RNA-star is a compute-intensive tool. If your data are very large, the job may really be too large to run at a public server like Galaxy Main https://usegalaxy.org. You might try mapping the data with an alternative tool, such as HISAT2 to see if that works. Jobs that are actually too large to run at public servers can be run on your own Galaxy that has been given sufficient resources. The FAQ below covers the options.
However, this type of memory error can also be triggered by problematic input content or formatting, so be sure to check for those types of problems first. Common causes and solution for data/input issues are also covered in the FAQ below.
Galaxy FAQs: https://galaxyproject.org/support/
- My job ended with an error. What can I do? https://galaxyproject.org/support/tool-error/
Thanks! Jen, Galaxy team