Trinity "Exceeded step memory limit" error?

Question: Trinity "Exceeded step memory limit" error?

6 months ago by

katharos • 10 wrote:

Hi,

I'm trying to assemble a de novo transcriptome using ~270 million paired end reads in Trinity on Galaxy main. The job appears to have finished, although when I look at the details in my history it says

"workdir is /pylon5/mc48nsp/xcgalaxy/main/staging/19349785/working Running Trinity from /local/2956986 slurmstepd: error: Exceeded step memory limit at some point."

However the assembly seems reasonable: ~223,786 transcripts is not surprising given the number of reads that went into it. Did the tool stop prematurely, or is this result ok?

Cheers and thanks, Katie

failure memory trinity usegalaxy.org assembly • 370 views

ADD COMMENT • link •

modified 4 months ago by Jennifer Hillman Jackson ♦ 25k • written 6 months ago by katharos • 10

Sounds like something similar just happened to me.

I tried running Trinity on a dataset with ~38 million reads. After 18 hours, it looked like it finished, everything was green, but the following message was in the history.

Running Trinity from /local/2985876 slurmstepd: error: Exceeded step memory limit at some point. slurmstepd: error: Exceeded job memory limit at some point.

Here's the published history. I emailed it to galaxy-bugs as well.

https://usegalaxy.org/u/coyk/h/unnamed-history

I tried to share this with the galaxy-bugs email instead of publishing the history but I got an error saying this wasn't a valid user.

Coyk

ADD REPLY • link written 6 months ago by dollykc • 10

Hello,

This type of error does indicate that the job ran out of memory during execution. You might want to try a rerun to see if the same results are produced. But also please consider sending the data in for review. The Galaxy pre-release for 18.05 is currently installed at Galaxy Main https://usegalaxy.org and we are looking at reported odd behavior. This message would not be expected if the result data is in a success state (green in color).

So, if the dataset is failed (red) a bug report can be sent in. If it is a putative "success" (green) with reported problems in the log, a share link to the history can be sent in instead. Please include the problematic Trinity dataset number and a link to this post for reference. Be sure to leave all input and output datasets active (undelete if necessary).

Support FAQs: https://galaxyproject.org/support/

Reporting Usage Issues or Software bugs https://galaxyproject.org/issues/

Thanks! Jen, Galaxy team

ADD REPLY • link written 6 months ago by Jennifer Hillman Jackson ♦ 25k

Thanks for the suggestion Jen!

It did go green, so I was a bit confused! I've just sent an email about it.

Cheers and thanks, Katie

ADD REPLY • link written 6 months ago by kmarshall • 0

4 months ago by

Jennifer Hillman Jackson ♦ 25k

United States

Jennifer Hillman Jackson ♦ 25k wrote:

Update: Trinity jobs that fail due to running out of memory resources can end up green in the history with a certain error message others have noted, specifically:

Exceeded step memory limit at some point.

We use a special cluster for these jobs and this is how they result when the job exceeds resources -- they do not turn red unless there is a true error, usually an input content issue. Some results may be returned but it is likely to be incomplete, or you should at least be aware that it might be.

Running more QA on your inputs can sometimes help to avoid this type of problem (https://galaxyproject.org/support/tool-error/#type-exceeds-memory-allocation), as can downsampling the input data to reduce the overall size of the job.

If you do need to run an entire large dataset through this tool, then moving to your own Galaxy where you can control the resources is the alternative, please see: https://galaxyproject.github.io/ && https://galaxyproject.org/choices/.

Thanks to all who shared histories and/or reported the issue. Jen, Galaxy team

ADD COMMENT • link modified 4 months ago • written 4 months ago by Jennifer Hillman Jackson ♦ 25k

Similar posts • Search »