Data import from EBI SRA to Galaxy does not work

Question: Data import from EBI SRA to Galaxy does not work

23 months ago by

Hungary

scholtz • 60 wrote:

Dear Galaxy Team,

There seems to be a persistent problem with importing fastq files from EBI SRA - I've been trying to upload data since about Christmas, and keep getting the same error message as reported by others. Although the alternative "Extract reads in FASTQ/A format from NCBI SRA" works, the fastq splitter is taking forever (memory problems?). I would appreciate if you could look into the problem - or should we try to contact the EBI staff? There is no announcement at their website about maintenance or anything...

Best regards,

Beata

ebi sra data import galaxy • 1.4k views

ADD COMMENT • link •

modified 21 months ago by Jennifer Hillman Jackson ♦ 25k • written 23 months ago by scholtz • 60

23 months ago by

Mo Heydarian ♦ 830

United States

Mo Heydarian ♦ 830 wrote:

Hello Beata,

I am also seeing an extended run time for FASTQ splitter.

An alternative to importing data with NCBI SRA Tools -> FASTQ splitter is to run NCBI SRA Tools with a list of SRR IDs, which will import paired end data as separate files corresponding to the forward and reverse reads. This way you won't have to run FASTQ splitter and your reads will be ready for analysis directly after import.

Note that when you use a list of SRR IDs to import your data with NCBI SRA Tools, every dataset provided will be imported in one list (collection). If you don't want to use collections, you should provide a list of individual SRR IDs to NCBI SRA Tools.

Thanks to Youri Hoogstrate for pointing out this usage of NCBI SRA Tools.

Hope this helps!

Cheers,

Mo Heydarian

ADD COMMENT • link written 23 months ago by Mo Heydarian ♦ 830

ADD REPLY • link written 22 months ago by Jennifer Hillman Jackson ♦ 25k

Dear All,

Some details to the workaround option suggested above (also, it seems to be faster than uploading from EBI): 1. Upload the SRR ID as a "list", (tabular file) using the NCBI SRA Tools - Extract reads in FASTQ/A format. Input type: List of SRA accession, one per line 2. Once uploaded, click on the "hide hidden" in the History panel 3. Unhide the forward/reverse data files

Proceed with the analysis as usual.

Hope this helps, and thanks a lot to the Galaxy team for pointing out this (lifesaving) option.

Beata

ADD REPLY • link written 21 months ago by scholtz • 60

23 months ago by

scholtz • 60

Hungary

scholtz • 60 wrote:

Hello Mo,

Thank you very much, I will try that for now. Also, this is the reponse I received from the EBI folks to my inquiry - the problem will hopefully be resolved by next week :)):

"On 2017.01.06., at 13:50, Marc Rossello mrosello@ebi.ac.uk wrote:

Dear colleagues,

The problems with the FTP servers that were limiting data downloads (or transferring files to Galaxy) have now largely been resolved and connections can be made and data downloaded. We are expecting the service to be stable by EOB today. It is still in an "at-risk" condition and the performance could be slower than usual until the beginning of next week.

Sorry for the inconvenience.

Regards, Marc"

ADD COMMENT • link written 23 months ago by scholtz • 60

22 months ago by

Jennifer Hillman Jackson ♦ 25k

United States

Jennifer Hillman Jackson ♦ 25k wrote:

UPDATE 2016-01-13

The technical team at EBI SRA sent an email this morning that states that the service is back up and running normally as of a few days ago. Please try your queries again and let us know if you have problems.

Thanks for your patience, Jen & the Galaxy team

ADD COMMENT • link written 22 months ago by Jennifer Hillman Jackson ♦ 25k

21 months ago by

Jennifer Hillman Jackson ♦ 25k

United States

Jennifer Hillman Jackson ♦ 25k wrote:

UPDATE 2016-02-13

Many issues have been reported again with this tool. We are in communication with the data provider and an update will posted back here from our team when more is known.

Workaround options:

Use the NCBI data retrieval tools when the data exists in both places.
Download the data locally, directly from the ENA using their methods (if possible, may take a few attempts). Then upload to Galaxy using FTP.

Thanks again to everyone for the patience! Jen, Galaxy team

ADD COMMENT • link modified 21 months ago • written 21 months ago by Jennifer Hillman Jackson ♦ 25k

Please log in to add an answer.

Similar posts • Search »