12 weeks ago by
United States
Hello,
This solution will work for you case.
- Change the datatype of the EBI SRA fastq datasets to
fastcssanger.gz. Do this for all datasets loaded from this source that are in color-space. The assigned datatype can be modified on the Edit Attribute forms under the "Datatype" tab (reach these forms by clicking on the pencil icon, per-dataset). Be sure to save after each change. NOTE: fastqsanger.gz is different than fastqcssanger.gz, so double check for the "cs" in the naming when making the change.
- Go back to the Edit Attributes forms and under the "Convert" tab, convert compressed to uncompressed -- and be sure to save again. This made the Fastq Groomer tool run faster for me. Given the data size, it might be necessary - or you can try skipping to step 3. Be aware that this processing will take many hours to run. My tests with compressed inputs have been running for 12 hours, have not completed, and still have the potential to fail (for exceeding memory/time resources).
- Use the tool Fastq Groomer, select those
fastqcssanger or fastqcssanger.gz datasets, and set the option Input FASTQ quality scores type to Color Space Sanger. Leave the remaining options at default. The result will be data converted to and assigned to either the fastqsanger or fastqsanger.gz format/datatype. Both can be used as inputs with most, but not all, tools (some older tools still require uncompressed inputs, example: Tophat won't accept compressed fastq data when in a dataset collection).
- Uncompressed fastq data can always be converted back to compressed to save space. It is the combined functions of uncompressing, doing the data change from color to base space, then compressing again -- all with the Fastq Groomer tool in one step -- that may present resource problems. Doing the manipulations distinctly has a greater chance of success. But you can try both. Sicking with compressed data has advantages (will save quota space), although you can always permanently delete (purge) those intermediate datasets later to get the space used back.
Thanks for your patience during testing. I wanted to make sure this would actually still work, and with your exact given data.
Jen, Galaxy team
I'm running a test to see how the data downloads and what datatype Galaxy assigns it. Most data from this source, when imported using the Fastq > Send to Galaxy links, will be transformed to have Sanger Fastq +33 scaled quality scores (fastqsanger). My test history is here if curious (test downloads are still running right now): https://usegalaxy.org/u/jen/h/test-history-ebi-sra-solid-fastq
The data is actually is in a compressed format, meaning the datatype should be assigned a "[datatype].gz" format, not "[datatype]" but there is a bug with the tool requiring one to reassign the datatype before using downstream tools. In most cases, this means reassigning the datatype from fastqsanger to fastqsanger.gz. That bug and the workaround details are covered in this prior post: https://biostar.usegalaxy.org/p/28718/
But let's see how these tests download first, then I'll give definitive advice about how to proceed. Most of Galaxy's tools no longer work with color-space (solid) data directly anymore. But there are ways to transform from solid fastq to sanger fastq within Galaxy (with some information loss, unfortunately). If needed, we'll explain how to do that in the final reply.
To better understand different fastq types, review the help on the FastqGroomer and/or Manipulate Fastq tool forms.
Thanks! Jen, Galaxy team
Well, the downloaded data was given the datatype "fastqsanger" even though it is in color space (not base space). Color space data is not used much anymore and the datatype sniffer or possibly the EBI SRA tool itself needs to be tuned to handle this type of uploaded content.
I'm going to test a few more things to see what will transform it into base space reads with the proper quality score encoding with the fewest steps. Then will open a development ticket(s) to prevent this from happening going forward. I'll write back with a link to that and the best workaround for you to use now. Feedback will likely be on Tuesday unless I can get to it over the holiday weekend, or someone else on this forum helps you first.
Thanks for your patience, Jen
Update: Still working on this and may or may not be able to give you a solution that works at Galaxy Main https://usegalaxy.org. I'll know after the new tests I have going finish (the first round failed and those tools/wrappers are unlikely to be updated for this purpose).
Why is this a problem?
Galaxy Main no longer supports tools/indexes for color-space analysis. The technology that created this data is much older and tools are not processing it as expected (as originally). Converting to an "Illumina-like" format is not the best way to process the data anyway. It is better to stay in color-space.
That said, there are wrappers in the Galaxy Tool Shed https://usegalaxy.org/toolshed that do work with color-space data directly. These could be used in a local Galaxy. Some are older, so if you decide to try this, proceed with the expectation that some testing will be needed and that the tool wrapper authors may no longer be supporting it (meaning, no changes if buggy).
More feedback soon and thanks for confirming that the bug report sent in was yours. An error message produced by a known issue with the Get Data > EBI SRA tool is identical to the one you encountered -- but the solution (if one can be found) will be different.