Fastq.gz files in shared Data Libraries

Question: Fastq.gz files in shared Data Libraries

3.4 years ago by

United Kingdom

rudigarude • 0 wrote:

Hi,

I have a production local version of galaxy running at our institution. After our illumina spits out the bcl files we run bcl2fastq and put it into our storage archive. What I want to be able to do is add these raw fastq.gz files as data libraries into galaxy without copying into galaxy. So far this works fine by mounting the storage archive in galaxy and adding the directories as datasets from filesystem paths.

The issue I've got is that bcl2fastq stores them as fastq.gz files and adding these as a data library in this way means that the autodetect file type assigns them as 'data'. I don't really want galaxy to attempt to uncompress anything like it does if you do a standard upload, and a lot of tools can work with fastq.gz files (e.g. fastqc and trinity). However since the files are assigned as format 'data', nothing works with them and there is no fastq.gz file type in the datatypes_conf.xml.

Has anyone dealt with a similar issue with working with fastq.gz's in data libraries or have any advice how to integrate these more easily?

Many thanks,

Martin

data libraries galaxy gunzip fastq.gz • 1.2k views

ADD COMMENT • link •

modified 3.4 years ago by Hotz, Hans-Rudolf • 1.8k • written 3.4 years ago by rudigarude • 0

3.4 years ago by

Hotz, Hans-Rudolf • 1.8k

Switzerland

Hotz, Hans-Rudolf • 1.8k wrote:

Hi Martin

Have you tried setting "File Format:" to 'fastq' or fastqsanger' instead of 'Auto-detect'?

We upload gzipped fastq files like this (with 'Upload files from filesystem paths', and 'Link to files without copying into Galaxy'). The files will then apear as "Data type: fastq" and "Peek: gzipped file". And we can use them in tools accepting gzipped fastq files.

Hope this helps, Hans-Rudolf

ADD COMMENT • link written 3.4 years ago by Hotz, Hans-Rudolf • 1.8k

Hi Hans-Rudolf,

I did try that but it didn't quite have the desired effect. It worked for programs that work with fastq.gz but then fools users into thinking they'll just work with programs that only accept fastq files.

So, having a bit of a play with it I found the 'archive_datatypes' data source in the tool shed that adds the fastq.gz datatype. So that allows the auto detect to work fine. I've now written a wrapper for gunzip so that these can then be uncompressed for programs that only work with uncompressed fastq files, and then added fastq.gz as an accepted file type for the few programs that work with the compressed version.

I think that gives a half decent solution to my problem at this point.

Cheers,

Martin

ADD REPLY • link written 3.4 years ago by rudigarude • 0

We are basically doing the same thing on our instance. The issue is ofcourse that with each tool update you will have to add the support for fastq.gz again.

I'm not sure if it helps, but maybe upvoting this ticket might highlight the need for this: https://trello.com/c/3RkTDnIn/345-666-support-gzipped-gz-compressed-versions-of-standard-datatypes

ADD REPLY • link written 3.4 years ago by Jelle Scholtalbers • 360

Great idea, I've upvoted.

ADD REPLY • link written 3.4 years ago by rudigarude • 0

Similar posts • Search »