Question: Fastq.gz files taking a long time to upload
1
gravatar for angela.m.mitchell
20 months ago by
angela.m.mitchell10 wrote:

Hello,

I am trying to upload two files, both of which are about 2.4 GB and in fastq.gz format. I used the ftp method, as direct upload does not work for files >2 GB. I seemed to have added them to my history, but it has been 7 hours, and they remain yellow with circling dots. Does it typically take this long just to upload the files? Is there something I should be doing to speed the process? I had previously done much smaller files as direct uploads (each being a few MB at most), and they were very fast (within a minute or so).

Thanks!

Angie

ADD COMMENTlink modified 20 months ago by Jennifer Hillman Jackson25k • written 20 months ago by angela.m.mitchell10
0
gravatar for Jennifer Hillman Jackson
20 months ago by
United States
Jennifer Hillman Jackson25k wrote:

Hello,

This seems too long. You refreshed the history panel (double circle icon at the top)?

Where are you using Galaxy? At http://usegalaxy.org, a local, a cloudman, some other public URL?

Thanks, Jen, Galaxy team

ADD COMMENTlink modified 20 months ago • written 20 months ago by Jennifer Hillman Jackson25k

Hello,

I am just using the galaxy main server at usegalaxy.orghttp://usegalaxy.org. I see that it says something about metadata is being auto detected.

Thanks!

On Oct 4, 2016, at 5:29 PM, Jennifer Hillman Jackson on Galaxy Biostar <notifications@biostars.org<a rel="nofollow" href="mailto:notifications@biostars.org">notifications@biostars.org> wrote:

Activity on a post you are following on Galaxy Biostarhttp://biostar.usegalaxy.org

User Jennifer Hillman Jacksonhttp://biostar.usegalaxy.org/u/254/ wrote Answer: Fastq.gz files taking a long time to uploadhttp://biostar.usegalaxy.org/p/19920/#19923:

Hello,

This seems too long.

Where are you using Galaxy? At http://usegalaxy.org, a local, a cloudman, some other public URL?

Thanks, Jen, Galaxy team

ADD REPLYlink written 20 months ago by angela.m.mitchell10

This still seems to be taking too long for fastq.qz datasets of this size. That said, the server was very busy/slow today. I would allow it to run. If it is still a problem tomorrow, write back and I'll go into your account and review anything odd with our administrator. Please leave the history items undeleted so we can look at them, even if they are still yellow/executing.

Sorry for the problems!

ADD REPLYlink modified 20 months ago • written 20 months ago by Jennifer Hillman Jackson25k

Will do--thank you!

On Oct 4, 2016, at 5:40 PM, Jennifer Hillman Jackson on Galaxy Biostar <notifications@biostars.org<a rel="nofollow" href="mailto:notifications@biostars.org">notifications@biostars.org> wrote:

Activity on a post you are following on Galaxy Biostarhttp://biostar.usegalaxy.org

User Jennifer Hillman Jacksonhttp://biostar.usegalaxy.org/u/254/ wrote Comment: Fastq.gz files taking a long time to uploadhttp://biostar.usegalaxy.org/p/19920/#19927:

This still seems to be taking to long. The server was very busy/slow today. I would allow it to run. If it is still a problem tomorrow, write back and I'll go into your account and review anything odd with our administrator. Please leave the history items undeleted so we can look at them.

Sorry for the problems!

ADD REPLYlink written 20 months ago by angela.m.mitchell10

Hello,

It looks like the upload is still stuck in the same place it was yesterday. Is there anything I should try?

Thanks!

Angela M. Mitchell, M.S. University of Colorado, Denver Department of Immunology and Microbiology Angela.M.Mitchell@ucdenver.eduAngela.M.Mitchell@ucdenver.edu 303-724-7216

From: Jennifer Hillman Jackson on Galaxy Biostar [mailto:notifications@biostars.org] Sent: Tuesday, October 04, 2016 5:40 PM To: Mitchell, Angela M (IMMU) Subject: [galaxy-biostar] Fastq.gz files taking a long time to upload

Activity on a post you are following on Galaxy Biostarhttp://biostar.usegalaxy.org

User Jennifer Hillman Jacksonhttp://biostar.usegalaxy.org/u/254/ wrote Comment: Fastq.gz files taking a long time to uploadhttp://biostar.usegalaxy.org/p/19920/#19927:

This still seems to be taking to long. The server was very busy/slow today. I would allow it to run. If it is still a problem tomorrow, write back and I'll go into your account and review anything odd with our administrator. Please leave the history items undeleted so we can look at them.

Sorry for the problems!

ADD REPLYlink written 20 months ago by angela.m.mitchell10

I am looking at your account right now, feedback soon. Jen

ADD REPLYlink written 20 months ago by Jennifer Hillman Jackson25k

Hello - There are two issues.

1) The compression on the data is not interpreted as valid by the server. It could be a true compression problem, or the specific flavor of compression is not recognized for some reason. Try uncompressing the data locally then loading it that way. If it will not uncompress locally, then the files are corrupt - you'll need new versions from the source.

2) These upload jobs failed due to the compression problem above, but did not get marked as red failed jobs in the UI. Our admin is looking into why this happened.

ADD REPLYlink written 20 months ago by Jennifer Hillman Jackson25k

Thank you for your help. I am uploading the unzipped files now via ftp. I’ll let you know if I have further issues.

Thanks!

Angela M. Mitchell, M.S. University of Colorado, Denver Department of Immunology and Microbiology Angela.M.Mitchell@ucdenver.eduAngela.M.Mitchell@ucdenver.edu 303-724-7216

From: Jennifer Hillman Jackson on Galaxy Biostar [mailto:notifications@biostars.org] Sent: Wednesday, October 05, 2016 11:42 AM To: Mitchell, Angela M (IMMU) Subject: [galaxy-biostar] Fastq.gz files taking a long time to upload

Activity on a post you are following on Galaxy Biostarhttp://biostar.usegalaxy.org

User Jennifer Hillman Jacksonhttp://biostar.usegalaxy.org/u/254/ wrote Comment: Fastq.gz files taking a long time to uploadhttp://biostar.usegalaxy.org/p/19920/#19948:

Hello - There are two issues.

1) The compression on the data is not interpreted as valid by the server. It could be a true compression problem, or the specific flavor of compression is not recognized for some reason. Try uncompressing the data locally then loading it that way. If it will not uncompress locally, then the files are corrupt - you'll need new versions from the source.

2) These upload jobs failed due to the compression problem above, but did not get marked as red failed jobs in the UI. Our admin is looking into why this happened.

ADD REPLYlink written 20 months ago by angela.m.mitchell10

I just wanted to give an update. I uncompressed the files and used ftp to upload fastq (rather than fastq.gz) files, which appeared to have worked.

Thanks for all of your help!

Angela M. Mitchell, M.S. University of Colorado, Denver Department of Immunology and Microbiology Angela.M.Mitchell@ucdenver.eduAngela.M.Mitchell@ucdenver.edu 303-724-7216

From: Jennifer Hillman Jackson on Galaxy Biostar [mailto:notifications@biostars.org] Sent: Wednesday, October 05, 2016 11:42 AM To: Mitchell, Angela M (IMMU) Subject: [galaxy-biostar] Fastq.gz files taking a long time to upload

Activity on a post you are following on Galaxy Biostarhttp://biostar.usegalaxy.org

User Jennifer Hillman Jacksonhttp://biostar.usegalaxy.org/u/254/ wrote Comment: Fastq.gz files taking a long time to uploadhttp://biostar.usegalaxy.org/p/19920/#19948:

Hello - There are two issues.

1) The compression on the data is not interpreted as valid by the server. It could be a true compression problem, or the specific flavor of compression is not recognized for some reason. Try uncompressing the data locally then loading it that way. If it will not uncompress locally, then the files are corrupt - you'll need new versions from the source.

2) These upload jobs failed due to the compression problem above, but did not get marked as red failed jobs in the UI. Our admin is looking into why this happened.

ADD REPLYlink written 20 months ago by angela.m.mitchell10
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 87 users visited in the last hour