Fastq.gz files taking a long time to upload

Question: Fastq.gz files taking a long time to upload

2.2 years ago by

Hello,

I am trying to upload two files, both of which are about 2.4 GB and in fastq.gz format. I used the ftp method, as direct upload does not work for files >2 GB. I seemed to have added them to my history, but it has been 7 hours, and they remain yellow with circling dots. Does it typically take this long just to upload the files? Is there something I should be doing to speed the process? I had previously done much smaller files as direct uploads (each being a few MB at most), and they were very fast (within a minute or so).

Thanks!

Angie

run-time upload ftp galaxy fastq.gz • 965 views

ADD COMMENT • link •

modified 2.2 years ago by Jennifer Hillman Jackson ♦ 25k • written 2.2 years ago by angela.m.mitchell • 10

2.2 years ago by

Jennifer Hillman Jackson ♦ 25k

United States

Jennifer Hillman Jackson ♦ 25k wrote:

Hello,

This seems too long. You refreshed the history panel (double circle icon at the top)?

Where are you using Galaxy? At http://usegalaxy.org, a local, a cloudman, some other public URL?

Thanks, Jen, Galaxy team

ADD COMMENT • link modified 2.2 years ago • written 2.2 years ago by Jennifer Hillman Jackson ♦ 25k

Hello,

I am just using the galaxy main server at usegalaxy.orghttp://usegalaxy.org. I see that it says something about metadata is being auto detected.

Thanks!

On Oct 4, 2016, at 5:29 PM, Jennifer Hillman Jackson on Galaxy Biostar <notifications@biostars.org<a rel="nofollow" href="mailto:notifications@biostars.org">notifications@biostars.org> wrote:

Activity on a post you are following on Galaxy Biostarhttp://biostar.usegalaxy.org

User Jennifer Hillman Jacksonhttp://biostar.usegalaxy.org/u/254/ wrote Answer: Fastq.gz files taking a long time to uploadhttp://biostar.usegalaxy.org/p/19920/#19923:

Hello,

This seems too long.

Where are you using Galaxy? At http://usegalaxy.org, a local, a cloudman, some other public URL?

Thanks, Jen, Galaxy team

ADD REPLY • link written 2.2 years ago by angela.m.mitchell • 10

This still seems to be taking too long for fastq.qz datasets of this size. That said, the server was very busy/slow today. I would allow it to run. If it is still a problem tomorrow, write back and I'll go into your account and review anything odd with our administrator. Please leave the history items undeleted so we can look at them, even if they are still yellow/executing.

Sorry for the problems!

ADD REPLY • link modified 2.2 years ago • written 2.2 years ago by Jennifer Hillman Jackson ♦ 25k

Will do--thank you!

On Oct 4, 2016, at 5:40 PM, Jennifer Hillman Jackson on Galaxy Biostar <notifications@biostars.org<a rel="nofollow" href="mailto:notifications@biostars.org">notifications@biostars.org> wrote:

Activity on a post you are following on Galaxy Biostarhttp://biostar.usegalaxy.org

User Jennifer Hillman Jacksonhttp://biostar.usegalaxy.org/u/254/ wrote Comment: Fastq.gz files taking a long time to uploadhttp://biostar.usegalaxy.org/p/19920/#19927:

This still seems to be taking to long. The server was very busy/slow today. I would allow it to run. If it is still a problem tomorrow, write back and I'll go into your account and review anything odd with our administrator. Please leave the history items undeleted so we can look at them.

Sorry for the problems!

ADD REPLY • link written 2.2 years ago by angela.m.mitchell • 10

Hello,

It looks like the upload is still stuck in the same place it was yesterday. Is there anything I should try?

Thanks!

Angela M. Mitchell, M.S. University of Colorado, Denver Department of Immunology and Microbiology Angela.M.Mitchell@ucdenver.eduAngela.M.Mitchell@ucdenver.edu 303-724-7216

From: Jennifer Hillman Jackson on Galaxy Biostar [mailto:notifications@biostars.org] Sent: Tuesday, October 04, 2016 5:40 PM To: Mitchell, Angela M (IMMU) Subject: [galaxy-biostar] Fastq.gz files taking a long time to upload

Activity on a post you are following on Galaxy Biostarhttp://biostar.usegalaxy.org

User Jennifer Hillman Jacksonhttp://biostar.usegalaxy.org/u/254/ wrote Comment: Fastq.gz files taking a long time to uploadhttp://biostar.usegalaxy.org/p/19920/#19927:

Sorry for the problems!

ADD REPLY • link written 2.2 years ago by angela.m.mitchell • 10

I am looking at your account right now, feedback soon. Jen

ADD REPLY • link written 2.1 years ago by Jennifer Hillman Jackson ♦ 25k

Hello - There are two issues.

1) The compression on the data is not interpreted as valid by the server. It could be a true compression problem, or the specific flavor of compression is not recognized for some reason. Try uncompressing the data locally then loading it that way. If it will not uncompress locally, then the files are corrupt - you'll need new versions from the source.

2) These upload jobs failed due to the compression problem above, but did not get marked as red failed jobs in the UI. Our admin is looking into why this happened.

ADD REPLY • link written 2.1 years ago by Jennifer Hillman Jackson ♦ 25k

Thank you for your help. I am uploading the unzipped files now via ftp. I’ll let you know if I have further issues.

Thanks!

Angela M. Mitchell, M.S. University of Colorado, Denver Department of Immunology and Microbiology Angela.M.Mitchell@ucdenver.eduAngela.M.Mitchell@ucdenver.edu 303-724-7216

From: Jennifer Hillman Jackson on Galaxy Biostar [mailto:notifications@biostars.org] Sent: Wednesday, October 05, 2016 11:42 AM To: Mitchell, Angela M (IMMU) Subject: [galaxy-biostar] Fastq.gz files taking a long time to upload

Activity on a post you are following on Galaxy Biostarhttp://biostar.usegalaxy.org

User Jennifer Hillman Jacksonhttp://biostar.usegalaxy.org/u/254/ wrote Comment: Fastq.gz files taking a long time to uploadhttp://biostar.usegalaxy.org/p/19920/#19948:

Hello - There are two issues.

2) These upload jobs failed due to the compression problem above, but did not get marked as red failed jobs in the UI. Our admin is looking into why this happened.

ADD REPLY • link written 2.1 years ago by angela.m.mitchell • 10

I just wanted to give an update. I uncompressed the files and used ftp to upload fastq (rather than fastq.gz) files, which appeared to have worked.

Thanks for all of your help!

Angela M. Mitchell, M.S. University of Colorado, Denver Department of Immunology and Microbiology Angela.M.Mitchell@ucdenver.eduAngela.M.Mitchell@ucdenver.edu 303-724-7216

Activity on a post you are following on Galaxy Biostarhttp://biostar.usegalaxy.org

User Jennifer Hillman Jacksonhttp://biostar.usegalaxy.org/u/254/ wrote Comment: Fastq.gz files taking a long time to uploadhttp://biostar.usegalaxy.org/p/19920/#19948:

Hello - There are two issues.

2) These upload jobs failed due to the compression problem above, but did not get marked as red failed jobs in the UI. Our admin is looking into why this happened.

ADD REPLY • link written 2.1 years ago by angela.m.mitchell • 10

Similar posts • Search »