Question: Problems with the Upload function at https://usegalaxy.org -- Workaround included, follow for updates
2
gravatar for Ath.T
11 days ago by
Ath.T20
Ath.T20 wrote:

Hello

I am new in Galaxy usegalaxy.org). I am trying to load fastq.gz files in galaxy from FTP, but it is very slow. The job still yellow with circling dots from 6 hours until now and I am still waiting for completing the loading. What should I do? Is it okay to take long time so I have to wait? Should I reload the files one by one, as the total size of the files is 60 GB? Do you have any suggestions?

Thanks

ADD COMMENTlink modified 11 days ago by Jennifer Hillman Jackson25k • written 11 days ago by Ath.T20
1
gravatar for Jennifer Hillman Jackson
11 days ago by
United States
Jennifer Hillman Jackson25k wrote:

Hello,

One of our web servers hosting Galaxy Main https://usegalaxy.org is under a very high load right now. Our administrators know what the issue is and will be correcting it with priority, yet it may be still a day or two out before fully resolved. Our team will write back with an update once things are back to normal.

Upload is impacted by this issue, resulting in unusually long execution times. Some jobs will never complete and/or eventually fail. The timing for Upload depends on a few factors, so the queued (grey) and execution (yellow) states vary in how long they take to run even under normal conditions. However, it would unusual for this kind of job to take over an hour even with a very large dataset.

The maximum Upload size for browsed files is 2 GB. For URL or FTP Upload data the maximum file size is 50 GB for most data and around 35 GB for BAM data. The initial FTP process itself can take much longer (depends on the connection and the data size), but once transferring data with the Upload tool into the history, under an hour is the expectation (and is usually much quicker). URL upload times can also vary based on file size/connection speed with the data source.

Workaround for now (rare advice! most reruns are discouraged, and that is still true for other tools): We suggest rerunning the Upload jobs. Kill the prior ones (permanently delete). If a job takes over an hour, then you probably hit the web server with the load problem, and you'll need to try again. Web services are randomly assigned -- so this may take a few attempts.

Thanks for posting the problem here and I hope the workaround helps to get your data loaded. I am going to modify the question title so others can find it and follow the ticket for updates.

Jen, Galaxy team

ADD COMMENTlink modified 11 days ago • written 11 days ago by Jennifer Hillman Jackson25k

Thanks Jen, I am having errors as follows - having run NGS: Deep Tools computeMatrix - I try to plotProfile (which worked a few times), but plotting failed & getting this error, for ex. in the failed job where plot image view normally appears - can you tell if it the high-demand server problem you referenced above in reply to upload hang, or does it mean I have a problem w/ one of my files or settings? ==========paste from plotProfiel image "info" window============ Fatal error: Exit code 1 () Traceback (most recent call last): File "/cvmfs/main.galaxyproject.org/deps/_conda/envs/mulled-v1-0fa290085c742a3ffee6a142d1fce47c178e1c289ff6f0c66023b506f3377842/bin/plotProfile", line 11, in <module> main(args)

File

thanks, HWB

ADD REPLYlink written 11 days ago by bass0

Hi, Would you please send in an email to galaxy-bugs@lists.galaxyproject.org from your registered account email address at https://usegalaxy.org. If the error dataset is red, the bug icon can be clicked on. If green, create a share link to the history and note the datasets numbers with the problem in the email. With either, please include a link back to this post.

How-to: https://galaxyproject.org/support/#troubleshooting >> https://galaxyproject.org/issues/

Next time, it is best to post a new question if the problem is not the same as the original question. Sometimes it is hard to tell, so this one is Ok. We can troubleshoot through email. There are no known issues with this tool but there have been some new cluster issues and this might be related. Thanks for reporting the problem! - Jen

ADD REPLYlink written 10 days ago by Jennifer Hillman Jackson25k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 87 users visited in the last hour