Errors at http://usegalaxy.org - Follow for details and updates

Question: Errors at http://usegalaxy.org - Follow for details and updates

2.0 years ago by

l.harris2 • 10

l.harris2 • 10 wrote:

Hi there,

I came across the following errow when uploading bam files (1gb - by FTP or upload tool).

I have not had this problem before. I was thinking it might have to do with the server maintence that just occured?

Kind regards, Lachlan

upload job usegalaxy.org ftp dataset • 1.7k views

ADD COMMENT • link •

modified 24 months ago by Jennifer Hillman Jackson ♦ 25k • written 2.0 years ago by l.harris2 • 10

Hello,

Correction: Data will load by FTP, but the jobs to move data into the history are delayed a bit longer than normal from executing (regular upload + the second upload step of FTP). This may be due to high server load. More feedback soon.

Thanks for reporting the problem, Jen, Galaxy team

ADD REPLY • link modified 24 months ago • written 24 months ago by Jennifer Hillman Jackson ♦ 25k

Update: Many job types will have execution problems right now. Our administrator and team are working to resolve this with priority. More updates to follow.

ADD REPLY • link modified 24 months ago • written 24 months ago by Jennifer Hillman Jackson ♦ 25k

24 months ago by

Nate Coraor ♦ 3.2k

United States

Nate Coraor ♦ 3.2k wrote:

Hi all,

Just to provide a bit of an update, we have been planning a storage system move from Corral 2 to Corral 3 (it is a storage service provided by our host, the Texas Advanced Computing Center (TACC)). The move was originally scheduled to take place about 2 weeks ago but not all data had copied yet and so we delayed. It was not supposed to happen last week either, but Corral 2 ran out of space and took usegalaxy.org down over the previous weekend, and so while it was already down we decided to finish the move.

Corral 3 is a much improved design over Corral 2 and supports higher data transfer speeds and overall capacity. At times in the past, usegalaxy.org performance could be affected by jobs running on other TACC systems. With Corral 3, this should be much less of an issue.

All data should be available now and there should not be any job errors related to the move from Corral 2 to Corral 3. If you encounter such errors, please report them.

Unfortunately, however, we seem to have run into a performance problem specific to Galaxy's VMs that is taking a lot of work/time to track down. Our gracious hosts at TACC have been hard at work on the problem, but at the moment, we are encountering extremely long (and widely variable) job wait times. The problem is not that jobs are actually taking longer to execute, it's that Galaxy's ability to start and finish them is severely delayed.

I apologize for the difficulties, and thank you for sticking with us. I hope this will be resolved by the end of the week and we'll have an even better usegalaxy.org than before.

--nate

ADD COMMENT • link modified 24 months ago • written 24 months ago by Nate Coraor ♦ 3.2k

24 months ago by

Jennifer Hillman Jackson ♦ 25k

United States

Jennifer Hillman Jackson ♦ 25k wrote:

Hello Lachlan,

The latest status for Galaxy Main (http://usegalaxy.org). In short, it is ready to use, just be aware of the following:

The FTP server is up now and accepting data transfers. Moving data into the history will queue a job. It will take a while to process as there is a backlog of jobs to process. Allow the job to execute - meaning, leave the job queued until it completes (do not stop/restart - this will place the job at the end of the queue again, further extending wait time).
Using the Upload tool directly will also queue a job. Delays are also to be expected with this job type.
Jobs starting using existing datasets are running but also with delays. There is a very large data migration in progress. The new banner at Galaxy Main http://usegalaxy.org explains:

Some Galaxy datasets created within the last week are temporarily unavailable while being copied to a new storage system. They will gradually become available as the copy progresses.
When the banner is removed, all data will be available.
If a job should fail during this time with an error, it is likely that the input datasets were not ready to use yet (still in the process of transfer). Wait for the banner to clear, then re-run the job. Error messages related to this can vary by tool, but are generally cluster related and/or data permission related. Click on the bug icon to view the entire error to read an error message in full. There is no need to submit bug reports for these, as we are aware of the problem and are sending out this same info for those that are reported.
If you are not sure if your error is related or not to the data transfer after reviewing, it is ok to send in a bug report and we can confirm and/or provide feedback about the issue, as normally, if unrelated.

Thanks for reporting the problem and for your patience. We believe these maintenance items will greatly improve the performance of the public Galaxy Main instance for all going forward.

Jen, Galaxy team

ADD COMMENT • link written 24 months ago by Jennifer Hillman Jackson ♦ 25k

UPDATE: Upload is failing at this time. A correction is being worked on. More once fixed. Jen

ADD REPLY • link written 24 months ago by Jennifer Hillman Jackson ♦ 25k

Thanks Jen, I appreciate the updates.

ADD REPLY • link written 24 months ago by l.harris2 • 10

Thumbs up for each status update. Good luck with fixes.

ADD REPLY • link written 24 months ago by raitis • 20

UPDATE2: Upload is now working for all tested datatypes. Please also try and report if any are problematic. Jen

Related question: How to know when migrated datasets are ready for use?

1 - Test rerun jobs periodically to see if available. Many are now.

2 - Wait until the banner on Main is removed. This will indicate that the maintenance cycle is complete. Errors that occur after that can be reported.

ADD REPLY • link written 24 months ago by Jennifer Hillman Jackson ♦ 25k

Hi Jen, Everything working as should now. Thank you for all your help. Galaxy is great!

Kind regards, Lachlan

ADD REPLY • link written 24 months ago by l.harris2 • 10

UPDATE3: The banner is removed.

If you are having job issue still, please see the details in this post for troubleshooting: https://biostar.usegalaxy.org/p/20791/#20808

If anyone has a new question related to this, please create a new post. Linking in this post is OK to give context. Please be specific about your issue(s). We may ask for a bug report if the problem cannot be solved here using general information.

Thanks! Jen, Galaxy team

ADD REPLY • link modified 24 months ago • written 24 months ago by Jennifer Hillman Jackson ♦ 25k

UPDATE4: Ongoing issues for certain functions. Details in several recent posts, many pointing back to this thread.

We'll continue to update this thread as status changes.

ADD REPLY • link modified 24 months ago • written 24 months ago by Jennifer Hillman Jackson ♦ 25k

UPDATE5: The data migration is completed, yet other issues continue at http://usegalaxy.org.

Options:

Rerun the jobs now. Many, if not most, will succeed.
Choose to wait until the server issues are fully resolved.
Use Galaxy another way. https://wiki.galaxyproject.org/BigPicture/Choices

Our thanks again to the community for their patience during this time. - Jen and the Galaxy team

ADD REPLY • link modified 24 months ago • written 24 months ago by Jennifer Hillman Jackson ♦ 25k

Please log in to add an answer.

Similar posts • Search »