Uploading small files are queued for hours

Question: Uploading small files are queued for hours

4.2 years ago by

Australia

Hi !

I've been trying to upload files via the FTP server 9.fna and .gff file). Every time it tells me that the server closed without sending data despite I'm connected in FileZila.

Because my files are small, I used the "Choose file" options. My documents are queued for hours now. Yesterday, it didn't upload at all (queued all day).

Is there a problem with Galaxy or am I doing something wrong ? It worked fine previously, I really don't know what's happening.

Please help, thanks

Morgane

queued uploading small files • 2.8k views

ADD COMMENT • link •

modified 4.2 years ago • written 4.2 years ago by morgane.moreau.info • 30

Hey Morgane,

Regarding the ftp transfer, I had the same problems with FileZilla in different computers even with the timeout option set to infinity. Then I just switched to WinSCP (http://winscp.net/eng/index.php), and now it works smoothly ;)
You should definitely give it a try.

ADD REPLY • link written 4.2 years ago by Guilherme Dias • 110

Thanks for your feedback !

Jen, yes I'm using the public main galaxy. Most of the time it doesn't work. I'm starting to think that there is a lag between upload to the server and being able to upload it into a history.

I will give a try with WinSCP if I run out of patience :). Thanks Guilherme.

ADD REPLY • link modified 4.2 years ago • written 4.2 years ago by morgane.moreau.info • 30

4.2 years ago by

Nate Coraor ♦ 3.2k

United States

Nate Coraor ♦ 3.2k wrote:

When uploading via the Upload Tool's "Upload File" button, are your upload jobs changing form the uploading (blue) state to the new/queued state (gray)? If they stay blue, most likely the transfer is being interrupted, and the request/console log in your browser's developer tools should have a bit more information on that.

I'd suggest using the new upload functionality, which includes a progress bar. You can access this using the small icon with a disk and up arrow located at the top right corner of the tools panel.

ADD COMMENT • link written 4.2 years ago by Nate Coraor ♦ 3.2k

Thanks Nate.

I tried to use the new upload functionality but it returned the same error. (server closed...).

ADD REPLY • link written 4.2 years ago by morgane.moreau.info • 30

Where do you see the "server closed" message, and is there any additional detail shown there? Can you provide a cut and paste or screenshot of the full error? Are your upload jobs changing form the uploading (blue) state to the new/queued state (gray)?

ADD REPLY • link written 4.2 years ago by Nate Coraor ♦ 3.2k

When I use the FTP upload (old school way) once I click on execute, the middle window (where is normally displays job has been successfully added to the queue) in between the Galaxy tools window on the left and the history window on the right, had a smiley face down :( and underneath it's written server closed without sending data. And the job never got uploaded into the history. I try again and again until it works.

If I do that in the new upload one I get an error message in red (I forgot what it said exactly tho, something about the server too).

For another file that I downloaded from a tut via upload file (old school way too), it stayed blue forever. I gave up on that one, followed the tut with my own samples. Although, that bugged as well but that 's another problem...

Hope that help, let me know if something is not clear.

ADD REPLY • link written 4.2 years ago by morgane.moreau.info • 30

Thanks for the details, this is really helpful. If you could try this in a different browser (I think you're using Chrome?) next time and see if it has the same problems, I'd appreciate it. This seems likely to be a connection problem between you and the Galaxy servers - I am not seeing anything in our logs from your address indicating dropped connections, so I am not sure whether your requests ever made it.

Some details such as the IP you're connecting from and the exact dates/times you receive this error would help us check the logs at the exact moments you ran in to the errors. You can send that information privately to galaxy-bugs@bx.psu.edu

Jobs that stay blue forever indicate that an upload (via the browser) was interrupted prior to its completion. This is why I recommend the new upload (it will soon become the default method), because the progress bar should indicate if your upload has stopped or failed.

ADD REPLY • link written 4.2 years ago by Nate Coraor ♦ 3.2k

Hi Nate,

It happened again ealier today so I sent this info to the address you gave me. It just happened again at 2.12 pm if you want to ave a look.

ADD REPLY • link written 4.2 years ago by morgane.moreau.info • 30

Hey Nate,

I just had the same problem as Morgane. The files are uploading (blue) for a long time and they never actually finish. I tried with the new upload functionality but it returns an error (Upload request failed). Ftp upload seems to work fine.

(importing EBI SRA files is also working fine)
*updating: Ftp transfer finishes and the files are listed when I click in "Upload file", but when I try to import them to the history the screen refreshes and an error message appears, something like: "The server finished the connection without sending the data".
I tried this in Chrome and IE but the error persists.

ADD REPLY • link modified 4.2 years ago • written 4.2 years ago by Guilherme Dias • 110

This was broken over the weekend, after our network outage. Unfortunately we didn't catch it until this morning. Is it still happening?

ADD REPLY • link written 4.2 years ago by Nate Coraor ♦ 3.2k

Now it's working fine. Thanks!

ADD REPLY • link written 4.2 years ago by Guilherme Dias • 110

4.2 years ago by

Jennifer Hillman Jackson ♦ 25k

United States

Jennifer Hillman Jackson ♦ 25k wrote:

Hi Morgane,

Upload and FTP connections should be fine at http://usegalaxy.org, the public Main Galaxy instance.

Are you using Galaxy somewhere else?

Best, Jen, Galaxy team

ADD COMMENT • link written 4.2 years ago by Jennifer Hillman Jackson ♦ 25k

I'm having a similar problem with jobs being stuck in the queue using the choose file option at the main public Galaxy. They've been there for nearly a day.

ADD REPLY • link written 4.2 years ago by Katherine.Holman • 0

Katherine, Are your uploads blue or gray? If blue, can you retry uploading using the new upload functionality as suggested in my reply? That will show a progress bar so you can see whether or not the files are actually being fully transmitted to the Galaxy servers.

How big are the files you are trying to upload?

Morgane, no, there is nothing going on with the Galaxy server right now, or anything related to this Saturday's scheduled downtime, that should prevent you from uploading files.

There was a bug with the workflow editor since yesterday that was fixed today.

Mapping jobs are going to be queued for a while - that is normal because the load on that cluster is high. You may want to try the new option for submitting to the Stampede cluster, which is available via a job resource selector at the bottom of the tool form on mapping tools.

ADD REPLY • link modified 4.2 years ago • written 4.2 years ago by Nate Coraor ♦ 3.2k

Thanks for this advises and providing alternatives :)

The Stampede test/development works fast. The normal one gets queued as well. Is the test one ok to use?

ADD REPLY • link written 4.2 years ago by morgane.moreau.info • 30

The development queues only have a 30 minute walltime, so you will not be able to do much real work there. Here's the information on the differences between the choices: https://wiki.galaxyproject.org/Main#Beta:_Job_execution_on_Stampede

The wait on Stampede's normal queue may be > 0, but it's probably still going to be less than the regular Galaxy cluster queue at this point.

ADD REPLY • link written 4.2 years ago by Nate Coraor ♦ 3.2k

Same as me... I start to wonder if there is not a problem with galaxy, maybe due to the maintenance planned on the week end ?? I don't know... I've been playing with Galaxy for about two weeks and now I want to redo the same things in a new (clean) workflow to be able to extract it and nothing works anymore... I finally got my files loaded but now the mapping doesn't work anymore..can't remove the duplicates either in another workflow. All my workflows are stuck :( I feel I'm going nowhere and loosing time...At least I'm not the only one this thing happens, so thanks for sharing...

ADD REPLY • link modified 4.2 years ago • written 4.2 years ago by morgane.moreau.info • 30

Please log in to add an answer.

Similar posts • Search »