Question: Files uploaded via FTP are disappearing
0
gravatar for haley.glass
20 days ago by
haley.glass0 wrote:

Hello,

I have been having issues with the FTP server. I am able to successfully upload my fasta files to the Galaxy FTP server via FileZilla on my Mac. I am then able to upload the files into Galaxy shortly after they are on the FTP server. However, if I go to Get Data -> Upload File -> Choose FTP File all of the files I uploaded are gone. They are also nonexistent in FileZilla as well. This is very frustrating because it takes a whole day to upload one file to the FTP server. What can i do to make sure this doesn't happen anymore?

Thanks!

ADD COMMENTlink modified 13 days ago by gopaldhas670 • written 20 days ago by haley.glass0

Hi Jen,

I uploaded about 10 files to the FTP server and then uploaded 4 of them to my Galaxy history. When I first went into the Upload File tool, all of the files were visible in the list. After that, when I went to upload the remaining files the next day, nothing was there anymore in either the tool or FileZilla.

ADD REPLYlink written 20 days ago by haley.glass0

FTP uploaded datasets remain in the server's FTP staging area for 3 days (at https://usegalaxy.org - other servers may have a different time set - you'd need to check with that server's admins/support team to find out). Could it be that the missing datasets were uploaded for longer than that when you went back the second day?

If data takes days to upload due to the connection speed, it would be a good strategy to FTP a few, move the data to a history, FTP a few more, move the data to a history, repeat as needed. You also might want to consider loading .gz compressed fastq data instead of uncompressed. You can always uncompressed or recompressed fastq dataset once in the history. https://galaxyproject.org/support/compressed-fastq/

I'll also run a quick test to make sure more is not going on and will post back once done.

ADD REPLYlink written 20 days ago by Jennifer Hillman Jackson24k

Update: I wasn't able to reproduce the problem with FTP upload files disappearing when logging out/logging in again (of Galaxy). I even had two that were a day or so old that were still in the list even after FTP loading more data, and loading some of that data into a history (but not all files).

From this test, it seems that your data left the FTP staging area after the three-day expiration.

ADD REPLYlink written 20 days ago by Jennifer Hillman Jackson24k
0
gravatar for Jennifer Hillman Jackson
20 days ago by
United States
Jennifer Hillman Jackson24k wrote:

Hello,

Does this mean that you moved the successfully FTP loaded data using the Upload tool into your history?

I am then able to upload the files into Galaxy shortly after they are on the FTP server.

And this means that when you go back to the Upload tool, the files are no longer listed? Both in Galaxy and in your FTP client server-side file listing?

However, if I go to Get Data -> Upload File -> Choose FTP File all of the files I uploaded are gone.

If true, then this can be expected. FTP uploaded data remains in the staging area for 3 days, then is removed server-side. It will not appear in the FTP server file listing (FTP client or Upload tool).

If this is not true, please explain the steps you are doing, in order, with a few more details so we can help troubleshoot more.

Thanks! Jen, Galaxy team

ADD COMMENTlink modified 20 days ago • written 20 days ago by Jennifer Hillman Jackson24k
0
gravatar for haley.glass
20 days ago by
haley.glass0 wrote:

I didn't know that there was a 3 day expiration, so that must have been what happened. I will be sure to add the data to the history right away in the future. Thanks for your help!

ADD COMMENTlink written 20 days ago by haley.glass0
0
gravatar for gopaldhas67
13 days ago by
gopaldhas670 wrote:

Move the data of the updated file FTP to the history so that it easy to upload... is it possible it might be replaced by another file or may get removed... https://babasupport.org/printer/epson-customer-service-number/588

ADD COMMENTlink written 13 days ago by gopaldhas670
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 67 users visited in the last hour