Question: Files uploaded via FTP are disappearing
0
haley.glass • 0 wrote:
Hello,
I have been having issues with the FTP server. I am able to successfully upload my fasta files to the Galaxy FTP server via FileZilla on my Mac. I am then able to upload the files into Galaxy shortly after they are on the FTP server. However, if I go to Get Data -> Upload File -> Choose FTP File all of the files I uploaded are gone. They are also nonexistent in FileZilla as well. This is very frustrating because it takes a whole day to upload one file to the FTP server. What can i do to make sure this doesn't happen anymore?
Thanks!
ADD COMMENT
• link
•
modified 26 days ago
by
sangitasekhavat2758 • 0
•
written
9 months ago by
haley.glass • 0
Hi Jen,
I uploaded about 10 files to the FTP server and then uploaded 4 of them to my Galaxy history. When I first went into the Upload File tool, all of the files were visible in the list. After that, when I went to upload the remaining files the next day, nothing was there anymore in either the tool or FileZilla.
FTP uploaded datasets remain in the server's FTP staging area for 3 days (at https://usegalaxy.org - other servers may have a different time set - you'd need to check with that server's admins/support team to find out). Could it be that the missing datasets were uploaded for longer than that when you went back the second day?
If data takes days to upload due to the connection speed, it would be a good strategy to FTP a few, move the data to a history, FTP a few more, move the data to a history, repeat as needed. You also might want to consider loading
.gzcompressed fastq data instead of uncompressed. You can always uncompressed or recompressed fastq dataset once in the history. https://galaxyproject.org/support/compressed-fastq/I'll also run a quick test to make sure more is not going on and will post back once done.
Update: I wasn't able to reproduce the problem with FTP upload files disappearing when logging out/logging in again (of Galaxy). I even had two that were a day or so old that were still in the list even after FTP loading more data, and loading some of that data into a history (but not all files).
From this test, it seems that your data left the FTP staging area after the three-day expiration.