Upload Of Large Files

Question: Upload Of Large Files

9.2 years ago by

Hello All, We have a local galaxy install at our facility that is running on a VM machine with an allocated memory of 2gigs. I have a user that has been trying to upload a ~6gig fastq file to galaxy for the past couple of days within our same subnet. She tried submitting the upload but the next day it still wasn't finished uploading, so I told her to cancel the upload and try again. The same problem occurred the second day after a day of trying. Should galaxy be able to support an upload of this size? Thanks, -Hans

galaxy • 965 views

ADD COMMENT • link •

modified 9.2 years ago by Greg Von Kuster • 840 • written 9.2 years ago by Hans Vasquez-Gross • 30

9.2 years ago by

Greg Von Kuster • 840

Greg Von Kuster • 840 wrote:

Hello Hans, What version of Galaxy are you running? We recently introduced enhancements to the upload utility for uploading files to a History - these were in change set 2583. These enhancements are currently under development for uploading files to a Data Library. If you are running an older version of Galaxy, are you running behind an Apache proxy? If so, you may want to increase the value of the TimeOut or ProxyTimeout directives. Greg Von Kuster Galaxy Development Team

ADD COMMENT • link written 9.2 years ago by Greg Von Kuster • 840

Another option is to compress the file before uploading it. It will reduce the bandwidth required in uploading and Galaxy will automatically decompress it once uploaded. Chris

ADD REPLY • link written 9.2 years ago by Chris Cole • 150

We did some testing and found that the upload isn't finishing because the file system that has the /tmp/ folder has too little space. I did a find for the .py files and greped for files containing /tmp/ and found that there are some files that have this path hardcoded in the scripts. We have the space on another file system that is located in /temp/galaxy_space. Is there anyway to change this configuration via a variable? If not, I'll try changing these locations manually to see if this fixes the problem. Thanks for the help, Hans Vasquez-Gross

ADD REPLY • link written 9.2 years ago by Hans Vasquez-Gross • 30

Hi Hans, /tmp is hardcoded as a last result, but everything uses Python's tempfile module, which has a set list of methods for determining where to put stuff. The easiest solution is to set $TEMP in the shell before starting Galaxy. We do this now to place tempfiles on the same filesystem as the database/files/ directory for faster upload processing (i.e. moves during upload processing just update an inode instead of moving data). --nate

ADD REPLY • link written 9.2 years ago by Nate Coraor ♦ 3.2k

Similar posts • Search »