Question: Upload Of Large Files
0
gravatar for Hans Vasquez-Gross
9.0 years ago by
Hans Vasquez-Gross30 wrote:
Hello All, We have a local galaxy install at our facility that is running on a VM machine with an allocated memory of 2gigs. I have a user that has been trying to upload a ~6gig fastq file to galaxy for the past couple of days within our same subnet. She tried submitting the upload but the next day it still wasn't finished uploading, so I told her to cancel the upload and try again. The same problem occurred the second day after a day of trying. Should galaxy be able to support an upload of this size? Thanks, -Hans
galaxy • 884 views
ADD COMMENTlink modified 9.0 years ago by Greg Von Kuster840 • written 9.0 years ago by Hans Vasquez-Gross30
0
gravatar for Greg Von Kuster
9.0 years ago by
Greg Von Kuster840 wrote:
Hello Hans, What version of Galaxy are you running? We recently introduced enhancements to the upload utility for uploading files to a History - these were in change set 2583. These enhancements are currently under development for uploading files to a Data Library. If you are running an older version of Galaxy, are you running behind an Apache proxy? If so, you may want to increase the value of the TimeOut or ProxyTimeout directives. Greg Von Kuster Galaxy Development Team
ADD COMMENTlink written 9.0 years ago by Greg Von Kuster840
Another option is to compress the file before uploading it. It will reduce the bandwidth required in uploading and Galaxy will automatically decompress it once uploaded. Chris
ADD REPLYlink written 9.0 years ago by Chris Cole150
We did some testing and found that the upload isn't finishing because the file system that has the /tmp/ folder has too little space. I did a find for the .py files and greped for files containing /tmp/ and found that there are some files that have this path hardcoded in the scripts. We have the space on another file system that is located in /temp/galaxy_space. Is there anyway to change this configuration via a variable? If not, I'll try changing these locations manually to see if this fixes the problem. Thanks for the help, Hans Vasquez-Gross
ADD REPLYlink written 9.0 years ago by Hans Vasquez-Gross30
Hi Hans, /tmp is hardcoded as a last result, but everything uses Python's tempfile module, which has a set list of methods for determining where to put stuff. The easiest solution is to set $TEMP in the shell before starting Galaxy. We do this now to place tempfiles on the same filesystem as the database/files/ directory for faster upload processing (i.e. moves during upload processing just update an inode instead of moving data). --nate
ADD REPLYlink written 9.0 years ago by Nate Coraor3.2k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 147 users visited in the last hour