Local Install - Understanding Upload File Concept

Question: Local Install - Understanding Upload File Concept

6.0 years ago by

mailing list • 840 wrote:

I'm confused if files are uploaded through the Galaxy server or through Apache? I already set up the Apache proxy explained here: http://wiki.galaxyproject.org/Admin/Config/Apache%20Proxy (Does that affect uploads?) My goal is to let users upload as large of files as possible through HTTP. So far I just tested and upload of a 140MB file and I got this error after a couple of minutes: 1: dsim-all-chromosome-r1.3.fasta.msg.updated.fasta 0 bytes An error occurred running this job:Traceback (most recent call last): File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line 384, in <module> __main__() File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line 359, in __main__ registry.load_datat I see this information in the bug report: Traceback (most recent call last): File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line 384, in <module> __main__() File "/misc/local/galaxy/galaxy-dist/tools/data_source/upload.py", line 359, in __main__ registry.load_datatypes( root_dir=sys.argv[1], config=sys.argv[2] ) File "/misc/local/galaxy/galaxy- dist/lib/galaxy/datatypes/registry.py", line 65, in load_datatypes tree = galaxy.util.parse_xml( config ) File "/misc/local/galaxy/galaxy-dist/lib/galaxy/util/__init__.py", line 143, in parse_xml tree = ElementTree.parse(fname) File "build/bdist.linux-x86_64/egg/elementtree/ElementTree.py", line 859, in parse File "build/bdist.linux-x86_64/egg/elementtree/ElementTree.py", line 576, in parse IOError: [Errno 2] No such file or directory: '/scratch/galaxy/tmptzuSYJ' Traceback (most recent call last): File "./scripts/set_metadata.py", line 118, in <module> __main__() File "./scripts/set_metadata.py", line 68, in __main__ datatypes_registry.load_datatypes( root_dir=config_root, config=datatypes_config ) File "/misc/local/galaxy/galaxy- dist/lib/galaxy/datatypes/registry.py", line 65, in load_datatypes tree = galaxy.util.parse_xml( config ) File "/misc/local/galaxy/galaxy-dist/lib/galaxy/util/__init__.py", line 143, in parse_xml tree = ElementTree.parse(fname) File "build/bdist.linux-x86_64/egg/elementtree/ElementTree.py", line 859, in parse File "build/bdist.linux-x86_64/egg/elementtree/ElementTree.py", line 576, in parse IOError: [Errno 2] No such file or directory: '/scratch/galaxy/tmptzuSYJ' Also here is what /scratch/galaxy looks like. It seems like the file it mentions does exist?? total 134M -rw-r--r-- 1 galaxy scicomp 20K Nov 13 13:35 tmpfCaiGX -rw-r--r-- 1 galaxy scicomp 20K Nov 14 13:36 tmpM2Wclq -rw-r--r-- 1 galaxy scicomp 20K Nov 14 13:35 tmpsIyPur -rw-r--r-- 1 galaxy scicomp 20K Nov 16 08:48 tmptY5RaM -rw-r--r-- 1 galaxy scicomp 20K Nov 16 08:54 tmptzuSYJ -rw-r--r-- 1 galaxy scicomp 20K Nov 15 15:47 tmpu77PEK -rw------- 1 galaxy scicomp 286 Nov 16 08:58 tmpykoBIq -rw------- 1 galaxy scicomp 134M Nov 16 08:58 upload_file_data_G3eZEg Thanks, Greg

galaxy • 1.6k views

ADD COMMENT • link •

modified 6.0 years ago by Nate Coraor ♦ 3.2k • written 6.0 years ago by mailing list • 840

6.0 years ago by

Nate Coraor ♦ 3.2k

United States

Nate Coraor ♦ 3.2k wrote:

The file is uploaded "through" Apache, but Apache doesn't do anything special with it other than stream the bits through to Galaxy. With nginx it's possible to have the proxy server actually handle the upload itself. Are you running the upload tool on a cluster? --nate

ADD COMMENT • link written 6.0 years ago by Nate Coraor ♦ 3.2k

Well I have Galaxy set up to send it's jobs to SGE/qsub but I haven't verified that it's doing that yet. I wanted to upload a file to test on first. Here are the changes I made to universe_wsgi.ini after copying the sample: -Greg

ADD REPLY • link written 6.0 years ago by mailing list • 840

I'm guessing that it's working, you can verify by having a look in your Galaxy server output/log file. new_file_path will need to be set to something that is accessible on the cluster. --nate

ADD REPLY • link written 6.0 years ago by Nate Coraor ♦ 3.2k

I guess I'm still confused about new_file_path. The upload tool uses that directory to store files temporariliy until the upload completes? And then moves it? (Will it clean up anything placed in the new_file_path directory?) Thanks, Greg

ADD REPLY • link written 6.0 years ago by mailing list • 840

Also what would I look for in the log file to know if it's issuing jobs? I started a new upload, but I don't see anything from the galaxy user when I run qstat. Thanks again. Greg

ADD REPLY • link written 6.0 years ago by mailing list • 840

If you follow the log, you should see messages from galaxy.jobs, and specifically, galaxy.jobs.runners.drmaa. --nate

ADD REPLY • link written 6.0 years ago by Nate Coraor ♦ 3.2k

No, I don't see anything like that in the log after I started the upload. Any ideas on how to see what might be going wrong there? -Greg

ADD REPLY • link written 6.0 years ago by mailing list • 840

Yes, the parameters of the upload tool are written to a temp file in this directory. It is cleaned up upon completion. A lot of other things use new_file_path, and not all of them clean themselves up, so it's a good idea to use an automated process to clean the directory. --nate

ADD REPLY • link written 6.0 years ago by Nate Coraor ♦ 3.2k

Similar posts • Search »