Question: Galaxy moves files prior to job completion
0
gravatar for djevo1
15 days ago by
djevo160
djevo160 wrote:

Below is an error obtained from our galaxy install. We have custom directories sets for the files directory and the jobs directory. Galaxy appears to move the files but then attempts to look for them at the original path. I have tried unsetting these options but the issue of it looking in the jobs_directory after the file has already been moved still persists.

galaxy.jobs DEBUG 2017-11-24 17:53:19,057 finish(): Moved 
/var/web_services/galaxy/database/jobs_directory/000/320/galaxy_dataset_431.dat 
to /mnt/ceph/home/galaxy/galaxy/files_directory/000/dataset_401.dat
galaxy.model.metadata DEBUG 2017-11-24 17:53:19,128 setting metadata 
externally failed for HistoryDatasetAssociation 431: [Errno 2] No such 
file or directory
galaxy.jobs.runners ERROR 2017-11-24 17:53:19,319 (320/179084) Job 
wrapper finish method failed
Traceback (most recent call last):
   File "/var/web_services/galaxy/lib/galaxy/jobs/runners/__init__.py", 
line 630, in finish_job
     job_state.job_wrapper.finish( stdout, stderr, exit_code )
   File "/var/web_services/galaxy/lib/galaxy/jobs/__init__.py", line 
1266, in finish
     dataset.datatype.set_meta( dataset, overwrite=False )
   File "/var/web_services/galaxy/lib/galaxy/datatypes/binary.py", line 
391, in set_meta
     exit_code = subprocess.call( args=command, stderr=open( 
stderr_name, 'wb' ) )
   File "/usr/lib64/python2.7/subprocess.py", line 524, in call
     return Popen(*popenargs, **kwargs).wait()
   File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
     errread, errwrite)
   File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
     raise child_exception
OSError: [Errno 2] No such file or directory
galaxy.jobs ERROR 2017-11-24 17:53:19,337 fail(): Missing output file in 
working directory: [Errno 2] No such file or directory: 
'/var/web_services/galaxy/database/jobs_directory/000/320/galaxy_dataset_431.dat'
ADD COMMENTlink modified 14 days ago by Nate Coraor3.1k • written 15 days ago by djevo160
0
gravatar for Nate Coraor
14 days ago by
Nate Coraor3.1k
United States
Nate Coraor3.1k wrote:

It looks like you're using the outputs_to_working_directory option in galaxy.ini? It looks like what's happening here is that Galaxy detects that setting metadata on the job's output datasets has failed and it's trying to do it again on the Galaxy server (retry_metadata_internally, which defaults to True).

I believe you've encountered a bug in this scenario, but it'd be good to know why setting metadata failed when the job was run on the cluster. You should find in the job's directory files named like metadata_results_HistoryDatasetAssociation_<id>_<random_chars>, this should contain an error if setting metadata failed. It might also be displayed just a few lines up in the log you've pasted.

ADD COMMENTlink written 14 days ago by Nate Coraor3.1k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 102 users visited in the last hour