Galaxy moves files prior to job completion

Question: Galaxy moves files prior to job completion

12 months ago by

djevo1 • 60

djevo1 • 60 wrote:

Below is an error obtained from our galaxy install. We have custom directories sets for the files directory and the jobs directory. Galaxy appears to move the files but then attempts to look for them at the original path. I have tried unsetting these options but the issue of it looking in the jobs_directory after the file has already been moved still persists.

galaxy.jobs DEBUG 2017-11-24 17:53:19,057 finish(): Moved 
/var/web_services/galaxy/database/jobs_directory/000/320/galaxy_dataset_431.dat 
to /mnt/ceph/home/galaxy/galaxy/files_directory/000/dataset_401.dat
galaxy.model.metadata DEBUG 2017-11-24 17:53:19,128 setting metadata 
externally failed for HistoryDatasetAssociation 431: [Errno 2] No such 
file or directory
galaxy.jobs.runners ERROR 2017-11-24 17:53:19,319 (320/179084) Job 
wrapper finish method failed
Traceback (most recent call last):
   File "/var/web_services/galaxy/lib/galaxy/jobs/runners/__init__.py", 
line 630, in finish_job
     job_state.job_wrapper.finish( stdout, stderr, exit_code )
   File "/var/web_services/galaxy/lib/galaxy/jobs/__init__.py", line 
1266, in finish
     dataset.datatype.set_meta( dataset, overwrite=False )
   File "/var/web_services/galaxy/lib/galaxy/datatypes/binary.py", line 
391, in set_meta
     exit_code = subprocess.call( args=command, stderr=open( 
stderr_name, 'wb' ) )
   File "/usr/lib64/python2.7/subprocess.py", line 524, in call
     return Popen(*popenargs, **kwargs).wait()
   File "/usr/lib64/python2.7/subprocess.py", line 711, in __init__
     errread, errwrite)
   File "/usr/lib64/python2.7/subprocess.py", line 1327, in _execute_child
     raise child_exception
OSError: [Errno 2] No such file or directory
galaxy.jobs ERROR 2017-11-24 17:53:19,337 fail(): Missing output file in 
working directory: [Errno 2] No such file or directory: 
'/var/web_services/galaxy/database/jobs_directory/000/320/galaxy_dataset_431.dat'

rna-seq directory galaxy python files • 368 views

ADD COMMENT • link •

modified 12 months ago by Nate Coraor ♦ 3.2k • written 12 months ago by djevo1 • 60

12 months ago by

Nate Coraor ♦ 3.2k

United States

Nate Coraor ♦ 3.2k wrote:

It looks like you're using the outputs_to_working_directory option in galaxy.ini? It looks like what's happening here is that Galaxy detects that setting metadata on the job's output datasets has failed and it's trying to do it again on the Galaxy server (retry_metadata_internally, which defaults to True).

I believe you've encountered a bug in this scenario, but it'd be good to know why setting metadata failed when the job was run on the cluster. You should find in the job's directory files named like metadata_results_HistoryDatasetAssociation_<id>_<random_chars>, this should contain an error if setting metadata failed. It might also be displayed just a few lines up in the log you've pasted.

ADD COMMENT • link written 12 months ago by Nate Coraor ♦ 3.2k

Similar posts • Search »