Question: Pulsar gives "[Errno 2] No such file or directory:" for data managers and UCSC Get Data
0
gravatar for y.hoogstrate
2.7 years ago by
y.hoogstrate460
Netherlands
y.hoogstrate460 wrote:

Dear Galaxy biostar users,

Our cluster is configured in such a way that Pulsar distributes the jobs over several nodes. This works like a charm for all tools, except for data managers and ucsc's get data tool, which all result in the following kind of error:

Traceback (most recent call last):
  File "/<galaxy-home-dir>/lib/galaxy/jobs/runners/pulsar.py", line 256, in __prepare_job
    job_wrapper.prepare( **prepare_kwds )
  File "/<galaxy-home-dir>/lib/galaxy/jobs/__init__.py", line 855, in prepare
    tool_evaluator.set_compute_environment( compute_environment, get_special=get_special )
  File "/<galaxy-home-dir>/lib/galaxy/tools/evaluation.py", line 97, in set_compute_environment
    self.tool.exec_before_job( self.app, inp_data, out_data, param_dict )
  File "/<galaxy-home-dir>/lib/galaxy/tools/__init__.py", line 2854, in exec_before_job
    out = open( json_filename, 'w' )
IOError: [Errno 2] No such file or directory: '/opt/pulsar/files/staging/1741/outputs/dataset_2157.dat'

I am not sure if this is a bad configuration in Pular or something else. For our setup it is not a huge problem at itself because we have redirected the data manager and ucsc jobs to the local runner, but it would be neat to distribute the data manager jobs over the compute nodes as well.

Does anyone have suggestions how to do/solve this?

Thanks in advance,

Youri

software error • 973 views
ADD COMMENTlink modified 2.7 years ago • written 2.7 years ago by y.hoogstrate460
1
gravatar for jmchilton
2.7 years ago by
jmchilton1.1k
United States
jmchilton1.1k wrote:

Youri - I don't think you are doing anything wrong here. Pulsar overrides the job building in a lot of ways - but there are many cracks where Galaxy plumbing allows things to get through. My guess here is that these outputs are being defined in galaxy.json and Pulsar isn't parsing that file to find outputs to write and send back to Galaxy. I have created an issue for this in Pulsar here https://github.com/galaxyproject/pulsar/issues/93.

For now is it possible to run these tools on the head node? I guess ucsc browser shouldn't be a problem but some data managers have serious compute requirements.

ADD COMMENTlink written 2.7 years ago by jmchilton1.1k
0
gravatar for y.hoogstrate
2.7 years ago by
y.hoogstrate460
Netherlands
y.hoogstrate460 wrote:

Hi John,

Thanks for your quick and clear answer.

"For now is it possible to run these tools on the head node?"
Yes, this is not a problem. The head node has multiple cores of which only 1 core is configured for local jobs. Also, data manager jobs are not being executed that often.

"I have created an issue for this in Pulsar"
Great, thank you! I will keep my eyes open. Whenever the time is there I am available for testing.

Youri

ADD COMMENTlink written 2.7 years ago by y.hoogstrate460
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 108 users visited in the last hour