Question: Help configuring dynamic resource allocation
0
gravatar for jdv
2.1 years ago by
jdv20
jdv20 wrote:

I'm trying (for the first time) to set up dynamic resource allocation for some jobs on a local Galaxy instance. I've read the docs here and can get a toy example working, but I know very little Python and don't fully understand how to access the job parameters from within the rules script. The tool is the phyml wrapper - there's a pruned-down snippet here. My rules file looks like this:

from galaxy.jobs import JobDestination
import os

def phyml(job):
    # Allocate extra time/cores to bootstrap jobs
    #support = job.parameters.???? - how to access branchSupport value?
    #if support > 0:
        #param_str = "-l walltime=168:00:00 -l nodes=1:ppn=60 -q batch"
    #else:
    param_str = "-l walltime=24:00:00 -l nodes=1:ppn=1 -q batch"
    return JobDestination(runner="torque", params={"nativeSpecification": param_str})

This works but obviously doesn't do any actual decision-making as is, and even though I'm sure it's very basic python I can't figure out the correct syntax to access the "branchSupport" value from the job. I'd greatly appreciate any help.

Thanks, Jeremy

galaxy python • 559 views
ADD COMMENTlink modified 2.1 years ago • written 2.1 years ago by jdv20
1
gravatar for jdv
2.1 years ago by
jdv20
jdv20 wrote:

For the record, I came up with the following working solution:

from galaxy.jobs import JobDestination
import os
import ast 

def phyml(job):

    # Allocate extra time/cores to bootstrap jobs
    params  = dict( [ p.name, p.value) for p in job.parameters ] ) 
    support = ast.literal_eval(params['support_condition'])['branchSupport']
    if int(support) > 0:
        param_str = "-l walltime=168:00:00 -l nodes=1:ppn=60 -q batch"
    else:
        param_str = "-l walltime=168:00:00 -l nodes=1:ppn=1 -q batch"
    return JobDestination(runner="torque", params={"nativeSpecification": param_str})

I finally figured out that the ast.literal_eval() was necessary because the value of the "support_condition" jobParameter was a string representation of a dict and not an actual dict. Not sure if there is a better syntax or method - again I'm not a Python coder as such. I would welcome suggestions or a pointer to the proper API documentation.

Dynamic resource allocation is such a useful or even critical feature for cluster environments - nice work dev team.

ADD COMMENTlink modified 2.1 years ago • written 2.1 years ago by jdv20
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 115 users visited in the last hour