Question: Passing job_resource_params_conf options to queue
0
gravatar for david.a.morais
2.7 years ago by
Canada
david.a.morais110 wrote:

Hi All,

 

I am trying to use the job_resource_params_conf options to allow users to set parameters for some queues (i.e. walltime, ppn) and sometimes select an already pre-configured queue.

The web interface is showing the options that I want to choose from, but after selecting the option when I execute the tool the job runs with default parameter.

I only modified the job_resource_params_conf.xml and job_config.xml, (besides the galaxy.ini) so I am not sure if any other file/script needs to be modified in order to pass these parameters to the jobs.

I am using torque pbs

Below is my job_resource_params_conf.xml and and the job_config.xml

 

 

  <param label="walltime" name="walltime" type="integer" size="3" min="1" max="120" value="" help="Maximum job time in hours, 'walltime' value (1-120). Leave blank to use default value."/>
  <param label="Queue" name="queue" type="select" help="Modify this option if you need more walltime, cores or memory">
    <option value="qwork">qwork 24 cores 32GB RAM walltime 72 Hours (default)</option>
    <option value="qtest3_short">qtest3_short, 24 cores, 32 RAM, walltime 15 min (short jobs. Lowest waiting time)</option>
    <option value="qwork_extended">qwork_extended 24 cores 32GB RAM walltime 120 Hours (CPU-time intensive)</option>
    <option value="qfat256">qfat256, 48 cores, 256 RAM, walltime 120 Hours (High memory requirement)</option>
    <option value="qfat512">qfat512, 48 cores, 512 RAM, walltime 48 Hours (Highest memory requirement. Limited walltime.)</option>
  </param>
</parameters>

 

and job_config.xml

 

 

<?xml version="1.0"?>
<job_conf>
    <plugins>
        <plugin id="pbs" type="runner" load="galaxy.jobs.runners.pbs:PBSJobRunner" worker="10"/>
        <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="4"/>
    </plugins>
    <handlers default="handlers">
        <handler id="handler0" tags="handlers"/>
        <handler id="handler1" tags="handlers"/>
        <handler id="handler2" tags="handlers"/>
    </handlers>
    <destinations default="qtest3">
       <destination id="qtest3" runner="pbs" >
           <param id="Resource_List">walltime=02:15:00</param>
           <param id="destination">qtest3@localhost</param>
      </destination>
       <destination id="qtest3_short" runner="pbs" >
           <param id="destination">qtest3@localhost</param>
            <param id="Resource_List">walltime=00:15:00</param>
      </destination>
       <destination id="qtest" runner="pbs" >
           <param id="Resource_List">walltime=00:15:00</param>
           <param id="destination">qtest</param>
      </destination>
      <destination id="qwork" runner="pbs" >
           <param id="Resource_List">walltime=120:00:00</param>
           <param id="destination">qwork</param>
        </destination>
        <destination id="qfat256" runner="pbs" >
            <param id="Resource_List">walltime=120:00:00</param>
            <param id="destination">qfat256</param>
        </destination>
        <destination id="qfat512" runner="pbs" >
            <param id="Resource_List">walltime=48:00:00</param>
            <param id="destination">qfat512</param>
         </destination>
           <destination id="local" runner="local"/>
    </destinations>

    <tools>
         <!-- Upload tools will run locally by default -->
         <tool id="toolshed.g2.bx.psu.edu/repos/devteam/fastqtofasta/3571553aeb20/fastqtofasta" destination="qtest3_short" resources="walltime"/>
        <tool id="toolshed.g2.bx.psu.edu/repos/geert-vandeweyer/sendmail/sendemail/0.0.1" destination="qtest3_short"  resources="walltime"/>
         <tool id="toolshed.g2.bx.psu.edu/repos/pjbriggs/trimmomatic/a60283899c6d/trimmomatic" destination="qtest3_short" resources="all"/>
    </tools>
    <resources default="default">
      <!-- Group different parameters defined in job_resource_params_conf.xml
                      together and assign these groups ids. Tool section below can map
           tools to different groups. This is experimental functionality!
      -->
      <group id="default">walltime</group>
      <group id="walltime">walltime</group>
      <group id="all">queue,walltime</group>
    </resources>
</job_conf>
job resource options • 1.0k views
ADD COMMENTlink modified 2.5 years ago by jmchilton1.1k • written 2.7 years ago by david.a.morais110

Hello,

This seems worth a check: What is the setting of 'job_config_file' in your galaxy.ini?

Galaxy is not loading job_conf.xml ?

ADD REPLYlink written 2.7 years ago by Jennifer Hillman Jackson25k

Hi Jennifer,

Galaxy is loading the job_conf.xml and the job_resource_params_conf.xml  as well.

What was happening is that when a user was choosing a 'Job Resource Parameters' from the interface these parameters where not passed to the scheduler, instead the job was sent to default queue.

After digging on the internet and look at several ansible playbooks (unfortunately the documentation of job_resource_params_conf is not very clear) I realized that I had to write a destination.py under '../rules/' and create a dynamic destination which receives these 'Job Resource Parameters' and send them to the scheduler.

Now everything is working fine.

 

If ever someone has the same problem I will be more than glad to help those person to configure the job_resource_params_conf.xml , job_config.xml and to create a dynamic rule.

 

Thanks a lot

David Morais

ADD REPLYlink written 2.7 years ago by david.a.morais110
0
gravatar for jmchilton
2.5 years ago by
jmchilton1.1k
United States
jmchilton1.1k wrote:

You must define a dynamic job destination (https://wiki.galaxyproject.org/Admin/Config/Jobs#Dynamic_Destination_Mapping) to consume these paramters  and produce a job destination that maps the parameters to the queue parameter you are wish to use.

If your job destination function has a parameter named "resource_params", a map of the parameters the user specified will be included in the request to define a job destination.

ADD COMMENTlink written 2.5 years ago by jmchilton1.1k

Thanks John,

I modify my job_config and created a few rules. Now all my jobs are Dynamic (with a default case).
This is working like a charm.

Cheers

ADD REPLYlink written 2.4 years ago by david.a.morais110

Hi David, 

we are facing exactly the same issue. Can you provide us your complete solution (including parts of all configured files)?

Cheers, Pavel

ADD REPLYlink written 2.4 years ago by pavel.fibich10

Hi Pavel,

For sure, Give me your email address and I will send all my rules and configurations to you.

Cheers

ADD REPLYlink written 2.4 years ago by david.a.morais110

Hi David,

thank you, just mylogin with @gmail.com as suffix.

Cheers, Pavel

ADD REPLYlink written 2.4 years ago by pavel.fibich10

HI @David/@Pavel,

Could one of you plz fwd me those instructions to resolve the issue mentioned above. I would really appreciate it. My email is balaszone at gmail dot com

Thanks, Bala

ADD REPLYlink modified 2.1 years ago • written 2.1 years ago by balaszone60

Hi David or Pavel,

I am facing exactly the same issue. Could one of you please provide me those instructions to resolve the issue? My email is fangpingmu at gmail dot com. Thank you very much.

ADD REPLYlink written 8 months ago by fangpingmu0

Hi David, I also got into the same trouble. Can you provide us your complete solution (including parts of all configured files)? My email is liuhui@fudan.edu.cn, Thanks a lot.

ADD REPLYlink written 18 months ago by hliu0
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 98 users visited in the last hour