Question: Making Galaxy Tools run on multiple cores/treads?
1
gravatar for vebaev
23 months ago by
vebaev130
vebaev130 wrote:

Dear all,

I have been trying for some days to make Galaxy run tools on more than 1 core (e.g. Bowtie to use 8 or more cores on my machine with 48 cores) by modifying and activating the file jpb_conf.xml but without any luck.

I will appreciate if anyone already done it to point me to a tutorial of this method or to the other method on installing a job manager which is also I think out of mine knowledge?

ADD COMMENTlink modified 23 months ago by Jennifer Hillman Jackson25k • written 23 months ago by vebaev130
1

I guess you deleted your comment. Anyway, bowtie-build is single-threaded. For the others, make sure you restart Galaxy, since the job_conf.xml file is only parsed when Galaxy starts.

ADD REPLYlink written 23 months ago by Devon Ryan1.9k

Yes , I couldnt paste my file config correctly as typing from my iPad. I restarted it. Let me see if I can paste it now....


<job_conf>
<plugins>
<plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="8"/>
<plugin id="multilocal" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="2"/>
</plugins>
<handlers>
<handler id="main"/>.
</handlers>

<destinations default="local">
<destination id="local" runner="local"/>
<destination id="multicore8" runner="multilocal">
<param id="local_slots">8</param>
</destination>
</destinations>

<tools>
<tool id="bowtie2" destination="multicore8"/>
<tool id="bowtie" destination="multicore8"/>
<tool id="bowtie-build" destination="multicore8"/>
<tool id="fastq_groomer_parallel" destination="multicore8"/>
<tool id="deeptools" destination="multicore8"/>
<tool id="cufflinks" destination="multicore8"/>
<tool id="cuffdiff" destination="multicore8"/>
<tool id="cuffmerge" destination="multicore8"/>
<tool id="tophat2" destination="multicore8"/>
</tools>
</job_conf>

ADD REPLYlink written 23 months ago by vebaev130

I don't see anything obviously wrong with that, just make sure you restart galaxy.

ADD REPLYlink written 23 months ago by Devon Ryan1.9k
2
gravatar for Devon Ryan
23 months ago by
Devon Ryan1.9k
Germany
Devon Ryan1.9k wrote:

Make sure you're modifying job_conf.xml rather than jpb_conf.xml. Below is the relevant section of my job_conf.xml for running bowtie2:

 <destinations default="slurm">
    <destination id="slurm10threads" runner="slurm">
        <param id="request_cpus">10</param>
        <param id="embed_metadata_in_job">False</param>
        <param id="nativeSpecification">-p Galaxy -n 10</param>
        <env file="/data/galaxy/galaxy-central/galaxy_venv/bin/activate" />
    </destination>
</destinations>
<tools>
    <tool id="bowtie2" destination="slurm10threads"/>

If you're using the "local" job runner, then you'll need something like:

    <destination id="local" runner="local">
        <param id="local_slots">4</param>
    </destination>

And then either set destination to local or just make that the default.

ADD COMMENTlink written 23 months ago by Devon Ryan1.9k

Could you help me figure out why I can't seem to make the NCBI_blast_plus package run multi-threaded? I have tried using different tool IDs, "blast", "ncbi_blast_plus" and "ncbi_plast_plus_tools". But none seem to work... ? Thanks for your help.

<?xml version="1.0"?>
<!-- A sample job config that explicitly configures job running the way it is configured by default (if there is no explicit config). -->
<job_conf>
    <plugins>
        <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="4"/>
        <plugin id="multilocal" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="1"/>
    </plugins>
    <handlers>
       <handler id="main" />
    </handlers>

  <destinations default="local">
        <destination id="local" runner="local"/>
        <destination id="multicore4" runner="multilocal">
          <param id="local_slots">4</param>
       </destination>
  </destinations>
    <tools>
      <tool id="bowtie2" destination="multicore4" />
      <tool id="bowtie"  destination="multicore4" />
      <tool id="deeptools"  destination="multicore4" />
      <tool id="tophat2" destination="multicore4" />>
      <tool id="blast" destination="multicore4" />
    </tools>
</job_conf>
ADD REPLYlink modified 7 months ago • written 7 months ago by TWV20
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 182 users visited in the last hour