Making Galaxy Tools run on multiple cores/treads?

Question: Making Galaxy Tools run on multiple cores/treads?

23 months ago by

vebaev • 130

vebaev • 130 wrote:

Dear all,

I have been trying for some days to make Galaxy run tools on more than 1 core (e.g. Bowtie to use 8 or more cores on my machine with 48 cores) by modifying and activating the file jpb_conf.xml but without any luck.

I will appreciate if anyone already done it to point me to a tutorial of this method or to the other method on installing a job manager which is also I think out of mine knowledge?

cores multicores multithreading galaxy local • 1.0k views

ADD COMMENT • link •

modified 23 months ago by Jennifer Hillman Jackson ♦ 25k • written 23 months ago by vebaev • 130

I guess you deleted your comment. Anyway, bowtie-build is single-threaded. For the others, make sure you restart Galaxy, since the job_conf.xml file is only parsed when Galaxy starts.

ADD REPLY • link written 23 months ago by Devon Ryan • 1.9k

Yes , I couldnt paste my file config correctly as typing from my iPad. I restarted it. Let me see if I can paste it now....

<job_conf>
<plugins>
<plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="8"/>
<plugin id="multilocal" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="2"/>
</plugins>
<handlers>
<handler id="main"/>.
</handlers>

ADD REPLY • link written 23 months ago by vebaev • 130

I don't see anything obviously wrong with that, just make sure you restart galaxy.

ADD REPLY • link written 23 months ago by Devon Ryan • 1.9k

23 months ago by

Devon Ryan • 1.9k

Germany

Devon Ryan • 1.9k wrote:

Make sure you're modifying job_conf.xml rather than jpb_conf.xml. Below is the relevant section of my job_conf.xml for running bowtie2:

 <destinations default="slurm">
    <destination id="slurm10threads" runner="slurm">
        <param id="request_cpus">10</param>
        <param id="embed_metadata_in_job">False</param>
        <param id="nativeSpecification">-p Galaxy -n 10</param>
        <env file="/data/galaxy/galaxy-central/galaxy_venv/bin/activate" />
    </destination>
</destinations>
<tools>
    <tool id="bowtie2" destination="slurm10threads"/>

If you're using the "local" job runner, then you'll need something like:

    <destination id="local" runner="local">
        <param id="local_slots">4</param>
    </destination>

And then either set destination to local or just make that the default.

ADD COMMENT • link written 23 months ago by Devon Ryan • 1.9k

ADD REPLY • link written 8 months ago by Jennifer Hillman Jackson ♦ 25k

Could you help me figure out why I can't seem to make the NCBI_blast_plus package run multi-threaded? I have tried using different tool IDs, "blast", "ncbi_blast_plus" and "ncbi_plast_plus_tools". But none seem to work... ? Thanks for your help.

<?xml version="1.0"?>
<!-- A sample job config that explicitly configures job running the way it is configured by default (if there is no explicit config). -->
<job_conf>
    <plugins>
        <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="4"/>
        <plugin id="multilocal" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="1"/>
    </plugins>
    <handlers>
       <handler id="main" />
    </handlers>

  <destinations default="local">
        <destination id="local" runner="local"/>
        <destination id="multicore4" runner="multilocal">
          <param id="local_slots">4</param>
       </destination>
  </destinations>
    <tools>
      <tool id="bowtie2" destination="multicore4" />
      <tool id="bowtie"  destination="multicore4" />
      <tool id="deeptools"  destination="multicore4" />
      <tool id="tophat2" destination="multicore4" />>
      <tool id="blast" destination="multicore4" />
    </tools>
</job_conf>

ADD REPLY • link modified 7 months ago • written 7 months ago by TWV • 20

Similar posts • Search »