Single and Multi threading on one machine

Question: Single and Multi threading on one machine

3.7 years ago by

Netherlands

I am trying to set up galaxy tools either single or multithreaded on a 24 core machine.

So what I would like to achieve is to have a maximum of 20 single core workers running and if a multi threaded job is started of lets say 6 cores then I can have a maximum of 20-6 = 14 single core runners and 1 multi core runner.

However I am having a very hard time to get this working and am probably missing something.

My multi threaded tool command looks like:

<command interpreter="python3.4">multhread.py '-input' '$input' '--seqtype' '$seqtype' '-f' 'TSV' '--applications' '$appl' '-output' '$outfile' -sourcedb XYZ '-num_threads' \${GALAXY_SLOTS:-6}
    </command>

My job_conf.xml file looks like this:

<job_conf>
    <plugins>
        <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="20"/>
    </plugins>
    <handlers>
        <handler id="main"/>
    </handlers>
    <destinations default="single">
        <destination id="single" runner="local">
        <param id="local_slots">20</param>
    </destination>
    </destinations>
</job_conf>

I digged a bit in the code and noticed:

            slots_statement = 'GALAXY_SLOTS="%d"; export GALAXY_SLOTS; GALAXY_SLOTS_CONFIGURED="1"; export GALAXY_SLOTS_CONFIGURED;' % ( int( slots ) )

Which is set to one as 1 runner is running, but shouldnt it in theory not be set to 6 as it consumes 6 slots cores?

threading multi • 1.7k views

ADD COMMENT • link •

modified 3.6 years ago by jmchilton ♦ 1.1k • written 3.7 years ago by jasperkoehorst • 10

3.6 years ago by

jmchilton ♦ 1.1k

United States

jmchilton ♦ 1.1k wrote:

What is 1 in this context - GALAXY_SLOTS or GALAXY_SLOTS_CONFIGURED? In your example GALAXY_SLOTS should be set to 20 which should cause all 20 of the local runner threads to each use 20 cores - this is not great.

Have you looked at this answer https://biostar.usegalaxy.org/p/10158/#10261?

Adapting it to your 20 core machine - you might want to have say two 8 core threads and 4 single core threads That would look something like this:

<job_conf>
<plugins>
<plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="4"/>
<plugin id="multilocal" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="2"/>
</plugins>
<destinations default="local">
<destination id="local" runner="local"/>
<destination id="multicore8" runner="multilocal">
<param id="local_slots">8</param>
</destination>
</destinations>
<tools>
   <tool id="bowtie2" destination="multicore8" />
   <tool id="bowtie" destination="multicore8" />
   <tool id="deeptools" destination="multicore8" />
   <tool id="cufflinks" destination="multicore8" />
   <tool id="cuffdiff" destination="multicore8" />
   <tool id="cuffmerge" destination="multicore8" />
   <tool id="tophat2" destination="multicore8" />
</tools>
</job_conf>

There are some things you can do to work through different permutations in this file - but ultimately Galaxy isn't a resource manager and you should probably install something like SLURM or Condor to make most effective use of your resources.

ADD COMMENT • link written 3.6 years ago by jmchilton ♦ 1.1k

Similar posts • Search »