Question: GALAXY_SLOTS not working
1
gravatar for tuto345
4.0 years ago by
tuto34520
France
tuto34520 wrote:

Hi All,

I have the strong impression that the GALAXY_SLOTS directive is not working. Bowtie2 runs on one thread through galaxy while the exact same command on the terminal runs with 4 threads (changing -p \${GALAXY_SLOTS:-4} to -p 4) and the exact same thing happens with parallel groomer.

Is there something that has to be configured on the admin side so that GALAXY_SLOTS is effectively used?

I'm on changeset 14925:2db0fb9594d6.

Thanks!

Cristian

software error galaxy slots • 2.1k views
ADD COMMENTlink modified 4.0 years ago • written 4.0 years ago by tuto34520
3
gravatar for Wolfgang Maier
4.0 years ago by
Germany
Wolfgang Maier600 wrote:

Hi,

${GALAXY_SLOTS:-4} is shell default value syntax, i.e. it says use the GALAXY_SLOTS environmental variable or, if that's not set, use 4.

In other words, 4 is not used generally, as you seem to assume, but ONLY if GALAXY_SLOTS is not set, but GALAXY_SLOTS should ALWAYS be set on every reasonably up-to-date version of Galaxy so your default never gets used.

GALAXY_SLOTS itself is determined by your job runner settings taken from job_conf.xml, which used to be in the main directory, but is now in the config folder.

I'm not an expert in writing these files so better wait for someone else to help out or try googling.

 

Best,

Wolfgang

ADD COMMENTlink written 4.0 years ago by Wolfgang Maier600
0
gravatar for tuto345
4.0 years ago by
tuto34520
France
tuto34520 wrote:

Hi Wolfgang,

GALAXY_SLOTS is not defined as an environment variable and the default is not used it most probably defaults to 1 somewhere. I could not determine where. I solved my problem by creating, in the jobs_conf.xml file, a destination for 4 threaded software, a handler for multithreaded jobs, a local plugin and individual entries for all the multithreaded software I found in the locally installed tools... I guess I interpreted the commentaries in the jobs_conf.xml correctly as now bowtie2 runs with 4 threads.

I must confess that it was a hit and miss process until everything was working. I still do not understand the relation of the threads defined in the galaxy.ini and the worker threads declared in the jobs_conf.xml file. I had to declare the web server and the handlers with 5 threads or otherwise I had an error.

in the end my config files are like this.

galaxy.ini

[server:web0]
use = egg:Paste#http
port = 8080
host = 127.0.0.1
use_threadpool = true
threadpool_workers = 5

#[server:web1]
#use = egg:Paste#http
#port = 8081
#host = 127.0.0.1
#use_threadpool = true
#threadpool_workers = 5

[server:handler0]
use = egg:Paste#http
port = 8082
host = 127.0.0.1
use_threadpool = true
threadpool_workers = 5

[server:handler1]
use = egg:Paste#http
port = 8083
host = 127.0.0.1
use_threadpool = true
threadpool_workers = 5

[server:handler2]
use = egg:Paste#http
port = 8084
host = 127.0.0.1
use_threadpool = true
threadpool_workers = 5

and job_conf.xml

<job_conf>
    <plugins>
        <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="10"/>
        <plugin id="multilocal" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="2"/>
    </plugins>
    <handlers default="handlers">
        <handler id="handler0" tags="handlers"/>
        <handler id="handler1" tags="handlers"/>
        <handler id="handler2" tags="multi"/>
    </handlers>
    <destinations default="local">
        <destination id="local" runner="local"/>
        <destination id="multicore4" runner="multilocal">
          <param id="local_slots">4</param> <!-- Specify GALAXY_SLOTS for local jobs. -->
          <param id="embed_metadata_in_job">True</param>
          <job_metrics />
        </destination>
        <destination id="multicore8" runner="multilocal">
          <param id="local_slots">8</param> <!-- Specify GALAXY_SLOTS for local jobs. -->
          <param id="embed_metadata_in_job">True</param>
          <job_metrics />
        </destination>
    </destinations>
    <tools>
    <tool id="bowtie2" handler="handler2" destination="multicore4" />
    <tool id="bowtie" handler="handler2" destination="multicore4" />
    <tool id="deeptools" handler="handler2" destination="multicore4" />
    <tool id="cufflinks" handler="handler2" destination="multicore4" />
    <tool id="cuffdiff" handler="handler2" destination="multicore4" />
    <tool id="cuffmerge" handler="handler2" destination="multicore4" />
    <tool id="tophat2" handler="handler2" destination="multicore4" />
    <tool id="fastq_groomer_parallel" handler="handler2" destination="multicore4" />
    </tools>
</job_conf>

Is this a reasonable configuration for a 40 cores server?

Any input on this would be appreciated!

Cristian

ADD COMMENTlink written 4.0 years ago by tuto34520
1

Hmmm... assuming you run enough jobs - there will only ever be two multilocal jobs running (or maybe 6 since there are multiple handlers) - so to my mind it doesn't make sense to allocate some at 4 cores and some at 8 - Galaxy is always just going to run two at a time - so perhaps just assign them all to 8?

Galaxy isn't a resource manager - so while you can configure it to restrict the number of cores at any given time - you cannot really do stuff like say fill this pool of N cores with 4 and 8 core jobs - so if the utilization of these nodes is important to you I would strongly suggest using a resource manager (such as SLURM or Condor) on this machine. Core utilization isn't the only down side to the local job runners - they will also kill your jobs everytime Galaxy restarts.

Also - since you only have one big box - I probably would only use one job handler. I have seen two or three farm things out to huge clusters so I don't think you need more than one per worker.

ADD REPLYlink written 4.0 years ago by jmchilton1.1k

Sorry for the late response and thank you for the tips.

I have installed torque and have problems with it. I will start another thread for that.

I would like to have your opinion on Pulsar as, in the near future, the idea is to send all the heavy/long jobs to our cluster (configured with torque/maui). Is it a good alternative to administer the resources of a single 40 core machine?

 

Thanks

Cristian

ADD REPLYlink written 3.9 years ago by tuto34520

Hi tuto345,

I have used the same lines provided by you above for galaxy.ini and job_conf.xml, but nevertheless when I run bowtie2 I see again 1 core is used? Is there somthing specific (e.g. tools names) I should look for my to work or the core should work out of the box? Thanks!

 

ADD REPLYlink modified 3.3 years ago • written 3.3 years ago by vebaev130

Hi Manekineko,

If you used the exact same lines, it should work. It did for me at least. Check that the names in the tool definition, especially destination, matches a destination id and that the runner is matches exactly the name of a plugin definition. You should restart galaxy after making the changes, but I guess that you did that.

I went with another approach as jmchilton adviced me. I installed torque on the single machine with only one handler and the ressources are properly handled, plus I have the ability to limit the number of jobs that a single user can run at the same time.

Hope it helps,

Cristian

ADD REPLYlink written 3.3 years ago by tuto34520

thanks :) I found out that I didn't uncomment the job_config_file in galaxy.ini so the job_config.xml wasn't loading at all :))

Anyway now is ok :) do you know how to know the exact tool name as I noticed after I put "bowtie2" and "samtools", only bowtie2 was running multicore, samtools not?

ADD REPLYlink written 3.3 years ago by vebaev130
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 172 users visited in the last hour