Question: CPU utilization with galaxy
3
gravatar for marcoabbestia
3.9 years ago by
European Union
marcoabbestia30 wrote:

Hi,

I launched an analysis of alignment with bwa in a local server using galaxy and after 17 days it gave me the results. When the software was running, I used the htop command and I saw that the process was using 1 CPU while the server has 32 CPU. I do not know if it's a problem with the installation of galaxy or if there is a way to increase the computing power. Please help me 

greetings

Marco

 

bwa cpu galaxy • 2.1k views
ADD COMMENTlink modified 3.9 years ago by jmchilton1.1k • written 3.9 years ago by marcoabbestia30
3
gravatar for jmchilton
3.9 years ago by
jmchilton1.1k
United States
jmchilton1.1k wrote:

I assume you haven't configured a job manager like SLURM or Torque (https://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer)?  These can help take advantage of large resources like this.

Even if you don't want to set up something like that - Galaxy can be configured to take advantage of more cores but it won't attempt to do much intelligent automatically. There is some discussion here - C: GALAXY_SLOTS not working

For your setup I might setup a config/job_conf.xml file like the following:

<job_conf>
    <plugins>
        <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="8"/>
        <plugin id="multilocal" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="2"/>
    </plugins>
    <handlers>
       <handler id="main" />
    </handlers>

  <destinations default="local">
        <destination id="local" runner="local"/>
        <destination id="multicore8" runner="multilocal">
          <param id="local_slots">8</param>
       </destination>
  </destinations>

    <tools>
      <tool id="bowtie2" destination="multicore8" />
      <tool id="bowtie"  destination="multicore8" />
      <tool id="deeptools"  destination="multicore8" />
      <tool id="cufflinks" destination="multicore8" />
      <tool id="cuffdiff"  destination="multicore8" />
      <tool id="cuffmerge" destination="multicore8" />
      <tool id="tophat2" destination="multicore8" />
    </tools>
</job_conf>

This will allow 8 single core jobs to run simultaneously and will reserve the other 16 cores to run two 8 core jobs and assigns bowtie2, bowtie, cufflinks, etc.... to run in these multi-core slots.

I hope this helps.

Update: Adjusted XML snippet to remove incorrect, extraneous attributes.

Update 2: Added missing handlers tag preventing this file from loading.

ADD COMMENTlink modified 3.3 years ago • written 3.9 years ago by jmchilton1.1k
1

Hi

I'm a complete Galaxy newbie, and am trying to configure a 16-core workstation to run, with an emphasis on speed-per-job. In other words, I'd like to dedicate as many cores to a given job as possible, and will run jobs sequentially.

This example job_conf.xml file was very well laid out and explained, and I've attempted to modify it to run 14 cores on the listed tools. However, it's not working, and it looks like only one core is working when I run Cufflinks (for example). Any idea what I'm doing wrong? Are there any other configuration files that I need to tweak? I have a feeling I'm doing some that is obviously wrong, and I apologize if I am.

<job_conf>
    <plugins>
        <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="2"/>
        <plugin id="multilocal" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="1"/>
    </plugins>
    <handlers>
       <handler id="main" />
    </handlers>

  <destinations default="local">
        <destination id="local" runner="local"/>
        <destination id="multicore14" runner="multilocal">
          <param id="local_slots">14</param>
       </destination>
  </destinations>

    <tools>
      <tool id="cuffdiff" destination="multicore14" />
      <tool id="cufflinks"  destination="multicore14" />
      <tool id="cuffmerge"  destination="multicore14" />
      <tool id="fastq_groomer" destination="multicore14" />
      <tool id="fastqc"  destination="multicore14" />
      <tool id="rgrnastar" destination="multicore14" />
      <tool id="trimmomatic" destination="multicore14" />
    </tools>
</job_conf>
ADD REPLYlink written 18 months ago by Andrew Norman20

Hi Andrew, Would you please post this as a new question? I'll come back after and remove this comment so there will not be duplication. Thanks! Jen, Galaxy team

ADD REPLYlink written 18 months ago by Jennifer Hillman Jackson25k

Sorry, I missed that you posted. https://biostar.usegalaxy.org/p/23270/. Hopefully, you worked this out already!

Related Q&A https://biostar.usegalaxy.org/p/21044/, https://biostar.usegalaxy.org/p/16880/

ADD REPLYlink modified 8 months ago • written 8 months ago by Jennifer Hillman Jackson25k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 148 users visited in the last hour