CPU utilization with galaxy

Question: CPU utilization with galaxy

3.9 years ago by

European Union

Hi,

I launched an analysis of alignment with bwa in a local server using galaxy and after 17 days it gave me the results. When the software was running, I used the htop command and I saw that the process was using 1 CPU while the server has 32 CPU. I do not know if it's a problem with the installation of galaxy or if there is a way to increase the computing power. Please help me

greetings

Marco

bwa cpu galaxy • 2.1k views

ADD COMMENT • link •

modified 3.9 years ago by jmchilton ♦ 1.1k • written 3.9 years ago by marcoabbestia • 30

3.9 years ago by

jmchilton ♦ 1.1k

United States

jmchilton ♦ 1.1k wrote:

I assume you haven't configured a job manager like SLURM or Torque (https://wiki.galaxyproject.org/Admin/Config/Performance/ProductionServer)? These can help take advantage of large resources like this.

Even if you don't want to set up something like that - Galaxy can be configured to take advantage of more cores but it won't attempt to do much intelligent automatically. There is some discussion here - C: GALAXY_SLOTS not working.

For your setup I might setup a config/job_conf.xml file like the following:

<job_conf>
<plugins>
<plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="8"/>
<plugin id="multilocal" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="2"/>
</plugins>
<handlers>
<handler id="main" />
</handlers>

This will allow 8 single core jobs to run simultaneously and will reserve the other 16 cores to run two 8 core jobs and assigns bowtie2, bowtie, cufflinks, etc.... to run in these multi-core slots.

I hope this helps.

Update: Adjusted XML snippet to remove incorrect, extraneous attributes.

Update 2: Added missing handlers tag preventing this file from loading.

ADD COMMENT • link modified 3.3 years ago • written 3.9 years ago by jmchilton ♦ 1.1k

I'm a complete Galaxy newbie, and am trying to configure a 16-core workstation to run, with an emphasis on speed-per-job. In other words, I'd like to dedicate as many cores to a given job as possible, and will run jobs sequentially.

This example job_conf.xml file was very well laid out and explained, and I've attempted to modify it to run 14 cores on the listed tools. However, it's not working, and it looks like only one core is working when I run Cufflinks (for example). Any idea what I'm doing wrong? Are there any other configuration files that I need to tweak? I have a feeling I'm doing some that is obviously wrong, and I apologize if I am.

<job_conf>
    <plugins>
        <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="2"/>
        <plugin id="multilocal" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="1"/>
    </plugins>
    <handlers>
       <handler id="main" />
    </handlers>

  <destinations default="local">
        <destination id="local" runner="local"/>
        <destination id="multicore14" runner="multilocal">
          <param id="local_slots">14</param>
       </destination>
  </destinations>

    <tools>
      <tool id="cuffdiff" destination="multicore14" />
      <tool id="cufflinks"  destination="multicore14" />
      <tool id="cuffmerge"  destination="multicore14" />
      <tool id="fastq_groomer" destination="multicore14" />
      <tool id="fastqc"  destination="multicore14" />
      <tool id="rgrnastar" destination="multicore14" />
      <tool id="trimmomatic" destination="multicore14" />
    </tools>
</job_conf>

ADD REPLY • link written 18 months ago by Andrew Norman • 20

Hi Andrew, Would you please post this as a new question? I'll come back after and remove this comment so there will not be duplication. Thanks! Jen, Galaxy team

ADD REPLY • link written 18 months ago by Jennifer Hillman Jackson ♦ 25k

Sorry, I missed that you posted. https://biostar.usegalaxy.org/p/23270/. Hopefully, you worked this out already!

ADD REPLY • link modified 8 months ago • written 8 months ago by Jennifer Hillman Jackson ♦ 25k

Similar posts • Search »