Question: Assign more CPU Cores for Jobs
2
gravatar for h.negronpagan
4 weeks ago by
h.negronpagan30 wrote:

Hi there,

This is my first post here.

I am currently running two local galaxy instances. My two computers have the following CPU Specs:

  1. Intel Xeon E5-2670 8-Core 16-Threads on Ubuntu 16.04
  2. Intel i7-6800k 6-Core 12-Threads on Ubuntu 16.04

I would like for each galaxy instance to run one simultaneous job, and use 6-Core 12-Threads on my first computer and 4-Core 8-Threads on my second computer. Note: I'm not trying to merge the instances. They're both sepparate instances. I have tried following this (https://biostar.usegalaxy.org/p/10158/) guide, but what that did was make my local instance run two simultaneous, single core jobs from the job queue.

The tools I am using are: cufflinks, tophat, bowtie, and fastq_groomer.

Thank you very much in advance.

Harry

rna-seq galaxy • 103 views
ADD COMMENTlink modified 4 weeks ago by Martin Čech ♦♦ 4.0k • written 4 weeks ago by h.negronpagan30

Are you trying to run a single job on two machines or am I misunderstanding this? That is not a feasible way of doing things I believe since there are other (&shared) hardware pieces that need to be in sync (like memory) to make the computation happen, we usually omit them when 'counting cores capacity' but they are integral to the process thus 'sharing' cores across machines is not a good idea I think.

ADD REPLYlink written 4 weeks ago by Martin Čech ♦♦ 4.0k
1

No, I will be running each machine locally (not from LAN) and each will have its own Galaxy Installation. I don't want to share resources between them or anything. One of these computers is at the lab, and the other is at my home (I work mostly from home). The problem is that the default galaxy configuration is not set to use all (or at least most) of the CPU resources, and the instructions provided in the link that I posted didn't help in making use of the extra CPU cores. It just limited the number of simultaneous jobs to two, but each only used one CPU core.

Edit: If I change the value on this line from 2 -> 4, then I can run 4 single core jobs simultaneously. Still can't get multicore jobs to run.

<plugin id="multilocal" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="2"/>

ADD REPLYlink modified 4 weeks ago • written 4 weeks ago by h.negronpagan30

enter image description here

This is my htop in terminal while I run fastq_groomer in galaxy

ADD REPLYlink modified 4 weeks ago • written 4 weeks ago by h.negronpagan30
0
gravatar for Martin Čech
4 weeks ago by
Martin Čech ♦♦ 4.0k
United States
Martin Čech ♦♦ 4.0k wrote:

Thanks for clarification.

cufflinks, tophat, bowtie Galaxy tools are multithread-ready as they have the GALAXY_SLOTS property in the wrapper defined.

There is some more documentation on how to set up slots in the job_conf here and here. In short what you want to do is to add <param id="local_slots">8</param> to your <destination> definition.

You can also use this script to check what the settings look like.

fastqgroomer from devteam seems to be a single-thread only but there is a different repository that may support parallelization.

ADD COMMENTlink written 4 weeks ago by Martin Čech ♦♦ 4.0k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 96 users visited in the last hour