Assign more CPU Cores for Jobs

Question: Assign more CPU Cores for Jobs

15 months ago by

h.negronpagan • 30

h.negronpagan • 30 wrote:

Hi there,

This is my first post here.

I am currently running two local galaxy instances. My two computers have the following CPU Specs:

Intel Xeon E5-2670 8-Core 16-Threads on Ubuntu 16.04
Intel i7-6800k 6-Core 12-Threads on Ubuntu 16.04

I would like for each galaxy instance to run one simultaneous job, and use 6-Core 12-Threads on my first computer and 4-Core 8-Threads on my second computer. Note: I'm not trying to merge the instances. They're both sepparate instances. I have tried following this (https://biostar.usegalaxy.org/p/10158/) guide, but what that did was make my local instance run two simultaneous, single core jobs from the job queue.

The tools I am using are: cufflinks, tophat, bowtie, and fastq_groomer.

Thank you very much in advance.

Harry

rna-seq galaxy • 634 views

ADD COMMENT • link •

modified 15 months ago by Martin Čech ♦♦ 4.9k • written 15 months ago by h.negronpagan • 30

Are you trying to run a single job on two machines or am I misunderstanding this? That is not a feasible way of doing things I believe since there are other (&shared) hardware pieces that need to be in sync (like memory) to make the computation happen, we usually omit them when 'counting cores capacity' but they are integral to the process thus 'sharing' cores across machines is not a good idea I think.

ADD REPLY • link written 15 months ago by Martin Čech ♦♦ 4.9k

No, I will be running each machine locally (not from LAN) and each will have its own Galaxy Installation. I don't want to share resources between them or anything. One of these computers is at the lab, and the other is at my home (I work mostly from home). The problem is that the default galaxy configuration is not set to use all (or at least most) of the CPU resources, and the instructions provided in the link that I posted didn't help in making use of the extra CPU cores. It just limited the number of simultaneous jobs to two, but each only used one CPU core.

Edit: If I change the value on this line from 2 -> 4, then I can run 4 single core jobs simultaneously. Still can't get multicore jobs to run.

ADD REPLY • link modified 15 months ago • written 15 months ago by h.negronpagan • 30

enter image description here

This is my htop in terminal while I run fastq_groomer in galaxy

ADD REPLY • link modified 15 months ago • written 15 months ago by h.negronpagan • 30

15 months ago by

Martin Čech ♦♦ 4.9k

United States

Martin Čech ♦♦ 4.9k wrote:

Thanks for clarification.

cufflinks, tophat, bowtie Galaxy tools are multithread-ready as they have the GALAXY_SLOTS property in the wrapper defined.

There is some more documentation on how to set up slots in the job_conf here and here. In short what you want to do is to add <param id="local_slots">8</param> to your <destination> definition.

You can also use this script to check what the settings look like.

fastqgroomer from devteam seems to be a single-thread only but there is a different repository that may support parallelization.

ADD COMMENT • link written 15 months ago by Martin Čech ♦♦ 4.9k

Similar posts • Search »