Question: Running Galaxy on a 64 CPU Core, 512 Gig RAM Server
0
gravatar for carlos
15 months ago by
carlos30
United States
carlos30 wrote:

Hi Everyone, we just setup a new dedicated Galaxy server that has 64 CPU Cores and 512 Gigs of RAM. We were hoping to decrease the run time for our pipelines, but we don't seem to notice any major difference from a former server with 4 CPU cores and 64 Gigs of RAM that was running Galaxy as well.


I read in one posting that we may need to setup a resource manager such as SLURM or Torque. Does this also involve setting up the dramaa or any other API so that jobs are routed to the resource manager? Is there any good updated guideline on how to do this? Please note that we are using a single stand alone server, not a cluster. We tried using our cluster that runs SLURM, but we could not manage to configure the runners correctly. For this reason we pulled a single node and setup Galaxy as a stand alone production server.


Please advise. Thank you !

ADD COMMENTlink modified 15 months ago • written 15 months ago by carlos30
2
gravatar for Devon Ryan
15 months ago by
Devon Ryan1.9k
Germany
Devon Ryan1.9k wrote:

I'd work more to get Galaxy to just use your cluster (ours uses our slurm cluster), that's a better long-term solution.

You don't need to use a resource manager for a single server, what you likely need to do is (A) increase the number of concurrent jobs that can run per user (B) increase the number of job handlers and (C) increase the number of cores used per job. Most likely you need mostly (C).

ADD COMMENTlink written 15 months ago by Devon Ryan1.9k
0
gravatar for carlos
15 months ago by
carlos30
United States
carlos30 wrote:

Davon, thanks for the information provided. Yes, I guess we'll try again to use SLURM on the cluster, but we did encounter many issues when we tried in the past. Would you have a good reference guide that I can use? On the cluster we are running Centos 7.3.

Meanwhile, I'll see if I can tweak the settings as you suggested. I assume all of these are found in the galaxy.ini, correct?

CL

ADD COMMENTlink written 15 months ago by carlos30
1

Advanced admin training materials are here: https://github.com/galaxyproject/dagobah-training

ADD REPLYlink written 15 months ago by Jennifer Hillman Jackson25k

I guess I just didn't run into many problems, just with the python environment (largely solved with the introduction of conda) and ensuring the file system was shared.

ADD REPLYlink written 15 months ago by Devon Ryan1.9k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 173 users visited in the last hour