Question: Configure galaxy run on a multi-core server
0
gravatar for fuyunzhishang
2.1 years ago by
fuyunzhishang0 wrote:

That question trouble me very long time, I spend lot time to figure it. I read the wiki: https://wiki.galaxyproject.org/Admin/Config/Performance/Scaling again and again, I still don't know what the nginx, uWSGI, wsgi, supervisor how to config, document not clearly show step by step, I just want use galaxy replace the analysis by command, please help me. I will be very thankful.

admin jobs config local • 1.4k views
ADD COMMENTlink modified 2.1 years ago by Jennifer Hillman Jackson25k • written 2.1 years ago by fuyunzhishang0

Could you give a bit more information about what exactly you need to scale that you can't figure out how to? The biggest thing to change is the job runner, specifically telling different tools that they can use more cores.

ADD REPLYlink written 2.1 years ago by Devon Ryan1.9k

Tank you first, I have a 32 core server, I just follow the https://wiki.galaxyproject.org/Admin/GetGalaxy for download galaxy to my server, I want to user galaxy for RNA-Seq, for now I just use bowtie, tophat, cufflinks that tools, follow the https://wiki.galaxyproject.org/Admin/Config/Performance/Scaling install uWSGI, nginx, supervisor and config they configure file. I also change the galaxy configure file galaxy.ini. I want to know the sample way to Scaling galaxy.

thank you again and my terrible English.

ADD REPLYlink modified 2.1 years ago • written 2.1 years ago by fuyunzhishang0

I skipped all of that and just used the docker container. After that, it's just a matter of changing jobs_conf.xml to allow the desired number of threads for each tool. If you already have Galaxy up and running, then have a look at that next.

ADD REPLYlink written 2.1 years ago by Devon Ryan1.9k

Hi, I try install docker galaxy on server like you said. But it can't work well, I install on a Centos server, I think is the compatibility problem? Is the docker galaxy just install on Ubuntu Server? And how to config the galaxy.ini file? Is that use "docker run -i -t -p 8080:80 bgruening/galaxy-stable /bin/bash" command to interactive session and edit "galaxy-central/confgi/config.ini" file, then run startup script to run the galaxy?

ADD REPLYlink modified 2.1 years ago • written 2.1 years ago by fuyunzhishang0

I happen to run that docker container on a CentOS 7 server, so there's no compatibility issue (the version of linux in the container and on the server don't need to match). The config file is /etc/galaxy/galaxy.ini, though you should really export a volume in for it to save stuff to (e.g. -v /some/path/for/galaxy:/export). That allows for persistence. After you edit it, you should supervisorctl restart galaxy:.

ADD REPLYlink written 2.1 years ago by Devon Ryan1.9k

I already execute "sudo docker run -d -p 8080:80 -p 9002:9002 -v /home/user/galaxy_storage/:/export/ bgruening/galaxy-stable" to run galaxy container on 8080 and supervisor on 9002 : enter image description here

enter image description here

But when I run "FASTQ Grooomer" convert file format enter image description here it can run the job enter image description here but just one python thread run enter image description here enter image description here

I know the config file store on server "home/user/galaxy_storage/galaxy-central/config", so I copy "galaxy.ini.docker_sample" as galaxy.ini and change it

enter image description here enter image description here

and next, I copy "job_conf.xml.docker_sample " as job_conf.xml, then restart the galaxy, it also run one python thread, I don't know why, so I edit the job_conf.xml like: enter image description here

I really don't know what to do, that hurt my heart!!!

ADD REPLYlink modified 2.1 years ago • written 2.1 years ago by fuyunzhishang0

That job will only ever run on one thread, that's the case with many programs and has nothing to do with Galaxy. If you want to test that multithreading is working, use an aligner like bowtie2 to test it.

ADD REPLYlink written 2.1 years ago by Devon Ryan1.9k
0
gravatar for christophe.habib
2.1 years ago by
France
christophe.habib340 wrote:

As Devon said you need to change the job runner, do you have any scheduler on your serv ?

Here is my setup with HTCondor, if you have any other one you will need to modify this obviously. You need to take care that the IPs are matching each others in the differents configuration files. This setup might not be perfect or even "good", I would gladly accept any suggestions. A little detail is that some parts of my config are specific to my hospital environment (proxy etc).

If i forgot to mention something important, please let me know guys.

in Galaxy.ini :

[uwsgi]
processes = 8
stats = 127.0.0.1:9191
socket = 127.0.0.1:4001
pythonpath = lib
threads = 4
logto = /home/galaxy/galaxy/uwsgi.log

single-interpreter = true
master = True

[server:handler0]
use = egg:Paste#http
port = 8090
host = 127.0.0.1
use_threadpool = true
threadpool_workers = 5

[server:handler1]
use = egg:Paste#http
port = 8091
host = 127.0.0.1
use_threadpool = true
threadpool_workers = 5

In job_conf.xml (you can modify the user job limit as you want, i think the limit should be the number of core).

<?xml version="1.0"?>
<job_conf>
    <plugins>
        <plugin id="condor" type="runner" load="galaxy.jobs.runners.condor:CondorJobRunner" workers="4"/>
        <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner"/>
    </plugins>

    <handlers default="handlers">
        <handler id="handler0" tags="handlers"/>
        <handler id="handler1" tags="handlers"/>
    </handlers>

    <destinations default="condor">
        <destination id="condor" runner="condor" />
        <destination id="local" runner="local"/>
    </destinations>

    <limits>
        <limit type="concurrent_jobs">14</limit>
        <limit type="registered_user_concurrent_jobs">13</limit>
        <limit type="unregistered_user_concurrent_jobs">0</limit>
        <limit type="job_walltime">48:00:00</limit>
    </limits>
</job_conf>

In Nginx conf I think this is the relevant part:

    location /galaxy {
        uwsgi_pass 127.0.0.1:4001;
        uwsgi_param UWSGI_SCHEME $scheme;
        include uwsgi_params;
    }
    location /for all static like suggested in the doc{
  }

location /_x_accel_redirect/ {
    internal;
    alias /;
}

In /etc/condor/condor_config.local you specify 1 slot partitionable so you can launch job using 1 core up to all your core:

NUM_SLOTS=1
NUM_SLOTS_TYPE_1=1
SLOT_TYPE_1=100%
SLOT_TYPE_1_PARTITIONABLE=true

In supervisor, here i didn't put NGINX in my supervisor conf but i think i should do it soon :

[program:galaxy_uwsgi]
command         = /home/galaxy/galaxy/.venv/bin/uwsgi --virtualenv /home/galaxy/galaxy/.venv --ini-paste /home/galaxy/galaxy/config/galaxy.ini --logdate --master --processes 2 --threads 2 --logto /home/galaxy/galaxy/uwsgi.log --http-socket 127.0.0.1:4001 --pythonpath lib --stats 127.0.0.1:9191
directory       = /home/galaxy/galaxy
umask           = 022
autostart       = true
autorestart     = true
startsecs       = 20
user            = galaxy
environment     = PATH="/home/galaxy/galaxy/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/galaxy/script/whatever",PYTHONHOME="/home/galaxy/galaxy/.venv",http_proxy="ifneeded:port",https_proxy="ifneeded:port"
numprocs        = 1
stopsignal      = INT
startretries    = 15

[program:handler]
command         = /home/galaxy/galaxy/.venv/bin/python ./lib/galaxy/main.py -c /home/galaxy/galaxy/config/galaxy.ini --server-name=handler%(process_num)s --log-file=/home/galaxy/galaxy/handler%(process_num)s.log
directory       = /home/galaxy/galaxy
process_name    = handler%(process_num)s
numprocs        = 2
umask           = 022
autostart       = true
autorestart     = true
startsecs       = 20
user            = galaxy
environment     = PYTHONHOME="/home/galaxy/galaxy/.venv",http_proxy="ifneeded:port",https_proxy="ifneeded:port"
startretries    = 15

[group:galaxy]
programs = handler, galaxy_uwsgi

I hope this will help you. I had a hard time finding all of these and to understand why some part didn't work (proxy, again, and other little detail like quote needed in my supervisor config only on my server). But I had some great help from all the galaxy team.

ADD COMMENTlink modified 2.1 years ago • written 2.1 years ago by christophe.habib340

Hey, thank your reply first, that means a lot for me. I use docker galaxy now, it already installed all the needed soft like uwsgi, nginx and supervisor. The problem for now is how to change the job runner in docker galaxy. I very appreciate you.

ADD REPLYlink written 2.1 years ago by fuyunzhishang0
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 151 users visited in the last hour