As Devon said you need to change the job runner, do you have any scheduler on your serv ?
Here is my setup with HTCondor, if you have any other one you will need to modify this obviously. You need to take care that the IPs are matching each others in the differents configuration files. This setup might not be perfect or even "good", I would gladly accept any suggestions. A little detail is that some parts of my config are specific to my hospital environment (proxy etc).
If i forgot to mention something important, please let me know guys.
in Galaxy.ini :
[uwsgi]
processes = 8
stats = 127.0.0.1:9191
socket = 127.0.0.1:4001
pythonpath = lib
threads = 4
logto = /home/galaxy/galaxy/uwsgi.log
single-interpreter = true
master = True
[server:handler0]
use = egg:Paste#http
port = 8090
host = 127.0.0.1
use_threadpool = true
threadpool_workers = 5
[server:handler1]
use = egg:Paste#http
port = 8091
host = 127.0.0.1
use_threadpool = true
threadpool_workers = 5
In job_conf.xml (you can modify the user job limit as you want, i think the limit should be the number of core).
<?xml version="1.0"?>
<job_conf>
<plugins>
<plugin id="condor" type="runner" load="galaxy.jobs.runners.condor:CondorJobRunner" workers="4"/>
<plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner"/>
</plugins>
<handlers default="handlers">
<handler id="handler0" tags="handlers"/>
<handler id="handler1" tags="handlers"/>
</handlers>
<destinations default="condor">
<destination id="condor" runner="condor" />
<destination id="local" runner="local"/>
</destinations>
<limits>
<limit type="concurrent_jobs">14</limit>
<limit type="registered_user_concurrent_jobs">13</limit>
<limit type="unregistered_user_concurrent_jobs">0</limit>
<limit type="job_walltime">48:00:00</limit>
</limits>
</job_conf>
In Nginx conf I think this is the relevant part:
location /galaxy {
uwsgi_pass 127.0.0.1:4001;
uwsgi_param UWSGI_SCHEME $scheme;
include uwsgi_params;
}
location /for all static like suggested in the doc{
}
location /_x_accel_redirect/ {
internal;
alias /;
}
In /etc/condor/condor_config.local you specify 1 slot partitionable so you can launch job using 1 core up to all your core:
NUM_SLOTS=1
NUM_SLOTS_TYPE_1=1
SLOT_TYPE_1=100%
SLOT_TYPE_1_PARTITIONABLE=true
In supervisor, here i didn't put NGINX in my supervisor conf but i think i should do it soon :
[program:galaxy_uwsgi]
command = /home/galaxy/galaxy/.venv/bin/uwsgi --virtualenv /home/galaxy/galaxy/.venv --ini-paste /home/galaxy/galaxy/config/galaxy.ini --logdate --master --processes 2 --threads 2 --logto /home/galaxy/galaxy/uwsgi.log --http-socket 127.0.0.1:4001 --pythonpath lib --stats 127.0.0.1:9191
directory = /home/galaxy/galaxy
umask = 022
autostart = true
autorestart = true
startsecs = 20
user = galaxy
environment = PATH="/home/galaxy/galaxy/.venv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/galaxy/script/whatever",PYTHONHOME="/home/galaxy/galaxy/.venv",http_proxy="ifneeded:port",https_proxy="ifneeded:port"
numprocs = 1
stopsignal = INT
startretries = 15
[program:handler]
command = /home/galaxy/galaxy/.venv/bin/python ./lib/galaxy/main.py -c /home/galaxy/galaxy/config/galaxy.ini --server-name=handler%(process_num)s --log-file=/home/galaxy/galaxy/handler%(process_num)s.log
directory = /home/galaxy/galaxy
process_name = handler%(process_num)s
numprocs = 2
umask = 022
autostart = true
autorestart = true
startsecs = 20
user = galaxy
environment = PYTHONHOME="/home/galaxy/galaxy/.venv",http_proxy="ifneeded:port",https_proxy="ifneeded:port"
startretries = 15
[group:galaxy]
programs = handler, galaxy_uwsgi
I hope this will help you. I had a hard time finding all of these and to understand why some part didn't work (proxy, again, and other little detail like quote needed in my supervisor config only on my server). But I had some great help from all the galaxy team.
Could you give a bit more information about what exactly you need to scale that you can't figure out how to? The biggest thing to change is the job runner, specifically telling different tools that they can use more cores.
Tank you first, I have a 32 core server, I just follow the https://wiki.galaxyproject.org/Admin/GetGalaxy for download galaxy to my server, I want to user galaxy for RNA-Seq, for now I just use bowtie, tophat, cufflinks that tools, follow the https://wiki.galaxyproject.org/Admin/Config/Performance/Scaling install uWSGI, nginx, supervisor and config they configure file. I also change the galaxy configure file galaxy.ini. I want to know the sample way to Scaling galaxy.
thank you again and my terrible English.
I skipped all of that and just used the docker container. After that, it's just a matter of changing jobs_conf.xml to allow the desired number of threads for each tool. If you already have Galaxy up and running, then have a look at that next.
Hi, I try install docker galaxy on server like you said. But it can't work well, I install on a Centos server, I think is the compatibility problem? Is the docker galaxy just install on Ubuntu Server? And how to config the galaxy.ini file? Is that use "docker run -i -t -p 8080:80 bgruening/galaxy-stable /bin/bash" command to interactive session and edit "galaxy-central/confgi/config.ini" file, then run startup script to run the galaxy?
I happen to run that docker container on a CentOS 7 server, so there's no compatibility issue (the version of linux in the container and on the server don't need to match). The config file is /etc/galaxy/galaxy.ini, though you should really export a volume in for it to save stuff to (e.g.
-v /some/path/for/galaxy:/export). That allows for persistence. After you edit it, you shouldsupervisorctl restart galaxy:.I already execute "sudo docker run -d -p 8080:80 -p 9002:9002 -v /home/user/galaxy_storage/:/export/ bgruening/galaxy-stable" to run galaxy container on 8080 and supervisor on 9002 :
But when I run "FASTQ Grooomer" convert file format
it can run the job
but just one python thread run
I know the config file store on server "home/user/galaxy_storage/galaxy-central/config", so I copy "galaxy.ini.docker_sample" as galaxy.ini and change it
and next, I copy "job_conf.xml.docker_sample " as job_conf.xml, then restart the galaxy, it also run one python thread, I don't know why, so I edit the job_conf.xml like:
I really don't know what to do, that hurt my heart!!!
That job will only ever run on one thread, that's the case with many programs and has nothing to do with Galaxy. If you want to test that multithreading is working, use an aligner like bowtie2 to test it.