Hi I have a local galaxy instance I ma trying to increase the number of default jobs that it can run beyond what i understand is a default of 4 jobs. I am an administrator,
I have looked at the information on this page which https://wiki.galaxyproject.org/Admin/Config/Jobs
If I understood correctly need to create a file with the following name /home/galaxy/galaxy/config/job_conf.xml
Which I have done with the following text
<job_conf> <plugins> <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="48"/> </plugins> <handlers> <handler id="main"/> </handlers> <destinations> <destination id="local" runner="local"/> </destinations> </job_conf>
Having done this I still am having a max number of jobs set at 4. Have I misunderstood ? Do I need to restart galaxy ? Thanks Guy
make sure you change the limits for 'destination_total_concurrent_jobs', see the example file: "job_conf.xml.sample_advanced"
Hi
I have looked in the folder /home/galaxy/galaxy/config/job_conf.xml and cannot see a file "job_conf.xml.sample_advanced". Would you be able to attach a copy or tell me where I can get a copy. What number should I assign to 'destination_total_concurrent_jobs'? 48 ? the same as for <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="48"/> .
Thanks
Guy
That file is on github.
Thanks got it
What number should I assign to 'destination_total_concurrent_jobs'? 48 ? the same as for <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="48"/> .
If you want a maximal number of concurrent jobs of 48, then set it to 48, If you wnat more jobs (maybe you wanna set up an additional runner) then set it higher
You need to restart Galaxy pretty much any time you update a file in
config/.I add similar issue before. Increasing the number of workers won't do the trick. You can read this I reported my attempt :
https://biostar.usegalaxy.org/p/16651/#16743
Hi I did have a look but to be honest it also reached the limits of systems admin capacity. Hopefully i will have more luck. Are you still working with 4 jobs? Thanks Guy
Hi Christophe
Would you be able to send me your /home/galaxy/galaxy/config/job_conf.xml file as I still cannot get a working version which increases the number of jobs Thanks
Guy
Hi Guy,
I think what you need is here in the job_conf.xml :
<limits> <limit type="concurrent_jobs">14</limit> <limit type="registered_user_concurrent_jobs">13</limit> <limit type="unregistered_user_concurrent_jobs">0</limit> <limit type="job_walltime">48:00:00</limit> </limits>
Did you try this ?
Thanks I did now restart it but due to the addition of the job_conf.xml file I now cannot restart galaxy. I get this in my log file.
Any ideas( I did have have old galaxy jobs that continued)
the error looks to be socket.error: [Errno 98] Address already in use
but I have no idea how to deal with this.
Thanks
Guy
"Address already in use" is most likely the culprit
any hints how to deal with this address issue? Thanks
Find whatever process is binding to the port and kill it.
you didn't kill the process, befor you re-started it
Thanks I sorted that out and killed it all. 'killall -u <username>'
AI can successfully restart galaxy now (after killing 'killall -u <username>). However if I have a file /home/galaxy/galaxy/config/job_conf.xml (I guess I do not need to use name "job_conf.xml.sample_advanced"
then I cannot successfully restart, presumably there is something wrong in this file.
I only altered the lines <plugin id="pbs" type="runner" load="galaxy.jobs.runners.pbs:PBSJobRunner" workers="48"/> and <limit type="destination_total_concurrent_jobs" id="local">48</limit>
is that sufficient or is there more I should have done?
Thanks for all the help Guy this is a link to the "job_conf.xml " file I am trying to use
Presumably you're getting an error message in a log somewhere, can you post it?
Edit: Are you actually trying to use the file you posted? That one seems...overly complicated, particularly if you don't happen to have all the stuff needed for all of the schedulers listed in there.
Yes I am trying to use the file I linked to can I guess you think I can simplify it I will get the log tomorrow. Thanks
Guy
Yes, you should be able to simplify it considerably (you won't need more than a couple dozen lines). I don't think that file was ever intended to be used in real life, it's just an example of various possibilities.
Only if I I have the "job_conf.xml " file I get an error in the terminal that I run the following
sh run.sh --daemon --log-file=/home/galaxy/log/galaxy.log
I get the error and nothing new in the galaxy.log
other text... Requirement already satisfied (use --upgrade to upgrade): ordereddict in ./.venv /lib/python2.6/site-packages (from kombu==3.0.30->-r requirements.txt (line 40)) You are using pip version 8.0.2+gx2, however version 8.1.2 is available. You should consider upgrading via the 'pip install --upgrade pip' command. Traceback (most recent call last): File "<string>", line 1, in <module> File "/home/galaxy/galaxy/lib/galaxy/dependencies/__init__.py", line 86, in op tional conditional = ConditionalDependencies( config_file ) File "/home/galaxy/galaxy/lib/galaxy/dependencies/__init__.py", line 20, in __ init__ self.parse_configs() File "/home/galaxy/galaxy/lib/galaxy/dependencies/__init__.py", line 29, in pa rse_configs for plugin in ElementTree.parse( job_conf_xml ).find( 'plugins' ).findall( ' plugin' ): File "/usr/lib64/python2.6/xml/etree/ElementTree.py", line 862, in parse tree.parse(source, parser) File "/usr/lib64/python2.6/xml/etree/ElementTree.py", line 586, in parse parser.feed(data) File "/usr/lib64/python2.6/xml/etree/ElementTree.py", line 1245, in feed self._parser.Parse(data, 0) xml.parsers.expat.ExpatError: not well-formed (invalid token): line 149, column 27
If you or anybody has a minimal 'job_conf.xml' file that will allow me to run more concurrent jobs I would be very grateful. I know I should be able to do this from the information on https://wiki.galaxyproject.org/Admin/Config/Jobs but it is rather difficult for me to follow.