Walltime on job

Question: Walltime on job_config

3.8 years ago by

Hi All,

I am installing Galaxy in our HPC facility and I would like some advices regarding setting walltime for jobs in the job_config.xml.

Is there a way to know what the main Galaxy job_config looks like regarding to walltime? Specifically I would like to know what is the appropriate walltime for tools like BWA vs Blast, or even what is the recommended walltime for sort or fastQC?

I understand that this depends on the size of the dataset. But what I am looking for are just a guideline to help me set up in our own facility.

Thanks a lot

galaxy tools • 822 views

ADD COMMENT • link •

modified 3.8 years ago by Nate Coraor ♦ 3.2k • written 3.8 years ago by David Anderson • 40

3.8 years ago by

Jennifer Hillman Jackson ♦ 25k

United States

Jennifer Hillman Jackson ♦ 25k wrote:

Hello,

Jobs tend to fail for memory more often than wall time on the public Main Galaxy, but much depends on your own data and supported tools.

The compute given to all jobs is currently:

36 hour walltime, 6 cores, 32 GB memory, low wait

We have a beta "long running" cluster (currently not in use) with alternate configurations. These can be found here:
http://wiki.galaxyproject.org/Main#Beta:_Job_execution_on_Stampede

Other running their own servers are welcome to offer more help. Jen, Galaxy team

ADD COMMENT • link written 3.8 years ago by Jennifer Hillman Jackson ♦ 25k

Thanks a lot Jennifer.

I am actually glad to know that you don't do a per tool configuration.

In our case we have a 24 cores 32GB of RAM. So I can easily emulate what you guys are doing.

But if someone else can share their config I would very much appreciate.

Cheers

ADD REPLY • link written 3.8 years ago by David Anderson • 40

3.8 years ago by

Nate Coraor ♦ 3.2k

United States

Nate Coraor ♦ 3.2k wrote:

Hi David,

The job_conf.xml for usegalaxy.org can be found here:

https://github.com/galaxyproject/usegalaxy-playbook/blob/master/templates/galaxy/usegalaxy.org/config/job_conf.xml.j2

As Jen mentioned, we currently have a 36 hour walltime, although there are certainly jobs that could run longer - we probably set it lower than what you might do with your own resources, for quality of service purposes at the public site.

ADD COMMENT • link written 3.8 years ago by Nate Coraor ♦ 3.2k

Thanks a lot Nate.
This was very helpful.

Just one more question. In the galaxy test I see that bowtie2, for example, has an option for the users choose the 'Job Resources Parameters' .
Is this a feature that can be accomplished with the most up-to-date (stable) code or this is something not in the stable version yet? If it is possible in the stable version could you point me out to a documentation where I can learn how to do it?
I would love to give the user the chance to choose different type of queues.

Thank a lot.

ADD REPLY • link modified 3.8 years ago • written 3.8 years ago by David Anderson • 40

Please log in to add an answer.

Similar posts • Search »