Cloud Man - job triggering

Question: Cloud Man - job triggering

3.5 years ago by

When I have jobs that have different memory and CPU requirements, is it possible to somehow specify what kind of node I want this job to be run on?

For example a high memory process/job I want to run on a amazon memory node, a high CPU process/job on a high CPU node. Sometimes I have jobs that can only have a single thread, but they need a lot of memory. A dirty way would be obviously (if I'm alone on the server) to start the corresponding computing node and then start the job.
BUT: I want this to work also in a workflow process and (optional) also for many users in parallel.

cloud man • 810 views

ADD COMMENT • link •

modified 3.5 years ago by Dannon Baker ♦ 3.7k • written 3.5 years ago by vowinkel.alexander • 10

3.5 years ago by

Dannon Baker ♦ 3.7k

United States

Dannon Baker ♦ 3.7k wrote:

On a cloudman-based instance, this might be a good bit of system-level work to get functional. Galaxy supports the notion of job destinations[1], but currently (to provide the simplest and most robust interface) cloudman assembles a 'flat' set of nodes to dispatch to regardless of what sort you have. Some nodes may get more slots than others, but all slots are equal as destinations with respect to memory available.

Someone else may chime in with better info, but hopefully this gives you a start as to what must be done.

https://wiki.galaxyproject.org/Admin/Config/Jobs#Mapping_Tools_To_Destinations

ADD COMMENT • link written 3.5 years ago by Dannon Baker ♦ 3.7k

And how is the "memory available" estimated? If I have a job, taking 2h, first hour little mem, 2nd hour very much ram. How is assured that an instance doesn't run out of memory (e.g. because another process is put on the same node during the 1st hour)?

ADD REPLY • link written 3.5 years ago by vowinkel.alexander • 10

This is more of a job scheduler question/task than a Galaxy configuration question. To do this, you'd probably want to map a single job to a node vs. to a core (i.e., 1 node = 1 queue slot; for example, see http://www.softpanorama.org/HPC/Grid_engine/sge_queues.shtml#Number_of_slots_per_per_host).

ADD REPLY • link written 3.5 years ago by Enis Afgan • 690

Similar posts • Search »