Question: Cloud Man - job triggering
0
gravatar for vowinkel.alexander
2.6 years ago by
vowinkel.alexander10 wrote:

When I have jobs that have different memory and CPU requirements, is it possible to somehow specify what kind of node I want this job to be run on?

For example a high memory process/job I want to run on a amazon memory node, a high CPU process/job on a high CPU node. Sometimes I have jobs that can only have a single thread, but they need a lot of memory. A dirty way would be obviously (if I'm alone on the server) to start the corresponding computing node and then start the job.
BUT: I want this to work also in a workflow process and (optional) also for many users in parallel. 

cloud man • 583 views
ADD COMMENTlink modified 2.6 years ago by Dannon Baker3.7k • written 2.6 years ago by vowinkel.alexander10
0
gravatar for Dannon Baker
2.6 years ago by
Dannon Baker3.7k
United States
Dannon Baker3.7k wrote:

On a cloudman-based instance, this might be a good bit of system-level work to get functional.  Galaxy supports the notion of job destinations[1], but currently (to provide the simplest and most robust interface) cloudman assembles a 'flat' set of nodes to dispatch to regardless of what sort you have.  Some nodes may get more slots than others, but all slots are equal as destinations with respect to memory available.

Someone else may chime in with better info, but hopefully this gives you a start as to what must be done.

https://wiki.galaxyproject.org/Admin/Config/Jobs#Mapping_Tools_To_Destinations

ADD COMMENTlink written 2.6 years ago by Dannon Baker3.7k

And how is the "memory available" estimated? If I have a job, taking 2h, first hour little mem, 2nd hour very much ram. How is assured that an instance doesn't run out of memory (e.g. because another process is put on the same node during the 1st hour)?

ADD REPLYlink written 2.6 years ago by vowinkel.alexander10

This is more of a job scheduler question/task than a Galaxy configuration question. To do this, you'd probably want to map a single job to a node vs. to a core (i.e., 1 node = 1 queue slot; for example, see http://www.softpanorama.org/HPC/Grid_engine/sge_queues.shtml#Number_of_slots_per_per_host).

ADD REPLYlink written 2.6 years ago by Enis Afgan620
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 113 users visited in the last hour