Question: switch between cloudman and local - use condor?
0
gravatar for ggeo1
17 months ago by
ggeo110
ggeo110 wrote:

Hello,

I have successfuly setup cloudman on amazon.

I want to ask this.

As a user I am running galaxy, either on my local PC, either on a server (not amazon!).

Is there a way to select where the workflow will run?For example, I want to run on my local.Next time I want to select to run on the amazon(cloudman).

So, there must be a way to communicate between galaxy and cloudman?An API maybe?

Thanks!

api galaxy cloudman • 544 views
ADD COMMENTlink modified 17 months ago • written 17 months ago by ggeo110
1
gravatar for onspotproductions
17 months ago by
United States
onspotproductions40 wrote:

There is a way to do this but it requires an external job runner to handle it. If I am correct, the DRM of choice to execute these types of runs has been HTCondor, but it will require some modification to the galaxy settings files to handle which jobs will run on each system. Ideally you would want it setup to run large jobs that are not parallizable on a single large EC2 instance (such as mapping) and all smaller jobs on the local machine/ server.

ADD COMMENTlink written 17 months ago by onspotproductions40

Thanks for the info.I was hopping that galaxy has an API for this job though.I will check condor.

ADD REPLYlink written 17 months ago by ggeo110

Hello, have you used condor?Do you know how to setup?Do I need to install condor on my local PC?Or as an ec2 innstance?And then, what about the cloudman on ec2?How can I combine/setup everything?Thanks!

ADD REPLYlink written 17 months ago by ggeo110

I have not done this yet myself and am not sure how to do it in combination with cloudman, but the setup would be and ec2 instance with galaxy and htcondor installed. Cloudman allows for dynamic creation of clusters which would be useful in this scenario, but I haven't tried this method yet. You then bind your local cluster to a fileshare of the location you installed galaxy. Your local cluster must also have htcondor installed. You would then refer to htcondor flocking for running jobs on remote clusters.

ADD REPLYlink written 17 months ago by onspotproductions40

One thing I should have mentioned is that all machines require mounting of the same NFS and there are some limitations when it comes to using NFS. You may want to look into NFSv4 to do this.

ADD REPLYlink written 17 months ago by onspotproductions40
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 65 users visited in the last hour