switch between cloudman and local

Question: switch between cloudman and local - use condor?

2.3 years ago by

ggeo1 • 10

ggeo1 • 10 wrote:

Hello,

I have successfuly setup cloudman on amazon.

I want to ask this.

As a user I am running galaxy, either on my local PC, either on a server (not amazon!).

Is there a way to select where the workflow will run?For example, I want to run on my local.Next time I want to select to run on the amazon(cloudman).

So, there must be a way to communicate between galaxy and cloudman?An API maybe?

Thanks!

api galaxy cloudman • 830 views

ADD COMMENT • link •

modified 2.3 years ago • written 2.3 years ago by ggeo1 • 10

2.3 years ago by

onspotproductions • 40

United States

onspotproductions • 40 wrote:

There is a way to do this but it requires an external job runner to handle it. If I am correct, the DRM of choice to execute these types of runs has been HTCondor, but it will require some modification to the galaxy settings files to handle which jobs will run on each system. Ideally you would want it setup to run large jobs that are not parallizable on a single large EC2 instance (such as mapping) and all smaller jobs on the local machine/ server.

ADD COMMENT • link written 2.3 years ago by onspotproductions • 40

Thanks for the info.I was hopping that galaxy has an API for this job though.I will check condor.

ADD REPLY • link written 2.3 years ago by ggeo1 • 10

Hello, have you used condor?Do you know how to setup?Do I need to install condor on my local PC?Or as an ec2 innstance?And then, what about the cloudman on ec2?How can I combine/setup everything?Thanks!

ADD REPLY • link written 2.3 years ago by ggeo1 • 10

I have not done this yet myself and am not sure how to do it in combination with cloudman, but the setup would be and ec2 instance with galaxy and htcondor installed. Cloudman allows for dynamic creation of clusters which would be useful in this scenario, but I haven't tried this method yet. You then bind your local cluster to a fileshare of the location you installed galaxy. Your local cluster must also have htcondor installed. You would then refer to htcondor flocking for running jobs on remote clusters.

ADD REPLY • link written 2.3 years ago by onspotproductions • 40

One thing I should have mentioned is that all machines require mounting of the same NFS and there are some limitations when it comes to using NFS. You may want to look into NFSv4 to do this.

ADD REPLY • link written 2.3 years ago by onspotproductions • 40

Similar posts • Search »