Latest
Open
RNA-Seq
ChIP-Seq
SNP
Assembly
Forum
Planet
All »
Home
Log In
Question: Galaxy Workflow Decomposition
1
gravatar for jtvivian
3.8 years ago by
jtvivian • 10
United States
jtvivian • 10 wrote:

Hey everyone,

Our group plans to run a variant calling pipeline on Amazon's EC2 server and in order to utilize the spot-pricing we have to handle for job termination at various steps.  Therefore, I plan to write a program that will decompose a galaxy workflow (.ga) and the associated tools (.xml) into discrete steps that can be represented as individual bash scripts or all piped through something like subprocess.

Before I start I figure I would ask the community if someone has already worked on something like this so I can use it as a reference or the tool itself.

Cheers,

John
workflow cluster galaxy cloud • 851 views
ADD COMMENT • link •
modified 3.8 years ago by Jennifer Hillman Jackson ♦ 25k • written 3.8 years ago by jtvivian • 10

Would this allow me to massively parallelize Galaxy workflows on EC2? How are you distributing the necessary tools?

ADD REPLY • link written 3.8 years ago by hannes • 0
0
gravatar for jtvivian
3.8 years ago by
jtvivian • 10
United States
jtvivian • 10 wrote:

Hi Hannes,

Once the pipeline has been decomposed into granular components then running it parallelized on EC2 should be relatively straightforward as any independent branches in the pipeline could theoretically be run on separate nodes if the data is all being shared via something like S3.  
 

In regards to your second question, tools are currently being distributed through docker containers.  Once these tools are in a final "frozen" state they should be pushed to DockerHub instead of simply shared as DockerFiles as there are potential dependency issues with building an image from a DockerFile as opposed to pulling an image from the Hub.

 

Cheers,
ADD COMMENT • link modified 3.8 years ago • written 3.8 years ago by jtvivian • 10
Please log in to add an answer.

Similar posts • Search »

Content
Help
Access
Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 173 users visited in the last hour