Queue Rules For Multi-Step Workflows

Question: Queue Rules For Multi-Step Workflows

5.4 years ago by

zuzmus • 10

zuzmus • 10 wrote:

Dear Galaxy managers, I would like to ask about the queuing rules for the workflows before processed on the server? I use my customized Galaxy workflow which contains 22 following steps (basically, filtering, trimming and format change of NGS data). I guess the server is generally very busy during last weeks/months (?), so my job was waiting about 24 hours in a queue (which would not be a problem), and then the first step of the workflow was processed, but the following 21 is again/still waiting (already for another couple of hours...). It makes me wondering about the queuing rules because I expected that the whole workflow is queued as one job.. Then my question is if the whole workflow, once submitted, is listed in the queue, or does the following step queue only after the previous step is finished (which would mean to wait the whole queue for each step of the workflow...)? I routinely used those wrokflows before (months ago) without any problems... I tried to search similar question in the archive before I posted this one... Thanks a lot for your answer, My best Zuzana Musilova -- Zuzana Musilova, PhD. (zuzmus@gmail.com) Zoological Institute University of Basel Vesalgasse 1, CH-4051 Basel Switzerland - Europe

galaxy • 1.4k views

ADD COMMENT • link •

modified 5.2 years ago by Jennifer Hillman Jackson ♦ 25k • written 5.4 years ago by zuzmus • 10

5.2 years ago by

Jennifer Hillman Jackson ♦ 25k

United States

Jennifer Hillman Jackson ♦ 25k wrote:

Hi Zuzana, Jobs queue individually. But different jobs may queue to different clusters, so if there are no dependencies, you may see jobs execute out of order. Workflows are still a benefit as this places jobs in the queue sooner than waiting for each job to finished (under normal circumstances - not the extended queue times seen now during the server reconfiguration). The banner on main provides updates about the status of Main, but right now NGS jobs are in hold. We expect these to start again tomorrow and then for the final stages of the upgrades to be complete within the next few weeks. Apologies for the slow reply to your question, Jen Galaxy team -- Jennifer Hillman-Jackson http://galaxyproject.org

ADD COMMENT • link written 5.2 years ago by Jennifer Hillman Jackson ♦ 25k

Hi Jennifer, Things are going better in the main server of Galaxy since the beginning of this week: thanks for these last improvements ! However, I started worflows of NGS analysis last monday (30th of september) for two histories (I guess something similar to Zuzana's worflow). Since I started it, all the steps of the worflow are "waiting to run" ("grey status"). I tried many times to start them by clicking on "Resume paused jobs" (in the history wheel-tool menu), but nothing happened. As you said in your reply to Zuzana, NGS jobs are still in hold so far. Do you confirm me I ought to wait (and avoid cancelling the worflow to start a new one, as it would end up in more delay)? At the same time, I am doing other NGS analysis with other histories (GATK Base Quality Recalibration and Unified Genotyper) and these jobs are working well. This time, I am doing all the steps sequentially, without worflows. Do individual NGS jobs run better than worflows or is it just variable from tool to tool ? Thank you for reply and for all your constant efforts to maintain this very useful plateform in shape ! Fabrice Le 03/10/2013 00:20, Jennifer Jackson a écrit : -- Fabrice Besnard Institute of Biology of the Ecole Normale Supérieure (IBENS) 46 rue d'Ulm, 75230 Paris cedex 05, France 8th floor. Office: Room 802. Lab: Room 817. mail: fbesnard@biologie.ens.fr Tel: +33-1-44-32-39-31

ADD REPLY • link written 5.2 years ago by Fabrice BESNARD • 130

Hello Fabrice, Glad to hear that, much more is on the way! Canceling and re-starting workflows will do nothing but move your work back to the end of the queue. I would leave queued jobs alone unless you no longer want to do that analysis anymore (permanently delete if really do not need). Worflows queue jobs sooner, so are a good strategy no matter the tool - all these do is add multiple tool jobs to your history at once. From there, it is just the same as if you started the jobs individually (but somehow at the same time!). It is important to note here that your jobs are distributed "per user". If you have a bunch of workflows running in different histories, then these will be competing for the job slot when it is your turn to have a job run. The job queued earliest will run. When the server is very busy, be sure to queue up what is most important first. Thanks for our great development team and active development community. They keep it all together! Take care! Jen Galaxy team -- Jennifer Hillman-Jackson http://galaxyproject.org

ADD REPLY • link written 5.2 years ago by Jennifer Hillman Jackson ♦ 25k

Similar posts • Search »