Question: Batch workflow submission thru UI is submitting non stop
2
gravatar for philipmabon
4.6 years ago by
philipmabon20
Canada
philipmabon20 wrote:

Hey,

 

I have a workflow that takes in two input datasets (forward and reverse fastq) and does some quality control checks using Fastqc Summary.

When I use my custom workflow API scripts on 200 strains, it works correctly, however, if I attempt the same number using the web interface is going into an infinite submitting.

I inspected the POST request in Chrome and everything looks correct.  It's sending the correct number of datasets, no duplicates. Also seems there is no specific number of datasets that cause the infinite submitting.

 

Has anyone else ran into this issue?

 

 

ui workflow galaxy dev • 985 views
ADD COMMENTlink modified 4.6 years ago by Martin Čech ♦♦ 4.9k • written 4.6 years ago by philipmabon20
2
gravatar for Dannon Baker
4.6 years ago by
Dannon Baker3.7k
United States
Dannon Baker3.7k wrote:

This was sorted out via IRC, but as a reference for anyone else looking -- the workflow/run post was taking too long to return and nginx was resubmitting to another worker.  The simplest immediate solution when running with a proxy + multiple workers is to make sure the request timeout is set long enough.

ADD COMMENTlink written 4.6 years ago by Dannon Baker3.7k
1

One other tip, in nginx the `proxy_next_upstream` directive can be used to prevent nginx from sending requests that timeout to additional upstreams. The downside is that it breaks "failover" functionality, so you need to take extra care to make sure your servers are functioning.

ADD REPLYlink written 4.6 years ago by Nate Coraor3.2k
1

Since it is hard to find a request timeout that can cover all cases I have modified our nginx config to be as follows:

location /workflow/run {

    proxy_pass http://galaxy_webapp/workflow/run;

 

    proxy_read_timeout 900;

    # disable next_upstream on timeout

    proxy_next_upstream error;

    proxy_set_header X-Forwarded-Host $host;

    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

 

    # custom 504 explaining workflow submission took too long probably ok though

    error_page 504 /504.html

}

 

 

ADD REPLYlink modified 4.6 years ago • written 4.6 years ago by eric.enns10

I don't know the nginx commands - but it would seem best to disable these proxy_next_upstream things for all POSTs to Galaxy instead of just /workflow/run.

ADD REPLYlink written 4.6 years ago by jmchilton1.1k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 180 users visited in the last hour