Batch workflow submission thru UI is submitting non stop

Question: Batch workflow submission thru UI is submitting non stop

4.6 years ago by

Canada

philipmabon • 20 wrote:

Hey,

I have a workflow that takes in two input datasets (forward and reverse fastq) and does some quality control checks using Fastqc Summary.

When I use my custom workflow API scripts on 200 strains, it works correctly, however, if I attempt the same number using the web interface is going into an infinite submitting.

I inspected the POST request in Chrome and everything looks correct. It's sending the correct number of datasets, no duplicates. Also seems there is no specific number of datasets that cause the infinite submitting.

Has anyone else ran into this issue?

ui workflow galaxy dev • 985 views

ADD COMMENT • link •

modified 4.6 years ago by Martin Čech ♦♦ 4.9k • written 4.6 years ago by philipmabon • 20

4.6 years ago by

Dannon Baker ♦ 3.7k

United States

Dannon Baker ♦ 3.7k wrote:

This was sorted out via IRC, but as a reference for anyone else looking -- the workflow/run post was taking too long to return and nginx was resubmitting to another worker. The simplest immediate solution when running with a proxy + multiple workers is to make sure the request timeout is set long enough.

ADD COMMENT • link written 4.6 years ago by Dannon Baker ♦ 3.7k

One other tip, in nginx the `proxy_next_upstream` directive can be used to prevent nginx from sending requests that timeout to additional upstreams. The downside is that it breaks "failover" functionality, so you need to take extra care to make sure your servers are functioning.

ADD REPLY • link written 4.6 years ago by Nate Coraor ♦ 3.2k

Since it is hard to find a request timeout that can cover all cases I have modified our nginx config to be as follows:

location /workflow/run {

proxy_pass http://galaxy_webapp/workflow/run;

proxy_read_timeout 900;

# disable next_upstream on timeout

proxy_next_upstream error;

proxy_set_header X-Forwarded-Host $host;

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

# custom 504 explaining workflow submission took too long probably ok though

error_page 504 /504.html

}

ADD REPLY • link modified 4.6 years ago • written 4.6 years ago by eric.enns • 10

I don't know the nginx commands - but it would seem best to disable these proxy_next_upstream things for all POSTs to Galaxy instead of just /workflow/run.

ADD REPLY • link written 4.6 years ago by jmchilton ♦ 1.1k

Similar posts • Search »