Question: REST API - can't find workflow step_id in a workflow invocation
1
gravatar for sfischer
2.8 years ago by
sfischer50
United States
sfischer50 wrote:

Hi,

i am using the REST API for workflows, as documented here:

https://galaxy.readthedocs.org/en/master/lib/galaxy.webapps.galaxy.api.html#module-galaxy.webapps.galaxy.api.workflows

I need to:

  1. GET a workflow invocation
  2. scrape its parameter values
  3. fire off a new run of that workflow based on those parameter values.  (Please don't ask why....)
  4. (I don't know in advance which workflow or invocation to use, so i can't hard-code anything)

This entails matching the steps in the invocation with the steps in the workflow, which is what I am having trouble with.

In the JSON returned by the api/workflow/{workflow_id} endpoint, i get a bunch of steps, each with what looks like a "step index", such as 1909417:

"steps": {
 "1909417": {
    "tool_id": "Cut1",
    "tool_version": "1.0.2",
    "id": ​1909417,
    "input_steps": { },
    "tool_inputs":  {
        "columnList": "{\"__class__\": \"RuntimeValue\"}",
        "input": "null",
        "delimiter": "\"T\""
    },
    "type": "tool",
    "annotation": null

},
...

The JSON to run a workflow looks like this: 

{
  "workflow_id" :"ba06423ff18d37cc",
  "history" : "groovy",
  "parameters" : {"1909417" : {"columnList" : "C1,C2"}},
  "ds_map":{"1909417" : {"src" : "hda", "id": "bbd44e69cb8906b587c210f637326cca"}}
}

The parameters and ds_map are keyed with the step index from the workflow:

My problem is that when I look at a workflow invocation, or its steps in detail, or its jobs in detail, i never see that step index.  

This is what the JSON for an invocation looks like (for the step above):

"steps": [
    {
        "workflow_step_uuid": "f2498dae-6e41-4dba-b1c2-74e984661980",
        "update_time": "2015-11-12T15:07:08.602957",
        "job_id": "bbd44e69cb8906b57a8032d7e2c63780",
        "state": "ok",
        "workflow_step_label": null,
        "order_index": ​0,
        "action": null,
        "model_class": "WorkflowInvocationStep",
        "workflow_step_id": "925f0024d97df8bb",
        "id": "7965aefdfdc0734d"
    }

]

And here is the associated job:

{
    "tool_id": "Cut1",
    "update_time": "2015-11-12T15:08:28.150810",
    "inputs": 
{
    "input": 
    {
        "src": "hda",
        "id": "bbd44e69cb8906b587c210f637326cca",
        "uuid": "acf3626f-c89d-4c55-918c-b1a2c746ef54"
    }
},
"outputs": 
{
    "out_file1": 
    {
        "src": "hda",
        "id": "bbd44e69cb8906b5f9839f2acf8a48d6",
        "uuid": "464c5960-3ec9-44db-9141-85f22c9b9f35"
    }
},
"exit_code": ​0,
"state": "ok",
"create_time": "2015-11-12T15:07:07.576559",
"params": 
    {
        "__workflow_invocation_uuid__": "\"09b2c6e0894f11e5a641005056a52a46\"",
        "delimiter": "\"T\"",
        "dbkey": "\"?\"",
        "columnList": "\"c1,c2\"",
        "chromInfo": "\"/galaxy-repl/localdata/chrom/?.len\""
    },
    "model_class": "Job",
    "id": "bbd44e69cb8906b57a8032d7e2c63780"

MY QUESTION IS:  how can i correlate a step in a workflow invocation with a step in its workflow?

Thanks,

Steve

api • 969 views
ADD COMMENTlink modified 2.8 years ago • written 2.8 years ago by sfischer50
0
gravatar for Dannon Baker
2.8 years ago by
Dannon Baker3.7k
United States
Dannon Baker3.7k wrote:

So the problem here is that the workflow endpoint does not encode step
ids, while the invocation one does.  I'll look into the right way to
resolve this going forward (probably encode at both endpoints), though
we need to do it in a backwards compatible way.

Anyway, in short, if you want to do this *right now*, the easiest
thing to do would be to use the encoding/decoding routines from Galaxy
to manually decode the workflowinvocation step ids to match back up to
workflow steps.

ADD COMMENTlink written 2.8 years ago by Dannon Baker3.7k
0
gravatar for sfischer
2.8 years ago by
sfischer50
United States
sfischer50 wrote:

Dannon,

Thanks very much. I had a guess that was it, but couldn't find it in the code base.

I do think it would be good, as you suggest, for galaxy to encode this value in the workflow info returned by the API call, to make it consistent with the invocation.

In the meantime, I can try the workaround you suggest.

Can you point me to the method that does the encoding?   Which file is it in, and which method is it?

Clarification:  I am using the REST API remotely, so my program does not have access to the galaxy codebase.  Instead, to do the recommended workaround, I'll need to duplicate the encoding algorithm in my program.   To do that, I need to see exactly how galaxy is doing its encoding.

Thanks very much,

Steve

 

 

ADD COMMENTlink modified 2.8 years ago • written 2.8 years ago by sfischer50

Yep, so what you'll want to do is use the SecurityHelper from galaxy.web.security (lib/galaxy/web/security/__init__.py)

In [1]: from galaxy.web.security import SecurityHelper

In [2]: sh = SecurityHelper(id_secret="USING THE DEFAULT IS NOT SECURE!")

In [3]: sh.decode_id("6fc9fbb81c497f69")

Out[3]: 82


(this, of course, only works if you know the id_secret for the server in question -- just saw your followup direct email)

ADD REPLYlink modified 2.8 years ago • written 2.8 years ago by Dannon Baker3.7k

Dannon,

Thanks.

It looks like this workaround isn't going to work for me:

1) i am operating remotely, so cannot import that code. (i am running my code, via REST, against usegalaxy.org)

2) i tried this in perl, locally:

% perl -e 'use Crypt::Blowfish; my $c = new Crypt::Blowfish "USING THE DEFAULT IS NOT SECURE!"; $e = $c->encrypt("!2091734"); print unpack("H16", $e), "\n";'

55fdda0bb0ad06b1

that value is not the right answer.

maybe the problem is that usegalaxy.org is not using the above value for its secret? it seems like without the id_secret, I can't do the workaround, is that right?

i sure hope you guys can fix this bug soon :)

Steve

ADD REPLYlink modified 2.8 years ago • written 2.8 years ago by sfischer50

Yep, I assumed you were talking about a server you administered.  The code was just an example -- usegalaxy.org definitely does not use the default secret_id and nobody else should either :)

I'm trying to think of another workaround for you, since even if I pushed a changeset flipping the encoding today it would be some time before we'd get it into the stable branch and update usegalaxy.org.

ADD REPLYlink written 2.8 years ago by Dannon Baker3.7k

Hi Dannon,

I have been prototyping on usegalaxy.org.   You're right we will soon have our own instance running.

While we wait for this to be fixed in the stable branch, can you provide a patch we can apply to our (upcoming) instance?  Just let me know what file to edit, and what the change would be.  I imagine it might be a single line of code.

Thanks,

Steve

ADD REPLYlink written 2.8 years ago by sfischer50

Yep, will get you a patch shortly.

ADD REPLYlink written 2.8 years ago by Dannon Baker3.7k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 75 users visited in the last hour