Question: Workflow import error - from release 16.07, to release 17.01
1
gravatar for matt.chambers42
18 months ago by
matt.chambers4250 wrote:

I'm trying to import workflows from a 16.07 instance to a 17.01 one. The tools are slightly different, so I'm getting a lovely Internal Server Error which tells me nothing. So I go to the log and find more detail that is almost, but not entirely, completely unhelpful:

129.114.104.5 - - [24/May/2017:12:13:47 -0500] "POST /workflow/import_workflow HTTP/1.0" 500 - "http://129.114.18.61/workflow/import_workflow" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
Error - <type 'exceptions.KeyError'="">: 'input_opts'
URL: http://129.114.18.61/workflow/import_workflow
File '/opt/galaxy/galaxy-app/lib/galaxy/web/framework/middleware/error.py', line 154 in __call__
  app_iter = self.application(environ, sr_checker)
File '/opt/galaxy/galaxy-app/.venv/local/lib/python2.7/site-packages/paste/recursive.py', line 85 in __call__
  return self.application(environ, start_response)
File '/opt/galaxy/galaxy-app/.venv/local/lib/python2.7/site-packages/paste/httpexceptions.py', line 640 in __call__
  return self.application(environ, start_response)
File '/opt/galaxy/galaxy-app/lib/galaxy/web/framework/base.py', line 134 in __call__
  return self.handle_request( environ, start_response )
File '/opt/galaxy/galaxy-app/lib/galaxy/web/framework/base.py', line 193 in handle_request
  body = method( trans, **kwargs )
File '/opt/galaxy/galaxy-app/lib/galaxy/webapps/galaxy/controllers/workflow.py', line 956 in import_workflow
  workflow, missing_tool_tups = self._workflow_from_dict( trans, data, source=src, add_to_menu=add_to_menu )
File '/opt/galaxy/galaxy-app/lib/galaxy/web/base/controller.py', line 1180 in _workflow_from_dict
  exact_tools=exact_tools,
File '/opt/galaxy/galaxy-app/lib/galaxy/managers/workflows.py', line 206 in build_workflow_from_dict
  exact_tools=exact_tools,
File '/opt/galaxy/galaxy-app/lib/galaxy/managers/workflows.py', line 298 in _workflow_from_dict
  module, step = self.__track_module_from_dict( trans, steps, steps_by_external_id, step_dict, exact_tools=exact_tools )
File '/opt/galaxy/galaxy-app/lib/galaxy/managers/workflows.py', line 850 in __track_module_from_dict
  module, step = self.__module_from_dict( trans, step_dict, exact_tools=exact_tools )
File '/opt/galaxy/galaxy-app/lib/galaxy/managers/workflows.py', line 897 in __module_from_dict
  module.save_to_step( step )
File '/opt/galaxy/galaxy-app/lib/galaxy/workflow/modules.py', line 880 in save_to_step
  step.tool_inputs = self.tool.params_to_strings( self.state.inputs, self.trans.app )
File '/opt/galaxy/galaxy-app/lib/galaxy/tools/__init__.py', line 1378 in params_to_strings
  return params_to_strings( self.inputs, params, app )
File '/opt/galaxy/galaxy-app/lib/galaxy/tools/parameters/__init__.py', line 149 in params_to_strings
  value = params[ key ].value_to_basic( value, app )
File '/opt/galaxy/galaxy-app/lib/galaxy/tools/parameters/grouping.py', line 91 in value_to_basic
  rval_dict[ input.name ] = input.value_to_basic( d[input.name], app )
KeyError: 'input_opts'


CGI Variables
-------------
  CONTENT_LENGTH: '531'
  CONTENT_TYPE: 'multipart/form-data; boundary=----WebKitFormBoundary3FMOXG9DsshH6EHO'
  HTTP_ACCEPT: 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'
  HTTP_ACCEPT_ENCODING: 'gzip, deflate'
  HTTP_ACCEPT_LANGUAGE: 'en-US,en;q=0.8,zh-CN;q=0.6,zh;q=0.4,so;q=0.2,hu;q=0.2'
  HTTP_CACHE_CONTROL: 'max-age=0'
  HTTP_CONNECTION: 'close'
  HTTP_COOKIE: 'galaxysession=c6ca0ddb55be603a14cbe07075a87ab2ebb9de99b46f305bd035b4cf143e2201a77f6cdf42ac961a'
  HTTP_HOST: '129.114.18.61'
  HTTP_ORIGIN: 'http://129.114.18.61'
  HTTP_REFERER: 'http://129.114.18.61/workflow/import_workflow'
  HTTP_UPGRADE_INSECURE_REQUESTS: '1'
  HTTP_USER_AGENT: 'Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'
  PATH_INFO: '/workflow/import_workflow'
  REMOTE_ADDR: '129.114.104.5'
  REQUEST_METHOD: 'POST'
  SERVER_NAME: '127.0.0.1'
  SERVER_PORT: '8080'
  SERVER_PROTOCOL: 'HTTP/1.0'


WSGI Variables
--------------
  application: <paste.recursive.RecursiveMiddleware object="" at="" 0x7f757f01bf90="">
  controller_action_key: u'web.workflow.import_workflow'
  is_api_request: False
  paste.cookies: (<SimpleCookie: galaxysession="c6ca0ddb55be603a14cbe07075a87ab2ebb9de99b46f305bd035b4cf143e2201a77f6cdf42ac961a">, 'galaxysession=c6ca0ddb55be603a14cbe07075a87ab2ebb9de99b46f305bd035b4cf143e2201a77f6cdf42ac961a')
  paste.expected_exceptions: [<class 'paste.httpexceptions.HTTPException'="">]
  paste.httpexceptions: <paste.httpexceptions.HTTPExceptionHandler object="" at="" 0x7f757f01bf10="">
  paste.httpserver.proxy.host: 'dummy'
  paste.httpserver.proxy.scheme: 'http'
  paste.httpserver.thread_pool: <paste.httpserver.ThreadPool object="" at="" 0x7f757fb1d210="">
  paste.recursive.forward: <paste.recursive.Forwarder from=""/>
  paste.recursive.include: <paste.recursive.Includer from=""/>
  paste.recursive.include_app_iter: <paste.recursive.IncluderAppIter from=""/>
  paste.recursive.script_name: ''
  paste.throw_errors: True
  request_id: '590675c640a411e78eacfa163e8ef98e'
  webob._body_file: (<_io.BufferedReader>, <socket._fileobject object="" at="" 0x7f758c077250="" length="531">)
  webob._parsed_post_vars: (MultiDict([('url', u'http://js-156-29.jetstream-cloud.org/u/chambm/w/human-sequence-database-search-variant-confirmation-jetstream-imported-from-uploaded-file/json'), ('file_data', u''), ('import_button', u'Import')]), <_io.BytesIO object at 0x7f7580124a10>)
  webob._parsed_query_vars: (GET([]), '')
  webob.is_body_seekable: True
  wsgi process: 'Multithreaded'
------------------------------------------------------------

Now, if I could see the argument values in the traceback instead of just the argument names, it should give me a strong idea about which part of the workflow is causing the problem. Apparently the inspect or cgitb modules in python will accomplish this, but I have no idea where to insert it so that it will take effect for the whole of Galaxy.

workflow import 17.01 galaxy 16.07 • 502 views
ADD COMMENTlink modified 18 months ago by Jennifer Hillman Jackson25k • written 18 months ago by matt.chambers4250

After a few false starts, I found that editing some code in lib/galaxy/web/framework/middleware/error.py achieved my desired effect. It is quite verbose because the workflow dictionary is huge, but it actually allowed me to pinpoint the problematic tool: Query to Tabular.

Here is the original error, cgitb style:

ADD REPLYlink written 18 months ago by matt.chambers4250

Will the workflow load if the latest version of the tool is installed first? https://toolshed.g2.bx.psu.edu/view/jjohnson/query_tabular/49ca06d66762

You may be doing that already, but just checking.

There have been some changes in the way dependencies are handled during tool installs in Galaxy version 17.01 and further improved in version 17.05 (the most current release). Upgrading the Galaxy instance would also be a good idea and might be enough to handle this case (load the workflow successfully, then edit it to resolve tool mismatches between the two Galaxy servers), but that would need to be tested.

ADD REPLYlink written 18 months ago by Jennifer Hillman Jackson25k

I posted as an answer intentionally. :) My main concern was with being able to more quickly clean information about exceptions from the Galaxy log (now and in the future). Although I should have posted the code change I made!

To fix this problem, I tried to manually update the JSON syntax of the old workflow (which used Query Tabular 0.1.3 while new instance has 3.0.0). I made a new workflow with 3.0.0 and copied parameters manually. Then I downloaded the new workflows' JSON and copied the tool_state part over to the old workflow. No dice, still got the input_opts KeyError when importing. So I tried importing the new workflow and I got the same error, i.e. I couldn't import the workflow I had just exported a minute earlier. So this has to be a Galaxy bug. I can't try this on main because it doesn't have Query Tabular installed. I'm running a Jetstream instance and 17.05 isn't an option yet. This is on 17.01.

ADD REPLYlink written 18 months ago by matt.chambers4250

Np :) I wanted to make sure we got this settled out as a usage versus code problem (that needs to be fixed) before this dropped off as answered.

I'll recruit some developer help for more troubleshooting. Thanks for the follow up! Jen

ADD REPLYlink written 18 months ago by Jennifer Hillman Jackson25k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 170 users visited in the last hour