Latest
Open
RNA-Seq
ChIP-Seq
SNP
Assembly
Forum
Planet
All »
Home
Log In
Question: Adding custom tools: resolving (external ) XML entities
1
gravatar for plindenbaum
4.5 years ago by
plindenbaum • 90
France
plindenbaum • 90 wrote:

Writing a XML description for some custom tools, I wanted to share things using XML entities declared in an external file.

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE tool  [
<!ENTITY % tool_entities SYSTEM "tool.ent">
%tool_entities;
]>
<tool (...)>
  <command>&java_exe; &jvm_options; -cp /lib/my.jar my.Tool (..)</command>

 

The file tool.ent:

<!ENTITY java_exe     "/usr/bin/java">
<!ENTITY jvm_options    " -Xmx500m ">

 

xmllint resolve/replace my entities in the XML file without any problem:

$ xmllint --noent tools/mytools/tool.xml

but the expat engine in galaxy doesn't seem to resolve it:

 

ExpatError: undefined entity &java_exe;: line 8, column 30

URL: http://172.18.254.211:8080/galaxy/admin/reload_tool
File '/commun/data/users/galaxy-dev/galaxy-dist/eggs/WebError-0.8a-py2.6.egg/weberror/evalexception/middleware.py', line 364 in respond
  app_iter = self.application(environ, detect_start_response)
File '/commun/data/users/galaxy-dev/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/recursive.py', line 84 in __call__
  return self.application(environ, start_response)
File '/commun/data/users/galaxy-dev/galaxy-dist/eggs/Paste-1.7.5.1-py2.6.egg/paste/httpexceptions.py', line 633 in __call__
  return self.application(environ, start_response)
File '/commun/data/users/galaxy-dev/galaxy-dist/lib/galaxy/web/framework/base.py', line 125 in __call__
  return self.handle_request( environ, start_response )
File '/commun/data/users/galaxy-dev/galaxy-dist/lib/galaxy/web/framework/base.py', line 182 in handle_request
  body = method( trans, **kwargs )
File '/commun/data/users/galaxy-dev/galaxy-dist/lib/galaxy/web/framework/__init__.py', line 228 in decorator
  return func( self, trans, *args, **kwargs )
File '/commun/data/users/galaxy-dev/galaxy-dist/lib/galaxy/web/base/controllers/admin.py', line 65 in reload_tool
  message, status = toolbox.reload_tool_by_id( tool_id )
File '/commun/data/users/galaxy-dev/galaxy-dist/lib/galaxy/tools/__init__.py', line 605 in reload_tool_by_id
  new_tool = self.load_tool( old_tool.config_file )
File '/commun/data/users/galaxy-dev/galaxy-dist/lib/galaxy/tools/__init__.py', line 581 in load_tool
  tree = self._load_and_preprocess_tool_xml( config_file )
File '/commun/data/users/galaxy-dev/galaxy-dist/lib/galaxy/tools/__init__.py', line 767 in _load_and_preprocess_tool_xml
  tree = parse_xml(config_file)
File '/commun/data/users/galaxy-dev/galaxy-dist/lib/galaxy/util/__init__.py', line 143 in parse_xml
  tree = ElementTree.parse(fname)
File '/commun/data/users/galaxy-dev/galaxy-dist/eggs/elementtree-1.2.6_20050316-py2.6.egg/elementtree/ElementTree.py', line 859 in parse
  tree.parse(source, parser)
File '/commun/data/users/galaxy-dev/galaxy-dist/eggs/elementtree-1.2.6_20050316-py2.6.egg/elementtree/ElementTree.py', line 583 in parse
  parser.feed(data)
File '/commun/data/users/galaxy-dev/galaxy-dist/eggs/elementtree-1.2.6_20050316-py2.6.egg/elementtree/ElementTree.py', line 1242 in feed
  self._parser.Parse(data, 0)
File '/commun/data/users/galaxy-dev/galaxy-dist/eggs/elementtree-1.2.6_20050316-py2.6.egg/elementtree/ElementTree.py', line 1198 in _default
  self._parser.ErrorColumnNumber)
ExpatError: undefined entity &java_exe;: line 8, column 30

 

can I tell galaxy to resolve XML entities ?

 

P.

UPDATE: I tested various way of settings macros (using <tokens/>, <xml/>, <macro><action></action></macro> ... ) . For example:

<macros>
  <macro name="java_exe">/usr/bin/java</macro>
  <macro name="jvm_options"> -Xmx500m </macro>
</macros>

 

and

(...)

 <macros>
    <import>macros.xml</import>
  </macros>
  <command> <expand macro="java_exe"/> <expand macro="jvm_options"/> (...)

(...)

 

but the macros in <command> don't seem to be resolved (I suspect they're replaced with empty string)

Edit2: here is my files so far. In <command>, I put the content of  @JAVA_EXE@ in a file, the ouput is "@JAVA_EXE@" (no substitution)

 

 

 

 

 
tool dtd xml custom • 2.6k views
ADD COMMENT • link •
modified 4.5 years ago • written 4.5 years ago by plindenbaum • 90

Please note that setting -Xmx and co is not needed to be set inside the tool. Are better way would be to set it via the job_conf.xml file. Here, an administrator can specify individual settings for every tool and adjust it to the cluster settings.

ADD REPLY • link written 4.5 years ago by Bjoern Gruening ♦ 5.1k

Likewise the location of the Java binary might be better done via an environment variable like PATH or something specific via a dependency, rather than hard coding like this?

ADD REPLY • link written 4.5 years ago by Peter Cock • 1.4k

Yes! Imho you can omit the path and call java directly. Galaxy will lookup in $PATH and will find java.

ADD REPLY • link written 4.5 years ago by Bjoern Gruening ♦ 5.1k

What is in the Galaxy logging when it calls the tool? That would show you the actual command line string built from the <command> tag and the user's parameter selection.
 

ADD REPLY • link written 4.5 years ago by Peter Cock • 1.4k

Cross posted to galaxy-dev http://lists.bx.psu.edu/pipermail/galaxy-dev/2014-May/019414.html / http://dev.list.galaxyproject.org/replacing-XML-fragments-in-mycustomtool-xml-tc4664538.html

ADD REPLY • link written 4.5 years ago by Peter Cock • 1.4k

Did this work once you updated Galaxy itself (as per the galaxy-dev discussion)?

ADD REPLY • link written 4.5 years ago by Peter Cock • 1.4k

sorry for the late response Peter, We are reorganizing our servers and the new server for galaxy with the latest version of galaxy isn't ready. I'll validate your answer anyway.

ADD REPLY • link written 4.4 years ago by plindenbaum • 90
3
gravatar for Peter Cock
4.5 years ago by
Peter Cock • 1.4k
European Union
Peter Cock • 1.4k wrote:

I presume resolving external entities would be frowned on from a security point of view.

You can probably achieve a similar result using Galaxy's XML macros - there is a fairly complex set in use here https://github.com/peterjc/galaxy_blast/tree/master/tools/ncbi_blast_plus putting reused XML (and Cheetah and RST) snippets into the ncbi_macros.xml file.

See https://wiki.galaxyproject.org/Admin/Tools/ToolConfigSyntax#Reusing_Repeated_Configuration_Elements

More details, in the tool wrapper you must explicitly import the macro file, for example ncbi_blastn_wrapper.xml has this snippet of XML:

    <macros>
        <token name="@BINARY@">blastn</token>
        <import>ncbi_macros.xml</import>
    </macros>

Later in the tool XML it references some of the defined XML macros, e.g. <expand macro="stdio" /> which will trigger the insertion of this snippet from ncbi_macros.xml:

        <stdio>
            <!-- Anything other than zero is an error -->
            <exit_code range="1:" />
            <exit_code range=":-1" />
            <!-- In case the return code has not been set properly check stderr too -->
            <regex match="Error:" />
            <regex match="Exception:" />
        </stdio>

That XML macro was defined in ncbi_macros.xml as follows:

    <xml name="stdio">
        <stdio>
            <!-- Anything other than zero is an error -->
            <exit_code range="1:" />
            <exit_code range=":-1" />
            <!-- In case the return code has not been set propery check stderr too -->
            <regex match="Error:" />
            <regex match="Exception:" />
        </stdio>
    </xml>

Note there are other non-XML macros in the same file, called tokens, for use within the RST help section or Cheetah command template (within the <command> tag which seems to be your specific interest). For example:

<command>blastn ... @THREADS@ ... </command>

Here @THREADS@ gets replaced with:

-num_threads "\${GALAXY_SLOTS:-8}"

This was defined in ncbi_macros.xml as:

<token name="@THREADS@">-num_threads "\${GALAXY_SLOTS:-8}"</token>
ADD COMMENT • link modified 4.5 years ago • written 4.5 years ago by Peter Cock • 1.4k

thank you. It doesn't work on my side. If I only define a chunk of text in <macros> should i define it as

<macros><token name="java_exe">/usr/bin/java</token></macros>

<macros><token name="@java_exe@">/usr/bin/java</token></macros>

<macros><macro name="java_exe">/usr/bin/java</macro></macros>

(...)

 

then, how should I replace it later in the <command> tag ?

<command> <macro name="java_exe"/> args </command>

<command> @java_exe@ args </command>

(...)

Thanks

P.

 

 

ADD REPLY • link modified 4.5 years ago • written 4.5 years ago by plindenbaum • 90

Updated again, use @java_exe@ both in the definition and the point of insertion.

ADD REPLY • link modified 4.5 years ago • written 4.5 years ago by Peter Cock • 1.4k
Please log in to add an answer.

Similar posts • Search »

Content
Help
Access
Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 169 users visited in the last hour