Galaxy tool development - how do I get a working samtools?

Question: Galaxy tool development - how do I get a working samtools?

11 months ago by

crashfrog • 0 wrote:

I'm trying to develop a Galaxy tool that has samtools 1.5 as a dependency, but nothing I declare in the "<requirements>" section succeeds in putting a working samtools on my PATH when the tool is executed by Galaxy.

With

<requirement type="package" version="1.5">samtools</requirement>

I get

samtools: error while loading shared libraries: libbz2.so.1.0: cannot open shared object file: No such file or directory

With

<requirement type="binary" version="1.5">samtools</requirement>

I get

Fatal error: Exit code 127 () ... /shared/000/399/tool_script.sh: line 9: samtools: command not found

Neither python-module or set_environment are appropriate here, and there are no other permitted values for requirement.type .

I know I'm not writing the first tool that has samtools as a dependency. How is this supposed to work?

galaxy samtools • 675 views

ADD COMMENT • link •

modified 11 months ago • written 11 months ago by crashfrog • 0

11 months ago by

y.hoogstrate • 460

Netherlands

y.hoogstrate • 460 wrote:

<requirement type="package" version="1.5">samtools</requirement>

should refer to the (i suppose bioconda) version of samtools 1.5

This is the way this should work I believe.

Can you run: <command>which -a samtools ; ldd `which samtools`</commmand> in your tool and post the output?

ADD COMMENT • link written 11 months ago by y.hoogstrate • 460

11 months ago by

crashfrog • 0

crashfrog • 0 wrote:

CLASSPATH=/nfs/sw/apps/varscan/varscan.2.3.9/VarScan.v2.3.9.jar ;export CLASSPATH;LOADEDMODULES=galaxy-super ;export LOADEDMODULES;PATH=/nfs/sw/lang/java/jre1.8.0_112/bin:/nfs/sw/apps/smalt/bin:/nfs/sw/apps/spades/spades-3.9.0/bin:/nfs/sw/apps/sratoolkit/sratoolkit.2.8.0-centos_linux64/bin:/nfs/sw/apps/ascp:/nfs/sw/apps/bwa/bwa-0.7.15/:/nfs/sw/apps/blast/ncbi-blast-2.6.0+/bin:/nfs/sw/apps/samtools/samtools-1.3.1/bin:/nfs/sw/apps/bowtie/bowtie2-2.2.9/:/nfs/sw/apps/snp/snp-pipeline-0.7.0/bin:/nfs/sw/apps/galaxy-dev/galaxy/.venv/bin:/tmp/404.1.all.q:/usr/local/bin:/bin:/usr/bin ;export PATH;_LMFILES_=/nfs/sw/Modules/galaxy-super ;export _LMFILES_;varscan() { java -jar /nfs/sw/apps/varscan/varscan.2.3.9/VarScan.v2.3.9.jar;};/nfs/sw/apps/galaxy-dev/galaxy/database/dependencies/_conda/envs/mulled-v1-ee953de1640e372d8d33733402c4e38671d17286210a6464d4805ae7ef8e67df/bin/samtools
    linux-vdso.so.1 =>  (0x00007ffdd58ce000)
    libz.so.1 => /nfs/sw/apps/galaxy-dev/galaxy/database/dependencies/_conda/envs/mulled-v1-ee953de1640e372d8d33733402c4e38671d17286210a6464d4805ae7ef8e67df/bin/../lib/libz.so.1 (0x00002b735a38e000)
    libm.so.6 => /lib64/libm.so.6 (0x00002b735a5a4000)
    libbz2.so.1.0 => not found
    liblzma.so.5 => /nfs/sw/apps/galaxy-dev/galaxy/database/dependencies/_conda/envs/mulled-v1-ee953de1640e372d8d33733402c4e38671d17286210a6464d4805ae7ef8e67df/bin/../lib/liblzma.so.5 (0x00002b735a8a6000)
    libcurl.so.4 => /nfs/sw/apps/galaxy-dev/galaxy/database/dependencies/_conda/envs/mulled-v1-ee953de1640e372d8d33733402c4e38671d17286210a6464d4805ae7ef8e67df/bin/../lib/libcurl.so.4 (0x00002b735aacb000)
    libcrypto.so.1.0.0 => /nfs/sw/apps/galaxy-dev/galaxy/database/dependencies/_conda/envs/mulled-v1-ee953de1640e372d8d33733402c4e38671d17286210a6464d4805ae7ef8e67df/bin/../lib/libcrypto.so.1.0.0 (0x00002b735ad3e000)
    libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b735b175000)
    libc.so.6 => /lib64/libc.so.6 (0x00002b735b391000)
    librt.so.1 => /lib64/librt.so.1 (0x00002b735b753000)
    libssh2.so.1 => /nfs/sw/apps/galaxy-dev/galaxy/database/dependencies/_conda/envs/mulled-v1-ee953de1640e372d8d33733402c4e38671d17286210a6464d4805ae7ef8e67df/bin/../lib/./libssh2.so.1 (0x00002b735b95b000)
    libssl.so.1.0.0 => /nfs/sw/apps/galaxy-dev/galaxy/database/dependencies/_conda/envs/mulled-v1-ee953de1640e372d8d33733402c4e38671d17286210a6464d4805ae7ef8e67df/bin/../lib/./libssl.so.1.0.0 (0x00002b735bb99000)
    libgssapi_krb5.so.2 => /nfs/sw/apps/galaxy-dev/galaxy/database/dependencies/_conda/envs/mulled-v1-ee953de1640e372d8d33733402c4e38671d17286210a6464d4805ae7ef8e67df/bin/../lib/./libgssapi_krb5.so.2 (0x00002b735be10000)
    libkrb5.so.3 => /nfs/sw/apps/galaxy-dev/galaxy/database/dependencies/_conda/envs/mulled-v1-ee953de1640e372d8d33733402c4e38671d17286210a6464d4805ae7ef8e67df/bin/../lib/./libkrb5.so.3 (0x00002b735c057000)
    libk5crypto.so.3 => /nfs/sw/apps/galaxy-dev/galaxy/database/dependencies/_conda/envs/mulled-v1-ee953de1640e372d8d33733402c4e38671d17286210a6464d4805ae7ef8e67df/bin/../lib/./libk5crypto.so.3 (0x00002b735c324000)
    libcom_err.so.3 => /nfs/sw/apps/galaxy-dev/galaxy/database/dependencies/_conda/envs/mulled-v1-ee953de1640e372d8d33733402c4e38671d17286210a6464d4805ae7ef8e67df/bin/../lib/./libcom_err.so.3 (0x00002b735c553000)
    libdl.so.2 => /lib64/libdl.so.2 (0x00002b735c756000)
    /lib64/ld-linux-x86-64.so.2 (0x00002b735a16a000)
    libkrb5support.so.0 => /nfs/sw/apps/galaxy-dev/galaxy/database/dependencies/_conda/envs/mulled-v1-ee953de1640e372d8d33733402c4e38671d17286210a6464d4805ae7ef8e67df/bin/../lib/././libkrb5support.so.0 (0x00002b735c95a000)
    libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00002b735cb65000)
    libresolv.so.2 => /lib64/libresolv.so.2 (0x00002b735cd69000)

ADD COMMENT • link written 11 months ago by crashfrog • 0

11 months ago by

crashfrog • 0

crashfrog • 0 wrote:

I know there's an issue in Conda's samtools package being built against dynamic libraries, and that the one in conda-forge is built against statics, but I've already changed galaxy.ini to use the recommended conda channel ordering, so it should be pulling the correct samtools dependencies.

ADD COMMENT • link written 11 months ago by crashfrog • 0

The path of samtools is /nfs/sw/apps/galaxy-dev/galaxy/database/dependencies/_conda/envs/mulled-v1-ee953de1640e372d8d33733402c4e38671d17286210a6464d4805ae7ef8e67df/bin/samtools and some of the libraries are in there as well. I think your galaxy and tools are working fine but the bioconda package is (as you mentioned already) improperly linked.

I think the mulled builds are specific for bioconda, so you are not using samtools from condaforge. You could shutdown galaxy and move the bioconda data, and see if a clean env does the job correctly:

mv /nfs/sw/apps/galaxy-dev/galaxy/database/dependencies/_conda /nfs/sw/apps/galaxy-dev/galaxy/database/dependencies/_conda.bak ; planemo serve ...xml

if this doesn't work you should move the directory back.

ADD REPLY • link written 11 months ago by y.hoogstrate • 460

I'm not using planemo, unfortunately. At this point I've changed the channel resolution order as specified in galaxy.ini (the default order has bioconda ahead of conda-forge) and reset the tool metadata, but it's still pulling mulled builds for samtools and causing the shared library error.

ADD REPLY • link written 11 months ago by crashfrog • 0

Not sure what your exact use-case is, but maybe you could use sambamba as drop-in replacement and revert back to bioconda. Changing the channel order, once it would work, may solve samtools but may break other tools. Since bioconda is supposed to be used as dependency resolver for Galaxy, this would be the handiest workaround I think. You need to pressure the bioconda ppl to solve it ;).

I've tested my instance of bioconda and got this:

conda create -n samtools150 samtools=1.5=0

conda create -n samtools151 samtools=1.5=1

conda create -n samtools152 samtools=1.5=2

source activate samtools150 ; ldd which samtools

libbz2.so.1.0 => /home/youri/anaconda2/envs/samtools150/bin/../lib/libbz2.so.1.0 (0x00007f31ee473000)

source activate samtools151 ; ldd which samtools

libbz2.so.1.0 => /home/youri/anaconda2/envs/samtools151/bin/../lib/libbz2.so.1.0 (0x00007fe51eaf4000)

source activate samtools152 ; ldd which samtools

libbz2.so.1.0 => /home/youri/anaconda2/envs/samtools152/bin/../lib/libbz2.so.1.0 (0x00007fdf796bf000)

Now I am a bit lost figuring out what the actual problem is because the appropriate linked libraries can be obtained via conda (at least on my instance).

ADD REPLY • link modified 11 months ago • written 11 months ago by y.hoogstrate • 460

The thing that doesn't make any sense is that

locate libbz2.so.1.0

returns

/usr/lib64/libbz2.so.1.0.6

so it's clear that libbz2 is present; but samtools can't find it anyway.

ADD REPLY • link written 11 months ago by crashfrog • 0

Please log in to add an answer.

Similar posts • Search »