Question: Bowtie2 error on locally installed Galaxy : unable to finish job
0
gravatar for chaoyuan
21 months ago by
chaoyuan80
chaoyuan80 wrote:

Hi, I've installed a local Galaxy. I used data_manager_fetch_genome_all_fasta to get a galGal4 genome. I then used data_manager_bowtie2_index_builder to build bowtie2 index. 

In my test run, I first passed my data through FastQC and then Fastq_groomer. Then I tried bowtie2. I can see the reference genomes and options without a problem. After I hit run, it seems to go through the bowtie2 process alright as I can see it start and finish in the process monitor, but it can not finish and gave me this error: 

Traceback (most recent call last):
  File "/home/joe/galaxy-dev/lib/galaxy/jobs/runners/local.py", line 128, in queue_job
    job_wrapper.finish( stdout, stderr, exit_code )
  File "/home/joe/galaxy-dev/lib/galaxy/jobs/__init__.py", line 1211, in finish
    dataset.datatype.set_meta( dataset, overwrite=False )
  File "/home/joe/galaxy-dev/lib/galaxy/datatypes/binary.py", line 339, in set_meta
    exit_code = subprocess.call( args=command, stderr=open( stderr_name, 'wb' ) )
  File "/usr/lib/python2.7/subprocess.py", line 522, in call
    return Popen(*popenargs, **kwargs).wait()
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

I tried the suggestion in http://stackoverflow.com/questions/18962785/oserror-errno-2-no-such-file-or-directory-while-using-python-subprocess-in-dj  and add shell=True keyword argument to line 339 in binary.py,  and then I had a different problem 

Traceback (most recent call last):
  File "/home/joe/galaxy-dev/lib/galaxy/jobs/runners/local.py", line 128, in queue_job
    job_wrapper.finish( stdout, stderr, exit_code )
  File "/home/joe/galaxy-dev/lib/galaxy/jobs/__init__.py", line 1211, in finish
    dataset.datatype.set_meta( dataset, overwrite=False )
  File "/home/joe/galaxy-dev/lib/galaxy/datatypes/binary.py", line 359, in set_meta
    raise Exception( "Error Setting BAM Metadata: %s" % stderr )
Exception: Error Setting BAM Metadata: index: 1: index: samtools: not found

It appears that the env.sh file was not executed. So I manually add the path to the samtools binary to the system PATH variable and make sure that it can be found on terminal. However, this problem persists. 

After several trials, it became clear to me that the bowtie2 process itself actually also calls samtools. It's after the bowtie2 process is finished and when the result is piped to samtools that the error occurs. While the bowtie2 process itself can locate where samtools is, a python program doesn't seem to be able to. I then download and build Samtools-1.2 from source. Now the Bowtie2 wrapper can find samtools, but I ended up with a standard error showing Samtools command usage, which would indicate that the command sent out by the wrapper is wrong? 

The Job command loos like this : 

bowtie2 -p ${GALAXY_SLOTS:-4} -x /home/joe/galaxy-dev/tool-data/galGal4/bowtie2_index/galGal4/galGal4 -U "/home/joe/galaxy-dev/database/files/000/dataset_5.dat" --very-fast-local | samtools view -Su - | samtools sort -o - - > /home/joe/galaxy-dev/database/files/000/dataset_22.dat

The first samtools command seems to have an extra '-', and the second samtools command seems to have extra '--'. Am I right about this? Where can I fix this ?

ADD COMMENTlink modified 4 months ago by Jennifer Hillman Jackson23k • written 21 months ago by chaoyuan80

Is Samtools installed? This looks like a dependency issue.

ADD REPLYlink written 21 months ago by Jennifer Hillman Jackson23k

Yes. It's installed along with the Bowtie2. (Dependency automatically handled). When Bowtie2 is running, I can see a Samtools running as well. If I uninstalled the Samtools installed by Galaxy, I can't really start the process at all. With the automatically installed Samtools, the error shows up apparently after the bowtie2 aligning process is finished. According to the Job Command-line, the bowtie2 result is piped to samtools. I believe this is where problem occurs. Please see my updated post. 

ADD REPLYlink written 21 months ago by chaoyuan80
2
gravatar for Daniel Blankenberg
21 months ago by
Daniel Blankenberg ♦♦ 1.7k
United States
Daniel Blankenberg ♦♦ 1.7k wrote:

The Cause of this error is due to the set_metadata process not having access to samtools when trying to index the BAM dataset's file. You should be able to setup a `default` samtools version and have that sourced through Galaxy's dependency resolution systems.

 

Alternatively, you can add `samtools` to the path for the environment+user for your job runner. 

ADD COMMENTlink written 21 months ago by Daniel Blankenberg ♦♦ 1.7k
1
gravatar for chaoyuan
21 months ago by
chaoyuan80
chaoyuan80 wrote:

It turns out that path to samtools is not added to system path variable, which I found odd because there was an env.sh recording the path to samtools. All the error I posted above ( including those following editing binary.py ) all stemmed from this very fact. Adding the path to samtools to system path variable solves this. There is no need to edit binary.py or bowtie2_wrapper.xml. 

ADD COMMENTlink modified 21 months ago • written 21 months ago by chaoyuan80
0
gravatar for vlad.seitan
15 months ago by
vlad.seitan0 wrote:

chaoyuan, do you mind saying how you got over this problem exactly? I am having the same issue and don't seem to be able to fix it. I have added the samtools bin folder (installed along with bowtie2 through the tool shed) to the system $PATH but still no luck. I am still getting an error message almost identical to yours.

The Galaxy framework encountered the following error while attempting to run the tool: Traceback (most recent call last): File "/Users/vladseitan/galaxy/lib/galaxy/jobs/runners/local.py", line 128, in queue_job job_wrapper.finish( stdout, stderr, exit_code ) File "/Users/vladseitan/galaxy/lib/galaxy/jobs/__init__.py", line 1257, in finish dataset.datatype.set_meta( dataset, overwrite=False ) File "/Users/vladseitan/galaxy/lib/galaxy/datatypes/binary.py", line 339, in set_meta exit_code = subprocess.call( args=command, stderr=open( stderr_name, 'wb' ) ) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 522, in call return Popen(popenargs, *kwargs).wait() File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 710, in __init__ errread, errwrite) File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 1335, in _execute_child raise child_exception OSError: [Errno 2] No such file or directory Tool execution generated the following error message: Unable to finish job

ADD COMMENTlink written 15 months ago by vlad.seitan0

I think yours is crashing on samtools sort, which since samtools 1.2 or 1.3 requires the samtools sort -o flag. Could you try using an older version of samtools (0.1.19)? If this solves your issue please report it here... The galaxy code should then be updated by using pysam.sort and pysam.index (https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/datatypes/binary.py#326 / https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/datatypes/binary.py#365)

ADD REPLYlink modified 15 months ago • written 15 months ago by y.hoogstrate450
0
gravatar for giovanninicola.bubici
4 months ago by
Italy
giovanninicola.bubici0 wrote:

Hello! I'm trying using a local instance of Galaxy and found the same problem with bowtie2. Can you explain in detail how can I add the path of samtools?

ADD COMMENTlink written 4 months ago by giovanninicola.bubici0

Install Samtools from the Tool Shed if you have not already (many Galaxy processes require this).

How to add the path is included in this reply: https://biostar.usegalaxy.org/p/20176/#20179

ADD REPLYlink written 4 months ago by Jennifer Hillman Jackson23k
0
gravatar for giovanninicola.bubici
4 months ago by
Italy
giovanninicola.bubici0 wrote:

Thank you! Samtools was already installed through Tool Shed, but Bowtie2 did not work. I solved the problem by installing Samtools to a local directory in the virtual machine.

ADD COMMENTlink written 4 months ago by giovanninicola.bubici0

Great! Glad you solved the problem.

ADD REPLYlink written 4 months ago by Jennifer Hillman Jackson23k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 120 users visited in the last hour