Gluster/Local storage integration with Galaxy

Question: Gluster/Local storage integration with Galaxy

4.5 years ago by

United States

johnbot1 • 0 wrote:

Hi,

Can someone give me the lowdown on Galaxy's Gluster integration mentioned in the January release update here? I configured a stand alone version of Galaxy on a single R3 Amazon EC2 instance and am looking for methods to upload/browse our directly mounted gluster file system (also in AWS) through the web interface. It would be great for many reasons to be able to tell the GUI to upload from locally mounted files. (speed, overcoming the 2GB upload limit etc.)

Thanks,

John

galaxy local files gluster • 1.1k views

ADD COMMENT • link •

modified 4.5 years ago by Dannon Baker ♦ 3.7k • written 4.5 years ago by johnbot1 • 0

4.5 years ago by

Dannon Baker ♦ 3.7k

United States

Dannon Baker ♦ 3.7k wrote:

While I don't have an answer regarding the Gluster support, the easiest way to add local file paths (for a cloudman instance or a local instance) is to edit universe_wsgi.ini to allow library_path_paste. This allows admin users to specify *any* system path in an upload box. It also provides an option to bypass the file copy and just use the file from an external path (that is, not in galaxy's database/ directory, but available on the filesystem).

ADD COMMENT • link written 4.5 years ago by Dannon Baker ♦ 3.7k

Thanks Dannon that's great information! I enabled that feature, set myself up as admin and tried to upload a file with the admin tools (telling it to only link to the file and not actually copy it) but received an error that the directory/file doesn't exist. I checked that the file existed, opened up permissions on it as much as possible and even moved it to /tmp from our main file system with the same result. To verify, i created a simple text file in the same /tmp directory and was able to add it to the library (linking only) without a problem. The bam file I'm trying to upload is 18GB so perhaps it's failing on that and I need to modify some python parameters etc.?

Date uploaded:Fri May 23 01:24:30 2014 (UTC)

File size:18.0 GB

UUID:34b1a621-2e83-4199-8a78-a001817a2f68

Data type:auto

Build:?

Miscellaneous information:Traceback (most recent call last): File "/home/ubuntu/galaxy-dist/tools/data_source/upload.py", line 390, in __main__() File "/home/ubuntu/galaxy-dist/tools/data_source/upload.py", line 379, in __main__ add_file( dataset, registry, j

Job Standard Error

Traceback (most recent call last):
  File "/home/ubuntu/galaxy-dist/tools/data_source/upload.py", line 390, in 
    __main__()
  File "/home/ubuntu/galaxy-dist/tools/data_source/upload.py", line 379, in __main__
    add_file( dataset, registry, json_file, output_path )
  File "/home/ubuntu/galaxy-dist/tools/data_source/upload.py", line 288, in add_file
    if datatype.dataset_content_needs_grooming( dataset.path ):
  File "/home/ubuntu/galaxy-dist/lib/galaxy/datatypes/binary.py", line 148, in dataset_content_needs_grooming
    version = self._get_samtools_version()
  File "/home/ubuntu/galaxy-dist/lib/galaxy/datatypes/binary.py", line 130, in _get_samtools_version
    output = subprocess.Popen( [ 'samtools' ], stderr=subprocess.PIPE, stdout=subprocess.PIPE ).communicate()[1]
  File "/usr/lib/python2.7/subprocess.py", line 710, in __init__
    errread, errwrite)
  File "/usr/lib/python2.7/subprocess.py", line 1327, in _execute_child
    raise child_exception
OSError: [Errno 2] No such file or directory

error

Database/Build:?

Number of data lines:None

Disk file:/tmp/total.gsnap.bam.sorted.bam

ADD REPLY • link written 4.5 years ago by johnbot1 • 0

Is the cluster currently running more than one node? One thing to keep in mind is that you'll need for these external files to be in a location that's shared via nfs with workers. By default, that's /mnt/galaxy*, and a few other locations (iirc), that you can see in /etc/exports. You can also add new locations by editing that file and executing something like `sudo /etc/init.d/nfs-server restart`

ADD REPLY • link written 4.5 years ago by Dannon Baker ♦ 3.7k

From the error message, it appears that the library upload process doesn't have access to the samtools binary.

ADD REPLY • link written 4.5 years ago by Daniel Blankenberg ♦♦ 1.7k

Hrmm, I just saw the "No such file or directory" bit and assumed it was talking about the input file, but you're right. That shouldn't be happening.

ADD REPLY • link written 4.5 years ago by Dannon Baker ♦ 3.7k

I'll go ahead and look through the permissions on the distribution directory to verify. I guess uploading a simple text file doesn't invoke that same samtools binary as happens when uploading a bam file?

ADD REPLY • link written 4.5 years ago by johnbot1 • 0

I've added execute permissions to ~/galaxy-dist/.hg/store/data/tools/samtools but receive the same errors. Any ideas on what else to look at?

Thanks

John

ADD REPLY • link written 4.5 years ago by johnbot1 • 0

Can you verify that the basic samtools command is available to the galaxy user in a terminal? If not, (which would be a bug), just adding it to the PATH environment variable should work.

ADD REPLY • link written 4.5 years ago by Dannon Baker ♦ 3.7k

Similar posts • Search »