Problems Starting Galaxy Cloudman

Question: Problems Starting Galaxy Cloudman

6.7 years ago by

Greg Edwards • 190 wrote:

Hi, I'm trying to restart my Galaxy Cloudman service, using the same approach that has been successful over the last couple of months .. - launch AMI 861460482541/galaxy-cloudman-2011-03-22 as m1.large - update from Cloudman console - copy in my tools etc etc - restart - away we go, all works However today the update fails, the log says ... /mnt/galaxyTools/galaxy-central/eggs/pysam-0.4.2_kanwei_b10f6e722e9a- py2.6-linux-x86_64-ucs4.egg/pysam/__init__.py:1: RuntimeWarning: __builtin__.file size changed, may indicate binary incompatibility from csamtools import * python path is: /mnt/galaxyTools/galaxy-central/eggs/numpy-1.6.0-py2.6-linux- x86_64-ucs4.egg, /mnt/galaxyTools/galaxy-central/eggs/pysam-0.4.2_kanwei_b10f6e722e9a- py2.6-linux-x86_64-ucs4.egg, /mnt/galaxyTools/galaxy-central/eggs/boto-2.2.2-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/Whoosh-0.3.18-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/pycrypto-2.0.1-py2.6-linux- x86_64-ucs4.egg, /mnt/galaxyTools/galaxy-central/eggs/python_lzo-1.08_2.03_static-py2.6 -linux-x86_64-ucs4.egg, /mnt/galaxyTools/galaxy- central/eggs/bx_python-0.7.1_7b95ff194725-py2.6-linux-x86_64-ucs4.egg, /mnt/galaxyTools/galaxy-central/eggs/amqplib-0.6.1-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/pexpect-2.4-py2.6.egg, /mnt/galaxyTools/galaxy- central/eggs/SQLAlchemy-0.5.6_dev_r6498-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/Babel-0.9.4-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/MarkupSafe-0.12-py2.6-linux- x86_64-ucs4.egg, /mnt/galaxyTools/galaxy-central/eggs/Mako-0.4.1-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/WebHelpers-0.2-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/simplejson-2.1.1-py2.6-linux- x86_64-ucs4.egg, /mnt/galaxyTools/galaxy-central/eggs/wchartype-0.1-py2.6.egg, /mnt/galaxyTools/galaxy- central/eggs/elementtree-1.2.6_20050316-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/docutils-0.7-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/WebOb-0.8.5-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/Routes-1.12.3-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/Cheetah-2.2.2-py2.6-linux- x86_64-ucs4.egg, /mnt/galaxyTools/galaxy-central/eggs/PasteDeploy-1.3.3-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/PasteScript-1.7.3-py2.6.egg, /mnt/galaxyTools/galaxy-central/eggs/Paste-1.6-py2.6.egg, /mnt/galaxyTools/galaxy-central/lib, /usr/lib/python2.6/, /usr/lib/python2.6/plat-linux2, /usr/lib/python2.6/lib-tk, /usr/lib/python2.6/lib-old, /usr/lib/python2.6/lib-dynload Traceback (most recent call last): File "/mnt/galaxyTools/galaxy-central/lib/galaxy/web/buildapp.py", line 82, in app_factory app = UniverseApplication( global_conf = global_conf, **kwargs ) File "/mnt/galaxyTools/galaxy-central/lib/galaxy/app.py", line 24, in __init__ self.config.check() File "/mnt/galaxyTools/galaxy-central/lib/galaxy/config.py", line 243, in check tree = parse_xml( config_filename ) File "/mnt/galaxyTools/galaxy-central/lib/galaxy/util/__init__.py", line 105, in parse_xml tree = ElementTree.parse(fname) File "/mnt/galaxyTools/galaxy-central/eggs/elementtree-1.2.6_2005031 6-py2.6.egg/elementtree/ElementTree.py", line 859, in parse tree.parse(source, parser) File "/mnt/galaxyTools/galaxy-central/eggs/elementtree-1.2.6_2005031 6-py2.6.egg/elementtree/ElementTree.py", line 576, in parse source = open(source, "rb") IOError: [Errno 2] No such file or directory: './migrated_tools_conf.xml' Removing PID file paster.pid While I'm here, I see a new Galaxy Cloudman AMI 072133624695/galaxy-cloudman-2012-02-26. I can't manage to start that, I get an error as below, with all types of instance, (tiny/small/medium/large). Is that a recommended AMI now ? It would be good to have a new updated AMI. [image: Inline image 1] Thanks ! -- Greg Edwards, Port Jackson Bioinformatics gedwards2@gmail.com

samtools bam • 1.2k views

ADD COMMENT • link •

modified 6.7 years ago • written 6.7 years ago by Greg Edwards • 190

6.7 years ago by

Dannon Baker ♦ 3.7k

United States

Dannon Baker ♦ 3.7k wrote:

Greg, The problem here is that the galaxy update failed to merge a change to run.sh because of minor customizations it has. We'll have a long term fix out for this soon, but for now what you can do is ssh in to your instance and update run.sh yourself prior to restarting galaxy. All you need to do is add 'migrated_tools_conf.xml.sample' to the SAMPLES in /mnt/galaxyTools/galaxy-central/run.sh, execute `sh run.sh --run- daemon` (or restart galaxy again from the admin page) and you should be good to go. That new AMI you're seeing is not owned by the Galaxy Team, and we don't actually know who made it. Keep using the same galaxy- cloudman-2011-03-22 for now (and we'll always have the most up-to-date AMI listed at usegalaxy.org/cloud) Because of Cloudman's modular volume design almost nothing resides on the AMI itself, so we can (and do) update the tools and index volumes without having to touch it. So while the AMI reflects a date of almost a year old, the galaxy tools volume (and thus the actual Galaxy instance you're running) has been updated much more recently. One last note- if you're updating and copying your tools in every time, you may want to try using the 'Persist changes' functionality available in the Cloudman admin panel. Once you've set your instance up how you want, if you click 'Persist changes to galaxyTools', it'll create a custom snapshot of your tools volume that will be used from that point forward with this instance. Let me know if you have any more issues, -Dannon

ADD COMMENT • link written 6.7 years ago by Dannon Baker ♦ 3.7k

Dannon, Thanks for that. Adding 'migrated_tools_conf.xml.sample' fixed up most problems. I'm left with a couple though. 1. The tools (apart from file upload with Get Data) don't initially execute. This includes my new tools and existing ones like Text Manipulation / Paste Files side-by-side. They stay in the grey "waiting to run" mode for about 15-30 minutes, then run. 2. When they finally run, they compute and report fine but finish in the Red error colour, and stderr has reported as below.This is both my tools and the system standard ones. WARNING:galaxy.datatypes.registry:Overriding conflicting datatype with extension 'coverage', using datatype from /mnt/galaxyData/tmp/tmp6o5hnt. There's an oddity I should mention. When I first fire up galaxy-cloudman-2011-03-22, a remnant of my add-on tools is already showing at the top of the Tools menu. "PTM Tools" is there, and in tools_conf.xml. But the tools are not in tools/ptmtools, so the menu is inactive. It's like I've corrupted the file systems associated with galaxy-cloudman-2011-03-22. I'm thinking of trying it all on the Singapore AWS node where I can't have touched the AMI's or file sets. This has all been on US East. In fact it's been on US East 1d, I could try on 1a, I never go there normally, but they probably share data. I could try on US West. I'll let you know, if I don't hear before. Greg. -- Greg Edwards, Port Jackson Bioinformatics gedwards2@gmail.com

ADD REPLY • link written 6.7 years ago by Greg Edwards • 190

Since Dannon is likely asleep, maybe this will help until he can respond: http://dev.list.galaxyproject.org/error-on-fastq-file-while-grooming- or-fastqc-report-or-fastqc-converter-td4396664.html ie: try removing the second coverage datatype definition from datatypes_conf.xml and restarting the Galaxy process. -- Ross Lazarus MBBS MPH; Associate Professor, Harvard Medical School; Head, Medical Bioinformatics, BakerIDI; Tel: +61 385321444;

ADD REPLY • link written 6.7 years ago by fubar ♦ 1.1k

Also Greg Starting a new cloudman instance with a different cluster name than previously used will always give you a clean environment and setup so you can continue to always use the us-east region. Cheers, Enis

ADD REPLY • link written 6.7 years ago by Enis Afgan • 690

6.7 years ago by

Greg Edwards • 190

Greg Edwards • 190 wrote:

Team, I'm up and running fine, thanks. Thanks Dannon for the info on 'migrated_tools_conf.xml.sample', that allowed a normal update. Thanks Ross for the info on coverage datatype definition from datatypes_conf.xml. Thanks Enis for the tip that a new clustername guarantees a clean new Galaxy. I've found something that may be of interest. A few days ago AWS EC2 added an m1.small and m1.medium model for 64-bit AMI's, and Galaxy starts ok with these. But it has a timing problem that I mentioned before. On m1.medium the jobs sit in the History queue in the gray waiting state for 1 to 30 minutes before they run. They run ok after that. On m1.small they sit there forever and don't run. Or they certainly don't run within 5 to 6 hours. Things work ok (or used to anyway) on t1.micro so I'm guessing there's a 'case' statement on instance type somewhere that is falling through for m1.medium and m1.small. (A wild guess but you know how it goes, it's usually simple :=) One last gremlin. My new tools are added at the top of tool_conf.xml and have always shown up at the top of the Tools list, in a local Galaxy and previous Cloud Galaxies. Now they show up near the bottom of the tools list. There's no obvious ordering happening like alphabetical tool name or file-date or name of tools/tooldir. Any ideas here appreciated. Thanks ! Greg E -- Greg Edwards, Port Jackson Bioinformatics gedwards2@gmail.com

ADD COMMENT • link written 6.7 years ago by Greg Edwards • 190

Please log in to add an answer.

Similar posts • Search »