Dear Galaxy Community
I have a small production instance of galaxy, installed following the tutorials from the galaxy team (using PostGres, NGINX, supervisord, virtualenv, etc.). Lately, I noticed that my galaxy was slow and laggy. Sometimes, it would not even refresh properly. I decided to restart Galaxy through supervisord (systemctl restart supervisord) and since them, I have a lot of issues.
First of all, restarting or stopping supervisord do not work properly (it freezes my terminal). Starting supervisord works, but galaxy is never available through my browser (and supervisorctl status states that the processes are running). Looking at the uwsgi.log file, I noticed that galaxy seems stalled while loading, as nothing happens after doing the following step:
galaxy.tools.toolbox.base DEBUG 2018-01-11 13:56:25,837 Loading tool panel finished (60.742 ms)
Trying to launch galaxy manually through "sh run.sh" lead the the same result. Nothing was done on our cluster lately: no updates, no config file changes. I only recently installed some galaxy tools, through the toolshed. I noticed also some very strange behaviors when galaxy is in this "stalled" process: I can not list files in the galaxy folder (using "ls"); it freezes, or access/copy files somewhere else (it also freezes).
If I launch galaxy manually, and kill it manually (sh run.sh, and then crtl+c when it is stalled), I get the following traceback
galaxy.tools.toolbox.base DEBUG 2018-01-11 13:56:25,837 Loading section: Sandbox Tools galaxy.tools.toolbox.base DEBUG 2018-01-11 13:56:25,837 Loading tool panel finished (60.742 ms) ^C^C^C^C^CTraceback (most recent call last): File "/mnt/extStorage/galaxy-app/lib/galaxy/webapps/galaxy/buildapp.py", line 58, in paste_app_factory app = galaxy.app.UniverseApplication(global_conf=global_conf, **kwargs) File "/mnt/extStorage/galaxy-app/lib/galaxy/app.py", line 117, in __init__ self._configure_toolbox() File "/mnt/extStorage/galaxy-app/lib/galaxy/config.py", line 955, in _configure_toolbox self.toolbox = tools.ToolBox(tool_configs, self.config.tool_path, self) File "/mnt/extStorage/galaxy-app/lib/galaxy/tools/__init__.py", line 226, in __init__ app=app, File "/mnt/extStorage/galaxy-app/lib/galaxy/tools/toolbox/base.py", line 1053, in __init__ super(BaseGalaxyToolBox, self).__init__(config_filenames, tool_root_dir, app) File "/mnt/extStorage/galaxy-app/lib/galaxy/tools/toolbox/base.py", line 85, in __init__ self._save_integrated_tool_panel() File "/mnt/extStorage/galaxy-app/lib/galaxy/tools/toolbox/integrated_panel.py", line 46, in _save_integrated_tool_panel self._write_integrated_tool_panel_config_file() File "/mnt/extStorage/galaxy-app/lib/galaxy/tools/toolbox/integrated_panel.py", line 105, in _write_integrated_tool_panel_config_file shutil.move(filename, destination) File "/usr/lib64/python2.7/shutil.py", line 293, in move os.rename(src, real_dst) KeyboardInterrupt
I have never seen this kind of issues before, and it really puzzles me. Anyone may have an idea?
Sincerely
David
