Newest FASTQC repository not containing any dependencies

24 months ago by

Martin Čech ♦♦ 4.9k

United States

Martin Čech ♦♦ 4.9k wrote:

Hi, the change you are seeing is part of a process to transfer our dependencies to the Conda package manager (as opposed to using Tool Shed as package manager). You correctly identified that there is no tool_dependencies.xml file in the new versions of some repos - this does not have to mean you have to maintain the dependency yourself - it often means that you can have Galaxy use Conda package manager to obtain and maintain this dependency based on the <requirement> tag in the tool xml itself.

More details about Conda and tool dependencies available at ourdocs https://docs.galaxyproject.org/en/master/admin/conda_faq.html

Please have a look there and feel free to ask any questions here, we will try to answer them.

ADD COMMENT • link written 24 months ago by Martin Čech ♦♦ 4.9k

Hi, thanks a lot for your prompt response. In view of the fact that Galaxy is moving towards a Conda-based dependency managing system, I decided to give it a shot. However, I encountered a couple of issues - please bear in mind I had not used Conda before and I am also quite new to Galaxy.

1) I could not get Galaxy to install Conda itself with the conda_auto_init option, though I tried quite a few times. In the end I decided to install Miniconda myself, and I put it in the database/dependencies/ directory to match the default directory in the Galaxy.ini configuration file. This way it seems to work fine: it is detected by Galaxy at start up and I dependencies can be successfully installed.

2) There is no indication that Conda is indeed working or that dependencies are being installed. It would be nice to see some kind of message whenever Conda installs dependencies.

Aside from that, now I am running the latest version of FASTQC without any issues, which is fantastic.

ADD REPLY • link written 24 months ago by Inquirer • 80

I am curious about Galaxy failing to initialize Conda with conda_auto_init = True. Do you have galaxy logs you could share with us?

As of the indications - Galaxy's UI is behind compared to the state of the backend. However in the 'manage installed tools' under Admin menu you can click on a repository and in the detail you can see Dependency Resolver Details section. Expand it and you will see what type of dependencies you have available(tool_shed_package or Conda).

ADD REPLY • link written 24 months ago by Martin Čech ♦♦ 4.9k

I have tried once more and this time Galaxy was capable of installing Conda by itself. I am not sure what I did wrong the last time, but it works. Sorry for the unjustified alarm.

ADD REPLY • link written 24 months ago by Inquirer • 80

Hi again. Sorry for coming back with the same, but now I am having trouble running the FASTQC version with Conda dependencies. To be honest I don't remember if I ever managed to do so, or I just assumed everything was fine when I saw that dependencies had been installed by Conda in the Dependency Resolver Details section. However, I have not changed much of my Galaxy instance that could be related to this in the last days; moreover, older versions of FASTQC using the tool_shed dependencies are functional.

In particular, I get the following error:

/home/galaxy/galaxy/database/jobs_directory/000/79/tool_script.sh: line 9: database/dependencies/conda/bin/activate: No such file or directory
/home/galaxy/galaxy/database/jobs_directory/000/79/galaxy_79.sh: line 33: database/dependencies/conda/bin/activate: No such file or directory

If I then go to the scripts that throw the error, I encounter the following:

[ "$CONDA_DEFAULT_ENV" = "/home/galaxy/galaxy/database/jobs_directory/000/79/conda-metadata-env" ] || . database/dependencies/conda/bin/activate '/home/galaxy/galaxy/database/jobs_directory/000/79/conda-metadata-env' 2>&1

I suspect that the problem here is that database/dependencies/conda/bin/activate (i.e. conda_prefix) is a relative path to Conda, which is probably not executed from the galaxy main directory as it is expected. Any ideas of what I am doing wrong? Thanks a lot and sorry for the inconvenience!

ADD REPLY • link written 24 months ago by Inquirer • 80

Can you show me conda-related part of your galaxy.ini? It seems like you may have set up path to conda manually because the folder name in database/dependencies/conda should have been database/dependencies/_conda.

ADD REPLY • link written 24 months ago by Martin Čech ♦♦ 4.9k

Indeed, I changed it manually at some point. I first installed conda in database/dependencies/_conda because I had issues with the automatic installation, as reported above. After I fixed some apparently malformed lines in my Galaxy.ini file, I changed the conda_prefix to database/dependencies/conda for my next -this time successful- attempt at enabling the automated installation of Conda.

That being said, I have managed to make FASTQC (as well as other tools) work by modifying the conda_prefix path in Galaxy ini and making it absolute instead of relative. However, I am not sure this is the optimal solution because 1) all other paths in this file are relative to the Galaxy main directory, and 2) it is said that the prefix + the job_working_directory path should be less than 50 characters. But as I said, at least for now, it works. I attach here the relevant section of the Galaxy.ini configuration file:

# conda_prefix is the location on the filesystem where Conda packages and environments are installed
# IMPORTANT: Due to a current limitation in conda, the total length of the
# conda_prefix and the job_working_directory path should be less than 50 characters!
conda_prefix = /home/galaxy/galaxy/database/dependencies/conda

# Override the Conda executable to use, it will default to the one on the
# PATH (if available) and then to <conda_prefix>/bin/conda
# conda_exec =
# Pass debug flag to conda commands.
# conda_debug = False
# conda channels to enable by default (http://conda.pydata.org/docs/custom-channels.html)
# conda_ensure_channels = conda-forge,r,bioconda,iuc
# Set to True to instruct Galaxy to look for and install missing tool
# dependencies before each job runs.
# conda_auto_install = True
# Set to True to perform additional checking of installed Conda environment
conda_verbose_install_check=True
# Set to True to instruct Galaxy to install Conda from the web automatically
# if it cannot find a local copy and conda_exec is not configured.
conda_auto_init = True

Besides, the Galaxy UI reports the following message when I click on the output of FASTQC:

discarding /home/galaxy/galaxy/database/dependencies/conda/bin from PATH
prepending /home/galaxy/galaxy/database/jobs_directory/000/101/conda-env/bin to PATH

Is it fine like this, even though the prefix exceeds 50 characters?

EDIT: Now I have tried the default configuration (i.e. commenting out the conda_prefix line) and it also works fine, with much less trouble. I have noticed that it also uses an absolute path, though it is not obvious just by looking at the Galaxy.ini file.

ADD REPLY • link modified 24 months ago • written 24 months ago by Inquirer • 80

I would leave the conda_prefix commented out. Galaxy will then use database/dependencies/_conda folder which is what you probably want. Please note that whatever you installed using the other conda (in /conda) you will have to install again.

For the most clean approach I would recommend removing the whole _conda folder, commenting out conda_prefix, and have Galaxy install Conda again.

ADD REPLY • link modified 23 months ago • written 23 months ago by Martin Čech ♦♦ 4.9k

Similar posts • Search »