Question: Import data via http link that requires a username and password?
2
gravatar for steve
14 months ago by
steve30
NYC
steve30 wrote:

I saw the information here on how to import files via FTP. Our users have access to their files via http (actually https), but the server requires them to enter a username and password to login and access the files. I tried to upload some data by going to '(Get Data) > Upload from Computer > Paste / Fetch data' and then entering the URL. However, when Galaxy tries to retrieve the file, it gets an error like this:

Enter username for Restricted Files at dataserver.edu: Unable to fetch https://dataserver.edu/results/lab/ChIP-Seq/alignment/bigwigs/sample1.bw EOF when reading a line

Obviously, this should be due to the server's login requirements.

Is there a way to enable users to login through Galaxy or somehow pass along their credentials?

secure upload ftp galaxy url • 460 views
ADD COMMENTlink modified 14 months ago by Jennifer Hillman Jackson23k • written 14 months ago by steve30
2
gravatar for Martin Čech
14 months ago by
Martin Čech ♦♦ 4.2k
United States
Martin Čech ♦♦ 4.2k wrote:

I think Galaxy does not have means to authenticate against FTP/HTTP server with user credentials.

I recommend directly connecting to the data location if possible and uploading the data to Galaxy's FTP server at ftp://usegalaxy.org

Alternatively you could proxy the connection through your station if you do not have direct access to the host machine.

ADD COMMENTlink modified 14 months ago • written 14 months ago by Martin Čech ♦♦ 4.2k

We are using our own self-hosted Galaxy Docker instance. I tried using that FTP wiki page guide to connect in CyberDuck using the same address & port that we are using the access the Galaxy instance via web browser (e.g. ftp://admin@ourserver.org:8080/ ) but could not get it to connect, it timed out instead. I think the http links would be more ideal because users would be able to upload it themselves and it would avoid the server permissions issues. Each user here has http access to their designated lab directory (on a separate server from our Galaxy) via browser login, so if there was a way to pass along the credentials when Galaxy tries to access the data it would make it easy for users to import data from their directories via http access.

ADD REPLYlink modified 14 months ago • written 14 months ago by steve30

sometimes the links can include the authentication in the url like http://someserver.us/file1.txt?username=test&password=secret - please check whether yours can do that

edit: this is fairly unsafe option even if it works since the password will be in the Galaxy's logs

ADD REPLYlink modified 14 months ago • written 14 months ago by Martin Čech ♦♦ 4.2k

yes it is possible to do this with http://username:password@someserver.us/file1.txt and yes this is obviously an unacceptable solution. The URL is recorded in the user's History, visible from the front page, among other places. Thanks.

ADD REPLYlink written 14 months ago by steve30

I think Galaxy does not have means to authenticate against FTP/HTTP server with user credentials.

Would it be non-trivial to develop a script/tool/module for Galaxy to do this? Supply a list of http links, and username/password to use, and access & import the data that way?

ADD REPLYlink modified 14 months ago • written 14 months ago by steve30
2

This depends on how hacky/unsafe you are willing to go. Galaxy does not have means to safely handle passwords other than the login passwords. I would probably write a standalone (non-galaxy) script that fetches the data and puts them in an import (or FTP) directory so then you can either add them to history via upload modal or to data libraries in Galaxy.

edit: you would probably want to wrap this script as a Galaxy's tool and make it put data in the current user's import folder

ADD REPLYlink modified 14 months ago • written 14 months ago by Martin Čech ♦♦ 4.2k

Thanks, I will look into this. I also saw this page which describes implementing the FTP uploads, and of course the pages on how to upload DataLibraries, but I am not sure if there is any relevant information on how get started with building a tool to interact with Galaxy's file importation systems. Are there any examples of similar tools? Maybe in the Galaxy or Galaxy Docker repositories?

ADD REPLYlink modified 14 months ago • written 14 months ago by steve30
2

Tutorials for Galaxy tool wrapping are here and here. I do not think you need to interact with Galaxy much. You need to show a form to user, let them put in the url, username and password and forward this to your script. Then fetch the data and copy to the proper folder.

ADD REPLYlink written 14 months ago by Martin Čech ♦♦ 4.2k
1

if you set this path in your Galaxy config and make your new import script to copy the data to the user's subfolder (named by their email) the user can then import them to a data library. Alternatively if you set the FTP import dir config in a similar way users will be able to use upload modal to add files from the folder to their histories.

ADD REPLYlink written 14 months ago by Martin Čech ♦♦ 4.2k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 114 users visited in the last hour