Import data via http link that requires a username and password?

Question: Import data via http link that requires a username and password?

2.2 years ago by

steve • 30

NYC

steve • 30 wrote:

I saw the information here on how to import files via FTP. Our users have access to their files via http (actually https), but the server requires them to enter a username and password to login and access the files. I tried to upload some data by going to '(Get Data) > Upload from Computer > Paste / Fetch data' and then entering the URL. However, when Galaxy tries to retrieve the file, it gets an error like this:

Enter username for Restricted Files at dataserver.edu: Unable to fetch https://dataserver.edu/results/lab/ChIP-Seq/alignment/bigwigs/sample1.bw EOF when reading a line

Obviously, this should be due to the server's login requirements.

Is there a way to enable users to login through Galaxy or somehow pass along their credentials?

secure upload ftp galaxy url • 1.1k views

ADD COMMENT • link •

modified 2.2 years ago by Jennifer Hillman Jackson ♦ 25k • written 2.2 years ago by steve • 30

2.2 years ago by

Martin Čech ♦♦ 4.9k

United States

Martin Čech ♦♦ 4.9k wrote:

I think Galaxy does not have means to authenticate against FTP/HTTP server with user credentials.

I recommend directly connecting to the data location if possible and uploading the data to Galaxy's FTP server at ftp://usegalaxy.org

Alternatively you could proxy the connection through your station if you do not have direct access to the host machine.

ADD COMMENT • link modified 2.2 years ago • written 2.2 years ago by Martin Čech ♦♦ 4.9k

We are using our own self-hosted Galaxy Docker instance. I tried using that FTP wiki page guide to connect in CyberDuck using the same address & port that we are using the access the Galaxy instance via web browser (e.g. ftp://admin@ourserver.org:8080/ ) but could not get it to connect, it timed out instead. I think the http links would be more ideal because users would be able to upload it themselves and it would avoid the server permissions issues. Each user here has http access to their designated lab directory (on a separate server from our Galaxy) via browser login, so if there was a way to pass along the credentials when Galaxy tries to access the data it would make it easy for users to import data from their directories via http access.

ADD REPLY • link modified 2.2 years ago • written 2.2 years ago by steve • 30

sometimes the links can include the authentication in the url like http://someserver.us/file1.txt?username=test&password=secret - please check whether yours can do that

edit: this is fairly unsafe option even if it works since the password will be in the Galaxy's logs

ADD REPLY • link modified 2.2 years ago • written 2.2 years ago by Martin Čech ♦♦ 4.9k

yes it is possible to do this with http://username:password@someserver.us/file1.txt and yes this is obviously an unacceptable solution. The URL is recorded in the user's History, visible from the front page, among other places. Thanks.

ADD REPLY • link written 2.2 years ago by steve • 30

I think Galaxy does not have means to authenticate against FTP/HTTP server with user credentials.

Would it be non-trivial to develop a script/tool/module for Galaxy to do this? Supply a list of http links, and username/password to use, and access & import the data that way?

ADD REPLY • link modified 2.2 years ago • written 2.2 years ago by steve • 30

This depends on how hacky/unsafe you are willing to go. Galaxy does not have means to safely handle passwords other than the login passwords. I would probably write a standalone (non-galaxy) script that fetches the data and puts them in an import (or FTP) directory so then you can either add them to history via upload modal or to data libraries in Galaxy.

edit: you would probably want to wrap this script as a Galaxy's tool and make it put data in the current user's import folder

ADD REPLY • link modified 2.2 years ago • written 2.2 years ago by Martin Čech ♦♦ 4.9k

Thanks, I will look into this. I also saw this page which describes implementing the FTP uploads, and of course the pages on how to upload DataLibraries, but I am not sure if there is any relevant information on how get started with building a tool to interact with Galaxy's file importation systems. Are there any examples of similar tools? Maybe in the Galaxy or Galaxy Docker repositories?

ADD REPLY • link modified 2.2 years ago • written 2.2 years ago by steve • 30

Tutorials for Galaxy tool wrapping are here and here. I do not think you need to interact with Galaxy much. You need to show a form to user, let them put in the url, username and password and forward this to your script. Then fetch the data and copy to the proper folder.

ADD REPLY • link written 2.2 years ago by Martin Čech ♦♦ 4.9k

if you set this path in your Galaxy config and make your new import script to copy the data to the user's subfolder (named by their email) the user can then import them to a data library. Alternatively if you set the FTP import dir config in a similar way users will be able to use upload modal to add files from the folder to their histories.

ADD REPLY • link written 2.2 years ago by Martin Čech ♦♦ 4.9k

Similar posts • Search »