Question: Help with local galaxy upload performance
0
gravatar for Philipp.rathert
3.0 years ago by
Germany
Philipp.rathert30 wrote:

Dear all,

 

I need some help.

 

I am running a Galaxy instance on a Mac pro (installed about 2 weeks ago) for our group. 4-5 people are supposed to use this computer frequently with no more than 3 at the same time. Due to the fact that I have no, and with no I really mean no, background in bioinformatics or informatics whatsoever I have a lot of questions.

 

A big issue is the upload speed of the data library function.

If I want to upload large files (>3GB) it takes ages. I stopped after 24h. The file was located on the same disk and writing 3.2gb into galaxy took this long?

Normally this should be a way to upload large data files without using FTP, but with me this is impossible.

Any idea what this could be?

Is this restricted via python or another tool?

 

In the meantime I will try to set up FTP upload, but It would be great if the issue could be fixed, I am afraid that this issue might affect other processes as well.

 

There are many more little problems which I am going to bring up soon but these are the most urgent ones I think.

 

Any help from somebody experienced (with OS X) is really appreciated!

ADD COMMENTlink modified 2.8 years ago • written 3.0 years ago by Philipp.rathert30

Hi Philip

 

Next time, it is better to submit 3 individual question. This makes it easier to answer and further down the line, easier for others to find the question/answers if they run into similar problems.

 

Let me try answer your third problem, but first:

 - how do you upload the data?  i.e which "Upload option" have you used?

 - have you set the "File Format" ?

 

Hans-Rudolf

ADD REPLYlink written 3.0 years ago by Hotz, Hans-Rudolf1.8k

Dear Hans-Rudolf,

I will seperate my next questions. I agree this makes more sense.

 

I used the data library function and tried to upload a single file but used the folder upload function as well. I had the File Format on autodetect most times but for the single file I set it to fasta once.

 

ADD REPLYlink modified 3.0 years ago • written 3.0 years ago by Philipp.rathert30

You should see a difference in the time it takes to upload the same file with or without pre-selecting 'fasta', don't you?

If I understand you correctly, you are using the 'upload directory of files' option....do you see a difference in the time it takes to upload the same file if you set "Copy data into Galaxy?" to 'link to files' ?

ADD REPLYlink written 3.0 years ago by Hotz, Hans-Rudolf1.8k

I do not see a difference if I select the file format or if I don't. I kill the upload after 12-24h when the file is still uploading.

However, I do see a big difference when I link the files. 10GB are there in 15 mins...But would this be a workaround for users as well?

I could allow folder upload to users and give them a shared folder on the machine?

 

ADD REPLYlink written 3.0 years ago by Philipp.rathert30

Don't know, as we don't offer data library generation to users

ADD REPLYlink written 3.0 years ago by Hotz, Hans-Rudolf1.8k

ok then this does not seem to be an option.

ADD REPLYlink written 3.0 years ago by Philipp.rathert30

post edited by philipp

ADD REPLYlink modified 3.0 years ago • written 3.0 years ago by Philipp.rathert30

post edited by philipp

ADD REPLYlink modified 3.0 years ago • written 3.0 years ago by Philipp.rathert30
1
gravatar for Philipp.rathert
2.8 years ago by
Germany
Philipp.rathert30 wrote:

We switched to docker now and this was a game changer!

I can highly recommend this...

ADD COMMENTlink written 2.8 years ago by Philipp.rathert30
0
gravatar for Guy Reeves
3.0 years ago by
Guy Reeves1.0k
Germany
Guy Reeves1.0k wrote:

Hi Philipp

I am not sure it will be the most direct solution for you, but like you I have limited experience with bioinformatics and I struggled to install an administer galaxy on various machines.  Since then I have swapped to installing galaxy using docker (you will need to install docker first on your machine) and have been able to get things working stably.  I have successfully enabled the FTP and regularly upload files of >5Gb

https://github.com/bgruening/docker-galaxy-stable 

Will work fine on a Mac.  Just a thought.  Thanks Guy

ADD COMMENTlink written 3.0 years ago by Guy Reeves1.0k
0
gravatar for Philipp.rathert
3.0 years ago by
Germany
Philipp.rathert30 wrote:

Thanks for the suggestion but i don't want to give up right now...

ADD COMMENTlink written 3.0 years ago by Philipp.rathert30
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 171 users visited in the last hour