Problem with uploading >2GB .fa file to Data Library in local Galaxy server

Question: Problem with uploading >2GB .fa file to Data Library in local Galaxy server

3.8 years ago by

Singapore

jchen015 • 80 wrote:

Dear senior members

I am the admin of my local Galaxy server (as shown here: https://www.dropbox.com/s/bn0oviy5hi8ms4n/admin.png?dl=0).

I have a hg19.fa file which is 3.2GB to upload to my local Galaxy server and understanding that the space ceiling constraint is 2GB, I decided to leverage on the Data Library to do so.

So I created a data library namely Feb. 2009 (hg19, GRCh37) (as shown here: https://www.dropbox.com/s/ixli6o1mc43sxyp/Creation%20of%20data%20library.png?dl=0).

I added a my dataset (hg19.fa) onto the library created (as shown here: https://www.dropbox.com/s/8g321ceo7pxgo7w/Uploading%20of%20data.png?dl=0) but it seems to be loading for ever.

Also, knowing that my hg19.fa file is located at my directory (/home/user/), I modified my config.ini file to like this - https://www.dropbox.com/s/vb8axye7os155rm/library_import.png?dl=0

May I know what is the issue here? Why is it taking forever to load? It is a 3.2GB file and currently, it has been 2.5hours......

Regards,

Julius

local_galaxy datalibrary • 1.3k views

ADD COMMENT • link •

modified 3.7 years ago by Martin Čech ♦♦ 4.9k • written 3.8 years ago by jchen015 • 80

3.7 years ago by

Martin Čech ♦♦ 4.9k

United States

Martin Čech ♦♦ 4.9k wrote:

Hello Julius,

the 2GB is a browser limit so if you want to put bigger dataset into Galaxy you need to use different approach. For public instances that would be an FTP server and for a private instance it would be Data Library Import.

The config you have set (library_import_dir) tells the Data Library application where to look for data, so the only thing you need to do is to put your hg19.fa into that folder and then select "Add Datasets from the library administration. You should see this form: https://www.dropbox.com/s/zb74n8uwklax1g5/Screenshot%202015-03-01%2011.45.19.png?dl=0

In there you select 'Upload directory of files' and point it to any folder within the folder specified in the config. Moreover you can opt out from copying data to Galaxy (saves space) and choose the option "Link to files...". The import should be fast because the file will not be copied - merely analyzed.

Some more details are available here: https://wiki.galaxyproject.org/Admin/DataLibraries/UploadingLibraryFiles

Alternatively, given you are running fairly recent version of Galaxy, you can use the Data Libraries Beta interface which will allow you to browse the filesystem folders (defined in config) in the UI and select individual files to import (you need to set allow_library_path_paste = True for that to work).

Sorry for the lack of documentation, this is something we are currently working on.

Let us know if you have more questions.

Martin, Galaxy Team

ADD COMMENT • link written 3.7 years ago by Martin Čech ♦♦ 4.9k

Similar posts • Search »