Question: Problem In Uploading The Data
gravatar for asifullah khan
8.3 years ago by
asifullah khan10 wrote:
Dear Officer, I am a new user to assemble 75bp illumina solexa data. I have done single read illumina sequencing of my DNA of interest. it is a single file of about 370MB. The data, in file is initially in FASTQ format but letter the data arrangement become in some different kind of FASTq format inside the file. I want to convert the whole data in to simple FASTA format using GALAXY tools. I see the interesting videos on your site about using galaxy and it seems very interesting. But here i fail to upload my 370MB data file as an initial step. I am using 100Mbp LANE networking. Are you people preferring any other special network connection for uploading such huge files on GALAXY.? Kindly guide in this aspect. Moreover Kindly if possible then send me simple perl script and there command lines usage description for converting any format of FASTq in to FASTA. Regards, Asifullah Khan Research officer, DNA sequencing Labs, ICCBS, University of Karachi, Pakistan.
galaxy • 1.4k views
ADD COMMENTlink modified 8.3 years ago by Nate Coraor3.2k • written 8.3 years ago by asifullah khan10
gravatar for Kevin
8.3 years ago by
Kevin150 wrote:
I have created a local install of galaxy and i can tell you it is not about network connection. As I am using a local install but the 'upload' is horribly slow. a 3Gb file can't finish in 24 hours. in the end, I used a python script to 'serve' the file via http protocol and entered a local LAN url for galaxy and it loaded up in minutes. For your case, I think I have no issues up to 250 Mb on their galaxy main server, but as the wiki and manual (plus my experience with a local instance) suggests you would do well to put the file on a webserver and give the url instead of upload via browser. Cheers Kevin
ADD COMMENTlink written 8.3 years ago by Kevin150
Dear Asifullah As Kevin says, you are much better off by using http or ftp url, Also i know there are issues if you refresh the browser while uploading, So I guess if uploading does not work normally, try putting on http or ftp server and give a url, and do not touch the browser untill uploading is over also you can alternatively use a perl or python script locally best Abhay Krishna
ADD REPLYlink written 8.3 years ago by Abhay Krishna70
I ran into this issue too repeatedly. Both with local install of Galaxy and the main instance. Files over roughly 2GB or so refused to upload, the page would still show them as uploading but nothing was happening. Compressing the same file so it was under 2GB and uploading worked fine both on the local instance and main. -Farhat -- Farhat Habib, PhD IISER Fellow, Biology Indian Institute of Science Education & Research, Sai Trinity Building, Garware Circle, Sutarwadi, Pashan, Pune-411 021. Office: +91 20 2590 8015 Cell: +91 973 0996 233
ADD REPLYlink written 8.3 years ago by Farhat Habib80
Hi Farhat and others, Thanks for reporting your experiences. Does it seem to be everyone's experience that direct file uploads of files > 2GB fails? We are looking in to more reliable methods for handling direct uploads, so any suggestions are welcome. --nate
ADD REPLYlink written 8.3 years ago by Nate Coraor3.2k
Hi Nate, While I do not have ideas to improve browser uploads but i do have some untested ideas for uploads for ppl who like me are unlikely to have access to the institute's ftp or http webserver to serve files. It might be a tad convoluted but have you considered dropbox integration for galaxy? I have terrible upload speeds for amazon S3 but that might be a consideration for some. with regards to local install, is there a way I can point galaxy directly to the file? or make it access it directly instead of my workaround by using python /perl scripts to emulate a webserver? Cheers Kevin
ADD REPLYlink written 8.3 years ago by Kevin150
Hi Kevin, step into your directory and execute: python -m SimpleHTTPServer 8082 Your files will be accesible from: http://ipaddress:8082/ Source: python-simplehttpserver/ Cheers, Jelle
ADD REPLYlink written 8.3 years ago by Jelle Scholtalbers360
Hi Kevin, Thanks for the suggestion. The donwside I see here is that it'd be a two step process to get it to S3 and then from there to Galaxy (and in addition, it's not free). Ideally we'd need a method to upload to Galaxy directly, but without the browser flakiness. Of course we can do this now with something like FTP, but a browser based solution would be more ideal. Not for history uploads, but you can with Library uploads (and then import the library item into your history): central/wiki/DataLibraries/UploadingFiles --nate
ADD REPLYlink written 8.3 years ago by Nate Coraor3.2k
Quick supplement to what Nate said, if you did want to ship data from dropbox -> galaxy, you could (temporarily) throw the file to be shipped in your dropbox public folder and then put that public download link into galaxy's upload box and it would work fine. -Dannon
ADD REPLYlink written 8.3 years ago by Dannon Baker3.7k
Yes, that would be fine for data that is not sensitive. I believe dropbox's API is publicly avail? perhaps there can be a dropbox user acct for each galaxy user acct? I acknowlegde Nate's concern that its an additional waste of bandwidth hence I mentioned it's a convoluted way of getting data across. I imagine it might be helpful for those that already use such services to store their data on the cloud. That said, has anyone considered the use of java uploaders that allow drag and drop like those for flickr and dropbox? Just tossing ideas about really. sorry if i sound like I don't know what I am saying. Cheers Kevin
ADD REPLYlink written 8.3 years ago by Kevin150
I haven't used dropbox, but flickr's browser uploader uses Flash, which loads the entire file into memory before sending (unfeasible for large files). Java is certainly something we've considered as an alternate in-browser uploader (which doesn't replace the regular form). We'd like to stay away from any sort of desktop uploader application. Much appreciated, thanks! --nate
ADD REPLYlink written 8.3 years ago by Nate Coraor3.2k
The 2 GB is not a Galaxy limitation. In fact, web browsers are in general uncapable of uploading very large files. See these links: HTML is just not the best way to move big files around... Florent
ADD REPLYlink written 8.3 years ago by Florent Angly370
gravatar for Nate Coraor
8.3 years ago by
Nate Coraor3.2k
United States
Nate Coraor3.2k wrote:
Hi Asifullah, Have you retried the upload? Occasionally, uploads may be interrupted. We do try to handle this and mark the dataset in an error state, but sometimes Galaxy is not able to detect the failure, and it never changes out of the uploading state. As others have said, if you have an FTP or web site on which you can place the file, and then put the URL to that file in the URL/text box on the upload form, this method is more reliable. Unfortunately, browsers are not well suited for sending large files, especially if over a slow or flaky connection. --nate
ADD COMMENTlink written 8.3 years ago by Nate Coraor3.2k
Please log in to add an answer.


Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 172 users visited in the last hour