Problem In Uploading The Data

Question: Problem In Uploading The Data

8.3 years ago by

Dear Officer, I am a new user to assemble 75bp illumina solexa data. I have done single read illumina sequencing of my DNA of interest. it is a single file of about 370MB. The data, in file is initially in FASTQ format but letter the data arrangement become in some different kind of FASTq format inside the file. I want to convert the whole data in to simple FASTA format using GALAXY tools. I see the interesting videos on your site about using galaxy and it seems very interesting. But here i fail to upload my 370MB data file as an initial step. I am using 100Mbp LANE networking. Are you people preferring any other special network connection for uploading such huge files on GALAXY.? Kindly guide in this aspect. Moreover Kindly if possible then send me simple perl script and there command lines usage description for converting any format of FASTq in to FASTA. Regards, Asifullah Khan Research officer, DNA sequencing Labs, ICCBS, University of Karachi, Pakistan.

galaxy • 1.4k views

ADD COMMENT • link •

modified 8.3 years ago by Nate Coraor ♦ 3.2k • written 8.3 years ago by asifullah khan • 10

8.3 years ago by

Kevin • 150

Kevin • 150 wrote:

I have created a local install of galaxy and i can tell you it is not about network connection. As I am using a local install but the 'upload' is horribly slow. a 3Gb file can't finish in 24 hours. in the end, I used a python script to 'serve' the file via http protocol and entered a local LAN url for galaxy and it loaded up in minutes. For your case, I think I have no issues up to 250 Mb on their galaxy main server, but as the wiki and manual (plus my experience with a local instance) suggests you would do well to put the file on a webserver and give the url instead of upload via browser. Cheers Kevin

ADD COMMENT • link written 8.3 years ago by Kevin • 150

Dear Asifullah As Kevin says, you are much better off by using http or ftp url, Also i know there are issues if you refresh the browser while uploading, So I guess if uploading does not work normally, try putting on http or ftp server and give a url, and do not touch the browser untill uploading is over also you can alternatively use a perl or python script locally http://seqanswers.com/forums/showthread.php?t=269 best Abhay Krishna

ADD REPLY • link written 8.3 years ago by Abhay Krishna • 70

I ran into this issue too repeatedly. Both with local install of Galaxy and the main instance. Files over roughly 2GB or so refused to upload, the page would still show them as uploading but nothing was happening. Compressing the same file so it was under 2GB and uploading worked fine both on the local instance and main. -Farhat -- Farhat Habib, PhD IISER Fellow, Biology Indian Institute of Science Education & Research, Sai Trinity Building, Garware Circle, Sutarwadi, Pashan, Pune-411 021. http://www.iiserpune.ac.in/~farhat Office: +91 20 2590 8015 Cell: +91 973 0996 233

ADD REPLY • link written 8.3 years ago by Farhat Habib • 80

Hi Farhat and others, Thanks for reporting your experiences. Does it seem to be everyone's experience that direct file uploads of files > 2GB fails? We are looking in to more reliable methods for handling direct uploads, so any suggestions are welcome. --nate

ADD REPLY • link written 8.3 years ago by Nate Coraor ♦ 3.2k

Hi Nate, While I do not have ideas to improve browser uploads but i do have some untested ideas for uploads for ppl who like me are unlikely to have access to the institute's ftp or http webserver to serve files. It might be a tad convoluted but have you considered dropbox integration for galaxy? I have terrible upload speeds for amazon S3 but that might be a consideration for some. with regards to local install, is there a way I can point galaxy directly to the file? or make it access it directly instead of my workaround by using python /perl scripts to emulate a webserver? Cheers Kevin

ADD REPLY • link written 8.3 years ago by Kevin • 150

Hi Kevin, step into your directory and execute: python -m SimpleHTTPServer 8082 Your files will be accesible from: http://ipaddress:8082/ Source: http://sysadmingeek.com/articles/serve-up-files-easily-with-the- python-simplehttpserver/ Cheers, Jelle

ADD REPLY • link written 8.3 years ago by Jelle Scholtalbers • 360

Hi Kevin, Thanks for the suggestion. The donwside I see here is that it'd be a two step process to get it to S3 and then from there to Galaxy (and in addition, it's not free). Ideally we'd need a method to upload to Galaxy directly, but without the browser flakiness. Of course we can do this now with something like FTP, but a browser based solution would be more ideal. Not for history uploads, but you can with Library uploads (and then import the library item into your history): http://bitbucket.org/galaxy/galaxy- central/wiki/DataLibraries/UploadingFiles --nate

ADD REPLY • link written 8.3 years ago by Nate Coraor ♦ 3.2k

Quick supplement to what Nate said, if you did want to ship data from dropbox -> galaxy, you could (temporarily) throw the file to be shipped in your dropbox public folder and then put that public download link into galaxy's upload box and it would work fine. -Dannon

ADD REPLY • link written 8.3 years ago by Dannon Baker ♦ 3.7k

Yes, that would be fine for data that is not sensitive. I believe dropbox's API is publicly avail? perhaps there can be a dropbox user acct for each galaxy user acct? I acknowlegde Nate's concern that its an additional waste of bandwidth hence I mentioned it's a convoluted way of getting data across. I imagine it might be helpful for those that already use such services to store their data on the cloud. That said, has anyone considered the use of java uploaders that allow drag and drop like those for flickr and dropbox? Just tossing ideas about really. sorry if i sound like I don't know what I am saying. Cheers Kevin

ADD REPLY • link written 8.3 years ago by Kevin • 150

I haven't used dropbox, but flickr's browser uploader uses Flash, which loads the entire file into memory before sending (unfeasible for large files). Java is certainly something we've considered as an alternate in-browser uploader (which doesn't replace the regular form). We'd like to stay away from any sort of desktop uploader application. Much appreciated, thanks! --nate

ADD REPLY • link written 8.3 years ago by Nate Coraor ♦ 3.2k

The 2 GB is not a Galaxy limitation. In fact, web browsers are in general uncapable of uploading very large files. See these links: http://kb.globalscape.com/Print10600.aspx http://www.motobit.com/help/ScptUtl/pa98.htm HTML is just not the best way to move big files around... Florent

ADD REPLY • link written 8.3 years ago by Florent Angly • 370

Similar posts • Search »