Question: Problems with FTP uploads to cloudman
0
gravatar for jsr2137
3.0 years ago by
jsr21370
United States
jsr21370 wrote:

Hi,

I am brand new to Galaxy and am currently having trouble uploading files to Cloudman via FTP. Specifically, I have been trying to upload several .bam files since last night using Filezilla, In the 16+ hour period, one of the .bam files uploaded successfully and I can access it in cloudman. The others have been continually failing and restarting during that whole period. Here is what I see as an upload fails and automatically restarts:

Command:    PASV
Response:    227 Entering Passive Mode (52,23,160,199,117,98).
Command:    LIST
Response:    150 Opening BINARY mode data connection for file list
Response:    226 Transfer complete
Command:    PASV
Response:    227 Entering Passive Mode (52,23,160,199,117,83).
Command:    STOR US-1584693.bam
Response:    150 Opening BINARY mode data connection for US-1584693.bam
Error:    Could not write to transfer socket: ECONNABORTED - Connection aborted
Response:    451 Transfer aborted. Input/output error
Error:    File transfer failed after transferring 1,370,750,976 bytes in 363 seconds
Status:    Starting upload of F:\Jeremy\ANALYZED_DATA\Project_15903_FUID1025093-2\US-1584693\US-1584693.bam
Status:    Retrieving directory listing of "/"...

 

This may be part of another question, but I thought I was perhaps getting this error for having not enough disk space (EBS). I assumed that of the 3 EBS volumes that were created with cloudman, the 20gb one was for user data (not sure if this is true). So, I made a snapshot, increased the volume to well above what was needed and attached this new EBS volume to my EC2 instance (and detached the older one). The instance worked fine after this, but FTP uploads gave me same error.

Any suggestions? Hopefully, I am just missing something obvious.

In case you are wondering why I am loading .bam files: Someone else ran cuffmerge with my data and I am trying to upload the .bam files and .gtf file to start cuffdiff with cloudman. This seemed reasonable, but I am new to this and not sure if this could be part of the problem. 

Thanks for any help you can provide,

Jeremy

ftp cloudman • 1.4k views
ADD COMMENTlink modified 3.0 years ago • written 3.0 years ago by jsr21370
1
gravatar for Enis Afgan
3.0 years ago by
Enis Afgan690
United States
Enis Afgan690 wrote:

I'm having trouble replicating this on a newly launched instance - I've tried uploading several .bam files from .7 to 4.5GB in size and they all worked fine (the 4.5GB one took about 1.5mins). Perhaps increasing verbosity of the FileZilla logs will give some more info (https://wiki.filezilla-project.org/Logs) - it could have something to do with the network. How old is your CloudMan instance? Can you check on your instance if file /usr/proftpd/etc/proftpd.conf has a line like the following: MasqueradeAddress <ip address="">

Re. file system size, the data gets uploaded to the /mnt/galaxy file system and running the df command will tell you if there is sufficient room to store the data.

ADD COMMENTlink modified 3.0 years ago • written 3.0 years ago by Enis Afgan690
0
gravatar for jsr2137
3.0 years ago by
jsr21370
United States
jsr21370 wrote:

Thanks for the help. I tried again today and still have same issue. Below is verbose log of error:

Status:    Starting upload of G:\ANALYZED_DATA\Project_15903_FUID1025093-3\US-1584765\US-1584765.bam
Trace:    CFtpControlSocket::ParseSubcommandResult(0)
Trace:    FileTransferSubcommandResult()
Status:    Retrieving directory listing of "/"...
Trace:    ftpcontrolsocket.cpp(1274): List called from other command   caller=000007FFFD4A6430
Trace:    CFtpControlSocket::ParseSubcommandResult(0)
Trace:    CFtpControlSocket::ListSubcommandResult()
Trace:    CFtpControlSocket::SendNextCommand()
Trace:    CFtpControlSocket::TransferSend()
Command:    PASV
Trace:    CFtpControlSocket::OnReceive()
Response:    227 Entering Passive Mode (54,208,166,236,117,137).
Trace:    CFtpControlSocket::TransferParseResponse()
Trace:    CFtpControlSocket::SendNextCommand()
Trace:    CFtpControlSocket::TransferSend()
Trace:    Binding data connection source IP to control connection source IP 140.142.102.198
Command:    LIST
Trace:    CTransferSocket::OnConnect
Trace:    CFtpControlSocket::OnReceive()
Response:    150 Opening BINARY mode data connection for file list
Trace:    CFtpControlSocket::TransferParseResponse()
Trace:    CFtpControlSocket::SendNextCommand()
Trace:    CFtpControlSocket::TransferSend()
Trace:    CTransferSocket::TransferEnd(1)
Trace:    CFtpControlSocket::TransferEnd()
Trace:    CFtpControlSocket::OnReceive()
Response:    226 Transfer complete
Trace:    CFtpControlSocket::TransferParseResponse()
Trace:    CFtpControlSocket::ResetOperation(0)
Trace:    CControlSocket::ResetOperation(0)
Trace:    CFtpControlSocket::ParseSubcommandResult(0)
Trace:    CFtpControlSocket::ListSubcommandResult()
Trace:    CFtpControlSocket::ResetOperation(0)
Trace:    CControlSocket::ResetOperation(0)
Trace:    CFtpControlSocket::ParseSubcommandResult(0)
Trace:    FileTransferSubcommandResult()
Trace:    CFtpControlSocket::SendNextCommand()
Trace:    FileTransferSend()
Trace:    CFtpControlSocket::SendNextCommand()
Trace:    CFtpControlSocket::TransferSend()
Command:    PASV
Trace:    CFtpControlSocket::OnReceive()
Response:    227 Entering Passive Mode (54,208,166,236,117,143).
Trace:    CFtpControlSocket::TransferParseResponse()
Trace:    CFtpControlSocket::SendNextCommand()
Trace:    CFtpControlSocket::TransferSend()
Command:    REST 880934912
Trace:    CFtpControlSocket::OnReceive()
Response:    350 Restarting at 880934912. Send STORE or RETRIEVE to initiate transfer
Trace:    CFtpControlSocket::TransferParseResponse()
Trace:    CFtpControlSocket::SendNextCommand()
Trace:    CFtpControlSocket::TransferSend()
Trace:    Binding data connection source IP to control connection source IP 140.142.102.198
Command:    STOR US-1584765.bam
Trace:    CTransferSocket::OnConnect
Trace:    CFtpControlSocket::OnReceive()
Response:    150 Opening BINARY mode data connection for US-1584765.bam
Trace:    CFtpControlSocket::TransferParseResponse()
Trace:    CFtpControlSocket::SendNextCommand()
Trace:    CFtpControlSocket::TransferSend()
Error:    Could not write to transfer socket: ECONNABORTED - Connection aborted
Trace:    CTransferSocket::TransferEnd(3)
Trace:    CFtpControlSocket::TransferEnd()
Trace:    CFtpControlSocket::OnReceive()
Response:    451 Transfer aborted. Input/output error
Trace:    CFtpControlSocket::TransferParseResponse()
Trace:    CFtpControlSocket::ResetOperation(2)
Trace:    CControlSocket::ResetOperation(2)
Trace:    CFtpControlSocket::ParseSubcommandResult(2)
Trace:    FileTransferSubcommandResult()
Trace:    CFtpControlSocket::ResetOperation(2)
Trace:    CControlSocket::ResetOperation(2)
Error:    File transfer failed after transferring 474,480,640 bytes in 126 seconds
Trace:    CFtpControlSocket::FileTransfer()
Status:    Starting upload of G:\ANALYZED_DATA\Project_15903_FUID1025093-3\US-1584765\US-1584765.bam

Also, I noticed there is message abotu an insecure server every time I connect with Filezilla:

Status:    Connection established, waiting for welcome message...
Status:    Insecure server, it does not support FTP over TLS.
Status:    Server does not support non-ASCII characters.
Status:    Connected
Status:    Retrieving directory listing...
Status:    Calculating timezone offset of server...
Status:    Timezone offset of server is 0 seconds.
Status:    Directory listing of "/" successful

Not sure what that means.

In regards to CloudMan, I started the instance this week, with the most current release from last month. I don't know how to check the other information you mentioned though. Is this available directly through the instance online (from a Windows machine)? I managed to get admin access on my account, but I assume I have to access the account differently to try what you suggested. Sorry, I imagine this is straightforward, just not really sure what I'm doing...

Thanks again.

 

ADD COMMENTlink written 3.0 years ago by jsr21370
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 126 users visited in the last hour