Problems with FTP uploads to cloudman

Question: Problems with FTP uploads to cloudman

3.1 years ago by

United States

jsr2137 • 0 wrote:

Hi,

I am brand new to Galaxy and am currently having trouble uploading files to Cloudman via FTP. Specifically, I have been trying to upload several .bam files since last night using Filezilla, In the 16+ hour period, one of the .bam files uploaded successfully and I can access it in cloudman. The others have been continually failing and restarting during that whole period. Here is what I see as an upload fails and automatically restarts:

Command:   PASV
Response:   227 Entering Passive Mode (52,23,160,199,117,98).
Command:   LIST
Response:   150 Opening BINARY mode data connection for file list
Response:   226 Transfer complete
Command:   PASV
Response:   227 Entering Passive Mode (52,23,160,199,117,83).
Command:   STOR US-1584693.bam
Response:   150 Opening BINARY mode data connection for US-1584693.bam
Error:   Could not write to transfer socket: ECONNABORTED - Connection aborted
Response:   451 Transfer aborted. Input/output error
Error:   File transfer failed after transferring 1,370,750,976 bytes in 363 seconds
Status:   Starting upload of F:\Jeremy\ANALYZED_DATA\Project_15903_FUID1025093-2\US-1584693\US-1584693.bam
Status:   Retrieving directory listing of "/"...

This may be part of another question, but I thought I was perhaps getting this error for having not enough disk space (EBS). I assumed that of the 3 EBS volumes that were created with cloudman, the 20gb one was for user data (not sure if this is true). So, I made a snapshot, increased the volume to well above what was needed and attached this new EBS volume to my EC2 instance (and detached the older one). The instance worked fine after this, but FTP uploads gave me same error.

Any suggestions? Hopefully, I am just missing something obvious.

In case you are wondering why I am loading .bam files: Someone else ran cuffmerge with my data and I am trying to upload the .bam files and .gtf file to start cuffdiff with cloudman. This seemed reasonable, but I am new to this and not sure if this could be part of the problem.

Thanks for any help you can provide,

Jeremy

ftp cloudman • 1.5k views

ADD COMMENT • link •

modified 3.1 years ago • written 3.1 years ago by jsr2137 • 0

3.1 years ago by

Enis Afgan • 690

United States

Enis Afgan • 690 wrote:

I'm having trouble replicating this on a newly launched instance - I've tried uploading several .bam files from .7 to 4.5GB in size and they all worked fine (the 4.5GB one took about 1.5mins). Perhaps increasing verbosity of the FileZilla logs will give some more info (https://wiki.filezilla-project.org/Logs) - it could have something to do with the network. How old is your CloudMan instance? Can you check on your instance if file /usr/proftpd/etc/proftpd.conf has a line like the following: MasqueradeAddress <ip address="">

Re. file system size, the data gets uploaded to the /mnt/galaxy file system and running the df command will tell you if there is sufficient room to store the data.

ADD COMMENT • link modified 3.1 years ago • written 3.1 years ago by Enis Afgan • 690

3.1 years ago by

jsr2137 • 0

United States

jsr2137 • 0 wrote:

Thanks for the help. I tried again today and still have same issue. Below is verbose log of error:

Status:   Starting upload of G:\ANALYZED_DATA\Project_15903_FUID1025093-3\US-1584765\US-1584765.bam
Trace:   CFtpControlSocket::ParseSubcommandResult(0)
Trace:   FileTransferSubcommandResult()
Status:   Retrieving directory listing of "/"...
Trace:   ftpcontrolsocket.cpp(1274): List called from other command caller=000007FFFD4A6430
Trace:   CFtpControlSocket::ParseSubcommandResult(0)
Trace:   CFtpControlSocket::ListSubcommandResult()
Trace:   CFtpControlSocket::SendNextCommand()
Trace:   CFtpControlSocket::TransferSend()
Command:   PASV
Trace:   CFtpControlSocket::OnReceive()
Response:   227 Entering Passive Mode (54,208,166,236,117,137).
Trace:   CFtpControlSocket::TransferParseResponse()
Trace:   CFtpControlSocket::SendNextCommand()
Trace:   CFtpControlSocket::TransferSend()
Trace:   Binding data connection source IP to control connection source IP 140.142.102.198
Command:   LIST
Trace:   CTransferSocket::OnConnect
Trace:   CFtpControlSocket::OnReceive()
Response:   150 Opening BINARY mode data connection for file list
Trace:   CFtpControlSocket::TransferParseResponse()
Trace:   CFtpControlSocket::SendNextCommand()
Trace:   CFtpControlSocket::TransferSend()
Trace:   CTransferSocket::TransferEnd(1)
Trace:   CFtpControlSocket::TransferEnd()
Trace:   CFtpControlSocket::OnReceive()
Response:   226 Transfer complete
Trace:   CFtpControlSocket::TransferParseResponse()
Trace:   CFtpControlSocket::ResetOperation(0)
Trace:   CControlSocket::ResetOperation(0)
Trace:   CFtpControlSocket::ParseSubcommandResult(0)
Trace:   CFtpControlSocket::ListSubcommandResult()
Trace:   CFtpControlSocket::ResetOperation(0)
Trace:   CControlSocket::ResetOperation(0)
Trace:   CFtpControlSocket::ParseSubcommandResult(0)
Trace:   FileTransferSubcommandResult()
Trace:   CFtpControlSocket::SendNextCommand()
Trace:   FileTransferSend()
Trace:   CFtpControlSocket::SendNextCommand()
Trace:   CFtpControlSocket::TransferSend()
Command:   PASV
Trace:   CFtpControlSocket::OnReceive()
Response:   227 Entering Passive Mode (54,208,166,236,117,143).
Trace:   CFtpControlSocket::TransferParseResponse()
Trace:   CFtpControlSocket::SendNextCommand()
Trace:   CFtpControlSocket::TransferSend()
Command:   REST 880934912
Trace:   CFtpControlSocket::OnReceive()
Response:   350 Restarting at 880934912. Send STORE or RETRIEVE to initiate transfer
Trace:   CFtpControlSocket::TransferParseResponse()
Trace:   CFtpControlSocket::SendNextCommand()
Trace:   CFtpControlSocket::TransferSend()
Trace:   Binding data connection source IP to control connection source IP 140.142.102.198
Command:   STOR US-1584765.bam
Trace:   CTransferSocket::OnConnect
Trace:   CFtpControlSocket::OnReceive()
Response:   150 Opening BINARY mode data connection for US-1584765.bam
Trace:   CFtpControlSocket::TransferParseResponse()
Trace:   CFtpControlSocket::SendNextCommand()
Trace:   CFtpControlSocket::TransferSend()
Error:   Could not write to transfer socket: ECONNABORTED - Connection aborted
Trace:   CTransferSocket::TransferEnd(3)
Trace:   CFtpControlSocket::TransferEnd()
Trace:   CFtpControlSocket::OnReceive()
Response:   451 Transfer aborted. Input/output error
Trace:   CFtpControlSocket::TransferParseResponse()
Trace:   CFtpControlSocket::ResetOperation(2)
Trace:   CControlSocket::ResetOperation(2)
Trace:   CFtpControlSocket::ParseSubcommandResult(2)
Trace:   FileTransferSubcommandResult()
Trace:   CFtpControlSocket::ResetOperation(2)
Trace:   CControlSocket::ResetOperation(2)
Error:   File transfer failed after transferring 474,480,640 bytes in 126 seconds
Trace:   CFtpControlSocket::FileTransfer()
Status:   Starting upload of G:\ANALYZED_DATA\Project_15903_FUID1025093-3\US-1584765\US-1584765.bam

Also, I noticed there is message abotu an insecure server every time I connect with Filezilla:

Status:   Connection established, waiting for welcome message...
Status:   Insecure server, it does not support FTP over TLS.
Status:   Server does not support non-ASCII characters.
Status:   Connected
Status:   Retrieving directory listing...
Status:   Calculating timezone offset of server...
Status:   Timezone offset of server is 0 seconds.
Status:   Directory listing of "/" successful

Not sure what that means.

In regards to CloudMan, I started the instance this week, with the most current release from last month. I don't know how to check the other information you mentioned though. Is this available directly through the instance online (from a Windows machine)? I managed to get admin access on my account, but I assume I have to access the account differently to try what you suggested. Sorry, I imagine this is straightforward, just not really sure what I'm doing...

Thanks again.

ADD COMMENT • link written 3.1 years ago by jsr2137 • 0

Please log in to add an answer.

Similar posts • Search »