Question: FTP upload issue at http://usegalaxy.org - FAQs, help, troubleshooting
1
gravatar for auradunbine
15 months ago by
auradunbine20
auradunbine20 wrote:

Hi! galaxy masters:

I have tried FTP upload about 2 month(July~Aug), and it is still not working. I sure my Filezilla is default setting(passive mode) and I also tried WINSCP, another FTP software, as follow the tutorial page. Both are fail to connect the galaxy server!! I use galaxy for 7 years, but I cannot do anything during these 2 months! Please help me!! Thanks a lot!! Below is the error message:

Status: Resolving address of usegalaxy.org
Status: Connecting to 129.114.60.56:21...
Status: Connection established, waiting for welcome message...
Status: Initializing TLS...
Status: Verifying certificate...
Status: TLS connection established.
Status: Logged in
Status: Retrieving directory listing...
Command:    PWD
Response:   257 "/" is the current directory
Command:    TYPE I
Response:   200 Type set to I
Command:    PASV
Response:   227 Entering Passive Mode (129,114,60,56,118,199).
Command:    MLSD
Error:  Connection timed out after 20 seconds of inactivity
Error:  Failed to retrieve directory listing
Status: Disconnected from server
Status: Resolving address of usegalaxy.org
Status: Connecting to 129.114.60.56:21...
Status: Connection established, waiting for welcome message...
Status: Initializing TLS...
Status: Verifying certificate...
Status: TLS connection established.
Status: Logged in
Status: Retrieving directory listing...
Command:    PWD
Response:   257 "/" is the current directory
Command:    TYPE I
Response:   200 Type set to I
Command:    PASV
Response:   227 Entering Passive Mode (129,114,60,56,118,221).
Command:    MLSD
Error:  Connection timed out after 20 seconds of inactivity
Error:  Failed to retrieve directory listing
upload ftp galaxy • 2.0k views
ADD COMMENTlink modified 14 months ago by Jennifer Hillman Jackson25k • written 15 months ago by auradunbine20

Hello - Some networks have difficulty connecting after the recent security changes to the FTP service (firewall issues). A solution is in progress and we expect it to be ready to test very soon. More updates then. Jen, Galaxy team

ADD REPLYlink written 15 months ago by Jennifer Hillman Jackson25k
1

Hi, Jen: Your assistance in this matter is deeply appreciated!! I'm looking forward the solutions!!

ADD REPLYlink written 15 months ago by auradunbine20

Update: The full correction is still in progress as a priority fix.

For an alternate description of how FTP upload is currently failing plus a workaround for loading large data (Upload via URL) please see: https://biostar.usegalaxy.org/p/24648/

ADD REPLYlink written 15 months ago by Jennifer Hillman Jackson25k

Thank you for all your assistance and I really appreciate your help in resolving the problem! I have tried both passive and active FTP with "require explicit FTP over TLS", but it's still not working. I'm sure my ftp configuration is following as the instruction. Below is the error message:

Status: Resolving address of usegalaxy.org
Status: Connecting to 129.114.60.60:21...
Status: Connection established, waiting for welcome message...
Status: Initializing TLS...
Status: Verifying certificate...
Status: TLS connection established.
Status: Logged in
Status: Retrieving directory listing...
Command:    PWD
Response:   257 "/" is the current directory
Command:    TYPE I
Response:   200 Type set to I
Command:    PASV
Response:   227 Entering Passive Mode (129,114,60,60,119,199).
Command:    MLSD
Error:  Connection timed out after 20 seconds of inactivity
Error:  Failed to retrieve directory listing
Status: Disconnected from server
Status: Resolving address of usegalaxy.org
Status: Connecting to 129.114.60.60:21...
Status: Connection established, waiting for welcome message...
Status: Initializing TLS...
Status: Verifying certificate...
Status: TLS connection established.
Status: Logged in
Status: Retrieving directory listing...
Command:    PWD
Response:   257 "/" is the current directory
Command:    TYPE I
Response:   200 Type set to I
Command:    PASV
Response:   227 Entering Passive Mode (129,114,60,60,118,138).
Command:    MLSD
Error:  Connection timed out after 20 seconds of inactivity
Error:  Failed to retrieve directory listing
ADD REPLYlink written 14 months ago by auradunbine20

This is the error message that is produced when an Implicit FTPS connection is made. Only Explicit FTPS is working at the moment. If you are using Filezilla, follow these instructions and connect with File > Site Manager > The site you set up for usegalaxy.org. Try passive and active mode as described in yesterday's update. If using another client, the setup can vary but should be easily found with a google. Or, consider asking a local admin for help.

The instructions in the Hub for using default passive mode with FTP/Filezilla are not valid at the moment. Avoid "Quick Connect".

ADD REPLYlink modified 14 months ago • written 14 months ago by Jennifer Hillman Jackson25k
2
gravatar for Jennifer Hillman Jackson
14 months ago by
United States
Jennifer Hillman Jackson25k wrote:

Update 2017-09-05 1:15 EST

Our admin Nate has made some changes but they are not complete yet. Use the first solution for now. We won't close the ticket out until the new configuration changes are fully in place.

Explicit FTPS: Working with either active or passive FTP. Which to use depends on whether the site allows active or not - just try one and then the other. If you are not sure how to configure your client for Explicit FTPS, see the Galaxy hub here or follow these general Filezilla instructions.

Implicit FTPS: NOT working, we are in the process of setting this up for people whose firewalls prevent the use of Explicit FTPS.

Thanks everyone for reporting the problems and for your patience! Jen

ADD COMMENTlink modified 14 months ago by Nate Coraor3.2k • written 14 months ago by Jennifer Hillman Jackson25k
0
gravatar for Jennifer Hillman Jackson
14 months ago by
United States
Jennifer Hillman Jackson25k wrote:

Update 2017-09-07

Both Explicit and Implicit FTPS are now functional.

Thanks, Jen & the Galaxy team

ADD COMMENTlink written 14 months ago by Jennifer Hillman Jackson25k

Hi Jen:

It seems that the new setting is still not working to us! I also tried to connect the galaxy server by some public wifi (without any firewall setting), it is not working, either. So I sure that the connection problem may not caused by the local firewall setting. Below is the latest error message:

Status: Resolving address of usegalaxy.org
Status: Connecting to 129.114.60.56:21...
Status: Connection established, waiting for welcome message...
Status: Initializing TLS...
Status: Verifying certificate...
Status: TLS connection established.
Status: Logged in
Status: Retrieving directory listing...
Command:    PWD
Response:   257 "/" is the current directory
Command:    TYPE I
Response:   200 Type set to I
Command:    PASV
Response:   227 Entering Passive Mode (129,114,60,56,117,157).
Command:    MLSD
Error:  Connection timed out after 20 seconds of inactivity
Error:  Failed to retrieve directory listing
Status: Disconnected from server
Status: Resolving address of usegalaxy.org
Status: Connecting to 129.114.60.56:21...
Status: Connection established, waiting for welcome message...
Status: Initializing TLS...
Status: Verifying certificate...
Status: TLS connection established.
Status: Logged in
Status: Retrieving directory listing...
Command:    PWD
Response:   257 "/" is the current directory
Command:    TYPE I
Response:   200 Type set to I
Command:    PASV
Response:   227 Entering Passive Mode (129,114,60,56,117,163).
Command:    MLSD
Error:  Connection timed out after 20 seconds of inactivity
Error:  Failed to retrieve directory listing
ADD REPLYlink written 14 months ago by auradunbine20
1

It looks like this was an explicit FTPS connection, could you try implicit FTPS and see if this has any effect for you? You can verify whether it's making an implicit connection by observing the port number, which should be 990, e.g.:

Status: Connecting to 129.114.60.56:990
ADD REPLYlink written 14 months ago by Nate Coraor3.2k

I tried it, but...........

Error message:

Status: Resolving address of usegalaxy.org Status: Connecting to 129.114.60.56:990... Error: Connection timed out after 20 seconds of inactivity Error: Could not connect to server Status: Waiting to retry... Status: Resolving address of usegalaxy.org Status: Connecting to 129.114.60.56:990... Error: Connection timed out after 20 seconds of inactivity Error: Could not connect to server Status: Disconnected from server Status: Resolving address of usegalaxy.org Status: Connecting to 129.114.60.56:990... Error: Connection timed out after 20 seconds of inactivity Error: Could not connect to server Status: Waiting to retry... Status: Resolving address of usegalaxy.org Status: Connecting to 129.114.60.56:990... Error: Connection timed out after 20 seconds of inactivity Error: Could not connect to server

ADD REPLYlink written 14 months ago by auradunbine20
1

Is the Windows firewall enabled on your system? Can you temporarily disable it while testing this connection?

ADD REPLYlink written 14 months ago by Nate Coraor3.2k

I just tested connections to usegalaxy.org, MacOS and Filezilla, and the FTP functionality is working as expected.

FAQ: https://galaxyproject.org/ftp-upload/

  • FTP, Filezilla > Settings, set at passive mode (saved), using "Quick connect" (implicit) form setting screenshot
  • FTP, Filezilla > Settings, set at active mode (saved), port 990 specified, using "Quick connect". Passive could also be set for this connection type. (implicit) form setting screenshot
  • FTP, File > Site Manager set up for usegalaxy.org as a site, using "Connect" on that form (explicit) form setting screenshots (there are two)

If these do not work, what is your OS? Do you want to share a screenshot of the connection type that matches one of the above and is not working?

ADD REPLYlink modified 14 months ago • written 14 months ago by Jennifer Hillman Jackson25k

It is working now!! Both explicit and implicit are successfully connected!!
Many thanks!!!!!

Status: Resolving address of usegalaxy.org
Status: Connecting to 129.114.60.60:21...
Status: Connection established, waiting for welcome message...
Response:   220-usegalaxy.org FTP
Response:    
Response:    This FTP server allows users to upload large data to the Galaxy server at:
Response:    
Response:      https://usegalaxy.org/
Response:    
Response:    Your login uses the same credentials as your Galaxy login - email address and
Response:    password.
Response:    
Response:    Once files are successfully uploaded, the list of files available for importing
Response:    in to a history will be displayed in the regular Upload tool.  For assistance,
Response:    please contact:
Response:    
Response:      galaxy-bugs@galaxyproject.org
Response:    
Response:    Files uploaded via FTP and not imported to a history within three days will be
Response:    deleted.
Response:   220 ProFTPD 1.3.5d Server usegalaxy.org FTP (Explicit SSL/TLS on port 21)) [::ffff:129.114.60.60]
Command:    AUTH TLS
Response:   234 AUTH TLS successful
Status: Initializing TLS...
Status: Verifying certificate...
Command:    USER auradunbine@gmail.com
Status: TLS/SSL connection established.
Response:   331 Password required for auradunbine@gmail.com
Command:    PASS ********
Response:   230 User auradunbine@gmail.com logged in
Command:    SYST
Response:   215 UNIX Type: L8
Command:    FEAT
Response:   211-Features:
Response:    EPRT
Response:    EPSV
Response:    MDTM
Response:    SSCN
Response:    TVFS
Response:    MFMT
Response:    SIZE
Response:    PROT
Response:    LANG bg-BG;en-US;es-ES;fr-FR;it-IT;ja-JP;ko-KR;ru-RU;zh-CN;zh-TW
Response:    CCC
Response:    PBSZ
Response:    AUTH TLS
Response:    MFF modify;UNIX.group;UNIX.mode;
Response:    REST STREAM
Response:    MLST modify*;perm*;size*;type*;unique*;UNIX.group*;UNIX.mode*;UNIX.owner*;
Response:    UTF8
Response:   211 End
Command:    OPTS UTF8 ON
Response:   200 UTF8 set to on
Command:    PBSZ 0
Response:   200 PBSZ 0 successful
Command:    PROT P
Response:   200 Protection set to Private
Status: Connected
Status: Retrieving directory listing...
Command:    PWD
Response:   257 "/" is the current directory
Command:    TYPE I
Response:   200 Type set to I
Command:    PASV
Response:   227 Entering Passive Mode (129,114,60,60,119,122).
Command:    MLSD
Response:   150 Opening BINARY mode data connection for MLSD
Response:   226 Transfer complete
Status: Directory listing successful
ADD REPLYlink written 14 months ago by auradunbine20

Great news, so glad! :)

ADD REPLYlink written 14 months ago by Jennifer Hillman Jackson25k

But I found there is still a problem for some computers, unavailable connection seems not due to firewall or OS. I don't know why....? (firewall closed, OS is Win10 or Win7)

Status: Resolving address of usegalaxy.org Status: Connecting to 129.114.60.60:21... Status: Connection established, waiting for welcome message... Status: Initializing TLS... Status: Verifying certificate... Status: TLS connection established. Status: Logged in Status: Retrieving directory listing... Command: PWD Response: 257 "/" is the current directory Command: TYPE I Response: 200 Type set to I Command: PASV Response: 227 Entering Passive Mode (129,114,60,60,117,85). Command: MLSD Error: Connection timed out after 20 seconds of inactivity Error: Failed to retrieve directory listing Status: Disconnected from server Status: Resolving address of usegalaxy.org Status: Connecting to 129.114.60.60:21... Status: Connection established, waiting for welcome message... Status: Initializing TLS... Status: Verifying certificate... Status: TLS connection established. Status: Logged in Status: Retrieving directory listing... Command: PWD Response: 257 "/" is the current directory Command: TYPE I Response: 200 Type set to I Command: PASV Response: 227 Entering Passive Mode (129,114,60,60,119,57). Command: MLSD Error: Connection timed out after 20 seconds of inactivity Error: Failed to retrieve directory listing

ADD REPLYlink written 14 months ago by auradunbine20

Just provided some "unavailable" connected information. OS: Win 10 (firewall closed)

Setting: FTP, File > Site Manager set up for usegalaxy.org as a site, using "Connect" on that form (explicit)

Status: Resolving address of usegalaxy.org Status: Connecting to 129.114.60.60:21... Status: Connection established, waiting for welcome message... Status: Initializing TLS... Status: Verifying certificate... Status: TLS connection established. Status: Logged in Status: Retrieving directory listing... Command: PWD Response: 257 "/" is the current directory Command: TYPE I Response: 200 Type set to I Command: PASV Response: 227 Entering Passive Mode (129,114,60,60,118,168). Command: MLSD

Error: The data connection could not be established: ETIMEDOUT - Connection attempt timed out

FTP, Filezilla > Settings, set at active mode (saved), port 990 specified, using "Quick connect". Passive could also be set for this connection type. (implicit)

Status: Resolving address of usegalaxy.org Status: Connecting to 129.114.60.56:990... Status: Connection attempt failed with "ETIMEDOUT - Connection attempt timed out", trying next address. Status: Connecting to 129.114.60.60:990... Status: Connection attempt failed with "ETIMEDOUT - Connection attempt timed out". Error: Could not connect to server Status: Waiting to retry... Status: Resolving address of usegalaxy.org Status: Connecting to 129.114.60.60:990... Status: Connection attempt failed with "ETIMEDOUT - Connection attempt timed out", trying next address. Status: Connecting to 129.114.60.56:990... Status: Connection attempt failed with "ETIMEDOUT - Connection attempt timed out". Error: Could not connect to server

ADD REPLYlink modified 14 months ago • written 14 months ago by auradunbine20

Thanks for the extra info about the problem. We are reviewing and will get back soon, Jen

ADD REPLYlink written 14 months ago by Jennifer Hillman Jackson25k

Is there a network administrator at your site who you can contact to help debug this issue from your side? I'm unable to reproduce this locally and it seems like something - either on the host or on that particular network - is blocking connections.

In the case of explicit FTPS, it succeeds in connecting on the control channel (port 21), but attempts to switch to passive mode - the PASV command. This instructs the FTP server on our side to create a random port (in our case, this will be between 30000 and 31000) for your client to connect to for data transfers. FileZilla attempts to get a directory listing from the data port with MLSD but times out when attempting to connect to the data port.

There could be multiple reasons for this, for example, there was a case we debugged a few weeks ago where the version of the FTP client (lftp) was too old to support TLS session reuse, which we require. A client upgrade fixed this. However, if you are using a recent version of FileZilla, this should not be an issue.

Another reason is that some firewalls inspect traffic to (remote) port 21 and watch for the response to the PASV command so they can allow outbound traffic from the client to the data port returned by PASV. Since the connection becomes encrypted after issuing AUTH TLS, the firewall cannot do this and thus connections to the data port fail.

As for the implicit FTPS connection, something is blocking your connection to the FTPS control channel (port 990). Some firewalls may only allow outbound connections to a certain list of ports, which may not include 990.

ADD REPLYlink written 14 months ago by Nate Coraor3.2k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 169 users visited in the last hour