Question: Galaxy Cloudman Login And Aws Instance
0
gravatar for Sun, Wenping [USA]
5.9 years ago by
Sun, Wenping [USA]230 wrote:
Dear galaxy members, I have question on galaxy running on cloud with the issue of authorization matter. I have launched aws galaxy cloudman instances to work with, shutting down the cluster afterwards. Then the 2nd time here are the steps that I did- 1. Launch the galaxy at cloudlaunch site 2. Register and then log in 3. Worked with data uploading and processing (the login information can see from pull-down menu), as well as the data history [cid:image001.jpg@01CDDEAF.44EBA770] 4. Go to aws console to check, find the instance is running 5. Copy the dns site followed by /cloud to check the cluster nodes, click "access galaxy" button there 6. On the galaxy window choose user login 7. And here is the window I have - I cannot login and no history data there. [cid:image002.jpg@01CDDEAF.44EBA770] My question: it seems that the two galaxy access is not the same. I rather to use the site from cloudlaunch site if it is preferred. However, I still have to go to the instance DNS site for shutting down the instance/cluster to avoid aws unnecessary charge. And, if I need to add cluster nodes, I still need to go to the DNS site. What is the appropriate procedure I need to follow to retain the data history and log in between two accesses, especially while shutting down and restart? I really appreciate your inputs and helps! Kind regards, Kathryn
galaxy • 1.5k views
ADD COMMENTlink modified 5.9 years ago by Dannon Baker3.7k • written 5.9 years ago by Sun, Wenping [USA]230
0
gravatar for Dannon Baker
5.9 years ago by
Dannon Baker3.7k
United States
Dannon Baker3.7k wrote:
It looks like you've worked on both the public galaxy site we provide main.g2.bx.psu.edu) and your own cloud instance. Data on the public instance won't be automatically migrated to the cloud instance and vice versa. All work done on the cloud site will remain there and persist after termination and restart of that cloud cluster. Work on main.g2.bx.psu.edu is completely separate and will remain there. -Dannon
ADD COMMENTlink written 5.9 years ago by Dannon Baker3.7k
Thanks and my question is-- How to shutting down the cloud if working on main.g2.bx.psu.edu? Thanks, Kathryn To: Sun, Wenping [USA] Cc: galaxy-user@bx.psu.edu Subject: [External] Re: [galaxy-user] galaxy cloudman login and aws instance It looks like you've worked on both the public galaxy site we provide main.g2.bx.psu.edu) and your own cloud instance. Data on the public instance won't be automatically migrated to the cloud instance and vice versa. All work done on the cloud site will remain there and persist after termination and restart of that cloud cluster. Work on main.g2.bx.psu.edu is completely separate and will remain there. -Dannon
ADD REPLYlink written 5.9 years ago by Sun, Wenping [USA]230
Ahh, ok. If you want to work exclusively on main and don't have the need for a cloud instance, just terminate (and permanently delete, if desired) cloud instances through the cloudman interface and keep using main.g2.bx.psu.edu. -Dannon
ADD REPLYlink written 5.9 years ago by Dannon Baker3.7k
Thanks. When I use main.g2.bx.psu.edu to access the galaxy, I did see the ec2 instance initiated from my aws console. Why this happen and what this instance doing? Meantime, what is the data security on main.g2.bx.psu.edu? If I use aws instance for the data works, do I have to terminate the ec2 instances/cluster and then launch the instance every time? Thank you, Kathryn To: Sun, Wenping [USA] Cc: galaxy-user@bx.psu.edu Subject: Re: [External] Re: [galaxy-user] galaxy cloudman login and aws instance Ahh, ok. If you want to work exclusively on main and don't have the need for a cloud instance, just terminate (and permanently delete, if desired) cloud instances through the cloudman interface and keep using main.g2.bx.psu.edu. -Dannon
ADD REPLYlink written 5.9 years ago by Sun, Wenping [USA]230
The ec2 instance is your personal instance that you have complete control over for customization, you don't have a personal quota, nobody else will be waiting in line to run jobs, etc. The server at main.g2.bx.psu.edu is our public galaxy instance that we provide and administer, but because it's a public service we do have quotas and sometimes processing queues, and so on. Regarding data security, here's the relevant snippet from the public galaxy welcome page: """This is a free, public, internet accessible resource. Data transfer and data storage are not encrypted. If there are restrictions on the way your research data can be stored and used, please consult your local institutional review board or the project PI before uploading it to any public site, including this Galaxy server. If you have protected data, large data storage requirements, or short deadlines you are encouraged to setup your own local Galaxy instance or run Galaxy on the cloud.""" Lastly, for aws, you don't *have* to terminate the ec2 instance and start it back up when you want to use it, but remember that for every hour of use Amazon does bill you. If you aren't running any jobs on the instance, it would make sense to terminate it and start it back up when you want it again.
ADD REPLYlink written 5.9 years ago by Dannon Baker3.7k
Thank you very much for the information. For the last part of using ec2 instances at aws, is there an option that I can control galaxy's operation at aws site? This way I can start/stop the instance when I need. I am not willing to terminate the instance since everything will be lost. Meantime, I am wondering the data structure on ec2 galaxy instance. Where is the location for the data uploaded from the galaxy gui? If I ssh the Ubuntu, where is the data directory for holding the work? Thank you, Kathryn To: Sun, Wenping [USA] Cc: galaxy-user@bx.psu.edu Subject: Re: [External] Re: [galaxy-user] galaxy cloudman login and aws instance The ec2 instance is your personal instance that you have complete control over for customization, you don't have a personal quota, nobody else will be waiting in line to run jobs, etc. The server at main.g2.bx.psu.edu is our public galaxy instance that we provide and administer, but because it's a public service we do have quotas and sometimes processing queues, and so on. Regarding data security, here's the relevant snippet from the public galaxy welcome page: """This is a free, public, internet accessible resource. Data transfer and data storage are not encrypted. If there are restrictions on the way your research data can be stored and used, please consult your local institutional review board or the project PI before uploading it to any public site, including this Galaxy server. If you have protected data, large data storage requirements, or short deadlines you are encouraged to setup your own local Galaxy instance or run Galaxy on the cloud.""" Lastly, for aws, you don't *have* to terminate the ec2 instance and start it back up when you want to use it, but remember that for every hour of use Amazon does bill you. If you aren't running any jobs on the instance, it would make sense to terminate it and start it back up when you want it again.
ADD REPLYlink written 5.9 years ago by Sun, Wenping [USA]230
We'd recommend you not ever stop/start through the aws console -- this is basically guaranteed to cause problems with your instance. Always use the cloudman interface (on your ec2 instance -- http://<ec2_instance>/cloud) to terminate, and then use usegalaxy.org/cloudlaunch to restart it. Everything will be in /mnt/galaxyData/files (which is symlinked to /mnt/galaxyTools/galaxy-central/database/files). Note that the filenames will be by id (like files/000/dataset_001.dat is dataset id 1, etc), and not particularly accessible in this format without a little extra legwork.
ADD REPLYlink written 5.9 years ago by Dannon Baker3.7k
The problem is that if use usegalaxy.org/cloudlaunch to restart galaxy, it will launch to main.g2.bx.psu.edu, where I have no control on the data security and running queues. Meanwhile, the correspondingly initiated ec2 instance is brand new and has no previous work records to track with. In this is the case, it would be great that if galaxy cloudman can be designed more aws instance friendly in the future. Thank you, Kathryn To: Sun, Wenping [USA] Cc: galaxy-user@bx.psu.edu Subject: Re: [External] Re: [galaxy-user] galaxy cloudman login and aws instance We'd recommend you not ever stop/start through the aws console -- this is basically guaranteed to cause problems with your instance. Always use the cloudman interface (on your ec2 instance -- http://<ec2_instance>/cloud) to terminate, and then use usegalaxy.org/cloudlaunch to restart it. Everything will be in /mnt/galaxyData/files (which is symlinked to /mnt/galaxyTools/galaxy-central/database/files). Note that the filenames will be by id (like files/000/dataset_001.dat is dataset id 1, etc), and not particularly accessible in this format without a little extra legwork.
ADD REPLYlink written 5.9 years ago by Sun, Wenping [USA]230
This isn't quite correct, but I think I see the confusion. When you launch an instance using usegalaxy.org/cloudlaunch it actually does start an instance in the cloud. A link to your new instance should be presented to you on a page saying something like "While it may take a few moments to boot, you will be able to access the cloud control panel at <your instance="" id="">". If you don't click on that link to your new instance but instead just click "Analyze Data" in the masthead of that page you're going to go back to usegalaxy.org, which is the same server as main.g2.bx.psu.edu. So, for your use in the cloud you want to launch using usegalaxy.org/cloudlaunch but make sure you're actually *using* the instance this starts for you, and *not* galaxyproject.orgmain.g2.bx.psu.edu), even though we use that server to launch your personal cloud instance.
ADD REPLYlink written 5.9 years ago by Dannon Baker3.7k
Please log in to add an answer.

Help
Access

Use of this site constitutes acceptance of our User Agreement and Privacy Policy.
Powered by Biostar version 16.09
Traffic: 167 users visited in the last hour