Sep 2, 2013

Get files from a finished EC2 Instance

Objective: Download our files from an EC2 instance on which denoising has finished.

Method: log into AWS and go to your instances.
Start the instance: Right-click > Start. (For long runs, we usually set an alarm to shutdown the instance when CPU use dropped to zero. So we have to start it back up again to download our data.)
Remove any alarms which may Stop your instance. In the column titled 'Alarm Status,' click on the link then the name of the alarm, then make sure 'Take this action' is unchecked.
Connect to the instance with ssh: Right-click > Connect (You can also use Terminal on Mac or Putty on Windows) Change user name to 'ubuntu' and select the path to the .qem certificate.
Remount Volumes. (If you need, you can check which volumes are attached.) Run sudo mount /dev/xvdf/ $HOME/data/. Then check that the data folder contains the files you need.
Connect to the instance with Cyberduck.
Download ALL the files!

You may consider compressing your files to save download time. In the directory, pick a file and type gzip YourFileName.fna/.
You can compress an entire folder with tar -czvf YourFolderName/.
Compression is particularly good with large repetitive files, so it's perfect for sequence data.

Results: The denoiser.log, centroids.fasta, singletons.fasta, denoiser_mapping.txt, denoised_clusters.txt, and denoised_seqs.fasta were downloaded from the two finished Instances.

3 hours over 5 hours

No comments:

Post a Comment