Amazon EC2: How to Setup an Instance and Bundle it

I know, that there are several documentations concerning Amazon’s Elastic Cloud Computing, though I’d like to describe some very first steps for using their infrastructure to build (Linux-) instances that fulfill your needs. Let’s not beat around the bush and start-over by setting up the very first instance.

The first thing needed is a client computer running any kind of the supported OSs by the Amazon API tools and of course the tools themselves. After you’ve installed those and configured all credentials the right way, we can create a new pair of SSH keys for our new project. Due to the location I’m currently in, I’ve chosen to use Amazon Instances in the western EU. Execute the following line on your command-line:

ec2-add-keypair –region eu-west-1 test-keypair

The result should look something like this:

[Deprecated] Xalan: org.apache.xml.res.XMLErrorResources_en_US
KEYPAIR test-keypair e1:1a:d1:a1:a1:1c:10:a1:b1:d1:cb:11:11:1a:11:11:f1:11:ae:fe
—–BEGIN RSA PRIVATE KEY—–
XXX
—–END RSA PRIVATE KEY—–

You can now copy the lines from BEGIN… until END… (including those two) into a file which you’ll be using as SSH-key for connecting to your instance. Don’t forget to chmod 600 it!

Next, let’s see what base-images for creating our instance we could you – first, provided by Amazon themselves:

ec2-describe-images –region eu-west-1 -o ‘amazon’

The list is contained of several different OSs and versions, although in our current project we can’t make use of any of those. Therefor, we’re now searching for a perfectly fitting, really good operating system:

ec2-describe-images –region eu-west-1 -a | grep -i debian

And yet, we receive another list with several different versions of the Debian Linux distribution. After we’ve picked the one we’d like to run, we should check what instances are currently up and running:

ec2-describe-instances –region eu-west-1

If you’re using Amazon’s EC2 for the first time, there shouldn’t be any items listed. We can now start our very first instance, by copying the instance’s identifier (in the second column, a string starting with ami-) and pasting it into our command:

ec2-run-instances –region eu-west-1 -k test-keypair -g ‘http/s’ -g ‘ssh’ ami-b8446fcc

In this command, we tell Amazon to start up a new instance that’s built on top of the ami-b8446fcc-image, using the key-pair we just created before and using some custom built firewall-rules named “http/s” (which allows us to connect to port 80 and 443) and “ssh” (port 22).

We need to wait a few seconds, until the instance comes up. We can use the describe-instances command from above to check the instance’s status:

ec2-describe-instances –region eu-west-1

As soon as it’s up and running, the “pending” column should have been replaced by a dynamically allocated hostname and the status “running”. Keep in mind, that this hostname/ip is dynamically allocated! If you want a fixed IP, you need to allocate and assign an Elastic IP – I’ll show you later how to do so.
If our firewall rules worked out, we can now connect using SSH:

ssh -i ~/Library/EC2/id_rsa-test-keypair root@ec2-11-111-11-111.eu-west-1.compute.amazonaws.com

By default, Amazon sets up a Small Instance, that provides around 10 GB of hard drive, an Dual-Core AMD Opteron 2218 with 2600 MHz and around 1.7 GB of RAM. Small instances, in comparison to the bigger ones, also still provide a real swap-partition which is limited to 895 MB. Here, you could run into problems when installing some Oracle for example, since the DB would like to have 1 GB or more Swap-space. If 895 MB should not be enough, the only way to enlarge it seems to be to use a file within your file-system for that. Either, by placing it somewhere into / or by using /mnt for that. /mnt is a special mount in Amazon instances which provides you 147 GB of additional volatile storage. Amazon doesn’t guarantee in any way the storage to be stable/solid or even backed up – and usually /mnt is only used for bundling your instance. You might now think “so, where could I place my data, if / is only providing me 10 GB of space and /mnt should not be used for storing sensitive data?” – well, there’s a third possibility named Elastic Block Storage. An EBS is being displayed to your instance as regular block-device (/dev/sdb, …) that’s format- and mountable. There, sensitive data could be stored, by bind-mounting the directories you’d like to have your data in. I’m not going to explain how this works, else I’ll never finish writing this brief documentation.

However, since we’re connected to the instance now, we can set it up the way we want it, with whatever software we need on it. I’ve assumed, that most people would set it up as regular LAMP-instance, therefor I’ve also added the firewall-rule for HTTP/HTTPS. After we finished installing the software we need, there’s one more package that should be plugged into the system: The Amazon AMI Tools. Simply wget them from http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.zip and unzip the package. Those tools make use of Ruby, so ensure having it installed on your instance. Also, you need to have the private key (pk-.pem) and the certificate (cert-.pem) you created the instance with somewhere within your instance’s filesystem, for later use.

Before we bundle up our system now, let’s come back to the topic we had before: The hostname/IP. Amazon allocates some dynamic address, unless you tell them to give you a fixed (called Elastic) IP. You can do that by simply executing:

ec2-allocate-address –region eu-west-1

… within your client’s command-line (where the Amazon API Tools have been installed – not on the instance!). As result you’ll get some IP address that has now being allocated by Amazon for you. The IP isn’t bound to any instance yet – it’s just allocated for you to be able to use it. ATTENTION: You pay for Elastic IPs as long as you do not assign them to an instance. Yes, that’s right. Amazon charges you for every allocated but unused IP hourly. By that, they want to prevent people “collecting” addresses, I guess. To assign the address you just received to your instance, simply run:

ec2-associate-address –region eu-west-1 -i i-11e11b1e 79.125.11.11

The i- is your actual instance’s ID, where the last, dot-separated number is the IP you’ve received. And yes, those are fake numbers – so don’t even try.

Now it could take a bit for Amazon to reconfigure the instance, but as soon as it finished, you should be able to re-connect to the instance using the IP you just assigned to it.

As last task for today, let’s bundle up the image the way we have it now. Bundling an image allows you more than just eating up your credit card’s limit by dumping your S3 buckets. On the one hand, with bundles you can recover machines that crashed or lost data within a few blinks and on the other, you can created new instances out of a bundle (talking about “scalability”).

For bundling, we use the AMI tools we installed. First of all, let’s create a directory for the bundle:

mkdir /mnt/myimage

After that, run the bundle-vol-tool:

ec2-bundle-vol -k pk-.pem -c cert-.pem -s -u -d /mnt/myimage/

This command takes several parameters for the private key, the certificate, the size of the resulting bundle in MB and your User-ID (without dashes). The User-ID can be found within your Account Information on Amazon’s EC2 site. The command should ask you, what architecture you’d like to bundle the system for – i386 should work out perfectly for what we’re doing. The following procedure could take some time, since the tool collections every peace of the system and builds a bundle into the directory we specified. As soon as the tool finished, we can upload our bundle to our S3-bucket:

ec2-upload-bundle –location EU -b -m /mnt/myimage/image.manifest.xml -a -s

Again, we need to specify some credentials (our access-key and the secret-key) for the upload to work. Also, we need to pick a globally unique bucket-name for uploading the bundle, what shouldn’t be that hard as long as you don’t try stuff like “linux” or other common words. The bundle will then be uploaded to your (private) bucket, so you have it for later use.

From within the web-interface you could now simply create new instances out of the uploaded bundle, without even knowing how the actual system was set up or having the Amazon API tools installed on your client.

Cool stuff, enjoy!