Cloud computing with Amazon's Elastic Compute Cloud
Cloud computing systems like Amazon's Elastic Compute Cloud (EC2) save power and overhead by taking the peak out of your server load.
One would expect Amazon to guard their infrastructure jealously, but piece by piece, Amazon has been opening up their infrastructure so that the rest of us can get our hands dirty playing with file storage, virtual servers, and even physical deliveries on the same kind of ludicrous scale Amazon uses every day.
Amazon Web Services (AWS) makes these systems available over a web services framework so that everything from using more storage space, to creating virtual servers, to requesting physical deliveries happens over SOAP. Instead of filling in forms each time you want more, less, or a different infrastructure, your code can stay as is and AWS provides the necessary services as needed.
Each of the Amazon web services comes with tools developed by Amazon, and a growing number are developed by third parties. Increasingly, third parties are building new and complex services on top of these basic services – for example, hugely scalable databases and web indexing. Amazon's "Elastic Compute Cloud" (EC2) provides virtual servers charged at an hourly rate from US$ 0.10 an hour, running on Amazon's huge number of servers spread across their data centers. EC2 gives you computing in a "cloud."
The term cloud computing can mean many different things – from Software as a Service (SaaS) to highly integrable services – but it also means that you don't worry about the infrastructure. EC2 runs on the Xen virtualization layer, but you don't have to worry about this – you just request more virtual servers and they appear. Cloud computing changes the way you provision servers because it makes rapid scaling easier and cheaper at peak demand times. Instead of spending several thousand dollars on five machines that spend 90% of their lives doing nothing, you can use five EC2 instances only when you need them.
One of the biggest differences EC2 presents is the use of Amazon Machine Images (AMIs), which are like server configurations or perhaps bootable CDs. EC2 uses these AMIs when it creates a new virtual server. In this article, I describe how to create your own AMI.
To start with EC2, set up an AWS account , then go the the EC2 homepage , where you can get the various keys you'll need. The easiest way to control your EC2 instances is with ElasticFox, a plugin for Firefox. Install ElasticFox from the Amazon Web Services site  and have a look around. The first step is to set up a virtual machine. In the center of the window, you'll see a list of AMIs. To create a new instance, select the appropriate AMI and click the I/O button.
A good place to start is by selecting ec2-public-images/fedora-core4-apache-mysql with the AMI ID ami-25b6534c. The new instance will appear in the list at the bottom after a few moments. When it says "running," right-click on it and copy the Public DNS into a browser. Now you should see what looks like a normal website running from your EC2 instance.
A host of AMIs are publicly available for PHP, Rails, Java, specialized number crunching, and other uses. The beauty of using AMIs is that they are honed for a particular purpose, so when your EC2 instance is running, it doesn't need any unnecessary software. This setup is somewhat different from traditional hosting, in which the server tends to contain all the software to run all your apps.
Creating an AMI
Creating an AMI takes a while, which can make things tricky at first, but once you have cracked it, the steps are pretty easy. AMIs can contain anything from a single service to all your applications and databases, so all your EC2 instances will look just like you want them to look. For example, if you deploy lots of websites that are based on the same software, you can push the software into the AMI so that you only have to upload the site itself to the EC2 instance.
Once the working AMI is set up, you can use it to create as many instances as you like. To create an AMI, create a Linux image containing all the required files and settings, bundle the image, and upload it to EC2, then register the uploaded image.
Creating the Linux Image
The first step is to create the Linux image in a "loopback file," which is used to simulate a hard disk and avoids the need to create the operating system on a separate drive. The dd command copies raw data into a file of a specified size – in this case, 1GB:
dd if=/dev/zero of=myimage.fs count=1024 bs=1M
The dd program works in units of blocks; count is the number of blocks to be copied and bs is the size of the blocks to use. Running this produces a completely empty file, inside which you'll create the linux image.
Next, you need to create a filesystem with mke2fs, which adds an ext3 filesystem in the file you just created:
mke2fs -F -j myimage.fs
If you haven't created filesystems in loopback devices before, this step might seem a bit weird. Think of it like this: The file myimage.fs_ now contains a filesystem that can be mounted just like an external hard drive:
sudo mount -o loop myimage.fs /mnt
This step mounts the filesystem on /mnt. The -o loop options makes you mount the filesystem as a loopback, rather than as a real disk drive. The whole loopback device arrangement should begin to fall into place now. Look around /mnt. All you'll see so far is the usual ext3 lost+found directory. Now you can create files and directories – this filesystem will hold a small version of Linux to use on EC2.
To begin your basic OS, you can use debootstrap, a program that sets up a basic Debian system on a given filesystem. If you need to install this program, it is simply a matter of entering apt-get install debootstrap. If you're doing this on another flavor of Linux, the steps are similar, although you might need to set up the OS differently, but for the example here, run debootstrap:
sudo debootstrap --arch i386 edgy mnt
While debootstrap runs, you'll see various retrieving and validating messages as it gets the required files and installs them in the loopback filesystem. When the program finishes, have another look at the filesystem in /mnt. Things should look familiar. With the next two commands, you finish off this task and move inside the new Linux image as though it has been booted all along:
sudo cp /etc/apt/sources.list /mnt/etc/apt/sources.list sudo chroot /mnt mount -t proc none proc
Now that you're in the image, change the password:
What you have now is an empty Debian image, which has pretty limited use. With the use of Aptitude, update the image and install an SSH server:
aptitude update aptitude upgrade aptitude install openssh-server
If you'd like Apache, enter:
aptitude install apache2
Next, you should address the network settings. Note that you'll edit /etc/network/interfaces in the image – it isn't your local machine any more. Use an editor to put the following into /etc/network/interfaces
auto lo iface lo inet loopback auto eth0 iface eth0 inet dhcp
and then add the following to /etc/fstab:
/dev/sda2 /mnt ext3 defaults 1 2 /dev/sda3 swap swap defaults 0 0
Now you're done with the image, but you can play around and perhaps install some software (e.g., subversion, MySQL, or anything else you use frequently). When you're finished, enter:
exit sudo umount /mnt
Now you have Linux in a file. By this stage, you can imagine how EC2 will use this to create instances. Any programs, code, or files required by your applications can be added to the new Linux image at this stage. Anything you add now will be on every EC2 instance you create – for example, just copy files directly into the mounted image. If you use some standard CMS software, you might grab a copy for the image: svn co http://svn.server/my_project/trunk mnt/var/www/html/my_project
Now you can set up databases, symlinks, config files, or anything else as you normally would. When all the files are set up, your Linux image is ready.
Understanding the Keys
EC2 comes with a whole stack of keys, access ids, certificates, and the like:
- Access key id – a 20-character string that identifies all the requests you make to Amazon's Web Services.
- Secret Access Key – a 40-character string that validates the access key.
- X.509 certificate file – a public key and a private key.
- Private key file.
KeyPairs connect you to running instances without the need to send your password in the clear. In the examples here, you generate them through ElasticFox.
Buy this article as PDF
New flaw in an old encryption scheme leaves the experts scrambling to disable SSL 3
Lennart Poettering wants to change the way Linux developers talk to each other.
Enterprise giant frees itself from ink and home PCs (and visa versa).
Mozilla’s product think tank sinks silently into history.
TODO group will focus on open source tools in large-scale environments.
New tool will look like GParted but support a wider range of storage technologies.
New public key pinning feature will help prevent man-in-the-middle attacks.
Carnegie Mellon researchers say 3 million pages could fall down the phishing hole in the next year.
The US government rolls new best-practice rules for protecting SSH.