Fresh Fruit

If you run virtual systems based on OpenStack, OpenNebula, or other cloud environments, you will notice that classic Linux systems like Debian, Ubuntu, and Red Hat are not actually made for use in cloud environments. In particular, the standard Linux variants are not very efficient with virtual machine maintenance.

The life cycle of a virtual system includes a variety of properties and events that are difficult to handle with the classical distributions. For instance, a minimal image for Ubuntu 4.14 might be as small as 300MB, but you can't actually do very much with a minimal image without having to install an assortment of additional packages. Moreover, all this additional software needs configuring. Management tools such as Puppet or Chef can help with configuring virtual machines, or you could rely on scripts at system startup time, but each VM needs all of this information all over again. Additionally, the number of active VMs can vary greatly in clouds, and Puppet and Chef are not particularly well adapted to frequent system fluctuation.

Also, what happens if a VM fails? Installing a cloud-based Pacemaker instance for virtual machines would be cumbersome and time consuming, and many clouds only compensate for failed VMs by way of orchestration.

CoreOS to the Rescue

The difficulties configuring a standard Linux distro for the cloud are actually more than enough to cause much wailing and gnashing of teeth, but don't panic! CoreOS  [1] aims to clear away the problems with classical Linux distributions in a cloud context. The CoreOS manufacturer, CoreOS Inc., released the first stable version in July.

To understand the role of CoreOS, you must understand the rise of the Docker container system [2], which is growing in popularity as a tool for supporting cloud environments. Docker lets you locate application-specific files and dependencies in a virtual container along with the application itself. Because all the files necessary for running the application are located in the container, the underlying host OS can be quite simple. CoreOS is essentially a framework for using Docker containers. All end-user-facing services then happen within Docker itself.

A container system like Docker lets admins move applications between multiple hosts easily: You can migrate user-created containers from one host to another or provide pre-configured containers to users as files on the Docker Image Store. A finished Docker container is basically autonomous: As long as the computer on which you want it to run has a Docker environment, it is totally happy.

The fact that CoreOS relies on Docker for applications causes some unexpected effects. The system does not have the traditional underpinnings; the CoreOS developers base their distribution on Chrome OS, rather than Debian or Ubuntu. Chrome OS is a reduced-to-the-max version of Linux, which Google uses on its own Chrome devices and which comes with basically nothing other than a web browser. (For example, Chrome OS does not have its own package manager on the lines of RPM or Debian's dpkg, which affects the way admins need to install updates for CoreOS.)

The minimal CoreOS system has very few specific parameters. As long as key values – such as the hostname and IP address – survive the reboot of a CoreOS system, updates are handled easily: You just need to shut down the old system and start a new system with a different and newer root disk – that's the whole update process.

Of course, minimal is a relative term. The CoreOS OpenStack image provided by the developers [3] weights in at 160MB, even as a Bzip2-compressed file (Figures 1-3). After unpacking, the image is 400MB, which is already more than Ubuntu's OpenStack image by more than 100MB.

Figure 1: A finished cluster of three VMs based on CoreOS looks like this in the OpenStack dashboard.

Figure 2: For OpenStack, CoreOS provides ready-to-use images of the alpha, beta, and stable channels.

Figure 3: The CoreOS Image for OpenStack is not exactly lightweight, but it does include everything you need for operating VMs.

However, the first impression is deceptive: CoreOS is significantly more than a mini-distribution and contains practically everything you need for operating services in the cloud. In addition to the Docker Container Manager (Figure 4) and built-in Docker support, CoreOS also includes a collection of useful tools to support its role as a cloud-based system. For instance, etcd [4] (Figure 5) and Fleet [5], which you will learn about later in this article, are tools built by in-house CoreOS developers to help solve specific problems associated with cloud-based systems.

Figure 4: Docker on Board: CoreOS comes with Docker 1.0.2 out the box; this means you can use all the services directly in Docker containers.

Figure 5: Using a separate configuration file, CoreOS VMs call the etcd discovery service after booting and also launch Fleet.

The CoreOS developers have improved the convenience factor in handling Docker images to the extent that you can now manage containers using a version control system such as Git. Moreover, it is easily possible to group all the files that are useful for running a specific application in a Docker container.

For instance, you could create a "traveling" Apache installation: Any host that can cope with Docker containers can also launch a container with a ready-to-run Apache installation. Deploying a full-featured Linux service is thus as easy as installing and launching an application in Mac OS X, where a single package contains all the necessary run-time components.

No Alternatives

The CoreOS root filesystem (/) is always mounted in read-only mode, so changes are impossible. Admins who rely on CoreOS have to make friends with the FastPatch system, a Ksplice-like feature that installs updates on a running system. Ultimately, using the FastPatch system is not so difficult, because the CoreOS developers left open a couple of useful back doors: In a typical update, the old root partition is not overwritten, but a second root is set up that contains the updated system. If something goes wrong with the update, the admin can bail out any time.

If you have a working root partition for CoreOS, a service contract lets you avoid downloading a complete second image for an update. The service package includes the CoreUpdate product, which is based on Google's Omaha project. CoreUpdate lets you manage updates across all nodes of the cluster.

etcd

The CoreOS developers created etcd to manage the configuration files in /etc. Of course, the term "file" is actually rather misleading in this context; CoreOS no longer stores its configurations in simple files (Figure 6) but uses etcd as a handy key-value store from which compatible services obtain configuration values.

Figure 6: Thanks to etcd, CoreOS has an amazingly tidy /etc directory.

The etcd instances on different VMs in a group can exchange the entire contents of the etcd key-value store, thus ensuring that all instances always have the current configuration.

Some clustering systems, such as those that use the Paxos protocol [6], have a coordinator – a dynamically selected node that is responsible for distributing configuration changes to all other nodes if the admin tweaks something in the cluster. CoreOS encounters an obstacle at this point: If an admin starts many CoreOS VMs in a cloud at the same time, the etcd instances do not know how to talk to one another (Figure 7).

Figure 7: The discovery service's etcd tokens only tell new cluster nodes how they can talk to the already existing machines.

The etcd project remedies this problem through the Discovery function: You can register your nodes at boot time with the service officially provided by CoreOS, or you can set up your own instance of the etcd discovery service, which you can easily operate locally. Immediately after starting, etcd connects with the specified address and thus discovers which other etcd instances exist. All other communication is then solely between the individual instances of etcd; the discovery service does not need to sniff the entire traffic.

For the CoreOS macrocosm, etcd is thus exactly the right solution. Admins of clusters already know that it is tedious and painstaking work to keep the content of /etc in sync across several cluster nodes; the problem is aggravated if cluster nodes constantly come and go – as is likely to be the case with CoreOS clusters comprising many instances. Anyone who hopes to take advantage of etcd outside of CoreOS will be disappointed. The tool has not yet seen much support in other projects.