Docker Talk – Docker swarms the Kubernetes community

Docker Talk – Docker swarms the Kubernetes community

© Images courtesy of Swapnil Bhartiya

© Images courtesy of Swapnil Bhartiya

Article from Issue 207/2018

At this year's DockerCon Europe, Docker announced that it is officially supporting the Google-sponsored Kubernetes ochestration engine – an unexpected development that surprised many observers.

At DockerCon EU in Copenhagen (Figure 1), Solomon Hykes, the founder of Docker, announced the availability of Kubernetes as an additional orchestrator along with Swarm, the company's own orchestrator.

Figure 1: The Docker faithful gathered at Copenhagen's Bella Center for this year's DockerCon EU.

The news took many by surprise. Why would Docker support an orchestration platform that competes with its own in-house tool? A closer look at this question reveals some insights into the direction of Docker as a company – and it also offers a fascinating window into the inner-workings of open source communities.

A Little History

When Docker hit version 1.0 in 2014, it became an instant hit with the developer community. Developers were able to develop, test, and deploy the same code in production that they ran on their laptops. CIOs and IT professionals were able to migrate applications from virtual machines to containers. New features and functionalities could be packaged inside easy-to-use containers and deployed immediately. They were able to innovate faster. They were able to stay ahead of their competitors. What was there to not to like in Docker containers? Businesses started to flock to containers.

But with big adoption comes big responsibilities.

This remarkable growth meant an ever-increasing user base and partner ecosystem. It meant more competitors. Docker faced growing pressure – from partners, customers, and competitors – to loosen its grip. Companies like CoreOS were not happy with things like run time and image format. CoreOS, in fact, came out with its own competing run time, called rkt, to address those concerns. There were talks of forking the Docker Engine, so companies could fix these problems.

Docker has been a champion of open source since the beginning, and it has been open sourcing pieces of Docker Engine since its early days. But the explosive adoption of Docker containers demanded more from the company. Container stakeholders also wanted standardization around two core components of Docker containers: the container run time and container image format. Under the umbrella of the Linux Foundation, a new Collaborative Project, named Open Container Initiative (OCI), was formed. The primary goal of OCI was to finalize the container run-time specification and container image format specification.

Over time, Docker unbundled more components from the Docker engine and released them as independent open source projects. These components included runc, containerd, SwarmKit, libnetwork, Notary, HyperKit, VPNKit, DataKit, and InfraKit.

Breaking the monolith of the Docker engine into smaller pieces made Docker modular and scalable, but it also posed new challenges for the company. Now Docker engineers had to gather these components and assemble them to build Docker.

Players were coming out with new use cases, expecting Docker to support their platforms. Docker ended up creating Docker for Windows, Docker for Mac OS, Docker for Cloud, Docker for different Linux distributions, Docker for Azure, Docker for AWS, and more. Every time Docker set out to build a specialized edition of Docker using open sourced components, its own production model started to fail. It experienced redundancies and wasted efforts.

To solve this problem, Docker created an assembly line within the company that allowed different teams to collaborate with each other, and with the ecosystem, to use these open sourced components. This approach cut down redundancy and made the entire process more efficient.

Docker decided to release this assembly line – the processes, tooling, components, and framework – as open source so the entire ecosystem could benefit from it. Docker called it the Moby Project. Moby is a framework that allows anyone to assemble their own container system using different components that were open sourced by Docker.

There was one more missing piece.

A Linux subsystem is a prerequisite for Docker, and it's not available in every environment that Docker supports. Docker had already done a lot of work with companies like Microsoft to add a subsystem so that Docker could run on different platforms. Docker released that work as open source and called it LinuxKit. It's a container-native toolkit that enables developers to create their own containerized operating systems that are secure, modular, and portable.

With LinuxKit and the Moby Project, Docker had achieved a Red Hat-like nirvana. The Moby Project was to Docker what Fedora was to Red Hat Enterprise Linux (RHEL). Moby was hosted on GitHub and became the upstream for Docker Editions, the same way Fedora is upstream for RHEL.

Docker Swarms Around Kubernetes

Docker was aware of Kubernetes' growing popularity as a great orchestration tool. Many Docker customers were toying with the idea of using Kubernetes to manage containers.

Why not give them what they want?

I saw many surprised looks at DockerCon from Docker users who criticized the company for being secretive about this Kubernetes support. "That's not how open source works; you don't keep things secret. Open source means you start talking about it from day zero," critics whispered.

But according to Patrick Chanezon, a member of the technical staff at Docker, "Moby is where all the action happens. All of the Kubernetes work was done in Moby; it's been going on for months. Some people may be surprised because they don't spend all of their time in our GitHub repos. What we announced here is the productization of that work."

Hykes told me that open source is not one-way traffic. If you want to do it right, you need to get engaged at the upstream level. That's exactly what Docker did when they decided to embrace Kubernetes. And Hykes stressed that you can't takes bits and pieces from an upstream project, use it in your product, and then try to keep up with the upstream. "When we realized that we had to support more than one orchestrator, we decided to do it the Docker way. It's a full-on upstream collaboration. We have been working on it for a while, and there is a very high level of engagement. It's a two-way upstream engagement with the Kubernetes community. I guess the Kubernetes community will get as much out of this engagement, in terms of code and operational hardening, as the rest of the Docker community."

Commoditization of Orchestrators

Despite all this excitement around orchestrators like Kubernetes and Swarm, Hykes said that eventually orchestrators will become commodities. When Docker was announced four years ago, everyone was excited about the container run time. Today we have a set of specifications from OCI for container run time and image format. It's commoditized.

"I think the same thing will happen with orchestration. Right now we have a lot of innovation and differentiation, but over a long period, there is going to be commoditization and people are going to converge to one of the two systems," said Hykes.

Two competing technologies being used by competitors is a norm in any industry. Eventually, one of the two technologies will fade out. What's unique about this situation is that Docker is embracing two competing technologies in its own product. Which will be the last orchestrator running?

During DockerCon, Docker engineers gave a demo displaying both Swarm and Kubernetes in the Universal Control Plane of Docker, showing how you can use either of the two to orchestrate your containers. A potential Docker customer may wonder which one to use – Swarm or Kubernetes? The short answer is whichever suits your needs. Swarm is an in-house project of Docker, whereas Kubernetes is a Google-led, community driven project hosted by CNCF. The projects have different goals and a different release cadence.

The broad scope of and developer base of the Kubernetes project raises the question of how Swarm could ever keep up. But according to Docker CEO Steve King, Swarm is not going away, assuring users that Docker will support the project for many years to come.

Docker needs to continue to invest in Swarm because it is the integrated orchestrator for the Docker Enterprise Edition. According to Chanezon, "The people who work on the open source project behind Swarm, called SwarmKit, have a very narrow focus: how to provide orchestration in the Docker platform. Because of that narrow focus, they can move faster than Kubernetes, where you have to work with all the companies who have invested in it to solve their own use cases. There is only one use case for us."

Swarm is actually a bit ahead of Kubernetes in terms of more advanced features needed by Docker customers. It's the Kubernetes community that then tries to catch up with those features. Chanezon gave an example of cryptographic node identity that was first introduced in Swarm and later implemented by Kubernetes. "Implementing both Swarm and Kubernetes on the Docker platform will allow us to innovate at the pace we want for our enterprise customers. When those features trickle down to Kubernetes, our customers will be able to leverage both."

On the other hand, Kubernetes has its own benefits for enterprise customers. In an interview, Docker Senior VP Scott Johnson highlighted the additional benefits customers get when they use Kubernetes with Docker Enterprise Edition. "Customers now have the ability to create their secure supply chain while using Kubernetes as an orchestrator for their run-time environment. They get compatibility across their deployment models – whether they use Linux, Windows, or a mainframe. They get benefits of Kubernetes and all the additional capabilities, especially around security, that Docker Enterprise Edition offers."

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

comments powered by Disqus
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters

Support Our Work

Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.

Learn More