Greg Kroah-Hartman is one of the friendliest faces of the Linux kernel community. He maintains the stable branch of the Linux kernel and participates in many more Linux projects. He works at the Linux Foundation as a fellow, a job that allows him to dedicate his time to the Linux kernel. We meet each other at almost every Linux-related conference. I sat down with Greg to talk about the kernel, 25 years of Linux, and what he does with the rest of his life.

Linux Magazine : What and how much has Linux achieved in these 25 years?

Greg Kroah-Hartman : We had this joke: "Total world domination was our goal." Everybody laughed at us. Then we did it, and nobody even noticed. I mean, we're everywhere. We're going places that Linux has never been. It's crazy to think that this project we all started working on for fun, myself included, has done this. It's insane.

I was talking to people the other night, and we were discussing that any chip that has over four megabytes of memory runs Linux. We're even trying to get it on smaller chips now. We are everywhere – from giant supercomputers to devices with less than 4MB of RAM.

LM : Linus started Linux in 1991, when did you get involved with it?

GKH : I started using Linux in the 1990s when Oracle first got ported to it because I was working for a company that ran Oracle on SCO machines. That was the first time I ran Linux. I used it a little bit and had fun building my own kernel, but I never saw anything that I could contribute.

In 1995, I was working for a barcode scanner company called PSC Inc. doing embedded software development. I wrote firmware for a barcode scanner that used USB. Linux didn't have a USB stack then. I had to test my USB scanner on all operating systems to make sure I got my USB code right. I saw Linux and watched the USB stack being developed for it. Linus and other developers started writing a little bit of code, and USB support started to get implemented. With that codebase, I saw little places where I could contribute, fixing odd bugs and adding small features. Linus took my patches and it was exciting. I said: "This is fun."

LM : Linus started Linux to scratch his itch, but you got involved due to your day job, or was there any personal itch to scratch?

GKH : It was part of my job, but I had my own itch. I had a USB device that didn't work on Linux. My wife went away on vacation for a weekend and said: "Why don't you write a driver? You always talk about that." I said OK, I'll write a driver. I wrote a driver over the weekend and submitted it, and I swear within an hour people came back pointing out problems telling me: This is wrong; this is wrong; this is wrong.

It felt awesome. They were critiquing my code, and I was learning from it. I always want to keep learning, so I said "Yes, you are right. This is wrong, this is wrong, and this is wrong." I iterated and fixed problems with it. It got accepted into the kernel. To me, it was fun.

I think feedback is very important. That feedback loop of people pointing out errors or problems with what you're doing is a very traditional, I guess scientific, method. And I love it. That's how we get better.

LM : At that time, you were working at a proprietary company. How different was the development inside the company as compared to open source? What kind of feedback would you get when you wrote code internally?

GKH : PSC Inc. was selling barcode scanners. We had a traditional model of software development, which you followed to write the code. I would get a lot of review about the specs, but when you wrote the code, there was nobody to review the code. It would go to testers and they would find problems. There would be a feedback loop in that model, but there was a gap. You write all this code, but you don't get any feedback at early stages as other developers are also writing their own stuff, and then all of it goes to testers. You didn't get feedback from fellow developers.

Anybody who has written code knows that it's better to find problems as early as possible. And that's the way Linux works. We find most bugs and problems before anybody actually runs the code. The code gets reviewed at a very early stage.

People send me patches. I was flying across the Atlantic for this conference, and I had 800 emails about just one subsystem. I went through them and found out that 200 of them applied, so I rejected the rest. Well, that's a bad percentage. That's fine. The point is that the review of the code was done immediately, at a very early stage. All of that was just from code review, before the code was even run. We go back and fix things immediately. That helps to catch things really early in the process, and that's good. It turns out to be a much better way of developing software.

LM : So how did you get from PSC to the Linux Foundation?

GKH : I wrote that driver a long time ago, and we didn't use Linux or anything; we were using embedded systems. Then things took off in 2000, around the dot-com boom. Suddenly there were all these new jobs. That was also the time when I was getting all these emails from people who were using the code that I had written. That's when I realized more people were using the stuff I wrote for free as compared to anything I ever got paid for (chuckles).

Even though the company was relatively successful – we sold barcode scanners that ran everywhere – those emails were exciting. I realized that it was so much fun. Then I got offered a job that essentially said: "You want to do this full time?" I was like "Yes!"

After PSC, I worked for WireX for a while before moving to IBM. I worked at IBM for a number of years and created the driver model with Pat Mochel, who was working at OSDL. Then I was on the IBM kernel team. After that, I moved to SUSE, and now I am a fellow of the Linux Foundation.

LM : What is a fellow of the Linux Foundation? What's your actual job?

GKH : My official job title is: "Linux Foundation can't tell me what to do, and I can't tell them what to do."(Chuckles) I do the stable kernel releases. I'm paid to be able to come and do my development without having to create a product or anything else like that, so it's fun.

I go around and help companies, teach them how to get involved in kernel development. Lately, I have been working with other projects to teach them how to scale. For example, Kubernetes hit a scaling wall recently with the number of contributors. I showed them how the kernel team solves these problems and helped them in figuring out that maybe they can change their development model and do better. A few years ago, I talked to Docker developers on how they can scale better. I have done that for some other companies and groups, like the Open Platform for Network Functions Virtualization (OPNFV) project.

LM : Has the development model changed over the years, or is it still the same?

GKH : The development model has changed a lot. We have changed it over time. We started Kernel Summit so we can meet each other and figure out how to continue to do this better. Part of our Kernel Summit is to talk about what is going wrong and what we need to do to fix it.

At one of the kernel summits, Larry McVoy showed us how to use BitKeeper, the distributed source-code control system. He showed us how we could distribute the source code and distribute development in a way that sped up the kernel development immensely.

Previously, I'd send an email to Linus with patches, he'd apply them to his local tree, and every week he would do a release. But you are left wondering whether he took your patch or not; you had to figure it out on your own. BitKeeper changed that. If I sent him patches, he'd push it out to his BitKeeper public tree, and I could see that.

Now it's even better with Git, because it's all distributed and scaled. I can work on the plane, without any email or Internet access. When I land, I push publicly to my kernel.org tree. The test infrastructure kicks in, and if the code passes all the tests, it moves to the next stage.

Now we have automatic testing and a distributed development model, which is so much faster, because we are dealing with lots and lots more people. Fifteen years ago, we were doing two patches an hour. At that time, we all thought that was insane; it was faster than anybody else. Now we're at eight patches an hour, and every year it keeps going up. The number of developers is increasing every release. The number of companies contributing is increasing. It's just crazy.

LM : You are often seen as second in charge after Linus. What real challenges do you see ahead for the Linux project?

GKH : I'm not second in charge. Linus and I have different roles. We deal with different people. Every year I keep saying there is no way we're going to scale faster if we're going to get bigger. We passed the other operating systems' development size and speed years and years ago. Now even Microsoft runs Linux. It's amazing. My worry is that the only thing that's going to hurt Linux is Linux itself. If we mess up. My goal is not to mess us up, and I try to help see the problems. We continue to talk about issues at the Kernel Summit and try to find ways to fix them. We are constantly paying attention, so that we are able to say that, yes, we're doing something stupid here, let's fix it. It's an iterative process.

LM : While we are talking about challenges, I recall Linus once said that Linux can't shrink anymore, and if you want to work on really small devices, you will have to end up with other alternatives. However I have talked to some companies that say you can still strip it down to do whatever you need…

GKH : I've been working a lot with those people that do that. You can run Linux on 4MB; it's a piece of cake. And you can get it down to 2MB. Marcel Hoffmann of Intel delivered a talk at LinuxCon on how to get it even smaller. The other night we were talking over beer (and I always get in trouble when I do that), and I said we can figure out a way and probably get it down to 512K, maybe even smaller. It really depends on what you want to use it for. Hoffmann, for example, is a Bluetooth guy who wants to make an IoT device. He wanted a Bluetooth stack, and we figured out a way to do it. But it's hard to even find chips that don't have four megabytes of RAM.

LM : The reason I asked that question was Google's new operating system, Fuchsia, which is targeted at really small devices. They have not made any public announcement, but it's on GitHub. Have you looked at it?

GKH : Yes, I have seen it, and I know those guys. If you look at the code, it's awesome.

LM : But they are not using the Linux kernel?

GKH : No, they're using Little Kernel. It's a tiny operating system that has been around for many years. It's on most Android phones; it's your boot loader. So what they did was they took Little Kernel and made it usable. It's gotten bigger now and has SMP x86 support. But it doesn't look like a normal operating system; it doesn't have a userspace component. Fuchsia works on adding userspace support to Little Kernel. Now you can run processes and do other fun things like running programs in Dart. Little Kernel is written by the same guys who wrote the original Android code, BeOS, and Palm. They're having fun making another operating system.

The good thing about Google is that they release thousands of different GitHub projects. This is just one of them, and it's been public for a long time. Google open sources a lot of stuff.

LM : How's your work/life balance? Do you work from home?

GKH : Yeah, I've worked from home ever since I started working for SUSE. Work/life balance is tricky. In the beginning I was really strict. I would say: "Okay, I'm working this hour to this hour and then that's it." But over time, I will be talking to my wife and ask, "What are you doing? Let's go have a coffee." Then you realize you can still get all your work done.

Since I work from home, I worry about my children, because I saw my father always go off to work, and he had very good work ethics. When my son was really little he would say: "No, my dad doesn't have a job; he just does emails in his pajamas."

LM : So Linus works from his bathrobe and you wear pajamas?

GKH : Yes, I work in pajamas. In my previous house, I had a basement office. We recently moved to Paris, and I have carved out an office space with a desk in our apartment.

My kids are growing up. My daughter has given talks at LinuxCon. She has worked LinuxCon events. She knows what I do and how this works. My son was at LinuxCon Japan last year, so they know what's going on and they see that it's a very different type of job.

I work from home, and I love it; it's great. But I travel a lot, too.

LM : So when you travel around the globe, talking to these companies, what's your goal?

GKH : My goal is: I want these companies to become part of the kernel community, because everybody contributes to Linux in a selfish way. They want Linux to work in a selfish manner. And that's fine, because when you contribute to Linux in a selfish manner, it benefits everybody.

Let me give you an example. It was the embedded guys who wanted good power management in Linux to save batteries. We got it to work, and now giant companies running giant supercomputers are saving billions of dollars thanks to the power management of Linux. It benefits everybody that somebody was being selfish.

LM : These days we hear a lot about security. Are we introducing more bugs due to Linux's popularity and its rising share of media attention?

GKH : I think the security issues have always been there. My very first job in Linux was to work on a secure version of Linux at WireX. The person in charge of that, Crispin Cowan, wrote some tools and modified gcc to help prevent stack overflows. We had a DARPA contract to do research on how to make things more secure, and we wrote some USENIX papers based on those results. That was fifteen years ago; we were already talking about how security needs to be done better; what we need is to be more proactive.

It's just that now people are finally realizing that, yes, this is a problem. It's always been there, and we need to pay more attention to it. We need to do better things. We need to avoid being reactionary. We need to do things in defensive ways, like adding code to the compilers to check for stock overflows automatically. Or look at all the GR security work that's happened and get that in.

Kees Cook from Google, and Konstantin Ryabitsev, the kernel.org maintainer, look at it like cars and airbags. You can make these big steel death machines that go really fast but you need to protect people inside them, so you need to be defensive. Now cars have crumple zones, airbags, and all these other safety things that aren't there to make the car go faster, but to protect the people. As kernel developers, we have to accept that we need to take the overhead of protecting things better and create crumple zones and airbags.

LM : In a previous conversation, you talked about how the kernel community is doing all this work, but the changes don't reach target devices, because vendors are not making it happen. Google is trying to change how Android devices are updated. What's needed, a cultural change or technological change?

GKH : It's totally a cultural change. It's the old model of "Hey if it works, don't touch it. It's not going to break."

Well, the world changes. If you look at the old version of Linux, it worked great back then based on the ideas of what we knew then, but the world changes. You never want to put an old version of Windows on the Internet now, and you wouldn't want to put an old version of Linux on the Internet now either because we know all the problems.

You have to be able to update. It's a cultural change to tell people that it's okay to accept changes. 12-13 years ago, the kernel community said we should make the guarantee that we will not break your system. We give you the safety of knowing that, if you upgrade, you're okay, and we have backed that up. We've proven we can do that.

It's overcoming that fear. People will accept change, but they will accept the change that they want; they don't want the other changes. They're worried about those changes breaking something.

To make such changes acceptable, you have to put an infrastructure in place to test the changes. And companies are doing a lot of work around that. Google is working on updating Android faster. They're working on infrastructure to make it easier for SoC vendors to do the testing so they can rely on it. They can test their machine, and they can put a new kernel in there and see if passes all the tests. If everything looks good, they can push it out to the world.

LM : Where do all these new kernel developers come from?

GKH : Over the years, you'll see we had people contribute to Linux, and then they go work for a company. Then they become a manager, then they become a vice president, and then they become a CTO, and they're still Linux developers.

One of the Japanese contributors worked for a company, and he wanted to show his engineers that anybody can contribute. He was a vice president, and he contributed documentation updates. Then he went to his engineers and said "I can do this, why can't you?" That's how Linux has succeeded.

Then people move around. The funny thing about the kernel community is that we all change jobs, but we all still do the same thing. It just keeps growing. It creates communities, people teach other people, and we get more people involved.

Efforts like Google Summer of Code and Outreachy get more and more people involved. The Eastern Europe Outreach program has grown so huge that there's a whole university pumping out kernel developers now. There's a class taught at the university where one of the projects is to contribute a code patch to the kernel.

These people are coming out of college with Linux Kernel experience, and then they're going off into other companies. It's been amazing. It's just growing that way.