Kernel Versions

Linus Torvalds announced that a 4.0 release would be coming up within the next year or so. He also clarified how he planned to structure future version numbers.

In ancient days, Linus tried a variety of approaches to version numbering, including the most famous scheme, which involved odd-numbered kernels (e.g., 2.3.x) accepting new features, and even-numbered kernels (e.g., 2.4.x) accepting bugfixes only.

That approach collapsed with the 2.6 tree, mostly because (newsflash) bugfixes are a pain in the butt, and developers prefer to code new features. It had reached the point where a significant number of developers would just go into hibernation during the stabilization series, which dragged out the process further.

Over the course of the 2.6 tree, Linus gradually abandoned that versioning scheme, leaving nothing in its place. Kernel development continued, and pressure mounted to find some way to stabilize the code. Eventually, Linus decided that because the vast majority of users ran distribution kernels, the distribution maintainers should handle the stabilization process. The official tree, although not completely ignoring the need for stabilization, would not prioritize it over active development.

The distribution maintainers managed their new official roles much as they had before  – distribution kernels had always modified the vanilla kernel anyway. But, various other stabilization efforts emerged as well.

Andrew Morton converted his -mm tree  – formerly a repository for patches he intended to send up to Linus – into the more formalized linux-next tree, where code could live and receive lots of testing and bugfixes before heading over to Linus's tree.

Greg Kroah-Hartman and a bunch of other folks also started an independent stabilization effort, essentially forking each official Linux release, putting out a few new sub-versions that had bugfixes only, until the next official release came out. Occasionally, Greg KH would choose one of the forked versions as a "long-term" stable tree, which would continue to stabilize well beyond the next official release. The long-term stable trees would typically be used at the core of a particular Linux distribution, while remaining available to everyone. Linus affectionately called the whole stable-tree process the "suckers tree," because of the huge amount of work it required to maintain.

Greg's "suckers tree," Andrew's linux-next tree, and the distribution trees all combined to provide a significant level of stabilization, leaving the official tree to focus on development.

But, without the even-number/odd-number cycle, when would the 2.6 tree ever make the jump to 2.7? The 2.6 releases kept piling up, and developers actually started complaining about the version numbers getting too big.

Finally, apparently just because the 2.6 tree had reached the uncomfortably large sub-version number of 2.6.39, Linus decided to come out with version 3.0. This didn't correspond with any huge change in the kernel internals. Linus apparently just did it because the version numbers had gotten too big for his liking.

In fact, to reduce the size even further, Linus replaced the x.y.z numbering system with an x.y system. Instead of 3.0.0, 3.0.1, 3.0.2, and so on, he released 3.0, 3.1, and 3.2. This left the third number available for the "sucker tree" to put out stable Linux versions 3.0.1, 3.0.2, 3.0.3, itself. As his own nod toward stability, Linus implemented a stricter release-candidate system with a brief merge window for new features and a brief stabilization window before putting out each new release.

Fast forward to the present day. Linus announced that he's planning to put out 4.0 sometime in the 3.19 time frame. He said, "I'm ok with 3.<low teens>, but I don't want us to get to the kinds of crazy numbers we had in the 2.x series, so at some point we're going to cut over from 3.x to 4.x, just to keep the numbers small and easy to remember."

No big plans. No significant meanings. Just a way to keep the version numbers small. However, Linus did also say that he wanted the 4.0 release to be a stabilization release. For that one release, he wanted everyone to pile into bugfixing and hold off on new features until 4.1.

As an indication of how far his thinking had come since the old even/odd days, he said, "I may be pessimistic, but I'd expect many developers would go 'Let's hunt bugs. Wait. Oooh, shiny' and go off doing some new feature after all instead. Or just take that release off. But I do wonder … Maybe it would be possible."

Among the various responses, Ingo Molnár said he didn't think the push for stability would be so hard for developers, but that "v3.19 would also probably be a super-stressful release to maintainers, as everyone would try to cram their feature in there," because "if anything important misses that window then it's a +5 months wait."

Ingo also questioned some of Linus's assumptions, saying, "It's not developers that typically determine the stability of a subsystem but _maintainers_, and the primary method of stabilization is, beyond being careful when merging a patch, to remember/monitor breakages and not merge new feature patches from a developer until fixable bugs are fixed by the developer."

Ingo said to Linus, "the main steering parameter of our kernel stabilization process is the maintainer directly above a developer, the first-hop maintainer. For 90-95% of the commits you are the second hop maintainer or higher. So whether in 4.0 you are going to take non-fixes will not directly affect the stabilization process and flow that is already in place." He suggested, "why not just do what worked so well for v3.0 and afterwards? Keep the existing process in place, don't upset it just due to a (comparably) silly number tweak."

Greg KH agreed with Ingo's idea that developers would try to cram features into 3.19 that weren't quite ready. He said, "I see it today when people are trying to guess as to what the next long-term-stable kernel is going to be and they throw thing in that are half-baked just because they know they can 'fix it up' later."

Greg felt his trees did a good job of stabilization. He said, "The fact that we had 4,000 bugfixes added to the 3.0-stable kernel is a testament to the fact that developers and users are paying more attention to the stable kernel releases." He pointed out that evangelizing solid practices for the -stable trees would probably accomplish more than a one-off push for 4.0 stabilization.

Elsewhere, Olof Johansson also pointed out that Linus's 4.0 stabilization idea seemed "more like a 'forced' extension of the -rc cycle by several weeks," because "we already do this for every release, where the last couple of weeks are strictly bugfixes only." He added, "If you're doing a 3.20/4.0 that is bugfixes only, then why release 3.19 at all? If the only difference between the two is said fixes, we'd be better off just holding on until the latter is released."

Alexander Holler also remarked that "The problem with many bugs is that workarounds don't get accepted. So the situation sometimes is that there is a driver which isn't that perfect and when you try to fix something, the patch is refused because it is as imperfect as the whole driver (even if the patch would cure some problems). So the only fix which won't be refused is a rewrite of the driver which often just isn't what people are willing to do."

Keith Curtis replied, "I think a better explanation is that there aren't metrics tracked by anyone, some important drivers are understaffed/abandoned, and some people don't care about their bug count." He suggested greater reliance on tools like Bugzilla, to track bugs and hold developers accountable for keeping their bug counts down.

The discussion ended for the moment, with an indication that the meaning of kernel version numbers continue to diminish, determined primarily by convenience rather than milestones in the code. Linus's desire for a 4.0 stabilization effort feels like a momentary whim, but indicates that stabilization remains near to his heart, and he'd like to find a useful way to address the issue.

Status of the 2.6.34.x Stable Tree

Aaro Koskinen was concerned that the 2.6.34.x stable series hadn't put out a new version for awhile, and Paul Gortmaker, the maintainer, had mentioned in July 2013 that he planned to end-of-life that series. Aaro asked if the EOL process had taken place, and if so, he suggested mentioning that on the kernel.org front page.

Paul replied, saying he did intend to put out another 2.6.34.x release and remarked, "It will be marked EOL at kernel.org when it is EOL. Those that care about it being EOL would have seen the message about it becoming EOL in the previous 2.6.34.x release announcement."

But H. Peter Anvin pointed out that it had been "over a year since the last 2.6.34 release, which lags the last 2.6.32 release by about five months. I am asking because someone just queried me privately about the status of 2.6.34. I'm worrying if people think that security patches are still being backported if in fact they aren't."

Paul replied that the final 2.6.34.x update would be out within a week, "with a focus on just clear CVE[-]like fixes and hence a relatively smaller queue size (i.e. nothing like 200 patches etc.)"

Paul said Peter could mark 2.6.34.x as EOL on kernel.org if he wanted, but Peter replied, "No, if you are genuinely planning another release soon that's fine."

Changing The ABI

David Howells recently broached a touchy subject. Some of the Kerberos developers wanted him to change the behavior of the add_key() and request_key() system calls, along with some kernel functions. David wanted to know if Linus Torvalds would be OK with that.

Specifically, he said, "they would like libkrb5 to fall back to using the session keyring if keyctl(KEYCTL_GET_PERSISTENT) fails with EOPNOTSUPP (say if CONFIG_PERSISTENT_KEYRINGS=n). However, this means that if you don't have a session keyring, kinit has one forcibly created for it by the kernel and then your credentials cache is deleted when kinit exits."

David saw about a half-dozen possible approaches to give the Kerberos developers what they wanted. Some of these did not require any changes to the kernel, but others did, and David seemed to feel that a combination of solutions would be best.

For example, he said, the Kerberos people could just "manually create a session keyring somewhere before it is needed." This way, they'd never run into the problem of kinit creating one and then deleting the credentials cache. This required no kernel changes, and David felt the Kerberos folks should probably do that.

On the other hand, David also liked a solution that did require kernel changes, mostly because, as he said, "the Kerberos people have a valid point. The current behaviour is poor." The idea he wanted to implement was "don't implicitly create a new anonymous keyring and don't implicitly set the session keyring to the user-session keyring, but rather just fall back to using the user-session keyring if there isn't a session keyring."

David said that this change would make no difference at all to most processes and would only affect programs that were run from pam_keyinit-managed login shells but that didn't use libkrb5 or keyrings.

The reason this is a touchy subject is because it would involve changing the way the kernel executes existing compiled software. It would change the kernel's ABI (application binary interface). As James Morris said in his succinct reply to David, "So there are existing apps which will see semantic changes? If so, we can't accept this change."

James suggested "creating a new system call with the desired behavior, and deprecating the current one (or at least, making it a wrapper for the new call)."

Linus Torvalds responded to David's idea, saying: "… the rule about ABI changes has always been: 'If somebody notices it, we revert it'." He went on:

IOW, it's not so much that you can't make changes, it's that you cannot make changes that break programs. And quite frankly, I have no good way to judge. Your [ABI change] _sounds_ fair, but who knows what odd things some distributions do. We'd have to be very careful, in *particular* we'd need to make it very very clear to the krb people that if the change screws over any old users, it gets reverted with extreme prejudice. At that point, maybe they say "never mind, we might as well just always make sure we have a session keyring."

And no, we are *not* going to say "we'll just stop doing this in the kernel and expect pam_keyinit to do it for us."

But James' suggestion to perhaps have a new version of add_key() with new semantics could work – if it's worth the pain (because we *would* have to maintain the old interface basically forever, so it would be more of a "the new system call doesn't really deprecate the old one, it just has more convenient semantics").

The discussion ended there. It's fascinating to see the way ABI changes are handled in kernel development. There are various nuances, such as being able to make such changes if truly no users are affected; in some cases, if a binary interface is simply broken, it might be fixed in a way that could break some user code. Also, if there were a security issue, that would trump ABI backward compatibility to some extent.

Aside from those few things, the constraint against breaking existing software seems sacrosanct. It didn't have to be that way. Linus might have decided that breaking very old software would be acceptable. It would certainly save the kernel some hassles, but Linus places a tremendous value on never breaking existing compiled code, unless there is absolutely no choice. Even deprecating a binary interface over a long period of time, to give developers a chance to fix their sources and compile new binaries, is frowned upon because, in some cases, the compiled binaries themselves are all that remain of a given project.