Zack's Kernel News

Kernel Gatherings Canceled Due to COVID-19

There are normally many Linux and open source conferences throughout the year. These have started to see some disruption due to the COVID-19 pandemic. In mid-March, Josef Bacik announced that one of the storage, memory, and networking summits would not be happening in April as planned. He said:

"We currently do not have concrete plans about how we will reschedule; the Linux Foundation is working very hard at getting us alternative dates as we speak. Once the new plans are concretely made, we will notify everyone again with the new plans.

"The tentative plan is to keep the attendees as they are if we reschedule within 2020. This includes anybody that declined for travel-related concerns. We will re-send all invitations again to the original invitees so it's clear that you have been invited.

"If we have to reschedule into 2021, then we will redo the CFP once we are closer to the actual date again and redo all of the invites and topics so we're as up to date as possible with the current state of the community.

"We will keep the current program committee, and I will continue to chair until we have the next LSF/MM/BPF."

Kernel Meta-Organizations Changing Due to COVID-19

In mid-March, Laura Abbott announced some changes to the Linux Foundation's Technical Advisory Board (TAB) charter that affected the way the 10 members were elected. In particular, there would be provisions for absentee voting, in case community members were not able to attend the events where elections were traditionally held. There would also be an expansion of voter eligibility, to include more than just the invitees to the Kernel Summit events.

The TAB is a very interesting construct. Technically part of the Linux Foundation, it typically consists of high-powered members of the kernel development community itself, acting as a liaison between the developer community and the Linux Foundation Board of Directors.

A number of folks had issues with Laura's changes, having to do with ballot stuffing and other potential cases of voter fraud. As Theodore Y. Ts'o pointed out, this issue is a lot more important now than it was five years ago. Nowadays, given the ongoing assaults on elections everywhere, it's not inconceivable that something like TAB elections could come under attack.

Ted said, "we need to be smart about how we pick the criteria [for identifying eligible voters]. Using a kernel.org account might be a good approach, since it would be a lot harder for a huge number of sock puppet accounts to meet that criteria. We don't have a final proposal for something which can be objectively measured, but can't be easily gamed by someone who is trying to subvert the system. It is pretty clear, though, that we need to have that clearly articulated, in writing, *before* we start the nomination for the next round of TAB candidates."

Ted also pointed out that there was probably no choice but to allow absentee voting, given the COVID-19 pandemic. He said, "it's only responsible to consider what we should do if in fact health and safety restrictions are such that we might not be able to hold *any* Linux systems conferences in 2020."

He proposed that "in that case, we might be forced to either keep TAB members in place beyond their original elected term, or we might have to go to a pure electronic voting for the upcoming TAB election. I very much hope that won't be the case, but we need to be prepared for that eventuality."

Jani Nikula pointed out some problems with requiring a kernel.org email address. He said, "it's at the kernel.org admins discretion to decide whether one is ultimately eligible or not. Do we want the kernel.org admin to have the final say on who gets to vote? Do we want to encourage people to have kernel.org accounts for no other reason than to vote? Furthermore, having a kernel.org account imposes the additional requirement that you're part of the kernel developers web of trust, i.e. that you've met other kernel developers in person. Which is a kind of awkward requirement."

But Greg Kroah-Hartman said that in fact, the whole point of requiring a kernel.org account was to make sure voters were part of the developer web of trust. He said, "That is the goal here; if you know of some other way to determine this, please let us know. We went through many iterations of this and at the moment, it is the best we can come up with."

He added, "Also, note that the 'kernel.org admin' is really a team of people who have been doing this for 9 years, it's not a single person responsible for giving out new accounts to people that do not meet the obvious requirement levels as published on kernel.org."

Greg also did acknowledge, "These are 'baby steps' we are taking here, to try to allow for remote voting. We are not saying this is the end-all-be-all solution, but you have to give Laura credit for coming up with this as 'better than nothing' which is what has been the case for the past decade or so."

Several folks pointed out that the charter changes were always an expansion of the voting pool, not a constriction of it. As Steven Rostedt said, "We are trying to extend who can vote beyond those that the charter allows. We are not preventing those that can vote under the current rules from voting. IIUC, we are trying to create absentee voting which we never had before. Thus, you can either vote the current way by getting [to] travel to wherever Kernel Summit is and attending the conference, or we can extend the charter so that if you can not come for whatever reason, you have an option to vote remotely, if you satisfy the new requirements. Remember, not attending means you do not satisfy the current requirements."

Steven added, "The TAB has bikeshed this a bit internally, and came up with the conclusion that kernel.org accounts is a very good 'first step'. If this proves to be a problem, we can look at something else. This is why we are being a bit vague in the changes so that if something better comes along we can switch to that. After some experience in various methods (if we try various methods), we could always make whatever method works best as an official method at a later time."

At a certain point in the conversation, James Bottomley offered some historical context. He said, "When the TAB charter was written (in 2006), the original reason was to prevent manipulation (real or imagined) by the committee who would then become the arbiters of nominations and thus able to influence who might run for the TAB. There are a couple of reasons for the electorate clause: When the TAB was formed, it was done by the kernel developers unhappy at the way OSDL (precursor organization to the LF) was behaving with regard to the kernel, who forced their way onto its board and formed the TAB to gain input and control on behalf of kernel developers, so the TAB was formed by kernel developer[s] for kernel developers and, since most other non-kernel open source groups had their own foundation like entities, keeping it kernel only wasn't seen as a problem. The other reason was that OSDL was a bit unhappy to be reformed in this way, and we foresaw that one way to dilute the reforming influence of the TAB would be to dilute kernel developer representation since they were the main community interested in that reform. When the OSDL became the LF, some of the initial antagonism and need for reform went away, and the elections were opened to the co-located conferences as a sign of improved trust."

Minimum Version Numbers for Build Tools

One of Linus Torvalds' development values is that the kernel should be buildable using tools that are very old. This way very old systems can continue to build fresh kernels without having to do major software upgrades. And this is good because not all hardware can tolerate major software upgrades.

So whenever Linux changes the minimum version number required for a given tool, it's a significant event. One of the big tools is the GNU C Compiler (GCC), but lesser tools are also important. Recently Borislav Petkov posted a patch updating the minimum version number for binutils, from version 2.21 to version 2.23.

Binutils is a set of GNU tools that are used specifically for writing software. They include an assembler, a linker, a profiler, and a lot of other truly amazing developer tools that are absolutely essential for Linux kernel development.

The problem, as patched by Alan Modra in 2012 (!), was in the linker – that part of binutils that combines a bunch of compiled binaries into a single executable program. To do that, the various files need to make reference to key memory locations within one another. And under some circumstances, the older version of the linker was pointing to the wrong locations.

So, as Borislav said, changing the required binutils version number was long overdue. He pointed out that the various Linux distributions already shipped with the fixed version of binutils, so no one using a standard distribution would even notice the new version requirement.

Kees Cook and Jason A. Donenfeld approved Borislav's patch, and Jason pointed out an immediate way to further clean up the kernel source tree, when he remarked, "then we'll be able to use ADX instructions without ifdefs."

In fact, Jason advocated bumping up the minimum binutils version number as high as 2.25, saying, "we could get rid of *all* of the CONFIG_AS_* ifdefs throughout."

However, that suggestion got lost in the wind. Bumping up to version 2.23 was already controversial enough. In particular, the timing of the patch was important. Borislav wanted to get the patch in the 5.7-rc development cycle after 5.7-rc1, to ensure it would be too late for the official 5.7 release, but would still get the longest possible period of testing, before coming out in the official 5.8 kernel release.

The goal was to get as much testing done by developers as possible, so that any users who still relied on the earlier version of binutils could be discovered, and any valid arguments not to bump up the version number could be revealed.

The controversy, however, was not about whether to bump up the version number – everyone agreed it was essential. No, the most controversial thing about Borislav's patch was in waiting until Linux 5.8. Both Kees and Jason had hoped to see this change go into the kernel in time for the 5.7 release. Jason said, "I don't think that filing these away in some subsystem tree will actually result in any more bugs being shaken out, than if we just do this at the very, very beginning of the 5.7 cycle."

But Borislav remarked wryly, "Are you or Kees going to deal with any fallout from upping the binutils version, rushed in in the last week before the merge window? Because I sure as hell am not. *Especially* if there's no big difference when it goes in."

At this point, Linus came into the discussion, saying:

"I think it's ok. It's not going to cause any _subtle_ failures; it's going to cause very clear 'oh, now it doesn't build' errors.

No?

And binutils 2.23 is what, 7+ years old by now and apparently had known failure cases too.

But if there are silent and subtle failures, that might be a reason to be careful. Are there?"

Borislav said he didn't know of any subtle failures, but better safe than sorry. He remarked, "a lot of hectic testing of last minute fixes in the past have taught me to take my time."

He acknowledged that binutils 2.23 had been tested for a very long time already, and that probably all would be well. And if Linus wanted the patch to go into 5.7, then who was he to argue. But he did say, "since it doesn't really matter when the patch goes in – there's always the next merge window – I would prefer to take our time and have it simmer in -next for max period."

Meanwhile Kees said he didn't really mind waiting. He remarked, "I have plenty of other hills to die on, so I have no urgency on this change. I actually thought it had already happened, since it was brought up a while ago. :) I am just excited to see it happen since it unblocks other work I've been touching. As long as it's 'eventually', I don't care when. :)"

Meanwhile Arvind Sankar asked, isn't this just a documentation patch? No actual code will change, so why is everyone being so careful about it?

Linus remarked that while this patch was indeed just a documentation change, "there's a second patch that knows that if we have binutils >= 2.22, then we don't need to check for AVX2 or ADX support, because we know it's there."

But Arvind also pointed out that the Linux kernel already hadn't supported pre-2.23 binutils since Linux 5.3 – the kernel build would actually break with anything less than binutils 2.23. So, as Arvind put it, "we've already gone 3 kernel versions without complaints."

That really put an end to all further discussion.

But it's nice to watch the kernel developers try to deal with this sort of change. They all really wanted to make the best choice that would maximize the possibility of identifying absolutely any user whose system might break as a result. So OK, Arvind gave us a smile at the end, realizing the entire discussion was moot. But it shows the sort of things that really matter in kernel development. Reliability.

Contrast that with so many devices and systems in the commercial world, where products go into end-of-life solely in order to force users to buy the next product. Linux has a very different philosophy, and it shows in the raw ubiquity of Linux. It is truly everywhere.