Dealing with Older GCC Versions

This past year, Linus Torvalds upped the supported version of GCC to version 4.9 or newer. The idea is that it's good to support older as well as newer versions, because then people running on old systems can still compile the kernel. At the same time, if no one in the world is using a particular older compiler, you might be able to simplify some kernel code by removing support for that compiler. So there's not only an incentive for the kernel developers to support older compilers, there's also an incentive for them to abandon very old compilers when they can safely get away with it.

It's an ever-advancing debate. Recently for example, Thomas Gleixner threw up his hands in disgust at some compiler behavior. Arnd Bergmann had posted some kernel patches to work around some ugly compiler behavior in GCC 4.9 that wasn't necessary in GCC 5 and newer.

Specifically, Arnd's code added some symbol references that were never actually used in the code. The symbols were then also referenced in unused inline functions, so as not to trigger a warning. Then, as Thomas pointed out, the symbols and the inline functions were all optimized out of the final compiled binary.

Thomas retched into his hand and said, "Bah, we really should have moved straight to GCC5 instead of upping it just to 4.9."

Nick Desaulniers and David Laight also retched into their hands, agreeing that this was a disgusting hack. David remarked, "It could be changed to use a symbol that the linker script already defines. However it does look like a workaround for a broken version of gcc."

A bunch of folks dove into GCC's documentation, trying to identify the intended behavior. But to some extent it was just an exercise, since the existing hack in the kernel code actually worked. A good thing too – because as Nick put it at one point, "From the link to GCC docs, the code in question doesn't even follow the pattern from the doc from informing the compiler of any dependency, it just looks like !@#$." And at another point Thomas remarked, "I clearly want a statement from the GCC people that this won't happen on pre gcc6 compilers and not just some 'works for me' statement based on a randomly picked compiler build."

At some point, Sedat Dilek said:

"There exist gcc-4.8 and gcc-4.9 for Debian/jessie where EOL was June 30, 2020 (see [] and [https://wiki.debian.org/LTS]).

"In the latest available version '4.9.2-10+deb8u1' I see no PR82602 was backported (see [] and [https://sources.debian.org/src/gcc-4.9/4.9.2-10+deb8u1/debian/patches/]).

"I am asking myself who is using such ancient compilers? Recently, I threw away GCC-8 from my Debian system.

"If this is a real problem with GCC version <= 5, so can this be moved to a GCC specific include header-file? Thinking of include/linux/compiler-gcc.h or include/linux/compiler_types.h with a GCC-VERSION check?"

Segher Boessenkool, an actual GCC developer, replied, "There is GCC 4.9.4; no one should use an older 4.9." And he went on, "Some distros have a GCC 4.8 as system compiler. We allow building GCC itself with a compiler that far back, for various reasons as well (and this is very sharp already, the last mainline GCC 4.8 release is from June 2015, not all that long ago at all)."

Segher also remarked, "Does anyone actually test the kernel with old compilers? It isn't hard to build a new compiler." To which Arnd replied (with a non-zero quantity of snark), "there are a number of notable kernel developers that intentionally stick to old versions because of compile speed. Each major compiler release adds about 4% overhead in total time to compile a kernel, so between gcc-4.6 and gcc-11 you add over 50% in build time."

Miguel Ojeda said to Segher, "I am usually in favor of raising the minimum whenever new hacks are required to be added. On the other hand, we already raised the version twice this year and it is not clear to me what is the minimum version we would need to go for to ensure this does not bite us."

Linus Torvalds agreed with that basic idea. And he went on to say:

"The good news is that I haven't seen a lot of pushback on the gcc version updates so far. I was expecting some complaints. I haven't seen a single one.

"That may be because people did end up finding it very onerous and complained internally on channels I haven't seen, but it might also be indicative of us having perhaps been a bit too timid about compiler version updates.

"However, in this case, can we just leave that old '__force_order' hack alone, and to work around the clang thing, just make a dummy definition of it anyway.

"Alternatively, just use the memory clobber. We use memory clobbers elsewhere in inline asms to make sure they are serialized; it's not normally a huge problem. Both clang and gcc should be smart enough to know that a memory clobber doesn't matter for things like local variables etc that might be on stack but have never had their address taken.

"Or are there other cases than that particular __force_order thing that people now worry about?"

On the particular question of Arnd's original patch, and how to make a proper fix, Arvind Sankar replied to Linus, "Actually, is a memory clobber required for correctness? Memory accesses probably shouldn't be reordered across a CRn write. Is asm volatile enough to stop that or do you need a memory clobber?"

A "memory clobber" is when you want to do an operation that might affect more than just the input and output variables, so you need to save some state during the operation and then restore that state afterwards. Doing the copy involves a tiny performance hit, so it's avoided where possible.

In this case, Linus said to Arvind:

"You do need a memory clobber if you really care about ordering wrt normal memory references.

"That said, I'm not convinced we do care here. Normal memory accesses (as seen by the compiler) should be entirely immune to any changes we do [to] wrt CRx registers.

"Because code that really fundamentally changes kernel mappings or access rules is already written in low-level assembler (eg the entry routines or bootup).

"Anything that relies on the more subtle changes (ie user space accesses etc) should already be ordered by other things – usually by the fact that they are also 'asm volatile'.

"But hey, maybe somebody can come up with an exception to that."

The discussion petered out around here, with no clear conclusions. Pavel Machek had the final word in the thread, remarking, "I do have [GCC] 4.9.2 on some systems. They work well, and are likely to compile significantly faster than newer ones. Please don't break them."

On-boarding New Kernel Hackers

John Wood recently made his first security patch for the kernel. But before we get into that, I'll say that there is no real "on-boarding" process for new kernel hackers. If you're a coder and you ask anyone how to join, they'll probably tell you to subscribe to the mailing list, start sending in patches, and you'll get the hang of things as you go.

So John wrote a patch, and Kees Cook sent it to the mailing list for reasons that will become clear in a moment. The basic idea of the patch, as John said in the commit message, was "to detect and mitigate a fork brute force attack." Later, Jann Horn would point out that John's code would protect vulnerable userspace code, rather than the kernel itself. And in particular, John said, "Attacks with the purpose to break ASLR or bypass canaries traditionally use some level of brute force with the help of the fork system call. This is possible since when creating a new process using fork its memory contents are the same as those of the parent process (the process that called the fork system call). So, the attacker can test the memory infinite times to find the correct memory values or the correct memory addresses without worrying about crashing the application."

Address Space Layout Randomization (ASLR) is a kernel feature that takes an eggbeater to RAM so that attackers can't tell which part of the egg to attack.

A canary is a little piece of data that you stick in memory after your variables and before the address you'll jump back to at the end of your function call. If an attacker tries to write beyond the end of a variable in order to change that return address to aim at their own malicious code, they'll overwrite the canary. Then, before returning from that function, you can check the canary. If it's been altered, you know the return address has been hacked, and you can avoid using it.

John offered a summary of another attempt to deal with this, the grsecurity project. Specifically, he said that grsecurity intentionally added delays to the kernel code as a way to frustrate the attack. However, since grsecurity only added those delays when a child process crashed (i.e., when the attacker tried a random attack that then failed), an attacker could avoid the delay by forking off tons of child processes first and then analyzing them all in turn. Then they could crash all they wanted, because the attacker would already have gotten access to all those child processes.

A second way to avoid grsecurity's delay, John said, was to fork off a child process, but then attack the parent process instead of the child. After the parent process crashed, the child would still be available to do another fork and repeat the process. The delay would not affect the attacker, because the child process would already exist and be ready for more forking.

John's patch addressed both of those attack vectors. He said, "the solution for the first bypass method is to detect a fast crash rate instead of only one simple crash. For the second bypass method the solution is to detect both the crash of parent and child processes. Moreover, as a mitigation method it is better to kill all the offending tasks involve[d] in the attack instead of use delays."

To detect a fast crash rate, John wanted to share some statistical data across all the processes forked off of the original parent process. Specifically, he said, "these statistics hold the timestamp for the first fork (case of a fork of task zero) or the timestamp for the execve system call (the other case). Also, hold the number of faults of all the tasks that share the same statistical data since the commented timestamp."

If an ungodly number of processes crash really quickly, John said, it's a sure sign of shenanigans. At that point, the kernel would simply kill all the processes in that particular statistical group (i.e., in the process being attacked). Boom! No more attack.

Kees, although posting the patch on John's behalf, also replied to the post, saying, "Thanks for this RFC! I'm excited to get this problem finally handled in the kernel."

At this point, James Morris asked Kees, "Why are you resending this? The author of the code [John] needs to be able to send and receive emails directly as part of development and maintenance."

And John explained, "I tried to send the full patch serie[s] by myself but my email got blocked. After get support from my email provider it told to me that my account is young, and due to its spam policie I am not allow, for now, to send a big amount of mails in a short period. They also informed me that soon I will be able to send more mails. The quantity increase with the age of the account."

Kees and Jann had a bunch of comments about John's implementation. In particular, Jann felt that gathering all that statistical data might not be necessary, and that maybe the most recent five forks were enough to know if someone was killing them at a rapid rate. Specifically Jann suggested, "You could keep a list of the timestamps of the last five crashes or so, and then take action if the last five crashes happened within (5-1)*crash_period_limit time."

John liked that idea and said he planned to change his implementation to match.

For the moment at least, John's patch – in whatever form it may morph into – doesn't seem particularly controversial. It solves a known problem or at least takes a hefty bite out of it. But at the same time, there are many problems that can come up. Especially with security patches, you never know when someone will stick their head up at the last minute and say, "hang on a sec, I've got a problem with this." And then maybe the patch is accepted, or maybe you go back to the drawing board.