Bottlenecks Discovered by Meta Engineers Lead to Scalability Improvements
Thanks to a small group of engineers at Meta, the Linux kernel module kernfs has been improved, which has led to some serious gains.
After engineers at Meta discovered that certain system daemons were getting stuck for minutes at a time on a massive collection of servers, Shakeel Butt, one of Meta's engineers, set about fixing it.
"Butt authored 18 patches for Linux 7.4 in enhancing the Kernfs code," said Michael Larabel of Phoronix. He continued, "The code improves the handling around different locks in Kernfs as well as fixing a few Kernfs bugs in the process."
Four particular patches, created to resolve this issue, saw serious performance gains when applied. In his message, Butt addressed two of the daemons that were causing the most issues happened to be ones that could least afford to lose performance: oomd (which decides what to kill when a machine runs out of memory) and below (which records the telemetry used to understand what happens after oomd acts).
Another issue Butts mentioned was kernfs_dop_revalidate(), which, according to Butts, "takes kernfs_rwsem for read once per path component of every walk into a kernfs mount. Linux rwsems do not permit reader lock stealing once a writer is queued…."
There's also this pull request from Christian Brauner, where he states, "This contains kernfs scalability work from Shakeel driven by what Meta sees in their fleet. kernfs_rwsem is per kernfs root, so sysfs and cgroupfs each have one for the whole machine, and every create, remove, and rename takes it for writing. rwsems don't let readers steal the lock once a writer is queued, so one cgroup being created or a device being renamed parks the entire incoming reader stream in uninterruptible sleep."
Larabel also posted a graph that shows how the Linux kernel performance improved after the patches were merged. This addition from Meta should give enterprise businesses using Linux a considerable boost when massive scaling comes into play.
