Improving Spinlock Efficiency

Alex Kogen, from Oracle, wanted to eke out a little more efficiency from spinlocks. Locks are part of any multitasking operating system – they make sure only one process can access any given system resource at a time. Spinlocks are locks that just loop around and around waiting for one process to let go of a resource so the next process can claim it. There are lots of different kinds of locks, depending on a ton of different special contexts. For example, an MCS lock is a particular type of spinlock named after the people who first described it: Mellor-Crummey and Scott.

In this case, Alex said, "In CNA [compact NUMA-aware], spinning threads are organized in two queues, a primary queue for threads running on the same node as the current lock holder, and a secondary queue for threads running on other nodes. After acquiring the MCS lock and before acquiring the spinlock, the MCS lock holder checks whether the next waiter in the primary queue (if it exists) is running on the same NUMA [non-uniform memory access] node. If it is not, that waiter is detached from the main queue and moved into the tail of the secondary queue. This way, we gradually filter the primary queue, leaving only waiters running on the same preferred NUMA node."

Alex was talking about having spinlocks be aware of the special context of the threads waiting for the resource they guard. Specifically, on systems with more than one CPU (which is pretty much all of them these days), threads on any CPU might be waiting for resources such as RAM on one specific CPU. But whenever a thread uses a resource on a different CPU from its own, access will be slower than a thread accessing that resource on its same CPU. Also, different CPUs may have RAM that's inherently faster or slower than RAM on another CPU. Those are called NUMA systems.

[...]