Canary vs. Overflow

A stack overflow overwrites parts of the stack. Protection can be provided by a canary, a bit sequence prior to the return address that the attacker cannot predict. If the patterns no longer match, that means somebody overwrote the stack. This is a warning, in the same way that a canary warned miners of mine gas.

A skillful attacker can work around this canary protection. For example, a technique called trampolining bends existing pointers in the program so that they point to the return address. However, this approach requires considerably more effort and the use of vulnerable pointers. Similarly, an attacker might try to read out the canary value and adjust the buffer overflow appropriately, but the programmer can counteract this step by randomizing the canary.

Shadow stacks provide a similar line of defense: They keep a copy of the return address, which the attacker is unlikely to access. The process checks whether the copy matches the original before it is returned [8].

The heap, as well as the stack, is sometimes vulnerable to overflows. Attackers often take advantage of use-after-free errors: The programmer releases memory with free(), but then uses the pointer again later on. Such errors can be corrected by more stringent checks when accessing memory areas in the kernel.

Invitation to Gamble

Attackers rely on well-known facts relating to the operating system's memory layout to inject their code (Figure 2). If the kernel randomizes the design of the addresses, attacking is like gambling. Although there are attacks like heap spraying, the overhead is growing. If both the kernel and the kernel modules are loaded at random addresses, and possibly in a random order, attacks on the kernel modules are more difficult. Similarly, random stack base addresses increase the effort required for a successful attack.

Figure 2: Normally, the distribution of the address space is defined on a 32-bit system of the I-386 architecture. The heap and stack are moving towards each other from two sides as their size grows dynamically.

The more chance comes into play, the more important it is for the attacker to find out something about the memory structure or the value of canaries through information exposure. Attackers take advantage of the fact that released memory is usually not overwritten. Uninitialized variables or format string vulnerabilities also help.

To avoid this kind of information gathering, it is important to overwrite the memory immediately after releasing it. In addition, functions can be accessed via IDs and a table, analogous to the interrupt vector table, instead of directly using their addresses. This precaution means that the addresses outside the kernel are not known and are more difficult for attackers to predict.

Modular Is Cool

Attackers who have managed to inject malicious code in spite of the resistance usually look to install a rootkit, which they can do simply by loading their own kernel module into the system. To prevent this, the admin could compile the kernel statically – without module support – but a static kernel is often inconvenient, unless it is used with an embedded devices. Alternatively, the Self-Protection project suggests that only a few locally logged-in users should be allowed to load modules.