One class of tools captures the events and function calls of compiled programs (and sometimes even of the entire system), producing logs known as traces. Programmers can use these logs to understand which function calls and actions lead to errors, and repeated entries could indicate race conditions or deadlocks. Reverse engineers might find debuggers useful, and kernel developers could gain insights into the typical processes between user space and the operating system kernel. Admins can use these tools to find out what's happening on a system, with traces helping them take a closer look at suspicious behavior. For example, if you monitor and log all calls to the system function open, a look at the debugger output could reveal which programs regularly produce temporary files of several gigabytes.

The candidate tools are Strace [1], LTTng [2], SystemTap [3], and DTrace [4]. In this article, I'll kick off with the classic tool, Strace, follow with the slightly more powerful tools, and finish off with DTrace, the most powerful tool in the test.

All candidates link into the Linux kernel, allowing them to monitor much of the system and keep an eye on things, like sockets, as well. In some cases, the debuggers capture network traffic and log the processor load or CPU time of individual actions, thus serving as profilers for developers and providing insights on performance bottlenecks to admins. The analysis works on any Linux system, without the need to inspect the source code of the application to be monitored or of the kernel.

[...]