When the performance analysis book by Brendan Gregg [1], which I had long been waiting for, appeared recently, I devoured it greedily. I even had access to a prerelease so that in January I was able to give the readers of Linux Magazine some tips on the kernel probes for throughput measurement by means of the Berkeley Packet Filters [2]. Performance guru Gregg also explains at the beginning of the book how he often uses classic command-line tools to find out what is causing a struggling server to suffer.

10 Commands

He starts off with a list of 10 command-line utilities (Listing 1) [3], which every Unix system understands, checks everyday things – such as how long the system has been running without rebooting (uptime) or whether there is anything noticeable in the system log (dmesg). The vmstat command looks for processes waiting for their turn on a CPU. It also checks if all RAM is occupied and the system is wildly swapping. Similarly, free shows the available memory.

Entering pidstat visualizes how processes are distributed on the system's CPUs; iostat determines whether the data exchange with the hard disk is the bottleneck. The utilization of individual CPUs is illustrated by mpstat, which also shows whether a single process is permanently blocking an entire CPU. The sar commands collect statistics about the network activity on a regular basis, including the throughput data. The best known of the tools, top, gives an overview of RAM usage and running processes, but according to Gregg, should be called last.

[...]