Performance analysis with iostat, sar, Ksar, collectd, and serverstats
Performance at a Glance
We describe five tools you can use to monitor and troubleshoot your system's performance.
Sooner or later, most server administrators realize they need to do some performance analysis. Whether you hope to detect bottlenecks or plan resources, reliable performance data is essential for any well-managed network. Also, historical records showing performance over an extended period makes it easier to forecast and adapt to changes.
The most important part of measuring performance values is that you don't start measuring right away. To begin, you need to consider carefully what information you want to obtain. Here are five suggestions for what is important during a performance analysis:
- Document your observations and results. You must allocate presumptions and conclusions based on your data.
- Set up a baseline of performance for your systems. This baseline is a defined initial state that you can use to qualify measurements.
- Measure the state of the system during the analysis before making a change. All data you collect for a defined system will help.
- Locate the bottlenecks in your system. Which resources limit performance?
- Make realistic assessments regarding which measures improve performance, and restrict yourself to these parts when optimizing.
In this article, I describe some tools you can use to monitor and troubleshoot system performance.
Iostat
Data for input and output operations are important in so far as the I/O system affects overall system performance in the case of bottlenecks. The parameter iowait is used as an example here; it is the time the CPU waits for I/O requests to be processed. Large I/O latencies can be a cause of high load on the system; but, they do not always need to be.
Iostat is part of the sysstat package that appears in the Ubuntu repositories as "System Performance Tools for Linux." Other programs included in this package are sar – which is described later – pidstat, mpstat, nfsiostat, and cifsiostat. The sysstat programs are consistent front ends to the /proc
filesystem of the Linux kernel. You cannot therefore provide any more statistics than the kernel offers for /proc
.
Iostat also provides the option -d
for querying device statistics. The tool delivers the CPU utilization report using the -c
option. Keep in mind that the first output by default contains the values since the system start. All other reports refer to the interval between individual outputs. If desired, you can skip the first report using the option -y
and receive only the current values.
Performance Indicators
Advanced output using -x
gives a parameter which, at first glance, is a factor for the utilization of devices for many users: %util (Listing 1).
Listing 1
iostat
A look at the man page, however, corrects the common mistaken belief that %util indicates the actual device usage: %util is the percentage of CPU time in which I/O requests were issued to the device. It may be true for individual traditional hard drives that a higher %util value is an indicator of good device utilization. However, RAID devices or SSDs also process several requests in parallel; the CPU time spent for the device is therefore put into perspective.
Better indicators of correct I/O utilization are %iowait in the CPU report and avgqu-sz and await in the disk report. %iowait is the percentage of CPU idle time during which I/O requests were still on the system. However, another pitfall is waiting here: %iowait is a special form of idle time and thus a sub-parameter of the actual CPU idle time. If a task that lets %iowait rise and a CPU-intensive task are running on a CPU core, iowait drops automatically. This situation arises because the second task consumes CPU time and there is thus no longer any idle time.
Listing 2 shows a simple experiment. The corresponding output from Top and iostat bring the idle times to light (Listing 3). If a computationally intensive task is running simultaneously with an I/O task, %iowait is zero, but %util has risen (Listing 4). As previously mentioned, the CPU-related idle times, as well as the values avgqu-sz and await proximal to the device are significant.
Listing 2
Task Set Experiment
Listing 3
Idle Processes
Listing 4
CPU-Intensive Task and I/O Task
Iostat is a tool that provides statistics on the block layer level. Requests are often applied in this context. They are structures that the Linux I/O scheduler works with. It passes them on to the device driver dispatch queue, which presents the last step in the Linux I/O chain.
The value avgqu-sz denotes the average length of the queue that was passed to the device. From time_in_queue and the iostat interval time_in_queue, it calculates the time in which requests had to wait for the device. If several in_flight requests have to wait, time_in_queue increases by the product of time_in_queue and the number of in_flight requests [1]. Finally, avgqu-sz can be calculated from the equation [2]:
(delta[time_in_queue]/interval)/1000.0
A simple example using five in_flight requests in 20ms with an iostat interval of five seconds would result in an avgqu-sz of 0.02. The following therefore applies for a device: The larger avgqu-sz is with a constant await time, the more requests it was able to handle.
The await value is in turn the average time spent waiting for an I/O request:
delta(read_ticks+write_ticks)/delta(read_IOs+write_IOs)
The product rule for several in_flight requests as discussed previously also applies for the ticks used here. It is possible to identify from the derivation of the two parameters that they are suitable as indicators for device utilization. Large await times with small avgu-sz are an indicator that it is necessary to wait for the requests to be processed by the device.
Using plotiostat
You can create graphs from the iostat output using detours. To this end, note the iostat calls in a logfile over a longer period:
iostat -xym 1 > iostat.log
It is best to run this command on a remote server in a screen session or using nohup
. Then, the command will not abort even if the SSH connection ends. A simple Python script called plotiostat.py3
[3] then generates a graphic from the logfile:
python plotiostat.py -f iostat.log -d sda -c rMB/s -c wMB/s
Finally, note that iostat provides figures at the block layer level. Performing an analysis at the application level is best done using tools such as iotop or atop, which display who is currently generating an I/O load at the process level in the style of the well-known program Top.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
TUXEDO Computers Unveils Linux Laptop Featuring AMD Ryzen CPU
This latest release is the first laptop to include the new CPU from Ryzen and Linux preinstalled.
-
XZ Gets the All-Clear
The back door xz vulnerability has been officially reverted for Fedora 40 and versions 38 and 39 were never affected.
-
Canonical Collaborates with Qualcomm on New Venture
This new joint effort is geared toward bringing Ubuntu and Ubuntu Core to Qualcomm-powered devices.
-
Kodi 21.0 Open-Source Entertainment Hub Released
After a year of development, the award-winning Kodi cross-platform, media center software is now available with many new additions and improvements.
-
Linux Usage Increases in Two Key Areas
If market share is your thing, you'll be happy to know that Linux is on the rise in two areas that, if they keep climbing, could have serious meaning for Linux's future.
-
Vulnerability Discovered in xz Libraries
An urgent alert for Fedora 40 has been posted and users should pay attention.
-
Canonical Bumps LTS Support to 12 years
If you're worried that your Ubuntu LTS release won't be supported long enough to last, Canonical has a surprise for you in the form of 12 years of security coverage.
-
Fedora 40 Beta Released Soon
With the official release of Fedora 40 coming in April, it's almost time to download the beta and see what's new.
-
New Pentesting Distribution to Compete with Kali Linux
SnoopGod is now available for your testing needs
-
Juno Computers Launches Another Linux Laptop
If you're looking for a powerhouse laptop that runs Ubuntu, the Juno Computers Neptune 17 v6 should be on your radar.