Monitoring with the Sysstat tool collection

Ins and Outs

If you have kernel 2.6.20 or newer, with the CONFIG_TASK_IO_ACCOUNTING option built in, you can also use pidstat to monitor the I/O load for individual processes or whole groups. On a system with the right support, typing

pidstat -d 2 3

will generate output as shown in Figure 3. Pidstat will list the processes generating I/O and even tell you the type (read or write), along with the data throughput (in Kbps).

To allow this to happen, you need a recent kernel, the packages from the Sysstat website, and a matching cron configuration.

Listing 1 shows the /etc/cron.d/sysstat file on Ubuntu. In the simplest of all cases, the following two lines in /etc/crontab are all you need:

*/10 * * * * root /usr/lib/sa/sa1 1 1
53 23 * * * root /usr/lib/sa/sa2 -A

The first entry starts sadc (the System Activity Data Collector), the server daemon that collects data every second and stores its findings under /var/log/sa. The logfile format is saXX, where XX is the date. The second part with sar -A just rotates the logfiles.

Listing 1

Cronjobs for Sysstat

01 # Global variables:
02 #
03 # our configuration file
04 DEFAULT=/etc/default/sysstat
05 #  default setting, overriden in the above file
06 ENABLED=false
07 SA1_OPTIONS=""
08 # Activity reports every 10 minutes everyday
09 5-55/10 * * * * root [ -x /usr/lib/sysstat/sa1 ] && { [ -r "$DEFAULT" ] && . "$DEFAULT" ;
10 [ "$ENABLED" = "true" ] && exec /usr/lib/sysstat/sa1 $SA1_OPTIONS 1 1 ; }
11 # Additional run at 23:59 to rotate the statistics file
12 59 23 * * * root [ -x /usr/lib/sysstat/sa1 ] && { [ -r "$DEFAULT" ] && . "$DEFAULT" ; [
13 "$ENABLED" = "true" ] && exec /usr/lib/sysstat/sa1 $SA1_OPTIONS 60 2 ; }

Iostat and Mpstat

Besides sar and pidstat, the Sysstat package also includes iostat and mpstat. Mpstat monitors the processor you pass in to it; iostat is popular with file, database, and mail server administrators as a tool for measuring system I/O load by drive, disk, partition, and filesystem.

For example,

iostat -m -n 10 2

monitors an NFS mount point and outputs the volume of data in megabytes (Listing 2). For a single partition, you can do the same by typing iostat -m -x sda2 10 2, as shown in Listing 3.

Listing 2

NFS Mount Point Activity

01 # iostat -m -n 10 2
02 Filesystem:  rMB_nor/s wMB_nor/s  rMB_dir/s  wMB_dir/s  rMB_svr/s  wMB_svr/s  rops/s  wops/s
03 server:/home      0,03      0,01       0,00       0,00       0,01       0,02    6,53      6,53
04
05 Filesystem:  rMB_nor/s wMB_nor/s  rMB_dir/s  wMB_dir/s  rMB_svr/s  wMB_svr/s  rops/s  wops/s
06 server:/home      6,65      6,64       0,00       0,00       6,65       0,81  252,40  252,40
07 #

Listing 3

I/O for a Partition

01 # iostat -m -x sda2 10 2
02 avg-cpu:   %user    %nice  %system %iowait  %steal  %idle
03             2,01     0,01     0,52    0,40    0,00  97,05
04
05 Device:   rrqm/s   wrqm/s      r/s     w/s   rMB/s  wMB/s  avgrq-sz  avgqu-sz  await  svctm  %util
06 sda2        0,71     3,32     1,09    0,85    0,02   0,02     38,13      0,06  28,37   2,87   0,56
07
08 avg-cpu:   %user    %nice  %system %iowait  %steal  %idle
09             6,14     0,00     0,81    0,33    0,00  92,71
10
11 Device:   rrqm/s   wrqm/s     r/s      w/s   rMB/s  wMB/s  avgrq-sz  avgqu-sz  await  svctm  %util
12 sda2        0,00    12,80     0,00    7,40    0,00   0,08     21,95      0,03   4,27   1,14   0,84
13 #

Absolutely Essential

The latest version of the Sysstat package includes some new features that many administrators have been waiting for – above all, the ability to monitor I/O load per process. Linux had some catching up to do in this regard. It is a pity you can't use this feature on older servers or on enterprise distributions. It is questionable whether distributions will respond with matching kernels and packages in future releases.

If you can do without detailed I/O statistics, a combination of sar and kSar will give you a flexible and comprehensive interface that lets you monitor multiple servers at the same time.

Even if you don't have a full-fledged monitoring system, you will still be able to identify recurring bottlenecks and take steps to fix any performance issues before a system crash.

Read full article as PDF:

062-064_systat.pdf  (570.81 kB)

Related content

  • Charly's Column Bootchart

    What could be more boring than watching a computer run through an error-free boot routine? Why watch for messages that may never appear? Every second you save adds to your valuable leisure time. Bootchart helps you find those valuable leisure moments.

  • System Diagnosis Tools

    To check on the health of a Linux system, administrators can turn to vmstat, iostat, netstat, and ifstat. Or, you can just use the versatile dstat, which combines the features of several tools in a single package.

  • Orca

    Monitor and troubleshoot Linux system performance with the free and powerful Orca.

  • Parallel Bash

    You don't need a heavy numeric mystery to benefit from the wonders of parallel processing. This article describes some simple techniques for parallelizing everyday bash scripts.

  • Top Ten Tops

    The famous Unix admin utility known as Top has many imitators. We take a look at some of the top Top tools.

comments powered by Disqus

Direct Download

Read full article as PDF:

062-064_systat.pdf  (570.81 kB)

News