Mix AND Match

As you might now expect, you can combine several of these powerful commands to offer a more granular output to limit your level of detail:

# lsof -i -a -u chris

The magical -a switch stands for AND, so this command should list all open ports for user chris. In this case, all I see is a browser connecting over TCP ports and a daemon called mdns running on UDP port 5353. Try it yourself.

Forget about slowing down the output with a clumsy grep, which then needs to be appended via a pipe as a suffix. By simply typing

# lsof /etc/*

you can see activity with open files mentioning the /etc directory.

Speaking of directories, you can even target a directory and its subdirectories specifically with the +D switch, or if you don't want to include subdirectories, you can turn off that functionality by using the +d option:

# lsof +D /var/log
# lsof +d /usr/local

Say you wanted to drill down into which process opened a particular file with a specific path. You could efficiently use the -t switch as follows:

# lsof -t /var/log/auth.log

Two more useful network options for lsof list TCP and UDP connections on all ports:

# lsof -i tcp
# lsof -i udp

To exclude any file opened by a process owned by the user daemon (e.g., a process with lots of output), you can enter:

# lsof -u ^daemon

For those of you who have used the watch command to check out what another command is doing, you might be pleasantly surprised to discover that lsof offers that functionality. The output is refreshed with the infinitely useful -r parameter:

# lsof -r5 -c avahi-daemon -a -i UDP

The line of equals signs (Figure 11) indicates each refresh.

Figure 11: The delimiting lines separate the five-second refreshes.

Debian Goodies

As I promised, I have a treat for Debian and Ubuntu users that hardly anyone I have encountered in sys admin circles has heard of. The secretive little package to which I'm referring is somewhat surprisingly called debian-goodies. As incongruous as the package name might sound, be assured that I'm entirely serious. You can install what is officially described as "small toolbox-style utilities for Debian systems" with the following command:

# apt-get install debian-goodies

The scope of sys admin fun, … er, increase in productivity, provided by these additional weapons is for another day. For now, I'll look at a single tool, checkrestart, which on its own is exceptionally useful.

Before I proceed, be warned that the output from checkrestart should not be used to make life and death decisions. In other words, every now and again information may change nanoseconds after the command is run; therefore, in such rare cases there's a minuscule chance that what you see is not what you get.

Now forewarned, step up to the lifesaver utility that is checkrestart. Sitting in the same package with commands such as dgrep, dzgrep, and debget, the powerful checkrestart is entirely based on lsof.

The checkrestart raison d'être is to probe libraries still in use by packages after an upgrade has been performed. Consider, for example, that you have a mail server that uses TLS encryption for some of its more secure connections and apache2 running with an SSL certificate or two installed.

If you then run a command to update all your repository information followed by a forced yes to upgrade any package that needs updating,

# aptitude update
# aptitude full-upgrade -y

you see in the resulting output that an OpenSSL upgrade is applied to your system automatically. You're aware that your OpenSSL upgrade affects the way you control your server remotely (i.e., via SSH), but you might have forgotten about your aforementioned mail server and web server.

To check for any packages that you might have overlooked restarting to effect the recent changes, simply run:

# checkrestart

The output is nice and clean and gives you some simple advice about files it has identified inside the main directory (/etc/init.d) used for starting and stopping daemons (Listing 4).

On newer systems, it also includes useful advice on how to start and stop daemons without the traditional /etc/init.d/daemon restart format; something along the lines of:

# service ssh restart

I can't tell you how many times I've performed package upgrades and forgotten – or more commonly not known about – a dependency in use by a service. Once you've restarted a service and checked that it has come up cleanly, you can be safe in the knowledge that you're running the latest security update and that you have not unwittingly performed a partial upgrade, with old libraries still in use.

Checkrestart also lists processes for services without a startup script (Listing  5). Moreover, the -p switch lists deleted files that belong to a package and kindly ignores deleted files that do not flag a package within the package manager:

# checkrestart -p

The other caveat worth mentioning is that certain upgrades, such as kernel upgrades, should generally be treated as the exception and almost always require a system reboot; however, checkrestart has undoubtedly saved me many a server reboot over the years.

With confidence, I can state that the information checkrestart offers helps me discover more about my servers and increases my knowledge about how packages interact and ultimately are set up to work on my systems. For example, I'm always forgetting that one package in particular pulls in an OpenSSL library and needs to be restarted after an upgrade.

If you're a Debian-based Linux user, I would highly recommend a quick peek at debian-goodies; it installs into a few hundred kilobytes and can be safely removed afterward if you're not going to use it in the future.

The End

Who would have thought that extra peace of mind could be achieved by simply listing open files on a system? The functionality of lsof takes many users by surprise. It's veritably brimming with features that make some system utilities pale in significance. Combined with other tools relevant to a particular task, it's a fantastic addition to any toolbox.

The lsof utility is surprisingly versatile and fast and outputs thoughtfully formatted information traversing several tricky aspects of a system. If you haven't used it before, I hope the brief insight given within this article will encourage more investigation.

Errors in the text and Table 2 of Martin Steigerwald's "Real-Time Monitoring Tools" article in issue 167 (pg. 62) were brought to our attention. We offer the corrections (highlighted) here.