Setting the Records Straight

When you are new to running servers and rightly worried about keeping downtime to a minimum, you inevitably find yourself facing a few problems – some predictable and others not so much. A particular aspect of systems that repeatedly causes consternation is logging and its concomitant storage.

To my mind, logging is one of the truly operational aspects of being a sys admin. Even with the best automation in the world, to get system logging right, you need to invest a little forethought along with very occasional housekeeping. After attempting to diagnose a few system problems, you soon come to realize that logs are absolutely essential. I'll reiterate that last statement by saying that they are not just important, they are truly key to keeping your servers functioning correctly.

Logfiles come in all different shapes and sizes and hold all kinds of information, both useful and relatively useless. Some logs are inconsequential if lost, whereas others are so critical to the operation of a server that their presence can mean the difference between a full server rebuild (sometimes adding several days of additional work to your busy week) or a five-minute health check.

With experience, you develop a level of trust in what your logs deliver and a degree of comfort in the level of detail you need to answer a number of questions relating to the kinds of problems you are often called on to investigate. How important is getting that level of detail correct? A few years ago, my least-favorite dialog box (on a certain popular GUI-based operating system) inconveniently popped up and reported the following: There has been an error.

No matter which way you squint, that dialog box isn't going to help much in solving what might be a life or death dilemma – in the sense of ongoing employment at least. The message I'm attempting to convey is: Logs are not only critical to the operation of your server, but they are also an aspect of your systems that needs more than just a flying blind, brief amount of consideration. Without a high level of logging detail, you are inevitably caught out unwittingly when presented with a large number of red herrings (i.e., clues) that lead you down the wrong path in your diagnosis.

Conversely, with a massive amount of detail your disks fill up so quickly that your remote server, at best, complains endlessly when trying to run its applications or, at worst, simply stops working, requiring expensive onsite engineers with hands-on access to recover it.

Doom-mongering aside, a Unix-flavored package called logrotate is so powerful that it swiftly takes care of many of the problems commonly associated with logging and storage without breaking a sweat. That said, it is still important to keep an eye on how your logs are behaving periodically, even with such a sophisticated tool in your toolkit.

Pause for Breath

Imagine a software package with functionality that lets you specify lots of well-considered parameters, including whether to send email after logfiles have reached a certain number, whether to compress logfiles to save disk space, and whether logs should be ignored or highlighted if they have no content (i.e., zero-byte files). The most important of all logrotate's features, however, is revealed by its name: The simple functionality of rotating logs is key to keeping your servers online.

Any additional bells and whistles won't be as useful to you as the simple act of purging old logfiles that are simply not useful any more to reduce disk space usage. Logrotate is so flexible that it can manage all of the above-mentioned tasks with the logfiles of almost any software package, making it a fantastic weapon in your sys admin arsenal.

The end result is peace of mind that your disks will never fill up, causing poor server performance. Coupled with the right level of logging (which is usually set within the configuration parameters of your applications and not within logrotate), you can, for all intents and purposes, strike logging off your daily housekeeping to-do list and relegate it to your monthly to-do list instead.

En Suite

To say the superhero that is logrotate is feature-filled is certainly no understatement. For a number of years now, logrotate has been so warmly embraced by the Linux community that many packages in the repositories ship conveniently with a ready-made logrotate profile that is dropped dutifully into the correct directory, with the path /etc/logrotate.d.

The result is that, give or take some very infrequent tweaks, everything related to pruning and purging your critical logs is taken care of automatically for you. Additionally, I have seen some less mature packages include an optional profile, even if it's not dropped into place when the package is installed.

However, until you delve into the multiple configuration options, you might not realize exactly what you should be doing with your logs. Now that I've established why it's so important to keep logs ticking over correctly to prevent disaster, I'll run through some basic logging scenarios, as well as a few that might not have occurred to you.

Status Quo

My systems run my preferred flavor of Linux – namely, Debian on my servers and Ubuntu on my desktops. However, like so many Linux packages, logrotate is probably nearly identical to use across all the popular distros, barring a few file path changes. Logrotate likely already exists on your system, but you might want to check at this stage with:

# apt-get install logrotate

Before forging ahead, I'll briefly mention where to check the innards of the package. A "state" file remembers who did what and when, in terms of when it last checked, changed, or rotated a log. That file is /var/lib/logrotate/status (Figure 1), which stores the last rotation recorded for each logfile by date.

Figure 1: The logrotate state file showing dates recorded for each logfile's last rotation.

Also, you don't have to worry about triggering logrotate, it simply runs as a cron job, sitting neatly inside the /etc/cron.daily/ directory. Thanks to cron's portability, then, you can simply move the 14 lines or so of configuration to another cron file and run it whenever you want.

The easiest way to dip your toe into the water with logrotate is to start looking at some of the bundled config files, which demonstrate how much of the hard work is already done for you by this well-written software package.