Understanding System Services with lnav

Tutorials – lnav

Article from Issue 196/2017
Author(s):

See what's going on in the background of your Linux box by analyzing the logfiles.

Broadly speaking, there are two types of software that run on a machine: interactive software that the user launches and controls and background software that runs quietly doing what it needs to do. The interactive software is what you think you're using most often – it's the web browsers and word processors and all the other bits and pieces you launch when you want to do something. However, if you take a look at the processes on your computer (just type ps ax in a terminal window), you'll see that most of the software running isn't anything you launched, but the stuff that runs in the background quietly getting on with what it's doing. For the most part, you don't have to think about this – the software just works, and you get to focus on the interactive software and use the computer in the way you want; however, every once in a while, you have to delve a little deeper and see how this background software is running.

Because this software runs in the background, you should never see an error message, a progress bar, or any of the other user interface clues that tell you how interactive software is performing. Instead, it sends all the output to logfiles. These are text files that usually live in the /var/log directory (see the "Init Systems" box). These logfiles are designed to be readable by both humans and machines, so they follow a fixed text format, but one that's sufficiently verbose to understand. The main log on most Linux boxes is the system log, which is /var/log/syslog on Debian and Ubuntu-based machines and /var/log/messages on many others. This is a text file, so you could open it in a text editor. However, it can be very large, so it's usually best to just view the end of it. You can do this from a terminal by typing:

tail /var/log/syslog

Init Systems

When you turn on your computer, the boot system (either the BIOS or EFI) kickstarts a bootloader (e.g., GRUB) that loads the kernel into memory and launches the initialization system. In the past, this init system was a set of shell scripts that launched all the background processes and made sure everything was running as it should. It's the output from this operation that goes into the syslog. In the past few years, most Linux systems have started using systemd as the init system. One of the biggest complaints about systemd is that it doesn't log to text files, but rather to binary files that are difficult to read. That previous statement isn't completely accurate – in fact it doesn't automatically log to text files, but in most Linux distros that use systemd you'll still find everything that you're used to in /var/log.

You can still use lnav even if your distro exclusively uses the binary logging in systemd; you just have to use journalctl to convert the binary logs into text logs first. The most basic way to do this is to just pipe all the output into lnav with

journalctl | lnav

If you do find yourself using this method of getting data into lnav, you might want to take a look through the journalctl options, because you can select what output to send into lnav.

On CentOS and other Red Hat-based machines, you'll have to change syslog for messages. You'll see that each line corresponds to a single event that a piece of background software wants to record. For example, it might be the software that handles your network connection logging some information about the interface, or cron running a periodic command. Each line can be useful, but it can be hard to discern what's going on by just looking at a logfile. Useful information can be hidden by pages and pages of routine output. The simplest way of pulling information out of these logfiles is to use grep, which searches text and only outputs lines that match. For example, If you want to find all the lines that contain the text "cron," you can run:

grep cron /var/log/syslog

There's far more power here than I have space to cover, but things quickly gets complex, especially if you want to analyze dates or numbers, because that isn't easy to do in Linux text processing (see the "Regular Expressions" box).

Regular Expressions

If you're going to be looking into logfiles, then learning regular expressions (also known as regex) is an absolute must. These can be fiendishly complex, but they don't have to be – learning just a few characters can give you quite a lot of power.

Put simply, regexes are strings of text that define a pattern that you want to search for. In the main text, I used the text "cron" to search for those four characters. This is perhaps the most basic usage, as any letter matches itself. The full stop (or period for our readers across the pond) is a wildcard that matches any single character, so cr.n will also match cron, but not croon. The asterisk (*) character is a modifier that matches the preceding character zero or more times. It's most useful when combined with the wildcard so that it'll match any block of text (e.g., cron.*root will match any line that contains the word "cron" followed by the word "root" regardless of what's between them).

These basics will get you quite a long way, but if you want a more detailed understanding of regular expressions, you can read about them on the Linux Voice website [1].

A New Hope

Lnav is a tool that hopes to make it easier to extract useful information from logfiles by making it easy to import them into a SQLite-based data store and then perform advanced queries on them simply and efficiently (for larger setups, see the "Alternative Options" box). You may find the tool in your distro's repositories; otherwise, you can grab a statically linked binary that should run on any Linux system from the lnav website [2]. Unzip this file, and you can run this tool from the command line by navigating to the unzipped directory and entering ./lnav.

Alternative Options

Lnav is a great option for home computers and small servers. However, if you have a big setup, it can be a struggle because it's not really designed for handling huge amounts of data coming from multiple servers.

Logstash is an excellent open source log amalgamator that pulls together logs from many computers and pushes them into an ElasticSearch database that can then be viewed using a visualization front end, such as Kibana or Graphite. In many ways, this is the same approach that lnav takes, just at a high-performance level, and typically it requires a dedicated server just to analyze your other servers' logs.

A number of excellent closed-source options, such as Splunk, do a great job of showing what's going on if you're happy using proprietary software in the heart of your server setup.

While these are excellent options for large organizations, I'm not aware of any other system that works as well as lnav for small setups.

There's no setup or configuration for lnav, and it automatically detects most log formats, so you only need to point this tool at the logfile you want to investigate. For example:

lnav /var/log/syslog

If you're running the statically linked version that you've just downloaded, you'll need to include a path to the executable (e.g., prepend the above command with ./ if you're in the same directory as the file).

At first glance, it'll look just like you've opened the text file in a text editor, but unlike a general purpose text application, lnav understands a little about the logfiles, so it can help you find what's happening. For example, press Shift+W to jump back to the previous warning message and Shift+E to jump to the last error message (without the Shift, W and E move forward to the next message, but until you've moved back in the file, there won't be anything forward).

If you want to display only lines matching a particular pattern, you can do this with filters. First enter command mode by pressing the colon, and then enter filter-in cron to show only the lines containing the text "cron" (see Figure 1). Similarly, you can use filter-out to show everything except these lines. These filters will be applied to the whole file, so until you cancel them (with Ctrl+R), any other action takes place on the filtered log, not the log as a whole.

Figure 1: In addition to all the searching options, the colored highlighting in lnav makes logs easier to understand.

Perhaps the biggest advantage of lnav over traditional text processing methods for looking through logfiles is that it makes it easy to search by date. You can scroll through by pressing D and Shift+D to move forward and backward by 24 hours (see the "Log Rotation" box if you can't find the day's data you are looking for), whereas pressing the numbers 1 to 6 go to the next nth minute past the hour (with Shift held down, they go to the previous nth minute).

Log Rotation

Usually, programs continually add new lines to the bottom of logfiles as new issues come up that need reporting. Lnav will automatically add these new lines to your session, so you don't need to worry too much about this. However, it can explain why you might get slightly different results when you run queries at different times. Obviously, if programs keep writing data to logfiles, they would eventually fill up the hard drive, and the computer would fail to start (this happens to most sys admins at least once in their careers). To keep everything logged long enough to be useful, almost all Linux distros rotate the logfiles. The exact process varies a lot between different distros (and is often customized by sys admins on servers), but usually each logfile is moved at the end of the day, and a new logfile is started. A certain number of days worth of logfiles are kept, and old ones are deleted (older logfiles may or may not be compressed to save space).

If you have a look in /var/log/, you'll probably see that for each logfile, there are several archives (usually with a number in the file name to indicate how many days old that file is). You can bring all these into a single lnav session by using wildcard expansion. For example, to analyze all the syslog files, you can run:

lnav /var/log/syslog*

Lnav can read both plain text files and compressed files, so there's no need to decompress older files before running this command.

As well as navigating by time, you can also amalgamate data by time. Press I to enter histogram mode (see Figure 2). Here, you'll get a line for each time period with the length of the highlighting varying depending on how many log messages there were at the time and the color of the highlighting showing the number of info, warning, and logfiles. In this mode, Z zooms in and Shift+Z zooms out. (See Figure 3 for other lnav hot keys.)

Figure 2: The highlighting in Histogram mode makes it easy to see how much activity has been going on over the log's period.
Figure 3: Lnav has a lot of features, but they're well documented [3] if you are having trouble.

By far the most powerful part of lnav is its ability to query the logfile using SQL. Press the semicolon to enter query mode, and then you can analyze the logfile using SQL statements. Everything is held in a single table called logline, so you can get everything with:

;select * from logline

If you look carefully, you'll see that this doesn't return all the lines in the table. Since the text lines in a logfile don't all hold the same type of data, lnav can't put them all in the same SQLite table. For example, a cron line might include details of what is run and by which user, while a dhclient line might include details of what IP addresses are assigned to a particular network interface. Lnav gets around this by only matching lines that match the pattern of the currently selected line (i.e., the top line in the terminal window) when you run a command. Press P before entering query mode to see the available columns for this log type (see Figure 4).

Figure 4: After pressing P, you can see the details lnav has managed to glean from the current log line.

I don't have time to delve fully into SQL here, but many of you will already be familiar with this database language, and SQLite supports all the major features, including aggregation, which can be a really useful tool for building up a high-level overview of what's going on in your logfiles. If you don't know SQL, then you can still find out most useful information using filters and the histogram mode in lnav.

One of the best features of the Linux command line is the ability to pipe the output of one command into the input of another. A really simple example of this pipes the bottom 10 lines of a logfile into a search for the text "cron," which you can get by running the command:

tail /var/log/syslog | grep cron

Fortunately, you don't loose this ability when using lnav, and you can send output to other programs. There are two commands for this: pipe-to and pipe-line-to. The first sends all the lines that you've bookmarked, and the second sends just the current line. Before looking at how to use these, you first need to know how to bookmark lines. The basic commands for this are m (which marks the current line), Shift+M (which marks all the lines between the last marked line and the top of the screen), and Shift+J (which marks the next line). When you mark a line, it becomes highlighted in the user interface, so you can see at a glance which lines are selected. Once you've selected the lines you are interested in, you can enter command mode by pressing the colon. The pipe-to command can be followed by any Linux commands, and it will send the data to them. For example, if you want to know how many lines you've highlighted, you can pipe them to the wc (word count) command with the l flag:

pipe-to wc -l

The pipe-line-to command works in much the same way, but it only sends the current line, regardless of what is or isn't marked.

Once you've found the lines you're looking for, you may well want to extract them for future use. This could be as simple as keeping a copy of them to use in the future, or you might be writing a report about a particular event and need to document what's happened. Whatever the purpose, you can get data out of lnav in a couple of ways. One way is to extract the bookmarked lines to another text file using the append-to <filename> or write-to <filename> commands. Alternatively, you can output the results of an SQL query as either CSV or JSON with write-to-csv <filename> or write-to-json <filename>.

This article has been a whirlwind tour of the main features of lnav and should give you a valuable new tool for finding out what's going on deep in the heart of your Linux box. Hopefully you'll find that next time something goes wrong, it should be a little easier to figure out.

Infos

  1. "Regular Expressions" by Marco Fioretti, Linux Voice, issue 12, pg. 92, https://www.linuxvoice.com/issues/012/regex12.pdf
  2. Logfile navigator: https://lnav.org/downloads/
  3. Lnav documentation: http://lnav.readthedocs.io/en/latest/

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Ghost Tracker

    During a long trek through the verbose syslog, really important warnings and errors are scattered along the path. Sometimes a missing message can be the decisive event. Sys admin columnist Charly has now hired a tracker to help him search for clues: Log File Navigator.

  • Tool Tips

    Six Linux tools reviewed: FISH 2.0.0, BAD 0.0.2, BLINK, FPING 3.5, LNAV 0.5.0, TOMB 1.4

  • logrotate

    Take charge of your installation’s logfiles with logrotate.

  • Glogg

    Programmers and Linux administrators appreciate the benefits of event logs. The Glogg tool is the perfect choice for searching even large logfiles.

  • Command Line: Searching Logfiles

    If your hardware or software goes on strike, or the graphical system or your Internet connection fail, checking the logfiles can often be a big help. In this month’s column, we will look at the command-line tools that can help you scour the depths of these critical files.

comments powered by Disqus
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters

Support Our Work

Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.

Learn More

News