Small Supervisor

A system's logfiles not only record failed login attempts by users, but they also log program errors and information about attacks. Admins therefore should keep a continuous eye on them. Tools such as LOGalyze [1], Logcheck [2], Logwatch [3], MultiTail [4], and SwatchDog [5] can help you here.

Unlike with large monitoring solutions like Nagios and Icinga, the minor variants focus on analyzing logfiles. They use fewer resources and can be set up much more quickly. They are therefore ideally suited for use on weak hardware and embedded devices like the Raspberry Pi, as well as on servers with few selected services.

All candidates use one or more logfiles and filter out important messages according to predefined rules. As an option, they can send the result by email to the administrator or output it on the command line. Admins can also add their own filter rules, usually in the form of regular expressions. Sometimes the developer provides a set of oft-needed rules. Powerful tools may also put together a report about the state of the system and indicate security problems. However, a comparison of the above candidates shows that these functions are not a matter of course (Table 1).

LOGalyze

LOGalyze [1] comes from the Hungarian company Zuriel Ltd. The formerly proprietary tool may now be available under GPLv2, but the developers are still keeping the source code under lock and key. The latest version 4.1.4 was released in December 2016, but it only fixes minor bugs from the almost four-year-old previous version. LOGalyze therefore still relies on Oracle's Java runtime environment in the completely outdated version 1.6.

A short and concise guide in PDF format provides a description of the installation. Administrators can configure LOGalyze using a supplied web application that requires one of the application servers Apache Tomcat, Jetty, GlassFish, or JBoss. The hopelessly outdated Tomcat 6.0.35 from 2011 is included with the installation package; however, it can be quickly booted using a prepared script.

Initially, administrators can create one or more collectors in the user interface. These collectors retrieve the log data via the network or from a file. Admins can switch each collector on or off individually. LOGalyze then generates statistics and reports from the imported data and summarizes all critical errors, for example, in a concise report (Figure 1).

Figure 1: LOGalyze displays statistics directly in the web interface as a pie, bar, or line chart. Here it is clear that the number of events dropped in recent times.

Admins can also create their own statistics and reports by clicking the corresponding criteria in the user interface. LOGalyze then generates either a PDF or CSV file on this basis. Admins can either download these files or have them sent by email.

They can also search the logfiles for terms. LOGalyze may not allow regular expressions, but it does link several search terms using operators like AND and OR. It stores frequently required search queries to allow quick retrieval later via mouse click (Figure 2). LOGalyze provides plenty of predefined searches which, among other things, quickly list all the errors from the syslog.

Figure 2: LOGalyze has found three events for the ubuntu computer.

Logcheck

The Debian project currently looks after Logcheck [2], which is available under GPLv2. It independently assesses logfiles for problems, security vulnerabilities, and possible intrusion attempts. After it's started, Logcheck accesses the syslog and the auth.log by default. However, you can use the tool on other logfiles.

Logcheck compares all records added since the last test with a load of preset filtering rules. Depending on the result, the tool either moves directly to the next entry or classifies it as an important system event, security issue, or warning. Logcheck then sends all the events from the last three categories in an email to the administrator or writes them in the standard output (Figure 3).

Figure 3: Ubuntu users can start Logcheck with the help of the logcheck user. The -o parameter outputs the events in the terminal. The outputs can be quite confusing.

Administrators can choose between three filter levels to maintain an overview: The highest, called paranoid, is intended for particularly safety-relevant systems with a few selected services. In this filter level, Logcheck provides an especially large number of detailed messages that it would discard in the other levels. The Server level is the default, and there are the least messages in the Workstation level. Logcheck sends warnings about security issues and attacks in each filter level. To ensure that the tool only reports each system event once, it remembers the last position in the logfiles to be assessed with the help of the Perl script Logtail.

All filter rules are available as regular expressions, so that admins can add their own, as desired (Figure 4). To provide a better overview, all of the expressions for a service, such as the Apache web server, are moved to a separate configuration file. When started, Logcheck automatically imports all configuration files. The developers kindly provide a package with several of these configuration files. However, the rules contained in this Logcheck database only cover a few basic errors and particularly important attack patterns.

Figure 4: Among other things, in the Server filter level, Logcheck discards all the events that correspond to these regular expressions in the /etc/logcheck/ignore.d.server/sudo file.

Most distributions have Logcheck in their repositories. On Debian systems, a cron job initiates Logcheck every hour, and the attentive tool is automatically activated at every system startup. Any suitable command-line program assumes the responsibility for sending emails – the task is assigned to Sendmail by default.

Logwatch

Like Logcheck, Logwatch [3] is waiting to be installed in the repositories of most major distributions. The tool is available under the MIT license and requires Perl 5.8. Once Logwatch starts, it accesses all the logs known to it and checks all the events from the last 24 hours in them. Admins can extend or shorten this observation period at their discretion.

Unlike with the competitor Logcheck, Logwatch generates a concise summary (Figures 5 and 6) from the read events. A separate section is devoted to each of the services running on the system, and the information displayed there is based on the respective service. For example, Logwatch lists all the packages installed in the past 24 hours for the package manager dpjk. If Logwatch was unable to meaningfully interpret an event, the tool simply attaches this event to the report as an attachment.

Figure 5: Logwatch can also generate the summary in text form …

The administrator informs Logwatch about numerous configuration files, about which services are running on the system, and about in which logfiles the services usually store their information. In turn, the configuration files are distributed across several subdirectories. Logwatch provides finished configuration files for many important and well-known system services, and some distributors supplement them with additional services. Thanks to these specifications, Logwatch scours more logfiles immediately after the installation than its competitor Logcheck.

A specialized Perl script analyzes a service's logfiles. For example, the /usr/share/logwatch/scripts/services/dpkg script processes the logfiles of the package manager dpkg. If Logwatch is to analyze an individually compiled service for the administrator, it needs to be able to write an evaluation script in Perl. A detailed how-to included with Logwatch helps here.

The analysis scripts import the events from the logfiles, generate a summary, and pass it on to Logwatch. Ultimately, the tool sends the collected results in an email, writes them to a file, or delivers them via stdout. In the past, Logwatch presented all information on a simple HTML page upon request (Figure 6). The final report also contains some system information, for example, the available disk space.

Figure 6: … or as a HTML file with rather small font.

Admins can also dictate the detail of Logwatch's report. There are a total of 10 detail levels available. The individual evaluation scripts determine which (additional) information each level of detail produces. The distributions generally start Logwatch via cron job once at night and send the generated report via email. The tool delegates the actual sending to Sendmail or another, freely selectable command-line program.

MultiTail

MultiTail [4] presents the ends of several text files in only one (terminal) window (Figure 7). Additional features were added over the course of time, in particular filter and monitoring functions for logfiles.

Figure 7: Admins can determine the order of the logfiles themselves in MultiTail. They just need to depict two logfiles one above the other.

The admins can thus filter the flood of information using regular expressions. If a regular expression applies, MultiTail launches an external program upon request. You can be sent email notifications, for example. Alternatively, MultiTail works like a visual pipe in that it writes the filtered information in files or forwards it to other processes. MultiTail can even act as a syslog server itself and accept outputs from other programs upon request – such as netstat.

The tool also highlights the row concerned and attracts attention via a beeper. An admin can highlight individual events in specific colors using regular expressions. For example, you can highlight in red all rows starting with Error. MultiTail also automatically converts inputs. It converts IP addresses into the appropriate domain name, converts signal numbers into names, and supplies each date in the local format.

However, administrators need to set up MultiTail completely on their own. Unlike Logcheck or Logwatch, the tool does not have any pre-made regular expressions. Admins also need to manually configure the forwarding and sending of emails. MultiTail is included with all major distributions and is available under GPLv2. The tool also comes with a detailed manual in HTML format.

SwatchDog

The Simple Log Watcher, Swatch for short [5], began as a small watchdog tool that monitored a syslog for activities. The program, which is available under GPLv2, now digests any logfiles. Formally, the command-line tool is called SwatchDog to avoid any confusion with a well-known Swiss watch manufacturer. However, in most distributions, it is in the swatch package, and the man page is the only documentation.

The tool itself consists of a small Perl script that assesses the logfiles stated via parameter (Figure 8). SwatchDog either goes through all the rows contained in the files or continuously monitors the file. In the latter case, SwatchDog can be started as a daemon and thus move to the background. Upon request, the tool also accepts log data via a pipe.

Figure 8: SwatchDog checks the whole /var/logs/auth.log file once here based on the --examine parameter.

In any case, you can specify in a configuration file for which events the tool needs to perform which actions. This configuration file uses SwatchDog's own syntax – Listing 1 shows a simple example. According to the instructions shown there, SwatchDog needs to search for the keywords warning and error. Administrators can use a regular expression for specifying the search pattern.

SwatchDog then performs all the actions that follow in the indented list. For example, echo outputs the corresponding line from the logfile on the console, while mail sends the message by email with the subject line Error occurred to tim@example.com. SwatchDog also calls any programs (exec) and forwards the affected event via pipe (pipe command). Perl experts can store Perl code, which the tool executes.

Users start SwatchDog on the command line by default. Administrators need to create a suitable cron job or systemd units themselves. Unlike with Logwatch, SwatchDog does not provide an example configuration. Users should therefore initially plan a bit of time to write a suitable configuration file.