As an alternative to the Prowl solution for sending text messages described in a previous article [1], another provider in the colorful world of phone apps, Pushover, now – for a one-off payment of $5 – lets you distribute 7,500 messages a month for the rest of your life through a web API to either iOS, Android, or desktop clients.

Rough and Ready Browser

On iOS or Android, the user logs in to the Pushover app, which then displays incoming messages as push notifications (Figure 1), even if the phone isn't being used and displays the lock screen. Additionally, Pushover offers native desktop clients for the Mac and a somewhat hacky browser solution for the Linux desktop.

Figure 1: The cell phone with the installed Pushover app shows a login attempt on the SSH server of the monitored Linux computer.

To install the desktop client in Chrome or Firefox, you go to the Pushover login page [2], enter your email address and password for your Pushover account, and then allow the browser to output notifications on your desktop. This only works while the browser is running and one tab is pointing to the Pushover website (Figure 2). Because I already do this for Gmail and Evernote on my home system, with the help of pinned tabs, an extra tab does not really matter.

Figure 2: A browser tab for the Pushover service enables pop-ups on the Linux desktop.

Tracking

Following the latest entries in a logfile like auth.log in /var/logs is not as easy as you might think. Even implementing a Unix function like tail -f (Figure 3), which every admin is likely to use several times a day, requires knowledge of the system seek() function, which you can use to advance the read cursor associated with a file handle to the end of a file.

Figure 3: Running tail -f against the /var/log/auth.log file shows successful and failed login attempts against the SSH daemon.

If read() fails to return any data, you have most likely reached the end of the file. But if additional lines do appear afterward, appended by another process in the meantime, then tail -f outputs the content. Even if the admin renames the file, the data-consuming process and the open file handle remain the same, and the reading program won't even notice.

But even tail -f can trip up if the distribution's logfile rotator steps in to shift the old file out of the way, then compresses the file and replaces it with a fresh, empty file. In this case, it would be fatal to keep tracking the open file handle with read(), because the fresh data would now undoubtedly be written to a completely different file.

Processes trailing logs can cater for this by periodically checking whether the file under the specified name still uses the same inode on the filesystem. If stat() shows that the inode has changed, the log analyzer needs to close the open file handle and open a new one on the new file with the same name.

Prefabricated

Luckily, no one actually needs to convert this logic into program code nowadays because several open source implementations already do the job perfectly. Python has pygtail [3], for example, which is supposedly a Python port of the widespread logcheck utility. If you do not want to write your own Python program and you can do without customizing the code to your needs, you can also use logcheck directly to parse the system logfile as the first leg in the alarm pipeline.

To install the Pygtail module for Python 3.x, use:

pip3 install pygtail

Listing 1 [4] imports the Pygtail module, and line 9 composes a path for the flag file required by Pygtail in the data directory below the user's home directory; in the case at hand, this is data/authwatch.auth.log.offset. This is where Pygtail stores the byte count of how far it got in the file; the next call carries on reading behind the offset. It will also output any new data, if available, or otherwise keep quiet.

Cron later calls the script at five-minute intervals and immediately says goodbye after its work is done, so it needs this persistent position marker in the offset file. The admin only has to create the ~/data directory once manually before using the script if it does not already exist.

Listing 1 also filters out regular events, such as entries in which keywords like CRON or Connection closed occur. Lines 15 and 16 use regular expressions, courtesy of the imported standard module re, to search for these entries.

If you have a flavor of Linux that relies on the much maligned systemd, you will not find an auth.log file, but you can use journalctl at the command line or the systemd Python bindings and their journal method to find the newest entries in the system log.

Instead of an offset into a file, the script then stores the timestamp of the last query in an extra file and jumps just beyond it for a call to seek_realtime(), to avoid reporting duplicates. In this case, the script does not need to worry about rotated logfiles because systemd abstracts such implementation details.