System Rescue

System rescue – it's definitely an important topic with lots of considerations. Do you go with "bare-metal restore" or just back up the data and all the configs? What about your database? Do you snapshot it, or replicate it and keep a transaction log? What about all the new NoSQL things? More to the point, how do you know when you need to do a system rescue?

Sometimes it's pretty obvious, like when some water spilled onto one of my machines; I stared in horror as the machine made a loud "pop" and the power supply killed the motherboard and then itself. Luckily, I didn't lose any data. Sometimes, however, it's not so clear when you have to do a system rescue. For example, if you find a corrupted file on your system, do you have other corrupted files? Short of opening them all and checking them, you don't know whether you have just one bad file or a completely corrupted filesystem.

File Integrity to the Rescue

Such problems have plagued administrators, well, since computers have had read/write data storage. The good news is that several mature tools can help you address the problems of managing files and ensuring that they are not modified or corrupted. Certain strategies are also helpful when designing and architecting systems to make things more robust. Ultimately, the goal is to prevent data corruption or improper modification as much as possible – by using file permissions, robust filesystems with journaling, and so on. Then, you need to ensure that you can detect file corruption and improper modification and, finally, restore things to a known good state. The two main tools for these tasks are Open Source Tripwire [1] and AIDE [2]. Neither has undergone major changes for a few years, mostly because they are fairly feature complete.

Tripwire

Tripwire, first written in 1992, is the granddaddy of file integrity tools. It quickly became popular and was eventually taken commercial, with an open source version remaining available. Open Source Tripwire hasn't undergone an update since late 2011. As I mentioned, it's pretty feature complete – except for hashing algorithms: Open Source Tripwire supports CRC-32 (trivial for an attacker to bypass), HAVAL (weaknesses were found as far back as 2004, so it's probably not a good choice), MD5, and SHA (both of which are showing their age).

Basically Open Source Tripwire doesn't support any modern hashing algorithms (e.g., SHA256 or SHA512). Although MD5 and SHA are hard to break, the skills of attackers keep improving, and it's unlikely that Open Source Tripwire will ever get support for modern hashing algorithms. It also seems to lack support for checking extended file attributes (xattr). Although it can check the basic file permissions (user, group, other), it can't check xattrs, meaning attackers can potentially add themselves to a file or directory and remain undetected. As such, if you have strong security requirements, you should probably consider moving away from Open Source Tripwire. Commercial versions of Tripwire are available, but I've never tried them because I'm not a big fan of closed source security.

AIDE

Luckily, you have a second option, AIDE. AIDE was created as a replacement for Tripwire and has had somewhat more active development. AIDE does support modern hashing algorithms such as SHA256 and SHA512, so the chances of an attacker modifying a file and managing to keep the hash the same on it are pretty nonexistent at this time (and probably for the next 10-20 years). AIDE also supports extended attributes, which is pretty important, because most Linux distributions now default to filesystems like ext4, XFS, and Btrfs, all of which support xattr by default.

Open Source Tripwire and AIDE operate in largely the same manner. You configure them to check certain files and directories, and they create a database of the file and directory permissions, ownership, size, access and modification times, a hash value of the data (if it's a file), and so on. You then run these tools periodically, and they recheck all the files to see whether anything has changed. If it has, the changes are logged, and you can configure the tools to email you a report.

I won't go into installation, because the tools are available as packages for virtually every distribution. Also, I won't cover configuration, because they have pretty solid default policies. I will, however, discuss where things can go horribly wrong and how to prevent that.