Root out rootkits
On the Trail
Detect rootkits, worms, and other malware with chkrootkit.
Sophisticated software can help you keep an eye out for malware, but by the time it's spotted, it's probably too late, costing you the time to repair and rebuild your system, as well as a loss of revenue if it's a production system. Chkrootkit is a proven, mature, and highly sophisticated tool that helps you detect a variety of malware in an ongoing process.
A True Story
A year or two after the turn of the millennium, I lost a server through the disastrous compromise of a popular web-based application. Thankfully, although the server was in use, there were no services on it that couldn't wait a few days before being restored.
The first clue that something was wrong was some strange filesystem behavior while logged in over SSH. I checked the Apache error logs, and – lo and behold – I found some output detailing a wget
transaction. The wget
file transfer was showing an illegitimate download from a nasty-looking URL.
Although the file name was unusual, what struck me was that all of the elements of the transaction were running under the apache
user (hence, the owner of the error.log
in question) and using the /var/log/apache
directory. This server was not without security, but after some forensic analysis once the server was taken off the network to avoid further local network damage, I discovered that a web application written in PHP had been compromised. PHP, of course, being run by the apache
user.
After gaining access via local user apache
, the miscreants had downloaded a rootkit without root permissions and managed to spawn a binary that gave them remote SSH access. From there, all hell broke loose.
First, when I accessed the server I noticed an unusual open port – not 22 or a port that I used for the SSH server (e.g., port 2222). Moreover, the /tmp
directory contained some slightly odd-looking temporary files. I suspect this is where the rootkit dropped its initial installation files after it had "broken free" of Apache's user, because /tmp
is usually writeable by all.
Second, when I ran a binary file, such as listing files in a directory, I noticed the server was a little clunky and unresponsive. It turns out the hacker had dropped alternative executables of system commands that fired up in their place.
Analysis later showed that at least 20 common binaries had been overwritten by a rootkit. These ranged from the simple ls
to more obscure commands. The nefarious attacker had put these alternatives in place to hide their presence on the system. Logs were also manipulated to show fewer details of the attacker's activities. However the attacker had either forgotten about Apache's error log or thought it was abstract enough not to be checked.
Seemingly simple but clever tricks, such as replacing the w
command, were employed. That command would normally offer some information on which users were logged on to the system and who was doing what. With the new version of the command, though, the logged-in hacker was essentially invisible until I dug deeper.
To avoid a full system rebuild, I downloaded a number of stock binaries from a source I trusted and overwrote those present on the filesystem. I then watched and waited patiently. Suddenly, I saw some network traffic to an IP address in Brazil and activity on the server that wasn't legitimate.
I repeated that procedure about three times, overwriting binaries again and again and adding firewall rules, but still something on the server was phoning home and letting the miscreants back in. Eventually I gave up, put a brand new disk in the box, and rebuilt the server from scratch. Incidentally, I would not have even tried dropping replacement binaries onto a production box; it is simply too risky. You've got to rip it up and start again if that ever happens on an important server.
One final, alarming, fact is that forensics proved that at midnight any login passwords (and users) were being covertly emailed back to an IP address in Brazil. That's not a good thing I'm sure that you'll agree.
In this article, I'll look at rootkits and chkrootkit
, a stalwart tool that helps identify machines that have been infected.
Roto-Rooter
In simple terms, rootkits essentially permit illegitimate access to a system against the owner's wishes. They come in all shapes and sizes and have most recently been discussed in relation to the UEFI (Unified Extensible Firmware Interface) "Secure Boot" functionality recently introduced. Secure Boot is meant to prevent the installation of rootkits into what used to be called the BIOS (Basic Input/Output System), so that your server or workstation isn't compromised before it has even booted up. To be clear, I'll be looking at operating system rootkits, as opposed to the boot-time equivalent.
One statement that probably needs to be reiterated is that if you lose a machine to a rootkit, you shouldn't risk your data for the sake of one hour's extra work; simply start again with a fresh build. The point I'm trying to make clear is that by the time you've spotted the evidence of a rootkit, it's probably too late. However, by using a clever tool to assist you, you at least can keep an eye on config mistakes that you make and potentially monitor unusual activity on your machines, even if they're not rootkit related. Whichever way you decide to use these sophisticated security tools, however, it should be a great learning experience. Now I'll look the venerable Chkrootkit utility.
Table 1 lists a sample of malware that the chkrootkit [1] tool can detect. This substantial list should be extremely worrying; just look at all the rootkit variants that have been discovered in the past. The Chkrootkit website also notes which tests are run to detect these kits (Table 2).
Table 1
Malware Currently Detectable by Chkrootkit.*
Rootkits |
|
AjaKit |
shv5 |
Ambient's rootkit (ARK) |
SK |
Anonoying rootkit |
Solaris rootkit |
Aquatica |
SucKIT |
Bobkit |
TRK |
dsc-rootkit |
t0rn (and variants); |
duarawkz |
t0rn v8.0 |
Ducoci rootkit |
Volc |
ESRK |
zaRwT |
FreeBSD rootkit |
ZK |
Fu rootkit |
|
George |
Worms |
Gold2 |
55808.A |
Hidrootkit |
Adore |
Illogic |
Lion |
Kenga3 |
LPD |
kenny-rk |
Lupper.Worm |
Linux Rootkit 64Bit |
Omega |
LOC rootkit |
Ramen |
LRK3, LRK4, LRK5, LRK6 (and variants) |
ShitC |
Madalin |
TC2 |
Maniac-RK |
Wormkit |
MithRa's rootkit |
x.c |
Monkit |
|
OpenBSD rk v1 |
LKMs |
Optickit |
Adore |
Pizdakit |
Enye |
rh[67]-shaper |
knark |
RK17 |
sebek |
Romanian rootkit |
|
Romanian rootkit |
Others |
rootedoor |
Operation Windigo |
RSHA |
OSX.RSPlug.A |
Shkit |
RST.b trojan |
Showtee |
Scalper |
shv4 |
Slapper A, B, C, and D |
*From the Chkrootkit website [1]. |
Table 2
Tests Run Against a System by Chkrootkit
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
LKMs (loadable kernel modules) are loaded dynamically into memory and interface directly with the kernel and its subsystems. When this type of access is successful, you have usually reached the point of no return. To see which kernel modules are in use, enter one of the following commands:
lsmod cat /proc/modules
The modules
file in the /proc
pseudo-filesystem contains all modules loaded into the kernel (Listing 1).
Listing 1
/proc/modules
The commands to query LKMs are useful if the rootkit in question is determined to include the source code of some nefarious kernel modules within its payload. Of course, if a compiler is not installed on your system, then the rootkit's code becomes a serious challenge to compile, and that's just what you want. It means the hacker would need to pull down precompiled binaries to your machine, which might not work correctly on your system for a number of reasons.
Take note that some distributions seem to insist that you leave a compiler like GCC installed. You can check for GCC's presence with:
whereis gcc make
Entering gcc -v
reveals the version. If safe to do so (i.e., nothing on your system requires that GCC remain there), you can remove the GCC compiler packages from a Debian distro with the following commands:
apt-get remove gcc apt-get remove build-essential
The build-essential
package drops about 20 tools onto your system that allow you to compile from source. I would never leave these packages installed on a production server for any period other than the five minutes every three months that I needed them.
Using Chkrootkit
To install Chkrootkit, you can use the usual method:
apt-get install chkrootkit
However, even from source [2], Chkrootkit is very simple to compile and execute the following:
make sense ./chkrootkit
After installing Chkrootkit using apt-get
on my Debian "Jessie" server, I ran it and received the output shown in Listing 2, abbreviated for space reasons.
Listing 2
Chkrootkit Output on a Systemd Server
Each rootkit shown in Table 1 is searched for by using a variety of techniques, such as checking for process names and other known footprints. If nothing is found, infected, or suspect, you're probably in good shape. Figure 1 shows the results you can get if you run chkrootkit
without the -x
(expert) or -q
(quiet) options.
If your system command binaries have not been overwritten, you will also see:
Checking 'grep'... not infected Checking 'su'... not infected Checking 'ifconfig'... not infected Checking 'killall'... not infected Checking 'top'... not infected Checking 'w'... not infected
Notice midway down Listing 2 that suspicious files were found. You might be a little surprised to learn that I'm not worried about these results. Quite commonly you will receive false positives from Chkrootkit. My guess is that these particular files are flagged because the .path
directories are hidden (i.e., they start with a dot).
If you're uncertain about suspicious output from Chkrootkit, you can use your favorite online search engine to investigate further or run Chkrootkit on a completely clean (offline) installation to confirm that the same alarms are triggered. If you check the bundled README.Debian
file for Chkrootkit, you receive some reassurance not to panic (Listing 3).
Listing 3
README.Debian File for Chkrootkit
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
SUSE Renames Several Products for Better Name Recognition
SUSE has been a very powerful player in the European market, but it knows it must branch out to gain serious traction. Will a name change do the trick?
-
ESET Discovers New Linux Malware
WolfsBane is an all-in-one malware that has hit the Linux operating system and includes a dropper, a launcher, and a backdoor.
-
New Linux Kernel Patch Allows Forcing a CPU Mitigation
Even when CPU mitigations can consume precious CPU cycles, it might not be a bad idea to allow users to enable them, even if your machine isn't vulnerable.
-
Red Hat Enterprise Linux 9.5 Released
Notify your friends, loved ones, and colleagues that the latest version of RHEL is available with plenty of enhancements.
-
Linux Sees Massive Performance Increase from a Single Line of Code
With one line of code, Intel was able to increase the performance of the Linux kernel by 4,000 percent.
-
Fedora KDE Approved as an Official Spin
If you prefer the Plasma desktop environment and the Fedora distribution, you're in luck because there's now an official spin that is listed on the same level as the Fedora Workstation edition.
-
New Steam Client Ups the Ante for Linux
The latest release from Steam has some pretty cool tricks up its sleeve.
-
Gnome OS Transitioning Toward a General-Purpose Distro
If you're looking for the perfectly vanilla take on the Gnome desktop, Gnome OS might be for you.
-
Fedora 41 Released with New Features
If you're a Fedora fan or just looking for a Linux distribution to help you migrate from Windows, Fedora 41 might be just the ticket.
-
AlmaLinux OS Kitten 10 Gives Power Users a Sneak Preview
If you're looking to kick the tires of AlmaLinux's upstream version, the developers have a purrfect solution.