A study in detecting network intruders
Uninvited Guests
The nightmare of any admin is a user who can't resisting clicking on an unknown attachment labeled Application.exe. This article draws on a real-world example to show how you can use built-in Linux resources to detect unauthorized traffic that might have been invited in by a trigger-happy user.
If a network monitoring process detects malware, a system administrator needs to identify the affected systems and contain the damage. A customer from a Microsoft-heavy environment recently came to me with a problem. Many of the clients on his network were infected with malware. Because the malware was quite sophisticated, the virus scanner did not help detect it. He wanted my help with finding all the infected clients.
To make things even more exciting, several versions of the malware appeared on the network. The variations in form meant that it was not easy to detect the malware using simple pattern matching with a filesystem scan. Fortunately, although the attackers were good at infiltration, they were not very skilled at connecting back to their Command and Control (C&C) server. This article describes our investigation and offers some tips on how to respond to similar attacks.
The Malware
The first generation of the malware attempted to connect to Telnet servers in Asia via TCP port 23. Since the Telnet protocol is hardly ever used on today's networks, this attack was quickly noticed. A single tcpdump
[1] command directed at the gateway returned some initial hits:
tcpdump -i eth0 tcp port 23
The packets had some peculiarities in the fields of the Telnet header, which were revealed by an analysis with Wireshark [2]. Headers are typically used to negotiate things like terminal emulation or even X forwarding. In the case of the malware, the headers contained nonsense, but we always found the same 4-byte sequence at the beginning of the packet.
The second generation of the malware tried to disguise itself as a DNS packet. This time it used UDP port 53, but it wanted to contact servers on the same Asian network. The desired DNS traffic on the network moved back and forth between clients and the Active Directory domain controller, because the domain controller machine was acting as a DNS server. Undesirable traffic was filtered out using tcpdump
:
tcpdump -i eth0 port 53 and not host IP_address_of_AD_controller
tcpdump
detected the traffic that went to port 53 but not to the Active Directory server. The analysis with Wireshark also showed that the sequence already known from the Telnet packets reappeared at the beginning of the packets. In addition, the packets were again not valid DNS packets.
The last version of the malware finally tried HTTP port 80, but it was caught due to invalid HTTP packets. In addition, starting in version 2 of the malware, DNS queries had already been noted against different hosts in three different subdomains. The victim of the attack had several sites, so there were several Internet gateways.
The Search
Although most of the network was Microsoft, the gateways and DNS servers fortunately ran on Linux. I was thus quickly able to launch a search for clues.
Many companies allow their employees outgoing access to the Internet via the company gateway and do not log the permitted traffic at the firewall. This approach has some disadvantages: If the logs had been run through a log management solution such as ELK [3] or Splunk [4], it would have been very easy to narrow down the computers because traffic via ports 23 or 53 is fairly atypical.
If web access is first routed through an internal enterprise proxy, computers that do not use this proxy but look for a direct connection to the outside through ports 80 (for HTTP) or 443 (for HTTPS) are at least suspicious.
If the firewall also blocks outgoing connections that do not originate from computers with known services (proxy, DNS server, mail server, and so on), the analyst can simply search for rejected packets that use an address outside the destination. A perfect world should have hardly any packets of this type.
Clients that do send these packets are either configured incorrectly (depending on destination and port) or should be inspected more closely by the admin.
Wireshark: The Forensic Scientist's Friend
Wireshark [2] is the industry's standard tool for investigating network incidents from logged packets. The large number of protocol modules available with Wireshark – for application protocols such as HTTP or SMTP – allows for more in-depth analysis.
At the same time, Wireshark massively supports the admin in finding a needle in a haystack. The software makes it possible to set filters for each protocol field. Figure 1 shows a Wireshark display filter for all HTTP packets that use the POST
method and go to the target host 192.168.1.1
. Wireshark also offers statistical functions that can help with your analysis.
Security experts advise against operating Wireshark with root privileges. Previous examples of errors in software's packet analysis modules have allowed attackers to inject malicious packets into the traffic. See the box entitled "Avoiding Root" for more on running Wireshark without root privileges.
Avoiding Root
The best way to make use of Wireshark without assigning it root privilege is to first create a .pcap
file with tcpdump
and then evaluate it with Wireshark using an unprivileged user ID or a named pipe. Create a pipe with the command:
mkfifo /tmp/wirepipe
Then call Wireshark using:
wireshark -k -i /tmp/wirepipe
Now you can now feed packets into this pipe using the root account and evaluate them using non-root Wireshark. Use the following command, to which you can add a filter for packet types if needed:
tcpdump -w /tmp/wirepipe
Alternatively, you can use the dumpcap
command:
dumpcap -w /tmp/wirepipe
Dumpcap [5] understands some options. For instance, you can specify the interface from which the packets should come. The -f
option lets you specify a filter in tcpdump
syntax.
The Conversations menu entry is the first port of call when searching for infected systems. You may also be able to use this option to find suspicious endpoints outside your own network. If you know the IP address of a control server, in the ideal case, you can quickly find all systems that communicate with it.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
So Long Neofetch and Thanks for the Info
Today is a day that every Linux user who enjoys bragging about their system(s) will mourn, as Neofetch has come to an end.
-
Ubuntu 24.04 Comes with a “Flaw"
If you're thinking you might want to upgrade from your current Ubuntu release to the latest, there's something you might want to consider before doing so.
-
Canonical Releases Ubuntu 24.04
After a brief pause because of the XZ vulnerability, Ubuntu 24.04 is now available for install.
-
Linux Servers Targeted by Akira Ransomware
A group of bad actors who have already extorted $42 million have their sights set on the Linux platform.
-
TUXEDO Computers Unveils Linux Laptop Featuring AMD Ryzen CPU
This latest release is the first laptop to include the new CPU from Ryzen and Linux preinstalled.
-
XZ Gets the All-Clear
The back door xz vulnerability has been officially reverted for Fedora 40 and versions 38 and 39 were never affected.
-
Canonical Collaborates with Qualcomm on New Venture
This new joint effort is geared toward bringing Ubuntu and Ubuntu Core to Qualcomm-powered devices.
-
Kodi 21.0 Open-Source Entertainment Hub Released
After a year of development, the award-winning Kodi cross-platform, media center software is now available with many new additions and improvements.
-
Linux Usage Increases in Two Key Areas
If market share is your thing, you'll be happy to know that Linux is on the rise in two areas that, if they keep climbing, could have serious meaning for Linux's future.
-
Vulnerability Discovered in xz Libraries
An urgent alert for Fedora 40 has been posted and users should pay attention.