Get deeper insights into your system with eBPF
Keen Observer
Use the eBPF in-kernel virtual machine to identify resource bottlenecks and optimize your installation.
eBPF [1] is a relatively new addition to the Linux kernel that takes over more monitoring, security, and networking duties from individual kernel modules. Originally called the Berkeley Packet Filter, BPF came to life in 1992 [2] in order to provide a better and optimized mechanism to filter packets.
BPF was first used as an HTTP packet filter in BSD. Several decades later, it was completely rehashed and took on new tasks. The new version of BPF is what is known as enhanced BPF or eBPF. In addition to various new features, eBPF also has a new mechanism to connect to the Linux kernel. Instead of just redirecting packets, eBPF can attach itself to any kernel event or any socket. eBPF is tightly integrated with the Linux kernel and can be used as an efficient mechanism for Linux tracing. You can also use eBPF behind the scenes on your Linux machines to discover performance issues and bottlenecks.
Get Started
eBPF requires a kernel newer than v4.4 and one that has been compiled with the CONFIG_BPF_SYSCALL option. Neither of these requirements should be a problem if you are using one of the mainstream distributions like Ubuntu and updating it regularly.
To experience the tracing benefits of ePBF, install the tools from BPF Compiler Collection commonly referred to as bcc [3]. Fire up a terminal in an Ubuntu installation and type:
$ sudo apt install bpfcc-tools linux-headers-$(uname -r)
This command will fetch the tools as well as the kernel headers for the kernel version that you are currently using. The bcc includes over 70 tools, and under Ubuntu, they are all installed in the /usr/sbin
directory and will have a -bpfcc
extension (Figure 1). The tools are, in fact, Python scripts that you can edit and modify as per your requirements, provided that you know what you're doing. If you are using another distro, refer to the bcc documentation [4] for distribution-specific installation instructions.
Keep a Close Watch
I'll start by showing how to keep an eye out for new processes using the execsnoop tool. This tool is especially useful for tracing processes that are short-lived, which is to say those processes that end before you can track them via the traditional process monitoring tools like top. Keeping an eye out for these processes that usually miss your attention might help you optimize your installation.
Fire up a terminal and type:
$ sudo execsnoop-bpfcc
The tool will now keep an eye out for new processes. To give it something to pick up, perform some action (Figure 2), such as firing up a new terminal. This action will produce the output shown in Listing 1.
Listing 1
execsnoop Output
As you can see from this truncated output, execsnoop prints one line of output for each new process. The output shows the parent process or command name under the PCOMM column, the PID of the process along with its parent PID, and the return value of the exec()
function under the RET column, as well as the command and any arguments under the ARGS column.
The -x
option can be used to include failed execution (Listing 2).
Listing 2
the execsnoop -x Option
You can similarly use the -t
option to include a timestamp column and the -n
option to match on a name. For instance, sudo execsnoop-bpcc -n ssh
will only catch processes where the command matches the specified name (in this case, ssh
).
A related tool is opensnoop, which enables you to trace file opens. The open()
system call brings up files for read and write operations, and keeping an eye on this process can reveal a lot of details about how a program works behind the scenes. It can, for instance, help you identify all the data files, config files, and log files that are associated with an app and the manner in which they are accessed by the app. If an application performs poorly, it could be because it is improperly configured and is wasting milliseconds trying to access files that don't exist.
The opensnoop tool traces the open()
system calls and prints a line for each call that's found, as shown in Listing 3.
Listing 3
Tracing open() Calls
The first column is the process ID of the process that invoked the open()
system call. The second column, COMM, displays the name of the process, and the third column displays the file descriptor as it was returned by open()
. Then comes the error value if any error code is returned by open()
, and the final column specifies the full path of the file used in the open()
system call.
On an active Linux installation, the output of opensnoop will be very difficult to follow. It'll be helpful if you filter the output with grep to make sure you catch hold of what you're looking for. Another good idea is to filter using the process ID of the process that is of interest to you with the -p
option or the -n
option to filter on process name, such as:
sudo opensnoop-bpfcc -n gnome-shell
Network Inspector
Another useful eBPF tool is tcpconnect
, which enables you to inspect active TCP connections by watching all connect()
system calls. The tcpconnect
tool prints all TCP connections, along with their source and destination addresses (Listing 4).
Listing 4
tcpconnect
The first and second column as usual list the process ID and the name of the process that is called connect(). The third column specifies whether you're using IPv4 or IPv6. The fourth and fifth columns are the source IP and destination IP for the connections, and the last column is the port number at the destination address.
One common option is -U
, which appends the UID of the processes to the output. You can then use it along with the -u
option to filter the output using UID, such as:
sudo tcpconnect-bpfcc -Uu 1000
Another tool that is useful for debugging TCP processes is tcpaccept (Figure 3), which traces the accept()
system call. The overhead of tcpaccept is negligible, and although netstat can do the job of both tcpconnect and tcpaccept, the two eBPF tools are more versatile. You can even use the -p
option with both the tools to watch specific processes, such as:
sudo tcpaccept-bpfcc -p 14786
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Canonical Bumps LTS Support to 12 years
If you're worried that your Ubuntu LTS release won't be supported long enough to last, Canonical has a surprise for you in the form of 12 years of security coverage.
-
Fedora 40 Beta Released Soon
With the official release of Fedora 40 coming in April, it's almost time to download the beta and see what's new.
-
New Pentesting Distribution to Compete with Kali Linux
SnoopGod is now available for your testing needs
-
Juno Computers Launches Another Linux Laptop
If you're looking for a powerhouse laptop that runs Ubuntu, the Juno Computers Neptune 17 v6 should be on your radar.
-
ZorinOS 17.1 Released, Includes Improved Windows App Support
If you need or desire to run Windows applications on Linux, there's one distribution intent on making that easier for you and its new release further improves that feature.
-
Linux Market Share Surpasses 4% for the First Time
Look out Windows and macOS, Linux is on the rise and has even topped ChromeOS to become the fourth most widely used OS around the globe.
-
KDE’s Plasma 6 Officially Available
KDE’s Plasma 6.0 "Megarelease" has happened, and it's brimming with new features, polish, and performance.
-
Latest Version of Tails Unleashed
Tails 6.0 is based on Debian 12 and includes GNOME 43.
-
KDE Announces New Slimbook V with Plenty of Power and KDE’s Plasma 6
If you're a fan of KDE Plasma, you'll be thrilled to hear they've announced a new Slimbook with an AMD CPU and the latest version of KDE Plasma desktop.
-
Monthly Sponsorship Includes Early Access to elementary OS 8
If you want to get a glimpse of what's in the pipeline for elementary OS 8, just set up a monthly sponsorship to help fund its continued existence.