Strace [1] is a valuable tool for monitoring and troubleshooting system calls. Unfortunately, it is sometimes difficult to understand raw system calls emitted by strace. For instance, the command shown in Listing 1 reveals lots of cryptic information.

mike@ldap:~$ strace touch
execve("/usr/bin/touch", ["touch"], 0x7fff2a3fea40 /* 41 vars */) = 0
brk(NULL)                               = 0x55c41812f000
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f0d06525000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
newfstatat(3, "", {st_mode=S_IFREG|0644, st_size=73439, ...}, AT_EMPTY_PATH) = 0
09 mmap(NULL, 73439, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f0d06513000
close(3)                                = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\20t\2\0\0\0\0\0"..., 832) = 832
pread64(3, "\6\0\0\0\4\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0@\0\0\0\0\0\0\0"..., 784, 64) = 784
newfstatat(3, "", {st_mode=S_IFREG|0755, st_size=1922136, ...}, AT_EMPTY_PATH) = 0

Falco [2] is an alternative tool that offers a more intuitive approach for monitoring and detecting system events. Falco is maintained by the Cloud Native Computing Foundation (CNFC) [3] and is designed to operate in distributed, containerized environments. However, you can also use Falco on a single Linux system.

Overview of Linux System Calls

In Linux, userspace programs make requests to the kernel via the glibc library [4]. Otherwise known as the GNU C library, glibc is made up of wrapper functions that invoke or make system calls – on behalf of userspace programs – to the Linux kernel. Other libraries, such as the musl library [5], also exist (see the box entitled "musl and glibc"), but glibc is the default library for most Linux distributions.

When the command chmod 700 is executed to assign permissions to a program, a system call or service request is made to the kernel to assign permissions to the program. Linux comes with support for many different system calls. Examples include:

• Filesystem operation: open(), close(), seek(), read(), write()
• Process management: fork(), exec(), exit(), kill(), wait()
• Memory management: brk(), mlock(), unlock(), munmap(), mmap(), sbrk()
• Interprocess communication: pipe(), socket(), shmget(), semget(), msgget()
• Device management: SetConsoleMode(), WriteConsole(), ReadConsole(), open(), close()

There are times when a program wants to make a system call explicitly. In this case, the glibc library provides the syscall function. The syscall function is useful when invoking a system call that has no wrapper function in the C library.

Another important component of system calls in Linux is the system call table. This table consists of the addresses of the kernel functions responsible for implementing specific system calls or system requests, such as open() or chmod(). The system call table is located within the kernel's address space.

So when a userspace program makes a system call via a glibc wrapper, it provides the system call number to the Linux kernel. The Linux kernel uses this number to search for the specific function in the system call table and executes the requested operation.

The following is a detailed flow of system calls in Linux:

1. A user program executes commands on the console.
2. Glibc initiates system calls to the kernel.
3. The glibc function copies the necessary arguments or parameters into registers rdi, rsi, rdx, rcx, r8, and r9. It also places the system call number into the rax register.
4. The library function then triggers a trap instruction or software interrupt, based on system architecture, which causes the processor to switch from user mode to kernel mode.
5. The Linux kernel saves the current state, request, and register values to preserve its context.
6. The kernel uses the system call number to look up the corresponding function in the system call table.
7. The kernel executes the corresponding function, which in turn executes the requested operation.
8. Once the function has finished its work, it returns values and, if necessary, error codes into the right registers and then hands over control to the user via glibc.

In addition to using system calls to interact with the kernel or request kernel services, there is another method Linux programs use to request kernel services. This new interface serves as a hideout for malware programs, avoiding security tools' monitoring system events via the traditional system call interface.

io_uring [6] is a Linux-specific system call interface designed for asynchronous I/O. io_uring is considered better than executing file operations through glibc functions, because it does not block file operations.

Although io_uring is a better alternative for I/O operations, it opens up another avenue for malicious software or programs to avoid being detected by security tools observing system calls through the GNU C libraries.

Why Monitor System Calls?

It is possible to observe system activities via system logs, but some activities might not appear in the system logs. Almost every program makes service calls to the Linux kernel through glibc or io_uring. Thus, security vendors prefer to hook into specific tracepoints in the kernel to track system events.

Monitoring system calls can help with debugging and can help you understand how applications use system resources like CPU, memory, and disk access. Once you know how a program manages a system's resources, you can identify or pinpoint performance bottlenecks to optimize performance.

Monitoring System Calls with Falco

Falco is a cloud native security tool that provides runtime security across cloud native environments. Falco is designed to monitor and detect events pre-defined as malicious via a rules file. Unlike strace, Falco does not reveal low-level details of system calls but rather outputs matching events to users via terminal or logfiles. It is easy to integrate Falco into distributed systems or platforms such as Kubernetes to track activities of containers running on Linux servers.

Falco is made up of a userspace program, a Falco driver located in the kernel space, and falcoctl for managing rules and other administrative tasks.

The Falco userspace program is a CLI tool for interacting with Falco. The userspace tool handles signals. In Linux, signals are used to announce events to processes. The userspace tool also parses information from the Falco driver, and it sends alerts to configured channels.

The Falco driver resides within the kernel space. The driver passes the raw system state to libscap and libsinsp to capture state events, extract non-Falco context, and filter state events. Both kernel module and eBPF source truth from system calls, whilst plugins like k8audit source truth from the Kubernetes events logs.

Falco supports the following drivers:

• Modern eBPF probe
• Legacy eBPF probe
• Kernel module

Plugins extend the functionality of Falco by adding new event sources and new fields that can extract information from events. Plugins are shared libraries that hook into the core functionalities of Falco to allow for:

• Adding new event sources that can be evaluated using Falco rules
• Adding the ability to define new fields that can extract information from events
• Interpreting the content of all the events captured in a data stream
• Adding events asynchronously in a given data stream

Without plugins such as k8audit, CloudTrail, and Okta, Falco only detects events via system calls. The falcoctl component installs rules and plugins and performs administrative tasks.