Systemd [1], an essential component in major GNU/Linux distributions today, is used for system startup/shutdown as the first process, process group control, subsystem containers, logging, hardware detection and management, and communication interfaces between processes. All these features pretty much sum up the problem: systemd interconnects a lot of services that were formerly modular and independent programs, each one separately configurable. Squeezing all of them into a distro creates dependencies that greatly increase system complexity and make it difficult to look in a single place to find problems.

For example if, during the boot process, the system just hangs at a certain point, with no usable GUI or terminal login and no visible log messages, it could be that some dependent units are just "waiting for each other" (which is, from systemd's point of view, not an error but a misconfiguration). Finding the culprit can then be difficult, especially when this happens in only one out of 100 (!) startup cases.

A lot of controversy is still flying around systemd concerning, on the con side, breaking Unix system principles of modularity and low interdependency of programs and, on the pro side, faster startup through massive parallelization, "all-in-one" system design, and possibly more flexibility.

Personally, I would like to stay out of the controversy, but after running my own tests and having brief discussions with systemd's main developer, it seems to me that systemd hasn't been well tested in "real hardware" environments, especially in conjunction with old computers with small amounts of RAM and slow media like DVDs or USB (1 … 2) flash disks, which are still accommodated in Knoppix. For virtualization and container-based systems on fast hardware, systemd is surely very useful.

You may know from last year's CeBIT presentation that I'm working on a "Knoppix for Raspberry Pi," which, on one hand, is less complicated than Intel/AMD hardware, because ARM architecture usually does not have a PCI bus system – instead using a "device tree" that defines the hard-wired addresses and I/O functions for the system – requiring less "hardware detection." On the other hand, the Pi is – for modern standards – a very lightweight system than cannot be upgraded with more RAM; it doesn't even have a SATA interface, and all data must pass through a controller with little available bandwidth. Therefore, I'd like to build a very light system for the Pi that does not inherit too many requirements. Basically, I want this simple BusyBox SysVinit configuration, with a single Bash script for startup/autoconfiguration (knoppix-autoconfig.sh), one for the graphical desktop (knoppix-startx.sh), and one for shutdown (knoppix-shutdown.sh):

# /etc/inittab for busybox init (Raspi Knoppix)
::sysinit:/etc/init.d/knoppix-autoconfig.sh
::respawn:/etc/init.d/knoppix-startx.sh
::shutdown:/etc/init.d/knoppix-shutdown.sh
ttyAMA0::respawn:-/bin/bash

The last line is for the Pi2-specific serial interface, which should run a Bash shell for debugging purposes.

All commands for system startup and shutdown are included in the three scripts. Some are called one after the other (to wait for dependencies, if present), and some are parallelized in a way that is known to work well. One main goal is to reach a startup time of only a few seconds, from power-on to the ready-to-use desktop.

Unfortunately (for my purpose), the Raspberry Pi distribution of Debian now also has switched to systemd, so it will be a challenge for me either to find a way to remove the systemd conglomerate of interconnected, parallelized services during startup and use the tiny SysVinit built into BusyBox instead, while also having to solve problems with Raspbian's systemd package dependencies when attempting to remove systemd, or to build a new distro from scratch.

In this article, I look at the first approach of removing systemd – at least as a boot system, if not more – and at keeping the system bootable and working.

Systemd – A Very Short Introduction

Before blindly attempting to remove a software package, it's a good idea to have a look at its purpose and function in the computer system. For SysVinit, this was rather simple: Depending on the runlevel (0 … 6) given at the boot command line, start and stop scripts in /etc/init.d/* are called for each installed and activated subsystem (e.g., "load kernel modules," "mount hard disks," or "start up the network"). They are numbered in the order of their calling, so if a program depends on another program, it can simply be named in chronological order after its dependency.

Systemd comprises not only the startup process (process 1, the systemd daemon), but also other services that partly replace formerly independent programs or functions (Table 1). Whereas SysVinit has commands like telinit, reboot, and shutdown/poweroff for switching runlevels, systemd has a front end called systemctl with many options (Table 2) and that also supersedes the symlink-based SysVinit runlevel files in /etc/init.d/* --> /etc/rc${RUNLEVEL}.d/*.

Systemd's journald replaces the formerly text-based logfiles in /var/log by context-sensitive binary files, and you need journalctl to read them. Important commands to know for debugging are shown in Table 3.

The /etc/systemd/system/* files override system defaults in /lib/systemd/system/*. The /run/systemd/system/* files are intermediately generated and take precedence over defaults. Systemd's startup config files (Table 4) should be managed with the Systemctl GUI (Table 5) and not edited or copied directly (which you can still do, regardless of this recommendation). Systemctl even has equivalents for the SysVinit reboot and shutdown commands (Table 6).

Even in this short introduction, you can imagine the full complexity of systemd. It is also important to know that systemd introduced its own pluggable authentication module (PAM) for session management in connection with systemd-logind. A process can ask about the owner of the console and graphical session by sending a message to D-Bus, which is then answered by systemd-logind, having learned about the user's login from the libpam-systemd component.

Experiment Prerequisites

For the experiments described here, I use a popular, lightweight distro – Lubuntu (the LXDE desktop variant of Ubuntu) – in the kvm virtualizer in order to reset everything to start after unsuccessful changes and be able to monitor everything from a working host system.

Also, because the old SysVinit packages have disappeared from the main repositories, I prepared one sysvinit.deb containing /sbin/init, to replace systemd. This should be installed first; otherwise, Lubuntu will complain about missing "essential system packages." The first step is always:

sudo dpkg --force-all -i sysvinit_2.88knoppix_i386.deb \
  initscripts_2.88knoppix_i386.deb \
  sysvinit-utils_2.88knoppix_i386.deb \
  sysv-rc_2.88knoppix_all.deb

Round 1: Just Remove systemd

For the first test, I wanted to see what would happen if I just removed systemd using the Apt standard packaging tool (Listing 1). On the plus side, this will free about 80MB of disk space …. Just kidding.

As expected, at command completion and reboot, the system comes up with a black screen. Removing the quiet option and adding debug in the GRUB boot menu helps somewhat in understanding why. Part of the problem is that way too many services are just gone with systemd, because of soft dependencies set in their package metadata, which affects network configuration and most of the graphical user interface. The other part of the problem is covered in the analysis in round 2.