Tips and tweaks for reducing Linux startup time

Compressing initrd

By default, Debian's initramfs-tool compresses the initrd with Gzip and adds all the drivers. The image is then only 31MB, but loading and unpacking takes quite a while. You can configure the behavior using the COMPRESS variable in /etc/initramfs-tools/initramfs.conf.

An alternative compression algorithm could provide improved performance: Facebook's Zstandard (or zstd for short) takes longer to compress but produces a smaller archive and unpacks it faster (see Table 1). At the cost of a larger image, LZ4 proves to be even snappier when it comes to unpacking. You might need to install these alternative compression tools before you can use them:

Table 1

Comparing Compression Methods

Algorithm

Size

initrd Unpack Time

Parameters

Gzip

31MB

419ms

COMPRESS=gzip

zstd

25MB

260ms

COMPRESS=zstd

LZ4

38MB

84ms

COMPRESS=lz4

sudo apt install zstd lz4

As you can see in Table 1, despite the larger archive, LZ4 needs only one fifth of the time compared to Gzip, and one third compared to zstd, to decompress in just 84ms. However, the results shown in Table 1 are only valid for the system used in our tests: Depending on the speed of the processor and the hard disk, zstd or Gzip might offer advantages on other systems.

Reducing the Scope

In many cases, you can optimize the startup process by reducing the initrd scope. The initramfs-tools-core package contains some tools you can use for analyzing the contents of initrd (Listing 4, line 1 and 3). Of the 1,377 files on the test system, 665 are kernel modules. You can extract the contents of initrd using the command in line 1 of Listing 5; this command lets you analyze the space used by each directory on the lab system (starting in Line 3).

Listing 4

Analyzing initrd Content

01 $ lsinitramfs /boot/initrd.img-5.9.0-3-amd64 | wc -l
02 1377
03 $ lsinitramfs /boot/initrd.img-5.9.0-3-amd64 | grep -c ko$
04 665

Listing 5

Analyzing Space Usage

01 $ unmkinitramfs /boot/initrd.img-5.9.0-3-amd64 5.9.0-3-amd64
02 $ cd 5.9.0-3-amd64
03 $ du -sh *
04 3.4M early
05 98M main
06 $ du -sh main/usr/lib/* | sort -rh
07 84M main/usr/lib/modules
08 9.0M main/usr/lib/x86_64-linux-gnu
09 2.3M main/usr/lib/firmware
10 200K main/usr/lib/udev
11 76K main/usr/lib/klibc-zlroYG-5K-Jd0h1P0SDWOT4wCA.so
12 16K main/usr/lib/systemd
13 16K main/usr/lib/modprobe.d
14 $ <B>du -sh main/usr/lib/modules/5.9.0-3-amd64/kernel/drivers/* | sort -rh | head -10<B>
15 29M main/usr/lib/modules/5.9.0-3-amd64/kernel/drivers/net
16 17M main/usr/lib/modules/5.9.0-3-amd64/kernel/drivers/scsi
17 2.6M main/usr/lib/modules/5.9.0-3-amd64/kernel/drivers/infiniband
18 2.5M main/usr/lib/modules/5.9.0-3-amd64/kernel/drivers/block
19 2.2M main/usr/lib/modules/5.9.0-3-amd64/kernel/drivers/usb
20 2.2M main/usr/lib/modules/5.9.0-3-amd64/kernel/drivers/hid
21 1.9M main/usr/lib/modules/5.9.0-3-amd64/kernel/drivers/ata
22 1.3M main/usr/lib/modules/5.9.0-3-amd64/kernel/drivers/mmc
23 940K main/usr/lib/modules/5.9.0-3-amd64/kernel/drivers/nvme
24 880K main/usr/lib/modules/5.9.0-3-amd64/kernel/drivers/input

The kernel modules take up the most space, as suspected. However, the diversity offered is only necessary if you want the installation to be compatible with a large number of systems and various scenarios. If the installation only has to work on a single computer, you have some room to optimize.

The MODULES=dep parameter in initramfs.conf makes sure that only those modules necessary for reading the root filesystem end up in initrd. After recreating the image, the size of initrd shrinks quite substantially. LZ4 is still the fastest way to unpack the results (see Table 2).

Table 2

Purified initrd

Algorithm

Size

initrd Unpack Time

Parameters

zstd

6.1MB

58ms

COMPRESS=zstd

LZ4

8.4MB

18ms

COMPRESS=lz4

The kernel modules now only occupy 4.6MB instead of 84MB, and the library directory now takes up most of the space. Another look at the files stored there reveals some big space guzzlers (Listing 6).

Listing 6

Analyzing Libraries

$ du -sh main/usr/lib/x86_64-linux-gnu/* | sort -rh | head -10
3.0M main/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
1.8M main/usr/lib/x86_64-linux-gnu/libc-2.31.so
1.3M main/usr/lib/x86_64-linux-gnu/libm-2.31.so
572K main/usr/lib/x86_64-linux-gnu/libpcre2-8.so.0.9.0
432K main/usr/lib/x86_64-linux-gnu/libdevmapper.so.1.02.1
412K main/usr/lib/x86_64-linux-gnu/libext2fs.so.2.4
368K main/usr/lib/x86_64-linux-gnu/libmount.so.1.1.0
332K main/usr/lib/x86_64-linux-gnu/libntfs-3g.so.883.0.0
320K main/usr/lib/x86_64-linux-gnu/libblkid.so.1.1.0
252K main/usr/lib/x86_64-linux-gnu/libfuse3.so.3.10.0

The libcrypto.so.1.1 library needs kmod (libkmod.so.2.3.5) to check the signature of Linux modules [1]. You could basically build kmod with --without-openssl. I tried this on the test system and thus removed the dependency, but this is not a good idea for security reasons. However, the question arises as to why libraries for NTFS and FUSE need to be in initrd. Many programs have a switch to output more detailed messages. Use the -v option for more verbose output with update-initramfs (Listing 7).

Listing 7

What Does update-initramfs Do?

$ sudo update-initramfs -uv
Available versions: 5.9.0-3-amd64
5.9.0-2-amd64
[...]
Calling hook ntfs_3g
Adding binary /bin/ntfs-3g
Adding binary-link /usr/lib/x86_64-linux-gnu/libntfs-3g.so.883
Adding binary /usr/lib/x86_64-linux-gnu/libntfs-3g.so.883.0.0
[...]

These hooks – a way for packages to integrate their own scripts – are located in the /usr/share/initramfs-tools/hooks/ directory and are executable. An unneeded script can be disabled by using the chmod command to remove the executable bit (Listing 8). Note that a system update can reverse the change. LVM and encrypted root partitions require dmsetup. On systems where /dev/mapper/ does not contain appropriate files, you can also disable the dmsetup hook.

Listing 8

Disabling Three Hooks

$ sudo chmod -x /usr/share/initramfs-tools/hooks/{fuse,ntfs_3g,thermal}
$ sudo update-initramfs -uv
[...]
/usr/share/initramfs-tools/hooks/ntfs_3g ignored: not executable
[...]

On systems with firmware that includes and applies the latest microcode updates (Listing 9), you do not also have to include these updates in initrd. On Intel systems, where Linux does not log the update, you can also disable the intel_microcode hook, which saves 500KB.

Listing 9

Checking Microcode Update

$ dmesg | grep microcode
[ 0.000000] microcode: microcode updated early to revision 0xde,   date = 2020-05-26
[ 0.529552] microcode: sig=0x906e9, pf=0x2, revision=0xde
[ 0.529744] microcode: Microcode Update Driver: v2.2.

All of these comparatively simple changes help the system unpack initrd in 14.5ms (Listing 10), where it took 419ms before – a massive 28 times faster.

Listing 10

Unpacking in Less Than 15ms

$ sudo dmsg
[...]
[ 0.398422] calling populate_rootfs+0x0/0x109 @ 1
[ 0.398453] Trying to unpack rootfs image as initramfs...
0.413173] Freeing initrd memory: 6968K
[ 0.413254] initcall populate_rootfs+0x0/0x109 returned 0 after 14480 usecs

According to the 80-20 rule, the last milliseconds require far more effort on the user's part. You could still remove a few Linux kernel modules. On the test system, for example, booting does not require the AHCI driver or the MMC driver. The MODULES=list option in initramfs.conf makes sure that only the modules listed in /etc/initramfs-tools/modules are added (Listing 11). This step reduces the time for unpacking and loading initrd to 9.5ms, which means that I have reached the target for the lab system (see the "Potential" box).

Listing 11

Explicitly Listed Modules

crc32c-intel
crc32c_generic
jbd2
mbcache
crc16
ext4
crct10dif_common
crct10dif-pclmul
crct10dif_generic
crc-t10dif
t10-pi
nvme-core
nvme

Potential

I should note that, without an encrypted root partition, it is possible to do without initrd altogether if the kernel has built-in drivers for the interface and the filesystem. This is not the case with Debian, so you would have to build the Linux kernel yourself and adjust the GRUB configuration /boot/grub/grub.cfg generated by update-grub each time. Because of the effort it takes to configure this option, I don't recommend it for systems where the necessary drivers aren't available by default.

Crypto Modules

The graph created by systemd-bootchart shows a large share for the dh_init and rsa_init methods in the total boot time (Figure 3). It turns out that these cryptographic modules run hardware tests that take some time (Listing 12). If you value security, you need to trust the developers that these tests are necessary. If you do not want to wait the 124ms, add the cryptomgr.notests option to the GRUB_CMDLINE_LINUX_DEFAULT variable from the /etc/default/grub.

Listing 12

Cryptography Test Time

$ sudo dmesg | grep dh_init
[ 0.144678] calling dh_init+0x0/0x20 @ 1
0.245015] initcall dh_init+0x0/0x20 returned 0 after 97656 usecs
$ sudo dmesg | grep rsa_init
[ 0.245015] calling rsa_init+0x0/0x50 @ 1
[ 0.275009] initcall rsa_init+0x0/0x50 returned 0 after 27343 usecs
Figure 3: Instead of 419ms, populate_rootfs now takes less than 10ms. dh_init, rsa_init, and acpi_init now occur in the first second.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Debian: 14 Seconds to Boot

    An article from the Debian-Administration.org community describes how to boot up Debian much faster on an Asus Eee 901. Its author, Phil Endecott, claims to do it within 14 seconds using his method.

  • Working with the Kernel

    If you work with third-party hardware drivers, or even if you just need to fix a broken system, someday you might need to upgrade the Linux kernel.

  • Knoppix 5.3.1: Lenny Basis, KDE 4.0 and Accessibility Software

    Klaus Knopper has released version 5.3.1 of his live Knoppix system. The codebase comes courtesy of Lenny, the next generation Debian.

  • ASK KLAUS!

    Klaus Knopper is the creator of Knoppix and co-founder of the LinuxTag expo. He currently works as a teacher, programmer, and consultant. If you have a configuration problem, or if you just want to learn more about how Linux works, send your questions to: klaus@linux-magazine. com

  • Optimizing the Kernel

    We explore some optimizations designed to deliver a smoother experience for desktop users.

comments powered by Disqus
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters

Support Our Work

Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.

Learn More

News