A modern compression tool
Command Line – zstd
Like other modern replacement commands, zstd offers significantly faster file compression than the standard archiving tools.
Unix-like systems have been around long enough that replacements are available for time-honored commands. For instance, tree
is a substitute for ls
, while apt
has unified apt-get
and the most popular of its associated scripts. Since 2015, one of the most popular substitutes has been Zstandard (zstd
) [1], a compression tool that is a simplification of gzip
that is significantly faster than standard archiving tools such as tar
, zip
, and bzip
.
An LZ77 lossless data compression algorithm [2] gives zstd
its speed. Algorithms in the LZ77 family are a form of sliding window compression, so-called because they encode in chunks of customizable sizes as they copy and verify results. Chunks that are too small do nothing to increase speed, but ones that are too large start to slow compression because they take longer to verify. Finding the right balance maximizes the speed of an operation. To help obtain the most efficient setting, zstd
can build and use a dictionary of settings for different types of files (see the "Using a Dictionary" section).
To LZ77 compression, zstd
adds two modern, high-speed entropy encoders [3], which are used at the end of compression. Huffman, with its out-of-order (OoO) execution, offers high speed operations, while Finite State Entropy (FSE), a more recent entropy encoder, is designed to ensure the accuracy of compression at high speeds. Armed with LZ77 compression and these two entropy encoders, zstd
easily outperforms other archiving commands, especially when the command options are carefully chosen.
Closely resembling gzip
, zstd
has associated commands that are the equivalent of some common options. For example, unzstd
is the equivalent of zstd -d
, which decompresses files. However, zstd
differs from gzip
in several ways that make it more user-friendly. To start with, zstd
does not delete original files by default like gzip
does. Instead, the original files are only deleted if the --remove
option is added to the command. In addition, by default, zstd
displays progress notifications for single files and displays a help file when an error occurs, behaviors that are turned off when the --quiet
(-q
) option is used.
Command Structure
Integers in zstd
options can be specified in kilobytes (KiB, Ki , K, or KB) or megabytes (MiB, Mi, M, or MB). All these abbreviations should come immediately after the integer, with no space between them.
Files compressed with zstd
have a .zst
extension. Using a standard command structure, zstd
compresses a file with:
zstd INPUT-FILE
An archive will be created in the same directory as the original file (Figure 1). If you add the --verbose
(-v
) option, you can see the compression details. You can compress multiple files using either regular expressions or by listing them after the command in a space-separated list. To create the compressed file in some other location, use the following format:
zstd -INPUT-FILE -o=OUTPUT-FILE
If you want to store multiple files in the same archive, you need to add tar
to the command. The simplest use of tar
involves no zstd
options:
tar --zstd -cf DIRECTORY.tar.zst DIRECTORY
If you want to use any zstd
options, you need to use tar
's -I
option and place the zstd
command and any of its options inside quotes (Figure 2). For example:
tar -I 'zstd --ultra -22' -cf DIRECTORY.tar.zst Directory/
At the most basic level, all that is needed is often a single option: --compress
(-z
) or the command zstd
to compress files; -d
, --decompress
, --uncompress,
or the command unzstd
to decompress files. If no other options are given, zstd
uses its defaults, which might not be the most efficient choices but could be sufficient for general purposes.
If you want more control over compression, other options exist. Using --list
(-l
), you can view information about compressed files, such as their size, compression ratio, and checksum (Figure 3). Additional information can be had by adding the --verbose
(-v
) option. If you do not want to use the default compression ratio of 3
, you can specify -#=1-19
, with a lower number offering greater speed of operation and a high number greater compression. To give a sense of performance, at 3
, zstd
compresses an average Linux kernel slightly faster than gzip
, while at 19
it is 26 times slower. However, the biggest gain in speed is in decompression, for which an archive made at level 3
is about 25 percent faster than gzip
, and one made at 19
is about 60 percent faster than gzip
. From these figures, you can deduce that zstd
's priorities are the degree of compression and the speed of decompression. By contrast, the speed of compression seems of secondary importance.
The speed of operation can be affected by the other options used. Using --fast=NUMBER
, you can add faster speeds of -5
to 2
, while --ultra=NUMBER
allows compression of 20
or greater. However, note that --fast
and --ultra
are not automatically enabled because they may have unintended effects. For instance, --fast
is more likely to produce a file that will not fit on an external drive, while a file produced using --ultra
may take an unacceptably long time to decompress. You may prefer to use --adapt=min=NUMBER,max=NUMBER
to have zstd
set the compression level as it judges best. In the same way, --rsyncable
can be specified to make zstd
adjust to work more efficiently when rsync
is used to connect to remote machines. If zstd
is compiled with multithread support, still another way to affect speed is to add the option --threads=#NUMBER
(-T#NUMBER
), which, when set to 0
, prompts zstd
to detect the number of available cores. Alternatively, the --single-thread
option restricts zstd
to one thread.
By default, zstd
defaults to its own .zst
format. However, it can also be used with several other common compression formats, including .gzip
, .xz
, .lzma
, and .lz4
. These can be specified with the option --format=FORMAT
. Once a compressed file is created, zstd
runs an integrity test that compares it to the original file.
Benchmarking
With all the options that affect zstd
's performance, you may want to experiment to find the most efficient command structure for compressions that you do regularly. zstd
can be compiled with several options for benchmarking, although only the long help, available with the -H
option, lists them all. With -b#NUMBER
, you can test a compression level. Rather than test compression levels one at a time, you can specify a starting level with -e#NUMBER
and the end of a range with -b#NUMBER
. You can also set the time taken to benchmark in seconds with -i#SECONDS
or cut the archive into blocks, specifying -i#SIZE
, which can be useful if you secure files in a cloud by storing them in several pieces. Should you want to know exactly how long an operation takes, you can add the option --priority=rt
(real-time).
Using a Dictionary
If you compress the same type of file regularly, you could be able to squeeze more compression from zstd
by creating a dictionary. Even though the files might be small, you might still be able to compress or decompress more efficiently with a dictionary, because zstd
will not have to read each file separately. However, the effort to create a dictionary could fail because the sample size is too small for general patterns to be observed or because zstd
is unable to find a means to streamline operations with a particular file type. Generally speaking, an effective dictionary requires several thousand samples, although you might manage to produce a dictionary with less. The only way to know is to try.
To create a dictionary, create a directory and add to it files of the same format. Then use the command structure:
zstd --train ##SAMPLE-DIRECTORY/*
If an error occurs, zstd
will stop and suggest how to correct it (Figure 4). If a dictionary is created, its default name will be dictionary
, but you can give it a more specific name with -o FILE
. Other options are listed in the long form help (-H
) but are seriously under-documented. A sample dictionary is available [4], as well as an industry standard [5] and a guide for creating a third-party dictionary builder [6], although none seem to exist yet. However you create a dictionary, you can use it when either compressing or decompressing by adding the -D DICTIONARY
option.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
Halcyon Creates Anti-Ransomware Protection for Linux
As more Linux systems are targeted by ransomware, Halcyon is stepping up its protection.
-
Valve and Arch Linux Announce Collaboration
Valve and Arch have come together for two projects that will have a serious impact on the Linux distribution.
-
Hacker Successfully Runs Linux on a CPU from the Early ‘70s
From the office of "Look what I can do," Dmitry Grinberg was able to get Linux running on a processor that was created in 1971.
-
OSI and LPI Form Strategic Alliance
With a goal of strengthening Linux and open source communities, this new alliance aims to nurture the growth of more highly skilled professionals.
-
Fedora 41 Beta Available with Some Interesting Additions
If you're a Fedora fan, you'll be excited to hear the beta version of the latest release is now available for testing and includes plenty of updates.
-
AlmaLinux Unveils New Hardware Certification Process
The AlmaLinux Hardware Certification Program run by the Certification Special Interest Group (SIG) aims to ensure seamless compatibility between AlmaLinux and a wide range of hardware configurations.
-
Wind River Introduces eLxr Pro Linux Solution
eLxr Pro offers an end-to-end Linux solution backed by expert commercial support.
-
Juno Tab 3 Launches with Ubuntu 24.04
Anyone looking for a full-blown Linux tablet need look no further. Juno has released the Tab 3.
-
New KDE Slimbook Plasma Available for Preorder
Powered by an AMD Ryzen CPU, the latest KDE Slimbook laptop is powerful enough for local AI tasks.
-
Rhino Linux Announces Latest "Quick Update"
If you prefer your Linux distribution to be of the rolling type, Rhino Linux delivers a beautiful and reliable experience.