A modern compression tool
Command Line – zstd
Like other modern replacement commands, zstd offers significantly faster file compression than the standard archiving tools.
Unix-like systems have been around long enough that replacements are available for time-honored commands. For instance, tree
is a substitute for ls
, while apt
has unified apt-get
and the most popular of its associated scripts. Since 2015, one of the most popular substitutes has been Zstandard (zstd
) [1], a compression tool that is a simplification of gzip
that is significantly faster than standard archiving tools such as tar
, zip
, and bzip
.
An LZ77 lossless data compression algorithm [2] gives zstd
its speed. Algorithms in the LZ77 family are a form of sliding window compression, so-called because they encode in chunks of customizable sizes as they copy and verify results. Chunks that are too small do nothing to increase speed, but ones that are too large start to slow compression because they take longer to verify. Finding the right balance maximizes the speed of an operation. To help obtain the most efficient setting, zstd
can build and use a dictionary of settings for different types of files (see the "Using a Dictionary" section).
To LZ77 compression, zstd
adds two modern, high-speed entropy encoders [3], which are used at the end of compression. Huffman, with its out-of-order (OoO) execution, offers high speed operations, while Finite State Entropy (FSE), a more recent entropy encoder, is designed to ensure the accuracy of compression at high speeds. Armed with LZ77 compression and these two entropy encoders, zstd
easily outperforms other archiving commands, especially when the command options are carefully chosen.
Closely resembling gzip
, zstd
has associated commands that are the equivalent of some common options. For example, unzstd
is the equivalent of zstd -d
, which decompresses files. However, zstd
differs from gzip
in several ways that make it more user-friendly. To start with, zstd
does not delete original files by default like gzip
does. Instead, the original files are only deleted if the --remove
option is added to the command. In addition, by default, zstd
displays progress notifications for single files and displays a help file when an error occurs, behaviors that are turned off when the --quiet
(-q
) option is used.
Command Structure
Integers in zstd
options can be specified in kilobytes (KiB, Ki , K, or KB) or megabytes (MiB, Mi, M, or MB). All these abbreviations should come immediately after the integer, with no space between them.
Files compressed with zstd
have a .zst
extension. Using a standard command structure, zstd
compresses a file with:
zstd INPUT-FILE
An archive will be created in the same directory as the original file (Figure 1). If you add the --verbose
(-v
) option, you can see the compression details. You can compress multiple files using either regular expressions or by listing them after the command in a space-separated list. To create the compressed file in some other location, use the following format:
zstd -INPUT-FILE -o=OUTPUT-FILE
If you want to store multiple files in the same archive, you need to add tar
to the command. The simplest use of tar
involves no zstd
options:
tar --zstd -cf DIRECTORY.tar.zst DIRECTORY
If you want to use any zstd
options, you need to use tar
's -I
option and place the zstd
command and any of its options inside quotes (Figure 2). For example:
tar -I 'zstd --ultra -22' -cf DIRECTORY.tar.zst Directory/
At the most basic level, all that is needed is often a single option: --compress
(-z
) or the command zstd
to compress files; -d
, --decompress
, --uncompress,
or the command unzstd
to decompress files. If no other options are given, zstd
uses its defaults, which might not be the most efficient choices but could be sufficient for general purposes.
If you want more control over compression, other options exist. Using --list
(-l
), you can view information about compressed files, such as their size, compression ratio, and checksum (Figure 3). Additional information can be had by adding the --verbose
(-v
) option. If you do not want to use the default compression ratio of 3
, you can specify -#=1-19
, with a lower number offering greater speed of operation and a high number greater compression. To give a sense of performance, at 3
, zstd
compresses an average Linux kernel slightly faster than gzip
, while at 19
it is 26 times slower. However, the biggest gain in speed is in decompression, for which an archive made at level 3
is about 25 percent faster than gzip
, and one made at 19
is about 60 percent faster than gzip
. From these figures, you can deduce that zstd
's priorities are the degree of compression and the speed of decompression. By contrast, the speed of compression seems of secondary importance.
The speed of operation can be affected by the other options used. Using --fast=NUMBER
, you can add faster speeds of -5
to 2
, while --ultra=NUMBER
allows compression of 20
or greater. However, note that --fast
and --ultra
are not automatically enabled because they may have unintended effects. For instance, --fast
is more likely to produce a file that will not fit on an external drive, while a file produced using --ultra
may take an unacceptably long time to decompress. You may prefer to use --adapt=min=NUMBER,max=NUMBER
to have zstd
set the compression level as it judges best. In the same way, --rsyncable
can be specified to make zstd
adjust to work more efficiently when rsync
is used to connect to remote machines. If zstd
is compiled with multithread support, still another way to affect speed is to add the option --threads=#NUMBER
(-T#NUMBER
), which, when set to 0
, prompts zstd
to detect the number of available cores. Alternatively, the --single-thread
option restricts zstd
to one thread.
By default, zstd
defaults to its own .zst
format. However, it can also be used with several other common compression formats, including .gzip
, .xz
, .lzma
, and .lz4
. These can be specified with the option --format=FORMAT
. Once a compressed file is created, zstd
runs an integrity test that compares it to the original file.
Benchmarking
With all the options that affect zstd
's performance, you may want to experiment to find the most efficient command structure for compressions that you do regularly. zstd
can be compiled with several options for benchmarking, although only the long help, available with the -H
option, lists them all. With -b#NUMBER
, you can test a compression level. Rather than test compression levels one at a time, you can specify a starting level with -e#NUMBER
and the end of a range with -b#NUMBER
. You can also set the time taken to benchmark in seconds with -i#SECONDS
or cut the archive into blocks, specifying -i#SIZE
, which can be useful if you secure files in a cloud by storing them in several pieces. Should you want to know exactly how long an operation takes, you can add the option --priority=rt
(real-time).
Using a Dictionary
If you compress the same type of file regularly, you could be able to squeeze more compression from zstd
by creating a dictionary. Even though the files might be small, you might still be able to compress or decompress more efficiently with a dictionary, because zstd
will not have to read each file separately. However, the effort to create a dictionary could fail because the sample size is too small for general patterns to be observed or because zstd
is unable to find a means to streamline operations with a particular file type. Generally speaking, an effective dictionary requires several thousand samples, although you might manage to produce a dictionary with less. The only way to know is to try.
To create a dictionary, create a directory and add to it files of the same format. Then use the command structure:
zstd --train ##SAMPLE-DIRECTORY/*
If an error occurs, zstd
will stop and suggest how to correct it (Figure 4). If a dictionary is created, its default name will be dictionary
, but you can give it a more specific name with -o FILE
. Other options are listed in the long form help (-H
) but are seriously under-documented. A sample dictionary is available [4], as well as an industry standard [5] and a guide for creating a third-party dictionary builder [6], although none seem to exist yet. However you create a dictionary, you can use it when either compressing or decompressing by adding the -D DICTIONARY
option.
Buy this article as PDF
(incl. VAT)
Buy Linux Magazine
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters
Support Our Work
Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.
News
-
TUXEDO Computers Unveils Linux Laptop Featuring AMD Ryzen CPU
This latest release is the first laptop to include the new CPU from Ryzen and Linux preinstalled.
-
XZ Gets the All-Clear
The back door xz vulnerability has been officially reverted for Fedora 40 and versions 38 and 39 were never affected.
-
Canonical Collaborates with Qualcomm on New Venture
This new joint effort is geared toward bringing Ubuntu and Ubuntu Core to Qualcomm-powered devices.
-
Kodi 21.0 Open-Source Entertainment Hub Released
After a year of development, the award-winning Kodi cross-platform, media center software is now available with many new additions and improvements.
-
Linux Usage Increases in Two Key Areas
If market share is your thing, you'll be happy to know that Linux is on the rise in two areas that, if they keep climbing, could have serious meaning for Linux's future.
-
Vulnerability Discovered in xz Libraries
An urgent alert for Fedora 40 has been posted and users should pay attention.
-
Canonical Bumps LTS Support to 12 years
If you're worried that your Ubuntu LTS release won't be supported long enough to last, Canonical has a surprise for you in the form of 12 years of security coverage.
-
Fedora 40 Beta Released Soon
With the official release of Fedora 40 coming in April, it's almost time to download the beta and see what's new.
-
New Pentesting Distribution to Compete with Kali Linux
SnoopGod is now available for your testing needs
-
Juno Computers Launches Another Linux Laptop
If you're looking for a powerhouse laptop that runs Ubuntu, the Juno Computers Neptune 17 v6 should be on your radar.