Command Line – zstd

Unix-like systems have been around long enough that replacements are available for time-honored commands. For instance, tree is a substitute for ls, while apt has unified apt-get and the most popular of its associated scripts. Since 2015, one of the most popular substitutes has been Zstandard (zstd) [1], a compression tool that is a simplification of gzip that is significantly faster than standard archiving tools such as tar, zip, and bzip.

An LZ77 lossless data compression algorithm [2] gives zstd its speed. Algorithms in the LZ77 family are a form of sliding window compression, so-called because they encode in chunks of customizable sizes as they copy and verify results. Chunks that are too small do nothing to increase speed, but ones that are too large start to slow compression because they take longer to verify. Finding the right balance maximizes the speed of an operation. To help obtain the most efficient setting, zstd can build and use a dictionary of settings for different types of files (see the "Using a Dictionary" section).

To LZ77 compression, zstd adds two modern, high-speed entropy encoders [3], which are used at the end of compression. Huffman, with its out-of-order (OoO) execution, offers high speed operations, while Finite State Entropy (FSE), a more recent entropy encoder, is designed to ensure the accuracy of compression at high speeds. Armed with LZ77 compression and these two entropy encoders, zstd easily outperforms other archiving commands, especially when the command options are carefully chosen.

[...]