Multifaceted backups with Kopia

Tutorial – Mastering Kopia

Article from Issue 275/2023
Author(s):

Data deduplication, encryption, compression, incremental backups, error correction, and support for snapshots and popular cloud storage services: Kopia delivers.

A good backup tool is like a dishwasher: It's not something most of us get excited about, but the degree to which it improves our daily lives is hard to overstate. And like with a dishwasher, no one really wants to spend time attending to a backup tool. Ideally, you'd want to set it up once and let it do its job with the push of a button or have it perform backups automatically, with no user interaction whatsoever.

Picking the right backup tool is not as trivial as choosing a dishwasher, though. Sure, you can whip up a simple shell script that backs up data to a different storage device using good old rsync. But in this day and age, it's simply not enough. If you're serious about keeping your data safe, you want to use a tool that supports incremental backups, deduplication, snapshots, and other useful features. For an offsite backup, you definitely want your backup tool to support mainstream storage services and encryption. On top of that, it wouldn't hurt if all of this were wrapped in a user-friendly interface.

It may sound like a pipe dream, but that's exactly what Kopia [1] has to offer. Plus, this cross-platform tool features a built-in web server and a dedicated desktop graphical application. And it goes without saying that you can use Kopia from the command line. In short, it's pretty much a perfect tool for keeping your data safe.

Getting Started with Kopia

Unsurprisingly, to use Kopia you have to install it on your system first, and the project supports practically every installation option imaginable. There are packages for most mainstream Linux distributions, there are Docker images for those who prefer to go the container route, there are AppImage packages, and you can even grab a single executable binary from the project's GitHub repository.

If you happen to use an Apt-based Linux distribution (Debian, Ubuntu, or Linux Mint), installing the latest version of Kopia is a matter of running the commands in Listing 1.

Listing 1

Apt-based Kopia Installation

curl -s https://kopia.io/signing-key | sudo gpg --dearmor -o /etc/apt/keyrings/kopia-keyring.gpg
echo "deb [signed-by=/etc/apt/keyrings/kopia-keyring.gpg] http://packages.kopia.io/apt/ stable main" | sudo tee /etc/apt/sources.list.d/kopia.list
sudo apt update
sudo apt install kopia

The official documentation also suggests installing the Kopia UI desktop application, but you don't really need it, because you can access and control Kopia via its web UI. In fact, the web UI offers the most straightforward way to learn Kopia's basics. To enable the web UI, you need to configure and start Kopia's built-in server. Normally, this involves creating a Kopia user, configuring permissions, and creating and enabling a certificate. The good news is that you don't need any of that if you only want to access Kopia from the same machine it runs on, and the machine itself is not accessible from outside of the local network. In this case, you can start Kopia's server with all security measures disabled using:

kopia server start --insecure --without-password --disable-csrf-token-checks

When the server is running, point the browser to http://127.0.0.1:51515 to access Kopia's web UI.

Starting the server manually is fine, but a better approach is to let the system do that automatically on boot through a systemd service. To do this, use the following commands to create a dedicated directory for systemd services, and then create a new systemd unit file and open it for editing:

mkdir -p ~/.config/systemd/user/
nano ~/.config/systemd/user/kopia.service

Enter the service definition in Listing 2 and save the changes.

Listing 2

Service Definition

[Unit]
Description=Kopia server
[Service]
Restart=always
ExecStart=kopia server start --insecure --without-password --disable-csrf-token-checks
ExecStop=/usr/bin/kill -HUP $MAINPID
[Install]
WantedBy=default.target

Use the following commands to enable and start the service as well as enable it on boot:

systemctl --user daemon-reload
systemctl --user enable kopia.service
systemctl --user start kopia.service
loginctl enable-linger $USER

With the Kopia server up and running, the next step is to create and configure a location for storing backups, called a backup repository. The landing page in the web UI lists all supported storage types (Figure 1), so creating and configuring a new backup repository is a matter of clicking the appropriate button (Figure 2). To set up an external USB storage device as a backup repository, click the Local Directory or NAS button (Figure 1).

Figure 1: Kopia supports a wide range of storage types.
Figure 2: Creating a new backup repository.

Assuming that the target storage device is connected and mounted, enter the root directory or a specific folder on the storage device in the Directory Path field, and click Next. Because Kopia encrypts backups, you must specify a password for the new repository. It goes without saying that if you lose or forget the password, you won't be able to access your backups. The Advanced Options section lets you configure additional settings. While you can leave most of the options at their defaults, you might want to enable the error correction feature that reduces the likelihood of data corruption caused by bitrot or hardware issues. To enable this feature, set the Error Correction Overhead option to the desired value. This value determines how much storage space is used for the error correction code. Keep in mind that the error correction functionality is still experimental. When you've configured the options, click Create Repository to create the backup repository.

Like most modern backup tools, Kopia doesn't simply mirror the data you want to keep safe. Instead the application uses the concept of snapshots. Every time you run a backup job, Kopia creates a snapshot, or a backup catalog that is frozen in time. The data in the snapshot reflects the directory structure and the state of each file as it was at the moment the snapshot was created. The snapshot approach has several advantages compared to a straight backup, key among them being the ability to restore previous versions of specific files and directories. On the downside, the snapshot-based backup approach requires more storage than the source. So it's a good idea to allocate as much storage space for use with Kopia as possible.

But if Kopia creates a new snapshot every time you run a backup job, wouldn't you need infinite storage to keep an infinite number of snapshots? Yes, but in reality, it's unlikely you'd need to keep all snapshots Kopia has ever created. Realistically, you'd want to keep a couple of yearly snapshots, weekly snapshots that go back a month or so, hourly snapshots for the last 48 hours, and so on. You can then let the backup tool automatically remove outdated snapshots, which frees up storage and ensures that you'll never run out of space.

This is exactly what Kopia's snapshot retention mechanism is designed to do. The application allows you to specify how many latest, daily, weekly, monthly, and annual snapshots you want to keep. It could be a bit tricky to figure out how many of each type of snapshots you might need. The good news is that Kopia features sensible defaults, so you don't have to deep-dive into the topic right from the start (Figure 3).

Figure 3: Kopia makes it possible to configure a snapshot retention policy.

To perform a backup (i.e., create a new snapshot), click the New Snapshot button, and enter the path to the directory with the data you want to back up. You can then configure snapshot settings using the available options. You can specify the number of snapshots you want Kopia to keep in the Snapshot Retention section (as already mentioned, you might want to leave the options at their defaults), and you can list the files and directories you want to exclude from the backup in the Files section (Figure 4).

Figure 4: You can exclude certain files and directories from the backup.

As any backup application worth its salt, Kopia makes it possible to set up a schedule for automatically creating snapshots (Figure 5). This can be done by configuring the available options in the Scheduling section. The most straightforward way to enable regular automatic backups is to choose the desired interval from the Snapshot Frequency drop-down list. This way, you can configure intervals from every 10 minutes to every 12 hours. If you want Kopia to automatically create snapshots every day at a specific time, specify one or several time entries in the Times of Day section. For example, if you want Kopia to create snapshots at 11am and 5pm, enter 11:00 and 17:00, each on a new line. And here's the clever part: If Kopia detects no new or modified files, it skips the scheduled snapshot operation, which avoids clogging the storage device with identical snapshots.

Figure 5: You can schedule automatic backups.

After you've configured the settings, press the Snapshot Now button to create a new backup snapshot.

If you take a look at backup snapshots on the storage device, you'll see directories with encrypted files in them. In other words, you can't directly access the backup data. Instead, you can browse the backups using Kopia's web UI or the Kopia desktop application. In the Snapshots section, click on the path defined as a backup source, and you should see a list of all snapshots. Click the desired snapshot entry, and you should see the source data as it was at the moment the snapshot was taken. You can traverse the directories to locate the file you need. Click on the file's link to download it. Kopia has another clever trick up its sleeve: It allows you to mount a snapshot as a local filesystem, so you can work with it using a file manager. To mount a snapshot, click the Mount as Local Filesystem button. This mounts the current snapshot and conveniently displays the path to its mount point.

Using Kopia's web UI or a mounted filesystem works fine if you need to restore one or two files or directories, but if you need to perform a full restore of the backed up data, there is a better way to do this (Figure 6). In Kopia's web UI, select the desired snapshot, press the Restore Files and Directories button, specify the destination directory for the restored data (this directory must be empty), and hit Begin Restore. Instead of restoring the entire snapshot, you can restore a single directory and all its content. To do this, navigate to the desired directory in the snapshot, and then press Restore Files and Directories.

Figure 6: Restoring data from a snapshot.

Using Kopia from the Command Line

The Kopia desktop application and the web UI offer a user-friendly way of using the application, but nothing beats the efficiency of the command line. In fact, you can set up a new backup repository, connect Kopia to it, and create a snapshot using the three simple commands in Listing 3.

Listing 3

Set Up, Connect, Create

kopia repository create filesystem --path /path/to/repository
kopia repository connect filesystem --path /path/to/repository
kopia snapshot create /path/to/source

To list all existing snapshots, run the kopia snapshot list command, and to restore data from a specific snapshot, use the kopia snapshot restore command followed by the hash of the desired snapshots and the path for saving the restored data:

kopia snapshot restore ke5ba82cc69841df04f5839102f0cd53d/path/to/restore/dir

In most cases, you're likely to keep multiple backup copies, and you can use Kopia to create and manage several repositories. Better still, Kopia makes it possible to synchronize the currently connected repository with another local or remote repository. This means that you can, for example, connect to a repository on a local storage device and then simply synchronize it with a remote repository. Unfortunately, you can only perform synchronization from the command line. Fortunately, it's just a matter of running a single command. You can synchronize the currently connected repository to a remote Backblaze B2 repository (where BUCKET is the actual name of an existing B2 bucket) with:

kopia repository sync-to b2 --bucket BUCKET

By default, the synchronization command doesn't synchronize deletions, but adding the --delete flag enables that:

kopia repository sync-to b2 --bucket BUCKET --delete

Conclusions

Hopefully by now you know enough to put Kopia to work on backup duties. But there is much, much more to the application beyond its basic functionality. If you are serious about backup, it's worth putting time and effort into learning and understanding Kopia's advanced features. Data loss is as inevitable as death and taxes, and sooner or later a backup will save your bacon. Kopia is equipped to do just that.

Infos

  1. Kopia: https://kopia.io/

The Author

Dmitri Popov has been writing exclusively about Linux and open source software for many years. His articles have appeared in Danish, British, US, German, Spanish, and Russian magazines and websites. You can find more on his website at http://cameracode.coffee.

Buy this article as PDF

Express-Checkout as PDF
Price $2.95
(incl. VAT)

Buy Linux Magazine

SINGLE ISSUES
 
SUBSCRIPTIONS
 
TABLET & SMARTPHONE APPS
Get it on Google Play

US / Canada

Get it on Google Play

UK / Australia

Related content

  • Introduction

    This month in Linux Voice and Elvie.

  • Back In Time

    Despite the importance of backups, many users still view the process as too complicated and too inconvenient. Back In Time makes the unloved backup less terrifying.

  • Restic/Jarg

    Restic and the Jarg front end appeal to users who want to see quick results without too much overhead.

  • Timeshift

    Timeshift lets users easily create, manage, and restore system snapshots.

  • Snapshot Tools

    Experts agree that you should keep a copy of your data, but restoring from incremental backups takes time and sometimes doesn't work as expected. Alternatively, you can capture your data in a snapshot. Read on for a review of some leading Linux snapshot tools.

comments powered by Disqus
Subscribe to our Linux Newsletters
Find Linux and Open Source Jobs
Subscribe to our ADMIN Newsletters

Support Our Work

Linux Magazine content is made possible with support from readers like you. Please consider contributing when you’ve found an article to be beneficial.

Learn More

News