In Sync

Not everyone is lucky enough to get through life with a single IT device for everything. Many Linux users have a distributed IT infrastructure, with a PC at home, another PC at work, and a laptop for the road.

In such a constellation, you always want to be able to access the current version of all files, regardless of which device you are currently using.

A typical solution to this dilemma is to take a working copy of all the relevant files with you, for example, on a USB stick. But if you don't pay meticulous attention, you could end up with several diverging versions of a file, none of which contains all the changes.

Only centralized data storage on a network drive in the cloud can help. However, the access speed and thus also your efficiency can suffer enormously. In addition, the client must be able to reach the server on the Internet at all times, which is often a problem when commuting or traveling.

For a really practicable solution to the problem, you need three components to work together: centralized data storage that is accessible on the Internet, local working copies of this pool on all devices, and a simple – preferably fully automated – method for synchronizing via a server and clients. Unison [1] is a popular open source tool that can handle the synchronization duties. This article shows how you can combine Unison with the OpenSSH secure shell [2] to connect your local systems to the cloud and synchronize your files (Figure 1).

Figure 1: The user files on three computers, two with a quasi-permanent and one with an occasional network connection, are synchronized over the Internet against a central server.

Setting Up SSH

SSH provides the basis for authenticating the client to the server and for transporting the data over the Internet with protection from unauthorized access.

See the project website for more on setting up OpenSSH [2]. On the server, the SSH service must listen on a defined port (typically port 22). If the server is behind a firewall or a router, you will need to forward the port.

Each client must have a secret key and its public counterpart. A typical scenario is to use an RSA key pair consisting of two files: ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub. If these keys are missing, or if you have forgotten the secret key's passphrase, you can generate a new key pair with ssh-keygen. To create a new passphrase, simply call ssh-keygen -p. You'll need to communicate each client's public key to the server. For example, if the server is located in the mycloud.com domain, you could enter:

$ $ ssh-copy-id client_name@server_name.mycloud.com

The key ends up in the ~/.ssh/authorized_keys file on the server. From now on, the way to connect from the client to the server is to enter the local secret key's passphrase; you won't have to rely on the remote server password.

To enter this passphrase only once per session on the client, use ssh-agents; this command launches automatically at login time on many current desktop systems. If you pass in the secret key unlocked by passphrase via ssh-add, then no further typing of the passphrase is required for each subsequent SSH connection during this session. You will therefore also want to run ssh-add at login by adding it to an autostart script.

Desktops typically use the graphical ssh-askpass dialog to query the passphrase. Some desktops, such as KDE Plasma, use their own implementation (for Plasma: ksshaskpass), which allows them to store the passphrase in the integrated password safe for further sessions after entering it. This step eliminates the need to re-enter the passphrase in the future.

The commands for installing OpenSSH on the server and the corresponding client on desktops and notebooks appear in Listing 1.

Setting Up Unison

For Unison to work properly, all the systems involved in replication – all the clients and the central server – must use the same software version. Both Unison's User Manual and Reference Guide refer explicitly to the fact that the data format of the archives can change between releases. Therefore it is recommended to generate Unison directly from the source code in order to be able to mix different distributions and their generations more easily in the working environment. This article uses the currently recommended stable Unison v2.48.4 [3].

To build Unison, you need GNU Make, the OCaml compiler, and some libraries. To install them on Debian and its offshoots including all necessary dependencies, use the command from line 1 of Listing 2. Then change to the home directory, download the Unison source code, unpack it, and change to the newly created source directory (lines 2 to 5).

Essentially, Unison consists of three executable files, which you build with the commands from lines 6 to 10 and install to /usr/local/bin (lines 11 to 17). The command-line-based Unison performs the actual sync and can be used both in the background and interactively at the command line. Finally, the unison-fsmonitor filesystem monitor is used in case of a permanent connection between client and server. When changes are made on one page, the client informs the other side and initiates a sync immediately.

The interactive alternative with a graphical user interface (GUI) is unison-gtk. It is especially recommended for mobile devices with only a temporary connection to the server and in case of inconsistencies between client and server. Inconsistencies inevitably arise as soon as files on both sides change for some reason without a timely sync being able to take place. Unison can no longer resolve this automatically. In such situations, the graphical client is clearer, making it easier to select the appropriate file version.

Configuring the Synchronization

The way Unison works is controlled by profile files, which can be nested if required (more about this later). These profiles follow semantics, the fundamentals of which I will explain with some examples. When you edit the files, be sure to consult the Unison manual, especially the chapters on profiles [4] and path specification [5].

As a userspace application, Unison stores its configuration in the ~/.unison directory; it is the individual user's responsibility to design this to suit their own needs. Consequently, the Unison profile common.prf (Listing 3) can only serve as an example; you will need to adapt it to meet your specific needs.

Line 4 in Listing 3 specifies the local directory that Unison will synchronize against the server and its directory (line 5). Line 7 defines the logfile that logs all individual adjustments.

In line 9, things start to get more interesting: Lines 4, 5, and 7 tell Unison to compare the entire content of the specified directories against each other. Lines 9 and 10 explicitly exclude two paths in the client root directory (/home/<Client>), Downloads/ and Maildir/ in this case, from the synchronization.

In addition, line 11 excludes all files and directories that have the specified pattern in their name. In this case, this means all the hidden files and directories (dotfiles) in the entire path from the root directory, for example, including the file /home/<Client>/<Folder>/.<File>. This also includes the /home/<Client>/.unison/ directory and its contents.

However, Unison needs to keep its profiles – only the ones from the .unison/ directory – in sync. This explains why line 13 first includes .unison/ again, but line 14 excludes its content (logs and archives). Line 15 then ensures that all files ending in .prf (i.e., the Unison profiles) are still synchronized.

Lines 17 to 19 do something similar for the hidden .gnupg/ directory. However, line 19 here explicitly includes only the three listed files in the synchronization.