W hen I took my first steps, it was frowned upon – and rightly so – to stay permanently logged into a system as root. To this day, the KDE graphical desktop environment displays a warning message if you so much as dare to log in as root. However, automation and orchestration were not an issue for many years, and many admins had no qualms about storing sensitive data such as passwords in plain text on disk.

It was not even standard practice to store system users' passwords in the encrypted /etc/shadow file. The "Linux Shadow Password HOWTO" guide [1] from March 1996 explained how to convert your system for the use of shadow passwords, but it took a while before shadow passwords finally became established on all Linux systems. When Debian GNU/Linux 3.0 (aka "woody") was released in 2002, it still prompted you to decide during the install whether you wanted to enable shadow passwords.

Today,storing passwords in plain text is considered an absolute no-no. Even services such as Postfix, which requires passwords to log in to an SMTP relay host can now store and read passwords in an encrypted form. Powerful tools are also available for popular automation tools such as Ansible or Puppet, making it easier to store passwords in an encrypted form and only access them at runtime. In terms of simple system security, you might think everything is looking good.

However, sharing sensitive content via the command line and a filesystem (except for ready-made solutions such as Ansible Vault) is still a Herculean task. In reality, this scenario is not as uncommon as many might think. Some organizations might still use netboot to install servers for manual configuration.

In order to set up the server's Intelligent Platform Management Interface (IPMI) server and access it via ipmi, you usually want to assign a unique password to the systems' out-of-band interfaces. If you now have 20 servers to deal with, you will also need 20 IPMI passwords, which need to be provided to your colleagues in a way that lets them continue to work on the setup. This raises the question of how to share the combination of username and password for each individual system with your coworkers. This question becomes even more urgent if you want to maintain the IPMI passwords in a central configuration directory, such as a Git directory.

Bull Baiting

Intuitively, most admins answer this question with GNU Privacy Guard (GnuPG). No matter how good this is from a security perspective, there is no escaping the fact that Pretty Good Privacy (PGP) and its free implementation, GnuPG, may be fine from a technical point of view, but neither have achieved the desired level of use.

For one thing, GnuPG is typically used in an email context, although using it on some operating systems is a problem. In theory, you could encrypt the IPMI passwords with GnuPG and then mail them to everyone who needs them. But this assumes that recipients not only have a working version of GnuPG in their email program, but they also have their own private key and have stored their key's public counterpart on a keyserver that is accessible to the sender.

GnuPG can also be used at the command line to encrypt and decrypt files. In this case, you could even reproduce the aforementioned example by involving a Git directory in some way. However there is no convenient way of doing so with GnuPG, and this is even more true if every employee is allowed to modify the encrypted file. Employees would first have to output the file locally in plain text, complete the edits, and encrypt the file again using the public keys of everyone who needs access. If you forget a key, the key's owner can no longer read the file despite having basic access rights.

Anyone who is familiar with the chaos that arises when several people collectively edit a LibreOffice Writer file in Nextcloud knows where this ends. Moreover, experience has shown that hacks like this only work in exceptional cases. Sooner or later, admins will find ways to work around the system. Other solutions such as Ansible Vault are easier to use, but they require all kinds of additional software and a separate setup. On top of this, some of the solutions do not allow the encrypted password files to be stored in a Git directory along with the rest of the configuration. All told, the situation is anything but satisfactory.

Better Solutions Needed

Kevin Niehage, a developer based in Berlin whose job partly involves dealing with security and data protection, had a very specific use case in mind when looking for a solution to this issue. He was busy converting his own IT infrastructure to run in containers. He wanted to be able to store and manage a service, along with the passwords and secrets associated with it, in Git, and he wanted to encrypt the sensitive data. This prompted him to create a solution for sharing sensitive data via a filesystem that was portable and as easy to use as possible.

Niehage didn't have to start from scratch: He had previously developed two encryption tools, v00 and v01, while working for his former employer, SysEleven. The tools' original task was to enable sharing of temporary passwords via an HTTP-based online service. You can still find the GitHub project [2], Shared-Secrets, in Niehage's account as well as the online service [3].

The idea behind the Shared-Secrets app is quite simple: When a provider creates a new access vector for a VPN connection for a customer, for example, the provider normally assigns a one-time password, which is sent to the customer as the next step. This is a one-time password, not because it is only used once, but because the customer can only retrieve it from the provider once. After that, the customer is responsible for keeping it safe. Technically, this only works if the provider makes the password available to the customer via a service set up specifically for this purpose, and the Shared-Secrets app developed by Niehage solves precisely this problem.

The construct works well: The tool can be rolled out quickly in its own Docker container and listens with SSL protection on the address exposed by the container manager. The service can then be used to receive passwords from users, encrypt them, and store them in a local database. The service's other task is to deliver a shared password to authorized users only once.

The database is primarily used to track passwords that have already been delivered. If a link is used to retrieve a password multiple times, Shared-Secrets recognizes this, allowing conclusions to be drawn about whether the password links have fallen into the wrong hands, for example, due to a compromised email account. As I mentioned earlier, Shared-Secrets uses two different algorithms for encryption: v00 is used for communication between the sending client and server, while v01 ensures encrypted delivery of the password to the client.

The v00 algorithm uses symmetric encryption (Figure 1). When the client defines a password with Shared-Secrets, the server does not even see the password sent to it (Figure 2). Consequently, a break-in attack on the server with the one-time passwords does not compromise all the stored passwords.

Figure 1: Shared-Secrets uses the v00 algorithm for symmetric encryption.

Figure 2: Shared-Secrets accepts a secret and encrypts it with a password if required.

The v01 algorithm, on the other hand, ensures that the string generated for output to the user side contains all the required parameters in the URL for once-only retrieval of the password. To allow this to happen, the server first displays a link that the client can use to retrieve the password again (Figure 3) on submission of the password. It uses hybrid encryption by combining the principle of symmetric encryption with a public key method. The server generates the RSA key for retrieving the secret dynamically; the key is Base64-encoded and part of the URL for retrieving the password. In addition, the server remembers the fingerprint of the RSA key used for decryption to prevent it from being requested again later.

Figure 3: Shared-Secrets generates a URL that contains all the details required for decrypting the password in Base64 encoding. Retrieval is only possible exactly once because the service remembers the keys' fingerprints.

The Shared-Secrets version used by SysEleven is version two, an almost complete rewrite from 2019. The first version still used GnuPG under the hood, but this was then discarded for various reasons. Because Niehage played a key role in working on Shared-Secrets, he knew what an everyday mechanism for storing encrypted passwords should look like. His ideas on the subject ultimately culminated in the v02enc tool.

Basics

Anyone who works with containers has undoubtedly encountered the fundamental problem of storing confidential details. As is so often the case in the open source world, v02enc was created because Niehage was dissatisfied with the existing solutions, such as age [4] and SOPS.

Niehage was not a fan of the age encryption tool for several reasons. Once an age file has been stored, it is impossible to find out for which users it was encrypted. It also is impossible to update an encrypted age file. Instead, you practically have to recreate it and encrypt it for the same recipients as the original file. If you do not have the list of the original recipients, it is practically impossible to change age files without running a high risk of excluding existing users.

SOPS also has some design flaws. It does resolve the problem of unknown recipients of an encrypted file by maintaining a keys.txt file for each file. However, if you want to encrypt several files for different groups of recipients in the same repository, you will find yourself faced with a flood of keys.txt files, which makes for extremely tricky handling.

Consequently, Niehage was faced with the task of imitating the functional principle of the v01 algorithm from his Shared-Secrets tool, but with symmetric encryption instead of hybrid encryption. Using v01 as a template, Niehage modified the tool so that it no longer used asymmetric RSA keys but instead could use symmetric keys from OpenSSL. He also extended the format's headers in order to be able to determine at an early stage during decryption whether or not the user had entered a correct password. A kind of local brute force attack with a password for each recipient for whom a file had been encrypted would take forever, especially for larger files to be decrypted.

After implementing these changes, the result was the v02 format, for which Niehage also designed several CLI tools, including v02enc and vim02enc, plus git02enc for Git integration. The complete source code for the tools can be found in Niehage's Git directory [5].