Stairway to the Cloud

Cloud storage is cheap nowadays, and you have plenty of storage providers to choose from. So, you have no excuse for not having an off-site backup system to keep your files safe. There is a fly in the ointment, however. Many cloud storage services want you to use their own proprietary graphical client applications. Worse still, some services don't provide Linux clients at all. Fortunately, there is rclone, a small open source utility that can talk to many popular cloud storage services, including Google Drive, Amazon S3, and hubiC. Additionally, rclone can handle local filesystems, so you can use it for local backup, too. The utility is straightforward in use, so there is no learning curve to speak of.

Deploying rclone

Written in Go, rclone is distributed as a self-contained binary file with no dependencies, and it will happily run on the x86, AMD64, and ARM platforms. Installing rclone is not difficult, but it does require a bit of manual work. Grab the latest release of the tool for the appropriate Linux platform from the project's website [1]. Unpack the downloaded archive and switch to the resulting directory in the terminal. Then, copy the binary executable to the /usr/local/bin/ directory and change the file's permissions:

sudo cp rclone /usr/local/bin/
sudo chown root:root /usr/local/bin/rclone
sudo chmod 755 /usr/local/bin/rclone

To install man pages containing rclone's documentation, run the following commands:

sudo mkdir -p /usr/local/share/man/man1
sudo cp rclone.1 /usr/local/share/man/man1/
sudo mandb

That's all there is to it. Alternatively, you can compile rclone from source, which is also simple to do. First, make sure that the Go programming language is installed on your system. On Debian and Ubuntu, you can install Go by running the apt-get install golang command as root. Create the ~/go directory, and point $GOPATH to it using the export GOPATH=$HOME/go command. Then, run the following command to download and compile rclone:

go get github.com/ncw/rclone

Once the operation is finished, you'll find the compiled binary file in the ~/go/bin directory. Copy the rclone file and change its permissions as described above.

Configuring rclone

To make rclone work with your preferred cloud storage service, you need to create a configuration file. The utility includes the special config subcommand that allows you to do just that. For example, suppose you want to use rclone with the hubiC service [2]. Run the rclone config command, press n when prompted to set a configuration password, and name the configuration profile remote. Then, enter the number corresponding to the hubiC option in the list of supported services. Leave the Hubic Client Id and Hubic Client Secret options empty and select the auto-configuration option. This will open the default browser and obtain an access token. Check and confirm the generated settings, and you are done.

Rclone uses a default browser for authenticating with cloud storage services, which works well on any system with a graphical desktop environment. But, what if you want to configure rclone on a remote machine without a graphical desktop? It can be done, but you still need another machine with a browser. Here is how it works.

Run the rclone config command and configure the options as described above until you reach the Use auto config? prompt. Press n and note the exact rsync authorize command (in the case of hubiC, the command is rsync authorize "hubic"). Run this command on the machine with a browser, copy the obtained access token, and paste it into the result > prompt on the remote machine.

If you've already configured rclone on a local machine, you can simply copy the .rclone.conf file to the remote machine. Usually, the configuration file is stored in the home directory (i.e., ~/.rclone.conf), but you can find the exact path by running the rclone -h command; The actual path to the configuration file will be shown next to the --config option.

Once rclone has been configured, run the rclone lsd remote: command, which returns all containers. Listing 1 shows the output of this command in the case of hubiC.

01     7639189 0001-01-01 00:00:00        21 default
02           0 0001-01-01 00:00:00         0 default_segments
03
04 Transferred:            0 Bytes (   0.00 kByte/s)
05 Errors:                 0
06 Checks:                 0
07 Transferred:            0
08 Elapsed time:       600ms

Using rclone

The rclone utility supports a few simple subcommands and options that allow you to access, manage, and use the remote storage; the copy and sync subcommands are probably the most important among them. As the name suggests, the copy subcommand copies the contents of the source directory to the remote destination.

This subcommand doesn't transfer unchanged files (it checks them by size, modification time, and md5sum hashes), and it doesn't delete files from the destination directory. If the destination directory doesn't exist, rclone automatically creates it. Most rclone commands have the simple rclone [OPTION] [SUBCOMMAND] <source> <destination> syntax, and here is what the command that copies the contents of a specified directory to the remote destination looks like:

rclone copy /path/to/source remote:destination

Similar to copy, the sync subcommand transfers files from the source directory to the destination, skipping unchanged files. When sync encounters files that don't exist in the source directory, the subcommand deletes them from the destination. In other words, sync keeps both source and destination directories in sync by modifying the destination. Because this operation (as well as some other rclone actions) is irreversible, it makes sense to test it first, for which rclone provides the handy --dry-run option. Add this option to the rclone sync command to check what files will be copied and deleted:

rclone --dry-run sync /path/to/source remote:destination

When using rclone, keep in mind that it copies and syncs the contents of directories and not the directories themselves. So, the rclone sync /home/user/Documents remote:Backup command copies the contents of the Documents directory (and not the directory and files in it) to the remote Backup directory.

The check subcommand can come in handy when you need to ensure that the files in the source and destination directories match:

rclone check /path/to/source remote:destination

This command compares files by their sizes and md5sum hashes and then shows a list of files that don't match.

Besides copy and sync, rclone supports several subcommands that let you view and manage remote storage. The lsd subcommand, for example, can be used to list all directories (also called containers and buckets) in the remote destination, and the ls subcommand shows all files in a specified remote directory:

rclone lsd remote:
rclone ls remote:dir

If you need to create or delete a remote directory, you can use the mkdir and rmdir subcommands for that:

rclone mkdir remote:new_dir
rclone rmdir remote:old_dir

The last command can remove a directory only if it's empty. If you want to delete a directory and its contents, use the purge subcommand:

rclone purge remote:old_dir

As a command-line tool, rclone supports a number of options that control its behavior. The --bwlimit option, for example, lets you limit the bandwidth available to rclone. This can be useful when the machine running rclone shares the Internet connections with other clients. Limiting the bandwidth ensures that other machines can access the Internet at a reasonable speed during the copy or sync operations. The bandwidth limit can be specified in kilobytes, megabytes, or gigabytes using the k, M, and G suffixes:

rclone --bwlimit=15M sync /path/to/source remote:destination

The command above limits the bandwidth to 15MBps.

By default, rclone runs four simultaneous file transfer operations, and you can adjust this number using the --transfers option. If you have a fast connection and remote storage service, you can increase the number of transfers:

rclone --transfers=7 copy /path/to/source remote:destination

Conversely, you might want to reduce the number of transfers if the remote service frequently times out or your Internet connection is on the slow side.

The --dry-run option mentioned previously allows you to test rclone operations without applying any changes. Finally, if you run rclone unattended, you might want to use the --log-file option as shown here

rclone --log-file=rclone.log sync /path/to/source remote:destination

to save rclone's output to a file for later reference.