Cloud backup

rclone

Whenever we write about backing up data, it's always with the caveat that it's a tedious but essential process. But two things have made it less tedious over the last decade – the first is cloud storage, as it means you no longer have to source your own off-site silo for your data, and the second is rsync. Rsync is an amazing tool that duplicates the contents of one filesystem to another and also does a great job of only pushing deltas rather than copying entire files each time. But it's complicated and requires considerable work if you need the source or destination to be somewhere in the cloud, and this is where rclone steps in.

Rclone's elevator pitch is "rsync for cloud storage," but to do this, it needs to know how to talk to lots of different cloud storage providers. Fortunately, it does – nearly 40 of them, including Dropbox, Google Drive, Amazon S3, Nextcloud, SFTP, WebDAV, and even your local filesystem. Adding a destination is an interactive process, started by typing rclone config, selecting New remote followed by your chosen cloud storage. This interactive process makes it easy to make the destination "read-only," for instance, and will typically open your default browser to allow you to authenticate the new client. It's quick and easy, and the excellent online documentation includes transcripts for each service if you get stuck. Backup is then as easy as typing rclone copy followed by the remote and local locations. Another brilliant feature supported on most clouds is server-side copy. This allows you to copy between two remote locations without the file or files passing through your local machine. It's perfect for transferring large amounts of data between Google and Amazon, for example, without affecting your local bandwidth at all.

Project Website

https://rclone.org

Apart from backup, one of the best uses for rclone is sending freshly scanned PDFs to Google Drive, where they'll automatically be processed by its fabulous OCR.

Data management