Backups with Borg and Rclone

Borg is an Open Source deduplicating backup program with impressive performance. Borgmatic is a tool to simplify the configuration of Borg. I will show how I use it with Rclone to backup files to remote cloud storage provider, Backblaze. Optionally, you can use healthchecks.io to notify you if it hasn’t been running.

Overview
- Workflow
Requirements
Setup
- Borg setup
Rclone setup
- Example config
- Example
Disaster recovery

Overview

Workflow

┌──────────┐                 ┌──────────────────┐
│          │                 │                  │
│   Borg   │◄────────────────┤ Source Directory │
│          │                 │                  │
└────────┬─┘                 └──────────────────┘
         │
         │                   ┌──────────────────┐
         └───────────────────►                  │
                             │ Target Directory │
         ┌───────────────────┤                  │
         │                   └──────────────────┘
         │
         │
         │
         │
┌────────▼───┐                ┌──────────────┐
│            │                │              │
│   Rclone   ├────────────────►  Backblaze   │
│            │                │              │
└────────────┘                └──────────────┘

Borg reads from a source directory
Borg compresses and deduplicates source
Borg writes encrypted data to target directory
Rclone reads from target directory
Rclone writes to remote cloud storage

Requirements

git
docker
docker-compose

Setup

Create a cloud account

Go to Backblaze to sign up for a free account. The first 10GB are free.

Create an App Key

An App Key is required for Rclone to authenticate. You will need this later, so save it someplace secure.

Borg setup

$ git clone https://github.com/borgmatic-collective/docker-borgmatic.git
$ cd docker-borgmatic/base/
$ cp .env.template .env

Edit .env

Edit the .env. See Layout for an explanation for each path.

Modify config

In VOLUME_ETC_BORGMATIC above, you will find config.yaml and crontab.txt.

Modify config.yaml to suit your needs, the reference is here. Set crontab.txt to when you want borg to run. You can use crontab.guru for help.

Initialize backup

You need to initialize the target directory using borg init. This step is important, follow closely.

“borg init” initializes an empty repository. A repository is a filesystem directory containing the deduplicated data of zero or more archives.

$ docker exec borgmatic \
sh -c "borg init --encryption=repokey-blake2 /mnt/borg-repository"

By default repositories initialized with this version will produce security
errors if written to with an older version (up to and including Borg 1.0.8).

If you want to use these older versions, you can disable the check by running:
borg upgrade --disable-tam /mnt/borg-repository

See https://borgbackup.readthedocs.io/en/stable/changes.html#pre-1-0-9-manifest-spoofing-vulnerability for details about the security implications.

IMPORTANT: you will need both KEY AND PASSPHRASE to access this repo!
If you used a repokey mode, the key is stored in the repo, but you should back it up separately.
Use "borg key export" to export the key, optionally in printable format.
Write down the passphrase. Store both at safe place(s).

Export key

You will probably want to export the key for safe storage.

IMPORTANT: you will need both KEY AND PASSPHRASE to access this repo! Save them someplace secure.

$ docker exec borgmatic \
sh -c "borg key export /mnt/borg-repository /root/key"

$ docker exec borgmatic \
sh -c "cat /root/key"
BORG_KEY e0a0c54d000d2d7fb05df873e192c355d3587f680a04d1e9ccdb77fad917e6e7
hqlhbGdvcml0aG2mc2hhMjU2pGRhdGHaAZ5sT8JH9P82qYmHLVnot5kkxRVPZIStq5W7ew
iIx6pQToz1skABlGe+qHoADJQAmjdpyUUlKmI03lI9koIz/i0AzPcYolDLXttoxDa+cRiV
x4CbuY2vfoYXnBAlpR1Up/T2wx9HZGLo3fGkhcrnlS6PvBcagf3J8L/rYitT+b9kULdbnL
...

$ docker exec borgmatic \
sh -c "rm -f /root/key"

Start Borgmatic

Start the container by running:

$ docker-compose up -d
Starting borgmatic ... done

$ docker ps
CONTAINER ID   IMAGE             COMMAND       CREATED          STATUS          PORTS     NAMES
3c76ad9c6866   b3vis/borgmatic   "/entry.sh"   24 minutes ago   Up 23 minutes             borgmatic

As long as the container is running, Borg will run as scheduled in crontab.txt.

Rclone setup

Create config

$ mkdir ./data/rclone
$ touch ./data/rclone/rclone.conf

Edit rclone.conf with your account and key you created in the cloud setup.

Example config

[remote]
type = b2
account = xxxxxxxxxxxx
key = xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

$ docker run -it --rm \
--volume /share/config/rclone:/config/rclone \
--volume /share/backup:/data:shared \
rclone/rclone sync /data remote:my-bucket

Test config

Running rclone lsd remote will show all directories/containers/buckets, if they exist

$ docker run --rm \
    -v ${PWD}/data/rclone:/config/rclone \
    rclone/rclone \
    lsd remote:
          -1 2021-12-18 00:12:13        -1 <name-of-bucket>

If you have an error in your config you may see something like:

$ docker run --rm \
    -v ${PWD}/data/rclone:/config/rclone \
    rclone/rclone \
    lsd remote:
2021/12/18 00:11:54 Failed to create file system for "remote:": failed to authorize account: failed to authenticate: Unknown 401  (401 bad_auth_token)

Automate Rclone

You can use cron to automate running Rclone backups on a schedule.

Example

$ crontab -e
0 2 * * * docker run -it --rm --volume /config/rclone:/config/rclone --volume /backups/repository:/data:shared rclone/rclone sync /data remote:<name of bucket> && curl -fsS -m 10 --retry 5 -o /dev/null https://hc-ping.com/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx

Disaster recovery

In the event of a disaster like your server losing a drive, you’ll want to restore from your cloud backups. To do so, follow the borgmatic guide on How to extract a backup.

Remember, you will need your passphrase set in BORG_PASSPHRASE and the encryption key you extracted. Without both of these you will not be able to recover from backup.