Restic Backups to Backblaze B2 with Object Lock

AI-generated

Restic and Backblaze B2 are a good combination for an off-site backup which does not need another server in my homelab. With B2 Object Lock, deleted backup data stays protected for a configured time, even if somebody gets access to the backup credentials.

I use Restic for Linux backups because it is encrypted on the client, deduplicates data and has a simple repository format. Backblaze B2 gives me cheap object storage without having to maintain another ZFS box, VPN or storage VM. This is especially useful for data from my Arch laptop and selected machines in my Proxmox homelab.

Why Object Lock matters for Restic

A normal Restic repository is not immutable. If ransomware gets root access and finds both the Restic password and storage credentials, it can run restic forget --prune or directly delete the repository objects.

Object Lock changes this part. Every newly uploaded object receives a retention date. Before that date, B2 refuses deletion. Restic can still write new backups and restore old data, but it cannot remove protected files.

For this setup I use these two retention layers:

  • B2 Object Lock retention: 30 days in Compliance mode
  • Restic snapshot retention: longer than 30 days, for example 45 daily backups plus weekly and monthly snapshots

The difference is important. If Restic tries to delete a snapshot before the Object Lock period is over, B2 rejects the request. That is expected, but Restic will return an error for the cleanup job.

Compliance mode is serious. A protected object cannot be deleted or have its retention shortened before the date expires. Create a test bucket first and make sure the retention time is correct. Do not start with several years just because it sounds safer.

Creating the B2 bucket and restricted key

Object Lock has to be enabled when creating the B2 bucket. It cannot simply be enabled later on an existing bucket with old data.

In the Backblaze web interface I create a new bucket in the region I want to use, enable Object Lock and set a default retention period of 30 days in Compliance mode. The default retention matters because Restic creates many objects automatically. I do not want to depend on each client setting a retention header correctly.

Next I create an application key limited to this single bucket. For the backup client it needs these capabilities:

  • list files
  • read files
  • write files
  • delete files

The delete capability looks strange at first. Restic needs it for normal repository maintenance. Object Lock still blocks deletion of protected objects, so the key cannot bypass the retention just because it has delete access.

I keep a second key for restores and checks. That one only needs list and read access. It is a small extra step, but a read-only restore key is useful when testing backups from another machine.

Object Lock protects existing backup data. It does not stop an attacker with write credentials from uploading garbage until your B2 account reaches its storage limit. I still use account alerts and do not store the write key on machines which do not create backups.

Configuring Restic with the B2 S3 API

Restic supports B2 directly, but I use B2's S3-compatible API here. It makes the endpoint and credentials easy to understand, and the same general setup also works with other S3 storage later.

On Arch I install Restic like this:

$ sudo pacman -S restic

For a Gentoo machine the package is available as well:

# emerge app-backup/restic

I place the credentials in a root-readable environment file. The endpoint must match the B2 bucket region. eu-central-003 below is only an example.

# install -d -m 0700 /etc/restic
# vim /etc/restic/b2.env
RESTIC_REPOSITORY=s3:https://s3.eu-central-003.backblazeb2.com/my-locked-backup-bucket/restic
RESTIC_PASSWORD_FILE=/etc/restic/password
AWS_ACCESS_KEY_ID=your-b2-key-id
AWS_SECRET_ACCESS_KEY=your-b2-application-key
# chmod 0600 /etc/restic/b2.env
# umask 077
# openssl rand -base64 48 > /etc/restic/password
# chmod 0600 /etc/restic/password

The Restic password encrypts the repository. The B2 key alone is not enough to read files. Losing the password means losing the backup, so I store it separately from the machine, together with recovery information.

I then load the file and initialise the repository once:

# set -a
# . /etc/restic/b2.env
# set +a
# restic init
created restic repository 8f1c2a4b at s3:https://s3.eu-central-003.backblazeb2.com/my-locked-backup-bucket/restic

Please note that knowledge of your password is required to access the repository.

Running backups and dealing with prune

A basic backup of local system configuration and home directories looks like this:

# set -a
# . /etc/restic/b2.env
# set +a
# restic backup /etc /home --exclude-caches --one-file-system

For databases, VM disks or application data I do not just copy live files blindly. PostgreSQL, MariaDB and some VM images need a snapshot or dump first. On my Proxmox machines I keep the VM backup process separate and put the exported backup files into Restic afterwards.

The retention command should keep snapshots longer than the B2 lock period:

# restic forget \
    --keep-daily 45 \
    --keep-weekly 12 \
    --keep-monthly 12

Later I run prune separately:

# restic prune

This is the part which can be a bit tricky. During the first 30 days, B2 can reject deletions made by prune. Restic may exit with an error although the repository itself is fine. The protected files remain in B2 and are cleaned up by a later prune run after their lock has expired.

I do not hide this error in a systemd timer. I log it and alert on it, but distinguish an expected Object Lock deletion failure from a failed backup. Storage use will also be higher than Restic's logical snapshot policy suggests, because locked pack files and old index files stay until B2 allows deletion.

Do not configure a B2 lifecycle rule which removes Restic files earlier than you expect. Object Lock should prevent this while retention is active, but mixing lifecycle cleanup and backup retention makes recovery harder to reason about.

For a broader Linux backup process, including retention planning and recovery procedures, the German Linux backup strategy training from cmt.de is worth a look.

Restore tests are part of the backup

A green backup job only proves that Restic uploaded something. I run repository checks regularly:

# restic check --read-data-subset=5%

I also restore a snapshot into a temporary directory from time to time:

# mkdir -p /var/tmp/restic-restore-test
# restic restore latest --target /var/tmp/restic-restore-test
# find /var/tmp/restic-restore-test -maxdepth 3 -type f | head

For important data I open files from the restore, not only list them. This also verifies that the password, B2 credentials and repository configuration are available when I actually need them.

This off-site copy complements local backups, not replaces them. My backup script for changed files is still useful for quick local recovery, while the B2 repository is the copy I want when a machine, its disks or its local backup storage are gone. I also keep boot recovery in mind, especially on encrypted systems using TPM2 LUKS unlock with Secure Boot.

If you know a better way to handle Restic prune errors with Object Lock, let me know.