Bare-Metal Restore: Rehearsing the Recovery You Dread

Every backup strategy I have ever written about on this site — Borg, Restic, Kopia, the whole 3-2-1 discipline — answers the question “can I get my files back.” None of them, by themselves, answer a harder question: if the machine itself is gone, cratered motherboard, stolen chassis, a fire that took the whole rack, how long until a replacement box is running the same services again, and does anyone actually know the steps. That is bare-metal restore, and it is the drill almost nobody rehearses, because rehearsing it means deliberately wrecking a working system to prove you can bring it back.

I do this drill twice a year on my most important box, and the first time I ran it I discovered my “backup” restored files perfectly onto an OS that had three packages missing, a different kernel version’s /etc/fstab syntax, and a systemd unit referencing a device path that no longer existed after a fresh install enumerated disks in a different order. The data was fine. The machine was not bootable. This post covers the tools and the runbook that fixed that.

Why file-level backup is not the same as bare-metal recovery

Restic, Borg and Kopia back up files. A bare-metal restore needs the whole picture: partition table, bootloader, kernel and initramfs matching the actual hardware, package selection, and every piece of configuration that lives outside a neat /home or /etc — LUKS headers, RAID metadata, network interface naming, and the quiet assumptions a running system makes about its own disk layout that only become visible when they are wrong. A file-level restore onto a freshly installed OS gets you your data back. It does not get you your server back, and the difference is measured in hours or days of manual reassembly exactly when you have the least patience for it.

There are two broad approaches: image the whole disk (Clonezilla) or script a bootable rebuild from a file-level backup plus a recorded system layout (Relax-and-Recover, ReaR). Both are worth knowing; they solve slightly different problems.

Approach 1: full-disk imaging with Clonezilla

Clonezilla boots from USB, images an entire disk or partition set block-by-block (with used-space-only imaging for supported filesystems, so it does not need to copy every empty sector), and can restore that image to identical or similar hardware. It is the right tool when “identical hardware” is a reasonable assumption — a homelab server you’d replace with the same model, or a VM you’d redeploy from a known image.

1
2
3
4

# Boot the target machine from a Clonezilla Live USB, then from the menu:
# device-image -> local_dev -> select the backup destination
# -> savedisk -> name the image -> select the source disk (e.g. sda)
# -> beginner mode -> -q2 (use partclone, the default and fastest)

The interactive menu is the normal way to drive it, but for scripted, unattended runs Clonezilla also has an ocs-sr command-line mode:

1

sudo ocs-sr -q2 -c -j2 -z1p -i 4096 -sfsck -p poweroff savedisk my-server-2024-07 sda

That backs up sda to an image named my-server-2024-07, compresses it, checks filesystems first, and powers off when done — the flags I actually use for an unattended nightly image of a small VM. Restoring is the mirror image, restoredisk in place of savedisk, onto a disk of equal or greater size.

Clonezilla’s honest limitation: it restores disk images, so a restore onto meaningfully different hardware (a different disk controller needing a different kernel driver, a different disk size requiring partition resizing) is not guaranteed to boot cleanly. It is superb for “this exact box died, I have an identical replacement” and less suited to “the building burned down and I’m rebuilding on whatever hardware I can source next week.”

Approach 2: scripted rebuild with Relax-and-Recover (ReaR)

ReaR takes a different angle: instead of imaging the whole disk, it captures the system layout (partition scheme, LVM, RAID, bootloader configuration) into a small recovery ISO, and restores the actual file data from your existing backup tool — Restic, Borg, rsync, tar, whatever you already run. That split matters because it decouples the two problems: your regular file-level backups stay your regular file-level backups, and ReaR is only responsible for making a new disk bootable and structured the same way, then handing off to your restore tool for the data.

Install and configure it on Debian/Ubuntu or RHEL-family systems:

1

sudo apt install rear

1
2
3
4
5
6

# /etc/rear/local.conf
OUTPUT=ISO
BACKUP=RESTIC
RESTIC_REPOSITORY="s3:https://s3.eu-central-1.example.com/homelab-offsite"
RESTIC_PASSWORD_FILE="/etc/rear/restic-password"
OUTPUT_URL="file:///mnt/backup-local/rear"

Generate the recovery ISO and (with BACKUP=RESTIC) trigger the underlying Restic backup in the same run:

1

sudo rear mkbackup

That produces a small bootable ISO in /mnt/backup-local/rear containing everything needed to recreate the disk layout, plus a Restic snapshot of the actual data taken at the same time. To restore: boot the recovery ISO on the new hardware, and from its rescue shell run:

1

rear recover

ReaR reads its embedded layout, repartitions the new disk to match (prompting if the new disk genuinely can’t match, e.g. it’s smaller), and then pulls the file data back from the Restic repository automatically. The result, on hardware that is at least broadly compatible, is a machine that boots into the same OS and configuration it had at backup time — this is the closest a homelab gets to the enterprise “boot from recovery media, walk away” restore experience.

Approach 3: a manual bootstrap from a plain file-level backup

Not every box justifies Clonezilla or ReaR’s setup overhead. For smaller VMs, a documented manual bootstrap from a Restic (or Kopia) backup is often good enough, provided the runbook is actually written down and actually tested:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

# 1. Install a minimal OS matching the original distro/version
# 2. Install the backup client
sudo apt install restic
# 3. Restore /etc first and diff it against the fresh install's /etc
#    before overwriting, since a blind overwrite can break the new
#    install's own package-manager state files
restic -r "$RESTIC_REPOSITORY" restore latest --target /mnt/restore --include /etc
diff -rq /mnt/restore/etc /etc
# 4. Restore data directories
restic -r "$RESTIC_REPOSITORY" restore latest --target / --include /home --include /var/www
# 5. Reinstall the package list captured at backup time
sudo dpkg --set-selections < /mnt/restore/etc/package-selections.txt
sudo apt-get dselect-upgrade

Step 5 only works if you were capturing dpkg --get-selections > /etc/package-selections.txt as part of the original backup — a small addition to any nightly job that saves hours of “which packages did this box actually have” archaeology later. This manual path is slower than ReaR and more error-prone, but it needs nothing extra installed on the source machine beyond what you already run, which makes it the pragmatic choice for a fleet of small, low-stakes containers or VMs where a full ReaR setup on each one isn’t worth the effort.

Why “identical hardware” quietly stops being true

A detail that catches people out on their second rehearsal, having passed their first: the assumption that a replacement box is “the same machine” tends to erode faster than expected. A server bought two or three years ago and its identically-specced replacement, ordered today because the original died, often ship with a different disk controller revision, a newer NVMe firmware with different sector reporting, or a network card that enumerates under a different driver and therefore a different interface name. None of that shows up until you are mid-restore and GRUB can’t find the disk it expects, or eth0 never comes up because the new card is enp3s0. This is not a hypothetical: it is the single most common reason a “should just work” bare-metal restore stalls, and it is precisely why a stale rehearsal is worse than an honest “we’ve never tested this” — a runbook that passed two years ago on hardware that no longer exists gives false confidence rather than none.

The fix is not exotic. Predictable network interface naming via systemd’s naming scheme, rather than pinning to a specific MAC in /etc/network/interfaces or a udev rule, survives a hardware swap far better. Booting with UEFI rather than legacy BIOS avoids a class of partition-table assumptions that differ between the two. And noting, in the runbook itself, which parts of the configuration are hardware-specific and therefore expected to need hand-editing after a restore onto different hardware, saves the person running the drill from wasting an hour convinced they’ve made a mistake when the runbook itself was always going to need a manual step at that point.

The rehearsal that goes wrong on purpose

The most useful version of this drill is not the one where everything works. Once a year, alongside the “does this succeed” rehearsal, I deliberately restore onto hardware that is not a close match — an old scratch machine with a different disk size and a different network card — specifically to find out which parts of the runbook are hardware-specific assumptions dressed up as universal steps. It nearly always surfaces something: a hard-coded disk device path, a partition size that no longer fits, a driver that isn’t in the target kernel’s default module set. Finding that on a Saturday afternoon against a disposable scratch box costs nothing. Finding it during an actual outage, against hardware you sourced in a hurry because the original box is gone, costs the entire point of having rehearsed at all.

The runbook: what to actually write down

None of these tools help if the knowledge of how to invoke them lives only in your head. A runbook for each critical machine should record, in a single document kept outside that machine (a wiki on a different host, or printed and filed, genuinely):

• Which restore method applies (Clonezilla image, ReaR, or manual bootstrap) and where the image/ISO/repository lives.
• The exact hardware assumptions (disk size, controller, whether the target must be identical hardware or can be “close enough”).
• Any manual post-restore steps: re-issuing a LUKS unlock key, re-registering with a licence server, updating a DNS record to point at the new machine’s address, reinstalling a hypervisor-specific guest agent.
• Who is expected to run this, and roughly how long it should take — a number you only get by actually timing a rehearsal.

Rehearsal cadence

A restore procedure that has never been executed is a hypothesis. I run a full bare-metal rehearsal twice a year on the machine I would miss most (currently a small server running DNS, the reverse proxy, and a couple of self-hosted apps), and a lighter manual-bootstrap check quarterly on everything else. The rehearsal is not a dry run on paper — it is genuinely wiping a spare disk or spinning up a scratch VM and running the real restore procedure against real backup data, timing it, and writing down every place the runbook was wrong or out of date. Every single rehearsal I have run has found at least one stale instruction: a changed IP range, a renamed disk, a config path that moved after an upgrade. That is the entire point of rehearsing — catching the drift while it is a mildly annoying Saturday afternoon exercise, not catching it during an actual outage.

Troubleshooting

Clonezilla restore boots to a GRUB rescue prompt. Usually a disk-size or partition-order mismatch between source and target. Boot the Clonezilla ISO again, mount the restored partition manually, chroot in, and run grub-install and update-grub against the new disk to regenerate the bootloader for its actual geometry.

ReaR’s rear mkbackup fails validation before it even starts. Run rear -D mkbackup for verbose debug output; the most common cause is a filesystem or LVM layout ReaR’s layout-detection code doesn’t recognise cleanly (some unusual multi-disk Btrfs setups, for instance) — check the ReaR compatibility matrix for your storage stack before relying on it as your only method.

The manual bootstrap restores /etc and networking breaks. This is the classic “restored config assumes hardware that no longer exists” problem — a /etc/udev/rules.d/70-persistent-net.rules or a systemd .link file pinning an interface name to a specific MAC address. Diff before overwriting (step 3 above exists specifically for this) and hand-merge network configuration rather than blindly restoring it.

A restore rehearsal takes so long nobody wants to repeat it. That is useful data, not a reason to skip the next one — it tells you your actual recovery time objective, and if it’s longer than your business or household can tolerate, that’s the finding the rehearsal exists to produce. Trim the rehearsal scope (test the riskiest machine fully, spot-check the rest) rather than abandoning the practice.

LUKS-encrypted disks won’t unlock after a ReaR or Clonezilla restore. The recovery media needs the unlock key or keyfile available to it, which is a chicken-and-egg problem if the key was only ever stored on the machine that just died. Keep an offline copy of LUKS headers and unlock keys as part of the same backup discipline covered in headless LUKS with remote unlock — a restore procedure that depends on a secret that only existed on the dead machine is not a restore procedure.

Verdict: is it worth the effort?

For anything you would genuinely be in trouble without — the machine running your DNS, your reverse proxy, your family’s photo library — yes, unambiguously. The gap between “I have backups” and “I can bring the machine back” is exactly the gap that turns a bad day into a bad week, and it only shows up when you deliberately go looking for it. Clonezilla is the quick, low-effort choice for identical-hardware replacement. ReaR is the more capable option when you want a genuinely bootable recovery image decoupled from your existing file-level backup tool. A documented manual bootstrap covers everything else at low setup cost. Pick based on how much that particular machine would hurt to lose, write the runbook down somewhere that survives the machine dying, and actually run the drill — twice a year, on a spare disk, for real. It is the single highest-value hour you can spend on backups after getting the backups themselves working.