RECOVERY ENGINEERING / PRINT EDITION

Rsnapshot Restoration Plan

Rebuild an independent, testable backup system for the Arch host while retaining the predecessor snapshot root and using its configuration as guidance, not as a template.

Working decisions: retain /mnt/stag/rsnapshot/ and use its existing interval hierarchy as the seed for a new rolling history. Preserve hard links and historical recovery points by making only verified, same-filesystem renames. Replace stale paths and unsafe live database copying. After each approved Brooklyn backup attempt, attempt to suspend Brooklyn through a dedicated restricted SSH key, while respecting Brooklyn's normal system sleep inhibitors.

1. Design Position

Keep

rsnapshot plus rsync hard-link rotations on a separate local disk.
Daily, weekly, monthly, and yearly recovery points.
Filesystem backup of configuration, web data, Maildir, and application state.
A dedicated logical database export captured before the snapshot.

Change

Retain the existing snapshot root; protect it with a manifest, verified namespace mapping, and same-filesystem moves only.
Use systemd timers, mount guards, locking, capacity checks, and failure units.
Replace the legacy MySQL dump with a consistent MariaDB 12 export.
Require explicit approval before remote pulls, Tide data, or power actions.

Not a substitute for Snapper: Snapper provides quick local Btrfs rollback. Rsnapshot provides independently stored, versioned file recovery on the separate /mnt/stag disk. A later off-host replica remains required for disaster recovery.

2. Evidence From The Legacy System

Legacy fact	Observed configuration	Modern interpretation
Snapshot root	`/mnt/stag/rsnapshot/`	RETAIN Keep the existing root and use its interval trees as a verified rolling seed. Never copy or move them across filesystems.
Retention	7 daily, 4 weekly, 12 monthly, 4 yearly	KEEP Start with the same recovery horizon, then revisit after real capacity measurements.
Schedule	The packaged `/etc/cron.d/rsnapshot` sample existed but every run line was commented.	CHANGE Use explicit systemd timers with `Persistent=true`; Cronie remains available for user cron jobs.
Database hook	`mysqldump --all-databases`	CHANGE Use a restricted backup account and `mariadb-dump --single-transaction --routines --events`.
Post hook	SSH command suspended `brooklyn` after each snapshot.	REPLACE The approved Brooklyn-specific job will attempt suspend after every backup attempt, success or failure, using a restricted key and explicit local logging. Remote `systemctl suspend` keeps its normal inhibitor behavior.
Remote pulls	Data was pulled from `brooklyn` and `cake` using Jim's SSH key.	CONFIGURE NOW Retain both source sets in the active configuration. Cake is reachable; Brooklyn is configured with Wake-on-LAN and is an immediate availability blocker, not deferred scope.

3. Recovery Map

The diagram separates data already local to this host from optional remote/legacy scopes. Solid lines are planned for the first usable backup. Dashed lines require a separate approval.

4. Retained-Root Namespace Migration

The existing snapshot directories are not merely old files. Rsnapshot links files across interval trees, so they are part of the recovery structure. The migration changes the configuration only after it proves where current data belongs in that structure.

Safety gate: no rsnapshot timer may run until the migration manifest, dry-run comparisons, destination decisions, and rotation choice below are approved. A normal rsnapshot rotation can remove the oldest interval directory, so an unreviewed first run is destructive to historical recovery points.

Existing snapshot namespace	Candidate current source	Migration treatment
`bonecrusher/etc/`	`/etc/`	RETAIN Compare, then continue using the existing canonical namespace.
`bonecrusher/srv/`	`/srv/`	RETAIN Compare current restored site content before first sync.
`bonecrusher/opt/`	`/opt/`	AUDIT Include only application paths that remain authoritative.
`bonecrusher/usr/local/`	`/usr/local/`	RETAIN Compare and preserve local tools/configuration.
`bonecrusher/var/www/`	`/var/www/`	RETAIN Compare and continue in the canonical web-content namespace.
`bonecrusher/var/mail/`	`/var/mail/`	RETAIN Compare Maildir metadata and message content before enabling sync.
`bonecrusher/var/spool/`	`/var/spool/`	AUDIT Include selected application spools, not transient queues blindly.
`bonecrusher/var/local/`	`/var/local/`	RETAIN Compare and include if present.
`bonecrusher/mysql/`	root-only MariaDB export staging	REPLACE CONTENT Retain the logical namespace but replace the legacy dump with a consistent current export.
`bonecrusher/var/lib/geneweb/`	GeneWeb absent	PRESERVE ONLY Do not sync until the backend returns and its authority is confirmed.
`bonecrusher/mnt/anvil/{mail,spool,www}/`	possible predecessor-mounted duplicates	COMPARE FIRST Do not rename or merge based on path similarity; establish whether each tree duplicates the canonical `var` subtree.
`bonecrusher/mnt/{sternum,hammer}/`	legacy archive disks	SEPARATE POLICY Retain history; include only after an authority and capacity decision.
`brooklyn/`	approved remote backup source	DEDICATED JOB After each backup attempt, log the result and attempt remote suspend through a forced-command key. A remote inhibitor may deliberately prevent sleep.
`cake/`	active remote backup source	CONFIGURE NOW Retain legacy home, Apache, web-root, and systemd sources. Cake is reachable and its configured source paths are present.
`laundry/`	legacy remote source	UNTOUCHED No pull or rename until separately approved.

Required migration method

Create a root-only manifest for every daily.*, weekly.*, monthly.*, and yearly.* tree. Record source namespace, candidate live source, size, newest modification time, owner, verification result, and decision.
Run non-destructive rsync --dry-run comparisons. A matching path name is insufficient evidence that the current and legacy datasets are equivalent.
For each approved path change, rename that subtree identically in every interval tier using same-filesystem mv within /mnt/stag/rsnapshot/. This preserves inode links; copying or cross-filesystem moves are prohibited.
Record every approved old-to-new mapping in the manifest and configuration comments so historical restores retain source provenance.
Configure rsnapshot to write the verified canonical namespace, then dry-run it before allowing any rotation.

5. First Backup Scope

Scope	Initial treatment	Reason
Configuration	INCLUDE `/etc`, `/root`, `/usr/local`, selected `/opt`	Required to rebuild services, cron, Apache, Keycloak, Postfix, and backup automation.
Content and mail	INCLUDE `/var/www`, `/srv`, `/var/mail`, application upload/data paths after audit	Contains restored web assets, static content, and Maildir.
MariaDB	DUMP into root-only staging, then snapshot the dump	Never use rsnapshot to copy live InnoDB data files as the database backup.
Docker	AUDIT bind mounts and named volumes; exclude overlay layers and images	Container images are reproducible. Persistent data is not.
Runtime/cache	EXCLUDE `/proc`, `/sys`, `/dev`, `/run`, `/tmp`, caches, Snapper snapshots, destination tree	Prevents recursion, noise, unsafe pseudo-files, and unnecessary capacity use.
`/mnt/sternum`	PRESERVE AND DEFER	Its historical snapshot namespace remains in place. A new sync policy needs a source-authority decision and capacity measurement before it is enabled.

6. Execution Phases

PHASE 0 Freeze rsnapshot scheduling. Inventory active application data, Docker volumes, MariaDB size, source sizes, and the exact mount identity for /mnt/stag. Establish approved free-space and failure thresholds before changing the retained tree.

PHASE 1 Build and approve the retained-root migration manifest. Compare every candidate current source against its legacy namespace with non-destructive rsync runs. Resolve duplicate var and mnt/anvil trees before configuration exists.

PHASE 2 Apply only approved same-filesystem subtree renames across every interval tier. Preserve historical provenance in the manifest. Do not copy data, collapse unverified duplicates, or rotate interval directories.

PHASE 3 Install rsnapshot and configure it against /mnt/stag/rsnapshot/. Add a preflight that rejects an unmounted or low-space target rather than writing into an accidental local directory. Use one_fs=1; add mounted data deliberately.

PHASE 4 Create a dedicated MariaDB backup credential and pre-snapshot service. Produce a consistent logical export with routines and events, validate completion, and retain root-only access. Use the existing bonecrusher/mysql namespace for the new export.

PHASE 5 Add systemd rsnapshot@daily, weekly, monthly, and yearly services/timers. Use Persistent=true, flock, ordering after the database export, and independent log/failure reporting. Initial schedule: daily after Jim's 03:28 Maildir sync. Implement Brooklyn as a separate job with a forced-command SSH key; after every completed attempt it records backup status, attempts remote systemctl suspend, and records whether Brooklyn slept or an inhibitor prevented it.

PHASE 6 Run rsnapshot configtest, dry-run the sync, and verify expected transfer volume. Perform a documented file, Maildir, configuration, and isolated MariaDB restore drill before the first rotation.

PHASE 7 Enable rolling retention only after the restore drill succeeds. Brooklyn and Cake are already active configured sources: Cake can run when an interval is approved; Brooklyn first needs restored LAN reachability. Add Tide, legacy archive updates, and off-host replication only through later scoped change sets.

7. Timer And Retention Policy

Approval point: rolling continuity is the working recommendation. Existing interval trees become the initial live history and age out under normal 7/4/12/4 retention. If historical intervals must never age out, first freeze them under a separate legacy-2026/ namespace. That alternative needs a new baseline and materially more capacity.

Interval	Retention	Planned cadence	Guardrails
daily	7	04:30 local time	After Jim's 03:28 Maildir sync; database export must succeed first.
weekly	4	Sunday 05:15	Never overlap with daily; same mount and capacity preflight.
monthly	12	Day 1, 06:00	Evaluate destination growth; include the separate legacy-archive run only after capacity approval.
yearly	4	January 1, 07:00	Restore sample before relying on the oldest tier.

8. Controls That Make This A Backup

Independent storage: the target is on /mnt/stag, not the root Btrfs disk.
Consistency: database exports are completed and checked before file snapshotting begins.
Failure visibility: timers log to journald and fail explicitly. Mail alerts remain secondary until outbound mail reputation is repaired.
Capacity discipline: no automatic first copy of the large predecessor archive; alert before the target approaches its approved threshold.
Least privilege: root-owned configuration, root-only database dump credentials, and dedicated noninteractive keys for any later remote source.
Recovery proof: a scheduled restore drill is a completion requirement, not a future aspiration.

9. Explicit Deferrals

Item	State	Condition to enable
Brooklyn remote pull and suspend	APPROVED DESIGN	Implement after source/reachability audit using a dedicated forced-command key. Suspend is attempted after every backup attempt, including failures, but no inhibitor is bypassed.
Tide/laundry sources	DEFERRED	Repair the failed Tide mount and decide which data is authoritative rather than duplicated.
Legacy `/mnt/sternum` archive	DEFERRED	Measure its baseline, reserve capacity, make one verified copy, then schedule monthly updates.
Frozen legacy history	DECISION REQUIRED	Choose rolling continuity or a permanent `legacy-2026/` namespace before the first interval rotation.
Off-host backup	NEXT DESIGN	Select encrypted external or remote storage after the local restore drill succeeds.

Revised 2026-08-08. Prepared from the legacy /mnt/debian/etc/rsnapshot.conf, its database and post-snapshot hooks, the retained snapshot tree, current mounted-storage observations, and the restored service inventory. This document is a controlled implementation plan, not evidence that rsnapshot has been installed or scheduled.