Skip to content

feat(backup): the volume sets leave prometheus — every run copies them to oracle and verify-backups checks they are still there (#535) - #550

Merged
Gerrrt merged 1 commit into
mainfrom
gerrrt/backup-volumes-oracle-beb3b2
Sep 19, 2026
Merged

Gerrrt merged 1 commit into
mainfrom
gerrrt/backup-volumes-oracle-beb3b2

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Sep 19, 2026

Copy link
Copy Markdown
Owner

What changed

make backup now copies every complete volume set to oracle as a step of the same run, after the set is written, verified and the stack is back up — atropos@10.0.99.30:backups/volumes/<STACK> by default, VOL_OFFHOST to change it — and fails if it cannot, so the weekly timer's ScheduledJobFailed is also the alarm for "the sets have stopped leaving this host". The daily verify-backups run asks oracle to hash every archive it holds, compares the answer with each set's MANIFEST (the column was computed on the bytes verify() had just decrypted, so this is the firewall copy's pull-back proof without the wire time) and pulls each MANIFEST back byte for byte. One retention knob, KEEP, both sides, with the clause that keeps it correct: the far side never removes a set this host still holds, so prune()'s newest-quiesced exception cannot become a weekly re-upload.

  • scripts/backup-volumes.sh: the far-side section (same transport and zsh-safe vocabulary as backup-firewall.sh, deliberately duplicated rather than shared), --local-only, --copy-only (the seed and the catch-up; never stops the stack), --list and --prune on both sides, KEEP and VOL_OFFHOST validated before any mode, and every mode that reads or changes a set now takes the lock.
  • scripts/restore-volumes.sh: passes --local-only to its pre-restore verify — a restore happens when things are broken.
  • No new unit, lock, metric or alert rule: backup.rules.yaml names no job on purpose and the generic pair carries both probes. Counted claims are unchanged.
  • restore-the-stack.md starts from the copy on oracle when prometheus is what died; schedule-maintenance.md, observability.md, roadmap.md, architecture.md, network.md, README.md, the sensitive-tier runbook and both unit headers say the sets leave.

Why

Closes the second half of #92 that ADR-0015 decided and left to build: a set on the disk it protects is a copy, not a backup. Refs #535, refs #92.

While benching this I found the live 2026-09-13 run failed on the loki-data sentinel; that is #468, already fixed by #469 on 2026-09-14, and the live checkout has it. The incomplete 20260913T033005Z set on prometheus is still there for a hand removal, as the script's own warning asks.

Blast radius

  • No segmentation or firewall change
  • No new published port
  • No credential outside secrets/*.sops.yaml — oracle receives age ciphertext only; the same robo key backup-firewall uses

Verification

  • make validate passes
  • Benched from the worktree against backups/volumes-bench on oracle: a 1.1 GB set copies and hashes clean in 1:53; a missing directory, a missing MANIFEST, a corrupted archive (reported as differs with the repair named), a dead incomplete directory (pruned) and strays (warned, never touched) each behave as the runbook says; KEEP=0, a .. in VOL_OFFHOST and --copy-only --hot are rejected. Scratch directory removed afterwards.
  • Deployed: after merge, on the main checkout, make backup ARGS=--copy-only seeds the five complete sets (~5.5 GB, about ten minutes), then a hand run of homelab-verify-backups.service; the first same-run copy is Sunday 03:30.
  • Docs updated

🤖 Generated with Claude Code

…m to oracle and verify-backups checks they are still there (#535)

For three weeks `make backup` ended by printing "copy the set to the backup
target — see #92", and nothing did. ADR-0015 decided the destination and left
the copying to build; this builds it on the model backup-firewall.sh set.

The copy is a step of the same job, after the set is written, verified and
the stack is back up: a run whose copy fails exits non-zero, so the weekly
timer's ScheduledJobFailed is also the alarm for "the sets have stopped
leaving this host". The daily verify-backups run asks oracle to hash every
archive it holds and compares the answer with each MANIFEST — the column was
computed on the bytes verify() had just decrypted, so it is the firewall's
pull-back proof without the wire time — and pulls each MANIFEST back byte for
byte. No new unit, lock, metric or rule: backup.rules.yaml names no job on
purpose, and the generic pair carries both probes.

Far side is atropos@10.0.99.30:backups/volumes/<STACK> (VOL_OFFHOST), per
stack because trinity runs the same script against the same oracle. One
retention knob, KEEP, both sides, with the clause that keeps it correct: the
far side never removes a set this host still holds, so prune()'s
newest-quiesced exception cannot become a weekly re-upload. Everything sent
over the wire is a validated stamp or volume name in single quotes — oracle's
login shell is zsh with nomatch on. New flags: --local-only, --copy-only (the
seed, and the catch-up after oracle was off; never stops the stack). KEEP is
now rejected unless a positive integer, and every mode that reads or changes
a set takes the lock, so a hand seed queues the daily verify rather than
racing it. restore-volumes.sh passes --local-only: a restore happens when
things are broken.

Benched from a worktree against a scratch directory on oracle: a 1.1 GB set
copies and hashes clean in under two minutes; a missing directory, a missing
MANIFEST, a corrupted archive and a dead incomplete directory each produce
the exit code and message the runbook describes, and strays are never
touched.

The restore runbook now starts from the copy on oracle when prometheus is
what died. The sets are not yet seeded there; that is the deploy step.

Refs #535, refs #92.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@Gerrrt
Gerrrt merged commit 0687063 into main Sep 19, 2026
3 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/backup-volumes-oracle-beb3b2 branch September 19, 2026 22:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant