Repository navigation
feat(backup): the volume sets leave prometheus — every run copies them to oracle and verify-backups checks they are still there (#535) - #550
Merged
Conversation
…m to oracle and verify-backups checks they are still there (#535) For three weeks `make backup` ended by printing "copy the set to the backup target — see #92", and nothing did. ADR-0015 decided the destination and left the copying to build; this builds it on the model backup-firewall.sh set. The copy is a step of the same job, after the set is written, verified and the stack is back up: a run whose copy fails exits non-zero, so the weekly timer's ScheduledJobFailed is also the alarm for "the sets have stopped leaving this host". The daily verify-backups run asks oracle to hash every archive it holds and compares the answer with each MANIFEST — the column was computed on the bytes verify() had just decrypted, so it is the firewall's pull-back proof without the wire time — and pulls each MANIFEST back byte for byte. No new unit, lock, metric or rule: backup.rules.yaml names no job on purpose, and the generic pair carries both probes. Far side is atropos@10.0.99.30:backups/volumes/<STACK> (VOL_OFFHOST), per stack because trinity runs the same script against the same oracle. One retention knob, KEEP, both sides, with the clause that keeps it correct: the far side never removes a set this host still holds, so prune()'s newest-quiesced exception cannot become a weekly re-upload. Everything sent over the wire is a validated stamp or volume name in single quotes — oracle's login shell is zsh with nomatch on. New flags: --local-only, --copy-only (the seed, and the catch-up after oracle was off; never stops the stack). KEEP is now rejected unless a positive integer, and every mode that reads or changes a set takes the lock, so a hand seed queues the daily verify rather than racing it. restore-volumes.sh passes --local-only: a restore happens when things are broken. Benched from a worktree against a scratch directory on oracle: a 1.1 GB set copies and hashes clean in under two minutes; a missing directory, a missing MANIFEST, a corrupted archive and a dead incomplete directory each produce the exit code and message the runbook describes, and strays are never touched. The restore runbook now starts from the copy on oracle when prometheus is what died. The sets are not yet seeded there; that is the deploy step. Refs #535, refs #92. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
make backupnow copies every complete volume set tooracleas a step of the same run, after the set is written, verified and the stack is back up —atropos@10.0.99.30:backups/volumes/<STACK>by default,VOL_OFFHOSTto change it — and fails if it cannot, so the weekly timer'sScheduledJobFailedis also the alarm for "the sets have stopped leaving this host". The dailyverify-backupsrun asksoracleto hash every archive it holds, compares the answer with each set's MANIFEST (the column was computed on the bytesverify()had just decrypted, so this is the firewall copy's pull-back proof without the wire time) and pulls each MANIFEST back byte for byte. One retention knob,KEEP, both sides, with the clause that keeps it correct: the far side never removes a set this host still holds, soprune()'s newest-quiesced exception cannot become a weekly re-upload.scripts/backup-volumes.sh: the far-side section (same transport and zsh-safe vocabulary asbackup-firewall.sh, deliberately duplicated rather than shared),--local-only,--copy-only(the seed and the catch-up; never stops the stack),--listand--pruneon both sides,KEEPandVOL_OFFHOSTvalidated before any mode, and every mode that reads or changes a set now takes the lock.scripts/restore-volumes.sh: passes--local-onlyto its pre-restore verify — a restore happens when things are broken.backup.rules.yamlnames no job on purpose and the generic pair carries both probes. Counted claims are unchanged.restore-the-stack.mdstarts from the copy onoraclewhenprometheusis what died;schedule-maintenance.md,observability.md,roadmap.md,architecture.md,network.md,README.md, the sensitive-tier runbook and both unit headers say the sets leave.Why
Closes the second half of #92 that ADR-0015 decided and left to build: a set on the disk it protects is a copy, not a backup. Refs #535, refs #92.
While benching this I found the live 2026-09-13 run failed on the loki-data sentinel; that is #468, already fixed by #469 on 2026-09-14, and the live checkout has it. The incomplete
20260913T033005Zset onprometheusis still there for a hand removal, as the script's own warning asks.Blast radius
secrets/*.sops.yaml—oraclereceives age ciphertext only; the samerobokeybackup-firewallusesVerification
make validatepassesbackups/volumes-benchonoracle: a 1.1 GB set copies and hashes clean in 1:53; a missing directory, a missing MANIFEST, a corrupted archive (reported as differs with the repair named), a dead incomplete directory (pruned) and strays (warned, never touched) each behave as the runbook says;KEEP=0, a..inVOL_OFFHOSTand--copy-only --hotare rejected. Scratch directory removed afterwards.make backup ARGS=--copy-onlyseeds the five complete sets (~5.5 GB, about ten minutes), then a hand run ofhomelab-verify-backups.service; the first same-run copy is Sunday 03:30.🤖 Generated with Claude Code