Repository navigation
Description
Off-host twice, offsite never
Three artefacts leave the monitoring host now, and every copy of each is on the same shelf under the same roof:
| Set | Made by | Copies | Since |
|---|---|---|---|
| The firewall export | make backup-firewall, nightly |
prometheus, oracle |
2026-09-03 (#92) |
| The volume sets | make backup, weekly |
prometheus, oracle |
2026-09-19 (#535) |
| The NAS set | make backup-nas, weekly |
prometheus, oracle |
2026-09-20 (#484) |
Each of those issues says the same sentence on its way out: off-host is not offsite. #92's status comment calls it "unchanged and still unowned"; roadmap.md carries it as a residual under #92 twice; build-the-nas.md §8 lists it under what the build leaves open. Nothing tracks it, and a residual four documents name and no issue owns is the thing this tracker exists to prevent.
What already exists next to it, and why it is not the same thing
#455 buys a drive kept at another address. That drive is ADR-0023's: the household's copy of Immich and Paperless-ngx, encrypted to a key the operator does not solely hold, opened once from the other person's device. Its reader is the household on a bad day. The three sets above are read by a technical successor on a different bad day, are encrypted to the estate's two age recipients, and are small — a few MB for the export and the NAS set, ~1.7 GB for the volumes. ADR-0024 keeps the two keys apart on purpose (#294's comment of 2026-09-08), so "put it all on the one drive" is a decision with a key question inside it, not a convenience.
The decision
- Same medium as Off-estate copy: ADR-0023's second-person proof #455's, two encryptions. One drive at another address carrying both the household archive and the estate's sets, each to its own recipients. Cheapest; one rotation to keep; the drive's disqualifiers in
back-up-the-age-key.mdapply as written. - A second medium, the one the second age recipient already lives on (ADR-0024). It is off-estate by construction and it visits — and a visit is the cadence the estate's sets could live with, given how little they change.
- Say it stays on the shelf, for the estate's own sets only: a fire that takes both laptops also takes the estate they describe, and a successor rebuilding from nothing would start from the repository, not from
grafana.db. ADR-0023 already accepts the tier being down; this would accept the estate's history being gone. Defensible, and it should be written down rather than arrived at.
Whichever it is: the cadence, the check that proves the far copy is current (verify-backups proves the shelf; nothing proves the other address), and the sentence in roadmap.md that stops saying "unowned".
Done when
An ADR or an amendment to ADR-0015 records where the estate's three sets go beyond the shelf, or that they do not; if they go, a copy has been made and proved once, and successor-handover.md says where it is without saying where it is.
Activity
Gerrrt commentedon Sep 20, 2026
Decided 2026-09-20, and the decision half is written down in #588. What follows is the half only the holder of the medium can do.
The decision
The second answer: the estate's three sets ride on the medium that holds the second age recipient (ADR-0024, #294), carried there by
make backup-offsite DEST=…on the ninety-day visit that medium already owes, and proved by the same visit — ADR-0048. One copy of record, the newest complete set of each kind, inoracle's layout so a restore is "copy the directory back".Why not the other two, briefly. #455's drive has the wrong reader and the wrong key — the household cannot open these sets and #294 keeps the keys apart on purpose — and it does not exist yet; on a bad day the successor would need the drive from one address and the key from another, where on the second recipient's medium they need one thing. The shelf is defensible for the volume sets and wrong for the firewall export, which is the one artefact here with a real rebuild cost and exactly what a successor standing up a replacement
morpheuswants first; the mechanism that carries it costs one directory on a medium that already visits.The cost is stated and accepted in the ADR and in
docs/security.md: that medium is now the one place where the estate's ciphertext sits beside a key that opens it. ADR-0015's refusal to giveoraclethat property stands; its argument turned on a powered, networked host in the same room. Losing the medium already means rotating every credential; the sets add the firewall's rule bodies and WAN address, the export's password hashes, andgrafana.dbto that disclosure.What is built
scripts/backup-offsite.sh— refuses a destination on this host's own filesystem by device number, re-verifies what the medium already holds before writing (the proof that last visit's bytes survived ninety days), copies MANIFEST-last, hashes the copy, keepsOFFSITE_KEEP(default 1) per kind and never the newest. 22 fixtures undermake validate; benched against the real sets to tmpfs.offsite-copyrow beside the two key proofs — no unit, the same ninety days —OffsiteCopyStalewith "mount the medium" advice, excluded fromScheduledJobStale, unit-tested.--list,--verify-onlyand--prunerun unwrapped so inspecting the medium never resets the clock.copy-the-backups-offsite.md, and the sentence inroadmap.mdthat stops saying unowned, plusbuild-the-nas.md§8, both restore runbooks,successor-handover.md(where the sets are, without where the medium is), ADR-0015's forward note.What closes this — one visit, off this host's disk
Read off the host today: the second recipient's proof series is
0, so that medium has not yet made its first visit. When it does, with it mounted on the monitoring host:make backup-offsite DEST=/path/to/the/medium ARGS=--list— what is here, what is there (nothing, the first time).make backup-offsite DEST=/path/to/the/medium— the copy of record. Expect threecopied …lines and the green closing line; a volume set is about 1.7 GB, so the medium needs a few gigabytes free, and the script refuses rather than fills.make secrets-verify-backup KEY=/path/to/the/medium/keys.txt— the visit's other job. Neither proof clears the other.Until then
ScheduledJobNeverRan{homelab_job="offsite-copy"}names the job after the nextmake install-timers, which is the honest state of a copy nobody has made. This issue closes on step 2's first green line, not on the merge.🤖 Generated with Claude Code
Merge origin/main into #577: the new decisions take their place in th…
fix(backup): a different filesystem is not a different machine — /dev…
docs(roadmap): #294's entry names the proof visit it is actually gate…
Merge pull request #609 from Gerrrt/gerrrt/second-age-recipient-sops-…
Gerrrt commentedon Sep 22, 2026
Still the visit, and four documents that had stopped waiting for it
Read off the host today, 2026-09-22 — the position is unchanged from the comment above, and
now says so out loud:
homelab_job_max_age_seconds{homelab_job="offsite-copy"}7776000— declared,install-timershas runhomelab_job_last_success_timestamp_seconds{homelab_job="offsite-copy"}ScheduledJobNeverRan{homelab_job="offsite-copy"}homelab_key_recipient_last_proof_timestamp_seconds{recipient="age1cutv5q…"}0The three sets on the shelf are current — firewall 11 h, NAS 1.4 d, volumes 2.5 d,
verify-backupsgreen 10 h ago — so a visit today would carry current bytes.#610 corrects four documents that described
the copy as already made. The one that mattered was
successor-handover.md: its asset tableand its closing paragraph both placed the newest of each set on the medium in flat present
tense, while the alert row thirty lines above called it "the honest state of a copy nobody
has made". A successor reads the table, not the alert row. Both now carry the qualifier
security.mdalready used, so they are true now and stay true afterwards. Also in it:roadmap.mdsaid #294's recipient had not landed when only its proof has not;README.mdcounted 48 ADRs against 49; and
validate.sh's comment still called/dev/shma destinationthe self-test copies to, which #596 made a refusal fixture. No behaviour changed.
What the audit turned up beside it, now owned rather than noted —
#611 (a copy missing one kind still prints the
green line and records ninety days; compounded by
backup-firewall.sh's cwd-relativeOUT_DIR), #612 (copy_exportsyncs beforewriting the sidecar), #613 (a stale
.partaccumulates, and one shape of it wedges the medium, with no runbook row),
#614 (the self-test reports 24 fixtures when
two silently skip, and no self-test runs in CI). #611 is the one that would actually cost
you: it is the failure mode this whole mechanism exists to prevent, one level up.
This still closes on step 2's first green line. Nothing above makes the copy, and no
removable medium is mounted on the monitoring host. When it is:
from the main checkout. Step 2 closes this; step 3 closes #294's last box; neither clears the
other.
🤖 Generated with Claude Code
Merge pull request #619 from Gerrrt/gerrrt/offsite-missing-kind-no-pr…
Gerrrt commentedon Sep 23, 2026
Done 2026-09-23: the first offsite copy, made and proved
make backup-offsite DEST=/mnt/usbonprometheus, on the medium that holdsage19mkg76v0…(#294, proved the same sitting):One catch, fixed in the same sitting. The archives are encrypted to the recipients the secrets file listed when each backup ran. The volume set
20260920T033007Zpredatesage19mkg76v0…, which was added 2026-09-22 20:56. So the key on the medium could not have opened the volume set beside it: every hash matched, but the ADR-0048 property did not hold. The NAS set (09-22 22:38) and the firewall export (09-23 04:31) came after the key and were fine.The fix was
make backupfor a fresh volume set, thenmake backup-offsiteagain. That replaced the 09-20 set with20260923T213613Z, and a decryption test with the medium's own key passed:So the medium now holds the newest of each kind beside a key that opens all three.
offsite-copyis recorded, andOffsiteCopyStalecounts ninety days from here.Worth knowing at the next visit: the hash check proves the bytes are intact, not that the key beside them opens them. After any recipient change, the next visit should carry sets made after it. The decryption test above is the proof.
🤖 Generated with Claude Code
Gerrrt commentedon Sep 23, 2026
Correction: the firewall export on the medium does not open with the medium's key
The closing comment above says the NAS set and the firewall export "came after the key and were fine". The export was not fine.
backup-firewall.sh'srecipient()took the first key of its.sops.yamlrule only, and passed it tosops --encrypt --age, which replaces the rule's recipient list rather than adding to it. Soconfig-20260923T043134Z.sops.yaml, like every export since 2026-09-09, is encrypted toage1yrdu996…alone.age19mkg76v0…, the key on the medium, cannot open it.Current state of the medium:
20260923T213613Zage -din the sitting20260922T223845Zconfig-20260923T043134ZThe fix is #660:
backup-firewall.shencrypts to every key of the rule, andbackup-offsite.shnow refuses to copy, or to count as proved, any set or export that a current recipient cannot open. Once it's merged:make backup-firewall, then one moremake backup-offsite DEST=…with the medium, and this check to confirm:🤖 Generated with Claude Code
Gerrrt commentedon Sep 23, 2026
Reopened 2026-09-23: the firewall export on the medium (
config-20260923T043134Z) is encrypted toage1yrdu996…only, so the medium's key cannot open it (correction above). This closes when #660 is merged, a fresh export is made withmake backup-firewall,make backup-offsitecarries it, andSOPS_AGE_KEY_FILE=<medium>/keys.txt sops -d … > /dev/nullopens it.Gerrrt commentedon Sep 23, 2026
Closed again 2026-09-23: all three kinds on the medium open with its key
After #660 (exports encrypted to every recipient;
backup-offsite.shrefuses a set a current key cannot open), a fresh export was made and carried:age19mkg76v0…20260923T213613Zage -din the sitting20260922T223845Zconfig-20260923T221711Zsops -daboveSince #660, a set that a current recipient cannot open is named
NOT heldand records no proof. So the green line on this copy now carries the property this issue exists for, not just matching hashes.🤖 Generated with Claude Code
fix(secrets): the lab's secrets live in git, not only on alexander (#667