Skip to content

The estate's three backup sets are off-host twice and offsite never: decide where they go beyond the shelf, or that they do not #573

Closed

Description

@Gerrrt

Off-host twice, offsite never

Three artefacts leave the monitoring host now, and every copy of each is on the same shelf under the same roof:

Set Made by Copies Since
The firewall export make backup-firewall, nightly prometheus, oracle 2026-09-03 (#92)
The volume sets make backup, weekly prometheus, oracle 2026-09-19 (#535)
The NAS set make backup-nas, weekly prometheus, oracle 2026-09-20 (#484)

Each of those issues says the same sentence on its way out: off-host is not offsite. #92's status comment calls it "unchanged and still unowned"; roadmap.md carries it as a residual under #92 twice; build-the-nas.md §8 lists it under what the build leaves open. Nothing tracks it, and a residual four documents name and no issue owns is the thing this tracker exists to prevent.

What already exists next to it, and why it is not the same thing

#455 buys a drive kept at another address. That drive is ADR-0023's: the household's copy of Immich and Paperless-ngx, encrypted to a key the operator does not solely hold, opened once from the other person's device. Its reader is the household on a bad day. The three sets above are read by a technical successor on a different bad day, are encrypted to the estate's two age recipients, and are small — a few MB for the export and the NAS set, ~1.7 GB for the volumes. ADR-0024 keeps the two keys apart on purpose (#294's comment of 2026-09-08), so "put it all on the one drive" is a decision with a key question inside it, not a convenience.

The decision

  • Same medium as Off-estate copy: ADR-0023's second-person proof #455's, two encryptions. One drive at another address carrying both the household archive and the estate's sets, each to its own recipients. Cheapest; one rotation to keep; the drive's disqualifiers in back-up-the-age-key.md apply as written.
  • A second medium, the one the second age recipient already lives on (ADR-0024). It is off-estate by construction and it visits — and a visit is the cadence the estate's sets could live with, given how little they change.
  • Say it stays on the shelf, for the estate's own sets only: a fire that takes both laptops also takes the estate they describe, and a successor rebuilding from nothing would start from the repository, not from grafana.db. ADR-0023 already accepts the tier being down; this would accept the estate's history being gone. Defensible, and it should be written down rather than arrived at.

Whichever it is: the cadence, the check that proves the far copy is current (verify-backups proves the shelf; nothing proves the other address), and the sentence in roadmap.md that stops saying "unowned".

Done when

An ADR or an amendment to ADR-0015 records where the estate's three sets go beyond the shelf, or that they do not; if they go, a copy has been made and proved once, and successor-handover.md says where it is without saying where it is.

Refs #92, #455, #535, #484, ADR-0015, ADR-0023, ADR-0024.

Activity

  1. added this to the Nothing blocks these milestone on Sep 20, 2026
  2. added
    securityA credential, an exposure, or a control that is weaker than recorded
    on Sep 20, 2026
  3. Gerrrt commented on Sep 20, 2026

    @Gerrrt
    OwnerAuthor

    Decided 2026-09-20, and the decision half is written down in #588. What follows is the half only the holder of the medium can do.

    The decision

    The second answer: the estate's three sets ride on the medium that holds the second age recipient (ADR-0024, #294), carried there by make backup-offsite DEST=… on the ninety-day visit that medium already owes, and proved by the same visit — ADR-0048. One copy of record, the newest complete set of each kind, in oracle's layout so a restore is "copy the directory back".

    Why not the other two, briefly. #455's drive has the wrong reader and the wrong key — the household cannot open these sets and #294 keeps the keys apart on purpose — and it does not exist yet; on a bad day the successor would need the drive from one address and the key from another, where on the second recipient's medium they need one thing. The shelf is defensible for the volume sets and wrong for the firewall export, which is the one artefact here with a real rebuild cost and exactly what a successor standing up a replacement morpheus wants first; the mechanism that carries it costs one directory on a medium that already visits.

    The cost is stated and accepted in the ADR and in docs/security.md: that medium is now the one place where the estate's ciphertext sits beside a key that opens it. ADR-0015's refusal to give oracle that property stands; its argument turned on a powered, networked host in the same room. Losing the medium already means rotating every credential; the sets add the firewall's rule bodies and WAN address, the export's password hashes, and grafana.db to that disclosure.

    What is built

    • scripts/backup-offsite.sh — refuses a destination on this host's own filesystem by device number, re-verifies what the medium already holds before writing (the proof that last visit's bytes survived ninety days), copies MANIFEST-last, hashes the copy, keeps OFFSITE_KEEP (default 1) per kind and never the newest. 22 fixtures under make validate; benched against the real sets to tmpfs.
    • The offsite-copy row beside the two key proofs — no unit, the same ninety days — OffsiteCopyStale with "mount the medium" advice, excluded from ScheduledJobStale, unit-tested. --list, --verify-only and --prune run unwrapped so inspecting the medium never resets the clock.
    • copy-the-backups-offsite.md, and the sentence in roadmap.md that stops saying unowned, plus build-the-nas.md §8, both restore runbooks, successor-handover.md (where the sets are, without where the medium is), ADR-0015's forward note.

    What closes this — one visit, off this host's disk

    Read off the host today: the second recipient's proof series is 0, so that medium has not yet made its first visit. When it does, with it mounted on the monitoring host:

    1. make backup-offsite DEST=/path/to/the/medium ARGS=--list — what is here, what is there (nothing, the first time).
    2. make backup-offsite DEST=/path/to/the/medium — the copy of record. Expect three copied … lines and the green closing line; a volume set is about 1.7 GB, so the medium needs a few gigabytes free, and the script refuses rather than fills.
    3. make secrets-verify-backup KEY=/path/to/the/medium/keys.txt — the visit's other job. Neither proof clears the other.

    Until then ScheduledJobNeverRan{homelab_job="offsite-copy"} names the job after the next make install-timers, which is the honest state of a copy nobody has made. This issue closes on step 2's first green line, not on the merge.

    🤖 Generated with Claude Code

  4. added 4 commits that reference this issue on Sep 21, 2026
  5. Gerrrt commented on Sep 22, 2026

    @Gerrrt
    OwnerAuthor

    Still the visit, and four documents that had stopped waiting for it

    Read off the host today, 2026-09-22 — the position is unchanged from the comment above, and
    now says so out loud:

    Series Value
    homelab_job_max_age_seconds{homelab_job="offsite-copy"} 7776000 — declared, install-timers has run
    homelab_job_last_success_timestamp_seconds{homelab_job="offsite-copy"} absent
    ScheduledJobNeverRan{homelab_job="offsite-copy"} firing
    homelab_key_recipient_last_proof_timestamp_seconds{recipient="age1cutv5q…"} 0

    The three sets on the shelf are current — firewall 11 h, NAS 1.4 d, volumes 2.5 d,
    verify-backups green 10 h ago — so a visit today would carry current bytes.

    #610 corrects four documents that described
    the copy as already made. The one that mattered was successor-handover.md: its asset table
    and its closing paragraph both placed the newest of each set on the medium in flat present
    tense, while the alert row thirty lines above called it "the honest state of a copy nobody
    has made"
    . A successor reads the table, not the alert row. Both now carry the qualifier
    security.md already used, so they are true now and stay true afterwards. Also in it:
    roadmap.md said #294's recipient had not landed when only its proof has not; README.md
    counted 48 ADRs against 49; and validate.sh's comment still called /dev/shm a destination
    the self-test copies to, which #596 made a refusal fixture. No behaviour changed.

    What the audit turned up beside it, now owned rather than noted —
    #611 (a copy missing one kind still prints the
    green line and records ninety days; compounded by backup-firewall.sh's cwd-relative
    OUT_DIR), #612 (copy_export syncs before
    writing the sidecar), #613 (a stale .part
    accumulates, and one shape of it wedges the medium, with no runbook row),
    #614 (the self-test reports 24 fixtures when
    two silently skip, and no self-test runs in CI). #611 is the one that would actually cost
    you: it is the failure mode this whole mechanism exists to prevent, one level up.

    This still closes on step 2's first green line. Nothing above makes the copy, and no
    removable medium is mounted on the monitoring host. When it is:

    make backup-offsite DEST=/path/to/the/medium ARGS=--list
    make backup-offsite DEST=/path/to/the/medium
    make secrets-verify-backup KEY=/path/to/the/medium/keys.txt

    from the main checkout. Step 2 closes this; step 3 closes #294's last box; neither clears the
    other.

    🤖 Generated with Claude Code

  6. added a commit that references this issue on Sep 22, 2026
  7. Gerrrt commented on Sep 23, 2026

    @Gerrrt
    OwnerAuthor

    Done 2026-09-23: the first offsite copy, made and proved

    make backup-offsite DEST=/mnt/usb on prometheus, on the medium that holds age19mkg76v0… (#294, proved the same sitting):

    -- first visit — nothing on the medium to re-verify
    copied 20260920T033007Z to /mnt/usb/backups/volumes/observability — every archive hashes to its MANIFEST entry
    copied 20260922T223845Z to /mnt/usb/backups/nas — every archive hashes to its MANIFEST entry
    copied config-20260923T043134Z.sops.yaml to /mnt/usb/backups/firewall — byte-identical, sha256 recorded beside it
    homelab-job job=offsite-copy exit=0 duration=33s success=1790199212
    

    One catch, fixed in the same sitting. The archives are encrypted to the recipients the secrets file listed when each backup ran. The volume set 20260920T033007Z predates age19mkg76v0…, which was added 2026-09-22 20:56. So the key on the medium could not have opened the volume set beside it: every hash matched, but the ADR-0048 property did not hold. The NAS set (09-22 22:38) and the firewall export (09-23 04:31) came after the key and were fine.

    The fix was make backup for a fresh volume set, then make backup-offsite again. That replaced the 09-20 set with 20260923T213613Z, and a decryption test with the medium's own key passed:

    age -d -i /mnt/usb/keys.txt …/20260923T213613Z/grafana-data.tar.gz.age > /dev/null && echo opens
    opens
    

    So the medium now holds the newest of each kind beside a key that opens all three. offsite-copy is recorded, and OffsiteCopyStale counts ninety days from here.

    Worth knowing at the next visit: the hash check proves the bytes are intact, not that the key beside them opens them. After any recipient change, the next visit should carry sets made after it. The decryption test above is the proof.

    🤖 Generated with Claude Code

  8. Gerrrt commented on Sep 23, 2026

    @Gerrrt
    OwnerAuthor

    Correction: the firewall export on the medium does not open with the medium's key

    The closing comment above says the NAS set and the firewall export "came after the key and were fine". The export was not fine. backup-firewall.sh's recipient() took the first key of its .sops.yaml rule only, and passed it to sops --encrypt --age, which replaces the rule's recipient list rather than adding to it. So config-20260923T043134Z.sops.yaml, like every export since 2026-09-09, is encrypted to age1yrdu996… alone. age19mkg76v0…, the key on the medium, cannot open it.

    Current state of the medium:

    Kind Opens with the medium's key
    Volumes 20260923T213613Z Yes, tested with age -d in the sitting
    NAS 20260922T223845Z Yes, made after the key was added
    Firewall config-20260923T043134Z No

    The fix is #660: backup-firewall.sh encrypts to every key of the rule, and backup-offsite.sh now refuses to copy, or to count as proved, any set or export that a current recipient cannot open. Once it's merged: make backup-firewall, then one more make backup-offsite DEST=… with the medium, and this check to confirm:

    SOPS_AGE_KEY_FILE=/mnt/usb/keys.txt sops -d /mnt/usb/backups/firewall/config-<new>.sops.yaml > /dev/null && echo opens

    🤖 Generated with Claude Code

  9. Gerrrt commented on Sep 23, 2026

    @Gerrrt
    OwnerAuthor

    Reopened 2026-09-23: the firewall export on the medium (config-20260923T043134Z) is encrypted to age1yrdu996… only, so the medium's key cannot open it (correction above). This closes when #660 is merged, a fresh export is made with make backup-firewall, make backup-offsite carries it, and SOPS_AGE_KEY_FILE=<medium>/keys.txt sops -d … > /dev/null opens it.

  10. reopened this on Sep 23, 2026
  11. Gerrrt commented on Sep 23, 2026

    @Gerrrt
    OwnerAuthor

    Closed again 2026-09-23: all three kinds on the medium open with its key

    After #660 (exports encrypted to every recipient; backup-offsite.sh refuses a set a current key cannot open), a fresh export was made and carried:

    SOPS_AGE_KEY_FILE=/mnt/usb/keys.txt sops -d /mnt/usb/backups/firewall/config-20260923T221711Z.sops.yaml > /dev/null && echo opens
    opens
    
    Kind On the medium Opens with age19mkg76v0…
    Volumes 20260923T213613Z Yes, age -d in the sitting
    NAS 20260922T223845Z Yes, made after the key was added; the offsite copy now checks this before copying
    Firewall config-20260923T221711Z Yes, sops -d above

    Since #660, a set that a current recipient cannot open is named NOT held and records no proof. So the green line on this copy now carries the property this issue exists for, not just matching hashes.

    🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    securityA credential, an exposure, or a control that is weaker than recorded

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions