Repository navigation
Description
What and why
Add a second age recipient to .sops.yaml and re-key secrets/observability.sops.yaml, so that decryption does not depend on one key held by one person.
This is the implementation half of #106, which states the risk and stops where the decision starts: ".sops.yaml supports multiple recipients, so the mechanism costs nothing; what it needs is somewhere to put the second key." That sentence is still the whole of the work. #106 stays open as the risk; this issue is where the second key's home gets decided and the re-key gets done.
Three things have changed since #106 was filed on 2026-08-21, and each of them pushes the same way.
ADR-0023 makes this a dependency rather than a nice-to-have. It requires an off-estate encrypted copy of the household's photographs and documents, and then rules on the key: "the off-estate copy is encrypted, and the key that opens it cannot be the one only the operator holds. #106 is this same problem one layer down." A single-holder key is now blocking a decision that is already accepted.
ADR-0023 also rules out the home #106 proposed for it. #106 suggests oracle — "an idle machine looking for a purpose, and holding a second recipient is a job that suits a machine with a slow disk and nothing else to do." ADR-0023 checked exactly that pattern and rejected it: oracle is 10.0.99.30, "the same VLAN, the same rack, the same power feed and the same room", so "every failure that motivates this document — fire, theft, flood, a mistake made with root, or simply the operator being unreachable — reaches both boxes in one event. The existing pattern therefore does not satisfy this constraint, and it is close enough to look as though it does." That reasoning is about the household archive, but it transfers to a decryption key without modification. oracle should be struck from #106's suggestion rather than inherited from it.
The handover page now depends on the answer. docs/runbooks/successor-handover.md (#124) tells an arriving operator that a successor who cannot decrypt has nothing, and has to point at #106 as an open issue rather than at a procedure. Right now the honest instruction to a technical trustee is "obtain the key from the person you are replacing", which fails in precisely the case a trustee exists for.
The decision this issue exists to make
Where the second key lives. The mechanism is settled and cheap; this is the part that is not.
| Option | For | Against |
|---|---|---|
| An offline key held by the technical second named on the break-glass card | Satisfies ADR-0023's off-estate constraint and the handover case in one move; the person already exists and is already named | Depends on that person storing it correctly, and make secrets-verify-backup cannot be run on their behalf |
| A second offline key stored in a different physical location, same holder | Survives fire, theft and flood at the rack | Does nothing for the operator being unreachable, which is half of what ADR-0023 is about |
oracle |
#106's own suggestion; zero human dependency | Ruled out by ADR-0023's reasoning above — same rack, same power, same room |
The counter-argument is in back-up-the-age-key.md and should be answered rather than dropped: "It is worth doing on the day the lab stops being a one-person project, and not before — every extra recipient is another key that can leak." The claim to weigh is whether ADR-0023 and a named technical second are that day.
Mechanism, once the home is chosen
Already documented in secrets/README.md; repeated here so the issue is self-contained.
age-keygenthe second keypair offline, never on the monitoring host.- Add its public half to
.sops.yamlas an additional recipient. sops updatekeys secrets/observability.sops.yamlfrom a host that can already decrypt.- Prove the new key actually opens it:
make secrets-verify-backup KEY=<the new key>. That target refuses the live key by device and inode and blanks the environment, so a pass means the second key did the work. - Confirm the first key still decrypts too — a re-key that drops a recipient is the failure mode here.
Done when
- The second key's home is decided and written down (the location itself stays out of the repository, as with the first —
back-up-the-age-key.mdexplains why). -
.sops.yamlcarries two recipients andsecrets/observability.sops.yamlis re-keyed. - Both keys independently verified with
make secrets-verify-backup. -
back-up-the-age-key.md§"What this still does not solve" updated — it currently defers this to "the day the lab stops being a one-person project", which will be the wrong text once this lands. -
successor-handover.md§"If you do not have the age key" updated to describe a second recipient rather than a one-key recovery path. -
secrets/README.md's second-host paragraph reviewed — it describes this same procedure for a different reason and should not end up contradicting the new text. - The age key has one holder and one copy #106 closed against this, or explicitly left open with what remains.
Whether ADR-0023's off-estate household archive key is this key or a separate one is in scope for the discussion here, since ADR-0023 ties them together.
Corrected 2026-09-19
The decision table above is decided: the second recipient is held by the technical second, on a removable medium, decided 2026-09-08 (#397). .sops.yaml's catch-all rule carries two recipients and secrets/observability.sops.yaml is re-keyed to both. #488 repaired the two recovery instructions the re-key broke; it did not run the proof.
The one box left is the done-when item "both keys independently verified with make secrets-verify-backup". The second copy cannot be proved from prometheus on the holder's behalf; back-up-the-age-key.md says clearing it for theirs "is a visit with the removable medium, not a timer". SecretsKeyBackupUnproven naming that recipient is ADR-0024's honest state until then. This closes on that visit.
Activity
Gerrrt commentedon Sep 8, 2026
Decided 2026-09-08, and the decision half is written down in
#397. What follows is the half
only a person can do.
The decision
The second recipient is held by the technical second named on the
break-glass card, generated where they keep it and never on the monitoring
host. It satisfies ADR-0023's off-estate constraint and the handover case in
one move, and it is the option this issue's table already favoured.
oracleis struck, for the reason ADR-0023 gave and ADR-0024 kept.
Two corrections to the body above, both already true when this was filed:
built the whole mechanism this issue describes:
make secrets-add-recipient,per-recipient proof (
SecretsKeyBackupUnprovennames the overduerecipient), and the runbook text. This issue was never referenced from any of
it; docs(secrets): name the second recipient's holder, and fix the roadmap's rejection (#294) #397 links the two.
back-up-the-age-key.mdreplaced it when ADR-0024 landed. What remained wasthe holder, which is what this decides.
ADR-0023's household archive key is not this key. Different artefact,
different reader, different failure: the archive is opened by the household on
a bad day, the repository's secrets by a technical trustee on a different bad
day, and one key doing both jobs means one leak takes both. Keep them separate;
if that is wrong, say so here and the runbook gets one sentence.
What closes this — three steps, all off this host
age-keygen -o <their medium>/keys.txt. Bring back the public halfonly;
scripts/add-recipient.shrefuses a key whose private half it canfind here.
make secrets-add-recipient PUBKEY=age1...— resolves thecreation_rulefrom the recipients the encrypted file already uses and re-keys in the same
run. Commit
.sops.yamlandsecrets/observability.sops.yaml.failure mode here:
make secrets-verify-backup KEY=<their copy>and again with the existingbackup key. The target refuses the live key by device and inode, so a pass
means the copy did the work.
Then tick the boxes above; the
back-up-the-age-key.mdandsuccessor-handover.mdboxes are #397, andsecrets/README.md's second-hostparagraph was reviewed and needs nothing — it already describes this procedure
and points at ADR-0024.
🤖 Generated with Claude Code
docs(secrets): name the second recipient's holder, and fix the roadma…
Merge pull request #397 from Gerrrt/gerrrt/second-recipient-holder-294
Gerrrt commentedon Sep 9, 2026
Re-keyed on 2026-09-09. Against the done-when list:
second named on the break-glass card; docs(secrets): name the second recipient's holder, and fix the roadmap's rejection (#294) #397 put it in
back-up-the-age-key.mdandsuccessor-handover.md, location withheld..sops.yamlcarries two recipients andsecrets/observability.sops.yamlis re-keyed. #412. The
keypair was generated where it lives; only
age1cutv5…travelled. Onething worth recording: the first attempt ran on a checkout that does not
hold the live key and
add-recipient.shrolled.sops.yamlback exactlyas written — "run this where the existing key is". The re-key has to
happen on the monitoring host, and now the runbook's reader knows that
from a real failure rather than a sentence.
make secrets-verify-backup.The live key still decrypts (checked on both the host and the branch).
The proof of each backup copy needs that copy on removable media at
this host — the existing backup key on its next visit, the new one when
the technical second's medium is here. Until then
SecretsKeyBackupUnprovennames the new recipient, which is ADR-0024'shonest state, not a fault.
back-up-the-age-key.md§"What this still does not solve" — docs(secrets): name the second recipient's holder, and fix the roadmap's rejection (#294) #397.successor-handover.md§"If you do not have the age key" — docs(secrets): name the second recipient's holder, and fix the roadmap's rejection (#294) #397.secrets/README.md's second-host paragraph — reviewed, needs nothing.What stays open here is the one box that needs a physical visit. Everything
else about this issue is done on #412's merge.
🤖 Generated with Claude Code
Merge pull request #412 from Gerrrt/gerrrt/second-recipient-added-294
Gerrrt commentedon Sep 20, 2026
Measured 2026-09-20, read-only, so the visit that closes this does not start by re-deriving the state.
./scripts/key-recipients.sh --list)age1yrdu996…andage1cutv5…, identical to.sops.yaml's catch-all rule — no driftage1yrdu996…last provedage1cutv5…last proved0inkey-recipients.prom, as written on the 2026-09-09 re-key)SecretsKeyBackupUnprovenage1cutv5…— theactiveAtreads 2026-09-18 because a Prometheus restart reset it, not because anything changeddefaultreceiver, repeats every 12h; no silence, deliberately (the runbook and ADR-0024 call this the honest state)Nothing in the repository needs to change for this to close: the two runbooks,
secrets/README.md, and the forward-pointers in.sops.yamlandsecrets/sensitive.example.yamlall read correctly after it does.The visit, when it happens
On the monitoring host, from the main checkout (
~/code/Gerrrt/HomeLab, not a worktree — #412 recorded why), with the technical second present:umask 077; cat > /dev/shm/restore-test.txt— paste, Ctrl-D. Not an editor;back-up-the-age-key.md§ "Do not open the copy in an editor" says why.make secrets-verify-backup KEY=<that path>and expect-- recipient age1cutv5…followed byok — … decrypts secrets/observability.sops.yaml (9/9 keys).grep age1cutv5 /var/lib/node_exporter/textfile_collector/key-recipients.promnow shows a non-zero timestamp; the alert resolves within about an hour (for: 1h).rm -f /dev/shm/restore-test.txtif step 1 used tmpfs. Unmount their medium.okline here and close. Nothing else to edit.The private half never leaves their medium or tmpfs, and never goes into a session transcript. If the run says
not one the secrets are encrypted to, the medium holds a different keypair from the one whose public half travelled on 2026-09-09; the fix ismake secrets-add-recipientwith the one it actually holds, then the proof again — not rotation.🤖 Generated with Claude Code
docs(roadmap): #294's entry names the proof visit it is actually gate…
Merge pull request #609 from Gerrrt/gerrrt/second-age-recipient-sops-…
Merge origin/main: #609 fixed the roadmap's #294 entry first, so take…
Merge pull request #658 from Gerrrt/gerrrt/replace-second-recipient-294
Gerrrt commentedon Sep 23, 2026
Done 2026-09-23: two recipients, both proved
The second holder's original key,
age1cutv5…, was never kept. It was replaced rather than proved:age19mkg76v0…added and the file re-keyedmake secrets-remove-recipientage1cutv5…removed and the file re-keyedage19mkg76v0…proved from its mediummake secrets-verify-backup:ok — keys.txt decrypts secrets/observability.sops.yaml (11/11 keys);key-recipients.promreads1790198834for itThe ciphertext and
.sops.yamlnow list exactlyage1yrdu996…andage19mkg76v0….age1yrdu996…was last proved on 2026-09-03, and its ninety days run out on 2026-12-02, on a clock separate from the new key's.The removed key still opens every older ciphertext in git history. It was lost, not exposed, so no credentials were rotated. If that ever changes,
back-up-the-age-key.md§ "Revocation is still rotation" applies.🤖 Generated with Claude Code
fix(secrets): two keys on the sensitive tier, and CI refuses one (#835)…