Skip to content

Add a second age recipient so decryption does not depend on one person #294

Closed

Description

@Gerrrt

What and why

Add a second age recipient to .sops.yaml and re-key secrets/observability.sops.yaml, so that decryption does not depend on one key held by one person.

This is the implementation half of #106, which states the risk and stops where the decision starts: ".sops.yaml supports multiple recipients, so the mechanism costs nothing; what it needs is somewhere to put the second key." That sentence is still the whole of the work. #106 stays open as the risk; this issue is where the second key's home gets decided and the re-key gets done.

Three things have changed since #106 was filed on 2026-08-21, and each of them pushes the same way.

ADR-0023 makes this a dependency rather than a nice-to-have. It requires an off-estate encrypted copy of the household's photographs and documents, and then rules on the key: "the off-estate copy is encrypted, and the key that opens it cannot be the one only the operator holds. #106 is this same problem one layer down." A single-holder key is now blocking a decision that is already accepted.

ADR-0023 also rules out the home #106 proposed for it. #106 suggests oracle — "an idle machine looking for a purpose, and holding a second recipient is a job that suits a machine with a slow disk and nothing else to do." ADR-0023 checked exactly that pattern and rejected it: oracle is 10.0.99.30, "the same VLAN, the same rack, the same power feed and the same room", so "every failure that motivates this document — fire, theft, flood, a mistake made with root, or simply the operator being unreachable — reaches both boxes in one event. The existing pattern therefore does not satisfy this constraint, and it is close enough to look as though it does." That reasoning is about the household archive, but it transfers to a decryption key without modification. oracle should be struck from #106's suggestion rather than inherited from it.

The handover page now depends on the answer. docs/runbooks/successor-handover.md (#124) tells an arriving operator that a successor who cannot decrypt has nothing, and has to point at #106 as an open issue rather than at a procedure. Right now the honest instruction to a technical trustee is "obtain the key from the person you are replacing", which fails in precisely the case a trustee exists for.

The decision this issue exists to make

Where the second key lives. The mechanism is settled and cheap; this is the part that is not.

Option For Against
An offline key held by the technical second named on the break-glass card Satisfies ADR-0023's off-estate constraint and the handover case in one move; the person already exists and is already named Depends on that person storing it correctly, and make secrets-verify-backup cannot be run on their behalf
A second offline key stored in a different physical location, same holder Survives fire, theft and flood at the rack Does nothing for the operator being unreachable, which is half of what ADR-0023 is about
oracle #106's own suggestion; zero human dependency Ruled out by ADR-0023's reasoning above — same rack, same power, same room

The counter-argument is in back-up-the-age-key.md and should be answered rather than dropped: "It is worth doing on the day the lab stops being a one-person project, and not before — every extra recipient is another key that can leak." The claim to weigh is whether ADR-0023 and a named technical second are that day.

Mechanism, once the home is chosen

Already documented in secrets/README.md; repeated here so the issue is self-contained.

  1. age-keygen the second keypair offline, never on the monitoring host.
  2. Add its public half to .sops.yaml as an additional recipient.
  3. sops updatekeys secrets/observability.sops.yaml from a host that can already decrypt.
  4. Prove the new key actually opens it: make secrets-verify-backup KEY=<the new key>. That target refuses the live key by device and inode and blanks the environment, so a pass means the second key did the work.
  5. Confirm the first key still decrypts too — a re-key that drops a recipient is the failure mode here.

Done when

  • The second key's home is decided and written down (the location itself stays out of the repository, as with the first — back-up-the-age-key.md explains why).
  • .sops.yaml carries two recipients and secrets/observability.sops.yaml is re-keyed.
  • Both keys independently verified with make secrets-verify-backup.
  • back-up-the-age-key.md §"What this still does not solve" updated — it currently defers this to "the day the lab stops being a one-person project", which will be the wrong text once this lands.
  • successor-handover.md §"If you do not have the age key" updated to describe a second recipient rather than a one-key recovery path.
  • secrets/README.md's second-host paragraph reviewed — it describes this same procedure for a different reason and should not end up contradicting the new text.
  • The age key has one holder and one copy #106 closed against this, or explicitly left open with what remains.

Whether ADR-0023's off-estate household archive key is this key or a separate one is in scope for the discussion here, since ADR-0023 ties them together.


Corrected 2026-09-19

The decision table above is decided: the second recipient is held by the technical second, on a removable medium, decided 2026-09-08 (#397). .sops.yaml's catch-all rule carries two recipients and secrets/observability.sops.yaml is re-keyed to both. #488 repaired the two recovery instructions the re-key broke; it did not run the proof.

The one box left is the done-when item "both keys independently verified with make secrets-verify-backup". The second copy cannot be proved from prometheus on the holder's behalf; back-up-the-age-key.md says clearing it for theirs "is a visit with the removable medium, not a timer". SecretsKeyBackupUnproven naming that recipient is ADR-0024's honest state until then. This closes on that visit.

Activity

  1. added
    securityA credential, an exposure, or a control that is weaker than recorded
    on Sep 4, 2026
  2. Gerrrt commented on Sep 8, 2026

    @Gerrrt
    OwnerAuthor

    Decided 2026-09-08, and the decision half is written down in
    #397. What follows is the half
    only a person can do.

    The decision

    The second recipient is held by the technical second named on the
    break-glass card
    , generated where they keep it and never on the monitoring
    host. It satisfies ADR-0023's off-estate constraint and the handover case in
    one move, and it is the option this issue's table already favoured. oracle
    is struck, for the reason ADR-0023 gave and ADR-0024 kept.

    Two corrections to the body above, both already true when this was filed:

    ADR-0023's household archive key is not this key. Different artefact,
    different reader, different failure: the archive is opened by the household on
    a bad day, the repository's secrets by a technical trustee on a different bad
    day, and one key doing both jobs means one leak takes both. Keep them separate;
    if that is wrong, say so here and the runbook gets one sentence.

    What closes this — three steps, all off this host

    1. With the technical second, on the medium they will keep:
      age-keygen -o <their medium>/keys.txt. Bring back the public half
      only; scripts/add-recipient.sh refuses a key whose private half it can
      find here.
    2. From the main checkout:
      make secrets-add-recipient PUBKEY=age1... — resolves the creation_rule
      from the recipients the encrypted file already uses and re-keys in the same
      run. Commit .sops.yaml and secrets/observability.sops.yaml.
    3. Prove both keys, separately — a re-key that drops a recipient is the
      failure mode here:
      make secrets-verify-backup KEY=<their copy> and again with the existing
      backup key. The target refuses the live key by device and inode, so a pass
      means the copy did the work.

    Then tick the boxes above; the back-up-the-age-key.md and
    successor-handover.md boxes are #397, and secrets/README.md's second-host
    paragraph was reviewed and needs nothing — it already describes this procedure
    and points at ADR-0024.

    🤖 Generated with Claude Code

  3. added 2 commits that reference this issue on Sep 8, 2026
  4. Gerrrt commented on Sep 9, 2026

    @Gerrrt
    OwnerAuthor

    Re-keyed on 2026-09-09. Against the done-when list:

    What stays open here is the one box that needs a physical visit. Everything
    else about this issue is done on #412's merge.

    🤖 Generated with Claude Code

  5. added a commit that references this issue on Sep 9, 2026
  6. Gerrrt commented on Sep 20, 2026

    @Gerrrt
    OwnerAuthor

    Measured 2026-09-20, read-only, so the visit that closes this does not start by re-deriving the state.

    Fact Value
    Recipients in the ciphertext (./scripts/key-recipients.sh --list) age1yrdu996… and age1cutv5…, identical to .sops.yaml's catch-all rule — no drift
    age1yrdu996… last proved 2026-09-03; its ninety days run out 2026-12-02
    age1cutv5… last proved never (0 in key-recipients.prom, as written on the 2026-09-09 re-key)
    SecretsKeyBackupUnproven firing for age1cutv5… — the activeAt reads 2026-09-18 because a Prometheus restart reset it, not because anything changed
    Delivery warning → default receiver, repeats every 12h; no silence, deliberately (the runbook and ADR-0024 call this the honest state)
    Visit nothing on the calendar

    Nothing in the repository needs to change for this to close: the two runbooks, secrets/README.md, and the forward-pointers in .sops.yaml and secrets/sensitive.example.yaml all read correctly after it does.

    The visit, when it happens

    On the monitoring host, from the main checkout (~/code/Gerrrt/HomeLab, not a worktree — #412 recorded why), with the technical second present:

    1. Mount their medium. If the key arrives as text instead, keep it off the disk:
      umask 077; cat > /dev/shm/restore-test.txt — paste, Ctrl-D. Not an editor; back-up-the-age-key.md § "Do not open the copy in an editor" says why.
    2. make secrets-verify-backup KEY=<that path> and expect
      -- recipient age1cutv5… followed by ok — … decrypts secrets/observability.sops.yaml (9/9 keys).
    3. grep age1cutv5 /var/lib/node_exporter/textfile_collector/key-recipients.prom now shows a non-zero timestamp; the alert resolves within about an hour (for: 1h).
    4. rm -f /dev/shm/restore-test.txt if step 1 used tmpfs. Unmount their medium.
    5. Paste the ok line here and close. Nothing else to edit.

    The private half never leaves their medium or tmpfs, and never goes into a session transcript. If the run says not one the secrets are encrypted to, the medium holds a different keypair from the one whose public half travelled on 2026-09-09; the fix is make secrets-add-recipient with the one it actually holds, then the proof again — not rotation.

    🤖 Generated with Claude Code

  7. added 4 commits that reference this issue on Sep 22, 2026
  8. Gerrrt commented on Sep 23, 2026

    @Gerrrt
    OwnerAuthor

    Done 2026-09-23: two recipients, both proved

    The second holder's original key, age1cutv5…, was never kept. It was replaced rather than proved:

    Step When Record
    age19mkg76v0… added and the file re-keyed 2026-09-22 20:56 UTC 6de8509
    Tool to remove a recipient: make secrets-remove-recipient 2026-09-23 20:07 UTC #658
    age1cutv5… removed and the file re-keyed 2026-09-23 21:23 UTC a2f369f
    age19mkg76v0… proved from its medium 2026-09-23 21:27:14 UTC make secrets-verify-backup: ok — keys.txt decrypts secrets/observability.sops.yaml (11/11 keys); key-recipients.prom reads 1790198834 for it

    The ciphertext and .sops.yaml now list exactly age1yrdu996… and age19mkg76v0…. age1yrdu996… was last proved on 2026-09-03, and its ninety days run out on 2026-12-02, on a clock separate from the new key's.

    The removed key still opens every older ciphertext in git history. It was lost, not exposed, so no credentials were rotated. If that ever changes, back-up-the-age-key.md § "Revocation is still rotation" applies.

    🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    securityA credential, an exposure, or a control that is weaker than recorded

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions