Skip to content

Back up certificates/ca-key.pem, and prove the copy #496

Closed

Description

@Gerrrt

certificates/ca-key.pem — the estate's root, the one that signs Grafana's
two leaves — is the only root secret in the estate with no custody story. The
age key has ADR-0024 and back-up-the-age-key.md: a second recipient off-host
and off-estate, proved on demand against the real ciphertext, nagged by
SecretsKeyBackupUnproven at ninety days. The tier's root goes to the same
offline medium by build-the-tier-ca.md. The estate's root has nothing — not
a copy, not a sentence.

Found while deciding #436, and recorded in
ADR-0043
decision 4 with the constraints below, so that this issue starts from them
rather than re-litigating them.

What the loss actually costs

Bounded, not fatal — and worth stating accurately, because the age key's
"unbounded downside" does not apply here. The root is pathlen:0, so there is
no intermediate to keep issuing; but there is a re-mint path, and it is
already written: make certs ARGS='--ca --force', two leaves reissued, one
scp -3 through the Mac for alexander's, and a re-trust everywhere the old
ca.pem was imported (successor-handover.md, ADR-0037's cost table).

Two things about that path are the real gap:

  • The trust-distribution list is unknown. ADR-0037: "the Mac's system
    store and Firefox's separate one, at least, and whatever else nobody wrote
    down." A re-trust of unknown size is the defect.
  • The deadline is not 825 days. It is the next time a leaf is needed —
    the next VLAN 30 service that wants TLS.

What the answer is not

  • Not SOPS-in-git. ADR-0037 rejected that for the tier's root: "a
    passphrase-encrypted CA private key in git is the exact artefact that was
    purged from this repository's history." This key is one of the ones purged.
  • Not a passphrase. gen-certs.sh closes that trade deliberately, in its
    header, and says how to reopen it if you disagree. This issue does not.

So the control is medium, not cryptography: one copy, on the offline medium
that already holds the age key's second copy and the tier's root, and nowhere
else. back-up-the-age-key.md's disqualifiers apply as written — not on the
monitoring host, not in a git tree, not in a synced folder.

What the proof would be

The age key's proof decrypts real ciphertext. A CA key's proof needs no
decryption at all: the public half of the backup against the public half of
ca.pem.

openssl rsa -in <backup> -pubout 2>/dev/null | sha256sum
openssl x509 -in certificates/ca.pem -pubkey -noout | sha256sum

Equal, or the copy is not this key. A sibling of scripts/verify-key-backup.sh
with the same refusal to run against the live key by device and inode, the
same "prints no secret" property, and a homelab_*_last_proof_timestamp_seconds
series so that a CaKeyBackupUnproven rule can nag the way ADR-0024's does.
One sentence from the ADR justifies the split: an age key has no re-mint path
and a CA does, which is why this key gets the age key's backup first and its
alerting second.

Order of work

  1. The copy, and the sentence in generate-certificates.md's file table that
    says it exists — plus successor-handover.md, where "both live only on
    prometheus" stops being true for one of the two.
  2. Write down where ca.pem is trusted, so the re-mint path has a list.
  3. The proof script and its series.
  4. The rule.

Refs #436, ADR-0043, ADR-0024, ADR-0037.


Corrected 2026-09-20

Steps 2–4 of the order of work are delivered by #549, merged this morning: the trust table in generate-certificates.md §4, scripts/verify-ca-key-backup.sh (public-key hash comparison, refuses the live key by device and inode, nine fixtures under make validate), scripts/ca-key-state.sh writing homelab_ca_key_backup_last_proof_timestamp_seconds, and CaKeyBackupUnproven in backup.rules.yaml with unit tests. homelab-ca-key-state.timer is installed on the monitoring host (first run 09:43 UTC today), so the rule fires within the hour on a 0 row — which is the honest state.

What is left is step 1, and it is yours with the medium in hand: the copy onto the offline medium that already holds the age key's second copy and the tier's root, then make certs-verify-backup KEY=<the copy>, then the two Mac rows in generate-certificates.md §4 confirmed from the Mac rather than from ADR-0037. The runbook is back-up-the-ca-key.md. This closes when the proof has run once and the alert clears; nothing merged shuts it.

Activity

  1. added this to the Nothing blocks these milestone on Sep 19, 2026
  2. added
    securityA credential, an exposure, or a control that is weaker than recorded
    on Sep 19, 2026
  3. Gerrrt commented on Sep 20, 2026

    @Gerrrt
    OwnerAuthor

    Step 1 delivered, 2026-09-20. The copy is on the offline medium and the proof has run against it:

    • make certs-verify-backup succeeded at 21:56:15 UTC; homelab_ca_key_backup_last_proof_timestamp_seconds{ca="estate",spki_sha256="1a8d9c61…"} went from 0 to 1789941375.
    • CaKeyBackupUnproven and ScheduledJobNeverRan{homelab_job="verify-ca-key-backup"} both left Prometheus's firing list by 21:56:37 UTC; Alertmanager shows neither.
    • The daily ca-key-state timer runs next at 09:38 UTC tomorrow and should carry the proof forward rather than reset it.

    What is still open before this closes: the two Mac rows in generate-certificates.md §4 confirmed on the device rather than from ADR-0037. That is a docs PR with Refs #496 once the Keychain Access and Firefox facts are in hand.

  4. Gerrrt commented on Sep 20, 2026

    @Gerrrt
    OwnerAuthor

    Closing. All four steps are delivered and the closing condition is met:

    • The copy is on the offline medium and was proved 2026-09-20 21:56 UTC against the key's public half (fingerprint 1a8d9c61…); the proof row went from 0 to a timestamp, and CaKeyBackupUnproven and ScheduledJobNeverRan{homelab_job="verify-ca-key-backup"} cleared by 21:56:37.
    • The two Mac rows in generate-certificates.md §4 are confirmed on the device, and both are Not imported: docs(certs): the Mac trusts the estate CA nowhere, confirmed on the device (#496) #580. The re-mint list is two rows shorter than ADR-0037 feared, and no browser on the Mac trusts the estate's CA today.
    • Next due: the proof again by 2026-12-19, or the day after any --ca --force, whichever comes first. The alert says which.
  5. added a commit that references this issue on Sep 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureA new capability or a changesecurityA credential, an exposure, or a control that is weaker than recorded

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions