Repository navigation
Description
certificates/ca-key.pem — the estate's root, the one that signs Grafana's
two leaves — is the only root secret in the estate with no custody story. The
age key has ADR-0024 and back-up-the-age-key.md: a second recipient off-host
and off-estate, proved on demand against the real ciphertext, nagged by
SecretsKeyBackupUnproven at ninety days. The tier's root goes to the same
offline medium by build-the-tier-ca.md. The estate's root has nothing — not
a copy, not a sentence.
Found while deciding #436, and recorded in
ADR-0043
decision 4 with the constraints below, so that this issue starts from them
rather than re-litigating them.
What the loss actually costs
Bounded, not fatal — and worth stating accurately, because the age key's
"unbounded downside" does not apply here. The root is pathlen:0, so there is
no intermediate to keep issuing; but there is a re-mint path, and it is
already written: make certs ARGS='--ca --force', two leaves reissued, one
scp -3 through the Mac for alexander's, and a re-trust everywhere the old
ca.pem was imported (successor-handover.md, ADR-0037's cost table).
Two things about that path are the real gap:
- The trust-distribution list is unknown. ADR-0037: "the Mac's system
store and Firefox's separate one, at least, and whatever else nobody wrote
down." A re-trust of unknown size is the defect. - The deadline is not 825 days. It is the next time a leaf is needed —
the next VLAN 30 service that wants TLS.
What the answer is not
- Not SOPS-in-git. ADR-0037 rejected that for the tier's root: "a
passphrase-encrypted CA private key in git is the exact artefact that was
purged from this repository's history." This key is one of the ones purged. - Not a passphrase.
gen-certs.shcloses that trade deliberately, in its
header, and says how to reopen it if you disagree. This issue does not.
So the control is medium, not cryptography: one copy, on the offline medium
that already holds the age key's second copy and the tier's root, and nowhere
else. back-up-the-age-key.md's disqualifiers apply as written — not on the
monitoring host, not in a git tree, not in a synced folder.
What the proof would be
The age key's proof decrypts real ciphertext. A CA key's proof needs no
decryption at all: the public half of the backup against the public half of
ca.pem.
openssl rsa -in <backup> -pubout 2>/dev/null | sha256sum
openssl x509 -in certificates/ca.pem -pubkey -noout | sha256sumEqual, or the copy is not this key. A sibling of scripts/verify-key-backup.sh
with the same refusal to run against the live key by device and inode, the
same "prints no secret" property, and a homelab_*_last_proof_timestamp_seconds
series so that a CaKeyBackupUnproven rule can nag the way ADR-0024's does.
One sentence from the ADR justifies the split: an age key has no re-mint path
and a CA does, which is why this key gets the age key's backup first and its
alerting second.
Order of work
- The copy, and the sentence in
generate-certificates.md's file table that
says it exists — plussuccessor-handover.md, where "both live only on
prometheus" stops being true for one of the two. - Write down where
ca.pemis trusted, so the re-mint path has a list. - The proof script and its series.
- The rule.
Refs #436, ADR-0043, ADR-0024, ADR-0037.
Corrected 2026-09-20
Steps 2–4 of the order of work are delivered by #549, merged this morning: the trust table in generate-certificates.md §4, scripts/verify-ca-key-backup.sh (public-key hash comparison, refuses the live key by device and inode, nine fixtures under make validate), scripts/ca-key-state.sh writing homelab_ca_key_backup_last_proof_timestamp_seconds, and CaKeyBackupUnproven in backup.rules.yaml with unit tests. homelab-ca-key-state.timer is installed on the monitoring host (first run 09:43 UTC today), so the rule fires within the hour on a 0 row — which is the honest state.
What is left is step 1, and it is yours with the medium in hand: the copy onto the offline medium that already holds the age key's second copy and the tier's root, then make certs-verify-backup KEY=<the copy>, then the two Mac rows in generate-certificates.md §4 confirmed from the Mac rather than from ADR-0037. The runbook is back-up-the-ca-key.md. This closes when the proof has run once and the alert clears; nothing merged shuts it.
Activity
Merge remote-tracking branch 'origin/main' into gerrrt/ca-key-backup-…
Merge remote-tracking branch 'origin/main' into gerrrt/ca-key-backup-…
Merge pull request #549 from Gerrrt/gerrrt/ca-key-backup-proof-86ed47
Gerrrt commentedon Sep 20, 2026
Step 1 delivered, 2026-09-20. The copy is on the offline medium and the proof has run against it:
make certs-verify-backupsucceeded at 21:56:15 UTC;homelab_ca_key_backup_last_proof_timestamp_seconds{ca="estate",spki_sha256="1a8d9c61…"}went from0to1789941375.CaKeyBackupUnprovenandScheduledJobNeverRan{homelab_job="verify-ca-key-backup"}both left Prometheus's firing list by 21:56:37 UTC; Alertmanager shows neither.ca-key-statetimer runs next at 09:38 UTC tomorrow and should carry the proof forward rather than reset it.What is still open before this closes: the two Mac rows in
generate-certificates.md§4 confirmed on the device rather than from ADR-0037. That is a docs PR withRefs #496once the Keychain Access and Firefox facts are in hand.Gerrrt commentedon Sep 20, 2026
Closing. All four steps are delivered and the closing condition is met:
1a8d9c61…); the proof row went from0to a timestamp, andCaKeyBackupUnprovenandScheduledJobNeverRan{homelab_job="verify-ca-key-backup"}cleared by 21:56:37.generate-certificates.md§4 are confirmed on the device, and both are Not imported: docs(certs): the Mac trusts the estate CA nowhere, confirmed on the device (#496) #580. The re-mint list is two rows shorter than ADR-0037 feared, and no browser on the Mac trusts the estate's CA today.--ca --force, whichever comes first. The alert says which.Merge pull request #580 from Gerrrt/gerrrt/ca-key-backup-proof-ca9590