Skip to content

Sensitive tier: step-ca, so issuing a certificate is an API call rather than a runbook #130

Closed

Description

@Gerrrt

Sub-issue of #102. Second in the build order, after #129.

What and why

step-ca Apache-2.0 Go/Docker.

Note it is not in awesome-selfhosted — that list delegates Identity Management and Federated Identity to awesome-sysadmin, so its absence there is a scope decision upstream, not a verdict on the software.

The estate already runs a private CA. scripts/gen-certs.sh and docs/runbooks/generate-certificates.md exist precisely because issuing a leaf certificate was, in the repo's own words, "a research project" before that tooling landed. That was the right fix at the time and it works — Grafana is served over TLS with it, and Prometheus scrapes it with ca_file and server_name rather than insecure_skip_verify.

What it does not solve is renewal. Every certificate that script issues expires on a date nobody is tracking, and #91's TLS-expiry checks exist to find out about it. step-ca inverts that: certificates are short-lived by design and renewed automatically, so expiry stops being an event.

The second reason is #97 — DNS for the MokerLink management UI, "so it is not reached by IP, and can hold a certificate that verifies". Every service that gets a name wants a certificate, and hand-issuing one per service is how the current approach stops scaling.

What it needs

  • Decide whether step-ca replaces the current CA or is issued beneath it. The CA was already replaced once (after the key leak documented in docs/security.md), and anything trusting the old one had to be re-pointed. Doing that again has a cost worth counting before starting.
  • The root key handling. This is the same class of problem as The age key has one holder and one copy #106: the age key has one holder and one copy. A CA root with the same property is the same mistake in a different format.
  • ACME provisioner for Caddy (Sensitive tier: Caddy as the reverse proxy and the only published port #129), so renewal is automatic rather than a runbook step.
  • scripts/gen-certs.sh and the certificates runbook get rewritten or retired — deliberately, not left to rot alongside the new path.

Which VLAN this touches

🔴 99 — Winterfell (management)

Security review

  • Does not weaken the default-deny posture between VLANs — adds no rule
  • Any new credential goes in secrets/*.sops.yaml — the CA root key is the sensitive artefact; it must not land in a tracked path, and .gitleaks.toml already carries a PEM private-key rule that would catch it
  • Any new device is added to docs/network.md with an OUI-truncated MAC — n/a

Depends on

The Winterfell mini PC in #102. Related: #91 (TLS-expiry checks), #97 (a name for the switch UI to hold a certificate for), #106 (single-holder key problem, same shape).


Corrected 2026-09-19

The comment of 2026-09-09 says "beneath the existing CA, as an intermediate — not a replacement". That is the sentence ADR-0037 was written to retract. The estate root carries pathlen:0, so a leaf beneath any intermediate fails path length constraint exceeded — measured, not inferred (ADR-0037's table; restated in ADR-0043). The tier has its own root, minted cold on prometheus; the root key never travels; trinity receives a bundle without it. build-the-tier-ca.md is the procedure.

Authored in #432. "Depends on #102" → #404.

Acceptance on trinity: the tree minted and installed per the runbook, and Caddy logs certificate obtained successfully for a seven-day leaf.

Activity

  1. Gerrrt commented on Sep 9, 2026

    @Gerrrt
    OwnerAuthor

    The first bullet is decided, and the service is authored in #415.

    Beneath the existing CA, as an intermediate — not a replacement. Decided
    2026-09-09. The root key already lives on the monitoring host and never
    leaves it; step ca init --root certificates/ca.pem --key certificates/ca-key.pem
    runs there and produces an intermediate whose tree populates step-ca's volume
    on trinity. Everything that trusts ca.pem today — Grafana's leaf,
    Prometheus's scrape, every browser — keeps trusting what step-ca issues, and
    nothing is re-pointed. The cost this issue asked to count is therefore zero on
    the trust side; what it buys is short-lived, auto-renewed leaves for every
    service behind Caddy.

    The root key handling. The root stays exactly where it is, with exactly
    the copies it has. step-ca holds only the intermediate, and the password that
    unlocks it is STEPCA_PASSWORD in secrets/sensitive.sops.yaml — the tier's
    own SOPS rule, its own age key on trinity, and the technical second's
    recipient (#294) joining that rule when the tier holds real data. So the
    "same mistake in a different format" this issue warned about is answered by
    the same mechanism that answered #106.

    Never auto-initialises. The image's entrypoint would mint a fresh root on
    first boot if config/ca.json were absent; a populated volume is what stops
    it, DOCKER_STEPCA_INIT_* is never set, and a bare make up on an empty
    trinity fails loudly on that file rather than quietly creating a second CA.

    ACME provisioner for Caddy — next, once the tree exists: the Caddyfile
    carries the tls { ca https://step-ca:9000/acme/acme/directory } block as a
    comment, and 9000 is exposed on the stack network and published nowhere.

    gen-certs.sh and the certificates runbook stay for now: they still
    issue the estate's leaves and the very intermediate this depends on. What
    retires is hand-issuance for the tier, and the runbook gains the
    intermediate procedure under #404.

    🤖 Generated with Claude Code

  2. added a commit that references this issue on Sep 10, 2026
  3. added this to the Wave 1 — trinity milestone on Sep 11, 2026
  4. Gerrrt commented on Sep 28, 2026

    @Gerrrt
    OwnerAuthor

    Acceptance on trinity, 2026-09-28: passes. Recorded in the changelog by #694.

    Caddy logged certificate obtained successfully for all six names. Caddy's container log was lost when it restarted at 16:25, so these lines come from Loki ({service_name="sensitive-caddy", host="trinity"}):

    04:32:18Z tls.obtain certificate obtained successfully  immich.matrix.elysium         issuer=step-ca:9000-acme-acme-directory
    04:32:19Z tls.obtain certificate obtained successfully  homeassistant.matrix.elysium  issuer=step-ca:9000-acme-acme-directory
    04:32:19Z tls.obtain certificate obtained successfully  vaultwarden.matrix.elysium    issuer=step-ca:9000-acme-acme-directory
    04:32:19Z tls.obtain certificate obtained successfully  adguard.matrix.elysium        issuer=step-ca:9000-acme-acme-directory
    04:32:20Z tls.obtain certificate obtained successfully  paperless.matrix.elysium      issuer=step-ca:9000-acme-acme-directory
    04:32:20Z tls.obtain certificate obtained successfully  trinity.matrix.elysium        issuer=step-ca:9000-acme-acme-directory
    

    The leaves last seven days and verify against the tier's root. I checked with openssl s_client -CAfile certificates/tier-ca.pem against 127.0.0.1:443. Every leaf's issuer is Matrix Elysium Sensitive Tier Intermediate CA.

    Name Verify notAfter
    trinity.matrix.elysium 0 (ok) Oct 5 04:32:20 2026 GMT
    homeassistant.matrix.elysium 0 (ok) Oct 5 04:32:19 2026 GMT
    adguard.matrix.elysium 0 (ok) Oct 5 04:32:19 2026 GMT
    immich.matrix.elysium 0 (ok) Oct 5 04:32:18 2026 GMT
    paperless.matrix.elysium 0 (ok) Oct 5 04:32:20 2026 GMT
    vaultwarden.matrix.elysium 0 (ok) Oct 5 04:32:19 2026 GMT

    The tree was installed per the runbook, and the root key never travelled:

    • The sensitive_step-ca-data volume holds certs/root_ca.crt, certs/intermediate_ca.crt and secrets/intermediate_ca_key. The key is 0600 and secrets/ is 0700.
    • find / -name root_ca_key on trinity returns nothing.
    • certificates/ on trinity holds only tier-ca.pem, so the bundle was shredded.
    • The root is CN=Matrix Elysium Sensitive Tier Root CA, SHA256 4C:C3:06:81:53:22:74:24:D7:2F:D9:B1:D7:29:DE:52:AC:FB:44:54:CC:8A:DE:37:00:9D:1F:22:D6:B9:56:5E.

    The provisioner matches ADR-0037 §3. The ACME provisioner in ca.json allows tls-alpn-01 only, and defaultTLSCertDuration and maxTLSCertDuration are both 168h0m0s.

    Still open elsewhere: #426, the expiry rule with a threshold in hours.

  5. Gerrrt commented on Sep 28, 2026

    @Gerrrt
    OwnerAuthor

    Acceptance verified on trinity (evidence above); recorded by #694.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureA new capability or a changesensitivestacks/sensitive on trinity (VLAN 99)

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions