diff --git a/Makefile b/Makefile index e3f06b88..d987847b 100644 --- a/Makefile +++ b/Makefile @@ -572,6 +572,14 @@ certs: ## Create the internal CA / issue a leaf (ARGS="--host x.matrix.elysium - @# refuses to render until they exist. ./scripts/gen-certs.sh $(ARGS) +.PHONY: tier-ca +tier-ca: ## The sensitive tier's own CA — mint it here, install it on trinity (ARGS="--mint" | "--install FILE" | "--list") + @# Not `certs`: that is the estate's CA and this is the tier's, and they are + @# deliberately two — the estate's root carries pathlen:0, so nothing beneath + @# it may be a CA (ADR-0037). The step binary runs from the pinned image, the + @# way promtool and caddy do; nothing is installed on the host. + ./scripts/tier-ca.sh $(ARGS) + .PHONY: gen-secret gen-secret: ## Generate a random secret (ARGS=--snmp for one per SNMP device) ./scripts/gen-secret.sh $(ARGS) diff --git a/README.md b/README.md index f3639ccb..d42d1266 100644 --- a/README.md +++ b/README.md @@ -81,9 +81,8 @@ documents for different readers. the Makefile, the scripts, the workflow and the runbooks resolves its image from `compose.yaml` too, so an image that is not pinned there cannot be run at all. -- **Documented decisions and runbooks.** 36 ADRs covering what was chosen -- **Documented decisions and runbooks.** 36 ADRs covering what was chosen - and what was rejected — including the costs accepted knowingly; 23 +- **Documented decisions and runbooks.** 37 ADRs covering what was chosen + and what was rejected — including the costs accepted knowingly; 24 runbooks for the operations that are easy to get wrong at 1am, one of which is the handover page a successor reads first. @@ -189,8 +188,8 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md). │ # authored ahead of it (ADR-0030, #266, #267) ├── stacks/sensitive/ # the household's tier — Caddy, step-ca, Home │ # Assistant, AdGuard Home, Immich, Paperless-ngx -│ # and Vaultwarden so far, for a host not yet -│ # built (ADR-0034, #404) +│ # and Vaultwarden so far; its own CA, leaves over +│ # ACME (ADR-0037); for a host not yet built (ADR-0034, #404) ├── secrets/ # SOPS-encrypted; see secrets/README.md ├── scripts/ # bootstrap, render, validate, pin-digests, purge ├── SECURITY.md # disclosure policy and known exposure diff --git a/docs/adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md b/docs/adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md new file mode 100644 index 00000000..2de8c2bb --- /dev/null +++ b/docs/adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md @@ -0,0 +1,217 @@ +# ADR-0037: Give the sensitive tier its own root, and issue beneath it over ACME + +**Status:** Accepted · 2026-09 + +## Context + +[#130](https://github.com/Gerrrt/HomeLab/issues/130) puts step-ca on the +sensitive tier and asks for two decisions before it is built: whether it +**replaces** the estate's CA or is **issued beneath** it, and how its root key +is handled — "the same class of problem as #106: the age key has one holder +and one copy. A CA root with the same property is the same mistake in a +different format." It also asks for an ACME provisioner so Caddy renews +without a runbook step, and for `scripts/gen-certs.sh` and +[`generate-certificates.md`](../runbooks/generate-certificates.md) to be +"rewritten or retired — deliberately, not left to rot alongside the new path". + +The stack's foundation landed on 2026-09-09 with the first decision already +written into it. `stacks/sensitive/compose.yaml`, its README and the roadmap +all said the same thing: *an intermediate beneath the lab CA, so nothing that +trusts `certificates/ca.pem` is re-pointed, and the root key never leaves the +monitoring host.* That is the natural answer, it costs nothing to re-trust, +and it was written before anyone read the root. + +### The lab root forbids it, and that was measured rather than inferred + +`gen-certs.sh` mints the estate's root with +`basicConstraints=critical,CA:TRUE,pathlen:0`. A path length of zero means no +CA certificate may sit between this root and a leaf — the root may sign +end-entity certificates and nothing else. Trial on 2026-09-09, against a +throwaway root minted by the same `openssl` line and the pinned step-ca image: + +| Step | Result | +| --- | --- | +| `step ca init --root ca.pem --key ca-key.pem …` | Succeeds. Copies the root, mints an intermediate, writes `config/ca.json` | +| `openssl verify -CAfile ca.pem intermediate_ca.crt` | `OK` — the intermediate is a valid certificate *of* the root | +| A leaf signed by that intermediate, verified through the chain | `error 25 at 2 depth lookup: path length constraint exceeded` | + +Every client in this estate is built on Go's `crypto/x509` — Caddy, Prometheus, +blackbox-exporter, step-ca's own health check — and it enforces the constraint +the same way. So the decision as written produces a CA that mints certificates +nothing accepts. It would have failed loudly, on the first `make up` on +`trinity`: step-ca's health check verifies its own serving certificate through +that chain against the root, so the container would never have gone healthy +and Caddy, which waits on it, would never have started. Loud, but it would +have read as a trust problem inside a container rather than as a property of +the root, and the fix would have been decided at 1am. + +`pathlen:0` is not a mistake to fix. It is the property that nothing beneath +the estate's root can ever become a CA, it has been on that root since the CA +was replaced after the key leak ([`security.md`](../security.md)), and it is +worth keeping. Lifting it means minting a new root, which is the replacement +the issue asked to have costed before it is started. + +### The cost of replacing, counted + +A new estate root with `pathlen:1` and the tier beneath it would need: + +- Grafana's leaf reissued on the monitoring host and the stack restarted; + Prometheus and blackbox read `ca.pem` by path, so they follow on restart. +- The lab guest's leaf reissued and carried with the new `ca.pem` to + `alexander` through the Mac, because `99 → 30` is closed + ([`build-the-lab-guest.md`](../runbooks/build-the-lab-guest.md) §5). +- Every device that imported `ca.pem` re-trusting the new one — the Mac's + system store and Firefox's separate one, at least, and whatever else nobody + wrote down. Until each is done, Grafana warns in that browser. +- The whole of it sequenced *before* the tier's tree is minted, on the + monitoring host, on a day the lab guest is also reachable. + +None of that is large. The CA was replaced once already and the procedure +exists. What decides it is not the size of the cost but who pays it and for +what: the estate's root would be reissued so that the household's devices can +trust it — and they have no reason to. + +### Two audiences, and the precedent for separating them + +The estate's CA has one client that is a person: the operator, opening Grafana +from Hicks. Everything else that verifies against it is a scrape or a probe on +the management VLAN. The tier's CA will be trusted by the household's phones +and laptops, for Vaultwarden, Immich and Home Assistant, and those devices +must import *some* root to reach the tier at all. Importing the estate's would +also have them vouch for a Grafana they cannot reach, on a segment they are +not allowed to see. + +This repository has already decided, twice, that things with separate trust +get separate keys. [`.sops.yaml`](../../.sops.yaml) gives the tier its own +encryption rule "not because the host is less trusted than the estate, but +because it has no use for the estate's secrets". +[ADR-0020](0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md) gives +the lab its own Prometheus and its own age key for the mirror-image reason. A +certificate authority is the same shape: the tier has no use for the estate's +root, and the estate has no use for a root that signs for the tier. + +### What `trinity` could forge + +[ADR-0008](0008-place-services-by-data-trust.md) puts the tier on the segment +where "compromise here is total", and gives that box the largest attack +surface in the estate — Immich's upload endpoint and Paperless's parsers on +one host. The intermediate key on that host signs for any name it is asked +for. Beneath the estate's root, a compromised `trinity` mints a certificate +for `grafana.matrix.elysium` that Prometheus, blackbox and the operator's +browser all accept. Beneath its own root, what it can forge is bounded to +what trusts the tier's root: the tier's own clients. That is not the reason +for the decision, but it is a consequence that points the same way. + +## Decision + +**The sensitive tier runs a certificate authority of its own — a root and an +intermediate minted on the monitoring host — and Caddy obtains every +certificate it serves from that intermediate over ACME.** The estate's CA is +untouched. Four parts. + +**1. A root of its own, minted where the estate's root lives.** +`make tier-ca ARGS=--mint` ([`scripts/tier-ca.sh`](../../scripts/tier-ca.sh)) +runs `step ca init` in the pinned step-ca image against a directory under +`certificates/` on the monitoring host. It produces the tree step-ca runs +from — `config/`, `certs/`, `secrets/`, `db/` — with the root at `pathlen:1` +and the intermediate at `pathlen:0`, which is the same shape as the estate's +root and for the same reason. It then adds the ACME provisioner offline, +against `config/ca.json`, rather than leaving it to the `--acme` default, so +that the challenge and lifetime restrictions below are in the file before the +CA ever starts. It refuses to run if a tree already exists, for the reason +`gen-certs.sh --ca` does: minting a new root invalidates every device that +trusts the old one, and that has to be deliberate. + +**2. The root key is cold, stays on the monitoring host, and never reaches +`trinity`.** What ships to `trinity` is a bundle the script builds from the +tree with `secrets/root_ca_key` left out, and the script refuses to report +success if the bundle contains it. This was checked rather than assumed: +`config/ca.json` references the intermediate's certificate and key and the +root's *certificate*, and nowhere the root's key — step-ca has no use for it +at runtime. step encrypts both keys at rest with the password it is given, +and that password is `STEPCA_PASSWORD` in `secrets/sensitive.sops.yaml`, +which decrypts on `trinity` and, by ADR-0024, for the technical second — and +**not** on the monitoring host, whose age key opens the estate's secrets and +not the tier's. So the root key file on the monitoring host is inert without +a secret that host cannot read, and using it — to mint a fresh intermediate +after `trinity` is rebuilt — is a two-host operation on purpose. + +That leaves the one-copy problem the issue names. It has the same shape as +the age key's and gets the same answer: +[`back-up-the-age-key.md`](../runbooks/back-up-the-age-key.md)'s offline copy +is where the tier's root key goes too, and +[`build-the-tier-ca.md`](../runbooks/build-the-tier-ca.md) says so. What +makes it a smaller problem than the age key's is how loss degrades: the +intermediate keeps issuing for its whole lifetime without the root, so a lost +root key costs nothing until the intermediate must be re-minted, and then it +costs a new root and a re-trust on the household's devices. Nothing goes +down. What is **rejected** is putting the root key in this repository under +SOPS: a passphrase-encrypted CA private key in git is the exact artefact +that was purged from this repository's history, and encrypting it with a +better cipher does not change what a reader of `security.md`'s table would +conclude on seeing it there again. + +**3. ACME, `tls-alpn-01` only, seven-day leaves.** The provisioner accepts one +challenge type. Caddy solves it on 443, which is already the tier's one +published port, so 80 stays unpublished ([ADR-0012](0012-publish-only-ports-with-an-off-host-consumer.md)) +and the CA never attempts `http-01` against a port nothing answers on. The +challenge is validated by step-ca connecting to the requested *name* from +inside the compose network, so every name Caddy serves is also a network +alias of the Caddy container — Docker's resolver answers `trinity.matrix.elysium` +with Caddy's address only because `compose.yaml` says so. A name in the +`Caddyfile` with no alias fails its first issuance with a DNS error at the +CA, which is loud and immediate. + +The leaf lifetime is 168 hours, default and maximum, and not step-ca's +default of 24. The lifetime is not a labour cost — Caddy renews at a third of +the way from expiry and nobody types anything — it is the **outage budget**: +the time between step-ca failing and the tier's handshakes failing. This is a +single box that [ADR-0023](0023-keep-the-household-recovery-path-outside-the-estate.md) +declines to make highly available, run by one person who is sometimes away for +a weekend. With 24-hour leaves that budget is as little as eight hours; with +seven-day leaves it is never less than two days and usually closer to five. +Short-lived is the point; short enough to fail over a Saturday is not. + +**4. `gen-certs.sh` and `generate-certificates.md` are rescoped, not +retired.** They are the *estate's* CA: Grafana on the monitoring host and on +the lab guest still serve leaves from it, and Grafana does not speak ACME. The +runbook now says which of the two CAs it is, and the script's header says what +it is no longer for. Its `pathlen:0` line is now commented as deliberate, +because this ADR is the record of what happens when that is forgotten. +Retiring the script waits on Grafana renewing from step-ca with +`step ca renew` on a timer — which needs 9000 published on `trinity` for its +first off-host consumer, and is a decision to take with ADR-0012 in hand, not +here. + +## Consequences + +- **Two private CAs, and the successor has to know there are two.** The + operator's devices import both roots; the household's import one. + `generate-certificates.md` opens with the table that says which is which, + and `successor-handover.md` names both keys among the things a clean clone + does not hold. +- **The comments that said "an intermediate beneath the lab CA" were wrong + and are corrected** in `compose.yaml`, the stack's README and the roadmap, + in this ADR's pull request. A decision recorded only in a comment is what + let it go unmeasured; this is the ADR it should have had. +- **The estate's TLS-expiry alerts do not fit seven-day leaves.** + `TlsCertificateExpiringSoon` fires at thirty days and + `TlsCertificateExpiryImminent` at seven ([#91](https://github.com/Gerrrt/HomeLab/issues/91)), + so a tier endpoint added to the lab-CA blackbox module would page + permanently. The tier's endpoints stay out of those rules until a rule with a + threshold in hours exists — [#426](https://github.com/Gerrrt/HomeLab/issues/426). +- **A dead step-ca is a dead tier within a week, and nothing pages on it + yet.** The estate's container-health alerting watches the monitoring host's + cAdvisor; `trinity` ships node and log telemetry only. Until #404 wires the + tier's containers into an alert, the seven-day budget is the only signal, + and it is a silent one. +- **Rebuilding `trinity` is `--install` again.** The bundle on the monitoring + host is the whole CA state that matters; the ACME accounts and issued + certificates in step-ca's database are disposable, because every leaf is + re-issued within a week anyway. Losing `trinity`'s disk costs nothing the + household's devices can see. +- **Reopened by:** Grafana moving onto step-ca, at which point the estate + trusts the tier's root for a reason and the estate's own CA can be argued + out of existence — a different ADR; or a client that must trust both roots + appearing, which is the first sign the two audiences are not as separate as + this ADR claims. diff --git a/docs/roadmap.md b/docs/roadmap.md index 19b66812..266c6d4d 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -439,9 +439,19 @@ what left this one unfireable for months. decided on top of it. What is outstanding is the build under #404, the rest of a stack, and four firewall rules the ADR counted as two. `stacks/sensitive/` exists since 2026-09-09 with its foundation — Caddy as the only published - port, step-ca as an intermediate beneath the lab CA — authored ahead of the - hardware the way `stacks/lab` was, and checked by everything `make validate` - runs, `caddy validate` included. AdGuard Home joined it the same day + port, step-ca as a certificate authority of the tier's own issuing to Caddy + over ACME — authored ahead of the hardware the way `stacks/lab` was, and + checked by everything `make validate` runs, `caddy validate` included. The + foundation first claimed step-ca would be an intermediate beneath the lab + CA; that root carries `pathlen:0` and cannot have one, which + [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md) + measured before deciding, along with where the root key lives (the + monitoring host, never `trinity`) and why the leaves are seven days. The + ACME path was proved on the monitoring host under a throwaway project; + what remains for [#130](https://github.com/Gerrrt/HomeLab/issues/130) is + the mint and install on the real host, under #404, and a TLS-expiry rule + sized for seven-day leaves ([#426](https://github.com/Gerrrt/HomeLab/issues/426)). + AdGuard Home joined it the same day ([#135](https://github.com/Gerrrt/HomeLab/issues/135)), in the shape ADR-0010 decided and not the one the service assumes: the blocklist set is a tracked file copied in on every start, the admin hash is in SOPS, port 53 diff --git a/docs/runbooks/build-the-tier-ca.md b/docs/runbooks/build-the-tier-ca.md new file mode 100644 index 00000000..bdcb776b --- /dev/null +++ b/docs/runbooks/build-the-tier-ca.md @@ -0,0 +1,239 @@ +# Runbook: Build the sensitive tier's certificate authority + +**Target:** step-ca on `trinity` (`10.0.99.40`), minted on `prometheus` (`10.0.99.20`) +**Time:** twenty minutes, across two hosts +**You will need:** a shell on both hosts, this repository checked out on each, +and `secrets/sensitive.sops.yaml` already created on `trinity` +(`make secrets-init STACK=sensitive`) + +This is the tier's own CA — **not** the estate's, and not beneath it. +[`generate-certificates.md`](generate-certificates.md) opens with the table +that tells the two apart, and +[ADR-0037](../adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md) +is why they are two: the estate's root carries `pathlen:0`, so a leaf beneath +any intermediate of it fails `path length constraint exceeded` in every client +this estate runs. Once this runbook is done, no certificate on the tier is +ever issued by hand again: Caddy asks step-ca over ACME for every name it +serves and renews each seven-day leaf on its own. + +## What moves where, and what never does + +| Artefact | Made on | Lives on | Travels? | +| --- | --- | --- | --- | +| `certificates/tier-ca/` — the whole tree, root key included | `prometheus` | `prometheus` | **No.** This is the CA | +| `certificates/tier-ca/secrets/root_ca_key` | `prometheus` | `prometheus`, and the age key's offline copy | **Never to `trinity`.** The install refuses a bundle that carries it | +| `certificates/tier-ca-bundle.tar` — config, both certificates, the intermediate's encrypted key | `prometheus` | `trinity`'s `step-ca-data` volume | Yes, once, by `scp`; delete the copy on `trinity` afterwards | +| `certificates/tier-ca.pem` — the root certificate | `prometheus` | Both hosts, and every device that reaches the tier | Yes, freely. Public | +| `STEPCA_PASSWORD` | You, with `make gen-secret` | `secrets/sensitive.sops.yaml` on `trinity` | To `prometheus` once, in a file, for the mint; shredded after | + +The intermediate's key is encrypted at rest with `STEPCA_PASSWORD`, and so +is the root's. `trinity`'s age key opens the tier's secrets and `prometheus`'s +does not, so the root key on `prometheus` is inert without a password that +host cannot decrypt — using it is a two-host operation on purpose. + +## 1. The password, on `trinity` first + +```bash +make gen-secret +make secrets-edit STACK=sensitive # STEPCA_PASSWORD: paste it +``` + +It has to be in SOPS *before* the mint, because the mint is where a second +copy is easiest to get wrong: `--mint` takes the password from a file and +step encrypts both keys with it, and if the value in SOPS differs by so much +as a trailing newline, step-ca on `trinity` cannot open its own key and the +container never goes healthy. Write it to a file for the trip to +`prometheus` — the script strips a trailing newline, so `printf` or an +editor both work — and shred that file when the mint is done. + +## 2. Mint, on `prometheus` + +```bash +make tier-ca ARGS="--mint --password-file /path/to/password" +``` + +Omit `--password-file` to be prompted instead. The script runs `step ca init` +in the pinned step-ca image — nothing is installed on the host — against +`certificates/tier-ca/`, mounted where the service will later run from, and +then adds the ACME provisioner against the resulting `config/ca.json` with +the three restrictions ADR-0037 decided: `tls-alpn-01` only, seven-day +leaves, nothing longer. It refuses to run over an existing tree without +`--force`, because a new root invalidates every device that trusts the old +one. + +Before it reports success it proves the result rather than trusting step: +the intermediate verifies against the root, the root is a CA, the +provisioner in `ca.json` carries exactly those claims, every path in +`ca.json` is one the service container will see, and the bundle it builds +does not contain the root key. It prints the root's SHA-256 fingerprint; +keep that line, you compare against it in step 4. + +```text +ok — minted + tree certificates/tier-ca (stays here; holds the root key) + root certificates/tier-ca.pem (public — this is what devices trust) + bundle certificates/tier-ca-bundle.tar (what travels to trinity; no root key) + root SHA256 5C:35:… +``` + +Then, still on `prometheus`: + +```bash +shred -u /path/to/password +``` + +> [!CAUTION] +> **`certificates/tier-ca/secrets/root_ca_key` stays on this host.** The +> bundle is built by listing files by name and never `secrets/` as a +> directory; `--install` refuses a bundle that carries the root key whatever +> built it. Copy the bundle. Do not `scp -r certificates/tier-ca`. + +Add the root key to the offline copy that holds the age key +([`back-up-the-age-key.md`](back-up-the-age-key.md)). It is useless without +`STEPCA_PASSWORD`, which is in SOPS, so the backup of the tier's root is +that copy plus this file — and losing both costs a new root and a re-trust on +every household device, not an outage: the intermediate on `trinity` keeps +issuing for its whole lifetime without it. + +## 3. Carry the bundle to `trinity` + +Both hosts are on Winterfell, so unlike the lab guest's certificate this is +one hop and no Mac. From `prometheus`: + +```bash +ssh @10.0.99.40 'mkdir -p HomeLab/certificates && chmod 700 HomeLab/certificates' +scp -p certificates/tier-ca-bundle.tar @10.0.99.40:HomeLab/certificates/ +``` + +`-p` keeps the mode (`0600`). The bundle holds the intermediate's key, +encrypted; treat it as a secret in transit and delete the copy on `trinity` +once step 4 has read it. + +## 4. Install, on `trinity` + +```bash +make tier-ca ARGS="--install certificates/tier-ca-bundle.tar" +``` + +That creates the `step-ca-data` volume with the labels compose stamps on its +own volumes — so `make up` adopts it silently rather than warning that it +"was not created by Docker Compose" — extracts the bundle into it as root, +hands the tree to step's uid with `secrets/` closed to `0700`, and writes +`certificates/tier-ca.pem` from the bundle's root certificate, which is what +`compose.yaml` mounts into Caddy as the trusted root for the ACME directory. +It refuses a volume that already holds a CA without `--force`, because +installing over one replaces the intermediate every leaf on the tier chains +to. + +It ends by reading the intermediate back out of the volume as the service's +own uid and printing the root's fingerprint: + +```text + root SHA256 5C:35:… (compare with what --mint printed) +``` + +**Compare it.** Same fingerprint, same root; anything else and the bundle is +not the one you minted. + +```bash +shred -u certificates/tier-ca-bundle.tar +make up STACK=sensitive +``` + +`make render` now passes — the tier's root is the only file under +`certificates/` this stack mounts — and `make up` starts step-ca, waits for +it to go healthy, then starts Caddy, which asks it for a certificate before +serving a byte. + +## 5. Verify — the certificate Caddy is serving, not the files on disk + +On `trinity`: + +```bash +docker logs sensitive-caddy 2>&1 | grep -i 'certificate obtained' +``` + +One line per name in the `Caddyfile`, within a few seconds of the start. +Then the handshake itself, verified against the tier's root: + +```bash +openssl s_client -connect 127.0.0.1:443 -servername trinity.matrix.elysium \ + -CAfile certificates/tier-ca.pem /dev/null \ + | grep -E 'Verify return|^issuer=' +``` + +`Verify return code: 0 (ok)` and an issuer of `Matrix Elysium Sensitive Tier +Intermediate CA`. The `notAfter` a week out is the design, not a mistake: + +```bash +openssl s_client -connect 127.0.0.1:443 -servername trinity.matrix.elysium \ + /dev/null | openssl x509 -noout -dates +``` + +From Hicks, in a browser, `https://trinity.matrix.elysium/` should warn until +step 6 and answer `404` after it — nothing is routed yet, and the 404 is the +tier's own text, which proves the whole path: name, firewall, Caddy, CA. + +This was proved once on the monitoring host, against a throwaway root, under +a throwaway compose project name, before the runbook was written. The stack +booted, step-ca went healthy, Caddy obtained a leaf for +`trinity.matrix.elysium` over `tls-alpn-01` from inside the compose network, +and the handshake verified against `tier-ca.pem`. `TIER_CA_PROJECT` in +`scripts/tier-ca.sh` exists for repeating that rehearsal without touching a +volume the real project would adopt. + +## 6. Trust the root on every device that reaches the tier + +Distribute `certificates/tier-ca.pem` — never anything else out of +`certificates/tier-ca/`. The operator's devices end up trusting two roots, +the estate's and this one; the household's trust only this one. + +```bash +# Debian/Ubuntu +sudo cp certificates/tier-ca.pem /usr/local/share/ca-certificates/matrix-elysium-tier.crt +sudo update-ca-certificates +``` + +Firefox keeps its own store: **Settings → Privacy & Security → Certificates → +View Certificates → Authorities → Import**. On a phone, install it as a +certificate profile and — on iOS — enable full trust for it under +**General → About → Certificate Trust Settings**, without which the profile +sits installed and untrusted. + +## Adding a service + +Three lines, in one commit, or the first issuance fails: + +1. A site block in the `Caddyfile` under the service's name. +2. That name under the `caddy` service's `networks.sensitive.aliases` in + `compose.yaml` — step-ca validates the challenge by dialling the name on + 443 from inside the compose network, and Docker's resolver answers with + Caddy's address only for names Caddy carries. Missing this fails with a + DNS error at the CA, in Caddy's log, on the first `make up`. +3. A host override on `morpheus` so browsers resolve it + ([`add-a-host-override.md`](add-a-host-override.md)). + +No certificate step. That is the point of this runbook. + +## When things change + +| Situation | What to do | +| --- | --- | +| `trinity` is rebuilt, or its disk is lost | Step 4 again with the bundle still on `prometheus`; step 5 to confirm. The ACME accounts and issued leaves in step-ca's database are disposable — every leaf is re-issued within a week regardless | +| The intermediate is suspected compromised | On `prometheus`, `make tier-ca ARGS="--mint --force"` mints a new root **and** intermediate — the root changes too, so every device re-trusts. Re-minting only the intermediate beneath the existing root is `step certificate create` against `certificates/tier-ca/` and is not scripted; ADR-0037 leaves it to the day it is needed | +| The root key on `prometheus` is lost | Nothing stops. The intermediate issues until 2036 without it. Before then, a new root: `--mint --force`, step 3 onward, and a re-trust on every household device | +| `STEPCA_PASSWORD` is lost | Both keys are unrecoverable; treat as the row above | +| step-ca is down | Caddy keeps serving each leaf until it expires, at least two days and at most seven after the CA stopped answering. Nothing pages on it yet ([#426](https://github.com/Gerrrt/HomeLab/issues/426)) — `docker ps` on `trinity` is the check until it does | + +## If something goes wrong + +| Symptom | Cause | Fix | +| --- | --- | --- | +| `make render` stops with *do NOT run `make certs ARGS=--ca` here* | `certificates/tier-ca.pem` is missing on `trinity` | Step 4. Do not do what the estate's error message would have said | +| step-ca never goes healthy; log says it cannot decrypt the key | `STEPCA_PASSWORD` in SOPS differs from what `--mint` was given | Fix SOPS if the file was right, or `--mint --force` with the SOPS value and step 3 onward | +| step-ca log: `open /tree/…: no such file` or any path outside `/home/step` | The tree was minted at the wrong mount point (a script older than this runbook) | `--mint --force` with the current script; it refuses to report a tree whose paths are wrong | +| Caddy log: `no such host` for a name, from the CA | The name is in the `Caddyfile` and not in `compose.yaml`'s aliases | Add the alias, `make up STACK=sensitive` | +| Caddy log: `x509: certificate signed by unknown authority` dialling `step-ca` | `/etc/caddy/tls/ca.pem` is not this CA's root — the estate's `ca.pem` was copied in by hand | Step 4 rewrites `tier-ca.pem` from the bundle | +| `--install` refuses: *already holds a CA* | The volume was populated before | `--force` only if replacing the intermediate is what you mean | +| `--install` refuses: *contains secrets/root_ca_key* | Someone tarred the tree by hand | Rebuild with `--mint` on `prometheus`; delete this copy | +| Browser warns after step 6 | The wrong root was imported, or Firefox's own store was skipped | Import `tier-ca.pem`, and into Firefox separately | diff --git a/docs/runbooks/generate-certificates.md b/docs/runbooks/generate-certificates.md index 06a6ee5b..e4a945e3 100644 --- a/docs/runbooks/generate-certificates.md +++ b/docs/runbooks/generate-certificates.md @@ -1,9 +1,31 @@ # Runbook: Generate the internal CA and leaf certificates -**Target:** the lab's own PKI, under `certificates/` +**Target:** the **estate's** PKI, under `certificates/` on the monitoring host **Time:** a couple of minutes **You will need:** nothing but `openssl` and this repository +> **There are two certificate authorities in this estate, and this runbook is +> one of them.** Check the table before issuing anything. +> +> | | The estate's CA (this runbook) | The sensitive tier's CA | +> | --- | --- | --- | +> | Issues for | Grafana on `prometheus` and on the lab guest | Everything Caddy serves on `trinity` | +> | Minted by | `make certs ARGS=--ca` — `openssl` | `make tier-ca ARGS=--mint` — step-ca, in its pinned image | +> | Leaves issued by | You, by hand, with `make certs` | step-ca over ACME; Caddy asks and renews on its own | +> | Leaf lifetime | 825 days | 7 days | +> | Root certificate | `certificates/ca.pem` | `certificates/tier-ca.pem` | +> | Root key lives | `prometheus`, and nowhere else | `prometheus`, and nowhere else — never on `trinity` | +> | Who trusts it | The operator's browsers; Prometheus and blackbox by `ca_file` | The household's devices, and the operator's | +> | Runbook | This one | [`build-the-tier-ca.md`](build-the-tier-ca.md) | +> +> They are two on purpose and cannot be one: this root carries `pathlen:0`, +> so nothing beneath it may be a CA, and a leaf beneath any intermediate of +> it fails `path length constraint exceeded` in every client here. +> [ADR-0037](../adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md) +> measured that and decided the rest. Nothing this runbook issues is for +> `trinity`; a `make certs` run there mints a third CA nothing trusts, and +> `make render STACK=sensitive` says so if the tier's root is missing. + The previous CA and leaf keys were committed to this public repository and had to be removed by rewriting every commit — see [`purge-git-history.md`](purge-git-history.md). Treat anything issued before @@ -119,6 +141,14 @@ Firefox keeps its own store and will not read the system one; import it under Leaves expire in 825 days. There is no automation and deliberately no cron: a lab with one certificate is better served by a reminder than by a renewal daemon nobody maintains. The reminder is an alert, not a calendar entry. + +That sentence is about *this* CA. The tier's certificates renew themselves — +Caddy re-issues each seven-day leaf from step-ca at a third of the way from +expiry, and nothing in this runbook applies to them. The day Grafana renews +from step-ca too, with `step ca renew` on a timer, this section and this +script retire together; that is ADR-0037's reopen condition, and it waits on +publishing step-ca's port to its first off-host consumer +([ADR-0012](../adr/0012-publish-only-ports-with-an-off-host-consumer.md)). blackbox-exporter probes Grafana by name and by address, verifies the chain against `ca.pem`, and reads the expiry off the handshake — so what is watched is the certificate actually being served, not a file on disk. @@ -186,6 +216,15 @@ Then issue them from step 1. ## Also worth knowing +**This root cannot have an intermediate**, and that is a property rather than +an omission: `pathlen:0` is what keeps a leaf key lifted from Grafana from +ever signing anything else. It is also why the sensitive tier's step-ca is a +root of its own instead of sitting beneath this one — a decision made after +the alternative was built and measured to fail +([ADR-0037](../adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)). +Lifting it means `--ca --force`, re-issuing every leaf, and re-trusting the +new `ca.pem` on every device that holds the old one. + The CA key is **not** passphrase-protected, which is a deliberate trade. A passphrase is what made the previous leak survivable, but it also makes every issuance interactive — which is how a lab ends up with one ancient certificate diff --git a/docs/runbooks/successor-handover.md b/docs/runbooks/successor-handover.md index d03ced17..d1428f19 100644 --- a/docs/runbooks/successor-handover.md +++ b/docs/runbooks/successor-handover.md @@ -195,6 +195,7 @@ inherits without knowing. | **The external heartbeat watcher** — a free-tier cron-monitor on somebody else's account | Whenever that account lapses | Nothing here can tell you. A watcher on this host would fail with the thing it watches, which is why it is off-host and therefore outside anything this repository can check | | **The age key backup goes unproven** | 90 days after the last verification, *per recipient* | `SecretsKeyBackupUnproven`, routed to the normal alert channel, naming the recipient — proving one copy does not clear another ([ADR-0024](../adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md)). A recipient never proved is recorded as never, and fires; `SecretsKeyRecipientsUnrecorded` fires instead if the record itself is missing ([#400](https://github.com/Gerrrt/HomeLab/issues/400)) | | **Grafana's leaf certificate** | 825 days from issue; the APC card's own certificate expires on its own clock | `TlsCertificateExpiringSoon` at 30 days, `TlsCertificateExpiryImminent` at 7 — read off the served handshake by `blackbox-exporter`, not off a file. Let it lapse and `up{job="grafana"}` goes to 0 as well | +| **The sensitive tier's certificates**, once `trinity` exists | Seven days after step-ca last answered Caddy — every leaf there is renewed automatically and lives a week, so the tier stays up for as long as its CA does ([ADR-0037](../adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)) | Nothing pages on it yet: the estate's expiry rules are sized for 825-day leaves and would fire permanently on seven-day ones, so the tier is deliberately outside them until [#426](https://github.com/Gerrrt/HomeLab/issues/426) lands. Until then a dead step-ca is found by a browser refusing the handshake, a week late | | **The UPS battery pack** | A pack was fitted 2026-08-28 and passed its self-test; packs are consumables and this one is on a biweekly test schedule | `UpsSelfTestFailed` and `UpsBatteryUnproven` key on the self-test result, which is the single honest signal this card emits — every charge, runtime and alarm value it reports was fabricated while the bay was empty. Two things remain open: the card's test *schedule* is unwatched ([#249](https://github.com/Gerrrt/HomeLab/issues/249)), and `upsBasicBatteryLastReplaceDate` still reads a pre-fit date, so it is not a usable record of the pack's age | | **Mains power to the monitoring path** | Any cut | The rack is on the UPS; the switch carrying `prometheus` and `oracle` is not, so both laptops keep running and go deaf. Stated in [`security.md`](../security.md#threat-model) | | **Container images going stale** | Continuously, once nobody merges | Dependabot proposes bumps and CI validates them, and convergence deploys a merge within the hour. An unattended estate simply stops receiving updates — nothing alerts on it | @@ -234,10 +235,15 @@ outgoing operator directly: - The pfSense, MokerLink, iLO and APC network-card **admin passwords**. None of them are in this repository in any form. -- The internal CA's private key. `certificates/` is gitignored and host-local, so - a clean clone has no CA at all — reissuing is +- The internal CAs' private keys — there are two, and both live only on + `prometheus`. `certificates/` is gitignored and host-local, so a clean clone + has no CA at all. The estate's: reissuing is [`generate-certificates.md`](generate-certificates.md) step 1, and the cost is - re-trusting the new `ca.pem` everywhere it was trusted. + re-trusting the new `ca.pem` everywhere it was trusted. The sensitive + tier's: its root key is cold on `prometheus` and its intermediate runs on + `trinity`, so nothing stops until the intermediate needs re-minting; + [`build-the-tier-ca.md`](build-the-tier-ca.md) covers both the rebuild and + the re-root, and the re-root costs a re-trust on every household device. - Access to whatever holds the alert webhooks and the heartbeat check. ### If you do not have the age key diff --git a/scripts/check_caddyfile.sh b/scripts/check_caddyfile.sh index a27187c1..0e972388 100755 --- a/scripts/check_caddyfile.sh +++ b/scripts/check_caddyfile.sh @@ -14,10 +14,14 @@ # genuinely understands the format is the thing that will run it. # # 1. `caddy validate` — adapts and provisions the config without starting it. -# Provisioning loads the TLS files the Caddyfile names, so a throwaway -# keypair is generated into a temp dir and mounted where compose.yaml -# mounts the real one. The real leaf never exists on a CI runner and must -# not: certificates/ is gitignored and stays that way. +# Provisioning loads the PEM files the Caddyfile names — the trusted root +# for its ACME issuer — so a throwaway self-signed certificate is +# generated into a temp dir and mounted where compose.yaml mounts the +# real one. Nothing is dialled: `--network none`, and provisioning an +# ACME issuer does not contact the directory; obtaining a certificate +# does, and that happens only when the server starts. The real root never +# exists on a CI runner and must not: certificates/ is gitignored and +# stays that way. # 2. `caddy fmt --diff` — the file is formatted the way `caddy fmt` would # write it, so a diff shows a change and not a reindent. # @@ -72,13 +76,13 @@ for stack in "${STACKS[@]}"; do work="$(mktemp -d)" trap 'rm -rf "${work}" 2>/dev/null || true' EXIT - # A throwaway pair, one day, unrelated to any CA this estate has ever run. - # It exists so `tls /etc/caddy/tls/cert.pem /etc/caddy/tls/key.pem` has files - # to load; it proves nothing about the real certificate and is not meant to. + # A throwaway certificate, one day, unrelated to any CA this estate has ever + # run. It exists so the issuer's `trusted_roots /etc/caddy/tls/ca.pem` has a + # PEM to parse; it proves nothing about the real root and is not meant to. openssl req -x509 -newkey rsa:2048 -nodes -days 1 \ -subj '/CN=caddyfile-validation' \ - -keyout "${work}/key.pem" -out "${work}/cert.pem" >/dev/null 2>&1 - cp "${work}/cert.pem" "${work}/ca.pem" + -keyout "${work}/key.pem" -out "${work}/ca.pem" >/dev/null 2>&1 + rm -f "${work}/key.pem" chmod 644 "${work}"/*.pem chmod 755 "${work}" @@ -93,8 +97,6 @@ for stack in "${STACKS[@]}"; do # image-for.sh above. run=(docker run --rm --network none \ -v "${work}/Caddyfile:/etc/caddy/Caddyfile:ro" \ - -v "${work}/cert.pem:/etc/caddy/tls/cert.pem:ro" \ - -v "${work}/key.pem:/etc/caddy/tls/key.pem:ro" \ -v "${work}/ca.pem:/etc/caddy/tls/ca.pem:ro" \ "${image}") diff --git a/scripts/gen-certs.sh b/scripts/gen-certs.sh index 555373f4..ef542dff 100755 --- a/scripts/gen-certs.sh +++ b/scripts/gen-certs.sh @@ -1,6 +1,15 @@ #!/usr/bin/env bash # -# Generate the lab's internal CA and the leaf certificates it signs. +# Generate the ESTATE'S internal CA and the leaf certificates it signs. +# +# The estate's, and only the estate's: Grafana on the monitoring host and on +# the lab guest serve leaves from this CA, and Prometheus and blackbox verify +# them against it. The sensitive tier on trinity has a certificate authority of +# its own — step-ca, with a root minted by scripts/tier-ca.sh — and every +# certificate Caddy serves there is issued over ACME and renewed without a +# runbook step. Nothing this script issues is for that host, and nothing that +# host serves comes from here. ADR-0037 says why they are two: the root below +# carries pathlen:0, so nothing beneath it may be a CA. # # Everything lands in certificates/, which is gitignored — along with *.pem and # *.key — because the previous set of these was committed to a public repository @@ -122,6 +131,13 @@ that holds it. Pass --force only if that is what you mean." fi info "creating the CA (${CA_DAYS} days)" + # pathlen:0 is deliberate and load-bearing: nothing beneath this root may be + # a CA, so a leaf key lifted from Grafana cannot be turned into a signer of + # anything else. It also means this root CANNOT have an intermediate — a + # leaf beneath one fails "path length constraint exceeded" in every client + # here — which is why the sensitive tier's step-ca has a root of its own + # rather than sitting beneath this one (ADR-0037, measured before deciding). + # Lifting it means re-minting this root and re-trusting it everywhere. openssl req -x509 -newkey rsa:4096 -sha256 -nodes \ -keyout "${CA_KEY}" -out "${CA_CRT}" -days "${CA_DAYS}" \ -subj "${CA_SUBJECT}" \ diff --git a/scripts/render-config.sh b/scripts/render-config.sh index 0d7d89a8..ab949813 100755 --- a/scripts/render-config.sh +++ b/scripts/render-config.sh @@ -85,8 +85,28 @@ report \"already exists\" and refuse — and --force would not help either. Then issue them — see docs/runbooks/generate-certificates.md." fi +# The fix depends on which stack, because the estate's CA is issued HERE and +# every other stack's certificates arrive from somewhere else. One message +# naming `make certs ARGS=--ca` for all of them is how a second — or, on +# trinity, a third — certificate authority gets minted by someone following +# the error text (build-the-lab-guest.md §5 carries the warning; this is the +# message it warns about). if ((${#absent[@]})); then - die "compose.yaml mounts these certificates, which are missing or empty: + case "${STACK}" in + sensitive) + die "compose.yaml mounts these certificates, which are missing or empty: +$(printf ' %s\n' "${absent[@]}") + +Caddy verifies step-ca's ACME directory against the tier's own root (ADR-0037). +That file is written on this host from the bundle minted on the monitoring +host — do NOT run \`make certs ARGS=--ca\` here, which would mint a CA nothing +trusts: + + make tier-ca ARGS=\"--install certificates/tier-ca-bundle.tar\" + +Full procedure in docs/runbooks/build-the-tier-ca.md." ;; + observability) + die "compose.yaml mounts these certificates, which are missing or empty: $(printf ' %s\n' "${absent[@]}") Grafana serves https from the leaf and Prometheus verifies it with the CA, so @@ -95,7 +115,15 @@ the stack cannot start without them. Generate them with: make certs ARGS=--ca make certs ARGS=\"--host grafana.matrix.elysium --ip 10.0.99.20 --dns grafana\" -Full procedure in docs/runbooks/generate-certificates.md." +Full procedure in docs/runbooks/generate-certificates.md." ;; + *) + die "compose.yaml mounts these certificates, which are missing or empty: +$(printf ' %s\n' "${absent[@]}") + +This stack's certificates are issued on the monitoring host and copied here — +do NOT run \`make certs ARGS=--ca\` on this host, which would mint a second CA. +See the stack's README and docs/runbooks/generate-certificates.md." ;; + esac fi unset absent clobbered cert diff --git a/scripts/tier-ca.sh b/scripts/tier-ca.sh new file mode 100755 index 00000000..2b33cc51 --- /dev/null +++ b/scripts/tier-ca.sh @@ -0,0 +1,382 @@ +#!/usr/bin/env bash +# +# The sensitive tier's certificate authority: mint it on the monitoring host, +# install it on trinity. ADR-0037 is the decision; this is the mechanism. +# +# The tier has a root of its own rather than an intermediate beneath the +# estate's CA, because the estate's root is minted with pathlen:0 and a leaf +# beneath any intermediate of it fails "path length constraint exceeded" +# (measured — the ADR has the table). Caddy on trinity then obtains every +# certificate it serves from the intermediate over ACME, so issuing one is a +# handshake and renewal is nobody's job. +# +# Everything lands under certificates/, which is gitignored, and the tree +# step-ca runs from is produced by `step ca init` in the pinned step-ca image — +# no step binary on the host, the same rule as promtool and caddy validate. +# +# Usage: +# scripts/tier-ca.sh --mint [--password-file FILE] monitoring host: create root + intermediate, +# write the bundle that ships to trinity +# scripts/tier-ca.sh --install BUNDLE trinity: populate step-ca's volume from +# the bundle, and write the root cert +# Caddy trusts +# scripts/tier-ca.sh --list show what exists here, and when it expires +# +# --password-file FILE the password that encrypts the CA keys at rest. It is +# STEPCA_PASSWORD in secrets/sensitive.sops.yaml — the +# two must match or step-ca cannot open its own key. +# Prompted for if omitted; never taken from argv. +# --force overwrite an existing tree (--mint) or a populated +# volume (--install). Minting a new root invalidates +# every device that trusts the old one. +# +# What the bundle holds, and what it does not: +# config/ca.json, config/defaults.json the CA's configuration, ACME included +# certs/root_ca.crt the root CERTIFICATE — public +# certs/intermediate_ca.crt the intermediate certificate +# secrets/intermediate_ca_key the intermediate's key, encrypted +# db/ empty; step-ca creates its state here +# +# secrets/root_ca_key NEVER. It stays in the tree on the +# monitoring host, and this script +# refuses to report a bundle that +# carries it. ca.json does not +# reference it; step-ca has no use for +# it at runtime. + +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +CERT_DIR="${REPO_ROOT}/certificates" +TREE="${CERT_DIR}/tier-ca" +BUNDLE="${CERT_DIR}/tier-ca-bundle.tar" +# The root certificate on its own, where compose.yaml mounts it into Caddy as +# the trusted root for the ACME directory, and what every device that reaches +# the tier imports. Public: distribute freely, like the estate's ca.pem. +ROOT_PEM="${CERT_DIR}/tier-ca.pem" + +# The compose project is `name: sensitive` and the volume is `step-ca-data`, +# so the volume Docker creates is this. --install creates it FIRST, with the +# labels compose stamps on its own volumes, so that `make up` adopts it silently +# instead of warning that it "was not created by Docker Compose" (measured: the +# labels are what compose checks; without them it warns and proceeds). +# TIER_CA_PROJECT exists for one reason: booting this stack under another +# project name on a host that is not trinity, to prove the ACME path against +# a throwaway root without touching a volume the real project would adopt. +COMPOSE_PROJECT="${TIER_CA_PROJECT:-sensitive}" +VOLUME_NAME="step-ca-data" +VOLUME="${COMPOSE_PROJECT}_${VOLUME_NAME}" + +CA_NAME="Matrix Elysium Sensitive Tier" +# Every name the CA's own serving certificate carries. `step-ca` is what Caddy +# dials on the compose network; `localhost` is what the container's healthcheck +# dials; the FQDN and the address are there so a future off-host client (the +# estate's Grafana renewing with `step ca renew`, ADR-0037's reopen) verifies +# without a re-mint. SANs are set at init and cost nothing to include. +CA_DNS=(step-ca localhost trinity.matrix.elysium 10.0.99.40) +CA_ADDRESS=":9000" +# One challenge type, and it is the one that runs over 443, the tier's only +# published port. 168h is the outage budget, not a labour cost — ADR-0037 §3. +ACME_CHALLENGE="tls-alpn-01" +LEAF_DEFAULT="168h" +LEAF_MAX="168h" +LEAF_MIN="5m" + +die() { printf '\033[0;31merror:\033[0m %s\n' "$*" >&2; exit 1; } +info() { printf '\033[0;34m--\033[0m %s\n' "$*"; } +ok() { printf '\033[0;32mok\033[0m — %s\n' "$*"; } +head_() { printf '\n\033[1m%s\033[0m\n' "$*"; } + +MODE="" +PASSWORD_FILE="" +BUNDLE_IN="" +FORCE=0 + +while (($#)); do + case "$1" in + --mint) MODE="mint"; shift ;; + --list) MODE="list"; shift ;; + --install) MODE="install"; BUNDLE_IN="${2:?--install needs the bundle path}"; shift 2 + [[ "${BUNDLE_IN}" != --* ]] || die "--install needs the bundle path, got a flag: ${BUNDLE_IN}" ;; + --password-file) PASSWORD_FILE="${2:?--password-file needs a path}"; shift 2 ;; + --force) FORCE=1; shift ;; + # Printed from the header rather than kept as a second copy — gen-certs.sh + # does the same, and for the same reason. + -h|--help) awk 'NR>1 && /^#/ {sub(/^# ?/, ""); print; next} NR>1 {exit}' "${BASH_SOURCE[0]}"; exit 0 ;; + *) die "unknown argument: $1" ;; + esac +done + +[[ -n "${MODE}" ]] || die "nothing to do — try --mint, --install BUNDLE, or --list (see --help)" + +# The daemon is checked before anything else so a missing one fails here, by +# name. The image itself is resolved at each call site on its own statement — +# scripts/check_image_pins.py traces `docker run` operands back to an +# assignment that mentions image-for.sh, and a function body is invisible to it. +require_docker() { + command -v docker >/dev/null 2>&1 || die "docker not found — the step binary runs from the pinned image" + docker info >/dev/null 2>&1 || die "docker daemon not reachable" +} +SENSITIVE_COMPOSE="${REPO_ROOT}/stacks/sensitive/compose.yaml" + +# --------------------------------------------------------------------------- +# --list +# --------------------------------------------------------------------------- +if [[ "${MODE}" == "list" ]]; then + head_ "certificates/tier-ca/" + if [[ ! -s "${TREE}/config/ca.json" ]]; then + printf ' (no tree yet — start with %s --mint)\n\n' "$(basename "$0")" + exit 0 + fi + for crt in root_ca intermediate_ca; do + path="${TREE}/certs/${crt}.crt" + [[ -s "${path}" ]] || { printf ' %-24s \033[0;31mMISSING\033[0m\n' "${crt}.crt"; continue; } + subject="$(openssl x509 -in "${path}" -noout -subject 2>/dev/null | sed 's/^subject=//')" + enddate="$(openssl x509 -in "${path}" -noout -enddate 2>/dev/null | cut -d= -f2)" + if openssl x509 -in "${path}" -noout -checkend 0 >/dev/null 2>&1; then + state="valid" + else + state="\033[0;31mEXPIRED\033[0m" + fi + printf ' %-24s %b until %s\n %s\n' "${crt}.crt" "${state}" "${enddate}" "${subject}" + done + printf ' %-24s %s\n' "root key" "$([[ -s "${TREE}/secrets/root_ca_key" ]] && echo 'present (stays here)' || echo 'ABSENT')" + printf ' %-24s %s\n' "bundle" "$([[ -s "${BUNDLE}" ]] && echo "${BUNDLE#"${REPO_ROOT}"/}" || echo '(not built)')" + printf ' %-24s %s\n\n' "root cert for Caddy" "$([[ -s "${ROOT_PEM}" ]] && echo "${ROOT_PEM#"${REPO_ROOT}"/}" || echo '(not written)')" + exit 0 +fi + +# --------------------------------------------------------------------------- +# --mint (monitoring host) +# --------------------------------------------------------------------------- +if [[ "${MODE}" == "mint" ]]; then + if [[ -s "${TREE}/config/ca.json" ]] && ((! FORCE)); then + die "${TREE#"${REPO_ROOT}"/}/config/ca.json already exists. +Minting a new root invalidates every device that trusts the current one — +every phone and laptop that imported ${ROOT_PEM#"${REPO_ROOT}"/}. Pass --force only if +that is what you mean." + fi + command -v openssl >/dev/null 2>&1 || die "openssl not found (used to verify the result)" + command -v python3 >/dev/null 2>&1 || die "python3 not found (used to read ca.json)" + require_docker + # The image this stack pins, never a floating tag — see scripts/image-for.sh. + IMAGE="$(COMPOSE_FILE="${SENSITIVE_COMPOSE}" "${REPO_ROOT}/scripts/image-for.sh" step-ca)" + + # 0700 on the directory, not just the keys — gen-certs.sh's reasoning. + mkdir -p "${CERT_DIR}" + chmod 700 "${CERT_DIR}" + umask 077 + + # The password: a file the operator names, or a prompt. Never argv, which + # every other user on the host can read out of /proc. The prompted copy + # lives on a private temp file for the length of the run and is shredded. + TMP="$(mktemp -d)" + trap 'shred -u "${TMP}"/* 2>/dev/null || rm -f "${TMP}"/*; rm -rf "${TMP}"' EXIT INT TERM + if [[ -n "${PASSWORD_FILE}" ]]; then + [[ -s "${PASSWORD_FILE}" ]] || die "password file is missing or empty: ${PASSWORD_FILE}" + cp "${PASSWORD_FILE}" "${TMP}/pw" + else + [[ -t 0 ]] || die "no --password-file and stdin is not a terminal — nothing to prompt on" + printf 'The password that encrypts the CA keys — STEPCA_PASSWORD in secrets/sensitive.sops.yaml.\n' + read -rsp 'password: ' pw1; printf '\n' + read -rsp 'again: ' pw2; printf '\n' + [[ "${pw1}" == "${pw2}" ]] || die "the two entries differ" + [[ -n "${pw1}" ]] || die "empty password" + printf '%s' "${pw1}" > "${TMP}/pw" + unset pw1 pw2 + fi + # A trailing newline in the file becomes part of the password step uses and + # the one in SOPS has none; strip it so the two cannot differ by one byte. + printf '%s' "$(cat "${TMP}/pw")" > "${TMP}/pw.trimmed" && mv -f "${TMP}/pw.trimmed" "${TMP}/pw" + chmod 600 "${TMP}/pw" + + if ((FORCE)) && [[ -d "${TREE}" ]]; then + info "removing the existing tree (--force)" + rm -rf "${TREE}" + fi + mkdir -p "${TREE}" + # The container runs as the operator so the tree is owned by the operator + # here; --install re-owns it to step's uid inside the volume on trinity. + # Mounted at /home/step — the image's STEPPATH and where compose.yaml mounts + # the volume — because `step ca init` writes ABSOLUTE paths into ca.json: + # minted under any other mount point, the CA on trinity fails to open + # /tree/certs/root_ca.crt, which is what the first trial did. + chmod 700 "${TREE}" + + dns_flags=() + for name in "${CA_DNS[@]}"; do dns_flags+=(--dns "${name}"); done + + info "minting the root and intermediate (${CA_NAME})" + docker run --rm --network none \ + --user "$(id -u):$(id -g)" \ + -e STEPPATH=/home/step -e HOME=/home/step \ + -v "${TREE}:/home/step" \ + -v "${TMP}/pw:/pw:ro" \ + --entrypoint step "${IMAGE}" \ + ca init \ + --deployment-type standalone \ + --name "${CA_NAME}" \ + "${dns_flags[@]}" \ + --address "${CA_ADDRESS}" \ + --provisioner operator \ + --password-file /pw \ + --provisioner-password-file /pw \ + >"${TMP}/init.log" 2>&1 || { cat "${TMP}/init.log" >&2; die "step ca init failed"; } + + # The ACME provisioner, added against the file rather than with `--acme` at + # init, so the challenge and lifetime claims are in ca.json before the CA + # ever starts — the default provisioner accepts every challenge type and + # issues 24h leaves, and neither is what ADR-0037 decided. + info "adding the ACME provisioner (${ACME_CHALLENGE} only, leaves ${LEAF_DEFAULT})" + docker run --rm --network none \ + --user "$(id -u):$(id -g)" \ + -e STEPPATH=/home/step -e HOME=/home/step \ + -v "${TREE}:/home/step" \ + --entrypoint step "${IMAGE}" \ + ca provisioner add acme --type ACME \ + --challenge "${ACME_CHALLENGE}" \ + --x509-min-dur "${LEAF_MIN}" \ + --x509-default-dur "${LEAF_DEFAULT}" \ + --x509-max-dur "${LEAF_MAX}" \ + --ca-config /home/step/config/ca.json \ + >"${TMP}/prov.log" 2>&1 || { cat "${TMP}/prov.log" >&2; die "step ca provisioner add failed"; } + + # ---- Prove the result before reporting it ------------------------------ + # The chain builds, the root is a CA, the provisioner carries exactly the + # claims above, and ca.json never mentions the root key. Each of these is + # a thing that would otherwise surface as a trust error on trinity. + openssl verify -CAfile "${TREE}/certs/root_ca.crt" "${TREE}/certs/intermediate_ca.crt" >/dev/null 2>&1 \ + || die "the intermediate does not verify against the root — refusing to report success" + openssl x509 -in "${TREE}/certs/root_ca.crt" -noout -ext basicConstraints 2>/dev/null | grep -q 'CA:TRUE' \ + || die "the root certificate is not a CA — refusing to report success" + grep -q 'root_ca_key' "${TREE}/config/ca.json" \ + && die "ca.json references the root key; the bundle would need it and it must not travel" + # Every path in ca.json must be the one the service container will see. + grep -oE '"/[^"]+"' "${TREE}/config/ca.json" | grep -qv '^"/home/step/' \ + && die "ca.json holds a path outside /home/step — the tree was minted at the wrong mount point" + python3 - "${TREE}/config/ca.json" "${ACME_CHALLENGE}" "${LEAF_DEFAULT}" "${LEAF_MAX}" <<'PY' \ + || die "the ACME provisioner in ca.json does not carry the expected claims" +import json, sys +cfg, challenge, default, maximum = sys.argv[1:] +provs = json.load(open(cfg))["authority"]["provisioners"] +acme = [p for p in provs if p.get("type") == "ACME"] +assert len(acme) == 1, f"expected one ACME provisioner, found {len(acme)}" +p = acme[0] +assert p.get("name") == "acme", p.get("name") +assert p.get("challenges") == [challenge], p.get("challenges") +claims = p.get("claims", {}) +def hours(d): # "168h0m0s" -> "168h" + return d.split("h")[0] + "h" if "h" in d else d +assert hours(claims.get("defaultTLSCertDuration", "")) == default, claims +assert hours(claims.get("maxTLSCertDuration", "")) == maximum, claims +PY + + # The root certificate on its own, for Caddy's trusted_roots and for every + # device that reaches the tier. + cp "${TREE}/certs/root_ca.crt" "${ROOT_PEM}" + chmod 644 "${ROOT_PEM}" + chmod 644 "${TREE}/certs/"*.crt + + # ---- The bundle that ships --------------------------------------------- + # Listed by name, never `secrets/`: that directory holds the root key. + info "building the bundle for trinity" + tar -C "${TREE}" -cf "${BUNDLE}" \ + config/ca.json config/defaults.json \ + certs/root_ca.crt certs/intermediate_ca.crt \ + secrets/intermediate_ca_key \ + db + chmod 600 "${BUNDLE}" + # The listing is captured once rather than piped into `grep -q`: with + # pipefail, grep closing the pipe on its first match hands tar SIGPIPE and + # the pipeline reports failure for a bundle that is fine (observed). + listing="$(tar -tf "${BUNDLE}")" + if grep -q 'root_ca_key' <<< "${listing}"; then + rm -f "${BUNDLE}" + die "the bundle contained the root key — deleted it; this is a bug in this script" + fi + + fingerprint="$(openssl x509 -in "${ROOT_PEM}" -noout -fingerprint -sha256 2>/dev/null | cut -d= -f2)" + ok "minted" + printf ' tree %s (stays here; holds the root key)\n' "${TREE#"${REPO_ROOT}"/}" + printf ' root %s (public — this is what devices trust)\n' "${ROOT_PEM#"${REPO_ROOT}"/}" + printf ' bundle %s (what travels to trinity; no root key)\n' "${BUNDLE#"${REPO_ROOT}"/}" + printf ' root SHA256 %s\n' "${fingerprint}" + printf '\nNext, on trinity — docs/runbooks/build-the-tier-ca.md:\n' + printf ' scp the bundle over, then\n' + printf ' make tier-ca ARGS="--install certificates/tier-ca-bundle.tar"\n\n' + exit 0 +fi + +# --------------------------------------------------------------------------- +# --install BUNDLE (trinity) +# --------------------------------------------------------------------------- +[[ -s "${BUNDLE_IN}" ]] || die "no bundle at ${BUNDLE_IN}" +require_docker +# The image this stack pins, never a floating tag — see scripts/image-for.sh. +IMAGE="$(COMPOSE_FILE="${SENSITIVE_COMPOSE}" "${REPO_ROOT}/scripts/image-for.sh" step-ca)" +BUNDLE_ABS="$(cd "$(dirname "${BUNDLE_IN}")" && pwd)/$(basename "${BUNDLE_IN}")" + +# Refuse a bundle that carries the root key, whatever built it. Listed once +# into a variable, not piped into `grep -q` — see the note in --mint. +listing="$(tar -tf "${BUNDLE_ABS}")" +if grep -q 'root_ca_key' <<< "${listing}"; then + die "this bundle contains secrets/root_ca_key. The root key never reaches trinity — +rebuild the bundle on the monitoring host with $(basename "$0") --mint, and delete this copy." +fi +grep -q '^config/ca.json$' <<< "${listing}" || die "not a tier-ca bundle: no config/ca.json inside" + +# A populated volume is a running CA's state. Overwriting it is a re-key of +# the intermediate, which is sometimes the point (a re-mint after a compromise) +# and never an accident. +if docker volume inspect "${VOLUME}" >/dev/null 2>&1; then + if docker run --rm --network none --user 0 -v "${VOLUME}:/home/step:ro" --entrypoint sh "${IMAGE}" \ + -c 'test -s /home/step/config/ca.json' 2>/dev/null && ((! FORCE)); then + die "volume ${VOLUME} already holds a CA (config/ca.json). +Installing over it replaces the intermediate every leaf on this tier chains to. +Pass --force only if that is what you mean — and expect Caddy to re-issue every +certificate on its next renewal." + fi +else + info "creating volume ${VOLUME} with compose's labels" + docker volume create \ + --label "com.docker.compose.project=${COMPOSE_PROJECT}" \ + --label "com.docker.compose.volume=${VOLUME_NAME}" \ + "${VOLUME}" >/dev/null +fi + +# Extract as root, then hand the tree to step's uid — the compose service runs +# as 1000:1000 with every capability dropped, so it must own what it reads. +# `secrets/` closes to 0700, and the bundle's own modes are not trusted. +info "populating ${VOLUME} from ${BUNDLE_IN}" +docker run --rm --network none --user 0 \ + -v "${VOLUME}:/home/step" \ + -v "${BUNDLE_ABS}:/bundle.tar:ro" \ + --entrypoint sh "${IMAGE}" -c ' + set -e + rm -rf /home/step/config /home/step/certs /home/step/secrets /home/step/db + tar -xf /bundle.tar -C /home/step + chown -R 1000:1000 /home/step + chmod 700 /home/step/secrets /home/step/config /home/step/db + chmod 600 /home/step/secrets/* + chmod 644 /home/step/certs/* + ' + +# The root certificate, where compose.yaml mounts it into Caddy. Public. +mkdir -p "${CERT_DIR}" +chmod 700 "${CERT_DIR}" +tar -xOf "${BUNDLE_ABS}" certs/root_ca.crt > "${ROOT_PEM}" +chmod 644 "${ROOT_PEM}" + +# Read the result back out of the volume, as step's own uid, so what is +# reported is what the service will see. +head_ "installed" +docker run --rm --network none --user 1000:1000 \ + -v "${VOLUME}:/home/step:ro" \ + --entrypoint step "${IMAGE}" \ + certificate inspect /home/step/certs/intermediate_ca.crt --short +fingerprint="$(openssl x509 -in "${ROOT_PEM}" -noout -fingerprint -sha256 2>/dev/null | cut -d= -f2 || true)" +[[ -n "${fingerprint}" ]] && printf ' root SHA256 %s (compare with what --mint printed)\n' "${fingerprint}" +printf ' root cert %s (Caddy trusts this; so should every device)\n' "${ROOT_PEM#"${REPO_ROOT}"/}" +printf '\nNext:\n' +printf ' make secrets-edit STACK=sensitive STEPCA_PASSWORD must be the password --mint was given\n' +printf ' make up STACK=sensitive\n\n' diff --git a/secrets/sensitive.example.yaml b/secrets/sensitive.example.yaml index bd34c5da..a28d4396 100644 --- a/secrets/sensitive.example.yaml +++ b/secrets/sensitive.example.yaml @@ -34,9 +34,14 @@ # on the day the tier holds real data, which is what ADR-0023 asks for the # household's secrets. # -# The password below is the one given to `step ca init` when the intermediate -# is minted on the monitoring host; the two must match or step-ca cannot open -# its own key. Long and random; it is typed nowhere after that. +# The password below is the one `make tier-ca ARGS=--mint` is given on the +# monitoring host when the tier's root and intermediate are minted; step +# encrypts both keys with it, and the two copies must match or step-ca cannot +# open its own key. Generate it with `make gen-secret`, put it here first, then +# pass it to --mint from a file (never argv) — docs/runbooks/build-the-tier-ca.md. +# It also unlocks the root key that stays on the monitoring host, whose age key +# cannot open THIS file — so using that root is a two-host operation on +# purpose (ADR-0037). Long and random; it is typed nowhere after that. STEPCA_PASSWORD: change-me-to-something-long # A bcrypt HASH, not the password (#135). AdGuard Home stores its admin diff --git a/stacks/sensitive/.env.example b/stacks/sensitive/.env.example index 49894131..b0d51f11 100644 --- a/stacks/sensitive/.env.example +++ b/stacks/sensitive/.env.example @@ -66,9 +66,9 @@ COMPOSE_PROFILES=ml # sets ALLOWED_HOSTS and CSRF_TRUSTED_ORIGINS in one go, so a browser reaching # the service under any other name gets a 400 on the page and a 403 on every # form. Reaching it under this one needs a host override on morpheus -# (paperless.matrix.elysium → 10.0.99.40, the seventh of ADR-0010's six) and, -# until step-ca issues a leaf per service, the name as a SAN on trinity's -# hand-issued certificate — compose.yaml carries the `make certs` line. +# (paperless.matrix.elysium → 10.0.99.40, the seventh of ADR-0010's six); the +# certificate is step-ca's over ACME, issued to the site block of the same +# name in the Caddyfile, which is also an alias on Caddy in compose.yaml. PAPERLESS_URL=https://paperless.matrix.elysium # The superuser created on first start; its password is PAPERLESS_ADMIN_PASSWORD diff --git a/stacks/sensitive/Caddyfile b/stacks/sensitive/Caddyfile index e7732d15..507a2413 100644 --- a/stacks/sensitive/Caddyfile +++ b/stacks/sensitive/Caddyfile @@ -11,74 +11,113 @@ admin off # The container's own healthcheck connects to https://localhost, whose SNI - # matches no site below; without a default, that handshake has no - # certificate to offer and the check fails on a proxy that is serving fine. + # matches no site below. default_sni covers a ClientHello with no SNI at + # all; fallback_sni covers one whose SNI matches nothing — busybox wget + # sends `localhost`, which is the second case. Without both, that + # handshake has no certificate to offer and the check fails on a proxy + # that is serving fine. default_sni trinity.matrix.elysium + fallback_sni trinity.matrix.elysium + + # 80 is not published (ADR-0012, .env.example), so the HTTP→HTTPS redirect + # Caddy would otherwise listen for on it is a listener nothing can reach. + # Disabled so the container's open ports are the ones compose.yaml says. + auto_https disable_redirects + + # Every certificate this file serves comes from step-ca over ACME, from + # the tier's own intermediate (ADR-0035, #130). `step-ca` resolves on the + # compose network; the directory URL's middle segment is the provisioner + # name scripts/tier-ca.sh creates. trusted_roots is the TIER'S root — the + # file compose.yaml mounts from certificates/tier-ca.pem — not the estate's + # ca.pem, which is a different CA and would reject the directory's + # certificate. The http-01 challenge is disabled here as well as at the + # CA: 80 is not published (ADR-0012), so leaving it enabled would only add + # a failed attempt before the one that works. + cert_issuer acme { + dir https://step-ca:9000/acme/acme/directory + trusted_roots /etc/caddy/tls/ca.pem + disable_http_challenge + } } -# Everything the tier serves, under one leaf for now. The certificate comes -# from scripts/gen-certs.sh (see compose.yaml); when step-ca's ACME provisioner -# is configured (#130) the `tls` line becomes +# Each name served here is ALSO a network alias of the caddy service in +# compose.yaml — step-ca validates the tls-alpn-01 challenge by dialling the +# name on 443 from inside the compose network, and Docker's resolver answers +# with Caddy's address only for names Caddy carries. A block added here +# without its alias fails its first issuance with a DNS error at the CA. # -# tls { -# ca https://step-ca:9000/acme/acme/directory -# ca_root /etc/caddy/tls/ca.pem -# } +# One site block per service, under its own name and therefore its own +# certificate. The next service adds a block of the same shape, its alias in +# compose.yaml, and a host override on morpheus pointing the name at +# 10.0.99.40 (docs/runbooks/add-a-host-override.md) — a name the resolver +# does not know never arrives. # -# and each service gets a name of its own beneath matrix.elysium, issued and -# renewed without a runbook step. -https:// { - tls /etc/caddy/tls/cert.pem /etc/caddy/tls/key.pem +# Home Assistant (#134). Its frontend runs over a websocket, which +# reverse_proxy upgrades without being told; nothing here needs a header +# Caddy does not already set — X-Forwarded-For is what its `trusted_proxies` +# line reads, and Caddy's fixed address is what it trusts. +homeassistant.matrix.elysium { + reverse_proxy home-assistant:8123 - # What the container's healthcheck asks for, on any Host. - handle /healthz { - respond "ok" 200 + log { + output stdout + format json } +} - # One block per service, matched on Host. Each name needs two things - # outside this file: a SAN on the leaf (compose.yaml's `make certs` line) - # and a host override on morpheus pointing it at 10.0.99.40 - # (docs/runbooks/add-a-host-override.md) — a Host the leaf does not carry - # fails the handshake, and a name the resolver does not know never arrives. - # - # Home Assistant (#134). Its frontend runs over a websocket, which - # reverse_proxy upgrades without being told; nothing here needs a header - # Caddy does not already set — X-Forwarded-For is what its - # `trusted_proxies` line reads, and Caddy's fixed address is what it trusts. - @homeassistant host homeassistant.matrix.elysium - handle @homeassistant { - reverse_proxy home-assistant:8123 +# AdGuard Home's UI (#135). Plain HTTP on the compose network; the DNS side +# of the same container is published on 53 directly, because DNS is not +# something this proxy can carry — compose.yaml, DIFFERENCE 7. +adguard.matrix.elysium { + reverse_proxy adguard:80 + + log { + output stdout + format json } +} - # AdGuard Home's UI (#135). Plain HTTP on the compose network; the DNS - # side of the same container is published on 53 directly, because DNS is - # not something this proxy can carry — compose.yaml, DIFFERENCE 7. - @adguard host adguard.matrix.elysium - handle @adguard { - reverse_proxy adguard:80 +# Immich (#132). The mobile app and the browser both talk to this name; the +# server takes uploads on the same port, and reverse_proxy imposes no body +# limit of its own. +immich.matrix.elysium { + reverse_proxy immich-server:2283 + + log { + output stdout + format json } +} - # Paperless-ngx (#133). reverse_proxy sets X-Forwarded-For, -Proto and - # -Host unasked, and compose.yaml's PAPERLESS_USE_X_FORWARD_HOST and - # PAPERLESS_PROXY_SSL_HEADER are what reads them — Django otherwise sees a - # plain-http request for the upstream's own name and rejects every POST. - @paperless host paperless.matrix.elysium - handle @paperless { - reverse_proxy paperless:8000 +# Paperless-ngx (#133). reverse_proxy sets X-Forwarded-For, -Proto and -Host +# unasked, and compose.yaml's PAPERLESS_USE_X_FORWARD_HOST and +# PAPERLESS_PROXY_SSL_HEADER are what reads them — Django otherwise sees a +# plain-http request for the upstream's own name and rejects every POST. +paperless.matrix.elysium { + reverse_proxy paperless:8000 + + log { + output stdout + format json } +} - # Vaultwarden (#131). Plain HTTP on the stack network; TLS ends here. Rocket - # serves the Bitwarden API, the web vault and the WebSocket notifications - # hub on the one port, and reverse_proxy upgrades WebSocket connections - # without being told to. 8080 is compose.yaml's ROCKET_PORT; X-Forwarded-For - # is what its IP_HEADER reads. - @vaultwarden host vaultwarden.matrix.elysium - handle @vaultwarden { - reverse_proxy vaultwarden:8080 +# Vaultwarden (#131). Plain HTTP on the stack network; TLS ends here. Rocket +# serves the Bitwarden API, the web vault and the WebSocket notifications hub +# on the one port, and reverse_proxy upgrades WebSocket connections without +# being told to. 8080 is compose.yaml's ROCKET_PORT; X-Forwarded-For is what +# its IP_HEADER reads. +vaultwarden.matrix.elysium { + reverse_proxy vaultwarden:8080 + + log { + output stdout + format json } +} - # ntfy and Homepage (#136, #137) add a block of the same - # shape above this line. What matches nothing lands here. +# The host's own name: nothing is served under it yet. +trinity.matrix.elysium { handle { respond "trinity: the sensitive tier. Nothing is routed under this name." 404 } @@ -88,3 +127,17 @@ https:// { format json } } + +# Any other name, and the container's own healthcheck. No hostname, so +# automatic HTTPS obtains nothing for this block; the handshake is served +# with the fallback certificate above. +https:// { + # What the container's healthcheck asks for, on any Host. + handle /healthz { + respond "ok" 200 + } + + handle { + respond "trinity: the sensitive tier. No such site." 404 + } +} diff --git a/stacks/sensitive/README.md b/stacks/sensitive/README.md index 5dcf4e62..3527ceb9 100644 --- a/stacks/sensitive/README.md +++ b/stacks/sensitive/README.md @@ -14,7 +14,7 @@ make up STACK=sensitive | Service | Image | Port | Purpose | | --- | --- | --- | --- | | caddy | `caddy` | 443 (https) | The tier's published HTTPS port. Terminates TLS, routes by name to every service behind it ([#129]) | -| step-ca | `smallstep/step-ca` | *internal* (9000) | The tier's certificate authority — an intermediate beneath the lab CA, so nothing that trusts `certificates/ca.pem` is re-pointed ([#130]) | +| step-ca | `smallstep/step-ca` | *internal* (9000) | The tier's certificate authority — a root of its own with an intermediate beneath it, issuing to Caddy over ACME ([#130], [ADR-0037]) | | home-assistant | `ghcr.io/home-assistant/home-assistant` | *internal* (8123) | Home automation, and what the `99 → 20` rule exists for — one pass, to the Hue bridge, scoped by [ADR-0035] ([#134]) | | adguard | `adguard/adguardhome` | 53 (dns), on `10.0.99.40` only | The DNS filter Unbound on `morpheus` forwards to under [ADR-0010] — never a client-facing resolver. The one port besides Caddy's, published to the firewall's forwarder and the blackbox prober and answered for nothing else; the UI is behind Caddy at `adguard.matrix.elysium` ([#135]) | | immich-server | `ghcr.io/immich-app/immich-server` | *internal* (2283) | The photo library — API and job workers in one container, reached as `https://immich.matrix.elysium` through Caddy ([#132]) | @@ -76,9 +76,11 @@ export/ untracked: where document_exporter writes. Likewise Secrets are `secrets/sensitive.sops.yaml`, encrypted to this stack's own rule in `.sops.yaml` — `trinity`'s key opens this file and nothing else of the -estate's (`secrets/sensitive.example.yaml` says why, and lists the six keys). Certificates live under -`certificates/`, untracked, and are issued on the monitoring host where the -CA key stays. +estate's (`secrets/sensitive.example.yaml` says why, and lists the six keys). The one file under +`certificates/` this stack reads is `tier-ca.pem`, the tier's root +certificate, written by `make tier-ca ARGS="--install …"` from the bundle +minted on the monitoring host; every leaf is obtained from step-ca at run +time and lives in Caddy's `/data` volume, never on disk here. ## Things worth knowing before editing @@ -87,36 +89,48 @@ CA key stays. `/config` are root-owned and a named volume inherits that — and names the command that re-checks the premise when the image moves. `cap_drop: [ALL]` plus `no-new-privileges` is what makes it acceptable; do not add capabilities - to make something else work. + to make something else work. step-ca keeps the same one capability for a + different, also measured, reason: its binary ships with the + `NET_BIND_SERVICE` file capability, and the kernel refuses to exec such a + file against a bounding set without it — the first boot failed on + `operation not permitted` before the process existed. It still runs as + `1000:1000`. - **No admin API.** `admin off` in the `Caddyfile` means a routing change is `make up STACK=sensitive`, which recreates the container, not `caddy reload`. The socket would have been unauthenticated and on the same network as every service it fronts. -- **step-ca is an intermediate, and its tree is not made here.** `step ca init` - runs on the monitoring host against `certificates/ca.pem` and its key; the - resulting `config/`, `certs/`, `secrets/` and `db/` populate the - `step-ca-data` volume on `trinity`. The image's entrypoint will not start - without `config/ca.json`, and `DOCKER_STEPCA_INIT_*` is never set, because - either would mint a root of its own. The key password is `STEPCA_PASSWORD` - in SOPS, written to a private tmpfs at start and nowhere on disk. -- **The certificate is hand-issued until ACME is wired.** `make certs - ARGS="--host trinity.matrix.elysium --ip 10.0.99.40 --dns trinity --dns - homeassistant.matrix.elysium --dns adguard.matrix.elysium --dns - immich.matrix.elysium --dns paperless.matrix.elysium --dns - vaultwarden.matrix.elysium"` on the monitoring host, three files copied over - as `build-the-lab-guest.md` §5 does it. `render-config.sh` refuses to render - while any of them is missing. - The `Caddyfile` carries the `tls { ca … }` block that replaces this once - [#130]'s ACME provisioner is configured, and per-service names stop being - a step. -- **The phones have to trust the lab CA.** The mobile app is the whole reason - Immich was chosen over PhotoPrism ([#132]), and it talks to +- **step-ca is a CA of the tier's own, and its tree is not made here.** Not + an intermediate beneath the estate's CA, which this file once claimed: that + root carries `pathlen:0`, and a leaf beneath any intermediate of it fails + `path length constraint exceeded` — measured, and decided in [ADR-0037]. + `make tier-ca ARGS=--mint` on the monitoring host mints the root and + intermediate in this image and writes a bundle *without the root key*; + `make tier-ca ARGS="--install …"` here populates the `step-ca-data` volume + from it and writes `certificates/tier-ca.pem`. The image's entrypoint will + not start without `config/ca.json`, and `DOCKER_STEPCA_INIT_*` is never + set, because either would mint a root of its own. The key password is + `STEPCA_PASSWORD` in SOPS, written to a private tmpfs at start and nowhere + on disk. [`build-the-tier-ca.md`](../../docs/runbooks/build-the-tier-ca.md) + is the procedure. +- **Every certificate is obtained over ACME, and every name is also an + alias.** The `Caddyfile`'s `cert_issuer acme` block points at step-ca's + directory and trusts the tier's root; leaves are seven days and Caddy + renews them. step-ca validates the `tls-alpn-01` challenge by dialling the + requested name on 443 from inside the compose network, so a name served in + the `Caddyfile` must also appear under the `caddy` service's network + `aliases` in `compose.yaml` — one line each, added together. A name with a + block and no alias fails its first issuance with a DNS error at the CA. +- **The phones have to trust the tier's root.** The mobile app is the whole + reason Immich was chosen over PhotoPrism ([#132]), and it talks to `https://immich.matrix.elysium` on a certificate a phone has never heard - of. `certificates/ca.pem` goes onto each phone as a user-installed root - before the app is pointed at the server; Android's Immich app honours a - user root, iOS needs the profile installed and then *enabled* under - Certificate Trust Settings, which is the step people miss. Nothing about - this changes when step-ca issues the leaf — the root is the same one. + of. `certificates/tier-ca.pem` — the tier's root, not the estate's + `ca.pem`, which vouches for nothing here ([ADR-0037]) — goes onto each + phone as a user-installed root before the app is pointed at the server; + Android's Immich app honours a user root, iOS needs the profile installed + and then *enabled* under Certificate Trust Settings, which is the step + people miss ([`build-the-tier-ca.md`](../../docs/runbooks/build-the-tier-ca.md) + §6). Nothing about this changes when a leaf renews — the root is the same + one. - **Immich runs as the deploying user, and the library disk has to be owned by that user.** `immich-server` carries `${RENDER_UID}:${RENDER_GID}` the way Alertmanager does in the estate's stack, and writes only under @@ -146,16 +160,16 @@ CA key stays. says so beside the key. - **80 is not published.** ADR-0012: a port is published when something off-host consumes it, and nothing consumes 80 — browsers on Hicks type - `https`, and ACME's `tls-alpn-01` challenge runs over 443. A redirect is a - later choice, made in `.env.example` and `compose.yaml` together. -- **Every routed name needs a SAN and a host override.** The `Caddyfile` - matches on Host, so `homeassistant.matrix.elysium` — and each name after - it, `vaultwarden.matrix.elysium` included — has to be on the leaf - (`compose.yaml`'s `make certs` line carries one `--dns` per name) and in - `morpheus`'s resolver, pointed at `10.0.99.40` - ([`add-a-host-override.md`](../../docs/runbooks/add-a-host-override.md)). - A name missing from the leaf fails the handshake; one missing from the - resolver never arrives. + `https`, and ACME's `tls-alpn-01` challenge runs over 443; the provisioner + accepts no other challenge, and the `Caddyfile` disables the redirect + listener Caddy would otherwise open on 80. A redirect is a later choice, + made in `.env.example`, `compose.yaml` and the `Caddyfile` together. +- **Every routed name needs a host override.** `homeassistant.matrix.elysium` + — and each name after it, `vaultwarden.matrix.elysium` included — has to be + in `morpheus`'s resolver, pointed at `10.0.99.40` + ([`add-a-host-override.md`](../../docs/runbooks/add-a-host-override.md)) — + a name the resolver does not know never arrives. The certificate side is + the alias bullet above, not a SAN: there is no leaf to put one on. - **Home Assistant is an ordinary member of the network, not `network_mode: host`.** Upstream's example uses host networking and `privileged` for discovery and USB. Discovery is mDNS and SSDP, which are link-local, and @@ -341,12 +355,13 @@ around it is here. for all of them. The token itself is held nowhere in the estate. Keep it with the operator's other credentials: it is precisely the thing this vault cannot hold for you. -- **The name has to be in the leaf.** The browser checks the certificate's SANs - before Caddy sees a Host header, so `vaultwarden.matrix.elysium` is on the - `make certs` line in `compose.yaml` and needs a host override - ([`add-a-host-override.md`](../../docs/runbooks/add-a-host-override.md)). Each - service adds its name there until step-ca's ACME provisioner issues per-name - leaves. +- **The name has its own certificate, and needs a host override.** The browser + checks the certificate's SANs before Caddy sees a Host header, and + `vaultwarden.matrix.elysium` gets a leaf of its own from step-ca because it + is a site block in the `Caddyfile` and an alias on Caddy in `compose.yaml` + (the alias bullet above). What it still needs from outside this stack is + the host override + ([`add-a-host-override.md`](../../docs/runbooks/add-a-host-override.md)). - **TOTP on both accounts at first login.** [ADR-0022]'s floor — the thing [ADR-0008] offered *in place of* SSO, and the one service in the tier where that substitute exists and matters most. Settings → Security → Two-step login @@ -445,13 +460,13 @@ it matters: volume, and the volume is populated by a procedure run on two hosts. A fresh `make up` on a bare `trinity` fails on `config/ca.json`, loudly and on purpose. -- **That the names resolve, or that the leaf carries them.** The host - overrides are a firewall change; `caddy validate` loads a throwaway pair, - and whether the real one carries `trinity.matrix.elysium` and every routed - name — `homeassistant`, `adguard`, `immich`, `paperless` and `vaultwarden`, - all under `matrix.elysium` — in its SANs is checked by the first browser, or - by `openssl x509 -noout -ext subjectAltName` on the monitoring host before - the files travel. +- **That ACME issuance works.** `caddy validate` provisions the issuer against + a throwaway root and dials nothing. Whether step-ca answers, validates the + challenge and signs is proved by the first `make up` — Caddy's log says + `certificate obtained successfully` and the served chain verifies against + `certificates/tier-ca.pem` — and it was proved once on the monitoring host + under a throwaway project before this was written + ([`build-the-tier-ca.md`](../../docs/runbooks/build-the-tier-ca.md) §*Verify*). - **That the limits fit the workload.** 4 cores and 3 GiB for OCR are a statement about the ProDesk made on a different machine. - **That Home Assistant keeps booting under its hardening.** It was booted @@ -478,6 +493,7 @@ it matters: [ADR-0022]: ../../docs/adr/0022-expire-the-sso-deferral-when-the-tier-holds-real-data.md [ADR-0023]: ../../docs/adr/0023-keep-the-household-recovery-path-outside-the-estate.md [ADR-0034]: ../../docs/adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md +[ADR-0037]: ../../docs/adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md [#129]: https://github.com/Gerrrt/HomeLab/issues/129 [#130]: https://github.com/Gerrrt/HomeLab/issues/130 [#131]: https://github.com/Gerrrt/HomeLab/issues/131 diff --git a/stacks/sensitive/compose.yaml b/stacks/sensitive/compose.yaml index e84545aa..8a9fe372 100644 --- a/stacks/sensitive/compose.yaml +++ b/stacks/sensitive/compose.yaml @@ -16,8 +16,10 @@ # containment argument in ADR-0008 — "containers sit on private bridge # networks with only the reverse proxy publishing a port" — is a property of # this file or it is a sentence in an ADR. step-ca (#130) issues the -# certificates behind it. Home Assistant (#134) is the first service behind -# them, and the shape every later one takes: `expose:` and no `ports:`. +# certificates behind it over ACME, from a root of the tier's own (ADR-0037). +# Home Assistant (#134) is the first service behind them, and the shape every +# later one takes: `expose:` and no `ports:`, a site block of its own in the +# Caddyfile, and its name as an alias on Caddy. # AdGuard Home (#135) is the second, and the one service that, by design, # publishes a port of its own — argued at DIFFERENCE 7 below. Immich (#132) # is the third, as four services — server, machine learning, Postgres, @@ -98,9 +100,24 @@ services: # the anchor's `networks: [sensitive]`; the subnet is declared at the # bottom of this file. `.1` is the bridge's own address, so `.2` is the # first a container can hold. + # + # And the aliases, for another reader: the compose network's resolver + # answers a name with a container's address only if the container carries + # that name. step-ca validates Caddy's tls-alpn-01 challenge by dialling + # the requested NAME on 443 from inside this network, so every name the + # Caddyfile serves is listed here too — a name served without an alias + # fails its first issuance with a DNS error at the CA, loudly. One alias + # per service, added in the same commit as its Caddyfile block (ADR-0037). networks: sensitive: ipv4_address: 172.28.99.2 + aliases: + - trinity.matrix.elysium + - homeassistant.matrix.elysium + - adguard.matrix.elysium + - immich.matrix.elysium + - paperless.matrix.elysium + - vaultwarden.matrix.elysium mem_limit: 128m memswap_limit: 128m cap_drop: [ALL] @@ -123,26 +140,19 @@ services: - /tmp:size=16m volumes: - ./Caddyfile:/etc/caddy/Caddyfile:ro + # ACME account and every issued leaf live here; disposable, because + # each leaf is re-issued within a week anyway (ADR-0037). - caddy-data:/data - caddy-config:/config - # The leaf comes from scripts/gen-certs.sh until step-ca issues it - # (#129: "initially, replaced by step-ca issuance once that lands"): - # make certs ARGS="--host trinity.matrix.elysium --ip 10.0.99.40 \ - # --dns trinity --dns homeassistant.matrix.elysium \ - # --dns adguard.matrix.elysium \ - # --dns immich.matrix.elysium --dns paperless.matrix.elysium \ - # --dns vaultwarden.matrix.elysium" - # One --dns per name the Caddyfile routes by, until step-ca issues each - # its own: the browser checks the SANs before Caddy ever sees the Host - # header, so a Host the leaf does not carry fails the handshake, not the - # route. Add a --dns here in the same commit as a route there. Issued on - # the monitoring host, where the CA key lives and stays; the three files - # travel the way the lab's did (build-the-lab-guest.md §5). - # render-config.sh refuses to render while any of these is missing, so - # `make up` cannot start a Caddy that would fail its first TLS handshake. - - ../../certificates/trinity.matrix.elysium.pem:/etc/caddy/tls/cert.pem:ro - - ../../certificates/trinity.matrix.elysium-key.pem:/etc/caddy/tls/key.pem:ro - - ../../certificates/ca.pem:/etc/caddy/tls/ca.pem:ro + # No leaf and no key are mounted: Caddy obtains both from step-ca over + # ACME (the Caddyfile's cert_issuer block). What it needs from disk is + # the TIER'S root certificate — not the estate's ca.pem, which is a + # different CA (ADR-0037) — to verify the ACME directory it dials. + # `make tier-ca ARGS="--install …"` writes this file on trinity from the + # bundle; render-config.sh refuses to render while it is missing, and + # its message says so rather than naming `make certs`, which on this host + # would mint a third CA nobody asked for. + - ../../certificates/tier-ca.pem:/etc/caddy/tls/ca.pem:ro ports: - "${BIND_ADDR:-0.0.0.0}:${HTTPS_PORT:-443}:443" depends_on: @@ -162,18 +172,31 @@ services: # --------------------------------------------------------------------------- # step-ca — the tier's certificate authority (#130) # --------------------------------------------------------------------------- - # An INTERMEDIATE beneath the lab CA, not a new root — decided on #130 and - # recorded here: everything that already trusts certificates/ca.pem keeps - # trusting what this issues, nothing is re-pointed, and the root key never - # leaves the monitoring host. The intermediate is minted there: + # A ROOT OF THE TIER'S OWN, with an intermediate beneath it — ADR-0037. This + # comment used to say "an intermediate beneath the lab CA, so nothing is + # re-pointed", and that was written before anyone read the lab root: it is + # minted with pathlen:0, so a leaf beneath any intermediate of it fails + # "path length constraint exceeded" in every Go client this estate runs. The + # ADR has the measurement and the cost of the alternative. + # + # Both are minted on the monitoring host, in this image, by # - # step ca init --root certificates/ca.pem --key certificates/ca-key.pem ... + # make tier-ca ARGS=--mint (scripts/tier-ca.sh) # - # and the resulting tree (config/, certs/, secrets/, db/) is what populates - # the step-ca-data volume on trinity — the procedure is #404's runbook. The - # image's entrypoint refuses to start without config/ca.json and would - # otherwise mint a fresh root of its own on first boot; a populated volume - # is what stops that, and DOCKER_STEPCA_INIT_* is deliberately never set. + # which writes the tree step-ca runs from (config/, certs/, secrets/, db/) + # under certificates/ there, and a bundle of it WITHOUT the root key. The + # bundle is what populates the step-ca-data volume on trinity, with + # `make tier-ca ARGS="--install …"` — docs/runbooks/build-the-tier-ca.md. + # The root key stays on the monitoring host, cold: ca.json never references + # it (checked by the script), so this service has no use for it. The image's + # entrypoint refuses to start without config/ca.json and would otherwise + # mint a fresh root of its own on first boot; a populated volume is what + # stops that, and DOCKER_STEPCA_INIT_* is deliberately never set. + # + # The ACME provisioner in that ca.json accepts tls-alpn-01 only, on 443 — + # so 80 stays unpublished (ADR-0012) — and issues seven-day leaves. The + # lifetime is the outage budget between this container failing and the + # tier's handshakes failing; ADR-0037 §3 says why it is not the 24h default. # # DIFFERENCE 3: the key password comes from SOPS by way of the environment # and a tmpfs file. step-ca wants a --password-file; the estate's rule is @@ -184,8 +207,10 @@ services: # the environment of either), and nothing on disk. `$$` is compose's # escape: the container sees `${STEPCA_PASSWORD}`, not this file's value. # - # Nothing publishes 9000. Caddy reaches it on this network for ACME once - # #130's provisioner is configured; a browser never needs it. + # Nothing publishes 9000. Caddy reaches it on this network for ACME; a + # browser never needs it. The day the estate's Grafana renews from here + # (ADR-0037's reopen) is the day it gets an off-host consumer and ADR-0012 + # says to publish it — not before. step-ca: <<: *service-defaults image: smallstep/step-ca:0.30.2@sha256:a2b17872915c193259b75a5474c398326f41bd199f0842093e52cf4182bc8270 @@ -194,6 +219,16 @@ services: memswap_limit: 256m user: "1000:1000" cap_drop: [ALL] + # Measured on the first boot, which the foundation commit never had: with + # every capability dropped, `exec /usr/local/bin/step-ca` fails with + # "operation not permitted" before main() runs. The image ships the binary + # with the NET_BIND_SERVICE file capability set (so it can bind 443 as a + # non-root user), and the kernel refuses to exec a file whose effective + # capability is outside the process's bounding set rather than running it + # unprivileged. Keeping that one capability in the bounding set is what + # makes the exec succeed; the process binds 9000 and would not need it. + # Same single capability Caddy keeps, for a different reason. + cap_add: [NET_BIND_SERVICE] security_opt: - no-new-privileges:true read_only: true @@ -214,9 +249,12 @@ services: - "9000" healthcheck: # `step ca health` is the CA asking itself, over TLS, against its own - # root — which is the lab CA's root once the tree is populated. The - # `step` binary ships in the image; check_compose_health.py --probe - # proves that on every run rather than trusting this comment. + # root — the tier's, out of the installed tree. `localhost` is in the + # serving certificate's SANs because tier-ca.sh puts it there at init; + # without it this check fails on hostname mismatch against a CA that is + # serving fine. The `step` binary ships in the image; + # check_compose_health.py --probe proves that on every run rather than + # trusting this comment. test: - CMD - step @@ -815,12 +853,12 @@ services: # to the one service that owns it, and stop only that. # # Reached as https://paperless.matrix.elysium through Caddy, on the existing - # 50→99 pass for 443 and no new rule. Two things outside this file have to be - # true first, and both belong to #404's build rather than to this file: the - # name needs a host override on morpheus (docs/runbooks/add-a-host-override.md) - # and, until step-ca's ACME provisioner issues a leaf per service, a SAN on - # the hand-issued one — the `make certs` line on Caddy's mounts above - # carries it. + # 50→99 pass for 443 and no new rule. One thing outside this file has to be + # true first, and it belongs to #404's build rather than to this file: the + # name needs a host override on morpheus (docs/runbooks/add-a-host-override.md). + # The certificate is step-ca's, obtained over ACME for the site block of + # this name in the Caddyfile; the alias on Caddy's network block above is + # what lets the CA validate it. # # ROOTLESS, as the operator. The image is s6-overlay over Django and # ordinarily starts as root, maps to uid 1000 through USERMAP_UID and drops