From 79fd90de6ac1791beac39646f34e7c1e583cd0e1 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Mon, 28 Sep 2026 21:10:50 +0000 Subject: [PATCH] feat(sensitive): ntfy in-house, and Alertmanager repointed at it (#136) Adds ntfy to the sensitive tier, behind Caddy at ntfy.matrix.elysium, and repoints Alertmanager's three real channels at it. urgent and security also keep a second webhook to their ntfy.sh topics, because a phone away from home cannot reach trinity: nothing on the tier is exposed, and WireGuard goes to the lab. The heartbeat stays on healthchecks.io, and check_alert_channels.py now fails a heartbeat pointed in-house. ntfy is deny-all. Two users come from SOPS through NTFY_AUTH_*: alertmanager may only write the three topics, by token, and phone may only read them. Measured on v2.28.0 under the compose hardening: 403s where expected, and a rotated token or hash is a 401 after restart, so the volume is disposable. Alertmanager sends the token as a header, not ?auth=, because Caddy logs URLs. It verifies against the tier's root through a per-receiver ca_file. A dead ntfy still pages. A new http_2xx_tier_ca blackbox probe, written out and enabled at cutover, raises EndpointUnreachable. That alert and AlertmanagerNotificationsFailing are both critical, so they reach urgent and its ntfy.sh copy; validate.sh pins the route. Not deployed. The cutover is in docs/runbooks/verify-the-alert-path.md. Refs #136 Co-Authored-By: Claude Opus 5.5 --- .github/workflows/ci.yml | 11 +- Makefile | 4 +- README.md | 4 +- docs/architecture.md | 2 +- docs/changelog.md | 38 +++++++ docs/observability.md | 17 ++- docs/roadmap.md | 8 +- .../runbooks/build-the-sensitive-tier-host.md | 21 ++-- docs/runbooks/restore-the-sensitive-tier.md | 3 +- docs/runbooks/verify-the-alert-path.md | 102 ++++++++++++++++- docs/security.md | 16 +++ scripts/backup-volumes.sh | 8 ++ scripts/check_alert_channels.py | 23 +++- scripts/render-config.sh | 31 +++++- scripts/seed-validation-env.sh | 13 +++ scripts/validate.sh | 8 +- secrets/observability.example.yaml | 37 +++++-- secrets/sensitive.example.yaml | 43 +++++++- .../alertmanager/alertmanager.yaml | 42 +++++++ stacks/observability/blackbox/blackbox.yaml | 28 ++++- stacks/observability/compose.yaml | 10 +- .../prometheus/rules/host.rules.yaml | 3 +- .../prometheus/rules/stack.rules.yaml | 9 +- .../prometheus/targets/blackbox.yaml | 33 ++++++ .../prometheus/tests/stack.test.yaml | 4 +- stacks/sensitive/Caddyfile | 14 +++ stacks/sensitive/README.md | 82 ++++++++++++-- stacks/sensitive/compose.yaml | 104 +++++++++++++++++- stacks/sensitive/ntfy/server.yml | 44 ++++++++ 29 files changed, 710 insertions(+), 52 deletions(-) create mode 100644 stacks/sensitive/ntfy/server.yml diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 2cd894bc..3706c516 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -518,7 +518,8 @@ jobs: path: .boot-proofs key: >- boot-proofs-${{ hashFiles('stacks/sensitive/compose.yaml', - 'stacks/sensitive/home-assistant/**', 'scripts/check_hardened_boot.sh') }} + 'stacks/sensitive/home-assistant/**', 'stacks/sensitive/ntfy/**', + 'scripts/seed-validation-env.sh', 'scripts/check_hardened_boot.sh') }} restore-keys: boot-proofs- - name: Boot Home Assistant under its hardening @@ -526,6 +527,14 @@ jobs: ./scripts/check_hardened_boot.sh --stack sensitive --service home-assistant --proof-cache .boot-proofs/proofs.txt + # ntfy exits at start on a malformed hash or token, which is why + # seed-validation-env.sh seeds it well-formed ones (#136); this is the + # step that would say so if it stopped. + - name: Boot ntfy under its hardening + run: >- + ./scripts/check_hardened_boot.sh --stack sensitive + --service ntfy --proof-cache .boot-proofs/proofs.txt + # --------------------------------------------------------------------------- secret-scan: name: Secret scan diff --git a/Makefile b/Makefile index e617b485..0347b2f8 100644 --- a/Makefile +++ b/Makefile @@ -659,10 +659,12 @@ gen-secret: ## Generate a random secret (ARGS=--snmp for one per SNMP device) ./scripts/gen-secret.sh $(ARGS) .PHONY: hash-password -hash-password: ## Bcrypt a password for AdGuard Home's admin account (prompted, never an argument) +hash-password: ## Bcrypt a password for AdGuard's admin or ntfy's users (prompted, never an argument) @# AdGuard Home keeps its admin password as a bcrypt hash, and the hash is @# what secrets/sensitive.sops.yaml holds as ADGUARD_ADMIN_PASSWORD_HASH — @# so the plaintext is typed once, here, and lives in the password manager. + @# ntfy's two NTFY_*_PASSWORD_HASH keys are made the same way: ntfy accepts + @# this cost-14 hash as it is (checked against the pinned image, #136). @# `caddy hash-password` prompts without echo on a terminal and reads one @# line from stdin without one, so `-t` is passed only when there is a tty @# to pass; either way the password is never an argument, never in shell diff --git a/README.md b/README.md index 180deedc..dde93ee4 100644 --- a/README.md +++ b/README.md @@ -208,8 +208,8 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md). │ # built 2026-09-27 (ADR-0030, #266, #267) ├── stacks/sensitive/ # the household's tier — Caddy, step-ca, Home │ # Assistant, AdGuard Home, Immich, Paperless-ngx, -│ # Vaultwarden and Homepage so far; its own CA, leaves over -│ # ACME (ADR-0037); on trinity since 2026-09-28 (ADR-0034, #404) +│ # Vaultwarden, Homepage and ntfy so far; its own CA, +│ # leaves over ACME (ADR-0037); on trinity since 2026-09-28 (ADR-0034, #404) ├── stacks/media/ # Jellyfin and Navidrome on smaug, under TrueNAS's │ # own Docker — deployed by hand, no secrets file │ # by decision (ADR-0040, #528) diff --git a/docs/architecture.md b/docs/architecture.md index c63bd447..6d9fccfc 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -190,7 +190,7 @@ hole from the monitoring VLAN into the monitored one. | `odin` (10.0.30.60) | 🟢 30 | [`stacks/soc`](../stacks/soc) | Wazuh (indexer, manager, dashboard), Velociraptor and Alloy: the security half of ADR-0007, placed by [ADR-0030](adr/0030-give-the-security-tooling-its-own-guest-and-its-own-stack.md) on a second guest on `Saruman` because `alexander`'s 8 GiB cannot hold both. Built 2026-09-27 by [`build-the-soc-guest.md`](runbooks/build-the-soc-guest.md), the stack authored and CI-validated ahead of the guest the way `stacks/lab` was ahead of `alexander` ([#266](https://github.com/Gerrrt/HomeLab/issues/266), [#267](https://github.com/Gerrrt/HomeLab/issues/267)); the six domain machines enrol as agents by GPO, which is the day those two issues close. Its Alloy pushes to `alexander`, never to VLAN 99, and the indexer's health is the one series that crosses into the lab's Prometheus. Alloy agent (Docker) | | `phoenix` (10.0.30.70) | 🟢 30 | *(none — a toolchain host, no Docker)* | The deployment host, built 2026-09-20: a Proxmox API token, an SSH key and a checkout, so that the Packer, OpenTofu and Ansible work after [#436](https://github.com/Gerrrt/HomeLab/issues/436) has somewhere to run from. A guest on `Saruman`, placed by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), which also decides that the estate's CA key stays on `prometheus` and does not follow the toolchain here. It holds no age key and converges nothing; it is the WireGuard endpoint of [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md), built 2026-09-22 ([#442](https://github.com/Gerrrt/HomeLab/issues/442)); the one thing it is meant to reach that no other guest does is `8006` on `Saruman`, by a single host-firewall rule — written 2026-09-20, when [#566](https://github.com/Gerrrt/HomeLab/issues/566) enabled the firewall the build had found disabled and narrowed the `local_network` alias that was admitting the whole segment underneath it. Its Alloy pushes to `alexander`, never to VLAN 99; the build is [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). Alloy agent (native package, `scripts/deploy-agent.sh`) | | `oracle` (10.0.99.30) | 🔴 99 | *(none — hand-run containers)* | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copies of the firewall export (`make backup-firewall`), of the weekly volume sets (`make backup`, [#535](https://github.com/Gerrrt/HomeLab/issues/535)) and of the media tier's state pulled off `smaug` (`make backup-nas`, [ADR-0045](adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md)). The estate's host for small off-host jobs — [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) | -| `trinity` (10.0.99.40) | 🔴 99 | [`stacks/sensitive`](../stacks/sensitive) | ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md). **Built and deployed 2026-09-28** ([#404](https://github.com/Gerrrt/HomeLab/issues/404), [`build-the-sensitive-tier-host.md`](runbooks/build-the-sensitive-tier-host.md)): Ubuntu 26.04 with both disks LUKS-encrypted and the root unlocked by the TPM ([ADR-0054](adr/0054-encrypt-trinitys-disks-and-seal-the-root-key-to-the-tpm.md)), all twelve containers healthy, and since that day AdGuard is the one forwarder behind `morpheus`'s resolver ([ADR-0055](adr/0055-forward-to-adguard-alone.md)). Not yet holding real data: the #404 step 10 gate stands. The stack: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github.com/Gerrrt/HomeLab/issues/129), [#130](https://github.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github.com/Gerrrt/HomeLab/issues/134)), Immich — four containers behind Caddy with a memory limit on each ([#132](https://github.com/Gerrrt/HomeLab/issues/132)) — Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github.com/Gerrrt/HomeLab/issues/131)) run beside them, with Homepage at `home.matrix.elysium` as the household's page of links to them, its three live tiles on read-only tokens ([#137](https://github.com/Gerrrt/HomeLab/issues/137)); ntfy follows. Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s | +| `trinity` (10.0.99.40) | 🔴 99 | [`stacks/sensitive`](../stacks/sensitive) | ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md). **Built and deployed 2026-09-28** ([#404](https://github.com/Gerrrt/HomeLab/issues/404), [`build-the-sensitive-tier-host.md`](runbooks/build-the-sensitive-tier-host.md)): Ubuntu 26.04 with both disks LUKS-encrypted and the root unlocked by the TPM ([ADR-0054](adr/0054-encrypt-trinitys-disks-and-seal-the-root-key-to-the-tpm.md)), all twelve containers healthy, and since that day AdGuard is the one forwarder behind `morpheus`'s resolver ([ADR-0055](adr/0055-forward-to-adguard-alone.md)). Not yet holding real data: the #404 step 10 gate stands. The stack: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github.com/Gerrrt/HomeLab/issues/129), [#130](https://github.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github.com/Gerrrt/HomeLab/issues/134)), Immich — four containers behind Caddy with a memory limit on each ([#132](https://github.com/Gerrrt/HomeLab/issues/132)) — Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github.com/Gerrrt/HomeLab/issues/131)) run beside them, with Homepage at `home.matrix.elysium` as the household's page of links to them, its three live tiles on read-only tokens ([#137](https://github.com/Gerrrt/HomeLab/issues/137)). ntfy ([#136](https://github.com/Gerrrt/HomeLab/issues/136)) is authored as the in-house receiver for Alertmanager's three real channels. It is not yet deployed and not yet cut over: [`verify-the-alert-path.md`](runbooks/verify-the-alert-path.md) is the step that makes it true. Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s | | `smaug` (10.0.40.30) | 🟡 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **Built, pooled and deployed**: TrueNAS on its boot SSD, the static above since 2026-09-16, the four inbound rules verified in position, the mirror `erebor` since 2026-09-18 and this stack running on it since 2026-09-19 — from a copy of the compose file on the pool, brought up with `docker compose` under TrueNAS's own Docker ([`build-the-nas.md`](runbooks/build-the-nas.md) §6). Jellyfin, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github.com/Gerrrt/HomeLab/issues/138)); and Audiobookshelf on 13378, **authored and not yet deployed**, whose clients are phones on Hicks and which therefore brings a fifth inbound rule, created when it is deployed ([#140](https://github.com/Gerrrt/HomeLab/issues/140), [ADR-0050](adr/0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md)). And Navidrome on 4533 for music, **authored and not yet deployed** on the same terms, whose Hicks pass already exists — created 2026-09-22, ahead of the service and of Audiobookshelf's ([#141](https://github.com/Gerrrt/HomeLab/issues/141)). The `media` share gets a Hicks pass on `445` for workstations, used by a second SMB user, `samwise`, so the televisions' `bilbo` stays theirs; it was **created on 2026-09-23** ([#523](https://github.com/Gerrrt/HomeLab/issues/523), [ADR-0051](adr/0051-let-hicks-workstations-mount-the-media-share-as-a-user-of-their-own.md)). Scraped by `prometheus` on `9100`; it pushes nothing, and runs no Alloy — the estate's first scraped host, and the reason [#256](https://github.com/Gerrrt/HomeLab/issues/256) was more than a line of YAML. That issue settled the fork TrueNAS opened in it: `node_exporter`, as a digest-pinned container in this stack rather than TrueNAS's own endpoint, so the existing `99 → 40:9100` pass, the `host-overview` dashboard and seven rules in `host.rules.yaml` all keep working unchanged. The `node` job and `prometheus/targets/node.yaml` are live, and the target with them since 2026-09-19, once the exporter answered from the pool. SMART reaches the same scrape by [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md): a root cron job in TrueNAS's UI runs the estate's collector and the exporter serves the file — still no agent, still nothing initiating from 40 — and patch state is deliberately not collected on an appliance ([#483](https://github.com/Gerrrt/HomeLab/issues/483)) | | `bahamut` (10.0.30.50) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 domain controller, PDC emulator and DNS for `ad.matrix.elysium` — Tier 0. Static, because every member finds a DC through DNS and the DCs *are* the DNS. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | | `leviathan` (10.0.30.51) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 second domain controller and DNS — Tier 0. Static, for the same reason. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | diff --git a/docs/changelog.md b/docs/changelog.md index 2db8e293..9e07fce3 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -19,6 +19,44 @@ docstring gives: it is a record, not a claim about now. ## 2026-09-28 +- **ntfy is authored for the sensitive tier, and Alertmanager is repointed + at it** ([#136](https://github.com/Gerrrt/HomeLab/issues/136)). It is not + deployed, and the issue stays open until the cutover in + [`verify-the-alert-path.md`](runbooks/verify-the-alert-path.md) has been seen + working on both phones. + - **The decision #136 asked for.** The in-house topics replace ntfy.sh for + all three real channels, and `urgent` and `security` send to their ntfy.sh + topics as well. A phone away from home cannot reach `trinity`: nothing on + the tier is exposed, and the WireGuard path goes to the lab. So an + in-house-only page would wait for Wi-Fi. The heartbeat stays on + healthchecks.io (#408), and `check_alert_channels.py` now fails a heartbeat + pointed at any `.matrix.elysium` host. + - **The auth.** ntfy is deny-all, with two users. `alertmanager` may only + write the three topics, by token; `phone` may only read them. Users, ACLs + and the token come from `NTFY_AUTH_*` in the environment, built from SOPS. + Measured on the pinned image (v2.28.0) under the compose file's hardening: + - Anonymous requests got 403 either way. + - The token could publish and got 403 on read. + - `phone` could read and got 403 on publish. + - A rotated token and a changed hash each turned the old credential into + a 401 after a restart. + + That last result is why `ntfy-data` is skipped by the backup rather than + archived. + - **Only the iPhone's wake-up leaves the house.** `upstream-base-url` sends + ntfy.sh the SHA-256 of the topic's URL, read off the image's own log, and + not the alert. + - **A dead ntfy still pages.** A blackbox probe verifies the tier's root with + the new `http_2xx_tier_ca` module. It is written out and enabled at + cutover. `AlertmanagerNotificationsFailing` and `EndpointUnreachable` are + critical, so they reach `urgent` and its ntfy.sh copy. `validate.sh` now + pins the `correctness` route alongside `availability`. + - **One thing the first boot taught.** ntfy exits at start on a malformed + bcrypt hash or token. `seed-validation-env.sh`'s `validation-only` would + have failed the new CI boot, so the ntfy seeds are well formed. + `check_hardened_boot.sh` could not run on `trinity` itself, because the + live stack owns the subnet. The same boot was run there under a throwaway + project with the subnet moved, and it came up healthy. - **Immich's restore is rehearsed on `trinity`, and the first real photos came before the gate that was meant to precede them** ([#132](https://github.com/Gerrrt/HomeLab/issues/132)). Two accounts diff --git a/docs/observability.md b/docs/observability.md index 252a62db..83f8f979 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -602,6 +602,20 @@ stops matching, and `amtool check-config` still reports SUCCESS — that mutatio was tried. `scripts/validate.sh` and CI therefore assert the table itself with `amtool config routes test --verify.receivers`, one assertion per row. +**Where the three real channels deliver** changed on +[#136](https://github.com/Gerrrt/HomeLab/issues/136). They now go to the +sensitive tier's own ntfy on `trinity`, deny-all, with Alertmanager publishing +by bearer token and verifying the leaf against the tier's root. `urgent` and +`security` also keep a second webhook to their ntfy.sh topic, because a phone +off the home network cannot reach the tier. Alertmanager counts each delivery +separately, so either half can fail without taking the other with it. If the +in-house ntfy is the thing that has failed, the page still goes out: +`EndpointUnreachable` (from the `http_2xx_tier_ca` probe of +`ntfy.matrix.elysium`) and `AlertmanagerNotificationsFailing` are both +critical, so both reach `urgent` and its ntfy.sh copy. The runbook has the +table and the cutover: +[`verify-the-alert-path.md`](runbooks/verify-the-alert-path.md#where-the-real-alerts-go). + Inhibit rules stop cascades: a down host suppresses its own disk warnings, a dead `snmp-exporter` suppresses the "every device is unreachable" storm that would otherwise follow, and a certificate inside seven days of expiry suppresses @@ -716,7 +730,8 @@ spending almost all of ntfy.sh's free daily budget, so real alerts were refused at the end of every day ([#407](https://github.com/Gerrrt/HomeLab/issues/407)). The heartbeat now pings a healthchecks.io check, period 5m and grace 15m, which emails when a ping does not arrive; the three real channels stay on ntfy and -have the budget to themselves. +have the budget to themselves. (Since #136 only two of them use ntfy.sh at all, +as the off-network copy of what the in-house ntfy receives; see *Routing*.) So now, if Prometheus stops evaluating, Alertmanager dies, or this host loses outbound network, something external notices — in principle. That is the diff --git a/docs/roadmap.md b/docs/roadmap.md index d62e252f..afa9de25 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -144,8 +144,12 @@ has been rehearsed on it. [#133](https://github.com/Gerrrt/HomeLab/issues/133) Paperless-ngx; [#134](https://github.com/Gerrrt/HomeLab/issues/134) Home Assistant ([ADR-0035](adr/0035-scope-the-99-to-20-rule-to-the-hue-bridge.md)); - [#136](https://github.com/Gerrrt/HomeLab/issues/136) ntfy, which decides - whether the in-house topic replaces the external ones or sits beside them; + [#136](https://github.com/Gerrrt/HomeLab/issues/136) ntfy — authored, with + the decision taken both ways at once: the in-house topics replace ntfy.sh for + all three real channels, and `urgent` and `security` page ntfy.sh as well, + because a phone away from home cannot reach the tier. The deploy and the + observed end-to-end check remain + (→ [runbook](runbooks/verify-the-alert-path.md#cutting-over-to-the-in-house-ntfy)); and [#137](https://github.com/Gerrrt/HomeLab/issues/137) Homepage, authored on 2026-09-28 against the running tier: `home.matrix.elysium`, a directory and not a status page, with live numbers only where a read-only token can diff --git a/docs/runbooks/build-the-sensitive-tier-host.md b/docs/runbooks/build-the-sensitive-tier-host.md index 023355de..e0723a58 100644 --- a/docs/runbooks/build-the-sensitive-tier-host.md +++ b/docs/runbooks/build-the-sensitive-tier-host.md @@ -322,7 +322,7 @@ hands off again, and `df -h /srv/immich` must show about 1.8T on `/dev/mapper/immich`. That proves the `crypttab` and `fstab` lines together, before any photograph depends on them. -## 6. Its own age key, and the seven secrets +## 6. Its own age key, and the fifteen secrets In your own terminal on `trinity`. None of this goes into a shared session. @@ -345,6 +345,11 @@ what each key is for. The values: | `PAPERLESS_SECRET_KEY` | `make gen-secret` | No | | `PAPERLESS_DBPASS` | `make gen-secret` | No | | `PAPERLESS_ADMIN_PASSWORD` | `make gen-secret` | **Yes**, it is a login | +| `HOMEPAGE_IMMICH_API_KEY`, `HOMEPAGE_PAPERLESS_TOKEN` | Minted in Immich and Paperless-ngx once they are up, each read-only — the clicks are beside the keys in the example file | No | +| `NTFY_ALERTMANAGER_TOKEN` | the pinned image's `ntfy token generate` — the command is beside the key | No. The monitoring host's `ALERTMANAGER_NTFY_TOKEN` is the other copy | +| `NTFY_ALERTMANAGER_PASSWORD_HASH` | `make gen-secret`, then `make hash-password` on it | No — the password is thrown away; this user logs in by token | +| `NTFY_PHONE_PASSWORD_HASH` | `make hash-password` | **The password** — each phone's ntfy app logs in with it | +| `NTFY_TOPIC_ALERTS`, `_URGENT`, `_SECURITY` | `make gen-secret`, three times | No. The monitoring host's receiver URLs end in the same three | ```bash make secrets-edit STACK=sensitive @@ -404,9 +409,9 @@ make backup-firewall **Host overrides**, as [`add-a-host-override.md`](add-a-host-override.md) describes. One entry, host `trinity`, domain `matrix.elysium`, address -`10.0.99.40`. The other five names go under *Additional Names for this Host*: +`10.0.99.40`. The other seven names go under *Additional Names for this Host*: -`homeassistant`, `immich`, `paperless`, `vaultwarden`, `adguard` +`homeassistant`, `immich`, `paperless`, `vaultwarden`, `adguard`, `home`, `ntfy` These are exactly the `Caddyfile`'s site names and the `caddy` service's aliases in `compose.yaml`. A later service adds its name in all three places. @@ -462,11 +467,11 @@ make ps STACK=sensitive make check-container-health STACK=sensitive ``` -All twelve services must be healthy, with the `ml` profile on as `.env.example` ships it. +All fourteen services must be healthy, with the `ml` profile on as `.env.example` ships it. Then the stack README's list, on the host it was written for: - **The CA tree and ACME.** [`build-the-tier-ca.md`](build-the-tier-ca.md) - §5, once for each of the six names: `certificate obtained` in Caddy's log, + §5, once for each of the eight names: `certificate obtained` in Caddy's log, and `Verify return code: 0` against `certificates/tier-ca.pem`. - **The library is on the USB disk.** `docker exec sensitive-immich-server df -h /data` shows the `/dev/mapper/immich` filesystem, not the root. @@ -515,6 +520,7 @@ floor, and recovery codes go in the password manager. | `https://paperless.matrix.elysium` | `admin` and the password from §6 | The profile's *Two-factor authentication* | | `https://immich.matrix.elysium` | The first sign-up is the admin | None. ADR-0022 records Immich as unable | | `https://adguard.matrix.elysium` | The password behind §6's hash | None — likewise | +| `https://ntfy.matrix.elysium` | User `phone` in the ntfy app on each phone, per the stack README's ntfy section; then the cutover in [`verify-the-alert-path.md`](verify-the-alert-path.md) | None. ntfy has none, and `phone` can only read alert text | Home Assistant's Hue integration is added **by address**, `10.0.20.20`, pressing the bridge's button when asked. That is the first traffic §8's pass @@ -552,8 +558,9 @@ make restore STACK=sensitive ARGS="--dry-run --from latest" ``` Both sides must be listed, and the dry run must pass. Ten volumes are -archived. `immich-model-cache` and `adguard-work` are skipped by name, so AdGuard -keeps answering the house's DNS while the rest of the stack is stopped. On +archived. `immich-model-cache`, `adguard-work` and `ntfy-data` are skipped by +name, so AdGuard keeps answering the house's DNS, and ntfy keeps delivering, +while the rest of the stack is stopped. On 2026-09-28, 150 of 150 lookups through `morpheus` succeeded during a backup. [`restore-the-sensitive-tier.md`](restore-the-sensitive-tier.md) §0 is the rest of what must be true. diff --git a/docs/runbooks/restore-the-sensitive-tier.md b/docs/runbooks/restore-the-sensitive-tier.md index 07900ce8..98cb9034 100644 --- a/docs/runbooks/restore-the-sensitive-tier.md +++ b/docs/runbooks/restore-the-sensitive-tier.md @@ -1,6 +1,6 @@ # Runbook: Restore the sensitive tier -**Target:** the twelve Docker data volumes on `trinity` (10.0.99.40), VLAN 99 — ten of which the backup archives +**Target:** the thirteen Docker data volumes on `trinity` (10.0.99.40), VLAN 99 — ten of which the backup archives **Time:** ten minutes for one volume; half an hour for the set on a rebuilt host **You will need:** a backup set, an age identity the set was encrypted to — `trinity`'s own key, or the technical second's — and the stack stopped; the @@ -26,6 +26,7 @@ most are rebuildable or re-fetched from somewhere else and five are not: | `step-ca-data` | The intermediate CA's tree | Re-minted on the monitoring host from the lab CA's key — [#404](https://github.com/Gerrrt/HomeLab/issues/404)'s procedure. Costs a runbook step, not data | | `caddy-data` | Caddy's storage: `instance.uuid`, the lock directory, later the ACME state | Recreated on the next start. Nothing here is worth a restore until step-ca issues leaves into it | | `caddy-config` | `autosave.json`, Caddy's copy of its last loaded config | Recreated on the next start from the `Caddyfile` | +| `ntfy-data` | ntfy's `user.db` and twelve hours of message cache | Nothing. **Not archived at all** ([#136](https://github.com/Gerrrt/HomeLab/issues/136)): the users, access list and token are re-provisioned from `secrets/sensitive.sops.yaml` on every start, and the cache is notifications already delivered. `make up` creates it empty and ntfy fills it | | `adguard-work` | AdGuard's blocklists, query log, statistics and UI sessions | Re-downloaded and re-accumulated. **Not archived at all since 2026-09-28**: archiving it meant stopping AdGuard, which since that day is the house's only DNS forwarder, and the query log is the household's browsing history. The settings are `adguard/AdGuardHome.yaml`, in git | So this runbook is mostly about those five, and [#131](https://github.com/Gerrrt/HomeLab/issues/131) diff --git a/docs/runbooks/verify-the-alert-path.md b/docs/runbooks/verify-the-alert-path.md index fb99a7f5..2cc13363 100644 --- a/docs/runbooks/verify-the-alert-path.md +++ b/docs/runbooks/verify-the-alert-path.md @@ -31,6 +31,105 @@ route proves delivery to a *different* URL than real alerts use, so it cannot see a deleted ntfy topic. The daily route travels the identical URL your warnings travel, but nothing machine-checks its absence — you do. +## Where the real alerts go + +Since [#136](https://github.com/Gerrrt/HomeLab/issues/136) the three real +channels are delivered to the sensitive tier's own ntfy, at +`https://ntfy.matrix.elysium` on `trinity`. They are no longer delivered to +ntfy.sh, with one deliberate exception: + +| Receiver | In-house ntfy | ntfy.sh | Why | +| --- | --- | --- | --- | +| `default` | yes | no | A slow scrape can wait until the phone is home | +| `urgent` | yes | **yes** | A page has to reach a phone that is not on the home network | +| `security` | yes | **yes** | Likewise | +| `heartbeat` | **never** | no | healthchecks.io. A watcher inside the house fails with the house | + +The ntfy.sh copies exist because nothing about the tier is exposed. A phone +off the home network cannot reach `trinity`, since the WireGuard path +([ADR-0042](../adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md)) +goes to the lab and not here. So an in-house-only page would wait in the cache +until the phone came home. With the heartbeat gone from ntfy.sh (#407), two +channels that fire on real faults sit far inside its free budget. + +Two consequences are worth holding in mind: + +- **The daily Watchdog proves the in-house path, not the ntfy.sh copies.** The + 24h route goes to `default`, which has no external twin. The copies are + proved by the cutover drill below and by every real page after it. The + comment on that route in `alertmanager.yaml` says why a daily Watchdog on the + loud channels would be worse than the gap. +- **A dead in-house ntfy still reaches you.** It surfaces as + `EndpointUnreachable` for `ntfy` (the blackbox probe) and as + `AlertmanagerNotificationsFailing`, once anything tries to page. Both are + critical, so both route to `urgent`, whose ntfy.sh copy does not depend on + the thing that failed. `scripts/validate.sh` pins those two routes. + +## Cutting over to the in-house ntfy + +Run this once, when #136 is deployed. Do it in order: each step's check is +what makes the next one mean anything. + +1. **Secrets on `trinity`.** Run `make secrets-edit STACK=sensitive` and set + the six `NTFY_*` keys, as `secrets/sensitive.example.yaml` describes them. + Keep the `phone` password in the password manager. +2. **Serve it.** On `trinity`, run `make up STACK=sensitive`, then look for + `certificate obtained` for `ntfy.matrix.elysium` in Caddy's log. +3. **The name.** Add `ntfy` under *Additional Names for this Host* on + `trinity`'s host override + ([`add-a-host-override.md`](add-a-host-override.md)). From `prometheus`, + this must print `{"healthy":true}`: + + ```bash + curl --cacert certificates/tier-ca.pem https://ntfy.matrix.elysium/v1/health + ``` + +4. **Deny-all, from outside.** An anonymous publish must answer `403`: + + ```bash + curl -s -o /dev/null -w '%{http_code}\n' --cacert certificates/tier-ca.pem \ + -d test https://ntfy.matrix.elysium/anything + ``` + +5. **The phones**, on Wi-Fi. The stack README's ntfy section has the setup. + Both phones must be subscribed to all three in-house topics and keep their + two ntfy.sh subscriptions. +6. **Secrets on `prometheus`.** Run `make secrets-edit`: + - Move the current `urgent` and `security` ntfy.sh URLs into + `ALERTMANAGER_URGENT_EXTERNAL_URL` and `ALERTMANAGER_SECURITY_EXTERNAL_URL`. + - Point the three `ALERTMANAGER_*_WEBHOOK_URL` keys at the in-house topics. + - Set `ALERTMANAGER_NTFY_TOKEN` to the token from step 1. + + Then run `make up`. Rendering refuses to start without + `certificates/tier-ca.pem`, which `make tier-ca ARGS=--mint` left on this host. +7. **The probe.** Verify from the running exporter, then uncomment the ntfy + target in `prometheus/targets/blackbox.yaml`. The command is in the comment + above it. +8. **Observe delivery, on both paths.** Fire a synthetic page and resolve it: + + ```bash + amtool alert add --alertmanager.url=http://localhost:9093 \ + alertname=AlertPathCutover severity=critical category=availability \ + --annotation=summary='#136 cutover: in-house and ntfy.sh' + # on Wi-Fi: it arrives on the in-house urgent topic, on both phones, + # readable rather than JSON + # on mobile data, Wi-Fi off: it arrives on the ntfy.sh urgent topic + amtool alert add --alertmanager.url=http://localhost:9093 \ + alertname=AlertPathCutover severity=critical category=availability \ + --end="$(date -u +%Y-%m-%dT%H:%M:%SZ)" + ``` + + Then lower the second Watchdog route as in *Confirming it actually works*, + below. It must arrive on the in-house alerts topic, and nowhere else. The + same check with `category=security` proves the security pair. +9. **Kill it, and be paged anyway.** Run `docker stop sensitive-ntfy` on + `trinity`. Within about ten minutes `EndpointUnreachable` for `ntfy` must + arrive on the ntfy.sh urgent topic. Then run `docker start sensitive-ntfy`. + +A 200 is not evidence. Only a notification seen on a phone is, and step 8 is +seen twice, on two networks. Write the times down here, the way the table +below was written, before the issue is closed. + ## Setting up the external watcher > **Done 2026-09-09.** One check on healthchecks.io, period 5m, grace 15m, @@ -164,7 +263,8 @@ Put it back to `24h` afterwards. | What you see | What it means | | --- | --- | | External check DOWN, daily heartbeat still arriving | The heartbeat URL is wrong or that specific destination is unreachable. The alert path itself is fine. | -| External check UP, daily heartbeat stopped | The **real alert channel** is broken — a deleted topic, a rotated URL. This is #67's original failure, and every real alert is being lost right now. | +| External check UP, daily heartbeat stopped | The **real alert channel** is broken — since #136 the in-house ntfy: a token that no longer matches, a topic renamed on one host and not the other, the phone's subscription or password. This is #67's original failure. Pages still reach you over the ntfy.sh copies; nothing else does. | +| `EndpointUnreachable` for `ntfy` on the ntfy.sh topic | The in-house ntfy, Caddy, or `trinity` itself. Warnings sent meanwhile reached no one — `default` has no second route — so read what fired in Alertmanager, through Grafana, once it is back. | | Both stopped | Prometheus, Alertmanager, or this host. Start with `docker compose ps` and `curl -s localhost:9093/-/healthy`. | | Both fine, but you expected an alert about something else | Not this runbook. The path works; check the rule, then the routing tree with `amtool config routes test`. | diff --git a/docs/security.md b/docs/security.md index d02647f1..d55ae7e0 100644 --- a/docs/security.md +++ b/docs/security.md @@ -443,6 +443,22 @@ assumption consistent with what they are. - CI runs `gitleaks` with rules specifically for SNMP communities, inline Grafana passwords, PEM private keys and age secret keys, and separately asserts that every `secrets/*.sops.yaml` is genuinely encrypted. +- **The alert path's credentials are in SOPS on two hosts, and one of them + crosses the CA boundary on purpose.** Since + [#136](https://github.com/Gerrrt/HomeLab/issues/136) Alertmanager delivers to + the tier's own ntfy. ntfy is deny-all, and the users, access list, token and + topic names are all `NTFY_*` keys in `secrets/sensitive.sops.yaml`. + Alertmanager's copy of the token is `ALERTMANAGER_NTFY_TOKEN` in the + observability file. It is sent as a header and never as ntfy's `?auth=` + parameter, because Caddy's access log records the URL and Alloy ships that + log to Loki. To verify the leaf, the estate's Alertmanager trusts the + **tier's** root, which + [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md) + otherwise keeps apart from the estate's. The trust is scoped: a `ca_file` + on those three webhook configs and on one blackbox module, not the + container's trust store. Nothing else on `prometheus` is asked to believe + that root. The `urgent` and `security` channels also keep their ntfy.sh + topics, where the name alone is the credential, as before. - **One service on the sensitive tier keeps its credentials outside SOPS, by necessity and on the record.** Home Assistant obtains device credentials through its own pairing flows — the Hue application key, the Ring token — diff --git a/scripts/backup-volumes.sh b/scripts/backup-volumes.sh index 489bb744..020b50e2 100755 --- a/scripts/backup-volumes.sh +++ b/scripts/backup-volumes.sh @@ -449,9 +449,17 @@ declare -A COMPANIONS=( # Its settings are adguard/AdGuardHome.yaml, in git. Not archiving it is # what takes AdGuard off the stop list: SERVICES is derived from the volumes # that ARE archived. +# ntfy-data (#136) holds two SQLite files and neither is a record. user.db is +# re-provisioned from NTFY_AUTH_* on every start — measured on the pinned +# image: a rotated token and a changed hash each replaced the stored one after +# a restart — and cache.db is at most twelve hours of notifications already +# delivered. ADR-0023 classes ntfy as Unclassed for the same reason. Skipping +# it also keeps ntfy running through a backup, which is when a failed backup +# would want to page. declare -A DISPOSABLE=( [immich-model-cache]="a model cache immich-machine-learning re-downloads on first use" [adguard-work]="blocklists, stats and a query log AdGuard rebuilds; archiving it would stop the house's only DNS forwarder" + [ntfy-data]="users re-provisioned from SOPS on every start, and twelve hours of notifications already delivered" [jellyfin-cache]="transcode scratch and image caches Jellyfin regenerates on demand" ) diff --git a/scripts/check_alert_channels.py b/scripts/check_alert_channels.py index 0d154a70..bb4a8a97 100755 --- a/scripts/check_alert_channels.py +++ b/scripts/check_alert_channels.py @@ -71,7 +71,9 @@ # Anchored on `url_file:` rather than on the path fragment, for the reason # render-config.sh records: a bare `secrets/[a-z_]+` also matches # `secrets/observability.sops.yaml` in a comment, and reports prose as a channel. -URL_FILE = re.compile(r"url_file:\s*(/etc/alertmanager/secrets/([a-z_]+))") +# `credentials_file` too since #136: the in-house ntfy's bearer token is read +# from the same directory and missing it fails the same way — at notify time. +URL_FILE = re.compile(r"(?:url|credentials)_file:\s*(/etc/alertmanager/secrets/([a-z_]+))") # The pairs render-config.sh writes, read out of that script rather than # duplicated here. Two copies of this list is exactly the drift the check is for. @@ -124,6 +126,12 @@ def failures() -> list[str]: "api.telegram.org", } UPTIME_KUMA_PATH = "/api/push/" +# Anything in the house's own domain — since #136 that includes the ntfy the +# real alerts go to. Worse than a push service, not the same: a watcher inside +# the house fails with the house, so the heartbeat would stop being delivered +# at the moment it had something to say. That one IS fixable from here, so it +# is a failure rather than a warning. +IN_HOUSE_SUFFIX = ".matrix.elysium" def warn(msg: str) -> None: @@ -182,6 +190,15 @@ def check_heartbeat_destination(container: str, name: str) -> None: bad(f"{name} is not a URL this can parse — the heartbeat may not deliver at all") return + if host == IN_HOUSE_SUFFIX[1:] or host.endswith(IN_HOUSE_SUFFIX): + bad( + f"{name} points at {host}, which is INSIDE the house. The dead man's " + f"switch has to live off this estate or it dies with it — point it " + f"back at the external watcher (docs/runbooks/verify-the-alert-path.md, " + f"#408)" + ) + return + if host in WATCHERS or UPTIME_KUMA_PATH in parsed.path: ok(f"{name} points at {host}, which watches for absence") return @@ -252,7 +269,7 @@ def main() -> int: for name in sorted(rendered - wanted): bad(f"AM_CHANNELS renders {name}, which no receiver reads") if wanted == rendered: - ok(f"{len(wanted)} url_file(s) declared and rendered: {', '.join(sorted(wanted))}") + ok(f"{len(wanted)} secret file(s) declared and rendered: {', '.join(sorted(wanted))}") out_dir = REPO / "stacks" / args.stack / "alertmanager" / ".rendered" @@ -327,7 +344,7 @@ def main() -> int: # thing this repository keeps writing checks to avoid. checked = ["config"] + (["files"] if checked_files else []) + (["container"] if checked_live else []) print( - f"\nalert channels OK — {len(wanted)} receiver URL(s), " + f"\nalert channels OK — {len(wanted)} secret file(s), " f"checked against: {', '.join(checked)}" ) # A warning that only appeared in scrollback would be the same silence this diff --git a/scripts/render-config.sh b/scripts/render-config.sh index 1caa6b90..143f36a7 100755 --- a/scripts/render-config.sh +++ b/scripts/render-config.sh @@ -115,7 +115,12 @@ the stack cannot start without them. Generate them with: make certs ARGS=--ca make certs ARGS=\"--host grafana.matrix.elysium --ip 10.0.99.20 --dns grafana\" -Full procedure in docs/runbooks/generate-certificates.md." ;; +Full procedure in docs/runbooks/generate-certificates.md. + +tier-ca.pem is different: it is the sensitive tier's root, which Alertmanager +and the blackbox exporter verify ntfy.matrix.elysium against (#136). It is +written on this host by \`make tier-ca ARGS=--mint\`, the step that minted it — +never by \`make certs\` (docs/runbooks/build-the-tier-ca.md)." ;; *) die "compose.yaml mounts these certificates, which are missing or empty: $(printf ' %s\n' "${absent[@]}") @@ -279,6 +284,9 @@ if [[ -f "${AM_CONFIG}" ]]; then ALERTMANAGER_URGENT_WEBHOOK_URL ALERTMANAGER_SECURITY_WEBHOOK_URL ALERTMANAGER_HEARTBEAT_URL + ALERTMANAGER_URGENT_EXTERNAL_URL + ALERTMANAGER_SECURITY_EXTERNAL_URL + ALERTMANAGER_NTFY_TOKEN ) fi @@ -364,12 +372,22 @@ fi # default channel's variable predates the other two and is not # ALERTMANAGER_DEFAULT_WEBHOOK_URL, and renaming a key in an encrypted file to # suit a loop is a worse trade than writing the pair out. +# +# Since #136 the list carries two more URLs and one thing that is not a URL. +# The *_external_url pair is the ntfy.sh copy of the urgent and security +# channels, for a phone off the home network; ntfy_token is the bearer token +# the in-house receivers present, read through `credentials_file` rather than +# `url_file`. It is rendered by the same loop because it has the same needs — +# a 0600 file, no trailing newline — and the cross-check below reads both keys. # --------------------------------------------------------------------------- AM_CHANNELS=( "ALERTMANAGER_WEBHOOK_URL:webhook_url" "ALERTMANAGER_URGENT_WEBHOOK_URL:urgent_url" "ALERTMANAGER_SECURITY_WEBHOOK_URL:security_url" "ALERTMANAGER_HEARTBEAT_URL:heartbeat_url" + "ALERTMANAGER_URGENT_EXTERNAL_URL:urgent_external_url" + "ALERTMANAGER_SECURITY_EXTERNAL_URL:security_external_url" + "ALERTMANAGER_NTFY_TOKEN:ntfy_token" ) AM_OUT_DIR="${STACK_DIR}/alertmanager/.rendered" if [[ -f "${AM_CONFIG}" ]]; then @@ -386,7 +404,8 @@ if [[ -f "${AM_CONFIG}" ]]; then done unset channel var file - # Every url_file alertmanager.yaml names must be one this loop just wrote. A + # Every url_file (and credentials_file) alertmanager.yaml names must be one + # this loop just wrote. A # url_file that does not exist is not a config error — Alertmanager reads it # at notify time, so the stack starts, amtool check-config passes, and the # first real alert is the thing that discovers the missing file. Adding a @@ -398,7 +417,7 @@ if [[ -f "${AM_CONFIG}" ]]; then # Anchored on `url_file:` rather than on the path fragment. A bare # `secrets/[a-z_]+` also matched `secrets/observability.sops.yaml` in this # file's own header comment, and reported the header as a missing channel. - done < <(grep -oE 'url_file:[[:space:]]*/etc/alertmanager/secrets/[a-z_]+' \ + done < <(grep -oE '(url|credentials)_file:[[:space:]]*/etc/alertmanager/secrets/[a-z_]+' \ "${AM_CONFIG}" \ | sed 's|.*/||' | sort -u) fi @@ -472,6 +491,12 @@ COMPOSE_VARS=( VAULTWARDEN_ADMIN_TOKEN HOMEPAGE_IMMICH_API_KEY HOMEPAGE_PAPERLESS_TOKEN + NTFY_ALERTMANAGER_PASSWORD_HASH + NTFY_PHONE_PASSWORD_HASH + NTFY_ALERTMANAGER_TOKEN + NTFY_TOPIC_ALERTS + NTFY_TOPIC_URGENT + NTFY_TOPIC_SECURITY INDEXER_PASSWORD DASHBOARD_PASSWORD API_PASSWORD diff --git a/scripts/seed-validation-env.sh b/scripts/seed-validation-env.sh index 4b50c255..fd4ab9dc 100755 --- a/scripts/seed-validation-env.sh +++ b/scripts/seed-validation-env.sh @@ -52,6 +52,19 @@ STACK="${REPO_ROOT}/stacks/${2:-observability}" echo "VAULTWARDEN_ADMIN_TOKEN=validation-only" echo "HOMEPAGE_IMMICH_API_KEY=validation-only" echo "HOMEPAGE_PAPERLESS_TOKEN=validation-only" + # ntfy validates these at start and exits on a malformed one, so a boot of + # it (check_hardened_boot.sh) needs them well-formed: a real bcrypt hash — + # of the string "validation-only", from the pinned image's `ntfy user + # hash` — with every `$` doubled for compose, as render-config.sh writes + # them, a token of the `tk_` + 29 shape, and three distinct topics. + # shellcheck disable=SC2016 # the $$ are for compose, not the shell + echo 'NTFY_ALERTMANAGER_PASSWORD_HASH=$$2a$$10$$3egG3fvpHgBzCUcV7m7xE.1aiIfVdTQxRjW0kYzefkJ8YPnyiaYWq' + # shellcheck disable=SC2016 + echo 'NTFY_PHONE_PASSWORD_HASH=$$2a$$10$$3egG3fvpHgBzCUcV7m7xE.1aiIfVdTQxRjW0kYzefkJ8YPnyiaYWq' + echo "NTFY_ALERTMANAGER_TOKEN=tk_validationonlyvalidationonly0" + echo "NTFY_TOPIC_ALERTS=validation-alerts" + echo "NTFY_TOPIC_URGENT=validation-urgent" + echo "NTFY_TOPIC_SECURITY=validation-security" echo "INDEXER_PASSWORD=validation-only" echo "DASHBOARD_PASSWORD=validation-only" echo "API_PASSWORD=validation-only" diff --git a/scripts/validate.sh b/scripts/validate.sh index 7869150e..69c1469b 100755 --- a/scripts/validate.sh +++ b/scripts/validate.sh @@ -257,6 +257,11 @@ elif ((${#AMTOOL[@]})); then # `continue: true` — the one place in the tree that does. That row is also # what catches a `severity` slip on the watchdog rule: with `info` it would # resolve to "null" and the dead man's switch would be silently disarmed. + # + # The `correctness` row is AlertmanagerNotificationsFailing, and the + # `availability` row is EndpointUnreachable — which covers the probe of the + # in-house ntfy. Since #136 both must reach `urgent`, the one receiver with + # an ntfy.sh copy, or a dead in-house ntfy is reported only to itself. routes_ok=1 while read -r expected labels; do [[ -n "${expected}" ]] || continue @@ -275,6 +280,7 @@ urgent severity=critical category=power security severity=critical category=security security severity=warning category=security urgent severity=critical category=availability +urgent severity=critical category=correctness default severity=warning category=capacity default severity=warning category=hardware default severity=warning category=monitoring @@ -282,7 +288,7 @@ null severity=info category=correctness ROUTES if ((routes_ok)); then - pass "${stack}: amtool config routes test (9 assertions)" + pass "${stack}: amtool config routes test (10 assertions)" else fail "${stack}: amtool config routes test" fi diff --git a/secrets/observability.example.yaml b/secrets/observability.example.yaml index e41d2559..eb787785 100644 --- a/secrets/observability.example.yaml +++ b/secrets/observability.example.yaml @@ -55,13 +55,36 @@ SNMP_PRIVPASS_ILO: change-me-ilo-priv # only a separate destination can make that difference audible, because that is # where the per-topic sound and do-not-disturb settings live. # -# With ntfy, three topics on one server is the whole setup — the topic is the -# last path segment, and a topic needs no creation step. Point them all at the -# same topic if you would rather have one channel; nothing breaks, but then the -# receiver names are lying again, so say so in docs/observability.md. -ALERTMANAGER_WEBHOOK_URL: https://ntfy.example.invalid/homelab-alerts -ALERTMANAGER_URGENT_WEBHOOK_URL: https://ntfy.example.invalid/homelab-urgent -ALERTMANAGER_SECURITY_WEBHOOK_URL: https://ntfy.example.invalid/homelab-security +# Since #136 these three are the sensitive tier's own ntfy, one topic each. +# The topic names are NTFY_TOPIC_ALERTS / _URGENT / _SECURITY in +# secrets/sensitive.sops.yaml, where ntfy grants access to them; copy each +# name here, after the host, and end every URL with ?template=alertmanager so +# the phone shows a title and a summary instead of JSON. Nothing about the +# token goes in the URL — it is ALERTMANAGER_NTFY_TOKEN below, sent as a +# header, because Caddy's access log records the URL. +ALERTMANAGER_WEBHOOK_URL: https://ntfy.matrix.elysium/change-me-alerts-topic?template=alertmanager +ALERTMANAGER_URGENT_WEBHOOK_URL: https://ntfy.matrix.elysium/change-me-urgent-topic?template=alertmanager +ALERTMANAGER_SECURITY_WEBHOOK_URL: https://ntfy.matrix.elysium/change-me-security-topic?template=alertmanager + +# The same urgent and security channels, AGAIN, on ntfy.sh — for a phone that +# is not on the home network and so cannot reach the in-house server (#136). +# On ntfy.sh the topic name is the credential (SECURITY.md), so these are two +# long random names, and they are the topics these two channels used before +# the in-house server existed: moving the old values here is the whole +# migration. No `?template=` — the ntfy.sh copies stay as they were. +ALERTMANAGER_URGENT_EXTERNAL_URL: https://ntfy.sh/change-me-to-a-long-random-topic +ALERTMANAGER_SECURITY_EXTERNAL_URL: https://ntfy.sh/change-me-to-another-long-random-topic + +# The bearer token the three in-house receivers present, and the SAME value as +# NTFY_ALERTMANAGER_TOKEN in secrets/sensitive.sops.yaml — two copies because +# the two hosts' age keys open two different files, on purpose. `tk_` and 29 +# lower-case letters and digits; make it on any host with docker: +# +# docker run --rm "$(COMPOSE_FILE=stacks/sensitive/compose.yaml \ +# ./scripts/image-for.sh ntfy)" token generate +# +# It can only publish to the three topics, and cannot read them. +ALERTMANAGER_NTFY_TOKEN: tk_changemechangemechangeme00 # The dead man's switch (#67). NOT an ntfy topic — this is a cron-monitor ping # URL from something that lives OFF this host: healthchecks.io, Cronitor, an diff --git a/secrets/sensitive.example.yaml b/secrets/sensitive.example.yaml index ad88e109..30b55247 100644 --- a/secrets/sensitive.example.yaml +++ b/secrets/sensitive.example.yaml @@ -1,15 +1,15 @@ --- # Template for secrets/sensitive.sops.yaml. # -# NINE KEYS. Read the list as a description of the stack: Caddy needs no +# FIFTEEN KEYS. Read the list as a description of the stack: Caddy needs no # credential — its certificate and key are files under certificates/, never # tracked, handled the way the estate's Grafana leaf is — step-ca needs # exactly one, the password that unlocks its intermediate key, AdGuard Home # needs one, the hash of its admin password (#135), Immich needs one, the # password its server presents to its own Postgres (#132), Vaultwarden needs # one, the hash of its admin token (#131), Paperless-ngx needs three (#133), -# and Homepage needs two, one read-only token per live widget (at the end of -# this file, #137). Home Assistant needs NONE +# Homepage needs two, one read-only token per live widget (#137), and ntfy +# needs six (at the end of this file, #136). Home Assistant needs NONE # from here, which is not the same as needing none. Its device credentials — # the Hue application key, the Ring token, everything a config flow produces # — are written by Home Assistant into its own store in the @@ -131,3 +131,40 @@ PAPERLESS_ADMIN_PASSWORD: change-me-to-something-long-and-random # dot this page refuses in another form. HOMEPAGE_IMMICH_API_KEY: change-me-to-an-immich-api-key-scoped-to-server-statistics HOMEPAGE_PAPERLESS_TOKEN: change-me-to-the-homepage-users-token + +# ntfy (#136). compose.yaml builds its users, access list and token from these +# on every start (DIFFERENCE 12), so changing one here and `make up` IS the +# rotation — there is no ntfy CLI step. +# +# NTFY_ALERTMANAGER_TOKEN the bearer token Alertmanager publishes +# with. The SAME value as +# ALERTMANAGER_NTFY_TOKEN in +# secrets/observability.sops.yaml; rotate +# both, trinity first. Made with the +# pinned image's `ntfy token generate` — +# the command is beside that key. +# NTFY_ALERTMANAGER_PASSWORD_HASH a bcrypt hash of a password nobody keeps. +# ntfy wants one per user, and this user +# authenticates by token alone: +# `make gen-secret`, then `make +# hash-password` on it, then forget it. +# NTFY_PHONE_PASSWORD_HASH the bcrypt hash of the password the ntfy +# app on each phone logs in with, as user +# `phone` (`make hash-password`). The +# password goes in the password manager. +# NTFY_TOPIC_ALERTS / _URGENT / _SECURITY +# three topic names, each `make gen-secret` +# output. The same three names end the URLs +# in secrets/observability.sops.yaml. With +# deny-all a name grants nothing by itself, +# but #136 asked that they be kept like a +# credential, and they are. +# +# Both hashes carry `$`; paste them as they are — render-config.sh escapes them +# for compose's .env, as it does AdGuard's. +NTFY_ALERTMANAGER_TOKEN: tk_changemechangemechangeme00 +NTFY_ALERTMANAGER_PASSWORD_HASH: change-me-to-the-output-of-make-hash-password +NTFY_PHONE_PASSWORD_HASH: change-me-to-the-output-of-make-hash-password +NTFY_TOPIC_ALERTS: change-me-to-something-long-and-random +NTFY_TOPIC_URGENT: change-me-to-something-long-and-random +NTFY_TOPIC_SECURITY: change-me-to-something-long-and-random diff --git a/stacks/observability/alertmanager/alertmanager.yaml b/stacks/observability/alertmanager/alertmanager.yaml index 7e56b0d0..ce6f0522 100644 --- a/stacks/observability/alertmanager/alertmanager.yaml +++ b/stacks/observability/alertmanager/alertmanager.yaml @@ -11,6 +11,9 @@ # at /etc/alertmanager/secrets/. Both the rendered directory and its contents # are gitignored. # +# Since #136 the three real channels go to the sensitive tier's own ntfy, and +# urgent and security go to ntfy.sh as well; the receivers section says why. +# # There are four channels and four destinations. They used to be three names # for one URL, which meant `urgent` and `default` differed only in how often # they repeated — a UPS on battery and a slow scrape landed in the same place @@ -72,6 +75,13 @@ route: # a deleted topic shows up as a missing daily heartbeat instead of as # nothing at all. The heartbeat route above cannot see that, because it # proves delivery to a different URL. + # + # Since #136 that URL is the in-house ntfy, so this proves the in-house + # path daily and says nothing about the two ntfy.sh copies urgent and + # security also send. Those are proved by the cutover drill in + # docs/runbooks/verify-the-alert-path.md and by every real page after it; + # a daily Watchdog on the loud channels would teach the phone to ignore + # them, which is worse than the gap. - receiver: default matchers: - alertname = "Watchdog" @@ -154,20 +164,52 @@ inhibit_rules: receivers: - name: "null" + # Since #136 the three real channels land on the sensitive tier's own ntfy, + # https://ntfy.matrix.elysium, over Caddy's 443 on this VLAN. Three settings + # come with that, identical in each in-house entry below, and the anchor + # keeps them identical: + # + # - The token rides in an Authorization header from a rendered file. Not + # ntfy's `?auth=` parameter, which would put it in the URL and Caddy's + # access log records the URL. + # - The certificate is verified against the TIER'S root, mounted from + # certificates/tier-ca.pem. Per receiver, not process-wide: nothing else + # Alertmanager talks to is asked to trust it (ADR-0037, docs/security.md). + # - Each URL ends `?template=alertmanager`, ntfy's built-in rendering of + # this payload, so a phone shows a title and a summary instead of JSON. + # + # urgent and security carry a SECOND webhook each, to their ntfy.sh topic, + # for the phone that is not on the home network and so cannot reach the + # first. Two entries in one receiver are independent deliveries: either can + # fail without stopping the other, and each is counted separately in + # alertmanager_notifications_failed_total. `default` has no external twin + # on purpose — a slow scrape can wait until the phone is home. - name: default webhook_configs: - url_file: /etc/alertmanager/secrets/webhook_url send_resolved: true + http_config: &in-house-ntfy + authorization: + type: Bearer + credentials_file: /etc/alertmanager/secrets/ntfy_token + tls_config: + ca_file: /etc/alertmanager/tls/tier-ca.pem - name: urgent webhook_configs: - url_file: /etc/alertmanager/secrets/urgent_url send_resolved: true + http_config: *in-house-ntfy + - url_file: /etc/alertmanager/secrets/urgent_external_url + send_resolved: true - name: security webhook_configs: - url_file: /etc/alertmanager/secrets/security_url send_resolved: true + http_config: *in-house-ntfy + - url_file: /etc/alertmanager/secrets/security_external_url + send_resolved: true # send_resolved: false, deliberately. If Prometheus stops sending Watchdog, # Alertmanager expires it after resolve_timeout and the pings simply stop — diff --git a/stacks/observability/blackbox/blackbox.yaml b/stacks/observability/blackbox/blackbox.yaml index c8e4c024..4e1b2f01 100644 --- a/stacks/observability/blackbox/blackbox.yaml +++ b/stacks/observability/blackbox/blackbox.yaml @@ -12,13 +12,14 @@ # serves, not by what would be nice: # # http_2xx_lab_ca https, certificate issued by certificates/ca.pem +# http_2xx_tier_ca https, certificate issued by the sensitive tier's step-ca # http_2xx_self_signed https, certificate the device made for itself # http_2xx_plain http, no TLS on offer # tcp_connect anything at all holding the port open # dns_answers a resolver, asked whether it still answers at all # dns_filters a resolver, asked whether it is still filtering # -# The two `dns` modules are a different shape from the four above them and the +# The two `dns` modules are a different shape from the five above them and the # block at the foot of this file says why: an http module is picked by what the # endpoint serves, a dns module carries its own question. # @@ -26,7 +27,7 @@ # verifies the chain — blackbox reads it off the handshake, before deciding # whether to trust it — and blackbox.rules.yaml turns that into the 30-day and # 7-day expiry alerts. Verification is a separate question from expiry, and the -# two https modules answer it differently on purpose. +# three https modules answer it differently on purpose. modules: # An HTTPS endpoint whose certificate the lab's own CA issued — Grafana, and @@ -54,6 +55,27 @@ modules: tls_config: ca_file: /etc/blackbox_exporter/tls/ca.pem + # An HTTPS endpoint on trinity, behind Caddy, whose certificate the sensitive + # tier's own step-ca issued (ADR-0037) — a different root from ca.pem, so a + # module of its own rather than a second ca_file on the one above, which + # would let either root vouch for either tier. The in-house ntfy is the + # target, and the reason: it is where Alertmanager delivers since #136, so + # its absence has to page through something other than itself — see the + # target's comment. Seven-day leaves renewed by Caddy mean the 30-day expiry + # warning should never fire here; if it does, renewal has stopped. + http_2xx_tier_ca: + prober: http + timeout: 10s + http: + valid_status_codes: [] # any 2xx + valid_http_versions: ["HTTP/1.1", "HTTP/2.0"] + method: GET + follow_redirects: true + preferred_ip_protocol: ip4 + ip_protocol_fallback: false + tls_config: + ca_file: /etc/blackbox_exporter/tls/tier-ca.pem + # An HTTPS endpoint whose certificate nothing issued — the management # interfaces on the APC card, the iLO and pfSense each present a certificate # they generated for themselves. insecure_skip_verify is not a shortcut @@ -110,7 +132,7 @@ modules: # --- Asking a resolver what it answered ------------------------------------ # - # The four modules above ask whether an endpoint can be reached. These two ask + # The five modules above ask whether an endpoint can be reached. These two ask # the question only a DNS server can be asked — *what did you say?* — and they # exist because ADR-0010 deliberately made one failure silent. # diff --git a/stacks/observability/compose.yaml b/stacks/observability/compose.yaml index 5a7ae66f..1a97f001 100644 --- a/stacks/observability/compose.yaml +++ b/stacks/observability/compose.yaml @@ -360,6 +360,11 @@ services: # Alertmanager does not expand environment variables, so each receiver # URL is read from a file via `url_file`. - ./alertmanager/.rendered:/etc/alertmanager/secrets:ro + # The SENSITIVE TIER'S root, not the estate's ca.pem (ADR-0037): the + # in-house ntfy's leaf is issued beneath it, and alertmanager.yaml names + # it as the ca_file of those receivers alone (#136). Public, like + # ca.pem; written on this host by `make tier-ca ARGS=--mint`. + - ../../certificates/tier-ca.pem:/etc/alertmanager/tls/tier-ca.pem:ro - alertmanager-data:/alertmanager ports: # Loopback only, and the only one of the four that can be (#70). Anyone @@ -539,7 +544,7 @@ services: # formality. # # Nothing in blackbox.yaml needs a capability. There is no `icmp` prober - # defined — the six modules are three http, one tcp_connect and two dns — + # defined — the seven modules are four http, one tcp_connect and two dns — # so CAP_NET_RAW is not wanted, and the exporter binds only 9115, above # 1024. Verified rather than assumed: this pinned digest was run with # --cap-drop ALL --security-opt no-new-privileges:true against the real @@ -566,6 +571,9 @@ services: # chain against it (#91). Changing this recreates the container, which # `make up` does; a config-only edit is a `make reload`. - ../../certificates/ca.pem:/etc/blackbox_exporter/tls/ca.pem:ro + # And the sensitive tier's root, for http_2xx_tier_ca: the probe of the + # in-house ntfy that Alertmanager now delivers to (#136). + - ../../certificates/tier-ca.pem:/etc/blackbox_exporter/tls/tier-ca.pem:ro # Never published to a host interface. Nothing outside the compose network # has any reason to ask this container to make a request on its behalf — # an open blackbox exporter is a server-side request forgery primitive. diff --git a/stacks/observability/prometheus/rules/host.rules.yaml b/stacks/observability/prometheus/rules/host.rules.yaml index c0abc8d6..242a8061 100644 --- a/stacks/observability/prometheus/rules/host.rules.yaml +++ b/stacks/observability/prometheus/rules/host.rules.yaml @@ -1013,7 +1013,8 @@ groups: # HostBatteryTempNotMeasured says, per host, that today is not that day. # It is `info` on purpose — true on both hosts for as long as these packs # are in, and a warning would be two twelve-hour nags forever on the - # channel whose ntfy budget the heartbeat already spends (#407). Until + # default channel. (It used to add "whose ntfy budget the heartbeat + # already spends (#407)"; the heartbeat left ntfy.sh on 2026-09-09.) Until # it resolves, the trackpad inspection in replace-the-laptop-cell.md §3 # is the only hot-cell detection there is. # diff --git a/stacks/observability/prometheus/rules/stack.rules.yaml b/stacks/observability/prometheus/rules/stack.rules.yaml index 96940c33..ffaaef08 100644 --- a/stacks/observability/prometheus/rules/stack.rules.yaml +++ b/stacks/observability/prometheus/rules/stack.rules.yaml @@ -90,6 +90,11 @@ groups: # was reachable and answering, and ntfy.sh was refusing with 429 # because the day's free message budget was spent (#407). "The # webhook is unreachable" is what a reader assumes, and it was wrong. + # + # CRITICAL IS LOAD-BEARING since #136. Three receivers now deliver to + # the in-house ntfy on trinity; if that is what fails, this alert is + # the one that says so, and only `urgent` carries an ntfy.sh copy that + # does not depend on it. validate.sh pins the route. expr: increase(alertmanager_notifications_failed_total[15m]) > 0 for: 5m labels: @@ -105,7 +110,9 @@ groups: (ntfy.sh's daily quota, #407), clientError or serverError means it answered with an error, and contextDeadlineExceeded means it did not answer at all. This alert travels the same path it reports on, - so if it reached you, at least one receiver still works. + so if it reached you, at least one receiver still works — and being + critical it goes to `urgent`, whose ntfy.sh copy reaches a phone + even when the in-house ntfy is the thing failing (#136). - alert: LokiIngestionStalled expr: | diff --git a/stacks/observability/prometheus/targets/blackbox.yaml b/stacks/observability/prometheus/targets/blackbox.yaml index 470ce1fb..15e4081b 100644 --- a/stacks/observability/prometheus/targets/blackbox.yaml +++ b/stacks/observability/prometheus/targets/blackbox.yaml @@ -103,6 +103,39 @@ host: prometheus via: compose +# --- The in-house ntfy, where alerts are delivered (#136) ------------------- +# Alertmanager's default, urgent and security receivers post to this service, +# so its absence must page through something other than itself. The probe +# fires EndpointUnreachable, which is critical and therefore routes to +# `urgent` — the receiver that also has an ntfy.sh copy (validate.sh pins the +# route). Without this target a dead ntfy would surface only as +# AlertmanagerNotificationsFailing, once something else had tried to page. +# +# By name only, with no `address` twin: the leaf comes from step-ca over ACME +# and carries the name and no IP SAN, and Caddy picks the site by SNI, so an +# https probe of 10.0.99.40 fails for a reason that says nothing about ntfy. +# `/v1/health` answers with no credential, deny-all notwithstanding. The +# module verifies the chain against the TIER'S root, which is also the only +# HTTPS check any of trinity's sites has, so its expiry series is the first +# evidence from outside that Caddy is renewing its seven-day leaves. +# +# NOT ENABLED until ntfy is serving on trinity and the name resolves — enabled +# early, it pages `urgent` from the moment it is loaded. The cutover in +# docs/runbooks/verify-the-alert-path.md verifies it from the running +# exporter and then uncomments it: +# +# docker exec blackbox-exporter wget -qO- \ +# 'http://localhost:9115/probe?target=https://ntfy.matrix.elysium/v1/health&module=http_2xx_tier_ca' \ +# | grep -E '^probe_(success|http_status_code|ssl_earliest_cert_expiry)' +# +# - targets: ["https://ntfy.matrix.elysium/v1/health"] +# labels: +# module: http_2xx_tier_ca +# name: ntfy +# role: alerting +# host: trinity +# via: dns + # --- Devices that otherwise only have SNMP --------------------------------- # An SNMP agent answering says nothing about the management UI a person would # reach for during an incident. These are the four devices in targets/snmp.yaml; diff --git a/stacks/observability/prometheus/tests/stack.test.yaml b/stacks/observability/prometheus/tests/stack.test.yaml index b027088c..73fb752a 100644 --- a/stacks/observability/prometheus/tests/stack.test.yaml +++ b/stacks/observability/prometheus/tests/stack.test.yaml @@ -116,7 +116,9 @@ tests: (ntfy.sh's daily quota, #407), clientError or serverError means it answered with an error, and contextDeadlineExceeded means it did not answer at all. This alert travels the same path it reports on, - so if it reached you, at least one receiver still works. + so if it reached you, at least one receiver still works — and being + critical it goes to `urgent`, whose ntfy.sh copy reaches a phone + even when the in-house ntfy is the thing failing (#136). # --- AlertmanagerNotificationsFailing: quiet on a flat counter ------------ # Alertmanager exports this counter for every integration it knows, at zero, diff --git a/stacks/sensitive/Caddyfile b/stacks/sensitive/Caddyfile index 29c4d57c..f0bb823d 100644 --- a/stacks/sensitive/Caddyfile +++ b/stacks/sensitive/Caddyfile @@ -129,6 +129,20 @@ home.matrix.elysium { } } +# ntfy (#136). Alertmanager on prometheus publishes here and the phones +# subscribe here; a subscription is a long-lived HTTP stream or a WebSocket, +# and reverse_proxy carries both unasked. The access log records the request +# URI, which is why Alertmanager's token travels in an Authorization header +# and never as ntfy's `?auth=` parameter. +ntfy.matrix.elysium { + reverse_proxy ntfy:8080 + + log { + output stdout + format json + } +} + # The host's own name: nothing is served under it yet. trinity.matrix.elysium { handle { diff --git a/stacks/sensitive/README.md b/stacks/sensitive/README.md index 3e842872..86177d72 100644 --- a/stacks/sensitive/README.md +++ b/stacks/sensitive/README.md @@ -29,13 +29,15 @@ make up STACK=sensitive | paperless-broker | `valkey/valkey` | *internal* (6379) | Paperless-ngx's task queue and cache — the one volume in this stack whose loss costs nothing | | vaultwarden | `vaultwarden/server` | *internal* (8080) | The household's password manager, at `https://vaultwarden.matrix.elysium` — Bitwarden's own clients and extensions, pointed at that URL ([#131]) | | homepage | `ghcr.io/gethomepage/homepage` | *internal* (3000) | The household's front page at `https://home.matrix.elysium`: what exists on this tier and where it lives. A directory, not a status page — three tiles read live numbers with read-only tokens, the rest are links ([#137]) | +| ntfy | `binwiederhier/ntfy` | *internal* (8080) | Where the estate's alerts arrive: Alertmanager on `prometheus` publishes to `https://ntfy.matrix.elysium` and the phones subscribe there. Deny-all, two declared users ([#136]) | -Thirteen services. Two are plumbing; Home Assistant and Vaultwarden are the first +Fourteen services. Two are plumbing; Home Assistant and Vaultwarden are the first household services and the shape every later one takes; AdGuard is the one the household uses without ever knowing it; four are Immich, the service [ADR-0008] -names as the price of putting the tier on Winterfell at all; and three are -Paperless-ngx, the archive of what the household cannot get back; and Homepage -is the page that tells the household the rest exist. What is +names as the price of putting the tier on Winterfell at all; three are +Paperless-ngx, the archive of what the household cannot get back; Homepage +is the page that tells the household the rest exist; and ntfy is the one the +estate uses, to tell the operator what is wrong with the rest. What is absent is as deliberate as what is here: - **No Prometheus, Loki or Grafana.** The lab has its own because its @@ -44,8 +46,9 @@ absent is as deliberate as what is here: `scripts/deploy-agent.sh`, pushing to `10.0.99.20`, needing no new rule and no new port. The agent is not in this compose file for the same reason it is not in the lab's: it is the estate's, deployed identically everywhere. -- **No other services yet.** ntfy ([#136]) arrives as Home Assistant, Immich, - Paperless-ngx, Vaultwarden and Homepage did: a service with +- **No other services planned.** The next one, whenever it comes, arrives as + Home Assistant, Immich, Paperless-ngx, Vaultwarden, Homepage and ntfy did: a + service with `expose:`, a block in the `Caddyfile`, a name on the leaf and in the resolver, its credential in SOPS where it takes one from outside, and a sentinel for its volume in `scripts/backup-volumes.sh`. A service in this file with `ports:` @@ -67,7 +70,7 @@ absent is as deliberate as what is here: ## Layout ```text -compose.yaml thirteen services, one network, health-gated ordering +compose.yaml fourteen services, one network, health-gated ordering Caddyfile every route the tier serves; validated in CI home-assistant/ configuration.yaml and packages/, mounted read-only over the volume Home Assistant writes its state to @@ -76,6 +79,7 @@ home-assistant/ configuration.yaml and packages/, mounted read-only adguard/AdGuardHome.yaml AdGuard Home's whole configuration, blocklists included homepage/ settings, services and widgets YAML — Homepage's whole configuration, mounted read-only +ntfy/server.yml ntfy's settings; its users and access list come from SOPS consume/ untracked: drop a scan here and Paperless-ngx imports and deletes it. Created by render-config.sh export/ untracked: where document_exporter writes. Likewise @@ -83,7 +87,7 @@ export/ untracked: where document_exporter writes. Likewise Secrets are `secrets/sensitive.sops.yaml`, encrypted to this stack's own rule in `.sops.yaml` — `trinity`'s key opens this file and nothing else of the -estate's (`secrets/sensitive.example.yaml` says why, and lists the nine keys). The one file under +estate's (`secrets/sensitive.example.yaml` says why, and lists every key). The one file under `certificates/` this stack reads is `tier-ca.pem`, the tier's root certificate, written by `make tier-ca ARGS="--install …"` from the bundle minted on the monitoring host; every leaf is obtained from step-ca at run @@ -452,6 +456,63 @@ that is kept: Paperless widgets got a 401 from the real services through Caddy and the tier CA, which is the whole path short of a valid token. +## ntfy + +Where the estate's alerts are delivered since [#136], replacing ntfy.sh for +every channel but two. `compose.yaml` has the service, its measurements, and +DIFFERENCE 12. [`verify-the-alert-path.md`](../../docs/runbooks/verify-the-alert-path.md) +has the routing it serves and the cutover. What has to be true around it is +here. + +- **Deny-all, and two users who can each do one thing.** `alertmanager` + publishes to the three topics by bearer token and cannot read them. `phone` + reads them by password and cannot publish. Nobody else can do either, + anonymous or not, and nobody can sign up. The list lives in SOPS and is + applied on every start, so a rotation is `make secrets-edit STACK=sensitive` + and `make up`. Rotating the token means the monitoring host's copy too, + `ALERTMANAGER_NTFY_TOKEN`. +- **Urgent and security go to ntfy.sh as well.** A phone off the home network + cannot reach this service: nothing on the tier is exposed, and the WireGuard + path goes to the lab. So the two channels that page carry a second webhook to + their old ntfy.sh topics. `default` does not, and a warning raised while you + are out waits in the twelve-hour cache until the phone is back on Wi-Fi. +- **The iPhone is woken through ntfy.sh, and learns nothing else from it.** + iOS delivers only through APNs, which only ntfy.sh can reach. So + `upstream-base-url` makes this server post a content-free poll request there: + the SHA-256 of the topic's URL, and nothing else, as the pinned image's own log + showed. The phone then fetches the message from here. ntfy.sh therefore + learns that a message exists, and when. It never learns what the message + says, and without this the iPhone sees an alert only when the app is opened. +- **The phones need the tier's root, the same as for Immich.** Each phone + needs the root installed and, on iOS, enabled + ([`build-the-tier-ca.md`](../../docs/runbooks/build-the-tier-ca.md) §6). + Then, in the ntfy app on each phone: + 1. Add the three in-house topics, with server + `https://ntfy.matrix.elysium` and user `phone`. The password is in the + password manager; the topic names are in `secrets/sensitive.sops.yaml`. + 2. Keep the two ntfy.sh subscriptions. They are the off-network pager. + 3. On iOS, leave the app's *default server* at `ntfy.sh`. The upstream + wake-ups arrive through it. + 4. Give `urgent` a sound that wakes you and `default` none. That per-topic + difference is why [#66] split the channels. + + Android's app has to honour a user-installed root for the first step to + work. Record here whether the Pixel's did. +- **A dead ntfy is reported through ntfy.sh.** A blackbox probe of + `/v1/health`, verified against the tier's root, raises `EndpointUnreachable`, + and the failed deliveries raise `AlertmanagerNotificationsFailing`. Both are + critical, so both route to `urgent` and its ntfy.sh copy. The probe is the + only HTTPS check any of this tier's sites has from outside. +- **Nothing to back up.** `ntfy-data` holds `user.db`, rebuilt from SOPS on + every start, and `cache.db`, at most twelve hours of notifications that were + already delivered. [ADR-0023] classes ntfy as unclassed for the same reason. + `backup-volumes.sh` skips the volume by name, which also keeps ntfy up + through a backup, the moment a failed backup would want to page. +- **No second factor.** ntfy has passwords and tokens. The only human account + is `phone`, which can read three topics of alert text and nothing else. + [ADR-0022] leaves ntfy out of its table for that reason: it authenticates no + household identity. + ## Backup and restore ```bash @@ -464,7 +525,9 @@ volume it cannot verify, so each of the ten volumes it archives has a sentinel entry there — `db.sqlite3` for Vaultwarden, read off a boot of the pinned image, beside the entries [#133] read off boots of every other — and `restore-volumes.sh` knows the uid each must come back owned by where that -uid is a constant. Two are skipped by name. `adguard-work` is skipped because +uid is a constant. Three are skipped by name. `ntfy-data` is skipped because +ntfy rebuilds its users from SOPS on every start and its cache is notifications +already delivered. `adguard-work` is skipped because archiving it would stop the house's only DNS forwarder, and nothing in it is worth restoring. `immich-model-cache` is skipped because a downloadable cache is not data, and a fresh host whose models have not been @@ -606,6 +669,7 @@ it matters: [#136]: https://github.com/Gerrrt/HomeLab/issues/136 [#137]: https://github.com/Gerrrt/HomeLab/issues/137 [#182]: https://github.com/Gerrrt/HomeLab/issues/182 +[#66]: https://github.com/Gerrrt/HomeLab/issues/66 [ADR-0035]: ../../docs/adr/0035-scope-the-99-to-20-rule-to-the-hue-bridge.md [#404]: https://github.com/Gerrrt/HomeLab/issues/404 [#428]: https://github.com/Gerrrt/HomeLab/issues/428 diff --git a/stacks/sensitive/compose.yaml b/stacks/sensitive/compose.yaml index 8257b66b..1b0dd117 100644 --- a/stacks/sensitive/compose.yaml +++ b/stacks/sensitive/compose.yaml @@ -25,7 +25,7 @@ # as three, and Vaultwarden (#131) the fifth — the one whose volume is the # vault, and the reason scripts/backup-volumes.sh learned this stack. # Homepage (#137) is the sixth, and the household's page of links to the other -# five. ntfy (#136) lands the way Home Assistant did. +# five. ntfy (#136) is the seventh, and where the estate's alerts now arrive. # # --------------------------------------------------------------------------- # On the comments in this file @@ -36,7 +36,7 @@ # NOT repeated here; two copies of an argument drift the same way two copies # of a config do. What IS written out here is every place this stack differs # from that one, because those are the only lines a reader cannot get from -# the sibling file. There are eleven such places, each marked DIFFERENCE. +# the sibling file. There are twelve such places, each marked DIFFERENCE. name: sensitive @@ -118,6 +118,7 @@ services: - paperless.matrix.elysium - vaultwarden.matrix.elysium - home.matrix.elysium + - ntfy.matrix.elysium mem_limit: 128m memswap_limit: 128m cap_drop: [ALL] @@ -579,6 +580,101 @@ services: retries: 3 start_period: 30s + # --------------------------------------------------------------------------- + # ntfy — where the estate's alerts are delivered (#136) + # --------------------------------------------------------------------------- + # Alertmanager on prometheus posts here, over Caddy's 443 on the same VLAN, + # and the phones subscribe here. The urgent and security channels ALSO keep + # their ntfy.sh topic, as a second webhook in alertmanager.yaml, because a + # phone off the home network cannot reach this service: nothing is exposed, + # and the WireGuard path (ADR-0042) goes to the lab, not here. The heartbeat + # is not here and never may be (#408): a watcher inside the house fails with + # the house. docs/runbooks/verify-the-alert-path.md has the whole picture. + # + # DIFFERENCE 12: auth is declared in the environment, and the environment is + # authoritative. `auth-default-access: deny-all` in ntfy/server.yml means an + # open relay is not one setting away, and the users, their ACLs and + # Alertmanager's token arrive as NTFY_AUTH_USERS / _ACCESS / _TOKENS, built + # below from SOPS. ntfy re-provisions its user.db from these on every start + # — measured on the pinned image: a changed hash and a rotated token each + # made the old credential answer 401 after a restart, and an ACL entry + # removed from the list answered 403. So rotation is `make secrets-edit` and + # `make up`, and the volume holds nothing that is not re-derived from here, + # which is why scripts/backup-volumes.sh skips it by name. + # + # Two users, each able to do one thing. `alertmanager` may WRITE the three + # topics and nothing else, by token; `phone` may READ them and nothing else, + # by password — its bcrypt hash is what SOPS holds (`make hash-password`). + # Measured with this exact list: anonymous publish and read 403, the token + # publishing 200 and reading 403, `phone` reading 200 and publishing 403, + # either against a topic outside the list 403, sign-up refused. + # + # The topic names come from SOPS too. With auth on, a name is not a + # credential the way it is on ntfy.sh (docs/security.md, the rotation of + # 2026-08-20), but #136 asked that they be treated as one, and a name + # guessable from this file is one less thing an attacker has to learn. + # + # upstream-base-url is ntfy.sh, for the iPhone. iOS delivers only through + # APNs, which only ntfy.sh can reach, so this server posts a content-free + # poll request there — the SHA-256 of the topic's URL and nothing else, + # read off the pinned image's own log — and the phone then fetches the + # message from here. The alert text never leaves the house; the fact and + # time of a message does. The Pixel keeps its own connection and needs none + # of it. + # + # Root with every capability dropped, as Vaultwarden (DIFFERENCE 2's + # argument): the pinned image has no /var/lib/ntfy at all, so the named + # volume is created root-owned and a non-root ntfy cannot open its own + # database. `docker diff` on the boot above showed nothing written outside + # it. Re-check when the image moves: + # + # docker run --rm --entrypoint ls -ld /var/lib/ntfy + # + # 128m: 14 MiB and ten tasks on that boot. Ten times, to bound a leak. + ntfy: + <<: *service-defaults + image: binwiederhier/ntfy:v2.28.0@sha256:6ef4b819f722fccdc036af611c4774cfdc2de821ab74fdd48bbf4c9d6f8973da + container_name: sensitive-ntfy + command: ["serve"] + mem_limit: 128m + memswap_limit: 128m + cap_drop: [ALL] + security_opt: + - no-new-privileges:true + read_only: true + tmpfs: + - /tmp:size=16m + # Folded scalars, one entry per line: YAML joins them with ", " and ntfy + # trims the space after each comma (checked on the pinned image). The + # `alertmanager` user's hash is of a password nobody keeps — ntfy requires + # one per user, and that user authenticates by token alone. + environment: + NTFY_AUTH_USERS: >- + alertmanager:${NTFY_ALERTMANAGER_PASSWORD_HASH:?set in secrets/sensitive.sops.yaml}:user, + phone:${NTFY_PHONE_PASSWORD_HASH:?set in secrets/sensitive.sops.yaml}:user + NTFY_AUTH_ACCESS: >- + alertmanager:${NTFY_TOPIC_ALERTS:?set in secrets/sensitive.sops.yaml}:wo, + alertmanager:${NTFY_TOPIC_URGENT:?set in secrets/sensitive.sops.yaml}:wo, + alertmanager:${NTFY_TOPIC_SECURITY:?set in secrets/sensitive.sops.yaml}:wo, + phone:${NTFY_TOPIC_ALERTS}:ro, + phone:${NTFY_TOPIC_URGENT}:ro, + phone:${NTFY_TOPIC_SECURITY}:ro + NTFY_AUTH_TOKENS: >- + alertmanager:${NTFY_ALERTMANAGER_TOKEN:?set in secrets/sensitive.sops.yaml}:alertmanager + volumes: + - ./ntfy/server.yml:/etc/ntfy/server.yml:ro + - ntfy-data:/var/lib/ntfy + expose: + - "8080" + healthcheck: + # Answers {"healthy":true} with no credential, deny-all notwithstanding. + # busybox wget; --probe proves it is in the image. + test: ["CMD", "wget", "--spider", "-q", "http://localhost:8080/v1/health"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 10s + # --------------------------------------------------------------------------- # Immich — the service ADR-0008 named as the cost of the decision (#132) # --------------------------------------------------------------------------- @@ -1223,3 +1319,7 @@ volumes: # paperless-media, one of the volumes here whose loss is a loss of function # rather than of a record — docs/runbooks/restore-the-sensitive-tier.md. vaultwarden-data: + # ntfy's user.db and twelve hours of message cache. The first is rebuilt from + # the environment on every start (DIFFERENCE 12) and the second has already + # been delivered, so scripts/backup-volumes.sh skips it by name. + ntfy-data: diff --git a/stacks/sensitive/ntfy/server.yml b/stacks/sensitive/ntfy/server.yml new file mode 100644 index 00000000..58b4f5e5 --- /dev/null +++ b/stacks/sensitive/ntfy/server.yml @@ -0,0 +1,44 @@ +--- +# ntfy's configuration (#136) — every setting, and no credential. +# +# The users, their access and Alertmanager's token are NOT here: they arrive +# as NTFY_AUTH_USERS / _ACCESS / _TOKENS from secrets/sensitive.sops.yaml, and +# compose.yaml (DIFFERENCE 12) says why the environment rather than this file. +# What is here is what the server is, and ntfy fills in what a key omits. + +# The URL the phones and Alertmanager use, and what ntfy hashes to name the +# upstream poll topic below — change it and the iPhone's subscription has to +# be made again. +base-url: "https://ntfy.matrix.elysium" + +# Plain HTTP on the compose network; TLS ends at Caddy, like every service here. +listen-http: ":8080" + +# Caddy is the only thing that can reach 8080, so X-Forwarded-For is the +# client's address on Hicks or Winterfell, and the per-visitor rate limits +# count phones rather than the proxy. +behind-proxy: true + +# Messages survive a restart for the default twelve hours, so a phone that was +# off-network when an alert fired still finds it when it reconnects. +cache-file: "/var/lib/ntfy/cache.db" + +# Deny-all is the whole security posture: anything not granted by an ACL in +# the environment is refused, anonymous or not. The default is read-write for +# everyone, which on a server Alertmanager publishes to is an open relay. +auth-file: "/var/lib/ntfy/user.db" +auth-default-access: "deny-all" + +# Accounts are declared, never created: no sign-up, and login only so the web +# app and the phones can present the `phone` user. +enable-signup: false +enable-login: true + +# iOS instant delivery. ntfy.sh receives a content-free poll request for the +# hashed topic and wakes the iPhone through APNs; the phone then fetches the +# message from this server. compose.yaml says what that does and does not +# disclose. +upstream-base-url: "https://ntfy.sh" + +# No attachment-cache-dir, so attachments are off: nothing that publishes here +# sends a file, and an upload path on this segment is a surface with no user.