From 66c687be1c243603dd87548e5ff24b75ac849bb8 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Tue, 29 Sep 2026 02:13:29 +0000 Subject: [PATCH] fix(observability): seven-day ACME leaves are not expiring certificates (#136) Enabling the ntfy probe (#715) paged `urgent` within ten minutes: "ntfy's certificate has 7 days left", on a leaf Caddy had just issued. Every sensitive-tier leaf is seven days from step-ca (ADR-0037), so it is always inside TlsCertificateExpiryImminent's 7-day threshold and the 30-day one. It would have repeated every 4h indefinitely. The comment in #712 that said the 30-day warning "should never fire here" was wrong. - The ntfy target gains `renewal: acme`. Both existing expiry rules select renewal!="acme", which is a no-op for every target that lacks the label. - New TlsAcmeRenewalStalled, critical, fires when an ACME leaf has under 36h left for 30m. Caddy renews at a third of the lifetime, about 56h, so this means about 20h of failed renewals. Critical so it reaches urgent's ntfy.sh copy: when that leaf lapses the in-house path goes too. - promtool cases: a healthy leaf at five days fires nothing; the same leaf at 30h fires only the new rule. - Rule counts in the docs move from 102 to 103. Refs #136 Co-Authored-By: Claude Opus 5.5 --- README.md | 4 +- docs/observability.md | 8 +-- docs/runbooks/deploy-stack.md | 2 +- docs/security.md | 2 +- stacks/observability/README.md | 2 +- .../prometheus/rules/blackbox.rules.yaml | 41 ++++++++++++- .../prometheus/targets/blackbox.yaml | 6 ++ .../prometheus/tests/blackbox.test.yaml | 57 +++++++++++++++++++ 8 files changed, 111 insertions(+), 11 deletions(-) diff --git a/README.md b/README.md index dde93ee4..6cf8dee0 100644 --- a/README.md +++ b/README.md @@ -69,7 +69,7 @@ documents for different readers. can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed identically everywhere. [How](docs/architecture.md#observability-data-flow) - **Dashboards and alerting as code.** 7 provisioned dashboards, 141 panels, and - 120 alert rules — 102 metric-based in Prometheus, 18 log-based in Loki — sharing + 121 alert rules — 103 metric-based in Prometheus, 18 log-based in Loki — sharing one Alertmanager routing tree. No dashboard exists only in a database. - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials, decrypted at deploy time into gitignored paths, with `git log` showing which @@ -195,7 +195,7 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md). . ├── stacks/observability/ # the deployed stack — one compose file, eight services │ ├── compose.yaml -│ ├── prometheus/ # config, file_sd targets, 102 alert rules +│ ├── prometheus/ # config, file_sd targets, 103 alert rules │ ├── alertmanager/ # routing and inhibition │ ├── loki/ # single-binary config + 18 LogQL rules │ ├── alloy/ # the agent config directory, shipped to every host diff --git a/docs/observability.md b/docs/observability.md index 83f8f979..4a668c0f 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -343,7 +343,7 @@ separates a quiet stream from a stopped one. ## Alerting -120 rules in total: 102 metric-based in `prometheus/rules/`, and 18 log-based in +121 rules in total: 103 metric-based in `prometheus/rules/`, and 18 log-based in `loki/rules/`. ### Log-based (Loki ruler) @@ -450,7 +450,7 @@ argument and for what to do when it exits 1. ### Metric-based (Prometheus) -102 rules across eleven files in `prometheus/rules/`: +103 rules across eleven files in `prometheus/rules/`: | File | Covers | | --- | --- | @@ -460,7 +460,7 @@ argument and for what to do when it exits 1. | `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling | | `stack.rules.yaml` | The stack watching itself: config reloads, rule evaluation, notification delivery, log ingestion, and the two cases `up == 0` structurally cannot see — a remote-writing agent that stops pushing, and a scraped target that stops being a target at all. The second is `ScrapeTargetDisappeared`, added with the first scraped host ([#256](https://github.com/Gerrrt/HomeLab/issues/256)): an emptied or unparseable `targets/node.yaml` makes the series vanish rather than fall to 0, so `InstanceDown` stays silent and `RemoteWriteJobStale` excludes scraped jobs by design. The target for `smaug` was written into `targets/node.yaml` disabled on 2026-09-17 and enabled on 2026-09-19, once the exporter answered from the pool. Split off `containers.rules.yaml` onto `component: stack` in [#81](https://github.com/Gerrrt/HomeLab/issues/81) so a Prometheus that cannot reload its config stops being filed as a container fault. Since [#575](https://github.com/Gerrrt/HomeLab/issues/575) also whether an Alertmanager silence is about to lapse or names no owning issue, read from the per-silence series `scripts/collect_silences.py` writes every fifteen minutes — `alertmanager_silences` is a count per state and cannot say which alert, when, or whose | | `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable | -| `blackbox.rules.yaml` | Whether an endpoint can actually be reached, from outside the service, and how many days its certificate has left — Grafana verified against the lab CA, the APC card's self-signed one read but not trusted, the wiki, Prometheus, Loki, Alertmanager and the switch UI over plain http. The iLO and pfSense UIs are written into `targets/blackbox.yaml` and left disabled: each needs a firewall pass from `10.0.99.20` that is a segmentation decision, not a monitoring one ([#91](https://github.com/Gerrrt/HomeLab/issues/91)) | +| `blackbox.rules.yaml` | Whether an endpoint can actually be reached, from outside the service, and how many days its certificate has left — the sensitive tier's seven-day ACME leaves excepted, which `TlsAcmeRenewalStalled` watches for a stalled renewal instead — Grafana verified against the lab CA, the APC card's self-signed one read but not trusted, the wiki, Prometheus, Loki, Alertmanager and the switch UI over plain http. The iLO and pfSense UIs are written into `targets/blackbox.yaml` and left disabled: each needs a firewall pass from `10.0.99.20` that is a segmentation decision, not a monitoring one ([#91](https://github.com/Gerrrt/HomeLab/issues/91)) | | `dns.rules.yaml` | Whether the house is still filtering DNS, asked directly at AdGuard Home on port 53 rather than through pfSense. Since [ADR-0055](adr/0055-forward-to-adguard-alone.md) AdGuard is the only forwarder, so `AdGuardNotAnswering` is **critical** at five minutes: the house cannot resolve outside names. `AdGuardNotFiltering` stays a warning, because a filter that fails open is a convenience lost, not an outage. The targets in `targets/blackbox-dns.yaml` are live since 2026-09-28, against AdGuard on `trinity` ([#126](https://github.com/Gerrrt/HomeLab/issues/126), [#404](https://github.com/Gerrrt/HomeLab/issues/404)) | | `backup.rules.yaml` | Whether the scheduled maintenance jobs are still being run at all — staleness, failure, never-ran, whether the age-key proof record exists to be held to its deadline, whether the CA key's offline copy has been proved lately ([#496](https://github.com/Gerrrt/HomeLab/issues/496)), and whether the newest backup sets have been carried onto the second recipient's medium within ninety days ([ADR-0048](adr/0048-carry-the-estates-backup-sets-with-the-second-recipient.md)) | | `deploy.rules.yaml` | Whether this host is running what the repository says — an uncommitted edit made on the host, a revision that did not verify, and how far behind `main` the host is. Reads the record `scripts/converge.sh` writes hourly ([#99](https://github.com/Gerrrt/HomeLab/issues/99), [ADR-0021](adr/0021-converge-on-a-timer-instead-of-deploying-over-ssh.md)) | @@ -473,7 +473,7 @@ as loaded and healthy and could not fire for any input ([#63](https://github.com `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which feed a rule synthetic series and assert it fires — paired with a case asserting it stays quiet, because a test that only ever expects silence would have passed -against the broken rule too. Coverage is eighty-two rules of 102 so far — the five +against the broken rule too. Coverage is eighty-three rules of 103 so far — all eight in `blackbox.rules.yaml`, both in `dns.rules.yaml`, `ContainerHighMemory`, `ContainerNearMemoryLimit`, `ContainerRestartLoop`, `ContainerCpuThrottled` and `PrometheusSizeRetentionActive`, `Watchdog`, the three iLO rules from diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index b96ce3d0..8e36c88b 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -118,7 +118,7 @@ Then in the UI: 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets take up to 45 seconds on their first scrape. -2. **Prometheus → Status → Rules.** 102 rules loaded, none in error. The +2. **Prometheus → Status → Rules.** 103 rules loaded, none in error. The page counts 96: the extra two are the recording rules, `homelab_suricata_expected_interface` and `homelab_battery_runtime_seconds`. Everything counted in this repository diff --git a/docs/security.md b/docs/security.md index d55ae7e0..87c6f81d 100644 --- a/docs/security.md +++ b/docs/security.md @@ -19,7 +19,7 @@ What this network is actually built to survive: | An attacker on the lab segment reaching the hypervisor's management plane | **Closed at the host, and watched.** `Saruman`'s Proxmox firewall admits `8006` and `22` from Hicks and `8006` from the deployment host only ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md)). It was found disabled and turned on on 2026-09-20 ([#566](https://github.com/Gerrrt/HomeLab/issues/566)); since [#576](https://github.com/Gerrrt/HomeLab/issues/576) `homelab_pve_firewall_enabled` is read every five minutes and `PveFirewallDisabled` pages after ten, so a `pve-firewall stop` left in place is noticed rather than found. `PveFirewallPolicyAccept` does the same for `policy_in` left at `ACCEPT`, which reads as enabled and admits the whole segment | | A range target with a path out | It has none — `ifrit`'s targets sit on a bridge with no physical port, on `172.30.30.0/24`, which the firewall does not route and on which nothing has a default route at all ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md)) | | Someone with the trusted Wi-Fi key quietly joining | Kea's lease log reaches Loki; `UnknownDeviceOnTrustedSegment` fires the first time a MAC appears on VLAN 50 in seven days ([ADR-0019](adr/0019-read-device-joins-from-the-dhcp-server.md)) | -| Losing visibility of a failure | 120 alert rules, 30 days of metrics and logs | +| Losing visibility of a failure | 121 alert rules, 30 days of metrics and logs | | Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana | | Mains power loss | **The rack, yes; the monitoring path, yes — on two laptop cells that were measured for the first time on 2026-09-12.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the TP-Link carrying `prometheus` and `oracle` has been on UPS power since 2026-09-08 ([#110](https://github.com/Gerrrt/HomeLab/issues/110)); the laptops ride a cut out on their own batteries, which `HostBatteryHealthLow` in `host.rules.yaml` now reads — `prometheus`'s cell was replaced on 2026-09-18 and reads 101 % of design, `oracle`'s is the original at 72 %, with its replacement bought on 2026-09-19 and in transit ([#531](https://github.com/Gerrrt/HomeLab/issues/531)) — and **`prometheus`'s runtime on its cell was measured on 2026-09-19 — about 2.5 hours from full at the stack's load — while `oracle`'s never has been**; since the same day the projection is recorded on every cut and pages under thirty minutes (`HostBatteryRuntimeLow`, [#532](https://github.com/Gerrrt/HomeLab/issues/532)), but neither pack reports a moving cell temperature, so this row is answered for the monitoring host, and for the other only as far as its cell being healthy — see below. **What the UPS cannot answer is what happens when the cut outlasts the pack: as of 2026-09-20 nothing shuts down on its signal, and everything on the PDU — `morpheus`, `Saruman`, `neo` and `smaug`, which is in the media room on a long cord from that PDU — stops uncleanly when the pack empties, about 47 minutes in at 21 % load by the card's own unmeasured estimate.** [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides that the firewall's NUT server halts `Saruman` and `smaug` first and itself last, and [`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) is the build, the forced-shutdown proof and the one mains pull that measures the pack; until those are done the decision is a configuration nobody has tested ([#574](https://github.com/Gerrrt/HomeLab/issues/574)) | | The estate being down while the person who runs it is unavailable | **Documentation, yes; data, not yet.** ADR-0011 puts the emergency tier on paper; [ADR-0023](adr/0023-keep-the-household-recovery-path-outside-the-estate.md) extends the same reasoning to the sensitive tier's data before that tier exists — see below | diff --git a/stacks/observability/README.md b/stacks/observability/README.md index 99aeb389..bcb69d75 100644 --- a/stacks/observability/README.md +++ b/stacks/observability/README.md @@ -32,7 +32,7 @@ prometheus/ prometheus.yaml scrape config; SNMP via file_sd targets/snmp.yaml SNMP targets — hot-reloaded, no restart needed targets/blackbox*.yaml probe targets, http and dns — hot-reloaded, no restart - rules/*.rules.yaml 102 alert rules across host/network/ups/containers/blackbox/dns/backup/ids/deploy + rules/*.rules.yaml 103 alert rules across host/network/ups/containers/blackbox/dns/backup/ids/deploy tests/*.test.yaml promtool unit tests — assert the rules can fire blackbox/blackbox.yaml probe modules — reachability, and what a resolver said alertmanager/ diff --git a/stacks/observability/prometheus/rules/blackbox.rules.yaml b/stacks/observability/prometheus/rules/blackbox.rules.yaml index d073c0a2..05bc0d04 100644 --- a/stacks/observability/prometheus/rules/blackbox.rules.yaml +++ b/stacks/observability/prometheus/rules/blackbox.rules.yaml @@ -97,10 +97,18 @@ groups: # is when a calendar reminder has clearly been missed. A `for` of an hour # on the warning costs nothing — expiry does not flap — and keeps a # single odd probe from paging. + # + # NOT FOR ACME LEAVES. A target labelled `renewal: acme` (the sensitive + # tier's, from its own step-ca) carries a seven-day certificate that is + # ALWAYS inside both thresholds, so both rules exclude it and + # TlsAcmeRenewalStalled below watches it instead. Learned the loud way: + # enabling the ntfy probe paged `urgent` within ten minutes with + # "ntfy's certificate has 7 days left" on a leaf Caddy had just issued + # (#136). - alert: TlsCertificateExpiringSoon expr: >- min by (name, role, host) - (probe_ssl_earliest_cert_expiry{job="blackbox"} - time()) + (probe_ssl_earliest_cert_expiry{job="blackbox", renewal!="acme"} - time()) / 86400 < 30 for: 1h labels: @@ -120,7 +128,7 @@ groups: - alert: TlsCertificateExpiryImminent expr: >- min by (name, role, host) - (probe_ssl_earliest_cert_expiry{job="blackbox"} - time()) + (probe_ssl_earliest_cert_expiry{job="blackbox", renewal!="acme"} - time()) / 86400 < 7 for: 10m labels: @@ -137,6 +145,35 @@ groups: Reissue now — docs/runbooks/generate-certificates.md — and restart whatever serves it. + # The ACME half of expiry. A seven-day leaf (ADR-0037) is renewed by + # Caddy once a third of its life remains, about 56 hours, and retried + # from then on, so a healthy leaf never reads below ~2.3 days. Under 36 + # hours means some twenty hours of failed renewals: step-ca down, the + # challenge failing, Caddy stuck. Critical, not warning, and deliberately + # so. The first leaf this watches is the in-house ntfy's, and when that + # lapses the in-house alert path goes with it, so the page has to reach + # `urgent` and its ntfy.sh copy. Every other name Caddy serves renews + # through the same step-ca, so one stalled leaf here is evidence about + # all of them. + - alert: TlsAcmeRenewalStalled + expr: >- + min by (name, role, host) + (probe_ssl_earliest_cert_expiry{job="blackbox", renewal="acme"} - time()) + / 3600 < 36 + for: 30m + labels: + component: blackbox + severity: critical + category: availability + annotations: + summary: "{{ $labels.name }}'s ACME certificate has {{ $value | printf \"%.0f\" }}h left and has not renewed" + description: >- + Caddy on {{ $labels.host }} should have renewed this leaf about a day + ago. Read Caddy's log for the renewal error first + (`docker logs sensitive-caddy 2>&1 | grep -i renew`), then check + step-ca is healthy — docs/runbooks/build-the-tier-ca.md. When it + lapses, every name on the tier stops verifying. + - alert: GatewayLatencyHigh # "Is it the house, or beyond the house?" — the first half of the # question #166 could not answer. On 2026-08-29 a workstation reported diff --git a/stacks/observability/prometheus/targets/blackbox.yaml b/stacks/observability/prometheus/targets/blackbox.yaml index 8818dd36..35b7aca7 100644 --- a/stacks/observability/prometheus/targets/blackbox.yaml +++ b/stacks/observability/prometheus/targets/blackbox.yaml @@ -6,6 +6,9 @@ # serves, see the table at the top of that file. # `name` is what the alert says out loud, so write it for whoever is woken up. # `host` is the machine the endpoint is expected on, as network.md names it. +# `renewal` is `acme` for a certificate something renews on its own on a short +# lifetime — today, the sensitive tier's — and absent otherwise. It +# selects which expiry rules apply (blackbox.rules.yaml). # `via` is how the endpoint was addressed: `dns` by the name a person types, # `address` by its IP, `compose` by its service name on the compose # network. A `dns` probe should have an `address` twin, because both @@ -136,6 +139,9 @@ role: alerting host: trinity via: dns + # A seven-day ACME leaf: the two 30/7-day expiry rules skip it and + # TlsAcmeRenewalStalled watches it instead (blackbox.rules.yaml). + renewal: acme # --- Devices that otherwise only have SNMP --------------------------------- # An SNMP agent answering says nothing about the management UI a person would diff --git a/stacks/observability/prometheus/tests/blackbox.test.yaml b/stacks/observability/prometheus/tests/blackbox.test.yaml index 9c57dc3e..f081eae9 100644 --- a/stacks/observability/prometheus/tests/blackbox.test.yaml +++ b/stacks/observability/prometheus/tests/blackbox.test.yaml @@ -506,3 +506,60 @@ tests: - eval_time: 40m alertname: InternetLatencyHigh exp_alerts: [] + + # --- ACME leaves: out of the 30/7-day rules, into their own --------------- + # The page this exists for: a seven-day leaf, freshly issued, read as "7 + # days left" and paged `urgent` the day the ntfy probe was enabled (#136). + # A healthy one at five days must fire neither estate expiry rule; the same + # leaf at 30 hours, a day after Caddy should have renewed it, must fire + # TlsAcmeRenewalStalled and only that. + - interval: 1m + input_series: + # Five days out at t=0, healthy. + - series: 'probe_ssl_earliest_cert_expiry{job="blackbox", instance="https://ntfy.matrix.elysium/v1/health", + name="ntfy", role="alerting", host="trinity", via="dns", renewal="acme"}' + values: "432000+0x180" + alert_rule_test: + - eval_time: 2h + alertname: TlsCertificateExpiringSoon + exp_alerts: [] + - eval_time: 2h + alertname: TlsCertificateExpiryImminent + exp_alerts: [] + - eval_time: 2h + alertname: TlsAcmeRenewalStalled + exp_alerts: [] + + - interval: 1m + input_series: + # 30 hours out at t=1h (108000 + 3600): a stalled renewal. + - series: 'probe_ssl_earliest_cert_expiry{job="blackbox", instance="https://ntfy.matrix.elysium/v1/health", + name="ntfy", role="alerting", host="trinity", via="dns", renewal="acme"}' + values: "111600+0x120" + alert_rule_test: + # `for: 30m`: pending at 20 minutes. + - eval_time: 20m + alertname: TlsAcmeRenewalStalled + exp_alerts: [] + - eval_time: 1h + alertname: TlsAcmeRenewalStalled + exp_alerts: + - exp_labels: + alertname: TlsAcmeRenewalStalled + name: ntfy + role: alerting + host: trinity + component: blackbox + severity: critical + category: availability + exp_annotations: + summary: "ntfy's ACME certificate has 30h left and has not renewed" + description: >- + Caddy on trinity should have renewed this leaf about a day + ago. Read Caddy's log for the renewal error first + (`docker logs sensitive-caddy 2>&1 | grep -i renew`), then check + step-ca is healthy — docs/runbooks/build-the-tier-ca.md. When it + lapses, every name on the tier stops verifying. + - eval_time: 1h + alertname: TlsCertificateExpiryImminent + exp_alerts: []