diff --git a/Makefile b/Makefile index 2d5f2278..79d95231 100644 --- a/Makefile +++ b/Makefile @@ -297,12 +297,16 @@ smart-state: ## Collect SMART health from THIS host's disks (needs root) and ren fi .PHONY: gateway-state -gateway-state: ## Collect the firewall's view of its uplinks (#353) +gateway-state: ## Collect the firewall's view of its uplinks and DDNS record (#353, #604) @# Two measurements per family: what pfSense reports, and whether traffic of @# that family actually leaves the building. They disagreed on 2026-09-07 — @# WAN_DHCP6 reported 100% loss while v6 reached the internet through it in @# 11ms, because dpinger was pointed at a link-local address that does not @# answer echo. One measurement alone cannot tell those apart. + @# + @# And whether the remote path's dynamic DNS name still resolves, at a + @# public resolver, to the WAN address (#604). Compared on the firewall: + @# the name and the address are both withheld, and only the verdict comes back. ./scripts/collect-gateway-state.sh --ssh $(FW_USER)@$(FW_HOST) --host morpheus .PHONY: silence-state diff --git a/README.md b/README.md index 04183ae6..188f2f8d 100644 --- a/README.md +++ b/README.md @@ -69,7 +69,7 @@ documents for different readers. can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed identically everywhere. [How](docs/architecture.md#observability-data-flow) - **Dashboards and alerting as code.** 7 provisioned dashboards, 141 panels, and - 112 alert rules — 94 metric-based in Prometheus, 18 log-based in Loki — sharing + 114 alert rules — 96 metric-based in Prometheus, 18 log-based in Loki — sharing one Alertmanager routing tree. No dashboard exists only in a database. - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials, decrypted at deploy time into gitignored paths, with `git log` showing which @@ -195,7 +195,7 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md). . ├── stacks/observability/ # the deployed stack — one compose file, eight services │ ├── compose.yaml -│ ├── prometheus/ # config, file_sd targets, 94 alert rules +│ ├── prometheus/ # config, file_sd targets, 96 alert rules │ ├── alertmanager/ # routing and inhibition │ ├── loki/ # single-binary config + 18 LogQL rules │ ├── alloy/ # the agent config directory, shipped to every host diff --git a/docs/adr/0044-answer-the-endpoint-with-dynamic-dns-from-morpheus.md b/docs/adr/0044-answer-the-endpoint-with-dynamic-dns-from-morpheus.md index ab389366..ead1357c 100644 --- a/docs/adr/0044-answer-the-endpoint-with-dynamic-dns-from-morpheus.md +++ b/docs/adr/0044-answer-the-endpoint-with-dynamic-dns-from-morpheus.md @@ -166,6 +166,17 @@ see *What would reopen this*. and it is not built here — this ADR records the decision, and a follow-up issue owns the probe. Until then the runbook's outside-the-house check is the only one, and it runs when a human runs it. + + > **Update · 2026-09-22.** The check is built + > ([#604](https://github.com/Gerrrt/HomeLab/issues/604)), in a different + > shape from the one this bullet names. There is no `dns` probe from + > inside. `scripts/collect-gateway-state.sh` sends a script to `morpheus` + > that reads the name from `config.xml` and the WAN address from the + > interface, then asks a public resolver on the firewall itself. Neither + > value leaves the box; only the verdict does, as + > `homelab_ddns_record_matches_wan`, and `DdnsRecordStale` fires when it + > has read 0 for an hour. The decision above is unchanged. What expires is + > this consequence's "unmonitored". - **One more credential lives outside the repository.** The provider account and its token are on the withheld list beside the endpoint, and the token is in `config.xml`, which diff --git a/docs/observability.md b/docs/observability.md index 16f6ea8c..0ecd22c4 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -343,7 +343,7 @@ separates a quiet stream from a stopped one. ## Alerting -112 rules in total: 94 metric-based in `prometheus/rules/`, and 18 log-based in +114 rules in total: 96 metric-based in `prometheus/rules/`, and 18 log-based in `loki/rules/`. ### Log-based (Loki ruler) @@ -450,12 +450,12 @@ argument and for what to do when it exits 1. ### Metric-based (Prometheus) -94 rules across eleven files in `prometheus/rules/`: +96 rules across eleven files in `prometheus/rules/`: | File | Covers | | --- | --- | | `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew and a clock with no time source at all — `HostClockUnsynchronised` reads `node_timex_sync_status`, because the offset reads zero once timesyncd has restored a clock that is wrong but stable, which is how a cleared RTC wrote fourteen minutes of samples three hours in the past and nothing noticed ([#519](https://github.com/Gerrrt/HomeLab/issues/519)) — reboots, and, for the two laptops, whether the shelf is on mains, whether the cell that carries them through a cut is still worth relying on ([#454](https://github.com/Gerrrt/HomeLab/issues/454), and [`runbooks/replace-the-laptop-cell.md`](runbooks/replace-the-laptop-cell.md) for the swap), how hot that cell is, whether its temperature is being measured at all — on 2026-09-19 it was not, on either host — and how many minutes the host has left once the cut arrives ([#532](https://github.com/Gerrrt/HomeLab/issues/532)); whether the wiki's drift check on `oracle` is still running ([#470](https://github.com/Gerrrt/HomeLab/issues/470)); and, for the NAS, whether every ZFS pool is online — `ZpoolNotOnline` reads `node_zfs_zpool_state`, added after `erebor` lost a disk on 2026-09-19 with the exporter hung ahead of it and `InstanceDown` the only page ([#558](https://github.com/Gerrrt/HomeLab/issues/558), [`runbooks/replace-the-nas-disk.md`](runbooks/replace-the-nas-disk.md)); and whether any drive's unsafe-shutdown count has grown in a day — `SmartDriveUnsafeShutdownsGrowing`, the measure of a cut the shutdown sequence [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides did not catch, the rule [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md) left to that issue when it carried `smaug`'s counter under the scrape ([#574](https://github.com/Gerrrt/HomeLab/issues/574)) | -| `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware and Smart Array cache. `shiva`'s Smart Storage Battery read failed from 2026-08-18 until it was replaced on 2026-09-02, with the array in write-through as a result, so stored metrics before that date show the failed pack — `IloBatteryCondition` names the spare part to order, and the controller rollups are deliberately read at *failed* rather than *degraded* ([#76](https://github.com/Gerrrt/HomeLab/issues/76)) | +| `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware and Smart Array cache. `shiva`'s Smart Storage Battery read failed from 2026-08-18 until it was replaced on 2026-09-02, with the array in write-through as a result, so stored metrics before that date show the failed pack — `IloBatteryCondition` names the spare part to order, and the controller rollups are deliberately read at *failed* rather than *degraded* ([#76](https://github.com/Gerrrt/HomeLab/issues/76)). Also whether the remote path's dynamic DNS name still resolves to the WAN address — `DdnsRecordStale` and `DdnsRecordUnchecked`, from the comparison `scripts/collect-gateway-state.sh` makes on the firewall every fifteen minutes ([#604](https://github.com/Gerrrt/HomeLab/issues/604), [ADR-0044](adr/0044-answer-the-endpoint-with-dynamic-dns-from-morpheus.md)); see *What the dynamic DNS series disclose* below | | `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. A pack was fitted on 2026-08-28 and passed its self-test, so these read real hardware; stored metrics older than that date are the card's fabricated values — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) | | `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling | | `stack.rules.yaml` | The stack watching itself: config reloads, rule evaluation, notification delivery, log ingestion, and the two cases `up == 0` structurally cannot see — a remote-writing agent that stops pushing, and a scraped target that stops being a target at all. The second is `ScrapeTargetDisappeared`, added with the first scraped host ([#256](https://github.com/Gerrrt/HomeLab/issues/256)): an emptied or unparseable `targets/node.yaml` makes the series vanish rather than fall to 0, so `InstanceDown` stays silent and `RemoteWriteJobStale` excludes scraped jobs by design. The target for `smaug` was written into `targets/node.yaml` disabled on 2026-09-17 and enabled on 2026-09-19, once the exporter answered from the pool. Split off `containers.rules.yaml` onto `component: stack` in [#81](https://github.com/Gerrrt/HomeLab/issues/81) so a Prometheus that cannot reload its config stops being filed as a container fault. Since [#575](https://github.com/Gerrrt/HomeLab/issues/575) also whether an Alertmanager silence is about to lapse or names no owning issue, read from the per-silence series `scripts/collect_silences.py` writes every fifteen minutes — `alertmanager_silences` is a count per state and cannot say which alert, when, or whose | @@ -473,7 +473,7 @@ as loaded and healthy and could not fire for any input ([#63](https://github.com `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which feed a rule synthetic series and assert it fires — paired with a case asserting it stays quiet, because a test that only ever expects silence would have passed -against the broken rule too. Coverage is seventy-four rules of 94 so far — the five +against the broken rule too. Coverage is seventy-six rules of 96 so far — the five in `blackbox.rules.yaml`, both in `dns.rules.yaml`, `ContainerHighMemory`, `ContainerNearMemoryLimit`, `ContainerRestartLoop`, `ContainerCpuThrottled` and `PrometheusSizeRetentionActive`, `Watchdog`, the three iLO rules from @@ -481,7 +481,8 @@ in `blackbox.rules.yaml`, both in `dns.rules.yaml`, `ContainerHighMemory`, `backup.test.yaml`, all five in `deploy.test.yaml`, `RemoteWriteJobStale`, `ScrapeTargetDisappeared`, `SuricataStopped`, the two gateway rules from -[#353](https://github.com/Gerrrt/HomeLab/issues/353), and all twenty-seven in +[#353](https://github.com/Gerrrt/HomeLab/issues/353), the two dynamic DNS rules from +[#604](https://github.com/Gerrrt/HomeLab/issues/604), and all twenty-seven in `host.rules.yaml` — `HostDiskWillFillIn24h` from [#189](https://github.com/Gerrrt/HomeLab/issues/189), six more from [#320](https://github.com/Gerrrt/HomeLab/issues/320), the four @@ -508,6 +509,25 @@ currently reports, because no service sets a CPU quota. Its tests are what make the rule's correctness checkable anyway, which is the #63 lesson applied before rather than after the fact ([#185](https://github.com/Gerrrt/HomeLab/issues/185)). +#### What the dynamic DNS series disclose + +`homelab_ddns_record_checked` and `homelab_ddns_record_matches_wan` are a +boolean each, labelled with `host` alone. The dynamic DNS name, its provider, +the address a resolver returned and the WAN address are all on +[`security.md`](security.md#what-this-repository-deliberately-does-not-publish)'s +withheld list, and none of them is in a series, a label or a log line. They do +not reach the monitoring host either: the collector sends a script to +`morpheus` over ssh on stdin, and the script reads the name from `config.xml` +and the address from the interface, asks `1.1.1.1`, and prints one word — +`match`, `stale` or `unchecked`. What the series do tell a reader is that the +estate has a dynamic DNS record, which ADR-0044 already publishes. + +`unchecked` is kept separate from `stale` on purpose. If the resolver does +not answer, that says nothing about the record, so `matches_wan` is left out +rather than set to 0 and only `DdnsRecordUnchecked` can fire. A record that is +deleted (NXDOMAIN, or an empty answer) is `stale`. So is a record that has one +right address and one wrong one, because a peer picks either. + Disk alerting is predictive rather than a fixed threshold — `predict_linear` over a 6-hour window, firing when the extrapolation reaches zero within a day *and* free space is already under 30%. A disk sitting at 86% and stable is not an diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index 74c6f64b..30a05707 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -118,7 +118,7 @@ Then in the UI: 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets take up to 45 seconds on their first scrape. -2. **Prometheus → Status → Rules.** 94 rules loaded, none in error. The +2. **Prometheus → Status → Rules.** 96 rules loaded, none in error. The page counts 96: the extra two are the recording rules, `homelab_suricata_expected_interface` and `homelab_battery_runtime_seconds`. Everything counted in this repository diff --git a/docs/runbooks/open-the-remote-path.md b/docs/runbooks/open-the-remote-path.md index ae3076f6..d37d1a74 100644 --- a/docs/runbooks/open-the-remote-path.md +++ b/docs/runbooks/open-the-remote-path.md @@ -112,6 +112,13 @@ section is the first step rather than a stop sign. Then the same query from a phone on mobile data. Both must print the WAN address. A stale answer within the record's TTL is normal for a few minutes after a forced update; a stale answer an hour later is the client. + + After this, the comparison is not only done by hand. `make gateway-state` + makes it on `morpheus` every fifteen minutes, and `DdnsRecordStale` fires + when the two addresses have differed for an hour + ([#604](https://github.com/Gerrrt/HomeLab/issues/604)). The phone check is + still worth doing once, because it is the only one made from outside the + house. 5. **Pick a listen port** that is not a well-known one. It goes on `security.md`'s withheld list beside the hostname and the WAN address, not into a commit message. @@ -573,6 +580,8 @@ and that the first lost device is when it stops being proportionate. | The client reaches the whole internet through the house | `AllowedIPs = 0.0.0.0/0` on the client | §4. This is the wide-`AllowedIPs` failure, and it is silent | | The client reaches Winterfell | A block in §7 is missing or ordered below the catch-all | Check rule order on the ImaginationLAN interface. Treat as a live segmentation failure and read the tripwire log | | `LabSegmentReachedInternalNetwork` fires | Either a real breach, or the rule was widened without the blocks | Both are urgent. Read the `filterlog` line: a `172.31.0.x` source is a peer, a `10.0.30.x` source is the lab | +| `DdnsRecordStale` fires | A public resolver answers the name with an address that is not the WAN's, or answers nothing. There are two causes. The **record is stale**: the updater failed, its token was rotated, or the provider dropped the entry. Or the **WAN address changed** — a new modem, or the spare firewall's different MAC — and the update has not landed | *Force update* on the row under *Services → Dynamic DNS* on `morpheus`. If the row's cached address is the WAN's and the record still differs, the problem is with the provider or the token, so read the row's status and the system log. Then §0 step 4. Until the record matches, the bare address on the withheld list is the way in | +| `DdnsRecordUnchecked` fires | The comparison could not run: `1.1.1.1` did not answer from `morpheus`, ssh to it failed, or no enabled Dynamic DNS entry exists. This says nothing about whether the record is right | `make gateway-state` and read its last line (`ddns=unchecked`). Then check that the entry is still enabled | | Everything works from the sofa and nothing from a hotel | You tested from inside the house | See §8's note. Hicks reaches the lab without any tunnel | | Results that contradict each other across one test run | The client rejoined the house Wi-Fi partway through; macOS prefers Wi-Fi over tethering | Turn Wi-Fi **off**, and re-read the route table before each measurement rather than trusting where you think you are | diff --git a/docs/security.md b/docs/security.md index e4ebe891..0e1cc4e6 100644 --- a/docs/security.md +++ b/docs/security.md @@ -18,7 +18,7 @@ What this network is actually built to survive: | An attacker on the lab segment reaching the hypervisor's BMC | **Accepted.** `shiva` stays on VLAN 30 by decision ([ADR-0033](adr/0033-keep-the-ilo-on-the-lab-segment.md)), hardened on 2026-09-09 — IPMI-over-LAN, SSH and Federation off, and its one path out of the segment deleted; a BMC compromise in the lab costs the lab, and the tripwire watches what it initiates | | A range target with a path out | It has none — `ifrit`'s targets sit on a bridge with no physical port, on `172.30.30.0/24`, which the firewall does not route and on which nothing has a default route at all ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md)) | | Someone with the trusted Wi-Fi key quietly joining | Kea's lease log reaches Loki; `UnknownDeviceOnTrustedSegment` fires the first time a MAC appears on VLAN 50 in seven days ([ADR-0019](adr/0019-read-device-joins-from-the-dhcp-server.md)) | -| Losing visibility of a failure | 112 alert rules, 30 days of metrics and logs | +| Losing visibility of a failure | 114 alert rules, 30 days of metrics and logs | | Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana | | Mains power loss | **The rack, yes; the monitoring path, yes — on two laptop cells that were measured for the first time on 2026-09-12.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the TP-Link carrying `prometheus` and `oracle` has been on UPS power since 2026-09-08 ([#110](https://github.com/Gerrrt/HomeLab/issues/110)); the laptops ride a cut out on their own batteries, which `HostBatteryHealthLow` in `host.rules.yaml` now reads — `prometheus`'s cell was replaced on 2026-09-18 and reads 101 % of design, `oracle`'s is the original at 72 %, with its replacement bought on 2026-09-19 and in transit ([#531](https://github.com/Gerrrt/HomeLab/issues/531)) — and **`prometheus`'s runtime on its cell was measured on 2026-09-19 — about 2.5 hours from full at the stack's load — while `oracle`'s never has been**; since the same day the projection is recorded on every cut and pages under thirty minutes (`HostBatteryRuntimeLow`, [#532](https://github.com/Gerrrt/HomeLab/issues/532)), but neither pack reports a moving cell temperature, so this row is answered for the monitoring host, and for the other only as far as its cell being healthy — see below. **What the UPS cannot answer is what happens when the cut outlasts the pack: as of 2026-09-20 nothing shuts down on its signal, and everything on the PDU — `morpheus`, `Saruman`, `neo` and `smaug`, which is in the media room on a long cord from that PDU — stops uncleanly when the pack empties, about 47 minutes in at 21 % load by the card's own unmeasured estimate.** [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides that the firewall's NUT server halts `Saruman` and `smaug` first and itself last, and [`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) is the build, the forced-shutdown proof and the one mains pull that measures the pack; until those are done the decision is a configuration nobody has tested ([#574](https://github.com/Gerrrt/HomeLab/issues/574)) | | The estate being down while the person who runs it is unavailable | **Documentation, yes; data, not yet.** ADR-0011 puts the emergency tier on paper; [ADR-0023](adr/0023-keep-the-household-recovery-path-outside-the-estate.md) extends the same reasoning to the sensitive tier's data before that tier exists — see below | diff --git a/scripts/collect-gateway-state.sh b/scripts/collect-gateway-state.sh index bbb189a5..2f9a7386 100755 --- a/scripts/collect-gateway-state.sh +++ b/scripts/collect-gateway-state.sh @@ -34,6 +34,24 @@ # The monitor and source columns carry the WAN address and the ISP's gateway, # and docs/security.md says the WAN address is not published. # +# THE DYNAMIC DNS RECORD, AS A COMPARISON (#604). ADR-0044 put the remote +# path's endpoint behind a record `morpheus` keeps current, and the WAN address +# is sticky — so a broken updater changes nothing visible until the one day the +# address moves, and on that day the operator is outside trying to get in. The +# question asked here is whether the name, as a PUBLIC resolver answers it, +# equals the address the WAN interface holds. The estate's own resolver would +# prove nothing: Unbound answers from its cache. +# +# The comparison runs ON THE FIREWALL. The hostname is on the withheld list and +# lives only in config.xml; the WAN address is withheld too. A script goes over +# ssh on stdin, reads both there, asks the resolver there, and prints one word. +# Neither value is in an argv, a variable or a log on this host, and the metric +# is that word as a boolean. +# +# "Could not compare" is kept apart from "does not match". A resolver that did +# not answer is a fact about the measurement, not about the record, and folding +# it into 0 would page for a stale record every time 1.1.1.1 blinked. +# # Usage: scripts/collect-gateway-state.sh --ssh USER@HOST --host NAME [--print] # scripts/collect-gateway-state.sh --self-test set -uo pipefail @@ -48,6 +66,9 @@ HOST_LABEL="" # measure its own egress policy and call it an uplink failure. PROBE4="${GATEWAY_PROBE4:-1.1.1.1}" PROBE6="${GATEWAY_PROBE6:-2606:4700:4700::1111}" +# The public resolver the record is asked of. An address, never a name: it is +# interpolated into the script the firewall runs, so it is checked for shape. +DDNS_RESOLVER="${GATEWAY_DDNS_RESOLVER:-1.1.1.1}" die() { printf '\033[0;31merror:\033[0m %s\n' "$*" >&2; exit 1; } @@ -79,6 +100,79 @@ parse_gatewaystatus() { ' } +# The verdict on one `drill NAME A` answer, given the WAN address as `wan`. +# Prints exactly one of match, stale or unchecked. It runs on the firewall and +# is kept here so --self-test exercises the same text. No single quotes in it: +# it travels inside a single-quoted shell assignment. +# +# NOERROR, every A equals wan (after any CNAME) match +# NOERROR, any A differs, or no A at all stale — deleted counts +# NXDOMAIN stale +# SERVFAIL, REFUSED, no header (timeout) unchecked +# +# An answer with one right A and one wrong one is stale: a peer picks either. +# shellcheck disable=SC2016 # awk's fields, not the shell's +DDNS_VERDICT_AWK=' + /^;; ->>HEADER<<-/ { + for (i = 1; i <= NF; i++) if ($i == "rcode:") { rcode = $(i + 1); sub(/,$/, "", rcode) } + } + /^;; ANSWER SECTION:/ { inans = 1; next } + inans && /^[[:space:]]*$/ { inans = 0 } + inans && $4 == "A" { n++; if ($5 != wan) bad++ } + END { + if (rcode == "NXDOMAIN") print "stale" + else if (rcode != "NOERROR") print "unchecked" + else if (n == 0 || bad > 0) print "stale" + else print "match" + } +' + +# The script the firewall runs for the comparison, on stdin to `sh -s`. It +# prints one word and nothing else. The PHP reads config.xml with pfSense's own +# config API and hands each enabled entry's name and interface address to the +# shell below it; both stay in that process on the firewall. +# +# The name is `host`, or `host.domainname` for the providers pfSense splits +# that way — the rule the Dynamic DNS status widget applies, reproduced because +# it is not in a shared include. `-v6` client types update AAAA records, which +# ADR-0044 decision 3 does not use, and are skipped. +remote_ddns_script() { + printf "resolver='%s'\n" "$DDNS_RESOLVER" + printf "verdict_awk='%s'\n" "$DDNS_VERDICT_AWK" + cat <<'SH' +entries="$(/usr/local/bin/php 2>/dev/null <<'PHP' +/dev/null | awk -v wan="$wan" "$verdict_awk")" + fi + case "$v" in + stale) result=stale ;; + match) ;; + *) [ "$result" = stale ] || result=unchecked ;; + esac +done <>HEADER<<- opcode: QUERY, rcode: %s, id: 4242\n;; flags: qr rd ra ; QUERY: 1, ANSWER: %s, AUTHORITY: 0, ADDITIONAL: 0\n;; QUESTION SECTION:\n;; home.example.net.\tIN\tA\n\n;; ANSWER SECTION:\n' "$1" "$2"; } + tail_=$'\n;; AUTHORITY SECTION:\n\n;; ADDITIONAL SECTION:\n\n;; Query time: 13 msec\n;; SERVER: 198.51.100.53' + verdict "record equals the WAN address" match \ + "$(hdr NOERROR 1) +home.example.net. 60 IN A 192.0.2.10 +${tail_}" + verdict "record holds another address" stale \ + "$(hdr NOERROR 1) +home.example.net. 60 IN A 198.51.100.7 +${tail_}" + verdict "a prefix of the WAN address is not a match" stale \ + "$(hdr NOERROR 1) +home.example.net. 60 IN A 192.0.2.1 +${tail_}" + verdict "NXDOMAIN is a stale record, not a failed check" stale \ + "$(hdr NXDOMAIN 0) +${tail_}" + verdict "NOERROR with no answer is a deleted record" stale \ + "$(hdr NOERROR 0) +${tail_}" + verdict "SERVFAIL is a failed check" unchecked \ + "$(hdr SERVFAIL 0) +${tail_}" + verdict "no reply at all is a failed check" unchecked "" + verdict "a CNAME chain ending at the WAN address matches" match \ + "$(hdr NOERROR 2) +www.example.net. 300 IN CNAME home.example.net. +home.example.net. 60 IN A 192.0.2.10 +${tail_}" + verdict "one right A and one wrong is stale" stale \ + "$(hdr NOERROR 2) +home.example.net. 60 IN A 192.0.2.10 +home.example.net. 60 IN A 198.51.100.7 +${tail_}" + # An A record in the authority or additional section is not the answer. + verdict "an A outside the answer section is ignored" stale \ + "$(hdr NOERROR 0) +;; AUTHORITY SECTION: + +;; ADDITIONAL SECTION: +ns.example.net. 60 IN A 192.0.2.10" + + # The firewall's script must at least parse as sh; it is never run here. + if remote_ddns_script | sh -n 2>/dev/null; then + printf '\033[0;32m PASS\033[0m %s\n' "the firewall's script parses" + else + printf '\033[0;31m FAIL\033[0m %s\n' "the firewall's script does not parse"; fail=1 + fi exit $fail fi @@ -123,6 +278,7 @@ while (($#)); do done [[ -n "$SSH_TARGET" ]] || die "--ssh USER@HOST is required" [[ -n "$HOST_LABEL" ]] || die "--host NAME is required" +[[ "$DDNS_RESOLVER" =~ ^[0-9A-Fa-f:.]+$ ]] || die "GATEWAY_DDNS_RESOLVER must be an address" STDERR_FILE="$(mktemp)"; trap 'rm -f "${STDERR_FILE}"' EXIT @@ -146,6 +302,16 @@ probe_family() { "$cmd -c 2 -t 5 $target >/dev/null 2>&1"; then echo 1; else echo 0; fi } +# The record comparison, on the firewall. Anything but the two words that are a +# verdict — ssh refused, a timeout, a script error — is `unchecked`, and none of +# it fails the gateway metrics above. +probe_ddns() { + local v + v="$(remote_ddns_script | timeout 60 ssh -o BatchMode=yes -o ConnectTimeout=10 \ + "$SSH_TARGET" 'sh -s' 2>/dev/null | tail -1)" + case "$v" in match|stale) echo "$v" ;; *) echo unchecked ;; esac +} + emit() { printf '# HELP homelab_gateway_status 1 when the firewall reports this gateway online.\n' printf '# TYPE homelab_gateway_status gauge\n' @@ -177,6 +343,20 @@ emit() { printf 'homelab_gateway_forwarding{host="%s",family="%s"} %s\n' \ "$HOST_LABEL" "$fam" "$(probe_family "$fam")" done + + local ddns; ddns="$(probe_ddns)" + printf '# HELP homelab_ddns_record_checked 1 when the firewall could compare its dynamic DNS record with its WAN address. Carries no name and no address.\n' + printf '# TYPE homelab_ddns_record_checked gauge\n' + printf 'homelab_ddns_record_checked{host="%s"} %s\n' \ + "$HOST_LABEL" "$([[ "$ddns" == unchecked ]] && echo 0 || echo 1)" + # Absent rather than 0 when the comparison did not run: a check that could + # not ask must not read as a record that answered wrong. + if [[ "$ddns" != unchecked ]]; then + printf '# HELP homelab_ddns_record_matches_wan 1 when a public resolver answers the dynamic DNS name with the WAN address. Carries no name and no address.\n' + printf '# TYPE homelab_ddns_record_matches_wan gauge\n' + printf 'homelab_ddns_record_matches_wan{host="%s"} %s\n' \ + "$HOST_LABEL" "$([[ "$ddns" == match ]] && echo 1 || echo 0)" + fi } if ((PRINT_ONLY)); then emit; exit 0; fi @@ -187,4 +367,6 @@ tmp="${PROM}.$$" emit > "${tmp}" || { rm -f "${tmp}"; die "could not write ${tmp}"; } [[ -s "${tmp}" ]] || { rm -f "${tmp}"; die "rendered no metrics"; } chmod 0644 "${tmp}"; mv -f "${tmp}" "${PROM}" -printf 'gateway-state host=%s gateways=%s\n' "$HOST_LABEL" "$(printf '%s\n' "$rows" | wc -l | tr -d ' ')" +printf 'gateway-state host=%s gateways=%s ddns=%s\n' "$HOST_LABEL" \ + "$(printf '%s\n' "$rows" | wc -l | tr -d ' ')" \ + "$(awk '/^homelab_ddns_record_matches_wan/ { print ($2 == 1 ? "match" : "stale"); f = 1 } END { if (!f) print "unchecked" }' "${PROM}")" diff --git a/stacks/observability/README.md b/stacks/observability/README.md index 8ba9ccaa..52d73fc9 100644 --- a/stacks/observability/README.md +++ b/stacks/observability/README.md @@ -32,7 +32,7 @@ prometheus/ prometheus.yaml scrape config; SNMP via file_sd targets/snmp.yaml SNMP targets — hot-reloaded, no restart needed targets/blackbox*.yaml probe targets, http and dns — hot-reloaded, no restart - rules/*.rules.yaml 94 alert rules across host/network/ups/containers/blackbox/dns/backup/ids/deploy + rules/*.rules.yaml 96 alert rules across host/network/ups/containers/blackbox/dns/backup/ids/deploy tests/*.test.yaml promtool unit tests — assert the rules can fire blackbox/blackbox.yaml probe modules — reachability, and what a resolver said alertmanager/ diff --git a/stacks/observability/prometheus/rules/network.rules.yaml b/stacks/observability/prometheus/rules/network.rules.yaml index 3e9a32ab..34cb9810 100644 --- a/stacks/observability/prometheus/rules/network.rules.yaml +++ b/stacks/observability/prometheus/rules/network.rules.yaml @@ -395,3 +395,53 @@ groups: gateway's Monitor IP at something that answers, or set it to not-monitored. Until then the gateway's real state cannot be read from its status. + + # --- The remote path's dynamic DNS record (#604) ----------------------- + # + # ADR-0044 keeps the WireGuard endpoint as a name updated by `morpheus`, + # and the WAN address is sticky: a broken updater hides until the day + # the address moves, and the person who finds out is outside the house. + # scripts/collect-gateway-state.sh compares the name, as a public + # resolver answers it, with the WAN address — on the firewall, so the + # series carries the verdict and neither value. + + - alert: DdnsRecordStale + # 1h is four consecutive 15-minute samples. A DHCP change pushes the + # update on the event and a free zone's TTL is about a minute, so a + # mismatch that outlasts an hour is not an update in flight. + expr: homelab_ddns_record_matches_wan == 0 + for: 1h + labels: + component: network + severity: warning + category: availability + annotations: + summary: "{{ $labels.host }}: the remote path's DNS name does not resolve to the WAN address" + description: >- + A public resolver answers the dynamic DNS name with something other + than the address the WAN holds, or with nothing. Either the record + is stale — the updater failed, its token was rotated, the provider + dropped the entry — or the WAN address changed and the update has + not landed. The tunnel cannot be reached by name until it matches. + First move: Services → Dynamic DNS on morpheus, Force update. Then + docs/runbooks/open-the-remote-path.md §0. + + - alert: DdnsRecordUnchecked + # The comparison could not run: the resolver did not answer, the + # firewall could not be asked, or config.xml holds no enabled entry. + # Without this a broken check reads as silence, which is the failure + # #604 exists to close. 6h, because a resolver blink is not news. + expr: homelab_ddns_record_checked == 0 + for: 6h + labels: + component: network + severity: warning + category: config + annotations: + summary: "{{ $labels.host }}: the dynamic DNS record has not been checked for 6h" + description: >- + The collector could not compare the dynamic DNS name with the WAN + address — the public resolver did not answer, ssh to the firewall + failed, or no enabled Dynamic DNS entry exists. This says nothing + about whether the record is right; it says nothing is watching it. + Run `make gateway-state` and read its last line. diff --git a/stacks/observability/prometheus/tests/network.test.yaml b/stacks/observability/prometheus/tests/network.test.yaml index ca70887e..fa0e9feb 100644 --- a/stacks/observability/prometheus/tests/network.test.yaml +++ b/stacks/observability/prometheus/tests/network.test.yaml @@ -559,3 +559,86 @@ tests: - eval_time: 45m alertname: GatewayDown exp_alerts: [] + + # --- DdnsRecordStale: fires once a mismatch outlasts an update ------------ + # The textfile is rewritten every 15 minutes; a 1m interval stands in for + # the scrape. Mismatch from 10m, so 45m is too soon and 80m is not. + - interval: 1m + input_series: + - series: 'homelab_ddns_record_checked{host="morpheus"}' + values: "1x120" + - series: 'homelab_ddns_record_matches_wan{host="morpheus"}' + values: "1x10 0x110" + alert_rule_test: + - eval_time: 45m + alertname: DdnsRecordStale + exp_alerts: [] + - eval_time: 80m + alertname: DdnsRecordStale + exp_alerts: + - exp_labels: + host: morpheus + component: network + severity: warning + category: availability + exp_annotations: + summary: "morpheus: the remote path's DNS name does not resolve to the WAN address" + description: >- + A public resolver answers the dynamic DNS name with something + other than the address the WAN holds, or with nothing. Either + the record is stale — the updater failed, its token was + rotated, the provider dropped the entry — or the WAN address + changed and the update has not landed. The tunnel cannot be + reached by name until it matches. First move: Services → + Dynamic DNS on morpheus, Force update. Then + docs/runbooks/open-the-remote-path.md §0. + - eval_time: 80m + alertname: DdnsRecordUnchecked + exp_alerts: [] + + # --- A check that cannot run is Unchecked, and never Stale ---------------- + # The collector omits matches_wan when the comparison did not run, so a + # resolver outage must not read as a wrong record. + - interval: 1m + input_series: + - series: 'homelab_ddns_record_checked{host="morpheus"}' + values: "0x480" + alert_rule_test: + - eval_time: 5h + alertname: DdnsRecordUnchecked + exp_alerts: [] + - eval_time: 7h + alertname: DdnsRecordUnchecked + exp_alerts: + - exp_labels: + host: morpheus + component: network + severity: warning + category: config + exp_annotations: + summary: "morpheus: the dynamic DNS record has not been checked for 6h" + description: >- + The collector could not compare the dynamic DNS name with the + WAN address — the public resolver did not answer, ssh to the + firewall failed, or no enabled Dynamic DNS entry exists. This + says nothing about whether the record is right; it says + nothing is watching it. Run `make gateway-state` and read its + last line. + - eval_time: 7h + alertname: DdnsRecordStale + exp_alerts: [] + + # --- A matching record stays quiet ---------------------------------------- + - interval: 1m + input_series: + - series: 'homelab_ddns_record_checked{host="morpheus"}' + values: "1x480" + - series: 'homelab_ddns_record_matches_wan{host="morpheus"}' + values: "1x480" + alert_rule_test: + - eval_time: 7h + alertname: DdnsRecordStale + exp_alerts: [] + - eval_time: 7h + alertname: DdnsRecordUnchecked + exp_alerts: []