From 883fcb68c22d09639c2d5ebe70be914ce85cdb3f Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Sun, 6 Sep 2026 18:28:22 +0000 Subject: [PATCH] feat(ups): watch the self-test schedule the battery control leans on (#249) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #93 set mjolnir to test itself every fortnight, and nothing in the stack could see that setting. upsTestResultsSummary is the LAST result: if the card reverts to never(5) after a config reset or a firmware update, it holds donePass forever and every rule in ups.rules.yaml stays correctly quiet. UpsBatteryUnproven cannot catch it — it matches noTestsInitiated(6) and the card reads 1 — and upsTestStartTime is sysUpTime-relative, so it is not a staleness signal either. THE VENDOR MIB WAS NOT THE COST THE ISSUE EXPECTED. #249 framed PowerNet as "a pinning decision, not a URL", because Schneider distribute it as a versioned download rather than a git ref: vendor 2.2 MB into the tree, or trust a moving vendor path. Neither is needed. The observium commit this repository ALREADY pins for UPS-MIB carries mibs/apc/PowerNet-MIB, so this adds a path against an existing pin and inherits its argument unchanged. One subtree, 1.3.6.1.4.1.318.1.1.1.7.2, never the enterprise root — the pfTablesAddrTable lesson one vendor along. Measured against the live card: snmp-walk 7 rows, 2 GETBULKs, 0.07s scrape 3 packets / 51 PDUs / 56 series -> 4 / 58 / 63 and the three the issue named arrive as predicted: upsAdvTestDiagnosticSchedule 8 upsAdvTestDiagnosticsResults 1 upsAdvTestLastDiagnosticsDate{upsAdvTestLastDiagnosticsDate="08/28/2026"} 1 Two rules, a pair on purpose: UpsSelfTestScheduleOff is the fast signal, true the moment the card reads never(5); UpsSelfTestStale is the backstop for a card that says it is scheduled and is not. Either alone leaves half the failure. The enumeration was read out of the MIB rather than assumed. An older PowerNet enumeration stopping at twelveWeeks(7) is in circulation and would read this card's 8 as out of range; the real one runs to fiftytwoWeeks(12). A unit test pins 8 as quiet for that reason — a rule firing for the estate's correct setting would be worse than no rule. UpsSelfTestStale counts distinct series over 21d rather than using `for:`. upsAdvTestLastDiagnosticsDate is a DisplayString, which snmp_exporter renders as a value-1 gauge carrying the date as a label, so "the date changed" is "a second series appeared" and no date parsing is needed. `for: 21d` would restart its clock whenever Prometheus does. It is gated on up{job="snmp"} because a dead exporter produces no new series either and would otherwise read as a card that stopped testing — a true alert about the wrong thing. tests/ups.test.yaml did not exist; nine rules were syntax-only, the standing #63 objected to. Seven cases now cover both new rules, including the dead-exporter trap and the fiftytwoWeeks case that stops someone tidying the expression into `<= 5 or >= 10`. Two corrections made on the way: `metric =~ 1|4|5` is a parse error, because =~ is a label matcher and not a value comparison, so the rule is three `or`ed comparisons; and docs/observability.md's promtool coverage claim was counted rather than incremented — 37 of 58, not the 35 of 56 it said. Co-Authored-By: Claude Opus 5 --- README.md | 4 +- docs/observability.md | 6 +- docs/roadmap.md | 49 +++++ docs/runbooks/deploy-stack.md | 2 +- docs/runbooks/fit-the-ups-battery.md | 15 +- docs/security.md | 28 +-- scripts/snmp-mibs.sh | 22 +++ stacks/observability/README.md | 2 +- .../prometheus/rules/ups.rules.yaml | 125 +++++++++++-- .../prometheus/tests/ups.test.yaml | 176 ++++++++++++++++++ .../snmp-exporter/generator.yaml | 31 +++ stacks/observability/snmp-exporter/snmp.yaml | 78 ++++++++ 12 files changed, 493 insertions(+), 45 deletions(-) create mode 100644 stacks/observability/prometheus/tests/ups.test.yaml diff --git a/README.md b/README.md index 0eee2133..62beadd9 100644 --- a/README.md +++ b/README.md @@ -58,7 +58,7 @@ documents for different readers. can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed identically everywhere. [How](docs/architecture.md#observability-data-flow) - **Dashboards and alerting as code.** 7 provisioned dashboards, 140 panels, and - 72 alert rules — 56 metric-based in Prometheus, 16 log-based in Loki — sharing + 74 alert rules — 58 metric-based in Prometheus, 16 log-based in Loki — sharing one Alertmanager routing tree. No dashboard exists only in a database. - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials, decrypted at deploy time into gitignored paths, with `git log` showing which @@ -175,7 +175,7 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md). . ├── stacks/observability/ # the deployed stack — one compose file, six services │ ├── compose.yaml -│ ├── prometheus/ # config, file_sd targets, 56 alert rules +│ ├── prometheus/ # config, file_sd targets, 58 alert rules │ ├── alertmanager/ # routing and inhibition │ ├── loki/ # single-binary config + 16 LogQL rules │ ├── alloy/ # the agent config directory, shipped to every host diff --git a/docs/observability.md b/docs/observability.md index 218043bb..96269afd 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -298,7 +298,7 @@ separates a quiet stream from a stopped one. ## Alerting -72 rules in total: 56 metric-based in `prometheus/rules/`, and 16 log-based in +74 rules in total: 58 metric-based in `prometheus/rules/`, and 16 log-based in `loki/rules/`. ### Log-based (Loki ruler) @@ -382,7 +382,7 @@ argument and for what to do when it exits 1. ### Metric-based (Prometheus) -56 rules across eleven files in `prometheus/rules/`: +58 rules across eleven files in `prometheus/rules/`: | File | Covers | | --- | --- | @@ -405,7 +405,7 @@ as loaded and healthy and could not fire for any input ([#63](https://github.com `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which feed a rule synthetic series and assert it fires — paired with a case asserting it stays quiet, because a test that only ever expects silence would have passed -against the broken rule too. Coverage is thirty-five rules of 56 so far — the five +against the broken rule too. Coverage is thirty-seven rules of 58 so far — the five in `blackbox.rules.yaml`, both in `dns.rules.yaml`, `ContainerHighMemory`, `ContainerNearMemoryLimit`, `ContainerRestartLoop`, `ContainerCpuThrottled` and `PrometheusSizeRetentionActive`, `Watchdog`, the three iLO rules from diff --git a/docs/roadmap.md b/docs/roadmap.md index 93223476..8d6668dd 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -619,6 +619,55 @@ months. ## Done +- [x] **[#249](https://github.com/Gerrrt/HomeLab/issues/249) Watch the UPS + self-test schedule.** 2026-09-06. `#93` set `mjolnir` to test itself every + fortnight and nothing in the stack could see that setting, so the control + keeping `upsTestResultsSummary` meaningful was itself unmonitored. + + **The vendor MIB turned out not to be the cost the issue expected.** #249 + framed adding PowerNet as "a pinning decision, not a URL" — Schneider + distribute it as a versioned download rather than a git ref, so the + options looked like vendoring 2.2 MB into the tree or trusting a moving + vendor path. Neither was needed: the observium commit this repository + **already pins** for UPS-MIB carries `mibs/apc/PowerNet-MIB` too. The line + added is a path against an existing pin, inheriting its argument unchanged. + + One subtree, `1.3.6.1.4.1.318.1.1.1.7.2`, never the enterprise root — the + `pfTablesAddrTable` lesson one vendor along. Measured against the live + card: 7 rows, 2 GETBULKs, 0.07s to walk, and end to end the scrape goes + from 3 packets / 51 PDUs / 56 series to 4 / 58 / 63. + + Two rules, and they are a pair on purpose. `UpsSelfTestScheduleOff` is the + fast signal, true the moment the card reads `never(5)`; + `UpsSelfTestStale` is the backstop for a card that says it is scheduled + and is not. Either alone leaves half the failure. + + The enumeration was read out of the MIB rather than assumed, because an + older PowerNet enumeration stopping at `twelveWeeks(7)` is in circulation + and would read this card's `8` as out of range — the real definition runs + to `fiftytwoWeeks(12)`. A unit test pins `8` as quiet for exactly that + reason: a rule that fired for the estate's correct setting would be worse + than no rule. + + `UpsSelfTestStale` counts distinct series over 21 days rather than using + `for:`, because `upsAdvTestLastDiagnosticsDate` is a DisplayString and + snmp_exporter renders it as a value-1 gauge carrying the date as a label — + so "the date changed" is "a second series appeared", and no date parsing + is needed. It is gated on `up{job="snmp"}` because a dead exporter + produces no new series either and would otherwise read as a card that + stopped testing. + + **`tests/ups.test.yaml` did not exist**; nine rules were syntax-only, + which is the standing #63 objected to. It does now, with seven cases + covering both new rules — including the dead-exporter trap and the + long-schedule case that stops someone tidying the expression into + `<= 5 or >= 10`. + + Still outstanding and not a monitoring question: the proof the schedule + *runs* rather than merely being set is the date advancing with nobody at + the card, due around 2026-09-11. As of 2026-09-06 it still reads + `08/28/2026`. `UpsSelfTestStale` is now what will say so if it does not. + - [x] **[#339](https://github.com/Gerrrt/HomeLab/issues/339) Fail when a declared timer is not actually installed.** 2026-09-06. `make check-timers` now asks `systemctl is-enabled` for every job in the `JOBS` diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index 2f01d551..66d39583 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -118,7 +118,7 @@ Then in the UI: 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets take up to 45 seconds on their first scrape. -2. **Prometheus → Status → Rules.** 56 rules loaded, none in error. The +2. **Prometheus → Status → Rules.** 58 rules loaded, none in error. The page counts 57: the extra one is `homelab_suricata_expected_interface`, the stack's only recording rule. Everything counted in this repository is alert rules, so the two numbers differ by one and always have. diff --git a/docs/runbooks/fit-the-ups-battery.md b/docs/runbooks/fit-the-ups-battery.md index 2f936871..12cb4132 100644 --- a/docs/runbooks/fit-the-ups-battery.md +++ b/docs/runbooks/fit-the-ups-battery.md @@ -25,12 +25,15 @@ un-silenced by hand at the right moment.** > advancing on its own — not the setting reading `8` — that proves the schedule > actually runs. > -> **Nothing in the stack can see any of that.** All three are PowerNet OIDs -> under `1.3.6.1.4.1.318`, and the `apc_ups` module walks the standard UPS-MIB -> only, so no rule in `ups.rules.yaml` can detect a card that has quietly -> stopped testing. That is -> [#249](https://github.com/Gerrrt/HomeLab/issues/249); until it lands, step 6 -> is verified by hand with `scripts/snmp-walk.sh` and by nothing else. +> **The stack can see all of that as of +> [#249](https://github.com/Gerrrt/HomeLab/issues/249).** The `apc_ups` module +> walks `1.3.6.1.4.1.318.1.1.1.7.2` alongside the standard UPS-MIB, and two +> rules in `ups.rules.yaml` watch it: `UpsSelfTestScheduleOff` fires the moment +> the schedule leaves a recurring value, and `UpsSelfTestStale` fires when the +> last-test date has not moved in 21 days — the fortnightly interval plus a +> week of grace. Step 6 is no longer a by-hand check, though +> `scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2` is still how +> you read the card directly when one of those fires. > > **One reading that was not reset.** `upsBasicBatteryLastReplaceDate` is > `08/15/2026` — thirteen days before the pack went in, on a date when this diff --git a/docs/security.md b/docs/security.md index 026052bc..efe3ea0e 100644 --- a/docs/security.md +++ b/docs/security.md @@ -16,7 +16,7 @@ What this network is actually built to survive: | A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it | | A range target with a path out | It has none — `ifrit`'s targets sit on a bridge with no physical port, on `172.30.30.0/24`, which the firewall does not route and on which nothing has a default route at all ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md)) | | Someone with the trusted Wi-Fi key quietly joining | Kea's lease log reaches Loki; `UnknownDeviceOnTrustedSegment` fires the first time a MAC appears on VLAN 50 in seven days ([ADR-0019](adr/0019-read-device-joins-from-the-dhcp-server.md)) | -| Losing visibility of a failure | 72 alert rules, 30 days of metrics and logs | +| Losing visibility of a failure | 74 alert rules, 30 days of metrics and logs | | Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana | | Mains power loss | **The rack, yes; the monitoring path, no.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the switch carrying `prometheus` and `oracle` still has no battery — see below | | The estate being down while the person who runs it is unavailable | **Documentation, yes; data, not yet.** ADR-0011 puts the emergency tier on paper; [ADR-0023](adr/0023-keep-the-household-recovery-path-outside-the-estate.md) extends the same reasoning to the sensitive tier's data before that tier exists — see below | @@ -329,17 +329,21 @@ fabricated values rather than measurements, so a dashboard or query whose range crosses that date is reading fiction on one side of it. And the card's test schedule is on but unwatched. Read off the NMC on 2026-09-03, `upsAdvTestDiagnosticSchedule` is `8` (biweeklySinceLastTest), which is what -*should* keep `1` from being a frozen last-known result — should, because -nothing here can confirm it still does. That OID is PowerNet, and -the `apc_ups` module walks the standard UPS-MIB only. Nothing here would notice -the card reverting to `never`: `upsTestResultsSummary` would hold `1` and every -rule would stay quiet. `UpsBatteryUnproven` cannot catch it either, because it -matches `6` (noTestsInitiated) and this card reads `1`. Note the shape of that — -the missing pack was visible in a MIB already walked, and the missing *schedule* -would not be. The manual check is in -[`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md), and -closing the gap is -[#249](https://github.com/Gerrrt/HomeLab/issues/249). +keeps `1` from being a frozen last-known result — and as of +[#249](https://github.com/Gerrrt/HomeLab/issues/249) that is confirmed rather +than assumed. The `apc_ups` module now walks `1.3.6.1.4.1.318.1.1.1.7.2` +alongside the standard UPS-MIB, and two rules watch it: `UpsSelfTestScheduleOff` +fires the moment the schedule reads unknown, at-turn-on or never, and +`UpsSelfTestStale` fires when the last-test date has not moved in 21 days. + +That closes a gap worth naming for its shape rather than its size. The missing +battery pack was visible in a MIB already walked — `upsTestResultsSummary = 4` +needed no new OIDs — and the missing *schedule* would not have been. +`upsTestResultsSummary` would have held `1` forever and every rule would have +stayed correctly quiet; `UpsBatteryUnproven` could not have caught it either, +because it matches `6` (noTestsInitiated) and this card reads `1`. A +healthy-looking metric with nothing underneath it is the failure this estate +keeps finding, one level up each time. ### Why SNMPv2c is still a weak point diff --git a/scripts/snmp-mibs.sh b/scripts/snmp-mibs.sh index 3c401c95..84142a5b 100755 --- a/scripts/snmp-mibs.sh +++ b/scripts/snmp-mibs.sh @@ -62,6 +62,14 @@ command -v tar >/dev/null 2>&1 || die "tar not found" NET_SNMP_URL='https://raw.githubusercontent.com/net-snmp/net-snmp/v5.9/mibs' FREEBSD_URL='https://raw.githubusercontent.com/freebsd/freebsd-src/release/14.2.0' UPS_MIB_URL='https://raw.githubusercontent.com/pgmillon/observium/9064fa3ae1ae634709198decf99aca7380693351/mibs/UPS-MIB' +# PowerNet, from the SAME observium commit as UPS-MIB above, which is the whole +# reason this line is cheap. #249 expected adding APC's vendor MIB to be "a +# pinning decision, not a URL" — Schneider distribute it as a versioned download +# rather than a git ref, so the options were vendoring 2.2 MB into the tree or +# trusting a moving vendor path. Neither is needed: the commit this repository +# already pins for UPS-MIB carries mibs/apc/PowerNet-MIB too, so this adds a +# path to an existing pin and inherits its argument unchanged. +POWERNET_MIB_URL='https://raw.githubusercontent.com/pgmillon/observium/9064fa3ae1ae634709198decf99aca7380693351/mibs/apc/PowerNet-MIB' HPE_URL='https://downloads.hpe.com/pub/softlib2/software1/pubsw-linux/p1580676047/v229101/upd11.85mib.tar.gz' CURL_OPTS=(-L -sS --retry 3 --retry-delay 3 --fail --max-time 180) @@ -98,6 +106,20 @@ done info "UPS-MIB" fetch UPS-MIB "${UPS_MIB_URL}" +# --------------------------------------------------------------------------- +# PowerNet, APC's own tree (1.3.6.1.4.1.318). UPS-MIB above covers battery, +# input, output and the LAST self-test result; it has no concept of a self-test +# SCHEDULE, which is the control #93 set and #249 found nothing watching. +# +# 2.2 MB and 70,663 lines for three scalars, which looks disproportionate and is +# not: the generator reads the whole tree to resolve three OIDs and emits only +# what generator.yaml asks for. The apc_ups module walks +# 1.3.6.1.4.1.318.1.1.1.7.2 and nothing else of 318 — walking the enterprise +# root is the pfTablesAddrTable mistake, one vendor along. +# --------------------------------------------------------------------------- +info "PowerNet-MIB (APC)" +fetch PowerNet-MIB "${POWERNET_MIB_URL}" + # --------------------------------------------------------------------------- # pfSense exposes pf through FreeBSD's bsnmpd. BEGEMOT-PF-MIB imports from # BEGEMOT-MIB, which imports from FOKUS-MIB; miss either and the generator diff --git a/stacks/observability/README.md b/stacks/observability/README.md index ca44ba7e..3bc92616 100644 --- a/stacks/observability/README.md +++ b/stacks/observability/README.md @@ -32,7 +32,7 @@ prometheus/ prometheus.yaml scrape config; SNMP via file_sd targets/snmp.yaml SNMP targets — hot-reloaded, no restart needed targets/blackbox*.yaml probe targets, http and dns — hot-reloaded, no restart - rules/*.rules.yaml 56 alert rules across host/network/ups/containers/blackbox/dns/backup/ids/deploy + rules/*.rules.yaml 58 alert rules across host/network/ups/containers/blackbox/dns/backup/ids/deploy tests/*.test.yaml promtool unit tests — assert the rules can fire blackbox/blackbox.yaml probe modules — reachability, and what a resolver said alertmanager/ diff --git a/stacks/observability/prometheus/rules/ups.rules.yaml b/stacks/observability/prometheus/rules/ups.rules.yaml index 5ab8775d..74fa5b9c 100644 --- a/stacks/observability/prometheus/rules/ups.rules.yaml +++ b/stacks/observability/prometheus/rules/ups.rules.yaml @@ -49,39 +49,39 @@ # 2026-09-20 (54f1715c-e57b-4322-8a6d-5435bc8e1bd8) was deleted on 2026-08-28, # so both rules are live. # -# WHAT THIS FILE STILL CANNOT SEE. +# WHAT THIS FILE COULD NOT SEE UNTIL #249. # -# Scheduled self-tests ARE enabled on the NMC. Read off the card on 2026-09-03: +# Scheduled self-tests are enabled on the NMC. Read off the card on 2026-09-06, +# matching what #249 recorded three days earlier: # # upsAdvTestDiagnosticSchedule 8 (biweeklySinceLastTest) # upsAdvTestDiagnosticsResults 1 (ok) # upsAdvTestLastDiagnosticsDate "08/28/2026" # -# All three are PowerNet (1.3.6.1.4.1.318.1.1.1.7.2.x), and the apc_ups module -# in snmp-exporter/generator.yaml walks 1.3.6.1.2.1.33.1 — the standard UPS-MIB -# — and nothing else. So none of it reaches Prometheus, and no rule below can -# be written against it. +# All three are PowerNet (1.3.6.1.4.1.318.1.1.1.7.2.x), and until #249 the +# apc_ups module walked 1.3.6.1.2.1.33.1 — the standard UPS-MIB — and nothing +# else. None of it reached Prometheus, so no rule here could be written against +# it, and that left a hole worth naming precisely. # -# That leaves one hole open. A value of 1 is the *last* result, not a fresh one: -# if the card stops testing, upsTestResultsSummary simply stops changing, and -# nothing here detects that. UpsBatteryUnproven cannot — it matches 6 -# (noTestsInitiated), and this card reads 1. upsTestStartTime is +# upsTestResultsSummary is the LAST result, not a fresh one. If the card stops +# testing it simply stops changing, and every rule below stays correctly quiet. +# UpsBatteryUnproven cannot catch it — that matches 6 (noTestsInitiated) and a +# card that has stopped testing still reads 1. upsTestStartTime is # sysUpTime-relative and does not survive an agent restart, so it is not a # usable staleness signal either. A card that quietly reverts to never(5) — a -# config reset, a firmware update, a restored configuration file — is +# config reset, a firmware update, a restored configuration file — was # indistinguishable, from inside this file, from one testing every fortnight. # -# The fix is to walk those three OIDs and alert on the schedule leaving the -# recurring values and on the last-test date going stale, which needs APC's -# PowerNet MIB added to scripts/snmp-mibs.sh and a pinning decision for it. That -# is issue #249, not this file. Until it lands, the check is by hand: +# UpsSelfTestScheduleOff and UpsSelfTestStale below close that, and they are a +# pair on purpose: the first is the fast signal, true the moment the card reads +# never(5); the second is the backstop for a card that says it is scheduled and +# is not. Either alone would leave half the failure. # -# scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2 -# -# and the thing that proves the schedule runs, rather than merely being set, is +# The thing that proves the schedule RUNS rather than merely being set is # upsAdvTestLastDiagnosticsDate advancing with nobody at the card — due around -# 2026-09-11 for a biweekly schedule last tested on 2026-08-28. Step 6 of -# docs/runbooks/fit-the-ups-battery.md. +# 2026-09-11 for a biweekly schedule last tested on 2026-08-28. As of 2026-09-06 +# it has not moved yet, which is expected and not yet evidence either way. +# UpsSelfTestStale is what will say so if it does not. # # Do not run self-tests by hand to keep the metric warm. A test transfers the # load to battery, and on a pack that is weak that is the outage the alert @@ -232,3 +232,88 @@ groups: way — scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2 — and if the schedule is off, turn it on rather than running tests by hand. + + - alert: UpsSelfTestScheduleOff + # The control #93 set, watched. Nothing above can see the schedule; this + # is the only rule that does, and it is the fast half of the pair. + # + # 1 unknown, 4 atTurnOn, 5 never — the three values that mean no + # recurring test. The full PowerNet enumeration is unknown(1), + # biweekly(2), weekly(3), atTurnOn(4), never(5), fourWeeks(6), + # twelveWeeks(7), biweeklySinceLastTest(8), weeklySinceLastTest(9), + # eightWeeks(10), twentysixWeeks(11), fiftytwoWeeks(12), read out of the + # MIB rather than assumed. An older enumeration stopping at 7 is in + # circulation and would read this card's 8 as out of range — a rule + # built on that would be wrong about the one value the card holds. + # + # 10, 11 and 12 are deliberately NOT here. They are real schedules, just + # long ones, and a card set to fifty-two weeks is caught by + # UpsSelfTestStale below rather than by this — which is the right + # division, because "not recurring at all" and "recurring too slowly to + # be a control" are different findings with different fixes. + # Written as three comparisons rather than a regex: `=~` is a LABEL + # matcher in PromQL and does not apply to a sample's value, so + # `metric =~ 1|4|5` is a parse error rather than the shorthand it looks + # like. + expr: | + upsAdvTestDiagnosticSchedule == 1 + or upsAdvTestDiagnosticSchedule == 4 + or upsAdvTestDiagnosticSchedule == 5 + for: 15m + labels: + component: ups + severity: warning + category: power + annotations: + summary: "{{ $labels.device }} is no longer scheduled to self-test" + description: >- + The card's automatic battery test schedule reads {{ $value }} — + unknown, at-turn-on or never — so no recurring test is coming. + upsTestResultsSummary will hold its last result indefinitely and + every other rule in this file will stay quiet. Set it back to + biweekly in the NMC; see docs/runbooks/fit-the-ups-battery.md. + + - alert: UpsSelfTestStale + # The backstop: a card that says it is scheduled and is not. + # + # NO DATE PARSING, and none is needed. upsAdvTestLastDiagnosticsDate is a + # DisplayString, which snmp_exporter renders as a value-1 gauge carrying + # the string as a label — verified against the live card, which returns + # + # upsAdvTestLastDiagnosticsDate{upsAdvTestLastDiagnosticsDate="08/28/2026"} 1 + # + # so each distinct date is a distinct series and counting series over a + # range counts TESTS. One distinct date in 21 days means the date has not + # changed in 21 days. + # + # 21 = the fortnightly schedule plus a week of grace, so a test that + # slips does not page. + # + # Expressed against the stored series rather than as a long `for:`, for + # the reason UpsBatteryUnproven states: a `for: 21d` restarts its clock + # whenever Prometheus does, so it would never fire on a box that is + # redeployed more often than three weeks. This reads history instead. + # + # GATED ON THE SCRAPE BEING UP, which is the trap in a rule shaped like + # this: a dead exporter produces no new series either, and would + # otherwise read as a card that stopped testing. `up` for the snmp job is + # the discriminator, and it makes the rule silent about a fault + # SnmpTargetDown already owns. + expr: | + count by (device) (last_over_time(upsAdvTestLastDiagnosticsDate[21d])) == 1 + and on(device) (up{job="snmp"} == 1) + labels: + component: ups + severity: warning + category: power + annotations: + summary: "{{ $labels.device }} has not completed a self-test in 21 days" + description: >- + upsAdvTestLastDiagnosticsDate has reported one distinct date for + three weeks against a fortnightly schedule, so the card is not + testing even if it says it is. That makes upsTestResultsSummary a + stale pass rather than a fresh one. Read the card with + `scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2`. + Do NOT run a test by hand to clear this — a test transfers the load + to battery, which on a weak pack is the outage the alert exists to + warn about. diff --git a/stacks/observability/prometheus/tests/ups.test.yaml b/stacks/observability/prometheus/tests/ups.test.yaml new file mode 100644 index 00000000..72c83a8a --- /dev/null +++ b/stacks/observability/prometheus/tests/ups.test.yaml @@ -0,0 +1,176 @@ +--- +# Unit tests for ups.rules.yaml. +# +# This file did not exist before #249. Nine rules were validated for syntax and +# nothing else, which is the standing #63 objected to — `promtool check rules` +# passes a rule that can never be true, and the two rules added by #249 are both +# in that risky shape: one compares a sample's value against an enumeration, and +# the other counts series over a range instead of using `for:`. +# +# The pairing is the point, and it is the argument containers.test.yaml makes at +# length: a case that only ever expects silence would pass against a rule that +# can never fire. Every rule below gets a firing case AND the production input, +# because the production input is what a broken rule is most likely to be +# accidentally right about. +# +# Only the two new rules are covered. The other nine are unchanged by #249 and +# are still syntax-only — stated here rather than left for someone to discover, +# the way stack.test.yaml names its own gaps. +rule_files: + - ../rules/ups.rules.yaml + +evaluation_interval: 1m + +tests: + # --- UpsSelfTestScheduleOff: fires on never(5) ---------------------------- + # The failure #249 exists for: a card that reverts to never after a config + # reset or a firmware update. Everything else in ups.rules.yaml stays quiet + # through this, because upsTestResultsSummary keeps reporting the last pass. + - interval: 1m + input_series: + - series: 'upsAdvTestDiagnosticSchedule{device="mjolnir",instance="10.0.99.10",job="snmp"}' + values: "5x30" + alert_rule_test: + - eval_time: 20m + alertname: UpsSelfTestScheduleOff + exp_alerts: + - exp_labels: + device: mjolnir + instance: 10.0.99.10 + job: snmp + component: ups + severity: warning + category: power + exp_annotations: + summary: "mjolnir is no longer scheduled to self-test" + description: >- + The card's automatic battery test schedule reads 5 — unknown, + at-turn-on or never — so no recurring test is coming. + upsTestResultsSummary will hold its last result indefinitely and + every other rule in this file will stay quiet. Set it back to + biweekly in the NMC; see docs/runbooks/fit-the-ups-battery.md. + + # --- UpsSelfTestScheduleOff: fires on atTurnOn(4) too --------------------- + # Not a duplicate of the case above. The rule is three `or`ed comparisons + # rather than a set membership, so each arm can be wrong independently — and 4 + # is the arm most easily lost, sitting between two that look more obviously + # like "off". + - interval: 1m + input_series: + - series: 'upsAdvTestDiagnosticSchedule{device="mjolnir",instance="10.0.99.10",job="snmp"}' + values: "4x30" + alert_rule_test: + - eval_time: 20m + alertname: UpsSelfTestScheduleOff + exp_alerts: + - exp_labels: + device: mjolnir + instance: 10.0.99.10 + job: snmp + component: ups + severity: warning + category: power + exp_annotations: + summary: "mjolnir is no longer scheduled to self-test" + description: >- + The card's automatic battery test schedule reads 4 — unknown, + at-turn-on or never — so no recurring test is coming. + upsTestResultsSummary will hold its last result indefinitely and + every other rule in this file will stay quiet. Set it back to + biweekly in the NMC; see docs/runbooks/fit-the-ups-battery.md. + + # --- UpsSelfTestScheduleOff: quiet on the value the card actually holds ---- + # 8 is biweeklySinceLastTest, read off mjolnir on 2026-09-06. This is the case + # that catches the enumeration mistake #249 warns about: an older PowerNet + # enumeration stops at twelveWeeks(7), and a rule written against it would + # treat 8 as out of range. A rule that fired for the estate's real, correct + # setting is worse than no rule. + - interval: 1m + input_series: + - series: 'upsAdvTestDiagnosticSchedule{device="mjolnir",instance="10.0.99.10",job="snmp"}' + values: "8x30" + alert_rule_test: + - eval_time: 20m + alertname: UpsSelfTestScheduleOff + exp_alerts: [] + + # --- UpsSelfTestScheduleOff: quiet on the long schedules ------------------ + # 12 is fiftytwoWeeks. Deliberately not in the rule: it is a real schedule, + # just a useless one, and UpsSelfTestStale is what says so. This fixture is + # what stops someone "tidying" the expression into `<= 5 or >= 10`. + - interval: 1m + input_series: + - series: 'upsAdvTestDiagnosticSchedule{device="mjolnir",instance="10.0.99.10",job="snmp"}' + values: "12x30" + alert_rule_test: + - eval_time: 20m + alertname: UpsSelfTestScheduleOff + exp_alerts: [] + + # --- UpsSelfTestStale: fires when the date has not moved in 21 days ------- + # One distinct date across the range, with the scrape up. The date lives in a + # label because snmp_exporter renders a DisplayString as a value-1 gauge, so + # "the date changed" is "a second series appeared" — which is what makes this + # countable without parsing anything. + # + # 22 days of samples at 30m so the 21d range is fully covered. + - interval: 30m + input_series: + - series: 'upsAdvTestLastDiagnosticsDate{device="mjolnir",instance="10.0.99.10",job="snmp",upsAdvTestLastDiagnosticsDate="08/28/2026"}' + values: "1x1056" + - series: 'up{device="mjolnir",instance="10.0.99.10",job="snmp"}' + values: "1x1056" + alert_rule_test: + - eval_time: 528h + alertname: UpsSelfTestStale + exp_alerts: + - exp_labels: + device: mjolnir + component: ups + severity: warning + category: power + exp_annotations: + summary: "mjolnir has not completed a self-test in 21 days" + description: >- + upsAdvTestLastDiagnosticsDate has reported one distinct date for + three weeks against a fortnightly schedule, so the card is not + testing even if it says it is. That makes upsTestResultsSummary + a stale pass rather than a fresh one. Read the card with + `scripts/snmp-walk.sh --device mjolnir + 1.3.6.1.4.1.318.1.1.1.7.2`. Do NOT run a test by hand to clear + this — a test transfers the load to battery, which on a weak + pack is the outage the alert exists to warn about. + + # --- UpsSelfTestStale: quiet when a test has happened --------------------- + # Two distinct dates inside the range, which is a card testing normally. The + # production state, and the proof the rule is counting dates rather than + # merely asserting the series exists. + - interval: 30m + input_series: + - series: 'upsAdvTestLastDiagnosticsDate{device="mjolnir",instance="10.0.99.10",job="snmp",upsAdvTestLastDiagnosticsDate="08/28/2026"}' + values: "1x528" + - series: 'upsAdvTestLastDiagnosticsDate{device="mjolnir",instance="10.0.99.10",job="snmp",upsAdvTestLastDiagnosticsDate="09/11/2026"}' + values: "_x528 1x528" + - series: 'up{device="mjolnir",instance="10.0.99.10",job="snmp"}' + values: "1x1056" + alert_rule_test: + - eval_time: 528h + alertname: UpsSelfTestStale + exp_alerts: [] + + # --- UpsSelfTestStale: quiet when the exporter is down ------------------- + # THE TRAP THIS RULE IS MOST LIKELY TO FALL INTO. A dead snmp target produces + # no new date series either, so without the `up` gate this rule would report a + # card that stopped testing whenever the exporter stopped scraping — a true + # alert about the wrong thing, raised against the wrong runbook. + # SnmpTargetDown owns that failure; this must stay silent for it. + - interval: 30m + input_series: + - series: 'upsAdvTestLastDiagnosticsDate{device="mjolnir",instance="10.0.99.10",job="snmp",upsAdvTestLastDiagnosticsDate="08/28/2026"}' + values: "1x1056" + - series: 'up{device="mjolnir",instance="10.0.99.10",job="snmp"}' + values: "0x1056" + alert_rule_test: + - eval_time: 528h + alertname: UpsSelfTestStale + exp_alerts: [] diff --git a/stacks/observability/snmp-exporter/generator.yaml b/stacks/observability/snmp-exporter/generator.yaml index 0ec12933..1c235032 100644 --- a/stacks/observability/snmp-exporter/generator.yaml +++ b/stacks/observability/snmp-exporter/generator.yaml @@ -48,6 +48,37 @@ modules: apc_ups: walk: - 1.3.6.1.2.1.33.1 # UPS-MIB::upsObjects + # APC's own tree, for the one thing UPS-MIB has no concept of: the self-test + # SCHEDULE. UPS-MIB carries upsTestResultsSummary — the LAST result — and + # nothing that says another test is ever coming. #93 set this card to test + # itself every fortnight and #249 found that the control keeping that + # result meaningful was itself unmonitored: if the card drops back to + # never(5) after a config reset or a firmware update, upsTestResultsSummary + # holds donePass forever and every rule in ups.rules.yaml stays correctly + # quiet. UpsBatteryUnproven cannot catch it either — it matches + # noTestsInitiated(6), and a card that has stopped testing still reads 1. + # + # ONE SUBTREE, NOT THE ENTERPRISE ROOT. 1.3.6.1.4.1.318 is enormous and + # walking it is the pfTablesAddrTable mistake one vendor along; the pfsense + # module above descends into .1.9 rather than taking it whole for exactly + # this reason. .7.2 is the diagnostics group: 7 rows, 2 GETBULKs, 0.07s + # measured against the live card on 2026-09-06. + # + # Read that day, and matching what #249 recorded three days earlier: + # + # .7.2.1.0 upsAdvTestDiagnosticSchedule 8 biweeklySinceLastTest + # .7.2.3.0 upsAdvTestDiagnosticsResults 1 ok + # .7.2.4.0 upsAdvTestLastDiagnosticsDate "08/28/2026" + # + # The enumeration matters and is the one thing easy to get wrong here. + # PowerNet's own definition runs unknown(1), biweekly(2), weekly(3), + # atTurnOn(4), never(5), fourWeeks(6), twelveWeeks(7), + # biweeklySinceLastTest(8), weeklySinceLastTest(9), eightWeeks(10), + # twentysixWeeks(11), fiftytwoWeeks(12) — read out of the MIB rather than + # assumed. An older enumeration stopping at 7 would read this card's 8 as + # out of range, and a rule built on that would be wrong about the one + # value the card actually holds. + - 1.3.6.1.4.1.318.1.1.1.7.2 # PowerNet-MIB::upsAdvTest (diagnostics) # pfSense (morpheus, 10.0.99.1) — BEGEMOT-PF-MIB, plus one HOST-RESOURCES # table. Packet filter counters, state table usage, per-interface pf diff --git a/stacks/observability/snmp-exporter/snmp.yaml b/stacks/observability/snmp-exporter/snmp.yaml index 7c5ace92..42a1cb61 100644 --- a/stacks/observability/snmp-exporter/snmp.yaml +++ b/stacks/observability/snmp-exporter/snmp.yaml @@ -28,6 +28,7 @@ modules: apc_ups: walk: - 1.3.6.1.2.1.33.1 + - 1.3.6.1.4.1.318.1.1.1.7.2 metrics: - name: upsIdentManufacturer oid: 1.3.6.1.2.1.33.1.1.1 @@ -378,6 +379,83 @@ modules: type: gauge help: The maximum line voltage allowed before the UPS system transfers to battery backup. - 1.3.6.1.2.1.33.1.9.10 + - name: upsAdvTestDiagnosticSchedule + oid: 1.3.6.1.4.1.318.1.1.1.7.2.1 + type: gauge + help: The UPS system's automatic battery test schedule. - 1.3.6.1.4.1.318.1.1.1.7.2.1 + enum_values: + 1: unknown + 2: biweekly + 3: weekly + 4: atTurnOn + 5: never + 6: fourWeeks + 7: twelveWeeks + 8: biweeklySinceLastTest + 9: weeklySinceLastTest + 10: eightWeeks + 11: twentysixWeeks + 12: fiftytwoWeeks + - name: upsAdvTestDiagnostics + oid: 1.3.6.1.4.1.318.1.1.1.7.2.2 + type: gauge + help: Setting this variable to testDiagnostics(2) causes the UPS to perform + a diagnostic self test - 1.3.6.1.4.1.318.1.1.1.7.2.2 + enum_values: + 1: noTestDiagnostics + 2: testDiagnostics + - name: upsAdvTestDiagnosticsResults + oid: 1.3.6.1.4.1.318.1.1.1.7.2.3 + type: gauge + help: The results of the last UPS diagnostics test performed. - 1.3.6.1.4.1.318.1.1.1.7.2.3 + enum_values: + 1: ok + 2: failed + 3: invalidTest + 4: testInProgress + - name: upsAdvTestLastDiagnosticsDate + oid: 1.3.6.1.4.1.318.1.1.1.7.2.4 + type: DisplayString + help: The date the last UPS diagnostics test was performed in mm/dd/yy format. + - 1.3.6.1.4.1.318.1.1.1.7.2.4 + - name: upsAdvTestRuntimeCalibration + oid: 1.3.6.1.4.1.318.1.1.1.7.2.5 + type: gauge + help: Setting this variable to performCalibration(2) causes the UPS to discharge + to calibrate the UPS - 1.3.6.1.4.1.318.1.1.1.7.2.5 + enum_values: + 1: noPerformCalibration + 2: performCalibration + 3: cancelCurrentCalibration + - name: upsAdvTestCalibrationResults + oid: 1.3.6.1.4.1.318.1.1.1.7.2.6 + type: gauge + help: The results of the last runtime calibration - 1.3.6.1.4.1.318.1.1.1.7.2.6 + enum_values: + 1: ok + 2: invalidCalibration + 3: calibrationInProgress + - name: upsAdvTestCalibrationDate + oid: 1.3.6.1.4.1.318.1.1.1.7.2.7 + type: DisplayString + help: The date the last UPS runtime calibration was performed in mm/dd/yy format. + - 1.3.6.1.4.1.318.1.1.1.7.2.7 + - name: upsAdvTestDiagnosticTime + oid: 1.3.6.1.4.1.318.1.1.1.7.2.8 + type: DisplayString + help: The time of the day to perform automatic battery test - 1.3.6.1.4.1.318.1.1.1.7.2.8 + - name: upsAdvTestDiagnosticDay + oid: 1.3.6.1.4.1.318.1.1.1.7.2.9 + type: gauge + help: The day of the week to perform automatic battery test. - 1.3.6.1.4.1.318.1.1.1.7.2.9 + enum_values: + 1: monday + 2: tuesday + 3: wednesday + 4: thursday + 5: friday + 6: saturday + 7: sunday ilo: walk: - 1.3.6.1.2.1.1