From a3566b36772da182a2fb74e665d015a85d35e261 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Thu, 3 Sep 2026 17:50:24 +0000 Subject: [PATCH 1/3] docs(ups): record that the card is already testing on a schedule (#93) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 6 of the fit-the-ups-battery runbook — enable scheduled self-tests on the NMC — turned out to need checking rather than doing. Read off mjolnir on 2026-09-03 with scripts/snmp-walk.sh: upsAdvTestDiagnosticSchedule 8 (biweeklySinceLastTest) upsAdvTestDiagnosticsResults 1 (ok) upsAdvTestLastDiagnosticsDate "08/28/2026" Six files had said the opposite since the pack went in. They now say what the device says, and #93 moves into Done. They also say the part that does not go away: all three are PowerNet OIDs, the apc_ups module walks the standard UPS-MIB only, and so a card that reverts to never(5) produces no alert and no changed metric — upsTestResultsSummary would hold 1 forever. Closing that means adding APC's MIB to snmp-mibs.sh, a new vendor source with its own pinning decision, and it is deliberately not done here. Until it is, step 6 is verified by hand, and what proves the schedule runs rather than merely being set is that date advancing unattended, due around 2026-09-11. Two findings on the way. upsBasicBatteryLastReplaceDate still reads 08/15/2026 for a pack fitted on the 28th, so the card's battery-age accounting is keyed to a date on which its own self-test was still aborting over an empty bay; the runbook and hardware.md carry it as an outstanding human step at the web UI. And the step 5 comparison was overstated everywhere it appeared: since the fit, upsEstimatedMinutesRemaining has sat on exactly 63 — the fabricated value — for 744 of 764 samples, so runtime cannot discriminate a real pack from an invented one. Voltage moving across 540-549 can, and that is now what the runbook, the rules header and the dashboard banner tell you to read. Co-Authored-By: Claude Opus 5 --- README.md | 19 ++- docs/hardware.md | 8 +- docs/roadmap.md | 48 ++++++-- docs/runbooks/fit-the-ups-battery.md | 110 +++++++++++++++--- docs/security.md | 18 +-- .../grafana/dashboards/ups-power.json | 4 +- .../prometheus/rules/ups.rules.yaml | 59 +++++++--- 7 files changed, 200 insertions(+), 66 deletions(-) diff --git a/README.md b/README.md index 1987df71..20a347ca 100644 --- a/README.md +++ b/README.md @@ -256,7 +256,7 @@ input and output voltage and runtime, under the banner that stood before a battery pack was fitted, when every battery figure was fabricated.](docs/images/ups-power.png) That capture is from 2026-08-22, before a pack was fitted on 2026-08-28. The -banner it shows has been rewritten twice since, and the self-test has now +banner it shows has been rewritten three times since, and the self-test has now passed, so the panels read measured values rather than invented ones — re-shoot the set with `make screenshots`. @@ -297,15 +297,14 @@ Open work is tracked in [`docs/roadmap.md`](docs/roadmap.md) is the narrative — what is outstanding and why it is in that order. -The current top items: put the UPS battery under scheduled test and rack the -shelf switch. The pack went in on 2026-08-28 and passed its self-test, so the -rack itself is protected — but the card is not testing on a schedule, which -leaves that pass a last-known result rather than a live one -([#93](https://github.com/Gerrrt/HomeLab/issues/93)); the switch between the -monitoring host and the network still has no battery at all, so both laptops -stay running and go deaf on a mains cut -([#110](https://github.com/Gerrrt/HomeLab/issues/110)); and get the firewall -backup off the machine it protects, and buy the spare that turns its restore +The current top items: rack the shelf switch, and get the firewall backup off +the machine it protects. The UPS is finished — a pack went into `mjolnir` on +2026-08-28, passed its self-test, and the card is set to test itself every +fortnight ([#93](https://github.com/Gerrrt/HomeLab/issues/93)) — but the switch +between the monitoring host and the network still has no battery at all, so both +laptops stay running and go deaf on a mains cut +([#110](https://github.com/Gerrrt/HomeLab/issues/110)); and the backup still +sits on the machine it protects, without the spare that turns its restore runbook from a hypothesis into something rehearsed ([#92](https://github.com/Gerrrt/HomeLab/issues/92)). diff --git a/docs/hardware.md b/docs/hardware.md index 9f5392ad..85c9a65f 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -61,9 +61,11 @@ revisions of this repository treated `shiva` as the hypervisor itself. - 1U rackmount tray for the ProDesk Mini[^ProDeskRackmount] - Sliding rails for the ProLiant[^Sliderail] - 1U universal rack mount for the APC[^Rail] -- APCRBC115 replacement battery cartridge for the APC, fitted 2026-08-28. - Scheduled self-tests on the card are still outstanding - ([#93](https://github.com/Gerrrt/HomeLab/issues/93)) +- APCRBC115 replacement battery cartridge for the APC, fitted 2026-08-28, + proven by a passing self-test the same day and under a biweekly schedule on + the card ([#93](https://github.com/Gerrrt/HomeLab/issues/93)). The card's + `upsBasicBatteryLastReplaceDate` still reads `08/15/2026` and wants resetting + to the fit date — it is the only record of the pack's age - 1U vented rack shelf, 4-post with square-hole mounting — on hand, for U4 and the unmanaged switch that feeds `prometheus` and `oracle`. It is not in the rack table above because it is not yet in the rack diff --git a/docs/roadmap.md b/docs/roadmap.md index 60dcff99..f5f42451 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -128,17 +128,6 @@ what left this one unfireable for months. protect. Unlike the firewall, they have now been restored — the whole stack was brought up on a restored set on 2026-08-29 and verified. Getting a copy off this host is the part that is still missing. -- **[#93](https://github.com/Gerrrt/HomeLab/issues/93) Replace the UPS battery.** - An APCRBC115 went into `mjolnir` on 2026-08-28 and passed its self-test the - same day: `upsTestResultsSummary` `4` → `1`, `upsBatteryVoltage` off its - fabricated `480`, runtime no longer pinned to exactly `63`. The - `UpsSelfTestFailed` silence was deleted rather than left to expire in - September, so the rule that would report a bad pack is live again. **What is - left is the last step: scheduled self-tests on the card.** Until they are on, - `1` is a last-known result with nothing refreshing it, and `UpsBatteryUnproven` - cannot detect a card that has quietly stopped testing — it matches `6` - (noTestsInitiated), and this one reads `1`. - → [runbook](runbooks/fit-the-ups-battery.md) - **[#110](https://github.com/Gerrrt/HomeLab/issues/110) Rack the shelf switch.** A 1U vented shelf in **U4**, carrying the unmanaged switch `prometheus` and `oracle` hang off. Both shelf machines are laptops, so on a mains cut they stay @@ -201,6 +190,43 @@ months. ## Done +- [x] **[#93](https://github.com/Gerrrt/HomeLab/issues/93) Replace the UPS + battery, delete the silence, put the card under scheduled test.** + 2026-09-03. The first two steps were done on 2026-08-28: an APCRBC115 into + `mjolnir`, `upsTestResultsSummary` `4` (aborted) → `1` (donePass) at + 22:45 UTC, and the `UpsSelfTestFailed` silence + `54f1715c-e57b-4322-8a6d-5435bc8e1bd8` deleted at 23:14 rather than left + to lapse on 2026-09-20 — nine minutes after the proving reading instead + of before it, which is the inversion the runbook exists to prevent and + which cost nothing only because the test passed. + + The third step turned out to need checking rather than doing. The card + reads `upsAdvTestDiagnosticSchedule` `8` (biweeklySinceLastTest), with + `upsAdvTestDiagnosticsResults` `1` (ok) and + `upsAdvTestLastDiagnosticsDate` `08/28/2026`. Six files had been asserting + the opposite since the fit; they now say what the device says. Whether the + schedule was set at the rack or has been the default all along, the NMC + will not say after the fact. + + **Nothing watches it.** All three are PowerNet OIDs and the `apc_ups` + module walks the standard UPS-MIB only, so a card that reverts to + `never(5)` produces no alert and no changed metric — `upsTestResultsSummary` + would simply hold `1` forever. Closing that means adding APC's MIB to + `scripts/snmp-mibs.sh`, which is a new vendor source with its own pinning + decision, and it is not done. Until it is, the check is + `scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2`, and + what proves the schedule *runs* rather than merely being set is that date + advancing unattended, due around 2026-09-11. + + Two smaller findings. `upsBasicBatteryLastReplaceDate` still reads + `08/15/2026` for a pack fitted on the 28th, so the card's battery-age + accounting is keyed to a date on which its own self-test was still + aborting over an empty bay. And runtime is a poor proof of a real pack on + this UPS: since the fit it has sat on exactly `63` — the fabricated + value — for 744 of 764 samples. Voltage moving across `540`-`549` is the + comparison that actually discriminates. + → [runbook](runbooks/fit-the-ups-battery.md) + - [x] **[#90](https://github.com/Gerrrt/HomeLab/issues/90) Detect Suricata being dead.** 2026-09-03. The roadmap line said the SNMP module does not expose a heartbeat; the firewall's agent already did. `bsnmpd` on diff --git a/docs/runbooks/fit-the-ups-battery.md b/docs/runbooks/fit-the-ups-battery.md index 71cbd7fb..f6775e21 100644 --- a/docs/runbooks/fit-the-ups-battery.md +++ b/docs/runbooks/fit-the-ups-battery.md @@ -3,18 +3,41 @@ **One rack visit, in an order that matters — and one alert that has to be un-silenced by hand at the right moment.** -> **Status — 2026-08-28: the pack is in and proven. The shelf is not.** +> **Status — 2026-09-03: the pack is in, proven, and testing on a schedule. +> The shelf is not racked.** > > The APCRBC115 was fitted, the silence was deleted, and a self-test passed: > `upsTestResultsSummary` went `4` (aborted) to `1` (donePass) at 22:45 UTC. -> Step 5's comparison holds — `upsBatteryVoltage` left its fabricated `480` for -> a float reading that varies, and the runtime estimate no longer sits on -> exactly `63`. Charge still reads exactly `100`, which is what a pack at float -> voltage looks like rather than a card that cannot see one. +> Step 5's comparison holds on one metric of the three — `upsBatteryVoltage` +> left its fabricated `480` for a float reading that moves across `540`–`549`. +> Runtime does not carry that weight: it has sat on exactly `63`, the fabricated +> value, for 744 of 764 samples since the fit. Charge still reads exactly `100`, +> which is what a pack at float voltage looks like rather than a card that +> cannot see one. > -> **Step 6 is outstanding.** Scheduled self-tests are not enabled on the NMC, so -> `1` is a last-known result with nothing refreshing it, and nothing in -> `ups.rules.yaml` detects a card that has quietly stopped testing. +> **Step 6 is done — checked 2026-09-03, and it was already true when checked.** +> The card reads `upsAdvTestDiagnosticSchedule` `8` (biweeklySinceLastTest), +> `upsAdvTestLastDiagnosticsDate` `08/28/2026` and +> `upsAdvTestDiagnosticsResults` `1` (ok). Whether the schedule was set during +> the rack visit or has been the card's default all along is not something the +> NMC will tell you after the fact; what is checkable is that it is on now. The +> next unattended test falls due on or about **2026-09-11**, and it is that date +> advancing on its own — not the setting reading `8` — that proves the schedule +> actually runs. +> +> **Nothing in the stack can see any of that.** All three are PowerNet OIDs +> under `1.3.6.1.4.1.318`, and the `apc_ups` module walks the standard UPS-MIB +> only, so no rule in `ups.rules.yaml` can detect a card that has quietly +> stopped testing. Until those OIDs are scraped, step 6 is verified by hand with +> `scripts/snmp-walk.sh` and by nothing else. +> +> **One reading that was not reset.** `upsBasicBatteryLastReplaceDate` is +> `08/15/2026` — thirteen days before the pack went in, on a date when this +> card's own self-test was still aborting over an empty bay. Step 2, item 5 asks +> for that date to come back reset and it did not, so the card's battery-age +> accounting, and the replacement date it projects from it, are keyed to +> something that did not happen. Harmless today; wrong in four years, when it is +> the only record of how old the pack is. > > The shelf and the switch move (step 2, items 1–4) were not done, so > [#110](https://github.com/Gerrrt/HomeLab/issues/110) is untouched and @@ -200,6 +223,15 @@ fabricating. Treat that as "the pack is not seen", not as "the pack is fine and the numbers happen to match" — go back to seating and the battery-replacement date before believing anything else on the dashboard. +**Read voltage first, and do not read too much into runtime.** Measured over the +six days after this pack went in, `upsBatteryVoltage` moved across `540`–`549` +and `upsEstimatedMinutesRemaining` sat on exactly `63` for 744 of 764 samples, +leaving it only for excursions to `45`, `60` and `137` under changing load. `63` +is also the number the card fabricated over an empty bay, so on any single +reading runtime cannot tell the two apart — a float voltage that varies can, and +charge cannot either, because it has read exactly `100` on every sample since +the fit. + ## 6. Enable scheduled self-tests A value of `1` is the *last* result, not a fresh one. If the NMC stops testing @@ -217,6 +249,43 @@ Do not go on running self-tests by hand to keep the metric warm. A test transfers the load to battery, and on a pack that is weak that is the outage the alert exists to warn about. +### Check it rather than assume it — and the check is not in Prometheus + +The setting lives in PowerNet, not in the UPS-MIB the `apc_ups` module walks, so +no Prometheus query can answer this. Read it off the card: + +```bash +scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2 +``` + +Three of those rows are the answer: + +| OID | Object | On `mjolnir`, 2026-09-03 | +| --- | --- | --- | +| `…7.2.1.0` | `upsAdvTestDiagnosticSchedule` | `8` — biweeklySinceLastTest | +| `…7.2.3.0` | `upsAdvTestDiagnosticsResults` | `1` — ok | +| `…7.2.4.0` | `upsAdvTestLastDiagnosticsDate` | `08/28/2026` | + +`upsAdvTestDiagnosticSchedule` reads `1` unknown, `2` biweekly, `3` weekly, `4` +atTurnOn, `5` never, `6` fourWeeks, `7` twelveWeeks, `8` biweeklySinceLastTest, +`9` weeklySinceLastTest — from APC's own PowerNet-MIB, not from the older +enumeration that stops at `7`. Anything in `1`, `4` or `5` means the card is not +testing on a schedule whatever the web UI last appeared to say. The two +`SinceLastTest` values are the ones to prefer: they count from the last test +rather than from power-on, so a reboot does not restart the clock. + +**The setting being on is not the same as the schedule running.** What proves it +is `upsAdvTestLastDiagnosticsDate` advancing with nobody at the card — on a +biweekly schedule, roughly fourteen days after the date it currently shows. Note +the date, come back after it, and read it again. + +Nothing watches any of this on your behalf: these OIDs are not scraped, so a +card that silently drops back to `never` produces no alert and no changed +metric. That gap is named in the header of `ups.rules.yaml`, and closing it +means adding PowerNet to `snmp-exporter/generator.yaml` — a new vendor MIB +source with its own pinning decision, which is why it is not folded into this +runbook. + ## 7. What becomes true afterwards `UpsBatteryLow`, `UpsChargeLow` and `UpsRuntimeCritical` become meaningful rules @@ -240,10 +309,11 @@ to settle before concluding anything. uplink to port 3 of the main switch, not the UPS. Nothing in this runbook touches VLAN configuration. -## Flipping the documents — the first half is done, the second is not +## Flipping the documents — all three passes are done -This happens in two passes, because "a pack is fitted" and "the UPS works" are -different claims and became true at different times. +This happened in three passes, because "a pack is fitted", "the UPS works" and +"the UPS keeps proving it works" are different claims and became true — or were +found to be true — at different times. **Done on 2026-08-28**, when the pack went in: every file that asserted there is no battery said something false from that moment, so they were moved to what was @@ -258,20 +328,26 @@ measured, and that scheduled self-tests are still off. That is `ups-power.json`, `ups.rules.yaml`, `docs/security.md`, `docs/observability.md`, `README.md` and `docs/roadmap.md`. -Two deliberate departures from the plan above. The banner panel was **rewritten +One deliberate departure from the plan above. The banner panel was **rewritten rather than deleted**, because there is still something true and non-obvious for it to say and keeping it left the panel count where it was, so the -`scripts/check_docs.py` panel-count coupling stayed untriggered. And **#93 was -not moved into Done**, because this roadmap entry defines it as *delete the -silence, then self-test, then enable scheduled tests* — the third is step 6, and -it is outstanding. +`scripts/check_docs.py` panel-count coupling stayed untriggered. + +**Done on 2026-09-03**, when step 6 was checked rather than performed: the card +was already set to `biweeklySinceLastTest`, so every file saying scheduled +self-tests are off was asserting something the device contradicts. The same set +moved again — `ups-power.json`, `ups.rules.yaml`, `docs/security.md`, +`README.md`, `docs/hardware.md` and `docs/roadmap.md` — this time to *on, and +unwatched*: the schedule is set, no rule can see it, and the proof is +`upsAdvTestLastDiagnosticsDate` advancing on its own around 2026-09-11. #93 +moved into Done in the same pass, because its third step is now satisfied. **Still to do:** | File | What changes | | --- | --- | | `docs/images/ups-power.png` | Re-capture the set with `make screenshots`. The committed image is from 2026-08-22 and shows the original no-battery banner; it needs the decrypted Grafana password, so it is a human step | -| `docs/roadmap.md` | Move #93 into Done — but only once step 6 is done and the card is testing on a schedule | +| The NMC | Reset `upsBasicBatteryLastReplaceDate`, which still reads `08/15/2026` for a pack fitted on the 28th. A human step at the card's web UI: every SNMP path in this repository reads and none of them writes | And when the shelf is racked, which is a separate visit and separate issue: diff --git a/docs/security.md b/docs/security.md index efbcccd0..aa219b34 100644 --- a/docs/security.md +++ b/docs/security.md @@ -230,13 +230,17 @@ rather than left to expire in September, so that rule is live again. Two things outlast the fix. Stored metrics older than 2026-08-28 *are* the fabricated values rather than measurements, so a dashboard or query whose range -crosses that date is reading fiction on one side of it. And scheduled self-tests -are not yet enabled on the card, which makes `1` a last-known result with -nothing refreshing it — `UpsBatteryUnproven` cannot catch a card that stops -testing, because it matches `6` (noTestsInitiated) and this one now reads `1`. -That is step 6 of -[`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md), and it is -outstanding. +crosses that date is reading fiction on one side of it. And the card's test +schedule is on but unwatched. Read off the NMC on 2026-09-03, +`upsAdvTestDiagnosticSchedule` is `8` (biweeklySinceLastTest), which is what +stops `1` from being a frozen last-known result — but that OID is PowerNet, and +the `apc_ups` module walks the standard UPS-MIB only. Nothing here would notice +the card reverting to `never`: `upsTestResultsSummary` would hold `1` and every +rule would stay quiet. `UpsBatteryUnproven` cannot catch it either, because it +matches `6` (noTestsInitiated) and this card reads `1`. Note the shape of that — +the missing pack was visible in a MIB already walked, and the missing *schedule* +would not be. The manual check, and what closing the gap would cost, are in +[`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md). ### Why SNMPv2c is still a weak point diff --git a/stacks/observability/grafana/dashboards/ups-power.json b/stacks/observability/grafana/dashboards/ups-power.json index 95e167b4..14c1c1b6 100644 --- a/stacks/observability/grafana/dashboards/ups-power.json +++ b/stacks/observability/grafana/dashboards/ups-power.json @@ -1,7 +1,7 @@ { "uid": "homelab-ups", "title": "UPS & Power", - "description": "APC Smart-UPS (mjolnir) via UPS-MIB. An APCRBC115 pack was fitted on 2026-08-28 and passed its self-test the same day, so the battery figures are measured — but anything before that date is fabricated, and scheduled self-tests are not yet enabled on the card. See docs/runbooks/fit-the-ups-battery.md.", + "description": "APC Smart-UPS (mjolnir) via UPS-MIB. An APCRBC115 pack was fitted on 2026-08-28 and passed its self-test the same day, so the battery figures are measured — but anything before that date is fabricated. The card tests itself every fortnight, which nothing on this dashboard can show you: the schedule is a PowerNet OID and this module walks the standard UPS-MIB only. See docs/runbooks/fit-the-ups-battery.md.", "tags": [ "homelab", "power", @@ -267,7 +267,7 @@ }, "options": { "mode": "markdown", - "content": "**The pack is fitted and proven.** An APCRBC115 went into `mjolnir` on 2026-08-28 and `upsTestResultsSummary` moved `4` (aborted) → `1` (donePass) at 22:45 UTC. `upsBatteryVoltage` left its fabricated `480` for a float reading that varies, and the runtime estimate moves instead of sitting on exactly `63` — the step 5 comparison a card cannot fake. Charge still reads exactly `100`, which is what a pack at float voltage looks like.\n\n**Anything on these panels before 2026-08-28 is fabricated.** With an empty bay the card invented a flat 100%, 63 minutes and 48.0 V. The trend panels below cross that boundary — widen the time range with that in mind.\n\n**Scheduled self-tests are not enabled yet.** `1` is the *last* result, not a fresh one; if the NMC never tests again this stops changing and nothing notices. `UpsBatteryUnproven` cannot catch it — that rule matches `6` (noTestsInitiated), and this card now reads `1`. Step 6 of `docs/runbooks/fit-the-ups-battery.md`." + "content": "**The pack is fitted and proven.** An APCRBC115 went into `mjolnir` on 2026-08-28 and `upsTestResultsSummary` moved `4` (aborted) → `1` (donePass) at 22:45 UTC. `upsBatteryVoltage` left its fabricated `480` for a float reading that moves across `540`–`549` — the step 5 comparison a card cannot fake. Runtime is *not* that comparison: it has sat on exactly `63`, the fabricated value, for 744 of 764 samples since the fit. Charge still reads exactly `100`, which is what a pack at float voltage looks like.\n\n**Anything on these panels before 2026-08-28 is fabricated.** With an empty bay the card invented a flat 100%, 63 minutes and 48.0 V. The trend panels below cross that boundary — widen the time range with that in mind.\n\n**The card tests itself every fortnight, and nothing here watches that.** Read off the NMC on 2026-09-03: `upsAdvTestDiagnosticSchedule` `8` (biweeklySinceLastTest), last test `08/28/2026`. Those are PowerNet OIDs and this module walks the standard UPS-MIB only, so if the card reverted to `never`, `upsTestResultsSummary` would hold `1` forever and no panel here would move. `UpsBatteryUnproven` cannot catch it either — it matches `6` (noTestsInitiated). Check by hand with `scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2`; step 6 of `docs/runbooks/fit-the-ups-battery.md`." } }, { diff --git a/stacks/observability/prometheus/rules/ups.rules.yaml b/stacks/observability/prometheus/rules/ups.rules.yaml index 67b094a3..36313178 100644 --- a/stacks/observability/prometheus/rules/ups.rules.yaml +++ b/stacks/observability/prometheus/rules/ups.rules.yaml @@ -3,10 +3,13 @@ # # These rules now read real hardware. An APCRBC115 pack was fitted on 2026-08-28 # and passed its self-test the same day: upsTestResultsSummary went 4 (aborted) -# to 1 (donePass) at 22:45 UTC, upsBatteryVoltage left its fabricated 480 for a -# float reading that varies, and the runtime estimate moves rather than sitting -# on exactly 63. Every rule here is deliberately loud and routed to the `urgent` -# receiver by category=power in alertmanager.yaml. +# to 1 (donePass) at 22:45 UTC, and upsBatteryVoltage left its fabricated 480 +# for a float reading that moves across 540-549. Runtime is the weaker half of +# that comparison and should not be leaned on: measured over the six days after +# the fit it sat on exactly 63 — the fabricated value — for 744 of 764 samples, +# leaving it only for excursions to 45, 60 and 137 under changing load. Every +# rule here is deliberately loud and routed to the `urgent` receiver by +# category=power in alertmanager.yaml. # # WHY THE PRE-FIT BASELINE IS STILL RECORDED HERE. # @@ -27,8 +30,8 @@ # # upsEstimatedChargeRemaining still reads exactly 100. On its own that is the # fabrication signature; here it is consistent with a pack at float voltage, -# because voltage and runtime both moved and the self-test passed. Charge alone -# was never the evidence. +# because voltage moved and the self-test passed. Charge alone was never the +# evidence, and neither, on this UPS, is runtime. # # THE ONE METRIC THAT DOES NOT LIE. # @@ -46,15 +49,38 @@ # 2026-09-20 (54f1715c-e57b-4322-8a6d-5435bc8e1bd8) was deleted on 2026-08-28, # so both rules are live. # -# WHAT IS STILL OUTSTANDING. +# WHAT THIS FILE STILL CANNOT SEE. # -# Scheduled self-tests are not enabled on the NMC. A value of 1 is the *last* -# result, not a fresh one: if the card stops testing the metric simply stops -# changing, and nothing here detects that. UpsBatteryUnproven cannot — it -# matches 6 (noTestsInitiated), and this card now reads 1. upsTestStartTime is +# Scheduled self-tests ARE enabled on the NMC. Read off the card on 2026-09-03: +# +# upsAdvTestDiagnosticSchedule 8 (biweeklySinceLastTest) +# upsAdvTestDiagnosticsResults 1 (ok) +# upsAdvTestLastDiagnosticsDate "08/28/2026" +# +# All three are PowerNet (1.3.6.1.4.1.318.1.1.1.7.2.x), and the apc_ups module +# in snmp-exporter/generator.yaml walks 1.3.6.1.2.1.33.1 — the standard UPS-MIB +# — and nothing else. So none of it reaches Prometheus, and no rule below can +# be written against it. +# +# That leaves one hole open. A value of 1 is the *last* result, not a fresh one: +# if the card stops testing, upsTestResultsSummary simply stops changing, and +# nothing here detects that. UpsBatteryUnproven cannot — it matches 6 +# (noTestsInitiated), and this card reads 1. upsTestStartTime is # sysUpTime-relative and does not survive an agent restart, so it is not a -# usable staleness signal either. Scheduled tests on the device are the control -# and this file cannot substitute for them; step 6 of +# usable staleness signal either. A card that quietly reverts to never(5) — a +# config reset, a firmware update, a restored configuration file — is +# indistinguishable, from inside this file, from one testing every fortnight. +# +# The fix is to walk those three OIDs and alert on the schedule leaving the +# recurring values and on the last-test date going stale, which needs APC's +# PowerNet MIB added to scripts/snmp-mibs.sh and a pinning decision for it. Not +# done here. Until it is, the check is by hand: +# +# scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2 +# +# and the thing that proves the schedule runs, rather than merely being set, is +# upsAdvTestLastDiagnosticsDate advancing with nobody at the card — due around +# 2026-09-11 for a biweekly schedule last tested on 2026-08-28. Step 6 of # docs/runbooks/fit-the-ups-battery.md. # # Do not run self-tests by hand to keep the metric warm. A test transfers the @@ -188,6 +214,7 @@ groups: annotations: summary: "UPS {{ $labels.device }} has not completed a self-test in 7 days" description: >- - The UPS has never proven it can carry the load. Enable scheduled - self-tests on the management card so this becomes live evidence - rather than an absence of it. + The UPS has never proven it can carry the load. The card is set to + biweeklySinceLastTest, so seven days of 6 means the schedule is not + running whatever the setting says — check it with + scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2. From 0e785a6812ec156e28bb344e25cbfc3485a1d7c7 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Thu, 3 Sep 2026 18:33:13 +0000 Subject: [PATCH 2/3] docs(ups): point the self-test-schedule gap at #249 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The four places that describe the gap left by #93 — the runbook banner and step 6, the ups.rules.yaml header, docs/security.md — each said the fix was not done and why. They now say where it is being done, and #249 goes into the roadmap's Monitoring section as open work. Co-Authored-By: Claude Opus 5 --- docs/roadmap.md | 15 ++++++++++++++- docs/runbooks/fit-the-ups-battery.md | 14 ++++++++------ docs/security.md | 6 ++++-- .../observability/prometheus/rules/ups.rules.yaml | 4 ++-- 4 files changed, 28 insertions(+), 11 deletions(-) diff --git a/docs/roadmap.md b/docs/roadmap.md index f5f42451..735baaf6 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -85,6 +85,17 @@ issues intact. Nothing was summarised away. sizeable: `pids_limit`, because tens of threads against a 10,000-thread abort is two orders of magnitude of daylight, and a byte ceiling on the TSDB, which is at a measurable steady state at day 28 of 30. +- **[#249](https://github.com/Gerrrt/HomeLab/issues/249) Scrape the UPS + self-test schedule.** [#93](https://github.com/Gerrrt/HomeLab/issues/93) left + `mjolnir` testing itself every fortnight and nothing able to see that it does. + The schedule, the last-test date and the last result are all PowerNet OIDs and + the `apc_ups` module walks the standard UPS-MIB only, so a card that reverts + to `never(5)` produces no alert and no changed metric — `upsTestResultsSummary` + holds `1` (donePass) forever. The missing pack was visible in a MIB already + walked; the missing schedule is not, which is the same failure one level up. + The real cost is not the two rules but a pinning decision for APC's MIB in + `scripts/snmp-mibs.sh`, which has no first-party git ref to point at. + → [runbook](runbooks/fit-the-ups-battery.md) - **[#12](https://github.com/Gerrrt/HomeLab/issues/12) Capture dashboard screenshots.** `make screenshots` does five of the seven; the Logs and Security dashboards are deliberately excluded. @@ -213,7 +224,9 @@ months. `never(5)` produces no alert and no changed metric — `upsTestResultsSummary` would simply hold `1` forever. Closing that means adding APC's MIB to `scripts/snmp-mibs.sh`, which is a new vendor source with its own pinning - decision, and it is not done. Until it is, the check is + decision; it is filed as + [#249](https://github.com/Gerrrt/HomeLab/issues/249) rather than done + here. Until it is, the check is `scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2`, and what proves the schedule *runs* rather than merely being set is that date advancing unattended, due around 2026-09-11. diff --git a/docs/runbooks/fit-the-ups-battery.md b/docs/runbooks/fit-the-ups-battery.md index f6775e21..55d5aae3 100644 --- a/docs/runbooks/fit-the-ups-battery.md +++ b/docs/runbooks/fit-the-ups-battery.md @@ -28,8 +28,9 @@ un-silenced by hand at the right moment.** > **Nothing in the stack can see any of that.** All three are PowerNet OIDs > under `1.3.6.1.4.1.318`, and the `apc_ups` module walks the standard UPS-MIB > only, so no rule in `ups.rules.yaml` can detect a card that has quietly -> stopped testing. Until those OIDs are scraped, step 6 is verified by hand with -> `scripts/snmp-walk.sh` and by nothing else. +> stopped testing. That is +> [#249](https://github.com/Gerrrt/HomeLab/issues/249); until it lands, step 6 +> is verified by hand with `scripts/snmp-walk.sh` and by nothing else. > > **One reading that was not reset.** `upsBasicBatteryLastReplaceDate` is > `08/15/2026` — thirteen days before the pack went in, on a date when this @@ -281,10 +282,11 @@ the date, come back after it, and read it again. Nothing watches any of this on your behalf: these OIDs are not scraped, so a card that silently drops back to `never` produces no alert and no changed -metric. That gap is named in the header of `ups.rules.yaml`, and closing it -means adding PowerNet to `snmp-exporter/generator.yaml` — a new vendor MIB -source with its own pinning decision, which is why it is not folded into this -runbook. +metric. That gap is named in the header of `ups.rules.yaml` and filed as +[#249](https://github.com/Gerrrt/HomeLab/issues/249) — closing it means adding +PowerNet to `snmp-exporter/generator.yaml`, a new vendor MIB source with its own +pinning decision, which is why it is an issue rather than a step in this +runbook. When it lands, this section becomes a PromQL query. ## 7. What becomes true afterwards diff --git a/docs/security.md b/docs/security.md index aa219b34..18fd2abf 100644 --- a/docs/security.md +++ b/docs/security.md @@ -239,8 +239,10 @@ the card reverting to `never`: `upsTestResultsSummary` would hold `1` and every rule would stay quiet. `UpsBatteryUnproven` cannot catch it either, because it matches `6` (noTestsInitiated) and this card reads `1`. Note the shape of that — the missing pack was visible in a MIB already walked, and the missing *schedule* -would not be. The manual check, and what closing the gap would cost, are in -[`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md). +would not be. The manual check is in +[`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md), and +closing the gap is +[#249](https://github.com/Gerrrt/HomeLab/issues/249). ### Why SNMPv2c is still a weak point diff --git a/stacks/observability/prometheus/rules/ups.rules.yaml b/stacks/observability/prometheus/rules/ups.rules.yaml index 36313178..8482ada3 100644 --- a/stacks/observability/prometheus/rules/ups.rules.yaml +++ b/stacks/observability/prometheus/rules/ups.rules.yaml @@ -73,8 +73,8 @@ # # The fix is to walk those three OIDs and alert on the schedule leaving the # recurring values and on the last-test date going stale, which needs APC's -# PowerNet MIB added to scripts/snmp-mibs.sh and a pinning decision for it. Not -# done here. Until it is, the check is by hand: +# PowerNet MIB added to scripts/snmp-mibs.sh and a pinning decision for it. That +# is issue #249, not this file. Until it lands, the check is by hand: # # scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2 # From 087c91eb7c3bc8095b50808c48a279c7e2cd7ba4 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Thu, 3 Sep 2026 18:38:26 +0000 Subject: [PATCH 3/3] docs(ups): stop the review's three overstatements MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit UpsBatteryUnproven's description claimed seven days at 6 (noTestsInitiated) means the schedule is not running. It does not: the card tests every fourteen days, so one whose counters have just been reset can sit at 6 for a fortnight while the schedule works. The description now says what the rule detects and sends the reader to the schedule, and a comment above the rule records why the window is deliberately shorter than the interval. docs/security.md said the schedule setting "stops" upsTestResultsSummary from freezing, in a paragraph whose whole point is that nothing here can confirm it still does. Now "should keep — should, because nothing here can confirm it". The runbook named only generator.yaml as the cost of closing #249, when the pinning decision lives in scripts/snmp-mibs.sh and make snmp-generate cannot resolve the OIDs until it fetches the MIB. Both named. Co-Authored-By: Claude Opus 5 --- docs/runbooks/fit-the-ups-battery.md | 11 ++++++---- docs/security.md | 3 ++- .../prometheus/rules/ups.rules.yaml | 22 +++++++++++++++---- 3 files changed, 27 insertions(+), 9 deletions(-) diff --git a/docs/runbooks/fit-the-ups-battery.md b/docs/runbooks/fit-the-ups-battery.md index 55d5aae3..2f936871 100644 --- a/docs/runbooks/fit-the-ups-battery.md +++ b/docs/runbooks/fit-the-ups-battery.md @@ -283,10 +283,13 @@ the date, come back after it, and read it again. Nothing watches any of this on your behalf: these OIDs are not scraped, so a card that silently drops back to `never` produces no alert and no changed metric. That gap is named in the header of `ups.rules.yaml` and filed as -[#249](https://github.com/Gerrrt/HomeLab/issues/249) — closing it means adding -PowerNet to `snmp-exporter/generator.yaml`, a new vendor MIB source with its own -pinning decision, which is why it is an issue rather than a step in this -runbook. When it lands, this section becomes a PromQL query. +[#249](https://github.com/Gerrrt/HomeLab/issues/249) — closing it takes two +files, not one: `scripts/snmp-mibs.sh` has to fetch APC's MIB before +`make snmp-generate` can resolve the new OIDs, and that is where the pinning +decision lives, alongside the walk added to `snmp-exporter/generator.yaml`. A +new vendor source with no first-party git ref to point at is why it is an issue +rather than a step in this runbook. When it lands, this section becomes a +PromQL query. ## 7. What becomes true afterwards diff --git a/docs/security.md b/docs/security.md index 6855c7cb..48a31a8e 100644 --- a/docs/security.md +++ b/docs/security.md @@ -233,7 +233,8 @@ fabricated values rather than measurements, so a dashboard or query whose range crosses that date is reading fiction on one side of it. And the card's test schedule is on but unwatched. Read off the NMC on 2026-09-03, `upsAdvTestDiagnosticSchedule` is `8` (biweeklySinceLastTest), which is what -stops `1` from being a frozen last-known result — but that OID is PowerNet, and +*should* keep `1` from being a frozen last-known result — should, because +nothing here can confirm it still does. That OID is PowerNet, and the `apc_ups` module walks the standard UPS-MIB only. Nothing here would notice the card reverting to `never`: `upsTestResultsSummary` would hold `1` and every rule would stay quiet. `UpsBatteryUnproven` cannot catch it either, because it diff --git a/stacks/observability/prometheus/rules/ups.rules.yaml b/stacks/observability/prometheus/rules/ups.rules.yaml index 8482ada3..5ab8775d 100644 --- a/stacks/observability/prometheus/rules/ups.rules.yaml +++ b/stacks/observability/prometheus/rules/ups.rules.yaml @@ -196,6 +196,14 @@ groups: # which a UPS has never demonstrated it can do its job, and it is # indistinguishable from a healthy one on every other metric. # + # The 7d window is shorter than this card's 14d test interval, so a + # card reset to 6 can trip this while its schedule is working. That is + # deliberate — the alert says "unproven for a week", which is worth + # knowing whatever the cause — but it is why the description sends you + # to the schedule rather than asserting it has failed. Widening the + # window to clear the interval would trade a week of blindness for a + # quieter alert, and this rule exists because quiet was the problem. + # # Expressed over a range rather than as `for: 7d`, deliberately. A long # `for` measures continuous pending time in Prometheus's own memory, so # any restart inside the window resets it — on a stack that gets @@ -214,7 +222,13 @@ groups: annotations: summary: "UPS {{ $labels.device }} has not completed a self-test in 7 days" description: >- - The UPS has never proven it can carry the load. The card is set to - biweeklySinceLastTest, so seven days of 6 means the schedule is not - running whatever the setting says — check it with - scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2. + upsTestResultsSummary has read 6 (noTestsInitiated) for seven days, + so the UPS has not proven it can carry the load inside that window. + That is what this rule detects and all it detects. It is not proof + the schedule has failed: this card is set to biweeklySinceLastTest, + a fourteen-day interval, so a card whose counters have just been + reset can sit at 6 for a fortnight while the schedule is working. + Read the schedule and the last-test date before concluding either + way — scripts/snmp-walk.sh --device mjolnir + 1.3.6.1.4.1.318.1.1.1.7.2 — and if the schedule is off, turn it on + rather than running tests by hand.