diff --git a/README.md b/README.md index 7853c424..2dcd70cb 100644 --- a/README.md +++ b/README.md @@ -256,7 +256,7 @@ input and output voltage and runtime, under the banner that stood before a battery pack was fitted, when every battery figure was fabricated.](docs/images/ups-power.png) That capture is from 2026-08-22, before a pack was fitted on 2026-08-28. The -banner it shows has been rewritten twice since, and the self-test has now +banner it shows has been rewritten three times since, and the self-test has now passed, so the panels read measured values rather than invented ones — re-shoot the set with `make screenshots`. @@ -297,17 +297,15 @@ Open work is tracked in [`docs/roadmap.md`](docs/roadmap.md) is the narrative — what is outstanding and why it is in that order. -The current top items: put the UPS battery under scheduled test and rack the -shelf switch. The pack went in on 2026-08-28 and passed its self-test, so the -rack itself is protected — but the card is not testing on a schedule, which -leaves that pass a last-known result rather than a live one -([#93](https://github.com/Gerrrt/HomeLab/issues/93)); the switch between the -monitoring host and the network still has no battery at all, so both laptops +The current top items: rack the shelf switch, and buy the spare ProDesk that +turns the firewall's restore runbook from a hypothesis into something rehearsed. +The UPS is finished — a pack went into `mjolnir` on 2026-08-28, passed its +self-test, and the card is set to test itself every fortnight +([#93](https://github.com/Gerrrt/HomeLab/issues/93)) — but the switch between +the monitoring host and the network still has no battery at all, so both laptops stay running and go deaf on a mains cut -([#110](https://github.com/Gerrrt/HomeLab/issues/110)); and buy the spare -ProDesk that turns the firewall's restore runbook from a hypothesis into -something rehearsed — the config export itself now leaves the monitoring host -nightly, so the spare is what is left +([#110](https://github.com/Gerrrt/HomeLab/issues/110)); and the config export +itself now leaves the monitoring host nightly, so the spare is what is left ([#92](https://github.com/Gerrrt/HomeLab/issues/92)). ## License diff --git a/docs/hardware.md b/docs/hardware.md index 9f5392ad..85c9a65f 100644 --- a/docs/hardware.md +++ b/docs/hardware.md @@ -61,9 +61,11 @@ revisions of this repository treated `shiva` as the hypervisor itself. - 1U rackmount tray for the ProDesk Mini[^ProDeskRackmount] - Sliding rails for the ProLiant[^Sliderail] - 1U universal rack mount for the APC[^Rail] -- APCRBC115 replacement battery cartridge for the APC, fitted 2026-08-28. - Scheduled self-tests on the card are still outstanding - ([#93](https://github.com/Gerrrt/HomeLab/issues/93)) +- APCRBC115 replacement battery cartridge for the APC, fitted 2026-08-28, + proven by a passing self-test the same day and under a biweekly schedule on + the card ([#93](https://github.com/Gerrrt/HomeLab/issues/93)). The card's + `upsBasicBatteryLastReplaceDate` still reads `08/15/2026` and wants resetting + to the fit date — it is the only record of the pack's age - 1U vented rack shelf, 4-post with square-hole mounting — on hand, for U4 and the unmanaged switch that feeds `prometheus` and `oracle`. It is not in the rack table above because it is not yet in the rack diff --git a/docs/roadmap.md b/docs/roadmap.md index 007436f2..91e8cd67 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -75,6 +75,17 @@ issues intact. Nothing was summarised away. sizeable: `pids_limit`, because tens of threads against a 10,000-thread abort is two orders of magnitude of daylight, and a byte ceiling on the TSDB, which is at a measurable steady state at day 28 of 30. +- **[#249](https://github.com/Gerrrt/HomeLab/issues/249) Scrape the UPS + self-test schedule.** [#93](https://github.com/Gerrrt/HomeLab/issues/93) left + `mjolnir` testing itself every fortnight and nothing able to see that it does. + The schedule, the last-test date and the last result are all PowerNet OIDs and + the `apc_ups` module walks the standard UPS-MIB only, so a card that reverts + to `never(5)` produces no alert and no changed metric — `upsTestResultsSummary` + holds `1` (donePass) forever. The missing pack was visible in a MIB already + walked; the missing schedule is not, which is the same failure one level up. + The real cost is not the two rules but a pinning decision for APC's MIB in + `scripts/snmp-mibs.sh`, which has no first-party git ref to point at. + → [runbook](runbooks/fit-the-ups-battery.md) - **[#12](https://github.com/Gerrrt/HomeLab/issues/12) Capture dashboard screenshots.** `make screenshots` does five of the seven; the Logs and Security dashboards are deliberately excluded. @@ -129,17 +140,6 @@ what left this one unfireable for months. on the host they protect. Unlike the firewall, they have been restored — the whole stack was brought up on a restored set on 2026-08-29 and verified — but nothing copies them anywhere. -- **[#93](https://github.com/Gerrrt/HomeLab/issues/93) Replace the UPS battery.** - An APCRBC115 went into `mjolnir` on 2026-08-28 and passed its self-test the - same day: `upsTestResultsSummary` `4` → `1`, `upsBatteryVoltage` off its - fabricated `480`, runtime no longer pinned to exactly `63`. The - `UpsSelfTestFailed` silence was deleted rather than left to expire in - September, so the rule that would report a bad pack is live again. **What is - left is the last step: scheduled self-tests on the card.** Until they are on, - `1` is a last-known result with nothing refreshing it, and `UpsBatteryUnproven` - cannot detect a card that has quietly stopped testing — it matches `6` - (noTestsInitiated), and this one reads `1`. - → [runbook](runbooks/fit-the-ups-battery.md) - **[#110](https://github.com/Gerrrt/HomeLab/issues/110) Rack the shelf switch.** A 1U vented shelf in **U4**, carrying the unmanaged switch `prometheus` and `oracle` hang off. Both shelf machines are laptops, so on a mains cut they stay @@ -205,6 +205,45 @@ months. ## Done +- [x] **[#93](https://github.com/Gerrrt/HomeLab/issues/93) Replace the UPS + battery, delete the silence, put the card under scheduled test.** + 2026-09-03. The first two steps were done on 2026-08-28: an APCRBC115 into + `mjolnir`, `upsTestResultsSummary` `4` (aborted) → `1` (donePass) at + 22:45 UTC, and the `UpsSelfTestFailed` silence + `54f1715c-e57b-4322-8a6d-5435bc8e1bd8` deleted at 23:14 rather than left + to lapse on 2026-09-20 — nine minutes after the proving reading instead + of before it, which is the inversion the runbook exists to prevent and + which cost nothing only because the test passed. + + The third step turned out to need checking rather than doing. The card + reads `upsAdvTestDiagnosticSchedule` `8` (biweeklySinceLastTest), with + `upsAdvTestDiagnosticsResults` `1` (ok) and + `upsAdvTestLastDiagnosticsDate` `08/28/2026`. Six files had been asserting + the opposite since the fit; they now say what the device says. Whether the + schedule was set at the rack or has been the default all along, the NMC + will not say after the fact. + + **Nothing watches it.** All three are PowerNet OIDs and the `apc_ups` + module walks the standard UPS-MIB only, so a card that reverts to + `never(5)` produces no alert and no changed metric — `upsTestResultsSummary` + would simply hold `1` forever. Closing that means adding APC's MIB to + `scripts/snmp-mibs.sh`, which is a new vendor source with its own pinning + decision; it is filed as + [#249](https://github.com/Gerrrt/HomeLab/issues/249) rather than done + here. Until it is, the check is + `scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2`, and + what proves the schedule *runs* rather than merely being set is that date + advancing unattended, due around 2026-09-11. + + Two smaller findings. `upsBasicBatteryLastReplaceDate` still reads + `08/15/2026` for a pack fitted on the 28th, so the card's battery-age + accounting is keyed to a date on which its own self-test was still + aborting over an empty bay. And runtime is a poor proof of a real pack on + this UPS: since the fit it has sat on exactly `63` — the fabricated + value — for 744 of 764 samples. Voltage moving across `540`-`549` is the + comparison that actually discriminates. + → [runbook](runbooks/fit-the-ups-battery.md) + - [x] **[#91](https://github.com/Gerrrt/HomeLab/issues/91) Probe the services this was filed for, and probe TLS expiry.** 2026-09-03. Seven named, five probed, plus the one the sentence about "four devices" implied: Grafana diff --git a/docs/runbooks/fit-the-ups-battery.md b/docs/runbooks/fit-the-ups-battery.md index 71cbd7fb..2f936871 100644 --- a/docs/runbooks/fit-the-ups-battery.md +++ b/docs/runbooks/fit-the-ups-battery.md @@ -3,18 +3,42 @@ **One rack visit, in an order that matters — and one alert that has to be un-silenced by hand at the right moment.** -> **Status — 2026-08-28: the pack is in and proven. The shelf is not.** +> **Status — 2026-09-03: the pack is in, proven, and testing on a schedule. +> The shelf is not racked.** > > The APCRBC115 was fitted, the silence was deleted, and a self-test passed: > `upsTestResultsSummary` went `4` (aborted) to `1` (donePass) at 22:45 UTC. -> Step 5's comparison holds — `upsBatteryVoltage` left its fabricated `480` for -> a float reading that varies, and the runtime estimate no longer sits on -> exactly `63`. Charge still reads exactly `100`, which is what a pack at float -> voltage looks like rather than a card that cannot see one. +> Step 5's comparison holds on one metric of the three — `upsBatteryVoltage` +> left its fabricated `480` for a float reading that moves across `540`–`549`. +> Runtime does not carry that weight: it has sat on exactly `63`, the fabricated +> value, for 744 of 764 samples since the fit. Charge still reads exactly `100`, +> which is what a pack at float voltage looks like rather than a card that +> cannot see one. > -> **Step 6 is outstanding.** Scheduled self-tests are not enabled on the NMC, so -> `1` is a last-known result with nothing refreshing it, and nothing in -> `ups.rules.yaml` detects a card that has quietly stopped testing. +> **Step 6 is done — checked 2026-09-03, and it was already true when checked.** +> The card reads `upsAdvTestDiagnosticSchedule` `8` (biweeklySinceLastTest), +> `upsAdvTestLastDiagnosticsDate` `08/28/2026` and +> `upsAdvTestDiagnosticsResults` `1` (ok). Whether the schedule was set during +> the rack visit or has been the card's default all along is not something the +> NMC will tell you after the fact; what is checkable is that it is on now. The +> next unattended test falls due on or about **2026-09-11**, and it is that date +> advancing on its own — not the setting reading `8` — that proves the schedule +> actually runs. +> +> **Nothing in the stack can see any of that.** All three are PowerNet OIDs +> under `1.3.6.1.4.1.318`, and the `apc_ups` module walks the standard UPS-MIB +> only, so no rule in `ups.rules.yaml` can detect a card that has quietly +> stopped testing. That is +> [#249](https://github.com/Gerrrt/HomeLab/issues/249); until it lands, step 6 +> is verified by hand with `scripts/snmp-walk.sh` and by nothing else. +> +> **One reading that was not reset.** `upsBasicBatteryLastReplaceDate` is +> `08/15/2026` — thirteen days before the pack went in, on a date when this +> card's own self-test was still aborting over an empty bay. Step 2, item 5 asks +> for that date to come back reset and it did not, so the card's battery-age +> accounting, and the replacement date it projects from it, are keyed to +> something that did not happen. Harmless today; wrong in four years, when it is +> the only record of how old the pack is. > > The shelf and the switch move (step 2, items 1–4) were not done, so > [#110](https://github.com/Gerrrt/HomeLab/issues/110) is untouched and @@ -200,6 +224,15 @@ fabricating. Treat that as "the pack is not seen", not as "the pack is fine and the numbers happen to match" — go back to seating and the battery-replacement date before believing anything else on the dashboard. +**Read voltage first, and do not read too much into runtime.** Measured over the +six days after this pack went in, `upsBatteryVoltage` moved across `540`–`549` +and `upsEstimatedMinutesRemaining` sat on exactly `63` for 744 of 764 samples, +leaving it only for excursions to `45`, `60` and `137` under changing load. `63` +is also the number the card fabricated over an empty bay, so on any single +reading runtime cannot tell the two apart — a float voltage that varies can, and +charge cannot either, because it has read exactly `100` on every sample since +the fit. + ## 6. Enable scheduled self-tests A value of `1` is the *last* result, not a fresh one. If the NMC stops testing @@ -217,6 +250,47 @@ Do not go on running self-tests by hand to keep the metric warm. A test transfers the load to battery, and on a pack that is weak that is the outage the alert exists to warn about. +### Check it rather than assume it — and the check is not in Prometheus + +The setting lives in PowerNet, not in the UPS-MIB the `apc_ups` module walks, so +no Prometheus query can answer this. Read it off the card: + +```bash +scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2 +``` + +Three of those rows are the answer: + +| OID | Object | On `mjolnir`, 2026-09-03 | +| --- | --- | --- | +| `…7.2.1.0` | `upsAdvTestDiagnosticSchedule` | `8` — biweeklySinceLastTest | +| `…7.2.3.0` | `upsAdvTestDiagnosticsResults` | `1` — ok | +| `…7.2.4.0` | `upsAdvTestLastDiagnosticsDate` | `08/28/2026` | + +`upsAdvTestDiagnosticSchedule` reads `1` unknown, `2` biweekly, `3` weekly, `4` +atTurnOn, `5` never, `6` fourWeeks, `7` twelveWeeks, `8` biweeklySinceLastTest, +`9` weeklySinceLastTest — from APC's own PowerNet-MIB, not from the older +enumeration that stops at `7`. Anything in `1`, `4` or `5` means the card is not +testing on a schedule whatever the web UI last appeared to say. The two +`SinceLastTest` values are the ones to prefer: they count from the last test +rather than from power-on, so a reboot does not restart the clock. + +**The setting being on is not the same as the schedule running.** What proves it +is `upsAdvTestLastDiagnosticsDate` advancing with nobody at the card — on a +biweekly schedule, roughly fourteen days after the date it currently shows. Note +the date, come back after it, and read it again. + +Nothing watches any of this on your behalf: these OIDs are not scraped, so a +card that silently drops back to `never` produces no alert and no changed +metric. That gap is named in the header of `ups.rules.yaml` and filed as +[#249](https://github.com/Gerrrt/HomeLab/issues/249) — closing it takes two +files, not one: `scripts/snmp-mibs.sh` has to fetch APC's MIB before +`make snmp-generate` can resolve the new OIDs, and that is where the pinning +decision lives, alongside the walk added to `snmp-exporter/generator.yaml`. A +new vendor source with no first-party git ref to point at is why it is an issue +rather than a step in this runbook. When it lands, this section becomes a +PromQL query. + ## 7. What becomes true afterwards `UpsBatteryLow`, `UpsChargeLow` and `UpsRuntimeCritical` become meaningful rules @@ -240,10 +314,11 @@ to settle before concluding anything. uplink to port 3 of the main switch, not the UPS. Nothing in this runbook touches VLAN configuration. -## Flipping the documents — the first half is done, the second is not +## Flipping the documents — all three passes are done -This happens in two passes, because "a pack is fitted" and "the UPS works" are -different claims and became true at different times. +This happened in three passes, because "a pack is fitted", "the UPS works" and +"the UPS keeps proving it works" are different claims and became true — or were +found to be true — at different times. **Done on 2026-08-28**, when the pack went in: every file that asserted there is no battery said something false from that moment, so they were moved to what was @@ -258,20 +333,26 @@ measured, and that scheduled self-tests are still off. That is `ups-power.json`, `ups.rules.yaml`, `docs/security.md`, `docs/observability.md`, `README.md` and `docs/roadmap.md`. -Two deliberate departures from the plan above. The banner panel was **rewritten +One deliberate departure from the plan above. The banner panel was **rewritten rather than deleted**, because there is still something true and non-obvious for it to say and keeping it left the panel count where it was, so the -`scripts/check_docs.py` panel-count coupling stayed untriggered. And **#93 was -not moved into Done**, because this roadmap entry defines it as *delete the -silence, then self-test, then enable scheduled tests* — the third is step 6, and -it is outstanding. +`scripts/check_docs.py` panel-count coupling stayed untriggered. + +**Done on 2026-09-03**, when step 6 was checked rather than performed: the card +was already set to `biweeklySinceLastTest`, so every file saying scheduled +self-tests are off was asserting something the device contradicts. The same set +moved again — `ups-power.json`, `ups.rules.yaml`, `docs/security.md`, +`README.md`, `docs/hardware.md` and `docs/roadmap.md` — this time to *on, and +unwatched*: the schedule is set, no rule can see it, and the proof is +`upsAdvTestLastDiagnosticsDate` advancing on its own around 2026-09-11. #93 +moved into Done in the same pass, because its third step is now satisfied. **Still to do:** | File | What changes | | --- | --- | | `docs/images/ups-power.png` | Re-capture the set with `make screenshots`. The committed image is from 2026-08-22 and shows the original no-battery banner; it needs the decrypted Grafana password, so it is a human step | -| `docs/roadmap.md` | Move #93 into Done — but only once step 6 is done and the card is testing on a schedule | +| The NMC | Reset `upsBasicBatteryLastReplaceDate`, which still reads `08/15/2026` for a pack fitted on the 28th. A human step at the card's web UI: every SNMP path in this repository reads and none of them writes | And when the shelf is racked, which is a separate visit and separate issue: diff --git a/docs/security.md b/docs/security.md index ae959986..48a31a8e 100644 --- a/docs/security.md +++ b/docs/security.md @@ -230,13 +230,20 @@ rather than left to expire in September, so that rule is live again. Two things outlast the fix. Stored metrics older than 2026-08-28 *are* the fabricated values rather than measurements, so a dashboard or query whose range -crosses that date is reading fiction on one side of it. And scheduled self-tests -are not yet enabled on the card, which makes `1` a last-known result with -nothing refreshing it — `UpsBatteryUnproven` cannot catch a card that stops -testing, because it matches `6` (noTestsInitiated) and this one now reads `1`. -That is step 6 of -[`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md), and it is -outstanding. +crosses that date is reading fiction on one side of it. And the card's test +schedule is on but unwatched. Read off the NMC on 2026-09-03, +`upsAdvTestDiagnosticSchedule` is `8` (biweeklySinceLastTest), which is what +*should* keep `1` from being a frozen last-known result — should, because +nothing here can confirm it still does. That OID is PowerNet, and +the `apc_ups` module walks the standard UPS-MIB only. Nothing here would notice +the card reverting to `never`: `upsTestResultsSummary` would hold `1` and every +rule would stay quiet. `UpsBatteryUnproven` cannot catch it either, because it +matches `6` (noTestsInitiated) and this card reads `1`. Note the shape of that — +the missing pack was visible in a MIB already walked, and the missing *schedule* +would not be. The manual check is in +[`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md), and +closing the gap is +[#249](https://github.com/Gerrrt/HomeLab/issues/249). ### Why SNMPv2c is still a weak point diff --git a/stacks/observability/grafana/dashboards/ups-power.json b/stacks/observability/grafana/dashboards/ups-power.json index 95e167b4..14c1c1b6 100644 --- a/stacks/observability/grafana/dashboards/ups-power.json +++ b/stacks/observability/grafana/dashboards/ups-power.json @@ -1,7 +1,7 @@ { "uid": "homelab-ups", "title": "UPS & Power", - "description": "APC Smart-UPS (mjolnir) via UPS-MIB. An APCRBC115 pack was fitted on 2026-08-28 and passed its self-test the same day, so the battery figures are measured — but anything before that date is fabricated, and scheduled self-tests are not yet enabled on the card. See docs/runbooks/fit-the-ups-battery.md.", + "description": "APC Smart-UPS (mjolnir) via UPS-MIB. An APCRBC115 pack was fitted on 2026-08-28 and passed its self-test the same day, so the battery figures are measured — but anything before that date is fabricated. The card tests itself every fortnight, which nothing on this dashboard can show you: the schedule is a PowerNet OID and this module walks the standard UPS-MIB only. See docs/runbooks/fit-the-ups-battery.md.", "tags": [ "homelab", "power", @@ -267,7 +267,7 @@ }, "options": { "mode": "markdown", - "content": "**The pack is fitted and proven.** An APCRBC115 went into `mjolnir` on 2026-08-28 and `upsTestResultsSummary` moved `4` (aborted) → `1` (donePass) at 22:45 UTC. `upsBatteryVoltage` left its fabricated `480` for a float reading that varies, and the runtime estimate moves instead of sitting on exactly `63` — the step 5 comparison a card cannot fake. Charge still reads exactly `100`, which is what a pack at float voltage looks like.\n\n**Anything on these panels before 2026-08-28 is fabricated.** With an empty bay the card invented a flat 100%, 63 minutes and 48.0 V. The trend panels below cross that boundary — widen the time range with that in mind.\n\n**Scheduled self-tests are not enabled yet.** `1` is the *last* result, not a fresh one; if the NMC never tests again this stops changing and nothing notices. `UpsBatteryUnproven` cannot catch it — that rule matches `6` (noTestsInitiated), and this card now reads `1`. Step 6 of `docs/runbooks/fit-the-ups-battery.md`." + "content": "**The pack is fitted and proven.** An APCRBC115 went into `mjolnir` on 2026-08-28 and `upsTestResultsSummary` moved `4` (aborted) → `1` (donePass) at 22:45 UTC. `upsBatteryVoltage` left its fabricated `480` for a float reading that moves across `540`–`549` — the step 5 comparison a card cannot fake. Runtime is *not* that comparison: it has sat on exactly `63`, the fabricated value, for 744 of 764 samples since the fit. Charge still reads exactly `100`, which is what a pack at float voltage looks like.\n\n**Anything on these panels before 2026-08-28 is fabricated.** With an empty bay the card invented a flat 100%, 63 minutes and 48.0 V. The trend panels below cross that boundary — widen the time range with that in mind.\n\n**The card tests itself every fortnight, and nothing here watches that.** Read off the NMC on 2026-09-03: `upsAdvTestDiagnosticSchedule` `8` (biweeklySinceLastTest), last test `08/28/2026`. Those are PowerNet OIDs and this module walks the standard UPS-MIB only, so if the card reverted to `never`, `upsTestResultsSummary` would hold `1` forever and no panel here would move. `UpsBatteryUnproven` cannot catch it either — it matches `6` (noTestsInitiated). Check by hand with `scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2`; step 6 of `docs/runbooks/fit-the-ups-battery.md`." } }, { diff --git a/stacks/observability/prometheus/rules/ups.rules.yaml b/stacks/observability/prometheus/rules/ups.rules.yaml index 67b094a3..5ab8775d 100644 --- a/stacks/observability/prometheus/rules/ups.rules.yaml +++ b/stacks/observability/prometheus/rules/ups.rules.yaml @@ -3,10 +3,13 @@ # # These rules now read real hardware. An APCRBC115 pack was fitted on 2026-08-28 # and passed its self-test the same day: upsTestResultsSummary went 4 (aborted) -# to 1 (donePass) at 22:45 UTC, upsBatteryVoltage left its fabricated 480 for a -# float reading that varies, and the runtime estimate moves rather than sitting -# on exactly 63. Every rule here is deliberately loud and routed to the `urgent` -# receiver by category=power in alertmanager.yaml. +# to 1 (donePass) at 22:45 UTC, and upsBatteryVoltage left its fabricated 480 +# for a float reading that moves across 540-549. Runtime is the weaker half of +# that comparison and should not be leaned on: measured over the six days after +# the fit it sat on exactly 63 — the fabricated value — for 744 of 764 samples, +# leaving it only for excursions to 45, 60 and 137 under changing load. Every +# rule here is deliberately loud and routed to the `urgent` receiver by +# category=power in alertmanager.yaml. # # WHY THE PRE-FIT BASELINE IS STILL RECORDED HERE. # @@ -27,8 +30,8 @@ # # upsEstimatedChargeRemaining still reads exactly 100. On its own that is the # fabrication signature; here it is consistent with a pack at float voltage, -# because voltage and runtime both moved and the self-test passed. Charge alone -# was never the evidence. +# because voltage moved and the self-test passed. Charge alone was never the +# evidence, and neither, on this UPS, is runtime. # # THE ONE METRIC THAT DOES NOT LIE. # @@ -46,15 +49,38 @@ # 2026-09-20 (54f1715c-e57b-4322-8a6d-5435bc8e1bd8) was deleted on 2026-08-28, # so both rules are live. # -# WHAT IS STILL OUTSTANDING. +# WHAT THIS FILE STILL CANNOT SEE. # -# Scheduled self-tests are not enabled on the NMC. A value of 1 is the *last* -# result, not a fresh one: if the card stops testing the metric simply stops -# changing, and nothing here detects that. UpsBatteryUnproven cannot — it -# matches 6 (noTestsInitiated), and this card now reads 1. upsTestStartTime is +# Scheduled self-tests ARE enabled on the NMC. Read off the card on 2026-09-03: +# +# upsAdvTestDiagnosticSchedule 8 (biweeklySinceLastTest) +# upsAdvTestDiagnosticsResults 1 (ok) +# upsAdvTestLastDiagnosticsDate "08/28/2026" +# +# All three are PowerNet (1.3.6.1.4.1.318.1.1.1.7.2.x), and the apc_ups module +# in snmp-exporter/generator.yaml walks 1.3.6.1.2.1.33.1 — the standard UPS-MIB +# — and nothing else. So none of it reaches Prometheus, and no rule below can +# be written against it. +# +# That leaves one hole open. A value of 1 is the *last* result, not a fresh one: +# if the card stops testing, upsTestResultsSummary simply stops changing, and +# nothing here detects that. UpsBatteryUnproven cannot — it matches 6 +# (noTestsInitiated), and this card reads 1. upsTestStartTime is # sysUpTime-relative and does not survive an agent restart, so it is not a -# usable staleness signal either. Scheduled tests on the device are the control -# and this file cannot substitute for them; step 6 of +# usable staleness signal either. A card that quietly reverts to never(5) — a +# config reset, a firmware update, a restored configuration file — is +# indistinguishable, from inside this file, from one testing every fortnight. +# +# The fix is to walk those three OIDs and alert on the schedule leaving the +# recurring values and on the last-test date going stale, which needs APC's +# PowerNet MIB added to scripts/snmp-mibs.sh and a pinning decision for it. That +# is issue #249, not this file. Until it lands, the check is by hand: +# +# scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2 +# +# and the thing that proves the schedule runs, rather than merely being set, is +# upsAdvTestLastDiagnosticsDate advancing with nobody at the card — due around +# 2026-09-11 for a biweekly schedule last tested on 2026-08-28. Step 6 of # docs/runbooks/fit-the-ups-battery.md. # # Do not run self-tests by hand to keep the metric warm. A test transfers the @@ -170,6 +196,14 @@ groups: # which a UPS has never demonstrated it can do its job, and it is # indistinguishable from a healthy one on every other metric. # + # The 7d window is shorter than this card's 14d test interval, so a + # card reset to 6 can trip this while its schedule is working. That is + # deliberate — the alert says "unproven for a week", which is worth + # knowing whatever the cause — but it is why the description sends you + # to the schedule rather than asserting it has failed. Widening the + # window to clear the interval would trade a week of blindness for a + # quieter alert, and this rule exists because quiet was the problem. + # # Expressed over a range rather than as `for: 7d`, deliberately. A long # `for` measures continuous pending time in Prometheus's own memory, so # any restart inside the window resets it — on a stack that gets @@ -188,6 +222,13 @@ groups: annotations: summary: "UPS {{ $labels.device }} has not completed a self-test in 7 days" description: >- - The UPS has never proven it can carry the load. Enable scheduled - self-tests on the management card so this becomes live evidence - rather than an absence of it. + upsTestResultsSummary has read 6 (noTestsInitiated) for seven days, + so the UPS has not proven it can carry the load inside that window. + That is what this rule detects and all it detects. It is not proof + the schedule has failed: this card is set to biweeklySinceLastTest, + a fourteen-day interval, so a card whose counters have just been + reset can sit at 6 for a fortnight while the schedule is working. + Read the schedule and the last-test date before concluding either + way — scripts/snmp-walk.sh --device mjolnir + 1.3.6.1.4.1.318.1.1.1.7.2 — and if the schedule is off, turn it on + rather than running tests by hand.