Repository navigation
Record that mjolnir's card is already testing on a schedule (#93) - #248
Conversation
Step 6 of the fit-the-ups-battery runbook — enable scheduled self-tests on the NMC — turned out to need checking rather than doing. Read off mjolnir on 2026-09-03 with scripts/snmp-walk.sh: upsAdvTestDiagnosticSchedule 8 (biweeklySinceLastTest) upsAdvTestDiagnosticsResults 1 (ok) upsAdvTestLastDiagnosticsDate "08/28/2026" Six files had said the opposite since the pack went in. They now say what the device says, and #93 moves into Done. They also say the part that does not go away: all three are PowerNet OIDs, the apc_ups module walks the standard UPS-MIB only, and so a card that reverts to never(5) produces no alert and no changed metric — upsTestResultsSummary would hold 1 forever. Closing that means adding APC's MIB to snmp-mibs.sh, a new vendor source with its own pinning decision, and it is deliberately not done here. Until it is, step 6 is verified by hand, and what proves the schedule runs rather than merely being set is that date advancing unattended, due around 2026-09-11. Two findings on the way. upsBasicBatteryLastReplaceDate still reads 08/15/2026 for a pack fitted on the 28th, so the card's battery-age accounting is keyed to a date on which its own self-test was still aborting over an empty bay; the runbook and hardware.md carry it as an outstanding human step at the web UI. And the step 5 comparison was overstated everywhere it appeared: since the fit, upsEstimatedMinutesRemaining has sat on exactly 63 — the fabricated value — for 744 of 764 samples, so runtime cannot discriminate a real pack from an invented one. Voltage moving across 540-549 can, and that is now what the runbook, the rules header and the dashboard banner tell you to read. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
🟡 Changes recommended
A few updated/added operational messages overstate what can be concluded from current metrics/config (notably around the 7-day “noTestsInitiated” alert and the documented closure steps), which could mislead on-call actions.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Updates the observability and operational documentation around the APC Smart-UPS (mjolnir) to reflect verified on-device state: scheduled self-tests are configured (biweeklySinceLastTest), while the monitoring stack still cannot scrape/alert on that schedule because it’s in PowerNet OIDs outside the current SNMP exporter walk.
Changes:
- Update the Prometheus UPS rules header and the Grafana UPS dashboard text to record the verified self-test schedule and explicitly call out the monitoring blind spot (PowerNet OIDs not scraped).
- Update runbook/roadmap/hardware/README narrative to move #93 to “done” based on the 2026-09-03 check, and refine the “voltage vs runtime” evidentiary guidance.
- Adjust alert/dashboard wording to describe manual verification steps and the limitations of existing UPS-MIB-derived metrics.
File summaries
| File | Description |
|---|---|
| stacks/observability/prometheus/rules/ups.rules.yaml | Documents verified scheduled-test state and monitoring gap; updates an alert annotation message. |
| stacks/observability/grafana/dashboards/ups-power.json | Updates dashboard description/banner to reflect schedule being set but not observable via current scrape. |
| README.md | Updates roadmap-style “top items” text to treat #93 as completed and prioritize remaining work. |
| docs/security.md | Updates security posture narrative to state the self-test schedule is enabled but not monitored. |
| docs/runbooks/fit-the-ups-battery.md | Updates runbook status and adds detailed “check schedule by hand” instructions and findings. |
| docs/roadmap.md | Moves #93 into “Done” with recorded verification details and remaining monitoring gap. |
| docs/hardware.md | Updates hardware inventory note for the UPS battery and outstanding reset of replace-date. |
Review details
- Files reviewed: 7/7 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
The four places that describe the gap left by #93 — the runbook banner and step 6, the ups.rules.yaml header, docs/security.md — each said the fix was not done and why. They now say where it is being done, and #249 goes into the roadmap's Monitoring section as open work. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…placement-e076f1 Two conflicts, both where main and this branch rewrote the same paragraph. docs/roadmap.md, Infrastructure: main rewrote #92 to "half done" now that `make backup-firewall` copies exports to oracle; this branch deleted the #93 entry below it. Kept main's #92 text without the #93 entry, which has moved into Done. docs/roadmap.md, Done: both sides added an entry dated 2026-09-03 — #91 from main, #93 from here. Both kept. README.md: main narrowed #92 to "the spare is what is left" now the export leaves the host nightly; this branch dropped #93 from the top items. Merged both — the top items are now the shelf switch and the spare. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
UpsBatteryUnproven's description claimed seven days at 6 (noTestsInitiated) means the schedule is not running. It does not: the card tests every fourteen days, so one whose counters have just been reset can sit at 6 for a fortnight while the schedule works. The description now says what the rule detects and sends the reader to the schedule, and a comment above the rule records why the window is deliberately shorter than the interval. docs/security.md said the schedule setting "stops" upsTestResultsSummary from freezing, in a paragraph whose whole point is that nothing here can confirm it still does. Now "should keep — should, because nothing here can confirm it". The runbook named only generator.yaml as the cost of closing #249, when the pinning decision lives in scripts/snmp-mibs.sh and make snmp-generate cannot resolve the OIDs until it fetches the MIB. Both named. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Closes #93.
The issue's last open step — "enable scheduled self-tests on the NMC" — turned
out to need checking rather than doing. Read off the card on 2026-09-03 with
scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2:…7.2.1.0upsAdvTestDiagnosticSchedule8— biweeklySinceLastTest…7.2.3.0upsAdvTestDiagnosticsResults1— ok…7.2.4.0upsAdvTestLastDiagnosticsDate08/28/20268is from APC's own PowerNet-MIB, whose enumeration runs to9— not theolder one that stops at
7. Whether the schedule was set at the rack on the28th or has been the card's default all along, the NMC will not say after the
fact. Six files had been asserting it was off; they now say what the device
says.
The other three steps were verified too, not assumed: silence
54f1715c-e57b-4322-8a6d-5435bc8e1bd8is absent from Alertmanager (notmerely expired),
upsTestResultsSummaryreads1, and no UPS alert is firing.What this deliberately does not do
Nothing in the stack can see that schedule. All three OIDs are PowerNet and the
apc_upsmodule walks the standard UPS-MIB only, so a card that reverts tonever(5)produces no alert and no changed metric —upsTestResultsSummarywould simply hold
1forever, andUpsBatteryUnprovencannot catch it becauseit matches
6.Closing that means adding APC's MIB to
scripts/snmp-mibs.sh— a new vendorsource needing its own pinning decision, in a file that argues at length about
exactly that — plus a regenerated
snmp.yamland two new rules. That is its ownchange, not a rider on this one. Every document here now names the gap and the
by-hand check instead of implying something watches it, and points at
#249, which is where it gets
closed — including the PowerNet pinning options and a staleness rule that needs
no date parsing.
Two findings on the way
upsBasicBatteryLastReplaceDatereads08/15/2026for a pack fitted on the28th — a date on which this card's own self-test was still aborting over an
empty bay. It is the only record of the pack's age. Carried as an outstanding
human step at the web UI; every SNMP path in this repository reads and none
of them writes.
upsEstimatedMinutesRemaininghas sat on exactly63— the fabricated value— for 744 of 764 samples, leaving it only for excursions to
45,60and137. Runtime cannot tell a real pack from an invented one on this UPS.Voltage moving across
540–549can, and that is now what the runbook, therules header and the dashboard banner tell you to read.
Still open
The proof that the schedule runs, rather than merely being set, is
upsAdvTestLastDiagnosticsDateadvancing with nobody at the card — due around2026-09-11. Worth a look then.
./scripts/validate.shpasses (2 skips: not the deployment checkout, stack notrunning here).
🤖 Generated with Claude Code