Skip to content

Record that mjolnir's card is already testing on a schedule (#93) - #248

Merged
Gerrrt merged 4 commits into
mainfrom
gerrrt/ups-battery-replacement-e076f1
Sep 3, 2026
Merged

Gerrrt merged 4 commits into
mainfrom
gerrrt/ups-battery-replacement-e076f1

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Sep 3, 2026 •

Copy link
Copy Markdown
Owner

Closes #93.

The issue's last open step — "enable scheduled self-tests on the NMC" — turned
out to need checking rather than doing. Read off the card on 2026-09-03 with
scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2:

OID Object Reads
…7.2.1.0 upsAdvTestDiagnosticSchedule 8 — biweeklySinceLastTest
…7.2.3.0 upsAdvTestDiagnosticsResults 1 — ok
…7.2.4.0 upsAdvTestLastDiagnosticsDate 08/28/2026

8 is from APC's own PowerNet-MIB, whose enumeration runs to 9 — not the
older one that stops at 7. Whether the schedule was set at the rack on the
28th or has been the card's default all along, the NMC will not say after the
fact. Six files had been asserting it was off; they now say what the device
says.

The other three steps were verified too, not assumed: silence
54f1715c-e57b-4322-8a6d-5435bc8e1bd8 is absent from Alertmanager (not
merely expired), upsTestResultsSummary reads 1, and no UPS alert is firing.

What this deliberately does not do

Nothing in the stack can see that schedule. All three OIDs are PowerNet and the
apc_ups module walks the standard UPS-MIB only, so a card that reverts to
never(5) produces no alert and no changed metric — upsTestResultsSummary
would simply hold 1 forever, and UpsBatteryUnproven cannot catch it because
it matches 6.

Closing that means adding APC's MIB to scripts/snmp-mibs.sh — a new vendor
source needing its own pinning decision, in a file that argues at length about
exactly that — plus a regenerated snmp.yaml and two new rules. That is its own
change, not a rider on this one. Every document here now names the gap and the
by-hand check instead of implying something watches it, and points at
#249, which is where it gets
closed — including the PowerNet pinning options and a staleness rule that needs
no date parsing.

Two findings on the way

  • upsBasicBatteryLastReplaceDate reads 08/15/2026 for a pack fitted on the
    28th — a date on which this card's own self-test was still aborting over an
    empty bay. It is the only record of the pack's age. Carried as an outstanding
    human step at the web UI; every SNMP path in this repository reads and none
    of them writes.
  • The step 5 comparison was overstated wherever it appeared. Since the fit,
    upsEstimatedMinutesRemaining has sat on exactly 63 — the fabricated value
    — for 744 of 764 samples, leaving it only for excursions to 45, 60 and
    137. Runtime cannot tell a real pack from an invented one on this UPS.
    Voltage moving across 540–549 can, and that is now what the runbook, the
    rules header and the dashboard banner tell you to read.

Still open

The proof that the schedule runs, rather than merely being set, is
upsAdvTestLastDiagnosticsDate advancing with nobody at the card — due around
2026-09-11. Worth a look then.

./scripts/validate.sh passes (2 skips: not the deployment checkout, stack not
running here).

🤖 Generated with Claude Code

Step 6 of the fit-the-ups-battery runbook — enable scheduled self-tests on
the NMC — turned out to need checking rather than doing. Read off mjolnir on
2026-09-03 with scripts/snmp-walk.sh:

  upsAdvTestDiagnosticSchedule   8    (biweeklySinceLastTest)
  upsAdvTestDiagnosticsResults   1    (ok)
  upsAdvTestLastDiagnosticsDate  "08/28/2026"

Six files had said the opposite since the pack went in. They now say what the
device says, and #93 moves into Done.

They also say the part that does not go away: all three are PowerNet OIDs, the
apc_ups module walks the standard UPS-MIB only, and so a card that reverts to
never(5) produces no alert and no changed metric — upsTestResultsSummary would
hold 1 forever. Closing that means adding APC's MIB to snmp-mibs.sh, a new
vendor source with its own pinning decision, and it is deliberately not done
here. Until it is, step 6 is verified by hand, and what proves the schedule
runs rather than merely being set is that date advancing unattended, due
around 2026-09-11.

Two findings on the way. upsBasicBatteryLastReplaceDate still reads 08/15/2026
for a pack fitted on the 28th, so the card's battery-age accounting is keyed to
a date on which its own self-test was still aborting over an empty bay; the
runbook and hardware.md carry it as an outstanding human step at the web UI.
And the step 5 comparison was overstated everywhere it appeared: since the fit,
upsEstimatedMinutesRemaining has sat on exactly 63 — the fabricated value — for
744 of 764 samples, so runtime cannot discriminate a real pack from an invented
one. Voltage moving across 540-549 can, and that is now what the runbook, the
rules header and the dashboard banner tell you to read.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Copilot AI lite review requested due to automatic review settings September 3, 2026 17:50

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

A few updated/added operational messages overstate what can be concluded from current metrics/config (notably around the 7-day “noTestsInitiated” alert and the documented closure steps), which could mislead on-call actions.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Updates the observability and operational documentation around the APC Smart-UPS (mjolnir) to reflect verified on-device state: scheduled self-tests are configured (biweeklySinceLastTest), while the monitoring stack still cannot scrape/alert on that schedule because it’s in PowerNet OIDs outside the current SNMP exporter walk.

Changes:

  • Update the Prometheus UPS rules header and the Grafana UPS dashboard text to record the verified self-test schedule and explicitly call out the monitoring blind spot (PowerNet OIDs not scraped).
  • Update runbook/roadmap/hardware/README narrative to move #93 to “done” based on the 2026-09-03 check, and refine the “voltage vs runtime” evidentiary guidance.
  • Adjust alert/dashboard wording to describe manual verification steps and the limitations of existing UPS-MIB-derived metrics.
File summaries
File Description
stacks/observability/prometheus/rules/ups.rules.yaml Documents verified scheduled-test state and monitoring gap; updates an alert annotation message.
stacks/observability/grafana/dashboards/ups-power.json Updates dashboard description/banner to reflect schedule being set but not observable via current scrape.
README.md Updates roadmap-style “top items” text to treat #93 as completed and prioritize remaining work.
docs/security.md Updates security posture narrative to state the self-test schedule is enabled but not monitored.
docs/runbooks/fit-the-ups-battery.md Updates runbook status and adds detailed “check schedule by hand” instructions and findings.
docs/roadmap.md Moves #93 into “Done” with recorded verification details and remaining monitoring gap.
docs/hardware.md Updates hardware inventory note for the UPS battery and outstanding reset of replace-date.
Review details
  • Files reviewed: 7/7 changed files
  • Comments generated: 3
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread docs/runbooks/fit-the-ups-battery.md Resolved
Comment thread docs/security.md Resolved
Gerrrt and others added 3 commits September 3, 2026 18:33
The four places that describe the gap left by #93 — the runbook banner and
step 6, the ups.rules.yaml header, docs/security.md — each said the fix was
not done and why. They now say where it is being done, and #249 goes into the
roadmap's Monitoring section as open work.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…placement-e076f1

Two conflicts, both where main and this branch rewrote the same paragraph.

docs/roadmap.md, Infrastructure: main rewrote #92 to "half done" now that
`make backup-firewall` copies exports to oracle; this branch deleted the #93
entry below it. Kept main's #92 text without the #93 entry, which has moved
into Done.

docs/roadmap.md, Done: both sides added an entry dated 2026-09-03 — #91 from
main, #93 from here. Both kept.

README.md: main narrowed #92 to "the spare is what is left" now the export
leaves the host nightly; this branch dropped #93 from the top items. Merged
both — the top items are now the shelf switch and the spare.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
UpsBatteryUnproven's description claimed seven days at 6 (noTestsInitiated)
means the schedule is not running. It does not: the card tests every fourteen
days, so one whose counters have just been reset can sit at 6 for a fortnight
while the schedule works. The description now says what the rule detects and
sends the reader to the schedule, and a comment above the rule records why the
window is deliberately shorter than the interval.

docs/security.md said the schedule setting "stops" upsTestResultsSummary from
freezing, in a paragraph whose whole point is that nothing here can confirm it
still does. Now "should keep — should, because nothing here can confirm it".

The runbook named only generator.yaml as the cost of closing #249, when the
pinning decision lives in scripts/snmp-mibs.sh and make snmp-generate cannot
resolve the OIDs until it fetches the MIB. Both named.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Gerrrt
Gerrrt merged commit 5eb4554 into main Sep 3, 2026
3 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/ups-battery-replacement-e076f1 branch September 3, 2026 19:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace the UPS battery, then delete the silence and enable scheduled self-tests

2 participants