Skip to content

feat(ups): watch the self-test schedule the battery control leans on (#249) - #347

Merged
Gerrrt merged 2 commits into
mainfrom
gerrrt/ups-selftest-249
Sep 6, 2026
Merged

Gerrrt merged 2 commits into
mainfrom
gerrrt/ups-selftest-249

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Closes #249.

#93 set mjolnir to test itself every fortnight, and nothing in the stack could
see that setting. upsTestResultsSummary is the last result: if the card
reverts to never(5) after a config reset or a firmware update it holds
donePass forever, and every rule in ups.rules.yaml stays correctly quiet.
UpsBatteryUnproven cannot catch it — it matches noTestsInitiated(6) and the
card reads 1 — and upsTestStartTime is sysUpTime-relative, so it is not a
staleness signal either.

The vendor MIB was not the cost the issue expected

#249 framed adding PowerNet as "a pinning decision, not a URL": Schneider
distribute it as a versioned download rather than a git ref, so the options
looked like vendoring 2.2 MB into the tree or trusting a moving vendor path.

Neither is needed. The observium commit this repository already pins for
UPS-MIB carries mibs/apc/PowerNet-MIB at the same SHA. This adds a path
against an existing pin and inherits its argument unchanged.

Cost, measured

One subtree — 1.3.6.1.4.1.318.1.1.1.7.2, never the enterprise root, which is
the pfTablesAddrTable lesson one vendor along.

before after
packets 3 4
PDUs 51 58
series 56 63
duration 0.016s 0.027s

The walk itself: 7 rows, 2 GETBULKs, 0.07s. The three OIDs the issue named
arrive exactly as predicted, including the date-as-label form the staleness rule
depends on:

upsAdvTestDiagnosticSchedule 8
upsAdvTestDiagnosticsResults 1
upsAdvTestLastDiagnosticsDate{upsAdvTestLastDiagnosticsDate="08/28/2026"} 1

Two rules, a pair on purpose

UpsSelfTestScheduleOff is the fast signal, true the moment the card reads
never(5). UpsSelfTestStale is the backstop for a card that says it is
scheduled and is not. Either alone leaves half the failure.

The enumeration was read out of the MIB, not assumed. An older PowerNet
enumeration stopping at twelveWeeks(7) is in circulation and would read this
card's 8 as out of range; the real one runs to fiftytwoWeeks(12). A unit
test pins 8 as quiet for exactly that reason — a rule that fired for the
estate's correct setting would be worse than no rule.

UpsSelfTestStale counts distinct series over 21 days rather than using for:.
upsAdvTestLastDiagnosticsDate is a DisplayString, which snmp_exporter
renders as a value-1 gauge carrying the date as a label, so "the date changed"
is "a second series appeared" — no date parsing. A for: 21d would restart
its clock whenever Prometheus does. It is gated on up{job="snmp"} because a
dead exporter produces no new series either, and would otherwise read as a card
that stopped testing — a true alert about the wrong thing, raised against the
wrong runbook.

tests/ups.test.yaml did not exist

Nine rules were syntax-only, which is the standing #63 objected to. Seven cases
now cover both new rules — including the dead-exporter trap and the
fiftytwoWeeks case that stops someone tidying the expression into
<= 5 or >= 10. promtool test rules: SUCCESS.

Two corrections made on the way

  • metric =~ 1|4|5 is a parse error — =~ is a label matcher, not a value
    comparison — so the rule is three ored comparisons.
  • docs/observability.md's promtool coverage claim was counted rather than
    incremented
    : 37 of 58, not the 35 of 56 it said.

Still outstanding, and not a monitoring question

The proof the schedule runs rather than merely being set is the date advancing
with nobody at the card, due around 2026-09-11. As of 2026-09-06 it still
reads 08/28/2026, which is expected and not yet evidence either way.
UpsSelfTestStale is now what will say so if it does not.

make validate passes.

🤖 Generated with Claude Code

Gerrrt and others added 2 commits September 6, 2026 18:28
…249)

#93 set mjolnir to test itself every fortnight, and nothing in the stack could
see that setting. upsTestResultsSummary is the LAST result: if the card reverts
to never(5) after a config reset or a firmware update, it holds donePass forever
and every rule in ups.rules.yaml stays correctly quiet. UpsBatteryUnproven
cannot catch it — it matches noTestsInitiated(6) and the card reads 1 — and
upsTestStartTime is sysUpTime-relative, so it is not a staleness signal either.

THE VENDOR MIB WAS NOT THE COST THE ISSUE EXPECTED. #249 framed PowerNet as "a
pinning decision, not a URL", because Schneider distribute it as a versioned
download rather than a git ref: vendor 2.2 MB into the tree, or trust a moving
vendor path. Neither is needed. The observium commit this repository ALREADY
pins for UPS-MIB carries mibs/apc/PowerNet-MIB, so this adds a path against an
existing pin and inherits its argument unchanged.

One subtree, 1.3.6.1.4.1.318.1.1.1.7.2, never the enterprise root — the
pfTablesAddrTable lesson one vendor along. Measured against the live card:

  snmp-walk    7 rows, 2 GETBULKs, 0.07s
  scrape       3 packets / 51 PDUs / 56 series  ->  4 / 58 / 63

and the three the issue named arrive as predicted:

  upsAdvTestDiagnosticSchedule 8
  upsAdvTestDiagnosticsResults 1
  upsAdvTestLastDiagnosticsDate{upsAdvTestLastDiagnosticsDate="08/28/2026"} 1

Two rules, a pair on purpose: UpsSelfTestScheduleOff is the fast signal, true
the moment the card reads never(5); UpsSelfTestStale is the backstop for a card
that says it is scheduled and is not. Either alone leaves half the failure.

The enumeration was read out of the MIB rather than assumed. An older PowerNet
enumeration stopping at twelveWeeks(7) is in circulation and would read this
card's 8 as out of range; the real one runs to fiftytwoWeeks(12). A unit test
pins 8 as quiet for that reason — a rule firing for the estate's correct
setting would be worse than no rule.

UpsSelfTestStale counts distinct series over 21d rather than using `for:`.
upsAdvTestLastDiagnosticsDate is a DisplayString, which snmp_exporter renders as
a value-1 gauge carrying the date as a label, so "the date changed" is "a second
series appeared" and no date parsing is needed. `for: 21d` would restart its
clock whenever Prometheus does. It is gated on up{job="snmp"} because a dead
exporter produces no new series either and would otherwise read as a card that
stopped testing — a true alert about the wrong thing.

tests/ups.test.yaml did not exist; nine rules were syntax-only, the standing #63
objected to. Seven cases now cover both new rules, including the dead-exporter
trap and the fiftytwoWeeks case that stops someone tidying the expression into
`<= 5 or >= 10`.

Two corrections made on the way: `metric =~ 1|4|5` is a parse error, because =~
is a label matcher and not a value comparison, so the rule is three `or`ed
comparisons; and docs/observability.md's promtool coverage claim was counted
rather than incremented — 37 of 58, not the 35 of 56 it said.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
# Conflicts:
#	docs/observability.md
#	docs/roadmap.md
@Gerrrt
Gerrrt merged commit c4af6a7 into main Sep 6, 2026
3 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/ups-selftest-249 branch September 6, 2026 18:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Scrape the UPS self-test schedule — nothing watches the control #93 leans on

1 participant