Repository navigation
feat(ups): watch the self-test schedule the battery control leans on (#249) - #347
Merged
Merged
Conversation
…249) #93 set mjolnir to test itself every fortnight, and nothing in the stack could see that setting. upsTestResultsSummary is the LAST result: if the card reverts to never(5) after a config reset or a firmware update, it holds donePass forever and every rule in ups.rules.yaml stays correctly quiet. UpsBatteryUnproven cannot catch it — it matches noTestsInitiated(6) and the card reads 1 — and upsTestStartTime is sysUpTime-relative, so it is not a staleness signal either. THE VENDOR MIB WAS NOT THE COST THE ISSUE EXPECTED. #249 framed PowerNet as "a pinning decision, not a URL", because Schneider distribute it as a versioned download rather than a git ref: vendor 2.2 MB into the tree, or trust a moving vendor path. Neither is needed. The observium commit this repository ALREADY pins for UPS-MIB carries mibs/apc/PowerNet-MIB, so this adds a path against an existing pin and inherits its argument unchanged. One subtree, 1.3.6.1.4.1.318.1.1.1.7.2, never the enterprise root — the pfTablesAddrTable lesson one vendor along. Measured against the live card: snmp-walk 7 rows, 2 GETBULKs, 0.07s scrape 3 packets / 51 PDUs / 56 series -> 4 / 58 / 63 and the three the issue named arrive as predicted: upsAdvTestDiagnosticSchedule 8 upsAdvTestDiagnosticsResults 1 upsAdvTestLastDiagnosticsDate{upsAdvTestLastDiagnosticsDate="08/28/2026"} 1 Two rules, a pair on purpose: UpsSelfTestScheduleOff is the fast signal, true the moment the card reads never(5); UpsSelfTestStale is the backstop for a card that says it is scheduled and is not. Either alone leaves half the failure. The enumeration was read out of the MIB rather than assumed. An older PowerNet enumeration stopping at twelveWeeks(7) is in circulation and would read this card's 8 as out of range; the real one runs to fiftytwoWeeks(12). A unit test pins 8 as quiet for that reason — a rule firing for the estate's correct setting would be worse than no rule. UpsSelfTestStale counts distinct series over 21d rather than using `for:`. upsAdvTestLastDiagnosticsDate is a DisplayString, which snmp_exporter renders as a value-1 gauge carrying the date as a label, so "the date changed" is "a second series appeared" and no date parsing is needed. `for: 21d` would restart its clock whenever Prometheus does. It is gated on up{job="snmp"} because a dead exporter produces no new series either and would otherwise read as a card that stopped testing — a true alert about the wrong thing. tests/ups.test.yaml did not exist; nine rules were syntax-only, the standing #63 objected to. Seven cases now cover both new rules, including the dead-exporter trap and the fiftytwoWeeks case that stops someone tidying the expression into `<= 5 or >= 10`. Two corrections made on the way: `metric =~ 1|4|5` is a parse error, because =~ is a label matcher and not a value comparison, so the rule is three `or`ed comparisons; and docs/observability.md's promtool coverage claim was counted rather than incremented — 37 of 58, not the 35 of 56 it said. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
# Conflicts: # docs/observability.md # docs/roadmap.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #249.
#93 set
mjolnirto test itself every fortnight, and nothing in the stack couldsee that setting.
upsTestResultsSummaryis the last result: if the cardreverts to
never(5)after a config reset or a firmware update it holdsdonePassforever, and every rule inups.rules.yamlstays correctly quiet.UpsBatteryUnprovencannot catch it — it matchesnoTestsInitiated(6)and thecard reads
1— andupsTestStartTimeissysUpTime-relative, so it is not astaleness signal either.
The vendor MIB was not the cost the issue expected
#249 framed adding PowerNet as "a pinning decision, not a URL": Schneider
distribute it as a versioned download rather than a git ref, so the options
looked like vendoring 2.2 MB into the tree or trusting a moving vendor path.
Neither is needed. The observium commit this repository already pins for
UPS-MIBcarriesmibs/apc/PowerNet-MIBat the same SHA. This adds a pathagainst an existing pin and inherits its argument unchanged.
Cost, measured
One subtree —
1.3.6.1.4.1.318.1.1.1.7.2, never the enterprise root, which isthe
pfTablesAddrTablelesson one vendor along.The walk itself: 7 rows, 2 GETBULKs, 0.07s. The three OIDs the issue named
arrive exactly as predicted, including the date-as-label form the staleness rule
depends on:
Two rules, a pair on purpose
UpsSelfTestScheduleOffis the fast signal, true the moment the card readsnever(5).UpsSelfTestStaleis the backstop for a card that says it isscheduled and is not. Either alone leaves half the failure.
The enumeration was read out of the MIB, not assumed. An older PowerNet
enumeration stopping at
twelveWeeks(7)is in circulation and would read thiscard's
8as out of range; the real one runs tofiftytwoWeeks(12). A unittest pins
8as quiet for exactly that reason — a rule that fired for theestate's correct setting would be worse than no rule.
UpsSelfTestStalecounts distinct series over 21 days rather than usingfor:.upsAdvTestLastDiagnosticsDateis aDisplayString, which snmp_exporterrenders as a value-1 gauge carrying the date as a label, so "the date changed"
is "a second series appeared" — no date parsing. A
for: 21dwould restartits clock whenever Prometheus does. It is gated on
up{job="snmp"}because adead exporter produces no new series either, and would otherwise read as a card
that stopped testing — a true alert about the wrong thing, raised against the
wrong runbook.
tests/ups.test.yamldid not existNine rules were syntax-only, which is the standing #63 objected to. Seven cases
now cover both new rules — including the dead-exporter trap and the
fiftytwoWeekscase that stops someone tidying the expression into<= 5 or >= 10.promtool test rules: SUCCESS.Two corrections made on the way
metric =~ 1|4|5is a parse error —=~is a label matcher, not a valuecomparison — so the rule is three
ored comparisons.docs/observability.md's promtool coverage claim was counted rather thanincremented: 37 of 58, not the 35 of 56 it said.
Still outstanding, and not a monitoring question
The proof the schedule runs rather than merely being set is the date advancing
with nobody at the card, due around 2026-09-11. As of 2026-09-06 it still
reads
08/28/2026, which is expected and not yet evidence either way.UpsSelfTestStaleis now what will say so if it does not.make validatepasses.🤖 Generated with Claude Code