Repository navigation
Description
#93 closed with mjolnir's
management card set to test itself every fortnight. Nothing in this stack can
see that setting, so the control that keeps upsTestResultsSummary meaningful
is itself unmonitored.
The gap
Read off the card on 2026-09-03 with
scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2:
| OID | Object | Reads |
|---|---|---|
…7.2.1.0 |
upsAdvTestDiagnosticSchedule |
8 — biweeklySinceLastTest |
…7.2.3.0 |
upsAdvTestDiagnosticsResults |
1 — ok |
…7.2.4.0 |
upsAdvTestLastDiagnosticsDate |
08/28/2026 |
All three are PowerNet (1.3.6.1.4.1.318). The apc_ups module walks
1.3.6.1.2.1.33.1 — the standard UPS-MIB — and nothing else.
So if the card drops back to never(5) — a config reset, a firmware update, a
restored configuration file, someone in the web UI — then:
upsTestResultsSummaryholds1(donePass) forever. It is the last result,
and nothing refreshes it.- Every rule in
ups.rules.yamlstays quiet, correctly, because none of them
can read a stale one. UpsBatteryUnprovencannot catch it: that rule matches6(noTestsInitiated)
and this card reads1.upsTestStartTimeissysUpTime-relative and does not survive an agent
restart, so it is not a usable staleness signal either.
Note the shape of it. The missing battery pack was visible in a MIB already
walked — upsTestResultsSummary = 4 needed no new OIDs, which
docs/security.md makes a point of. The missing schedule would not be. This
is the same failure mode one level up: a healthy-looking metric with nothing
underneath it.
What to add
Three scalars, and only those three. Do not walk 1.3.6.1.4.1.318 — the
pfTablesAddrTable lesson in generator.yaml is what that costs. The subtree
1.3.6.1.4.1.318.1.1.1.7.2 is 7 rows and answered in 0.21s over two GETBULKs.
Two rules to go with them:
UpsSelfTestScheduleOff—upsAdvTestDiagnosticSchedulein{1, 4, 5}
(unknown, atTurnOn, never). Warning,category: power. This is the fast
signal: the control is gone the moment it reads5.UpsSelfTestStale— the last-test date has not moved in 21 days
(14 + a week of grace). The backstop for a card that says it is scheduled and
is not.
upsAdvTestDiagnosticSchedule is 1 unknown, 2 biweekly, 3 weekly, 4
atTurnOn, 5 never, 6 fourWeeks, 7 twelveWeeks, 8 biweeklySinceLastTest,
9 weeklySinceLastTest — from APC's own PowerNet-MIB, not the older
enumeration that stops at 7. Getting that wrong reads 8 as out-of-range.
The staleness rule needs no date parsing
upsAdvTestLastDiagnosticsDate is a DisplayString ("08/28/2026"), which
snmp_exporter renders as a value-1 gauge carrying the string as a label —
confirmed against what is already scraped:
upsIdentModel{device="mjolnir", upsIdentModel="Smart-UPS X 1500"} 1
Each distinct date is therefore a distinct series, so counting series over a
range counts tests:
count by (device) (
last_over_time(upsAdvTestLastDiagnosticsDate[21d])
) == 1
One distinct date in 21 days means the date has not changed in 21 days. It
reads the stored series rather than counting pending time, so a Prometheus
restart does not reset it — the same reasoning UpsBatteryUnproven is built on,
and for the same reason a for: 21d would be useless here.
Two things to check when implementing: 21d must sit inside retention (30d /
12GB, and prometheus_tsdb_time_retentions_total was 0 last time it was
looked at), and the rule must be gated on the scrape being up so a dead exporter
does not read as a card that stopped testing.
The actual cost: a new vendor MIB source
scripts/snmp-mibs.sh fetches nothing from APC today, and adding it is a
pinning decision, not a URL. That file argues the case at length: the UPS-MIB
is pinned to a commit precisely because a moved MIB does not fail, it renders a
different snmp.yaml and the diff arrives looking like it came from nowhere.
The other three sources are first-party refs on projects that do not move
release tags.
PowerNet has no equivalent — Schneider's distribution is a versioned download,
not a git ref. Options, roughly in order of preference:
- Vendor the file into the repository. It is one text file, unlike the 5.5 MB
HPE bundle that motivated not vendoring; check the licence. - Pin the vendor download by version and verify a checksum in
fetch(). - Take it from snmp_exporter's own generator mibs, matching whatever upstream
considers canonical, and pin the commit.
Whichever, make snmp-generate must be re-run and the metric-count guard will
compare before and after, as designed.
Done when
-
scripts/snmp-mibs.shfetches PowerNet, pinned, with the reasoning
written down next to the existing pins -
apc_upsingenerator.yamlwalks the three OIDs and no more of318 -
snmp.yamlregenerated withmake snmp-generate -
UpsSelfTestScheduleOffandUpsSelfTestStaleinups.rules.yaml, with
atests/ups.test.yaml— that file has no promtool tests today -
docs/runbooks/fit-the-ups-battery.mdstep 6 stops saying the check is by
hand, andups.rules.yaml's "WHAT THIS FILE STILL CANNOT SEE" header
shrinks to what is still true -
docs/security.md— the paragraph that currently names this gap
Related
- #93 — where this came from
docs/runbooks/fit-the-ups-battery.md
step 6, which carries the by-hand check this replaces
One thing worth waiting for, or at least noting: the proof that the schedule
runs rather than merely being set is upsAdvTestLastDiagnosticsDate advancing
with nobody at the card, due around 2026-09-11. If it has not moved by then,
this issue stops being a monitoring gap and becomes a fault on the card.
Activity
docs(ups): point the self-test-schedule gap at #249
docs(ups): stop the review's three overstatements
Merge pull request #347 from Gerrrt/gerrrt/ups-selftest-249
Merge pull request #348 from Gerrrt/gerrrt/ups-stale-false-positive