Skip to content

fix(ups): UpsSelfTestStale fired the moment it deployed (#249) - #348

Merged
Gerrrt merged 1 commit into
mainfrom
gerrrt/ups-stale-false-positive
Sep 6, 2026
Merged

Gerrrt merged 1 commit into
mainfrom
gerrrt/ups-stale-false-positive

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Sep 6, 2026

Copy link
Copy Markdown
Owner

Caught on the live stack twenty minutes after #249 converged, by looking at the
alert list rather than by anything telling me.

UpsSelfTestStale counts distinct upsAdvTestLastDiagnosticsDate series over 21
days and alerts when there is one. A metric that has just been introduced has
exactly one distinct value by construction.
There was a single sample in the
window, twenty minutes old:

count_over_time(upsAdvTestLastDiagnosticsDate[21d])  ->  1
alerts: [('firing', 'UpsSelfTestStale')]

It was firing for a reason that had nothing to do with the card, and it would
have stayed firing until the next real self-test on 2026-09-11. That is the same
class of defect this repository keeps recording — an alert that is true for a
reason other than the one it names — and I shipped it in the change that was
supposed to be careful about exactly that.

The fix

Also gate on the series having existed:

and on(device) present_over_time(upsAdvTestLastDiagnosticsDate[1d] offset 20d)

so the count only means what it claims once there are twenty days of history. A
day-wide probe rather than an instant one, so a brief exporter outage exactly
twenty days back does not silence the rule for three weeks.

The cost, stated rather than hidden: this rule says nothing for the first
twenty days after the metric appears, or after a Prometheus wipe. That is
correct — before then there is genuinely no evidence — and it is better than a
rule that pages on the day it ships.

Regression test

Two days of samples: enough for the count to be 1, nowhere near enough for the
history gate. The case fails without the fix and passes with it.
promtool test rules: SUCCESS across all eight UPS cases.

make validate passes.

🤖 Generated with Claude Code

Caught on the live stack twenty minutes after #249 converged. The rule counts
distinct upsAdvTestLastDiagnosticsDate series over 21 days and alerts when there
is one — but a metric that has just been introduced has exactly one distinct
value by construction. There was a single sample in the window, twenty minutes
old, and the alert was firing for a reason that had nothing to do with the card.
It would have stayed firing until the next real self-test on 2026-09-11.

The same class of defect this repository keeps recording: an alert that is true
for a reason other than the one it names.

Gated on present_over_time(upsAdvTestLastDiagnosticsDate[1d] offset 20d), so the
count only means what it claims once twenty days of history exist. A day-wide
probe rather than an instant one, so a brief exporter outage exactly twenty days
back does not silence the rule for three weeks.

The cost, stated rather than hidden: this says nothing for the first twenty days
after the metric appears or after a Prometheus wipe. That is correct — before
then there is genuinely no evidence — and it is better than a rule that pages on
the day it ships.

Regression test added with two days of samples: enough for the count to be 1,
nowhere near enough for the history gate, so the case fails without the fix.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Gerrrt
Gerrrt merged commit 056deef into main Sep 6, 2026
3 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/ups-stale-false-positive branch September 6, 2026 18:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant