Repository navigation
fix(ups): UpsSelfTestStale fired the moment it deployed (#249) - #348
Merged
Merged
Conversation
Caught on the live stack twenty minutes after #249 converged. The rule counts distinct upsAdvTestLastDiagnosticsDate series over 21 days and alerts when there is one — but a metric that has just been introduced has exactly one distinct value by construction. There was a single sample in the window, twenty minutes old, and the alert was firing for a reason that had nothing to do with the card. It would have stayed firing until the next real self-test on 2026-09-11. The same class of defect this repository keeps recording: an alert that is true for a reason other than the one it names. Gated on present_over_time(upsAdvTestLastDiagnosticsDate[1d] offset 20d), so the count only means what it claims once twenty days of history exist. A day-wide probe rather than an instant one, so a brief exporter outage exactly twenty days back does not silence the rule for three weeks. The cost, stated rather than hidden: this says nothing for the first twenty days after the metric appears or after a Prometheus wipe. That is correct — before then there is genuinely no evidence — and it is better than a rule that pages on the day it ships. Regression test added with two days of samples: enough for the count to be 1, nowhere near enough for the history gate, so the case fails without the fix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Caught on the live stack twenty minutes after #249 converged, by looking at the
alert list rather than by anything telling me.
UpsSelfTestStalecounts distinctupsAdvTestLastDiagnosticsDateseries over 21days and alerts when there is one. A metric that has just been introduced has
exactly one distinct value by construction. There was a single sample in the
window, twenty minutes old:
It was firing for a reason that had nothing to do with the card, and it would
have stayed firing until the next real self-test on 2026-09-11. That is the same
class of defect this repository keeps recording — an alert that is true for a
reason other than the one it names — and I shipped it in the change that was
supposed to be careful about exactly that.
The fix
Also gate on the series having existed:
so the count only means what it claims once there are twenty days of history. A
day-wide probe rather than an instant one, so a brief exporter outage exactly
twenty days back does not silence the rule for three weeks.
The cost, stated rather than hidden: this rule says nothing for the first
twenty days after the metric appears, or after a Prometheus wipe. That is
correct — before then there is genuinely no evidence — and it is better than a
rule that pages on the day it ships.
Regression test
Two days of samples: enough for the count to be
1, nowhere near enough for thehistory gate. The case fails without the fix and passes with it.
promtool test rules: SUCCESS across all eight UPS cases.make validatepasses.🤖 Generated with Claude Code