Repository navigation
Description
Split out of #120, which was closed on the code being correct and delivery being
proven — not on the end-to-end drill, which has never been run.
Everything machine-checkable about the alerting path now checks out. What has
never happened is a human watching the switch go red.
A dead man's switch nobody has seen go red is indistinguishable from one that
does not work.
That is #120's own standard, and this is the half of it still outstanding.
What is already true
Verified on 2026-09-04, so this issue does not need to re-establish any of it:
- Four receivers on four distinct URLs, all four rendered and mounted.
group_interval: 1m,repeat_interval: 5mon the heartbeat route after fix(alertmanager): heartbeat pinged at half its advertised rate (#120) #284 —
before that fix it pinged at 600s, which made the 15m grace below sit under a
single missed ping.- Measured inter-ping gaps of 300s / 310s / 351s.
- Zero
clientErroracross every ping, so the far end is accepting them. Note
what that does and does not prove: the endpoint exists, not that anything
alarms when the pings stop.
The drill
docs/runbooks/verify-the-alert-path.md, "Confirming it actually works":
docker stop alertmanager
# wait out the grace window — 15m
# the external check must report DOWN *and notify you*
docker start alertmanager
# must return green within one repeat_intervalBest run when you are near the phone and not on the clock.
While you are in there, two things only you can read back
Both are settings on the external check, invisible from this host, and both were
sized against a repeat_interval that was wrong until #284:
- Period 5m, grace 15m. If the check was created before fix(alertmanager): heartbeat pinged at half its advertised rate (#120) #284 it may have
been retuned to cope with the 600s cadence — a grace stretched to 20m+ to stop
false pages. That compensation is now wrong in the other direction and should
go back to 15m. - The check's own notification must not land on this stack's ntfy topic.
If both go to the same topic, a deleted topic takes out the alert and the
warning about the alert together — the No dead man's switch on the notification path #67 failure, one level up. Email is
fine here; it fails independently.
Done when
- The check has been observed DOWN inside the grace window, with the
notification arriving somewhere that is not this stack's webhook. - It returned green within one
repeat_intervalafterdocker start. - Period and grace confirmed at 5m / 15m.
- A line in this issue saying it was seen, and on what date — so the next person
does not have to wonder whether it was ever tested.
Activity
Gerrrt commentedon Sep 5, 2026
Duplicate of #288 — same drill, split out of #120 twice on 2026-09-04, 42 minutes apart.
Keeping #288 because its "done when" is the more complete of the two: it carries the 24h-route confirmation via the temporary-
repeat_intervalmethod, which this issue omits. Everything this issue had that #288 did not — the post-#284 measurements, the period/grace retune check, and the ntfy-topic warning — has been transplanted into #288 (comment) before closing, so nothing is lost.The drill itself is still outstanding. It needs a human near a phone and cannot be run from this host.
🤖 Generated with Claude Code