Skip to content

Watch the dead man's switch actually trip, once #312

Closed as duplicate
Closed as duplicate

Description

@Gerrrt

Split out of #120, which was closed on the code being correct and delivery being
proven — not on the end-to-end drill, which has never been run.

Everything machine-checkable about the alerting path now checks out. What has
never happened is a human watching the switch go red.

A dead man's switch nobody has seen go red is indistinguishable from one that
does not work.

That is #120's own standard, and this is the half of it still outstanding.

What is already true

Verified on 2026-09-04, so this issue does not need to re-establish any of it:

  • Four receivers on four distinct URLs, all four rendered and mounted.
  • group_interval: 1m, repeat_interval: 5m on the heartbeat route after fix(alertmanager): heartbeat pinged at half its advertised rate (#120) #284 —
    before that fix it pinged at 600s, which made the 15m grace below sit under a
    single missed ping.
  • Measured inter-ping gaps of 300s / 310s / 351s.
  • Zero clientError across every ping, so the far end is accepting them. Note
    what that does and does not prove: the endpoint exists, not that anything
    alarms when the pings stop.

The drill

docs/runbooks/verify-the-alert-path.md, "Confirming it actually works":

docker stop alertmanager
# wait out the grace window — 15m
# the external check must report DOWN *and notify you*
docker start alertmanager
# must return green within one repeat_interval

Best run when you are near the phone and not on the clock.

While you are in there, two things only you can read back

Both are settings on the external check, invisible from this host, and both were
sized against a repeat_interval that was wrong until #284:

  1. Period 5m, grace 15m. If the check was created before fix(alertmanager): heartbeat pinged at half its advertised rate (#120) #284 it may have
    been retuned to cope with the 600s cadence — a grace stretched to 20m+ to stop
    false pages. That compensation is now wrong in the other direction and should
    go back to 15m.
  2. The check's own notification must not land on this stack's ntfy topic.
    If both go to the same topic, a deleted topic takes out the alert and the
    warning about the alert together — the No dead man's switch on the notification path #67 failure, one level up. Email is
    fine here; it fails independently.

Done when

  • The check has been observed DOWN inside the grace window, with the
    notification arriving somewhere that is not this stack's webhook.
  • It returned green within one repeat_interval after docker start.
  • Period and grace confirmed at 5m / 15m.
  • A line in this issue saying it was seen, and on what date — so the next person
    does not have to wonder whether it was ever tested.

Activity

  1. Gerrrt commented on Sep 5, 2026

    @Gerrrt
    OwnerAuthor

    Duplicate of #288 — same drill, split out of #120 twice on 2026-09-04, 42 minutes apart.

    Keeping #288 because its "done when" is the more complete of the two: it carries the 24h-route confirmation via the temporary-repeat_interval method, which this issue omits. Everything this issue had that #288 did not — the post-#284 measurements, the period/grace retune check, and the ntfy-topic warning — has been transplanted into #288 (comment) before closing, so nothing is lost.

    The drill itself is still outstanding. It needs a human near a phone and cannot be run from this host.

    🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureA new capability or a change

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions