Skip to content

Watch the alerting dead man's switch trip, once #288

Closed

Description

@Gerrrt

Split out of #120, which is closed: the three notification secrets are filled in, the five-receiver config is live, and each receiver reads its own non-empty secret. What #120 could not close is its own last instruction:

Then, once, actually watch it trip. A dead man's switch nobody has seen go red is indistinguishable from one that does not work.

The mechanism is built and verifiable from the repository. The drill is not, and there is no record of it having been run. That is a different kind of task with a different definition of done, which is why it is here rather than holding #120 open.

What exists today

  • watchdog.rules.yaml emits the always-firing Watchdog alert.
  • alertmanager.yaml:61-67 routes it to the heartbeat receiver with group_wait: 0s, group_interval: 1m, repeat_interval: 5m, continue: true.
  • A second Watchdog route sends 24h to the real alert channel — the half that catches a webhook which is well-formed, reachable, returns 200 and points at nothing (No dead man's switch on the notification path #67's actual failure).
  • ALERTMANAGER_HEARTBEAT_URL is set and the external check is green.

A green check that has never been seen go red is the thing this issue exists to fix.

The procedure

Already written, in docs/runbooks/verify-the-alert-path.md under "Confirming it actually works":

docker stop alertmanager
# wait out the grace window — 15m by default
# the external check must report DOWN and notify you
docker start alertmanager
# it must return to green within one repeat_interval

Nothing needs to be written. This is execution, and then a line somewhere saying it happened.

The runbook's second paragraph there covers the daily half without waiting a day: temporarily lower repeat_interval on the second Watchdog route, make reload, and check the real channel.

Pick the window deliberately

docker stop alertmanager for the length of the grace window means the estate has no alert delivery for ~15 minutes. Prometheus keeps evaluating and Alertmanager will deliver what is still firing when it comes back, but anything that resolves inside the window is never notified. Two constraints from the runbook's own "timing is coupled" section apply:

  • Not Sunday 03:30 — homelab-backup-volumes quiesces the stack then, which stops Prometheus evaluating and therefore stops the heartbeat. Overlapping the two makes the result unreadable.
  • The external check's notification must not be this stack's webhook. If it is, the test cannot report its own failure.

Done when

  • The external check has been observed DOWN and sent its notification.
  • It returned green within one repeat_interval after docker start.
  • The 24h route to the real channel confirmed by the temporary-repeat_interval method.
  • The date recorded — verify-the-alert-path.md is the natural place, the way restore-the-firewall.md carries "until that has been done once, this runbook is a hypothesis".

That last box is what stops this being re-litigated in six months, and it is the same gap #92 has: a mechanism built, a drill never run, and nothing in the repository that says which.

Activity

  1. Gerrrt commented on Sep 5, 2026

    @Gerrrt
    OwnerAuthor

    Folding in #312, which was split out of #120 independently 42 minutes after this
    one and describes the same drill. Closing that as the duplicate and keeping this
    one, because its "done when" is the more complete of the two — it carries the
    24h-route check, which #312 omits. Everything #312 had that this issue did not is
    below, so nothing is lost by closing it.

    Updated state of the mechanism, verified 2026-09-04

    This supersedes the numbers in the body above, all of it post-#284:

    • Four receivers on four distinct URLs, all four rendered and mounted.
    • group_interval: 1m, repeat_interval: 5m on the heartbeat route. Before
      fix(alertmanager): heartbeat pinged at half its advertised rate (#120) #284 it pinged at 600s
      , which put the 15m grace window under a single
      missed ping — the switch could not have tripped reliably even in principle.
    • Measured inter-ping gaps: 300s / 310s / 351s.
    • Zero clientError across every ping. Note precisely what that does and does
      not prove: the endpoint exists and accepts posts. It says nothing about
      whether anything alarms when the pings stop, which is the entire question this
      issue exists to answer.

    Two settings only you can read back

    Both live on the external check, are invisible from this host, and were sized
    against a repeat_interval that was wrong until #284:

    1. Period 5m, grace 15m. If the check was created before fix(alertmanager): heartbeat pinged at half its advertised rate (#120) #284 it may have
      been retuned to cope with the 600s cadence — a grace stretched to 20m+ to stop
      false pages. That compensation is now wrong in the other direction and should
      go back to 15m.
    2. The check's own notification must not land on this stack's ntfy topic.
      If both go to the same topic, a deleted topic takes out the alert and the
      warning about the alert together — the No dead man's switch on the notification path #67 failure, one level up. Email is
      fine here; it fails independently. This is already implied by the body's
      "must not be this stack's webhook", but the ntfy-topic version is the one that
      actually bites.

    Scheduling

    Beyond the Sunday 03:30 exclusion above: best run when you are near the phone and
    not on the clock. The estate has no alert delivery for the ~15 minutes the switch
    is stopped.

    Done when — merged list

    • The external check observed DOWN inside the grace window, with its
      notification arriving somewhere that is not this stack's webhook or
      ntfy topic.
    • Returned green within one repeat_interval after docker start.
    • Period and grace confirmed at 5m / 15m.
    • The 24h route to the real channel confirmed by the
      temporary-repeat_interval method.
    • The date recorded in docs/runbooks/verify-the-alert-path.md, the way
      restore-the-firewall.md carries "until that has been done once, this
      runbook is a hypothesis".

    Nothing here is actionable from this host — it needs a human watching a phone —
    so this stays open until someone runs it.

    🤖 Generated with Claude Code

  2. Gerrrt commented on Sep 6, 2026

    @Gerrrt
    OwnerAuthor

    Blocked on #359. Checked the
    preconditions before running the drill, and it cannot pass as written.

    heartbeat_url points at ntfy.sh, the same service as the three real alert
    channels (distinct topics, so not the "same topic" trap this issue's runbook
    warns about). ntfy is a push service — it delivers what it is sent and has no
    notion of an expected interval, so it cannot notice that a heartbeat stopped
    arriving. There is no external check to observe going DOWN.

    So the first two acceptance boxes here have nothing to test:

    • The external check has been observed DOWN and sent its notification.
    • It returned green within one repeat_interval after docker start.

    The heartbeat itself is healthy — roughly 5 webhook notifications per 30
    minutes, matching repeat_interval: 5m. It is being delivered to a topic nobody
    is waiting on.

    I did not run the drill. docker stop alertmanager for the 15-minute grace
    window would take alert delivery down for a quarter of an hour to demonstrate a
    silence that is already demonstrable from the configuration at no cost. If the
    outage is wanted anyway to document it, that is a deliberate choice rather than
    this issue's default.

    The third box — the 24h route to the real channel via a temporary
    repeat_interval — is also not something I can close alone: lowering it proves
    Alertmanager sent the notification, which is exactly #67's
    trap of a webhook that returns 200 and points at nothing. Confirming receipt
    needs a person looking at the channel.

    This issue stays open and becomes runnable once #359 puts a real watcher in
    place.

    🤖 Generated with Claude Code

  3. Gerrrt commented on Sep 8, 2026

    @Gerrrt
    OwnerAuthor

    Checked again on 2026-09-08, from the monitoring host, with the repository's
    own tool:

    $ python3 scripts/check_alert_channels.py --files --live observability
      WARN  heartbeat_url points at ntfy.sh, which is a PUSH service and cannot detect absence.
    

    So this is where #359 left it: #359 closed on detecting the gap (the
    deploy-time warning above, #382), not on filling it. There is still no
    external watcher, and the first two boxes here still have nothing to observe.
    The heartbeat itself is healthy — five pings per 25 minutes to a topic nobody
    is waiting on.

    The whole procedure, in order, for the day it is run

    Everything below is off this host or needs a person watching a phone; none of
    it is a repository change.

    1. Create the watcher. One check on a service that lives off this host and
      expects a ping — healthchecks.io, Cronitor, Dead Man's Snitch, or a
      self-hosted Uptime Kuma push monitor; check_alert_channels.py --live
      recognises all of them and will turn the warning into a pass. Period 5m,
      grace 15m
      , named so a 3am notification explains itself.
    2. Point its notification somewhere that is not this stack. Not this
      stack's webhook, not any of its four ntfy topics. Email is fine; it fails
      independently. A watcher that reports through the thing it watches cannot
      report its own failure — that is No dead man's switch on the notification path #67 one level up.
    3. Install the URL. make secrets-edit from the main checkout,
      ALERTMANAGER_HEARTBEAT_URL → the new check's ping URL, then
      make render && make reload. make up's deploy-time check should now say
      PASS for the heartbeat destination; the check should go green within one
      repeat_interval (5m).
    4. Pick the window. Not Sunday 03:30 (homelab-backup-volumes stops
      Prometheus and the heartbeat with it, and the result is unreadable). Near
      the phone, not on the clock: the estate has no alert delivery for the
      ~15 minutes the switch is stopped.
    5. Trip it. docker stop alertmanager from the main checkout; wait out the
      grace; the check must report DOWN and its notification must arrive on the
      channel from step 2. docker start alertmanager; it must return green
      within one repeat_interval.
    6. The 24h half, in the same sitting. Lower repeat_interval on the
      second Watchdog route in alertmanager.yaml (the one to default,
      currently 24h), make reload, confirm the Watchdog notification arrives
      on the normal ntfy channel, put it back, make reload. Mind the runbook's
      own warning: a grace under 10m turns every make reload into a page, so do
      this after step 5 has settled, not during it.
    7. Write the date down. docs/runbooks/verify-the-alert-path.md, in the
      house banner form the other runbooks use —
      > **Done YYYY-MM-DD.** … — replacing the This has not been done
      paragraph, and naming which watcher service and where its notification
      goes (not the URL). That line is what stops this being re-litigated.

    A recommendation on splitting it

    Step 6 is runnable today without a watcher, and I chose not to run it alone.
    Real alerts already arrive on that channel daily, so it proves little on its
    own; and a runbook that says "half of the drill was done on the 8th and the
    other half later" is exactly the kind of record that gets re-litigated. One
    sitting, one date. If you would rather have the 24h half proven now, it is a
    five-minute job from the main checkout and I can do it on request.

    🤖 Generated with Claude Code

  4. Gerrrt commented on Sep 9, 2026

    @Gerrrt
    OwnerAuthor

    Unblocked. The external watcher exists as of 2026-09-09: a healthchecks.io
    check, period 5m, grace 15m, notifying by email — not through ntfy, for the
    reason #407 gives. ALERTMANAGER_HEARTBEAT_URL points at it
    (#408), the deploy-time check
    reads the destination as hc-ping.com and passes, and 20 pings landed in the
    first two hours with zero failures.

    So the first two boxes now have something to test, and the drill is the whole
    of what is left. In one sitting:

    1. Not Sunday 03:30. Phone in hand; email is where the DOWN notice lands.
    2. From the main checkout: docker stop alertmanager. Wait out the 15-minute
      grace. The check must go DOWN and the email must arrive.
    3. docker start alertmanager. The check must return to green within one
      repeat_interval (5m).
    4. Lower the second Watchdog route's repeat_interval in alertmanager.yaml
      (the default one, 24h), make reload, confirm the Watchdog notification
      arrives on the normal ntfy channel, put it back, make reload. Do this
      after step 3 has settled — the runbook notes a make reload inside the
      grace window can itself trip the check.
    5. Record the date in verify-the-alert-path.md under Confirming it actually
      works
      , in the > **Done YYYY-MM-DD.** form; the watcher-setup banner
      already carries its own date and says the drill is what remains.

    Say when you are starting and I will do steps 2–4 from the main checkout on
    your word, with you watching the phone, and write step 5 afterwards.

    🤖 Generated with Claude Code

  5. Gerrrt commented on Sep 9, 2026

    @Gerrrt
    OwnerAuthor

    Run on 2026-09-09, both halves in one sitting, phone in hand. Every box in
    the merged list above is ticked by an observation rather than a claim.

    Box Evidence
    External check observed DOWN inside the grace window, notifying somewhere that is not this stack's webhook or ntfy topic docker stop alertmanager 03:07:37 UTC; healthchecks.io went DOWN and emailed at 03:25:58 — 18 minutes, against period 5m + grace 15m
    Returned green within one repeat_interval after docker start started 03:25:58, ready 03:26:03; first ping delivered 03:27:57, 2 minutes
    Period and grace confirmed at 5m / 15m set when the check was created earlier tonight; the 18-minute trip is consistent with exactly those
    24h route confirmed by the temporary-repeat_interval method route lowered to 2m and reloaded 03:28:29; Watchdog arrived on the normal ntfy channel at 03:28; route restored from git and reloaded 03:33:16, main checkout clean
    Date recorded in verify-the-alert-path.md #409, in the > **Done 2026-09-09** form the other runbooks use

    Zero delivery failures across the window, and nothing paged that should not
    have — Alertmanager was stopped, so nothing could.

    One observation from the daily half, recorded in the runbook rather than
    fixed: a route whose repeat_interval sits below the root's 5m
    group_interval repeats on the 5m tick, not at its own interval. The first
    notification still goes out immediately, so the test works; it is the same
    coupling the heartbeat route already documents, and it does not touch the
    real 24h setting.

    The watcher this issue was blocked on since #359 was set up the same evening
    (#408), which also took the heartbeat's ~240 messages a day off the ntfy
    budget (#407). Closing on #409's merge.

    🤖 Generated with Claude Code

  6. added a commit that references this issue on Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureA new capability or a change

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions