Repository navigation
Description
Split out of #120, which is closed: the three notification secrets are filled in, the five-receiver config is live, and each receiver reads its own non-empty secret. What #120 could not close is its own last instruction:
Then, once, actually watch it trip. A dead man's switch nobody has seen go red is indistinguishable from one that does not work.
The mechanism is built and verifiable from the repository. The drill is not, and there is no record of it having been run. That is a different kind of task with a different definition of done, which is why it is here rather than holding #120 open.
What exists today
watchdog.rules.yamlemits the always-firingWatchdogalert.alertmanager.yaml:61-67routes it to theheartbeatreceiver withgroup_wait: 0s,group_interval: 1m,repeat_interval: 5m,continue: true.- A second
Watchdogroute sends 24h to the real alert channel — the half that catches a webhook which is well-formed, reachable, returns 200 and points at nothing (No dead man's switch on the notification path #67's actual failure). ALERTMANAGER_HEARTBEAT_URLis set and the external check is green.
A green check that has never been seen go red is the thing this issue exists to fix.
The procedure
Already written, in docs/runbooks/verify-the-alert-path.md under "Confirming it actually works":
docker stop alertmanager
# wait out the grace window — 15m by default
# the external check must report DOWN and notify you
docker start alertmanager
# it must return to green within one repeat_intervalNothing needs to be written. This is execution, and then a line somewhere saying it happened.
The runbook's second paragraph there covers the daily half without waiting a day: temporarily lower repeat_interval on the second Watchdog route, make reload, and check the real channel.
Pick the window deliberately
docker stop alertmanager for the length of the grace window means the estate has no alert delivery for ~15 minutes. Prometheus keeps evaluating and Alertmanager will deliver what is still firing when it comes back, but anything that resolves inside the window is never notified. Two constraints from the runbook's own "timing is coupled" section apply:
- Not Sunday 03:30 —
homelab-backup-volumesquiesces the stack then, which stops Prometheus evaluating and therefore stops the heartbeat. Overlapping the two makes the result unreadable. - The external check's notification must not be this stack's webhook. If it is, the test cannot report its own failure.
Done when
- The external check has been observed DOWN and sent its notification.
- It returned green within one
repeat_intervalafterdocker start. - The 24h route to the real channel confirmed by the temporary-
repeat_intervalmethod. - The date recorded —
verify-the-alert-path.mdis the natural place, the wayrestore-the-firewall.mdcarries "until that has been done once, this runbook is a hypothesis".
That last box is what stops this being re-litigated in six months, and it is the same gap #92 has: a mechanism built, a drill never run, and nothing in the repository that says which.
Activity
Gerrrt commentedon Sep 5, 2026
Folding in #312, which was split out of #120 independently 42 minutes after this
one and describes the same drill. Closing that as the duplicate and keeping this
one, because its "done when" is the more complete of the two — it carries the
24h-route check, which #312 omits. Everything #312 had that this issue did not is
below, so nothing is lost by closing it.
Updated state of the mechanism, verified 2026-09-04
This supersedes the numbers in the body above, all of it post-#284:
group_interval: 1m,repeat_interval: 5mon the heartbeat route. Beforefix(alertmanager): heartbeat pinged at half its advertised rate (#120) #284 it pinged at 600s, which put the 15m grace window under a single
missed ping — the switch could not have tripped reliably even in principle.
clientErroracross every ping. Note precisely what that does and doesnot prove: the endpoint exists and accepts posts. It says nothing about
whether anything alarms when the pings stop, which is the entire question this
issue exists to answer.
Two settings only you can read back
Both live on the external check, are invisible from this host, and were sized
against a
repeat_intervalthat was wrong until #284:been retuned to cope with the 600s cadence — a grace stretched to 20m+ to stop
false pages. That compensation is now wrong in the other direction and should
go back to 15m.
If both go to the same topic, a deleted topic takes out the alert and the
warning about the alert together — the No dead man's switch on the notification path #67 failure, one level up. Email is
fine here; it fails independently. This is already implied by the body's
"must not be this stack's webhook", but the ntfy-topic version is the one that
actually bites.
Scheduling
Beyond the Sunday 03:30 exclusion above: best run when you are near the phone and
not on the clock. The estate has no alert delivery for the ~15 minutes the switch
is stopped.
Done when — merged list
notification arriving somewhere that is not this stack's webhook or
ntfy topic.
repeat_intervalafterdocker start.temporary-
repeat_intervalmethod.docs/runbooks/verify-the-alert-path.md, the wayrestore-the-firewall.mdcarries "until that has been done once, thisrunbook is a hypothesis".
Nothing here is actionable from this host — it needs a human watching a phone —
so this stays open until someone runs it.
🤖 Generated with Claude Code
Gerrrt commentedon Sep 6, 2026
Blocked on #359. Checked the
preconditions before running the drill, and it cannot pass as written.
heartbeat_urlpoints atntfy.sh, the same service as the three real alertchannels (distinct topics, so not the "same topic" trap this issue's runbook
warns about). ntfy is a push service — it delivers what it is sent and has no
notion of an expected interval, so it cannot notice that a heartbeat stopped
arriving. There is no external check to observe going DOWN.
So the first two acceptance boxes here have nothing to test:
repeat_intervalafterdocker start.The heartbeat itself is healthy — roughly 5 webhook notifications per 30
minutes, matching
repeat_interval: 5m. It is being delivered to a topic nobodyis waiting on.
I did not run the drill.
docker stop alertmanagerfor the 15-minute gracewindow would take alert delivery down for a quarter of an hour to demonstrate a
silence that is already demonstrable from the configuration at no cost. If the
outage is wanted anyway to document it, that is a deliberate choice rather than
this issue's default.
The third box — the 24h route to the real channel via a temporary
repeat_interval— is also not something I can close alone: lowering it provesAlertmanager sent the notification, which is exactly #67's
trap of a webhook that returns 200 and points at nothing. Confirming receipt
needs a person looking at the channel.
This issue stays open and becomes runnable once #359 puts a real watcher in
place.
🤖 Generated with Claude Code
Gerrrt commentedon Sep 8, 2026
Checked again on 2026-09-08, from the monitoring host, with the repository's
own tool:
So this is where #359 left it: #359 closed on detecting the gap (the
deploy-time warning above, #382), not on filling it. There is still no
external watcher, and the first two boxes here still have nothing to observe.
The heartbeat itself is healthy — five pings per 25 minutes to a topic nobody
is waiting on.
The whole procedure, in order, for the day it is run
Everything below is off this host or needs a person watching a phone; none of
it is a repository change.
expects a ping — healthchecks.io, Cronitor, Dead Man's Snitch, or a
self-hosted Uptime Kuma push monitor;
check_alert_channels.py --liverecognises all of them and will turn the warning into a pass. Period 5m,
grace 15m, named so a 3am notification explains itself.
stack's webhook, not any of its four ntfy topics. Email is fine; it fails
independently. A watcher that reports through the thing it watches cannot
report its own failure — that is No dead man's switch on the notification path #67 one level up.
make secrets-editfrom the main checkout,ALERTMANAGER_HEARTBEAT_URL→ the new check's ping URL, thenmake render && make reload.make up's deploy-time check should now sayPASS for the heartbeat destination; the check should go green within one
repeat_interval(5m).homelab-backup-volumesstopsPrometheus and the heartbeat with it, and the result is unreadable). Near
the phone, not on the clock: the estate has no alert delivery for the
~15 minutes the switch is stopped.
docker stop alertmanagerfrom the main checkout; wait out thegrace; the check must report DOWN and its notification must arrive on the
channel from step 2.
docker start alertmanager; it must return greenwithin one
repeat_interval.repeat_intervalon thesecond Watchdog route in
alertmanager.yaml(the one todefault,currently
24h),make reload, confirm the Watchdog notification arriveson the normal ntfy channel, put it back,
make reload. Mind the runbook'sown warning: a grace under 10m turns every
make reloadinto a page, so dothis after step 5 has settled, not during it.
docs/runbooks/verify-the-alert-path.md, in thehouse banner form the other runbooks use —
> **Done YYYY-MM-DD.** …— replacing the This has not been doneparagraph, and naming which watcher service and where its notification
goes (not the URL). That line is what stops this being re-litigated.
A recommendation on splitting it
Step 6 is runnable today without a watcher, and I chose not to run it alone.
Real alerts already arrive on that channel daily, so it proves little on its
own; and a runbook that says "half of the drill was done on the 8th and the
other half later" is exactly the kind of record that gets re-litigated. One
sitting, one date. If you would rather have the 24h half proven now, it is a
five-minute job from the main checkout and I can do it on request.
🤖 Generated with Claude Code
Gerrrt commentedon Sep 9, 2026
Unblocked. The external watcher exists as of 2026-09-09: a healthchecks.io
check, period 5m, grace 15m, notifying by email — not through ntfy, for the
reason #407 gives.
ALERTMANAGER_HEARTBEAT_URLpoints at it(#408), the deploy-time check
reads the destination as
hc-ping.comand passes, and 20 pings landed in thefirst two hours with zero failures.
So the first two boxes now have something to test, and the drill is the whole
of what is left. In one sitting:
docker stop alertmanager. Wait out the 15-minutegrace. The check must go DOWN and the email must arrive.
docker start alertmanager. The check must return to green within onerepeat_interval(5m).repeat_intervalinalertmanager.yaml(the
defaultone,24h),make reload, confirm the Watchdog notificationarrives on the normal ntfy channel, put it back,
make reload. Do thisafter step 3 has settled — the runbook notes a
make reloadinside thegrace window can itself trip the check.
verify-the-alert-path.mdunder Confirming it actuallyworks, in the
> **Done YYYY-MM-DD.**form; the watcher-setup banneralready carries its own date and says the drill is what remains.
Say when you are starting and I will do steps 2–4 from the main checkout on
your word, with you watching the phone, and write step 5 afterwards.
🤖 Generated with Claude Code
Gerrrt commentedon Sep 9, 2026
Run on 2026-09-09, both halves in one sitting, phone in hand. Every box in
the merged list above is ticked by an observation rather than a claim.
docker stop alertmanager03:07:37 UTC; healthchecks.io went DOWN and emailed at 03:25:58 — 18 minutes, against period 5m + grace 15mrepeat_intervalafterdocker startrepeat_intervalmethodverify-the-alert-path.md> **Done 2026-09-09**form the other runbooks useZero delivery failures across the window, and nothing paged that should not
have — Alertmanager was stopped, so nothing could.
One observation from the daily half, recorded in the runbook rather than
fixed: a route whose
repeat_intervalsits below the root's 5mgroup_intervalrepeats on the 5m tick, not at its own interval. The firstnotification still goes out immediately, so the test works; it is the same
coupling the heartbeat route already documents, and it does not touch the
real 24h setting.
The watcher this issue was blocked on since #359 was set up the same evening
(#408), which also took the heartbeat's ~240 messages a day off the ntfy
budget (#407). Closing on #409's merge.
🤖 Generated with Claude Code
Merge pull request #409 from Gerrrt/gerrrt/deadmans-switch-drill-288