Repository navigation
Description
#118 and #119 are merged but not deployed. Both add SOPS keys that only you
can fill in, so make up will refuse to run until they exist. This is the
intended behaviour — render-config.sh dies naming the missing keys rather than
starting a stack with half a routing tree — but it means the change is sitting
in main doing nothing.
The three keys
make secrets-edit| Key | What it is |
|---|---|
ALERTMANAGER_URGENT_WEBHOOK_URL |
ntfy topic for criticals |
ALERTMANAGER_SECURITY_WEBHOOK_URL |
ntfy topic for category=security |
ALERTMANAGER_HEARTBEAT_URL |
cron-monitor ping URL, not an ntfy topic |
For the first two, three topics on one ntfy server is the whole setup — the topic
is the last path segment and needs no creation step. The point of separate topics
is the per-topic sound and do-not-disturb settings on the phone; pointing them all
at the same topic works and breaks nothing, but then the receiver names are lying
again, which is what #66 was about.
The third comes from something that lives off this host — healthchecks.io
(free tier covers one check), Cronitor, or an Uptime Kuma push monitor on another
machine. A watcher on the monitoring host fails at the same moment as the thing
it is watching.
make upThen verify
docs/runbooks/verify-the-alert-path.md is the full procedure. The short version:
docker exec alertmanager ls /etc/alertmanager/secrets/ # four files
amtool alert add --alertmanager.url=http://localhost:9093 \
alertname=RouteTest severity=warning category=security # lands on the security topicSet the external check's period to 5m and grace to 15m — both are tied to
repeat_interval on the heartbeat route, and the runbook says what breaks if you
move one without the other. Point that check's own notification somewhere that is
not this stack's webhook; if both land on the same topic, a deleted topic
takes out the alert and the warning about the alert together.
Then, once, actually watch it trip. A dead man's switch nobody has seen go red is
indistinguishable from one that does not work:
docker stop alertmanager # external check must go DOWN inside the grace window
docker start alertmanager # must return green within one repeat_intervalUntil then — the current state, precisely
The running Alertmanager still holds the old config in memory (verified via
/api/v2/status: three receivers, default and urgent both on webhook_url),
because it only re-reads on reload or restart and neither has happened. Every
alert is being delivered normally right now.
alertmanager/.rendered/ holds only webhook_url, unchanged. Nothing points at
a placeholder — throwaway URLs used to exercise render-config.sh were removed
after testing.
The one gap: alertmanager.yaml is a bind mount, so an unexpected restart
before the keys are filled in would load the new five-receiver config and find
urgent_url, security_url and heartbeat_url missing. Warnings on the
default route would still deliver; criticals and security alerts would not.
Uptime is currently 3.85 days with RestartCount=0 and restart: unless-stopped,
so this needs a reboot or a manual restart to happen — but do not make reload
or restart the stack before running make secrets-edit.
Activity
Gerrrt commentedon Sep 4, 2026
The state section in this issue is out of date, and the verification step found a real bug.
The deploy already happened
Checked against the live stack today:
secrets/observability.sops.yaml..rendered/holds four distinct files (not justwebhook_url), no placeholders./etc/alertmanager/secrets/./api/v2/statusshows the five-receiver config with all foururl_filepaths — not the three-receiver config described above.2026-09-04T02:54Z; 124 webhook notifications, zero failures.So the "one gap" this issue warns about is closed, and the warning against
make reloadno longer applies.What was still broken
The heartbeat route had
group_interval: 5malongsiderepeat_interval: 5m. A group is only reconsidered on agroup_intervaltick, and only then asks whetherrepeat_intervalhas elapsed — so the +5m tick landed just before the deadline it was testing, skipped, and fired at +10m.Measured rather than inferred:
increase(alertmanager_notifications_total{integration="webhook"}[6h])= 37.03. OnlyWatchdogroutes toheartbeat, and the two firing warnings ondefaultrepeat at 12h, so that is 36 pings in 6h — exactly 600s each. A true 5m heartbeat is 72.Nothing inside the stack could show this: every notification succeeded and
alertmanager_notifications_failed_totalwas zero everywhere.This mattered for the instructions above. The rule "grace ≥ 2 ×
repeat_interval" exists so one missed ping is tolerated and two are a fault. Against a real 10m cadence, the 15m grace this issue specifies was already under a single missed ping — the nextmake reloador the Sunday 03:30 backup would have paged for a healthy stack.Fixed in #284 (
group_interval: 1m). Routing untouched; 8/8amtoolroute assertions unchanged.Remaining, and why this issue should stay open
make render && make reloadfrom the main checkout once fix(alertmanager): heartbeat pinged at half its advertised rate (#120) #284 merges.docker stop alertmanager.Gerrrt commentedon Sep 4, 2026
Done — the keys are filled in and the stack is running the new configuration. Closing.
Verified on the live stack, 2026-09-04
The running Alertmanager now holds the five-receiver config, where this issue recorded three:
Each notifying receiver reads its own secret, and the four secret files exist, are non-empty, and are all different from each other:
That covers the concern raised in the description about pointing them all at the same topic — the four destinations are genuinely distinct, so the receiver names are not lying.
All four keys are present in
secrets/observability.sops.yaml:ALERTMANAGER_WEBHOOK_URL,ALERTMANAGER_URGENT_WEBHOOK_URL,ALERTMANAGER_SECURITY_WEBHOOK_URL,ALERTMANAGER_HEARTBEAT_URL.The gap the description called out — an unexpected restart loading the five-receiver config against missing keys — is closed by the same fact: the config is loaded and the keys are there, so criticals and security alerts route rather than silently failing.
One step from the description I cannot show evidence for
Deploying is verifiable from here; the
docker stop alertmanager→ external check goes DOWN inside the grace window →docker start→ green within onerepeat_intervalrehearsal is not, and I found no record of it. Same argument as #92's restore rehearsal: the mechanism is built, the drill is what tells you it works.If that has already been done, this is a no-op. If not, it is worth its own issue rather than keeping this one open for it — the deploy that this issue is titled for is finished, and a rehearsal is a different kind of task with a different definition of done.
Merge pull request #284 from Gerrrt/gerrrt/notification-secrets-deplo…
Gerrrt commentedon Sep 4, 2026
Filed the outstanding rehearsal as #288 so it is not lost with this issue closed.
Gerrrt commentedon Sep 4, 2026
Closing. Summary of what this issue actually turned out to be.
The secrets and the deploy
Already done before this session started — all four keys present, four distinct rendered files, four mounted in the container, five-receiver config live. The "current state" section in the description was stale; see the earlier comment.
What the verification step found
The heartbeat route had
group_intervalequal torepeat_interval(both 5m), which made Alertmanager ping at 600s, not 300s — every notification a success,_failed_totalzero, config file reading5m. Fixed in #284 withgroup_interval: 1m.This mattered because it silently voided the sizing rule in this issue: at a real 10m cadence the 15m grace specified above was already under a single missed ping, so the next reload or the Sunday backup would have paged for a healthy stack.
Deploying it also surfaced a second trap worth recording:
alertmanager.yamlis a single-file bind mount, pinned to its inode.git pullrenames a new file over the old one, so the container keeps reading the unlinked original andPOST /-/reloadreturns HTTP 200 having re-read stale config. Plainup -ddoes not fix it either — it recreates only on a service-definition change.--force-recreateis required, and verification has to happen inside the container, not against the host file.Verified live
group_interval: 1m,repeat_interval: 5m, 5 receivers, 4url_filepathsclientErroracross all pings, which means the heartbeat endpoint is accepting them — so a check does exist on the far endNot verified, and deliberately recorded as such
The switch has never been watched to trip.
docker stop alertmanager→ external check goes DOWN inside the grace window →docker start→ green within onerepeat_intervalhas not been run, and the external check's own period/grace settings have not been read back by anyone but their owner.By this issue's own standard — "a dead man's switch nobody has seen go red is indistinguishable from one that does not work" — that half is still untested. Closing on the code being correct and delivery being proven, not on the end-to-end drill.
Gerrrt commentedon Sep 4, 2026
The untested half is now tracked separately in #312 — watching the switch trip once, plus reading back the external check's period and grace, which are the two things that cannot be verified from the monitoring host.