Skip to content

Fill in the three new notification secrets, then deploy #118 and #119 #120

Closed

Description

@Gerrrt

#118 and #119 are merged but not deployed. Both add SOPS keys that only you
can fill in, so make up will refuse to run until they exist. This is the
intended behaviour — render-config.sh dies naming the missing keys rather than
starting a stack with half a routing tree — but it means the change is sitting
in main doing nothing.

The three keys

make secrets-edit
Key What it is
ALERTMANAGER_URGENT_WEBHOOK_URL ntfy topic for criticals
ALERTMANAGER_SECURITY_WEBHOOK_URL ntfy topic for category=security
ALERTMANAGER_HEARTBEAT_URL cron-monitor ping URL, not an ntfy topic

For the first two, three topics on one ntfy server is the whole setup — the topic
is the last path segment and needs no creation step. The point of separate topics
is the per-topic sound and do-not-disturb settings on the phone; pointing them all
at the same topic works and breaks nothing, but then the receiver names are lying
again, which is what #66 was about.

The third comes from something that lives off this host — healthchecks.io
(free tier covers one check), Cronitor, or an Uptime Kuma push monitor on another
machine. A watcher on the monitoring host fails at the same moment as the thing
it is watching.

make up

Then verify

docs/runbooks/verify-the-alert-path.md is the full procedure. The short version:

docker exec alertmanager ls /etc/alertmanager/secrets/   # four files
amtool alert add --alertmanager.url=http://localhost:9093 \
  alertname=RouteTest severity=warning category=security  # lands on the security topic

Set the external check's period to 5m and grace to 15m — both are tied to
repeat_interval on the heartbeat route, and the runbook says what breaks if you
move one without the other. Point that check's own notification somewhere that is
not this stack's webhook; if both land on the same topic, a deleted topic
takes out the alert and the warning about the alert together.

Then, once, actually watch it trip. A dead man's switch nobody has seen go red is
indistinguishable from one that does not work:

docker stop alertmanager    # external check must go DOWN inside the grace window
docker start alertmanager   # must return green within one repeat_interval

Until then — the current state, precisely

The running Alertmanager still holds the old config in memory (verified via
/api/v2/status: three receivers, default and urgent both on webhook_url),
because it only re-reads on reload or restart and neither has happened. Every
alert is being delivered normally right now.

alertmanager/.rendered/ holds only webhook_url, unchanged. Nothing points at
a placeholder — throwaway URLs used to exercise render-config.sh were removed
after testing.

The one gap: alertmanager.yaml is a bind mount, so an unexpected restart
before the keys are filled in would load the new five-receiver config and find
urgent_url, security_url and heartbeat_url missing. Warnings on the
default route would still deliver; criticals and security alerts would not.
Uptime is currently 3.85 days with RestartCount=0 and restart: unless-stopped,
so this needs a reboot or a manual restart to happen — but do not make reload
or restart the stack before running make secrets-edit.

Filed from the session that merged #117, #118 and #119.

Activity

  1. Gerrrt commented on Sep 4, 2026

    @Gerrrt
    OwnerAuthor

    The state section in this issue is out of date, and the verification step found a real bug.

    The deploy already happened

    Checked against the live stack today:

    • All four keys present in secrets/observability.sops.yaml.
    • .rendered/ holds four distinct files (not just webhook_url), no placeholders.
    • The container mounts all four at /etc/alertmanager/secrets/.
    • /api/v2/status shows the five-receiver config with all four url_file paths — not the three-receiver config described above.
    • Alertmanager recreated 2026-09-04T02:54Z; 124 webhook notifications, zero failures.

    So the "one gap" this issue warns about is closed, and the warning against make reload no longer applies.

    What was still broken

    The heartbeat route had group_interval: 5m alongside repeat_interval: 5m. A group is only reconsidered on a group_interval tick, and only then asks whether repeat_interval has elapsed — so the +5m tick landed just before the deadline it was testing, skipped, and fired at +10m.

    Measured rather than inferred: increase(alertmanager_notifications_total{integration="webhook"}[6h]) = 37.03. Only Watchdog routes to heartbeat, and the two firing warnings on default repeat at 12h, so that is 36 pings in 6h — exactly 600s each. A true 5m heartbeat is 72.

    Nothing inside the stack could show this: every notification succeeded and alertmanager_notifications_failed_total was zero everywhere.

    This mattered for the instructions above. The rule "grace ≥ 2 × repeat_interval" exists so one missed ping is tolerated and two are a fault. Against a real 10m cadence, the 15m grace this issue specifies was already under a single missed ping — the next make reload or the Sunday 03:30 backup would have paged for a healthy stack.

    Fixed in #284 (group_interval: 1m). Routing untouched; 8/8 amtool route assertions unchanged.

    Remaining, and why this issue should stay open

    1. make render && make reload from the main checkout once fix(alertmanager): heartbeat pinged at half its advertised rate (#120) #284 merges.
    2. External check at period 5m, grace 15m — correct as written after fix(alertmanager): heartbeat pinged at half its advertised rate (#120) #284, wrong before it.
    3. Actually watch it trip: docker stop alertmanager.
  2. Gerrrt commented on Sep 4, 2026

    @Gerrrt
    OwnerAuthor

    Done — the keys are filled in and the stack is running the new configuration. Closing.

    Verified on the live stack, 2026-09-04

    The running Alertmanager now holds the five-receiver config, where this issue recorded three:

    receivers: null, default, urgent, security, heartbeat
    

    Each notifying receiver reads its own secret, and the four secret files exist, are non-empty, and are all different from each other:

    $ docker exec alertmanager ls -l /etc/alertmanager/secrets/
    48  webhook_url
    40  urgent_url
    40  security_url
    40  heartbeat_url

    That covers the concern raised in the description about pointing them all at the same topic — the four destinations are genuinely distinct, so the receiver names are not lying.

    All four keys are present in secrets/observability.sops.yaml: ALERTMANAGER_WEBHOOK_URL, ALERTMANAGER_URGENT_WEBHOOK_URL, ALERTMANAGER_SECURITY_WEBHOOK_URL, ALERTMANAGER_HEARTBEAT_URL.

    The gap the description called out — an unexpected restart loading the five-receiver config against missing keys — is closed by the same fact: the config is loaded and the keys are there, so criticals and security alerts route rather than silently failing.

    One step from the description I cannot show evidence for

    Then, once, actually watch it trip. A dead man's switch nobody has seen go red is indistinguishable from one that does not work.

    Deploying is verifiable from here; the docker stop alertmanager → external check goes DOWN inside the grace window → docker start → green within one repeat_interval rehearsal is not, and I found no record of it. Same argument as #92's restore rehearsal: the mechanism is built, the drill is what tells you it works.

    If that has already been done, this is a no-op. If not, it is worth its own issue rather than keeping this one open for it — the deploy that this issue is titled for is finished, and a rehearsal is a different kind of task with a different definition of done.

  3. added a commit that references this issue on Sep 4, 2026
  4. Gerrrt commented on Sep 4, 2026

    @Gerrrt
    OwnerAuthor

    Filed the outstanding rehearsal as #288 so it is not lost with this issue closed.

  5. Gerrrt commented on Sep 4, 2026

    @Gerrrt
    OwnerAuthor

    Closing. Summary of what this issue actually turned out to be.

    The secrets and the deploy

    Already done before this session started — all four keys present, four distinct rendered files, four mounted in the container, five-receiver config live. The "current state" section in the description was stale; see the earlier comment.

    What the verification step found

    The heartbeat route had group_interval equal to repeat_interval (both 5m), which made Alertmanager ping at 600s, not 300s — every notification a success, _failed_total zero, config file reading 5m. Fixed in #284 with group_interval: 1m.

    This mattered because it silently voided the sizing rule in this issue: at a real 10m cadence the 15m grace specified above was already under a single missed ping, so the next reload or the Sunday backup would have paged for a healthy stack.

    Deploying it also surfaced a second trap worth recording: alertmanager.yaml is a single-file bind mount, pinned to its inode. git pull renames a new file over the old one, so the container keeps reading the unlinked original and POST /-/reload returns HTTP 200 having re-read stale config. Plain up -d does not fix it either — it recreates only on a service-definition change. --force-recreate is required, and verification has to happen inside the container, not against the host file.

    Verified live

    • group_interval: 1m, repeat_interval: 5m, 5 receivers, 4 url_file paths
    • Measured inter-ping gaps of 300s / 310s / 351s (was 600s); the window with 1m ticks against a 5m deadline is 300–360s
    • Zero clientError across all pings, which means the heartbeat endpoint is accepting them — so a check does exist on the far end

    Not verified, and deliberately recorded as such

    The switch has never been watched to trip. docker stop alertmanager → external check goes DOWN inside the grace window → docker start → green within one repeat_interval has not been run, and the external check's own period/grace settings have not been read back by anyone but their owner.

    By this issue's own standard — "a dead man's switch nobody has seen go red is indistinguishable from one that does not work" — that half is still untested. Closing on the code being correct and delivery being proven, not on the end-to-end drill.

  6. Gerrrt commented on Sep 4, 2026

    @Gerrrt
    OwnerAuthor

    The untested half is now tracked separately in #312 — watching the switch trip once, plus reading back the external check's period and grace, which are the two things that cannot be verified from the monitoring host.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureA new capability or a change

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions