Skip to content

fix(alertmanager): heartbeat pinged at half its advertised rate (#120) - #284

Merged
Gerrrt merged 1 commit into
mainfrom
gerrrt/notification-secrets-deploy-631bde
Sep 4, 2026
Merged

Gerrrt merged 1 commit into
mainfrom
gerrrt/notification-secrets-deploy-631bde

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Sep 4, 2026

Copy link
Copy Markdown
Owner

Closes part of #120.

The deploy in #120 is already done

The issue's "current state" section is stale. Verified on the live stack:

  • All four keys are present in secrets/observability.sops.yaml.
  • .rendered/ holds four distinct files, no placeholders.
  • The container mounts all four at /etc/alertmanager/secrets/.
  • /api/v2/status shows the five-receiver config, with all four url_file
    paths — not the three-receiver config the issue describes.
  • Alertmanager was recreated 2026-09-04T02:54Z; 124 webhook notifications,
    zero failures.

So the secrets were filled in and the stack deployed some time after the issue
was written. What had not happened is the verification — and that turned up a
real defect.

The defect

The heartbeat route set group_interval: 5m alongside repeat_interval: 5m.

A group is only reconsidered on a group_interval tick, and only then asks
whether repeat_interval has elapsed. With the two equal, the +5m tick lands
fractionally before the deadline it is testing, skips, and fires at +10m.

Measured, not inferred:

increase(alertmanager_notifications_total{integration="webhook"}[6h]) = 37.03

Only Watchdog routes to heartbeat; the two firing warnings on default
repeat at 12h and contribute ~1 over that window. 36 pings in 6h is exactly
600s per ping. A true 5m heartbeat is 72.

Nothing inside the stack could see this. Every notification succeeded,
alertmanager_notifications_failed_total was zero across every reason, and the
config file said 5m.

Why it mattered more than a factor of two

It silently voided the runbook's own sizing rule. "Grace ≥ 2 ×
repeat_interval"
exists so that one missed ping is a tolerated hiccup and
two consecutive misses are a fault. Against a real 10m cadence, the documented
15m grace was already under a single missed ping — so the next make reload,
or the Sunday 03:30 backup window, would have paged for a stack that was
perfectly healthy. The issue instructs setting period 5m / grace 15m; that
would have been wrong on arrival.

The change

group_interval: 1m on the heartbeat route only. Tick granularity is now finer
than the deadline, so the ping lands at 5m and never later than 6m, and the
documented 5m/15m external sizing becomes correct as written.

Routing is untouched — the 8 amtool config routes test assertions pass
unchanged, and amtool check-config still reports 5 receivers.

The runbook gains the constraint and a query to check the real rate rather
than reading it off the config, since this is a failure mode the config cannot
show you.

Verification

  • amtool check-config — SUCCESS, 5 receivers
  • ./scripts/validate.sh — all checks passed (2 skipped: not the deployment
    checkout, no stack running in this worktree)
  • 8/8 route assertions unchanged

Still needs the maintainer

Not doable from this session, and #120 stays open for them:

  1. Deploy — make render && make reload from the main checkout
    (/home/robo/code/Gerrrt/HomeLab); a worktree render writes a file no
    container mounts.
  2. Create/confirm the external check with period 5m, grace 15m — now
    correct, given this fix.
  3. Watch it actually trip once (docker stop alertmanager).

🤖 Generated with Claude Code

The heartbeat route set group_interval and repeat_interval both to 5m. A
group is only reconsidered on a group_interval tick, and only then asks
whether repeat_interval has elapsed, so the +5m tick arrived fractionally
before the deadline it was testing, skipped, and fired at +10m.

The failure is invisible from inside the stack: every notification still
succeeds, alertmanager_notifications_failed_total stays at zero, and the
config says 5m. Measured on the live stack at 600s per ping against the
300s this file claimed — 37 webhook notifications in 6h where a true 5m
heartbeat is 72.

It also silently voided the runbook's sizing rule. "Grace >= 2 x
repeat_interval" was picked so one missed ping is tolerated and two are a
fault; against a real 10m cadence the documented 15m grace was already
under a single missed ping, so the next reload or the Sunday backup would
have paged for a stack that was fine.

group_interval: 1m makes the tick granularity finer than the deadline, so
the ping lands at 5m and never later than 6m. Routing is untouched — the
8 amtool route assertions pass unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Gerrrt
Gerrrt merged commit 3e327ed into main Sep 4, 2026
3 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/notification-secrets-deploy-631bde branch September 4, 2026 22:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant