Repository navigation
fix(alertmanager): heartbeat pinged at half its advertised rate (#120) - #284
Merged
Merged
Conversation
The heartbeat route set group_interval and repeat_interval both to 5m. A group is only reconsidered on a group_interval tick, and only then asks whether repeat_interval has elapsed, so the +5m tick arrived fractionally before the deadline it was testing, skipped, and fired at +10m. The failure is invisible from inside the stack: every notification still succeeds, alertmanager_notifications_failed_total stays at zero, and the config says 5m. Measured on the live stack at 600s per ping against the 300s this file claimed — 37 webhook notifications in 6h where a true 5m heartbeat is 72. It also silently voided the runbook's sizing rule. "Grace >= 2 x repeat_interval" was picked so one missed ping is tolerated and two are a fault; against a real 10m cadence the documented 15m grace was already under a single missed ping, so the next reload or the Sunday backup would have paged for a stack that was fine. group_interval: 1m makes the tick granularity finer than the deadline, so the ping lands at 5m and never later than 6m. Routing is untouched — the 8 amtool route assertions pass unchanged. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes part of #120.
The deploy in #120 is already done
The issue's "current state" section is stale. Verified on the live stack:
secrets/observability.sops.yaml..rendered/holds four distinct files, no placeholders./etc/alertmanager/secrets/./api/v2/statusshows the five-receiver config, with all foururl_filepaths — not the three-receiver config the issue describes.
zero failures.
So the secrets were filled in and the stack deployed some time after the issue
was written. What had not happened is the verification — and that turned up a
real defect.
The defect
The heartbeat route set
group_interval: 5malongsiderepeat_interval: 5m.A group is only reconsidered on a
group_intervaltick, and only then askswhether
repeat_intervalhas elapsed. With the two equal, the +5m tick landsfractionally before the deadline it is testing, skips, and fires at +10m.
Measured, not inferred:
Only
Watchdogroutes toheartbeat; the two firing warnings ondefaultrepeat at 12h and contribute ~1 over that window. 36 pings in 6h is exactly
600s per ping. A true 5m heartbeat is 72.
Nothing inside the stack could see this. Every notification succeeded,
alertmanager_notifications_failed_totalwas zero across every reason, and theconfig file said
5m.Why it mattered more than a factor of two
It silently voided the runbook's own sizing rule. "Grace ≥ 2 ×
repeat_interval" exists so that one missed ping is a tolerated hiccup andtwo consecutive misses are a fault. Against a real 10m cadence, the documented
15m grace was already under a single missed ping — so the next
make reload,or the Sunday 03:30 backup window, would have paged for a stack that was
perfectly healthy. The issue instructs setting period 5m / grace 15m; that
would have been wrong on arrival.
The change
group_interval: 1mon the heartbeat route only. Tick granularity is now finerthan the deadline, so the ping lands at 5m and never later than 6m, and the
documented 5m/15m external sizing becomes correct as written.
Routing is untouched — the 8
amtool config routes testassertions passunchanged, and
amtool check-configstill reports 5 receivers.The runbook gains the constraint and a query to check the real rate rather
than reading it off the config, since this is a failure mode the config cannot
show you.
Verification
amtool check-config— SUCCESS, 5 receivers./scripts/validate.sh— all checks passed (2 skipped: not the deploymentcheckout, no stack running in this worktree)
Still needs the maintainer
Not doable from this session, and #120 stays open for them:
make render && make reloadfrom the main checkout(
/home/robo/code/Gerrrt/HomeLab); a worktree render writes a file nocontainer mounts.
correct, given this fix.
docker stop alertmanager).🤖 Generated with Claude Code