You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
ContainerHighMemory cannot fire — no service sets a memory limit #63
container_memory_working_set_bytes{name!=""}
/ container_spec_memory_limit_bytes{name!=""} > 0.90
and container_spec_memory_limit_bytes{name!=""} > 0
The guard is correct and the alert is unreachable, because no service in compose.yaml sets a memory limit — there is no mem_limit, no cpus, no deploy.resources, no pids_limit and no ulimits on any of the six.
Live Prometheus returns 0 series for container_spec_memory_limit_bytes, so
the alert cannot fire for any input.
This is the same failure class the repository already documents twice — ups.rules.yaml:6 on the missing battery pack, and the snmp-exporter
healthcheck comment on /health returning 404 — and the note there applies
here: a rule that cannot fire is worse than no rule, because the dashboard shows
it loaded.
Two ways out
Set limits. Gives the alert something to measure, and bounds the blast
radius of a leak on a host with 7.6 GB and no swap headroom to spare. Needs a
number per service that will not cause an OOM kill under normal load.
Rewrite it against the host total.container_memory_working_set_bytes / machine_memory_bytes needs no limits and still catches a runaway.
Whichever, the count in the docs should stop including a rule that cannot fire.
Activity
docs(roadmap): track the memory limits as their own item
feat(alerts): catch a remote-writing agent that stops pushing
fix(observability): make the security dashboard legible, and say what…
feat(observability): bound each service's memory, and its swap (#114)
feat(observability): bound each service's memory, and its swap (#114)
fix(rules): prove ContainerCpuThrottled can fire, and say that it can…
feat(lab): watch the domain's clock, and scrape rather than receive (#…
feat(lab): watch the domain's clock, and scrape rather than receive (#…