Skip to content

ContainerHighMemory cannot fire — no service sets a memory limit #63

Closed

Description

@Gerrrt

containers.rules.yaml:29-36:

container_memory_working_set_bytes{name!=""}
/ container_spec_memory_limit_bytes{name!=""} > 0.90
and container_spec_memory_limit_bytes{name!=""} > 0

The guard is correct and the alert is unreachable, because no service in
compose.yaml sets a memory limit — there is no mem_limit, no cpus, no
deploy.resources, no pids_limit and no ulimits on any of the six.

Live Prometheus returns 0 series for container_spec_memory_limit_bytes, so
the alert cannot fire for any input.

This is the same failure class the repository already documents twice —
ups.rules.yaml:6 on the missing battery pack, and the snmp-exporter
healthcheck comment on /health returning 404 — and the note there applies
here: a rule that cannot fire is worse than no rule, because the dashboard shows
it loaded.

Two ways out

  • Set limits. Gives the alert something to measure, and bounds the blast
    radius of a leak on a host with 7.6 GB and no swap headroom to spare. Needs a
    number per service that will not cause an OOM kill under normal load.
  • Rewrite it against the host total. container_memory_working_set_bytes / machine_memory_bytes needs no limits and still catches a runaway.

Whichever, the count in the docs should stop including a rule that cannot fire.

Found while verifying #12.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions