Skip to content

Replace the UPS battery, then delete the silence and enable scheduled self-tests #93

Closed

Description

@Gerrrt

mjolnir has no battery pack fitted. A mains loss is an immediate hard
shutdown of the entire rack, with no warning. docs/security.md:18 lists this
as "Not currently defended."

A replacement APCRBC115 cartridge is on order.

Buy the 1U shelf with it — #110

This is one purchase, not two. The pack on its own leaves the monitoring path
half-protected.

prometheus and oracle hang off an unmanaged switch that has no battery at
all. Both of those machines are laptops with their own batteries, so on a mains
cut they stay running and go deaf — a monitoring host that survives the event
it exists to observe but cannot report it is a strange way to lose an evidence
trail. Fitting the UPS pack protects the rack and does nothing for that switch.

#110 racks it: a 1U vented shelf
(~$25–45) in U4, spec already settled at the rack — 4-post frame, square holes,
full shelf with rear support rather than a cantilever, cage nuts plus screws.

Ordering them together is the point. Ordering the pack alone and getting to the
shelf later is the version where the gap stays open for however long "later"
turns out to be.

The management card fabricates a healthy battery

ups.rules.yaml:6 and the UPS dashboard both say so, at length. The NMC
reports upsBatteryStatus 2 (normal), upsEstimatedChargeRemaining 100,
upsEstimatedMinutesRemaining 63 and upsAlarmsPresent 0 — for a pack that is
not there. Six panel titles on the dashboard end in "(fabricated — no battery
fitted)".

Only UpsSelfTestFailed and UpsBatteryUnproven can detect the real
condition, keyed on upsTestResultsSummary (observed 4 = aborted).

When the pack arrives, in this order

1. Delete the silence immediately — do not wait for it to expire.

UpsSelfTestFailed is silenced in Alertmanager until 2026-09-20
(54f1715c-e57b-4322-8a6d-5435bc8e1bd8). It routes on category=power to the
urgent receiver with group_wait: 0s and repeat_interval: 30m, so leaving
it firing meant paging every half hour about a condition already known — which is
how an urgent receiver stops being read.

Between fitting the pack and 20 September, that silence would suppress
UpsSelfTestFailed on a UPS that can finally report honestly — including a
new pack that is itself faulty or badly seated
, which is exactly when you want
to hear about it.

curl -sS -X DELETE \
  http://localhost:9093/api/v2/silence/54f1715c-e57b-4322-8a6d-5435bc8e1bd8

If the pack has not arrived when the silence lapses, the alert comes back and
asks again. That is deliberate. Do not extend it without checking.

2. Run one self-test from the management card and confirm
upsTestResultsSummary reads 1 (donePass), not 4 (aborted).

3. Enable scheduled self-tests on the NMC, so the metric stays live evidence
rather than a stale last-known result.

4. Expect the fabricated metrics to become real — charge, runtime and
voltage start reporting actual values for the first time, which turns
UpsBatteryLow, UpsChargeLow and UpsRuntimeCritical into meaningful rules
instead of decorative ones.

Tracked in docs/roadmap.md; filed as an issue so it has a place to be discussed and closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    featureA new capability or a change

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions