Repository navigation
Description
mjolnir has no battery pack fitted. A mains loss is an immediate hard
shutdown of the entire rack, with no warning. docs/security.md:18 lists this
as "Not currently defended."
A replacement APCRBC115 cartridge is on order.
Buy the 1U shelf with it — #110
This is one purchase, not two. The pack on its own leaves the monitoring path
half-protected.
prometheus and oracle hang off an unmanaged switch that has no battery at
all. Both of those machines are laptops with their own batteries, so on a mains
cut they stay running and go deaf — a monitoring host that survives the event
it exists to observe but cannot report it is a strange way to lose an evidence
trail. Fitting the UPS pack protects the rack and does nothing for that switch.
#110 racks it: a 1U vented shelf
(~$25–45) in U4, spec already settled at the rack — 4-post frame, square holes,
full shelf with rear support rather than a cantilever, cage nuts plus screws.
Ordering them together is the point. Ordering the pack alone and getting to the
shelf later is the version where the gap stays open for however long "later"
turns out to be.
The management card fabricates a healthy battery
ups.rules.yaml:6 and the UPS dashboard both say so, at length. The NMC
reports upsBatteryStatus 2 (normal), upsEstimatedChargeRemaining 100,
upsEstimatedMinutesRemaining 63 and upsAlarmsPresent 0 — for a pack that is
not there. Six panel titles on the dashboard end in "(fabricated — no battery
fitted)".
Only UpsSelfTestFailed and UpsBatteryUnproven can detect the real
condition, keyed on upsTestResultsSummary (observed 4 = aborted).
When the pack arrives, in this order
1. Delete the silence immediately — do not wait for it to expire.
UpsSelfTestFailed is silenced in Alertmanager until 2026-09-20
(54f1715c-e57b-4322-8a6d-5435bc8e1bd8). It routes on category=power to the
urgent receiver with group_wait: 0s and repeat_interval: 30m, so leaving
it firing meant paging every half hour about a condition already known — which is
how an urgent receiver stops being read.
Between fitting the pack and 20 September, that silence would suppress
UpsSelfTestFailed on a UPS that can finally report honestly — including a
new pack that is itself faulty or badly seated, which is exactly when you want
to hear about it.
curl -sS -X DELETE \
http://localhost:9093/api/v2/silence/54f1715c-e57b-4322-8a6d-5435bc8e1bd8If the pack has not arrived when the silence lapses, the alert comes back and
asks again. That is deliberate. Do not extend it without checking.
2. Run one self-test from the management card and confirm
upsTestResultsSummary reads 1 (donePass), not 4 (aborted).
3. Enable scheduled self-tests on the NMC, so the metric stays live evidence
rather than a stale last-known result.
4. Expect the fabricated metrics to become real — charge, runtime and
voltage start reporting actual values for the first time, which turns
UpsBatteryLow, UpsChargeLow and UpsRuntimeCritical into meaningful rules
instead of decorative ones.
Tracked in docs/roadmap.md; filed as an issue so it has a place to be discussed and closed.
Activity
docs(ups): point the self-test-schedule gap at #249
Merge remote-tracking branch 'origin/main' into gerrrt/ups-battery-re…
Merge pull request #248 from Gerrrt/gerrrt/ups-battery-replacement-e0…