From c09123d505b2a3a8b2a7ca2b218c5427415d4cf3 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Wed, 7 Oct 2026 06:25:03 +0000 Subject: [PATCH 1/2] feat(observability): expiry gauges for the four management consoles' certificates (#857) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The pfSense GUI, the iLO, Saruman's :8006 and the TrueNAS UI each answer Hicks only, so no blackbox probe could read their certificates. ADR-0084 takes the issue's option 2: scripts/collect-cert-expiry.sh handshakes with each console from inside its own segment and writes homelab_cert_expiry_timestamp_seconds, with no new firewall pass. - pfsense-ui: read on morpheus over the ssh `make gateway-state` makes - pve-ui, ilo-ui: read on Saruman by a new agent unit (same VLAN as the iLO) - truenas-ui: read on smaug by a root cron job (build-the-nas.md §6.10) Rules: ManagementCertificateExpiringSoon (30d), ...ExpiryImminent (7d, inhibits the warning), ManagementCertificateUnchecked, CertExpiryStateStale. Refs #857: the Saruman and smaug halves need someone at a console. Co-Authored-By: Claude Opus 5.5 --- Makefile | 9 +- README.md | 8 +- ...ificate-expiry-from-inside-each-segment.md | 101 +++++++++ docs/changelog.md | 25 +++ docs/observability.md | 21 +- docs/runbooks/build-the-nas.md | 54 +++++ docs/runbooks/deploy-stack.md | 2 +- docs/runbooks/generate-certificates.md | 27 +++ docs/runbooks/schedule-maintenance.md | 2 +- docs/security.md | 2 +- scripts/README.md | 1 + scripts/collect-cert-expiry.sh | 209 ++++++++++++++++++ scripts/install-agent-collectors.sh | 1 + stacks/observability/README.md | 2 +- .../alertmanager/alertmanager.yaml | 7 + .../prometheus/rules/host.rules.yaml | 108 +++++++++ .../prometheus/targets/blackbox.yaml | 6 + .../prometheus/tests/cert-expiry.test.yaml | 206 +++++++++++++++++ systemd/README.md | 3 +- systemd/agent/homelab-cert-expiry.service | 46 ++++ systemd/agent/homelab-cert-expiry.timer | 17 ++ 21 files changed, 842 insertions(+), 15 deletions(-) create mode 100644 docs/adr/0084-read-management-certificate-expiry-from-inside-each-segment.md create mode 100755 scripts/collect-cert-expiry.sh create mode 100644 stacks/observability/prometheus/tests/cert-expiry.test.yaml create mode 100644 systemd/agent/homelab-cert-expiry.service create mode 100644 systemd/agent/homelab-cert-expiry.timer diff --git a/Makefile b/Makefile index f69fd100..49818a3a 100644 --- a/Makefile +++ b/Makefile @@ -361,7 +361,7 @@ smart-state: ## Collect SMART health from THIS host's disks (needs root) and ren fi .PHONY: gateway-state -gateway-state: ## Collect the firewall's view of its uplinks and DDNS record (#353, #604) +gateway-state: ## Collect the firewall's view of its uplinks, DDNS record and GUI certificate (#353, #604, #857) @# Two measurements per family: what pfSense reports, and whether traffic of @# that family actually leaves the building. They disagreed on 2026-09-07 — @# WAN_DHCP6 reported 100% loss while v6 reached the internet through it in @@ -372,6 +372,13 @@ gateway-state: ## Collect the firewall's view of its uplinks and DDNS record (#3 @# public resolver, to the WAN address (#604). Compared on the firewall: @# the name and the address are both withheld, and only the verdict comes back. ./scripts/collect-gateway-state.sh --ssh $(FW_USER)@$(FW_HOST) --host morpheus + @# + @# And when the GUI's certificate expires (#857, ADR-0084), read off the + @# handshake on the firewall itself: the GUI is blocked from this host on + @# purpose, and the ssh it already allows is enough. Rides on this target so + @# it needs no new timer; a failure fails the job like any other. + ./scripts/collect-cert-expiry.sh --ssh $(FW_USER)@$(FW_HOST) --host morpheus \ + --probe pfsense-ui=morpheus=127.0.0.1:443 .PHONY: silence-state silence-state: ## Collect Alertmanager's silences as metrics (#575) diff --git a/README.md b/README.md index 5111eefc..e4ef2cf1 100644 --- a/README.md +++ b/README.md @@ -95,7 +95,7 @@ documents for different readers. can't run an agent (firewall, switch, UPS, iLO). One agent config, deployed identically everywhere. [How](docs/architecture.md#observability-data-flow) - **Dashboards and alerting as code.** 8 provisioned dashboards, 156 panels, and - 163 alert rules — 144 metric-based in Prometheus, 19 log-based in Loki — sharing + 167 alert rules — 148 metric-based in Prometheus, 19 log-based in Loki — sharing one Alertmanager routing tree. No dashboard exists only in a database. - **Secrets encrypted in-repo with SOPS + age.** Per-device credentials, decrypted at deploy time into gitignored paths, with `git log` showing which @@ -125,7 +125,7 @@ documents for different readers. the Makefile, the scripts, the workflow and the runbooks resolves its image from `compose.yaml` too, so an image that is not pinned there cannot be run at all. -- **Documented decisions and runbooks.** 80 ADRs covering what was chosen +- **Documented decisions and runbooks.** 81 ADRs covering what was chosen and what was rejected — including the costs accepted knowingly; 45 runbooks for the operations that are easy to get wrong at 1am, one of which is the handover page a successor reads first. Every critical alert links to @@ -191,7 +191,7 @@ inbound passes may still reach it. Data flow and the maintained Mermaid topology . ├── stacks/ │ ├── observability/ # the estate's stack on prometheus — ten services -│ │ ├── prometheus/ # config, file_sd targets, 144 alert rules +│ │ ├── prometheus/ # config, file_sd targets, 148 alert rules │ │ ├── alertmanager/ # routing and inhibition │ │ ├── loki/ # single-binary config + 19 LogQL rules │ │ ├── alloy/ # the agent config directory, shipped to every host @@ -217,7 +217,7 @@ inbound passes may still reach it. Data flow and the maintained Mermaid topology │ ├── architecture.md network.md hardware.md │ ├── observability.md security.md roadmap.md changelog.md │ ├── diagrams/ # the network diagram (SVG) and its predecessors -│ ├── adr/ # 80 ADRs — architecture decision records +│ ├── adr/ # 81 ADRs — architecture decision records │ └── runbooks/ # 45 runbooks; successor-handover.md is the front door └── Makefile # make help ``` diff --git a/docs/adr/0084-read-management-certificate-expiry-from-inside-each-segment.md b/docs/adr/0084-read-management-certificate-expiry-from-inside-each-segment.md new file mode 100644 index 00000000..a248f3f3 --- /dev/null +++ b/docs/adr/0084-read-management-certificate-expiry-from-inside-each-segment.md @@ -0,0 +1,101 @@ +# ADR-0084: Read the management consoles' certificate expiry from inside each segment + +**Status:** Accepted · 2026-10 + +## Context + +Four management consoles serve self-signed certificates that nothing watched +([#857](https://github.com/Gerrrt/HomeLab/issues/857)): + +| Console | Where | Who can reach it | +| --- | --- | --- | +| pfSense GUI | `morpheus`, `10.0.99.1:443` | Hicks. "Block HTTPS to pfSense" on the Winterfell interface is an explicit rule | +| iLO | `shiva`, `10.0.30.10:443`, VLAN 30 | Hicks. VLAN 99 passes SNMP to it and nothing else ([ADR-0033](0033-keep-the-ilo-on-the-lab-segment.md)) | +| Proxmox UI | `Saruman`, `10.0.30.110:8006` | Hicks and `phoenix` | +| TrueNAS UI | `smaug`, `10.0.40.30:443` | Hicks | + +Each expires on its own schedule. For example, the pfSense GUI's certificate +was regenerated on 2026-09-28 and runs out on 2027-04-16. Until now, the first +sign of an expiry would have been a browser warning on the day someone needed +the console. + +The estate already watches certificate expiry. `TlsCertificateExpiringSoon` +reads `probe_ssl_earliest_cert_expiry` off a blackbox handshake (#91). That +needs the monitoring host to complete a handshake, and it can reach none of +these four. Probes for the iLO and pfSense were written and left disabled in +`targets/blackbox.yaml`, waiting on a firewall pass each. + +Three options were weighed: + +1. **A blackbox probe per console, with one firewall pass each from + `10.0.99.20`.** This reuses the existing rules unchanged. But each pass is a + segmentation decision, and the comment above the disabled probes says so. + The pfSense one gives the machine every agent pushes to a path to the + firewall's login page. The iLO one undoes the part of ADR-0033 that kept + VLAN 99 to SNMP. Four passes would be opened for a date that changes about + once a year. +2. **Read the expiry from inside each console's own segment, from a host that + can already reach it, and ship it as a textfile gauge.** No new pass is + needed. +3. **A calendar entry per certificate.** This is cheap, but it is exactly the + remembering that the estate is built to avoid, and nothing would notice a + certificate reissued early and the entry left stale. + +## Decision + +**Option 2.** `scripts/collect-cert-expiry.sh` makes a TLS handshake with each +console from a host already able to reach it. It writes +`homelab_cert_expiry_timestamp_seconds{endpoint, host}` and +`homelab_cert_checked{endpoint, host}` into that host's textfile directory. + +| `endpoint` | Read from | Runs as | +| --- | --- | --- | +| `pfsense-ui` | morpheus itself, `127.0.0.1:443`, over the ssh `make gateway-state` already makes | that target, every 15 minutes | +| `pve-ui` | Saruman itself, `127.0.0.1:8006` | `homelab-cert-expiry.timer`, daily | +| `ilo-ui` | Saruman, `10.0.30.10:443`, which is the same VLAN, so no pf rule is crossed | the same timer | +| `truenas-ui` | smaug itself, `127.0.0.1:443` | a root cron job, the [ADR-0047](0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md) mechanism | + +The decisions inside it: + +- **The handshake, not a file.** This is the principle the blackbox rules + already follow. What matters is the certificate being served, and a file can + be replaced while the daemon still holds the old one. On pfSense, the route + to the file goes through `config.xml`, and a search of that file has printed + a private key into a transcript before. `s_client` does not verify, so a + self-signed or expired certificate still reports its date. +- **The parsing happens on Linux.** The probe is POSIX `sh` and returns + openssl's `notAfter=` line, and GNU `date` turns it into epoch seconds on the + collecting host. morpheus is FreeBSD and its `date` is not GNU. +- **A failed read is not an expiry.** An unread console writes + `homelab_cert_checked 0` and no expiry sample, the + `homelab_ddns_record_checked` pattern. `ManagementCertificateUnchecked` + fires after six hours. Without it, a console that refused the handshake would + make the expiry alerts quietly impossible. +- **The rules are their own.** `ManagementCertificateExpiringSoon` (30 days, + warning) and `ManagementCertificateExpiryImminent` (7 days, critical) mirror + the blackbox pair's tiers and reasons, aggregated `min by (endpoint, host)`. + An inhibit in `alertmanager.yaml` makes a certificate page once. + `CertExpiryStateStale` watches the file's mtime, as `SmartStateStale` does. +- **The old-TLS fallback.** A handshake that fails is retried once at + `SECLEVEL=0`, because iLO 4 may not meet OpenSSL 3's default level. Nothing + is sent and nothing read is trusted except a date, so this weakens nothing. + +## Consequences + +- No firewall rule changes, and `docs/firewall-claims.yaml` does not move. +- The two disabled blackbox probes stay disabled. This collector replaces them + for **expiry only**. Whether the iLO or pfSense GUI is *reachable* from the + monitoring host remains the separate decision their comment describes, and + this ADR does not take it. +- Two of the four gauges need someone at a console to start them. Saruman's + collector is installed from the Mac with + `make install-agent-collectors AGENT=root@10.0.30.110 ARGS="--only cert-expiry"`, because VLAN 99 cannot + reach VLAN 30. smaug's is a cron job added in TrueNAS's UI + ([`build-the-nas.md`](../runbooks/build-the-nas.md) §6.10). The pfSense one + starts with the next `make gateway-state` after merging. +- Resolution is a day for three of the four. Against a 30-day warning this + costs nothing. +- If a console moves to a certificate the estate's CA or step-ca issues and the + monitoring host is ever allowed to reach it, a blackbox probe is better + because it also proves reachability. Its row here should then be retired, not + kept as a second reading. diff --git a/docs/changelog.md b/docs/changelog.md index d2dbb4cb..41fbfff2 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -19,6 +19,30 @@ docstring gives: it is a record, not a claim about now. ## 2026-10-07 +- **The four management consoles' certificates have expiry gauges** + ([#857](https://github.com/Gerrrt/HomeLab/issues/857), + [ADR-0084](adr/0084-read-management-certificate-expiry-from-inside-each-segment.md)). + These are the pfSense GUI, the iLO, `Saruman`'s `:8006` and the TrueNAS UI. + Each answers Hicks only, so no blackbox probe could read them. + `scripts/collect-cert-expiry.sh` makes the handshake inside each console's + own segment and writes `homelab_cert_expiry_timestamp_seconds`, and no + firewall pass was added. + - **Where each is read.** pfSense's is read on the firewall over the ssh + `make gateway-state` already makes. Saruman's and the iLO's are read from + Saruman by a new agent timer, since Saruman shares the iLO's segment. + smaug's is read by a root cron job. + - **New rules.** `ManagementCertificateExpiringSoon` warns at 30 days and + `ManagementCertificateExpiryImminent` pages at 7. + `ManagementCertificateUnchecked` covers a console that will not answer, and + `CertExpiryStateStale` covers a collector that stopped. + - **Measured from the monitoring host on 2026-10-07.** morpheus serves its + GUI certificate on `127.0.0.1:443`, `notAfter` 2027-04-16 16:37:44 UTC. + That matches the 2026-09-28 regeneration. + - **Not yet live: the Saruman and smaug halves.** Each needs someone at a + console: `make install-agent-collectors` from the Mac for Saruman, and + [`build-the-nas.md`](runbooks/build-the-nas.md) §6.10 for smaug. #857 + stays open until all four `endpoint` series exist. + - **`trinity`'s backup sets are re-verified nightly, here and on `oracle`** ([#856](https://github.com/Gerrrt/HomeLab/issues/856), [ADR-0064](adr/0064-copy-immichs-library-to-oracle-until-the-off-estate-copy-exists.md)). @@ -39,6 +63,7 @@ docstring gives: it is a record, not a claim about now. entry. It took about three minutes. - **The fixture.** `backup-library.sh --self-test` now flips one byte in a set and truncates another, and `--verify-only --all` fails on both. +||||||| parent of 4ab91f2 (wip #857) - **Syslog stores only the senders it names.** [#844](https://github.com/Gerrrt/HomeLab/issues/844) found that any host diff --git a/docs/observability.md b/docs/observability.md index 8bde86c2..0716e9a7 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -344,7 +344,7 @@ separates a quiet stream from a stopped one. ## Alerting -163 rules in total: 144 metric-based in `prometheus/rules/`, and 19 log-based in +167 rules in total: 148 metric-based in `prometheus/rules/`, and 19 log-based in `loki/rules/`. ### Log-based (Loki ruler) @@ -466,17 +466,17 @@ argument and for what to do when it exits 1. ### Metric-based (Prometheus) -144 rules across twelve files in `prometheus/rules/`: +148 rules across twelve files in `prometheus/rules/`: | File | Covers | | --- | --- | -| `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew and a clock with no time source at all — `HostClockUnsynchronised` reads `node_timex_sync_status`, because the offset reads zero once timesyncd has restored a clock that is wrong but stable, which is how a cleared RTC wrote fourteen minutes of samples three hours in the past and nothing noticed ([#519](https://github.com/Gerrrt/HomeLab/issues/519)) — reboots, and, for the two laptops, whether the shelf is on mains, whether the cell that carries them through a cut is still worth relying on ([#454](https://github.com/Gerrrt/HomeLab/issues/454), and [`runbooks/replace-the-laptop-cell.md`](runbooks/replace-the-laptop-cell.md) for the swap), how hot that cell is, whether its temperature is being measured at all — on 2026-09-19 it was not, on either host — and how many minutes the host has left once the cut arrives ([#532](https://github.com/Gerrrt/HomeLab/issues/532)); whether the wiki's drift check on `oracle` is still running ([#470](https://github.com/Gerrrt/HomeLab/issues/470)); and, for the NAS, whether every ZFS pool is online — `ZpoolNotOnline` reads `node_zfs_zpool_state`, added after `erebor` lost a disk on 2026-09-19 with the exporter hung ahead of it and `InstanceDown` the only page ([#558](https://github.com/Gerrrt/HomeLab/issues/558), [`runbooks/replace-the-nas-disk.md`](runbooks/replace-the-nas-disk.md)); and whether any drive's unsafe-shutdown count has grown in a day — `SmartDriveUnsafeShutdownsGrowing`, whose page says when `smaug` recorded a clean power-off the same day, because mains removed after a clean shutdown ticks the S3520 and a clean shutdown alone does not ([#746](https://github.com/Gerrrt/HomeLab/issues/746)), the measure of a cut the shutdown sequence [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides did not catch, the rule [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md) left to that issue when it carried `smaug`'s counter under the scrape ([#574](https://github.com/Gerrrt/HomeLab/issues/574)); and, for `Saruman`, how full its LVM-thin pools are — which `node_filesystem_*` cannot see — and whether its Proxmox firewall is still on with an inbound policy that drops, from two agent collectors ([#538](https://github.com/Gerrrt/HomeLab/issues/538), [#576](https://github.com/Gerrrt/HomeLab/issues/576)); and whether the `tc` mirror that feeds Zeek on `fenrir` is still delivering packets from every port of the lab bridge — `ZeekMirrorInactive`, Zeek's absence detection answered on the hypervisor because the lab's Loki has no ruler ([#437](https://github.com/Gerrrt/HomeLab/issues/437), [ADR-0068](adr/0068-mirror-the-lab-bridge-to-zeek-with-tc-not-open-vswitch.md)); and whether a guest the hypervisor tags `disposable` has outlived the fortnight it was built for — `DisposableGuestOutlived`, read from each guest's own `qm config` rather than from any list of which guests should exist, which is also why `HypervisorGuestStopped` stays quiet for such a guest, with `GuestConfigUnreadable` covering the read itself failing ([#438](https://github.com/Gerrrt/HomeLab/issues/438), [ADR-0071](adr/0071-run-disposable-investigations-on-a-guest-that-is-destroyed.md)) | +| `host.rules.yaml` | Instance down, predictive disk fill, memory, load, clock skew and a clock with no time source at all — `HostClockUnsynchronised` reads `node_timex_sync_status`, because the offset reads zero once timesyncd has restored a clock that is wrong but stable, which is how a cleared RTC wrote fourteen minutes of samples three hours in the past and nothing noticed ([#519](https://github.com/Gerrrt/HomeLab/issues/519)) — reboots, and, for the two laptops, whether the shelf is on mains, whether the cell that carries them through a cut is still worth relying on ([#454](https://github.com/Gerrrt/HomeLab/issues/454), and [`runbooks/replace-the-laptop-cell.md`](runbooks/replace-the-laptop-cell.md) for the swap), how hot that cell is, whether its temperature is being measured at all — on 2026-09-19 it was not, on either host — and how many minutes the host has left once the cut arrives ([#532](https://github.com/Gerrrt/HomeLab/issues/532)); whether the wiki's drift check on `oracle` is still running ([#470](https://github.com/Gerrrt/HomeLab/issues/470)); and, for the NAS, whether every ZFS pool is online — `ZpoolNotOnline` reads `node_zfs_zpool_state`, added after `erebor` lost a disk on 2026-09-19 with the exporter hung ahead of it and `InstanceDown` the only page ([#558](https://github.com/Gerrrt/HomeLab/issues/558), [`runbooks/replace-the-nas-disk.md`](runbooks/replace-the-nas-disk.md)); and whether any drive's unsafe-shutdown count has grown in a day — `SmartDriveUnsafeShutdownsGrowing`, whose page says when `smaug` recorded a clean power-off the same day, because mains removed after a clean shutdown ticks the S3520 and a clean shutdown alone does not ([#746](https://github.com/Gerrrt/HomeLab/issues/746)), the measure of a cut the shutdown sequence [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides did not catch, the rule [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md) left to that issue when it carried `smaug`'s counter under the scrape ([#574](https://github.com/Gerrrt/HomeLab/issues/574)); and, for `Saruman`, how full its LVM-thin pools are — which `node_filesystem_*` cannot see — and whether its Proxmox firewall is still on with an inbound policy that drops, from two agent collectors ([#538](https://github.com/Gerrrt/HomeLab/issues/538), [#576](https://github.com/Gerrrt/HomeLab/issues/576)); and whether the `tc` mirror that feeds Zeek on `fenrir` is still delivering packets from every port of the lab bridge — `ZeekMirrorInactive`, Zeek's absence detection answered on the hypervisor because the lab's Loki has no ruler ([#437](https://github.com/Gerrrt/HomeLab/issues/437), [ADR-0068](adr/0068-mirror-the-lab-bridge-to-zeek-with-tc-not-open-vswitch.md)); and whether a guest the hypervisor tags `disposable` has outlived the fortnight it was built for — `DisposableGuestOutlived`, read from each guest's own `qm config` rather than from any list of which guests should exist, which is also why `HypervisorGuestStopped` stays quiet for such a guest, with `GuestConfigUnreadable` covering the read itself failing ([#438](https://github.com/Gerrrt/HomeLab/issues/438), [ADR-0071](adr/0071-run-disposable-investigations-on-a-guest-that-is-destroyed.md)); and how many days the four management consoles' certificates have left — the pfSense GUI, the iLO, `Saruman`'s `:8006` and the TrueNAS UI, none of which the blackbox exporter can reach — `ManagementCertificateExpiringSoon` at 30 days and `ManagementCertificateExpiryImminent` at 7, read from a handshake made inside each console's own segment, with `ManagementCertificateUnchecked` for a console that would not answer and `CertExpiryStateStale` for a collector that stopped ([#857](https://github.com/Gerrrt/HomeLab/issues/857), [ADR-0084](adr/0084-read-management-certificate-expiry-from-inside-each-segment.md)) | | `network.rules.yaml` | SNMP reachability, pf not running, state table, switch links, iLO hardware and Smart Array cache. `shiva`'s Smart Storage Battery read failed from 2026-08-18 until it was replaced on 2026-09-02, with the array in write-through as a result, so stored metrics before that date show the failed pack — `IloBatteryCondition` names the spare part to order, and the controller rollups are deliberately read at *failed* rather than *degraded* ([#76](https://github.com/Gerrrt/HomeLab/issues/76)). Also whether the remote path's dynamic DNS name still resolves to the WAN address — `DdnsRecordStale` and `DdnsRecordUnchecked`, from the comparison `scripts/collect-gateway-state.sh` makes on the firewall every fifteen minutes ([#604](https://github.com/Gerrrt/HomeLab/issues/604), [ADR-0044](adr/0044-answer-the-endpoint-with-dynamic-dns-from-morpheus.md)); see *What the dynamic DNS series disclose* below | | `ups.rules.yaml` | On battery, low battery, runtime, load, temperature. A pack was fitted on 2026-08-28 and passed its self-test, so these read real hardware; stored metrics older than that date are the card's fabricated values — see [`runbooks/fit-the-ups-battery.md`](runbooks/fit-the-ups-battery.md) | | `containers.rules.yaml` | Restart loops, OOM kills, memory, throttling | | `stack.rules.yaml` | The stack watching itself: config reloads, rule evaluation, notification delivery, log ingestion, and the two cases `up == 0` structurally cannot see — a remote-writing agent that stops pushing, and a scraped target that stops being a target at all. The second is `ScrapeTargetDisappeared`, added with the first scraped host ([#256](https://github.com/Gerrrt/HomeLab/issues/256)): an emptied or unparseable `targets/node.yaml` makes the series vanish rather than fall to 0, so `InstanceDown` stays silent and `RemoteWriteJobStale` excludes scraped jobs by design. The target for `smaug` was written into `targets/node.yaml` disabled on 2026-09-17 and enabled on 2026-09-19, once the exporter answered from the pool. Split off `containers.rules.yaml` onto `component: stack` in [#81](https://github.com/Gerrrt/HomeLab/issues/81) so a Prometheus that cannot reload its config stops being filed as a container fault. Since [#575](https://github.com/Gerrrt/HomeLab/issues/575) also whether an Alertmanager silence is about to lapse or names no owning issue, read from the per-silence series `scripts/collect_silences.py` writes every fifteen minutes — `alertmanager_silences` is a count per state and cannot say which alert, when, or whose | | `watchdog.rules.yaml` | One rule that always fires, so that its absence is detectable | -| `blackbox.rules.yaml` | Whether an endpoint can actually be reached, from outside the service, and how many days its certificate has left — the sensitive tier's seven-day ACME leaves excepted, which an hours pair watches for a stalled renewal instead — `TlsAcmeRenewalLate` at 48h, `TlsAcmeRenewalStalled` at 36h ([#426](https://github.com/Gerrrt/HomeLab/issues/426)) — Grafana verified against the lab CA, the APC card's self-signed one read but not trusted, the wiki, Prometheus, Loki and Alertmanager over plain http (the switch UI's probes were removed on 2026-09-06, `targets/blackbox.yaml`), and — the other way round — that the ingest proxy on `10.0.99.20:9090` and `:3100` still refuses a request with no token (`IngestAuthNotEnforced`, [#182](https://github.com/Gerrrt/HomeLab/issues/182)). The iLO and pfSense UIs are written into `targets/blackbox.yaml` and left disabled: each needs a firewall pass from `10.0.99.20` that is a segmentation decision, not a monitoring one ([#91](https://github.com/Gerrrt/HomeLab/issues/91)) | +| `blackbox.rules.yaml` | Whether an endpoint can actually be reached, from outside the service, and how many days its certificate has left — the sensitive tier's seven-day ACME leaves excepted, which an hours pair watches for a stalled renewal instead — `TlsAcmeRenewalLate` at 48h, `TlsAcmeRenewalStalled` at 36h ([#426](https://github.com/Gerrrt/HomeLab/issues/426)) — Grafana verified against the lab CA, the APC card's self-signed one read but not trusted, the wiki, Prometheus, Loki and Alertmanager over plain http (the switch UI's probes were removed on 2026-09-06, `targets/blackbox.yaml`), and — the other way round — that the ingest proxy on `10.0.99.20:9090` and `:3100` still refuses a request with no token (`IngestAuthNotEnforced`, [#182](https://github.com/Gerrrt/HomeLab/issues/182)). The iLO and pfSense UIs are written into `targets/blackbox.yaml` and left disabled: each needs a firewall pass from `10.0.99.20` that is a segmentation decision, not a monitoring one ([#91](https://github.com/Gerrrt/HomeLab/issues/91)). Their certificates' expiry is watched without that pass, from `host.rules.yaml` ([#857](https://github.com/Gerrrt/HomeLab/issues/857)) | | `dns.rules.yaml` | Whether the house is still filtering DNS, asked directly at AdGuard Home on port 53 rather than through pfSense. Since [ADR-0055](adr/0055-forward-to-adguard-alone.md) AdGuard is the only forwarder, so `AdGuardNotAnswering` is **critical** at five minutes: the house cannot resolve outside names. `AdGuardNotFiltering` stays a warning, because a filter that fails open is a convenience lost, not an outage. The targets in `targets/blackbox-dns.yaml` are live since 2026-09-28, against AdGuard on `trinity` ([#126](https://github.com/Gerrrt/HomeLab/issues/126), [#404](https://github.com/Gerrrt/HomeLab/issues/404)) | | `backup.rules.yaml` | Whether the scheduled maintenance jobs are still being run at all — staleness, failure, never-ran, whether the age-key proof record exists to be held to its deadline, whether the CA key's offline copy has been proved lately ([#496](https://github.com/Gerrrt/HomeLab/issues/496)), whether the newest backup sets have been carried onto the second recipient's medium within ninety days ([ADR-0048](adr/0048-carry-the-estates-backup-sets-with-the-second-recipient.md)), and whether the household's copy has been carried to the holder's drive within ninety days and proved by the holder within a year ([ADR-0073](adr/0073-carry-the-household-copy-on-a-drive-the-holder-keeps.md)) | | `deploy.rules.yaml` | Whether each host that pulls — `prometheus` and `trinity` ([#533](https://github.com/Gerrrt/HomeLab/issues/533)) — is running what the repository says: an uncommitted edit made on the host, a revision that did not verify, and how far behind `main` it is, and a host whose record stopped arriving. Reads the record `scripts/converge.sh` writes hourly ([#99](https://github.com/Gerrrt/HomeLab/issues/99), [ADR-0021](adr/0021-converge-on-a-timer-instead-of-deploying-over-ssh.md)) | @@ -502,7 +502,7 @@ as loaded and healthy and could not fire for any input ([#63](https://github.com `prometheus/tests/*.test.yaml` holds `promtool test rules` unit tests, which feed a rule synthetic series and assert it fires — paired with a case asserting it stays quiet, because a test that only ever expects silence would have passed -against the broken rule too. Coverage is 144 rules of 144 so far ([#843](https://github.com/Gerrrt/HomeLab/issues/843)) — all ten +against the broken rule too. Coverage is 148 rules of 148 so far ([#843](https://github.com/Gerrrt/HomeLab/issues/843)) — all ten in `blackbox.rules.yaml`, all three in `dns.rules.yaml`, `GatewayFilesystemCritical`, `ContainerHighMemory`, `ContainerNearMemoryLimit`, `ContainerRestartLoop`, `ContainerCpuThrottled`, the three container-state rules from @@ -573,6 +573,17 @@ covers all of them. The spindles are left out of the second path on purpose. Reading them there too would page twice for one disk. +#### What the certificate expiry series disclose + +`homelab_cert_expiry_timestamp_seconds` and `homelab_cert_checked` carry an +`endpoint` name (`pfsense-ui`, `pve-ui`, `ilo-ui`, `truenas-ui`) and the +`host` the console belongs to, and nothing else. No address, subject or +fingerprint leaves the host that made the handshake. A console's expiry date +says when a self-signed certificate was made, which is not on +[`security.md`](security.md#what-this-repository-deliberately-does-not-publish)'s +withheld list, and the consoles themselves are already named in +[`network.md`](network.md) ([#857](https://github.com/Gerrrt/HomeLab/issues/857)). + #### What the dynamic DNS series disclose `homelab_ddns_record_checked` and `homelab_ddns_record_matches_wan` are a diff --git a/docs/runbooks/build-the-nas.md b/docs/runbooks/build-the-nas.md index e75b993c..6bcf19a1 100644 --- a/docs/runbooks/build-the-nas.md +++ b/docs/runbooks/build-the-nas.md @@ -1998,6 +1998,60 @@ Prometheus. Then stop one deliberately, for example within the scrape interval and firing ten minutes later. Start it again with `docker start media-navidrome`. +### §6.10 — Turn certificate expiry collection on + +The TrueNAS UI's certificate expires on its own schedule, and nothing on the +monitoring host can see it: 443 answers Hicks only, which §1 set up on purpose +([#857](https://github.com/Gerrrt/HomeLab/issues/857), +[ADR-0084](../adr/0084-read-management-certificate-expiry-from-inside-each-segment.md)). +`scripts/collect-cert-expiry.sh` makes the handshake from this host to its own +UI and writes `homelab_cert_expiry_timestamp_seconds{endpoint="truenas-ui"}` +into the directory §6.4 made. It runs as a root cron job, like the other +collectors in this section. `ManagementCertificateExpiringSoon` warns at 30 +days, and `CertExpiryStateStale` fires if the job stops. + +**1. Fetch it and read one run**, in the TrueNAS shell: + +```bash +cd /mnt/erebor/apps/stack \ + && curl -fsSLO https://raw.githubusercontent.com/Gerrrt/HomeLab/main/scripts/collect-cert-expiry.sh \ + && chmod 0755 collect-cert-expiry.sh +PATH=/usr/sbin:/usr/bin:/sbin:/bin /bin/bash /mnt/erebor/apps/stack/collect-cert-expiry.sh --print --host smaug --probe truenas-ui=smaug=127.0.0.1:443 +``` + +`--print` writes no file. Expect a +`homelab_cert_expiry_timestamp_seconds{endpoint="truenas-ui",host="smaug"}` +line and `homelab_cert_checked … 1`. If `checked` is `0` and there is no +expiry line, the UI is bound to its address rather than to every interface: +run it again with `127.0.0.1:443` replaced by `10.0.40.30:443`, and use that +in step 2. + +**2. The cron job.** **System → Advanced Settings → Cron Jobs → Add**: + +```text +PATH=/usr/sbin:/usr/bin:/sbin:/bin TEXTFILE_DIR=/mnt/erebor/apps/textfile timeout 60 /bin/bash /mnt/erebor/apps/stack/collect-cert-expiry.sh --host smaug --probe truenas-ui=smaug=127.0.0.1:443 +``` + +| Field | Value | Why | +| --- | --- | --- | +| Description | `homelab cert-expiry (#857)` | So the next person finds the decision from the job | +| Command | the block above, exactly | `--host smaug` names the file `cert-expiry-smaug.prom`; `timeout 60` because a wedged UI would otherwise hang the job | +| Run As User | `root` | Only because `/mnt/erebor/apps/textfile` is root-owned `0755` | +| Schedule | daily, `08:50` | The warning is at 30 days; `CertExpiryStateStale` fires at two days, two missed runs | +| Hide Standard Output | **on** | Success is one line and TrueNAS would mail it daily | +| Hide Standard Error | **off** | A failure is the thing worth seeing | + +**3. Confirm**, from the monitoring host after the first scheduled run: + +```bash +curl -s http://10.0.40.30:9100/metrics | grep -E '^homelab_cert_|^node_textfile_scrape_error' +``` + +The expiry line, `homelab_cert_checked` at `1`, and `node_textfile_scrape_error` +at `0`. **When the UI's certificate is reissued** (System → Certificates), the +next day's file carries the new date; see +[`generate-certificates.md`](generate-certificates.md#management-console-certificates). + ## §7 — Verify > **As of 2026-09-19:** the monitoring-host line holds in both halves, the diff --git a/docs/runbooks/deploy-stack.md b/docs/runbooks/deploy-stack.md index 199c7bb9..eb391a42 100644 --- a/docs/runbooks/deploy-stack.md +++ b/docs/runbooks/deploy-stack.md @@ -146,7 +146,7 @@ Then in the UI: 1. **Prometheus → Status → Targets.** Every job `UP`. The four `snmp` targets take up to 45 seconds on their first scrape. 2. **Prometheus → Status → Rules.** Every rule loaded, none in error. The - page lists 150: the 144 alert rules this repository counts everywhere + page lists 154: the 148 alert rules this repository counts everywhere else, plus the six recording rules: `homelab_suricata_expected_interface`, `homelab_battery_runtime_seconds`, and the two directions each of diff --git a/docs/runbooks/generate-certificates.md b/docs/runbooks/generate-certificates.md index 7837a931..fa858531 100644 --- a/docs/runbooks/generate-certificates.md +++ b/docs/runbooks/generate-certificates.md @@ -194,6 +194,33 @@ verifies, so an expired leaf fails it outright and `EndpointUnreachable` fires for Grafana — alongside `up{job="grafana"}` going to 0, since Prometheus verifies the same chain on its scrape. +## Management console certificates + +Four consoles serve self-signed certificates that this CA never issued, and +the blackbox probes above cannot reach any of them: each answers Hicks only. +`scripts/collect-cert-expiry.sh` reads each one's expiry off a handshake made +inside its own segment and writes `homelab_cert_expiry_timestamp_seconds`; +`ManagementCertificateExpiringSoon` warns at 30 days and +`ManagementCertificateExpiryImminent` pages at 7 +([#857](https://github.com/Gerrrt/HomeLab/issues/857), +[ADR-0084](../adr/0084-read-management-certificate-expiry-from-inside-each-segment.md)). + +| `endpoint` | Console | Read by | Reissue | +| --- | --- | --- | --- | +| `pfsense-ui` | pfSense GUI on `morpheus` | `make gateway-state` on the monitoring host, over its ssh, every 15 minutes | `pfSsh.php playback generateguicert` in the firewall's shell. That is how the 2026-09-28 regeneration was done, and it gives 398 days, so the 2027-04-16 expiry recurs | +| `pve-ui` | Proxmox UI on `Saruman`, `:8006` | `homelab-cert-expiry.timer` on Saruman, daily | `pvecm updatecerts --force`, then `systemctl restart pveproxy`, at Saruman's console | +| `ilo-ui` | iLO on `shiva`, `10.0.30.10` | the same timer on Saruman, which shares its segment | The iLO's own UI: **Administration → Security → SSL Certificate** | +| `truenas-ui` | TrueNAS UI on `smaug` | a root cron job on smaug ([`build-the-nas.md`](build-the-nas.md) §6.10), daily | **Credentials → Certificates**, then select it under **System → General → GUI** | + +After reissuing, the alert clears at the next collection. For `pfsense-ui` that +is within 15 minutes. For the others it is the next day, or sooner if the +collector is run by hand: `systemctl start homelab-cert-expiry` on Saruman, or +the cron job's **Run Now** on smaug. + +A console whose certificate cannot be read raises +`ManagementCertificateUnchecked` after six hours rather than going silent. Run +the collector with `--print` to see which probe failed. + ## If something goes wrong | Symptom | Cause | Fix | diff --git a/docs/runbooks/schedule-maintenance.md b/docs/runbooks/schedule-maintenance.md index 6457daf3..4b0c99c4 100644 --- a/docs/runbooks/schedule-maintenance.md +++ b/docs/runbooks/schedule-maintenance.md @@ -160,7 +160,7 @@ and `morpheus` is FreeBSD with no apt at all. runs Alloy but has no checkout of this repository — `oracle` — gets the collectors and their own timers installed directly, by `make install-agent-collectors AGENT=user@host`. It ships every collector the script's -`COLLECTORS` table names — `patch-state`, `smart-state`, `pve-version`, +`COLLECTORS` table names — `patch-state`, `smart-state`, `pve-version`, `cert-expiry`, `guest-state`, `thin-pool-state`, `guest-disk-state`, `pve-firewall-state`, `iso-store-state`, `zeek-mirror-state`, `pbs-task-state` and `drift-check`, plus the two rows that collect nothing, `zeek-mirror` and `prune-images`, and one scheduled job, `zeek-archive-prune` — and checks each host's requirements **per collector**, so a host without apt still gets SMART and the one it cannot have diff --git a/docs/security.md b/docs/security.md index 4dce9a72..9201e041 100644 --- a/docs/security.md +++ b/docs/security.md @@ -21,7 +21,7 @@ What this network is actually built to survive: | Someone on the lab segment reading the domain's metrics | **Scoped, unauthenticated, and accepted.** `windows_exporter` listens on `9182` on all six of ADR-0029's guests, on the segment that exists to hold attackers. Each guest's Windows Firewall admits `10.0.30.40` (`alexander`) alone ([`build-the-lab-domain.md`](runbooks/build-the-lab-domain.md) §7). That rule is the difference between an endpoint a `/24` sweep finds and one you have to go looking for, and it is a residual, not a control: the exporter has no authentication, so whoever owns `alexander` reads every gauge. It has no write API, so what leaks is the shape of the host (services, disks, uptime, the licence clock), the same class as `node_exporter` on the NAS below | | A range target with a path out | It has none — `ifrit`'s targets sit on a bridge with no physical port, on `172.30.30.0/24`, which the firewall does not route and on which nothing has a default route at all ([ADR-0014](adr/0014-put-ifrit-on-imaginationlan-and-give-the-targets-no-route.md), [ADR-0017](adr/0017-buy-ifrit-for-iops-and-keep-the-range-disposable.md)) | | Someone with the trusted Wi-Fi key quietly joining | Kea's lease log reaches Loki; `UnknownDeviceOnTrustedSegment` fires the first time a MAC appears on VLAN 50 in seven days ([ADR-0019](adr/0019-read-device-joins-from-the-dhcp-server.md)) | -| Losing visibility of a failure | 163 alert rules, 30 days of metrics and logs | +| Losing visibility of a failure | 167 alert rules, 30 days of metrics and logs | | Someone on a reachable VLAN silencing an alert to hide a failure | Alertmanager binds to `127.0.0.1`; silences go through authenticated Grafana | | Mains power loss | **The rack, yes; the monitoring path, yes — on two laptop cells that were measured for the first time on 2026-09-12.** A pack fitted to `mjolnir` on 2026-08-28 passed its self-test; the TP-Link carrying `prometheus` and `oracle` has been on UPS power since 2026-09-08 ([#110](https://github.com/Gerrrt/HomeLab/issues/110)); the laptops ride a cut out on their own batteries, which `HostBatteryHealthLow` in `host.rules.yaml` now reads — `prometheus`'s cell was replaced on 2026-09-18 and reads 101 % of design, `oracle`'s is the original at 72 %, with its replacement bought on 2026-09-19 and in transit ([#531](https://github.com/Gerrrt/HomeLab/issues/531)) — and **`prometheus`'s runtime on its cell was measured on 2026-09-19 — about 2.5 hours from full at the stack's load — while `oracle`'s never has been**; since the same day the projection is recorded on every cut and pages under thirty minutes (`HostBatteryRuntimeLow`, [#532](https://github.com/Gerrrt/HomeLab/issues/532)), but neither pack reports a moving cell temperature, so this row is answered for the monitoring host, and for the other only as far as its cell being healthy — see below. **What the UPS cannot answer is what happens when the cut outlasts the pack: as of 2026-09-20 nothing shuts down on its signal, and everything on the PDU — `morpheus`, `Saruman`, `neo` and `smaug`, which is in the media room on a long cord from that PDU — stops uncleanly when the pack empties, about 47 minutes in at 21 % load by the card's own unmeasured estimate.** [ADR-0049](adr/0049-shut-down-on-the-ups-from-a-nut-server-on-the-firewall.md) decides that the firewall's NUT server halts `Saruman` and `smaug` first and itself last, and [`shut-down-on-the-ups.md`](runbooks/shut-down-on-the-ups.md) is the build, the forced-shutdown proof and the one mains pull that measures the pack; until those are done the decision is a configuration nobody has tested ([#574](https://github.com/Gerrrt/HomeLab/issues/574)) | | The estate being down while the person who runs it is unavailable | **Documentation, yes; data, not yet off the estate.** ADR-0011 puts the emergency tier on paper; [ADR-0023](adr/0023-keep-the-household-recovery-path-outside-the-estate.md) extends the same reasoning to the sensitive tier's data. That tier has held real photographs since 2026-09-28, and their off-estate copy is built but has no holder yet ([#455](https://github.com/Gerrrt/HomeLab/issues/455)) — see below | diff --git a/scripts/README.md b/scripts/README.md index 42dc5ea5..329a43ce 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -117,6 +117,7 @@ they describe. The rest are installed on the host they describe by | `collect-smart-state.sh`, `render-smart-baselines.sh` | SMART health for disks the iLO cannot see, and the known reallocated-sector baselines ([ADR-0046](../docs/adr/0046-record-a-known-static-smart-count-as-a-baseline-not-a-silence.md)) | | `collect-zpool-state.sh`, `collect-truenas-version.sh` | `smaug`'s pool leaves and TrueNAS version | | `collect-gateway-state.sh` | The firewall's view of its uplinks and its DDNS record | +| `collect-cert-expiry.sh` | When the management consoles' certificates expire, read off a handshake inside each console's segment ([ADR-0084](../docs/adr/0084-read-management-certificate-expiry-from-inside-each-segment.md)) | | `collect_silences.py` | Alertmanager's silences, one series each, so a silence is watched rather than remembered | | `collect-container-state.sh` | Whether each container of a compose project is running | | `collect-guest-state.sh`, `collect-guest-disk-state.sh`, `collect-thin-pool-state.sh` | `Saruman`'s guests: running, how full, and how full its thin pools are | diff --git a/scripts/collect-cert-expiry.sh b/scripts/collect-cert-expiry.sh new file mode 100755 index 00000000..52cefa5b --- /dev/null +++ b/scripts/collect-cert-expiry.sh @@ -0,0 +1,209 @@ +#!/usr/bin/env bash +# +# When the management certificates expire, as metrics (#857, ADR-0084). +# +# WHAT NOTHING WAS WATCHING. Four consoles serve certificates that no probe +# reads: the pfSense GUI on morpheus, the iLO on shiva, Saruman's Proxmox UI on +# :8006 and the TrueNAS UI on smaug. Each is reachable from Hicks and nowhere +# else, which is deliberate, so the blackbox exporter on the monitoring host +# cannot see any of them — the iLO and pfSense probes are written out and +# disabled in targets/blackbox.yaml for exactly that reason. The first sign of +# an expiry would be a browser warning on the day someone needs the console. +# +# WHY IT RUNS INSIDE EACH SEGMENT. ADR-0084: opening four management ports to +# the monitoring host is four segmentation decisions made for a date that +# changes once a year. Instead this runs where the endpoint already is — +# +# pfsense-ui on the monitoring host, over the ssh to morpheus that +# `make gateway-state` already makes, against 127.0.0.1:443 +# pve-ui on Saruman, against 127.0.0.1:8006 +# ilo-ui on Saruman, against 10.0.30.10:443 — same VLAN, no pf rule +# truenas-ui on smaug, from the ADR-0047 root cron +# +# and nothing new is opened between segments. +# +# IT READS THE HANDSHAKE, NOT A FILE. The same principle as the blackbox expiry +# rules: what matters is the leaf the endpoint actually serves. A file on disk +# can be renewed while the daemon still holds the old one, and on pfSense the +# file route goes near config.xml, which has printed a private key into a +# transcript before (changelog, 2026-09-28). openssl s_client does not verify +# by default, which is what is wanted here: a self-signed or expired leaf must +# still report its date. +# +# OLD TLS. iLO 4 may not complete a handshake at OpenSSL 3's default security +# level. A failed read is retried once at SECLEVEL=0. That weakens nothing: +# nothing is sent and nothing read is trusted except a date. +# +# THE PROBE SCRIPT IS POSIX sh and runs wherever the endpoint is — locally, or +# on morpheus over ssh, where the login shell is not bash and `date` is not GNU. +# It prints one line per probe and the date arithmetic happens here, on Linux. +# +# "Could not read" is kept apart from "expiring". A probe that fails writes +# homelab_cert_checked 0 and no expiry sample, the homelab_ddns_record_checked +# pattern: a console that is briefly down must not look like a certificate that +# expired, and a reader that has quietly stopped working must not look like +# good news. +# +# Usage: scripts/collect-cert-expiry.sh [--ssh USER@HOST] [--host NAME] [--print] +# --probe ENDPOINT=HOST=ADDR:PORT [...] +# scripts/collect-cert-expiry.sh --self-test +# +# --probe endpoint name, the host label the series carries (`-` for this +# host's hostname), and the address to handshake with +# --host names the output file, cert-expiry-.prom; defaults to +# this host's hostname +# --ssh run the probes on that host instead of here +set -uo pipefail + +TEXTFILE_DIR="${TEXTFILE_DIR:-/var/lib/node_exporter/textfile_collector}" +PRINT_ONLY=0 +SSH_TARGET="" +FILE_HOST="" +PROBES=() + +die() { printf '\033[0;31merror:\033[0m %s\n' "$*" >&2; exit 1; } + +# Everything in a probe ends up interpolated into a script another host runs, +# so each part is checked for shape before it gets there. +valid_probe() { + [[ "$1" =~ ^[a-z0-9][a-z0-9-]*=([A-Za-z0-9][A-Za-z0-9._-]*|-)=[A-Za-z0-9][A-Za-z0-9.-]*:[0-9]{1,5}$ ]] +} + +# The script that does the handshakes. Takes "ENDPOINT ADDR:PORT" pairs and +# prints "ENDPOINT notAfter=" or "ENDPOINT -". +probe_script() { + cat <<'EOF' +read_end() { + timeout 20 openssl s_client -connect "$1" $2 /dev/null \ + | openssl x509 -noout -enddate 2>/dev/null +} +EOF + local pair endpoint addr + # shellcheck disable=SC2016 # $e and $(...) belong to the generated script + for pair in "$@"; do + endpoint="${pair%% *}"; addr="${pair#* }" + printf 'e=$(read_end %s "") ; [ -n "$e" ] || e=$(read_end %s "-cipher DEFAULT@SECLEVEL=0")\n' "$addr" "$addr" + printf 'printf "%%s %%s\\n" %s "${e:--}"\n' "$endpoint" + done +} + +# "notAfter=Apr 16 16:37:44 2027 GMT" -> epoch seconds, or nothing. +to_epoch() { + local s="${1#notAfter=}" + [[ -n "$s" && "$s" != "$1" ]] || return 0 + date -u -d "$s" +%s 2>/dev/null || true +} + +# Results ("ENDPOINT HOSTLABEL EPOCH-or-empty" per line) -> exposition format. +render() { + local lines="$1" endpoint hostlabel epoch + printf '# HELP homelab_cert_expiry_timestamp_seconds When the certificate this endpoint serves expires (notAfter).\n' + printf '# TYPE homelab_cert_expiry_timestamp_seconds gauge\n' + while read -r endpoint hostlabel epoch; do + [[ -n "$endpoint" && -n "$epoch" ]] || continue + printf 'homelab_cert_expiry_timestamp_seconds{endpoint="%s",host="%s"} %s\n' "$endpoint" "$hostlabel" "$epoch" + done <<<"$lines" + printf '# HELP homelab_cert_checked 1 when the endpoint completed a handshake and its certificate was read.\n' + printf '# TYPE homelab_cert_checked gauge\n' + while read -r endpoint hostlabel epoch; do + [[ -n "$endpoint" ]] || continue + printf 'homelab_cert_checked{endpoint="%s",host="%s"} %s\n' "$endpoint" "$hostlabel" "$([[ -n "$epoch" ]] && echo 1 || echo 0)" + done <<<"$lines" +} + +if [[ "${1:-}" == "--self-test" ]]; then + fail=0 + ok() { printf '\033[0;32m PASS\033[0m %s\n' "$1"; } + bad() { printf '\033[0;31m FAIL\033[0m %s\n' "$1"; fail=1; } + same() { if [[ "$2" == "$3" ]]; then ok "$1"; else bad "$1"; printf ' got %s\n expected %s\n' "$2" "$3"; fi; } + + # morpheus's leaf as read on 2026-10-07. + same "openssl's date to epoch" "$(to_epoch 'notAfter=Apr 16 16:37:44 2027 GMT')" 1807893464 + same "a single-digit day" "$(to_epoch 'notAfter=Jan 2 03:04:05 2030 GMT')" 1893553445 + same "a failed read is empty" "$(to_epoch '-')" "" + same "garbage is empty" "$(to_epoch 'notAfter=whenever')" "" + + out="$(render $'pve-ui Saruman 1807893464\nilo-ui shiva ')" + same "a read endpoint has both series" \ + "$(grep -c 'endpoint="pve-ui"' <<<"$out")" 2 + same "an unread endpoint has no expiry sample" \ + "$(grep -c '^homelab_cert_expiry_timestamp_seconds{endpoint="ilo-ui"' <<<"$out")" 0 + same "an unread endpoint is checked 0" \ + "$(grep '^homelab_cert_checked{endpoint="ilo-ui"' <<<"$out")" 'homelab_cert_checked{endpoint="ilo-ui",host="shiva"} 0' + + for p in 'pfsense-ui=morpheus=127.0.0.1:443' 'pve-ui=-=127.0.0.1:8006' 'ilo-ui=shiva=10.0.30.10:443'; do + if valid_probe "$p"; then ok "accepts $p"; else bad "accepts $p"; fi + done + # shellcheck disable=SC2016 # the $(id) is the injection being refused + for p in 'x=y=1.2.3.4' 'pve-ui=-=127.0.0.1:8006;reboot' 'a=$(id)=h:1' 'UI=h=h:1' 'a=h=h:1 2'; do + if valid_probe "$p"; then bad "rejects ${p}"; else ok "rejects ${p}"; fi + done + + if probe_script "pfsense-ui 127.0.0.1:443" "ilo-ui 10.0.30.10:443" | sh -n 2>/dev/null; then + ok "the probe script parses as sh" + else + bad "the probe script does not parse as sh" + fi + exit $fail +fi + +while (($#)); do + case "$1" in + --print) PRINT_ONLY=1; shift ;; + --ssh) SSH_TARGET="${2:-}"; shift 2 ;; + --host) FILE_HOST="${2:-}"; shift 2 ;; + --probe) PROBES+=("${2:-}"); shift 2 ;; + *) die "unknown argument $1" ;; + esac +done +((${#PROBES[@]})) || die "at least one --probe ENDPOINT=HOST=ADDR:PORT is required" +[[ -n "$FILE_HOST" ]] || FILE_HOST="$(hostname)" +[[ "$FILE_HOST" =~ ^[A-Za-z0-9][A-Za-z0-9._-]*$ ]] || die "--host ${FILE_HOST@Q} is not a host name" + +pairs=(); labels=() +for p in "${PROBES[@]}"; do + valid_probe "$p" || die "--probe ${p@Q} is not ENDPOINT=HOST=ADDR:PORT" + endpoint="${p%%=*}"; rest="${p#*=}"; hostlabel="${rest%%=*}"; addr="${rest#*=}" + [[ "$hostlabel" == - ]] && hostlabel="$(hostname)" + pairs+=("${endpoint} ${addr}"); labels+=("${endpoint} ${hostlabel}") +done + +STDERR_FILE="$(mktemp)"; trap 'rm -f "${STDERR_FILE}"' EXIT +if [[ -n "$SSH_TARGET" ]]; then + raw="$(probe_script "${pairs[@]}" \ + | ssh -o BatchMode=yes -o ConnectTimeout=10 "$SSH_TARGET" 'sh -s' 2>"${STDERR_FILE}")" +else + raw="$(probe_script "${pairs[@]}" | sh -s 2>"${STDERR_FILE}")" +fi +# No line at all means the probes never ran — ssh failed, or there is no +# openssl. That is the job failing, not every certificate being unreadable, so +# it exits non-zero and leaves the previous file for the staleness rules. +if [[ -z "$raw" ]]; then + detail="$(tr -d '\r' < "${STDERR_FILE}" | grep -v '^$' | tail -2 | paste -sd'; ' -)" + die "the probes did not run${SSH_TARGET:+ on ${SSH_TARGET}}${detail:+ — ${detail}}" +fi + +results="" +for l in "${labels[@]}"; do + endpoint="${l%% *}"; hostlabel="${l#* }" + line="$(grep -m1 "^${endpoint} " <<<"$raw")" + epoch="$(to_epoch "${line#* }")" + results+="${endpoint} ${hostlabel} ${epoch}"$'\n' +done + +if ((PRINT_ONLY)); then render "$results"; exit 0; fi + +PROM="${TEXTFILE_DIR}/cert-expiry-${FILE_HOST}.prom" +[[ -d "${TEXTFILE_DIR}" ]] || die "no ${TEXTFILE_DIR}" +tmp="${PROM}.$$" +render "$results" > "${tmp}" || { rm -f "${tmp}"; die "could not write ${tmp}"; } +chmod 0644 "${tmp}" +mv -f "${tmp}" "${PROM}" +while read -r endpoint _ epoch; do + [[ -n "$endpoint" ]] || continue + if [[ -n "$epoch" ]]; then + printf 'cert-expiry %s expires %s\n' "$endpoint" "$(date -u -d "@${epoch}" +%F)" + else + printf 'cert-expiry %s could not be read\n' "$endpoint" + fi +done <<<"$results" diff --git a/scripts/install-agent-collectors.sh b/scripts/install-agent-collectors.sh index 1d5620fc..cd8bbbd2 100755 --- a/scripts/install-agent-collectors.sh +++ b/scripts/install-agent-collectors.sh @@ -113,6 +113,7 @@ COLLECTORS=( "patch-state scripts/collect-patch-state.sh apt-patch-state.prom /usr/bin/apt-get" "smart-state scripts/collect-smart-state.sh smart-state-HOST.prom /usr/sbin/smartctl" "pve-version scripts/collect-pve-version.sh pve-version.prom /usr/bin/pveversion" + "cert-expiry scripts/collect-cert-expiry.sh cert-expiry-HOST.prom /usr/bin/pveproxy" "guest-state scripts/collect-guest-state.sh guest-state.prom /usr/sbin/qm" "thin-pool-state scripts/collect-thin-pool-state.sh thin-pool-state.prom /usr/sbin/lvs" "guest-disk-state scripts/collect-guest-disk-state.sh guest-disk-state.prom /usr/sbin/qm" diff --git a/stacks/observability/README.md b/stacks/observability/README.md index 83c5ac80..e471b0f1 100644 --- a/stacks/observability/README.md +++ b/stacks/observability/README.md @@ -53,7 +53,7 @@ prometheus/ targets/snmp.yaml SNMP targets — hot-reloaded, no restart needed targets/node.yaml node_exporter scrapes, for the host that runs no Alloy (smaug) targets/blackbox*.yaml probe targets, http, dns and latency — hot-reloaded, no restart - rules/*.rules.yaml 144 alert rules: host, network, ups, containers, blackbox, + rules/*.rules.yaml 148 alert rules: host, network, ups, containers, blackbox, dns, backup, ids, deploy, stack, internet and watchdog tests/*.test.yaml promtool unit tests — assert the rules can fire blackbox/blackbox.yaml probe modules — reachability, and what a resolver said diff --git a/stacks/observability/alertmanager/alertmanager.yaml b/stacks/observability/alertmanager/alertmanager.yaml index cd710261..af59d1ec 100644 --- a/stacks/observability/alertmanager/alertmanager.yaml +++ b/stacks/observability/alertmanager/alertmanager.yaml @@ -177,6 +177,13 @@ inhibit_rules: target_matchers: [alertname = "TlsAcmeRenewalLate"] equal: ["name"] + # And for the management consoles nothing can probe, whose expiry comes from + # a collector rather than the handshake blackbox makes (#857). `endpoint` is + # that collector's name for the console, unique across the estate. + - source_matchers: [alertname = "ManagementCertificateExpiryImminent"] + target_matchers: [alertname = "ManagementCertificateExpiringSoon"] + equal: ["endpoint"] + receivers: - name: "null" diff --git a/stacks/observability/prometheus/rules/host.rules.yaml b/stacks/observability/prometheus/rules/host.rules.yaml index 4ab7e587..48ea256b 100644 --- a/stacks/observability/prometheus/rules/host.rules.yaml +++ b/stacks/observability/prometheus/rules/host.rules.yaml @@ -1851,3 +1851,111 @@ groups: to outlast. If this is prometheus, that is the monitoring host: shut it down cleanly rather than letting the cell decide. HostOnBattery says why the mains is out. + + # Management console certificates (#857, ADR-0084): the pfSense GUI, the + # iLO, Saruman's :8006 and the TrueNAS UI. TlsCertificateExpiringSoon + # cannot see them — each answers Hicks only, so the blackbox exporter on + # the monitoring host never completes a handshake — and so these read + # scripts/collect-cert-expiry.sh's gauge instead, written by a handshake + # made inside each console's own segment. + # + # The same two tiers and thresholds as the blackbox pair, for the same + # reasons: 30 days is enough to reissue by hand, 7 is a reminder that was + # missed. The critical one inhibits the warning in alertmanager.yaml. + # `min by (endpoint, host)` so the alert carries only the console's name + # and the machine it belongs to — the iLO's series is collected on + # Saruman (instance="Saruman") and is about shiva (host="shiva"). + - alert: ManagementCertificateExpiringSoon + expr: >- + min by (endpoint, host) + (homelab_cert_expiry_timestamp_seconds - time()) + / 86400 < 30 + for: 1h + labels: + component: host + severity: warning + category: availability + annotations: + summary: >- + {{ $labels.endpoint }} on {{ $labels.host }} has a certificate + with {{ $value | printf "%.0f" }} days left + description: >- + Read from the handshake with the console itself, so this is the + certificate actually being served. Each of these is self-signed and + reissued from its own UI or shell — + docs/runbooks/generate-certificates.md#management-console-certificates + says how for each one. The alert resolves on the next collection + after the new certificate is served: within 15 minutes for + pfsense-ui, the next day for the others. + + - alert: ManagementCertificateExpiryImminent + expr: >- + min by (endpoint, host) + (homelab_cert_expiry_timestamp_seconds - time()) + / 86400 < 7 + for: 10m + labels: + component: host + severity: critical + category: availability + annotations: + runbook_url: "https://github.com/Gerrrt/HomeLab/blob/main/docs/runbooks/generate-certificates.md#management-console-certificates" + summary: >- + {{ $labels.endpoint }} on {{ $labels.host }} has a certificate + with {{ $value | printf "%.0f" }} days left + description: >- + Under a week, or already expired if the number is negative. The + console keeps answering, but browsers start refusing it, and this + is the console someone reaches for when something else is already + wrong. Reissue it now from the runbook. + + # The reader could not complete a handshake. Kept apart from the expiry + # rules on purpose (the homelab_ddns_record_checked pattern): an unread + # certificate has no expiry sample, so without this a console that + # stopped answering — or an iLO that OpenSSL refuses to talk to — would + # make the expiry alerts quietly impossible. Six hours, because a console + # restarting or a host rebooting is not worth a word. + - alert: ManagementCertificateUnchecked + expr: homelab_cert_checked == 0 + for: 6h + labels: + component: host + severity: warning + category: config + annotations: + summary: "{{ $labels.endpoint }} on {{ $labels.host }}: its certificate could not be read" + description: >- + The collector on {{ $labels.instance }} ran but got no certificate + from {{ $labels.endpoint }}, so its expiry is not being watched. Run + the collector by hand with --print to see the probe: for + pfsense-ui that is `make gateway-state` on the monitoring host, on + Saruman `systemctl start homelab-cert-expiry` and its journal, on + smaug the cron job's command. A console that is down says so + elsewhere; one that is up and refusing the handshake is a TLS + setting changed on the device. + + - alert: CertExpiryStateStale + # The collector has stopped rewriting its file, on any host. The + # SmartStateStale shape and reasons: the .prom persists and is + # re-served on every scrape, so the expiry samples stay present and a + # year away while nobody is reading them. On the monitoring host the + # writer is `make gateway-state`, which ScheduledJobStale watches too; + # on Saruman and smaug it is a timer and a TrueNAS cron job that + # nothing else knows about. Two days against a daily schedule. + expr: >- + time() - node_textfile_mtime_seconds{file=~".*cert-expiry-.*\\.prom"} > 2 * 86400 + for: 30m + labels: + component: host + severity: warning + category: correctness + annotations: + summary: "{{ $labels.instance }} has not rewritten {{ $labels.file }} in two days" + description: >- + The certificate expiry textfile is {{ $value | humanizeDuration }} + old, so ManagementCertificateExpiringSoon is reading dates that + nobody has checked since. On Saruman the writer is + homelab-cert-expiry.timer; on smaug a cron job in TrueNAS's UI + (System → Advanced Settings → Cron Jobs; build-the-nas.md §6.10); + on the monitoring host `make gateway-state`. Check the job still + exists and that its last run wrote the file. diff --git a/stacks/observability/prometheus/targets/blackbox.yaml b/stacks/observability/prometheus/targets/blackbox.yaml index 3344d4ca..ff75ca25 100644 --- a/stacks/observability/prometheus/targets/blackbox.yaml +++ b/stacks/observability/prometheus/targets/blackbox.yaml @@ -237,6 +237,12 @@ # default GUI certificate is 398 days and nothing else watches it — # but not worth having by accident. # +# Their CERTIFICATES are watched without either pass (#857, ADR-0084): +# scripts/collect-cert-expiry.sh reads pfSense's on the firewall over the ssh +# `make gateway-state` already makes, and the iLO's from Saruman, which shares +# its segment. So the case for these probes is reachability alone now, not +# expiry — weigh the pass on that. +# # Once the rule exists, verify from the running exporter before uncommenting: # # docker exec blackbox-exporter wget -qO- \ diff --git a/stacks/observability/prometheus/tests/cert-expiry.test.yaml b/stacks/observability/prometheus/tests/cert-expiry.test.yaml new file mode 100644 index 00000000..9da34e40 --- /dev/null +++ b/stacks/observability/prometheus/tests/cert-expiry.test.yaml @@ -0,0 +1,206 @@ +--- +# Unit tests for the management console certificate rules in host.rules.yaml +# (#857, ADR-0084). In their own file because host.test.yaml evaluates every +# 5 minutes and these want a minute's resolution for the `for` cases. +# +# The arithmetic is blackbox.test.yaml's: an expiry of D days at eval time T +# is D × 86400 + T in seconds, so the summary's day count is exact. Every +# firing case has a quiet partner — a test that only expects silence would +# pass against a rule that can never fire. +# +# The labels are the real shapes. Saruman's Alloy sets instance="Saruman" and +# the collector's own host label is the console's machine, so the iLO's series +# is instance="Saruman", host="shiva". smaug's comes through the media +# node_exporter scrape with job="node". + +rule_files: + - ../rules/host.rules.yaml + +evaluation_interval: 1m + +tests: + # --- Expiry: 20 days out, beside one a year away ------------------------- + # 1735200 = 20 × 86400 + 7200, so at 2h the division is exactly 20. + - interval: 1m + input_series: + - series: 'homelab_cert_expiry_timestamp_seconds{endpoint="pve-ui", host="Saruman", + instance="Saruman", job="Saruman-metrics"}' + values: "1735200+0x180" + - series: 'homelab_cert_expiry_timestamp_seconds{endpoint="pfsense-ui", host="morpheus", + instance="prometheus", job="prometheus-metrics"}' + values: "34567200+0x180" + alert_rule_test: + # Under 30 days from t=0, but `for: 1h`. + - eval_time: 30m + alertname: ManagementCertificateExpiringSoon + exp_alerts: [] + + # `instance` and `job` are gone: `min by (endpoint, host)` keeps only + # what names the console. + - eval_time: 2h + alertname: ManagementCertificateExpiringSoon + exp_alerts: + - exp_labels: + alertname: ManagementCertificateExpiringSoon + endpoint: pve-ui + host: Saruman + component: host + severity: warning + category: availability + exp_annotations: + summary: "pve-ui on Saruman has a certificate with 20 days left" + description: >- + Read from the handshake with the console itself, so this is the + certificate actually being served. Each of these is self-signed and + reissued from its own UI or shell — + docs/runbooks/generate-certificates.md#management-console-certificates + says how for each one. The alert resolves on the next collection + after the new certificate is served: within 15 minutes for + pfsense-ui, the next day for the others. + + - eval_time: 2h + alertname: ManagementCertificateExpiryImminent + exp_alerts: [] + + # --- Expiry: 3 days out, the iLO as read from Saruman ---------------------- + # 262800 = 3 × 86400 + 3600; at 2h that is 2.96 days, printed as 3. Both + # tiers fire at Prometheus; alertmanager.yaml's inhibit keeps the warning off + # the phone, which promtool cannot see. + - interval: 1m + input_series: + - series: 'homelab_cert_expiry_timestamp_seconds{endpoint="ilo-ui", host="shiva", + instance="Saruman", job="Saruman-metrics"}' + values: "262800+0x180" + alert_rule_test: + # `for: 10m` on the critical tier. + - eval_time: 5m + alertname: ManagementCertificateExpiryImminent + exp_alerts: [] + + - eval_time: 2h + alertname: ManagementCertificateExpiryImminent + exp_alerts: + - exp_labels: + alertname: ManagementCertificateExpiryImminent + endpoint: ilo-ui + host: shiva + component: host + severity: critical + category: availability + exp_annotations: + runbook_url: "https://github.com/Gerrrt/HomeLab/blob/main/docs/runbooks/generate-certificates.md#management-console-certificates" + summary: "ilo-ui on shiva has a certificate with 3 days left" + description: >- + Under a week, or already expired if the number is negative. The + console keeps answering, but browsers start refusing it, and this + is the console someone reaches for when something else is already + wrong. Reissue it now from the runbook. + + - eval_time: 2h + alertname: ManagementCertificateExpiringSoon + exp_alerts: + - exp_labels: + alertname: ManagementCertificateExpiringSoon + endpoint: ilo-ui + host: shiva + component: host + severity: warning + category: availability + exp_annotations: + summary: "ilo-ui on shiva has a certificate with 3 days left" + description: >- + Read from the handshake with the console itself, so this is the + certificate actually being served. Each of these is self-signed and + reissued from its own UI or shell — + docs/runbooks/generate-certificates.md#management-console-certificates + says how for each one. The alert resolves on the next collection + after the new certificate is served: within 15 minutes for + pfsense-ui, the next day for the others. + + # --- Unchecked: a console that stopped answering -------------------------- + # No expiry sample exists for truenas-ui — the collector leaves it out — so + # the expiry rules cannot fire for it, and this is the only thing that says + # so. pve-ui read fine and must stay quiet. + - interval: 1m + input_series: + - series: 'homelab_cert_checked{endpoint="truenas-ui", host="smaug", instance="smaug", job="node"}' + values: "0x480" + - series: 'homelab_cert_checked{endpoint="pve-ui", host="Saruman", instance="Saruman", job="Saruman-metrics"}' + values: "1x480" + alert_rule_test: + # A console restarting is not worth a word: `for: 6h`. + - eval_time: 5h + alertname: ManagementCertificateUnchecked + exp_alerts: [] + + - eval_time: 7h + alertname: ManagementCertificateUnchecked + exp_alerts: + - exp_labels: + alertname: ManagementCertificateUnchecked + endpoint: truenas-ui + host: smaug + instance: smaug + job: node + component: host + severity: warning + category: config + exp_annotations: + summary: "truenas-ui on smaug: its certificate could not be read" + description: >- + The collector on smaug ran but got no certificate from + truenas-ui, so its expiry is not being watched. Run the collector + by hand with --print to see the probe: for pfsense-ui that is + `make gateway-state` on the monitoring host, on Saruman + `systemctl start homelab-cert-expiry` and its journal, on smaug + the cron job's command. A console that is down says so + elsewhere; one that is up and refusing the handshake is a TLS + setting changed on the device. + + # --- Stale: smaug's cron job stopped ------------------------------------- + # Written at t=0 and never again: 49 hours old at 49h, past the two-day line + # for longer than `for`. A different textfile at the same age must not match. + - interval: 5m + input_series: + - series: 'node_textfile_mtime_seconds{instance="smaug", job="node", file="/textfile/cert-expiry-smaug.prom"}' + values: "0x700" + - series: 'node_textfile_mtime_seconds{instance="smaug", job="node", file="/textfile/truenas-version.prom"}' + values: "0x700" + alert_rule_test: + - eval_time: 47h + alertname: CertExpiryStateStale + exp_alerts: [] + + - eval_time: 49h + alertname: CertExpiryStateStale + exp_alerts: + - exp_labels: + alertname: CertExpiryStateStale + instance: smaug + job: node + file: /textfile/cert-expiry-smaug.prom + component: host + severity: warning + category: correctness + exp_annotations: + summary: "smaug has not rewritten /textfile/cert-expiry-smaug.prom in two days" + description: >- + The certificate expiry textfile is 2d 1h 0m 0s old, so + ManagementCertificateExpiringSoon is reading dates that nobody has + checked since. On Saruman the writer is homelab-cert-expiry.timer; + on smaug a cron job in TrueNAS's UI (System → Advanced Settings → + Cron Jobs; build-the-nas.md §6.10); on the monitoring host `make + gateway-state`. Check the job still exists and that its last run + wrote the file. + + # --- Stale: quiet while a daily job keeps rewriting it ------------------- + # Saruman's file, rewritten every day: never more than a day old. + - interval: 1h + input_series: + - series: 'node_textfile_mtime_seconds{instance="Saruman", job="Saruman-metrics", + file="/var/lib/node_exporter/textfile_collector/cert-expiry-Saruman.prom"}' + values: "0x23 86400x23 172800x23" + alert_rule_test: + - eval_time: 71h + alertname: CertExpiryStateStale + exp_alerts: [] diff --git a/systemd/README.md b/systemd/README.md index c3faa2d4..97c54130 100644 --- a/systemd/README.md +++ b/systemd/README.md @@ -56,7 +56,7 @@ editing units, not installing them. | `homelab-pkg-state` | daily 09:00 | `morpheus`'s package state, over SSH | | `homelab-recipient-state` | daily 09:15 | Which age recipients can open the secrets, and when each was proved | | `homelab-ca-key-state` | daily 09:30 | The CA key's fingerprint, and when its offline copy was last proved | -| `homelab-gateway-state` | every 15 min | The firewall's view of its uplinks and its DDNS record | +| `homelab-gateway-state` | every 15 min | The firewall's view of its uplinks, its DDNS record and its GUI certificate's expiry | | `homelab-silence-state` | every 15 min, at :05 | Alertmanager's silences, as metrics | | `homelab-snmp-verify` | Wednesday 06:30 | Every SNMP device answers to its current credential | | `homelab-check-versions` | Wednesday 06:45 | The documented OS versions against what the hosts report | @@ -93,6 +93,7 @@ for it in `stacks/lab/prometheus/rules/lab.rules.yaml`. | `homelab-prune-images` | Monday 04:00 | hosts that run Docker | Removes unused images | | `homelab-drift-check` | daily 06:30 | `oracle` | The wiki's drift check | | `homelab-pve-version` | daily 08:15 | `Saruman` | The Proxmox VE version | +| `homelab-cert-expiry` | daily 08:30 | `Saruman` | When its own `:8006` certificate and the iLO's expire ([#857](https://github.com/Gerrrt/HomeLab/issues/857)) | | `homelab-pve-firewall-state` | every 5 min | `Saruman` | Whether the Proxmox firewall is on | | `homelab-guest-state` | every 10 min | `Saruman` | Which guests are running | | `homelab-guest-disk-state` | every 10 min | `Saruman` | How full the guests' filesystems are | diff --git a/systemd/agent/homelab-cert-expiry.service b/systemd/agent/homelab-cert-expiry.service new file mode 100644 index 00000000..9e7de617 --- /dev/null +++ b/systemd/agent/homelab-cert-expiry.service @@ -0,0 +1,46 @@ +# When Saruman's console certificates expire, written where the Alloy agent will +# find it (#857, ADR-0084). +# +# TWO ENDPOINTS, BOTH READ FROM HERE. Proxmox's own UI on :8006, and the iLO of +# the machine Saruman runs on (shiva, 10.0.30.10). The monitoring host cannot +# reach either: VLAN 99 passes SNMP to the iLO and nothing else (ADR-0033), and +# Saruman's UI answers Hicks and phoenix only. Saruman shares the iLO's segment, +# so its handshake with 10.0.30.10 crosses no pf rule at all — which is the +# whole reason the iLO is read from here rather than given a firewall pass. +# +# THE NETWORK IS NEEDED, unlike the other agent collectors here, so there is no +# PrivateNetwork=true. It is limited to IP sockets instead. +# +# ROOT ONLY FOR THE DIRECTORY, as with pve-version: a handshake needs no +# privilege, and the textfile directory is root-owned. +[Unit] +Description=HomeLab: record when Saruman's and the iLO's console certificates expire +Documentation=file:///home/robo/code/Gerrrt/HomeLab/docs/runbooks/generate-certificates.md +ConditionPathIsDirectory=/var/lib/node_exporter/textfile_collector +ConditionPathExists=/usr/bin/pveproxy +Wants=network-online.target +After=network-online.target + +[Service] +Type=oneshot +User=root +Group=root +ExecStart=/usr/local/bin/homelab-collect-cert-expiry --probe pve-ui=-=127.0.0.1:8006 --probe ilo-ui=shiva=10.0.30.10:443 +TimeoutStartSec=120 +Restart=no +Nice=10 +NoNewPrivileges=true +RestrictSUIDSGID=true +LockPersonality=true +RestrictRealtime=true +PrivateTmp=true +ProtectHome=true +ProtectSystem=strict +ReadWritePaths=/var/lib/node_exporter/textfile_collector +ProtectKernelTunables=true +ProtectKernelModules=true +ProtectControlGroups=true +RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX +StandardOutput=journal +StandardError=journal +SyslogIdentifier=homelab-cert-expiry diff --git a/systemd/agent/homelab-cert-expiry.timer b/systemd/agent/homelab-cert-expiry.timer new file mode 100644 index 00000000..f623a770 --- /dev/null +++ b/systemd/agent/homelab-cert-expiry.timer @@ -0,0 +1,17 @@ +[Unit] +Description=HomeLab: daily console certificate expiry record +Documentation=file:///home/robo/code/Gerrrt/HomeLab/docs/runbooks/generate-certificates.md + +[Timer] +Unit=homelab-cert-expiry.service +# Daily. The warning is at 30 days, so a day of resolution costs nothing, and +# CertExpiryStateStale allows two days before calling the file stale. +# +# Persistent=true so a host that was off at 08:30 records on its next boot. +OnCalendar=*-*-* 08:30:00 +RandomizedDelaySec=900 +Persistent=true +AccuracySec=1m + +[Install] +WantedBy=timers.target From b51b93cdd6d24a62c4339cba95a138a9aee4846b Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Wed, 7 Oct 2026 13:42:48 +0000 Subject: [PATCH 2/2] docs(changelog): drop two conflict base sections left in by my resolutions (#857) Both conflict resolutions on this branch kept zdiff3's `|||||||` base section, stale copies of other entries. Rebuilt from main's copy at the merge parent plus this PR's one entry. Co-Authored-By: Claude Opus 5.5 --- docs/changelog.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/docs/changelog.md b/docs/changelog.md index 3fa890d6..283c5206 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -42,7 +42,6 @@ docstring gives: it is a record, not a claim about now. console: `make install-agent-collectors` from the Mac for Saruman, and [`build-the-nas.md`](runbooks/build-the-nas.md) §6.10 for smaug. #857 stays open until all four `endpoint` series exist. -||||||| c191bca - **#182's deployment, recorded late** ([#182](https://github.com/Gerrrt/HomeLab/issues/182), @@ -157,7 +156,6 @@ docstring gives: it is a record, not a claim about now. entry. It took about three minutes. - **The fixture.** `backup-library.sh --self-test` now flips one byte in a set and truncates another, and `--verify-only --all` fails on both. -||||||| parent of 4ab91f2 (wip #857) - **Syslog stores only the senders it names.** [#844](https://github.com/Gerrrt/HomeLab/issues/844) found that any host