Repository navigation
feat(observability): expiry gauges for the four management consoles' certificates (#857) - #960
Merged
Merged
Conversation
…certificates (#857) The pfSense GUI, the iLO, Saruman's :8006 and the TrueNAS UI each answer Hicks only, so no blackbox probe could read their certificates. ADR-0084 takes the issue's option 2: scripts/collect-cert-expiry.sh handshakes with each console from inside its own segment and writes homelab_cert_expiry_timestamp_seconds, with no new firewall pass. - pfsense-ui: read on morpheus over the ssh `make gateway-state` makes - pve-ui, ilo-ui: read on Saruman by a new agent unit (same VLAN as the iLO) - truenas-ui: read on smaug by a root cron job (build-the-nas.md §6.10) Rules: ManagementCertificateExpiringSoon (30d), ...ExpiryImminent (7d, inhibits the warning), ManagementCertificateUnchecked, CertExpiryStateStale. Refs #857: the Saruman and smaug halves need someone at a console. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Copilot review overview
🟡 Changes recommended
Collector error handling can hang execution or report success without publishing metrics correctly.
Review effort: Balanced
Findings: 2
Open (2)
What changed in this PR
Adds certificate-expiry monitoring for four management consoles without new firewall access, addressing #857. Saruman and smaug still require manual deployment.
Changes:
- Adds a TLS certificate collector with gateway, agent-timer, and TrueNAS cron integration.
- Adds four alerts, rule tests, and warning inhibition.
- Documents the decision, deployment steps, and renewal procedures.
| File | Description |
|---|---|
systemd/README.md |
Documents collector schedules. |
systemd/agent/homelab-cert-expiry.timer |
Schedules daily collection. |
systemd/agent/homelab-cert-expiry.service |
Probes Proxmox and iLO. |
stacks/observability/README.md |
Updates alert totals. |
stacks/observability/prometheus/tests/cert-expiry.test.yaml |
Tests expiry and collection alerts. |
stacks/observability/prometheus/targets/blackbox.yaml |
Clarifies disabled probes’ remaining purpose. |
stacks/observability/prometheus/rules/host.rules.yaml |
Adds four certificate-monitoring alerts. |
stacks/observability/alertmanager/alertmanager.yaml |
Inhibits warnings during critical alerts. |
scripts/README.md |
Lists the new collector. |
scripts/install-agent-collectors.sh |
Registers agent installation. |
scripts/collect-cert-expiry.sh |
Collects and publishes certificate metrics. |
README.md |
Updates alert and ADR counts. |
Makefile |
Adds pfSense collection to gateway-state. |
docs/security.md |
Updates alert totals. |
docs/runbooks/schedule-maintenance.md |
Lists the new agent collector. |
docs/runbooks/generate-certificates.md |
Documents console certificate renewal. |
docs/runbooks/deploy-stack.md |
Updates expected rule counts. |
docs/runbooks/build-the-nas.md |
Adds TrueNAS collection setup. |
docs/observability.md |
Describes alerts and disclosed metrics. |
docs/changelog.md |
Records implementation and pending deployment. |
docs/adr/0084-read-management-certificate-expiry-from-inside-each-segment.md |
Records the segment-local collection decision. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Gerrrt
enabled auto-merge (squash)
October 7, 2026 13:29
….prom cannot be installed Conflicts were the counted claims and the changelog; counts re-derived by check_docs.py (150 Prometheus rules, 169 total, 82 ADRs). Review on #960: - a trailing --probe/--ssh/--host made `shift 2` fail without moving and the parser loop forever; each now needs a value or dies - chmod and mv of the rendered file were unchecked under no errexit; a failure now removes the temp file and exits non-zero Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tions (#857) Both conflict resolutions on this branch kept zdiff3's `|||||||` base section, stale copies of other entries. Rebuilt from main's copy at the merge parent plus this PR's one entry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ording) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gerrrt
added a commit
that referenced
this pull request
Oct 7, 2026
…082s (#1015) * docs(adr): renumber the lab host-keys ADR to 0085; main had two ADR-0082s #962 (the wiki over TLS) and #931 (lab credential hygiene) each landed an ADR-0082, minutes apart, so check_docs fails on main ("ADR-0082 is claimed by 2 files … renumber the one that landed second") and the README's count is one short. #931 landed second, so its ADR moves. 0083 and 0084 are claimed by open PRs (#928, #959, #960), so it takes 0085. Every reference to it moves with it; the wiki's ADR-0082 references are untouched. README: 83 ADRs. Refs #846 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(adr): record the renumber in ADR-0085, as ADR-0049 records its own The convention from ADR-0049: a renumbered ADR opens with a note naming the number it landed as, the PR that took that number first and when, the rule that chose it, and that the decision is untouched. Refs #846 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 7, 2026
Gerrrt
added a commit
that referenced
this pull request
Oct 7, 2026
main renumbered the host-keys ADR to 0085 in #1015, with a dated note on where the number came from; 0083 and 0084 were taken by #928 and #960. This branch's own renumber to 0083 is dropped in favour of main's, so the PR keeps only the grant fix in the runbook, the clearer error in lab-known-hosts.sh, and the As-run row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Refs #857. The issue stays open until all four series exist, because two of them need someone at a console (see below).
What
Four management consoles serve self-signed certificates that nothing watched: the pfSense GUI, the iLO, Saruman's
:8006and the TrueNAS UI. Each answers Hicks only, so the blackbox exporter can read none of them. This takes the issue's option 2, recorded as ADR-0084: read each certificate's expiry from inside its own segment, with no new firewall pass.scripts/collect-cert-expiry.shmakes a TLS handshake (openssl s_client) with each console from a host that can already reach it, and writes two series per endpoint:homelab_cert_expiry_timestamp_seconds{endpoint,host}homelab_cert_checked{endpoint,host}A failed read writes
checked 0and no expiry sample, the DDNS pattern. The probe is POSIXsh, so it runs on morpheus over ssh too, and the date parsing happens on Linux.pfsense-ui is read on morpheus (
127.0.0.1:443) by the existingmake gateway-statetarget, every 15 minutes, with no new timer.pve-ui and ilo-ui are read on Saruman by a new agent unit,
homelab-cert-expiry, daily. The iLO shares Saruman's segment, so no pf rule is crossed.truenas-ui is read on smaug by a root cron job (ADR-0047 mechanism), set up in
build-the-nas.md§6.10.Rules in
host.rules.yaml:ManagementCertificateExpiringSoon(30d, warning)ManagementCertificateExpiryImminent(7d, critical, inhibits the warning)ManagementCertificateUncheckedCertExpiryStateStaleTests are in
tests/cert-expiry.test.yaml.Docs: a new "Management console certificates" section in
generate-certificates.md, which the rules link to. The disabled iLO and pfSense blackbox probes stay disabled, and their comment now says the case for them is reachability alone.Verified
make gateway-stateagainst the live firewall withTEXTFILE_DIRset to scratch. It wrotehomelab_cert_expiry_timestamp_seconds{endpoint="pfsense-ui",host="morpheus"} 1807893464(2027-04-16 16:37:44 UTC), which matches the 2026-09-28 regeneration.checked 0and no expiry line.--self-testsuite passes all 16 cases.promtool test rulespassescert-expiry.test.yaml.check_docs.pypasses with the counts updated: 148 metric rules, 167 in total, 81 ADRs.After merge: two hand steps
make install-agent-collectors AGENT=root@10.0.30.110 ARGS="--only cert-expiry".iLO 4 may need the
SECLEVEL=0retry the script already makes. If it still will not handshake,ManagementCertificateUncheckedsays so after 6h.build-the-nas.md§6.10, at the TrueNAS shell and UI.Check from Prometheus:
count by (endpoint) (homelab_cert_expiry_timestamp_seconds)should list all four.🤖 Generated with Claude Code