Skip to content

feat(observability): expiry gauges for the four management consoles' certificates (#857) - #960

Merged
Gerrrt merged 6 commits into
mainfrom
gerrrt/cert-expiry-monitoring-071c85
Oct 7, 2026
Merged

Gerrrt merged 6 commits into
mainfrom
gerrrt/cert-expiry-monitoring-071c85

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Oct 7, 2026

Copy link
Copy Markdown
Owner

Refs #857. The issue stays open until all four series exist, because two of them need someone at a console (see below).

What

Four management consoles serve self-signed certificates that nothing watched: the pfSense GUI, the iLO, Saruman's :8006 and the TrueNAS UI. Each answers Hicks only, so the blackbox exporter can read none of them. This takes the issue's option 2, recorded as ADR-0084: read each certificate's expiry from inside its own segment, with no new firewall pass.

  • scripts/collect-cert-expiry.sh makes a TLS handshake (openssl s_client) with each console from a host that can already reach it, and writes two series per endpoint:

    • homelab_cert_expiry_timestamp_seconds{endpoint,host}
    • homelab_cert_checked{endpoint,host}

    A failed read writes checked 0 and no expiry sample, the DDNS pattern. The probe is POSIX sh, so it runs on morpheus over ssh too, and the date parsing happens on Linux.

  • pfsense-ui is read on morpheus (127.0.0.1:443) by the existing make gateway-state target, every 15 minutes, with no new timer.

  • pve-ui and ilo-ui are read on Saruman by a new agent unit, homelab-cert-expiry, daily. The iLO shares Saruman's segment, so no pf rule is crossed.

  • truenas-ui is read on smaug by a root cron job (ADR-0047 mechanism), set up in build-the-nas.md §6.10.

  • Rules in host.rules.yaml:

    • ManagementCertificateExpiringSoon (30d, warning)
    • ManagementCertificateExpiryImminent (7d, critical, inhibits the warning)
    • ManagementCertificateUnchecked
    • CertExpiryStateStale

    Tests are in tests/cert-expiry.test.yaml.

  • Docs: a new "Management console certificates" section in generate-certificates.md, which the rules link to. The disabled iLO and pfSense blackbox probes stay disabled, and their comment now says the case for them is reachability alone.

Verified

  • I ran make gateway-state against the live firewall with TEXTFILE_DIR set to scratch. It wrote homelab_cert_expiry_timestamp_seconds{endpoint="pfsense-ui",host="morpheus"} 1807893464 (2027-04-16 16:37:44 UTC), which matches the 2026-09-28 regeneration.
  • A probe at a closed port wrote checked 0 and no expiry line.
  • The --self-test suite passes all 16 cases. promtool test rules passes cert-expiry.test.yaml. check_docs.py passes with the counts updated: 148 metric rules, 167 in total, 81 ADRs.

After merge: two hand steps

  1. Saruman, from the Mac, because VLAN 99 cannot reach VLAN 30:
    make install-agent-collectors AGENT=root@10.0.30.110 ARGS="--only cert-expiry".
    iLO 4 may need the SECLEVEL=0 retry the script already makes. If it still will not handshake, ManagementCertificateUnchecked says so after 6h.
  2. smaug: build-the-nas.md §6.10, at the TrueNAS shell and UI.

Check from Prometheus: count by (endpoint) (homelab_cert_expiry_timestamp_seconds) should list all four.

🤖 Generated with Claude Code

…certificates (#857)

The pfSense GUI, the iLO, Saruman's :8006 and the TrueNAS UI each answer
Hicks only, so no blackbox probe could read their certificates. ADR-0084
takes the issue's option 2: scripts/collect-cert-expiry.sh handshakes
with each console from inside its own segment and writes
homelab_cert_expiry_timestamp_seconds, with no new firewall pass.

- pfsense-ui: read on morpheus over the ssh `make gateway-state` makes
- pve-ui, ilo-ui: read on Saruman by a new agent unit (same VLAN as the iLO)
- truenas-ui: read on smaug by a root cron job (build-the-nas.md §6.10)

Rules: ManagementCertificateExpiringSoon (30d), ...ExpiryImminent (7d,
inhibits the warning), ManagementCertificateUnchecked, CertExpiryStateStale.

Refs #857: the Saruman and smaug halves need someone at a console.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 7, 2026 07:10

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Collector error handling can hang execution or report success without publishing metrics correctly.

Review effort: Balanced
Findings: 2 Medium severity

Open (2)
What changed in this PR

Adds certificate-expiry monitoring for four management consoles without new firewall access, addressing #857. Saruman and smaug still require manual deployment.

Changes:

  • Adds a TLS certificate collector with gateway, agent-timer, and TrueNAS cron integration.
  • Adds four alerts, rule tests, and warning inhibition.
  • Documents the decision, deployment steps, and renewal procedures.
File Description
systemd/​README.md Documents collector schedules.
systemd/​agent/​homelab-cert-expiry.timer Schedules daily collection.
systemd/​agent/​homelab-cert-expiry.service Probes Proxmox and iLO.
stacks/​observability/​README.md Updates alert totals.
stacks/​observability/​prometheus/​tests/​cert-expiry.test.yaml Tests expiry and collection alerts.
stacks/​observability/​prometheus/​targets/​blackbox.yaml Clarifies disabled probes’ remaining purpose.
stacks/​observability/​prometheus/​rules/​host.rules.yaml Adds four certificate-monitoring alerts.
stacks/​observability/​alertmanager/​alertmanager.yaml Inhibits warnings during critical alerts.
scripts/​README.md Lists the new collector.
scripts/​install-agent-collectors.sh Registers agent installation.
scripts/​collect-cert-expiry.sh Collects and publishes certificate metrics.
README.md Updates alert and ADR counts.
Makefile Adds pfSense collection to gateway-state.
docs/​security.md Updates alert totals.
docs/​runbooks/​schedule-maintenance.md Lists the new agent collector.
docs/​runbooks/​generate-certificates.md Documents console certificate renewal.
docs/​runbooks/​deploy-stack.md Updates expected rule counts.
docs/​runbooks/​build-the-nas.md Adds TrueNAS collection setup.
docs/​observability.md Describes alerts and disclosed metrics.
docs/​changelog.md Records implementation and pending deployment.
docs/​adr/​0084-read-management-certificate-expiry-from-inside-each-segment.md Records the segment-local collection decision.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread scripts/collect-cert-expiry.sh Resolved
Comment thread scripts/collect-cert-expiry.sh Resolved
@Gerrrt
Gerrrt enabled auto-merge (squash) October 7, 2026 13:29
Gerrrt and others added 3 commits October 7, 2026 13:41
….prom cannot be installed

Conflicts were the counted claims and the changelog; counts re-derived
by check_docs.py (150 Prometheus rules, 169 total, 82 ADRs).

Review on #960:
- a trailing --probe/--ssh/--host made `shift 2` fail without moving
  and the parser loop forever; each now needs a value or dies
- chmod and mv of the rendered file were unchecked under no errexit;
  a failure now removes the temp file and exits non-zero

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…tions (#857)

Both conflict resolutions on this branch kept zdiff3's `|||||||` base
section, stale copies of other entries. Rebuilt from main's copy at the
merge parent plus this PR's one entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ording)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gerrrt added a commit that referenced this pull request Oct 7, 2026
…082s (#1015)

* docs(adr): renumber the lab host-keys ADR to 0085; main had two ADR-0082s

#962 (the wiki over TLS) and #931 (lab credential hygiene) each landed
an ADR-0082, minutes apart, so check_docs fails on main ("ADR-0082 is
claimed by 2 files … renumber the one that landed second") and the
README's count is one short. #931 landed second, so its ADR moves.
0083 and 0084 are claimed by open PRs (#928, #959, #960), so it takes
0085. Every reference to it moves with it; the wiki's ADR-0082
references are untouched. README: 83 ADRs.

Refs #846

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(adr): record the renumber in ADR-0085, as ADR-0049 records its own

The convention from ADR-0049: a renumbered ADR opens with a note naming
the number it landed as, the PR that took that number first and when,
the rule that chose it, and that the decision is untouched.

Refs #846

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Gerrrt and others added 2 commits October 7, 2026 14:07
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Gerrrt
Gerrrt merged commit 51a95dc into main Oct 7, 2026
8 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/cert-expiry-monitoring-071c85 branch October 7, 2026 14:30
Gerrrt added a commit that referenced this pull request Oct 7, 2026
main renumbered the host-keys ADR to 0085 in #1015, with a dated note
on where the number came from; 0083 and 0084 were taken by #928 and
#960. This branch's own renumber to 0083 is dropped in favour of
main's, so the PR keeps only the grant fix in the runbook, the clearer
error in lab-known-hosts.sh, and the As-run row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants