Skip to content

feat(ingest): serve 9090 and 3100 over TLS, and move every client to https (#764) - #959

Merged
Gerrrt merged 1 commit into
mainfrom
gerrrt/ingest-ports-tls-flip
Oct 7, 2026
Merged

Gerrrt merged 1 commit into
mainfrom
gerrrt/ingest-ports-tls-flip

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Oct 7, 2026 •

Copy link
Copy Markdown
Owner

Closes #764. This is the second of two PRs. The first, #932, gave every client trust in the estate CA while the URLs stayed http.

Prerequisites, done 2026-10-07

Every client already trusts the estate CA, so this PR only flips the scheme.

  • feat(ingest): every client can verify the estate CA, ahead of TLS on 9090 and 3100 (#764) #932 is deployed on prometheus.
    • ingest-proxy mounts the prometheus.matrix.elysium leaf. It can read the 0640 key, and the leaf verifies against the committed CA.
    • The running blackbox exporter has the lab-CA ca_file on http_401_ingest_refused.
    • All four ingest probes are green.
  • trinity: its converge applied feat(ingest): every client can verify the estate CA, ahead of TLS on 9090 and 3100 (#764) #932 at 16:28. Home Assistant and Homepage were recreated with REQUESTS_CA_BUNDLE and NODE_EXTRA_CA_CERTS. Both are healthy, and each bundle contains the estate CA, checked by fingerprint.
  • Agents. Each holds /etc/alloy/ingest-ca.pem with INGEST_CA_FILE set.
    • oracle and trinity were redeployed with deploy-agent.sh, and both passed its arrival check.
    • Saruman was redeployed from the Mac. Its arrival check was asked from prometheus, since Hicks can't reach 9090: samples about 15 s old, a Loki line from that second, and no drops or refusals.

The branch is rebuilt on main and holds this one commit. Its ADR is 0086, because #928, #960 and the host-keys renumber took 0083–0085.

What changes

  • Caddyfile. tls /etc/caddy/tls/cert.pem /etc/caddy/tls/key.pem on the :9090 and :3100 sites, through an ingest_tls snippet. :2020 stays plain. auto_https off stays too, because it controls certificate management and redirects, not TLS.
  • check_caddyfile.sh mounts a throwaway certificate and key at those paths, because caddy validate loads them. Without the mounts it fails, which I checked.
  • deploy-agent.sh. For the estate it uses https:// URLs and passes --cacert with the committed ingest-ca.pem to every arrival-check curl. The lab (--monitoring-host 10.0.30.40) stays on http.
  • Clients. Homepage's five Prometheus widgets and Home Assistant's rack-power rest resource now use https://. Both verify through the bundles feat(ingest): every client can verify the estate CA, ahead of TLS on 9090 and 3100 (#764) #932 added.
  • Blackbox.
    • The health probes are https on http_2xx_lab_ca, so TlsCertificateExpiringSoon now covers the new leaf.
    • The refusal probes are https too.
    • Rule-test fixtures are updated, and promtool passes.
  • ADR-0086 records the decision, and ADR-0067 gets a forward note.
  • Docs. SECURITY.md, security.md, architecture.md, network.md, observability.md, generate-certificates.md, the stack README and the changelog no longer say plain HTTP. The README's ADR count is now 80.

Measured

On the pinned Caddy image (2.11.7, re-run after main's bump from 2.11.4), run the way ingest-proxy runs (uid 65534 plus the operator's group, read-only root, the same tmpfs mounts), with the real leaf and stub upstreams. A curl client verified against the committed CA through the 10.0.99.20 IP SAN:

Request Result
/-/healthy and /ready 200, verified
query, no token 401 Bearer realm="homelab-ingest"
query, reader token 200
Loki delete, no token 401
plain http:// 400
https without the CA refused (verify 20)
wrong hostname refused (verify 1)

Cutover (after merge)

  1. Run make converge in the main checkout on prometheus straight away, rather than waiting for :29.
  2. Run make up on trinity.
  3. Run deploy-agent.sh for oracle and trinity, and Saruman's from the Mac, all straight away. On the Mac, export INGEST_TOKEN and INGEST_TOKEN_READER first; both are in Vaultwarden. Saruman's arrival check is skipped from the Mac, so I'll run it from prometheus.

Until each agent is redeployed, the proxy refuses it:

  • Metrics: the remote-write WAL holds them, so nothing is lost.
  • Logs: loki.write drops them after its retries, so a few minutes of logs can be lost. LogEntriesDropped will say how many.

Then confirm:

  • probe_success is 1 for all four ingest targets;
  • probe_ssl_earliest_cert_expiry{name=~"prometheus|loki"} is present;
  • Homepage's widgets and the HA sensor show values;
  • curl http://10.0.99.20:9090/ fails.

🤖 Generated with Claude Code

Base automatically changed from gerrrt/ingest-ports-tls-642bf2 to main October 7, 2026 14:05
Gerrrt added a commit that referenced this pull request Oct 7, 2026
…082s (#1015)

* docs(adr): renumber the lab host-keys ADR to 0085; main had two ADR-0082s

#962 (the wiki over TLS) and #931 (lab credential hygiene) each landed
an ADR-0082, minutes apart, so check_docs fails on main ("ADR-0082 is
claimed by 2 files … renumber the one that landed second") and the
README's count is one short. #931 landed second, so its ADR moves.
0083 and 0084 are claimed by open PRs (#928, #959, #960), so it takes
0085. Every reference to it moves with it; the wiki's ADR-0082
references are untouched. README: 83 ADRs.

Refs #846

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* docs(adr): record the renumber in ADR-0085, as ADR-0049 records its own

The convention from ADR-0049: a renumbered ADR opens with a note naming
the number it landed as, the PR that took that number first and when,
the rule that chose it, and that the decision is untouched.

Refs #846

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@Gerrrt
Gerrrt force-pushed the gerrrt/ingest-ports-tls-flip branch from 0da2939 to fbd8552 Compare October 7, 2026 15:18
…https (#764)

The second of two steps; the first landed every client's trust in the
estate CA while their URLs stayed http.

- Caddyfile: `tls cert key` with the prometheus.matrix.elysium leaf on
  both published sites; :2020 stays plain. check_caddyfile.sh mounts a
  throwaway pair, since validate loads the files.
- deploy-agent.sh: https URLs and `curl --cacert` on the committed CA
  for the estate; the lab's proxy stays http.
- Homepage's five widgets and Home Assistant's rack-power sensor on
  https; blackbox health probes to http_2xx_lab_ca, refusal probes to
  https, rule-test fixtures to match.
- ADR-0083, a forward note on ADR-0067, and the docs whose plain-HTTP
  claims this ends.

Closes #764

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Gerrrt
Gerrrt force-pushed the gerrrt/ingest-ports-tls-flip branch from fbd8552 to f1ed9c4 Compare October 7, 2026 17:20
@Gerrrt
Gerrrt marked this pull request as ready for review October 7, 2026 17:28
Copilot AI balanced review requested due to automatic review settings October 7, 2026 17:29
@Gerrrt
Gerrrt merged commit c1efb8d into main Oct 7, 2026
9 of 10 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/ingest-ports-tls-flip branch October 7, 2026 17:34

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

The coordinated production cutover needs corrected rollout guidance and human verification across all clients.

4 open findings
What changed in this PR

Completes the estate ingest TLS migration after #932 established client trust, while leaving lab and internal HTTP endpoints unchanged.

Changes:

  • Enables TLS on ingest ports 9090 and 3100.
  • Moves agents, query clients and blackbox probes to verified HTTPS.
  • Updates alert fixtures, security documentation and the rollout decision.
File Description
stacks/​sensitive/​homepage/​services.yaml Switches five widgets to HTTPS.
stacks/​sensitive/​home-assistant/​packages/​homelab_rack.yaml Switches the rack-power query to HTTPS.
stacks/​sensitive/​compose.yaml Updates client trust comments only.
stacks/​observability/​README.md Documents TLS on published ingest ports.
stacks/​observability/​prometheus/​tests/​stack.test.yaml Updates remote-write URL fixtures.
stacks/​observability/​prometheus/​tests/​blackbox.test.yaml Updates HTTPS probe fixtures.
stacks/​observability/​prometheus/​targets/​blackbox.yaml Enables verified HTTPS probes.
stacks/​observability/​Caddyfile Loads the certificate on both ingest listeners.
stacks/​observability/​blackbox/​blackbox.yaml Updates probe-module comments only.
stacks/​observability/​alloy/​config.alloy Updates example URLs in comments only.
SECURITY.md Revises the estate’s cleartext-token residual.
scripts/​deploy-agent.sh Configures HTTPS endpoints and CA-verified arrival checks.
scripts/​check_caddyfile.sh Updates certificate-validation comments only.
README.md Increments the ADR count.
docs/​security.md Distinguishes estate TLS from lab HTTP.
docs/​runbooks/​generate-certificates.md Updates TLS consumers and trust distribution.
docs/​observability.md Updates agent onboarding guidance.
docs/​network.md Documents encrypted ingest traffic.
docs/​changelog.md Records the TLS migration.
docs/​architecture.md Updates published-port security descriptions.
docs/​adr/​0086-serve-the-ingest-ports-over-tls-under-the-estate-ca.md Records TLS design and cutover consequences.
docs/​adr/​0067-authenticate-the-ingest-ports-with-a-token-per-client.md Adds a forward note to the TLS decision.

🧠 Review effort: Balanced


💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +92 to +94
The agents on `oracle` and `Saruman` change only when `deploy-agent.sh` runs.
So they are redeployed straight after the monitoring host applies the flip,
and until then the proxy refuses them.
Comment on lines +100 to +102
- **A short gap at cutover is accepted.** For metrics, the agent's
remote-write WAL holds them until it is redeployed. For logs, `loki.write`
retries and then drops, so a few minutes of `oracle`'s and `Saruman`'s logs
- **A short gap at cutover is accepted.** For metrics, the agent's
remote-write WAL holds them until it is redeployed. For logs, `loki.write`
retries and then drops, so a few minutes of `oracle`'s and `Saruman`'s logs
can be lost, and `LogEntriesDropped` says how many. To keep the gap short,

rest:
- resource: http://10.0.99.20:9090/api/v1/query
- resource: https://10.0.99.20:9090/api/v1/query
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Serve the ingest ports over TLS

2 participants