Repository navigation
feat(ingest): serve 9090 and 3100 over TLS, and move every client to https (#764) - #959
Merged
Merged
Conversation
Gerrrt
added a commit
that referenced
this pull request
Oct 7, 2026
…082s (#1015) * docs(adr): renumber the lab host-keys ADR to 0085; main had two ADR-0082s #962 (the wiki over TLS) and #931 (lab credential hygiene) each landed an ADR-0082, minutes apart, so check_docs fails on main ("ADR-0082 is claimed by 2 files … renumber the one that landed second") and the README's count is one short. #931 landed second, so its ADR moves. 0083 and 0084 are claimed by open PRs (#928, #959, #960), so it takes 0085. Every reference to it moves with it; the wiki's ADR-0082 references are untouched. README: 83 ADRs. Refs #846 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs(adr): record the renumber in ADR-0085, as ADR-0049 records its own The convention from ADR-0049: a renumbered ADR opens with a note naming the number it landed as, the PR that took that number first and when, the rule that chose it, and that the decision is untouched. Refs #846 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Gerrrt
force-pushed
the
gerrrt/ingest-ports-tls-flip
branch
from
October 7, 2026 15:18
0da2939 to
fbd8552
Compare
…https (#764) The second of two steps; the first landed every client's trust in the estate CA while their URLs stayed http. - Caddyfile: `tls cert key` with the prometheus.matrix.elysium leaf on both published sites; :2020 stays plain. check_caddyfile.sh mounts a throwaway pair, since validate loads the files. - deploy-agent.sh: https URLs and `curl --cacert` on the committed CA for the estate; the lab's proxy stays http. - Homepage's five widgets and Home Assistant's rack-power sensor on https; blackbox health probes to http_2xx_lab_ca, refusal probes to https, rule-test fixtures to match. - ADR-0083, a forward note on ADR-0067, and the docs whose plain-HTTP claims this ends. Closes #764 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Gerrrt
force-pushed
the
gerrrt/ingest-ports-tls-flip
branch
from
October 7, 2026 17:20
fbd8552 to
f1ed9c4
Compare
Gerrrt
marked this pull request as ready for review
October 7, 2026 17:28
There was a problem hiding this comment.
🔵 Needs a closer look
The coordinated production cutover needs corrected rollout guidance and human verification across all clients.
4 open findings
What changed in this PR
Completes the estate ingest TLS migration after #932 established client trust, while leaving lab and internal HTTP endpoints unchanged.
Changes:
- Enables TLS on ingest ports 9090 and 3100.
- Moves agents, query clients and blackbox probes to verified HTTPS.
- Updates alert fixtures, security documentation and the rollout decision.
| File | Description |
|---|---|
stacks/sensitive/homepage/services.yaml |
Switches five widgets to HTTPS. |
stacks/sensitive/home-assistant/packages/homelab_rack.yaml |
Switches the rack-power query to HTTPS. |
stacks/sensitive/compose.yaml |
Updates client trust comments only. |
stacks/observability/README.md |
Documents TLS on published ingest ports. |
stacks/observability/prometheus/tests/stack.test.yaml |
Updates remote-write URL fixtures. |
stacks/observability/prometheus/tests/blackbox.test.yaml |
Updates HTTPS probe fixtures. |
stacks/observability/prometheus/targets/blackbox.yaml |
Enables verified HTTPS probes. |
stacks/observability/Caddyfile |
Loads the certificate on both ingest listeners. |
stacks/observability/blackbox/blackbox.yaml |
Updates probe-module comments only. |
stacks/observability/alloy/config.alloy |
Updates example URLs in comments only. |
SECURITY.md |
Revises the estate’s cleartext-token residual. |
scripts/deploy-agent.sh |
Configures HTTPS endpoints and CA-verified arrival checks. |
scripts/check_caddyfile.sh |
Updates certificate-validation comments only. |
README.md |
Increments the ADR count. |
docs/security.md |
Distinguishes estate TLS from lab HTTP. |
docs/runbooks/generate-certificates.md |
Updates TLS consumers and trust distribution. |
docs/observability.md |
Updates agent onboarding guidance. |
docs/network.md |
Documents encrypted ingest traffic. |
docs/changelog.md |
Records the TLS migration. |
docs/architecture.md |
Updates published-port security descriptions. |
docs/adr/0086-serve-the-ingest-ports-over-tls-under-the-estate-ca.md |
Records TLS design and cutover consequences. |
docs/adr/0067-authenticate-the-ingest-ports-with-a-token-per-client.md |
Adds a forward note to the TLS decision. |
🧠 Review effort: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+92
to
+94
| The agents on `oracle` and `Saruman` change only when `deploy-agent.sh` runs. | ||
| So they are redeployed straight after the monitoring host applies the flip, | ||
| and until then the proxy refuses them. |
Comment on lines
+100
to
+102
| - **A short gap at cutover is accepted.** For metrics, the agent's | ||
| remote-write WAL holds them until it is redeployed. For logs, `loki.write` | ||
| retries and then drops, so a few minutes of `oracle`'s and `Saruman`'s logs |
| - **A short gap at cutover is accepted.** For metrics, the agent's | ||
| remote-write WAL holds them until it is redeployed. For logs, `loki.write` | ||
| retries and then drops, so a few minutes of `oracle`'s and `Saruman`'s logs | ||
| can be lost, and `LogEntriesDropped` says how many. To keep the gap short, |
|
|
||
| rest: | ||
| - resource: http://10.0.99.20:9090/api/v1/query | ||
| - resource: https://10.0.99.20:9090/api/v1/query |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Closes #764. This is the second of two PRs. The first, #932, gave every client trust in the estate CA while the URLs stayed http.
Prerequisites, done 2026-10-07
Every client already trusts the estate CA, so this PR only flips the scheme.
prometheus.ingest-proxymounts theprometheus.matrix.elysiumleaf. It can read the 0640 key, and the leaf verifies against the committed CA.ca_fileonhttp_401_ingest_refused.REQUESTS_CA_BUNDLEandNODE_EXTRA_CA_CERTS. Both are healthy, and each bundle contains the estate CA, checked by fingerprint./etc/alloy/ingest-ca.pemwithINGEST_CA_FILEset.oracleandtrinitywere redeployed withdeploy-agent.sh, and both passed its arrival check.Sarumanwas redeployed from the Mac. Its arrival check was asked fromprometheus, since Hicks can't reach 9090: samples about 15 s old, a Loki line from that second, and no drops or refusals.The branch is rebuilt on main and holds this one commit. Its ADR is 0086, because #928, #960 and the host-keys renumber took 0083–0085.
What changes
tls /etc/caddy/tls/cert.pem /etc/caddy/tls/key.pemon the:9090and:3100sites, through aningest_tlssnippet.:2020stays plain.auto_https offstays too, because it controls certificate management and redirects, not TLS.check_caddyfile.shmounts a throwaway certificate and key at those paths, becausecaddy validateloads them. Without the mounts it fails, which I checked.deploy-agent.sh. For the estate it useshttps://URLs and passes--cacertwith the committedingest-ca.pemto every arrival-check curl. The lab (--monitoring-host 10.0.30.40) stays on http.restresource now usehttps://. Both verify through the bundles feat(ingest): every client can verify the estate CA, ahead of TLS on 9090 and 3100 (#764) #932 added.http_2xx_lab_ca, soTlsCertificateExpiringSoonnow covers the new leaf.SECURITY.md,security.md,architecture.md,network.md,observability.md,generate-certificates.md, the stack README and the changelog no longer say plain HTTP. The README's ADR count is now 80.Measured
On the pinned Caddy image (2.11.7, re-run after main's bump from 2.11.4), run the way
ingest-proxyruns (uid 65534 plus the operator's group, read-only root, the same tmpfs mounts), with the real leaf and stub upstreams. A curl client verified against the committed CA through the10.0.99.20IP SAN:/-/healthyand/readyBearer realm="homelab-ingest"http://Cutover (after merge)
make convergein the main checkout onprometheusstraight away, rather than waiting for :29.make upon trinity.deploy-agent.shfor oracle and trinity, and Saruman's from the Mac, all straight away. On the Mac, exportINGEST_TOKENandINGEST_TOKEN_READERfirst; both are in Vaultwarden. Saruman's arrival check is skipped from the Mac, so I'll run it fromprometheus.Until each agent is redeployed, the proxy refuses it:
loki.writedrops them after its retries, so a few minutes of logs can be lost.LogEntriesDroppedwill say how many.Then confirm:
probe_successis 1 for all four ingest targets;probe_ssl_earliest_cert_expiry{name=~"prometheus|loki"}is present;curl http://10.0.99.20:9090/fails.🤖 Generated with Claude Code