Repository navigation
Description
What is true
The lab's Prometheus and Loki on alexander are published to the whole of VLAN 30, unauthenticated. VLAN 30 is the segment ADR-0007 exists to hold attackers on, and #449 is about to make deliberately weaker.
stacks/lab/compose.yaml:120-121sets--web.enable-remote-write-receiverand--web.enable-lifecycle.stacks/lab/compose.yaml:155publishes"${BIND_ADDR:-0.0.0.0}:${PROMETHEUS_PORT:-9090}:9090".:197does the same for Loki on 3100.stacks/lab/.env.example:13setsBIND_ADDR=0.0.0.0.stacks/lab/loki/loki-config.yaml:21hasauth_enabled: false, and:58-59hasretention_enabled: true/delete_request_store: filesystem. That combination turns Loki's delete API on.
Any host on VLAN 30 can therefore:
POST /-/quitor/-/reloadto the lab Prometheus;- forge series through remote-write;
- submit Loki delete requests for log ranges from odin, phoenix and fenrir.
That is the evidence store for the segment's incidents. The compose comment at :134-140 names the risk ("an unauthenticated remote-write receiver on the segment that exists to hold attackers … also a delete-series API pointed at the lab's own evidence"). ADR-0030 records that the ports are published. Nothing records the lack of authentication as an accepted residual.
Why this is the estate's #182, again
SECURITY.md lists exactly this risk for the estate's ingest ports. #182 and ADR-0067 fixed it on 10.0.99.20 with a Caddy proxy and a bearer token per client. The lab never got the same treatment, and the lab is the segment where it matters most.
What closes it
Do in the lab what ADR-0067 did for the estate:
- Bind 9090 and 3100 to
127.0.0.1, or to the compose network only. - Add a
caddyingest proxy tostacks/lab. Reuse the(ingest_auth)and(refuse)snippets fromstacks/observability/Caddyfile, with one token per client (odin, phoenix, fenrir) fromsecrets/lab.sops.yaml. - Drop
--web.enable-lifecycle. If a reload path is needed, keep it reachable only from inside the container network. - Remove Loki's delete path from what the proxy forwards, or turn off
delete_request_store, if nothing in the lab uses deletes. - Add a lab copy of
IngestAuthNotEnforced(a probe that fails if an unauthenticated write is accepted). It is non-paging per ADR-0020, but visible. - Ship the tokens to the three agents'
config.alloy, through the samedeploy-agent.shpath the estate uses.
Rollout order: clients first, then the proxy refuses unauthenticated requests. That is the same order ADR-0067's runbook used.
Done when
- From any VLAN 30 address other than the proxy,
curl -X POST http://10.0.30.x:9090/-/quitand an unauthenticatedPOST /loki/api/v1/pushare refused. - The three agents still deliver.
- Ideally this lands before Give the lab domain users and deliberate weaknesses, one tag at a time #449 ships its weaknesses.
Refs #182, #764, #449. Found in the 2026-10-03 repository review.
Activity
docs(lab): bring every description in line with the proxy, and stop c…
fix(lab): put the lab's Prometheus and Loki behind a token-per-client…
docs: correct reachability, collection, secret and proxy claims from …
feat(lab): give golem its own token at the ingest proxy (#485) (#896)
feat(sensor): fenrir's own age key and its ingest token (#834) (#905)
feat(lab): the four ingest tokens the proxy checks (#834) (#907)
Gerrrt commentedon Oct 4, 2026
Done, and deployed on 2026-10-04. Checked against Done when:
/-/quitand an unauthenticated Loki push are refused. From fenrir,POST http://10.0.30.40:9090/-/quitandPOST http://10.0.30.40:3100/loki/api/v1/pushboth returned401. On alexander,POST /loki/api/v1/deleteandPOST /api/v1/writereturned401as well. Caddy's access log, which records only refusals, showed those four and nothing else.count by (instance) (timestamp(up{instance=~"odin|fenrir|phoenix|golem"}) > time() - 120)on alexander returned all four (odin 5, fenrir 3, phoenix 2, golem 2), each pushing through the proxy with its own token. That's one more client than the issue named: golem, added by feat(lab): give golem its own token at the ingest proxy (#485) #896.How it shipped:
Rolled out clients first, per
stacks/lab/README.md: odin, fenrir, phoenix and golem, thenmake up STACK=labon alexander.The script's false alarm on golem turned out to be golem being idle: its logs were arriving, which a hand-sent
loggerline proved.deploy-agent.sh's check is fixed in #908.🤖 Posted by Claude Code
fix(secrets): odin's ingest token, which the soc stack requires, is i…
docs(sensor): the compose header says make up, and that fenrir is built
docs(sensor): the compose header says make up, and that fenrir is bui…