Skip to content

The lab Prometheus and Loki take unauthenticated writes, /-/quit and log deletion from the whole of VLAN 30 #834

Closed

Description

@Gerrrt

What is true

The lab's Prometheus and Loki on alexander are published to the whole of VLAN 30, unauthenticated. VLAN 30 is the segment ADR-0007 exists to hold attackers on, and #449 is about to make deliberately weaker.

  • stacks/lab/compose.yaml:120-121 sets --web.enable-remote-write-receiver and --web.enable-lifecycle.
  • stacks/lab/compose.yaml:155 publishes "${BIND_ADDR:-0.0.0.0}:${PROMETHEUS_PORT:-9090}:9090". :197 does the same for Loki on 3100. stacks/lab/.env.example:13 sets BIND_ADDR=0.0.0.0.
  • stacks/lab/loki/loki-config.yaml:21 has auth_enabled: false, and :58-59 has retention_enabled: true / delete_request_store: filesystem. That combination turns Loki's delete API on.

Any host on VLAN 30 can therefore:

  • POST /-/quit or /-/reload to the lab Prometheus;
  • forge series through remote-write;
  • submit Loki delete requests for log ranges from odin, phoenix and fenrir.

That is the evidence store for the segment's incidents. The compose comment at :134-140 names the risk ("an unauthenticated remote-write receiver on the segment that exists to hold attackers … also a delete-series API pointed at the lab's own evidence"). ADR-0030 records that the ports are published. Nothing records the lack of authentication as an accepted residual.

Why this is the estate's #182, again

SECURITY.md lists exactly this risk for the estate's ingest ports. #182 and ADR-0067 fixed it on 10.0.99.20 with a Caddy proxy and a bearer token per client. The lab never got the same treatment, and the lab is the segment where it matters most.

What closes it

Do in the lab what ADR-0067 did for the estate:

  • Bind 9090 and 3100 to 127.0.0.1, or to the compose network only.
  • Add a caddy ingest proxy to stacks/lab. Reuse the (ingest_auth) and (refuse) snippets from stacks/observability/Caddyfile, with one token per client (odin, phoenix, fenrir) from secrets/lab.sops.yaml.
  • Drop --web.enable-lifecycle. If a reload path is needed, keep it reachable only from inside the container network.
  • Remove Loki's delete path from what the proxy forwards, or turn off delete_request_store, if nothing in the lab uses deletes.
  • Add a lab copy of IngestAuthNotEnforced (a probe that fails if an unauthenticated write is accepted). It is non-paging per ADR-0020, but visible.
  • Ship the tokens to the three agents' config.alloy, through the same deploy-agent.sh path the estate uses.

Rollout order: clients first, then the proxy refuses unauthenticated requests. That is the same order ADR-0067's runbook used.

Done when

Refs #182, #764, #449. Found in the 2026-10-03 repository review.

Activity

  1. added
    bugSomething isn't working
    securityA credential, an exposure, or a control that is weaker than recorded
    labstacks/lab on alexander (VLAN 30)
    on Oct 3, 2026
  2. Gerrrt commented on Oct 4, 2026

    @Gerrrt
    OwnerAuthor

    Done, and deployed on 2026-10-04. Checked against Done when:

    • From a VLAN 30 address other than the proxy, /-/quit and an unauthenticated Loki push are refused. From fenrir, POST http://10.0.30.40:9090/-/quit and POST http://10.0.30.40:3100/loki/api/v1/push both returned 401. On alexander, POST /loki/api/v1/delete and POST /api/v1/write returned 401 as well. Caddy's access log, which records only refusals, showed those four and nothing else.
    • The agents still deliver. After the switch, count by (instance) (timestamp(up{instance=~"odin|fenrir|phoenix|golem"}) > time() - 120) on alexander returned all four (odin 5, fenrir 3, phoenix 2, golem 2), each pushing through the proxy with its own token. That's one more client than the issue named: golem, added by feat(lab): give golem its own token at the ingest proxy (#485) #896.

    How it shipped:

    Rolled out clients first, per stacks/lab/README.md: odin, fenrir, phoenix and golem, then make up STACK=lab on alexander.

    The script's false alarm on golem turned out to be golem being idle: its logs were arriving, which a hand-sent logger line proved. deploy-agent.sh's check is fixed in #908.

    🤖 Posted by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinglabstacks/lab on alexander (VLAN 30)securityA credential, an exposure, or a control that is weaker than recorded

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions