Skip to content

Lab domain credential hygiene: a blank-password guard, WinRM scope, a token off the command line, host keys #846

Closed

Description

@Gerrrt

What is true

Four small gaps in how the lab domain's build and configuration handle credentials. None is severe alone. Together they are the weak end of a domain that #449 is about to make deliberately weak on top.

1. A blank Administrator password is accepted without a word.

  • ansible/inventory/group_vars/all.yaml:29-30: lab_admin_password: "{{ lookup('ansible.builtin.env', 'LAB_ADMIN_PASSWORD') }}". If phoenix.env was not sourced, that is "".
  • roles/base/tasks/main.yml:118-125 then runs win_user … password: "{{ lab_admin_password }}" update_password: always under no_log: true, so any error is hidden as well.
  • Packer has the guard Ansible lacks (packer/variables.pkr.hcl:96-109).

2. WinRM is open to the whole segment, unencrypted, during Packer Windows builds.

  • packer/windows/scripts/bootstrap.ps1:32-35 enables Basic auth with AllowUnencrypted and opens TCP 5985 with no -RemoteAddress.
  • windows.pkr.hcl:148-149 sets winrm_insecure = true and winrm_use_ssl = false.
  • The build password crosses VLAN 30 as cleartext Basic auth, readable by any guest there. ADR-0077's own reason for not using WinRM is that "NTLM over HTTP is one of the protocols this segment exists to have attacked."

3. packer-smoke.sh puts the Proxmox token on a command line.

  • scripts/packer-smoke.sh:67,77 builds AUTH="Authorization: PVEAPIToken=…=${PROXMOX_TOKEN_SECRET}" and passes it as curl -H "${AUTH}". It is visible in ps and /proc/*/cmdline on phoenix.
  • scripts/deploy-agent.sh:484-486 already does this properly: printf 'header = …' | curl -K -.

4. Host keys are unchecked while domain-admin passwords cross SSH.

  • ansible/ansible.cfg:16 has host_key_checking = false.
  • ADR-0077 §2 accepts this because every rebuild replaces the keys. That reasoning does not weigh that LAB_ADMIN_PASSWORD and LAB_DSRM_PASSWORD then cross a channel an ARP spoofer can sit in the middle of. docs/security.md:773 acknowledges ARP spoofing on this segment.

What closes it

  1. A pre_tasks ansible.builtin.assert at the top of lab-domain.yml: both variables are defined and | length >= 14, with a message naming phoenix.env.
  2. -RemoteAddress 10.0.30.70 (phoenix) on the WinRM firewall rule in bootstrap.ps1, the same restriction openssh.ps1 uses. Better still: an HTTPS listener with a self-signed certificate created in bootstrap.ps1, and winrm_use_ssl = true.
  3. Switch packer-smoke.sh to the curl -K - pattern.
  4. Optional, as an amendment to ADR-0077 §2: after tofu apply, read each clone's host key through the guest agent (qm guest exec … cat /etc/ssh/ssh_host_ed25519_key.pub, or the Windows equivalent). The token already has VM.GuestAgent.*. Write a per-run known_hosts, then turn host key checking on.

Done when

  • Running lab-domain.yml without phoenix.env fails in the first task.
  • A Windows build's WinRM listener refuses connections from any address but phoenix.
  • ps on phoenix during a smoke run shows no token.

Refs ADR-0077, #448, #449. Found in the 2026-10-03 repository review.

Activity

  1. added this to the automation milestone on Oct 3, 2026
  2. added
    bugSomething isn't working
    securityA credential, an exposure, or a control that is weaker than recorded
    labstacks/lab on alexander (VLAN 30)
    on Oct 3, 2026
  3. Gerrrt commented on Oct 7, 2026

    @Gerrrt
    OwnerAuthor

    Closed by #931 (merged 2026-10-07 13:48 UTC as be31da3), with a follow-up in #1018. All three "done when" conditions were proven on the live lab on 2026-10-07:

    • Running lab-domain.yml without phoenix.env fails at the first task. The guard is a play on lab_domain with any_errors_fatal, tagged always. With the env unset it fails before any role runs, with and without --limit titan. With it set, the following plays run on all six. Its output shows the expression, never a value.
    • A Windows build's WinRM refuses connections from any address but phoenix. During the 912 build, 12 s after Packer connected to WinRM (13:50:56 UTC), ports 5985 and 5986 on the build VM timed out from odin (10.0.30.60) and were refused/filtered from Saruman (10.0.30.110). Meanwhile Packer, from phoenix, provisioned over HTTPS on 5986. The template built in 36m48s and passed packer-smoke.sh 912.
    • ps on phoenix during a smoke run shows no token. A root sampler ran ps -eo args | grep -c PVEAPIToken once a second for the whole 912 smoke run: 299 samples, 0 hits.

    Host keys, item 4 (ADR-0083, renumbered from 0082 in #1018):

    • scripts/lab-known-hosts.sh pinned all six on phoenix.
    • verify.yml passed on all six with host-key checking on.
    • Earlier, a key with one character changed was refused with REMOTE HOST IDENTIFICATION HAS CHANGED.

    Found along the way and fixed:

    Not yet checked: SetupComplete.log on a clone of the new templates showing the HTTPS listener, rule and certificate removed. The 912 smoke clone was destroyed before it could be read. The 911 smoke run will use --keep for it.

    🤖 Generated with Claude Code

  4. Gerrrt commented on Oct 7, 2026

    @Gerrrt
    OwnerAuthor

    Reopening: I closed this before every check had been done, and the one I'd left open fails.

    SetupComplete.log on a smoke clone of the new 911 (2026-10-07 15:24 UTC) shows the certificate step failing with rc=1:

    Remove-Item : A parameter cannot be found that matches parameter name 'DeleteKey'.
    

    -DeleteKey was added in #931 from review and never ran on Windows before merging. In Windows PowerShell 5.1 it doesn't work on certificates sent down a pipeline, so the packer-winrm certificate stays on every clone of 911 and 912. Everything else held on the clone: WinRM is stopped and disabled, the HTTPS listener and the packer-winrm-https rule are gone, and no inbound rule is open on 5985 or 5986. The leftover is a self-signed certificate that nothing trusts, the same on every clone. I couldn't confirm whether its private key is still on disk: removing it by path deleted the certificate but reported "Keyset does not exist".

    The fix (PR to follow):

    • delete the certificate and its key in the task that starts sysprep, so the template never contains them;
    • keep a corrected removal on the clone as a backstop;
    • drop the AllowUnencrypted reset, which the build never turns on and which errors on the clone's Public network.

    It's proven by a 912 rebuild and a smoke run with --keep. I'll close this again once that passes.

    🤖 Generated with Claude Code

  5. reopened this on Oct 7, 2026
  6. Gerrrt commented on Oct 7, 2026

    @Gerrrt
    OwnerAuthor

    Closing again: the check that failed now passes.

    The certificate cleanup is fixed by #1031, which is merged. That PR deletes the build's WinRM private key, then its certificate, in the sysprep task before generalising. The clone runs the same script again as a backstop, and a certificate is never removed while its key can't be. Both templates were rebuilt from it on 2026-10-07 and checked on a --keep smoke clone:

    Template Key deleted before sysprep Pre-sysprep run Clone-side run Backstop rc Left on the clone Smoke test, ps
    912, Server 2025 CNG f23a6ba4…, 18:28:15 left: 0 left: 0 0 nothing: no certificate, no rule, WinRM disabled passed, 0/277
    911, Windows 11 CNG 21aea7e1…, 19:50:52 left: 0 left: 0 0 nothing: no certificate, no rule, WinRM disabled passed, 0/348

    The other conditions were proven earlier and stand: the blank-password guard, WinRM scoped to phoenix during a build, no token on argv, and pinned host keys.

    Still open: the six domain guests rebuilt from 15:38 UTC today were cloned from the templates as they were before #1031. Each still carries that build's packer-winrm certificate, and probably its key. WinRM is disabled and closed on all six, so that is a leftover, not an opening. Removing it from them is follow-up work; it isn't reopened here.

    🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workinglabstacks/lab on alexander (VLAN 30)securityA credential, an exposure, or a control that is weaker than recorded

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions