Skip to content

feat(tofu): provision lab guests with OpenTofu, state encrypted before the first apply (#445) - #803

Merged
Gerrrt merged 4 commits into
mainfrom
gerrrt/lab-vm-opentofu-provision-825058
Oct 2, 2026
Merged

Gerrrt merged 4 commits into
mainfrom
gerrrt/lab-vm-opentofu-provision-825058

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Oct 2, 2026 •

Copy link
Copy Markdown
Owner

Refs #445. This lands the code and every guard. The first apply and its two on-host proofs wait for template 901 (#440), so this does not close the issue.

What

  • tofu/ (ADR-0076) uses bpg/proxmox, pinned exactly.
    • State encryption: the encryption config sets enforced = true on both state and plan, so a plaintext write is refused rather than made.
    • Clones: full clones only, hardcoded in the module.
    • Pools: derived from the guests, so a pool is created and destroyed with them.
    • The one guest: the only guest is proof guest 998, declared only under -var proof=true.
    • The six domain guests (150–155) stay hand-managed until Configure with Ansible, and build ADR-0029's six guests from the pipeline rather than by hand #448.
  • The state key is a pbkdf2 passphrase in phoenix.env.
    • Escrow: a copy goes in secrets/tofu.sops.yaml, encrypted to the estate's two recipients. phoenix can write that copy and cannot read it.
    • Why not age: OpenTofu has no age key provider, and ADR-0043 and ADR-0074 keep age keys off phoenix.
  • The guards the issue asks for, before the first apply:
    • .gitignore covers state, plans and .terraform/.
    • check-tracked-artefacts.sh asserts the same patterns are untracked, and gains its first --self-test (14 fixtures).
    • .gitleaks.toml gains tfstate-plaintext. It matches plaintext state by content under any name and stays silent on OpenTofu's encrypted wrapper.
    • docs/security.md records phoenix as a secret-bearing host.
  • scripts/check-tofu-state-encryption.sh proves the encryption rather than assuming it.
    • In CI, with --self-test (10 fixtures): it copies the real tofu/encryption.tf next to a terraform_data canary. It then proves:
      • the canary is absent from the state;
      • the plan is ciphertext;
      • a wrong passphrase cannot read the state;
      • enforced refuses an unencrypted method.
      • It also proves the inverse: without the file, the canary is present and the gitleaks rule matches.
      • git add of the state is refused, and git add -f fails check-tracked-artefacts.sh.
    • On phoenix: the same script checks the real state, with the secret piped in on stdin.
  • CI: lint.sh runs tofu fmt -check, then init -backend=false -lockfile=readonly, then validate. It uses ghcr.io/opentofu/opentofu, pinned in stacks/observability/compose.yaml.
  • Docs: a new runbook, docs/runbooks/provision-lab-guests.md, plus updates to the jumpbox runbook, roadmap, changelog and README counts.

Proved locally

All checks below ran with tofu 1.13.1, and all passed:

  • check-tofu-state-encryption.sh --self-test (10 fixtures)
  • check-tracked-artefacts.sh --self-test (14 fixtures)
  • the tofu runs in lint.sh, plus shellcheck and yamllint
  • check_docs.py and check_sops_rules.py
  • The new gitleaks regex matches nothing in the tree or its history.

Mutation checks, run on copies:

  • Removing enforced = true turns the encryption suite red.
  • Removing the state block turns it red.
  • Removing the .gitignore lines turns it red.
  • Removing the new patterns from check-tracked-artefacts.sh turns its suite red.

Not run locally: there is no Docker on this machine, so the image path of both tofu checks is first exercised by this PR's CI. check_mounted_config.py --self-test fails locally for the same reason, and that is unrelated to this change.

Still open (runbook §2–§4, on phoenix, after #440)

  1. Grant Pool.Allocate on /pool/proof. Whether Proxmox accepts that ACL before the pool exists is unverified.
  2. Escrow the passphrase and compare its hash on prometheus.
  3. Apply proof guest 998. Its password must not occur in the real state, git add must refuse the state, and git add -f must turn check-tracked-artefacts.sh red.
  4. Destroy the guest and its pool, and record the run in §6. Then Provision lab VMs with OpenTofu, and encrypt the state before the first apply #445 can close.

🤖 Generated with Claude Code

…e the first apply (#445)

OpenTofu over Terraform because it encrypts state (ADR-0075). The key is a
pbkdf2 passphrase in phoenix.env, escrowed in secrets/tofu.sops.yaml to the
estate's two recipients: OpenTofu has no age key provider and ADR-0043 keeps
age keys off phoenix. enforced = true on state and plan.

Guards land before any apply: .gitignore, check-tracked-artefacts.sh patterns
and its first --self-test, a tfstate-plaintext gitleaks rule, and phoenix
recorded in docs/security.md as secret-bearing.
check-tofu-state-encryption.sh --self-test proves the real encryption.tf
against a canary, and that the checks fail without it.

The first apply (proof guest 998) and its on-host proofs wait for template
901 (#440). The six hand-built domain guests stay out until #448.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings October 2, 2026 04:37

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Pool teardown removes its ACL and the encryption test does not independently verify both enforcement guards.

Review effort: Balanced
Findings: 1 High severity · 1 Medium severity · 1 Low severity

Open (3)
What changed in this PR

Adds OpenTofu-based Proxmox guest provisioning with encrypted state, validation guards, and operational documentation.

Changes:

  • Adds a reusable full-clone guest module and proof guest.
  • Enforces and tests state/plan encryption.
  • Adds repository safeguards, CI integration, ADR, and runbooks.
File Description
tofu/​versions.tf Pins OpenTofu and providers.
tofu/​variables.tf Defines proof and SSH-key inputs.
tofu/​README.md Documents the OpenTofu tree.
tofu/​providers.tf Configures Proxmox access.
tofu/​modules/​guest/​variables.tf Defines guest module inputs.
tofu/​modules/​guest/​outputs.tf Exposes guest identifiers and addresses.
tofu/​modules/​guest/​main.tf Implements full-clone guests.
tofu/​guests.tf Declares proof guest and managed pools.
tofu/​encryption.tf Enforces state and plan encryption.
tofu/​.terraform.lock.hcl Locks provider artifacts.
stacks/​observability/​compose.yaml Pins the OpenTofu CI image.
secrets/​tofu.example.yaml Defines the escrow secret shape.
secrets/​README.md Documents passphrase escrow.
scripts/​lint.sh Adds OpenTofu formatting and validation.
scripts/​check-tracked-artefacts.sh Rejects tracked OpenTofu artifacts.
scripts/​check-tofu-state-encryption.sh Tests encryption safeguards.
scripts/​check_container_health.py Excludes profiled OpenTofu service.
README.md Updates documentation counts.
docs/​security.md Records phoenix secrets and controls.
docs/​runbooks/​provision-lab-guests.md Adds provisioning and recovery procedures.
docs/​runbooks/​build-the-jumpbox.md Extends phoenix credential setup.
docs/​roadmap.md Records OpenTofu implementation status.
docs/​changelog.md Adds the dated implementation record.
docs/​adr/​0075-provision-lab-guests-with-opentofu-and-encrypt-its-state-from-the-first-apply.md Records the architecture decision.
.gitleaks.toml Detects plaintext state.
.gitignore Ignores state, plans, and caches.
Files not reviewed (1)
  • tofu/.terraform.lock.hcl: Generated file

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread tofu/guests.tf Resolved
Comment thread scripts/check-tofu-state-encryption.sh Resolved
Comment thread docs/runbooks/provision-lab-guests.md Resolved
… loses its grant (#445)

The enforced fixture swapped the state and plan methods together, so either
block losing enforced = true stayed green behind the other. One fixture per
block now; deleting either line turns its own fixture red.

Proxmox's pool delete drops the pool's ACL (delete_pool_acl), so the proof
teardown removes phoenix's /pool/proof grant. The runbook, ADR-0075 and
guests.tf say so, and the runbook re-grants before each proof rather than
widening the grant to /pool.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Gerrrt
Gerrrt enabled auto-merge (squash) October 2, 2026 05:05
Gerrrt and others added 2 commits October 1, 2026 22:07
…u-provision-825058

# Conflicts:
#	docs/changelog.md
main took ADR-0075 for the SSO re-acceptance (#799), so this branch's ADR
becomes ADR-0076 and every reference to it moves; references to main's
ADR-0075 are untouched. README counts both: 76 ADRs, 42 runbooks. The
changelog keeps both sides' 2026-10-01 entries.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Gerrrt
Gerrrt merged commit c6849c7 into main Oct 2, 2026
7 of 8 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/lab-vm-opentofu-provision-825058 branch October 2, 2026 05:13
Gerrrt added a commit that referenced this pull request Oct 2, 2026
…ine-3a8fc3

main took ADR-0076 for OpenTofu (#803), so this branch's ADR is now
ADR-0077 and its references follow; main's ADR-0075 and ADR-0076
references are untouched. README counts 77 ADRs and main's 42
runbooks. lint.sh keeps both new linters, tofu then ansible-lint.

#445 landed leaving the six hand-managed and their move into tofu/ to
#448 (ADR-0076 decision 7), with no MAC input on the guest module.
ADR-0077 decision 6 and the docs now say the MACs are pinned when the
six are declared there, rather than that #445 pins them.

Refs #448

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants