From 0af07753f1f606ec7bd1b6341c442f0d98cca091 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Tue, 22 Sep 2026 22:22:43 +0000 Subject: [PATCH 1/2] feat(media): add Navidrome to stacks/media, with its state in the NAS pull MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Navidrome joins Jellyfin on smaug as the tier's music server. It speaks the Subsonic API, so its clients are existing phone apps. The service shape comes from a scratch boot of the pinned image (0.64.1): - It runs as 65534, read-only, with cap_drop ALL. The image declares no user. - /cache is a 65534-owned tmpfs. The image has no /cache, and a root named volume there aborts the process a second after it reports ready. - Plugins are off, which keeps a 0700 /data/plugins out of what the backup user reads. - The wget healthcheck hits /ping, which answers 200 before any admin exists. - The admin is created with `navidrome user create --admin` from the TrueNAS shell. After that the web form that hands it to the first visitor answers 403. There is still no secrets file (#528). The issue said no new firewall rule was needed. The 50 -> 40 it relied on is two port-scoped passes, 443 and 8096, and the phones are on Hicks. So a fifth pass, 50 -> 10.0.40.30:4533, is written into build-the-nas §0.5, and the new §6.5 creates it along with the deploy. Navidrome's database holds playlists and play counts, so it is backed up the way Jellyfin's watch history is. backup-nas.sh now pulls two archives, jellyfin-config and navidrome-data, from one snapshot. It sizes both before transferring either, and names a directory missing from the snapshot. It verifies each set against its own MANIFEST, so the one-archive sets already retained stay valid. Authored, not deployed: the deploy waits on the mirror being whole (#558). Refs #141 Co-Authored-By: Claude Opus 5.5 --- .github/dependabot.yml | 6 +- .gitignore | 6 +- Makefile | 4 +- README.md | 12 +- docs/architecture.md | 2 +- docs/changelog.md | 20 +++ docs/network.md | 6 +- docs/observability.md | 3 +- docs/roadmap.md | 4 +- docs/runbooks/build-the-nas.md | 140 +++++++++++++++--- docs/runbooks/restore-the-stack.md | 7 +- docs/runbooks/schedule-maintenance.md | 6 +- docs/runbooks/successor-handover.md | 24 ++-- docs/security.md | 18 ++- scripts/backup-nas.sh | 200 +++++++++++++++++--------- scripts/backup-volumes.sh | 15 +- stacks/media/.env.example | 42 +++++- stacks/media/README.md | 120 +++++++++++----- stacks/media/compose.yaml | 121 +++++++++++++++- systemd/homelab-backup-nas.service | 5 +- systemd/homelab-backup-nas.timer | 2 +- 21 files changed, 593 insertions(+), 170 deletions(-) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 38ce99a4..783c38d3 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -84,9 +84,9 @@ updates: # README repeat the claim. The pins and make validate did reach it. This is # the third. # - # Two images now, not one: #256 added a node_exporter beside Jellyfin, because - # this host is scraped rather than running an agent. They share nothing, so no - # groups — each bump is its own PR. + # Three images now: #256 added a node_exporter beside Jellyfin, because this + # host is scraped rather than running an agent, and #141 added Navidrome. + # They share nothing, so no groups — each bump is its own PR. - package-ecosystem: docker-compose directory: /stacks/media schedule: diff --git a/.gitignore b/.gitignore index 11ed4971..4aac8383 100644 --- a/.gitignore +++ b/.gitignore @@ -48,9 +48,9 @@ stacks/*/snmp-exporter/mibs/ # `make backup` writes age-encrypted volume archives to backups/volumes//, # and `make restore` writes a pre-restore snapshot of what it is about to # replace to backups/volumes/.pre-restore-/. `make backup-nas` writes -# the age-encrypted pull of Jellyfin's state off smaug to backups/nas//. -# `make backup-firewall` writes the SOPS-encrypted pfSense config to -# backups/firewall/. None of it belongs in git. +# the age-encrypted pull of Jellyfin's and Navidrome's state off smaug to +# backups/nas//. `make backup-firewall` writes the SOPS-encrypted +# pfSense config to backups/firewall/. None of it belongs in git. # # The firewall config carries the WAN address, the full rule set and user # password hashes — docs/security.md says those are deliberately unpublished, diff --git a/Makefile b/Makefile index 79d95231..075f83ea 100644 --- a/Makefile +++ b/Makefile @@ -722,12 +722,12 @@ backup: ## Quiesce the stack, archive its volumes to ./backups/, verify, copy to STACK=$(STACK) ./scripts/backup-volumes.sh $(ARGS) .PHONY: backup-nas -backup-nas: ## Pull Jellyfin's state off smaug from its newest ZFS snapshot, encrypt and verify +backup-nas: ## Pull Jellyfin's and Navidrome's state off smaug from its newest ZFS snapshot, encrypt and verify @# The one backup that leaves this host to FETCH rather than to deliver. @# smaug cannot run backup-volumes.sh — no age, no checkout, no key, and @# ADR-0016 forbids it initiating anything upward — so this host reads a @# snapshot of erebor/apps over the 99 → 40:22 pass and encrypts what - @# arrives here (ADR-0045). Jellyfin is never stopped: the snapshot is the + @# arrives here (ADR-0045). Nothing is stopped: the snapshot is the @# quiesce. NAS_KEEP and not KEEP, for the reason backup-firewall gives @# for FW_KEEP. Sets land in backups/nas/, apart from the volume sets, and @# `make verify-backups` reads both. The copy to oracle is a step of this diff --git a/README.md b/README.md index 188f2f8d..69a4cd16 100644 --- a/README.md +++ b/README.md @@ -178,7 +178,7 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md). | Firewall / routing | [pfSense on FreeBSD 16](docs/network.md) | VLANs, DHCP, default-deny | | Virtualisation | Proxmox VE | Lab hypervisor | | Storage | [TrueNAS 25.10](docs/runbooks/build-the-nas.md) | `smaug`: 2× 18 TB ZFS mirror `erebor`, the SMB share, and the Docker the media stack runs under | -| Media | [Jellyfin](stacks/media) | Quick Sync transcoding on the NAS; the one stack deployed from TrueNAS rather than by `make deploy` | +| Media | [Jellyfin, Navidrome](stacks/media) | Quick Sync transcoding on the NAS; the one stack deployed from TrueNAS rather than by `make deploy` | | Lab observability | [Prometheus, Loki, Grafana](stacks/lab) | On `alexander`, a guest on `Saruman`, with its own Prometheus; only liveness crosses to the estate's, never telemetry | | Metrics | [Prometheus](stacks/observability/prometheus) | 30-day retention capped at 12 GiB, remote-write receiver | | Logs | [Loki](stacks/observability/loki) | Single-binary, filesystem storage | @@ -210,9 +210,9 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md). │ # Assistant, AdGuard Home, Immich, Paperless-ngx │ # and Vaultwarden so far; its own CA, leaves over │ # ACME (ADR-0037); for a host not yet built (ADR-0034, #404) -├── stacks/media/ # Jellyfin on smaug, under TrueNAS's own Docker — -│ # deployed by hand, no secrets file by decision -│ # (ADR-0040, #528) +├── stacks/media/ # Jellyfin and Navidrome on smaug, under TrueNAS's +│ # own Docker — deployed by hand, no secrets file +│ # by decision (ADR-0040, #528) ├── secrets/ # SOPS-encrypted; see secrets/README.md ├── scripts/ # bootstrap, render, validate, pin-digests, purge ├── SECURITY.md # disclosure policy and known exposure @@ -371,8 +371,8 @@ self-test, and the card is set to test itself every fortnight switch between the monitoring host and the network draws from it too, racked in U4 ([#110](https://github.com/Gerrrt/HomeLab/issues/110)); and the config export leaves the monitoring host nightly, the volume backup sets weekly -([#535](https://github.com/Gerrrt/HomeLab/issues/535)) and Jellyfin's state -weekly by way of it ([#484](https://github.com/Gerrrt/HomeLab/issues/484)), +([#535](https://github.com/Gerrrt/HomeLab/issues/535)) and the media tier's +state weekly by way of it ([#484](https://github.com/Gerrrt/HomeLab/issues/484)), so the rehearsal is what is left ([#92](https://github.com/Gerrrt/HomeLab/issues/92)). ## License diff --git a/docs/architecture.md b/docs/architecture.md index fc38fc38..c16208b1 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -191,7 +191,7 @@ hole from the monitoring VLAN into the monitored one. | `phoenix` (10.0.30.70) | 🟢 30 | *(none — a toolchain host, no Docker)* | The deployment host, built 2026-09-20: a Proxmox API token, an SSH key and a checkout, so that the Packer, OpenTofu and Ansible work after [#436](https://github.com/Gerrrt/HomeLab/issues/436) has somewhere to run from. A guest on `Saruman`, placed by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), which also decides that the estate's CA key stays on `prometheus` and does not follow the toolchain here. It holds no age key and converges nothing; it is the WireGuard endpoint of [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md), built 2026-09-22 ([#442](https://github.com/Gerrrt/HomeLab/issues/442)); the one thing it is meant to reach that no other guest does is `8006` on `Saruman`, by a single host-firewall rule — written 2026-09-20, when [#566](https://github.com/Gerrrt/HomeLab/issues/566) enabled the firewall the build had found disabled and narrowed the `local_network` alias that was admitting the whole segment underneath it. Its Alloy pushes to `alexander`, never to VLAN 99; the build is [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). Alloy agent (native package, `scripts/deploy-agent.sh`) | | `oracle` (10.0.99.30) | 🔴 99 | *(none — hand-run containers)* | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copies of the firewall export (`make backup-firewall`), of the weekly volume sets (`make backup`, [#535](https://github.com/Gerrrt/HomeLab/issues/535)) and of Jellyfin's state pulled off `smaug` (`make backup-nas`, [ADR-0045](adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md)). The estate's host for small off-host jobs — [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) | | `trinity` (10.0.99.40) | 🔴 99 | [`stacks/sensitive`](../stacks/sensitive) | **Not built yet** — ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md), after the firewall restore is rehearsed on it ([#404](https://github.com/Gerrrt/HomeLab/issues/404)). The foundation is authored: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github.com/Gerrrt/HomeLab/issues/129), [#130](https://github.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github.com/Gerrrt/HomeLab/issues/134)), Immich — four containers behind Caddy with a memory limit on each ([#132](https://github.com/Gerrrt/HomeLab/issues/132)) — Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github.com/Gerrrt/HomeLab/issues/131)) are authored as well; the rest follow. Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s | -| `smaug` (10.0.40.30) | 🟡 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **Built, pooled and deployed**: TrueNAS on its boot SSD, the static above since 2026-09-16, the four inbound rules verified in position, the mirror `erebor` since 2026-09-18 and this stack running on it since 2026-09-19 — from a copy of the compose file on the pool, brought up with `docker compose` under TrueNAS's own Docker ([`build-the-nas.md`](runbooks/build-the-nas.md) §6). Jellyfin alone, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github.com/Gerrrt/HomeLab/issues/138)). Scraped by `prometheus` on `9100`; it pushes nothing, and runs no Alloy — the estate's first scraped host, and the reason [#256](https://github.com/Gerrrt/HomeLab/issues/256) was more than a line of YAML. That issue settled the fork TrueNAS opened in it: `node_exporter`, as a digest-pinned container in this stack rather than TrueNAS's own endpoint, so the existing `99 → 40:9100` pass, the `host-overview` dashboard and seven rules in `host.rules.yaml` all keep working unchanged. The `node` job and `prometheus/targets/node.yaml` are live, and the target with them since 2026-09-19, once the exporter answered from the pool. SMART reaches the same scrape by [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md): a root cron job in TrueNAS's UI runs the estate's collector and the exporter serves the file — still no agent, still nothing initiating from 40 — and patch state is deliberately not collected on an appliance ([#483](https://github.com/Gerrrt/HomeLab/issues/483)) | +| `smaug` (10.0.40.30) | 🟡 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **Built, pooled and deployed**: TrueNAS on its boot SSD, the static above since 2026-09-16, the four inbound rules verified in position, the mirror `erebor` since 2026-09-18 and this stack running on it since 2026-09-19 — from a copy of the compose file on the pool, brought up with `docker compose` under TrueNAS's own Docker ([`build-the-nas.md`](runbooks/build-the-nas.md) §6). Jellyfin, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github.com/Gerrrt/HomeLab/issues/138)); and Navidrome for music, publishing 4533 for the Subsonic apps on the phones on Hicks through a fifth pass — authored in the stack and deployed by [`build-the-nas.md`](runbooks/build-the-nas.md) §6.5 once the mirror is whole ([#141](https://github.com/Gerrrt/HomeLab/issues/141)). Scraped by `prometheus` on `9100`; it pushes nothing, and runs no Alloy — the estate's first scraped host, and the reason [#256](https://github.com/Gerrrt/HomeLab/issues/256) was more than a line of YAML. That issue settled the fork TrueNAS opened in it: `node_exporter`, as a digest-pinned container in this stack rather than TrueNAS's own endpoint, so the existing `99 → 40:9100` pass, the `host-overview` dashboard and seven rules in `host.rules.yaml` all keep working unchanged. The `node` job and `prometheus/targets/node.yaml` are live, and the target with them since 2026-09-19, once the exporter answered from the pool. SMART reaches the same scrape by [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md): a root cron job in TrueNAS's UI runs the estate's collector and the exporter serves the file — still no agent, still nothing initiating from 40 — and patch state is deliberately not collected on an appliance ([#483](https://github.com/Gerrrt/HomeLab/issues/483)) | | `bahamut` (10.0.30.50) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 domain controller, PDC emulator and DNS for `ad.matrix.elysium` — Tier 0. Static, because every member finds a DC through DNS and the DCs *are* the DNS. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | | `leviathan` (10.0.30.51) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 second domain controller and DNS — Tier 0. Static, for the same reason. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | | `titan` (10.0.30.52) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 file and member server — the shares, and the NTLM relay target that only exists because 2025 requires outbound SMB signing and not inbound — Tier 1. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | diff --git a/docs/changelog.md b/docs/changelog.md index e4dbd6bc..9268308d 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -19,6 +19,26 @@ docstring gives: it is a record, not a claim about now. ## 2026-09-22 +- **[#141](https://github.com/Gerrrt/HomeLab/issues/141) Navidrome is + authored into `stacks/media`, and the issue was wrong twice.** It said no + new firewall rule would be needed; the `50 → 40` it leaned on is two + port-scoped passes, `443` and `8096`, and the phones Navidrome serves are on + Hicks — so a fifth, `50 → 10.0.40.30:4533`, is written into + `build-the-nas.md` §0.5 and made by the new §6.5. It said nothing about + backup; playlists and play counts are music's watch history, so + `backup-nas.sh` now pulls two archives, `jellyfin-config` and + `navidrome-data`, out of one snapshot, and verifies each set against its own + MANIFEST so the one-archive sets already on the shelf stay valid. + + **The pinned image decided three lines.** It has no `/cache`, and a named + volume mounted there as root kills the process a second after it reports + ready, so the cache is a `65534`-owned tmpfs. Plugins are off, which keeps + a `0700` directory out of what the backup user reads. And an admin made + with `navidrome user create --admin` from the TrueNAS shell closes the web + form that would otherwise hand the admin to the first visitor — so the + stack still has no secrets file (#528). Not deployed: the mirror is one + disk until [#558](https://github.com/Gerrrt/HomeLab/issues/558). + - **[#455](https://github.com/Gerrrt/HomeLab/issues/455) The off-estate drive is bought, and buying it settles less than the row implied.** A WD Elements Portable 5 TB, `WDBU6Y0050BBK-WESN`, bus-powered USB 3.2 Gen 1. The second diff --git a/docs/network.md b/docs/network.md index b748be5b..cfe3857f 100644 --- a/docs/network.md +++ b/docs/network.md @@ -341,7 +341,11 @@ Televisions and consoles. Internet only. on `9100` and `22` — four host-scoped, port-scoped passes above *Block access to CasaBonita* on their interfaces. Everything else on every other segment is still refused, and the televisions need no rule at all because they share this - broadcast domain with the server. + broadcast domain with the server. **A fifth is authored and not created:** + Hicks → `10.0.40.30:4533`, for the Subsonic apps on the phones to reach + Navidrome ([#141](https://github.com/Gerrrt/HomeLab/issues/141)), in + [`build-the-nas.md`](runbooks/build-the-nas.md) §0.5 and made by §6.5 on + the day Navidrome is deployed. Until then, four is the count. - **What answers on `9100` is `node_exporter`**, which makes this the one host in the estate that Prometheus *scrapes* rather than is pushed to ([#256](https://github.com/Gerrrt/HomeLab/issues/256), diff --git a/docs/observability.md b/docs/observability.md index 0ecd22c4..835d3fbc 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -784,7 +784,8 @@ has stopped leaving this host"; since [#535](https://github.com/Gerrrt/HomeLab/issues/535) `backup-volumes` does the same with each weekly set; and since [#484](https://github.com/Gerrrt/HomeLab/issues/484) `backup-nas` — the one -job that first *fetches* from another host, Jellyfin's state off `smaug` — +job that first *fetches* from another host, Jellyfin's and Navidrome's +state off `smaug` — copies its set the same way. `verify-backups` hashes the far side of both set directories every morning. No rule names any of the three; the generic pair carries them all. diff --git a/docs/roadmap.md b/docs/roadmap.md index 96233dac..ae04350e 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -173,7 +173,9 @@ workstation can mount the share. that can fail** have no gate and no order between them. - **[#140](https://github.com/Gerrrt/HomeLab/issues/140) Audiobookshelf and [#141](https://github.com/Gerrrt/HomeLab/issues/141) Navidrome** have no - gate but the mirror being whole, which is #558. + gate but the mirror being whole, which is #558. Navidrome is authored in + `stacks/media` and waits only on that; deploying it is + [`build-the-nas.md`](runbooks/build-the-nas.md) §6.5, fifth pass included. ## Saruman: the domain, then the SOC diff --git a/docs/runbooks/build-the-nas.md b/docs/runbooks/build-the-nas.md index 3ba16e09..dc07a380 100644 --- a/docs/runbooks/build-the-nas.md +++ b/docs/runbooks/build-the-nas.md @@ -216,16 +216,17 @@ the box and in the firewall, not here. The static is set on the host and the reservation is set on the server, and both are done because either alone is a single point of drift. -### §0.5 — Create the four rules, in order and in position +### §0.5 — Create the rules, in order and in position -**Position is the whole difficulty.** All four sit above a deny that has been -in place since 2025; appended where new rules naturally land they would match +**Position is the whole difficulty.** Every one of them sits above a deny that +has been in place since 2025; appended where new rules naturally land they would match nothing, and *"can I reach the NAS"* would still pass for the wrong reason. | On interface | Protocol / source → destination | Description | Position | | --- | --- | --- | --- | | Hicks (50) | `tcp` `vlan50 net` → `10.0.40.30` port `443` | `Allow HTTPS to smaug` | **above** *Block access to CasaBonita* | | Hicks (50) | `tcp` `vlan50 net` → `10.0.40.30` port `8096` | `Allow 8096 to smaug` | **above** *Block access to CasaBonita* | +| Hicks (50) | `tcp` `vlan50 net` → `10.0.40.30` port `4533` | `Allow 4533 to smaug` | **above** *Block access to CasaBonita* | | Winterfell (99) | `tcp` `10.0.99.20` → `10.0.40.30` port `9100` | `Allow 9100 to smaug` | **above** *Block access to CasaBonita* | | Winterfell (99) | `tcp` `10.0.99.20` → `10.0.40.30` port `22` | `Allow SSH to smaug` | **above** *Block access to CasaBonita* | @@ -236,6 +237,15 @@ the description is what §0.6 matches on and a description naming one port is unambiguous about which rule answered. **Set these descriptions exactly**; they are load-bearing in the next section, not decoration. +**The fifth came later, for Navidrome** ([#141](https://github.com/Gerrrt/HomeLab/issues/141)). +The four above were created on 2026-09-16; `Allow 4533 to smaug` is added by +§6.5, when Navidrome is. Its consumer is the Subsonic apps on the phones, and +the phones are on Hicks, so it is the `8096` rule's shape on another port — +[ADR-0012](../adr/0012-publish-only-ports-with-an-off-host-consumer.md)'s named +off-host consumer, and nothing wider. The issue that proposed Navidrome said no +rule would be needed; that was written against a `50 → 40` pass that turned out +to be two port-scoped ones. + **The Hicks rule's ports differ from ADR-0016's table, and deliberately.** That table says `22,8096`, which assumed a box administered over SSH — ADR-0016 decided Ubuntu Server. `smaug` runs TrueNAS, which is administered over HTTPS, @@ -271,8 +281,8 @@ pfctl -sr -vv \ Each pass must appear **above** the *Block access to CasaBonita* rule on its own interface: `Allow 9100`/`Allow SSH` before the block on `igc0.99`, and -`Allow HTTPS`/`Allow 8096` before it on `igc0.50`. **Read the order, not the -numbers.** `-vv` numbers each ruleset from zero rather than counting output +`Allow HTTPS`/`Allow 8096`/`Allow 4533` before it on `igc0.50` — the last +once §6.5 has created it. **Read the order, not the numbers.** `-vv` numbers each ruleset from zero rather than counting output lines, so its `@` indices match neither `pfctl -sr | grep -n` nor anything written down here — they are a printing artefact, and only the sequence is a fact about the firewall. @@ -292,6 +302,13 @@ reads: nc -z -w3 10.0.40.30 443 && echo "WRONG: 99 can reach 443" || echo "correct: blocked" ``` +Once Navidrome is up (§6.5), `4533` is the same test on its own port — +refused from here, answered from Hicks: + +```bash +nc -z -w3 10.0.40.30 4533 && echo "WRONG: 99 can reach 4533" || echo "correct: blocked" +``` + Then read **the tripwire counter on `igc0.40`** ([#223](https://github.com/Gerrrt/HomeLab/issues/223)). It matches packets *originating* on CasaBonita, and the return traffic for a session Hicks opened @@ -465,8 +482,8 @@ decision made deliberately rather than drifted into. | Dataset | Record size | atime | What it holds | Backed up | | --- | --- | --- | --- | --- | -| `erebor/media` | `1M` | off | films, music, the library | **no** | -| `erebor/apps` | default | off | the compose files, and Jellyfin's `/config` as a bind mount | **yes** — §4.1 and §6.2 | +| `erebor/media` | `1M` | off | films, music (under `music/`, which Navidrome reads), the library | **no** | +| `erebor/apps` | default | off | the compose files, Jellyfin's `/config` and Navidrome's `/data` as bind mounts | **yes** — §4.1 and §6.2 | **Why the split.** ADR-0008 already ruled the library replaceable — its loss is *"annoying rather than catastrophic"* — and backing up 18 TB of re-downloadable @@ -545,13 +562,13 @@ ls -1 /mnt/erebor/apps/.zfs/snapshot/ > four the preset wrote and that one. > > **No workstation can mount this share, and that was found by trying.** The -> Hicks rules from §0.5 pass `443` and `8096` to `smaug` and nothing else; -> SMB is `445`, so a Hicks machine that reaches the TrueNAS UI and Jellyfin -> gets nothing from `\\10.0.40.30\media`. Only devices already on +> Hicks rules from §0.5 pass `443` and `8096` to `smaug` and nothing else +> (and `4533` since §6.5, which changes nothing here); SMB is `445`, so a +> Hicks machine that reaches the TrueNAS UI and Jellyfin gets nothing from `\\10.0.40.30\media`. Only devices already on > CasaBonita can mount it, and those are televisions. Getting a film onto the > library today means the console shell — `mkdir` and `curl` under > `/mnt/erebor/media/` — which is how the test clip in §6.1 arrived. Whether -> the answer is a fifth Hicks rule on `445` or something else is +> the answer is another Hicks rule on `445` or something else is > [#523](https://github.com/Gerrrt/HomeLab/issues/523)'s; it is recorded > here because the share exists and looks usable and is not. @@ -615,6 +632,17 @@ mkdir -p /mnt/erebor/apps/jellyfin/config \ && chmod 755 /mnt/erebor/apps/jellyfin/config ``` +**Navidrome's the same way**, as `NAVIDROME_DATA_PATH` — and the music +directory it reads, `MUSIC_PATH`, which is created rather than left for Docker +to make as root on first `up`. The library's own ACL (§5's `everyone@` Basic +Read, inherited) is what lets uid 65534 read what lands in it: + +```bash +mkdir -p /mnt/erebor/apps/navidrome/data /mnt/erebor/media/music \ + && chown 65534:65534 /mnt/erebor/apps/navidrome/data \ + && chmod 755 /mnt/erebor/apps/navidrome/data +``` + > **Migrating the state that already exists.** The stack ran from 2026-09-19 > with `/config` as a named volume, so on this host the directory above is > not empty on first use — it is filled from the volume, with Jellyfin @@ -650,11 +678,23 @@ stat -c '%g %G' /dev/dri/renderD128 docker compose up -d && docker compose ps ``` -Jellyfin binds `8096` and reads `erebor/media`. `node-exporter` binds `9100` -and is the whole of how this host is monitored — see §6.1. Jellyfin's state -is the bind mount above, on `erebor/apps`, which is what §4.1 snapshots and -§6.2 pulls; its cache is a named volume on `erebor/ix-apps`, and is not -backed up by anything, by decision. +Jellyfin binds `8096` and reads `erebor/media`. Navidrome binds `4533` and +reads `erebor/media/music`. `node-exporter` binds `9100` and is the whole of +how this host is monitored — see §6.1. Jellyfin's and Navidrome's state are +the bind mounts above, on `erebor/apps`, which is what §4.1 snapshots and +§6.2 pulls; Jellyfin's cache is a named volume on `erebor/ix-apps` and +Navidrome's is a tmpfs, and neither is backed up by anything, by decision. + +**Create Navidrome's admin immediately after the first `up`, from this +shell.** Until one exists, the first visitor to `:4533` is offered the form +that creates it — every console on this segment and every workstation on +Hicks. Doing it here closes that route (the web form answers `403` once an +admin exists) and keeps the password out of every environment variable, which +is why this stack still has no secrets file. `-it` because it prompts twice: + +```bash +docker exec -it media-navidrome navidrome user create --admin -u -n +``` **Updating the stack is the same two `curl` lines and `docker compose up -d` again.** Nothing on this host pulls from `main` on its own: there is no @@ -769,7 +809,8 @@ is what makes step 7 safe. 2. **Give the user read on the dataset, and nothing else.** From the console shell, read what the Apps preset set on the dataset root before changing - it, and record both lines here the way §5 recorded `bilbo`'s: + it, and record both lines here the way §5 recorded `bilbo`'s. Navidrome's + `navidrome/data` joined later and inherits the entry; §6.5 step 6 proves it: ```bash zfs get -H acltype,aclmode erebor/apps \ @@ -870,7 +911,7 @@ is what makes step 7 safe. > the `igc0.40` tripwire read **0 packets** on 2026-09-20 with the pull > done. Step 9 is the commit this block landed in. -### §6.3 — Restore Jellyfin's state +### §6.3 — Restore the media tier's state Two cases, in the order to try them. @@ -915,6 +956,14 @@ The proof is the same as the migration's: the users and the watch history are back in the web UI. `--numeric-owner` on both ends is what keeps `65534` as `65534` across two hosts that spell it differently. +**Navidrome is the same two cases with its own names.** A set from +2026-09-22 on holds `navidrome-data.tar.gz.age` beside Jellyfin's, both read +from the one snapshot, and the MANIFEST row says it goes to the `navidrome` +service's `/data`. Swap `jellyfin`/`jellyfin/config`/`jellyfin-config` for +`navidrome`/`navidrome/data`/`navidrome-data` in every line above, and prove +it by logging in: the users, playlists and favourites are back. Sets older +than that hold Jellyfin's archive only, and say so in their MANIFEST. + ### §6.4 — Turn SMART collection on, and confirm the boot disk's row The exporter §6.1 stood up reads nothing a drive says: it is uid 65534, @@ -1091,6 +1140,54 @@ reopens it. > The cron job is as the table says: daily 08:30, `root`, stdout hidden, > stderr not. +### §6.5 — Add Navidrome to the running stack + +[#141](https://github.com/Gerrrt/HomeLab/issues/141). §6 now describes a stack +with Navidrome in it; this is the order in which it joins the one that was +deployed on 2026-09-19 without it. The order matters for one reason: the +weekly pull in §6.2 expects `navidrome/data` in every snapshot once the +monitoring host's checkout carries this change. So **the directory has to be +in a snapshot before the next Saturday 03:30 UTC after that checkout moves**, +or that run fails naming the directory. It fails loudly and writes nothing, +but it is still a missed week. + +**Gate:** the mirror is whole ([#558](https://github.com/Gerrrt/HomeLab/issues/558)). +[`roadmap.md`](../roadmap.md) puts no other gate on this. + +1. **The fifth pass.** Create `Allow 4533 to smaug` on Hicks exactly as the + §0.5 table gives it, and read its position from `morpheus` with the + §0.6 `pfctl` line: it sits above *Block access to CasaBonita* on `igc0.50`. +2. **The directories**, as root on this host — the Navidrome block in §6. +3. **Re-fetch the two files and bring it up** — the `curl` lines in §6, + then `docker compose up -d`. Jellyfin and `node-exporter` are unchanged + and compose leaves them running. +4. **Create the admin, at once** — the `docker exec -it … user create` + line in §6. Then `docker compose ps` shows `media-navidrome` healthy. +5. **Prove the reach.** From a Hicks workstation, + `curl -s http://10.0.40.30:4533/ping` prints `.`. From the monitoring + host, the §0.6 `nc` line for `4533` reads *correct: blocked*. From a + phone on Hicks, a Subsonic client logs in with the admin and plays a + track from `music/`. +6. **Prove the backup user can read it**, the way §6.2 step 5 did for + Jellyfin, against the first snapshot taken **after** step 3. Run it from + the Data Protection page by hand, as §4.1 did, rather than waiting: + + ```bash + ssh frodo@10.0.40.30 "tar -cf /dev/null --numeric-owner -C '/mnt/erebor/apps/.zfs/snapshot//navidrome/data' . && echo readable" + ``` + + `ND_PLUGINS_ENABLED=false` is what keeps a `0700` directory out of it, + and this proves that it did. +7. **Move the monitoring host's checkout, then run the pull by hand**, as + §6.2 step 6 did. The set it writes has two archives, and `--list` then + `make verify-backups` read both: + + ```bash + make backup-nas && make backup-nas ARGS=--list && make verify-backups + ``` + +> **Not yet done.** Authored 2026-09-22. Waits on #558. + ## §7 — Verify > **As of 2026-09-19:** the monitoring-host line holds in both halves, the @@ -1128,9 +1225,12 @@ reopens it. - A television on CasaBonita finds Jellyfin and plays something **without** any firewall rule being involved -- A Hicks workstation reaches `https://10.0.40.30` and `http://10.0.40.30:8096` +- A Hicks workstation reaches `https://10.0.40.30` and `http://10.0.40.30:8096`, + and — once §6.5 is done — a phone on Hicks plays a track through a Subsonic + app pointed at `http://10.0.40.30:4533` - The monitoring host reaches `9100` and **nothing else**. Both halves are - checkable now: `443` and `8096` must be refused from the monitoring host, and + checkable now: `443`, `8096` and `4533` must be refused from the monitoring + host, and `node_exporter` must be answering — §6.1 is what stands it up, and [#256](https://github.com/Gerrrt/HomeLab/issues/256) settled that it is `node_exporter` rather than TrueNAS's own endpoint. `up{job="node"}` should be diff --git a/docs/runbooks/restore-the-stack.md b/docs/runbooks/restore-the-stack.md index 01c59435..f5f9478b 100644 --- a/docs/runbooks/restore-the-stack.md +++ b/docs/runbooks/restore-the-stack.md @@ -307,11 +307,12 @@ curl -sG http://localhost:9090/api/v1/query \ --- -## Jellyfin's state is a different set +## The media tier's state is a different set `make restore` and everything above it read `backups/volumes/`, the sets of -the stack on this host. Jellyfin's state on `smaug` is pulled into -`backups/nas/` by `make backup-nas` — one archive per set, read out of a ZFS +the stack on this host. Jellyfin's and Navidrome's state on `smaug` is pulled +into `backups/nas/` by `make backup-nas` — one archive per service in each +set (sets before 2026-09-22 hold Jellyfin's alone), read out of a ZFS snapshot rather than a stopped container, and copied to `oracle` in the same run under `backups/nas` beside these sets ([ADR-0045](../adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md)) — diff --git a/docs/runbooks/schedule-maintenance.md b/docs/runbooks/schedule-maintenance.md index 6039072b..ad17dcac 100644 --- a/docs/runbooks/schedule-maintenance.md +++ b/docs/runbooks/schedule-maintenance.md @@ -71,8 +71,8 @@ until that is done. One job's output *arrives* from another host before it leaves: `backup-nas` reaches into `smaug` over `99 → 40:22` — the direction [ADR-0016](../adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) -requires of CasaBonita, which may initiate nothing — reads Jellyfin's state -out of the dataset's newest ZFS snapshot, encrypts it here, and then copies +requires of CasaBonita, which may initiate nothing — reads Jellyfin's and +Navidrome's state out of the dataset's newest ZFS snapshot, encrypts it here, and then copies the set to `oracle` by the same helpers and under the same rules as the volume sets ([ADR-0045](../adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md)). @@ -628,7 +628,7 @@ expected rather than a second fault. | `backup-firewall` exits 1 with *off-host copy FAILED* | `oracle` is down, its host key is not in `robo`'s `known_hosts`, or this host's key is not authorised there | The export was written locally and is intact. Repair the path to `oracle` — [`restore-the-firewall.md`](restore-the-firewall.md) §0 — and the next run copies every file that never left | | `backup-volumes` exits 1 with *off-host copy FAILED* | The same three causes, or `oracle` ran out of room | The set was written and verified here and the stack is up. Repair the path, then `make backup ARGS=--copy-only` — it copies every set `oracle` lacks without stopping the stack, and the next weekly run would do the same | | `verify-backups` exits 1 naming a set on `oracle` as *missing* or *differs* | The copy of that set never landed, was removed, or its bytes no longer hash to the manifest | Every local set still decrypts, or the message would say so first. *Missing*: `make backup ARGS=--copy-only`. *Differs*: nothing removes it for you — look at it, `rm -rf` that one directory on `oracle`, then the same command. [`restore-the-stack.md`](restore-the-stack.md) §0 | -| `backup-nas` exits 1 | The message says which: *cannot reach* means SSH is off on `smaug`, the `99 → 40:22` pass is out of position, or the key or host key is missing; *newest snapshot … is N hours old* means the periodic task on `smaug` has stopped; *tar could not read* means Jellyfin wrote a file `frodo` cannot read | Nothing is stopped on either side and the last complete set is intact. [`build-the-nas.md`](build-the-nas.md) §6.2 is the setup this checks against, and names the fallback for the third case. A snapshot named in the *future* means `NAS_SNAPSHOT_TZ` is not `smaug`'s zone | +| `backup-nas` exits 1 | The message says which: *cannot reach* means SSH is off on `smaug`, the `99 → 40:22` pass is out of position, or the key or host key is missing; *newest snapshot … is N hours old* means the periodic task on `smaug` has stopped; *tar could not read* means Jellyfin or Navidrome wrote a file `frodo` cannot read; *cannot read … the directory does not exist* names a service whose directory is not in that snapshot — Navidrome not yet deployed (§6.5) | Nothing is stopped on either side and the last complete set is intact. [`build-the-nas.md`](build-the-nas.md) §6.2 is the setup this checks against, and names the fallback for the third case. A snapshot named in the *future* means `NAS_SNAPSHOT_TZ` is not `smaug`'s zone | | `dashboards-drift` exits 1 | Grafana holds a dashboard edit that is not committed | Not a fault. Run `make dashboards-export`, read `git diff`, commit it. If the diff is empty but the job still fails, Grafana is down or `make render` has never run here | | `loki-coverage` exits 1 | A Loki alerting rule cannot see a host that is producing exactly the lines it hunts | Not an outage — nothing is broken, but an alert cannot fire for that host, which is how [#261](https://github.com/Gerrrt/HomeLab/issues/261) went unnoticed. The FAIL line names the rule, the host and the log type the lines are arriving under; the fix is usually an `or` branch on the rule for that host's stream. A `WARN` is the latent form — the rule cannot reach the host at all, but nothing there matches it today — and does not fail the job | | `firewall-claims` exits 1 | A segmentation claim in `docs/firewall-claims.yaml` no longer matches the running ruleset | Not an outage, and the firewall is not the thing that is wrong — a document is. The FAIL line names the interface, the segment and the direction: *now reaches X* means a block was removed or a VLAN was added, *no longer reaches X* means a block landed and the prose still describes the world before it. Re-derive with `scripts/check_firewall_claims.py --derive`, then move the prose that cites it — `docs/network.md` and `docs/security.md`. Never edit an ADR in place: [ADR-0001](../adr/0001-record-architecture-decisions.md) makes them immutable, so a stale one gets a marked amendment or a superseding ADR | diff --git a/docs/runbooks/successor-handover.md b/docs/runbooks/successor-handover.md index b0172aee..d276e3af 100644 --- a/docs/runbooks/successor-handover.md +++ b/docs/runbooks/successor-handover.md @@ -146,8 +146,8 @@ make validate `make validate` runs everything CI runs, plus one check that is about *this host*: whether the stack runs here while no `homelab-*` units are installed. The -schedule covers convergence, the volume backups, the pull of Jellyfin's state -off `smaug`, the firewall export, the SNMP verification and the dashboard +schedule covers convergence, the volume backups, the pull of Jellyfin's and +Navidrome's state off `smaug`, the firewall export, the SNMP verification and the dashboard drift check — [`schedule-maintenance.md`](schedule-maintenance.md) has the table and the staleness threshold on each. @@ -244,9 +244,9 @@ outgoing operator directly: - The pfSense, MokerLink, iLO and APC network-card **admin passwords**. None of them are in this repository in any form. -- Jellyfin's `admin` on `smaug`, and TrueNAS's own web-UI admin. Neither is in - this repository in any form — `stacks/media` has no secrets file by - decision, and [`security.md`](../security.md) § Secrets says why. You do not +- Jellyfin's and Navidrome's admins on `smaug`, and TrueNAS's own web-UI + admin. None is in this repository in any form — `stacks/media` has no + secrets file by decision, and [`security.md`](../security.md) § Secrets says why. You do not need Jellyfin's: from a workstation on Hicks, *Forgot Password* on `http://10.0.40.30:8096` writes a PIN file into the container's config volume, the login page then takes the PIN, and the file is read from a @@ -257,8 +257,16 @@ outgoing operator directly: ``` That flow only answers an address Jellyfin counts as local, which every - RFC 1918 range is until its *LAN networks* setting says otherwise. TrueNAS's - admin has no such path from here; get it from the outgoing operator. + RFC 1918 range is until its *LAN networks* setting says otherwise. + Navidrome's is reset from the same TrueNAS shell, and prompts for the new + password: + + ```bash + docker exec -it media-navidrome navidrome user edit -u --set-password -n + ``` + + TrueNAS's admin has no such path from here; get it from the outgoing + operator. - The internal CAs' private keys — there are two, and both live on `prometheus` with one offline copy each on the medium that holds the age key. `certificates/` is gitignored and host-local, so a clean clone has no @@ -349,7 +357,7 @@ new keypair whose public half matches nothing. | --- | --- | | The whole observability stack (`make down`) | The record, and only the record. Nothing in the house depends on it, dashboards and alert rules are in git, and metrics and logs re-accumulate. `grafana-data` is the exception — users, annotations and any un-exported dashboard edit live only there | | `stacks/lab` | Nothing. It is committed and deployable and has never been deployed | -| `stacks/media` | Film night, and the watch positions if `jellyfin-config` goes with it. Nothing else depends on it, and [ADR-0008](../adr/0008-place-services-by-data-trust.md) already ruled the library replaceable | +| `stacks/media` | Film night and music, and the watch positions and playlists if `jellyfin-config` and `navidrome-data` go with it. Nothing else depends on it, and [ADR-0008](../adr/0008-place-services-by-data-trust.md) already ruled the library replaceable | | `homelab-converge.timer` | Automatic deployment. The host stays on whatever revision it is on until someone runs `make up`, which is exactly how it worked before | | `dashboards-drift` | The daily proof that Grafana holds no uncommitted dashboard edit | | Suricata on the terminal segments | Detection, not connectivity. It is alert-only — `Block Offenders` is off on both interfaces and stays off. [`enable-suricata.md`](enable-suricata.md) | diff --git a/docs/security.md b/docs/security.md index 0e1cc4e6..031dd2d9 100644 --- a/docs/security.md +++ b/docs/security.md @@ -275,6 +275,13 @@ existed. The residual it leaves, accepted, is one more service on the NAS with one more key that reads it — a key that lives on the host already holding the estate's age identity, and reads a directory that includes Jellyfin's users' password hashes. +A fifth is written and not yet made: Hicks → `10.0.40.30:4533`, for +Navidrome's phone clients ([#141](https://github.com/Gerrrt/HomeLab/issues/141)), +the `8096` rule's shape on one more port, created by +[`build-the-nas.md`](runbooks/build-the-nas.md) §6.5 with the deploy. It +adds no residual `8096` did not already have: Navidrome's first admin is made +from the TrueNAS shell before the port is ever reachable from Hicks, and the +library behind it is mounted read-only. [`network.md`](network.md) holds the current list. **Skids' does not exist.** `10.0.99.40 → 10.0.20.104:80,443/tcp` — Home Assistant to the Hue bridge, the one device on that segment with a local API — still waits above the block that @@ -433,10 +440,13 @@ assumption consistent with what they are. protects it is a hash at rest, VLAN 40's terminal property, the `50 → 40` passes being the only way in, and [ADR-0008](adr/0008-place-services-by-data-trust.md)'s blast radius — a - media server whose data is replaceable. Audiobookshelf - ([#140](https://github.com/Gerrrt/HomeLab/issues/140)) and Navidrome - ([#141](https://github.com/Gerrrt/HomeLab/issues/141)) create their first - user the same way and join this bullet when they land. What would retire + media server whose data is replaceable. Navidrome + ([#141](https://github.com/Gerrrt/HomeLab/issues/141)) is the same class: + its admin is made by `navidrome user create --admin` in the container, + prompted for on the TrueNAS shell, and kept as a hash in `navidrome.db` on + the bind mount `backup-nas.sh` pulls. Audiobookshelf + ([#140](https://github.com/Gerrrt/HomeLab/issues/140)) creates its first + user the same way and joins this bullet when it lands. What would retire it: a service on that tier taking a credential from outside. Then the tier gets `secrets/media.sops.yaml` and a rule of its own under [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md), diff --git a/scripts/backup-nas.sh b/scripts/backup-nas.sh index 2a2e126c..c1c80797 100755 --- a/scripts/backup-nas.sh +++ b/scripts/backup-nas.sh @@ -1,18 +1,19 @@ #!/usr/bin/env bash # -# Pull Jellyfin's state off smaug, encrypt it here, and prove the archive is -# readable before calling the run a success. +# Pull the media tier's state off smaug, encrypt it here, and prove each +# archive is readable before calling the run a success. # # WHAT THIS PROTECTS, AND WHAT IT DOES NOT # # erebor/apps on smaug — Jellyfin's database, users, watch history, resume -# positions and metadata: the half of the media tier ADR-0008 ruled -# irreplaceable, and the one build-the-nas.md §4 declared backed up for a week -# before anything backed it up (#484). It does NOT protect erebor/media, the +# positions and metadata, and Navidrome's database, playlists, favourites and +# play counts (#141): the half of the media tier ADR-0008 ruled irreplaceable, +# and the one build-the-nas.md §4 declared backed up for a week before +# anything backed it up (#484). It does NOT protect erebor/media, the # library, which ADR-0008 ruled replaceable and 18 TB of which would not fit # anywhere in this estate; nor jellyfin-cache, transcode scratch that -# backup-volumes.sh lists as DISPOSABLE by name; nor erebor/ix-apps, Docker's -# images and that cache volume. Those omissions are decisions (ADR-0045), and +# backup-volumes.sh lists as DISPOSABLE by name; nor Navidrome's cache, a +# tmpfs; nor erebor/ix-apps, Docker's images and that cache volume. Those omissions are decisions (ADR-0045), and # stacks/media/README.md states them where the stack is. # # WHY IT PULLS, AND WHY THIS IS NOT backup-volumes.sh @@ -35,16 +36,17 @@ # tar's stdout on smaug becomes age's stdin here, and nothing unencrypted # touches this disk. # -# WHY IT READS A SNAPSHOT AND NEVER STOPS JELLYFIN +# WHY IT READS A SNAPSHOT AND NEVER STOPS JELLYFIN OR NAVIDROME # # backup-volumes.sh stops the stack because a copy of a live SQLite database -# is a file that looks like a backup. Stopping Jellyfin over ssh would need the +# is a file that looks like a backup. Stopping either server over ssh would need the # docker socket on smaug, which is root, for a user whose whole design is that # it can read one directory and do nothing else. A ZFS snapshot is the quiesce # instead: TrueNAS takes one of erebor/apps every night (the periodic task # build-the-nas.md §4 records), and this reads the newest one through # .zfs/snapshot/. A snapshot is atomic across the dataset, so jellyfin.db, its -# -wal and its -shm are captured at one instant — the crash image SQLite's WAL +# -wal and its -shm are captured at one instant — and so are navidrome.db and +# its pair, since BOTH archives of a set are read out of the same snapshot — the crash image SQLite's WAL # mode is designed to recover from. That is a stronger claim than # backup-volumes.sh's --hot can make and a weaker one than its quiesce, and the # manifest says `snapshot` rather than `quiesced` for that reason. Two things @@ -85,6 +87,10 @@ # du -sk '//' sizing, and the read preflight # tar --numeric-owner -czf - -C '//' . # +# The last two once per archive in ARCHIVES, whose subpaths are fixed in this +# file rather than taken from the environment — they are the stack's layout, +# and .env.example is where that is written. +# # Compression runs on smaug, so the wire carries gzip and age runs here. # # THE USER ON THE FAR SIDE @@ -94,7 +100,7 @@ # the read with the same tar, to /dev/null. The key is this host's operator # key, the one that already reaches morpheus and oracle. Known limit, stated # rather than engineered away: whoever holds that key can read Jellyfin's -# configuration, which includes its users' password hashes; what lands here is +# and Navidrome's databases, which include their users' password hashes; what lands here is # ciphertext to the same two recipients that can open grafana.db. # # WHY NAS_KEEP AND NOT KEEP @@ -131,7 +137,6 @@ # NAS_SSH_TARGET default frodo@10.0.40.30 the host the state is pulled FROM # NAS_OFFHOST default atropos@10.0.99.30:backups/nas where the set is copied TO # NAS_SNAPSHOT_ROOT default /mnt/erebor/apps/.zfs/snapshot -# NAS_SUBPATH default jellyfin/config under each snapshot # NAS_SNAPSHOT_TZ default America/Los_Angeles the zone the names are in # NAS_SNAPSHOT_MAX_AGE default 172800 seconds; older fails the run # NAS_KEEP default 7 complete sets to retain @@ -153,15 +158,32 @@ VOL_OFFHOST="${NAS_OFFHOST:-atropos@10.0.99.30:backups/nas}" source "${REPO_ROOT}/scripts/backup-volumes.sh" OUT_DIR="${REPO_ROOT}/backups/nas" -VOL="jellyfin-config" -VOLUMES=("${VOL}") -# verify_remote_archives honours this when set; one archive per set here, so -# it never is. +# One archive per media service with state, each a directory under the +# snapshot. The archive names are backup-volumes.sh's table keys — SENTINEL and +# COMPANIONS carry them, and verify() refuses one that is not there — and the +# service and mount are what the MANIFEST row records, so a restore knows where +# each archive goes back. A set holds every archive listed here, all read from +# ONE snapshot: a set is one instant of the tier, not two. +ARCHIVES=(jellyfin-config navidrome-data) +declare -A ARCHIVE_SUBPATH=( + [jellyfin-config]="jellyfin/config" + [navidrome-data]="navidrome/data" +) +declare -A ARCHIVE_SERVICE=( + [jellyfin-config]="jellyfin" + [navidrome-data]="navidrome" +) +declare -A ARCHIVE_MOUNT=( + [jellyfin-config]="/config" + [navidrome-data]="/data" +) +VOLUMES=("${ARCHIVES[@]}") +# verify_remote_archives honours this when set; a set's archives are read +# from its own MANIFEST here instead, so it never is. ONLY_VOLUMES=() NAS_SSH_TARGET="${NAS_SSH_TARGET:-frodo@10.0.40.30}" NAS_SNAPSHOT_ROOT="${NAS_SNAPSHOT_ROOT:-/mnt/erebor/apps/.zfs/snapshot}" -NAS_SUBPATH="${NAS_SUBPATH:-jellyfin/config}" NAS_SNAPSHOT_TZ="${NAS_SNAPSHOT_TZ:-America/Los_Angeles}" NAS_SNAPSHOT_MAX_AGE="${NAS_SNAPSHOT_MAX_AGE:-172800}" NAS_KEEP="${NAS_KEEP:-7}" @@ -186,15 +208,26 @@ fi [[ ${NAS_SNAPSHOT_ROOT} =~ ^/[A-Za-z0-9._/-]+$ && ${NAS_SNAPSHOT_ROOT} != *..* ]] \ || die "NAS_SNAPSHOT_ROOT must be an absolute path of letters, digits, . _ - and /, got '${NAS_SNAPSHOT_ROOT}'" NAS_SNAPSHOT_ROOT="${NAS_SNAPSHOT_ROOT%/}" -[[ ${NAS_SUBPATH} =~ ^[A-Za-z0-9._-][A-Za-z0-9._/-]*$ && ${NAS_SUBPATH} != *..* && ${NAS_SUBPATH} != */ ]] \ - || die "NAS_SUBPATH must be a relative path of letters, digits, . _ - and /, got '${NAS_SUBPATH}'" +# The subpaths are this file's own, but they are interpolated into the far +# side's shell like the rest, so they are held to the same class. +for a in "${ARCHIVES[@]}"; do + sp="${ARCHIVE_SUBPATH[$a]:-}" + [[ ${sp} =~ ^[A-Za-z0-9._-][A-Za-z0-9._/-]*$ && ${sp} != *..* && ${sp} != */ ]] \ + || die "the subpath for ${a} must be a relative path of letters, digits, . _ - and /, got '${sp}'" + [[ -n ${ARCHIVE_SERVICE[$a]:-} && -n ${ARCHIVE_MOUNT[$a]:-} ]] \ + || die "${a} has no service or mount in this file's tables" +done +unset a sp [[ ${NAS_SNAPSHOT_TZ} =~ ^[A-Za-z0-9/_+-]+$ && -f /usr/share/zoneinfo/${NAS_SNAPSHOT_TZ} ]] \ || die "NAS_SNAPSHOT_TZ is not a zone this host knows: '${NAS_SNAPSHOT_TZ}'" NAS_USER="${NAS_SSH_TARGET%@*}" check_sentinel_table -[[ -n ${SENTINEL[$VOL]:-} ]] \ - || die "no sentinel for ${VOL} in backup-volumes.sh — this script will not write a backup it cannot verify" +for a in "${ARCHIVES[@]}"; do + [[ -n ${SENTINEL[$a]:-} ]] \ + || die "no sentinel for ${a} in backup-volumes.sh — this script will not write a backup it cannot verify" +done +unset a # The ssh to smaug. NOT remote(): that name is backup-volumes.sh's and points # at oracle, and the copy step below relies on it doing so. SSH is that @@ -249,7 +282,8 @@ pick_snapshot() { # --------------------------------------------------------------------------- # Arguments — the shape backup-volumes.sh uses, minus the modes that only make # sense with a compose file: no --hot (a snapshot is the whole point), no -# --inventory, no --project, no --only (one archive per set). +# --inventory, no --project, no --only (a set is one instant of the tier, and +# half of one is not a set). # --------------------------------------------------------------------------- usage() { sed -n 's|^# \{0,1\}||; /^Usage:/,/^$/p' "$0" | head -24; } @@ -329,8 +363,19 @@ case "${MODE}" in if ((${#targets[@]} == 0)) || [[ -z ${targets[0]} ]]; then die "no complete sets in ${OUT_DIR}" fi + # Each set against its OWN MANIFEST, not against ARCHIVES: the sets + # written before Navidrome existed hold jellyfin-config alone, truthfully, + # and stay verifiable until retention retires them. The far-side check + # below reads the MANIFEST the same way. A set whose MANIFEST lists + # nothing is refused rather than passed. for t in "${targets[@]}"; do - verify_set "${t}" || failed=1 + mapfile -t in_set < <(manifest_volumes "${t}") + if ((${#in_set[@]} == 0)); then + red "$(basename "${t}"): the MANIFEST lists no archives" + failed=1 + continue + fi + verify_set "${t}" "${in_set[@]}" || failed=1 done # The far side is checked even when a local set failed — different # findings, different repairs — unless --local-only says oracle may be @@ -388,7 +433,8 @@ fi # has no secrets file — it needs no secrets, and #528 owns whether it ever # gets one — so the file is the observability stack's, whose recipients are # the estate's two keys (ADR-0024). Whoever can open grafana.db is exactly who -# should be able to open Jellyfin's users table, and both are the same class +# should be able to open Jellyfin's and Navidrome's users tables, and all +# three are the same class # of thing. When #528 lands a media file under the same catch-all rule it # carries the same two keys, and NAS_RECIPIENT_STACK=media says so. mapfile -t AGE_RECIPIENTS < <("${REPO_ROOT}/scripts/key-recipients.sh" --list --stack "${NAS_RECIPIENT_STACK}") @@ -402,19 +448,27 @@ nas_remote true \ || die "cannot reach ${NAS_SSH_TARGET} non-interactively — SSH is off on smaug (build-the-nas.md §6.2 turns it on), the 99 → 40:22 pass is not in position, this host's key is not in ${NAS_USER}'s authorized keys, or smaug's host key is not in ~/.ssh/known_hosts" pick_snapshot -SRC="${NAS_SNAPSHOT_ROOT}/${SNAPSHOT}/${NAS_SUBPATH}" # du is the sizing pass and the read preflight in one: it exits non-zero on # any directory it cannot enter, which is the permission fault §6.2 exists to -# prevent, found before a set directory exists rather than after. -kb="$(nas_remote "du -sk '${SRC}'" | awk '{print $1}')" \ - || die "cannot read ${SRC} as ${NAS_USER} — the subpath is wrong, or ${NAS_USER} lacks read on something under it (build-the-nas.md §6.2)" -[[ ${kb} =~ ^[0-9]+$ ]] || die "du on ${SRC} returned '${kb}', not a size" +# prevent, found before a set directory exists rather than after. Every +# archive is sized before any is pulled, so a missing directory for the +# second one fails the run before the first has spent a transfer. +declare -A SRC=() +total_kb=0 +for a in "${ARCHIVES[@]}"; do + SRC[$a]="${NAS_SNAPSHOT_ROOT}/${SNAPSHOT}/${ARCHIVE_SUBPATH[$a]}" + kb="$(nas_remote "du -sk '${SRC[$a]}'" | awk '{print $1}')" \ + || die "cannot read ${SRC[$a]} as ${NAS_USER} — the directory does not exist in ${SNAPSHOT} (is ${ARCHIVE_SERVICE[$a]} deployed, and was it before that snapshot?), or ${NAS_USER} lacks read on something under it (build-the-nas.md §6.2)" + [[ ${kb} =~ ^[0-9]+$ ]] || die "du on ${SRC[$a]} returned '${kb}', not a size" + info "${SRC[$a]} is $(human $((kb * 1024))) uncompressed" + total_kb=$((total_kb + kb)) +done avail_kb="$(df --output=avail -k "${OUT_DIR}" | tail -1 | tr -d ' ')" -if ((avail_kb < kb * 11 / 10)); then - die "only $(human $((avail_kb * 1024))) free at ${OUT_DIR}, and ${SRC} holds $(human $((kb * 1024))) uncompressed — refusing to start" +if ((avail_kb < total_kb * 11 / 10)); then + die "only $(human $((avail_kb * 1024))) free at ${OUT_DIR}, and the set holds $(human $((total_kb * 1024))) uncompressed — refusing to start" fi -info "${SRC} is $(human $((kb * 1024))) uncompressed" +unset a kb STAMP="$(date -u +%Y%m%dT%H%M%SZ)" SET_DIR="${OUT_DIR}/${STAMP}" @@ -422,70 +476,84 @@ SET_DIR="${OUT_DIR}/${STAMP}" mkdir "${SET_DIR}" || die "set ${STAMP} already exists" started=${SECONDS} -part="${SET_DIR}/${VOL}.tar.gz.age.part" -info "archiving ${VOL} from ${NAS_SSH_TARGET}:${SRC}" - -# tar's stdout on smaug is age's stdin here; nothing plaintext touches this -# disk. The pipeline's statuses are read one at a time because they mean -# different things: 255 is ssh itself, and tar's 1 and 2 are the two faults -# the header argues about. -set +e -nas_remote "tar --numeric-owner -czf - -C '${SRC}' ." | age "${AGE_ARGS[@]}" --output "${part}" -rcs=("${PIPESTATUS[@]}") -set -e +declare -A ARCHIVE_BYTES=() ARCHIVE_SHA=() +part="" fail_archive() { red "$*" - rm -f "${part}" + [[ -n ${part} ]] && rm -f "${part}" red "the incomplete set is at ${SET_DIR} — no manifest written, nothing pruned" exit 1 } -case "${rcs[0]}" in - 0) ;; - 255) fail_archive "ssh to ${NAS_SSH_TARGET} failed mid-transfer" ;; - 1) fail_archive "tar reported a file changing while it read ${SRC} — that cannot happen on a snapshot, so this is not one: ${NAS_SNAPSHOT_ROOT} is not a .zfs/snapshot directory, or ${NAS_SUBPATH} is a live path" ;; - 2) fail_archive "tar could not read something under ${SRC} as ${NAS_USER} — a file Jellyfin wrote without world read; build-the-nas.md §6.2 names the fallback" ;; - *) fail_archive "tar on ${NAS_SSH_TARGET} exited ${rcs[0]}" ;; -esac -((rcs[1] == 0)) || fail_archive "age failed to write ${part}" -# Strict, never lenient: --hot's carve-out is for a copy of a live volume, and -# this read a snapshot. A missing sentinel here means the archive is not of -# Jellyfin's config directory, whatever the path was called. -if ! verify "${part}" "${VOL}" 0; then - red "the archive did not verify — the incomplete set is at ${SET_DIR}, no manifest written, nothing pruned" - exit 1 -fi -ARCHIVE_BYTES="$(stat -c %s "${part}")" -ARCHIVE_SHA="$(sha256sum "${part}" | awk '{print $1}')" -mv "${part}" "${SET_DIR}/${VOL}.tar.gz.age" +for a in "${ARCHIVES[@]}"; do + part="${SET_DIR}/${a}.tar.gz.age.part" + info "archiving ${a} from ${NAS_SSH_TARGET}:${SRC[$a]}" + + # tar's stdout on smaug is age's stdin here; nothing plaintext touches this + # disk. The pipeline's statuses are read one at a time because they mean + # different things: 255 is ssh itself, and tar's 1 and 2 are the two faults + # the header argues about. + set +e + nas_remote "tar --numeric-owner -czf - -C '${SRC[$a]}' ." | age "${AGE_ARGS[@]}" --output "${part}" + rcs=("${PIPESTATUS[@]}") + set -e + + case "${rcs[0]}" in + 0) ;; + 255) fail_archive "ssh to ${NAS_SSH_TARGET} failed mid-transfer" ;; + 1) fail_archive "tar reported a file changing while it read ${SRC[$a]} — that cannot happen on a snapshot, so this is not one: ${NAS_SNAPSHOT_ROOT} is not a .zfs/snapshot directory" ;; + 2) fail_archive "tar could not read something under ${SRC[$a]} as ${NAS_USER} — a file ${ARCHIVE_SERVICE[$a]} wrote without world read; build-the-nas.md §6.2 names the fallback" ;; + *) fail_archive "tar on ${NAS_SSH_TARGET} exited ${rcs[0]}" ;; + esac + ((rcs[1] == 0)) || fail_archive "age failed to write ${part}" + + # Strict, never lenient: --hot's carve-out is for a copy of a live volume, + # and this read a snapshot. A missing sentinel here means the archive is not + # of the directory its name says, whatever the path was called. + if ! verify "${part}" "${a}" 0; then + fail_archive "${a} did not verify" + fi + ARCHIVE_BYTES[$a]="$(stat -c %s "${part}")" + ARCHIVE_SHA[$a]="$(sha256sum "${part}" | awk '{print $1}')" + mv "${part}" "${SET_DIR}/${a}.tar.gz.age" + part="" +done +unset a # Written last: its presence is what marks the set complete. The same # tab-separated shape backup-volumes.sh writes, so list_sets, prune and a # reader who knows one knows the other; the three extra keys say what was -# read and when the far side froze it. +# read and when the far side froze it. `source` is the snapshot directory, +# since the archives are its subdirectories and each row names its own mount. { printf '# %s set %s\n' "$(basename "$0")" "${STAMP}" printf 'stack\t%s\n' media printf 'project\t%s\n' media printf 'mode\t%s\n' snapshot - printf 'source\t%s:%s\n' "${NAS_SSH_TARGET}" "${SRC}" + printf 'source\t%s:%s\n' "${NAS_SSH_TARGET}" "${NAS_SNAPSHOT_ROOT}/${SNAPSHOT}" printf 'snapshot\t%s\n' "${SNAPSHOT}" printf 'snapshot_epoch\t%s\n' "${SNAPSHOT_EPOCH}" printf 'recipient\t%s\n' "${AGE_RECIPIENT}" printf 'archiver\t%s\n' "ssh-tar" printf 'downtime\t0\n' printf '#volume\tservice\tmount\tbytes\tsha256\n' - printf '%s\t%s\t%s\t%s\t%s\n' "${VOL}" jellyfin /config "${ARCHIVE_BYTES}" "${ARCHIVE_SHA}" + for a in "${ARCHIVES[@]}"; do + printf '%s\t%s\t%s\t%s\t%s\n' "${a}" "${ARCHIVE_SERVICE[$a]}" "${ARCHIVE_MOUNT[$a]}" "${ARCHIVE_BYTES[$a]}" "${ARCHIVE_SHA[$a]}" + done } > "${SET_DIR}/MANIFEST" +set_bytes=0 +for a in "${ARCHIVES[@]}"; do set_bytes=$((set_bytes + ARCHIVE_BYTES[$a])); done +unset a + # Local retention before the copy, for backup-volumes.sh's reason: a set past # KEEP here is past it there, and copying it first moves what prune_offhost # then removes. prune printf '\n' -green "wrote ${SET_DIR#"${REPO_ROOT}"/} — ${VOL} from ${SNAPSHOT}, $(human "${ARCHIVE_BYTES}"), Jellyfin never stopped, $((SECONDS - started))s total" +green "wrote ${SET_DIR#"${REPO_ROOT}"/} — ${ARCHIVES[*]} from ${SNAPSHOT}, $(human "${set_bytes}"), nothing stopped, $((SECONDS - started))s total" printf '\n' if ((LOCAL_ONLY)); then diff --git a/scripts/backup-volumes.sh b/scripts/backup-volumes.sh index 6a44700d..a23fae11 100755 --- a/scripts/backup-volumes.sh +++ b/scripts/backup-volumes.sh @@ -339,6 +339,14 @@ human() { numfmt --to=iec --suffix=B "$1" 2>/dev/null || printf '%sB' "$1"; } # checkpoints it into a 536 KB main file with no -wal at all. A ZFS snapshot # taken while Jellyfin runs carries all three files (db, -wal, -shm) at one # instant, which is what makes reading one consistent; see COMPANIONS. +# +# Navidrome's (navidrome-data) is the second archive backup-nas.sh pulls, from +# the same snapshot, and for the same reason is declared by no compose file. +# Read off a boot of the pinned image on 2026-09-22 as uid 65534 with +# ND_PLUGINS_ENABLED=false: ./navidrome.db exists within a second of start, +# before the first scan finishes, with a 4 MB -wal and a -shm beside it, and +# ./artwork appears as the scanner extracts covers. Nothing else is written +# there — the cache is a tmpfs on ND_CACHEFOLDER. declare -A SENTINEL=( [prometheus-data]="./chunks_head" [loki-data]="./chunks" @@ -357,6 +365,7 @@ declare -A SENTINEL=( [paperless-db-data]="./18/docker/PG_VERSION" [paperless-broker-data]="./dump.rdb" [jellyfin-config]="./data/jellyfin.db" + [navidrome-data]="./navidrome.db" ) # Reported when absent, never fatal. These cover the fresh-volume case, where @@ -388,6 +397,7 @@ declare -A COMPANIONS=( [paperless-db-data]="./18/docker/base ./18/docker/pg_wal" [paperless-broker-data]="" [jellyfin-config]="./data/jellyfin.db-wal ./config/system.xml ./metadata ./plugins" + [navidrome-data]="./navidrome.db-wal ./artwork" ) # Volumes archived by NOTHING, each with the reason — the third table, and @@ -404,8 +414,9 @@ declare -A COMPANIONS=( # `STACK=media backup-volumes.sh --inventory` say the true thing — nothing in # that file is this script's to archive — instead of dying over a sentinel. # What IS archived from that host is backup-nas.sh's, and it is a bind mount -# on erebor/apps rather than a volume, which is why no jellyfin-config is -# declared there at all (ADR-0045). +# on erebor/apps rather than a volume, which is why no jellyfin-config or +# navidrome-data is declared there at all (ADR-0045). Navidrome's cache is a +# tmpfs, so it needs no entry here. declare -A DISPOSABLE=( [immich-model-cache]="a model cache immich-machine-learning re-downloads on first use" [jellyfin-cache]="transcode scratch and image caches Jellyfin regenerates on demand" diff --git a/stacks/media/.env.example b/stacks/media/.env.example index f44bf6bc..d422e394 100644 --- a/stacks/media/.env.example +++ b/stacks/media/.env.example @@ -5,9 +5,10 @@ # `make up STACK=media` and scripts/render-config.sh do not run on that host # (README.md says why). Secrets do NOT belong here; see secrets/README.md. This # stack has none, by decision (#528; docs/security.md § Secrets): Jellyfin's -# admin account is created through its own setup wizard on first run and lives -# in its database, not in an environment variable, so nothing a secrets file -# held could reach the container. +# admin account is created through its own setup wizard on first run, and +# Navidrome's by `navidrome user create --admin` in the container, and each +# lives in its own database rather than in an environment variable, so nothing +# a secrets file held could reach either container. # The user Jellyfin runs as. Not root — the image declares no user and would # run as root if this were unset, and nothing else in this estate does. @@ -81,6 +82,32 @@ SMART_TEXTFILE_PATH=/mnt/erebor/apps/textfile # backed up, is JELLYFIN_CONFIG_PATH above and not this path. MEDIA_PATH=/mnt/erebor/media +# The user Navidrome runs as. The image declares no user either, so this is +# the line that keeps it off root. It owns NAVIDROME_DATA_PATH below and has +# to be able to read MUSIC_PATH — the same pair, and the same premise, as +# JELLYFIN_UID/GID above. +NAVIDROME_UID=65534 +NAVIDROME_GID=65534 + +# Navidrome's own state — its SQLite database (users, playlists, favourites, +# play counts) and extracted artwork — bind-mounted at /data. +# +# A directory on erebor/apps beside Jellyfin's, for every reason +# JELLYFIN_CONFIG_PATH gives: it is what the nightly snapshot holds and what +# scripts/backup-nas.sh pulls from it. Created `0755`, owned by +# NAVIDROME_UID:NAVIDROME_GID, before the first `up` with this line present +# (build-the-nas.md §6). Its cache is NOT here: it is a tmpfs in compose.yaml, +# so the pull carries no transcodes. +NAVIDROME_DATA_PATH=/mnt/erebor/apps/navidrome/data + +# The music, mounted read-only into Navidrome at /music. +# +# A subdirectory of MEDIA_PATH above — the same unbacked-up dataset, with +# Jellyfin's films beside it — so Navidrome walks the music and nothing else. +# Jellyfin still sees it through /media, which is harmless: the two servers +# read, neither writes. +MUSIC_PATH=/mnt/erebor/media/music + # The address Jellyfin's 8096 is published on. # # 0.0.0.0 on smaug means its one interface on CasaBonita, 10.0.40.30 — the host @@ -105,3 +132,12 @@ JELLYFIN_BIND_ADDR=0.0.0.0 # Everything already on CasaBonita reaches it regardless; docs/security.md # records that as the residual it is. NODE_EXPORTER_BIND_ADDR=0.0.0.0 + +# The address Navidrome's 4533 is published on. +# +# 0.0.0.0 for the reason JELLYFIN_BIND_ADDR gives, and a key of its own for +# the reason NODE_EXPORTER_BIND_ADDR gives: 4533 is published for the phones +# on Hicks, a different consumer from 8096's televisions. Who may reach it +# across the boundary is the fifth pass, `50 → 10.0.40.30:4533` in +# build-the-nas.md §0.5 — not this line. +NAVIDROME_BIND_ADDR=0.0.0.0 diff --git a/stacks/media/README.md b/stacks/media/README.md index cc6e755f..47db562a 100644 --- a/stacks/media/README.md +++ b/stacks/media/README.md @@ -1,6 +1,6 @@ # Media stack -[ADR-0008]'s media tier — the household's media server — on `smaug` +[ADR-0008]'s media tier — the household's media and music servers — on `smaug` (`10.0.40.30`, CasaBonita / VLAN 40), the ThinkServer TS150 that [#413] bought and [ADR-0040] gave TrueNAS. **Deployed 2026-09-19**, from a copy of this directory's two files on the pool, under TrueNAS's own Docker; @@ -18,26 +18,37 @@ fetched the same way and run by TrueNAS's cron — a change to it reaches cd /mnt/erebor/apps/stack && docker compose up -d ``` -`.env.example` names `JELLYFIN_CONFIG_PATH`, a directory on `erebor/apps` -that has to exist, owned by `65534:65534`, before the first `up` — §6 of the -runbook creates it, and says why it is a bind mount and not a volume. +`.env.example` names `JELLYFIN_CONFIG_PATH` and `NAVIDROME_DATA_PATH`, +directories on `erebor/apps` that have to exist, owned by `65534:65534`, +before the first `up` — §6 of the runbook creates them, and says why they are +bind mounts and not volumes. | Service | Image | Port | Purpose | | --- | --- | --- | --- | | jellyfin | `jellyfin/jellyfin` | 8096 (http), on the segment | The media server the televisions reach directly, with Quick Sync hardware transcoding on the E3-1225 v6's HD P630 ([#138], [ADR-0016]) | +| navidrome | `deluan/navidrome` | 4533 (http), on the segment and to Hicks | The music server, over the Subsonic API, for the apps on the phones ([#141]) | | node-exporter | `prom/node-exporter` | 9100 (http), to `10.0.99.20` only | How this host is monitored at all — Prometheus scrapes it, because nothing on this segment may push ([#256], [ADR-0016]); it also serves the SMART textfile a root cron job on the host writes ([#483], [ADR-0047]) | -Two services, and only one of them is the tier. `node-exporter` is here -because of the section below; everything else in this file is about Jellyfin. +Three services, and two of them are the tier. `node-exporter` is here +because of the section below; everything else in this file is about Jellyfin +and Navidrome. -One media service, and that is a decision rather than a starting point. [ADR-0016] +Jellyfin is the one video server, and that is a decision rather than a +starting point. [ADR-0016] builds **Jellyfin alone** and adds Plex only if a screen on 40 turns out to have no working Jellyfin client — the LG OLED, which is the primary screen, has one. Plex authenticates its clients through `plex.tv` even on a local network, which would make a household service that works today depend on a third party staying up and keeping its terms. That is a deferral against a -test nobody has run, not a rejection. Audiobookshelf ([#140]) and Navidrome -([#141]) land here later, in the shape this file already has. +test nobody has run, not a rejection. + +**Navidrome is here for music, which is the weakest thing Jellyfin does** +([#141]). It speaks the Subsonic API, so the clients are a dozen mature apps +on every platform that this repository will never maintain, and it reads the +same pool from `erebor/media/music`, read-only. Its clients are phones, and +the phones are on Hicks, which is why it needs the fifth pass below where +Jellyfin needed none. Audiobookshelf ([#140]) lands here later, in the same +shape. ## Why there is no reverse proxy, and why that is not an omission @@ -61,22 +72,23 @@ every television would have to trust, and a second thing to be down. | Who | Reaches it how | | --- | --- | | Televisions on CasaBonita | Natively, same broadcast domain — the firewall never sees the packet | -| A Hicks workstation | Two of the four rules in [`build-the-nas.md`] §0.5 — `50 → 10.0.40.30:443` and `50 → 10.0.40.30:8096`, one per port | +| A Hicks workstation | Two of the rules in [`build-the-nas.md`] §0.5 — `50 → 10.0.40.30:443` and `50 → 10.0.40.30:8096`, one per port | | Prometheus, on `9100` | A third — `10.0.99.20 → 10.0.40.30:9100` | | Prometheus, on `22` | The fourth — `10.0.99.20 → 10.0.40.30:22`, inert until [`build-the-nas.md`] §6.2 switches SSH on for the backup pull, as `frodo` with one key and read access to `erebor/apps` ([ADR-0045]) | +| A phone on Hicks, on `4533` | The fifth — `50 → 10.0.40.30:4533`, authored for Navidrome and created by §6.5 with its deploy | | Everything else on the estate | Not at all — default deny | [ADR-0012] asks for a named off-host consumer before a port is published, and -here there are three: every screen in the house, one workstation, and the -monitoring host. +here there are four: every screen in the house, one workstation, the phones +and the monitoring host. -Neither published port is private to its consumer, and the reason is the same -for both: everything already on CasaBonita shares this broadcast domain and -reaches them without the firewall seeing a packet. For 8096 that is the whole -point. For 9100 it is a residual — an unauthenticated read of this host's -filesystems, uptime and load, by the televisions — and `docs/security.md` -records it rather than the firewall rule being mistaken for a boundary it is -not. +No published port is private to its consumer, and the reason is the same +for all three: everything already on CasaBonita shares this broadcast domain +and reaches them without the firewall seeing a packet. For 8096 and 4533 that +is the whole point. For 9100 it is a residual — an unauthenticated read of +this host's filesystems, uptime and load, by the televisions — and +`docs/security.md` records it rather than the firewall rule being mistaken for +a boundary it is not. ## Why this host is scraped, and runs no agent @@ -127,7 +139,7 @@ to be rediscovered. ## The backup split -**Jellyfin's state is backed up. The library is not.** That is deliberate, it +**Jellyfin's and Navidrome's state is backed up. The library is not.** That is deliberate, it is what [#138] demanded a deliberate answer on, and it is carried in the volume layout rather than in a policy document: @@ -135,7 +147,9 @@ volume layout rather than in a policy document: | --- | --- | --- | | `${JELLYFIN_CONFIG_PATH}` → `/config` | database, users, **watch history, resume positions**, metadata | **yes** — `scripts/backup-nas.sh`, weekly, from a ZFS snapshot of `erebor/apps` | | `jellyfin-cache` | transcode scratch, image caches | no — regenerable | -| `${MEDIA_PATH}` → `/media` | the library itself | no — see below | +| `${NAVIDROME_DATA_PATH}` → `/data` | Navidrome's database — **users, playlists, favourites, play counts** — and extracted artwork | **yes** — the same pull, the same snapshot, a second archive in the set | +| `/cache` (tmpfs) | Navidrome's transcodes and resized artwork | no — regenerable, and gone on restart | +| `${MEDIA_PATH}` → `/media`, `${MUSIC_PATH}` → `/music` | the library itself | no — see below | [ADR-0008] already ruled the library replaceable and its loss *"annoying rather than catastrophic"*, so backing up 18 TB of re-acquirable files would spend the @@ -151,19 +165,22 @@ backed up ([#484]). [ADR-0045] settles it: `/config` is a **bind mount** on `erebor/apps`, TrueNAS snapshots that dataset nightly, and the monitoring host **pulls** the newest snapshot's copy over the `99 → 40:22` pass — the rule [ADR-0016] wrote for exactly this and nothing else — encrypting it on arrival -with `age` and copying the set on to `oracle` in the same run. Jellyfin is -never stopped; the snapshot is the quiesce. Nothing on this host initiates +with `age` and copying the set on to `oracle` in the same run. Navidrome's +`/data` is a second bind mount on the same dataset and a second archive in the +same set, read out of the same snapshot, so the two are always one instant +([#141]). Neither server is ever stopped; the snapshot is the quiesce. Nothing on this host initiates anything, which is the terminal property [ADR-0016] keeps. [`build-the-nas.md`] §6.2 turns the pull on and §6.3 restores from it. **What is not backed up, on purpose:** `erebor/media`, the library; `jellyfin-cache`, which `scripts/backup-volumes.sh` lists as disposable by -name; and `erebor/ix-apps`, Docker's images and that cache volume. And +name; Navidrome's `/cache`, a tmpfs; and `erebor/ix-apps`, Docker's images and that cache volume. And `scripts/backup-volumes.sh` does not run against this stack at all — `STACK=media` reports nothing to archive, which is the true answer here. -The library is mounted **read-only**. Jellyfin keeps metadata in `/config` by -default, so it has no need to write there — and a media server on the segment +The library is mounted **read-only**, into both servers. Jellyfin keeps +metadata in `/config` by default and Navidrome in its database, so neither has +any need to write there — and a media server on the segment with the game consoles should not be able to delete the thing it serves. ## Where the admin credential lives, and why there is no secrets file @@ -198,17 +215,29 @@ the segment the `50 → 40` passes were made for. Before the wizard has been run at all there is no admin to reset, and `/health` reads `Degraded`, which is why the healthcheck ignores the body. -Audiobookshelf ([#140]) and Navidrome ([#141]) are the same shape — a first -user created through the UI, no admin password from the environment — and -join this section rather than `secrets/` when they land. Their way back in -without the password: Navidrome ships -`navidrome user edit --user --set-password`; Audiobookshelf's root is -reset by editing its user store under `/config`. +**Navidrome is the same class, and joined this section on 2026-09-22** +([#141]). Its admin is created from the TrueNAS shell, not the web form — +`docker exec -it media-navidrome navidrome user create --admin -u -n`, +which prompts for the password — immediately after the first `up` +([`build-the-nas.md`] §6), because until an admin exists the first visitor to +`:4533` is offered the form that creates one. Once one exists the form +answers `403`. The hash is in `navidrome.db` on `NAVIDROME_DATA_PATH`, the +password in the password manager beside Jellyfin's. Its way back in without +the password is the same binary, from the same shell: + +```bash +docker exec -it media-navidrome navidrome user edit -u --set-password -n +``` + +Audiobookshelf ([#140]) is the same shape — a first user created through the +UI, no admin password from the environment — and joins this section rather +than `secrets/` when it lands; its root is reset by editing its user store +under `/config`. ## What was measured rather than assumed -Every non-obvious line in `compose.yaml` came off the pinned image on -2026-09-16, not off upstream's documentation: +Every non-obvious line in `compose.yaml` came off the pinned image, not off +upstream's documentation. Jellyfin's on 2026-09-16: - **It declares no user**, so it would run as root unless told otherwise. Nothing else in this estate does, and neither does this. @@ -224,6 +253,29 @@ Every non-obvious line in `compose.yaml` came off the pinned image on is a restart loop. - **65 MiB idle RSS**, which is what the 2 GiB ceiling is a ceiling over. +Navidrome's on 2026-09-22, from a scratch boot as `65534`, read-only and +cap-dropped, with a generated track in the library: + +- **It declares no user either**, and its `/data` and `/music` are root `755` + rather than `777`. `/data` is a bind mount created `65534`-owned; `/music` + is only read. +- **The image has no `/cache`, and a named volume there kills it.** Docker + makes the mount point root `755`, Navidrome cannot create + `/cache/plugins`, and the process aborts a second after logging that it is + ready. The cache is a tmpfs owned by `65534` instead — and it is off + `/data` on purpose, so the backup does not carry transcodes. +- **`ND_PLUGINS_ENABLED=false`** stops it creating a `0700` + `/data/plugins`, the one directory it wrote that the backup user could + not be assumed to read. +- **`/ping` answers `200` before any admin exists**, so the healthcheck has + no bootstrap problem of the kind Jellyfin's `/health` has. `wget`, `curl`, + `busybox` and `nc` are all present; the healthcheck uses `wget`. +- **An admin made with `navidrome user create --admin` in the running + container closes the web form**: `/auth/createAdmin` answered `403` + afterwards, and the login took the password. +- **49 MiB idle RSS** after the first scan, which is what the 1 GiB ceiling + — covering the 384 MiB tmpfs as well — is a ceiling over. + ## The check that passed [ADR-0040] keeps the media stack in this repository on the strength of Quick diff --git a/stacks/media/compose.yaml b/stacks/media/compose.yaml index dd5b0a83..49b18aeb 100644 --- a/stacks/media/compose.yaml +++ b/stacks/media/compose.yaml @@ -19,12 +19,13 @@ # by TrueNAS's cron rather than by this file (§6.4). # # --------------------------------------------------------------------------- -# Why this stack is one service +# Why this stack is two media services # --------------------------------------------------------------------------- # ADR-0016 decided Jellyfin alone, and Plex only if a screen on 40 turns out to # have no working Jellyfin client — a deferral against a test nobody has run, -# not a rejection. Audiobookshelf (#140) and Navidrome (#141) land here later, -# in the shape this file already has. +# not a rejection. Navidrome (#141) joined it on 2026-09-22 for music, which +# is the weakest thing Jellyfin does. Audiobookshelf (#140) lands here later, +# in the same shape. # # --------------------------------------------------------------------------- # Why there is no reverse proxy here, and why that is not an oversight @@ -195,6 +196,113 @@ services: # figure covers a start that has one to read. start_period: 60s + # --------------------------------------------------------------------------- + # Navidrome — the music server the phones reach, over Subsonic (#141) + # --------------------------------------------------------------------------- + # Jellyfin can serve music and does it worst of everything it does. Navidrome + # speaks the Subsonic API, so the client problem is solved by a dozen mature + # apps this repository will never have to maintain. It reads the same pool + # Jellyfin does, from its own subdirectory, read-only. + # + # MEASURED 2026-09-22 on the pinned image — the lines below that differ from + # Jellyfin's differ because of these: + # + # 1. The image declares NO user, like Jellyfin, so it would run as root. + # Its `/data` and `/music` are `0:0` mode **755**, unlike Jellyfin's + # 777 — which does not matter for /data, a bind mount created 65534-owned + # before first start (build-the-nas.md §6), and does not matter for + # /music, which is only read. + # + # 2. The image has NO /cache, and that is fatal as a named volume: Docker + # creates the mount point root 0755, Navidrome as 65534 cannot create + # `/cache/plugins`, and the process ABORTS a second after reporting + # ready. Hence the uid-owned tmpfs below rather than a volume. + # + # 3. `curl`, `wget`, `busybox` and `nc` are all present — Alpine. The + # healthcheck uses busybox-style wget, the repository's default shape. + # + # 4. `/ping` answers **200** from first boot, BEFORE any admin exists — so + # there is no bootstrap deadlock of the kind Jellyfin's /health has. + # + # 5. It boots read-only as uid 65534 with /tmp and /cache as tmpfs, + # scans a library and answers within about two seconds. + # + # 6. Idle RSS after a first scan: **49 MiB** of anon memory. + # + # 7. Until an admin exists, the first visitor to 4533 gets to CREATE it — + # every console on this segment and every workstation on Hicks. §6 + # closes that window from the TrueNAS shell, immediately after `up`, + # with `navidrome user create --admin`; after that the web route answers + # 403. Nothing is handed in from the environment, which is why this + # stack still has no secrets file (#528). + navidrome: + <<: *service-defaults + image: deluan/navidrome:0.64.1@sha256:df22d661b8d0322c33754d999b567150df9e131a47ae53be71fed0154bf8cef5 + container_name: media-navidrome + user: "${NAVIDROME_UID:?}:${NAVIDROME_GID:?}" + environment: + # The image sets ND_MUSICFOLDER=/music and ND_DATAFOLDER=/data itself; + # they are repeated so the mounts below and the paths the process uses + # are written in one place rather than one here and one in a Dockerfile. + ND_MUSICFOLDER: /music + ND_DATAFOLDER: /data + # Off /data, so the backup pull does not carry a transcode cache — see + # MEASURED (2) for why it is a tmpfs. + ND_CACHEFOLDER: /cache + # Anonymous usage reporting to insights.navidrome.org. CasaBonita has + # internet, and a household music server has no reason to use it for + # this. The log reads `Insight Collector is DISABLED`. + ND_ENABLEINSIGHTSCOLLECTOR: "false" + # WASM plugins, which nothing here uses. Off, it stops Navidrome + # creating a 0700 /data/plugins — a directory the backup user would + # have to be proven able to read — and a wazero compile cache. + ND_PLUGINS_ENABLED: "false" + # A ceiling to catch a leak, not a measurement, on the same terms as + # Jellyfin's. Over 49 MiB idle, with room for a full scan of a real library + # and for the /cache tmpfs below, whose pages are charged to this limit. + mem_limit: 1g + memswap_limit: 1g + cap_drop: [ALL] + security_opt: + - no-new-privileges:true + read_only: true + tmpfs: + - /tmp:size=64m + # Transcodes, resized artwork and background images: Navidrome caps + # each at 100 MB by default, and all of it regenerates. A tmpfs rather + # than a volume because of MEASURED (2), and owned by the user: line + # because a tmpfs is root's otherwise. + - /cache:size=384m,uid=65534,gid=65534,mode=0700 + volumes: + # Navidrome's own state: its SQLite database — users, playlists, + # favourites, play counts — and extracted artwork. Jellyfin's watch + # history for music, and backed up the same way: a bind mount on + # erebor/apps that scripts/backup-nas.sh reads out of the dataset's + # newest snapshot (ADR-0045). Created 0755 65534:65534 before first + # start (§6). + - ${NAVIDROME_DATA_PATH:?}:/data + # READ-ONLY, for Jellyfin's reason: a server on the segment with the game + # consoles should not be able to delete what it serves. Navidrome never + # writes to the library unless told to, and the watcher it starts on + # /music works through a read-only mount — measured. + - ${MUSIC_PATH:?}:/music:ro + # PUBLISHED, with a named off-host consumer as ADR-0012 asks: the Subsonic + # apps on the phones, which live on Hicks, through the fifth pass in + # build-the-nas.md §0.5 (`50 → 10.0.40.30:4533`). Everything on CasaBonita + # reaches it natively, as it reaches 8096. + ports: + - "${NAVIDROME_BIND_ADDR:?}:4533:4533" + healthcheck: + # `/ping` and the status code, which is 200 before and after the admin + # exists — see MEASURED (4). + test: ["CMD", "wget", "--spider", "-q", "http://localhost:4533/ping"] + interval: 30s + timeout: 5s + retries: 3 + # About two seconds on a first boot. A start with a large library to + # scan still answers /ping first; the scan runs behind it. + start_period: 30s + # --------------------------------------------------------------------------- # node_exporter — how this host is monitored at all (#256, ADR-0016) # --------------------------------------------------------------------------- @@ -359,9 +467,10 @@ networks: - subnet: 172.28.40.0/24 volumes: - # ONE volume, and it is the one that is not backed up. Jellyfin's state is - # the bind mount above, on erebor/apps, so that it can be snapshotted and - # pulled — see the mount comment and ADR-0045. What is declared here is + # ONE volume, and it is the one that is not backed up. Jellyfin's state and + # Navidrome's are bind mounts above, on erebor/apps, so that they can be + # snapshotted and pulled — see the mount comments and ADR-0045. Navidrome's + # cache is a tmpfs, not a volume, for the reason its MEASURED (2) gives. What is declared here is # transcode scratch and regenerable image caches, and scripts/backup-volumes.sh # lists it as DISPOSABLE by name, for the reason stacks/sensitive gives for # immich-model-cache; `STACK=media backup-volumes.sh --inventory` therefore diff --git a/systemd/homelab-backup-nas.service b/systemd/homelab-backup-nas.service index a15bc04c..f6cf208c 100644 --- a/systemd/homelab-backup-nas.service +++ b/systemd/homelab-backup-nas.service @@ -1,4 +1,5 @@ -# Weekly pull of Jellyfin's state off smaug (#484, ADR-0045). +# Weekly pull of Jellyfin's and Navidrome's state off smaug (#484, #141, +# ADR-0045). # # WHY IT IS ITS OWN UNIT AND NOT A ROW IN homelab-backup-volumes. That unit # stops the observability stack and archives volumes on this host's docker @@ -14,7 +15,7 @@ # # See homelab-backup-volumes.service for why each directive below is set. [Unit] -Description=HomeLab: pull Jellyfin's state off smaug from its newest snapshot +Description=HomeLab: pull the media tier's state off smaug from its newest snapshot Documentation=file:///home/robo/code/Gerrrt/HomeLab/docs/runbooks/schedule-maintenance.md Wants=network-online.target After=network-online.target diff --git a/systemd/homelab-backup-nas.timer b/systemd/homelab-backup-nas.timer index 22f1cf52..734ea0bc 100644 --- a/systemd/homelab-backup-nas.timer +++ b/systemd/homelab-backup-nas.timer @@ -1,5 +1,5 @@ [Unit] -Description=HomeLab: weekly pull of Jellyfin's state off smaug +Description=HomeLab: weekly pull of the media tier's state off smaug Documentation=file:///home/robo/code/Gerrrt/HomeLab/docs/runbooks/schedule-maintenance.md [Timer] From 702fb3a67c80b8f8456b4293e892ce2e0965f853 Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Tue, 22 Sep 2026 22:32:23 +0000 Subject: [PATCH 2/2] docs(media): record the fifth pass to smaug as created MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Allow 4533 to smaug now exists on Hicks (opt2 / igc0.50), directly under Allow 8096 to smaug and above Block access to CasaBonita. It was inserted from morpheus's shell as a copy of the 8096 rule with the port changed, and saved as a config revision named for #141. `pfctl -sr -vv` reads https, 8096, 4533 and then the block on igc0.50. The monitoring host is refused on 4533, and the igc0.40 tripwire reads 0 packets. The rule came before the service because a pass to a port with no listener opens nothing. The rest of §6.5 still waits on #558. Refs #141 Co-Authored-By: Claude Opus 5.5 --- docs/architecture.md | 2 +- docs/changelog.md | 4 +++- docs/network.md | 11 ++++++----- docs/roadmap.md | 4 ++-- docs/runbooks/build-the-nas.md | 17 +++++++++++------ docs/security.md | 9 +++++---- stacks/media/README.md | 2 +- 7 files changed, 29 insertions(+), 20 deletions(-) diff --git a/docs/architecture.md b/docs/architecture.md index c16208b1..72a16fd4 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -191,7 +191,7 @@ hole from the monitoring VLAN into the monitored one. | `phoenix` (10.0.30.70) | 🟢 30 | *(none — a toolchain host, no Docker)* | The deployment host, built 2026-09-20: a Proxmox API token, an SSH key and a checkout, so that the Packer, OpenTofu and Ansible work after [#436](https://github.com/Gerrrt/HomeLab/issues/436) has somewhere to run from. A guest on `Saruman`, placed by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), which also decides that the estate's CA key stays on `prometheus` and does not follow the toolchain here. It holds no age key and converges nothing; it is the WireGuard endpoint of [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md), built 2026-09-22 ([#442](https://github.com/Gerrrt/HomeLab/issues/442)); the one thing it is meant to reach that no other guest does is `8006` on `Saruman`, by a single host-firewall rule — written 2026-09-20, when [#566](https://github.com/Gerrrt/HomeLab/issues/566) enabled the firewall the build had found disabled and narrowed the `local_network` alias that was admitting the whole segment underneath it. Its Alloy pushes to `alexander`, never to VLAN 99; the build is [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). Alloy agent (native package, `scripts/deploy-agent.sh`) | | `oracle` (10.0.99.30) | 🔴 99 | *(none — hand-run containers)* | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copies of the firewall export (`make backup-firewall`), of the weekly volume sets (`make backup`, [#535](https://github.com/Gerrrt/HomeLab/issues/535)) and of Jellyfin's state pulled off `smaug` (`make backup-nas`, [ADR-0045](adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md)). The estate's host for small off-host jobs — [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) | | `trinity` (10.0.99.40) | 🔴 99 | [`stacks/sensitive`](../stacks/sensitive) | **Not built yet** — ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md), after the firewall restore is rehearsed on it ([#404](https://github.com/Gerrrt/HomeLab/issues/404)). The foundation is authored: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github.com/Gerrrt/HomeLab/issues/129), [#130](https://github.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github.com/Gerrrt/HomeLab/issues/134)), Immich — four containers behind Caddy with a memory limit on each ([#132](https://github.com/Gerrrt/HomeLab/issues/132)) — Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github.com/Gerrrt/HomeLab/issues/131)) are authored as well; the rest follow. Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s | -| `smaug` (10.0.40.30) | 🟡 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **Built, pooled and deployed**: TrueNAS on its boot SSD, the static above since 2026-09-16, the four inbound rules verified in position, the mirror `erebor` since 2026-09-18 and this stack running on it since 2026-09-19 — from a copy of the compose file on the pool, brought up with `docker compose` under TrueNAS's own Docker ([`build-the-nas.md`](runbooks/build-the-nas.md) §6). Jellyfin, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github.com/Gerrrt/HomeLab/issues/138)); and Navidrome for music, publishing 4533 for the Subsonic apps on the phones on Hicks through a fifth pass — authored in the stack and deployed by [`build-the-nas.md`](runbooks/build-the-nas.md) §6.5 once the mirror is whole ([#141](https://github.com/Gerrrt/HomeLab/issues/141)). Scraped by `prometheus` on `9100`; it pushes nothing, and runs no Alloy — the estate's first scraped host, and the reason [#256](https://github.com/Gerrrt/HomeLab/issues/256) was more than a line of YAML. That issue settled the fork TrueNAS opened in it: `node_exporter`, as a digest-pinned container in this stack rather than TrueNAS's own endpoint, so the existing `99 → 40:9100` pass, the `host-overview` dashboard and seven rules in `host.rules.yaml` all keep working unchanged. The `node` job and `prometheus/targets/node.yaml` are live, and the target with them since 2026-09-19, once the exporter answered from the pool. SMART reaches the same scrape by [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md): a root cron job in TrueNAS's UI runs the estate's collector and the exporter serves the file — still no agent, still nothing initiating from 40 — and patch state is deliberately not collected on an appliance ([#483](https://github.com/Gerrrt/HomeLab/issues/483)) | +| `smaug` (10.0.40.30) | 🟡 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **Built, pooled and deployed**: TrueNAS on its boot SSD, the static above since 2026-09-16, the four inbound rules verified in position, the mirror `erebor` since 2026-09-18 and this stack running on it since 2026-09-19 — from a copy of the compose file on the pool, brought up with `docker compose` under TrueNAS's own Docker ([`build-the-nas.md`](runbooks/build-the-nas.md) §6). Jellyfin, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github.com/Gerrrt/HomeLab/issues/138)); and Navidrome for music, publishing 4533 for the Subsonic apps on the phones on Hicks through a fifth pass, created 2026-09-22 — authored in the stack and deployed by [`build-the-nas.md`](runbooks/build-the-nas.md) §6.5 once the mirror is whole ([#141](https://github.com/Gerrrt/HomeLab/issues/141)). Scraped by `prometheus` on `9100`; it pushes nothing, and runs no Alloy — the estate's first scraped host, and the reason [#256](https://github.com/Gerrrt/HomeLab/issues/256) was more than a line of YAML. That issue settled the fork TrueNAS opened in it: `node_exporter`, as a digest-pinned container in this stack rather than TrueNAS's own endpoint, so the existing `99 → 40:9100` pass, the `host-overview` dashboard and seven rules in `host.rules.yaml` all keep working unchanged. The `node` job and `prometheus/targets/node.yaml` are live, and the target with them since 2026-09-19, once the exporter answered from the pool. SMART reaches the same scrape by [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md): a root cron job in TrueNAS's UI runs the estate's collector and the exporter serves the file — still no agent, still nothing initiating from 40 — and patch state is deliberately not collected on an appliance ([#483](https://github.com/Gerrrt/HomeLab/issues/483)) | | `bahamut` (10.0.30.50) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 domain controller, PDC emulator and DNS for `ad.matrix.elysium` — Tier 0. Static, because every member finds a DC through DNS and the DCs *are* the DNS. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | | `leviathan` (10.0.30.51) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 second domain controller and DNS — Tier 0. Static, for the same reason. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | | `titan` (10.0.30.52) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 file and member server — the shares, and the NTLM relay target that only exists because 2025 requires outbound SMB signing and not inbound — Tier 1. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | diff --git a/docs/changelog.md b/docs/changelog.md index 9268308d..58412d1b 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -37,7 +37,9 @@ docstring gives: it is a record, not a claim about now. with `navidrome user create --admin` from the TrueNAS shell closes the web form that would otherwise hand the admin to the first visitor — so the stack still has no secrets file (#528). Not deployed: the mirror is one - disk until [#558](https://github.com/Gerrrt/HomeLab/issues/558). + disk until [#558](https://github.com/Gerrrt/HomeLab/issues/558). The + fifth pass was created the same day, ahead of it, and read in position + from `morpheus` with the `igc0.40` tripwire still at zero. - **[#455](https://github.com/Gerrrt/HomeLab/issues/455) The off-estate drive is bought, and buying it settles less than the row implied.** A WD Elements diff --git a/docs/network.md b/docs/network.md index cfe3857f..beae53f4 100644 --- a/docs/network.md +++ b/docs/network.md @@ -341,11 +341,12 @@ Televisions and consoles. Internet only. on `9100` and `22` — four host-scoped, port-scoped passes above *Block access to CasaBonita* on their interfaces. Everything else on every other segment is still refused, and the televisions need no rule at all because they share this - broadcast domain with the server. **A fifth is authored and not created:** - Hicks → `10.0.40.30:4533`, for the Subsonic apps on the phones to reach - Navidrome ([#141](https://github.com/Gerrrt/HomeLab/issues/141)), in - [`build-the-nas.md`](runbooks/build-the-nas.md) §0.5 and made by §6.5 on - the day Navidrome is deployed. Until then, four is the count. + broadcast domain with the server. **A fifth since 2026-09-22:** Hicks → + `10.0.40.30:4533`, for the Subsonic apps on the phones to reach Navidrome + ([#141](https://github.com/Gerrrt/HomeLab/issues/141)), created ahead of + the service and verified in position from `morpheus` + ([`build-the-nas.md`](runbooks/build-the-nas.md) §0.5, §6.5). Until + Navidrome is deployed it matches nothing, because nothing listens there. - **What answers on `9100` is `node_exporter`**, which makes this the one host in the estate that Prometheus *scrapes* rather than is pushed to ([#256](https://github.com/Gerrrt/HomeLab/issues/256), diff --git a/docs/roadmap.md b/docs/roadmap.md index ae04350e..4ed770dd 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -174,8 +174,8 @@ workstation can mount the share. - **[#140](https://github.com/Gerrrt/HomeLab/issues/140) Audiobookshelf and [#141](https://github.com/Gerrrt/HomeLab/issues/141) Navidrome** have no gate but the mirror being whole, which is #558. Navidrome is authored in - `stacks/media` and waits only on that; deploying it is - [`build-the-nas.md`](runbooks/build-the-nas.md) §6.5, fifth pass included. + `stacks/media`, its fifth pass exists, and it waits only on that; deploying + it is the rest of [`build-the-nas.md`](runbooks/build-the-nas.md) §6.5. ## Saruman: the domain, then the SOC diff --git a/docs/runbooks/build-the-nas.md b/docs/runbooks/build-the-nas.md index dc07a380..a682e6ca 100644 --- a/docs/runbooks/build-the-nas.md +++ b/docs/runbooks/build-the-nas.md @@ -238,8 +238,8 @@ unambiguous about which rule answered. **Set these descriptions exactly**; they are load-bearing in the next section, not decoration. **The fifth came later, for Navidrome** ([#141](https://github.com/Gerrrt/HomeLab/issues/141)). -The four above were created on 2026-09-16; `Allow 4533 to smaug` is added by -§6.5, when Navidrome is. Its consumer is the Subsonic apps on the phones, and +The four above were created on 2026-09-16; `Allow 4533 to smaug` on +2026-09-22, by §6.5 step 1, ahead of the rest of that section. Its consumer is the Subsonic apps on the phones, and the phones are on Hicks, so it is the `8096` rule's shape on another port — [ADR-0012](../adr/0012-publish-only-ports-with-an-off-host-consumer.md)'s named off-host consumer, and nothing wider. The issue that proposed Navidrome said no @@ -281,8 +281,7 @@ pfctl -sr -vv \ Each pass must appear **above** the *Block access to CasaBonita* rule on its own interface: `Allow 9100`/`Allow SSH` before the block on `igc0.99`, and -`Allow HTTPS`/`Allow 8096`/`Allow 4533` before it on `igc0.50` — the last -once §6.5 has created it. **Read the order, not the numbers.** `-vv` numbers each ruleset from zero rather than counting output +`Allow HTTPS`/`Allow 8096`/`Allow 4533` before it on `igc0.50`. **Read the order, not the numbers.** `-vv` numbers each ruleset from zero rather than counting output lines, so its `@` indices match neither `pfctl -sr | grep -n` nor anything written down here — they are a printing artefact, and only the sequence is a fact about the firewall. @@ -563,7 +562,7 @@ ls -1 /mnt/erebor/apps/.zfs/snapshot/ > > **No workstation can mount this share, and that was found by trying.** The > Hicks rules from §0.5 pass `443` and `8096` to `smaug` and nothing else -> (and `4533` since §6.5, which changes nothing here); SMB is `445`, so a +> (and `4533` since 2026-09-22, which changes nothing here); SMB is `445`, so a > Hicks machine that reaches the TrueNAS UI and Jellyfin gets nothing from `\\10.0.40.30\media`. Only devices already on > CasaBonita can mount it, and those are televisions. Getting a film onto the > library today means the console shell — `mkdir` and `curl` under @@ -1186,7 +1185,13 @@ but it is still a missed week. make backup-nas && make backup-nas ARGS=--list && make verify-backups ``` -> **Not yet done.** Authored 2026-09-22. Waits on #558. +> **Step 1 done 2026-09-22**, ahead of the gate — a pass to a port nothing +> listens on opens nothing. Inserted directly beneath `Allow 8096 to smaug` +> on Hicks from `morpheus`'s shell rather than the UI, as a copy of that rule +> with the port changed, and saved as a config revision named for #141. +> `pfctl -sr -vv` then read `https`, `8096`, `4533` and the block, in that +> order, on `igc0.50`; the monitoring host was refused on `4533`; the +> `igc0.40` tripwire read **0 packets**. Steps 2–7 wait on #558. ## §7 — Verify diff --git a/docs/security.md b/docs/security.md index 031dd2d9..80c72859 100644 --- a/docs/security.md +++ b/docs/security.md @@ -275,10 +275,11 @@ existed. The residual it leaves, accepted, is one more service on the NAS with one more key that reads it — a key that lives on the host already holding the estate's age identity, and reads a directory that includes Jellyfin's users' password hashes. -A fifth is written and not yet made: Hicks → `10.0.40.30:4533`, for -Navidrome's phone clients ([#141](https://github.com/Gerrrt/HomeLab/issues/141)), -the `8096` rule's shape on one more port, created by -[`build-the-nas.md`](runbooks/build-the-nas.md) §6.5 with the deploy. It +A fifth exists since 2026-09-22: Hicks → `10.0.40.30:4533`, for Navidrome's +phone clients ([#141](https://github.com/Gerrrt/HomeLab/issues/141)), the +`8096` rule's shape on one more port, verified in position from `morpheus` +and made ahead of the service ([`build-the-nas.md`](runbooks/build-the-nas.md) +§6.5), so until Navidrome is deployed it opens a port nothing answers on. It adds no residual `8096` did not already have: Navidrome's first admin is made from the TrueNAS shell before the port is ever reachable from Hicks, and the library behind it is mounted read-only. diff --git a/stacks/media/README.md b/stacks/media/README.md index 47db562a..eb794ac7 100644 --- a/stacks/media/README.md +++ b/stacks/media/README.md @@ -75,7 +75,7 @@ every television would have to trust, and a second thing to be down. | A Hicks workstation | Two of the rules in [`build-the-nas.md`] §0.5 — `50 → 10.0.40.30:443` and `50 → 10.0.40.30:8096`, one per port | | Prometheus, on `9100` | A third — `10.0.99.20 → 10.0.40.30:9100` | | Prometheus, on `22` | The fourth — `10.0.99.20 → 10.0.40.30:22`, inert until [`build-the-nas.md`] §6.2 switches SSH on for the backup pull, as `frodo` with one key and read access to `erebor/apps` ([ADR-0045]) | -| A phone on Hicks, on `4533` | The fifth — `50 → 10.0.40.30:4533`, authored for Navidrome and created by §6.5 with its deploy | +| A phone on Hicks, on `4533` | The fifth — `50 → 10.0.40.30:4533`, for Navidrome, created 2026-09-22 (§6.5 step 1) | | Everything else on the estate | Not at all — default deny | [ADR-0012] asks for a named off-host consumer before a port is published, and