From 1b3a62cb65303f774a19c290e05790e107feed4d Mon Sep 17 00:00:00 2001 From: Garrett Allen <98648590+Gerrrt@users.noreply.github.com> Date: Tue, 22 Sep 2026 22:23:07 +0000 Subject: [PATCH] feat(media): author Audiobookshelf beside Jellyfin, behind a fifth Hicks pass MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Audiobookshelf joins stacks/media for the one thing Jellyfin does not do: listening progress that follows a person between devices. It is authored and validated here; deploying it on smaug is build-the-nas.md §6.5, which waits on the mirror being whole (#558). The issue said no rule beyond #138's 50→40 would be needed. There is no such rule - Hicks reaches smaug on 443 and 8096, one pass per port - and the phones are on Hicks, so the service brings a fifth pass, `Allow 13378 to smaug`, created when it is deployed. ADR-0050 records that, audiobooks only with podcasts deferred behind a read-only library, and the backup shape. Measured on the pinned image first: it binds port 80 by default, which a capability-less non-root process cannot (PORT=13378); /config and /metadata do not exist in the image; its entrypoint is tini (init: false, as for Immich); the database is in rollback-journal mode; 91 MiB idle. The admin reset was proved on a scratch boot, not described. backup-nas.sh now pulls a table of archives from one snapshot instead of one path, so Audiobookshelf's database leaves the NAS with Jellyfin's. Its row is `pending` until §6.5's Done commit flips it: skipped by name while its directory is absent, so merging this does not fail Saturday's Jellyfin pull. Verification reads each set against its own MANIFEST, so the retained Jellyfin-only sets still verify - both were re-verified, and a bench pull on oracle covered the skip, a missing sentinel and two archives. backup-volumes.sh's compose parser learns that a ${VAR}-spelled source is a host path, which it already was. Refs #140 Co-Authored-By: Claude Opus 5.5 --- .github/dependabot.yml | 6 +- .gitignore | 2 +- Makefile | 9 +- README.md | 6 +- ...ita-inward-and-keep-it-terminal-outward.md | 6 + ...ellyfins-state-from-a-snapshot-over-ssh.md | 10 + ...he-media-tier-behind-a-fifth-hicks-pass.md | 132 ++++++++++ docs/architecture.md | 4 +- docs/changelog.md | 32 +++ docs/network.md | 9 +- docs/observability.md | 2 +- docs/roadmap.md | 11 +- docs/runbooks/build-the-nas.md | 150 ++++++++++- docs/runbooks/restore-the-stack.md | 13 +- docs/runbooks/schedule-maintenance.md | 3 +- docs/runbooks/successor-handover.md | 4 +- docs/security.md | 28 ++- scripts/backup-nas.sh | 238 ++++++++++++------ scripts/backup-volumes.sh | 26 +- stacks/media/.env.example | 38 ++- stacks/media/README.md | 98 ++++++-- stacks/media/compose.yaml | 124 ++++++++- systemd/homelab-backup-nas.service | 5 +- systemd/homelab-backup-nas.timer | 2 +- 24 files changed, 802 insertions(+), 156 deletions(-) create mode 100644 docs/adr/0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 38ce99a4..e9ce74b7 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -84,9 +84,9 @@ updates: # README repeat the claim. The pins and make validate did reach it. This is # the third. # - # Two images now, not one: #256 added a node_exporter beside Jellyfin, because - # this host is scraped rather than running an agent. They share nothing, so no - # groups — each bump is its own PR. + # Three images now: #256 added a node_exporter beside Jellyfin, because this + # host is scraped rather than running an agent, and #140 added Audiobookshelf, + # from ghcr.io. They share nothing, so no groups — each bump is its own PR. - package-ecosystem: docker-compose directory: /stacks/media schedule: diff --git a/.gitignore b/.gitignore index 11ed4971..f8cfdf15 100644 --- a/.gitignore +++ b/.gitignore @@ -48,7 +48,7 @@ stacks/*/snmp-exporter/mibs/ # `make backup` writes age-encrypted volume archives to backups/volumes//, # and `make restore` writes a pre-restore snapshot of what it is about to # replace to backups/volumes/.pre-restore-/. `make backup-nas` writes -# the age-encrypted pull of Jellyfin's state off smaug to backups/nas//. +# the age-encrypted pull of the media tier's state off smaug to backups/nas//. # `make backup-firewall` writes the SOPS-encrypted pfSense config to # backups/firewall/. None of it belongs in git. # diff --git a/Makefile b/Makefile index 79d95231..4b1bf890 100644 --- a/Makefile +++ b/Makefile @@ -722,13 +722,14 @@ backup: ## Quiesce the stack, archive its volumes to ./backups/, verify, copy to STACK=$(STACK) ./scripts/backup-volumes.sh $(ARGS) .PHONY: backup-nas -backup-nas: ## Pull Jellyfin's state off smaug from its newest ZFS snapshot, encrypt and verify +backup-nas: ## Pull the media tier's state off smaug from its newest ZFS snapshot, encrypt and verify @# The one backup that leaves this host to FETCH rather than to deliver. @# smaug cannot run backup-volumes.sh — no age, no checkout, no key, and @# ADR-0016 forbids it initiating anything upward — so this host reads a @# snapshot of erebor/apps over the 99 → 40:22 pass and encrypts what - @# arrives here (ADR-0045). Jellyfin is never stopped: the snapshot is the - @# quiesce. NAS_KEEP and not KEEP, for the reason backup-firewall gives + @# arrives here (ADR-0045) — Jellyfin's and Audiobookshelf's state, one + @# archive each, from one snapshot (ADR-0050). Nothing on smaug is stopped: + @# the snapshot is the quiesce. NAS_KEEP and not KEEP, for the reason backup-firewall gives @# for FW_KEEP. Sets land in backups/nas/, apart from the volume sets, and @# `make verify-backups` reads both. The copy to oracle is a step of this @# target too, by the helpers `backup` gained under #535 — same far-side @@ -740,7 +741,7 @@ backup-nas: ## Pull Jellyfin's state off smaug from its newest ZFS snapshot, enc verify-backups: ## Re-verify every retained set of both kinds: the volume sets and the NAS set @# What homelab-verify-backups.timer runs nightly. Two directories, one @# job: backups/volumes/ is verified against the stack's derived volume - @# list and backups/nas/ against its own, and a media set in the volume + @# list and backups/nas/ each set against its own MANIFEST, and a media set in the volume @# directory would fail the first — which is why they are apart, and why @# one target walks both rather than a second unit doing the second half. @# Both halves run even when the first fails, so one morning's journal diff --git a/README.md b/README.md index 188f2f8d..9113e240 100644 --- a/README.md +++ b/README.md @@ -98,7 +98,7 @@ documents for different readers. the Makefile, the scripts, the workflow and the runbooks resolves its image from `compose.yaml` too, so an image that is not pinned there cannot be run at all. -- **Documented decisions and runbooks.** 49 ADRs covering what was chosen +- **Documented decisions and runbooks.** 50 ADRs covering what was chosen and what was rejected — including the costs accepted knowingly; 34 runbooks for the operations that are easy to get wrong at 1am, one of which is the handover page a successor reads first. @@ -178,7 +178,7 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md). | Firewall / routing | [pfSense on FreeBSD 16](docs/network.md) | VLANs, DHCP, default-deny | | Virtualisation | Proxmox VE | Lab hypervisor | | Storage | [TrueNAS 25.10](docs/runbooks/build-the-nas.md) | `smaug`: 2× 18 TB ZFS mirror `erebor`, the SMB share, and the Docker the media stack runs under | -| Media | [Jellyfin](stacks/media) | Quick Sync transcoding on the NAS; the one stack deployed from TrueNAS rather than by `make deploy` | +| Media | [Jellyfin, Audiobookshelf](stacks/media) | Quick Sync transcoding on the NAS, and audiobooks with synced progress (authored, not yet deployed — [#140](https://github.com/Gerrrt/HomeLab/issues/140)); the one stack deployed from TrueNAS rather than by `make deploy` | | Lab observability | [Prometheus, Loki, Grafana](stacks/lab) | On `alexander`, a guest on `Saruman`, with its own Prometheus; only liveness crosses to the estate's, never telemetry | | Metrics | [Prometheus](stacks/observability/prometheus) | 30-day retention capped at 12 GiB, remote-write receiver | | Logs | [Loki](stacks/observability/loki) | Single-binary, filesystem storage | @@ -371,7 +371,7 @@ self-test, and the card is set to test itself every fortnight switch between the monitoring host and the network draws from it too, racked in U4 ([#110](https://github.com/Gerrrt/HomeLab/issues/110)); and the config export leaves the monitoring host nightly, the volume backup sets weekly -([#535](https://github.com/Gerrrt/HomeLab/issues/535)) and Jellyfin's state +([#535](https://github.com/Gerrrt/HomeLab/issues/535)) and the media tier's state weekly by way of it ([#484](https://github.com/Gerrrt/HomeLab/issues/484)), so the rehearsal is what is left ([#92](https://github.com/Gerrrt/HomeLab/issues/92)). diff --git a/docs/adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md b/docs/adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md index 9784b730..53028251 100644 --- a/docs/adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md +++ b/docs/adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md @@ -35,6 +35,12 @@ > TrueNAS has no `age`, a key on the NAS would buy nothing, and the property > this section wanted holds more strongly with no archive on the NAS at all. > The direction — pulled by `prometheus` over `99 → 40:22` — is as written. +> +> **A fifth rule, 2026-09-22.** Audiobookshelf joins the media tier and its +> clients are phones on Hicks, so it gets `vlan50 net → 10.0.40.30:13378`, a +> pass of the same kind as the table below, created when it is deployed +> ([ADR-0050](0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md)). The table is not edited; the +> rule set the runbook keeps is the current one. ## Context diff --git a/docs/adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md b/docs/adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md index 6a8d057e..ceef39c9 100644 --- a/docs/adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md +++ b/docs/adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md @@ -6,6 +6,16 @@ writes the first snapshot schedule decision 4 of [ADR-0040](0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md) asked for +> [!NOTE] +> **The pull reads a list, since 2026-09-22.** Audiobookshelf's state joins +> Jellyfin's as a second archive in the same set, read from the same +> snapshot, and `backup-nas.sh` now reads a table of archive → subpath where +> this ADR describes one path; a service authored before it is deployed is +> `pending` in that table and skipped by name while its directory is absent +> ([ADR-0050](0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md)). Everything below — the +> direction, the user, the snapshot as quiesce, encryption on arrival — is +> unchanged. + ## Context [`build-the-nas.md`](../runbooks/build-the-nas.md) §4 splits the pool into diff --git a/docs/adr/0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md b/docs/adr/0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md new file mode 100644 index 00000000..68e6d358 --- /dev/null +++ b/docs/adr/0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md @@ -0,0 +1,132 @@ +# ADR-0050: Add Audiobookshelf to the media tier behind a fifth Hicks pass, and pull its state with Jellyfin's + +**Status:** Accepted · 2026-09 · adds a service to the tier +[ADR-0008](0008-place-services-by-data-trust.md) created, a rule to the four +[ADR-0016](0016-open-casabonita-inward-and-keep-it-terminal-outward.md) became, +and a second archive to the pull +[ADR-0045](0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md) built; +decides [#140](https://github.com/Gerrrt/HomeLab/issues/140) + +## Context + +[#140](https://github.com/Gerrrt/HomeLab/issues/140) proposed Audiobookshelf, +an audiobook server with its own phone apps, for one property Jellyfin lacks: +**listening progress that follows a person between devices.** Jellyfin tracks +position the way it tracks a film, and nobody listens to a twenty-hour book in +one room. + +ADR-0008 does not name it. Its test is the trust of the data, not the +function, and an audiobook library is the same kind of data as the films on +the same pool: replaceable, and annoying rather than catastrophic to lose. So +the placement is not in question. The issue said it was a small addition to a +box being built anyway, and that it needed **"no new rule beyond the 50→40 +that #138 already adds"**. + +That last clause is wrong, and it is the reason this is an ADR rather than a +changelog line. There is no 50→40 rule. There are two Hicks passes, one per +port — `Allow HTTPS to smaug` on `443` and `Allow 8096 to smaug` — +host-scoped and port-scoped, as ADR-0016 asked of every rule into CasaBonita, +and as [`build-the-nas.md`](../runbooks/build-the-nas.md) §0.5 created them on +2026-09-16. The SMB share found the same edge by trying on 2026-09-18 +([#523](https://github.com/Gerrrt/HomeLab/issues/523)). Audiobookshelf listens +on `13378`. And Jellyfin's argument for publishing to its segment was that its +clients are televisions *on* that segment; Audiobookshelf's clients are phones, +and the phones are on Hicks ([`network.md`](../network.md)). **This is the +first service on the NAS whose only real consumers are across a segment +boundary.** + +The backup has the same shape of problem. ADR-0045's pull reads one directory, +`jellyfin/config`, into one archive. Audiobookshelf's state — the database that +holds every listener's position, which is the thing this service exists for — +would sit on `erebor/apps`, be snapshotted every night, and never leave the +NAS. + +Measured on the pinned image on 2026-09-22, read-only as `65534`: + +- It declares no user and binds `PORT=80`, which a process with no + capabilities cannot. `PORT=13378` in the environment fixes both at once. +- `/config` holds `absdatabase.sqlite` in rollback-journal mode, not WAL; + `/metadata` holds covers, per-item metadata, logs and transient transcode + segments. Neither directory exists in the image. +- Its own scheduled backups are off by default. +- 91 MiB idle RSS, 119 MiB with a transcode running. + +## Decision + +1. **Audiobookshelf joins `stacks/media` on `smaug`**, in Jellyfin's shape: + non-root, `read_only`, `cap_drop: ALL`, pinned by digest, healthchecked + with what the image ships, and a memory ceiling over a measured number. + Its state is two bind mounts under one directory on `erebor/apps`, which + the nightly snapshot already covers. + +2. **A fifth pass: `vlan50 net → 10.0.40.30:13378`**, described + `Allow 13378 to smaug`, above *Block access to CasaBonita* on Hicks. It is + a rule of its own and not a port added to the `8096` rule, for the reason + the Hicks pair is split: §0.6 matches on the description. Its named + consumer, as [ADR-0012](0012-publish-only-ports-with-an-off-host-consumer.md) + requires, is the household's phones. It is **created when the service is + deployed** (§6.5), not when this merges — a pass to a port nothing answers + on cannot be proved. + +3. **Audiobooks only. Podcasts are deferred, not declined.** Auto-download is + the one feature that makes this service reach out on a schedule, and it + needs a folder it can write into on `erebor/media`. The library is mounted + read-only, for Jellyfin's reason — a server on the segment with the game + consoles should not be able to delete what it serves — and that mount is + what makes podcasts impossible rather than merely unused. This reopens + when someone wants a podcast on this server rather than in a phone's own + app. The answer then is a separate writable folder under `erebor/media`, + so the audiobook library stays read-only, and a note in `security.md` + that the segment now fetches on a timer. CasaBonita has internet egress + already, so this is a data-path change, not a firewall change. + +4. **The pull carries one archive per service, all from one snapshot, in one + set.** `scripts/backup-nas.sh` reads a table of archive → subpath rather + than one path. Audiobookshelf's is `audiobookshelf-state`, the parent of + its two mounts, with `./config/absdatabase.sqlite` as its sentinel. A set + is every archive or no MANIFEST, as before. Verification reads each set + against its own MANIFEST, so the sets written before this hold + `jellyfin-config` alone and are still complete. + +5. **A service authored before it is deployed is `pending` in that table.** + `stacks/media` is deployed by hand, later, and Audiobookshelf's deploy + waits on the mirror being whole + ([#558](https://github.com/Gerrrt/HomeLab/issues/558)). A pending row whose + directory is absent from the snapshot is skipped, and the skip is logged by + name. A pending row whose directory is present is pulled and verified like + any other. The deploy's Done commit flips it to `required`. After that, a + missing directory fails the run. Without this, merging the service would + have failed every Saturday's Jellyfin pull until the deploy. With it, the + repository still states in plain text whether the service is deployed. + +6. **No secrets file**, by [#528](https://github.com/Gerrrt/HomeLab/issues/528)'s + reasoning: the root user is created in the first-run screen and kept as a + hash in the database, where no environment variable reaches. The way back + in without the password was measured. Stop the container, clear the root + user's hash with a one-off `node` against the database, then sign in with a + blank password and set a new one. + +## Consequences + +- **Hicks reaches one more port on the NAS.** The service behind it requires + a login for everything but its health and status endpoints. Everything on + CasaBonita reaches it natively as well, as it reaches `8096`. +- **The monitoring host still reaches `9100` and `22` and nothing else.** It + cannot probe `13378`, so the container healthcheck is the only liveness + check, which is the position Jellyfin is in + ([#570](https://github.com/Gerrrt/HomeLab/issues/570)). +- **`frodo` reads one more directory**, and it includes the Audiobookshelf + users' password hashes. What lands on the monitoring host is ciphertext to + the same two recipients as before. ADR-0045's stated limit grows by one + service, and its shape does not change. +- **Every set grows by the size of the database and the metadata**, which is + kilobytes on a first boot and megabytes on a real library. ABS's own + auto-backups stay off so the sets do not also carry zipped copies of the + same database. +- **The skip in decision 5 is the one place the pull does less than its table + says**, and it is bounded. It applies only to a row the repository marks + pending, and only while that row's directory does not exist. Each skip + prints a line naming §6.5. +- **ADR-0016 and ADR-0045 are not superseded.** ADR-0016's rule set grows by a + row of the same kind, and ADR-0045's mechanism stays the same apart from + reading a list where it read one path. Both gain a pointer here. diff --git a/docs/architecture.md b/docs/architecture.md index fc38fc38..a3e9ffaf 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -189,9 +189,9 @@ hole from the monitoring VLAN into the monitored one. | `alexander` (10.0.30.40) | 🟢 30 | [`stacks/lab`](../stacks/lab) | Prometheus, Loki, Grafana, Alloy — the lab's own observability, which never remote-writes to VLAN 99 ([ADR-0007](adr/0007-defensive-estate-and-offensive-range.md), [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md)). A guest on `Saruman`, not the hypervisor; Alloy agent (Docker) | | `odin` (10.0.30.60) | 🟢 30 | [`stacks/soc`](../stacks/soc) | **Not built yet** — Wazuh (indexer, manager, dashboard), Velociraptor and Alloy: the security half of ADR-0007, placed by [ADR-0030](adr/0030-give-the-security-tooling-its-own-guest-and-its-own-stack.md) on a second guest on `Saruman` because `alexander`'s 8 GiB cannot hold both. The stack is authored and CI-validated ahead of the guest, the way `stacks/lab` was ahead of `alexander`; the build is [`build-the-soc-guest.md`](runbooks/build-the-soc-guest.md), behind [#414](https://github.com/Gerrrt/HomeLab/issues/414). The six domain machines report to it as agents; its Alloy pushes to `alexander`, never to VLAN 99, and the indexer's health is the one series that crosses into the lab's Prometheus. Alloy agent (Docker) | | `phoenix` (10.0.30.70) | 🟢 30 | *(none — a toolchain host, no Docker)* | The deployment host, built 2026-09-20: a Proxmox API token, an SSH key and a checkout, so that the Packer, OpenTofu and Ansible work after [#436](https://github.com/Gerrrt/HomeLab/issues/436) has somewhere to run from. A guest on `Saruman`, placed by [ADR-0043](adr/0043-keep-the-ca-on-prometheus-and-build-phoenix-as-the-deployment-host.md), which also decides that the estate's CA key stays on `prometheus` and does not follow the toolchain here. It holds no age key and converges nothing; it is the WireGuard endpoint of [ADR-0042](adr/0042-terminate-the-remote-path-on-the-lab-and-route-it.md), built 2026-09-22 ([#442](https://github.com/Gerrrt/HomeLab/issues/442)); the one thing it is meant to reach that no other guest does is `8006` on `Saruman`, by a single host-firewall rule — written 2026-09-20, when [#566](https://github.com/Gerrrt/HomeLab/issues/566) enabled the firewall the build had found disabled and narrowed the `local_network` alias that was admitting the whole segment underneath it. Its Alloy pushes to `alexander`, never to VLAN 99; the build is [`build-the-jumpbox.md`](runbooks/build-the-jumpbox.md). Alloy agent (native package, `scripts/deploy-agent.sh`) | -| `oracle` (10.0.99.30) | 🔴 99 | *(none — hand-run containers)* | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copies of the firewall export (`make backup-firewall`), of the weekly volume sets (`make backup`, [#535](https://github.com/Gerrrt/HomeLab/issues/535)) and of Jellyfin's state pulled off `smaug` (`make backup-nas`, [ADR-0045](adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md)). The estate's host for small off-host jobs — [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) | +| `oracle` (10.0.99.30) | 🔴 99 | *(none — hand-run containers)* | The Lemmiwinks wiki and its Postgres, since 2025-11-12 ([ADR-0011](adr/0011-keep-the-wiki-internal.md)); Alloy agent (Docker, `scripts/deploy-agent.sh`); the off-host copies of the firewall export (`make backup-firewall`), of the weekly volume sets (`make backup`, [#535](https://github.com/Gerrrt/HomeLab/issues/535)) and of the media tier's state pulled off `smaug` (`make backup-nas`, [ADR-0045](adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md)). The estate's host for small off-host jobs — [ADR-0015](adr/0015-give-oracle-the-off-host-jobs.md) | | `trinity` (10.0.99.40) | 🔴 99 | [`stacks/sensitive`](../stacks/sensitive) | **Not built yet** — ADR-0008's sensitive tier on the ProDesk 600 G4 of [ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md), after the firewall restore is rehearsed on it ([#404](https://github.com/Gerrrt/HomeLab/issues/404)). The foundation is authored: Caddy as the published HTTPS port and step-ca issuing beneath the tier's own root rather than the estate's, which is left untouched ([#129](https://github.com/Gerrrt/HomeLab/issues/129), [#130](https://github.com/Gerrrt/HomeLab/issues/130), [ADR-0037](adr/0037-give-the-sensitive-tier-its-own-root-and-issue-beneath-it-over-acme.md)), with AdGuard Home behind Caddy and publishing 53 to the firewall's forwarder alone ([#135](https://github.com/Gerrrt/HomeLab/issues/135), [ADR-0010](adr/0010-keep-the-resolver-on-the-gateway.md)); Home Assistant ([#134](https://github.com/Gerrrt/HomeLab/issues/134)), Immich — four containers behind Caddy with a memory limit on each ([#132](https://github.com/Gerrrt/HomeLab/issues/132)) — Paperless-ngx with a Postgres and a Valkey of its own ([#133](https://github.com/Gerrrt/HomeLab/issues/133)) and Vaultwarden ([#131](https://github.com/Gerrrt/HomeLab/issues/131)) are authored as well; the rest follow. Alloy agent (Docker, `scripts/deploy-agent.sh`), pushing to `prometheus` like `oracle`'s | -| `smaug` (10.0.40.30) | 🟡 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **Built, pooled and deployed**: TrueNAS on its boot SSD, the static above since 2026-09-16, the four inbound rules verified in position, the mirror `erebor` since 2026-09-18 and this stack running on it since 2026-09-19 — from a copy of the compose file on the pool, brought up with `docker compose` under TrueNAS's own Docker ([`build-the-nas.md`](runbooks/build-the-nas.md) §6). Jellyfin alone, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github.com/Gerrrt/HomeLab/issues/138)). Scraped by `prometheus` on `9100`; it pushes nothing, and runs no Alloy — the estate's first scraped host, and the reason [#256](https://github.com/Gerrrt/HomeLab/issues/256) was more than a line of YAML. That issue settled the fork TrueNAS opened in it: `node_exporter`, as a digest-pinned container in this stack rather than TrueNAS's own endpoint, so the existing `99 → 40:9100` pass, the `host-overview` dashboard and seven rules in `host.rules.yaml` all keep working unchanged. The `node` job and `prometheus/targets/node.yaml` are live, and the target with them since 2026-09-19, once the exporter answered from the pool. SMART reaches the same scrape by [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md): a root cron job in TrueNAS's UI runs the estate's collector and the exporter serves the file — still no agent, still nothing initiating from 40 — and patch state is deliberately not collected on an appliance ([#483](https://github.com/Gerrrt/HomeLab/issues/483)) | +| `smaug` (10.0.40.30) | 🟡 40 | [`stacks/media`](../stacks/media) | [ADR-0008](adr/0008-place-services-by-data-trust.md)'s media tier on the ThinkServer TS150 of [#413](https://github.com/Gerrrt/HomeLab/issues/413), placed and addressed by [ADR-0016](adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) and running TrueNAS rather than Ubuntu Server by [ADR-0040](adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md). **Built, pooled and deployed**: TrueNAS on its boot SSD, the static above since 2026-09-16, the four inbound rules verified in position, the mirror `erebor` since 2026-09-18 and this stack running on it since 2026-09-19 — from a copy of the compose file on the pool, brought up with `docker compose` under TrueNAS's own Docker ([`build-the-nas.md`](runbooks/build-the-nas.md) §6). Jellyfin, publishing 8096 to the segment because the televisions reach it natively and no firewall rule is involved at all ([#138](https://github.com/Gerrrt/HomeLab/issues/138)); and Audiobookshelf on 13378, **authored and not yet deployed**, whose clients are phones on Hicks and which therefore brings a fifth inbound rule, created when it is deployed ([#140](https://github.com/Gerrrt/HomeLab/issues/140), [ADR-0050](adr/0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md)). Scraped by `prometheus` on `9100`; it pushes nothing, and runs no Alloy — the estate's first scraped host, and the reason [#256](https://github.com/Gerrrt/HomeLab/issues/256) was more than a line of YAML. That issue settled the fork TrueNAS opened in it: `node_exporter`, as a digest-pinned container in this stack rather than TrueNAS's own endpoint, so the existing `99 → 40:9100` pass, the `host-overview` dashboard and seven rules in `host.rules.yaml` all keep working unchanged. The `node` job and `prometheus/targets/node.yaml` are live, and the target with them since 2026-09-19, once the exporter answered from the pool. SMART reaches the same scrape by [ADR-0047](adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md): a root cron job in TrueNAS's UI runs the estate's collector and the exporter serves the file — still no agent, still nothing initiating from 40 — and patch state is deliberately not collected on an appliance ([#483](https://github.com/Gerrrt/HomeLab/issues/483)) | | `bahamut` (10.0.30.50) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 domain controller, PDC emulator and DNS for `ad.matrix.elysium` — Tier 0. Static, because every member finds a DC through DNS and the DCs *are* the DNS. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | | `leviathan` (10.0.30.51) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 second domain controller and DNS — Tier 0. Static, for the same reason. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | | `titan` (10.0.30.52) | 🟢 30 | *(none — Windows)* | **Not built yet** — Windows Server 2025 file and member server — the shares, and the NTLM relay target that only exists because 2025 requires outbound SMB signing and not inbound — Tier 1. Scraped by `alexander` on `9182`; it pushes nothing, and runs no Alloy ([ADR-0029](adr/0029-size-the-lab-domain-and-separate-its-namespace-and-clock.md)) | diff --git a/docs/changelog.md b/docs/changelog.md index e4dbd6bc..9d0a4988 100644 --- a/docs/changelog.md +++ b/docs/changelog.md @@ -19,6 +19,38 @@ docstring gives: it is a record, not a claim about now. ## 2026-09-22 +- **[#140](https://github.com/Gerrrt/HomeLab/issues/140) Audiobookshelf is + authored for `stacks/media`, and the issue's one-line firewall claim was + wrong.** It said no rule beyond #138's 50→40 would be needed; there is no + such rule, only `443` and `8096` passes, one per port, and the phones are on + Hicks. So the service brings a fifth pass, `Allow 13378 to smaug`, created + when it is deployed rather than now. + [ADR-0050](adr/0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md) + records that, and records audiobooks only, with podcasts deferred behind a + read-only library mount. It also records that `scripts/backup-nas.sh` + pulls a table of archives, not one path, all from one snapshot. Without + that, the database holding every listener's position would have been + snapshotted nightly and never left the NAS. + + **Measured on the pinned image before a line was written.** It binds port + 80 by default, which a capability-less non-root process cannot. `/config` + and `/metadata` don't exist in the image. Its entrypoint is `tini`. The + database is in rollback-journal mode, not WAL. Idle RSS is 91 MiB. The + admin reset was proved on a scratch boot rather than described from + upstream: clear `root`'s hash, sign in blank, and the old password is + refused. + + **The pull learned to wait for a deploy.** The mirror is degraded (#558), + and the deploy waits on it. A pull that demanded Audiobookshelf's directory + would therefore have failed every Saturday's Jellyfin set until then. The + row is `pending`: skipped by name while its directory is absent, pulled + strictly once it is present, and flipped to `required` by §6.5's Done + commit. Verification now reads each set against its own MANIFEST, so the + two retained Jellyfin-only sets still verify. Both were re-verified with + the new script, and a bench pull against a fake snapshot tree on `oracle` + covered all three branches: skipped, refused for a missing sentinel with no + MANIFEST, and two archives verified. + - **[#455](https://github.com/Gerrrt/HomeLab/issues/455) The off-estate drive is bought, and buying it settles less than the row implied.** A WD Elements Portable 5 TB, `WDBU6Y0050BBK-WESN`, bus-powered USB 3.2 Gen 1. The second diff --git a/docs/network.md b/docs/network.md index b748be5b..520966d2 100644 --- a/docs/network.md +++ b/docs/network.md @@ -341,7 +341,14 @@ Televisions and consoles. Internet only. on `9100` and `22` — four host-scoped, port-scoped passes above *Block access to CasaBonita* on their interfaces. Everything else on every other segment is still refused, and the televisions need no rule at all because they share this - broadcast domain with the server. + broadcast domain with the server. **A fifth is specified and not yet + created**: Hicks to `10.0.40.30:13378`, `Allow 13378 to smaug`, for + Audiobookshelf, whose clients are the phones on this list rather than the + televisions + ([ADR-0050](adr/0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md)). + It is created when the service is deployed + ([`build-the-nas.md`](runbooks/build-the-nas.md) §6.5), and this bullet + says five on that day. - **What answers on `9100` is `node_exporter`**, which makes this the one host in the estate that Prometheus *scrapes* rather than is pushed to ([#256](https://github.com/Gerrrt/HomeLab/issues/256), diff --git a/docs/observability.md b/docs/observability.md index 0ecd22c4..87f99554 100644 --- a/docs/observability.md +++ b/docs/observability.md @@ -784,7 +784,7 @@ has stopped leaving this host"; since [#535](https://github.com/Gerrrt/HomeLab/issues/535) `backup-volumes` does the same with each weekly set; and since [#484](https://github.com/Gerrrt/HomeLab/issues/484) `backup-nas` — the one -job that first *fetches* from another host, Jellyfin's state off `smaug` — +job that first *fetches* from another host, the media tier's state off `smaug` — copies its set the same way. `verify-backups` hashes the far side of both set directories every morning. No rule names any of the three; the generic pair carries them all. diff --git a/docs/roadmap.md b/docs/roadmap.md index 96233dac..7ebd34ee 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -171,9 +171,14 @@ workstation can mount the share. - **[#523](https://github.com/Gerrrt/HomeLab/issues/523) A rule for the share** and **[#570](https://github.com/Gerrrt/HomeLab/issues/570) a healthcheck that can fail** have no gate and no order between them. -- **[#140](https://github.com/Gerrrt/HomeLab/issues/140) Audiobookshelf and - [#141](https://github.com/Gerrrt/HomeLab/issues/141) Navidrome** have no - gate but the mirror being whole, which is #558. +- **[#140](https://github.com/Gerrrt/HomeLab/issues/140) Audiobookshelf** is + authored — the service, a fifth Hicks pass it needs and the #140 text said + it did not, and its archive in the NAS pull, `pending` until deployed + ([ADR-0050](adr/0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md)). + What is left is [`build-the-nas.md`](runbooks/build-the-nas.md) §6.5 on + `smaug`, gated on the mirror being whole, which is #558. + [#141](https://github.com/Gerrrt/HomeLab/issues/141) Navidrome has no gate + but the same one. ## Saruman: the domain, then the SOC diff --git a/docs/runbooks/build-the-nas.md b/docs/runbooks/build-the-nas.md index 3ba16e09..89c2a9cf 100644 --- a/docs/runbooks/build-the-nas.md +++ b/docs/runbooks/build-the-nas.md @@ -216,11 +216,15 @@ the box and in the firewall, not here. The static is set on the host and the reservation is set on the server, and both are done because either alone is a single point of drift. -### §0.5 — Create the four rules, in order and in position +### §0.5 — Create the rules, in order and in position -**Position is the whole difficulty.** All four sit above a deny that has been +**Position is the whole difficulty.** Every one sits above a deny that has been in place since 2025; appended where new rules naturally land they would match nothing, and *"can I reach the NAS"* would still pass for the wrong reason. +The first four were created on 2026-09-16. The fifth is Audiobookshelf's +([ADR-0050](../adr/0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md)) +and is created as step 1 of §6.5, not before — a pass to a port nothing +answers on is a rule that cannot be proved. | On interface | Protocol / source → destination | Description | Position | | --- | --- | --- | --- | @@ -228,6 +232,7 @@ nothing, and *"can I reach the NAS"* would still pass for the wrong reason. | Hicks (50) | `tcp` `vlan50 net` → `10.0.40.30` port `8096` | `Allow 8096 to smaug` | **above** *Block access to CasaBonita* | | Winterfell (99) | `tcp` `10.0.99.20` → `10.0.40.30` port `9100` | `Allow 9100 to smaug` | **above** *Block access to CasaBonita* | | Winterfell (99) | `tcp` `10.0.99.20` → `10.0.40.30` port `22` | `Allow SSH to smaug` | **above** *Block access to CasaBonita* | +| Hicks (50) | `tcp` `vlan50 net` → `10.0.40.30` port `13378` | `Allow 13378 to smaug` | **above** *Block access to CasaBonita* — §6.5 | **Four, where [ADR-0016](../adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) wrote three.** The Hicks pass is one rule per port rather than one rule @@ -255,6 +260,14 @@ before, and not for administration. and the firewall never sees the packets. That is the whole of what ADR-0008 bought by putting the server with its clients. +**The fifth rule is the exception to that, and the reason is who listens.** +Audiobookshelf's clients are phones, and the phones are on Hicks — so unlike +Jellyfin its consumers are across a segment boundary. [#140](https://github.com/Gerrrt/HomeLab/issues/140) +said no rule beyond the 50→40 one would be needed, on the assumption that +rule was a segment pass; it is a pair of port passes, and 13378 is in neither. +It is a pass of its own rather than a port added to `Allow 8096 to smaug`, for +the reason the Hicks pair is split: the description is what §0.6 matches on. + ### §0.6 — Prove the rules did what you meant **Do this from `morpheus` and from the monitoring host, not from the pfSense @@ -271,7 +284,8 @@ pfctl -sr -vv \ Each pass must appear **above** the *Block access to CasaBonita* rule on its own interface: `Allow 9100`/`Allow SSH` before the block on `igc0.99`, and -`Allow HTTPS`/`Allow 8096` before it on `igc0.50`. **Read the order, not the +`Allow HTTPS`/`Allow 8096` — and, once §6.5 has created it, `Allow 13378` — +before it on `igc0.50`. **Read the order, not the numbers.** `-vv` numbers each ruleset from zero rather than counting output lines, so its `@` indices match neither `pfctl -sr | grep -n` nor anything written down here — they are a printing artefact, and only the sequence is a @@ -466,7 +480,7 @@ decision made deliberately rather than drifted into. | Dataset | Record size | atime | What it holds | Backed up | | --- | --- | --- | --- | --- | | `erebor/media` | `1M` | off | films, music, the library | **no** | -| `erebor/apps` | default | off | the compose files, and Jellyfin's `/config` as a bind mount | **yes** — §4.1 and §6.2 | +| `erebor/apps` | default | off | the compose files, Jellyfin's `/config` and — from §6.5 — Audiobookshelf's `config/` and `metadata/`, all as bind mounts | **yes** — §4.1 and §6.2 | **Why the split.** ADR-0008 already ruled the library replaceable — its loss is *"annoying rather than catastrophic"* — and backing up 18 TB of re-downloadable @@ -870,7 +884,7 @@ is what makes step 7 safe. > the `igc0.40` tripwire read **0 packets** on 2026-09-20 with the pull > done. Step 9 is the commit this block landed in. -### §6.3 — Restore Jellyfin's state +### §6.3 — Restore the media tier's state Two cases, in the order to try them. @@ -915,6 +929,33 @@ The proof is the same as the migration's: the users and the watch history are back in the web UI. `--numeric-owner` on both ends is what keeps `65534` as `65534` across two hosts that spell it differently. +**Audiobookshelf's state** (once §6.5 has run) restores the same two ways, +from the same snapshot or the same set — its archive is +`audiobookshelf-state`, and it holds `config/` and `metadata/` together, so +it unpacks into the parent directory rather than into `config`. From a +snapshot: + +```bash +docker compose stop audiobookshelf \ + && cp -a /mnt/erebor/apps/.zfs/snapshot//audiobookshelf/. /mnt/erebor/apps/audiobookshelf/ \ + && docker compose up -d +``` + +From a set, stream `audiobookshelf-state.tar.gz.age` to the staging +directory exactly as above, then on this host as root: + +```bash +docker compose stop audiobookshelf \ + && tar --numeric-owner -xzf /mnt/erebor/apps/home/frodo/restore/audiobookshelf-state.tar.gz -C /mnt/erebor/apps/audiobookshelf \ + && chown -R 65534:65534 /mnt/erebor/apps/audiobookshelf \ + && docker compose up -d \ + && rm -r /mnt/erebor/apps/home/frodo/restore +``` + +The proof is a book resuming where a phone left it. The library itself is not +in the set — it is on `erebor/media`, replaceable by ADR-0008 — so a book +whose files are gone comes back as an item with its position and no audio. + ### §6.4 — Turn SMART collection on, and confirm the boot disk's row The exporter §6.1 stood up reads nothing a drive says: it is uid 65534, @@ -1091,6 +1132,98 @@ reopens it. > The cron job is as the table says: daily 08:30, `root`, stdout hidden, > stderr not. +### §6.5 — Add Audiobookshelf + +[#140](https://github.com/Gerrrt/HomeLab/issues/140), +[ADR-0050](../adr/0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md). +The service is in `compose.yaml` from the day it merged, and **nothing here +has been done on this host.** The roadmap holds it until the mirror is whole +([#558](https://github.com/Gerrrt/HomeLab/issues/558)): its state lands on +`erebor/apps` like Jellyfin's, and a pool of one disk is not the place to +start accumulating a new thing worth backing up. + +Until this section runs, `scripts/backup-nas.sh` carries +`audiobookshelf-state` as **pending** and skips it by name when its directory +is absent from the snapshot, so the weekly Jellyfin pull is not broken by a +service that is not there yet. Step 8 is where that stops being allowed. + +1. **Create the fifth rule** in §0.5's table, in the pfSense UI: Hicks (50), + `tcp`, `vlan50 net` → `10.0.40.30` port `13378`, description exactly + **`Allow 13378 to smaug`**, **above** *Block access to CasaBonita*. Then + read its position from `morpheus` with §0.6's command — `Allow 13378` + before the block on `igc0.50`, beside `Allow HTTPS` and `Allow 8096`. + +2. **Create the state directory and the library folder**, as root, from the + console shell. The image ships neither `/config` nor `/metadata`, so there + is no world-writable default to fall back on as Jellyfin had; the + container runs as `65534` and has to own both. The library folder is on + `erebor/media` beside the films and inherits §5's ACL, whose `everyone@` + read entry is what lets `65534` read it: + + ```bash + mkdir -p /mnt/erebor/apps/audiobookshelf/config /mnt/erebor/apps/audiobookshelf/metadata \ + && chown -R 65534:65534 /mnt/erebor/apps/audiobookshelf \ + && chmod 755 /mnt/erebor/apps/audiobookshelf /mnt/erebor/apps/audiobookshelf/config /mnt/erebor/apps/audiobookshelf/metadata \ + && mkdir -p /mnt/erebor/media/audiobooks + ``` + +3. **Re-fetch the stack's two files and bring it up** — §6's two `curl` lines + for `compose.yaml` and `.env.example`, then: + + ```bash + cd /mnt/erebor/apps/stack && docker compose up -d && docker compose ps + ``` + + `media-audiobookshelf` should read `healthy` within a minute; it answered in + five seconds on the bench. + +4. **Create the root user** from a Hicks workstation or phone at + `http://10.0.40.30:13378`. The password goes in the password manager beside + Jellyfin's `admin`, and nowhere in this repository (#528). Then **Libraries + → Add**: type *Books*, folder `/audiobooks`. + +5. **Leave two settings at their defaults, and read that they are.** Under + **Settings → Backups**, *automatic backups* stays **off**: the snapshot + pull below already carries the database, and ABS's own zips would land in + `/metadata/backups` and ride every set as copies of themselves. And no + podcast library is created — the library mount is read-only, so + auto-download could not write anyway, and ADR-0050 records why that is a + deferral and not an oversight. + +6. **A phone on Hicks** signs in with the Audiobookshelf app at + `http://10.0.40.30:13378`, plays a minute of something, and a second + device signed in as the same user resumes at that minute. That is the + property #140 was opened for; read it, do not infer it from the server + answering. + +7. **Prove the backup user can read it**, the day after (the nightly snapshot + has to have been taken with the directory in it) — §6.2 step 5's check, + against the new subpath: + + ```bash + ssh frodo@10.0.40.30 "tar -cf /dev/null --numeric-owner -C '/mnt/erebor/apps/.zfs/snapshot//audiobookshelf' . && echo readable" + ``` + + Then, on the monitoring host from the deployment checkout, a set by hand. + It must list **both** archives and print `./config/absdatabase.sqlite + present`: + + ```bash + make backup-nas && make backup-nas ARGS=--list && make verify-backups + ``` + +8. **Flip the row to required.** In `scripts/backup-nas.sh`'s `NAS_ARCHIVES`, + `audiobookshelf-state` changes from `pending` to `required`, in the commit + that fills the Done block below. From then on a missing directory fails the + pull by name instead of being skipped. + +9. **Re-read the tripwire** from `morpheus`: the `igc0.40` packet count is + still **zero**. The app talks to the server; nothing on the server talks + out to anything but the internet it already had. + +> **Not yet done.** The date, the set stamp from step 7 and the two readings +> from steps 6 and 9 go here, in the commit that flips step 8. + ## §7 — Verify > **As of 2026-09-19:** the monitoring-host line holds in both halves, the @@ -1128,9 +1261,10 @@ reopens it. - A television on CasaBonita finds Jellyfin and plays something **without** any firewall rule being involved -- A Hicks workstation reaches `https://10.0.40.30` and `http://10.0.40.30:8096` +- A Hicks workstation reaches `https://10.0.40.30` and `http://10.0.40.30:8096`, + and — once §6.5 has run — a Hicks phone reaches `http://10.0.40.30:13378` - The monitoring host reaches `9100` and **nothing else**. Both halves are - checkable now: `443` and `8096` must be refused from the monitoring host, and + checkable now: `443`, `8096` and `13378` must be refused from the monitoring host, and `node_exporter` must be answering — §6.1 is what stands it up, and [#256](https://github.com/Gerrrt/HomeLab/issues/256) settled that it is `node_exporter` rather than TrueNAS's own endpoint. `up{job="node"}` should be @@ -1165,7 +1299,7 @@ reopens it. PBS**, whose sync job wants another PBS instance — on TrueNAS that is PBS in a VM or a change to an NFS/SMB datastore. [#485](https://github.com/Gerrrt/HomeLab/issues/485) carries it. -- **Offsite** — owned, since 2026-09-20. §6.2 gets Jellyfin's state off +- **Offsite** — owned, since 2026-09-20. §6.2 gets the media tier's state off `smaug`, onto the monitoring host and onto `oracle`, and every one of those copies is on the same shelf under the same roof — the position the volume sets and the firewall export are in. [ADR-0048](../adr/0048-carry-the-estates-backup-sets-with-the-second-recipient.md) diff --git a/docs/runbooks/restore-the-stack.md b/docs/runbooks/restore-the-stack.md index 01c59435..d783e95d 100644 --- a/docs/runbooks/restore-the-stack.md +++ b/docs/runbooks/restore-the-stack.md @@ -307,14 +307,15 @@ curl -sG http://localhost:9090/api/v1/query \ --- -## Jellyfin's state is a different set +## The media tier's state is a different set `make restore` and everything above it read `backups/volumes/`, the sets of -the stack on this host. Jellyfin's state on `smaug` is pulled into -`backups/nas/` by `make backup-nas` — one archive per set, read out of a ZFS -snapshot rather than a stopped container, and copied to `oracle` in the same -run under `backups/nas` beside these sets -([ADR-0045](../adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md)) — +the stack on this host. Jellyfin's and Audiobookshelf's state on `smaug` is +pulled into `backups/nas/` by `make backup-nas` — one archive per service, all +from one ZFS snapshot rather than a stopped container, and copied to `oracle` +in the same run under `backups/nas` beside these sets +([ADR-0045](../adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md), +[ADR-0050](../adr/0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md)) — and it is restored on `smaug`, by hand, per [`build-the-nas.md`](build-the-nas.md) §6.3. Nothing in this runbook touches it, and `make restore` cannot: the volumes it would write do not exist here. diff --git a/docs/runbooks/schedule-maintenance.md b/docs/runbooks/schedule-maintenance.md index 6039072b..8be29173 100644 --- a/docs/runbooks/schedule-maintenance.md +++ b/docs/runbooks/schedule-maintenance.md @@ -71,7 +71,8 @@ until that is done. One job's output *arrives* from another host before it leaves: `backup-nas` reaches into `smaug` over `99 → 40:22` — the direction [ADR-0016](../adr/0016-open-casabonita-inward-and-keep-it-terminal-outward.md) -requires of CasaBonita, which may initiate nothing — reads Jellyfin's state +requires of CasaBonita, which may initiate nothing — reads Jellyfin's and +Audiobookshelf's state out of the dataset's newest ZFS snapshot, encrypts it here, and then copies the set to `oracle` by the same helpers and under the same rules as the volume sets diff --git a/docs/runbooks/successor-handover.md b/docs/runbooks/successor-handover.md index b0172aee..56eec943 100644 --- a/docs/runbooks/successor-handover.md +++ b/docs/runbooks/successor-handover.md @@ -146,7 +146,7 @@ make validate `make validate` runs everything CI runs, plus one check that is about *this host*: whether the stack runs here while no `homelab-*` units are installed. The -schedule covers convergence, the volume backups, the pull of Jellyfin's state +schedule covers convergence, the volume backups, the pull of the media tier's state off `smaug`, the firewall export, the SNMP verification and the dashboard drift check — [`schedule-maintenance.md`](schedule-maintenance.md) has the table and the @@ -349,7 +349,7 @@ new keypair whose public half matches nothing. | --- | --- | | The whole observability stack (`make down`) | The record, and only the record. Nothing in the house depends on it, dashboards and alert rules are in git, and metrics and logs re-accumulate. `grafana-data` is the exception — users, annotations and any un-exported dashboard edit live only there | | `stacks/lab` | Nothing. It is committed and deployable and has never been deployed | -| `stacks/media` | Film night, and the watch positions if `jellyfin-config` goes with it. Nothing else depends on it, and [ADR-0008](../adr/0008-place-services-by-data-trust.md) already ruled the library replaceable | +| `stacks/media` | Film night and the audiobooks, and the watch and listening positions if `/mnt/erebor/apps` goes with it. Nothing else depends on it, and [ADR-0008](../adr/0008-place-services-by-data-trust.md) already ruled the library replaceable | | `homelab-converge.timer` | Automatic deployment. The host stays on whatever revision it is on until someone runs `make up`, which is exactly how it worked before | | `dashboards-drift` | The daily proof that Grafana holds no uncommitted dashboard edit | | Suricata on the terminal segments | Detection, not connectivity. It is alert-only — `Block Offenders` is off on both interfaces and stays off. [`enable-suricata.md`](enable-suricata.md) | diff --git a/docs/security.md b/docs/security.md index 0e1cc4e6..23c7fd01 100644 --- a/docs/security.md +++ b/docs/security.md @@ -274,7 +274,15 @@ read-only user, reachable from `10.0.99.20` alone by the rule that already existed. The residual it leaves, accepted, is one more service on the NAS with one more key that reads it — a key that lives on the host already holding the estate's age identity, and reads a directory that includes -Jellyfin's users' password hashes. +Jellyfin's users' password hashes — and, once Audiobookshelf is deployed, +its users' hashes too, in the same pull +([ADR-0050](adr/0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md)). +**A fifth is specified and not created:** `vlan50 net → 10.0.40.30:13378` +for Audiobookshelf, whose clients are phones on Hicks. The issue that +proposed it said no new rule would be needed; the Hicks passes are per +port, so one is. It is created when the service is deployed +([`build-the-nas.md`](runbooks/build-the-nas.md) §6.5), because a pass to a +port nothing answers on cannot be proved. [`network.md`](network.md) holds the current list. **Skids' does not exist.** `10.0.99.40 → 10.0.20.104:80,443/tcp` — Home Assistant to the Hue bridge, the one device on that segment with a local API — still waits above the block that @@ -290,7 +298,9 @@ game consoles share that broadcast domain and the firewall never sees those packets — the same property that lets them reach Jellyfin, working the other way. node_exporter has no write API, so the exposure is disclosure of the host's shape: filesystems, uptime, load. Accepted, and the same class as the -unauthenticated ports the observability stack publishes. +unauthenticated ports the observability stack publishes. Audiobookshelf's +`13378` is reachable the same way and is not the same residual: everything +there but its health and status endpoints wants a login. The row ADR-0008wrote as `99 → 20` is narrower than it read: one host to one device on two ports, with the twenty other devices on Skids still unreachable from anywhere, @@ -424,8 +434,11 @@ assumption consistent with what they are. the deviation and what would retire it. - **The media tier keeps its admin credential outside SOPS too, and has no secrets file at all.** Jellyfin's `admin` is created by its own setup wizard - and kept as a hash in `jellyfin.db` inside the `jellyfin-config` volume; no - environment variable or rendered file is a way to hand it in, so + and kept as a hash in `jellyfin.db` under its `/config` bind mount, and + Audiobookshelf's `root` is created on its first-run screen and kept as a + hash in `absdatabase.sqlite` + ([ADR-0050](adr/0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md)); + no environment variable or rendered file is a way to hand either in, so `stacks/media` has no `secrets/media.*` and no `.sops.yaml` rule — on purpose, decided on [#528](https://github.com/Gerrrt/HomeLab/issues/528), and not because the stack was deployed by hand. The plaintext is in the @@ -433,10 +446,9 @@ assumption consistent with what they are. protects it is a hash at rest, VLAN 40's terminal property, the `50 → 40` passes being the only way in, and [ADR-0008](adr/0008-place-services-by-data-trust.md)'s blast radius — a - media server whose data is replaceable. Audiobookshelf - ([#140](https://github.com/Gerrrt/HomeLab/issues/140)) and Navidrome - ([#141](https://github.com/Gerrrt/HomeLab/issues/141)) create their first - user the same way and join this bullet when they land. What would retire + media server whose data is replaceable. Navidrome + ([#141](https://github.com/Gerrrt/HomeLab/issues/141)) creates its first + user the same way and joins this bullet when it lands. What would retire it: a service on that tier taking a credential from outside. Then the tier gets `secrets/media.sops.yaml` and a rule of its own under [ADR-0020](adr/0020-run-the-lab-stack-in-a-guest-with-its-own-prometheus.md), diff --git a/scripts/backup-nas.sh b/scripts/backup-nas.sh index 2a2e126c..c1b1f0a4 100755 --- a/scripts/backup-nas.sh +++ b/scripts/backup-nas.sh @@ -1,16 +1,20 @@ #!/usr/bin/env bash # -# Pull Jellyfin's state off smaug, encrypt it here, and prove the archive is -# readable before calling the run a success. +# Pull the media tier's state off smaug, encrypt it here, and prove every +# archive is readable before calling the run a success. # # WHAT THIS PROTECTS, AND WHAT IT DOES NOT # # erebor/apps on smaug — Jellyfin's database, users, watch history, resume -# positions and metadata: the half of the media tier ADR-0008 ruled -# irreplaceable, and the one build-the-nas.md §4 declared backed up for a week -# before anything backed it up (#484). It does NOT protect erebor/media, the -# library, which ADR-0008 ruled replaceable and 18 TB of which would not fit -# anywhere in this estate; nor jellyfin-cache, transcode scratch that +# positions and metadata, and Audiobookshelf's database and metadata, which is +# where every listener's place in every book lives (ADR-0050): the half of the +# media tier ADR-0008 ruled irreplaceable, and the one build-the-nas.md §4 +# declared backed up for a week before anything backed it up (#484). One +# archive per service, all read from the SAME snapshot, in one set — so a +# restore never has to ask which night each service's state came from. It +# does NOT protect erebor/media, the library, which ADR-0008 ruled replaceable +# and 18 TB of which would not fit anywhere in this estate; nor jellyfin-cache, +# transcode scratch that # backup-volumes.sh lists as DISPOSABLE by name; nor erebor/ix-apps, Docker's # images and that cache volume. Those omissions are decisions (ADR-0045), and # stacks/media/README.md states them where the stack is. @@ -35,7 +39,7 @@ # tar's stdout on smaug becomes age's stdin here, and nothing unencrypted # touches this disk. # -# WHY IT READS A SNAPSHOT AND NEVER STOPS JELLYFIN +# WHY IT READS A SNAPSHOT AND NEVER STOPS JELLYFIN OR AUDIOBOOKSHELF # # backup-volumes.sh stops the stack because a copy of a live SQLite database # is a file that looks like a backup. Stopping Jellyfin over ssh would need the @@ -45,10 +49,13 @@ # build-the-nas.md §4 records), and this reads the newest one through # .zfs/snapshot/. A snapshot is atomic across the dataset, so jellyfin.db, its # -wal and its -shm are captured at one instant — the crash image SQLite's WAL -# mode is designed to recover from. That is a stronger claim than -# backup-volumes.sh's --hot can make and a weaker one than its quiesce, and the -# manifest says `snapshot` rather than `quiesced` for that reason. Two things -# follow. tar can never report "file changed as we read it" on a snapshot, so +# mode is designed to recover from. Audiobookshelf's absdatabase.sqlite runs +# in rollback-journal mode instead (measured; ADR-0050), and the same instant +# freezes a hot journal beside it if a write was in flight, which SQLite rolls +# back on open — the other crash image it is designed to recover from. That is +# a stronger claim than backup-volumes.sh's --hot can make and a weaker one +# than its quiesce, and the manifest says `snapshot` rather than `quiesced` for +# that reason. Two things follow. tar can never report "file changed as we read it" on a snapshot, so # exit 1 from it is treated as the premise breaking rather than as noise. And a # snapshot that has stopped being taken is a backup that has silently stopped # being current, so the newest snapshot's age is checked against @@ -71,17 +78,22 @@ # pulled in. The sets are NOT in backups/volumes/: verify_set() there checks a # set against the current stack's volume list, so a media set in that directory # would fail the nightly verification of the observability sets, and the two -# retentions would count against one KEEP. +# retentions would count against one KEEP. For the same reason --verify-only +# here hands verify_set() each set's OWN manifest list rather than letting it +# default to NAS_ARCHIVES: the sets written before Audiobookshelf joined hold +# jellyfin-config alone, and are complete for what they were. # # WHAT CROSSES THE WIRE # -# Four commands, all handed to the far side's login shell, which on TrueNAS is +# Five commands, all handed to the far side's login shell, which on TrueNAS is # zsh with nomatch on — backup-firewall.sh's rules apply verbatim: no glob, no # bash-only syntax, every path single-quoted and built only from values -# validated below and from a snapshot name that matched SNAPSHOT_RE. +# validated below and from a snapshot name that matched SNAPSHOT_RE. `test -d` +# runs once per pending archive, and the last two once per archive pulled. # # true the reachability preflight # ls -1 '' the snapshot names +# test -d '//' is a pending service deployed yet # du -sk '//' sizing, and the read preflight # tar --numeric-owner -czf - -C '//' . # @@ -93,9 +105,9 @@ # erebor/apps and nothing else — build-the-nas.md §6.2 creates it and proves # the read with the same tar, to /dev/null. The key is this host's operator # key, the one that already reaches morpheus and oracle. Known limit, stated -# rather than engineered away: whoever holds that key can read Jellyfin's -# configuration, which includes its users' password hashes; what lands here is -# ciphertext to the same two recipients that can open grafana.db. +# rather than engineered away: whoever holds that key can read Jellyfin's and +# Audiobookshelf's state, which includes their users' password hashes; what +# lands here is ciphertext to the same two recipients that can open grafana.db. # # WHY NAS_KEEP AND NOT KEEP # @@ -131,7 +143,6 @@ # NAS_SSH_TARGET default frodo@10.0.40.30 the host the state is pulled FROM # NAS_OFFHOST default atropos@10.0.99.30:backups/nas where the set is copied TO # NAS_SNAPSHOT_ROOT default /mnt/erebor/apps/.zfs/snapshot -# NAS_SUBPATH default jellyfin/config under each snapshot # NAS_SNAPSHOT_TZ default America/Los_Angeles the zone the names are in # NAS_SNAPSHOT_MAX_AGE default 172800 seconds; older fails the run # NAS_KEEP default 7 complete sets to retain @@ -153,15 +164,48 @@ VOL_OFFHOST="${NAS_OFFHOST:-atropos@10.0.99.30:backups/nas}" source "${REPO_ROOT}/scripts/backup-volumes.sh" OUT_DIR="${REPO_ROOT}/backups/nas" -VOL="jellyfin-config" -VOLUMES=("${VOL}") -# verify_remote_archives honours this when set; one archive per set here, so -# it never is. + +# What a set holds, in the order it is pulled: archive name, the directory +# under each snapshot it is read from, the service and container mounts the +# MANIFEST records against it, and whether the service is deployed. The name +# is the SENTINEL key in backup-volumes.sh and the file name in the set; the +# subpath is the bind mount stacks/media/.env.example names, relative to +# erebor/apps. Not an environment override, deliberately: these are facts +# about the compose file, and a line in /etc/default/homelab-timers that +# changed one would be a backup of the wrong directory that still verified. +# +# `required` or `pending`, and pending is the only way an archive leaves a set +# without the run failing. stacks/media is authored here and deployed on smaug +# by hand, later (build-the-nas.md §6), so a service can exist in compose.yaml +# for weeks before its directory exists on the pool — Audiobookshelf's waited +# on the mirror being whole (#558). A pending row whose directory is ABSENT +# from the snapshot is skipped with a line naming the runbook section; one +# whose directory is PRESENT is pulled and verified exactly like a required +# one. The commit that records the deploy's Done block flips it to required, +# and from then on a missing directory fails the run by name. The skip is +# therefore only possible while this file says, in the repository, that the +# service is not deployed. +NAS_ARCHIVES=( + "jellyfin-config|jellyfin/config|jellyfin|/config|required" + "audiobookshelf-state|audiobookshelf|audiobookshelf|/config,/metadata|pending" +) +VOLUMES=() +declare -A NAS_SUBPATH=() NAS_SERVICE=() NAS_MOUNT=() NAS_STATE=() +for a in "${NAS_ARCHIVES[@]}"; do + IFS='|' read -r v p s m st <<<"${a}" + VOLUMES+=("${v}") + NAS_SUBPATH["${v}"]="${p}" + NAS_SERVICE["${v}"]="${s}" + NAS_MOUNT["${v}"]="${m}" + NAS_STATE["${v}"]="${st}" +done +unset a v p s m st +# verify_remote_archives honours this when set; this script never narrows a +# set, so it never is, and the far side is read against each MANIFEST. ONLY_VOLUMES=() NAS_SSH_TARGET="${NAS_SSH_TARGET:-frodo@10.0.40.30}" NAS_SNAPSHOT_ROOT="${NAS_SNAPSHOT_ROOT:-/mnt/erebor/apps/.zfs/snapshot}" -NAS_SUBPATH="${NAS_SUBPATH:-jellyfin/config}" NAS_SNAPSHOT_TZ="${NAS_SNAPSHOT_TZ:-America/Los_Angeles}" NAS_SNAPSHOT_MAX_AGE="${NAS_SNAPSHOT_MAX_AGE:-172800}" NAS_KEEP="${NAS_KEEP:-7}" @@ -186,15 +230,22 @@ fi [[ ${NAS_SNAPSHOT_ROOT} =~ ^/[A-Za-z0-9._/-]+$ && ${NAS_SNAPSHOT_ROOT} != *..* ]] \ || die "NAS_SNAPSHOT_ROOT must be an absolute path of letters, digits, . _ - and /, got '${NAS_SNAPSHOT_ROOT}'" NAS_SNAPSHOT_ROOT="${NAS_SNAPSHOT_ROOT%/}" -[[ ${NAS_SUBPATH} =~ ^[A-Za-z0-9._-][A-Za-z0-9._/-]*$ && ${NAS_SUBPATH} != *..* && ${NAS_SUBPATH} != */ ]] \ - || die "NAS_SUBPATH must be a relative path of letters, digits, . _ - and /, got '${NAS_SUBPATH}'" [[ ${NAS_SNAPSHOT_TZ} =~ ^[A-Za-z0-9/_+-]+$ && -f /usr/share/zoneinfo/${NAS_SNAPSHOT_TZ} ]] \ || die "NAS_SNAPSHOT_TZ is not a zone this host knows: '${NAS_SNAPSHOT_TZ}'" NAS_USER="${NAS_SSH_TARGET%@*}" check_sentinel_table -[[ -n ${SENTINEL[$VOL]:-} ]] \ - || die "no sentinel for ${VOL} in backup-volumes.sh — this script will not write a backup it cannot verify" +for v in "${VOLUMES[@]}"; do + # The table is this file's, but the subpaths still reach a remote shell, so + # they are held to the class every other interpolated value is. + [[ ${NAS_SUBPATH[$v]} =~ ^[A-Za-z0-9._-][A-Za-z0-9._/-]*$ && ${NAS_SUBPATH[$v]} != *..* && ${NAS_SUBPATH[$v]} != */ ]] \ + || die "subpath for ${v} must be a relative path of letters, digits, . _ - and /, got '${NAS_SUBPATH[$v]}'" + [[ -n ${SENTINEL[$v]:-} ]] \ + || die "no sentinel for ${v} in backup-volumes.sh — this script will not write a backup it cannot verify" + [[ ${NAS_STATE[$v]} == required || ${NAS_STATE[$v]} == pending ]] \ + || die "state for ${v} must be required or pending, got '${NAS_STATE[$v]}'" +done +unset v # The ssh to smaug. NOT remote(): that name is backup-volumes.sh's and points # at oracle, and the copy step below relies on it doing so. SSH is that @@ -249,7 +300,7 @@ pick_snapshot() { # --------------------------------------------------------------------------- # Arguments — the shape backup-volumes.sh uses, minus the modes that only make # sense with a compose file: no --hot (a snapshot is the whole point), no -# --inventory, no --project, no --only (one archive per set). +# --inventory, no --project, no --only (a set is every archive or nothing). # --------------------------------------------------------------------------- usage() { sed -n 's|^# \{0,1\}||; /^Usage:/,/^$/p' "$0" | head -24; } @@ -329,8 +380,12 @@ case "${MODE}" in if ((${#targets[@]} == 0)) || [[ -z ${targets[0]} ]]; then die "no complete sets in ${OUT_DIR}" fi + # Each set against its own MANIFEST, not against NAS_ARCHIVES — see WHY + # IT SOURCES backup-volumes.sh. A set whose MANIFEST lists nothing falls + # back to the full list inside verify_set() and fails there, loudly. for t in "${targets[@]}"; do - verify_set "${t}" || failed=1 + mapfile -t held < <(manifest_volumes "${t}") + verify_set "${t}" "${held[@]}" || failed=1 done # The far side is checked even when a local set failed — different # findings, different repairs — unless --local-only says oracle may be @@ -402,19 +457,52 @@ nas_remote true \ || die "cannot reach ${NAS_SSH_TARGET} non-interactively — SSH is off on smaug (build-the-nas.md §6.2 turns it on), the 99 → 40:22 pass is not in position, this host's key is not in ${NAS_USER}'s authorized keys, or smaug's host key is not in ~/.ssh/known_hosts" pick_snapshot -SRC="${NAS_SNAPSHOT_ROOT}/${SNAPSHOT}/${NAS_SUBPATH}" +declare -A SRC=() BYTES=() SHA=() +for v in "${VOLUMES[@]}"; do SRC["${v}"]="${NAS_SNAPSHOT_ROOT}/${SNAPSHOT}/${NAS_SUBPATH[$v]}"; done + +# A pending service's directory, asked about by itself: `test -d` exits 1 for +# absent and ssh exits 255 for everything else, so "not deployed yet" is never +# confused with "could not ask". What is left in VOLUMES is what this set +# holds, and the MANIFEST is written from it. +held=() +for v in "${VOLUMES[@]}"; do + if [[ ${NAS_STATE[$v]} == pending ]]; then + set +e + nas_remote "test -d '${SRC[$v]}'" + rc=$? + set -e + case "${rc}" in + 0) ;; + 1) + info "skipping ${v}: ${SRC[$v]} does not exist — ${NAS_SERVICE[$v]} is pending in NAS_ARCHIVES and not deployed on smaug yet (build-the-nas.md §6.5)" + continue + ;; + *) die "cannot ask ${NAS_SSH_TARGET} whether ${SRC[$v]} exists (exit ${rc})" ;; + esac + fi + held+=("${v}") +done +VOLUMES=("${held[@]}") +unset held rc # du is the sizing pass and the read preflight in one: it exits non-zero on # any directory it cannot enter, which is the permission fault §6.2 exists to -# prevent, found before a set directory exists rather than after. -kb="$(nas_remote "du -sk '${SRC}'" | awk '{print $1}')" \ - || die "cannot read ${SRC} as ${NAS_USER} — the subpath is wrong, or ${NAS_USER} lacks read on something under it (build-the-nas.md §6.2)" -[[ ${kb} =~ ^[0-9]+$ ]] || die "du on ${SRC} returned '${kb}', not a size" +# prevent, found before a set directory exists rather than after. Every +# archive's directory is read before any is written, so a service whose +# directory was never created (build-the-nas.md §6.5) fails the run here, by +# name, and not half way through a set. +total_kb=0 +for v in "${VOLUMES[@]}"; do + kb="$(nas_remote "du -sk '${SRC[$v]}'" | awk '{print $1}')" \ + || die "cannot read ${SRC[$v]} as ${NAS_USER} — the directory does not exist on smaug, or ${NAS_USER} lacks read on something under it (build-the-nas.md §6.2)" + [[ ${kb} =~ ^[0-9]+$ ]] || die "du on ${SRC[$v]} returned '${kb}', not a size" + info "${SRC[$v]} is $(human $((kb * 1024))) uncompressed" + total_kb=$((total_kb + kb)) +done avail_kb="$(df --output=avail -k "${OUT_DIR}" | tail -1 | tr -d ' ')" -if ((avail_kb < kb * 11 / 10)); then - die "only $(human $((avail_kb * 1024))) free at ${OUT_DIR}, and ${SRC} holds $(human $((kb * 1024))) uncompressed — refusing to start" +if ((avail_kb < total_kb * 11 / 10)); then + die "only $(human $((avail_kb * 1024))) free at ${OUT_DIR}, and the set holds $(human $((total_kb * 1024))) uncompressed — refusing to start" fi -info "${SRC} is $(human $((kb * 1024))) uncompressed" STAMP="$(date -u +%Y%m%dT%H%M%SZ)" SET_DIR="${OUT_DIR}/${STAMP}" @@ -422,43 +510,49 @@ SET_DIR="${OUT_DIR}/${STAMP}" mkdir "${SET_DIR}" || die "set ${STAMP} already exists" started=${SECONDS} -part="${SET_DIR}/${VOL}.tar.gz.age.part" -info "archiving ${VOL} from ${NAS_SSH_TARGET}:${SRC}" - -# tar's stdout on smaug is age's stdin here; nothing plaintext touches this -# disk. The pipeline's statuses are read one at a time because they mean -# different things: 255 is ssh itself, and tar's 1 and 2 are the two faults -# the header argues about. -set +e -nas_remote "tar --numeric-owner -czf - -C '${SRC}' ." | age "${AGE_ARGS[@]}" --output "${part}" -rcs=("${PIPESTATUS[@]}") -set -e +part="" fail_archive() { red "$*" - rm -f "${part}" + [[ -n ${part} ]] && rm -f "${part}" red "the incomplete set is at ${SET_DIR} — no manifest written, nothing pruned" exit 1 } -case "${rcs[0]}" in - 0) ;; - 255) fail_archive "ssh to ${NAS_SSH_TARGET} failed mid-transfer" ;; - 1) fail_archive "tar reported a file changing while it read ${SRC} — that cannot happen on a snapshot, so this is not one: ${NAS_SNAPSHOT_ROOT} is not a .zfs/snapshot directory, or ${NAS_SUBPATH} is a live path" ;; - 2) fail_archive "tar could not read something under ${SRC} as ${NAS_USER} — a file Jellyfin wrote without world read; build-the-nas.md §6.2 names the fallback" ;; - *) fail_archive "tar on ${NAS_SSH_TARGET} exited ${rcs[0]}" ;; -esac -((rcs[1] == 0)) || fail_archive "age failed to write ${part}" -# Strict, never lenient: --hot's carve-out is for a copy of a live volume, and -# this read a snapshot. A missing sentinel here means the archive is not of -# Jellyfin's config directory, whatever the path was called. -if ! verify "${part}" "${VOL}" 0; then - red "the archive did not verify — the incomplete set is at ${SET_DIR}, no manifest written, nothing pruned" - exit 1 -fi -ARCHIVE_BYTES="$(stat -c %s "${part}")" -ARCHIVE_SHA="$(sha256sum "${part}" | awk '{print $1}')" -mv "${part}" "${SET_DIR}/${VOL}.tar.gz.age" +for v in "${VOLUMES[@]}"; do + part="${SET_DIR}/${v}.tar.gz.age.part" + info "archiving ${v} from ${NAS_SSH_TARGET}:${SRC[$v]}" + + # tar's stdout on smaug is age's stdin here; nothing plaintext touches this + # disk. The pipeline's statuses are read one at a time because they mean + # different things: 255 is ssh itself, and tar's 1 and 2 are the two faults + # the header argues about. + set +e + nas_remote "tar --numeric-owner -czf - -C '${SRC[$v]}' ." | age "${AGE_ARGS[@]}" --output "${part}" + rcs=("${PIPESTATUS[@]}") + set -e + + case "${rcs[0]}" in + 0) ;; + 255) fail_archive "ssh to ${NAS_SSH_TARGET} failed mid-transfer" ;; + 1) fail_archive "tar reported a file changing while it read ${SRC[$v]} — that cannot happen on a snapshot, so this is not one: ${NAS_SNAPSHOT_ROOT} is not a .zfs/snapshot directory, or ${NAS_SUBPATH[$v]} is a live path" ;; + 2) fail_archive "tar could not read something under ${SRC[$v]} as ${NAS_USER} — a file ${NAS_SERVICE[$v]} wrote without world read; build-the-nas.md §6.2 names the fallback" ;; + *) fail_archive "tar on ${NAS_SSH_TARGET} exited ${rcs[0]}" ;; + esac + ((rcs[1] == 0)) || fail_archive "age failed to write ${part}" + + # Strict, never lenient: --hot's carve-out is for a copy of a live volume, + # and this read a snapshot. A missing sentinel here means the archive is not + # of that service's state directory, whatever the path was called. + if ! verify "${part}" "${v}" 0; then + red "${v} did not verify — the incomplete set is at ${SET_DIR}, no manifest written, nothing pruned" + exit 1 + fi + BYTES["${v}"]="$(stat -c %s "${part}")" + SHA["${v}"]="$(sha256sum "${part}" | awk '{print $1}')" + mv "${part}" "${SET_DIR}/${v}.tar.gz.age" + part="" +done # Written last: its presence is what marks the set complete. The same # tab-separated shape backup-volumes.sh writes, so list_sets, prune and a @@ -469,14 +563,16 @@ mv "${part}" "${SET_DIR}/${VOL}.tar.gz.age" printf 'stack\t%s\n' media printf 'project\t%s\n' media printf 'mode\t%s\n' snapshot - printf 'source\t%s:%s\n' "${NAS_SSH_TARGET}" "${SRC}" + printf 'source\t%s:%s\n' "${NAS_SSH_TARGET}" "${NAS_SNAPSHOT_ROOT}/${SNAPSHOT}" printf 'snapshot\t%s\n' "${SNAPSHOT}" printf 'snapshot_epoch\t%s\n' "${SNAPSHOT_EPOCH}" printf 'recipient\t%s\n' "${AGE_RECIPIENT}" printf 'archiver\t%s\n' "ssh-tar" printf 'downtime\t0\n' printf '#volume\tservice\tmount\tbytes\tsha256\n' - printf '%s\t%s\t%s\t%s\t%s\n' "${VOL}" jellyfin /config "${ARCHIVE_BYTES}" "${ARCHIVE_SHA}" + for v in "${VOLUMES[@]}"; do + printf '%s\t%s\t%s\t%s\t%s\n' "${v}" "${NAS_SERVICE[$v]}" "${NAS_MOUNT[$v]}" "${BYTES[$v]}" "${SHA[$v]}" + done } > "${SET_DIR}/MANIFEST" # Local retention before the copy, for backup-volumes.sh's reason: a set past @@ -485,7 +581,7 @@ mv "${part}" "${SET_DIR}/${VOL}.tar.gz.age" prune printf '\n' -green "wrote ${SET_DIR#"${REPO_ROOT}"/} — ${VOL} from ${SNAPSHOT}, $(human "${ARCHIVE_BYTES}"), Jellyfin never stopped, $((SECONDS - started))s total" +green "wrote ${SET_DIR#"${REPO_ROOT}"/} — ${VOLUMES[*]} from ${SNAPSHOT}, $(human "$(manifest_bytes "${SET_DIR}")"), nothing on smaug stopped, $((SECONDS - started))s total" printf '\n' if ((LOCAL_ONLY)); then diff --git a/scripts/backup-volumes.sh b/scripts/backup-volumes.sh index 6a44700d..c4ea7f3e 100755 --- a/scripts/backup-volumes.sh +++ b/scripts/backup-volumes.sh @@ -121,7 +121,7 @@ # WHEN SOURCED # # scripts/backup-nas.sh sources this file rather than running it: it pulls -# Jellyfin's state off smaug over ssh (ADR-0045), a job this script's docker- +# the media tier's state off smaug over ssh (ADR-0045), a job this script's docker- # and-quiesce shape cannot do, and it wants the same sentinel table, the same # verify() and the same set layout rather than a second copy of them. The # guard just above the argument parsing returns to the caller, so everything @@ -339,6 +339,16 @@ human() { numfmt --to=iec --suffix=B "$1" 2>/dev/null || printf '%sB' "$1"; } # checkpoints it into a 536 KB main file with no -wal at all. A ZFS snapshot # taken while Jellyfin runs carries all three files (db, -wal, -shm) at one # instant, which is what makes reading one consistent; see COMPANIONS. +# +# Audiobookshelf's (audiobookshelf-state) is the second such archive name, and +# the one directory holds both of its bind mounts, config/ and metadata/ +# (ADR-0050). Read off a boot of the pinned image on 2026-09-22, read-only as +# 65534: ./config/absdatabase.sqlite exists before the listener answers, with +# ./config/migrations beside it; ./metadata/items//metadata.json appears +# on the first scan and ./metadata/logs on the first boot. The database is in +# rollback-journal mode, not WAL (`PRAGMA journal_mode` reads `delete`), so +# there is no -wal to list: a hot -journal is only ever present mid-write, and +# a snapshot that caught one carries it at the same instant as the database. declare -A SENTINEL=( [prometheus-data]="./chunks_head" [loki-data]="./chunks" @@ -357,6 +367,7 @@ declare -A SENTINEL=( [paperless-db-data]="./18/docker/PG_VERSION" [paperless-broker-data]="./dump.rdb" [jellyfin-config]="./data/jellyfin.db" + [audiobookshelf-state]="./config/absdatabase.sqlite" ) # Reported when absent, never fatal. These cover the fresh-volume case, where @@ -388,6 +399,7 @@ declare -A COMPANIONS=( [paperless-db-data]="./18/docker/base ./18/docker/pg_wal" [paperless-broker-data]="" [jellyfin-config]="./data/jellyfin.db-wal ./config/system.xml ./metadata ./plugins" + [audiobookshelf-state]="./config/migrations ./metadata/items ./metadata/logs" ) # Volumes archived by NOTHING, each with the reason — the third table, and @@ -403,9 +415,9 @@ declare -A COMPANIONS=( # on smaug, where this script does not, so the entry exists to make # `STACK=media backup-volumes.sh --inventory` say the true thing — nothing in # that file is this script's to archive — instead of dying over a sentinel. -# What IS archived from that host is backup-nas.sh's, and it is a bind mount -# on erebor/apps rather than a volume, which is why no jellyfin-config is -# declared there at all (ADR-0045). +# What IS archived from that host is backup-nas.sh's, and it is bind mounts +# on erebor/apps rather than volumes, which is why neither jellyfin-config nor +# audiobookshelf-state is declared there at all (ADR-0045, ADR-0050). declare -A DISPOSABLE=( [immich-model-cache]="a model cache immich-machine-learning re-downloads on first use" [jellyfin-cache]="transcode scratch and image caches Jellyfin regenerates on demand" @@ -439,7 +451,11 @@ parse_compose() { in_services && /^ volumes:[[:space:]]*$/ { inlist = 1; next } in_services && /^ [A-Za-z_<]/ { inlist = 0 } in_services && inlist && /^ - / { - if ($2 ~ /^[.\/]/) next + # A bind mount is a host path, and one spelled as a variable is still + # one: stacks/media names every host path through .env, and a source + # like ${X:?}/config would otherwise split on its own colon into a + # "volume" called ${X that two mounts share (#140). + if ($2 ~ /^[.\/$]/) next split($2, p, ":") if (p[1] == "" || p[2] == "") next printf "mount\t%s\t%s\t%s\n", p[1], svc, p[2] diff --git a/stacks/media/.env.example b/stacks/media/.env.example index f44bf6bc..b08732c8 100644 --- a/stacks/media/.env.example +++ b/stacks/media/.env.example @@ -5,9 +5,9 @@ # `make up STACK=media` and scripts/render-config.sh do not run on that host # (README.md says why). Secrets do NOT belong here; see secrets/README.md. This # stack has none, by decision (#528; docs/security.md § Secrets): Jellyfin's -# admin account is created through its own setup wizard on first run and lives -# in its database, not in an environment variable, so nothing a secrets file -# held could reach the container. +# admin account, and Audiobookshelf's root user, are created through their own +# first-run screens and live in their databases, not in environment variables, +# so nothing a secrets file held could reach either container. # The user Jellyfin runs as. Not root — the image declares no user and would # run as root if this were unset, and nothing else in this estate does. @@ -92,6 +92,38 @@ MEDIA_PATH=/mnt/erebor/media # default deny sees to everything else. JELLYFIN_BIND_ADDR=0.0.0.0 +# The user Audiobookshelf runs as — nobody:nogroup, for Jellyfin's reason. The +# image declares no user and creates /config and /metadata itself, so this pair +# has to own AUDIOBOOKSHELF_STATE_PATH's two directories before the first `up`, +# and has to be able to read AUDIOBOOKS_PATH, which erebor/media's `everyone@` +# read entry (build-the-nas.md §5) already grants. +AUDIOBOOKSHELF_UID=65534 +AUDIOBOOKSHELF_GID=65534 + +# Audiobookshelf's state: `config/` (the database — users, libraries and every +# listener's position in every book) and `metadata/` (covers, per-item +# metadata, logs), bind-mounted at /config and /metadata. +# +# One PARENT directory on erebor/apps rather than two keys, because the backup +# pulls it as one archive: scripts/backup-nas.sh reads `audiobookshelf` out of +# the same snapshot as `jellyfin/config` (ADR-0050). A directory and not a +# child dataset for Jellyfin's reason — the snapshot task is not recursive. +# Created, with both children, `0755` 65534:65534 before the first `up` +# (build-the-nas.md §6.5). +AUDIOBOOKSHELF_STATE_PATH=/mnt/erebor/apps/audiobookshelf + +# The audiobook library, mounted READ-ONLY at /audiobooks. A directory in the +# library dataset beside the films — replaceable media, not backed up, by +# ADR-0008's test. Read-only is also what keeps podcasts off: auto-download +# needs a writable folder, and ADR-0050 deferred it. +AUDIOBOOKS_PATH=/mnt/erebor/media/audiobooks + +# The address Audiobookshelf's 13378 is published on. 0.0.0.0 means 10.0.40.30, +# as for the two keys either side of it. Its consumers are the phones on Hicks, +# through the fifth pass — `vlan50 net → 10.0.40.30:13378`, build-the-nas.md +# §0.5 — and not this line. +AUDIOBOOKSHELF_BIND_ADDR=0.0.0.0 + # The address node_exporter's 9100 is published on. # # Same reasoning as JELLYFIN_BIND_ADDR above — smaug is single-homed, so 0.0.0.0 diff --git a/stacks/media/README.md b/stacks/media/README.md index cc6e755f..2765be94 100644 --- a/stacks/media/README.md +++ b/stacks/media/README.md @@ -18,26 +18,35 @@ fetched the same way and run by TrueNAS's cron — a change to it reaches cd /mnt/erebor/apps/stack && docker compose up -d ``` -`.env.example` names `JELLYFIN_CONFIG_PATH`, a directory on `erebor/apps` -that has to exist, owned by `65534:65534`, before the first `up` — §6 of the -runbook creates it, and says why it is a bind mount and not a volume. +`.env.example` names `JELLYFIN_CONFIG_PATH` and `AUDIOBOOKSHELF_STATE_PATH`, +directories on `erebor/apps` that have to exist, owned by `65534:65534`, +before the first `up` — §6 and §6.5 of the runbook create them, and say why +they are bind mounts and not volumes. | Service | Image | Port | Purpose | | --- | --- | --- | --- | | jellyfin | `jellyfin/jellyfin` | 8096 (http), on the segment | The media server the televisions reach directly, with Quick Sync hardware transcoding on the E3-1225 v6's HD P630 ([#138], [ADR-0016]) | +| audiobookshelf | `ghcr.io/advplyr/audiobookshelf` | 13378 (http), to Hicks through the fifth pass | Audiobooks, with listening progress that syncs between a person's devices ([#140], [ADR-0050]). **Authored, not yet deployed** — [`build-the-nas.md`] §6.5 | | node-exporter | `prom/node-exporter` | 9100 (http), to `10.0.99.20` only | How this host is monitored at all — Prometheus scrapes it, because nothing on this segment may push ([#256], [ADR-0016]); it also serves the SMART textfile a root cron job on the host writes ([#483], [ADR-0047]) | -Two services, and only one of them is the tier. `node-exporter` is here -because of the section below; everything else in this file is about Jellyfin. +Three services, and two of them are the tier. `node-exporter` is here +because of the section below. -One media service, and that is a decision rather than a starting point. [ADR-0016] +Jellyfin is the one video service, and that is a decision rather than a +starting point. [ADR-0016] builds **Jellyfin alone** and adds Plex only if a screen on 40 turns out to have no working Jellyfin client — the LG OLED, which is the primary screen, has one. Plex authenticates its clients through `plex.tv` even on a local network, which would make a household service that works today depend on a third party staying up and keeping its terms. That is a deferral against a -test nobody has run, not a rejection. Audiobookshelf ([#140]) and Navidrome -([#141]) land here later, in the shape this file already has. +test nobody has run, not a rejection. Audiobookshelf ([#140]) joined it on +[ADR-0008]'s test — an audiobook library is the same kind of data in the same +place — for the one thing Jellyfin does not do: progress that follows a +listener from a phone to a car to a speaker. **Audiobooks only**: the library +is mounted read-only, so podcast auto-download, the one feature that would +make it fetch on a schedule and write into `erebor/media`, is deferred by +[ADR-0050] rather than left switched off by default. Navidrome ([#141]) lands +here later, in the shape this file already has. ## Why there is no reverse proxy, and why that is not an omission @@ -61,19 +70,24 @@ every television would have to trust, and a second thing to be down. | Who | Reaches it how | | --- | --- | | Televisions on CasaBonita | Natively, same broadcast domain — the firewall never sees the packet | -| A Hicks workstation | Two of the four rules in [`build-the-nas.md`] §0.5 — `50 → 10.0.40.30:443` and `50 → 10.0.40.30:8096`, one per port | +| A Hicks workstation | Two of the rules in [`build-the-nas.md`] §0.5 — `50 → 10.0.40.30:443` and `50 → 10.0.40.30:8096`, one per port | +| A Hicks phone, on `13378` | The fifth — `50 → 10.0.40.30:13378`, `Allow 13378 to smaug` — **specified, and created only when [`build-the-nas.md`] §6.5 deploys Audiobookshelf** ([ADR-0050]) | | Prometheus, on `9100` | A third — `10.0.99.20 → 10.0.40.30:9100` | -| Prometheus, on `22` | The fourth — `10.0.99.20 → 10.0.40.30:22`, inert until [`build-the-nas.md`] §6.2 switches SSH on for the backup pull, as `frodo` with one key and read access to `erebor/apps` ([ADR-0045]) | +| Prometheus, on `22` | The fourth rule — `10.0.99.20 → 10.0.40.30:22`, inert until [`build-the-nas.md`] §6.2 switches SSH on for the backup pull, as `frodo` with one key and read access to `erebor/apps` ([ADR-0045]) | | Everything else on the estate | Not at all — default deny | [ADR-0012] asks for a named off-host consumer before a port is published, and -here there are three: every screen in the house, one workstation, and the -monitoring host. - -Neither published port is private to its consumer, and the reason is the same -for both: everything already on CasaBonita shares this broadcast domain and -reaches them without the firewall seeing a packet. For 8096 that is the whole -point. For 9100 it is a residual — an unauthenticated read of this host's +here there are four: every screen in the house, one workstation, the phones, +and the monitoring host. The phones are the first consumers of a service on +this host that are across a segment boundary rather than on it — Jellyfin's +case for publishing to the segment was that its clients live there, and +Audiobookshelf's do not. + +No published port is private to its consumer, and the reason is the same +for all three: everything already on CasaBonita shares this broadcast domain +and reaches them without the firewall seeing a packet. For 8096 that is the +whole point. For 13378 it costs nothing, because nothing there answers +without a login. For 9100 it is a residual — an unauthenticated read of this host's filesystems, uptime and load, by the televisions — and `docs/security.md` records it rather than the firewall rule being mistaken for a boundary it is not. @@ -127,15 +141,16 @@ to be rediscovered. ## The backup split -**Jellyfin's state is backed up. The library is not.** That is deliberate, it +**Jellyfin's and Audiobookshelf's state is backed up. The library is not.** That is deliberate, it is what [#138] demanded a deliberate answer on, and it is carried in the volume layout rather than in a policy document: | Volume / mount | What it holds | Backed up | | --- | --- | --- | | `${JELLYFIN_CONFIG_PATH}` → `/config` | database, users, **watch history, resume positions**, metadata | **yes** — `scripts/backup-nas.sh`, weekly, from a ZFS snapshot of `erebor/apps` | +| `${AUDIOBOOKSHELF_STATE_PATH}/config` → `/config`, `…/metadata` → `/metadata` | the database — users, libraries, **every listener's position in every book** — and covers, per-item metadata, logs | **yes** — the same pull, the same snapshot, archive `audiobookshelf-state` ([ADR-0050]); `pending` in the script until §6.5 runs, and skipped by name while its directory is absent | | `jellyfin-cache` | transcode scratch, image caches | no — regenerable | -| `${MEDIA_PATH}` → `/media` | the library itself | no — see below | +| `${MEDIA_PATH}` → `/media`, `${AUDIOBOOKS_PATH}` → `/audiobooks` | the library itself | no — see below | [ADR-0008] already ruled the library replaceable and its loss *"annoying rather than catastrophic"*, so backing up 18 TB of re-acquirable files would spend the @@ -198,12 +213,26 @@ the segment the `50 → 40` passes were made for. Before the wizard has been run at all there is no admin to reset, and `/health` reads `Degraded`, which is why the healthcheck ignores the body. -Audiobookshelf ([#140]) and Navidrome ([#141]) are the same shape — a first -user created through the UI, no admin password from the environment — and -join this section rather than `secrets/` when they land. Their way back in -without the password: Navidrome ships -`navidrome user edit --user --set-password`; Audiobookshelf's root is -reset by editing its user store under `/config`. +**Audiobookshelf's `root`** is the same shape: created on its first-run +screen, kept as a bcrypt hash in `absdatabase.sqlite`, and in the password +manager beside Jellyfin's. Its way back in was measured on 2026-09-22 rather +than assumed. From the stack directory on `smaug`, with the service stopped, +clear the hash with the image's own `node` and `sqlite3` module: + +```bash +docker compose stop audiobookshelf \ + && docker compose run --rm --no-deps -T --entrypoint node audiobookshelf -e \ + 'const s=require("/app/node_modules/sqlite3");const d=new s.Database("/config/absdatabase.sqlite");d.run("UPDATE users SET pash=NULL WHERE type=\"root\"",function(e){console.log(e||("cleared "+this.changes));d.close()})' \ + && docker compose up -d +``` + +`root` then signs in with a **blank** password, and the old one is refused; +set a new one under **Settings → Users** at once, because until then anyone +on Hicks or CasaBonita who types `root` is the administrator. + +Navidrome ([#141]) is the same shape again and joins this section rather than +`secrets/` when it lands; it ships +`navidrome user edit --user --set-password`. ## What was measured rather than assumed @@ -224,6 +253,24 @@ Every non-obvious line in `compose.yaml` came off the pinned image on is a restart loop. - **65 MiB idle RSS**, which is what the 2 GiB ceiling is a ceiling over. +Audiobookshelf's lines came off its pinned image on 2026-09-22, booted +read-only as `65534` with a generated two-minute book, a root user, a scan, a +progress write and a forced transcode. `compose.yaml` numbers all nine; the +ones that decide a line: + +- **It binds `PORT=80` by default**, which a process with no capabilities + cannot — hence `PORT=13378` in the environment. +- **`/config` and `/metadata` do not exist in the image.** The server creates + them, so the bind directories are created and chowned first; there is no + image mode to lean on. +- **`wget` and `busybox` are present, `curl` is not** — Jellyfin's opposite. +- **Its entrypoint is `tini`**, so the service sets `init: false`, as the two + Immich images do in `stacks/sensitive`. +- **The database is in rollback-journal mode, not WAL**, and progress is its + `mediaProgresses` table. Its own auto-backups are off by default and stay + off. +- **91 MiB idle, 119 MiB transcoding**, under a 1 GiB ceiling. + ## The check that passed [ADR-0040] keeps the media stack in this repository on the strength of Quick @@ -248,6 +295,7 @@ reopen condition is closed; the stack stays here. [ADR-0040]: ../../docs/adr/0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md [ADR-0045]: ../../docs/adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md [ADR-0047]: ../../docs/adr/0047-collect-smaug-smart-through-a-root-cron-and-the-textfile-collector.md +[ADR-0050]: ../../docs/adr/0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md [`build-the-nas.md`]: ../../docs/runbooks/build-the-nas.md [#138]: https://github.com/Gerrrt/HomeLab/issues/138 [#140]: https://github.com/Gerrrt/HomeLab/issues/140 diff --git a/stacks/media/compose.yaml b/stacks/media/compose.yaml index dd5b0a83..1f9a335f 100644 --- a/stacks/media/compose.yaml +++ b/stacks/media/compose.yaml @@ -19,12 +19,15 @@ # by TrueNAS's cron rather than by this file (§6.4). # # --------------------------------------------------------------------------- -# Why this stack is one service +# Why this stack is two media services and not three # --------------------------------------------------------------------------- # ADR-0016 decided Jellyfin alone, and Plex only if a screen on 40 turns out to # have no working Jellyfin client — a deferral against a test nobody has run, -# not a rejection. Audiobookshelf (#140) and Navidrome (#141) land here later, -# in the shape this file already has. +# not a rejection. Audiobookshelf (#140, ADR-0050) joined it on the argument +# that an audiobook library is the same kind of data in the same place, and +# brought the one thing Jellyfin does not do: listening progress that follows +# a person between a phone, a car and a speaker. Navidrome (#141) lands here +# later, in the shape this file already has. # # --------------------------------------------------------------------------- # Why there is no reverse proxy here, and why that is not an oversight @@ -195,6 +198,114 @@ services: # figure covers a start that has one to read. start_period: 60s + # --------------------------------------------------------------------------- + # Audiobookshelf — audiobooks, with progress that syncs (#140, ADR-0050) + # --------------------------------------------------------------------------- + # Its clients are phones, not televisions, and the phones are on Hicks — so + # unlike Jellyfin this is a service whose consumers are across a segment + # boundary, and it is reached through the fifth pass in build-the-nas.md + # §0.5 (`50 → 10.0.40.30:13378`) rather than by sharing a broadcast domain. + # + # MEASURED 2026-09-22 on the pinned image, booted read-only as 65534 with a + # generated two-minute book, a root user, a scan, a progress write and a + # forced transcode: + # + # 1. The image declares NO user, like Jellyfin. And it sets `PORT=80`, + # which a non-root process with every capability dropped cannot bind — + # hence the environment line below, and 13378 because that is the port + # upstream's documentation and the phone apps' setup screens assume. + # + # 2. `/config` and `/metadata` do NOT EXIST in the image; the server + # creates them. So there is no image mode to lean on as Jellyfin does, + # and both bind directories are created 0755 65534:65534 before the + # first `up` (build-the-nas.md §6.5). + # + # 3. `wget`, `busybox` and `nc` are present; `curl` is NOT — Jellyfin's + # opposite, and node_exporter's twin, which decides the healthcheck. + # + # 4. Its entrypoint is `tini`. Under the anchor's `init: true` tini starts + # as PID 2 and warns on every boot that it will not reap — the finding + # stacks/sensitive records for both Immich images, and the same answer: + # the image's own init is the init. + # + # 5. It writes nothing outside /config and /metadata. Transcode segments + # for a client that cannot play the file directly land in + # /metadata/streams, not /tmp; the tmpfs below is for node itself. + # + # 6. The database is `absdatabase.sqlite` in ROLLBACK-JOURNAL mode, not + # WAL: `PRAGMA journal_mode` reads `delete`, and a progress write left + # no -wal beside it. A snapshot taken mid-transaction freezes a hot + # journal with it, which SQLite rolls back on open — consistent, the + # same property the snapshot gives Jellyfin's WAL. Progress is the + # `mediaProgresses` table in that file. + # + # 7. Its own scheduled backups are OFF by default ("Auto Backups are + # disabled" at boot). Leave them off: they would write zipped copies of + # the database into /metadata/backups, which the snapshot pull already + # covers, and grow every set with copies of itself. + # + # 8. Files are created 0644 and directories 0755, so `frodo`'s read on + # erebor/apps (build-the-nas.md §6.2) needs nothing new. + # + # 9. It answered `/healthcheck` five seconds into a first boot. Idle RSS of + # node after a scan: **91 MiB**; **119 MiB** with a transcode running. + audiobookshelf: + <<: *service-defaults + image: ghcr.io/advplyr/audiobookshelf:2.36.1@sha256:3528a93b6442ffe54bd46771bbbab7c97084e1101071586d9dc2254f30bb4358 + container_name: media-audiobookshelf + # See MEASURED (4). + init: false + user: "${AUDIOBOOKSHELF_UID:?}:${AUDIOBOOKSHELF_GID:?}" + environment: + # See MEASURED (1). The in-container port equals the published one so a + # log line naming the port names the one a client uses. + PORT: "13378" + # A ceiling to catch a leak, not a measurement, as for Jellyfin. The + # numbers it is a ceiling over are 91 MiB idle and 119 MiB transcoding; + # the headroom is for scanning a real library, which is ffprobe over every + # file and the only work that will move it. + mem_limit: 1g + memswap_limit: 1g + cap_drop: [ALL] + security_opt: + - no-new-privileges:true + read_only: true + tmpfs: + - /tmp:size=64m + volumes: + # ALL of its state, in two directories under ONE parent on erebor/apps, + # which is what lets scripts/backup-nas.sh pull it as one archive from + # the same snapshot as Jellyfin's (ADR-0050, extending ADR-0045). + # + # /config is the database: users, libraries and — the reason this service + # exists — every listener's position in every book. /metadata is covers, + # per-item metadata, logs, and transient transcode segments; small, and + # pulled with the database rather than argued apart from it. + - ${AUDIOBOOKSHELF_STATE_PATH:?}/config:/config + - ${AUDIOBOOKSHELF_STATE_PATH:?}/metadata:/metadata + # READ-ONLY, for Jellyfin's reason: a server on the segment with the game + # consoles should not be able to delete the thing it serves. It is also + # the line that makes podcasts impossible rather than merely unused — + # auto-download needs a writable folder, and ADR-0050 deferred it. + - ${AUDIOBOOKS_PATH:?}:/audiobooks:ro + # PUBLISHED for the phones on Hicks, through the 13378 pass in + # build-the-nas.md §0.5 — ADR-0012's named off-host consumer. Everything + # on CasaBonita reaches it too, as it reaches 8096; unlike node_exporter's + # 9100 that is not a disclosure, because nothing here answers without a + # login. + ports: + - "${AUDIOBOOKSHELF_BIND_ADDR:?}:13378:13378" + healthcheck: + # busybox wget, because the image has it and has no curl — MEASURED (3). + # `/healthcheck` and not `/ping`: both answer 200 without a login, but + # `/ping` writes a log line per call, which at this interval is 2,880 + # lines a day saying nothing. + test: ["CMD", "wget", "--spider", "-q", "http://localhost:13378/healthcheck"] + interval: 30s + timeout: 5s + retries: 3 + start_period: 60s + # --------------------------------------------------------------------------- # node_exporter — how this host is monitored at all (#256, ADR-0016) # --------------------------------------------------------------------------- @@ -359,9 +470,10 @@ networks: - subnet: 172.28.40.0/24 volumes: - # ONE volume, and it is the one that is not backed up. Jellyfin's state is - # the bind mount above, on erebor/apps, so that it can be snapshotted and - # pulled — see the mount comment and ADR-0045. What is declared here is + # ONE volume, and it is the one that is not backed up. Jellyfin's state and + # Audiobookshelf's are the bind mounts above, on erebor/apps, so that they + # can be snapshotted and pulled — see the mount comments, ADR-0045 and + # ADR-0050. What is declared here is # transcode scratch and regenerable image caches, and scripts/backup-volumes.sh # lists it as DISPOSABLE by name, for the reason stacks/sensitive gives for # immich-model-cache; `STACK=media backup-volumes.sh --inventory` therefore diff --git a/systemd/homelab-backup-nas.service b/systemd/homelab-backup-nas.service index a15bc04c..4455984a 100644 --- a/systemd/homelab-backup-nas.service +++ b/systemd/homelab-backup-nas.service @@ -1,4 +1,5 @@ -# Weekly pull of Jellyfin's state off smaug (#484, ADR-0045). +# Weekly pull of the media tier's state off smaug — Jellyfin's and +# Audiobookshelf's (#484, ADR-0045, ADR-0050). # # WHY IT IS ITS OWN UNIT AND NOT A ROW IN homelab-backup-volumes. That unit # stops the observability stack and archives volumes on this host's docker @@ -14,7 +15,7 @@ # # See homelab-backup-volumes.service for why each directive below is set. [Unit] -Description=HomeLab: pull Jellyfin's state off smaug from its newest snapshot +Description=HomeLab: pull the media tier's state off smaug from its newest snapshot Documentation=file:///home/robo/code/Gerrrt/HomeLab/docs/runbooks/schedule-maintenance.md Wants=network-online.target After=network-online.target diff --git a/systemd/homelab-backup-nas.timer b/systemd/homelab-backup-nas.timer index 22f1cf52..734ea0bc 100644 --- a/systemd/homelab-backup-nas.timer +++ b/systemd/homelab-backup-nas.timer @@ -1,5 +1,5 @@ [Unit] -Description=HomeLab: weekly pull of Jellyfin's state off smaug +Description=HomeLab: weekly pull of the media tier's state off smaug Documentation=file:///home/robo/code/Gerrrt/HomeLab/docs/runbooks/schedule-maintenance.md [Timer]