You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
make backup's volume sets stay on the host they protect: ADR-0015's copy to oracle is decided and unbuilt #535
docs/roadmap.md's backup paragraph: "The volume sets make backup writes still sit on the host they protect … ADR-0015 sends them to oracle … and leaves only the copying to build." The firewall's configuration already makes that trip (#92's first half, verified 2026-09-03). The volume archives — Prometheus's TSDB, Loki, Grafana, and on trinity one day the tier's — do not.
A backup on the disk it backs up is a copy, not a backup. ADR-0015 chose oracle as the destination and the decision has sat inside #92's roadmap bullet, where it stopped belonging when ADR-0034 re-scoped that issue to the firewall rehearsal.
What to build
A step in the backup timer, after the archive is written and verified, that copies the set to oracle over the path the firewall copy already uses (same user, atropos; same key; same directory convention).
Retention on oracle: how many sets, and who prunes them. The firewall copy's answer is probably the right one.
restore-the-stack.md gains the sentence "the set is on oracle at …", because a restore runbook that starts from a host that has just died is a runbook that starts nowhere.
Done when
A set written on prometheus is on oracle within the same timer run, a rule fires when one is not, and the restore runbook starts from the remote copy.
Built and deployed by #550 (merged 2026-09-19 22:58 UTC).
On oracle now:backups/volumes/observability/ holds the five complete sets (20260829 → 20260906), seeded from the main checkout with make backup ARGS=--copy-only without stopping the stack, and make backup ARGS='--verify-only --all' then passed on both sides — every archive hashes to its MANIFEST entry there, every MANIFEST byte-identical. The daily verify-backups run does the same from tomorrow 05:30.
What closes this: the "same timer run" half of Done when is Sunday 2026-09-20 03:30 — the first homelab-backup-volumes run with the copy step in it. A new stamp under backups/volumes/observability on oracle after it, with homelab_job_last_exit_code{homelab_job="backup-volumes"} == 0, is the proof; a failed copy is ScheduledJobFailed within ten minutes, by design (no new rule — backup.rules.yaml names no job).
The two unit files changed only in their comments (VOL_OFFHOST named as a host fact; the verify unit now says it reaches oracle). sudo ./scripts/install-timers.sh --install re-copies them when convenient; the running units behave identically without it.
A second directory rides this copy.#554 (for #484) pulls Jellyfin's state off smaug into backups/nas/ on the monitoring host and copies it to oracle under backups/nas — beside backups/volumes/<stack> — by the helpers this issue's PR landed: backup-nas.sh sets VOL_OFFHOST to that directory before it sources backup-volumes.sh, so the value the volume unit puts in /etc/default/homelab-timers is never inherited, and every rule (verify by far-side hash, prune to KEEP, .part-then-mv, no globs over the wire) applies unchanged. make verify-backups walks both directories on both sides. Nothing here changes what this issue still owes: the seed and the first same-run copy on the deployment checkout.
Done when, met. The first timer run with the copy step in it, Sun 2026-09-20 03:30 UTC: set 20260920T033007Z written (5 volumes, 1.7 GB, stack down 170 s), copied to atropos@10.0.99.30:backups/volumes/observability in the same run and every archive hashed to its MANIFEST entry there, homelab_job_last_exit_code{homelab_job="backup-volumes"} 0, 356 s in total. oracle now holds six complete sets, KEEP 7. A rule fires when a copy is missing by construction (the copy failing fails the job, and the daily verify fails on a set that is not there — both exercised by hand in #550's bench), and restore-the-stack.md §3 starts from the copy on oracle. Closing.
Activity
Merge pull request #550 from Gerrrt/gerrrt/backup-volumes-oracle-beb3b2
Gerrrt commentedon Sep 19, 2026
Built and deployed by #550 (merged 2026-09-19 22:58 UTC).
On
oraclenow:backups/volumes/observability/holds the five complete sets (20260829 → 20260906), seeded from the main checkout withmake backup ARGS=--copy-onlywithout stopping the stack, andmake backup ARGS='--verify-only --all'then passed on both sides — every archive hashes to its MANIFEST entry there, every MANIFEST byte-identical. The dailyverify-backupsrun does the same from tomorrow 05:30.What closes this: the "same timer run" half of Done when is Sunday 2026-09-20 03:30 — the first
homelab-backup-volumesrun with the copy step in it. A new stamp underbackups/volumes/observabilityonoracleafter it, withhomelab_job_last_exit_code{homelab_job="backup-volumes"} == 0, is the proof; a failed copy isScheduledJobFailedwithin ten minutes, by design (no new rule —backup.rules.yamlnames no job).Two loose ends, neither blocking:
backups/volumes/20260913T033005Zis still onprometheusfor a hand removal, as the script's own warning asks; it was correctly not copied.VOL_OFFHOSTnamed as a host fact; the verify unit now says it reachesoracle).sudo ./scripts/install-timers.sh --installre-copies them when convenient; the running units behave identically without it.Gerrrt commentedon Sep 19, 2026
A second directory rides this copy. #554 (for #484) pulls Jellyfin's state off
smaugintobackups/nas/on the monitoring host and copies it tooracleunderbackups/nas— besidebackups/volumes/<stack>— by the helpers this issue's PR landed:backup-nas.shsetsVOL_OFFHOSTto that directory before it sourcesbackup-volumes.sh, so the value the volume unit puts in/etc/default/homelab-timersis never inherited, and every rule (verify by far-side hash, prune toKEEP,.part-then-mv, no globs over the wire) applies unchanged.make verify-backupswalks both directories on both sides. Nothing here changes what this issue still owes: the seed and the first same-run copy on the deployment checkout.Refs #484, refs #554.
feat(backup): jellyfin's state is pulled off smaug from a ZFS snapsho…
Gerrrt commentedon Sep 20, 2026
Done when, met. The first timer run with the copy step in it, Sun 2026-09-20 03:30 UTC: set
20260920T033007Zwritten (5 volumes, 1.7 GB, stack down 170 s), copied toatropos@10.0.99.30:backups/volumes/observabilityin the same run and every archive hashed to its MANIFEST entry there,homelab_job_last_exit_code{homelab_job="backup-volumes"}0, 356 s in total.oraclenow holds six complete sets,KEEP7. A rule fires when a copy is missing by construction (the copy failing fails the job, and the daily verify fails on a set that is not there — both exercised by hand in #550's bench), andrestore-the-stack.md§3 starts from the copy onoracle. Closing.feat(sensitive): a nightly backup timer on trinity (#404, #131)
feat(sensitive): a nightly backup timer on trinity (#404, #131) (#697)
docs(sensitive): Immich's restore rehearsed on trinity, and the photo…
docs(sensitive): Immich's restore rehearsed on trinity, and the photo…