Skip to content

make backup's volume sets stay on the host they protect: ADR-0015's copy to oracle is decided and unbuilt #535

Closed

Description

@Gerrrt

Decided and unbuilt

docs/roadmap.md's backup paragraph: "The volume sets make backup writes still sit on the host they protect … ADR-0015 sends them to oracle … and leaves only the copying to build." The firewall's configuration already makes that trip (#92's first half, verified 2026-09-03). The volume archives — Prometheus's TSDB, Loki, Grafana, and on trinity one day the tier's — do not.

A backup on the disk it backs up is a copy, not a backup. ADR-0015 chose oracle as the destination and the decision has sat inside #92's roadmap bullet, where it stopped belonging when ADR-0034 re-scoped that issue to the firewall rehearsal.

What to build

  • A step in the backup timer, after the archive is written and verified, that copies the set to oracle over the path the firewall copy already uses (same user, atropos; same key; same directory convention).
  • Retention on oracle: how many sets, and who prunes them. The firewall copy's answer is probably the right one.
  • verify-backups learns to check the remote copy exists and matches, or the BackupCopyMissing-shaped rule that The heartbeat receiver is not a dead man's switch — nothing watches for its absence #359's dead-man pattern suggests.
  • restore-the-stack.md gains the sentence "the set is on oracle at …", because a restore runbook that starts from a host that has just died is a runbook that starts nowhere.

Done when

A set written on prometheus is on oracle within the same timer run, a rule fires when one is not, and the restore runbook starts from the remote copy.

Refs #92, ADR-0015.

Activity

  1. added this to the Nothing blocks these milestone on Sep 19, 2026
  2. added
    securityA credential, an exposure, or a control that is weaker than recorded
    on Sep 19, 2026
  3. added a commit that references this issue on Sep 19, 2026
  4. Gerrrt commented on Sep 19, 2026

    @Gerrrt
    OwnerAuthor

    Built and deployed by #550 (merged 2026-09-19 22:58 UTC).

    On oracle now: backups/volumes/observability/ holds the five complete sets (20260829 → 20260906), seeded from the main checkout with make backup ARGS=--copy-only without stopping the stack, and make backup ARGS='--verify-only --all' then passed on both sides — every archive hashes to its MANIFEST entry there, every MANIFEST byte-identical. The daily verify-backups run does the same from tomorrow 05:30.

    What closes this: the "same timer run" half of Done when is Sunday 2026-09-20 03:30 — the first homelab-backup-volumes run with the copy step in it. A new stamp under backups/volumes/observability on oracle after it, with homelab_job_last_exit_code{homelab_job="backup-volumes"} == 0, is the proof; a failed copy is ScheduledJobFailed within ten minutes, by design (no new rule — backup.rules.yaml names no job).

    Two loose ends, neither blocking:

  5. Gerrrt commented on Sep 19, 2026

    @Gerrrt
    OwnerAuthor

    A second directory rides this copy. #554 (for #484) pulls Jellyfin's state off smaug into backups/nas/ on the monitoring host and copies it to oracle under backups/nas — beside backups/volumes/<stack> — by the helpers this issue's PR landed: backup-nas.sh sets VOL_OFFHOST to that directory before it sources backup-volumes.sh, so the value the volume unit puts in /etc/default/homelab-timers is never inherited, and every rule (verify by far-side hash, prune to KEEP, .part-then-mv, no globs over the wire) applies unchanged. make verify-backups walks both directories on both sides. Nothing here changes what this issue still owes: the seed and the first same-run copy on the deployment checkout.

    Refs #484, refs #554.

  6. Gerrrt commented on Sep 20, 2026

    @Gerrrt
    OwnerAuthor

    Done when, met. The first timer run with the copy step in it, Sun 2026-09-20 03:30 UTC: set 20260920T033007Z written (5 volumes, 1.7 GB, stack down 170 s), copied to atropos@10.0.99.30:backups/volumes/observability in the same run and every archive hashed to its MANIFEST entry there, homelab_job_last_exit_code{homelab_job="backup-volumes"} 0, 356 s in total. oracle now holds six complete sets, KEEP 7. A rule fires when a copy is missing by construction (the copy failing fails the job, and the daily verify fails on a set that is not there — both exercised by hand in #550's bench), and restore-the-stack.md §3 starts from the copy on oracle. Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    securityA credential, an exposure, or a control that is weaker than recorded

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions