Skip to content

fix(backup): a visit missing a kind carries what it has and records no proof (#611) - #619

Merged
Gerrrt merged 3 commits into
mainfrom
gerrrt/offsite-missing-kind-no-proof-611
Sep 22, 2026
Merged

Gerrrt merged 3 commits into
mainfrom
gerrrt/offsite-missing-kind-no-proof-611

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Sep 22, 2026

Copy link
Copy Markdown
Owner

What changed

backup-offsite.sh no longer records a proof when a backup kind has nothing
complete under it, and backup-firewall.sh's local paths are anchored to the
repository instead of the working directory. Both halves of
#611.

Why

The bug, reproduced against origin/main before anything was changed, on a
tree holding a volume set and a firewall export and no NAS set:

warning: no complete NAS set under <tree>/backups/nas
copied 20260921T000000Z ... every archive hashes to its MANIFEST entry
copied config-20260921T000000Z.sops.yaml ... sha256 recorded beside it
the medium holds the newest of each kind, proved — offsite, not just off-host.
>>> exit=0

That exit 0 is what run-scheduled.sh records as
homelab_job_last_success_timestamp_seconds{homelab_job="offsite-copy"}, so
OffsiteCopyStale stays quiet for ninety days over a medium missing a set.

Same tree, after:

copied 20260921T000000Z ... every archive hashes to its MANIFEST entry
copied config-20260921T000000Z.sops.yaml ... sha256 recorded beside it
the medium holds volumes, firewall, and that verifies.
NOT held: nas — nothing complete under <tree>/backups/nas, run `make backup-nas`
No proof is recorded while a kind is missing, so this run does not buy ninety
days of OffsiteCopyStale silence.
>>> exit=1

The copy is not refused; the proof is. That was #611's open question — hard
failure or loud success — and it is the one judgement call here rather than a
defect. Refusing the whole run was the third option and is not taken: it
strands two good sets over a backup somebody may never have run. Say so on the
issue if you want it stricter; it is the exit 1 and nothing else.

Two things the issue does not mention

There are two false green lines, not one. #611 cites the one after the
copy. The nothing to copy path made the same claim about the same three kinds
and exited 0 just as readily, so a medium already holding two of three reported
a clean proof on every subsequent run — arguably the worse of the two, since
it is the steady state. Both now go through finish().

SOPS_POLICY was relative too. #611 names OUT_DIR; SOPS_POLICY=".sops.yaml"
sat on the line below with the same defect. Anchoring only the one named would
have left the other half. backup-volumes.sh:158 and backup-nas.sh:155 were
already absolute — this brings the odd one out into line.

The two fixes are complementary rather than alternatives: anchoring removes the
likeliest cause of a gap, and refusing the proof catches whatever still makes
one — a backup that simply stopped running, which is the case the ninety-day
deadline exists for.

Blast radius

  • No change to network segmentation or firewall rules
  • No new port published to a VLAN that could not already reach the service
  • No credential added outside secrets/*.sops.yaml

--list, --verify-only and --prune all return before the copy path and are
untouched — checked, because the --list pre-flight before a visit must stay
exit 0 and informational.

One behaviour change worth stating plainly: a run that would previously have
recorded a success now exits 1 when a kind is missing, so
ScheduledJobFailed{homelab_job="offsite-copy"} can fire where it could not
before. That is the point. On this host it changes nothing today — all three
kinds are present and complete.

Verification

  • make validate passes — all checks passed (3 skipped — NOT run, not verified)
  • backup-offsite.sh --self-test — 31 PASS, 0 FAIL, up from 24
  • The seven new fixtures fail against the code they replace
  • Deployed — n/a, but see below

Seven fixtures added. They cover the copy path, the nothing to copy path, and
a control: filling the gap turns the same run green, which proves the refusal
is the missing kind and not the fixture.

validate.sh's (24 fixtures) label is hand-written and would have gone
stale, so it follows in its own commit. It is a label, not an assertion —
#614 still owns making it
derived and making the two conditional fixtures stop skipping silently.

This does not take effect on merge. scripts/ runs from the checkout on
the monitoring host, and the converge timer is report-only, so the nightly
backup-firewall keeps the old behaviour until that checkout is updated.

Using Refs rather than a close keyword, as
#596's fix did — close #611 by
hand when you are satisfied.

Refs #611, #614, #573, ADR-0048.

🤖 Generated with Claude Code

Gerrrt and others added 3 commits September 22, 2026 19:27
…o proof (#611)

backup-offsite.sh warned about a kind with nothing complete under it and then
carried on, so a run that copied two of three printed "the medium holds the
newest of each kind, proved" in green and exited 0. run-scheduled.sh records
that exit as homelab_job_last_success_timestamp_seconds{homelab_job=
"offsite-copy"}, so OffsiteCopyStale went quiet for ninety days over a medium
that was missing a set. The sentence was false and the silence was bought
with it.

Reproduced against origin/main before changing anything, on a tree holding a
volume set and a firewall export and no NAS set:

  warning: no complete NAS set under <tree>/backups/nas
  copied 20260921T000000Z ... every archive hashes to its MANIFEST entry
  copied config-20260921T000000Z.sops.yaml ... sha256 recorded beside it
  the medium holds the newest of each kind, proved — offsite, not just
  off-host. Keep it out of the house.
  >>> exit=0

THE COPY IS NOT REFUSED, THE PROOF IS. Someone who has never run
`make backup-nas` should still be able to carry the other two off the estate,
and a visit that carried two kinds beats one that carried none. So the run
copies everything it has, names what is absent, and exits non-zero. Same tree,
after:

  copied 20260921T000000Z ... every archive hashes to its MANIFEST entry
  copied config-20260921T000000Z.sops.yaml ... sha256 recorded beside it
  the medium holds volumes, firewall, and that verifies.
  NOT held: nas — nothing complete under <tree>/backups/nas, run `make backup-nas`
  No proof is recorded while a kind is missing, so this run does not buy
  ninety days of OffsiteCopyStale silence.
  >>> exit=1

That was the open question on #611 — hard failure or loud success — and it is
the one thing here that is a judgement rather than a defect. Refusing the whole
run was the third option and is not taken: it strands two good sets over a
backup somebody may never have run.

#611 cites one green line. There are two, and the other is not in the issue:
the "nothing to copy" path made the same claim about the same three kinds and
exited 0 just as readily, so a medium already holding two of three reported a
clean proof on every subsequent run. Both now go through finish().

backup-firewall.sh's OUT_DIR is anchored to the repository, the cause #611
identifies. SOPS_POLICY was relative too and is anchored with it — fixing only
the path the issue names would have left the other half. These agreed with
where backup-offsite.sh looks only because the systemd unit sets
WorkingDirectory; run by hand from anywhere else the export landed where the
offsite copy does not read. backup-volumes.sh:158 and backup-nas.sh:155 were
already absolute.

The two fixes are complementary rather than alternatives, as the issue
suggests: anchoring removes the likeliest cause, and refusing the proof catches
whatever still produces a gap — a backup that simply stopped running, which is
the case the ninety-day deadline exists for.

Seven fixtures added, and they fail against the code they replace. One proves
the refusal is the missing kind and not the fixture: filling the gap turns the
same run green. Self-test 31 PASS, 0 FAIL; `make validate` passes.

Refs #611, #573, ADR-0048.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The label in validate.sh is written by hand and said 24, which was right until
the previous commit added seven. It is a label and not an assertion — validate
reads the self-test's exit code — so a stale number fails nothing and is
exactly the decorative count #614 is about. Updated rather than left, and #614
still owns making it derived and making the two conditional fixtures stop
skipping silently.

Refs #611, #614.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Gerrrt
Gerrrt merged commit d0e2556 into main Sep 22, 2026
3 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/offsite-missing-kind-no-proof-611 branch September 22, 2026 19:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

backup-offsite records a proved ninety days for a copy that is missing a set

1 participant