Repository navigation
fix(backup): a visit missing a kind carries what it has and records no proof (#611) - #619
Merged
Merged
Conversation
…o proof (#611) backup-offsite.sh warned about a kind with nothing complete under it and then carried on, so a run that copied two of three printed "the medium holds the newest of each kind, proved" in green and exited 0. run-scheduled.sh records that exit as homelab_job_last_success_timestamp_seconds{homelab_job= "offsite-copy"}, so OffsiteCopyStale went quiet for ninety days over a medium that was missing a set. The sentence was false and the silence was bought with it. Reproduced against origin/main before changing anything, on a tree holding a volume set and a firewall export and no NAS set: warning: no complete NAS set under <tree>/backups/nas copied 20260921T000000Z ... every archive hashes to its MANIFEST entry copied config-20260921T000000Z.sops.yaml ... sha256 recorded beside it the medium holds the newest of each kind, proved — offsite, not just off-host. Keep it out of the house. >>> exit=0 THE COPY IS NOT REFUSED, THE PROOF IS. Someone who has never run `make backup-nas` should still be able to carry the other two off the estate, and a visit that carried two kinds beats one that carried none. So the run copies everything it has, names what is absent, and exits non-zero. Same tree, after: copied 20260921T000000Z ... every archive hashes to its MANIFEST entry copied config-20260921T000000Z.sops.yaml ... sha256 recorded beside it the medium holds volumes, firewall, and that verifies. NOT held: nas — nothing complete under <tree>/backups/nas, run `make backup-nas` No proof is recorded while a kind is missing, so this run does not buy ninety days of OffsiteCopyStale silence. >>> exit=1 That was the open question on #611 — hard failure or loud success — and it is the one thing here that is a judgement rather than a defect. Refusing the whole run was the third option and is not taken: it strands two good sets over a backup somebody may never have run. #611 cites one green line. There are two, and the other is not in the issue: the "nothing to copy" path made the same claim about the same three kinds and exited 0 just as readily, so a medium already holding two of three reported a clean proof on every subsequent run. Both now go through finish(). backup-firewall.sh's OUT_DIR is anchored to the repository, the cause #611 identifies. SOPS_POLICY was relative too and is anchored with it — fixing only the path the issue names would have left the other half. These agreed with where backup-offsite.sh looks only because the systemd unit sets WorkingDirectory; run by hand from anywhere else the export landed where the offsite copy does not read. backup-volumes.sh:158 and backup-nas.sh:155 were already absolute. The two fixes are complementary rather than alternatives, as the issue suggests: anchoring removes the likeliest cause, and refusing the proof catches whatever still produces a gap — a backup that simply stopped running, which is the case the ninety-day deadline exists for. Seven fixtures added, and they fail against the code they replace. One proves the refusal is the missing kind and not the fixture: filling the gap turns the same run green. Self-test 31 PASS, 0 FAIL; `make validate` passes. Refs #611, #573, ADR-0048. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…g-kind-no-proof-611
The label in validate.sh is written by hand and said 24, which was right until the previous commit added seven. It is a label and not an assertion — validate reads the self-test's exit code — so a stale number fails nothing and is exactly the decorative count #614 is about. Updated rather than left, and #614 still owns making it derived and making the two conditional fixtures stop skipping silently. Refs #611, #614. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This was referenced Sep 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
backup-offsite.shno longer records a proof when a backup kind has nothingcomplete under it, and
backup-firewall.sh's local paths are anchored to therepository instead of the working directory. Both halves of
#611.
Why
The bug, reproduced against
origin/mainbefore anything was changed, on atree holding a volume set and a firewall export and no NAS set:
That exit 0 is what
run-scheduled.shrecords ashomelab_job_last_success_timestamp_seconds{homelab_job="offsite-copy"}, soOffsiteCopyStalestays quiet for ninety days over a medium missing a set.Same tree, after:
The copy is not refused; the proof is. That was #611's open question — hard
failure or loud success — and it is the one judgement call here rather than a
defect. Refusing the whole run was the third option and is not taken: it
strands two good sets over a backup somebody may never have run. Say so on the
issue if you want it stricter; it is the
exit 1and nothing else.Two things the issue does not mention
There are two false green lines, not one. #611 cites the one after the
copy. The
nothing to copypath made the same claim about the same three kindsand exited 0 just as readily, so a medium already holding two of three reported
a clean proof on every subsequent run — arguably the worse of the two, since
it is the steady state. Both now go through
finish().SOPS_POLICYwas relative too. #611 namesOUT_DIR;SOPS_POLICY=".sops.yaml"sat on the line below with the same defect. Anchoring only the one named would
have left the other half.
backup-volumes.sh:158andbackup-nas.sh:155werealready absolute — this brings the odd one out into line.
The two fixes are complementary rather than alternatives: anchoring removes the
likeliest cause of a gap, and refusing the proof catches whatever still makes
one — a backup that simply stopped running, which is the case the ninety-day
deadline exists for.
Blast radius
secrets/*.sops.yaml--list,--verify-onlyand--pruneall return before the copy path and areuntouched — checked, because the
--listpre-flight before a visit must stayexit 0 and informational.
One behaviour change worth stating plainly: a run that would previously have
recorded a success now exits 1 when a kind is missing, so
ScheduledJobFailed{homelab_job="offsite-copy"}can fire where it could notbefore. That is the point. On this host it changes nothing today — all three
kinds are present and complete.
Verification
make validatepasses —all checks passed (3 skipped — NOT run, not verified)backup-offsite.sh --self-test— 31 PASS, 0 FAIL, up from 24Seven fixtures added. They cover the copy path, the
nothing to copypath, anda control: filling the gap turns the same run green, which proves the refusal
is the missing kind and not the fixture.
validate.sh's(24 fixtures)label is hand-written and would have gonestale, so it follows in its own commit. It is a label, not an assertion —
#614 still owns making it
derived and making the two conditional fixtures stop skipping silently.
This does not take effect on merge.
scripts/runs from the checkout onthe monitoring host, and the converge timer is report-only, so the nightly
backup-firewallkeeps the old behaviour until that checkout is updated.Using
Refsrather than a close keyword, as#596's fix did — close #611 by
hand when you are satisfied.
Refs #611, #614, #573, ADR-0048.
🤖 Generated with Claude Code