Skip to content

fix(backup): a different filesystem is not a different machine — /dev/shm is refused, and the self-test stops calling it a medium - #596

Merged
Gerrrt merged 2 commits into
mainfrom
gerrrt/offsite-refuse-tmpfs-9f3c21
Sep 21, 2026
Merged

Gerrrt merged 2 commits into
mainfrom
gerrrt/offsite-refuse-tmpfs-9f3c21

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Sep 21, 2026

Copy link
Copy Markdown
Owner

What happened

An hour after #588 merged, the first real invocation was make backup-offsite DEST=/dev/shm. It ran to its green line:

copied 20260920T033007Z to /dev/shm/backups/volumes/observability — every archive hashes to its MANIFEST entry
the medium holds the newest of each kind, proved — offsite, not just off-host. Keep it out of the house.

1.7 GB into tmpfs on a host with 216 MB free, gone on the next reboot, and a recorded offsite-copy success buying ninety days of silence from OffsiteCopyStale for a copy that existed nowhere. That is the failure class ADR-0048 exists to prevent — a copy read with confidence — reproduced by the tool written to prevent it.

The guard compared filesystems. /dev/shm is a different filesystem. It is also this host's RAM. The check was never the claim the ADR made for it.

The fix

Three refusals, ordered so each gives its own diagnosis rather than whichever fires first:

Destination Refusal
tmpfs, ramfs, overlay, squashfs, the synthetic ones is a … filesystem, which is not a medium
Under /tmp, /run, /dev, /proc, /sys which this host clears or recreates
The filesystem the sets live on on the same filesystem as the sets it would copy (the original check, kept)

The middle one exists because a host with a separate /tmp partition passes the other two. Then a warning, not a refusal, when the destination does not report as removable: a second internal disk and a network mount are legitimate for someone who has decided so, and no check here can establish what the ADR actually requires — that the medium leaves the house.

The self-test is where this was really written in

It handed the copy fixtures /dev/shm and called it the medium, on the reasoning that it is a different filesystem from /tmp. True, and the exact mistake — the trap got written into the tests instead of caught by them.

A real stand-in needs a loop mount, which needs root, which CI does not have. So the copy fixtures set OFFSITE_UNSAFE_SKIP_MEDIUM_CHECKS, which prints a loud warning on any run that sets it, and every refusal gets a fixture with it unset — tmpfs first. The same-filesystem fixture moves out of /tmp so the volatile-path check does not shadow the one under test. 24 fixtures, up from 22.

Documents

ADR-0048 is immutable, so it gains a note recording the correction rather than an edit, plus a second note saying what no check can establish. The runbook now says plainly what a green line does not prove, with /dev/shm named as the case that bit, and the failure table gains the three new symptoms. The same overclaim is corrected in schedule-maintenance.md, install-timers.sh and backup.rules.yaml.

Done on the host, outside this PR

  • The 1.7 GB tmpfs copy removed; memory went from 216 MB free back to 1.9 GB.
  • offsite-copy.prom removed, so the false success no longer vouches for anything. No copy has been made, which is again what the metrics say.

#573's first real visit has still not happened.

Refs #573.

🤖 Generated with Claude Code

Gerrrt and others added 2 commits September 21, 2026 05:32
…/shm is refused, and the self-test stops calling it a medium

`make backup-offsite DEST=/dev/shm` ran to its green line on 2026-09-21, an
hour after the script merged: 1.7 GB into tmpfs on a host with 216 MB free,
gone on the next reboot, and a recorded `offsite-copy` success buying ninety
days of silence from OffsiteCopyStale for a copy that existed nowhere. The
guard compared filesystems and tmpfs is a different filesystem. That check was
never the claim the ADR made for it.

Three refusals now, ordered so each gives its own diagnosis: an in-memory or
synthetic filesystem (tmpfs, ramfs, overlay and friends) wherever it is
mounted; a destination under /tmp, /run, /dev, /proc or /sys whatever is
mounted there, since a separate /tmp partition passes the other two; and the
filesystem the sets already live on, which is the original device check, kept.
Then a warning — not a refusal — when the destination does not report as
removable, because a second internal disk and a network mount are legitimate
for someone who has decided so, and no check here can establish the property
the ADR actually requires: that the medium leaves the house.

The self-test is where this was really written in. It handed the copy fixtures
/dev/shm and called it the medium, on the reasoning that it is a different
filesystem from /tmp — true, and the exact mistake. A real stand-in needs a
loop mount, which needs root, which CI does not have, so the copy fixtures now
set OFFSITE_UNSAFE_SKIP_MEDIUM_CHECKS (which prints a loud warning on any run
that sets it) and every refusal gets a fixture with it unset, tmpfs first.
The same-filesystem fixture moves out of /tmp so the volatile-path check does
not shadow the one under test. 24 fixtures, up from 22.

ADR-0048 is immutable, so it gains a note recording the correction rather than
an edit. The runbook says plainly what a green line does not prove, and the
failure table gains the three new symptoms. The same overclaim is corrected in
schedule-maintenance.md, install-timers.sh and backup.rules.yaml.

The false success on the monitoring host was cleared by hand and the tmpfs
copy removed; #573's first real visit has still not happened.

Refs #573.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Gerrrt
Gerrrt merged commit 1afd68e into main Sep 21, 2026
3 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/offsite-refuse-tmpfs-9f3c21 branch September 21, 2026 06:20
Gerrrt added a commit that referenced this pull request Sep 22, 2026
…etes

The only conflict is scripts/validate.sh, and both sides edit the block this
branch removes. #610 rewrote the comment above backup-offsite.sh's self-test to
say that /dev/shm is a refusal FIXTURE and not a destination — a different
filesystem that is still this host's RAM, which is the #596 regression — and to
widen the refusal list to an in-memory filesystem and a tree cleared on boot.

Taking this branch's side would have dropped that correction with the comment it
lives in, so it is carried into scripts/self-tests.sh's header instead, which is
where the reason each suite exists now lives. Nothing else on main touches this.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant