Skip to content

test(ci): every fixture suite runs on the pull request, and the counts stop being decorative - #618

Merged
Gerrrt merged 4 commits into
mainfrom
gerrrt/self-tests-ci-614
Sep 22, 2026
Merged

Gerrrt merged 4 commits into
mainfrom
gerrrt/self-tests-ci-614

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Sep 22, 2026

Copy link
Copy Markdown
Owner

What changed

scripts/self-tests.sh discovers and runs every embedded fixture suite in the repository. scripts/validate.sh and .github/workflows/ci.yml both delegate to it, and it asserts both callers the way lint.sh asserts its own.

The head_ "Collector parsers" section in validate.sh — eight hand-written blocks — becomes a three-line delegation with a --skips-file, mirroring the Lint section exactly.

Why

The nine suites are the only test their scripts have. No host here can be asked for a pending security update, a pfSense system upgrade or a drive with 850 pending sectors on demand, and VLAN 99 cannot open TCP/22 to VLAN 30 to watch the real output even once. All nine ran in validate.sh and in no CI job, so they guarded a make validate a contributor may not run and guarded the pull request not at all — the #68 asymmetry, for the third time. This is the second and third bullets of #614.

Discovered, not listed

A hand-kept list is a second copy of one fact, and this repository had already paid for it:

scripts/collect-gateway-state.sh has a --self-test documented in its own usage at line 38 that validate.sh never called and no CI job ever ran.

Nothing said so. It is one of the nine now, and it passes. A script counts as a suite if it dispatches --self-test — four shapes in this tree — so a usage line or a comment cannot be mistaken for one, and a script that advertises the flag without handling it fails loudly when run.

The counts stop being decorative

PASS backup-offsite.sh --self-test (24 fixtures) was printed whether 24 fixtures ran or 22 did: two skip when $HOME is not writable or /dev/shm is absent, and the suite still exits 0. That is #614's first bullet, and the two that can vanish are the same-filesystem refusal and the tmpfs refusal, which is the #596 regression itself.

Counts are now read from each suite's own PASS lines. With HOME=/nonexistent the line reads 23, the skip is printed instead of discarded into /dev/null, and validate.sh folds it into SKIPPED so the summary stays true. --require-all makes it a failure — the contract lint.sh and install-timers.sh already use.

Blast radius

  • Segmentation unchanged — CI and local tooling only, no host changes.
  • No published ports touched.
  • No credentials touched. Every suite already ran locally under make validate; none needs a daemon, a network or a secret.

Verification

Verified by breaking things, not by watching them pass:

Break Result
a fixture given a wrong expectation job fails, the suite's own diagnostic surfaced indented
HOME=/nonexistent count drops to 23, SKIP printed; under --require-all, red
either caller removed caller assertion fails, naming the missing one

Plus: ./scripts/self-tests.sh → 94 fixtures across 9 suites; ./scripts/lint.sh clean; make validate passes (3 skips, all "not the deployment checkout").

One aside worth recording: the first make validate failed on observability: promtool check config, which re-ran green three times in a row and is the known local flake under a busy shared docker daemon — nothing in this diff is Prometheus-related.

Known limit, stated rather than left to be found

The count is descriptive, not a target: a fixture deleted from a suite lowers the number and nothing objects, because nothing knows what the number should be. Pinning the total would be a counted claim of the kind check_docs.py keeps, and it is a deliberate non-goal — the cost is a bump on every PR that adds a fixture, and the case it catches is one the diff already shows. What this does catch is what reaches CI silently: a fixture that ran and disagreed, and a fixture that did not run at all. The script header says so.

Interaction with #615

#615 adds a tenth suite, check_compose_health.py --self-test, with its own dedicated CI step. Whichever merges second, discovery picks that suite up automatically — its dispatch shape is already one of the four matched here. The dedicated step then becomes redundant and can be dropped; it is fast and harmless until then.

Refs #614, #68

🤖 Generated with Claude Code

…s stop being decorative

The nine embedded fixture suites are the only test their scripts have — no host
here can be asked for a pending security update, a pfSense system upgrade or a
drive with 850 pending sectors, and VLAN 99 cannot open TCP/22 to VLAN 30 to
watch the real output once. All nine ran in scripts/validate.sh and in no CI
job, so they guarded a `make validate` a contributor may not run and guarded the
pull request not at all. That is the #68 asymmetry, for the third time (#614).

scripts/self-tests.sh discovers and runs them; validate.sh and ci.yml both
delegate to it, and it asserts both callers the way lint.sh asserts its own.

Discovered rather than listed, because a hand-kept list is a second copy of one
fact and this repository had already paid for it: collect-gateway-state.sh has
a --self-test documented in its own usage at line 38 that validate.sh never
called and no CI job ever ran. Nothing said so. It is one of the nine now. A
script counts as a suite if it DISPATCHES the flag, which is four shapes in
this tree, so a usage line cannot be mistaken for a suite.

The counts are derived from each suite's own PASS lines instead of being typed
into validate.sh. `PASS backup-offsite.sh --self-test (24 fixtures)` was printed
whether 24 fixtures ran or 22 did: two of them skip when $HOME is not writable
or /dev/shm is absent, and the suite still exits 0. With $HOME unwritable the
line now reads 23, the skip is printed rather than discarded into /dev/null, and
validate.sh folds it into the SKIPPED count so the summary stays true.

--require-all makes such a skip a failure, which is the contract lint.sh and
install-timers.sh already use and what #614 asked for: in CI the same-filesystem
and tmpfs refusals must run, the second being the #596 regression itself.

Verified by breaking things rather than by watching them pass: a fixture given
a wrong expectation fails the job and its own diagnostic is surfaced indented;
$HOME=/nonexistent turns the count to 23 and, under --require-all, red; removing
either caller fails the caller assertion. A deleted fixture only lowers the
count, and the header says so rather than leaving it to be discovered — pinning
the total would be a counted claim, and that is a deliberate non-goal.

The section is renamed from "Collector parsers", which had stopped being true:
two of the suites are neither collectors nor parsers.

Refs #614, #68

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Gerrrt and others added 2 commits September 22, 2026 19:26
…etes

The only conflict is scripts/validate.sh, and both sides edit the block this
branch removes. #610 rewrote the comment above backup-offsite.sh's self-test to
say that /dev/shm is a refusal FIXTURE and not a destination — a different
filesystem that is still this host's RAM, which is the #596 regression — and to
widen the refusal list to an in-memory filesystem and a tree cleared on boot.

Taking this branch's side would have dropped that correction with the comment it
lives in, so it is carried into scripts/self-tests.sh's header instead, which is
where the reason each suite exists now lives. Nothing else on main touches this.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…iring goes

#615 landed check_compose_health.py --self-test with a step of its own in ci.yml
and a block of its own in validate.sh. Its dispatch shape is one of the four
scripts/self-tests.sh matches, so it is discovered with the other nine and both
copies are redundant — running it twice locally and reporting a hand-typed
"(11 fixtures)" beside a derived one. Both are removed; the suite still runs in
both places, through the discovery.

The ordering #615 asked for is kept rather than dropped with the step. Its
comment wanted the retry fixtures to report BEFORE the six minutes of registry
traffic the probe step spends, and the self-tests step sat after it. That step
now runs first among the real work in the validate job, ahead of the .env seed
and every pull — it needs no daemon, no network and no secret, and finishes in
seconds. What #615 wanted for one suite, all ten now get.

The conflict in validate.sh is the collector section this branch deletes, taken
from this side. Worth recording what main had done to it meanwhile: the
backup-offsite count was hand-bumped from 24 to 31. That is the second such bump
since this branch opened and precisely the maintenance this removes — the
derived count reads 31 here without anyone having typed it.

112 fixtures across 10 suites.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Gerrrt

Gerrrt commented Sep 22, 2026

Copy link
Copy Markdown
Owner Author

Updated for main now that #615 has merged, and its dedicated wiring is dropped as anticipated.

check_compose_health.py --self-test arrived with a step of its own in ci.yml and a block of its own in validate.sh. Its dispatch shape is one of the four self-tests.sh matches, so it is discovered with the other nine — both copies were redundant, running it twice locally and printing a hand-typed (11 fixtures) beside a derived one. Both are gone. The suite still runs in both places, through the discovery.

#615's ordering is kept rather than dropped with its step. That comment wanted the retry fixtures reported before the six minutes of registry traffic the probe step spends, and the self-tests step sat after it. It now runs first among the real work in the validate job — ahead of the .env seed and every pull — since it needs no daemon, no network and no secret and finishes in seconds. What #615 wanted for one suite, all ten now get.

The count bumped again while this branch was open. main moved backup-offsite.sh from (24 fixtures) to (31 fixtures) by hand — the second such bump since this opened, and exactly the maintenance this removes. The derived line reads 31 without anyone having typed it:

  PASS backup-offsite.sh --self-test (31 fixtures)
  PASS check_compose_health.py --self-test (11 fixtures)
  ...
112 fixtures across 10 suites passed

Re-verified after the merge: lint.sh clean, make validate green with check_compose_health.py --self-test reporting exactly once, ci.yml parses, both scripts pass bash -n.


Generated by Claude Code

… so they cannot no-op

`printf 'x' | dd ... seek=10 conv=notrunc` writes 'x' at byte 10 of a file made
of /dev/urandom. One run in 256 per site, that byte already IS 'x': the file
does not change, the verify correctly reports no corruption, and the fixture
that exists to prove corruption is caught fails instead. Two tamper sites, so
about one run in 128.

Harmless while this suite ran only under `make validate`, which is where it has
been since it shipped. Putting it in front of every pull request turned it into
CI going red for a reason no diff could cause — the #602 complaint exactly — and
it landed on the first run that gated one, on the export site:

  FAIL a tampered export on the medium fails the verify
       exit 0, wanted 1; wanted output containing: differs from its recorded sha256
       | every set on the medium verifies — 3 item(s)

The tamper now reads the byte and writes (byte + 1) % 256, which changes the
file whatever was there and leaves its length alone. Verified against 0x78, the
value that broke it, and against 0x00 and 0xff for the wraparound; 40 consecutive
self-test runs, no failures.

Worth stating plainly, because it is the argument for #614: this is a fixture
that could pass without testing anything, in the suite that proves an offsite
backup has not been corrupted, and nothing had ever run it anywhere it would be
noticed.

Refs #614

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Gerrrt
Gerrrt merged commit 15511b0 into main Sep 22, 2026
3 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/self-tests-ci-614 branch September 22, 2026 20:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant