Repository navigation
test(ci): every fixture suite runs on the pull request, and the counts stop being decorative - #618
Conversation
…s stop being decorative The nine embedded fixture suites are the only test their scripts have — no host here can be asked for a pending security update, a pfSense system upgrade or a drive with 850 pending sectors, and VLAN 99 cannot open TCP/22 to VLAN 30 to watch the real output once. All nine ran in scripts/validate.sh and in no CI job, so they guarded a `make validate` a contributor may not run and guarded the pull request not at all. That is the #68 asymmetry, for the third time (#614). scripts/self-tests.sh discovers and runs them; validate.sh and ci.yml both delegate to it, and it asserts both callers the way lint.sh asserts its own. Discovered rather than listed, because a hand-kept list is a second copy of one fact and this repository had already paid for it: collect-gateway-state.sh has a --self-test documented in its own usage at line 38 that validate.sh never called and no CI job ever ran. Nothing said so. It is one of the nine now. A script counts as a suite if it DISPATCHES the flag, which is four shapes in this tree, so a usage line cannot be mistaken for a suite. The counts are derived from each suite's own PASS lines instead of being typed into validate.sh. `PASS backup-offsite.sh --self-test (24 fixtures)` was printed whether 24 fixtures ran or 22 did: two of them skip when $HOME is not writable or /dev/shm is absent, and the suite still exits 0. With $HOME unwritable the line now reads 23, the skip is printed rather than discarded into /dev/null, and validate.sh folds it into the SKIPPED count so the summary stays true. --require-all makes such a skip a failure, which is the contract lint.sh and install-timers.sh already use and what #614 asked for: in CI the same-filesystem and tmpfs refusals must run, the second being the #596 regression itself. Verified by breaking things rather than by watching them pass: a fixture given a wrong expectation fails the job and its own diagnostic is surfaced indented; $HOME=/nonexistent turns the count to 23 and, under --require-all, red; removing either caller fails the caller assertion. A deleted fixture only lowers the count, and the header says so rather than leaving it to be discovered — pinning the total would be a counted claim, and that is a deliberate non-goal. The section is renamed from "Collector parsers", which had stopped being true: two of the suites are neither collectors nor parsers. Refs #614, #68 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…etes The only conflict is scripts/validate.sh, and both sides edit the block this branch removes. #610 rewrote the comment above backup-offsite.sh's self-test to say that /dev/shm is a refusal FIXTURE and not a destination — a different filesystem that is still this host's RAM, which is the #596 regression — and to widen the refusal list to an in-memory filesystem and a tree cleared on boot. Taking this branch's side would have dropped that correction with the comment it lives in, so it is carried into scripts/self-tests.sh's header instead, which is where the reason each suite exists now lives. Nothing else on main touches this. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…iring goes #615 landed check_compose_health.py --self-test with a step of its own in ci.yml and a block of its own in validate.sh. Its dispatch shape is one of the four scripts/self-tests.sh matches, so it is discovered with the other nine and both copies are redundant — running it twice locally and reporting a hand-typed "(11 fixtures)" beside a derived one. Both are removed; the suite still runs in both places, through the discovery. The ordering #615 asked for is kept rather than dropped with the step. Its comment wanted the retry fixtures to report BEFORE the six minutes of registry traffic the probe step spends, and the self-tests step sat after it. That step now runs first among the real work in the validate job, ahead of the .env seed and every pull — it needs no daemon, no network and no secret, and finishes in seconds. What #615 wanted for one suite, all ten now get. The conflict in validate.sh is the collector section this branch deletes, taken from this side. Worth recording what main had done to it meanwhile: the backup-offsite count was hand-bumped from 24 to 31. That is the second such bump since this branch opened and precisely the maintenance this removes — the derived count reads 31 here without anyone having typed it. 112 fixtures across 10 suites. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Updated for
#615's ordering is kept rather than dropped with its step. That comment wanted the retry fixtures reported before the six minutes of registry traffic the probe step spends, and the self-tests step sat after it. It now runs first among the real work in the The count bumped again while this branch was open. Re-verified after the merge: Generated by Claude Code |
… so they cannot no-op `printf 'x' | dd ... seek=10 conv=notrunc` writes 'x' at byte 10 of a file made of /dev/urandom. One run in 256 per site, that byte already IS 'x': the file does not change, the verify correctly reports no corruption, and the fixture that exists to prove corruption is caught fails instead. Two tamper sites, so about one run in 128. Harmless while this suite ran only under `make validate`, which is where it has been since it shipped. Putting it in front of every pull request turned it into CI going red for a reason no diff could cause — the #602 complaint exactly — and it landed on the first run that gated one, on the export site: FAIL a tampered export on the medium fails the verify exit 0, wanted 1; wanted output containing: differs from its recorded sha256 | every set on the medium verifies — 3 item(s) The tamper now reads the byte and writes (byte + 1) % 256, which changes the file whatever was there and leaves its length alone. Verified against 0x78, the value that broke it, and against 0x00 and 0xff for the wraparound; 40 consecutive self-test runs, no failures. Worth stating plainly, because it is the argument for #614: this is a fixture that could pass without testing anything, in the suite that proves an offsite backup has not been corrupted, and nothing had ever run it anywhere it would be noticed. Refs #614 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
What changed
scripts/self-tests.shdiscovers and runs every embedded fixture suite in the repository.scripts/validate.shand.github/workflows/ci.ymlboth delegate to it, and it asserts both callers the waylint.shasserts its own.The
head_ "Collector parsers"section invalidate.sh— eight hand-written blocks — becomes a three-line delegation with a--skips-file, mirroring the Lint section exactly.Why
The nine suites are the only test their scripts have. No host here can be asked for a pending security update, a pfSense system upgrade or a drive with 850 pending sectors on demand, and VLAN 99 cannot open TCP/22 to VLAN 30 to watch the real output even once. All nine ran in
validate.shand in no CI job, so they guarded amake validatea contributor may not run and guarded the pull request not at all — the #68 asymmetry, for the third time. This is the second and third bullets of #614.Discovered, not listed
A hand-kept list is a second copy of one fact, and this repository had already paid for it:
Nothing said so. It is one of the nine now, and it passes. A script counts as a suite if it dispatches
--self-test— four shapes in this tree — so a usage line or a comment cannot be mistaken for one, and a script that advertises the flag without handling it fails loudly when run.The counts stop being decorative
PASS backup-offsite.sh --self-test (24 fixtures)was printed whether 24 fixtures ran or 22 did: two skip when$HOMEis not writable or/dev/shmis absent, and the suite still exits 0. That is #614's first bullet, and the two that can vanish are the same-filesystem refusal and the tmpfs refusal, which is the #596 regression itself.Counts are now read from each suite's own PASS lines. With
HOME=/nonexistentthe line reads 23, the skip is printed instead of discarded into/dev/null, andvalidate.shfolds it intoSKIPPEDso the summary stays true.--require-allmakes it a failure — the contractlint.shandinstall-timers.shalready use.Blast radius
make validate; none needs a daemon, a network or a secret.Verification
Verified by breaking things, not by watching them pass:
HOME=/nonexistent--require-all, redPlus:
./scripts/self-tests.sh→ 94 fixtures across 9 suites;./scripts/lint.shclean;make validatepasses (3 skips, all "not the deployment checkout").One aside worth recording: the first
make validatefailed onobservability: promtool check config, which re-ran green three times in a row and is the known local flake under a busy shared docker daemon — nothing in this diff is Prometheus-related.Known limit, stated rather than left to be found
The count is descriptive, not a target: a fixture deleted from a suite lowers the number and nothing objects, because nothing knows what the number should be. Pinning the total would be a counted claim of the kind
check_docs.pykeeps, and it is a deliberate non-goal — the cost is a bump on every PR that adds a fixture, and the case it catches is one the diff already shows. What this does catch is what reaches CI silently: a fixture that ran and disagreed, and a fixture that did not run at all. The script header says so.Interaction with #615
#615 adds a tenth suite,
check_compose_health.py --self-test, with its own dedicated CI step. Whichever merges second, discovery picks that suite up automatically — its dispatch shape is already one of the four matched here. The dedicated step then becomes redundant and can be dropped; it is fast and harmless until then.Refs #614, #68
🤖 Generated with Claude Code