Skip to content

perf(ci): cache the probe's proof, not its images, so a docs-only diff pulls nothing - #625

Merged
Gerrrt merged 1 commit into
mainfrom
gerrrt/probe-proof-cache-602
Sep 22, 2026
Merged

Gerrrt merged 1 commit into
mainfrom
gerrrt/probe-proof-cache-602

Conversation

@Gerrrt

@Gerrrt Gerrrt commented Sep 22, 2026

Copy link
Copy Markdown
Owner

What changed

check_compose_health.py gains --proof-cache <path>. CI restores it with actions/cache and passes it to the probe step. make validate passes no cache path and is unchanged.

Why

What --probe establishes is "binary B execs inside image reference R". Every R here is digest-pinned, so that is a claim about immutable bytes — it cannot change while the digest does not. Re-establishing it on every pull request is what made this step pull roughly 8 GB from ghcr.io for diffs that were Markdown only.

#607 fixed the anonymous half of #602. Measurement since shows that was not enough. On 2026-09-22, run 35776982436 with ghcr.io: authenticated as Gerrrt:

20:00:00  pulling ghcr.io/immich-app/immich-server:v3.2.2@sha256:79cc1623...
20:00:12  registry said toomanyrequests, retrying in 5s
20:00:27  registry said toomanyrequests, retrying in 20s
20:00:47  registry said toomanyrequests, retrying in 60s
20:04:13  could not pull ... toomanyrequests: retry-after: 264.525µs, allowed: 44000/minute

postgres:17-vectorchord exhausted the same budget in the same minute and recovered, while immich-server did not — the retry is a coin toss at this volume. And the registry's own retry-after is 264 microseconds, so there is no honest number a longer pause could be derived from. Option 3 stays rejected, now on measurement rather than on principle.

Caching the images was the wrong shape. The six ghcr.io images alone are over 6 GB (paperless-ngx 2.5 GB, immich-server 2.18 GB, immich-machine-learning 1.29 GB, …). That is most of this repository's 10 GB cache budget, and it would still have to be restored on every run. The proofs are a few KB of digests.

This is not the path filter #602 rules out twice

--probe unconditional unchanged — still fails rather than skips with no daemon
every target accounted for yes; one whose proof is not held is pulled and exec'd as before
only digest-pinned refs cacheable enforced on the way in and out — a tag can move, a digest cannot
prober changes the cache carries a hash of check_compose_health.py; a mismatch discards every entry
pin bump restore-keys restores the previous proofs, so only the digest that moved is pulled

The summary reports carried proofs separately from binaries exec'd, never folded into one number — different claims, and one count covering both is the (24 fixtures) problem of #614.

Blast radius

  • Segmentation unchanged — CI and one script, no host changes.
  • No published ports touched.
  • No credentials touched. .probe-proofs/ is gitignored; it holds digests and binary names, nothing sensitive.

Verification

Measured on all five stacks:

cold   25 proofs established, usual runtime
warm   1 second, 0 pulls, 0 probes — every proof carried

A moved digest re-probes exactly that image and nothing else:

-- 5 of 6 image(s) already proved at this digest — not pulled
1 binary/binaries exec'd inside 1 pinned image(s) in 1s; 5 image(s) carried a proof

Also: --proof-cache without --probe is refused; editing the prober prints "proof cache was written by a different prober — proving again" and reproves all 7; make validate green; lint.sh and check_image_pins.py clean.

Ten new fixtures cover the reuse rules — 21 in this suite, 122 across the repo. They exist because the failure mode is silent and one-directional: a cache trusted too readily prints the green line having probed nothing, which is the state --probe exists to eliminate (#79). They pin that a tag is never cached, that a proof from a different prober is discarded, and that a corrupt, malformed or missing cache is no proofs rather than an exception.

What this does not settle

Whether it actually holds is a CI observation, not a PR claim — the same standard #607 was held to, and #607 did not survive it. Refs #602, so the issue stays open until several pull requests have run clean. The first real evidence is this PR's own second run, when the cache is warm.

actions/cache@v4 is a new dependency; Dependabot's github-actions ecosystem already covers it. #607 avoided an action where a plain command would do — here there is no plain equivalent.

Refs #602

🤖 Generated with Claude Code

…f pulls nothing

What --probe establishes is "binary B execs inside image reference R". Every R
here is digest-pinned, so that is a claim about immutable bytes: it cannot change
while the digest does not. Re-establishing it on every pull request is what made
this step pull roughly 8 GB from ghcr.io for diffs that were Markdown only.

ghcr.io throttled those pulls anonymously on 2026-09-21, and again on 2026-09-22
with a working docker login, where the retry spent its whole 5/20/60 budget and
the fourth attempt still failed. #607 fixed the anonymous half; measurement after
it shows authentication alone does not clear the limit. In that run
immich-server exhausted the budget and failed while postgres exhausted the same
budget in the same minute and recovered, and the registry's own retry-after was
264 microseconds — so there is no honest number a longer pause could be derived
from, which is why option 3 stays rejected (#602).

So cache the proof rather than the image. The six ghcr.io images alone are over
6 GB and would take most of this repository's 10 GB cache budget and still have
to be restored every run; the proofs are a few KB of digests.

Nothing about the check weakens, and this is deliberately not the path filter
#602 rules out twice:

  * --probe stays unconditional and still fails rather than skips with no
    daemon. Every target is still accounted for; one whose proof is not held is
    pulled and exec'd exactly as before.
  * ONLY digest-pinned references are cacheable, on the way in and on the way
    out. A tag can be moved under you between two runs; a digest cannot.
  * The cache carries a hash of this file, so editing the prober voids every
    proof it made rather than leaving stale verdicts in place.
  * restore-keys makes a pin bump cheap rather than total: the exact key misses,
    the prefix key restores the previous proofs, and only the digest that
    actually moved is pulled.

The summary reports carried proofs separately from binaries exec'd, never folded
into one number — they are different claims, and one count covering both is the
"(24 fixtures)" problem #614 was about.

Ten fixtures cover the reuse rules, because the failure mode is silent and
one-directional: a cache trusted too readily prints the green line having probed
nothing, which is the state --probe exists to eliminate (#79). They pin that a
tag is never cached, that a proof from a different prober is discarded, and that
a corrupt, malformed or missing cache is no proofs rather than an exception.

Measured on all five stacks: cold 25 proofs in the usual time, warm 1 s with
zero pulls and zero probes. `make validate` passes no cache path and is
unchanged — a workstation's daemon already holds the images.

Refs #602

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Gerrrt
Gerrrt merged commit c933ccc into main Sep 22, 2026
3 checks passed
@Gerrrt
Gerrrt deleted the gerrrt/probe-proof-cache-602 branch September 22, 2026 21:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant