Repository navigation
feat(restore-jupyter-cache): the templates restore the execution cache, keyed on the environment too (#270) - #280
Conversation
…ge (#270) The execution cache's key is to hash the environment files, with the build cache's fallbacks, so the jobs that restore it now pass a salted harness/environment.yml, and expect jupyter-cache-{env hash}--{md hash}-{run id}. unit-exec-seed asserts a genuine miss, which the bare jupyter-cache- fallback used to rule out. A new job, unit-exec-env-miss, restores the seed's lectures byte for byte under another environment file and asserts a miss: the key used to hash the lectures alone, and restored the seed's own cache. Its control restores after a lecture edit with the seed's environment, and must fall back to the seed's cache, as nearly every pull request does. bjc-restore-exec checks that build-jupyter-cache saves under the key the restore looks up. The header's second constraint now says why every generated-key restore passes a salted environment. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WT8bMZiq7Bm5J79q1N3EWq
…'s key hashes the environment (#270) The execution cache's key hashed only the lectures, and a restore fell back to any execution cache. .jupyter_cache revalidates each notebook against its code, but not against the packages that executed it, so a pull request that changed environment.yml restored outputs made with the old packages, re-executed nothing, and passed. The build cache's key has never crossed an environment change. build-jupyter-cache now saves the execution cache as jupyter-cache-{env hash}-{update hash}-{md hash}-{run id}, from the same step as the build cache's key, and logs both. restore-jupyter-cache looks it up with the build cache's fallbacks after the lectures' own prefix: jupyter-cache-{env}-{update}-, then jupyter-cache-{env}-. So both types miss on the same changes, and environment and environment-update apply to both. The status report's "normal for" list loses "After lecture content changes (execution cache)", which a lecture change never caused, since it falls back. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WT8bMZiq7Bm5J79q1N3EWq
…270) A restored build cache brings back what the last cache build made, including what the source has deleted since. Step 1 (#275) removes the restored html, but _build/jupyter and _build/latex stay, and a build that makes the notebooks or the PDF over them copies a removed lecture's notebook, or a deleted PDF, into the site. The restored _build saves no work beyond .jupyter_cache: a fresh checkout makes Sphinx rewrite every page. Both templates now restore cache-type 'execution', _build/.jupyter_cache alone, which leaves nothing of the kind, and is about a sixth of the download in lecture-dp. Its key now hashes the environment, so a pull request that changes environment.yml still misses and executes every lecture. The steps are renamed "Restore execution cache", and the headers, prerequisites and the pdf-check hand-off say which cache. cache.yml's header names both caches. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WT8bMZiq7Bm5J79q1N3EWq
…an environment change (#270) The restore chapter recommends the execution cache, and says when the build cache is still the one to restore: a build that offers the PDF or the notebooks without building them. Its minimal and save-cache examples restore `execution`, a new example shows the build cache, and "In the templates" says how a repository that copied an older template moves. The keys table and "Why the keys are what they are" give the execution cache's new key and fallbacks, which never cross an `environment` change; the troubleshooting entries follow. ARCHITECTURE's "Two caches" puts the execution cache first, as the one the templates restore, and the build cache as the default's. The build-jupyter-cache chapter gives the new key. Getting started, composition, publish-gh-pages and build-lectures say the templates restore the execution cache, and the migration note drops any interim prune of the restored _build. TESTING's restore row adds unit-exec-env-miss; PLAN's preview-speed row drops #270. The CHANGELOG's two entries are under Changed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WT8bMZiq7Bm5J79q1N3EWq
There was a problem hiding this comment.
Note
Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.
🟡 Changes recommended
A reviewer-raised documentation nit remains unresolved, where the getting-started guide’s wording about pruning the restored _build is broader than the provided command example that targets _build/html specifically.
1 open issue
What changed in this PR
This pull request updates the QuantEcon GitHub Actions templates and caching actions so CI/publish workflows restore only the Jupyter execution cache (_build/.jupyter_cache) and ensures the execution cache key is environment-aware (includes environment + environment-update hashes). This prevents reusing notebook outputs produced under a different package set after environment changes, while avoiding stale _build artifacts being republished.
Changes:
- Make the execution cache key include environment + update + source hashes, with environment-scoped restore fallbacks.
- Switch
templates/ci.ymlandtemplates/publish.ymlto restorecache-type: executionby default. - Update harness tests to assert correct miss/fallback behavior on environment vs. lecture changes; align docs/architecture/changelog with the new semantics.
Notes (nits):
docs/user/getting-started.md: wording suggests pruning all_build, but the example (jb clean . --html) is HTML-specific.templates/cache.yml: header comment still reads like it only generates the build cache, though it now creates both caches.
Assessment: This pull request is ready to be approved.
| File | Review focus / summary |
|---|---|
templates/publish.yml |
Restore step updated to explicitly restore the execution cache (cache-type: execution) and adjust explanatory comments. |
templates/ci.yml |
Restore step updated to restore execution cache by default; comments updated to reflect env-keyed behavior and expected misses. |
templates/cache.yml |
Template workflow context for producing caches; header wording nit noted re: “build cache” vs. two caches. |
restore-jupyter-cache/action.yml |
Execution-mode key construction and restore fallbacks updated to include environment/update hashes; messaging/docs strings adjusted. |
docs/user/getting-started.md |
Guidance updated to match execution-cache restore in templates; nit noted about _build vs _build/html phrasing. |
docs/user/composition.md |
Documentation aligned so environment files apply to/cache-key both cache types. |
docs/user/actions/restore-jupyter-cache.md |
Expanded guidance/examples/keys table to recommend execution cache and document new key + fallback behavior. |
docs/user/actions/publish-gh-pages.md |
Updated to reflect template behavior restoring execution cache in publish workflow. |
docs/user/actions/build-lectures.md |
Updated wording to avoid implying only the build cache is involved; aligns with new template behavior. |
docs/user/actions/build-jupyter-cache.md |
Updated cache key documentation for execution cache and clarified which cache templates restore. |
docs/dev/TESTING.md |
Test matrix updated to include execution-cache env-miss coverage (e.g., unit-exec-env-miss). |
docs/dev/PLAN.md |
Planning note updated to reflect completion of the related work item (drops #270 reference). |
docs/dev/ARCHITECTURE.md |
“Two caches” documentation revised to describe execution cache first and its updated keying/restore behavior. |
CHANGELOG.md |
Unreleased entries added documenting execution-cache key change and template behavior update. |
build-jupyter-cache/action.yml |
Cache key generation consolidated; execution cache save now uses the computed env-aware key; logs updated. |
.github/workflows/test-actions.yml |
Harness extended/updated to assert new execution-cache key format and correct miss/fallback behavior. |
Review effort: Balanced
…ader both caches (#270) Getting started's migration note said to drop an interim prune of the restored `_build`, while its example, `jb clean . --html`, prunes `_build/html` alone; the note names `_build/html` again. cache.yml's header still spoke of one cache in its title and its success and failure lines; it now speaks of both. From Copilot's review of #280. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WT8bMZiq7Bm5J79q1N3EWq
|
Copilot's review summary also noted that Claude Code (on behalf of mmcky) |
Brings in #280 (#270). CHANGELOG: main's ### Changed under [Unreleased] stays first, and this branch's ### Fixed follows it. PLAN.md: main's Preview speed row, without the #216 row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WT8bMc6gKgTU85ZHgMKnXB
Brings in #280, #281, #283 and #284. CHANGELOG keeps main's [Unreleased] entries and puts #142's in Changed; PLAN keeps main's rows and drops #142 from the Container validation row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WT8bMZzgrBo6FK6GsstoUY
* release: v0.17.0 Cut [0.17.0] - 2026-11-10 under an empty [Unreleased]: setup-latex (#283), setup-environment's install-latex on the LaTeX cache (#284), the execution cache keyed on the environment and the templates restoring it (#280), preview-cloudflare's deploy-url (#281), and :latest promoted only after the smoke test (#274). PLAN's release line: 9 actions with setup-latex, v0.17.0 the latest release, @v0 on v0.16.0 until this release's canary and the lecture matrix have passed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WT8bMbZoC2dpGt3sPyBjyd * docs(plan): format the release line's v0.17.0 as code, as its other versions are Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01WT8bMbZoC2dpGt3sPyBjyd --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

#160's step 1 (#275) made a build-cache restore remove the restored
<path>/html. That stopped a deleted page or static file from being published again. It left_build/jupyterand_build/latex, as the last cache build made them, including a removed lecture's notebook and a deleted PDF. A workflow that buildsjupyterorpdflatexover the restored build cache, and copies their output into the site, still publishes those. The canary'sci.ymldoes that, and so does either template with its download steps uncommented. The rest of the restored_buildsaves no work: a fresh checkout gives every source file a new modification time, so Sphinx rewrites every page. Only_build/.jupyter_cachespares anything, by sparing notebook execution.So this PR is step 2: the templates restore
cache-type: 'execution',_build/.jupyter_cachealone. That could not be done as things stood, because of how the two caches were keyed:environment;So under
execution, a pull request that changedenvironment.yml, an Anaconda baseline bump for one, restored outputs made with the old packages, re-executed nothing, and passed. This PR gives the execution cache the build cache's environment keying first. The maintainer agreed the design and its three decisions on #270, comment.cache-type: executionbuild-jupyter-cachesavesjupyter-cache-<md>-<run id>jupyter-cache-<env>-<update>-<md>-<run id>jupyter-cache-(any execution cache)jupyter-cache-<env>-<update>-, thenjupyter-cache-<env>-: the build cache's, exactlyenvironmentbuilddoesenvironment-updatefilejupyter-cache-<env>-, asbuilddoes (the maintainer's decision 3)environment,environment-updateinputs<env>,<update>and<md>are the hashes ofenvironment,environment-updateand the.mdfiles undersource-dir.restore-jupyter-cache's default staysbuild, and its build restore is unchanged.Changes
build-jupyter-cache/action.ymlENV_HASH,UPDATE_HASHandSOURCE_HASHthroughenv:, and logs both. "Save execution cache" takessteps.cache-key.outputs.exec-keyin place of its inlinehashFiles. Theenvironmentandenvironment-updatedescriptions say the hash is part of both caches' keys. No new input or output:cache-keyis still the build cache'srestore-jupyter-cache/action.ymlenvironmentandenvironment-updatedescriptions lose "Withcache-type: build" and "Ignored for the execution cache". The status report's "No cache found" list loses "After lecture content changes (execution cache)". A lecture change never caused a miss, since it falls back, and "After environment.yml changes" now covers both typestemplates/ci.ymlcache-type: 'execution'and a comment on why. The header, the prerequisites, the pdf-check hand-off comment and theexecute_notebooksnote say which cachetemplates/publish.ymlwith: cache-type: 'execution'and the same comment. The header changes to matchtemplates/cache.ymlinstall-latexlines are untouched, since the latex session may edit them.github/workflows/test-actions.ymldocs/user/actions/restore-jupyter-cache.md• recommends
execution, and says when the build cache is still the one: a build that offers the PDF or the notebooks without building them;• restores
executionin its minimal andsave-cacheexamples, and adds an example for the build cache;• "In the templates" says how a repository that copied an older template moves;
• the keys table, "Why the keys are what they are", the last paragraph of "The restored HTML is removed" and three troubleshooting entries follow
docs/user/actions/build-jupyter-cache.mddocs/dev/ARCHITECTURE.mddocs/user/getting-started.md,composition.md,actions/publish-gh-pages.md,actions/build-lectures.mdbuild-lectures.mdis the chapter only, not the actiondocs/dev/TESTING.mdrestore-jupyter-cache's row addsunit-exec-env-missdocs/dev/PLAN.mdCHANGELOG.md[Unreleased],### Changed, with noThe tests
jupyter-cache-fallback. It now says why every restore with generated keys passes a saltedenvironment. Both types' fallbacks now include its hash, and with no file the hash is empty, so the fallback would match other runs' caches.unit-exec-seedwrites and passes a saltedharness/environment.yml, and asserts a genuine miss, asunit-build-seeddoes.unit-exec-restoreandunit-exec-restore-savepass it, and expect the new key.unit-exec-env-miss. It restores the seed's lectures, byte for byte, under a differently salted environment file, and asserts a miss with no_build/.jupyter_cache. Then, as the control, it restores with the seed's environment and an edited lecture, and asserts a fallback hit underjupyter-cache-<env>--with the seed's STAMP. That control is what nearly every pull request does. It is inharness-summary'sneeds:beside the otherunit-exec-*jobs.bjc-restore-execpasses the fixture'senvironment, and expects the new key.bjc-smokeruns this PR's ownbuild-jupyter-cache, whose save steps are its own, not@v0's, so this round trip shows the save and the restore agree.Callers
I checked every QuantEcon workflow that calls either action, through code search in the org and across GitHub, and read each hit's workflows through the API:
lecture-dp,ci.ymlandpublish.ymlbuildv0moves, itscache.ymlsaves the execution cache under the new key, which it does not restoreci.ymlcache-type: 'build', withsave-cache: 'true'test-lecture-theme-sphinxbuild-jupyter-cacheonlyNothing outside QuantEcon calls either action. No workflow restores⚠️ marker. A workflow that did would miss once after
execution, which is why the CHANGELOG entry carries nov0moves, until its next cache build: slower, not failing, unless it also setsfail-on-miss: 'true'. The entry says so.A repository that copied an older template keeps the build cache, with step 1's prune, until it changes:
ci.yml,cache-type: 'build'becomescache-type: 'execution'on the restore step;publish.yml, the restore step, which passes no inputs, gainswith: cache-type: 'execution'.If its
cache.ymlgivesbuild-jupyter-cacheanenvironmentorenvironment-update, the restore steps should be given the same, as before. A site that offers the PDF or the notebooks from the build cache without building them keepsbuild.What I checked, and how
The issue, against
origin/mainate40924a, in the agreed check, posted on #270. Every claim held, but most cited lines had moved since9644f17:restore-jupyter-cache/action.yml:167-185;templates/ci.yml:157-162andtemplates/publish.yml:91-92;:753,:786and:2646.The check found two things the issue did not say:
environment.yml, so the hash would have been empty.Storage, for M3 (
gh api …/actions/caches):lecture-dplecture-python.myst's zipped_buildartifact is about 300 MiB. The maintainer kept the full_buildsave unchanged, sincerestore-jupyter-cache's default restores it. Retiring it, or making it opt-in, goes with a later change of that default.The new checks fail on today's actions. I pushed the harness commit alone, 8d1f898, on
main, to a throwaway branch, and dispatched the harness. Two jobs failed at their new assertions:unit-exec-seedreportedcache-hit: expected [false], got [true] with key [jupyter-cache-0402c945…-42316732919]. That is another run's execution cache frommain, through the bare fallback.bjc-restore-execreportedcache-key: expected [jupyter-cache-c263993c…--c0d06946…-42345491853], got [jupyter-cache-c0d06946…-42345491853]. Today's save key has no environment hash.The seed's failure skipped its three dependents. So I pushed one more commit to the throwaway branch, for the demonstration only:
continue-on-erroron the seed's miss assertion, so the dependents run against today's actions too. That run failed them all:unit-exec-env-miss, the new test:cache-hit: expected [false], got [true] with key [jupyter-cache-3fc5c339…-42345836349]: outputs executed under another environment were restored (#270). Today's key hashes the lectures alone, so it restored the seed's own cache under a different environment file.unit-exec-restoreandunit-exec-restore-save:cache-key: expected [jupyter-cache-46c0469a…--3fc5c339…-42345836349], got [jupyter-cache-3fc5c339…-42345836349].bjc-restore-exec: as in the first run.I then deleted the throwaway branch.
Locally, with actionlint 1.7.12 (its release's SHA-256 checked) and shellcheck 0.11.0:
python3 scripts/generate-docs.py --check --check-examples --actionlint …,actionlintandpython3 scripts/lint-shell.pyare clean, and the CHANGELOG diff adds lines only under[Unreleased].This branch's harness run is this PR's own check: every changed job must pass with the new keys.
What to watch after merge
Nothing runs at merge. The PR changes no
containers/**file, and neithersetup-environmentnorbuild-lectures, so it needn't wait for Monday's lecture-matrix run. The templates reach a repository only when copied, and the key change reaches callers whenv0moves to the release that carries it. The harness onmainsaves new-format keys from the merge on.For the staged release, the canary should exercise this. On a staging branch, switch the canary's
ci.ymltocache-type: 'execution', keepingsave-cache: 'true', then show:cache.ymlrun logs the execution cache key asjupyter-cache-<env>--<md>-<run id>.executionrestore with a matched key, and its build executes only the lectures it changed._staticfile gets a preview where the lecture's page, its_notebooks/<lecture>.ipynband the static file return 404, while a control page returns 200. The notebook is the case step 1 could not fix: underbuild, the canary buildsjupyterover the restored cache, and the notebook comes back.environment.ymlreports "No cache found", and executes every lecture.Not in this PR
restore-jupyter-cache's default frombuildtoexecution. It is a separate decision, best taken oncelecture-dpand the canary have run onexecutionfor a while. It would need a_buildsave's future goes with it.lecture-dpand the canary toexecution, a one-line edit each, after the release. The canary's is the organiser's at release time.environment-update, which it shares with the build cache. A stricter rule for both would mean the templates' restore steps passingenvironment-updatewhenevercache.ymldoes.Closes #270.
🤖 Generated with Claude Code
https://claude.ai/code/session_01WT8bMZiq7Bm5J79q1N3EWq