Skip to content

feat: Add caching support for container mode builds #18

Open

Description

@mmcky

Speed up container-mode setup-environment by avoiding a fresh package install every run. Scope is now narrow: only the per-lecture environment-update delta install is uncached, and no consumer pays for it today.

The common stack is pre-baked on the Anaconda 2026.06 baseline (containers/quantecon/environment.yml:11; lean-image pins at containers/quantecon-build/environment.yml:20-29), reaching consumers via the image tag, not the v0.9.0 release. Undone is the literal ask: analyse packages common across the lecture repos and add them to quantecon-build. containers/quantecon-build/environment.yml:3 still routes every lecture-specific package (quantecon, wbgapi, jax) to the lecture repo's own environment file; that delta install is the whole remaining scope.

The default path is zero-cost: environment-update defaults to '' (setup-environment/action.yml:17) and container mode runs a no-op (setup-environment/action.yml:85-93), shipped in v0.5.0. No consumer passes it — QuantEcon/lecture-dp's ci.yml and cache.yml do not — and the #92 baseline below shows no delta install among the costs.

actions/cache can see /opt/conda — under container: every step runs inside the container, JS actions included — so isolation was never the obstacle. The real obstacles, hit by the #100 harness in standard mode: tar ownership/permissions on restore (setup-environment/action.yml:122-127), and restoring a cached env over the image's own. The workspace-copy workaround (into .cache/, back over site-packages) is out; the pip-download-cache and mounted-volume variants need re-deriving from these. Two shipped comments still assert the wrong rationale (Conda env outside the workspace mount, actions/cache on the host) and link back here.

Next steps

Keep open at low priority, per PLAN.md:115. Origin: PR #17, container-aware setup-environment. #107 (conda env update without -n, wrong env) lands before any work here; #92 has the measured preview-build baselines.

Activity

  1. added
    enhancementImprovement to existing content or functionality
    low-priorityNice to have, no time pressure
    on Aug 11, 2026
  2. quantecon-services commented on Sep 24, 2026

    @quantecon-services
    Collaborator

    Triage note, 2026-09-24: this issue's "correct the rationale shipped in the code comments" item was meant to ride with #107. #107 closed on 2026-09-24 without it. The wrong text is still at setup-environment/action.yml:61-67 ("env outside workspace mount", "actions/cache on host", the unmeasured ~30–60 s) and at setup-environment/README.md:9 and :126-127. That correction is small and needn't wait for the rest of this low-priority issue.


    Generated by Claude Code

  3. mmcky commented on Nov 6, 2026

    @mmcky
    ContributorAuthor

    Checked against main at 9644f17 (2026-11-06). Two of the three wrong texts are gone: setup-environment/README.md is now a generated signpost, and neither its rationale nor the "~30-60 s" figure survives anywhere. The setup-environment chapter now states it correctly: in container mode a delta is installed afresh on every run (docs/user/actions/setup-environment.md:142). The third remains, the comment at setup-environment/action.yml:118-125, which still says the Conda environment "lives outside the workspace mount, so actions/cache cannot persist it". The images are on the Anaconda 2026.07 baseline now, not 2026.06.

    The rest still has no consumer. An org-wide code search finds no workflow outside this repository that passes environment-update. The one measurement, the harness's env-container-update job in run 42210229573, took 13 s to install a one-package pip delta into the lean image, against 89 s to pull the image. Two merges since make the remedies easier: #245 builds and smoke-tests an image change on its pull request, so baking a common package such as quantecon into the images is a tested change; and #255 stopped the scheduled rebuilds, so a cache keyed on the image's digest would stay valid until the next image merge.

    Proposed: fix the comment now, as a one-commit change with no release of its own, saying that container mode caches nothing, and that a restored environment would be several GB and would mix with the image's own (#209). Keep this issue parked with a trigger: when a repository first passes environment-update (most likely lecture-python.myst's full chain, for its ML libraries), time its setup-environment step and its image pull. If the delta is a large share of the run, prefer baking its common packages into the images, then a pip and Conda download cache keyed on the image digest and the delta file, and only last a cache of the environment itself.

    Claude Code (on behalf of mmcky)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementImprovement to existing content or functionalitylow-priorityNice to have, no time pressure

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions