Skip to content

setup-environment: bumping cache-version does not give a fresh Conda environment #205

Closed

Description

@quantecon-services

Found while writing the setup-environment chapter (#189, #204).

What happens

In standard mode, the Conda environment cache has one exact key and two restore-keys fallbacks. Each fallback restores the most recent cache whose key starts with it:

key:         conda-<os>-<environment-name>-py<python-version>-<hash of environment>-<cache-version>
fallback 1:  conda-<os>-<environment-name>-py<python-version>-<hash of environment>-
fallback 2:  conda-<os>-<environment-name>-py<python-version>-

cache-version is only in the exact key. When it is bumped from v1 to v2:

  1. The exact key …-<hash>-v2 misses.
  2. Fallback 1 matches …-<hash>-v1 and restores the old environment.
  3. conda env update --prune runs on it, and the job saves the result under …-v2.

So a bump forces one conda env update, which editing environment.yml already does, and never a fresh environment. The input used to say "bump to force a rebuild". #204 corrects the description to match what happens, and this issue fixes the behaviour.

The fallbacks themselves are deliberate: since #33, an edit to environment.yml updates the latest environment instead of rebuilding it. The flaw is only that no fallback includes cache-version, and fallback 1 matches exactly the bump case: the same file under a different version. The harness's "cold" job shows a fallback restoring across keys: it restored conda-Linux-quantecon-py3.13-cc8e…-v1, another run's cache, through fallback 2.

Why it matters

A bump is how a repository asks for a clean slate: when a restored environment is in a state that conda env update only patches. The CHANGELOG has a precedent, the pre-#100 cache path, where a partial restore "could leave a mangled env behind for conda env update to patch". Today the only clean slate is deleting the repository's conda- caches by hand, which needs write access and happens outside code review.

No production consumer is affected yet. lecture-dp and the canary reach setup-environment only through build-jupyter-cache, inside the quantecon-build container, and container mode has no Conda cache. It matters for the templates' standard-runner alternative and for any repository outside QuantEcon.

Fix

Move cache-version ahead of the Python version and the hash, so that the one fallback never leaves a version:

key:       conda-<os>-<environment-name>-<cache-version>-py<python-version>-<hash of environment>
fallback:  conda-<os>-<environment-name>-<cache-version>-py<python-version>-
Change Today After
Bump cache-version Restores the old environment and updates it Nothing matches, so a fresh build
Edit environment.yml Restores the latest environment and updates it Same, within the version
Change python-version or environment-name Fresh build Same
  • The key format changes, so every repository's first standard-mode run after the release builds from scratch once, as after the testing: publish/preview/cache action logic has no automated tests #100 cache-path fix. No repository pays that today.
  • The cache-version description, the chapter's cache-key table and troubleshooting, and QUICK-REFERENCE's cache advice change with the code.
  • It changes behaviour, so it gets a CHANGELOG entry and ships in the next release.

Test

A harness job runs after env-seed, on the same salted fixture, with a salted cache-version, and asserts that conda-cache-hit is empty, a true miss. On the current layout that job fails: fallback 1 finds env-seed's cache and the output is 'false'. The extra cache it saves is about 119 MB per run.

Activity

  1. quantecon-services commented on Sep 28, 2026

    @quantecon-services
    CollaboratorAuthor

    Checked on the canary, in standard mode, against its own environment.yml: the full Anaconda 2025.12 stack. The run pinned setup-environment to #206's merge commit, 560053c (QuantEcon/test-actions-lecture-intro#70, run). Each job salts the file per run.

    Case Expected Got
    Same file, same version Exact hit 'true', and the lectures built on it
    New cache-version Nothing restored Empty, so a fresh build
    Edited file, same version Restored through the fallback seed's …-v1-py3.13-0ca79db8… restored, so 'false'

    So the fix holds on a real lecture environment. The in-repo harness now checks both halves on its fixture: env-version-bump from #206, and env-file-edit in #208.

    The run also found a separate bug, older than #206, which failed the last case after the restore: #209. A restore lands on top of the environment setup-miniconda has just created, so a cache whose versions differ from a fresh conda create produces a mix of both. On the canary that broke pip, and conda env update then failed.


    Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething is wrong or broken

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions