Skip to content

test-containers-lectures: absorb transient raw-GitHub 429s when building lecture repos #102

Closed

Description

@mmcky

A test-containers-lectures.yml run failed on lectures/hansen_singleton_1983.md in QuantEcon/lecture-python.myst — CellExecutionError / HTTP Error 429: Too Many Requests: a hidden cell fetches its dataset from raw GitHub at build time (lectures/hansen_singleton_1983.md:1432-1439).

The matrix builds several repos at once from GitHub's shared runner IPs, so raw GitHub rate-limits it — non-deterministically: the same notebook passed on the other container in the same run. It surfaced during the Anaconda 2026.06 migration (#95), but the cause is the fetch, not the migration or pandas.

Lecture-side fixes do not work: local _static/ paths break Colab, since QuantEcon/lecture-python.notebooks is a flat .ipynb directory with no _static/ (QuantEcon/lecture-python.myst#972, closed for that reason). It is not lecture-specific either — lectures/hansen_singleton_1982.md:1000, lectures/pandas_panel.md:68,188,505 and lectures/phillips_drifts_volatilities.md:103 fetch over raw GitHub too. Moving datasets to data-lectures does not help: pandas_panel.md already reads from that repo over the same github.com/.../raw/ host.

No retry, backoff or 429 handling exists here. The only mitigation, per-repo serialization at .github/workflows/test-containers-lectures.yml:46-48 (v0.7.0, re-scoped by run id in v0.9.0, #103), pairs the two containers for one repo; the three repos at :55-60 still build in parallel. Flakes also file tracker noise: on any non-workflow_dispatch failure (:138), notify-failure (:132-154) runs scripts/create-ci-failure-issue.sh → "🔴 CI failure: Test Container Builds".

Fix: retry the HTML build stage. With execute_notebooks: "cache" (lectures/_config.yml:21) and lecture-repo/_build/.jupyter_cache persisting across steps in the job — why stages 2 and 3 already skip execution — a retry re-executes only the failed notebooks, not the full 30-60 minute run, so no need to gate on the 429 string.

  • Give the "Build HTML (Execute Notebooks)" step (:95-102) an id and continue-on-error: true, then a conditional second attempt on steps.<id>.outcome == 'failure'
  • Give the retry a distinct failure-artifact-name — build-lectures uploads execution reports under that name on failure (build-lectures/action.yml:38,179), so a repeated name collides with the first attempt's artifact
  • Validate with a maintainer workflow_dispatch after merge — it triggers only on workflow_run and workflow_dispatch (:22-27), so a PR cannot exercise it

Acceptance: a single transient 429 no longer fails the matrix; a persistent failure still does.

While in this workflow: restore the missing matrix repo

lecture-python-programming was dropped from the matrix (:54-64) in an unannounced swap in 81e96ff — unlike lecture-jax, which is commented out at :61-64 with a TODO and an issue link. containers/VALIDATION.md:25,42,46,50 and TESTING.md:11 still advertise the 4-repo matrix, so the only place lecture content executes on the images silently lost a quarter of its coverage while the docs kept claiming it.

It belongs in this PR rather than its own: it is four lines in the file you are already editing, and the repo has six build-time raw-GitHub fetches plus yfinance calls, so the retry above is the prerequisite for adding it without immediately reintroducing the flake. Note it is also the repo pinning anaconda=2026.07 against the containers' 2026.06.

  • Add the lecture-python-programming matrix row (current repo name, no .myst suffix)
  • Correct the 4-repo claims in containers/VALIDATION.md and TESTING.md to match

Activity

  1. added
    bugSomething is wrong or broken
    good first issueSelf-contained and friendly to newcomers
    on Jul 8, 2026
  2. mmcky commented on Jul 22, 2026

    @mmcky
    ContributorAuthor

    Transferring this issue to QuantEcon/actions — on reflection this is only relevant to the container-validation workflow, where the test matrix makes multiple calls to GitHub from shared runner IPs while building several repos and hits raw-GitHub rate limits. The right fix is on the CI/container side (e.g. retries or caching), not in the lectures themselves.

    We won't be changing the data-fetch approach in the lectures: switching to local _static/ paths would break the Colab integration, since the published notebooks (lecture-python.notebooks) are a flat directory of .ipynb files with no _static/ folder — every Colab user would hit a FileNotFoundError. That's why the lectures read vendored data over raw GitHub URLs (see also QuantEcon/lecture-python.myst#972, closed for the same reason). Longer term, a shared data-lectures repository will give us a cleaner home for these datasets.

  3. added
    bugSomething is wrong or broken
    good first issueSelf-contained and friendly to newcomers
    on Jul 22, 2026
  4. added
    infrastructureSubstantial CI / build / deploy / tooling / automation work, or behaviour-preserving restructuring
    and removed
    bugSomething is wrong or broken
    on Aug 11, 2026
  5. changed the title [-]hansen_singleton_1983: build-time `pd.read_csv` of a vendored CSV over a raw-GitHub URL flakes on HTTP 429[/-] [+]test-containers-lectures: absorb transient raw-GitHub 429s when building lecture repos[/+] on Aug 11, 2026
  6. added a commit that references this issue on Sep 28, 2026
  7. added a commit that references this issue on Sep 28, 2026
  8. added a commit that references this issue on Sep 28, 2026
  9. added a commit that references this issue on Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    good first issueSelf-contained and friendly to newcomersinfrastructureSubstantial CI / build / deploy / tooling / automation work, or behaviour-preserving restructuring

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions