Repository navigation
Description
lecture-python.myst in the weekly container-validation matrix roughly doubled from ~54 minutes to over 120 over two weeks, and now hits the job ceiling. #140 restores the alerting that should have surfaced this three weeks ago; this issue is the diagnosis and the capacity decision it exposed.
What the run history shows
Last green sweep was 2026-07-27. Durations for lecture-python.myst, both images, since:
| Sweep | · quantecon |
· quantecon-build |
|---|---|---|
| 2026-07-27 | green | green |
| 2026-08-03 | 120m (ceiling) | 120m (ceiling) |
| 2026-08-05 | 120m (ceiling) | 120m (ceiling) |
| 2026-08-10 | 109m | 120m (ceiling) |
The other two repos are stable and nowhere near the ceiling: lecture-python-intro 12–19m, lecture-python-advanced.myst 43–52m. So this is specific to one repo, not a runner-wide or image-wide slowdown.
The 08-10 leg finishing at 109m is the more alarming datapoint than the timeouts — it means the job now passes or fails on the margin, so the matrix will flap rather than fail cleanly.
What to work out
- Whether the cause is upstream content (lectures added or made heavier in
lecture-python.myst), the container baseline, or execution behaviour — agit logover that repo'slectures/across the window, against a manual dispatch at a pinned older image, separates the two - Whether both images regressed by the same factor, which would point at content rather than the image
- Whether the GPU-dependent lectures are executing on CPU runners and dominating the time
The capacity decision this forces
timeout-minutes: 120 at .github/workflows/test-containers-lectures.yml:37 is a single ceiling shared by every leg, sized for the smallest. Options, in rough order of preference:
- Split the three builders into separate jobs, so
html(where execution happens) is timed and reported independently ofpdflatex/jupyterrendering — also makes the next regression legible without a stopwatch - Raise the ceiling for the known-heavy repos only, per matrix entry rather than per workflow
- Reduce what this repo validates weekly, if the answer is that a full three-builder pass over the largest lecture repo is no longer the right weekly cost
Preventive half
Slow drift is invisible here by construction: the only symptom is duration, and nothing records it. #140 now reports per-leg duration inside the failure issue, which helps only once something already broke.
- Record per-leg durations to
$GITHUB_STEP_SUMMARYon every run, and warn when a leg exceeds the previous successful run's time by a threshold
Comparing against the previous successful run (via the runs API) rather than a checked-in baseline is deliberate: these durations track upstream lecture content this repo does not control, so a committed baseline would go stale on every upstream lecture addition and become noise nobody maintains — which is how containers/VALIDATION.md's timing table died.
This is the third silent-duration failure in this repo, after #103 (nine weeks of cold-but-green container builds) and #83 (two months of 40-minute cold lecture builds). In all three the only observable symptom was time.
Activity
ci(lecture-validation): a per-repo timeout, 180 min for lecture-pytho…
mmcky commentedon Nov 9, 2026
Checked against
mainat5a80d5c, from the jobs API for runs 60–79. The capacity half of this issue is done: #197 made the ceiling per repository, 180 min forlecture-python.myst, and logs each runner's CPU. The preventive half is still needed.lecture-python.myst's green legs crept from 129–130 min on 2026-10-05 to 174 min on 2026-11-05, 97% of its limit, and nothing short of the limit would have reported it. 13 of the 17 green legs since #197 would have hit the old 120.Two parts of the proposal change. Splitting the builders saves at most 10 min a leg, since the HTML step is 72–163 of it, and the API already gives each step's time. Comparing against the previous successful run is too noisy: two legs of one run, on identical inputs, took 130 and 79 min, so the median of a leg's last five green runs is the baseline instead.
The maintainer's decisions, 2026-11-10, taking the recommended set:
durationsjob after the matrix writes each leg's minutes, its HTML minutes, its share of the ceiling and its change against that median to the run's summary;lecture-python.myst's ceiling goes to 240 min, for lecture-python.myst: CPU build time sits at the lecture-validation ceiling #196, to come back down once the lecture is faster.CI only, with no CHANGELOG entry. One PR closes this issue and #196. It merges after #286, on the organiser's word.
Claude Code (on behalf of mmcky), organiser session; the check by the mini:actions-containers session
ci(lecture-validation): say where yq comes from, and what happens wit…
ci(lecture-validation): report each leg's duration and warn as it nea…
docs(plan): #188 is closed; #282 and #291 join the open issues (#292)