Skip to content

test-containers-lectures: lecture-python.myst doubled to the 120-minute ceiling #141

Closed

Description

@mmcky

lecture-python.myst in the weekly container-validation matrix roughly doubled from ~54 minutes to over 120 over two weeks, and now hits the job ceiling. #140 restores the alerting that should have surfaced this three weeks ago; this issue is the diagnosis and the capacity decision it exposed.

What the run history shows

Last green sweep was 2026-07-27. Durations for lecture-python.myst, both images, since:

Sweep · quantecon · quantecon-build
2026-07-27 green green
2026-08-03 120m (ceiling) 120m (ceiling)
2026-08-05 120m (ceiling) 120m (ceiling)
2026-08-10 109m 120m (ceiling)

The other two repos are stable and nowhere near the ceiling: lecture-python-intro 12–19m, lecture-python-advanced.myst 43–52m. So this is specific to one repo, not a runner-wide or image-wide slowdown.

The 08-10 leg finishing at 109m is the more alarming datapoint than the timeouts — it means the job now passes or fails on the margin, so the matrix will flap rather than fail cleanly.

What to work out

  • Whether the cause is upstream content (lectures added or made heavier in lecture-python.myst), the container baseline, or execution behaviour — a git log over that repo's lectures/ across the window, against a manual dispatch at a pinned older image, separates the two
  • Whether both images regressed by the same factor, which would point at content rather than the image
  • Whether the GPU-dependent lectures are executing on CPU runners and dominating the time

The capacity decision this forces

timeout-minutes: 120 at .github/workflows/test-containers-lectures.yml:37 is a single ceiling shared by every leg, sized for the smallest. Options, in rough order of preference:

  • Split the three builders into separate jobs, so html (where execution happens) is timed and reported independently of pdflatex/jupyter rendering — also makes the next regression legible without a stopwatch
  • Raise the ceiling for the known-heavy repos only, per matrix entry rather than per workflow
  • Reduce what this repo validates weekly, if the answer is that a full three-builder pass over the largest lecture repo is no longer the right weekly cost

Preventive half

Slow drift is invisible here by construction: the only symptom is duration, and nothing records it. #140 now reports per-leg duration inside the failure issue, which helps only once something already broke.

  • Record per-leg durations to $GITHUB_STEP_SUMMARY on every run, and warn when a leg exceeds the previous successful run's time by a threshold

Comparing against the previous successful run (via the runs API) rather than a checked-in baseline is deliberate: these durations track upstream lecture content this repo does not control, so a committed baseline would go stale on every upstream lecture addition and become noise nobody maintains — which is how containers/VALIDATION.md's timing table died.

This is the third silent-duration failure in this repo, after #103 (nine weeks of cold-but-green container builds) and #83 (two months of 40-minute cold lecture builds). In all three the only observable symptom was time.

Activity

  1. added
    infrastructureSubstantial CI / build / deploy / tooling / automation work, or behaviour-preserving restructuring
    on Aug 11, 2026
  2. mmcky commented on Nov 9, 2026

    @mmcky
    ContributorAuthor

    Checked against main at 5a80d5c, from the jobs API for runs 60–79. The capacity half of this issue is done: #197 made the ceiling per repository, 180 min for lecture-python.myst, and logs each runner's CPU. The preventive half is still needed. lecture-python.myst's green legs crept from 129–130 min on 2026-10-05 to 174 min on 2026-11-05, 97% of its limit, and nothing short of the limit would have reported it. 13 of the 17 green legs since #197 would have hit the old 120.

    Two parts of the proposal change. Splitting the builders saves at most 10 min a leg, since the HTML step is 72–163 of it, and the API already gives each step's time. Comparing against the previous successful run is too noisy: two legs of one run, on identical inputs, took 130 and 79 min, so the median of a leg's last five green runs is the baseline instead.

    The maintainer's decisions, 2026-11-10, taking the recommended set:

    • a durations job after the matrix writes each leg's minutes, its HTML minutes, its share of the ceiling and its change against that median to the run's summary;
    • it warns when a green leg passes 80% of its ceiling, or runs more than 50% and 15 min over its median. The warnings are annotations and summary lines only, and file no issue;
    • lecture-python.myst's ceiling goes to 240 min, for lecture-python.myst: CPU build time sits at the lecture-validation ceiling #196, to come back down once the lecture is faster.

    CI only, with no CHANGELOG entry. One PR closes this issue and #196. It merges after #286, on the organiser's word.

    Claude Code (on behalf of mmcky), organiser session; the check by the mini:actions-containers session

  3. added a commit that references this issue on Nov 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    high-priorityAddress sooninfrastructureSubstantial CI / build / deploy / tooling / automation work, or behaviour-preserving restructuring

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions