Repository navigation
fix(ci): alert when a lecture-matrix leg times out - #140
Merged
Merged
Conversation
The weekly container-validation matrix has been non-green since 2026-08-03 and filed nothing. Three consecutive sweeps (08-03, 08-05, 08-10) had both `lecture-python.myst` legs hit the 120-minute ceiling while every other leg passed, and `Report failure` was skipped every time. A job killed by `timeout-minutes` concludes *cancelled*, not *failed*, so `failure()` is false and the guard is silently disarmed by exactly the regression the matrix exists to catch — a build slow enough to stop finishing. Same class as #83 and #103; the same guard-polarity lesson #123 already encoded in `build-jupyter-cache`, never carried across to here. `always()` is needed because `failure()` is false once a needed job is cancelled. A run superseded by the workflow-level `cancel-in-progress` now reaches the notify job too — accepted deliberately: supersedes are rare on a weekly `workflow_run` trigger, the issue is deduplicated rather than duplicated, and a spare issue is far cheaper than another three-week silence. `create-ci-failure-issue.sh` carried the same defect independently: it listed only legs with `conclusion == "failure"`, so a timed-out leg produced an empty list under a "failed" heading. It now includes cancelled legs, labels them, and reports each leg's duration so slow drift is visible in the issue itself. Does not address why `lecture-python.myst` roughly doubled from ~54m to >120m over two weeks, or the fact that two of six legs now sit within ten minutes of the ceiling. Both are diagnosis, tracked separately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
7 tasks
Contributor
There was a problem hiding this comment.
Pull request overview
Restores reliable failure alerting for the weekly container-validation lecture matrix by ensuring timeouts (job cancelled due to timeout-minutes) trigger notifications and are clearly reported in the created tracking issue.
Changes:
- Update the notification job guard to alert on both
failureandcancelledresults from the matrix job, usingalways()so it still runs when upstream is cancelled. - Improve the failure-issue body generation to include cancelled legs and report per-leg durations, making timeout/slow-drift regressions visible directly in the issue.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
scripts/create-ci-failure-issue.sh |
Include cancelled jobs in the reported “failed jobs” list and add duration reporting per job. |
.github/workflows/test-containers-lectures.yml |
Make the Report failure job run for both failed and timed-out (cancelled) matrix outcomes, with rationale documented inline. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
9 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The weekly container-validation matrix has been non-green since 2026-08-03 and filed nothing. Found while assessing the July testing audit.
lecture-python.mystlegsReport failureLast green sweep was 2026-07-27. No failure issue exists.
Cause
A job killed by
timeout-minutesconcludescancelled, notfailure. The guard was a barefailure(), which is false in that case — so the alerting is silently disarmed by precisely the regression this matrix exists to catch: a build that got slow enough to stop finishing. Every other leg passed, so nothing else signalled either.This is the third instance of the class in this repo — #103 (nine weeks green-but-cold) and #83 (two months of silent cache failures) — and it is the same guard-polarity lesson #123 encoded in
build-jupyter-cache, which was never carried across to this workflow.Two independent defects, both fixed
The guard. Now
always() && … && (needs.test.result == 'failure' || needs.test.result == 'cancelled').always()is required becausefailure()is false once a needed job is cancelled. The cost is that a run superseded by the workflow-levelcancel-in-progressalso reaches the notify job. That is accepted deliberately and commented in place: supersedes are rare on a weeklyworkflow_runtrigger, the filed issue is deduplicated rather than duplicated, and a spare issue is much cheaper than another three-week silence.The alert body.
create-ci-failure-issue.shfiltered legs onconclusion == "failure"alone, so a timed-out leg produced an empty list under a "failed" heading — the alert would have fired with nothing in it. It now includes cancelled legs, labels them, and reports each leg's duration, so slow drift is visible in the issue rather than requiring a click into the run. Verified against run 31350902651, where it renders:shellcheckandactionlintboth clean on the changed files.Deliberately not in scope
Why
lecture-python.mystroughly doubled from ~54m to >120m in two weeks, and the fact that two of six legs now sit within ten minutes of the ceiling. Those are diagnosis and capacity decisions, tracked separately — this PR only restores the signal that should have surfaced them three weeks ago.🤖 Generated with Claude Code