Skip to content

fix(ci): alert when a lecture-matrix leg times out - #140

Merged
mmcky merged 1 commit into
mainfrom
fix/container-matrix-timeout-alerting
Aug 11, 2026
Merged

mmcky merged 1 commit into
mainfrom
fix/container-matrix-timeout-alerting

Conversation

@mmcky

@mmcky mmcky commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

The weekly container-validation matrix has been non-green since 2026-08-03 and filed nothing. Found while assessing the July testing audit.

Sweep Run lecture-python.myst legs Report failure
2026-08-03 30781114809 both hit the 120-minute ceiling skipped
2026-08-05 30982565528 both hit the ceiling skipped
2026-08-10 31350902651 one at the ceiling, the other 109m skipped

Last green sweep was 2026-07-27. No failure issue exists.

Cause

A job killed by timeout-minutes concludes cancelled, not failure. The guard was a bare failure(), which is false in that case — so the alerting is silently disarmed by precisely the regression this matrix exists to catch: a build that got slow enough to stop finishing. Every other leg passed, so nothing else signalled either.

This is the third instance of the class in this repo — #103 (nine weeks green-but-cold) and #83 (two months of silent cache failures) — and it is the same guard-polarity lesson #123 encoded in build-jupyter-cache, which was never carried across to this workflow.

Two independent defects, both fixed

The guard. Now always() && … && (needs.test.result == 'failure' || needs.test.result == 'cancelled'). always() is required because failure() is false once a needed job is cancelled. The cost is that a run superseded by the workflow-level cancel-in-progress also reaches the notify job. That is accepted deliberately and commented in place: supersedes are rare on a weekly workflow_run trigger, the filed issue is deduplicated rather than duplicated, and a spare issue is much cheaper than another three-week silence.

The alert body. create-ci-failure-issue.sh filtered legs on conclusion == "failure" alone, so a timed-out leg produced an empty list under a "failed" heading — the alert would have fired with nothing in it. It now includes cancelled legs, labels them, and reports each leg's duration, so slow drift is visible in the issue rather than requiring a click into the run. Verified against run 31350902651, where it renders:

- `lecture-python.myst · quantecon-build` — cancelled after 120m (timed out, or the run was superseded)

shellcheck and actionlint both clean on the changed files.

Deliberately not in scope

Why lecture-python.myst roughly doubled from ~54m to >120m in two weeks, and the fact that two of six legs now sit within ten minutes of the ceiling. Those are diagnosis and capacity decisions, tracked separately — this PR only restores the signal that should have surfaced them three weeks ago.

🤖 Generated with Claude Code

The weekly container-validation matrix has been non-green since 2026-08-03
and filed nothing. Three consecutive sweeps (08-03, 08-05, 08-10) had both
`lecture-python.myst` legs hit the 120-minute ceiling while every other leg
passed, and `Report failure` was skipped every time.

A job killed by `timeout-minutes` concludes *cancelled*, not *failed*, so
`failure()` is false and the guard is silently disarmed by exactly the
regression the matrix exists to catch — a build slow enough to stop
finishing. Same class as #83 and #103; the same guard-polarity lesson #123
already encoded in `build-jupyter-cache`, never carried across to here.

`always()` is needed because `failure()` is false once a needed job is
cancelled. A run superseded by the workflow-level `cancel-in-progress` now
reaches the notify job too — accepted deliberately: supersedes are rare on
a weekly `workflow_run` trigger, the issue is deduplicated rather than
duplicated, and a spare issue is far cheaper than another three-week
silence.

`create-ci-failure-issue.sh` carried the same defect independently: it
listed only legs with `conclusion == "failure"`, so a timed-out leg
produced an empty list under a "failed" heading. It now includes cancelled
legs, labels them, and reports each leg's duration so slow drift is visible
in the issue itself.

Does not address why `lecture-python.myst` roughly doubled from ~54m to
>120m over two weeks, or the fact that two of six legs now sit within ten
minutes of the ceiling. Both are diagnosis, tracked separately.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Restores reliable failure alerting for the weekly container-validation lecture matrix by ensuring timeouts (job cancelled due to timeout-minutes) trigger notifications and are clearly reported in the created tracking issue.

Changes:

  • Update the notification job guard to alert on both failure and cancelled results from the matrix job, using always() so it still runs when upstream is cancelled.
  • Improve the failure-issue body generation to include cancelled legs and report per-leg durations, making timeout/slow-drift regressions visible directly in the issue.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
scripts/create-ci-failure-issue.sh Include cancelled jobs in the reported “failed jobs” list and add duration reporting per job.
.github/workflows/test-containers-lectures.yml Make the Report failure job run for both failed and timed-out (cancelled) matrix outcomes, with rationale documented inline.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@mmcky
mmcky merged commit e56d616 into main Aug 11, 2026
18 checks passed
@mmcky
mmcky deleted the fix/container-matrix-timeout-alerting branch August 11, 2026 02:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants