Skip to content

🔴 CI failure: Test Container Builds #159

Closed

Description

@github-actions

🔴 Test Container Builds failed

Run 31988218374
Trigger workflow_run
Branch main
Commit e56d616
Failed at 2026-08-17 06:31 UTC

Failed jobs

  • lecture-python.myst · quantecon — cancelled after 120m (timed out, or the run was superseded)
  • lecture-python.myst · quantecon-build — cancelled after 120m (timed out, or the run was superseded)

Nobody was watching this run, so this issue is the alert. The run log shows what broke, and any job that uploads artifacts has attached its build output and failure reports to the run.

Close this issue once the workflow is green again — the next failure opens a fresh one.

Activity

  1. added
    bugSomething is wrong or broken
    infrastructureSubstantial CI / build / deploy / tooling / automation work, or behaviour-preserving restructuring
    on Aug 17, 2026
  2. github-actions commented on Aug 24, 2026

    @github-actions
    Author

    🔴 Test Container Builds failed

    Run 32683799203
    Trigger workflow_run
    Branch main
    Commit e56d616
    Failed at 2026-08-24 06:40 UTC

    Failed jobs

    • lecture-python.myst · quantecon-build — cancelled after 120m (timed out, or the run was superseded)
    • lecture-python.myst · quantecon — cancelled after 120m (timed out, or the run was superseded)

    Nobody was watching this run, so this issue is the alert. The run log shows what broke, and any job that uploads artifacts has attached its build output and failure reports to the run.

    Close this issue once the workflow is green again — the next failure opens a fresh one.

  3. github-actions commented on Aug 31, 2026

    @github-actions
    Author

    🔴 Test Container Builds failed

    Run 33350472160
    Trigger workflow_run
    Branch main
    Commit e56d616
    Failed at 2026-08-31 06:04 UTC

    Failed jobs

    • lecture-python.myst · quantecon-build — cancelled after 120m (timed out, or the run was superseded)

    Nobody was watching this run, so this issue is the alert. The run log shows what broke, and any job that uploads artifacts has attached its build output and failure reports to the run.

    Close this issue once the workflow is green again — the next failure opens a fresh one.

  4. github-actions commented on Sep 7, 2026

    @github-actions
    Author

    🔴 Test Container Builds failed

    Run 34076061797
    Trigger workflow_run
    Branch main
    Commit e56d616
    Failed at 2026-09-07 06:07 UTC

    Failed jobs

    • lecture-python.myst · quantecon-build — cancelled after 120m (timed out, or the run was superseded)

    Nobody was watching this run, so this issue is the alert. The run log shows what broke, and any job that uploads artifacts has attached its build output and failure reports to the run.

    Close this issue once the workflow is green again — the next failure opens a fresh one.

  5. github-actions commented on Sep 18, 2026

    @github-actions
    Author

    🔴 Test Container Builds failed

    Run 35309577029
    Trigger workflow_run
    Branch main
    Commit 3adfcf3
    Failed at 2026-09-18 06:54 UTC

    Failed jobs

    • lecture-python.myst · quantecon — cancelled after 107m (timed out, or the run was superseded)
    • lecture-python.myst · quantecon-build — cancelled after 106m (timed out, or the run was superseded)

    Nobody was watching this run, so this issue is the alert. The run log shows what broke, and any job that uploads artifacts has attached its build output and failure reports to the run.

    Close this issue once the workflow is green again — the next failure opens a fresh one.

  6. github-actions commented on Sep 18, 2026

    @github-actions
    Author

    🔴 Test Container Builds failed

    Run 35316939330
    Trigger workflow_run
    Branch main
    Commit 195b9f2
    Failed at 2026-09-18 10:30 UTC

    Failed jobs

    • lecture-python.myst · quantecon-build — cancelled after 120m (timed out, or the run was superseded)
    • lecture-python-intro · quantecon-build — failure after 12m
    • lecture-python.myst · quantecon — failure after 95m

    Nobody was watching this run, so this issue is the alert. The run log shows what broke, and any job that uploads artifacts has attached its build output and failure reports to the run.

    Close this issue once the workflow is green again — the next failure opens a fresh one.

  7. github-actions commented on Sep 21, 2026

    @github-actions
    Author

    🔴 Test Container Builds failed

    Run 35554068185
    Trigger workflow_run
    Branch main
    Commit 44c0273
    Failed at 2026-09-21 05:41 UTC

    Failed jobs

    • lecture-python.myst · quantecon-build — cancelled after 120m (timed out, or the run was superseded)
    • lecture-python.myst · quantecon — failure after 77m

    Nobody was watching this run, so this issue is the alert. The run log shows what broke, and any job that uploads artifacts has attached its build output and failure reports to the run.

    Close this issue once the workflow is green again — the next failure opens a fresh one.

  8. github-actions commented on Sep 23, 2026

    @github-actions
    Author

    🔴 Test Container Builds failed

    Run 35926916398
    Trigger workflow_run
    Branch main
    Commit b8b1b33
    Failed at 2026-09-23 22:37 UTC

    Failed jobs

    • lecture-python-intro · quantecon — failure after 23m
    • lecture-python-advanced.myst · quantecon-build — cancelled after 25m (timed out, or the run was superseded)
    • lecture-python.myst · quantecon — cancelled after 25m (timed out, or the run was superseded)
    • lecture-python-advanced.myst · quantecon — cancelled after 25m (timed out, or the run was superseded)
    • lecture-python-intro · quantecon-build — cancelled after 2m (timed out, or the run was superseded)
    • lecture-python.myst · quantecon-build — cancelled after 25m (timed out, or the run was superseded)

    Nobody was watching this run, so this issue is the alert. The run log shows what broke, and any job that uploads artifacts has attached its build output and failure reports to the run.

    Close this issue once the workflow is green again — the next failure opens a fresh one.

  9. github-actions commented on Sep 23, 2026

    @github-actions
    Author

    🔴 Test Container Builds failed

    Run 35929322686
    Trigger workflow_run
    Branch main
    Commit df44407
    Failed at 2026-09-23 22:45 UTC

    Failed jobs

    • lecture-python-intro · quantecon-build — cancelled after 7m (timed out, or the run was superseded)
    • lecture-python.myst · quantecon-build — cancelled after 7m (timed out, or the run was superseded)
    • lecture-python-advanced.myst · quantecon — cancelled after 7m (timed out, or the run was superseded)
    • lecture-python-advanced.myst · quantecon-build — cancelled after 7m (timed out, or the run was superseded)
    • lecture-python.myst · quantecon — cancelled after 7m (timed out, or the run was superseded)
    • lecture-python-intro · quantecon — cancelled after 7m (timed out, or the run was superseded)

    Nobody was watching this run, so this issue is the alert. The run log shows what broke, and any job that uploads artifacts has attached its build output and failure reports to the run.

    Close this issue once the workflow is green again — the next failure opens a fresh one.

  10. github-actions commented on Sep 23, 2026

    @github-actions
    Author

    🔴 Test Container Builds failed

    Run 35930060149
    Trigger workflow_run
    Branch main
    Commit b20c274
    Failed at 2026-09-23 23:15 UTC

    Failed jobs

    • lecture-python.myst · quantecon — cancelled after 29m (timed out, or the run was superseded)
    • lecture-python-advanced.myst · quantecon-build — cancelled after 29m (timed out, or the run was superseded)
    • lecture-python-advanced.myst · quantecon — cancelled after 28m (timed out, or the run was superseded)
    • lecture-python.myst · quantecon-build — cancelled after 28m (timed out, or the run was superseded)
    • lecture-python-intro · quantecon-build — cancelled after 14m (timed out, or the run was superseded)

    Nobody was watching this run, so this issue is the alert. The run log shows what broke, and any job that uploads artifacts has attached its build output and failure reports to the run.

    Close this issue once the workflow is green again — the next failure opens a fresh one.

  11. github-actions commented on Sep 24, 2026

    @github-actions
    Author

    🔴 Test Container Builds failed

    Run 35932591513
    Trigger workflow_run
    Branch main
    Commit 0e514bf
    Failed at 2026-09-24 02:50 UTC

    Failed jobs

    • lecture-python.myst · quantecon — failure after 118m
    • lecture-python.myst · quantecon-build — failure after 95m

    Nobody was watching this run, so this issue is the alert. The run log shows what broke, and any job that uploads artifacts has attached its build output and failure reports to the run.

    Close this issue once the workflow is green again — the next failure opens a fresh one.

  12. github-actions commented on Sep 24, 2026

    @github-actions
    Author

    🔴 Test Container Builds failed

    Run 35963034770
    Trigger workflow_run
    Branch main
    Commit 64aeb2f
    Failed at 2026-09-24 09:21 UTC

    Failed jobs

    • lecture-python.myst · quantecon — failure after 74m
    • lecture-python.myst · quantecon-build — failure after 119m

    Nobody was watching this run, so this issue is the alert. The run log shows what broke, and any job that uploads artifacts has attached its build output and failure reports to the run.

    Close this issue once the workflow is green again — the next failure opens a fresh one.

  13. quantecon-services commented on Sep 24, 2026

    @quantecon-services
    Collaborator

    Triage note, 2026-09-24. The latest run (35963034770, on 64aeb2f) is red for two separate reasons:

    1. A real lecture failure. Both lecture-python.myst legs fail "Build HTML" with a CellExecutionError in lectures/bayes_nonconj.md. That is lecture content; no issue tracks it yet, and it belongs in QuantEcon/lecture-python.myst.
    2. The timeout. The quantecon-build leg also used 116 of its 120 minutes on notebooks. That is test-containers-lectures: lecture-python.myst doubled to the 120-minute ceiling #141. imp_sample alone took 31.7 of 70 minutes on quantecon and 43.1 of 116 on quantecon-build.

    The work to make this matrix trustworthy is now tracked by the Project #188 (#102, #141, #142). This alert stays unparented and should close once a run is green.


    Generated by Claude Code

  14. quantecon-services commented on Sep 24, 2026

    @quantecon-services
    Collaborator

    Why this has been red since 2026-09-18. Two problems in lecture-python.myst cause it. Neither is in QuantEcon/actions or its images.

    1. bayes_nonconj fails since numpyro 0.22.0 (PyPI, 2026-09-18 08:51Z), which the lecture installs unpinned (!pip install numpyro jax arviz, bayes_nonconj.md:24). 0.22.0 validates distribution arguments by default. The truncated log-normal prior (truncated_lognormal(), bayes_nonconj.md:333-336) is an ExpTransform of a TruncatedNormal(high=0). Its declared support is Positive rather than (0, 1], so NUTS initialisation can propose θ > 1, and dist.Binomial now raises.
    2. The HTML build's CPU time sits at the job's 120-minute timeout-minutes, so whether a leg passes depends on the runner it lands on.

    Tested repos, each on quantecon and quantecon-build: lecture-python-intro, lecture-python.myst, lecture-python-advanced.myst. lecture-jax is commented out.

    Evidence

    • The switch from pass to fail matches the upload. In run 35316939330, bayes_nonconj passed at 07:05Z, before the upload (105510635506). It failed at 09:02Z, after it (105510635773). The lecture file has not changed since June. Every run since has failed it, on both images, 10.9–15.7 s into the notebook, and it is the only warning in each log.
    • A local run with only numpyro changed reproduces it. On 0.21.0 all cells pass. On 0.22.0, bayes_nonconj.md:338 (mcmc_ln = run_nuts(...)) fails with ValueError: BinomialProbs distribution got invalid probs parameter. The CI logs show only CellExecutionError, so the CI traceback itself is inferred. The reports artifacts that hold it expire after 7 days, and the first expires about 2026-09-25.
    • The timeout is a separate problem. 105510635506 was cancelled at exactly 120:00 even though bayes_nonconj passed in it. Two HTML builds of the same lecture sha, on full images rebuilt from the same sources, took 71.9 min (107515495294) and 116.5 min (107422724233). imp_sample (numba → JAX rewrite in caf3ddc, 2026-08-02) is 35–46% of the HTML time; phillips_drifts_volatilities and sargent_surico are next. Cancellations at 120 min have recurred on 08-03, 08-10, 08-17, 08-24, 08-31 and 09-07 (test-containers-lectures: lecture-python.myst doubled to the 120-minute ceiling #141), so the green 09-14 run was the exception.
    • The images are ruled out. They ship no jax, numpyro or arviz. The image from before 195b9f2 also passed bayes_nonconj on 09-18, before the upload, and 195b9f2 moved no Python package.
    • The lecture-python-intro lean failure on 09-18 (105510635540, observed_distributions) was a one-off. The same leg passed earlier that day and has been green since. The fetch that failed is unconfirmed; its err.log is artifact 10535844150, which expires about 2026-09-25 07:07Z.

    Fixes

    • lecture-python.myst, the real fix: give the prior its true support. Tested on numpyro 0.21.0 and 0.22.0; all cells pass on both.
      class _UnitLogNormal(dist.TransformedDistribution):
          support = dist.constraints.interval(0.0, 1.0)
      return _UnitLogNormal(base, dist.transforms.ExpTransform())
      Pinning numpyro<0.22 in that cell is a stopgap only: it downgrades the environment for every later numpyro notebook in the build. init_strategy=init_to_median alone is not enough, because the later SVI cell fails the same way.
    • lecture-python.myst: make imp_sample fast on CPU again, either with its numba estimator or smaller sample sizes without a GPU.
    • QuantEcon/actions:

    Fixing bayes_nonconj alone would still leave 2 of the 4 latest lecture-python.myst legs at the ceiling.

    Release signal (#110 decision 4): QuantEcon/actions and its images did not cause this failure. Until both lecture-python.myst fixes land, a red lecture-python.myst leg says nothing about the images. The intro and advanced legs are still a valid signal.


    Generated by Claude Code

  15. github-actions commented on Sep 25, 2026

    @github-actions
    Author

    🔴 Test Container Builds failed

    Run 36122900556
    Trigger workflow_run
    Branch main
    Commit 29e148a
    Failed at 2026-09-25 14:07 UTC

    Failed jobs

    • lecture-python.myst · quantecon-build — failure after 108m
    • lecture-python.myst · quantecon — failure after 123m

    Nobody was watching this run, so this issue is the alert. The run log shows what broke, and any job that uploads artifacts has attached its build output and failure reports to the run.

    Close this issue once the workflow is green again — the next failure opens a fresh one.

  16. github-actions commented on Sep 28, 2026

    @github-actions
    Author

    🔴 Test Container Builds failed

    Run 36370003884
    Trigger workflow_run
    Branch main
    Commit c0ac111
    Failed at 2026-09-28 06:08 UTC

    Failed jobs

    • lecture-python.myst · quantecon-build — failure after 94m
    • lecture-python.myst · quantecon — failure after 124m

    Nobody was watching this run, so this issue is the alert. The run log shows what broke, and any job that uploads artifacts has attached its build output and failure reports to the run.

    Close this issue once the workflow is green again — the next failure opens a fresh one.

  17. added a commit that references this issue on Sep 28, 2026
  18. mmcky commented on Oct 5, 2026

    @mmcky
    Contributor

    Closing: the container test matrix is green again. Run 37256340227 on 2026-10-05 (4550f69) passed all eight legs. It is the first green run since 2026-09-14.

    Both causes named in the 2026-09-24 triage are addressed:

    #141 stays open. lecture-python.myst's build time is still the underlying problem, and its legs already use about three quarters of the new ceiling. A later failure opens a new alert.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething is wrong or brokeninfrastructureSubstantial CI / build / deploy / tooling / automation work, or behaviour-preserving restructuring

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions