Container validation: no image reaches :latest untested, and the lecture matrix is a trustworthy signal

[quantecon-services]

Goal: no container image reaches :latest without being tested, and the weekly lecture-build matrix (test-containers-lectures.yml) is red only when something is really broken.

Where we stand (verified 2026-09-28)

Next: #141. #102 is closed: #203 fixed it on 2026-09-28 with a CI-side retry, and #144 was closed unmerged. #197 already gave lecture-python.myst a 180-minute limit and records each leg's runner CPU. What #141 still needs:

• its preventive half: record every leg's duration, and warn when one grows past the previous green run's;

• a decision on the runtime itself (lecture-python.myst: CPU build time sits at the lecture-validation ceiling #196).

• The latest matrix run (36370003884) was the weekly run on 2026-09-28, on c0ac111, and the first with ci(lecture-validation): retry a failed HTML build once with --all; restore lecture-python-programming to the matrix #203's retry.

  • It failed only in the two lecture-python.myst legs, and only on lectures/bayes_nonconj.md. That is numpyro 0.22's BinomialProbs distribution got invalid probs parameter, the failure every run has hit since 2026-09-18.
  • Both legs retried the HTML stage and failed again, as a persistent failure should.
  • The fix is bayes_nonconj: declare the truncated log-normal prior's (0, 1] support (numpyro 0.22) lecture-python.myst#1065, which had not landed by that run.
  • The other six legs passed.
  • The matrix has been red since 2026-08-03 (alert: 🔴 CI failure: Test Container Builds #159), apart from one green run on 2026-09-14.

• Those two legs took 95 min (quantecon-build) and 125 min (quantecon), inside the 180-minute limit ci(lecture-validation): per-repo timeout (180 min for lecture-python.myst); log the runner CPU #197 gave lecture-python.myst. The quantecon leg's first HTML attempt alone took 120 minutes, so at the old 120-minute ceiling that leg would have been cancelled. test-containers-lectures: lecture-python.myst doubled to the 120-minute ceiling #141 stays open for the runtime itself (lecture-python.myst: CPU build time sits at the lecture-validation ceiling #196).

• Both images moved to the anaconda=2026.07 baseline the matrix repos pin (deps(containers): move both images to the Anaconda 2026.07 baseline #199, merged 2026-09-25). The build and the smoke test were green, and the matrix runs since show no new failure.

• The matrix covers four repos again. ci(lecture-validation): retry a failed HTML build once with --all; restore lecture-python-programming to the matrix #203 restored lecture-python-programming beside lecture-python-intro, lecture-python.myst and lecture-python-advanced.myst. lecture-jax is commented out.

• Nothing validates an image before :latest moves (containers: no validation runs before the tag every consumer pins moves #142). Any merge touching containers/** rebuilds and pushes :latest immediately. deps: bump the conda group across 2 directories with 1 update #162, feat(containers): Node 24 LTS from nodejs.org, out of conda #166, test(containers): exercise the real theme and FreeFont path in the smoke fixture (#108) #171 and deps(containers): move both images to the Anaconda 2026.07 baseline #199 all went out that way.

Plan

The work items and their order are the sub-issues below.

Sequencing rationale:

1. test-containers-lectures: absorb transient raw-GitHub 429s when building lecture repos #102 is done: ci(lecture-validation): retry a failed HTML build once with --all; restore lecture-python-programming to the matrix #203 closed it on 2026-09-28.
2. test-containers-lectures: lecture-python.myst doubled to the 120-minute ceiling #141 comes next, because a matrix that times out cannot tell a healthy image from a broken one.
3. containers: no validation runs before the tag every consumer pins moves #142 is the larger change. Its own body says its phase 2 (main-<sha> tag, retag :latest on green) should follow the release-gate fixture work in The release gate needs its own fixture repo — the canary cannot be trusted as one #136. The Spike: settle the two gate-mechanism assumptions before building for them #143 spike may re-scope The release gate needs its own fixture repo — the canary cannot be trusted as one #136, so what phase 2 can reuse waits on the Spike: settle the two gate-mechanism assumptions before building for them #143 decision (see Next steps after the 2026-09-28 session: release gating (#138) and loose ends #212).

Related work

• Tracking: release gating — get to a candidate that is verified before v0 moves #138, release gating for the actions' v0 tag. It has the same structural problem for a different artefact. It is kept separate because its goal is the v0 move, not images.
• Optimize preview builds: phased plan for the fastest PR previews (tracking) #92, preview-build speed. Two of its items are container changes:
  • zstd layers for quantecon-build, plus a tag input for test-container.yml #150 pushes a :zstd tag and smoke-tests it before :latest, which overlaps containers: no validation runs before the tag every consumer pins moves #142's phase 2;
  • Ship containers/quantecon-preview: slim TeX-free preview image with pinned deploy CLI #151 adds a new quantecon-preview image that this validation will need to cover.
• Build-time monitoring across lecture repos, with published benchmarks #186, build-time monitoring. test-containers-lectures: lecture-python.myst doubled to the 120-minute ceiling #141's per-leg duration summary is a small first instance of it.

Out of scope

• The bayes_nonconj.md failure itself: that fix belongs in lecture-python.myst.
• Image content changes such as the Anaconda baseline and pins. Those follow the dependency policy in docs/dev/PLAN.md.