Skip to content

ci(workflows): make apt fetches fail fast and retry instead of hanging - #507

Merged
Milhouszhang merged 1 commit into
berabuddies:masterfrom
Milhouszhang:ci/apt-retry-timeout
Jul 9, 2026
Merged

ci(workflows): make apt fetches fail fast and retry instead of hanging#507
Milhouszhang merged 1 commit into
berabuddies:masterfrom
Milhouszhang:ci/apt-retry-timeout

Conversation

@Milhouszhang

Copy link
Copy Markdown
Collaborator

PR #506 的 CI 挂在 Desktop backend (test + clippy) 上,一行 Rust 都没编译就被杀了。

根因

Ubuntu 镜像偶发「接受连接但不再发数据」。apt 默认既没有读超时,也不重试,于是整个 apt-get install 无限期挂起,直到 job 的 timeout-minutes 把它杀掉。

run 29012965487 的实测时间线:

时刻 事件
10:51:49 Install system dependencies 开始,apt-get update 正常(10.2 MB / 2s)
11:03:27 开始拉 libwebkit2gtk-4.1-0(25.1 MB)
停滞 8.5 分钟,零输出
11:11:56 ##[error]The operation was canceled. ← job 启动后正好 20 分钟

后续 rust-toolchain / nextest / rustfmt / clippy 全部 skipped。同一 run 里 Rust (build + test) 的 apt 步骤 6 秒跑完,说明不是网络整体故障,是那个镜像在大包上抽风。

修法

调大 timeout-minutes 不是解 —— 那只会让它卡得更久。真正要做的是让停滞的连接自己失败,然后重试

  • Acquire::http::Timeout=30 —— 30 秒收不到数据就断开,把「无限挂起」变成「一次失败」
  • Acquire::Retries=3 —— 失败后自动重试(会换镜像)

两个用到 apt 的 job(rustdesktop-backend)共用一个 APT_RESILIENCE env var。不引入任何第三方 action。

备选方案(本 PR 未采用)

awalsh128/cache-apt-pkgs-action 缓存 desktop-backend 那 79 个包,提速明显,但引入第三方依赖 + 缓存一致性风险。跟「修 flake」不是一个关注点,留作后续。

验证

  • APT_RESILIENCE 在 GHA 默认的 bash -e 下正确词分割成 -o Acquire::http::Timeout=30 -o Acquire::Retries=3 四个参数(本 PR 改的两个 job 都没有 shell: 覆盖,只有 desktop/playwright job 有 defaults.run.working-directory)。
  • YAML 解析通过;本 PR 的 CI 自身即是这段改动的执行验证。

🤖 Generated with Claude Code

The Ubuntu mirror intermittently accepts a connection and then stops
sending data. apt has no read timeout by default, so it blocks forever:
run 29012965487 stalled 8.5 min on the 25 MB libwebkit2gtk-4.1-0 fetch
and the Desktop backend job was killed by its 20 min timeout-minutes
without compiling a single line of Rust.

Set a 30s read timeout so a stalled transfer aborts, and 3 retries so
apt re-fetches (picking another mirror) instead of failing the job.
Applied to both apt-using jobs via a shared env var.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Milhouszhang
Milhouszhang merged commit d601d76 into berabuddies:master Jul 9, 2026
10 of 12 checks passed
@Milhouszhang
Milhouszhang deleted the ci/apt-retry-timeout branch July 9, 2026 13:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant