From 6d44f42b470ca317db52055774c28c2858dd08ab Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sun, 12 Jul 2026 01:51:25 +0800 Subject: [PATCH 01/10] chore(build): make mcpp the only project build system --- .agents/skills/mcpp-style-ref/SKILL.md | 6 +- .agents/skills/xlings-build/SKILL.md | 48 +++---- .../xlings-build/references/platforms.md | 65 --------- .agents/skills/xlings-contributing/SKILL.md | 15 ++- AGENTS.md | 21 +-- src/agent/skills/contributing.cppm | 13 +- src/core/xself/init.cppm | 2 +- tests/unit/test_interface_protocol.cpp | 12 +- xmake.lua | 125 ------------------ xmake/packages/libarchive.lua | 115 ---------------- 10 files changed, 47 insertions(+), 375 deletions(-) delete mode 100644 .agents/skills/xlings-build/references/platforms.md delete mode 100644 xmake.lua delete mode 100644 xmake/packages/libarchive.lua diff --git a/.agents/skills/mcpp-style-ref/SKILL.md b/.agents/skills/mcpp-style-ref/SKILL.md index 52b9451d..fc81773a 100644 --- a/.agents/skills/mcpp-style-ref/SKILL.md +++ b/.agents/skills/mcpp-style-ref/SKILL.md @@ -145,15 +145,15 @@ xlings install gcc@15 -y 参考本仓库 `src/` 目录结构： -- `xmake.lua`：配置 `set_languages("c++23")`、`set_policy("build.c++.modules", true)` +- `mcpp.toml`：声明 C++23 模块目标、工具链和依赖 - `add_files("main.cpp")`、`add_files("**.cppm")` 添加源文件 - 可执行目标与静态库目标分离（如 `mcpp-style-ref` 主程序、`error` 静态库） 构建： ```bash -xmake build -xmake run +mcpp build +mcpp test ``` ## 适用场景 diff --git a/.agents/skills/xlings-build/SKILL.md b/.agents/skills/xlings-build/SKILL.md index 4bbde321..7c7e4a29 100644 --- a/.agents/skills/xlings-build/SKILL.md +++ b/.agents/skills/xlings-build/SKILL.md @@ -1,6 +1,6 @@ --- name: xlings-build -description: 构建 xlings 项目的三平台操作指南，覆盖 Linux、macOS、Windows 上的工具链准备、`xmake` 配置、release 脚本构建和构建后验证。Use this skill when Codex needs to build, package, or verify xlings locally or in CI, especially when aligning with `.github/workflows/*` and `tools/*_release.*`, or when setting up musl-gcc / LLVM / MSVC toolchains for this repository. +description: 构建 xlings 项目的三平台操作指南，覆盖 Linux、macOS、Windows 上的工具链准备、`mcpp` 构建、release 脚本和构建后验证。 --- # XLINGS Build @@ -10,7 +10,7 @@ description: 构建 xlings 项目的三平台操作指南，覆盖 Linux、macOS Use this skill as the build source of truth for xlings. Prefer repository CI and release scripts over ad-hoc command invention. This repo has two build products: -- C++23 core via `xmake` +- C++23 core via `mcpp` - Rust `xvm` / `xvm-shim` via `cargo` Prefer the platform release scripts for final packaging: @@ -22,9 +22,9 @@ Prefer the platform release scripts for final packaging: Follow this order unless the user explicitly wants only one phase: -1. Install `xlings` first if the workflow depends on its bundled `xmake` or toolchain packages. +1. Install `xlings` first so `.xlings.json` can provide mcpp. 2. Install the platform toolchain. -3. Configure `xmake`. +3. Confirm `mcpp.toml` and the selected target. 4. Build the repo or run the release script. 5. Verify the built binary with `-h`. 6. Run a network smoke test with `xlings install d2x -y` when the environment allows network access. @@ -64,25 +64,16 @@ sudo chown "$(id -u):$(id -g)" /home/xlings/.xlings_data/lib ln -sfn "$MUSL_SDK/x86_64-linux-musl/lib/libc.so" /home/xlings/.xlings_data/lib/ld-musl-x86_64.so.1 ``` -### 3) Configure xmake +### 3) Build with mcpp -Always configure with the musl SDK and keep `-y` so xmake can pull missing helper packages automatically. - -```bash -xmake f -c -p linux -m release \ - --sdk="$MUSL_SDK" \ - --cross=x86_64-linux-musl- \ - --cc="$CC" \ - --cxx="$CXX" \ - -y -``` +`mcpp.toml` declares the musl target and toolchain. ### 4) Build Quick local build: ```bash -xmake build xlings -y +mcpp build --target x86_64-linux-musl cargo build --manifest-path core/xvm/Cargo.toml --release ``` @@ -115,16 +106,16 @@ source ~/.bashrc ### 2) Install toolchain -CI-aligned command: +CI-aligned toolchain: ```bash -brew install xmake llvm@20 +xlings install ``` -Then configure with LLVM: +Then build with the toolchain declared by `mcpp.toml`: ```bash -xmake f -p macosx -m release --toolchain=llvm --sdk=/opt/homebrew/opt/llvm@20 -y +mcpp build ``` ### 3) Build @@ -132,7 +123,7 @@ xmake f -p macosx -m release --toolchain=llvm --sdk=/opt/homebrew/opt/llvm@20 -y Quick local build: ```bash -xmake build xlings -y +mcpp build cargo build --manifest-path core/xvm/Cargo.toml --release ``` @@ -154,26 +145,20 @@ Also keep the release-script expectation in mind: the final binary should not re ## Windows Build -Use the system MSVC environment. This repo's Windows CI does not install a compiler with xlings; it configures `xmake` for the default MSVC toolchain. +Use the LLVM toolchain declared by `mcpp.toml`, matching Windows CI. ### 1) Prepare tools -Install `xmake` and ensure Rust is available. In CI, this is handled by `xmake-io/github-action-setup-xmake@v1`. +Install xlings/mcpp and ensure Rust is available. Optional: install xlings as a runtime/package-manager smoke-test target after build. -### 2) Configure xmake - -```powershell -xmake f -p windows -m release -y -``` - ### 3) Build Quick local build: ```powershell -xmake build xlings -y +mcpp build cargo build --manifest-path core/xvm/Cargo.toml --release ``` @@ -211,11 +196,10 @@ If network access is not available, say that the smoke test was skipped rather t ## Practical Notes - On Linux, `musl-gcc@15.1.0` is required for `import std` in the C++23 build. -- The root `xmake.lua` only handles the C++ target. `xim` tasks live under a separate xmake project. +- `mcpp.toml` is the only C++ project definition. - Prefer release scripts for packaging because they also assemble `xim`, `xvm`, config files, bundled `xmake`, and post-build verification. - When the user asks for exact CI parity, read the workflow file first and mirror it exactly rather than relying on memory. ## References -- Read [references/platforms.md](references/platforms.md) for platform-by-platform commands and file outputs. - Read [references/links.md](references/links.md) for workflow, release script, and project links. diff --git a/.agents/skills/xlings-build/references/platforms.md b/.agents/skills/xlings-build/references/platforms.md deleted file mode 100644 index 4e094e22..00000000 --- a/.agents/skills/xlings-build/references/platforms.md +++ /dev/null @@ -1,65 +0,0 @@ -# XLINGS Build Platforms - -## Linux - -Install and configure: - -```bash -curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/refs/heads/main/tools/other/quick_install.sh | bash -source ~/.bashrc -xlings install musl-gcc@15.1.0 -y -xlings info musl-gcc - -MUSL_SDK="${XLINGS_HOME:-$HOME/.xlings}/data/xpkgs/musl-gcc/15.1.0" -export CC=x86_64-linux-musl-gcc -export CXX=x86_64-linux-musl-g++ -export PATH="$MUSL_SDK/bin:$PATH" - -xmake f -c -p linux -m release \ - --sdk="$MUSL_SDK" \ - --cross=x86_64-linux-musl- \ - --cc="$CC" \ - --cxx="$CXX" \ - -y -``` - -Build outputs: - -- `build/linux/x86_64/release/xlings` -- `core/xvm/target/release/xvm` for plain cargo build -- `build/xlings-*-linux-x86_64.tar.gz` for release script build - -## macOS - -CI-aligned toolchain setup: - -```bash -brew install xmake llvm@20 -xmake f -p macosx -m release --toolchain=llvm --sdk=/opt/homebrew/opt/llvm@20 -y -``` - -Build outputs: - -- `build/macosx/arm64/release/xlings` -- `core/xvm/target/release/xvm` -- `build/xlings-*-macosx-arm64.tar.gz` - -## Windows - -CI-aligned setup: - -```powershell -xmake f -p windows -m release -y -``` - -Build outputs: - -- `build\windows\x64\release\xlings.exe` -- `core\xvm\target\release\xvm.exe` -- `build\xlings-*-windows-x86_64.zip` - -## Release Scripts - -- Linux: `tools/linux_release.sh` -- macOS: `tools/macos_release.sh` -- Windows: `tools/windows_release.ps1` diff --git a/.agents/skills/xlings-contributing/SKILL.md b/.agents/skills/xlings-contributing/SKILL.md index 81c69a84..e01a1793 100644 --- a/.agents/skills/xlings-contributing/SKILL.md +++ b/.agents/skills/xlings-contributing/SKILL.md @@ -18,7 +18,7 @@ This skill defines the standard contribution flow for AI agents working on the x curl -fsSL https://raw.githubusercontent.com/openxlings/xlings/main/tools/other/quick_install.sh | bash # From repo root — install build dependencies -xlings install # reads .xlings.json → xmake, cmake, ninja, toolchain +xlings install # reads .xlings.json → installs mcpp # Switch to the correct dev toolchain xlings use gcc@16.1.0 # Linux dev build (avoids musl/glibc link conflicts) @@ -27,8 +27,8 @@ xlings use gcc@16.1.0 # Linux dev build (avoids musl/glibc link conflicts) ### 2. Verify build works ```bash -xmake f --local_libxpkg=/path/to/libxpkg -y # only if modifying upstream libxpkg -xmake build xlings # ~15s on warm cache +mcpp build +mcpp test ``` ### 3. Repository structure awareness @@ -54,7 +54,7 @@ tests/ ├── e2e/ # End-to-end shell tests │ ├── project_test_lib.sh # Shared test helpers │ └── fixtures/ # Test fixture packages -└── (unit tests via xmake build xlings_tests) +└── (unit tests via `mcpp test`) ``` ## Standard Contribution Flow @@ -130,11 +130,12 @@ Key helpers from `project_test_lib.sh`: ### Step 5: Build + test locally ```bash -# Build -xmake build xlings +# Build and unit tests +mcpp build +mcpp test # Run your test -XLINGS_BIN=$(find build -path '*/release/xlings' -type f | head -1) \ +XLINGS_BIN=$(find target -path '*/bin/xlings' -type f | head -1) \ bash tests/e2e/_test.sh # Run existing tests to check for regressions diff --git a/AGENTS.md b/AGENTS.md index 11e01d88..4612a836 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -48,7 +48,7 @@ tests/ │ ├── project_test_lib.sh # Shared helpers (find_xlings_bin, run_xlings) │ ├── fixtures/ # Test fixture packages │ └── subos_xpkg_*.sh # SubOS-as-xpkg e2e tests -└── (unit tests via xmake build xlings_tests) +└── (unit tests via `mcpp test`) .agents/ ├── docs/ # Agent working docs (see .agents/docs/README.md) @@ -59,19 +59,19 @@ tests/ ## Build System -Single build tool: **xmake** (C++23 modules). +Single build tool: **mcpp** (C++23 modules). ```bash # Setup (from repo root): -xlings install # installs xmake, cmake, ninja, toolchain from .xlings.json +xlings install # installs mcpp from .xlings.json xlings use gcc@16.1.0 # switch to glibc-linked dev toolchain # Build: -xmake build xlings # dev binary → build/linux/x86_64/release/xlings +mcpp build # dev binary → target///bin/xlings # Test: -xmake build xlings_tests && xmake run xlings_tests # unit tests -XLINGS_BIN=$(find build -path '*/release/xlings' -type f | head -1) \ +mcpp test # unit tests +XLINGS_BIN=$(find target -path '*/bin/xlings' -type f | head -1) \ bash tests/e2e/.sh # e2e tests ``` @@ -110,11 +110,12 @@ Use `project_test_lib.sh` helpers. Key functions: ### Upstream dependency -`mcpplibs/libxpkg` provides the xpkg loader/executor. Referenced via xmake dep: -```lua -add_requires("mcpplibs-xpkg 0.0.41") +`mcpplibs/libxpkg` provides the xpkg loader/executor. Referenced via `mcpp.toml`: +```toml +[dependencies.mcpplibs] +xpkg = "0.0.42" ``` -For joint development: `xmake f --local_libxpkg=/path/to/libxpkg` +For joint development, use mcpp's local dependency override/workspace mechanism. ## Agent Skills diff --git a/src/agent/skills/contributing.cppm b/src/agent/skills/contributing.cppm index 9c2f5b43..abec1442 100644 --- a/src/agent/skills/contributing.cppm +++ b/src/agent/skills/contributing.cppm @@ -31,7 +31,7 @@ ABSOLUTE RULES — NEVER violate these 2. NEVER trigger a release without CI passing on the PR first. 3. NEVER amend or force-push commits that others may have pulled. 4. ALWAYS create a feature branch from the latest main. -5. ALWAYS run `xmake build xlings` and verify before pushing. +5. ALWAYS run `mcpp build` and `mcpp test` before pushing. 6. ALWAYS use `xlings install` for tools, never apt/brew/curl. ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ @@ -54,8 +54,9 @@ STEP 2: Implement the change - Do NOT add comments/docstrings/refactors beyond what's asked STEP 3: Build and verify locally - xmake build xlings - xmake run xlings # manual smoke test + mcpp build + mcpp test + # manual smoke test STEP 4: Commit git add @@ -121,14 +122,14 @@ BUILD ENVIRONMENT ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Setup (from repo root): - xlings install # installs xmake, cmake, ninja, toolchain + xlings install # installs mcpp xlings use gcc@16.1.0 # dev build toolchain (Linux) Build: - xmake build xlings # dev binary + mcpp build # dev binary Test: - xmake build xlings_tests && xmake run xlings_tests # unit tests + mcpp test # unit tests Release scripts (CI uses these): Linux: tools/linux_release.sh (musl-gcc static binary) diff --git a/src/core/xself/init.cppm b/src/core/xself/init.cppm index 26a2f887..bf8be9cf 100644 --- a/src/core/xself/init.cppm +++ b/src/core/xself/init.cppm @@ -9,7 +9,7 @@ import xlings.platform; // Cross-version compat (currently: legacy alias cleanup, profile upgrade). // See compact/xself.cppm — each compat lives in its own version sub-namespace. import xlings.core.xself.compat; -// Generated at build time from config/shell/*.{sh,fish,ps1}; see xmake.lua. +// Generated at build time from config/shell/*.{sh,fish,ps1}; see mcpp.toml. import xlings.core.xself.profile_resources; namespace xlings::xself { diff --git a/tests/unit/test_interface_protocol.cpp b/tests/unit/test_interface_protocol.cpp index e6b1a63b..2cdd6735 100644 --- a/tests/unit/test_interface_protocol.cpp +++ b/tests/unit/test_interface_protocol.cpp @@ -20,8 +20,7 @@ import xlings.libs.json; namespace { // Resolve once on first call and cache the absolute path. mcpp places binaries -// under target///bin; keep the legacy xmake build layout as -// a fallback while the repository still supports older local workflows. +// under target///bin. std::string find_xlings_binary_() { namespace fs = std::filesystem; static const std::string cached = []() -> std::string { @@ -58,15 +57,6 @@ std::string find_xlings_binary_() { return absolute_or_original(newest_mcpp_candidate); } - static const char* platforms[] = {"linux", "macosx", "windows"}; - static const char* archs[] = {"x86_64", "arm64", "x64"}; - static const char* modes[] = {"release", "debug"}; - for (auto* p : platforms) for (auto* a : archs) for (auto* m : modes) { - fs::path candidate = fs::path("build") / p / a / m / exe; - if (fs::is_regular_file(candidate, ec)) { - return absolute_or_original(candidate); - } - } return std::string{}; }(); return cached; diff --git a/xmake.lua b/xmake.lua deleted file mode 100644 index 34c48d87..00000000 --- a/xmake.lua +++ /dev/null @@ -1,125 +0,0 @@ -add_rules("mode.debug", "mode.release") - -set_languages("c++23") - --- GCC 15 ICE workaround: -O1/-O2 crash in tree-ssa-dce with C++23 modules -if is_plat("linux") then - add_cxxflags("-Og", {force = true}) -end - -add_repositories("mcpplibs-index https://github.com/mcpplibs/mcpplibs-index.git") - --- Project-private package overrides. See xmake/packages/libarchive.lua --- for why we ship our own libarchive definition (TL;DR: xmake-repo's --- libarchive uses 7-Zip LZMA SDK as a dep, libarchive's cmake actually --- needs xz-utils' liblzma → .tar.xz extraction silently degrades to a --- fork-exec of the system `xz` binary, which doesn't exist on --- musl-static minimal containers or Windows). -includes("xmake/packages/libarchive.lua") - --- Local-libxpkg override for cross-repo joint debugging. --- Usage: --- xmake f --local_libxpkg=/path/to/mcpplibs/libxpkg ... --- When set, we treat the local checkout as the source of mcpplibs-xpkg --- instead of fetching from the released xrepo package. Useful while --- iterating on libxpkg + xlings together (e.g., schema split for --- build/runtime deps). Empty string (default) → use released package. -option("local_libxpkg") - set_default("") - set_showmenu(true) - set_description("Path to local mcpplibs/libxpkg checkout for dev builds") -option_end() - -add_requires("cmdline 0.0.2") -add_requires("ftxui 6.1.9") -add_requires("mcpplibs-capi-lua") --- mcpplibs-xpkg: prefer a local source checkout for joint debugging --- when --local_libxpkg=/path/to/libxpkg is set. We pull the upstream --- xmake.lua targets in via `includes()` so the consumer's build sees --- our edits without going through xrepo's package cache (which keys --- by released-version hash and ignores in-flight source changes). --- Otherwise fall back to the released package from xrepo. -if has_config("local_libxpkg") and get_config("local_libxpkg") ~= "" then - includes(path.join(get_config("local_libxpkg"), "xmake.lua")) -else - add_requires("mcpplibs-xpkg 0.0.41") -end -add_requires("gtest 1.15.2") -add_requires("mcpplibs-tinyhttps 0.2.0") --- libarchive's compression backends. Force `system = false` so xmake --- builds them from source under our musl-cross toolchain instead of --- picking up the host's glibc-built /usr/lib copies, which can't be --- linked into a musl-static binary. Required for the linux release --- build; harmless on macOS/Windows. --- --- Note `xz` (xz-utils / liblzma) replaces the upstream --- `lzma` (7-Zip LZMA SDK). See xmake/packages/libarchive.lua header --- comment for why; the override of libarchive itself wires `xz` in as --- a dep so libarchive's cmake actually finds liblzma. -add_requires("zlib", { system = false }) -add_requires("lz4", { system = false }) -add_requires("bzip2", { system = false }) -add_requires("zstd", { system = false }) -add_requires("xz", { system = false }) -add_requires("libarchive-xlings 3.8.7") - --- C++23 main binary -target("xlings") - set_kind("binary") - add_files("src/main.cpp") - add_files("src/**.cppm") - add_includedirs("src/libs/json") - add_packages("cmdline", "ftxui", "mcpplibs-capi-lua") - if has_config("local_libxpkg") and get_config("local_libxpkg") ~= "" then - add_deps("mcpplibs-xpkg", "mcpplibs-xpkg-loader", - "mcpplibs-xpkg-index", "mcpplibs-xpkg-lua-stdlib", - "mcpplibs-xpkg-executor") - else - add_packages("mcpplibs-xpkg") - end - add_packages("mcpplibs-tinyhttps", "libarchive-xlings") - set_policy("build.c++.modules", true) - - if is_plat("macosx") then - set_toolchains("llvm") - elseif is_plat("linux") then - add_ldflags("-static", {force = true}) - elseif is_plat("windows") then - -- libarchive's XAR format parser uses xmllite (CreateXmlReader); - -- xmake-repo's libarchive package_def only adds advapi32. Add - -- xmllite here so the link step finds CreateXmlReader. - add_syslinks("xmllite") - end - --- Unit tests -target("xlings_tests") - set_kind("binary") - set_default(false) - set_rundir("$(projectdir)") - -- InterfaceProtocol tests spawn the real xlings binary via popen, - -- so make sure it's available when `xmake build xlings_tests` runs. - add_deps("xlings") - add_files("tests/unit/*.cpp") - add_files("src/**.cppm") - add_includedirs("src/libs/json") - add_packages("cmdline", "ftxui", "mcpplibs-capi-lua", "gtest") - if has_config("local_libxpkg") and get_config("local_libxpkg") ~= "" then - add_deps("mcpplibs-xpkg", "mcpplibs-xpkg-loader", - "mcpplibs-xpkg-index", "mcpplibs-xpkg-lua-stdlib", - "mcpplibs-xpkg-executor") - else - add_packages("mcpplibs-xpkg") - end - add_packages("mcpplibs-tinyhttps", "libarchive-xlings") - set_policy("build.c++.modules", true) - - if is_plat("macosx") then - set_toolchains("llvm") - elseif is_plat("linux") then - add_ldflags("-static", {force = true}) - elseif is_plat("windows") then - -- libarchive's XAR format parser uses xmllite (CreateXmlReader); - -- xmake-repo's libarchive package_def only adds advapi32. Add - -- xmllite here so the link step finds CreateXmlReader. - add_syslinks("xmllite") - end diff --git a/xmake/packages/libarchive.lua b/xmake/packages/libarchive.lua deleted file mode 100644 index 677aa00b..00000000 --- a/xmake/packages/libarchive.lua +++ /dev/null @@ -1,115 +0,0 @@ --- libarchive package override (project-private). --- --- Why this file exists --- -------------------- --- xmake-repo's `libarchive` package definition declares its compression --- backend deps as: --- --- add_deps("zlib", "bzip2", "lz4", "zstd", "lzma") --- --- Four of those names match the cmake `find_package` names that --- libarchive's CMakeLists.txt probes during build (`ZLIB`, `BZip2`, --- `lz4`, `zstd`) — so the libarchive build wires them in correctly. --- --- The fifth, `lzma`, does NOT. xmake-repo's `lzma` package is the --- 7-Zip LZMA SDK (header `LzmaLib.h`, function `LzmaCompress`). What --- libarchive's cmake actually probes is `find_package(LibLZMA)` — --- xz-utils' liblzma (header `lzma.h`, function `lzma_code`). Different --- API surface entirely. xrepo's package for that is `xz`. --- --- Net effect of the upstream `add_deps("lzma")` line: cmake's --- LibLZMA detection fails, libarchive falls back to its --- `archive_read_support_filter_xz_external` path that fork-exec's the --- system `xz` / `lzma` binary at runtime. Symptoms: --- * Linux musl-static deployments that lack a host `xz` binary → --- .tar.xz / .tar.lzma extraction fails at runtime. --- * Windows builds → no system `xz.exe` → same failure. --- * Even on hosts where it works, every .tar.xz extract is a --- fork-exec round-trip (slow, temp-file IO). --- --- Concrete xim-pkgindex packages affected today: node@* (linux uses --- node-vXXX-linux-x64.tar.xz) and llvm (macos arm64 uses .tar.xz). --- --- Override strategy --- ----------------- --- Re-declare libarchive as `libarchive-xlings`, set_base("libarchive") --- to inherit URL list / version table / etc., then: --- * swap `lzma` → `xz` in the deps so the cmake find_package call --- resolves to xrepo-built liblzma instead of falling back; --- * pass explicit `-DENABLE_LZMA=ON` (and the other backends) so --- cmake's auto-detection cannot silently disable any of them on a --- stripped builder image. --- --- Including this file from the project's root xmake.lua is enough; --- xmake's `package()` block registers a private package alongside any --- xmake-repo packages and `add_requires("libarchive-xlings ...")` --- picks it up. --- --- When this can be removed --- ------------------------ --- After xmake-repo merges a fix that changes `add_deps("lzma")` to --- `add_deps("xz")` in their libarchive package def. Track at: --- https://github.com/xmake-io/xmake-repo/tree/dev/packages/l/libarchive - -package("libarchive-xlings") - - set_base("libarchive") - - -- Pin the same versions xmake-repo ships, so set_base inherits the - -- URL templates and we just need version → sha256 mappings. - add_versions("3.8.7", "4b787cca6697a95c7725e45293c973c208cbdc71ae2279f30ef09f52472b9166") - add_versions("3.8.6", "213269b05aac957c98f6e944774bb438d0bd168a2ec60b9e4f8d92035925821c") - - -- Replace the upstream deps list. `xz` is xz-utils / liblzma - -- (provides lzma.h), the one libarchive's CMakeLists actually - -- probes via find_package(LibLZMA). The other four match what - -- libarchive expects already. - add_deps("cmake") - add_deps("zlib", "bzip2", "lz4", "zstd", "xz") - - if is_plat("windows") then - add_syslinks("advapi32") - end - - on_install("windows", "linux", "macosx", function (package) - local configs = { - -- Match xmake-repo's defaults... - "-DENABLE_TEST=OFF", - "-DENABLE_CAT=OFF", - "-DENABLE_TAR=OFF", - "-DENABLE_CPIO=OFF", - "-DENABLE_OPENSSL=OFF", - "-DENABLE_PCREPOSIX=OFF", - "-DENABLE_LibGCC=OFF", - "-DENABLE_CNG=OFF", - "-DENABLE_ICONV=OFF", - "-DENABLE_ACL=OFF", - "-DENABLE_EXPAT=OFF", - "-DENABLE_LIBXML2=OFF", - "-DENABLE_LIBB2=OFF", - -- ...and force-enable every compression backend we ship a - -- dep for. cmake's auto-detection silently disables a - -- backend if the find_package probe fails on the build - -- host (e.g. minimal CI image with no system xz-dev), - -- which is precisely what produced the original bug. Be - -- explicit so the build either uses the xrepo-built - -- backend or fails loudly. - "-DENABLE_ZLIB=ON", - "-DENABLE_BZip2=ON", - "-DENABLE_LZ4=ON", - "-DENABLE_ZSTD=ON", - "-DENABLE_LZMA=ON", - } - table.insert(configs, "-DCMAKE_BUILD_TYPE=" .. (package:debug() and "Debug" or "Release")) - table.insert(configs, "-DBUILD_SHARED_LIBS=" .. (package:config("shared") and "ON" or "OFF")) - if not package:config("shared") then - package:add("defines", "LIBARCHIVE_STATIC") - end - import("package.tools.cmake").install(package, configs) - end) - - on_test(function (package) - assert(package:has_cfuncs("archive_version_number", {includes = "archive.h"})) - end) - -package_end() From cbbdf0f236c0c94f729542cb2a24244f21e647d0 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sun, 12 Jul 2026 01:58:59 +0800 Subject: [PATCH 02/10] fix(xim): make download cache commits transactional --- ...sue-356-partial-download-cache-fix-plan.md | 387 ++++++++++++++++++ src/core/xim/downloader.cppm | 369 ++++++++++++++--- src/core/xim/installer.cppm | 4 + src/core/xim/libxpkg/types/type.cppm | 1 + src/platform.cppm | 1 + src/platform/unix.cppm | 63 +++ src/platform/windows.cppm | 60 +++ tests/unit/test_main.cpp | 265 +++++++++++- tests/unit/test_mirror.cpp | 26 ++ 9 files changed, 1126 insertions(+), 50 deletions(-) create mode 100644 .agents/docs/2026-07-11-issue-356-partial-download-cache-fix-plan.md diff --git a/.agents/docs/2026-07-11-issue-356-partial-download-cache-fix-plan.md b/.agents/docs/2026-07-11-issue-356-partial-download-cache-fix-plan.md new file mode 100644 index 00000000..a2f2482b --- /dev/null +++ b/.agents/docs/2026-07-11-issue-356-partial-download-cache-fix-plan.md @@ -0,0 +1,387 @@ +# Issue #356：无 SHA256 下载残片进入缓存的根因与修复方案 + +> 日期：2026-07-11 +> 状态：实施中（动态更新） +> 关联 Issue：[openxlings/xlings#356](https://github.com/openxlings/xlings/issues/356) +> 分析基线：`openxlings/xlings@37fedb7`、`openxlings/xim-pkgindex@b894897` + +## 1. 结论 + +Issue #356 成立。最佳修复不是单独增加一个大小检查，也不是只给 `mcpp` 补 SHA256，而是以下三层组合： + +1. **立即止血：给官方 `XLINGS_RES` 资源补 SHA256。** 先迁移 `mcpp` 的 0.0.81、`latest` 当前指向版本与其他活跃版本，再逐步回填可验证版本；索引形态必须先通过旧客户端兼容矩阵，不能未经验证直接发布 `res=true` 条目。 +2. **根因修复：下载到唯一临时文件，验收成功后原子提交。** 网络传输、进程中断、候选镜像校验失败期间都不能写最终缓存路径。 +3. **缓存策略修复：删除 HEAD 失败时“任意非空文件即命中”的规则。** 无 SHA256 的离线复用只接受由新下载事务写出的完整性 sidecar，且记录大小必须与当前文件一致；旧缓存没有完整提交证据时不能命中。 + +同时应在 `mcpplibs/tinyhttps` 修正 chunked 响应提前 EOF 被误判为正常结束的问题，并将响应的实际字节数、期望字节数和最终来源 URL 返回给 xlings。这样 Content-Length 校验属于传输层，缓存提交属于 xlings，制品哈希属于索引/发布链，责任边界清晰。 + +不建议把“libarchive 能打开并读到第一个 header”作为主要验收条件。截断通常发生在归档后半段，首 header 可正常读取；完整遍历则会让大型工具链在真正解压前额外做一次完整解压读取，成本过高。 + +## 2. 事实与影响范围 + +### 2.1 Issue 场景的可验证事实 + +- 在 `xim-pkgindex@b894897` 快照中，`mcpp@0.0.81` 使用裸 `"XLINGS_RES"`，解析出的 `DownloadTask::sha256` 为空。下列数量同样是该提交的审计快照，不代表实施时仓库 HEAD。 +- 0.0.81 的上游 release 已提供每个平台的 `.sha256` 文件，`tools/mirror_res.sh` 也已镜像这些 sidecar，校验数据并不缺失，只是没有进入索引条目。 +- Linux x86_64 制品的权威大小为 `12,628,937` 字节，SHA256 为 `47c41529a00930ad701a76bb53e0847220c0764eb1f8e6cf6d515c45fea8cfcc`。从 `xlings-res/mcpp` 下载的镜像制品与上游 sidecar 一致。 +- Issue 中“正常包约 30 MB+”是估计值，不准确；114 KB 文件仍显著小于权威大小，不影响缺陷成立。 +- 当前 `xim-pkgindex` 有 26 个包文件包含裸 `XLINGS_RES`，共 336 个裸条目；其中 `mcpp` 占 208 个，`xlings` 占 60 个。当前没有包使用已经被 xlings 支持的 `res=true + sha256-by-arch` 形态。 + +### 2.2 当前代码路径 + +```mermaid +flowchart LR + A[索引裸 XLINGS_RES] --> B[DownloadTask.sha256 为空] + B --> C[tinyhttps 直接写最终 destFile] + C -->|进程被杀/异常退出| D[最终路径残留半文件] + D --> E[下次安装执行 HEAD] + E -->|HEAD 失败| F[只检查 localSize > 0] + F --> G[误报缓存命中] + G --> H[libarchive 解压] + H --> I[Truncated tar archive] +``` + +关键位置： + +- `src/core/xim/installer.cppm:1201-1225`：裸 `XLINGS_RES` 生成 URL，但没有生成校验值。 +- `src/core/xim/downloader.cppm:362-409`：无 SHA256 时执行 HEAD 缓存判断；HEAD 失败后任意非空文件都会成功返回。 +- `src/core/xim/downloader.cppm:439-478`：`tinyhttps` 直接以最终 `destFile` 为下载目标。 +- `src/core/xim/downloader.cppm:481-500`：1 KiB 下限只能拒绝极小错误页，无法识别 114 KB 或更大的截断文件。 +- `src/core/xim/downloader.cppm:502-523`：只有声明 SHA256 时才做强校验；无 SHA256 sidecar 不记录“完整提交”或实际文件大小。 +- `tests/unit/test_main.cpp:832-870`：现有测试只验证 sidecar 的读写，没有验证 HEAD-fail 缓存准入策略。 + +### 2.3 Issue 根因描述需要补充的两点 + +第一，当前依赖的 `mcpplibs-tinyhttps` 0.2.0 在响应声明 `Content-Length > 0` 时，已经使用 `read_exact` 读取全部字节；提前断线会返回 `Read error`，xlings 外层随后删除目标文件。因此“普通断线一定留下并接受半文件”并不完整。稳定入口是： + +- 进程在外层清理前被强制终止； +- 机器掉电或进程崩溃； +- 旧版本/其他路径已经留下最终路径半文件； +- chunked 响应在终止 chunk 到来前断开。 + +第二，`mcpplibs-tinyhttps` 0.2.0 到 0.2.3 的 chunked 读取逻辑相同：空的 chunk-size 行会被 `parse_hex` 当作 0，随后按合法终止 chunk 返回成功。这是独立的传输完整性缺陷，必须在上游修复；单靠 xlings 的 HEAD 检查无法可靠补偿。 + +## 3. 方案比较 + +| 方案 | 优点 | 缺点 | 结论 | +|---|---|---|---| +| A. 只给 mcpp/XLINGS_RES 补 SHA256 | 最快；现有 hash cache path 可直接识别并删除 114 KB 文件；抵御镜像内容漂移 | 无法保护第三方无 SHA256 包；下载中断仍直接污染最终路径；发布机器人仍可能生成新裸条目 | 作为 P0 止血，不是完整修复 | +| B. 只增加 Content-Length/归档检查 | 能覆盖部分无哈希下载；改动较局部 | Content-Length 路径底层已经严格读取；chunked/connection-close 没有可靠长度；首 header 不能发现后半段截断；完整归档扫描代价高 | 不推荐作为主方案 | +| C. SHA256 迁移 + 事务下载 + 严格缓存准入 | 同时阻止新残片、修复旧缓存误命中、保留可证明的离线复用；适用于官方和第三方包 | 涉及 tinyhttps、xlings、xim-pkgindex/发布链三个仓库，需要分阶段落地 | **推荐** | + +## 4. 推荐设计 + +### 4.1 传输层：明确“完整响应”的判定 + +在 `mcpplibs/tinyhttps` 中： + +1. `Content-Length` 响应继续要求 `bytesWritten == contentLength`，并把这两个值返回给调用方。 +2. chunked 响应只有读到语法合法的 `0\r\n` 终止 chunk 和尾部结束行才算成功；空 size line、非法十六进制、缺少 chunk 尾部 CRLF、提前 EOF 都返回错误。 +3. connection-close 响应在 HTTP 语义上没有可比较的期望长度，只能报告 `expectedBytes = null`，不能伪造完整性保证。 +4. `DownloadToFileResult`/xlings 包装结果至少携带： + - `bytesWritten` + - `expectedBytes`（可选） + - `finalUrl` 或被接受的候选 URL + - 明确的传输错误类型 + +这层只判断 HTTP 消息是否完整，不判断制品是否可信。 + +### 4.2 xlings 下载层：临时文件验收后原子提交 + +`download_one` 不再把 `destFile` 直接交给 tinyhttps： + +1. 在缓存判断前取得 per-destination 跨进程锁，并持有到提交/失败清理完成，避免两个 xlings 进程同时判定 miss、删除或替换同一目标。锁实现放入 platform 层，Linux/macOS 使用 advisory lock，Windows 使用独占文件句柄。 +2. 为每次下载生成同目录、唯一的 `stagingFile`，例如 `.part.`。同目录保证最终 `rename` 不跨文件系统。 +3. tinyhttps 的所有候选 URL 都写 `stagingFile`。失败、取消或校验拒绝时只删除临时文件，不触碰已有的已提交缓存。 +4. `onVerify` 对 `stagingFile` 计算 SHA256，不能继续捕获最终 `destFile`。 +5. 候选传输成功后依次执行： + - 若有期望长度，检查实际长度一致； + - 若有 SHA256，检查 hash； + - 若无 SHA256 且是归档，仅保留现有的极小错误页检查作为快速诊断，不把它当完整性证明。 +6. 所有验收通过后，在锁内执行可恢复提交：已有无效目标先改名为同目录 backup，`stagingFile` 再 rename 为 `destFile`；第二步失败则恢复 backup，成功后删除 backup。该流程在两次 rename 之间最终路径短暂不存在，因此不能称为全过程原子替换。 +7. 锁取得后先执行恢复状态机：只有 backup 时恢复为 live；live 与 backup 同时存在时保留 live 并删除已确认过期的 backup；live、backup、staging 的其他组合 fail closed 并记录诊断。若把机器掉电纳入保证，还需分别定义 POSIX `fsync`/目录同步和 Windows `FlushFileBuffers` 的持久化顺序。 +8. 提交最终文件后，原子写入 sidecar。若在“文件已提交、sidecar 未写”之间崩溃，结果只是下次保守重下，不会误接收半文件。 +9. 只有持有目标锁时才能清理该目标的过期 `.part.*`/backup；不要删除其他进程可能仍在写的临时文件，也不要把临时文件当缓存候选。 + +临时文件名必须支持并发 xlings 进程，不能使用固定 `.part`。唯一 ID 可由进程 ID、线程 ID、单调计数器和随机数组合；不需要暴露为公共接口。锁等待必须响应取消并有超时，超时错误应指出被占用的缓存目标。 + +### 4.3 sidecar v2：记录“已完整提交”而不假装是密码学证明 + +现有 sidecar 只有 `Last-Modified` 和 `ETag`。升级为带版本的记录： + +```text +format: 2 +complete: true +size: 12628937 +source-url: +cache-identity: +last-modified: +etag: +``` + +含义： + +- `complete=true` 只表示该文件经过新事务流程成功提交，不等价于可信 hash。 +- `size` 是提交时实际大小，用于发现后续截断、覆盖或磁盘损坏。 +- `source-url` 记录真正成功的候选。当前代码下载可能从 fallback 成功，却只 HEAD 主 URL；新设计应优先探测实际来源。 +- `cache-identity` 必须绑定包、解析后的版本、平台、架构以及规范化主 URL 或官方资源键。HEAD 失败离线复用时必须匹配，防止不同任务因文件名和大小相同而误用彼此缓存。 +- `source-url`、`size`、ETag、Last-Modified 应直接来自成功 GET 的结果，避免提交后再 HEAD 带来的额外失败和 TOCTOU。 +- sidecar 自身用临时文件 + rename 写入，避免半行记录。 +- 老 sidecar 不含 `format/complete/size`，按 legacy 处理，不能获得 HEAD-fail 离线准入资格。 + +### 4.4 缓存准入矩阵 + +| 条件 | 处理 | +|---|---| +| 声明 SHA256，缓存 hash 匹配 | 命中 | +| 声明 SHA256，缓存 hash 不匹配 | 删除文件与 sidecar，重新下载 | +| 无 SHA256，HEAD 成功且远端长度与本地一致，ETag/Last-Modified 与 sidecar 一致 | 命中 | +| 无 SHA256，HEAD 成功且无远端新鲜度字段，但长度一致、sidecar v2 完整且记录大小一致 | 可命中，记录 weak-cache 日志 | +| 无 SHA256，HEAD 失败，sidecar v2 完整且记录大小与本地一致 | 允许离线命中，并明确警告“非密码学验证” | +| 无 SHA256，HEAD 失败，只有 legacy sidecar 或任意非空文件 | **不得命中**；尝试 GET，GET 也失败则明确报错 | +| 存在 `.part.*`，最终文件不存在 | 清理或忽略 `.part.*`，重新下载 | + +该矩阵保留了 0.4.15 引入的 airline-friendly 目标，但把“存在文件”升级为“存在新事务完整提交证据”。 + +### 4.5 解压失败后的自愈 + +事务下载不能证明无 SHA256、无长度的 connection-close 内容在业务上完整。对归档增加一层反应式自愈： + +- `extract_archive` 将错误至少区分为输入归档错误和本地写入错误。 +- `open`、`next_header`、`read_data_block` 的格式/截断错误视为输入归档错误；删除对应缓存和 sidecar，使下一次安装必然重新下载。 +- `write_header`、`write_data_block`、磁盘空间、权限错误属于本地环境错误，不删除可能完全有效的大型缓存。 +- 本轮不做“下载后完整预扫描归档”，避免 LLVM/GCC 等大包在解压前额外读取和解压一次。 + +### 4.6 官方资源层：让 SHA256 成为默认契约 + +当前工作树的自定义解析路径支持以下索引形态，但旧客户端兼容性尚未证明： + +```lua +["0.0.81"] = { + res = true, + sha256 = { + x86_64 = "", + aarch64 = "", + }, +}, +``` + +因此 mcpp 止血不需要等待新的 xpkg schema，但发布到全局索引前必须选择经旧客户端验证的表达。优先测试以下双兼容候选： + +~~~lua +["0.0.81"] = { + url = "XLINGS_RES", + sha256 = { + x86_64 = "", + aarch64 = "", + }, +}, +~~~ + +新客户端读取架构 hash 后继续把 `url` sentinel 展开为官方资源 URL；旧客户端仍有机会按原 `XLINGS_RES` 路径安装。是否成立必须用真实旧版本验证，不能仅靠静态推断。迁移建议： + +1. 先给 `mcpp` 的 0.0.81、`latest` 解析后的目标版本及其他活跃镜像版本补 hash；`latest` 本身继续作为 `ref`，验收其最终 `DownloadTask::sha256` 非空。 +2. `version-check.py` 的 `res_versioned` apply 路径不再写裸 sentinel。它应读取发布产物的 sidecar/manifest，生成带校验的 res 条目；缺任一受支持平台/架构的 hash 时 fail closed，不开不完整 PR。 +3. `mirror_res.sh` 在上传后不仅检查 HTTP 200，还要下载或流式计算每个镜像制品的 SHA256，与上游值比较。 +4. xlings 自身 release 当前没有 `.sha256` 资产。发布链应为每个归档生成 sidecar，或统一发布机器可读 manifest，再允许机器人生成带 hash 的 res 条目。 +5. 审计其余 24 个含裸 `XLINGS_RES` 的配方，按“能取得权威 hash → 迁移；不能取得 → 保留兼容但走严格事务缓存”的顺序处理。不要让全量历史回填阻塞 mcpp 止血。 + +长期建议统一发布一个 manifest，字段至少为 `{version, platform, arch, filename, size, sha256}`。sidecar 可继续供人和简单脚本使用，索引机器人消费 manifest，避免在脚本中硬编码平台文件名。 + +## 5. 实施顺序 + +### P0：当天止血（xim-pkgindex / 发布脚本） + +- 迁移 `mcpp@0.0.81` 和当前 `latest` 的三平台条目，使用上游已发布的 `.sha256`。 +- 修改 mcpp 的 index bump 流程，使新版本不再生成裸 `XLINGS_RES`。 +- 验证 GLOBAL 与 CN 镜像的 hash 都等于索引值。 + +这一阶段上线后，现有 114 KB 缓存会在 SHA cache path 被识别为 mismatch、删除并重新下载。 + +### P1：通用根因修复（mcpplibs/tinyhttps + xlings） + +- tinyhttps 修正 chunked EOF 判定并扩充结果元数据。 +- xlings 升级 tinyhttps 依赖。 +- `download_one` 改为 staging → verify → atomic rename。 +- sidecar 升级为 v2，删除无条件非空缓存 fallback。 + +### P2：自愈与生态收口 + +- 结构化区分归档输入错误与本地写入错误，输入错误时驱逐缓存。 +- 为 xlings/其他官方 res 发布生成 checksum manifest。 +- 分批迁移其余官方 `XLINGS_RES` 条目，并在索引 CI 中禁止新增无 SHA256 的官方二进制归档。 + +## 6. 测试计划 + +### 6.1 mcpplibs/tinyhttps + +- `Content-Length=N`，服务端只发送 ` X2 -> X3 -> X4 ---------+--> xlings release -> index 更新 -> 生态 E2E +T1 -> tinyhttps release ------+ +L1 -> libxpkg release -> L2 --+ +I1 ---------------------------+ +I2 依赖 L1/L2 的兼容矩阵通过，但不阻塞 I1 的 mcpp 止血 +~~~ + +- `X1` 与 `T1`、`L1`、`I1` 可并行；同一仓库内部保持顺序，避免把根因修复、schema 和发布脚本揉成一个不可审查 PR。 +- `X1`～`X4` 可以合并为一个 xlings PR，但提交按工作包分层，测试必须随对应实现提交。 +- `L2` 必须等 `L1` 有可引用版本或 commit；不能在 xlings 中保留第二套“临时兼容解析器”。 +- `I1` 只迁移有权威 hash 的活跃资源；`I2` 才做全量审计，不允许为了追求迁移率伪造或从不可信镜像推导 hash。 + +### 9.2 X1：缓存决策与 sidecar v2 + +**修改范围**：`src/core/xim/downloader.cppm`、`tests/unit/test_main.cpp`。 + +1. 先把缓存准入抽成不访问网络的决策函数，输入包含：是否声明 SHA256、文件存在/大小、HEAD 结果、远端大小/新鲜度字段、sidecar 版本/complete/size；输出为 `Hit`、`Redownload` 或 `OfflineUnverifiedHit`。 +2. 先增加失败测试：HEAD 失败 + 114 KiB legacy 文件必须 `Redownload`；HEAD 失败 + v2 complete 且大小一致必须 `OfflineUnverifiedHit`；v2 大小不一致必须 `Redownload`。 +3. sidecar reader 同时读取 legacy 与 v2；writer 只生成 v2，并以同目录临时文件 + rename 提交。 +4. 删除 `HEAD failed && localSize > 0` 的旧准入分支；离线复用只接受 v2 完整提交证据。 + +**2026-07-12 实施记录**： + +- 已为 `DownloadTask` 增加稳定 `cacheIdentity`，由解析后的包名、版本、平台、架构和资源键组成。 +- sidecar reader 兼容 legacy，严格校验 v2 的 `format/complete/size/cache-identity`；writer 只写 v2。 +- 已删除 HEAD 失败时任意非空缓存命中，只有 v2 的大小和 identity 同时匹配才返回 `OfflineUnverifiedHit`。 +- RED 证据：`mcpp test` 曾因缺少 `CacheAdmissionInput_` / `decide_cache_admission_` 编译失败。 +- GREEN 证据：`XimDownloaderTest` 7/7 通过；完整 `mcpp test` 10 个测试二进制全部通过。 +- 测试门禁同时修复 `test_mirror` 对真实用户 `XLINGS_HOME` 的污染：测试进程现在使用独立临时 home，不再读取用户的 `github-mirrors.json`。 + +### 9.3 X2：文件下载事务 + +**修改范围**：`src/core/xim/downloader.cppm`、下载器单元/集成测试。 + +1. 每次 GET 使用 `.part..`；`tinyhttps::DownloadOptions::destFile` 和 `onVerify` 都指向 staging。 +2. 下载、候选校验、大小检查或取消失败时删除本次 staging，不删除已有已提交目标。 +3. 验收后执行 `dest -> backup`、`staging -> dest`、删除 backup；第二步失败必须恢复旧目标。 +4. 最终文件提交后再原子写 sidecar v2。文件已提交但 sidecar 未提交时，下次按保守 miss 处理。 +5. 测试必须覆盖：候选一 hash 失败、候选二成功；取消；提交失败回滚；遗留 `.part.*` 不被当缓存。 + +**2026-07-12 实施记录**： + +- tinyhttps 的目标和逐候选 `onVerify` 已统一改为唯一 sibling staging 文件，最终路径不再承接网络写入。 +- SHA mismatch、传输失败、取消和极小错误页只删除本次 staging；旧目标保留到新内容验收成功。 +- 提交使用 `live -> backup -> staging -> live`，第二次 rename 失败会恢复旧目标；该语义明确称为“可恢复提交”，不宣称两次 rename 之间最终路径始终存在。 +- 已加入传输和 HEAD probe 测试 seam；失败保留旧目标、预取消零传输、候选 hash fallback、backup 后故障注入恢复测试均通过。 +- 全套 `mcpp test`：10 passed，0 failed。 + +### 9.4 X3：跨进程并发 + +**修改范围**：`src/platform.cppm`、对应三平台实现、`src/core/xim/downloader.cppm`、测试。 + +1. 以最终目标路径派生锁文件；缓存判定、下载、提交和 sidecar 写入均在锁内。 +2. Unix 使用 advisory file lock，Windows 使用不共享写入的文件句柄；等待检查取消并有明确超时错误。 +3. 清理只处理锁持有者能够证明为过期的 staging/backup，不按通配符删除其他进程文件。 +4. 并发测试断言只发生一次有效提交，等待者随后命中已提交缓存。 + +**2026-07-12 实施记录**： + +- `platform::FileLock` 已实现：Linux/macOS 使用 `flock`，Windows 使用独占 `CreateFileW`；等待支持取消和 10 分钟超时。 +- 锁已覆盖文件缓存判定、传输、提交和 sidecar 写入。 +- 持锁后的恢复状态机已覆盖“仅 backup 时恢复 live”“live + backup 时保留 live”“清理同目标遗留 staging”。 +- 同进程双句柄竞争、取消等待、延迟释放、两种崩溃恢复测试通过；完整 `mcpp test` 10/10 通过。 +- 尚缺真实双进程竞争测试以及 macOS/Windows CI 证据，因此本项仍为“进行中”。 + +### 9.5 X4：解压失败自愈 + +**修改范围**：`src/core/xim/extract.cppm`、`src/core/xim/installer.cppm`、相关测试。 + +1. 解压结果区分 `InvalidInputArchive` 与 `LocalWriteFailure`。 +2. 前者删除下载缓存及 sidecar，后者保留缓存并返回环境错误。 +3. 不增加下载后的完整归档预扫描；测试直接构造截断归档和不可写目标。 + +### 9.6 T1：tinyhttps 协议完整性 + +在独立仓库和独立 PR 中补协议级本地 HTTP server 测试，再修 chunk parser。成功结果公开 `bytesWritten`、可选 `expectedBytes`、`finalUrl`；xlings 在升级依赖后使用这些字段写 sidecar和日志。connection-close 响应的 `expectedBytes` 必须为空，不能被描述为长度已验证。 + +### 9.7 L1/L2：唯一 xpkg 入口 + +`libxpkg` 接受并归一化原有版本资源项以及 `xpm.source = "xlings-res" | `，输出平台、架构、最终 URL、SHA256 和 fallback 的统一资源对象。兼容逻辑只位于 libxpkg 的 compat 模块。xlings 升级依赖并删除 `load_platform_entries_()`；删除前必须用同一组 fixture 对旧字符串 URL、`XLINGS_RES`、mirror table、`ref`、单 hash 和多架构 hash 做前后结果对比。 + +### 9.8 I1/I2：索引迁移与老客户端保护 + +新增 `xpm.source` 是可选字段，不会改变旧的 `xpm[platform][version]` 条目；因此迁移采用“双轨兼容”，而不是立即重写所有历史条目： + +- 旧 `"XLINGS_RES"` 条目继续保留并可被旧客户端读取。 +- 新版本推荐使用 `xpm.source = "xlings-res"`，版本项保留原平台/版本位置并补 SHA256。 +- 在确认目标老版本客户端会把 `source` 当保留元数据而不是版本键之前，不对官方索引批量加入该字段；兼容测试必须实际运行至少当前稳定版和修复版解析同一 fixture。 +- 如果旧客户端会误把 `source` 当平台或版本，官方索引只先落地旧语法的 `res = true + sha256-by-arch` 或显式资源表，待兼容版本覆盖后再迁移 `source`。 +- 索引指针/制品发布机制可继续让旧用户固定读取旧 artifact，但这只能作为发布回滚手段，不能替代解析兼容测试。 + +### 9.9 发布门禁与最终验收 + +1. 每个仓库从干净的 `main` 建独立分支/worktree，不覆盖当前工作区已有改动。 +2. PR 描述包含根因、行为变化、测试命令和跨仓依赖；所有 Linux、macOS、Windows 必需检查通过。 +3. 先发布 tinyhttps/libxpkg，再升级 xlings 依赖与版本；xlings release 必须生成各平台资产和 SHA256/manifest。 +4. release 自动或人工生成索引 PR；核对 GLOBAL、CN 资源 hash 与 manifest 一致后合并。 +5. 用隔离 `XLINGS_HOME` 验证：旧稳定客户端仍能安装未迁移条目；新客户端能安装新旧条目；预置 114 KiB 错误缓存后安装 mcpp 会驱逐、重下并成功解压；离线 v2 完整缓存可复用。 +6. 只有 Issue #356 关闭、所有跨仓 PR/release/index 证据链接回填本节后，`R1` 才能标记完成。 + +## 10. 最终建议 + +按 **P0 索引止血 → P1 事务缓存根治 → P2 发布生态收口** 的顺序实施。若只能先做一个改动，应先给 mcpp 补 SHA256，因为它能立即修复 Issue 中的用户；但关闭 Issue #356 应以 P1 完成并覆盖“HEAD 失败 + legacy 半文件”和“chunked 提前 EOF”回归测试为准，不能只以 mcpp 当前版本恢复安装为准。 diff --git a/src/core/xim/downloader.cppm b/src/core/xim/downloader.cppm index df627556..cea57493 100644 --- a/src/core/xim/downloader.cppm +++ b/src/core/xim/downloader.cppm @@ -72,10 +72,136 @@ std::string lower_hex_(std::string_view s) { // Format: one "key: value" per line, only `last-modified` and `etag` // recognized. Anything else is ignored. Missing sidecar = no metadata. struct MetaSidecar_ { + int format { 1 }; + bool complete { false }; + std::int64_t size { -1 }; + std::string sourceUrl; + std::string cacheIdentity; std::string lastModified; std::string etag; }; +std::filesystem::path unique_sibling_path_( + const std::filesystem::path& base, + std::string_view marker) { + static std::atomic_uint64_t sequence { 0 }; + auto path = base; + path += std::format( + ".{}.{}.{}", marker, + std::chrono::steady_clock::now().time_since_epoch().count(), + sequence.fetch_add(1)); + return path; +} + +bool commit_staging_file_(const std::filesystem::path& staging, + const std::filesystem::path& destination, + std::string& error, + bool failAfterBackupForTest = false) { + namespace fs = std::filesystem; + std::error_code ec; + auto backup = unique_sibling_path_(destination, "old"); + const bool hadDestination = fs::exists(destination, ec); + if (ec) { + error = std::format("failed to inspect {}: {}", + destination.string(), ec.message()); + return false; + } + if (hadDestination) { + fs::rename(destination, backup, ec); + if (ec) { + error = std::format("failed to preserve {}: {}", + destination.string(), ec.message()); + return false; + } + } + if (failAfterBackupForTest) { + if (hadDestination) { + fs::rename(backup, destination, ec); + } + error = "injected commit failure after backup"; + return false; + } + fs::rename(staging, destination, ec); + if (ec) { + auto commitError = ec.message(); + if (hadDestination) { + std::error_code restoreEc; + fs::rename(backup, destination, restoreEc); + if (restoreEc) { + error = std::format( + "failed to commit {} ({}) and restore previous file ({})", + destination.string(), commitError, restoreEc.message()); + return false; + } + } + error = std::format("failed to commit {}: {}", + destination.string(), commitError); + return false; + } + if (hadDestination) { + fs::remove(backup, ec); + } + return true; +} + +bool recover_download_transaction_( + const std::filesystem::path& destination, + std::string& error) { + namespace fs = std::filesystem; + std::error_code ec; + std::vector backups; + std::vector stagingFiles; + const auto backupPrefix = destination.filename().string() + ".old."; + const auto stagingPrefix = destination.filename().string() + ".part."; + auto parent = destination.parent_path(); + for (auto it = fs::directory_iterator(parent, ec); + !ec && it != fs::directory_iterator{}; it.increment(ec)) { + auto name = it->path().filename().string(); + if (name.starts_with(backupPrefix)) backups.push_back(it->path()); + else if (name.starts_with(stagingPrefix)) stagingFiles.push_back(it->path()); + } + if (ec) { + error = std::format("failed to inspect download transaction for {}: {}", + destination.string(), ec.message()); + return false; + } + + std::ranges::sort(backups); + const bool liveExists = fs::exists(destination, ec); + if (ec) { + error = std::format("failed to inspect {}: {}", + destination.string(), ec.message()); + return false; + } + if (!liveExists && !backups.empty()) { + auto restore = backups.back(); + backups.pop_back(); + fs::rename(restore, destination, ec); + if (ec) { + error = std::format("failed to restore interrupted download {}: {}", + destination.string(), ec.message()); + return false; + } + } + for (const auto& path : backups) { + fs::remove(path, ec); + if (ec) { + error = std::format("failed to remove stale backup {}: {}", + path.string(), ec.message()); + return false; + } + } + for (const auto& path : stagingFiles) { + fs::remove(path, ec); + if (ec) { + error = std::format("failed to remove stale staging file {}: {}", + path.string(), ec.message()); + return false; + } + } + return true; +} + std::optional read_meta_sidecar_(const std::filesystem::path& p) { std::ifstream in(p); if (!in) return std::nullopt; @@ -93,18 +219,133 @@ std::optional read_meta_sidecar_(const std::filesystem::path& p) { }; trim(key); trim(val); for (auto& c : key) c = static_cast(std::tolower(static_cast(c))); - if (key == "last-modified") m.lastModified = std::move(val); - else if (key == "etag") m.etag = std::move(val); + if (key == "format") { + int parsed {}; + auto [ptr, err] = std::from_chars(val.data(), val.data() + val.size(), parsed); + if (err != std::errc{} || ptr != val.data() + val.size()) return std::nullopt; + m.format = parsed; + } else if (key == "complete") { + if (val == "true") m.complete = true; + else if (val == "false") m.complete = false; + else return std::nullopt; + } else if (key == "size") { + std::int64_t parsed {}; + auto [ptr, err] = std::from_chars(val.data(), val.data() + val.size(), parsed); + if (err != std::errc{} || ptr != val.data() + val.size() || parsed < 0) { + return std::nullopt; + } + m.size = parsed; + } else if (key == "source-url") { + m.sourceUrl = std::move(val); + } else if (key == "cache-identity") { + m.cacheIdentity = std::move(val); + } else if (key == "last-modified") { + m.lastModified = std::move(val); + } else if (key == "etag") { + m.etag = std::move(val); + } + } + if (m.format != 1 && m.format != 2) return std::nullopt; + if (m.format == 2 + && (!m.complete || m.size < 0 || m.cacheIdentity.empty())) { + return std::nullopt; } return m; } -void write_meta_sidecar_(const std::filesystem::path& p, - const tinyhttps::RemoteFileMeta& meta) { - std::ofstream out(p, std::ios::trunc); - if (!out) return; - if (!meta.lastModified.empty()) out << "last-modified: " << meta.lastModified << "\n"; - if (!meta.etag.empty()) out << "etag: " << meta.etag << "\n"; +bool write_meta_sidecar_(const std::filesystem::path& p, + const tinyhttps::RemoteFileMeta& meta, + std::int64_t size, + std::string_view cacheIdentity, + std::string_view sourceUrl) { + auto staging = unique_sibling_path_(p, "tmp"); + { + std::ofstream out(staging, std::ios::trunc); + if (!out) return false; + out << "format: 2\n"; + out << "complete: true\n"; + out << "size: " << size << "\n"; + out << "source-url: " << sourceUrl << "\n"; + out << "cache-identity: " << cacheIdentity << "\n"; + if (!meta.lastModified.empty()) out << "last-modified: " << meta.lastModified << "\n"; + if (!meta.etag.empty()) out << "etag: " << meta.etag << "\n"; + out.flush(); + if (!out) { + std::error_code ec; + std::filesystem::remove(staging, ec); + return false; + } + } + std::string error; + if (!commit_staging_file_(staging, p, error)) { + std::error_code ec; + std::filesystem::remove(staging, ec); + return false; + } + return true; +} + +enum class CacheAdmission_ { + Hit, + OfflineUnverifiedHit, + Redownload, +}; + +struct CacheAdmissionInput_ { + std::int64_t localSize { -1 }; + bool headSucceeded { false }; + std::int64_t remoteSize { -1 }; + std::string remoteLastModified; + std::string remoteEtag; + std::optional sidecar; + std::string expectedCacheIdentity; +}; + +CacheAdmission_ decide_cache_admission_(const CacheAdmissionInput_& input) { + if (input.localSize <= 0) return CacheAdmission_::Redownload; + + if (!input.headSucceeded) { + if (!input.sidecar) return CacheAdmission_::Redownload; + const auto& stored = *input.sidecar; + const bool committedV2 = + stored.format == 2 + && stored.complete + && stored.size == input.localSize + && !input.expectedCacheIdentity.empty() + && stored.cacheIdentity == input.expectedCacheIdentity; + return committedV2 + ? CacheAdmission_::OfflineUnverifiedHit + : CacheAdmission_::Redownload; + } + + const bool sizeMatch = + input.remoteSize > 0 && input.remoteSize == input.localSize; + if (!sizeMatch) return CacheAdmission_::Redownload; + + if (!input.sidecar) return CacheAdmission_::Hit; + const auto& stored = *input.sidecar; + const bool freshMatch = + (!input.remoteLastModified.empty() + && input.remoteLastModified == stored.lastModified) + || (!input.remoteEtag.empty() && input.remoteEtag == stored.etag); + const bool noFreshnessEvidence = + stored.lastModified.empty() && stored.etag.empty(); + return (freshMatch || noFreshnessEvidence) + ? CacheAdmission_::Hit + : CacheAdmission_::Redownload; +} + +struct DownloadTestHooks_ { + std::function transferOverride; + std::function queryRemoteMeta; +}; + +tinyhttps::RemoteFileMeta query_remote_meta_( + const std::string& url, + const DownloadTestHooks_* hooks) { + if (hooks && hooks->queryRemoteMeta) return hooks->queryRemoteMeta(url); + return tinyhttps::query_remote_meta(url); } // Derive the destination directory name from a git URL, e.g. @@ -240,7 +481,8 @@ DownloadResult git_clone_one(const DownloadTask& task) { // Download a single file using libcurl with real-time progress callback. DownloadResult download_one(const DownloadTask& task, std::function onProgress = nullptr, - CancellationToken* cancel = nullptr) { + CancellationToken* cancel = nullptr, + const DownloadTestHooks_* testHooks = nullptr) { namespace fs = std::filesystem; DownloadResult result; @@ -334,6 +576,23 @@ DownloadResult download_one(const DownloadTask& task, sidecarPath += ".meta"; result.localFile = destFile; + auto lockPath = destFile; + lockPath += ".lock"; + platform::FileLock cacheLock; + std::string lockError; + auto cancelled = [cancel] { + return cancel && (cancel->is_paused() || cancel->is_cancelled()); + }; + if (!cacheLock.acquire( + lockPath, std::chrono::minutes{10}, cancelled, lockError)) { + result.error = std::move(lockError); + return result; + } + if (!recover_download_transaction_(destFile, lockError)) { + result.error = std::move(lockError); + return result; + } + // ── Cache hit path 1: sha256 verified (cheapest, most reliable) ── // If the recipe declares a sha256 and the on-disk file matches, we're // byte-identical to upstream — skip download outright. @@ -350,7 +609,7 @@ DownloadResult download_one(const DownloadTask& task, // sha mismatch: stale/corrupt cache. Remove before falling through // so the next download_to_file starts from a clean slate (defends // against a future tinyhttps that might enable Range/resume). - fs::remove(destFile, ec); + // Keep the previous file until a verified replacement is ready. } // ── Cache hit path 2: HEAD-based freshness (when sha256 is unset) ── @@ -360,46 +619,41 @@ DownloadResult download_one(const DownloadTask& task, tinyhttps::RemoteFileMeta probedMeta; bool probedMetaValid = false; if (fs::exists(destFile) && task.sha256.empty()) { - probedMeta = tinyhttps::query_remote_meta(task.url); + probedMeta = query_remote_meta_(task.url, testHooks); probedMetaValid = true; if (probedMeta.ok) { std::error_code sec; auto localSize = static_cast(fs::file_size(destFile, sec)); - std::string storedLM, storedETag; - if (auto stored = read_meta_sidecar_(sidecarPath)) { - storedLM = std::move(stored->lastModified); - storedETag = std::move(stored->etag); - } - bool sizeMatch = (probedMeta.contentLength > 0) - && (localSize == probedMeta.contentLength); - // Strong freshness signal: server's Last-Modified or ETag - // matches what we recorded the last time we downloaded. - bool freshMatch = - (!probedMeta.lastModified.empty() && probedMeta.lastModified == storedLM) - || (!probedMeta.etag.empty() && probedMeta.etag == storedETag); - // Weak signal: no sidecar (legacy file from before this code), - // but the size matches what the server reports right now. - bool weakMatch = sizeMatch && storedLM.empty() && storedETag.empty(); - - if (sizeMatch && (freshMatch || weakMatch)) { + auto admission = decide_cache_admission_({ + .localSize = sec ? -1 : localSize, + .headSucceeded = true, + .remoteSize = probedMeta.contentLength, + .remoteLastModified = probedMeta.lastModified, + .remoteEtag = probedMeta.etag, + .sidecar = read_meta_sidecar_(sidecarPath), + .expectedCacheIdentity = task.cacheIdentity, + }); + if (admission == CacheAdmission_::Hit) { log::debug("already downloaded (HEAD cache hit, size={}): {}", localSize, destFile.string()); result.success = true; return result; } - // Stale: drop both file and sidecar before re-downloading. - fs::remove(destFile, ec); - fs::remove(sidecarPath, ec); + // Stale: retain the previous committed pair until the replacement + // has transferred and passed all acceptance checks. } else { - // HEAD failed (offline, server blocks HEAD, 4xx, etc.). If we - // already have a non-empty cached file, prefer it over failing - // — being airline-friendly is worth the small risk of serving - // a stale payload when sha256 is unset. std::error_code sec; - auto localSize = fs::file_size(destFile, sec); - if (!sec && localSize > 0) { - log::warn("HEAD probe failed for {} ({}); using cached file: {}", + auto localSize = static_cast(fs::file_size(destFile, sec)); + auto admission = decide_cache_admission_({ + .localSize = sec ? -1 : localSize, + .headSucceeded = false, + .sidecar = read_meta_sidecar_(sidecarPath), + .expectedCacheIdentity = task.cacheIdentity, + }); + if (admission == CacheAdmission_::OfflineUnverifiedHit) { + log::warn("HEAD probe failed for {} ({}); using transaction-committed " + "cache without cryptographic verification: {}", task.url, probedMeta.error.empty() ? "unknown" : probedMeta.error, destFile.string()); @@ -433,16 +687,19 @@ DownloadResult download_one(const DownloadTask& task, // .agents/docs/2026-06-04-github-asset-adaptive-mirror.md. urls = mirror::adaptive::reorder(std::move(urls), !task.sha256.empty()); + auto stagingFile = unique_sibling_path_(destFile, "part"); + // Use in-process tinyhttps for all downloads (streaming progress). // When a CancellationToken is available, wire isCancelled so ESC aborts. { tinyhttps::DownloadOptions opts; - opts.destFile = destFile; + opts.destFile = stagingFile; opts.urls = std::move(urls); opts.retryCount = 3; opts.connectTimeoutSec = 30; opts.maxTimeSec = 600; opts.onProgress = onProgress; + if (testHooks) opts.transferOverride = testHooks->transferOverride; if (cancel) { opts.isCancelled = [cancel] { return cancel->is_paused() || cancel->is_cancelled(); }; } @@ -461,9 +718,9 @@ DownloadResult download_one(const DownloadTask& task, // whole download with the remaining candidates untried. if (!task.sha256.empty()) { auto want = lower_hex_(task.sha256); - opts.onVerify = [destFile, want, &task](const std::string& u) + opts.onVerify = [stagingFile, want, &task](const std::string& u) -> std::string { - auto digest = sha256::hex_file(destFile); + auto digest = sha256::hex_file(stagingFile); if (digest && *digest == want) return {}; return std::format( "sha256 mismatch for {} (source {}): got {}, want {}", @@ -473,6 +730,7 @@ DownloadResult download_one(const DownloadTask& task, auto dlResult = tinyhttps::download_file(opts); if (!dlResult.success) { + fs::remove(stagingFile, ec); result.error = dlResult.error; return result; } @@ -487,14 +745,14 @@ DownloadResult download_one(const DownloadTask& task, // misleading libarchive "unrecognized format". Skip when sha256 // is declared — that's a stronger check than size alone. if (task.sha256.empty() && looks_like_archive_filename_(destFile)) { - auto sz = fs::file_size(destFile, ec); + auto sz = fs::file_size(stagingFile, ec); if (!ec && sz < kMinPlausibleArchiveBytes_) { result.error = std::format( "{}: downloaded payload is only {} bytes — likely an " "error stub returned as 200 OK (no sha256 declared to " "cross-check). URL: {}", task.name, sz, task.url); - fs::remove(destFile, ec); + fs::remove(stagingFile, ec); return result; } } @@ -503,22 +761,35 @@ DownloadResult download_one(const DownloadTask& task, // onVerify above already gated acceptance; in-process hash, no // dependency on a host `sha256sum` binary, which stock macOS lacks). if (!task.sha256.empty()) { - auto digest = sha256::hex_file(destFile); + auto digest = sha256::hex_file(stagingFile); if (!digest || *digest != lower_hex_(task.sha256)) { result.error = std::format("SHA256 mismatch for {}", task.name); - fs::remove(destFile, ec); + fs::remove(stagingFile, ec); return result; } - } else { + } + + std::string commitError; + if (!commit_staging_file_(stagingFile, destFile, commitError)) { + fs::remove(stagingFile, ec); + result.error = std::move(commitError); + return result; + } + + if (task.sha256.empty()) { // No sha256 declared: persist server-reported Last-Modified / ETag // alongside the payload so the next install can use a HEAD probe // to decide cache freshness instead of re-downloading. if (!probedMetaValid) { - probedMeta = tinyhttps::query_remote_meta(task.url); + probedMeta = query_remote_meta_(task.url, testHooks); probedMetaValid = true; } - if (probedMeta.ok && (!probedMeta.lastModified.empty() || !probedMeta.etag.empty())) { - write_meta_sidecar_(sidecarPath, probedMeta); + std::error_code sizeEc; + auto committedSize = static_cast(fs::file_size(destFile, sizeEc)); + if (!sizeEc && !task.cacheIdentity.empty()) { + write_meta_sidecar_( + sidecarPath, probedMeta, committedSize, + task.cacheIdentity, task.url); } } diff --git a/src/core/xim/installer.cppm b/src/core/xim/installer.cppm index 0c0a3b60..dc9c919f 100644 --- a/src/core/xim/installer.cppm +++ b/src/core/xim/installer.cppm @@ -1218,6 +1218,10 @@ public: task.name = detail_::plan_key_(node); task.url = res.url; task.sha256 = res.sha256; + task.cacheIdentity = std::format( + "{}/{}/{}/{}/{}", + node.name, version, platform, hostArch, + useResFallbacks ? "xlings-res" : res.url); task.destDir = detail_::runtime_dir_(node, dataDir); if (useResFallbacks) { diff --git a/src/core/xim/libxpkg/types/type.cppm b/src/core/xim/libxpkg/types/type.cppm index 4d46f5e1..5befe3fd 100644 --- a/src/core/xim/libxpkg/types/type.cppm +++ b/src/core/xim/libxpkg/types/type.cppm @@ -146,6 +146,7 @@ struct DownloadTask { std::string name; std::string url; std::string sha256; + std::string cacheIdentity; std::filesystem::path destDir; std::vector fallbackUrls; // tried in order when url fails }; diff --git a/src/platform.cppm b/src/platform.cppm index 8677d96c..ca55f0a3 100644 --- a/src/platform.cppm +++ b/src/platform.cppm @@ -41,6 +41,7 @@ namespace platform { export using platform_impl::Icon; export using platform_impl::atomic_replace_executable; export using platform_impl::atomic_swap_paths; + export using platform_impl::FileLock; // ── Execution identity (root / sudo awareness) ────────────────── // Single source of truth for "who am I / who should own the files I diff --git a/src/platform/unix.cppm b/src/platform/unix.cppm index f59d2baa..c3e23467 100644 --- a/src/platform/unix.cppm +++ b/src/platform/unix.cppm @@ -2,12 +2,14 @@ module; #include #include +#include #if defined(__linux__) || defined(__APPLE__) #include #include #include #include #include +#include #include #endif #if defined(__linux__) @@ -26,6 +28,67 @@ import std; namespace xlings { namespace platform_impl { + export class FileLock { + public: + FileLock() = default; + FileLock(const FileLock&) = delete; + FileLock& operator=(const FileLock&) = delete; + FileLock(FileLock&& other) noexcept : fd_(std::exchange(other.fd_, -1)) {} + FileLock& operator=(FileLock&& other) noexcept { + if (this != &other) { + release(); + fd_ = std::exchange(other.fd_, -1); + } + return *this; + } + ~FileLock() { release(); } + + bool acquire(const std::filesystem::path& path, + std::chrono::milliseconds timeout, + const std::function& cancelled, + std::string& error) { + release(); + fd_ = ::open(path.c_str(), O_CREAT | O_RDWR, 0600); + if (fd_ < 0) { + error = std::format("failed to open lock {}: {}", + path.string(), std::strerror(errno)); + return false; + } + auto deadline = std::chrono::steady_clock::now() + timeout; + while (::flock(fd_, LOCK_EX | LOCK_NB) != 0) { + if (errno != EWOULDBLOCK && errno != EAGAIN) { + error = std::format("failed to lock {}: {}", + path.string(), std::strerror(errno)); + release(); + return false; + } + if (cancelled && cancelled()) { + error = "cancelled while waiting for cache lock"; + release(); + return false; + } + if (std::chrono::steady_clock::now() >= deadline) { + error = std::format("timed out waiting for cache lock {}", + path.string()); + release(); + return false; + } + std::this_thread::sleep_for(std::chrono::milliseconds{50}); + } + return true; + } + + void release() { + if (fd_ < 0) return; + ::flock(fd_, LOCK_UN); + ::close(fd_); + fd_ = -1; + } + + private: + int fd_ { -1 }; + }; + // Query the controlling terminal for its background color via the // OSC-11 sequence (xterm spec, supported by xterm / iTerm2 / Alacritty // / Kitty / WezTerm / modern Windows Terminal). Returns std::nullopt diff --git a/src/platform/windows.cppm b/src/platform/windows.cppm index 58cbac96..78970207 100644 --- a/src/platform/windows.cppm +++ b/src/platform/windows.cppm @@ -17,6 +17,66 @@ import xlings.runtime.cancellation; namespace xlings { namespace platform_impl { + export class FileLock { + public: + FileLock() = default; + FileLock(const FileLock&) = delete; + FileLock& operator=(const FileLock&) = delete; + FileLock(FileLock&& other) noexcept + : handle_(std::exchange(other.handle_, INVALID_HANDLE_VALUE)) {} + FileLock& operator=(FileLock&& other) noexcept { + if (this != &other) { + release(); + handle_ = std::exchange(other.handle_, INVALID_HANDLE_VALUE); + } + return *this; + } + ~FileLock() { release(); } + + bool acquire(const std::filesystem::path& path, + std::chrono::milliseconds timeout, + const std::function& cancelled, + std::string& error) { + release(); + auto deadline = std::chrono::steady_clock::now() + timeout; + while (true) { + handle_ = ::CreateFileW( + path.wstring().c_str(), + GENERIC_READ | GENERIC_WRITE, + 0, nullptr, OPEN_ALWAYS, + FILE_ATTRIBUTE_NORMAL, nullptr); + if (handle_ != INVALID_HANDLE_VALUE) return true; + auto code = ::GetLastError(); + if (code != ERROR_SHARING_VIOLATION + && code != ERROR_LOCK_VIOLATION) { + error = std::format( + "failed to lock {}: Windows error {}", + path.string(), code); + return false; + } + if (cancelled && cancelled()) { + error = "cancelled while waiting for cache lock"; + return false; + } + if (std::chrono::steady_clock::now() >= deadline) { + error = std::format("timed out waiting for cache lock {}", + path.string()); + return false; + } + std::this_thread::sleep_for(std::chrono::milliseconds{50}); + } + } + + void release() { + if (handle_ == INVALID_HANDLE_VALUE) return; + ::CloseHandle(handle_); + handle_ = INVALID_HANDLE_VALUE; + } + + private: + HANDLE handle_ { INVALID_HANDLE_VALUE }; + }; + export constexpr char PATH_SEPARATOR = ';'; export constexpr std::string_view OS_NAME = "windows"; diff --git a/tests/unit/test_main.cpp b/tests/unit/test_main.cpp index 9d48dc85..cd2be060 100644 --- a/tests/unit/test_main.cpp +++ b/tests/unit/test_main.cpp @@ -33,6 +33,7 @@ import xlings.core.xim.downloader; import xlings.runtime; import xlings.capabilities; import xlings.libs.tinyhttps; +import xlings.libs.sha256; import mcpplibs.xpkg; import mcpplibs.cmdline; @@ -844,9 +845,16 @@ TEST(XimDownloaderTest, MetaSidecarRoundTrip) { meta.lastModified = "Wed, 21 Oct 2015 07:28:00 GMT"; meta.etag = "\"abc123\""; - xlings::xim::write_meta_sidecar_(path, meta); + ASSERT_TRUE(xlings::xim::write_meta_sidecar_( + path, meta, 1234, + "test/1.0.0/linux/x86_64/url", + "https://example.test/payload.tar.gz")); auto roundtrip = xlings::xim::read_meta_sidecar_(path); ASSERT_TRUE(roundtrip.has_value()); + EXPECT_EQ(roundtrip->format, 2); + EXPECT_TRUE(roundtrip->complete); + EXPECT_EQ(roundtrip->size, 1234); + EXPECT_EQ(roundtrip->cacheIdentity, "test/1.0.0/linux/x86_64/url"); EXPECT_EQ(roundtrip->lastModified, meta.lastModified); EXPECT_EQ(roundtrip->etag, meta.etag); @@ -869,6 +877,261 @@ TEST(XimDownloaderTest, MetaSidecarRoundTrip) { fs::remove_all(tmp); } +TEST(XimDownloaderTest, HeadFailureRejectsLegacyNonEmptyCache) { + xlings::xim::CacheAdmissionInput_ input { + .localSize = 114 * 1024, + .headSucceeded = false, + .sidecar = xlings::xim::MetaSidecar_ { + .lastModified = "Wed, 21 Oct 2015 07:28:00 GMT", + }, + .expectedCacheIdentity = "mcpp/0.0.81/linux/x86_64/xlings-res", + }; + + EXPECT_EQ( + xlings::xim::decide_cache_admission_(input), + xlings::xim::CacheAdmission_::Redownload); +} + +TEST(XimDownloaderTest, HeadFailureAcceptsMatchingCommittedV2Cache) { + xlings::xim::CacheAdmissionInput_ input { + .localSize = 12628937, + .headSucceeded = false, + .sidecar = xlings::xim::MetaSidecar_ { + .format = 2, + .complete = true, + .size = 12628937, + .cacheIdentity = "mcpp/0.0.81/linux/x86_64/xlings-res", + }, + .expectedCacheIdentity = "mcpp/0.0.81/linux/x86_64/xlings-res", + }; + + EXPECT_EQ( + xlings::xim::decide_cache_admission_(input), + xlings::xim::CacheAdmission_::OfflineUnverifiedHit); +} + +TEST(XimDownloaderTest, HeadFailureRejectsV2CacheWithWrongSizeOrIdentity) { + xlings::xim::CacheAdmissionInput_ input { + .localSize = 114 * 1024, + .headSucceeded = false, + .sidecar = xlings::xim::MetaSidecar_ { + .format = 2, + .complete = true, + .size = 12628937, + .cacheIdentity = "other/0.0.81/linux/x86_64/xlings-res", + }, + .expectedCacheIdentity = "mcpp/0.0.81/linux/x86_64/xlings-res", + }; + + EXPECT_EQ( + xlings::xim::decide_cache_admission_(input), + xlings::xim::CacheAdmission_::Redownload); +} + +TEST(XimDownloaderTest, FailedTransferPreservesCommittedDestination) { + namespace fs = std::filesystem; + auto tmp = fs::temp_directory_path() / "xim_download_transaction_failure"; + fs::remove_all(tmp); + fs::create_directories(tmp); + auto destination = tmp / "payload.tar.gz"; + { + std::ofstream out(destination); + out << "previous-good-payload"; + } + + xlings::xim::DownloadTask task { + .name = "transaction-test", + .url = "https://example.test/payload.tar.gz", + .cacheIdentity = "transaction-test/1/linux/x86_64/url", + .destDir = tmp, + }; + xlings::xim::DownloadTestHooks_ hooks; + hooks.queryRemoteMeta = [](const std::string&) { + return xlings::tinyhttps::RemoteFileMeta{ + .ok = true, + .contentLength = 999, + }; + }; + hooks.transferOverride = [](const std::string&, const fs::path& path) { + std::ofstream(path) << "partial"; + return xlings::tinyhttps::DownloadFileResult{false, "connection reset"}; + }; + + auto result = xlings::xim::download_one(task, nullptr, nullptr, &hooks); + EXPECT_FALSE(result.success); + EXPECT_EQ(xlings::platform::read_file_to_string(destination.string()), + "previous-good-payload"); + for (const auto& entry : fs::directory_iterator(tmp)) { + EXPECT_FALSE(entry.path().filename().string().contains(".part.")); + } + fs::remove_all(tmp); +} + +TEST(XimDownloaderTest, HashRejectedCandidateCommitsFallbackFromStaging) { + namespace fs = std::filesystem; + auto tmp = fs::temp_directory_path() / "xim_download_transaction_fallback"; + fs::remove_all(tmp); + fs::create_directories(tmp); + + const std::string goodPayload = "fallback-good-payload"; + auto expectedHash = xlings::sha256::hex(goodPayload); + xlings::xim::DownloadTask task { + .name = "fallback-test", + .url = "https://mirror.test/payload.bin", + .sha256 = expectedHash, + .cacheIdentity = "fallback-test/1/linux/x86_64/url", + .destDir = tmp, + .fallbackUrls = {"https://origin.test/payload.bin"}, + }; + int attempts = 0; + xlings::xim::DownloadTestHooks_ hooks; + hooks.transferOverride = [&](const std::string&, const fs::path& path) { + std::ofstream(path) << (++attempts == 1 ? "bad" : goodPayload); + return xlings::tinyhttps::DownloadFileResult{true, {}}; + }; + + auto result = xlings::xim::download_one(task, nullptr, nullptr, &hooks); + ASSERT_TRUE(result.success) << result.error; + EXPECT_EQ(attempts, 2); + EXPECT_EQ(xlings::platform::read_file_to_string(result.localFile.string()), + goodPayload); + for (const auto& entry : fs::directory_iterator(tmp)) { + EXPECT_FALSE(entry.path().filename().string().contains(".part.")); + } + fs::remove_all(tmp); +} + +TEST(XimDownloaderTest, CancelledDownloadPreservesCommittedDestination) { + namespace fs = std::filesystem; + auto tmp = fs::temp_directory_path() / "xim_download_transaction_cancel"; + fs::remove_all(tmp); + fs::create_directories(tmp); + auto destination = tmp / "payload.bin"; + std::ofstream(destination) << "previous-good-payload"; + + xlings::xim::DownloadTask task { + .name = "cancel-test", + .url = "https://example.test/payload.bin", + .cacheIdentity = "cancel-test/1/linux/x86_64/url", + .destDir = tmp, + }; + int transfers = 0; + xlings::xim::DownloadTestHooks_ hooks; + hooks.queryRemoteMeta = [](const std::string&) { + return xlings::tinyhttps::RemoteFileMeta{ + .ok = true, + .contentLength = 999, + }; + }; + hooks.transferOverride = [&](const std::string&, const fs::path&) { + ++transfers; + return xlings::tinyhttps::DownloadFileResult{true, {}}; + }; + xlings::CancellationToken cancellation; + cancellation.cancel(); + + auto result = xlings::xim::download_one( + task, nullptr, &cancellation, &hooks); + EXPECT_FALSE(result.success); + EXPECT_EQ(result.error, "cancelled"); + EXPECT_EQ(transfers, 0); + EXPECT_EQ(xlings::platform::read_file_to_string(destination.string()), + "previous-good-payload"); + fs::remove_all(tmp); +} + +TEST(XimDownloaderTest, CommitFailureAfterBackupRestoresPreviousFile) { + namespace fs = std::filesystem; + auto tmp = fs::temp_directory_path() / "xim_download_commit_restore"; + fs::remove_all(tmp); + fs::create_directories(tmp); + auto destination = tmp / "payload.bin"; + auto staging = tmp / "payload.bin.part.test"; + std::ofstream(destination) << "previous-good-payload"; + std::ofstream(staging) << "replacement-payload"; + + std::string error; + EXPECT_FALSE(xlings::xim::commit_staging_file_( + staging, destination, error, true)); + EXPECT_EQ(error, "injected commit failure after backup"); + EXPECT_EQ(xlings::platform::read_file_to_string(destination.string()), + "previous-good-payload"); + EXPECT_TRUE(fs::exists(staging)); + fs::remove_all(tmp); +} + +TEST(XimDownloaderTest, FileLockWaitsForOwnerAndHonorsCancellation) { + namespace fs = std::filesystem; + auto tmp = fs::temp_directory_path() / "xim_download_file_lock"; + fs::remove_all(tmp); + fs::create_directories(tmp); + auto path = tmp / "payload.lock"; + + xlings::platform::FileLock owner; + std::string error; + ASSERT_TRUE(owner.acquire( + path, std::chrono::seconds{1}, {}, error)) << error; + + xlings::platform::FileLock cancelledWaiter; + EXPECT_FALSE(cancelledWaiter.acquire( + path, std::chrono::seconds{1}, [] { return true; }, error)); + EXPECT_EQ(error, "cancelled while waiting for cache lock"); + + std::jthread releaser([&] { + std::this_thread::sleep_for(std::chrono::milliseconds{100}); + owner.release(); + }); + xlings::platform::FileLock waiter; + error.clear(); + auto start = std::chrono::steady_clock::now(); + ASSERT_TRUE(waiter.acquire( + path, std::chrono::seconds{1}, {}, error)) << error; + EXPECT_GE( + std::chrono::steady_clock::now() - start, + std::chrono::milliseconds{50}); + fs::remove_all(tmp); +} + +TEST(XimDownloaderTest, RecoveryRestoresBackupWhenLiveIsMissing) { + namespace fs = std::filesystem; + auto tmp = fs::temp_directory_path() / "xim_download_recover_backup"; + fs::remove_all(tmp); + fs::create_directories(tmp); + auto destination = tmp / "payload.bin"; + auto backup = tmp / "payload.bin.old.100.1"; + auto staging = tmp / "payload.bin.part.100.1"; + std::ofstream(backup) << "previous-good-payload"; + std::ofstream(staging) << "partial"; + + std::string error; + ASSERT_TRUE(xlings::xim::recover_download_transaction_( + destination, error)) << error; + EXPECT_EQ(xlings::platform::read_file_to_string(destination.string()), + "previous-good-payload"); + EXPECT_FALSE(fs::exists(backup)); + EXPECT_FALSE(fs::exists(staging)); + fs::remove_all(tmp); +} + +TEST(XimDownloaderTest, RecoveryKeepsLiveAndRemovesStaleBackup) { + namespace fs = std::filesystem; + auto tmp = fs::temp_directory_path() / "xim_download_recover_live"; + fs::remove_all(tmp); + fs::create_directories(tmp); + auto destination = tmp / "payload.bin"; + auto backup = tmp / "payload.bin.old.100.1"; + std::ofstream(destination) << "committed-payload"; + std::ofstream(backup) << "previous-payload"; + + std::string error; + ASSERT_TRUE(xlings::xim::recover_download_transaction_( + destination, error)) << error; + EXPECT_EQ(xlings::platform::read_file_to_string(destination.string()), + "committed-payload"); + EXPECT_FALSE(fs::exists(backup)); + fs::remove_all(tmp); +} + // ============================================================ // xim installer tests // ============================================================ diff --git a/tests/unit/test_mirror.cpp b/tests/unit/test_mirror.cpp index cb94cb17..80ee2c92 100644 --- a/tests/unit/test_mirror.cpp +++ b/tests/unit/test_mirror.cpp @@ -15,6 +15,32 @@ namespace fs = std::filesystem; namespace { +class MirrorTestEnvironment final : public testing::Environment { +public: + void SetUp() override { + home_ = fs::temp_directory_path() + / std::format("xlings-mirror-test-{}", + std::chrono::steady_clock::now().time_since_epoch().count()); + fs::create_directories(home_); +#ifdef _WIN32 + _putenv_s("XLINGS_HOME", home_.string().c_str()); +#else + ::setenv("XLINGS_HOME", home_.string().c_str(), 1); +#endif + } + + void TearDown() override { + std::error_code ec; + fs::remove_all(home_, ec); + } + +private: + fs::path home_; +}; + +[[maybe_unused]] auto* mirror_test_environment_ = + testing::AddGlobalTestEnvironment(new MirrorTestEnvironment); + // Helper: any item in the list whose URL starts with the given prefix. bool any_starts_with(const std::vector& urls, std::string_view prefix) { return std::ranges::any_of(urls, [&](const auto& u) { From 92faef1ca4804eec3bcd2ef7ed66fcee852d9c1b Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sun, 12 Jul 2026 02:02:04 +0800 Subject: [PATCH 03/10] fix(xim): evict only invalid archive caches --- ...sue-356-partial-download-cache-fix-plan.md | 10 ++- src/core/xim/extract.cppm | 75 ++++++++++++++----- src/core/xim/installer.cppm | 30 +++++++- tests/unit/test_main.cpp | 34 +++++++++ 4 files changed, 127 insertions(+), 22 deletions(-) diff --git a/.agents/docs/2026-07-11-issue-356-partial-download-cache-fix-plan.md b/.agents/docs/2026-07-11-issue-356-partial-download-cache-fix-plan.md index a2f2482b..7a13b251 100644 --- a/.agents/docs/2026-07-11-issue-356-partial-download-cache-fix-plan.md +++ b/.agents/docs/2026-07-11-issue-356-partial-download-cache-fix-plan.md @@ -271,7 +271,7 @@ etag: | X1 | `openxlings/xlings` | 缓存决策纯函数与 sidecar v2 | 已完成（本地） | `mcpp test`：legacy/v2/size/identity 回归通过；全套 10/10 测试二进制通过 | | X2 | `openxlings/xlings` | 唯一 staging 文件、验收后可恢复提交 | 已完成（本地） | 失败/取消保留旧目标；hash fallback；backup 后失败恢复；全套 `mcpp test` 通过 | | X3 | `openxlings/xlings` | 同目标并发锁与崩溃恢复 | 进行中 | POSIX/Windows 实现及本地测试通过；待双进程测试与三平台 CI | -| X4 | `openxlings/xlings` | 归档输入错误驱逐缓存 | 待开始 | 截断错误驱逐、权限/写入错误保留缓存 | +| X4 | `openxlings/xlings` | 归档输入错误驱逐缓存 | 已完成（本地） | 非法/截断输入驱逐；本地目标写错误保留；全套 `mcpp test` 通过 | | T1 | `mcpplibs/tinyhttps` | chunked EOF 严格判定与传输元数据 | 待开始 | 协议测试全部通过并发布新版本 | | L1 | `mcpplibs/libxpkg` | `xpm.source` 解析、兼容、资源归一化 | 待开始 | 旧写法及新 xlings-res/URL template 契约测试 | | L2 | `openxlings/xlings` | 删除 `load_platform_entries_()` 重复解析并接入 libxpkg | 待开始 | xpkg 解析只剩一个入口；旧索引安装回归通过 | @@ -355,6 +355,14 @@ I2 依赖 L1/L2 的兼容矩阵通过，但不阻塞 I1 的 mcpp 止血 2. 前者删除下载缓存及 sidecar，后者保留缓存并返回环境错误。 3. 不增加下载后的完整归档预扫描；测试直接构造截断归档和不可写目标。 +**2026-07-12 实施记录**： + +- 新增兼容 API `extract_archive_detailed()`，返回 `InvalidInputArchive`、`LocalWriteFailure` 或 `Internal`；原 `extract_archive()` 字符串接口保留。 +- open/next-header/read-data/不安全归档路径归为输入错误；创建目录、write-header/write-data/finish-entry 归为本地写错误。 +- installer 只在 `InvalidInputArchive` 时删除归档和 `.meta`；本地写错误保留缓存。 +- 非法归档驱逐及“目标是普通文件”本地写失败保留缓存测试通过；未增加归档预扫描。 +- 完整 `mcpp test`：10 passed，0 failed。 + ### 9.6 T1：tinyhttps 协议完整性 在独立仓库和独立 PR 中补协议级本地 HTTP server 测试，再修 chunk parser。成功结果公开 `bytesWritten`、可选 `expectedBytes`、`finalUrl`；xlings 在升级依赖后使用这些字段写 sidecar和日志。connection-close 响应的 `expectedBytes` 必须为空，不能被描述为长度已验证。 diff --git a/src/core/xim/extract.cppm b/src/core/xim/extract.cppm index 7ad330c3..6d578ef9 100644 --- a/src/core/xim/extract.cppm +++ b/src/core/xim/extract.cppm @@ -10,6 +10,21 @@ import std; export namespace xlings::xim { +enum class ExtractErrorKind { + InvalidInputArchive, + LocalWriteFailure, + Internal, +}; + +struct ExtractError { + ExtractErrorKind kind { ExtractErrorKind::Internal }; + std::string message; +}; + +std::expected +extract_archive_detailed(const std::filesystem::path& archive, + const std::filesystem::path& destDir); + // In-process archive extraction backed by libarchive. // // Replaces the previous popen("tar xf …") path that suffered from a @@ -124,7 +139,7 @@ la_int64_t const_root_lookup_uid_(void*, const char*, la_int64_t) { return 0; } la_int64_t const_root_lookup_gid_(void*, const char*, la_int64_t) { return 0; } // Read each block of an entry's payload from `src` and write to `dst`. -std::expected +std::expected copy_entry_data_(struct archive* src, struct archive* dst) { const void* buff = nullptr; std::size_t size = 0; @@ -133,11 +148,15 @@ copy_entry_data_(struct archive* src, struct archive* dst) { int r = ::archive_read_data_block(src, &buff, &size, &offset); if (r == ARCHIVE_EOF) return {}; if (r < ARCHIVE_OK) { - return std::unexpected("read_data_block: " + libarchive_error_(src)); + return std::unexpected(ExtractError{ + ExtractErrorKind::InvalidInputArchive, + "read_data_block: " + libarchive_error_(src)}); } r = ::archive_write_data_block(dst, buff, size, offset); if (r < ARCHIVE_OK) { - return std::unexpected("write_data_block: " + libarchive_error_(dst)); + return std::unexpected(ExtractError{ + ExtractErrorKind::LocalWriteFailure, + "write_data_block: " + libarchive_error_(dst)}); } } } @@ -160,23 +179,25 @@ void ensure_archive_locale_() { } // namespace detail_ -std::expected -extract_archive(const std::filesystem::path& archive, - const std::filesystem::path& destDir) { +std::expected +extract_archive_detailed(const std::filesystem::path& archive, + const std::filesystem::path& destDir) { namespace fs = std::filesystem; detail_::ensure_archive_locale_(); std::error_code ec; fs::create_directories(destDir, ec); if (ec) { - return std::unexpected(std::format( - "create_directories({}) failed: {}", - destDir.string(), ec.message())); + return std::unexpected(ExtractError{ + ExtractErrorKind::LocalWriteFailure, + std::format("create_directories({}) failed: {}", + destDir.string(), ec.message())}); } if (!fs::exists(archive)) { - return std::unexpected(std::format( - "archive does not exist: {}", archive.string())); + return std::unexpected(ExtractError{ + ExtractErrorKind::InvalidInputArchive, + std::format("archive does not exist: {}", archive.string())}); } // Resolve symlinks in the destination root before handing paths to @@ -205,7 +226,9 @@ extract_archive(const std::filesystem::path& archive, if (!src || !dst) { cleanup(); - return std::unexpected("libarchive: failed to allocate handles"); + return std::unexpected(ExtractError{ + ExtractErrorKind::Internal, + "libarchive: failed to allocate handles"}); } ::archive_read_support_filter_all(src); @@ -239,7 +262,8 @@ extract_archive(const std::filesystem::path& archive, std::string err = std::format("open {}: {}", archive.string(), detail_::libarchive_error_(src)); cleanup(); - return std::unexpected(std::move(err)); + return std::unexpected(ExtractError{ + ExtractErrorKind::InvalidInputArchive, std::move(err)}); } for (;;) { @@ -249,7 +273,8 @@ extract_archive(const std::filesystem::path& archive, if (r < ARCHIVE_WARN) { std::string err = "next_header: " + detail_::libarchive_error_(src); cleanup(); - return std::unexpected(std::move(err)); + return std::unexpected(ExtractError{ + ExtractErrorKind::InvalidInputArchive, std::move(err)}); } // Reroot the entry under destDir. archive_entry_pathname is a @@ -258,7 +283,9 @@ extract_archive(const std::filesystem::path& archive, auto safeRel = detail_::check_safe_pathname_(original.c_str()); if (!safeRel) { cleanup(); - return std::unexpected(std::move(safeRel).error()); + return std::unexpected(ExtractError{ + ExtractErrorKind::InvalidInputArchive, + std::move(safeRel).error()}); } auto rebased = (canonicalDest / *safeRel).lexically_normal().string(); ::archive_entry_set_pathname(entry, rebased.c_str()); @@ -272,7 +299,9 @@ extract_archive(const std::filesystem::path& archive, auto safeHl = detail_::check_safe_pathname_(hardlink.c_str()); if (!safeHl) { cleanup(); - return std::unexpected(std::move(safeHl).error()); + return std::unexpected(ExtractError{ + ExtractErrorKind::InvalidInputArchive, + std::move(safeHl).error()}); } auto rebasedHl = (canonicalDest / *safeHl).lexically_normal().string(); ::archive_entry_set_hardlink(entry, rebasedHl.c_str()); @@ -303,7 +332,8 @@ extract_archive(const std::filesystem::path& archive, "write_header({}): {}", rebased, detail_::libarchive_error_(dst)); cleanup(); - return std::unexpected(std::move(err)); + return std::unexpected(ExtractError{ + ExtractErrorKind::LocalWriteFailure, std::move(err)}); } } @@ -319,7 +349,8 @@ extract_archive(const std::filesystem::path& archive, "finish_entry({}): {}", rebased, detail_::libarchive_error_(dst)); cleanup(); - return std::unexpected(std::move(err)); + return std::unexpected(ExtractError{ + ExtractErrorKind::LocalWriteFailure, std::move(err)}); } } @@ -327,4 +358,12 @@ extract_archive(const std::filesystem::path& archive, return canonicalDest; } +std::expected +extract_archive(const std::filesystem::path& archive, + const std::filesystem::path& destDir) { + auto result = extract_archive_detailed(archive, destDir); + if (!result) return std::unexpected(std::move(result).error().message); + return *result; +} + } // namespace xlings::xim diff --git a/src/core/xim/installer.cppm b/src/core/xim/installer.cppm index dc9c919f..0f178f02 100644 --- a/src/core/xim/installer.cppm +++ b/src/core/xim/installer.cppm @@ -25,6 +25,19 @@ import xlings.runtime.cancellation; export namespace xlings::xim { +bool evict_invalid_archive_cache_( + const std::filesystem::path& archive, + const ExtractError& error) { + if (error.kind != ExtractErrorKind::InvalidInputArchive) return false; + std::error_code ec; + bool removed = std::filesystem::remove(archive, ec); + auto sidecar = archive; + sidecar += ".meta"; + ec.clear(); + std::filesystem::remove(sidecar, ec); + return removed; +} + namespace detail_ { namespace lua = mcpplibs::capi::lua; @@ -1387,11 +1400,22 @@ public: } // Extract into the same runtime dir as the download auto runtimeDir = dlIt->second.localFile.parent_path(); - auto extracted = extract_archive(dlIt->second.localFile, runtimeDir); + auto extracted = extract_archive_detailed( + dlIt->second.localFile, runtimeDir); if (!extracted) { - log::error("extract failed for {}: {}", node.name, extracted.error()); + auto error = std::move(extracted).error(); + if (evict_invalid_archive_cache_( + dlIt->second.localFile, error)) { + log::warn( + "evicted invalid archive cache for {}: {}", + node.name, dlIt->second.localFile.string()); + } + log::error("extract failed for {}: {}", + node.name, error.message); if (onStatus) { - onStatus({ node.name, InstallPhase::Failed, 0.0f, extracted.error() }); + onStatus({ + node.name, InstallPhase::Failed, 0.0f, + error.message}); } continue; } diff --git a/tests/unit/test_main.cpp b/tests/unit/test_main.cpp index cd2be060..342de311 100644 --- a/tests/unit/test_main.cpp +++ b/tests/unit/test_main.cpp @@ -3484,6 +3484,40 @@ TEST(Extract, MissingArchiveReturnsError) { EXPECT_FALSE(r.has_value()); } +TEST(Extract, InvalidArchiveIsClassifiedAndEvictedWithSidecar) { + ExtractFixture fx; + auto archive = fx.tmp / "truncated.tar.gz"; + auto sidecar = std::filesystem::path(archive.string() + ".meta"); + std::ofstream(archive) << "not-a-complete-archive"; + std::ofstream(sidecar) << "format: 2\n"; + + auto result = xlings::xim::extract_archive_detailed( + archive, fx.tmp / "invalid-out"); + ASSERT_FALSE(result.has_value()); + EXPECT_EQ(result.error().kind, + xlings::xim::ExtractErrorKind::InvalidInputArchive); + EXPECT_TRUE(xlings::xim::evict_invalid_archive_cache_( + archive, result.error())); + EXPECT_FALSE(std::filesystem::exists(archive)); + EXPECT_FALSE(std::filesystem::exists(sidecar)); +} + +TEST(Extract, LocalWriteFailureDoesNotEvictValidInput) { + ExtractFixture fx; + auto archive = fx.make_tar_gz(); + auto invalidDestination = fx.tmp / "destination-is-a-file"; + std::ofstream(invalidDestination) << "not-a-directory"; + + auto result = xlings::xim::extract_archive_detailed( + archive, invalidDestination); + ASSERT_FALSE(result.has_value()); + EXPECT_EQ(result.error().kind, + xlings::xim::ExtractErrorKind::LocalWriteFailure); + EXPECT_FALSE(xlings::xim::evict_invalid_archive_cache_( + archive, result.error())); + EXPECT_TRUE(std::filesystem::exists(archive)); +} + TEST(Extract, RejectsPathTraversal) { // Build a tarball containing an entry with "../escape.txt". libarchive // with ARCHIVE_EXTRACT_SECURE_NODOTDOT must refuse to extract the From 848079cf87d1933a2595d131b4c8a576ba272ad1 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sun, 12 Jul 2026 02:15:55 +0800 Subject: [PATCH 04/10] docs(issue356): record upstream implementation progress --- ...ssue-356-partial-download-cache-fix-plan.md | 18 ++++++++++++++++-- 1 file changed, 16 insertions(+), 2 deletions(-) diff --git a/.agents/docs/2026-07-11-issue-356-partial-download-cache-fix-plan.md b/.agents/docs/2026-07-11-issue-356-partial-download-cache-fix-plan.md index 7a13b251..5ddbc6a4 100644 --- a/.agents/docs/2026-07-11-issue-356-partial-download-cache-fix-plan.md +++ b/.agents/docs/2026-07-11-issue-356-partial-download-cache-fix-plan.md @@ -272,8 +272,8 @@ etag: | X2 | `openxlings/xlings` | 唯一 staging 文件、验收后可恢复提交 | 已完成（本地） | 失败/取消保留旧目标；hash fallback；backup 后失败恢复；全套 `mcpp test` 通过 | | X3 | `openxlings/xlings` | 同目标并发锁与崩溃恢复 | 进行中 | POSIX/Windows 实现及本地测试通过；待双进程测试与三平台 CI | | X4 | `openxlings/xlings` | 归档输入错误驱逐缓存 | 已完成（本地） | 非法/截断输入驱逐；本地目标写错误保留；全套 `mcpp test` 通过 | -| T1 | `mcpplibs/tinyhttps` | chunked EOF 严格判定与传输元数据 | 待开始 | 协议测试全部通过并发布新版本 | -| L1 | `mcpplibs/libxpkg` | `xpm.source` 解析、兼容、资源归一化 | 待开始 | 旧写法及新 xlings-res/URL template 契约测试 | +| T1 | `mcpplibs/tinyhttps` | chunked EOF 严格判定与传输元数据 | PR CI 已通过，待发布 | [tinyhttps#9](https://github.com/mcpplibs/tinyhttps/pull/9)；Linux mcpp build/test 通过；待合并并发布 0.2.9 | +| L1 | `mcpplibs/libxpkg` | `xpm.source` 解析、兼容、资源归一化 | PR 待 CI/发布 | [libxpkg#26](https://github.com/openxlings/libxpkg/pull/26)；loader/compat 26/26 通过；待 CI、合并并发布 0.0.43 | | L2 | `openxlings/xlings` | 删除 `load_platform_entries_()` 重复解析并接入 libxpkg | 待开始 | xpkg 解析只剩一个入口；旧索引安装回归通过 | | I1 | `openxlings/xim-pkgindex` | mcpp 活跃版本补 hash，生成器禁止新裸条目 | 待开始 | GLOBAL/CN hash 一致；索引 CI 通过 | | I2 | `openxlings/xim-pkgindex` | 官方资源分批迁移与新旧客户端兼容测试 | 待开始 | 旧客户端读取保留条目，新客户端读取推荐条目 | @@ -367,10 +367,24 @@ I2 依赖 L1/L2 的兼容矩阵通过，但不阻塞 I1 的 mcpp 止血 在独立仓库和独立 PR 中补协议级本地 HTTP server 测试，再修 chunk parser。成功结果公开 `bytesWritten`、可选 `expectedBytes`、`finalUrl`；xlings 在升级依赖后使用这些字段写 sidecar和日志。connection-close 响应的 `expectedBytes` 必须为空，不能被描述为长度已验证。 +**2026-07-12 实施记录**： + +- [tinyhttps#9](https://github.com/mcpplibs/tinyhttps/pull/9) 已实现严格 chunk-size/CRLF/trailer EOF 判定，并公开 `bytesWritten`、可选 `expectedBytes`、`finalUrl`、`etag` 和 `lastModified`。 +- `mcpp test` 的本地 HTTPS 与协议回归测试通过；PR 的 Linux mcpp CI 已通过。 +- `mcpp.toml` 已升至 0.2.9；尚未合并、打 tag 和发布，因此 T1 不能标记完成。 + ### 9.7 L1/L2：唯一 xpkg 入口 `libxpkg` 接受并归一化原有版本资源项以及 `xpm.source = "xlings-res" | `，输出平台、架构、最终 URL、SHA256 和 fallback 的统一资源对象。兼容逻辑只位于 libxpkg 的 compat 模块。xlings 升级依赖并删除 `load_platform_entries_()`；删除前必须用同一组 fixture 对旧字符串 URL、`XLINGS_RES`、mirror table、`ref`、单 hash 和多架构 hash 做前后结果对比。 +**2026-07-12 L1 实施记录**： + +- [libxpkg#26](https://github.com/openxlings/libxpkg/pull/26) 新增 `mcpplibs.xpkg.compat`，`resolve_resource()` 成为旧资源写法、`xlings-res` 与 URL template 的统一归一化入口。 +- loader 将根级/平台级 `source` 解析为元数据并明确跳过版本迭代；原 platform/version 模型保持不变。 +- 契约测试覆盖显式 URL 优先级、根级/平台级 source、`${name/version/os/arch/arch_alias/ext}`、mirror 展开、ref/cycle、旧 `XLINGS_RES`、`res=true`、per-arch map 和 per-arch SHA256；loader/compat 26/26 通过。 +- `mcpp build` 通过。完整 `mcpp test` 的 4 个 elfpatch executor 失败已在干净 `main@9e934be` 原样复现，确认不是 L1 回归。 +- `mcpp.toml` 已升至 0.0.43；尚待 PR CI、合并和发布，随后才能开始 L2 的依赖升级与重复解析器删除。 + ### 9.8 I1/I2：索引迁移与老客户端保护 新增 `xpm.source` 是可选字段，不会改变旧的 `xpm[platform][version]` 条目；因此迁移采用“双轨兼容”，而不是立即重写所有历史条目： From 737940db5e964470ed4bab2935dabccd35c43835 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sun, 12 Jul 2026 02:19:58 +0800 Subject: [PATCH 05/10] test(xim): verify cache lock across processes --- ...sue-356-partial-download-cache-fix-plan.md | 5 +- tests/unit/test_main.cpp | 55 +++++++++++++++++++ 2 files changed, 58 insertions(+), 2 deletions(-) diff --git a/.agents/docs/2026-07-11-issue-356-partial-download-cache-fix-plan.md b/.agents/docs/2026-07-11-issue-356-partial-download-cache-fix-plan.md index 5ddbc6a4..2fb10bb4 100644 --- a/.agents/docs/2026-07-11-issue-356-partial-download-cache-fix-plan.md +++ b/.agents/docs/2026-07-11-issue-356-partial-download-cache-fix-plan.md @@ -270,7 +270,7 @@ etag: |---|---|---|---|---| | X1 | `openxlings/xlings` | 缓存决策纯函数与 sidecar v2 | 已完成（本地） | `mcpp test`：legacy/v2/size/identity 回归通过；全套 10/10 测试二进制通过 | | X2 | `openxlings/xlings` | 唯一 staging 文件、验收后可恢复提交 | 已完成（本地） | 失败/取消保留旧目标；hash fallback；backup 后失败恢复；全套 `mcpp test` 通过 | -| X3 | `openxlings/xlings` | 同目标并发锁与崩溃恢复 | 进行中 | POSIX/Windows 实现及本地测试通过；待双进程测试与三平台 CI | +| X3 | `openxlings/xlings` | 同目标并发锁与崩溃恢复 | 已完成（本地） | POSIX/Windows 实现、真实双进程竞争及恢复测试通过；待 PR 三平台 CI | | X4 | `openxlings/xlings` | 归档输入错误驱逐缓存 | 已完成（本地） | 非法/截断输入驱逐；本地目标写错误保留；全套 `mcpp test` 通过 | | T1 | `mcpplibs/tinyhttps` | chunked EOF 严格判定与传输元数据 | PR CI 已通过，待发布 | [tinyhttps#9](https://github.com/mcpplibs/tinyhttps/pull/9)；Linux mcpp build/test 通过；待合并并发布 0.2.9 | | L1 | `mcpplibs/libxpkg` | `xpm.source` 解析、兼容、资源归一化 | PR 待 CI/发布 | [libxpkg#26](https://github.com/openxlings/libxpkg/pull/26)；loader/compat 26/26 通过；待 CI、合并并发布 0.0.43 | @@ -345,7 +345,8 @@ I2 依赖 L1/L2 的兼容矩阵通过，但不阻塞 I1 的 mcpp 止血 - 锁已覆盖文件缓存判定、传输、提交和 sidecar 写入。 - 持锁后的恢复状态机已覆盖“仅 backup 时恢复 live”“live + backup 时保留 live”“清理同目标遗留 staging”。 - 同进程双句柄竞争、取消等待、延迟释放、两种崩溃恢复测试通过；完整 `mcpp test` 10/10 通过。 -- 尚缺真实双进程竞争测试以及 macOS/Windows CI 证据，因此本项仍为“进行中”。 +- 新增真实子进程持锁测试：父进程在子进程写入 ready 标记后竞争同一锁，必须等待子进程释放，再完成提交；Linux 定向测试通过。 +- 实现与本地测试已完成；在 PR 的 Linux、macOS、Windows CI 全绿前，本项只标记“已完成（本地）”。 ### 9.5 X4：解压失败自愈 diff --git a/tests/unit/test_main.cpp b/tests/unit/test_main.cpp index 342de311..a257530d 100644 --- a/tests/unit/test_main.cpp +++ b/tests/unit/test_main.cpp @@ -1092,6 +1092,50 @@ TEST(XimDownloaderTest, FileLockWaitsForOwnerAndHonorsCancellation) { fs::remove_all(tmp); } +TEST(XimDownloaderTest, FileLockSerializesIndependentProcesses) { + namespace fs = std::filesystem; + auto tmp = fs::temp_directory_path() + / std::format("xim_download_process_lock_{}", xlings::platform::get_pid()); + fs::remove_all(tmp); + fs::create_directories(tmp); + auto lock_path = tmp / "payload.lock"; + auto ready_path = tmp / "child.ready"; + auto executable = xlings::platform::get_executable_path(); + ASSERT_FALSE(executable.empty()); + + auto command = std::format( + "\"{}\" --file-lock-child \"{}\" \"{}\"", + executable.string(), lock_path.string(), ready_path.string()); + auto child = xlings::platform::spawn_command(command); + ASSERT_GT(child.pid, 0); + + auto ready_deadline = std::chrono::steady_clock::now() + + std::chrono::seconds{3}; + while (!fs::exists(ready_path) + && std::chrono::steady_clock::now() < ready_deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds{10}); + } + + xlings::platform::FileLock waiter; + std::string error; + auto start = std::chrono::steady_clock::now(); + bool acquired = false; + if (fs::exists(ready_path)) { + acquired = waiter.acquire( + lock_path, std::chrono::seconds{2}, {}, error); + } + auto waited = std::chrono::steady_clock::now() - start; + auto [child_status, child_output] = xlings::platform::wait_or_kill( + child, nullptr, std::chrono::seconds{3}); + + EXPECT_TRUE(fs::exists(ready_path)) << child_output; + EXPECT_TRUE(acquired) << error << "\n" << child_output; + EXPECT_GE(waited, std::chrono::milliseconds{150}); + EXPECT_EQ(child_status, 0) << child_output; + waiter.release(); + fs::remove_all(tmp); +} + TEST(XimDownloaderTest, RecoveryRestoresBackupWhenLiveIsMissing) { namespace fs = std::filesystem; auto tmp = fs::temp_directory_path() / "xim_download_recover_backup"; @@ -3933,6 +3977,17 @@ TEST(DownloaderArchiveSniff, WorksWithFullPaths) { #ifndef XLINGS_USE_GTEST_MAIN int main(int argc, char** argv) { + if (argc == 4 && std::string_view(argv[1]) == "--file-lock-child") { + xlings::platform::FileLock lock; + std::string error; + if (!lock.acquire(argv[2], std::chrono::seconds{2}, {}, error)) { + std::cerr << error << '\n'; + return 2; + } + std::ofstream(argv[3]) << "ready"; + std::this_thread::sleep_for(std::chrono::milliseconds{400}); + return 0; + } ::testing::InitGoogleTest(&argc, argv); return RUN_ALL_TESTS(); } From 54ab6951391cc28d498a7ed72a4b1c4df576afc4 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sun, 12 Jul 2026 02:37:33 +0800 Subject: [PATCH 06/10] feat(xim): centralize resource resolution in libxpkg --- ...sue-356-partial-download-cache-fix-plan.md | 16 +- ...6-07-11-xpkg-resource-expression-design.md | 398 +++++++++++++++++ src/core/xim/downloader.cppm | 36 +- src/core/xim/installer.cppm | 421 +++--------------- src/libs/tinyhttps.cppm | 26 +- tests/unit/test_main.cpp | 114 +++++ 6 files changed, 646 insertions(+), 365 deletions(-) create mode 100644 .agents/docs/2026-07-11-xpkg-resource-expression-design.md diff --git a/.agents/docs/2026-07-11-issue-356-partial-download-cache-fix-plan.md b/.agents/docs/2026-07-11-issue-356-partial-download-cache-fix-plan.md index 2fb10bb4..5f759ad0 100644 --- a/.agents/docs/2026-07-11-issue-356-partial-download-cache-fix-plan.md +++ b/.agents/docs/2026-07-11-issue-356-partial-download-cache-fix-plan.md @@ -272,9 +272,9 @@ etag: | X2 | `openxlings/xlings` | 唯一 staging 文件、验收后可恢复提交 | 已完成（本地） | 失败/取消保留旧目标；hash fallback；backup 后失败恢复；全套 `mcpp test` 通过 | | X3 | `openxlings/xlings` | 同目标并发锁与崩溃恢复 | 已完成（本地） | POSIX/Windows 实现、真实双进程竞争及恢复测试通过；待 PR 三平台 CI | | X4 | `openxlings/xlings` | 归档输入错误驱逐缓存 | 已完成（本地） | 非法/截断输入驱逐；本地目标写错误保留；全套 `mcpp test` 通过 | -| T1 | `mcpplibs/tinyhttps` | chunked EOF 严格判定与传输元数据 | PR CI 已通过，待发布 | [tinyhttps#9](https://github.com/mcpplibs/tinyhttps/pull/9)；Linux mcpp build/test 通过；待合并并发布 0.2.9 | -| L1 | `mcpplibs/libxpkg` | `xpm.source` 解析、兼容、资源归一化 | PR 待 CI/发布 | [libxpkg#26](https://github.com/openxlings/libxpkg/pull/26)；loader/compat 26/26 通过；待 CI、合并并发布 0.0.43 | -| L2 | `openxlings/xlings` | 删除 `load_platform_entries_()` 重复解析并接入 libxpkg | 待开始 | xpkg 解析只剩一个入口；旧索引安装回归通过 | +| T1 | `mcpplibs/tinyhttps` | chunked EOF 严格判定与传输元数据 | 已完成 | [tinyhttps#9](https://github.com/mcpplibs/tinyhttps/pull/9) 已合并；[0.2.9 release](https://github.com/mcpplibs/tinyhttps/releases/tag/0.2.9)；索引 [mcpp-index#68](https://github.com/mcpplibs/mcpp-index/pull/68) 待最终 CI/合并 | +| L1 | `mcpplibs/libxpkg` | `xpm.source`、平台兼容、资源归一化 | 已完成 | [libxpkg#26](https://github.com/openxlings/libxpkg/pull/26)、[#27](https://github.com/openxlings/libxpkg/pull/27) 已合并；[0.0.44 release](https://github.com/openxlings/libxpkg/releases/tag/v0.0.44) | +| L2 | `openxlings/xlings` | 删除 `load_platform_entries_()` 重复解析并接入 libxpkg | 已完成（本地） | 重复 parser/sandbox/template 展开器已删除；统一 compat 测试与完整 `mcpp test` 10/10 通过；待正式索引依赖和三平台 CI | | I1 | `openxlings/xim-pkgindex` | mcpp 活跃版本补 hash，生成器禁止新裸条目 | 待开始 | GLOBAL/CN hash 一致；索引 CI 通过 | | I2 | `openxlings/xim-pkgindex` | 官方资源分批迁移与新旧客户端兼容测试 | 待开始 | 旧客户端读取保留条目，新客户端读取推荐条目 | | R1 | 全生态 | PR、三平台 CI、版本升级、release、索引更新 | 待开始 | PR 合并、release 资产、索引 PR、端到端安装记录 | @@ -372,7 +372,9 @@ I2 依赖 L1/L2 的兼容矩阵通过，但不阻塞 I1 的 mcpp 止血 - [tinyhttps#9](https://github.com/mcpplibs/tinyhttps/pull/9) 已实现严格 chunk-size/CRLF/trailer EOF 判定，并公开 `bytesWritten`、可选 `expectedBytes`、`finalUrl`、`etag` 和 `lastModified`。 - `mcpp test` 的本地 HTTPS 与协议回归测试通过；PR 的 Linux mcpp CI 已通过。 -- `mcpp.toml` 已升至 0.2.9；尚未合并、打 tag 和发布，因此 T1 不能标记完成。 +- `mcpp.toml` 已升至 0.2.9；PR 已合并并发布 [0.2.9](https://github.com/mcpplibs/tinyhttps/releases/tag/0.2.9)。 +- xlings wrapper 已消费实际/期望字节、最终 URL、ETag 和 Last-Modified；长度不一致在提交缓存前拒绝，GET 元数据进入 sidecar。 +- mcpp 默认索引 PR [#68](https://github.com/mcpplibs/mcpp-index/pull/68) 已建立，GLOBAL/CN 同字节 SHA256 已验证，待完整 CI 和 artifact 发布。 ### 9.7 L1/L2：唯一 xpkg 入口 @@ -384,7 +386,11 @@ I2 依赖 L1/L2 的兼容矩阵通过，但不阻塞 I1 的 mcpp 止血 - loader 将根级/平台级 `source` 解析为元数据并明确跳过版本迭代；原 platform/version 模型保持不变。 - 契约测试覆盖显式 URL 优先级、根级/平台级 source、`${name/version/os/arch/arch_alias/ext}`、mirror 展开、ref/cycle、旧 `XLINGS_RES`、`res=true`、per-arch map 和 per-arch SHA256；loader/compat 26/26 通过。 - `mcpp build` 通过。完整 `mcpp test` 的 4 个 elfpatch executor 失败已在干净 `main@9e934be` 原样复现，确认不是 L1 回归。 -- `mcpp.toml` 已升至 0.0.43；尚待 PR CI、合并和发布，随后才能开始 L2 的依赖升级与重复解析器删除。 +- 兼容审计发现 0.0.43 loader 缺少目标平台 sandbox 上下文，不能安全替换 xlings 的旧 parser；因此没有把 0.0.43 推入默认索引。 +- [libxpkg#27](https://github.com/openxlings/libxpkg/pull/27) 增加 `LoaderContext { platform, arch }`，统一支持 `is_host/is_plat/is_arch`、`os.host/os.arch` 和 `_RUNTIME`，并把 per-arch 缺失收敛为 compat fail-closed。 +- #26/#27 CI 均通过并已合并，最终发布 [libxpkg 0.0.44](https://github.com/openxlings/libxpkg/releases/tag/v0.0.44)。 +- xlings 已删除 `load_platform_entries_()`、本地 Lua sandbox 和本地 template 展开器；平台上下文加载和资源归一化全部调用 libxpkg。 +- xlings 侧 source/ref/template/mirror/legacy/per-arch 契约测试与完整 `mcpp test` 10/10 通过；尚待 0.0.44 默认索引 PR 合并后切换正式依赖并跑三平台 CI。 ### 9.8 I1/I2：索引迁移与老客户端保护 diff --git a/.agents/docs/2026-07-11-xpkg-resource-expression-design.md b/.agents/docs/2026-07-11-xpkg-resource-expression-design.md new file mode 100644 index 00000000..eeddc4e7 --- /dev/null +++ b/.agents/docs/2026-07-11-xpkg-resource-expression-design.md @@ -0,0 +1,398 @@ +# xpm 资源表达与兼容设计 + +> 日期：2026-07-11 +> 修订：2026-07-12 +> 状态：已实现，待随 xlings 发布 +> 适用版本：libxpkg 0.0.44+ +> 范围：libxpkg 解析/compat、xlings 安装器、官方 xim-pkgindex + +## 1. 结论 + +不重新设计 `xpm`。平台和版本两层结构保持不变： + +```lua +xpm = { + linux = { + ["1.0.0"] = { + url = "https://example.test/foo-1.0.0-linux-x86_64.tar.gz", + sha256 = "", + }, + }, +} +``` + +只增加可选默认来源： + +```lua +xpm.source = "xlings-res" +``` + +或： + +```lua +xpm.source = "https://github.com/acme/foo/releases/download/${version}/foo-${version}-${os}-${arch}.tar.gz" +``` + +版本项、mirror、`ref`、显式 URL 和多架构结构仍使用原模型。`source` 只减少重复，不建立第二套 versions/targets DSL。 + +## 2. 设计目标 + +需要同时满足： + +1. 官方 xlings-res 不再为每个平台、每个版本重复写 `"XLINGS_RES"`。 +2. 同一 URL 模板的多个版本不再重复 URL。 +3. 单 hash、per-arch hash、架构别名和 mirror 继续可用。 +4. 不规则版本可用显式 URL 覆盖默认来源。 +5. 旧包继续工作；兼容逻辑只存在于 libxpkg。 +6. xlings 不再执行第二套 Lua 资源解析器。 + +## 3. 权威职责边界 + +```text +libxpkg loader + 读取 package/xpm，注入目标 platform/arch 兼容上下文 + ↓ +libxpkg compat::resolve_resource + ref、source 优先级、arch、hash、template、mirror 归一化 + ↓ +xlings installer + 选择资源服务器、生成 xlings-res URL/fallback、形成下载任务 + ↓ +xlings downloader + mirror 尝试、SHA256、事务缓存、sidecar +``` + +libxpkg 是唯一解析、兼容和资源归一化入口。xlings 只保留产品配置相关策略，例如用户选择的资源服务器和 fallback 列表。 + +## 4. source 语法 + +### 4.1 根级 source + +作用于所有平台： + +```lua +xpm = { + source = "xlings-res", + + linux = { + ["1.0.0"] = {}, + }, + macosx = { + ["1.0.0"] = {}, + }, +} +``` + +### 4.2 平台级 source + +覆盖根级 source： + +```lua +xpm = { + source = "xlings-res", + + windows = { + source = "https://vendor.test/foo/${version}/foo-${arch_alias}.zip", + ["1.0.0"] = { + arch_alias = { x86_64 = "win64" }, + sha256 = { x86_64 = "" }, + }, + }, +} +``` + +### 4.3 source 值 + +仅支持两类字符串： + +- `"xlings-res"`：由 xlings 根据资源服务器配置生成 URL。 +- URL template：普通 HTTP(S) URL，可包含占位符。 + +libxpkg 对 `xlings-res` 不硬编码服务器地址；它返回 `SourceKind::XlingsRes`，由 xlings 生成主 URL 和 fallback。 + +## 5. 版本项 + +使用默认 source 且不声明 hash： + +```lua +["1.0.0"] = {} +``` + +单 hash： + +```lua +["1.0.0"] = { sha256 = "" } +``` + +per-arch hash： + +```lua +["1.0.0"] = { + sha256 = { + x86_64 = "", + aarch64 = "", + }, +} +``` + +显式覆盖默认 source： + +```lua +["1.0.1"] = { + url = "https://special.test/foo-1.0.1.tar.gz", + sha256 = "", +} +``` + +版本别名： + +```lua +["latest"] = { ref = "1.0.1" } +``` + +`ref` 支持多跳；目标缺失或形成环时 fail closed。 + +## 6. URL template + +支持： + +| 占位符 | 值 | +|---|---| +| `${name}` | package name | +| `${version}` | 跟随 ref 后的最终版本 | +| `${os}` | `linux` / `macosx` / `windows` | +| `${arch}` | 规范化架构，如 `x86_64` / `aarch64` | +| `${arch_alias}` | 当前版本的架构别名；缺省等于 `${arch}` | +| `${ext}` | Windows 为 `zip`，其他为 `tar.gz`，调用者可覆盖 | + +示例： + +```lua +xpm = { + source = "https://github.com/acme/foo/releases/download/${version}/foo-${version}-${os}-${arch_alias}.${ext}", + + linux = { + ["1.0.0"] = { + arch_alias = { + x86_64 = "amd64", + aarch64 = "arm64", + }, + sha256 = { + x86_64 = "", + aarch64 = "", + }, + }, + }, +} +``` + +template 同样应用于 mirror URL。 + +## 7. 当前支持的资源写法 + +### A. 字符串 URL + +```lua +["1.0.0"] = "https://example.test/foo.tar.gz" +``` + +### B. 旧 XLINGS_RES sentinel + +```lua +["1.0.0"] = "XLINGS_RES" +``` + +继续兼容，但新包推荐使用 `xpm.source = "xlings-res"`。 + +### C. URL + SHA256 + +```lua +["1.0.0"] = { url = "https://example.test/foo.tar.gz", sha256 = "" } +``` + +### D. mirror table + +```lua +["1.0.0"] = { + url = { + GLOBAL = "https://global.test/foo.tar.gz", + CN = "https://cn.test/foo.tar.gz", + }, + sha256 = "", +} +``` + +### E. URL template + per-arch SHA256 + +```lua +["1.0.0"] = { + url = "https://example.test/foo-${version}-${arch_alias}.tar.gz", + arch_alias = { x86_64 = "amd64", aarch64 = "arm64" }, + sha256 = { x86_64 = "", aarch64 = "" }, +} +``` + +### F. per-arch resource map + +```lua +["1.0.0"] = { + x86_64 = { url = "https://example.test/foo-amd64.tar.gz", sha256 = "" }, + aarch64 = { url = "https://example.test/foo-arm64.tar.gz", sha256 = "" }, +} +``` + +### G. `res = true` + +```lua +["1.0.0"] = { + res = true, + sha256 = { x86_64 = "", aarch64 = "" }, +} +``` + +这是已发布的历史 V2 输入，libxpkg 继续兼容；新包不推荐使用，推荐上提为 `xpm.source = "xlings-res"`。 + +## 8. 归一化优先级 + +```text +1. 定位 platform/version。 +2. 跟随 ref 到最终版本，并检测 missing target/cycle。 +3. 若存在 per-arch resource map，严格选择 host arch。 +4. 选择单 hash 或 host arch 对应 hash。 +5. 版本显式 URL 优先。 +6. 平台 source 次之。 +7. 根级 source 最后。 +8. 展开主 URL 和 mirrors 的 template。 +``` + +以下情况 fail closed： + +- 平台或版本不存在。 +- ref 目标不存在或形成环。 +- per-arch resource map 非空但缺少 host arch。 +- per-arch SHA256 表非空但缺少 host arch。 +- 最终既没有 URL 也没有 source。 + +hash 完全缺失仍允许下载，以兼容旧包；这不等于完整性已验证。 + +## 9. 两个最佳范例 + +### 9.1 官方 xlings-res + +```lua +package = { + spec = "2", + name = "mcpp", + archs = { "x86_64", "aarch64" }, + + xpm = { + source = "xlings-res", + + linux = { + ["latest"] = { ref = "0.0.87" }, + ["0.0.87"] = { + sha256 = { + x86_64 = "", + aarch64 = "", + }, + }, + }, + + macosx = { + ["latest"] = { ref = "0.0.87" }, + ["0.0.87"] = { + sha256 = { + x86_64 = "", + aarch64 = "", + }, + }, + }, + + windows = { + ["latest"] = { ref = "0.0.87" }, + ["0.0.87"] = { + sha256 = { x86_64 = "" }, + }, + }, + }, +} +``` + +### 9.2 用户自定义 release + mirror + 特殊版本覆盖 + +```lua +package = { + spec = "2", + name = "foo", + archs = { "x86_64", "aarch64" }, + + xpm = { + source = "https://github.com/acme/foo/releases/download/v${version}/foo-${os}-${arch_alias}.${ext}", + + linux = { + ["latest"] = { ref = "2.1.0" }, + ["2.1.0"] = { + arch_alias = { x86_64 = "amd64", aarch64 = "arm64" }, + sha256 = { x86_64 = "", aarch64 = "" }, + }, + ["2.0.0"] = { + url = { + GLOBAL = "https://legacy.test/foo-2.0.0-linux.tar.gz", + CN = "https://cn.test/foo-2.0.0-linux.tar.gz", + }, + sha256 = "", + }, + }, + + windows = { + source = "https://downloads.acme.test/foo/${version}/foo-${arch_alias}.zip", + ["2.1.0"] = { + arch_alias = { x86_64 = "win64" }, + sha256 = { x86_64 = "" }, + }, + }, + }, +} +``` + +## 10. Loader 平台兼容 + +旧包可能在 Lua 顶层使用： + +```lua +if is_host("linux") then + package.xpm.linux = { ... } +end +``` + +libxpkg 0.0.44 增加 `LoaderContext { platform, arch }`，统一提供： + +- `is_host()` / `is_plat()` +- `is_arch()` +- `os.host()` / `os.arch()` +- `_RUNTIME.platform` / `_RUNTIME.arch` + +xlings 安装阶段通过该上下文加载包，因此删除旧的 `load_platform_entries_()` 不会牺牲这类历史兼容性。 + +## 11. 老客户端与官方索引迁移 + +新增字段对新客户端是可选元数据，但部分旧 libxpkg/xlings loader 会把根级 `source` 当作平台或把平台级 `source` 当作版本。因此不能仅凭“字段是追加的”推断旧客户端安全。 + +官方索引迁移遵循： + +1. 历史 `"XLINGS_RES"` 条目不重写，老客户端继续读取。 +2. 新版本先补权威 SHA256；没有完整 hash 时不伪造。 +3. 实际运行当前稳定旧客户端和修复版解析/安装同一 fixture。 +4. 只有旧客户端不会误解析 `source`，才可在其可见索引轨道加入推荐写法。 +5. 如果旧客户端不兼容，利用版本化 index artifact/pointer 保留旧快照；新语法只进入要求新 xlings 的索引轨道，或暂时继续发布旧 sentinel + hash 结构。 + +索引 artifact/pointer 能提供回滚和旧快照，但不能替代解析兼容测试。 + +## 12. 实现与验证 + +- libxpkg 0.0.43：实现 `source`、compat 归一化和旧资源模型测试。 +- libxpkg 0.0.44：增加平台 loader context 和 per-arch fail-closed。 +- xlings：删除 `load_platform_entries_()`、本地 template 展开器和第二套 Lua sandbox；安装器只调用 libxpkg。 +- tinyhttps 0.2.9：返回实际/期望字节、最终 URL、ETag 和 Last-Modified，供事务缓存 sidecar 使用。 + +最终发布门禁以 issue #356 实施文档的 PR、CI、release、索引和 E2E 台账为准。 diff --git a/src/core/xim/downloader.cppm b/src/core/xim/downloader.cppm index cea57493..5b2bea2f 100644 --- a/src/core/xim/downloader.cppm +++ b/src/core/xim/downloader.cppm @@ -688,6 +688,7 @@ DownloadResult download_one(const DownloadTask& task, urls = mirror::adaptive::reorder(std::move(urls), !task.sha256.empty()); auto stagingFile = unique_sibling_path_(destFile, "part"); + tinyhttps::DownloadFileResult transferResult; // Use in-process tinyhttps for all downloads (streaming progress). // When a CancellationToken is available, wire isCancelled so ESC aborts. @@ -728,14 +729,24 @@ DownloadResult download_one(const DownloadTask& task, }; } - auto dlResult = tinyhttps::download_file(opts); - if (!dlResult.success) { + transferResult = tinyhttps::download_file(opts); + if (!transferResult.success) { fs::remove(stagingFile, ec); - result.error = dlResult.error; + result.error = transferResult.error; return result; } } + if (transferResult.expectedBytes + && transferResult.bytesWritten != *transferResult.expectedBytes) { + result.error = std::format( + "incomplete transfer for {}: wrote {} of {} bytes", + task.name, transferResult.bytesWritten, + *transferResult.expectedBytes); + fs::remove(stagingFile, ec); + return result; + } + // Size sanity check for archives without sha256. When a recipe // omits sha256 we have no way to cross-check content authenticity, // so a CDN serving 200 OK + a tiny error stub (e.g., gitcode's @@ -780,16 +791,29 @@ DownloadResult download_one(const DownloadTask& task, // No sha256 declared: persist server-reported Last-Modified / ETag // alongside the payload so the next install can use a HEAD probe // to decide cache freshness instead of re-downloading. - if (!probedMetaValid) { + tinyhttps::RemoteFileMeta committedMeta { + .ok = transferResult.success, + .contentLength = transferResult.expectedBytes.value_or(-1), + .lastModified = transferResult.lastModified, + .etag = transferResult.etag, + }; + if (committedMeta.lastModified.empty() && committedMeta.etag.empty() + && !transferResult.expectedBytes && !probedMetaValid) { probedMeta = query_remote_meta_(task.url, testHooks); probedMetaValid = true; } + if (committedMeta.lastModified.empty() && committedMeta.etag.empty() + && !transferResult.expectedBytes && probedMetaValid) { + committedMeta = probedMeta; + } std::error_code sizeEc; auto committedSize = static_cast(fs::file_size(destFile, sizeEc)); if (!sizeEc && !task.cacheIdentity.empty()) { write_meta_sidecar_( - sidecarPath, probedMeta, committedSize, - task.cacheIdentity, task.url); + sidecarPath, committedMeta, committedSize, + task.cacheIdentity, + transferResult.finalUrl.empty() + ? task.url : transferResult.finalUrl); } } diff --git a/src/core/xim/installer.cppm b/src/core/xim/installer.cppm index 0f178f02..8e7f7261 100644 --- a/src/core/xim/installer.cppm +++ b/src/core/xim/installer.cppm @@ -2,8 +2,9 @@ export module xlings.core.xim.installer; import std; import mcpplibs.xpkg; +import mcpplibs.xpkg.loader; +import mcpplibs.xpkg.compat; import mcpplibs.xpkg.executor; -import mcpplibs.capi.lua; import xlings.core.xim.libxpkg.types.type; import xlings.core.xim.index; import xlings.core.xim.catalog; @@ -40,8 +41,6 @@ bool evict_invalid_archive_cache_( namespace detail_ { -namespace lua = mcpplibs::capi::lua; - std::string effective_store_name_(std::string_view namespaceName, std::string_view name) { return package_store_name(namespaceName, name); } @@ -180,36 +179,6 @@ std::string build_xlings_res_url_(std::string_view pkgName, return build_xlings_res_url_with_server_(default_res_server_(), pkgName, version, platform); } -// Expand a V2 URL template. Supported placeholders: -// ${name} ${version} ${os} ${arch} ${arch_alias} ${ext} -// `arch` is the canonical host arch (x86_64/aarch64); `arch_alias` maps it -// to an upstream token (falls back to the canonical arch when unmapped). -// `ext` mirrors the XLINGS_RES convention: zip on windows, tar.gz elsewhere. -std::string expand_url_template_(std::string tmpl, - std::string_view name, - std::string_view version, - std::string_view platform, - std::string_view arch, - const std::unordered_map& arch_alias) { - std::string alias{arch}; - if (auto it = arch_alias.find(std::string(arch)); it != arch_alias.end()) - alias = it->second; - std::string ext = (std::string(platform) == "windows") ? "zip" : "tar.gz"; - auto sub = [&](std::string_view key, std::string_view val) { - std::string needle = std::string("${") + std::string(key) + "}"; - for (auto pos = tmpl.find(needle); pos != std::string::npos; - pos = tmpl.find(needle, pos + val.size())) - tmpl.replace(pos, needle.size(), val); - }; - sub("name", name); - sub("version", version); - sub("os", platform); - sub("arch", arch); - sub("arch_alias", alias); - sub("ext", ext); - return tmpl; -} - // Build fallback URLs from all candidate resource servers (excluding the selected one) std::vector build_xlings_res_fallback_urls_(std::string_view pkgName, std::string_view version, @@ -226,6 +195,53 @@ std::vector build_xlings_res_fallback_urls_(std::string_view pkgNam return fallbacks; } +struct DownloadResource_ { + std::string version; + std::string url; + std::string sha256; + std::unordered_map mirrors; + bool useResFallbacks { false }; +}; + +std::expected resolve_download_resource_( + const mcpplibs::xpkg::PlatformMatrix& matrix, + std::string_view name, + std::string_view requestedVersion, + std::string_view platform, + std::string_view arch, + std::string_view preferredMirror) { + auto resolved = mcpplibs::xpkg::resolve_resource(matrix, { + .name = std::string(name), + .version = std::string(requestedVersion), + .platform = std::string(platform), + .arch = std::string(arch), + }); + if (!resolved) return std::unexpected(resolved.error()); + + DownloadResource_ result { + .version = resolved->version, + .url = resolved->url, + .sha256 = resolved->sha256, + .mirrors = resolved->mirrors, + .useResFallbacks = resolved->kind + == mcpplibs::xpkg::SourceKind::XlingsRes, + }; + if (result.useResFallbacks) { + result.url = build_xlings_res_url_(name, result.version, platform); + } + + auto preferred = preferredMirror.empty() + ? std::string_view{"GLOBAL"} + : preferredMirror; + if (auto it = result.mirrors.find(std::string(preferred)); + it != result.mirrors.end()) { + result.url = it->second; + } + if (result.url.empty()) + return std::unexpected("resolved resource URL is empty"); + return result; +} + bool has_directory_entries_(const std::filesystem::path& dir) { std::error_code ec; if (!std::filesystem::exists(dir, ec) || !std::filesystem::is_directory(dir, ec)) return false; @@ -818,224 +834,6 @@ bool run_config_hook_(const PlanNode& node, return true; } -bool register_platform_loader_sandbox_(lua::State* L, const std::string& platform) { - auto quoted = "'" + platform + "'"; - auto script = - "import = function(...) return setmetatable({}, { __index = function() return function() end end }) end\n" - "function is_host(name) return name == " + quoted + " end\n" - "format = string.format\n" - "_RUNTIME = { platform = " + quoted + " }\n" - "os.host = function() return " + quoted + " end\n" - "os.arch = os.arch or function() return 'arm64' end\n" - "os.isfile = os.isfile or function() return false end\n" - "os.isdir = os.isdir or function() return false end\n" - "os.scriptdir = os.scriptdir or function() return '.' end\n" - "os.dirs = os.dirs or function() return {} end\n" - "os.files = os.files or function() return {} end\n" - "os.exists = os.exists or function() return false end\n" - "os.tryrm = os.tryrm or function() end\n" - "os.trymv = os.trymv or function() end\n" - "os.mv = os.mv or function() return true end\n" - "os.cp = os.cp or function() return true end\n" - "os.iorun = os.iorun or function() return nil end\n" - "os.cd = os.cd or function() end\n" - "os.mkdir = os.mkdir or function() end\n" - "os.sleep = os.sleep or function() end\n" - "path = path or {}\n" - "path.join = path.join or function(...) " - " local parts = {} " - " for i = 1, select('#', ...) do " - " local v = select(i, ...) " - " if v ~= nil then parts[#parts+1] = tostring(v) end " - " end " - " return table.concat(parts, '/') " - "end\n" - "path.filename = path.filename or function(p) return type(p)=='string' and (p:match('[^/\\\\]+$') or p) or '' end\n" - "path.directory = path.directory or function(p) return type(p)=='string' and (p:match('(.*)[/\\\\]') or '.') or '.' end\n" - "path.basename = path.basename or function(p) return type(p)=='string' and (p:match('[^/\\\\]+$') or p) or '' end\n" - "io.readfile = io.readfile or function() return '' end\n" - "io.writefile = io.writefile or function() end\n" - "try = try or function(block) pcall(block[1]) end\n" - "cprint = cprint or print\n" - "string.replace = string.replace or function(s, old, new) return s:gsub(old, new) end\n" - "string.split = string.split or function(s, sep) " - " local r = {} " - " for m in (s .. sep):gmatch('(.-)' .. sep) do r[#r+1] = m end " - " return r " - "end\n" - "raise = raise or function() end\n" - "runtime = setmetatable({}, { __index = function() return function() return '' end end })\n" - "system = setmetatable({}, { __index = function() return function() return '' end end })\n" - "libxpkg = setmetatable({}, { __index = function() return setmetatable({}, { __index = function() return function() return '' end end }) end })\n"; - - return lua::L_dostring(L, script.c_str()) == lua::OK; -} - -std::unordered_map -load_platform_entries_(const std::filesystem::path& pkgFile, const std::string& platform) { - std::unordered_map entries; - - auto* L = lua::L_newstate(); - if (!L) return entries; - lua::L_openlibs(L); - - auto closeLua = [&]() { - if (L) { - lua::close(L); - L = nullptr; - } - }; - - if (!register_platform_loader_sandbox_(L, platform)) { - closeLua(); - return entries; - } - if (lua::L_dofile(L, pkgFile.string().c_str()) != lua::OK) { - closeLua(); - return entries; - } - - lua::getglobal(L, "package"); - if (lua::type(L, -1) != lua::TTABLE) { - closeLua(); - return entries; - } - - auto packageIdx = lua::gettop(L); - lua::getfield(L, packageIdx, "xpm"); - if (lua::type(L, -1) != lua::TTABLE) { - closeLua(); - return entries; - } - - auto xpmIdx = lua::gettop(L); - lua::getfield(L, xpmIdx, platform.c_str()); - if (lua::type(L, -1) != lua::TTABLE) { - closeLua(); - return entries; - } - - auto platformIdx = lua::gettop(L); - lua::pushnil(L); - while (lua::next(L, platformIdx)) { - std::string version; - if (lua::type(L, -2) == lua::TSTRING) version = lua::tostring(L, -2); - if (!version.empty() && version != "deps" && version != "inherits") { - mcpplibs::xpkg::PlatformResource res; - if (lua::type(L, -1) == lua::TTABLE) { - auto read_field = [&](const char* key) -> std::string { - lua::getfield(L, -1, key); - std::string val; - if (lua::type(L, -1) == lua::TSTRING) val = lua::tostring(L, -1); - lua::pop(L, 1); - return val; - }; - res.url = read_field("url"); - // Handle url table: { GLOBAL = "...", CN = "..." } - if (res.url.empty()) { - lua::getfield(L, -1, "url"); - if (lua::type(L, -1) == lua::TTABLE) { - lua::pushnil(L); - while (lua::next(L, -2)) { - if (lua::type(L, -2) == lua::TSTRING && lua::type(L, -1) == lua::TSTRING) - res.mirrors[lua::tostring(L, -2)] = lua::tostring(L, -1); - lua::pop(L, 1); - } - if (auto it = res.mirrors.find("GLOBAL"); it != res.mirrors.end()) - res.url = it->second; - else if (!res.mirrors.empty()) - res.url = res.mirrors.begin()->second; - } - lua::pop(L, 1); - } - res.sha256 = read_field("sha256"); - res.ref = read_field("ref"); - - // ---- V2 multi-arch shapes (mirrors libxpkg xpkg-loader) ---- - int resIdx = lua::gettop(L); // resource table (absolute index) - - auto read_str_map = [&](const char* key, - std::unordered_map& out, - bool canon_keys) { - lua::getfield(L, resIdx, key); - if (lua::type(L, -1) == lua::TTABLE) { - lua::pushnil(L); - while (lua::next(L, -2)) { - if (lua::type(L, -2) == lua::TSTRING && lua::type(L, -1) == lua::TSTRING) { - std::string k = lua::tostring(L, -2); - out[canon_keys ? mcpplibs::xpkg::normalize_arch(k) : k] = - lua::tostring(L, -1); - } - lua::pop(L, 1); - } - } - lua::pop(L, 1); - }; - - // Scheme C / res: `sha256` is a per-arch TABLE (string read above - // returned ""). arch_alias is the optional ${arch_alias} mapping. - read_str_map("sha256", res.sha256_by_arch, true); - read_str_map("arch_alias", res.arch_alias, true); - - lua::getfield(L, resIdx, "res"); - res.is_res = lua::toboolean(L, -1); - lua::pop(L, 1); - - // Scheme B: per-arch resource map. Detected when no single-arch - // url/ref/sha256 and no template/res markers are present. - if (res.url.empty() && res.ref.empty() && res.sha256.empty() - && res.sha256_by_arch.empty() && !res.is_res) { - lua::pushnil(L); - while (lua::next(L, resIdx)) { - if (lua::type(L, -2) == lua::TSTRING && lua::type(L, -1) == lua::TTABLE) { - std::string canon = - mcpplibs::xpkg::normalize_arch(lua::tostring(L, -2)); - if (canon == "x86_64" || canon == "aarch64" || canon == "x86") { - int archIdx = lua::gettop(L); - mcpplibs::xpkg::ArchResource ar; - lua::getfield(L, archIdx, "url"); - if (lua::type(L, -1) == lua::TSTRING) ar.url = lua::tostring(L, -1); - lua::pop(L, 1); - if (ar.url.empty()) { - lua::getfield(L, archIdx, "url"); - if (lua::type(L, -1) == lua::TTABLE) { - lua::pushnil(L); - while (lua::next(L, -2)) { - if (lua::type(L, -2) == lua::TSTRING - && lua::type(L, -1) == lua::TSTRING) - ar.mirrors[lua::tostring(L, -2)] = - lua::tostring(L, -1); - lua::pop(L, 1); - } - if (auto it = ar.mirrors.find("GLOBAL"); - it != ar.mirrors.end()) - ar.url = it->second; - else if (!ar.mirrors.empty()) - ar.url = ar.mirrors.begin()->second; - } - lua::pop(L, 1); - } - lua::getfield(L, archIdx, "sha256"); - if (lua::type(L, -1) == lua::TSTRING) ar.sha256 = lua::tostring(L, -1); - lua::pop(L, 1); - res.archs[canon] = std::move(ar); - } - } - lua::pop(L, 1); - } - } - } else if (lua::type(L, -1) == lua::TSTRING) { - res.url = lua::tostring(L, -1); - } - entries[version] = std::move(res); - } - lua::pop(L, 1); - } - - closeLua(); - return entries; -} - } // namespace detail_ using InstallRequestHandler = std::function&)>; @@ -1106,21 +904,12 @@ public: for (auto& node : plan.nodes) { if (node.alreadyInstalled) continue; - std::expected pkg = - catalog_ - ? catalog_->load_package(PackageMatch{ - .rawName = node.rawName, - .name = node.name, - .version = node.version, - .namespaceName = node.namespaceName, - .canonicalName = node.canonicalName, - .repoName = node.repoName, - .pkgFile = node.pkgFile, - .storeRoot = node.storeRoot, - .scope = node.scope, - .installed = node.alreadyInstalled, - }) - : index_->load_package(node.name); + const std::string hostArch = + mcpplibs::xpkg::normalize_arch(detail_::detect_arch_()); + auto pkg = mcpplibs::xpkg::load_package(node.pkgFile, { + .platform = platform, + .arch = hostArch, + }); if (!pkg) { log::warn("skipping {}: {}", node.name, pkg.error()); continue; @@ -1149,103 +938,33 @@ public: } } - auto platformEntries = detail_::load_platform_entries_(node.pkgFile, platform); - if (platformEntries.empty()) { - auto platformIt = pkg->xpm.entries.find(platform); - if (platformIt != pkg->xpm.entries.end()) { - platformEntries = platformIt->second; - } - } - if (platformEntries.empty()) continue; - - std::string version = node.version; - // Follow ref chain - auto verIt = platformEntries.find(version); - if (verIt != platformEntries.end() && !verIt->second.ref.empty()) { - version = verIt->second.ref; - verIt = platformEntries.find(version); - } - - if (verIt == platformEntries.end()) continue; - - auto& res = verIt->second; - - // ---- V2 install-time arch resolution ---- - // Resolve the host arch (canonical x86_64/aarch64). When all V2 - // fields are empty the legacy single-arch path below is untouched. - const std::string hostArch = - mcpplibs::xpkg::normalize_arch(detail_::detect_arch_()); - bool resGenerated = false; - if (!res.archs.empty()) { - // Scheme B: per-arch resource map (fail-closed on miss). - auto ait = res.archs.find(hostArch); - if (ait == res.archs.end()) { - log::warn("skipping {}: no resource for arch '{}' in version {}", - node.name, hostArch, version); - continue; - } - res.url = ait->second.url; - res.sha256 = ait->second.sha256; - res.mirrors = ait->second.mirrors; - } else if (res.is_res) { - // res shape: XLINGS_RES auto-URL + per-arch checksum. - auto sit = res.sha256_by_arch.find(hostArch); - if (sit == res.sha256_by_arch.end()) { - log::warn("skipping {}: no checksum for arch '{}' in version {}", - node.name, hostArch, version); - continue; - } - res.url = detail_::build_xlings_res_url_(node.name, version, platform); - res.sha256 = sit->second; - resGenerated = true; - } else if (!res.sha256_by_arch.empty()) { - // Scheme C: URL template + per-arch checksum. - auto sit = res.sha256_by_arch.find(hostArch); - if (sit == res.sha256_by_arch.end()) { - log::warn("skipping {}: no checksum for arch '{}' in version {}", - node.name, hostArch, version); - continue; - } - res.url = detail_::expand_url_template_( - res.url, node.name, version, platform, hostArch, res.arch_alias); - res.sha256 = sit->second; - } - - bool isXlingsRes = (res.url == "XLINGS_RES"); - if (isXlingsRes) { - res.url = detail_::build_xlings_res_url_(node.name, version, platform); - } - bool useResFallbacks = isXlingsRes || resGenerated; - - // Mirror selection: prefer dlConfig.preferredMirror, fallback others - if (!res.mirrors.empty()) { - auto preferred = dlConfig.preferredMirror.empty() ? "GLOBAL" : dlConfig.preferredMirror; - if (auto it = res.mirrors.find(preferred); it != res.mirrors.end()) { - res.url = it->second; - } + auto resource = detail_::resolve_download_resource_( + pkg->xpm, node.name, node.version, platform, hostArch, + dlConfig.preferredMirror); + if (!resource) { + log::warn("skipping {}: {}", node.name, resource.error()); + continue; } - if (res.url.empty()) continue; - DownloadTask task; task.name = detail_::plan_key_(node); - task.url = res.url; - task.sha256 = res.sha256; + task.url = resource->url; + task.sha256 = resource->sha256; task.cacheIdentity = std::format( "{}/{}/{}/{}/{}", - node.name, version, platform, hostArch, - useResFallbacks ? "xlings-res" : res.url); + node.name, resource->version, platform, hostArch, + resource->useResFallbacks ? "xlings-res" : resource->url); task.destDir = detail_::runtime_dir_(node, dataDir); - if (useResFallbacks) { + if (resource->useResFallbacks) { task.fallbackUrls = detail_::build_xlings_res_fallback_urls_( - node.name, version, platform); + node.name, resource->version, platform); } // Add remaining mirrors as fallbacks - if (!res.mirrors.empty()) { - for (auto& [key, mirrorUrl] : res.mirrors) { - if (mirrorUrl != res.url) { + if (!resource->mirrors.empty()) { + for (auto& [key, mirrorUrl] : resource->mirrors) { + if (mirrorUrl != resource->url) { task.fallbackUrls.push_back(mirrorUrl); } } diff --git a/src/libs/tinyhttps.cppm b/src/libs/tinyhttps.cppm index 49fcbfb3..1bc5c4a6 100644 --- a/src/libs/tinyhttps.cppm +++ b/src/libs/tinyhttps.cppm @@ -11,6 +11,11 @@ export namespace xlings::tinyhttps { struct DownloadFileResult { bool success { false }; std::string error; + std::int64_t bytesWritten { 0 }; + std::optional expectedBytes; + std::string finalUrl; + std::string etag; + std::string lastModified; }; struct DownloadOptions { @@ -284,7 +289,14 @@ DownloadFileResult download_once( auto result = client.download_to_file(url, dest, progress, cancel); if (result.ok() && !stalled) { - return {true, {}}; + return { + .success = true, + .bytesWritten = result.bytesWritten, + .expectedBytes = result.expectedBytes, + .finalUrl = result.finalUrl, + .etag = result.etag, + .lastModified = result.lastModified, + }; } if (stalled) { return {false, std::format( @@ -292,8 +304,16 @@ DownloadFileResult download_once( "(set XLINGS_DOWNLOAD_LOW_SPEED=off to disable the watchdog)", lowSpeedLimitBytes, lowSpeedTimeSec)}; } - return {false, result.error.empty() - ? "HTTP " + std::to_string(result.statusCode) : result.error}; + return { + .success = false, + .error = result.error.empty() + ? "HTTP " + std::to_string(result.statusCode) : result.error, + .bytesWritten = result.bytesWritten, + .expectedBytes = result.expectedBytes, + .finalUrl = result.finalUrl, + .etag = result.etag, + .lastModified = result.lastModified, + }; } } // namespace detail_ diff --git a/tests/unit/test_main.cpp b/tests/unit/test_main.cpp index a257530d..e21bc9ed 100644 --- a/tests/unit/test_main.cpp +++ b/tests/unit/test_main.cpp @@ -1001,6 +1001,70 @@ TEST(XimDownloaderTest, HashRejectedCandidateCommitsFallbackFromStaging) { fs::remove_all(tmp); } +TEST(TinyhttpsWrapperTest, ReturnsAcceptedCandidateTransferMetadata) { + namespace fs = std::filesystem; + auto tmp = fs::temp_directory_path() / "tinyhttps_transfer_metadata"; + fs::remove_all(tmp); + fs::create_directories(tmp); + + xlings::tinyhttps::DownloadOptions options; + options.destFile = tmp / "payload.bin"; + options.urls = {"https://origin.test/payload.bin"}; + options.retryCount = 0; + options.transferOverride = [](const std::string&, const fs::path& path) { + std::ofstream(path) << "payload"; + return xlings::tinyhttps::DownloadFileResult { + .success = true, + .bytesWritten = 7, + .expectedBytes = 7, + .finalUrl = "https://cdn.test/final.bin", + .etag = "etag-1", + .lastModified = "Sat, 12 Jul 2026 00:00:00 GMT", + }; + }; + + auto result = xlings::tinyhttps::download_file(options); + ASSERT_TRUE(result.success) << result.error; + EXPECT_EQ(result.bytesWritten, 7); + ASSERT_TRUE(result.expectedBytes.has_value()); + EXPECT_EQ(*result.expectedBytes, 7); + EXPECT_EQ(result.finalUrl, "https://cdn.test/final.bin"); + EXPECT_EQ(result.etag, "etag-1"); + fs::remove_all(tmp); +} + +TEST(XimDownloaderTest, RejectsIncompleteReportedTransferBeforeCommit) { + namespace fs = std::filesystem; + auto tmp = fs::temp_directory_path() / "xim_download_incomplete_metadata"; + fs::remove_all(tmp); + fs::create_directories(tmp); + auto destination = tmp / "payload.bin"; + std::ofstream(destination) << "previous-good-payload"; + + xlings::xim::DownloadTask task { + .name = "incomplete-test", + .url = "https://origin.test/payload.bin", + .cacheIdentity = "incomplete-test/1/linux/x86_64/url", + .destDir = tmp, + }; + xlings::xim::DownloadTestHooks_ hooks; + hooks.transferOverride = [](const std::string&, const fs::path& path) { + std::ofstream(path) << "bad"; + return xlings::tinyhttps::DownloadFileResult { + .success = true, + .bytesWritten = 3, + .expectedBytes = 100, + }; + }; + + auto result = xlings::xim::download_one(task, nullptr, nullptr, &hooks); + EXPECT_FALSE(result.success); + EXPECT_NE(result.error.find("wrote 3 of 100 bytes"), std::string::npos); + EXPECT_EQ(xlings::platform::read_file_to_string(destination.string()), + "previous-good-payload"); + fs::remove_all(tmp); +} + TEST(XimDownloaderTest, CancelledDownloadPreservesCommittedDestination) { namespace fs = std::filesystem; auto tmp = fs::temp_directory_path() / "xim_download_transaction_cancel"; @@ -1136,6 +1200,56 @@ TEST(XimDownloaderTest, FileLockSerializesIndependentProcesses) { fs::remove_all(tmp); } +TEST(XimInstallerResourceTest, ResolvesXlingsResSourceAndFinalRefVersion) { + mcpplibs::xpkg::PlatformMatrix matrix; + matrix.source = "xlings-res"; + matrix.entries["linux"]["latest"].ref = "1.0.0"; + matrix.entries["linux"]["1.0.0"].sha256_by_arch["x86_64"] = "hash-x86"; + + auto resolved = xlings::xim::detail_::resolve_download_resource_( + matrix, "tool", "latest", "linux", "amd64", "GLOBAL"); + ASSERT_TRUE(resolved.has_value()) << resolved.error(); + EXPECT_EQ(resolved->version, "1.0.0"); + EXPECT_EQ(resolved->sha256, "hash-x86"); + EXPECT_TRUE(resolved->useResFallbacks); + EXPECT_NE(resolved->url.find("/tool/releases/download/1.0.0/"), + std::string::npos); +} + +TEST(XimInstallerResourceTest, ResolvesTemplateAliasAndPreferredMirror) { + mcpplibs::xpkg::PlatformMatrix matrix; + matrix.source = "https://origin.test/${version}/tool-${arch_alias}.${ext}"; + auto& resource = matrix.entries["linux"]["2.0.0"]; + resource.sha256_by_arch["x86_64"] = "hash-x86"; + resource.arch_alias["x86_64"] = "amd64"; + resource.mirrors["CN"] = "https://cn.test/${version}/tool-${arch_alias}.tar.gz"; + + auto resolved = xlings::xim::detail_::resolve_download_resource_( + matrix, "tool", "2.0.0", "linux", "x86_64", "CN"); + ASSERT_TRUE(resolved.has_value()) << resolved.error(); + EXPECT_EQ(resolved->url, "https://cn.test/2.0.0/tool-amd64.tar.gz"); + EXPECT_EQ(resolved->sha256, "hash-x86"); + EXPECT_FALSE(resolved->useResFallbacks); +} + +TEST(XimInstallerResourceTest, PreservesLegacyXlingsResAndFailsClosedOnArchMiss) { + mcpplibs::xpkg::PlatformMatrix legacy; + legacy.entries["linux"]["1.0.0"].url = "XLINGS_RES"; + auto resolved = xlings::xim::detail_::resolve_download_resource_( + legacy, "legacy", "1.0.0", "linux", "x86_64", "GLOBAL"); + ASSERT_TRUE(resolved.has_value()) << resolved.error(); + EXPECT_TRUE(resolved->useResFallbacks); + + mcpplibs::xpkg::PlatformMatrix per_arch; + per_arch.entries["linux"]["1.0.0"].archs["x86_64"] = { + .url = "https://example.test/x86.tar.gz", + .sha256 = "hash-x86", + }; + auto missing = xlings::xim::detail_::resolve_download_resource_( + per_arch, "tool", "1.0.0", "linux", "aarch64", "GLOBAL"); + EXPECT_FALSE(missing.has_value()); +} + TEST(XimDownloaderTest, RecoveryRestoresBackupWhenLiveIsMissing) { namespace fs = std::filesystem; auto tmp = fs::temp_directory_path() / "xim_download_recover_backup"; From ef00abcecaeef7fa85e50cd068e32f7cfa41e054 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sun, 12 Jul 2026 02:39:22 +0800 Subject: [PATCH 07/10] docs(build): clarify xmake is not a runtime dependency --- .agents/skills/xlings-build/SKILL.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/.agents/skills/xlings-build/SKILL.md b/.agents/skills/xlings-build/SKILL.md index 7c7e4a29..a4cc9a26 100644 --- a/.agents/skills/xlings-build/SKILL.md +++ b/.agents/skills/xlings-build/SKILL.md @@ -95,7 +95,7 @@ If using the packaged release tree, prefer `./bin/xlings -h` inside the assemble ## macOS Build -The checked-in CI uses Homebrew `llvm@20` plus `xmake`. If the user insists on an xlings-first workflow, install xlings first for consistency, but the canonical repo reference is still the CI setup. +The checked-in CI uses the toolchain declared by `mcpp.toml` and builds the C++ project with `mcpp`. Installing xlings first only provides the project-declared development tools; xlings does not implicitly install xmake. ### 1) Install xlings @@ -197,7 +197,7 @@ If network access is not available, say that the smoke test was skipped rather t - On Linux, `musl-gcc@15.1.0` is required for `import std` in the C++23 build. - `mcpp.toml` is the only C++ project definition. -- Prefer release scripts for packaging because they also assemble `xim`, `xvm`, config files, bundled `xmake`, and post-build verification. +- Prefer release scripts for packaging because they also assemble the xlings binary, `xvm`, config files, package-index data, and post-build verification. The release does not bundle xmake as an internal runtime dependency. - When the user asks for exact CI parity, read the workflow file first and mirror it exactly rather than relying on memory. ## References From 0dc28e496a76c3bde9063406567b68f9cdff4bce Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sun, 12 Jul 2026 01:58:32 +0800 Subject: [PATCH 08/10] fix(ci): avoid recursive mcpp bootstrap pin --- .github/workflows/xlings-ci-aarch64.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/.github/workflows/xlings-ci-aarch64.yml b/.github/workflows/xlings-ci-aarch64.yml index be1eec4a..0bd4085d 100644 --- a/.github/workflows/xlings-ci-aarch64.yml +++ b/.github/workflows/xlings-ci-aarch64.yml @@ -71,6 +71,10 @@ jobs: git clone --depth 1 https://github.com/mcpp-community/mcpp /tmp/mcpp-src cd /tmp/mcpp-src mcpp self config --mirror GLOBAL 2>/dev/null || true + # We are bootstrapping mcpp itself with the already-installed mcpp. + # Do not let the cloned project's .xlings.json recursively request + # an older mcpp version that may have left the rolling package index. + mv .xlings.json .xlings.json.bootstrap mcpp build MCPP=$(realpath "$(find target -type f -name mcpp -printf '%T@ %p\n' | sort -rn | head -1 | cut -d' ' -f2)") test -x "$MCPP" From 464aac81fe6f2cf3ac1a5c34a70451f321e072f6 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sun, 12 Jul 2026 02:18:32 +0800 Subject: [PATCH 09/10] fix(ci): reinitialize mcpp registry consistently --- .github/workflows/release.yml | 28 +++++++++++++--------- .github/workflows/xlings-ci-aarch64.yml | 4 ++-- .github/workflows/xlings-ci-linux-e2e.yml | 8 ++++--- .github/workflows/xlings-ci-linux-root.yml | 8 ++++--- .github/workflows/xlings-ci-linux.yml | 8 ++++--- .github/workflows/xlings-ci-macos.yml | 8 ++++--- .github/workflows/xlings-ci-windows.yml | 8 ++++--- 7 files changed, 44 insertions(+), 28 deletions(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 7fb722db..70196663 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -67,14 +67,16 @@ jobs: - name: Install build dependencies (mcpp) run: | cd "$GITHUB_WORKSPACE" + xlings config --mirror GLOBAL xlings install -y mcpp --version mcpp self version - - name: Refresh mcpp package index cache + - name: Reinitialize mcpp registry (GLOBAL) run: | - rm -rf "$HOME/.mcpp/registry/data/mcpplibs" - rm -rf "$GITHUB_WORKSPACE/.mcpp/registry/data/mcpplibs" + rm -rf "$HOME/.mcpp/registry" + rm -rf "$GITHUB_WORKSPACE/.mcpp/registry" + mcpp self config --mirror GLOBAL - name: Prepare fixture index repo run: | @@ -129,7 +131,7 @@ jobs: - name: Bootstrap mcpp + refresh index (latest, GLOBAL) run: | - xlings config --mirror GLOBAL 2>/dev/null || true + xlings config --mirror GLOBAL xlings update -y 2>/dev/null || xlings update 2>/dev/null || true xlings install mcpp -y mcpp --version @@ -138,7 +140,7 @@ jobs: run: | git clone --depth 1 https://github.com/mcpp-community/mcpp /tmp/mcpp-src cd /tmp/mcpp-src - mcpp self config --mirror GLOBAL 2>/dev/null || true + mcpp self config --mirror GLOBAL mcpp build MCPP=$(realpath "$(find target -type f -name mcpp -printf '%T@ %p\n' | sort -rn | head -1 | cut -d' ' -f2)") test -x "$MCPP" @@ -200,14 +202,16 @@ jobs: - name: Install build dependencies (mcpp) run: | cd "$GITHUB_WORKSPACE" + xlings config --mirror GLOBAL xlings install -y mcpp --version mcpp self version - - name: Refresh mcpp package index cache + - name: Reinitialize mcpp registry (GLOBAL) run: | - rm -rf "$HOME/.mcpp/registry/data/mcpplibs" - rm -rf "$GITHUB_WORKSPACE/.mcpp/registry/data/mcpplibs" + rm -rf "$HOME/.mcpp/registry" + rm -rf "$GITHUB_WORKSPACE/.mcpp/registry" + mcpp self config --mirror GLOBAL - name: Build (macos_release) env: @@ -279,15 +283,17 @@ jobs: shell: pwsh run: | Set-Location $env:GITHUB_WORKSPACE + xlings config --mirror GLOBAL xlings install -y mcpp --version mcpp self version - - name: Refresh mcpp package index cache + - name: Reinitialize mcpp registry (GLOBAL) shell: pwsh run: | - Remove-Item -Recurse -Force "$env:USERPROFILE\.mcpp\registry\data\mcpplibs" -ErrorAction SilentlyContinue - Remove-Item -Recurse -Force "$env:GITHUB_WORKSPACE\.mcpp\registry\data\mcpplibs" -ErrorAction SilentlyContinue + Remove-Item -Recurse -Force "$env:USERPROFILE\.mcpp\registry" -ErrorAction SilentlyContinue + Remove-Item -Recurse -Force "$env:GITHUB_WORKSPACE\.mcpp\registry" -ErrorAction SilentlyContinue + mcpp self config --mirror GLOBAL - name: Build (windows_release) shell: pwsh diff --git a/.github/workflows/xlings-ci-aarch64.yml b/.github/workflows/xlings-ci-aarch64.yml index 0bd4085d..03e75aa3 100644 --- a/.github/workflows/xlings-ci-aarch64.yml +++ b/.github/workflows/xlings-ci-aarch64.yml @@ -61,7 +61,7 @@ jobs: - name: Bootstrap mcpp + GLOBAL mirror run: | - xlings config --mirror GLOBAL 2>/dev/null || true + xlings config --mirror GLOBAL xlings update -y 2>/dev/null || xlings update 2>/dev/null || true xlings install mcpp -y mcpp --version @@ -70,7 +70,7 @@ jobs: run: | git clone --depth 1 https://github.com/mcpp-community/mcpp /tmp/mcpp-src cd /tmp/mcpp-src - mcpp self config --mirror GLOBAL 2>/dev/null || true + mcpp self config --mirror GLOBAL # We are bootstrapping mcpp itself with the already-installed mcpp. # Do not let the cloned project's .xlings.json recursively request # an older mcpp version that may have left the rolling package index. diff --git a/.github/workflows/xlings-ci-linux-e2e.yml b/.github/workflows/xlings-ci-linux-e2e.yml index 0b11222c..43566e62 100644 --- a/.github/workflows/xlings-ci-linux-e2e.yml +++ b/.github/workflows/xlings-ci-linux-e2e.yml @@ -72,14 +72,16 @@ jobs: - name: Install build dependencies (mcpp) run: | cd "$GITHUB_WORKSPACE" + xlings config --mirror GLOBAL xlings install -y mcpp --version mcpp self version - - name: Refresh mcpp package index cache + - name: Reinitialize mcpp registry (GLOBAL) run: | - rm -rf "$HOME/.mcpp/registry/data/mcpplibs" - rm -rf "$GITHUB_WORKSPACE/.mcpp/registry/data/mcpplibs" + rm -rf "$HOME/.mcpp/registry" + rm -rf "$GITHUB_WORKSPACE/.mcpp/registry" + mcpp self config --mirror GLOBAL - name: Prepare fixture index repo run: | diff --git a/.github/workflows/xlings-ci-linux-root.yml b/.github/workflows/xlings-ci-linux-root.yml index 7b348375..5455077c 100644 --- a/.github/workflows/xlings-ci-linux-root.yml +++ b/.github/workflows/xlings-ci-linux-root.yml @@ -69,13 +69,15 @@ jobs: - name: Install build dependencies (mcpp) run: | cd "$GITHUB_WORKSPACE" + xlings config --mirror GLOBAL xlings install -y mcpp --version - - name: Refresh mcpp package index cache + - name: Reinitialize mcpp registry (GLOBAL) run: | - rm -rf "$HOME/.mcpp/registry/data/mcpplibs" - rm -rf "$GITHUB_WORKSPACE/.mcpp/registry/data/mcpplibs" + rm -rf "$HOME/.mcpp/registry" + rm -rf "$GITHUB_WORKSPACE/.mcpp/registry" + mcpp self config --mirror GLOBAL - name: Prepare unit fixture index repo run: | diff --git a/.github/workflows/xlings-ci-linux.yml b/.github/workflows/xlings-ci-linux.yml index 89948b97..7ff66532 100644 --- a/.github/workflows/xlings-ci-linux.yml +++ b/.github/workflows/xlings-ci-linux.yml @@ -68,14 +68,16 @@ jobs: - name: Install build dependencies (mcpp) run: | cd "$GITHUB_WORKSPACE" + xlings config --mirror GLOBAL xlings install -y mcpp --version mcpp self version - - name: Refresh mcpp package index cache + - name: Reinitialize mcpp registry (GLOBAL) run: | - rm -rf "$HOME/.mcpp/registry/data/mcpplibs" - rm -rf "$GITHUB_WORKSPACE/.mcpp/registry/data/mcpplibs" + rm -rf "$HOME/.mcpp/registry" + rm -rf "$GITHUB_WORKSPACE/.mcpp/registry" + mcpp self config --mirror GLOBAL - name: Prepare unit fixture index repo run: | diff --git a/.github/workflows/xlings-ci-macos.yml b/.github/workflows/xlings-ci-macos.yml index d804305a..fc389e66 100644 --- a/.github/workflows/xlings-ci-macos.yml +++ b/.github/workflows/xlings-ci-macos.yml @@ -64,14 +64,16 @@ jobs: - name: Install build dependencies (mcpp) run: | cd "$GITHUB_WORKSPACE" + xlings config --mirror GLOBAL xlings install -y mcpp --version mcpp self version - - name: Refresh mcpp package index cache + - name: Reinitialize mcpp registry (GLOBAL) run: | - rm -rf "$HOME/.mcpp/registry/data/mcpplibs" - rm -rf "$GITHUB_WORKSPACE/.mcpp/registry/data/mcpplibs" + rm -rf "$HOME/.mcpp/registry" + rm -rf "$GITHUB_WORKSPACE/.mcpp/registry" + mcpp self config --mirror GLOBAL - name: Prepare unit fixture index repo run: | diff --git a/.github/workflows/xlings-ci-windows.yml b/.github/workflows/xlings-ci-windows.yml index 1aa572a5..3216552f 100644 --- a/.github/workflows/xlings-ci-windows.yml +++ b/.github/workflows/xlings-ci-windows.yml @@ -72,15 +72,17 @@ jobs: shell: pwsh run: | Set-Location $env:GITHUB_WORKSPACE + xlings config --mirror GLOBAL xlings install -y mcpp --version mcpp self version - - name: Refresh mcpp package index cache + - name: Reinitialize mcpp registry (GLOBAL) shell: pwsh run: | - Remove-Item -Recurse -Force "$env:USERPROFILE\.mcpp\registry\data\mcpplibs" -ErrorAction SilentlyContinue - Remove-Item -Recurse -Force "$env:GITHUB_WORKSPACE\.mcpp\registry\data\mcpplibs" -ErrorAction SilentlyContinue + Remove-Item -Recurse -Force "$env:USERPROFILE\.mcpp\registry" -ErrorAction SilentlyContinue + Remove-Item -Recurse -Force "$env:GITHUB_WORKSPACE\.mcpp\registry" -ErrorAction SilentlyContinue + mcpp self config --mirror GLOBAL - name: Prepare unit fixture index repo shell: bash From 73ff6be5b74fb5a83c48325bb39316b3485fff11 Mon Sep 17 00:00:00 2001 From: sunrisepeak Date: Sun, 12 Jul 2026 02:23:34 +0800 Subject: [PATCH 10/10] fix(ci): disable project pin before mcpp bootstrap --- .github/workflows/release.yml | 3 +++ .github/workflows/xlings-ci-aarch64.yml | 2 +- 2 files changed, 4 insertions(+), 1 deletion(-) diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 70196663..3c1fed76 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -140,6 +140,9 @@ jobs: run: | git clone --depth 1 https://github.com/mcpp-community/mcpp /tmp/mcpp-src cd /tmp/mcpp-src + # Avoid recursively resolving mcpp's development pin while the + # already-installed mcpp bootstraps the source tree. + mv .xlings.json .xlings.json.bootstrap mcpp self config --mirror GLOBAL mcpp build MCPP=$(realpath "$(find target -type f -name mcpp -printf '%T@ %p\n' | sort -rn | head -1 | cut -d' ' -f2)") diff --git a/.github/workflows/xlings-ci-aarch64.yml b/.github/workflows/xlings-ci-aarch64.yml index 03e75aa3..1916443a 100644 --- a/.github/workflows/xlings-ci-aarch64.yml +++ b/.github/workflows/xlings-ci-aarch64.yml @@ -70,11 +70,11 @@ jobs: run: | git clone --depth 1 https://github.com/mcpp-community/mcpp /tmp/mcpp-src cd /tmp/mcpp-src - mcpp self config --mirror GLOBAL # We are bootstrapping mcpp itself with the already-installed mcpp. # Do not let the cloned project's .xlings.json recursively request # an older mcpp version that may have left the rolling package index. mv .xlings.json .xlings.json.bootstrap + mcpp self config --mirror GLOBAL mcpp build MCPP=$(realpath "$(find target -type f -name mcpp -printf '%T@ %p\n' | sort -rn | head -1 | cut -d' ' -f2)") test -x "$MCPP"