diff --git a/.agents/docs/2026-07-11-issue-356-partial-download-cache-fix-plan.md b/.agents/docs/2026-07-11-issue-356-partial-download-cache-fix-plan.md new file mode 100644 index 00000000..2b3897bf --- /dev/null +++ b/.agents/docs/2026-07-11-issue-356-partial-download-cache-fix-plan.md @@ -0,0 +1,419 @@ +# Issue #356:无 SHA256 下载残片进入缓存的根因与修复方案 + +> 日期:2026-07-11 +> 状态:实施中(动态更新) +> 关联 Issue:[openxlings/xlings#356](https://github.com/openxlings/xlings/issues/356) +> 分析基线:`openxlings/xlings@37fedb7`、`openxlings/xim-pkgindex@b894897` + +## 1. 结论 + +Issue #356 成立。最佳修复不是单独增加一个大小检查,也不是只给 `mcpp` 补 SHA256,而是以下三层组合: + +1. **立即止血:给官方 `XLINGS_RES` 资源补 SHA256。** 先迁移 `mcpp` 的 0.0.81、`latest` 当前指向版本与其他活跃版本,再逐步回填可验证版本;索引形态必须先通过旧客户端兼容矩阵,不能未经验证直接发布 `res=true` 条目。 +2. **根因修复:下载到唯一临时文件,验收成功后原子提交。** 网络传输、进程中断、候选镜像校验失败期间都不能写最终缓存路径。 +3. **缓存策略修复:删除 HEAD 失败时“任意非空文件即命中”的规则。** 无 SHA256 的离线复用只接受由新下载事务写出的完整性 sidecar,且记录大小必须与当前文件一致;旧缓存没有完整提交证据时不能命中。 + +同时应在 `mcpplibs/tinyhttps` 修正 chunked 响应提前 EOF 被误判为正常结束的问题,并将响应的实际字节数、期望字节数和最终来源 URL 返回给 xlings。这样 Content-Length 校验属于传输层,缓存提交属于 xlings,制品哈希属于索引/发布链,责任边界清晰。 + +不建议把“libarchive 能打开并读到第一个 header”作为主要验收条件。截断通常发生在归档后半段,首 header 可正常读取;完整遍历则会让大型工具链在真正解压前额外做一次完整解压读取,成本过高。 + +## 2. 事实与影响范围 + +### 2.1 Issue 场景的可验证事实 + +- 在 `xim-pkgindex@b894897` 快照中,`mcpp@0.0.81` 使用裸 `"XLINGS_RES"`,解析出的 `DownloadTask::sha256` 为空。下列数量同样是该提交的审计快照,不代表实施时仓库 HEAD。 +- 0.0.81 的上游 release 已提供每个平台的 `.sha256` 文件,`tools/mirror_res.sh` 也已镜像这些 sidecar,校验数据并不缺失,只是没有进入索引条目。 +- Linux x86_64 制品的权威大小为 `12,628,937` 字节,SHA256 为 `47c41529a00930ad701a76bb53e0847220c0764eb1f8e6cf6d515c45fea8cfcc`。从 `xlings-res/mcpp` 下载的镜像制品与上游 sidecar 一致。 +- Issue 中“正常包约 30 MB+”是估计值,不准确;114 KB 文件仍显著小于权威大小,不影响缺陷成立。 +- 当前 `xim-pkgindex` 有 26 个包文件包含裸 `XLINGS_RES`,共 336 个裸条目;其中 `mcpp` 占 208 个,`xlings` 占 60 个。当前没有包使用已经被 xlings 支持的 `res=true + sha256-by-arch` 形态。 + +### 2.2 当前代码路径 + +```mermaid +flowchart LR + A[索引裸 XLINGS_RES] --> B[DownloadTask.sha256 为空] + B --> C[tinyhttps 直接写最终 destFile] + C -->|进程被杀/异常退出| D[最终路径残留半文件] + D --> E[下次安装执行 HEAD] + E -->|HEAD 失败| F[只检查 localSize > 0] + F --> G[误报缓存命中] + G --> H[libarchive 解压] + H --> I[Truncated tar archive] +``` + +关键位置: + +- `src/core/xim/installer.cppm:1201-1225`:裸 `XLINGS_RES` 生成 URL,但没有生成校验值。 +- `src/core/xim/downloader.cppm:362-409`:无 SHA256 时执行 HEAD 缓存判断;HEAD 失败后任意非空文件都会成功返回。 +- `src/core/xim/downloader.cppm:439-478`:`tinyhttps` 直接以最终 `destFile` 为下载目标。 +- `src/core/xim/downloader.cppm:481-500`:1 KiB 下限只能拒绝极小错误页,无法识别 114 KB 或更大的截断文件。 +- `src/core/xim/downloader.cppm:502-523`:只有声明 SHA256 时才做强校验;无 SHA256 sidecar 不记录“完整提交”或实际文件大小。 +- `tests/unit/test_main.cpp:832-870`:现有测试只验证 sidecar 的读写,没有验证 HEAD-fail 缓存准入策略。 + +### 2.3 Issue 根因描述需要补充的两点 + +第一,当前依赖的 `mcpplibs-tinyhttps` 0.2.0 在响应声明 `Content-Length > 0` 时,已经使用 `read_exact` 读取全部字节;提前断线会返回 `Read error`,xlings 外层随后删除目标文件。因此“普通断线一定留下并接受半文件”并不完整。稳定入口是: + +- 进程在外层清理前被强制终止; +- 机器掉电或进程崩溃; +- 旧版本/其他路径已经留下最终路径半文件; +- chunked 响应在终止 chunk 到来前断开。 + +第二,`mcpplibs-tinyhttps` 0.2.0 到 0.2.3 的 chunked 读取逻辑相同:空的 chunk-size 行会被 `parse_hex` 当作 0,随后按合法终止 chunk 返回成功。这是独立的传输完整性缺陷,必须在上游修复;单靠 xlings 的 HEAD 检查无法可靠补偿。 + +## 3. 方案比较 + +| 方案 | 优点 | 缺点 | 结论 | +|---|---|---|---| +| A. 只给 mcpp/XLINGS_RES 补 SHA256 | 最快;现有 hash cache path 可直接识别并删除 114 KB 文件;抵御镜像内容漂移 | 无法保护第三方无 SHA256 包;下载中断仍直接污染最终路径;发布机器人仍可能生成新裸条目 | 作为 P0 止血,不是完整修复 | +| B. 只增加 Content-Length/归档检查 | 能覆盖部分无哈希下载;改动较局部 | Content-Length 路径底层已经严格读取;chunked/connection-close 没有可靠长度;首 header 不能发现后半段截断;完整归档扫描代价高 | 不推荐作为主方案 | +| C. SHA256 迁移 + 事务下载 + 严格缓存准入 | 同时阻止新残片、修复旧缓存误命中、保留可证明的离线复用;适用于官方和第三方包 | 涉及 tinyhttps、xlings、xim-pkgindex/发布链三个仓库,需要分阶段落地 | **推荐** | + +## 4. 推荐设计 + +### 4.1 传输层:明确“完整响应”的判定 + +在 `mcpplibs/tinyhttps` 中: + +1. `Content-Length` 响应继续要求 `bytesWritten == contentLength`,并把这两个值返回给调用方。 +2. chunked 响应只有读到语法合法的 `0\r\n` 终止 chunk 和尾部结束行才算成功;空 size line、非法十六进制、缺少 chunk 尾部 CRLF、提前 EOF 都返回错误。 +3. connection-close 响应在 HTTP 语义上没有可比较的期望长度,只能报告 `expectedBytes = null`,不能伪造完整性保证。 +4. `DownloadToFileResult`/xlings 包装结果至少携带: + - `bytesWritten` + - `expectedBytes`(可选) + - `finalUrl` 或被接受的候选 URL + - 明确的传输错误类型 + +这层只判断 HTTP 消息是否完整,不判断制品是否可信。 + +### 4.2 xlings 下载层:临时文件验收后原子提交 + +`download_one` 不再把 `destFile` 直接交给 tinyhttps: + +1. 在缓存判断前取得 per-destination 跨进程锁,并持有到提交/失败清理完成,避免两个 xlings 进程同时判定 miss、删除或替换同一目标。锁实现放入 platform 层,Linux/macOS 使用 advisory lock,Windows 使用独占文件句柄。 +2. 为每次下载生成同目录、唯一的 `stagingFile`,例如 `.part.`。同目录保证最终 `rename` 不跨文件系统。 +3. tinyhttps 的所有候选 URL 都写 `stagingFile`。失败、取消或校验拒绝时只删除临时文件,不触碰已有的已提交缓存。 +4. `onVerify` 对 `stagingFile` 计算 SHA256,不能继续捕获最终 `destFile`。 +5. 候选传输成功后依次执行: + - 若有期望长度,检查实际长度一致; + - 若有 SHA256,检查 hash; + - 若无 SHA256 且是归档,仅保留现有的极小错误页检查作为快速诊断,不把它当完整性证明。 +6. 所有验收通过后,在锁内执行可恢复提交:已有无效目标先改名为同目录 backup,`stagingFile` 再 rename 为 `destFile`;第二步失败则恢复 backup,成功后删除 backup。该流程在两次 rename 之间最终路径短暂不存在,因此不能称为全过程原子替换。 +7. 锁取得后先执行恢复状态机:只有 backup 时恢复为 live;live 与 backup 同时存在时保留 live 并删除已确认过期的 backup;live、backup、staging 的其他组合 fail closed 并记录诊断。若把机器掉电纳入保证,还需分别定义 POSIX `fsync`/目录同步和 Windows `FlushFileBuffers` 的持久化顺序。 +8. 提交最终文件后,原子写入 sidecar。若在“文件已提交、sidecar 未写”之间崩溃,结果只是下次保守重下,不会误接收半文件。 +9. 只有持有目标锁时才能清理该目标的过期 `.part.*`/backup;不要删除其他进程可能仍在写的临时文件,也不要把临时文件当缓存候选。 + +临时文件名必须支持并发 xlings 进程,不能使用固定 `.part`。唯一 ID 可由进程 ID、线程 ID、单调计数器和随机数组合;不需要暴露为公共接口。锁等待必须响应取消并有超时,超时错误应指出被占用的缓存目标。 + +### 4.3 sidecar v2:记录“已完整提交”而不假装是密码学证明 + +现有 sidecar 只有 `Last-Modified` 和 `ETag`。升级为带版本的记录: + +```text +format: 2 +complete: true +size: 12628937 +source-url: +cache-identity: +last-modified: +etag: +``` + +含义: + +- `complete=true` 只表示该文件经过新事务流程成功提交,不等价于可信 hash。 +- `size` 是提交时实际大小,用于发现后续截断、覆盖或磁盘损坏。 +- `source-url` 记录真正成功的候选。当前代码下载可能从 fallback 成功,却只 HEAD 主 URL;新设计应优先探测实际来源。 +- `cache-identity` 必须绑定包、解析后的版本、平台、架构以及规范化主 URL 或官方资源键。HEAD 失败离线复用时必须匹配,防止不同任务因文件名和大小相同而误用彼此缓存。 +- `source-url`、`size`、ETag、Last-Modified 应直接来自成功 GET 的结果,避免提交后再 HEAD 带来的额外失败和 TOCTOU。 +- sidecar 自身用临时文件 + rename 写入,避免半行记录。 +- 老 sidecar 不含 `format/complete/size`,按 legacy 处理,不能获得 HEAD-fail 离线准入资格。 + +### 4.4 缓存准入矩阵 + +| 条件 | 处理 | +|---|---| +| 声明 SHA256,缓存 hash 匹配 | 命中 | +| 声明 SHA256,缓存 hash 不匹配 | 删除文件与 sidecar,重新下载 | +| 无 SHA256,HEAD 成功且远端长度与本地一致,ETag/Last-Modified 与 sidecar 一致 | 命中 | +| 无 SHA256,HEAD 成功且无远端新鲜度字段,但长度一致、sidecar v2 完整且记录大小一致 | 可命中,记录 weak-cache 日志 | +| 无 SHA256,HEAD 失败,sidecar v2 完整且记录大小与本地一致 | 允许离线命中,并明确警告“非密码学验证” | +| 无 SHA256,HEAD 失败,只有 legacy sidecar 或任意非空文件 | **不得命中**;尝试 GET,GET 也失败则明确报错 | +| 存在 `.part.*`,最终文件不存在 | 清理或忽略 `.part.*`,重新下载 | + +该矩阵保留了 0.4.15 引入的 airline-friendly 目标,但把“存在文件”升级为“存在新事务完整提交证据”。 + +### 4.5 解压失败后的自愈 + +事务下载不能证明无 SHA256、无长度的 connection-close 内容在业务上完整。对归档增加一层反应式自愈: + +- `extract_archive` 将错误至少区分为输入归档错误和本地写入错误。 +- `open`、`next_header`、`read_data_block` 的格式/截断错误视为输入归档错误;删除对应缓存和 sidecar,使下一次安装必然重新下载。 +- `write_header`、`write_data_block`、磁盘空间、权限错误属于本地环境错误,不删除可能完全有效的大型缓存。 +- 本轮不做“下载后完整预扫描归档”,避免 LLVM/GCC 等大包在解压前额外读取和解压一次。 + +### 4.6 官方资源层:让 SHA256 成为默认契约 + +当前工作树的自定义解析路径支持以下索引形态,但旧客户端兼容性尚未证明: + +```lua +["0.0.81"] = { + res = true, + sha256 = { + x86_64 = "", + aarch64 = "", + }, +}, +``` + +因此 mcpp 止血不需要等待新的 xpkg schema,但发布到全局索引前必须选择经旧客户端验证的表达。优先测试以下双兼容候选: + +~~~lua +["0.0.81"] = { + url = "XLINGS_RES", + sha256 = { + x86_64 = "", + aarch64 = "", + }, +}, +~~~ + +新客户端读取架构 hash 后继续把 `url` sentinel 展开为官方资源 URL;旧客户端仍有机会按原 `XLINGS_RES` 路径安装。是否成立必须用真实旧版本验证,不能仅靠静态推断。迁移建议: + +1. 先给 `mcpp` 的 0.0.81、`latest` 解析后的目标版本及其他活跃镜像版本补 hash;`latest` 本身继续作为 `ref`,验收其最终 `DownloadTask::sha256` 非空。 +2. `version-check.py` 的 `res_versioned` apply 路径不再写裸 sentinel。它应读取发布产物的 sidecar/manifest,生成带校验的 res 条目;缺任一受支持平台/架构的 hash 时 fail closed,不开不完整 PR。 +3. `mirror_res.sh` 在上传后不仅检查 HTTP 200,还要下载或流式计算每个镜像制品的 SHA256,与上游值比较。 +4. xlings 自身 release 当前没有 `.sha256` 资产。发布链应为每个归档生成 sidecar,或统一发布机器可读 manifest,再允许机器人生成带 hash 的 res 条目。 +5. 审计其余 24 个含裸 `XLINGS_RES` 的配方,按“能取得权威 hash → 迁移;不能取得 → 保留兼容但走严格事务缓存”的顺序处理。不要让全量历史回填阻塞 mcpp 止血。 + +长期建议统一发布一个 manifest,字段至少为 `{version, platform, arch, filename, size, sha256}`。sidecar 可继续供人和简单脚本使用,索引机器人消费 manifest,避免在脚本中硬编码平台文件名。 + +## 5. 实施顺序 + +### P0:当天止血(xim-pkgindex / 发布脚本) + +- 迁移 `mcpp@0.0.81` 和当前 `latest` 的三平台条目,使用上游已发布的 `.sha256`。 +- 修改 mcpp 的 index bump 流程,使新版本不再生成裸 `XLINGS_RES`。 +- 验证 GLOBAL 与 CN 镜像的 hash 都等于索引值。 + +这一阶段上线后,现有 114 KB 缓存会在 SHA cache path 被识别为 mismatch、删除并重新下载。 + +### P1:通用根因修复(mcpplibs/tinyhttps + xlings) + +- tinyhttps 修正 chunked EOF 判定并扩充结果元数据。 +- xlings 升级 tinyhttps 依赖。 +- `download_one` 改为 staging → verify → atomic rename。 +- sidecar 升级为 v2,删除无条件非空缓存 fallback。 + +### P2:自愈与生态收口 + +- 结构化区分归档输入错误与本地写入错误,输入错误时驱逐缓存。 +- 为 xlings/其他官方 res 发布生成 checksum manifest。 +- 分批迁移其余官方 `XLINGS_RES` 条目,并在索引 CI 中禁止新增无 SHA256 的官方二进制归档。 + +## 6. 测试计划 + +### 6.1 mcpplibs/tinyhttps + +- `Content-Length=N`,服务端只发送 ` X2 -> X3 -> X4 ---------+--> xlings release -> index 更新 -> 生态 E2E +T1 -> tinyhttps release ------+ +L1 -> libxpkg release -> L2 --+ +I1 ---------------------------+ +I2 依赖 L1/L2 的兼容矩阵通过,但不阻塞 I1 的 mcpp 止血 +~~~ + +- `X1` 与 `T1`、`L1`、`I1` 可并行;同一仓库内部保持顺序,避免把根因修复、schema 和发布脚本揉成一个不可审查 PR。 +- `X1`~`X4` 可以合并为一个 xlings PR,但提交按工作包分层,测试必须随对应实现提交。 +- `L2` 必须等 `L1` 有可引用版本或 commit;不能在 xlings 中保留第二套“临时兼容解析器”。 +- `I1` 只迁移有权威 hash 的活跃资源;`I2` 才做全量审计,不允许为了追求迁移率伪造或从不可信镜像推导 hash。 + +### 9.2 X1:缓存决策与 sidecar v2 + +**修改范围**:`src/core/xim/downloader.cppm`、`tests/unit/test_main.cpp`。 + +1. 先把缓存准入抽成不访问网络的决策函数,输入包含:是否声明 SHA256、文件存在/大小、HEAD 结果、远端大小/新鲜度字段、sidecar 版本/complete/size;输出为 `Hit`、`Redownload` 或 `OfflineUnverifiedHit`。 +2. 先增加失败测试:HEAD 失败 + 114 KiB legacy 文件必须 `Redownload`;HEAD 失败 + v2 complete 且大小一致必须 `OfflineUnverifiedHit`;v2 大小不一致必须 `Redownload`。 +3. sidecar reader 同时读取 legacy 与 v2;writer 只生成 v2,并以同目录临时文件 + rename 提交。 +4. 删除 `HEAD failed && localSize > 0` 的旧准入分支;离线复用只接受 v2 完整提交证据。 + +**2026-07-12 实施记录**: + +- 已为 `DownloadTask` 增加稳定 `cacheIdentity`,由解析后的包名、版本、平台、架构和资源键组成。 +- sidecar reader 兼容 legacy,严格校验 v2 的 `format/complete/size/cache-identity`;writer 只写 v2。 +- 已删除 HEAD 失败时任意非空缓存命中,只有 v2 的大小和 identity 同时匹配才返回 `OfflineUnverifiedHit`。 +- RED 证据:`mcpp test` 曾因缺少 `CacheAdmissionInput_` / `decide_cache_admission_` 编译失败。 +- GREEN 证据:`XimDownloaderTest` 7/7 通过;完整 `mcpp test` 10 个测试二进制全部通过。 +- 测试门禁同时修复 `test_mirror` 对真实用户 `XLINGS_HOME` 的污染:测试进程现在使用独立临时 home,不再读取用户的 `github-mirrors.json`。 + +### 9.3 X2:文件下载事务 + +**修改范围**:`src/core/xim/downloader.cppm`、下载器单元/集成测试。 + +1. 每次 GET 使用 `.part..`;`tinyhttps::DownloadOptions::destFile` 和 `onVerify` 都指向 staging。 +2. 下载、候选校验、大小检查或取消失败时删除本次 staging,不删除已有已提交目标。 +3. 验收后执行 `dest -> backup`、`staging -> dest`、删除 backup;第二步失败必须恢复旧目标。 +4. 最终文件提交后再原子写 sidecar v2。文件已提交但 sidecar 未提交时,下次按保守 miss 处理。 +5. 测试必须覆盖:候选一 hash 失败、候选二成功;取消;提交失败回滚;遗留 `.part.*` 不被当缓存。 + +**2026-07-12 实施记录**: + +- tinyhttps 的目标和逐候选 `onVerify` 已统一改为唯一 sibling staging 文件,最终路径不再承接网络写入。 +- SHA mismatch、传输失败、取消和极小错误页只删除本次 staging;旧目标保留到新内容验收成功。 +- 提交使用 `live -> backup -> staging -> live`,第二次 rename 失败会恢复旧目标;该语义明确称为“可恢复提交”,不宣称两次 rename 之间最终路径始终存在。 +- 已加入传输和 HEAD probe 测试 seam;失败保留旧目标、预取消零传输、候选 hash fallback、backup 后故障注入恢复测试均通过。 +- 全套 `mcpp test`:10 passed,0 failed。 + +### 9.4 X3:跨进程并发 + +**修改范围**:`src/platform.cppm`、对应三平台实现、`src/core/xim/downloader.cppm`、测试。 + +1. 以最终目标路径派生锁文件;缓存判定、下载、提交和 sidecar 写入均在锁内。 +2. Unix 使用 advisory file lock,Windows 使用不共享写入的文件句柄;等待检查取消并有明确超时错误。 +3. 清理只处理锁持有者能够证明为过期的 staging/backup,不按通配符删除其他进程文件。 +4. 并发测试断言只发生一次有效提交,等待者随后命中已提交缓存。 + +**2026-07-12 实施记录**: + +- `platform::FileLock` 已实现:Linux/macOS 使用 `flock`,Windows 使用独占 `CreateFileW`;等待支持取消和 10 分钟超时。 +- 锁已覆盖文件缓存判定、传输、提交和 sidecar 写入。 +- 持锁后的恢复状态机已覆盖“仅 backup 时恢复 live”“live + backup 时保留 live”“清理同目标遗留 staging”。 +- 同进程双句柄竞争、取消等待、延迟释放、两种崩溃恢复测试通过;完整 `mcpp test` 10/10 通过。 +- 新增真实子进程持锁测试:父进程在子进程写入 ready 标记后竞争同一锁,必须等待子进程释放,再完成提交;Linux 定向测试通过。 +- [xlings#359](https://github.com/openxlings/xlings/pull/359) 最终实现 HEAD 的 Linux、macOS、Windows、aarch64/QEMU、Linux E2E、root 静态构建和 root 身份验证七项检查均通过;Windows CI 同时验证了句柄释放生命周期和真实子进程命令行。 + +### 9.5 X4:解压失败自愈 + +**修改范围**:`src/core/xim/extract.cppm`、`src/core/xim/installer.cppm`、相关测试。 + +1. 解压结果区分 `InvalidInputArchive` 与 `LocalWriteFailure`。 +2. 前者删除下载缓存及 sidecar,后者保留缓存并返回环境错误。 +3. 不增加下载后的完整归档预扫描;测试直接构造截断归档和不可写目标。 + +**2026-07-12 实施记录**: + +- 新增兼容 API `extract_archive_detailed()`,返回 `InvalidInputArchive`、`LocalWriteFailure` 或 `Internal`;原 `extract_archive()` 字符串接口保留。 +- open/next-header/read-data/不安全归档路径归为输入错误;创建目录、write-header/write-data/finish-entry 归为本地写错误。 +- installer 只在 `InvalidInputArchive` 时删除归档和 `.meta`;本地写错误保留缓存。 +- 非法归档驱逐及“目标是普通文件”本地写失败保留缓存测试通过;未增加归档预扫描。 +- 完整 `mcpp test`:10 passed,0 failed。 + +### 9.6 T1:tinyhttps 协议完整性 + +在独立仓库和独立 PR 中补协议级本地 HTTP server 测试,再修 chunk parser。成功结果公开 `bytesWritten`、可选 `expectedBytes`、`finalUrl`;xlings 在升级依赖后使用这些字段写 sidecar和日志。connection-close 响应的 `expectedBytes` 必须为空,不能被描述为长度已验证。 + +**2026-07-12 实施记录**: + +- [tinyhttps#9](https://github.com/mcpplibs/tinyhttps/pull/9) 已实现严格 chunk-size/CRLF/trailer EOF 判定,并公开 `bytesWritten`、可选 `expectedBytes`、`finalUrl`、`etag` 和 `lastModified`。 +- `mcpp test` 的本地 HTTPS 与协议回归测试通过;PR 的 Linux mcpp CI 已通过。 +- `mcpp.toml` 已升至 0.2.9;PR 已合并并发布 [0.2.9](https://github.com/mcpplibs/tinyhttps/releases/tag/0.2.9)。 +- xlings wrapper 已消费实际/期望字节、最终 URL、ETag 和 Last-Modified;长度不一致在提交缓存前拒绝,GET 元数据进入 sidecar。 +- mcpp 默认索引 PR [#68](https://github.com/mcpplibs/mcpp-index/pull/68) 六项 CI 全绿后已合并(`8c75ace`);GLOBAL/CN 同字节 SHA256 已验证,正式 index artifact workflow `29178149741` 发布成功。 + +### 9.7 L1/L2:唯一 xpkg 入口 + +`libxpkg` 接受并归一化原有版本资源项以及 `xpm.source = "xlings-res" | `,输出平台、架构、最终 URL、SHA256 和 fallback 的统一资源对象。兼容逻辑只位于 libxpkg 的 compat 模块。xlings 升级依赖并删除 `load_platform_entries_()`;删除前必须用同一组 fixture 对旧字符串 URL、`XLINGS_RES`、mirror table、`ref`、单 hash 和多架构 hash 做前后结果对比。 + +**2026-07-12 L1 实施记录**: + +- [libxpkg#26](https://github.com/openxlings/libxpkg/pull/26) 新增 `mcpplibs.xpkg.compat`,`resolve_resource()` 成为旧资源写法、`xlings-res` 与 URL template 的统一归一化入口。 +- loader 将根级/平台级 `source` 解析为元数据并明确跳过版本迭代;原 platform/version 模型保持不变。 +- 契约测试覆盖显式 URL 优先级、根级/平台级 source、`${name/version/os/arch/arch_alias/ext}`、mirror 展开、ref/cycle、旧 `XLINGS_RES`、`res=true`、per-arch map 和 per-arch SHA256;loader/compat 26/26 通过。 +- `mcpp build` 通过。完整 `mcpp test` 的 4 个 elfpatch executor 失败已在干净 `main@9e934be` 原样复现,确认不是 L1 回归。 +- 兼容审计发现 0.0.43 loader 缺少目标平台 sandbox 上下文,不能安全替换 xlings 的旧 parser;因此没有把 0.0.43 推入默认索引。 +- [libxpkg#27](https://github.com/openxlings/libxpkg/pull/27) 增加 `LoaderContext { platform, arch }`,统一支持 `is_host/is_plat/is_arch`、`os.host/os.arch` 和 `_RUNTIME`,并把 per-arch 缺失收敛为 compat fail-closed。 +- #26/#27 CI 均通过并已合并,最终发布 [libxpkg 0.0.44](https://github.com/openxlings/libxpkg/releases/tag/v0.0.44)。 +- xlings 已删除 `load_platform_entries_()`、本地 Lua sandbox 和本地 template 展开器;平台上下文加载和资源归一化全部调用 libxpkg。 +- libxpkg 0.0.44 默认索引及 artifact 已发布;xlings 已切换正式依赖 0.0.44,并与 tinyhttps 0.2.9 一起从重置后的 GLOBAL 索引解析。`mcpp build && mcpp test` 10/10 通过,其中 `test_main` 214 项无失败(197 通过、17 项因本地无 xim-pkgindex fixture 跳过);尚待 PR 三平台 CI。 + +### 9.8 I1/I2:索引迁移与老客户端保护 + +新增 `xpm.source` 是可选字段,不会改变旧的 `xpm[platform][version]` 条目;因此迁移采用“双轨兼容”,而不是立即重写所有历史条目: + +- 旧 `"XLINGS_RES"` 条目继续保留并可被旧客户端读取。 +- 新版本推荐使用 `xpm.source = "xlings-res"`;迁移期的版本项仍写 `{ url = "XLINGS_RES", sha256 = { ... } }`,让 V1/V2/修复版都能取到 URL。 +- 在确认目标老版本客户端会把 `source` 当保留元数据而不是版本键之前,不对官方索引批量加入该字段;兼容测试必须实际运行至少当前稳定版和修复版解析同一 fixture。 +- 如果旧客户端会误把 `source` 当平台或版本,官方索引只先落地 `{ url = "XLINGS_RES", sha256 = { ... } }`;不能只用 `res=true` 或 `{}`,因为 V1 客户端会失去 URL。 +- 2026-07-12 实测确认 pointer 是滚动当前快照,0.4.52+ 默认读取当前 pointer,更老客户端仍拉取索引 `main`;版本化 artifact 只支持人工复现/回滚,不会自动把旧客户端固定在旧索引。因此迁移前客户端矩阵是硬门禁。 + +当前迁移 PR 为 [xim-pkgindex#352](https://github.com/openxlings/xim-pkgindex/pull/352):只迁移已取得权威 hash 的 mcpp 0.0.67、0.0.81、0.0.87。三版本、四资产、GLOBAL/CN 共 24 次完整 GET 均与权威 SHA256 一致且镜像逐字节相同;0.4.49、0.4.62 和本地修复版在隔离 home 中完成 9/9 安装。生成器会验证平台/架构集合、binary、sidecar 和流式 SHA256,任何缺失或不一致均整次 fail closed 且不改配方。 + +### 9.9 发布门禁与最终验收 + +1. 每个仓库从干净的 `main` 建独立分支/worktree,不覆盖当前工作区已有改动。 +2. PR 描述包含根因、行为变化、测试命令和跨仓依赖;所有 Linux、macOS、Windows 必需检查通过。 +3. 先发布 tinyhttps/libxpkg,再升级 xlings 依赖与版本;xlings release 必须生成各平台资产和 SHA256/manifest。 +4. release 自动或人工生成索引 PR;核对 GLOBAL、CN 资源 hash 与 manifest 一致后合并。 +5. 用隔离 `XLINGS_HOME` 验证:旧稳定客户端仍能安装未迁移条目;新客户端能安装新旧条目;预置 114 KiB 错误缓存后安装 mcpp 会驱逐、重下并成功解压;离线 v2 完整缓存可复用。 +6. 只有 Issue #356 关闭、所有跨仓 PR/release/index 证据链接回填本节后,`R1` 才能标记完成。 + +## 10. 最终建议 + +按 **P0 索引止血 → P1 事务缓存根治 → P2 发布生态收口** 的顺序实施。若只能先做一个改动,应先给 mcpp 补 SHA256,因为它能立即修复 Issue 中的用户;但关闭 Issue #356 应以 P1 完成并覆盖“HEAD 失败 + legacy 半文件”和“chunked 提前 EOF”回归测试为准,不能只以 mcpp 当前版本恢复安装为准。 diff --git a/.agents/docs/2026-07-11-xpkg-resource-expression-design.md b/.agents/docs/2026-07-11-xpkg-resource-expression-design.md new file mode 100644 index 00000000..4405b1ad --- /dev/null +++ b/.agents/docs/2026-07-11-xpkg-resource-expression-design.md @@ -0,0 +1,398 @@ +# xpm 资源表达与兼容设计 + +> 日期:2026-07-11 +> 修订:2026-07-12 +> 状态:已实现,待随 xlings 发布 +> 适用版本:libxpkg 0.0.44+ +> 范围:libxpkg 解析/compat、xlings 安装器、官方 xim-pkgindex + +## 1. 结论 + +不重新设计 `xpm`。平台和版本两层结构保持不变: + +```lua +xpm = { + linux = { + ["1.0.0"] = { + url = "https://example.test/foo-1.0.0-linux-x86_64.tar.gz", + sha256 = "", + }, + }, +} +``` + +只增加可选默认来源: + +```lua +xpm.source = "xlings-res" +``` + +或: + +```lua +xpm.source = "https://github.com/acme/foo/releases/download/${version}/foo-${version}-${os}-${arch}.tar.gz" +``` + +版本项、mirror、`ref`、显式 URL 和多架构结构仍使用原模型。`source` 只减少重复,不建立第二套 versions/targets DSL。 + +## 2. 设计目标 + +需要同时满足: + +1. 官方 xlings-res 不再为每个平台、每个版本重复写 `"XLINGS_RES"`。 +2. 同一 URL 模板的多个版本不再重复 URL。 +3. 单 hash、per-arch hash、架构别名和 mirror 继续可用。 +4. 不规则版本可用显式 URL 覆盖默认来源。 +5. 旧包继续工作;兼容逻辑只存在于 libxpkg。 +6. xlings 不再执行第二套 Lua 资源解析器。 + +## 3. 权威职责边界 + +```text +libxpkg loader + 读取 package/xpm,注入目标 platform/arch 兼容上下文 + ↓ +libxpkg compat::resolve_resource + ref、source 优先级、arch、hash、template、mirror 归一化 + ↓ +xlings installer + 选择资源服务器、生成 xlings-res URL/fallback、形成下载任务 + ↓ +xlings downloader + mirror 尝试、SHA256、事务缓存、sidecar +``` + +libxpkg 是唯一解析、兼容和资源归一化入口。xlings 只保留产品配置相关策略,例如用户选择的资源服务器和 fallback 列表。 + +## 4. source 语法 + +### 4.1 根级 source + +作用于所有平台: + +```lua +xpm = { + source = "xlings-res", + + linux = { + ["1.0.0"] = {}, + }, + macosx = { + ["1.0.0"] = {}, + }, +} +``` + +### 4.2 平台级 source + +覆盖根级 source: + +```lua +xpm = { + source = "xlings-res", + + windows = { + source = "https://vendor.test/foo/${version}/foo-${arch_alias}.zip", + ["1.0.0"] = { + arch_alias = { x86_64 = "win64" }, + sha256 = { x86_64 = "" }, + }, + }, +} +``` + +### 4.3 source 值 + +仅支持两类字符串: + +- `"xlings-res"`:由 xlings 根据资源服务器配置生成 URL。 +- URL template:普通 HTTP(S) URL,可包含占位符。 + +libxpkg 对 `xlings-res` 不硬编码服务器地址;它返回 `SourceKind::XlingsRes`,由 xlings 生成主 URL 和 fallback。 + +## 5. 版本项 + +使用默认 source 且不声明 hash: + +```lua +["1.0.0"] = {} +``` + +单 hash: + +```lua +["1.0.0"] = { sha256 = "" } +``` + +per-arch hash: + +```lua +["1.0.0"] = { + sha256 = { + x86_64 = "", + aarch64 = "", + }, +} +``` + +显式覆盖默认 source: + +```lua +["1.0.1"] = { + url = "https://special.test/foo-1.0.1.tar.gz", + sha256 = "", +} +``` + +版本别名: + +```lua +["latest"] = { ref = "1.0.1" } +``` + +`ref` 支持多跳;目标缺失或形成环时 fail closed。 + +## 6. URL template + +支持: + +| 占位符 | 值 | +|---|---| +| `${name}` | package name | +| `${version}` | 跟随 ref 后的最终版本 | +| `${os}` | `linux` / `macosx` / `windows` | +| `${arch}` | 规范化架构,如 `x86_64` / `aarch64` | +| `${arch_alias}` | 当前版本的架构别名;缺省等于 `${arch}` | +| `${ext}` | Windows 为 `zip`,其他为 `tar.gz`,调用者可覆盖 | + +示例: + +```lua +xpm = { + source = "https://github.com/acme/foo/releases/download/${version}/foo-${version}-${os}-${arch_alias}.${ext}", + + linux = { + ["1.0.0"] = { + arch_alias = { + x86_64 = "amd64", + aarch64 = "arm64", + }, + sha256 = { + x86_64 = "", + aarch64 = "", + }, + }, + }, +} +``` + +template 同样应用于 mirror URL。 + +## 7. 当前支持的资源写法 + +### A. 字符串 URL + +```lua +["1.0.0"] = "https://example.test/foo.tar.gz" +``` + +### B. 旧 XLINGS_RES sentinel + +```lua +["1.0.0"] = "XLINGS_RES" +``` + +继续兼容,但新包推荐使用 `xpm.source = "xlings-res"`。 + +### C. URL + SHA256 + +```lua +["1.0.0"] = { url = "https://example.test/foo.tar.gz", sha256 = "" } +``` + +### D. mirror table + +```lua +["1.0.0"] = { + url = { + GLOBAL = "https://global.test/foo.tar.gz", + CN = "https://cn.test/foo.tar.gz", + }, + sha256 = "", +} +``` + +### E. URL template + per-arch SHA256 + +```lua +["1.0.0"] = { + url = "https://example.test/foo-${version}-${arch_alias}.tar.gz", + arch_alias = { x86_64 = "amd64", aarch64 = "arm64" }, + sha256 = { x86_64 = "", aarch64 = "" }, +} +``` + +### F. per-arch resource map + +```lua +["1.0.0"] = { + x86_64 = { url = "https://example.test/foo-amd64.tar.gz", sha256 = "" }, + aarch64 = { url = "https://example.test/foo-arm64.tar.gz", sha256 = "" }, +} +``` + +### G. `res = true` + +```lua +["1.0.0"] = { + res = true, + sha256 = { x86_64 = "", aarch64 = "" }, +} +``` + +这是已发布的历史 V2 输入,libxpkg 继续兼容;新包不推荐使用,推荐上提为 `xpm.source = "xlings-res"`。 + +## 8. 归一化优先级 + +```text +1. 定位 platform/version。 +2. 跟随 ref 到最终版本,并检测 missing target/cycle。 +3. 若存在 per-arch resource map,严格选择 host arch。 +4. 选择单 hash 或 host arch 对应 hash。 +5. 版本显式 URL 优先。 +6. 平台 source 次之。 +7. 根级 source 最后。 +8. 展开主 URL 和 mirrors 的 template。 +``` + +以下情况 fail closed: + +- 平台或版本不存在。 +- ref 目标不存在或形成环。 +- per-arch resource map 非空但缺少 host arch。 +- per-arch SHA256 表非空但缺少 host arch。 +- 最终既没有 URL 也没有 source。 + +hash 完全缺失仍允许下载,以兼容旧包;这不等于完整性已验证。 + +## 9. 两个最佳范例 + +### 9.1 官方 xlings-res + +```lua +package = { + spec = "2", + name = "mcpp", + archs = { "x86_64", "aarch64" }, + + xpm = { + source = "xlings-res", + + linux = { + ["latest"] = { ref = "0.0.87" }, + ["0.0.87"] = { + sha256 = { + x86_64 = "", + aarch64 = "", + }, + }, + }, + + macosx = { + ["latest"] = { ref = "0.0.87" }, + ["0.0.87"] = { + sha256 = { + x86_64 = "", + aarch64 = "", + }, + }, + }, + + windows = { + ["latest"] = { ref = "0.0.87" }, + ["0.0.87"] = { + sha256 = { x86_64 = "" }, + }, + }, + }, +} +``` + +### 9.2 用户自定义 release + mirror + 特殊版本覆盖 + +```lua +package = { + spec = "2", + name = "foo", + archs = { "x86_64", "aarch64" }, + + xpm = { + source = "https://github.com/acme/foo/releases/download/v${version}/foo-${os}-${arch_alias}.${ext}", + + linux = { + ["latest"] = { ref = "2.1.0" }, + ["2.1.0"] = { + arch_alias = { x86_64 = "amd64", aarch64 = "arm64" }, + sha256 = { x86_64 = "", aarch64 = "" }, + }, + ["2.0.0"] = { + url = { + GLOBAL = "https://legacy.test/foo-2.0.0-linux.tar.gz", + CN = "https://cn.test/foo-2.0.0-linux.tar.gz", + }, + sha256 = "", + }, + }, + + windows = { + source = "https://downloads.acme.test/foo/${version}/foo-${arch_alias}.zip", + ["2.1.0"] = { + arch_alias = { x86_64 = "win64" }, + sha256 = { x86_64 = "" }, + }, + }, + }, +} +``` + +## 10. Loader 平台兼容 + +旧包可能在 Lua 顶层使用: + +```lua +if is_host("linux") then + package.xpm.linux = { ... } +end +``` + +libxpkg 0.0.44 增加 `LoaderContext { platform, arch }`,统一提供: + +- `is_host()` / `is_plat()` +- `is_arch()` +- `os.host()` / `os.arch()` +- `_RUNTIME.platform` / `_RUNTIME.arch` + +xlings 安装阶段通过该上下文加载包,因此删除旧的 `load_platform_entries_()` 不会牺牲这类历史兼容性。 + +## 11. 老客户端与官方索引迁移 + +新增字段对新客户端是可选元数据,但部分旧 libxpkg/xlings loader 会把根级 `source` 当作平台或把平台级 `source` 当作版本。因此不能仅凭“字段是追加的”推断旧客户端安全。 + +官方索引迁移遵循: + +1. 历史 `"XLINGS_RES"` 条目不重写,老客户端继续读取。 +2. 新版本先补权威 SHA256;没有完整 hash 时不伪造。 +3. 实际运行当前稳定旧客户端和修复版解析/安装同一 fixture。 +4. 只有旧客户端不会误解析 `source`,才可在其可见索引轨道加入推荐写法。 +5. 如果旧客户端不兼容,新语法只进入明确要求新 xlings 的索引轨道,或暂时继续发布 `{ url = "XLINGS_RES", sha256 = { ... } }` 双兼容结构。 + +当前 pointer 是滚动快照,旧客户端不会自动固定旧 artifact;更老客户端还会直接拉取索引 `main`。版本化 artifact 只能人工复现/回滚,不能替代解析兼容测试或作为默认保护机制。 + +## 12. 实现与验证 + +- libxpkg 0.0.43:实现 `source`、compat 归一化和旧资源模型测试。 +- libxpkg 0.0.44:增加平台 loader context 和 per-arch fail-closed。 +- xlings:删除 `load_platform_entries_()`、本地 template 展开器和第二套 Lua sandbox;安装器只调用 libxpkg。 +- tinyhttps 0.2.9:返回实际/期望字节、最终 URL、ETag 和 Last-Modified,供事务缓存 sidecar 使用。 + +最终发布门禁以 issue #356 实施文档的 PR、CI、release、索引和 E2E 台账为准。 diff --git a/mcpp.lock b/mcpp.lock index 37799b95..786cb7bf 100644 --- a/mcpp.lock +++ b/mcpp.lock @@ -27,13 +27,13 @@ hash = "fnv1a:8ee8a8e51ac69885" [package."mcpplibs.tinyhttps"] namespace = "mcpplibs" -version = "0.2.8" -source = "index+mcpplibs@0.2.8" -hash = "fnv1a:bc63967ab318ede6" +version = "0.2.9" +source = "index+mcpplibs@0.2.9" +hash = "fnv1a:3465dd0bd5d7aa20" [package."mcpplibs.xpkg"] namespace = "mcpplibs" -version = "0.0.42" -source = "index+mcpplibs@0.0.42" -hash = "fnv1a:841e4283e2553732" +version = "0.0.44" +source = "index+mcpplibs@0.0.44" +hash = "fnv1a:750b1f1ed374ad3d" diff --git a/mcpp.toml b/mcpp.toml index 9042e6f1..038899bf 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,6 +1,6 @@ [package] name = "xlings" -version = "0.4.61" +version = "0.4.63" description = "Universal package management infrastructure tool with SubOS isolation" license = "Apache-2.0" repo = "https://github.com/openxlings/xlings" @@ -39,8 +39,8 @@ libarchive = "3.8.7" [dependencies.mcpplibs] cmdline = "0.0.2" -xpkg = "0.0.42" -tinyhttps = "0.2.8" +xpkg = "0.0.44" +tinyhttps = "0.2.9" capi.lua = "0.0.3" [dev-dependencies] diff --git a/src/core/config.cppm b/src/core/config.cppm index cc61d359..51800aad 100644 --- a/src/core/config.cppm +++ b/src/core/config.cppm @@ -13,7 +13,7 @@ import xlings.core.xvm.db; namespace xlings { export struct Info { - static constexpr std::string_view VERSION = "0.4.62"; + static constexpr std::string_view VERSION = "0.4.63"; static constexpr std::string_view REPO = "https://github.com/openxlings/xlings"; }; diff --git a/src/core/xim/downloader.cppm b/src/core/xim/downloader.cppm index df627556..01d7bca2 100644 --- a/src/core/xim/downloader.cppm +++ b/src/core/xim/downloader.cppm @@ -72,10 +72,136 @@ std::string lower_hex_(std::string_view s) { // Format: one "key: value" per line, only `last-modified` and `etag` // recognized. Anything else is ignored. Missing sidecar = no metadata. struct MetaSidecar_ { + int format { 1 }; + bool complete { false }; + std::int64_t size { -1 }; + std::string sourceUrl; + std::string cacheIdentity; std::string lastModified; std::string etag; }; +std::filesystem::path unique_sibling_path_( + const std::filesystem::path& base, + std::string_view marker) { + static std::atomic_uint64_t sequence { 0 }; + auto path = base; + path += std::format( + ".{}.{}.{}", marker, + std::chrono::steady_clock::now().time_since_epoch().count(), + sequence.fetch_add(1)); + return path; +} + +bool commit_staging_file_(const std::filesystem::path& staging, + const std::filesystem::path& destination, + std::string& error, + bool failAfterBackupForTest = false) { + namespace fs = std::filesystem; + std::error_code ec; + auto backup = unique_sibling_path_(destination, "old"); + const bool hadDestination = fs::exists(destination, ec); + if (ec) { + error = std::format("failed to inspect {}: {}", + destination.string(), ec.message()); + return false; + } + if (hadDestination) { + fs::rename(destination, backup, ec); + if (ec) { + error = std::format("failed to preserve {}: {}", + destination.string(), ec.message()); + return false; + } + } + if (failAfterBackupForTest) { + if (hadDestination) { + fs::rename(backup, destination, ec); + } + error = "injected commit failure after backup"; + return false; + } + fs::rename(staging, destination, ec); + if (ec) { + auto commitError = ec.message(); + if (hadDestination) { + std::error_code restoreEc; + fs::rename(backup, destination, restoreEc); + if (restoreEc) { + error = std::format( + "failed to commit {} ({}) and restore previous file ({})", + destination.string(), commitError, restoreEc.message()); + return false; + } + } + error = std::format("failed to commit {}: {}", + destination.string(), commitError); + return false; + } + if (hadDestination) { + fs::remove(backup, ec); + } + return true; +} + +bool recover_download_transaction_( + const std::filesystem::path& destination, + std::string& error) { + namespace fs = std::filesystem; + std::error_code ec; + std::vector backups; + std::vector stagingFiles; + const auto backupPrefix = destination.filename().string() + ".old."; + const auto stagingPrefix = destination.filename().string() + ".part."; + auto parent = destination.parent_path(); + for (auto it = fs::directory_iterator(parent, ec); + !ec && it != std::default_sentinel; it.increment(ec)) { + auto name = it->path().filename().string(); + if (name.starts_with(backupPrefix)) backups.push_back(it->path()); + else if (name.starts_with(stagingPrefix)) stagingFiles.push_back(it->path()); + } + if (ec) { + error = std::format("failed to inspect download transaction for {}: {}", + destination.string(), ec.message()); + return false; + } + + std::ranges::sort(backups); + const bool liveExists = fs::exists(destination, ec); + if (ec) { + error = std::format("failed to inspect {}: {}", + destination.string(), ec.message()); + return false; + } + if (!liveExists && !backups.empty()) { + auto restore = backups.back(); + backups.pop_back(); + fs::rename(restore, destination, ec); + if (ec) { + error = std::format("failed to restore interrupted download {}: {}", + destination.string(), ec.message()); + return false; + } + } + for (const auto& path : backups) { + fs::remove(path, ec); + if (ec) { + error = std::format("failed to remove stale backup {}: {}", + path.string(), ec.message()); + return false; + } + } + for (const auto& path : stagingFiles) { + fs::remove(path, ec); + if (ec) { + error = std::format("failed to remove stale staging file {}: {}", + path.string(), ec.message()); + return false; + } + } + return true; +} + std::optional read_meta_sidecar_(const std::filesystem::path& p) { std::ifstream in(p); if (!in) return std::nullopt; @@ -93,18 +219,133 @@ std::optional read_meta_sidecar_(const std::filesystem::path& p) { }; trim(key); trim(val); for (auto& c : key) c = static_cast(std::tolower(static_cast(c))); - if (key == "last-modified") m.lastModified = std::move(val); - else if (key == "etag") m.etag = std::move(val); + if (key == "format") { + int parsed {}; + auto [ptr, err] = std::from_chars(val.data(), val.data() + val.size(), parsed); + if (err != std::errc{} || ptr != val.data() + val.size()) return std::nullopt; + m.format = parsed; + } else if (key == "complete") { + if (val == "true") m.complete = true; + else if (val == "false") m.complete = false; + else return std::nullopt; + } else if (key == "size") { + std::int64_t parsed {}; + auto [ptr, err] = std::from_chars(val.data(), val.data() + val.size(), parsed); + if (err != std::errc{} || ptr != val.data() + val.size() || parsed < 0) { + return std::nullopt; + } + m.size = parsed; + } else if (key == "source-url") { + m.sourceUrl = std::move(val); + } else if (key == "cache-identity") { + m.cacheIdentity = std::move(val); + } else if (key == "last-modified") { + m.lastModified = std::move(val); + } else if (key == "etag") { + m.etag = std::move(val); + } + } + if (m.format != 1 && m.format != 2) return std::nullopt; + if (m.format == 2 + && (!m.complete || m.size < 0 || m.cacheIdentity.empty())) { + return std::nullopt; } return m; } -void write_meta_sidecar_(const std::filesystem::path& p, - const tinyhttps::RemoteFileMeta& meta) { - std::ofstream out(p, std::ios::trunc); - if (!out) return; - if (!meta.lastModified.empty()) out << "last-modified: " << meta.lastModified << "\n"; - if (!meta.etag.empty()) out << "etag: " << meta.etag << "\n"; +bool write_meta_sidecar_(const std::filesystem::path& p, + const tinyhttps::RemoteFileMeta& meta, + std::int64_t size, + std::string_view cacheIdentity, + std::string_view sourceUrl) { + auto staging = unique_sibling_path_(p, "tmp"); + { + std::ofstream out(staging, std::ios::trunc); + if (!out) return false; + out << "format: 2\n"; + out << "complete: true\n"; + out << "size: " << size << "\n"; + out << "source-url: " << sourceUrl << "\n"; + out << "cache-identity: " << cacheIdentity << "\n"; + if (!meta.lastModified.empty()) out << "last-modified: " << meta.lastModified << "\n"; + if (!meta.etag.empty()) out << "etag: " << meta.etag << "\n"; + out.flush(); + if (!out) { + std::error_code ec; + std::filesystem::remove(staging, ec); + return false; + } + } + std::string error; + if (!commit_staging_file_(staging, p, error)) { + std::error_code ec; + std::filesystem::remove(staging, ec); + return false; + } + return true; +} + +enum class CacheAdmission_ { + Hit, + OfflineUnverifiedHit, + Redownload, +}; + +struct CacheAdmissionInput_ { + std::int64_t localSize { -1 }; + bool headSucceeded { false }; + std::int64_t remoteSize { -1 }; + std::string remoteLastModified; + std::string remoteEtag; + std::optional sidecar; + std::string expectedCacheIdentity; +}; + +CacheAdmission_ decide_cache_admission_(const CacheAdmissionInput_& input) { + if (input.localSize <= 0) return CacheAdmission_::Redownload; + + if (!input.headSucceeded) { + if (!input.sidecar) return CacheAdmission_::Redownload; + const auto& stored = *input.sidecar; + const bool committedV2 = + stored.format == 2 + && stored.complete + && stored.size == input.localSize + && !input.expectedCacheIdentity.empty() + && stored.cacheIdentity == input.expectedCacheIdentity; + return committedV2 + ? CacheAdmission_::OfflineUnverifiedHit + : CacheAdmission_::Redownload; + } + + const bool sizeMatch = + input.remoteSize > 0 && input.remoteSize == input.localSize; + if (!sizeMatch) return CacheAdmission_::Redownload; + + if (!input.sidecar) return CacheAdmission_::Hit; + const auto& stored = *input.sidecar; + const bool freshMatch = + (!input.remoteLastModified.empty() + && input.remoteLastModified == stored.lastModified) + || (!input.remoteEtag.empty() && input.remoteEtag == stored.etag); + const bool noFreshnessEvidence = + stored.lastModified.empty() && stored.etag.empty(); + return (freshMatch || noFreshnessEvidence) + ? CacheAdmission_::Hit + : CacheAdmission_::Redownload; +} + +struct DownloadTestHooks_ { + std::function transferOverride; + std::function queryRemoteMeta; +}; + +tinyhttps::RemoteFileMeta query_remote_meta_( + const std::string& url, + const DownloadTestHooks_* hooks) { + if (hooks && hooks->queryRemoteMeta) return hooks->queryRemoteMeta(url); + return tinyhttps::query_remote_meta(url); } // Derive the destination directory name from a git URL, e.g. @@ -240,7 +481,8 @@ DownloadResult git_clone_one(const DownloadTask& task) { // Download a single file using libcurl with real-time progress callback. DownloadResult download_one(const DownloadTask& task, std::function onProgress = nullptr, - CancellationToken* cancel = nullptr) { + CancellationToken* cancel = nullptr, + const DownloadTestHooks_* testHooks = nullptr) { namespace fs = std::filesystem; DownloadResult result; @@ -334,6 +576,23 @@ DownloadResult download_one(const DownloadTask& task, sidecarPath += ".meta"; result.localFile = destFile; + auto lockPath = destFile; + lockPath += ".lock"; + platform::FileLock cacheLock; + std::string lockError; + auto cancelled = [cancel] { + return cancel && (cancel->is_paused() || cancel->is_cancelled()); + }; + if (!cacheLock.acquire( + lockPath, std::chrono::minutes{10}, cancelled, lockError)) { + result.error = std::move(lockError); + return result; + } + if (!recover_download_transaction_(destFile, lockError)) { + result.error = std::move(lockError); + return result; + } + // ── Cache hit path 1: sha256 verified (cheapest, most reliable) ── // If the recipe declares a sha256 and the on-disk file matches, we're // byte-identical to upstream — skip download outright. @@ -350,7 +609,7 @@ DownloadResult download_one(const DownloadTask& task, // sha mismatch: stale/corrupt cache. Remove before falling through // so the next download_to_file starts from a clean slate (defends // against a future tinyhttps that might enable Range/resume). - fs::remove(destFile, ec); + // Keep the previous file until a verified replacement is ready. } // ── Cache hit path 2: HEAD-based freshness (when sha256 is unset) ── @@ -360,46 +619,41 @@ DownloadResult download_one(const DownloadTask& task, tinyhttps::RemoteFileMeta probedMeta; bool probedMetaValid = false; if (fs::exists(destFile) && task.sha256.empty()) { - probedMeta = tinyhttps::query_remote_meta(task.url); + probedMeta = query_remote_meta_(task.url, testHooks); probedMetaValid = true; if (probedMeta.ok) { std::error_code sec; auto localSize = static_cast(fs::file_size(destFile, sec)); - std::string storedLM, storedETag; - if (auto stored = read_meta_sidecar_(sidecarPath)) { - storedLM = std::move(stored->lastModified); - storedETag = std::move(stored->etag); - } - bool sizeMatch = (probedMeta.contentLength > 0) - && (localSize == probedMeta.contentLength); - // Strong freshness signal: server's Last-Modified or ETag - // matches what we recorded the last time we downloaded. - bool freshMatch = - (!probedMeta.lastModified.empty() && probedMeta.lastModified == storedLM) - || (!probedMeta.etag.empty() && probedMeta.etag == storedETag); - // Weak signal: no sidecar (legacy file from before this code), - // but the size matches what the server reports right now. - bool weakMatch = sizeMatch && storedLM.empty() && storedETag.empty(); - - if (sizeMatch && (freshMatch || weakMatch)) { + auto admission = decide_cache_admission_({ + .localSize = sec ? -1 : localSize, + .headSucceeded = true, + .remoteSize = probedMeta.contentLength, + .remoteLastModified = probedMeta.lastModified, + .remoteEtag = probedMeta.etag, + .sidecar = read_meta_sidecar_(sidecarPath), + .expectedCacheIdentity = task.cacheIdentity, + }); + if (admission == CacheAdmission_::Hit) { log::debug("already downloaded (HEAD cache hit, size={}): {}", localSize, destFile.string()); result.success = true; return result; } - // Stale: drop both file and sidecar before re-downloading. - fs::remove(destFile, ec); - fs::remove(sidecarPath, ec); + // Stale: retain the previous committed pair until the replacement + // has transferred and passed all acceptance checks. } else { - // HEAD failed (offline, server blocks HEAD, 4xx, etc.). If we - // already have a non-empty cached file, prefer it over failing - // — being airline-friendly is worth the small risk of serving - // a stale payload when sha256 is unset. std::error_code sec; - auto localSize = fs::file_size(destFile, sec); - if (!sec && localSize > 0) { - log::warn("HEAD probe failed for {} ({}); using cached file: {}", + auto localSize = static_cast(fs::file_size(destFile, sec)); + auto admission = decide_cache_admission_({ + .localSize = sec ? -1 : localSize, + .headSucceeded = false, + .sidecar = read_meta_sidecar_(sidecarPath), + .expectedCacheIdentity = task.cacheIdentity, + }); + if (admission == CacheAdmission_::OfflineUnverifiedHit) { + log::warn("HEAD probe failed for {} ({}); using transaction-committed " + "cache without cryptographic verification: {}", task.url, probedMeta.error.empty() ? "unknown" : probedMeta.error, destFile.string()); @@ -433,16 +687,20 @@ DownloadResult download_one(const DownloadTask& task, // .agents/docs/2026-06-04-github-asset-adaptive-mirror.md. urls = mirror::adaptive::reorder(std::move(urls), !task.sha256.empty()); + auto stagingFile = unique_sibling_path_(destFile, "part"); + tinyhttps::DownloadFileResult transferResult; + // Use in-process tinyhttps for all downloads (streaming progress). // When a CancellationToken is available, wire isCancelled so ESC aborts. { tinyhttps::DownloadOptions opts; - opts.destFile = destFile; + opts.destFile = stagingFile; opts.urls = std::move(urls); opts.retryCount = 3; opts.connectTimeoutSec = 30; opts.maxTimeSec = 600; opts.onProgress = onProgress; + if (testHooks) opts.transferOverride = testHooks->transferOverride; if (cancel) { opts.isCancelled = [cancel] { return cancel->is_paused() || cancel->is_cancelled(); }; } @@ -461,9 +719,9 @@ DownloadResult download_one(const DownloadTask& task, // whole download with the remaining candidates untried. if (!task.sha256.empty()) { auto want = lower_hex_(task.sha256); - opts.onVerify = [destFile, want, &task](const std::string& u) + opts.onVerify = [stagingFile, want, &task](const std::string& u) -> std::string { - auto digest = sha256::hex_file(destFile); + auto digest = sha256::hex_file(stagingFile); if (digest && *digest == want) return {}; return std::format( "sha256 mismatch for {} (source {}): got {}, want {}", @@ -471,13 +729,24 @@ DownloadResult download_one(const DownloadTask& task, }; } - auto dlResult = tinyhttps::download_file(opts); - if (!dlResult.success) { - result.error = dlResult.error; + transferResult = tinyhttps::download_file(opts); + if (!transferResult.success) { + fs::remove(stagingFile, ec); + result.error = transferResult.error; return result; } } + if (transferResult.expectedBytes + && transferResult.bytesWritten != *transferResult.expectedBytes) { + result.error = std::format( + "incomplete transfer for {}: wrote {} of {} bytes", + task.name, transferResult.bytesWritten, + *transferResult.expectedBytes); + fs::remove(stagingFile, ec); + return result; + } + // Size sanity check for archives without sha256. When a recipe // omits sha256 we have no way to cross-check content authenticity, // so a CDN serving 200 OK + a tiny error stub (e.g., gitcode's @@ -487,14 +756,14 @@ DownloadResult download_one(const DownloadTask& task, // misleading libarchive "unrecognized format". Skip when sha256 // is declared — that's a stronger check than size alone. if (task.sha256.empty() && looks_like_archive_filename_(destFile)) { - auto sz = fs::file_size(destFile, ec); + auto sz = fs::file_size(stagingFile, ec); if (!ec && sz < kMinPlausibleArchiveBytes_) { result.error = std::format( "{}: downloaded payload is only {} bytes — likely an " "error stub returned as 200 OK (no sha256 declared to " "cross-check). URL: {}", task.name, sz, task.url); - fs::remove(destFile, ec); + fs::remove(stagingFile, ec); return result; } } @@ -503,22 +772,48 @@ DownloadResult download_one(const DownloadTask& task, // onVerify above already gated acceptance; in-process hash, no // dependency on a host `sha256sum` binary, which stock macOS lacks). if (!task.sha256.empty()) { - auto digest = sha256::hex_file(destFile); + auto digest = sha256::hex_file(stagingFile); if (!digest || *digest != lower_hex_(task.sha256)) { result.error = std::format("SHA256 mismatch for {}", task.name); - fs::remove(destFile, ec); + fs::remove(stagingFile, ec); return result; } - } else { + } + + std::string commitError; + if (!commit_staging_file_(stagingFile, destFile, commitError)) { + fs::remove(stagingFile, ec); + result.error = std::move(commitError); + return result; + } + + if (task.sha256.empty()) { // No sha256 declared: persist server-reported Last-Modified / ETag // alongside the payload so the next install can use a HEAD probe // to decide cache freshness instead of re-downloading. - if (!probedMetaValid) { - probedMeta = tinyhttps::query_remote_meta(task.url); + tinyhttps::RemoteFileMeta committedMeta { + .ok = transferResult.success, + .contentLength = transferResult.expectedBytes.value_or(-1), + .lastModified = transferResult.lastModified, + .etag = transferResult.etag, + }; + if (committedMeta.lastModified.empty() && committedMeta.etag.empty() + && !transferResult.expectedBytes && !probedMetaValid) { + probedMeta = query_remote_meta_(task.url, testHooks); probedMetaValid = true; } - if (probedMeta.ok && (!probedMeta.lastModified.empty() || !probedMeta.etag.empty())) { - write_meta_sidecar_(sidecarPath, probedMeta); + if (committedMeta.lastModified.empty() && committedMeta.etag.empty() + && !transferResult.expectedBytes && probedMetaValid) { + committedMeta = probedMeta; + } + std::error_code sizeEc; + auto committedSize = static_cast(fs::file_size(destFile, sizeEc)); + if (!sizeEc && !task.cacheIdentity.empty()) { + write_meta_sidecar_( + sidecarPath, committedMeta, committedSize, + task.cacheIdentity, + transferResult.finalUrl.empty() + ? task.url : transferResult.finalUrl); } } diff --git a/src/core/xim/extract.cppm b/src/core/xim/extract.cppm index 7ad330c3..6d578ef9 100644 --- a/src/core/xim/extract.cppm +++ b/src/core/xim/extract.cppm @@ -10,6 +10,21 @@ import std; export namespace xlings::xim { +enum class ExtractErrorKind { + InvalidInputArchive, + LocalWriteFailure, + Internal, +}; + +struct ExtractError { + ExtractErrorKind kind { ExtractErrorKind::Internal }; + std::string message; +}; + +std::expected +extract_archive_detailed(const std::filesystem::path& archive, + const std::filesystem::path& destDir); + // In-process archive extraction backed by libarchive. // // Replaces the previous popen("tar xf …") path that suffered from a @@ -124,7 +139,7 @@ la_int64_t const_root_lookup_uid_(void*, const char*, la_int64_t) { return 0; } la_int64_t const_root_lookup_gid_(void*, const char*, la_int64_t) { return 0; } // Read each block of an entry's payload from `src` and write to `dst`. -std::expected +std::expected copy_entry_data_(struct archive* src, struct archive* dst) { const void* buff = nullptr; std::size_t size = 0; @@ -133,11 +148,15 @@ copy_entry_data_(struct archive* src, struct archive* dst) { int r = ::archive_read_data_block(src, &buff, &size, &offset); if (r == ARCHIVE_EOF) return {}; if (r < ARCHIVE_OK) { - return std::unexpected("read_data_block: " + libarchive_error_(src)); + return std::unexpected(ExtractError{ + ExtractErrorKind::InvalidInputArchive, + "read_data_block: " + libarchive_error_(src)}); } r = ::archive_write_data_block(dst, buff, size, offset); if (r < ARCHIVE_OK) { - return std::unexpected("write_data_block: " + libarchive_error_(dst)); + return std::unexpected(ExtractError{ + ExtractErrorKind::LocalWriteFailure, + "write_data_block: " + libarchive_error_(dst)}); } } } @@ -160,23 +179,25 @@ void ensure_archive_locale_() { } // namespace detail_ -std::expected -extract_archive(const std::filesystem::path& archive, - const std::filesystem::path& destDir) { +std::expected +extract_archive_detailed(const std::filesystem::path& archive, + const std::filesystem::path& destDir) { namespace fs = std::filesystem; detail_::ensure_archive_locale_(); std::error_code ec; fs::create_directories(destDir, ec); if (ec) { - return std::unexpected(std::format( - "create_directories({}) failed: {}", - destDir.string(), ec.message())); + return std::unexpected(ExtractError{ + ExtractErrorKind::LocalWriteFailure, + std::format("create_directories({}) failed: {}", + destDir.string(), ec.message())}); } if (!fs::exists(archive)) { - return std::unexpected(std::format( - "archive does not exist: {}", archive.string())); + return std::unexpected(ExtractError{ + ExtractErrorKind::InvalidInputArchive, + std::format("archive does not exist: {}", archive.string())}); } // Resolve symlinks in the destination root before handing paths to @@ -205,7 +226,9 @@ extract_archive(const std::filesystem::path& archive, if (!src || !dst) { cleanup(); - return std::unexpected("libarchive: failed to allocate handles"); + return std::unexpected(ExtractError{ + ExtractErrorKind::Internal, + "libarchive: failed to allocate handles"}); } ::archive_read_support_filter_all(src); @@ -239,7 +262,8 @@ extract_archive(const std::filesystem::path& archive, std::string err = std::format("open {}: {}", archive.string(), detail_::libarchive_error_(src)); cleanup(); - return std::unexpected(std::move(err)); + return std::unexpected(ExtractError{ + ExtractErrorKind::InvalidInputArchive, std::move(err)}); } for (;;) { @@ -249,7 +273,8 @@ extract_archive(const std::filesystem::path& archive, if (r < ARCHIVE_WARN) { std::string err = "next_header: " + detail_::libarchive_error_(src); cleanup(); - return std::unexpected(std::move(err)); + return std::unexpected(ExtractError{ + ExtractErrorKind::InvalidInputArchive, std::move(err)}); } // Reroot the entry under destDir. archive_entry_pathname is a @@ -258,7 +283,9 @@ extract_archive(const std::filesystem::path& archive, auto safeRel = detail_::check_safe_pathname_(original.c_str()); if (!safeRel) { cleanup(); - return std::unexpected(std::move(safeRel).error()); + return std::unexpected(ExtractError{ + ExtractErrorKind::InvalidInputArchive, + std::move(safeRel).error()}); } auto rebased = (canonicalDest / *safeRel).lexically_normal().string(); ::archive_entry_set_pathname(entry, rebased.c_str()); @@ -272,7 +299,9 @@ extract_archive(const std::filesystem::path& archive, auto safeHl = detail_::check_safe_pathname_(hardlink.c_str()); if (!safeHl) { cleanup(); - return std::unexpected(std::move(safeHl).error()); + return std::unexpected(ExtractError{ + ExtractErrorKind::InvalidInputArchive, + std::move(safeHl).error()}); } auto rebasedHl = (canonicalDest / *safeHl).lexically_normal().string(); ::archive_entry_set_hardlink(entry, rebasedHl.c_str()); @@ -303,7 +332,8 @@ extract_archive(const std::filesystem::path& archive, "write_header({}): {}", rebased, detail_::libarchive_error_(dst)); cleanup(); - return std::unexpected(std::move(err)); + return std::unexpected(ExtractError{ + ExtractErrorKind::LocalWriteFailure, std::move(err)}); } } @@ -319,7 +349,8 @@ extract_archive(const std::filesystem::path& archive, "finish_entry({}): {}", rebased, detail_::libarchive_error_(dst)); cleanup(); - return std::unexpected(std::move(err)); + return std::unexpected(ExtractError{ + ExtractErrorKind::LocalWriteFailure, std::move(err)}); } } @@ -327,4 +358,12 @@ extract_archive(const std::filesystem::path& archive, return canonicalDest; } +std::expected +extract_archive(const std::filesystem::path& archive, + const std::filesystem::path& destDir) { + auto result = extract_archive_detailed(archive, destDir); + if (!result) return std::unexpected(std::move(result).error().message); + return *result; +} + } // namespace xlings::xim diff --git a/src/core/xim/installer.cppm b/src/core/xim/installer.cppm index 0c0a3b60..6aa88789 100644 --- a/src/core/xim/installer.cppm +++ b/src/core/xim/installer.cppm @@ -2,8 +2,9 @@ export module xlings.core.xim.installer; import std; import mcpplibs.xpkg; +import mcpplibs.xpkg.loader; +import mcpplibs.xpkg.compat; import mcpplibs.xpkg.executor; -import mcpplibs.capi.lua; import xlings.core.xim.libxpkg.types.type; import xlings.core.xim.index; import xlings.core.xim.catalog; @@ -25,9 +26,20 @@ import xlings.runtime.cancellation; export namespace xlings::xim { -namespace detail_ { +bool evict_invalid_archive_cache_( + const std::filesystem::path& archive, + const ExtractError& error) { + if (error.kind != ExtractErrorKind::InvalidInputArchive) return false; + std::error_code ec; + bool removed = std::filesystem::remove(archive, ec); + auto sidecar = archive; + sidecar += ".meta"; + ec.clear(); + std::filesystem::remove(sidecar, ec); + return removed; +} -namespace lua = mcpplibs::capi::lua; +namespace detail_ { std::string effective_store_name_(std::string_view namespaceName, std::string_view name) { return package_store_name(namespaceName, name); @@ -167,36 +179,6 @@ std::string build_xlings_res_url_(std::string_view pkgName, return build_xlings_res_url_with_server_(default_res_server_(), pkgName, version, platform); } -// Expand a V2 URL template. Supported placeholders: -// ${name} ${version} ${os} ${arch} ${arch_alias} ${ext} -// `arch` is the canonical host arch (x86_64/aarch64); `arch_alias` maps it -// to an upstream token (falls back to the canonical arch when unmapped). -// `ext` mirrors the XLINGS_RES convention: zip on windows, tar.gz elsewhere. -std::string expand_url_template_(std::string tmpl, - std::string_view name, - std::string_view version, - std::string_view platform, - std::string_view arch, - const std::unordered_map& arch_alias) { - std::string alias{arch}; - if (auto it = arch_alias.find(std::string(arch)); it != arch_alias.end()) - alias = it->second; - std::string ext = (std::string(platform) == "windows") ? "zip" : "tar.gz"; - auto sub = [&](std::string_view key, std::string_view val) { - std::string needle = std::string("${") + std::string(key) + "}"; - for (auto pos = tmpl.find(needle); pos != std::string::npos; - pos = tmpl.find(needle, pos + val.size())) - tmpl.replace(pos, needle.size(), val); - }; - sub("name", name); - sub("version", version); - sub("os", platform); - sub("arch", arch); - sub("arch_alias", alias); - sub("ext", ext); - return tmpl; -} - // Build fallback URLs from all candidate resource servers (excluding the selected one) std::vector build_xlings_res_fallback_urls_(std::string_view pkgName, std::string_view version, @@ -213,10 +195,57 @@ std::vector build_xlings_res_fallback_urls_(std::string_view pkgNam return fallbacks; } +struct DownloadResource_ { + std::string version; + std::string url; + std::string sha256; + std::unordered_map mirrors; + bool useResFallbacks { false }; +}; + +std::expected resolve_download_resource_( + const mcpplibs::xpkg::PlatformMatrix& matrix, + std::string_view name, + std::string_view requestedVersion, + std::string_view platform, + std::string_view arch, + std::string_view preferredMirror) { + auto resolved = mcpplibs::xpkg::resolve_resource(matrix, { + .name = std::string(name), + .version = std::string(requestedVersion), + .platform = std::string(platform), + .arch = std::string(arch), + }); + if (!resolved) return std::unexpected(resolved.error()); + + DownloadResource_ result { + .version = resolved->version, + .url = resolved->url, + .sha256 = resolved->sha256, + .mirrors = resolved->mirrors, + .useResFallbacks = resolved->kind + == mcpplibs::xpkg::SourceKind::XlingsRes, + }; + if (result.useResFallbacks) { + result.url = build_xlings_res_url_(name, result.version, platform); + } + + auto preferred = preferredMirror.empty() + ? std::string_view{"GLOBAL"} + : preferredMirror; + if (auto it = result.mirrors.find(std::string(preferred)); + it != result.mirrors.end()) { + result.url = it->second; + } + if (result.url.empty()) + return std::unexpected("resolved resource URL is empty"); + return result; +} + bool has_directory_entries_(const std::filesystem::path& dir) { std::error_code ec; if (!std::filesystem::exists(dir, ec) || !std::filesystem::is_directory(dir, ec)) return false; - return std::filesystem::directory_iterator(dir, ec) != std::filesystem::directory_iterator{}; + return std::filesystem::directory_iterator(dir, ec) != std::default_sentinel; } bool stage_extracted_payload_(const std::filesystem::path& extractRoot, @@ -226,7 +255,8 @@ bool stage_extracted_payload_(const std::filesystem::path& extractRoot, if (!fs::exists(extractRoot, ec) || !fs::is_directory(extractRoot, ec)) return false; std::vector entries; - for (fs::directory_iterator it(extractRoot, ec), end; !ec && it != end; it.increment(ec)) { + for (fs::directory_iterator it(extractRoot, ec); + !ec && it != std::default_sentinel; it.increment(ec)) { entries.push_back(it->path()); } if (ec || entries.empty()) return false; @@ -280,7 +310,8 @@ bool stage_extracted_payload_(const std::filesystem::path& extractRoot, } std::vector payloadEntries; - for (fs::directory_iterator it(payloadRoot, ec), end; !ec && it != end; it.increment(ec)) { + for (fs::directory_iterator it(payloadRoot, ec); + !ec && it != std::default_sentinel; it.increment(ec)) { payloadEntries.push_back(it->path()); } if (ec) return false; @@ -805,224 +836,6 @@ bool run_config_hook_(const PlanNode& node, return true; } -bool register_platform_loader_sandbox_(lua::State* L, const std::string& platform) { - auto quoted = "'" + platform + "'"; - auto script = - "import = function(...) return setmetatable({}, { __index = function() return function() end end }) end\n" - "function is_host(name) return name == " + quoted + " end\n" - "format = string.format\n" - "_RUNTIME = { platform = " + quoted + " }\n" - "os.host = function() return " + quoted + " end\n" - "os.arch = os.arch or function() return 'arm64' end\n" - "os.isfile = os.isfile or function() return false end\n" - "os.isdir = os.isdir or function() return false end\n" - "os.scriptdir = os.scriptdir or function() return '.' end\n" - "os.dirs = os.dirs or function() return {} end\n" - "os.files = os.files or function() return {} end\n" - "os.exists = os.exists or function() return false end\n" - "os.tryrm = os.tryrm or function() end\n" - "os.trymv = os.trymv or function() end\n" - "os.mv = os.mv or function() return true end\n" - "os.cp = os.cp or function() return true end\n" - "os.iorun = os.iorun or function() return nil end\n" - "os.cd = os.cd or function() end\n" - "os.mkdir = os.mkdir or function() end\n" - "os.sleep = os.sleep or function() end\n" - "path = path or {}\n" - "path.join = path.join or function(...) " - " local parts = {} " - " for i = 1, select('#', ...) do " - " local v = select(i, ...) " - " if v ~= nil then parts[#parts+1] = tostring(v) end " - " end " - " return table.concat(parts, '/') " - "end\n" - "path.filename = path.filename or function(p) return type(p)=='string' and (p:match('[^/\\\\]+$') or p) or '' end\n" - "path.directory = path.directory or function(p) return type(p)=='string' and (p:match('(.*)[/\\\\]') or '.') or '.' end\n" - "path.basename = path.basename or function(p) return type(p)=='string' and (p:match('[^/\\\\]+$') or p) or '' end\n" - "io.readfile = io.readfile or function() return '' end\n" - "io.writefile = io.writefile or function() end\n" - "try = try or function(block) pcall(block[1]) end\n" - "cprint = cprint or print\n" - "string.replace = string.replace or function(s, old, new) return s:gsub(old, new) end\n" - "string.split = string.split or function(s, sep) " - " local r = {} " - " for m in (s .. sep):gmatch('(.-)' .. sep) do r[#r+1] = m end " - " return r " - "end\n" - "raise = raise or function() end\n" - "runtime = setmetatable({}, { __index = function() return function() return '' end end })\n" - "system = setmetatable({}, { __index = function() return function() return '' end end })\n" - "libxpkg = setmetatable({}, { __index = function() return setmetatable({}, { __index = function() return function() return '' end end }) end })\n"; - - return lua::L_dostring(L, script.c_str()) == lua::OK; -} - -std::unordered_map -load_platform_entries_(const std::filesystem::path& pkgFile, const std::string& platform) { - std::unordered_map entries; - - auto* L = lua::L_newstate(); - if (!L) return entries; - lua::L_openlibs(L); - - auto closeLua = [&]() { - if (L) { - lua::close(L); - L = nullptr; - } - }; - - if (!register_platform_loader_sandbox_(L, platform)) { - closeLua(); - return entries; - } - if (lua::L_dofile(L, pkgFile.string().c_str()) != lua::OK) { - closeLua(); - return entries; - } - - lua::getglobal(L, "package"); - if (lua::type(L, -1) != lua::TTABLE) { - closeLua(); - return entries; - } - - auto packageIdx = lua::gettop(L); - lua::getfield(L, packageIdx, "xpm"); - if (lua::type(L, -1) != lua::TTABLE) { - closeLua(); - return entries; - } - - auto xpmIdx = lua::gettop(L); - lua::getfield(L, xpmIdx, platform.c_str()); - if (lua::type(L, -1) != lua::TTABLE) { - closeLua(); - return entries; - } - - auto platformIdx = lua::gettop(L); - lua::pushnil(L); - while (lua::next(L, platformIdx)) { - std::string version; - if (lua::type(L, -2) == lua::TSTRING) version = lua::tostring(L, -2); - if (!version.empty() && version != "deps" && version != "inherits") { - mcpplibs::xpkg::PlatformResource res; - if (lua::type(L, -1) == lua::TTABLE) { - auto read_field = [&](const char* key) -> std::string { - lua::getfield(L, -1, key); - std::string val; - if (lua::type(L, -1) == lua::TSTRING) val = lua::tostring(L, -1); - lua::pop(L, 1); - return val; - }; - res.url = read_field("url"); - // Handle url table: { GLOBAL = "...", CN = "..." } - if (res.url.empty()) { - lua::getfield(L, -1, "url"); - if (lua::type(L, -1) == lua::TTABLE) { - lua::pushnil(L); - while (lua::next(L, -2)) { - if (lua::type(L, -2) == lua::TSTRING && lua::type(L, -1) == lua::TSTRING) - res.mirrors[lua::tostring(L, -2)] = lua::tostring(L, -1); - lua::pop(L, 1); - } - if (auto it = res.mirrors.find("GLOBAL"); it != res.mirrors.end()) - res.url = it->second; - else if (!res.mirrors.empty()) - res.url = res.mirrors.begin()->second; - } - lua::pop(L, 1); - } - res.sha256 = read_field("sha256"); - res.ref = read_field("ref"); - - // ---- V2 multi-arch shapes (mirrors libxpkg xpkg-loader) ---- - int resIdx = lua::gettop(L); // resource table (absolute index) - - auto read_str_map = [&](const char* key, - std::unordered_map& out, - bool canon_keys) { - lua::getfield(L, resIdx, key); - if (lua::type(L, -1) == lua::TTABLE) { - lua::pushnil(L); - while (lua::next(L, -2)) { - if (lua::type(L, -2) == lua::TSTRING && lua::type(L, -1) == lua::TSTRING) { - std::string k = lua::tostring(L, -2); - out[canon_keys ? mcpplibs::xpkg::normalize_arch(k) : k] = - lua::tostring(L, -1); - } - lua::pop(L, 1); - } - } - lua::pop(L, 1); - }; - - // Scheme C / res: `sha256` is a per-arch TABLE (string read above - // returned ""). arch_alias is the optional ${arch_alias} mapping. - read_str_map("sha256", res.sha256_by_arch, true); - read_str_map("arch_alias", res.arch_alias, true); - - lua::getfield(L, resIdx, "res"); - res.is_res = lua::toboolean(L, -1); - lua::pop(L, 1); - - // Scheme B: per-arch resource map. Detected when no single-arch - // url/ref/sha256 and no template/res markers are present. - if (res.url.empty() && res.ref.empty() && res.sha256.empty() - && res.sha256_by_arch.empty() && !res.is_res) { - lua::pushnil(L); - while (lua::next(L, resIdx)) { - if (lua::type(L, -2) == lua::TSTRING && lua::type(L, -1) == lua::TTABLE) { - std::string canon = - mcpplibs::xpkg::normalize_arch(lua::tostring(L, -2)); - if (canon == "x86_64" || canon == "aarch64" || canon == "x86") { - int archIdx = lua::gettop(L); - mcpplibs::xpkg::ArchResource ar; - lua::getfield(L, archIdx, "url"); - if (lua::type(L, -1) == lua::TSTRING) ar.url = lua::tostring(L, -1); - lua::pop(L, 1); - if (ar.url.empty()) { - lua::getfield(L, archIdx, "url"); - if (lua::type(L, -1) == lua::TTABLE) { - lua::pushnil(L); - while (lua::next(L, -2)) { - if (lua::type(L, -2) == lua::TSTRING - && lua::type(L, -1) == lua::TSTRING) - ar.mirrors[lua::tostring(L, -2)] = - lua::tostring(L, -1); - lua::pop(L, 1); - } - if (auto it = ar.mirrors.find("GLOBAL"); - it != ar.mirrors.end()) - ar.url = it->second; - else if (!ar.mirrors.empty()) - ar.url = ar.mirrors.begin()->second; - } - lua::pop(L, 1); - } - lua::getfield(L, archIdx, "sha256"); - if (lua::type(L, -1) == lua::TSTRING) ar.sha256 = lua::tostring(L, -1); - lua::pop(L, 1); - res.archs[canon] = std::move(ar); - } - } - lua::pop(L, 1); - } - } - } else if (lua::type(L, -1) == lua::TSTRING) { - res.url = lua::tostring(L, -1); - } - entries[version] = std::move(res); - } - lua::pop(L, 1); - } - - closeLua(); - return entries; -} - } // namespace detail_ using InstallRequestHandler = std::function&)>; @@ -1093,21 +906,12 @@ public: for (auto& node : plan.nodes) { if (node.alreadyInstalled) continue; - std::expected pkg = - catalog_ - ? catalog_->load_package(PackageMatch{ - .rawName = node.rawName, - .name = node.name, - .version = node.version, - .namespaceName = node.namespaceName, - .canonicalName = node.canonicalName, - .repoName = node.repoName, - .pkgFile = node.pkgFile, - .storeRoot = node.storeRoot, - .scope = node.scope, - .installed = node.alreadyInstalled, - }) - : index_->load_package(node.name); + const std::string hostArch = + mcpplibs::xpkg::normalize_arch(detail_::detect_arch_()); + auto pkg = mcpplibs::xpkg::load_package(node.pkgFile, { + .platform = platform, + .arch = hostArch, + }); if (!pkg) { log::warn("skipping {}: {}", node.name, pkg.error()); continue; @@ -1136,99 +940,33 @@ public: } } - auto platformEntries = detail_::load_platform_entries_(node.pkgFile, platform); - if (platformEntries.empty()) { - auto platformIt = pkg->xpm.entries.find(platform); - if (platformIt != pkg->xpm.entries.end()) { - platformEntries = platformIt->second; - } - } - if (platformEntries.empty()) continue; - - std::string version = node.version; - // Follow ref chain - auto verIt = platformEntries.find(version); - if (verIt != platformEntries.end() && !verIt->second.ref.empty()) { - version = verIt->second.ref; - verIt = platformEntries.find(version); - } - - if (verIt == platformEntries.end()) continue; - - auto& res = verIt->second; - - // ---- V2 install-time arch resolution ---- - // Resolve the host arch (canonical x86_64/aarch64). When all V2 - // fields are empty the legacy single-arch path below is untouched. - const std::string hostArch = - mcpplibs::xpkg::normalize_arch(detail_::detect_arch_()); - bool resGenerated = false; - if (!res.archs.empty()) { - // Scheme B: per-arch resource map (fail-closed on miss). - auto ait = res.archs.find(hostArch); - if (ait == res.archs.end()) { - log::warn("skipping {}: no resource for arch '{}' in version {}", - node.name, hostArch, version); - continue; - } - res.url = ait->second.url; - res.sha256 = ait->second.sha256; - res.mirrors = ait->second.mirrors; - } else if (res.is_res) { - // res shape: XLINGS_RES auto-URL + per-arch checksum. - auto sit = res.sha256_by_arch.find(hostArch); - if (sit == res.sha256_by_arch.end()) { - log::warn("skipping {}: no checksum for arch '{}' in version {}", - node.name, hostArch, version); - continue; - } - res.url = detail_::build_xlings_res_url_(node.name, version, platform); - res.sha256 = sit->second; - resGenerated = true; - } else if (!res.sha256_by_arch.empty()) { - // Scheme C: URL template + per-arch checksum. - auto sit = res.sha256_by_arch.find(hostArch); - if (sit == res.sha256_by_arch.end()) { - log::warn("skipping {}: no checksum for arch '{}' in version {}", - node.name, hostArch, version); - continue; - } - res.url = detail_::expand_url_template_( - res.url, node.name, version, platform, hostArch, res.arch_alias); - res.sha256 = sit->second; - } - - bool isXlingsRes = (res.url == "XLINGS_RES"); - if (isXlingsRes) { - res.url = detail_::build_xlings_res_url_(node.name, version, platform); - } - bool useResFallbacks = isXlingsRes || resGenerated; - - // Mirror selection: prefer dlConfig.preferredMirror, fallback others - if (!res.mirrors.empty()) { - auto preferred = dlConfig.preferredMirror.empty() ? "GLOBAL" : dlConfig.preferredMirror; - if (auto it = res.mirrors.find(preferred); it != res.mirrors.end()) { - res.url = it->second; - } + auto resource = detail_::resolve_download_resource_( + pkg->xpm, node.name, node.version, platform, hostArch, + dlConfig.preferredMirror); + if (!resource) { + log::warn("skipping {}: {}", node.name, resource.error()); + continue; } - if (res.url.empty()) continue; - DownloadTask task; task.name = detail_::plan_key_(node); - task.url = res.url; - task.sha256 = res.sha256; + task.url = resource->url; + task.sha256 = resource->sha256; + task.cacheIdentity = std::format( + "{}/{}/{}/{}/{}", + node.name, resource->version, platform, hostArch, + resource->useResFallbacks ? "xlings-res" : resource->url); task.destDir = detail_::runtime_dir_(node, dataDir); - if (useResFallbacks) { + if (resource->useResFallbacks) { task.fallbackUrls = detail_::build_xlings_res_fallback_urls_( - node.name, version, platform); + node.name, resource->version, platform); } // Add remaining mirrors as fallbacks - if (!res.mirrors.empty()) { - for (auto& [key, mirrorUrl] : res.mirrors) { - if (mirrorUrl != res.url) { + if (!resource->mirrors.empty()) { + for (auto& [key, mirrorUrl] : resource->mirrors) { + if (mirrorUrl != resource->url) { task.fallbackUrls.push_back(mirrorUrl); } } @@ -1383,11 +1121,22 @@ public: } // Extract into the same runtime dir as the download auto runtimeDir = dlIt->second.localFile.parent_path(); - auto extracted = extract_archive(dlIt->second.localFile, runtimeDir); + auto extracted = extract_archive_detailed( + dlIt->second.localFile, runtimeDir); if (!extracted) { - log::error("extract failed for {}: {}", node.name, extracted.error()); + auto error = std::move(extracted).error(); + if (evict_invalid_archive_cache_( + dlIt->second.localFile, error)) { + log::warn( + "evicted invalid archive cache for {}: {}", + node.name, dlIt->second.localFile.string()); + } + log::error("extract failed for {}: {}", + node.name, error.message); if (onStatus) { - onStatus({ node.name, InstallPhase::Failed, 0.0f, extracted.error() }); + onStatus({ + node.name, InstallPhase::Failed, 0.0f, + error.message}); } continue; } @@ -2038,7 +1787,7 @@ public: auto parent = installDir.parent_path(); std::error_code listEc; auto first = std::filesystem::directory_iterator(parent, listEc); - if (!listEc && first == std::filesystem::directory_iterator{}) { + if (!listEc && first == std::default_sentinel) { std::error_code rmEc; if (std::filesystem::remove(parent, rmEc)) { log::debug("swept empty package dir: {}", parent.string()); diff --git a/src/core/xim/libxpkg/types/type.cppm b/src/core/xim/libxpkg/types/type.cppm index 4d46f5e1..5befe3fd 100644 --- a/src/core/xim/libxpkg/types/type.cppm +++ b/src/core/xim/libxpkg/types/type.cppm @@ -146,6 +146,7 @@ struct DownloadTask { std::string name; std::string url; std::string sha256; + std::string cacheIdentity; std::filesystem::path destDir; std::vector fallbackUrls; // tried in order when url fails }; diff --git a/src/libs/tinyhttps.cppm b/src/libs/tinyhttps.cppm index 49fcbfb3..1bc5c4a6 100644 --- a/src/libs/tinyhttps.cppm +++ b/src/libs/tinyhttps.cppm @@ -11,6 +11,11 @@ export namespace xlings::tinyhttps { struct DownloadFileResult { bool success { false }; std::string error; + std::int64_t bytesWritten { 0 }; + std::optional expectedBytes; + std::string finalUrl; + std::string etag; + std::string lastModified; }; struct DownloadOptions { @@ -284,7 +289,14 @@ DownloadFileResult download_once( auto result = client.download_to_file(url, dest, progress, cancel); if (result.ok() && !stalled) { - return {true, {}}; + return { + .success = true, + .bytesWritten = result.bytesWritten, + .expectedBytes = result.expectedBytes, + .finalUrl = result.finalUrl, + .etag = result.etag, + .lastModified = result.lastModified, + }; } if (stalled) { return {false, std::format( @@ -292,8 +304,16 @@ DownloadFileResult download_once( "(set XLINGS_DOWNLOAD_LOW_SPEED=off to disable the watchdog)", lowSpeedLimitBytes, lowSpeedTimeSec)}; } - return {false, result.error.empty() - ? "HTTP " + std::to_string(result.statusCode) : result.error}; + return { + .success = false, + .error = result.error.empty() + ? "HTTP " + std::to_string(result.statusCode) : result.error, + .bytesWritten = result.bytesWritten, + .expectedBytes = result.expectedBytes, + .finalUrl = result.finalUrl, + .etag = result.etag, + .lastModified = result.lastModified, + }; } } // namespace detail_ diff --git a/src/platform.cppm b/src/platform.cppm index 8677d96c..1525f17d 100644 --- a/src/platform.cppm +++ b/src/platform.cppm @@ -41,6 +41,7 @@ namespace platform { export using platform_impl::Icon; export using platform_impl::atomic_replace_executable; export using platform_impl::atomic_swap_paths; + export using platform_impl::FileLock; // ── Execution identity (root / sudo awareness) ────────────────── // Single source of truth for "who am I / who should own the files I @@ -123,12 +124,11 @@ namespace platform { if (!std::filesystem::exists(path, ec)) return; platform_impl::lchown_path_(path, inv->uid, inv->gid); if (recursive && std::filesystem::is_directory(path, ec)) { - auto end = std::filesystem::recursive_directory_iterator{}; for (auto it = std::filesystem::recursive_directory_iterator( path, std::filesystem::directory_options::skip_permission_denied, ec); - !ec && it != end; it.increment(ec)) { + !ec && it != std::default_sentinel; it.increment(ec)) { platform_impl::lchown_path_(it->path(), inv->uid, inv->gid); } } diff --git a/src/platform/unix.cppm b/src/platform/unix.cppm index f59d2baa..c3e23467 100644 --- a/src/platform/unix.cppm +++ b/src/platform/unix.cppm @@ -2,12 +2,14 @@ module; #include #include +#include #if defined(__linux__) || defined(__APPLE__) #include #include #include #include #include +#include #include #endif #if defined(__linux__) @@ -26,6 +28,67 @@ import std; namespace xlings { namespace platform_impl { + export class FileLock { + public: + FileLock() = default; + FileLock(const FileLock&) = delete; + FileLock& operator=(const FileLock&) = delete; + FileLock(FileLock&& other) noexcept : fd_(std::exchange(other.fd_, -1)) {} + FileLock& operator=(FileLock&& other) noexcept { + if (this != &other) { + release(); + fd_ = std::exchange(other.fd_, -1); + } + return *this; + } + ~FileLock() { release(); } + + bool acquire(const std::filesystem::path& path, + std::chrono::milliseconds timeout, + const std::function& cancelled, + std::string& error) { + release(); + fd_ = ::open(path.c_str(), O_CREAT | O_RDWR, 0600); + if (fd_ < 0) { + error = std::format("failed to open lock {}: {}", + path.string(), std::strerror(errno)); + return false; + } + auto deadline = std::chrono::steady_clock::now() + timeout; + while (::flock(fd_, LOCK_EX | LOCK_NB) != 0) { + if (errno != EWOULDBLOCK && errno != EAGAIN) { + error = std::format("failed to lock {}: {}", + path.string(), std::strerror(errno)); + release(); + return false; + } + if (cancelled && cancelled()) { + error = "cancelled while waiting for cache lock"; + release(); + return false; + } + if (std::chrono::steady_clock::now() >= deadline) { + error = std::format("timed out waiting for cache lock {}", + path.string()); + release(); + return false; + } + std::this_thread::sleep_for(std::chrono::milliseconds{50}); + } + return true; + } + + void release() { + if (fd_ < 0) return; + ::flock(fd_, LOCK_UN); + ::close(fd_); + fd_ = -1; + } + + private: + int fd_ { -1 }; + }; + // Query the controlling terminal for its background color via the // OSC-11 sequence (xterm spec, supported by xterm / iTerm2 / Alacritty // / Kitty / WezTerm / modern Windows Terminal). Returns std::nullopt diff --git a/src/platform/windows.cppm b/src/platform/windows.cppm index 58cbac96..78970207 100644 --- a/src/platform/windows.cppm +++ b/src/platform/windows.cppm @@ -17,6 +17,66 @@ import xlings.runtime.cancellation; namespace xlings { namespace platform_impl { + export class FileLock { + public: + FileLock() = default; + FileLock(const FileLock&) = delete; + FileLock& operator=(const FileLock&) = delete; + FileLock(FileLock&& other) noexcept + : handle_(std::exchange(other.handle_, INVALID_HANDLE_VALUE)) {} + FileLock& operator=(FileLock&& other) noexcept { + if (this != &other) { + release(); + handle_ = std::exchange(other.handle_, INVALID_HANDLE_VALUE); + } + return *this; + } + ~FileLock() { release(); } + + bool acquire(const std::filesystem::path& path, + std::chrono::milliseconds timeout, + const std::function& cancelled, + std::string& error) { + release(); + auto deadline = std::chrono::steady_clock::now() + timeout; + while (true) { + handle_ = ::CreateFileW( + path.wstring().c_str(), + GENERIC_READ | GENERIC_WRITE, + 0, nullptr, OPEN_ALWAYS, + FILE_ATTRIBUTE_NORMAL, nullptr); + if (handle_ != INVALID_HANDLE_VALUE) return true; + auto code = ::GetLastError(); + if (code != ERROR_SHARING_VIOLATION + && code != ERROR_LOCK_VIOLATION) { + error = std::format( + "failed to lock {}: Windows error {}", + path.string(), code); + return false; + } + if (cancelled && cancelled()) { + error = "cancelled while waiting for cache lock"; + return false; + } + if (std::chrono::steady_clock::now() >= deadline) { + error = std::format("timed out waiting for cache lock {}", + path.string()); + return false; + } + std::this_thread::sleep_for(std::chrono::milliseconds{50}); + } + } + + void release() { + if (handle_ == INVALID_HANDLE_VALUE) return; + ::CloseHandle(handle_); + handle_ = INVALID_HANDLE_VALUE; + } + + private: + HANDLE handle_ { INVALID_HANDLE_VALUE }; + }; + export constexpr char PATH_SEPARATOR = ';'; export constexpr std::string_view OS_NAME = "windows"; diff --git a/tests/unit/test_interface_protocol.cpp b/tests/unit/test_interface_protocol.cpp index 2cdd6735..ee495dae 100644 --- a/tests/unit/test_interface_protocol.cpp +++ b/tests/unit/test_interface_protocol.cpp @@ -41,7 +41,7 @@ std::string find_xlings_binary_() { fs::file_time_type newest_mcpp_time {}; if (fs::exists("target", ec)) { for (auto it = fs::recursive_directory_iterator("target", ec); - !ec && it != fs::recursive_directory_iterator(); it.increment(ec)) { + !ec && it != std::default_sentinel; it.increment(ec)) { if (!it->is_regular_file(ec)) continue; if (it->path().filename() != exe) continue; if (it->path().parent_path().filename() != "bin") continue; diff --git a/tests/unit/test_main.cpp b/tests/unit/test_main.cpp index 9d48dc85..fa749dfe 100644 --- a/tests/unit/test_main.cpp +++ b/tests/unit/test_main.cpp @@ -33,6 +33,7 @@ import xlings.core.xim.downloader; import xlings.runtime; import xlings.capabilities; import xlings.libs.tinyhttps; +import xlings.libs.sha256; import mcpplibs.xpkg; import mcpplibs.cmdline; @@ -844,9 +845,16 @@ TEST(XimDownloaderTest, MetaSidecarRoundTrip) { meta.lastModified = "Wed, 21 Oct 2015 07:28:00 GMT"; meta.etag = "\"abc123\""; - xlings::xim::write_meta_sidecar_(path, meta); + ASSERT_TRUE(xlings::xim::write_meta_sidecar_( + path, meta, 1234, + "test/1.0.0/linux/x86_64/url", + "https://example.test/payload.tar.gz")); auto roundtrip = xlings::xim::read_meta_sidecar_(path); ASSERT_TRUE(roundtrip.has_value()); + EXPECT_EQ(roundtrip->format, 2); + EXPECT_TRUE(roundtrip->complete); + EXPECT_EQ(roundtrip->size, 1234); + EXPECT_EQ(roundtrip->cacheIdentity, "test/1.0.0/linux/x86_64/url"); EXPECT_EQ(roundtrip->lastModified, meta.lastModified); EXPECT_EQ(roundtrip->etag, meta.etag); @@ -869,6 +877,465 @@ TEST(XimDownloaderTest, MetaSidecarRoundTrip) { fs::remove_all(tmp); } +TEST(XimDownloaderTest, HeadFailureRejectsLegacyNonEmptyCache) { + xlings::xim::CacheAdmissionInput_ input { + .localSize = 114 * 1024, + .headSucceeded = false, + .sidecar = xlings::xim::MetaSidecar_ { + .lastModified = "Wed, 21 Oct 2015 07:28:00 GMT", + }, + .expectedCacheIdentity = "mcpp/0.0.81/linux/x86_64/xlings-res", + }; + + EXPECT_EQ( + xlings::xim::decide_cache_admission_(input), + xlings::xim::CacheAdmission_::Redownload); +} + +TEST(XimDownloaderTest, HeadFailureAcceptsMatchingCommittedV2Cache) { + xlings::xim::CacheAdmissionInput_ input { + .localSize = 12628937, + .headSucceeded = false, + .sidecar = xlings::xim::MetaSidecar_ { + .format = 2, + .complete = true, + .size = 12628937, + .cacheIdentity = "mcpp/0.0.81/linux/x86_64/xlings-res", + }, + .expectedCacheIdentity = "mcpp/0.0.81/linux/x86_64/xlings-res", + }; + + EXPECT_EQ( + xlings::xim::decide_cache_admission_(input), + xlings::xim::CacheAdmission_::OfflineUnverifiedHit); +} + +TEST(XimDownloaderTest, HeadFailureRejectsV2CacheWithWrongSizeOrIdentity) { + xlings::xim::CacheAdmissionInput_ input { + .localSize = 114 * 1024, + .headSucceeded = false, + .sidecar = xlings::xim::MetaSidecar_ { + .format = 2, + .complete = true, + .size = 12628937, + .cacheIdentity = "other/0.0.81/linux/x86_64/xlings-res", + }, + .expectedCacheIdentity = "mcpp/0.0.81/linux/x86_64/xlings-res", + }; + + EXPECT_EQ( + xlings::xim::decide_cache_admission_(input), + xlings::xim::CacheAdmission_::Redownload); +} + +TEST(XimDownloaderTest, FailedTransferPreservesCommittedDestination) { + namespace fs = std::filesystem; + auto tmp = fs::temp_directory_path() / "xim_download_transaction_failure"; + fs::remove_all(tmp); + fs::create_directories(tmp); + auto destination = tmp / "payload.tar.gz"; + { + std::ofstream out(destination); + out << "previous-good-payload"; + } + + xlings::xim::DownloadTask task { + .name = "transaction-test", + .url = "https://example.test/payload.tar.gz", + .cacheIdentity = "transaction-test/1/linux/x86_64/url", + .destDir = tmp, + }; + xlings::xim::DownloadTestHooks_ hooks; + hooks.queryRemoteMeta = [](const std::string&) { + return xlings::tinyhttps::RemoteFileMeta{ + .ok = true, + .contentLength = 999, + }; + }; + hooks.transferOverride = [](const std::string&, const fs::path& path) { + std::ofstream(path) << "partial"; + return xlings::tinyhttps::DownloadFileResult{false, "connection reset"}; + }; + + auto result = xlings::xim::download_one(task, nullptr, nullptr, &hooks); + EXPECT_FALSE(result.success); + EXPECT_EQ(xlings::platform::read_file_to_string(destination.string()), + "previous-good-payload"); + for (const auto& entry : fs::directory_iterator(tmp)) { + EXPECT_FALSE(entry.path().filename().string().contains(".part.")); + } + fs::remove_all(tmp); +} + +TEST(XimDownloaderTest, HashRejectedCandidateCommitsFallbackFromStaging) { + namespace fs = std::filesystem; + auto tmp = fs::temp_directory_path() / "xim_download_transaction_fallback"; + fs::remove_all(tmp); + fs::create_directories(tmp); + + const std::string goodPayload = "fallback-good-payload"; + auto expectedHash = xlings::sha256::hex(goodPayload); + xlings::xim::DownloadTask task { + .name = "fallback-test", + .url = "https://mirror.test/payload.bin", + .sha256 = expectedHash, + .cacheIdentity = "fallback-test/1/linux/x86_64/url", + .destDir = tmp, + .fallbackUrls = {"https://origin.test/payload.bin"}, + }; + int attempts = 0; + xlings::xim::DownloadTestHooks_ hooks; + hooks.transferOverride = [&](const std::string&, const fs::path& path) { + std::ofstream(path) << (++attempts == 1 ? "bad" : goodPayload); + return xlings::tinyhttps::DownloadFileResult{true, {}}; + }; + + auto result = xlings::xim::download_one(task, nullptr, nullptr, &hooks); + ASSERT_TRUE(result.success) << result.error; + EXPECT_EQ(attempts, 2); + EXPECT_EQ(xlings::platform::read_file_to_string(result.localFile.string()), + goodPayload); + for (const auto& entry : fs::directory_iterator(tmp)) { + EXPECT_FALSE(entry.path().filename().string().contains(".part.")); + } + fs::remove_all(tmp); +} + +TEST(TinyhttpsWrapperTest, ReturnsAcceptedCandidateTransferMetadata) { + namespace fs = std::filesystem; + auto tmp = fs::temp_directory_path() / "tinyhttps_transfer_metadata"; + fs::remove_all(tmp); + fs::create_directories(tmp); + + xlings::tinyhttps::DownloadOptions options; + options.destFile = tmp / "payload.bin"; + options.urls = {"https://origin.test/payload.bin"}; + options.retryCount = 0; + options.transferOverride = [](const std::string&, const fs::path& path) { + std::ofstream(path) << "payload"; + return xlings::tinyhttps::DownloadFileResult { + .success = true, + .bytesWritten = 7, + .expectedBytes = 7, + .finalUrl = "https://cdn.test/final.bin", + .etag = "etag-1", + .lastModified = "Sat, 12 Jul 2026 00:00:00 GMT", + }; + }; + + auto result = xlings::tinyhttps::download_file(options); + ASSERT_TRUE(result.success) << result.error; + EXPECT_EQ(result.bytesWritten, 7); + ASSERT_TRUE(result.expectedBytes.has_value()); + EXPECT_EQ(*result.expectedBytes, 7); + EXPECT_EQ(result.finalUrl, "https://cdn.test/final.bin"); + EXPECT_EQ(result.etag, "etag-1"); + fs::remove_all(tmp); +} + +TEST(XimDownloaderTest, RejectsIncompleteReportedTransferBeforeCommit) { + namespace fs = std::filesystem; + auto tmp = fs::temp_directory_path() / "xim_download_incomplete_metadata"; + fs::remove_all(tmp); + fs::create_directories(tmp); + auto destination = tmp / "payload.bin"; + std::ofstream(destination) << "previous-good-payload"; + + xlings::xim::DownloadTask task { + .name = "incomplete-test", + .url = "https://origin.test/payload.bin", + .cacheIdentity = "incomplete-test/1/linux/x86_64/url", + .destDir = tmp, + }; + xlings::xim::DownloadTestHooks_ hooks; + hooks.transferOverride = [](const std::string&, const fs::path& path) { + std::ofstream(path) << "bad"; + return xlings::tinyhttps::DownloadFileResult { + .success = true, + .bytesWritten = 3, + .expectedBytes = 100, + }; + }; + + auto result = xlings::xim::download_one(task, nullptr, nullptr, &hooks); + EXPECT_FALSE(result.success); + EXPECT_NE(result.error.find("wrote 3 of 100 bytes"), std::string::npos); + EXPECT_EQ(xlings::platform::read_file_to_string(destination.string()), + "previous-good-payload"); + fs::remove_all(tmp); +} + +TEST(XimDownloaderTest, PersistsAcceptedGetMetadataInCommittedSidecar) { + namespace fs = std::filesystem; + auto tmp = fs::temp_directory_path() / "xim_download_get_metadata"; + fs::remove_all(tmp); + fs::create_directories(tmp); + + xlings::xim::DownloadTask task { + .name = "metadata-test", + .url = "https://origin.test/payload.bin", + .cacheIdentity = "metadata-test/1/linux/x86_64/url", + .destDir = tmp, + }; + xlings::xim::DownloadTestHooks_ hooks; + hooks.transferOverride = [](const std::string&, const fs::path& path) { + std::ofstream(path) << "payload"; + return xlings::tinyhttps::DownloadFileResult { + .success = true, + .bytesWritten = 7, + .expectedBytes = 7, + .finalUrl = "https://cdn.test/final.bin", + .etag = "etag-get", + .lastModified = "Sat, 12 Jul 2026 00:00:00 GMT", + }; + }; + + auto result = xlings::xim::download_one(task, nullptr, nullptr, &hooks); + ASSERT_TRUE(result.success) << result.error; + auto sidecar = xlings::xim::read_meta_sidecar_(result.localFile.string() + ".meta"); + ASSERT_TRUE(sidecar.has_value()); + EXPECT_EQ(sidecar->format, 2); + EXPECT_TRUE(sidecar->complete); + EXPECT_EQ(sidecar->size, 7); + EXPECT_EQ(sidecar->etag, "etag-get"); + EXPECT_EQ(sidecar->sourceUrl, "https://cdn.test/final.bin"); + EXPECT_EQ(sidecar->cacheIdentity, task.cacheIdentity); + fs::remove_all(tmp); +} + +TEST(XimDownloaderTest, CancelledDownloadPreservesCommittedDestination) { + namespace fs = std::filesystem; + auto tmp = fs::temp_directory_path() / "xim_download_transaction_cancel"; + fs::remove_all(tmp); + fs::create_directories(tmp); + auto destination = tmp / "payload.bin"; + std::ofstream(destination) << "previous-good-payload"; + + xlings::xim::DownloadTask task { + .name = "cancel-test", + .url = "https://example.test/payload.bin", + .cacheIdentity = "cancel-test/1/linux/x86_64/url", + .destDir = tmp, + }; + int transfers = 0; + xlings::xim::DownloadTestHooks_ hooks; + hooks.queryRemoteMeta = [](const std::string&) { + return xlings::tinyhttps::RemoteFileMeta{ + .ok = true, + .contentLength = 999, + }; + }; + hooks.transferOverride = [&](const std::string&, const fs::path&) { + ++transfers; + return xlings::tinyhttps::DownloadFileResult{true, {}}; + }; + xlings::CancellationToken cancellation; + cancellation.cancel(); + + auto result = xlings::xim::download_one( + task, nullptr, &cancellation, &hooks); + EXPECT_FALSE(result.success); + EXPECT_EQ(result.error, "cancelled"); + EXPECT_EQ(transfers, 0); + EXPECT_EQ(xlings::platform::read_file_to_string(destination.string()), + "previous-good-payload"); + fs::remove_all(tmp); +} + +TEST(XimDownloaderTest, CommitFailureAfterBackupRestoresPreviousFile) { + namespace fs = std::filesystem; + auto tmp = fs::temp_directory_path() / "xim_download_commit_restore"; + fs::remove_all(tmp); + fs::create_directories(tmp); + auto destination = tmp / "payload.bin"; + auto staging = tmp / "payload.bin.part.test"; + std::ofstream(destination) << "previous-good-payload"; + std::ofstream(staging) << "replacement-payload"; + + std::string error; + EXPECT_FALSE(xlings::xim::commit_staging_file_( + staging, destination, error, true)); + EXPECT_EQ(error, "injected commit failure after backup"); + EXPECT_EQ(xlings::platform::read_file_to_string(destination.string()), + "previous-good-payload"); + EXPECT_TRUE(fs::exists(staging)); + fs::remove_all(tmp); +} + +TEST(XimDownloaderTest, FileLockWaitsForOwnerAndHonorsCancellation) { + namespace fs = std::filesystem; + auto tmp = fs::temp_directory_path() / "xim_download_file_lock"; + fs::remove_all(tmp); + fs::create_directories(tmp); + auto path = tmp / "payload.lock"; + + xlings::platform::FileLock owner; + std::string error; + ASSERT_TRUE(owner.acquire( + path, std::chrono::seconds{1}, {}, error)) << error; + + xlings::platform::FileLock cancelledWaiter; + EXPECT_FALSE(cancelledWaiter.acquire( + path, std::chrono::seconds{1}, [] { return true; }, error)); + EXPECT_EQ(error, "cancelled while waiting for cache lock"); + + std::jthread releaser([&] { + std::this_thread::sleep_for(std::chrono::milliseconds{100}); + owner.release(); + }); + xlings::platform::FileLock waiter; + error.clear(); + auto start = std::chrono::steady_clock::now(); + ASSERT_TRUE(waiter.acquire( + path, std::chrono::seconds{1}, {}, error)) << error; + EXPECT_GE( + std::chrono::steady_clock::now() - start, + std::chrono::milliseconds{50}); + waiter.release(); + releaser.join(); + fs::remove_all(tmp); +} + +TEST(XimDownloaderTest, FileLockSerializesIndependentProcesses) { + namespace fs = std::filesystem; + auto tmp = fs::temp_directory_path() + / std::format("xim_download_process_lock_{}", xlings::platform::get_pid()); + fs::remove_all(tmp); + fs::create_directories(tmp); + auto lock_path = tmp / "payload.lock"; + auto ready_path = tmp / "child.ready"; + auto executable = xlings::platform::get_executable_path(); + ASSERT_FALSE(executable.empty()); + + auto command = std::format( + "\"{}\" --file-lock-child \"{}\" \"{}\"", + executable.string(), lock_path.string(), ready_path.string()); +#ifdef _WIN32 + // spawn_command invokes cmd.exe /c. Its parser removes one outer quote + // pair, so preserve the quotes around the executable and arguments by + // wrapping the complete command once more. + command = "\"" + command + "\""; +#endif + auto child = xlings::platform::spawn_command(command); + ASSERT_GT(child.pid, 0); + + auto ready_deadline = std::chrono::steady_clock::now() + + std::chrono::seconds{3}; + while (!fs::exists(ready_path) + && std::chrono::steady_clock::now() < ready_deadline) { + std::this_thread::sleep_for(std::chrono::milliseconds{10}); + } + + xlings::platform::FileLock waiter; + std::string error; + auto start = std::chrono::steady_clock::now(); + bool acquired = false; + if (fs::exists(ready_path)) { + acquired = waiter.acquire( + lock_path, std::chrono::seconds{2}, {}, error); + } + auto waited = std::chrono::steady_clock::now() - start; + auto [child_status, child_output] = xlings::platform::wait_or_kill( + child, nullptr, std::chrono::seconds{3}); + + EXPECT_TRUE(fs::exists(ready_path)) << child_output; + EXPECT_TRUE(acquired) << error << "\n" << child_output; + EXPECT_GE(waited, std::chrono::milliseconds{150}); + EXPECT_EQ(child_status, 0) << child_output; + waiter.release(); + fs::remove_all(tmp); +} + +TEST(XimInstallerResourceTest, ResolvesXlingsResSourceAndFinalRefVersion) { + mcpplibs::xpkg::PlatformMatrix matrix; + matrix.source = "xlings-res"; + matrix.entries["linux"]["latest"].ref = "1.0.0"; + matrix.entries["linux"]["1.0.0"].sha256_by_arch["x86_64"] = "hash-x86"; + + auto resolved = xlings::xim::detail_::resolve_download_resource_( + matrix, "tool", "latest", "linux", "amd64", "GLOBAL"); + ASSERT_TRUE(resolved.has_value()) << resolved.error(); + EXPECT_EQ(resolved->version, "1.0.0"); + EXPECT_EQ(resolved->sha256, "hash-x86"); + EXPECT_TRUE(resolved->useResFallbacks); + EXPECT_NE(resolved->url.find("/tool/releases/download/1.0.0/"), + std::string::npos); +} + +TEST(XimInstallerResourceTest, ResolvesTemplateAliasAndPreferredMirror) { + mcpplibs::xpkg::PlatformMatrix matrix; + matrix.source = "https://origin.test/${version}/tool-${arch_alias}.${ext}"; + auto& resource = matrix.entries["linux"]["2.0.0"]; + resource.sha256_by_arch["x86_64"] = "hash-x86"; + resource.arch_alias["x86_64"] = "amd64"; + resource.mirrors["CN"] = "https://cn.test/${version}/tool-${arch_alias}.tar.gz"; + + auto resolved = xlings::xim::detail_::resolve_download_resource_( + matrix, "tool", "2.0.0", "linux", "x86_64", "CN"); + ASSERT_TRUE(resolved.has_value()) << resolved.error(); + EXPECT_EQ(resolved->url, "https://cn.test/2.0.0/tool-amd64.tar.gz"); + EXPECT_EQ(resolved->sha256, "hash-x86"); + EXPECT_FALSE(resolved->useResFallbacks); +} + +TEST(XimInstallerResourceTest, PreservesLegacyXlingsResAndFailsClosedOnArchMiss) { + mcpplibs::xpkg::PlatformMatrix legacy; + legacy.entries["linux"]["1.0.0"].url = "XLINGS_RES"; + auto resolved = xlings::xim::detail_::resolve_download_resource_( + legacy, "legacy", "1.0.0", "linux", "x86_64", "GLOBAL"); + ASSERT_TRUE(resolved.has_value()) << resolved.error(); + EXPECT_TRUE(resolved->useResFallbacks); + + mcpplibs::xpkg::PlatformMatrix per_arch; + per_arch.entries["linux"]["1.0.0"].archs["x86_64"] = { + .url = "https://example.test/x86.tar.gz", + .sha256 = "hash-x86", + }; + auto missing = xlings::xim::detail_::resolve_download_resource_( + per_arch, "tool", "1.0.0", "linux", "aarch64", "GLOBAL"); + EXPECT_FALSE(missing.has_value()); +} + +TEST(XimDownloaderTest, RecoveryRestoresBackupWhenLiveIsMissing) { + namespace fs = std::filesystem; + auto tmp = fs::temp_directory_path() / "xim_download_recover_backup"; + fs::remove_all(tmp); + fs::create_directories(tmp); + auto destination = tmp / "payload.bin"; + auto backup = tmp / "payload.bin.old.100.1"; + auto staging = tmp / "payload.bin.part.100.1"; + std::ofstream(backup) << "previous-good-payload"; + std::ofstream(staging) << "partial"; + + std::string error; + ASSERT_TRUE(xlings::xim::recover_download_transaction_( + destination, error)) << error; + EXPECT_EQ(xlings::platform::read_file_to_string(destination.string()), + "previous-good-payload"); + EXPECT_FALSE(fs::exists(backup)); + EXPECT_FALSE(fs::exists(staging)); + fs::remove_all(tmp); +} + +TEST(XimDownloaderTest, RecoveryKeepsLiveAndRemovesStaleBackup) { + namespace fs = std::filesystem; + auto tmp = fs::temp_directory_path() / "xim_download_recover_live"; + fs::remove_all(tmp); + fs::create_directories(tmp); + auto destination = tmp / "payload.bin"; + auto backup = tmp / "payload.bin.old.100.1"; + std::ofstream(destination) << "committed-payload"; + std::ofstream(backup) << "previous-payload"; + + std::string error; + ASSERT_TRUE(xlings::xim::recover_download_transaction_( + destination, error)) << error; + EXPECT_EQ(xlings::platform::read_file_to_string(destination.string()), + "committed-payload"); + EXPECT_FALSE(fs::exists(backup)); + fs::remove_all(tmp); +} + // ============================================================ // xim installer tests // ============================================================ @@ -3221,6 +3688,40 @@ TEST(Extract, MissingArchiveReturnsError) { EXPECT_FALSE(r.has_value()); } +TEST(Extract, InvalidArchiveIsClassifiedAndEvictedWithSidecar) { + ExtractFixture fx; + auto archive = fx.tmp / "truncated.tar.gz"; + auto sidecar = std::filesystem::path(archive.string() + ".meta"); + std::ofstream(archive) << "not-a-complete-archive"; + std::ofstream(sidecar) << "format: 2\n"; + + auto result = xlings::xim::extract_archive_detailed( + archive, fx.tmp / "invalid-out"); + ASSERT_FALSE(result.has_value()); + EXPECT_EQ(result.error().kind, + xlings::xim::ExtractErrorKind::InvalidInputArchive); + EXPECT_TRUE(xlings::xim::evict_invalid_archive_cache_( + archive, result.error())); + EXPECT_FALSE(std::filesystem::exists(archive)); + EXPECT_FALSE(std::filesystem::exists(sidecar)); +} + +TEST(Extract, LocalWriteFailureDoesNotEvictValidInput) { + ExtractFixture fx; + auto archive = fx.make_tar_gz(); + auto invalidDestination = fx.tmp / "destination-is-a-file"; + std::ofstream(invalidDestination) << "not-a-directory"; + + auto result = xlings::xim::extract_archive_detailed( + archive, invalidDestination); + ASSERT_FALSE(result.has_value()); + EXPECT_EQ(result.error().kind, + xlings::xim::ExtractErrorKind::LocalWriteFailure); + EXPECT_FALSE(xlings::xim::evict_invalid_archive_cache_( + archive, result.error())); + EXPECT_TRUE(std::filesystem::exists(archive)); +} + TEST(Extract, RejectsPathTraversal) { // Build a tarball containing an entry with "../escape.txt". libarchive // with ARCHIVE_EXTRACT_SECURE_NODOTDOT must refuse to extract the @@ -3636,6 +4137,17 @@ TEST(DownloaderArchiveSniff, WorksWithFullPaths) { #ifndef XLINGS_USE_GTEST_MAIN int main(int argc, char** argv) { + if (argc == 4 && std::string_view(argv[1]) == "--file-lock-child") { + xlings::platform::FileLock lock; + std::string error; + if (!lock.acquire(argv[2], std::chrono::seconds{2}, {}, error)) { + std::cerr << error << '\n'; + return 2; + } + std::ofstream(argv[3]) << "ready"; + std::this_thread::sleep_for(std::chrono::milliseconds{400}); + return 0; + } ::testing::InitGoogleTest(&argc, argv); return RUN_ALL_TESTS(); } diff --git a/tests/unit/test_mirror.cpp b/tests/unit/test_mirror.cpp index cb94cb17..80ee2c92 100644 --- a/tests/unit/test_mirror.cpp +++ b/tests/unit/test_mirror.cpp @@ -15,6 +15,32 @@ namespace fs = std::filesystem; namespace { +class MirrorTestEnvironment final : public testing::Environment { +public: + void SetUp() override { + home_ = fs::temp_directory_path() + / std::format("xlings-mirror-test-{}", + std::chrono::steady_clock::now().time_since_epoch().count()); + fs::create_directories(home_); +#ifdef _WIN32 + _putenv_s("XLINGS_HOME", home_.string().c_str()); +#else + ::setenv("XLINGS_HOME", home_.string().c_str(), 1); +#endif + } + + void TearDown() override { + std::error_code ec; + fs::remove_all(home_, ec); + } + +private: + fs::path home_; +}; + +[[maybe_unused]] auto* mirror_test_environment_ = + testing::AddGlobalTestEnvironment(new MirrorTestEnvironment); + // Helper: any item in the list whose URL starts with the given prefix. bool any_starts_with(const std::vector& urls, std::string_view prefix) { return std::ranges::any_of(urls, [&](const auto& u) {