diff --git a/.agents/docs/2026-08-30-openkal-0.10-ecosystem-plan.md b/.agents/docs/2026-08-30-openkal-0.10-ecosystem-plan.md new file mode 100644 index 0000000..e17fc27 --- /dev/null +++ b/.agents/docs/2026-08-30-openkal-0.10-ecosystem-plan.md @@ -0,0 +1,445 @@ +# openkal 0.10 与整个生态的同步方案 + +2026-08-30 · 跨 9 个包 · 施工中 +前置:`2026-08-30-issue13-full-sweep-design.md`(缺陷定位)、 +`2026-08-30-issue13-exec-search-and-what-it-hid.md` + +--- + +## 0. 这份方案为什么存在 + +上一轮把四条缺陷**上报**给了规范和后端。那是不够的 —— 上报之后消费者手里的东西 +一点没变。这一轮把四条**做掉**:改规范、改所有实现、改消费者、发布、在干净沙箱里 +用真实索引验证。 + +⚠️⚠️ **并且规范一动,整个图必须整体移动。** 这不是洁癖,是 mcpp 的解析规则: +版本要求是精确的、不向上浮动,而 `openkal` 是图里所有包的**共享依赖**。所以 + + openkal-musl → openkal 0.10.0 + openkal-linux → openkal 0.9.0 + +会得到 `irreconcilable versions` 而不是"各用各的"。**这个失败我这一轮已经真实撞到 +过一次**(openkal-llvm-runtime 钉 openkal-musl 0.9.0,而我发了 0.10.0),所以它不是 +推测。 + +⇒ 规范升一个版本 = 五个实现 + 两个消费者 + 索引,一个都不能落。 + +--- + +## 1. 依赖图(实测,不是回忆) + +| 包 | 现在钉的 openkal | 自身版本 | 提供 fs / process / task | +| --- | --- | --- | --- | +| openkal-linux | 0.9.0 | 0.8.0 | ✓ ✓ ✓ | +| openkal-macos | 0.8.0 | 0.5.0(本地落后,已发 0.6.1) | ✓ ✓ ✓ | +| openkal-windows | 0.9.0 | 0.4.0 | ✓ ✓ ✓ | +| openkal-opensbi | 0.9.0 | 0.3.0 | ✓ ✗ ✗ | +| openkal-uefi | 0.9.0 | 0.3.0 | ✗ ✗ ✗ | +| ~~openkal-macos-abi~~ | — | — | ⚠️ **不是一个包** —— 见 §1.1 | +| **openkal-musl** | 0.9.0 | 0.10.0 | 消费者 | +| **openkal-llvm-runtime** | 经 musl | 0.6.0 | 消费者 | +| std-freestanding-alloc-kal | 0.8.0 | 0.1.1 | 消费者 | + +### 1.1 ⚠️⚠️ 我把同一个包数成了两个,而这正是"全局 review"要抓的东西 + +第一次清点时,`~/workspace/github/mcpplibs/` 下有 `openkal-macos` 和 +`openkal-macos-abi` 两个目录,两者 `mcpp.toml` 里的版本还不一样(0.5.0 / 0.6.0), +于是我把它们当成两个实现,各自实现了一遍、各自提交了一遍。 + +**它们是同一个仓库 `mcpplibs/openkal-macos` 的两个 git worktree**,只是分别停在 +不同分支上 —— 版本不同是因为分支不同,不是因为包不同。`name` 字段两边都是 +`openkal-macos`。 + +⇒ **实现是五个,不是六个。** 重复的那次提交没有到达远端(远端 +`declarations-0.10` 只有一份),第二个 worktree 已还回它原来的 +`abi/one-idea-one-spelling` 分支。 + +⚠️ 教训写在这里而不是抹掉:**目录名不是包名**。清点依赖图要读 +`mcpp.toml` 的 `name`,不是读目录。我下次清点先按 `name` 去重。 + +### 1.2 ⭐ 把索引整个扫一遍,而不是从正在改的包往外推 + +上面那句「本轮不扩大范围去动 `std-freestanding-alloc-kal`」是**写错了**,而且错的 +不是结论是方法:「生态闭环」不允许有一个包留在旧图里。所以改成扫 +`mcpp-index/pkgs/` 里**每一个提到 openkal 的文件**,再逐个核对「它自称的版本」与 +「它的消费者要求的版本」。扫出三处: + +| 包 | 问题 | 处理 | +|---|---|---| +| `openkal-opensbi` | 提交叫「0.4.0」、依赖已移到 openkal 0.10.0,**自己的 `version` 还是 0.3.0**;而 openkal-musl 0.11.0 要求的正是 0.4.0 | 补版本号 | +| `openkal-uefi` | 同上,一模一样 | 补版本号 | +| `std-freestanding-alloc-kal` | 钉在 `openkal = "0.9.0"`,**根本不在原清单里** | 0.1.3 + 重钉 0.10.0,PR #3 | + +⚠️ 前两条的形态值得记:**一次「只改依赖」的重钉,最容易漏掉的就是自己的版本号。** +tag 是 0.4.0、清单自称 0.3.0,发出去就是一个不认自己名字的包,而那条精确要求指着它。 + +⭐ 后一条说明**方法比结论重要**:它是被 `std-freestanding` 以 `^0.1.0` 引的,顺着 +「谁依赖 openkal-musl」往外推永远推不到它;只有反过来问「谁提到了 openkal」才会 +出现。核对完全链之后,**只有那两个自称版本对不上**,其余六个都吻合。 + +ⓘ 另有一处只记录、不动手:索引里 `openkal-libc` 和 `openkal-musl` 是**同一个包的 +两个条目**——`mcpplibs/openkal-libc` 仓库的 `mcpp.toml` 里 `name` 写的就是 +`openkal-musl`,两边 0.10.0 的发布时间戳精确到同一秒。这正是 1.1 那个「一个东西 +两个名字」的老问题,只是这次落在索引里。它早于本轮改动。 + +⚠️ **clause 9 的后果**:「An implementation exports the names of the interfaces it +provides and no other」。新操作属于 `openkal.fs` / `openkal.process` / +`openkal.task` 这三组,所以**提供该接口的实现必须导出这些名字**,不能只是不实现。 +能力上做不到的用 `kal_fs_props` 位说明 —— 这正是链接操作的既有模型。 +⇒ opensbi 要补四个 `kal_fs_*`;uefi 不提供 fs,不受影响,但仍要跟着重钉版本。 + +--- + +## 2. 规范加了什么(openkal 0.10) + +| 声明 | 解决 | 由哪个 props 位守门 | +| --- | --- | --- | +| `kal_fs_lock` / `kal_fs_unlock` + `KAL_LOCK_*` | 锁形同虚设 | `KAL_FS_PROP_LOCKS`(新,1<<5) | +| `kal_fs_set_modified_at` | 目录时间无路可走 | 复用 `KAL_FS_PROP_MODIFIED_TIME` | +| `kal_fs_capacity` | `fs::space` | `KAL_FS_PROP_CAPACITY`(新,1<<6) | +| `kal_process_spawn_bound` | `kill` 打不到 fork+exec 的程序 | `KAL_PROCESS_PROP_BOUND_LIFETIME`(新,1<<5) | +| `kal_task_parallelism` | `hardware_concurrency()` 静默答 1 | 复用 `KAL_TASK_PROP_PARALLEL` | + +全部是**新增声明**,没有改动任何既有声明,符合 clause 8。 + +--- + +## 3. ⭐ 全局设计 review —— 我自己的四处问题 + +写完之后按规范自己的规则逐条过了一遍,**发现四处问题并已修**。记在这里,因为 +"我 review 过了"和"review 发现了什么"是两回事。 + +### 3.1 props 位的位置放错了(已修) + +新位写成 `1<<5`、`1<<6` 没错,但**插在 `MAKE_LINKS`(1<<4)之前**,文件读起来位序 +是乱的。位值不冲突,所以编译和行为都对 —— 这正是它容易留下的原因。已按位序排好。 + +### 3.2 `kal_fs_capacity` 的措辞自相矛盾(已修) + +原文写「Either position may be left clear by an implementation that cannot answer +it, **which reports kal_err_not_supported instead**」—— 前半句说留空,后半句说报错, +两个不同的处置写成了一句。 + +⇒ 改成两级,且与 `kal_task_parallelism` 用**同一个约定**: +整个操作做不到 → `kal_err_not_supported`;某一个位置分辨不出 → 写 0, +而 0 的意思是"说不出"而不是"没有"。**调用者必须能把"没有空间"和"没有回答"分开。** + +### 3.3 ⚠️⚠️ 锁的持有者写成了"程序",而这会把一个著名的坑写进规范(已修) + +原文写「An implementation shall release what a program held when the program +ends」。这听起来对,但它默许了 POSIX `fcntl` 最老那套语义:**一个程序只要关闭该节点 +的任意一个描述符,它在这个节点上的全部锁就没了** —— 于是一个把同一个文件打开两次的 +库会自己毁掉自己的锁。 + +⇒ 改成:**持有者是这个 `kal_file`**,由 `kal_fs_close_file` 和程序结束释放,并明写 +实现**不得**暴露那套旧语义。Linux/macOS 都有更新的形式(持有者恰好就是打开的文件), +那才是该建在上面的东西。 + +⭐ 同时补上了"为什么这件事非得在线下面做":调用者**能**用 `KAL_OPEN_EXCLUSIVE` 加 +一个名字自己造互斥,造不出来的是**持有者死掉时的释放**。这是只有环境能给的那一半。 + +### 3.4 两个 spawn 变体不能组合,而这必须写明(已修) + +`kal_process_spawn_bound` 与既有的 `kal_process_spawn_with` **没有**组合形式。 +按 clause 8 只能新增声明,那么"授予目录 + 绑定寿命"就得是第四个声明。 + +⇒ 不声明,并写明理由:一个接口就是这样先有四个 spawn 再有八个的。等真有东西需要时 +再加(clause 8 允许)。⚠️ 但必须同时警告:**调用者不能退而用 +`kal_process_spawn_with` 并假设它绑定** —— 它不绑定,而"程序活得比调用者久"正是这条 +要消掉的失败。 + +### 3.5 复核过、结论是没问题的几条 + +- **clause 6.4**(有些资源永远做不到的操作不可采纳):锁由 `kal_fs_props` 守门,与 + 链接操作**完全同构** —— 规范自己的文字就说这正是链接可采纳的原因。✓ +- **clause 5.3**(结构布局冻结):没有新增或改动任何结构。✓ +- **clause 8**:五个都是新增声明,零改动。✓ +- **类型**:`start`/`len` 用 `kal_u64`(与 `kal_fs_truncate` 一致),`mode` 用 + `kal_uintptr`(与 `kal_fs_open` 的 flags 一致),`kal_task_parallelism` 返回 + `kal_uintptr`(与 `kal_task_current` 一致)。clause 5.4 排除的类型一个没用。✓ +- **错误词汇**:锁的"会阻塞"用 `kal_err_again`,这正是 `fcntl` 拼作 EAGAIN、 + 调用者据以轮询的那个。✓ + +--- + +## 4. 任务与依赖顺序 + +发布顺序由依赖图决定,**不能并行**: + +``` +① openkal 0.10.0(规范) + ↓ +② 五个实现同时重钉 + 实现新操作 + openkal-linux 0.9.0 / macos 0.7.0 / windows 0.5.0 / opensbi 0.4.0 / uefi 0.4.0 + ↓ +③ openkal-musl 0.11.0(消费:真锁、真 kill、目录时间、cpu 数、statvfs) + ↓ +④ openkal-llvm-runtime 0.7.0(重钉) + ↓ +⑤ mcpp-index 逐个条目 + ↓ +⑥ 干净沙箱 + 真实索引 + CN 镜像的闭环验证 +``` + +已完成的前置(不在上面这条链里): +**openkal-linux 0.8.0** —— spawn 用 CLOEXEC 管道回报失败的 exec(#20),已合并。 +实测:发布版 0.7.1 对"存在但不是程序"报**成功**、随后调用者死在 127;打了补丁后 +`posix_spawn` 报 EACCES、`execve` 返回 EACCES。 + +--- + +## 5. ⑥ 闭环验证要验什么(判据,不是"能编过") + +| # | 观察 | 现在(0.9 生态) | +| --- | --- | --- | +| 1 | 两个进程,第二个 `F_SETLK` **拿不到**锁 | 两个都拿到 | +| 2 | `fork`+`execve` 起的程序被 `kill` 后**真的停了** | 跑完全程 | +| 3 | `hardware_concurrency()` == 宿主的核数 | 1 | +| 4 | `std::filesystem::space()` 有答案 | ENOSYS | +| 5 | 锁目录的 `last_write_time(dir, t)` 走规范说的路 | 走的是规范外的路 | +| 6 | 起一个"存在但不是程序"的名字 → EACCES,调用者活着 | 已在 openkal-linux 0.8.0 修掉 | +| 7 | **每个依赖 openkal 的包仍然构建得起来** | — | +| 8 | 轮询过的流,一次 `read` 拿回**写方写的那一段**,不是一个字节 | 一次一字节 | +| 9 | 一个条件变量上两个等待者,广播把**第二个**也放出来 | 永远醒不来 | + +⚠️ 第 7 条是这一轮新加的,而且是最容易漏的:前六条都是"新功能对不对",第 7 条是 +"我有没有把别人弄坏"。规范一动就是整图移动,所以它必须逐包验证而不是抽查。 + +⚠️⚠️ **第 8、9 条是消费者的测试逼出来的,不是我想出来的**,而这正是前七条的方法 +问题:1–6 是照着「规范新加了什么」写的,7 是照着「我可能弄坏什么」写的,**没有 +一条是照着「消费者拿它做什么」写的**。8 和 9 两条缺陷都活过了本仓全部既有判据, +其中第 9 条本仓早就有「四个上下文抢一把锁八万次」而一路全绿——**抢锁不是等条件 +变量,一个等待者不是两个**。 + +⭐ 两条都已经写成本仓自己的判据(第 8 条在 `examples/subprocess`,第 9 条在 +`examples/posix`),并且**双向验证过**:摘掉修复它们变红。不双向验证的判据不算判据。 + +--- + +## 6. 进度(2026-08-30,施工中) + +| 阶段 | 状态 | +| --- | --- | +| openkal-linux 0.8.0 —— exec 失败回报(#20) | ✅ **已合并**。实测:0.7.1 对"存在但不是程序"报成功、调用者死在 127;修后 EACCES 且 `execve` 返回 | +| openkal 0.10.0 规范 | PR #25,CI 跑中,**1 项红**(见下) | +| openkal-linux 0.9.0 | PR #22。**十条判据全绿**:锁真排他(同一进程第二个打开的文件被拒)、容量 1500829MB、目录时间盖上、并行度 32 | +| openkal-macos 0.7.0 | PR #16。四条实现,`spawn_bound` 拒绝(无从被起映像内部武装的原语) | +| openkal-windows 0.5.0 | PR #16。四条实现,`spawn_bound` 记录为"能做但未实测,故不声明" | +| openkal-opensbi 0.4.0 / openkal-uefi 0.4.0 | PR #13 / #10,纯重钉 | +| openkal-musl 0.11.0 | 已推分支。**实测两条"曾记为修不了"的都真修好了**(见下) | +| openkal-llvm-runtime 0.7.0 | 待 musl 发布后重钉 | +| mcpp-index / 沙箱闭环 | 待整链绿 | + +### 6.1 ⭐ openkal-musl 侧的两条决定性读数 + +``` +锁: parent ACQUIRED / child EAGAIN => only one holds it (原来两个都拿到) +kill: fork + execve -> program was killed (原来跑完全程) +``` + +两条都是 0.10.0 的分歧表里写着"本仓修不了"的。规范加了词之后它们就不再是分歧。 + +### 6.2 ⚠️ 一项 CI 未过,待日志 + +`conformance (openkal-windows, msvc@system)` 红。运行未结束前 GitHub 不给日志, +待取。其余 linux/macos/windows-llvm 各腿在跑。 + +⚠️ 在整链绿之前**没有任何东西被发布**,所以消费者手里的生态没有被动过 —— 这是 +这一轮可以随时停在这里而不留下半成品的原因。 + +--- + +## 7. 用报告者的真实工程验证(`/home/speak/workspace/scode/re-cloud-code`) + +不再靠探针猜,直接构建他们的工程。**三条新缺陷是他们自己的测试套件发现的,我写的 +探针一条都没碰到。** + +### 7.1 ✅ 交叉编译两个架构都通,而且不需要他们那个绕过脚本 + +``` +x86_64-linux-musl 静态, INTERP=0, 未定义符号=0, 132M +aarch64-linux-musl 静态, INTERP=0, 未定义符号=0, 135M +``` + +`scripts/build-static.sh` 的开头写着「bypassing mcpp's gcc-pinned musl target」—— +**现在原生路径就能走通**,那个脚本可以退役。 + +⚠️ aarch64 之前不通的原因不是编译,是**清单里少三行**: +`openkal-llvm-runtime` 只声明在 `[target.x86_64-linux-musl.dependencies]` 下。 +错误信息本身就说清楚了:目标的 C 库由工具链约定提供,而 `[toolchain]` 指名 llvm +把约定顶掉了,于是这个目标在图里没有任何东西供给 C 库。加同样的块即可。 + +产物跑起来:进入备用屏、画帧、退出 0。`OPENKAL_MUSL_TRACE=enosys` 只报一个 +268(`fchmodat`)——就是双方已达成一致归他们的那条。 + +### 7.2 ⚠️⚠️ `chdir` 到不了被起的程序(新,且是他们 cwd 测试挂掉的真因) + +实测,带宿主对照: + +``` +after chdir(/tmp) : /tmp +started program pwd : <原目录> ← openkal-musl +started program pwd : /tmp ← 宿主 +``` + +链条:`okm_chdir` 只改本库自己的表(`okm_cwd_dir`/`g_cwd`),因为 **openkal 没有 +任何操作能改一个运行中程序的工作目录**;而 openkal-linux 用 `execveat(dirfd, …)`, +那个 dirfd 只用来**解析名字**,不改 cwd。 + +⚠️ 并且 openkal-linux 的源码注释写着「The started program's working directory is +the directory supplied here」——**它自称的这条性质它并没有实现**。规范的 +`process.h` 则对被起程序的工作目录只字未提。 + +⇒ 这是规范缺口 + 后端注释与实现不符,两处都要报。 + +### 7.3 ⚠️ 没有进程组,所以超时杀不掉后代(新) + +他们的超时逻辑是标准做法:子进程 `setpgid(0,0)` 自立组,父侧同步 `setpgid(pid,pid)`, +超时 `killpg` 整组。这个端口上没有组:父侧那次 `setpgid(pid,pid)` 报 EPERM, +`kill(-pgid)` 报 ESRCH。他们有兜底所以**直接子进程杀得掉**(「期限杀 → 124」是绿的), +但 bash 后台起的 `sleep` 没人杀 ⇒ 两条「无残留进程」判据红。 + +⚠️ 这正是我在 §3.4 评估 `kal_process_spawn_bound` 时**排除掉的那个替代方案** +(让 terminate 杀进程组)——排除的理由仍然成立(新组会脱离终端前台组),但这说明 +「杀掉一棵进程树」是消费者真实需要的东西,而 `spawn_bound` 只解决了直接子进程那一层。 + +### 7.4 ✅ 增量分块 —— 一次 `poll` 只留一个字节,于是流式变成了逐字节(已修) + +`echo one; sleep 0.4; echo two` 要求两段**分次**到达。内容顺序是对的 +(「stdout 全量按序」绿),分块不对。 + +根因在 `okm_syscall.c` 的 `do_read`:openkal 没有就绪查询,所以 `poll` 做一次**有界 +传输**并把读到的字节留下(`okm_take_ahead`)。但留下的只有**一个**字节,随后的 +`read` 就把那一个字节直接返回了——一个 4 字节的行要四次 `read` 才读完。 + +⭐ 实测,同一个探针,宿主作对照: + +``` +openkal-musl "o" "n" "e" "." "t" "w" "o" "." ← 八次,每次一字节 +宿主 "one." "two." ← 两次 +修好之后 "one." "two." ← 与宿主逐字节一致 +``` + +⇒ 修法:留下的那个字节交出去之后,若调用者的缓冲还有空间且**还能再做一次有界读**, +就接着把余下的读完再一起返回。`held + more`,一次系统调用一段。 + +⚠️⚠️ **这就是 openkal-linux#13 最初那句「只输出 1 字节」的真正含义**,而我此前把它 +读成了「输出被截断」。它没有被截断——它是**一次只到一个字节**,内容一个不少。 +半年前的报告用的词是准确的,读错的是我。 + +⭐ 并且这一条是**四条里唯一一条不需要动规范**的:`kal_stream` 的读已经够用,错的是 +这个端口怎么用它。7.2 / 7.3 都得等规范。 + +### 7.5 ✅ aarch64 的测试第一次真的跑起来了(qemu-user + binfmt_misc) + +此前 aarch64 **一次都没跑过**。原因还是 §7.1 那三行清单:六个 package 里 +**只有 `apps/cloudcode` 补了 aarch64 的块**,六个库 package 一个都没补,于是 +`mcpp test --target aarch64-linux-musl` 在解析阶段就停了,根本没到编译。 + +⭐ 更好的写法是不要按架构重复,`cfg` 谓词一条覆盖两个架构: + +```toml +[target.'cfg(all(os = "linux", env = "musl"))'.dependencies] +openkal-llvm-runtime = "0.7.0" +``` + +实测这条谓词两个架构都解析得到,`x86_64` 与 `aarch64` 各自的块可以合成这一条。 + +跑法上不需要虚拟机:产物是**静态**的,宿主已注册 `qemu-aarch64` 的 binfmt_misc, +所以 `mcpp test --target aarch64-linux-musl` 直接就地执行,不用起 VM,也不用改 +测试脚本。 + +⚠️ **两个架构的失败集合完全一致**——这本身是一条读数:挂掉的没有一条是与架构相关的, +全部是 7.2 / 7.3 那两个语义缺口,以及双方已达成一致归项目侧的权限位。 + +### 7.6 ⚠️⚠️ 两个等待者的条件变量永远醒不来(新,而且这是最严重的一条) + +`test_chat_approval` 在 musl 上 300s 超时,零输出。**宿主对照 0.06s 通过**——这一对 +数字就是全部诊断,也正是为什么必须跑宿主对照。 + +根因链条,三处证据互相印证: + +1. `__okm_futex` 只实现了 `FUTEX_WAIT` 和 `FUTEX_WAKE`,`FUTEX_REQUEUE` 落到默认 + 分支答 `ENOSYS`——**而且不发系统调用,所以 strace 里看不见**。 +2. musl 的 `unlock_requeue`(`pthread_cond_timedwait.c:48`)先放开屏障再请求 + requeue;两次请求都失败之后,**没有任何剩下的路径会唤醒谁**。 +3. strace 的终态是两个上下文各自阻塞在 `FUTEX_WAIT_PRIVATE, 2`,而 `2` 正是 + `lock()` 里 `a_cas(l,1,2)` 之后等待的那个值。 + +⭐⭐ **要两个等待者才会犯,这就是它活到现在的原因。** 那次 requeue 只在 +`node.prev` 非空时发出——也就是**有第二个上下文排在被释放者后面**时。一个等待者 +永远碰不到它,而一个等待者是绝大多数程序的样子。 + +⇒ 修法:**唤醒代替搬移**。openkal 没有「把等待者从一个地址搬到另一个」的操作,但 +musl 的等待者永远在 `while (a_cas(l, 0, 2))` 里,所以在原地址上唤醒它,它重读那个 +字、拿到调用者刚放开的锁、继续往下走。丢掉的是一趟调度,不是一个结果。 + +实测:同一条测试,300s 超时 → **0.10s 通过**。 + +⚠️ **这条是本仓自己的缺陷,不需要动规范**,和 7.4 一样。四条里两条是规范缺口 +(7.2 / 7.3),两条是这个端口怎么用现有原语的问题(7.4 / 7.6)。 + +⭐ 并且它暴露了判据方法本身的毛病:`examples/posix` 里早就有「四个上下文抢一把锁 +八万次」,一路全绿——**抢锁不是等条件变量,一个等待者不是两个**。已补一条只看 +「第二个等待者有没有被放出来」的判据,并且**双向验证过**:带修复三条 ok,把修复 +摘掉探针就停在这一块,报「it did not return; where it was」。 + +⇒ 7.2 / 7.3 / 7.4 / 7.6 都不在 issue 13 的清单里,也不在我这一轮加的任何判据里。 +**判据是照着已知缺陷写的,消费者的测试是照着他们要做的事写的**,后者找到了前者 +找不到的东西。 + +--- + +## 8. 两个架构的最终读数,以及每一条失败的归属 + +修完一字节读(7.4)和 requeue(7.6)之后,**两个架构逐条一致**: + +| package | x86_64 | aarch64 | +|---|---|---| +| kaos | 2/1(3 条断言红) | 2/1(同) | +| tui | 21/0 | 21/0 | +| agent-core | 42/9 | 42/9 | +| llm | 11/0 | 11/0 | +| oauth | 13/2 | 13/2 | +| pi-tui | 7/0 | 7/0 | +| **合计** | **96 通过 / 12 失败** | 同上 | + +⚠️ **每一条归属都有判别依据,不是看名字猜的**: + +**项目侧 8 条** +- 权限位 7 条(agent-core 5 + oauth 2)。双方早已达成一致,`fchmodat` 是唯一还在 + 报 ENOSYS 的那个号(268)。 +- `test_chat_live` 1 条。⭐ **宿主目标上也红**(0 passed / 1 failed,0.02s),报 + `config.toml is missing defaultProvider`——缺配置,与 openkal 无关。 + +**openkal 侧 4 条**——**只剩 §7.2 / §7.3 那两类语义缺口** +- cwd:`test_chat_tools`、`test_chat_mcp`,加 kaos 的 `cwd: 子进程 chdir 生效` +- 进程组:`test_chat_bashbg`(`任务未在 10s 内终态`),加 kaos 两条「无残留进程」 + +⭐ **判别法只有一个,就是同一套跑宿主目标。** `test_chat_approval` 宿主 0.06s 绿、 +musl 300s 不返回 ⇒ 本仓的;`test_chat_live` 宿主也红 ⇒ 不是本仓的。凭断言名字归类 +会把这两条都归错。 + +### 8.1 项目侧要改的两处(与失败无关,是构建用法) + +⚠️ **一、六个库 package 都缺 aarch64 的依赖块。** 只有 `apps/cloudcode` 补了,所以 +`mcpp test --target aarch64-linux-musl` 在解析阶段就停,**根本没到编译器**—— +「aarch64 跑不了」不是编译问题。 + +⭐ 更好的写法是不按架构重复,一条 `cfg` 谓词覆盖两个架构(实测两个架构都解析得到): + +```toml +# 七个 mcpp.toml 里都把 +[target.x86_64-linux-musl.dependencies] +# 换成 +[target.'cfg(all(os = "linux", env = "musl"))'.dependencies] +openkal-llvm-runtime = "0.7.0" +``` + +⚠️ **二、`scripts/build-static.sh` 可以退役。** 它开头写着「bypassing mcpp's +gcc-pinned musl target」,而原生路径现在两个架构都通,产物静态、`INTERP=0`、 +未定义符号 0。跑测试也不需要虚拟机:宿主注册了 `qemu-aarch64` 的 binfmt_misc, +静态产物直接就地执行。 + +ⓘ 我没有向他们的仓库推任何东西(`cloud-teahouse/re-cloud-code` 不是我的),本地 +那些清单改动是为了打通链条,验证完会还原。 diff --git a/.agents/docs/2026-08-30-openkal-0.11-start-design.md b/.agents/docs/2026-08-30-openkal-0.11-start-design.md new file mode 100644 index 0000000..4b085d7 --- /dev/null +++ b/.agents/docs/2026-08-30-openkal-0.11-start-design.md @@ -0,0 +1,119 @@ +# openkal 0.11 提案：把 spawn 的变体收敛成一个「怎么起」的描述 + +> 状态：**待拍板,一行代码都还没写。** 0.10 的六个 PR 正在合流,这份是 0.10 之后的事。 + +## 0. 为什么现在必须谈这个 + +0.10 之前,spawn 的变体是两个;0.10 加了第三个;**真实消费者的测试又要求第四和第五个**。 +`process.h` 自己的注释早就警告过这条路的尽头: + +> Declaring every combination is how an interface acquires four spawns and then +> eight, so the combination is declared when something needs it and not before. + +⭐ 现在「something needs it」到了,而且是**两件事同时到**。所以要谈的不是「再加一个变体」, +是**这个族该长成什么样**。 + +## 1. 两个缺口(都由报告者的测试套件发现,不是我猜的) + +### 1.1 说不出「程序在哪个目录里跑」 + +`kal_process_spawn(base, path, …)` 的 `base` 只**解析名字**。被起的程序的工作目录是 +**实现自己的**,调用者碰不到。 + +实测(宿主作对照): + +``` +chdir(/tmp) 之后 : /tmp +被起程序的 pwd : <原目录> ← openkal-musl +被起程序的 pwd : /tmp ← 宿主 +``` + +⚠️ **后端修不了。** `fchdir(b)` 会让 openkal-linux 那句注释成真而行为不会变好:`b` 是名字 +落在哪个 preopen 就是哪个,`/usr/bin/sh` 的 `b` 就是根。**给程序起名**和**说它在哪儿跑** +是两个目录,接口只带了一个。 + +⚠️ 并且这**不是**「openkal 拒绝可变 cwd」那条设计的问题。那条拒绝的理由是 +「a working directory that can be changed is shared mutable state between execution +contexts」——**这个理由完全成立,而且不适用于这里**:在**起程序的那一刻**说明它在哪儿跑, +是每次 spawn 各自说一次的、不可变的、不被任何两个上下文共享的东西。⇒ 该拒绝的继续拒绝, +缺的是另一件事。 + +### 1.2 说不出「连它起的东西一起杀」 + +`kal_process_terminate` 只到被起的那一个。shell 起到后台的东西没人管。 + +消费者的写法是标准做法(子进程 `setpgid(0,0)` 自立组,超时 `killpg` 整组),这个端口上 +`setpgid` 报 EPERM、`kill(-pgid)` 报 ESRCH。他们有兜底所以**直接子进程杀得掉**, +后代杀不掉。 + +⚠️ 0.10 的 `KAL_PROCESS_PROP_BOUND_LIFETIME` 解决的是**另一层**:它把被起程序绑在调用者 +的命上(`PR_SET_PDEATHSIG`),够不到孙子。两条不重叠。 + +## 2. ⭐ 提案：一个 `kal_process_start`,而不是第四、第五个 spawn + +```c +/* 声明时定死,clause 5.3 */ +struct kal_start { + struct kal_dir base; /* `path' 相对谁解析 */ + struct kal_dir work; /* 程序在哪个目录里跑 */ + const struct kal_preopen* grants; /* 交给它的目录,可为空 */ + kal_uintptr grant_count; + kal_uintptr flags; /* KAL_START_BOUND_LIFETIME | KAL_START_OWN_JOB */ +}; + +int kal_process_start(const struct kal_start*, + const char* path, kal_uintptr path_len, + const char** argv, const kal_uintptr* argv_lens, kal_uintptr argc, + const char** envp, const kal_uintptr* envp_lens, kal_uintptr envc, + const struct kal_spawn_streams* streams, + struct kal_process* out); +``` + +`KAL_START_OWN_JOB`:被起的程序**和它起的一切**构成一个单位,`kal_process_terminate` +把这个单位整个结束。Linux 是新进程组或 cgroup,Windows 是 job object,macOS 是进程组。 + +### 2.1 为什么是 flags 而不是再开变体 + +因为**两个缺口都要求「每次 spawn 各自选」**,而不是「实现要么总这样要么总不这样」: + +⚠️ 新进程组会**脱离终端前台组**,于是带界面的程序里,子上下文读终端会拿到 SIGTTIN 停住。 +我在 0.10 评估 `spawn_bound` 时就是因为这个排除了「让 terminate 杀进程组」——**那个理由 +现在仍然成立**。三个流全是管道的调用者(跑 shell 的那种)要这个行为;交互式的调用者绝不要。 + +⇒ 这个区别**只有按次表达才对**。做成默认行为是错的,做成实现属性也是错的。flags 正好。 + +⭐ 而 flags 与 openkal 的既有风格一致:`kal_*_props` 本来就是位集,实现对做不到的那一位 +答 `kal_err_not_supported`,和现在 `spawn_bound` 的约定一模一样。 + +### 2.2 ⚠️ 这个提案的代价,先说清楚 + +**它让 `spawn` / `spawn_with` / `spawn_bound` 三个都变成冗余的,而 clause 8 不许删。** +末态是四个声明,其中三个是历史。 + +这不好看。但另一条路的末态是**八个、然后十六个**,而且每加一个都要在五个实现里各写一遍。 +⇒ 我认为「一个通用形式 + 三个历史留下的」是可辩护的末态,「十六个 spawn」不是。 + +**第二个代价**:`struct kal_start` 一旦声明就冻住(clause 5.3),将来再有新需求,要么进 +`flags`(只够表达开关,不够表达带参数的东西),要么又开一个。⇒ flags 里能表达的将来能加, +带新参数的将来仍然会疼。这一点提案**没有**解决,只是把疼痛推远了。 + +## 3. 备选,以及我为什么不选它们 + +| 方案 | 为什么不选 | +|---|---| +| 加 `kal_process_spawn_in(work, base, …)` | 只解决 1.1;1.2 还要再来一个;组合还要再来。正是注释警告的那条路 | +| 让 `base` 同时当工作目录 | 1.1 里已经证明这不成立:`/usr/bin/sh` 的 base 是根 | +| 把工作目录塞进 `grants` 里约定一个名字(如 `"."`) | 隐式,违反「一个意思一种拼法」 | +| 加一个改 cwd 的操作 | openkal 明确拒绝过,**而且拒绝得对**——见 1.1 末段 | +| 什么都不加,让消费者自己绕 | 他们已经在绕了(`scripts/build-static.sh`),而绕不过 1.2 | + +## 4. 落地顺序(若拍板要做) + +1. openkal:声明 + SPEC 条目 + SURFACE + conformance 三节 +2. 五个实现各自实现;做不到的位不声明,答 `kal_err_not_supported` +3. openkal-musl:`posix_spawn` 的 `addchdir` / `setpgroup` 接到 `work` / `OWN_JOB` +4. openkal-llvm-runtime 跟版本 +5. 用报告者的工程验证:kaos 的三条红判据、agent-core 的三条,应当全绿 + +⚠️ **必须在 0.10 全部合流、发布、进 index 之后再开始**,否则两条链在同一批仓库里交叉, +版本要求又是精确的,任何一个包对不上整张图都解析不了。 diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index a384189..888b680 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -75,7 +75,7 @@ jobs: # primitive that copies an address space and starts a context in the # copy, and inventing one would be the simulation clause 3.1 # forbids. `--no-fork` asserts the refusal. - - { name: 'windows, gcc', os: windows-2022, toolchain: 'gcc@16.1.0', target: 'x86_64-windows-gnu', net: 'yes', fork: '--no-fork', shell: '--no-shell', abort: '--abort-terminated', dirtime: '--no-dir-time' } + - { name: 'windows, gcc', os: windows-2022, toolchain: 'gcc@16.1.0', target: 'x86_64-windows-gnu', net: 'yes', fork: '--no-fork', shell: '--no-shell', abort: '--abort-terminated', dirtime: '--dir-time' } defaults: run: shell: bash diff --git a/README.md b/README.md index c234890..b68acdf 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ the claim can be checked rather than repeated. ```toml [dependencies] -openkal-musl = "0.10.0" +openkal-musl = "0.11.0" ``` It names no implementation and no platform: a C library is the one consumer that @@ -41,6 +41,7 @@ consumer needs to answer it without asking. | this package | is carried by | | --- | --- | +| 0.11.0 | `openkal-llvm-runtime = "0.7.0"` | | 0.10.0 | `openkal-llvm-runtime = "0.6.0"` | | 0.9.0 | `openkal-llvm-runtime = "0.5.0"` | | 0.7.0 | `openkal-llvm-runtime = "0.4.0"` | @@ -75,8 +76,8 @@ architecture. Replacing that one header is the whole of the redirection; the `musl/PATCHES.md` lists the whole of what is not unmodified: **four patched lines**, all of one kind — a machine word carried through a variable declared -`long`, which is not a machine word on one of the three targets — and **ten -replaced sources**. Five of the ten are replaced for the same reason: each +`long`, which is not a machine word on one of the three targets — and **eleven +replaced sources**. Five of the eleven are replaced for the same reason: each reads the shape of one particular environment rather than asking a kernel for something. @@ -157,12 +158,14 @@ answer that leaves a program wrong without telling it. | closing a standard stream in a program being started | `posix_spawn_file_actions_addclose(&fa, 0…2)` makes the spawn report `ENOSYS`; above position two it is performed, because nothing there is inherited | openkal has no value meaning "no stream", and the value that looks like one — zero — means the opposite: the stream the caller has. Accepting the action and not performing it would hand a program the standard input its caller had just taken away. | | starting a program upon a stream whose handle is zero | a caller that redirects its **output** onto its own standard input and then starts a program gets `ENOSYS` | `kal_spawn_streams` reserves zero for inheritance and `kal_stream` reserves nothing, so an implementation whose streams are the environment's own descriptors hands out zero for standard input. The two agree at position zero and cannot be told apart anywhere else. Reported upstream; refused here rather than answered wrongly. | | ~~a version a program can read~~ | **answered since 0.9.0.** `uname`'s `release` field is this package's version, and `OPENKAL_MUSL_TRACE=enosys` names it on the error stream once per process before the program runs | It was the string literal `0.5.0` through every release after 0.5.0, so a program that asked was not left without an answer -- it was given a false one. ⚠️ It therefore MOVES AT EVERY RELEASE: nothing here or in musl reads it (`gethostname` and `getdomainname` are musl's only consumers of `uname` and both read `nodename`), but a program above it that compares the field against a fixed string will see it change. `sysname` is `openkal` and not `Linux`, so nothing can have been reading it as a kernel version. | -| ~~**setting** the modification time of a directory~~ | **answered since 0.10.0 where the implementation can open a directory**, which Linux and macOS can and Windows cannot — its `kal_fs_open` names `FILE_NON_DIRECTORY_FILE`, so `utimensat` on a directory is still refused there. **Reading** it was never affected and is correct everywhere | the port used to ask for `KAL_OPEN_READ \| KAL_OPEN_WRITE` unconditionally, which a directory refuses; it now asks what the name refers to and opens a directory for reading only. ⚠️ **That is outside what `fs.h` states** — the interface requires `KAL_OPEN_WRITE` for `kal_fs_set_modified` and names `kal_fs_open_dir`, which yields a `kal_dir`, as the way to open a directory, while `kal_fs_set_modified` has no `kal_dir` form. So there is no stated route to a directory's time at all; one has been asked for. A file still asks for exactly what the interface requires. Note that libc++ gives both overloads of `last_write_time` the same name in the message it throws, so a failure did not say which of the two had failed — and the one that worked was the one a consumer reported as broken. | -| ~~a lock on a file~~ | **`fcntl(F_SETLK)`, `F_SETLKW` and `F_GETLK` report `ENOSYS` since 0.10.0.** ⚠️ They used to answer 0 and do nothing, so **two programs took one exclusive lock and both were told they had it**; `F_GETLK` left the caller's word untouched, which reads as "somebody holds this" — for ever, so a loop waiting for a lock to be released never left it. `flock` has no case and reports `ENOSYS` too | openkal has no locking operation. ⭐ **Unlike the permission row, this refusal is temporary**: `fcntl(F_SETLK)` on Linux and macOS and `LockFileEx` on Windows all exist and all take a byte range, so every environment beneath openkal can perform it — what is missing is a word in the specification, and one has been asked for (a `kal_fs_lock` beside a `kal_fs_props` position, admitted on exactly the grounds the link operations were). It cannot be composed here meanwhile: a lock built from `KAL_OPEN_EXCLUSIVE` and a name beside the file is released by nobody when its holder dies, so a program that ended abnormally while holding one would be locked out of its own file for ever. | +| ~~**setting** the modification time of a directory~~ | **answered everywhere since 0.11.0.** ⚠️ 0.10.0 answered it only where the implementation could open a directory --- Linux and macOS could and Windows could not, because its `kal_fs_open` names `FILE_NON_DIRECTORY_FILE`. **Reading** it was never affected | 0.10.0 reached a directory's time by opening the directory for READING and stamping that, which worked and was **outside anything `fs.h` stated** --- there was no route to a directory's time at all. openkal 0.10 added `kal_fs_set_modified_at`, which takes a NAME, and every implementation answers it: the Windows one opens for the attribute alone, which reaches a directory where opening a FILE cannot. ⭐ The divergence this row recorded was caused by a missing declaration, and it went away when the declaration arrived. | +| ~~a lock on a file~~ | **`fcntl(F_SETLK)` and `F_SETLKW` take a real lock since 0.11.0.** ⚠️ They used to answer 0 and do nothing, so **two programs took one exclusive lock and both were told they had it**. ⭐ The holder is the **open file**, not the program: a SECOND open file of one name is refused here, where the older process-held form would have granted it and a library that opened one file twice would have destroyed its own lock. `F_GETLK` is still refused, and `flock` has no case | 0.10.0 refused all three and said the refusal was **temporary** in a way the permission row is not — every environment beneath openkal can lock a byte range and what was missing was a word. openkal 0.10 added `kal_fs_lock` with `KAL_FS_PROP_LOCKS`, and this is what that record said would happen. `F_GETLK` remains refused because it asks whether a lock **would** block without taking one, and openkal has no operation that answers a question without performing it — the absence clause 6.3 records for readiness. | | whether a file may be executed | `access(path, X_OK)` answers **yes for anything that exists**, and starting a name that exists and cannot be run still ends the caller with 127 | `kal_node_info` carries `writable` and no other permission, so "it is there" is the whole of what this port can answer. The two halves are the same gap: the enquiry cannot tell, and neither can the check `posix_spawn` makes before starting. openkal-linux knows — its own duplicate is the thing that fails — and has been asked to report it. | | descriptors above 2 crossing into a started program | a started program receives standard input, output and error and **nothing else**; a non-close-on-exec descriptor 4 is not there, and `fcntl(F_SETFD, 0)` upon one therefore changes nothing | `kal_spawn_streams` has exactly three positions and openkal has no general form for placing a stream at position *n*. `posix_spawn_file_actions_adddup2` above position two is already refused rather than accepted, so the two agree; only implicit inheritance is lost. A general form has been asked for. | -| how many processors there are | `sched_getaffinity` reports `ENOSYS`, so `std::thread::hardware_concurrency()` and `sysconf(_SC_NPROCESSORS_ONLN)` answer **1** | ⚠️ this one is silent: a program sizing a pool of workers gets one worker and no error. `openkal.task` says whether contexts run in parallel (`KAL_TASK_PROP_PARALLEL`) and not how many can; an enquiry has been asked for beside that word. | -| volume capacity, hard links, named pipes | `statvfs` (`std::filesystem::space`), `link` (`create_hard_link`), `mkfifo` and `socketpair` report `ENOSYS` | openkal has no operation for any of them. `kal_fs_link_create` makes a node whose content is a name — a symbolic link — and there is no hard link; `kal_process_channel` is a pipe in one direction, so a bidirectional pair is not one of them. Each is a loud absence rather than a wrong answer, which is why none is composed here. | +| **the working directory a started program runs in** | `chdir` succeeds and moves what *this* program resolves names against, and a program started afterwards runs in the directory the caller was in **before** it. `posix_spawn_file_actions_addchdir_np` is not answered | ⚠️ this one is silent, and it is the one a consumer is most likely to meet: the call reports success, the caller's own paths follow it, and only the started program disagrees. `kal_process_spawn`'s `base` is what the program's NAME resolves against, and resolving a name is not entering a directory — so naming a program and naming where it runs are two directories and openkal 0.10 carries one. ⭐ It cannot be composed here either: `chdir` in a copy before starting would need openkal to have an operation that moves a running program's working directory, and it deliberately has none. A per-spawn form has been asked for; see `.agents/docs/2026-08-30-openkal-0.11-start-design.md`. | +| **terminating what a started program itself started** | `kill` reaches a program this one started; a program *it* started is not reached. `setpgid` across processes reports `EPERM` and `setsid` reports `EPERM`, so `kill(-pgid)` finds no group and reports `ESRCH` | there are no process groups. ⭐ The refusals are honest rather than convenient — `setpgid(0, 0)` succeeds because in a world with no groups a context already is its own group of one, and that is a true answer rather than a fiction; every form that asks to move *another* process is refused. What this costs is the ordinary timeout: a shell that backgrounds work is killed and the work it backgrounded survives. `KAL_PROCESS_PROP_BOUND_LIFETIME` (0.10) binds a started program to its caller's life and does not reach a grandchild, so it does not answer this. Asked for in the same 0.11 note. | +| ~~how many processors there are~~ | **answered since 0.11.0.** `sched_getaffinity`, and therefore `std::thread::hardware_concurrency()` and `sysconf(_SC_NPROCESSORS_ONLN)`, report the real count | ⚠️ it used to be **silent**: a program sizing a pool of workers got one worker and no error. openkal 0.10 added `kal_task_parallelism`. ⭐ Zero from that enquiry means *cannot say* and is reported here as a refusal rather than as a bitmap of one processor, because musl would read the latter as a fact this port had invented. | +| hard links, named pipes, a bidirectional pair | `link` (`create_hard_link`), `mkfifo` and `socketpair` report `ENOSYS`. **`statvfs` (`std::filesystem::space`) is answered since 0.11.0** | openkal has no operation for the first three. `kal_fs_link_create` makes a node whose content is a name — a symbolic link — and there is no hard link; `kal_process_channel` is a pipe in one direction, so a bidirectional pair is not one of them. Volume capacity was in this row until openkal 0.10 added `kal_fs_capacity`. | | an alternate signal stack | `sigaltstack` reports `ENOSYS` since 0.10.0 | it used to report success and install nothing, and the enquiry that would have caught it answered 0 with a zeroed record. There are no signals here, so there is nothing for such a stack to be. | **⭐ What carries confinement here, since a mode word does not.** A program that @@ -381,11 +384,13 @@ beneath openkal are such environments. ⚠️ **This paragraph used to add that a caller cannot distinguish it. A caller can, and saying otherwise is what kept anyone from looking.** Three differences -are known and `musl/PATCHES.md` states each: a program that cannot be started -(**answered since 0.10.0** — the name is asked about first, so `execvp` can -search a PATH), a `kill` that reaches the waiting copy rather than the program -(**not answered**; use `posix_spawn`, `system` or `popen` where a caller needs -to stop what it started), and the identifier the started program reports. +were known and `musl/PATCHES.md` states each. Two are now answered: a program +that cannot be started (**0.10.0** — the name is asked about first, so `execvp` +can search a PATH; and **openkal-linux 0.8.0**, which reports an exec that +failed, so a name that exists and cannot be run is refused too), and a `kill` +that reached the waiting copy rather than the program (**0.11.0** — openkal 0.10 +added `kal_process_spawn_bound`, and `execve`, alone, asks for it). What remains +is the identifier the started program reports. **A program named without a suffix** is tried with one environment's suffix second, which is what every C library for that environment does. It is here diff --git a/examples/posix/src/main.c b/examples/posix/src/main.c index e075c95..1ae9603 100644 --- a/examples/posix/src/main.c +++ b/examples/posix/src/main.c @@ -27,6 +27,20 @@ static void *worker(void *p) { return p; } +/* Two contexts waiting upon ONE condition variable, which is a different thing + * from two contexts contending a mutex --- see where this is called. */ +static pthread_mutex_t cm = PTHREAD_MUTEX_INITIALIZER; +static pthread_cond_t cv = PTHREAD_COND_INITIALIZER; +static int cv_go, cv_done, cv_waiting; +static void *cv_waiter(void *p) { + pthread_mutex_lock(&cm); + cv_waiting++; + while (!cv_go) pthread_cond_wait(&cv, &cm); + cv_done++; + pthread_mutex_unlock(&cm); + return p; +} + int main(int argc, char **argv, char **envp) { /* The copy this program starts, so that openkal.process is observed by its * effect rather than by its return value. */ @@ -173,6 +187,50 @@ int main(int argc, char **argv, char **envp) { check(counter == 80000, "80000 increments, none lost"); printf(" counter=%d\n", counter); + /* ⚠️⚠️ TWO WAITERS UPON ONE CONDITION VARIABLE, AND THE COUNT IS THE POINT. + * + * The section above starts four contexts and contends a mutex hard, and it + * passed throughout a defect that stopped this port dead: a broadcast reaches + * musl's `unlock_requeue', which asks for `FUTEX_REQUEUE' --- and that request + * is made ONLY when a second context is queued behind the one being released. + * One waiter never reaches it. Answering it with ENOSYS therefore cost + * nothing at all until a program had two, and then it cost everything: the + * second waiter was never woken and the program did not end. + * + * ⭐ SO THE OBSERVATION IS NOT "A CONDITION VARIABLE WORKS". It is that a + * SECOND waiter is released, because the first one always was. Found in a + * consumer's own test suite rather than here, and this is the line that would + * have found it: every probe in this file was written from a list of known + * defects, and a defect that needs two waiters is not on such a list until + * something with two waiters runs. */ + { + pthread_t w[2]; + int made = 0; + for (int i = 0; i < 2; i++) if (pthread_create(&w[i], NULL, cv_waiter, NULL) == 0) made++; + + /* ⚠️ BOTH MUST BE WAITING BEFORE THE BROADCAST, or one is never queued + * behind the other, the requeue is never requested, and the observation + * below holds for a reason that has nothing to do with what it checks. */ + for (int spin = 0; spin < 400; spin++) { + pthread_mutex_lock(&cm); + const int ready = cv_waiting; + pthread_mutex_unlock(&cm); + if (ready == made) break; + struct timespec ms = { 0, 5000000 }; + nanosleep(&ms, NULL); + } + pthread_mutex_lock(&cm); + const int both_waiting = cv_waiting == made; + cv_go = 1; + pthread_cond_broadcast(&cv); + pthread_mutex_unlock(&cm); + + for (int i = 0; i < made; i++) pthread_join(w[i], NULL); + check(made == 2, "two contexts started to wait upon one condition variable"); + check(both_waiting, "both were waiting when the broadcast was made"); + check(cv_done == 2, "the broadcast released the SECOND waiter and not only the first"); + } + /* memory */ size_t big = 1 << 20; char *p = malloc(big); diff --git a/examples/subprocess/src/main.c b/examples/subprocess/src/main.c index c35003b..8bafb9d 100644 --- a/examples/subprocess/src/main.c +++ b/examples/subprocess/src/main.c @@ -64,6 +64,7 @@ #define _GNU_SOURCE #include #include +#include #include #include #include @@ -122,6 +123,14 @@ static int child_mode(int argc, char** argv) usleep((unsigned)atoi(argv[i + 1]) * 1000u); _exit(7); } + /* Two segments with a gap between them, so a reader that polls has to + * come back twice and each arrival has a size worth measuring. */ + if (strcmp(argv[i], "--child-two-segments") == 0) { + (void)!write(STDOUT_FILENO, "one.", 4); + usleep(400u * 1000u); + (void)!write(STDOUT_FILENO, "two.", 4); + _exit(0); + } } return -1; } @@ -713,6 +722,68 @@ int main(int argc, char** argv) else unsetenv("PATH"); } + /* ⚠️⚠️ WHAT ONE `read' RETURNS AFTER A `poll', AND THE SIZE IS THE WHOLE + * OBSERVATION. + * + * openkal has no readiness enquiry, so `poll' here performs a bounded + * transfer and keeps what it produced --- one byte, deliberately, because a + * larger read-ahead would be a second buffer under stdio's. The defect was + * that `read' then returned ONLY that byte: a caller that polls goes straight + * back to `poll', which keeps another, so a stream arrived one byte per + * iteration for ever. + * + * ⭐ EVERY BYTE WAS DELIVERED AND IN ORDER, WHICH IS WHY IT SURVIVED. A caller + * that concatenates sees exactly the right bytes; only a caller that looks at + * the BOUNDARIES sees anything wrong, and then it sees a lot --- a reader + * scanning each arrival for a word finds none, because `two.' arrives as `t' + * and `wo.'. openkal-linux#13's first report called this "only output one + * byte", and it was not truncation. + * + * ⇒ So this does not check the bytes. It checks that the FIRST arrival is the + * segment the writer wrote, which is the thing that was wrong. */ + { + int fds[2]; + if (pipe(fds) != 0) { + check(0, "a pipe is created for the segment observation"); + } else { + posix_spawn_file_actions_t fa; + posix_spawn_file_actions_init(&fa); + posix_spawn_file_actions_adddup2(&fa, fds[1], STDOUT_FILENO); + posix_spawn_file_actions_addclose(&fa, fds[0]); + + pid_t sp = -1; + char* av[] = { argv[0], (char*)"--child-two-segments", NULL }; + const int e = posix_spawn(&sp, argv[0], &fa, NULL, av, environ); + posix_spawn_file_actions_destroy(&fa); + close(fds[1]); + + if (e != 0) { + check(0, "a program writing two segments starts"); + close(fds[0]); + } else { + struct pollfd pf = { fds[0], POLLIN, 0 }; + char seg[64]; + long first = -1; + if (poll(&pf, 1, 5000) > 0) first = (long)read(fds[0], seg, sizeof seg); + check(first == 4 && memcmp(seg, "one.", 4) == 0, + "a polled read returns the segment the writer wrote, not one byte of it"); + if (first != 4) printf("note: first arrival was %ld byte(s)\n", first); + + /* And the second segment, which must not have been folded into the + * first --- a reader that got all eight bytes at once would satisfy + * the observation above for the wrong reason. */ + pf.revents = 0; + long second = -1; + if (poll(&pf, 1, 5000) > 0) second = (long)read(fds[0], seg, sizeof seg); + check(second == 4 && memcmp(seg, "two.", 4) == 0, + "and the second segment arrives separately, as its writer wrote it"); + close(fds[0]); + int s = 0; + waitpid(sp, &s, 0); + } + } + } + printf("-- failures: %d --\n", failures); return failures ? 1 : 0; } diff --git a/examples/surface/src/main.c b/examples/surface/src/main.c index c65330d..79373d2 100644 --- a/examples/surface/src/main.c +++ b/examples/surface/src/main.c @@ -118,6 +118,13 @@ int main(int argc, char** argv) const long open_max = sysconf(_SC_OPEN_MAX); check(open_max > 0, "the greatest number of descriptors is answered"); + /* ⚠️⚠️ IT ANSWERED 1, SILENTLY. `hardware_concurrency()' reads this, so a + * program sizing a pool of workers got one worker and no error. openkal 0.10 + * added the enquiry beneath it. This asserts only that it is a real count + * and not the fallback, because the number itself is the machine's. */ + const long cpus = sysconf(_SC_NPROCESSORS_ONLN); + check(cpus > 0, "how many contexts can run at once is answered"); + struct rlimit rl; const int got = getrlimit(RLIMIT_NOFILE, &rl); check(got == 0 && rl.rlim_cur > 0, "and the limit it is read from is the same enquiry"); @@ -129,13 +136,17 @@ int main(int argc, char** argv) /* ⚠️⚠️ THESE THREE ANSWERED 0 AND DID NOTHING. Measured against the host: * two programs took one exclusive lock and BOTH were told they had it. * - * ⭐ THE REFUSAL IS TEMPORARY IN A WAY `chmod' IS NOT, and this observation - * is written so that it says so. `fcntl(F_SETLK)' and `LockFileEx' both - * exist and both take a byte range, so every environment beneath openkal - * CAN do this; what is missing is a word in the specification, which has - * been asked for. When it arrives this observation is the one that changes, - * and it should change to "a second program is refused the lock" rather - * than be deleted. */ + * 0.10.0 refused them and said the refusal was TEMPORARY in a way `chmod' + * is not --- every environment beneath openkal can lock a byte range, and + * what was missing was a word in the specification. That comment said this + * observation was the one that would change when the word arrived, and that + * it should change to "a second holder is refused" rather than be deleted. + * + * ⭐ openkal 0.10 IS THAT WORD, so it changed, and this is now the + * observation it said it would become. `kal_fs_lock' states the holder as + * the open FILE, which is why a SECOND open file of one name is refused + * here --- the older process-held form would have granted it, and a library + * that opened one file twice would have destroyed its own lock. */ { const int fd = open("surface.lock", O_RDWR | O_CREAT | O_TRUNC, 0644); check(fd >= 0, "a file to ask about locking can be made"); @@ -145,15 +156,42 @@ int main(int argc, char** argv) fl.l_type = F_WRLCK; fl.l_whence = SEEK_SET; fl.l_start = 0; fl.l_len = 0; errno = 0; - refuses(fcntl(fd, F_SETLK, &fl), ENOSYS, - "taking a lock is refused rather than granted and not taken"); + check(fcntl(fd, F_SETLK, &fl) == 0, "an exclusive lock is taken"); + + /* ⭐ THE OBSERVATION THAT TELLS THE TWO FORMS APART, and it needs no + * second program: a SECOND OPEN FILE of the same name, here. The + * process-held form grants this, because the holder is the process + * and the process already holds it. The open-file form refuses it, + * and openkal states the open-file form. */ + const int again = open("surface.lock", O_RDWR); + check(again >= 0, "the same name can be opened a second time"); + if (again >= 0) { + struct flock two; + memset(&two, 0, sizeof two); + two.l_type = F_WRLCK; two.l_whence = SEEK_SET; + two.l_start = 0; two.l_len = 0; + errno = 0; + refuses(fcntl(again, F_SETLK, &two), EAGAIN, + "and a second OPEN FILE is refused, not granted"); + close(again); + } + + struct flock un; + memset(&un, 0, sizeof un); + un.l_type = F_UNLCK; un.l_whence = SEEK_SET; un.l_start = 0; un.l_len = 0; + check(fcntl(fd, F_SETLK, &un) == 0, "the lock is released"); + /* ⚠️ AND THE ENQUIRY IS STILL REFUSED, WHICH IS NOT AN OVERSIGHT. + * `F_GETLK' asks whether a lock WOULD block without taking one, and + * openkal has no operation that answers a question without + * performing it. Taking the lock and releasing it would answer, and + * would also take a lock the caller did not ask for. */ struct flock q; memset(&q, 0, sizeof q); q.l_type = F_WRLCK; q.l_whence = SEEK_SET; q.l_start = 0; q.l_len = 0; errno = 0; refuses(fcntl(fd, F_GETLK, &q), ENOSYS, - "and asking who holds one is refused rather than answered wrongly"); + "but asking WHETHER one would block is still refused"); close(fd); } unlink("surface.lock"); diff --git a/mcpp.toml b/mcpp.toml index bdde15a..1a157b3 100644 --- a/mcpp.toml +++ b/mcpp.toml @@ -1,7 +1,7 @@ [package] namespace = "mcpplibs" name = "openkal-musl" -version = "0.10.0" +version = "0.11.0" description = "musl 1.2.5 redirected onto openkal: one C library, ported once, above every implementation of the specification rather than above one kernel." license = "Apache-2.0" @@ -15,7 +15,7 @@ authors = ["mcpplibs"] repo = "https://github.com/mcpplibs/openkal-musl" [dependencies] -openkal = "0.9.0" +openkal = "0.10.0" # An ordinary consumer of openkal declares the specification and leaves the # choice of implementation to whoever builds the program, which is what the @@ -30,10 +30,10 @@ openkal = "0.9.0" # # The consequence for a program is that it names this package and nothing else. [target.'cfg(os = "linux")'.dependencies] -openkal-linux = { version = "0.7.1", features = ["standalone"] } +openkal-linux = { version = "0.9.0", features = ["standalone"] } [target.'cfg(os = "macos")'.dependencies] -openkal-macos = { version = "0.6.1", features = ["standalone"] } +openkal-macos = { version = "0.7.0", features = ["standalone"] } # ⚠️ FIRST STEP TOWARD A BARE MACHINE, AND NOT THE WHOLE OF IT. # @@ -44,7 +44,7 @@ openkal-macos = { version = "0.6.1", features = ["standalone"] } # runtime that receives control, and a C library configured for an environment # with no process to exit from. So this declares the implementation and stops. [target.'cfg(os = "none")'.dependencies] -openkal-opensbi = { version = "0.3.0", features = ["standalone"] } +openkal-opensbi = { version = "0.4.0", features = ["standalone"] } # ⭐ WHICH OPENKAL INTERFACES THE IMPLEMENTATION BENEATH IS EXPECTED TO PROVIDE. # @@ -69,7 +69,7 @@ openkal-opensbi = { version = "0.3.0", features = ["standalone"] } defines = ["OKM_HAS_FS=0", "OKM_HAS_PROCESS=0", "OKM_HAS_TASK=0"] [target.'cfg(windows)'.dependencies] -openkal-windows = { version = "0.4.0", features = ["standalone"] } +openkal-windows = { version = "0.5.0", features = ["standalone"] } # The feature macros musl's own build establishes. # @@ -126,6 +126,14 @@ sources = [ "!musl/src/mman/mmap.c", "!musl/src/internal/syscall_ret.c", "!musl/src/unistd/getcwd.c", + # ⚠️⚠️ THE THIRD SOURCE OF ONE KIND, AND IT WAS UNREACHABLE UNTIL 0.11.0. + # musl's `fcntl' reads its variable argument as an `unsigned long', which holds + # a pointer on every system musl was written for and thirty-two bits on one + # this port builds for --- so a `struct flock *' arrived with its top half + # gone. Every command answered before 0.11.0 took an integer or took a pointer + # nothing followed, so the truncation faulted nothing; a real `F_SETLK' follows + # it, and it faulted on the first attempt. Replaced by port/src/okm_fcntl.c. + "!musl/src/fcntl/fcntl.c", # Replaced by port/src/okm_phdr.c: musl answers this from the auxiliary # vector, which the replaced __libc_start_main does not read. "!musl/src/ldso/dl_iterate_phdr.c", diff --git a/musl/PATCHES.md b/musl/PATCHES.md index dbd776d..6274f27 100644 --- a/musl/PATCHES.md +++ b/musl/PATCHES.md @@ -71,7 +71,7 @@ carry it in a `long`. ## The sources this port replaces, and why each -Ten, and the list in the manifest carries the same reasons. Five read the shape +Eleven, and the list in the manifest carries the same reasons. Five read the shape of one environment directly. Two carry a machine word through a variable declared `long`. Two more were found only by running the result. And one is replaced because another already was: @@ -103,6 +103,40 @@ pointer: the value never becomes an integer at all. That is correct on every system musl was written for and is not correct on one that writes a volume first, and openkal does not say which a system does. +⚠️⚠️ `src/fcntl/fcntl.c` is the THIRD of that kind, and it survived three releases +of a port that already names the kind twice. + +It reads its variable argument as an `unsigned long`: + + unsigned long arg; + arg = va_arg(ap, unsigned long); + case F_SETLK: return syscall(SYS_fcntl, fd, cmd, (void *)arg); + +which holds a pointer on every system musl was written for and thirty-two bits +on one this port builds for. A caller passing a `struct flock *` had the top half +of it discarded **before this port saw it**. + +⭐ **It was unreachable until 0.11.0**, which is why it survived. Every command +this library answered took an integer, or took a pointer it never followed: +`F_SETLK` returned 0 and did nothing, and then reported `ENOSYS`. A truncated +pointer that nothing dereferences is a truncated pointer nothing reports. openkal +0.10 gave this port a real lock, `F_SETLK` began following the pointer, and it +faulted on the first attempt. + +⚠️ The register file names the type rather than the symptom: + + page fault on read access to 0x00000000fe2ffec2 + rax:00000000fe2ffec0 rsp:00007ffffe2fc7a0 + movzxw 0x02(%rax), %eax + +`rax` is the caller's pointer with its top thirty-two bits gone, and the offset +it faults at — two — is `l_whence`, the first field this port reads. + +⚠️ And the vararg TYPE is part of the calling convention rather than a detail: +`va_arg(ap, unsigned long)` and `va_arg(ap, uintptr_t)` read different numbers of +bytes where the two differ, so this is not a cast applied afterwards. Reading it +as the narrower type has already lost the half by then. + ## What has no equivalent on one object format `port/include/features.h` records a measurement rather than a patch: on PE, as @@ -152,16 +186,25 @@ and the first two were found by a consumer rather than here: the caller. What is still not answered is a name that exists and cannot be executed — openkal reports no execute permission, so that one still ends the caller with 127. Asked of openkal-linux, which knows and does not report it. -2. **`kill` does not reach a program started this way.** After `fork` and +2. **`kill` did not reach a program started this way.** After `fork` and `execve` there are three images, not two: the copy waits for the program it - started. A signal sent to the identifier the parent holds reaches the waiter, - which dies — and the parent is told the program died on that signal, while - the program runs to completion, unsupervised. Measured, with the host as - control: identical status words, opposite outcomes. **Not answered here.** - openkal has no way to say "this program's lifetime is bound to mine", and - `kal_process_terminate` is right to terminate only what it was given. Asked - of the specification. Until then a caller that needs to stop what it started - should use `posix_spawn`, `system` or `popen`, where `kill` does reach. + started, and a signal sent to the identifier the parent holds reached the + waiter. The parent was told the program died on that signal while the program + ran to completion, unsupervised. Measured, with the host as control: + identical status words, opposite outcomes. + + ⚠️ **This entry used to end "Not answered here", and it is answered now.** + openkal had no way to say "this program's lifetime is bound to mine", and + `kal_process_terminate` was right to terminate only what it was given — so + what was missing was a word, not a mechanism. openkal 0.10 added + `kal_process_spawn_bound`, and **since 0.11.0 `execve` asks for it**. + `posix_spawn` does not and must not: a POSIX child outlives its parent. + + ⚠️ A backend may decline the binding — openkal-macos has no primitive that + arms it from inside the started image, and openkal-windows has not measured + its own. There this falls back to the unbound spawn rather than refusing to + start the program at all, and the divergence is the one this entry used to + describe. `KAL_PROCESS_PROP_BOUND_LIFETIME` is what a caller asks. 3. **The identifier the started program reports is not the caller's**, because there are two images where a system with the operation would have one. diff --git a/port/src/okm_fcntl.c b/port/src/okm_fcntl.c new file mode 100644 index 0000000..1f1e22e --- /dev/null +++ b/port/src/okm_fcntl.c @@ -0,0 +1,96 @@ +/* Operations upon an open file description. + * + * ⚠️⚠️ THE ELEVENTH SOURCE THIS PORT REPLACES, AND IT IS THE THIRD OF ONE KIND: + * A MACHINE WORD CARRIED THROUGH A VARIABLE DECLARED `long'. + * + * musl's own reads its variable argument as an `unsigned long': + * + * unsigned long arg; + * arg = va_arg(ap, unsigned long); + * ... + * case F_SETLK: return syscall(SYS_fcntl, fd, cmd, (void *)arg); + * + * That is correct on every system musl was written for, where a `long' holds a + * pointer. It is not correct on one this port builds for, where it holds + * thirty-two bits and a pointer holds sixty-four --- so a caller passing a + * `struct flock *' had the top half of it discarded before the port saw it. + * + * ⭐⭐ AND IT WAS UNREACHABLE UNTIL 0.11.0, WHICH IS WHY IT SURVIVED THREE + * RELEASES OF A PORT THAT ALREADY NAMED THIS DEFECT TWICE. + * + * Every command this library answered before took an integer, or took a pointer + * it never followed: `F_SETLK' returned 0 and did nothing, and later `ENOSYS'. + * A truncated pointer that nothing dereferences is a truncated pointer nothing + * reports. openkal 0.10 gave this port a real lock, `F_SETLK' began following + * the pointer, and the defect became a fault on the first attempt. + * + * ⚠️ MEASURED, and the register file names the type rather than the symptom: + * + * page fault on read access to 0x00000000fe2ffec2 + * rax:00000000fe2ffec0 rsp:00007ffffe2fc7a0 + * winlk+0x1e67f: movzxw 0x02(%rax), %eax + * + * `rax' is the caller's `struct flock *' with its top thirty-two bits gone, and + * the offset it faults at --- two --- is `l_whence', the first field this port + * reads. The pointer was already ruined when the port received it. + * + * ⇒ Replaced rather than patched, which is what this port does with the other + * two of this kind (`mmap' and `getcwd'): the value never becomes a narrower + * integer at all. `uintptr_t' is the type that holds a pointer on every target, + * and it is what `va_arg' is asked for. + * + * ⚠️ THE VARARG TYPE IS PART OF THE CALLING CONVENTION AND NOT A DETAIL. + * `va_arg(ap, unsigned long)' and `va_arg(ap, uintptr_t)' read different numbers + * of bytes where the two types differ, so this is not a cast applied afterwards + * --- reading it as the narrower type has already lost the half by then. + */ +#define _GNU_SOURCE +#include +#include +#include +#include +#include "syscall.h" + +int fcntl(int fd, int cmd, ...) +{ + /* The one changed line, and the whole of the change. */ + uintptr_t arg; + va_list ap; + va_start(ap, cmd); + arg = va_arg(ap, uintptr_t); + va_end(ap); + if (cmd == F_SETFL) arg |= O_LARGEFILE; + if (cmd == F_SETLKW) return syscall_cp(SYS_fcntl, fd, cmd, (void *)arg); + if (cmd == F_GETOWN) { + struct f_owner_ex ex; + int ret = __syscall(SYS_fcntl, fd, F_GETOWN_EX, &ex); + if (ret == -EINVAL) return __syscall(SYS_fcntl, fd, cmd, (void *)arg); + if (ret) return __syscall_ret(ret); + return ex.type == F_OWNER_PGRP ? -ex.pid : ex.pid; + } + if (cmd == F_DUPFD_CLOEXEC) { + int ret = __syscall(SYS_fcntl, fd, F_DUPFD_CLOEXEC, arg); + if (ret != -EINVAL) { + if (ret >= 0) + __syscall(SYS_fcntl, ret, F_SETFD, FD_CLOEXEC); + return __syscall_ret(ret); + } + ret = __syscall(SYS_fcntl, fd, F_DUPFD_CLOEXEC, 0); + if (ret != -EINVAL) { + if (ret >= 0) __syscall(SYS_close, ret); + return __syscall_ret(-EINVAL); + } + ret = __syscall(SYS_fcntl, fd, F_DUPFD, arg); + if (ret >= 0) __syscall(SYS_fcntl, ret, F_SETFD, FD_CLOEXEC); + return __syscall_ret(ret); + } + switch (cmd) { + case F_SETLK: + case F_GETLK: + case F_GETOWN_EX: + case F_SETOWN_EX: + return syscall(SYS_fcntl, fd, cmd, (void *)arg); + default: + return syscall(SYS_fcntl, fd, cmd, arg); + } +} diff --git a/port/src/okm_opt.h b/port/src/okm_opt.h index 26d7498..c756579 100644 --- a/port/src/okm_opt.h +++ b/port/src/okm_opt.h @@ -59,6 +59,10 @@ #define okm_fs_rename kal_fs_rename #define okm_fs_file_info kal_fs_file_info #define okm_fs_set_modified kal_fs_set_modified +#define okm_fs_set_modified_at kal_fs_set_modified_at +#define okm_fs_lock kal_fs_lock +#define okm_fs_unlock kal_fs_unlock +#define okm_fs_capacity kal_fs_capacity #define okm_fs_list_begin kal_fs_list_begin #define okm_fs_list_next kal_fs_list_next @@ -149,6 +153,14 @@ static inline int okm_fs_rename(struct kal_dir, const char*, kal_uintptr, static inline int okm_fs_file_info(struct kal_file, kal_u32, struct kal_node_info*) { return kal_err_not_supported; } static inline int okm_fs_set_modified(struct kal_file, kal_u64) { return kal_err_not_supported; } +static inline int okm_fs_set_modified_at(struct kal_dir, const char*, kal_uintptr, + kal_u64) { return kal_err_not_supported; } +static inline int okm_fs_lock(struct kal_file, kal_u64, kal_u64, + kal_uintptr) { return kal_err_not_supported; } +static inline int okm_fs_unlock(struct kal_file, kal_u64, + kal_u64) { return kal_err_not_supported; } +static inline int okm_fs_capacity(struct kal_dir, kal_u64*, + kal_u64*) { return kal_err_not_supported; } static inline int okm_fs_list_begin(struct kal_dir, kal_uintptr*) { return kal_err_not_supported; } static inline int okm_fs_list_next(struct kal_dir, kal_uintptr*, char*, kal_uintptr, kal_uintptr*, int*) { return kal_err_not_supported; } @@ -158,6 +170,7 @@ static inline int okm_fs_list_next(struct kal_dir, kal_uintptr*, char*, kal_uint #if OKM_HAS_PROCESS #define okm_process_spawn kal_process_spawn +#define okm_process_spawn_bound kal_process_spawn_bound #define okm_process_wait kal_process_wait #define okm_process_terminate kal_process_terminate #define okm_process_close kal_process_close @@ -193,6 +206,11 @@ static inline int okm_process_spawn(struct kal_dir, const char*, kal_uintptr, const char**, const kal_uintptr*, kal_uintptr, const struct kal_spawn_streams*, struct kal_process*) { return kal_err_not_supported; } +static inline int okm_process_spawn_bound(struct kal_dir, const char*, kal_uintptr, + const char**, const kal_uintptr*, kal_uintptr, + const char**, const kal_uintptr*, kal_uintptr, + const struct kal_spawn_streams*, + struct kal_process*) { return kal_err_not_supported; } static inline int okm_process_wait(struct kal_process, int*, int*) { return kal_err_not_supported; } static inline int okm_process_terminate(struct kal_process) { return kal_err_not_supported; } @@ -208,6 +226,13 @@ static inline void okm_process_close(struct kal_process) {} #define okm_task_wait kal_task_wait #define okm_task_wake kal_task_wake +/* ⭐ WEAK, BECAUSE openkal 0.10 ADDED IT AND A BACKEND MAY NOT HAVE FOLLOWED. + * Every route in this port that reaches an optional operation tests the + * reference before calling; this one is reached from `sched_getaffinity', which + * a program asks once at startup and must not fault in. */ +extern __typeof(kal_task_parallelism) kal_task_parallelism __attribute__((__weak__)); +#define okm_task_parallelism kal_task_parallelism + /* The identity of the execution context in progress. */ #define OKM_CONTEXT_ID() kal_task_current() @@ -253,6 +278,12 @@ static inline int okm_task_wake(const kal_u32*, kal_uintptr, * per-context table working unchanged rather than growing a second shape. */ #define OKM_CONTEXT_ID() ((kal_uintptr)1) +/* Where contexts are withheld there is exactly one, and this says so rather + * than refusing: a program sizing itself against one context on a machine that + * has one is sizing itself correctly. */ +static inline kal_uintptr okm_task_parallelism_absent(void) { return 1; } +#define okm_task_parallelism okm_task_parallelism_absent + #endif /* OKM_HAS_TASK */ #endif /* OKM_OPT_H */ diff --git a/port/src/okm_spawn.c b/port/src/okm_spawn.c index f667e80..424053b 100644 --- a/port/src/okm_spawn.c +++ b/port/src/okm_spawn.c @@ -257,10 +257,63 @@ static int seed(struct kal_spawn_streams* s, int* placed) return 0; } +/* ⭐⭐ WHETHER THE STARTED PROGRAM MAY OUTLIVE THIS ONE, WHICH IS THE THING + * `execve' MEANS AND NOTHING ELSE HERE DOES. + * + * `execve' is composed as starting a program and ending with its status, so + * there are three images where a system with the operation has two: this one, + * the copy that waits, and the program. A signal aimed at the identifier the + * caller holds reaches the WAITER --- measured with a host as control: identical + * status words, opposite outcomes, the caller told the program died while the + * program ran to completion, unsupervised. openkal-linux#13. + * + * openkal 0.10 adds `kal_process_spawn_bound', which says the thing that could + * not be said. It is asked for ONLY where `execve' is meant, because an ordinary + * `posix_spawn' means the opposite: POSIX children outlive their parents. + * + * ⚠️ AND A BACKEND MAY DECLINE IT, in which case this falls back to the + * unbound spawn rather than refusing to start the program at all. A caller of + * `execve' that gets an unbound program is where this port has always been; a + * caller that gets no program is worse. The difference is recorded in + * musl/PATCHES.md and is what `KAL_PROCESS_PROP_BOUND_LIFETIME' is for. */ +static int start_program(int bound, struct okm_at* at, + const char** a_ptr, const kal_uintptr* a_len, int argc, + const char** e_ptr, const kal_uintptr* e_len, int envc, + struct kal_spawn_streams* streams, + struct kal_process* child) +{ + if (bound && (okm_process_props() & KAL_PROCESS_PROP_BOUND_LIFETIME)) { + const int e = okm_process_spawn_bound(at->base, at->rel, slen(at->rel), + a_ptr, a_len, (kal_uintptr)argc, + e_ptr, e_len, (kal_uintptr)envc, + streams, child); + if (e != kal_err_not_supported) return e; + } + return okm_process_spawn(at->base, at->rel, slen(at->rel), + a_ptr, a_len, (kal_uintptr)argc, + e_ptr, e_len, (kal_uintptr)envc, + streams, child); +} + +int __okm_spawn_common(pid_t* restrict res, const char* restrict path, + const posix_spawn_file_actions_t* fa, + const posix_spawnattr_t* restrict attr, + char* const argv[restrict], char* const envp[restrict], + int bound); + int __posix_spawn(pid_t* restrict res, const char* restrict path, const posix_spawn_file_actions_t* fa, const posix_spawnattr_t* restrict attr, char* const argv[restrict], char* const envp[restrict]) +{ + return __okm_spawn_common(res, path, fa, attr, argv, envp, 0); +} + +int __okm_spawn_common(pid_t* restrict res, const char* restrict path, + const posix_spawn_file_actions_t* fa, + const posix_spawnattr_t* restrict attr, + char* const argv[restrict], char* const envp[restrict], + int bound) { if (!res || !path) return EINVAL; if (attr && (attr->__flags & ~(POSIX_SPAWN_SETSIGDEF | POSIX_SPAWN_SETSIGMASK))) @@ -423,10 +476,8 @@ int __posix_spawn(pid_t* restrict res, const char* restrict path, * started program reads its own name through kal_env_arg(0), so a caller * that did not supply it could not predict what the program would read. */ struct kal_process child; - int e = okm_process_spawn(at.base, at.rel, slen(at.rel), - a_ptr, a_len, (kal_uintptr)argc, - e_ptr, e_len, (kal_uintptr)envc, - &streams, &child); + int e = start_program(bound, &at, a_ptr, a_len, argc, + e_ptr, e_len, envc, &streams, &child); /* ⭐ THE ONE ENVIRONMENT THAT SPELLS A PROGRAM WITH A SUFFIX IS ANSWERED * BEFORE THIS POINT AND NOT AFTER IT. diff --git a/port/src/okm_syscall.c b/port/src/okm_syscall.c index e1d9e56..505d1a4 100644 --- a/port/src/okm_syscall.c +++ b/port/src/okm_syscall.c @@ -86,6 +86,12 @@ extern __typeof(kal_timeout_wait_process) kal_timeout_wait_process __attribute__ int __posix_spawn(pid_t* restrict, const char* restrict, const posix_spawn_file_actions_t*, const posix_spawnattr_t* restrict, char* const[restrict], char* const[restrict]); +/* The same, and whether the started program may outlive this one. Only + * SYS_execve passes 1: `posix_spawn' means the opposite, because a POSIX child + * outlives its parent. */ +int __okm_spawn_common(pid_t* restrict, const char* restrict, + const posix_spawn_file_actions_t*, const posix_spawnattr_t* restrict, + char* const[restrict], char* const[restrict], int bound); #define OKM_PAGE 4096 @@ -142,7 +148,37 @@ static syscall_arg_t do_read(int fd, void* buf, size_t len) * result every caller of `read' already handles. */ const long held = okm_take_ahead(d, buf, len); if (held == OKM_AHEAD_EOF) return 0; - if (held) return (syscall_arg_t)held; + if (held) { + /* ⚠️⚠️ AND WHATEVER ELSE IS ALREADY THERE, BECAUSE ONE BYTE ON ITS OWN + * TURNED EVERY POLLED READ INTO A POLLED READ OF ONE BYTE. + * + * The enquiry takes a byte to make its answer true. Returning only that + * byte is a legal short read --- and a caller that polls goes straight + * back to `poll', which takes another byte, so the whole of a stream + * arrives ONE BYTE PER ITERATION, for ever. Measured against the host, + * on `echo one; sleep 0.4; echo two': + * + * here "o" "n" "e" "." "t" "w" "o" "." eight chunks + * host "one." "two." two + * + * ⚠️ Every byte is delivered and in order, so a caller that concatenates + * sees the right bytes --- which is why this survived: the defect is + * invisible to anyone who does not look at the BOUNDARIES. A caller that + * scans a chunk for a word finds none, because `two' arrives as `t' and + * `wo'. openkal-linux#13's first report said the program `only output + * one byte', and this is that. + * + * ⭐ THE REMEDY IS THE OPERATION THE ENQUIRY ITSELF IS BUILT ON. A bound + * of `now' asks for whatever has already arrived and does not wait, so + * the byte and the rest of what is there come back together. Where the + * environment beneath declines `openkal.timeout' there is no such + * operation, and the single byte is what can be honestly returned. */ + if (len > 1 && okm_can_bound()) { + const long more = okm_timed_read(s, (char*)buf + 1, len - 1, OKM_NOW_NS); + if (more > 0) return (syscall_arg_t)(held + more); + } + return (syscall_arg_t)held; + } if (d->flags & O_NONBLOCK) return okm_timed_read(s, buf, len, OKM_NOW_NS); @@ -1181,12 +1217,11 @@ syscall_arg_t __okm_syscall(syscall_arg_t n, syscall_arg_t a1, syscall_arg_t a2, * `kal_fs_set_modified' takes a `kal_file' and has no `kal_dir' form. * So there is no stated route to a directory's time at all. * - * ⇒ The intent is stated first and the fallback is taken only for a - * directory, so a FILE still asks for exactly what the interface - * requires. An implementation that cannot do it returns an error and - * that error is passed on unchanged: this is not a simulation and not a - * silent success, it is one operation attempted a second way. Recorded - * in musl/PATCHES.md and asked of the specification. */ + * ⭐⭐ ASKED OF THE SPECIFICATION, AND openkal 0.10 ANSWERED IT. + * `kal_fs_set_modified_at' takes a NAME, so a directory is now reached by + * a stated route rather than by opening it for reading and hoping. The + * older way is kept below for a backend that has not followed yet, and is + * tried only where the new operation is absent. */ struct kal_node_info kind = { .self_size = sizeof kind }; const int ke = okm_fs_info(at.base, at.rel, slen(at.rel), 0, KAL_INFO_KIND, &kind); @@ -1199,6 +1234,13 @@ syscall_arg_t __okm_syscall(syscall_arg_t n, syscall_arg_t a1, syscall_arg_t a2, ? KAL_OPEN_READ : (KAL_OPEN_READ | KAL_OPEN_WRITE); + { + const int se = okm_fs_set_modified_at(at.base, at.rel, + slen(at.rel), when); + if (se != kal_err_not_supported) + return se == kal_ok ? 0 : -okm_errno(se); + } + struct kal_file f; int e = okm_fs_open(at.base, at.rel, slen(at.rel), want, &f); if (e != kal_ok) return -okm_errno(e); @@ -1485,38 +1527,63 @@ syscall_arg_t __okm_syscall(syscall_arg_t n, syscall_arg_t a1, syscall_arg_t a2, return 0; } /* ⚠️⚠️ THESE THREE ANSWERED `0' AND DID NOTHING, SO EVERY LOCK WAS - * GRANTED AND NO LOCK EXISTED. + * GRANTED AND NO LOCK EXISTED. Measured with the host as control: two + * programs took one exclusive lock and BOTH were told they had it. * - * Measured, with the host as control: two programs took an exclusive - * lock on one file and BOTH were told they had it; on the host the - * second is refused with EAGAIN. Anything protecting a write with a - * lock --- a state file, a single-instance guard, a database --- had no - * protection and no way to find out. + * 0.10.0 refused them, and said the refusal was TEMPORARY in a way the + * permission one is not --- every environment beneath openkal can lock a + * byte range, and what was missing was a word in the specification. * - * ⚠️ `F_GETLK' was worse, because its answer pointed the other way. - * POSIX says it writes `F_UNLCK' into `l_type' when nothing would - * block; leaving the caller's word untouched returns the `F_WRLCK' the - * caller conventionally put there before asking, so the answer read - * "somebody holds this" --- for ever. A loop waiting for a lock to be - * released never left it. + * ⭐ openkal 0.10 IS THAT WORD. `kal_fs_lock' states the holder as the + * open FILE and requires release when the program ends however it ends, + * which is the half a caller could never have built for itself. */ + case F_SETLK: case F_SETLKW: { + if (d->kind != OKM_FILE) return -EBADF; + const struct flock* fl = (const struct flock*)a3; + if (!fl) return -EFAULT; + /* openkal takes a position and a length; POSIX takes a position, a + * length and where the position is measured from. Only the third + * needs translating, and only two of its three forms can be. */ + kal_u64 start; + if (fl->l_whence == SEEK_SET) start = (kal_u64)fl->l_start; + else if (fl->l_whence == SEEK_CUR) { + kal_u64 here = 0; + if (okm_fs_seek(d->file, 0, KAL_SEEK_CURRENT, &here) != kal_ok) + return -EINVAL; + start = here + (kal_u64)fl->l_start; + } else { + /* SEEK_END, which openkal has no form of: the length a range is + * measured back from is not a thing this interface reports at the + * moment the lock is taken. Refused rather than computed from a + * size that may already have changed. */ + return -EINVAL; + } + /* ⭐ ZERO MEANS `TO THE END, HOWEVER FAR THAT COMES TO BE' IN BOTH, + * so it is passed rather than translated. */ + const kal_u64 len = (kal_u64)fl->l_len; + + if (fl->l_type == F_UNLCK) { + const int e = okm_fs_unlock(d->file, start, len); + return e == kal_ok ? 0 : -okm_errno(e); + } + kal_uintptr mode = (fl->l_type == F_RDLCK) ? KAL_LOCK_SHARED + : KAL_LOCK_EXCLUSIVE; + if ((int)a2 == F_SETLKW) mode |= KAL_LOCK_WAIT; + const int e = okm_fs_lock(d->file, start, len, mode); + return e == kal_ok ? 0 : -okm_errno(e); + } + /* ⚠️ AND THE ENQUIRY IS STILL REFUSED, WHICH IS NOT AN OVERSIGHT. * - * ⭐ REFUSED, AND THE REFUSAL IS TEMPORARY IN A WAY `chmod' IS NOT. - * `chmod' is declined because a FAT volume, a UEFI partition and a - * Windows access-control list do not share a model. Locking is the - * opposite: `fcntl(F_SETLK)', `fcntl(F_SETLK)' and `LockFileEx' all - * exist and all take a byte range, so every environment beneath openkal - * can perform it. What is missing is a WORD in the specification, and - * one has been asked for --- `kal_fs_lock' beside a `kal_fs_props' - * position, which is exactly how link operations were admitted. When it - * lands these three lines become an implementation and no caller - * changes. + * `F_GETLK' asks whether a lock WOULD block without taking one, and + * openkal has no operation that answers a question without performing + * it --- the same absence clause 6.3 records for readiness. Taking the + * lock and releasing it again would answer, and would also take a lock + * the caller did not ask for, hand a spurious `no' to a caller that + * already holds one, and be stale the moment it returned. * - * ⚠️ It cannot be composed here in the meantime. A lock built out of - * `KAL_OPEN_EXCLUSIVE' and a name beside the file would be released by - * nobody when its holder died --- a program that ended abnormally while - * holding one would be locked out of its own file for ever, which is a - * worse failure than the refusal and a much harder one to read. */ - case F_SETLK: case F_SETLKW: case F_GETLK: return -ENOSYS; + * ⭐ A refusal is what a caller can act upon; `F_SETLK' answers the + * question `F_GETLK' is usually asked in order to answer. */ + case F_GETLK: return -ENOSYS; default: return -EINVAL; } } @@ -1738,8 +1805,13 @@ syscall_arg_t __okm_syscall(syscall_arg_t n, syscall_arg_t a1, syscall_arg_t a2, * recorded in musl/PATCHES.md. */ case SYS_execve: { pid_t child = 0; - const int e = __posix_spawn(&child, (const char*)a1, 0, 0, - (char* const*)a2, (char* const*)a3); + /* ⭐ BOUND, WHICH IS THE THING `execve' MEANS. The started program + * stands in for this one, so it does not outlive it --- and until + * openkal 0.10 there was no way to say so, which is why a `kill' aimed + * at this image reached the copy that waits and left the program + * running to completion, unsupervised. */ + const int e = __okm_spawn_common(&child, (const char*)a1, 0, 0, + (char* const*)a2, (char* const*)a3, 1); if (e) return -e; int st = 0; if (do_wait4((int)child, &st, 0, 0) < 0) kal_exit(127); @@ -2062,6 +2134,78 @@ syscall_arg_t __okm_syscall(syscall_arg_t n, syscall_arg_t a1, syscall_arg_t a2, * * musl reaches this through `prlimit64' first and falls back to * `getrlimit' only on ENOSYS, so answering this one answers both. */ + /* ⭐⭐ IT ANSWERED 1, SILENTLY, AND A POOL OF WORKERS WAS SIZED AGAINST IT. + * + * musl's `sysconf(_SC_NPROCESSORS_ONLN)' reaches this, and with no case it + * fell back to 1 --- so `std::thread::hardware_concurrency()' answered 1 + * with no error and a program that sizes itself got one worker and no way + * to know. Measured: 1 here against 32 on the same machine's own C library. + * openkal 0.10 adds the enquiry, and it reports the set THIS context may + * run on rather than the set the machine has. + * + * ⚠️ ZERO IS `CANNOT SAY' AND IS NOT ONE, so it is reported as a refusal + * rather than as a bitmap of one processor: musl would read the latter as a + * fact and this port would be inventing it. */ +#ifdef SYS_sched_getaffinity + case SYS_sched_getaffinity: { + const size_t cap = (size_t)a2; + unsigned char* out = (unsigned char*)a3; + if (!out || cap < sizeof(unsigned long)) return -EINVAL; + if (!okm_task_parallelism) return -ENOSYS; + const kal_uintptr n = okm_task_parallelism(); + if (n == 0) return -ENOSYS; + + for (size_t i = 0; i < cap; i++) out[i] = 0; + /* The identity of the processors is not reported by openkal --- only how + * many --- so the lowest `n' positions are set. musl counts the bits and + * asks nothing else of them. */ + size_t bits = (size_t)n; + if (bits > cap * 8) bits = cap * 8; + for (size_t i = 0; i < bits; i++) out[i / 8] |= (unsigned char)(1u << (i % 8)); + const size_t words = (bits + 8 * sizeof(unsigned long) - 1) + / (8 * sizeof(unsigned long)); + return (syscall_arg_t)(words * sizeof(unsigned long)); + } +#endif + + /* How much the volume holds, which `std::filesystem::space' reaches. */ +#if defined(SYS_statfs) || defined(SYS_fstatfs) + case +# ifdef SYS_statfs + SYS_statfs +# else + SYS_fstatfs +# endif + : { + struct okm_statfs { + unsigned long type, bsize; + uint64_t blocks, bfree, bavail, files, ffree; + struct { int v[2]; } fsid; + unsigned long namelen, frsize, flags, spare[4]; + }* out = (void*)a2; + if (!out) return -EFAULT; + struct okm_at at; + const syscall_arg_t r = okm_resolve(AT_FDCWD, (const char*)a1, &at, 0); + if (r) return r; + struct kal_dir d; + int e = okm_fs_open_dir(at.base, ".", 1, &d); + if (e != kal_ok) return -okm_errno(e); + kal_u64 total = 0, avail = 0; + e = okm_fs_capacity(d, &total, &avail); + okm_fs_close_dir(d); + if (e != kal_ok) return -okm_errno(e); + + for (unsigned i = 0; i < sizeof *out; i++) ((char*)out)[i] = 0; + /* openkal reports BYTES and this record counts blocks, so a block size + * is chosen and the counts are derived from it rather than invented. */ + out->bsize = out->frsize = OKM_PAGE; + out->blocks = total / OKM_PAGE; + out->bfree = out->bavail = avail / OKM_PAGE; + out->namelen = 255; + return 0; + } +#endif + #ifdef SYS_prlimit64 case SYS_prlimit64: { struct okm_rlimit64 { uint64_t cur, max; }; diff --git a/port/src/okm_thread.c b/port/src/okm_thread.c index 11e87ea..6e0149c 100644 --- a/port/src/okm_thread.c +++ b/port/src/okm_thread.c @@ -175,6 +175,55 @@ syscall_arg_t __okm_futex(const int* addr, int op, int val, const struct timespe if (e != kal_ok) return -okm_errno(e); return (long)woken; } + /* ⚠️⚠️ WOKEN HERE RATHER THAN MOVED THERE, AND WITHOUT THIS CASE A CONDITION + * VARIABLE WITH TWO WAITERS STOPPED FOR EVER. + * + * `FUTEX_REQUEUE' asks for `val' waiters upon `addr' to be woken and a + * further `val2' to be MOVED to a second address, so that a thread released + * by a condition variable goes straight to waiting upon the mutex instead of + * waking only to block again. openkal has kal_task_wait and kal_task_wake and + * nothing that moves a waiter between two addresses --- so this used to reach + * the arm below and answer ENOSYS. + * + * ⚠️ AND MUSL DOES NOT CHECK. `unlock_requeue' in pthread_cond_timedwait.c + * releases the barrier and then makes this request; when it fails there is no + * remaining path that wakes anyone, so the next waiter in the list sleeps + * until the program is killed: + * + * a_store(l, 0); + * if (w) __wake(l, 1, 1); + * else __syscall(SYS_futex, l, FUTEX_REQUEUE|FUTEX_PRIVATE, 0, 1, r) != -ENOSYS + * || __syscall(SYS_futex, l, FUTEX_REQUEUE, 0, 1, r); + * + * ⭐ IT TAKES TWO WAITERS, WHICH IS WHY IT SURVIVED. That call is reached only + * when `node.prev' is set --- when a second context is queued behind the one + * being released. One waiter upon a condition variable never reaches it, and + * one waiter is what almost every program has. Measured against a host: a + * consumer's test that has two contexts wait upon one variable passes there in + * 0.06s and did not finish here in 300. + * + * ⇒ Waking is a correct substitute for moving, and the loop musl wakes into is + * why. A waiter is always inside `while (a_cas(l, 0, 2))', so a context woken + * upon `addr' re-reads the word, takes the lock the caller has just released, + * and proceeds. What is lost is the journey: it wakes, and may block again + * upon the mutex, where a move would have left it blocked once. That is a cost + * in scheduling and not in correctness, and it is the whole difference. + * + * ⚠️ `val2' arrives in the argument this port declares as a deadline, because + * that is the register the operation puts it in. It is a count here. */ + case FUTEX_REQUEUE: { + const kal_intptr move = (kal_intptr)t; + /* Either count unbounded makes the sum unbounded, and the sum is what is + * woken --- added rather than saturated, a pair of "all" would wrap to + * none and stop exactly what this case exists to release. */ + const kal_uintptr n = (val < 0 || move < 0) + ? (kal_uintptr)-1 + : (kal_uintptr)val + (kal_uintptr)move; + kal_uintptr woken = 0; + const int e = okm_task_wake((const kal_u32*)addr, n, &woken); + if (e != kal_ok) return -okm_errno(e); + return (long)woken; + } default: return -ENOSYS; } diff --git a/tools/cross-build-macos.sh b/tools/cross-build-macos.sh index 3aa8ccd..97c32bf 100755 --- a/tools/cross-build-macos.sh +++ b/tools/cross-build-macos.sh @@ -65,7 +65,7 @@ cd "$here" # Kept in step with mcpp.toml, INCLUDING that system's own exclusions: # okm_phdr.c answers dl_iterate_phdr from an ELF header and that format has none. -skip='__libc_start_main|__init_tls|__set_thread_area|clone|posix_spawn|posix_spawnp|mmap|syscall_ret|getcwd|dl_iterate_phdr|okm_phdr|cache' +skip='__libc_start_main|__init_tls|__set_thread_area|clone|posix_spawn|posix_spawnp|mmap|syscall_ret|getcwd|fcntl|dl_iterate_phdr|okm_phdr|cache' for f in musl/src/*/*.c musl/src/malloc/mallocng/*.c port/src/*.c port/src/*.S; do base=$(basename "$f"); base=${base%.*} [[ "$base" =~ ^($skip)$ ]] && continue diff --git a/tools/probe-cross-macos.sh b/tools/probe-cross-macos.sh index d42db34..3ed6ba6 100755 --- a/tools/probe-cross-macos.sh +++ b/tools/probe-cross-macos.sh @@ -86,7 +86,7 @@ cd "$here" # SECOND statement of what mcpp.toml already states, and a second statement is # a thing that falls behind the first. It fell behind on the release that added # the tenth entry, and it is this job that said so. -skip='__libc_start_main|__init_tls|__set_thread_area|clone|posix_spawn|posix_spawnp|mmap|syscall_ret|getcwd|dl_iterate_phdr|okm_phdr|cache' +skip='__libc_start_main|__init_tls|__set_thread_area|clone|posix_spawn|posix_spawnp|mmap|syscall_ret|getcwd|fcntl|dl_iterate_phdr|okm_phdr|cache' units=0 for f in musl/src/*/*.c musl/src/malloc/mallocng/*.c port/src/*.c port/src/*.S; do base=$(basename "$f"); base=${base%.*}