feat(SGLANG-ORACLE-LEASE-WHEEL): the wheel route runs the oracle in a lease, and the box rebooted DURING the leg that would have given a ratio (#1265) - #1831
Merged
Conversation
…se, and that is still not a run (#1265) W1 read the wheels and planned a route out of the forbidden container path. It measured files, and it said so: `## What is measured here, and what is not` exists precisely because a correct wheel is not an oracle. This is W2, which wrote the driver W1 owed and ran it on `dgx:gpu0`. ## What the lease measured One `rc run` job, `86282a1a-6e07-4099-b2e8-f4768aa714e8`, worker `rc-worker-4b8lj`, 2026-08-23T20:35:04Z to 21:03:34Z, exit 0. No `ssh`, no container image, no file mutex outside the lease. It downloaded both wheels from PyPI and hashed THE BYTES THAT LANDED, against the values W1 committed: 12,716,006 / `1c2d2602…` and 34,243,333 / `727e4bc5…`, both `WHEEL_SHA_OK=1`. A remote hash would have proved nothing about the file on that disk. It installed them in upstream's own order and from upstream's own index (`docker/Dockerfile:210,242`), then asserted the installed tree from `cd /`: manifest_files=3338 derived_files=3338 missing=0 extra=0 differing=0 IDENTITY OK: 3338 files match the manifest for pin f63458b5… IDENTITY_RC=0 `sglang/_version.py` sets `__commit_id__ = None`, so that manifest is the only commit assertion the installed package can carry. It is now asserted. ## Two things W1 left open that this closes **The FlashInfer JIT-cache difference is CLOSED, not accepted.** `docker/Dockerfile:21` defaults that stage off and nobody has read `sha256:d0a667e`, so W1 could only record the difference. The matching aarch64 wheel exists and now installs: `JITCACHE_STATE=INSTALLED`, `flashinfer-jit-cache==0.6.12+cu130`. Without it the first request compiles kernels in-process and the first leg measures the compiler. **The manifest no longer lands unreached.** `scripts/sglang_lease_identity.py` reads it in the lease and `tests/scripts/test_sglang_lease_identity.py` reads it in CI, where it is the only executing code that opens the file. That suite also mutates every guarantee the gate claims — one changed byte, a missing file, an extra file, a cwd that is not `/`, and a byte-identical SOURCE TREE — and each one goes red. Every mutation asserts that it APPLIED before reading the verdict, because a mutation that never applied reads as a passing test. `scripts/dgx-sglang-low-concurrency.sh` is NOT patched. Its missing half is the forbidden path, which is why W1 said it owed a new driver rather than a repair. The replacement is `scripts/rc-sglang-oracle-lease.sh`. ## What this does NOT establish, and why the flag does not move No model has been served by this route. The serving job is submitted and queued behind two other sessions on the same device, which is the lease working rather than failing. `AGENTS.md` §"Measure gateability" wants the oracle to "demonstrably build and run the model", and the spec's five exit conditions require all five in ONE recorded run. Two are met. Three are not. `gateable` stays `no`, `evidence` stays `#1265`, and the spec carries the verdict as a table so a later reader cannot mistake two-of-five for five. `LGC_RC=4` reproduced #1354 on a fourth job: inside a lease the SM clock is sampled and never pinned. Nothing in this change rests on the fenced `flashinfer_cubin-0.6.12` architecture-token reading, which W1 inherited and W2 did not re-derive either. No issue is closed. #1265 stays open for the model run. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
Bring the branch up to `af320abb2` so the commit-trailer and commit-style gates EXECUTE rather than skip. A branch behind `origin/main` makes both report nothing about this tree, which reads as green and is not. No conflict, and no file this row touches moved on `main`. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…ns that RAN, and the trap that made a second filename necessary (#1265) The header printed a plausible recipe rather than the one used. Two things were missing and both cost something to learn. `--idle-timeout` was absent. The job's quiet phases are a gigabyte-scale download and a 52 GiB copy, and the idle killer counts the job's own stdout. The heartbeat inside the script is what covers that, and the header did not say so, so a reader could remove one without knowing it guarded the other. The serve phase is staged as `run2.sh` and not as a rewrite of `run.sh`. bash reads a script LAZILY, so overwriting the file a running job is executing corrupts that job. On 2026-08-23 the install job was mid-`pip install` when the serve script was ready, and staging beside it rather than over it is the only reason both ran. That is a property of the fleet, not of this script, so it belongs in the recipe. The staged copy at `/workspace/sglang-w2/run2.sh` is byte-identical to this file, sha256 `3a610c59b32eaf8d8b7ef7472c3d9fa84b4eca0f0b3a0c72c250e4bd159c50d3`. No behaviour changes; the diff is comment lines only. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…n a lease, so gateable goes yes — and the box rebooted under the next leg (#1265) The serving job ran. Five of five exit conditions hold, so `gateable` moves to `yes` and `evidence` moves off `#1265`. The route is a PyPI wheel in a virtual environment inside an `rc` lease: no `ssh`, no container image, no file mutex outside the lease. ## Conditions 3 and 4 Job `b9e7709d`, `dgx:gpu0`, from 21:51:56Z. The virtual environment from the install job survived, so it re-asserted `IDENTITY_RC=0` at 3338 of 3338 in eleven seconds and never touched PyPI — a second assertion on a second job. It copied the 55.6 GB checkpoint off CIFS and served `Qwen3_5ForConditionalGeneration` to readiness in **454 s** on `/health_generate`, with decode CUDA graphs captured at bs 1 to 32. The backends come from the server's own log, not from the source reading that predicted them: attention_backend='flashinfer' moe_runner_backend='auto' mamba_backend='triton' linear_attn_backend='triton' "Attention backend not specified. Use flashinfer backend by default." "Using hybrid linear attention backend for hybrid GDN models." `moe_runner_backend='auto'` is NAMED and UNEXERCISED. Qwen3.8-27B has no expert layer, so that is a declared default of a path this model never enters. A MoE arm is owed and the spec says so. One timed c1 leg: **6 of 6 requests, 0 failed, 768 output tokens — exactly 6 x 128 — in 173.23 s** over `/v1/completions`, the client path the 2026-07-28 image run drove. **The warm JIT cache is measurable.** The discarded warmup reached 4.431 output tok/s and the timed leg 4.433, a 0.05% difference. That is what installing `flashinfer-jit-cache` bought, and it is the reason W1 was right to call that difference closable rather than acceptable. ## The c8 leg is VOID because the machine rebooted under it Everything stopped at 22:27:00Z. My first reading was that the job had been killed, because the client streaming it had just died and this fleet is recorded as killing a job whose client detaches. **That reading was wrong**, and the field that falsified it is one no part of the job was watching: the next job on `dgx:gpu0`, at 22:32:47Z, printed `boot_id=26394f62…` against this row's `02d5a76f…`, with the kernel PID counter down from 3510 to 594. Same pod name on both sides. The box rebooted. `.agents/environment.md` told this row to print `boot_id` in every leased job that loads anything large. That instruction is the only reason the cause is known, and it is why the spec now carries a reboot rather than a guess. **The watchdog did not fire and could not have.** Its floor was 5,000 MB; the last eight samples read 17074, 17251, 17269, 17296, 17323, 17338, 14935 and 15449 MB. The machine died with 15 GB available. That reproduces a rule already in `.agents/environment.md` — a sampling watchdog cannot guard the reboot class of failure, because a userspace sampler dies with the kernel — and the reproduction is the only new part. The floor stays in the script because it costs nothing, and it is not protection. The reboot is also what answers the teardown question the job never got to print: a reboot returns everything, and another session's job ran on the box five minutes later. The assertion is still made rather than inferred, by `scripts/rc-sglang-lease-reap.sh`, which matches the venv interpreter path and never a launcher name. ## What the flag does NOT claim Reachability, not a floor. No vllm.cpp arm ran beside any of this, the clock cannot be pinned in a lease and spread 7.59% against the 5% ceiling at 84 C with software thermal slowdown active, and **no ratio is derivable or offered**. `docs/benchmarks/open-gaps.md` still lists every SGLang floor arm as unrun, and #1265 stays open for the c8 point, the MoE arm, and the teardown assertion. `evidence` points at the spec rather than at a new file under `docs/bench-evidence/`, because a second copy of this record is a measurement of one file stored inside another and nothing would keep it in sync. One stale record rides along: `.agents/environment.md` said this route was specified and not run. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
Bring the branch up to `3574065e7` so the commit-trailer and commit-style gates EXECUTE rather than skip. A branch behind `origin/main` makes both report nothing about this tree, which reads as green and is not. One conflict, in the `SUITES` array of `scripts/agent-preflight.sh`: main appended three suites while this branch appended one. Resolved the way AGENTS.md §Records prescribes for a keyed record rather than by taking git's word for it — the COMPLETE target-branch file was taken, the one scoped edit was applied to it again, and `diff` against `origin/main` was run to confirm the result differs by exactly the single added line. An automatic three-way merge of an array like this is usually right and occasionally drops an entry, and nothing downstream would have said so. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…ng job never printed, made by a job that did (#1265) Job `b9e7709d` died with the box under it, so it never reached its own teardown and never said whether anything of ours was left holding the GPU. Inferring that from "a reboot returns everything" is reasoning, not an assertion, and this repository has been wrong about a stranded engine before: a `pkill -f` on a launcher name once left an EngineCore holding 23 GB across three jobs. So a job asserted it. `0f84b66d-1c30-4de5-bdb8-ee7b058f284a` on `dgx:gpu0` at 23:10:48Z: boot_id 26394f62-37c5-4fc9-885a-c8faba9d35ac COMPUTE_APPS=0 SGENV_PROCS=0 NOTHING STRANDED: no process is running our venv interpreter. It matches on the venv INTERPRETER PATH, `/tmp/sgenv/bin/python`, and never on a script name, which is the match that caused the strand it is guarding against. Three things follow. The GPU holds no compute process. Nothing is running this row's environment. And the `boot_id` it read is the post-reboot one, so a THIRD job confirms the reboot rather than the same reading being quoted twice. `/tmp/sgenv` and `/tmp/ckpt38` are both gone: the `du -sh` of each printed nothing. A re-run therefore pays the whole install again, about 28 minutes plus 1.75 GB for the JIT cache, and another 20 minutes to re-copy the 52 GB checkpoint off CIFS. The `serve` phase already reinstalls when the environment is absent, which is why that path exists. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
… as prose, the reboot table named no tree, and c8 never issued c8 (#1265) Three record-accuracy findings from the fresh review of #1831. The verdict is unchanged and no measured number moves. THE REAP JOB ARCHIVED NOTHING. `scripts/rc-sglang-lease-reap.sh` wrote only to stdout. Unlike its siblings it never set `$OUT` and never ran `exec > >(tee -a "$OUT/job.log")`, so `/mnt/nas_share/rc/sglang-w2/out/` held no `reap-*` directory and a sweep of the whole share found `DONE_MARKER_SGLANG_W2_REAP` in the script and in no output anywhere. `COMPUTE_APPS=0`, `SGENV_PROCS=0` and the job id were prose in one spec, and that prose was striking through an `## Owed` item. The output was not lost, though, and the honest repair is to say where it lives rather than to withdraw the closure. The resource controller keeps its own copy: `rc logs 0f84b66d-1c30-4de5-bdb8-ee7b058f284a` returns all 412 bytes, and the spec now quotes them verbatim instead of paraphrasing four of them. A copy sits at `out/reap-20260823T231048Z/job.log` with a `PROVENANCE.txt` saying it is post-hoc and not a job-written artifact. The script now tees like `install` and `serve`, and refuses rather than running unarchived if it cannot create `$OUT`. THE REBOOT TABLE NAMED NO TREE. Two of its four rows are another row's artifact, cited as "another session's, on the same box". They are `/mnt/nas_share/rc/gdn-moe-packed-ba/logs/gate-ab.log` lines 1-3 and 18-20, and the table now says so, prints those six lines, and gives line 8 of this row's own two job logs for the other two. CONCURRENCY 8 WAS NEVER ISSUED. The spec said the collapse "landed inside an 8-concurrency leg" and advised lowering the concurrency. `sglang-c8.log` is 0 bytes, and the server log's last four entries are the whole life of that leg: a readiness `GET /v1/models` at 22:26:41, one `Prefill batch, #new-seq: 1` at 22:26:47, one `Decode batch, #running-req: 1` at 22:26:54, then nothing. The leg had reached only its `warmup_requests=1` request; the 48-prompt main loop never started. ONE request was in flight at the moment of death, so the load was c1 and lowering the concurrency is a guess, not a lever this run supports. Why the box went down stays open, and no replacement guess is offered. The same wording is corrected in the oracle file and the matrix row, which both carried it. Two pre-existing gate-scope items the review raised are filed rather than fixed, because neither is this change's defect and each needs work outside its scope. #1832: the manifest count 3338 is gated only against itself, and dropping a real file while decrementing the header leaves the suite at rc=0. #1833: both registrations of the identity suite are deletable at rc=0, and a control on an unrelated suite shows the SUITES array itself is ungated. Both are indexed and listed under `## Owed`. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
Bring the branch up to `df1ee2058` so the commit-trailer and commit-style gates EXECUTE rather than skip. A branch behind `origin/main` makes both report nothing about this tree, which reads as green and is not. No conflict. `.agents/issue-index.md` auto-merged under the union driver, which is exactly the shape that has silently dropped a tail row here before, so the result was checked rather than trusted: 662 rows, both of this change's appended rows present at the tail, and no duplicated issue number. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
Bring the branch up to `4345f34a5` so the commit-trailer and commit-style gates EXECUTE rather than skip. A branch behind `origin/main` makes both report nothing about this tree, which reads as green and is not. One conflict, in the `SUITES` array of `scripts/agent-preflight.sh`: main appended `test_ci_walk_base` while this branch appended `test_sglang_lease_identity`. Resolved the way AGENTS.md §Records prescribes for a keyed record rather than by taking git's word for it — the COMPLETE target-branch file was taken, this branch's one scoped edit was applied to it again, and `diff` against `origin/main` confirms the result differs by exactly the single added line. An automatic three-way merge of an array like this is usually right and occasionally drops an entry, and nothing downstream would have said so. `.agents/issue-index.md` auto-merged under the union driver, which is the shape that has silently dropped a tail row here before, so the result was checked rather than trusted: this change's two appended rows are present and no issue number is duplicated. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
localai-bot
changed the title
Bring the branch up to `4f0d44ca2` so the commit-trailer and commit-style gates EXECUTE rather than skip. The previous run reported SKIP for both with the reason that `origin/main` was not an ancestor of HEAD, which reads as a clean run to a caller that looks only at the status and is not one. `4f0d44ca2` is the hugo lane fix (#1830). It and this branch both edit `.github/workflows/ci.yml`, so the merge was verified by hand rather than trusted: both `HUGO_VERSION` keys, the `peaceiris/actions-hugo` setup step, the `test_ci_site_lane` step and this branch's `test_sglang_lease_identity` step are all present, the `SUITES` array still holds exactly 61 entries with none dropped, and `.agents/issue-index.md` kept its tail rows (#1828, #1754, #1833) rather than losing one to the union driver. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
Bring the branch up to `0043efb43` so the commit-trailer and commit-style gates execute against current `main` rather than skip. `.github/workflows/ci.yml` and `.agents/issue-index.md` both auto-merged. AGENTS.md forbids taking git's word for a keyed record, so both were checked by hand: `ci.yml` still carries the `peaceiris/actions-hugo` step, both `HUGO_VERSION` keys, the `test_ci_site_lane` step and this branch's `test_sglang_lease_identity` step, parses as 19 jobs with 51 steps in `agent-record`, the `SUITES` array still holds 61 entries, and the index kept rows #1828, #1832 and #1833 while gaining main's new row. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
Bring the branch up to `e4b47d218` so the commit-trailer and commit-style gates execute against current `main` rather than skip. `scripts/agent-preflight.sh` conflicted in the `SUITES` array: `main` appended `test_rc_stage_checkpoint` while this branch appended `test_sglang_lease_identity`. AGENTS.md forbids accepting an automatic three-way merge of a keyed record, so the resolution took `origin/main`'s COMPLETE file and re-applied this branch's single scoped edit on top. `git diff --numstat origin/main -- scripts/agent-preflight.sh` is `1 0`: one line added, none removed, which is the proof that no unrelated entry was lost. `.agents/issue-index.md` union-merged and kept rows #1828, #1832 and #1833. `.github/workflows/ci.yml` still carries the `peaceiris/actions-hugo` step, both `HUGO_VERSION` keys, the `test_ci_site_lane` step and this branch's `test_sglang_lease_identity` step, and parses as 19 jobs. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The SGLang oracle has been
gateable = nosince 2026-08-18, because the routethat measured it on 2026-07-28 was a container image and
rc runhas no--image. This lands the replacement route: the two PyPI wheels, installedinside an
rclease, in the order upstream's owndocker/Dockerfile:210,242installs them.
Both wheels are hashed inside the job, not before it, and both matched. The
installed tree is asserted by content against the committed manifest: 3338 of
3338 files, 0 missing, 0 extra, 0 differing, at pin
f63458b5. That assertionis the only one available, because the PyPI wheel sets
__commit_id__ = Noneand so carries no runtime claim about the commit it was built from.
The oracle then served
Qwen3_5ForConditionalGeneration(Qwen3.8-27B bf16,55.6 GB) to readiness in 454 s with decode CUDA graphs captured, and completed
one timed c1 leg through
/v1/completions: 6 of 6 requests completed, 0 failed,768 output tokens against 6 x 128 expected.
gateabletherefore returns toyeson the wheel route.No speed ratio is derivable from this run and none is offered. The c8 leg is
recorded VOID because the host rebooted DURING it at 22:27:00Z: the next job on
dgx:gpu0readboot_id=26394f62against this row's02d5a76f, with thekernel PID counter down from 3510 to 594, in
/mnt/nas_share/rc/gdn-moe-packed-ba/logs/gate-ab.loglines 1-3 and 18-20. The5,000 MB
MemAvailablewatchdog never fired and could not have, because themachine died with 15,449 MB available, which is a sampling watchdog failing to
guard the reboot class. The clock spread was 7.59% against a 5% ceiling at 84 C.
moe_runner_backendis named by the server's own log but is unexercised,because a dense model has no expert layer.
DURING, and not under. That leg is not a concurrency-8 datapoint of any kind:
sglang-c8.logis 0 bytes, and the server log's last four entries are its wholelife — a readiness
GET /v1/modelsat 22:26:41, onePrefill batch, #new-seq: 1at 22:26:47, one
Decode batch, #running-req: 1at 22:26:54, then nothing. Theleg had reached only its
warmup_requests=1request and the 48-prompt main loopnever started, so ONE request was in flight at the moment of death. Why the box
went down is OPEN, and lowering the concurrency is not a lever this run supports.
The teardown assertion the rebooting job never printed was made separately by
scripts/rc-sglang-lease-reap.shin job0f84b66dat 23:10:48Z —COMPUTE_APPS=0,SGENV_PROCS=0. That script wrote only to stdout and, unlike its siblings, neverteed to
$OUT, so it archived nothing: the output survived only in the resourcecontroller's job store and is read back with
rc logs 0f84b66d-1c30-4de5-bdb8-ee7b058f284a, quoted verbatim in the spec andcopied post-hoc to
out/reap-20260823T231048Z/job.logbeside aPROVENANCE.txtsaying it is a copy. The script now tees like
installandserve.scripts/dgx-sglang-low-concurrency.shstays unrunnable and is deliberatelyNOT patched; it has no execution half to patch. The replacement driver is
scripts/rc-sglang-oracle-lease.sh.The identity gate runs inside a lease, which CI has none of. What CI holds is
that the gate goes red on each defect it promises to catch, and that the
committed manifest agrees with the pin block it stands in for. That suite is
also the only executing code that opens the manifest, which is what stops a
453 KB record landing unread.
#1265 stays open for the owed items: the c8 point, a MoE arm, any ratio, and a
watchdog that works on this box.
Two pre-existing gate-scope items raised by the fresh review are filed rather
than fixed here, because neither is this change's defect and each needs work
outside its scope. Both are indexed in
.agents/issue-index.mdand listed under## Owedof the spec.3338is gated only against itself. It appears in noexecuting code, and the only test touching it compares
file_countwithlen(files)in the same document, so dropping a real file while decrementingthe header leaves the suite at
rc=0on a manifest asserting a different tree.Repairing it needs a re-derivation on a second independent install.
rc=0—the
SUITESentry and the whole CI step alike — and a control on an unrelatedsuite behaves identically, so the
SUITESarray itself is ungated. Repairingit means giving
check-test-registration.pya population rule, which is asemantic checker change owing its own row and red-first evidence.
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]