feat(#810 A3): NemotronH gets its ABI driver, and the reason its gate has not run gets corrected - #1128
Merged
Merged
Conversation
… has not run gets corrected The A2-P spec's `## Owed` lists three things. Two land here. The third did not run, and the reason on record for why it has not run turned out to be false. `examples/nemotron_h_gen` is the driver the governing spec §6.1 describes: modelled on `kimi_linear_gen`, one project include (`vllm.h`), linked against `vllm::shared`, and absent from `example-abi-allowlist.txt` because it reaches no internal header. `deepseek_v4_gen` and `laguna_gen` were not copied; both drive a bespoke forward through internal headers and are the transition state that allowlist exists to retire. The golden is JSON rather than Kimi's `.npy` plus raw-i32 pair, so the driver carries a scanner for that one committed shape. It refuses a shape it does not recognise rather than returning an empty vector, because an empty vector downstream is a comparison over zero elements and a comparison over zero elements reports a perfect score. That is also why a pass needs three things at once, checked in the driver instead of left to whoever reads the log: something was compared, everything compared matched, and every row was compared to the golden's full width. Those guards were exercised against a real engine rather than argued for. On a small local checkpoint the driver exits 0 on a full-width match, 1 on a divergence, 2 on each of five malformed goldens, and 4 when `--steps 4` is run against a width-8 golden. That last one matched every token it looked at and still refused, which is the case a driver without the check would have reported as 4/4. `docs/USAGE.md` gains the weights block AGENTS.md wants in the change that makes a capability reachable: repo, revision, staged path, byte total, the sha256 of the first shard, and every arm named including the four refused ones. The revision was verified rather than copied. Its stale row claiming the forward refuses the first step is corrected, because a row that reads as broken after A2-P fixed it is as wrong as one that reads as working. The larger correction is the pending cause. A2-P recorded the gate as pending on CONTENTION, at loadavg 211 with 3 of 119 GB free. Re-measured under a lease that box is idle: loadavg 0.36, 115 of 119 GB, GPU 0%, checkpoint present and revision-verified against its own LFS record. The gate is blocked on something else, that nothing on that host can build a CUDA binary, and the three ways to try are enumerated in the benchmark record with the host facts separated from the agent-access one. #775 is the precedent: a cause left standing long enough stops being read, and by then it has changed underneath. `scripts/runner-routing-allowlist.txt:26` stays, and that was decided by mutation rather than by reading. Deleting it takes the routing checker from OK to ERROR naming `ForwardNemotronHForCausalLM returns HostLogits`; the tree was restored byte-for-byte. A2-Q2b removes it, and the allowlist was not widened. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
…ch machine it was measured on The previous commit recorded a pending cause built on a probe that ran on the dgx HOST while describing the `rc` worker CONTAINER. For this purpose those are different machines, and reporting one as the other produced a confident, specific and wrong conclusion: that no CUDA binary could be built at all. Measured inside `rc run`, the container is Ubuntu 24.04 running as uid 0 and carries gcc, g++, cmake, ninja, make, python3, git and apt, with the GB10 visible to nvidia-smi and working DNS. Only `nvcc` is genuinely absent, and apt's own nvidia-cuda-toolkit is 12.0.140, too old for sm_121a, so CUDA 13.x installs from the NVIDIA repo. Neither docker nor sudo is involved anywhere, and the ask for them is withdrawn. The HOST toolchain finding is real and stands, but it gates nothing, because the host is not where work runs. The checkpoint sentences were wrong the same way. `findmnt` on the host shows `/usr/local/nas_share` mounted from `//192.168.68.102/Data` over cifs, and the checkpoint resolves under BOTH that path and `~/ckpt`, so `.env`'s CHECKPOINT_ROOT is correct and must not be "fixed". The mount line is quoted in the benchmark record rather than paraphrased, so the record carries the measurement instead of an attribution. Whether the CONTAINER can see that path is left OPEN and is claimed by nobody: the probe queued to answer it was killed to free the queue, and step 6 of the A3 job is now the only thing that will settle it. Corrected in place rather than quietly replaced. A known-false claim left inside a landed record is worse than the original error, because the next reader cannot tell it apart from the parts that were right, and the two-machine confusion is the part worth keeping. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
`origin/main` moved again while the record corrections were being written, including ENV-GPU-LEASE-METHODOLOGY (#1141), which is the policy this row's own lease discipline follows. None of the incoming commits touches `examples/`, `docs/USAGE.md` or the NemotronH sources, and the merge reported no conflict. A clean merge is not a building merge, so the merged tree is rebuilt rather than assumed before the gate is rerun. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5 [claude-code]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The A2-P spec's
## Owedlists two things this row carries and one it cannot:the thin ABI driver, the
docs/USAGE.mdweights block that rides with it, andthe A3 end-to-end token gate. The first two land here. The third did not run,
and the reason on record for why it has not run turned out to be wrong.
Issue: #810.
examples/nemotron_h_genModelled on
examples/kimi_linear_genexactly as the governing spec §6.1requires: one project include (
vllm.h), linked againstvllm::shared, and noscripts/example-abi-allowlist.txtrow, because it reaches no internal header.check-surface-coverage.pycounts 18 example units with the same 8 reachinginternal headers as before.
deepseek_v4_genandlaguna_genwere deliberatelynot copied; both drive a bespoke forward through internal headers and are the
transition state that allowlist exists to retire.
The golden is JSON rather than the
.npyplus raw-i32 pair Kimi's battery uses,so the driver carries a small scanner for that one committed shape. It refuses a
shape it does not recognise instead of returning an empty vector, because an
empty vector downstream is a comparison over zero elements and a comparison over
zero elements reports a perfect score.
The guards are proven armed against a real engine, not argued for. The
driver is model-agnostic, so it was exercised on a small local checkpoint:
STRICT PASS, exit 0DIVERGENCE, exit 1--steps 4against golden width 8, 4 of 4 matchedSHORT, exit 4The third row is the one that matters. It matched every token it looked at and
still refused, because it had looked at half the golden.
The A3 gate has not run, and its cause was wrong TWICE
First cause, dead: contention. The A2-P spec records
dgx.casaat loadavg 211with 3 of 119 GB. Re-measured under a lease, the box is idle at loadavg 0.36 with
115 of 119 GB free and the GPU at 0%, checkpoint present and revision-verified.
Second cause, also dead, and it was mine: this branch briefly claimed no CUDA
binary could be built at all. That was a HOST measurement reported as a
CONTAINER measurement, and for this purpose they are different machines.
Measured inside
rc run, the worker container is Ubuntu 24.04 running as uid 0with
gcc,g++,cmake,ninja,make,python3,gitandaptpresent,the GB10 visible to
nvidia-smi, and working DNS. Onlynvccis absent, andapt's
nvidia-cuda-toolkit12.0.140 is too old for sm_121a, so CUDA 13.xinstalls from the NVIDIA repo. Neither
dockernorsudois involved, and theearlier ask for them is withdrawn. The host toolchain finding
(#1019) is real but gates
nothing, because the host is not where work runs.
The checkpoint sentences were wrong the same way, and are corrected against
findmntrather than paraphrase:The checkpoint resolves under BOTH
/usr/local/nas_share/checkpoints/...and~/ckpt/..., so.env'sCHECKPOINT_ROOTis correct and was not "fixed".Corrected in place, with the wrong claim kept beside the correction. A
known-false line left inside a landed record is worse than the original error,
because the next reader cannot tell it apart from the parts that were right, and
the two-machine confusion is the part worth keeping.
Genuinely outstanding: install
nvccin the build container, and whether thatcontainer can see
$CHECKPOINT_ROOT, which is OPEN and claimed by nobody.docs/BENCHMARKS.mdrecords the gate as pending a named resource, never as apass. Arm 2 of the gate (three prompts concurrent and interleaved) stays blocked
by design: G-SAFE refuses
num_reqs > 1and A2-B owns that clause.scripts/runner-routing-allowlist.txt:26STAYS, decided by mutationnemotron_h.cpp:1031-1034still refuses the NVFP4lm_headon a non-CPU queue,so the forward returns
HostLogits. Deleting the entry in a scratch copy takescheck-runner-routing-consistency.pyfromOKtoERRORnamingForwardNemotronHForCausalLM returns HostLogits, exit 1; the tree was restoredbyte-for-byte. A2-Q2b removes it. The allowlist was not widened.
Records
docs/USAGE.mdgains the weights block AGENTS.md requires in the change thatmakes the capability reachable: repo, revision, staged path, byte total, the
verified sha256 of the first shard, and every arm named, including the four
refused ones. Its stale "refuses to run" row is corrected, and it says plainly
that no token gate result exists yet. No lifecycle state changed, so no
docs/STATUS.mdwrite is owed.FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]