Skip to content

record(MODEL-TEXT-nemotron-h): the row is blocked on a kernel that landed, and a file glob is why nobody saw it - #1081

Merged
localai-bot merged 28 commits into
mainfrom
row/RECORDS-NEMOTRON-H-MATRIX-1074
Aug 22, 2026
Merged

record(MODEL-TEXT-nemotron-h): the row is blocked on a kernel that landed, and a file glob is why nobody saw it#1081
localai-bot merged 28 commits into
mainfrom
row/RECORDS-NEMOTRON-H-MATRIX-1074

Conversation

@localai-bot

@localai-bot localai-bot commented Aug 16, 2026

Copy link
Copy Markdown
Collaborator

The NemotronH model-matrix row said the architecture was BLOCKED on
KERNEL-SSM-MAMBA (#496)
because "the Mamba2 SSD core is unported". That kernel landed on 2026-08-13.
The row also carried ☐ required for its spec and none for its evidence while
five specs were committed and the paged forward, the device MoE arm and the ABI
driver had all landed. This change corrects the record. It does not move the
row's lifecycle state.

FOLLOWING_AGENTS_PROTOCOL

Why a directory listing hid the blocker

#496 landed its host arm at
47960a009 (W1) and its CUDA arm at 43a6c5518 (W2). The kernel is
src/vt/cuda/cuda_mamba2_ssd.cuh — a 692-line header, #included at
cuda_gdn.cu:48 and registered as kMamba2ChunkScan at cuda_gdn.cu:6669, not
a translation unit of its own. A src/vt/*mamba* FILE GLOB therefore returns
nothing and reads as absence, and that glob is the search both the row's text and
#1074 rested on.
nemotron_h.cpp:597 calls vt::Mamba2ChunkScan today, and nemotron_h.cpp:354
calls vt::MoeRelu2. #496 stays open for its GENERIC MambaSpec producer, which
this architecture does not need, because A1 made the runner read the model's own
KV spec.

Worth carrying: #496's own issue TITLE still asserts "Mamba2 SSD core is
unported"
, so the false claim has a second live copy this change cannot reach.

Three claims in the row were FALSE and are corrected: the KERNEL-SSM-MAMBA
block, the non-gated relu² MoE "exists nowhere locally", and ModelOpt
MIXED_PRECISION loading "exists nowhere locally". One was TRUE and stays: the
DeepSeek-style MTP head does not exist locally, and it remains owed as W5.

The row STAYS INVENTORIED

This is the part an earlier draft of this description got wrong, and the
correction is the point of the change. That draft moved the row INVENTORIED ->
PARTIAL with the mark 🚧, the rollup INVENTORIED 324->323 and PARTIAL
22->23, and the two prose counts 53->54. None of that is in this diff.

Measured on this head against origin/main:

  • the row's lifecycle cell is `INVENTORIED` on both sides;
  • the rollup table .agents/model-matrix.md:76-86 is BYTE-IDENTICAL to main's
    INVENTORIED 324, PARTIAL 22, ACTIVE 10, SPIKE 9, BLOCKED 5, DONE
    3, READY 3, GATING 1, total 377;
  • scripts/check-agent-record.py derives MODEL=377 and its model inventory
    {rows: 324, memberships: 373, architectures: 356, targets: 310, modules: 261}
    unchanged.

The lifecycle move is left to whoever re-runs the gate against main. What
survives is the part that was never about a state change: the row's FALSE claims,
and its corrected code and test anchors.

The A3 gate PASSES, and it belongs to a named tree

An earlier draft recorded this gate as PENDING, because it read
#1221's pull request BODY as the
current state. That body's "Still owed: the sm_121a green-after re-run" was true
when written and stale by the time it was read. A body is a snapshot of its
author's knowledge at write time, not a live field.

The artefact is on the NAS at /usr/local/nas_share/rc/nh1157/. Verbatim from
gate_fixed.out:

TOKEN MATCH: 96/96 over 3 prompt(s) (full rows=3, short rows=0, mode=decode)
STRICT PASS

against the pinned oracle vllm=0.23.1rc1.dev1511+g555967922 on
nemotron-3.5-lightning-30b-nvfp4 at revision
29f2d1746d8f41e316523194b19018707749b1b1.

It is the DEVICE leg. The binary is libvllm 0.0.3+cuda, and cfg.log
records fp4-mma, cutlass-nvfp4 and cutlass-fp8 as ENABLED for [121a]. The
run logs Asynchronous scheduling is enabled (max_concurrent_batches=2), which
is precisely where device_token_ids is non-null; on the host queue it is always
null and the #1157 defect
cannot arise. The decisive evidence is neither of those: the same binary on the
same checkpoint with only nemotron_h_device.cpp reverted to the fix's
parent scores 4/24 (full rows=0, short rows=3) and bails at 8 generated tokens.
A host-leg run would have been unmoved by that revert. The delta is the proof;
the pass on its own is not.

The pass was measured on the tree that landed as 0ea5d249f, and there is no
run against current main.
#1221 MERGED on 2026-08-18 as 0ea5d249f, which is
main's last touch of nemotron_h_device.cpp, so the repair the 96/96 depended
on is on main. That is not the same fact as a gated main: the 96/96 was taken
on that branch tree, main has advanced since, and nobody has re-run it. Every
row here therefore names the SHA the measurement belongs to and records no pass
on main. An earlier draft of this description instead called that branch "OPEN
and CONFLICTING" with "the merge is owed", and quoted a shipped cell as reading
"on #1221's OPEN branch, not on main". Neither is true here, and that quoted
text ships nowhere: grep -c "OPEN branch" over docs/ and
.agents/model-matrix.md is 0.

No throughput, latency or memory figure is claimed anywhere in this change.
gate_fixed.out carries 264.4s to load and 327-343s per 32-token prompt. That is
a correctness run whose lm_head and 46 FP8 mamba projections still execute
host-side. Both public rows say "NOT a benchmark" in those words. One config
caveat travels with the run: --gpu-memory-utilization 0.92 did not size the KV
pool, which fell back to 256 blocks
(#83).

The stale ## Owed bullet for #1217

The ## Owed entry for #1217
in .agents/specs/nemotron-h-model.md was written while #1221 was open, and this
branch ADDED it: origin/main carries the phrase "open and unmerged" zero times.
It was false three ways at its own head, and the last commit repairs it.

the bullet said this head has
#1221 "is open and unmerged" merged 2026-08-18 as 0ea5d249f
#1217 "is recorded NOWHERE in this tree" .agents/issue-index.md:404 and nemotron-h-a2p-paged-forward.md:901
"the standing reason this row's end-to-end gate is not a pass" the same file's ## Now records 96/96 STRICT PASS

The "recorded nowhere" claim WAS true at b626be75a, which is verifiable —
git show b626be75a:.agents/issue-index.md | grep -c 1217 is 0. The merge
8a00afcbe brought both records in. Both anchors were re-derived at this head:
issue-index.md:404 carries the row under the owning row
MODEL-NEMOTRON-H-ABI-A2P, and nemotron-h-a2p-paged-forward.md:901 is the
first bullet under that spec's ## 11. Owed.

The duplicate-key argument for withholding an index row is kept as history and
marked expired. Appending a row now would create the very duplicate that argument
was written to avoid, because the row is already on main. Nothing is owed to
this reconcile for #1217.
The issue stays open as a seam defect owned by
MODEL-NEMOTRON-H-ABI-A2P.

Two defects found and fixed in flow

docs/BENCHMARKS.md carried the key NemotronH paged forward twice on this
branch and once on main, because the branch APPENDED where it should have
UPDATED, and the two rows contradicted each other (4/24 with a re-run pending,
versus 96/96). They are collapsed to one row on the key. grep -c for that key
is now 1 here and 1 on main. Nothing gates this:
scripts/check-public-doc-tables.py holds the page's size and shape, not its key
uniqueness. The row's position moved when the duplicate was collapsed and it now
follows Record-anchor ratchet; the key and content are correct, and reordering a
file that other pull requests edit was not worth the conflict.

docs/STATUS.md was restructured under this branch by
#1491, which reorganised it by
surface and deleted the per-model table, so the merge conflicted for real. The
page is resolved to main's whole and the record is re-expressed as a one-line
in-place edit to the Text generation row.

A latent defect this branch introduced, and the replay that removes it

The row's corrected cell-6 prose linked #496's spec inline as
[#496](…) ([spec](specs/mamba2-ssd.md)). scripts/check-doc-checkpoint.py's
spec_for_row takes the first specs/*.md link anywhere in the row line, so
that parenthetical shadowed the row's own spec: MODEL-TEXT-nemotron-h resolved
to mamba2-ssd.md instead of nemotron-h-model.md. On origin/main the same row
resolves correctly to nemotron-h-model.md, so this branch introduced it.

This is not a gate artifact, and the squash would not have absorbed it. The
shadowing lands and persists: every future lifecycle move of this row would
demand a ## Now from mamba2-ssd.md, a spec whose author has no reason to
touch it, with nothing pointing back at the row that caused it. The fix is
correct on its own merits; the gate going red was how it was found, not why it
is being made.

It surfaced because #1577, #1609 and #1621 landed on main under this branch and
gave the checker sight of the states it had been blind to — PARTIAL joined
STATES, INVENTORIED joined the new RECORD_STATES. 414a8002b moves this row
PARTIAL -> INVENTORIED, which is a real move the old checker could not see.

errors_for is strictly per-commit (commits_in_range -> errors_for(paths, parent, commit)), so no later commit can pay the debt. The 27 branch commits were
therefore replayed with git filter-branch --index-filter, scoped
rewrite-1081 --not origin/main so that no commit of main was rewritten.

Proof, not assertion:

  • origin/main at replay time (e100e64e1) was NOT rewritten: it still resolves
    as itself and remains an ancestor of the head.
  • git diff <old-head> <new-head> is one file, one line — the parenthetical.
  • All 27 old/new pairs compared: the only path that differs is
    .agents/model-matrix.md, and author name, author email, author date,
    committer name, committer email, committer date and subject are byte-identical
    on every pair. The five commits that never contained the string kept their
    original SHAs, because the filter is a no-op on them.
  • The shadowing string now occurs 0 times across all 27 commits, and
    mamba2-ssd.md is still linked from the row's Spike/spec cell, so no
    reference is lost.

Which projections this change owes, and why

scripts/check-doc-checkpoint.py was instrumented on this exact diff rather than
reasoned about. classify() returns classes {'feature_surface'} and
lifecycle_moves [], and MEASUREMENT_RECORDS is
('.agents/benchmark-record.md',), which this change does not touch.

An earlier revision of this description said the checker's STATES tuple
"contains neither INVENTORIED nor PARTIAL, so it is structurally blind to
this row's state". That was true when measured and is false at this head, and
the conclusion survives while the reason inverts. After #1577, #1609 and #1621 the
gate SEES both states and reports no lifecycle move across origin/main -> head,
because the row genuinely does not move: INVENTORIED on both sides. A gate that
looks and finds nothing is a stronger result than a gate that cannot look.

docs/STATUS.md, docs/BENCHMARKS.md and the spec's ## Now are therefore
written because AGENTS.md ## Public documents owes them, not because a gate
asked. docs/FEATURES.md IS touched, at :151: the NemotronHForCausalLM row
said nothing about the end-to-end token gate before, and its correctness column
now reads "A3 e2e gate 96/96 STRICT PASS on GB10 at 0ea5d249f (#1221); NO run
against current main".

The docs/STATUS.md edit takes oversized_cells from 2 to 3 against a ratchet of
44, and long_paragraphs stays 0 against 75. No ratchet is raised.

.agents/issue-index.md

GitHub does not run the merge=union driver .gitattributes:7 sets, so this
branch has repeatedly discarded the union auto-merge, taken main's file whole,
and re-appended only the rows whose KEY main lacks: #1074 and #1080. main
already carries a #1068 row, so that one is not re-appended.

Measured on this head against origin/main: 565 rows, of which main's 563 are
all present, in order, byte-identical; 0 removed; exactly 2 added.

scripts/check-issue-index-append-only.py exits 0. The file is not a strict BYTE
prefix of main's, because main's newer rows follow this branch's two; the
append-only rule is about rows rather than bytes.

#1080 is re-measured rather than carried: after this change repairs
nemotron-h-model.md's ## 7. Now to ## Now, 15 specs still write the
numbered spelling, up from the twelve #1080 was filed against. A growing
population is the argument for the checker-semantics fix over a rename sweep, and
both need their own spec and a red-before test. It is listed under ## Owed in
nemotron-h-model.md.

Reconciling with #1415, which landed on both Nemotron rows

main gained the A2-Q2b device lm_head (#1415) while this branch was open, and
it edited the same two pages. These were real content conflicts, not the
union-driver artifact the index produces. Both were resolved BY KEY, and neither
side is discarded.

docs/BENCHMARKS.md was POSITIONAL rather than a same-key dispute. This branch had
already moved NemotronH paged forward below Record-anchor ratchet when it
collapsed that key's duplicate, and #1415 inserted a NEW key at the old offset.
Three-way: main did not touch NemotronH paged forward since the merge base, so
this branch's version stands, and NemotronH host re-expansion / decode token is a
main-only key taken verbatim. Each key occurs exactly once.

The reflex resolution here is wrong, and nothing would have caught it. Taking
main's version of the conflicted region — the ordinary move when the other side
looks newer — would have re-created the exact duplicate this pull request exists
to remove, because our copy of that key lives further down the file and the
conflicted region is not where the key now is. It would also have looked clean:
scripts/check-public-doc-tables.py holds these pages' size and shape and does
NOT test key uniqueness, which is
#1543. A gate could not have
caught the regression. Only decomposing the conflict BY KEY, and asking the
three-way question "did the other side actually touch this key", separated a
position collision from a content disagreement. Anyone resolving a conflict in a
keyed projection in this tree should assume the same trap.

docs/FEATURES.md is the same key, NemotronHForCausalLM, changed by both sides
in disjoint cells. Cell 3 changed only here, adding the A3 gate result. Cell 4
changed only on main, moving device lm_head out of the owed list and recording
it UNMEASURED. The row is composed from both, so nothing is adjudicated and nothing
is dropped.

#1415 landing does not gate this row, and this merge does not upgrade the
claim.
Cell 3 still reads "NO run against current main". A device lm_head
that reaches the device is not a run of this row's end-to-end token gate. FP8
mamba stays owed, because A2-Q1 (#1289) is still DRAFT.

Verified rather than trusted: every other key in both pages is byte-identical to
main; docs/STATUS.md auto-merged with Text generation as its only differing
key, no key added or lost; the model-matrix rollup is byte-identical and the row is
still INVENTORIED on both sides.

Gates

Run on 1b741c66baec6c9660aa80af3a82c0f5eca24dd9, this pull request's head, with
origin/main at e7f38e0c01ff1d85d18ef64eadd98996743ca061.

scripts/agent-preflight.sh --fail-on-skip reports ZERO gates SKIPPED, and
doc-checkpoint range is ok. RANGE_COUNT is 28. Both trailer gates were
proven to READ that range by mutation rather than read off a green line: a
detached scratch commit with no trailer block makes check-commit-trailers.py
exit 1 naming 3aeb80e9b769, and a scratch commit whose subject ends in a period
makes check-commit-style.py exit 1 naming 487921ac602c. The tree was restored
to 1b741c66b after each, verified by SHA and a clean git status.

doc-checkpoint is green for the right reason, and that was falsified too.
check-doc-checkpoint.py --commit 4dff5158b (the replayed 414a8002b) exits 0
while the PARTIAL -> INVENTORIED move is still present; renaming that spec's
## Now heading in a scratch amend takes it to exit 1 with
".agents/specs/nemotron-h-model.md has no ## Now section". So the gate passes
because the obligation is satisfied, not because the move vanished.

One trap worth recording: the FIRST attempt at that mutation read as passing. It
was run from a tree checked out at 4dff5158b, which predates the merge that
brought #1577 in, so it executed the OLD blind checker against the new commit.
The instrument was the stale thing, not the result.

Bare, on the same head, each exit 0:

check-doc-checkpoint.py --base e7f38e0c0 --head HEAD   OK
check-public-doc-tables.py                             OK
check-supported-models.py                              OK: exactly the 40 registered architectures
check-agent-record.py                                  ENGINE=169 MODEL=377 QUANT=84 KERNEL=57 BACKEND=85 ANCHOR-ROT=37
check-issue-index-append-only.py                       OK: issue index append-only
check-symbol-anchors.py                                OK every symbol anchor still finds what it names
check-now-current.py                                   OK

scripts/agent-preflight.sh exits 1 on test_cpu_x86_llamacpp_floor ALONE. That
is #618 and not a finding: the
harness reported waiting for quiet: 15s busy=109% builders=0 load=29.97. The
test reads scripts/cpu-x86-llamacpp-floor.sh and
docs/bench-evidence/cpu-x86-llamacpp-20260811.md, and this change touches
neither — it touches six record files and zero code files.

Inherited, not introduced

windows-msvc-cpu and windows-msvc-vulkan fail on every pull request and have
no main baseline (#584,
#968). This change has zero code
files and cannot reach a compiler.

Closes #1074

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

mudler added 8 commits August 16, 2026 23:14
…o, and a file glob is why nobody saw it

The NemotronH model-matrix row described `main` as of 2026-08-12 and had not
moved since. It read `INVENTORIED`, blocked on `KERNEL-SSM-MAMBA` (#496),
and named three things that "exist nowhere locally". Every one of those claims
was re-checked against `main` at `10002648199cfbbaf1e423f7c80cacb2f4b56366`
rather than inherited, including the ones the filing issue asserted.

Two of the three now exist. The non-gated relu2 MoE landed at `4d0c399e1`:
`vt::MoeRelu2`, called from `nemotron_h.cpp:354`. ModelOpt `MIXED_PRECISION`
per-module loading landed at `1bc5ef82c`. The third, the MTP head, is still
genuinely owed as W5, and the loader defers its 270 tensors by name.

The blocker itself is the finding. #496 W1 landed the three Mamba2 host
references at `47960a009` and W2 landed the CUDA arm at `43a6c5518`. The
kernel is `src/vt/cuda/cuda_mamba2_ssd.cuh`, a header included by
`cuda_gdn.cu` rather than a translation unit of its own, so a `src/vt/*mamba*`
file glob finds nothing and reads as absence. That is exactly the search this
row's text and the filing issue both rested on. NemotronH calls those ops at
`nemotron_h.cpp:597,620` today. #496 stays open for a GENERIC `MambaSpec`
producer, which this model does not need, because A1 made the runner read the
model's own KV spec.

The row moves `INVENTORIED` to `PARTIAL`, with the rollup, the checklist entry
and the projections that move owes. It does not move to `ACTIVE`, and the
reason is a rule rather than a judgement: `check-agent-record.py` requires an
`ACTIVE` row to name a `CLAIM-*` row that a claim source carries, no claim
record claims this row, and authoring one for another session's in-flight work
would be a fabricated record. `PARTIAL` is what the existing evidence backs,
and it is what the two Qwen3.5 text-only arms carry for the same posture.

Nothing here claims reachability. The forward reached through
`ModelRegistry::Forward` is the host reference, so `nemotron_h_registry.cpp:162`
still refuses paged and batched decode by name. There is no
`examples/nemotron_h_gen`, no ABI token gate, and no throughput, latency or
memory number, and `docs/BENCHMARKS.md` records that as a gap rather than a
number.

Found while doing this: `check-doc-checkpoint.py` matches a spec's live-position
section with `^##\s+Now\s*$`, and thirteen specs write it as `## N. Now`, so a
lifecycle move reds on a section that is present. Filed as #1080. This spec is
the thirteenth and is repaired here, because this change is what makes it the
spec a moving row links. The other twelve are listed under `## Owed`.

Closes #1074

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
…-- issue-index reconciled BY HAND

`origin/main` moved while this reconcile was written. The union driver merged
`.agents/issue-index.md` by interleaving: it put this branch's two rows before
main's `#1066` row, so main's file stopped being a prefix of ours. The index is
an append-only log and the prefix property is what makes it one, so the
auto-merge was discarded and main's file was taken whole, with this branch's
two rows re-appended at the end. Verified: main's blob is a byte prefix of the
result, 290 rows, no duplicate issue number.

Nothing else conflicted. `docs/BENCHMARKS.md`, `docs/FEATURES.md` and
`docs/STATUS.md` were touched on both sides in different places, and the diff
against `origin/main` after the merge is exactly this branch's additions.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
`main` advanced and GitHub reported CONFLICTING on `.agents/issue-index.md`.
GitHub does not run the `merge=union` driver `.gitattributes:7` sets, so the
resolution happens locally where it does.

The union auto-merge of the index was DISCARDED rather than trusted: main's
file taken wholesale, only this row's two rows re-appended, main's file
asserted a strict PREFIX of the result. 293 rows, zero duplicate keys.

`.agents/model-matrix.md` is a KEYED table, not an append-only log, and a union
merge DUPLICATES a keyed row rather than merging it -- that has happened in
this tree before. Verified it was NOT auto-merged (only issue-index.md,
docs/BENCHMARKS.md and docs/FEATURES.md were), and that the NemotronH key
appears exactly ONCE as a table row (`:286`), its other occurrence being the
separate checklist table at `:145`.

`check-agent-record.py` and `check-model-checklist.py` both exit 0.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
`main` advanced (#1084 and #1085 landed) and this branch conflicted for two
different reasons, resolved two different ways.

`.agents/issue-index.md` is append-only under `merge=union`, which GitHub does
not run -- so the union auto-merge was DISCARDED, main's file taken wholesale,
and only this row's own rows re-appended. Main's file is a strict PREFIX of
the result, zero duplicate keys.

`docs/FEATURES.md` was a GENUINE content conflict, not a merge-driver
artifact: both sides added a `NemotronHForCausalLM` row -- this branch's, and
main's from #1085. A keyed table gets ONE row per key carrying BOTH sides'
content, so the two were merged rather than one being chosen:

  - from this branch: the forward computes on landed `vt::Mamba2*` ops (#496)
    -- the finding that retired this row's false blocker -- and the loader's
    quantization breakdown (5935 NVFP4 g16, 46 FP8 W8A8, bf16);
  - from #1085: the A2-Q2a device MoE arm (23 blocks, NVFP4 g16 Marlin) and
    the sharper unreachability statement, G-SAFE refuses FIRST with A2-P
    owning the wiring;
  - from both: no e2e gate and no number.

The merged row is 6 parts with a maximum cell of 214 against the 220 cap, so
it fits without deleting anyone else's entry -- the "cap the entry, never the
file" rule doing its job.

`check-public-doc-tables.py` and `check-agent-record.py` both exit 0, and the
tree carries no conflict markers.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
`main` advanced (#1082 A2-P spec landed) and GitHub reported CONFLICTING on
`.agents/issue-index.md`. GitHub does not run the `merge=union` driver that
`.gitattributes:7` sets, so the resolution happens locally where it does.

The union auto-merge was DISCARDED: main's file taken wholesale, only rows
whose KEY main lacks re-appended, and main's file asserted to be a strict
PREFIX of the result. Zero duplicate keys.

The earlier `docs/FEATURES.md` conflict on this branch -- both sides adding a
`NemotronHForCausalLM` row -- stays resolved as one merged row per key,
carrying the Mamba2-ops finding and the loader breakdown from this branch and
the A2-Q2a device arm and G-SAFE wording from #1085.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
…s row, not merged into it

`main` moved 54 commits ahead of this branch and its NemotronH row is NEWER than
this branch's: A2-R `598226e96`, A2-P `a6df72777`, A2-Q2a and the A3 driver
`c83b96934` all appended to the same cell after this branch was cut. Merging the
two texts would have carried this branch's "the step still REFUSES paged decode
because the reached forward is the HOST reference" over the top of a paged
forward that has since landed.

So no keyed record was auto-merged. `.agents/model-matrix.md`, `docs/FEATURES.md`,
`docs/BENCHMARKS.md`, `docs/STATUS.md` and `.agents/specs/nemotron-h-model.md`
were taken from `b626be75a` WHOLE -- the tree was asserted byte-identical to main
before any edit -- and the scoped edit was then written again against what main
actually says. `.agents/issue-index.md` is append-only under `merge=union`, which
GitHub does not run, so the union auto-merge was discarded, main's file taken
whole, and only the rows whose KEY main lacks re-appended: #1074 and #1080. Main
already carries a #1068 row, so that one is NOT re-appended. Main's blob is a
strict BYTE PREFIX of the result (341815 of 344811 bytes), 366 rows, zero
duplicate keys.

What survives the rewrite is what is still true of main: the row reads
`INVENTORIED`, its `Spike/spec` cell reads `☐ required` against five committed
specs, its evidence cell reads `none`, and it still says the row is BLOCKED on
`KERNEL-SSM-MAMBA` (#496) because "the Mamba2 SSD core is unported". That last
claim is false and re-measured here.

What does NOT survive: this branch's `docs/BENCHMARKS.md` row said the reached
forward is the host reference and named A2-P as owed. Both are stale, and main
already carries a correct `MODEL-NEMOTRON-H-ABI-A2P` row saying the A3 gate is
PENDING. The replacement row is keyed to the MODEL row instead and points at that
one rather than restating it. This branch's `docs/FEATURES.md` edit is dropped
entirely: main's row already describes A2-P, A2-Q2a and G-SAFE correctly, and no
feature surface moves in this change.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
…, and it was re-appended by hand

`origin/main` advanced two commits (#1216, #1237) while this reconcile was being
written. Neither touches `.agents/model-matrix.md`, `docs/STATUS.md`,
`docs/BENCHMARKS.md` or `.agents/specs/nemotron-h-model.md`, so the scoped edit
did not have to be rewritten a second time.

`.agents/issue-index.md` did overlap: main gained three rows and the union driver
interleaved them, putting this branch's two rows before main's new ones and
destroying the prefix property that makes the file an append-only log. GitHub
does not run that driver anyway. So the auto-merge was DISCARDED, main's file
taken whole, and only the rows whose KEY main lacks re-appended: #1074 and #1080.
Asserted: main's blob is a strict BYTE prefix of the result (346246 of 349242
bytes), 369 rows, zero duplicate keys.

The delta against `cdfade6ab` is byte-for-byte the same five files it was against
`b626be75a`.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
…sted

`origin/main` advanced again (#1248) while this branch was being pushed, and this
time it DID touch three of the same files: `.agents/model-matrix.md`,
`docs/BENCHMARKS.md` and `docs/STATUS.md`. Git auto-merged all three, and
AGENTS.md `## Records` says never to accept that for a keyed record.

So the auto-merge was checked rather than trusted. `git diff` against
`10fe7f475` returns 13 changed lines in `.agents/model-matrix.md` and exactly
one each in `docs/BENCHMARKS.md` and `docs/STATUS.md`, and every one of them is
this branch's own scoped edit: the NemotronH row, the rollup, the two prose
counts, the checklist entry, the open-gap row and the status clause. #1248's
`MODEL-SPEC-deepseek-v4-dspark-...` row at `:546`, its DSpark benchmark row and
its DSpark status paragraph are byte-for-byte equal to main's. No unrelated key
moved.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
@localai-bot localai-bot changed the title record(MODEL-TEXT-nemotron-h): the row's blocker landed 23 commits ago, and a file glob is why nobody saw it record(MODEL-TEXT-nemotron-h): the row is blocked on a kernel that landed, and a file glob is why nobody saw it Aug 18, 2026
…e fourth time

`origin/main` advanced twice more (#1245, #1186). Only `.agents/issue-index.md`
overlaps, and it overlaps every time because it is the one file in this change
that every other branch also appends to. That is why GitHub keeps reporting this
pull request CONFLICTING: it does not run the `merge=union` driver
`.gitattributes:7` sets, so the conflict is real on the forge and absent locally.

Resolved the only way that keeps the file an append-only log: the auto-merge was
DISCARDED, main's file taken whole, and only the rows whose KEY main lacks
re-appended -- #1074 and #1080. Asserted: main's blob is a strict BYTE prefix of
the result (346713 of 349709 bytes), 370 rows, zero duplicate keys.

Nothing else overlapped. The delta against `ae581da3e` is the same five files it
was against `10fe7f475`.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
mudler added 2 commits August 18, 2026 16:38
…DING because I read a body written before its own job ran

The previous commit here wrote the A3 end-to-end token gate as `PENDING` and
said no GB10 96/96 existed. That was wrong, and the way it was wrong is worth
keeping: I searched `origin/main` and five `row/MODEL-NEMOTRON-H-*` branches,
found nothing, and then took #1221's PULL REQUEST BODY as the current state. Its
"Still owed: the sm_121a green-after re-run" was TRUE WHEN WRITTEN and stale by
the time I read it. A body is a snapshot of its author's knowledge at write time,
not a live field, and no amount of branch searching corrects for that, because
the artifact was never in git at all.

It is on the NAS. `/usr/local/nas_share/rc/nh1157/gate_fixed.out`:

    TOKEN MATCH: 96/96 over 3 prompt(s) (full rows=3, short rows=0, mode=decode)
    STRICT PASS

against the pinned oracle `vllm=0.23.1rc1.dev1511+g555967922` on
`nemotron-3.5-lightning-30b-nvfp4` at revision `29f2d1746d8f41e316523194b19018707749b1b1`.
Verified by reading the files, not by trusting the report of them.

It is the DEVICE leg. The binary is `libvllm 0.0.3+cuda` and `cfg.log` records
`fp4-mma`, `cutlass-nvfp4` and `cutlass-fp8` `ENABLED for [121a]`. The run logs
`Asynchronous scheduling is enabled (max_concurrent_batches=2)`, which is exactly
where `device_token_ids` is non-null; on the host queue it is always null and the
#1157 defect cannot arise. The decisive one is neither: the same binary on the
same checkpoint with ONLY `nemotron_h_device.cpp` reverted to the fix's parent
scores `4/24 (full rows=0, short rows=3)` and bails at 8 tokens. A host-leg run
would have been unmoved by that revert. The delta is the proof; the pass alone
is not.

So the row now says the gate PASSES and says whose it is. It belongs to #1221,
branch `row/MODEL-NEMOTRON-H-ABI-A2P-1157-fix` at `6e9e8955`, OPEN and
`CONFLICTING`. `main`'s last touch of `nemotron_h_device.cpp` is `a6df72777`
(A2-P), so `main` does not carry the fix and is not gated. A gate that passed and
a tree that is gated are different facts, and this record keeps them apart.

No performance number is claimed anywhere. `gate_fixed.out` carries 264.4s to
load and 327-343s per 32-token prompt; that is a correctness run whose `lm_head`
and 46 FP8 mamba projections still execute host-side. Both public rows say so in
the words "NOT a benchmark", so nothing can later lift them into a grid. One
config caveat also travels: `--gpu-memory-utilization 0.92` did not size the KV
pool, which fell back to 256 blocks (#83).

The row stays `PARTIAL`. `ACTIVE` still fails the `CLAIM-*` rule, and a gate that
passes off `main` is not a reason to move a lifecycle state.

`docs/FEATURES.md` moves in the same commit, because it has to. This commit
edits `.agents/model-matrix.md`, which `scripts/check-doc-checkpoint.py` reads as
a `feature_surface` change by path (`FEATURE_SURFACE_FILES`), and that class owes
`docs/FEATURES.md` in the SAME commit -- the checker walks a range one commit at
a time, so no later commit can pay the debt. The row for
`NemotronHForCausalLM` said nothing at all about the end-to-end token gate while
the matrix said it was PENDING, and it would have kept saying nothing now that
the gate passes. It now carries the same two facts the matrix does, in the
correctness column and in that column's own words: 96/96 `STRICT PASS` on GB10,
on #1221's open branch and not on `main`. The added text is 90 characters, which
leaves the cell at 192 of its 220-character cap and the row at 499 of 600, so no
other row pays for it.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
`main` gained #1221 while this branch was open, so `docs/BENCHMARKS.md` and
`docs/STATUS.md` conflicted on the NemotronH rows. Resolved by taking main's
rows as the base and correcting the one claim that main is now STALE on.

Main says "GB10 read 4/24 ... sm_121a re-run pending a lease" and "sm_121a
re-run pending". That was true when #1221 was written. The re-run has since
happened: the A3 gate PASSES on GB10 at 96/96, `STRICT PASS`, mode=decode,
against `vllm=0.23.1rc1.dev1511+g555967922` on `nemotron-3.5-lightning-30b-nvfp4`
revision `29f2d1746d8f41e316523194b19018707749b1b1`, with a red-before of 4/24
on the SAME binary and checkpoint with only `nemotron_h_device.cpp` reverted.
Artefacts: `/usr/local/nas_share/rc/nh1157/{gate_fixed,gate_red,cfg}.out`.
`cfg.log` shows `fp4-mma`, `cutlass-nvfp4`, `cutlass-fp8`, `marlin-nvfp4` and
`fa2` all `ENABLED for [121a]`, so the build was not degraded, and both legs log
`Asynchronous scheduling is enabled`, which is the path where
`device_token_ids` is live and the defect lived.

The delta, not the pass, is the proof: on the host leg reverting the fix would
have changed nothing.

STILL NOT A BENCHMARK. The 264.4 s load and 43 405 MB peak are recorded as a
correctness run and labelled as such, because NVFP4 `lm_head` (A2-Q2b) and the
46 FP8 W8A8 mamba projections (A2-Q1, #940) still execute host-side. No
throughput, latency or memory ratio is claimed on any axis.

The index was reconciled by discard-and-reapply: main's file taken wholesale,
only rows whose key main lacks re-appended, main's file a strict PREFIX.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
mudler added 5 commits August 19, 2026 11:32
…tted

`main` is green again (#1297 fixed the gcc-16 `getpid` red; the qwen
`build-test-cpu` red had already been fixed by `281b4bc76`), so this branch is
merged forward to drop the stale inherited failures.

`docs/BENCHMARKS.md` was a genuine content conflict: this branch and main
inserted different rows at the same point. BOTH are kept, main's first. No row
dropped, no ratchet raised.

ANCHOR REPAIR. The merge rotted one citation and `check-agent-record.py`
caught it: `stale' 33 > baseline 32`, naming
`nemotron_h_device.cpp:1412 expected NemotronHPagedForward`. Main's count is
exactly 32 with RC=0, and diffing the two report lists isolated the single
extra entry as this row's, so it was repaired rather than the baseline raised.
The symbol now lives at `:1474`.

Worth recording because it nearly shipped: the citation is a markdown link,
and the DISPLAY TEXT and the LINK TARGET are two separate spellings of the
same line number. Repairing only `nemotron_h_device.cpp:1412` left
`[nemotron_h_device.cpp:1474](...cpp#L1412)` -- a citation that READ correct
and POINTED wrong, and which a reader checking the rendered text would have
called fresh. Both halves are now `1474`.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
GitHub does not run the `merge=union` driver `.gitattributes:7` sets, so a
branch touching `.agents/issue-index.md` re-conflicts whenever main appends a
row. Resolved locally where the driver runs: the union auto-merge was
DISCARDED, main's file taken wholesale, only rows whose KEY main lacks
re-appended, and main's file asserted a strict PREFIX with zero duplicate keys.
Any keyed-table collision kept BOTH sides, main's first, so no row is dropped.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
GitHub does not run the `merge=union` driver `.gitattributes:7` sets, so a
branch touching `.agents/issue-index.md` re-conflicts whenever main appends a
row. Resolved locally where the driver runs: the union auto-merge was
DISCARDED, main's file taken wholesale, only rows whose KEY main lacks
re-appended, and main's file asserted a strict PREFIX with zero duplicate keys.
Any keyed-table collision kept BOTH sides, main's first, so no row is dropped.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
GitHub does not run the `merge=union` driver `.gitattributes:7` sets, so a
branch touching `.agents/issue-index.md` re-conflicts whenever main appends a
row. Resolved locally where the driver runs: the union auto-merge was
DISCARDED, main's file taken wholesale, only rows whose KEY main lacks
re-appended, and main's file asserted a strict PREFIX with zero duplicate keys.
Any keyed-table collision kept BOTH sides, main's first, so no row is dropped.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
GitHub does not run the `merge=union` driver `.gitattributes:7` sets, so a
branch touching `.agents/issue-index.md` re-conflicts whenever main appends a
row. Resolved locally where the driver runs: the union auto-merge was
DISCARDED, main's file taken wholesale, only rows whose KEY main lacks
re-appended, and main's file asserted a strict PREFIX with zero duplicate keys.
Any keyed-table collision kept BOTH sides, main's first, so no row is dropped.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
Catch-up merge so the branch's gates run against the tree the records will land
on. `git merge-tree --write-tree` reported no conflict.

`.agents/issue-index.md` is the one keyed record both sides touch, and it is
verified by key rather than trusted to the automatic merge: `origin/main` holds
451 rows, this tree holds 453, all 451 of main's rows are present in main's own
order, and the two extra rows are this branch's own `#1074` and `#1080` entries.
No key is duplicated.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
@localai-bot
localai-bot force-pushed the row/RECORDS-NEMOTRON-H-MATRIX-1074 branch from f30c2da to c598f1b Compare August 20, 2026 04:05
mudler added 2 commits August 20, 2026 07:55
…ce was reddening this branch (#1074)

FOLLOWING_AGENTS_PROTOCOL

`build-test-cpu` and both `sanitize-cpu` jobs were red here on one test,
`175 - test_nemotron_h_paged_forward`, and the defect was never in this branch,
which carries no product code at all. #1371 left `CPU_ATTN` as a name in a
candidate list with no registrar, so the CPU inherited FlashAttention's
`head_size % 8` constraint with no fallback. `9ecaf1bb3` (#1392) fixes that, and
`f07f96e1c` (#1444) is the current `origin/main` that carries it. Merging the
older SHA would have left this branch behind again, and `commit-trailers` and
`commit-style` SKIP on a branch that is behind.

Four files are touched by both sides, and each was verified rather than trusted
to the automatic merge. `.agents/issue-index.md` is the keyed record: main
carries 462 rows and this tree carries 464, all 462 of main's rows are present
byte-identical and in main's own order, the preamble is byte-identical, and the
two added rows are this branch's own `#1074` and `#1080`. Nothing is removed and
no key is duplicated. `docs/BENCHMARKS.md`, `docs/FEATURES.md` and
`docs/STATUS.md` are projections whose two sides edit different rows, and that
is measured rather than asserted: the added and removed content lines of
`merged vs main` hash equal to this branch's own delta, and those of
`merged vs branch` hash equal to main's own delta, for all three files.

No file is touched beyond what the merge itself requires.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
…nion-append as a conflict (#1074)

FOLLOWING_AGENTS_PROTOCOL

This branch went `CONFLICTING`/`DIRTY` the moment #1409 merged, because both
append a row to `.agents/issue-index.md` and GitHub does not apply the
`merge=union` driver that `.gitattributes` declares for that path. It is the
known forge artifact rather than a real conflict, and that was established
before the merge rather than assumed: `git merge-tree --write-tree` over
`origin/main` and this branch's tip exits 0 and writes tree
`2404d135933ba312a147157ccab9bf315502567b`, with no conflicted path. The
resolution is to re-merge, not to debug the forge.

Two keyed records are touched by both sides, and each was verified by key rather
than trusted to the automatic merge.

`.agents/issue-index.md` stays a pure append: main carries 471 rows and this
tree carries 473, all 471 of main's rows are present byte-identical and in
main's own order, the preamble is byte-identical, the two added rows are this
branch's own `#1074` and `#1080`, nothing is removed and no key is duplicated.

`.agents/model-matrix.md` is the one that needed care, and it is NEW to this
merge -- main did not touch it when this branch was last merged. It holds 373
keyed rows on both sides, 0 missing and 0 added and 0 duplicated, and exactly
ONE row differs from main: `MODEL-TEXT-nemotron-h-nemotron-hfor-causal-lm`,
which is the row this reconcile exists to move. The file also stores DERIVED
COUNTS beside its rows, which a textual merge cannot reason about, so they were
checked rather than accepted: this branch moves `53 -> 54` engaged,
`324 -> 323` `INVENTORIED` and `22 -> 23` `PARTIAL`, and main's own edits to the
two `Qwen3_5*` rows touch ZERO count lines and change no lifecycle state -- they
renumber `file:line` anchors inside rows that stay `PARTIAL`. The merged counts
are therefore still the counts of the merged rows.

`docs/BENCHMARKS.md`, `docs/FEATURES.md` and `docs/STATUS.md` are projections
whose two sides edit different rows. Measured rather than asserted, for all
three and for the model matrix: the added and removed content lines of
`merged vs main` hash equal to this branch's own delta, and those of
`merged vs branch` hash equal to main's own delta.

No file is touched beyond what the merge itself requires.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
mudler added 4 commits August 21, 2026 07:59
…o its model row is resolved to main's page (#1074)

FOLLOWING_AGENTS_PROTOCOL

`docs/STATUS.md` is a REAL conflict this time, not the `merge=union` forge
artifact the previous two merges were. #1491 (`aee6c48d6`) cut the page from
2906 lines and 65 table rows to 98 and 27, reorganising it BY SURFACE rather
than by model, and the per-model table this branch edits no longer exists. The
row the branch changes has no counterpart to merge into.

The resolution is main's page, taken whole and byte for byte, so this commit
carries no trace of the old structure. Re-expressing what that row recorded is a
content decision and is deliberately NOT in this commit; it is the next one, so
a reviewer can read the merge and the decision separately.

The other keyed records merged clean and were verified by key rather than
trusted to the automatic merge. `.agents/issue-index.md` stays a pure append:
main carries 502 rows and this tree 504, all 502 of main's rows byte-identical
and in main's own order, the preamble byte-identical, the two added rows this
branch's own `#1074` and `#1080`, nothing removed and no key duplicated.
`.agents/model-matrix.md` holds 373 keyed rows on both sides with 0 missing, 0
added and 0 duplicated.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
…ncile rested on expired (#1074)

FOLLOWING_AGENTS_PROTOCOL

This reconcile was written while [#1221](#1221)
was OPEN, and it moved the row `INVENTORIED` -> `PARTIAL` on the strength of an
A3 gate that read 96/96 `STRICT PASS` on GB10. #1221 MERGED on 2026-08-18 as
`0ea5d249f`, which is now `main`'s last touch of `nemotron_h_device.cpp`, so
every "blocked on #1221 landing" sentence in this branch became false.

What is recorded now is narrower than what the branch originally claimed, and
deliberately so. The A3 gate read 96/96 `STRICT PASS` (mode=decode) on GB10,
MEASURED ON THE TREE THAT LANDED AS `0ea5d249f`. That fix is on `main`. **No run
against current `main` exists**, because `main` has advanced many commits past
the measured tree. Naming the SHA a measurement belongs to is the whole point:
an evidence line that names a tree it was not measured on has cost this
repository before, and this change does not add another.

**The row therefore STAYS `INVENTORIED`.** The lifecycle move is left to whoever
re-runs the gate against `main`, and the derived counts in
`.agents/model-matrix.md` are restored to main's `53`/`324`/`22` with the
engaged-architecture row withdrawn. Measured rather than asserted: the row-state
histogram over all 373 keyed rows is byte-identical to main's, so this change
moves NO row's lifecycle state, and the count block is byte-identical to main's.
What survives from the original reconcile is the part that was never about
#1221: the row's `KERNEL-SSM-MAMBA` block was FALSE, and its corrected code and
test anchors stay, so a later state move has its contract already written.

`docs/BENCHMARKS.md` carried the key `NemotronH paged forward`
(`MODEL-NEMOTRON-H-ABI-A2P`) TWICE on this branch and once on main, because the
branch APPENDED where it should have UPDATED. The two rows contradicted each
other: main's said GB10 read 4/24 with an sm_121a re-run pending a lease, and
the branch's said GB10 read 96/96. The lease and the re-run happened, so main's
row is stale on its face. They are collapsed to ONE row on the key byte-identical
to main's, carrying the current projection. Nothing gates this:
`check-public-doc-tables.py` holds the page's size and shape and not its key
uniqueness, so it would have landed silently.

`docs/STATUS.md` gets a one-line in-place edit to the `Text generation` row's
"Open gate or limitation" cell, which is where #1491 routed this after it
reorganised the page by surface. That is the shape `af25bd251`, `5702d8f83` and
`d995c52f0` established for `Speculative decoding` and the diffusion row, and it
satisfies the `lifecycle` obligation without growing a shrink-only page: 1
insertion, 1 deletion, 98 lines and 27 table rows before and after. Model-level
detail is NOT reintroduced; it stays in
[`docs/models/nemotron-3-5-lightning.md`](../docs/models/nemotron-3-5-lightning.md),
whose "no token gate result exists" line remains true of `main`.

The obligation still fires with no lifecycle move because
`check-doc-checkpoint.py` adds the `lifecycle` class for a measurement change as
well as a state change, so `STATUS` and `BENCHMARKS` are both owed and both
written.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
… open, and this branch added that text (#1074)

The `## Owed` entry for #1217 was written while #1221 was open and it kept three
claims that are false at this head. It called #1221 "open and unmerged", although
#1221 merged on 2026-08-18 as `0ea5d249f`, which the previous commit's own
subject says. It said #1217 "is recorded NOWHERE in this tree", although the
merge `8a00afcbe` brought both records in: `.agents/issue-index.md:403` carries
the index row under the owning row `MODEL-NEMOTRON-H-ABI-A2P`, and
`nemotron-h-a2p-paged-forward.md:901` lists the issue under that spec's
`## 11. Owed`. That claim was true at `b626be75a` and is not true here. And it
named #1217 "the standing reason this row's end-to-end gate is not a pass", which
contradicts `## Now` in the same file, where the A3 gate reads 96/96
`STRICT PASS`.

`git diff 6b48edb 414a800` shows this branch ADDED the phrase, and
`origin/main` carries it zero times, so the stale text is this pull request's to
remove rather than an inherited defect.

The bullet now records where the record landed and cites both anchors, which
were re-derived at this head. The duplicate-key argument for withholding an index
row is kept as history and marked expired: appending a row now would create the
very duplicate that argument was written to avoid, because the row is already on
`main`. Nothing is owed to this reconcile for #1217. The reason the row's gate is
not recorded as a pass on `main` is restated as `## Now` gives it, which is that
the 96/96 was measured on the #1221 branch tree and no run against current `main`
exists.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
…as behind it (#1074)

`scripts/agent-preflight.sh --fail-on-skip` SKIPPED both `commit-trailers` and
`commit-style`, because `origin/main` was not an ancestor of this head. A SKIP
reports nothing about this tree, so neither gate had read it. This merge is what
lets them run.

`origin/main` is `04f1cead6f94c01f8fd4e669148f7a9dbcf0b132`. The five commits are a container-publish repair
(#1560), a BPE row close (#1558), two README edits (#1554, #1302), and a
clock-gate spec (#1553). They touch 19 files, and three of them are files this
pull request also edits: `.agents/issue-index.md`, `docs/BENCHMARKS.md` and
`docs/STATUS.md`. `git merge-tree --write-tree` reported no conflict on any of
the three, and the append-only and public-document gates are rerun on the merge
result rather than assumed from that report.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
mudler added 4 commits August 21, 2026 09:32
#1074)

`origin/main` moved to `947e5f648552192d08cad4e636953be57bf34ffb` while this branch's gates were running, so the
trailer gates would have SKIPPED again on the next run. A SKIP reports nothing
about this tree.

The one commit is the MiniMax-Music3 DiT profile (#1550). It touches 11 files,
and two are files this pull request also edits: `.agents/issue-index.md` and
`docs/BENCHMARKS.md`. Both auto-merged, and the append-only and public-document
gates are rerun on the merge result rather than assumed.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
…e this branch reasons about (#1074)

`origin/main` moved to `5453e571dd412e63de53bd6ae66b5a94cd7df1e3` while this branch's CI was running, so the
trailer gates would SKIP again and `build-newest-gcc` was still running against a
base that predates its own fix. A SKIP reports nothing about this tree.

Three of the 32 matter to this pull request beyond the merge. #1577, #1609 and
#1621 all change `scripts/check-doc-checkpoint.py`'s record states, and this
branch's description asserted that the gate's `STATES` tuple could see neither
`INVENTORIED` nor `PARTIAL`. That claim is re-derived against the merged checker
rather than carried, because a stale premise about a gate is the defect this
branch already had once.

#1581 landed the `process_id.h` seam that had reddened `build-newest-gcc` since
W5, so that job's earlier red on this branch was stale CI rather than a finding.

Four files this pull request edits also moved on `main`:
`.agents/issue-index.md`, `docs/BENCHMARKS.md`, `docs/FEATURES.md` and
`docs/STATUS.md`. The append-only and public-document gates are rerun on the
merge result rather than assumed from a clean auto-merge.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
…n landed 32 commits (#1074)

`.agents/issue-index.md` is append-only, so every row that lands above #1217's
shifts it down. The 32 commits merged from `origin/main` in the previous commit
appended enough rows to move it from `:403` to `:404`, and the `## Owed` bullet
cited the old line.

Re-derived at this head rather than adjusted by arithmetic: `grep -n '^| \[#1217\]'`
reports `404`, and the a2p spec's bullet is unmoved at `:901` under `## 11. Owed`
at `:899`. This is the anchor-rot #1143 names, inside one pull request rather than
across releases.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
…ted (#1074)

Worktrees share `.git`, so another session's fetch advanced `origin/main` to
`e100e64e1f6c553c8e16932d332cfc24a34e8d20` under this one. The trailer gates resolve their base from
`origin/main`, so they would SKIP without this merge, and a SKIP reports nothing
about this tree.

The append-only and public-document gates are rerun on the merge result.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
@localai-bot
localai-bot force-pushed the row/RECORDS-NEMOTRON-H-MATRIX-1074 branch from 9ab3031 to f0d70d6 Compare August 22, 2026 00:02
…ncile edits (#1074)

`origin/main` is `e7f38e0c01ff1d85d18ef64eadd98996743ca061`, which now carries the A2-Q2b device `lm_head`
(#1415). `git merge-tree` reported real content conflicts in `docs/BENCHMARKS.md`
and `docs/FEATURES.md`, not the union-driver artifact the index produces. Both are
resolved BY KEY, and neither discards a side.

`docs/BENCHMARKS.md` was POSITIONAL, not a same-key dispute. This branch had
already moved `NemotronH paged forward` below `Record-anchor ratchet` when it
collapsed that key's duplicate, and #1415 inserted a NEW key at the old offset.
Three-way: `main` did not touch `NemotronH paged forward` since the merge base,
so this branch's version stands, and `NemotronH host re-expansion / decode token`
is a main-only key taken verbatim. Each key now occurs exactly once, which is the
defect this pull request exists to fix and must not reintroduce.

`docs/FEATURES.md` is the same key, `NemotronHForCausalLM`, changed by both
sides -- in DISJOINT CELLS. Cell 3 changed only here, adding the A3 gate result.
Cell 4 changed only on `main`, moving device `lm_head` out of the owed list and
recording it UNMEASURED. The row is composed from both, so no fact is adjudicated
and no content is dropped.

The A3 claim is NOT upgraded by this merge. Cell 3 still reads
"NO run against current `main`", and #1415 landing device `lm_head` is not a run
of this row's gate. FP8 mamba stays owed, because A2-Q1 (#1289) is still DRAFT.

Verified rather than trusted: every other key in both pages is byte-identical to
`main`; `docs/STATUS.md` auto-merged with `Text generation` as the only differing
key; the index is a pure append (2 added, 0 removed, main's 563 rows in order);
and the model-matrix rollup is byte-identical with the row still `INVENTORIED`.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
@localai-bot
localai-bot merged commit 6ec2cf8 into main Aug 22, 2026
22 of 27 checks passed
localai-bot added a commit that referenced this pull request Aug 22, 2026
…e that never tests key uniqueness (#1656)

`.agents/issue-index.md` did not track
[#1543](#1543). Verified with a
positive control rather than a bare grep, because a null grep proves the
terms wrong before it proves anything absent: the index carries 563 rows
on `origin/main`, `issues/1217` resolves to exactly one of them, and
`issues/1543` resolves to none, on `origin/main` and on the head of
every open pull request. 18 of the 19 open heads carry the index and
answer the control with 1 and #1543 with 0; the nineteenth, PR #361,
predates the file, so `git show` exits 128 there and the row cannot be
present. The issue was filed and never indexed, so the three places
AGENTS.md requires to agree, the index, the owning row's spec, and the
pull request body, agreed on nothing.

This change appends one row and changes nothing else.

## What #1543 says, and why the checker is not touched here

`scripts/check-public-doc-tables.py` gates the projection documents for
size and shape, the section and paragraph counts, the cell cap, the
shrink-only oversized-cell ratchet, and never for key uniqueness. A
public document is defined as the keyed current projection, one row per
key carrying what is true now, and no gate enforces the "one" part. On
[PR #1081](#1081)
`docs/BENCHMARKS.md` carried the key `**NemotronH paged forward**
(MODEL-NEMOTRON-H-ABI-A2P, #810)` twice with contradictory content, one
row reporting a GB10 read of 4/24 with the re-run pending and the other
reporting the same gate passing 96/96, and the checker returned rc=0 on
that file.

That is a checker-semantics change, which AGENTS.md routes to its own
row, spec, red-before test or mutation, and green-after evidence. So
this change indexes the issue and leaves
`scripts/check-public-doc-tables.py` untouched, and #1543 stays open.

## Why the row names an owning row rather than a spec

AGENTS.md gives an index row two admissible shapes: it names an owning
row ID, or it names a spec that lists the issue under `## Owed`.
`ENG-RECORD-CONFLICT-SURFACES` is already the standing owner of
public-document projection-gate semantics. #460, #495, #498, #507 and
#1055 all sit there, and #1055's own row records in words that
redesigning the doc budgets is that row's scope. That group is named at
exactly this width on purpose: the five do not all sit in one script,
since #460's `check_links` is defined in `scripts/check-agent-record.py`
and #498's retired `MAX_README_CHARS` lives in
`scripts/check-readme-structure.py`, so calling them
`check-public-doc-tables.py` semantics would be false of two of them.
The owner is also live rather than historical: the row is `READY`, not
`DONE`, and its spec already names
`tests/scripts/test_check_public_doc_tables.py` as a red-before suite,
so its declared test surface is the checker #1543 indicts.

The `## Owed` shape was considered and rejected on its meaning. `##
Owed` names the spec whose own work left the debt, and no spec here did.
The defect surfaced during the review of PR #1081, whose separate repair
collapses the duplicate rows into one without closing the blindness,
because the next append reproduces it. Naming an owner is also not a
claim that the scope is already written, so the row says in its own text
that the fixing row still owes the spec and the red-before, and it
carries the two cautions the issue records: the assertion has to be
scoped per table, since `audit-live-rows --check` reports three IDs
living in two matrices, BACKEND-CPU, BACKEND-CUDA-SM121 and
BACKEND-VULKAN, so a naive global check reds `main` on day one; and the
red-before is available rather than hypothetical, because PR #1081's own
pre-repair head contains a real duplicate.

## What the review changed

The fresh review reproduced the premise, the duplicate count, the
ownership mutation and the gates, confirmed the owner choice on stronger
grounds than the first draft gave, and returned one minor finding: two
enumerations inside the appended cell were imprecise. The cell is
append-only, so it can never be corrected once it lands, and both were
corrected on the branch before it did.

The first is the script attribution above. The second claimed the only
duplicate-key checks under `scripts/` were three named files.
`scripts/check-agent-record.py` holds two more, verified at this head:
`check_matrices` refuses a `duplicate ID` over the matrix inventories,
and `check_issue_index` refuses `issue #N listed twice`, which is the
refusal this very row had to be checked against before it was appended.
That file also reads `docs/BENCHMARKS.md`, through `markdown_files`. The
substantive claim survives and is now stated as what it always was, that
no checker asserts uniqueness on a projection document, and
`check_matrices` is named as prior art, because it is the shape the
fixing row should mirror and an immutable row implying there was none
would cost someone that discovery later. The prior art is cited as
`path::symbol` rather than `path:line`, because
`ENG-RECORD-ANCHOR-RATCHET` measured that every stale anchor found in
its 2026-08-13/14 campaign was still in range, so a line number in a row
that can never be edited rots silently.

`origin/main` advanced by two commits during the review, so it is merged
here with an authored, trailered message rather than by `--no-edit`,
which produces the trailerless commit the trailer gate rejects. The
range against the merge base remains a pure append,
`check-issue-index-append-only.py` reports zero removed lines, and the
pull request still shows one added line in one file.

## Evidence

Ownership is gated, not decorative. `check_issue_index` counts the rows
that name neither an owner nor a spec `## Owed` entry and refuses any
count above `UNOWNED_HIGH_WATER`, currently 33. Mutating this row's
owner cell to a dash in the worktree takes the count to 34 and reds
`check-agent-record.py` with `#1543` named in the message, which also
proves the checker read this worktree and not the shared checkout, a
distinction that has silently inverted verdicts here before. The tree
was restored byte-for-byte afterwards and the mutation was confirmed to
have applied before the run.

Green after, re-run on the corrected and merged head:
`check-agent-record.py` rc=0, `check-issue-index-append-only.py` rc=0,
and `scripts/agent-preflight.sh --fail-on-skip` with zero skips. The two
trailer gates were confirmed to have RUN rather than read as green:
`origin/main` resolved to a commit, so `BASE_UNRESOLVED` does not apply;
`git merge-base --is-ancestor` exited 0, so neither `ANCESTRY_UNKNOWN`
nor the behind-base arm applies; and `git rev-list --count` exited 0
printing `RANGE_COUNT=3`, a numeric count above zero, so `RANGE_UNKNOWN`
does not apply and `commit-trailers` and `commit-style` both took their
run arm over the one commit under judgement.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

---------

Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
localai-bot pushed a commit that referenced this pull request Aug 22, 2026
…icated a scoreboard key (#1250)

FOLLOWING_AGENTS_PROTOCOL

This branch was 104 commits behind `origin/main` and `git merge-tree` refused
it, so `commit-trailers` and `commit-style` were SKIPPING rather than passing.
This merge takes `origin/main` db648fb and nothing else. Three files
conflicted and each was resolved by KEY, never by hunk.

`docs/BENCHMARKS.md` is the one that mattered, and the conflicted region was
NOT where this row's key lives. The base carried
`**NemotronH paged forward**` directly after `**Developer/row protocol**`.
This branch edited that row in place at that offset. Main MOVED the row two
positions down, past `**Record-anchor ratchet**`, and inserted a different key,
`**NemotronH host re-expansion / decode token**` (`A2-Q2b`), at the offset the
move vacated. Git therefore conflicted this branch's edit against a row it has
nothing to do with, and auto-merged main's relocated copy in BELOW the conflict
as an addition. The conflicted working tree held TWO
`**NemotronH paged forward**` rows, at lines 16 and 21. Taking either side
wholesale lands a duplicate key, and `scripts/check-public-doc-tables.py` does
not test key uniqueness (#1543), so it would have landed silently. This is the
#1081 shape again.

The row was therefore rebuilt rather than chosen. `docs/BENCHMARKS.md` is taken
from `origin/main` byte-for-byte, verified by an empty diff before any edit, and
the single `**NemotronH paged forward**` row is then replaced in place. The
result is one changed line, zero added, zero removed, exactly one row under each
of the two keys, and main's `A2-Q2b` row untouched.

Main had ALSO edited that row's third and fourth cells, so this was a real
two-sided edit and not only a displacement. The rebuilt row keeps this branch's
measurement, which is the row's purpose and which main's cells predate, and it
keeps main's provenance caveat that the branch's own text dropped: the 96/96 was
read on `5325b7b97` and NOT on current `main`. Main's remaining cell content
is superseded rather than discarded -- `NOT a benchmark` and the 264.4 s load
were true only while no benchmark existed, and this row now carries one. The
rebuilt row measures 589 characters against the 600-character ENTRY budget; the
forensics stay in `.agents/benchmark-record.md`, which is where that budget's
error message sends them.

`docs/STATUS.md` was restructured by #1491 (`aee6c48d6`) from 65 rows to 27,
organised by surface instead of by model, with model detail routed to
`docs/models/`. The per-model `Nemotron-3.5-Lightning-30B-A3B` row this branch
edited NO LONGER EXISTS on that page, so re-applying the edit by key would have
asserted on a missing key. Following the post-restructure convention
(`af25bd251`, `5702d8f83`, `d995c52f0`), the page is taken from
`origin/main` byte-for-byte and one line changes in place: the speed result is
added to the existing `Text generation` row's open-gate cell, beside the
Nemotron sentence main already keeps there. One changed line, zero added, zero
removed, still 27 rows, and no model-level detail is reintroduced. That cell was
already over `MAX_CELL_CHARS`, so the shrink-only `oversized_cells` ratchet
does not move: it measures 3 against a cap of 44, as it does on `main`.

`.agents/benchmark-record.md` is a prepend-ordered append-only log and both
sides prepended a new section at the same offset, which is the whole conflict.
Nothing is chosen between them: main's file is taken whole and this branch's
245-line `MODEL-NEMOTRON-H-ABI-A2P` section is inserted after main's newer
`A2-Q2b` section, keeping the newest-first order. Verified rather than assumed
-- 245 added and 0 removed, and all 26,720 of main's lines are present,
byte-identical, in main's own order.

`.agents/issue-index.md` auto-merged and was verified by key anyway: main
carries 575 rows and this tree carries 577, the two added rows are this branch's
own #1250 and #1253, nothing is removed, no key is duplicated, and all 594 of
main's lines are byte-identical and in order.

`examples/nemotron_h_gen/main.cpp` auto-merged. Main touched it once
(`7fce87b57`) and in a disjoint region, measured rather than trusted: the added
and removed content lines of merged-versus-main hash equal to this branch's own
delta for that file.

No file is touched beyond what the merge itself requires.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]
localai-bot added a commit that referenced this pull request Aug 22, 2026
…e 0.001392x, load 2.12x faster, and the GPU idle for 93.7% of our decode (#1250) (#1251)

There was no Nemotron number on any axis. `docs/BENCHMARKS.md` carried
`MODEL-NEMOTRON-H-ABI-A2P` as "No speed number, by the unit's own rule".
This
branch takes the first ones, against the pinned vLLM oracle, which
completed a
model run inside an `rc` lease for the first time.

FOLLOWING_AGENTS_PROTOCOL

## The numbers

Both legs on `dgx:gpu0` through `rc run`, same box, same boot id, same
checkpoint directory, same three pre-tokenized prompts, same 32 greedy
tokens
with `ignore_eos`, batch 1 sequential, two legs over one engine load on
each
side. Tree `5325b7b970b67f97a77834e907fc34fb2990b71e`, which contains
`0ea5d249f` (#1221). **`enforce_eager=False` throughout: CUDA graphs
were on and
were never disabled.**

| axis | ours | pinned vLLM | ratio |
|---|---|---|---|
| per output token, warm (n=5) | 10.3194 s | 0.014369 s | **718.2x
slower** |
| output throughput, batch 1 | 0.09691 tok/s | 69.595 tok/s |
**0.001392x** |
| engine load | 280.9 s | 596.3 s | **0.4711x, we are 2.12x FASTER** |
| peak host memory | 44,616 MB | 70,974 MB | 0.629x raw, not
like-for-like |
| KV pool | 8192 tokens | 644,096 tokens | 78.6x, could NOT be matched |

**Every timing leg is a gated leg.** Ours read `TOKEN MATCH: 96/96 over
3
prompt(s) (full rows=3, short rows=0, mode=decode)` on BOTH legs,
`STRICT PASS`,
192 of 192 tokens, so no number here comes from a configuration whose
tokens were
not compared in the same process. The same-binary A/B over one load is
1.0016
warm and the five warm prompts spread 0.245%.

The build was not degraded: CUDA 13.3.73, `CFG_RC=0`, `BUILD_RC=0`, zero
compile
errors, and `fp4-mma` / `cutlass-nvfp4` / `cutlass-fp8` / `marlin-nvfp4`
/ `fa2`
each `ENABLED for [121a]`, counted as `FEATURE_LINES_SEEN=5
DEGRADED_FEATURE_LINES=0` rather than eyeballed. The box was idle:
`--query-compute-apps` empty, 4,892 of 122,502 MB used.

## Two refusals that travel with the ratios

They are recorded beside the numbers rather than netted out of them.

**The clock gate refuses this pair.** `gpu_clock_state compare` exits 1
with six
reasons: ours 6.31% busy and vLLM 31.05% busy against a 50% floor, vLLM
spread
5.14% against a 5.0% ceiling, ours throttled `SwPowerCap`, persistence
`Disabled` on both. Same boot id, both medians 2411 MHz,
`median_offset_pct`
0.0. What the refusal cannot do is explain the result: the tool's own
recorded
basis is 0.7548 points of kernel time per point of clock, and the gap is
718x.

**The KV pools could not be matched.** vLLM's hybrid allocator resolves
its own
block size -- 512 here, 4192 at other settings -- against our 32, so a
token
capacity match is the most that was ever available and this run did not
get one.
The memory axis is therefore reported raw, and the straggler still
resident in
the oracle's window (PID 40514, 22,986 MiB, the EngineCore this row's
own
watchdog killed) is named rather than subtracted.

## What names the bottleneck, and why it is not a ceiling

**`nvidia-smi` reported GPU utilization 0% in 2,019 of 2,155 samples of
our
window. The GPU was busy in 6.31% of it.** That is the driver's own
answer, not
a reading of the source, and the clock helper's refusal to attribute
such a
window IS the finding: this decode is not GPU work.

It matches the mechanism exactly. Of 52 layers, 6 are attention and stay
on the
device, 23 MoE layers run on the device through the NVFP4 Marlin arm,
and 23
Mamba2 layers bounce -- the normed hidden is downloaded, the mixer runs
on the
CPU queue, the result is uploaded -- once per layer per token; then
`NemotronHHostLmHead` projects on the host because
`src/vllm/model_executor/models/nemotron_h.cpp::NemotronHHostLmHead`
refuses the NVFP4 `lm_head` on a non-CPU queue.

That citation is a SYMBOL rather than a line, here and in the spec and
in the
index row, and the change is a repair. All three said
`nemotron_h.cpp:1031-1034`;
the refusal is at `:1036` now and those lines hold `H`, `V` and two size
checks.
`scripts/check-symbol-anchors.py` says in its own docstring that it
"does not
verify LINE citations. There is no honest way to", so nothing reported
it. The
index row is the one that had to be fixed before this merges: that file
is
append-only, a landed row may never be edited, and the same checker
lists it in
`FROZEN_FILES` because "a stale anchor inside one is unrepairable by
construction".

## The lever is confirmed, and still ungated

A three-leg discriminator moved those 23 layers on-device and reported
both
halves of this row's prediction rather than speed alone:

| leg | arm | token gate | warm s/token | GPU busy |
|---|---|---|---|---|
| `a3_hostmamba` | host bounce, the SHIPPED default | **96/96 `STRICT
PASS`** | 10.1502 | 7.86% |
| `a3_off` | device `Mamba2ChunkScan` | **95/96 `DIVERGENCE`** | 1.3898
| 10.18% |
| `a3_on` | device `Mamba2StateUpdate` | **95/96 `DIVERGENCE`** | 1.3947
| 10.27% |

**7.28x** warm, on the same cold-prompt exclusion used throughout, and
the busy
fraction rises as predicted. (A 6.64x figure for the same pair includes
the cold
prompt on both sides.)

**It is not a parity number.** Both device legs read 95/96, and
AGENTS.md
establishes the token gate BEFORE a performance result is accepted, so
718.2x
stays the gated figure and the device arm's ~97x-vs-oracle is carried as
an
ungated projection. #1289 is DRAFT for exactly this. And no ceiling is
in sight:
at ~10.2% busy the decode is still ~90% GPU-idle, so A2-Q1 banks 7.28x
without
closing the gap, and A2-Q2b's `lm_head` owes the same test.

**#940 was never the live pointer** -- it has been closed since
2026-08-16. The
live tracker is #1289, blocked by #1388. Corrected everywhere this
branch cited
it; the two pre-existing #940 rows in the issue index belong to other
rows and
are untouched.

**Two findings went to #1388**, which had neither. Its "arch- or
host-specific,
not arm-specific" conclusion came from two DEVICE arms plus a passing
Thor, with
no host-arm leg on GB10; there are now three such legs and all pass
96/96 on the
same binary and box, so the divergence does track the arm. And the
diverging row
is prompt 2 -- which is also the row where the pinned oracle failed to
reproduce
its OWN committed golden (`matched=26` of 32) once its resolved
`block_size`
moved to 512. That is evidence for the benign-near-tie branch and
against a
wrong recurrent carry.

## The keyed records were rebuilt twice, and the second conflict was
POSITIONAL

An automatic three-way merge on this branch had damaged four keyed
records: it
duplicated the `NemotronH paged forward` and `Record-anchor ratchet`
rows in
`docs/BENCHMARKS.md`, duplicated this row's `#1250` index row, and
DROPPED
`ENG-EXPERT-STREAM-DEVICE`'s row from `docs/STATUS.md` and its section
from
`.agents/benchmark-record.md`.

Re-basing on current `main` produced the same class of defect from the
opposite
direction, and nothing would have caught it. The base carried
`**NemotronH paged forward**` directly after `**Developer/row
protocol**`, and
this branch edited it there. Main MOVED that row two positions down,
past
`**Record-anchor ratchet**`, and inserted a DIFFERENT key,
`**NemotronH host re-expansion / decode token**` (`A2-Q2b`), at the
offset the
move vacated. Git therefore conflicted this branch's edit against a row
it has
nothing to do with, and auto-merged main's relocated copy in BELOW the
conflict
as an addition. The conflicted tree held TWO `**NemotronH paged
forward**` rows.
Taking either side wholesale lands a duplicate key, and
`scripts/check-public-doc-tables.py` does not test key uniqueness
(#1543), so it
would have landed silently. This is the #1081 shape again.

Every one was rebuilt the way AGENTS.md prescribes -- take main's
complete
version, re-apply the scoped edit, verify unrelated keys byte-for-byte.
Each
public doc now differs from main by exactly one line, the index carries
main's
rows unchanged with two rows added and no duplicate keys, and the record
is
main's file with this row's one 245-line section inserted and nothing
removed.

"Strict prefix" is the wrong word for the index and it is worth being
exact
about, because the two claims are not the same guarantee. The two rows
do NOT
land at the end. Main's own later rows follow them, which is the
ordinary result
of two branches appending concurrently.

No offset or trailing-line count is quoted here, deliberately. That
number moves
with every merge of `main` -- it read 118 one base ago and 141 at this
head --
and `squash_merge_commit_message = PR_BODY` makes this description an
immutable
commit message, so a coordinate into a moving file is the same defect
here as a
`file:line` anchor is in a record. What IS true is the property the
append-only
rule actually needs, and it is measured at the head this pull request
pushes:
main's lines are an exact ordered subsequence of this tree's, the diff
is a
SINGLE insert opcode, nothing is removed, and no issue key appears
twice.
Main had also edited that scoreboard row's own cells, so the rebuilt row
keeps
this branch's measurement AND main's provenance caveat that the branch's
text
dropped: the 96/96 was read on `5325b7b97`, not on current `main`.

Two repairs to this branch's own `.agents/benchmark-record.md` section
ride
along, because that file is append-only too and this is the last cheap
moment
for either. Its section header carried its dated parenthetical TWICE; of
the 378
`## ` headers in that file it was the only one shaped that way, so it is
a slip
and not a convention, and it is the line a reader greps. And the section
carried
2026-08-18 while sitting between two 2026-08-19 sections; it now sits
below both,
so the file's newest-first order holds. Same 245 lines, different place.

`docs/STATUS.md` needed a different answer. #1491 rewrote that page from
65 rows
to 27, organised by surface rather than by model, with model detail
routed to
`docs/models/`, so the per-model `Nemotron-3.5-Lightning-30B-A3B` row
this branch
edited NO LONGER EXISTS there and a by-key re-apply would assert on a
missing
key. The speed result goes into the existing `Text generation` row's
open-gate
cell instead, one line changed in place, and the page keeps main's 27
rows.

## Also here

**`--num-blocks`, `--kv-cache-memory` and `--repeat` on
`nemotron-h-gen`.** The
driver could not state its own KV budget and inherited the 256-block
fallback
the engine takes when `gpu_memory_utilization` is accepted and does not
size the
pool (#83); both legs of the A3 gate logged that warning, and this run
does not.
`--repeat` runs the battery again over one load, and the verdict reads
grand
totals, so a battery that passed once and diverged on the second leg
cannot
report a pass.

**#1253**, filed and fixed in this flow. The oracle could not run a
model in a
lease for want of `python3-dev` -- Triton compiles `cuda_utils.c` at
runtime and
the failure surfaces four frames up as `Engine core initialization
failed ...
Failed core proc(s): {}`, an empty proc set naming nothing. And `nvcc
--version`
is not the toolkit postcondition: the worker ships a partial CUDA 13.0
whose
compiler answers while cuBLAS dev is absent, so CMake printed all five
feature
lines and then failed with `Target "vllm" links to CUDA::cublasLt but
the target
was not found`. #1185's recorded hazard for the oracle -- a reboot of
the box in
the step after `torch.compile` -- did NOT occur; the watchdog held.

Closes #1250.

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5 [claude-code]

---------

Co-authored-by: Ettore Di Giacinto <mudler@localai.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

records: the NemotronH model-matrix row still reads INVENTORIED and blocked on #496, 23 commits later

2 participants