Name the hbg Graph body after tasks, and hold its membership in graph_context

docs/dfx/args-dump.md

@@ -81,7 +81,7 @@ manifest's `bin_file` is `null`, and every `bin_size` is `0`.
 Level 3 deliberately inherits Level 1's selector semantics; it does not add a
 second mask. `tensormap_and_ringbuffer` registers selection metadata in its
 AICPU per-task table. `host_build_graph` embeds the same mask, ambiguity flags,
-and scalar dtypes in each H2D task image, including cached Graph node
+and scalar dtypes in each H2D task image, including cached in-graph task
 definitions. The device collector consumes either source identically. On `a5`,
 however, the resulting tensor bytes remain untrusted under #1560, so payload
 restoration stays outside this change until that issue is fixed.

docs/dfx/hbg-bind-phases.md

@@ -27,7 +27,7 @@ line per segment per bind at `LOG_TIMING`:
 | ------- | -------------- |
 | `args` | staging readable caller tensors H2D and exposing their existing host buffers to orchestration; pure outputs skip both |
 | `arena_build`, `static_arena`, `gm_heap`, `shared_mem`, `runtime_init` | arena layout, GM heap and shared-memory bring-up |
-| `host_orch` | **all** orchestration: every task submitted, every Graph node recorded, the Definition built |
+| `host_orch` | **all** orchestration: every task submitted, every in-graph task recorded, the Definition built |
 | `graph_upload` | one H2D of the block holding every Definition object, and binding each Graph task to the one with its key. The recorders built the objects in that block's host staging during `host_orch`, so this segment writes their headers and copies in only what did not fit |
 | `arena_h2d` | one H2D of the arena's copied zone and the shared-memory image |
 | `host_view_close` | closing per-run tensor-access regions and any optional device mappings; the current bind path installs none (`count=0 bytes=0`) |
@@ -72,7 +72,7 @@ device lock for the whole job (see
 | Entry point | `SceneTestCase` at level 2, run through the module runner | standalone `main.py`, which owns its L3 `Worker` |
 | Devices | 1 | 2 (EP2/TP2) |
 | Host tasks | 47 | 1131 |
-| Graph replays | 40, of a 277-node Definition | 20, of a 743-node Definition |
+| Graph replays | 40, of a 277-task Definition | 20, of a 743-task Definition |
 | Graph boundary | 26 tensors | 118 tensors, 31 scalars |
 | First-run compile | seconds | **minutes** (369 kernel sources + an 11.6k-line orchestration) |
 | Parameters | host fixture per run | child memory, and `--skip-golden` leaves it uninitialized |
@@ -412,7 +412,7 @@ orchestration views, so it performs no `halHostRegister` calls and reports
 the H2D work but removes that registration side.
 
 Three of these deserve reading together. `host_orch` is the whole story on dsv4 —
-839 `submit_task`, 743 `record_node` and 272 `alloc_tensors` per bind against qwen's
+839 `submit_task`, 743 `record_in_graph_task` and 272 `alloc_tensors` per bind against qwen's
 5, 277 and 2 — and its 2.3 ms of scatter is why a claim about it needs a
 sub-counter rather than a stopwatch. At the pinned commit, `args` plus
 `host_view_close` are two orders of magnitude above everything else while being

docs/investigations/2026-08-hbg-graph-block-decomposition.md

@@ -3,6 +3,11 @@
 **Date**: 2026-08-19, revised 2026-08-20
 **Verdict**: adopted at seven Definitions, once #1929 removed the single recording slot the first measurement failed against — the win is smaller than the orchestration window alone suggests, because seven Definitions are seven images to upload
 
+> **Names in this entry are the ones in use at the time.** A task inside a Graph
+> body was a "node" and the record phase was `record_node`; they are now an
+> in-graph task and `record_in_graph_task`. `strace_timing.py` still accepts the
+> old phase name, so logs from this investigation remain readable.
+
 ## Question
 
 `examples/a2a3/host_build_graph/deepseek_v4_flash_decode` casts its 43-layer

docs/investigations/2026-08-host-orch-phase-tail-is-page-faults.md

@@ -1,5 +1,12 @@
 # 2026-08 — The host-orchestration phase tail is page faults, not the code in the phase
 
+> **Phase and type names in this entry are the ones the tooling emitted at the
+> time.** The per-in-graph-task record phase was `record_node` and is now
+> `record_in_graph_task`; `GraphRecordedNode` is now `RecordedInGraphTask` and
+> `recording.nodes` is `recording.tasks`. The measurements and the archived run
+> directories below keep the old spellings because that is what the logs say, and
+> `strace_timing.py` still accepts `record_node` so those logs remain readable.
+
 ## Question
 
 `host_build_graph`'s host-side bind path shows two shapes on every swimlane of the

docs/investigations/README.md

@@ -84,7 +84,7 @@ that ...".
 
 Newest first.
 
-- [2026-08 — The host-orchestration phase tail is page faults, not the code in the phase](2026-08-host-orch-phase-tail-is-page-faults.md) — root cause of the two shapes on every hbg host swimlane: 447 of the 449 `record_node` calls above 10 µs took a minor fault, 19% of calls carry 79% of the phase, and a fault costs 14–33 µs here against 1.7 µs off-tree because the process's own `mmap`/`munmap` holds `mmap_lock` against every faulting thread — three 64 MiB unmaps in an off-tree reproducer recreate the whole distribution. The fault count is deterministic (1063/1065/1168 per two orchestrations) and drops to 29 when glibc keeps freed memory; the cost per fault varies 2.4× between runs of the same binary, which is the measurement noise that hid this for fourteen iterations. Refutes THP (`PR_SET_THP_DISABLE` leaves the count unchanged), preemption, and node shape; records why the tunables are not a fix (`args` regresses, and the probe build's −53% is the probe amplifying its own subject). Amended 2026-08-23: recording the Definition image into the retained upload staging (8 × ~126 KB per dsv4 bind, previously a vector per recording) moved `graph_upload`'s faults 38 → 1 per bind but left `host_orch`'s count unresolvable in both directions, because a freed 126 KB block is reused without re-faulting — size against glibc's mmap and trim thresholds, not byte count, decides what shows up in this tail. Amended 2026-08-25: retaining the 82 MB SM mirror on the runner (one buffer per pipeline slot instead of one per bind) removes an `mmap` + `munmap` of that size per bind — `hblkhd` stops returning to its pre-bind value on 6 of 6 binds, in both arms of two interleaved repetitions — and shows that a retained buffer must be handed over **uninitialized**: the first implementation used `std::vector::resize`, whose value-initialization faulted in all 20132 pages of the window on each rank's cold bind (~20k minflt against ~1100) and left the whole 82 MB resident. `host_orch`'s warm-bind fault count resolves in neither direction (base [1160, 1256] over eight binds, retained [181, 1268]), since the mirror is ~6 THP faults of a ~1200-fault bind; an earlier attribution of a warm-bind rise to glibc's dynamic mmap threshold is retracted there. That amendment also closes the entry's "Where a fix would go" list — items 1–3 shipped as #1981, item 4 as #1988 plus #2013, and item 3's flat-region form as #2015 — and records that none of them reached the ~1100 faults the submitting thread takes per bind, which is what is left. Amended again the same day: the claim in that amendment that pre-sizing the recorder's node and tensor storage would only help a cold bind is **retracted** — a slot-creation counter shows warm dsv4 binds still creating 1336 node slots of 1679, because #1981's retention is per thread while the pool hands bodies out through one shared FIFO. Reserving each node's own buffer to the cap makes it exactly one page and is *worse* than main (minflt 1070 → 2540); packing every body's tensors into one never-grown bump region is what helps (`record_node` warm min 1702/3423 → 1239/1563 µs). **Amended 2026-08-25 (last)**: the *count × price* framing this entry opened with is refuted by two arms pointing opposite ways — glibc keeping freed memory removes 86% of the faults and buys **no** time, while #2015 removes 6% and buys 29–43%. The count is not a lever; only the price is, and the in-tree `mmap_lock` writer that sets it is **`mprotect`**, which glibc uses to open a non-main arena (26 calls per bind in that band; those arenas only grow, `MADV_DONTNEED` there is 0). Every earlier strace here traced `madvise`/`mmap`/`munmap`/`brk` and not `mprotect`, which is why the in-tree source of the exclusion went unfound for three rounds. Consequence: the residual ~1100 resident-page re-faults per bind — survived #1981, #1988, #2013, #2015, mechanism undetermined — showed **no measurable latency change** when the tunable arm removed 86% of them, so they are **not currently established as a performance defect**; userspace tools are exhausted (`mincore` and pagemap both report presence, not writability), so pricing them at all needs `bpftrace` on `handle_mm_fault`. **Corrected 2026-08-26**: they are a **warm-up cost and they end** — at `f40cacf30`, `host_orch`'s per-bind `minflt` runs 989, 983 (cold), then 114, 173, 54, 3, 13, 13, 11, 8, and reaches 0 by the sixth bind on three independent runs. Every per-bind count in this entry came from three-to-six-round runs divided by the bind count, so each averaged two cold binds and three or four still-decaying ones under a steady-state label; the quantity being divided was never per-bind. Nothing "survived" the four changes, and the mechanism left undetermined for three rounds turned out not to need determining. Control plane at that commit: 0.529 ms min / 0.570 median over 8 warm binds (`host_orch` 0.341/0.364). The reusable half of this — that dropping the cold bind does not reach the steady state — is now a trap in [hbg-bind-phases.md](../dfx/hbg-bind-phases.md). Also carries the per-site fault table (two of its top three sites were deleted by #2019 and #2015 within days — keep the method, not the numbers) and the three tooling traps that each produced a wrong conclusion first
+- [2026-08 — The host-orchestration phase tail is page faults, not the code in the phase](2026-08-host-orch-phase-tail-is-page-faults.md) — root cause of the two shapes on every hbg host swimlane: 447 of the 449 `record_node` (now `record_in_graph_task`) calls above 10 µs took a minor fault, 19% of calls carry 79% of the phase, and a fault costs 14–33 µs here against 1.7 µs off-tree because the process's own `mmap`/`munmap` holds `mmap_lock` against every faulting thread — three 64 MiB unmaps in an off-tree reproducer recreate the whole distribution. The fault count is deterministic (1063/1065/1168 per two orchestrations) and drops to 29 when glibc keeps freed memory; the cost per fault varies 2.4× between runs of the same binary, which is the measurement noise that hid this for fourteen iterations. Refutes THP (`PR_SET_THP_DISABLE` leaves the count unchanged), preemption, and node shape; records why the tunables are not a fix (`args` regresses, and the probe build's −53% is the probe amplifying its own subject). Amended 2026-08-23: recording the Definition image into the retained upload staging (8 × ~126 KB per dsv4 bind, previously a vector per recording) moved `graph_upload`'s faults 38 → 1 per bind but left `host_orch`'s count unresolvable in both directions, because a freed 126 KB block is reused without re-faulting — size against glibc's mmap and trim thresholds, not byte count, decides what shows up in this tail. Amended 2026-08-25: retaining the 82 MB SM mirror on the runner (one buffer per pipeline slot instead of one per bind) removes an `mmap` + `munmap` of that size per bind — `hblkhd` stops returning to its pre-bind value on 6 of 6 binds, in both arms of two interleaved repetitions — and shows that a retained buffer must be handed over **uninitialized**: the first implementation used `std::vector::resize`, whose value-initialization faulted in all 20132 pages of the window on each rank's cold bind (~20k minflt against ~1100) and left the whole 82 MB resident. `host_orch`'s warm-bind fault count resolves in neither direction (base [1160, 1256] over eight binds, retained [181, 1268]), since the mirror is ~6 THP faults of a ~1200-fault bind; an earlier attribution of a warm-bind rise to glibc's dynamic mmap threshold is retracted there. That amendment also closes the entry's "Where a fix would go" list — items 1–3 shipped as #1981, item 4 as #1988 plus #2013, and item 3's flat-region form as #2015 — and records that none of them reached the ~1100 faults the submitting thread takes per bind, which is what is left. Amended again the same day: the claim in that amendment that pre-sizing the recorder's node and tensor storage would only help a cold bind is **retracted** — a slot-creation counter shows warm dsv4 binds still creating 1336 node slots of 1679, because #1981's retention is per thread while the pool hands bodies out through one shared FIFO. Reserving each node's own buffer to the cap makes it exactly one page and is *worse* than main (minflt 1070 → 2540); packing every body's tensors into one never-grown bump region is what helps (`record_node` warm min 1702/3423 → 1239/1563 µs). **Amended 2026-08-25 (last)**: the *count × price* framing this entry opened with is refuted by two arms pointing opposite ways — glibc keeping freed memory removes 86% of the faults and buys **no** time, while #2015 removes 6% and buys 29–43%. The count is not a lever; only the price is, and the in-tree `mmap_lock` writer that sets it is **`mprotect`**, which glibc uses to open a non-main arena (26 calls per bind in that band; those arenas only grow, `MADV_DONTNEED` there is 0). Every earlier strace here traced `madvise`/`mmap`/`munmap`/`brk` and not `mprotect`, which is why the in-tree source of the exclusion went unfound for three rounds. Consequence: the residual ~1100 resident-page re-faults per bind — survived #1981, #1988, #2013, #2015, mechanism undetermined — showed **no measurable latency change** when the tunable arm removed 86% of them, so they are **not currently established as a performance defect**; userspace tools are exhausted (`mincore` and pagemap both report presence, not writability), so pricing them at all needs `bpftrace` on `handle_mm_fault`. **Corrected 2026-08-26**: they are a **warm-up cost and they end** — at `f40cacf30`, `host_orch`'s per-bind `minflt` runs 989, 983 (cold), then 114, 173, 54, 3, 13, 13, 11, 8, and reaches 0 by the sixth bind on three independent runs. Every per-bind count in this entry came from three-to-six-round runs divided by the bind count, so each averaged two cold binds and three or four still-decaying ones under a steady-state label; the quantity being divided was never per-bind. Nothing "survived" the four changes, and the mechanism left undetermined for three rounds turned out not to need determining. Control plane at that commit: 0.529 ms min / 0.570 median over 8 warm binds (`host_orch` 0.341/0.364). The reusable half of this — that dropping the cold bind does not reach the steady state — is now a trap in [hbg-bind-phases.md](../dfx/hbg-bind-phases.md). Also carries the per-site fault table (two of its top three sites were deleted by #2019 and #2015 within days — keep the method, not the numbers) and the three tooling traps that each produced a wrong conclusion first
 - [2026-08 — hbg: per-block Graph Definitions and cross-layer reuse](2026-08-hbg-graph-block-decomposition.md) — adopted at seven Definitions covering all 43 layers, after #1929 replaced the single recording slot the first attempt measured against (it demoted a Graph whose key differed from the in-flight recording's: 79 of 82 intended submissions recorded, host tasks rose 1131 → 1486). Now host submissions 1131 → 129, `host_orch` −44% and `sm_h2d` −85%, but `graph_upload` +207% for seven images instead of one, so the control plane nets −17% at the per-phase floor and **nothing at the median** — a predictable cost traded for a lower floor and a 133%-wide spread that depends on seven recording threads getting CPU. Keeps the structural map that made the reuse provable (367 kernels → 169 classes / 132 by code alone; which blocks can share a Definition and why the hash-routed MoE cannot), the two indices and the last layer's different `hc_post` destination, and the arithmetic that a recorded node costs about what a submitted task costs so break-even sits near three occurrences
 - [2026-08 — hbg: uploading Graph Definitions once as shared device objects](2026-08-hbg-graph-definition-single-upload.md) — cut the per-replay 130 KB Definition re-serialization (image build 931→24 µs, orch total −54%), and confirmed the H2D stage is latency- not bandwidth-bound. **Amended 2026-08-18**: a `--rounds 3` split shows 12.19 of the residual 12.9 ms is the *one-time* `rtMalloc`+`memset` of 40 execution-storage blocks (~53 MB), not per-call latency — real per-call is ~17 µs, so batching the reference submissions is worth ≤0.6 ms, and 88% of this change's own cold-start gain (−1.879 ms in that split) came from execution storage shrinking rather than from the byte reduction it targeted. **Amended 2026-08-25**: the one-time verify gate this change introduced no longer hashes anything — `content_hash`, `verify_state` and the whole-image zero-fill are removed, since `graph_definition_array` plus `bind_graph_topology` already bound every device-side read, so the three ~1 MB passes per dsv4 bind bought no safety the structural checks did not
 - [2026-07 — Why qwen3_14b_decode held a device for 406 s, and the four answers that were wrong](2026-07-qwen-scene-test-406s-decomposition.md) — root cause was torch thread oversubscription in goldens, not any of the big things: the reference walks 3584 tiny slice ops per layer and torch sizes its pool from the core count, so a 320-core host paid 6.35 s/layer against 1.05 s at 4. Capping to 8 (#1601) cut the golden 359 s → ~40 s and let the case rejoin the sweep. Records the measured decomposition (compile 59 s / fixture 13 s / golden 359 s / device tens of ms), how to split queue wait from card-held work via the `npu-lock` timestamps, and why golden caching, vectorisation, a nightly split and `skip_golden` were each dropped