diff --git a/docs/dfx/args-dump.md b/docs/dfx/args-dump.md index 2a47905626..f20cef217a 100644 --- a/docs/dfx/args-dump.md +++ b/docs/dfx/args-dump.md @@ -81,7 +81,7 @@ manifest's `bin_file` is `null`, and every `bin_size` is `0`. Level 3 deliberately inherits Level 1's selector semantics; it does not add a second mask. `tensormap_and_ringbuffer` registers selection metadata in its AICPU per-task table. `host_build_graph` embeds the same mask, ambiguity flags, -and scalar dtypes in each H2D task image, including cached Graph node +and scalar dtypes in each H2D task image, including cached in-graph task definitions. The device collector consumes either source identically. On `a5`, however, the resulting tensor bytes remain untrusted under #1560, so payload restoration stays outside this change until that issue is fixed. diff --git a/docs/dfx/hbg-bind-phases.md b/docs/dfx/hbg-bind-phases.md index e7d0d3fd1e..fcb8d04c24 100644 --- a/docs/dfx/hbg-bind-phases.md +++ b/docs/dfx/hbg-bind-phases.md @@ -27,7 +27,7 @@ line per segment per bind at `LOG_TIMING`: | ------- | -------------- | | `args` | staging readable caller tensors H2D and exposing their existing host buffers to orchestration; pure outputs skip both | | `arena_build`, `static_arena`, `gm_heap`, `shared_mem`, `runtime_init` | arena layout, GM heap and shared-memory bring-up | -| `host_orch` | **all** orchestration: every task submitted, every Graph node recorded, the Definition built | +| `host_orch` | **all** orchestration: every task submitted, every in-graph task recorded, the Definition built | | `graph_upload` | one H2D of the block holding every Definition object, and binding each Graph task to the one with its key. The recorders built the objects in that block's host staging during `host_orch`, so this segment writes their headers and copies in only what did not fit | | `arena_h2d` | one H2D of the arena's copied zone and the shared-memory image | | `host_view_close` | closing per-run tensor-access regions and any optional device mappings; the current bind path installs none (`count=0 bytes=0`) | @@ -72,7 +72,7 @@ device lock for the whole job (see | Entry point | `SceneTestCase` at level 2, run through the module runner | standalone `main.py`, which owns its L3 `Worker` | | Devices | 1 | 2 (EP2/TP2) | | Host tasks | 47 | 1131 | -| Graph replays | 40, of a 277-node Definition | 20, of a 743-node Definition | +| Graph replays | 40, of a 277-task Definition | 20, of a 743-task Definition | | Graph boundary | 26 tensors | 118 tensors, 31 scalars | | First-run compile | seconds | **minutes** (369 kernel sources + an 11.6k-line orchestration) | | Parameters | host fixture per run | child memory, and `--skip-golden` leaves it uninitialized | @@ -412,7 +412,7 @@ orchestration views, so it performs no `halHostRegister` calls and reports the H2D work but removes that registration side. Three of these deserve reading together. `host_orch` is the whole story on dsv4 — -839 `submit_task`, 743 `record_node` and 272 `alloc_tensors` per bind against qwen's +839 `submit_task`, 743 `record_in_graph_task` and 272 `alloc_tensors` per bind against qwen's 5, 277 and 2 — and its 2.3 ms of scatter is why a claim about it needs a sub-counter rather than a stopwatch. At the pinned commit, `args` plus `host_view_close` are two orders of magnitude above everything else while being diff --git a/docs/investigations/2026-08-hbg-graph-block-decomposition.md b/docs/investigations/2026-08-hbg-graph-block-decomposition.md index ac334d9e5d..1ac02a158c 100644 --- a/docs/investigations/2026-08-hbg-graph-block-decomposition.md +++ b/docs/investigations/2026-08-hbg-graph-block-decomposition.md @@ -3,6 +3,11 @@ **Date**: 2026-08-19, revised 2026-08-20 **Verdict**: adopted at seven Definitions, once #1929 removed the single recording slot the first measurement failed against — the win is smaller than the orchestration window alone suggests, because seven Definitions are seven images to upload +> **Names in this entry are the ones in use at the time.** A task inside a Graph +> body was a "node" and the record phase was `record_node`; they are now an +> in-graph task and `record_in_graph_task`. `strace_timing.py` still accepts the +> old phase name, so logs from this investigation remain readable. + ## Question `examples/a2a3/host_build_graph/deepseek_v4_flash_decode` casts its 43-layer diff --git a/docs/investigations/2026-08-host-orch-phase-tail-is-page-faults.md b/docs/investigations/2026-08-host-orch-phase-tail-is-page-faults.md index 601e5aa0fe..33ec020804 100644 --- a/docs/investigations/2026-08-host-orch-phase-tail-is-page-faults.md +++ b/docs/investigations/2026-08-host-orch-phase-tail-is-page-faults.md @@ -1,5 +1,12 @@ # 2026-08 — The host-orchestration phase tail is page faults, not the code in the phase +> **Phase and type names in this entry are the ones the tooling emitted at the +> time.** The per-in-graph-task record phase was `record_node` and is now +> `record_in_graph_task`; `GraphRecordedNode` is now `RecordedInGraphTask` and +> `recording.nodes` is `recording.tasks`. The measurements and the archived run +> directories below keep the old spellings because that is what the logs say, and +> `strace_timing.py` still accepts `record_node` so those logs remain readable. + ## Question `host_build_graph`'s host-side bind path shows two shapes on every swimlane of the diff --git a/docs/investigations/README.md b/docs/investigations/README.md index 5cb3d7545f..5acb46c6ab 100644 --- a/docs/investigations/README.md +++ b/docs/investigations/README.md @@ -84,7 +84,7 @@ that ...". Newest first. -- [2026-08 — The host-orchestration phase tail is page faults, not the code in the phase](2026-08-host-orch-phase-tail-is-page-faults.md) — root cause of the two shapes on every hbg host swimlane: 447 of the 449 `record_node` calls above 10 µs took a minor fault, 19% of calls carry 79% of the phase, and a fault costs 14–33 µs here against 1.7 µs off-tree because the process's own `mmap`/`munmap` holds `mmap_lock` against every faulting thread — three 64 MiB unmaps in an off-tree reproducer recreate the whole distribution. The fault count is deterministic (1063/1065/1168 per two orchestrations) and drops to 29 when glibc keeps freed memory; the cost per fault varies 2.4× between runs of the same binary, which is the measurement noise that hid this for fourteen iterations. Refutes THP (`PR_SET_THP_DISABLE` leaves the count unchanged), preemption, and node shape; records why the tunables are not a fix (`args` regresses, and the probe build's −53% is the probe amplifying its own subject). Amended 2026-08-23: recording the Definition image into the retained upload staging (8 × ~126 KB per dsv4 bind, previously a vector per recording) moved `graph_upload`'s faults 38 → 1 per bind but left `host_orch`'s count unresolvable in both directions, because a freed 126 KB block is reused without re-faulting — size against glibc's mmap and trim thresholds, not byte count, decides what shows up in this tail. Amended 2026-08-25: retaining the 82 MB SM mirror on the runner (one buffer per pipeline slot instead of one per bind) removes an `mmap` + `munmap` of that size per bind — `hblkhd` stops returning to its pre-bind value on 6 of 6 binds, in both arms of two interleaved repetitions — and shows that a retained buffer must be handed over **uninitialized**: the first implementation used `std::vector::resize`, whose value-initialization faulted in all 20132 pages of the window on each rank's cold bind (~20k minflt against ~1100) and left the whole 82 MB resident. `host_orch`'s warm-bind fault count resolves in neither direction (base [1160, 1256] over eight binds, retained [181, 1268]), since the mirror is ~6 THP faults of a ~1200-fault bind; an earlier attribution of a warm-bind rise to glibc's dynamic mmap threshold is retracted there. That amendment also closes the entry's "Where a fix would go" list — items 1–3 shipped as #1981, item 4 as #1988 plus #2013, and item 3's flat-region form as #2015 — and records that none of them reached the ~1100 faults the submitting thread takes per bind, which is what is left. Amended again the same day: the claim in that amendment that pre-sizing the recorder's node and tensor storage would only help a cold bind is **retracted** — a slot-creation counter shows warm dsv4 binds still creating 1336 node slots of 1679, because #1981's retention is per thread while the pool hands bodies out through one shared FIFO. Reserving each node's own buffer to the cap makes it exactly one page and is *worse* than main (minflt 1070 → 2540); packing every body's tensors into one never-grown bump region is what helps (`record_node` warm min 1702/3423 → 1239/1563 µs). **Amended 2026-08-25 (last)**: the *count × price* framing this entry opened with is refuted by two arms pointing opposite ways — glibc keeping freed memory removes 86% of the faults and buys **no** time, while #2015 removes 6% and buys 29–43%. The count is not a lever; only the price is, and the in-tree `mmap_lock` writer that sets it is **`mprotect`**, which glibc uses to open a non-main arena (26 calls per bind in that band; those arenas only grow, `MADV_DONTNEED` there is 0). Every earlier strace here traced `madvise`/`mmap`/`munmap`/`brk` and not `mprotect`, which is why the in-tree source of the exclusion went unfound for three rounds. Consequence: the residual ~1100 resident-page re-faults per bind — survived #1981, #1988, #2013, #2015, mechanism undetermined — showed **no measurable latency change** when the tunable arm removed 86% of them, so they are **not currently established as a performance defect**; userspace tools are exhausted (`mincore` and pagemap both report presence, not writability), so pricing them at all needs `bpftrace` on `handle_mm_fault`. **Corrected 2026-08-26**: they are a **warm-up cost and they end** — at `f40cacf30`, `host_orch`'s per-bind `minflt` runs 989, 983 (cold), then 114, 173, 54, 3, 13, 13, 11, 8, and reaches 0 by the sixth bind on three independent runs. Every per-bind count in this entry came from three-to-six-round runs divided by the bind count, so each averaged two cold binds and three or four still-decaying ones under a steady-state label; the quantity being divided was never per-bind. Nothing "survived" the four changes, and the mechanism left undetermined for three rounds turned out not to need determining. Control plane at that commit: 0.529 ms min / 0.570 median over 8 warm binds (`host_orch` 0.341/0.364). The reusable half of this — that dropping the cold bind does not reach the steady state — is now a trap in [hbg-bind-phases.md](../dfx/hbg-bind-phases.md). Also carries the per-site fault table (two of its top three sites were deleted by #2019 and #2015 within days — keep the method, not the numbers) and the three tooling traps that each produced a wrong conclusion first +- [2026-08 — The host-orchestration phase tail is page faults, not the code in the phase](2026-08-host-orch-phase-tail-is-page-faults.md) — root cause of the two shapes on every hbg host swimlane: 447 of the 449 `record_node` (now `record_in_graph_task`) calls above 10 µs took a minor fault, 19% of calls carry 79% of the phase, and a fault costs 14–33 µs here against 1.7 µs off-tree because the process's own `mmap`/`munmap` holds `mmap_lock` against every faulting thread — three 64 MiB unmaps in an off-tree reproducer recreate the whole distribution. The fault count is deterministic (1063/1065/1168 per two orchestrations) and drops to 29 when glibc keeps freed memory; the cost per fault varies 2.4× between runs of the same binary, which is the measurement noise that hid this for fourteen iterations. Refutes THP (`PR_SET_THP_DISABLE` leaves the count unchanged), preemption, and node shape; records why the tunables are not a fix (`args` regresses, and the probe build's −53% is the probe amplifying its own subject). Amended 2026-08-23: recording the Definition image into the retained upload staging (8 × ~126 KB per dsv4 bind, previously a vector per recording) moved `graph_upload`'s faults 38 → 1 per bind but left `host_orch`'s count unresolvable in both directions, because a freed 126 KB block is reused without re-faulting — size against glibc's mmap and trim thresholds, not byte count, decides what shows up in this tail. Amended 2026-08-25: retaining the 82 MB SM mirror on the runner (one buffer per pipeline slot instead of one per bind) removes an `mmap` + `munmap` of that size per bind — `hblkhd` stops returning to its pre-bind value on 6 of 6 binds, in both arms of two interleaved repetitions — and shows that a retained buffer must be handed over **uninitialized**: the first implementation used `std::vector::resize`, whose value-initialization faulted in all 20132 pages of the window on each rank's cold bind (~20k minflt against ~1100) and left the whole 82 MB resident. `host_orch`'s warm-bind fault count resolves in neither direction (base [1160, 1256] over eight binds, retained [181, 1268]), since the mirror is ~6 THP faults of a ~1200-fault bind; an earlier attribution of a warm-bind rise to glibc's dynamic mmap threshold is retracted there. That amendment also closes the entry's "Where a fix would go" list — items 1–3 shipped as #1981, item 4 as #1988 plus #2013, and item 3's flat-region form as #2015 — and records that none of them reached the ~1100 faults the submitting thread takes per bind, which is what is left. Amended again the same day: the claim in that amendment that pre-sizing the recorder's node and tensor storage would only help a cold bind is **retracted** — a slot-creation counter shows warm dsv4 binds still creating 1336 node slots of 1679, because #1981's retention is per thread while the pool hands bodies out through one shared FIFO. Reserving each node's own buffer to the cap makes it exactly one page and is *worse* than main (minflt 1070 → 2540); packing every body's tensors into one never-grown bump region is what helps (`record_node` warm min 1702/3423 → 1239/1563 µs). **Amended 2026-08-25 (last)**: the *count × price* framing this entry opened with is refuted by two arms pointing opposite ways — glibc keeping freed memory removes 86% of the faults and buys **no** time, while #2015 removes 6% and buys 29–43%. The count is not a lever; only the price is, and the in-tree `mmap_lock` writer that sets it is **`mprotect`**, which glibc uses to open a non-main arena (26 calls per bind in that band; those arenas only grow, `MADV_DONTNEED` there is 0). Every earlier strace here traced `madvise`/`mmap`/`munmap`/`brk` and not `mprotect`, which is why the in-tree source of the exclusion went unfound for three rounds. Consequence: the residual ~1100 resident-page re-faults per bind — survived #1981, #1988, #2013, #2015, mechanism undetermined — showed **no measurable latency change** when the tunable arm removed 86% of them, so they are **not currently established as a performance defect**; userspace tools are exhausted (`mincore` and pagemap both report presence, not writability), so pricing them at all needs `bpftrace` on `handle_mm_fault`. **Corrected 2026-08-26**: they are a **warm-up cost and they end** — at `f40cacf30`, `host_orch`'s per-bind `minflt` runs 989, 983 (cold), then 114, 173, 54, 3, 13, 13, 11, 8, and reaches 0 by the sixth bind on three independent runs. Every per-bind count in this entry came from three-to-six-round runs divided by the bind count, so each averaged two cold binds and three or four still-decaying ones under a steady-state label; the quantity being divided was never per-bind. Nothing "survived" the four changes, and the mechanism left undetermined for three rounds turned out not to need determining. Control plane at that commit: 0.529 ms min / 0.570 median over 8 warm binds (`host_orch` 0.341/0.364). The reusable half of this — that dropping the cold bind does not reach the steady state — is now a trap in [hbg-bind-phases.md](../dfx/hbg-bind-phases.md). Also carries the per-site fault table (two of its top three sites were deleted by #2019 and #2015 within days — keep the method, not the numbers) and the three tooling traps that each produced a wrong conclusion first - [2026-08 — hbg: per-block Graph Definitions and cross-layer reuse](2026-08-hbg-graph-block-decomposition.md) — adopted at seven Definitions covering all 43 layers, after #1929 replaced the single recording slot the first attempt measured against (it demoted a Graph whose key differed from the in-flight recording's: 79 of 82 intended submissions recorded, host tasks rose 1131 → 1486). Now host submissions 1131 → 129, `host_orch` −44% and `sm_h2d` −85%, but `graph_upload` +207% for seven images instead of one, so the control plane nets −17% at the per-phase floor and **nothing at the median** — a predictable cost traded for a lower floor and a 133%-wide spread that depends on seven recording threads getting CPU. Keeps the structural map that made the reuse provable (367 kernels → 169 classes / 132 by code alone; which blocks can share a Definition and why the hash-routed MoE cannot), the two indices and the last layer's different `hc_post` destination, and the arithmetic that a recorded node costs about what a submitted task costs so break-even sits near three occurrences - [2026-08 — hbg: uploading Graph Definitions once as shared device objects](2026-08-hbg-graph-definition-single-upload.md) — cut the per-replay 130 KB Definition re-serialization (image build 931→24 µs, orch total −54%), and confirmed the H2D stage is latency- not bandwidth-bound. **Amended 2026-08-18**: a `--rounds 3` split shows 12.19 of the residual 12.9 ms is the *one-time* `rtMalloc`+`memset` of 40 execution-storage blocks (~53 MB), not per-call latency — real per-call is ~17 µs, so batching the reference submissions is worth ≤0.6 ms, and 88% of this change's own cold-start gain (−1.879 ms in that split) came from execution storage shrinking rather than from the byte reduction it targeted. **Amended 2026-08-25**: the one-time verify gate this change introduced no longer hashes anything — `content_hash`, `verify_state` and the whole-image zero-fill are removed, since `graph_definition_array` plus `bind_graph_topology` already bound every device-side read, so the three ~1 MB passes per dsv4 bind bought no safety the structural checks did not - [2026-07 — Why qwen3_14b_decode held a device for 406 s, and the four answers that were wrong](2026-07-qwen-scene-test-406s-decomposition.md) — root cause was torch thread oversubscription in goldens, not any of the big things: the reference walks 3584 tiny slice ops per layer and torch sizes its pool from the core count, so a 320-core host paid 6.35 s/layer against 1.05 s at 4. Capping to 8 (#1601) cut the golden 359 s → ~40 s and let the case rejoin the sweep. Records the measured decomposition (compile 59 s / fixture 13 s / golden 359 s / device tens of ms), how to split queue wait from card-held work via the `npu-lock` timestamps, and why golden caching, vectorisation, a nightly split and `skip_golden` were each dropped diff --git a/examples/a2a3/host_build_graph/deepseek_v4_flash_decode/README.md b/examples/a2a3/host_build_graph/deepseek_v4_flash_decode/README.md index 73842fee3c..c9c16c5d8c 100644 --- a/examples/a2a3/host_build_graph/deepseek_v4_flash_decode/README.md +++ b/examples/a2a3/host_build_graph/deepseek_v4_flash_decode/README.md @@ -26,7 +26,7 @@ untouched. Eight Definitions cover all 43 layers: -- `csa_attn_block` (50 nodes) / `csa_moe_block` (32) and `hca_attn_block` (35) / +- `csa_attn_block` (50 in-graph tasks) / `csa_moe_block` (32) and `hca_attn_block` (35) / `hca_moe_block` (31) — the decoder loop's two alternating layer shapes, layers 2..41, plus layer 42 replaying `csa_attn_block` and `hca_moe_block`. `csa_moe_block` records two Definitions: its routing kernel is `route_hash_1` @@ -34,7 +34,7 @@ Eight Definitions cover all 43 layers: key, so the predicate is a Graph config value rather than a host-side `if` the first recorded layer would settle for every replay. - `swa_attn_block` (28) — the two peeled sliding-window attentions of layers 0 - and 1. Their nodes are pairwise alpha-equivalent, so layer 1 replays what layer + and 1. Their in-graph tasks are pairwise alpha-equivalent, so layer 1 replays what layer 0 recorded. - `hash_moe_l0_block` (31) / `hash_moe_l1_block` (31) — the peeled MoE scopes. These cannot share a Definition: `dispatch_wait` folds the MoE epoch in as a @@ -96,14 +96,14 @@ from the same body submitted task by task, so they are worth keeping written down: 1. **The recorder inferred dependencies from the allocation site, not the last - writer.** A recorded node's fanin came only from tensor args classified - `INTERNAL` — which names whichever node's packed window holds the bytes, i.e. + writer.** A recorded task's fanin came only from tensor args classified + `INTERNAL` — which names whichever in-graph task's packed window holds the bytes, i.e. the allocator — plus explicit `set_dependencies`. Every write-then-read through an `alloc_tensors` buffer or a boundary view was therefore unordered, and a Definition replayed a DAG the body does not have when its tasks are submitted individually. Measured on the pre-split single-Definition form of this body: 1348 edges against the 2143 the ordinary path computes for the same - tasks, 543 of 561 comparable nodes short. On device that ran + tasks, 543 of 561 comparable in-graph tasks short. On device that ran `csa_slots_build_valid_qk_plan` before the `topk` that fills its input, so `qk_pv_1` gathered KV pages at addresses the bus rejected. The recorder now runs the same `compute_task_fanin` / `register_task_outputs` the ring path diff --git a/simpler_setup/tools/strace_timing.py b/simpler_setup/tools/strace_timing.py index c9deba1b99..ca1a63a2d8 100644 --- a/simpler_setup/tools/strace_timing.py +++ b/simpler_setup/tools/strace_timing.py @@ -951,6 +951,14 @@ def load_host_phase_records(paths): } ) +# Phases a recorder worker emits, so a record carrying a tid of its own belongs +# on the recorder lane rather than the main one. "record_node" is the name the +# runtime emitted for an in-graph task before it was renamed; logs and the +# archived runs cited in docs/investigations/ still carry it, and an unknown +# phase name here is silently attributed to host_main rather than rejected, so +# both spellings stay accepted. +_RECORD_WORKER_PHASE_NAMES = frozenset({"record_in_graph_task", "record_node", "build_definition"}) + def host_record_spans(spans, passes): """Turn phase records into spans nested under their pass's ``bind``. @@ -1004,7 +1012,7 @@ def host_record_spans(spans, passes): else: name = f"{_PREPARE_SPAN}.host_orch.{phase}" depth = parent.depth + 2 - if phase in {"record_node", "build_definition"} and is_record_worker: + if phase in _RECORD_WORKER_PHASE_NAMES and is_record_worker: phase_thread = "graph_record_worker" elif phase == "graph_submit": phase_thread = "graph_submit_main" diff --git a/simpler_setup/tools/swimlane_converter.py b/simpler_setup/tools/swimlane_converter.py index 62e7c1cc57..8212a15ab1 100644 --- a/simpler_setup/tools/swimlane_converter.py +++ b/simpler_setup/tools/swimlane_converter.py @@ -125,10 +125,10 @@ def format_task_display(task_id): return f"r{ring}t{local}" -def _decode_graph_node_task_id(task_id): - """Decode Scheduler-owned Graph-node ids. +def _decode_in_graph_task_id(task_id): + """Decode Scheduler-owned in-graph task ids. - ``host_build_graph`` puts a materialized node in id space 1 (IN_GRAPH) with + ``host_build_graph`` puts a materialized in-graph task in id space 1 (IN_GRAPH) with ``local=(graph_local_id << 10) | task_index``; the stream-visible outer Graph task stays in space 0 (GLOBAL). See src/common/host_build_graph/task_id_encoding.h. """ @@ -140,7 +140,7 @@ def _decode_graph_node_task_id(task_id): def _collect_graph_execution_instances(tasks, scheduler_phases): # noqa: PLR0912 - """Join Graph-node rows to their outer GraphPrepare records.""" + """Join in-graph task rows to their outer GraphPrepare records.""" prepare_by_outer = defaultdict(list) dummy_rows = [] for thread_idx, records in enumerate(scheduler_phases or []): @@ -155,17 +155,17 @@ def _collect_graph_execution_instances(tasks, scheduler_phases): # noqa: PLR091 rows_by_outer = defaultdict(list) for task in tasks: - decoded = _decode_graph_node_task_id(task.get("task_id")) + decoded = _decode_in_graph_task_id(task.get("task_id")) if decoded is not None: - outer_task_id, node_index = decoded - rows_by_outer[outer_task_id].append((task, node_index)) + outer_task_id, task_index = decoded + rows_by_outer[outer_task_id].append((task, task_index)) dummy_by_outer = defaultdict(list) for record, thread_idx in dummy_rows: - decoded = _decode_graph_node_task_id(record.get("task_id")) + decoded = _decode_in_graph_task_id(record.get("task_id")) if decoded is not None: - outer_task_id, node_index = decoded - dummy_by_outer[outer_task_id].append((record, node_index, thread_idx)) + outer_task_id, task_index = decoded + dummy_by_outer[outer_task_id].append((record, task_index, thread_idx)) instances = [] for outer_task_id, prepare_records in prepare_by_outer.items(): @@ -173,8 +173,8 @@ def _collect_graph_execution_instances(tasks, scheduler_phases): # noqa: PLR091 aicpu_rows = dummy_by_outer.get(outer_task_id, []) if not rows and not aicpu_rows: continue - node_indices = {node_index for _, node_index in rows} - node_indices.update(node_index for _, node_index, _ in aicpu_rows) + task_indices = {task_index for _, task_index in rows} + task_indices.update(task_index for _, task_index, _ in aicpu_rows) starts = [ task.get("dispatch_time_us", _task_slice_start_us(task)) if task.get("dispatch_time_us", -1) >= 0 @@ -193,7 +193,7 @@ def _collect_graph_execution_instances(tasks, scheduler_phases): # noqa: PLR091 "outer_task_id": outer_task_id, "rows": rows, "aicpu_rows": aicpu_rows, - "visible_node_indices": sorted(node_indices), + "visible_task_indices": sorted(task_indices), "execution_start_us": min(starts), "execution_end_us": max(ends), "prepare_start_us": prepare_start_us, @@ -1443,23 +1443,23 @@ def generate_chrome_trace_json( # noqa: PLR0912, PLR0913, PLR0915 ) for instance in graph_instances: outer_display = format_task_display(instance["outer_task_id"]) - node_indices = instance["visible_node_indices"] + task_indices = instance["visible_task_indices"] events.append( { "args": { "outer_task_id": instance["outer_task_id"], - "visible_node_count": len(node_indices), - "visible_node_index_min": min(node_indices), - "visible_node_index_max": max(node_indices), + "visible_in_graph_task_count": len(task_indices), + "visible_in_graph_task_index_min": min(task_indices), + "visible_in_graph_task_index_max": max(task_indices), "prepare_slice_count": instance["prepare_slice_count"], "prepare_duration_us": instance["prepare_duration_us"], "execution_start_us": instance["execution_start_us"], "execution_duration_us": instance["execution_end_us"] - instance["execution_start_us"], - "synthetic_id_layout": "ring1:(outer_task_id << 10) | node_index", + "synthetic_id_layout": "ring1:(outer_task_id << 10) | in_graph_task_index", }, "cat": "graph_execution", "cname": "rail_animation", - "name": f"GraphExecution({outer_display}, {len(node_indices)} visible nodes)", + "name": f"GraphExecution({outer_display}, {len(task_indices)} visible in-graph tasks)", "ph": "X", "pid": 5, "tid": 5000 + instance["lane_idx"], diff --git a/src/a2a3/runtime/host_build_graph/docs/RUNTIME_LOGIC.md b/src/a2a3/runtime/host_build_graph/docs/RUNTIME_LOGIC.md index bbd4528811..a5e7a8cd3c 100644 --- a/src/a2a3/runtime/host_build_graph/docs/RUNTIME_LOGIC.md +++ b/src/a2a3/runtime/host_build_graph/docs/RUNTIME_LOGIC.md @@ -140,8 +140,8 @@ signal, so such a queue accepts one push and then reports full. The ramp is mandatory but it is a function of `capacity` alone, so `SchedulerState::seed_queue_slots()` writes it on the device rather than `bind` shipping 1,775,616 bytes of it. The ready queues are still *not* bounded to -`total_tasks`: graph execution expands a GRAPH task into on-device nodes that push -past the host task count, so every slot must carry a valid sequence. +`total_tasks`: graph execution expands a GRAPH task into on-device in-graph tasks that +push past the host task count, so every slot must carry a valid sequence. Both run before the boot thread publishes `runtime_init_ready_`, which is what releases the peer threads into the dispatch loop, so no push can observe an diff --git a/src/a2a3/runtime/host_build_graph/docs/profiling_levels.md b/src/a2a3/runtime/host_build_graph/docs/profiling_levels.md index 2db0be30ca..f17d9c3b5d 100644 --- a/src/a2a3/runtime/host_build_graph/docs/profiling_levels.md +++ b/src/a2a3/runtime/host_build_graph/docs/profiling_levels.md @@ -246,7 +246,7 @@ so records and spans read against each other with no alignment step. | Group | Kinds | | ----- | ----- | | Bind segments (partition the stage) | `args`, `arena_build`, `static_arena`, `gm_heap`, `shared_mem`, `runtime_init`, `host_orch`, `graph_upload`, `sm_h2d`, `arena_h2d`, `host_view_close` | -| Orchestrator operations (inside `host_orch`) | `submit_task`, `alloc_tensors`, `record_node`, `graph_submit`, `build_definition`, `graph_begin`, `recording_wait`, `graph_commit`, `submit_admit`, `record_handoff`, `generated_args` | +| Orchestrator operations (inside `host_orch`) | `submit_task`, `alloc_tensors`, `record_in_graph_task`, `graph_submit`, `build_definition`, `graph_begin`, `recording_wait`, `graph_commit`, `submit_admit`, `record_handoff`, `generated_args` | Three of the orchestrator kinds end with a task submitted — `submit_task`, `alloc_tensors`, `graph_submit` — so their count is the bind's `total_tasks` @@ -266,7 +266,7 @@ the wrong one produces a number that reads as data and is not: - **A record is an interval** — one operation, start to end. Its `detail` says *which* operation (a task id, a Graph key, the submission index) or *how much* - it covered (`build_definition`'s node count, `recording_wait`'s in-flight + it covered (`build_definition`'s in-graph task count, `recording_wait`'s in-flight count). That is the whole contract. - **A quantity about a segment is an attribute** — `bytes=`, `heap_used=`, `spilled=`, `minflt=`, `nvcsw=`. It goes in the segment's attribute string, @@ -326,7 +326,7 @@ python -m pytest --platform --device 0 --enable-chip-swimlane They come from per-kind counters, not from the record pool. The counters use lock-free atomic additions across the main and recording-worker lanes, with every phase isolated on its own cache line so concurrent `graph_submit` and - `record_node` updates do not false-share. The per-event pool is armed when the + `record_in_graph_task` updates do not false-share. The per-event pool is armed when the artifact is wanted (`SIMPLER_HBG_HOST_PHASE_RECORDS_ENABLE` *and* an output prefix) or whenever the chip swimlane is at `ORCH_PHASES`; a steady-state run satisfies neither, so it pays no pool append and no artifact lock at all. A @@ -344,7 +344,7 @@ python -m pytest --platform --device 0 --enable-chip-swimlane This is the channel to read for a distribution or a per-event timeline; the summed lines cannot express either. Every record carries its producer Linux tid. `strace_timing.py --swimlane --host-phase-records ` draws each record - inside the matching `chip.run.bind`; `record_node` and `build_definition` + inside the matching `chip.run.bind`; `record_in_graph_task` and `build_definition` appear on the `graph record worker` lane, while outer `graph_submit` events appear on the `graph submit main` lane. diff --git a/src/a2a3/runtime/host_build_graph/host/graph_recorder_pool.h b/src/a2a3/runtime/host_build_graph/host/graph_recorder_pool.h index 3a5cea9eff..bf8cfefeee 100644 --- a/src/a2a3/runtime/host_build_graph/host/graph_recorder_pool.h +++ b/src/a2a3/runtime/host_build_graph/host/graph_recorder_pool.h @@ -143,7 +143,7 @@ class GraphAsyncRecordingState { size_t owned_args_index; { - std::lock_guard lock(mutex_); + std::scoped_lock lock(mutex_); if (stopping_ || free_owned_args_count_ == 0 || job_count_ == kJobCapacity) return false; owned_args_index = free_owned_args_[--free_owned_args_count_]; } @@ -175,8 +175,8 @@ class GraphAsyncRecordingState { cv_.notify_one(); // graph_begin() has already installed the keyed in-flight entry and // submitted the zero-heap outer shell. Enqueuing the private job is - // therefore the last dependency of the caller; graph_prepare() and all - // node recording may start after later shells are submitted. + // therefore the last dependency of the caller; graph_prepare() and the + // recording of the body may start after later shells are submitted. return true; } @@ -236,11 +236,11 @@ class GraphAsyncRecordingState { // still reports ready and prewarm() fails: the lazy stand-up in // graph_recording_reset is the backstop for the workers start() adds later. if (!graph_recorder_stand_up_storage()) { - std::lock_guard lock(mutex_); + std::scoped_lock lock(mutex_); storage_failed_ = true; } { - std::lock_guard lock(mutex_); + std::scoped_lock lock(mutex_); ready_workers_++; } cv_.notify_all(); @@ -262,7 +262,7 @@ class GraphAsyncRecordingState { current.function(owned_args_[current.owned_args_index].args()); current.function = {}; { - std::lock_guard lock(mutex_); + std::scoped_lock lock(mutex_); free_owned_args_[free_owned_args_count_++] = current.owned_args_index; active_jobs_--; } @@ -273,7 +273,7 @@ class GraphAsyncRecordingState { void shutdown() { wait(); { - std::lock_guard lock(mutex_); + std::scoped_lock lock(mutex_); stopping_ = true; } cv_.notify_all(); diff --git a/src/a2a3/runtime/host_build_graph/host/host_phase_trace.cpp b/src/a2a3/runtime/host_build_graph/host/host_phase_trace.cpp index 174e0fcf22..12d1eb8567 100644 --- a/src/a2a3/runtime/host_build_graph/host/host_phase_trace.cpp +++ b/src/a2a3/runtime/host_build_graph/host/host_phase_trace.cpp @@ -45,7 +45,7 @@ namespace { // path performs no atomic read-modify-write at all. Sharing them cost the emitting // thread about 0.8 us per record at eight producers, purely in cache-line // ownership: every producer of a Graph workload records the same kind -// (`record_node`), so a per-kind counter is a single line eight threads fight +// (`record_in_graph_task`), so a per-kind counter is a single line eight threads fight // over, and the `alignas(64)` that separates one kind from another does nothing // about that. // @@ -307,7 +307,7 @@ void host_phase_record_bind(uint32_t kind, uint64_t start_ns, const char *attrs, void host_phase_trace_begin(const void *host_api) { TraceState &s = state(); - std::lock_guard lock(s.lifecycle_mutex); + std::scoped_lock lock(s.lifecycle_mutex); s.active.store(false, std::memory_order_release); drain_in_flight_records(s); s.api = static_cast(host_api); @@ -340,7 +340,7 @@ void host_phase_trace_note_submitted(uint64_t submitted_tasks) { void host_phase_trace_end() { TraceState &s = state(); - std::lock_guard lifecycle_lock(s.lifecycle_mutex); + std::scoped_lock lifecycle_lock(s.lifecycle_mutex); if (!s.active.load(std::memory_order_relaxed)) { return; } diff --git a/src/a2a3/runtime/host_build_graph/host/runtime_maker.cpp b/src/a2a3/runtime/host_build_graph/host/runtime_maker.cpp index ef555f55df..90d8f006e4 100644 --- a/src/a2a3/runtime/host_build_graph/host/runtime_maker.cpp +++ b/src/a2a3/runtime/host_build_graph/host/runtime_maker.cpp @@ -471,7 +471,7 @@ bool bind_graph_definitions( return false; } GraphExecutionStorageLayout storage_layout{}; - if (definition->task_count == 0 || definition->task_count > GRAPH_MAX_NODES || + if (definition->task_count == 0 || definition->task_count > MAX_IN_GRAPH_TASKS || definition->full_key != upload->full_key || !graph_execution_storage_layout( static_cast(definition->task_count), definition->tensor_arg_count, @@ -492,21 +492,27 @@ bool bind_graph_definitions( return false; } const uintptr_t storage_addr = outer_base + definition->required_heap; - if (storage_addr % alignof(GraphNodeStorage) != 0) { + if (storage_addr % alignof(InGraphTaskStorage) != 0) { LOG_ERROR("host-orch: Graph runtime storage address is misaligned"); return false; } PackedDefinition &packed_definition = object_it->second; if (!packed_definition.populations_ready) { - const GraphNodeDefinition *nodes = - graph_definition_array(*definition, definition->off_nodes, definition->task_count); - if (nodes == nullptr) { - LOG_ERROR("host-orch: invalid Graph Definition node array"); + const InGraphTaskDefinition *tasks = graph_definition_array( + *definition, definition->off_in_graph_tasks, definition->task_count + ); + if (tasks == nullptr) { + LOG_ERROR("host-orch: invalid Graph Definition in-graph task array"); return false; } for (uint32_t i = 0; i < definition->task_count; ++i) { + // Sizing takes the kind materialize will give this task. add_task + // singles out GRAPH and routes everything else by shape, and a Graph + // body member is never the shell, so the shape decides. Derived here + // the same way the device derives it, so the two cannot drift. + const ActiveMask mask(tasks[i].active_mask); packed_definition.ready_queue_populations.add_task( - ActiveMask(nodes[i].active_mask), TaskAttrs(nodes[i].task_attrs), TaskKind::GRAPH_NODE + mask, TaskAttrs(tasks[i].task_attrs), mask.is_dummy() ? TaskKind::DUMMY : TaskKind::KERNEL ); } packed_definition.populations_ready = true; @@ -839,7 +845,7 @@ int32_t run_host_orchestration( reinterpret_cast(gm_heap) < HEAP_VIRTUAL_BASE && "device memory reaches into the virtual heap window" ); // The alignment bind_graph_definitions checked on the virtual base — a Graph - // task's runtime storage must land on alignof(GraphNodeStorage) — carries to + // task's runtime storage must land on alignof(InGraphTaskStorage) — carries to // the real base only while the two are congruent: both are aligned to // kDefaultBaseAlign, and that covers the storage's own requirement. static_assert( @@ -847,8 +853,8 @@ int32_t run_host_orchestration( "the virtual heap base must share the committed region's alignment" ); static_assert( - alignof(GraphNodeStorage) <= DeviceArena::kDefaultBaseAlign, - "a Graph node's storage alignment must be covered by the heap region's base alignment" + alignof(InGraphTaskStorage) <= DeviceArena::kDefaultBaseAlign, + "an in-graph task's storage alignment must be covered by the heap region's base alignment" ); always_assert(reinterpret_cast(gm_heap) % DeviceArena::kDefaultBaseAlign == 0); const sm_layout::HeapRebase heap_rebase{reinterpret_cast(gm_heap), heap_bytes}; diff --git a/src/a2a3/runtime/host_build_graph/orchestration/orchestration_api.h b/src/a2a3/runtime/host_build_graph/orchestration/orchestration_api.h index b43aae394b..15d41dc1b3 100644 --- a/src/a2a3/runtime/host_build_graph/orchestration/orchestration_api.h +++ b/src/a2a3/runtime/host_build_graph/orchestration/orchestration_api.h @@ -580,7 +580,7 @@ static inline GraphSubmitResult rt_submit_graph_impl(uint64_t graph_key, const G // Handing the recording to a worker. Measured at 10-75 us per start and covered // by no other record: it runs after rt_graph_begin returns, so a swimlane shows // it as a gap with no recorder active — which is what it is, the recorder has - // not reached its first node yet. + // not reached its first in-graph task yet. rt_record_orch_phase(HostPhaseKind::OrchRecordHandoff, _begun_ns, _exit_ns, graph_key); } _phase.count++; diff --git a/src/a2a3/runtime/host_build_graph/runtime/async_wait.h b/src/a2a3/runtime/host_build_graph/runtime/async_wait.h index 3c7b10186d..2fc1176b5c 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/async_wait.h +++ b/src/a2a3/runtime/host_build_graph/runtime/async_wait.h @@ -124,7 +124,7 @@ struct AsyncWaitEntry { struct AsyncPollResult { int32_t completed{0}; // Host-submitted stream tasks completed. - int32_t resolved{0}; // All task completions, including internal Graph nodes. + int32_t resolved{0}; // All task completions, including in-graph tasks. int32_t error_code{SIMPLER_ERROR_NONE}; ChipTaskSlotState *failed_slot_state{nullptr}; }; diff --git a/src/a2a3/runtime/host_build_graph/runtime/orchestrator_core/orchestrator.cpp b/src/a2a3/runtime/host_build_graph/runtime/orchestrator_core/orchestrator.cpp index cf66c916cf..6e78e6c7ea 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/orchestrator_core/orchestrator.cpp +++ b/src/a2a3/runtime/host_build_graph/runtime/orchestrator_core/orchestrator.cpp @@ -303,8 +303,8 @@ struct GraphRecordedScalarSourceRef { size_t source_index{0}; }; -// A node's dispatch predicate, held as the operand tensor plus the element index -// within it rather than the absolute address submit would resolve. The tensor is +// A recorded task's dispatch predicate, held as the operand tensor plus the element +// index within it rather than the absolute address submit would resolve. The tensor is // copied because the caller only lends it for the duration of the submit call. struct GraphRecordedPredicate { simpler::hbg::Tensor operand; @@ -315,7 +315,7 @@ struct GraphRecordedPredicate { PredicateOp op{PredicateOp::NONE}; }; -struct GraphRecordedNode { +struct RecordedInGraphTask { std::array kernel_ids{}; ActiveMask active_mask{}; TaskAttrs task_attrs{}; @@ -323,10 +323,10 @@ struct GraphRecordedNode { int16_t total_required_subtasks{0}; size_t total_output_size{0}; uintptr_t record_packed_base{0}; - // This node's slice of the recording's node tensor pool, as an offset so the node + // This task's slice of the recording's tensor pool, as an offset so the task // carries no address into storage the recording owns. The element addresses are handed - // to the caller through TaskOutputTensors and have to stay valid while every later node - // records, which the pool satisfies by being allocated at the cap and never growing. + // to the caller through TaskOutputTensors and have to stay valid while the rest of the + // body records, which the pool satisfies by being allocated at the cap and never growing. uint32_t tensor_offset{0}; uint32_t tensor_count{0}; // Ranges into the recording's flat arrays. tensor_sources has one entry per @@ -336,18 +336,18 @@ struct GraphRecordedNode { uint32_t scalar_count{0}; uint32_t fanin_offset{0}; uint32_t fanin_count{0}; - // Index into the recording's predicates, or -1 when the node carries none. + // Index into the recording's predicates, or -1 when the task carries none. int32_t predicate_index{-1}; ArgsDumpTaskMetadata dump_metadata; - // Restore the state a fresh node has, without giving up `tensors`'s buffer -- which is - // the point of reusing a slot. A reused slot that keeps any field of the previous body + // Restore the state a freshly recorded task has, in a slot the previous body left + // behind. A reused slot that keeps any field of the previous body // records a Definition that body never had, and predicate_index and dump_metadata are // written only on the paths that have one, so neither can be left to the fill. // - // Field by field rather than `*this = GraphRecordedNode{}`: the latter is immune to - // fields added later, but it costs a second write of the whole struct on every node - // and measured 350-700 us per bind on dsv4's 1679 nodes. The static_assert below is + // Field by field rather than `*this = RecordedInGraphTask{}`: the latter is immune to + // fields added later, but it costs a second write of the whole struct on every recorded + // task and measured 350-700 us per bind on dsv4's 1679 tasks. The static_assert below is // the cheap half of that guarantee -- adding a field breaks the build here, which is // where the reader is told to extend this function. void reset() { @@ -374,19 +374,19 @@ struct GraphRecordedNode { // from the previous body into the next recording -- silently, as a Definition that body // never had. Adding a field changes this size, so the build stops here instead. static_assert( - sizeof(GraphRecordedNode) == 104, "GraphRecordedNode gained or lost a field: extend reset() to match, then " - "update this size" + sizeof(RecordedInGraphTask) == 104, "RecordedInGraphTask gained or lost a field: extend reset() to match, then " + "update this size" ); -// One recorded node's scratch output window. reserve_heap_scratch is a pure bump -// and a node stores the aligned size it advanced by, so consecutive windows abut: +// One recorded task's scratch output window. reserve_heap_scratch is a pure bump +// and a task stores the aligned size it advanced by, so consecutive windows abut: // held in record order these are sorted and disjoint, which is what lets an -// address lookup binary search instead of walking every producer. A node with no +// address lookup binary search instead of walking every producer. A task with no // output advances nothing and owns no entry. struct GraphRecordedOutputRange { uintptr_t begin; uintptr_t end; - uint32_t node_index; + uint32_t task_index; }; // The Graph boundary as the submitting thread captured it, deep-copied because the @@ -412,32 +412,32 @@ struct GraphRecording { // boundary matching while this thread records. const GraphBoundary *boundary{nullptr}; bool unsupported{false}; - std::vector nodes; - // How many of `nodes` this recording has filled. The array itself is never cleared - // and graph_recording_reserve_storage sizes it to the node cap, so a body is recorded - // into slots that already exist: a recorded node makes no allocation at all. - size_t node_count{0}; - // Every recorded node's tensor arguments, packed end to end in one region this - // recording bumps through, and the reason a node holds an offset rather than its own - // buffer: a body's tensors then occupy the bytes they need instead of a page per node - // (a per-node buffer at the cap is 32 x 128 B = exactly one page, so dsv4's 1679 nodes + std::vector tasks; + // How many of `tasks` this recording has filled. The array itself is never cleared + // and graph_recording_reserve_storage sizes it to the in-graph task cap, so a body is + // recorded into slots that already exist: a recorded task makes no allocation at all. + size_t task_count{0}; + // Every recorded task's tensor arguments, packed end to end in one region this + // recording bumps through, and the reason a task holds an offset rather than its own + // buffer: a body's tensors then occupy the bytes they need instead of a page per task + // (a per-task buffer at the cap is 32 x 128 B = exactly one page, so dsv4's 1679 tasks // touched 1679 pages to hold ~210 KB). Allocated once per thread at the cap and never - // grown, which is what keeps a node's borrowed element addresses valid for the rest of + // grown, which is what keeps a task's borrowed element addresses valid for the rest of // the recording. `new[]` default-initializes a trivially-default-constructible Tensor, - // so the region costs no page until a body writes one; each element a node uses is + // so the region costs no page until a body writes one; each element a task uses is // value-initialized before it is filled. - std::unique_ptr node_tensor_pool; - uint32_t node_tensor_cursor{0}; - // Flat per-node arrays, indexed by the ranges on GraphRecordedNode. Held here rather - // than on each node so recording a graph pays no allocation per node per array, and - // reserved to the node cap by graph_recording_reserve_storage so it pays no growth - // either. + std::unique_ptr task_tensor_pool; + uint32_t task_tensor_cursor{0}; + // Flat per-task arrays, indexed by the ranges on RecordedInGraphTask. Held here rather + // than on each recorded task so recording a graph pays no allocation per task per array, + // and reserved to the in-graph task cap by graph_recording_reserve_storage so it pays no + // growth either. std::vector tensor_sources; std::vector scalars; std::vector scalar_sources; std::vector internal_fanins; std::vector output_ranges; - // Indexed by GraphRecordedNode::predicate_index; only predicated nodes + // Indexed by RecordedInGraphTask::predicate_index; only predicated tasks // contribute an entry. std::vector predicates; // Hazard state for the recorded body, owned per recorder thread because @@ -448,14 +448,14 @@ struct GraphRecording { // (register_task_outputs, STEP 4). The shadow-record path replaces // submit_task_common wholesale, so without a map of its own the recorder // can only see the edges tensor-source classification yields — and that - // classification answers "which node's packed window holds these bytes", + // classification answers "which recorded task's packed window holds these bytes", // i.e. who ALLOCATED the buffer, never who wrote it last. A body that // allocates once with alloc_tensors and then writes in place with add_inout - // (the shape every generated orchestration uses) would therefore record a - // node with no edge to its actual producer, and the Definition would replay + // (the shape every generated orchestration uses) would therefore record an + // in-graph task with no edge to its actual producer, and the Definition would replay // a DAG the same body never had when submitted task by task. ChipTensorMap tensor_map{}; - // Set once both the hazard map and the node tensor pool are up, and only then: the two + // Set once both the hazard map and the tensor pool are up, and only then: the two // allocate, so a flag set by the first would let a thread whose second allocation // failed skip the stand-up on its next recording and record through a null pool. bool storage_ready{false}; @@ -468,8 +468,8 @@ struct GraphRecording { bool in_manual_scope() const { return scope_stack_top >= manual_begin_depth; } - simpler::hbg::Tensor *node_tensors(const GraphRecordedNode &node) const { - return node_tensor_pool.get() + node.tensor_offset; + simpler::hbg::Tensor *task_tensors(const RecordedInGraphTask &task) const { + return task_tensor_pool.get() + task.tensor_offset; } const GraphTaskArgs *boundary_args() const { return boundary == nullptr ? nullptr : boundary->args; } @@ -639,7 +639,7 @@ template GraphRecordedScalarSourceRef graph_classify_scalar(const GraphRecording &recording, const ArgT &args, int32_t scalar_index) { if (recording.boundary_args() == nullptr) return {}; - // Identity, not type: an internal node's Arg and the boundary Arg have + // Identity, not type: an in-graph task's Arg and the boundary Arg have // different capacities, so compare the addresses through void. if (static_cast(&args) == static_cast(recording.boundary_args()) && scalar_index < recording.boundary_args()->scalar_count()) { @@ -664,7 +664,7 @@ graph_classify_scalar(const GraphRecording &recording, const ArgT &args, int32_t } // Entry capacity for one recorded body's hazard map. A Definition is capped at -// GRAPH_MAX_NODES nodes and each node registers at most its INOUT/OUTPUT_EXISTING +// MAX_IN_GRAPH_TASKS tasks and each recorded task registers at most its INOUT/OUTPUT_EXISTING // args, so this bounds the worst realistic body while staying a small fraction of // the ordinary path's whole-orchestration pool (CHIP_TENSORMAP_POOL_SIZE). Exhausting // it marks the recording unsupported, which graph_commit reports as @@ -672,18 +672,18 @@ graph_classify_scalar(const GraphRecording &recording, const ArgT &args, int32_t // is no ordinary-path fallback left to take. constexpr int32_t GRAPH_RECORD_TENSORMAP_POOL_SIZE = 16384; -// Elements in the recording's node tensor pool: every node a body can hold, times every -// tensor argument a node can carry. An in-cap body therefore always fits, and the bump -// cursor is checked anyway because a body that overshoots GRAPH_MAX_NODES keeps recording -// so it can finish. -constexpr size_t GRAPH_RECORD_NODE_TENSOR_POOL_ELEMS = - static_cast(GRAPH_MAX_NODES) * static_cast(CORE_MAX_TENSOR_ARGS); +// Elements in the recording's tensor pool: every in-graph task a body can hold, times +// every tensor argument one such task can carry. An in-cap body therefore always fits, and +// the bump cursor is checked anyway because a body that overshoots MAX_IN_GRAPH_TASKS keeps +// recording so it can finish. +constexpr size_t GRAPH_RECORD_TENSOR_POOL_ELEMS = + static_cast(MAX_IN_GRAPH_TASKS) * static_cast(CORE_MAX_TENSOR_ARGS); // The graph_local_id a recorded task's IN_GRAPH id carries. A recorded task belongs // to no Graph task yet -- every shell replaying the Definition re-mints the id with // its own local id at materialize -- so record time names a task by its index alone, // and the id's low field is that index and nothing else. That is what keeps the -// index inside the GRAPH_MAX_NODES task chains the recording's hazard map is +// index inside the MAX_IN_GRAPH_TASKS task chains the recording's hazard map is // dimensioned for. constexpr uint32_t GRAPH_RECORD_NO_OWNING_GRAPH = 0; @@ -691,7 +691,7 @@ constexpr uint32_t GRAPH_RECORD_NO_OWNING_GRAPH = 0; // reported to the caller, which abandons the recording rather than producing a // Definition with inferred edges missing. bool graph_recording_init_tensor_map(GraphRecording &recording) { - return recording.tensor_map.init(CHIP_TENSORMAP_NUM_BUCKETS, GRAPH_RECORD_TENSORMAP_POOL_SIZE, GRAPH_MAX_NODES); + return recording.tensor_map.init(CHIP_TENSORMAP_NUM_BUCKETS, GRAPH_RECORD_TENSORMAP_POOL_SIZE, MAX_IN_GRAPH_TASKS); } // The recorder thread's own storage for the body it is recording, and the reason none @@ -726,7 +726,7 @@ void unbind_recorder_boundary() { } // Stand this thread's retained storage up at the cap, so no body it records grows any of -// it and no recorded node allocates. +// it and no recorded task allocates. // // Capacity kept across recordings is otherwise the high-water mark of the bodies this // thread happened to record, and which body a thread gets is decided by one FIFO the @@ -735,20 +735,21 @@ void unbind_recorder_boundary() { // binds: a thread that recorded a narrow body first extends its slots and reallocates // every array the first time a wider one lands on it, on whatever bind that happens to // be. Measured on dsv4, whose eight Definitions differ in size, a warm bind still created -// 1336 node slots of 1679. Standing everything up at the cap makes a thread's storage -// independent of the order it saw bodies in. +// 1336 of the 1679 `tasks` slots that body needed. Standing everything up at the cap makes a +// thread's storage independent of the order it saw bodies in. // -// Each bound is a per-node cap times the node cap, so these are the recorded body's own -// limits rather than a worst case invented here: the tensor pool and tensor_sources are -// one entry per tensor argument (CORE_MAX_TENSOR_ARGS), the two scalar arrays one per -// scalar argument (CORE_MAX_SCALAR_ARGS), and predicates and output_ranges at most one per -// node. +// Each bound is a per-task cap times the in-graph task cap, so these are the recorded +// body's own limits rather than a worst case invented here: the tensor pool and +// tensor_sources are one entry per tensor argument (CORE_MAX_TENSOR_ARGS), the two scalar +// arrays one per scalar argument (CORE_MAX_SCALAR_ARGS), and predicates and output_ranges at +// most one per task. // // internal_fanins is the one array left growing, and the reason is the size it grows to -// rather than the bound it could reach. It has no per-node cap: CHIP_MAX_FANIN bounds a -// ring task's inline fanin, but a Graph node's producers travel in the Definition's own -// CSR, which the scheduler reads directly, so the only limits are uint16 producer indices -// and each producer being an earlier node — a structural 1024 x 1023 / 2 edges, 4.2 MB. +// rather than the bound it could reach. It has no per-in-graph-task cap: CHIP_MAX_FANIN +// bounds a global task's inline fanin, but an in-graph task's producers travel in the +// Definition's own CSR, which the scheduler reads directly, so the only limits are uint16 +// producer indices and each producer being an earlier task of the same body — a structural +// 1024 x 1023 / 2 edges, 4.2 MB. // What decides whether growth costs anything is not that bound but whether a reallocation // crosses glibc's mmap threshold, since a freed block below it is reused off the heap // without re-faulting (see the entry cited above). A dsv4 body holds ~630 edges, 5 KB, two @@ -764,20 +765,20 @@ void unbind_recorder_boundary() { // Returns false when the pool cannot be allocated, which the caller treats like a hazard // map it could not stand up. bool graph_recording_reserve_storage(GraphRecording &recording) { - constexpr size_t kNodeCap = GRAPH_MAX_NODES; - recording.node_tensor_pool.reset(new (std::nothrow) simpler::hbg::Tensor[GRAPH_RECORD_NODE_TENSOR_POOL_ELEMS]); - if (recording.node_tensor_pool == nullptr) return false; - recording.nodes.resize(kNodeCap); - recording.tensor_sources.reserve(kNodeCap * static_cast(CORE_MAX_TENSOR_ARGS)); - recording.scalars.reserve(kNodeCap * static_cast(CORE_MAX_SCALAR_ARGS)); - recording.scalar_sources.reserve(kNodeCap * static_cast(CORE_MAX_SCALAR_ARGS)); - recording.output_ranges.reserve(kNodeCap); - recording.predicates.reserve(kNodeCap); + constexpr size_t kInGraphTaskCap = MAX_IN_GRAPH_TASKS; + recording.task_tensor_pool.reset(new (std::nothrow) simpler::hbg::Tensor[GRAPH_RECORD_TENSOR_POOL_ELEMS]); + if (recording.task_tensor_pool == nullptr) return false; + recording.tasks.resize(kInGraphTaskCap); + recording.tensor_sources.reserve(kInGraphTaskCap * static_cast(CORE_MAX_TENSOR_ARGS)); + recording.scalars.reserve(kInGraphTaskCap * static_cast(CORE_MAX_SCALAR_ARGS)); + recording.scalar_sources.reserve(kInGraphTaskCap * static_cast(CORE_MAX_SCALAR_ARGS)); + recording.output_ranges.reserve(kInGraphTaskCap); + recording.predicates.reserve(kInGraphTaskCap); return true; } // Bind this thread's storage to one in-flight entry and empty it. Returns false when the -// hazard map or the node tensor pool cannot be stood up, which is only reachable on the +// hazard map or the tensor pool cannot be stood up, which is only reachable on the // thread's first recording. // Stand this thread's storage up once, or report that it could not be. Idempotent. // @@ -792,7 +793,7 @@ bool graph_recording_stand_up(GraphRecording &recording) { return false; } } catch (const std::bad_alloc &) { - // The node tensor pool is a nothrow new, but the flat arrays are vectors whose + // The tensor pool is a nothrow new, but the flat arrays are vectors whose // resize/reserve throw. This also runs on a recorder worker as it starts, where an // escaping exception terminates the process instead of letting the pool's prewarm // report the failure. @@ -804,14 +805,14 @@ bool graph_recording_stand_up(GraphRecording &recording) { } // Bind this thread's storage to one in-flight entry and empty it. Returns false when the -// hazard map or the node tensor pool cannot be stood up. +// hazard map or the tensor pool cannot be stood up. bool graph_recording_reset(GraphRecording &recording, const GraphInflightRecording &entry) { - // A body over GRAPH_MAX_NODES is abandoned, but it still grew every array to its real + // A body over MAX_IN_GRAPH_TASKS is abandoned, but it still grew every array to its real // size while it ran. Handing that to the next recording would retain storage for a // Definition that can never be published, unbounded, for the process's life -- so an // over-cap recording gives its storage back instead of passing it on. This is what - // makes the bound documented on GraphRecording::node_count true rather than nominal. - if (recording.nodes.size() > GRAPH_MAX_NODES) { + // makes the bound documented on GraphRecording::task_count true rather than nominal. + if (recording.tasks.size() > MAX_IN_GRAPH_TASKS) { recording = GraphRecording{}; } if (!graph_recording_stand_up(recording)) { @@ -826,9 +827,9 @@ bool graph_recording_reset(GraphRecording &recording, const GraphInflightRecordi recording.manual_begin_depth = CHIP_MAX_SCOPE_DEPTH; // clear() keeps each array's capacity, and the stand-up above reserved every one of // them to what a body at the cap needs, so no body a thread records can grow one. - // nodes is deliberately not cleared: see GraphRecording::node_count. - recording.node_count = 0; - recording.node_tensor_cursor = 0; + // tasks is deliberately not cleared: see GraphRecording::task_count. + recording.task_count = 0; + recording.task_tensor_cursor = 0; recording.tensor_sources.clear(); recording.scalars.clear(); recording.scalar_sources.clear(); @@ -839,12 +840,12 @@ bool graph_recording_reset(GraphRecording &recording, const GraphInflightRecordi } bool graph_classify_tensor( - const GraphRecording &recording, const GraphRecordedNode ¤t, int32_t task_index, + const GraphRecording &recording, const RecordedInGraphTask ¤t, int32_t task_index, const simpler::hbg::Tensor &tensor, GraphRecordedTensorSourceRef *source ) { if (graph_tensor_from_boundary(recording, tensor, source)) return true; const uintptr_t tensor_addr = static_cast(tensor.buffer.addr); - // The node being recorded is not in output_ranges yet — its entry is appended + // The task being recorded is not in output_ranges yet — its entry is appended // once its own tensors are classified — so its window is tested here, and a // hit is OWN_OUTPUT rather than a dependency. if (current.record_packed_base != 0 && current.total_output_size != 0 && @@ -869,7 +870,7 @@ bool graph_classify_tensor( const GraphRecordedOutputRange &range = *(after - 1); if (tensor_addr >= range.end) return false; source->source = GraphRecordedTensorSource::INTERNAL; - source->source_index = range.node_index; + source->source_index = range.task_index; source->packed_offset = tensor_addr - range.begin; return true; } @@ -948,9 +949,9 @@ T *graph_image_section(std::byte *image, uint32_t offset) { // Counts, section offsets and total_bytes for the image this recording produces, // settled without writing any of it so the destination can be claimed at the // exact size. required_heap comes from the fill, which is the pass that walks the -// nodes in order. +// tasks in order. std::optional graph_layout_definition(const GraphRecording &recording) { - if (recording.unsupported || recording.node_count == 0 || recording.node_count > GRAPH_MAX_NODES || + if (recording.unsupported || recording.task_count == 0 || recording.task_count > MAX_IN_GRAPH_TASKS || recording.boundary_tensors().empty() || recording.boundary_tensors().size() > UINT16_MAX || recording.boundary_tensors().size() != recording.boundary_types().size() || recording.boundary_args() == nullptr) { @@ -962,10 +963,10 @@ std::optional graph_layout_definition(const GraphRecording &rec size_t total_fanins = 0; size_t root_count = 0; size_t predicate_count = 0; - // node_count, not nodes.size(): the array keeps the slots a longer body left behind, + // task_count, not tasks.size(): the array keeps the slots a longer body left behind, // and those are not part of this recording. - for (size_t node = 0; node < recording.node_count; ++node) { - const GraphRecordedNode &source = recording.nodes[node]; + for (size_t i = 0; i < recording.task_count; ++i) { + const RecordedInGraphTask &source = recording.tasks[i]; if (source.tensor_count > UINT32_MAX - total_tensors || source.scalar_count > UINT32_MAX - total_scalars || source.fanin_count > UINT32_MAX - total_fanins || source.tensor_source_offset > recording.tensor_sources.size() || @@ -988,7 +989,7 @@ std::optional graph_layout_definition(const GraphRecording &rec GraphDefinition definition{}; definition.full_key = recording.full_key; - definition.task_count = static_cast(recording.node_count); + definition.task_count = static_cast(recording.task_count); definition.edge_count = static_cast(total_fanins); definition.root_count = static_cast(root_count); definition.boundary_count = static_cast(recording.boundary_tensors().size()); @@ -1007,13 +1008,15 @@ std::optional graph_layout_definition(const GraphRecording &rec definition.execution_storage_bytes = static_cast(execution_storage_bytes); size_t image_bytes = sizeof(GraphDefinition); - if (!graph_layout_section(recording.node_count + 1, &image_bytes, &definition.off_fanout_offsets) || + if (!graph_layout_section(recording.task_count + 1, &image_bytes, &definition.off_fanout_offsets) || !graph_layout_section(total_fanins, &image_bytes, &definition.off_fanout_indices) || - !graph_layout_section(recording.node_count + 1, &image_bytes, &definition.off_fanin_offsets) || + !graph_layout_section(recording.task_count + 1, &image_bytes, &definition.off_fanin_offsets) || !graph_layout_section(total_fanins, &image_bytes, &definition.off_fanin_indices) || !graph_layout_section(root_count, &image_bytes, &definition.off_root_indices) || - !graph_layout_section(recording.node_count, &image_bytes, &definition.off_node_offsets) || - !graph_layout_section(recording.node_count, &image_bytes, &definition.off_nodes) || + !graph_layout_section(recording.task_count, &image_bytes, &definition.off_in_graph_task_offsets) || + !graph_layout_section( + recording.task_count, &image_bytes, &definition.off_in_graph_tasks + ) || !graph_layout_section(total_tensors, &image_bytes, &definition.off_tensors) || !graph_layout_section(total_tensors, &image_bytes, &definition.off_tensor_sources) || !graph_layout_section(total_scalars, &image_bytes, &definition.off_scalars) || @@ -1052,8 +1055,8 @@ bool graph_fill_definition(const GraphRecording &recording, GraphDefinition defi auto *fanin_offsets = graph_image_section(image, definition.off_fanin_offsets); auto *fanin_indices = graph_image_section(image, definition.off_fanin_indices); auto *roots = graph_image_section(image, definition.off_root_indices); - auto *node_offsets = graph_image_section(image, definition.off_node_offsets); - auto *nodes = graph_image_section(image, definition.off_nodes); + auto *in_graph_task_offsets = graph_image_section(image, definition.off_in_graph_task_offsets); + auto *tasks = graph_image_section(image, definition.off_in_graph_tasks); auto *tensors = graph_image_section(image, definition.off_tensors); auto *tensor_sources = graph_image_section(image, definition.off_tensor_sources); auto *scalars = graph_image_section(image, definition.off_scalars); @@ -1069,16 +1072,16 @@ bool graph_fill_definition(const GraphRecording &recording, GraphDefinition defi // A producer's fanout count is accumulated across the consumer walk below and then // prefix-summed in place, so every entry has to start at zero — including [0], // which nothing else writes and which the device checks is zero. - std::fill_n(fanout_offsets, recording.node_count + 1, 0U); + std::fill_n(fanout_offsets, recording.task_count + 1, 0U); fanin_offsets[0] = 0; - for (size_t i = 0; i < recording.node_count; ++i) { - const GraphRecordedNode &source = recording.nodes[i]; + for (size_t i = 0; i < recording.task_count; ++i) { + const RecordedInGraphTask &source = recording.tasks[i]; if (source.total_output_size > static_cast(INT32_MAX) || source.tensor_count > static_cast(INT32_MAX) || source.scalar_count > static_cast(INT32_MAX) || source.fanin_count > UINT16_MAX) { return false; } - node_offsets[i] = required_heap; + in_graph_task_offsets[i] = required_heap; const uint64_t output_bytes = CHIP_ALIGN_UP(source.total_output_size, CHIP_ALIGN_SIZE); if (required_heap > UINT64_MAX - output_bytes) return false; required_heap += output_bytes; @@ -1092,19 +1095,19 @@ bool graph_fill_definition(const GraphRecording &recording, GraphDefinition defi } fanin_offsets[i + 1] = static_cast(fanin_cursor); - GraphNodeDefinition &node = nodes[i]; - std::copy(source.kernel_ids.begin(), source.kernel_ids.end(), std::begin(node.kernel_id)); - node.active_mask = source.active_mask.raw(); - node.task_attrs = source.task_attrs.raw(); - node.logical_block_num = source.logical_block_num; - node.total_required_subtasks = source.total_required_subtasks; - node.tensor_count = static_cast(source.tensor_count); - node.scalar_count = static_cast(source.scalar_count); - node.total_output_size = static_cast(source.total_output_size); - node.tensor_offset = static_cast(tensor_cursor); - node.scalar_offset = static_cast(scalar_cursor); - node.dump_metadata = source.dump_metadata; - node.predicate_slot = 0; + InGraphTaskDefinition &task = tasks[i]; + std::copy(source.kernel_ids.begin(), source.kernel_ids.end(), std::begin(task.kernel_id)); + task.active_mask = source.active_mask.raw(); + task.task_attrs = source.task_attrs.raw(); + task.logical_block_num = source.logical_block_num; + task.total_required_subtasks = source.total_required_subtasks; + task.tensor_count = static_cast(source.tensor_count); + task.scalar_count = static_cast(source.scalar_count); + task.total_output_size = static_cast(source.total_output_size); + task.tensor_offset = static_cast(tensor_cursor); + task.scalar_offset = static_cast(scalar_cursor); + task.dump_metadata = source.dump_metadata; + task.predicate_slot = 0; if (source.predicate_index >= 0) { if (static_cast(source.predicate_index) >= recording.predicates.size()) return false; const GraphRecordedPredicate &recorded = recording.predicates[source.predicate_index]; @@ -1118,9 +1121,9 @@ bool graph_fill_definition(const GraphRecording &recording, GraphDefinition defi packed.elem_size = recorded.elem_size; packed.op = static_cast(recorded.op); predicates[predicate_cursor] = packed; - node.predicate_slot = static_cast(++predicate_cursor); + task.predicate_slot = static_cast(++predicate_cursor); } - const simpler::hbg::Tensor *source_tensors = recording.node_tensors(source); + const simpler::hbg::Tensor *source_tensors = recording.task_tensors(source); for (size_t t = 0; t < source.tensor_count; ++t) { if (source_tensors[t].ndims > MAX_TENSOR_DIMS) return false; tensors[tensor_cursor] = graph_tensor_pack(source_tensors[t]); @@ -1149,10 +1152,10 @@ bool graph_fill_definition(const GraphRecording &recording, GraphDefinition defi return false; } definition.required_heap = required_heap; - for (size_t i = 0; i < recording.node_count; ++i) + for (size_t i = 0; i < recording.task_count; ++i) fanout_offsets[i + 1] += fanout_offsets[i]; - std::vector cursors(fanout_offsets, fanout_offsets + recording.node_count); - for (size_t consumer = 0; consumer < recording.node_count; ++consumer) { + std::vector cursors(fanout_offsets, fanout_offsets + recording.task_count); + for (size_t consumer = 0; consumer < recording.task_count; ++consumer) { for (uint32_t f = fanin_offsets[consumer]; f < fanin_offsets[consumer + 1]; ++f) { const size_t producer = fanin_indices[f]; fanout_indices[cursors[producer]++] = static_cast(consumer); @@ -1448,7 +1451,7 @@ static bool prepare_task( out->slot_state->logical_block_num = block_num; out->slot_state->active_mask = active_mask; out->slot_state->task_attrs = task_attrs; - out->slot_state->task_kind = active_mask ? TaskKind::KERNEL : TaskKind::DUMMY; + out->slot_state->task_kind = active_mask.is_dummy() ? TaskKind::DUMMY : TaskKind::KERNEL; // Reclaim gate: seed last_consumer to self, so a producer with no consumers // is retirable once completed_watermark >= its own id. Each fanin edge bumps // it in append_fanin_or_fail. completion_flags for this slot were cleared @@ -1472,8 +1475,8 @@ void OrchestratorState::begin_scope(ScopeMode mode) { // A Graph replays as a flat DAG with no scope structure: scope boundaries only // shape scheduling on the ordinary path, and the shadow-record path submits no // ordinary tasks. So a scope inside a Graph body must not touch the real scope - // stack. Its manual/auto mode still matters, though — the recorder infers a - // node's producers with the same compute_task_fanin the ordinary path uses, and + // stack. Its manual/auto mode still matters, though — the recorder infers a recorded + // task's producers with the same compute_task_fanin the ordinary path uses, and // that inference is suppressed inside a manual scope — so the depth is tracked on // the recording instead. if (GraphRecording *recording = active_graph_recording(orch); recording != nullptr) { @@ -2074,17 +2077,16 @@ bool graph_finalize_pending_submissions(OrchestratorState *orch, GraphHostState return true; } -// Record one internal Graph node while recording, without consuming a -// task-table slot. Builds the node's metadata and materialized outputs -// exactly as submit_task_common would, but assigns output buffers from the -// bit-63 virtual address range and derives internal fanins from tensor-source -// classification — so no task slot, tensormap entry, fanin-pool entry, or upload -// is produced for the node. The resulting Definition is later attached to the -// outer GRAPH shells already submitted by the main thread. The returned -// TaskOutputTensors borrow the node's own tensor storage; moving the node into -// recording.nodes keeps those addresses valid because the inner buffer is -// transferred, not copied. -TaskOutputTensors graph_record_submit_node( +// Record one in-graph task while recording, without consuming a task-table +// slot. Builds the task's metadata and materialized outputs exactly as +// submit_task_common would, but assigns output buffers from the bit-63 virtual +// address range and derives internal fanins from tensor-source classification — so +// no task-table slot, tensormap entry, fanin-pool entry, or upload is produced for +// it. The resulting Definition is later attached to the outer GRAPH shells already +// submitted by the main thread. The returned TaskOutputTensors point into the +// recording's tensor pool, which is allocated at the cap and never grows, so they +// stay valid for the rest of the recording. +TaskOutputTensors graph_record_submit_in_graph_task( OrchestratorState *orch, const CoreTaskArgs &args, ActiveMask active_mask, TaskAttrs task_attrs, int32_t aic_kernel_id, int32_t aiv0_kernel_id, int32_t aiv1_kernel_id ) { @@ -2092,16 +2094,16 @@ TaskOutputTensors graph_record_submit_node( TaskOutputTensors result; GraphRecording &recording = *active_graph_recording(orch); - const size_t node_index = recording.node_count; + const size_t task_index = recording.task_count; // A recorded task lives in the IN_GRAPH id space, so an id the body hands // around says which of the two kinds of thing it names without any arithmetic: // an IN_GRAPH id is a task of this body, indexed by its low field; a GLOBAL id is // a task submitted before the Graph, which nothing in the body may depend on. const TaskId task_id = - simpler::hbg::make_in_graph_task(GRAPH_RECORD_NO_OWNING_GRAPH, static_cast(node_index)); + simpler::hbg::make_in_graph_task(GRAPH_RECORD_NO_OWNING_GRAPH, static_cast(task_index)); result.set_task_id(task_id); - if (node_index >= GRAPH_MAX_NODES || args.has_error) { + if (task_index >= MAX_IN_GRAPH_TASKS || args.has_error) { recording.unsupported = true; } @@ -2116,61 +2118,59 @@ TaskOutputTensors graph_record_submit_node( const uintptr_t packed_base_addr = GRAPH_RECORD_VIRTUAL_BASE + recording.next_virtual_offset; recording.next_virtual_offset += aligned_output; - // The node is filled in place, in the slot it will keep. Reusing the slot is what - // keeps its `tensors` buffer across recordings; reset() puts the rest of the slot - // back to a fresh node's state. Growing `nodes` moves the slots, but a slot's - // borrowed output addresses live in that heap buffer and a move only transfers its - // pointer, so they stay valid -- the same reason the old push_back(std::move(node)) - // was sound. - if (node_index >= recording.nodes.size()) recording.nodes.emplace_back(); - GraphRecordedNode &node = recording.nodes[node_index]; - node.reset(); - node.kernel_ids[static_cast(SubtaskSlot::AIC)] = aic_kernel_id; - node.kernel_ids[static_cast(SubtaskSlot::AIV0)] = aiv0_kernel_id; - node.kernel_ids[static_cast(SubtaskSlot::AIV1)] = aiv1_kernel_id; - node.active_mask = active_mask; - node.task_attrs = task_attrs; - node.task_attrs.set_early_resolve(false); - node.logical_block_num = args.launch_spec.block_num(); + // The task is filled in place, in the slot it will keep, and reset() puts the rest of + // the slot back to a freshly recorded task's state. An over-cap body grows `tasks`, + // which moves the slots, but the addresses handed to the caller live in the recording's + // tensor pool rather than in a slot, so a move cannot invalidate them. + if (task_index >= recording.tasks.size()) recording.tasks.emplace_back(); + RecordedInGraphTask &task = recording.tasks[task_index]; + task.reset(); + task.kernel_ids[static_cast(SubtaskSlot::AIC)] = aic_kernel_id; + task.kernel_ids[static_cast(SubtaskSlot::AIV0)] = aiv0_kernel_id; + task.kernel_ids[static_cast(SubtaskSlot::AIV1)] = aiv1_kernel_id; + task.active_mask = active_mask; + task.task_attrs = task_attrs; + task.task_attrs.set_early_resolve(false); + task.logical_block_num = args.launch_spec.block_num(); // Mirror prepare_task's contract: block_num must be positive and the subtask // count must fit int16_t. An out-of-contract value marks asynchronous // recording unsupported and makes commit fail-fast, rather than baking a // truncated or negative count into the cached Definition (which the device - // would expand into a node that never completes). + // would expand into an in-graph task that never completes). const int32_t required_subtasks = - static_cast(node.logical_block_num) * __builtin_popcount(active_mask.core_mask()); - if (node.logical_block_num <= 0 || required_subtasks > std::numeric_limits::max()) { + static_cast(task.logical_block_num) * __builtin_popcount(active_mask.core_mask()); + if (task.logical_block_num <= 0 || required_subtasks > std::numeric_limits::max()) { recording.unsupported = true; - node.total_required_subtasks = 0; + task.total_required_subtasks = 0; } else { - node.total_required_subtasks = static_cast(required_subtasks); + task.total_required_subtasks = static_cast(required_subtasks); } - node.record_packed_base = packed_base_addr; - node.total_output_size = aligned_output; + task.record_packed_base = packed_base_addr; + task.total_output_size = aligned_output; // Build the tensor list exactly as TaskPayload::init: inputs/inouts copy // the caller's simpler::hbg::Tensor; outputs materialize from the create-info onto the - // scratch buffer and carry this node's owner id. + // scratch buffer and carry the recorded task's owner id. const int32_t tensor_count = args.tensor_count(); - // Claim this node's slice of the pool. The cursor is a pure bump, so slices abut and a + // Claim this task's slice of the pool. The cursor is a pure bump, so slices abut and a // body holds its tensors in the bytes they need; nothing is ever returned to it, since // the whole pool is reset by the next recording. - if (static_cast(recording.node_tensor_cursor) + static_cast(tensor_count) > - GRAPH_RECORD_NODE_TENSOR_POOL_ELEMS) { + if (static_cast(recording.task_tensor_cursor) + static_cast(tensor_count) > + GRAPH_RECORD_TENSOR_POOL_ELEMS) { recording.unsupported = true; return result; } - node.tensor_offset = recording.node_tensor_cursor; - node.tensor_count = static_cast(tensor_count); - recording.node_tensor_cursor += static_cast(tensor_count); - simpler::hbg::Tensor *node_tensors = recording.node_tensors(node); + task.tensor_offset = recording.task_tensor_cursor; + task.tensor_count = static_cast(tensor_count); + recording.task_tensor_cursor += static_cast(tensor_count); + simpler::hbg::Tensor *task_tensors = recording.task_tensors(task); // Value-initialized before the fill, not merely claimed: the slice holds whatever the // previous body left in it, and simpler::hbg::Tensor::init_from writes strides only up // to the new tensor's ndims, so a narrower tensor would inherit a wider one's trailing // strides. - std::fill_n(node_tensors, static_cast(tensor_count), simpler::hbg::Tensor{}); + std::fill_n(task_tensors, static_cast(tensor_count), simpler::hbg::Tensor{}); for (int32_t i = 0; i < tensor_count; ++i) { - simpler::hbg::Tensor &slot_tensor = node_tensors[static_cast(i)]; + simpler::hbg::Tensor &slot_tensor = task_tensors[static_cast(i)]; if (args.tag(i) != TensorArgType::OUTPUT) { slot_tensor.copy(args.tensor(i).ref()); } else { @@ -2182,39 +2182,39 @@ TaskOutputTensors graph_record_submit_node( } } // The addresses handed out here are into the pool, which never moves, so they stay - // valid for every later node of this recording. + // valid for the rest of this recording. for (int32_t i = 0; i < tensor_count; ++i) { - if (args.tag(i) == TensorArgType::OUTPUT) result.materialize_output(node_tensors[static_cast(i)]); + if (args.tag(i) == TensorArgType::OUTPUT) result.materialize_output(task_tensors[static_cast(i)]); } - node.scalar_offset = static_cast(recording.scalars.size()); - node.scalar_count = static_cast(args.scalar_count()); + task.scalar_offset = static_cast(recording.scalars.size()); + task.scalar_count = static_cast(args.scalar_count()); recording.scalars.insert(recording.scalars.end(), args.scalars(), args.scalars() + args.scalar_count()); #if SIMPLER_DFX - node.dump_metadata.dump_arg_mask = args.dump_arg_mask(); - node.dump_metadata.dump_arg_flags = args.dump_arg_index_ambiguous_mask(); - memcpy(node.dump_metadata.scalar_dtypes, args.scalar_dtypes(), args.scalar_count() * sizeof(uint8_t)); + task.dump_metadata.dump_arg_mask = args.dump_arg_mask(); + task.dump_metadata.dump_arg_flags = args.dump_arg_index_ambiguous_mask(); + memcpy(task.dump_metadata.scalar_dtypes, args.scalar_dtypes(), args.scalar_count() * sizeof(uint8_t)); #endif // Classify each scalar's source: a plain literal is static Definition data, // while a value copied from a boundary scalar is refreshed on replay. A // mutable tracked boundary scalar is not supported and falls back. - recording.scalar_sources.resize(static_cast(node.scalar_offset) + node.scalar_count); + recording.scalar_sources.resize(static_cast(task.scalar_offset) + task.scalar_count); for (int32_t i = 0; i < args.scalar_count(); ++i) { GraphRecordedScalarSourceRef source = graph_classify_scalar(recording, args, i); if (source.source == GraphRecordedScalarSource::INVALIDATED_BOUNDARY) recording.unsupported = true; - recording.scalar_sources[static_cast(node.scalar_offset) + static_cast(i)] = source; + recording.scalar_sources[static_cast(task.scalar_offset) + static_cast(i)] = source; } // Classify each tensor's source, then derive internal fanins from the // INTERNAL classifications plus any explicit internal dependency. - node.tensor_source_offset = static_cast(recording.tensor_sources.size()); - recording.tensor_sources.resize(static_cast(node.tensor_source_offset) + tensor_count); + task.tensor_source_offset = static_cast(recording.tensor_sources.size()); + recording.tensor_sources.resize(static_cast(task.tensor_source_offset) + tensor_count); for (int32_t i = 0; i < tensor_count; ++i) { // The out-pointer is used only for the duration of the call, so pointing - // it into the flat array is safe even though a later node grows that array. + // it into the flat array is safe even though a later task of this body grows it. if (!graph_classify_tensor( - recording, node, static_cast(node_index), node_tensors[static_cast(i)], - &recording.tensor_sources[static_cast(node.tensor_source_offset) + static_cast(i)] + recording, task, static_cast(task_index), task_tensors[static_cast(i)], + &recording.tensor_sources[static_cast(task.tensor_source_offset) + static_cast(i)] )) { recording.unsupported = true; } @@ -2226,18 +2226,18 @@ TaskOutputTensors graph_record_submit_node( // predicate creates no dependency here any more than it does on the ordinary // path: the caller declares one, and the explicit-dep loop below records it. // - // Gated on the recorded attribute, not on args: a kernel-less node never + // Gated on the recorded attribute, not on args: a kernel-less task never // dispatches, so submit_dummy_task and alloc_tensors drop the predicate the - // caller set. Reading args here instead would record a predicate the node's + // caller set. Reading args here instead would record a predicate the task's // own attribute denies, and materialize rejects a Definition whose two halves // disagree. - if (node.task_attrs.has_predicate()) { + if (task.task_attrs.has_predicate()) { const CoreTaskPredicate &pred = args.predicate(); GraphRecordedPredicate recorded; recorded.op = pred.op; recorded.target = pred.target; const simpler::hbg::Tensor *operand = pred.operand.tensor; - // OWN_OUTPUT would read the node's own output before the node runs, so it + // OWN_OUTPUT would read the task's own output before the task runs, so it // names no value the predicate could be evaluating. An index vector that // leaves the operand's extent is caught here too: materialize would // otherwise reject the baked offset on the device, where the failure is a @@ -2246,7 +2246,7 @@ TaskOutputTensors graph_record_submit_node( operand == nullptr ? 0 : operand->compute_flat_offset(pred.operand.indices, pred.operand.ndims); if (operand == nullptr || operand->ndims > MAX_TENSOR_DIMS || pred.operand.ndims > operand->ndims || flat_offset < operand->start_offset || flat_offset - operand->start_offset >= operand->extent_elem_cache || - !graph_classify_tensor(recording, node, static_cast(node_index), *operand, &recorded.source) || + !graph_classify_tensor(recording, task, static_cast(task_index), *operand, &recorded.source) || recorded.source.source == GraphRecordedTensorSource::OWN_OUTPUT) { recording.unsupported = true; } else { @@ -2254,31 +2254,31 @@ TaskOutputTensors graph_record_submit_node( recorded.elem_offset = flat_offset - operand->start_offset; recorded.elem_size = static_cast(get_element_size(operand->dtype)); } - node.predicate_index = static_cast(recording.predicates.size()); + task.predicate_index = static_cast(recording.predicates.size()); recording.predicates.push_back(recorded); } - node.fanin_offset = static_cast(recording.internal_fanins.size()); - // Dedup within this node's own range: the flat array's earlier entries belong - // to earlier nodes. - auto add_fanin = [&recording, &node](size_t producer) { - const auto begin = recording.internal_fanins.begin() + node.fanin_offset; + task.fanin_offset = static_cast(recording.internal_fanins.size()); + // Dedup within this task's own range: the flat array's earlier entries belong + // to the body's earlier tasks. + auto add_fanin = [&recording, &task](size_t producer) { + const auto begin = recording.internal_fanins.begin() + task.fanin_offset; if (std::find(begin, recording.internal_fanins.end(), producer) == recording.internal_fanins.end()) { recording.internal_fanins.push_back(producer); } }; for (uint32_t i = 0; i < static_cast(tensor_count); ++i) { - const GraphRecordedTensorSourceRef &source = recording.tensor_sources[node.tensor_source_offset + i]; + const GraphRecordedTensorSourceRef &source = recording.tensor_sources[task.tensor_source_offset + i]; if (source.source == GraphRecordedTensorSource::INTERNAL) add_fanin(source.source_index); } // Inferred hazards, on the same terms as the ordinary path. The loop above only - // names the node that ALLOCATED each buffer; every write-then-read through a + // names the in-graph task that ALLOCATED each buffer; every write-then-read through a // buffer someone else allocated — an alloc_tensors output written in place // with add_inout, or a view of a boundary tensor — needs the last-writer // lookup compute_task_fanin performs. Running the very same function against // the recording's own map is what keeps a Definition's edge set equal to the - // one the body gets when its tasks are submitted individually. + // one the body gets when the ordinary path submits its tasks one at a time. // // Producers outside the recording window are dropped: they are reached // through boundary tensors, and the outer Graph shell already carries those @@ -2292,29 +2292,29 @@ TaskOutputTensors graph_record_submit_node( args.explicit_deps_data(), }; const bool manual_scope = recording.in_manual_scope(); - if (!recording.storage_ready || node_index >= GRAPH_MAX_NODES) { + if (!recording.storage_ready || task_index >= MAX_IN_GRAPH_TASKS) { // An over-cap body is already abandoned, and its task ids have run past // the index field make_in_graph_task packs them into, so registering one // would key the map outside its task chains. recording.unsupported = true; } else if (recording.tensor_map.free_entries() < count_registrable_outputs(dep_inputs, manual_scope)) { - // Recording one more node would assert inside new_entry(). Abandon the + // Recording one more task would assert inside new_entry(). Abandon the // Definition instead, so the run fails by name at graph_commit rather // than on a hard assert here. LOG_WARN( - "[GraphExecution] recording hazard map exhausted at node %zu (%d entries); Graph abandoned", node_index, - GRAPH_RECORD_TENSORMAP_POOL_SIZE + "[GraphExecution] recording hazard map exhausted at in-graph task %zu (%d entries); Graph abandoned", + task_index, GRAPH_RECORD_TENSORMAP_POOL_SIZE ); recording.unsupported = true; } else { - auto emit_inferred = [&add_fanin, node_index](TaskId producer) -> bool { + auto emit_inferred = [&add_fanin, task_index](TaskId producer) -> bool { // A GLOBAL producer is a task submitted before the Graph. The outer shell // was submitted through the ordinary path against this same boundary, so // its own fanin already orders the whole body behind that task and the // Definition carries no edge of its own. if (simpler::hbg::is_global_task(producer)) return true; const uint32_t producer_index = simpler::hbg::task_local_id(producer); - if (producer_index < static_cast(node_index)) { + if (producer_index < static_cast(task_index)) { add_fanin(static_cast(producer_index)); } return true; @@ -2343,7 +2343,7 @@ TaskOutputTensors graph_record_submit_node( continue; } const uint32_t dep_index = simpler::hbg::task_local_id(dep); - if (dep_index >= static_cast(node_index)) { + if (dep_index >= static_cast(task_index)) { // A task of this body that is not yet recorded: the Definition's edges are // acyclic by construction, so a forward reference cannot be expressed. recording.unsupported = true; @@ -2352,22 +2352,21 @@ TaskOutputTensors graph_record_submit_node( add_fanin(static_cast(dep_index)); } - node.fanin_count = static_cast(recording.internal_fanins.size() - node.fanin_offset); - if (node.record_packed_base != 0 && node.total_output_size != 0 && - node.total_output_size <= UINTPTR_MAX - node.record_packed_base) { - const uintptr_t begin = node.record_packed_base; - const uintptr_t end = begin + node.total_output_size; + task.fanin_count = static_cast(recording.internal_fanins.size() - task.fanin_offset); + if (task.record_packed_base != 0 && task.total_output_size != 0 && + task.total_output_size <= UINTPTR_MAX - task.record_packed_base) { + const uintptr_t begin = task.record_packed_base; + const uintptr_t end = begin + task.total_output_size; // The sorted-and-disjoint property the lookup depends on, checked rather // than assumed: a mid-recording heap rollback would break it, and today - // the only rollback is graph_end's, after every node is recorded. + // the only rollback is graph_end's, after every task of the body is recorded. always_assert(recording.output_ranges.empty() || recording.output_ranges.back().end <= begin); - recording.output_ranges.push_back({begin, end, static_cast(node_index)}); + recording.output_ranges.push_back({begin, end, static_cast(task_index)}); } - // Published last, as push_back(std::move(node)) used to be: until this advances, the - // slot is not part of the recording, so nothing that scans the recorded nodes can see - // the node being built. - recording.node_count = node_index + 1; - ORCH_PHASE_END(HostPhaseKind::OrchRecordNode, task_id.raw); + // Published last: until this advances, the slot is not part of the recording, so + // nothing that scans the recorded tasks can see the task being built. + recording.task_count = task_index + 1; + ORCH_PHASE_END(HostPhaseKind::OrchRecordInGraphTask, task_id.raw); return result; } @@ -2586,7 +2585,7 @@ bool OrchestratorState::graph_end() { } const bool built = layout.has_value() && graph_fill_definition(*recording, *layout, image); if (built) { - ORCH_PHASE_END(HostPhaseKind::OrchBuildDefinition, recording->node_count); + ORCH_PHASE_END(HostPhaseKind::OrchBuildDefinition, recording->task_count); } const GraphDefinition *header = built ? graph_record_definition(*state, record) : nullptr; if (header == nullptr) { @@ -2596,7 +2595,7 @@ bool OrchestratorState::graph_end() { return false; } LOG_DEBUG( - "[GraphExecution] define key=0x%llx nodes=%u bytes=%u", static_cast(header->full_key), + "[GraphExecution] define key=0x%llx tasks=%u bytes=%u", static_cast(header->full_key), header->task_count, header->total_bytes ); bool ready = false; @@ -2741,7 +2740,7 @@ TaskOutputTensors OrchestratorState::submit_task(const MixedKernels &mixed_kerne } if (active_graph_recording(orch) != nullptr) { - return graph_record_submit_node( + return graph_record_submit_in_graph_task( orch, args, active_mask, task_attrs, normalized.aic_kernel_id, normalized.aiv0_kernel_id, normalized.aiv1_kernel_id ); @@ -2784,7 +2783,7 @@ TaskOutputTensors OrchestratorState::submit_dummy_task(const CoreTaskArgs &args) task_attrs.set_timing_slot(args.task_timing_slot()); if (active_graph_recording(orch) != nullptr) { - return graph_record_submit_node( + return graph_record_submit_in_graph_task( orch, args, ActiveMask{}, task_attrs, INVALID_KERNEL_ID, INVALID_KERNEL_ID, INVALID_KERNEL_ID ); } @@ -2832,12 +2831,12 @@ TaskOutputTensors OrchestratorState::alloc_tensors(const CoreTaskArgs &args) { return TaskOutputTensors{}; } - // A Graph body may allocate. The allocation records as a kernel-less node — - // the same shape submit_dummy_task records — and replay reserves the - // intermediate heap for every internal node anyway, so the outputs land at + // A Graph body may allocate. The allocation records as a kernel-less in-graph + // task — the same shape submit_dummy_task records — and replay reserves the + // intermediate heap for every in-graph task anyway, so the outputs land at // addresses the replayed Definition derives for itself. if (active_graph_recording(orch) != nullptr) { - return graph_record_submit_node( + return graph_record_submit_in_graph_task( orch, args, ActiveMask{}, TaskAttrs{}, INVALID_KERNEL_ID, INVALID_KERNEL_ID, INVALID_KERNEL_ID ); } diff --git a/src/a2a3/runtime/host_build_graph/runtime/runtime_types.h b/src/a2a3/runtime/host_build_graph/runtime/runtime_types.h index a49e8a5a4c..eb7a6b5a9b 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/runtime_types.h +++ b/src/a2a3/runtime/host_build_graph/runtime/runtime_types.h @@ -109,7 +109,7 @@ // base before the image travels. Nothing dereferences an address in this window. inline constexpr uint64_t HEAP_VIRTUAL_BASE = 1ULL << 62; -// Base of the address range Graph recording hands to an internal node's packed +// Base of the address range Graph recording hands to an in-graph task's packed // outputs. Recorded addresses are never dereferenced: they exist so // graph_classify_tensor can tell an internal producer's output from a boundary // tensor by address-range containment alone, and the Definition stores them as @@ -206,11 +206,20 @@ struct TaskAllocResult { bool failed() const { return task_id < 0; } }; +/** + * What a task is, independent of where it belongs. + * + * KERNEL and DUMMY are leaves: KERNEL dispatches to cores, DUMMY carries only + * dependencies. GRAPH is a container — a shell that expands into its own body of + * tasks and completes when they all have. + * + * Membership is not a kind: a task inside a Graph body is an ordinary KERNEL or + * DUMMY, and `graph_context` names the Graph it belongs to. + */ enum class TaskKind : uint8_t { KERNEL = 0, DUMMY = 1, GRAPH = 2, - GRAPH_NODE = 3, }; struct OutputLayout { @@ -316,7 +325,7 @@ struct TaskPayload { // fanin holds flat position-independent producer local task ids. A producer is // named by its local id alone, so no per-edge indirection is stored. Scanned by // classify_fanin_state against the shared-memory completion_flags. Hard-capped at - // CHIP_MAX_FANIN (no dep-pool spill). Unbound on a Graph node, whose + // CHIP_MAX_FANIN (no dep-pool spill). Unbound on an in-graph task, whose // dependencies live in the Definition's fanin CSR instead. simpler::hbg::SelfRelativePtr tensors; simpler::hbg::SelfRelativePtr scalars; @@ -395,7 +404,7 @@ struct TaskPayload { * Point this payload's three argument regions at pool-resident storage. Must run * before prefetch() and init(), which dereference them. * - * A Graph node passes nullptr for fanin: its dependencies come from the + * An in-graph task passes nullptr for fanin: its dependencies come from the * Definition's CSR, so the region does not exist and fanin_count stays 0. */ void bind_regions(simpler::hbg::Tensor *tensor_region, uint64_t *scalar_region, int32_t *fanin_region) { @@ -601,9 +610,24 @@ struct alignas(64) ChipTaskSlotState { // Graph-only scheduling metadata occupies the former tail padding, keeping // the slot state at one cache line and preserving the 40-byte descriptor // ABI consumed by AICore. Readiness uses the shared intrusive wake-list - // fields above; this index identifies the node in the saved fanin CSR. + // fields above; this index identifies the task in the saved fanin CSR. // Ordinary tasks leave both Graph fields -1/null. - int32_t graph_node_index{-1}; + int32_t in_graph_task_index{-1}; + // Graph membership, and which of the two Graph structs this points at is + // decided by task_kind rather than by anything stored here: + // + // nullptr an ordinary task, in no Graph + // != nullptr, task_kind == GRAPH the outer Graph task, pointing at the + // shared GraphDefinition until localize + // swaps in its GraphExecution + // != nullptr, task_kind != GRAPH an in-graph task, pointing at the + // GraphExecution it belongs to + // + // So every reader must test task_kind before casting, and complete_task + // routes on exactly that pair: a null context or a GRAPH kind takes the + // ordinary global fanout, anything else is counted against its Graph. A + // localize that fails puts this back to nullptr (scheduler_cold_path.cpp), + // so the outer task cannot be mistaken for an in-graph one. void *graph_context{nullptr}; int32_t claim_block_range(int32_t block_limit, int32_t max_count, int32_t &start) { @@ -640,8 +664,8 @@ struct alignas(64) ChipTaskSlotState { /** * Reset dynamic scheduling fields to their pristine values. Called once per - * slot as the orchestrator claims it in prepare_task, and again as a Graph - * node's storage is materialized — whole-graph-resident hbg has no + * slot as the orchestrator claims it in prepare_task, and again as an + * in-graph task's storage is materialized — whole-graph-resident hbg has no * execution-time slot recycle. Skips payload/task (bound once) and * task_state (the orchestrator sets PENDING when it populates the slot). * wake_list_head starts nullptr (open for registration), NOT SENTINEL. @@ -652,7 +676,7 @@ struct alignas(64) ChipTaskSlotState { any_subtask_deferred.store(false, std::memory_order_relaxed); completed_subtasks.store(0, std::memory_order_relaxed); next_block_idx.store(0, std::memory_order_relaxed); - graph_node_index = -1; + in_graph_task_index = -1; graph_context = nullptr; task_kind = TaskKind::KERNEL; // Note: active_mask and task_attrs are per-submit-constant fields diff --git a/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler.h b/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler.h index 7c9853cec0..bd76d2b2d2 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler.h +++ b/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler.h @@ -902,12 +902,12 @@ struct SchedulerState { // Scheduler polling only chooses which already-wired producer a consumer // waits on at this instant; it never recomputes producer relationships. int32_t graph_first_unmet_producer(const GraphExecution &execution, const ChipTaskSlotState &consumer) const { - const uint32_t node_index = static_cast(consumer.graph_node_index); - const uint32_t begin = execution.fanin_offsets[node_index]; - const uint32_t end = execution.fanin_offsets[node_index + 1]; + const uint32_t task_index = static_cast(consumer.in_graph_task_index); + const uint32_t begin = execution.fanin_offsets[task_index]; + const uint32_t end = execution.fanin_offsets[task_index + 1]; for (uint32_t edge = begin; edge < end; ++edge) { const uint16_t producer_index = execution.fanin_indices[edge]; - const ChipTaskSlotState &producer = execution.node_at(producer_index).slot; + const ChipTaskSlotState &producer = execution.task_at(producer_index).slot; if (producer.task_state.load(std::memory_order_acquire) != CHIP_TASK_COMPLETED) { return static_cast(producer_index); } @@ -935,7 +935,7 @@ struct SchedulerState { push_ready_routed(consumer); return; } - producer = &execution.node_at(unmet_producer).slot; + producer = &execution.task_at(unmet_producer).slot; } } @@ -948,7 +948,7 @@ struct SchedulerState { if (unmet_producer < 0) { push_ready_routed(waiter); } else { - register_graph_wake(execution, &execution.node_at(unmet_producer).slot, waiter); + register_graph_wake(execution, &execution.task_at(unmet_producer).slot, waiter); } consumers_rescanned++; waiter = next; @@ -964,7 +964,7 @@ struct SchedulerState { // here — they reach the ready queue through their producers' wake list. int32_t graph_route_ready_roots(GraphExecution &execution) { if (execution.outer_slot == nullptr || !graph_execution_external_ready(execution)) return 0; - const int32_t published = execution.published_nodes.load(std::memory_order_acquire); + const int32_t published = execution.published_tasks.load(std::memory_order_acquire); int32_t routed = 0; while (true) { int32_t i = execution.route_cursor.load(std::memory_order_relaxed); @@ -975,33 +975,33 @@ struct SchedulerState { continue; } if (execution.fanin_offsets[i] == execution.fanin_offsets[i + 1]) { - push_ready_routed(&execution.node_at(i).slot); + push_ready_routed(&execution.task_at(i).slot); routed++; } } return routed; } - // Register each newly materialized node [first, last) on its first unmet + // Register each newly materialized in-graph task [first, last) on its first unmet // producer (or route it immediately when every producer already completed), // publish the range for routing, and route any roots the external gate now // admits. Runs single-owner per graph via the prepare-queue slot, so the // range never overlaps another thread's. register_graph_wake and // graph_first_unmet_producer are safe against a producer completing - // concurrently, which is what lets a node dispatch before the whole graph is + // concurrently, which is what lets a task dispatch before the whole graph is // materialized. void graph_incremental_publish(GraphExecution &execution, int32_t first, int32_t last) { for (int32_t i = first; i < last; ++i) { if (execution.fanin_offsets[i] == execution.fanin_offsets[i + 1]) continue; // root - ChipTaskSlotState &node = execution.node_at(i).slot; - const int32_t unmet = graph_first_unmet_producer(execution, node); + ChipTaskSlotState &task = execution.task_at(i).slot; + const int32_t unmet = graph_first_unmet_producer(execution, task); if (unmet < 0) { - push_ready_routed(&node); + push_ready_routed(&task); } else { - register_graph_wake(execution, &execution.node_at(unmet).slot, &node); + register_graph_wake(execution, &execution.task_at(unmet).slot, &task); } } - execution.published_nodes.store(last, std::memory_order_release); + execution.published_tasks.store(last, std::memory_order_release); graph_route_ready_roots(execution); } @@ -1013,16 +1013,16 @@ struct SchedulerState { } GraphMaterializeResult prepare_graph_task( - ChipTaskSlotState &outer_slot, int32_t max_nodes = GRAPH_MATERIALIZE_SLICE_NODES, - int32_t *nodes_materialized = nullptr + ChipTaskSlotState &outer_slot, int32_t max_tasks = GRAPH_MATERIALIZE_SLICE_TASKS, + int32_t *tasks_materialized = nullptr ) { GraphExecution *execution = graph_execution_from_outer_slot(outer_slot); if (execution == nullptr) return GraphMaterializeResult::INVALID; - const int32_t before = execution->materialized_nodes; + const int32_t before = execution->materialized_tasks; const GraphMaterializeResult result = - graph_execution_materialize_slice(outer_slot, *execution, max_nodes, nodes_materialized); + graph_execution_materialize_slice(outer_slot, *execution, max_tasks, tasks_materialized); if (result == GraphMaterializeResult::PENDING || result == GraphMaterializeResult::PREPARED) { - graph_incremental_publish(*execution, before, execution->materialized_nodes); + graph_incremental_publish(*execution, before, execution->materialized_tasks); } if (result == GraphMaterializeResult::PREPARED && graph_execution_external_ready(*execution)) { activate_prepared_graph(*execution); @@ -1051,7 +1051,11 @@ struct SchedulerState { #endif ) { TaskCompletionOutcome outcome; - if (slot_state.task_kind != TaskKind::GRAPH_NODE) { + // A task in a Graph body retires into its execution's counters; everything else + // — including the outer GRAPH shell — is a task of the run and releases its + // fanout. graph_context is null for the common case, so this short-circuits + // before the kind is read. + if (slot_state.graph_context == nullptr || slot_state.task_kind == TaskKind::GRAPH) { #if SIMPLER_SCHED_PROFILING CompletionStats stats = on_task_complete(slot_state, thread_idx); outcome.fanout_edges = static_cast(stats.fanout_edges); @@ -1062,27 +1066,29 @@ struct SchedulerState { return outcome; } - GraphExecution *execution = graph_execution_from_slot(slot_state); - if (execution == nullptr || execution->definition == nullptr || execution->nodes == nullptr) { + // Membership is established by the branch above: graph_context names this task's + // execution, and the shell case has already returned. + GraphExecution *execution = static_cast(slot_state.graph_context); + if (execution->definition == nullptr || execution->tasks == nullptr) { outcome.error_code = SIMPLER_ERROR_INVALID_ARGS; return outcome; } - // Incremental activation routes a node before the graph reaches ACTIVE, so a - // node can legitimately complete while the graph is still MATERIALIZING or - // PREPARED. Only SUBMITTED (execution not yet bound) and COMPLETED - // (execution already retired) are invalid states for a node completion. + // Incremental activation routes an in-graph task before the graph reaches + // ACTIVE, so one can legitimately complete while the graph is still + // MATERIALIZING or PREPARED. Only SUBMITTED (execution not yet bound) and + // COMPLETED (execution already retired) are invalid states for such a completion. const GraphExecutionState graph_state = graph_execution_state(*execution); if (graph_state < GraphExecutionState::MATERIALIZING || graph_state > GraphExecutionState::ACTIVE) { outcome.error_code = SIMPLER_ERROR_INVALID_ARGS; return outcome; } - const int32_t saved_node_index = slot_state.graph_node_index; - if (saved_node_index < 0) { + const int32_t saved_task_index = slot_state.in_graph_task_index; + if (saved_task_index < 0) { outcome.error_code = SIMPLER_ERROR_INVALID_ARGS; return outcome; } - const uint32_t node_index = static_cast(saved_node_index); - if (node_index >= static_cast(execution->node_count)) { + const uint32_t task_index = static_cast(saved_task_index); + if (task_index >= static_cast(execution->task_count)) { outcome.error_code = SIMPLER_ERROR_INVALID_ARGS; return outcome; } @@ -1093,11 +1099,11 @@ struct SchedulerState { slot_state.mark_completed(); outcome.fanout_edges = drain_graph_wake_list(*execution, slot_state); - const bool graph_completed = graph_execution_complete_node(*execution); - graph_execution_retire_node(*execution); + const bool graph_completed = graph_execution_complete_in_graph_task(*execution); + graph_execution_retire_in_graph_task(*execution); if (!graph_completed) return outcome; - // Internal nodes count as zero stream tasks. The final node publishes + // Internal tasks count as zero stream tasks. The final in-graph task publishes // the outer task exactly once, waking external consumers and // contributing the one task the host actually submitted. if (execution->outer_slot != nullptr) { diff --git a/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_dispatch.cpp b/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_dispatch.cpp index 22cf302219..99ba55679e 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_dispatch.cpp +++ b/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_dispatch.cpp @@ -1299,7 +1299,7 @@ int32_t SchedulerContext::resolve_and_dispatch(Runtime *runtime, int32_t thread_ // independently, then meet in GraphExecution::state. // // Keep this ahead of dummy/regular dispatch so a ready Graph can expose - // its root nodes without waiting for an otherwise unrelated dispatch + // its root tasks without waiting for an otherwise unrelated dispatch // pass. Limiting the work to one activation and one bounded prepare // slice per loop prevents a large definition from monopolizing a // scheduler thread. @@ -1328,9 +1328,9 @@ int32_t SchedulerContext::resolve_and_dispatch(Runtime *runtime, int32_t thread_ uint64_t graph_prepare_t0 = chip_swimlane_level_ >= ChipSwimlaneLevel::SCHED_PHASES ? get_sys_cnt_aicpu() : 0; #endif - int32_t nodes_materialized = 0; + int32_t tasks_materialized = 0; GraphMaterializeResult result = - sched_->prepare_graph_task(*prepare_slot, GRAPH_MATERIALIZE_SLICE_NODES, &nodes_materialized); + sched_->prepare_graph_task(*prepare_slot, GRAPH_MATERIALIZE_SLICE_TASKS, &tasks_materialized); if (result == GraphMaterializeResult::PENDING || result == GraphMaterializeResult::BUSY) { if (!sched_->push_graph_prepare(prepare_slot, prepare_task_id, thread_idx)) { fail_scheduler(runtime, thread_idx, SIMPLER_ERROR_READY_QUEUE_OVERFLOW); @@ -1340,7 +1340,7 @@ int32_t SchedulerContext::resolve_and_dispatch(Runtime *runtime, int32_t thread_ fail_scheduler(runtime, thread_idx, SIMPLER_ERROR_INVALID_ARGS); break; } - if (nodes_materialized > 0 || result == GraphMaterializeResult::PREPARED) { + if (tasks_materialized > 0 || result == GraphMaterializeResult::PREPARED) { made_progress = true; } #if SIMPLER_DFX @@ -1348,7 +1348,7 @@ int32_t SchedulerContext::resolve_and_dispatch(Runtime *runtime, int32_t thread_ uint64_t graph_prepare_t1 = get_sys_cnt_aicpu(); chip_swimlane_aicpu_record_graph_prepare( thread_idx, graph_prepare_t0, graph_prepare_t1, chip_swimlane.sched_loop_count, prepare_task_id, - static_cast(nodes_materialized) + static_cast(tasks_materialized) ); _t0_phase = graph_prepare_t1; } diff --git a/src/a2a3/runtime/host_build_graph/runtime/shared_memory.h b/src/a2a3/runtime/host_build_graph/runtime/shared_memory.h index 13d5117930..7af119c046 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/shared_memory.h +++ b/src/a2a3/runtime/host_build_graph/runtime/shared_memory.h @@ -392,7 +392,7 @@ struct HeapRebase { // Translate one address the image carries. Anything below HEAP_VIRTUAL_BASE is a // real device address the caller owns — a boundary tensor, or an unset field left // at 0 — and is returned untouched. At or above it, the address came from the -// graph heap: a recorded node's outputs live in its Definition as offsets, so no +// graph heap: a recorded in-graph task's outputs live in its Definition as offsets, so no // Graph-recording address (>= GRAPH_RECORD_VIRTUAL_BASE) reaches the image, and // the committed-heap bound below rejects one rather than classifying by it. inline uint64_t rebased_heap_addr(uint64_t addr, const HeapRebase &rebase) noexcept { @@ -480,7 +480,7 @@ inline uint64_t compact_live_image( const auto *mirror_fanin = reinterpret_cast(mirror_base + from.fanin_pool); const auto *mirror_tensors = reinterpret_cast(mirror_base + from.tensor_pool); const auto *mirror_scalars = reinterpret_cast(mirror_base + from.scalar_pool); - // An unbound region stays unbound: a Graph node's payload never gets a fanin + // An unbound region stays unbound: an in-graph task's payload never gets a fanin // region, and its count is 0, so no consumer resolves it. A bound one is inside // its own mirror pool by construction — the only binder is a bump cursor on that // pool — and the translation below depends on it, so it is asserted rather than diff --git a/src/a2a3/runtime/host_build_graph/runtime/submit_types.h b/src/a2a3/runtime/host_build_graph/runtime/submit_types.h index e8be0426ae..cf46391850 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/submit_types.h +++ b/src/a2a3/runtime/host_build_graph/runtime/submit_types.h @@ -136,9 +136,14 @@ class ActiveMask { uint8_t core_mask() const { return raw_ & 0x07u; } + // No subtask slot is active, so nothing dispatches to a core. Sole definition + // of "dummy": both ResourceShape::DUMMY below and TaskKind::DUMMY derive from + // it, so the two cannot disagree. + bool is_dummy() const { return core_mask() == 0; } + ResourceShape to_shape() const { + if (is_dummy()) return ResourceShape::DUMMY; uint8_t cmask = core_mask(); - if (cmask == 0) return ResourceShape::DUMMY; int bit_count = __builtin_popcount(cmask); if (bit_count >= 2) return ResourceShape::MIX; if (cmask & SUBTASK_MASK_AIC) return ResourceShape::AIC; diff --git a/src/a2a3/runtime/host_build_graph/runtime/tensormap.h b/src/a2a3/runtime/host_build_graph/runtime/tensormap.h index 76173bcbc5..336b53f8ba 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/tensormap.h +++ b/src/a2a3/runtime/host_build_graph/runtime/tensormap.h @@ -569,7 +569,7 @@ struct ChipTensorMap { const int32_t task_slot = local_id; // A producer's low id field is a task chain index directly, so the id space a // caller inserts under has to be the one this map was dimensioned for: a - // whole-run map takes task capacity, a Graph recording's takes GRAPH_MAX_NODES. + // whole-run map takes task capacity, a Graph recording's takes MAX_IN_GRAPH_TASKS. debug_assert(task_slot >= 0 && task_slot < max_tasks); entry->producer_task_id = producer_task_id; diff --git a/src/a2a3/runtime/host_build_graph/runtime/types.h b/src/a2a3/runtime/host_build_graph/runtime/types.h index c6e113913a..f58ab8fb64 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/types.h +++ b/src/a2a3/runtime/host_build_graph/runtime/types.h @@ -88,7 +88,7 @@ enum class ScopeMode : uint8_t { * LIFETIME — single-pass only: * Internally this class stores pointers into the submitting task's tensor * storage: the region named by TaskPayload::tensors for a plain submit, the - * GraphRecording node's tensors for a submit inside a Graph body. Both belong to + * GraphRecording in-graph task's tensors for a submit inside a Graph body. Both belong to * one orchestration pass, which the next bind rebuilds over the same bytes. * Therefore the TaskOutputTensors instance, the const simpler::hbg::Tensor& returned by * get_ref(), and any pointer derived from either MUST NOT outlive the @@ -696,7 +696,7 @@ inline constexpr uint32_t GRAPH_MAX_SCALAR_ARGS = 64; // Boundary arguments of a Graph. Sized independently of CoreTaskArgs because the // outer GRAPH payload carries the whole boundary, while materialize stages only -// one node's arguments at a time. The compact boundary values live in that +// one in-graph task's arguments at a time. The compact boundary values live in that // payload's argument-pool regions, so widening these caps costs pool bytes only // for Graphs that use them; TaskPayload itself stays fixed-size. using GraphTaskArgs = Arg; diff --git a/src/a5/runtime/host_build_graph/docs/RUNTIME_LOGIC.md b/src/a5/runtime/host_build_graph/docs/RUNTIME_LOGIC.md index bbd4528811..a5e7a8cd3c 100644 --- a/src/a5/runtime/host_build_graph/docs/RUNTIME_LOGIC.md +++ b/src/a5/runtime/host_build_graph/docs/RUNTIME_LOGIC.md @@ -140,8 +140,8 @@ signal, so such a queue accepts one push and then reports full. The ramp is mandatory but it is a function of `capacity` alone, so `SchedulerState::seed_queue_slots()` writes it on the device rather than `bind` shipping 1,775,616 bytes of it. The ready queues are still *not* bounded to -`total_tasks`: graph execution expands a GRAPH task into on-device nodes that push -past the host task count, so every slot must carry a valid sequence. +`total_tasks`: graph execution expands a GRAPH task into on-device in-graph tasks that +push past the host task count, so every slot must carry a valid sequence. Both run before the boot thread publishes `runtime_init_ready_`, which is what releases the peer threads into the dispatch loop, so no push can observe an diff --git a/src/a5/runtime/host_build_graph/docs/profiling_levels.md b/src/a5/runtime/host_build_graph/docs/profiling_levels.md index 2db0be30ca..f17d9c3b5d 100644 --- a/src/a5/runtime/host_build_graph/docs/profiling_levels.md +++ b/src/a5/runtime/host_build_graph/docs/profiling_levels.md @@ -246,7 +246,7 @@ so records and spans read against each other with no alignment step. | Group | Kinds | | ----- | ----- | | Bind segments (partition the stage) | `args`, `arena_build`, `static_arena`, `gm_heap`, `shared_mem`, `runtime_init`, `host_orch`, `graph_upload`, `sm_h2d`, `arena_h2d`, `host_view_close` | -| Orchestrator operations (inside `host_orch`) | `submit_task`, `alloc_tensors`, `record_node`, `graph_submit`, `build_definition`, `graph_begin`, `recording_wait`, `graph_commit`, `submit_admit`, `record_handoff`, `generated_args` | +| Orchestrator operations (inside `host_orch`) | `submit_task`, `alloc_tensors`, `record_in_graph_task`, `graph_submit`, `build_definition`, `graph_begin`, `recording_wait`, `graph_commit`, `submit_admit`, `record_handoff`, `generated_args` | Three of the orchestrator kinds end with a task submitted — `submit_task`, `alloc_tensors`, `graph_submit` — so their count is the bind's `total_tasks` @@ -266,7 +266,7 @@ the wrong one produces a number that reads as data and is not: - **A record is an interval** — one operation, start to end. Its `detail` says *which* operation (a task id, a Graph key, the submission index) or *how much* - it covered (`build_definition`'s node count, `recording_wait`'s in-flight + it covered (`build_definition`'s in-graph task count, `recording_wait`'s in-flight count). That is the whole contract. - **A quantity about a segment is an attribute** — `bytes=`, `heap_used=`, `spilled=`, `minflt=`, `nvcsw=`. It goes in the segment's attribute string, @@ -326,7 +326,7 @@ python -m pytest --platform --device 0 --enable-chip-swimlane They come from per-kind counters, not from the record pool. The counters use lock-free atomic additions across the main and recording-worker lanes, with every phase isolated on its own cache line so concurrent `graph_submit` and - `record_node` updates do not false-share. The per-event pool is armed when the + `record_in_graph_task` updates do not false-share. The per-event pool is armed when the artifact is wanted (`SIMPLER_HBG_HOST_PHASE_RECORDS_ENABLE` *and* an output prefix) or whenever the chip swimlane is at `ORCH_PHASES`; a steady-state run satisfies neither, so it pays no pool append and no artifact lock at all. A @@ -344,7 +344,7 @@ python -m pytest --platform --device 0 --enable-chip-swimlane This is the channel to read for a distribution or a per-event timeline; the summed lines cannot express either. Every record carries its producer Linux tid. `strace_timing.py --swimlane --host-phase-records ` draws each record - inside the matching `chip.run.bind`; `record_node` and `build_definition` + inside the matching `chip.run.bind`; `record_in_graph_task` and `build_definition` appear on the `graph record worker` lane, while outer `graph_submit` events appear on the `graph submit main` lane. diff --git a/src/a5/runtime/host_build_graph/host/graph_recorder_pool.h b/src/a5/runtime/host_build_graph/host/graph_recorder_pool.h index 3a5cea9eff..bf8cfefeee 100644 --- a/src/a5/runtime/host_build_graph/host/graph_recorder_pool.h +++ b/src/a5/runtime/host_build_graph/host/graph_recorder_pool.h @@ -143,7 +143,7 @@ class GraphAsyncRecordingState { size_t owned_args_index; { - std::lock_guard lock(mutex_); + std::scoped_lock lock(mutex_); if (stopping_ || free_owned_args_count_ == 0 || job_count_ == kJobCapacity) return false; owned_args_index = free_owned_args_[--free_owned_args_count_]; } @@ -175,8 +175,8 @@ class GraphAsyncRecordingState { cv_.notify_one(); // graph_begin() has already installed the keyed in-flight entry and // submitted the zero-heap outer shell. Enqueuing the private job is - // therefore the last dependency of the caller; graph_prepare() and all - // node recording may start after later shells are submitted. + // therefore the last dependency of the caller; graph_prepare() and the + // recording of the body may start after later shells are submitted. return true; } @@ -236,11 +236,11 @@ class GraphAsyncRecordingState { // still reports ready and prewarm() fails: the lazy stand-up in // graph_recording_reset is the backstop for the workers start() adds later. if (!graph_recorder_stand_up_storage()) { - std::lock_guard lock(mutex_); + std::scoped_lock lock(mutex_); storage_failed_ = true; } { - std::lock_guard lock(mutex_); + std::scoped_lock lock(mutex_); ready_workers_++; } cv_.notify_all(); @@ -262,7 +262,7 @@ class GraphAsyncRecordingState { current.function(owned_args_[current.owned_args_index].args()); current.function = {}; { - std::lock_guard lock(mutex_); + std::scoped_lock lock(mutex_); free_owned_args_[free_owned_args_count_++] = current.owned_args_index; active_jobs_--; } @@ -273,7 +273,7 @@ class GraphAsyncRecordingState { void shutdown() { wait(); { - std::lock_guard lock(mutex_); + std::scoped_lock lock(mutex_); stopping_ = true; } cv_.notify_all(); diff --git a/src/a5/runtime/host_build_graph/host/host_phase_trace.cpp b/src/a5/runtime/host_build_graph/host/host_phase_trace.cpp index 174e0fcf22..12d1eb8567 100644 --- a/src/a5/runtime/host_build_graph/host/host_phase_trace.cpp +++ b/src/a5/runtime/host_build_graph/host/host_phase_trace.cpp @@ -45,7 +45,7 @@ namespace { // path performs no atomic read-modify-write at all. Sharing them cost the emitting // thread about 0.8 us per record at eight producers, purely in cache-line // ownership: every producer of a Graph workload records the same kind -// (`record_node`), so a per-kind counter is a single line eight threads fight +// (`record_in_graph_task`), so a per-kind counter is a single line eight threads fight // over, and the `alignas(64)` that separates one kind from another does nothing // about that. // @@ -307,7 +307,7 @@ void host_phase_record_bind(uint32_t kind, uint64_t start_ns, const char *attrs, void host_phase_trace_begin(const void *host_api) { TraceState &s = state(); - std::lock_guard lock(s.lifecycle_mutex); + std::scoped_lock lock(s.lifecycle_mutex); s.active.store(false, std::memory_order_release); drain_in_flight_records(s); s.api = static_cast(host_api); @@ -340,7 +340,7 @@ void host_phase_trace_note_submitted(uint64_t submitted_tasks) { void host_phase_trace_end() { TraceState &s = state(); - std::lock_guard lifecycle_lock(s.lifecycle_mutex); + std::scoped_lock lifecycle_lock(s.lifecycle_mutex); if (!s.active.load(std::memory_order_relaxed)) { return; } diff --git a/src/a5/runtime/host_build_graph/host/runtime_maker.cpp b/src/a5/runtime/host_build_graph/host/runtime_maker.cpp index 72e4a717cf..7b1f9a705b 100644 --- a/src/a5/runtime/host_build_graph/host/runtime_maker.cpp +++ b/src/a5/runtime/host_build_graph/host/runtime_maker.cpp @@ -476,7 +476,7 @@ bool bind_graph_definitions( return false; } GraphExecutionStorageLayout storage_layout{}; - if (definition->task_count == 0 || definition->task_count > GRAPH_MAX_NODES || + if (definition->task_count == 0 || definition->task_count > MAX_IN_GRAPH_TASKS || definition->full_key != upload->full_key || !graph_execution_storage_layout( static_cast(definition->task_count), definition->tensor_arg_count, @@ -497,21 +497,27 @@ bool bind_graph_definitions( return false; } const uintptr_t storage_addr = outer_base + definition->required_heap; - if (storage_addr % alignof(GraphNodeStorage) != 0) { + if (storage_addr % alignof(InGraphTaskStorage) != 0) { LOG_ERROR("host-orch: Graph runtime storage address is misaligned"); return false; } PackedDefinition &packed_definition = object_it->second; if (!packed_definition.populations_ready) { - const GraphNodeDefinition *nodes = - graph_definition_array(*definition, definition->off_nodes, definition->task_count); - if (nodes == nullptr) { - LOG_ERROR("host-orch: invalid Graph Definition node array"); + const InGraphTaskDefinition *tasks = graph_definition_array( + *definition, definition->off_in_graph_tasks, definition->task_count + ); + if (tasks == nullptr) { + LOG_ERROR("host-orch: invalid Graph Definition in-graph task array"); return false; } for (uint32_t i = 0; i < definition->task_count; ++i) { + // Sizing takes the kind materialize will give this task. add_task + // singles out GRAPH and routes everything else by shape, and a Graph + // body member is never the shell, so the shape decides. Derived here + // the same way the device derives it, so the two cannot drift. + const ActiveMask mask(tasks[i].active_mask); packed_definition.ready_queue_populations.add_task( - ActiveMask(nodes[i].active_mask), TaskAttrs(nodes[i].task_attrs), TaskKind::GRAPH_NODE + mask, TaskAttrs(tasks[i].task_attrs), mask.is_dummy() ? TaskKind::DUMMY : TaskKind::KERNEL ); } packed_definition.populations_ready = true; @@ -855,7 +861,7 @@ int32_t run_host_orchestration( reinterpret_cast(gm_heap) < HEAP_VIRTUAL_BASE && "device memory reaches into the virtual heap window" ); // The alignment bind_graph_definitions checked on the virtual base — a Graph - // task's runtime storage must land on alignof(GraphNodeStorage) — carries to + // task's runtime storage must land on alignof(InGraphTaskStorage) — carries to // the real base only while the two are congruent: both are aligned to // kDefaultBaseAlign, and that covers the storage's own requirement. static_assert( @@ -863,8 +869,8 @@ int32_t run_host_orchestration( "the virtual heap base must share the committed region's alignment" ); static_assert( - alignof(GraphNodeStorage) <= DeviceArena::kDefaultBaseAlign, - "a Graph node's storage alignment must be covered by the heap region's base alignment" + alignof(InGraphTaskStorage) <= DeviceArena::kDefaultBaseAlign, + "an in-graph task's storage alignment must be covered by the heap region's base alignment" ); always_assert(reinterpret_cast(gm_heap) % DeviceArena::kDefaultBaseAlign == 0); const sm_layout::HeapRebase heap_rebase{reinterpret_cast(gm_heap), heap_bytes}; diff --git a/src/a5/runtime/host_build_graph/orchestration/orchestration_api.h b/src/a5/runtime/host_build_graph/orchestration/orchestration_api.h index b43aae394b..15d41dc1b3 100644 --- a/src/a5/runtime/host_build_graph/orchestration/orchestration_api.h +++ b/src/a5/runtime/host_build_graph/orchestration/orchestration_api.h @@ -580,7 +580,7 @@ static inline GraphSubmitResult rt_submit_graph_impl(uint64_t graph_key, const G // Handing the recording to a worker. Measured at 10-75 us per start and covered // by no other record: it runs after rt_graph_begin returns, so a swimlane shows // it as a gap with no recorder active — which is what it is, the recorder has - // not reached its first node yet. + // not reached its first in-graph task yet. rt_record_orch_phase(HostPhaseKind::OrchRecordHandoff, _begun_ns, _exit_ns, graph_key); } _phase.count++; diff --git a/src/a5/runtime/host_build_graph/runtime/async_wait.h b/src/a5/runtime/host_build_graph/runtime/async_wait.h index 3c7b10186d..2fc1176b5c 100644 --- a/src/a5/runtime/host_build_graph/runtime/async_wait.h +++ b/src/a5/runtime/host_build_graph/runtime/async_wait.h @@ -124,7 +124,7 @@ struct AsyncWaitEntry { struct AsyncPollResult { int32_t completed{0}; // Host-submitted stream tasks completed. - int32_t resolved{0}; // All task completions, including internal Graph nodes. + int32_t resolved{0}; // All task completions, including in-graph tasks. int32_t error_code{SIMPLER_ERROR_NONE}; ChipTaskSlotState *failed_slot_state{nullptr}; }; diff --git a/src/a5/runtime/host_build_graph/runtime/orchestrator_core/orchestrator.cpp b/src/a5/runtime/host_build_graph/runtime/orchestrator_core/orchestrator.cpp index cf66c916cf..6e78e6c7ea 100644 --- a/src/a5/runtime/host_build_graph/runtime/orchestrator_core/orchestrator.cpp +++ b/src/a5/runtime/host_build_graph/runtime/orchestrator_core/orchestrator.cpp @@ -303,8 +303,8 @@ struct GraphRecordedScalarSourceRef { size_t source_index{0}; }; -// A node's dispatch predicate, held as the operand tensor plus the element index -// within it rather than the absolute address submit would resolve. The tensor is +// A recorded task's dispatch predicate, held as the operand tensor plus the element +// index within it rather than the absolute address submit would resolve. The tensor is // copied because the caller only lends it for the duration of the submit call. struct GraphRecordedPredicate { simpler::hbg::Tensor operand; @@ -315,7 +315,7 @@ struct GraphRecordedPredicate { PredicateOp op{PredicateOp::NONE}; }; -struct GraphRecordedNode { +struct RecordedInGraphTask { std::array kernel_ids{}; ActiveMask active_mask{}; TaskAttrs task_attrs{}; @@ -323,10 +323,10 @@ struct GraphRecordedNode { int16_t total_required_subtasks{0}; size_t total_output_size{0}; uintptr_t record_packed_base{0}; - // This node's slice of the recording's node tensor pool, as an offset so the node + // This task's slice of the recording's tensor pool, as an offset so the task // carries no address into storage the recording owns. The element addresses are handed - // to the caller through TaskOutputTensors and have to stay valid while every later node - // records, which the pool satisfies by being allocated at the cap and never growing. + // to the caller through TaskOutputTensors and have to stay valid while the rest of the + // body records, which the pool satisfies by being allocated at the cap and never growing. uint32_t tensor_offset{0}; uint32_t tensor_count{0}; // Ranges into the recording's flat arrays. tensor_sources has one entry per @@ -336,18 +336,18 @@ struct GraphRecordedNode { uint32_t scalar_count{0}; uint32_t fanin_offset{0}; uint32_t fanin_count{0}; - // Index into the recording's predicates, or -1 when the node carries none. + // Index into the recording's predicates, or -1 when the task carries none. int32_t predicate_index{-1}; ArgsDumpTaskMetadata dump_metadata; - // Restore the state a fresh node has, without giving up `tensors`'s buffer -- which is - // the point of reusing a slot. A reused slot that keeps any field of the previous body + // Restore the state a freshly recorded task has, in a slot the previous body left + // behind. A reused slot that keeps any field of the previous body // records a Definition that body never had, and predicate_index and dump_metadata are // written only on the paths that have one, so neither can be left to the fill. // - // Field by field rather than `*this = GraphRecordedNode{}`: the latter is immune to - // fields added later, but it costs a second write of the whole struct on every node - // and measured 350-700 us per bind on dsv4's 1679 nodes. The static_assert below is + // Field by field rather than `*this = RecordedInGraphTask{}`: the latter is immune to + // fields added later, but it costs a second write of the whole struct on every recorded + // task and measured 350-700 us per bind on dsv4's 1679 tasks. The static_assert below is // the cheap half of that guarantee -- adding a field breaks the build here, which is // where the reader is told to extend this function. void reset() { @@ -374,19 +374,19 @@ struct GraphRecordedNode { // from the previous body into the next recording -- silently, as a Definition that body // never had. Adding a field changes this size, so the build stops here instead. static_assert( - sizeof(GraphRecordedNode) == 104, "GraphRecordedNode gained or lost a field: extend reset() to match, then " - "update this size" + sizeof(RecordedInGraphTask) == 104, "RecordedInGraphTask gained or lost a field: extend reset() to match, then " + "update this size" ); -// One recorded node's scratch output window. reserve_heap_scratch is a pure bump -// and a node stores the aligned size it advanced by, so consecutive windows abut: +// One recorded task's scratch output window. reserve_heap_scratch is a pure bump +// and a task stores the aligned size it advanced by, so consecutive windows abut: // held in record order these are sorted and disjoint, which is what lets an -// address lookup binary search instead of walking every producer. A node with no +// address lookup binary search instead of walking every producer. A task with no // output advances nothing and owns no entry. struct GraphRecordedOutputRange { uintptr_t begin; uintptr_t end; - uint32_t node_index; + uint32_t task_index; }; // The Graph boundary as the submitting thread captured it, deep-copied because the @@ -412,32 +412,32 @@ struct GraphRecording { // boundary matching while this thread records. const GraphBoundary *boundary{nullptr}; bool unsupported{false}; - std::vector nodes; - // How many of `nodes` this recording has filled. The array itself is never cleared - // and graph_recording_reserve_storage sizes it to the node cap, so a body is recorded - // into slots that already exist: a recorded node makes no allocation at all. - size_t node_count{0}; - // Every recorded node's tensor arguments, packed end to end in one region this - // recording bumps through, and the reason a node holds an offset rather than its own - // buffer: a body's tensors then occupy the bytes they need instead of a page per node - // (a per-node buffer at the cap is 32 x 128 B = exactly one page, so dsv4's 1679 nodes + std::vector tasks; + // How many of `tasks` this recording has filled. The array itself is never cleared + // and graph_recording_reserve_storage sizes it to the in-graph task cap, so a body is + // recorded into slots that already exist: a recorded task makes no allocation at all. + size_t task_count{0}; + // Every recorded task's tensor arguments, packed end to end in one region this + // recording bumps through, and the reason a task holds an offset rather than its own + // buffer: a body's tensors then occupy the bytes they need instead of a page per task + // (a per-task buffer at the cap is 32 x 128 B = exactly one page, so dsv4's 1679 tasks // touched 1679 pages to hold ~210 KB). Allocated once per thread at the cap and never - // grown, which is what keeps a node's borrowed element addresses valid for the rest of + // grown, which is what keeps a task's borrowed element addresses valid for the rest of // the recording. `new[]` default-initializes a trivially-default-constructible Tensor, - // so the region costs no page until a body writes one; each element a node uses is + // so the region costs no page until a body writes one; each element a task uses is // value-initialized before it is filled. - std::unique_ptr node_tensor_pool; - uint32_t node_tensor_cursor{0}; - // Flat per-node arrays, indexed by the ranges on GraphRecordedNode. Held here rather - // than on each node so recording a graph pays no allocation per node per array, and - // reserved to the node cap by graph_recording_reserve_storage so it pays no growth - // either. + std::unique_ptr task_tensor_pool; + uint32_t task_tensor_cursor{0}; + // Flat per-task arrays, indexed by the ranges on RecordedInGraphTask. Held here rather + // than on each recorded task so recording a graph pays no allocation per task per array, + // and reserved to the in-graph task cap by graph_recording_reserve_storage so it pays no + // growth either. std::vector tensor_sources; std::vector scalars; std::vector scalar_sources; std::vector internal_fanins; std::vector output_ranges; - // Indexed by GraphRecordedNode::predicate_index; only predicated nodes + // Indexed by RecordedInGraphTask::predicate_index; only predicated tasks // contribute an entry. std::vector predicates; // Hazard state for the recorded body, owned per recorder thread because @@ -448,14 +448,14 @@ struct GraphRecording { // (register_task_outputs, STEP 4). The shadow-record path replaces // submit_task_common wholesale, so without a map of its own the recorder // can only see the edges tensor-source classification yields — and that - // classification answers "which node's packed window holds these bytes", + // classification answers "which recorded task's packed window holds these bytes", // i.e. who ALLOCATED the buffer, never who wrote it last. A body that // allocates once with alloc_tensors and then writes in place with add_inout - // (the shape every generated orchestration uses) would therefore record a - // node with no edge to its actual producer, and the Definition would replay + // (the shape every generated orchestration uses) would therefore record an + // in-graph task with no edge to its actual producer, and the Definition would replay // a DAG the same body never had when submitted task by task. ChipTensorMap tensor_map{}; - // Set once both the hazard map and the node tensor pool are up, and only then: the two + // Set once both the hazard map and the tensor pool are up, and only then: the two // allocate, so a flag set by the first would let a thread whose second allocation // failed skip the stand-up on its next recording and record through a null pool. bool storage_ready{false}; @@ -468,8 +468,8 @@ struct GraphRecording { bool in_manual_scope() const { return scope_stack_top >= manual_begin_depth; } - simpler::hbg::Tensor *node_tensors(const GraphRecordedNode &node) const { - return node_tensor_pool.get() + node.tensor_offset; + simpler::hbg::Tensor *task_tensors(const RecordedInGraphTask &task) const { + return task_tensor_pool.get() + task.tensor_offset; } const GraphTaskArgs *boundary_args() const { return boundary == nullptr ? nullptr : boundary->args; } @@ -639,7 +639,7 @@ template GraphRecordedScalarSourceRef graph_classify_scalar(const GraphRecording &recording, const ArgT &args, int32_t scalar_index) { if (recording.boundary_args() == nullptr) return {}; - // Identity, not type: an internal node's Arg and the boundary Arg have + // Identity, not type: an in-graph task's Arg and the boundary Arg have // different capacities, so compare the addresses through void. if (static_cast(&args) == static_cast(recording.boundary_args()) && scalar_index < recording.boundary_args()->scalar_count()) { @@ -664,7 +664,7 @@ graph_classify_scalar(const GraphRecording &recording, const ArgT &args, int32_t } // Entry capacity for one recorded body's hazard map. A Definition is capped at -// GRAPH_MAX_NODES nodes and each node registers at most its INOUT/OUTPUT_EXISTING +// MAX_IN_GRAPH_TASKS tasks and each recorded task registers at most its INOUT/OUTPUT_EXISTING // args, so this bounds the worst realistic body while staying a small fraction of // the ordinary path's whole-orchestration pool (CHIP_TENSORMAP_POOL_SIZE). Exhausting // it marks the recording unsupported, which graph_commit reports as @@ -672,18 +672,18 @@ graph_classify_scalar(const GraphRecording &recording, const ArgT &args, int32_t // is no ordinary-path fallback left to take. constexpr int32_t GRAPH_RECORD_TENSORMAP_POOL_SIZE = 16384; -// Elements in the recording's node tensor pool: every node a body can hold, times every -// tensor argument a node can carry. An in-cap body therefore always fits, and the bump -// cursor is checked anyway because a body that overshoots GRAPH_MAX_NODES keeps recording -// so it can finish. -constexpr size_t GRAPH_RECORD_NODE_TENSOR_POOL_ELEMS = - static_cast(GRAPH_MAX_NODES) * static_cast(CORE_MAX_TENSOR_ARGS); +// Elements in the recording's tensor pool: every in-graph task a body can hold, times +// every tensor argument one such task can carry. An in-cap body therefore always fits, and +// the bump cursor is checked anyway because a body that overshoots MAX_IN_GRAPH_TASKS keeps +// recording so it can finish. +constexpr size_t GRAPH_RECORD_TENSOR_POOL_ELEMS = + static_cast(MAX_IN_GRAPH_TASKS) * static_cast(CORE_MAX_TENSOR_ARGS); // The graph_local_id a recorded task's IN_GRAPH id carries. A recorded task belongs // to no Graph task yet -- every shell replaying the Definition re-mints the id with // its own local id at materialize -- so record time names a task by its index alone, // and the id's low field is that index and nothing else. That is what keeps the -// index inside the GRAPH_MAX_NODES task chains the recording's hazard map is +// index inside the MAX_IN_GRAPH_TASKS task chains the recording's hazard map is // dimensioned for. constexpr uint32_t GRAPH_RECORD_NO_OWNING_GRAPH = 0; @@ -691,7 +691,7 @@ constexpr uint32_t GRAPH_RECORD_NO_OWNING_GRAPH = 0; // reported to the caller, which abandons the recording rather than producing a // Definition with inferred edges missing. bool graph_recording_init_tensor_map(GraphRecording &recording) { - return recording.tensor_map.init(CHIP_TENSORMAP_NUM_BUCKETS, GRAPH_RECORD_TENSORMAP_POOL_SIZE, GRAPH_MAX_NODES); + return recording.tensor_map.init(CHIP_TENSORMAP_NUM_BUCKETS, GRAPH_RECORD_TENSORMAP_POOL_SIZE, MAX_IN_GRAPH_TASKS); } // The recorder thread's own storage for the body it is recording, and the reason none @@ -726,7 +726,7 @@ void unbind_recorder_boundary() { } // Stand this thread's retained storage up at the cap, so no body it records grows any of -// it and no recorded node allocates. +// it and no recorded task allocates. // // Capacity kept across recordings is otherwise the high-water mark of the bodies this // thread happened to record, and which body a thread gets is decided by one FIFO the @@ -735,20 +735,21 @@ void unbind_recorder_boundary() { // binds: a thread that recorded a narrow body first extends its slots and reallocates // every array the first time a wider one lands on it, on whatever bind that happens to // be. Measured on dsv4, whose eight Definitions differ in size, a warm bind still created -// 1336 node slots of 1679. Standing everything up at the cap makes a thread's storage -// independent of the order it saw bodies in. +// 1336 of the 1679 `tasks` slots that body needed. Standing everything up at the cap makes a +// thread's storage independent of the order it saw bodies in. // -// Each bound is a per-node cap times the node cap, so these are the recorded body's own -// limits rather than a worst case invented here: the tensor pool and tensor_sources are -// one entry per tensor argument (CORE_MAX_TENSOR_ARGS), the two scalar arrays one per -// scalar argument (CORE_MAX_SCALAR_ARGS), and predicates and output_ranges at most one per -// node. +// Each bound is a per-task cap times the in-graph task cap, so these are the recorded +// body's own limits rather than a worst case invented here: the tensor pool and +// tensor_sources are one entry per tensor argument (CORE_MAX_TENSOR_ARGS), the two scalar +// arrays one per scalar argument (CORE_MAX_SCALAR_ARGS), and predicates and output_ranges at +// most one per task. // // internal_fanins is the one array left growing, and the reason is the size it grows to -// rather than the bound it could reach. It has no per-node cap: CHIP_MAX_FANIN bounds a -// ring task's inline fanin, but a Graph node's producers travel in the Definition's own -// CSR, which the scheduler reads directly, so the only limits are uint16 producer indices -// and each producer being an earlier node — a structural 1024 x 1023 / 2 edges, 4.2 MB. +// rather than the bound it could reach. It has no per-in-graph-task cap: CHIP_MAX_FANIN +// bounds a global task's inline fanin, but an in-graph task's producers travel in the +// Definition's own CSR, which the scheduler reads directly, so the only limits are uint16 +// producer indices and each producer being an earlier task of the same body — a structural +// 1024 x 1023 / 2 edges, 4.2 MB. // What decides whether growth costs anything is not that bound but whether a reallocation // crosses glibc's mmap threshold, since a freed block below it is reused off the heap // without re-faulting (see the entry cited above). A dsv4 body holds ~630 edges, 5 KB, two @@ -764,20 +765,20 @@ void unbind_recorder_boundary() { // Returns false when the pool cannot be allocated, which the caller treats like a hazard // map it could not stand up. bool graph_recording_reserve_storage(GraphRecording &recording) { - constexpr size_t kNodeCap = GRAPH_MAX_NODES; - recording.node_tensor_pool.reset(new (std::nothrow) simpler::hbg::Tensor[GRAPH_RECORD_NODE_TENSOR_POOL_ELEMS]); - if (recording.node_tensor_pool == nullptr) return false; - recording.nodes.resize(kNodeCap); - recording.tensor_sources.reserve(kNodeCap * static_cast(CORE_MAX_TENSOR_ARGS)); - recording.scalars.reserve(kNodeCap * static_cast(CORE_MAX_SCALAR_ARGS)); - recording.scalar_sources.reserve(kNodeCap * static_cast(CORE_MAX_SCALAR_ARGS)); - recording.output_ranges.reserve(kNodeCap); - recording.predicates.reserve(kNodeCap); + constexpr size_t kInGraphTaskCap = MAX_IN_GRAPH_TASKS; + recording.task_tensor_pool.reset(new (std::nothrow) simpler::hbg::Tensor[GRAPH_RECORD_TENSOR_POOL_ELEMS]); + if (recording.task_tensor_pool == nullptr) return false; + recording.tasks.resize(kInGraphTaskCap); + recording.tensor_sources.reserve(kInGraphTaskCap * static_cast(CORE_MAX_TENSOR_ARGS)); + recording.scalars.reserve(kInGraphTaskCap * static_cast(CORE_MAX_SCALAR_ARGS)); + recording.scalar_sources.reserve(kInGraphTaskCap * static_cast(CORE_MAX_SCALAR_ARGS)); + recording.output_ranges.reserve(kInGraphTaskCap); + recording.predicates.reserve(kInGraphTaskCap); return true; } // Bind this thread's storage to one in-flight entry and empty it. Returns false when the -// hazard map or the node tensor pool cannot be stood up, which is only reachable on the +// hazard map or the tensor pool cannot be stood up, which is only reachable on the // thread's first recording. // Stand this thread's storage up once, or report that it could not be. Idempotent. // @@ -792,7 +793,7 @@ bool graph_recording_stand_up(GraphRecording &recording) { return false; } } catch (const std::bad_alloc &) { - // The node tensor pool is a nothrow new, but the flat arrays are vectors whose + // The tensor pool is a nothrow new, but the flat arrays are vectors whose // resize/reserve throw. This also runs on a recorder worker as it starts, where an // escaping exception terminates the process instead of letting the pool's prewarm // report the failure. @@ -804,14 +805,14 @@ bool graph_recording_stand_up(GraphRecording &recording) { } // Bind this thread's storage to one in-flight entry and empty it. Returns false when the -// hazard map or the node tensor pool cannot be stood up. +// hazard map or the tensor pool cannot be stood up. bool graph_recording_reset(GraphRecording &recording, const GraphInflightRecording &entry) { - // A body over GRAPH_MAX_NODES is abandoned, but it still grew every array to its real + // A body over MAX_IN_GRAPH_TASKS is abandoned, but it still grew every array to its real // size while it ran. Handing that to the next recording would retain storage for a // Definition that can never be published, unbounded, for the process's life -- so an // over-cap recording gives its storage back instead of passing it on. This is what - // makes the bound documented on GraphRecording::node_count true rather than nominal. - if (recording.nodes.size() > GRAPH_MAX_NODES) { + // makes the bound documented on GraphRecording::task_count true rather than nominal. + if (recording.tasks.size() > MAX_IN_GRAPH_TASKS) { recording = GraphRecording{}; } if (!graph_recording_stand_up(recording)) { @@ -826,9 +827,9 @@ bool graph_recording_reset(GraphRecording &recording, const GraphInflightRecordi recording.manual_begin_depth = CHIP_MAX_SCOPE_DEPTH; // clear() keeps each array's capacity, and the stand-up above reserved every one of // them to what a body at the cap needs, so no body a thread records can grow one. - // nodes is deliberately not cleared: see GraphRecording::node_count. - recording.node_count = 0; - recording.node_tensor_cursor = 0; + // tasks is deliberately not cleared: see GraphRecording::task_count. + recording.task_count = 0; + recording.task_tensor_cursor = 0; recording.tensor_sources.clear(); recording.scalars.clear(); recording.scalar_sources.clear(); @@ -839,12 +840,12 @@ bool graph_recording_reset(GraphRecording &recording, const GraphInflightRecordi } bool graph_classify_tensor( - const GraphRecording &recording, const GraphRecordedNode ¤t, int32_t task_index, + const GraphRecording &recording, const RecordedInGraphTask ¤t, int32_t task_index, const simpler::hbg::Tensor &tensor, GraphRecordedTensorSourceRef *source ) { if (graph_tensor_from_boundary(recording, tensor, source)) return true; const uintptr_t tensor_addr = static_cast(tensor.buffer.addr); - // The node being recorded is not in output_ranges yet — its entry is appended + // The task being recorded is not in output_ranges yet — its entry is appended // once its own tensors are classified — so its window is tested here, and a // hit is OWN_OUTPUT rather than a dependency. if (current.record_packed_base != 0 && current.total_output_size != 0 && @@ -869,7 +870,7 @@ bool graph_classify_tensor( const GraphRecordedOutputRange &range = *(after - 1); if (tensor_addr >= range.end) return false; source->source = GraphRecordedTensorSource::INTERNAL; - source->source_index = range.node_index; + source->source_index = range.task_index; source->packed_offset = tensor_addr - range.begin; return true; } @@ -948,9 +949,9 @@ T *graph_image_section(std::byte *image, uint32_t offset) { // Counts, section offsets and total_bytes for the image this recording produces, // settled without writing any of it so the destination can be claimed at the // exact size. required_heap comes from the fill, which is the pass that walks the -// nodes in order. +// tasks in order. std::optional graph_layout_definition(const GraphRecording &recording) { - if (recording.unsupported || recording.node_count == 0 || recording.node_count > GRAPH_MAX_NODES || + if (recording.unsupported || recording.task_count == 0 || recording.task_count > MAX_IN_GRAPH_TASKS || recording.boundary_tensors().empty() || recording.boundary_tensors().size() > UINT16_MAX || recording.boundary_tensors().size() != recording.boundary_types().size() || recording.boundary_args() == nullptr) { @@ -962,10 +963,10 @@ std::optional graph_layout_definition(const GraphRecording &rec size_t total_fanins = 0; size_t root_count = 0; size_t predicate_count = 0; - // node_count, not nodes.size(): the array keeps the slots a longer body left behind, + // task_count, not tasks.size(): the array keeps the slots a longer body left behind, // and those are not part of this recording. - for (size_t node = 0; node < recording.node_count; ++node) { - const GraphRecordedNode &source = recording.nodes[node]; + for (size_t i = 0; i < recording.task_count; ++i) { + const RecordedInGraphTask &source = recording.tasks[i]; if (source.tensor_count > UINT32_MAX - total_tensors || source.scalar_count > UINT32_MAX - total_scalars || source.fanin_count > UINT32_MAX - total_fanins || source.tensor_source_offset > recording.tensor_sources.size() || @@ -988,7 +989,7 @@ std::optional graph_layout_definition(const GraphRecording &rec GraphDefinition definition{}; definition.full_key = recording.full_key; - definition.task_count = static_cast(recording.node_count); + definition.task_count = static_cast(recording.task_count); definition.edge_count = static_cast(total_fanins); definition.root_count = static_cast(root_count); definition.boundary_count = static_cast(recording.boundary_tensors().size()); @@ -1007,13 +1008,15 @@ std::optional graph_layout_definition(const GraphRecording &rec definition.execution_storage_bytes = static_cast(execution_storage_bytes); size_t image_bytes = sizeof(GraphDefinition); - if (!graph_layout_section(recording.node_count + 1, &image_bytes, &definition.off_fanout_offsets) || + if (!graph_layout_section(recording.task_count + 1, &image_bytes, &definition.off_fanout_offsets) || !graph_layout_section(total_fanins, &image_bytes, &definition.off_fanout_indices) || - !graph_layout_section(recording.node_count + 1, &image_bytes, &definition.off_fanin_offsets) || + !graph_layout_section(recording.task_count + 1, &image_bytes, &definition.off_fanin_offsets) || !graph_layout_section(total_fanins, &image_bytes, &definition.off_fanin_indices) || !graph_layout_section(root_count, &image_bytes, &definition.off_root_indices) || - !graph_layout_section(recording.node_count, &image_bytes, &definition.off_node_offsets) || - !graph_layout_section(recording.node_count, &image_bytes, &definition.off_nodes) || + !graph_layout_section(recording.task_count, &image_bytes, &definition.off_in_graph_task_offsets) || + !graph_layout_section( + recording.task_count, &image_bytes, &definition.off_in_graph_tasks + ) || !graph_layout_section(total_tensors, &image_bytes, &definition.off_tensors) || !graph_layout_section(total_tensors, &image_bytes, &definition.off_tensor_sources) || !graph_layout_section(total_scalars, &image_bytes, &definition.off_scalars) || @@ -1052,8 +1055,8 @@ bool graph_fill_definition(const GraphRecording &recording, GraphDefinition defi auto *fanin_offsets = graph_image_section(image, definition.off_fanin_offsets); auto *fanin_indices = graph_image_section(image, definition.off_fanin_indices); auto *roots = graph_image_section(image, definition.off_root_indices); - auto *node_offsets = graph_image_section(image, definition.off_node_offsets); - auto *nodes = graph_image_section(image, definition.off_nodes); + auto *in_graph_task_offsets = graph_image_section(image, definition.off_in_graph_task_offsets); + auto *tasks = graph_image_section(image, definition.off_in_graph_tasks); auto *tensors = graph_image_section(image, definition.off_tensors); auto *tensor_sources = graph_image_section(image, definition.off_tensor_sources); auto *scalars = graph_image_section(image, definition.off_scalars); @@ -1069,16 +1072,16 @@ bool graph_fill_definition(const GraphRecording &recording, GraphDefinition defi // A producer's fanout count is accumulated across the consumer walk below and then // prefix-summed in place, so every entry has to start at zero — including [0], // which nothing else writes and which the device checks is zero. - std::fill_n(fanout_offsets, recording.node_count + 1, 0U); + std::fill_n(fanout_offsets, recording.task_count + 1, 0U); fanin_offsets[0] = 0; - for (size_t i = 0; i < recording.node_count; ++i) { - const GraphRecordedNode &source = recording.nodes[i]; + for (size_t i = 0; i < recording.task_count; ++i) { + const RecordedInGraphTask &source = recording.tasks[i]; if (source.total_output_size > static_cast(INT32_MAX) || source.tensor_count > static_cast(INT32_MAX) || source.scalar_count > static_cast(INT32_MAX) || source.fanin_count > UINT16_MAX) { return false; } - node_offsets[i] = required_heap; + in_graph_task_offsets[i] = required_heap; const uint64_t output_bytes = CHIP_ALIGN_UP(source.total_output_size, CHIP_ALIGN_SIZE); if (required_heap > UINT64_MAX - output_bytes) return false; required_heap += output_bytes; @@ -1092,19 +1095,19 @@ bool graph_fill_definition(const GraphRecording &recording, GraphDefinition defi } fanin_offsets[i + 1] = static_cast(fanin_cursor); - GraphNodeDefinition &node = nodes[i]; - std::copy(source.kernel_ids.begin(), source.kernel_ids.end(), std::begin(node.kernel_id)); - node.active_mask = source.active_mask.raw(); - node.task_attrs = source.task_attrs.raw(); - node.logical_block_num = source.logical_block_num; - node.total_required_subtasks = source.total_required_subtasks; - node.tensor_count = static_cast(source.tensor_count); - node.scalar_count = static_cast(source.scalar_count); - node.total_output_size = static_cast(source.total_output_size); - node.tensor_offset = static_cast(tensor_cursor); - node.scalar_offset = static_cast(scalar_cursor); - node.dump_metadata = source.dump_metadata; - node.predicate_slot = 0; + InGraphTaskDefinition &task = tasks[i]; + std::copy(source.kernel_ids.begin(), source.kernel_ids.end(), std::begin(task.kernel_id)); + task.active_mask = source.active_mask.raw(); + task.task_attrs = source.task_attrs.raw(); + task.logical_block_num = source.logical_block_num; + task.total_required_subtasks = source.total_required_subtasks; + task.tensor_count = static_cast(source.tensor_count); + task.scalar_count = static_cast(source.scalar_count); + task.total_output_size = static_cast(source.total_output_size); + task.tensor_offset = static_cast(tensor_cursor); + task.scalar_offset = static_cast(scalar_cursor); + task.dump_metadata = source.dump_metadata; + task.predicate_slot = 0; if (source.predicate_index >= 0) { if (static_cast(source.predicate_index) >= recording.predicates.size()) return false; const GraphRecordedPredicate &recorded = recording.predicates[source.predicate_index]; @@ -1118,9 +1121,9 @@ bool graph_fill_definition(const GraphRecording &recording, GraphDefinition defi packed.elem_size = recorded.elem_size; packed.op = static_cast(recorded.op); predicates[predicate_cursor] = packed; - node.predicate_slot = static_cast(++predicate_cursor); + task.predicate_slot = static_cast(++predicate_cursor); } - const simpler::hbg::Tensor *source_tensors = recording.node_tensors(source); + const simpler::hbg::Tensor *source_tensors = recording.task_tensors(source); for (size_t t = 0; t < source.tensor_count; ++t) { if (source_tensors[t].ndims > MAX_TENSOR_DIMS) return false; tensors[tensor_cursor] = graph_tensor_pack(source_tensors[t]); @@ -1149,10 +1152,10 @@ bool graph_fill_definition(const GraphRecording &recording, GraphDefinition defi return false; } definition.required_heap = required_heap; - for (size_t i = 0; i < recording.node_count; ++i) + for (size_t i = 0; i < recording.task_count; ++i) fanout_offsets[i + 1] += fanout_offsets[i]; - std::vector cursors(fanout_offsets, fanout_offsets + recording.node_count); - for (size_t consumer = 0; consumer < recording.node_count; ++consumer) { + std::vector cursors(fanout_offsets, fanout_offsets + recording.task_count); + for (size_t consumer = 0; consumer < recording.task_count; ++consumer) { for (uint32_t f = fanin_offsets[consumer]; f < fanin_offsets[consumer + 1]; ++f) { const size_t producer = fanin_indices[f]; fanout_indices[cursors[producer]++] = static_cast(consumer); @@ -1448,7 +1451,7 @@ static bool prepare_task( out->slot_state->logical_block_num = block_num; out->slot_state->active_mask = active_mask; out->slot_state->task_attrs = task_attrs; - out->slot_state->task_kind = active_mask ? TaskKind::KERNEL : TaskKind::DUMMY; + out->slot_state->task_kind = active_mask.is_dummy() ? TaskKind::DUMMY : TaskKind::KERNEL; // Reclaim gate: seed last_consumer to self, so a producer with no consumers // is retirable once completed_watermark >= its own id. Each fanin edge bumps // it in append_fanin_or_fail. completion_flags for this slot were cleared @@ -1472,8 +1475,8 @@ void OrchestratorState::begin_scope(ScopeMode mode) { // A Graph replays as a flat DAG with no scope structure: scope boundaries only // shape scheduling on the ordinary path, and the shadow-record path submits no // ordinary tasks. So a scope inside a Graph body must not touch the real scope - // stack. Its manual/auto mode still matters, though — the recorder infers a - // node's producers with the same compute_task_fanin the ordinary path uses, and + // stack. Its manual/auto mode still matters, though — the recorder infers a recorded + // task's producers with the same compute_task_fanin the ordinary path uses, and // that inference is suppressed inside a manual scope — so the depth is tracked on // the recording instead. if (GraphRecording *recording = active_graph_recording(orch); recording != nullptr) { @@ -2074,17 +2077,16 @@ bool graph_finalize_pending_submissions(OrchestratorState *orch, GraphHostState return true; } -// Record one internal Graph node while recording, without consuming a -// task-table slot. Builds the node's metadata and materialized outputs -// exactly as submit_task_common would, but assigns output buffers from the -// bit-63 virtual address range and derives internal fanins from tensor-source -// classification — so no task slot, tensormap entry, fanin-pool entry, or upload -// is produced for the node. The resulting Definition is later attached to the -// outer GRAPH shells already submitted by the main thread. The returned -// TaskOutputTensors borrow the node's own tensor storage; moving the node into -// recording.nodes keeps those addresses valid because the inner buffer is -// transferred, not copied. -TaskOutputTensors graph_record_submit_node( +// Record one in-graph task while recording, without consuming a task-table +// slot. Builds the task's metadata and materialized outputs exactly as +// submit_task_common would, but assigns output buffers from the bit-63 virtual +// address range and derives internal fanins from tensor-source classification — so +// no task-table slot, tensormap entry, fanin-pool entry, or upload is produced for +// it. The resulting Definition is later attached to the outer GRAPH shells already +// submitted by the main thread. The returned TaskOutputTensors point into the +// recording's tensor pool, which is allocated at the cap and never grows, so they +// stay valid for the rest of the recording. +TaskOutputTensors graph_record_submit_in_graph_task( OrchestratorState *orch, const CoreTaskArgs &args, ActiveMask active_mask, TaskAttrs task_attrs, int32_t aic_kernel_id, int32_t aiv0_kernel_id, int32_t aiv1_kernel_id ) { @@ -2092,16 +2094,16 @@ TaskOutputTensors graph_record_submit_node( TaskOutputTensors result; GraphRecording &recording = *active_graph_recording(orch); - const size_t node_index = recording.node_count; + const size_t task_index = recording.task_count; // A recorded task lives in the IN_GRAPH id space, so an id the body hands // around says which of the two kinds of thing it names without any arithmetic: // an IN_GRAPH id is a task of this body, indexed by its low field; a GLOBAL id is // a task submitted before the Graph, which nothing in the body may depend on. const TaskId task_id = - simpler::hbg::make_in_graph_task(GRAPH_RECORD_NO_OWNING_GRAPH, static_cast(node_index)); + simpler::hbg::make_in_graph_task(GRAPH_RECORD_NO_OWNING_GRAPH, static_cast(task_index)); result.set_task_id(task_id); - if (node_index >= GRAPH_MAX_NODES || args.has_error) { + if (task_index >= MAX_IN_GRAPH_TASKS || args.has_error) { recording.unsupported = true; } @@ -2116,61 +2118,59 @@ TaskOutputTensors graph_record_submit_node( const uintptr_t packed_base_addr = GRAPH_RECORD_VIRTUAL_BASE + recording.next_virtual_offset; recording.next_virtual_offset += aligned_output; - // The node is filled in place, in the slot it will keep. Reusing the slot is what - // keeps its `tensors` buffer across recordings; reset() puts the rest of the slot - // back to a fresh node's state. Growing `nodes` moves the slots, but a slot's - // borrowed output addresses live in that heap buffer and a move only transfers its - // pointer, so they stay valid -- the same reason the old push_back(std::move(node)) - // was sound. - if (node_index >= recording.nodes.size()) recording.nodes.emplace_back(); - GraphRecordedNode &node = recording.nodes[node_index]; - node.reset(); - node.kernel_ids[static_cast(SubtaskSlot::AIC)] = aic_kernel_id; - node.kernel_ids[static_cast(SubtaskSlot::AIV0)] = aiv0_kernel_id; - node.kernel_ids[static_cast(SubtaskSlot::AIV1)] = aiv1_kernel_id; - node.active_mask = active_mask; - node.task_attrs = task_attrs; - node.task_attrs.set_early_resolve(false); - node.logical_block_num = args.launch_spec.block_num(); + // The task is filled in place, in the slot it will keep, and reset() puts the rest of + // the slot back to a freshly recorded task's state. An over-cap body grows `tasks`, + // which moves the slots, but the addresses handed to the caller live in the recording's + // tensor pool rather than in a slot, so a move cannot invalidate them. + if (task_index >= recording.tasks.size()) recording.tasks.emplace_back(); + RecordedInGraphTask &task = recording.tasks[task_index]; + task.reset(); + task.kernel_ids[static_cast(SubtaskSlot::AIC)] = aic_kernel_id; + task.kernel_ids[static_cast(SubtaskSlot::AIV0)] = aiv0_kernel_id; + task.kernel_ids[static_cast(SubtaskSlot::AIV1)] = aiv1_kernel_id; + task.active_mask = active_mask; + task.task_attrs = task_attrs; + task.task_attrs.set_early_resolve(false); + task.logical_block_num = args.launch_spec.block_num(); // Mirror prepare_task's contract: block_num must be positive and the subtask // count must fit int16_t. An out-of-contract value marks asynchronous // recording unsupported and makes commit fail-fast, rather than baking a // truncated or negative count into the cached Definition (which the device - // would expand into a node that never completes). + // would expand into an in-graph task that never completes). const int32_t required_subtasks = - static_cast(node.logical_block_num) * __builtin_popcount(active_mask.core_mask()); - if (node.logical_block_num <= 0 || required_subtasks > std::numeric_limits::max()) { + static_cast(task.logical_block_num) * __builtin_popcount(active_mask.core_mask()); + if (task.logical_block_num <= 0 || required_subtasks > std::numeric_limits::max()) { recording.unsupported = true; - node.total_required_subtasks = 0; + task.total_required_subtasks = 0; } else { - node.total_required_subtasks = static_cast(required_subtasks); + task.total_required_subtasks = static_cast(required_subtasks); } - node.record_packed_base = packed_base_addr; - node.total_output_size = aligned_output; + task.record_packed_base = packed_base_addr; + task.total_output_size = aligned_output; // Build the tensor list exactly as TaskPayload::init: inputs/inouts copy // the caller's simpler::hbg::Tensor; outputs materialize from the create-info onto the - // scratch buffer and carry this node's owner id. + // scratch buffer and carry the recorded task's owner id. const int32_t tensor_count = args.tensor_count(); - // Claim this node's slice of the pool. The cursor is a pure bump, so slices abut and a + // Claim this task's slice of the pool. The cursor is a pure bump, so slices abut and a // body holds its tensors in the bytes they need; nothing is ever returned to it, since // the whole pool is reset by the next recording. - if (static_cast(recording.node_tensor_cursor) + static_cast(tensor_count) > - GRAPH_RECORD_NODE_TENSOR_POOL_ELEMS) { + if (static_cast(recording.task_tensor_cursor) + static_cast(tensor_count) > + GRAPH_RECORD_TENSOR_POOL_ELEMS) { recording.unsupported = true; return result; } - node.tensor_offset = recording.node_tensor_cursor; - node.tensor_count = static_cast(tensor_count); - recording.node_tensor_cursor += static_cast(tensor_count); - simpler::hbg::Tensor *node_tensors = recording.node_tensors(node); + task.tensor_offset = recording.task_tensor_cursor; + task.tensor_count = static_cast(tensor_count); + recording.task_tensor_cursor += static_cast(tensor_count); + simpler::hbg::Tensor *task_tensors = recording.task_tensors(task); // Value-initialized before the fill, not merely claimed: the slice holds whatever the // previous body left in it, and simpler::hbg::Tensor::init_from writes strides only up // to the new tensor's ndims, so a narrower tensor would inherit a wider one's trailing // strides. - std::fill_n(node_tensors, static_cast(tensor_count), simpler::hbg::Tensor{}); + std::fill_n(task_tensors, static_cast(tensor_count), simpler::hbg::Tensor{}); for (int32_t i = 0; i < tensor_count; ++i) { - simpler::hbg::Tensor &slot_tensor = node_tensors[static_cast(i)]; + simpler::hbg::Tensor &slot_tensor = task_tensors[static_cast(i)]; if (args.tag(i) != TensorArgType::OUTPUT) { slot_tensor.copy(args.tensor(i).ref()); } else { @@ -2182,39 +2182,39 @@ TaskOutputTensors graph_record_submit_node( } } // The addresses handed out here are into the pool, which never moves, so they stay - // valid for every later node of this recording. + // valid for the rest of this recording. for (int32_t i = 0; i < tensor_count; ++i) { - if (args.tag(i) == TensorArgType::OUTPUT) result.materialize_output(node_tensors[static_cast(i)]); + if (args.tag(i) == TensorArgType::OUTPUT) result.materialize_output(task_tensors[static_cast(i)]); } - node.scalar_offset = static_cast(recording.scalars.size()); - node.scalar_count = static_cast(args.scalar_count()); + task.scalar_offset = static_cast(recording.scalars.size()); + task.scalar_count = static_cast(args.scalar_count()); recording.scalars.insert(recording.scalars.end(), args.scalars(), args.scalars() + args.scalar_count()); #if SIMPLER_DFX - node.dump_metadata.dump_arg_mask = args.dump_arg_mask(); - node.dump_metadata.dump_arg_flags = args.dump_arg_index_ambiguous_mask(); - memcpy(node.dump_metadata.scalar_dtypes, args.scalar_dtypes(), args.scalar_count() * sizeof(uint8_t)); + task.dump_metadata.dump_arg_mask = args.dump_arg_mask(); + task.dump_metadata.dump_arg_flags = args.dump_arg_index_ambiguous_mask(); + memcpy(task.dump_metadata.scalar_dtypes, args.scalar_dtypes(), args.scalar_count() * sizeof(uint8_t)); #endif // Classify each scalar's source: a plain literal is static Definition data, // while a value copied from a boundary scalar is refreshed on replay. A // mutable tracked boundary scalar is not supported and falls back. - recording.scalar_sources.resize(static_cast(node.scalar_offset) + node.scalar_count); + recording.scalar_sources.resize(static_cast(task.scalar_offset) + task.scalar_count); for (int32_t i = 0; i < args.scalar_count(); ++i) { GraphRecordedScalarSourceRef source = graph_classify_scalar(recording, args, i); if (source.source == GraphRecordedScalarSource::INVALIDATED_BOUNDARY) recording.unsupported = true; - recording.scalar_sources[static_cast(node.scalar_offset) + static_cast(i)] = source; + recording.scalar_sources[static_cast(task.scalar_offset) + static_cast(i)] = source; } // Classify each tensor's source, then derive internal fanins from the // INTERNAL classifications plus any explicit internal dependency. - node.tensor_source_offset = static_cast(recording.tensor_sources.size()); - recording.tensor_sources.resize(static_cast(node.tensor_source_offset) + tensor_count); + task.tensor_source_offset = static_cast(recording.tensor_sources.size()); + recording.tensor_sources.resize(static_cast(task.tensor_source_offset) + tensor_count); for (int32_t i = 0; i < tensor_count; ++i) { // The out-pointer is used only for the duration of the call, so pointing - // it into the flat array is safe even though a later node grows that array. + // it into the flat array is safe even though a later task of this body grows it. if (!graph_classify_tensor( - recording, node, static_cast(node_index), node_tensors[static_cast(i)], - &recording.tensor_sources[static_cast(node.tensor_source_offset) + static_cast(i)] + recording, task, static_cast(task_index), task_tensors[static_cast(i)], + &recording.tensor_sources[static_cast(task.tensor_source_offset) + static_cast(i)] )) { recording.unsupported = true; } @@ -2226,18 +2226,18 @@ TaskOutputTensors graph_record_submit_node( // predicate creates no dependency here any more than it does on the ordinary // path: the caller declares one, and the explicit-dep loop below records it. // - // Gated on the recorded attribute, not on args: a kernel-less node never + // Gated on the recorded attribute, not on args: a kernel-less task never // dispatches, so submit_dummy_task and alloc_tensors drop the predicate the - // caller set. Reading args here instead would record a predicate the node's + // caller set. Reading args here instead would record a predicate the task's // own attribute denies, and materialize rejects a Definition whose two halves // disagree. - if (node.task_attrs.has_predicate()) { + if (task.task_attrs.has_predicate()) { const CoreTaskPredicate &pred = args.predicate(); GraphRecordedPredicate recorded; recorded.op = pred.op; recorded.target = pred.target; const simpler::hbg::Tensor *operand = pred.operand.tensor; - // OWN_OUTPUT would read the node's own output before the node runs, so it + // OWN_OUTPUT would read the task's own output before the task runs, so it // names no value the predicate could be evaluating. An index vector that // leaves the operand's extent is caught here too: materialize would // otherwise reject the baked offset on the device, where the failure is a @@ -2246,7 +2246,7 @@ TaskOutputTensors graph_record_submit_node( operand == nullptr ? 0 : operand->compute_flat_offset(pred.operand.indices, pred.operand.ndims); if (operand == nullptr || operand->ndims > MAX_TENSOR_DIMS || pred.operand.ndims > operand->ndims || flat_offset < operand->start_offset || flat_offset - operand->start_offset >= operand->extent_elem_cache || - !graph_classify_tensor(recording, node, static_cast(node_index), *operand, &recorded.source) || + !graph_classify_tensor(recording, task, static_cast(task_index), *operand, &recorded.source) || recorded.source.source == GraphRecordedTensorSource::OWN_OUTPUT) { recording.unsupported = true; } else { @@ -2254,31 +2254,31 @@ TaskOutputTensors graph_record_submit_node( recorded.elem_offset = flat_offset - operand->start_offset; recorded.elem_size = static_cast(get_element_size(operand->dtype)); } - node.predicate_index = static_cast(recording.predicates.size()); + task.predicate_index = static_cast(recording.predicates.size()); recording.predicates.push_back(recorded); } - node.fanin_offset = static_cast(recording.internal_fanins.size()); - // Dedup within this node's own range: the flat array's earlier entries belong - // to earlier nodes. - auto add_fanin = [&recording, &node](size_t producer) { - const auto begin = recording.internal_fanins.begin() + node.fanin_offset; + task.fanin_offset = static_cast(recording.internal_fanins.size()); + // Dedup within this task's own range: the flat array's earlier entries belong + // to the body's earlier tasks. + auto add_fanin = [&recording, &task](size_t producer) { + const auto begin = recording.internal_fanins.begin() + task.fanin_offset; if (std::find(begin, recording.internal_fanins.end(), producer) == recording.internal_fanins.end()) { recording.internal_fanins.push_back(producer); } }; for (uint32_t i = 0; i < static_cast(tensor_count); ++i) { - const GraphRecordedTensorSourceRef &source = recording.tensor_sources[node.tensor_source_offset + i]; + const GraphRecordedTensorSourceRef &source = recording.tensor_sources[task.tensor_source_offset + i]; if (source.source == GraphRecordedTensorSource::INTERNAL) add_fanin(source.source_index); } // Inferred hazards, on the same terms as the ordinary path. The loop above only - // names the node that ALLOCATED each buffer; every write-then-read through a + // names the in-graph task that ALLOCATED each buffer; every write-then-read through a // buffer someone else allocated — an alloc_tensors output written in place // with add_inout, or a view of a boundary tensor — needs the last-writer // lookup compute_task_fanin performs. Running the very same function against // the recording's own map is what keeps a Definition's edge set equal to the - // one the body gets when its tasks are submitted individually. + // one the body gets when the ordinary path submits its tasks one at a time. // // Producers outside the recording window are dropped: they are reached // through boundary tensors, and the outer Graph shell already carries those @@ -2292,29 +2292,29 @@ TaskOutputTensors graph_record_submit_node( args.explicit_deps_data(), }; const bool manual_scope = recording.in_manual_scope(); - if (!recording.storage_ready || node_index >= GRAPH_MAX_NODES) { + if (!recording.storage_ready || task_index >= MAX_IN_GRAPH_TASKS) { // An over-cap body is already abandoned, and its task ids have run past // the index field make_in_graph_task packs them into, so registering one // would key the map outside its task chains. recording.unsupported = true; } else if (recording.tensor_map.free_entries() < count_registrable_outputs(dep_inputs, manual_scope)) { - // Recording one more node would assert inside new_entry(). Abandon the + // Recording one more task would assert inside new_entry(). Abandon the // Definition instead, so the run fails by name at graph_commit rather // than on a hard assert here. LOG_WARN( - "[GraphExecution] recording hazard map exhausted at node %zu (%d entries); Graph abandoned", node_index, - GRAPH_RECORD_TENSORMAP_POOL_SIZE + "[GraphExecution] recording hazard map exhausted at in-graph task %zu (%d entries); Graph abandoned", + task_index, GRAPH_RECORD_TENSORMAP_POOL_SIZE ); recording.unsupported = true; } else { - auto emit_inferred = [&add_fanin, node_index](TaskId producer) -> bool { + auto emit_inferred = [&add_fanin, task_index](TaskId producer) -> bool { // A GLOBAL producer is a task submitted before the Graph. The outer shell // was submitted through the ordinary path against this same boundary, so // its own fanin already orders the whole body behind that task and the // Definition carries no edge of its own. if (simpler::hbg::is_global_task(producer)) return true; const uint32_t producer_index = simpler::hbg::task_local_id(producer); - if (producer_index < static_cast(node_index)) { + if (producer_index < static_cast(task_index)) { add_fanin(static_cast(producer_index)); } return true; @@ -2343,7 +2343,7 @@ TaskOutputTensors graph_record_submit_node( continue; } const uint32_t dep_index = simpler::hbg::task_local_id(dep); - if (dep_index >= static_cast(node_index)) { + if (dep_index >= static_cast(task_index)) { // A task of this body that is not yet recorded: the Definition's edges are // acyclic by construction, so a forward reference cannot be expressed. recording.unsupported = true; @@ -2352,22 +2352,21 @@ TaskOutputTensors graph_record_submit_node( add_fanin(static_cast(dep_index)); } - node.fanin_count = static_cast(recording.internal_fanins.size() - node.fanin_offset); - if (node.record_packed_base != 0 && node.total_output_size != 0 && - node.total_output_size <= UINTPTR_MAX - node.record_packed_base) { - const uintptr_t begin = node.record_packed_base; - const uintptr_t end = begin + node.total_output_size; + task.fanin_count = static_cast(recording.internal_fanins.size() - task.fanin_offset); + if (task.record_packed_base != 0 && task.total_output_size != 0 && + task.total_output_size <= UINTPTR_MAX - task.record_packed_base) { + const uintptr_t begin = task.record_packed_base; + const uintptr_t end = begin + task.total_output_size; // The sorted-and-disjoint property the lookup depends on, checked rather // than assumed: a mid-recording heap rollback would break it, and today - // the only rollback is graph_end's, after every node is recorded. + // the only rollback is graph_end's, after every task of the body is recorded. always_assert(recording.output_ranges.empty() || recording.output_ranges.back().end <= begin); - recording.output_ranges.push_back({begin, end, static_cast(node_index)}); + recording.output_ranges.push_back({begin, end, static_cast(task_index)}); } - // Published last, as push_back(std::move(node)) used to be: until this advances, the - // slot is not part of the recording, so nothing that scans the recorded nodes can see - // the node being built. - recording.node_count = node_index + 1; - ORCH_PHASE_END(HostPhaseKind::OrchRecordNode, task_id.raw); + // Published last: until this advances, the slot is not part of the recording, so + // nothing that scans the recorded tasks can see the task being built. + recording.task_count = task_index + 1; + ORCH_PHASE_END(HostPhaseKind::OrchRecordInGraphTask, task_id.raw); return result; } @@ -2586,7 +2585,7 @@ bool OrchestratorState::graph_end() { } const bool built = layout.has_value() && graph_fill_definition(*recording, *layout, image); if (built) { - ORCH_PHASE_END(HostPhaseKind::OrchBuildDefinition, recording->node_count); + ORCH_PHASE_END(HostPhaseKind::OrchBuildDefinition, recording->task_count); } const GraphDefinition *header = built ? graph_record_definition(*state, record) : nullptr; if (header == nullptr) { @@ -2596,7 +2595,7 @@ bool OrchestratorState::graph_end() { return false; } LOG_DEBUG( - "[GraphExecution] define key=0x%llx nodes=%u bytes=%u", static_cast(header->full_key), + "[GraphExecution] define key=0x%llx tasks=%u bytes=%u", static_cast(header->full_key), header->task_count, header->total_bytes ); bool ready = false; @@ -2741,7 +2740,7 @@ TaskOutputTensors OrchestratorState::submit_task(const MixedKernels &mixed_kerne } if (active_graph_recording(orch) != nullptr) { - return graph_record_submit_node( + return graph_record_submit_in_graph_task( orch, args, active_mask, task_attrs, normalized.aic_kernel_id, normalized.aiv0_kernel_id, normalized.aiv1_kernel_id ); @@ -2784,7 +2783,7 @@ TaskOutputTensors OrchestratorState::submit_dummy_task(const CoreTaskArgs &args) task_attrs.set_timing_slot(args.task_timing_slot()); if (active_graph_recording(orch) != nullptr) { - return graph_record_submit_node( + return graph_record_submit_in_graph_task( orch, args, ActiveMask{}, task_attrs, INVALID_KERNEL_ID, INVALID_KERNEL_ID, INVALID_KERNEL_ID ); } @@ -2832,12 +2831,12 @@ TaskOutputTensors OrchestratorState::alloc_tensors(const CoreTaskArgs &args) { return TaskOutputTensors{}; } - // A Graph body may allocate. The allocation records as a kernel-less node — - // the same shape submit_dummy_task records — and replay reserves the - // intermediate heap for every internal node anyway, so the outputs land at + // A Graph body may allocate. The allocation records as a kernel-less in-graph + // task — the same shape submit_dummy_task records — and replay reserves the + // intermediate heap for every in-graph task anyway, so the outputs land at // addresses the replayed Definition derives for itself. if (active_graph_recording(orch) != nullptr) { - return graph_record_submit_node( + return graph_record_submit_in_graph_task( orch, args, ActiveMask{}, TaskAttrs{}, INVALID_KERNEL_ID, INVALID_KERNEL_ID, INVALID_KERNEL_ID ); } diff --git a/src/a5/runtime/host_build_graph/runtime/runtime_types.h b/src/a5/runtime/host_build_graph/runtime/runtime_types.h index 6fe3a8ab80..eb7a6b5a9b 100644 --- a/src/a5/runtime/host_build_graph/runtime/runtime_types.h +++ b/src/a5/runtime/host_build_graph/runtime/runtime_types.h @@ -109,7 +109,7 @@ // base before the image travels. Nothing dereferences an address in this window. inline constexpr uint64_t HEAP_VIRTUAL_BASE = 1ULL << 62; -// Base of the address range Graph recording hands to an internal node's packed +// Base of the address range Graph recording hands to an in-graph task's packed // outputs. Recorded addresses are never dereferenced: they exist so // graph_classify_tensor can tell an internal producer's output from a boundary // tensor by address-range containment alone, and the Definition stores them as @@ -206,11 +206,20 @@ struct TaskAllocResult { bool failed() const { return task_id < 0; } }; +/** + * What a task is, independent of where it belongs. + * + * KERNEL and DUMMY are leaves: KERNEL dispatches to cores, DUMMY carries only + * dependencies. GRAPH is a container — a shell that expands into its own body of + * tasks and completes when they all have. + * + * Membership is not a kind: a task inside a Graph body is an ordinary KERNEL or + * DUMMY, and `graph_context` names the Graph it belongs to. + */ enum class TaskKind : uint8_t { KERNEL = 0, DUMMY = 1, GRAPH = 2, - GRAPH_NODE = 3, }; struct OutputLayout { @@ -316,7 +325,7 @@ struct TaskPayload { // fanin holds flat position-independent producer local task ids. A producer is // named by its local id alone, so no per-edge indirection is stored. Scanned by // classify_fanin_state against the shared-memory completion_flags. Hard-capped at - // CHIP_MAX_FANIN (no dep-pool spill). Unbound on a Graph node, whose + // CHIP_MAX_FANIN (no dep-pool spill). Unbound on an in-graph task, whose // dependencies live in the Definition's fanin CSR instead. simpler::hbg::SelfRelativePtr tensors; simpler::hbg::SelfRelativePtr scalars; @@ -395,7 +404,7 @@ struct TaskPayload { * Point this payload's three argument regions at pool-resident storage. Must run * before prefetch() and init(), which dereference them. * - * A Graph node passes nullptr for fanin: its dependencies come from the + * An in-graph task passes nullptr for fanin: its dependencies come from the * Definition's CSR, so the region does not exist and fanin_count stays 0. */ void bind_regions(simpler::hbg::Tensor *tensor_region, uint64_t *scalar_region, int32_t *fanin_region) { @@ -598,9 +607,27 @@ struct alignas(64) ChipTaskSlotState { // ranges through claim_block_range(). std::atomic next_block_idx{0}; - // Graph scheduling metadata occupies the slot's tail padding. Ordinary - // Ordinary tasks keep the index invalid and the context null. - int32_t graph_node_index{-1}; + // Graph-only scheduling metadata occupies the former tail padding, keeping + // the slot state at one cache line and preserving the 40-byte descriptor + // ABI consumed by AICore. Readiness uses the shared intrusive wake-list + // fields above; this index identifies the task in the saved fanin CSR. + // Ordinary tasks leave both Graph fields -1/null. + int32_t in_graph_task_index{-1}; + // Graph membership, and which of the two Graph structs this points at is + // decided by task_kind rather than by anything stored here: + // + // nullptr an ordinary task, in no Graph + // != nullptr, task_kind == GRAPH the outer Graph task, pointing at the + // shared GraphDefinition until localize + // swaps in its GraphExecution + // != nullptr, task_kind != GRAPH an in-graph task, pointing at the + // GraphExecution it belongs to + // + // So every reader must test task_kind before casting, and complete_task + // routes on exactly that pair: a null context or a GRAPH kind takes the + // ordinary global fanout, anything else is counted against its Graph. A + // localize that fails puts this back to nullptr (scheduler_cold_path.cpp), + // so the outer task cannot be mistaken for an in-graph one. void *graph_context{nullptr}; int32_t claim_block_range(int32_t block_limit, int32_t max_count, int32_t &start) { @@ -637,8 +664,8 @@ struct alignas(64) ChipTaskSlotState { /** * Reset dynamic scheduling fields to their pristine values. Called once per - * slot as the orchestrator claims it in prepare_task, and again as a Graph - * node's storage is materialized — whole-graph-resident hbg has no + * slot as the orchestrator claims it in prepare_task, and again as an + * in-graph task's storage is materialized — whole-graph-resident hbg has no * execution-time slot recycle. Skips payload/task (bound once) and * task_state (the orchestrator sets PENDING when it populates the slot). * wake_list_head starts nullptr (open for registration), NOT SENTINEL. @@ -649,7 +676,7 @@ struct alignas(64) ChipTaskSlotState { any_subtask_deferred.store(false, std::memory_order_relaxed); completed_subtasks.store(0, std::memory_order_relaxed); next_block_idx.store(0, std::memory_order_relaxed); - graph_node_index = -1; + in_graph_task_index = -1; graph_context = nullptr; task_kind = TaskKind::KERNEL; // Note: active_mask and task_attrs are per-submit-constant fields diff --git a/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler.h b/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler.h index 7c9853cec0..bd76d2b2d2 100644 --- a/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler.h +++ b/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler.h @@ -902,12 +902,12 @@ struct SchedulerState { // Scheduler polling only chooses which already-wired producer a consumer // waits on at this instant; it never recomputes producer relationships. int32_t graph_first_unmet_producer(const GraphExecution &execution, const ChipTaskSlotState &consumer) const { - const uint32_t node_index = static_cast(consumer.graph_node_index); - const uint32_t begin = execution.fanin_offsets[node_index]; - const uint32_t end = execution.fanin_offsets[node_index + 1]; + const uint32_t task_index = static_cast(consumer.in_graph_task_index); + const uint32_t begin = execution.fanin_offsets[task_index]; + const uint32_t end = execution.fanin_offsets[task_index + 1]; for (uint32_t edge = begin; edge < end; ++edge) { const uint16_t producer_index = execution.fanin_indices[edge]; - const ChipTaskSlotState &producer = execution.node_at(producer_index).slot; + const ChipTaskSlotState &producer = execution.task_at(producer_index).slot; if (producer.task_state.load(std::memory_order_acquire) != CHIP_TASK_COMPLETED) { return static_cast(producer_index); } @@ -935,7 +935,7 @@ struct SchedulerState { push_ready_routed(consumer); return; } - producer = &execution.node_at(unmet_producer).slot; + producer = &execution.task_at(unmet_producer).slot; } } @@ -948,7 +948,7 @@ struct SchedulerState { if (unmet_producer < 0) { push_ready_routed(waiter); } else { - register_graph_wake(execution, &execution.node_at(unmet_producer).slot, waiter); + register_graph_wake(execution, &execution.task_at(unmet_producer).slot, waiter); } consumers_rescanned++; waiter = next; @@ -964,7 +964,7 @@ struct SchedulerState { // here — they reach the ready queue through their producers' wake list. int32_t graph_route_ready_roots(GraphExecution &execution) { if (execution.outer_slot == nullptr || !graph_execution_external_ready(execution)) return 0; - const int32_t published = execution.published_nodes.load(std::memory_order_acquire); + const int32_t published = execution.published_tasks.load(std::memory_order_acquire); int32_t routed = 0; while (true) { int32_t i = execution.route_cursor.load(std::memory_order_relaxed); @@ -975,33 +975,33 @@ struct SchedulerState { continue; } if (execution.fanin_offsets[i] == execution.fanin_offsets[i + 1]) { - push_ready_routed(&execution.node_at(i).slot); + push_ready_routed(&execution.task_at(i).slot); routed++; } } return routed; } - // Register each newly materialized node [first, last) on its first unmet + // Register each newly materialized in-graph task [first, last) on its first unmet // producer (or route it immediately when every producer already completed), // publish the range for routing, and route any roots the external gate now // admits. Runs single-owner per graph via the prepare-queue slot, so the // range never overlaps another thread's. register_graph_wake and // graph_first_unmet_producer are safe against a producer completing - // concurrently, which is what lets a node dispatch before the whole graph is + // concurrently, which is what lets a task dispatch before the whole graph is // materialized. void graph_incremental_publish(GraphExecution &execution, int32_t first, int32_t last) { for (int32_t i = first; i < last; ++i) { if (execution.fanin_offsets[i] == execution.fanin_offsets[i + 1]) continue; // root - ChipTaskSlotState &node = execution.node_at(i).slot; - const int32_t unmet = graph_first_unmet_producer(execution, node); + ChipTaskSlotState &task = execution.task_at(i).slot; + const int32_t unmet = graph_first_unmet_producer(execution, task); if (unmet < 0) { - push_ready_routed(&node); + push_ready_routed(&task); } else { - register_graph_wake(execution, &execution.node_at(unmet).slot, &node); + register_graph_wake(execution, &execution.task_at(unmet).slot, &task); } } - execution.published_nodes.store(last, std::memory_order_release); + execution.published_tasks.store(last, std::memory_order_release); graph_route_ready_roots(execution); } @@ -1013,16 +1013,16 @@ struct SchedulerState { } GraphMaterializeResult prepare_graph_task( - ChipTaskSlotState &outer_slot, int32_t max_nodes = GRAPH_MATERIALIZE_SLICE_NODES, - int32_t *nodes_materialized = nullptr + ChipTaskSlotState &outer_slot, int32_t max_tasks = GRAPH_MATERIALIZE_SLICE_TASKS, + int32_t *tasks_materialized = nullptr ) { GraphExecution *execution = graph_execution_from_outer_slot(outer_slot); if (execution == nullptr) return GraphMaterializeResult::INVALID; - const int32_t before = execution->materialized_nodes; + const int32_t before = execution->materialized_tasks; const GraphMaterializeResult result = - graph_execution_materialize_slice(outer_slot, *execution, max_nodes, nodes_materialized); + graph_execution_materialize_slice(outer_slot, *execution, max_tasks, tasks_materialized); if (result == GraphMaterializeResult::PENDING || result == GraphMaterializeResult::PREPARED) { - graph_incremental_publish(*execution, before, execution->materialized_nodes); + graph_incremental_publish(*execution, before, execution->materialized_tasks); } if (result == GraphMaterializeResult::PREPARED && graph_execution_external_ready(*execution)) { activate_prepared_graph(*execution); @@ -1051,7 +1051,11 @@ struct SchedulerState { #endif ) { TaskCompletionOutcome outcome; - if (slot_state.task_kind != TaskKind::GRAPH_NODE) { + // A task in a Graph body retires into its execution's counters; everything else + // — including the outer GRAPH shell — is a task of the run and releases its + // fanout. graph_context is null for the common case, so this short-circuits + // before the kind is read. + if (slot_state.graph_context == nullptr || slot_state.task_kind == TaskKind::GRAPH) { #if SIMPLER_SCHED_PROFILING CompletionStats stats = on_task_complete(slot_state, thread_idx); outcome.fanout_edges = static_cast(stats.fanout_edges); @@ -1062,27 +1066,29 @@ struct SchedulerState { return outcome; } - GraphExecution *execution = graph_execution_from_slot(slot_state); - if (execution == nullptr || execution->definition == nullptr || execution->nodes == nullptr) { + // Membership is established by the branch above: graph_context names this task's + // execution, and the shell case has already returned. + GraphExecution *execution = static_cast(slot_state.graph_context); + if (execution->definition == nullptr || execution->tasks == nullptr) { outcome.error_code = SIMPLER_ERROR_INVALID_ARGS; return outcome; } - // Incremental activation routes a node before the graph reaches ACTIVE, so a - // node can legitimately complete while the graph is still MATERIALIZING or - // PREPARED. Only SUBMITTED (execution not yet bound) and COMPLETED - // (execution already retired) are invalid states for a node completion. + // Incremental activation routes an in-graph task before the graph reaches + // ACTIVE, so one can legitimately complete while the graph is still + // MATERIALIZING or PREPARED. Only SUBMITTED (execution not yet bound) and + // COMPLETED (execution already retired) are invalid states for such a completion. const GraphExecutionState graph_state = graph_execution_state(*execution); if (graph_state < GraphExecutionState::MATERIALIZING || graph_state > GraphExecutionState::ACTIVE) { outcome.error_code = SIMPLER_ERROR_INVALID_ARGS; return outcome; } - const int32_t saved_node_index = slot_state.graph_node_index; - if (saved_node_index < 0) { + const int32_t saved_task_index = slot_state.in_graph_task_index; + if (saved_task_index < 0) { outcome.error_code = SIMPLER_ERROR_INVALID_ARGS; return outcome; } - const uint32_t node_index = static_cast(saved_node_index); - if (node_index >= static_cast(execution->node_count)) { + const uint32_t task_index = static_cast(saved_task_index); + if (task_index >= static_cast(execution->task_count)) { outcome.error_code = SIMPLER_ERROR_INVALID_ARGS; return outcome; } @@ -1093,11 +1099,11 @@ struct SchedulerState { slot_state.mark_completed(); outcome.fanout_edges = drain_graph_wake_list(*execution, slot_state); - const bool graph_completed = graph_execution_complete_node(*execution); - graph_execution_retire_node(*execution); + const bool graph_completed = graph_execution_complete_in_graph_task(*execution); + graph_execution_retire_in_graph_task(*execution); if (!graph_completed) return outcome; - // Internal nodes count as zero stream tasks. The final node publishes + // Internal tasks count as zero stream tasks. The final in-graph task publishes // the outer task exactly once, waking external consumers and // contributing the one task the host actually submitted. if (execution->outer_slot != nullptr) { diff --git a/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_dispatch.cpp b/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_dispatch.cpp index 22cf302219..99ba55679e 100644 --- a/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_dispatch.cpp +++ b/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_dispatch.cpp @@ -1299,7 +1299,7 @@ int32_t SchedulerContext::resolve_and_dispatch(Runtime *runtime, int32_t thread_ // independently, then meet in GraphExecution::state. // // Keep this ahead of dummy/regular dispatch so a ready Graph can expose - // its root nodes without waiting for an otherwise unrelated dispatch + // its root tasks without waiting for an otherwise unrelated dispatch // pass. Limiting the work to one activation and one bounded prepare // slice per loop prevents a large definition from monopolizing a // scheduler thread. @@ -1328,9 +1328,9 @@ int32_t SchedulerContext::resolve_and_dispatch(Runtime *runtime, int32_t thread_ uint64_t graph_prepare_t0 = chip_swimlane_level_ >= ChipSwimlaneLevel::SCHED_PHASES ? get_sys_cnt_aicpu() : 0; #endif - int32_t nodes_materialized = 0; + int32_t tasks_materialized = 0; GraphMaterializeResult result = - sched_->prepare_graph_task(*prepare_slot, GRAPH_MATERIALIZE_SLICE_NODES, &nodes_materialized); + sched_->prepare_graph_task(*prepare_slot, GRAPH_MATERIALIZE_SLICE_TASKS, &tasks_materialized); if (result == GraphMaterializeResult::PENDING || result == GraphMaterializeResult::BUSY) { if (!sched_->push_graph_prepare(prepare_slot, prepare_task_id, thread_idx)) { fail_scheduler(runtime, thread_idx, SIMPLER_ERROR_READY_QUEUE_OVERFLOW); @@ -1340,7 +1340,7 @@ int32_t SchedulerContext::resolve_and_dispatch(Runtime *runtime, int32_t thread_ fail_scheduler(runtime, thread_idx, SIMPLER_ERROR_INVALID_ARGS); break; } - if (nodes_materialized > 0 || result == GraphMaterializeResult::PREPARED) { + if (tasks_materialized > 0 || result == GraphMaterializeResult::PREPARED) { made_progress = true; } #if SIMPLER_DFX @@ -1348,7 +1348,7 @@ int32_t SchedulerContext::resolve_and_dispatch(Runtime *runtime, int32_t thread_ uint64_t graph_prepare_t1 = get_sys_cnt_aicpu(); chip_swimlane_aicpu_record_graph_prepare( thread_idx, graph_prepare_t0, graph_prepare_t1, chip_swimlane.sched_loop_count, prepare_task_id, - static_cast(nodes_materialized) + static_cast(tasks_materialized) ); _t0_phase = graph_prepare_t1; } diff --git a/src/a5/runtime/host_build_graph/runtime/shared_memory.h b/src/a5/runtime/host_build_graph/runtime/shared_memory.h index 13d5117930..7af119c046 100644 --- a/src/a5/runtime/host_build_graph/runtime/shared_memory.h +++ b/src/a5/runtime/host_build_graph/runtime/shared_memory.h @@ -392,7 +392,7 @@ struct HeapRebase { // Translate one address the image carries. Anything below HEAP_VIRTUAL_BASE is a // real device address the caller owns — a boundary tensor, or an unset field left // at 0 — and is returned untouched. At or above it, the address came from the -// graph heap: a recorded node's outputs live in its Definition as offsets, so no +// graph heap: a recorded in-graph task's outputs live in its Definition as offsets, so no // Graph-recording address (>= GRAPH_RECORD_VIRTUAL_BASE) reaches the image, and // the committed-heap bound below rejects one rather than classifying by it. inline uint64_t rebased_heap_addr(uint64_t addr, const HeapRebase &rebase) noexcept { @@ -480,7 +480,7 @@ inline uint64_t compact_live_image( const auto *mirror_fanin = reinterpret_cast(mirror_base + from.fanin_pool); const auto *mirror_tensors = reinterpret_cast(mirror_base + from.tensor_pool); const auto *mirror_scalars = reinterpret_cast(mirror_base + from.scalar_pool); - // An unbound region stays unbound: a Graph node's payload never gets a fanin + // An unbound region stays unbound: an in-graph task's payload never gets a fanin // region, and its count is 0, so no consumer resolves it. A bound one is inside // its own mirror pool by construction — the only binder is a bump cursor on that // pool — and the translation below depends on it, so it is asserted rather than diff --git a/src/a5/runtime/host_build_graph/runtime/submit_types.h b/src/a5/runtime/host_build_graph/runtime/submit_types.h index e8be0426ae..cf46391850 100644 --- a/src/a5/runtime/host_build_graph/runtime/submit_types.h +++ b/src/a5/runtime/host_build_graph/runtime/submit_types.h @@ -136,9 +136,14 @@ class ActiveMask { uint8_t core_mask() const { return raw_ & 0x07u; } + // No subtask slot is active, so nothing dispatches to a core. Sole definition + // of "dummy": both ResourceShape::DUMMY below and TaskKind::DUMMY derive from + // it, so the two cannot disagree. + bool is_dummy() const { return core_mask() == 0; } + ResourceShape to_shape() const { + if (is_dummy()) return ResourceShape::DUMMY; uint8_t cmask = core_mask(); - if (cmask == 0) return ResourceShape::DUMMY; int bit_count = __builtin_popcount(cmask); if (bit_count >= 2) return ResourceShape::MIX; if (cmask & SUBTASK_MASK_AIC) return ResourceShape::AIC; diff --git a/src/a5/runtime/host_build_graph/runtime/tensormap.h b/src/a5/runtime/host_build_graph/runtime/tensormap.h index 76173bcbc5..336b53f8ba 100644 --- a/src/a5/runtime/host_build_graph/runtime/tensormap.h +++ b/src/a5/runtime/host_build_graph/runtime/tensormap.h @@ -569,7 +569,7 @@ struct ChipTensorMap { const int32_t task_slot = local_id; // A producer's low id field is a task chain index directly, so the id space a // caller inserts under has to be the one this map was dimensioned for: a - // whole-run map takes task capacity, a Graph recording's takes GRAPH_MAX_NODES. + // whole-run map takes task capacity, a Graph recording's takes MAX_IN_GRAPH_TASKS. debug_assert(task_slot >= 0 && task_slot < max_tasks); entry->producer_task_id = producer_task_id; diff --git a/src/a5/runtime/host_build_graph/runtime/types.h b/src/a5/runtime/host_build_graph/runtime/types.h index c6e113913a..f58ab8fb64 100644 --- a/src/a5/runtime/host_build_graph/runtime/types.h +++ b/src/a5/runtime/host_build_graph/runtime/types.h @@ -88,7 +88,7 @@ enum class ScopeMode : uint8_t { * LIFETIME — single-pass only: * Internally this class stores pointers into the submitting task's tensor * storage: the region named by TaskPayload::tensors for a plain submit, the - * GraphRecording node's tensors for a submit inside a Graph body. Both belong to + * GraphRecording in-graph task's tensors for a submit inside a Graph body. Both belong to * one orchestration pass, which the next bind rebuilds over the same bytes. * Therefore the TaskOutputTensors instance, the const simpler::hbg::Tensor& returned by * get_ref(), and any pointer derived from either MUST NOT outlive the @@ -696,7 +696,7 @@ inline constexpr uint32_t GRAPH_MAX_SCALAR_ARGS = 64; // Boundary arguments of a Graph. Sized independently of CoreTaskArgs because the // outer GRAPH payload carries the whole boundary, while materialize stages only -// one node's arguments at a time. The compact boundary values live in that +// one in-graph task's arguments at a time. The compact boundary values live in that // payload's argument-pool regions, so widening these caps costs pool bytes only // for Graphs that use them; TaskPayload itself stays fixed-size. using GraphTaskArgs = Arg; diff --git a/src/common/host_build_graph/docs/GRAPH_EXECUTION.md b/src/common/host_build_graph/docs/GRAPH_EXECUTION.md index 9dfc097c91..b0127901d9 100644 --- a/src/common/host_build_graph/docs/GRAPH_EXECUTION.md +++ b/src/common/host_build_graph/docs/GRAPH_EXECUTION.md @@ -1,13 +1,16 @@ # Graph Execution Graph Execution is available only in the `host_build_graph` runtime. A Graph is -a composite incore task: it is submitted and completed once like an AIC, AIV, -MIX, or SPMD task, but contains a recorded task DAG. +a composite incore task: it occupies one task window slot and completes once, +like any other task of the run, but contains a recorded DAG of in-graph tasks. It +is a container in the same sense an SPMD task is — SPMD expands one slot into +`logical_block_num` blocks, a Graph expands one slot into its recorded in-graph +tasks — where an AIC, AIV or MIX task is a leaf that dispatches straight to cores. Every invocation places exactly one `GRAPH` task in the host task window. On a first miss, the caller immediately submits an outer task shell keyed by Graph identity while a recording thread records the DAG off the ordinary submit path. Internal -submissions build host-only node metadata and assign output addresses from a +submissions build host-only in-graph task metadata and assign output addresses from a private bit-63 virtual range instead of consuming task-window slots or heap. Later calls for the same in-flight identity submit more shells without waiting for recording, and a call for a *different* identity opens its own recording on @@ -15,8 +18,8 @@ its own thread rather than waiting. At orchestration completion, the caller join every recording and fills each shell's heap range and Definition content hash. Cached invocations submit the same one `GRAPH` task directly — a cache hit never waits on a recording. In both cases the device Scheduler expands the saved -topology and dispatches the internal nodes; the Host Orchestrator never submits -those nodes as tasks of the run itself. +topology and dispatches the in-graph tasks; the Host Orchestrator never submits +them as tasks of the run itself. Boundary contracts are checked before an in-flight shell is accepted. Once a shell has entered the task/dependency sequence, an unsupported construct found @@ -85,13 +88,13 @@ There are no public `GraphArgs`, `GraphBindings`, `Patch`, or `ScalarRef` types. The boundary is represented by `CoreTaskArgs`. Boundary scalars are pass-through bindings. Forward them directly with -`node_args.add_scalar(args.scalar(i))` or `copy_scalars_from(args, i, count)` +`task_args.add_scalar(args.scalar(i))` or `copy_scalars_from(args, i, count)` so recording can retain their source indices. Ordinary C++ value transformations do not retain boundary provenance. Both -`node_args.add_scalar(args.scalar(i) + 1)` and copying `args.scalar(i)` into a +`task_args.add_scalar(args.scalar(i) + 1)` and copying `args.scalar(i)` into a local arithmetic variable before calling `add_scalar` produce an ordinary -static node scalar. That value is stored in the Definition, and later cache +static in-graph task scalar. That value is stored in the Definition, and later cache hits reuse the first invocation's value without a warning. The runtime cannot distinguish such a derived value from an intentional static literal after the C++ expression has produced a plain arithmetic value. Compute the derived value @@ -120,7 +123,7 @@ with the unmodified boundary value. - Boundary storage is caller-owned. `INPUT`, `INOUT`, `OUTPUT_EXISTING`, and `NO_DEP` are supported. A boundary `TensorCreateInfo` tagged `OUTPUT` is not. - Early-resolve hints apply while recording the first invocation. Replayed - internal nodes use the saved completion topology without the hint. + in-graph tasks use the saved completion topology without the hint. - A recorded task may depend on a Graph-external producer when that producer is the creator of a boundary ChipTensor. The outer Graph owns that dependency on replay; arbitrary cross-boundary explicit dependencies remain unsupported. @@ -128,7 +131,7 @@ with the unmodified boundary value. into an absolute GM address, which no Definition can hold, so the Definition stores the operand tensor's classified source plus the element index within it and materialize resolves the pair per execution. The operand may be a - boundary ChipTensor or another node's output; the predicate itself creates no + boundary ChipTensor or another in-graph task's output; the predicate itself creates no dependency, exactly as on the ordinary path, so the caller still declares one on the operand's producer. @@ -209,7 +212,7 @@ and is not part of the Graph key. Recording uses host-only C++ state: -- `std::vector` for nodes, tensors, scalars, fanins, and pending uploads; +- `std::vector` for in-graph tasks, tensors, scalars, fanins, and pending uploads; - `std::unordered_map` for the per-run Definition cache; - `std::unordered_map` for the recordings in flight, keyed by Graph identity and holding each entry by `std::unique_ptr`, guarded by a mutex and completion @@ -235,7 +238,7 @@ therefore leaves the shared task allocator unchanged. `graph_begin` computes the Graph identity before the body is recorded. On a cache miss, the calling thread allocates a zero-heap outer task shell, records its boundary dependency edges, and returns. A recording thread receives a deep -copy of the boundary arguments, records the internal nodes in the private virtual +copy of the boundary arguments, records the in-graph tasks in the private virtual address range, and builds and hashes the Definition. The first call waits only until that private job has been installed in the recorder queue; it does not wait for the operating system to schedule the thread or for `graph_prepare` to bind @@ -257,7 +260,7 @@ Growth happens inside the submission that needs it, so it lands on the submittin thread: a workload whose Definition count exceeds the prewarmed count pays a `pthread_create` (measured 32-74 us each) in the middle of its submission burst. Recording touches no shared allocator state and each recording classifies Tensor -sources only against its own nodes and its own boundary, so two recordings +sources only against its own in-graph tasks and its own boundary, so two recordings sharing the `GRAPH_RECORD_VIRTUAL_BASE` range cannot see each other's addresses. What serializes is only the per-identity rule: at most one recording per Graph key, which the keyed in-flight map enforces. @@ -280,7 +283,7 @@ state unrepresentable rather than merely unlikely. Calls for the same identity while recording is in flight follow the same shell submission path on the calling thread. Their task IDs and TensorMap producers therefore enter the ordinary program-order sequence while its recording thread is -executing `record_node` and `build_definition`. +executing `record_in_graph_task` and `build_definition`. **Ordinary submissions do not join the recorders either.** `rt_submit_task`, `rt_submit_dummy_task` and `alloc_tensors` proceed while any number of Definitions @@ -316,23 +319,23 @@ stopped and four recording threads running. With the barrier only at completion, the recorders rather than the submitter become the tail. Host phase records therefore show `graph_submit`, `submit_task` and -`alloc_tensors` on the main lane overlapping `record_node` and +`alloc_tensors` on the main lane overlapping `record_in_graph_task` and `build_definition` on the recording lanes. At `graph_end`, recording is compacted into one contiguous, pointer-free POD Definition. It contains: -- node order and AIC/AIV/MIX/SPMD kernel metadata; +- in-graph task order and AIC/AIV/MIX/SPMD kernel metadata; - `root_indices` plus both directions of the immutable topology: fanin CSR and fanout CSR; -- one packed-heap offset per node; -- each node's ChipTensor source: +- one packed-heap offset per in-graph task; +- each in-graph task's ChipTensor source: `BOUNDARY_EXACT`, `BOUNDARY_VIEW`, `INTERNAL`, or `OWN_OUTPUT`; - fixed scalar values plus boundary-scalar source indices; - fixed boundary signatures and alias representatives. The header also carries a content hash of the complete Definition image. The -device execution pool requires this hash, the Graph key, and the node count to +device execution pool requires this hash, the Graph key, and the in-graph task count to all match before reusing a resident Definition. A new run may record different metadata under the same function identity, so key-only reuse is not safe. @@ -342,7 +345,7 @@ Tensors use the fixed-width `GraphTensor` wire POD rather than the copy with no raw Host pointers and no relocation pass. Before materialization, the Scheduler recomputes the Definition content hash -and validates section ranges, topology indices, node heap offsets, the outer +and validates section ranges, topology indices, in-graph task heap offsets, the outer heap extent, ChipTensor metadata, and ChipTensor-source bounds. Invalid wire data is rejected before an offset participates in pointer arithmetic. @@ -356,8 +359,8 @@ For a cache hit, the Host Orchestrator: 1. validates the fixed boundary contract; 2. reserves one task-window slot; 3. reserves one heap block large enough for every internal intermediate, plus - the Definition's `execution_storage_bytes` for node storage and its argument - pools; + the Definition's `execution_storage_bytes` for in-graph task storage and its + argument pools; 4. computes only external fanin and boundary tensormap effects; 5. emits one outer `GRAPH` task; 6. stores boundary values in the outer task's ordinary compact argument pools. @@ -368,19 +371,19 @@ outer payload as a kernel payload; device localization reads the compact values directly, so boundary metadata does not need to expand to full `ChipTensor` records on either side of H2D. -Internal nodes consume no task-table slots. Their descriptor, payload, slot +In-graph tasks consume no task-table slots. Their descriptor, payload, slot state, and argument pools live in the tail of the outer `GRAPH` task's own heap block, -past `required_heap`: `[GraphExecution][GraphNodeStorage...][tensor pool][scalar pool]`. One +past `required_heap`: `[GraphExecution][InGraphTaskStorage...][tensor pool][scalar pool]`. One `TaskAllocator::alloc` covers both the packed outputs and this execution storage, so they are reclaimed together without a separate device allocation or release path. -A node payload holds no argument array of its own — it names each region by a delta, -like any other payload. Its pools are the last two regions of the execution storage, -sized by the Definition's `tensor_arg_count` / `scalar_arg_count` and indexed by the -node's own `tensor_offset` / `scalar_offset`, so a node's arguments occupy the same -span in the pool as in the Definition's arg table. There is no fanin region: node -dependencies come from the Definition's fanin CSR, so a node's `fanin_count` stays 0 -and its fanin delta unbound. +An in-graph task's payload holds no argument array of its own — it names each region by +a delta, like any other payload. Its pools are the last two regions of the execution +storage, sized by the Definition's `tensor_arg_count` / `scalar_arg_count` and indexed by +the task's own `tensor_offset` / `scalar_offset`, so its arguments occupy the same +span in the pool as in the Definition's arg table. There is no fanin region: in-graph +task dependencies come from the Definition's fanin CSR, so such a task's `fanin_count` +stays 0 and its fanin delta unbound. The Host computes the execution-storage size before allocating the outer task's heap. It points the outer slot's existing `graph_context` at the shared device @@ -389,7 +392,7 @@ other task's argument pools. The copied arena zone and compact shared-memory image travel in one H2D. During the parallel initial classify, the Scheduler constructs `GraphExecution` in the outer heap tail, binds it to that Definition and the outer payload, and replaces `graph_context` with the execution pointer. -Node storage remains untouched until bounded materialization begins. +In-graph task storage remains untouched until bounded materialization begins. ## Scheduler flow @@ -412,29 +415,29 @@ inside a run. A Graph is placed in two independent control flows: -- `graph_prepare_queue`: materialize the saved nodes even while external fanin - is still pending; +- `graph_prepare_queue`: materialize the saved in-graph tasks even while external + fanin is still pending; - `graph_ready_queue`: signal that the outer Graph's external fanin is ready. Core-owning Scheduler threads pop at most one item from each queue per loop. A -prepare call expands at most four nodes and requeues unfinished work, +prepare call expands at most four in-graph tasks and requeues unfinished work, interleaving Graph expansion with normal scheduling. Preparation and external readiness set two bits in one atomic activation gate. -Whichever operation sets the second bit activates the saved root nodes exactly -once. +Whichever operation sets the second bit activates the saved root in-graph tasks +exactly once. Internal dependency readiness borrows the completion-state polling idea, but dependency wiring remains an Orchestrator responsibility: - recording constructs both fanin and fanout CSR in the immutable Definition; -- materialization builds each node's runnable state from the Definition, +- materialization builds each in-graph task's runnable state from the Definition, resolving its Tensor addresses against the boundary image and its producers' packed windows; - materialization registers each non-root on one producer selected from its saved fanin CSR; -- a node's release/acquire `task_state` is its Graph-local completion flag, so - internal nodes need neither shared-memory completion flags nor task-table slots; +- an in-graph task's release/acquire `task_state` is its Graph-local completion flag, so + such tasks need neither shared-memory completion flags nor task-table slots; - producer completion closes and drains only its current wake-list rather than traversing the saved fanout CSR; - a woken consumer scans its saved fanin CSR and either enters its shape queue @@ -457,8 +460,8 @@ outer GRAPH -> final internal completion completes the outer GRAPH ``` -Internal nodes count as zero tasks of the run itself. The final node completes -the one outer Graph task, publishes that task's completion flag, wakes +In-graph tasks count as zero tasks of the run itself. The last one to complete +finishes the one outer Graph task, publishes that task's completion flag, wakes external consumers, and contributes one to the host-visible completion count. Localization or materialization failure is fail-fast: the Scheduler latches an @@ -486,11 +489,11 @@ builds: ChipTensor's creator; - an unclassifiable internal ChipTensor source, including a dispatch predicate's operand tensor; -- a dispatch predicate whose operand is the predicated node's own output; +- a dispatch predicate whose operand is the predicated in-graph task's own output; - a dispatch predicate whose index vector leaves the operand tensor's extent; - a boundary-derived scalar accessed through mutable `scalar()`; - runtime allocation inside the Graph body; -- more than 1024 internal nodes; +- more than 1024 in-graph tasks; - insufficient heap capacity while deferred shells are finalized. An AICPU execution-pool or materialization failure happens after the outer @@ -498,7 +501,7 @@ Graph has already been submitted. It therefore latches a Scheduler fatal error instead of falling back; leaving the outer task pending would otherwise wedge completion. -Explicit dependencies between recorded internal nodes are preserved when they +Explicit dependencies between recorded in-graph tasks are preserved when they are otherwise supported; ordinary ChipTensor dependencies are always preserved. ## DFX diff --git a/src/common/host_build_graph/graph_execution.cpp b/src/common/host_build_graph/graph_execution.cpp index 381fc260e2..b464f37e8f 100644 --- a/src/common/host_build_graph/graph_execution.cpp +++ b/src/common/host_build_graph/graph_execution.cpp @@ -21,25 +21,25 @@ namespace { GraphExecution *acquire_execution_storage( - uintptr_t storage_addr, size_t storage_bytes, int32_t node_count, uint32_t tensor_arg_count, + uintptr_t storage_addr, size_t storage_bytes, int32_t task_count, uint32_t tensor_arg_count, uint32_t scalar_arg_count ) { GraphExecutionStorageLayout layout{}; - // GraphNodeStorage, not GraphExecution: the node array's alignment is the widest - // the storage carries, and nodes_offset only rounds up relative to this base, so - // an under-aligned base would leave every alignas(64) node entry misaligned. - if (storage_addr == 0 || storage_addr % alignof(GraphNodeStorage) != 0 || - !graph_execution_storage_layout(node_count, tensor_arg_count, scalar_arg_count, &layout) || + // InGraphTaskStorage, not GraphExecution: the in-graph task array's alignment is the widest + // the storage carries, and tasks_offset only rounds up relative to this base, so + // an under-aligned base would leave every alignas(64) in-graph task entry misaligned. + if (storage_addr == 0 || storage_addr % alignof(InGraphTaskStorage) != 0 || + !graph_execution_storage_layout(task_count, tensor_arg_count, scalar_arg_count, &layout) || layout.total_bytes > storage_bytes) { return nullptr; } auto *execution = new (reinterpret_cast(storage_addr)) GraphExecution{}; - execution->node_count = node_count; - execution->remaining_nodes.store(node_count, std::memory_order_relaxed); + execution->task_count = task_count; + execution->remaining_tasks.store(task_count, std::memory_order_relaxed); auto *base = reinterpret_cast(execution); - execution->node_storage = reinterpret_cast(base + layout.nodes_offset); - execution->node_tensor_pool = reinterpret_cast(base + layout.tensors_offset); - execution->node_scalar_pool = reinterpret_cast(base + layout.scalars_offset); + execution->task_storage = reinterpret_cast(base + layout.tasks_offset); + execution->task_tensor_pool = reinterpret_cast(base + layout.tensors_offset); + execution->task_scalar_pool = reinterpret_cast(base + layout.scalars_offset); return execution; } @@ -64,9 +64,9 @@ bool bind_graph_topology(GraphExecution &execution) { // GRAPH_MAX_SCALAR_ARGS, not MAX_SCALAR_ARGS: this counts the scalars the // Graph BOUNDARY carries, which the recorder sizes with // GraphTaskArgs = Arg and the - // outer Graph payload hands it to GraphExecution, never through a node + // outer Graph payload hands it to GraphExecution, never through an in-graph task // payload. MAX_SCALAR_ARGS is the per-AICore-task cap (16) and applies to - // GraphNodeDefinition::scalar_count below, which is checked separately; using + // InGraphTaskDefinition::scalar_count below, which is checked separately; using // it here rejected every boundary wider than one kernel call could take. if (definition.boundary_scalar_count > GRAPH_MAX_SCALAR_ARGS) return false; const uint32_t *fanin_offsets = @@ -83,12 +83,12 @@ bool bind_graph_topology(GraphExecution &execution) { graph_definition_array(definition, definition.off_fanout_indices, definition.edge_count); const uint16_t *roots = graph_definition_array(definition, definition.off_root_indices, definition.root_count); - const GraphNodeDefinition *nodes = - graph_definition_array(definition, definition.off_nodes, definition.task_count); - const uint64_t *node_offsets = - graph_definition_array(definition, definition.off_node_offsets, definition.task_count); - if (fanin_offsets == nullptr || fanout_offsets == nullptr || roots == nullptr || nodes == nullptr || - node_offsets == nullptr || + const InGraphTaskDefinition *tasks = + graph_definition_array(definition, definition.off_in_graph_tasks, definition.task_count); + const uint64_t *in_graph_task_offsets = + graph_definition_array(definition, definition.off_in_graph_task_offsets, definition.task_count); + if (fanin_offsets == nullptr || fanout_offsets == nullptr || roots == nullptr || tasks == nullptr || + in_graph_task_offsets == nullptr || (definition.edge_count != 0 && (fanin_indices == nullptr || fanout_indices == nullptr)) || fanin_offsets[0] != 0 || fanout_offsets[0] != 0 || fanin_offsets[definition.task_count] != definition.edge_count || @@ -99,22 +99,22 @@ bool bind_graph_topology(GraphExecution &execution) { uint64_t required_heap = 0; constexpr uint8_t VALID_ACTIVE_MASK = (1U << SUBTASK_SLOT_COUNT) - 1U; for (uint32_t i = 0; i < definition.task_count; ++i) { - const GraphNodeDefinition &node = nodes[i]; - if (node_offsets[i] != required_heap || node.total_output_size < 0 || node.tensor_count < 0 || - node.tensor_count > MAX_TENSOR_ARGS || node.scalar_count < 0 || node.scalar_count > MAX_SCALAR_ARGS || - node.tensor_offset > definition.tensor_arg_count || - static_cast(node.tensor_count) > definition.tensor_arg_count - node.tensor_offset || - node.scalar_offset > definition.scalar_arg_count || - static_cast(node.scalar_count) > definition.scalar_arg_count - node.scalar_offset || - (node.active_mask & ~VALID_ACTIVE_MASK) != 0 || node.logical_block_num <= 0 || - node.total_required_subtasks < 0) { + const InGraphTaskDefinition &task = tasks[i]; + if (in_graph_task_offsets[i] != required_heap || task.total_output_size < 0 || task.tensor_count < 0 || + task.tensor_count > MAX_TENSOR_ARGS || task.scalar_count < 0 || task.scalar_count > MAX_SCALAR_ARGS || + task.tensor_offset > definition.tensor_arg_count || + static_cast(task.tensor_count) > definition.tensor_arg_count - task.tensor_offset || + task.scalar_offset > definition.scalar_arg_count || + static_cast(task.scalar_count) > definition.scalar_arg_count - task.scalar_offset || + (task.active_mask & ~VALID_ACTIVE_MASK) != 0 || task.logical_block_num <= 0 || + task.total_required_subtasks < 0) { return false; } for (int32_t slot = 0; slot < SUBTASK_SLOT_COUNT; ++slot) { - const bool active = (node.active_mask & (1U << slot)) != 0; - if (active != (node.kernel_id[slot] != INVALID_KERNEL_ID)) return false; + const bool active = (task.active_mask & (1U << slot)) != 0; + if (active != (task.kernel_id[slot] != INVALID_KERNEL_ID)) return false; } - const uint64_t output_bytes = CHIP_ALIGN_UP(static_cast(node.total_output_size), CHIP_ALIGN_SIZE); + const uint64_t output_bytes = CHIP_ALIGN_UP(static_cast(task.total_output_size), CHIP_ALIGN_SIZE); if (output_bytes > definition.required_heap - required_heap) return false; required_heap += output_bytes; } @@ -169,13 +169,13 @@ GraphDefinition *graph_definition_object_framed(GraphDefinitionHeader &header) { // Rebind one Definition tensor template onto this execution. A BOUNDARY_* ref // takes the invocation's boundary tensor; an INTERNAL / OWN_OUTPUT ref takes the -// producer node's materialized output base. `node_index` is the consuming node, -// which bounds a producer reference to a node that is already constructed. +// producer in-graph task's materialized output base. `task_index` is the consuming +// task, which bounds a producer reference to a task that is already constructed. // Returns false when the ref addresses no valid source — the Definition is then // invalid, since every ref is written by the recorder from a classified source. bool graph_rebind_tensor( - const GraphExecution &execution, const GraphNodeDefinition *nodes, const uint64_t *node_offsets, - const GraphTensor &tensor_template, const GraphTensorSourceRef &ref, int32_t node_index, GraphTensor *rebound_out + const GraphExecution &execution, const InGraphTaskDefinition *tasks, const uint64_t *in_graph_task_offsets, + const GraphTensor &tensor_template, const GraphTensorSourceRef &ref, int32_t task_index, GraphTensor *rebound_out ) { GraphTensor rebound = tensor_template; if (!graph_tensor_wire_valid(rebound)) return false; @@ -195,17 +195,17 @@ bool graph_rebind_tensor( } else if (ref.source == static_cast(GraphTensorSource::INTERNAL) || ref.source == static_cast(GraphTensorSource::OWN_OUTPUT)) { const bool own_output = ref.source == static_cast(GraphTensorSource::OWN_OUTPUT); - const int32_t producer_index = own_output ? node_index : static_cast(ref.source_index); - if (producer_index < 0 || producer_index > node_index || (own_output && ref.source_index != node_index) || - (!own_output && producer_index == node_index)) { + const int32_t producer_index = own_output ? task_index : static_cast(ref.source_index); + if (producer_index < 0 || producer_index > task_index || (own_output && ref.source_index != task_index) || + (!own_output && producer_index == task_index)) { return false; } - TaskDescriptor &producer = execution.node_at(producer_index).task; - const uint64_t producer_bytes = static_cast(nodes[producer_index].total_output_size); + TaskDescriptor &producer = execution.task_at(producer_index).task; + const uint64_t producer_bytes = static_cast(tasks[producer_index].total_output_size); const uintptr_t producer_base = reinterpret_cast(producer.packed_buffer_base); if (ref.packed_offset > producer_bytes || rebound.buffer_size > producer_bytes - ref.packed_offset || ref.packed_offset > UINTPTR_MAX - producer_base || - ref.packed_offset > UINT64_MAX - node_offsets[producer_index]) { + ref.packed_offset > UINT64_MAX - in_graph_task_offsets[producer_index]) { return false; } rebound.buffer_addr = producer_base + ref.packed_offset; @@ -279,7 +279,7 @@ GraphExecution *graph_execution_localize(ChipTaskSlotState &outer_slot) { const GraphDefinition *definition = graph_definition_object_framed(*definition_header); TaskPayload &payload = *outer_slot.payload; if (definition == nullptr || definition->total_bytes == 0 || definition->task_count == 0 || - definition->task_count > GRAPH_MAX_NODES || + definition->task_count > MAX_IN_GRAPH_TASKS || payload.tensor_count != static_cast(definition->boundary_count) || payload.scalar_count != static_cast(definition->boundary_scalar_count) || (payload.tensor_count != 0 && payload.tensor_data() == nullptr) || @@ -307,7 +307,7 @@ GraphExecution *graph_execution_localize(ChipTaskSlotState &outer_slot) { execution->boundary_scalars = payload.scalar_data(); execution->boundary_scalar_count = static_cast(payload.scalar_count); if (!bind_graph_topology(*execution)) { - execution->retired_nodes.store(execution->node_count, std::memory_order_relaxed); + execution->retired_tasks.store(execution->task_count, std::memory_order_relaxed); graph_execution_mark_completed(*execution); return nullptr; } @@ -316,12 +316,12 @@ GraphExecution *graph_execution_localize(ChipTaskSlotState &outer_slot) { } GraphMaterializeResult graph_execution_materialize_slice( - ChipTaskSlotState &outer_slot, GraphExecution &execution, int32_t max_nodes, int32_t *nodes_materialized + ChipTaskSlotState &outer_slot, GraphExecution &execution, int32_t max_tasks, int32_t *tasks_materialized ) { - if (nodes_materialized != nullptr) *nodes_materialized = 0; + if (tasks_materialized != nullptr) *tasks_materialized = 0; if (outer_slot.task_kind != TaskKind::GRAPH || outer_slot.task == nullptr || - outer_slot.task->packed_buffer_base == nullptr || max_nodes <= 0 || execution.definition == nullptr || - execution.node_storage == nullptr) { + outer_slot.task->packed_buffer_base == nullptr || max_tasks <= 0 || execution.definition == nullptr || + execution.task_storage == nullptr) { return GraphMaterializeResult::INVALID; } @@ -343,22 +343,22 @@ GraphMaterializeResult graph_execution_materialize_slice( execution.materialize_busy.store(0, std::memory_order_release); return GraphMaterializeResult::BUSY; } - // Incremental activation reads producer slots through execution.nodes + // Incremental activation reads producer slots through execution.tasks // while the graph is still materializing, so publish the storage base - // once, before the first range. Topological node order guarantees every - // producer index a materialized node references is already constructed, + // once, before the first range. Topological task order guarantees every + // producer index a materialized task references is already constructed, // and materialize_busy serializes this with any concurrent slice. - execution.nodes = execution.node_storage; + execution.tasks = execution.task_storage; } else if (state != GraphExecutionState::MATERIALIZING) { execution.materialize_busy.store(0, std::memory_order_release); return GraphMaterializeResult::INVALID; } const GraphDefinition &definition = *execution.definition; - const GraphNodeDefinition *nodes = - graph_definition_array(definition, definition.off_nodes, definition.task_count); - const uint64_t *node_offsets = - graph_definition_array(definition, definition.off_node_offsets, definition.task_count); + const InGraphTaskDefinition *tasks = + graph_definition_array(definition, definition.off_in_graph_tasks, definition.task_count); + const uint64_t *in_graph_task_offsets = + graph_definition_array(definition, definition.off_in_graph_task_offsets, definition.task_count); const GraphTensor *definition_tensors = definition.tensor_arg_count == 0 ? nullptr : @@ -381,7 +381,7 @@ GraphMaterializeResult graph_execution_materialize_slice( definition.predicate_count == 0 ? nullptr : graph_definition_array(definition, definition.off_predicates, definition.predicate_count); - if (nodes == nullptr || node_offsets == nullptr || + if (tasks == nullptr || in_graph_task_offsets == nullptr || (definition.tensor_arg_count != 0 && (definition_tensors == nullptr || tensor_sources == nullptr)) || (definition.scalar_arg_count != 0 && (definition_scalars == nullptr || scalar_sources == nullptr)) || (definition.predicate_count != 0 && predicates == nullptr)) { @@ -389,14 +389,14 @@ GraphMaterializeResult graph_execution_materialize_slice( return GraphMaterializeResult::INVALID; } - const int32_t first = execution.materialized_nodes; - const int32_t last = std::min(execution.node_count, first + max_nodes); + const int32_t first = execution.materialized_tasks; + const int32_t last = std::min(execution.task_count, first + max_tasks); const uintptr_t outer_base = reinterpret_cast(outer_slot.task->packed_buffer_base); for (int32_t i = first; i < last; ++i) { - GraphNodeStorage *storage = &execution.node_at(i); - if (i >= execution.constructed_nodes) { - storage = new (storage) GraphNodeStorage; - execution.constructed_nodes++; + InGraphTaskStorage *storage = &execution.task_at(i); + if (i >= execution.constructed_tasks) { + storage = new (storage) InGraphTaskStorage; + execution.constructed_tasks++; } TaskDescriptor &task = storage->task; TaskPayload &payload = storage->payload; @@ -405,13 +405,13 @@ GraphMaterializeResult graph_execution_materialize_slice( task.task_id = simpler::hbg::make_in_graph_task( simpler::hbg::task_local_id(outer_slot.task->task_id), static_cast(i) ); - const GraphNodeDefinition &source = nodes[i]; - const uint64_t node_offset = node_offsets[i]; + const InGraphTaskDefinition &source = tasks[i]; + const uint64_t task_offset = in_graph_task_offsets[i]; const uint64_t output_bytes = CHIP_ALIGN_UP(static_cast(source.total_output_size), CHIP_ALIGN_SIZE); for (int k = 0; k < SUBTASK_SLOT_COUNT; ++k) task.kernel_id[k] = source.kernel_id[k]; - task.packed_buffer_base = reinterpret_cast(outer_base + node_offset); - task.packed_buffer_end = reinterpret_cast(outer_base + node_offset + output_bytes); + task.packed_buffer_base = reinterpret_cast(outer_base + task_offset); + task.packed_buffer_end = reinterpret_cast(outer_base + task_offset + output_bytes); slot.reset_for_reuse(); slot.task_state.store(CHIP_TASK_PENDING, std::memory_order_relaxed); @@ -420,8 +420,10 @@ GraphMaterializeResult graph_execution_materialize_slice( slot.task_attrs = TaskAttrs(source.task_attrs); slot.total_required_subtasks = source.total_required_subtasks; slot.logical_block_num = source.logical_block_num; - slot.graph_node_index = i; - slot.task_kind = TaskKind::GRAPH_NODE; + slot.in_graph_task_index = i; + // A task in a Graph body is an ordinary leaf, classified by the same rule as + // one submitted outside a Graph. Its membership is carried by graph_context. + slot.task_kind = slot.active_mask.is_dummy() ? TaskKind::DUMMY : TaskKind::KERNEL; slot.graph_context = &execution; payload.tensor_count = source.tensor_count; payload.scalar_count = source.scalar_count; @@ -435,40 +437,40 @@ GraphMaterializeResult graph_execution_materialize_slice( execution.materialize_busy.store(0, std::memory_order_release); return GraphMaterializeResult::INVALID; } - // A node's arguments occupy the same span in this execution's pools as in the - // Definition's arg tables, so the region starts at the node's own offset. No + // A task's arguments occupy the same span in this execution's pools as in the + // Definition's arg tables, so the region starts at the task's own offset. No // fanin region: its dependencies come from the Definition's CSR, and // reset_graph_payload below keeps fanin_count at 0. payload.bind_regions( - execution.node_tensor_pool + source.tensor_offset, execution.node_scalar_pool + source.scalar_offset, + execution.task_tensor_pool + source.tensor_offset, execution.task_scalar_pool + source.scalar_offset, nullptr ); - simpler::hbg::Tensor *node_tensors = payload.tensor_data(); + simpler::hbg::Tensor *task_tensors = payload.tensor_data(); for (int32_t j = 0; j < source.tensor_count; ++j) { const uint32_t tensor_index = source.tensor_offset + static_cast(j); GraphTensor rebound; if (!graph_rebind_tensor( - execution, nodes, node_offsets, definition_tensors[tensor_index], tensor_sources[tensor_index], i, - &rebound + execution, tasks, in_graph_task_offsets, definition_tensors[tensor_index], + tensor_sources[tensor_index], i, &rebound )) { execution.materialize_busy.store(0, std::memory_order_release); return GraphMaterializeResult::INVALID; } execution.consumed_tensor_args++; - graph_tensor_unpack(rebound, &node_tensors[j]); + graph_tensor_unpack(rebound, &task_tensors[j]); } - uint64_t *node_scalars = payload.scalar_data(); + uint64_t *task_scalars = payload.scalar_data(); for (int32_t j = 0; j < source.scalar_count; ++j) { const uint32_t scalar_index = source.scalar_offset + static_cast(j); const GraphScalarSourceRef &ref = scalar_sources[scalar_index]; if (ref.source == static_cast(GraphScalarSource::STATIC_VALUE)) { - node_scalars[j] = definition_scalars[scalar_index]; + task_scalars[j] = definition_scalars[scalar_index]; } else if (ref.source == static_cast(GraphScalarSource::BOUNDARY)) { if (ref.source_index >= execution.boundary_scalar_count || execution.boundary_scalars == nullptr) { execution.materialize_busy.store(0, std::memory_order_release); return GraphMaterializeResult::INVALID; } - node_scalars[j] = execution.boundary_scalars[ref.source_index]; + task_scalars[j] = execution.boundary_scalars[ref.source_index]; } else { execution.materialize_busy.store(0, std::memory_order_release); return GraphMaterializeResult::INVALID; @@ -476,26 +478,26 @@ GraphMaterializeResult graph_execution_materialize_slice( } reset_graph_payload(payload); // The attribute bit and the predicate slot are written together by the - // recorder. A Definition where they disagree would either route the node + // recorder. A Definition where they disagree would either route the task // through a predicate the scheduler never reads, or leave a resolved // predicate that no dispatch consults. if (slot.task_attrs.has_predicate() != (source.predicate_slot != 0)) { execution.materialize_busy.store(0, std::memory_order_release); return GraphMaterializeResult::INVALID; } - // Resolved after the reset, which clears the predicate every node starts from. + // Resolved after the reset, which clears the predicate every task starts from. if (source.predicate_slot != 0) { const uint32_t predicate_index = static_cast(source.predicate_slot) - 1; GraphTensor operand; // OWN_OUTPUT is a valid source for a tensor arg but never for an - // operand: it would bind the predicate to the buffer this node has + // operand: it would bind the predicate to the buffer this task has // yet to write, so the dispatch decision would read whatever the heap // last held. The recorder refuses it; so does the image reader. if (predicate_index >= definition.predicate_count || predicates[predicate_index].operand_source.source == static_cast(GraphTensorSource::OWN_OUTPUT) || !graph_rebind_tensor( - execution, nodes, node_offsets, predicates[predicate_index].operand, + execution, tasks, in_graph_task_offsets, predicates[predicate_index].operand, predicates[predicate_index].operand_source, i, &operand ) || !graph_predicate_resolve(operand, predicates[predicate_index], &payload.predicate)) { @@ -504,15 +506,15 @@ GraphMaterializeResult graph_execution_materialize_slice( } } } - execution.materialized_nodes = last; - if (nodes_materialized != nullptr) *nodes_materialized = last - first; + execution.materialized_tasks = last; + if (tasks_materialized != nullptr) *tasks_materialized = last - first; - if (last < execution.node_count) { + if (last < execution.task_count) { execution.materialize_busy.store(0, std::memory_order_release); return GraphMaterializeResult::PENDING; } - // Every node's [tensor_offset, tensor_offset + tensor_count) range is bounds- + // Every task's [tensor_offset, tensor_offset + tensor_count) range is bounds- // checked on its own. This total additionally requires the ranges to account // for the whole tensor array, rejecting a Definition that under- or // over-consumes it. diff --git a/src/common/host_build_graph/graph_execution.h b/src/common/host_build_graph/graph_execution.h index 44174875d1..b2607b5f9a 100644 --- a/src/common/host_build_graph/graph_execution.h +++ b/src/common/host_build_graph/graph_execution.h @@ -22,12 +22,12 @@ #include "runtime_types.h" #include "tensor.h" -inline constexpr uint32_t GRAPH_MAX_NODES = 1024; +inline constexpr uint32_t MAX_IN_GRAPH_TASKS = 1024; static_assert( - GRAPH_MAX_NODES <= (1u << simpler::hbg::IN_GRAPH_TASK_INDEX_BITS), - "a node index must fit the low field of an IN_GRAPH task id" + MAX_IN_GRAPH_TASKS <= (1u << simpler::hbg::IN_GRAPH_TASK_INDEX_BITS), + "an in-graph task index must fit the low field of an IN_GRAPH task id" ); -inline constexpr int32_t GRAPH_MATERIALIZE_SLICE_NODES = 4; +inline constexpr int32_t GRAPH_MATERIALIZE_SLICE_TASKS = 4; enum class GraphTensorSource : uint8_t { BOUNDARY_EXACT = 0, @@ -79,7 +79,7 @@ struct GraphScalarSourceRef { uint8_t reserved; }; -// Wire representation of a node's dispatch predicate. The operand's absolute GM +// Wire representation of an in-graph task's dispatch predicate. The operand's absolute GM // address is not replay-invariant, so the Definition names the tensor the // operand element sits in plus its element offset within that tensor; // materialize rebinds the tensor for the execution and resolves the pair into @@ -97,15 +97,15 @@ struct GraphPredicate { uint8_t reserved[6]; }; -struct GraphNodeDefinition { +struct InGraphTaskDefinition { int32_t kernel_id[SUBTASK_SLOT_COUNT]; uint8_t active_mask; uint8_t task_attrs; int16_t logical_block_num; int16_t total_required_subtasks; - // One-based index into the Definition's predicate array; 0 means the node - // carries no dispatch predicate. Biased so that a zeroed GraphNodeDefinition - // is a valid predicate-free node. Predicated nodes are rare, so the + // One-based index into the Definition's predicate array; 0 means the task + // carries no dispatch predicate. Biased so that a zeroed InGraphTaskDefinition + // is a valid predicate-free task. Predicated tasks are rare, so the // predicates live in their own array rather than inline. uint16_t predicate_slot; int32_t tensor_count; @@ -170,17 +170,18 @@ struct GraphDefinition { uint32_t tensor_arg_count; uint32_t scalar_arg_count; uint32_t predicate_count; - // Bytes the GraphExecution header, node array and node argument pools need in - // the outer GRAPH task's heap tail. Invocation boundaries live in the outer - // task payload's compact argument-pool regions instead. + // Bytes the GraphExecution header, in-graph task array and in-graph task + // argument pools need in the outer Graph task's heap tail. Invocation + // boundaries live in the outer task payload's compact argument-pool regions + // instead. uint32_t execution_storage_bytes; uint32_t off_fanout_offsets; uint32_t off_fanout_indices; uint32_t off_fanin_offsets; uint32_t off_fanin_indices; uint32_t off_root_indices; - uint32_t off_node_offsets; - uint32_t off_nodes; + uint32_t off_in_graph_task_offsets; + uint32_t off_in_graph_tasks; uint32_t off_tensors; uint32_t off_tensor_sources; uint32_t off_scalars; @@ -195,8 +196,8 @@ static_assert(std::is_trivially_copyable_v); static_assert(std::is_standard_layout_v); static_assert(std::is_trivially_copyable_v); static_assert(std::is_standard_layout_v); -static_assert(std::is_trivially_copyable_v); -static_assert(std::is_standard_layout_v); +static_assert(std::is_trivially_copyable_v); +static_assert(std::is_standard_layout_v); static_assert(std::is_trivially_copyable_v); static_assert(std::is_standard_layout_v); static_assert(std::is_trivially_copyable_v); @@ -211,7 +212,7 @@ static_assert(std::is_standard_layout_v); // type that asked for more would make every one of those stores undefined, with // no diagnostic. static_assert( - alignof(GraphNodeDefinition) <= alignof(std::max_align_t) && alignof(GraphTensor) <= alignof(std::max_align_t) && + alignof(InGraphTaskDefinition) <= alignof(std::max_align_t) && alignof(GraphTensor) <= alignof(std::max_align_t) && alignof(GraphTensorSourceRef) <= alignof(std::max_align_t) && alignof(GraphScalarSourceRef) <= alignof(std::max_align_t) && alignof(GraphBoundarySignature) <= alignof(std::max_align_t) && @@ -312,13 +313,13 @@ enum class GraphMaterializeResult : uint8_t { PREPARED = 3, }; -struct alignas(64) GraphNodeStorage { +struct alignas(64) InGraphTaskStorage { TaskDescriptor task; ChipTaskSlotState slot; - // The payload carries its argument regions as deltas into pools past the node - // array, so its size is the same for every node and the slot names it by a delta + // The payload carries its argument regions as deltas into pools past the task + // array, so its size is the same for every task and the slot names it by a delta // from the slot's own address. Field order here therefore constrains nothing, and - // node_at strides the storage by this type. + // task_at strides the storage by this type. TaskPayload payload; }; @@ -331,27 +332,27 @@ struct GraphExecution { // per-submission gate object. std::atomic state{static_cast(GraphExecutionState::SUBMITTED)}; std::atomic materialize_busy{0}; - std::atomic remaining_nodes{0}; - std::atomic retired_nodes{0}; - // Incremental activation: nodes in [0, published_nodes) are fully + std::atomic remaining_tasks{0}; + std::atomic retired_tasks{0}; + // Incremental activation: tasks in [0, published_tasks) are fully // materialized and registered, so a route pass may consider them. route_cursor - // is the next such node index a route pass will claim; roots below it have + // is the next such task index a route pass will claim; roots below it have // been pushed to the ready queue exactly once. Both advance monotonically and // reset per (re)submission. - std::atomic published_nodes{0}; + std::atomic published_tasks{0}; std::atomic route_cursor{0}; - int32_t node_count{0}; - int32_t materialized_nodes{0}; - int32_t constructed_nodes{0}; + int32_t task_count{0}; + int32_t materialized_tasks{0}; + int32_t constructed_tasks{0}; uint32_t consumed_tensor_args{0}; ChipTaskSlotState *outer_slot{nullptr}; - GraphNodeStorage *nodes{nullptr}; - GraphNodeStorage *node_storage{nullptr}; - // This execution's node argument pools, in the storage tail past node_storage. - // Every node payload's tensor and scalar deltas point here; its pool position is + InGraphTaskStorage *tasks{nullptr}; + InGraphTaskStorage *task_storage{nullptr}; + // This execution's task argument pools, in the storage tail past task_storage. + // Every task payload's tensor and scalar deltas point here; its pool position is // the Definition's tensor_offset / scalar_offset. - simpler::hbg::Tensor *node_tensor_pool{nullptr}; - uint64_t *node_scalar_pool{nullptr}; + simpler::hbg::Tensor *task_tensor_pool{nullptr}; + uint64_t *task_scalar_pool{nullptr}; const GraphDefinition *definition{nullptr}; const uint32_t *fanin_offsets{nullptr}; const uint16_t *fanin_indices{nullptr}; @@ -360,25 +361,26 @@ struct GraphExecution { const uint64_t *boundary_scalars{nullptr}; uint32_t boundary_scalar_count{0}; - GraphNodeStorage &node_at(int32_t index) const { return node_storage[index]; } + InGraphTaskStorage &task_at(int32_t index) const { return task_storage[index]; } }; -static_assert(std::is_trivially_destructible_v); -// The tensor pool starts right after the node array, and the scalar pool starts +static_assert(std::is_trivially_destructible_v); +// The tensor pool starts right after the in-graph task array, and the scalar pool starts // after a whole number of ChipTensors. static_assert( - alignof(GraphNodeStorage) % alignof(simpler::hbg::Tensor) == 0, - "a node entry must be at least simpler::hbg::Tensor-aligned: the tensor pool follows the node array" + alignof(InGraphTaskStorage) % alignof(simpler::hbg::Tensor) == 0, + "an in-graph task entry must be at least simpler::hbg::Tensor-aligned: the tensor pool follows the " + "in-graph task array" ); static_assert( sizeof(simpler::hbg::Tensor) % alignof(uint64_t) == 0, "the tensor stride must keep the scalar pool aligned" ); static_assert(std::is_trivially_destructible_v); // The whole storage is aligned for its widest member, so one base check covers the -// header as well as the node array that follows it. +// header as well as the in-graph task array that follows it. static_assert( - alignof(GraphNodeStorage) % alignof(GraphExecution) == 0, - "the node array's alignment must subsume the execution header's" + alignof(InGraphTaskStorage) % alignof(GraphExecution) == 0, + "the in-graph task array's alignment must subsume the execution header's" ); static_assert(sizeof(GraphTensor) <= sizeof(simpler::hbg::Tensor)); @@ -388,42 +390,42 @@ inline constexpr size_t graph_boundary_tensor_pool_slots(uint32_t tensor_count) } // The outer GRAPH task's heap tail occupies -// [GraphExecution][GraphNodeStorage x node_count][simpler::hbg::Tensor x tensor_arg_count] +// [GraphExecution][InGraphTaskStorage x task_count][simpler::hbg::Tensor x tensor_arg_count] // [uint64_t x scalar_arg_count]. // -// The last two regions are the node payloads' argument pools, indexed by the +// The last two regions are the in-graph task payloads' argument pools, indexed by the // Definition's own tensor_offset / scalar_offset — which is why the Definition's -// arg-table counts size them rather than a per-node sum: node i's arguments occupy -// [offset, offset + count) in both the table and the pool. There is no fanin region: -// node dependencies live in the Definition's fanin CSR, so a node's fanin_count stays -// 0 and its fanin delta unbound. +// arg-table counts size them rather than a per-task sum: in-graph task i's arguments +// occupy [offset, offset + count) in both the table and the pool. There is no fanin +// region: an in-graph task's dependencies live in the Definition's fanin CSR, so its +// fanin_count stays 0 and its fanin delta unbound. struct GraphExecutionStorageLayout { - size_t nodes_offset; + size_t tasks_offset; size_t tensors_offset; size_t scalars_offset; size_t total_bytes; }; inline bool graph_execution_storage_layout( - int32_t node_count, uint32_t tensor_arg_count, uint32_t scalar_arg_count, GraphExecutionStorageLayout *out + int32_t task_count, uint32_t tensor_arg_count, uint32_t scalar_arg_count, GraphExecutionStorageLayout *out ) { - if (out == nullptr || node_count <= 0 || node_count > static_cast(GRAPH_MAX_NODES)) { + if (out == nullptr || task_count <= 0 || task_count > static_cast(MAX_IN_GRAPH_TASKS)) { return false; } - constexpr size_t ALIGNMENT = alignof(GraphNodeStorage); - out->nodes_offset = (sizeof(GraphExecution) + ALIGNMENT - 1) & ~(ALIGNMENT - 1); - out->tensors_offset = out->nodes_offset + static_cast(node_count) * sizeof(GraphNodeStorage); + constexpr size_t ALIGNMENT = alignof(InGraphTaskStorage); + out->tasks_offset = (sizeof(GraphExecution) + ALIGNMENT - 1) & ~(ALIGNMENT - 1); + out->tensors_offset = out->tasks_offset + static_cast(task_count) * sizeof(InGraphTaskStorage); out->scalars_offset = out->tensors_offset + static_cast(tensor_arg_count) * sizeof(simpler::hbg::Tensor); out->total_bytes = out->scalars_offset + static_cast(scalar_arg_count) * sizeof(uint64_t); return true; } inline bool graph_execution_storage_bytes( - int32_t node_count, uint32_t tensor_arg_count, uint32_t scalar_arg_count, size_t *storage_bytes + int32_t task_count, uint32_t tensor_arg_count, uint32_t scalar_arg_count, size_t *storage_bytes ) { GraphExecutionStorageLayout layout{}; if (storage_bytes == nullptr || - !graph_execution_storage_layout(node_count, tensor_arg_count, scalar_arg_count, &layout)) { + !graph_execution_storage_layout(task_count, tensor_arg_count, scalar_arg_count, &layout)) { return false; } *storage_bytes = layout.total_bytes; @@ -432,13 +434,9 @@ inline bool graph_execution_storage_bytes( GraphExecution *graph_execution_localize(ChipTaskSlotState &outer_slot); GraphMaterializeResult graph_execution_materialize_slice( - ChipTaskSlotState &outer_slot, GraphExecution &execution, int32_t max_nodes, int32_t *nodes_materialized = nullptr + ChipTaskSlotState &outer_slot, GraphExecution &execution, int32_t max_tasks, int32_t *tasks_materialized = nullptr ); -inline GraphExecution *graph_execution_from_slot(ChipTaskSlotState &slot) { - return slot.task_kind == TaskKind::GRAPH_NODE ? static_cast(slot.graph_context) : nullptr; -} - // An outer GRAPH slot's graph_context holds the shared Definition's device address // until graph_execution_localize replaces it with the execution, so this cast is only // valid after that call. What makes it safe is the boot sequence, not this slot: every @@ -494,14 +492,14 @@ inline bool graph_execution_signal_external_ready(GraphExecution &execution) { GRAPH_EXECUTION_EXTERNAL_READY) == 0; } -inline bool graph_execution_complete_node(GraphExecution &execution) { - return execution.remaining_nodes.fetch_sub(1, std::memory_order_acq_rel) == 1; +inline bool graph_execution_complete_in_graph_task(GraphExecution &execution) { + return execution.remaining_tasks.fetch_sub(1, std::memory_order_acq_rel) == 1; } inline void graph_execution_mark_completed(GraphExecution &execution) { graph_execution_set_state(execution, GraphExecutionState::COMPLETED); } -inline void graph_execution_retire_node(GraphExecution &execution) { - execution.retired_nodes.fetch_add(1, std::memory_order_release); +inline void graph_execution_retire_in_graph_task(GraphExecution &execution) { + execution.retired_tasks.fetch_add(1, std::memory_order_release); } diff --git a/src/common/host_build_graph/graph_host_state.h b/src/common/host_build_graph/graph_host_state.h index 8319e61306..e1988b2b52 100644 --- a/src/common/host_build_graph/graph_host_state.h +++ b/src/common/host_build_graph/graph_host_state.h @@ -72,14 +72,14 @@ struct GraphHostDefinitionList { GraphHostStatePtr make_graph_host_state(const GraphDefinitionArena &arena); /** - * Stand the calling thread's recording storage up — hazard map, node slots, the flat - * per-node arrays and the node tensor pool — without recording anything. + * Stand the calling thread's recording storage up — hazard map, in-graph task slots, the + * flat per-task arrays and the task tensor pool — without recording anything. * * A recorder worker calls this once as it starts, so the allocations land at callable * registration rather than inside the first bind that worker serves, and a failure is * reported where the caller can still act on it. It is an optimization, not the only * stand-up point: a worker the pool creates after prewarm, and a thread whose storage was - * dropped for overshooting the node cap, still stand up lazily on their next recording. + * dropped for overshooting the in-graph task cap, still stand up lazily on their next recording. * * @return false when an allocation failed; the failure is also counted for * graph_recorder_storage_failures(), which is how the host notices across the diff --git a/src/common/host_build_graph/self_relative_ptr.h b/src/common/host_build_graph/self_relative_ptr.h index 2199c5a852..cdf736e11d 100644 --- a/src/common/host_build_graph/self_relative_ptr.h +++ b/src/common/host_build_graph/self_relative_ptr.h @@ -22,7 +22,7 @@ * addresses inside it is invariant under the move while a raw pointer is not. * Every user has to satisfy that precondition: a GLOBAL task's payload and * descriptor live in the same shared-memory image as its slot state, and a Graph - * node's live in the same GraphNodeStorage. + * task's live in the same InGraphTaskStorage. * * A zero delta means unbound — a field can never coincide with its own target. * Zeroed memory therefore reads as null, which is what the slot's pristine state diff --git a/src/common/host_build_graph/task_id_encoding.h b/src/common/host_build_graph/task_id_encoding.h index 52319b271e..c078d53ddf 100644 --- a/src/common/host_build_graph/task_id_encoding.h +++ b/src/common/host_build_graph/task_id_encoding.h @@ -49,7 +49,7 @@ namespace simpler::hbg { enum class TaskIdSpace : uint32_t { GLOBAL = 0, IN_GRAPH = 1 }; // An in-graph task's low bits pack its Graph task's local id above its index -// within that Graph. graph_execution.h asserts GRAPH_MAX_NODES fits the index. +// within that Graph. graph_execution.h asserts MAX_IN_GRAPH_TASKS fits the index. inline constexpr uint32_t IN_GRAPH_TASK_INDEX_BITS = 10; constexpr TaskId make_global_task(uint32_t local_id) { diff --git a/src/common/platform/include/aicpu/chip_swimlane_collector_aicpu.h b/src/common/platform/include/aicpu/chip_swimlane_collector_aicpu.h index 1f4cb64485..7e7ba4a43b 100644 --- a/src/common/platform/include/aicpu/chip_swimlane_collector_aicpu.h +++ b/src/common/platform/include/aicpu/chip_swimlane_collector_aicpu.h @@ -238,7 +238,7 @@ void chip_swimlane_aicpu_record_predicated_skip( /** Record one bounded Scheduler-side Graph materialization slice. */ void chip_swimlane_aicpu_record_graph_prepare( int thread_idx, uint64_t start_time, uint64_t end_time, uint32_t loop_iter, uint64_t task_id, - uint32_t nodes_materialized + uint32_t tasks_materialized ); /** diff --git a/src/common/platform/include/common/chip_swimlane_profiling.h b/src/common/platform/include/common/chip_swimlane_profiling.h index af29c2b459..a9dcc63bfb 100644 --- a/src/common/platform/include/common/chip_swimlane_profiling.h +++ b/src/common/platform/include/common/chip_swimlane_profiling.h @@ -542,7 +542,7 @@ enum class ChipSwimlaneSchedPhaseKind : uint32_t { // phase_data.dummy_task identity payload as DummyTask. // Outer (sched lane): one bounded Graph Definition materialization slice. // phase_data.graph_task identifies the ring-0 outer Graph task and - // tasks_processed is the number of nodes patched in this slice. + // tasks_processed is the number of in-graph tasks patched in this slice. GraphPrepare = 13, }; @@ -625,7 +625,7 @@ static_assert(sizeof(ChipSwimlaneAicpuOrchPhaseRecord) == 32, "ChipSwimlaneAicpu * * `payload` is kind-discriminated: a task id for the kinds that submit a task * (see host_phase_kind_submits_task), otherwise a per-kind detail count such as - * a byte or node count. Readers must consult `kind` before interpreting it. + * a byte or in-graph task count. Readers must consult `kind` before interpreting it. */ struct HostPhaseRecord { uint64_t start_ns; @@ -712,8 +712,8 @@ inline const char *host_phase_kind_name(HostPhaseKind kind) { return "submit_task"; case HostPhaseKind::OrchAllocTensors: return "alloc_tensors"; - case HostPhaseKind::OrchRecordNode: - return "record_node"; + case HostPhaseKind::OrchRecordInGraphTask: + return "record_in_graph_task"; case HostPhaseKind::OrchGraphSubmit: return "graph_submit"; case HostPhaseKind::OrchBuildDefinition: diff --git a/src/common/platform/include/common/host_phase_kind.h b/src/common/platform/include/common/host_phase_kind.h index c885e0d2c0..1ea65a96bd 100644 --- a/src/common/platform/include/common/host_phase_kind.h +++ b/src/common/platform/include/common/host_phase_kind.h @@ -52,18 +52,18 @@ enum class HostPhaseKind : uint32_t { BindArenaH2d, BindHostViewClose, // Recorded by the host orchestrator (orchestrator_core/orchestrator.cpp). - OrchSubmitTask, // submit_task_common: one ordinary task - OrchAllocTensors, // prepare_task: one alloc_tensors slot - OrchRecordNode, // graph_record_submit_node: one recorded Graph node - OrchGraphSubmit, // graph_submit_definition: one outer GRAPH task - OrchBuildDefinition, // graph_layout_definition + graph_fill_definition: one image - OrchGraphBegin, // graph_begin: the whole entry, OrchGraphSubmit nested inside - OrchRecordingWait, // graph_commit's wait for the last recorder to finish - OrchGraphCommit, // graph_commit: the wait plus back-patching every shell + OrchSubmitTask, // submit_task_common: one ordinary task + OrchAllocTensors, // prepare_task: one alloc_tensors slot + OrchRecordInGraphTask, // graph_record_submit_in_graph_task: one recorded in-graph task + OrchGraphSubmit, // graph_submit_definition: one outer GRAPH task + OrchBuildDefinition, // graph_layout_definition + graph_fill_definition: one image + OrchGraphBegin, // graph_begin: the whole entry, OrchGraphSubmit nested inside + OrchRecordingWait, // graph_commit's wait for the last recorder to finish + OrchGraphCommit, // graph_commit: the wait plus back-patching every shell // Recorded in the orchestration .so (orchestration_api.h), which measures the // three submission segments the runtime cannot see. OrchSubmitAdmit, // rt_submit_graph: entry until the Graph is admitted - OrchRecordHandoff, // rt_graph_begin's return until the recorder's first node + OrchRecordHandoff, // rt_graph_begin's return until the recorder's first in-graph task OrchGeneratedArgs, // between two submissions: the generated code's own arg setup Count }; diff --git a/src/common/platform/shared/aicpu/chip_swimlane_collector_aicpu.cpp b/src/common/platform/shared/aicpu/chip_swimlane_collector_aicpu.cpp index d38999f103..8591a6f8c3 100644 --- a/src/common/platform/shared/aicpu/chip_swimlane_collector_aicpu.cpp +++ b/src/common/platform/shared/aicpu/chip_swimlane_collector_aicpu.cpp @@ -989,12 +989,12 @@ void chip_swimlane_aicpu_record_predicated_skip( void chip_swimlane_aicpu_record_graph_prepare( int thread_idx, uint64_t start_time, uint64_t end_time, uint32_t loop_iter, uint64_t task_id, - uint32_t nodes_materialized + uint32_t tasks_materialized ) { auto *record = acquire_sched_phase_record(thread_idx); if (record == nullptr) return; fill_sched_phase_record( - record, ChipSwimlaneSchedPhaseKind::GraphPrepare, start_time, end_time, loop_iter, nodes_materialized, + record, ChipSwimlaneSchedPhaseKind::GraphPrepare, start_time, end_time, loop_iter, tasks_materialized, /*shared_at_start=*/nullptr, /*shared_at_end=*/nullptr ); record->phase_data.graph_task.local_id = static_cast(task_id); diff --git a/tests/st/a2a3/host_build_graph/graph_predicated_dispatch/kernels/orchestration/graph_predicated_dispatch_orch.cpp b/tests/st/a2a3/host_build_graph/graph_predicated_dispatch/kernels/orchestration/graph_predicated_dispatch_orch.cpp index a5a89e4a58..a80e0bc322 100644 --- a/tests/st/a2a3/host_build_graph/graph_predicated_dispatch/kernels/orchestration/graph_predicated_dispatch_orch.cpp +++ b/tests/st/a2a3/host_build_graph/graph_predicated_dispatch/kernels/orchestration/graph_predicated_dispatch_orch.cpp @@ -86,7 +86,7 @@ void layer(const GraphTaskArgs &args, int variant) { const simpler::hbg::Tensor gate_view = boundary_gate.view(view_shape.data(), view_offset.data()); // Graph-internal predicate operand: its address is the recording's virtual - // one, so replay can only read it once materialize has rebound the node. + // one, so replay can only read it once materialize has rebound the tensor. const std::array gate_shape{GATE_ELEMS}; TensorCreateInfo gate_info(gate_shape.data(), static_cast(gate_shape.size()), DataType::INT32); CoreTaskArgs gate_args; diff --git a/tests/st/a5/host_build_graph/graph_predicated_dispatch/kernels/orchestration/graph_predicated_dispatch_orch.cpp b/tests/st/a5/host_build_graph/graph_predicated_dispatch/kernels/orchestration/graph_predicated_dispatch_orch.cpp index a5a89e4a58..a80e0bc322 100644 --- a/tests/st/a5/host_build_graph/graph_predicated_dispatch/kernels/orchestration/graph_predicated_dispatch_orch.cpp +++ b/tests/st/a5/host_build_graph/graph_predicated_dispatch/kernels/orchestration/graph_predicated_dispatch_orch.cpp @@ -86,7 +86,7 @@ void layer(const GraphTaskArgs &args, int variant) { const simpler::hbg::Tensor gate_view = boundary_gate.view(view_shape.data(), view_offset.data()); // Graph-internal predicate operand: its address is the recording's virtual - // one, so replay can only read it once materialize has rebound the node. + // one, so replay can only read it once materialize has rebound the tensor. const std::array gate_shape{GATE_ELEMS}; TensorCreateInfo gate_info(gate_shape.data(), static_cast(gate_shape.size()), DataType::INT32); CoreTaskArgs gate_args; diff --git a/tests/st/host_build_graph_validation/kernels/orchestration/validation_orch.cpp b/tests/st/host_build_graph_validation/kernels/orchestration/validation_orch.cpp index ca0343373c..bd9de6b5a0 100644 --- a/tests/st/host_build_graph_validation/kernels/orchestration/validation_orch.cpp +++ b/tests/st/host_build_graph_validation/kernels/orchestration/validation_orch.cpp @@ -49,7 +49,7 @@ simpler::hbg::Tensor tensor_with_unbound_owner(const simpler::hbg::Tensor &exter // An IN_GRAPH id names storage inside one Graph task's body, not a task-table slot, // so it can never be a fanin producer. Declaring one as an explicit dependency is // the caller error append_fanin_or_fail rejects. -void submit_task_depending_on_graph_node() { +void submit_task_depending_on_in_graph_task() { const TaskId deps[1] = {simpler::hbg::make_in_graph_task(/*graph_local_id=*/1, /*task_index=*/0)}; CoreTaskArgs args; args.launch_spec.set_block_num(1); @@ -85,7 +85,7 @@ __attribute__((visibility("default"))) void aicpu_orchestration_entry(const Chip set_tensor_data(tensor_with_unbound_owner(external), 1, index, 7); return; case 4: - submit_task_depending_on_graph_node(); + submit_task_depending_on_in_graph_task(); return; default: rt_report_fatal( diff --git a/tests/st/host_build_graph_validation/test_host_build_graph_validation.py b/tests/st/host_build_graph_validation/test_host_build_graph_validation.py index f44d16f1a8..4878573ac0 100644 --- a/tests/st/host_build_graph_validation/test_host_build_graph_validation.py +++ b/tests/st/host_build_graph_validation/test_host_build_graph_validation.py @@ -38,7 +38,7 @@ "mixed_subtask_overflow": 1, "unbound_owner_read": 2, "unbound_owner_write": 3, - "graph_node_dependency": 4, + "in_graph_task_dependency": 4, } diff --git a/tests/ut/cpp/a2a3/test_graph_activation.cpp b/tests/ut/cpp/a2a3/test_graph_activation.cpp index 670e98a5e5..aabceac203 100644 --- a/tests/ut/cpp/a2a3/test_graph_activation.cpp +++ b/tests/ut/cpp/a2a3/test_graph_activation.cpp @@ -11,7 +11,7 @@ /** * Deterministic tests for incremental graph activation. * - * Under incremental activation a graph node may reach the ready queue before the + * Under incremental activation an in-graph task may reach the ready queue before the * whole GRAPH task is materialized, so a producer can complete while a later * consumer is still being registered. Scene tests hit that interleaving only * probabilistically; these host-side tests force it, exercising the exact path @@ -55,18 +55,18 @@ class GraphActivationTest : public ::testing::Test { sm_arena.release(); } - // One GraphExecution node whose slot is a routable single-block KERNEL/AIC + // One in-graph task whose slot is a routable single-block KERNEL/AIC // task in the given completion state, with its payload wired the way // materialization leaves it for the wake/route path. - static void init_graph_node(GraphNodeStorage &node, int32_t node_index, ChipTaskState state) { - memset(&node, 0, sizeof(GraphNodeStorage)); - node.slot.task_state.store(state); - node.slot.graph_node_index = node_index; - node.slot.active_mask = ActiveMask(SUBTASK_MASK_AIC); - node.slot.task_kind = TaskKind::KERNEL; - node.slot.total_required_subtasks = 1; - node.slot.logical_block_num = 1; - node.slot.payload.set(&node.payload); + static void init_in_graph_task(InGraphTaskStorage &task, int32_t task_index, ChipTaskState state) { + memset(&task, 0, sizeof(InGraphTaskStorage)); + task.slot.task_state.store(state); + task.slot.in_graph_task_index = task_index; + task.slot.active_mask = ActiveMask(SUBTASK_MASK_AIC); + task.slot.task_kind = TaskKind::KERNEL; + task.slot.total_required_subtasks = 1; + task.slot.logical_block_num = 1; + task.slot.payload.set(&task.payload); } }; @@ -74,84 +74,83 @@ class GraphActivationTest : public ::testing::Test { // == SENTINEL) reaches graph_first_unmet_producer, which reads task_state and // routes it — it is never lost on the closed wake list. TEST_F(GraphActivationTest, WakeRoutesConsumerWhenProducerCompletedBeforeRegister) { - auto nodes = std::make_unique(2); - init_graph_node(nodes[0], 0, CHIP_TASK_COMPLETED); // producer, already completed - init_graph_node(nodes[1], 1, CHIP_TASK_PENDING); // consumer of node 0 - nodes[0].slot.wake_list_head.store(WAKE_LIST_SENTINEL); // its wake list already drained + auto tasks = std::make_unique(2); + init_in_graph_task(tasks[0], 0, CHIP_TASK_COMPLETED); // producer, already completed + init_in_graph_task(tasks[1], 1, CHIP_TASK_PENDING); // consumer of task 0 + tasks[0].slot.wake_list_head.store(WAKE_LIST_SENTINEL); // its wake list already drained - std::vector fanin_offsets{0, 0, 1}; // node 0 is a root; node 1 <- {0} + std::vector fanin_offsets{0, 0, 1}; // task 0 is a root; task 1 <- {0} std::vector fanin_indices{0}; GraphExecution exec{}; - exec.nodes = exec.node_storage = nodes.get(); + exec.tasks = exec.task_storage = tasks.get(); exec.fanin_offsets = fanin_offsets.data(); exec.fanin_indices = fanin_indices.data(); - sched.register_graph_wake(exec, &nodes[0].slot, &nodes[1].slot); + sched.register_graph_wake(exec, &tasks[0].slot, &tasks[1].slot); ChipTaskSlotState *out[2]; ASSERT_EQ(sched.get_ready_tasks_batch(sched.ready_queues, ResourceShape::AIC, out, 2), 1) << "consumer must route to ready, not hang on the SENTINEL wake list"; - EXPECT_EQ(out[0], &nodes[1].slot); + EXPECT_EQ(out[0], &tasks[1].slot); } -// graph_incremental_publish routes a node whose producers are all COMPLETED at -// publish time, and wake-chains a node with a still-pending producer so it +// graph_incremental_publish routes an in-graph task whose producers are all COMPLETED at +// publish time, and wake-chains one with a still-pending producer so it // routes exactly once that producer completes and drains its wake list. TEST_F(GraphActivationTest, IncrementalPublishRoutesCompletedDepsAndWakeChainsPending) { - auto nodes = std::make_unique(4); - init_graph_node(nodes[0], 0, CHIP_TASK_COMPLETED); // root, completed - init_graph_node(nodes[1], 1, CHIP_TASK_PENDING); // root, pending - init_graph_node(nodes[2], 2, CHIP_TASK_PENDING); // consumer of node 0 (completed) - init_graph_node(nodes[3], 3, CHIP_TASK_PENDING); // consumer of node 1 (pending) + auto tasks = std::make_unique(4); + init_in_graph_task(tasks[0], 0, CHIP_TASK_COMPLETED); // root, completed + init_in_graph_task(tasks[1], 1, CHIP_TASK_PENDING); // root, pending + init_in_graph_task(tasks[2], 2, CHIP_TASK_PENDING); // consumer of task 0 (completed) + init_in_graph_task(tasks[3], 3, CHIP_TASK_PENDING); // consumer of task 1 (pending) - std::vector fanin_offsets{0, 0, 0, 1, 2}; // node 2 <- {0}, node 3 <- {1} + std::vector fanin_offsets{0, 0, 0, 1, 2}; // task 2 <- {0}, task 3 <- {1} std::vector fanin_indices{0, 1}; GraphExecution exec{}; - exec.nodes = exec.node_storage = nodes.get(); + exec.tasks = exec.task_storage = tasks.get(); exec.fanin_offsets = fanin_offsets.data(); exec.fanin_indices = fanin_indices.data(); sched.graph_incremental_publish(exec, 0, 4); - EXPECT_EQ(exec.published_nodes.load(), 4); + EXPECT_EQ(exec.published_tasks.load(), 4); ChipTaskSlotState *out[4]; ASSERT_EQ(sched.get_ready_tasks_batch(sched.ready_queues, ResourceShape::AIC, out, 4), 1) << "only the consumer whose producers are all COMPLETED routes at publish time"; - EXPECT_EQ(out[0], &nodes[2].slot); + EXPECT_EQ(out[0], &tasks[2].slot); - nodes[1].slot.task_state.store(CHIP_TASK_COMPLETED); - sched.drain_graph_wake_list(exec, nodes[1].slot); + tasks[1].slot.task_state.store(CHIP_TASK_COMPLETED); + sched.drain_graph_wake_list(exec, tasks[1].slot); ASSERT_EQ(sched.get_ready_tasks_batch(sched.ready_queues, ResourceShape::AIC, out, 4), 1) << "the wake-chained consumer must route once its pending producer completes"; - EXPECT_EQ(out[0], &nodes[3].slot); + EXPECT_EQ(out[0], &tasks[3].slot); } -// Incremental activation dispatches a node before the graph reaches ACTIVE, so -// complete_task must accept a node completion while the graph is MATERIALIZING or +// Incremental activation dispatches an in-graph task before the graph reaches ACTIVE, so +// complete_task must accept such a completion while the graph is MATERIALIZING or // PREPARED, and reject it only for SUBMITTED (not yet bound) or COMPLETED // (already retired). TEST_F(GraphActivationTest, CompleteTaskAcceptsCompletionBeforeActive) { GraphDefinition definition{}; auto complete_in_state = [&](GraphExecutionState state) { - auto node = std::make_unique(1); - memset(node.get(), 0, sizeof(GraphNodeStorage)); - node[0].slot.task_kind = TaskKind::GRAPH_NODE; - node[0].slot.graph_node_index = 0; - node[0].slot.total_required_subtasks = 1; - node[0].slot.payload.set(&node[0].payload); + auto task = std::make_unique(1); + memset(task.get(), 0, sizeof(InGraphTaskStorage)); + task[0].slot.in_graph_task_index = 0; + task[0].slot.total_required_subtasks = 1; + task[0].slot.payload.set(&task[0].payload); GraphExecution exec{}; exec.definition = &definition; - exec.nodes = exec.node_storage = node.get(); - exec.node_count = 1; - exec.remaining_nodes.store(1); + exec.tasks = exec.task_storage = task.get(); + exec.task_count = 1; + exec.remaining_tasks.store(1); exec.outer_slot = nullptr; graph_execution_set_state(exec, state); - node[0].slot.graph_context = &exec; + task[0].slot.graph_context = &exec; #if SIMPLER_SCHED_PROFILING - return sched.complete_task(node[0].slot, 0).error_code; + return sched.complete_task(task[0].slot, 0).error_code; #else - return sched.complete_task(node[0].slot).error_code; + return sched.complete_task(task[0].slot).error_code; #endif }; @@ -161,3 +160,31 @@ TEST_F(GraphActivationTest, CompleteTaskAcceptsCompletionBeforeActive) { EXPECT_EQ(complete_in_state(GraphExecutionState::SUBMITTED), SIMPLER_ERROR_INVALID_ARGS); EXPECT_EQ(complete_in_state(GraphExecutionState::COMPLETED), SIMPLER_ERROR_INVALID_ARGS); } + +// The outer Graph task completes as a task of the run, not into its execution's +// counters. It is the one slot where a non-null graph_context does NOT mean "in a +// Graph body": before localize swaps in the GraphExecution the shell's context is +// the shared GraphDefinition, so the `task_kind == GRAPH` half of complete_task's +// predicate is the only thing keeping the two apart. Drop it and this slot's +// Definition gets read as an execution -- a silent static_cast onto another +// struct's layout, no fault and no error code. +TEST_F(GraphActivationTest, CompleteTaskTakesTheOrdinaryPathForTheOuterGraphTask) { + GraphDefinition definition{}; + TaskDescriptor outer_task{}; + outer_task.task_id = simpler::hbg::make_global_task(0); + + ChipTaskSlotState slot{}; + slot.task.set(&outer_task); + slot.task_kind = TaskKind::GRAPH; + slot.graph_context = &definition; + +#if SIMPLER_SCHED_PROFILING + const SchedulerState::TaskCompletionOutcome outcome = sched.complete_task(slot, 0); +#else + const SchedulerState::TaskCompletionOutcome outcome = sched.complete_task(slot); +#endif + + EXPECT_EQ(outcome.error_code, SIMPLER_ERROR_NONE); + EXPECT_EQ(outcome.stream_tasks_completed, 1) << "the outer Graph task is one completed task of the run"; + EXPECT_EQ(slot.task_state.load(std::memory_order_relaxed), CHIP_TASK_COMPLETED); +} diff --git a/tests/ut/cpp/a2a3/test_hbg_task_allocator.cpp b/tests/ut/cpp/a2a3/test_hbg_task_allocator.cpp index 9ee460662e..7ad599162e 100644 --- a/tests/ut/cpp/a2a3/test_hbg_task_allocator.cpp +++ b/tests/ut/cpp/a2a3/test_hbg_task_allocator.cpp @@ -292,7 +292,7 @@ TEST_F(HbgTaskAllocatorTest, LatchedFatalShortCircuitsReserveDeferredHeap) { EXPECT_EQ(allocator.heap_top(), 0u); } -// Graph recording addresses its internal nodes' outputs from +// Graph recording addresses its in-graph tasks' outputs from // GRAPH_RECORD_VIRTUAL_BASE upward and classifies internal vs boundary tensor // sources by address-range containment alone. A real heap that reached into that // range would silently misclassify, so init() refuses it. diff --git a/tests/ut/cpp/a5/test_graph_activation.cpp b/tests/ut/cpp/a5/test_graph_activation.cpp index 670e98a5e5..aabceac203 100644 --- a/tests/ut/cpp/a5/test_graph_activation.cpp +++ b/tests/ut/cpp/a5/test_graph_activation.cpp @@ -11,7 +11,7 @@ /** * Deterministic tests for incremental graph activation. * - * Under incremental activation a graph node may reach the ready queue before the + * Under incremental activation an in-graph task may reach the ready queue before the * whole GRAPH task is materialized, so a producer can complete while a later * consumer is still being registered. Scene tests hit that interleaving only * probabilistically; these host-side tests force it, exercising the exact path @@ -55,18 +55,18 @@ class GraphActivationTest : public ::testing::Test { sm_arena.release(); } - // One GraphExecution node whose slot is a routable single-block KERNEL/AIC + // One in-graph task whose slot is a routable single-block KERNEL/AIC // task in the given completion state, with its payload wired the way // materialization leaves it for the wake/route path. - static void init_graph_node(GraphNodeStorage &node, int32_t node_index, ChipTaskState state) { - memset(&node, 0, sizeof(GraphNodeStorage)); - node.slot.task_state.store(state); - node.slot.graph_node_index = node_index; - node.slot.active_mask = ActiveMask(SUBTASK_MASK_AIC); - node.slot.task_kind = TaskKind::KERNEL; - node.slot.total_required_subtasks = 1; - node.slot.logical_block_num = 1; - node.slot.payload.set(&node.payload); + static void init_in_graph_task(InGraphTaskStorage &task, int32_t task_index, ChipTaskState state) { + memset(&task, 0, sizeof(InGraphTaskStorage)); + task.slot.task_state.store(state); + task.slot.in_graph_task_index = task_index; + task.slot.active_mask = ActiveMask(SUBTASK_MASK_AIC); + task.slot.task_kind = TaskKind::KERNEL; + task.slot.total_required_subtasks = 1; + task.slot.logical_block_num = 1; + task.slot.payload.set(&task.payload); } }; @@ -74,84 +74,83 @@ class GraphActivationTest : public ::testing::Test { // == SENTINEL) reaches graph_first_unmet_producer, which reads task_state and // routes it — it is never lost on the closed wake list. TEST_F(GraphActivationTest, WakeRoutesConsumerWhenProducerCompletedBeforeRegister) { - auto nodes = std::make_unique(2); - init_graph_node(nodes[0], 0, CHIP_TASK_COMPLETED); // producer, already completed - init_graph_node(nodes[1], 1, CHIP_TASK_PENDING); // consumer of node 0 - nodes[0].slot.wake_list_head.store(WAKE_LIST_SENTINEL); // its wake list already drained + auto tasks = std::make_unique(2); + init_in_graph_task(tasks[0], 0, CHIP_TASK_COMPLETED); // producer, already completed + init_in_graph_task(tasks[1], 1, CHIP_TASK_PENDING); // consumer of task 0 + tasks[0].slot.wake_list_head.store(WAKE_LIST_SENTINEL); // its wake list already drained - std::vector fanin_offsets{0, 0, 1}; // node 0 is a root; node 1 <- {0} + std::vector fanin_offsets{0, 0, 1}; // task 0 is a root; task 1 <- {0} std::vector fanin_indices{0}; GraphExecution exec{}; - exec.nodes = exec.node_storage = nodes.get(); + exec.tasks = exec.task_storage = tasks.get(); exec.fanin_offsets = fanin_offsets.data(); exec.fanin_indices = fanin_indices.data(); - sched.register_graph_wake(exec, &nodes[0].slot, &nodes[1].slot); + sched.register_graph_wake(exec, &tasks[0].slot, &tasks[1].slot); ChipTaskSlotState *out[2]; ASSERT_EQ(sched.get_ready_tasks_batch(sched.ready_queues, ResourceShape::AIC, out, 2), 1) << "consumer must route to ready, not hang on the SENTINEL wake list"; - EXPECT_EQ(out[0], &nodes[1].slot); + EXPECT_EQ(out[0], &tasks[1].slot); } -// graph_incremental_publish routes a node whose producers are all COMPLETED at -// publish time, and wake-chains a node with a still-pending producer so it +// graph_incremental_publish routes an in-graph task whose producers are all COMPLETED at +// publish time, and wake-chains one with a still-pending producer so it // routes exactly once that producer completes and drains its wake list. TEST_F(GraphActivationTest, IncrementalPublishRoutesCompletedDepsAndWakeChainsPending) { - auto nodes = std::make_unique(4); - init_graph_node(nodes[0], 0, CHIP_TASK_COMPLETED); // root, completed - init_graph_node(nodes[1], 1, CHIP_TASK_PENDING); // root, pending - init_graph_node(nodes[2], 2, CHIP_TASK_PENDING); // consumer of node 0 (completed) - init_graph_node(nodes[3], 3, CHIP_TASK_PENDING); // consumer of node 1 (pending) + auto tasks = std::make_unique(4); + init_in_graph_task(tasks[0], 0, CHIP_TASK_COMPLETED); // root, completed + init_in_graph_task(tasks[1], 1, CHIP_TASK_PENDING); // root, pending + init_in_graph_task(tasks[2], 2, CHIP_TASK_PENDING); // consumer of task 0 (completed) + init_in_graph_task(tasks[3], 3, CHIP_TASK_PENDING); // consumer of task 1 (pending) - std::vector fanin_offsets{0, 0, 0, 1, 2}; // node 2 <- {0}, node 3 <- {1} + std::vector fanin_offsets{0, 0, 0, 1, 2}; // task 2 <- {0}, task 3 <- {1} std::vector fanin_indices{0, 1}; GraphExecution exec{}; - exec.nodes = exec.node_storage = nodes.get(); + exec.tasks = exec.task_storage = tasks.get(); exec.fanin_offsets = fanin_offsets.data(); exec.fanin_indices = fanin_indices.data(); sched.graph_incremental_publish(exec, 0, 4); - EXPECT_EQ(exec.published_nodes.load(), 4); + EXPECT_EQ(exec.published_tasks.load(), 4); ChipTaskSlotState *out[4]; ASSERT_EQ(sched.get_ready_tasks_batch(sched.ready_queues, ResourceShape::AIC, out, 4), 1) << "only the consumer whose producers are all COMPLETED routes at publish time"; - EXPECT_EQ(out[0], &nodes[2].slot); + EXPECT_EQ(out[0], &tasks[2].slot); - nodes[1].slot.task_state.store(CHIP_TASK_COMPLETED); - sched.drain_graph_wake_list(exec, nodes[1].slot); + tasks[1].slot.task_state.store(CHIP_TASK_COMPLETED); + sched.drain_graph_wake_list(exec, tasks[1].slot); ASSERT_EQ(sched.get_ready_tasks_batch(sched.ready_queues, ResourceShape::AIC, out, 4), 1) << "the wake-chained consumer must route once its pending producer completes"; - EXPECT_EQ(out[0], &nodes[3].slot); + EXPECT_EQ(out[0], &tasks[3].slot); } -// Incremental activation dispatches a node before the graph reaches ACTIVE, so -// complete_task must accept a node completion while the graph is MATERIALIZING or +// Incremental activation dispatches an in-graph task before the graph reaches ACTIVE, so +// complete_task must accept such a completion while the graph is MATERIALIZING or // PREPARED, and reject it only for SUBMITTED (not yet bound) or COMPLETED // (already retired). TEST_F(GraphActivationTest, CompleteTaskAcceptsCompletionBeforeActive) { GraphDefinition definition{}; auto complete_in_state = [&](GraphExecutionState state) { - auto node = std::make_unique(1); - memset(node.get(), 0, sizeof(GraphNodeStorage)); - node[0].slot.task_kind = TaskKind::GRAPH_NODE; - node[0].slot.graph_node_index = 0; - node[0].slot.total_required_subtasks = 1; - node[0].slot.payload.set(&node[0].payload); + auto task = std::make_unique(1); + memset(task.get(), 0, sizeof(InGraphTaskStorage)); + task[0].slot.in_graph_task_index = 0; + task[0].slot.total_required_subtasks = 1; + task[0].slot.payload.set(&task[0].payload); GraphExecution exec{}; exec.definition = &definition; - exec.nodes = exec.node_storage = node.get(); - exec.node_count = 1; - exec.remaining_nodes.store(1); + exec.tasks = exec.task_storage = task.get(); + exec.task_count = 1; + exec.remaining_tasks.store(1); exec.outer_slot = nullptr; graph_execution_set_state(exec, state); - node[0].slot.graph_context = &exec; + task[0].slot.graph_context = &exec; #if SIMPLER_SCHED_PROFILING - return sched.complete_task(node[0].slot, 0).error_code; + return sched.complete_task(task[0].slot, 0).error_code; #else - return sched.complete_task(node[0].slot).error_code; + return sched.complete_task(task[0].slot).error_code; #endif }; @@ -161,3 +160,31 @@ TEST_F(GraphActivationTest, CompleteTaskAcceptsCompletionBeforeActive) { EXPECT_EQ(complete_in_state(GraphExecutionState::SUBMITTED), SIMPLER_ERROR_INVALID_ARGS); EXPECT_EQ(complete_in_state(GraphExecutionState::COMPLETED), SIMPLER_ERROR_INVALID_ARGS); } + +// The outer Graph task completes as a task of the run, not into its execution's +// counters. It is the one slot where a non-null graph_context does NOT mean "in a +// Graph body": before localize swaps in the GraphExecution the shell's context is +// the shared GraphDefinition, so the `task_kind == GRAPH` half of complete_task's +// predicate is the only thing keeping the two apart. Drop it and this slot's +// Definition gets read as an execution -- a silent static_cast onto another +// struct's layout, no fault and no error code. +TEST_F(GraphActivationTest, CompleteTaskTakesTheOrdinaryPathForTheOuterGraphTask) { + GraphDefinition definition{}; + TaskDescriptor outer_task{}; + outer_task.task_id = simpler::hbg::make_global_task(0); + + ChipTaskSlotState slot{}; + slot.task.set(&outer_task); + slot.task_kind = TaskKind::GRAPH; + slot.graph_context = &definition; + +#if SIMPLER_SCHED_PROFILING + const SchedulerState::TaskCompletionOutcome outcome = sched.complete_task(slot, 0); +#else + const SchedulerState::TaskCompletionOutcome outcome = sched.complete_task(slot); +#endif + + EXPECT_EQ(outcome.error_code, SIMPLER_ERROR_NONE); + EXPECT_EQ(outcome.stream_tasks_completed, 1) << "the outer Graph task is one completed task of the run"; + EXPECT_EQ(slot.task_state.load(std::memory_order_relaxed), CHIP_TASK_COMPLETED); +} diff --git a/tests/ut/cpp/common/test_hbg_graph_cache.cpp b/tests/ut/cpp/common/test_hbg_graph_cache.cpp index 82698eeb1c..3522b36e5e 100644 --- a/tests/ut/cpp/common/test_hbg_graph_cache.cpp +++ b/tests/ut/cpp/common/test_hbg_graph_cache.cpp @@ -61,24 +61,24 @@ make_test_definition(uint64_t graph_key, uint64_t boundary_address, uint32_t bou std::vector fanout_offsets{0, 1, 1}; std::vector fanout_indices{1}; std::vector roots{0}; - std::vector node_offsets{0, 64}; - std::vector nodes(2); - for (GraphNodeDefinition &node : nodes) { - std::fill(std::begin(node.kernel_id), std::end(node.kernel_id), INVALID_KERNEL_ID); - node.kernel_id[0] = 42; - node.active_mask = 1; - node.logical_block_num = 1; - node.total_required_subtasks = 1; - node.tensor_count = 1; - node.scalar_count = 1; - node.total_output_size = 64; + std::vector in_graph_task_offsets{0, 64}; + std::vector tasks(2); + for (InGraphTaskDefinition &task : tasks) { + std::fill(std::begin(task.kernel_id), std::end(task.kernel_id), INVALID_KERNEL_ID); + task.kernel_id[0] = 42; + task.active_mask = 1; + task.logical_block_num = 1; + task.total_required_subtasks = 1; + task.tensor_count = 1; + task.scalar_count = 1; + task.total_output_size = 64; } - nodes[0].dump_metadata.dump_arg_mask = uint64_t{1} << 0; - nodes[0].dump_metadata.scalar_dtypes[0] = static_cast(DataType::FLOAT32); - nodes[1].dump_metadata.dump_arg_mask = uint64_t{1} << 1; - nodes[1].dump_metadata.scalar_dtypes[0] = static_cast(DataType::INT32); - nodes[1].tensor_offset = 1; - nodes[1].scalar_offset = 1; + tasks[0].dump_metadata.dump_arg_mask = uint64_t{1} << 0; + tasks[0].dump_metadata.scalar_dtypes[0] = static_cast(DataType::FLOAT32); + tasks[1].dump_metadata.dump_arg_mask = uint64_t{1} << 1; + tasks[1].dump_metadata.scalar_dtypes[0] = static_cast(DataType::INT32); + tasks[1].tensor_offset = 1; + tasks[1].scalar_offset = 1; std::vector tensors{make_test_tensor(boundary_address), make_test_tensor(boundary_address)}; tensors[1].buffer_size = 32; std::vector tensor_sources(2); @@ -106,8 +106,8 @@ make_test_definition(uint64_t graph_key, uint64_t boundary_address, uint32_t bou definition.off_fanout_offsets = append_section(image, fanout_offsets); definition.off_fanout_indices = append_section(image, fanout_indices); definition.off_root_indices = append_section(image, roots); - definition.off_node_offsets = append_section(image, node_offsets); - definition.off_nodes = append_section(image, nodes); + definition.off_in_graph_task_offsets = append_section(image, in_graph_task_offsets); + definition.off_in_graph_tasks = append_section(image, tasks); definition.off_tensors = append_section(image, tensors); definition.off_tensor_sources = append_section(image, tensor_sources); definition.off_scalars = append_section(image, scalars); @@ -162,11 +162,11 @@ class AlignedStorage { public: explicit AlignedStorage(size_t bytes, uint8_t fill = 0) : bytes_(bytes) { - data_ = ::operator new(bytes, std::align_val_t(alignof(GraphNodeStorage))); + data_ = ::operator new(bytes, std::align_val_t(alignof(InGraphTaskStorage))); std::memset(data_, fill, bytes); } - ~AlignedStorage() { ::operator delete(data_, std::align_val_t(alignof(GraphNodeStorage))); } + ~AlignedStorage() { ::operator delete(data_, std::align_val_t(alignof(InGraphTaskStorage))); } void *data() const { return data_; } uint8_t *bytes() const { return static_cast(data_); } @@ -285,39 +285,39 @@ TEST(GraphScalarProvenance, ForwardedScalarRetainsBoundarySource) { boundary_args.anchor_scalar_sources(); CoreTaskArgs forwarded_args; forwarded_args.copy_scalars_from(boundary_args, 1, 1); - CoreTaskArgs node_args; + CoreTaskArgs task_args; - node_args.copy_scalars_from(forwarded_args, 0, 1); + task_args.copy_scalars_from(forwarded_args, 0, 1); - EXPECT_EQ(node_args.scalar_source(0), static_cast(&std::as_const(boundary_args).scalar(1))); + EXPECT_EQ(task_args.scalar_source(0), static_cast(&std::as_const(boundary_args).scalar(1))); } TEST(GraphScalarProvenance, MutableAccessInvalidatesForwardedSource) { CoreTaskArgs boundary_args; boundary_args.add_scalar(uint32_t{17}); boundary_args.anchor_scalar_sources(); - CoreTaskArgs node_args; - node_args.copy_scalars_from(boundary_args, 0, 1); - ASSERT_NE(node_args.scalar_source(0), nullptr); + CoreTaskArgs task_args; + task_args.copy_scalars_from(boundary_args, 0, 1); + ASSERT_NE(task_args.scalar_source(0), nullptr); - node_args.scalar(0) = 18; + task_args.scalar(0) = 18; - EXPECT_EQ(node_args.scalar_source(0), nullptr); + EXPECT_EQ(task_args.scalar_source(0), nullptr); EXPECT_EQ( - node_args.invalidated_scalar_source(0), static_cast(&std::as_const(boundary_args).scalar(0)) + task_args.invalidated_scalar_source(0), static_cast(&std::as_const(boundary_args).scalar(0)) ); } TEST(GraphExecutionStorage, ComputesAlignedExactSize) { - constexpr int32_t NODE_COUNT = 7; + constexpr int32_t TASK_COUNT = 7; constexpr uint32_t TENSOR_ARGS = 11; constexpr uint32_t SCALAR_ARGS = 5; GraphExecutionStorageLayout layout{}; - ASSERT_TRUE(graph_execution_storage_layout(NODE_COUNT, TENSOR_ARGS, SCALAR_ARGS, &layout)); - EXPECT_EQ(layout.nodes_offset % alignof(GraphNodeStorage), 0U); - EXPECT_GE(layout.nodes_offset, sizeof(GraphExecution)); - EXPECT_EQ(layout.tensors_offset, layout.nodes_offset + NODE_COUNT * sizeof(GraphNodeStorage)); + ASSERT_TRUE(graph_execution_storage_layout(TASK_COUNT, TENSOR_ARGS, SCALAR_ARGS, &layout)); + EXPECT_EQ(layout.tasks_offset % alignof(InGraphTaskStorage), 0U); + EXPECT_GE(layout.tasks_offset, sizeof(GraphExecution)); + EXPECT_EQ(layout.tensors_offset, layout.tasks_offset + TASK_COUNT * sizeof(InGraphTaskStorage)); EXPECT_EQ(layout.tensors_offset % alignof(simpler::hbg::Tensor), 0U); EXPECT_EQ(layout.scalars_offset, layout.tensors_offset + TENSOR_ARGS * sizeof(simpler::hbg::Tensor)); EXPECT_EQ(layout.total_bytes, layout.scalars_offset + SCALAR_ARGS * sizeof(uint64_t)); @@ -357,25 +357,25 @@ TEST(GraphBoundaryPool, WidestBoundaryExceedsOneSlotBudget) { } // The pools are sized by the Definition's arg tables, so a Definition whose -// nodes declare fewer arguments reserves less. +// tasks declare fewer arguments reserves less. TEST(GraphExecutionStorage, NarrowerArgTablesReserveLess) { - constexpr int32_t NODE_COUNT = 4; + constexpr int32_t TASK_COUNT = 4; size_t wide = 0; size_t narrow = 0; - ASSERT_TRUE(graph_execution_storage_bytes(NODE_COUNT, 32, 16, &wide)); - ASSERT_TRUE(graph_execution_storage_bytes(NODE_COUNT, 4, 2, &narrow)); + ASSERT_TRUE(graph_execution_storage_bytes(TASK_COUNT, 32, 16, &wide)); + ASSERT_TRUE(graph_execution_storage_bytes(TASK_COUNT, 4, 2, &narrow)); EXPECT_LT(narrow, wide); } -TEST(GraphExecutionStorage, RejectsInvalidNodeCount) { +TEST(GraphExecutionStorage, RejectsInvalidInGraphTaskCount) { size_t storage_bytes = 0; EXPECT_FALSE(graph_execution_storage_bytes(0, 1, 1, &storage_bytes)); EXPECT_FALSE(graph_execution_storage_bytes(-1, 1, 1, &storage_bytes)); - EXPECT_FALSE(graph_execution_storage_bytes(static_cast(GRAPH_MAX_NODES) + 1, 1, 1, &storage_bytes)); - // A Definition with no arguments at all still needs its node array. + EXPECT_FALSE(graph_execution_storage_bytes(static_cast(MAX_IN_GRAPH_TASKS) + 1, 1, 1, &storage_bytes)); + // A Definition with no arguments at all still needs its in-graph task array. EXPECT_TRUE(graph_execution_storage_bytes(1, 0, 0, &storage_bytes)); - EXPECT_GE(storage_bytes, sizeof(GraphExecution) + sizeof(GraphNodeStorage)); + EXPECT_GE(storage_bytes, sizeof(GraphExecution) + sizeof(InGraphTaskStorage)); } // A resubmission gets the same heap tail back, so the bytes it starts from are @@ -403,33 +403,33 @@ TEST(GraphExecutionReplay, ResubmissionRebuildsFromDefinition) { outer_slot.task.set(&outer_task); outer_slot.graph_context = execution; - // The execution and node storage both occupy the outer heap tail after + // The execution and in-graph task storage both occupy the outer heap tail after // required_heap. EXPECT_EQ(static_cast(execution), heap.execution()); EXPECT_EQ(graph_execution_materialize_slice(outer_slot, *execution, 2), GraphMaterializeResult::PREPARED); - GraphNodeStorage &node = execution->node_at(0); - ASSERT_EQ(node.payload.scalar_count, 1); - ASSERT_EQ(node.payload.tensor_count, 1); - EXPECT_EQ(node.payload.scalar_data()[0], 17U); - EXPECT_EQ(execution->node_at(1).payload.scalar_data()[0], 18U); - EXPECT_EQ(node.payload.dump_metadata.dump_arg_mask, uint64_t{1} << 0); - EXPECT_EQ(node.payload.dump_metadata.scalar_dtypes[0], static_cast(DataType::FLOAT32)); - EXPECT_EQ(execution->node_at(1).payload.dump_metadata.dump_arg_mask, uint64_t{1} << 1); - EXPECT_EQ(execution->node_at(1).payload.dump_metadata.scalar_dtypes[0], static_cast(DataType::INT32)); + InGraphTaskStorage &storage = execution->task_at(0); + ASSERT_EQ(storage.payload.scalar_count, 1); + ASSERT_EQ(storage.payload.tensor_count, 1); + EXPECT_EQ(storage.payload.scalar_data()[0], 17U); + EXPECT_EQ(execution->task_at(1).payload.scalar_data()[0], 18U); + EXPECT_EQ(storage.payload.dump_metadata.dump_arg_mask, uint64_t{1} << 0); + EXPECT_EQ(storage.payload.dump_metadata.scalar_dtypes[0], static_cast(DataType::FLOAT32)); + EXPECT_EQ(execution->task_at(1).payload.dump_metadata.dump_arg_mask, uint64_t{1} << 1); + EXPECT_EQ(execution->task_at(1).payload.dump_metadata.scalar_dtypes[0], static_cast(DataType::INT32)); graph_execution_mark_completed(*execution); - execution->retired_nodes.store(2, std::memory_order_release); + execution->retired_tasks.store(2, std::memory_order_release); outer_task.task_id = simpler::hbg::make_global_task(8); // Poison every field the rebuild is responsible for restoring. A replay that // preserved any of them would leave the poison observable. - node.task.kernel_id[0] = 314; - node.slot.active_mask = ActiveMask(3); - node.payload.scalar_data()[0] = 2718; - execution->node_at(1).payload.scalar_data()[0] = 31415; - node.payload.tensor_data()[0].version = 1618; - node.slot.completed_subtasks.store(1, std::memory_order_relaxed); - node.payload.dispatch_fanin.store(1, std::memory_order_relaxed); + storage.task.kernel_id[0] = 314; + storage.slot.active_mask = ActiveMask(3); + storage.payload.scalar_data()[0] = 2718; + execution->task_at(1).payload.scalar_data()[0] = 31415; + storage.payload.tensor_data()[0].version = 1618; + storage.slot.completed_subtasks.store(1, std::memory_order_relaxed); + storage.payload.dispatch_fanin.store(1, std::memory_order_relaxed); execution = heap.initialize_execution(definition_object, reinterpret_cast(second_boundary.data()), 99); ASSERT_NE(execution, nullptr); @@ -438,23 +438,23 @@ TEST(GraphExecutionReplay, ResubmissionRebuildsFromDefinition) { EXPECT_EQ(static_cast(execution), heap.execution()); EXPECT_EQ(graph_execution_materialize_slice(outer_slot, *execution, 2), GraphMaterializeResult::PREPARED); - EXPECT_EQ(node.task.kernel_id[0], 42); - EXPECT_EQ(node.slot.active_mask.raw(), 1); - EXPECT_EQ(node.payload.scalar_data()[0], 99U); - EXPECT_EQ(execution->node_at(1).payload.scalar_data()[0], 18U); - EXPECT_EQ(node.payload.tensor_data()[0].version, 0); - EXPECT_EQ(node.task.task_id, simpler::hbg::make_in_graph_task(/*graph_local_id=*/8, /*task_index=*/0)); - EXPECT_EQ(node.task.packed_buffer_base, heap.base()); - EXPECT_EQ(node.payload.tensor_data()[0].buffer.addr, reinterpret_cast(second_boundary.data())); - EXPECT_EQ(execution->node_at(1).payload.tensor_data()[0].buffer.addr, reinterpret_cast(heap.base() + 16)); - EXPECT_EQ(node.slot.completed_subtasks.load(std::memory_order_relaxed), 0); - EXPECT_EQ(node.payload.dispatch_fanin.load(std::memory_order_relaxed), 0); - EXPECT_EQ(node.payload.dump_metadata.dump_arg_mask, uint64_t{1} << 0); + EXPECT_EQ(storage.task.kernel_id[0], 42); + EXPECT_EQ(storage.slot.active_mask.raw(), 1); + EXPECT_EQ(storage.payload.scalar_data()[0], 99U); + EXPECT_EQ(execution->task_at(1).payload.scalar_data()[0], 18U); + EXPECT_EQ(storage.payload.tensor_data()[0].version, 0); + EXPECT_EQ(storage.task.task_id, simpler::hbg::make_in_graph_task(/*graph_local_id=*/8, /*task_index=*/0)); + EXPECT_EQ(storage.task.packed_buffer_base, heap.base()); + EXPECT_EQ(storage.payload.tensor_data()[0].buffer.addr, reinterpret_cast(second_boundary.data())); + EXPECT_EQ(execution->task_at(1).payload.tensor_data()[0].buffer.addr, reinterpret_cast(heap.base() + 16)); + EXPECT_EQ(storage.slot.completed_subtasks.load(std::memory_order_relaxed), 0); + EXPECT_EQ(storage.payload.dispatch_fanin.load(std::memory_order_relaxed), 0); + EXPECT_EQ(storage.payload.dump_metadata.dump_arg_mask, uint64_t{1} << 0); } // The boundary scalar pool is bounded by the Graph boundary contract // (GRAPH_MAX_SCALAR_ARGS), not by a single task payload's MAX_SCALAR_ARGS — -// a node stages at most MAX_SCALAR_ARGS entries from it, but the pool itself +// an in-graph task stages at most MAX_SCALAR_ARGS entries from it, but the pool itself // may be wider. TEST(GraphExecutionReplay, MaterializesBoundaryScalarPoolWiderThanTaskPayload) { constexpr uint64_t GRAPH_KEY_VALUE = 0x1234; @@ -480,7 +480,7 @@ TEST(GraphExecutionReplay, MaterializesBoundaryScalarPoolWiderThanTaskPayload) { outer_slot.graph_context = execution; EXPECT_EQ(graph_execution_materialize_slice(outer_slot, *execution, 2), GraphMaterializeResult::PREPARED); - EXPECT_EQ(execution->node_storage[0].payload.scalar_data()[0], 21U); + EXPECT_EQ(execution->task_storage[0].payload.scalar_data()[0], 21U); } TEST(GraphExecutionReplay, RejectsBoundaryScalarPoolBeyondContract) { @@ -640,10 +640,13 @@ TEST(GraphExecutionErrors, GraphPrepareQueueOverflowIsReported) { EXPECT_EQ(header.sched_error_bitmap.load(std::memory_order_acquire), 1U << 3); } -TEST(GraphExecutionErrors, InvalidNodeCompletionIsReported) { +TEST(GraphExecutionErrors, InvalidInGraphTaskCompletionIsReported) { SchedulerState scheduler{}; ChipTaskSlotState slot{}; - slot.task_kind = TaskKind::GRAPH_NODE; + // Membership without a usable execution: graph_context names one, but neither its + // definition nor its in-graph task array was ever bound. + GraphExecution execution{}; + slot.graph_context = &execution; const SchedulerState::TaskCompletionOutcome outcome = scheduler.complete_task(slot); @@ -651,23 +654,22 @@ TEST(GraphExecutionErrors, InvalidNodeCompletionIsReported) { EXPECT_EQ(outcome.stream_tasks_completed, 0); } -TEST(GraphExecutionProgress, InternalNodeResolutionIsNotAHostCompletion) { +TEST(GraphExecutionProgress, InGraphTaskResolutionIsNotAHostCompletion) { SchedulerState scheduler{}; GraphDefinition definition{}; - GraphNodeStorage node{}; + InGraphTaskStorage task{}; GraphExecution execution{}; execution.definition = &definition; - execution.nodes = &node; - execution.node_storage = &node; - execution.node_count = 1; - execution.remaining_nodes.store(1, std::memory_order_relaxed); + execution.tasks = &task; + execution.task_storage = &task; + execution.task_count = 1; + execution.remaining_tasks.store(1, std::memory_order_relaxed); graph_execution_set_state(execution, GraphExecutionState::ACTIVE, std::memory_order_relaxed); - node.slot.task_kind = TaskKind::GRAPH_NODE; - node.slot.graph_context = &execution; - node.slot.graph_node_index = 0; + task.slot.graph_context = &execution; + task.slot.in_graph_task_index = 0; AsyncWaitList wait_list{}; - wait_list.entries[0].slot_state = &node.slot; + wait_list.entries[0].slot_state = &task.slot; wait_list.entries[0].task_token = simpler::hbg::make_global_task(1); wait_list.entries[0].normal_done = true; wait_list.count = 1; @@ -709,25 +711,25 @@ TEST(GraphExecutionMaterialize, DirtyStorageYieldsValidExecution) { // Every observable scheduling field must be a materialize-written value, // not the 0xAA fill: state machine, counters and atomics all start from // values only the device side wrote. - for (int32_t i = 0; i < execution->node_count; ++i) { - const GraphNodeStorage &node = execution->node_at(i); - ASSERT_EQ(node.slot.task_state.load(std::memory_order_relaxed), CHIP_TASK_PENDING); - ASSERT_EQ(node.slot.task_kind, TaskKind::GRAPH_NODE); - ASSERT_EQ(node.slot.completed_subtasks.load(std::memory_order_relaxed), 0); - ASSERT_EQ(node.payload.dispatch_fanin.load(std::memory_order_relaxed), 0); - ASSERT_EQ(node.payload.tensor_count, 1); - ASSERT_EQ(node.payload.scalar_count, 1); + for (int32_t i = 0; i < execution->task_count; ++i) { + const InGraphTaskStorage &storage = execution->task_at(i); + ASSERT_EQ(storage.slot.task_state.load(std::memory_order_relaxed), CHIP_TASK_PENDING); + ASSERT_EQ(storage.slot.task_kind, TaskKind::KERNEL); + ASSERT_EQ(storage.slot.completed_subtasks.load(std::memory_order_relaxed), 0); + ASSERT_EQ(storage.payload.dispatch_fanin.load(std::memory_order_relaxed), 0); + ASSERT_EQ(storage.payload.tensor_count, 1); + ASSERT_EQ(storage.payload.scalar_count, 1); // A tensor address of 0xAAAAAAAAAAAAAAAA would mean the fill leaked // through into a field the scheduler later dereferences. - ASSERT_NE(node.payload.tensor_data()[0].buffer.addr, 0xAAAAAAAAAAAAAAAAULL); - // make_test_definition assigns node i the heap offset 64*i, so the + ASSERT_NE(storage.payload.tensor_data()[0].buffer.addr, 0xAAAAAAAAAAAAAAAAULL); + // make_test_definition assigns in-graph task i the heap offset 64*i, so the // packed window starts at outer_base + 64*i, not at outer_base. - ASSERT_EQ(node.task.packed_buffer_base, static_cast(heap.base() + static_cast(i) * 64)); + ASSERT_EQ(storage.task.packed_buffer_base, static_cast(heap.base() + static_cast(i) * 64)); } - EXPECT_EQ(execution->materialized_nodes, execution->node_count); + EXPECT_EQ(execution->materialized_tasks, execution->task_count); EXPECT_EQ(execution->consumed_tensor_args, 2U); - // Localize and materialize read the boundary and write node arguments; neither may + // Localize and materialize read the boundary and write in-graph task arguments; neither may // touch the tensor region past the packed boundary values. const std::byte *tail = heap.boundary_tail(1); for (size_t i = 0; i < heap.boundary_tail_bytes(1); ++i) { diff --git a/tests/ut/cpp/common/test_hbg_graph_definition_arena.cpp b/tests/ut/cpp/common/test_hbg_graph_definition_arena.cpp index 70b77cef10..75b3ee020a 100644 --- a/tests/ut/cpp/common/test_hbg_graph_definition_arena.cpp +++ b/tests/ut/cpp/common/test_hbg_graph_definition_arena.cpp @@ -93,10 +93,10 @@ class HbgGraphDefinitionArenaTest : public ::testing::Test { return reinterpret_cast(image); } - // Record one Graph of `node_count` chained nodes under `graph_key`. The chain + // Record one Graph of `task_count` chained tasks under `graph_key`. The chain // makes the image's size a function of the count, so two keys recorded with // different counts cannot come out byte-identical and share one Definition. - void record_graph(uint64_t graph_key, int node_count, const simpler::hbg::Tensor &boundary, const uint32_t *shape) { + void record_graph(uint64_t graph_key, int task_count, const simpler::hbg::Tensor &boundary, const uint32_t *shape) { GraphTaskArgs boundary_args; boundary_args.add_input(boundary); const GraphScopeResult scope = orch.graph_begin(graph_key, boundary_args, 0x1736); @@ -104,12 +104,12 @@ class HbgGraphDefinitionArenaTest : public ::testing::Test { ASSERT_TRUE(scope.task_id.is_valid()); ASSERT_TRUE(orch.graph_prepare(scope.recording_handle, boundary_args)); simpler::hbg::Tensor input = boundary; - for (int i = 0; i < node_count; ++i) { - CoreTaskArgs node_args; - node_args.add_input(input); + for (int i = 0; i < task_count; ++i) { + CoreTaskArgs task_args; + task_args.add_input(input); TensorCreateInfo output(shape, 1, DataType::UINT32); - node_args.add_output(output); - TaskOutputTensors outputs = orch.submit_dummy_task(node_args); + task_args.add_output(output); + TaskOutputTensors outputs = orch.submit_dummy_task(task_args); ASSERT_TRUE(outputs.task_id().is_valid()); input = outputs.get_ref(0); } diff --git a/tests/ut/cpp/common/test_hbg_graph_recording_bounds.cpp b/tests/ut/cpp/common/test_hbg_graph_recording_bounds.cpp index 490ed68b9e..a377a0c9e9 100644 --- a/tests/ut/cpp/common/test_hbg_graph_recording_bounds.cpp +++ b/tests/ut/cpp/common/test_hbg_graph_recording_bounds.cpp @@ -10,7 +10,7 @@ */ /** * A recorded task's id is what its entry in the recording's hazard map is keyed on, - * and that map holds GRAPH_MAX_NODES task chains. An IN_GRAPH id's low field is the + * and that map holds MAX_IN_GRAPH_TASKS task chains. An IN_GRAPH id's low field is the * task's index within its body, so the key is in range however far into a run the * Graph begins — which a GLOBAL id carrying the allocator's own local id would not * be. @@ -80,7 +80,7 @@ class HbgGraphRecordingBoundsTest : public ::testing::Test { } }; -// A Graph recorded after GRAPH_MAX_NODES ordinary tasks. Its recorded task registers +// A Graph recorded after MAX_IN_GRAPH_TASKS ordinary tasks. Its recorded task registers // its outputs in the recording hazard map keyed on its own id, so that id decides // whether the key lands inside the map's task chains. TEST_F(HbgGraphRecordingBoundsTest, RecordedTaskIsKeyedByItsIndexNotByTheRunsNumbering) { @@ -91,12 +91,12 @@ TEST_F(HbgGraphRecordingBoundsTest, RecordedTaskIsKeyedByItsIndexNotByTheRunsNum orch.begin_scope(); // Move the allocator's local-id counter past the recording map's task-chain // count, the way any run that submits a while before its first Graph does. - for (uint32_t i = 0; i < GRAPH_MAX_NODES; ++i) { + for (uint32_t i = 0; i < MAX_IN_GRAPH_TASKS; ++i) { CoreTaskArgs filler; filler.add_input(boundary); ASSERT_TRUE(orch.submit_dummy_task(filler).task_id().is_valid()) << "filler task " << i; } - ASSERT_EQ(orch.task_allocator.active_count(), static_cast(GRAPH_MAX_NODES)); + ASSERT_EQ(orch.task_allocator.active_count(), static_cast(MAX_IN_GRAPH_TASKS)); GraphTaskArgs boundary_args; boundary_args.add_input(boundary); @@ -107,17 +107,17 @@ TEST_F(HbgGraphRecordingBoundsTest, RecordedTaskIsKeyedByItsIndexNotByTheRunsNum // An INOUT operand is what makes the recorded task register an output: the // recording's hazard map exists for exactly the write-in-place shape. - CoreTaskArgs node_args; - node_args.add_inout(boundary); - const TaskId node_id = orch.submit_dummy_task(node_args).task_id(); - ASSERT_TRUE(node_id.is_valid()); - EXPECT_EQ(simpler::hbg::task_id_space(node_id), simpler::hbg::TaskIdSpace::IN_GRAPH) + CoreTaskArgs task_args; + task_args.add_inout(boundary); + const TaskId in_graph_task_id = orch.submit_dummy_task(task_args).task_id(); + ASSERT_TRUE(in_graph_task_id.is_valid()); + EXPECT_EQ(simpler::hbg::task_id_space(in_graph_task_id), simpler::hbg::TaskIdSpace::IN_GRAPH) << "a recorded task must not take a GLOBAL id: nothing resolves it against the task table, and its low " "field is what keys the recording's hazard map"; - EXPECT_EQ(simpler::hbg::task_local_id(node_id), 0u) + EXPECT_EQ(simpler::hbg::task_local_id(in_graph_task_id), 0u) << "the first recorded task's low field is task index 0, independent of how many tasks the run has " "already allocated"; - EXPECT_LT(simpler::hbg::task_local_id(node_id), GRAPH_MAX_NODES); + EXPECT_LT(simpler::hbg::task_local_id(in_graph_task_id), MAX_IN_GRAPH_TASKS); ASSERT_TRUE(orch.graph_end()); } @@ -147,9 +147,9 @@ TEST_F(HbgGraphRecordingBoundsTest, PreGraphProducerOfABoundaryTensorContributes ASSERT_TRUE(graph.recording); ASSERT_TRUE(orch.graph_prepare(graph.recording_handle, boundary_args)); - CoreTaskArgs node_args; - node_args.add_inout(boundary); - ASSERT_TRUE(orch.submit_dummy_task(node_args).task_id().is_valid()); + CoreTaskArgs task_args; + task_args.add_inout(boundary); + ASSERT_TRUE(orch.submit_dummy_task(task_args).task_id().is_valid()); ASSERT_TRUE(orch.graph_end()); const GraphHostDefinitionList published = graph_host_definitions(*graph_state); diff --git a/tests/ut/cpp/common/test_hbg_graph_submit_failure.cpp b/tests/ut/cpp/common/test_hbg_graph_submit_failure.cpp index 4e02ca5a50..c783344189 100644 --- a/tests/ut/cpp/common/test_hbg_graph_submit_failure.cpp +++ b/tests/ut/cpp/common/test_hbg_graph_submit_failure.cpp @@ -46,9 +46,9 @@ class HbgGraphSubmitFailureTest : public ::testing::Test { std::vector definition_staging; GraphDefinitionArena arena{}; - // A Graph task's heap allocation covers its nodes' packed outputs *and* the + // A Graph task's heap allocation covers its tasks' packed outputs *and* the // execution storage the device materializes into, so the pool has to hold a - // GraphExecution header plus one GraphNodeStorage (~5 KB) on top of the + // GraphExecution header plus one InGraphTaskStorage (~5 KB) on top of the // outputs. 4 KB used to be enough when the storage came from a separate // device allocation. static constexpr size_t HEAP_BYTES = 64 * 1024; @@ -113,11 +113,11 @@ TEST_F(HbgGraphSubmitFailureTest, InFlightGraphInvocationsReserveHeapOnlyAtCommi EXPECT_EQ(graph_host_upload_count(*graph_state), 2u); ASSERT_TRUE(orch.graph_prepare(first.recording_handle, boundary_args)); - CoreTaskArgs node_args; - node_args.add_input(boundary); + CoreTaskArgs task_args; + task_args.add_input(boundary); TensorCreateInfo recorded_output(shape, 1, DataType::UINT32); - node_args.add_output(recorded_output); - ASSERT_TRUE(orch.submit_dummy_task(node_args).task_id().is_valid()); + task_args.add_output(recorded_output); + ASSERT_TRUE(orch.submit_dummy_task(task_args).task_id().is_valid()); ASSERT_TRUE(orch.graph_end()); EXPECT_EQ(orch.task_allocator.heap_top(), 0u); @@ -156,7 +156,7 @@ TEST_F(HbgGraphSubmitFailureTest, InFlightGraphInvocationsReserveHeapOnlyAtCommi // // That overlap is held together only by field partitioning: under // recording_mutex the main thread reads boundary_tensors / boundary_types / -// boundary_scalar_count, while the worker writes boundary_args / nodes / +// boundary_scalar_count, while the worker writes boundary_args / tasks / // next_virtual_offset / unsupported without it (graph_prepare skips the mutex on // purpose, so a submit burst cannot starve it). Nothing enforces that split, so // this pins the functional contract that depends on it — and gives TSAN a window @@ -182,7 +182,7 @@ TEST_F(HbgGraphSubmitFailureTest, WorkerRecordsWhileMainThreadSubmitsSameHashShe bool prepared = false; bool main_done_submitting = false; bool prepare_ok = false; - bool node_ok = false; + bool task_ok = false; bool end_ok = false; std::thread worker([&]() { @@ -205,11 +205,11 @@ TEST_F(HbgGraphSubmitFailureTest, WorkerRecordsWhileMainThreadSubmitsSameHashShe }); } - CoreTaskArgs node_args; - node_args.add_input(boundary); + CoreTaskArgs task_args; + task_args.add_input(boundary); TensorCreateInfo recorded_output(shape, 1, DataType::UINT32); - node_args.add_output(recorded_output); - node_ok = orch.submit_dummy_task(node_args).task_id().is_valid(); + task_args.add_output(recorded_output); + task_ok = orch.submit_dummy_task(task_args).task_id().is_valid(); end_ok = orch.graph_end(); }); @@ -232,7 +232,7 @@ TEST_F(HbgGraphSubmitFailureTest, WorkerRecordsWhileMainThreadSubmitsSameHashShe worker.join(); ASSERT_TRUE(prepare_ok); - ASSERT_TRUE(node_ok); + ASSERT_TRUE(task_ok); ASSERT_TRUE(end_ok); EXPECT_FALSE(second.recording); EXPECT_FALSE(second.execute_block); @@ -290,11 +290,11 @@ TEST_F(HbgGraphSubmitFailureTest, AbortedRecordingLatchesFatalAtCommit) { ASSERT_TRUE(graph.task_id.is_valid()); ASSERT_TRUE(orch.graph_prepare(graph.recording_handle, boundary_args)); - CoreTaskArgs node_args; - node_args.add_input(boundary); + CoreTaskArgs task_args; + task_args.add_input(boundary); TensorCreateInfo recorded_output(shape, 1, DataType::UINT32); - node_args.add_output(recorded_output); - ASSERT_TRUE(orch.submit_dummy_task(node_args).task_id().is_valid()); + task_args.add_output(recorded_output); + ASSERT_TRUE(orch.submit_dummy_task(task_args).task_id().is_valid()); orch.graph_abort(graph.recording_handle); ASSERT_FALSE(orch.fatal) << "Abort alone must not latch; the shell is still finalizable in principle"; @@ -323,11 +323,11 @@ TEST_F(HbgGraphSubmitFailureTest, AutoScopeNestedInManualScopeRefusesTheRecordin orch.begin_scope(ScopeMode::MANUAL); orch.begin_scope(ScopeMode::AUTO); - CoreTaskArgs node_args; - node_args.add_input(boundary); + CoreTaskArgs task_args; + task_args.add_input(boundary); TensorCreateInfo recorded_output(shape, 1, DataType::UINT32); - node_args.add_output(recorded_output); - ASSERT_TRUE(orch.submit_dummy_task(node_args).task_id().is_valid()); + task_args.add_output(recorded_output); + ASSERT_TRUE(orch.submit_dummy_task(task_args).task_id().is_valid()); orch.end_scope(); orch.end_scope(); @@ -338,10 +338,10 @@ TEST_F(HbgGraphSubmitFailureTest, AutoScopeNestedInManualScopeRefusesTheRecordin EXPECT_TRUE(orch.fatal) << "a shell whose Definition never arrived cannot be completed"; } -// A Graph body may allocate. The allocation records as a kernel-less node, the -// same shape submit_dummy_task records, so the recording stays publishable and +// A Graph body may allocate. The allocation records as a kernel-less in-graph task, +// the same shape submit_dummy_task records, so the recording stays publishable and // the commit latches no fatal. -TEST_F(HbgGraphSubmitFailureTest, RuntimeAllocationInsideTheBodyRecordsAKernellessNode) { +TEST_F(HbgGraphSubmitFailureTest, RuntimeAllocationInsideTheBodyRecordsAKernellessInGraphTask) { std::array storage{}; uint32_t shape[] = {static_cast(storage.size())}; simpler::hbg::Tensor boundary = simpler::hbg::make_tensor_external(storage.data(), shape, 1); @@ -377,12 +377,12 @@ TEST_F(HbgGraphSubmitFailureTest, FaninFailureLatchesFatalWithoutPartialUpload) ASSERT_TRUE(graph.recording); ASSERT_TRUE(orch.graph_prepare(graph.recording_handle, boundary_args)); - CoreTaskArgs node_args; - node_args.add_input(boundary); + CoreTaskArgs task_args; + task_args.add_input(boundary); TensorCreateInfo recorded_output(shape, 1, DataType::UINT32); - node_args.add_output(recorded_output); + task_args.add_output(recorded_output); const uint64_t heap_top_before_record = orch.task_allocator.heap_top(); - ASSERT_TRUE(orch.submit_dummy_task(node_args).task_id().is_valid()); + ASSERT_TRUE(orch.submit_dummy_task(task_args).task_id().is_valid()); EXPECT_EQ(orch.task_allocator.heap_top(), heap_top_before_record); ASSERT_TRUE(orch.graph_end()); EXPECT_EQ(orch.task_allocator.heap_top(), heap_top_before_record); @@ -421,9 +421,9 @@ TEST_F(HbgGraphSubmitFailureTest, CachedGraphUsesFinalTaskWindowSlot) { ASSERT_TRUE(graph.recording); ASSERT_TRUE(orch.graph_prepare(graph.recording_handle, boundary_args)); - CoreTaskArgs node_args; - node_args.add_input(boundary); - ASSERT_TRUE(orch.submit_dummy_task(node_args).task_id().is_valid()); + CoreTaskArgs task_args; + task_args.add_input(boundary); + ASSERT_TRUE(orch.submit_dummy_task(task_args).task_id().is_valid()); ASSERT_TRUE(orch.graph_end()); ASSERT_EQ(orch.task_allocator.active_count(), 1); @@ -455,7 +455,7 @@ TEST_F(HbgGraphSubmitFailureTest, CachedGraphUsesFinalTaskWindowSlot) { // it. class HbgGraphPredicateRejectionTest : public HbgGraphSubmitFailureTest { protected: - // Records one predicated node into a fresh Graph and asserts the recording + // Records one predicated in-graph task into a fresh Graph and asserts the recording // refused it. `build_predicate` receives the boundary tensor. template void expect_recording_refused(uint64_t graph_key, BuildPredicate build_predicate) { @@ -470,14 +470,14 @@ class HbgGraphPredicateRejectionTest : public HbgGraphSubmitFailureTest { EXPECT_TRUE(graph.recording); EXPECT_TRUE(orch.graph_prepare(graph.recording_handle, boundary_args)); - CoreTaskArgs node_args; - node_args.add_input(boundary); + CoreTaskArgs task_args; + task_args.add_input(boundary); TensorCreateInfo recorded_output(shape, 1, DataType::INT32); - node_args.add_output(recorded_output); + task_args.add_output(recorded_output); MixedKernels mixed{}; mixed.aiv0_kernel_id = 0; - node_args.set_predicate(build_predicate(boundary)); - EXPECT_TRUE(orch.submit_task(mixed, node_args).task_id().is_valid()); + task_args.set_predicate(build_predicate(boundary)); + EXPECT_TRUE(orch.submit_task(mixed, task_args).task_id().is_valid()); EXPECT_THROW(orch.graph_end(), AssertionError) << "an unrecordable predicate must not publish"; orch.graph_abort(graph.recording_handle); @@ -505,7 +505,7 @@ TEST_F(HbgGraphPredicateRejectionTest, OperandIndexOutsideTheExtentAbortsTheReco } TEST_F(HbgGraphPredicateRejectionTest, OperandOnAnUnclassifiableTensorAbortsTheRecording) { - // Neither a boundary tensor nor any recorded node's output, so the recorder + // Neither a boundary tensor nor any recorded task's output, so the recorder // cannot name a base the replay could rebind against. std::array foreign_storage{}; uint32_t shape[] = {static_cast(foreign_storage.size())}; @@ -516,12 +516,12 @@ TEST_F(HbgGraphPredicateRejectionTest, OperandOnAnUnclassifiableTensorAbortsTheR }); } -// A kernel-less node never dispatches, so submit_dummy_task and alloc_tensors +// A kernel-less in-graph task never dispatches, so submit_dummy_task and alloc_tensors // drop the caller's predicate exactly as they do on the ordinary path. Recording -// must drop it too: a node whose Definition claimed a predicate its own attribute +// must drop it too: a task whose Definition claimed a predicate its own attribute // denies is rejected by materialize, on the device, for a value the scheduler was // never going to read. -TEST_F(HbgGraphPredicateRejectionTest, PredicateOnAKernellessNodeIsNotRecorded) { +TEST_F(HbgGraphPredicateRejectionTest, PredicateOnAKernellessInGraphTaskIsNotRecorded) { std::array storage{}; uint32_t shape[] = {static_cast(storage.size())}; simpler::hbg::Tensor boundary = simpler::hbg::make_tensor_external(storage.data(), shape, 1, DataType::INT32); @@ -533,14 +533,14 @@ TEST_F(HbgGraphPredicateRejectionTest, PredicateOnAKernellessNodeIsNotRecorded) ASSERT_TRUE(graph.recording); ASSERT_TRUE(orch.graph_prepare(graph.recording_handle, boundary_args)); - CoreTaskArgs node_args; - node_args.add_input(boundary); + CoreTaskArgs task_args; + task_args.add_input(boundary); TensorCreateInfo recorded_output(shape, 1, DataType::INT32); - node_args.add_output(recorded_output); + task_args.add_output(recorded_output); // Out of extent, which a recorded predicate would reject — proving the // predicate never reached the recorder rather than merely passing its checks. - node_args.set_predicate(predicate_on(boundary, 16)); - ASSERT_TRUE(orch.submit_dummy_task(node_args).task_id().is_valid()); + task_args.set_predicate(predicate_on(boundary, 16)); + ASSERT_TRUE(orch.submit_dummy_task(task_args).task_id().is_valid()); EXPECT_TRUE(orch.graph_end()) << "a dropped predicate must not make the body unrecordable"; orch.graph_commit(); @@ -579,17 +579,17 @@ TEST_F(HbgGraphSubmitFailureTest, ASecondKeyRecordsAlongsideTheFirst) { TensorCreateInfo recorded_output(shape, 1, DataType::UINT32); ASSERT_TRUE(orch.graph_prepare(first.recording_handle, args_a)); - CoreTaskArgs node_a; - node_a.add_input(boundary_a); - node_a.add_output(recorded_output); - ASSERT_TRUE(orch.submit_dummy_task(node_a).task_id().is_valid()); + CoreTaskArgs task_a; + task_a.add_input(boundary_a); + task_a.add_output(recorded_output); + ASSERT_TRUE(orch.submit_dummy_task(task_a).task_id().is_valid()); ASSERT_TRUE(orch.graph_end()); ASSERT_TRUE(orch.graph_prepare(second.recording_handle, args_b)); - CoreTaskArgs node_b; - node_b.add_input(boundary_b); - node_b.add_output(recorded_output); - ASSERT_TRUE(orch.submit_dummy_task(node_b).task_id().is_valid()); + CoreTaskArgs task_b; + task_b.add_input(boundary_b); + task_b.add_output(recorded_output); + ASSERT_TRUE(orch.submit_dummy_task(task_b).task_id().is_valid()); ASSERT_TRUE(orch.graph_end()); // One commit drains and back-patches both keys' deferred shells. @@ -628,10 +628,10 @@ TEST_F(HbgGraphSubmitFailureTest, ConcurrentDefinitionsFinalizeInSubmissionOrder for (size_t i = kGraphCount; i-- > 0;) { ASSERT_TRUE(orch.graph_prepare(graphs[i].recording_handle, args)) << "Graph " << i; - CoreTaskArgs node_args; - node_args.add_input(boundary); - node_args.add_output(recorded_output); - ASSERT_TRUE(orch.submit_dummy_task(node_args).task_id().is_valid()) << "Graph " << i; + CoreTaskArgs task_args; + task_args.add_input(boundary); + task_args.add_output(recorded_output); + ASSERT_TRUE(orch.submit_dummy_task(task_args).task_id().is_valid()) << "Graph " << i; ASSERT_TRUE(orch.graph_end()) << "Graph " << i; } @@ -674,10 +674,10 @@ TEST_F(HbgGraphSubmitFailureTest, ACachedGraphReplaysWhileAnotherKeyRecords) { const GraphScopeResult first = orch.graph_begin(0x1903, args_a, 0x1736); ASSERT_TRUE(first.recording); ASSERT_TRUE(orch.graph_prepare(first.recording_handle, args_a)); - CoreTaskArgs node_a; - node_a.add_input(boundary_a); - node_a.add_output(recorded_output); - ASSERT_TRUE(orch.submit_dummy_task(node_a).task_id().is_valid()); + CoreTaskArgs task_a; + task_a.add_input(boundary_a); + task_a.add_output(recorded_output); + ASSERT_TRUE(orch.submit_dummy_task(task_a).task_id().is_valid()); ASSERT_TRUE(orch.graph_end()); orch.graph_commit(); ASSERT_FALSE(orch.fatal); @@ -695,10 +695,10 @@ TEST_F(HbgGraphSubmitFailureTest, ACachedGraphReplaysWhileAnotherKeyRecords) { EXPECT_GT(orch.task_allocator.heap_top(), 0u); ASSERT_TRUE(orch.graph_prepare(second.recording_handle, args_b)); - CoreTaskArgs node_b; - node_b.add_input(boundary_b); - node_b.add_output(recorded_output); - ASSERT_TRUE(orch.submit_dummy_task(node_b).task_id().is_valid()); + CoreTaskArgs task_b; + task_b.add_input(boundary_b); + task_b.add_output(recorded_output); + ASSERT_TRUE(orch.submit_dummy_task(task_b).task_id().is_valid()); ASSERT_TRUE(orch.graph_end()); orch.graph_commit(); EXPECT_FALSE(orch.fatal); @@ -733,10 +733,10 @@ TEST_F(HbgGraphSubmitFailureTest, AnOrdinaryAllocationInterleavesWithADeferredSh // Only then does the recording finish and the shell claim its block. ASSERT_TRUE(orch.graph_prepare(graph.recording_handle, boundary_args)); - CoreTaskArgs node_args; - node_args.add_input(boundary); - node_args.add_output(recorded_output); - ASSERT_TRUE(orch.submit_dummy_task(node_args).task_id().is_valid()); + CoreTaskArgs task_args; + task_args.add_input(boundary); + task_args.add_output(recorded_output); + ASSERT_TRUE(orch.submit_dummy_task(task_args).task_id().is_valid()); ASSERT_TRUE(orch.graph_end()); orch.graph_commit(); @@ -757,10 +757,10 @@ TEST_F(HbgGraphSubmitFailureTest, AnOrdinaryAllocationInterleavesWithADeferredSh // A Graph's boundary tensor can be an upstream task's output, which lives in the // graph heap and therefore carries an address out of HEAP_VIRTUAL_BASE's window // while recording — three address classes are in play at once, the third being -// GRAPH_RECORD_VIRTUAL_BASE for the recorded nodes' own outputs. Recording must +// GRAPH_RECORD_VIRTUAL_BASE for the recorded tasks' own outputs. Recording must // still classify such a tensor as a boundary: graph_tensor_from_boundary matches // on equality, not on range containment, and the windows do not overlap. If it -// fell through to the recorded-output ranges instead, the node would be marked +// fell through to the recorded-output ranges instead, the task would be marked // unsupported and the whole Graph would silently drop to the ordinary path. // // The Definition describes a boundary by its shape, strides, buffer size and type @@ -793,11 +793,11 @@ TEST_F(HbgGraphSubmitFailureTest, RecordsAGraphWhoseBoundaryLivesInTheHeapWindow EXPECT_TRUE(graph.task_id.is_valid()); EXPECT_TRUE(orch.graph_prepare(graph.recording_handle, boundary_args)); - CoreTaskArgs node_args; - node_args.add_input(boundary); + CoreTaskArgs task_args; + task_args.add_input(boundary); TensorCreateInfo recorded_output(shape, 1, DataType::UINT32); - node_args.add_output(recorded_output); - EXPECT_TRUE(orch.submit_dummy_task(node_args).task_id().is_valid()); + task_args.add_output(recorded_output); + EXPECT_TRUE(orch.submit_dummy_task(task_args).task_id().is_valid()); EXPECT_TRUE(orch.graph_end()); orch.graph_commit(); EXPECT_FALSE(orch.fatal); diff --git a/tests/ut/cpp/common/test_hbg_ready_queue_seed.cpp b/tests/ut/cpp/common/test_hbg_ready_queue_seed.cpp index 5c547768df..31868e5aba 100644 --- a/tests/ut/cpp/common/test_hbg_ready_queue_seed.cpp +++ b/tests/ut/cpp/common/test_hbg_ready_queue_seed.cpp @@ -151,7 +151,7 @@ TEST(HbgReadyQueueSizing, DerivesCapacityForEachReachablePopulation) { populations.add_task(ActiveMask(SUBTASK_MASK_AIC), TaskAttrs{}, TaskKind::KERNEL, 9400); populations.add_task(ActiveMask(SUBTASK_MASK_AIV0), sync_start, TaskKind::KERNEL, 3); - populations.add_task(ActiveMask(SUBTASK_MASK_AIC | SUBTASK_MASK_AIV0), predicate, TaskKind::GRAPH_NODE, 5); + populations.add_task(ActiveMask(SUBTASK_MASK_AIC | SUBTASK_MASK_AIV0), predicate, TaskKind::KERNEL, 5); populations.add_task(ActiveMask{}, TaskAttrs{}, TaskKind::DUMMY); populations.add_task(ActiveMask(SUBTASK_MASK_AIV1), predicated_sync_start, TaskKind::KERNEL, 3); populations.add_task(ActiveMask{}, TaskAttrs{}, TaskKind::GRAPH, 7); diff --git a/tests/ut/cpp/common/test_hbg_slot_claim.cpp b/tests/ut/cpp/common/test_hbg_slot_claim.cpp index 2ab93a423b..711bb3911f 100644 --- a/tests/ut/cpp/common/test_hbg_slot_claim.cpp +++ b/tests/ut/cpp/common/test_hbg_slot_claim.cpp @@ -88,7 +88,7 @@ class HbgSlotClaimTest : public ::testing::Test { state.any_subtask_deferred.store(true, std::memory_order_relaxed); state.completed_subtasks.store(7, std::memory_order_relaxed); state.next_block_idx.store(3, std::memory_order_relaxed); - state.graph_node_index = 11; + state.in_graph_task_index = 11; sm_handle->header->tasks.completion_flags[slot].store(1, std::memory_order_relaxed); } @@ -144,9 +144,9 @@ TEST_F(HbgSlotClaimTest, GraphOuterTaskClaimsAPoisonedSlot) { ASSERT_NE(graph.recording_handle, nullptr); ASSERT_TRUE(orch.graph_prepare(graph.recording_handle, boundary_args)); - CoreTaskArgs node_args; - node_args.add_input(boundary); - ASSERT_TRUE(orch.submit_dummy_task(node_args).task_id().is_valid()); + CoreTaskArgs task_args; + task_args.add_input(boundary); + ASSERT_TRUE(orch.submit_dummy_task(task_args).task_id().is_valid()); ASSERT_TRUE(orch.graph_end()); ASSERT_EQ(orch.task_allocator.active_count(), 1); @@ -170,9 +170,9 @@ TEST_F(HbgSlotClaimTest, CachedGraphReplayClaimsAPoisonedSlot) { const GraphScopeResult recorded = orch.graph_begin(0x51ADC1A2, boundary_args, 0x1736); ASSERT_TRUE(recorded.recording); ASSERT_TRUE(orch.graph_prepare(recorded.recording_handle, boundary_args)); - CoreTaskArgs node_args; - node_args.add_input(boundary); - ASSERT_TRUE(orch.submit_dummy_task(node_args).task_id().is_valid()); + CoreTaskArgs task_args; + task_args.add_input(boundary); + ASSERT_TRUE(orch.submit_dummy_task(task_args).task_id().is_valid()); ASSERT_TRUE(orch.graph_end()); poison_slot(1); diff --git a/tests/ut/cpp/common/test_hbg_sm_compaction.cpp b/tests/ut/cpp/common/test_hbg_sm_compaction.cpp index 41fb08f07a..6d6d6a5d9a 100644 --- a/tests/ut/cpp/common/test_hbg_sm_compaction.cpp +++ b/tests/ut/cpp/common/test_hbg_sm_compaction.cpp @@ -105,7 +105,7 @@ class Mirror { payloads()[i].fanin_data()[j] = static_cast(0x50 + i * 0x10 + j); } slot_states()[i].last_consumer_local_id = static_cast(i); - slot_states()[i].graph_node_index = static_cast(200 + i); + slot_states()[i].in_graph_task_index = static_cast(200 + i); slot_states()[i].bind_buffers(&payloads()[i], &descriptors()[i]); completion_flags()[i].store(static_cast(i & 1), std::memory_order_relaxed); } @@ -228,7 +228,7 @@ TEST(HbgSmCompaction, CarriesEveryLiveSlotsContent) { EXPECT_EQ(compacted.payload_at(i)->tensor_count, TENSORS_PER_TASK) << "slot " << i; EXPECT_EQ(compacted.payload_at(i)->tensor_data()[0].buffer.addr, 0x1000 + i * 0x10) << "slot " << i; EXPECT_EQ(compacted.slot_states()[i].last_consumer_local_id, static_cast(i)) << "slot " << i; - EXPECT_EQ(compacted.slot_states()[i].graph_node_index, static_cast(200 + i)) << "slot " << i; + EXPECT_EQ(compacted.slot_states()[i].in_graph_task_index, static_cast(200 + i)) << "slot " << i; EXPECT_EQ(compacted.completion_flags()[i].load(std::memory_order_relaxed), static_cast(i & 1)) << "slot " << i; } diff --git a/tests/ut/cpp/common/test_host_phase_records.cpp b/tests/ut/cpp/common/test_host_phase_records.cpp index 6fe53d06b1..3f51134944 100644 --- a/tests/ut/cpp/common/test_host_phase_records.cpp +++ b/tests/ut/cpp/common/test_host_phase_records.cpp @@ -52,7 +52,7 @@ TEST(HostPhaseRecords, DisarmedPoolCollectsNothing) { // The producer's record call must tolerate a null pool — that is the common // configuration, where only the per-kind counters run. - record_n(pool, HostPhaseKind::OrchRecordNode, 10); + record_n(pool, HostPhaseKind::OrchRecordInGraphTask, 10); store.finish(0, /*invocation_id=*/9); EXPECT_TRUE(store.records().empty()); EXPECT_EQ(store.total_records(), 0u); @@ -64,7 +64,7 @@ TEST(HostPhaseRecords, RecordsSurviveInOrderWithinOneBuffer) { HostPhaseRecordPool *pool = store.arm(true); ASSERT_NE(pool, nullptr); - record_n(pool, HostPhaseKind::OrchRecordNode, 337, /*first_start=*/1000); + record_n(pool, HostPhaseKind::OrchRecordInGraphTask, 337, /*first_start=*/1000); store.finish(0, /*invocation_id=*/9); const auto records = store.records(); @@ -73,7 +73,7 @@ TEST(HostPhaseRecords, RecordsSurviveInOrderWithinOneBuffer) { EXPECT_EQ(store.dropped_records(), 0u); for (size_t i = 0; i < records.size(); ++i) { EXPECT_EQ(records[i].start_ns, 1000u + i) << "record " << i << " out of rotation order"; - EXPECT_EQ(records[i].kind, static_cast(HostPhaseKind::OrchRecordNode)); + EXPECT_EQ(records[i].kind, static_cast(HostPhaseKind::OrchRecordInGraphTask)); } } @@ -116,7 +116,7 @@ TEST(HostPhaseRecords, ConcurrentProducersLoseNothingAndDoNotOverlapSlots) { while (!go.load(std::memory_order_acquire)) {} // Disjoint start_ns ranges, so a lost or duplicated record is visible // as a hole or a repeat rather than needing per-thread bookkeeping. - record_n(pool, HostPhaseKind::OrchRecordNode, kPerThread, /*first_start=*/1 + t * kPerThread, t); + record_n(pool, HostPhaseKind::OrchRecordInGraphTask, kPerThread, /*first_start=*/1 + t * kPerThread, t); }); } go.store(true, std::memory_order_release); @@ -144,7 +144,7 @@ TEST(HostPhaseRecords, OverflowDropsAndCountsWithoutLosingEarlierRecords) { const uint32_t capacity = static_cast(HostPhaseRecordStore::capacity()); const uint32_t overshoot = 25; - record_n(pool, HostPhaseKind::OrchRecordNode, capacity + overshoot, /*first_start=*/1); + record_n(pool, HostPhaseKind::OrchRecordInGraphTask, capacity + overshoot, /*first_start=*/1); store.finish(0, /*invocation_id=*/9); const auto records = store.records(); @@ -174,7 +174,7 @@ TEST(HostPhaseRecords, AStoreReusingAnAddressDoesNotInheritACachedLane) { new (at_address) HostPhaseRecordStore(); HostPhaseRecordPool *pool = at_address->arm(true); ASSERT_NE(pool, nullptr); - record_n(pool, HostPhaseKind::OrchRecordNode, 100, /*first_start=*/1); + record_n(pool, HostPhaseKind::OrchRecordInGraphTask, 100, /*first_start=*/1); at_address->finish(0, /*invocation_id=*/9); ASSERT_EQ(at_address->records().size(), 100u); at_address->~HostPhaseRecordStore(); @@ -183,7 +183,7 @@ TEST(HostPhaseRecords, AStoreReusingAnAddressDoesNotInheritACachedLane) { new (at_address) HostPhaseRecordStore(); pool = at_address->arm(true); ASSERT_NE(pool, nullptr); - record_n(pool, HostPhaseKind::OrchRecordNode, 100, /*first_start=*/1000); + record_n(pool, HostPhaseKind::OrchRecordInGraphTask, 100, /*first_start=*/1000); at_address->finish(0, /*invocation_id=*/9); const auto records = at_address->records(); @@ -199,7 +199,7 @@ TEST(HostPhaseRecords, ReArmClearsThePreviousPass) { HostPhaseRecordStore store; HostPhaseRecordPool *pool = store.arm(true); ASSERT_NE(pool, nullptr); - record_n(pool, HostPhaseKind::OrchRecordNode, 50); + record_n(pool, HostPhaseKind::OrchRecordInGraphTask, 50); store.finish(7, /*invocation_id=*/9); EXPECT_EQ(store.records().size(), 50u); EXPECT_EQ(store.submitted_tasks(), 7u); @@ -219,7 +219,7 @@ TEST(HostPhaseRecords, ProducerThreadIdSurvivesTheArtifactStore) { HostPhaseRecordPool *pool = store.arm(true); ASSERT_NE(pool, nullptr); - record_n(pool, HostPhaseKind::OrchRecordNode, 1, /*first_start=*/10, /*thread_id=*/101); + record_n(pool, HostPhaseKind::OrchRecordInGraphTask, 1, /*first_start=*/10, /*thread_id=*/101); record_n(pool, HostPhaseKind::OrchGraphSubmit, 1, /*first_start=*/20, /*thread_id=*/202); store.finish(1, /*invocation_id=*/9); @@ -238,7 +238,7 @@ TEST(HostPhaseRecords, SubmitProjectionKeepsOnlyTaskSubmittingKinds) { // pass's total_tasks, and the sub-operations do not count. record_n(pool, HostPhaseKind::OrchSubmitTask, 5); record_n(pool, HostPhaseKind::OrchAllocTensors, 2); - record_n(pool, HostPhaseKind::OrchRecordNode, 277); + record_n(pool, HostPhaseKind::OrchRecordInGraphTask, 277); record_n(pool, HostPhaseKind::OrchGraphSubmit, 40); record_n(pool, HostPhaseKind::OrchBuildDefinition, 1); record_n(pool, HostPhaseKind::BindHostOrch, 1); @@ -274,7 +274,7 @@ TEST(HostPhaseRecords, APassIsWrittenAtMostOnceAndTheNextPassAppends) { HostPhaseRecordStore store; HostPhaseRecordPool *pool = store.arm(true); ASSERT_NE(pool, nullptr); - record_n(pool, HostPhaseKind::OrchRecordNode, 4); + record_n(pool, HostPhaseKind::OrchRecordInGraphTask, 4); store.finish(4, /*invocation_id=*/11); EXPECT_EQ(store.write_records_jsonl(path), 0); diff --git a/tests/ut/py/test_strace_timing.py b/tests/ut/py/test_strace_timing.py index 1e45fb5b79..74b7e48be8 100644 --- a/tests/ut/py/test_strace_timing.py +++ b/tests/ut/py/test_strace_timing.py @@ -985,7 +985,7 @@ def test_host_record_spans_nest_bind_segments_and_orchestrator_operations(tmp_pa "inv": 5, "records": [ {"phase": "args", "start_ns": 1_000, "end_ns": 1_100, "detail": 4096, "tid": 9}, - {"phase": "record_node", "start_ns": 1_150, "end_ns": 1_230, "detail": 4, "tid": 42}, + {"phase": "record_in_graph_task", "start_ns": 1_150, "end_ns": 1_230, "detail": 4, "tid": 42}, {"phase": "graph_submit", "start_ns": 1_200, "end_ns": 1_250, "detail": 77, "tid": 9}, ], } @@ -1001,7 +1001,7 @@ def test_host_record_spans_nest_bind_segments_and_orchestrator_operations(tmp_pa assert by_name["chip.run.bind.host_orch.graph_submit"].depth == bind_depth + 2 assert by_name["chip.run.bind.args"].ts == 1_000 assert by_name["chip.run.bind.args"].dur == 100 - assert by_name["chip.run.bind.host_orch.record_node"].tid == 42 + assert by_name["chip.run.bind.host_orch.record_in_graph_task"].tid == 42 assert by_name["chip.run.bind.host_orch.graph_submit"].tid == 9 trace = to_host_swimlane(spans + out) @@ -1014,6 +1014,39 @@ def test_host_record_spans_nest_bind_segments_and_orchestrator_operations(tmp_pa assert lane_names[42] == "graph record worker" +def test_host_record_spans_put_the_pre_rename_record_phase_on_the_recorder_lane(): + """A log written before the phase was renamed still lands on the recorder lane. + + The runtime emitted `record_node` for what is now `record_in_graph_task`, and + an unrecognised phase name is attributed to host_main rather than rejected — + so dropping the old spelling would silently redraw every archived log's + recorder work onto the main lane. + """ + spans = list(parse_spans([_span_record(pid=9, tid=9, inv=5, name="chip.run.bind", ts=900, dur=500)])) + passes = [ + { + "pid": 9, + "inv": 5, + "records": [ + {"phase": "record_node", "start_ns": 1_000, "end_ns": 1_080, "detail": 4, "tid": 42}, + ], + } + ] + + out, orphaned, _ = host_record_spans(spans, passes) + + assert orphaned == 0 + by_name = {span.name: span for span in out} + assert by_name["chip.run.bind.host_orch.record_node"].tid == 42 + trace = to_host_swimlane(spans + out) + lane_names = { + event["tid"]: event["args"]["name"] + for event in trace["traceEvents"] + if event["ph"] == "M" and event["name"] == "thread_name" + } + assert lane_names[42] == "graph record worker" + + def test_host_record_spans_keep_legacy_recording_on_main_lane(): spans = list(parse_spans([_span_record(pid=9, tid=9, inv=5, name="chip.run.bind", ts=900, dur=500)])) passes = [ @@ -1021,7 +1054,7 @@ def test_host_record_spans_keep_legacy_recording_on_main_lane(): "pid": 9, "inv": 5, "records": [ - {"phase": "record_node", "start_ns": 1_000, "end_ns": 1_080, "detail": 4}, + {"phase": "record_in_graph_task", "start_ns": 1_000, "end_ns": 1_080, "detail": 4}, {"phase": "build_definition", "start_ns": 1_100, "end_ns": 1_180, "detail": 4}, {"phase": "graph_submit", "start_ns": 1_200, "end_ns": 1_250, "detail": 77}, ], diff --git a/tests/ut/py/test_swimlane_converter.py b/tests/ut/py/test_swimlane_converter.py index 3cefe1600a..ebd6aa3278 100644 --- a/tests/ut/py/test_swimlane_converter.py +++ b/tests/ut/py/test_swimlane_converter.py @@ -490,9 +490,9 @@ def test_graph_prepare_phases_create_graph_execution_envelopes(tmp_path): out = tmp_path / "trace.json" outer_a = 3 outer_b = 7 - node_a0 = (1 << 32) | (outer_a << 10) - node_a1 = (1 << 32) | ((outer_a << 10) | 1) - node_b0 = (1 << 32) | (outer_b << 10) + task_a0 = (1 << 32) | (outer_a << 10) + task_a1 = (1 << 32) | ((outer_a << 10) | 1) + task_b0 = (1 << 32) | (outer_b << 10) scheduler_phases = [ [ { @@ -519,9 +519,9 @@ def test_graph_prepare_phases_create_graph_execution_envelopes(tmp_path): ] ] tasks = [ - _task_row(node_a0, 0, dispatch=2.0, start=2.2, end=3.0, receive=2.1), - _task_row(node_a1, 1, dispatch=3.2, start=3.4, end=4.0, receive=3.3), - _task_row(node_b0, 0, dispatch=5.3, start=5.5, end=6.0, receive=5.4), + _task_row(task_a0, 0, dispatch=2.0, start=2.2, end=3.0, receive=2.1), + _task_row(task_a1, 1, dispatch=3.2, start=3.4, end=4.0, receive=3.3), + _task_row(task_b0, 0, dispatch=5.3, start=5.5, end=6.0, receive=5.4), ] sc.generate_chrome_trace_json(tasks, str(out), scheduler_phases=scheduler_phases, core_to_thread=[0, 0]) @@ -534,7 +534,7 @@ def test_graph_prepare_phases_create_graph_execution_envelopes(tmp_path): ) graph_events = [event for event in events if event.get("cat") == "graph_execution"] assert [event["args"]["outer_task_id"] for event in graph_events] == [outer_a, outer_b] - assert graph_events[0]["args"]["visible_node_count"] == 2 + assert graph_events[0]["args"]["visible_in_graph_task_count"] == 2 assert graph_events[0]["args"]["prepare_slice_count"] == 2 assert graph_events[0]["ts"] == 1.0 assert graph_events[0]["dur"] == 4.0