Refactor: retire hbg logic that no longer fits host orchestration (1/n)

docs/investigations/2026-08-hbg-consumer-wait-cannot-observe-device.md

@@ -0,0 +1,102 @@
+# hbg: `rt_set_tensor_data`'s consumer-wait cannot observe device progress
+
+**Date**: 2026-08-28
+**Verdict**: **not fixed** — identified while retiring host-orchestration leftovers
+(#2068), left alone because the fix changes `rt_get_tensor_data` /
+`rt_set_tensor_data` semantics and first needs a verdict on which behavior is
+intended.
+
+## The defect
+
+`wait_for_tensor_ready()` in `src/common/host_build_graph/shared/runtime_core.cpp`
+has two halves. Under `host_build_graph` they have different fates, and the
+difference is not what a reader would guess.
+
+`rt_set_tensor_data` calls it with `wait_for_consumers = true`, which spins on:
+
+```cpp
+SharedMemoryTaskHeader &cons_tasks = orch.sm_header->tasks;
+while (cons_tasks.completed_watermark.load(std::memory_order_acquire) < slot.last_consumer_local_id) {
+```
+
+`orch.sm_header` is the **host mirror**. Three facts settle what that loop does:
+
+| Fact | Where |
+| ---- | ----- |
+| The mirror's watermark is initialized to `-1` | `shared/shared_memory.cpp` (`init_header`) |
+| `update_completed_watermark()` has exactly one caller, on the device | `{arch}/runtime/host_build_graph/runtime/scheduler/scheduler.h` (`on_mixed_task_complete`) |
+| `last_consumer_local_id` is seeded to the task's **own** local id at submit, i.e. `>= 0` | `shared/orchestrator.cpp` (`prepare_task`, and the Graph outer shell) |
+
+The device advances the watermark in *its* copy of the image; nothing writes the
+host's after `init_header`. So the comparison is `-1 < own_id`, which is
+**unconditionally true**, and the wait can only end at
+`TENSOR_DATA_TIMEOUT_MS` = 15 s with `SIMPLER_ERROR_TENSOR_WAIT_TIMEOUT`.
+
+**There is no subset that works.** In particular a producer completed inline on
+the host is *not* an exception: the seed is its own id rather than `-1`, so a
+producer with no consumers at all still fails the comparison. (An earlier draft
+of #2068's description claimed that exception; it is wrong, and this entry
+exists partly to keep that claim from being believed.)
+
+The producer half is a different story. `rt_get_tensor_data` calls with
+`wait_for_consumers = false`, which spins on `slot.task_state` instead — and
+`alloc_tensors` does call `mark_completed()` on the host for a hidden-alloc task.
+So **that** half does have a working case: a producer completed inline during
+orchestration is observable, and only a producer left to the device is not.
+
+## Why it looks intentional
+
+The function's own comment already says the timeout is not a synchronization
+mechanism:
+
+> The host builds the complete graph before device scheduling starts, so a live
+> device producer cannot complete during this call; the timeout remains a
+> defensive failure backstop rather than a synchronization mechanism for
+> orchestration code.
+
+And `ChipTaskSlotState::last_consumer_local_id`'s comment calls itself
+"inert-but-scaffolded for parity". Both readings are consistent with "hbg
+orchestration is a bind-time pass, so nothing it waits on can be in flight".
+
+What that reasoning does not cover is the seeded value. If the wait is meant to be
+a no-op whenever no consumer is live, the mirror-side comparison should be
+vacuous — and it would be, had the seed stayed `-1`. Seeding it to the task's own
+id makes the loop always spin, which turns a "defensive backstop" into a
+guaranteed 15 s stall for any caller that reaches it.
+
+## Why it was not fixed in #2068
+
+Everything else in that PR could be settled by grep and reachability: a field with
+no writer, a branch with no reachable condition, a name for an event that cannot
+happen. This one cannot. Two readings both fit the code, and they imply opposite
+fixes:
+
+1. **The degradation is intended.** hbg orchestration cannot observe device
+   progress by construction, so `rt_set_tensor_data`'s consumer-wait is
+   meaningless there and should be removed — along with the
+   `last_consumer_local_id` maintenance that only feeds it — rather than made to
+   work. `rt_set_tensor_data` then documents that it must not be called against a
+   tensor whose consumers are device-side.
+2. **A read is missing.** The wait is supposed to work, and the host should be
+   reading the device's watermark (a D2H of the header field, or a mapped view)
+   instead of its own mirror.
+
+Choosing wrongly is worse than leaving it: (1) deletes a guard that some caller
+may be relying on to fail loudly, and (2) adds a D2H to a path that currently
+touches no device memory, on a runtime whose bind latency is actively being
+optimized.
+
+## What a fix needs first
+
+- **Which callers reach the consumer-wait at all.** `rt_set_tensor_data` is
+  orchestration-facing API, so the callers are outside this repo. Nothing in
+  `examples/` or `tests/st/` exercises it against a device-side consumer today —
+  which is also why the 15 s stall has never been reported.
+- **Whether the same reading applies to the producer half.** It has a working case
+  (inline-completed producers), so its timeout is not vacuous in the same way, and
+  the two halves may not want the same treatment.
+
+## Related
+
+- [`2026-08-host-orch-phase-tail-is-page-faults.md`](2026-08-host-orch-phase-tail-is-page-faults.md)
+  — why adding a D2H to the bind path is not free in the way it looks.

docs/investigations/README.md

@@ -84,6 +84,8 @@ that ...".
 
 Newest first.
 
+- [2026-08 — hbg: `rt_set_tensor_data`'s consumer-wait cannot observe device progress](2026-08-hbg-consumer-wait-cannot-observe-device.md) — **not fixed**, found while retiring host-orchestration leftovers (#2068). `wait_for_tensor_ready(wait_for_consumers=true)` spins on the **host mirror's** `completed_watermark`, which only the device advances — in its own copy — so the comparison is `-1 < last_consumer_local_id`, unconditionally true, and the wait can only end at the 15 s `TENSOR_DATA_TIMEOUT_MS`. **No subset works**: a producer completed inline on the host is not an exception, because the seed is the task's own id rather than `-1`, so even a producer with no consumers stalls (an earlier draft of #2068's description claimed that exception — it is wrong). The producer half is different and does have a working case, since it reads `task_state`, which `alloc_tensors` sets host-side. Left alone because two readings fit the code and imply opposite fixes — delete the wait as meaningless under host orchestration, or add the D2H it is missing — and picking wrong either drops a guard some out-of-repo caller may rely on or puts a device read on the bind path
+
 - [2026-08 — The host-orchestration phase tail is page faults, not the code in the phase](2026-08-host-orch-phase-tail-is-page-faults.md) — root cause of the two shapes on every hbg host swimlane: 447 of the 449 `record_node` (now `record_in_graph_task`) calls above 10 µs took a minor fault, 19% of calls carry 79% of the phase, and a fault costs 14–33 µs here against 1.7 µs off-tree because the process's own `mmap`/`munmap` holds `mmap_lock` against every faulting thread — three 64 MiB unmaps in an off-tree reproducer recreate the whole distribution. The fault count is deterministic (1063/1065/1168 per two orchestrations) and drops to 29 when glibc keeps freed memory; the cost per fault varies 2.4× between runs of the same binary, which is the measurement noise that hid this for fourteen iterations. Refutes THP (`PR_SET_THP_DISABLE` leaves the count unchanged), preemption, and node shape; records why the tunables are not a fix (`args` regresses, and the probe build's −53% is the probe amplifying its own subject). Amended 2026-08-23: recording the Definition image into the retained upload staging (8 × ~126 KB per dsv4 bind, previously a vector per recording) moved `graph_upload`'s faults 38 → 1 per bind but left `host_orch`'s count unresolvable in both directions, because a freed 126 KB block is reused without re-faulting — size against glibc's mmap and trim thresholds, not byte count, decides what shows up in this tail. Amended 2026-08-25: retaining the 82 MB SM mirror on the runner (one buffer per pipeline slot instead of one per bind) removes an `mmap` + `munmap` of that size per bind — `hblkhd` stops returning to its pre-bind value on 6 of 6 binds, in both arms of two interleaved repetitions — and shows that a retained buffer must be handed over **uninitialized**: the first implementation used `std::vector::resize`, whose value-initialization faulted in all 20132 pages of the window on each rank's cold bind (~20k minflt against ~1100) and left the whole 82 MB resident. `host_orch`'s warm-bind fault count resolves in neither direction (base [1160, 1256] over eight binds, retained [181, 1268]), since the mirror is ~6 THP faults of a ~1200-fault bind; an earlier attribution of a warm-bind rise to glibc's dynamic mmap threshold is retracted there. That amendment also closes the entry's "Where a fix would go" list — items 1–3 shipped as #1981, item 4 as #1988 plus #2013, and item 3's flat-region form as #2015 — and records that none of them reached the ~1100 faults the submitting thread takes per bind, which is what is left. Amended again the same day: the claim in that amendment that pre-sizing the recorder's node and tensor storage would only help a cold bind is **retracted** — a slot-creation counter shows warm dsv4 binds still creating 1336 node slots of 1679, because #1981's retention is per thread while the pool hands bodies out through one shared FIFO. Reserving each node's own buffer to the cap makes it exactly one page and is *worse* than main (minflt 1070 → 2540); packing every body's tensors into one never-grown bump region is what helps (`record_node` warm min 1702/3423 → 1239/1563 µs). **Amended 2026-08-25 (last)**: the *count × price* framing this entry opened with is refuted by two arms pointing opposite ways — glibc keeping freed memory removes 86% of the faults and buys **no** time, while #2015 removes 6% and buys 29–43%. The count is not a lever; only the price is, and the in-tree `mmap_lock` writer that sets it is **`mprotect`**, which glibc uses to open a non-main arena (26 calls per bind in that band; those arenas only grow, `MADV_DONTNEED` there is 0). Every earlier strace here traced `madvise`/`mmap`/`munmap`/`brk` and not `mprotect`, which is why the in-tree source of the exclusion went unfound for three rounds. Consequence: the residual ~1100 resident-page re-faults per bind — survived #1981, #1988, #2013, #2015, mechanism undetermined — showed **no measurable latency change** when the tunable arm removed 86% of them, so they are **not currently established as a performance defect**; userspace tools are exhausted (`mincore` and pagemap both report presence, not writability), so pricing them at all needs `bpftrace` on `handle_mm_fault`. **Corrected 2026-08-26**: they are a **warm-up cost and they end** — at `f40cacf30`, `host_orch`'s per-bind `minflt` runs 989, 983 (cold), then 114, 173, 54, 3, 13, 13, 11, 8, and reaches 0 by the sixth bind on three independent runs. Every per-bind count in this entry came from three-to-six-round runs divided by the bind count, so each averaged two cold binds and three or four still-decaying ones under a steady-state label; the quantity being divided was never per-bind. Nothing "survived" the four changes, and the mechanism left undetermined for three rounds turned out not to need determining. Control plane at that commit: 0.529 ms min / 0.570 median over 8 warm binds (`host_orch` 0.341/0.364). The reusable half of this — that dropping the cold bind does not reach the steady state — is now a trap in [hbg-bind-phases.md](../dfx/hbg-bind-phases.md). Also carries the per-site fault table (two of its top three sites were deleted by #2019 and #2015 within days — keep the method, not the numbers) and the three tooling traps that each produced a wrong conclusion first
 - [2026-08 — hbg: per-block Graph Definitions and cross-layer reuse](2026-08-hbg-graph-block-decomposition.md) — adopted at seven Definitions covering all 43 layers, after #1929 replaced the single recording slot the first attempt measured against (it demoted a Graph whose key differed from the in-flight recording's: 79 of 82 intended submissions recorded, host tasks rose 1131 → 1486). Now host submissions 1131 → 129, `host_orch` −44% and `sm_h2d` −85%, but `graph_upload` +207% for seven images instead of one, so the control plane nets −17% at the per-phase floor and **nothing at the median** — a predictable cost traded for a lower floor and a 133%-wide spread that depends on seven recording threads getting CPU. Keeps the structural map that made the reuse provable (367 kernels → 169 classes / 132 by code alone; which blocks can share a Definition and why the hash-routed MoE cannot), the two indices and the last layer's different `hc_post` destination, and the arithmetic that a recorded node costs about what a submitted task costs so break-even sits near three occurrences
 - [2026-08 — hbg: uploading Graph Definitions once as shared device objects](2026-08-hbg-graph-definition-single-upload.md) — cut the per-replay 130 KB Definition re-serialization (image build 931→24 µs, orch total −54%), and confirmed the H2D stage is latency- not bandwidth-bound. **Amended 2026-08-18**: a `--rounds 3` split shows 12.19 of the residual 12.9 ms is the *one-time* `rtMalloc`+`memset` of 40 execution-storage blocks (~53 MB), not per-call latency — real per-call is ~17 µs, so batching the reference submissions is worth ≤0.6 ms, and 88% of this change's own cold-start gain (−1.879 ms in that split) came from execution storage shrinking rather than from the byte reduction it targeted. **Amended 2026-08-25**: the one-time verify gate this change introduced no longer hashes anything — `content_hash`, `verify_state` and the whole-image zero-fill are removed, since `graph_definition_array` plus `bind_graph_topology` already bound every device-side read, so the three ~1 MB passes per dsv4 bind bought no safety the structural checks did not

docs/troubleshooting/device-error-codes.md

@@ -32,9 +32,13 @@ grep -E "orch_error_code=|sched_error_code=|sub_class=|error detail:" <run log>
 The `host_build_graph` runtime runs orchestration on the host, transfers the
 prepared image to the AICPU, and runs scheduling there. The
 `tensormap_and_ringbuffer` runtime runs both orchestration and scheduling on the
-AICPU. On a fatal condition the runtime **latches** a code into the shared-memory
-header; the host reads it back in `validate_runtime_impl` and prints the lines
-above.
+AICPU. On a fatal condition the runtime **latches** a code, which the host reads
+back in `validate_runtime_impl` to print the lines above. Where it is latched
+follows where the reporter runs: a scheduler code goes into the shared-memory
+header, and so does an orchestrator code under `tensormap_and_ringbuffer`, whose
+orchestrator is on the AICPU. `host_build_graph`'s orchestrator is host-side, so
+its code stays in host memory (`OrchestratorState::fatal_code`) and never crosses
+to the device — a refused bind reports it without the device having run at all.
 
 At most one of `orch_error_code` (1-11) and `sched_error_code` (100+) is ever
 non-zero. `runtime_status` is just the latched code negated.
@@ -210,7 +214,7 @@ enforces coverage. Edit those and the log carries the new code correctly:
 | ---- | ----- |
 | runtime code names / descriptions / hints | `src/common/runtime_status/error_names.h` |
 | host-side CANN names / descriptions / hints | `src/common/platform/include/host/acl_error_names.h` |
-| `SCHEDULER_TIMEOUT` sub-class labels | `src/{arch}/runtime/{host_build_graph,tensormap_and_ringbuffer}/common/runtime_status.h` |
+| `SCHEDULER_TIMEOUT` sub-class labels | `src/common/host_build_graph/runtime_status.h`, `src/{arch}/runtime/tensormap_and_ringbuffer/common/runtime_status.h` |
 | completeness test | `tests/ut/cpp/common/test_error_code_names.cpp` |
 
 **This page does not need updating for a new code** — deliberately. The tables
@@ -220,7 +224,8 @@ time, so there is nothing to drift out of sync.
 
 ## References
 
-- Code definitions: `src/{arch}/runtime/{host_build_graph,tensormap_and_ringbuffer}/common/runtime_status.h`
+- Code definitions: `src/common/host_build_graph/runtime_status.h`,
+  `src/{arch}/runtime/tensormap_and_ringbuffer/common/runtime_status.h`
 - Host print site: `.../host/runtime_maker.cpp` (`validate_runtime_impl`)
 - Sub-class logic: `.../runtime/scheduler/scheduler_cold_path.cpp` (`classify_stall_reason`)
 - End-to-end negative tests: `tests/st/runtime_fatal_codes/`

src/a2a3/runtime/host_build_graph/aicpu/aicpu_executor.cpp

@@ -71,9 +71,8 @@ static int32_t read_runtime_status(Runtime *runtime) {
     }
 
     auto *header = static_cast<SharedMemoryHeader *>(sm);
-    int32_t orch_error_code = header->orch_error_code.load(std::memory_order_acquire);
     int32_t sched_error_code = header->sched_error_code.load(std::memory_order_acquire);
-    return runtime_status_from_error_codes(orch_error_code, sched_error_code);
+    return runtime_status_from_error_code(sched_error_code);
 }
 
 static RuntimeContext *rt{nullptr};
@@ -312,7 +311,7 @@ int32_t AicpuExecutor::run(Runtime *runtime) {
 
             sched_ctx_.bind_runtime(rt);
 
-            // Latch the host-built task count (on_orchestration_done sets total_tasks_)
+            // Latch the host-built task count (on_graph_attached sets total_tasks_)
             // BEFORE the runtime_init_ready_ release below — that store is the barrier
             // that unblocks the scheduler threads. Otherwise they would acquire
             // runtime_init_ready_ with total_tasks_=0 and race to an early exit before
@@ -322,9 +321,9 @@ int32_t AicpuExecutor::run(Runtime *runtime) {
             // called it in run_host_orchestration; the orchestrator's own
             // task-allocator pointers name host memory the device never reads, so
             // mark_done()'s active_count() read would dereference it and fault the
-            // AICPU. on_orchestration_done only needs total_tasks and the scalar
+            // AICPU. on_graph_attached only needs total_tasks and the scalar
             // orchestrator.inline_completed_tasks, both already valid.
-            sched_ctx_.on_orchestration_done(runtime, rt, thread_idx, runtime->host_total_tasks);
+            sched_ctx_.on_graph_attached(rt, thread_idx, runtime->host_total_tasks);
             LOG_INFO("Thread %d: host-orch boot complete (%d tasks)", thread_idx, runtime->host_total_tasks);
         }
 
@@ -360,7 +359,7 @@ int32_t AicpuExecutor::run(Runtime *runtime) {
     // Every AICPU thread schedules its assigned cores.
     if (!sched_ctx_.is_completed()) {
         if (rt == nullptr) {
-            LOG_ERROR("Thread %d: rt is null after orchestrator error, skipping dispatch", thread_idx);
+            LOG_ERROR("Thread %d: rt is null after a failed boot, skipping dispatch", thread_idx);
         } else {
             sched_ctx_.bind_runtime(rt);
             // 3S+1P: the last thread is the core-less resolution (P) thread; the

src/a2a3/runtime/host_build_graph/docs/RUNTIME_LOGIC.md

@@ -52,8 +52,9 @@ For each run, the host:
 5. finalizes task counts and the graph image; and
 6. copies the shared-memory image and the arena's copied zone to the device.
 
-An orchestration fatal stops this sequence and is propagated through
-`orch_error_code`.
+An orchestration fatal stops this sequence before the upload. The orchestrator
+runs on the host, so its code is latched in `OrchestratorState::fatal_code` and
+never reaches shared memory; the bind maps it onto the status the caller sees.
 
 ### 2.3 Device Execution and Teardown
 
@@ -335,9 +336,11 @@ a slot is used, preventing masked-slot aliasing. See
 
 ## 9. Errors and Diagnostics
 
-The runtime latches orchestration and scheduler errors in shared memory and maps
-them to the negative run status observed by the host. Important validation paths
-include:
+The runtime latches a fatal code and maps it to the negative run status the host
+observes. The two reporters latch in different places: a scheduler code goes into
+the shared-memory header, and an orchestration code into
+`OrchestratorState::fatal_code` in host memory, since this runtime's orchestrator
+runs on the host. Important validation paths include:
 
 - invalid arguments (`-5`);
 - sync-start residency violations (`-7`);