From ba213d6094caed3465c3cfcc3fa316b9ac481177 Mon Sep 17 00:00:00 2001 From: poursoul Date: Fri, 28 Aug 2026 20:24:57 -0700 Subject: [PATCH] Refactor: retire hbg logic that no longer fits host orchestration host_build_graph moved orchestration to the host, but much of its code still assumes the tensormap_and_ringbuffer world it was copied from, where an orchestrator and a scheduler share an AICPU and talk through shared memory. This retires what no longer fits: state nothing writes, branches nothing can reach, and names for events that cannot happen. One stage of that cleanup, not all of it. Scope is what could be established as dead or misnamed without changing behavior; a behavioral item found on the way is recorded at the end instead of fixed. Shared-memory state the host orchestrator vacated: - orchestrator_done: written by mark_done(), read by nothing. - total_size and task_descriptors_offset: only init/print/validate fodder; the device resolves every segment through sm_layout's offset arithmetic and bounds the region with attach_populated's image_bytes. - orch_error_code: the orchestrator is host-side, so its latched code never had a device reader. The bind already refuses to upload an image whose orchestration failed, which made the field provably zero on the device and all three scheduler branches testing it unreachable. Both struct sizes are unchanged (128 / 192 bytes) -- alignas padding absorbs the removals -- but nearly every surviving field moves, so the whole header pair is an ABI break for anyone mixing artifacts: - SharedMemoryTaskHeader: task_descriptors_offset is deleted at offset 64 and task_descriptors takes its alignas(64), so every field from there on shifts back 8 bytes -- task_payloads 80->72, slot_states 88->80, completion_flags 96->88, and total_tasks 104->96. total_tasks is the one the device reads out of the H2D'd header, so it now carries a layout assert of its own; the segment pointers are host-side only. - SharedMemoryHeader: sched_error_bitmap/code/thread move from 148/152/156 to 128/132/136. The host .so and the AICPU .so share both structs, so mixing an old and a new build reads the scheduler code -- and the device's task total -- at the wrong offset. pip install rebuilds every variant together, so this is a bisect hazard rather than a live one. Fatal state, one field instead of two: OrchestratorState::fatal_code. It replaces `bool fatal` outright, and a report naming no code latches SIMPLER_ERROR_EXPLICIT_ORCH_FATAL, so "is fatal" and "which code" can no longer disagree; is_fatal() derives the predicate rather than storing it twice. That normalization also applies before the log line, which otherwise read FATAL(code=0, latched=9) on that path -- a zero that looks like "no error", with the latched= suffix implying a competing writer that does not exist. The field is atomic and latched by CAS, because report_fatal does not run on one thread. A recording worker reaches it for everything the recording cannot answer locally with its `unsupported` flag: every submit entry validates its arguments ahead of its recording branch (submit_task, alloc_tensors), and rt_report_fatal is public enough for a Graph body to call directly. Meanwhile the bind thread reads is_fatal() at every entry. The old atomic CAS on the shared-memory field was what made first-writer-wins hold across those threads, so moving the state host-side has to keep it rather than fall back to a check-then-set. TaskAllocator::report_capacity_exhausted latches the same field under the same rule, stated at its own write rather than inherited from alloc()'s guard, so the two writers cannot drift. Pairing the recording lifecycle, which that cross-thread reporter makes reachable: a prepared recording must leave RECORDING, and graph_end is what retires it. rt_graph_end short-circuited on is_fatal() and returned true, so a fatal reported inside a body left the entry in RECORDING with its recorder thread_locals still bound -- and graph_commit's drain waits on recording_cv for exactly that transition, with no timeout, on the bind thread that is already failing. graph_end now retires the entry itself on a fatal and reports false, and the short-circuit is gone so that branch is reachable. The invariant is that graph_end retires the entry it bound on every path that has one, which is why no caller pairs a false return with an abort. It must not: graph_commit frees a drained entry after releasing recording_mutex, so a second abort would take that mutex and still touch freed memory. graph_end says so at its definition and rt_graph_end says what its false means, since the older `return true` on a null op was the only reading under which a caller-side abort looked justified. Names describing events that cannot happen here. Both were copied from tmr, where they are accurate -- that runtime's orchestrator thread does finish and does exit: - on_orchestration_done -> on_graph_attached. It runs on the boot leader, after attach_populated, and latches total_tasks_, sizes the per-S completed-task queues, folds inline_completed_tasks, and writes the DFX core map. The event it follows is the attach. - handle_orchestrator_exit -> check_exit_conditions. It runs on every dispatch-loop iteration, testing a latched scheduler error plus the completion count. With the orchestrator branch gone its error half was a duplicate of check_idle_fatal_error, so it now delegates that half and keeps only the count test the idle path has no count to make. tmr's functions of the same names are untouched, and so is the docs/investigations/ entry citing handle_orchestrator_exit -- that analysis is about tmr's copy, so a repo-wide rename would corrupt it. Other dead surface, each with no caller or no consumer: set_scheduler(), SharedMemoryHandle::validate() / print_layout(), completed_tasks_count(), CHIP_TASK_CONSUMED (hbg never advances past COMPLETED; tmr's enumerator of the same name stays), the sched_error_code store in derive_ready_queue_capacities together with its SharedMemoryHeader parameter, and runtime_status_from_error_codes' second parameter, which becomes runtime_status_from_error_code. The arch-local runtime/graph_{cache,execution,host_state}.h are deleted: each was a one-line forward to its src/common/host_build_graph twin. They outlived the move because the orchestration .so's include path carries src/common but not src/common/host_build_graph, so orchestration_api.h's bare-name includes could only resolve through the shim; naming the header with its prefix resolves it with no include-path change. Comments and docs the change falsified, or that were already wrong: - ChipTaskSlotState::task_state was documented as merely a host-visible mirror. For an IN_GRAPH task -- which holds no slot in the SM table and therefore no completion_flags byte -- it is the readiness truth the device itself polls via graph_first_unmet_producer. Only a GLOBAL task's task_state mirrors completion_flags. - The same paragraph credited an "allocator deadlock detector" as a reader; hbg's allocator reaches an immediate capacity verdict and never touches task_state. - ChipTaskState still said readiness derives from fanin_refcount, a field the polling model removed, and dispatch_fanin was described as its dual and as seeded "at wiring". - "rt is null after orchestrator error" named a cause that cannot reach the device; rt is null only after a failed boot. - FATAL_ERROR_CHECK_INTERVAL said the idle path checks an orchestrator error; it checks for a latched scheduler error. - device-error-codes.md said both runtimes latch into the shared-memory header, and its runtime_status.h paths were stale for hbg. Two tests cover the pairing, one per side of the wrapper boundary. test_hbg_graph_submit_failure covers the orchestrator: a worker prepares a recording, its body reports a fatal, and the case pins that end declines, that the entry left RECORDING, that the worker's thread_locals came back (a second recording on the same thread prepares), that graph_commit returns, and that the body's code survived commit's own SIMPLER_ERROR_INVALID_ARGS report. Without graph_end's fatal branch it walks into the unsupported-Definition debug_assert instead. test_hbg_graph_async_submit covers the wrapper itself, which the case above cannot reach: it calls OrchestratorState::graph_end() directly, so nothing in tests/ut/cpp exercised rt_graph_end at all -- neither the short-circuit nor a caller-side abort would have been caught. That suite already drives rt_submit_graph_impl against a stubbed ops table, so the new case latches a fatal from inside the recorded body and pins both halves: graph_end is reached (end_calls == 1, and it observes the fatal) and graph_abort is not (abort_calls == 0). Restoring either defect reddens it -- the old is_fatal() short-circuit fails the first two assertions, the caller-side abort fails the third. The stub's is_fatal and graph_end now read that fatal flag rather than answering false/true unconditionally, which leaves the existing four cases in that suite unchanged since none of them latches one. Found and left alone: wait_for_tensor_ready(wait_for_consumers=true) spins on the host mirror's completed_watermark, and update_completed_- watermark()'s only caller is the device scheduler, advancing the device's own copy. The host's stays at the -1 init_header wrote, while the wait compares it against last_consumer_local_id, which prepare_task seeds to the producer's own local id -- so the comparison is unsatisfiable for every producer, including one with no consumers at all, and that wait can only ever reach its 15 s timeout. Fixing it changes rt_get_tensor_data's external semantics and first needs a verdict on whether the degradation to a defensive backstop is intended. --- ...hbg-consumer-wait-cannot-observe-device.md | 102 +++++++++++++++ docs/investigations/README.md | 2 + docs/troubleshooting/device-error-codes.md | 15 ++- .../host_build_graph/aicpu/aicpu_executor.cpp | 11 +- .../host_build_graph/docs/RUNTIME_LOGIC.md | 13 +- .../host_build_graph/host/runtime_maker.cpp | 23 ++-- .../orchestration/orchestration_api.h | 38 ++++-- .../host_build_graph/runtime/graph_cache.h | 14 --- .../runtime/graph_execution.h | 14 --- .../runtime/graph_host_state.h | 14 --- .../runtime/scheduler/scheduler.h | 14 ++- .../runtime/scheduler/scheduler_cold_path.cpp | 74 ++++------- .../runtime/scheduler/scheduler_context.h | 25 ++-- .../runtime/scheduler/scheduler_dispatch.cpp | 5 +- .../runtime/scheduler/scheduler_types.h | 2 +- .../host_build_graph/aicpu/aicpu_executor.cpp | 11 +- .../host_build_graph/docs/RUNTIME_LOGIC.md | 13 +- .../host_build_graph/host/runtime_maker.cpp | 23 ++-- .../orchestration/orchestration_api.h | 38 ++++-- .../host_build_graph/runtime/graph_cache.h | 14 --- .../runtime/graph_execution.h | 14 --- .../runtime/graph_host_state.h | 14 --- .../runtime/scheduler/scheduler.h | 14 ++- .../runtime/scheduler/scheduler_cold_path.cpp | 74 ++++------- .../runtime/scheduler/scheduler_context.h | 25 ++-- .../runtime/scheduler/scheduler_dispatch.cpp | 5 +- .../runtime/scheduler/scheduler_types.h | 2 +- .../host/ready_queue_sizing.cpp | 7 +- src/common/host_build_graph/orchestrator.h | 32 ++++- .../host_build_graph/ready_queue_sizing.h | 4 +- src/common/host_build_graph/runtime.h | 2 +- src/common/host_build_graph/runtime_status.h | 13 +- src/common/host_build_graph/runtime_types.h | 59 +++++---- .../host_build_graph/shared/orchestrator.cpp | 70 +++++++---- .../host_build_graph/shared/runtime_core.cpp | 14 +-- .../host_build_graph/shared/runtime_init.cpp | 15 ++- .../host_build_graph/shared/shared_memory.cpp | 52 -------- src/common/host_build_graph/shared_memory.h | 64 ++++------ src/common/host_build_graph/task_allocator.h | 21 +++- .../common/test_hbg_graph_async_submit.cpp | 72 ++++++++++- .../test_hbg_graph_definition_arena.cpp | 6 +- .../common/test_hbg_graph_submit_failure.cpp | 117 ++++++++++++++---- .../cpp/common/test_hbg_ready_queue_seed.cpp | 6 +- .../ut/cpp/common/test_hbg_sm_compaction.cpp | 4 +- 44 files changed, 654 insertions(+), 517 deletions(-) create mode 100644 docs/investigations/2026-08-hbg-consumer-wait-cannot-observe-device.md delete mode 100644 src/a2a3/runtime/host_build_graph/runtime/graph_cache.h delete mode 100644 src/a2a3/runtime/host_build_graph/runtime/graph_execution.h delete mode 100644 src/a2a3/runtime/host_build_graph/runtime/graph_host_state.h delete mode 100644 src/a5/runtime/host_build_graph/runtime/graph_cache.h delete mode 100644 src/a5/runtime/host_build_graph/runtime/graph_execution.h delete mode 100644 src/a5/runtime/host_build_graph/runtime/graph_host_state.h diff --git a/docs/investigations/2026-08-hbg-consumer-wait-cannot-observe-device.md b/docs/investigations/2026-08-hbg-consumer-wait-cannot-observe-device.md new file mode 100644 index 0000000000..90c93f7f89 --- /dev/null +++ b/docs/investigations/2026-08-hbg-consumer-wait-cannot-observe-device.md @@ -0,0 +1,102 @@ +# hbg: `rt_set_tensor_data`'s consumer-wait cannot observe device progress + +**Date**: 2026-08-28 +**Verdict**: **not fixed** — identified while retiring host-orchestration leftovers +(#2068), left alone because the fix changes `rt_get_tensor_data` / +`rt_set_tensor_data` semantics and first needs a verdict on which behavior is +intended. + +## The defect + +`wait_for_tensor_ready()` in `src/common/host_build_graph/shared/runtime_core.cpp` +has two halves. Under `host_build_graph` they have different fates, and the +difference is not what a reader would guess. + +`rt_set_tensor_data` calls it with `wait_for_consumers = true`, which spins on: + +```cpp +SharedMemoryTaskHeader &cons_tasks = orch.sm_header->tasks; +while (cons_tasks.completed_watermark.load(std::memory_order_acquire) < slot.last_consumer_local_id) { +``` + +`orch.sm_header` is the **host mirror**. Three facts settle what that loop does: + +| Fact | Where | +| ---- | ----- | +| The mirror's watermark is initialized to `-1` | `shared/shared_memory.cpp` (`init_header`) | +| `update_completed_watermark()` has exactly one caller, on the device | `{arch}/runtime/host_build_graph/runtime/scheduler/scheduler.h` (`on_mixed_task_complete`) | +| `last_consumer_local_id` is seeded to the task's **own** local id at submit, i.e. `>= 0` | `shared/orchestrator.cpp` (`prepare_task`, and the Graph outer shell) | + +The device advances the watermark in *its* copy of the image; nothing writes the +host's after `init_header`. So the comparison is `-1 < own_id`, which is +**unconditionally true**, and the wait can only end at +`TENSOR_DATA_TIMEOUT_MS` = 15 s with `SIMPLER_ERROR_TENSOR_WAIT_TIMEOUT`. + +**There is no subset that works.** In particular a producer completed inline on +the host is *not* an exception: the seed is its own id rather than `-1`, so a +producer with no consumers at all still fails the comparison. (An earlier draft +of #2068's description claimed that exception; it is wrong, and this entry +exists partly to keep that claim from being believed.) + +The producer half is a different story. `rt_get_tensor_data` calls with +`wait_for_consumers = false`, which spins on `slot.task_state` instead — and +`alloc_tensors` does call `mark_completed()` on the host for a hidden-alloc task. +So **that** half does have a working case: a producer completed inline during +orchestration is observable, and only a producer left to the device is not. + +## Why it looks intentional + +The function's own comment already says the timeout is not a synchronization +mechanism: + +> The host builds the complete graph before device scheduling starts, so a live +> device producer cannot complete during this call; the timeout remains a +> defensive failure backstop rather than a synchronization mechanism for +> orchestration code. + +And `ChipTaskSlotState::last_consumer_local_id`'s comment calls itself +"inert-but-scaffolded for parity". Both readings are consistent with "hbg +orchestration is a bind-time pass, so nothing it waits on can be in flight". + +What that reasoning does not cover is the seeded value. If the wait is meant to be +a no-op whenever no consumer is live, the mirror-side comparison should be +vacuous — and it would be, had the seed stayed `-1`. Seeding it to the task's own +id makes the loop always spin, which turns a "defensive backstop" into a +guaranteed 15 s stall for any caller that reaches it. + +## Why it was not fixed in #2068 + +Everything else in that PR could be settled by grep and reachability: a field with +no writer, a branch with no reachable condition, a name for an event that cannot +happen. This one cannot. Two readings both fit the code, and they imply opposite +fixes: + +1. **The degradation is intended.** hbg orchestration cannot observe device + progress by construction, so `rt_set_tensor_data`'s consumer-wait is + meaningless there and should be removed — along with the + `last_consumer_local_id` maintenance that only feeds it — rather than made to + work. `rt_set_tensor_data` then documents that it must not be called against a + tensor whose consumers are device-side. +2. **A read is missing.** The wait is supposed to work, and the host should be + reading the device's watermark (a D2H of the header field, or a mapped view) + instead of its own mirror. + +Choosing wrongly is worse than leaving it: (1) deletes a guard that some caller +may be relying on to fail loudly, and (2) adds a D2H to a path that currently +touches no device memory, on a runtime whose bind latency is actively being +optimized. + +## What a fix needs first + +- **Which callers reach the consumer-wait at all.** `rt_set_tensor_data` is + orchestration-facing API, so the callers are outside this repo. Nothing in + `examples/` or `tests/st/` exercises it against a device-side consumer today — + which is also why the 15 s stall has never been reported. +- **Whether the same reading applies to the producer half.** It has a working case + (inline-completed producers), so its timeout is not vacuous in the same way, and + the two halves may not want the same treatment. + +## Related + +- [`2026-08-host-orch-phase-tail-is-page-faults.md`](2026-08-host-orch-phase-tail-is-page-faults.md) + — why adding a D2H to the bind path is not free in the way it looks. diff --git a/docs/investigations/README.md b/docs/investigations/README.md index 5acb46c6ab..70ad4c0b5b 100644 --- a/docs/investigations/README.md +++ b/docs/investigations/README.md @@ -84,6 +84,8 @@ that ...". Newest first. +- [2026-08 — hbg: `rt_set_tensor_data`'s consumer-wait cannot observe device progress](2026-08-hbg-consumer-wait-cannot-observe-device.md) — **not fixed**, found while retiring host-orchestration leftovers (#2068). `wait_for_tensor_ready(wait_for_consumers=true)` spins on the **host mirror's** `completed_watermark`, which only the device advances — in its own copy — so the comparison is `-1 < last_consumer_local_id`, unconditionally true, and the wait can only end at the 15 s `TENSOR_DATA_TIMEOUT_MS`. **No subset works**: a producer completed inline on the host is not an exception, because the seed is the task's own id rather than `-1`, so even a producer with no consumers stalls (an earlier draft of #2068's description claimed that exception — it is wrong). The producer half is different and does have a working case, since it reads `task_state`, which `alloc_tensors` sets host-side. Left alone because two readings fit the code and imply opposite fixes — delete the wait as meaningless under host orchestration, or add the D2H it is missing — and picking wrong either drops a guard some out-of-repo caller may rely on or puts a device read on the bind path + - [2026-08 — The host-orchestration phase tail is page faults, not the code in the phase](2026-08-host-orch-phase-tail-is-page-faults.md) — root cause of the two shapes on every hbg host swimlane: 447 of the 449 `record_node` (now `record_in_graph_task`) calls above 10 µs took a minor fault, 19% of calls carry 79% of the phase, and a fault costs 14–33 µs here against 1.7 µs off-tree because the process's own `mmap`/`munmap` holds `mmap_lock` against every faulting thread — three 64 MiB unmaps in an off-tree reproducer recreate the whole distribution. The fault count is deterministic (1063/1065/1168 per two orchestrations) and drops to 29 when glibc keeps freed memory; the cost per fault varies 2.4× between runs of the same binary, which is the measurement noise that hid this for fourteen iterations. Refutes THP (`PR_SET_THP_DISABLE` leaves the count unchanged), preemption, and node shape; records why the tunables are not a fix (`args` regresses, and the probe build's −53% is the probe amplifying its own subject). Amended 2026-08-23: recording the Definition image into the retained upload staging (8 × ~126 KB per dsv4 bind, previously a vector per recording) moved `graph_upload`'s faults 38 → 1 per bind but left `host_orch`'s count unresolvable in both directions, because a freed 126 KB block is reused without re-faulting — size against glibc's mmap and trim thresholds, not byte count, decides what shows up in this tail. Amended 2026-08-25: retaining the 82 MB SM mirror on the runner (one buffer per pipeline slot instead of one per bind) removes an `mmap` + `munmap` of that size per bind — `hblkhd` stops returning to its pre-bind value on 6 of 6 binds, in both arms of two interleaved repetitions — and shows that a retained buffer must be handed over **uninitialized**: the first implementation used `std::vector::resize`, whose value-initialization faulted in all 20132 pages of the window on each rank's cold bind (~20k minflt against ~1100) and left the whole 82 MB resident. `host_orch`'s warm-bind fault count resolves in neither direction (base [1160, 1256] over eight binds, retained [181, 1268]), since the mirror is ~6 THP faults of a ~1200-fault bind; an earlier attribution of a warm-bind rise to glibc's dynamic mmap threshold is retracted there. That amendment also closes the entry's "Where a fix would go" list — items 1–3 shipped as #1981, item 4 as #1988 plus #2013, and item 3's flat-region form as #2015 — and records that none of them reached the ~1100 faults the submitting thread takes per bind, which is what is left. Amended again the same day: the claim in that amendment that pre-sizing the recorder's node and tensor storage would only help a cold bind is **retracted** — a slot-creation counter shows warm dsv4 binds still creating 1336 node slots of 1679, because #1981's retention is per thread while the pool hands bodies out through one shared FIFO. Reserving each node's own buffer to the cap makes it exactly one page and is *worse* than main (minflt 1070 → 2540); packing every body's tensors into one never-grown bump region is what helps (`record_node` warm min 1702/3423 → 1239/1563 µs). **Amended 2026-08-25 (last)**: the *count × price* framing this entry opened with is refuted by two arms pointing opposite ways — glibc keeping freed memory removes 86% of the faults and buys **no** time, while #2015 removes 6% and buys 29–43%. The count is not a lever; only the price is, and the in-tree `mmap_lock` writer that sets it is **`mprotect`**, which glibc uses to open a non-main arena (26 calls per bind in that band; those arenas only grow, `MADV_DONTNEED` there is 0). Every earlier strace here traced `madvise`/`mmap`/`munmap`/`brk` and not `mprotect`, which is why the in-tree source of the exclusion went unfound for three rounds. Consequence: the residual ~1100 resident-page re-faults per bind — survived #1981, #1988, #2013, #2015, mechanism undetermined — showed **no measurable latency change** when the tunable arm removed 86% of them, so they are **not currently established as a performance defect**; userspace tools are exhausted (`mincore` and pagemap both report presence, not writability), so pricing them at all needs `bpftrace` on `handle_mm_fault`. **Corrected 2026-08-26**: they are a **warm-up cost and they end** — at `f40cacf30`, `host_orch`'s per-bind `minflt` runs 989, 983 (cold), then 114, 173, 54, 3, 13, 13, 11, 8, and reaches 0 by the sixth bind on three independent runs. Every per-bind count in this entry came from three-to-six-round runs divided by the bind count, so each averaged two cold binds and three or four still-decaying ones under a steady-state label; the quantity being divided was never per-bind. Nothing "survived" the four changes, and the mechanism left undetermined for three rounds turned out not to need determining. Control plane at that commit: 0.529 ms min / 0.570 median over 8 warm binds (`host_orch` 0.341/0.364). The reusable half of this — that dropping the cold bind does not reach the steady state — is now a trap in [hbg-bind-phases.md](../dfx/hbg-bind-phases.md). Also carries the per-site fault table (two of its top three sites were deleted by #2019 and #2015 within days — keep the method, not the numbers) and the three tooling traps that each produced a wrong conclusion first - [2026-08 — hbg: per-block Graph Definitions and cross-layer reuse](2026-08-hbg-graph-block-decomposition.md) — adopted at seven Definitions covering all 43 layers, after #1929 replaced the single recording slot the first attempt measured against (it demoted a Graph whose key differed from the in-flight recording's: 79 of 82 intended submissions recorded, host tasks rose 1131 → 1486). Now host submissions 1131 → 129, `host_orch` −44% and `sm_h2d` −85%, but `graph_upload` +207% for seven images instead of one, so the control plane nets −17% at the per-phase floor and **nothing at the median** — a predictable cost traded for a lower floor and a 133%-wide spread that depends on seven recording threads getting CPU. Keeps the structural map that made the reuse provable (367 kernels → 169 classes / 132 by code alone; which blocks can share a Definition and why the hash-routed MoE cannot), the two indices and the last layer's different `hc_post` destination, and the arithmetic that a recorded node costs about what a submitted task costs so break-even sits near three occurrences - [2026-08 — hbg: uploading Graph Definitions once as shared device objects](2026-08-hbg-graph-definition-single-upload.md) — cut the per-replay 130 KB Definition re-serialization (image build 931→24 µs, orch total −54%), and confirmed the H2D stage is latency- not bandwidth-bound. **Amended 2026-08-18**: a `--rounds 3` split shows 12.19 of the residual 12.9 ms is the *one-time* `rtMalloc`+`memset` of 40 execution-storage blocks (~53 MB), not per-call latency — real per-call is ~17 µs, so batching the reference submissions is worth ≤0.6 ms, and 88% of this change's own cold-start gain (−1.879 ms in that split) came from execution storage shrinking rather than from the byte reduction it targeted. **Amended 2026-08-25**: the one-time verify gate this change introduced no longer hashes anything — `content_hash`, `verify_state` and the whole-image zero-fill are removed, since `graph_definition_array` plus `bind_graph_topology` already bound every device-side read, so the three ~1 MB passes per dsv4 bind bought no safety the structural checks did not diff --git a/docs/troubleshooting/device-error-codes.md b/docs/troubleshooting/device-error-codes.md index b7067b6f15..00cb338bdf 100644 --- a/docs/troubleshooting/device-error-codes.md +++ b/docs/troubleshooting/device-error-codes.md @@ -32,9 +32,13 @@ grep -E "orch_error_code=|sched_error_code=|sub_class=|error detail:" The `host_build_graph` runtime runs orchestration on the host, transfers the prepared image to the AICPU, and runs scheduling there. The `tensormap_and_ringbuffer` runtime runs both orchestration and scheduling on the -AICPU. On a fatal condition the runtime **latches** a code into the shared-memory -header; the host reads it back in `validate_runtime_impl` and prints the lines -above. +AICPU. On a fatal condition the runtime **latches** a code, which the host reads +back in `validate_runtime_impl` to print the lines above. Where it is latched +follows where the reporter runs: a scheduler code goes into the shared-memory +header, and so does an orchestrator code under `tensormap_and_ringbuffer`, whose +orchestrator is on the AICPU. `host_build_graph`'s orchestrator is host-side, so +its code stays in host memory (`OrchestratorState::fatal_code`) and never crosses +to the device — a refused bind reports it without the device having run at all. At most one of `orch_error_code` (1-11) and `sched_error_code` (100+) is ever non-zero. `runtime_status` is just the latched code negated. @@ -210,7 +214,7 @@ enforces coverage. Edit those and the log carries the new code correctly: | ---- | ----- | | runtime code names / descriptions / hints | `src/common/runtime_status/error_names.h` | | host-side CANN names / descriptions / hints | `src/common/platform/include/host/acl_error_names.h` | -| `SCHEDULER_TIMEOUT` sub-class labels | `src/{arch}/runtime/{host_build_graph,tensormap_and_ringbuffer}/common/runtime_status.h` | +| `SCHEDULER_TIMEOUT` sub-class labels | `src/common/host_build_graph/runtime_status.h`, `src/{arch}/runtime/tensormap_and_ringbuffer/common/runtime_status.h` | | completeness test | `tests/ut/cpp/common/test_error_code_names.cpp` | **This page does not need updating for a new code** — deliberately. The tables @@ -220,7 +224,8 @@ time, so there is nothing to drift out of sync. ## References -- Code definitions: `src/{arch}/runtime/{host_build_graph,tensormap_and_ringbuffer}/common/runtime_status.h` +- Code definitions: `src/common/host_build_graph/runtime_status.h`, + `src/{arch}/runtime/tensormap_and_ringbuffer/common/runtime_status.h` - Host print site: `.../host/runtime_maker.cpp` (`validate_runtime_impl`) - Sub-class logic: `.../runtime/scheduler/scheduler_cold_path.cpp` (`classify_stall_reason`) - End-to-end negative tests: `tests/st/runtime_fatal_codes/` diff --git a/src/a2a3/runtime/host_build_graph/aicpu/aicpu_executor.cpp b/src/a2a3/runtime/host_build_graph/aicpu/aicpu_executor.cpp index 86193f2ec2..1dfee5ec96 100644 --- a/src/a2a3/runtime/host_build_graph/aicpu/aicpu_executor.cpp +++ b/src/a2a3/runtime/host_build_graph/aicpu/aicpu_executor.cpp @@ -71,9 +71,8 @@ static int32_t read_runtime_status(Runtime *runtime) { } auto *header = static_cast(sm); - int32_t orch_error_code = header->orch_error_code.load(std::memory_order_acquire); int32_t sched_error_code = header->sched_error_code.load(std::memory_order_acquire); - return runtime_status_from_error_codes(orch_error_code, sched_error_code); + return runtime_status_from_error_code(sched_error_code); } static RuntimeContext *rt{nullptr}; @@ -312,7 +311,7 @@ int32_t AicpuExecutor::run(Runtime *runtime) { sched_ctx_.bind_runtime(rt); - // Latch the host-built task count (on_orchestration_done sets total_tasks_) + // Latch the host-built task count (on_graph_attached sets total_tasks_) // BEFORE the runtime_init_ready_ release below — that store is the barrier // that unblocks the scheduler threads. Otherwise they would acquire // runtime_init_ready_ with total_tasks_=0 and race to an early exit before @@ -322,9 +321,9 @@ int32_t AicpuExecutor::run(Runtime *runtime) { // called it in run_host_orchestration; the orchestrator's own // task-allocator pointers name host memory the device never reads, so // mark_done()'s active_count() read would dereference it and fault the - // AICPU. on_orchestration_done only needs total_tasks and the scalar + // AICPU. on_graph_attached only needs total_tasks and the scalar // orchestrator.inline_completed_tasks, both already valid. - sched_ctx_.on_orchestration_done(runtime, rt, thread_idx, runtime->host_total_tasks); + sched_ctx_.on_graph_attached(rt, thread_idx, runtime->host_total_tasks); LOG_INFO("Thread %d: host-orch boot complete (%d tasks)", thread_idx, runtime->host_total_tasks); } @@ -360,7 +359,7 @@ int32_t AicpuExecutor::run(Runtime *runtime) { // Every AICPU thread schedules its assigned cores. if (!sched_ctx_.is_completed()) { if (rt == nullptr) { - LOG_ERROR("Thread %d: rt is null after orchestrator error, skipping dispatch", thread_idx); + LOG_ERROR("Thread %d: rt is null after a failed boot, skipping dispatch", thread_idx); } else { sched_ctx_.bind_runtime(rt); // 3S+1P: the last thread is the core-less resolution (P) thread; the diff --git a/src/a2a3/runtime/host_build_graph/docs/RUNTIME_LOGIC.md b/src/a2a3/runtime/host_build_graph/docs/RUNTIME_LOGIC.md index a5e7a8cd3c..788c93eb35 100644 --- a/src/a2a3/runtime/host_build_graph/docs/RUNTIME_LOGIC.md +++ b/src/a2a3/runtime/host_build_graph/docs/RUNTIME_LOGIC.md @@ -52,8 +52,9 @@ For each run, the host: 5. finalizes task counts and the graph image; and 6. copies the shared-memory image and the arena's copied zone to the device. -An orchestration fatal stops this sequence and is propagated through -`orch_error_code`. +An orchestration fatal stops this sequence before the upload. The orchestrator +runs on the host, so its code is latched in `OrchestratorState::fatal_code` and +never reaches shared memory; the bind maps it onto the status the caller sees. ### 2.3 Device Execution and Teardown @@ -335,9 +336,11 @@ a slot is used, preventing masked-slot aliasing. See ## 9. Errors and Diagnostics -The runtime latches orchestration and scheduler errors in shared memory and maps -them to the negative run status observed by the host. Important validation paths -include: +The runtime latches a fatal code and maps it to the negative run status the host +observes. The two reporters latch in different places: a scheduler code goes into +the shared-memory header, and an orchestration code into +`OrchestratorState::fatal_code` in host memory, since this runtime's orchestrator +runs on the host. Important validation paths include: - invalid arguments (`-5`); - sync-start residency violations (`-7`); diff --git a/src/a2a3/runtime/host_build_graph/host/runtime_maker.cpp b/src/a2a3/runtime/host_build_graph/host/runtime_maker.cpp index e6b85f5e8c..bd0754b4dc 100644 --- a/src/a2a3/runtime/host_build_graph/host/runtime_maker.cpp +++ b/src/a2a3/runtime/host_build_graph/host/runtime_maker.cpp @@ -56,9 +56,9 @@ #include "host_build_graph/runtime_status.h" #include "host_build_graph/common.h" #include "host_build_graph/dep_gen_host_graph.h" -#include "../runtime/graph_execution.h" +#include "host_build_graph/graph_execution.h" #include "host_build_graph/host_tensor_access.h" -#include "../runtime/graph_host_state.h" +#include "host_build_graph/graph_host_state.h" #include "host_build_graph/host_phase_trace.h" #include "host_build_graph/orchestrator.h" #include "host_build_graph/ready_queue_sizing.h" @@ -345,9 +345,8 @@ static int32_t read_runtime_status(Runtime *runtime, const HostApi *api, SharedM return 0; } - int32_t orch_error_code = host_header->orch_error_code.load(std::memory_order_relaxed); int32_t sched_error_code = host_header->sched_error_code.load(std::memory_order_relaxed); - return runtime_status_from_error_codes(orch_error_code, sched_error_code); + return runtime_status_from_error_code(sched_error_code); } namespace { @@ -734,14 +733,12 @@ int32_t run_host_orchestration( // described — a heap or tensormap exhaustion drops tasks, a fanin overflow drops // edges. Uploading it would launch the device on an incomplete graph and surface // the cause as whatever the device notices second, usually a scheduler timeout. - const int32_t orch_error = sm_layout::orch_error_code_addr(host_sm)->load(std::memory_order_acquire); - if (orch_error != SIMPLER_ERROR_NONE || orchestrator.fatal) { + if (orchestrator.is_fatal()) { // The latched code is the diagnosis, so it is what the caller sees — through the // same mapping the run path uses, since a caller cannot tell which of the two - // noticed. A fatal with no code left to read is the only generic failure. - const int32_t status = orch_error != SIMPLER_ERROR_NONE ? - runtime_status_from_error_codes(orch_error, SIMPLER_ERROR_NONE) : - PTO_RUNTIME_ERR_INTERNAL; + // noticed. + const int32_t orch_error = orchestrator.fatal_code.load(std::memory_order_acquire); + const int32_t status = runtime_status_from_error_code(orch_error); LOG_RUNTIME_FAILURE(orch_error, SIMPLER_ERROR_NONE, status); LOG_ERROR( "host-orch: refusing to upload an incomplete graph after %" PRIu64 " heap bytes", @@ -800,8 +797,7 @@ int32_t run_host_orchestration( } ReadyQueueCapacities ready_queue_capacities{}; - const int32_t ready_queue_status = - derive_ready_queue_capacities(ready_queue_populations, *host_sm_handle.header, &ready_queue_capacities); + const int32_t ready_queue_status = derive_ready_queue_capacities(ready_queue_populations, &ready_queue_capacities); if (ready_queue_status != 0) { LOG_ERROR( "host-orch: ready queue reachable population exceeds %" PRIu64 " (ready=%" PRIu64 "/%" PRIu64 "/%" PRIu64 @@ -1395,9 +1391,8 @@ extern "C" int validate_runtime_impl(Runtime *runtime, const HostApi *api, int e runtime_status = read_runtime_status(runtime, api, &host_header); } if (runtime_status != 0) { - int32_t orch_error_code = host_header.orch_error_code.load(std::memory_order_relaxed); int32_t sched_error_code = host_header.sched_error_code.load(std::memory_order_relaxed); - LOG_RUNTIME_FAILURE(orch_error_code, sched_error_code, runtime_status); + LOG_RUNTIME_FAILURE(SIMPLER_ERROR_NONE, sched_error_code, runtime_status); } if (skip_tensor_copy_back) { diff --git a/src/a2a3/runtime/host_build_graph/orchestration/orchestration_api.h b/src/a2a3/runtime/host_build_graph/orchestration/orchestration_api.h index ddca64d52c..6ee478a923 100644 --- a/src/a2a3/runtime/host_build_graph/orchestration/orchestration_api.h +++ b/src/a2a3/runtime/host_build_graph/orchestration/orchestration_api.h @@ -44,15 +44,15 @@ #include // Type headers needed by orchestration -#include "host_build_graph/common.h" // framework_bind_runtime / framework_current_runtime -#include "common/host_phase_kind.h" // HostPhaseKind, for the phase records below -#include "graph_cache.h" // Graph Execution key and result helpers -#include "graph_host_state.h" // GRAPH_MAX_DEFINITIONS -#include "host_build_graph/runtime_types.h" // SIMPLER_ERROR_* -#include "host_build_graph/submit_types.h" // MixedKernels, INVALID_KERNEL_ID, subtask slots -#include "types.h" // Arg, TaskOutputTensors, TensorArgType -#include "task_args.h" // ChipStorageTaskArgs, simpler::hbg::Tensor -#include "tensor.h" // simpler::hbg::Tensor, TensorCreateInfo +#include "host_build_graph/common.h" // framework_bind_runtime / framework_current_runtime +#include "common/host_phase_kind.h" // HostPhaseKind, for the phase records below +#include "host_build_graph/graph_cache.h" // Graph Execution key and result helpers +#include "host_build_graph/graph_host_state.h" // GRAPH_MAX_DEFINITIONS +#include "host_build_graph/runtime_types.h" // SIMPLER_ERROR_* +#include "host_build_graph/submit_types.h" // MixedKernels, INVALID_KERNEL_ID, subtask slots +#include "types.h" // Arg, TaskOutputTensors, TensorArgType +#include "task_args.h" // ChipStorageTaskArgs, simpler::hbg::Tensor +#include "tensor.h" // simpler::hbg::Tensor, TensorCreateInfo // ============================================================================= // simpler::hbg::Tensor Factory Helpers @@ -294,11 +294,20 @@ static inline void rt_graph_abort(void *recording_handle) { // Finish the recording pass and publish its Definition. The calling thread // finalizes the already-submitted outer Graph shells in rt_graph_commit. +// +// No is_fatal() short-circuit: a prepared recording has to leave RECORDING even when +// the run has already failed, or graph_commit's drain never completes. graph_end +// retires the entry it bound on every path that has one — published, unsupported, +// key-mismatched, or fatal — so `false` here means "no Definition", never "the entry +// is still yours to retire". That is why no caller pairs this with an abort. +// +// The null-op guard is defensive only: hbg's ops table always carries graph_end +// (runtime_core.cpp), and a table without it would have no graph_begin either, so +// no recording could be open to end. It reports false rather than the older `true` +// because nothing was published. static inline bool rt_graph_end() { RuntimeContext *rt = current_runtime(); - if (rt->ops->is_fatal(rt) || rt->ops->graph_end == nullptr) { - return true; - } + if (rt->ops->graph_end == nullptr) return false; return rt->ops->graph_end(rt); } @@ -544,6 +553,11 @@ static inline GraphSubmitResult rt_submit_graph_impl(uint64_t graph_key, const G return; } invoke(record_args); + // Not paired with an abort: graph_end retires the entry it bound on + // every path that has one, including the fatal one. A second abort + // would race graph_commit's drain, which frees the entry after + // releasing recording_mutex — so the mutex this would take is no + // protection against touching it. (void)rt_graph_end(); } catch (...) { rt_graph_abort(handle); diff --git a/src/a2a3/runtime/host_build_graph/runtime/graph_cache.h b/src/a2a3/runtime/host_build_graph/runtime/graph_cache.h deleted file mode 100644 index e99f37bb89..0000000000 --- a/src/a2a3/runtime/host_build_graph/runtime/graph_cache.h +++ /dev/null @@ -1,14 +0,0 @@ -/* - * Copyright (c) PyPTO Contributors. - * This program is free software, you can redistribute it and/or modify it under the terms and conditions of - * CANN Open Software License Agreement Version 2.0 (the "License"). - * Please refer to the License for details. You may not use this file except in compliance with the License. - * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, - * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. - * See LICENSE in the root of the software repository for the full text of the License. - * ----------------------------------------------------------------------------------------------------------- - */ - -#pragma once - -#include "host_build_graph/graph_cache.h" diff --git a/src/a2a3/runtime/host_build_graph/runtime/graph_execution.h b/src/a2a3/runtime/host_build_graph/runtime/graph_execution.h deleted file mode 100644 index c2176ce4d3..0000000000 --- a/src/a2a3/runtime/host_build_graph/runtime/graph_execution.h +++ /dev/null @@ -1,14 +0,0 @@ -/* - * Copyright (c) PyPTO Contributors. - * This program is free software, you can redistribute it and/or modify it under the terms and conditions of - * CANN Open Software License Agreement Version 2.0 (the "License"). - * Please refer to the License for details. You may not use this file except in compliance with the License. - * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, - * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. - * See LICENSE in the root of the software repository for the full text of the License. - * ----------------------------------------------------------------------------------------------------------- - */ - -#pragma once - -#include "host_build_graph/graph_execution.h" diff --git a/src/a2a3/runtime/host_build_graph/runtime/graph_host_state.h b/src/a2a3/runtime/host_build_graph/runtime/graph_host_state.h deleted file mode 100644 index 885c97660e..0000000000 --- a/src/a2a3/runtime/host_build_graph/runtime/graph_host_state.h +++ /dev/null @@ -1,14 +0,0 @@ -/* - * Copyright (c) PyPTO Contributors. - * This program is free software, you can redistribute it and/or modify it under the terms and conditions of - * CANN Open Software License Agreement Version 2.0 (the "License"). - * Please refer to the License for details. You may not use this file except in compliance with the License. - * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, - * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. - * See LICENSE in the root of the software repository for the full text of the License. - * ----------------------------------------------------------------------------------------------------------- - */ - -#pragma once - -#include "host_build_graph/graph_host_state.h" diff --git a/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler.h b/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler.h index 56bfc800f5..a560efb729 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler.h +++ b/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler.h @@ -14,11 +14,13 @@ * * The Scheduler is responsible for: * 1. Maintaining per-resource-shape ready queues - * 2. Polling-completion dependency resolution: a task is ready when every - * producer named in its inline fanin has set its completion_flags byte; - * a producer publishes completion + drains its wake list on finish - * 3. Publishing the host-visible task_state mirror (PENDING -> COMPLETED) and - * advancing the completed_watermark (consumer-retirement signal) + * 2. Polling-completion dependency resolution: a GLOBAL task is ready when every + * producer named in its inline fanin has set its completion_flags byte, an + * IN_GRAPH one when every producer in its Graph's fanin wire has reached + * task_state == COMPLETED (that table has no flag bytes); a producer publishes + * completion + drains its wake list on finish + * 3. Publishing task_state (PENDING -> COMPLETED) and advancing the + * completed_watermark (consumer-retirement signal) * 4. Two-stage mixed-task completion (subtask done bits -> mixed-task complete) * * The Scheduler runs on Device AI_CPU. host_build_graph is scheduler-only (the @@ -38,7 +40,7 @@ #include "utils/device_arena.h" #include "aicpu/platform_regs.h" // get_reg_ptr / RegId for the early-dispatch doorbell #include "async_wait.h" -#include "graph_execution.h" +#include "host_build_graph/graph_execution.h" #include "host_build_graph/task_id_encoding.h" #include "host_build_graph/task_allocator.h" #include "host_build_graph/runtime_types.h" diff --git a/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_cold_path.cpp b/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_cold_path.cpp index 29c54f266c..d0e33be23b 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_cold_path.cpp +++ b/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_cold_path.cpp @@ -52,30 +52,30 @@ void SchedulerContext::fail_scheduler(Runtime *runtime, int32_t thread_idx, int3 } } -LoopAction SchedulerContext::handle_orchestrator_exit( - int32_t thread_idx, SharedMemoryHeader *header, Runtime *runtime, int32_t &task_count -) { +LoopAction +SchedulerContext::check_latched_sched_error(int32_t thread_idx, SharedMemoryHeader *header, Runtime *runtime) { if (completed_.load(std::memory_order_acquire)) { return LoopAction::BREAK_LOOP; } - int32_t orch_err = header->orch_error_code.load(std::memory_order_acquire); - if (orch_err != SIMPLER_ERROR_NONE) { + int32_t sched_err = header->sched_error_code.load(std::memory_order_acquire); + if (sched_err != SIMPLER_ERROR_NONE) { LOG_ERROR( - "Thread %d: Fatal error (code=%d), sending EXIT_SIGNAL to all cores. " + "Thread %d: Scheduler fatal error detected (code=%d), sending EXIT_SIGNAL to all cores. " "completed_tasks=%d, total_tasks=%d", - thread_idx, orch_err, completed_tasks_.load(std::memory_order_relaxed), total_tasks_ + thread_idx, sched_err, completed_tasks_.load(std::memory_order_relaxed), total_tasks_ ); if (!completed_.exchange(true, std::memory_order_acq_rel)) { emergency_shutdown(runtime); } return LoopAction::BREAK_LOOP; } - int32_t sched_err = header->sched_error_code.load(std::memory_order_acquire); - if (sched_err != SIMPLER_ERROR_NONE) { - LOG_ERROR("Thread %d: Scheduler fatal error detected (code=%d)", thread_idx, sched_err); - if (!completed_.exchange(true, std::memory_order_acq_rel)) { - emergency_shutdown(runtime); - } + return LoopAction::NONE; +} + +LoopAction SchedulerContext::check_exit_conditions( + int32_t thread_idx, SharedMemoryHeader *header, Runtime *runtime, int32_t &task_count +) { + if (check_latched_sched_error(thread_idx, header, runtime) == LoopAction::BREAK_LOOP) { return LoopAction::BREAK_LOOP; } @@ -91,26 +91,7 @@ LoopAction SchedulerContext::handle_orchestrator_exit( } LoopAction SchedulerContext::check_idle_fatal_error(int32_t thread_idx, SharedMemoryHeader *header, Runtime *runtime) { - if (completed_.load(std::memory_order_acquire)) { - return LoopAction::BREAK_LOOP; - } - int32_t orch_err = header->orch_error_code.load(std::memory_order_acquire); - if (orch_err != SIMPLER_ERROR_NONE) { - LOG_ERROR("Thread %d: Fatal error detected (code=%d), sending EXIT_SIGNAL to all cores", thread_idx, orch_err); - if (!completed_.exchange(true, std::memory_order_acq_rel)) { - emergency_shutdown(runtime); - } - return LoopAction::BREAK_LOOP; - } - int32_t sched_err = header->sched_error_code.load(std::memory_order_acquire); - if (sched_err != SIMPLER_ERROR_NONE) { - LOG_ERROR("Thread %d: Scheduler fatal error detected (code=%d)", thread_idx, sched_err); - if (!completed_.exchange(true, std::memory_order_acq_rel)) { - emergency_shutdown(runtime); - } - return LoopAction::BREAK_LOOP; - } - return LoopAction::NONE; + return check_latched_sched_error(thread_idx, header, runtime); } // ============================================================================= @@ -906,7 +887,7 @@ int32_t SchedulerContext::post_handshake_init(Runtime *runtime) { // Initialize task counters. Task count comes from shared memory. // 0 is the correct count at boot: the graph is not attached yet, and - // on_orchestration_done latches the host-built total before releasing any + // on_graph_attached latches the host-built total before releasing any // scheduler thread. total_tasks_ = 0; completed_tasks_.store(0, std::memory_order_release); @@ -1028,15 +1009,13 @@ void SchedulerContext::bind_runtime(RuntimeContext *rt) { } // ============================================================================= -// Post-orchestration bookkeeping. Runs once on the boot leader after the -// host-built image is attached; latches total_tasks_ and folds inline-completed -// tasks (or shuts down on a fatal orchestration error). classify_ready_ is -// released after this call and is what publishes total_tasks_ to the peer threads, -// which acquire it before classify_partition reads the count. +// Post-attach bookkeeping. Runs once on the boot leader after the host-built +// image is attached; latches total_tasks_, sizes the per-S queues to it, and +// folds inline-completed tasks. classify_ready_ is released after this call and +// is what publishes total_tasks_ to the peer threads, which acquire it before +// classify_partition reads the count. // ============================================================================= -void SchedulerContext::on_orchestration_done( - Runtime *runtime, RuntimeContext *rt, [[maybe_unused]] int32_t thread_idx, int32_t total_tasks -) { +void SchedulerContext::on_graph_attached(RuntimeContext *rt, [[maybe_unused]] int32_t thread_idx, int32_t total_tasks) { total_tasks_ = total_tasks; // Allocate the per-S CompletedTaskQueues here on the boot leader, before it @@ -1065,17 +1044,6 @@ void SchedulerContext::on_orchestration_done( #endif } - // Check for fatal error from orchestration; if so, shut down immediately. - int32_t orch_err = 0; - if (sched_->sm_header) { - orch_err = sched_->sm_header->orch_error_code.load(std::memory_order_relaxed); - } - if (orch_err != SIMPLER_ERROR_NONE) { - if (!completed_.exchange(true, std::memory_order_acq_rel)) { - emergency_shutdown(runtime); - } - } - // The polling initial classify (seed the ready queues + wake lists for the // whole graph) runs AFTER this, partitioned across all AICPU threads in // classify_partition() — see AicpuExecutor::run. It is kept out of this diff --git a/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_context.h b/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_context.h index 5283b9d461..d50960e76b 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_context.h +++ b/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_context.h @@ -155,18 +155,18 @@ class SchedulerContext { // Orchestrator threads (core_trackers_[thread_idx].core_num() == 0) are a no-op. int32_t shutdown(int32_t thread_idx); - // Run all post-orchestration scheduler bookkeeping: + // Run all post-attach scheduler bookkeeping, once, on the boot leader: // - publishes core assignments to the perf collector (SIMPLER_DFX) - // - latches submitted task count from shared memory + // - latches the host-built task count + // - sizes the per-S completed-task queues to it // - folds inline_completed_tasks into completed_tasks_ - // (skipped on fatal error — emergency_shutdown runs instead) - // Callers must invoke rt_orchestration_done(rt) before this — that - // step belongs to the orchestrator lifecycle, not the scheduler. - void on_orchestration_done(Runtime *runtime, RuntimeContext *rt, int32_t thread_idx, int32_t total_tasks); + // The orchestration this graph came from ran to completion on the host, so + // there is no orchestrator lifecycle to hook here: the event is the attach. + void on_graph_attached(RuntimeContext *rt, int32_t thread_idx, int32_t total_tasks); // Seed the ready queues + wake lists for the whole graph at boot. Called by // every AICPU thread on a disjoint slice of the submitted-task range, after - // on_orchestration_done and before runtime_init_ready_ (the caller barriers + // on_graph_attached and before runtime_init_ready_ (the caller barriers // all threads between the two). Concurrency-safe: push_ready_routed and // register_wake are the same lock-free primitives used during the run. void classify_partition(int32_t thread_idx, int32_t nthreads); @@ -181,7 +181,6 @@ class SchedulerContext { int32_t aic_count() const { return aic_count_; } int32_t aiv_count() const { return aiv_count_; } bool is_completed() const { return completed_.load(std::memory_order_acquire); } - int32_t completed_tasks_count() const { return completed_tasks_.load(std::memory_order_acquire); } friend class SchedulerContextTestPeer; @@ -523,8 +522,16 @@ class SchedulerContext { // Cold path: exit checks, stall diagnostics, profiling (scheduler_cold_path.cpp) // ========================================================================= + // The latched-error test both exit checks below share. Deliberately NOT marked + // cold/noinline: it is inlined into each of them, so neither pays a frame for + // the shared half — check_exit_conditions runs on every dispatch pass. + LoopAction check_latched_sched_error(int32_t thread_idx, SharedMemoryHeader *header, Runtime *runtime); + + // Dispatch-loop exit check: latched scheduler error, then the completion + // count against the run's task total. check_idle_fatal_error below is the + // error-only half, for the idle path that has no count to compare. __attribute__((noinline, cold)) LoopAction - handle_orchestrator_exit(int32_t thread_idx, SharedMemoryHeader *header, Runtime *runtime, int32_t &task_count); + check_exit_conditions(int32_t thread_idx, SharedMemoryHeader *header, Runtime *runtime, int32_t &task_count); __attribute__((noinline, cold)) LoopAction check_idle_fatal_error(int32_t thread_idx, SharedMemoryHeader *header, Runtime *runtime); diff --git a/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_dispatch.cpp b/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_dispatch.cpp index 787d284961..e874594bb6 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_dispatch.cpp +++ b/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_dispatch.cpp @@ -952,8 +952,7 @@ int32_t SchedulerContext::run_resolution_thread(Runtime *runtime, int32_t thread #endif int32_t published_task_count = 0; - if (handle_orchestrator_exit(thread_idx, header, runtime, published_task_count) == LoopAction::BREAK_LOOP) - break; + if (check_exit_conditions(thread_idx, header, runtime, published_task_count) == LoopAction::BREAK_LOOP) break; int32_t resolved_this_pass = 0; bool resolved_any = false; @@ -1301,7 +1300,7 @@ int32_t SchedulerContext::resolve_and_dispatch(Runtime *runtime, int32_t thread_ #endif int32_t task_count = 0; if (!tracker.has_any_running_cores()) { - LoopAction action = handle_orchestrator_exit(thread_idx, header, runtime, task_count); + LoopAction action = check_exit_conditions(thread_idx, header, runtime, task_count); if (action == LoopAction::BREAK_LOOP) break; } diff --git a/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_types.h b/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_types.h index d1c9cfb92d..feedd2c4df 100644 --- a/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_types.h +++ b/src/a2a3/runtime/host_build_graph/runtime/scheduler/scheduler_types.h @@ -60,7 +60,7 @@ constexpr int32_t MAX_AICPU_THREADS = PLATFORM_MAX_AICPU_THREADS; // independent of the wall-clock timeout below: small enough to fire a few times // before the budget expires, large enough not to flood device_log. constexpr int32_t STALL_LOG_INTERVAL = 480000; -constexpr int32_t FATAL_ERROR_CHECK_INTERVAL = 1024; // Check orchestrator error every N idle iters +constexpr int32_t FATAL_ERROR_CHECK_INTERVAL = 1024; // Check for a latched scheduler error every N idle iters // Wall-clock budget for declaring "no progress = scheduler timeout". Replaces // the per-thread iteration-count cap that once lived here as MAX_IDLE_ITERATIONS diff --git a/src/a5/runtime/host_build_graph/aicpu/aicpu_executor.cpp b/src/a5/runtime/host_build_graph/aicpu/aicpu_executor.cpp index 86193f2ec2..1dfee5ec96 100644 --- a/src/a5/runtime/host_build_graph/aicpu/aicpu_executor.cpp +++ b/src/a5/runtime/host_build_graph/aicpu/aicpu_executor.cpp @@ -71,9 +71,8 @@ static int32_t read_runtime_status(Runtime *runtime) { } auto *header = static_cast(sm); - int32_t orch_error_code = header->orch_error_code.load(std::memory_order_acquire); int32_t sched_error_code = header->sched_error_code.load(std::memory_order_acquire); - return runtime_status_from_error_codes(orch_error_code, sched_error_code); + return runtime_status_from_error_code(sched_error_code); } static RuntimeContext *rt{nullptr}; @@ -312,7 +311,7 @@ int32_t AicpuExecutor::run(Runtime *runtime) { sched_ctx_.bind_runtime(rt); - // Latch the host-built task count (on_orchestration_done sets total_tasks_) + // Latch the host-built task count (on_graph_attached sets total_tasks_) // BEFORE the runtime_init_ready_ release below — that store is the barrier // that unblocks the scheduler threads. Otherwise they would acquire // runtime_init_ready_ with total_tasks_=0 and race to an early exit before @@ -322,9 +321,9 @@ int32_t AicpuExecutor::run(Runtime *runtime) { // called it in run_host_orchestration; the orchestrator's own // task-allocator pointers name host memory the device never reads, so // mark_done()'s active_count() read would dereference it and fault the - // AICPU. on_orchestration_done only needs total_tasks and the scalar + // AICPU. on_graph_attached only needs total_tasks and the scalar // orchestrator.inline_completed_tasks, both already valid. - sched_ctx_.on_orchestration_done(runtime, rt, thread_idx, runtime->host_total_tasks); + sched_ctx_.on_graph_attached(rt, thread_idx, runtime->host_total_tasks); LOG_INFO("Thread %d: host-orch boot complete (%d tasks)", thread_idx, runtime->host_total_tasks); } @@ -360,7 +359,7 @@ int32_t AicpuExecutor::run(Runtime *runtime) { // Every AICPU thread schedules its assigned cores. if (!sched_ctx_.is_completed()) { if (rt == nullptr) { - LOG_ERROR("Thread %d: rt is null after orchestrator error, skipping dispatch", thread_idx); + LOG_ERROR("Thread %d: rt is null after a failed boot, skipping dispatch", thread_idx); } else { sched_ctx_.bind_runtime(rt); // 3S+1P: the last thread is the core-less resolution (P) thread; the diff --git a/src/a5/runtime/host_build_graph/docs/RUNTIME_LOGIC.md b/src/a5/runtime/host_build_graph/docs/RUNTIME_LOGIC.md index a5e7a8cd3c..788c93eb35 100644 --- a/src/a5/runtime/host_build_graph/docs/RUNTIME_LOGIC.md +++ b/src/a5/runtime/host_build_graph/docs/RUNTIME_LOGIC.md @@ -52,8 +52,9 @@ For each run, the host: 5. finalizes task counts and the graph image; and 6. copies the shared-memory image and the arena's copied zone to the device. -An orchestration fatal stops this sequence and is propagated through -`orch_error_code`. +An orchestration fatal stops this sequence before the upload. The orchestrator +runs on the host, so its code is latched in `OrchestratorState::fatal_code` and +never reaches shared memory; the bind maps it onto the status the caller sees. ### 2.3 Device Execution and Teardown @@ -335,9 +336,11 @@ a slot is used, preventing masked-slot aliasing. See ## 9. Errors and Diagnostics -The runtime latches orchestration and scheduler errors in shared memory and maps -them to the negative run status observed by the host. Important validation paths -include: +The runtime latches a fatal code and maps it to the negative run status the host +observes. The two reporters latch in different places: a scheduler code goes into +the shared-memory header, and an orchestration code into +`OrchestratorState::fatal_code` in host memory, since this runtime's orchestrator +runs on the host. Important validation paths include: - invalid arguments (`-5`); - sync-start residency violations (`-7`); diff --git a/src/a5/runtime/host_build_graph/host/runtime_maker.cpp b/src/a5/runtime/host_build_graph/host/runtime_maker.cpp index d0f6dca224..4c08150b3f 100644 --- a/src/a5/runtime/host_build_graph/host/runtime_maker.cpp +++ b/src/a5/runtime/host_build_graph/host/runtime_maker.cpp @@ -56,9 +56,9 @@ #include "host_build_graph/runtime_status.h" #include "host_build_graph/common.h" #include "host_build_graph/dep_gen_host_graph.h" -#include "../runtime/graph_execution.h" +#include "host_build_graph/graph_execution.h" #include "host_build_graph/host_tensor_access.h" -#include "../runtime/graph_host_state.h" +#include "host_build_graph/graph_host_state.h" #include "host_build_graph/host_phase_trace.h" #include "host_build_graph/orchestrator.h" #include "host_build_graph/ready_queue_sizing.h" @@ -345,9 +345,8 @@ static int32_t read_runtime_status(Runtime *runtime, const HostApi *api, SharedM return 0; } - int32_t orch_error_code = host_header->orch_error_code.load(std::memory_order_relaxed); int32_t sched_error_code = host_header->sched_error_code.load(std::memory_order_relaxed); - return runtime_status_from_error_codes(orch_error_code, sched_error_code); + return runtime_status_from_error_code(sched_error_code); } namespace { @@ -734,14 +733,12 @@ int32_t run_host_orchestration( // described — a heap or tensormap exhaustion drops tasks, a fanin overflow drops // edges. Uploading it would launch the device on an incomplete graph and surface // the cause as whatever the device notices second, usually a scheduler timeout. - const int32_t orch_error = sm_layout::orch_error_code_addr(host_sm)->load(std::memory_order_acquire); - if (orch_error != SIMPLER_ERROR_NONE || orchestrator.fatal) { + if (orchestrator.is_fatal()) { // The latched code is the diagnosis, so it is what the caller sees — through the // same mapping the run path uses, since a caller cannot tell which of the two - // noticed. A fatal with no code left to read is the only generic failure. - const int32_t status = orch_error != SIMPLER_ERROR_NONE ? - runtime_status_from_error_codes(orch_error, SIMPLER_ERROR_NONE) : - PTO_RUNTIME_ERR_INTERNAL; + // noticed. + const int32_t orch_error = orchestrator.fatal_code.load(std::memory_order_acquire); + const int32_t status = runtime_status_from_error_code(orch_error); LOG_RUNTIME_FAILURE(orch_error, SIMPLER_ERROR_NONE, status); LOG_ERROR( "host-orch: refusing to upload an incomplete graph after %" PRIu64 " heap bytes", @@ -800,8 +797,7 @@ int32_t run_host_orchestration( } ReadyQueueCapacities ready_queue_capacities{}; - const int32_t ready_queue_status = - derive_ready_queue_capacities(ready_queue_populations, *host_sm_handle.header, &ready_queue_capacities); + const int32_t ready_queue_status = derive_ready_queue_capacities(ready_queue_populations, &ready_queue_capacities); if (ready_queue_status != 0) { LOG_ERROR( "host-orch: ready queue reachable population exceeds %" PRIu64 " (ready=%" PRIu64 "/%" PRIu64 "/%" PRIu64 @@ -1395,9 +1391,8 @@ extern "C" int validate_runtime_impl(Runtime *runtime, const HostApi *api, int e runtime_status = read_runtime_status(runtime, api, &host_header); } if (runtime_status != 0) { - int32_t orch_error_code = host_header.orch_error_code.load(std::memory_order_relaxed); int32_t sched_error_code = host_header.sched_error_code.load(std::memory_order_relaxed); - LOG_RUNTIME_FAILURE(orch_error_code, sched_error_code, runtime_status); + LOG_RUNTIME_FAILURE(SIMPLER_ERROR_NONE, sched_error_code, runtime_status); } if (skip_tensor_copy_back) { diff --git a/src/a5/runtime/host_build_graph/orchestration/orchestration_api.h b/src/a5/runtime/host_build_graph/orchestration/orchestration_api.h index ddca64d52c..6ee478a923 100644 --- a/src/a5/runtime/host_build_graph/orchestration/orchestration_api.h +++ b/src/a5/runtime/host_build_graph/orchestration/orchestration_api.h @@ -44,15 +44,15 @@ #include // Type headers needed by orchestration -#include "host_build_graph/common.h" // framework_bind_runtime / framework_current_runtime -#include "common/host_phase_kind.h" // HostPhaseKind, for the phase records below -#include "graph_cache.h" // Graph Execution key and result helpers -#include "graph_host_state.h" // GRAPH_MAX_DEFINITIONS -#include "host_build_graph/runtime_types.h" // SIMPLER_ERROR_* -#include "host_build_graph/submit_types.h" // MixedKernels, INVALID_KERNEL_ID, subtask slots -#include "types.h" // Arg, TaskOutputTensors, TensorArgType -#include "task_args.h" // ChipStorageTaskArgs, simpler::hbg::Tensor -#include "tensor.h" // simpler::hbg::Tensor, TensorCreateInfo +#include "host_build_graph/common.h" // framework_bind_runtime / framework_current_runtime +#include "common/host_phase_kind.h" // HostPhaseKind, for the phase records below +#include "host_build_graph/graph_cache.h" // Graph Execution key and result helpers +#include "host_build_graph/graph_host_state.h" // GRAPH_MAX_DEFINITIONS +#include "host_build_graph/runtime_types.h" // SIMPLER_ERROR_* +#include "host_build_graph/submit_types.h" // MixedKernels, INVALID_KERNEL_ID, subtask slots +#include "types.h" // Arg, TaskOutputTensors, TensorArgType +#include "task_args.h" // ChipStorageTaskArgs, simpler::hbg::Tensor +#include "tensor.h" // simpler::hbg::Tensor, TensorCreateInfo // ============================================================================= // simpler::hbg::Tensor Factory Helpers @@ -294,11 +294,20 @@ static inline void rt_graph_abort(void *recording_handle) { // Finish the recording pass and publish its Definition. The calling thread // finalizes the already-submitted outer Graph shells in rt_graph_commit. +// +// No is_fatal() short-circuit: a prepared recording has to leave RECORDING even when +// the run has already failed, or graph_commit's drain never completes. graph_end +// retires the entry it bound on every path that has one — published, unsupported, +// key-mismatched, or fatal — so `false` here means "no Definition", never "the entry +// is still yours to retire". That is why no caller pairs this with an abort. +// +// The null-op guard is defensive only: hbg's ops table always carries graph_end +// (runtime_core.cpp), and a table without it would have no graph_begin either, so +// no recording could be open to end. It reports false rather than the older `true` +// because nothing was published. static inline bool rt_graph_end() { RuntimeContext *rt = current_runtime(); - if (rt->ops->is_fatal(rt) || rt->ops->graph_end == nullptr) { - return true; - } + if (rt->ops->graph_end == nullptr) return false; return rt->ops->graph_end(rt); } @@ -544,6 +553,11 @@ static inline GraphSubmitResult rt_submit_graph_impl(uint64_t graph_key, const G return; } invoke(record_args); + // Not paired with an abort: graph_end retires the entry it bound on + // every path that has one, including the fatal one. A second abort + // would race graph_commit's drain, which frees the entry after + // releasing recording_mutex — so the mutex this would take is no + // protection against touching it. (void)rt_graph_end(); } catch (...) { rt_graph_abort(handle); diff --git a/src/a5/runtime/host_build_graph/runtime/graph_cache.h b/src/a5/runtime/host_build_graph/runtime/graph_cache.h deleted file mode 100644 index e99f37bb89..0000000000 --- a/src/a5/runtime/host_build_graph/runtime/graph_cache.h +++ /dev/null @@ -1,14 +0,0 @@ -/* - * Copyright (c) PyPTO Contributors. - * This program is free software, you can redistribute it and/or modify it under the terms and conditions of - * CANN Open Software License Agreement Version 2.0 (the "License"). - * Please refer to the License for details. You may not use this file except in compliance with the License. - * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, - * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. - * See LICENSE in the root of the software repository for the full text of the License. - * ----------------------------------------------------------------------------------------------------------- - */ - -#pragma once - -#include "host_build_graph/graph_cache.h" diff --git a/src/a5/runtime/host_build_graph/runtime/graph_execution.h b/src/a5/runtime/host_build_graph/runtime/graph_execution.h deleted file mode 100644 index c2176ce4d3..0000000000 --- a/src/a5/runtime/host_build_graph/runtime/graph_execution.h +++ /dev/null @@ -1,14 +0,0 @@ -/* - * Copyright (c) PyPTO Contributors. - * This program is free software, you can redistribute it and/or modify it under the terms and conditions of - * CANN Open Software License Agreement Version 2.0 (the "License"). - * Please refer to the License for details. You may not use this file except in compliance with the License. - * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, - * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. - * See LICENSE in the root of the software repository for the full text of the License. - * ----------------------------------------------------------------------------------------------------------- - */ - -#pragma once - -#include "host_build_graph/graph_execution.h" diff --git a/src/a5/runtime/host_build_graph/runtime/graph_host_state.h b/src/a5/runtime/host_build_graph/runtime/graph_host_state.h deleted file mode 100644 index 885c97660e..0000000000 --- a/src/a5/runtime/host_build_graph/runtime/graph_host_state.h +++ /dev/null @@ -1,14 +0,0 @@ -/* - * Copyright (c) PyPTO Contributors. - * This program is free software, you can redistribute it and/or modify it under the terms and conditions of - * CANN Open Software License Agreement Version 2.0 (the "License"). - * Please refer to the License for details. You may not use this file except in compliance with the License. - * THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, - * INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. - * See LICENSE in the root of the software repository for the full text of the License. - * ----------------------------------------------------------------------------------------------------------- - */ - -#pragma once - -#include "host_build_graph/graph_host_state.h" diff --git a/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler.h b/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler.h index 56bfc800f5..a560efb729 100644 --- a/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler.h +++ b/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler.h @@ -14,11 +14,13 @@ * * The Scheduler is responsible for: * 1. Maintaining per-resource-shape ready queues - * 2. Polling-completion dependency resolution: a task is ready when every - * producer named in its inline fanin has set its completion_flags byte; - * a producer publishes completion + drains its wake list on finish - * 3. Publishing the host-visible task_state mirror (PENDING -> COMPLETED) and - * advancing the completed_watermark (consumer-retirement signal) + * 2. Polling-completion dependency resolution: a GLOBAL task is ready when every + * producer named in its inline fanin has set its completion_flags byte, an + * IN_GRAPH one when every producer in its Graph's fanin wire has reached + * task_state == COMPLETED (that table has no flag bytes); a producer publishes + * completion + drains its wake list on finish + * 3. Publishing task_state (PENDING -> COMPLETED) and advancing the + * completed_watermark (consumer-retirement signal) * 4. Two-stage mixed-task completion (subtask done bits -> mixed-task complete) * * The Scheduler runs on Device AI_CPU. host_build_graph is scheduler-only (the @@ -38,7 +40,7 @@ #include "utils/device_arena.h" #include "aicpu/platform_regs.h" // get_reg_ptr / RegId for the early-dispatch doorbell #include "async_wait.h" -#include "graph_execution.h" +#include "host_build_graph/graph_execution.h" #include "host_build_graph/task_id_encoding.h" #include "host_build_graph/task_allocator.h" #include "host_build_graph/runtime_types.h" diff --git a/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_cold_path.cpp b/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_cold_path.cpp index 29c54f266c..d0e33be23b 100644 --- a/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_cold_path.cpp +++ b/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_cold_path.cpp @@ -52,30 +52,30 @@ void SchedulerContext::fail_scheduler(Runtime *runtime, int32_t thread_idx, int3 } } -LoopAction SchedulerContext::handle_orchestrator_exit( - int32_t thread_idx, SharedMemoryHeader *header, Runtime *runtime, int32_t &task_count -) { +LoopAction +SchedulerContext::check_latched_sched_error(int32_t thread_idx, SharedMemoryHeader *header, Runtime *runtime) { if (completed_.load(std::memory_order_acquire)) { return LoopAction::BREAK_LOOP; } - int32_t orch_err = header->orch_error_code.load(std::memory_order_acquire); - if (orch_err != SIMPLER_ERROR_NONE) { + int32_t sched_err = header->sched_error_code.load(std::memory_order_acquire); + if (sched_err != SIMPLER_ERROR_NONE) { LOG_ERROR( - "Thread %d: Fatal error (code=%d), sending EXIT_SIGNAL to all cores. " + "Thread %d: Scheduler fatal error detected (code=%d), sending EXIT_SIGNAL to all cores. " "completed_tasks=%d, total_tasks=%d", - thread_idx, orch_err, completed_tasks_.load(std::memory_order_relaxed), total_tasks_ + thread_idx, sched_err, completed_tasks_.load(std::memory_order_relaxed), total_tasks_ ); if (!completed_.exchange(true, std::memory_order_acq_rel)) { emergency_shutdown(runtime); } return LoopAction::BREAK_LOOP; } - int32_t sched_err = header->sched_error_code.load(std::memory_order_acquire); - if (sched_err != SIMPLER_ERROR_NONE) { - LOG_ERROR("Thread %d: Scheduler fatal error detected (code=%d)", thread_idx, sched_err); - if (!completed_.exchange(true, std::memory_order_acq_rel)) { - emergency_shutdown(runtime); - } + return LoopAction::NONE; +} + +LoopAction SchedulerContext::check_exit_conditions( + int32_t thread_idx, SharedMemoryHeader *header, Runtime *runtime, int32_t &task_count +) { + if (check_latched_sched_error(thread_idx, header, runtime) == LoopAction::BREAK_LOOP) { return LoopAction::BREAK_LOOP; } @@ -91,26 +91,7 @@ LoopAction SchedulerContext::handle_orchestrator_exit( } LoopAction SchedulerContext::check_idle_fatal_error(int32_t thread_idx, SharedMemoryHeader *header, Runtime *runtime) { - if (completed_.load(std::memory_order_acquire)) { - return LoopAction::BREAK_LOOP; - } - int32_t orch_err = header->orch_error_code.load(std::memory_order_acquire); - if (orch_err != SIMPLER_ERROR_NONE) { - LOG_ERROR("Thread %d: Fatal error detected (code=%d), sending EXIT_SIGNAL to all cores", thread_idx, orch_err); - if (!completed_.exchange(true, std::memory_order_acq_rel)) { - emergency_shutdown(runtime); - } - return LoopAction::BREAK_LOOP; - } - int32_t sched_err = header->sched_error_code.load(std::memory_order_acquire); - if (sched_err != SIMPLER_ERROR_NONE) { - LOG_ERROR("Thread %d: Scheduler fatal error detected (code=%d)", thread_idx, sched_err); - if (!completed_.exchange(true, std::memory_order_acq_rel)) { - emergency_shutdown(runtime); - } - return LoopAction::BREAK_LOOP; - } - return LoopAction::NONE; + return check_latched_sched_error(thread_idx, header, runtime); } // ============================================================================= @@ -906,7 +887,7 @@ int32_t SchedulerContext::post_handshake_init(Runtime *runtime) { // Initialize task counters. Task count comes from shared memory. // 0 is the correct count at boot: the graph is not attached yet, and - // on_orchestration_done latches the host-built total before releasing any + // on_graph_attached latches the host-built total before releasing any // scheduler thread. total_tasks_ = 0; completed_tasks_.store(0, std::memory_order_release); @@ -1028,15 +1009,13 @@ void SchedulerContext::bind_runtime(RuntimeContext *rt) { } // ============================================================================= -// Post-orchestration bookkeeping. Runs once on the boot leader after the -// host-built image is attached; latches total_tasks_ and folds inline-completed -// tasks (or shuts down on a fatal orchestration error). classify_ready_ is -// released after this call and is what publishes total_tasks_ to the peer threads, -// which acquire it before classify_partition reads the count. +// Post-attach bookkeeping. Runs once on the boot leader after the host-built +// image is attached; latches total_tasks_, sizes the per-S queues to it, and +// folds inline-completed tasks. classify_ready_ is released after this call and +// is what publishes total_tasks_ to the peer threads, which acquire it before +// classify_partition reads the count. // ============================================================================= -void SchedulerContext::on_orchestration_done( - Runtime *runtime, RuntimeContext *rt, [[maybe_unused]] int32_t thread_idx, int32_t total_tasks -) { +void SchedulerContext::on_graph_attached(RuntimeContext *rt, [[maybe_unused]] int32_t thread_idx, int32_t total_tasks) { total_tasks_ = total_tasks; // Allocate the per-S CompletedTaskQueues here on the boot leader, before it @@ -1065,17 +1044,6 @@ void SchedulerContext::on_orchestration_done( #endif } - // Check for fatal error from orchestration; if so, shut down immediately. - int32_t orch_err = 0; - if (sched_->sm_header) { - orch_err = sched_->sm_header->orch_error_code.load(std::memory_order_relaxed); - } - if (orch_err != SIMPLER_ERROR_NONE) { - if (!completed_.exchange(true, std::memory_order_acq_rel)) { - emergency_shutdown(runtime); - } - } - // The polling initial classify (seed the ready queues + wake lists for the // whole graph) runs AFTER this, partitioned across all AICPU threads in // classify_partition() — see AicpuExecutor::run. It is kept out of this diff --git a/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_context.h b/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_context.h index 5283b9d461..d50960e76b 100644 --- a/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_context.h +++ b/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_context.h @@ -155,18 +155,18 @@ class SchedulerContext { // Orchestrator threads (core_trackers_[thread_idx].core_num() == 0) are a no-op. int32_t shutdown(int32_t thread_idx); - // Run all post-orchestration scheduler bookkeeping: + // Run all post-attach scheduler bookkeeping, once, on the boot leader: // - publishes core assignments to the perf collector (SIMPLER_DFX) - // - latches submitted task count from shared memory + // - latches the host-built task count + // - sizes the per-S completed-task queues to it // - folds inline_completed_tasks into completed_tasks_ - // (skipped on fatal error — emergency_shutdown runs instead) - // Callers must invoke rt_orchestration_done(rt) before this — that - // step belongs to the orchestrator lifecycle, not the scheduler. - void on_orchestration_done(Runtime *runtime, RuntimeContext *rt, int32_t thread_idx, int32_t total_tasks); + // The orchestration this graph came from ran to completion on the host, so + // there is no orchestrator lifecycle to hook here: the event is the attach. + void on_graph_attached(RuntimeContext *rt, int32_t thread_idx, int32_t total_tasks); // Seed the ready queues + wake lists for the whole graph at boot. Called by // every AICPU thread on a disjoint slice of the submitted-task range, after - // on_orchestration_done and before runtime_init_ready_ (the caller barriers + // on_graph_attached and before runtime_init_ready_ (the caller barriers // all threads between the two). Concurrency-safe: push_ready_routed and // register_wake are the same lock-free primitives used during the run. void classify_partition(int32_t thread_idx, int32_t nthreads); @@ -181,7 +181,6 @@ class SchedulerContext { int32_t aic_count() const { return aic_count_; } int32_t aiv_count() const { return aiv_count_; } bool is_completed() const { return completed_.load(std::memory_order_acquire); } - int32_t completed_tasks_count() const { return completed_tasks_.load(std::memory_order_acquire); } friend class SchedulerContextTestPeer; @@ -523,8 +522,16 @@ class SchedulerContext { // Cold path: exit checks, stall diagnostics, profiling (scheduler_cold_path.cpp) // ========================================================================= + // The latched-error test both exit checks below share. Deliberately NOT marked + // cold/noinline: it is inlined into each of them, so neither pays a frame for + // the shared half — check_exit_conditions runs on every dispatch pass. + LoopAction check_latched_sched_error(int32_t thread_idx, SharedMemoryHeader *header, Runtime *runtime); + + // Dispatch-loop exit check: latched scheduler error, then the completion + // count against the run's task total. check_idle_fatal_error below is the + // error-only half, for the idle path that has no count to compare. __attribute__((noinline, cold)) LoopAction - handle_orchestrator_exit(int32_t thread_idx, SharedMemoryHeader *header, Runtime *runtime, int32_t &task_count); + check_exit_conditions(int32_t thread_idx, SharedMemoryHeader *header, Runtime *runtime, int32_t &task_count); __attribute__((noinline, cold)) LoopAction check_idle_fatal_error(int32_t thread_idx, SharedMemoryHeader *header, Runtime *runtime); diff --git a/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_dispatch.cpp b/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_dispatch.cpp index a5305285d2..c6c1a04cd7 100644 --- a/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_dispatch.cpp +++ b/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_dispatch.cpp @@ -953,8 +953,7 @@ int32_t SchedulerContext::run_resolution_thread(Runtime *runtime, int32_t thread #endif int32_t published_task_count = 0; - if (handle_orchestrator_exit(thread_idx, header, runtime, published_task_count) == LoopAction::BREAK_LOOP) - break; + if (check_exit_conditions(thread_idx, header, runtime, published_task_count) == LoopAction::BREAK_LOOP) break; int32_t resolved_this_pass = 0; bool resolved_any = false; @@ -1302,7 +1301,7 @@ int32_t SchedulerContext::resolve_and_dispatch(Runtime *runtime, int32_t thread_ #endif int32_t task_count = 0; if (!tracker.has_any_running_cores()) { - LoopAction action = handle_orchestrator_exit(thread_idx, header, runtime, task_count); + LoopAction action = check_exit_conditions(thread_idx, header, runtime, task_count); if (action == LoopAction::BREAK_LOOP) break; } diff --git a/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_types.h b/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_types.h index e3307c0cbd..f36c224926 100644 --- a/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_types.h +++ b/src/a5/runtime/host_build_graph/runtime/scheduler/scheduler_types.h @@ -75,7 +75,7 @@ constexpr int32_t MAX_AICPU_THREADS = PLATFORM_MAX_AICPU_THREADS; // independent of the wall-clock timeout below: small enough to fire a few times // before the budget expires, large enough not to flood device_log. constexpr int32_t STALL_LOG_INTERVAL = 480000; -constexpr int32_t FATAL_ERROR_CHECK_INTERVAL = 1024; // Check orchestrator error every N idle iters +constexpr int32_t FATAL_ERROR_CHECK_INTERVAL = 1024; // Check for a latched scheduler error every N idle iters // Wall-clock budget for declaring "no progress = scheduler timeout". Replaces // the per-thread iteration-count cap that once lived here as MAX_IDLE_ITERATIONS diff --git a/src/common/host_build_graph/host/ready_queue_sizing.cpp b/src/common/host_build_graph/host/ready_queue_sizing.cpp index c71a9840a7..e0948ffba7 100644 --- a/src/common/host_build_graph/host/ready_queue_sizing.cpp +++ b/src/common/host_build_graph/host/ready_queue_sizing.cpp @@ -84,11 +84,8 @@ bool ReadyQueuePopulations::derive_capacities(ReadyQueueCapacities *capacities) return true; } -int32_t derive_ready_queue_capacities( - const ReadyQueuePopulations &populations, SharedMemoryHeader &sm_header, ReadyQueueCapacities *capacities -) { +int32_t derive_ready_queue_capacities(const ReadyQueuePopulations &populations, ReadyQueueCapacities *capacities) { if (populations.derive_capacities(capacities)) return 0; - sm_header.sched_error_code.store(SIMPLER_ERROR_READY_QUEUE_OVERFLOW, std::memory_order_release); - return runtime_status_from_error_codes(SIMPLER_ERROR_NONE, SIMPLER_ERROR_READY_QUEUE_OVERFLOW); + return runtime_status_from_error_code(SIMPLER_ERROR_READY_QUEUE_OVERFLOW); } diff --git a/src/common/host_build_graph/orchestrator.h b/src/common/host_build_graph/orchestrator.h index 65872c8460..31935c13b9 100644 --- a/src/common/host_build_graph/orchestrator.h +++ b/src/common/host_build_graph/orchestrator.h @@ -27,7 +27,9 @@ #pragma once +#include #include +#include #include "common/chip_swimlane_profiling.h" #include "host_build_graph/task_allocator.h" @@ -87,9 +89,21 @@ struct OrchestratorState { int32_t total_aiv_count{0}; // AIV cores (= 2 × clusters on standard hardware) // === FATAL ERROR === - // Fatal error flag (single-thread access by orchestrator, no atomic needed) - // Cross-thread notification uses shared memory orch_error_code (atomic) - bool fatal; + // The whole fatal state, in one field: the first code latched, or + // SIMPLER_ERROR_NONE while the orchestration is healthy. First-writer-wins, so + // it names the failure that started the cascade rather than the last symptom, + // and the bind maps it onto the status the caller sees. A report that supplies + // no code is latched as SIMPLER_ERROR_EXPLICIT_ORCH_FATAL, which is what keeps + // "is fatal" and "which code" from being two separately-settable things. + // + // Atomic because the reporters are not one thread. A recording worker reaches + // report_fatal for everything the recording cannot answer locally with its + // `unsupported` flag: every submit entry validates its arguments ahead of the + // recording branch (submit_task, alloc_tensors), and the public rt_report_fatal + // is callable from a Graph body too. Meanwhile the bind thread reads is_fatal() + // at every entry. The CAS in orch_mark_fatal is what makes first-writer-wins + // hold across those threads rather than only within one. + std::atomic fatal_code{SIMPLER_ERROR_NONE}; // Hidden alloc tasks complete synchronously inside the orchestrator and // therefore bypass the executor's normal worker-completion counter path. @@ -126,6 +140,8 @@ struct OrchestratorState { int64_t bytes_allocated; #endif + bool is_fatal() const { return fatal_code.load(std::memory_order_acquire) != SIMPLER_ERROR_NONE; } + bool in_manual_scope() const { return scope_stack_top >= manual_begin_depth; } // === Cold-path API (defined in orchestrator.cpp) === @@ -141,7 +157,6 @@ struct OrchestratorState { // and must not orchestrate. bool init(void *sm_base, void *gm_heap, uint64_t heap_size, uint64_t max_tasks, SchedulerState *scheduler); - void set_scheduler(SchedulerState *scheduler); void report_fatal(int32_t error_code, const char *func, const char *fmt, ...); void begin_scope(ScopeMode mode = ScopeMode::AUTO); void end_scope(); @@ -160,6 +175,15 @@ struct OrchestratorState { void mark_done(); }; +// task_allocator holds a pointer back to this object's own fatal_code, so relocating +// an OrchestratorState by value would leave the allocator latching into the source. +// The atomic member already makes both operations ill-formed; asserting it says so on +// purpose rather than leaving the self-reference resting on that side effect. +static_assert( + !std::is_move_assignable_v && !std::is_copy_assignable_v, + "OrchestratorState holds a pointer into itself (task_allocator's fatal_code); assigning one would retarget it" +); + // ============================================================================= // Orchestrator Profiling Data // ============================================================================= diff --git a/src/common/host_build_graph/ready_queue_sizing.h b/src/common/host_build_graph/ready_queue_sizing.h index b265494702..04d6253598 100644 --- a/src/common/host_build_graph/ready_queue_sizing.h +++ b/src/common/host_build_graph/ready_queue_sizing.h @@ -27,6 +27,4 @@ struct ReadyQueuePopulations { bool derive_capacities(ReadyQueueCapacities *capacities) const; }; -int32_t derive_ready_queue_capacities( - const ReadyQueuePopulations &populations, SharedMemoryHeader &sm_header, ReadyQueueCapacities *capacities -); +int32_t derive_ready_queue_capacities(const ReadyQueuePopulations &populations, ReadyQueueCapacities *capacities); diff --git a/src/common/host_build_graph/runtime.h b/src/common/host_build_graph/runtime.h index 26e487a8fa..042d1fccd1 100644 --- a/src/common/host_build_graph/runtime.h +++ b/src/common/host_build_graph/runtime.h @@ -169,7 +169,7 @@ class Runtime { uint64_t func_id_to_addr_[RUNTIME_MAX_FUNC_ID]; // Total tasks the host orchestrator submitted, handed to the scheduler by - // SchedulerContext::on_orchestration_done. host_build_graph builds the whole + // SchedulerContext::on_graph_attached. host_build_graph builds the whole // graph on the host, so this scalar is the count's only carrier: the shared // memory header holds no task counter for the boot thread to read. int32_t host_total_tasks; diff --git a/src/common/host_build_graph/runtime_status.h b/src/common/host_build_graph/runtime_status.h index 9fbed0da9a..a450f8bd46 100644 --- a/src/common/host_build_graph/runtime_status.h +++ b/src/common/host_build_graph/runtime_status.h @@ -47,12 +47,13 @@ // push into a ready queue found no free slot (full, or window > capacity) #define SIMPLER_ERROR_READY_QUEUE_OVERFLOW 104 -static inline int32_t runtime_status_from_error_codes(int32_t orch_error_code, int32_t sched_error_code) { - if (orch_error_code != SIMPLER_ERROR_NONE) { - return orch_error_code < 0 ? orch_error_code : -orch_error_code; - } - if (sched_error_code != SIMPLER_ERROR_NONE) { - return sched_error_code < 0 ? sched_error_code : -sched_error_code; +// Maps one latched code onto the status the caller sees. Both families negate the +// same way, and a single call site only ever holds one of them: the orchestrator +// runs to completion on the host before the device starts, so an orchestrator code +// is latched during the bind and a scheduler code only after it. +static inline int32_t runtime_status_from_error_code(int32_t error_code) { + if (error_code != SIMPLER_ERROR_NONE) { + return error_code < 0 ? error_code : -error_code; } return 0; } diff --git a/src/common/host_build_graph/runtime_types.h b/src/common/host_build_graph/runtime_types.h index b01e3e17ba..9e7f22a872 100644 --- a/src/common/host_build_graph/runtime_types.h +++ b/src/common/host_build_graph/runtime_types.h @@ -175,8 +175,10 @@ constexpr uint64_t TENSOR_DATA_TIMEOUT_MS = 15000; // 15 s * PENDING -> COMPLETED * * The slot stays in PENDING from submit through "ready in queue" and "running - * on a worker"; readiness and running-vs-idle are derived from fanin_refcount - * and per-core running_slot_state respectively, not from task_state itself. + * on a worker": readiness comes from the producers' completion state, and + * running-vs-idle from the per-core running_slot_state -- neither from this + * field. Which completion state carries readiness depends on the task's id + * space; see ChipTaskSlotState below. * * Conditions: * PENDING->COMPLETED: all subtasks finish (set by scheduler) or task is a @@ -187,9 +189,8 @@ constexpr uint64_t TENSOR_DATA_TIMEOUT_MS = 15000; // 15 s * completed_watermark instead. */ typedef enum { - CHIP_TASK_PENDING = 0, // Submitted; awaiting fanin, queued, or dispatched - CHIP_TASK_COMPLETED = 1, // Execution finished, output may still be in use - CHIP_TASK_CONSUMED = 2 // Unused: host_build_graph never advances past COMPLETED + CHIP_TASK_PENDING = 0, // Submitted; awaiting fanin, queued, or dispatched + CHIP_TASK_COMPLETED = 1 // Execution finished, output may still be in use } ChipTaskState; /** @@ -344,9 +345,10 @@ struct TaskPayload { // the completed mask stable for its single launch owner, whether staging is local // or uses the global drain fallback. alignas(64) std::atomic staged_core_mask[EARLY_DISPATCH_CORE_MASK_WORDS]{}; - // Early-dispatch CANDIDATE detection (event-driven, dual of fanin_refcount): - // seeded at wiring with producers already complete, then a flagged producer - // bumps each consumer after all of its logical blocks are published. + // Early-dispatch CANDIDATE detection, event-driven and counted rather than + // polled: seeded to 0 at submit with the producers already complete, then a + // flagged producer bumps each consumer after all of its logical blocks are + // published (propagate_dispatch_fanin). // dispatch_fanin == fanin_actual_count <=> every producer is // flagged-and-fully-published or was // pre-completed => this task is an early-dispatch candidate (push early_dispatch_queues[shape]). @@ -547,15 +549,24 @@ static_assert(sizeof(simpler::hbg::Tensor) == 128, "simpler::hbg::Tensor must be * Per-task slot scheduling state (scheduler-private, NOT in shared memory) * * 64 bytes = one cache line. Under the polling completion model a task's - * readiness is derived from its producers' completion_flags (in the SM - * header); producer completion is published by setting this task's own - * completion_flag + draining its wake list. There is no fanout adjacency, - * refcount, or per-task lock here. + * readiness is derived from its producers' completion state; producer completion + * is published by marking this task complete + draining its wake list. There is + * no fanout adjacency, refcount, or per-task lock here. * - * task_state is retained (a COMPLETED store on completion) because the HOST - * still polls it: the completion-wait in runtime_core.cpp, the allocator - * deadlock detector, and the cold-path stall dump. completion_flags is the - * device-side readiness truth; task_state is the host-visible mirror. + * Which field carries that completion state depends on which task table the slot + * belongs to, and both are load-bearing: + * + * - A GLOBAL task holds a slot in the SM task table, so its readiness truth is + * `completion_flags[local_id]` — a byte-per-slot array, which is what lets a + * fanin scan read many producers out of one cache line. `task_state` is then + * a mirror, polled by the host completion-wait in runtime_core.cpp. + * - An IN_GRAPH task lives in its Graph's own storage and has no slot in that + * table, hence no flag byte. `task_state` IS its readiness truth, read on the + * device by graph_first_unmet_producer; only the outer Graph shell (a GLOBAL + * task) gets a flag when the body finishes. + * + * So a completion publishes both for a GLOBAL task and `task_state` alone for an + * IN_GRAPH one. */ struct alignas(64) ChipTaskSlotState { // Highest local task id among this slot's consumers. Reclaim gate: the slot @@ -565,9 +576,11 @@ struct alignas(64) ChipTaskSlotState { // bumped via max() at submit for each consumer. int32_t last_consumer_local_id; - // Host-visible completion mirror. PENDING at submit; COMPLETED at - // on_mixed_task_complete. Read by the host completion-wait / deadlock - // detector / cold-path dump; the device readiness path uses completion_flags. + // Completion state. PENDING at submit; COMPLETED at whichever completion path + // owns this slot. For an IN_GRAPH task this is the readiness truth the device + // itself polls (graph_first_unmet_producer); for a GLOBAL task it mirrors + // completion_flags[slot], which is what the device reads instead. Also read by + // the host completion-wait and the cold-path stall dump. std::atomic task_state; // --- Per-slot constant, re-bound by orch::prepare_task each submit --- @@ -652,10 +665,10 @@ struct alignas(64) ChipTaskSlotState { task.set(t); } - // Host-visible completion mirror. The device readiness truth - // (completion_flags[slot]) is published by the scheduler's - // on_mixed_task_complete; this store makes the same fact visible to the - // host completion-wait / deadlock detector. + // Publishes completion. For an IN_GRAPH task this store is the whole + // publication — that task has no completion_flags byte. For a GLOBAL task it + // accompanies the completion_flags[slot] store that on_mixed_task_complete + // makes, and is the copy the host completion-wait reads. void mark_completed() { task_state.store(CHIP_TASK_COMPLETED, std::memory_order_release); } void mark_any_subtask_deferred() { any_subtask_deferred.store(true, std::memory_order_release); } diff --git a/src/common/host_build_graph/shared/orchestrator.cpp b/src/common/host_build_graph/shared/orchestrator.cpp index 6b12e117af..0bfbbb7327 100644 --- a/src/common/host_build_graph/shared/orchestrator.cpp +++ b/src/common/host_build_graph/shared/orchestrator.cpp @@ -234,41 +234,49 @@ __attribute__((weak, visibility("hidden"))) uint64_t host_phase_now_ns() { retur } while (0) #endif +// A report that names no code still means "fatal", so it is latched -- and logged -- +// under the explicit-fatal code rather than as a zero that reads like "no error". +static constexpr int32_t normalized_fatal_code(int32_t error_code) { + return error_code == SIMPLER_ERROR_NONE ? SIMPLER_ERROR_EXPLICIT_ORCH_FATAL : error_code; +} + +// First-writer-wins, so the latched code names the failure that started the +// cascade. The CAS is load-bearing rather than decorative: a recording worker and +// the bind thread both reach here (see OrchestratorState::fatal_code). +// TaskAllocator::report_capacity_exhausted writes the same field under the same rule. static int32_t orch_mark_fatal(OrchestratorState *orch, int32_t error_code) { always_assert(orch != nullptr); - orch->fatal = true; - if (error_code == SIMPLER_ERROR_NONE || orch->sm_header == nullptr) { - return SIMPLER_ERROR_NONE; - } - + const int32_t code = normalized_fatal_code(error_code); int32_t expected = SIMPLER_ERROR_NONE; - std::atomic &orch_error_code = orch->sm_header->orch_error_code; - if (orch_error_code.compare_exchange_strong(expected, error_code, std::memory_order_acq_rel)) { - return error_code; + if (orch->fatal_code.compare_exchange_strong(expected, code, std::memory_order_acq_rel)) { + return code; } + // A failed exchange loads the winner's code into `expected`. return expected; } static void orch_report_fatal_v(OrchestratorState *orch, int32_t error_code, const char *func, const char *fmt, va_list args) { - int32_t latched_code = orch_mark_fatal(orch, error_code); + const int32_t reported = normalized_fatal_code(error_code); + // Differs from `reported` only when an earlier fatal already owns the field. + const int32_t latched_code = orch_mark_fatal(orch, reported); if (fmt == nullptr || fmt[0] == '\0') { - if (latched_code != SIMPLER_ERROR_NONE && latched_code != error_code) { - unified_log_error(func, "FATAL(code=%d, latched=%d)", error_code, latched_code); + if (latched_code != reported) { + unified_log_error(func, "FATAL(code=%d, latched=%d)", reported, latched_code); } else { - unified_log_error(func, "FATAL(code=%d)", error_code); + unified_log_error(func, "FATAL(code=%d)", reported); } return; } std::array message{}; vsnprintf(message.data(), message.size(), fmt, args); - if (latched_code != SIMPLER_ERROR_NONE && latched_code != error_code) { - unified_log_error(func, "FATAL(code=%d, latched=%d): %s", error_code, latched_code, message.data()); + if (latched_code != reported) { + unified_log_error(func, "FATAL(code=%d, latched=%d): %s", reported, latched_code, message.data()); return; } - unified_log_error(func, "FATAL(code=%d): %s", error_code, message.data()); + unified_log_error(func, "FATAL(code=%d): %s", reported, message.data()); } void OrchestratorState::report_fatal(int32_t error_code, const char *func, const char *fmt, ...) { @@ -1469,7 +1477,7 @@ static bool prepare_task( void OrchestratorState::begin_scope(ScopeMode mode) { auto *orch = this; - if (orch->fatal) { + if (orch->is_fatal()) { return; } // A Graph replays as a flat DAG with no scope structure: scope boundaries only @@ -1514,7 +1522,7 @@ void OrchestratorState::begin_scope(ScopeMode mode) { void OrchestratorState::end_scope() { auto *orch = this; - if (orch->fatal) { + if (orch->is_fatal()) { return; } // Matches begin_scope: a scope inside a Graph body never touches the real @@ -1547,13 +1555,13 @@ void OrchestratorState::end_scope() { // become large enough while the host waits: latch // SIMPLER_ERROR_TENSORMAP_OVERFLOW and bail rather than letting new_entry()'s hard // assert fire mid-registration. Returns false when the pool is exhausted or a -// fatal is already latched by another party. +// fatal is already latched. static bool ensure_tensormap_capacity(OrchestratorState *orch, int32_t needed) { ChipTensorMap &tm = orch->tensor_map; if (tm.free_entries() >= needed) { return true; } - if (orch->sm_header->orch_error_code.load(std::memory_order_acquire) != SIMPLER_ERROR_NONE) { + if (orch->is_fatal()) { return false; } @@ -2559,12 +2567,27 @@ void OrchestratorState::graph_abort(void *recording_handle) { // Finish the background recording and publish the Definition. The main // thread finalizes the already-submitted outer Graph tasks in graph_commit. +// +// Retires the entry it bound on every path below that has one, so a caller never has +// to pair a `false` return with an abort — and must not: graph_commit frees a drained +// entry after releasing recording_mutex, so a second abort would take that mutex and +// still touch freed memory. bool OrchestratorState::graph_end() { GraphHostState *state = graph_state_from(this); GraphRecording *recording = active_graph_recording(this); GraphInflightRecording *entry = g_active_graph_entry; if (state == nullptr || recording == nullptr || entry == nullptr) return false; + // A fatal latched anywhere in the run ends this pass with no Definition, but the + // entry still has to leave RECORDING: graph_commit's drain blocks until every + // in-flight entry has, and nothing else transitions this one. Returning early + // instead would park the entry — and this thread's recorder thread_locals — for + // the rest of the process, and hang the bind that is already failing. + if (is_fatal()) { + graph_abort(entry); + return false; + } + ORCH_PHASE_START(); std::optional layout = graph_layout_definition(*recording); // The claim is what decides where this thread writes, so it precedes the fill @@ -2670,7 +2693,7 @@ TaskOutputTensors OrchestratorState::submit_task(const MixedKernels &mixed_kerne // Orchestration API should short-circuit after fatal, but keep this entry // robust as a no-op in case a caller reaches it directly. - if (orch->fatal) { + if (orch->is_fatal()) { return TaskOutputTensors{}; } @@ -2760,7 +2783,7 @@ TaskOutputTensors OrchestratorState::submit_task(const MixedKernels &mixed_kerne TaskOutputTensors OrchestratorState::submit_dummy_task(const CoreTaskArgs &args) { auto *orch = this; - if (orch->fatal) { + if (orch->is_fatal()) { return TaskOutputTensors{}; } @@ -2797,7 +2820,7 @@ TaskOutputTensors OrchestratorState::alloc_tensors(const CoreTaskArgs &args) { auto *orch = this; // Orchestration API should short-circuit after fatal, but keep this entry // robust as a no-op in case a caller reaches it directly. - if (orch->fatal) { + if (orch->is_fatal()) { return TaskOutputTensors{}; } @@ -2892,7 +2915,7 @@ TaskOutputTensors OrchestratorState::alloc_tensors(const CoreTaskArgs &args) { // codegen task there is no Arg-driven hint to honor here, so mark it // unconditionally. prepared.slot_state->task_attrs.set_early_resolve(true); - prepared.slot_state->mark_completed(); // host-visible task_state mirror + prepared.slot_state->mark_completed(); // GLOBAL task, so task_state is the host-visible mirror // Polling: pre-set the device-visible completion_flags byte in the H2D // image. Consumers poll completion_flags (not task_state), so a hidden-alloc // producer completed here on the host must publish its flag too — otherwise @@ -2929,7 +2952,6 @@ void OrchestratorState::mark_done() { if (total_tasks > 0) { LOG_DEBUG("=== [Orchestrator] total_tasks=%d ===", total_tasks); } - orch->sm_header->orchestrator_done.store(1, std::memory_order_release); orch->scope_stack_top = -1; orch->manual_begin_depth = CHIP_MAX_SCOPE_DEPTH; #if !SIMPLER_ORCH_PROFILING && SIMPLER_DFX diff --git a/src/common/host_build_graph/shared/runtime_core.cpp b/src/common/host_build_graph/shared/runtime_core.cpp index 325d1b3913..4870cf8766 100644 --- a/src/common/host_build_graph/shared/runtime_core.cpp +++ b/src/common/host_build_graph/shared/runtime_core.cpp @@ -154,7 +154,7 @@ void rt_orchestration_done(RuntimeContext *rt) { rt->inline_completed_tasks = rt->orchestrator->inline_completed_tasks; } -static bool is_fatal_impl(RuntimeContext *rt) { return rt->orchestrator->fatal; } +static bool is_fatal_impl(RuntimeContext *rt) { return rt->orchestrator->is_fatal(); } void rt_report_fatal(RuntimeContext *rt, int32_t error_code, const char *func, const char *fmt, ...) { va_list args; @@ -231,9 +231,9 @@ static bool wait_for_tensor_ready( while (slot.task_state.load(std::memory_order_acquire) < CHIP_TASK_COMPLETED) { SPIN_WAIT_HINT(); if ((++spin_count & 1023) == 0) { - // A fatal latched elsewhere (e.g. the scheduler-side wiring - // deadlock detector) breaks this wait; cold path only. - if (orch.sm_header->orch_error_code.load(std::memory_order_acquire) != SIMPLER_ERROR_NONE) { + // A fatal latched earlier in this orchestration breaks the wait; + // cold path only. + if (orch.is_fatal()) { failed = true; return; } @@ -262,9 +262,9 @@ static bool wait_for_tensor_ready( while (cons_tasks.completed_watermark.load(std::memory_order_acquire) < slot.last_consumer_local_id) { SPIN_WAIT_HINT(); if ((++spin_count & 1023) == 0) { - // A fatal latched elsewhere (e.g. the scheduler-side wiring - // deadlock detector) breaks this wait; cold path only. - if (orch.sm_header->orch_error_code.load(std::memory_order_acquire) != SIMPLER_ERROR_NONE) { + // A fatal latched earlier in this orchestration breaks the wait; + // cold path only. + if (orch.is_fatal()) { failed = true; return; } diff --git a/src/common/host_build_graph/shared/runtime_init.cpp b/src/common/host_build_graph/shared/runtime_init.cpp index 6374d71847..3db42a8707 100644 --- a/src/common/host_build_graph/shared/runtime_init.cpp +++ b/src/common/host_build_graph/shared/runtime_init.cpp @@ -19,6 +19,7 @@ * original files and the aicpu build only. */ +#include #include #include @@ -205,18 +206,18 @@ void SchedulerState::destroy() { bool OrchestratorState::init( void *sm_base, void *gm_heap, uint64_t heap_size, uint64_t max_tasks, SchedulerState *scheduler_arg ) { - auto *orch = this; - *orch = OrchestratorState{}; + // Reset in place rather than by move-assignment: fatal_code is a std::atomic, + // which is neither copy- nor move-assignable, and a re-init has to clear every + // field the previous pass left behind (the pool cursors below rely on it). + this->~OrchestratorState(); + auto *orch = new (static_cast(this)) OrchestratorState{}; always_assert(max_tasks > 0); orch->sm_header = reinterpret_cast(sm_base); - orch->fatal = false; orch->scheduler = scheduler_arg; - auto *orch_err = sm_layout::orch_error_code_addr(sm_base); - - orch->task_allocator.init(static_cast(max_tasks), gm_heap, heap_size, orch_err); + orch->task_allocator.init(static_cast(max_tasks), gm_heap, heap_size, &orch->fatal_code); // The mirror's argument pools. Offset arithmetic on the same base as sm_header, // so it holds for whichever SM this orchestrator was pointed at. The cursors @@ -246,8 +247,6 @@ bool OrchestratorState::init( return true; } -void OrchestratorState::set_scheduler(SchedulerState *scheduler) { this->scheduler = scheduler; } - // ============================================================================= // Top-level runtime arena // ============================================================================= diff --git a/src/common/host_build_graph/shared/shared_memory.cpp b/src/common/host_build_graph/shared/shared_memory.cpp index e47a53d447..ba8b7d9dba 100644 --- a/src/common/host_build_graph/shared/shared_memory.cpp +++ b/src/common/host_build_graph/shared/shared_memory.cpp @@ -146,17 +146,7 @@ void SharedMemoryHandle::init_header() { // orchestration, so the real value is not known here yet. header->tasks.total_tasks = 0; - header->orchestrator_done.store(0, std::memory_order_relaxed); - - // Layout info. The descriptors are the first segment, so their offset is where - // the header's own padded size ends — pitch-independent, unlike every segment - // after them. - header->tasks.task_descriptors_offset = CHIP_ALIGN_UP(sizeof(SharedMemoryHeader), CHIP_ALIGN_SIZE); - - header->total_size = sm_size; - // Error reporting - header->orch_error_code.store(SIMPLER_ERROR_NONE, std::memory_order_relaxed); header->sched_error_bitmap.store(0, std::memory_order_relaxed); header->sched_error_code.store(SIMPLER_ERROR_NONE, std::memory_order_relaxed); header->sched_error_thread.store(-1, std::memory_order_relaxed); @@ -167,45 +157,3 @@ void SharedMemoryHandle::init_header() { // count, not the size the table was dimensioned for. The device reads no slot // past total_tasks, so the unclaimed tail is left uninitialized. } - -// ============================================================================= -// Debug Utilities -// ============================================================================= - -void SharedMemoryHandle::print_layout() { - if (!header) return; - - SharedMemoryHeader *h = header; - - LOG_DEBUG("=== Shared Memory Layout ==="); - LOG_DEBUG("Base address: %p", sm_base); - LOG_DEBUG("Total size: %" PRIu64 " bytes", h->total_size); - LOG_DEBUG("Task table:"); - LOG_DEBUG( - " descriptors_off: %" PRIu64 " (0x%" PRIx64 ")", h->tasks.task_descriptors_offset, - h->tasks.task_descriptors_offset - ); - LOG_DEBUG(" completed_wm: %d", h->tasks.completed_watermark.load(std::memory_order_acquire)); - LOG_DEBUG("orchestrator_done: %d", h->orchestrator_done.load(std::memory_order_acquire)); - LOG_DEBUG("Error state:"); - LOG_DEBUG(" orch_error_code: %d", h->orch_error_code.load(std::memory_order_relaxed)); - LOG_DEBUG(" sched_error_bitmap: 0x%x", h->sched_error_bitmap.load(std::memory_order_relaxed)); - LOG_DEBUG(" sched_error_code: %d", h->sched_error_code.load(std::memory_order_relaxed)); - LOG_DEBUG(" sched_error_thread: %d", h->sched_error_thread.load(std::memory_order_relaxed)); - LOG_DEBUG("================================"); -} - -bool SharedMemoryHandle::validate() { - if (!sm_base) return false; - if (!header) return false; - - const SharedMemoryHeader *h = header; - - // Check that offsets are within bounds - if (h->tasks.task_descriptors_offset >= h->total_size) return false; - - // Check pointer alignment - if ((uintptr_t)h->tasks.task_descriptors % CHIP_ALIGN_SIZE != 0) return false; - - return true; -} diff --git a/src/common/host_build_graph/shared_memory.h b/src/common/host_build_graph/shared_memory.h index 138857dfcf..ef66143acb 100644 --- a/src/common/host_build_graph/shared_memory.h +++ b/src/common/host_build_graph/shared_memory.h @@ -15,7 +15,7 @@ * * Memory Layout: * +---------------------------+ - * | SharedMemoryHeader | (completion watermark + sync + error state) + * | SharedMemoryHeader | (completion watermark + scheduler error state) * +---------------------------+ * | TaskDescriptor[] | * | TaskPayload[] | @@ -49,8 +49,8 @@ struct SharedMemoryHandle; /** * The task table's header in shared memory. * - * Groups the completion watermark, layout info, and the pointers to the four - * slot-pitched segments. Pointers are host-side only (set by setup_pointers, + * Groups the completion watermark, the run's task total, and the pointers to the + * four slot-pitched segments. Pointers are host-side only (set by setup_pointers, * invalid on device). * * The run's task total sits here too, as a plain scalar. The graph is complete @@ -68,11 +68,8 @@ struct alignas(64) SharedMemoryTaskHeader { // (concurrent CAS-advance by completing threads). alignas(64) std::atomic completed_watermark; - // Layout metadata (set once at init) - alignas(64) uint64_t task_descriptors_offset; // Offset from SM base, in bytes - // Segment pointers (host-side, set by setup_pointers) - TaskDescriptor *task_descriptors; + alignas(64) TaskDescriptor *task_descriptors; TaskPayload *task_payloads; ChipTaskSlotState *slot_states; @@ -80,7 +77,13 @@ struct alignas(64) SharedMemoryTaskHeader { // 0 = pending, 1 = task fully COMPLETED. Writer = the task's completer at // on_mixed_task_complete; reader = consumer fanin polling (is_completion_flag_set). // Cleared per-slot in orch::prepare_task as each slot is claimed. Indexed by - // local task id, like every other segment. + // local task id, like every other segment — so it covers GLOBAL tasks only. An + // IN_GRAPH task holds no slot here and publishes completion through its own + // ChipTaskSlotState::task_state instead; the Graph's outer shell is the GLOBAL + // task that carries a flag for the whole body. + // + // A hidden-alloc task is the one flag the host presets to 1: it completes during + // orchestration, and a consumer polls this array rather than task_state. std::atomic *completion_flags; // Tasks this run submitted, i.e. the slot count the four segments above are @@ -135,41 +138,36 @@ struct alignas(64) SharedMemoryTaskHeader { static_assert(sizeof(SharedMemoryTaskHeader) == 128, "SharedMemoryTaskHeader layout drift"); static_assert( - offsetof(SharedMemoryTaskHeader, task_descriptors_offset) == 64, - "SharedMemoryTaskHeader task_descriptors_offset layout drift" + offsetof(SharedMemoryTaskHeader, task_descriptors) == 64, "SharedMemoryTaskHeader task_descriptors layout drift" ); +// The device reads this one out of the H2D'd header, so it is pinned separately from the +// segment pointers above, which are host-side only. +static_assert(offsetof(SharedMemoryTaskHeader, total_tasks) == 96, "SharedMemoryTaskHeader total_tasks layout drift"); /** * Shared memory header structure * - * Contains the task table's header plus the run's global sync and error state. + * Contains the task table's header plus the scheduler's error state. */ struct alignas(CHIP_ALIGN_SIZE) SharedMemoryHeader { // === TASK TABLE HEADER (set once at init) === SharedMemoryTaskHeader tasks; - // === GLOBAL FIELDS === - std::atomic orchestrator_done; // Flag: orchestration complete - - // Total shared memory size (for validation) - uint64_t total_size; - // === ERROR REPORTING === - // Orchestrator fatal error code (Orchestrator → Scheduler, AICPU → Host) - // Non-zero signals fatal error. Written by orchestrator, read by scheduler and host. - std::atomic orch_error_code; - - // Scheduler error state (Scheduler → Host, independent of orchestrator) - // Written by scheduler threads on timeout; read by orchestrator and host. + // Scheduler error state. Written by scheduler threads on timeout; read by the + // scheduler's own cold path and, after a failed run, by the host through a D2H + // copy of this header. The orchestrator runs on the host and latches its own + // fatal code in OrchestratorState, so no orchestrator error crosses here. std::atomic sched_error_bitmap; // Bit X set = thread X had error std::atomic sched_error_code; // Last scheduler error code (last-writer-wins) std::atomic sched_error_thread; // Thread index of last error writer }; static_assert(sizeof(SharedMemoryHeader) == 192, "SharedMemoryHeader layout drift"); -static_assert(offsetof(SharedMemoryHeader, total_size) == 136, "SharedMemoryHeader total_size layout drift"); -static_assert(offsetof(SharedMemoryHeader, orch_error_code) == 144, "SharedMemoryHeader orch_error_code layout drift"); +static_assert( + offsetof(SharedMemoryHeader, sched_error_bitmap) == 128, "SharedMemoryHeader sched_error_bitmap layout drift" +); // ============================================================================= // Shared Memory Handle @@ -230,8 +228,6 @@ struct SharedMemoryHandle { attach_populated(void *sm_base, uint64_t sm_size, uint64_t max_tasks, uint64_t live_slots, uint64_t image_bytes); void destroy(); - void print_layout(); - bool validate(); private: void init_header(); @@ -246,10 +242,10 @@ struct SharedMemoryHandle { // ============================================================================= // // When the host pre-builds a runtime-arena image, it needs the device-side -// addresses of several SM sub-fields (the task header, -// task_descriptors arrays, orch_error_code) so it can wire them into the -// orchestrator / scheduler init_data path without dereferencing the SM — -// the SM lives in device memory and cannot be touched from host. +// addresses of several SM sub-fields (the task header, the task_descriptors +// arrays) so it can wire them into the scheduler init_data path without +// dereferencing the SM — the SM lives in device memory and cannot be touched +// from host. // // These helpers compute those addresses by offset arithmetic on the SM // device base. Pure pointer math, no loads/stores; safe to call from host. @@ -257,12 +253,6 @@ struct SharedMemoryHandle { // own setup_pointers), so values are guaranteed consistent across sides. namespace sm_layout { -inline std::atomic *orch_error_code_addr(void *sm_dev_base) noexcept { - return reinterpret_cast *>( - static_cast(sm_dev_base) + offsetof(SharedMemoryHeader, orch_error_code) - ); -} - inline SharedMemoryTaskHeader *task_header_addr(void *sm_dev_base) noexcept { return reinterpret_cast( static_cast(sm_dev_base) + offsetof(SharedMemoryHeader, tasks) diff --git a/src/common/host_build_graph/task_allocator.h b/src/common/host_build_graph/task_allocator.h index 9723caa7a7..693b8b12e0 100644 --- a/src/common/host_build_graph/task_allocator.h +++ b/src/common/host_build_graph/task_allocator.h @@ -21,6 +21,7 @@ #pragma once #include +#include #include #include @@ -57,9 +58,10 @@ class TaskAllocator { /** * Initialize the allocator with its task capacity and heap resources. * - * All pointer arguments are device addresses (live in SM / GM heap); this - * function only stores them, no dereferences, so it is safe to invoke - * from host code that constructs a prebuilt arena image. + * `heap_base` is a device address (the GM heap); this function only stores it, + * no dereferences, so it is safe to invoke from host code that constructs a + * prebuilt arena image. `error_code_ptr` is the host-side orchestrator's own + * fatal_code, dereferenced only from the host as the allocator runs. * * `capacity` is the number of task slots the caller's task table holds — what * the bind resolved from runtime_env.ring_task_window, defaulting to @@ -164,6 +166,8 @@ class TaskAllocator { uint64_t heap_top_ = 0; // Current heap allocation pointer // --- Shared --- + // The orchestrator's own fatal_code. Atomic for the same reason it is there: the + // bind thread and a Graph recording worker both latch into it. std::atomic *error_code_ptr_ = nullptr; // ========================================================================= @@ -230,9 +234,14 @@ class TaskAllocator { ); } LOG_ERROR("========================================"); - if (error_code_ptr_) { - int32_t code = heap_blocked ? SIMPLER_ERROR_HEAP_RING_DEADLOCK : SIMPLER_ERROR_FLOW_CONTROL_DEADLOCK; - error_code_ptr_->store(code, std::memory_order_release); + // First-writer-wins, matching orch_mark_fatal, which latches the same field. + // alloc() already declines once a code is latched, so in practice this is the + // first writer -- but the rule is stated here rather than inherited from that + // guard, so the two writers cannot drift apart. + if (error_code_ptr_ != nullptr) { + const int32_t code = heap_blocked ? SIMPLER_ERROR_HEAP_RING_DEADLOCK : SIMPLER_ERROR_FLOW_CONTROL_DEADLOCK; + int32_t expected = SIMPLER_ERROR_NONE; + error_code_ptr_->compare_exchange_strong(expected, code, std::memory_order_acq_rel); } } }; diff --git a/tests/ut/cpp/common/test_hbg_graph_async_submit.cpp b/tests/ut/cpp/common/test_hbg_graph_async_submit.cpp index 01a156c7dd..8370068336 100644 --- a/tests/ut/cpp/common/test_hbg_graph_async_submit.cpp +++ b/tests/ut/cpp/common/test_hbg_graph_async_submit.cpp @@ -51,6 +51,14 @@ struct FakeRuntime { int begin_calls{0}; int prepare_calls{0}; int end_calls{0}; + // A recorded body reports a fatal on the recorder thread while the bind thread + // reads it, which is what the runtime's own fatal_code is atomic for. + std::atomic fatal{false}; + // graph_end's view of the fatal at the moment it ran, so a test can tell "end + // was reached after the body latched" from "end was reached at all". + bool end_saw_fatal{false}; + std::atomic abort_calls{0}; + const void *abort_handle{nullptr}; // Written from a recorder worker (the recorded body calls rt_graph_commit) and from // the main thread, so it cannot be a plain int. std::atomic commit_calls{0}; @@ -80,7 +88,7 @@ static_assert(offsetof(FakeRuntime, pending_scope_mode) == offsetof(RuntimeConte FakeRuntime *as_fake(RuntimeContext *rt) { return reinterpret_cast(rt); } -bool fake_is_fatal(RuntimeContext *) { return false; } +bool fake_is_fatal(RuntimeContext *rt) { return as_fake(rt)->fatal.load(std::memory_order_acquire); } GraphScopeResult fake_graph_begin(RuntimeContext *rt, uint64_t, const GraphTaskArgs &) { FakeRuntime &fake = *as_fake(rt); @@ -137,14 +145,21 @@ bool fake_graph_prepare(RuntimeContext *rt, void *recording_handle, const GraphT return true; } -void fake_graph_abort(RuntimeContext *, void *) {} +void fake_graph_abort(RuntimeContext *rt, void *recording_handle) { + FakeRuntime &fake = *as_fake(rt); + fake.abort_handle = recording_handle; + fake.abort_calls.fetch_add(1, std::memory_order_acq_rel); +} bool fake_graph_end(RuntimeContext *rt) { FakeRuntime &fake = *as_fake(rt); std::lock_guard lock(fake.mutex); fake.end_calls++; fake.submit_thread = std::this_thread::get_id(); - return true; + // Mirrors the real graph_end's contract: on a fatal it retires the entry itself + // and reports that no Definition was published. + fake.end_saw_fatal = fake.fatal.load(std::memory_order_acquire); + return !fake.end_saw_fatal; } void fake_graph_commit(RuntimeContext *rt) { as_fake(rt)->commit_calls++; } @@ -395,3 +410,54 @@ TEST(HbgGraphAsyncSubmit, RecordingReadsAnOwnedCopyOfTheBoundary) { EXPECT_NE(fake.recorded_tensor_storage, caller_tensor_storage) << "the worker must not read simpler::hbg::Tensor storage the caller owns"; } + +// A fatal reported inside a recorded body has to reach graph_end. graph_end is the +// only thing that transitions the in-flight entry out of RECORDING, and +// graph_commit's drain blocks on recording_cv until every entry has — with no +// timeout, on the bind thread. rt_graph_end used to short-circuit on is_fatal() and +// report success, which skipped the retire and hung that drain. +// +// The same case pins the other half: the wrapper must NOT follow a declined end with +// an abort. graph_end retires the entry it bound on every path that has one, so a +// caller-side abort is always a second one — and graph_commit frees a drained entry +// after releasing recording_mutex, so the second abort has nothing left to +// synchronize against and writes to freed memory. +// +// Neither half is reachable from a test that calls OrchestratorState::graph_end() +// directly: both live in the wrapper, which is why this case is here and not with the +// orchestrator's own graph tests. +TEST(HbgGraphAsyncSubmit, AFatalInsideARecordedBodyReachesGraphEndAndAbortsNothing) { + FakeRuntime fake{}; + fake.ops = &kFakeOps; + framework_bind_runtime(reinterpret_cast(&fake)); + + uint32_t storage[4]{}; + uint32_t shape[] = {4}; + simpler::hbg::Tensor boundary = simpler::hbg::make_tensor_external(storage, shape, 1); + GraphTaskArgs args; + args.add_input(boundary); + + // No later submission to overlap with, so release the handshake gate rather than + // letting fake_graph_prepare sit out a timeout nothing will satisfy. + fake.later_submit_entered = true; + + int body_calls = 0; + { + ScopeGuard scope; + (void)rt_submit_graph_impl(0x1722, args, [&](const GraphTaskArgs &) { + body_calls++; + // Stands in for the body's own rt_report_fatal: from the wrapper's side a + // fatal is just is_fatal() turning true partway through the pass. + fake.fatal.store(true, std::memory_order_release); + }); + } + // Drains the recorder pool through the ops table, as orchestration completion does. + rt_graph_commit(); + framework_bind_runtime(nullptr); + + EXPECT_EQ(body_calls, 1); + EXPECT_EQ(fake.end_calls, 1) << "a fatal must not stop the recording pass from reaching graph_end"; + EXPECT_TRUE(fake.end_saw_fatal) << "graph_end has to observe the fatal — it is the retire point"; + EXPECT_EQ(fake.abort_calls.load(std::memory_order_acquire), 0) + << "graph_end retires its own entry, so a caller-side abort would be a second one racing graph_commit's free"; +} diff --git a/tests/ut/cpp/common/test_hbg_graph_definition_arena.cpp b/tests/ut/cpp/common/test_hbg_graph_definition_arena.cpp index abc694236a..4fa2ef9ea8 100644 --- a/tests/ut/cpp/common/test_hbg_graph_definition_arena.cpp +++ b/tests/ut/cpp/common/test_hbg_graph_definition_arena.cpp @@ -128,7 +128,7 @@ TEST_F(HbgGraphDefinitionArenaTest, ObjectsAreBuiltInTheArenaAtAlignedDisjointOf record_graph(0x1715, 1, boundary, shape); record_graph(0x1716, 3, boundary, shape); orch.graph_commit(); - ASSERT_FALSE(orch.fatal); + ASSERT_FALSE(orch.is_fatal()); const GraphHostDefinitionList definitions = graph_host_definitions(*graph_state); ASSERT_EQ(definitions.entries.size(), 2u); @@ -190,7 +190,7 @@ TEST_F(HbgGraphDefinitionArenaTest, AnArenaWithNoRoomSpillsAndStillPublishesTheI orch.begin_scope(); record_graph(0x1715, 2, boundary, shape); orch.graph_commit(); - ASSERT_FALSE(orch.fatal) << "outgrowing the arena must cost a copy, not the run"; + ASSERT_FALSE(orch.is_fatal()) << "outgrowing the arena must cost a copy, not the run"; const GraphHostDefinitionList definitions = graph_host_definitions(*graph_state); ASSERT_EQ(definitions.entries.size(), 1u); @@ -221,7 +221,7 @@ TEST_F(HbgGraphDefinitionArenaTest, AnArenaTooSmallForAnObjectSpillsIt) { orch.begin_scope(); record_graph(0x1715, 1, boundary, shape); orch.graph_commit(); - ASSERT_FALSE(orch.fatal); + ASSERT_FALSE(orch.is_fatal()); const GraphHostDefinitionList definitions = graph_host_definitions(*graph_state); ASSERT_EQ(definitions.entries.size(), 1u); diff --git a/tests/ut/cpp/common/test_hbg_graph_submit_failure.cpp b/tests/ut/cpp/common/test_hbg_graph_submit_failure.cpp index ecef313fe5..707194de82 100644 --- a/tests/ut/cpp/common/test_hbg_graph_submit_failure.cpp +++ b/tests/ut/cpp/common/test_hbg_graph_submit_failure.cpp @@ -122,7 +122,7 @@ TEST_F(HbgGraphSubmitFailureTest, InFlightGraphInvocationsReserveHeapOnlyAtCommi EXPECT_EQ(orch.task_allocator.heap_top(), 0u); orch.graph_commit(); - EXPECT_FALSE(orch.fatal); + EXPECT_FALSE(orch.is_fatal()); EXPECT_GT(orch.task_allocator.heap_top(), 0u); const std::optional first_upload = graph_host_upload(*graph_state, 0); const std::optional second_upload = graph_host_upload(*graph_state, 1); @@ -244,7 +244,7 @@ TEST_F(HbgGraphSubmitFailureTest, WorkerRecordsWhileMainThreadSubmitsSameHashShe EXPECT_EQ(orch.task_allocator.heap_top(), 0u) << "no shell may take heap before commit"; orch.graph_commit(); - ASSERT_FALSE(orch.fatal); + ASSERT_FALSE(orch.is_fatal()); ASSERT_EQ(graph_host_upload_count(*graph_state), 3u); const GraphHostDefinitionList definitions = graph_host_definitions(*graph_state); @@ -297,10 +297,87 @@ TEST_F(HbgGraphSubmitFailureTest, AbortedRecordingLatchesFatalAtCommit) { ASSERT_TRUE(orch.submit_dummy_task(task_args).task_id().is_valid()); orch.graph_abort(graph.recording_handle); - ASSERT_FALSE(orch.fatal) << "Abort alone must not latch; the shell is still finalizable in principle"; + ASSERT_FALSE(orch.is_fatal()) << "Abort alone must not latch; the shell is still finalizable in principle"; orch.graph_commit(); - EXPECT_TRUE(orch.fatal) << "A shell whose Definition never arrived cannot be completed"; + EXPECT_TRUE(orch.is_fatal()) << "A shell whose Definition never arrived cannot be completed"; +} + +// A recording worker reaches report_fatal for anything the recording cannot answer +// locally: every submit entry validates its arguments ahead of its recording branch, +// and the public rt_report_fatal is callable from a body. Two things have to hold +// afterwards, and neither is about the code that was latched. +// +// The entry has to leave RECORDING. graph_commit's drain blocks until every in-flight +// entry has, and on this path graph_end is the only thing the worker calls that can +// perform the transition — so a graph_end that returns early on the fatal turns a +// reported error into a hang on the bind thread. +// +// And the worker's recorder thread_locals have to be released. The recorder pool +// outlives the run, so a thread that keeps them bound fails graph_prepare's +// already-recording guard for every later recording it is handed. +TEST_F(HbgGraphSubmitFailureTest, AFatalDuringRecordingRetiresTheEntryAndFreesTheRecorderThread) { + std::array storage{}; + uint32_t shape[] = {static_cast(storage.size())}; + simpler::hbg::Tensor boundary = simpler::hbg::make_tensor_external(storage.data(), shape, 1); + GraphTaskArgs boundary_args; + boundary_args.add_input(boundary); + + orch.begin_scope(); + // Two keys, so the worker has a second recording to prove its thread_locals came + // back. Both open before any fatal is latched. + const GraphScopeResult first = orch.graph_begin(0x1720, boundary_args, 0x1736); + const GraphScopeResult second = orch.graph_begin(0x1721, boundary_args, 0x1736); + ASSERT_TRUE(first.recording); + ASSERT_TRUE(second.recording); + + bool first_prepare_ok = false; + bool first_end_ok = true; + bool second_prepare_ok = false; + bool second_end_ok = true; + bool retired_entry_refuses_prepare = false; + + std::thread worker([&]() { + // Worker-owned boundary copies, alive until each recording ends. + GraphTaskArgs first_args; + first_args.add_input(boundary); + first_prepare_ok = orch.graph_prepare(first.recording_handle, first_args); + if (!first_prepare_ok) return; + + // The body reports a fatal. This is the cross-thread write fatal_code is + // atomic for: the bind thread reads it through is_fatal() at every entry. + orch.report_fatal(SIMPLER_ERROR_EXPLICIT_ORCH_FATAL, "recorded_body", "%s", "the body reported a fatal"); + first_end_ok = orch.graph_end(); + + GraphTaskArgs second_args; + second_args.add_input(boundary); + second_prepare_ok = orch.graph_prepare(second.recording_handle, second_args); + if (!second_prepare_ok) return; + second_end_ok = orch.graph_end(); + + // With this thread's thread_locals proven clear by the prepare above, the only + // reason left to refuse the first handle is that its entry is no longer + // RECORDING -- which is the transition graph_commit's drain waits for. + GraphTaskArgs retry_args; + retry_args.add_input(boundary); + retired_entry_refuses_prepare = !orch.graph_prepare(first.recording_handle, retry_args); + }); + worker.join(); + + ASSERT_TRUE(first_prepare_ok); + EXPECT_FALSE(first_end_ok) << "a fatal publishes no Definition, so end must decline"; + EXPECT_TRUE(second_prepare_ok) << "graph_end must release the recorder thread_locals it bound"; + EXPECT_FALSE(second_end_ok); + EXPECT_TRUE(retired_entry_refuses_prepare) << "a fatal must leave the entry out of RECORDING"; + + // Returns rather than blocking: the drain's predicate is already satisfied, because + // both entries left RECORDING above. + orch.graph_commit(); + + EXPECT_TRUE(orch.is_fatal()); + EXPECT_EQ(orch.fatal_code.load(std::memory_order_acquire), SIMPLER_ERROR_EXPLICIT_ORCH_FATAL) + << "first-writer-wins: commit's own SIMPLER_ERROR_INVALID_ARGS must not displace the body's code"; + EXPECT_EQ(graph_host_definitions(*graph_state).entries.size(), 0u) << "no Definition may be published"; } // The ordinary path reports SIMPLER_ERROR_INVALID_ARGS for an auto scope opened @@ -335,7 +412,7 @@ TEST_F(HbgGraphSubmitFailureTest, AutoScopeNestedInManualScopeRefusesTheRecordin EXPECT_THROW(orch.graph_end(), AssertionError) << "an auto scope inside a manual one must not publish"; orch.graph_abort(graph.recording_handle); orch.graph_commit(); - EXPECT_TRUE(orch.fatal) << "a shell whose Definition never arrived cannot be completed"; + EXPECT_TRUE(orch.is_fatal()) << "a shell whose Definition never arrived cannot be completed"; } // A Graph body may allocate. The allocation records as a kernel-less in-graph task, @@ -362,7 +439,7 @@ TEST_F(HbgGraphSubmitFailureTest, RuntimeAllocationInsideTheBodyRecordsAKernelle EXPECT_TRUE(orch.graph_end()); orch.graph_commit(); - EXPECT_FALSE(orch.fatal); + EXPECT_FALSE(orch.is_fatal()); } TEST_F(HbgGraphSubmitFailureTest, FaninFailureLatchesFatalWithoutPartialUpload) { @@ -388,7 +465,7 @@ TEST_F(HbgGraphSubmitFailureTest, FaninFailureLatchesFatalWithoutPartialUpload) EXPECT_EQ(orch.task_allocator.heap_top(), heap_top_before_record); orch.graph_commit(); EXPECT_GT(orch.task_allocator.heap_top(), heap_top_before_record); - ASSERT_FALSE(orch.fatal); + ASSERT_FALSE(orch.is_fatal()); const size_t uploads_before_failure = graph_host_upload_count(*graph_state); CoreTaskArgs producer_args; @@ -402,10 +479,8 @@ TEST_F(HbgGraphSubmitFailureTest, FaninFailureLatchesFatalWithoutPartialUpload) EXPECT_TRUE(replay.execute_block); EXPECT_FALSE(replay.recording); EXPECT_FALSE(replay.task_id.is_valid()); - EXPECT_TRUE(orch.fatal); - EXPECT_EQ( - sm_handle->header->orch_error_code.load(std::memory_order_acquire), SIMPLER_ERROR_FANIN_CAPACITY_EXCEEDED - ); + EXPECT_TRUE(orch.is_fatal()); + EXPECT_EQ(orch.fatal_code.load(std::memory_order_acquire), SIMPLER_ERROR_FANIN_CAPACITY_EXCEEDED); EXPECT_EQ(graph_host_upload_count(*graph_state), uploads_before_failure); } @@ -439,7 +514,7 @@ TEST_F(HbgGraphSubmitFailureTest, CachedGraphUsesFinalTaskWindowSlot) { EXPECT_EQ(simpler::hbg::task_local_id(replay.task_id), static_cast(allocator.capacity() - 1)); EXPECT_EQ(allocator.active_count(), allocator.capacity()); EXPECT_EQ(allocator.active_count(), allocator.capacity()); - EXPECT_EQ(sm_handle->header->orch_error_code.load(std::memory_order_acquire), SIMPLER_ERROR_NONE); + EXPECT_EQ(orch.fatal_code.load(std::memory_order_acquire), SIMPLER_ERROR_NONE); } // The constructs a predicate can present that no Definition can express. Each is @@ -482,7 +557,7 @@ class HbgGraphPredicateRejectionTest : public HbgGraphSubmitFailureTest { EXPECT_THROW(orch.graph_end(), AssertionError) << "an unrecordable predicate must not publish"; orch.graph_abort(graph.recording_handle); orch.graph_commit(); - EXPECT_TRUE(orch.fatal) << "a shell whose Definition never arrived cannot be completed"; + EXPECT_TRUE(orch.is_fatal()) << "a shell whose Definition never arrived cannot be completed"; } static CoreTaskPredicate predicate_on(const simpler::hbg::Tensor &operand, uint32_t index) { @@ -544,7 +619,7 @@ TEST_F(HbgGraphPredicateRejectionTest, PredicateOnAKernellessInGraphTaskIsNotRec EXPECT_TRUE(orch.graph_end()) << "a dropped predicate must not make the body unrecordable"; orch.graph_commit(); - EXPECT_FALSE(orch.fatal); + EXPECT_FALSE(orch.is_fatal()); } // Distinct Graph keys record concurrently. A Definition the run has not seen @@ -594,7 +669,7 @@ TEST_F(HbgGraphSubmitFailureTest, ASecondKeyRecordsAlongsideTheFirst) { // One commit drains and back-patches both keys' deferred shells. orch.graph_commit(); - EXPECT_FALSE(orch.fatal); + EXPECT_FALSE(orch.is_fatal()); const GraphScopeResult replay_a = orch.graph_begin(0x1901, args_a, 0x1736); EXPECT_FALSE(replay_a.execute_block) << "the first key's Definition must be cached"; @@ -636,7 +711,7 @@ TEST_F(HbgGraphSubmitFailureTest, ConcurrentDefinitionsFinalizeInSubmissionOrder } orch.graph_commit(); - ASSERT_FALSE(orch.fatal); + ASSERT_FALSE(orch.is_fatal()); ASSERT_EQ(graph_host_upload_count(*graph_state), kGraphCount); const char *previous_end = nullptr; @@ -680,7 +755,7 @@ TEST_F(HbgGraphSubmitFailureTest, ACachedGraphReplaysWhileAnotherKeyRecords) { ASSERT_TRUE(orch.submit_dummy_task(task_a).task_id().is_valid()); ASSERT_TRUE(orch.graph_end()); orch.graph_commit(); - ASSERT_FALSE(orch.fatal); + ASSERT_FALSE(orch.is_fatal()); // Key B is now recording and stays that way for the rest of the test. const GraphScopeResult second = orch.graph_begin(0x1904, args_b, 0x1736); @@ -701,7 +776,7 @@ TEST_F(HbgGraphSubmitFailureTest, ACachedGraphReplaysWhileAnotherKeyRecords) { ASSERT_TRUE(orch.submit_dummy_task(task_b).task_id().is_valid()); ASSERT_TRUE(orch.graph_end()); orch.graph_commit(); - EXPECT_FALSE(orch.fatal); + EXPECT_FALSE(orch.is_fatal()); } // An ordinary task submitted while a recording is in flight takes its heap @@ -740,7 +815,7 @@ TEST_F(HbgGraphSubmitFailureTest, AnOrdinaryAllocationInterleavesWithADeferredSh ASSERT_TRUE(orch.graph_end()); orch.graph_commit(); - EXPECT_FALSE(orch.fatal); + EXPECT_FALSE(orch.is_fatal()); EXPECT_GT(orch.task_allocator.heap_top(), heap_after_ordinary) << "the shell's block sits above the ordinary task's, not before it"; SharedMemoryTaskHeader &tasks = sm_handle->header->tasks; @@ -800,7 +875,7 @@ TEST_F(HbgGraphSubmitFailureTest, RecordsAGraphWhoseBoundaryLivesInTheHeapWindow EXPECT_TRUE(orch.submit_dummy_task(task_args).task_id().is_valid()); EXPECT_TRUE(orch.graph_end()); orch.graph_commit(); - EXPECT_FALSE(orch.fatal); + EXPECT_FALSE(orch.is_fatal()); // Each call uses its own graph_key, so it publishes exactly one Definition // and appends exactly one upload. That upload names this call's full_key @@ -902,7 +977,7 @@ TEST_F(HbgGraphSubmitFailureTest, AHiddenAllocTaskLeavesItsDispatchPredicateDefi args.add_output(output); const TaskOutputTensors outputs = orch.alloc_tensors(args); ASSERT_TRUE(outputs.task_id().is_valid()); - ASSERT_FALSE(orch.fatal); + ASSERT_FALSE(orch.is_fatal()); const uint64_t slot = simpler::hbg::task_local_id(outputs.task_id()); ASSERT_LT(slot, static_cast(kPoisonedSlots)) << "the submitted slot must be one this test poisoned"; diff --git a/tests/ut/cpp/common/test_hbg_ready_queue_seed.cpp b/tests/ut/cpp/common/test_hbg_ready_queue_seed.cpp index 095f8b0efe..349193915c 100644 --- a/tests/ut/cpp/common/test_hbg_ready_queue_seed.cpp +++ b/tests/ut/cpp/common/test_hbg_ready_queue_seed.cpp @@ -187,16 +187,14 @@ TEST(HbgReadyQueueSizing, RejectsMergedPopulationPastReservationLimit) { EXPECT_FALSE(first.derive_capacities(&capacities)); } -TEST(HbgReadyQueueSizing, BindRejectionReturnsAndStoresReadyQueueOverflow) { +TEST(HbgReadyQueueSizing, BindRejectionReturnsReadyQueueOverflow) { ReadyQueuePopulations populations{}; populations.add_task(ActiveMask(SUBTASK_MASK_AIC), TaskAttrs{}, TaskKind::KERNEL, READY_QUEUE_CAPACITY_LIMIT + 1); - SharedMemoryHeader header{}; ReadyQueueCapacities capacities{}; - const int32_t status = derive_ready_queue_capacities(populations, header, &capacities); + const int32_t status = derive_ready_queue_capacities(populations, &capacities); EXPECT_EQ(status, -SIMPLER_ERROR_READY_QUEUE_OVERFLOW); - EXPECT_EQ(header.sched_error_code.load(std::memory_order_acquire), SIMPLER_ERROR_READY_QUEUE_OVERFLOW); } TEST(HbgReadyQueueSizing, InitializesEveryLogicalQueueCapacityFromLayout) { diff --git a/tests/ut/cpp/common/test_hbg_sm_compaction.cpp b/tests/ut/cpp/common/test_hbg_sm_compaction.cpp index 47cf8a4a65..37ebf238be 100644 --- a/tests/ut/cpp/common/test_hbg_sm_compaction.cpp +++ b/tests/ut/cpp/common/test_hbg_sm_compaction.cpp @@ -74,7 +74,6 @@ class Mirror { const auto off = sm_layout::segment_offsets(WINDOW); auto *header = reinterpret_cast(image_.base()); auto &tasks = header->tasks; - tasks.task_descriptors_offset = off.descriptors; tasks.completed_watermark.store(-1, std::memory_order_relaxed); tasks.total_tasks = static_cast(SUBMITTED); tasks.task_descriptors = descriptors(); @@ -235,7 +234,6 @@ TEST(HbgSmCompaction, CarriesEveryLiveSlotsContent) { // The header's pitch-independent fields come across; the mirror slot past the // prefix does not. auto &tasks = reinterpret_cast(compacted.image.base())->tasks; - EXPECT_EQ(tasks.task_descriptors_offset, sm_layout::segment_offsets(WINDOW).descriptors); EXPECT_EQ(tasks.completed_watermark.load(std::memory_order_relaxed), -1); // The device bounds its completed_watermark walk with this, and the restack is // the only thing that carries it there. @@ -301,7 +299,7 @@ TEST(HbgSmCompaction, ZeroSubmittedShipsTheHeaderAlone) { EXPECT_EQ(compacted.bytes, compacted.off(0).end); EXPECT_LT(compacted.bytes, sm_layout::segment_offsets(1).end); auto &tasks = reinterpret_cast(compacted.image.base())->tasks; - EXPECT_EQ(tasks.task_descriptors_offset, sm_layout::segment_offsets(WINDOW).descriptors); + EXPECT_EQ(tasks.total_tasks, static_cast(SUBMITTED)); EXPECT_EQ(tasks.task_descriptors, nullptr); }