diff --git a/docs/dfx/dep-gen.md b/docs/dfx/dep-gen.md index 05983f5cbc..1c8be95b14 100644 --- a/docs/dfx/dep-gen.md +++ b/docs/dfx/dep-gen.md @@ -75,6 +75,23 @@ inputs to each submit are captured and the graph is reconstructed afterwards. immediately and know to update the annotated mirror. - **Output.** `/deps.json` — strided-Tensor schema with `tasks[]`, `tensors[]`, and tensor-annotated `edges[]` (see §4). +- **Edges are as-constructed, not as-reduced.** Both passes replay the same + fanin construction, so `deps.json` records the pre-reduction edge set. + The `tensormap_and_ringbuffer` runtime's bounded bitmap transitive + reduction (`reduce_wait_edges`, applied when the builder flushes in + `submit_task_common`) runs *after* construction and clears the `wait` + flag on a direct edge already covered by a WAIT path through the + producer's own transitive ancestors. A diamond `A→B→C` + `A→C` still + shows `A→C` with its constructed flags. The differential gate is + unaffected — both passes replay the same construction. What the + reduction changes is the edge's flags: a redundant `WAIT|RETAIN` edge + demotes to RETAIN-only and a redundant `WAIT`-only edge drops to + `DEP_NONE`, with the entry kept either way, so retention and + pin-release accounting are preserved and only readiness enforcement + is relaxed. + [`wait_reduction_sim`](../../simpler_setup/tools/README.md#wait_reduction_sim) + replays the reduction offline over a capture to measure its coverage + against the full-DAG upper bound. ### 2.2 Host orchestration (`host_build_graph`) diff --git a/simpler_setup/tools/README.md b/simpler_setup/tools/README.md index 9cf4258e29..df443df854 100644 --- a/simpler_setup/tools/README.md +++ b/simpler_setup/tools/README.md @@ -17,6 +17,7 @@ no repo checkout required. - **[hbg_bind_phases](#hbg_bind_phases)** — `host_build_graph` `bind`-stage phases from `bind phase=` log markers → per-phase min/median/max plus the control-plane total - **[dump_viewer](#dump_viewer)** — inspect / export args dumps (see [docs/args-dump.md](../../docs/dfx/args-dump.md) for full workflow) - **[deps_viewer](#deps_viewer)** — `deps.json` (dep_gen) → text or pan/zoom HTML dependency graph +- **[wait_reduction_sim](#wait_reduction_sim)** — `deps.json` (dep_gen) → bounded-bitmap WAIT reduction coverage vs the full-DAG upper bound, per BL For CLIs that allow an omitted input, auto-detection paths (`outputs/*/chip_swimlane_records.json`, `outputs/*/args_dump/`) are resolved @@ -591,6 +592,57 @@ python -m simpler_setup.tools.dump_viewer outputs/_/args_dump/ --index --- +## wait_reduction_sim + +Measure how many redundant WAIT edges the `tensormap_and_ringbuffer` +runtime's bounded reachability bitmap reduction would remove from a real +dependency graph, against the exact full-DAG transitive reduction as the +upper bound (issue #1376 acceptance #9). Decides the production bitmap +window (BL) from data instead of guesswork. + +### Overview + +`wait_reduction_sim` reads the same `deps.json` the +[`deps_viewer`](#deps_viewer) consumes (edges are as-constructed, i.e. +pre-reduction, so one capture serves baseline and comparison alike). It +reconstructs the global submission order from the `tasks[]` record order, +OR-accumulates edge flags per `(pred, succ)` pair, and runs two models over +the WAIT subgraph: + +- **Full reduction** — exact transitive reachability over the whole DAG: + the upper bound any reducer could reach. +- **Online bitmap** — a faithful mirror of the runtime's + `reduce_wait_edges` (frozen per-task `R[t]`, two-pass `direct`/`via` + fold, `d > BL` window misses kept) at each requested window size. + +The report includes per-BL removal counts, `WAIT|RETAIN → RETAIN` demotions +vs pure WAIT drops, window and cross-ring misses, the producer→consumer +submission-distance CDF, and the estimated reduction in readiness fanout +nodes / dependency-pool entries. `DepGenRecord` does not preserve explicit +dependency kinds yet (#1827), so removal counts are accurate while the report +marks affected demote-vs-drop classifications as uncertain. + +### Usage + +```bash +# Capture once (dep_gen records pre-construction edges; see docs/dfx/dep-gen.md) +pytest examples/a5/tensormap_and_ringbuffer/qwen3_14b_decode --platform a5 --enable-dep-gen + +# Compare BL=64/128/256 (default) against the upper bound +python -m simpler_setup.tools.wait_reduction_sim outputs/_/deps.json + +# Machine-readable output, e.g. to diff two captures +python -m simpler_setup.tools.wait_reduction_sim deps_a.json deps_b.json --json report.json +``` + +Reading the output: when `BL=64 removed ≈ upper_bound`, the single-word +window already saturates the graph's redundancy and larger windows buy +nothing; when the `pct_pairs_within_window` column is well below 100 for a +BL, the graph has far-apart producer/consumer pairs that only a wider +window could cover. + +--- + ## Shared Configuration ### Input File Format diff --git a/simpler_setup/tools/__init__.py b/simpler_setup/tools/__init__.py index f68fe4505b..75c728bf4e 100644 --- a/simpler_setup/tools/__init__.py +++ b/simpler_setup/tools/__init__.py @@ -14,6 +14,7 @@ - ``sched_overhead_analysis``: scheduler overhead deep-dive - ``critical_path`` : chip swimlane critical-path compute/stall analysis - ``deps_viewer`` : deps.json -> text or pan/zoom HTML dependency graph +- ``wait_reduction_sim`` : deps.json -> bounded-bitmap WAIT reduction coverage vs full-DAG bound - ``dump_viewer`` : inspect args dumps - ``strace_timing`` : per-stage / per-round timing from [STRACE] log markers - ``hbg_bind_phases`` : per-phase host_build_graph bind statistics from `bind phase=` markers diff --git a/simpler_setup/tools/wait_reduction_sim.py b/simpler_setup/tools/wait_reduction_sim.py new file mode 100644 index 0000000000..879849658d --- /dev/null +++ b/simpler_setup/tools/wait_reduction_sim.py @@ -0,0 +1,318 @@ +#!/usr/bin/env python3 +# Copyright (c) PyPTO Contributors. +# This program is free software, you can redistribute it and/or modify it under the terms and conditions of +# CANN Open Software License Agreement Version 2.0 (the "License"). +# Please refer to the License for details. You may not use this file except in compliance with the License. +# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +# INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +# See LICENSE in the root of the software repository for the full text of the License. +# ----------------------------------------------------------------------------------------------------------- +"""Offline WAIT-edge reduction coverage simulator (issue #1376). + +Reads one or more ``deps.json`` captures (see ``docs/dfx/dep-gen.md``), +reconstructs the global submission order from the ``tasks[]`` record order, +and measures how many redundant WAIT edges the runtime's bounded reachability +bitmap would remove — against the exact full-graph transitive reduction as +the upper bound. + +Two models run over the same WAIT graph: + +- **Full reduction** (upper bound): exact transitive reachability on the whole + DAG. An edge ``p -> t`` is redundant when ``p`` reaches ``t`` through any + other WAIT path of length >= 2. This is what a host-resident full-DAG + reducer (e.g. the hbg Definition pass) could remove. +- **Online bitmap** (the runtime algorithm): a faithful mirror of + ``reduce_wait_edges`` — per-task frozen ``R[t]`` bitmap over the last BL + submissions, two-pass ``direct``/``via`` fold, ``d > BL`` window misses kept + conservatively, ``d == BL`` direct bit without the shift. + +Run for BL=64/128/256 and compare removal counts and the seq-distance CDF to +decide the production window size (issue acceptance #9). + +Usage:: + + python -m simpler_setup.tools.wait_reduction_sim DEPS_JSON... [--bl 64,128,256] [--json OUT.json] + +Modeling notes: + +- Edges are OR-accumulated per ``(pred, succ)`` pair across their records + (same convention ``deps.json`` documents for consumers). Only pairs whose + accumulated flags contain ``wait`` participate in reachability; the + ``wait``-only vs ``wait|retain`` split decides whether a removal is a pure + drop or a RETAIN-only demotion. +- ``deps.json`` predates any runtime reduction: both replay passes record the + as-constructed edge set, so the same capture serves as input for baseline + and comparison alike. +- ``alloc_tensors`` tasks bypass the dep_gen capture point and appear only as + edge ``pred`` values. They are inserted into the submission order just + before their first consumer reference, which is the earliest position the + runtime could have submitted them; their own fanin is empty by construction. +- Captures made before the ``flags`` field existed carry only ``source``; the + source-to-flags mapping (creator -> wait|retain, tensormap -> wait, + explicit -> wait|retain) reconstructs the replay's conservative flags. +- ``DepGenRecord`` does not preserve the kind of an explicit dependency + (issue #1827), so an explicit runtime WAIT-only edge appears as + WAIT|RETAIN in ``deps.json``. Removal counts remain valid, but the report + marks demote-vs-drop classifications involving such pairs as uncertain. +""" + +from __future__ import annotations + +import argparse +import json +import sys +from collections import deque +from pathlib import Path + +SOURCE_FLAGS = { + "creator": ("wait", "retain"), + "tensormap": ("wait",), + "explicit": ("wait", "retain"), +} + + +def _edge_flags(edge: dict) -> frozenset[str]: + """Per-record edge flags, falling back to the source-derived mapping.""" + flags = edge.get("flags") + if flags is not None: + return frozenset(flags) + return frozenset(SOURCE_FLAGS.get(edge.get("source", ""), ("wait", "retain"))) + + +def load_wait_graph( + path: Path, +) -> tuple[ + list[str], + dict[str, int], + dict[tuple[str, str], frozenset[str]], + frozenset[tuple[str, str]], +]: + """Return graph data and pairs whose RETAIN classification is uncertain. + + The submission order is the ``tasks[]`` record order; alloc tasks that + never appear in ``tasks[]`` are inserted immediately before their first + consumer reference. + """ + data = json.loads(path.read_text()) + order: list[str] = [t["task_id"] for t in data.get("tasks", [])] + all_flags: dict[tuple[str, str], set[str]] = {} + explicit_pairs: set[tuple[str, str]] = set() + certain_retain_pairs: set[tuple[str, str]] = set() + for edge in data.get("edges", []): + pair = (edge["pred"], edge["succ"]) + flags = _edge_flags(edge) + # Every record of a pair contributes, RETAIN-only ones included: the + # pair is wait overall if any record waits, and retain overall if any + # record retains (OR-accumulate). + all_flags.setdefault(pair, set()).update(flags) + if edge.get("source") == "explicit": + explicit_pairs.add(pair) + elif "retain" in flags: + certain_retain_pairs.add(pair) + # The WAIT graph is the subset that gates readiness somewhere; a pair no + # record waits on is not one of its edges. + pair_flags = {pair: flags for pair, flags in all_flags.items() if "wait" in flags} + # Insert unseen preds before their first consumer's position in `order`. + seen = set(order) + for pred, _succ in pair_flags: + if pred in seen: + continue + # Find the earliest consumer that is itself in the recorded order. + pos = len(order) + for p2, s2 in pair_flags: + if p2 == pred and s2 in seen: + pos = min(pos, order.index(s2)) + order.insert(pos, pred) + seen.add(pred) + seq = {task_id: i for i, task_id in enumerate(order)} + uncertain_retain_pairs = (explicit_pairs & pair_flags.keys()) - certain_retain_pairs + return ( + order, + seq, + {pair: frozenset(f) for pair, f in pair_flags.items()}, + frozenset(uncertain_retain_pairs), + ) + + +def full_reduction(order: list[str], pair_flags: dict[tuple[str, str], frozenset[str]]) -> set[tuple[str, str]]: + """Exact full-DAG transitive reduction of the WAIT graph (upper bound).""" + nodes = set(order) + for pred, succ in pair_flags: + nodes.add(pred) + nodes.add(succ) + succ_map: dict[str, set[str]] = {n: set() for n in nodes} + indeg: dict[str, int] = {n: 0 for n in nodes} + for u, v in pair_flags: + if v not in succ_map[u]: + succ_map[u].add(v) + indeg[v] += 1 + queue = deque(n for n, d in indeg.items() if d == 0) + topo: list[str] = [] + while queue: + n = queue.popleft() + topo.append(n) + for m in succ_map[n]: + indeg[m] -= 1 + if indeg[m] == 0: + queue.append(m) + if len(topo) != len(nodes): + raise ValueError("WAIT graph contains a cycle; reduction is ill-defined") + reach: dict[str, set[str]] = {} + redundant: set[tuple[str, str]] = set() + for u in reversed(topo): + indirect: set[str] = set() + for v in succ_map[u]: + indirect |= reach[v] + for v in succ_map[u]: + if v in indirect: + redundant.add((u, v)) + reach[u] = indirect | succ_map[u] + return redundant + + +def online_bitmap( + order: list[str], + seq: dict[str, int], + pair_flags: dict[tuple[str, str], frozenset[str]], + bl: int, +) -> set[tuple[str, str]]: + """Mirror of the runtime reduce_wait_edges at window size ``bl``.""" + mask = (1 << bl) - 1 + wait_preds: dict[str, list[str]] = {} + for pred, succ in pair_flags: + wait_preds.setdefault(succ, []).append(pred) + r_bitmaps: dict[str, int] = {t: 0 for t in order} + removed: set[tuple[str, str]] = set() + for t in order: + seq_t = seq[t] + direct = 0 + via = 0 + tracked: dict[str, int] = {} + for p in wait_preds.get(t, []): + d = seq_t - seq[p] + if d < 1 or d > bl: + continue # window miss: kept conservatively by the runtime too + tracked[p] = d + direct |= 1 << (d - 1) + if d < bl: + via |= r_bitmaps[p] << d + # Python ints are unbounded; the mask emulates the bl-wide register so + # bits shifted past the window drop exactly as they would in silicon. + r_bitmaps[t] = (direct | via) & mask + for p, d in tracked.items(): + if via & (1 << (d - 1)): + removed.add((p, t)) + return removed + + +def percentile(sorted_vals: list[int], frac: float) -> int: + if not sorted_vals: + return 0 + idx = min(int(frac * len(sorted_vals)), len(sorted_vals) - 1) + return sorted_vals[idx] + + +def simulate(path: Path, bls: list[int]) -> dict: + order, seq, pair_flags, uncertain_retain_pairs = load_wait_graph(path) + wait_pairs = {p for p, f in pair_flags.items() if "wait" in f} + redundant = full_reduction(order, pair_flags) + + pair_distances = {(p, s): seq[s] - seq[p] for (p, s) in wait_pairs} + distances = sorted(d for d in pair_distances.values() if d >= 0) + cross_ring_pairs = {(pred, succ) for (pred, succ) in wait_pairs if (int(pred) >> 32) != (int(succ) >> 32)} + report: dict = { + "file": str(path), + "tasks": len(order), + "wait_pairs": len(wait_pairs), + "full_reduction_upper_bound": len(redundant), + "full_demote_to_retain": sum(1 for pair in redundant if "retain" in pair_flags.get(pair, frozenset())), + "full_pure_drop": sum(1 for pair in redundant if "retain" not in pair_flags.get(pair, frozenset())), + "full_retain_classification_uncertain": len(redundant & uncertain_retain_pairs), + "seq_distance": { + "p50": percentile(distances, 0.50), + "p90": percentile(distances, 0.90), + "p99": percentile(distances, 0.99), + "max": distances[-1] if distances else 0, + }, + "windows": {}, + } + for bl in bls: + removed = online_bitmap(order, seq, pair_flags, bl) + within = sum(1 for d in distances if d <= bl) + window_miss_pairs = {pair for pair, d in pair_distances.items() if d < 1 or d > bl} + redundant_within = {pair for pair in redundant if pair_distances[pair] <= bl} + cross_ring_redundant = redundant & cross_ring_pairs + report["windows"][str(bl)] = { + "removed": len(removed), + "pct_of_upper_bound": (round(100.0 * len(removed) / len(redundant), 2) if redundant else 0.0), + "demote_to_retain": sum(1 for pair in removed if "retain" in pair_flags.get(pair, frozenset())), + "pure_drop": sum(1 for pair in removed if "retain" not in pair_flags.get(pair, frozenset())), + "retain_classification_uncertain": len(removed & uncertain_retain_pairs), + "pairs_within_window": within, + "pct_pairs_within_window": (round(100.0 * within / len(distances), 2) if distances else 0.0), + "window_miss_wait_pairs": len(window_miss_pairs), + "redundant_window_misses": len(redundant & window_miss_pairs), + "bitmap_misses_within_window": len(redundant_within - removed), + "cross_ring_wait_pairs": len(cross_ring_pairs), + "cross_ring_redundant_wait_pairs": len(cross_ring_redundant), + "cross_ring_removed": len(removed & cross_ring_pairs), + "cross_ring_misses": len(cross_ring_redundant - removed), + "estimated_readiness_fanout_nodes_removed": len(removed), + "estimated_dep_pool_entries_removed": len(removed), + } + return report + + +def main(argv: list[str] | None = None) -> int: + parser = argparse.ArgumentParser( + prog="wait_reduction_sim", + description="Measure bounded-bitmap WAIT reduction coverage against the full-DAG upper bound.", + ) + parser.add_argument("deps_json", nargs="+", type=Path, help="deps.json capture(s)") + parser.add_argument( + "--bl", + default="64,128,256", + help="comma-separated bitmap window sizes to simulate (default 64,128,256)", + ) + parser.add_argument("--json", type=Path, default=None, help="also write the report as JSON") + args = parser.parse_args(argv) + + bls = [int(x) for x in args.bl.split(",") if x.strip()] + reports = [] + for path in args.deps_json: + try: + reports.append(simulate(path, bls)) + except (OSError, ValueError, KeyError) as exc: + print(f"error: {path}: {exc}", file=sys.stderr) + return 1 + + for r in reports: + print(f"== {r['file']}") + print( + f" tasks={r['tasks']} wait_pairs={r['wait_pairs']} " + f"upper_bound={r['full_reduction_upper_bound']} " + f"(demote {r['full_demote_to_retain']} / drop {r['full_pure_drop']}; " + f"classification uncertain {r['full_retain_classification_uncertain']})" + ) + cdf = r["seq_distance"] + print(f" seq distance: p50={cdf['p50']} p90={cdf['p90']} p99={cdf['p99']} max={cdf['max']}") + for bl in bls: + w = r["windows"][str(bl)] + print( + f" BL={bl:>3}: removed={w['removed']:>5} " + f"({w['pct_of_upper_bound']:>6.2f}% of upper bound; " + f"demote {w['demote_to_retain']} / drop {w['pure_drop']}; " + f"window misses {w['redundant_window_misses']}; " + f"cross-ring misses {w['cross_ring_misses']}; " + f"{w['pct_pairs_within_window']:.1f}% pairs within window)" + ) + print() + + if args.json is not None: + args.json.write_text(json.dumps(reports, indent=1)) + print(f"json report written to {args.json}") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/src/a2a3/runtime/tensormap_and_ringbuffer/docs/RUNTIME_LOGIC.md b/src/a2a3/runtime/tensormap_and_ringbuffer/docs/RUNTIME_LOGIC.md index 7e834a17c5..991f667daa 100644 --- a/src/a2a3/runtime/tensormap_and_ringbuffer/docs/RUNTIME_LOGIC.md +++ b/src/a2a3/runtime/tensormap_and_ringbuffer/docs/RUNTIME_LOGIC.md @@ -392,7 +392,9 @@ When `OrchestratorState::submit_task` processes parameters: | `is_tensor[16]` | Whether each parameter is tensor or scalar | | `param_count` | Number of valid parameters | | `fanin_slot_states[]` | Producer slot state pointers (used by `on_task_release`) | -| `fanin_actual_count` | Actual fanin count | +| `fanin_actual_count` | Total fanin edge count (all flag combinations, including RETAIN-only and reduction-dropped entries) | +| `fanin_wait_count` | DEP_WAIT edge count after reduction — the readiness denominator | +| `early_dispatch_blocked` | set when reduction dropped an edge to a producer that does not allow early resolve; `early_dispatch_target()` = `fanin_wait_count` + this, so that producer keeps this task ineligible for early dispatch exactly as it did before reduction | ### 6.2 Task State Machine @@ -426,6 +428,7 @@ Key members: - `scope_tasks[]`, `scope_begins[]`, `scope_stack_top`: scope nesting stack (flat buffer partitioned by level) - `scheduler`: pointer to scheduler state (for Orch-side wiring helpers and ready queue access) - `gm_heap_base`, `gm_heap_size`: GM heap for output buffers +- `wait_reach[CHIP_MAX_RING_DEPTH]`, `submit_seq`: per-slot frozen WAIT-ancestor bitmap plus its global submission sequence (`WaitReachEntry`, 16 B per slot), backing the step-5.5 bounded reduction. Orchestrator-private runtime-arena storage, never read by scheduler threads. Costs `16 B x window_size` per ring — **1 MiB** at the default 4 rings x 16384 slots, and linear in `runtime_env.ring_task_window` if a ring is enlarged. ### 7.2 Task Submission Flow (`OrchestratorState::submit_task`) @@ -437,7 +440,8 @@ Key members: | 3 | **Lookup**: for each INPUT/INOUT param, search TensorMap for producers; collect producer pointers in `FaninBuilder` | | 4 | **Insert**: register OUTPUT/INOUT args in TensorMap | | 5 | **Record fanin metadata**: store producer edges (slot pointer + `DepFlags` packed in the low bits) in `payload->fanin_inline_edges[]` (+ spill pool if >64); claim each live producer by incrementing `fanout_count` under that producer's `fanout_lock`. Creator edges are `DEP_WAIT\|DEP_RETAIN`, tensormap-modifier edges `DEP_WAIT`. This step runs **before** `payload.init()`. | -| 6 | **Orch-side wiring / ready publish**: the orchestrator wires live fanout edges into the per-ring dep_pool; zero-fanin and already-completed fanin tasks publish directly to ready queues. Only `DEP_WAIT` edges gate readiness — they count toward `fanin_count` and are linked onto the producer's `fanout_head` for completion notification. A `DEP_WAIT`-only edge releases its submit→wire retention pin **at wiring** (and on the already-completed fast path), so its producer can be CONSUMED without waiting for this consumer; a `DEP_RETAIN` edge keeps the pin until this consumer's `on_task_release`. A hypothetical `RETAIN`-only edge (none exist yet) would neither gate readiness nor link a fanout node — it only holds the lifetime pin. | +| 5.5 | **Bounded transitive reduction** (`reduce_wait_edges`, issue #1376): publish this task's frozen 64-bit WAIT-ancestor bitmap over the last `WAIT_REACH_WINDOW` global submissions, then clear `DEP_WAIT` on any direct edge already covered by a WAIT path through another producer's transitive ancestors (`WAIT\|RETAIN` demotes to RETAIN-only; `WAIT`-only drops to `DEP_NONE`, the entry stays for pin accounting). Candidates beyond the window keep their WAIT. Runs before the payload flush, so `fanin_wait_count` reflects the reduced readiness set. | +| 6 | **Orch-side wiring / ready publish**: the orchestrator wires live fanout edges into the per-ring dep_pool; zero-fanin and already-completed fanin tasks publish directly to ready queues. Only `DEP_WAIT` edges gate readiness — they count toward `fanin_count` and are linked onto the producer's `fanout_head` for completion notification. A `DEP_WAIT`-only edge releases its submit→wire retention pin **at wiring** (and on the already-completed fast path), so its producer can be CONSUMED without waiting for this consumer; a `DEP_RETAIN` edge keeps the pin until this consumer's `on_task_release`. A `RETAIN`-only edge — produced by step 5.5 when it demotes a redundant `WAIT\|RETAIN` edge — neither gates readiness nor links a fanout node; it only holds the lifetime pin until this consumer's `on_task_release`. | > **Note**: Fanout wiring is now completed before publish in the orchestrator submit path. > Scheduler threads consume ready queues directly. diff --git a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/orchestrator.cpp b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/orchestrator.cpp index 399131a03c..41d8e6d660 100644 --- a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/orchestrator.cpp +++ b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/orchestrator.cpp @@ -246,6 +246,9 @@ struct FaninBuilder { spill_pool(spill_pool) {} int32_t count{0}; // total fanin edges (all flag combinations) int32_t wait_count{0}; // edges carrying DEP_WAIT — sizes readiness accounting + // Set when reduction cleared DEP_WAIT on an edge to a producer that does + // not allow early resolve; flushed to TaskPayload::early_dispatch_blocked. + bool reduced_unflagged_producer{false}; int32_t spill_start{0}; OrchestratorState *orch{nullptr}; uint32_t seen_epoch{0}; @@ -257,6 +260,15 @@ struct FaninBuilder { return for_each_fanin_storage(inline_slots, count, spill_start, spill_pool, static_cast(fn)); } + // Mutating walk over the same edges for_each visits, handing the callback + // the entry itself so flags can be rewritten (used by the reduction pass). + template + void for_each_entry(Fn &&fn) { + for (int32_t i = 0; i < count; i++) { + fn(entry_at(i)); + } + } + bool mark_seen(uint8_t prod_ring, int32_t prod_slot) { if (prod_ring >= CHIP_MAX_RING_DEPTH || prod_slot < 0) { return false; @@ -436,6 +448,100 @@ static bool all_claimed_fanin_allow_early_resolve(const FaninBuilder &fanin_buil }); } +// Publish this task's frozen WAIT-ancestor bitmap. Runs once per submit, before +// the slot becomes a producer of any later submit (single orchestrator thread), +// so a reader always observes the entry of the task generation that owns the +// slot. +static void publish_wait_reach(OrchestratorState *orch, uint8_t ring, int32_t slot, uint64_t ancestors) { + orch->wait_reach[ring][slot].ancestors = ancestors; +} + +// Resolve a producer slot pointer to its (ring, slot) index. ring_id is +// per-slot invariant; pointer subtraction from the ring's slot_states base +// never dereferences a possibly-reused slot. +static bool slot_index_of(const OrchestratorState *orch, ChipTaskSlotState *p, uint8_t &ring, int32_t &slot) { + ring = p->ring_id; + if (ring >= CHIP_MAX_RING_DEPTH) return false; + const SharedMemoryRingHeader &r = orch->sm_header->rings[ring]; + slot = static_cast(p - r.slot_states); + return slot >= 0 && static_cast(slot) < r.task_window_size; +} + +// Two-pass bounded transitive reduction of the WAIT graph, run once per submit +// on the fully-built fanin, before it is flushed to the payload and wired. +// +// Pass 1 folds `direct` (this task's WAIT producers, one bit each at distance +// d-1) and `via` (ancestors reachable through another direct WAIT producer q: +// R[q] << d, distance addition), then publishes R[t] = direct | via. Pass 2 +// clears DEP_WAIT on any tracked candidate whose bit is set in `via` — some +// direct producer proves a WAIT path p -> ... -> q -> t — demoting +// WAIT|RETAIN to RETAIN and WAIT-only to DEP_NONE (the entry stays in storage +// so its submit-claim pin is released by the !DEP_RETAIN paths). Both passes +// recompute d from the same seqs and fold with OR, so candidate enumeration +// order (creator vs tensormap vs explicit) cannot change the result. +// +// Only DEP_WAIT edges participate in reachability: a RETAIN-only candidate +// sets no bit and merges no ancestors, because the last hop to t must be a +// WAIT edge for the path to carry ordering. A candidate with +// d > WAIT_REACH_WINDOW keeps its WAIT and contributes nothing to R[t] — it +// and all its ancestors are unrepresentable in the window. d == WAIT_REACH_ +// WINDOW sets only the direct bit: the shift would be undefined and every +// ancestor of that producer already lies outside t's window. +// +// Every producer in the builder is pinned for this submit: append_fanin_or_fail +// claimed it with fanout_count++ under the producer's fanout_lock, atomically +// with the consumed/generation check, for every edge regardless of DepFlags. +// Slot rebind requires the CONSUMED flip (which requires +// fanout_refcount == fanout_count) plus allocator reclaim, so a pinned +// producer's slot — and therefore its wait_reach entry and seq — still belongs +// to the claimed task for the whole reduction pass. The bitmap is a frozen +// value published at the producer's own submit, so unlike a fanin-pointer +// walk there is no staleness dimension beyond slot identity, and slot identity +// is settled by the pin. +static void reduce_wait_edges(OrchestratorState *orch, FaninBuilder *b, uint8_t ring, int32_t slot, uint64_t seq_t) { + uint64_t direct = 0; + uint64_t via = 0; + b->for_each([&](ChipTaskSlotState *p, DepFlags flags) { + if (!dep_has_wait(flags)) return; + uint8_t pring; + int32_t pslot; + if (!slot_index_of(orch, p, pring, pslot)) return; + const WaitReachEntry &entry = orch->wait_reach[pring][pslot]; + uint64_t d = seq_t - entry.seq; // global seq: ring-independent, unsigned-exact + if (d == 0 || d > static_cast(WAIT_REACH_WINDOW)) return; + direct |= 1ull << (d - 1); + if (d < static_cast(WAIT_REACH_WINDOW)) { + via |= entry.ancestors << d; + } + }); + // A via bit lands at index (i + d) for ancestor bit i >= 0 of a producer at + // distance d >= 1, so index 0 is unreachable: the nearest direct producer + // can never be proven redundant. A set bit 0 means the shift-merge has + // drifted (the classic form is shifting by d - 1) and reduction is about to + // drop an edge nothing covers. + always_assert((via & 1ull) == 0 && "via bit 0 set: distance-1 producers are not reducible"); + publish_wait_reach(orch, ring, slot, direct | via); + if (via == 0) return; + b->for_each_entry([&](FaninSpillEntry &entry) { + DepFlags f = entry.flags(); + if (!dep_has_wait(f)) return; + uint8_t pring; + int32_t pslot; + if (!slot_index_of(orch, entry.slot_state(), pring, pslot)) return; + uint64_t d = seq_t - orch->wait_reach[pring][pslot].seq; + if (d == 0 || d > static_cast(WAIT_REACH_WINDOW)) return; + if ((via & (1ull << (d - 1))) == 0) return; + // A producer that does not allow early resolve never propagates + // dispatch_fanin. Record that before the edge leaves wait_count, so the + // early-dispatch target keeps the unit this producer used to hold. + if (!entry.slot_state()->task_attrs.allow_early_resolve()) { + b->reduced_unflagged_producer = true; + } + entry.set(entry.slot_state(), static_cast(f & ~DEP_WAIT)); + b->wait_count--; + }); +} + void OrchestratorState::mark_dep_pool_position(ChipTaskSlotState &slot_state) { SchedulerState *sched = scheduler; auto &rss = sched->ring_sched_states[slot_state.ring_id]; @@ -463,6 +569,9 @@ void OrchestratorState::wire_fanin_task(ChipTaskSlotState &slot_state, int32_t w if (dep_has_wait(flags)) { producer->lock_fanout(); int32_t pstate = producer->task_state.load(std::memory_order_acquire); + // Only WAIT producers reach this check. An unflagged producer whose + // edge reduction demoted is not one of them, and is accounted for by + // early_dispatch_blocked raising early_dispatch_target instead. if (!early_disqualified && !producer->task_attrs.allow_early_resolve()) { early_disqualified = true; } @@ -482,8 +591,9 @@ void OrchestratorState::wire_fanin_task(ChipTaskSlotState &slot_state, int32_t w // consumer is linked. With the consumer now on fanout_head (or already // seen as completed), release it so the producer can be CONSUMED without // waiting for this consumer. A DEP_RETAIN edge keeps the pin until the - // consumer's on_task_release. - if (dep_has_wait(flags) && !dep_has_retain(flags)) { + // consumer's on_task_release. A reduction-dropped (DEP_NONE) edge is + // released here too: it never links, so wiring is its only release point. + if (!dep_has_retain(flags)) { // Wiring-phase atomics (this release, plus the lock_fanout / dep_pool.prepend / // fanin_refcount ops around it) are not bucketed: g_orch_args_atomic_count // covers the submit/dep-claim phase only, whose g_orch_args_cycle window has @@ -500,7 +610,8 @@ void OrchestratorState::wire_fanin_task(ChipTaskSlotState &slot_state, int32_t w int32_t dispatch_fanin = payload->dispatch_fanin.fetch_add(early_seed, std::memory_order_acq_rel) + early_seed; // A fully pre-completed fanin routes normally. If any producer was live, // the exact-full increment must enqueue the early candidate. - if (completed_fanin != payload->fanin_actual_count && dispatch_fanin == payload->fanin_actual_count) { + int32_t target = payload->early_dispatch_target(); + if (completed_fanin != target && dispatch_fanin == target) { sched->try_enqueue_early_dispatch_candidate(slot_state); } } @@ -548,6 +659,7 @@ struct PreparedTask { TaskDescriptor *task = nullptr; TaskPayload *payload = nullptr; ChipTaskSlotState *slot_state = nullptr; + uint64_t seq = 0; // global submission sequence assigned in prepare_task }; static OutputLayout calculate_output_layout(const CoreTaskArgs &args) { @@ -632,6 +744,20 @@ static bool prepare_task( out->slot_state->reset_for_reuse(); out->slot_state->fanin_count = 0; + // Assign the global submission sequence and stamp the slot's side entry + // before any consumer can observe the slot: every task-consuming path runs + // on this single thread, and a later submit reaches this slot as a + // producer only after this submit returns. alloc_tensors shares this + // path, so hidden alloc tasks get a seq without entering + // submit_task_common. + // + // Both fields are written together so the entry always describes one task + // generation. An empty bitmap contributes no `via` bit, so a slot whose + // submit fails between here and reduce_wait_edges' publication carries a + // conservative entry rather than the previous generation's ancestors. + out->seq = orch->submit_seq++; + orch->wait_reach[ring_id][out->alloc_result.slot] = WaitReachEntry{0, out->seq}; + out->payload->prefetch(args.tensor_count(), args.scalar_count()); // Re-bind payload/task pointers each submit. Value is per-slot constant @@ -1037,19 +1163,28 @@ static TaskOutputTensors submit_task_common( // the producer's fanout_lock. Doing it there (rather than a separate pass // here) is what prevents a producer from transitioning to CONSUMED between // the dependency decision and the claim. + // + // Bounded transitive reduction runs on the fully-built fanin, before it is + // flushed to the payload and wired, so RETAIN-only and dropped edges are + // reflected in every downstream count. Publication of R[t] happens inside, + // so every slot that can later be read as a producer carries its own + // bitmap. + reduce_wait_edges(orch, &fanin_builder, ring_id, prepared.alloc_result.slot, prepared.seq); int32_t inline_count = std::min(fanin_builder.count, CHIP_FANIN_INLINE_CAP); - // Every fanin edge produced here carries DEP_WAIT (creator = WAIT|RETAIN, - // modifier = WAIT, explicit defaults to WAIT|RETAIN or opts into WAIT), so - // wait_count == count. fanin_actual_count therefore doubles as the WAIT-edge - // count that the early-dispatch threshold (dispatch_fanin, which counts only - // WAIT producers) is compared against. A future RETAIN-only edge would break - // that equality and must carry its own WAIT-edge count for that comparison. + // fanin_actual_count is the TOTAL edge count (for_each iteration and + // on_task_release walk every edge, including RETAIN-only and + // reduction-dropped ones); fanin_wait_count is the readiness WAIT-edge count + // after reduction, always <= fanin_actual_count. The early-dispatch + // denominator is early_dispatch_target(), which re-adds the unit a + // reduced-away unflagged producer used to hold. always_assert( - fanin_builder.wait_count == fanin_builder.count && - "fanin_actual_count is the early-dispatch WAIT denominator; a non-WAIT edge needs a separate count" + fanin_builder.wait_count <= fanin_builder.count && + "fanin_wait_count is the readiness WAIT denominator and never exceeds the edge total" ); // Store fanin metadata in payload for scheduler to iterate payload.fanin_actual_count = fanin_builder.count; + payload.fanin_wait_count = fanin_builder.wait_count; + payload.early_dispatch_blocked = fanin_builder.reduced_unflagged_producer ? 1 : 0; // fanin_builder.count is finalized here and submit runs once per task, so // each dense consumer emits one debug message when enabled. This checks > // THRESHOLD because the count lands at its final total here. @@ -1117,14 +1252,15 @@ static TaskOutputTensors submit_task_common( int32_t ready_seed = fanin_builder.wait_count + 1; cur_slot_state.fanin_count = ready_seed; if (all_claimed_fanin_allow_early_resolve(fanin_builder)) { - payload.dispatch_fanin.store(fanin_builder.count, std::memory_order_release); + payload.dispatch_fanin.store(fanin_builder.wait_count, std::memory_order_release); } cur_slot_state.fanin_refcount.store(ready_seed, std::memory_order_release); // wire_fanin_task is skipped here, so its ordering-only pin release runs - // on this path too: an edge without retention drops its submit->wire pin - // so the (already completed) producer can be CONSUMED. + // on this path too: an edge without retention — including a + // reduction-dropped DEP_NONE edge — drops its submit->wire pin so the + // (already completed) producer can be CONSUMED. for_each_fanin_slot_state(payload, [&](ChipTaskSlotState *producer, DepFlags flags) { - if (dep_has_wait(flags) && !dep_has_retain(flags)) { + if (!dep_has_retain(flags)) { sched->release_producer(*producer); // wiring-phase atomic, not bucketed (see wire_fanin_task) } }); @@ -1327,8 +1463,15 @@ TaskOutputTensors OrchestratorState::alloc_tensors(const CoreTaskArgs &args) { outputs.set_task_id(prepared.task_id); payload.init(args, outputs, prepared.alloc_result, layout); payload.fanin_actual_count = 0; + payload.fanin_wait_count = 0; + payload.early_dispatch_blocked = 0; payload.fanin_spill_start = 0; payload.fanin_spill_pool = &orch->rings[simpler::tmr::task_ring(prepared.task_id)].fanin_pool; + // A hidden alloc task has no WAIT predecessors: its reachability bitmap is + // empty and its seq was stamped by prepare_task, so a consumer reading it + // as a creator producer sees a valid, conservative entry rather than a + // stale slot generation. + publish_wait_reach(orch, simpler::tmr::task_ring(prepared.task_id), prepared.alloc_result.slot, 0); CYCLE_COUNT_LAP(g_orch_args_cycle); if (prepared.slot_state != nullptr) { diff --git a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/orchestrator.h b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/orchestrator.h index 126fc56f0c..1aec6a89f6 100644 --- a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/orchestrator.h +++ b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/orchestrator.h @@ -45,6 +45,7 @@ struct OrchestratorLayout { size_t off_fanin_pool[CHIP_MAX_RING_DEPTH]; size_t off_fanin_seen_epoch[CHIP_MAX_RING_DEPTH]; + size_t off_wait_reach[CHIP_MAX_RING_DEPTH]; size_t off_scope_tasks; size_t off_scope_begins; ChipTensorMapLayout tensor_map; @@ -71,6 +72,18 @@ struct OrchestratorState { uint32_t *fanin_seen_epoch[CHIP_MAX_RING_DEPTH]; uint32_t fanin_seen_current_epoch{1}; + // Per-slot frozen WAIT-ancestor reachability (bitmap + submit seq), + // indexed [ring][slot]. Orchestrator-private runtime-arena storage, never + // read by scheduler threads. A slot's entry is valid only while the slot + // holds the task that published it: every consumer that reads a producer's + // entry holds that producer's submit-claim pin (fanout_count), so the slot + // cannot be rebound under the read. + WaitReachEntry *wait_reach[CHIP_MAX_RING_DEPTH]; + + // Global submission sequence. Assigned once per prepared task across all + // rings; unsigned subtraction preserves recent distances across uint64 wrap. + uint64_t submit_seq{0}; + // === TENSOR MAP (Private) === ChipTensorMap tensor_map; // Producer lookup diff --git a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/runtime_types.h b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/runtime_types.h index 4ef15b716d..2e0fd14720 100644 --- a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/runtime_types.h +++ b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/runtime_types.h @@ -96,6 +96,12 @@ // Fanin storage #define CHIP_FANIN_INLINE_CAP 64 +// Bounded WAIT-ancestor reachability window, in tasks. One native word; the +// shift-merge in reduce_wait_edges relies on WAIT_REACH_WINDOW == 64 (a +// d == WAIT_REACH_WINDOW shift would be undefined behavior, and every ancestor +// of such a producer is already outside the window anyway). +inline constexpr int WAIT_REACH_WINDOW = 64; + // Dependency-degree diagnostic: log once at debug level when a task's fanin or // a producer's fanout first exceeds this degree, so dense dependency graphs can // be inspected without adding noise to normal runtime logs. @@ -193,6 +199,17 @@ struct FaninSpillEntry { }; static_assert(sizeof(FaninSpillEntry) == sizeof(uintptr_t)); +// Per-slot frozen WAIT-ancestor reachability. `ancestors` bit i set means the +// task with submit_seq == seq - i - 1 has a WAIT path to this slot's task; +// published once at the owning submit, immutable for the slot's task +// generation. `seq` is the global submission sequence of that task. Both +// fields are written by the orchestrator thread only. +struct WaitReachEntry { + uint64_t ancestors; + uint64_t seq; +}; +static_assert(sizeof(WaitReachEntry) == 16, "WaitReachEntry is one 16B stride"); + /** * Dependency list entry (singly-linked list node) * Stored in DepListPool ring buffer. @@ -299,7 +316,8 @@ struct TaskPayload { // Candidate detection is the event-driven dual of fanin_refcount. Wiring // seeds producers already complete, and flagged producers increment the // count only after all logical blocks are launch-visible. Equality with - // fanin_actual_count makes the consumer eligible for early dispatch. + // fanin_wait_count (only DEP_WAIT producers link onto fanout_head and bump + // it) makes the consumer eligible for early dispatch. std::atomic dispatch_fanin{0}; // CONSUMER side: fully-published + pre-completed producers // Claimed-but-unpublished blocks are not launch-visible. Seq_cst updates // pair with early_dispatch_state so final publication cannot be lost when @@ -322,6 +340,30 @@ struct TaskPayload { // records that producer release observed the owner; only cancellation clears // ownership before payload reinitialization. std::atomic early_sync_drain_state{EARLY_SYNC_DRAIN_NONE}; + // Set when transitive reduction cleared DEP_WAIT on an edge whose producer + // does not allow early resolve. Such a producer never propagates + // dispatch_fanin, so while it was in fanin_wait_count the count was + // permanently short and this task could not early-dispatch. Reduction takes + // it out of that count, so the unreachable unit is carried here instead -- + // see early_dispatch_target(). Occupies alignment padding ahead of + // fanin_wait_count, so the payload layout is unchanged. + uint8_t early_dispatch_blocked{0}; + // Number of DEP_WAIT fanin edges (readiness-bearing), <= fanin_actual_count. + // fanin_actual_count is the TOTAL edge count that for_each iteration and + // on_task_release walk (including RETAIN-only and reduction-dropped edges); + // this is the WAIT-edge count the early-dispatch threshold compares + // dispatch_fanin against. They differ once transitive reduction produces + // RETAIN-only or dropped edges that carry no DEP_WAIT. Occupies the padding + // between the early-dispatch block and the 64B-aligned predicate, so the + // payload layout is unchanged. + int32_t fanin_wait_count{0}; + // The count dispatch_fanin must hit for this task to be an early-dispatch + // candidate. Only producers linked onto a fanout_head ever bump + // dispatch_fanin, so it can reach at most fanin_wait_count: a nonzero + // early_dispatch_blocked therefore makes this target permanently + // unreachable, which is exactly what the reduced-away producer did while it + // was still counted. + int32_t early_dispatch_target() const { return fanin_wait_count + early_dispatch_blocked; } // === Cache line 9 (byte 576) — dispatch predicate (AICPU-only) === // Offset is a fixed 576, independent of MAX_TENSOR_ARGS / MAX_SCALAR_ARGS. // AICore never reads it — args are materialized from the tensor_count / tensors @@ -415,6 +457,13 @@ struct TaskPayload { // TaskPayload layout verification (offsetof requires complete type). static_assert(offsetof(TaskPayload, fanin_spill_pool) == 16, "spill pool pointer layout drift"); static_assert(offsetof(TaskPayload, fanin_inline_edges) == 24, "inline fanin array must follow spill metadata"); +static_assert( + offsetof(TaskPayload, fanin_wait_count) >= + offsetof(TaskPayload, early_sync_drain_state) + sizeof(TaskPayload::early_sync_drain_state) && + offsetof(TaskPayload, fanin_wait_count) + sizeof(TaskPayload::fanin_wait_count) <= + offsetof(TaskPayload, predicate), + "fanin_wait_count occupies the padding between the early-dispatch block and the 64B-aligned predicate" +); static_assert( offsetof(TaskPayload, predicate) == 576, "dispatch predicate occupies cache line 9 at fixed byte 576 (before tensors, never moves)" diff --git a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler.h b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler.h index 7013e5acf5..0dd8a8c862 100644 --- a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler.h +++ b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler.h @@ -957,7 +957,7 @@ struct SchedulerState { // FLAGGED producer `p` publishes blocks (normal dispatch, early-dispatch release, or // sync_start staging), but no-ops until every logical block is launch-visible. Only then // does it walk p's fanout and bump each consumer's - // dispatch_fanin. A consumer whose dispatch_fanin reaches fanin_actual_count (= every + // dispatch_fanin. A consumer whose dispatch_fanin reaches early_dispatch_target() (= every // producer is flagged-and-fully-dispatched, or was already complete when the consumer was // wired) is an early-dispatch candidate: CAS NONE->STAGING (exactly-once) and push to // early_dispatch_queues[shape] (or early_sync_start_queue for a require_sync_start cohort) @@ -997,15 +997,21 @@ struct SchedulerState { for (; edge != nullptr; edge = edge->next) { ChipTaskSlotState *c = edge->slot_state; if (c->task_attrs.has_predicate()) continue; // predicated consumers never early-dispatch - // Compare to fanin_actual_count (the real producer-edge count), NOT - // fanin_count: fanin_count = fanin_actual_count + 1 (a self/wiring +1 that - // ready_fanin gets but dispatch_fanin does not). dispatch_fanin starts at - // the wiring-time flagged-pre-completed seed and is bumped here by flagged - // producers; reaching fanin_actual_count means every producer is - // flagged-and-fully-published or was pre-completed. An unflagged producer leaves the - // seed short and never bumps, so this stays unreachable for that consumer. + // Compare to early_dispatch_target(), NOT fanin_count or + // fanin_actual_count: fanin_count = fanin_wait_count + 1 (a + // self/wiring +1 that ready_fanin gets but dispatch_fanin does not), + // and fanin_actual_count also counts RETAIN-only and + // reduction-dropped edges, which never link onto fanout_head and so + // never bump dispatch_fanin. dispatch_fanin starts at the + // wiring-time flagged-pre-completed seed and is bumped here by + // flagged producers; reaching the target means every WAIT producer + // is flagged-and-fully-published or was pre-completed. An unflagged + // producer leaves the seed short and never bumps -- whether it is + // still a WAIT producer or was reduced away and left its unit in + // early_dispatch_blocked -- so this stays unreachable for that + // consumer either way. int32_t nf = c->payload->dispatch_fanin.fetch_add(1, std::memory_order_acq_rel) + 1; - if (nf != c->payload->fanin_actual_count) continue; + if (nf != c->payload->early_dispatch_target()) continue; try_enqueue_early_dispatch_candidate(*c); } } diff --git a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/shared/runtime_init.cpp b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/shared/runtime_init.cpp index 28288a7304..abd511dd7a 100644 --- a/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/shared/runtime_init.cpp +++ b/src/a2a3/runtime/tensormap_and_ringbuffer/runtime/shared/runtime_init.cpp @@ -333,6 +333,10 @@ OrchestratorLayout OrchestratorState::reserve_layout( const size_t seen_epoch_bytes = CHIP_ALIGN_UP(static_cast(task_window_sizes[r]) * sizeof(uint32_t), CHIP_ALIGN_SIZE); layout.off_fanin_seen_epoch[r] = arena.reserve(seen_epoch_bytes, CHIP_ALIGN_SIZE); + + const size_t wait_reach_bytes = + CHIP_ALIGN_UP(static_cast(task_window_sizes[r]) * sizeof(WaitReachEntry), CHIP_ALIGN_SIZE); + layout.off_wait_reach[r] = arena.reserve(wait_reach_bytes, CHIP_ALIGN_SIZE); } layout.off_scope_tasks = arena.reserve(static_cast(layout.scope_tasks_cap) * sizeof(uintptr_t), alignof(ChipTaskSlotState *)); @@ -399,6 +403,13 @@ bool OrchestratorState::init_data_from_layout( auto *seen_epoch = static_cast(arena.region_ptr(layout.off_fanin_seen_epoch[r])); memset(seen_epoch, 0, seen_epoch_bytes); orch->fanin_seen_epoch[r] = seen_epoch; + + const size_t wait_reach_bytes = CHIP_ALIGN_UP( + static_cast(layout.tensor_map.task_window_sizes[r]) * sizeof(WaitReachEntry), CHIP_ALIGN_SIZE + ); + auto *wait_reach = static_cast(arena.region_ptr(layout.off_wait_reach[r])); + memset(wait_reach, 0, wait_reach_bytes); + orch->wait_reach[r] = wait_reach; } if (!orch->tensor_map.init_data_from_layout(layout.tensor_map, arena)) { @@ -428,6 +439,10 @@ bool OrchestratorState::reset_for_reuse( orch->gm_heap_size = total_heap_size; orch->fatal = false; orch->inline_completed_tasks = 0; + // The wait_reach entries are not cleared here: every slot publishes its + // bitmap before any submit can read it as a producer, so entries left from + // a previous run are unreachable. + orch->submit_seq = 0; uint32_t next_epoch = orch->fanin_seen_current_epoch + 1; if (next_epoch == 0) { @@ -481,6 +496,7 @@ void OrchestratorState::wire_arena_pointers( for (int r = 0; r < CHIP_MAX_RING_DEPTH; r++) { orch->rings[r].fanin_pool.base = static_cast(arena.region_ptr(layout.off_fanin_pool[r])); orch->fanin_seen_epoch[r] = static_cast(arena.region_ptr(layout.off_fanin_seen_epoch[r])); + orch->wait_reach[r] = static_cast(arena.region_ptr(layout.off_wait_reach[r])); } orch->tensor_map.wire_arena_pointers(layout.tensor_map, arena); orch->scope_tasks = static_cast(arena.region_ptr(layout.off_scope_tasks)); @@ -494,6 +510,7 @@ void OrchestratorState::destroy() { for (int r = 0; r < CHIP_MAX_RING_DEPTH; r++) { orch->rings[r].fanin_pool.base = nullptr; orch->fanin_seen_epoch[r] = nullptr; + orch->wait_reach[r] = nullptr; } orch->scope_tasks = nullptr; orch->scope_begins = nullptr; diff --git a/src/a5/runtime/tensormap_and_ringbuffer/docs/RUNTIME_LOGIC.md b/src/a5/runtime/tensormap_and_ringbuffer/docs/RUNTIME_LOGIC.md index 8353aced25..f031e4a671 100644 --- a/src/a5/runtime/tensormap_and_ringbuffer/docs/RUNTIME_LOGIC.md +++ b/src/a5/runtime/tensormap_and_ringbuffer/docs/RUNTIME_LOGIC.md @@ -402,7 +402,9 @@ When `OrchestratorState::submit_task` processes parameters: | `is_tensor[16]` | Whether each parameter is tensor or scalar | | `param_count` | Number of valid parameters | | `fanin_slot_states[]` | Producer slot state pointers (used by `on_task_release`) | -| `fanin_actual_count` | Actual fanin count | +| `fanin_actual_count` | Total fanin edge count (all flag combinations, including RETAIN-only and reduction-dropped entries) | +| `fanin_wait_count` | DEP_WAIT edge count after reduction — the readiness denominator | +| `early_dispatch_blocked` | set when reduction dropped an edge to a producer that does not allow early resolve; `early_dispatch_target()` = `fanin_wait_count` + this, so that producer keeps this task ineligible for early dispatch exactly as it did before reduction | ### 6.2 Task State Machine @@ -436,6 +438,7 @@ Key members: - `scope_tasks[]`, `scope_begins[]`, `scope_stack_top`: scope nesting stack (flat buffer partitioned by level) - `scheduler`: pointer to scheduler state (for Orch-side wiring helpers and ready queue access) - `gm_heap_base`, `gm_heap_size`: GM heap for output buffers +- `wait_reach[CHIP_MAX_RING_DEPTH]`, `submit_seq`: per-slot frozen WAIT-ancestor bitmap plus its global submission sequence (`WaitReachEntry`, 16 B per slot), backing the step-5.5 bounded reduction. Orchestrator-private runtime-arena storage, never read by scheduler threads. Costs `16 B x window_size` per ring — **1 MiB** at the default 4 rings x 16384 slots, and linear in `runtime_env.ring_task_window` if a ring is enlarged. ### 7.2 Task Submission Flow (`OrchestratorState::submit_task`) @@ -447,7 +450,8 @@ Key members: | 3 | **Lookup**: for each INPUT/INOUT param, search TensorMap for producers; collect producer pointers in `FaninBuilder` | | 4 | **Insert**: register OUTPUT/INOUT args in TensorMap | | 5 | **Record fanin metadata**: store producer edges (slot pointer + `DepFlags` packed in the low bits) in `payload->fanin_inline_edges[]` (+ spill pool if >64); claim each live producer by incrementing `fanout_count` under that producer's `fanout_lock`. Creator edges are `DEP_WAIT\|DEP_RETAIN`, tensormap-modifier edges `DEP_WAIT`. This step runs **before** `payload.init()`. | -| 6 | **Orch-side wiring / ready publish**: the orchestrator wires live fanout edges into the per-ring dep_pool; zero-fanin and already-completed fanin tasks publish directly to ready queues. Only `DEP_WAIT` edges gate readiness — they count toward `fanin_count` and are linked onto the producer's `fanout_head` for completion notification. A `DEP_WAIT`-only edge releases its submit→wire retention pin **at wiring** (and on the already-completed fast path), so its producer can be CONSUMED without waiting for this consumer; a `DEP_RETAIN` edge keeps the pin until this consumer's `on_task_release`. A hypothetical `RETAIN`-only edge (none exist yet) would neither gate readiness nor link a fanout node — it only holds the lifetime pin. | +| 5.5 | **Bounded transitive reduction** (`reduce_wait_edges`, issue #1376): publish this task's frozen 64-bit WAIT-ancestor bitmap over the last `WAIT_REACH_WINDOW` global submissions, then clear `DEP_WAIT` on any direct edge already covered by a WAIT path through another producer's transitive ancestors (`WAIT\|RETAIN` demotes to RETAIN-only; `WAIT`-only drops to `DEP_NONE`, the entry stays for pin accounting). Candidates beyond the window keep their WAIT. Runs before the payload flush, so `fanin_wait_count` reflects the reduced readiness set. | +| 6 | **Orch-side wiring / ready publish**: the orchestrator wires live fanout edges into the per-ring dep_pool; zero-fanin and already-completed fanin tasks publish directly to ready queues. Only `DEP_WAIT` edges gate readiness — they count toward `fanin_count` and are linked onto the producer's `fanout_head` for completion notification. A `DEP_WAIT`-only edge releases its submit→wire retention pin **at wiring** (and on the already-completed fast path), so its producer can be CONSUMED without waiting for this consumer; a `DEP_RETAIN` edge keeps the pin until this consumer's `on_task_release`. A `RETAIN`-only edge — produced by step 5.5 when it demotes a redundant `WAIT\|RETAIN` edge — neither gates readiness nor links a fanout node; it only holds the lifetime pin until this consumer's `on_task_release`. | > **Note**: Fanout wiring is now completed before publish in the orchestrator submit path. > Scheduler threads consume ready queues directly. diff --git a/src/a5/runtime/tensormap_and_ringbuffer/runtime/orchestrator.cpp b/src/a5/runtime/tensormap_and_ringbuffer/runtime/orchestrator.cpp index 52581bcc03..4dbf2eb13a 100644 --- a/src/a5/runtime/tensormap_and_ringbuffer/runtime/orchestrator.cpp +++ b/src/a5/runtime/tensormap_and_ringbuffer/runtime/orchestrator.cpp @@ -240,6 +240,9 @@ struct FaninBuilder { spill_pool(spill_pool) {} int32_t count{0}; // total fanin edges (all flag combinations) int32_t wait_count{0}; // edges carrying DEP_WAIT — sizes readiness accounting + // Set when reduction cleared DEP_WAIT on an edge to a producer that does + // not allow early resolve; flushed to TaskPayload::early_dispatch_blocked. + bool reduced_unflagged_producer{false}; int32_t spill_start{0}; OrchestratorState *orch{nullptr}; uint32_t seen_epoch{0}; @@ -251,6 +254,15 @@ struct FaninBuilder { return for_each_fanin_storage(inline_slots, count, spill_start, spill_pool, static_cast(fn)); } + // Mutating walk over the same edges for_each visits, handing the callback + // the entry itself so flags can be rewritten (used by the reduction pass). + template + void for_each_entry(Fn &&fn) { + for (int32_t i = 0; i < count; i++) { + fn(entry_at(i)); + } + } + bool mark_seen(uint8_t prod_ring, int32_t prod_slot) { if (prod_ring >= CHIP_MAX_RING_DEPTH || prod_slot < 0) { return false; @@ -429,6 +441,100 @@ static bool all_claimed_fanin_allow_early_resolve(const FaninBuilder &fanin_buil }); } +// Publish this task's frozen WAIT-ancestor bitmap. Runs once per submit, before +// the slot becomes a producer of any later submit (single orchestrator thread), +// so a reader always observes the entry of the task generation that owns the +// slot. +static void publish_wait_reach(OrchestratorState *orch, uint8_t ring, int32_t slot, uint64_t ancestors) { + orch->wait_reach[ring][slot].ancestors = ancestors; +} + +// Resolve a producer slot pointer to its (ring, slot) index. ring_id is +// per-slot invariant; pointer subtraction from the ring's slot_states base +// never dereferences a possibly-reused slot. +static bool slot_index_of(const OrchestratorState *orch, ChipTaskSlotState *p, uint8_t &ring, int32_t &slot) { + ring = p->ring_id; + if (ring >= CHIP_MAX_RING_DEPTH) return false; + const SharedMemoryRingHeader &r = orch->sm_header->rings[ring]; + slot = static_cast(p - r.slot_states); + return slot >= 0 && static_cast(slot) < r.task_window_size; +} + +// Two-pass bounded transitive reduction of the WAIT graph, run once per submit +// on the fully-built fanin, before it is flushed to the payload and wired. +// +// Pass 1 folds `direct` (this task's WAIT producers, one bit each at distance +// d-1) and `via` (ancestors reachable through another direct WAIT producer q: +// R[q] << d, distance addition), then publishes R[t] = direct | via. Pass 2 +// clears DEP_WAIT on any tracked candidate whose bit is set in `via` — some +// direct producer proves a WAIT path p -> ... -> q -> t — demoting +// WAIT|RETAIN to RETAIN and WAIT-only to DEP_NONE (the entry stays in storage +// so its submit-claim pin is released by the !DEP_RETAIN paths). Both passes +// recompute d from the same seqs and fold with OR, so candidate enumeration +// order (creator vs tensormap vs explicit) cannot change the result. +// +// Only DEP_WAIT edges participate in reachability: a RETAIN-only candidate +// sets no bit and merges no ancestors, because the last hop to t must be a +// WAIT edge for the path to carry ordering. A candidate with +// d > WAIT_REACH_WINDOW keeps its WAIT and contributes nothing to R[t] — it +// and all its ancestors are unrepresentable in the window. d == WAIT_REACH_ +// WINDOW sets only the direct bit: the shift would be undefined and every +// ancestor of that producer already lies outside t's window. +// +// Every producer in the builder is pinned for this submit: append_fanin_or_fail +// claimed it with fanout_count++ under the producer's fanout_lock, atomically +// with the consumed/generation check, for every edge regardless of DepFlags. +// Slot rebind requires the CONSUMED flip (which requires +// fanout_refcount == fanout_count) plus allocator reclaim, so a pinned +// producer's slot — and therefore its wait_reach entry and seq — still belongs +// to the claimed task for the whole reduction pass. The bitmap is a frozen +// value published at the producer's own submit, so unlike a fanin-pointer +// walk there is no staleness dimension beyond slot identity, and slot identity +// is settled by the pin. +static void reduce_wait_edges(OrchestratorState *orch, FaninBuilder *b, uint8_t ring, int32_t slot, uint64_t seq_t) { + uint64_t direct = 0; + uint64_t via = 0; + b->for_each([&](ChipTaskSlotState *p, DepFlags flags) { + if (!dep_has_wait(flags)) return; + uint8_t pring; + int32_t pslot; + if (!slot_index_of(orch, p, pring, pslot)) return; + const WaitReachEntry &entry = orch->wait_reach[pring][pslot]; + uint64_t d = seq_t - entry.seq; // global seq: ring-independent, unsigned-exact + if (d == 0 || d > static_cast(WAIT_REACH_WINDOW)) return; + direct |= 1ull << (d - 1); + if (d < static_cast(WAIT_REACH_WINDOW)) { + via |= entry.ancestors << d; + } + }); + // A via bit lands at index (i + d) for ancestor bit i >= 0 of a producer at + // distance d >= 1, so index 0 is unreachable: the nearest direct producer + // can never be proven redundant. A set bit 0 means the shift-merge has + // drifted (the classic form is shifting by d - 1) and reduction is about to + // drop an edge nothing covers. + always_assert((via & 1ull) == 0 && "via bit 0 set: distance-1 producers are not reducible"); + publish_wait_reach(orch, ring, slot, direct | via); + if (via == 0) return; + b->for_each_entry([&](FaninSpillEntry &entry) { + DepFlags f = entry.flags(); + if (!dep_has_wait(f)) return; + uint8_t pring; + int32_t pslot; + if (!slot_index_of(orch, entry.slot_state(), pring, pslot)) return; + uint64_t d = seq_t - orch->wait_reach[pring][pslot].seq; + if (d == 0 || d > static_cast(WAIT_REACH_WINDOW)) return; + if ((via & (1ull << (d - 1))) == 0) return; + // A producer that does not allow early resolve never propagates + // dispatch_fanin. Record that before the edge leaves wait_count, so the + // early-dispatch target keeps the unit this producer used to hold. + if (!entry.slot_state()->task_attrs.allow_early_resolve()) { + b->reduced_unflagged_producer = true; + } + entry.set(entry.slot_state(), static_cast(f & ~DEP_WAIT)); + b->wait_count--; + }); +} + void OrchestratorState::mark_dep_pool_position(ChipTaskSlotState &slot_state) { SchedulerState *sched = scheduler; auto &rss = sched->ring_sched_states[slot_state.ring_id]; @@ -457,6 +563,9 @@ void OrchestratorState::wire_fanin_task(ChipTaskSlotState &slot_state, int32_t w if (dep_has_wait(flags)) { producer->lock_fanout(); int32_t pstate = producer->task_state.load(std::memory_order_acquire); + // Only WAIT producers reach this check. An unflagged producer whose + // edge reduction demoted is not one of them, and is accounted for by + // early_dispatch_blocked raising early_dispatch_target instead. if (!early_disqualified && !producer->task_attrs.allow_early_resolve()) { early_disqualified = true; } @@ -476,8 +585,9 @@ void OrchestratorState::wire_fanin_task(ChipTaskSlotState &slot_state, int32_t w // consumer is linked. With the consumer now on fanout_head (or already // seen as completed), release it so the producer can be CONSUMED without // waiting for this consumer. A DEP_RETAIN edge keeps the pin until the - // consumer's on_task_release. - if (dep_has_wait(flags) && !dep_has_retain(flags)) { + // consumer's on_task_release. A reduction-dropped (DEP_NONE) edge is + // released here too: it never links, so wiring is its only release point. + if (!dep_has_retain(flags)) { // Wiring-phase atomics (this release, plus the lock_fanout / dep_pool.prepend / // fanin_refcount ops around it) are not bucketed: g_orch_args_atomic_count // covers the submit/dep-claim phase only, whose g_orch_args_cycle window has @@ -493,7 +603,8 @@ void OrchestratorState::wire_fanin_task(ChipTaskSlotState &slot_state, int32_t w int32_t dispatch_fanin = payload->dispatch_fanin.fetch_add(early_seed, std::memory_order_acq_rel) + early_seed; // Fully pre-completed fanin routes normally. If any producer was live, // the exact-full increment must enqueue the early candidate. - if (completed_fanin != payload->fanin_actual_count && dispatch_fanin == payload->fanin_actual_count) { + int32_t target = payload->early_dispatch_target(); + if (completed_fanin != target && dispatch_fanin == target) { sched->try_enqueue_early_dispatch_candidate(slot_state); } } @@ -540,6 +651,7 @@ struct PreparedTask { TaskDescriptor *task = nullptr; TaskPayload *payload = nullptr; ChipTaskSlotState *slot_state = nullptr; + uint64_t seq = 0; // global submission sequence assigned in prepare_task }; static OutputLayout calculate_output_layout(const CoreTaskArgs &args) { @@ -624,6 +736,20 @@ static bool prepare_task( out->slot_state->reset_for_reuse(); out->slot_state->fanin_count = 0; + // Assign the global submission sequence and stamp the slot's side entry + // before any consumer can observe the slot: every task-consuming path runs + // on this single thread, and a later submit reaches this slot as a + // producer only after this submit returns. alloc_tensors shares this + // path, so hidden alloc tasks get a seq without entering + // submit_task_common. + // + // Both fields are written together so the entry always describes one task + // generation. An empty bitmap contributes no `via` bit, so a slot whose + // submit fails between here and reduce_wait_edges' publication carries a + // conservative entry rather than the previous generation's ancestors. + out->seq = orch->submit_seq++; + orch->wait_reach[ring_id][out->alloc_result.slot] = WaitReachEntry{0, out->seq}; + out->payload->prefetch(args.tensor_count(), args.scalar_count()); // Re-bind payload/task pointers each submit. Value is per-slot constant @@ -1039,19 +1165,28 @@ static TaskOutputTensors submit_task_common( // the producer's fanout_lock. Doing it there (rather than a separate pass // here) is what prevents a producer from transitioning to CONSUMED between // the dependency decision and the claim. + // + // Bounded transitive reduction runs on the fully-built fanin, before it is + // flushed to the payload and wired, so RETAIN-only and dropped edges are + // reflected in every downstream count. Publication of R[t] happens inside, + // so every slot that can later be read as a producer carries its own + // bitmap. + reduce_wait_edges(orch, &fanin_builder, ring_id, prepared.alloc_result.slot, prepared.seq); int32_t inline_count = std::min(fanin_builder.count, CHIP_FANIN_INLINE_CAP); - // Every fanin edge produced here carries DEP_WAIT (creator = WAIT|RETAIN, - // modifier = WAIT, explicit defaults to WAIT|RETAIN or opts into WAIT), so - // wait_count == count. fanin_actual_count therefore doubles as the WAIT-edge - // count that the early-dispatch threshold (dispatch_fanin, which counts only - // WAIT producers) is compared against. A future RETAIN-only edge would break - // that equality and must carry its own WAIT-edge count for that comparison. + // fanin_actual_count is the TOTAL edge count (for_each iteration and + // on_task_release walk every edge, including RETAIN-only and + // reduction-dropped ones); fanin_wait_count is the readiness WAIT-edge count + // after reduction, always <= fanin_actual_count. The early-dispatch + // denominator is early_dispatch_target(), which re-adds the unit a + // reduced-away unflagged producer used to hold. always_assert( - fanin_builder.wait_count == fanin_builder.count && - "fanin_actual_count is the early-dispatch WAIT denominator; a non-WAIT edge needs a separate count" + fanin_builder.wait_count <= fanin_builder.count && + "fanin_wait_count is the readiness WAIT denominator and never exceeds the edge total" ); // Store fanin metadata in payload for scheduler to iterate payload.fanin_actual_count = fanin_builder.count; + payload.fanin_wait_count = fanin_builder.wait_count; + payload.early_dispatch_blocked = fanin_builder.reduced_unflagged_producer ? 1 : 0; // fanin_builder.count is finalized here and submit runs once per task, so // each dense consumer emits one debug message when enabled. This checks > // THRESHOLD because the count lands at its final total here. @@ -1119,14 +1254,15 @@ static TaskOutputTensors submit_task_common( int32_t ready_seed = fanin_builder.wait_count + 1; cur_slot_state.fanin_count = ready_seed; if (all_claimed_fanin_allow_early_resolve(fanin_builder)) { - payload.dispatch_fanin.store(fanin_builder.count, std::memory_order_release); + payload.dispatch_fanin.store(fanin_builder.wait_count, std::memory_order_release); } cur_slot_state.fanin_refcount.store(ready_seed, std::memory_order_release); // wire_fanin_task is skipped here, so its ordering-only pin release runs - // on this path too: an edge without retention drops its submit->wire pin - // so the (already completed) producer can be CONSUMED. + // on this path too: an edge without retention — including a + // reduction-dropped DEP_NONE edge — drops its submit->wire pin so the + // (already completed) producer can be CONSUMED. for_each_fanin_slot_state(payload, [&](ChipTaskSlotState *producer, DepFlags flags) { - if (dep_has_wait(flags) && !dep_has_retain(flags)) { + if (!dep_has_retain(flags)) { sched->release_producer(*producer); // wiring-phase atomic, not bucketed (see wire_fanin_task) } }); @@ -1329,8 +1465,15 @@ TaskOutputTensors OrchestratorState::alloc_tensors(const CoreTaskArgs &args) { outputs.set_task_id(prepared.task_id); payload.init(args, outputs, prepared.alloc_result, layout); payload.fanin_actual_count = 0; + payload.fanin_wait_count = 0; + payload.early_dispatch_blocked = 0; payload.fanin_spill_start = 0; payload.fanin_spill_pool = &orch->rings[simpler::tmr::task_ring(prepared.task_id)].fanin_pool; + // A hidden alloc task has no WAIT predecessors: its reachability bitmap is + // empty and its seq was stamped by prepare_task, so a consumer reading it + // as a creator producer sees a valid, conservative entry rather than a + // stale slot generation. + publish_wait_reach(orch, simpler::tmr::task_ring(prepared.task_id), prepared.alloc_result.slot, 0); CYCLE_COUNT_LAP(g_orch_args_cycle); if (prepared.slot_state != nullptr) { diff --git a/src/a5/runtime/tensormap_and_ringbuffer/runtime/orchestrator.h b/src/a5/runtime/tensormap_and_ringbuffer/runtime/orchestrator.h index 126fc56f0c..1aec6a89f6 100644 --- a/src/a5/runtime/tensormap_and_ringbuffer/runtime/orchestrator.h +++ b/src/a5/runtime/tensormap_and_ringbuffer/runtime/orchestrator.h @@ -45,6 +45,7 @@ struct OrchestratorLayout { size_t off_fanin_pool[CHIP_MAX_RING_DEPTH]; size_t off_fanin_seen_epoch[CHIP_MAX_RING_DEPTH]; + size_t off_wait_reach[CHIP_MAX_RING_DEPTH]; size_t off_scope_tasks; size_t off_scope_begins; ChipTensorMapLayout tensor_map; @@ -71,6 +72,18 @@ struct OrchestratorState { uint32_t *fanin_seen_epoch[CHIP_MAX_RING_DEPTH]; uint32_t fanin_seen_current_epoch{1}; + // Per-slot frozen WAIT-ancestor reachability (bitmap + submit seq), + // indexed [ring][slot]. Orchestrator-private runtime-arena storage, never + // read by scheduler threads. A slot's entry is valid only while the slot + // holds the task that published it: every consumer that reads a producer's + // entry holds that producer's submit-claim pin (fanout_count), so the slot + // cannot be rebound under the read. + WaitReachEntry *wait_reach[CHIP_MAX_RING_DEPTH]; + + // Global submission sequence. Assigned once per prepared task across all + // rings; unsigned subtraction preserves recent distances across uint64 wrap. + uint64_t submit_seq{0}; + // === TENSOR MAP (Private) === ChipTensorMap tensor_map; // Producer lookup diff --git a/src/a5/runtime/tensormap_and_ringbuffer/runtime/runtime_types.h b/src/a5/runtime/tensormap_and_ringbuffer/runtime/runtime_types.h index b467034bae..c62acbe5d6 100644 --- a/src/a5/runtime/tensormap_and_ringbuffer/runtime/runtime_types.h +++ b/src/a5/runtime/tensormap_and_ringbuffer/runtime/runtime_types.h @@ -96,6 +96,12 @@ // Fanin storage #define CHIP_FANIN_INLINE_CAP 64 +// Bounded WAIT-ancestor reachability window, in tasks. One native word; the +// shift-merge in reduce_wait_edges relies on WAIT_REACH_WINDOW == 64 (a +// d == WAIT_REACH_WINDOW shift would be undefined behavior, and every ancestor +// of such a producer is already outside the window anyway). +inline constexpr int WAIT_REACH_WINDOW = 64; + // Dependency-degree diagnostic: log once at debug level when a task's fanin or // a producer's fanout first exceeds this degree, so dense dependency graphs can // be inspected without adding noise to normal runtime logs. @@ -193,6 +199,17 @@ struct FaninSpillEntry { }; static_assert(sizeof(FaninSpillEntry) == sizeof(uintptr_t)); +// Per-slot frozen WAIT-ancestor reachability. `ancestors` bit i set means the +// task with submit_seq == seq - i - 1 has a WAIT path to this slot's task; +// published once at the owning submit, immutable for the slot's task +// generation. `seq` is the global submission sequence of that task. Both +// fields are written by the orchestrator thread only. +struct WaitReachEntry { + uint64_t ancestors; + uint64_t seq; +}; +static_assert(sizeof(WaitReachEntry) == 16, "WaitReachEntry is one 16B stride"); + /** * Dependency list entry (singly-linked list node) * Stored in DepListPool ring buffer. @@ -299,7 +316,8 @@ struct TaskPayload { // Candidate detection is the event-driven dual of fanin_refcount. Wiring // seeds producers already complete, and flagged producers increment the // count only after all logical blocks are launch-visible. Equality with - // fanin_actual_count makes the consumer eligible for early dispatch. + // fanin_wait_count (only DEP_WAIT producers link onto fanout_head and bump + // it) makes the consumer eligible for early dispatch. std::atomic dispatch_fanin{0}; // CONSUMER side: fully-published + pre-completed producers // Claimed-but-unpublished blocks are not launch-visible. Seq_cst updates // pair with early_dispatch_state so final publication cannot be lost when @@ -322,6 +340,30 @@ struct TaskPayload { // records that producer release observed the owner; only cancellation clears // ownership before payload reinitialization. std::atomic early_sync_drain_state{EARLY_SYNC_DRAIN_NONE}; + // Set when transitive reduction cleared DEP_WAIT on an edge whose producer + // does not allow early resolve. Such a producer never propagates + // dispatch_fanin, so while it was in fanin_wait_count the count was + // permanently short and this task could not early-dispatch. Reduction takes + // it out of that count, so the unreachable unit is carried here instead -- + // see early_dispatch_target(). Occupies alignment padding ahead of + // fanin_wait_count, so the payload layout is unchanged. + uint8_t early_dispatch_blocked{0}; + // Number of DEP_WAIT fanin edges (readiness-bearing), <= fanin_actual_count. + // fanin_actual_count is the TOTAL edge count that for_each iteration and + // on_task_release walk (including RETAIN-only and reduction-dropped edges); + // this is the WAIT-edge count the early-dispatch threshold compares + // dispatch_fanin against. They differ once transitive reduction produces + // RETAIN-only or dropped edges that carry no DEP_WAIT. Occupies the padding + // between the early-dispatch block and the 64B-aligned predicate, so the + // payload layout is unchanged. + int32_t fanin_wait_count{0}; + // The count dispatch_fanin must hit for this task to be an early-dispatch + // candidate. Only producers linked onto a fanout_head ever bump + // dispatch_fanin, so it can reach at most fanin_wait_count: a nonzero + // early_dispatch_blocked therefore makes this target permanently + // unreachable, which is exactly what the reduced-away producer did while it + // was still counted. + int32_t early_dispatch_target() const { return fanin_wait_count + early_dispatch_blocked; } // === Cache line 9 (byte 576) — dispatch predicate (AICPU-only) === // Offset is a fixed 576, independent of MAX_TENSOR_ARGS / MAX_SCALAR_ARGS. // AICore never reads it — args are materialized from the tensor_count / tensors @@ -415,6 +457,13 @@ struct TaskPayload { // TaskPayload layout verification (offsetof requires complete type). static_assert(offsetof(TaskPayload, fanin_spill_pool) == 16, "spill pool pointer layout drift"); static_assert(offsetof(TaskPayload, fanin_inline_edges) == 24, "inline fanin array must follow spill metadata"); +static_assert( + offsetof(TaskPayload, fanin_wait_count) >= + offsetof(TaskPayload, early_sync_drain_state) + sizeof(TaskPayload::early_sync_drain_state) && + offsetof(TaskPayload, fanin_wait_count) + sizeof(TaskPayload::fanin_wait_count) <= + offsetof(TaskPayload, predicate), + "fanin_wait_count occupies the padding between the early-dispatch block and the 64B-aligned predicate" +); static_assert( offsetof(TaskPayload, predicate) == 576, "dispatch predicate occupies cache line 9 at fixed byte 576 (before tensors, never moves)" diff --git a/src/a5/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler.h b/src/a5/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler.h index ced25fe2bf..dda0496987 100644 --- a/src/a5/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler.h +++ b/src/a5/runtime/tensormap_and_ringbuffer/runtime/scheduler/scheduler.h @@ -963,8 +963,13 @@ struct SchedulerState { for (; edge != nullptr; edge = edge->next) { ChipTaskSlotState *c = edge->slot_state; if (c->task_attrs.has_predicate()) continue; + // early_dispatch_target() is the WAIT-edge count plus any unit held + // by a reduced-away unflagged producer: only DEP_WAIT producers link + // onto fanout_head and bump dispatch_fanin, so RETAIN-only and + // reduction-dropped edges (still counted by fanin_actual_count) stay + // out of the count while still being able to hold the target short. int32_t nf = c->payload->dispatch_fanin.fetch_add(1, std::memory_order_acq_rel) + 1; - if (nf != c->payload->fanin_actual_count) continue; + if (nf != c->payload->early_dispatch_target()) continue; try_enqueue_early_dispatch_candidate(*c); } } diff --git a/src/a5/runtime/tensormap_and_ringbuffer/runtime/shared/runtime_init.cpp b/src/a5/runtime/tensormap_and_ringbuffer/runtime/shared/runtime_init.cpp index 0ac724d72f..782e50301d 100644 --- a/src/a5/runtime/tensormap_and_ringbuffer/runtime/shared/runtime_init.cpp +++ b/src/a5/runtime/tensormap_and_ringbuffer/runtime/shared/runtime_init.cpp @@ -350,6 +350,10 @@ OrchestratorLayout OrchestratorState::reserve_layout( const size_t seen_epoch_bytes = CHIP_ALIGN_UP(static_cast(task_window_sizes[r]) * sizeof(uint32_t), CHIP_ALIGN_SIZE); layout.off_fanin_seen_epoch[r] = arena.reserve(seen_epoch_bytes, CHIP_ALIGN_SIZE); + + const size_t wait_reach_bytes = + CHIP_ALIGN_UP(static_cast(task_window_sizes[r]) * sizeof(WaitReachEntry), CHIP_ALIGN_SIZE); + layout.off_wait_reach[r] = arena.reserve(wait_reach_bytes, CHIP_ALIGN_SIZE); } layout.off_scope_tasks = arena.reserve(static_cast(layout.scope_tasks_cap) * sizeof(uintptr_t), alignof(ChipTaskSlotState *)); @@ -416,6 +420,13 @@ bool OrchestratorState::init_data_from_layout( auto *seen_epoch = static_cast(arena.region_ptr(layout.off_fanin_seen_epoch[r])); memset(seen_epoch, 0, seen_epoch_bytes); orch->fanin_seen_epoch[r] = seen_epoch; + + const size_t wait_reach_bytes = CHIP_ALIGN_UP( + static_cast(layout.tensor_map.task_window_sizes[r]) * sizeof(WaitReachEntry), CHIP_ALIGN_SIZE + ); + auto *wait_reach = static_cast(arena.region_ptr(layout.off_wait_reach[r])); + memset(wait_reach, 0, wait_reach_bytes); + orch->wait_reach[r] = wait_reach; } if (!orch->tensor_map.init_data_from_layout(layout.tensor_map, arena)) { @@ -445,6 +456,10 @@ bool OrchestratorState::reset_for_reuse( orch->gm_heap_size = total_heap_size; orch->fatal = false; orch->inline_completed_tasks = 0; + // The wait_reach entries are not cleared here: every slot publishes its + // bitmap before any submit can read it as a producer, so entries left from + // a previous run are unreachable. + orch->submit_seq = 0; uint32_t next_epoch = orch->fanin_seen_current_epoch + 1; if (next_epoch == 0) { @@ -503,6 +518,7 @@ void OrchestratorState::wire_arena_pointers( for (int r = 0; r < CHIP_MAX_RING_DEPTH; r++) { orch->rings[r].fanin_pool.base = static_cast(arena.region_ptr(layout.off_fanin_pool[r])); orch->fanin_seen_epoch[r] = static_cast(arena.region_ptr(layout.off_fanin_seen_epoch[r])); + orch->wait_reach[r] = static_cast(arena.region_ptr(layout.off_wait_reach[r])); } orch->tensor_map.wire_arena_pointers(layout.tensor_map, arena); orch->scope_tasks = static_cast(arena.region_ptr(layout.off_scope_tasks)); @@ -516,6 +532,7 @@ void OrchestratorState::destroy() { for (int r = 0; r < CHIP_MAX_RING_DEPTH; r++) { orch->rings[r].fanin_pool.base = nullptr; orch->fanin_seen_epoch[r] = nullptr; + orch->wait_reach[r] = nullptr; } orch->scope_tasks = nullptr; orch->scope_begins = nullptr; diff --git a/tests/ut/cpp/a2a3/test_orchestrator_fanin.cpp b/tests/ut/cpp/a2a3/test_orchestrator_fanin.cpp index d1f5201fcd..fed24cc9ae 100644 --- a/tests/ut/cpp/a2a3/test_orchestrator_fanin.cpp +++ b/tests/ut/cpp/a2a3/test_orchestrator_fanin.cpp @@ -12,6 +12,7 @@ #include #include +#include #include #include @@ -238,6 +239,667 @@ TEST_F(OrchestratorFaninTest, AllCompletedFastPathReleasesWaitOnlyPin) { EXPECT_EQ(producer_slot.fanout_refcount.load(), rc_before + 1); } +// Bounded reachability bitmap reduction (issue #1376) +// --------------------------------------------------------------------------- + +// Helper: fetch a task's slot state from the SM handle. +static ChipTaskSlotState &slot_of(SharedMemoryHandle *sm, const TaskOutputTensors &t) { + return sm->header->rings[simpler::tmr::task_ring(t.task_id())].get_slot_state_by_task_id( + static_cast(simpler::tmr::task_local_id(t.task_id())) + ); +} + +// Diamond A -> B -> C plus direct A -> C, all conservative RETAIN edges: the +// direct A -> C WAIT is covered by the transitive path, so it demotes to +// RETAIN-only and drops out of the readiness count. +TEST_F(OrchestratorFaninTest, DiamondReducesRedundantWaitToRetainOnly) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + TaskId ac[] = {a.task_id(), b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + auto &b_slot = slot_of(sm_handle, b); + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_actual_count, 2); + EXPECT_EQ(payload->fanin_wait_count, 1); + bool saw_retain_only = false, saw_wait_retain = false; + for (int i = 0; i < payload->fanin_actual_count; i++) { + ChipTaskSlotState *p = payload->fanin_inline_edges[i].slot_state(); + DepFlags f = payload->fanin_inline_edges[i].flags(); + if (p == &a_slot) { + EXPECT_EQ(f, DEP_RETAIN); + saw_retain_only = true; + } else if (p == &b_slot) { + EXPECT_EQ(f, DEP_WAIT | DEP_RETAIN); + saw_wait_retain = true; + } + } + EXPECT_TRUE(saw_retain_only); + EXPECT_TRUE(saw_wait_retain); +} + +// Same diamond but the direct A -> C is ordering-only: the cleared edge becomes +// DEP_NONE and stays in storage (fanin_actual_count unchanged) so its +// submit-claim pin is still released by the !DEP_RETAIN paths. +TEST_F(OrchestratorFaninTest, DiamondDropsRedundantWaitOnlyEdge) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + TaskId ac[] = {b.task_id(), a.task_id()}; + DepFlags kinds[] = {DEP_WAIT | DEP_RETAIN, DEP_WAIT}; + CoreTaskArgs c_args; + c_args.set_dependencies_with_kinds(ac, kinds, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_actual_count, 2); + EXPECT_EQ(payload->fanin_wait_count, 1); + EXPECT_EQ(payload->fanin_inline_edges[1].slot_state(), &a_slot); + EXPECT_EQ(payload->fanin_inline_edges[1].flags(), DEP_NONE); +} + +// Early-dispatch accounting survives reduction. submit_dummy_task tasks do not +// allow early resolve, so the diamond's reduced A -> C edge points at an +// unflagged producer: C keeps the unit that producer held in fanin_wait_count, +// leaving early_dispatch_target() unreachable exactly as it was before +// reduction. +TEST_F(OrchestratorFaninTest, ReducedEdgeToUnflaggedProducerBlocksEarlyDispatch) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + TaskId ac[] = {b.task_id(), a.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_actual_count, 2); + EXPECT_EQ(payload->fanin_wait_count, 1); + EXPECT_EQ(payload->early_dispatch_blocked, 1); + EXPECT_EQ(payload->early_dispatch_target(), 2); +} + +// The same diamond with a producer that DOES allow early resolve: reduction +// costs the consumer nothing, because that producer would have propagated. +TEST_F(OrchestratorFaninTest, ReducedEdgeToFlaggedProducerLeavesEarlyDispatchOpen) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + slot_of(sm_handle, a).task_attrs.set_early_resolve(true); + + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + TaskId ac[] = {b.task_id(), a.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_wait_count, 1); + EXPECT_EQ(payload->early_dispatch_blocked, 0); + EXPECT_EQ(payload->early_dispatch_target(), 1); +} + +// A -> B -> C -> D plus direct A -> D (no A -> C, no B -> D): the covering path +// is longer than one hop, so only the transitive bitmap can prove A redundant. +TEST_F(OrchestratorFaninTest, Depth3ChainReducesBeyondOneHop) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + TaskId bc[] = {b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(bc, 1); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + TaskId ad[] = {a.task_id(), c.task_id()}; + CoreTaskArgs d_args; + d_args.set_dependencies(ad, 2); + TaskOutputTensors d = orch.submit_dummy_task(d_args); + ASSERT_TRUE(d.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + TaskPayload *payload = slot_of(sm_handle, d).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_wait_count, 1); + for (int i = 0; i < payload->fanin_actual_count; i++) { + if (payload->fanin_inline_edges[i].slot_state() == &a_slot) { + EXPECT_EQ(payload->fanin_inline_edges[i].flags(), DEP_RETAIN); + } + } +} + +// Two producers with no path between them: nothing to prove, both WAITs stay. +TEST_F(OrchestratorFaninTest, IndependentProducersAreNotReduced) { + orch.begin_scope(); + + CoreTaskArgs a_args, b_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(a.task_id().is_valid()); + ASSERT_TRUE(b.task_id().is_valid()); + + TaskId deps[] = {a.task_id(), b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(deps, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_actual_count, 2); + EXPECT_EQ(payload->fanin_wait_count, 2); + EXPECT_EQ(payload->fanin_inline_edges[0].flags(), DEP_WAIT | DEP_RETAIN); + EXPECT_EQ(payload->fanin_inline_edges[1].flags(), DEP_WAIT | DEP_RETAIN); +} + +// A -> B ordered by a WAIT-only (ordering-only) edge still puts A in R[B]: any +// WAIT edge carries ordering, so the covering path proves reachability and the +// direct A -> C is reduced. (The reduction's reachability semantics differ +// from a fanin-pointer walk here — WAIT-only cover is a valid witness.) +TEST_F(OrchestratorFaninTest, WaitOnlyCoveringEdgeStillProvesReachability) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + DepFlags wait_kind[] = {DEP_WAIT}; + CoreTaskArgs b_args; + b_args.set_dependencies_with_kinds(ab, wait_kind, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + TaskId ac[] = {a.task_id(), b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_wait_count, 1); + for (int i = 0; i < payload->fanin_actual_count; i++) { + if (payload->fanin_inline_edges[i].slot_state() == &a_slot) { + EXPECT_EQ(payload->fanin_inline_edges[i].flags(), DEP_RETAIN); + } + } +} + +// A producer farther back than WAIT_REACH_WINDOW submissions keeps its WAIT: +// the window cannot represent it, so the edge is retained conservatively. +TEST_F(OrchestratorFaninTest, WindowMissBeyondBlKeepsWait) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + // Fillers push seq(A) out of the window: seq distance to C exceeds 64. + CoreTaskArgs filler_args; + for (int i = 0; i < WAIT_REACH_WINDOW; i++) { + TaskOutputTensors f = orch.submit_dummy_task(filler_args); + ASSERT_TRUE(f.task_id().is_valid()); + } + + TaskId ac[] = {a.task_id(), b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_wait_count, 2); + for (int i = 0; i < payload->fanin_actual_count; i++) { + if (payload->fanin_inline_edges[i].slot_state() == &a_slot) { + EXPECT_EQ(payload->fanin_inline_edges[i].flags(), DEP_WAIT | DEP_RETAIN); + } + } +} + +// d(A -> C) == WAIT_REACH_WINDOW exactly: the direct bit is the last window +// bit, and the close covering producer B (d == 1, with A at bit 62 of R[B]) +// still shifts A's bit onto it, so the edge reduces. Proves the +// d == WAIT_REACH_WINDOW guard does not block valid boundary reduction. +TEST_F(OrchestratorFaninTest, BoundaryAtBlStillReducesViaCloseProducer) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + // 62 fillers: seq(A)=0, seq(B)=1, fillers 2..63, C=64 -> d(A->C)=64. + CoreTaskArgs filler_args; + for (int i = 0; i < WAIT_REACH_WINDOW - 2; i++) { + TaskOutputTensors f = orch.submit_dummy_task(filler_args); + ASSERT_TRUE(f.task_id().is_valid()); + } + + TaskId ac[] = {a.task_id(), b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_wait_count, 1); + for (int i = 0; i < payload->fanin_actual_count; i++) { + if (payload->fanin_inline_edges[i].slot_state() == &a_slot) { + EXPECT_EQ(payload->fanin_inline_edges[i].flags(), DEP_RETAIN); + } + } +} + +// Unsigned sequence subtraction preserves recent distances across uint64 wrap. +TEST_F(OrchestratorFaninTest, SequenceWrapPreservesRecentReachability) { + orch.submit_seq = std::numeric_limits::max() - 1; + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + TaskId ac[] = {a.task_id(), b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_wait_count, 1); + for (int i = 0; i < payload->fanin_actual_count; i++) { + if (payload->fanin_inline_edges[i].slot_state() == &a_slot) { + EXPECT_EQ(payload->fanin_inline_edges[i].flags(), DEP_RETAIN); + } + } +} + +// Nested scopes move tasks onto different rings; the global submission +// sequence makes cross-ring candidates participate in the same window. +TEST_F(OrchestratorFaninTest, CrossRingCandidateUsesGlobalSequence) { + orch.begin_scope(); // ring 0 + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + ASSERT_EQ(simpler::tmr::task_ring(a.task_id()), 0); + + orch.begin_scope(); // ring 1 + ASSERT_EQ(orch.current_ring_id(), 1); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + ASSERT_EQ(simpler::tmr::task_ring(b.task_id()), 1); + + TaskId ac[] = {a.task_id(), b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_wait_count, 1); + for (int i = 0; i < payload->fanin_actual_count; i++) { + if (payload->fanin_inline_edges[i].slot_state() == &a_slot) { + EXPECT_EQ(payload->fanin_inline_edges[i].flags(), DEP_RETAIN); + } + } +} + +// alloc_tensors' hidden task never enters submit_task_common; it must still +// publish an empty reachability bitmap so a consumer reading it as a creator +// producer sees a conservative entry, never a stale slot generation. +TEST_F(OrchestratorFaninTest, AllocTensorProducerPublishesEmptyReach) { + orch.begin_scope(); + + std::vector create_infos; + CoreTaskArgs alloc_args; + add_runtime_output_arg(alloc_args, create_infos, 4); + TaskOutputTensors alloc = orch.alloc_tensors(alloc_args); + ASSERT_TRUE(alloc.task_id().is_valid()); + + // Consumer depends on the alloc task explicitly; a second consumer chain + // through the first proves distances stay correct across the alloc entry. + TaskId deps[] = {alloc.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(deps, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + ASSERT_FALSE(orch.fatal); + + TaskId cd[] = {alloc.task_id(), b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(cd, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + ASSERT_FALSE(orch.fatal); + + auto &alloc_slot = slot_of(sm_handle, alloc); + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_wait_count, 1); + for (int i = 0; i < payload->fanin_actual_count; i++) { + if (payload->fanin_inline_edges[i].slot_state() == &alloc_slot) { + EXPECT_EQ(payload->fanin_inline_edges[i].flags(), DEP_RETAIN); + } + } +} + +// Runtime reuse leaves the side array uncleared; publication by the new slot +// owner replaces stale bits before a later consumer can read them. +TEST_F(OrchestratorFaninTest, RuntimeReuseOverwritesStaleSlotBitmap) { + orch.wait_reach[0][0].ancestors = std::numeric_limits::max(); + orch.wait_reach[0][0].seq = 1234; + + ASSERT_TRUE(sm_handle->init(sm_handle->sm_base, sm_handle->sm_size, CHIP_TASK_WINDOW_SIZE, 4096)); + uint64_t heap_sizes[CHIP_MAX_RING_DEPTH]; + uint64_t task_window_sizes[CHIP_MAX_RING_DEPTH]; + for (int r = 0; r < CHIP_MAX_RING_DEPTH; r++) { + heap_sizes[r] = 4096; + task_window_sizes[r] = CHIP_TASK_WINDOW_SIZE; + } + ASSERT_TRUE(orch.reset_for_reuse(orch_layout, sm_handle->sm_base, gm_heap.data(), heap_sizes, task_window_sizes)); + sched.reset_for_reuse(sched_layout, sm_handle->sm_base); + + EXPECT_EQ(orch.wait_reach[0][0].ancestors, std::numeric_limits::max()); + orch.begin_scope(); + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + ASSERT_EQ(simpler::tmr::task_local_id(a.task_id()), 0u); + EXPECT_EQ(orch.wait_reach[0][0].ancestors, 0); + EXPECT_EQ(orch.wait_reach[0][0].seq, 0); + + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + TaskId ac[] = {a.task_id(), b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + EXPECT_EQ(slot_of(sm_handle, c).payload->fanin_wait_count, 1); +} + +// Acceptance #5: candidate discovery order must not change the reduced graph. +// Same diamond with the dependency arrays in both orders. +TEST_F(OrchestratorFaninTest, DiscoveryOrderDoesNotChangeReduction) { + for (int reverse = 0; reverse < 2; reverse++) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + TaskId ac[] = {a.task_id(), b.task_id()}; + TaskId ca[] = {b.task_id(), a.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(reverse ? ca : ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + auto &b_slot = slot_of(sm_handle, b); + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_actual_count, 2); + EXPECT_EQ(payload->fanin_wait_count, 1); + for (int i = 0; i < payload->fanin_actual_count; i++) { + ChipTaskSlotState *p = payload->fanin_inline_edges[i].slot_state(); + if (p == &a_slot) { + EXPECT_EQ(payload->fanin_inline_edges[i].flags(), DEP_RETAIN); + } else if (p == &b_slot) { + EXPECT_EQ(payload->fanin_inline_edges[i].flags(), DEP_WAIT | DEP_RETAIN); + } + } + + orch.end_scope(); + } +} + +// The reduction covers spill-region candidates too — no inline cap. The +// redundant pair lands past CHIP_FANIN_INLINE_CAP so the cleared edge lives in +// the spill pool. +TEST_F(OrchestratorFaninTest, SpillRegionCandidatesAreReduced) { + orch.begin_scope(); + + constexpr int kOldProducers = CHIP_FANIN_INLINE_CAP + 1; + std::vector old_producers; + old_producers.reserve(kOldProducers); + for (int i = 0; i < kOldProducers; i++) { + CoreTaskArgs args; + old_producers.push_back(orch.submit_dummy_task(args)); + ASSERT_TRUE(old_producers.back().task_id().is_valid()); + } + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + std::vector deps; + std::vector kinds; + deps.reserve(kOldProducers + 2); + kinds.reserve(kOldProducers + 2); + for (auto &producer : old_producers) { + deps.push_back(producer.task_id()); + kinds.push_back(DEP_WAIT | DEP_RETAIN); + } + deps.push_back(b.task_id()); + kinds.push_back(DEP_WAIT | DEP_RETAIN); + deps.push_back(a.task_id()); // redundant, lands in the spill region + kinds.push_back(DEP_WAIT | DEP_RETAIN); + + CoreTaskArgs c_args; + c_args.set_dependencies_with_kinds(deps.data(), kinds.data(), static_cast(deps.size())); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_actual_count, kOldProducers + 2); + EXPECT_EQ(payload->fanin_wait_count, kOldProducers + 1); + ASSERT_NE(payload->fanin_spill_pool, nullptr); + bool found = false; + auto check = [&](ChipTaskSlotState *p, DepFlags f) { + if (p == &a_slot) { + EXPECT_EQ(f, DEP_RETAIN); + found = true; + } + }; + FaninPool &pool = *payload->fanin_spill_pool; + int32_t spill_count = payload->fanin_actual_count - CHIP_FANIN_INLINE_CAP; + ASSERT_GT(spill_count, 0); + for (int i = 0; i < spill_count; i++) { + FaninSpillEntry &e = pool.base[(payload->fanin_spill_start % pool.capacity + i) % pool.capacity]; + check(e.slot_state(), e.flags()); + } + EXPECT_TRUE(found); +} + +// A reduction-dropped (DEP_NONE) edge on the all-completed fast path releases +// its submit-claim pin exactly once — there, not again at on_task_release. +TEST_F(OrchestratorFaninTest, AllCompletedFastPathReleasesDroppedEdgePin) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + DepFlags wait_kind[] = {DEP_WAIT}; + CoreTaskArgs b_args; + b_args.set_dependencies_with_kinds(ab, wait_kind, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + auto &b_slot = slot_of(sm_handle, b); + // Both completed: the consumer takes the all-completed fast path. + a_slot.task_state.store(CHIP_TASK_COMPLETED, std::memory_order_release); + b_slot.task_state.store(CHIP_TASK_COMPLETED, std::memory_order_release); + int32_t a_rc_before = a_slot.fanout_refcount.load(); + + // Direct A -> C is ordering-only so the reduction drops it to DEP_NONE. + TaskId ac[] = {b.task_id(), a.task_id()}; + DepFlags kinds[] = {DEP_WAIT | DEP_RETAIN, DEP_WAIT}; + CoreTaskArgs c_args; + c_args.set_dependencies_with_kinds(ac, kinds, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_wait_count, 1); + // The dropped edge's pin was released by the fast path. + EXPECT_EQ(a_slot.fanout_refcount.load(), a_rc_before + 1); +} + +// A RETAIN-only survivor of the reduction keeps its producer pinned past +// wiring: only the consumer's on_task_release drops that pin. +TEST_F(OrchestratorFaninTest, ReducedRetainOnlyEdgeHoldsProducerUntilConsumerRelease) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + TaskId ac[] = {a.task_id(), b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + auto &c_slot = slot_of(sm_handle, c); + TaskPayload *payload = c_slot.payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_wait_count, 1); + + int32_t a_rc_before = a_slot.fanout_refcount.load(); + sched.on_task_release(c_slot); + // The RETAIN-only edge's pin is released by on_task_release. + EXPECT_EQ(a_slot.fanout_refcount.load(), a_rc_before + 1); +} + +// Zero-fanin and single-fanin tasks still publish their (mostly empty) +// bitmaps; a later chain consuming them must reduce normally. +TEST_F(OrchestratorFaninTest, ZeroFaninTaskPublishesEmptyBitmap) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); // zero fanin -> R=0 + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); // single fanin + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + TaskId ac[] = {a.task_id(), b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + ASSERT_FALSE(orch.fatal); + + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_wait_count, 1); +} + TEST_F(OrchestratorFaninTest, SubmitPathHeapDeadlockLogReportsRingAndRealHeapState) { std::vector create_infos; create_infos.reserve(8); diff --git a/tests/ut/cpp/a2a3/test_wiring.cpp b/tests/ut/cpp/a2a3/test_wiring.cpp index fad7e20587..79f2178ad7 100644 --- a/tests/ut/cpp/a2a3/test_wiring.cpp +++ b/tests/ut/cpp/a2a3/test_wiring.cpp @@ -129,6 +129,7 @@ TEST_F(WiringTest, NoFaninTaskBecomesReady) { init_slot(task_slot, CHIP_TASK_PENDING, 0, 1); payload.fanin_actual_count = 0; + payload.fanin_wait_count = 0; task_slot.payload = &payload; task_slot.task = &desc; @@ -164,6 +165,7 @@ TEST_F(WiringTest, WireTaskAllProducersEarlyFinished) { // Consumer task with 2 fanins init_slot(task_slot, CHIP_TASK_PENDING, 0, 1); payload.fanin_actual_count = 2; + payload.fanin_wait_count = 2; payload.fanin_inline_edges[0].set(&producer_slots[0], DEP_WAIT | DEP_RETAIN); payload.fanin_inline_edges[1].set(&producer_slots[1], DEP_WAIT | DEP_RETAIN); @@ -201,6 +203,7 @@ TEST_F(WiringTest, WireTaskProducersPendingTaskNotReady) { init_slot(task_slot, CHIP_TASK_PENDING, 0, 1); payload.fanin_actual_count = 2; + payload.fanin_wait_count = 2; payload.fanin_inline_edges[0].set(&producer_slots[0], DEP_WAIT | DEP_RETAIN); payload.fanin_inline_edges[1].set(&producer_slots[1], DEP_WAIT | DEP_RETAIN); task_slot.payload = &payload; @@ -263,6 +266,7 @@ TEST_F(WiringTest, WireTaskMixedProducerStates) { init_slot(task_slot, CHIP_TASK_PENDING, 0, 1); payload.fanin_actual_count = 3; + payload.fanin_wait_count = 3; for (int i = 0; i < 3; i++) { payload.fanin_inline_edges[i].set(&producers[i], DEP_WAIT | DEP_RETAIN); } @@ -306,6 +310,7 @@ TEST_F(WiringTest, WireTaskAllFlaggedPrecompletedSeedsDispatchFanin) { init_slot(task_slot, CHIP_TASK_PENDING, 0, 1); payload.fanin_actual_count = 2; + payload.fanin_wait_count = 2; payload.fanin_inline_edges[0].set(&producer_slots[0], DEP_WAIT | DEP_RETAIN); payload.fanin_inline_edges[1].set(&producer_slots[1], DEP_WAIT | DEP_RETAIN); task_slot.payload = &payload; @@ -334,6 +339,7 @@ TEST_F(WiringTest, WireTaskUnflaggedPrecompletedProducerDoesNotSeed) { init_slot(task_slot, CHIP_TASK_PENDING, 0, 1); payload.fanin_actual_count = 1; + payload.fanin_wait_count = 1; payload.fanin_inline_edges[0].set(&producer, DEP_WAIT | DEP_RETAIN); task_slot.payload = &payload; task_slot.task = &desc; @@ -359,6 +365,7 @@ TEST_F(WiringTest, WireTaskOneUnflaggedProducerDisqualifiesSeed) { init_slot(task_slot, CHIP_TASK_PENDING, 0, 1); payload.fanin_actual_count = 2; + payload.fanin_wait_count = 2; payload.fanin_inline_edges[0].set(&producers[0], DEP_WAIT | DEP_RETAIN); payload.fanin_inline_edges[1].set(&producers[1], DEP_WAIT | DEP_RETAIN); task_slot.payload = &payload; @@ -369,6 +376,73 @@ TEST_F(WiringTest, WireTaskOneUnflaggedProducerDisqualifiesSeed) { EXPECT_EQ(payload.dispatch_fanin.load(), 0); // disqualified: seed stays 0 } +// Post-reduction twin of EarlyDispatchBlockedByUnflaggedProducer: reduction +// demoted the unflagged producer's edge out of fanin_wait_count, so +// early_dispatch_blocked carries the unit it used to hold and the flagged +// producer alone still cannot reach early_dispatch_target(). +TEST_F(WiringTest, ReducedEdgeToUnflaggedProducerStillBlocksEarlyDispatch) { + alignas(64) ChipTaskSlotState task_slot; + alignas(64) ChipTaskSlotState p_flagged, q_unflagged; + alignas(64) TaskPayload payload; + memset(&payload, 0, sizeof(payload)); + TaskDescriptor desc{}; + + init_slot(p_flagged, CHIP_TASK_PENDING, 1, 1); + p_flagged.task_attrs.set_early_resolve(true); + init_slot(q_unflagged, CHIP_TASK_PENDING, 1, 1); + q_unflagged.task_attrs.set_early_resolve(false); + + init_slot(task_slot, CHIP_TASK_PENDING, 0, 1); + payload.fanin_actual_count = 2; + payload.fanin_wait_count = 1; // q's WAIT was cleared by reduction + payload.early_dispatch_blocked = 1; // ... and q does not allow early resolve + payload.fanin_inline_edges[0].set(&p_flagged, DEP_WAIT | DEP_RETAIN); + payload.fanin_inline_edges[1].set(&q_unflagged, DEP_NONE); + task_slot.payload = &payload; + task_slot.task = &desc; + + wire_fanin(task_slot, 1); + sched.record_published_blocks(p_flagged, p_flagged.logical_block_num); + sched.propagate_dispatch_fanin(p_flagged); + + EXPECT_EQ(payload.early_dispatch_target(), 2); + EXPECT_EQ(payload.dispatch_fanin.load(), 1); // p alone can never reach 2 + EXPECT_EQ(payload.early_dispatch_state.load(), EARLY_DISPATCH_NONE); +} + +// Control for the pair above: the same reduced shape whose reduced-away producer +// IS flagged leaves the target at fanin_wait_count, so reduction does not cost +// this consumer its early-dispatch candidacy. +TEST_F(WiringTest, ReducedEdgeToFlaggedProducerKeepsEarlyDispatch) { + alignas(64) ChipTaskSlotState task_slot; + alignas(64) ChipTaskSlotState p_flagged, q_flagged; + alignas(64) TaskPayload payload; + memset(&payload, 0, sizeof(payload)); + TaskDescriptor desc{}; + + init_slot(p_flagged, CHIP_TASK_PENDING, 1, 1); + p_flagged.task_attrs.set_early_resolve(true); + init_slot(q_flagged, CHIP_TASK_PENDING, 1, 1); + q_flagged.task_attrs.set_early_resolve(true); + + init_slot(task_slot, CHIP_TASK_PENDING, 0, 1); + payload.fanin_actual_count = 2; + payload.fanin_wait_count = 1; + payload.early_dispatch_blocked = 0; // reduction saw no unflagged producer + payload.fanin_inline_edges[0].set(&p_flagged, DEP_WAIT | DEP_RETAIN); + payload.fanin_inline_edges[1].set(&q_flagged, DEP_NONE); + task_slot.payload = &payload; + task_slot.task = &desc; + + wire_fanin(task_slot, 1); + sched.record_published_blocks(p_flagged, p_flagged.logical_block_num); + sched.propagate_dispatch_fanin(p_flagged); + + EXPECT_EQ(payload.early_dispatch_target(), 1); + EXPECT_EQ(payload.dispatch_fanin.load(), 1); + EXPECT_EQ(payload.early_dispatch_state.load(), EARLY_DISPATCH_STAGING); +} + TEST_F(WiringTest, EarlyDispatchWaitsForAllProducerBlocksPublished) { // A flagged, still-pending producer seeds nothing at wiring (not // pre-completed); only publishing every logical block bumps the consumer @@ -385,6 +459,7 @@ TEST_F(WiringTest, EarlyDispatchWaitsForAllProducerBlocksPublished) { init_slot(task_slot, CHIP_TASK_PENDING, 0, 1); payload.fanin_actual_count = 1; + payload.fanin_wait_count = 1; payload.fanin_inline_edges[0].set(&producer, DEP_WAIT | DEP_RETAIN); task_slot.payload = &payload; task_slot.task = &desc; @@ -516,6 +591,7 @@ TEST_F(WiringTest, LateWiredFullyPublishedProducerStillSeedsEarlyDispatch) { init_slot(consumer, CHIP_TASK_PENDING, 0, 1); consumer_payload.fanin_actual_count = 1; + consumer_payload.fanin_wait_count = 1; consumer_payload.fanin_inline_edges[0].set(&producer, DEP_WAIT | DEP_RETAIN); consumer.payload = &consumer_payload; consumer.task = &consumer_desc; @@ -544,6 +620,7 @@ TEST_F(WiringTest, WiringSeedEnqueuesAfterConcurrentPropagation) { init_slot(consumer, CHIP_TASK_PENDING, 0, 1); consumer_payload.fanin_actual_count = 3; + consumer_payload.fanin_wait_count = 3; for (int i = 0; i < 3; i++) consumer_payload.fanin_inline_edges[i].set(&producers[i], DEP_WAIT | DEP_RETAIN); consumer.payload = &consumer_payload; @@ -1006,6 +1083,7 @@ TEST_F(WiringTest, EarlyDispatchBlockedByUnflaggedProducer) { init_slot(task_slot, CHIP_TASK_PENDING, 0, 1); payload.fanin_actual_count = 2; + payload.fanin_wait_count = 2; payload.fanin_inline_edges[0].set(&p_flagged, DEP_WAIT | DEP_RETAIN); payload.fanin_inline_edges[1].set(&q_unflagged, DEP_WAIT | DEP_RETAIN); task_slot.payload = &payload; @@ -1036,6 +1114,7 @@ TEST_F(WiringTest, UnflaggedProducerDoesNotPropagate) { init_slot(consumer, CHIP_TASK_PENDING, 1, 1); consumer.payload = &cons_payload; cons_payload.fanin_actual_count = 1; + cons_payload.fanin_wait_count = 1; DepListEntry dep{}; dep.slot_state = &consumer; @@ -1067,6 +1146,7 @@ TEST_F(WiringTest, FlaggedPrecompletedCreatorTransparentToEarlyDispatch) { init_slot(task_slot, CHIP_TASK_PENDING, 0, 1); payload.fanin_actual_count = 2; + payload.fanin_wait_count = 2; payload.fanin_inline_edges[0].set(&creator, DEP_WAIT | DEP_RETAIN); payload.fanin_inline_edges[1].set(&compute, DEP_WAIT | DEP_RETAIN); task_slot.payload = &payload; @@ -1148,6 +1228,7 @@ TEST_F(WiringTest, OnTaskReleaseReleasesProducers) { init_slot(task_slot, CHIP_TASK_COMPLETED, 3, 1); payload.fanin_actual_count = 2; + payload.fanin_wait_count = 2; payload.fanin_inline_edges[0].set(&producers[0], DEP_WAIT | DEP_RETAIN); payload.fanin_inline_edges[1].set(&producers[1], DEP_WAIT | DEP_RETAIN); // Need a valid fanin_spill_pool even though we don't spill @@ -1191,6 +1272,7 @@ TEST_F(WiringTest, OrderingOnlyReleasedAtWiringRetentionHeldUntilRelease) { init_slot(task_slot, CHIP_TASK_PENDING, 0, 1); payload.fanin_actual_count = 2; + payload.fanin_wait_count = 2; payload.fanin_inline_edges[0].set(&wait_producer, DEP_WAIT); payload.fanin_inline_edges[1].set(&retain_producer, DEP_WAIT | DEP_RETAIN); FaninPool dummy_pool{}; @@ -1244,6 +1326,8 @@ TEST_F(WiringTest, ReleaseHonorsRetainFlagInSpillRegion) { e->set(&spill_retain, DEP_WAIT | DEP_RETAIN); payload.fanin_actual_count = CHIP_FANIN_INLINE_CAP + 1; + + payload.fanin_wait_count = CHIP_FANIN_INLINE_CAP + 1; payload.fanin_spill_start = spill_start; payload.fanin_spill_pool = &spill_pool; task_slot.payload = &payload; @@ -1330,6 +1414,7 @@ TEST_F(WiringTest, NoEdgePublishRecordsDepPoolMark) { init_slot(task_slot, CHIP_TASK_PENDING, 0, 1); payload.fanin_actual_count = 0; + payload.fanin_wait_count = 0; task_slot.payload = &payload; task_slot.task = &desc; diff --git a/tests/ut/cpp/a5/test_orchestrator_fanin.cpp b/tests/ut/cpp/a5/test_orchestrator_fanin.cpp index 0616252269..fa31439a88 100644 --- a/tests/ut/cpp/a5/test_orchestrator_fanin.cpp +++ b/tests/ut/cpp/a5/test_orchestrator_fanin.cpp @@ -12,6 +12,7 @@ #include #include +#include #include #include @@ -250,6 +251,667 @@ TEST_F(OrchestratorFaninTest, AllCompletedFastPathReleasesWaitOnlyPin) { EXPECT_EQ(producer_slot.fanout_refcount.load(), rc_before + 1); } +// Bounded reachability bitmap reduction (issue #1376) +// --------------------------------------------------------------------------- + +// Helper: fetch a task's slot state from the SM handle. +static ChipTaskSlotState &slot_of(SharedMemoryHandle *sm, const TaskOutputTensors &t) { + return sm->header->rings[simpler::tmr::task_ring(t.task_id())].get_slot_state_by_task_id( + static_cast(simpler::tmr::task_local_id(t.task_id())) + ); +} + +// Diamond A -> B -> C plus direct A -> C, all conservative RETAIN edges: the +// direct A -> C WAIT is covered by the transitive path, so it demotes to +// RETAIN-only and drops out of the readiness count. +TEST_F(OrchestratorFaninTest, DiamondReducesRedundantWaitToRetainOnly) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + TaskId ac[] = {a.task_id(), b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + auto &b_slot = slot_of(sm_handle, b); + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_actual_count, 2); + EXPECT_EQ(payload->fanin_wait_count, 1); + bool saw_retain_only = false, saw_wait_retain = false; + for (int i = 0; i < payload->fanin_actual_count; i++) { + ChipTaskSlotState *p = payload->fanin_inline_edges[i].slot_state(); + DepFlags f = payload->fanin_inline_edges[i].flags(); + if (p == &a_slot) { + EXPECT_EQ(f, DEP_RETAIN); + saw_retain_only = true; + } else if (p == &b_slot) { + EXPECT_EQ(f, DEP_WAIT | DEP_RETAIN); + saw_wait_retain = true; + } + } + EXPECT_TRUE(saw_retain_only); + EXPECT_TRUE(saw_wait_retain); +} + +// Same diamond but the direct A -> C is ordering-only: the cleared edge becomes +// DEP_NONE and stays in storage (fanin_actual_count unchanged) so its +// submit-claim pin is still released by the !DEP_RETAIN paths. +TEST_F(OrchestratorFaninTest, DiamondDropsRedundantWaitOnlyEdge) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + TaskId ac[] = {b.task_id(), a.task_id()}; + DepFlags kinds[] = {DEP_WAIT | DEP_RETAIN, DEP_WAIT}; + CoreTaskArgs c_args; + c_args.set_dependencies_with_kinds(ac, kinds, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_actual_count, 2); + EXPECT_EQ(payload->fanin_wait_count, 1); + EXPECT_EQ(payload->fanin_inline_edges[1].slot_state(), &a_slot); + EXPECT_EQ(payload->fanin_inline_edges[1].flags(), DEP_NONE); +} + +// Early-dispatch accounting survives reduction. submit_dummy_task tasks do not +// allow early resolve, so the diamond's reduced A -> C edge points at an +// unflagged producer: C keeps the unit that producer held in fanin_wait_count, +// leaving early_dispatch_target() unreachable exactly as it was before +// reduction. +TEST_F(OrchestratorFaninTest, ReducedEdgeToUnflaggedProducerBlocksEarlyDispatch) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + TaskId ac[] = {b.task_id(), a.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_actual_count, 2); + EXPECT_EQ(payload->fanin_wait_count, 1); + EXPECT_EQ(payload->early_dispatch_blocked, 1); + EXPECT_EQ(payload->early_dispatch_target(), 2); +} + +// The same diamond with a producer that DOES allow early resolve: reduction +// costs the consumer nothing, because that producer would have propagated. +TEST_F(OrchestratorFaninTest, ReducedEdgeToFlaggedProducerLeavesEarlyDispatchOpen) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + slot_of(sm_handle, a).task_attrs.set_early_resolve(true); + + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + TaskId ac[] = {b.task_id(), a.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_wait_count, 1); + EXPECT_EQ(payload->early_dispatch_blocked, 0); + EXPECT_EQ(payload->early_dispatch_target(), 1); +} + +// A -> B -> C -> D plus direct A -> D (no A -> C, no B -> D): the covering path +// is longer than one hop, so only the transitive bitmap can prove A redundant. +TEST_F(OrchestratorFaninTest, Depth3ChainReducesBeyondOneHop) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + TaskId bc[] = {b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(bc, 1); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + TaskId ad[] = {a.task_id(), c.task_id()}; + CoreTaskArgs d_args; + d_args.set_dependencies(ad, 2); + TaskOutputTensors d = orch.submit_dummy_task(d_args); + ASSERT_TRUE(d.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + TaskPayload *payload = slot_of(sm_handle, d).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_wait_count, 1); + for (int i = 0; i < payload->fanin_actual_count; i++) { + if (payload->fanin_inline_edges[i].slot_state() == &a_slot) { + EXPECT_EQ(payload->fanin_inline_edges[i].flags(), DEP_RETAIN); + } + } +} + +// Two producers with no path between them: nothing to prove, both WAITs stay. +TEST_F(OrchestratorFaninTest, IndependentProducersAreNotReduced) { + orch.begin_scope(); + + CoreTaskArgs a_args, b_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(a.task_id().is_valid()); + ASSERT_TRUE(b.task_id().is_valid()); + + TaskId deps[] = {a.task_id(), b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(deps, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_actual_count, 2); + EXPECT_EQ(payload->fanin_wait_count, 2); + EXPECT_EQ(payload->fanin_inline_edges[0].flags(), DEP_WAIT | DEP_RETAIN); + EXPECT_EQ(payload->fanin_inline_edges[1].flags(), DEP_WAIT | DEP_RETAIN); +} + +// A -> B ordered by a WAIT-only (ordering-only) edge still puts A in R[B]: any +// WAIT edge carries ordering, so the covering path proves reachability and the +// direct A -> C is reduced. (The reduction's reachability semantics differ +// from a fanin-pointer walk here — WAIT-only cover is a valid witness.) +TEST_F(OrchestratorFaninTest, WaitOnlyCoveringEdgeStillProvesReachability) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + DepFlags wait_kind[] = {DEP_WAIT}; + CoreTaskArgs b_args; + b_args.set_dependencies_with_kinds(ab, wait_kind, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + TaskId ac[] = {a.task_id(), b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_wait_count, 1); + for (int i = 0; i < payload->fanin_actual_count; i++) { + if (payload->fanin_inline_edges[i].slot_state() == &a_slot) { + EXPECT_EQ(payload->fanin_inline_edges[i].flags(), DEP_RETAIN); + } + } +} + +// A producer farther back than WAIT_REACH_WINDOW submissions keeps its WAIT: +// the window cannot represent it, so the edge is retained conservatively. +TEST_F(OrchestratorFaninTest, WindowMissBeyondBlKeepsWait) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + // Fillers push seq(A) out of the window: seq distance to C exceeds 64. + CoreTaskArgs filler_args; + for (int i = 0; i < WAIT_REACH_WINDOW; i++) { + TaskOutputTensors f = orch.submit_dummy_task(filler_args); + ASSERT_TRUE(f.task_id().is_valid()); + } + + TaskId ac[] = {a.task_id(), b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_wait_count, 2); + for (int i = 0; i < payload->fanin_actual_count; i++) { + if (payload->fanin_inline_edges[i].slot_state() == &a_slot) { + EXPECT_EQ(payload->fanin_inline_edges[i].flags(), DEP_WAIT | DEP_RETAIN); + } + } +} + +// d(A -> C) == WAIT_REACH_WINDOW exactly: the direct bit is the last window +// bit, and the close covering producer B (d == 1, with A at bit 62 of R[B]) +// still shifts A's bit onto it, so the edge reduces. Proves the +// d == WAIT_REACH_WINDOW guard does not block valid boundary reduction. +TEST_F(OrchestratorFaninTest, BoundaryAtBlStillReducesViaCloseProducer) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + // 62 fillers: seq(A)=0, seq(B)=1, fillers 2..63, C=64 -> d(A->C)=64. + CoreTaskArgs filler_args; + for (int i = 0; i < WAIT_REACH_WINDOW - 2; i++) { + TaskOutputTensors f = orch.submit_dummy_task(filler_args); + ASSERT_TRUE(f.task_id().is_valid()); + } + + TaskId ac[] = {a.task_id(), b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_wait_count, 1); + for (int i = 0; i < payload->fanin_actual_count; i++) { + if (payload->fanin_inline_edges[i].slot_state() == &a_slot) { + EXPECT_EQ(payload->fanin_inline_edges[i].flags(), DEP_RETAIN); + } + } +} + +// Unsigned sequence subtraction preserves recent distances across uint64 wrap. +TEST_F(OrchestratorFaninTest, SequenceWrapPreservesRecentReachability) { + orch.submit_seq = std::numeric_limits::max() - 1; + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + TaskId ac[] = {a.task_id(), b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_wait_count, 1); + for (int i = 0; i < payload->fanin_actual_count; i++) { + if (payload->fanin_inline_edges[i].slot_state() == &a_slot) { + EXPECT_EQ(payload->fanin_inline_edges[i].flags(), DEP_RETAIN); + } + } +} + +// Nested scopes move tasks onto different rings; the global submission +// sequence makes cross-ring candidates participate in the same window. +TEST_F(OrchestratorFaninTest, CrossRingCandidateUsesGlobalSequence) { + orch.begin_scope(); // ring 0 + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + ASSERT_EQ(simpler::tmr::task_ring(a.task_id()), 0); + + orch.begin_scope(); // ring 1 + ASSERT_EQ(orch.current_ring_id(), 1); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + ASSERT_EQ(simpler::tmr::task_ring(b.task_id()), 1); + + TaskId ac[] = {a.task_id(), b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_wait_count, 1); + for (int i = 0; i < payload->fanin_actual_count; i++) { + if (payload->fanin_inline_edges[i].slot_state() == &a_slot) { + EXPECT_EQ(payload->fanin_inline_edges[i].flags(), DEP_RETAIN); + } + } +} + +// alloc_tensors' hidden task never enters submit_task_common; it must still +// publish an empty reachability bitmap so a consumer reading it as a creator +// producer sees a conservative entry, never a stale slot generation. +TEST_F(OrchestratorFaninTest, AllocTensorProducerPublishesEmptyReach) { + orch.begin_scope(); + + std::vector create_infos; + CoreTaskArgs alloc_args; + add_runtime_output_arg(alloc_args, create_infos, 4); + TaskOutputTensors alloc = orch.alloc_tensors(alloc_args); + ASSERT_TRUE(alloc.task_id().is_valid()); + + // Consumer depends on the alloc task explicitly; a second consumer chain + // through the first proves distances stay correct across the alloc entry. + TaskId deps[] = {alloc.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(deps, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + ASSERT_FALSE(orch.fatal); + + TaskId cd[] = {alloc.task_id(), b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(cd, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + ASSERT_FALSE(orch.fatal); + + auto &alloc_slot = slot_of(sm_handle, alloc); + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_wait_count, 1); + for (int i = 0; i < payload->fanin_actual_count; i++) { + if (payload->fanin_inline_edges[i].slot_state() == &alloc_slot) { + EXPECT_EQ(payload->fanin_inline_edges[i].flags(), DEP_RETAIN); + } + } +} + +// Runtime reuse leaves the side array uncleared; publication by the new slot +// owner replaces stale bits before a later consumer can read them. +TEST_F(OrchestratorFaninTest, RuntimeReuseOverwritesStaleSlotBitmap) { + orch.wait_reach[0][0].ancestors = std::numeric_limits::max(); + orch.wait_reach[0][0].seq = 1234; + + ASSERT_TRUE(sm_handle->init(sm_handle->sm_base, sm_handle->sm_size, CHIP_TASK_WINDOW_SIZE, 4096)); + uint64_t heap_sizes[CHIP_MAX_RING_DEPTH]; + uint64_t task_window_sizes[CHIP_MAX_RING_DEPTH]; + for (int r = 0; r < CHIP_MAX_RING_DEPTH; r++) { + heap_sizes[r] = 4096; + task_window_sizes[r] = CHIP_TASK_WINDOW_SIZE; + } + ASSERT_TRUE(orch.reset_for_reuse(orch_layout, sm_handle->sm_base, gm_heap.data(), heap_sizes, task_window_sizes)); + sched.reset_for_reuse(sched_layout, sm_handle->sm_base); + + EXPECT_EQ(orch.wait_reach[0][0].ancestors, std::numeric_limits::max()); + orch.begin_scope(); + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + ASSERT_EQ(simpler::tmr::task_local_id(a.task_id()), 0u); + EXPECT_EQ(orch.wait_reach[0][0].ancestors, 0); + EXPECT_EQ(orch.wait_reach[0][0].seq, 0); + + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + TaskId ac[] = {a.task_id(), b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + EXPECT_EQ(slot_of(sm_handle, c).payload->fanin_wait_count, 1); +} + +// Acceptance #5: candidate discovery order must not change the reduced graph. +// Same diamond with the dependency arrays in both orders. +TEST_F(OrchestratorFaninTest, DiscoveryOrderDoesNotChangeReduction) { + for (int reverse = 0; reverse < 2; reverse++) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + TaskId ac[] = {a.task_id(), b.task_id()}; + TaskId ca[] = {b.task_id(), a.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(reverse ? ca : ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + auto &b_slot = slot_of(sm_handle, b); + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_actual_count, 2); + EXPECT_EQ(payload->fanin_wait_count, 1); + for (int i = 0; i < payload->fanin_actual_count; i++) { + ChipTaskSlotState *p = payload->fanin_inline_edges[i].slot_state(); + if (p == &a_slot) { + EXPECT_EQ(payload->fanin_inline_edges[i].flags(), DEP_RETAIN); + } else if (p == &b_slot) { + EXPECT_EQ(payload->fanin_inline_edges[i].flags(), DEP_WAIT | DEP_RETAIN); + } + } + + orch.end_scope(); + } +} + +// The reduction covers spill-region candidates too — no inline cap. The +// redundant pair lands past CHIP_FANIN_INLINE_CAP so the cleared edge lives in +// the spill pool. +TEST_F(OrchestratorFaninTest, SpillRegionCandidatesAreReduced) { + orch.begin_scope(); + + constexpr int kOldProducers = CHIP_FANIN_INLINE_CAP + 1; + std::vector old_producers; + old_producers.reserve(kOldProducers); + for (int i = 0; i < kOldProducers; i++) { + CoreTaskArgs args; + old_producers.push_back(orch.submit_dummy_task(args)); + ASSERT_TRUE(old_producers.back().task_id().is_valid()); + } + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + std::vector deps; + std::vector kinds; + deps.reserve(kOldProducers + 2); + kinds.reserve(kOldProducers + 2); + for (auto &producer : old_producers) { + deps.push_back(producer.task_id()); + kinds.push_back(DEP_WAIT | DEP_RETAIN); + } + deps.push_back(b.task_id()); + kinds.push_back(DEP_WAIT | DEP_RETAIN); + deps.push_back(a.task_id()); // redundant, lands in the spill region + kinds.push_back(DEP_WAIT | DEP_RETAIN); + + CoreTaskArgs c_args; + c_args.set_dependencies_with_kinds(deps.data(), kinds.data(), static_cast(deps.size())); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_actual_count, kOldProducers + 2); + EXPECT_EQ(payload->fanin_wait_count, kOldProducers + 1); + ASSERT_NE(payload->fanin_spill_pool, nullptr); + bool found = false; + auto check = [&](ChipTaskSlotState *p, DepFlags f) { + if (p == &a_slot) { + EXPECT_EQ(f, DEP_RETAIN); + found = true; + } + }; + FaninPool &pool = *payload->fanin_spill_pool; + int32_t spill_count = payload->fanin_actual_count - CHIP_FANIN_INLINE_CAP; + ASSERT_GT(spill_count, 0); + for (int i = 0; i < spill_count; i++) { + FaninSpillEntry &e = pool.base[(payload->fanin_spill_start % pool.capacity + i) % pool.capacity]; + check(e.slot_state(), e.flags()); + } + EXPECT_TRUE(found); +} + +// A reduction-dropped (DEP_NONE) edge on the all-completed fast path releases +// its submit-claim pin exactly once — there, not again at on_task_release. +TEST_F(OrchestratorFaninTest, AllCompletedFastPathReleasesDroppedEdgePin) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + DepFlags wait_kind[] = {DEP_WAIT}; + CoreTaskArgs b_args; + b_args.set_dependencies_with_kinds(ab, wait_kind, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + auto &b_slot = slot_of(sm_handle, b); + // Both completed: the consumer takes the all-completed fast path. + a_slot.task_state.store(CHIP_TASK_COMPLETED, std::memory_order_release); + b_slot.task_state.store(CHIP_TASK_COMPLETED, std::memory_order_release); + int32_t a_rc_before = a_slot.fanout_refcount.load(); + + // Direct A -> C is ordering-only so the reduction drops it to DEP_NONE. + TaskId ac[] = {b.task_id(), a.task_id()}; + DepFlags kinds[] = {DEP_WAIT | DEP_RETAIN, DEP_WAIT}; + CoreTaskArgs c_args; + c_args.set_dependencies_with_kinds(ac, kinds, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_wait_count, 1); + // The dropped edge's pin was released by the fast path. + EXPECT_EQ(a_slot.fanout_refcount.load(), a_rc_before + 1); +} + +// A RETAIN-only survivor of the reduction keeps its producer pinned past +// wiring: only the consumer's on_task_release drops that pin. +TEST_F(OrchestratorFaninTest, ReducedRetainOnlyEdgeHoldsProducerUntilConsumerRelease) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + TaskId ac[] = {a.task_id(), b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + + auto &a_slot = slot_of(sm_handle, a); + auto &c_slot = slot_of(sm_handle, c); + TaskPayload *payload = c_slot.payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_wait_count, 1); + + int32_t a_rc_before = a_slot.fanout_refcount.load(); + sched.on_task_release(c_slot); + // The RETAIN-only edge's pin is released by on_task_release. + EXPECT_EQ(a_slot.fanout_refcount.load(), a_rc_before + 1); +} + +// Zero-fanin and single-fanin tasks still publish their (mostly empty) +// bitmaps; a later chain consuming them must reduce normally. +TEST_F(OrchestratorFaninTest, ZeroFaninTaskPublishesEmptyBitmap) { + orch.begin_scope(); + + CoreTaskArgs a_args; + TaskOutputTensors a = orch.submit_dummy_task(a_args); // zero fanin -> R=0 + ASSERT_TRUE(a.task_id().is_valid()); + TaskId ab[] = {a.task_id()}; + CoreTaskArgs b_args; + b_args.set_dependencies(ab, 1); // single fanin + TaskOutputTensors b = orch.submit_dummy_task(b_args); + ASSERT_TRUE(b.task_id().is_valid()); + + TaskId ac[] = {a.task_id(), b.task_id()}; + CoreTaskArgs c_args; + c_args.set_dependencies(ac, 2); + TaskOutputTensors c = orch.submit_dummy_task(c_args); + ASSERT_TRUE(c.task_id().is_valid()); + ASSERT_FALSE(orch.fatal); + + TaskPayload *payload = slot_of(sm_handle, c).payload; + ASSERT_NE(payload, nullptr); + EXPECT_EQ(payload->fanin_wait_count, 1); +} + TEST_F(OrchestratorFaninTest, SubmitPathHeapDeadlockLogReportsRingAndRealHeapState) { std::vector create_infos; create_infos.reserve(8); diff --git a/tests/ut/cpp/a5/test_wiring.cpp b/tests/ut/cpp/a5/test_wiring.cpp index ad48cbae1a..1846b2b064 100644 --- a/tests/ut/cpp/a5/test_wiring.cpp +++ b/tests/ut/cpp/a5/test_wiring.cpp @@ -146,6 +146,7 @@ TEST_F(WiringTest, NoFaninTaskBecomesReady) { init_slot(task_slot, CHIP_TASK_PENDING, 0, 1); payload.fanin_actual_count = 0; + payload.fanin_wait_count = 0; task_slot.payload = &payload; task_slot.task = &desc; @@ -181,6 +182,7 @@ TEST_F(WiringTest, WireTaskAllProducersEarlyFinished) { // Consumer task with 2 fanins init_slot(task_slot, CHIP_TASK_PENDING, 0, 1); payload.fanin_actual_count = 2; + payload.fanin_wait_count = 2; payload.fanin_inline_edges[0].set(&producer_slots[0], DEP_WAIT | DEP_RETAIN); payload.fanin_inline_edges[1].set(&producer_slots[1], DEP_WAIT | DEP_RETAIN); @@ -218,6 +220,7 @@ TEST_F(WiringTest, WireTaskProducersPendingTaskNotReady) { init_slot(task_slot, CHIP_TASK_PENDING, 0, 1); payload.fanin_actual_count = 2; + payload.fanin_wait_count = 2; payload.fanin_inline_edges[0].set(&producer_slots[0], DEP_WAIT | DEP_RETAIN); payload.fanin_inline_edges[1].set(&producer_slots[1], DEP_WAIT | DEP_RETAIN); task_slot.payload = &payload; @@ -260,6 +263,7 @@ TEST_F(WiringTest, WireTaskMixedProducerStates) { init_slot(task_slot, CHIP_TASK_PENDING, 0, 1); payload.fanin_actual_count = 3; + payload.fanin_wait_count = 3; for (int i = 0; i < 3; i++) { payload.fanin_inline_edges[i].set(&producers[i], DEP_WAIT | DEP_RETAIN); } @@ -468,6 +472,7 @@ TEST_F(WiringTest, OnTaskReleaseReleasesProducers) { init_slot(task_slot, CHIP_TASK_COMPLETED, 3, 1); payload.fanin_actual_count = 2; + payload.fanin_wait_count = 2; payload.fanin_inline_edges[0].set(&producers[0], DEP_WAIT | DEP_RETAIN); payload.fanin_inline_edges[1].set(&producers[1], DEP_WAIT | DEP_RETAIN); // Need a valid fanin_spill_pool even though we don't spill @@ -511,6 +516,7 @@ TEST_F(WiringTest, OrderingOnlyReleasedAtWiringRetentionHeldUntilRelease) { init_slot(task_slot, CHIP_TASK_PENDING, 0, 1); payload.fanin_actual_count = 2; + payload.fanin_wait_count = 2; payload.fanin_inline_edges[0].set(&wait_producer, DEP_WAIT); payload.fanin_inline_edges[1].set(&retain_producer, DEP_WAIT | DEP_RETAIN); FaninPool dummy_pool{}; @@ -564,6 +570,8 @@ TEST_F(WiringTest, ReleaseHonorsRetainFlagInSpillRegion) { e->set(&spill_retain, DEP_WAIT | DEP_RETAIN); payload.fanin_actual_count = CHIP_FANIN_INLINE_CAP + 1; + + payload.fanin_wait_count = CHIP_FANIN_INLINE_CAP + 1; payload.fanin_spill_start = spill_start; payload.fanin_spill_pool = &spill_pool; task_slot.payload = &payload; @@ -866,6 +874,7 @@ TEST_F(WiringTest, FaninPoolReclaimsOnceWithheldProgressIsPublished) { auto &payload = ring->get_payload_by_task_id(0); payload.fanin_actual_count = CHIP_FANIN_INLINE_CAP + 1; + payload.fanin_wait_count = CHIP_FANIN_INLINE_CAP + 1; payload.fanin_spill_start = 1; payload.fanin_spill_pool = &pool; @@ -943,6 +952,7 @@ TEST_F(WiringTest, NoEdgePublishRecordsDepPoolMark) { init_slot(task_slot, CHIP_TASK_PENDING, 0, 1); payload.fanin_actual_count = 0; + payload.fanin_wait_count = 0; task_slot.payload = &payload; task_slot.task = &desc; diff --git a/tests/ut/py/test_wait_reduction_sim.py b/tests/ut/py/test_wait_reduction_sim.py new file mode 100644 index 0000000000..f92a397e2e --- /dev/null +++ b/tests/ut/py/test_wait_reduction_sim.py @@ -0,0 +1,101 @@ +#!/usr/bin/env python3 +# Copyright (c) PyPTO Contributors. +# This program is free software, you can redistribute it and/or modify it under the terms and conditions of +# CANN Open Software License Agreement Version 2.0 (the "License"). +# Please refer to the License for details. You may not use this file except in compliance with the License. +# THIS SOFTWARE IS PROVIDED ON AN "AS IS" BASIS, WITHOUT WARRANTIES OF ANY KIND, EITHER EXPRESS OR IMPLIED, +# INCLUDING BUT NOT LIMITED TO NON-INFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE. +# See LICENSE in the root of the software repository for the full text of the License. +# ----------------------------------------------------------------------------------------------------------- + +import json + +from simpler_setup.tools.wait_reduction_sim import full_reduction, load_wait_graph, online_bitmap, simulate + + +def _pair_flags(edges) -> dict[tuple[str, str], frozenset[str]]: + return {edge: frozenset(("wait", "retain")) for edge in edges} + + +def test_online_bitmap_matches_full_reduction_inside_window(): + order = [str(i) for i in range(5)] + seq = {task_id: i for i, task_id in enumerate(order)} + possible_edges = [(str(i), str(j)) for i in range(5) for j in range(i + 1, 5)] + + for mask in range(1 << len(possible_edges)): + edges = {edge for i, edge in enumerate(possible_edges) if mask & (1 << i)} + flags = _pair_flags(edges) + exact = full_reduction(order, flags) + for bl in (1, 2, 3, 4): + actual = online_bitmap(order, seq, flags, bl) + exact_inside_window = {edge for edge in exact if seq[edge[1]] - seq[edge[0]] <= bl} + assert actual == exact_inside_window + + +def test_simulate_reports_window_cross_ring_and_resource_reductions(tmp_path): + ring1_b = str(1 << 32) + ring1_c = str((1 << 32) + 1) + data = { + "tasks": [{"task_id": task_id} for task_id in ("0", ring1_b, ring1_c, "3")], + "edges": [ + {"pred": "0", "succ": ring1_b, "source": "creator", "flags": ["wait", "retain"]}, + {"pred": ring1_b, "succ": ring1_c, "source": "creator", "flags": ["wait", "retain"]}, + {"pred": "0", "succ": ring1_c, "source": "explicit", "flags": ["wait", "retain"]}, + {"pred": ring1_c, "succ": "3", "source": "creator", "flags": ["wait", "retain"]}, + {"pred": "0", "succ": "3", "source": "creator", "flags": ["wait", "retain"]}, + ], + } + path = tmp_path / "deps.json" + path.write_text(json.dumps(data)) + + report = simulate(path, [2]) + window = report["windows"]["2"] + assert report["full_reduction_upper_bound"] == 2 + assert report["full_retain_classification_uncertain"] == 1 + assert window["removed"] == 1 + assert window["retain_classification_uncertain"] == 1 + assert window["redundant_window_misses"] == 1 + assert window["bitmap_misses_within_window"] == 0 + assert window["cross_ring_redundant_wait_pairs"] == 1 + assert window["cross_ring_removed"] == 1 + assert window["cross_ring_misses"] == 0 + assert window["estimated_readiness_fanout_nodes_removed"] == 1 + assert window["estimated_dep_pool_entries_removed"] == 1 + + +def test_load_wait_graph_inserts_hidden_alloc_before_first_consumer(tmp_path): + data = { + "tasks": [{"task_id": "1"}, {"task_id": "2"}], + "edges": [ + {"pred": "99", "succ": "1", "source": "creator", "flags": ["wait", "retain"]}, + {"pred": "1", "succ": "2", "source": "tensormap", "flags": ["wait"]}, + ], + } + path = tmp_path / "deps.json" + path.write_text(json.dumps(data)) + + order, seq, flags, uncertain = load_wait_graph(path) + assert order == ["99", "1", "2"] + assert seq == {"99": 0, "1": 1, "2": 2} + assert flags[("99", "1")] == frozenset(("wait", "retain")) + assert not uncertain + + +def test_load_wait_graph_or_accumulates_retain_only_records(tmp_path): + data = { + "tasks": [{"task_id": "1"}, {"task_id": "2"}, {"task_id": "3"}], + "edges": [ + {"pred": "1", "succ": "2", "source": "tensormap", "flags": ["retain"]}, + {"pred": "1", "succ": "2", "source": "tensormap", "flags": ["wait"]}, + {"pred": "2", "succ": "3", "source": "tensormap", "flags": ["retain"]}, + ], + } + path = tmp_path / "deps.json" + path.write_text(json.dumps(data)) + + _order, _seq, flags, _uncertain = load_wait_graph(path) + # A record that does not wait still contributes its retain to the pair, so + # a reduced ("1", "2") counts as a demotion rather than a pure drop. + assert flags[("1", "2")] == frozenset(("wait", "retain")) + # A pair no record waits on is not an edge of the WAIT graph. + assert ("2", "3") not in flags