Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .github/workflows/_st-sim-a2a3.yml
Original file line number Diff line number Diff line change
Expand Up @@ -120,6 +120,14 @@ jobs:
--require-pto-isa --manual "${{ inputs.manual_mode == 'only' && 'only' || 'include' }}" \
--enable-chip-swimlane --enable-dep-gen

- name: chip_swimlane smoke (a2a3 host_build_graph)
if: inputs.include_dfx_smokes
run: |
.venv/bin/python -m pytest tests/st/a2a3/host_build_graph/dfx/chip_swimlane/ \
--platform a2a3sim --device 0-15 -p no:xdist --pto-session-timeout 600 \
--require-pto-isa --manual "${{ inputs.manual_mode == 'only' && 'only' || 'include' }}" \
--enable-chip-swimlane 3

- name: PMU smoke (a2a3)
if: inputs.include_dfx_smokes
run: |
Expand Down
8 changes: 8 additions & 0 deletions .github/workflows/_st-sim-a5.yml
Original file line number Diff line number Diff line change
Expand Up @@ -113,6 +113,14 @@ jobs:
--require-pto-isa --manual "${{ inputs.manual_mode == 'only' && 'only' || 'include' }}" \
--enable-chip-swimlane --enable-dep-gen

- name: chip_swimlane smoke (a5 host_build_graph)
if: inputs.include_dfx_smokes
run: |
.venv/bin/python -m pytest tests/st/a5/host_build_graph/dfx/chip_swimlane/ \
--platform a5sim --device 0-15 -p no:xdist --pto-session-timeout 600 \
--require-pto-isa --manual "${{ inputs.manual_mode == 'only' && 'only' || 'include' }}" \
--enable-chip-swimlane 3

- name: PMU smoke (a5)
if: inputs.include_dfx_smokes
run: |
Expand Down
68 changes: 49 additions & 19 deletions docs/dfx/chip-swimlane-profiling.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,12 +45,16 @@ available.
`chip_swimlane_records.json` with `deps.json` from
[`dep_gen`](dep-gen.md) at post-process time; see
[§3.5](#35-dependency-arrows-from-dep_gen).
- **AICPU scheduler phases** — per-iteration breakdown into seven
mutually time-exclusive **outer** phases (`complete` / `async_poll`
/ `dispatch` / `release` / `dummy` / `early_dispatch` / `drain`), plus
`resolve`, `drain_prepare`, and `drain_publish` **inner** phases. `resolve`
is rendered on a sibling scheduler sub-lane with the same `Sched_N` label,
while the drain sub-phases are nested within their `drain` bar,
- **AICPU scheduler phases** — per-iteration breakdown into mutually
time-exclusive **outer** phases (`complete` / `async_poll` / `dispatch` /
`release` / `dummy` / `early_dispatch` / `drain` / `graph_prepare`), plus
nested phases.
In `tensormap_and_ringbuffer`, `resolve` is nested within `complete` or
`dummy`; in `host_build_graph`, `resolve`, `async_poll`, and `dummy` are
standalone, mutually exclusive phases on the dedicated P thread. HBG
`resolve` uses that P thread's main scheduler lane; TMR's nested `resolve`
uses a sibling scheduler sub-lane. The drain sub-phases are nested within
their `drain` bar,
and two **separate-lane**
phases (`dummy_task` and `predicated_skip`, sampled immediately before
`on_task_complete()` begins dependency resolution and rendered as synthetic
Expand Down Expand Up @@ -257,13 +261,14 @@ field but render differently in Perfetto:
| Phase | Role | Lane | `tasks_processed` semantic |
| ----- | ---- | ---- | -------------------------- |
| `complete` | outer | sched (pid=2) | FIN'd subtasks + sub-block retires this iter |
| `async_poll` | outer | sched | async-wait (SDMA/RoCE/URMA/CCU) subtasks completed this iter; split from `complete` |
| `async_poll` | outer | sched | async-wait completions resolved; zero means polling consumed CPU without completing work |
| `dispatch` | outer | sched | subtasks published this iter |
| `release` | outer | sched | deferred-release slots drained this iter |
| `dummy` | outer | sched | `dummy_ready_queue` entries handled this iter (explicit dummies and false-predicate tasks) |
| `early_dispatch` | outer | sched | blocks staged by speculative early-dispatch this pass |
| `drain` | outer | sched | blocks staged by this thread's global sync-start drain pass |
| `resolve` | inner | sched sub-lane, same `Sched_N` label as its outer lane | consumers visited in `on_task_complete` |
| `graph_prepare` | outer | sched | Graph Definition nodes expanded this pass |
| `resolve` | inner (TMR); P-thread outer (HBG) | TMR sched sub-lane; HBG P sched lane | consumers visited in `on_task_complete` (TMR); completed SPSC slots (HBG) |
| `drain_prepare` | inner | sched, nested in `drain` | subtasks prepared for global sync-start publication |
| `drain_publish` | inner | sched, nested in `drain` | subtasks published during global sync-start staging |
| `dummy_task` | separate-lane | Worker View AICPU_N (pid=4) | one dummy entering `on_task_complete()`; full identity is in `task_id` |
Expand All @@ -274,14 +279,32 @@ orchestrator submit path, so it has no swimlane lane. Read its cost
from `g_orch_fanin_cycle` in the device-log orch breakdown (the
`fanin` line) instead.

Outer phases are mutually time-exclusive within an iter. The converter renders
`resolve` on a sibling `Sched_N` tid so flow arrows attach to the outer
`complete`/`dummy` lane; `drain_prepare` and `drain_publish` remain on the
scheduler lane and are time-contained by `drain`. Separate-lane phases are
routed to a different lane by the converter
Outer phases are mutually time-exclusive within an iter. In
`tensormap_and_ringbuffer`, the converter renders `resolve` on a sibling
`Sched_N` tid because it is time-contained by the outer `complete`/`dummy`
lane. In `host_build_graph`, standalone `resolve` stays beside `async_poll` and
`dummy` on the P thread's main scheduler lane. `drain_prepare` and
`drain_publish` remain on the scheduler lane and are time-contained by `drain`.
Separate-lane phases are routed to a different lane by the converter
(Worker View AICPU_N), so they never overlap visually with the sched lane
bars even when their timestamps fall inside an outer span.

On the HBG P thread, consecutive empty async-wait polls are compacted into one
`async_poll(0)` record. Its duration is the exact sum of time spent inside the
poll calls, anchored at the point where the aggregate is flushed; it is not a
wall-clock envelope over the intervening loop bookkeeping. The aggregate is
flushed before `resolve` or `dummy`, when a poll resolves work or reports an
error, and when P exits. This keeps polling cost visible without exporting one
record per spin. A non-zero `tasks_processed` counts every resolved async-wait
entry, including internal Graph nodes, rather than only host-submitted stream
tasks. The compacted record's `shared_at_start` snapshot comes from the first
poll in the aggregate, while `loop_iter` names the iteration that flushes the
aggregate. Because the displayed start timestamp is synthesized from summed
poll CPU time, neither field identifies one wall-clock iteration boundary.
The converter still emits the record's real `shared_at_end` snapshot on the
global ready-queue counter track; only the aggregate's start-side metadata has
the synthesized-timestamp caveat.

Legacy phases (`scan` / `poll` / `idle` / `fanout` / `prestage`)
are still parsed for old captures but current a2a3/a5 builds no
longer emit them. Renames: `fanout` → `resolve`, `prestage` →
Expand Down Expand Up @@ -323,9 +346,10 @@ in. The trace contains:
blocks (level >= 4).
- **AICPU Scheduler** (pid=2) — per-iteration scheduler phase
blocks coloured by `phase` (level >= 3). Outer phases appear as sibling bars
on each scheduler thread's first `Sched_N` lane. `resolve` appears on an
adjacent `Sched_N` sub-lane, while `drain_prepare` and `drain_publish` nest
within `drain`.
on each scheduler thread's first `Sched_N` lane. TMR's nested `resolve`
appears on an adjacent `Sched_N` sub-lane; HBG's standalone `resolve` stays
on the P thread's first lane. `drain_prepare` and `drain_publish` nest within
`drain`.
- **Scheduler View** (pid=3) — task-execution overlay using AICPU
dispatch/finish timestamps (level >= 2), with the same labels
as Worker View.
Expand Down Expand Up @@ -377,6 +401,11 @@ report is printed to stdout; it correlates AICPU phase records
with the device log to attribute each scheduler iteration to a
specific overhead source.

The scheduler-budget parser counts every mutually exclusive outer phase and
standalone HBG P-thread `resolve` bars. It excludes only `resolve` records whose
timestamps are contained by a TMR `complete` or `dummy` parent, preventing the
nested TMR work from being counted twice.

### 3.4 Adding human-readable names

Lane labels degrade in two steps:
Expand Down Expand Up @@ -677,10 +706,11 @@ Both architectures use split phase streams:

- `ChipSwimlaneAicpuSchedPhaseRecord` (64 B) — one record per **emitted
phase**, not per scheduler iteration: a single iteration routinely emits
several (e.g. Complete, AsyncPoll, Dispatch, Release, plus the Resolve
inner phase). `ChipSwimlaneSchedPhaseKind` spans the outer phases
several (e.g. Complete, AsyncPoll, Dispatch, Release, plus Resolve).
`ChipSwimlaneSchedPhaseKind` spans the outer phases
(Complete, Dispatch, Release, Dummy, EarlyDispatch, AsyncPoll, Drain,
GraphPrepare), the inner ones (Resolve, DrainPrepare, DrainPublish) and
GraphPrepare), runtime-specific Resolve, the inner drain phases
(DrainPrepare, DrainPublish), and
the separate-lane markers (DummyTask, PredicatedSkip) — see §3.2 for how
each is rendered. Carries loop_iter + tasks_processed + pop_hit /
pop_miss deltas and queue-depth snapshots.
Expand Down
15 changes: 14 additions & 1 deletion docs/dfx/sched-overhead-model.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,10 +103,23 @@ python -m simpler_setup.tools.swimlane_converter <perf>.json \
| 1 | Overhead verdict — per-engine + system `all`/`has` overhead (% of makespan) |
| 2 | aicore switch — per-core min/mean/max, overhead-vs-independent split, makespan bound |
| 3 / 4 | Head OH / Tail OH distributions |
| 5 | AICPU scheduler-loop budget — ns/loop, phase split, pop hit-rate, fanout/fanin |
| 5 | AICPU scheduler-loop budget — separate S/P ns/loop, all mutually exclusive outer phases, standalone HBG P-thread Resolve, pop hit-rate, fanout/fanin |
| 6 | Critical-path attribution — compute vs scheduler-injected µs on the makespan path |

```bash
python -m simpler_setup.tools.sched_overhead_analysis \
--chip-swimlane-records-json <perf>.json --deps-json <deps>.json
```

For TMR captures, Resolve is nested in Complete or Dummy and is excluded from
the phase total to avoid double counting. For HBG captures, Resolve is
standalone work on the P thread and is included. Empty HBG async polling is
reported as compact `AsyncPoll(0)` bars, so its measured CPU cost contributes
to the scheduler budget instead of being reconstructed as idle. HBG's S
threads detect AICore FIN and dispatch work, while its P thread resolves
completion state and dependencies. Part 5 reports their loop rates separately;
the Tail-OH-to-loop comparison uses only S-thread loops because Tail OH ends at
FIN detection, before P-thread resolution begins. Phase totals still sum all
threads and therefore include P-thread CPU cost. The phase table lists only
phases represented by the capture, so runtime-specific absent phases do not
appear as synthetic zero rows.
14 changes: 14 additions & 0 deletions simpler_setup/scene_test.py
Original file line number Diff line number Diff line change
Expand Up @@ -1905,6 +1905,20 @@ def task_orch(orch, _args, _cfg, _ns=ns, _test_args=test_args, _config=config):
# pytest auto test method
# ------------------------------------------------------------------

@staticmethod
def _effective_enable_chip_swimlane(request) -> int:
"""Return the multi-round-safe chip-swimlane level for extension hooks."""
return effective_diagnostic_options(
request.config.getoption("--rounds", default=1),
chip_swimlane=request.config.getoption("--enable-chip-swimlane", default=0),
dump_args=0,
pmu=0,
dep_gen=False,
scope_stats=False,
swimlane_overhead=False,
warn=False,
).chip_swimlane

@staticmethod
def _effective_enable_dep_gen(request) -> bool:
"""Return the multi-round-safe dep-gen setting for extension hooks.
Expand Down
Loading
Loading