[Feature] Let consecutive runs queue on the same stream instead of round-tripping through a host sync

[ChaoWao]

Summary

Consecutive a2a3 runs are gated by a host-side blocking sync: the executor waits for run N's streams to drain before run N+1's kernels are submitted. But a stream is an ordered queue — if run N+1's AICore and AICPU kernels were enqueued on the same pair while N is still executing, the driver would serialize them for free, and the host would stop paying per-run launch latency between runs.

This asks whether we can let consecutive runs queue naturally on the stream instead of round-tripping through a host sync.

Motivation / Use Case

Today device execution is strictly serial, and the serialization is enforced on the host rather than by the queue:

• The execution claim from try_acquire_native_run is exclusive, and simpler_finalize_run releases it (c_api_shared.cpp:965) only after drain_execution (:919).
• drain_execution → reap_run() → sync_stream_pair() is a blocking aclrtSynchronizeStreamWithTimeout on both streams.
• Launch requires the claim, so run N+1 cannot submit until run N has drained and finalized.

Depth-two pipelining does not change this. What overlaps today is only run N+1's preparation — arena build, host orchestration, args H2D — against run N's device execution. That is what the two arena banks and two host Runtime buffers exist for (HOST_PER_RUN). The submission itself does not overlap, so between every pair of runs the device is idle for a host round trip: sync return → validate → finalize → claim release → next launch.

Ordering correctness would not depend on the host at all if both runs sat on the same queue, which makes the host sync a latency cost rather than a safety mechanism.

Note this is not what #1717 proposes. #1717 overlaps the device launch with the host-orchestration build inside one run (its main beneficiary is cold start), and it states that "run-to-run pipelining (prepared-successor) already hides O + launch for steady-state runs". Only preparation is pipelined; the launch is still behind the claim and the sync. This issue is the run-to-run device overlap that assumption implies but that does not exist yet.

Proposed API / Behavior

Enqueue run N+1's kernel pair while run N is still in flight, and let the stream order them:

• Admit two launched runs, not one launched plus one prepared, in the execution-claim contract.
• Keep the AICPU and AICore streams distinct. The AICPU Run kernel spins in the handshake waiting for the AICore workers, so collapsing them onto one queue would leave the AICore submission behind a spin that can never end.
• Completion becomes per-run rather than "sync the pair": each run needs its own completion fence to drain and validate against, instead of a whole-stream synchronize.

Alternatives Considered

Keeping the blocking sync and instead shortening the host round trip (cheaper validate, earlier claim release). That reduces the gap but does not remove it, and it leaves the host on the critical path between every pair of runs.

Additional Context

Known blockers — these are the reason this is a question and not a patch. Each is runner-wide state that two in-flight runs would race on:

1. device_wall_dev_ptr_ is a single runner-wide buffer. ensure_device_wall_buffer() allocates it once; every run's kernel_args.args.device_wall_data_base points at the same block; it is re-initialized per run and read back after drain. Two runs in flight clobber each other's phase/timing records — silently wrong DFX data, not a crash. It would have to become per-slot.

2. The host-side handshake clear in launch_run. It zeroes workers[i].aicore_done before submitting, and its own comment notes "the workers region persists across runs in the pooled arena" and that the clear exists to stop the AICPU sweep from "reading a prior run's report". If run N+1's clear executed while run N were still running, it would stomp N's live handshake. Arena banks are per-slot, so this may already be isolated — it needs confirming, not assuming.

3. Code publication becomes a drain barrier. A stream with queued work cannot be destroyed, so the stale-stream recreation that Fix A2A3 repeated-run AICore stream regression #1807 introduced would have to wait for the queue to empty. That is correct semantics — new code must not be executed by cores holding old instructions — and publication only happens on register_callable, so the cost is rare. But the design has to state it rather than discover it.

4. RunStreamPair's ownership model assumes one owner. It records the single PreparedExecution that submitted the pair and refuses ensure() while an owner is set (Refactor: submit every a2a3 run on one AICore/AICPU stream pair #1852). With two in-flight runs that becomes a set of in-flight runs, and "readiness" stops meaning "idle".

Related: #1717 (intra-run launch/host-build overlap), #984 (streaming orchestrator design space), #1807 (publication-keyed stream reuse), #1852 (one stream pair per runner, which is what makes "queue on the same stream" the natural shape).