diff --git a/docs/comm-domain.md b/docs/comm-domain.md index b7db6f67c4..f2de05c3ae 100644 --- a/docs/comm-domain.md +++ b/docs/comm-domain.md @@ -237,6 +237,19 @@ the resident `KernelArgs`, and injects it into every run's kernel SDMA streams and its kernels read a zero workspace address. The workspace is released at Worker finalize by ordinary stream/manager teardown. +**`enable_sdma` is a defect quarantine, not a capability switch, and it has an +exit condition.** Selecting an engine already belongs to the kernel, which reads +whichever addresses were injected; the runtime would otherwise provision every +engine the platform supports. SDMA is the exception because +`SdmaWorkspaceManager::Init()` is indivisible — the 16 KB workspace *is* the +descriptor table for 48 CP-process STARS streams, so there is no way to have the +address without holding the streams — and a Worker holding those streams gets a +single device-reset attempt instead of three after an AICore fault. Defaulting it +on would therefore put every ordinary Worker in that population, which is exactly +the regression [the investigation](investigations/2026-07-a2a3-sdma-fault-teardown.md) +records. The flag disappears once CANN bounds the final CP-process stream release, +or once pto-isa offers an `Init()` that separates the workspace from the streams. + Provisioning also warms the SDMA control path once, in the same call: a vector-only AICore ELF (`sdma_warmup_kernel.o`, staged per arch under `build/lib//sdma_warmup/`) walks every channel so the first @@ -260,7 +273,7 @@ and issue #1425. `enable_sdma` is currently honored only by the a2a3 onboard `tensormap_and_ringbuffer` runtime; host-build-graph, simulation, a5, and provider-disabled builds fail Worker init fast when it is set. A5 provisions its communication-context SDMA workspace by default; this is separate from -the callable-declared workspace mechanism controlled by `enable_sdma`. +the Worker-level workspace mechanism controlled by `enable_sdma`. --- diff --git a/src/common/platform/include/aicpu/aicpu_device_config.h b/src/common/platform/include/aicpu/aicpu_device_config.h index 88051e7013..4b9a65a658 100644 --- a/src/common/platform/include/aicpu/aicpu_device_config.h +++ b/src/common/platform/include/aicpu/aicpu_device_config.h @@ -55,9 +55,10 @@ int get_scheduler_timeout_ms(); * kind (see DmaWorkspaceKind). Published by simpler_aicpu_init (from * InitArgs.dma_workspace_addr[]) into a resident-SO array; the scheduler * copies each slot into every core's GlobalContext, so kernels read it via - * get_dma_workspace(args, kind). 0 means no callable has provisioned that - * engine. A callable that declares the engine is rejected before launch if the - * platform cannot provide a non-zero address. Out-of-range kinds are ignored. + * get_dma_workspace(args, kind). 0 means this Worker did not provision that + * engine — the platform does not support it, or the Worker declined it. A + * kernel that reads 0 is responsible for its own fallback; nothing rejects it + * before launch. Out-of-range kinds are ignored. */ void set_dma_workspace_addr(int kind, unsigned long long addr); diff --git a/src/common/platform/include/common/dma_workspace.h b/src/common/platform/include/common/dma_workspace.h index de601d7904..f6bdbf5e15 100644 --- a/src/common/platform/include/common/dma_workspace.h +++ b/src/common/platform/include/common/dma_workspace.h @@ -11,22 +11,34 @@ /** * Async-DMA engine workspace kinds. * - * The runtime provisions a per-device scratch workspace for each async-DMA - * engine it supports and injects the device addresses into every core's - * GlobalContext, so kernels obtain them via get_dma_workspace(args, kind) - * without threading them as user args. One slot per engine, indexed by this - * enum; DMA_WORKSPACE_KIND_COUNT sizes the injected array end to end - * (InitArgs, the resident AICPU config, and GlobalContext). + * The runtime provisions a per-device scratch workspace and injects the device + * addresses into every core's GlobalContext, so kernels obtain them via + * get_dma_workspace(args, kind) without threading them as user args. Selecting + * an engine is the kernel's decision, not the runtime's: every provisioned + * address is injected, and a kind the runtime did not provision reads back 0. + * One slot per engine, indexed by this enum; DMA_WORKSPACE_KIND_COUNT sizes the + * injected array end to end (InitArgs, the resident AICPU config, and + * GlobalContext). * * simpler-owned and deliberately independent of pto-isa's comm::DmaEngine — * the kernel maps this kind to the pto-isa engine tag at the call boundary. * Shared by host (provisioning + InitArgs) and device (scheduler + kernels), * so it carries no dependencies beyond the enum itself. * + * A Worker provisions an engine only when it both requested that engine and the + * device supports it (dma_workspace_supported_mask()), so an unrequested or + * unsupported kind stays 0. SDMA is the only engine anyone declines today, and + * the only one worth declining: provisioning it is inseparable from holding 48 + * CP-process STARS streams whose post-fault release CANN bounds neither with a + * completion fence nor with a portable timeout — see + * docs/investigations/2026-07-a2a3-sdma-fault-teardown.md. A Worker declines it + * by leaving `enable_sdma` off, which is the default. + * * Current support matrix: SDMA is available only on a2a3 onboard with the * tensormap_and_ringbuffer runtime. URMA is reserved for the future a5 * per-domain provider. Host-build-graph, simulation, a5, and builds without - * the a2a3 PTO-SDMA provider reject non-empty requirements at registration. + * the a2a3 PTO-SDMA provider reject a request for SDMA during Worker + * initialization, when the workspace would be provisioned. */ #ifndef PLATFORM_COMMON_DMA_WORKSPACE_H_