[Code Health] TensorMap pool is a fixed 65536 for both runtimes: 100-200x oversized on hbg, derivable from ring_task_window on tmr

[ChaoWao]

Category

Technical Debt (cleanup, refactor)

Component

TensorMap

Description

CHIP_TENSORMAP_POOL_SIZE is a fixed 65536 entries in all four copies of
runtime_types.h. Nothing can change it — there is no runtime_env field, no env
var, no CallConfig path — so it is a compile-time number that every workload
shares. Measured peak occupancy says it is 100-200x oversized on host_build_graph
and about 5x oversized on tensormap_and_ringbuffer, and on hbg its exhaustion is a
hard failure whose only stated remedy is editing a macro the user cannot reach.

The two runtimes' maps are structurally different, so the fix is different for
each. This is the part worth writing down, because the same constant name hides
two different resources:

	tensormap_and_ringbuffer	host_build_graph
Storage	device arena, offsets wired to device addresses by wire_arena_pointers	host heap, std::unique_ptr members, never copied to the device
Reclaim	entries free as tasks retire	device completion frees nothing; an entry frees only when dependency computation finds its producer semantically covered
On exhaustion	back-pressure — wait_for_tensormap_entries spins, with a ~500 ms deadlock backstop	fatal SIMPLER_ERROR_TENSORMAP_OVERFLOW
Therefore	a window	a soft ceiling on the whole orchestration

Measured peak occupancy

next_entry_idx is exactly the high-water mark of concurrently live entries:
new_entry() takes from the free list first and bumps the cursor only when the free
list is empty, so a slot is added only when every earlier slot is live. Logged once
per orchestration at rt_orchestration_done (temporary probe, not committed):

case	runtime	map	peak	capacity	used
dsv4 FLASH decode (43 layers)	hbg	whole orchestration	285	65536	0.4%
dsv4 FLASH decode	hbg	per recorder thread (one Definition)	255	16384	1.6%
paged_attention_unroll	hbg	whole orchestration	768	65536	1.2%
graph_execution	hbg	whole orchestration	3	65536	0.005%
dsv4 FLASH decode	tmr	whole orchestration	5319-13304	65536	8-20%
paged_attention_unroll	tmr	whole orchestration	252	65536	0.4%
batch_paged_attention	tmr	whole orchestration	21	65536	0.03%

The tmr peak varies 5319 -> 13304 across four binds of one run, because it is a
reclaiming window: how many entries are live at once depends on how fast the device
retires tasks, not only on the graph.

Memory cost

tmr — measured arena_size = 23610304 (23.6 MB) for dsv4, of which the tensormap
is:

entry_pool         65536 x 128 B          = 8.00 MB
free_entry_list    65536 x 8 B            = 0.50 MB
buckets + epochs   4096 x (8 + 4) B       = 0.05 MB
task_entry_heads   4 rings x 16384 x 12 B = 0.75 MB
                                           --------
                                            9.30 MB  = 39% of the runtime arena

Held once on the device and once as the cached host mirror
(store_prebuilt_runtime_image keeps host_arena), per (callable, config). The
image is cached across binds, so this is resident memory plus a one-off ~24 MB H2D,
not per-bind latency.

hbg — all host heap: 8.5 MB for the orchestration map plus 8 recorder threads x
2.1 MB = about 25 MB per rank process, against a measured need of ~36 KB and
~32 KB.

The tmr size is derivable from an existing knob

Re-running dsv4-tmr with runtime_env.ring_task_window changed:

ring_task_window	tensormap peak	peak / window
1024	1278-1297	1.25
4096	4981-4985	1.22
16384	5319-13304	0.32-0.81

Below 16384 the peak tracks the window at a tight 1.22-1.25x — the window is the
binding constraint. At 16384 it stops binding and the graph's own live set takes
over, which is why the peak lands under the window and starts to vary. A resource
that holds a fixed 1.25x ratio to an existing knob does not need a number of its own.

One thing this study did not settle: the pool is shared across all four rings,
and all three experiments sized every ring the same, so the data cannot distinguish
k x max(ring_task_window) from k x sum(ring_task_window). The 1024 point (peak
1278, not ~5120) leans toward max, but dsv4 may simply not reach the deeper ring
indices. A per-ring-unequal run is needed before committing to a formula.

Location

• src/a2a3/runtime/tensormap_and_ringbuffer/runtime/runtime_types.h:74-75 — CHIP_TENSORMAP_POOL_SIZE (65536), CHIP_TENSORMAP_NUM_BUCKETS (4096)
• src/a5/runtime/tensormap_and_ringbuffer/runtime/runtime_types.h:74-75 — same
• src/a2a3/runtime/host_build_graph/runtime/runtime_types.h:96-97 — same
• src/a5/runtime/host_build_graph/runtime/runtime_types.h:96-97 — same
• src/{a2a3,a5}/runtime/host_build_graph/runtime/orchestrator_core/orchestrator.cpp:675 — GRAPH_RECORD_TENSORMAP_POOL_SIZE (16384), the per-recorder-thread map
• src/{a2a3,a5}/runtime/tensormap_and_ringbuffer/runtime/shared/tensormap.cpp:84-87 — reserve_layout_default, which hardcodes the constant into the device arena
• src/{a2a3,a5}/runtime/host_build_graph/runtime/shared/tensormap.cpp:123-125 — init_default, the host-heap counterpart
• src/{a2a3,a5}/runtime/host_build_graph/runtime/orchestrator_core/orchestrator.cpp — the fatal exhaustion path, "Increase CHIP_TENSORMAP_POOL_SIZE (current: %d)"
• src/{a2a3,a5}/runtime/tensormap_and_ringbuffer/runtime/orchestrator.cpp — the back-pressure counterpart with the same unreachable advice

Proposed Fix

Three steps, in this order — the first is independent of the unsettled question above.

1. hbg: grow the pool in chunks instead of sizing it up front. The hbg map is host
heap with no wire format and no device arena, so it can grow. The only constraint is
that entries are reached by pointer (bucket chains, the free-entry stack, per-task
chains), so the backing must not move: a chunked pool (entry_at(i) ->
chunks[i >> k][i & mask], append a chunk when the cursor reaches the end, never
move an existing one) satisfies that. Start at ~1024 entries (128 KB) and grow.

This is not only ~24 MB of host memory per rank process. It removes a correctness
cliff: today a large enough non-Graph hbg orchestration hits
SIMPLER_ERROR_TENSORMAP_OVERFLOW and the error tells the user to raise a macro they
cannot reach. With growth there is no ceiling to hit and the message can go away.
Pure host code, so cpput can cover it.

2. tmr: derive the pool from ring_task_window instead of the constant. Settle
max vs sum with a per-ring-unequal run first, then size it as a small multiple —
2x covers every measured point with 1.6x headroom. A user who sets
ring_task_window to 4096 then gets a 1 MB tensormap instead of 8 MB, without
learning what a tensormap is. Note that under-sizing here costs orchestration stalls,
not a deadlock: entries free as already-submitted tasks retire, so progress does not
depend on the blocked submission. That is the same trade ring_task_window itself
already makes, not a new risk.

3. Derive CHIP_TENSORMAP_NUM_BUCKETS from the pool size (a load factor) rather
than fixing it at 4096. Only ~50 KB, so this is tidiness rather than savings — worth
doing in the same change as step 2, not on its own.

The reduce user-facing configuration motivation behind this is worth stating
plainly: the tensormap is not a user knob today, so there is nothing to delete
there. The user-facing sizing surface is exactly
runtime_env.{ring_task_window, ring_heap, ring_dep_pool}. What this issue offers
that motivation is a precedent — a resource that holds a fixed ratio to the task
window can be derived away entirely. ring_dep_pool is the same shape (per-ring
fanin spill, which should also scale with the window) and is the most likely knob to
be removable next.

Related: #1920 (hbg ready queues sized by a fixed constant), #1956 (derive the hbg
graph heap rather than a 256 MB constant) — both the same class of fixed-constant
sizing. #1582 (a2a3 vs a5 tmr divergence) matters because every constant above exists
in two arch copies that must move together.

Measured at commit 3c89eaa on a2a3 (Ascend 910),
Linux aarch64, CANN 9.0.0, driver 26.0.rc1. All probes were reverted; the tree carries
no part of this study.

Priority

Medium (minor risk, should fix in next few releases)