[CI] st-network1-onboard-a2a3 fails from remote-node device init (507033), not from the change under test

[ChaoWao]

Platform

a2a3 (Ascend 910B/C hardware)

Runtime Variant

tensormap_and_ringbuffer — but the job fails before any runtime code executes.

Description

st-network1-onboard-a2a3 is failing across unrelated PRs, and the one
failure whose log I read fails in remote-node device initialization, before
any test workload runs:

RuntimeError: remote L3 session open failed for worker 1:
  remote L3 session startup failed for worker 1: remote session startup:
    File ".../simpler/remote_l3_session.py", line 1299, in run_session
      inner_worker.init(_startup_deadline=startup_deadline)
    File ".../simpler/worker.py", line 8331, in _await_children_ready
      raise RuntimeError(f"{kind} worker {i} (pid {pids[i]}) failed during init: ...")
  RuntimeError: chip worker 0 (pid 336220) failed during init:
    chip_process dev=14 init: RuntimeError: simpler_init failed with code 507033

The local side came up fine ([chip_process pid=3406204 dev=12] ready). What
failed is the remote machine's chip worker on dev=14 — a device the local
job never allocated, so the local --device 12,13 reservation says nothing
about whether it was free.

Filing rather than re-running to green because this lane is red on several PRs
at once, the failing test set rotates between runs, and one test
(global_tload_mpirun_l3) fails identically on two unrelated changes. That
combination is the signature of the environment rather than of any one change —
and per discipline.md §5, a lane
re-run to green without recording why is indistinguishable from one hiding a
real regression.

Observations

#	PR	Job	Failing test(s)	Mechanism
1	#2262	104729208186	test_global_tload_mixed_l3_network1, global_tload_mixed_l3, vector_add_mixed_l3, compute_then_tload_mixed_l3 (4)	remote chip_process dev=14 init: simpler_init failed with code 507033 (log read)
2	#2262 (re-run)	104745555245	global_tload_mpirun_l3	not read — run still in progress, so only the annotation is available
3	#2252	104743937959	global_tload_mpirun_l3	not read
4	#2251	104723385181	not read	not read
5	#2263	—	—	passed

So it is not a constant red: #2263 went green on the same lane. Rows 2 and 3
are the same test on two unrelated changes.

Two things I did not establish and that the next person should not assume
from this issue:

• What 507033 denotes. It is absent from docs/troubleshooting/device-error-codes.md,
  from src/, and from CANN 9.0.0's acl/acl_base.h as a searchable literal.
  The table in that doc stops at 507046; 507033 should be added once
  somebody resolves it.
• Whether rows 2–4 share row 1's mechanism. Only the test identity matches.
  Reading row 2's log needs its run to finish first (gh run view --job ... --log-failed
  refuses while any sibling job is pending).

Suggested next step

Resolve 507033, then decide whether the remote node needs the same device
reservation discipline the local side gets. If dev=14 on the remote machine
can be occupied by another user while this lane runs, the lane has no exclusive
claim on the hardware it needs and will keep failing at a rate set by unrelated
load — which the rotating test set is consistent with.

Git Commit ID

Observed against 22385d2b0 as base (rows 1–2 on 4779267a4).

NPU Kind

Ascend 910B/C — runner a2a3pod-1, remote node reached through the network1 config.

Host Platform

Linux (aarch64)

Related

• [CI] st-pod-onboard-a2a3 fails ~half of runs from pod-runner network, not from the change under test #1768 — same shape on a different lane (st-pod-onboard-a2a3, pod-runner network). Same runner family (a2a3pod-1); different failure mechanism, so tracked separately.
• [Bug] Allreduce L3 cases intermittently hit an AICore UB out-of-bounds in full-suite onboard runs #1489 — intermittent allreduce L3 failures in full-suite onboard runs (st-onboard-a2a3).

[ChaoWao]

Row 2's log is now readable (the run finished), and it closes the gap I left above — but it is a different mechanism from row 1, so this lane has at least two.

Row 2 (global_tload_mpirun_l3, job 104745555245) never reaches a device-init failure. It gets a device-side AIV exception mid-run:

EZ9999[PID: 860208] ... The error from device(chipId:6, dieId:1), serial number is 47,
  there is an exception of fftsplus aivector error
→ aclrtSynchronizeStreamWithTimeout (AICPU) failed: 507018
→ recover_device_or_mark_unusable: bounded device drain failed: 507015
→ validate_runtime_impl: sched_error_code=100 SCHEDULER_TIMEOUT
→ RuntimeError: Worker.close(): teardown left resources un-reclaimed (leaked):
    1 global comm domain(s)
→ RuntimeError: child teardown: pid 680931 exited with code 1

So the corrected picture:

#	Mode	Where	Surfaces as
1	simpler_init fails with 507033 before any workload	remote node, dev=14	remote L3 session open failure
2	fftsplus aivector error mid-run	remote node (its mono_ns domain, not the local one)	507018 → SCHEDULER_TIMEOUT → teardown leaks 1 global comm domain → child exit 1

Two things this changes about the issue as filed:

• The title is too narrow. It names the 507033 init path, which is only mode 1. Mode 2 is a device AIV exception, and fftsplus aivector error puts it in the same family as [Bug] Allreduce L3 cases intermittently hit an AICore UB out-of-bounds in full-suite onboard runs #1489 (VEC instruction error: the ub address out of bounds on the allreduce L3 cases) rather than anything about network1 wiring. Whether mode 2 belongs here or as a comment on [Bug] Allreduce L3 cases intermittently hit an AICore UB out-of-bounds in full-suite onboard runs #1489 is a maintainer call; I have left it here so both observations stay with the lane that produced them.
• The comm-domain leak is a second-order symptom worth naming separately. Worker.close() reporting 1 global comm domain(s) un-reclaimed is what turns a device exception into a child exit 1, and it is the thing the harness actually reports. If a device error on this path is expected to be survivable, that teardown should reclaim rather than assert.

Still unresolved and unchanged from the issue body: what 507033 denotes (absent from docs/troubleshooting/device-error-codes.md, from src/, and from CANN 9.0.0's acl/acl_base.h as a searchable literal), and the mechanisms behind rows 3 and 4.