Problem
When a TP worker fails to map or unmap KV tensors, the current IPC path can still return a success response. That hides the actual failure from the coordinator and makes the later wait path fail only with a generic timeout.
There is also a context issue in the KV tensor map callback: the callback can fall back to the default (pp_rank=0, group_id=0) values instead of using the manager's recorded pipeline rank and broadcast group id. In TP/PP deployments this can report or wait on the wrong group context.
Expected behavior
- Worker-side
map_to_kv_tensors / unmap_from_kv_tensors failures should be propagated to the IPC caller.
- KV tensor creation checks should preserve the manager's
pp_rank and group_id.
- Timeout errors while waiting for KV tensors should include the last observed exception so the root cause is visible in logs.
Notes
I have a small fix branch that keeps this scoped to TP IPC / KV tensor error propagation.
Problem
When a TP worker fails to map or unmap KV tensors, the current IPC path can still return a success response. That hides the actual failure from the coordinator and makes the later wait path fail only with a generic timeout.
There is also a context issue in the KV tensor map callback: the callback can fall back to the default
(pp_rank=0, group_id=0)values instead of using the manager's recorded pipeline rank and broadcast group id. In TP/PP deployments this can report or wait on the wrong group context.Expected behavior
map_to_kv_tensors/unmap_from_kv_tensorsfailures should be propagated to the IPC caller.pp_rankandgroup_id.Notes
I have a small fix branch that keeps this scoped to TP IPC / KV tensor error propagation.