Dlrover master fail with worker running

[ljye2023]

Describe the bug
when some code error happens, dlrover master failed, some worker still running. the running worker would repeat this error message

[2026-03-12 14:50:48,019] [ERROR] [function_util.py:114:wrapped] <_InactiveRpcError of RPC that terminated with:

To Reproduce
Steps to reproduce the unexpected case:

• create job with some rank would run error.

Expected behavior
when dlrover master failed, the rest running and pending pod would be stopped.

APP Info (please complete the following information):

• DLRover: [0.6.1]

[ljye2023]

This change would cause all node removed, including the teminated pods. So I'd like to reclaim the requirement

1. During failure recovery:

• Preserve abnormal pods as much as possible
• Reschedule new pods

2. When a task exits:

• Delete pods in Running, Pending, or Init state
• Retain pods that are already in terminating state