BatchSandbox remains Failed after the same Pod recovers to Running and Ready

[jianpingpei]

Summary

A BatchSandbox can remain permanently in Failed even when the same Pod later recovers to Running and Ready after a transient container-creation failure.

This leaves the control-plane state inconsistent with the actual Pod and causes clients that observed Failed to abandon an otherwise running sandbox.

Observed timeline

The following sequence was observed on a single-Pod BatchSandbox:

1. The Pod was scheduled successfully; its sandbox, init container, image pull, and sidecar startup all succeeded.

2. Creating the main container took longer than the runtime deadline and kubelet emitted:

   Create container agent error: context deadline exceeded
   

3. A retry shortly afterward reported:

   container name ... already existed
   

4. The BatchSandbox phase changed from Pending to Failed.

5. About 85 seconds later, kubelet successfully created and started the same main container.

6. The Pod became Running and Ready, but the BatchSandbox remained Failed indefinitely.

There was no rescheduling or replacement Pod involved.

Current behavior

applySteadyRuntimePhase returns immediately when the existing BatchSandbox phase is Failed:

if status.Phase == sandboxv1alpha1.BatchSandboxPhaseFailed {
    return
}

Therefore a later reconciliation cannot restore Succeed, even when all observed Pods are Running and Ready and no current failure remains.

Expected behavior

Transient container-creation failures should not leave a BatchSandbox permanently inconsistent with a recovered Pod.

Possible approaches:

• avoid making the BatchSandbox terminal while the same Pod can still complete container creation; or
• allow a narrowly scoped Failed -> Succeed reconciliation when the same Pod subsequently becomes Running and Ready and the recorded failure is no longer present.

The recovery rule should not hide genuine terminal failures or treat a replacement Pod as recovery without an explicit policy.

Suggested regression test

1. Start with a Pending BatchSandbox and a Pod reporting a transient container creation failure.
2. Reconcile it to Failed.
3. Update the same Pod to Running, Ready, with the main container running and no active waiting/terminated failure.
4. Reconcile again.
5. Verify the BatchSandbox no longer remains stale in Failed (according to the chosen recovery semantics).

This is related to status convergence, but distinct from detecting a main-container restart after a sandbox was already Ready.

[luffy0x]

@jianpingpei, thank you for the detailed report.

I would like to work on this issue.

Before implementation, I plan to clarify and cover the recovery semantics with a focused regression test: a Failed BatchSandbox may recover only when the same Pod later becomes Running and Ready, the main container is running, and no active failure remains. A replacement Pod should not be treated as the same recovery path without an explicit policy.

My proposed scope is:

• add a Pending -> Failed -> same-Pod recovery regression test;
• implement the narrowly scoped state transition;
• preserve genuine terminal failures and replacement-Pod behavior.

Please let me know whether this scope matches the intended direction.

中文

@jianpingpei，感谢你提供详细的问题描述。

我想认领并处理这个 issue。

在实现之前，我计划先明确恢复语义，并通过一个聚焦的回归测试覆盖该场景：只有当同一个 Pod 后续恢复为 Running 且 Ready、主容器处于运行状态，并且当前不存在活跃失败条件时，Failed 状态的 BatchSandbox 才可以恢复。除非有明确策略，否则替换后的 Pod 不应被视为同一恢复路径。

我计划的实现范围是：

• 添加 Pending -> Failed -> 同一 Pod 恢复 的回归测试；
• 实现严格限定范围的状态转换；
• 保持真实终态失败和替换 Pod 的现有行为。

请确认这个范围是否符合项目预期的实现方向。

[jianpingpei]

Thanks for taking this on. Your proposed scope matches the intended direction.

Please go ahead with the focused regression test and narrowly scoped recovery transition. One important acceptance point is to preserve Pod identity so that a replacement Pod, including one that may reuse the same name, is not treated as recovery of the original Pod. Genuine terminal failures should continue to remain terminal.

Please link the PR to this issue when it is ready.

[luffy0x]

PR opened: https://github.com/opensandbox-group/OpenSandbox/pull/1523\n\nIt adds same-Pod UID-aware recovery, rejects same-name replacement Pods, preserves terminal resume failures, and includes tests, generated CRD output, and documentation.