Skip to content

[BUG] Fix: ClusterBaseModel stuck in deletion for terminated nodes #597

Description

@YouNeedCryDear

What happened?

Currently, handleModelDeletion blocks and waits for all associated model status ConfigMaps to be marked as Deleted. However, if a node is terminated unexpectedly, its ConfigMap is never updated.

What did you expect to happen?

Fix a bug where ClusterBaseModel and BaseModel custom resources get stuck in the Deleting state when the nodes they were deployed to have been terminated from the cluster.

The controller must be updated to handle this edge case gracefully by:

  1. Ignoring missing nodes: In handleModelDeletion, check if the node corresponding to a ConfigMap exists in the cluster. If it returns errors.IsNotFound(), ignore the ConfigMap when counting modelsNotDeleted.
  2. Garbage collecting stale ConfigMaps: Update both handleModelDeletion and processModelStatus to automatically issue a Delete on the stale ConfigMap when a Node is confirmed as missing (NotFound). This prevents unbounded growth of orphaned ConfigMaps.

How can we reproduce it (as minimally and precisely as possible)?

Terminates nodes from the cluster then delete the CBM.

# Example YAML configuration that reproduces the issue

Anything else we need to know?

Environment

  • OME version: 0.1.5
  • Kubernetes version (use kubectl version):
  • Cloud provider or hardware configuration:
  • OS (e.g., from /etc/os-release):
  • Runtime (SGLang, vLLM, etc.) and version:
  • Model being served (if applicable):
  • Install method (Helm, kubectl, etc.):

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions