What happened?
Currently, handleModelDeletion blocks and waits for all associated model status ConfigMaps to be marked as Deleted. However, if a node is terminated unexpectedly, its ConfigMap is never updated.
What did you expect to happen?
Fix a bug where ClusterBaseModel and BaseModel custom resources get stuck in the Deleting state when the nodes they were deployed to have been terminated from the cluster.
The controller must be updated to handle this edge case gracefully by:
- Ignoring missing nodes: In
handleModelDeletion, check if the node corresponding to a ConfigMap exists in the cluster. If it returns errors.IsNotFound(), ignore the ConfigMap when counting modelsNotDeleted.
- Garbage collecting stale ConfigMaps: Update both
handleModelDeletion and processModelStatus to automatically issue a Delete on the stale ConfigMap when a Node is confirmed as missing (NotFound). This prevents unbounded growth of orphaned ConfigMaps.
How can we reproduce it (as minimally and precisely as possible)?
Terminates nodes from the cluster then delete the CBM.
# Example YAML configuration that reproduces the issue
Anything else we need to know?
Environment
- OME version: 0.1.5
- Kubernetes version (use
kubectl version):
- Cloud provider or hardware configuration:
- OS (e.g., from
/etc/os-release):
- Runtime (SGLang, vLLM, etc.) and version:
- Model being served (if applicable):
- Install method (Helm, kubectl, etc.):
What happened?
Currently,
handleModelDeletionblocks and waits for all associated model status ConfigMaps to be marked asDeleted. However, if a node is terminated unexpectedly, its ConfigMap is never updated.What did you expect to happen?
Fix a bug where
ClusterBaseModelandBaseModelcustom resources get stuck in theDeletingstate when the nodes they were deployed to have been terminated from the cluster.The controller must be updated to handle this edge case gracefully by:
handleModelDeletion, check if the node corresponding to a ConfigMap exists in the cluster. If it returnserrors.IsNotFound(), ignore the ConfigMap when countingmodelsNotDeleted.handleModelDeletionandprocessModelStatusto automatically issue aDeleteon the staleConfigMapwhen aNodeis confirmed as missing (NotFound). This prevents unbounded growth of orphaned ConfigMaps.How can we reproduce it (as minimally and precisely as possible)?
Terminates nodes from the cluster then delete the CBM.
# Example YAML configuration that reproduces the issueAnything else we need to know?
Environment
kubectl version):/etc/os-release):