[kubernetes] Pods with wrong image getting stuck forever

[aivanou]

🐛 Bug

Module (check all that applies):

• torchx.spec
• torchx.component
• torchx.apps
• torchx.runtime
• torchx.cli
• torchx.schedulers
• torchx.pipelines
• torchx.aws
• torchx.examples
• other

When job started with image that does not exist, it is getting stuck forever. We need to provide better experience in propagating the errors to the users via torchx.

Repro:

torchx run --scheduler kubernetes --scheduler_args namespace=default,queue=default examples/apps/dist_cifar/component.py:trainer --image dummy_image --rdzv_backend=etcd-v2 --rdzv_endpoint=etcd-server:2379 --nnodes 2 -- --epochs 1 --output_path s3://torchx-test/aivanou


torchx status $job

The second command will always show pending

[d4l3k]

volcano-sh/volcano#1765

[d4l3k]

Looks like to support this we'll have to directly query all of the pods.

[kiukchung]

is this still an issue? can we fix as part of 0.1.2?

[d4l3k]

It's more on the volcano side. Kubernetes infinitely retries when images don't exist

[kiukchung]

@d4l3k any issues on volcano side that you can link here to help us track it?

[d4l3k]

two comments up :) volcano-sh/volcano#1765