ray scheduler either appends a random string to my specified app_id, or ignores it entirely

[starpit]

🐛 Bug

When I submit, e.g.

torchx run -s ray -cfg app_id=myOwnUniqueString ...

it seems that the ray scheduler always appends a "unique" 64-bit suffix to whatever I specify. I could understand doing this for cases where an app_id was not explicitly provided from the CLI, but if I am giving you an app_id to use, I kinda expect you to use it?

Using the utils.python builtin, at least only a random string is tacked on the end of what I provide. When using the utils.sh builtin, none of what I specify in the -cfg app_id=... actually appears in the resulting ray job id.

Module (check all that applies):

• torchx.spec
• torchx.component
• torchx.apps
• torchx.runtime
• torchx.cli
• torchx.schedulers
• torchx.pipelines
• torchx.aws
• torchx.examples
• other

To Reproduce

Steps to reproduce the behavior:

1. torchx run -s ray -cfg app_id=somethingSomething
2. observe that the ray job has a job_id of somethingSomething-64bitsofadditionalhex

Expected behavior

I think the ray scheduler should use what it is given, and only append random bits if it is using its own default value for app_id.

Environment

• torchx version (e.g. 0.1.0rc1): 0.2.0
• Python version: any
• OS (e.g., Linux): any
• How you installed torchx (conda, pip, source, docker): pip
• Docker image and tag (if using docker): n/a
• Git commit (if installed from source): n/a
• Execution environment (on-prem, AWS, GCP, Azure etc): n/a
• Any other relevant information: n/a

Additional context

[d4l3k]

@starpit can share a bit more about why you want to override the app_id?

We haven't supported setting app_id on the scheduler via the config options and always take it from the component for most schedulers. The app_id field in RayJob is an internal implementation detail

We also should really be throwing errors for unrecognized fields like this

All of the supported cfg options are listed under "Config Options" in the docs https://pytorch.org/torchx/main/schedulers/ray.html

[starpit]

the requirement is that the job ids be either specifiable, or knowable by another program (the program that calls torchx run). e.g. with dist.ddp, i can use --name to specify an id, but then the ray scheduler tacks some random bits on the end.

this wouldn't be the end of the world, if it were possible to know what id torchx assigned. unless i have missed something (possible!), it doesn't seem straightforward to do this? the output of torchx run is not structured, so it is not straightforward to parse. neither is there a torchx run -o json?

[d4l3k]

We don't document this but the output from torchx run will return only the job ID via stdout. All non-structured data is outputted to stderr. We use it for some of our integration tests https://github.com/pytorch/torchx/blob/main/torchx/cli/cmd_run.py#L217-L218

For more advanced integrations I'd recommend using the runner API interface https://pytorch.org/torchx/main/runner.html

[starpit]

Thanks! The caller in this case is not python code, so using the CLI is the only option.