KEP-2170: Add validation to Torch numProcPerNode field

[astefanutti]

What this PR does / why we need it:

Change the TorchMLPolicy numProcPerNode type to intstr.IntOrString and add CRD validation.

Checklist:

• Docs included if any changes are user facing

[andreyvelich]

I am also thinking about changing numProcPerNode to IntOrString. However, it is not a standard type in Python.
Thus, users should know what does this type mean: K8sIoApimachineryPkgUtilIntstrIntOrString.
Or we need to make numProcPerNode: Union[int,str] type.

Do we have any plans to expose numProcPerNode parameters to the Trainer object in SDK: https://github.com/kubeflow/trainer/blob/master/sdk/kubeflow/trainer/types/types.py#L73?

Or it will be always auto-populated based on num_nodes and resources_per_node ?

[andreyvelich]

cc @kubeflow/wg-training-leads @Electronic-Waste

[astefanutti]

Yes, I also think there should be a parameter exposed in the SDK. It is only loosely coupled with resources_per_node.

The Kubernetes Python client declares IntOrString fields as object, but Union[int,str] would be better.

[google-oss-prow]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign johnugeorge for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment