KEP-2170: Adding validation webhook for v2 trainjob #2307

akshaychitneni · 2024-10-24T21:34:20Z

Adds validation webhook for v2 trainjob.
Relates to #2209

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in Fixes #<issue number>, #<issue number>, ... format, will close the issue(s) when PR gets merged):
Fixes # #2209

Checklist:

Docs included if any changes are user facing

akshaychitneni · 2024-10-24T21:35:28Z

cc @tenzen-y @andreyvelich

coveralls · 2024-10-25T17:12:16Z

Pull Request Test Coverage Report for Build 11784298214

Details

6 of 6 (100.0%) changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage remained the same at 100.0%

Totals
Change from base Build 11758410179:	0.0%
Covered Lines:	78
Relevant Lines:	78

💛 - Coveralls

tenzen-y

Thank you for taking this, and moving this forward.
And Sorry for the delay.

pkg/controller.v2/trainjob_controller.go

tenzen-y · 2024-10-28T19:06:14Z

pkg/runtime.v2/core/clustertrainingruntime.go

+		Namespace: new.Namespace,
+		Name:      new.Spec.RuntimeRef.Name,


Have you ever seen the isseus when we use the old object names?

Why do we get new object here and not old ?

Here I am validating updated object instead of the existing one

pkg/runtime.v2/core/trainingruntime.go

pkg/runtime.v2/framework/plugins/jobset/jobset.go

tenzen-y · 2024-10-28T19:12:31Z

pkg/runtime.v2/framework/plugins/jobset/jobset.go

@@ -140,3 +143,115 @@ func (j *JobSet) ReconcilerBuilders() []runtime.ReconcilerBuilder {
 		},
 	}
 }
+
+func (j *JobSet) Validate(oldObj, newObj *kubeflowv2.TrainJob, runtimeInfo *runtime.Info) (admission.Warnings, field.ErrorList) {


It seems that there are some conflicts between @andreyvelich PR and this.
@akshaychitneni Could you consult with @andreyvelich, then which PRs should we merge into the main, first.

I rebased with @andreyvelich's changes

tenzen-y · 2024-10-28T19:20:28Z

pkg/webhook.v2/setup.go

@@ -31,7 +31,7 @@ func Setup(mgr ctrl.Manager, runtimes map[string]runtime.Runtime) (string, error
 		return kubeflowv2.TrainingRuntimeKind, err
 	}
 	if err := setupWebhookForTrainJob(mgr, runtimes); err != nil {
-		return "TrainJob", err
+		return kubeflowv2.TrainJobKind, err


Nice catch.

pkg/webhook.v2/trainjob_webhook.go

tenzen-y · 2024-10-28T19:24:35Z

pkg/webhook.v2/trainjob_webhook.go

Cool! This is what I imagined architechture in my KubeflowJobPipeline framework design phase.

tenzen-y · 2024-10-28T19:25:43Z

test/integration/framework/framework.go

-	failedCtrlName, err := controllerv2.SetupControllers(mgr, runtimes)
-	gomega.ExpectWithOffset(1, err).NotTo(gomega.HaveOccurred(), "controller", failedCtrlName)
-	gomega.ExpectWithOffset(1, failedCtrlName).To(gomega.BeEmpty())
+	if startControllers {


Have you ever seen any issues like null pointer when we start the controllers for webhook testing, right?

I think I have seen but we might not need to start the controllers just to validate create/update requests and leave to reconciler tests to cover reconciliation

That sounds reasonable. Thanks!

Interesting, maybe we should do the same for Kueue/JobSet integration tests cc @kannon92 @ahg-g

@akshaychitneni Why we didn't remove controller start from here:

trainer/test/integration/framework/framework.go

Lines 105 to 115 in c1fc937

failedCtrlName, err := controller.SetupControllers(mgr, runtimes, ctrlpkg.Options{

// controller-runtime v0.19+ validates controller names are unique, to make sure

// exported Prometheus metrics for each controller do not conflict. The current check

// relies on static state that's not compatible with testing execution model.

// See the following resources for more context:

// https://github.com/kubernetes-sigs/controller-runtime/pull/2902#issuecomment-2284194683

// https://github.com/kubernetes-sigs/controller-runtime/issues/2994

SkipNameValidation: ptr.To(true),

})

gomega.ExpectWithOffset(1, err).NotTo(gomega.HaveOccurred(), "controller", failedCtrlName)

gomega.ExpectWithOffset(1, failedCtrlName).To(gomega.BeEmpty())

?

akshaychitneni · 2025-03-14T17:04:56Z

@tenzen-y @andreyvelich thanks for your help with reviewing this PR. I have addressed all comments. However, I see e2e with notebook is failing due to job not moving to complete state within wait time as I see 2 of 3 pods remain in running state. Is it a known issue?

andreyvelich · 2025-03-14T17:11:46Z

For some reason, some of the pods are restarting after the master node is complete:

+ kubectl get pods
NAME                                  READY   STATUS      RESTARTS      AGE
t89ae0aa340f-trainer-node-0-0-mg98q   0/1     Completed   0             45s
t89ae0aa340f-trainer-node-0-1-fjwsr   1/1     Running     1 (13s ago)   45s
t89ae0aa340f-trainer-node-0-2-ffm9r   1/1     Running     1 (13s ago)   45s

I don't see same behaviour in other PRs: https://github.com/kubeflow/trainer/actions/runs/13859046765/job/38782615386?pr=2521

Did you try to run this Notebook locally on Kind cluster with your changes ?

pkg/constants/constants.go

andreyvelich

Thanks for this great contribution @akshaychitneni!
/lgtm
/assign @tenzen-y @astefanutti @Electronic-Waste

tenzen-y

Thank you for this!
I added small comments.

pkg/runtime/framework/plugins/torch/torch.go

test/integration/controller/trainjob_controller_test.go

test/integration/webhooks/trainjob_test.go

Signed-off-by: Akshay Chitneni <[email protected]>

akshaychitneni · 2025-03-16T01:53:30Z

@tenzen-y addressed your comments. Please take a look

tenzen-y

Thank you for this great contribution!
/lgtm
/approve

google-oss-prow · 2025-03-16T03:41:28Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tenzen-y

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [tenzen-y]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Signed-off-by: Akshay Chitneni <[email protected]> Co-authored-by: Akshay Chitneni <[email protected]>

google-oss-prow bot requested review from jinchihe and kuizhiqing October 24, 2024 21:34

google-oss-prow bot added the size/XL label Oct 24, 2024

akshaychitneni force-pushed the webhookv2 branch 5 times, most recently from 892a40b to f1a06c4 Compare October 25, 2024 16:36

google-oss-prow bot added size/L and removed size/XL labels Oct 25, 2024

akshaychitneni force-pushed the webhookv2 branch 3 times, most recently from ce983eb to 736a759 Compare October 25, 2024 17:09

akshaychitneni force-pushed the webhookv2 branch 2 times, most recently from f85da83 to ba32e68 Compare October 25, 2024 18:23

google-oss-prow bot added size/XL and removed size/L labels Oct 25, 2024

akshaychitneni force-pushed the webhookv2 branch from ba32e68 to 20136ef Compare October 25, 2024 18:46

google-oss-prow bot added size/L and removed size/XL labels Oct 25, 2024

akshaychitneni force-pushed the webhookv2 branch from 20136ef to 0aa9ee0 Compare October 25, 2024 18:50

tenzen-y reviewed Oct 28, 2024

View reviewed changes

akshaychitneni force-pushed the webhookv2 branch from 0aa9ee0 to 1b675c5 Compare November 3, 2024 17:52

google-oss-prow bot added size/XL and removed size/L labels Nov 3, 2024

akshaychitneni force-pushed the webhookv2 branch 3 times, most recently from a3ea261 to 4b4d76e Compare November 6, 2024 17:46

akshaychitneni force-pushed the webhookv2 branch 2 times, most recently from 3737792 to 75caeeb Compare March 14, 2025 01:02

google-oss-prow bot added size/XL and removed size/L labels Mar 14, 2025

akshaychitneni force-pushed the webhookv2 branch 3 times, most recently from 8668adf to a93176f Compare March 14, 2025 17:00

andreyvelich reviewed Mar 14, 2025

View reviewed changes

pkg/constants/constants.go Outdated Show resolved Hide resolved

akshaychitneni force-pushed the webhookv2 branch from a93176f to 94b0937 Compare March 14, 2025 23:49

andreyvelich reviewed Mar 15, 2025

View reviewed changes

google-oss-prow bot assigned astefanutti, Electronic-Waste, tenzen-y and andreyvelich Mar 15, 2025

google-oss-prow bot added the lgtm label Mar 15, 2025

tenzen-y reviewed Mar 15, 2025

View reviewed changes

Adding v2 trainjob validation webhook

aba3e4b

Signed-off-by: Akshay Chitneni <[email protected]>

akshaychitneni force-pushed the webhookv2 branch from 94b0937 to aba3e4b Compare March 16, 2025 01:51

google-oss-prow bot added size/L and removed lgtm size/XL labels Mar 16, 2025

tenzen-y reviewed Mar 16, 2025

View reviewed changes

google-oss-prow bot added the lgtm label Mar 16, 2025

google-oss-prow bot added the approved label Mar 16, 2025

google-oss-prow bot merged commit 250c116 into kubeflow:master Mar 16, 2025
15 checks passed

mahdikhashan pushed a commit to mahdikhashan/trainer that referenced this pull request Mar 16, 2025

KEP-2170: Adding validation webhook for v2 trainjob (kubeflow#2307)

aad6edc

Signed-off-by: Akshay Chitneni <[email protected]> Co-authored-by: Akshay Chitneni <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KEP-2170: Adding validation webhook for v2 trainjob #2307

KEP-2170: Adding validation webhook for v2 trainjob #2307

akshaychitneni commented Oct 24, 2024 •

edited

Loading

akshaychitneni commented Oct 24, 2024

coveralls commented Oct 25, 2024 •

edited

Loading

tenzen-y left a comment

tenzen-y Oct 28, 2024

andreyvelich Nov 7, 2024

akshaychitneni Nov 11, 2024

tenzen-y Oct 28, 2024

akshaychitneni Nov 6, 2024 •

edited

Loading

tenzen-y Oct 28, 2024

tenzen-y Oct 28, 2024

tenzen-y Oct 28, 2024

akshaychitneni Oct 30, 2024

tenzen-y Feb 28, 2025

andreyvelich Mar 11, 2025

akshaychitneni commented Mar 14, 2025

andreyvelich commented Mar 14, 2025

andreyvelich left a comment

tenzen-y left a comment

akshaychitneni commented Mar 16, 2025

tenzen-y left a comment

google-oss-prow bot commented Mar 16, 2025

	failedCtrlName, err := controller.SetupControllers(mgr, runtimes, ctrlpkg.Options{
	// controller-runtime v0.19+ validates controller names are unique, to make sure
	// exported Prometheus metrics for each controller do not conflict. The current check
	// relies on static state that's not compatible with testing execution model.
	// See the following resources for more context:
	// https://github.com/kubernetes-sigs/controller-runtime/pull/2902#issuecomment-2284194683
	// https://github.com/kubernetes-sigs/controller-runtime/issues/2994
	SkipNameValidation: ptr.To(true),
	})
	gomega.ExpectWithOffset(1, err).NotTo(gomega.HaveOccurred(), "controller", failedCtrlName)
	gomega.ExpectWithOffset(1, failedCtrlName).To(gomega.BeEmpty())

KEP-2170: Adding validation webhook for v2 trainjob #2307

KEP-2170: Adding validation webhook for v2 trainjob #2307

Conversation

akshaychitneni commented Oct 24, 2024 • edited Loading

akshaychitneni commented Oct 24, 2024

coveralls commented Oct 25, 2024 • edited Loading

Pull Request Test Coverage Report for Build 11784298214

Details

💛 - Coveralls

tenzen-y left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akshaychitneni Nov 6, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

akshaychitneni commented Mar 14, 2025

andreyvelich commented Mar 14, 2025

andreyvelich left a comment

Choose a reason for hiding this comment

tenzen-y left a comment

Choose a reason for hiding this comment

akshaychitneni commented Mar 16, 2025

tenzen-y left a comment

Choose a reason for hiding this comment

google-oss-prow bot commented Mar 16, 2025

akshaychitneni commented Oct 24, 2024 •

edited

Loading

coveralls commented Oct 25, 2024 •

edited

Loading

akshaychitneni Nov 6, 2024 •

edited

Loading