ManifestWorkReplicaSet Rollout Plugin #160

youngbupark · 2025-10-28T08:56:03Z

This is the initial draft for supporting rollout plugin in ManifestWorkReplicaSet Work Controller
Note: rollback will be added when we propose MWRS automatic rollback enhancement.

openshift-ci · 2025-10-28T08:56:07Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: youngbupark
Once this PR has been reviewed and has the lgtm label, please assign qiujian16 for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

youngbupark · 2025-10-28T09:03:23Z

enhancements/sig-architecture/227-manifestworkreplicaset-rollout-plugin/README.md

+The following service defines the contract between Work Controller and the plugin. Each call must be idempotent, stateless, and time-bounded (≤30 s) to ensure consistent controller reconciliation. Plugin server must implement the following APIs. The helpers to implement server and clients will be implemented in [ocm/sdk-go](https://github.com/open-cluster-management-io/sdk-go) repository. 
+
+```proto
+// RolloutPluginService is the service for the rollout plugin.


Here is the initial commit of gRPC server proto - open-cluster-management-io/sdk-go#154

Note: The implementation can change as we develop.

enhancements/sig-architecture/227-manifestworkreplicaset-rollout-plugin/README.md

qiujian16 · 2025-10-29T02:19:21Z

enhancements/sig-architecture/227-manifestworkreplicaset-rollout-plugin/README.md

+// RolloutPluginService is the service for the rollout plugin.
+service RolloutPluginService {
+  // Initialize initializes the plugin.
+  rpc Initialize(InitializeRequest) returns (InitializeResponse);


I think we will need some clarification on error handling. What happens when a specific call fails? How would mwrs consumer to know and debug.

Good call. I will add error handling section.

enhancements/sig-architecture/227-manifestworkreplicaset-rollout-plugin/README.md

qiujian16 · 2025-10-29T02:24:15Z

enhancements/sig-architecture/227-manifestworkreplicaset-rollout-plugin/README.md

+  // If the validation is completed successfully, the plugin should return a OK result.
+  // If the validation is still in progress, the plugin should return a INPROGRESS result.
+  // If the validation is failed, the plugin should return a FAILED result.
+  rpc ValidateRollout(RolloutPluginRequest) returns (ValidateResponse);


When will this be called in mwrs reconciler? I think a flow on when these APIs will be called in mwrs controller will be helpful.

This doc includes mermaid sequence diagram to describe how each hook will be called. Please check this out.

qiujian16 · 2025-10-29T02:26:11Z

enhancements/sig-architecture/227-manifestworkreplicaset-rollout-plugin/README.md

+
+  // BeginRollout is called before the manifestwork resource is applied.
+  // It is used to prepare the rollout.
+  rpc BeginRollout(RolloutPluginRequest) returns (google.protobuf.Empty);


Does it mean any spec change on manifestwork will trigger this? What if placement changes but mw spec does not change in mwrs?

this BeginRollout will be called before creating manifestwork in each cluster (please see the mermaid sequence diagram) BeginRollout will be called whenever MWRS creates or update manifestwork in each cluster namespace.

qiujian16 · 2025-10-29T02:27:33Z

enhancements/sig-architecture/227-manifestworkreplicaset-rollout-plugin/README.md

+| False | False or not set | SUCCEEDED | Succeeded | Work has been successfully applied |
+| Unknown/Not set | Any | N/A | Progressing | Conservative fallback: treat as still progressing |
+
+The following state machine shows the expected transitions between rollout statuses:


is this only enabled when rollout strategy in mwrs is set?

yes correct.

qiujian16 · 2025-10-29T02:30:20Z

enhancements/sig-architecture/227-manifestworkreplicaset-rollout-plugin/README.md

+    # Plugin configuration
+    plugin:
+      # the image name of plugin sidecar
+      image: quay.io/open-cluster-management/my-rollout-plugin:v0.0.1


I am not sure if setting image only is enough if the plugin also need to wire with cloud service or mesh which mean the plugin also need some customized args or secret.

Make sense. hmm. in this case, it might be better to run plugin separately than sidecar, similar to envoy extension filter. WDYT? then we can also support multiple plugins.

qiujian16 · 2025-10-29T02:31:10Z

enhancements/sig-architecture/227-manifestworkreplicaset-rollout-plugin/README.md

+
+## Alternatives
+
+- Implement plugin server as a standalone service: In order to run plugin server securely, we need to enable TLS connection. Using a sidecar, we can avoid the complex network and security configuration.


on the other hand, plugin can be easily customized.

It might be better to run standalone services. Do you prefer running plugin service as a standalone service? @qiujian16

I think so, the plugin API might need to be authenticated also if it wires to a cloud service, or the placement needs to use its own service account to call the plugin.

Agree. I updated the design.

haoqing0110 · 2025-10-30T03:46:34Z

enhancements/sig-architecture/227-manifestworkreplicaset-rollout-plugin/README.md

+  - rolloutStatus: The current [cluster rollout status](https://github.com/open-cluster-management-io/sdk-go/blob/main/pkg/apis/cluster/v1alpha1/rollout.go#L23-L39) (e.g., ToApply, Progressing, Succeeded, Failed, TimeOut, Skip).
+  - manifestRevisionName: The name of the manifest revision applied to the cluster.
+
+### Configure custom plugin for work controller


Once a plugin is enabled, what's the behavior of a normal mwrs rollout which does not need to call any plugin?

Once a plugin is enabled, what's the behavior of a normal mwrs rollout which does not need to call any plugin?

Plugin will be applied to all mwrs. @haoqing0110 do you think it is better to make it opt-in ?

enhancements/sig-architecture/227-manifestworkreplicaset-rollout-plugin/README.md

youngbupark · 2025-11-18T10:58:18Z

enhancements/sig-architecture/227-manifestworkreplicaset-rollout-plugin/README.md

+    workDriver: kube
+    # plugin configuration
+    plugins:
+      - name: my-rollout


@qiujian16 @haoqing0110 rather than using sidecar model, I agree to use standalone service. here is new cluster-manager resource model to register the plugin. new one supports multiple plugins and users can select its plugin if they need.

youngbupark · 2025-11-18T10:59:05Z

enhancements/sig-architecture/227-manifestworkreplicaset-rollout-plugin/README.md

+  // ProgressRollout is called after the manifestwork is applied.
+  // Whenever the feedbacks are updated, this method will be called.
+  // The plugin can execute the rollout logic based on the feedback status changes.
+  rpc ProgressRollout(RolloutPluginRequest) returns (google.protobuf.Empty);


@annelaucg I removed Rollback specific operation as we discussed yesterday. it is much simpler.

youngbupark · 2025-11-18T11:01:48Z

enhancements/sig-architecture/227-manifestworkreplicaset-rollout-plugin/README.md

+
+### Using plugin in MWRS
+
+Plugin is an opt-in feature. User can use the registered plugin by setting `.spec.placementRefs[*].rolloutStrategy.plugin`. The reconciler makes sure that plugin is available and show its availability in `PluginLoaded` status condition.


@haoqing0110 the feedback makes sense. Plugin must be an opt-in feature. to minimize the impact on the existing resource, we should not use plugin by default although it is configured in clustermanager. Otherwise, it can change the behavior of mwrs resource which user may not expect.

@qiujian16 / based on @haoqing0110 feedback, I proposed new MWRS resource change to make plugin opt-in even if cluster-manager resource has registered plugins.

ok, note the work-controller will not be allowed to access clustermanager CR. So plugin setting will need to be set on work-controller's flag, or setting in a configmap by cluster manager, and work-controller will mount the configmap which makes sense to me also.

I will update it.

qiujian16 · 2025-11-18T13:36:07Z

enhancements/sig-architecture/227-manifestworkreplicaset-rollout-plugin/README.md

+    workDriver: kube
+    # plugin configuration
+    plugins:
+      - name: my-rollout


qiujian16 · 2025-11-18T13:40:15Z

enhancements/sig-architecture/227-manifestworkreplicaset-rollout-plugin/README.md

+
+### Using plugin in MWRS
+
+Plugin is an opt-in feature. User can use the registered plugin by setting `.spec.placementRefs[*].rolloutStrategy.plugin`. The reconciler makes sure that plugin is available and show its availability in `PluginLoaded` status condition.


ok, note the work-controller will not be allowed to access clustermanager CR. So plugin setting will need to be set on work-controller's flag, or setting in a configmap by cluster manager, and work-controller will mount the configmap which makes sense to me also.

qiujian16 · 2025-11-18T13:43:59Z

enhancements/sig-architecture/227-manifestworkreplicaset-rollout-plugin/README.md

+          # optional. secretRef is the reference for ca
+          secretRef: 
+            name: my-rollout-ca
+            namespace: default


namespace might not be needed. The secret has to be put in the open-cluster-management-hub ns.

annelaucg · 2025-11-19T00:02:54Z

enhancements/sig-architecture/227-manifestworkreplicaset-rollout-plugin/README.md

+        Work->>Work: Create rollout handler
+        Work->>Work: Find rollout/removed/timeout candidate clusters (RolloutResult)
+        alt timeout clusters exists and .spec.placementRefs[*].rolloutStrategy.abortOnFailure is true
+          Note over Work, PluginServer: Start automatic abort


Also need to set abort when Degraded or ValidateFailed.

So this will also need to be set during the ProgressRollout func and the ValidateRolloutFunc is that right?

No this will be set by MWRS controller. Please see the proposal in #164

annelaucg · 2025-11-19T00:04:36Z

enhancements/sig-architecture/227-manifestworkreplicaset-rollout-plugin/README.md

+      rolloutStrategy:
+        type: Progressive
+        # plugin is optional.
+        plugin: my-rollout


Do we need any failsafe if the plugin that the user adds causes issues? Or is that dependent on the user?

Can you please elaborate on failsafe ? In general, user should be able to recognize it. using plugin is opt-in feature. MWRS should show the error in status if Plugin is throwing error or unavailable. It might be better to stop the reconciler loop rather than self-resolving the problem.

haoqing0110 · 2025-11-20T03:50:06Z

enhancements/sig-architecture/227-manifestworkreplicaset-rollout-plugin/README.md

+
+* `BeginRollout()`: Called before applying the ManifestWork to a target cluste to roll out new revision, allowing the plugin to perform any necessary preparations.
+* `ProgressRollout()`: Called during every reconciliation loop to report the current rollout status to the plugin.
+* `ValidateRolloutCompletion()`: Called after the `Progressing` condition on the target cluster's ManifestWork becomes False. This hook enables post-rollout testing before the status is set to Succeeded. For example, the plugin could use this call to create a new ManifestWork that runs a Kubernetes Job for verification. **Note that this hook must be called within the `ProgressDeadline` timeout.**


When the rollout status is "Validating", what's the LastTransitionTime of it? LastTransitionTime is used to calculate timeout clusters.

LastTransitionTime will be the time when Progressing (ManifestWork) == False which is the same as the current behavior.

I will make it clear.

youngbupark added 2 commits October 27, 2025 13:00

Initial PR for rollout plugin

ea4ddcb

revise

a5b74b1

openshift-ci bot added the do-not-merge/work-in-progress label Oct 28, 2025

youngbupark mentioned this pull request Oct 28, 2025

✨ Initial commit of rollout plugin protobuf open-cluster-management-io/sdk-go#154

Open

youngbupark commented Oct 28, 2025

View reviewed changes

enhancements/sig-architecture/227-manifestworkreplicaset-rollout-plugin/README.md Outdated Show resolved Hide resolved

revise

d266d3c

youngbupark changed the title ~~KEP: ManifestWorkReplicaSet Rollout Plugin~~ ManifestWorkReplicaSet Rollout Plugin Oct 28, 2025

add risk

48ea733

youngbupark marked this pull request as ready for review October 29, 2025 02:00

openshift-ci bot removed the do-not-merge/work-in-progress label Oct 29, 2025

openshift-ci bot requested review from deads2k and qiujian16 October 29, 2025 02:00

qiujian16 reviewed Oct 29, 2025

View reviewed changes

haoqing0110 reviewed Oct 30, 2025

View reviewed changes

annelaucg reviewed Oct 31, 2025

View reviewed changes

enhancements/sig-architecture/227-manifestworkreplicaset-rollout-plugin/README.md Outdated Show resolved Hide resolved

youngbupark mentioned this pull request Nov 12, 2025

Support rollback for ManifestWorkReplicaSet #164

Open

youngbupark added 6 commits November 16, 2025 12:16

revise

840f9f3

updated

416c028

revised

2308fb0

updated

a5e9a40

updated

528d25b

wip

66f298e

youngbupark commented Nov 18, 2025

View reviewed changes

qiujian16 reviewed Nov 18, 2025

View reviewed changes

annelaucg reviewed Nov 19, 2025

View reviewed changes

haoqing0110 reviewed Nov 20, 2025

View reviewed changes


		## Alternatives

		- Implement plugin server as a standalone service: In order to run plugin server securely, we need to enable TLS connection. Using a sidecar, we can avoid the complex network and security configuration. No newline at end of file


		### Using plugin in MWRS

		Plugin is an opt-in feature. User can use the registered plugin by setting `.spec.placementRefs[*].rolloutStrategy.plugin`. The reconciler makes sure that plugin is available and show its availability in `PluginLoaded` status condition.

ManifestWorkReplicaSet Rollout Plugin #160

Are you sure you want to change the base?

ManifestWorkReplicaSet Rollout Plugin #160

Conversation

youngbupark commented Oct 28, 2025

Uh oh!

openshift-ci bot commented Oct 28, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haoqing0110 Oct 30, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

youngbupark Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

youngbupark Nov 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

haoqing0110 Oct 30, 2025 •

edited

Loading

youngbupark Nov 18, 2025 •

edited

Loading

youngbupark Nov 18, 2025 •

edited

Loading