[release-4.16] OCPBUGS-59647: Reduce Frequency of Update Requests for Copied CSVs (#3597)

[tmshort]

Cherry-pick from release-4.17

• (bugfix): reduce frequency of update requests for CSVs

by adding annotations to copied CSVs that are populated with hashes of the non-status fields and the status fields.

This seems to be how this was intended to work, but was not actually working this way because the annotations never actually existed on the copied CSV. This resulted in a hot loop of update requests being made on all copied CSVs.

• update unit tests

• updates to test so far

• Small changes

• Add metadata drift guard to copyToNamespace

Since we switched to a PartialObjectMetadata cache to save memory, we lost visibility into copied CSV spec and status fields, and the reintroduced nonStatusCopyHash/statusCopyHash annotations only partially solved the problem. Manual edits to a copied CSV could still go undetected, causing drift without reconciliation.

This commit adds two new annotations: olm.operatorframework.io/observedGeneration and olm.operatorframework.io/observedResourceVersion. It implements a mechanism to guard agains metadata drift at the top of the existing-copy path in copyToNamespace. If a stored observedGeneration or observedResourceVersion no longer matches the live object, the operator now:

  • Updates the spec and hash annotations
  • Updates the status subresource
  • Records the new generation and resourceVersion in the guard annotations

Because the guard only fires when its annotations are already present, all existing unit tests pass unchanged. We preserve the memory benefits of the metadata‐only informer, avoid extra GETs, and eliminate unnecessary API churn.

Future work may explore a WithTransform informer to regain full object visibility with minimal memory impact.

• Tests for metadata guard

Verifies that exactly three updates (spec, status, guard) are issued when the observedGeneration doesn’t match.

• Persist observed annotations on all status updates

• GCI the file

• Use TransformFunc

Unit tests not updated

• Update operatorgroup tests to compile

• Restore operatorgroup_test from master

Remove metadatalister

• Remove more PartialObjectMetadata

• Remove hashes from operator_test

• Fix error messages for static-analysis

• Update test annotations and test client

• Rename pruning to listerwatcher

• Set resync to 6h

• Add CSV copy revert syncer

• Log tweaks

• Consolidate revert and gc syncers

• Add logging and reduce the amount of metadata in the TransformFunc

• Handle the copy CSV revert via a requeue of the primary CSV

• Revert "Set resync to 6h"

This reverts commit 855f940a2199bd4071c51f14ef44728550bf13cf.

• Add delete handler for copied csv

• Revert whitespace change

• Rename function, fix comment

---

Upstream-repository: operator-lifecycle-manager
Upstream-commit: d055f28750cf62f966f566d36990fff5285c7a71 (cherry picked from commit bc111a9) (cherry picked from commit 882eb21) (cherry picked from commit e4bc847)

[openshift-ci-robot]

@tmshort: This pull request references Jira Issue OCPBUGS-59647, which is invalid:

• expected dependent Jira Issue OCPBUGS-59253 to be in one of the following states: VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA), but it is ON_QA instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Cherry-pick from release-4.17

• (bugfix): reduce frequency of update requests for CSVs

by adding annotations to copied CSVs that are populated with hashes of the non-status fields and the status fields.

This seems to be how this was intended to work, but was not actually working this way because the annotations never actually existed on the copied CSV. This resulted in a hot loop of update requests being made on all copied CSVs.

• update unit tests

• updates to test so far

• Small changes

• Add metadata drift guard to copyToNamespace

Since we switched to a PartialObjectMetadata cache to save memory, we lost visibility into copied CSV spec and status fields, and the reintroduced nonStatusCopyHash/statusCopyHash annotations only partially solved the problem. Manual edits to a copied CSV could still go undetected, causing drift without reconciliation.

This commit adds two new annotations: olm.operatorframework.io/observedGeneration and olm.operatorframework.io/observedResourceVersion. It implements a mechanism to guard agains metadata drift at the top of the existing-copy path in copyToNamespace. If a stored observedGeneration or observedResourceVersion no longer matches the live object, the operator now:

 • Updates the spec and hash annotations
 • Updates the status subresource
 • Records the new generation and resourceVersion in the guard annotations

Because the guard only fires when its annotations are already present, all existing unit tests pass unchanged. We preserve the memory benefits of the metadata‐only informer, avoid extra GETs, and eliminate unnecessary API churn.

Future work may explore a WithTransform informer to regain full object visibility with minimal memory impact.

• Tests for metadata guard

Verifies that exactly three updates (spec, status, guard) are issued when the observedGeneration doesn’t match.

• Persist observed annotations on all status updates

• GCI the file

• Use TransformFunc

Unit tests not updated

• Update operatorgroup tests to compile

• Restore operatorgroup_test from master

Remove metadatalister

• Remove more PartialObjectMetadata

• Remove hashes from operator_test

• Fix error messages for static-analysis

• Update test annotations and test client

• Rename pruning to listerwatcher

• Set resync to 6h

• Add CSV copy revert syncer

• Log tweaks

• Consolidate revert and gc syncers

• Add logging and reduce the amount of metadata in the TransformFunc

• Handle the copy CSV revert via a requeue of the primary CSV

• Revert "Set resync to 6h"

This reverts commit 855f940a2199bd4071c51f14ef44728550bf13cf.

• Add delete handler for copied csv

• Revert whitespace change

• Rename function, fix comment

---

Upstream-repository: operator-lifecycle-manager
Upstream-commit: d055f28750cf62f966f566d36990fff5285c7a71 (cherry picked from commit bc111a9) (cherry picked from commit 882eb21) (cherry picked from commit e4bc847)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: tmshort

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• DOWNSTREAM_OWNERS [tmshort]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[tmshort]

I should've been using go 1.21 for building/testing this... hence the verify failures.

[tmshort]

/retest-required

[kuiwang02]

/hold

I am sorry to hold it because I need time to pre-verify it.
after it is done with ok, I will unhold it. thanks

[tmshort]

/retest-required

[kuiwang02]

@tmshort when I pre-verify it, I find the update is still mass operation although it is less than previous release.
I am fear that it still cause customer issue.
I update the ticket https://issues.redhat.com/browse/OCPBUGS-59647# with comment.
could you please check it? thanks

by the way, when I verify the issue on 4.17 (4.16 to 4.17), it has same behavior. but I do not find the behavior when I verify it on4.19.

[tmshort]

/retest-required

[tmshort]

/retest-required

[tmshort]

/retest-required

[openshift-ci]

@tmshort: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-gcp-olm	c2fb6b2	link	true	/test e2e-gcp-olm

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.