fix: check ephemeral metadata is set before delete

[com6056]

We're seeing issues with rollout sync failing due to errors like this:

roCtx.reconcile err Operation cannot be fulfilled on pods \"REDACTED\": the object has been modified; please apply your changes to the latest version and try again"

It seems that we're trying to delete the metadata on every pod every sync, even if the pod doesn't have the keys. This means that in environments with high pod churn, this loop can get stuck for a very long time. This simply adds a check to make sure the key is actually present in the metadata before we delete it and mark it as modified.

Before the fix with the new test:

go test ./utils/replicaset/...
--- FAIL: TestSyncEphemeralPodMetadata (0.00s)
    canary_test.go:1353:
        	Error Trace:	/Users/jrodgers/argo-rollouts/utils/replicaset/canary_test.go:1353
        	Error:      	Should be false
        	Test:       	TestSyncEphemeralPodMetadata
time="2025-01-30T13:43:28-08:00" level=warning msg="Failed to determine existing ephemeral metadata from annotation: foo"
time="2025-01-30T13:43:28-08:00" level=info msg="ComputePodTemplateHash hash changed (new hash: 6c8776dbbb, old hash: fcb6c6ff9)" namespace=default rollout=red
time="2025-01-30T13:43:28-08:00" level=info msg="Pod template change detected (new: 8657b54cf, old: different-hash)" namespace=default rollout=nginx
time="2025-01-30T13:43:28-08:00" level=info msg="Canary steps change detected (new: 5ffbfbbd64, old: different-hash)" namespace=default rollout=nginx
time="2025-01-30T13:43:28-08:00" level=warning msg="Unable to convert ReplicaSet revision to int: strconv.Atoi: parsing \"not-an-int\": invalid syntax" ReplicaSet= namespace= rollout=ro
time="2025-01-30T13:43:28-08:00" level=warning msg="Unable to convert ReplicaSet revision to int: strconv.Atoi: parsing \"not-an-int\": invalid syntax" ReplicaSet= namespace= rollout=ro
FAIL
FAIL	github.com/argoproj/argo-rollouts/utils/replicaset	0.685s
FAIL

After the fix with the new test:

go test ./utils/replicaset/...
ok  	github.com/argoproj/argo-rollouts/utils/replicaset	(cached)

Checklist:

• Either (a) I've created an enhancement proposal and discussed it with the community, (b) this is a bug fix, or (c) this is a chore.
• The title of the PR is (a) conventional with a list of types and scopes found here, (b) states what changed, and (c) suffixes the related issues number. E.g. "fix(controller): Updates such and such. Fixes #1234".
• I've signed my commits with DCO
• I have written unit and/or e2e tests for my change. PRs without these are unlikely to be merged.
• My builds are green. Try syncing with master if they are not.
• My organization is added to USERS.md.

[github-actions]

Published E2E Test Results

  4 files    4 suites   3h 8m 36s ⏱️
113 tests 104 ✅  7 💤 2 ❌
454 runs  424 ✅ 28 💤 2 ❌

For more details on these failures, see this check.

Results for commit 68b5f6c.

♻️ This comment has been updated with latest results.

[github-actions]

Published Unit Test Results

2 291 tests   2 291 ✅  2m 59s ⏱️
  128 suites      0 💤
    1 files        0 ❌

Results for commit 68b5f6c.

♻️ This comment has been updated with latest results.

[sonarqubecloud]

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[com6056]

This same condition is being checked already right above this