Fix istio cert rotation bug for issue #4744

[jonathanelbailey]

Description

This PR fixes a known issue when Emissary is configured to connect to an Istio mTLS network. When Emissary's Istio sidecar initiates a cert rotation the istio-certs secret generates a cache entry, but does not generate a delta since the istio-certs cache entry does not map to an actual resource on the Kubernetes cluster. This can eventually resolve itself if a delta is generated that can initiate a change, but other times it results in an unrecoverable error that can cause traffic disruption without custom health checks.

Related Issues

#4744

Testing

• Replicated the issue using recommended steps from Emissary Ingress - does not reread Istio certificates after a few days #4744 (comment)
• Using this test image that includes the fix, the test case could no longer be replicated
• Tested on Istio 1.23.2
• Added parametrized unit tests for improved coverage of this test case

Checklist

• Does my change need to be backported to a previous release?

  • What backport versions were discussed with the Maintainers in the Issue?

• I made sure to update CHANGELOG.md.

  Remember, the CHANGELOG needs to mention:

  • Any new features
  • Any changes to our included version of Envoy
  • Any non-backward-compatible changes
  • Any deprecations

• This is unlikely to impact how Ambassador performs at scale.

  Remember, things that might have an impact at scale include:

  • Any significant changes in memory use that might require adjusting the memory limits
  • Any significant changes in CPU use that might require adjusting the CPU limits
  • Anything that might change how many replicas users should use
  • Changes that impact data-plane latency/scalability

• My change is adequately tested.

  Remember when considering testing:

  • Your change needs to be specifically covered by tests.
    • Tests need to cover all the states where your change is relevant: for example, if you add a behavior that can be enabled or disabled, you'll need tests that cover the enabled case and tests that cover the disabled case. It's not sufficient just to test with the behavior enabled.
  • You also need to make sure that the entire area being changed has adequate test coverage.
    • If existing tests don't actually cover the entire area being changed, add tests.
    • This applies even for aspects of the area that you're not changing – check the test coverage, and improve it if needed!
  • We should lean on the bulk of code being covered by unit tests, but...
  • ... an end-to-end test should cover the integration points

• I updated CONTRIBUTING.md with any special dev tricks I had to use to work on this code efficiently.

• The changes in this PR have been reviewed for security concerns and adherence to security best practices.

[kflynn]

Hey @jonathanelbailey, are you on the CNCF Slack by any chance? I'd like to chat about this one... 🙂