Add steps for running conformance tests on Envoy Gateway

[arkodg]

/kind documentation
/kind test
/area conformance

What this PR does / why we need it:

• Adds steps showing how to run conformance tests on Envoy Gateway
• This should help
  • Increase velocity of conformance test contributions since tests can be easier to validate
  • Give more confidence to conformance test PR reviewers by knowing the outcome of the test result on an implementation
• The current documentation on https://gateway-api.sigs.k8s.io/concepts/conformance/#3-conformance-tests does not seem adequate enough for conformance tests contributors to validate their changes on implementations

[k8s-ci-robot]

Adding the "do-not-merge/release-note-label-needed" label because no release-note block was detected, please follow our release note process to remove it.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: arkodg
Once this PR has been reviewed and has the lgtm label, please assign shaneutt for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[arkodg]

ptal @robscott

[robscott]

Thanks @arkodg!

[robscott]

I'm torn on this script. It feels like it should either exist in Envoy Gateway or in a more generic place within GW API. If it were within GW API, it seems like it could just be a generic script for creating a kind cluster with metal LB for the sake of testing with ~any in-cluster implementation of Gateway API.

[arkodg]

ah would you prefer a curl / wget that fetches this script and runs it ?

yah its a generic script which can be used for any implementation - should i move it under hack/implementations/common ?

[youngnick]

I think that a curl or wget that fetches this script from EG is probably better for now.

It might be good to have something similar in the Gateway API repo, but that's probably a larger discussion that we shouldn't block the rest of this PR for.

[mlavacca]

We already have a place where such a piece of documentation should be put: https://github.com/kubernetes-sigs/gateway-api/tree/main/conformance/reports/v1.1.0/envoy-gateway. I guess that running CONFORMANCE_REPORT_PATH=conformance-report-k8s.yaml make experimental-conformance reproduces the same result as here (and if not, I think we should update the reports README instead of putting this script along with README here). I lean toward leaving this kind of script out of the Gateway API repo

[arkodg]

over the past years, Ive seen many contributors find it hard to test their PRs associated with adding/modifying conformance tests, This was an attempt to make it easier for a contributor to say Hey I tested this Conformance Test PR on these X platforms by following the steps in ..., thereby making it easier for new contributors to come in and pick up conformance issues, contributions seems to be on the decline rn.

The steps in the conformance tests are not sufficient.

Happy to close this PR if the maintainers prefer that

[shaneutt]

README's for our reporting implementations seem entirely fine (and quite helpful for users), but I don't want to set a precedent for implementation-specific executable code as something that we host here in the upstream (and if that precedent has already been set somehow, we should re-look at that).

[robscott]

but I don't want to set a precedent for implementation-specific executable code as something that we host here in the upstream

Mostly agree with this reasoning, but I think this specific script would be useful for ~any implementation that could be deployed inside a cluster. I think I'd be fine with either bumping this script up to a more generic place that could be used to test with any in-cluster implementation, or just updating the directions to describe how to run conformance tests with Envoy Gateway by pulling this script in from a remote location.

I'd also agree that we shouldn't open this broader idea to just any implementation, maybe we can limit it to implementations that have a solid track record of delivering passing conformance reports?

[shaneutt]

I can be persuaded, putting it in a general place and so forth would certainly be preferable.

[arkodg]

ive moved the create-cluster.sh to hack/implementations/common so any implementation can peruse it

[robscott]

/retest

[shaneutt]

Strong preference to not host the script here, but the README to explain how the tests are run for this are very appreciated, thank you for taking the time to do that.

unblock