Separate conformance tests dependencies

[rikatz]

What type of PR is this?
/kind cleanup

What this PR does / why we need it:
Separate conformance tests dependencies from main tests, and allow generating a binary to run conformance on a different machine

Which issue(s) this PR fixes:
Fixes #3318

Does this PR introduce a user-facing change?:

Move conformance dependencies out of primary go.mod

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: rikatz
Once this PR has been reviewed and has the lgtm label, please assign aojea for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[kflynn]

This looks really good, thanks! One question from my end.

[robscott]

Thanks @rikatz, this is great!

[robscott]

What's left that still requires a controller-runtime dependency?

[rikatz]

the pkg/test uses envtest, which is part of controller-runtime.

Probably as part of the VAP tests effort we can split it as well, and get rid once for all of controller-runtime as an API dependency

[dprotaso]

I believe introducing a new go module into the repository will now require additional changes to the release process

See: https://go.dev/doc/modules/managing-source?utm_source=chatgpt.com#multiple-module-source

[dprotaso]

Technically you don't need this given you have the go.work folder

and allow generating a binary to run conformance on a different machine

Also having the replace will prevent go install and go run from working.

[rikatz]

what's the case of having "go install" or "go run" here as we don't have a "main"?

I agree that replacement may be problematic in case we try to do some "go install", but "go run" works fine:

go run ./tools/gepstoc/main.go 

the "tools" directory has its own go.mod already. Did I missed something?

[dprotaso]

Curious what this pseudo version means exactly?

In order for the build to work I think we'll need to have automation to update this module version

eg.

1. A commit makes a change to the sigs.k8s.io/gateway-api module
2. Automation then needs to bump the version in this module file

Unsure if we want to do this as part of every release (more error prone) or every commit (then you could build the conformance binary at HEAD). One caveat is we'd want to make sure this doesn't trigger an infinite loop.

[rikatz]

The pseudo version is related to the replacement we are doing, mostly saying "we don't care, use the latest from the replacement"

Similar to Kubernetes main repo: https://github.com/kubernetes/kubernetes/blob/master/go.mod#L88

As we are not building binaries, but APIs, it makes sense that we keep the "go.mod" of the API as clean as possible, with the "cost" IMO of the inner modules that we don't provide to consumers with these kind of tweaks.

[rikatz]

@dprotaso btw, as you are here, do you mind trying to build knative with the changes above? I may try doing it as well with knative and other implementations, mostly willing to see if something will break on someone's build env