Issue 3138 - Conformance Tests for BackendTLSPolicy - normative #3212

candita · 2024-07-23T20:36:20Z

What type of PR is this?

/kind test
/area conformance

What this PR does / why we need it:

Add a normative test of Gateway API BackendTLSPolicy implementations.

Which issue(s) this PR fixes:
Fixes #3138

Does this PR introduce a user-facing change?:

NONE

k8s-ci-robot · 2024-07-23T20:36:22Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

conformance/echo-basic/echo-basic.go

k8s-ci-robot · 2024-08-13T00:38:37Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: candita
Once this PR has been reviewed and has the lgtm label, please assign mlavacca for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

candita · 2024-08-13T20:18:30Z

/test pull-gateway-api-verify

candita · 2024-08-14T20:02:18Z

/test pull-gateway-api-verify

candita · 2024-08-19T17:16:41Z

/test pull-gateway-api-verify

candita · 2024-08-19T18:51:48Z

/test pull-gateway-api-verify

candita · 2024-08-19T19:09:29Z

/test pull-gateway-api-verify

candita · 2024-08-19T19:56:40Z

/test pull-gateway-api-verify

candita · 2024-08-19T20:36:48Z

/test pull-gateway-api-verify

candita · 2024-08-19T22:33:00Z

/test pull-gateway-api-verify

candita · 2024-08-19T23:16:00Z

/test pull-gateway-api-test

youngnick

This LGTM, but I think there will be a conflict with the work @mlavacca is doing in #3287. That's still draft, so this is fine, but wanted to make sure the RPs were linked.

conformance/tests/backendtlspolicy-normative.go

BackendTLSPolicy.

…e original as well. See https://github.com/kubernetes-sigs/gateway-api/pull/2745/files#diff-cf5b4a9b433acc91f8c7cc2dc802e29aa712c8d820887742f3bc5ae45b7a9d0fR24-R28

howardjohn · 2024-08-28T22:09:01Z

conformance/echo-basic/echo-basic.go

+	// If the request was made to URI backendTLS, then get the server name indication and
+	// add it to the RequestAssertions.  It will be echoed back later.
+	if strings.Contains(r.RequestURI, "backendTLS") {
+		sni, err = sniffForSNI(r.RemoteAddr)


I don't get how this works. Are we supposed to send an HTTP request and then send a TLS request later?

If we are just serving this as TLS already you can just do r.TLS.ServerName I think

The idea is that you send an HTTP or HTTPS request and the BackendTLSPolicy adds what is needed for the backend request to succeed (certificate and SNI). The echo service finds the SNI and echoes it back to the test to prove the BackendTLSPolicy was used.

I still need to dig into this a bit deeper, but I think #3212 (comment) is relevant to this question.

The logic seems to be: on an HTTP request, on a new TCP listener, accept a new connection, get the TLS handshake data, then close the connection.

Should it not be: accept an HTTPS request and report the SNI?

conformance/base/manifests.yaml - fix yaml conformance/tests/backendtlspolicy.yaml - fix yaml conformance/tests/tlsroute-simple-same-namespace.go - rename cert for sharing conformance/utils/suite/conformance.go - fix a bug in cleanup-base-resources flag application conformance/utils/suite/suite.go - rename cert for sharing

shaneutt

Looking good! Thanks @candita!!! Mostly minor comments.

Also I was digging into the test details a bit when I got pulled away from a meeting so I still need to come back and do another round of review here, but please check my comments and LMKWYT 👍

shaneutt · 2024-08-30T12:19:51Z

conformance/base/manifests.yaml

+      volumes:
+      - name: secret-volume
+        secret:
+          secretName: backend-tls-checks-certificate


When we have a reference to a resource in our test manifests and the referenced resource is dynamically generated, it can be helpful to leave future code readers a breadcrumb about where/how that gets created:

Suggested change

secretName: backend-tls-checks-certificate

# this secret is generated dynamically by the test suite

secretName: backend-tls-checks-certificate

Super minor thing of course, but not harmful either.

shaneutt · 2024-08-30T12:21:47Z

conformance/echo-basic/echo-basic.go

 	"net/http"
 	"os"
 	"regexp"
 	"strconv"
 	"strings"
 	"time"

+	"github.com/paultag/sniff/parser"
+


I thought we had a linter that would actually ensure we had three groups (one for stdlib, one for external, one for local), but maybe not?

Suggested change

shaneutt · 2024-08-30T12:24:39Z

conformance/echo-basic/echo-basic.go

+	PeerCertificates []string `json:"peerCertificates,omitempty"`
+	// ServerName is the SNI.
+	ServerName         string `json:"serverName"`


Might it be better just to spell the whole thing out?

Suggested change

PeerCertificates []string `json:"peerCertificates,omitempty"`

// ServerName is the SNI.

ServerName string `json:"serverName"`

PeerCertificates []string `json:"peerCertificates,omitempty"`

ServerNameIndication string `json:"serverName"`

WDYT?

IMNSHO, "SNI" is a term of art for a particular thing within the TLS while "ServerNameIndication" could be that or could be some other random thing. I'd leave it as "ServerName" and change the comment to

// ServerName is the name sent from the peer using SNI

or some such.

And with that I shall try to stop bikeshedding... 😉

I'm fine with Flynn's suggestion 👍

shaneutt · 2024-08-30T12:25:43Z

conformance/tests/backendtlspolicy.yaml

+    caCertificateRefs:
+    - group: ""
+      kind: Secret
+      name: "backend-tls-checks-certificate"


Similar to above, a breadcrumb comment can't hurt much:

Suggested change

name: "backend-tls-checks-certificate"

# this secret is generated dynamically by the test suite

name: "backend-tls-checks-certificate"

shaneutt · 2024-08-30T12:29:21Z

conformance/utils/http/http.go

+	// SNI is the server name indication seen by the backend.
+	SNI string


My personal preference would be to spell it out:

Suggested change

// SNI is the server name indication seen by the backend.

SNI string

// ServerNameIndication (SNI) indicates which hostname the client is attempting to connect to.

ServerNameIndication string

However if you feel very strongly otherwise I don't consider this blocking: consider it resolved at your discretion.

OK OK OK, I guess one more. I'd probably have this be ServerName too. 🤷‍♂️

I don't feel strongly, I'm fine to go with Flynn's suggestion

shaneutt · 2024-08-30T12:38:29Z

conformance/utils/kubernetes/certificate.go

+		return fmt.Errorf("failed creating key: %w", err)
+	}
+	return nil
+}


I know it's sometimes hard to justify the value, but ideally we should add some unit tests for these new functions. If nothing else, because it's a good habit.

shaneutt · 2024-08-30T12:38:55Z

conformance/utils/suite/conformance.go

-		suite.Applier.MustApplyWithCleanup(t, suite.Client, suite.TimeoutConfig, manifestLocation, true)
+		suite.Applier.MustApplyWithCleanup(t, suite.Client, suite.TimeoutConfig, manifestLocation, suite.Cleanup)


Oh did we just have this available but never honored it here this whole time? 😂

Thanks for noticing and fixing that 👍

shaneutt · 2024-08-30T12:40:01Z

pkg/features/features.go

+// ExperimentalPolicyFeatures includes all the supported features for experimental policies.
+var ExperimentalPolicyFeatures = sets.New(


A minor thing, but we sort of have a precedent for naming these which is <API/Topic><Conformance>Features (e.g. HTTPRouteExperimentalFeatures), so maybe:

Suggested change

// ExperimentalPolicyFeatures includes all the supported features for experimental policies.

var ExperimentalPolicyFeatures = sets.New(

// PolicyExperimentalFeatures includes all the supported features for experimental policies.

var PolicyExperimentalFeatures = sets.New(

Might be preferable for consistency?

This set will likely be replaced by #3287, but I think it is not a sufficient reason to block this PR. In case this will be merged first, I'll adapt #3287 accordingly.

shaneutt · 2024-08-30T12:52:20Z

conformance/tests/backendtlspolicy.go

+					Response:  http.Response{StatusCode: 200},
+					Backend:   "infra-backend-v1",
+					Namespace: "gateway-conformance-infra",
+					SNI:       "abc.example.com",


We're copying this string literal in several places, and it seems like we're tying it to test cases where TLS is in play. Perhaps we should

make the hostname something more explicit, like perhaps including "gateway api" and "tls testing" or something?

use a constant to keep track and use it in the multiple places it's being referred to right now

shaneutt · 2024-08-30T13:13:40Z

conformance/tests/backendtlspolicy.go

+					Namespace: "gateway-conformance-infra",
+					SNI:       "abc.example.com",
+				})
+		})


Digging into the MakeRequestAndExpectEventuallyConsistentResponse code one thing I noticed is that it appears that is always sends the http:// scheme?

Meaning: we're sending requests to the Gateway in the clear, and then proving that the Gateway added the correct backend TLS configuration on the backend and then served the response back out in the clear.

So (if I'm not misunderstanding anything) that's a perfectly valid test but to me it kind of begs the question: just for completeness should we not also test when the frontend connection has a distinct TLS configuration and verify the fingerprint on the clientside, and then verify the different fingerprint on the server side just to to thoroughly prove that the implementations are working properly?

Note that when I dug into this I was in a bit of a rush because I had a meeting about to start, so it's possible I'm misunderstanding, but take a look and LMKWYT and I'll come back and take another look at this myself soon 🤔

Yes, I agree, except we have a separate test for frontend TLS in TLSRouteSimpleSameNamespace This PR adds a test just for backend TLS. Adding a test for both frontend and backend TLS is needed.

There are a couple of subtests that are missing. I'd like to get just one merged before following up with the others. This is my to do list:

add a test case where the backend tls validation is misconfigured and ensure it returns an error

add a test case where frontend connection is also TLS

add a test case where frontend connection is also TLS and misconfigured

Ok cool, thanks for the heads up. Those test cases sound good, if you think it's feasible I might just add the point that we should test the fingerprints at each level as this helps to ensure more definitively everything is working exactly as expected.

I'm very much down with getting one valuable test in for now and follow up with the others. I would just ask if they're not already, get those test cases laid out in a follow-up issue somewhere, and then please feel free to consider this comment resolved at your discretion. 👍

k8s-ci-robot · 2024-09-04T02:26:44Z

PR needs rebase.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · 2024-09-10T20:23:49Z

@candita: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-gateway-api-crds-validation-1	`63b3c23`	link	true	`/test pull-gateway-api-crds-validation-1`
pull-gateway-api-crds-validation-2	`63b3c23`	link	true	`/test pull-gateway-api-crds-validation-2`
pull-gateway-api-crds-validation-3	`63b3c23`	link	true	`/test pull-gateway-api-crds-validation-3`
pull-gateway-api-crds-validation-4	`63b3c23`	link	true	`/test pull-gateway-api-crds-validation-4`
pull-gateway-api-crds-validation-5	`63b3c23`	link	true	`/test pull-gateway-api-crds-validation-5`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

k8s-ci-robot requested review from michaelbeaumont and youngnick July 23, 2024 20:36

whitneygriffith reviewed Aug 1, 2024

View reviewed changes

conformance/echo-basic/echo-basic.go Outdated Show resolved Hide resolved

candita force-pushed the issue3138-BackendTLSPolicy-echoserver branch from 76c8e10 to 6d9ab9e Compare August 13, 2024 00:38

candita force-pushed the issue3138-BackendTLSPolicy-echoserver branch from 0ec34a8 to 75551a0 Compare August 19, 2024 17:12

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 19, 2024

candita force-pushed the issue3138-BackendTLSPolicy-echoserver branch from 75551a0 to 7626aaa Compare August 19, 2024 18:48

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Aug 19, 2024

candita force-pushed the issue3138-BackendTLSPolicy-echoserver branch from 7626aaa to 1bc71f0 Compare August 19, 2024 19:09

candita force-pushed the issue3138-BackendTLSPolicy-echoserver branch from 1bc71f0 to 99e7eac Compare August 19, 2024 19:55

candita force-pushed the issue3138-BackendTLSPolicy-echoserver branch from 99e7eac to b774245 Compare August 19, 2024 20:35

candita force-pushed the issue3138-BackendTLSPolicy-echoserver branch from b774245 to 91488aa Compare August 19, 2024 22:30

candita marked this pull request as ready for review August 20, 2024 00:28

k8s-ci-robot requested review from kflynn and robscott August 20, 2024 00:28

candita changed the title ~~WIP Issue 3138 - Conformance Tests for BackendTLSPolicy - normative~~ Issue 3138 - Conformance Tests for BackendTLSPolicy - normative Aug 20, 2024

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 20, 2024

youngnick reviewed Aug 22, 2024

View reviewed changes

conformance/tests/backendtlspolicy-normative.go Outdated Show resolved Hide resolved

This was referenced Aug 22, 2024

Conformance: Signal Stability of Tests #3009

Closed

feat: features metadata #3287

Merged

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 24, 2024

candita force-pushed the issue3138-BackendTLSPolicy-echoserver branch from 2dc68d8 to b8ea586 Compare August 27, 2024 01:46

candita added 3 commits August 26, 2024 21:58

Issue 3138 - add normative conformance test for

7ef7d0f

BackendTLSPolicy.

Fix where the new go module needed by echo-basic resides, but keep th…

c2a5a2a

…e original as well. See https://github.com/kubernetes-sigs/gateway-api/pull/2745/files#diff-cf5b4a9b433acc91f8c7cc2dc802e29aa712c8d820887742f3bc5ae45b7a9d0fR24-R28

Rebase and make requested updates.

c2bb7ec

candita force-pushed the issue3138-BackendTLSPolicy-echoserver branch from b8ea586 to c2bb7ec Compare August 27, 2024 01:59

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Aug 27, 2024

shaneutt self-requested a review August 27, 2024 15:22

howardjohn reviewed Aug 28, 2024

View reviewed changes

shaneutt requested review from howardjohn, youngnick and whitneygriffith August 29, 2024 13:16

candita force-pushed the issue3138-BackendTLSPolicy-echoserver branch from 0ad1310 to 63b3c23 Compare August 29, 2024 17:07

shaneutt reviewed Aug 30, 2024

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 4, 2024

shaneutt added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Sep 18, 2024

shaneutt added this to the v1.3.0 milestone Sep 18, 2024

	secretName: backend-tls-checks-certificate
	# this secret is generated dynamically by the test suite
	secretName: backend-tls-checks-certificate

	name: "backend-tls-checks-certificate"
	# this secret is generated dynamically by the test suite
	name: "backend-tls-checks-certificate"

		// SNI is the server name indication seen by the backend.
		SNI string

		suite.Applier.MustApplyWithCleanup(t, suite.Client, suite.TimeoutConfig, manifestLocation, true)
		suite.Applier.MustApplyWithCleanup(t, suite.Client, suite.TimeoutConfig, manifestLocation, suite.Cleanup)

		// ExperimentalPolicyFeatures includes all the supported features for experimental policies.
		var ExperimentalPolicyFeatures = sets.New(

Issue 3138 - Conformance Tests for BackendTLSPolicy - normative #3212

Are you sure you want to change the base?

Issue 3138 - Conformance Tests for BackendTLSPolicy - normative #3212

Conversation

candita commented Jul 23, 2024

k8s-ci-robot commented Jul 23, 2024

k8s-ci-robot commented Aug 13, 2024

candita commented Aug 13, 2024

candita commented Aug 14, 2024

candita commented Aug 19, 2024

candita commented Aug 19, 2024

candita commented Aug 19, 2024

candita commented Aug 19, 2024

candita commented Aug 19, 2024

candita commented Aug 19, 2024

candita commented Aug 19, 2024

youngnick left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

candita Aug 28, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shaneutt left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

shaneutt Aug 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

k8s-ci-robot commented Sep 4, 2024

k8s-ci-robot commented Sep 10, 2024

candita Aug 28, 2024 •

edited

Loading

shaneutt Aug 30, 2024 •

edited

Loading