CLOUDP-356376 Allow Search to work with x509 cluster auth #568

fealebenpae · 2025-11-03T18:41:42Z

Summary

With the switch to gRPC for mongot in #527 it's now possible for Search to be deployed against clusters that use x509 internal authentication. This removes the validation check for the internal cluster authentication mode and adds a new e2e test based on the existing search_enterprise_tls test that verifies existing functionality against a x509 cluster.

Proof of Work

New test.

Checklist

Have you linked a jira ticket and/or is the ticket in the title?
Have you checked whether your jira ticket required DOCSP changes?
Have you added changelog file?
- use skip-changelog label if not needed
- refer to Changelog files and Release Notes section in CONTRIBUTING.md for more details

github-actions · 2025-11-03T18:42:56Z

⚠️ (this preview might not be accurate if the PR is not rebased on current master branch)

MCK 1.6.0 Release Notes

New Features

MongoDBCommunity: Added support to configure custom cluster domain via newly introduced spec.clusterDomain resource field. If spec.clusterDomain is not set, environment variable CLUSTER_DOMAIN is used as cluster domain. If the environment variable CLUSTER_DOMAIN is also not set, operator falls back to cluster.local as default cluster domain.
Helm Chart: Introduced two new helm fields operator.podSecurityContext and operator.securityContext that can be used to configure securityContext for Operator deployment through Helm Chart.
MongoDBSearch: Switch to gRPC and mTLS for internal communication
Since MCK 1.4 the mongod and mongot processess communicated using the MongoDB Wire Protocol and used keyfile authentication. This release switches that to gRPC with mTLS authentication. gRPC will allow for load-balancing search queries against multiple mongot processes in the future, and mTLS decouples the internal cluster authentication mode and credentials among mongod processes from the connection to the mongot process. The Operator will automatically enable gRPC for existing and new workloads, and will enable mTLS authentication if both Database Server and MongoDBSearch resource are configured for TLS.
MongoDBSearch: MongoDB deployments using X509 internal cluster authentication are now supported. Previously MongoDB Search required SCRAM authentication among members of a MongoDB replica set. Note: SCRAM client authentication is still required, this change merely relaxes the requirements on internal cluster authentication.

Bug Fixes

Fixed parsing of the customEnvVars Helm value when values contain = characters.
ReplicaSet: Blocked disabling TLS and changing member count simultaneously. These operations must now be applied separately to prevent configuration inconsistencies.

Other Changes

Simplified MongoDB Search setup: Removed the custom Search Coordinator polyfill (a piece of compatibility code previously needed to add the required permissions), as MongoDB 8.2.0 and later now include the necessary permissions via the built-in searchCoordinator role.
kubectl-mongodb plugin: cosign, the signing tool that is used to sign kubectl-mongodb plugin binaries, has been updated to version 3.0.2. With this change, released binaries will be bundled with .bundle files containing both signature and certificate information. For more information on how to verify signatures using new cosign version please refer to -> https://github.com/sigstore/cosign/blob/v3.0.2/doc/cosign_verify-blob.md

fealebenpae · 2025-11-04T12:20:49Z

docker/mongodb-kubernetes-tests/tests/search/search_enterprise_x509_cluster_auth.py

+    resource["spec"]["security"]["authentication"] = {
+        "enabled": True,
+        "modes": ["X509", "SCRAM"],
+        "agents": {"mode": "X509"},
+        "internalCluster": "X509",
+    }


This is the only addition to the code already in the e2e_search_enterprise_tls test.

fealebenpae · 2025-11-04T12:21:46Z

docker/mongodb-kubernetes-tests/tests/search/search_enterprise_x509_cluster_auth.py

+    create_agent_tls_certs(issuer, namespace, mdb.name, "certs")
+    create_x509_mongodb_tls_certs(issuer, namespace, mdb.name, f"certs-{mdb.name}-clusterfile")
+    create_x509_mongodb_tls_certs(issuer, namespace, mdb.name, f"certs-{mdb.name}-cert", mdb.get_members())


The other change, necessary to populate the secrets expected by the MongoDB CR change above.

lsierant · 2025-11-04T12:39:10Z

changelog/20251103_feature_mongodbsearch_mongodb_deployments_using_x509.md

+date: 2025-11-03
+---
+
+* **MongoDBSearch**: MongoDB deployments using X509 internal cluster authentication are now supported. Previously MongoDB Search required SCRAM authentication among members of a MongoDB replica set.


I think it's worth mentioning that mongot itself still cannot authenticate using x509, so SCRAM auth must be enabled for clients to connect (but it might be both x509, but SCRAM is mandatory for search)

lsierant · 2025-11-04T12:45:05Z

docker/mongodb-kubernetes-tests/tests/search/search_enterprise_x509_cluster_auth.py

+        f"{mdbs.name}-search",
+        replicas=1,
+        service_name=search_service_name,
+        additional_domains=[f"{search_service_name}.{namespace}.svc.cluster.local"],


Is this service FQDN actually validated by mongod when connecting to mongot? It should, right?

Also when we switch to L7 our proxy component should probably present the same cert, or at least cert issued for the same service FQDN?

I'm actually not a 100% if mongod performs hostname validation. I'll comment this out and see what happens.

re. L7 proxies - the proxy will terminate the TLS connection from mongod, and establish its own TLS connection to the mongots it represents. It can use the same TLS certificate as mongot, or its own (though it too will have to be signed by the same CA). We'll sort this out separately with load balancing, however, and adjust as necessary. This is just a copy of the existing TLS test we have.

@lsierant it does seem that when mongod connects to mongot the hostname is validated against mongot's TLS certificate. Commenting out the additional_domains specification here causes the createSearchIndex command to fail with a Error connecting to Search Index Management service message.

vinilage

LGTM!

vinilage · 2025-11-05T15:04:30Z

changelog/20251103_feature_mongodbsearch_mongodb_deployments_using_x509.md

+---
+kind: feature
+date: 2025-11-03
+---
+
+* **MongoDBSearch**: MongoDB deployments using X509 internal cluster authentication are now supported. Previously MongoDB Search required SCRAM authentication among members of a MongoDB replica set. Note: SCRAM client authentication is still required, this change merely relaxes the requirements on internal cluster authentication.
+


MaciejKaras · 2025-11-06T08:47:48Z

controllers/searchcontroller/enterprise_search_source_test.go

 		},
-		// Internal cluster authentication tests
-		{
-			name:                "X509 internal cluster auth not supported",


shouldn't we keep this test case and just verify that we don't throw an error? As I understand there are three options for internalClusterAuth right now:

SCRAM

X5909

empty

Shouldn't we validate if received internalClusterAuth that is outside of the accepted values?

The existing e2e tests cover the SCRAM internal auth cases, and the new e2e test covers x509 so I thought since the Validate method no longer concerns itself with internal cluster auth at all those unit test cases are no longer relevant, but I agree - it costs nothing to have them still.

MaciejKaras

I proly don't understand completely this change, so let me ask some questions. Basically this change boils down to removing

if r.Spec.Security.GetInternalClusterAuthenticationMode() == util.X509 {
		return xerrors.New("MongoDBSearch does not support X.509 internal cluster authentication")
	}

...and in result allowing to provide x509 internal cluster authentication. Is that correct?

fealebenpae · 2025-11-06T08:56:05Z

@MaciejKaras exactly - not supporting x509 internal cluster authentication was a mongot limitation that was lifted with the change in #527, so this PR just removes our validation check for x509 auth and adds a test.

CLOUDP-356376 Allow Search to work with x509 cluster auth

0f77ed9

fealebenpae marked this pull request as ready for review November 4, 2025 08:46

fealebenpae requested review from a team and vinilage as code owners November 4, 2025 08:46

fealebenpae requested review from MaciejKaras and viveksinghggits November 4, 2025 08:46

fealebenpae marked this pull request as draft November 4, 2025 08:46

fealebenpae and others added 2 commits November 4, 2025 10:47

Merge branch 'master' into fealebenpae/search-x509-cluster-auth

046af5a

remove test added to the wrong task group

84950c1

fealebenpae marked this pull request as ready for review November 4, 2025 12:18

fealebenpae commented Nov 4, 2025

View reviewed changes

lsierant reviewed Nov 4, 2025

View reviewed changes

lsierant approved these changes Nov 4, 2025

View reviewed changes

changelog

612a4d2

vinilage approved these changes Nov 5, 2025

View reviewed changes

MaciejKaras reviewed Nov 6, 2025

View reviewed changes

MaciejKaras approved these changes Nov 6, 2025

View reviewed changes

fealebenpae and others added 2 commits November 6, 2025 11:37

restore unit tests

6998d9f

Merge branch 'master' into fealebenpae/search-x509-cluster-auth

ff0c564

fealebenpae merged commit 2cf14d4 into master Nov 6, 2025
34 of 37 checks passed

fealebenpae deleted the fealebenpae/search-x509-cluster-auth branch November 6, 2025 13:01

CLOUDP-356376 Allow Search to work with x509 cluster auth #568

CLOUDP-356376 Allow Search to work with x509 cluster auth #568

Uh oh!

Conversation

fealebenpae commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Proof of Work

Checklist

Uh oh!

github-actions bot commented Nov 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

MCK 1.6.0 Release Notes

New Features

Bug Fixes

Other Changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vinilage left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaciejKaras Nov 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaciejKaras left a comment

Choose a reason for hiding this comment

Uh oh!

fealebenpae commented Nov 6, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

fealebenpae commented Nov 3, 2025 •

edited

Loading

github-actions bot commented Nov 3, 2025 •

edited

Loading

MaciejKaras Nov 6, 2025 •

edited

Loading