KEP-4317: (Pod Certificates) Update for beta #5565

ahmedtd · 2025-09-24T02:10:26Z

Update KEP-4317 for beta.

Concrete changes:

Add a config field to PodCertificateRequests and PodCertificateProjection as discussed in SIG-Auth.
Specify metrics that should be exported by kube-apiserver and kubelet.
Answer other beta questions in the KEP template

Issue link: Pod Certificates #4317

keps/sig-auth/4317-pod-certificates/README.md

stlaz · 2025-09-25T11:42:54Z

keps/sig-auth/4317-pod-certificates/README.md

+Several example signers built on the alpha feature set are available in the
+[mesh-examples repository](https://github.com/ahmedtd/mesh-example).


I wonder if we should just add this to a new story, something like this:

#### Story 2 I am a cluster owner. I would like to run a service in mu cluster and force it to use mTLS for server/client authentication. I expect other applications in my cluster to be communicating with my service. I want the cluster to provide a simple, yet secure way for these applications to request client certificates that they can use to authenticate to the new service. Several example signers built on the alpha feature set are available in the [mesh-examples repository](https://github.com/ahmedtd/mesh-example).

I would say if we would be able to reference signers already pre-existing would be possible easier for people to adopt it and give examples on writing their own if required

stlaz · 2025-09-25T11:44:32Z

keps/sig-auth/4317-pod-certificates/README.md

+* Add e2e test to exercise the feature end-to-end
+* Implement out-of-tree signers for several scenarios to prove that the feature
+  is sufficient for real use cases.  Accomplished at
+  [mesh-examples](https://github.com/ahmedtd/mesh-example).


should we have something in sigs.k8s.io eventually?

I think that makes sense, but we would probably have to commit to maintaining them until they are upstreamed.

I would expect some of these to land in k/k in KCM eventually.

Yeah, I think the main question, is do we want to ship a SIG Auth project of some demo signers, that might eventually get specced out in a KEP and moved into k/k?

My preference is to skip that step and go straight into KCM. TLS (and potentially mTLS) for Kube services is something I want all clusters to have.

keps/sig-auth/4317-pod-certificates/README.md

stlaz · 2025-09-25T11:51:20Z

keps/sig-auth/4317-pod-certificates/README.md

 If support for the new key type is introduced in version N, then we have the
 following skew cases:


What about API version skews?

Can you elaborate? The cases I'm discussing are about which version of the Kubernetes API each component was built against.

stlaz · 2025-09-25T11:53:56Z

keps/sig-auth/4317-pod-certificates/README.md

+4. Disable the PodCertificaterequest feature gate.
+5. Disable the certificates/v1beta1 API.


Wrong item numbering :)

Is there a specific order of components in which to disable the FG? If Disable the PodCertificaterequest feature gate implies kubelet, it should be explicitly mentioned.

stlaz · 2025-09-25T11:58:54Z

keps/sig-auth/4317-pod-certificates/README.md


 ###### How can a rollout or rollback fail? Can it impact already running workloads?

+During rollout (enabling the certificates/v1beta1 API and PodCertificate feature


I don't remember this particular detail from the implementation - is it possible to do these enablements in parallel? Won't a missing API cause for the feature to be still considered disabled on the kubelet side/fail initialization of a PCR watch?

Hmmm, you're right, I think we need the feature gate to be enabled on all kube-apiserver replicas before it is enabled on any kubelet instance. I've updated the rollout/rollback steps to document this, and updated this section.

keps/sig-auth/4317-pod-certificates/README.md

stlaz · 2025-09-25T12:16:55Z

keps/sig-auth/4317-pod-certificates/README.md

 -->

+`kubelet` will report metrics that show the feature is in use:
+* `podcertificate_time_to_expiration` A gauge vector, faceted on namespace, pod


You cannot use metrics labels with unbounded dimensions, such as names. This is to prevent DoS-ing your metrics scraper as this can easily cause memory exhaustion.

Applies to all suggested metrics.

My thinking here is:

There is a fixed upper limit on the number of pods served by each kubelet, and thus fixed upper limit on the pod namespace and pod name.

Each pod probably only mounts a few certificates, and thus signer name is bounded (though not explicitly).

This could be problematic for the kube-apiserver metrics, though. I'll check and see if there are any that include namespace / pod name today.

Some reference metrics:

https://kubernetes.io/docs/reference/instrumentation/metrics/#:~:text=kube_pod_resource_limit --- exported by the scheduler, faceted on namespace and pod name

apiserver_certificates_registry_csr_requested_duration_total --- exported by kube-apiserver, faceted on signerName, but with the caveat that only kubernetes.io signers are specifically reported.

These don't seem consistent to me --- there are potentially thousands of namespaces and tens of thousands of pods within a cluster, and (likely) ten or fewer signer names.

Reading through the code change in kubernetes/kubernetes#102523, it seems like the real problems was that Prometheus counter vectors by default continue to hang on to facets, even if they are no longer getting data. We can fix that with

I think we might need some input from SIG Instrumentation about how to handle this case. The metrics instrumentation guide says this:

Notable exceptions are exporters like kube-state-metrics, which expose per-pod or per-deployment metrics, which are theoretically unbound over time as one could constantly create new ones, with new names. However, they have a reasonable upper bound for a given size of infrastructure they refer to and its typical frequency of changes.

In general, “external” labels like pod name, node name (any object name), & namespace do not belong in the instrumentation itself (the exception being kube-state-metrics). They are to be attached to metrics by the collecting system that has the external knowledge (blog post).

I'm not quite sure how to actually take action on that guidance. It seems clear that some of these metrics I am proposing should actually be exported by kube-state-metrics, though.

I think we might need some input from SIG Instrumentation about how to handle this case.

That would be best.

Isnt this state management of the metric - more reason to have the metrics in kube-state-metrics (provided the data is readily available in some API) which will do the state management implicitly?

That is correct, but won't the information be available in the PodCertificateRequest status?

These get vacuumed very quickly, and given a PCR, you can generally only tell which pod it is related to, not which volume in the pod. That might be OK, I guess.

But I think to make diagnosis of issues via kube-state-metrics more bulletproof, we would want the pod status to have a field that maps volumes to their current refresh and expiration times.

kube-state-metrics could then export gauges showing time left until refresh and expiration for each volume of each pod.

long thread. I'm everyone has commented on cardinality. I share the concern. Perhaps it's more useful to know a count of how many successful/failed refreshes by signer and how many expired/current certs by signer

OK, I've cut the metrics down to one kubelet metric, reporting a gauge of currently-tracked pod certificate projections, bucketing them by signer name and the current state of the certificate. I think this strikes a good balance between cardinality and ability to pinpoint problems.

The new metric looks good to me 👍

stlaz · 2025-09-25T12:26:36Z

keps/sig-auth/4317-pod-certificates/README.md

+
+`kube-apiserver` will report the following metrics to help operators assess the
+health of signer implementations:
+* `podcertificate_processed_latencies`: A histogram vector, faceted on signer name,


I suspect we don't expect there would be too many signers, or that there would be a great turnover so I guess the signer name is probably ok for these metrics.

I would be keen to say this might not be the case. In a real case scenarios we could have a secure environments with KArpenter to scale out our workloads for example and further to that various signers depends on what their function would be or maybe different CAs.

kannon92 · 2025-09-28T17:46:54Z

Please request a PRR review since you are promoting to beta.

SergeyKanzhelev · 2025-10-08T16:55:58Z

keps/sig-auth/4317-pod-certificates/README.md

+  //
+  // Keys should be namespaced, in the form `<domainName>/<keyname>`.  Signers
+  // should document the keys and values they support.  Signers should deny
+  // requests that contain keys they do not recognize.


what would be length and characters limits here? Do we ever expect very long keys or values? If we want to start with restricting unicode command characters, we need to state it early - it will be hard to restrict later.

I think we would do validation on the total size of the field, as well as copying whatever validation is applied to metadata annotations.

This is related to https://github.com/kubernetes/enhancements/pull/5565/files#r2417192704.

Please specify the limitations in spec. In metadata key is not allowing unicode and value allows any characters. If keys are expected to be ascii (no internationalization needed) and values allow the control characters, it is fine. It just can lead to discussions like kubernetes/kubernetes#132104 in future

I've updated the spec to mention that the field is subject to the same validations as the metadata.annotations field, and include a summary description of that validation.

ahmedtd · 2025-10-08T17:42:16Z

Please request a PRR review since you are promoting to beta.

I think I have done this by including the PRR file in the PR.

kannon92 · 2025-10-09T15:42:38Z

keps/sig-auth/4317-pod-certificates/README.md

  * Node UID
 * A requested maximum duration for the certificate lifetime.
 * The signer name from which a certificate is being requested.
+* A free-form `map[string]string` to allow pod authors to communicate additional


AFAIK kubernetes api conventions discourage the use of maps in the API.

https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/api-conventions.md#lists-of-named-subobjects-preferred-over-maps

Is this just suggesting exposing values to metadata on the pod?

For this I just copied the structure of annotations. But a list of kv-pairs would also be a valid approach.

I'm going to keep this as the same structure as annotations, since that lets us benefit from the annotation validation.

kannon92

PRR shadow:

This is not my area of expertise but I find the PRR very well thought out.

I have some questions on the use of a map[string]string in userConfig but I'll leave that to sigs to decide.

/assign @deads2k

deads2k

Some comments, nothing serious but the metrics need to be sorted out. PRR lgtm too.

deads2k · 2025-10-09T19:49:50Z

keps/sig-auth/4317-pod-certificates/README.md

+  // Keys should be namespaced, in the form `<domainName>/<keyname>`.  Signers
+  // should document the keys and values they support.  Signers should deny
+  // requests that contain keys they do not recognize.


Clarify that kubernetes does verify or restrict these key/value pairs in any way.

deads2k · 2025-10-09T19:52:10Z

keps/sig-auth/4317-pod-certificates/README.md

+  // Keys must be namespaced in the form `<domainName>/<keyName`>.  Signer
+  // implementations should document what UserConfig keys they support. Signer
+  // implementations should reject PodCertificateRequests that contain user config that
+  // they do not recognize with a condition of type "Denied" and reason "InvalidUserConfig".


same doc here about kubernetes not providing any guarantees about these name/value pairs.

deads2k · 2025-10-09T19:59:32Z

keps/sig-auth/4317-pod-certificates/README.md

+  - serving certificates
+  - issuing client certificates


yes to both please.

These are both implemented in my demo repo

deads2k · 2025-10-09T19:59:45Z

keps/sig-auth/4317-pod-certificates/README.md

+  is sufficient for real use cases. These scenarios should include issuing of:
+  - serving certificates
+  - issuing client certificates
+  - SPIFFE certificates (?)


@enj opinion on requiring?

We should certainly make it possible, so that projects like Istio and Cilium can leverage it, if they want to. I don't think anything regarding SPIFFE should be included directly in KCM in the future though.

I have implemented basic SPIFFE cert support in my demo repo. If we decide to move some of them into a sig-auth repo, we don't need to move that one.

deads2k · 2025-10-09T20:00:03Z

keps/sig-auth/4317-pod-certificates/README.md

+  - SPIFFE certificates (?)
+  [mesh-examples](https://github.com/ahmedtd/mesh-example).
+* Migrate kubelet implementation to use the beta API.
+* Deprecate and remove the alpha API.


we don't require removal for beta.

We're going to remove it just to keep life simple, I think.

deads2k · 2025-10-09T20:06:24Z

keps/sig-auth/4317-pod-certificates/README.md

 -->

+`kubelet` will report metrics that show the feature is in use:
+* `podcertificate_time_to_expiration` A gauge vector, faceted on namespace, pod


long thread. I'm everyone has commented on cardinality. I share the concern. Perhaps it's more useful to know a count of how many successful/failed refreshes by signer and how many expired/current certs by signer

ahmedtd · 2025-10-14T06:22:11Z

Whoops, I'm not sure how I closed this.

deads2k · 2025-10-14T15:23:12Z

PRR lgtm. Changes lgtm. I'm happy with metrics separated by signer name.

/approve
/hold for @enj

enj

First pass.

enj · 2025-10-14T15:05:07Z

keps/sig-auth/4317-pod-certificates/README.md

+  //
+  // Signers should document the keys and values they support.  Signers should
+  // deny requests that contain keys they do not recognize.
+  UserAnnotations map[string]string `json:"userAnnotations,omitempty" protobuf:"bytes,11,opt,name=userAnnotations"`


Please add something like untrusted or unverified to the name of the field.

OK, I've added Unverified in front of the field in PodCertificateRequestSpec. I've left the PodCertificateProjection field as "userAnnotations", since it seems weird to make the user write out that their requests are unverified.

enj · 2025-10-14T15:07:22Z

keps/sig-auth/4317-pod-certificates/README.md

+  //
+  // Entries are subject to the same validation as object metadata annotations.
+  // Each key must be valid label key, which is a case-insensitive ASCII string,
+  // optionally prefixed with a DNS 1123 subdomain and a slash.  No restrictions


I thought we wanted these to be domain qualified?

I wanted to reuse the same validation logic that we have adopted for annotations, since that's likely to be already well thought out. We can add an additional restriction that the key must be domain-qualified.

enj · 2025-10-14T15:09:24Z

keps/sig-auth/4317-pod-certificates/README.md

+  // does not recognize one of the keys passed in userConfig, or if the signer
+  // otherwise considers the userConfig of the request to be invalid.
+  PodCertificateRequestConditionInvalidUserConfig string = "InvalidUserConfig"


I think this will need to be updated to match the other field's name.

Oh, yes, done.

enj · 2025-10-14T15:11:31Z

keps/sig-auth/4317-pod-certificates/README.md

 	//
 	// Valid values are "RSA3072", "RSA4096", "ECDSAP256",
-	// "ECDSAP384", and "ED25519".  If left empty, Kubelet defaults to "ECDSAP256".
+	// "ECDSAP384", and "ED25519".


Did we make the field required?

Yes, you have to specify a value in the pod spec.

enj · 2025-10-14T15:13:47Z

keps/sig-auth/4317-pod-certificates/README.md

+* Add e2e test to exercise the feature end-to-end
+* Implement out-of-tree signers for several scenarios to prove that the feature
+  is sufficient for real use cases.  Accomplished at
+  [mesh-examples](https://github.com/ahmedtd/mesh-example).


I would expect some of these to land in k/k in KCM eventually.

enj · 2025-10-14T15:15:40Z

keps/sig-auth/4317-pod-certificates/README.md

+  is sufficient for real use cases. These scenarios should include issuing of:
+  - serving certificates
+  - issuing client certificates
+  - SPIFFE certificates (?)


We should certainly make it possible, so that projects like Istio and Cilium can leverage it, if they want to. I don't think anything regarding SPIFFE should be included directly in KCM in the future though.

enj · 2025-10-14T15:54:38Z

keps/sig-auth/4317-pod-certificates/README.md


 ###### How can this feature be enabled / disabled in a live cluster?

 At alpha, enabling this feature in a live cluster will require the following steps.


Suggested change

At alpha, enabling this feature in a live cluster will require the following steps.

At beta, enabling this feature in a live cluster will require the following steps.

enj

Mostly minor comments.

enj · 2025-10-15T14:42:24Z

keps/sig-auth/4317-pod-certificates/README.md

 * kube-apiserver will then truncate `maxExpirationSeconds` to the signer's
  configured maximum duration.


I thought we removed this too?

Yes, you're right, removed.

enj · 2025-10-15T14:43:58Z

keps/sig-auth/4317-pod-certificates/README.md

+  // the signer implementation.  Kubernetes does not restrict or validate this
+  // metadata in any way.
+  //
+  // These values are copied verbatim into the `spec.UnverifiedUserAnnotations` field of


Suggested change

// These values are copied verbatim into the `spec.UnverifiedUserAnnotations` field of

// These values are copied verbatim into the `spec.unverifiedUserAnnotations` field of

enj · 2025-10-15T14:55:14Z

keps/sig-auth/4317-pod-certificates/README.md

 * Confirm that the issued chain (if one is set) consists of valid certificates,
  that do indeed form a chain to each other.


We don't check that the certs form a chain, just that all the certs are valid.

We also know that Go is going to tighten cert validation so let's be stricter on this new API from the start:

no DNSNames entries are "" or contain .. or start/end with .

EmailAddresses all pass mail.ParseAddress

enj · 2025-10-15T19:23:53Z

/lgtm
/approve
/hold cancel

k8s-ci-robot · 2025-10-15T19:24:02Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ahmedtd, deads2k, enj

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~keps/prod-readiness/OWNERS~~ [deads2k]
~~keps/sig-auth/OWNERS~~ [deads2k,enj]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Sep 24, 2025

k8s-ci-robot requested review from aramase and liggitt September 24, 2025 02:10

k8s-ci-robot added kind/kep Categorizes KEP tracking issues and PRs modifying the KEP directory sig/auth Categorizes an issue or PR as relevant to SIG Auth. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 24, 2025

enj added this to SIG Auth Sep 24, 2025

enj moved this to Needs Triage in SIG Auth Sep 24, 2025

ahmedtd force-pushed the podcertbeta branch from 9212583 to f881201 Compare September 24, 2025 17:45

stlaz reviewed Sep 25, 2025

View reviewed changes

ahmedtd force-pushed the podcertbeta branch 3 times, most recently from 6d57df8 to b32a96f Compare October 7, 2025 21:23

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 7, 2025

SergeyKanzhelev reviewed Oct 8, 2025

View reviewed changes

kannon92 reviewed Oct 9, 2025

View reviewed changes

k8s-ci-robot assigned deads2k Oct 9, 2025

deads2k reviewed Oct 10, 2025

View reviewed changes

ahmedtd force-pushed the podcertbeta branch from 8dc9c4d to 8fe66b9 Compare October 14, 2025 06:14

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Oct 14, 2025

ahmedtd closed this Oct 14, 2025

ahmedtd force-pushed the podcertbeta branch from 8fe66b9 to acf00df Compare October 14, 2025 06:14

github-project-automation bot moved this from Needs Triage to Closed / Done in SIG Auth Oct 14, 2025

ahmedtd reopened this Oct 14, 2025

github-project-automation bot moved this from Closed / Done to Needs Triage in SIG Auth Oct 14, 2025

ahmedtd force-pushed the podcertbeta branch from acf00df to 03b77da Compare October 14, 2025 07:14

k8s-ci-robot added size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 14, 2025

ahmedtd force-pushed the podcertbeta branch from 03b77da to 4cfc1e8 Compare October 14, 2025 07:24

k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Oct 14, 2025

enj requested changes Oct 14, 2025

View reviewed changes

github-project-automation bot moved this from Needs Triage to Changes Requested in SIG Auth Oct 14, 2025

ahmedtd force-pushed the podcertbeta branch from 4cfc1e8 to a2f27fb Compare October 14, 2025 17:39

rayandas mentioned this pull request Oct 15, 2025

Pod Certificates #4317

Open

11 tasks

enj reviewed Oct 15, 2025

View reviewed changes

KEP-4317 (Pod Certificates): Update for beta

53c7294

ahmedtd force-pushed the podcertbeta branch from a2f27fb to 53c7294 Compare October 15, 2025 19:05

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 15, 2025

k8s-ci-robot assigned enj Oct 15, 2025

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 15, 2025

k8s-ci-robot merged commit f8cff47 into kubernetes:master Oct 15, 2025
4 checks passed

k8s-ci-robot added this to the v1.35 milestone Oct 15, 2025

github-project-automation bot moved this from Changes Requested to Closed / Done in SIG Auth Oct 15, 2025

		Several example signers built on the alpha feature set are available in the
		[mesh-examples repository](https://github.com/ahmedtd/mesh-example).

		If support for the new key type is introduced in version N, then we have the
		following skew cases:

		4. Disable the PodCertificaterequest feature gate.
		5. Disable the certificates/v1beta1 API.


		###### How can a rollout or rollback fail? Can it impact already running workloads?

		During rollout (enabling the certificates/v1beta1 API and PodCertificate feature


		###### How can this feature be enabled / disabled in a live cluster?

		At alpha, enabling this feature in a live cluster will require the following steps.

	At alpha, enabling this feature in a live cluster will require the following steps.
	At beta, enabling this feature in a live cluster will require the following steps.

		* kube-apiserver will then truncate `maxExpirationSeconds` to the signer's
		configured maximum duration.

	// These values are copied verbatim into the `spec.UnverifiedUserAnnotations` field of
	// These values are copied verbatim into the `spec.unverifiedUserAnnotations` field of

		* Confirm that the issued chain (if one is set) consists of valid certificates,
		that do indeed form a chain to each other.

KEP-4317: (Pod Certificates) Update for beta #5565

KEP-4317: (Pod Certificates) Update for beta #5565

Uh oh!

Conversation

ahmedtd commented Sep 24, 2025 • edited by enj Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

richabanker Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dgrisonnet Oct 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kannon92 commented Sep 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SergeyKanzhelev Oct 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ahmedtd commented Oct 8, 2025

Uh oh!

Choose a reason for hiding this comment

ahmedtd commented Sep 24, 2025 •

edited by enj

Loading

richabanker Oct 8, 2025 •

edited

Loading

dgrisonnet Oct 15, 2025 •

edited

Loading

kannon92 commented Sep 28, 2025 •

edited

Loading

SergeyKanzhelev Oct 8, 2025 •

edited

Loading