KEP for statusz page for Kubernetes Components

[richabanker]

• One-line PR description: Add a KEP for statusz page in Kubernetes components.

• Issue link: Statusz for Kubernetes Components #4827

[richabanker]

/assign @logicalhan

[jpbetz]

/approve
For PRR. Alpha criteria looks straightforward here.

[richabanker]

/assign @rexagod

[jpbetz]

/approve
For PRR

[SergeyKanzhelev]

/lgtm

[vinayakankugoyal]

for kubelet can you please add details about what the RBAC permissions will be for accessing this endpoint?

system:monitoring does not have the ability to call the kubelet's endpoints AFAIK. What subresource will allow a user access to kubelet's statsz endpoint?

[richabanker]

Hmm, I was assuming that system:monitoring group would suffice for kubelet as well, i.e. we could add the new endpoints to this set of rules for system:monitoring role.
cc @liggitt for his advice here.

[cjcullen]

Does this endpoint expect to be programmatically queried? I was assuming no, which makes it feel different than the endpoints that are granted to system:monitoring.

[vinayakankugoyal]

@richabanker - I think it would help to clarify what you will be adding. Can you add a markdown code section of what changes you will make to the system:monitoring ClusterRole.

[vinayakankugoyal]

I don't think this will work. Here is how I tested this.

Create a serviceaccount

kubectl create serviceaccount hello --namespace default

Create a binding to system:monitoring ClusterRole

kubectl create clusterrolebinding --clusterrole=system:monitoring --serviceaccount default:hello helloissystemmonitor 

Call the kubelet metrics endpoint via apiserver proxy.

kubectl get --raw /api/v1/nodes/$(kubectl get nodes --no-headers | awk '{print $1}')/proxy/metrics --token $(kubectl create token hello)

Error from server (Forbidden): nodes "gke-test-default-pool-93fcb923-nt6x" is forbidden: User "system:serviceaccount:default:hello" cannot get resource "nodes/proxy" in API group "" at the cluster scope

Now lets try directly accessing the kubelet API of the node

Create a file called pod.yaml with these contents

apiVersion: v1
kind: Pod
metadata:
  name: ubuntu-sleeper
spec:
  serviceAccount: hello
  containers:
  - name: ubuntu
    image: ubuntu:latest # Use the latest Ubuntu image
    command: ["/bin/bash", "-c", "apt-get update && apt-get install -y curl jq && sleep infinity"] # Start a bash shell
    env:
    - name: NODE_IP
      valueFrom:
        fieldRef:
          fieldPath: status.hostIP

Create a pod

kubectl apply -f pod.yaml 

Exec into the pod

kubectl exec ubuntu-sleeper -it -- /bin/bash 

Run the following command inside the pod

root@ubuntu-sleeper:/# curl -k https://$NODE_IP:10250/metrics -H "Authorization: Bearer $(cat /var/run/secrets/kubernetes.io/serviceaccount/
token)"
Forbidden (user=system:serviceaccount:default:hello, verb=get, resource=nodes, subresource=metrics)

Now lets check the permissions that hello has

kubectl auth can-i --list --as system:serviceaccount:default:hello              ok  test kube 
Warning: the list may be incomplete: webhook authorizer does not support user rule resolution
Resources                                       Non-Resource URLs                      Resource Names   Verbs
selfsubjectreviews.authentication.k8s.io        []                                     []               [create]
selfsubjectaccessreviews.authorization.k8s.io   []                                     []               [create]
selfsubjectrulesreviews.authorization.k8s.io    []                                     []               [create]
                                                [/.well-known/openid-configuration/]   []               [get]
                                                [/.well-known/openid-configuration]    []               [get]
                                                [/api/*]                               []               [get]
                                                [/api]                                 []               [get]
                                                [/apis/*]                              []               [get]
                                                [/apis]                                []               [get]
                                                [/healthz/*]                           []               [get]
                                                [/healthz]                             []               [get]
                                                [/healthz]                             []               [get]
                                                [/healthz]                             []               [get]
                                                [/livez/*]                             []               [get]
                                                [/livez]                               []               [get]
                                                [/livez]                               []               [get]
                                                [/livez]                               []               [get]
                                                [/metrics/slis]                        []               [get]
                                                [/metrics]                             []               [get]
                                                [/openapi/*]                           []               [get]
                                                [/openapi]                             []               [get]
                                                [/openid/v1/jwks/]                     []               [get]
                                                [/openid/v1/jwks]                      []               [get]
                                                [/readyz/*]                            []               [get]
                                                [/readyz]                              []               [get]
                                                [/readyz]                              []               [get]
                                                [/readyz]                              []               [get]
                                                [/version/]                            []               [get]
                                                [/version/]                            []               [get]
                                                [/version]                             []               [get]
                                                [/version]                             []               [get]

notice that it has /metrics permissions but that won't give it permissions for kubelet. For kubelet you will have to extend the rbac to something like nodes/statusz or something like that.

[richabanker]

I have added a markdown for changes that we can do to system:monitoring role to allow access to kubelet's /statusz. Does this look ok now @vinayakankugoyal ?

[vinayakankugoyal]

Hmm... I am not sure about this.

It seems like system:monitoring did not previously have permissions to call kubelet API do we really want to grant this permission?

[richabanker]

I added these changes to system:monitoring as placeholder for now. We can have a final decision on this once @tallclair (sig-node) is available for comments. Also filed kubernetes/kubernetes#127990 to track the discussion.

cc @SergeyKanzhelev

[vinayakankugoyal]

Hmm... I am not sure about this.

It seems like system:monitoring did not previously have permissions to call kubelet API do we really want to grant this permission?

[vinayakankugoyal]

You can already do this by calling, why not just use that?

kubectl get --raw /version
{
  "major": "1",
  "minor": "31",
  "gitVersion": "v1.31.1-gke.1472000",
  "gitCommit": "e21141bf6951009a138dda22624a11674fb9be02",
  "gitTreeState": "clean",
  "buildDate": "2024-09-23T20:27:37Z",
  "goVersion": "go1.22.6 X:boringcrypto",
  "compiler": "gc",
  "platform": "linux/amd64"
}%          

[vinayakankugoyal]

This information is also exposed via a metric

kubectl get --raw /metrics|grep "^kubernetes_build_info.*"                                                                               INT|INT err  1m 10s  test kube 
kubernetes_build_info{build_date="2024-09-23T20:27:37Z",compiler="gc",git_commit="e21141bf6951009a138dda22624a11674fb9be02",git_tree_state="clean",git_version="v1.31.1-gke.500",go_version="go1.22.6 X:boringcrypto",major="1",minor="31",platform="linux/amd64"} 1

[richabanker]

I understand that the proposed statusz endpoint might seem a bit redundant given the existing /version endpoint. However, statusz offers a dedicated space for comprehensive component information, going beyond just version details. It has the potential to serve as a central debugging hub, providing not only component status but also links to other relevant z-pages for deeper observability.

[SergeyKanzhelev]

I am fine with deciding on it during implementation

/lgtm

[vinayakankugoyal]

/lgtm

[logicalhan]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jpbetz, logicalhan, richabanker, SergeyKanzhelev

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• keps/prod-readiness/OWNERS [jpbetz]
• keps/sig-instrumentation/OWNERS [logicalhan]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment