[KEP-5573]: Remove cgroup v1 support #5574
Conversation
k8s-ci-robot
added
the
cncf-cla: yes
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: kannon92 The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
kind/kep
| - Default value for `FailCgroupV1` changed to `true` | ||
| - Updated warning messages and events for unsupported status | ||
| - Documentation updates in kubernetes/enhancements repository | ||
| - All e2e testing removed for cgroup v1. |
There was a problem hiding this comment.
If we apply the plan like below:
- Disable cgroupv1 by default in 1.35
- Declare deprecation of cgroupv1 support officially
- Wait for two releases and remove all cgroupv1 related code
We may have roughly two steps: 1. to change default and add deprecation message; 2 lock the value to true, and remove related code & tests.
There was a problem hiding this comment.
Deprecation should be declared before disabling it by default?
There was a problem hiding this comment.
let's rephrase this item as:
| - All e2e testing removed for cgroup v1. | |
| - Subset of tests will run on cgroupv1 offering sanity check of cgroupv1 functioning. Since the cgroupv1 was moved to the maintenance mode, only older features are supported on cgroupv1. |
There was a problem hiding this comment.
I disagree with this. This essentially makes our KEP moot. We are still committing to maintaining and testing cgroup v1.
There was a problem hiding this comment.
I think we have to do so for a period to ease into the dropping of support. we do have v1 support in containerd tests still so we can maintain those at least for the alpha of this feature
There was a problem hiding this comment.
okay. I'll update this.
Is it okay to rely on only containerd for cgroup v1 support while CRIO we will no longer test on cgroup v1 in upstream?
Worst case we could use centos8.
There was a problem hiding this comment.
since runtimes are an implementation of the CRI interface, that is the only piece Kubernetes is responsible, I think that is a legit decision for SIG node to say the feature is covered that way ... but is just my opinion
| --> | ||
|
|
||
| **Upgrade considerations**: | ||
| - Clusters upgrading to Kubernetes v1.35+ on cgroup v1 hosts will fail to start kubelet unless `FailCgroupV1` is set to false |
There was a problem hiding this comment.
For kubeadm view, init/upgrade precheck should fail for cgroup v1.
This should be applied after the kubelet change.
|
|
||
| #### GA | ||
|
|
||
| - Default value for `FailCgroupV1` changed to `true` |
There was a problem hiding this comment.
also fail-cgroupv1 kubelet flag.
There was a problem hiding this comment.
We don't need to set the flag. We prefer kubelet config fields over flags now.
Setting the kubelet config is sufficient.
| - Kubelet configuration documentation | ||
| - Migration guides | ||
| - Release notes | ||
| - Blog posts about the transition |
There was a problem hiding this comment.
I may help in this task. I tried to write a blog about cgroup v2 in kubernetes/website#47342 which is not completed yet.
|
|
||
| To (unsupported): | ||
| ```golang | ||
| klog.Warning("cgroup v1 detected. cgroup v1 support is unsupported and will be removed in a future release. Please migrate to cgroup v2. More information at https://git.k8s.io/enhancements/keps/sig-node/5573-cgroup-v1-unsupported") |
There was a problem hiding this comment.
will be removed in a future release
Which release?
There was a problem hiding this comment.
I updated. Removal would be 1.38.
There was a problem hiding this comment.
In this case, we might also provide context somewhere that 1.38 means a rough timeline of ~EOY 2026, given the other timelines here will probably mostly be calendar dates versus k8s release aligned (distros phasing support, systemd etc)
kannon92
force-pushed
the
move-cgroup-v1-unsupported
branch
from
0b216d8 to
2fde3ad
Compare
kannon92
force-pushed
the
move-cgroup-v1-unsupported
branch
from
2fde3ad to
59dadda
Compare
|
FYI @aojea @stmcginnis @medyagh @afbjorklund |
kannon92
changed the title
There was a problem hiding this comment.
I do think this is something we need to do eventually, and there is clearly evidence that the linux ecosystem is moving this direction, but I think there's still a question of timing.
I don't see much evidence provided that non-bleeding-edge distros are done heading this way (including both those that we run "production" clusters on, and those that developers may locally test on with many tools, yes including kind, I'd be happy to not have to deal with this but we need to consider the impact).
In the linked issue I only see mentions of some major managed clusters, and fedora / bleeding-edge systemd...
|
|
||
| The motivation builds on the rationale established in KEP-4569: | ||
| - The Linux kernel community has made cgroup v2 the focus for new features | ||
| - Major Linux distributions are phasing out cgroup v1 support |
There was a problem hiding this comment.
Do we have a more detailed timeline for this? Because any hosts on cgroup v1 running minikube --vm-driver=docker, kind, k3d etc will be inheriting cgroup v1/v2 from the host.
Many of our users will not be on bleeding edge distros, so even when systemd drops support in the latest release they may still be on cgroup v1.
There was a problem hiding this comment.
For me, it will be easier to focus on cgroupv2, but I still commonly see users on v1 ...
There was a problem hiding this comment.
I can only dig up fedora ecosystem as I know that better.
RHEL 9.4 has deprecated cgroup v1 and RHEL 10 has removed cgroup for cgroup v1.
in Openshift we have deprecated cgroup v1 in 4.18 and block upgrades to 4.19 (1.31 to 1.32).
There was a problem hiding this comment.
adding a reference to other distros like debian/ubuntu will make the case stronger , specially since you are explicitly calling that out "Major Linux distributions are phasing out cgroup v1 support" , so pick some of them and explain their timelines to justify the dates.
If I have to pick I think debian will be the most important since it is completely community driven and widely used
There was a problem hiding this comment.
it will be another year of support for cgroupv1. If we will recieve overwhelming feedback on this - we can push the removal part later. But disabling by default and declaring that it is deprecated must happen so we will raise awarenesess early and fullfil the required deprecation timeline.
This may be an alterative way to say this:
| - Major Linux distributions are phasing out cgroup v1 support | |
| - Linux distributions are phasing out cgroup v1 support as they adopt the newer versions of systemd |
There was a problem hiding this comment.
+1 @aojea
Systemd is not the only way to run Kubernetes (even if it is the best supported and most widely used), I think we're centering a little too much on that here.
I know we have certified distros that are not using systemd.
There was a problem hiding this comment.
I know we have certified distros that are not using systemd.
What does "certified" means in this context?
There was a problem hiding this comment.
What does "certified" means in this context?
Sorry, very confusing and overloaded terms here -- I mean conformant Kubernetes distros (not linux distros) which in turn are not using systemd, I know at least one fairly popular off the top of my head, I didn't want to focus the conversation on that though, I just mean that systemd is not universal in supported clusters.
| 1. Continue monitoring cgroup v2 CI jobs to ensure stability | ||
| 2. Add specific tests for the new default behavior | ||
| 3. Ensure all new tests use cgroup v2 hosts | ||
| 4. Remove all cgroup v1 test lanes. |
There was a problem hiding this comment.
This term "test lanes" is usually only used within SIG Node when talking about node e2e jobs, but to be clear, we have to have all CI off of cgroup v1, which includes any of the cluster e2es.
Are we FYI-ing SIG Cluster Lifecycle, for example?
There was a problem hiding this comment.
@pacoxu and @neolit123 have weighed in on the issue.
There was a problem hiding this comment.
for sig cl one good thing is that most of the tools use kubeadm and it uses the k/system-validators library that has been throwing a warning about cgroups v1 for a couple (?) of releases. kOps doesn't use the system-validators, but they have dropped cgroups v1 support already, as i read somewhere.
the concerns are still there. users ignore warnings, and if we start erroring in preflight they will start the arguments about distro foo, being very popular but still has issue bar with enabling cgroups v2 by default. same story was with the old kernel errors on preflight. for that, sig cl will be the front liner, but we are just going to have to link to this kep and say - complain to sig node if you really want to.
@kannon92 , i suggest you advertise this kep work on the sig-node, sig-cl and k8s dev mailing lists so that people are aware of the work even before this merges.
There was a problem hiding this comment.
@pacoxu and @neolit123 have weighed in on the issue.
Yeah, that's not quite the same thing though, I'd want to FYI minikube, kops, CAP*, etc. just in case, if not other folks interested in the list.
@kannon92 , i suggest you advertise this kep work on the sig-node, sig-cl and k8s dev mailing lists so that people are aware of the work even before this merges.
+1, even folks not nominally working on node need to be aware of a deprecation this large
There was a problem hiding this comment.
I sent an email to [email protected], sig-node and sig-cluster-lifecycle.
There was a problem hiding this comment.
hi @kannon92 i think it is better to ensure cri-dockerd supports cgroups v2
Mirantis/cri-dockerd#493
minikube's default container runtime is cri-dockerd and we would need to ensure we dont break them
so I suggest helping cri-dockerd project to support v2 first then do this
There was a problem hiding this comment.
CRI-Dockerd is perhaps not the ideal default choice ... cgroupv2 GA-ed in Kubernetes multiple years ago
see also: https://www.k8s.dev/docs/code/cri-api-dev-policies/
cri-dockerd is explicitly not in scope for SIG node as a participating runtime with support required to GA features ... (that decision is not new and didn't involve me, just an FYI .. it was sent to the kubernetes mailing lists including the required dev@ ...)
| The motivation builds on the rationale established in KEP-4569: | ||
| - The Linux kernel community has made cgroup v2 the focus for new features | ||
| - Major Linux distributions are phasing out cgroup v1 support | ||
| - [systemd](https://github.com/systemd/systemd/releases/tag/v258) and other critical components are moving beyond cgroup v1 |
There was a problem hiding this comment.
| - [systemd](https://github.com/systemd/systemd/releases/tag/v258) and other critical components are moving beyond cgroup v1 | |
| - systemd announced deprecation 1.5 years ago in [v256](https://github.com/systemd/systemd/releases/tag/v256-rc3) and [removal](https://github.com/systemd/systemd/releases/tag/v258). Other critical components are moving beyond cgroup v1 |
|
|
||
| #### Stable | ||
|
|
||
| - Remove all cgroup v1 related code. |
There was a problem hiding this comment.
| - Remove all cgroup v1 related code. | |
| - All new wide-spread blocking issues, discovered during the migration to cgroup v2 are mitigated | |
| - Remove all cgroup v1 related code. |
There was a problem hiding this comment.
This is pretty vague. I am not aware of blocking issues.
Claude code was used to aide in development of this KEP.