Clarify node drain requirement for kubelet minor version upgrades

[avestuk]

Addresses the issue raised in #46356 where I suggested that the language here could be made more clear.

This change clarifies the requirement that nodes are drained before a minor version upgrade takes place. The old wording implied that a minor version upgrade of the kubelet was not supported at all.

[k8s-ci-robot]

Welcome @avestuk!

It looks like this is your first PR to kubernetes/website 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/website has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign palnabarun for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• content/en/releases/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[netlify]

✅ Pull request preview available for checking

Built without sensitive environment variables

Name	Link
🔨 Latest commit	35d517a
🔍 Latest deploy log	https://app.netlify.com/sites/kubernetes-io-main-staging/deploys/675083dd65521b0008d6fe8e
😎 Deploy Preview	https://deploy-preview-46359--kubernetes-io-main-staging.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[avestuk]

@sftim @salaxander you both happy with the latest version?

Also thank you for the reviews!

[liggitt]

I think the "to a newer minor version of Kubernetes" bit is redundant.

I also think the content of kubelet-managed directories is not necessarily cross-minor-version compatible. I'd like sig-node's input on highlighting the need to clear those directories before an in-place upgrade, something like this:

Suggested change

	In-place minor version `kubelet` upgrades, to a newer minor version of Kubernetes, without a node drain, are not supported.
	In-place minor version `kubelet` upgrades without draining pods and clearing the content of the kubelet-managed directories are not supported.

[sftim]

OK, so combining the thinking so far, I suggest:

Suggested change

	In-place minor version `kubelet` upgrades, to a newer minor version of Kubernetes, without a node drain, are not supported.
	In-place `kubelet` upgrades, without first draining pods from the node **and** clearing the content
	of the kubelet-managed directories, are only supported for upgrades to a newer patch release within
	the same minor version of Kubernetes.

This makes for a problem, though: can we hyperlink “clearing the content of the kubelet-managed directories” to the specific text about how to do that? If not, we're almost certainly going to get issues filed about adding more detail.

[avestuk]

More information about this clearing of directories would definitely be good because the current documentation doesn't state anything about needing to do this - see https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/upgrading-linux-nodes/

[avestuk]

@liggitt Is there somebody from sig-node you'd recommend asking for a consult on this?

[liggitt]

probably folks in https://github.com/kubernetes/community/tree/master/sig-node#technical-leads or delegates

cc @kubernetes/sig-node-pr-reviews
/cc @dchen1107 @derekwaynecarr @mrunalp

[avestuk]

@mrunalp @dchen1107 @derekwaynecarr anything I can do to help move this PR forward?

[SergeyKanzhelev]

Theoretically those files should be compatible unless specified otherwise in release notes. But after the drain happened, some of them have little sense. But committing on compatibility is not ideal.

I think the ask here is to come up with the docs page on what files kubelet may be using. Is there any suggestion on whether it will be best as admin actions section? Or perhaps kubelet configuration? Any advice would be helpful.

Off the top of my head, the files include:

• static pods manifests /etc/kubernetes/manifests (must be compatible unless some removed fields in pod API, which is rare (never?))
• topology manager checkpoints (have little sense to keep when the node was drained)
• kubelet configuration file and additional drop-in configuration files. Mostly compatible
• device plugin directory - may not be able to delete as Device plugins may be implemented as DaemonSets ignoring the bad node state. And those will be running and keeping the socket in /var/lib/kubelet/device-plugins/.
• Kubelet lock file: /var/run/kubelet.lock. Likely will not change, but safe to clean up if it was left over.

I can bring this to tomorrow's meeting to find volunteer to help document this.

[sftim]

Best place for a list of files that the kubelet uses: within https://kubernetes.io/docs/reference/node/

Note that many of these are defaults or conventions; eg the kubelet even used to be able to fetch static Pod manifests via HTTP, /etc/kubernetes/manifests is just the most common place to put them.

[neolit123]

old thread on this topic:
#12326 (comment)
but clearing contents on kubelet managed directories was never mentioned before?
that includes /var/lib/kubelet/pki, which i don't think the users would want to clear.

I think the ask here is to come up with the docs page on what files kubelet may be using.

CPU checkpoint files too maybe?
https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cm/cpumanager/state/checkpoint.go#L31-L33

[SergeyKanzhelev]

#47425

[sftim]

Maybe we also need a new task page: prepare a node for an in-place upgrade.

[ffromani]

/cc

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all PRs.

This bot triages PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the PR is closed

You can:

• Mark this PR as fresh with /remove-lifecycle stale
• Close this PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[avestuk]

/remove-lifecycle stale

[avestuk]

@kubernetes/sig-node-pr-reviews could somebody have a look at this please?

[k8s-ci-robot]

@avestuk: Reiterating the mentions to trigger a notification:
@kubernetes/sig-node-pr-reviews

In response to this:

@kubernetes/sig-node-pr-reviews could somebody have a look at this please?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[avestuk]

@mrunalp, @derekwaynecarr ,@dchen1107 do any of you think you'd be able to take a look at this? It's a one liner? If not could you provide an estimate of when you might be able to look at it?

[SergeyKanzhelev]

@mrunalp, @derekwaynecarr ,@dchen1107 do any of you think you'd be able to take a look at this? It's a one liner? If not could you provide an estimate of when you might be able to look at it?

the proposed line has redundancy. Please rephrase. Also since we have added a kubelet reference doc - maybe a good idea to reference it and suggest that state may need to be cleared as well if kubelet was not terminated gracefully for the upgrade.

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: avestuk / name: Alex Vest (1157906, 35d517a)

[avestuk]

@SergeyKanzhelev I've attempted to address the redunancy and I've linked to the kubelet reference documentation though I don't find this to be a complete solution. As a user my question on reading the kubelet reference link would be "which directories do I need to delete?".