Add CRI-based node provider

[mikeroySoft]

What would you like to be added:

With there being so many CRI-compatible runtimes available now, it would be great to have some choice to the runtime that KIND works with.

The proposal is to add a CRI-based provider that can leverage CRI in order to integrate with whatever CRI-compatible runtime may be available.

Why is this needed:

Without this, if we wanted to use a different runtime provider, a specific integration would be needed (see Podman). Instead, if there was a CRI-based provider, we could direct KIND to use that and it would then be unnecessary to maintain specific integrations, so long as the runtime implementation conformed to CRI spec (such as containerd cri plugin, for example).

[mauilion]

I am excited about this idea! @BenTheElder has some concerns around networking and ensuring that "nodes" get static addressing but maybe we can discuss this proposal at the next office hours.

[BenTheElder]

So fwiw we need to leverage things outside of CRI today (also podman and docker are NOT CRI compatible anyhow), maintaining those is not going anywhere, but I do think a CRI provider is a good idea, especially if we can leverage crictl so as not to pull k8s.io/kubernetes into the kind binary.

[BenTheElder]

CRI is a great promise but it really is just the API kubelet needs and effectively has two production implementations. We also want e.g. ignite support at some point.

I am also happy to support this but calling it
/priority backlog
As it's not strictly needed for our primary charter (testing kubernetes itself, some local app developer usage) but could be really cool especially for some niche usage.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[frittentheke]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[frittentheke]

/remove-lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[mattysweeps]

/remove-lifecycle rotten

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale

[tao12345666333]

/remove-lifecycle stale

[BenTheElder]

I'm not terribly convinced that this is an approach we should continue to consider. I know that the original motivation was resolved with a different approach.

I don't think anyone is running CRI outside of Kubernetes, it might well be called "kubelet runtime interface".

I'm sure some people are using containerd without Kubernetes, with the containerd API, perhaps. https://github.com/containerd/nerdctl might make a suitable provider, but that's a different ask (and one we haven't actually heard yet).

@frittentheke @mattysweeps @tao12345666333 do you have more concrete use cases?

Networking has also proven to be a significant pain point between providers (and within our existing providers), amongst other things. I don't think we could support something quite this general reliably even if we had a sufficient use-case.

[mattysweeps]

I thought a CRI interface would be useful for running Kind clusters beside or inside existing Kubernetes clusters. The benefit being an isolated cluster which uses real kubelets under the hood but can be easily torn down. I'm not sure how feasible this is though.

[BenTheElder]

So CRI provides an interface for kubelet to talk to the container runtime. The only way to control a kubelet is either static pod manifests on disk in the appropriate directory if enabled, or being the API server it is talkong to. You cannot share a kubelet.

Similarly, a CRI instance cannot be operated by two kubelets in any existing implementation.

KIND is also not a good approach for isolation, it is a leaky abstraction with the shared host kernel and not everything being namespaced in Linux. For isolation purposes you need to use a VM based solution or different bare metal machines.

Support for CRI would allow you to run different CRI implementations just for the purpose of running kind, which would provide some flexibility in providing a runtime, but is a pretty unrealistic scenario for most users I think. Instead we're directly supporting multiple options for running containers through interfaces not meant for exclusive access by a kubelet (so far podman and docker, neither of which implement CRI but are widely used for running containers from multiple different processes).

[frittentheke]

@BenTheElder my voice proposing CRI support in KIND was to allow e.g. containerd to be used as high level (container) runtime ... via its CRI API (to not create yet another strong binding). Actually with no relation to Kubernetes or the kubelet there (which admittedly is the main source for CRI calls and interactions nowadays).

If using CRI would then automagically allow for fancy new low-level runtimes such as Kata containers or others (see https://www.inovex.de/blog/containers-docker-containerd-nabla-kata-firecracker/) to be used, was not my main reason behind the suggestion. Networking is already implemented in a pluggable and flexible way via the CNI, so there already is abstraction there.

[mattysweeps]

So CRI provides an interface for kubelet to talk to the container runtime. The only way to control a kubelet is either static pod manifests on disk in the appropriate directory if enabled, or being the API server it is talkong to. You cannot share a kubelet.

Similarly, a CRI instance cannot be operated by two kubelets in any existing implementation.

KIND is also not a good approach for isolation, it is a leaky abstraction with the shared host kernel and not everything being namespaced in Linux. For isolation purposes you need to use a VM based solution or different bare metal machines.

Support for CRI would allow you to run different CRI implementations just for the purpose of running kind, which would provide some flexibility in providing a runtime, but is a pretty unrealistic scenario for most users I think. Instead we're directly supporting multiple options for running containers through interfaces not meant for exclusive access by a kubelet (so far podman and docker, neither of which implement CRI but are widely used for running containers from multiple different processes).

I'm sorry. I said kubelet but what I meant to say was the container runtime.
One could query an existing kubernetes cluster to get the CRI configuration for a given kubelet node, then talk to that node's container runtime and build a kind cluster from it. But you'd also have to have direct access to the node.

This is such a small and theoretical use case that it doesn't make sense for Kind to support it.

There might still be benefits to supporting CRI from an abstraction point of view. Like @frittentheke has mentioned: implementing the CRI API might add flexibility over time as new or special case runtimes are created.

[BenTheElder]

If using CRI would then automagically allow for fancy new low-level runtimes such as Kata containers or others (see https://www.inovex.de/blog/containers-docker-containerd-nabla-kata-firecracker/) to be used, was not my main reason behind the suggestion.

It would not. Even with the existing docker/podman integrations if you set kata as the default runtime kind will not work, it will need specific workarounds to deal with the differences. We have other issues open related to this (#1772, #705).

One could query an existing kubernetes cluster to get the CRI configuration for a given kubelet node, then talk to that node's container runtime and build a kind cluster from it. But you'd also have to have direct access to the node.

That would not work well, a CRI client controls all pods. There's no namespacing. Kubelet expects to manage all containers. There should really only be one CRI client. Also as I said kind is leaky and should not be running alongside your production pods (!) It's also already possible to run inside another container if you really want to run it on kubernetes nodes.

There might still be benefits to supporting CRI from an abstraction point of view. Like @frittentheke has mentioned: implementing the CRI API might add flexibility over time as new or special case runtimes are created.

Yeah, that sounds nice in theory, but it would only be true for high level runtimes, and see the caveats above: You can't actually handle new low level runtimes (runsc, kata, etc.) with this and high level runtimes (cri-o, containerd) still only have two in existence, so theoretically enabling a third that hasn't shown up in the ~five years since CRI was announced doesn't really seem useful.

[BenTheElder]

I think if we add another backend it makes more sense to support nerdctl as an analog to docker #2317 (comment)

running a cri-o setup in order to just run kind seems wildly unlikely, and cri-o or containerd as the two CRI options both don't make a ton of sense to run just for stuff like kind versus docker or podman. cri-dockerd we already target docker directly.