Cluster create never finishes when given podSubnet smaller than /24

[justenwalker]

What happened:

Whe configuring a pod subnet, a subnet smaller than /24 will cause the kube-controller-manager-kind-control-plane to enter a CrashLoop due to this error:

# k -n kube-system logs kube-controller-manager-kind-control-plane | grep cidr
F0114 15:54:20.877255       1 node_ipam_controller.go:110] Controller: Invalid --cluster-cidr, mask size of cluster CIDR must be less than --node-cidr-mask-size

What you expected to happen:

If there is a minimum size of a podSubnet, the config should validate that the subnet given meets the criteria. Otherwise it should function normally (perhaps by adjusting the CIDR mask)

How to reproduce it:

$ cat <<EOF > ./cluster.yaml
apiVersion: kind.sigs.k8s.io/v1alpha3
kind: Cluster
networking:
  podSubnet: 172.16.64.0/26
EOF
$ kind create cluster --config=./cluster.yaml
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.15.6) 🖼
 ✓ Preparing nodes 📦
 ✓ Creating kubeadm config 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
 ✗ Waiting ≤ 10m0s for control-plane = Ready ⏳
 • WARNING: Timed out waiting for Ready ⚠️
Cluster creation complete. You can now use the cluster with:

export KUBECONFIG="$(kind get kubeconfig-path --name="kind")"
kubectl cluster-info
$ docker exec -it kind-control-plane kubectl --kubeconfig=/etc/kubernetes/admin.conf -n kube-system get pod kube-controller-manager-kind-control-plane
NAME                                         READY   STATUS             RESTARTS   AGE
kube-controller-manager-kind-control-plane   0/1     CrashLoopBackOff   7          15m

Anything else we need to know?:

host $ docker exec -it kind-control-plane /bin/sh
docker $ export KUBECONFIG=/etc/kubernetes/admin.conf
docker $ kubectl -n kube-system get pod  kube-controller-manager-kind-control-plane
NAME                                         READY   STATUS             RESTARTS   AGE
kube-controller-manager-kind-control-plane   0/1     CrashLoopBackOff   5          6m13s
docker $ kubectl -n kube-system logs kube-controller-manager-kind-control-plane | grep cidr
F0114 16:04:31.674717       1 node_ipam_controller.go:110] Controller: Invalid --cluster-cidr, mask size of cluster CIDR must be less than --node-cidr-mask-size
docker $ kubectl -n kube-system get pod kube-controller-manager-kind-control-plane -o yaml
apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/config.hash: 5956ad6fb3d745e44e5f3d05dc4cc747
    kubernetes.io/config.mirror: 5956ad6fb3d745e44e5f3d05dc4cc747
    kubernetes.io/config.seen: "2020-01-14T15:59:17.0778636Z"
    kubernetes.io/config.source: file
  creationTimestamp: "2020-01-14T15:59:55Z"
  labels:
    component: kube-controller-manager
    tier: control-plane
  name: kube-controller-manager-kind-control-plane
  namespace: kube-system
  resourceVersion: "572"
  selfLink: /api/v1/namespaces/kube-system/pods/kube-controller-manager-kind-control-plane
  uid: d648a806-1fd3-4ed5-b1e0-6acb8ebca5d6
spec:
  containers:
  - command:
    - kube-controller-manager
    - --allocate-node-cidrs=true
    - --authentication-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --authorization-kubeconfig=/etc/kubernetes/controller-manager.conf
    - --bind-address=127.0.0.1
    - --client-ca-file=/etc/kubernetes/pki/ca.crt
    - --cluster-cidr=172.16.64.0/26
    - --cluster-signing-cert-file=/etc/kubernetes/pki/ca.crt
    - --cluster-signing-key-file=/etc/kubernetes/pki/ca.key
    - --controllers=*,bootstrapsigner,tokencleaner
    - --enable-hostpath-provisioner=true
    - --kubeconfig=/etc/kubernetes/controller-manager.conf
    - --leader-elect=true
    - --node-cidr-mask-size=24
    - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt
    - --root-ca-file=/etc/kubernetes/pki/ca.crt
    - --service-account-private-key-file=/etc/kubernetes/pki/sa.key
    - --use-service-account-credentials=true
... <snip> ...

Environment:

$ kind version
v0.5.1
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.1", GitCommit:"4485c6f18cee9a5d3c3b4e523bd27972b1b53892", GitTreeState:"clean", BuildDate:"2019-07-18T09:18:22Z", GoVersion:"go1.12.5", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.7", GitCommit:"6c143d35bb11d74970e7bc0b6c45b6bfdffc0bd4", GitTreeState:"clean", BuildDate:"2019-12-11T12:34:17Z", GoVersion:"go1.12.12", Compiler:"gc", Platform:"linux/amd64"}
$ docker info | grep -i version
 Server Version: 19.03.5
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Kernel Version: 4.9.184-linuxkit
$ cat /etc/os-release
NAME="Ubuntu"
VERSION="18.04.3 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.3 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

[BenTheElder]

With a newer kind version and increased -v we can get more details

I'm not actually sure what requires this other than obviously we need enough IPs for the pods

[justenwalker]

If newer versions of Kind have easier ways to troubleshoot that's great.

The issue I'm bringing up is that this subnet configuration could be validated before submission so that you get a more useful error than F0114 15:54:20.877255 1 node_ipam_controller.go:110] Controller: Invalid --cluster-cidr, mask size of cluster CIDR must be less than --node-cidr-mask-size

[BenTheElder]

ah ok, that's fun.
I bet kind or kubeadm neeeds to be overridding these. cc @neolit123 I vaguely remember you commenting on an issue like this

[neolit123]

The issue I'm bringing up is that this subnet configuration could be validated before submission so that you get a more useful error than F0114 15:54:20.877255 1 node_ipam_controller.go:110] Controller: Invalid --cluster-cidr, mask size of cluster CIDR must be less than --node-cidr-mask-size

kubeadm has some special casing for the cluster CIDR mask for IPv6, but for IPv4 it does not (e.g.) take the cluster CIDR mask and increase it by one to satisfy the Node CIDR semantic of the KCM.

if you'd like kubeadm to behave differently, please log an issue in the kubernetes/kubeadm tracker.

thanks
/close

[k8s-ci-robot]

@neolit123: Closing this issue.

In response to this:

The issue I'm bringing up is that this subnet configuration could be validated before submission so that you get a more useful error than F0114 15:54:20.877255 1 node_ipam_controller.go:110] Controller: Invalid --cluster-cidr, mask size of cluster CIDR must be less than --node-cidr-mask-size

kubeadm has some special casing for the cluster CIDR mask for IPv6, but for IPv4 it does not (e.g.) take the cluster CIDR mask and increase it by one to satisfy the Node CIDR semantic of the KCM.

if you'd like kubeadm to behave differently, please log an issue in the kubernetes/kubeadm tracker.

thanks
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[BenTheElder]

it might be reasonable for kind do this calculation and pass it on to kubeadm, since we're providing an option for the CIDR at a level above kubeadm (and e.g. also passing it to our CNI impl).

[BenTheElder]

(pass it on to kubeadm by way of patching the ---node-cidr-mask-size in the generated config)

[justenwalker]

@neolit123 / @BenTheElder It's not that I want kubernetes/kubeadmin to behave differently; I want Kind to validate the cluster config so it doesn't pass options to kubeadmin that are known to fail: ie -- providing an IPv4 CIDR that is too small.

I think it would be less confusing if kind kicked back and error that said "hey, that podSubnet you passed is way too small and the cluster wont come up" instead of kind saying "all good" and then finding out later that its in a crash-loop

[neolit123]

it might be reasonable for kind do this calculation and pass it on to kubeadm, since we're providing an option for the CIDR at a level above kubeadm (and e.g. also passing it to our CNI impl).

it is possible for kind to manage this too.

minimal configuration:

apiVersion: kubeadm.k8s.io/v1beta2
kind: ClusterConfiguration
kubernetesVersion: v1.16.0
networking:
  podSubnet: <some-cidr>
controllerManager:
  extraArgs:
    node-cidr-mask-size: <mask-adapted-from-some-cidr>

[neolit123]

@neolit123 / @BenTheElder It's not that I want kubernetes/kubeadmin to behave differently; I want Kind to validate the cluster config so it doesn't pass options to kubeadmin that are known to fail: ie -- providing an IPv4 CIDR that is too small.

using kubeadm without kind, the same failure will be observed.
therefore, it is not a bad idea to have the validation on this lower level.

[aojea]

You can not validate this because you don't know the node-cidr-mask-size in advance and you can not assume that the user is going to use the default value, because that is only used by the controller-manager, not cluster wide.

This is a network planning problem, because user says "my cluster has network 192.168.0.0/28"
but user does not change the node-cidr-mask (that is only used by the controller-manager) , then the controller-manager uses a /24, and since it is bigger it breaks.

Cluster-CIDR is problematic, there are a lot of issues about it, but this is the one that describes it better
kubernetes/kubernetes#46508 (comment)

[aojea]

scratch my last comment, lack of coffee, kubeadm is already doing the calculation, just need to be updated

[BenTheElder]

@aojea I defer to you on this one. Perhaps at least we should add a docs note.

[aojea]

/assign