Failed to create cluster: failed to apply overlay network: exit status 1

[kkimdev]

Hi I tried running kind and encountered the following errors. Full log attached at the bottom.

Notable lines:

• Error processing tar file(exit status 1): write /usr/local/bin/kube-controller-manager: no space left on device
• [WARNING ImagePull]: failed to pull image [k8s.gcr.io/kube-controller-manager-amd64:v1.11.3]: exit status 1
• [WARNING ImagePull]: failed to pull image [k8s.gcr.io/etcd-amd64:3.2.18]: exit status 1
• error: unable to recognize "https://cloud.weave.works/k8s/net?k8s-version=Q2xpZW50IFZlcnNpb246IHZlcnNpb24uSW5mb3tNYWpvcjoiMSIsIE1pbm9yOiIxMSIsIEdpdFZlcnNpb246InYxLjExLjMiLCBHaXRDb21taXQ6ImE0NTI5NDY0ZTQ2MjljMjEyMjRiM2Q1MmVkZmUwZWE5MWIwNzI4NjIiLCBHaXRUcmVlU3RhdGU6ImNsZWFuIiwgQnVpbGREYXRlOiIyMDE4LTA5LTIwVDA5OjAyOjUyWiIsIEdvVmVyc2lvbjoiZ28xLjEwLjMiLCBDb21waWxlcjoiZ2MiLCBQbGF0Zm9ybToibGludXgvYW1kNjQifQo=": Get https://172.17.0.2:6443/api?timeout=32s: dial tcp 172.17.0.2:6443: connect: connection refused
• FATA[1007-16:12:53] Failed to create cluster: failed to apply overlay network: exit status 1

So I guess there are two issues here:

• Failing to install kube-controller-manager. I'm not sure why it complained no space left on device. I'm using Docker Toolbox for Windows https://docs.docker.com/toolbox/toolbox_install_windows/ and the disk was far from full.

~$ df
Filesystem                Size      Used Available Use% Mounted on
tmpfs                   896.1M    244.0M    652.0M  27% /
tmpfs                   497.8M    104.0K    497.7M   0% /dev/shm
/dev/sda1                17.9G      5.3G     11.6G  32% /mnt/sda1
cgroup                  497.8M         0    497.8M   0% /sys/fs/cgroup
c/Users                 463.5G    276.1G    187.4G  60% /c/Users
/dev/sda1                17.9G      5.3G     11.6G  32% /mnt/sda1/var/lib/docker```

• Early exit on error. I think the create process should have stopped at the initial "no space left on device" error but it went on a while.

$ ~/go/bin/kind create
INFO[1007-16:08:47] Image: kindest/node:v1.11.3 present locally
INFO[1007-16:08:47] Running: /usr/bin/docker [docker run -d --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run --tmpfs /var/lib/docker:exec -v /lib/modules:/lib/modules:ro --hostname kind-1-control-plane --name kind-1-control-plane --label io.k8s.sigs.kind.cluster=1 --expose 6443 --publish-all --entrypoint=/usr/local/bin/entrypoint kindest/node:v1.11.3 /sbin/init]
kind-1-control-plane
f9d9e4e6e2f0: Loading layer 1.378 MB/1.378 MB
7d9ec20bda54: Loading layer 185.2 MB/185.2 MB
Loaded image: k8s.gcr.io/kube-apiserver:v1.11.3
Loaded image: gcr.io/google_containers/kube-apiserver:v1.11.3
582b548209e1: Loading layer  44.2 MB/44.2 MB
e20569a478ed: Loading layer 3.358 MB/3.358 MB
59eabcbba10c: Loading layer 51.92 MB/51.92 MB
Loaded image: k8s.gcr.io/kube-proxy:v1.11.3
Loaded image: gcr.io/google_containers/kube-proxy:v1.11.3
d3a980f536d3: Loading layer 153.8 MB/153.8 MB
Error processing tar file(exit status 1): write /usr/local/bin/kube-controller-manager: no space left on device
1fb1aa2fa388: Loading layer  55.5 MB/55.5 MB
Loaded image: k8s.gcr.io/kube-scheduler:v1.11.3
Loaded image: gcr.io/google_containers/kube-scheduler:v1.11.3
REPOSITORY                                      TAG                 IMAGE ID            CREATED             SIZE
gcr.io/google_containers/kube-apiserver-amd64   v1.11.3             932fff609f2d        2 weeks ago         186 MB
gcr.io/google_containers/kube-apiserver         v1.11.3             932fff609f2d        2 weeks ago         186 MB
k8s.gcr.io/kube-apiserver-amd64                 v1.11.3             932fff609f2d        2 weeks ago         186 MB
k8s.gcr.io/kube-apiserver                       v1.11.3             932fff609f2d        2 weeks ago         186 MB
gcr.io/google_containers/kube-proxy-amd64       v1.11.3             8b7daab5f48f        2 weeks ago         97.6 MB
gcr.io/google_containers/kube-proxy             v1.11.3             8b7daab5f48f        2 weeks ago         97.6 MB
k8s.gcr.io/kube-proxy-amd64                     v1.11.3             8b7daab5f48f        2 weeks ago         97.6 MB
k8s.gcr.io/kube-proxy                           v1.11.3             8b7daab5f48f        2 weeks ago         97.6 MB
k8s.gcr.io/kube-scheduler-amd64                 v1.11.3             ab997a31b99f        2 weeks ago         56.7 MB
k8s.gcr.io/kube-scheduler                       v1.11.3             ab997a31b99f        2 weeks ago         56.7 MB
gcr.io/google_containers/kube-scheduler-amd64   v1.11.3             ab997a31b99f        2 weeks ago         56.7 MB
gcr.io/google_containers/kube-scheduler         v1.11.3             ab997a31b99f        2 weeks ago         56.7 MB
INFO[1007-16:08:55] Using KubeadmConfig:

# config generated by kind
apiVersion: kubeadm.k8s.io/v1alpha2
kind: MasterConfiguration
kubernetesVersion: v1.11.3
# TODO(bentheelder): fix this upstream!
# we need nsswitch.conf so we use /etc/hosts
# https://github.com/kubernetes/kubernetes/issues/69195
apiServerExtraVolumes:
- name: nsswitch
  mountPath: /etc/nsswitch.conf
  hostPath: /etc/nsswitch.conf
  writeable: false
  pathType: FileOrCreate
clusterName: kind-1
# on docker for mac we have to expose the api server via port forward,
# so we need to ensure the cert is valid for localhost so we can talk
# to the cluster after rewriting the kubeconfig to point to localhost
apiServerCertSANs: [localhost]


[init] using Kubernetes version: v1.11.3
[preflight] running pre-flight checks
        [WARNING FileContent--proc-sys-net-bridge-bridge-nf-call-iptables]: /proc/sys/net/bridge/bridge-nf-call-iptables does not exist
        [WARNING Swap]: running with swap on is not supported. Please disable swap
        [WARNING FileExisting-crictl]: crictl not found in system path
I1007 23:08:54.584002     368 kernel_validator.go:81] Validating kernel version
I1007 23:08:54.584592     368 kernel_validator.go:96] Validating kernel config
[preflight/images] Pulling images required for setting up a Kubernetes cluster
[preflight/images] This might take a minute or two, depending on the speed of your internet connection
[preflight/images] You can also perform this action in beforehand using 'kubeadm config images pull'
        [WARNING ImagePull]: failed to pull image [k8s.gcr.io/kube-controller-manager-amd64:v1.11.3]: exit status 1
        [WARNING ImagePull]: failed to pull image [k8s.gcr.io/etcd-amd64:3.2.18]: exit status 1
[kubelet] Writing kubelet environment file with flags to file "/var/lib/kubelet/kubeadm-flags.env"
[kubelet] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[preflight] Activating the kubelet service
[certificates] Generated ca certificate and key.
[certificates] Generated apiserver certificate and key.
[certificates] apiserver serving cert is signed for DNS names [kind-1-control-plane kubernetes kubernetes.default kubernetes.default.svc kubernetes.default.svc.cluster.local localhost] and IPs [10.96.0.1 172.17.0.2]
[certificates] Generated apiserver-kubelet-client certificate and key.
[certificates] Generated sa key and public key.
[certificates] Generated front-proxy-ca certificate and key.
[certificates] Generated front-proxy-client certificate and key.
[certificates] Generated etcd/ca certificate and key.
[certificates] Generated etcd/server certificate and key.
[certificates] etcd/server serving cert is signed for DNS names [kind-1-control-plane localhost] and IPs [127.0.0.1 ::1][certificates] Generated etcd/peer certificate and key.
[certificates] etcd/peer serving cert is signed for DNS names [kind-1-control-plane localhost] and IPs [172.17.0.2 127.0.0.1 ::1]
[certificates] Generated etcd/healthcheck-client certificate and key.
[certificates] Generated apiserver-etcd-client certificate and key.
[certificates] valid certificates and keys now exist in "/etc/kubernetes/pki"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/admin.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/kubelet.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/controller-manager.conf"
[kubeconfig] Wrote KubeConfig file to disk: "/etc/kubernetes/scheduler.conf"
[controlplane] Adding extra host path mount "nsswitch" to "kube-apiserver"
[controlplane] wrote Static Pod manifest for component kube-apiserver to "/etc/kubernetes/manifests/kube-apiserver.yaml"[controlplane] wrote Static Pod manifest for component kube-controller-manager to "/etc/kubernetes/manifests/kube-controller-manager.yaml"
[controlplane] wrote Static Pod manifest for component kube-scheduler to "/etc/kubernetes/manifests/kube-scheduler.yaml"[etcd] Wrote Static Pod manifest for a local etcd instance to "/etc/kubernetes/manifests/etcd.yaml"
[init] waiting for the kubelet to boot up the control plane as Static Pods from directory "/etc/kubernetes/manifests"
[init] this might take a minute or longer if the control plane images have to be pulled

The connection to the server 172.17.0.2:6443 was refused - did you specify the right host or port?
error: unable to recognize "https://cloud.weave.works/k8s/net?k8s-version=Q2xpZW50IFZlcnNpb246IHZlcnNpb24uSW5mb3tNYWpvcjoiMSIsIE1pbm9yOiIxMSIsIEdpdFZlcnNpb246InYxLjExLjMiLCBHaXRDb21taXQ6ImE0NTI5NDY0ZTQ2MjljMjEyMjRiM2Q1MmVkZmUwZWE5MWIwNzI4NjIiLCBHaXRUcmVlU3RhdGU6ImNsZWFuIiwgQnVpbGREYXRlOiIyMDE4LTA5LTIwVDA5OjAyOjUyWiIsIEdvVmVyc2lvbjoiZ28xLjEwLjMiLCBDb21waWxlcjoiZ2MiLCBQbGF0Zm9ybToibGludXgvYW1kNjQifQo=": Get https://172.17.0.2:6443/api?timeout=32s: dial tcp 172.17.0.2:6443: connect: connection refused
FATA[1007-16:12:53] Failed to create cluster: failed to apply overlay network: exit status 1

[BenTheElder]

This is almost definitely: #22, the current tradeoffs use memory too heavily for docker for Windows / docker for Mac. Even after that issue is closed though most likely we need to recommend that users increase the docker VM / hypervisor memory allocation. At least on docker for Mac the default is too small to fit much of a cluster.

The "out of disk" is likely in the docker overlay inside the container which is a tmpfs currently. I'll switch it to a volume finally today 😬

[kkimdev]

Oh I see. Thanks! =D

[kkimdev]

Increased the ram to 2gb and now it runs smoothly :)

[neolit123]

2GB is indeed required for smooth operations in kubeadm.
ref kubernetes/kubeadm#1052

currently we don't have warnings or system checks as this requires a bunch of OS specific calls that we don't want to include (yet).

one option was to print a message at startup "make sure you have (these-system-reqs)".

[BenTheElder]

Thanks @neolit123 this seems like something I should add to #39

Also, I'm not sure why I chose to allow image-sideloading to fail. Other errors do cause early exit on cluster create.

[BenTheElder]

#22 is now fixed, I'll continue to work on docs and checks for this 👍