Is kind w/ podman provisioner for installing k8s v1.24.0 in a WSL instance? #2766

Unveraenderbar · 2022-05-13T16:37:56Z

I know that kind+podman cannot run rootless in a WSL instance -- but installation when running as superuser works fine for k8s v1.23.6
(using podman as runtime).
However, when trying to install k8s v1.24.0 within the same environment, the kubelets in the podman containers fail w/ errors

$ sudo sh -c 'podman exec $(podman ps --filter label=io.x-k8s.kind.role=control-plane -q) journalctl -xeu kubelet' | grep -A 10 'kubeletVersion="v1.24.0"'
May 13 10:34:17 k8s-v1.24.0-control-plane kubelet[1044]: I0513 10:34:17.026601    1044 server.go:399] "Kubelet version" kubeletVersion="v1.24.0"
May 13 10:34:17 k8s-v1.24.0-control-plane kubelet[1044]: I0513 10:34:17.026627    1044 server.go:401] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
May 13 10:34:17 k8s-v1.24.0-control-plane kubelet[1044]: I0513 10:34:17.027340    1044 server.go:813] "Client rotation is on, will bootstrap in background"
May 13 10:34:17 k8s-v1.24.0-control-plane kubelet[1044]: I0513 10:34:17.028219    1044 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubele
t-client-current.pem".
May 13 10:34:17 k8s-v1.24.0-control-plane kubelet[1044]: I0513 10:34:17.028665    1044 container_manager_linux.go:848] "CPUAccounting not enabled for process" pid=1044
May 13 10:34:17 k8s-v1.24.0-control-plane kubelet[1044]: I0513 10:34:17.028683    1044 container_manager_linux.go:851] "MemoryAccounting not enabled for process" pid=10
44
May 13 10:34:17 k8s-v1.24.0-control-plane kubelet[1044]: I0513 10:34:17.028733    1044 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc
/kubernetes/pki/ca.crt"
May 13 10:34:17 k8s-v1.24.0-control-plane kubelet[1044]: W0513 10:34:17.029821    1044 sysinfo.go:203] Nodes topology is not available, providing CPU topology
May 13 10:34:17 k8s-v1.24.0-control-plane kubelet[1044]: Error: failed to run Kubelet: invalid configuration: cgroup ["kubelet"] has some missing paths: /sys/fs/cgroup/
systemd/kubelet.slice
May 13 10:34:17 k8s-v1.24.0-control-plane kubelet[1044]: Usage:
May 13 10:34:17 k8s-v1.24.0-control-plane kubelet[1044]:   kubelet [flags]
...

I suppose this is triggered by the woefully outdated WSL 5 10 kernel as it (as far as I understand it) sets up a weird mix of cgroups and cgroups2 mounts.
So, is kind + k8s v1.24.0 + podman out of reach on WSL instances, or can the behaviour shown above considered a bug?
Or am I missing a piece of configuration that would enable kind to configure cgroups for v1.24.0 the same way it does for older k8s versions?

The text was updated successfully, but these errors were encountered:

BenTheElder · 2022-05-13T16:40:17Z

https://github.com/kubernetes-sigs/kind/releases/tag/v0.13.0#breaking-changes
#2765

It would be helpful to know more about the environment e.g. the kind version

We don't have WSL environment #1529, definitely not WSL + podman. We have community documented WSL + docker.

Unveraenderbar · 2022-05-13T16:45:14Z

Sorry -- it's kind v0.13.0

BenTheElder · 2022-05-13T16:49:03Z

can you share podman info ?

IIRC wsl distros don't run systemd, it sounds like (cgroupv1, cgroupfs) on the host is broken with the v1.24 kind change #2765

Unveraenderbar · 2022-05-13T16:59:32Z

It's podman 3.4, installed from OpenSuse kubic repository (I tried also 4.1.0 compiled from sources, but w/ the same result)

host:
  arch: amd64
  buildahVersion: 1.23.1
  cgroupControllers: []
  cgroupManager: cgroupfs
  cgroupVersion: v1
  conmon:
    package: 'conmon: /usr/libexec/podman/conmon'
    path: /usr/libexec/podman/conmon
    version: 'conmon version 2.1.0, commit: '
  cpus: 16
  distribution:
    codename: focal
    distribution: ubuntu
    version: "20.04"
  eventLogger: file
  hostname: P7560-027WX
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 5.10.16.3-microsoft-standard-WSL2
  linkmode: dynamic
  logDriver: k8s-file
  memFree: 9635094528
  memTotal: 26234175488
  ociRuntime:
    name: crun
    package: 'crun: /usr/bin/crun'
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: ea1fe3938eefa14eb707f1d22adff4db670645d6
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    exists: true
    path: /tmp/podman-run-1000/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: 'slirp4netns: /usr/bin/slirp4netns'
    version: |-
      slirp4netns version 1.1.8
      commit: unknown
      libslirp: 4.3.1-git
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.4.3
  swapFree: 7515918336
  swapTotal: 7516192768
  uptime: 29h 30m 48s (Approximately 1.21 days)
plugins:
  log:
  - k8s-file
  - none
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - docker.io
  - quay.io
store:
  configFile: /home/user/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: vfs
  graphOptions: {}
  graphRoot: /home/user/.local/share/containers/storage
  graphStatus: {}
  imageStore:
    number: 7
  runRoot: /tmp/containers-user-1000/containers
  volumePath: /home/user/.local/share/containers/storage/volumes
version:
  APIVersion: 3.4.2
  Built: 0
  BuiltTime: Thu Jan  1 01:00:00 1970
  GitCommit: ""
  GoVersion: go1.16.6
  OsArch: linux/amd64
  Version: 3.4.2

BenTheElder · 2022-05-13T17:03:28Z

Thanks

cgroupManager: cgroupfs
cgroupVersion: v1

Continues to look very similar to #2765

BenTheElder · 2022-05-13T17:07:14Z

#2765 (comment) for how to force v1.24.0 kind image to handle cgroups the same as v1.23.6 until we can resolve this bug.

Unveraenderbar · 2022-05-13T17:10:12Z

oops forgot to run podman info as superuser (as it has to be done when running kind under WSL) -- but of course, it's the very same regarding cgroups.

Unveraenderbar · 2022-05-13T17:11:15Z

#2765 (comment) for how to force v1.24.0 kind image to handle cgroups the same as v1.23.6 until we can resolve this bug.

Thanks a lot!!
I'll try to force cgroups ASAP

BenTheElder · 2022-05-13T17:13:09Z

Can you also try running as superuser with kind v0.13.0 kind create cluster --retain (using the defaults just like this, no additional flags) , kind export logs (after it fails) and then you can run kind delete cluster to cleanup afterwards, and share the resulting logs directory from kind export logs?

--retain prevents immediate cleanup on failure, we can see what the node logs are this way and delete after.

I spotted something in the node logs in #2765 and I want to see if we have something similar here.

Unveraenderbar · 2022-05-13T17:50:41Z

Here the logs
kind-logs.tar.gz

BenTheElder · 2022-05-13T17:53:27Z

May 13 17:19:43 k8s-v1.24.0-control-plane kubelet[248]: Error: failed to run Kubelet: invalid configuration: cgroup ["kubelet"] has some missing paths: /sys/fs/cgroup/systemd/kubelet.slice

thanks!

Unveraenderbar · 2022-05-13T18:01:23Z

#2765 (comment) for how to force v1.24.0 kind image to handle cgroups the same as v1.23.6 until we can resolve this bug.

With the configuration referenced above, a kind cluster w/ v1.24.0 on a WSL2 is up&running -- very cool, thanks again!

BenTheElder · 2022-05-13T18:16:49Z

Great! so this is the same root bug as #2765, we'll work on fixing it so this workaround isn't needed in the future, sorry about that!

BenTheElder · 2022-05-13T20:35:48Z

bentheelder/kind-node:v1.24.0 has #2767, but I lack the handy access to testing this environment, if you could try kind v0.13.0 with kind create cluster --image=bentheelder/kind-node:v1.24.0 --retain and report success or share the logs that would be great!

Unveraenderbar · 2022-05-13T21:31:21Z

Seems to crash, w/ the same kubelet error message
kind-log-2767.tar.gz

BenTheElder · 2022-05-19T04:50:38Z

Hi @Unveraenderbar, sorry for the delay.

I've tested an updated fix in #2767 on colima xref #2778 which should be the same root issue (kind v0.13.0 / v1.24 imagse brokene on non-systemd hosts)

Please let me know if this updated image works with kind v0.13.0
kind create cluster --image=bentheelder/kind-node:v1.24.0@sha256:cf92db251fb3f420634ec46d9c222812fee4d75f1e0105a325e57679b0f57370

Hopefully that should be it, and we'll cut another release soon with this fix.

Unveraenderbar · 2022-05-19T09:43:50Z

Hi @BenTheElder,
using your image @cf92db25, kind (running w/ superuser privileges) provisions k8s fine under WSL (w/ standard distro Ubuntu 20.04 + podman 3.4.7).
Lot of thanks for fixing this so quick!
And looking forward to kind v0.14 :)

BenTheElder · 2022-05-19T21:00:23Z

https://github.com/kubernetes-sigs/kind/releases/tag/v0.14.0

Unveraenderbar added the kind/support Categorizes issue or PR as a support question. label May 13, 2022

BenTheElder mentioned this issue May 13, 2022

[rancher desktop] v0.13.0 not starting in Mac (Intel) #2765

Closed

BenTheElder self-assigned this May 13, 2022

This was referenced May 13, 2022

entrypoint cleanup + non-systemd-host fix #2767

Merged

[WSL 2] kind v0.13.0 can't create a cluster #2769

Closed

kind v0.13.0 does not work with Colima 0.4.1 #2778

Closed

BenTheElder closed this as completed May 19, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Is kind w/ podman provisioner for installing k8s v1.24.0 in a WSL instance? #2766

Is kind w/ podman provisioner for installing k8s v1.24.0 in a WSL instance? #2766

Unveraenderbar commented May 13, 2022

BenTheElder commented May 13, 2022 •

edited

Loading

Unveraenderbar commented May 13, 2022

BenTheElder commented May 13, 2022

Unveraenderbar commented May 13, 2022

BenTheElder commented May 13, 2022

BenTheElder commented May 13, 2022 •

edited

Loading

Unveraenderbar commented May 13, 2022

Unveraenderbar commented May 13, 2022

BenTheElder commented May 13, 2022

Unveraenderbar commented May 13, 2022

BenTheElder commented May 13, 2022

Unveraenderbar commented May 13, 2022

BenTheElder commented May 13, 2022

BenTheElder commented May 13, 2022

Unveraenderbar commented May 13, 2022

BenTheElder commented May 19, 2022

Unveraenderbar commented May 19, 2022

BenTheElder commented May 19, 2022

Is kind w/ podman provisioner for installing k8s v1.24.0 in a WSL instance? #2766

Is kind w/ podman provisioner for installing k8s v1.24.0 in a WSL instance? #2766

Comments

Unveraenderbar commented May 13, 2022

BenTheElder commented May 13, 2022 • edited Loading

Unveraenderbar commented May 13, 2022

BenTheElder commented May 13, 2022

Unveraenderbar commented May 13, 2022

BenTheElder commented May 13, 2022

BenTheElder commented May 13, 2022 • edited Loading

Unveraenderbar commented May 13, 2022

Unveraenderbar commented May 13, 2022

BenTheElder commented May 13, 2022

Unveraenderbar commented May 13, 2022

BenTheElder commented May 13, 2022

Unveraenderbar commented May 13, 2022

BenTheElder commented May 13, 2022

BenTheElder commented May 13, 2022

Unveraenderbar commented May 13, 2022

BenTheElder commented May 19, 2022

Unveraenderbar commented May 19, 2022

BenTheElder commented May 19, 2022

BenTheElder commented May 13, 2022 •

edited

Loading

BenTheElder commented May 13, 2022 •

edited

Loading