Can't run KIND on ChromeOS Linux VM

[adrianludwin]

What happened: I tried to run kind (v0.4.0) on the Linux VM in Chrome OS 75.0.3770.144:

aludwin@penguin:~$ kind create cluster
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.15.0) 🖼
 ✓ Preparing nodes 📦 
 ✗ Creating kubeadm config 📜 
Error: failed to create cluster: failed to get kubernetes version from node: failed to get file: exit status 1

Based on the error and a quick look at the source code, I decided to try to run kindest/node manually, and got the following result:

aludwin@penguin:~$ docker run kindest/node:v1.15.0 -it /bin/bash
mount: /sys: permission denied.

I suspect (but have not tried to confirm) that this comes from the following lines in images/base/entrypoint:

fix_mount() {
  echo 'INFO: ensuring we can execute /bin/mount even with userns-remap'
  # necessary only when userns-remap is enabled on the host, but harmless
  # The binary /bin/mount should be owned by root and have the setuid bit
  chown root:root /bin/mount
  chmod -s /bin/mount

  # This is a workaround to an AUFS bug that might cause `Text file
  # busy` on `mount` command below. See more details in
  # https://github.com/moby/moby/issues/9547
  sync

  echo 'INFO: remounting /sys read-only'
  # systemd-in-a-container should have read only /sys
  # https://www.freedesktop.org/wiki/Software/systemd/ContainerInterface/
  # however, we need other things from `docker run --privileged` ...
  # and this flag also happens to make /sys rw, amongst other things
  mount -o remount,ro /sys

What you expected to happen: Cluster gets created.

How to reproduce it (as minimally and precisely as possible):

If my suspicion that the problems is in the entrypoint is correct, docker run kindest/node:v1.15.0 -it /bin/bash should be sufficient. If that's wrong, kind create cluster reliably reproduces.

Anything else we need to know?:

Environment:

• kind version: v0.4.0
• Kubernetes version: n/a
• Docker version: 19.03.1
• OS: Chrome OS 75.0.3770.144. VM is running Debian 9.

[BenTheElder]

We need to do that remount.

There is also a previous bug for this. I don't think kind will work inside the chromeOS sandbox at the moment.

[BenTheElder]

aludwin@penguin:~$ docker run kindest/node:v1.15.0 -it /bin/bash
mount: /sys: permission denied.

Permission is denied because this command line is missing flags related to that (which kind will include).

If you run kind with --loglevel=debug you can see more detail including this command.

[BenTheElder]

#462

[adrianludwin]

Thanks for the fast response, sorry for missing the other bug. Do we know if this is the last remaining blocker for Chrome OS? And is there a bug assigned to Chrome OS?

…

Sent from my phone

On Fri, 9 Aug 2019, 11:30 am Benjamin Elder, ***@***.***> wrote: We need to do that remount. There is also a previous bug for this. I don't think kind will work inside the chromeOS sandbox at the moment. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#763?email_source=notifications&email_token=AE43PZDDKRT5ZXNF5SWEMPLQDWES7A5CNFSM4IKVC4MKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD37ADNI#issuecomment-519963061>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AE43PZBOERWT6PVTZHUEVE3QDWES7ANCNFSM4IKVC4MA> .

[BenTheElder]

We don't know what the blocker is yet, and I don't have access to a setup suitable for testing this. The existing bug doesn't have enough details either.

This may not even be considered a bug on their end. Kubernetes / the container runtimes need relatively elevated permissions to run at the moment, I suspect the sandbox is too secure :+)

[adrianludwin]

Re #462: I did see that bug but I thought this was something different. My computer successfully got past the "preparing nodes" step but died on the next "Creating kubeadm config" step.

Can I provide any more details that would help with this? Agreed that there's a chance ChromeOS won't want to "fix" this but I'm motivated to push on it a bit :)

Here's the result of the debug logs:

aludwin@penguin:~$ kind create cluster --loglevel=debug
DEBU[12:04:09] Running: /usr/bin/docker [docker ps -q -a --no-trunc --filter label=io.k8s.sigs.kind.cluster --format {{.Names}}\t{{.Label "io.k8s.sigs.kind.cluster"}}] 
Creating cluster "kind" ...
DEBU[12:04:09] Running: /usr/bin/docker [docker inspect --type=image kindest/node:v1.15.0] 
INFO[12:04:09] Image: kindest/node:v1.15.0 present locally  
 ✓ Ensuring node image (kindest/node:v1.15.0) 🖼
DEBU[12:04:09] Running: /usr/bin/docker [docker info --format '{{json .SecurityOptions}}'] 
DEBU[12:04:09] Running: /usr/bin/docker [docker run -d -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname kind-control-plane --name kind-control-plane --label io.k8s.sigs.kind.cluster=kind --label io.k8s.sigs.kind.role=control-plane --expose 43283 --publish=127.0.0.1:43283:6443 kindest/node:v1.15.0@sha256:b4d092fd2b507843dd096fe6c85d06a27a0cbd740a0b32a880fe61aba24bb478] 
 ✓ Preparing nodes 📦 
DEBU[12:04:13] Running: /usr/bin/docker [docker ps -q -a --no-trunc --filter label=io.k8s.sigs.kind.cluster --format {{.Names}}\t{{.Label "io.k8s.sigs.kind.cluster"}} --filter label=io.k8s.sigs.kind.cluster=kind] 
DEBU[12:04:13] Running: /usr/bin/docker [docker inspect -f {{index .Config.Labels "io.k8s.sigs.kind.role"}} kind-control-plane] 
DEBU[12:04:14] Running: /usr/bin/docker [docker exec --privileged kind-control-plane cat /kind/version] 
 ✗ Creating kubeadm config 📜 
DEBU[12:04:14] Running: /usr/bin/docker [docker ps -q -a --no-trunc --filter label=io.k8s.sigs.kind.cluster --format {{.Names}}\t{{.Label "io.k8s.sigs.kind.cluster"}} --filter label=io.k8s.sigs.kind.cluster=kind] 
DEBU[12:04:14] Running: /usr/bin/docker [docker rm -f -v kind-control-plane] 
Error: failed to create cluster: failed to get kubernetes version from node: failed to get file: exit status 1

I'll try to reproduce this more precisely.

[adrianludwin]

Running the docker run command on its own, without -d, has the same result I got initially:

aludwin@penguin:~$ docker run -t --privileged --security-opt seccomp=unconfined --tmpfs /tmp --tmpfs /run -v /lib/modules:/lib/modules:ro --hostname kind-control-plane --name kind-control-plane --label io.k8s.sigs.kind.cluster=kind --label io.k8s.sigs.kind.role=control-plane --expose 43283 --publish=127.0.0.1:43283:6443 kindest/node:v1.15.0@sha256:b4d092fd2b507843dd096fe6c85d06a27a0cbd740a0b32a880fe61aba24bb478
mount: /sys: permission denied.

[BenTheElder]

How was docker installed?
What other security mechanisms does this system have?
Can you do mounts in a container at all?
If you run the same thing but -it --entrypoint=/bin/bash can you do mount -l to list the mounts?

[adrianludwin]

How was docker installed?

I followed the instructions for Docker-on-Pixelbook (basically sudo apt-get install docker-ce), using the Debian repo (not Ubuntu). I didn't need any of the stuff about a blacklisted syscall - it Just Worked for me when I installed it.

What other security mechanisms does this system have?

I have no idea. AFAIK it's stock ChromeOS but it's managed by my employer (Google) so they may have some additional stuff on it.

Can you do mounts in a container at all?

I can do a bind mount, yes:

$ mkdir /mnt/test
$ mkdir /home/aludwin
$ echo hi > /mnt/test/foo
$ mount --bind /mnt/test /home/aludwin
$ cat /home/aludwin/foo
hi

My VM and container don't have any block devices other than vdb and I'm not sure how to mount that (mount: /home/aludwin: special device vdb does not exist.). Is this enough to answer your question?

If you run the same thing but -it --entrypoint=/bin/bash can you do mount -l to list the mounts?

Yup:

root@kind-control-plane:/# mount -l
/dev/vdb on / type btrfs (rw,relatime,discard,space_cache,user_subvol_rm_allowed,subvolid=379,subvol=/lxd/storage-pools/default/containers/penguin/rootfs/var/lib/docker/btrfs/subvolumes/e0cd2ec9e5417357be09b8d36540d00f596f0f0d127a346faba8f67096bbb990)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev type tmpfs (rw,nosuid,size=65536k,mode=755,uid=1000000,gid=1000000)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=1000005,mode=620,ptmxmode=666)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,relatime,mode=755,uid=1000000,gid=1000000)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,name=systemd)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,pids)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,perf_event)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls,net_prio)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,hugetlb)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
mqueue on /dev/mqueue type mqueue (rw,nosuid,nodev,noexec,relatime)
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k,uid=1000000,gid=1000000)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,noexec,relatime,uid=1000000,gid=1000000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,noexec,relatime,uid=1000000,gid=1000000)
/dev/vdb on /etc/resolv.conf type btrfs (rw,relatime,discard,space_cache,user_subvol_rm_allowed,subvolid=265,subvol=/lxd/storage-pools/default/containers/penguin/rootfs/var/lib/docker/containers/c7cf5109d3d26d6ae53bf527f2ed790d169a8d8e85edb6adb3fd137f22ae518b/resolv.conf)
/dev/vdb on /etc/hostname type btrfs (rw,relatime,discard,space_cache,user_subvol_rm_allowed,subvolid=265,subvol=/lxd/storage-pools/default/containers/penguin/rootfs/var/lib/docker/containers/c7cf5109d3d26d6ae53bf527f2ed790d169a8d8e85edb6adb3fd137f22ae518b/hostname)
/dev/vdb on /etc/hosts type btrfs (rw,relatime,discard,space_cache,user_subvol_rm_allowed,subvolid=265,subvol=/lxd/storage-pools/default/containers/penguin/rootfs/var/lib/docker/containers/c7cf5109d3d26d6ae53bf527f2ed790d169a8d8e85edb6adb3fd137f22ae518b/hosts)
/dev/vdb on /usr/lib/modules type btrfs (ro,relatime,discard,space_cache,user_subvol_rm_allowed,subvolid=265,subvol=/lxd/storage-pools/default/containers/penguin/rootfs/lib/modules)
/dev/vdb on /var/lib/containerd type btrfs (rw,relatime,discard,space_cache,user_subvol_rm_allowed,subvolid=265,subvol=/lxd/storage-pools/default/containers/penguin/rootfs/var/lib/docker/volumes/56ea3686b15c669ecb98b632b1fc1094e1c6bd2b5679f060ce74f4e48b4c457b/_data)
devtmpfs on /dev/null type devtmpfs (rw,nosuid,noexec,relatime,size=5572304k,nr_inodes=1393076,mode=755)
devtmpfs on /dev/random type devtmpfs (rw,nosuid,noexec,relatime,size=5572304k,nr_inodes=1393076,mode=755)
devtmpfs on /dev/full type devtmpfs (rw,nosuid,noexec,relatime,size=5572304k,nr_inodes=1393076,mode=755)
devtmpfs on /dev/tty type devtmpfs (rw,nosuid,noexec,relatime,size=5572304k,nr_inodes=1393076,mode=755)
devtmpfs on /dev/zero type devtmpfs (rw,nosuid,noexec,relatime,size=5572304k,nr_inodes=1393076,mode=755)
devtmpfs on /dev/urandom type devtmpfs (rw,nosuid,noexec,relatime,size=5572304k,nr_inodes=1393076,mode=755)
devpts on /dev/console type devpts (rw,nosuid,noexec,relatime,gid=1000005,mode=620,ptmxmode=666)

[BenTheElder]

xref: https://bugs.chromium.org/p/chromium/issues/detail?id=878034

[rosera]

Hey, I am highly interested in getting this working on ChromeOS (I raised the original ticket reference https://bugs.chromium.org/p/chromium/issues/detail?id=878034). Thanks very much for the tips regarding debugging - very helpful.

I plan to investigate a bit more over the coming weekend. My thoughts at the moment relate to the container being restricted on the accessible ports available to Crostini. K8s ports are not on the existing list of default ports. I believe you can enable them, so it might be trial and error for this to work. Beyond that there is also the use of a privileged container - which I understood was not supported on Crostini? This is more of an issue and one which I do not think will be enabled due to security concerns.

[BenTheElder]

Hi! 👋

K8s ports are not on the existing list of default ports

you can configure kind to use a particular port with kind create cluster --config=config.yaml and config.yaml:

apiVersion: kind.sigs.k8s.io/v1alpha3
kind: Config
networking:
  apiServerPort: 6443

Beyond that there is also the use of a privileged container - which I understood was not supported on Crostini? This is more of an issue and one which I do not think will be enabled due to security concerns.

That's a bit more of a blocker.

Each kind "node" docker container is --privileged in order to run containerd / pods on the inside, each kubelet needs its own container runtime. To run pods we need proc, cgroups, mounting ...

It might be possible to hack up kind to use usermode Linux instead but that's going to be fairly involved.

If I read the logs from @adrianludwin correctly though, we actually manage to create those containers, but it fails once we try to use mount inside them. Besides the re-mounting we do*, I believe kubelet needs permission to mount in order to setup pods etc.

* (changing /sys to read only, docker forces /sys to a read-write mount if you specify --privileged),

[BenTheElder]

dupe: #1137

I don't really have the capacity to dig into this further at the moment, and I don't think any of our other usual contributors do. AFAICT, the chromeOS sandbox is too restrictive to run Kubernetes.

This is probably generally a good thing 🙃 as kind is certainly not the most secure thing to be doing, but there's not much we can do here.

At this point I'm going to recommend that we add this to known issues and close out this bug in favor of the upstream chromeOS issue.

[cbandy]

If I understand correctly, that upstream bug is targetting minikube (rather than kind specifically) and also it might be ... done/terminal/finished/closed?

Status: Verified (Closed)

[BenTheElder]

I don't know if the upstream bug is actually fixed, but if you read through the comments the upstream bug expands to more generally trying to run some form of kubernetes-like-thing including kind, minikube, k3s...

kind is not doing anything special for operating systems otherwise, we only require privileged docker containers and whatever kernel features kubernetes requires.

[BenTheElder]

See some more background on /sys specifically in #1474, but this is a red-herring anyhow, you cannot get far without permission to manipulate mounts (you can't really run containers / kubelet without this...), AIUI chromeOS sandbox is preventing this, not /sys specifically, so #1474 will not change that chromeOS cannot work.

[stmcginnis]

Can this be closed now? Notes regarding ChromeOS were added in #1138 - I don't think there is much more that can be done from the kind side of things unless or until changes are made to ChromeOS to enable this.

[BenTheElder]

Yeah, the crostini sandbox is too locked down. That makes sense really, Kubernetes is also for managing Linux and uses many privileged kernel APIs

[rosera]

FYI - ChromeOS supports minikube. I havent tried Kind, so not sure if this works. I wrote how to run Minikube here https://richrose.dev/posts/chromeos/productivity/chromeos-minikube/

[ck1125]

Thanks for sharing @rosera Sounds promising...will give it a whir and report back...

[BenTheElder]

thanks @rosera, cross posting this at kubernetes/minikube#6411

I havent tried Kind, so not sure if this works.

KIND doesn't use VMs (it uses container nodes), but if you can create arbitrary VMs you could create a VM to run kind in.

[keyonjie]

sn't use VMs (it uses container no

Thanks for sharing @rosera . I am going to try what you have shared. Can you share more details about your environment? e.g. we should perform all commands on ChromeOS/crostini/debian (or ubuntu? 20.04?)

[rosera]

I used ChromeOS debian (Crostini) environment, no additional changes were required. I used a Pixelbook i7 16Gb and a Asus CN60 i7 16Gb to test. I posted the blog post over a year ago, so may require some changes?

[keyonjie]

@rosera it fails at my chromebook (Debian 11):
$ minikube start --driver=kvm2
😄 minikube v1.26.1 on Debian 11.4 (amd64)
✨ Using the kvm2 driver based on existing profile
👍 Starting control plane node minikube in cluster minikube
🔄 Restarting existing kvm2 VM for "minikube" ...
🤦 StartHost failed, but will try again: driver start: error creating VM: virError(Code=38, Domain=0, Message='Unable to set XATTR trusted.libvirt.security.dac on /var/lib/libvirt/qemu/domain-3-minikube/master-key.aes: Operation not permitted')
🔄 Restarting existing kvm2 VM for "minikube" ...
😿 Failed to start kvm2 VM. Running "minikube delete" may fix it: driver start: error creating VM: virError(Code=38, Domain=0, Message='Unable to set XATTR trusted.libvirt.security.dac on /var/lib/libvirt/qemu/domain-4-minikube/master-key.aes: Operation not permitted')

❌ Exiting due to GUEST_PROVISION: Failed to start host: driver start: error creating VM: virError(Code=38, Domain=0, Message='Unable to set XATTR trusted.libvirt.security.dac on /var/lib/libvirt/qemu/domain-4-minikube/master-key.aes: Operation not permitted')

╭───────────────────────────────────────────────────────────────────────────────────────────╮
│ │
│ 😿 If the above advice does not help, please let us know: │
│ 👉 https://github.com/kubernetes/minikube/issues/new/choose │
│ │
│ Please run minikube logs --file=logs.txt and attach logs.txt to the GitHub issue. │
│ │
╰───────────────────────────────────────────────────────────────────────────────────────────╯

[keyonjie]

oh, it works after changing the /etc/libvirt/qemu.conf and rebooting the chrome book! Thanks @cbandy !
add these lines:
user = "root"
group = "root"
remember_owner = 0

[BenTheElder]

If you can get an arbitrary VM running you should be able to run docker or podman + kind inside that using the usual linux install etc. 👀

[rosera]

Thanks @keyonjie I updated my blog with the additional steps.

[dex4er]

I would like to inform you I can run KinD on ChromeOS: either with docker or with podman (running from root).

First I enabled nesting mode for LXC container: it is requirement for Docker on ChromeOS. After ctrl+alt+t in Chrome:

vmc launch termina
lxc config set penguin security.nesting true
lxc restart penguin

Then:

cat > cluster.yaml << END 
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  KubeletInUserNamespace: true
END
kind create cluster --config cluster.yaml

The reason why KinD fails is LXC container misses some devices (/dev/kmsg, /dev/mapper/control) then kubelet panics.

Also rootless setup doesn't work because either it doesn't have bridge-nf-call-iptables or cgroups v2. I think it is not a problem if KubeletInUserNamespace is used and docker or podman are started from root.

[BenTheElder]

xref: #3101

[maxamillion]

@dex4er this doesn't seem to work for debian 12 bookworm crostini, is there a new trick that needs to be enacted?

$ docker --version
Docker version 24.0.7, build afdd53b

$ kind --version
kind version 0.20.0

$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

To verify the nesting in vmc:

(termina) chronos@localhost ~ $ lxc config get penguin security.nesting
true

And finally, attempting to create a cluster:

$ cat > cluster.yaml << END
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  KubeletInUserNamespace: true
END

$ kind create cluster --config cluster.yaml
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.27.3) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✗ Starting control-plane 🕹️
Deleted nodes: ["kind-control-plane"]
ERROR: failed to create cluster: failed to init node with kubeadm: command "docker exec --privileged kind-control-plane kubeadm init --skip-phases=preflight --config=/kind/kubeadm.conf --skip-token-print --v=6" failed with error: exit status 1

Thoughts?

Thank you in advance!

[dex4er]

@maxamillion I'm sorry. It works for me: very fresh ChromeOS with bookworm (fresh install, not an upgrade) with Docker 24.0.7 and kind v0.20.0:

dex4er@penguin:~/tmp$ cat cluster.yaml 
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
featureGates:
  KubeletInUserNamespace: true
dex4er@penguin:~/tmp$ kind create cluster --config cluster.yaml 
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.27.3) 🖼 
 ✓ Preparing nodes 📦  
 ✓ Writing configuration 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Not sure what to do next? 😅  Check out https://kind.sigs.k8s.io/docs/user/quick-start/
dex4er@penguin:~/tmp$ docker ps
CONTAINER ID   IMAGE                  COMMAND                  CREATED          STATUS          PORTS                       NAMES
862e3ffebc57   kindest/node:v1.27.3   "/usr/local/bin/entr…"   54 seconds ago   Up 52 seconds   127.0.0.1:41295->6443/tcp   kind-control-plane
dex4er@penguin:~/tmp$ kubectl version
Client Version: v1.29.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.3

You can try docker logs -f kind-control-plane for troubleshooting. Also:

$ docker exec -it kind-control-plane bash
root@kind-control-plane:/# journalctl -u kubelet