bug(al2): QEMU User Static Fails on Amazon EKS Node 1.30 (v20240924)

[smittnacht]

What happened:
Since the release of amazon-eks-node-1.30-v20240924, running multiarch/qemu-user-static fails to register binfmt handlers properly.

While debugging with Docker, I observed this behavior on both a standalone environment and an EKS 1.30 cluster using the same AMI.

On Instance with amazon-eks-node-1.30-v20240917

[root@ip-172-31-34-162 bin]# systemctl start docker
[root@ip-172-31-34-162 bin]# docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
Setting /usr/bin/qemu-armeb-static as binfmt interpreter for armeb
Setting /usr/bin/qemu-sparc-static as binfmt interpreter for sparc
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
Setting /usr/bin/qemu-sparc64-static as binfmt interpreter for sparc64
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
Setting /usr/bin/qemu-aarch64_be-static as binfmt interpreter for aarch64_be
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
Setting /usr/bin/qemu-riscv32-static as binfmt interpreter for riscv32
Setting /usr/bin/qemu-riscv64-static as binfmt interpreter for riscv64
Setting /usr/bin/qemu-xtensa-static as binfmt interpreter for xtensa
Setting /usr/bin/qemu-xtensaeb-static as binfmt interpreter for xtensaeb
Setting /usr/bin/qemu-microblaze-static as binfmt interpreter for microblaze
Setting /usr/bin/qemu-microblazeel-static as binfmt interpreter for microblazeel
Setting /usr/bin/qemu-or1k-static as binfmt interpreter for or1k
Setting /usr/bin/qemu-hexagon-static as binfmt interpreter for hexagon
[root@ip-172-31-34-162 bin]# docker run --rm --platform linux/arm64 -t arm64v8/ubuntu uname -m
aarch64

On Instance with amazon-eks-node-1.30-v20240924 the command doesn't work as expected

[root@ip-172-31-44-104 bin]# systemctl start docker
[root@ip-172-31-44-104 bin]# docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
Setting /usr/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /usr/bin/qemu-arm-static as binfmt interpreter for arm
Setting /usr/bin/qemu-armeb-static as binfmt interpreter for armeb
Setting /usr/bin/qemu-sparc-static as binfmt interpreter for sparc
Setting /usr/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
Setting /usr/bin/qemu-sparc64-static as binfmt interpreter for sparc64
Setting /usr/bin/qemu-ppc-static as binfmt interpreter for ppc
Setting /usr/bin/qemu-ppc64-static as binfmt interpreter for ppc64
Setting /usr/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
Setting /usr/bin/qemu-m68k-static as binfmt interpreter for m68k
Setting /usr/bin/qemu-mips-static as binfmt interpreter for mips
Setting /usr/bin/qemu-mipsel-static as binfmt interpreter for mipsel
Setting /usr/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
Setting /usr/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
Setting /usr/bin/qemu-mips64-static as binfmt interpreter for mips64
Setting /usr/bin/qemu-mips64el-static as binfmt interpreter for mips64el
Setting /usr/bin/qemu-sh4-static as binfmt interpreter for sh4
Setting /usr/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
Setting /usr/bin/qemu-s390x-static as binfmt interpreter for s390x
Setting /usr/bin/qemu-aarch64-static as binfmt interpreter for aarch64
Setting /usr/bin/qemu-aarch64_be-static as binfmt interpreter for aarch64_be
Setting /usr/bin/qemu-hppa-static as binfmt interpreter for hppa
Setting /usr/bin/qemu-riscv32-static as binfmt interpreter for riscv32
Setting /usr/bin/qemu-riscv64-static as binfmt interpreter for riscv64
Setting /usr/bin/qemu-xtensa-static as binfmt interpreter for xtensa
Setting /usr/bin/qemu-xtensaeb-static as binfmt interpreter for xtensaeb
Setting /usr/bin/qemu-microblaze-static as binfmt interpreter for microblaze
Setting /usr/bin/qemu-microblazeel-static as binfmt interpreter for microblazeel
Setting /usr/bin/qemu-or1k-static as binfmt interpreter for or1k
Setting /usr/bin/qemu-hexagon-static as binfmt interpreter for hexagon
[root@ip-172-31-44-104 bin]# docker run --rm --platform linux/arm64 -t arm64v8/ubuntu uname -m
exec /usr/bin/uname: exec format error

What you expected to happen:
QEMU should register with binfmt correctly, allowing cross-architecture execution as before.

How to reproduce it (as minimally and precisely as possible):

• launch instance with amazon-eks-node-1.30-v20240917
• install docker (yum install docker)
• start docker daemon (systemctl start docker)
• install qemu-user-static (docker run --rm --privileged multiarch/qemu-user-static --reset -p yes)
• check if it works (docker run --rm --platform linux/arm64 -t arm64v8/ubuntu uname -m)

Environment:

• AWS Region: -
• Instance Type(s): -
• Cluster Kubernetes version: 1.30
• Node Kubernetes version: 1.30
• AMI Version: ami-08ca62f441108a7c6

[cartermckinnon]

What version of Docker was installed in each case?

[smittnacht]

What version of Docker was installed in each case?

on instance with AMI amazon-eks-node-1.30-v20240924
sh-4.2$ docker --version
Docker version 25.0.6, build 32b99dd

on instance with AMI amazon-eks-node-1.30-v20240917
sh-4.2$ docker --version
Docker version 25.0.6, build 32b99dd

so both use the exact same version

[cartermckinnon]

@smittnacht I think you have a support case open for this? The issue is due to a kernel change in Amazon Linux that should be addressed in the next AL release. That will land in the EKS AMI builds as soon as it's available. I'll leave this issue open for tracking.