docker in kubernetes

[steviehs]

For the syncosync build, we have set up our own gitlab runners on kubernetes and run a docker inside it, which worked from time to time...

now, there is a strange issue:
from CustomPiOS everything looks fine... until the chroot with qemuaarch64 happens: qemu states a wrong exec format...
I have tried now several hours to debug - therefore I forked CustomPiOS to add debug info - but I have no glue...

https://gitlab.com/syncosync/syncosync/-/jobs/2221508446

shows the pipeline constantly failing and the issue

https://gitlab.com/syncosync/syncosync/-/jobs/2221508446#L816 states the architecture of "touch"
https://gitlab.com/syncosync/syncosync/-/jobs/2221508446#L852 states the architecture is wrong...

Any ideas what could get wrong?

Running the docker locally on my debian works fine... so I guess, it has something to do with loop mounts and privileges or so.

Sorry for my vague request, but I am rather glueless... and I hate the idea of setting up bare metal idling around to build a new distro once a week...

[guysoft]

The error is

/chroot_script: line 81: /usr/bin/touch: cannot execute binary file: Exec format error

usr/bin/touch: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=55caec4d0916cca9fdb58167c5a7a8b78a9d2d2e, for GNU/Linux 3.7.0, stripped
+ ls -la usr/bin/touch

It would seem that this line is not really executing qemu-aarch64-static:

chroot . usr/bin/qemu-aarch64-static /bin/bash /chroot_script

If anyone has gitlab experience that could help

[woodcoder]

I ended up here while trying to run CustomPiOS on Ubuntu 22.04 under Parallels on new Apple silicon (ARM64).

Initially the Exec format error happened when trying to run /bin/bash at line 69 of the custompios script. This happened because the assumption that arm(fh) 32-bit code is supported by all aarch64 chips is not true. [See this discussion for more info, but lack of support can be detected by calls to dpkg --print-foreign-architectures or lscpu | grep op-mode.]

Hacking the custompios script to use qemu-arm-static in this case allowed me to run bash successfully - but then gave me the exact same error as in this issue - the Exec format error for /usr/bin/touch.

I believe this is because while the shell runs under emulation, other programs that the shell starts aren't automatically run under qemu.

In my situation I eventually traced this to binfmts not being setup correctly - which is what normally solves this problem. My workaround is to create the missing /var/lib/binfmts/qemu-arm config with the arm magic numbers and enable it with sudo update-binfmts --enable qemu-arm. This allows the ELF 32-bit code to run on the M1/M2 chips and CustomPiOS to run out of the box (I could even undo my changes to the custompios script).
__

For this issue, I therefore think it's likely that qemu-aarch64-static is being executed successfully (otherwise the error would been seen for bash, not touch).

So instead I wondered if this issue might be solved by running update-binfmts --enable qemu-aarch64 - the way that the custompios script does when .dockerenv is present?

[guysoft]

Update binfmts should happen here:

CustomPiOS/src/custompios

Line 21 in 63da54b

update-binfmts --enable qemu-aarch64