Sharing my first contact experience

[jmaerki]

Hey there,
when I first stumbled over Kairos a short while ago, I thought: exactly what I'm looking for. Since I've hit a few road bumps trying it out (over about 2 days), I thought I'd share that...

Some background on where I come from: I'm running multiple k3s clusters for myself and others and I'm about to help a client bring up a more modern infrastructure but I have to keep things as simple as possible, so Kairos looks like a good candidate to help with just that. I'm also looking for a good way to set up edge clusters "as-a-box" for environments with no k8s knowledge. In my experiments, I have a solid iPXE setup that I use in my network to PXE boot all sorts of bare metal and VMs (on Proxmox and VirtualBox).

First of all, what cost me quite some time was the fact that I managed to overlook the "Note" on the "Getting Started" page. When I had the first installs working, I was wondering where my k3s was. I'm not sure how to better document that for guys like me who lose themselves in details and manage to miss the bigger points. I have no defense for my failure.

I haven't progressed to custom images, yet, so my installs always seem to load the images from the internet. Even though I have a 1 GBit/s internet line, the processes seemed rather slow compared to booting a basic Debian install. The boot log doesn't really inform that much over boot/copy progress. On my slower test machines, it could take over 2 minutes to have something show up indicating that booting/installing is still running properly. Also, machine shutdowns after a fresh install from the k3s provider images seems very slow. I had to learn to trust the process and be patient.

I tried different images. I run my own k3s nodes mostly on Debian Bullseye nowadays, so I wanted to try that. I started out on Kairos 1.5 and had some cases where progress really just stopped (hang) which made me look towards alpine-ubuntu and opensuse-leap. After finding 1.6.1 things got better. But after finding out that the Debian variant seems to be on "testing" repos, I started looking towards ubuntu-lts-22 which I ended up going for with my latest tests. With this I'm now able to reproducibly PXE-install k3s 1.24.10 nodes on VirtualBox and Proxmox. However, a bare-metal install via PXE boot on an HP t520 thin client containing a 256GB NGFF SATA SSD reproducibly yields a non-bootable system (falling back to PXE booting). Just to be sure that the t520 box isn't the problem, I PXE booted Debian Bullseye installer and ran a full manual install just to verify I can boot the machine after the installation. After that I switched to the ubuntu-lts-22 (kairos-ubuntu-22-lts-v1.6.1-k3sv1.24.10+k3s1) again and ended up with an unbootable installation. Not sure how to proceed here, except maybe to try opensuse or another image again to compare.

On the VM side, I ran into some trouble that might have to do with APIC. I had to set up VMs on both VirtualBox and Proxmox with a single core and I/O APIC disabled to install. After installation, I was able to up the core count >1.

In another experiment I tried a manual install like this:

    $ kairos-agent manual-install http://centos-vm/pxe/cloud-init/kairos-test.yaml
    open http://centos-vm/pxe/cloud-init/kairos-test.yaml: no such file or directory
    $ curl -v http://centos-vm/pxe/cloud-init/kairos-test.yaml
    # works! Cannot resolve host with curl, but with IP address it finds it but kairos-agent still brings the same error.

I didn't go further there. In cases like this, it would have been useful to have ping and nslookup available from the core image. I went back to PXE booting after that.

Another slightly confusing thing is finding different syntaxes for setting up users. There's the cloud-init native root-level users tag. And the differently structured users tag under /stages/network also took some trial & error to get working. I still need to clean up there, but I can SSH into the machine with a key.

Next steps for me now: trying AuroraBoot on ESXi in the client's environment where there is no PXE boot infrastructure, yet. I will probably also need to try building my own images to get a non-US keyboard mapping, NTP, some additional packages and stuff. I would have loved to have my...

keyboard:
  layout: ch

...in the cloud-init doing something. But I guess I need to solve that differently. Also the P2P feature looks really interesting.

So I guess I'm still in the learning curve and I'm missing some bits and pieces to get everything solved quickly. But Kairos looks promising and I like most of what I'm seeing so far. I guess having so many permutations of potential features and environments makes it very hard to get things flying. The documentation is impressive but with so many little details, one has to be on top of their game to read it carefully. I hope my writing this down was half way useful or at least interesting.

[Itxaka]

@jmaerki This is incredible feedback, thanks for the write up!!

[mauromorales]

Regarding the fallback to PXE you might need a workaround telling 'config_url: ""' so it doesn't try to look into pxe again ... it's a workaround and a fix is coming

[jmaerki]

Regarding the fallback to PXE you might need a workaround telling 'config_url: ""' so it doesn't try to look into pxe again ... it's a workaround and a fix is coming

Thanks, Mauro, but that booted normally into the installer displaying a QR code, as I would have expected. It doesn't seem to recognize that there already is an install when PXE booting.

❯ cat mac-ec8eb5d54aba.ipxe
#!ipxe

echo Setting up for HP t520...
set config ""
chain ../kairos/kairos-ubuntu-22-lts-v1.6.1-k3sv1.24.10.ipxe

...and...

❯ cat ../kairos/kairos-ubuntu-22-lts-v1.6.1-k3sv1.24.10.ipxe
#!ipxe

set version ubuntu-22-lts-v1.6.1-k3sv1.24.10
set url http://centos-vm/pxe/kairos

set kernel kairos-${version}-kernel
set initrd kairos-${version}-initrd
set rootfs kairos-${version}.squashfs

set dns 192.168.0.2
# Configure interface
ifconf

isset ${config}

# set cmdline extra.values=1
kernel ${url}/${kernel} initrd=${initrd} rd.neednet=1 ip=dhcp rd.cos.disable rd.debug root=live:${url}/${rootfs} netboot nodepair.enable config_url=${config} console=tty1 console=ttyS0 ${cmdline}
initrd ${url}/${initrd}
boot

I also tried removing the root parameter but that let's the boot process hang after connecting the NIC.

[mauromorales]

hmm sorry to hear, maybe it only worked for us because we were using AuroraBoot 🧐 ? ... @mudler do you have an idea what might be the cause of this?

[jmaerki]

Ok, so I disabled PXE on my dnsmasq and booted the t520 with AuroraBoot. Took me a moment. My CentOS 7 still has docker 1.31.1 and that caused "/usr/bin/docker-current: missing signature key.". So I wrote a quick k8s Pod:

apiVersion: v1
kind: Pod
metadata:
  namespace: default
  name: auroraboot
spec:
  hostNetwork: true
  containers:
  - name: auroraboot-container
    image: quay.io/kairos/auroraboot
    args:
    - "--set"
    - "container_image=quay.io/kairos/kairos-opensuse-leap:v1.6.1-k3sv1.24.10-k3s1"
    - "--cloud-config"
    - "http://centos-vm/pxe/cloud-init/kairos-test.yaml"
    imagePullPolicy: IfNotPresent
  restartPolicy: Never

That finally worked. The t520 netbooted, installed, shut down and I turned it back on and it was unbootable just like in the previous attempt. Too bad I don't have another free bare metal box that is not an HP Thin Client to try right now.

[jmaerki]

As some long-hanging fruit, I tried Kairos on a HP t620 (slightly bigger but almost the same machine as the t510), but I'm seeing the same. I need to figure out how I can test on a non-HP box so see if anything's different.

[mauromorales]

I agree that the users on different levels of the configuration is confusing. Being able to have them at the different stages is clear, but not needed if you're just getting started or don't need that level of detail. On the other hand, defining at the root of the config is simpler, but as soon as I know about stages makes me wonder at which stage those different root configurations are applied. Maybe we can do a better job at adding documentation about all this, and have a better way to walk the reader through docs, depending on their level of experience.

[jbalonso]

I'd like to point out I see you found that your experience with kairos-agent manual-install is a known-issue: #764 (fix coming in #963). I'm linking it here for the benefit of future readers.

[jmaerki]

Thanks for all your feedback so far. I've taken a few next steps. One thing I've abandoned for now is building a Debian Bullseye core image. I had to cut my time when I got to the step where I wanted to build a provider image with k3s based on the new core image. I guess I'd have to dive a lot deeper into that. I wondered whether there were plans to actually replace the provider images by a normal bundle. Some evidence of that can be found in the bundle repo but it looks like this is not actively pursued.
Anyway, since I had to pull myself out of too many rabbit holes, I went with a vanilla Ubuntu 22 LTS + k3s image and AuroraBoot'ed a single k3s node on the customer's ESXi environment and that came up beautifully with no hassle. And this morning I wrote a Dockerfile to install some additional packages, built it with TeamCity, pushed it to my private registry and tried an upgrade of the node. That let me run into #932 where that work-around helped me a lot.
Next on the list is bootstrapping Argo CD on that test cluster and get my usual setup running on that. Kairos looks real sexy right now since it ticks most of my boxes to be able to build a GitOpsy setup with minimal manual steps to bring up a cluster or new nodes or to rebuild a failed node.