feat(al2023): disable default nodeadm phases

[bobh66]

I am converting our EKS custom nodegroup configuration from AL2 to AL2023 and I can't find a way to modify the NodeConfig dynamically at runtime in userdata. I'm testing with the AL2023 1.29 0116 EKS AMI.

In AL2 we calculate the values for various kubelet flags in a userdata shell script and pass them to bootstrap.sh on the command line. This includes things like CPU reservation settings which are really hard to know ahead of time. We also need to reboot the node to activate the CPU reservations and other features that are not enabled initially and require a reboot, so we do that after we call bootstrap.sh.

In AL2023 I have tried generating a NodeConfig dynamically in the userdata script and then calling nodeadm init --skip run -c file:///var/tmp/nodeconfig.txt which seems to work, sort of, sometimes. I am seeing issues with the nodes not coming back up after the reboot, which appears to be caused by the issues that ensureEKSNetworkConfiguration was designed to fix, so I changed the code to not skip the run.

That fixed the networking problem but now the kubelet is ignoring the custom configuration in the 00-nodeadm.conf

I have a feeling that either I'm missing something simple and that there is an easier way to do this, or that this use case is not supported and I have to come up with another solution.

Is there a way to generate NodeConfig configuration at boot time and inject that into the kubelet?

Thanks in advance

[cartermckinnon]

If you can stay on the static user data path, that's a safer road; but we'll see what kind of escape hatch we can add for use cases like this. I think the problem you're having boils down to not being able to disable the default config phase, which happens prior to cloud-init.

[bobh66]

Unfortunately static configuration would add a lot more overhead as we would need to maintain or generate different NodeConfigs based on the instance type and application requirements. Being able to do it at boot time is a critical step in keeping things simple.

I was able to work around it by creating a reboot service that runs after nodeadm-run service and disables itself. So we create a basic NodeConfig in metadata that gets consumed by the nodeadm-config service, then we dynamically create a complete NodeConfig inside userdata and run nodeadm init -c file:// to process it, and then let nodeadm-run do it's thing before the reboot.

I'm curious what sort of nodeadm-related configuration is "expected" to happen in cloud-init since the documentation mentions it but doesn't provide any examples of what to configure or how to configure it inside cloud-init.

Thanks

[ndbaker1]

could you just run systemctl disable nodeadm-config before rebooting in your user data (and continue invoking nodeadm init) if the config service is clobbering your customization?

FWIW, our "expected" usage of cloud-init has always been an escape hatch for modifying files generated by nodeadm-config using your preferred tooling (Like adding flags/configs to the kubelet config.json with jq or etc). This was designed to have no upper bound on flexibility, but hadn't imagined folks generating entire NodeConfigs.

If that solves your use case we can help extend the docs around the nodeadm lifecycle and how to best opt-out

[cartermckinnon]

Being able to do it at boot time is a critical step in keeping things simple.

Completely understand! I'm mostly concerned with avoiding footguns in the default path, not opposed to making this more flexible.

what sort of nodeadm-related configuration is "expected" to happen in cloud-init

The happy path is you never call nodeadm directly. The NodeConfig is a declarative API to configure the instance, which is "implemented" by nodeadm. So, you include your NodeConfig in your user data, and nodeadm retrieves + applies it behind the scenes.

On our AL2023 AMIs, your cloud-init scripts execute between the two "phases" of nodeadm init (config and run). This allows you to install additional software or configuration after all of the inert config has been rendered, but prior to kubelet, containerd, etc. being started.

[cartermckinnon]

cloud-init has always been an escape hatch for modifying files generated by nodeadm-config using your preferred tooling

We do want to avoid this whenever possible, because our experience has been that shell scripts in user data are error-prone, particularly when the EKS tooling touches the same file/service. If we can add something to NodeConfig that removes the need to touch the same files that nodeadm does, please let us know!

[bobh66]

@ndbaker1

could you just run systemctl disable nodeadm-config before rebooting in your user data (and continue invoking nodeadm init) if the config service is clobbering your customization?

We didn't really have any issues with the config getting clobbered once we figured out that we could provide a minimal config for nodeadm-config to process and then rerun nodeadm init with the updated configuration.

In our case the reboot in cloud-init was causing the nodeadm-run service to never run and patch up the ethernet configuration so we ended up getting DHCP enabled on all interfaces which caused the node to become unreachable.

FWIW, our "expected" usage of cloud-init has always been an escape hatch for modifying files generated by nodeadm-config using your preferred tooling (Like adding flags/configs to the kubelet config.json with jq or etc). This was designed to have no upper bound on flexibility, but hadn't imagined folks generating entire NodeConfigs.

That makes sense - I was just trying to continue using nodeadm for configuration so I could be sure that it would do what it wanted to do with the NodeConfig data and I wouldn't have to figure out and maintain the specific files to update with jq (read: I'm lazy).

[bobh66]

@cartermckinnon

This allows you to install additional software or configuration after all of the inert config has been rendered, but prior to kubelet, containerd, etc. being started.

I'm guessing that most people don't configure the kubelet itself at this stage. If we didn't have to worry about CPU-specific configuration for a wide variety of instance types it would be a lot easier.

Thanks for the responses @cartermckinnon and @ndbaker1 - I think we're doing things "right" or at least as right as we can for now.