[FEATURE] Memory Limits (#494, @konradmalik) #494
Conversation
|
Also, this may be interesting for you: k3s-io/k3s#3005 (good progress lately: https://twitter.com/_AkihiroSuda_/status/1366689973672402945) |
Thanks, I'll follow and investigate this one. It seems it requires running systemd inside the container... we'll see. |
|
LGTM! |
|
Hi @konradmalik , I just tested this on my Linux machine and it seems to fail 🤔 $ k3d cluster create memtest --servers-memory 1g --agents 1 --agents-memory 1.5g
# ...
$ kubectl get nodes -o go-template='{{range $index, $item := .items}}{{ $item.metadata.name }}{{ ":"}}{{ $item.status.capacity.memory }}{{ "\n" }}{{ end }}'
k3d-memtest-agent-0:32475116Ki
k3d-memtest-server-0:32475116Ki
$ docker inspect k3d-memtest-server-0 | jq '.[0].HostConfig.Memory'
1073741824
$ docker inspect k3d-memtest-agent-0 | jq '.[0].HostConfig.Memory'
1610612736Note: edac folder is present on my machine. |
|
ha, no wonder it did not work. Volume binding was reversed 😄 any idea what could cause the order to change? For sure was working before 🤔 anyway, the newest commit fixes this |
|
Hi @konradmalik , finally I found some time to get back to this :)
Nothing changes afaik and the order should always have been Anyway, I can confirm that it works now on my machine (Linux, edac present). $ kubectl get nodes -o go-template='{{range $index, $item := .items}}{{ $item.metadata.name }}{{ ":"}}{{ $item.status.capacity.memory }}{{ "\n" }}{{ end }}'
k3d-memtest-server-0:1073741Ki
k3d-memtest-agent-0:1610612KiUpdate 1: ✔️ Windows w/ Docker for Desktop w/ WSL2 backend |
iwilltry42
changed the title
Author: Konrad Malik <[email protected]> Date: Mon Mar 29 17:52:15 2021 +0200 [FEATURE] Memory Limits (#494, @konradmalik)
What
Docker based memory limit per node container. Separate limits for servers and agents. Also, per single node limiting is possible when creating nodes one after another using
k3d node create.Why
See issue #491 and mentions.
This is an implementation of docker-based memory limiting per node type (server/agent) that tries to make those limits also visible to kubelet via cadvisor, so that
kubectl describe nodegives proper (limited) memory capacity and does not schedule pods when limits are reached.In my tests it works as expected, below is a simple demonstration:
Create a cluster:
Docker stats:
Nodes description (memory fragment):
Here, provision something that will surely reach the limits. I deployed 10 replicas of a service that needs 100m CPU and 300Mi memory. The 8th replica is in a pending state due to lack of memory:
Let's see the resources:
Look good I guess? 😉
Implications
The route taken was vastly different from what was previously tried. After investigating cadvisor source code it seems that it uses
/proc/meminfoto read the available total memory and swap. The main idea behind this implementation is to simply create a minimal, fakememinfofile and mount it as read-only to the node containers.Other stuff worth mentioning in a "tldr" version below:
free,topas well as for example prometheus node exporter etc. but those were already "broken" for k3d as they showed host, not container resourcesserversMemory,agentsMemoryoptions under "runtime" yaml confservers-memory,agents-memorycli args undercluster creatememoryarg undernode createdockerlimitsFurther work
Potentially TODO is a check that someone wants to create containers that have larger memory limit (in total) than the current machine allows. This is not validated currently but maybe should not be checked in the first place? When creating servers and agents without limits, the total memory limit is always larger than the memory of the host.
CPU limits need a little bit more work but I'm pretty sure they can be implemented in a very similar way. I can start doing this after receiving positive feedback on this oneSeems like cores number is read from cpuinfo as a last resort. First, the topology of the cpu is built based on the sysfs. Minimal mocking and mounting cpus under sysfs crashes k3s, so full-fledged solution would be needed. In short, cpu seem like a no-go at the moment, unless something comes to my mind along the way.
Edit 1:
Moved from singular files to a folder for easier further mocking.
Currently working on a "edac" folder mock. The above solution for memory limit works on a non-ecc memory. I cannot test, but I think it won't work on ecc memory, as the true, larger capacity will be read from the "edac" folder. We cannot just mount fake edac always because when it does not exist on the host (for example my macbook) it will fail. Sysfs is read only in docker, so docker cannot create this folder if it does not exist in the system. The solution is to check if it exists but this gets tricky on docker for mac and docker for windows.
Edit 2:
Mocking edac to force using meminfo is now working (checked on linux with existing edac folder, but no ecc memory) and is made through an in-container directory existence check so it is universal and should work across linux, windows and mac.