Skip to content

Commit 17fb8e5

Browse files
committed
add some screenshots from testing today
Signed-off-by: clux <[email protected]>
1 parent 2ea93ab commit 17fb8e5

File tree

3 files changed

+24
-4
lines changed

3 files changed

+24
-4
lines changed

docs/blog/posts/2024-06-11-reflector-memory.md

+24-4
Original file line numberDiff line numberDiff line change
@@ -152,11 +152,31 @@ The initial [synthetic benchmarks](https://github.com/kube-rs/kube/pull/1494#iss
152152

153153
Whether the ad-hoc synthetic benchmarks are in any way realistic going forwards remains to be seen. How much you can get likely depends on a range of factors from allocator choice to usage patterns.
154154

155-
__So far__, we have seen controllers with a basically unchanged profile, some with small improvements in the 10-20% range, and one [50% drop in a real-world controller](https://github.com/kube-rs/kube/pull/1494#issuecomment-2126694967) from testing.
155+
__So far__, we have seen controllers with a basically unchanged profile, some with small improvements in the 10-20% range, one [50% drop in a real-world controller](https://github.com/kube-rs/kube/pull/1494#issuecomment-2126694967) from testing, the same controller dropping 80% with page tuning (see below).
156156

157-
When using the standard `ListWatch` [InitialListStrategy], the default [Config::page_size] of `500` will undermine this optimization, because individual pages are still kept in the watcher while they are being sent out one-by-one. Setting the page size to `50` has been necessary for me to get anything close to the benchmarks.
157+
If you are using the standard `ListWatch` [InitialListStrategy], the default [Config::page_size] of `500` will undermine this optimization, because individual pages are still kept in the watcher while they are being sent out one-by-one. Setting the page size to `50` has been necessary for me to get anything close to the benchmarks.
158158

159-
So for now; YMMV. Try setting the `page_size` and please [reach out](https://discord.gg/tokio) with more results!
159+
So for now; YMMV. Try setting the `page_size`, and [chat about](https://discord.gg/tokio) / [share](https://github.com/kube-rs/kube/discussions) your results!
160+
161+
### Examples
162+
Two examples from my own deployment testing today.
163+
164+
#### Optimized Metadata Controller
165+
A metadata controller watching 2000 objects (all in stores), doing 6000 reconciles an hour, using **9MB of RAM**.
166+
167+
![memory drop from a metadata controller having 2000 objects in its stores](mem-lc-drop.png)
168+
169+
Yellow == `0.91.0`, green/red = `0.92.0` with defaults, orange = `0.92.0` with reduced page size (50).
170+
171+
This has seen the biggest change, dropping ~85% of its memory usage, but it also is also doing all the biggest [[optimization]] tricks (`metadata_watcher`, page_size 50, pruning of managed fields), and it has basically no other cached data.
172+
173+
#### KS Controller
174+
A controller for [flux kustomizations](https://fluxcd.io/flux/components/kustomize/kustomizations/) storing and reconciling about 200 `ks` objects without any significant optimization techniques.
175+
![memory drop from a ks controller with 200 larger objects in its stores](mem-dn-drop.png)
176+
177+
LHS == `0.91.0`, middle = `0.92.0` with defaults (many spot instance pods), RHS from 16:00 is `0.92.0` with reduced page size (50).
178+
179+
This controller saw a ~30% improvement overall, but not until reducing the page size. I assume this did not see any default improvement because the 200 objects fit comfortably within a single page and were cached internally in the `watcher` as before.
160180

161181
## Thoughts for the future
162182

@@ -178,7 +198,7 @@ If you are using a custom store **please see the new [watcher::Event]** and make
178198
- `Deleted` -> `Delete`
179199
- `Restarted` -> change to `InitApply` with 2 new arms:
180200
* Create new arms for `Init` marking start (allocate a temporary buffer)
181-
* buffer objects from `InitApply`
201+
* buffer objects from `InitApply` (you get one object at a time, no need to loop)
182202
* Swap store in `InitDone` and deallocate the old buffer
183203

184204
See the above `Store::apply_watcher_event` code for pointers.

docs/blog/posts/mem-dn-drop.png

16.1 KB
Loading

docs/blog/posts/mem-lc-drop.png

13.2 KB
Loading

0 commit comments

Comments
 (0)