Allow configuration of the maximum number of event loop threads

[mhnap]

Tell us more about this new feature.

Currently, it's not possible to set a custom number of event loop threads:

mountpoint-s3/mountpoint-s3-client/src/s3_crt_client.rs

Line 261 in ac6c177

let mut event_loop_group = EventLoopGroup::new_default(&allocator, None, || {}).unwrap();

mountpoint-s3/mountpoint-s3-crt/src/io/event_loop.rs

Line 114 in ac6c177

let max_threads = max_threads.unwrap_or(0);

By default, the value is 0 which translates to the number of available processors on the machine / 2 (to exclude hyper-threads).
This is not a bad default value. But I run a couple of benchmarks (with a changed version) and they show that increasing this value can boost performance.
Also worth noting is that not all EC2 instance types support hyper-threading, so it could make even more sense to change this value manually.

[arsh]

Can you share more info about your benchmarks? What are you measuring, what is the read / write pattern, how many processors, if in AWS, which EC2 instance?

I tried these changes on reading a 10GB file sequentially and did not see any improvements.

[mhnap]

Can you share more info about your benchmarks?

Yes, sure. Sorry for the late response, I tried to make additional benchmarks.

Here is some general info about my benchmarks:

Data	Value
Objects count	150,000
Total size, GiB	300
--max-fuse-threads	16/32
--max-event-loop-threads	2/4
--max-cache-size	1024
--metadata-ttl	indefinite
EC2 type	m5a.xlarge/m7g.xlarge

What are you measuring, what is the read / write pattern, how many processors, if in AWS, which EC2 instance?

I was measuring the total time spent listing each directory while also reading every file simultaneously (16 or 32 is an internal number of concurrent reads). My main goal was to find the best configuration that would traverse the whole bucket and read all objects most quickly.

I tried these changes on reading a 10GB file sequentially and did not see any improvements.

Worth noting that after additional benchmarks, I also mostly didn't see any performance improvements for increasing the --max-event-loop-threads value from 2 to 4. However, a small boost (10-15%) can sometimes be observed for some configurations. It's hard to tell one exact case when this would definitely help cause in my benchmarks there are also a lot of additional conditions such as whether the whole file or some part of it would be read, other CPU/Network consuming processes are running, etc. Maybe it would be better to make some less complex cases only for listing and reading each file for benchmarks.

Indeed, I'm not sure whether the --max-event-loop-threads will be a generally useful parameter, but I think it still would be great to expose it for further performance investigations.