Multithreading and other speed improvements

[ancly2008]

Testing has found that a 4-core CPU device (ARM, X86 platform) has a relatively low bandwidth for P2P testing using iperf3. When observing with htop, it seems that maybe only 2 CPUs are used. How to set n3n to fully utilize 4 CPUs? thx

[hamishcoleman]

What speeds are you achieving?. What speeds do you expect? What are you comparing against? What arch/speed/cpu are you using for the testing ? The main n3n processing all happens on a single thread. There are a number of compile time speedup options in the code, but any binary compiled that way is likely to not be using generic assembly instructions and thus may not work on all cpus. There are vague plans for detecting cpu at runtime and activating the right speedups. There are also moves planned for the core to add some threading support. Neither of these are simple to just enable, and the time available for development is limited.

…

On 22 April 2025 8:37:15 pm GMT+12:00, ancly2008 ***@***.***> wrote: ancly2008 created an issue (n42n/n3n#93) Testing has found that a 4-core CPU device (ARM, X86 platform) has a relatively low bandwidth for P2P testing using iperf3. When observing with htop, it seems that maybe only 2 CPUs are used. How to set n3n to fully utilize 4 CPUs? thx

-- Sent from my Android device with K-9 Mail. Please excuse my brevity.

[ancly2008]

Thank you for your reply, and I am glad to hear about the plan to support multithreading.
Sorry for not specifying the testing situation before!
My tests were conducted in the OpenWRT environment, and the test results are as follows
1, p2w_r619ac wifi router , ipq40xx cpu (ARMv7 Processor rev 5 (v7l) x 4 (716MHz,) , Iperf3 testing upload and download, all at around 20-30Mbps , if use wireguard , Iperf3 testing can reach 200Mbps, while other software similar to n3n that performs well can reach 90-120Mbps ,Can n3n be increased to 80Mbps

2, RAX3000M wifi router , MT7981 (ARM Cortex-A53 x2 1.3GHz) . Iperf3 testing upload and download, all at around 60Mbps , if use wireguard , Iperf3 testing can reach 350Mbps, while other software similar to n3n that performs well can reach 200Mbps , Can n3n be increased to 200Mbps

3 , x86 router , Intel CPU (4core , 2000.148 MHz) , Iperf3 testing upload and download, all at around 200Mbps , if use wireguard , Iperf3 testing can reach 600Mbps, while other software similar to n3n that performs well can reach 450Mbps , Can n3n be increased to 400Mbps

More thanks!

[hamishcoleman]

Thanks for those details, that should help with coming up with a baseline to consider future improvements.

Have you investigated the speed improvements from using the compile-time config options to cater for your exact system? (See https://github.com/n42n/n3n/blob/main/doc/BuildConfig.md#optimisation-options)

[ancly2008]

Thank you for your reply ,
I have added CFLAGS="-O3 -march=native" CFLAGS="-DSPECK_ARM_NEON" in ARM platform , and added CFLAGS="-O3" in X86 platform , I have tested the new n3n-edge, but the testing did not improve, and it is unclear where the problem lies
btw, test with ciper =Speck or ChaCha20 .

[ancly2008]

Can I increase the packet buff to improve transmission speed ?
I tried to modify
#define N2N_SN_PKTBUF_SIZE 2048 to 8192 or other
#define N2N_PKT_BUF_SIZE 2048 to 8192 or other
but it would cause compilation failure .

[hamishcoleman]

Those defines generally only affect the size of the buffer memory allocated internally. I do not know why changing them would have caused a compile failure, but that is probably not worth investigating now.

you can change the tuntap.mtu value (but not to a value larger than the PKT BUF SIZE) but any protocol UDP packets generated by N3N that are larger than the underlying transport MTU (this would often be the ethernet or pppoe MTU) will cause transport-layer fragmentation.

Fragmentation can cause small packet drops to have quite a large affect on performance and reliability and increase both latency and memory usage.

Sadly, I do not believe there are any quick wins to improve performance (There is no "#define MAKE_SPEED_SLOW" in the code that we can simply remove) and while it is a goal, the current focus of all my N3N work is on adding IPv6 support.

[ancly2008]

I see , thanks

[dimerr]

Hi. I did a lot of profiling and patching because of extreme CPU load and very low speed. ARMv7-A 900MHz could only do 27Mbit without compression and encryption. Key problem is excessive function/sys calls in packet processing loops:

• Hundreds of thousands time() calls ( turned to be a rather low impact, but still, more than half of those calls are excessive and can be cached)
• traceEvent is the main performance killer, loops are full of traceEvent calls with heavy functions as params, which are executed on per-packet basis.
• any other functions which are called from loop and might be not very effective, with O(n) scans in hash tables, or math etc

This increased speed 3-4 times and now CPU is not 100% all the time even, but still very high load, around 75% at 66Mbit/s. While VTUN on same system does around 20-35%. I'm not sure is this due to only double work n3n does to encap to UDP and then write to TAP and back, but most likely there's more place for optimizations.

[hamishcoleman]

@dimerr Thanks for looking at that. I believe your conclusions match my guesses, but it is good to have some confirmation. Once I get the current IPv6 work done, I hope to add some performance tracking framework so that changes to improve performance can be measured.

• I plan to add a single time call in the mainloop and then use that for all actions from that event, but that has not happened yet
• I've been removing traceEvents with heavy params from the hot path as I see them for a couple of years now, but not focusing on that.
• I believe the current hash table is not the most performant, but I've not got the performance data to test that theory. I also am planning to split up the packet processing so it is clearer in flame-graphs where the time is being spent and can target the cpu hogs.

Are your patches available anywhere? Maybe you could submit them as a PR?

[dimerr]

Since I'm not a real programmer, i doubt it's a good idea to show that mess to anyone :D Mostly its a deep mess, due to numerous research sessions. Need to sync with current codebase, but permanently busy with something else...

[dimerr]

@hamishcoleman https://github.com/dimerr/n3n I did some rework based on latest commit, this gives around 2.5times improvement on ARMv7 SoC's that i use. But only for reference...

[hamishcoleman]

Wow, thanks, I'll try and have a look through that. Thats quite a large diff to consume all in one go - is there any chance that you had smaller patches for the evolution of the change?

Your repository doesnt appear to have run the github actions either - do you know if all the standard tests are passing?

Can you share what errors you got that caused you to need the no strict aliasing? (And which SoCs you tested on?)

[ancly2008]

@dimerr ，Thanks for your optimization. I have tested the following two improvements in openwrt environment：
1, The p2w device iperf test was increased from about 25m to 40-50m
2, After enabling user authentication in the x86 environment, the previous memory problem was solved and the memory is now stable at 0.42m