ECN bleaching - RFC6040 not supported

[mrjstrand]

Just discovered that it currently seems impossible to use ZeroTeir with L4S which is relying on handling of ECN bits according to RFC6040 across tunnels. Without supporting RFC6040 congestion controllers and rate adaptors that is using L4S is not possible to use through ZeroTeir. I.e. TCP Prague, BBRv2, AppleQUIC, SCReAM etc... will not be possible to use any Congestion Experienced (CE) indication from L4S enabled intermediate nodes and needs to fall back to e.g. loss based legacy behavior. This will potentially break some time sensitive applications (like remote gaming and other remote controlled Use Cases) and will limit the usability of ZeroTeir.

• What you expect to be happening.
  => ECN bits should be copied from inner to outer IP header when entering the tunnel and from outer to inner IP header when exiting the tunnel,

• What is actually happening?
  => ECN bits are not copied either from inner to outer or from outer to inner IP header.

• Any steps to reproduce the error.
  => from a ZeroTeir endpoint run: ping -Q 0x01
  => Check ECN code point on the outer IP header and inner IP header with tcpdump both on sender and receiver side
  tcpdump -vv -i <zeroTeir Interface>
  tcpdump -vv -i <outer Interface>
  => Inner ECN codepoint will be 1 both on sender and receiver side
  => outer ECN codepoint will be 0 (or potentially something else) both on sender and receiver side whatever value the outer header gets it is not copied to the inner IP header.

• Any relevant console output or screenshots.
  => ZeroTeir Interface example
  01:10:32.923748 IP (tos 0x1,ECT(1), ttl 64, id 36966, offset 0, flags [DF], proto UDP (17), length 1228
  => Outer Interface example
  01:32:41.533323 IP (tos 0x0, ttl 254, id 9091, offset 0, flags [none], proto UDP (17), length 1294)

• What operating system and ZeroTier version. Please try the latest ZeroTier release.
  => Linux 4.9.253-tegra#1 SMP PREEMPT Wed Sep 1 09:57:04 CST 2021 aarch64 aarch64 aarch64 GNU/Linux
  => $ zerotier-cli -v
  1.12.2
  => Installed with:
  curl -s https://install.zerotier.com | sudo bash

[joseph-henry]

Thank you for this. I can think of a few situations where this was likely the culprit for some issues. We'll work on testing this idea.

[laduke]

What is an easy way to test the effects of this?

[mrjstrand]

As described you can test with ping and check inner and outer header with tcpdump. Like this from a ZT end point: ping -Q 0x01 <another ZT endpoint>

…

On Tue, 12 Mar 2024, 19:45 Travis LaDuke, ***@***.***> wrote: What is an easy way to test the effects of this? — Reply to this email directly, view it on GitHub <#2246 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BDC22RVINBEIZ6RUIQTJS5DYX5EOHAVCNFSM6AAAAABEIEOM4OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJSGMZDENRRHE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[laduke]

Thanks. I see, but what is a test closer a real usecase or application that works better if these bits are set correctly?

[mrjstrand]

To test using L4S you can create a bottleneck with a tool called Jens: https://github.com/telekom/aml-jens then use a L4S enabled congestion control e.g the SCReAM test application: https://github.com/EricssonResearch/scream or e.g. TCP Prague: https://github.com/L4STeam/linux The SCReAM test app provides quite detailed logs over what is observed at the endpoints while the Jens tool provides observation at the bottleneck as well as CE markings. Den tis 12 mars 2024 kl 20:16 skrev Travis LaDuke ***@***.***

…

: Thanks. I see, but what is a test closer a real usecase or application that works better if these bits are set correctly? — Reply to this email directly, view it on GitHub <#2246 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/BDC22RTANEIAOGOOXML5DBLYX5IBRAVCNFSM6AAAAABEIEOM4OVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJSGM3DQMZSGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>