U45N: Unacceptable TX/RX performance

[Jinbei43]

After I connected 2 U45N cards with QSFP28 100Gb cables, loaded ONIC driver, and test with iPerf3 application. I found that throughput performance is quite lower than it should be around 4-7 Gb/s (I know it won't reach 100 Gb/s line-rate because of in-host bottleneck, but it shouldn't be that slow.). What happened? And this case seems to have similar problem, but at least he got better throughput (._.).

And if I test to a certain point (trying to find that critical point), the latency will continue to increase (up to tens of thousands of milliseconds) until it becomes uncommunicable (even Ping ICMP). It makes me feel like the card's queue is full, and this problem goes away when I shut down and turn on my system. Is it possible to solve this problem sustainably and trace my card operation status more than ethtool or dmesg?

Thank you so much
PS: I also have issues with QDMA subsystem, which I have posted here.

[johnwinans]

What sort of cables are you using?
According to UltraScale+ Devices Integrated 100G Ethernet Subsystem LogiCORE IP Product Guide (PG203), the free CMAC license does not include 100GBASE-CR4 (which is the version with link training for copper.)
I have yet to test my U45N cards. But the "Licensing and Ordering" section of the above manual suggests that the CAUI-4 interface requires fiber cables. (For example, 100GBASE-SR4/multimode fiber for short hops.)
I'd love to hear about any experience you have with different cable types.

[Jinbei43]

Thank you for your suggestion, I read PG203 product guide and now understand what you said. I use 2 cables: CR4 and SR4. So, I set up connection like below figure. However, I can see link status "UP" and same stats results ("show port info" in testpmd and "ethtool -S") for both cable types (also, Ping ICMP and iPerf3).

How about your experience with U45N cards + OpenNIC? It's very rare to find someone to use this same card model!

[johnwinans]

I agree. It is unfortunate that the AU45N is not discussed all that often. But it is nice to see that someone did the basic integration needed for an out of the box open-nic build.

I have a short passive DAC cable laying around that I hope to test with just to see how it does.

I plugged it into my Arista 7060CX and can see that it appears as:

Port       Name           Status       Vlan     Duplex Speed  Type         Flags
...
Et31/1                    notconnect   1        full   100G   100GBASE-CR4
...

After some googling around and reading about cables, maybe the copper cables will be OK at a short distance.
I have to get the qspi flash module integrated before I can comfortably iterate my RTL design. FYI - I expect to implement my appliaction in the FPGA and then use bypass mode DMA to deliver state information about my app to the main memory.
I do not expect to have a 'normal' kernel Ethernet driver loaded.

[johnwinans]

I can't imagine that you'd see any improvement by adding a switch in between your two cards. But it might be an interesting data point worth collecting.