Re: NAT performance regression caused by vlan GRO support

[Toshiaki Makita <makita.toshiaki@lab.ntt.co.jp>]

On 2019/04/05 5:22, Rafał Miłecki wrote:
> On 04.04.2019 17:17, Toshiaki Makita wrote:
>> On 19/04/04 (木) 21:57:15, Rafał Miłecki wrote:
>>> I'd like to report a regression that goes back to the 2015. I know
>>> it's damn
>>> late, but the good thing is, the regression is still easy to
>>> reproduce, verify &
>>> revert.
>>>
>>> Long story short, starting with the commit 66e5133f19e9 ("vlan: Add
>>> GRO support
>>> for non hardware accelerated vlan") - which first hit kernel 4.2 - NAT
>>> performance of my router dropped by 30% - 40%.
>>>
>>> My hardware is BCM47094 SoC (dual core ARM) with integrated network
>>> controller
>>> and external BCM53012 switch.
>>>
>>> Relevant setup:
>>> * SoC network controller is wired to the hardware switch
>>> * Switch passes 802.1q frames with VID 1 to four LAN ports
>>> * Switch passes 802.1q frames with VID 2 to WAN port
>>> * Linux does NAT for LAN (eth0.1) to WAN (eth0.2)
>>> * Linux uses pfifo and "echo 2 > rps_cpus"
>>> * Ryzen 5 PRO 2500U (x86_64) laptop connected to a LAN port
>>> * Intel i7-2670QM laptop connected to a WAN port
>>> * Speed of LAN to WAN measured using iperf & TCP over 10 minutes
>>>
>>> 1) 5.1.0-rc3
>>> [  6]  0.0-600.0 sec  39.9 GBytes   572 Mbits/sec
>>>
>>> 2) 5.1.0-rc3 + rtcache patch
>>> [  6]  0.0-600.0 sec  40.0 GBytes   572 Mbits/sec
>>>
>>> 3) 5.1.0-rc3 + disable GRO support
>>> [  6]  0.0-300.4 sec  27.5 GBytes   786 Mbits/sec
>>>
>>> 4) 5.1.0-rc3 + rtcache patch + disable GRO support
>>> [  6]  0.0-600.0 sec  65.6 GBytes   939 Mbits/sec
>>
>> Did you test it with disabling GRO by ethtool -K?
> 
> Oh, I didn't know about such possibility! I just tested:
> 1) Kernel with GRO support left in place (no local patch disabling it)
> 2) ethtool -K eth0 gro off
> and it bumped my NAT performance from 576 Mb/s to 939 Mb/s. I can reliably
> break/fix NAT performance by just calling ethtool -K eth0 gro on/off.
> 
> 
>> Is this the result with your reverting patch?
> 
> Previous results were coming from kernel with patched
> vlan_offload_init() - see
> diff at the end of my first e-mail.
> 
> 
>> It's late night in Japan so I think I will try to reproduce it tomorrow.

My test results:

Receiving packets from eth0.10, forwarding them to eth0.20 and applying
MASQUERADE on eth0.20, using i40e 25G NIC on kernel 4.20.13.
Disabled rxvlan by ethtool -K to exercise vlan_gro_receive().
Measured TCP throughput by netperf.

GRO on : 17 Gbps
GRO off:  5 Gbps

So I failed to reproduce your problem.

Would you check the CPU usage by "mpstat -P ALL" or similar (like "sar
-u ALL -P ALL") to check if the traffic is able to consume 100% CPU on
your machine?

If CPU is 100%, perf may help us analyze your problem. If it's
available, try running below while testing:
# perf record -a -g -- sleep 5

And then run this after testing:
# perf report --no-child

-- 
Toshiaki Makita