Re: NAT performance regression caused by vlan GRO support

From: "Rafał Miłecki" <zajec5@gmail.com>
To: Toshiaki Makita <toshiaki.makita1@gmail.com>
Cc: netdev@vger.kernel.org, "David S. Miller" <davem@davemloft.net>,
	Toshiaki Makita <makita.toshiaki@lab.ntt.co.jp>,
	Stefano Brivio <sbrivio@redhat.com>,
	Sabrina Dubroca <sd@queasysnail.net>,
	David Ahern <dsahern@gmail.com>, Felix Fietkau <nbd@nbd.name>,
	Jo-Philipp Wich <jo@mein.io>,
	Koen Vandeputte <koen.vandeputte@ncentric.com>
Subject: Re: NAT performance regression caused by vlan GRO support
Date: Thu, 4 Apr 2019 22:22:02 +0200	[thread overview]
Message-ID: <53588a9f-8cc8-0ee5-0947-8ab2b2e56f15@gmail.com> (raw)
In-Reply-To: <75961408-fd62-0f12-bd4b-79008b27576c@gmail.com>

On 04.04.2019 17:17, Toshiaki Makita wrote:
> On 19/04/04 (木) 21:57:15, Rafał Miłecki wrote:
>> I'd like to report a regression that goes back to the 2015. I know it's damn
>> late, but the good thing is, the regression is still easy to reproduce, verify &
>> revert.
>>
>> Long story short, starting with the commit 66e5133f19e9 ("vlan: Add GRO support
>> for non hardware accelerated vlan") - which first hit kernel 4.2 - NAT
>> performance of my router dropped by 30% - 40%.
>>
>> My hardware is BCM47094 SoC (dual core ARM) with integrated network controller
>> and external BCM53012 switch.
>>
>> Relevant setup:
>> * SoC network controller is wired to the hardware switch
>> * Switch passes 802.1q frames with VID 1 to four LAN ports
>> * Switch passes 802.1q frames with VID 2 to WAN port
>> * Linux does NAT for LAN (eth0.1) to WAN (eth0.2)
>> * Linux uses pfifo and "echo 2 > rps_cpus"
>> * Ryzen 5 PRO 2500U (x86_64) laptop connected to a LAN port
>> * Intel i7-2670QM laptop connected to a WAN port
>> * Speed of LAN to WAN measured using iperf & TCP over 10 minutes
>>
>> 1) 5.1.0-rc3
>> [  6]  0.0-600.0 sec  39.9 GBytes   572 Mbits/sec
>>
>> 2) 5.1.0-rc3 + rtcache patch
>> [  6]  0.0-600.0 sec  40.0 GBytes   572 Mbits/sec
>>
>> 3) 5.1.0-rc3 + disable GRO support
>> [  6]  0.0-300.4 sec  27.5 GBytes   786 Mbits/sec
>>
>> 4) 5.1.0-rc3 + rtcache patch + disable GRO support
>> [  6]  0.0-600.0 sec  65.6 GBytes   939 Mbits/sec
> 
> Did you test it with disabling GRO by ethtool -K?

Oh, I didn't know about such possibility! I just tested:
1) Kernel with GRO support left in place (no local patch disabling it)
2) ethtool -K eth0 gro off
and it bumped my NAT performance from 576 Mb/s to 939 Mb/s. I can reliably
break/fix NAT performance by just calling ethtool -K eth0 gro on/off.

> Is this the result with your reverting patch?

Previous results were coming from kernel with patched vlan_offload_init() - see
diff at the end of my first e-mail.

> It's late night in Japan so I think I will try to reproduce it tomorrow.

Thank you!