* NAT performance regression caused by vlan GRO support
@ 2019-04-04 12:57 Rafał Miłecki
2019-04-04 15:17 ` Toshiaki Makita
2019-04-07 11:53 ` Rafał Miłecki
0 siblings, 2 replies; 16+ messages in thread
From: Rafał Miłecki @ 2019-04-04 12:57 UTC (permalink / raw)
To: netdev, David S. Miller, Toshiaki Makita
Cc: Stefano Brivio, Sabrina Dubroca, David Ahern, Felix Fietkau,
Jo-Philipp Wich, Koen Vandeputte
[-- Attachment #1: Type: text/plain, Size: 1951 bytes --]
Hello,
I'd like to report a regression that goes back to the 2015. I know it's damn
late, but the good thing is, the regression is still easy to reproduce, verify &
revert.
Long story short, starting with the commit 66e5133f19e9 ("vlan: Add GRO support
for non hardware accelerated vlan") - which first hit kernel 4.2 - NAT
performance of my router dropped by 30% - 40%.
My hardware is BCM47094 SoC (dual core ARM) with integrated network controller
and external BCM53012 switch.
Relevant setup:
* SoC network controller is wired to the hardware switch
* Switch passes 802.1q frames with VID 1 to four LAN ports
* Switch passes 802.1q frames with VID 2 to WAN port
* Linux does NAT for LAN (eth0.1) to WAN (eth0.2)
* Linux uses pfifo and "echo 2 > rps_cpus"
* Ryzen 5 PRO 2500U (x86_64) laptop connected to a LAN port
* Intel i7-2670QM laptop connected to a WAN port
* Speed of LAN to WAN measured using iperf & TCP over 10 minutes
1) 5.1.0-rc3
[ 6] 0.0-600.0 sec 39.9 GBytes 572 Mbits/sec
2) 5.1.0-rc3 + rtcache patch
[ 6] 0.0-600.0 sec 40.0 GBytes 572 Mbits/sec
3) 5.1.0-rc3 + disable GRO support
[ 6] 0.0-300.4 sec 27.5 GBytes 786 Mbits/sec
4) 5.1.0-rc3 + rtcache patch + disable GRO support
[ 6] 0.0-600.0 sec 65.6 GBytes 939 Mbits/sec
5) 4.1.15 + rtcache patch
934 Mb/s
6) 4.3.4 + rtcache patch
565 Mb/s
As you can see I can achieve a big performance gain by disabling/reverting a
GRO support. Getting up to 65% faster NAT makes a huge difference and ideally
I'd like to get that with upstream Linux code.
Could someone help me and check the reported commit/code, please? Is there
any other info I can provide or anything I can test for you?
--- a/net/8021q/vlan_core.c
+++ b/net/8021q/vlan_core.c
@@ -545,6 +545,8 @@ static int __init vlan_offload_init(void)
{
unsigned int i;
+ return -ENOTSUPP;
+
for (i = 0; i < ARRAY_SIZE(vlan_packet_offloads); i++)
dev_add_offload(&vlan_packet_offloads[i]);
[-- Attachment #2: .config --]
[-- Type: application/x-config, Size: 77368 bytes --]
^ permalink raw reply [flat|nested] 16+ messages in thread* Re: NAT performance regression caused by vlan GRO support 2019-04-04 12:57 NAT performance regression caused by vlan GRO support Rafał Miłecki @ 2019-04-04 15:17 ` Toshiaki Makita 2019-04-04 20:22 ` Rafał Miłecki 2019-04-07 11:53 ` Rafał Miłecki 1 sibling, 1 reply; 16+ messages in thread From: Toshiaki Makita @ 2019-04-04 15:17 UTC (permalink / raw) To: Rafał Miłecki Cc: netdev, David S. Miller, Toshiaki Makita, Stefano Brivio, Sabrina Dubroca, David Ahern, Felix Fietkau, Jo-Philipp Wich, Koen Vandeputte Hi Rafał, On 19/04/04 (木) 21:57:15, Rafał Miłecki wrote: > Hello, > > I'd like to report a regression that goes back to the 2015. I know it's > damn > late, but the good thing is, the regression is still easy to reproduce, > verify & > revert. > > Long story short, starting with the commit 66e5133f19e9 ("vlan: Add GRO > support > for non hardware accelerated vlan") - which first hit kernel 4.2 - NAT > performance of my router dropped by 30% - 40%. > > My hardware is BCM47094 SoC (dual core ARM) with integrated network > controller > and external BCM53012 switch. > > Relevant setup: > * SoC network controller is wired to the hardware switch > * Switch passes 802.1q frames with VID 1 to four LAN ports > * Switch passes 802.1q frames with VID 2 to WAN port > * Linux does NAT for LAN (eth0.1) to WAN (eth0.2) > * Linux uses pfifo and "echo 2 > rps_cpus" > * Ryzen 5 PRO 2500U (x86_64) laptop connected to a LAN port > * Intel i7-2670QM laptop connected to a WAN port > * Speed of LAN to WAN measured using iperf & TCP over 10 minutes > > 1) 5.1.0-rc3 > [ 6] 0.0-600.0 sec 39.9 GBytes 572 Mbits/sec > > 2) 5.1.0-rc3 + rtcache patch > [ 6] 0.0-600.0 sec 40.0 GBytes 572 Mbits/sec > > 3) 5.1.0-rc3 + disable GRO support > [ 6] 0.0-300.4 sec 27.5 GBytes 786 Mbits/sec > > 4) 5.1.0-rc3 + rtcache patch + disable GRO support > [ 6] 0.0-600.0 sec 65.6 GBytes 939 Mbits/sec Did you test it with disabling GRO by ethtool -K? Is this the result with your reverting patch? It's late night in Japan so I think I will try to reproduce it tomorrow. Thanks. > > 5) 4.1.15 + rtcache patch > 934 Mb/s > > 6) 4.3.4 + rtcache patch > 565 Mb/s > > As you can see I can achieve a big performance gain by > disabling/reverting a > GRO support. Getting up to 65% faster NAT makes a huge difference and > ideally > I'd like to get that with upstream Linux code. > > Could someone help me and check the reported commit/code, please? Is there > any other info I can provide or anything I can test for you? > > > --- a/net/8021q/vlan_core.c > +++ b/net/8021q/vlan_core.c > @@ -545,6 +545,8 @@ static int __init vlan_offload_init(void) > { > unsigned int i; > > + return -ENOTSUPP; > + > for (i = 0; i < ARRAY_SIZE(vlan_packet_offloads); i++) > dev_add_offload(&vlan_packet_offloads[i]); ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: NAT performance regression caused by vlan GRO support 2019-04-04 15:17 ` Toshiaki Makita @ 2019-04-04 20:22 ` Rafał Miłecki 2019-04-05 4:26 ` Toshiaki Makita 0 siblings, 1 reply; 16+ messages in thread From: Rafał Miłecki @ 2019-04-04 20:22 UTC (permalink / raw) To: Toshiaki Makita Cc: netdev, David S. Miller, Toshiaki Makita, Stefano Brivio, Sabrina Dubroca, David Ahern, Felix Fietkau, Jo-Philipp Wich, Koen Vandeputte On 04.04.2019 17:17, Toshiaki Makita wrote: > On 19/04/04 (木) 21:57:15, Rafał Miłecki wrote: >> I'd like to report a regression that goes back to the 2015. I know it's damn >> late, but the good thing is, the regression is still easy to reproduce, verify & >> revert. >> >> Long story short, starting with the commit 66e5133f19e9 ("vlan: Add GRO support >> for non hardware accelerated vlan") - which first hit kernel 4.2 - NAT >> performance of my router dropped by 30% - 40%. >> >> My hardware is BCM47094 SoC (dual core ARM) with integrated network controller >> and external BCM53012 switch. >> >> Relevant setup: >> * SoC network controller is wired to the hardware switch >> * Switch passes 802.1q frames with VID 1 to four LAN ports >> * Switch passes 802.1q frames with VID 2 to WAN port >> * Linux does NAT for LAN (eth0.1) to WAN (eth0.2) >> * Linux uses pfifo and "echo 2 > rps_cpus" >> * Ryzen 5 PRO 2500U (x86_64) laptop connected to a LAN port >> * Intel i7-2670QM laptop connected to a WAN port >> * Speed of LAN to WAN measured using iperf & TCP over 10 minutes >> >> 1) 5.1.0-rc3 >> [ 6] 0.0-600.0 sec 39.9 GBytes 572 Mbits/sec >> >> 2) 5.1.0-rc3 + rtcache patch >> [ 6] 0.0-600.0 sec 40.0 GBytes 572 Mbits/sec >> >> 3) 5.1.0-rc3 + disable GRO support >> [ 6] 0.0-300.4 sec 27.5 GBytes 786 Mbits/sec >> >> 4) 5.1.0-rc3 + rtcache patch + disable GRO support >> [ 6] 0.0-600.0 sec 65.6 GBytes 939 Mbits/sec > > Did you test it with disabling GRO by ethtool -K? Oh, I didn't know about such possibility! I just tested: 1) Kernel with GRO support left in place (no local patch disabling it) 2) ethtool -K eth0 gro off and it bumped my NAT performance from 576 Mb/s to 939 Mb/s. I can reliably break/fix NAT performance by just calling ethtool -K eth0 gro on/off. > Is this the result with your reverting patch? Previous results were coming from kernel with patched vlan_offload_init() - see diff at the end of my first e-mail. > It's late night in Japan so I think I will try to reproduce it tomorrow. Thank you! ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: NAT performance regression caused by vlan GRO support 2019-04-04 20:22 ` Rafał Miłecki @ 2019-04-05 4:26 ` Toshiaki Makita 2019-04-05 5:48 ` Rafał Miłecki 0 siblings, 1 reply; 16+ messages in thread From: Toshiaki Makita @ 2019-04-05 4:26 UTC (permalink / raw) To: Rafał Miłecki Cc: Toshiaki Makita, netdev, David S. Miller, Stefano Brivio, Sabrina Dubroca, David Ahern, Felix Fietkau, Jo-Philipp Wich, Koen Vandeputte On 2019/04/05 5:22, Rafał Miłecki wrote: > On 04.04.2019 17:17, Toshiaki Makita wrote: >> On 19/04/04 (木) 21:57:15, Rafał Miłecki wrote: >>> I'd like to report a regression that goes back to the 2015. I know >>> it's damn >>> late, but the good thing is, the regression is still easy to >>> reproduce, verify & >>> revert. >>> >>> Long story short, starting with the commit 66e5133f19e9 ("vlan: Add >>> GRO support >>> for non hardware accelerated vlan") - which first hit kernel 4.2 - NAT >>> performance of my router dropped by 30% - 40%. >>> >>> My hardware is BCM47094 SoC (dual core ARM) with integrated network >>> controller >>> and external BCM53012 switch. >>> >>> Relevant setup: >>> * SoC network controller is wired to the hardware switch >>> * Switch passes 802.1q frames with VID 1 to four LAN ports >>> * Switch passes 802.1q frames with VID 2 to WAN port >>> * Linux does NAT for LAN (eth0.1) to WAN (eth0.2) >>> * Linux uses pfifo and "echo 2 > rps_cpus" >>> * Ryzen 5 PRO 2500U (x86_64) laptop connected to a LAN port >>> * Intel i7-2670QM laptop connected to a WAN port >>> * Speed of LAN to WAN measured using iperf & TCP over 10 minutes >>> >>> 1) 5.1.0-rc3 >>> [ 6] 0.0-600.0 sec 39.9 GBytes 572 Mbits/sec >>> >>> 2) 5.1.0-rc3 + rtcache patch >>> [ 6] 0.0-600.0 sec 40.0 GBytes 572 Mbits/sec >>> >>> 3) 5.1.0-rc3 + disable GRO support >>> [ 6] 0.0-300.4 sec 27.5 GBytes 786 Mbits/sec >>> >>> 4) 5.1.0-rc3 + rtcache patch + disable GRO support >>> [ 6] 0.0-600.0 sec 65.6 GBytes 939 Mbits/sec >> >> Did you test it with disabling GRO by ethtool -K? > > Oh, I didn't know about such possibility! I just tested: > 1) Kernel with GRO support left in place (no local patch disabling it) > 2) ethtool -K eth0 gro off > and it bumped my NAT performance from 576 Mb/s to 939 Mb/s. I can reliably > break/fix NAT performance by just calling ethtool -K eth0 gro on/off. > > >> Is this the result with your reverting patch? > > Previous results were coming from kernel with patched > vlan_offload_init() - see > diff at the end of my first e-mail. > > >> It's late night in Japan so I think I will try to reproduce it tomorrow. My test results: Receiving packets from eth0.10, forwarding them to eth0.20 and applying MASQUERADE on eth0.20, using i40e 25G NIC on kernel 4.20.13. Disabled rxvlan by ethtool -K to exercise vlan_gro_receive(). Measured TCP throughput by netperf. GRO on : 17 Gbps GRO off: 5 Gbps So I failed to reproduce your problem. Would you check the CPU usage by "mpstat -P ALL" or similar (like "sar -u ALL -P ALL") to check if the traffic is able to consume 100% CPU on your machine? If CPU is 100%, perf may help us analyze your problem. If it's available, try running below while testing: # perf record -a -g -- sleep 5 And then run this after testing: # perf report --no-child -- Toshiaki Makita ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: NAT performance regression caused by vlan GRO support 2019-04-05 4:26 ` Toshiaki Makita @ 2019-04-05 5:48 ` Rafał Miłecki 2019-04-05 7:11 ` Rafał Miłecki 0 siblings, 1 reply; 16+ messages in thread From: Rafał Miłecki @ 2019-04-05 5:48 UTC (permalink / raw) To: Toshiaki Makita Cc: Toshiaki Makita, netdev, David S. Miller, Stefano Brivio, Sabrina Dubroca, David Ahern, Felix Fietkau, Jo-Philipp Wich, Koen Vandeputte On 05.04.2019 06:26, Toshiaki Makita wrote: > On 2019/04/05 5:22, Rafał Miłecki wrote: >> On 04.04.2019 17:17, Toshiaki Makita wrote: >>> On 19/04/04 (木) 21:57:15, Rafał Miłecki wrote: >>>> I'd like to report a regression that goes back to the 2015. I know >>>> it's damn >>>> late, but the good thing is, the regression is still easy to >>>> reproduce, verify & >>>> revert. >>>> >>>> Long story short, starting with the commit 66e5133f19e9 ("vlan: Add >>>> GRO support >>>> for non hardware accelerated vlan") - which first hit kernel 4.2 - NAT >>>> performance of my router dropped by 30% - 40%. >>>> >>>> My hardware is BCM47094 SoC (dual core ARM) with integrated network >>>> controller >>>> and external BCM53012 switch. >>>> >>>> Relevant setup: >>>> * SoC network controller is wired to the hardware switch >>>> * Switch passes 802.1q frames with VID 1 to four LAN ports >>>> * Switch passes 802.1q frames with VID 2 to WAN port >>>> * Linux does NAT for LAN (eth0.1) to WAN (eth0.2) >>>> * Linux uses pfifo and "echo 2 > rps_cpus" >>>> * Ryzen 5 PRO 2500U (x86_64) laptop connected to a LAN port >>>> * Intel i7-2670QM laptop connected to a WAN port >>>> * Speed of LAN to WAN measured using iperf & TCP over 10 minutes >>>> >>>> 1) 5.1.0-rc3 >>>> [ 6] 0.0-600.0 sec 39.9 GBytes 572 Mbits/sec >>>> >>>> 2) 5.1.0-rc3 + rtcache patch >>>> [ 6] 0.0-600.0 sec 40.0 GBytes 572 Mbits/sec >>>> >>>> 3) 5.1.0-rc3 + disable GRO support >>>> [ 6] 0.0-300.4 sec 27.5 GBytes 786 Mbits/sec >>>> >>>> 4) 5.1.0-rc3 + rtcache patch + disable GRO support >>>> [ 6] 0.0-600.0 sec 65.6 GBytes 939 Mbits/sec >>> >>> Did you test it with disabling GRO by ethtool -K? >> >> Oh, I didn't know about such possibility! I just tested: >> 1) Kernel with GRO support left in place (no local patch disabling it) >> 2) ethtool -K eth0 gro off >> and it bumped my NAT performance from 576 Mb/s to 939 Mb/s. I can reliably >> break/fix NAT performance by just calling ethtool -K eth0 gro on/off. >> >> >>> Is this the result with your reverting patch? >> >> Previous results were coming from kernel with patched >> vlan_offload_init() - see >> diff at the end of my first e-mail. >> >> >>> It's late night in Japan so I think I will try to reproduce it tomorrow. > > My test results: > > Receiving packets from eth0.10, forwarding them to eth0.20 and applying > MASQUERADE on eth0.20, using i40e 25G NIC on kernel 4.20.13. > Disabled rxvlan by ethtool -K to exercise vlan_gro_receive(). > Measured TCP throughput by netperf. > > GRO on : 17 Gbps > GRO off: 5 Gbps > > So I failed to reproduce your problem. :( Thanks for trying & checking that! > Would you check the CPU usage by "mpstat -P ALL" or similar (like "sar > -u ALL -P ALL") to check if the traffic is able to consume 100% CPU on > your machine? 1) ethtool -K eth0 gro on + iperf running (577 Mb/s) root@OpenWrt:/# mpstat -P ALL 10 3 Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU) 16:33:40 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 16:33:50 all 0.00 0.00 0.00 0.00 0.00 58.79 0.00 0.00 41.21 16:33:50 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 16:33:50 1 0.00 0.00 0.00 0.00 0.00 17.58 0.00 0.00 82.42 16:33:50 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 16:34:00 all 0.00 0.00 0.05 0.00 0.00 59.44 0.00 0.00 40.51 16:34:00 0 0.00 0.00 0.10 0.00 0.00 99.90 0.00 0.00 0.00 16:34:00 1 0.00 0.00 0.00 0.00 0.00 18.98 0.00 0.00 81.02 16:34:00 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 16:34:10 all 0.00 0.00 0.00 0.00 0.00 59.59 0.00 0.00 40.41 16:34:10 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 16:34:10 1 0.00 0.00 0.00 0.00 0.00 19.18 0.00 0.00 80.82 Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle Average: all 0.00 0.00 0.02 0.00 0.00 59.27 0.00 0.00 40.71 Average: 0 0.00 0.00 0.03 0.00 0.00 99.97 0.00 0.00 0.00 Average: 1 0.00 0.00 0.00 0.00 0.00 18.58 0.00 0.00 81.42 2) ethtool -K eth0 gro off + iperf running (941 Mb/s) root@OpenWrt:/# mpstat -P ALL 10 3 Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU) 16:34:39 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 16:34:49 all 0.00 0.00 0.05 0.00 0.00 86.91 0.00 0.00 13.04 16:34:49 0 0.00 0.00 0.10 0.00 0.00 78.22 0.00 0.00 21.68 16:34:49 1 0.00 0.00 0.00 0.00 0.00 95.60 0.00 0.00 4.40 16:34:49 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 16:34:59 all 0.00 0.00 0.10 0.00 0.00 87.06 0.00 0.00 12.84 16:34:59 0 0.00 0.00 0.20 0.00 0.00 79.72 0.00 0.00 20.08 16:34:59 1 0.00 0.00 0.00 0.00 0.00 94.41 0.00 0.00 5.59 16:34:59 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 16:35:09 all 0.00 0.00 0.05 0.00 0.00 85.71 0.00 0.00 14.24 16:35:09 0 0.00 0.00 0.10 0.00 0.00 79.42 0.00 0.00 20.48 16:35:09 1 0.00 0.00 0.00 0.00 0.00 92.01 0.00 0.00 7.99 Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle Average: all 0.00 0.00 0.07 0.00 0.00 86.56 0.00 0.00 13.37 Average: 0 0.00 0.00 0.13 0.00 0.00 79.12 0.00 0.00 20.75 Average: 1 0.00 0.00 0.00 0.00 0.00 94.01 0.00 0.00 5.99 3) System idle (no iperf) root@OpenWrt:/# mpstat -P ALL 10 1 Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU) 16:35:31 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle 16:35:41 all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 16:35:41 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 16:35:41 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle Average: all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 Average: 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 > If CPU is 100%, perf may help us analyze your problem. If it's > available, try running below while testing: > # perf record -a -g -- sleep 5 > > And then run this after testing: > # perf report --no-child I can see my CPU 0 is fully loaded when using "gro on". I'll try perf now. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: NAT performance regression caused by vlan GRO support 2019-04-05 5:48 ` Rafał Miłecki @ 2019-04-05 7:11 ` Rafał Miłecki 2019-04-05 7:14 ` Felix Fietkau 0 siblings, 1 reply; 16+ messages in thread From: Rafał Miłecki @ 2019-04-05 7:11 UTC (permalink / raw) To: Toshiaki Makita Cc: Toshiaki Makita, netdev, David S. Miller, Stefano Brivio, Sabrina Dubroca, David Ahern, Felix Fietkau, Jo-Philipp Wich, Koen Vandeputte On 05.04.2019 07:48, Rafał Miłecki wrote: > On 05.04.2019 06:26, Toshiaki Makita wrote: >> My test results: >> >> Receiving packets from eth0.10, forwarding them to eth0.20 and applying >> MASQUERADE on eth0.20, using i40e 25G NIC on kernel 4.20.13. >> Disabled rxvlan by ethtool -K to exercise vlan_gro_receive(). >> Measured TCP throughput by netperf. >> >> GRO on : 17 Gbps >> GRO off: 5 Gbps >> >> So I failed to reproduce your problem. > > :( Thanks for trying & checking that! > > >> Would you check the CPU usage by "mpstat -P ALL" or similar (like "sar >> -u ALL -P ALL") to check if the traffic is able to consume 100% CPU on >> your machine? > > 1) ethtool -K eth0 gro on + iperf running (577 Mb/s) > root@OpenWrt:/# mpstat -P ALL 10 3 > Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU) > > 16:33:40 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle > 16:33:50 all 0.00 0.00 0.00 0.00 0.00 58.79 0.00 0.00 41.21 > 16:33:50 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 > 16:33:50 1 0.00 0.00 0.00 0.00 0.00 17.58 0.00 0.00 82.42 > > 16:33:50 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle > 16:34:00 all 0.00 0.00 0.05 0.00 0.00 59.44 0.00 0.00 40.51 > 16:34:00 0 0.00 0.00 0.10 0.00 0.00 99.90 0.00 0.00 0.00 > 16:34:00 1 0.00 0.00 0.00 0.00 0.00 18.98 0.00 0.00 81.02 > > 16:34:00 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle > 16:34:10 all 0.00 0.00 0.00 0.00 0.00 59.59 0.00 0.00 40.41 > 16:34:10 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 > 16:34:10 1 0.00 0.00 0.00 0.00 0.00 19.18 0.00 0.00 80.82 > > Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle > Average: all 0.00 0.00 0.02 0.00 0.00 59.27 0.00 0.00 40.71 > Average: 0 0.00 0.00 0.03 0.00 0.00 99.97 0.00 0.00 0.00 > Average: 1 0.00 0.00 0.00 0.00 0.00 18.58 0.00 0.00 81.42 > > > 2) ethtool -K eth0 gro off + iperf running (941 Mb/s) > root@OpenWrt:/# mpstat -P ALL 10 3 > Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU) > > 16:34:39 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle > 16:34:49 all 0.00 0.00 0.05 0.00 0.00 86.91 0.00 0.00 13.04 > 16:34:49 0 0.00 0.00 0.10 0.00 0.00 78.22 0.00 0.00 21.68 > 16:34:49 1 0.00 0.00 0.00 0.00 0.00 95.60 0.00 0.00 4.40 > > 16:34:49 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle > 16:34:59 all 0.00 0.00 0.10 0.00 0.00 87.06 0.00 0.00 12.84 > 16:34:59 0 0.00 0.00 0.20 0.00 0.00 79.72 0.00 0.00 20.08 > 16:34:59 1 0.00 0.00 0.00 0.00 0.00 94.41 0.00 0.00 5.59 > > 16:34:59 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle > 16:35:09 all 0.00 0.00 0.05 0.00 0.00 85.71 0.00 0.00 14.24 > 16:35:09 0 0.00 0.00 0.10 0.00 0.00 79.42 0.00 0.00 20.48 > 16:35:09 1 0.00 0.00 0.00 0.00 0.00 92.01 0.00 0.00 7.99 > > Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle > Average: all 0.00 0.00 0.07 0.00 0.00 86.56 0.00 0.00 13.37 > Average: 0 0.00 0.00 0.13 0.00 0.00 79.12 0.00 0.00 20.75 > Average: 1 0.00 0.00 0.00 0.00 0.00 94.01 0.00 0.00 5.99 > > > 3) System idle (no iperf) > root@OpenWrt:/# mpstat -P ALL 10 1 > Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU) > > 16:35:31 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle > 16:35:41 all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 > 16:35:41 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 > 16:35:41 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 > > Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle > Average: all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 > Average: 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 > Average: 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 > > >> If CPU is 100%, perf may help us analyze your problem. If it's >> available, try running below while testing: >> # perf record -a -g -- sleep 5 >> >> And then run this after testing: >> # perf report --no-child > > I can see my CPU 0 is fully loaded when using "gro on". I'll try perf now. I guess its GRO + csum_partial() to be blamed for this performance drop. Maybe csum_partial() is very fast on your powerful machine and few extra calls don't make a difference? I can imagine it affecting much slower home router with ARM cores. 1) ethtool -K eth0 gro on Samples: 34K of event 'cycles', Event count (approx.): 10041345370 Overhead Command Shared Object Symbol + 25,46% ksoftirqd/0 [kernel.kallsyms] [k] csum_partial + 8,82% ksoftirqd/0 [kernel.kallsyms] [k] v7_dma_inv_range + 6,03% swapper [kernel.kallsyms] [k] arch_cpu_idle + 4,08% ksoftirqd/0 [kernel.kallsyms] [k] v7_dma_clean_range + 3,82% ksoftirqd/0 [kernel.kallsyms] [k] l2c210_inv_range + 3,14% swapper [kernel.kallsyms] [k] rcu_idle_exit + 3,00% ksoftirqd/0 [kernel.kallsyms] [k] l2c210_clean_range + 2,43% ksoftirqd/0 [kernel.kallsyms] [k] bgmac_start_xmit + 1,24% swapper [kernel.kallsyms] [k] csum_partial + 1,20% swapper [kernel.kallsyms] [k] do_idle + 1,19% swapper [kernel.kallsyms] [k] skb_segment + 1,19% ksoftirqd/0 [kernel.kallsyms] [k] arm_dma_unmap_page + 1,00% ksoftirqd/0 [kernel.kallsyms] [k] bgmac_poll + 0,95% ksoftirqd/0 [kernel.kallsyms] [k] __slab_free.constprop.3 + 0,80% ksoftirqd/0 [kernel.kallsyms] [k] skb_release_data + 0,77% swapper [kernel.kallsyms] [k] __dev_queue_xmit + 0,73% ksoftirqd/0 [kernel.kallsyms] [k] build_skb + 0,68% ksoftirqd/0 [kernel.kallsyms] [k] skb_segment + 0,66% ksoftirqd/0 [kernel.kallsyms] [k] mmiocpy + 0,66% ksoftirqd/0 [kernel.kallsyms] [k] skb_checksum_help + 0,65% ksoftirqd/0 [kernel.kallsyms] [k] dev_gro_receive + 0,64% ksoftirqd/0 [kernel.kallsyms] [k] page_address + 0,62% ksoftirqd/0 [kernel.kallsyms] [k] __qdisc_run + 0,62% ksoftirqd/0 [kernel.kallsyms] [k] dma_cache_maint_page + 0,59% swapper [kernel.kallsyms] [k] __kmalloc_track_caller + 0,59% swapper [kernel.kallsyms] [k] mmiocpy + 0,58% ksoftirqd/0 [kernel.kallsyms] [k] sch_direct_xmit + 0,55% ksoftirqd/0 [kernel.kallsyms] [k] mmioset + 0,52% ksoftirqd/0 [kernel.kallsyms] [k] inet_gro_receive 0,49% ksoftirqd/0 [kernel.kallsyms] [k] netdev_alloc_frag 0,47% swapper [kernel.kallsyms] [k] __netif_receive_skb_core 0,45% swapper [kernel.kallsyms] [k] kmem_cache_alloc 0,45% ksoftirqd/0 [kernel.kallsyms] [k] __skb_checksum 0,43% swapper [kernel.kallsyms] [k] v7_dma_clean_range 0,39% ksoftirqd/0 [kernel.kallsyms] [k] kmem_cache_alloc 0,36% ksoftirqd/0 [kernel.kallsyms] [k] qdisc_dequeue_head 0,36% ksoftirqd/0 [kernel.kallsyms] [k] arm_dma_map_page 0,35% swapper [kernel.kallsyms] [k] mmioset 0,34% ksoftirqd/0 [kernel.kallsyms] [k] tcp_gro_receive 0,33% swapper [kernel.kallsyms] [k] __copy_skb_header 0,33% ksoftirqd/0 [kernel.kallsyms] [k] kmem_cache_free 0,32% ksoftirqd/0 [kernel.kallsyms] [k] netif_skb_features 0,30% swapper [kernel.kallsyms] [k] netif_skb_features 0,30% ksoftirqd/0 [kernel.kallsyms] [k] __skb_flow_dissect 2) ethtool -K eth0 gro off Samples: 39K of event 'cycles', Event count (approx.): 13065826851 Overhead Command Shared Object Symbol + 11,09% swapper [kernel.kallsyms] [k] v7_dma_inv_range + 5,86% ksoftirqd/1 [kernel.kallsyms] [k] v7_dma_clean_range + 5,77% swapper [kernel.kallsyms] [k] l2c210_inv_range + 5,38% swapper [kernel.kallsyms] [k] __irqentry_text_end + 4,44% swapper [kernel.kallsyms] [k] bcma_host_soc_read32 + 3,28% ksoftirqd/1 [kernel.kallsyms] [k] __netif_receive_skb_core + 3,25% ksoftirqd/1 [kernel.kallsyms] [k] l2c210_clean_range + 2,70% swapper [kernel.kallsyms] [k] arch_cpu_idle + 2,25% swapper [kernel.kallsyms] [k] bgmac_poll + 2,14% ksoftirqd/1 [kernel.kallsyms] [k] bgmac_start_xmit + 1,79% ksoftirqd/1 [kernel.kallsyms] [k] __dev_queue_xmit + 1,36% ksoftirqd/1 [kernel.kallsyms] [k] skb_vlan_untag + 1,11% swapper [kernel.kallsyms] [k] __skb_flow_dissect + 1,07% ksoftirqd/1 [kernel.kallsyms] [k] netif_skb_features + 0,98% ksoftirqd/1 [kernel.kallsyms] [k] ip_rcv_core.constprop.3 + 0,92% ksoftirqd/1 [kernel.kallsyms] [k] sch_direct_xmit + 0,90% ksoftirqd/1 [kernel.kallsyms] [k] __local_bh_enable_ip + 0,86% ksoftirqd/1 [kernel.kallsyms] [k] nf_hook_slow + 0,82% swapper [kernel.kallsyms] [k] net_rx_action + 0,80% ksoftirqd/1 [kernel.kallsyms] [k] validate_xmit_skb.constprop.30 + 0,75% swapper [kernel.kallsyms] [k] build_skb + 0,72% ksoftirqd/1 [kernel.kallsyms] [k] ip_forward + 0,71% ksoftirqd/1 [kernel.kallsyms] [k] br_handle_frame_finish + 0,71% ksoftirqd/1 [kernel.kallsyms] [k] skb_pull_rcsum + 0,65% swapper [kernel.kallsyms] [k] arm_dma_unmap_page + 0,59% ksoftirqd/1 [kernel.kallsyms] [k] ip_finish_output2 + 0,59% swapper [kernel.kallsyms] [k] __skb_get_hash + 0,58% swapper [kernel.kallsyms] [k] dma_cache_maint_page + 0,55% ksoftirqd/1 [kernel.kallsyms] [k] fdb_find_rcu + 0,54% swapper [kernel.kallsyms] [k] bcma_host_soc_write32 + 0,53% ksoftirqd/1 [kernel.kallsyms] [k] vlan_do_receive + 0,52% ksoftirqd/1 [kernel.kallsyms] [k] memmove + 0,52% swapper [kernel.kallsyms] [k] rcu_idle_exit + 0,51% ksoftirqd/1 [kernel.kallsyms] [k] ip_rcv + 0,51% ksoftirqd/1 [kernel.kallsyms] [k] dev_hard_start_xmit 0,49% ksoftirqd/1 [kernel.kallsyms] [k] ip_output 0,46% ksoftirqd/1 [kernel.kallsyms] [k] vlan_dev_hard_start_xmit 0,45% swapper [kernel.kallsyms] [k] enqueue_to_backlog 0,42% swapper [kernel.kallsyms] [k] netdev_alloc_frag 0,42% swapper [kernel.kallsyms] [k] skb_release_data 0,41% ksoftirqd/1 [kernel.kallsyms] [k] ip_forward_finish 0,40% ksoftirqd/1 [kernel.kallsyms] [k] br_handle_frame 0,37% ksoftirqd/1 [kernel.kallsyms] [k] mmiocpy 0,37% ksoftirqd/1 [kernel.kallsyms] [k] page_address 0,36% ksoftirqd/0 [kernel.kallsyms] [k] v7_dma_inv_range 0,36% ksoftirqd/1 [kernel.kallsyms] [k] memcmp 0,36% ksoftirqd/1 [kernel.kallsyms] [k] netif_receive_skb_internal 0,34% swapper [kernel.kallsyms] [k] page_address 0,34% swapper [kernel.kallsyms] [k] mmioset 0,33% ksoftirqd/1 [kernel.kallsyms] [k] br_pass_frame_up 0,33% ksoftirqd/1 [kernel.kallsyms] [k] neigh_connected_output 0,33% swapper [kernel.kallsyms] [k] kmem_cache_alloc 0,31% ksoftirqd/1 [kernel.kallsyms] [k] mmioset 0,30% ksoftirqd/1 [kernel.kallsyms] [k] ip_finish_output 0,30% ksoftirqd/1 [kernel.kallsyms] [k] bcma_bgmac_write ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: NAT performance regression caused by vlan GRO support 2019-04-05 7:11 ` Rafał Miłecki @ 2019-04-05 7:14 ` Felix Fietkau 2019-04-05 7:58 ` Toshiaki Makita 0 siblings, 1 reply; 16+ messages in thread From: Felix Fietkau @ 2019-04-05 7:14 UTC (permalink / raw) To: Rafał Miłecki, Toshiaki Makita Cc: Toshiaki Makita, netdev, David S. Miller, Stefano Brivio, Sabrina Dubroca, David Ahern, Jo-Philipp Wich, Koen Vandeputte On 2019-04-05 09:11, Rafał Miłecki wrote: > On 05.04.2019 07:48, Rafał Miłecki wrote: >> On 05.04.2019 06:26, Toshiaki Makita wrote: >>> My test results: >>> >>> Receiving packets from eth0.10, forwarding them to eth0.20 and applying >>> MASQUERADE on eth0.20, using i40e 25G NIC on kernel 4.20.13. >>> Disabled rxvlan by ethtool -K to exercise vlan_gro_receive(). >>> Measured TCP throughput by netperf. >>> >>> GRO on : 17 Gbps >>> GRO off: 5 Gbps >>> >>> So I failed to reproduce your problem. >> >> :( Thanks for trying & checking that! >> >> >>> Would you check the CPU usage by "mpstat -P ALL" or similar (like "sar >>> -u ALL -P ALL") to check if the traffic is able to consume 100% CPU on >>> your machine? >> >> 1) ethtool -K eth0 gro on + iperf running (577 Mb/s) >> root@OpenWrt:/# mpstat -P ALL 10 3 >> Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU) >> >> 16:33:40 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >> 16:33:50 all 0.00 0.00 0.00 0.00 0.00 58.79 0.00 0.00 41.21 >> 16:33:50 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 >> 16:33:50 1 0.00 0.00 0.00 0.00 0.00 17.58 0.00 0.00 82.42 >> >> 16:33:50 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >> 16:34:00 all 0.00 0.00 0.05 0.00 0.00 59.44 0.00 0.00 40.51 >> 16:34:00 0 0.00 0.00 0.10 0.00 0.00 99.90 0.00 0.00 0.00 >> 16:34:00 1 0.00 0.00 0.00 0.00 0.00 18.98 0.00 0.00 81.02 >> >> 16:34:00 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >> 16:34:10 all 0.00 0.00 0.00 0.00 0.00 59.59 0.00 0.00 40.41 >> 16:34:10 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 >> 16:34:10 1 0.00 0.00 0.00 0.00 0.00 19.18 0.00 0.00 80.82 >> >> Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >> Average: all 0.00 0.00 0.02 0.00 0.00 59.27 0.00 0.00 40.71 >> Average: 0 0.00 0.00 0.03 0.00 0.00 99.97 0.00 0.00 0.00 >> Average: 1 0.00 0.00 0.00 0.00 0.00 18.58 0.00 0.00 81.42 >> >> >> 2) ethtool -K eth0 gro off + iperf running (941 Mb/s) >> root@OpenWrt:/# mpstat -P ALL 10 3 >> Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU) >> >> 16:34:39 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >> 16:34:49 all 0.00 0.00 0.05 0.00 0.00 86.91 0.00 0.00 13.04 >> 16:34:49 0 0.00 0.00 0.10 0.00 0.00 78.22 0.00 0.00 21.68 >> 16:34:49 1 0.00 0.00 0.00 0.00 0.00 95.60 0.00 0.00 4.40 >> >> 16:34:49 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >> 16:34:59 all 0.00 0.00 0.10 0.00 0.00 87.06 0.00 0.00 12.84 >> 16:34:59 0 0.00 0.00 0.20 0.00 0.00 79.72 0.00 0.00 20.08 >> 16:34:59 1 0.00 0.00 0.00 0.00 0.00 94.41 0.00 0.00 5.59 >> >> 16:34:59 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >> 16:35:09 all 0.00 0.00 0.05 0.00 0.00 85.71 0.00 0.00 14.24 >> 16:35:09 0 0.00 0.00 0.10 0.00 0.00 79.42 0.00 0.00 20.48 >> 16:35:09 1 0.00 0.00 0.00 0.00 0.00 92.01 0.00 0.00 7.99 >> >> Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >> Average: all 0.00 0.00 0.07 0.00 0.00 86.56 0.00 0.00 13.37 >> Average: 0 0.00 0.00 0.13 0.00 0.00 79.12 0.00 0.00 20.75 >> Average: 1 0.00 0.00 0.00 0.00 0.00 94.01 0.00 0.00 5.99 >> >> >> 3) System idle (no iperf) >> root@OpenWrt:/# mpstat -P ALL 10 1 >> Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU) >> >> 16:35:31 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >> 16:35:41 all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 >> 16:35:41 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 >> 16:35:41 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 >> >> Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >> Average: all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 >> Average: 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 >> Average: 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 >> >> >>> If CPU is 100%, perf may help us analyze your problem. If it's >>> available, try running below while testing: >>> # perf record -a -g -- sleep 5 >>> >>> And then run this after testing: >>> # perf report --no-child >> >> I can see my CPU 0 is fully loaded when using "gro on". I'll try perf now. > > I guess its GRO + csum_partial() to be blamed for this performance drop. > > Maybe csum_partial() is very fast on your powerful machine and few extra calls > don't make a difference? I can imagine it affecting much slower home router with > ARM cores. Most high performance Ethernet devices implement hardware checksum offload, which completely gets rid of this overhead. Unfortunately, the BCM53xx/47xx Ethernet MAC doesn't have this, which is why you're getting such crappy performance. - Felix ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: NAT performance regression caused by vlan GRO support 2019-04-05 7:14 ` Felix Fietkau @ 2019-04-05 7:58 ` Toshiaki Makita 2019-04-05 8:12 ` Rafał Miłecki 2019-04-05 10:18 ` Toke Høiland-Jørgensen 0 siblings, 2 replies; 16+ messages in thread From: Toshiaki Makita @ 2019-04-05 7:58 UTC (permalink / raw) To: Felix Fietkau, Rafał Miłecki Cc: Toshiaki Makita, netdev, David S. Miller, Stefano Brivio, Sabrina Dubroca, David Ahern, Jo-Philipp Wich, Koen Vandeputte On 2019/04/05 16:14, Felix Fietkau wrote: > On 2019-04-05 09:11, Rafał Miłecki wrote: >> On 05.04.2019 07:48, Rafał Miłecki wrote: >>> On 05.04.2019 06:26, Toshiaki Makita wrote: >>>> My test results: >>>> >>>> Receiving packets from eth0.10, forwarding them to eth0.20 and applying >>>> MASQUERADE on eth0.20, using i40e 25G NIC on kernel 4.20.13. >>>> Disabled rxvlan by ethtool -K to exercise vlan_gro_receive(). >>>> Measured TCP throughput by netperf. >>>> >>>> GRO on : 17 Gbps >>>> GRO off: 5 Gbps >>>> >>>> So I failed to reproduce your problem. >>> >>> :( Thanks for trying & checking that! >>> >>> >>>> Would you check the CPU usage by "mpstat -P ALL" or similar (like "sar >>>> -u ALL -P ALL") to check if the traffic is able to consume 100% CPU on >>>> your machine? >>> >>> 1) ethtool -K eth0 gro on + iperf running (577 Mb/s) >>> root@OpenWrt:/# mpstat -P ALL 10 3 >>> Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU) >>> >>> 16:33:40 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >>> 16:33:50 all 0.00 0.00 0.00 0.00 0.00 58.79 0.00 0.00 41.21 >>> 16:33:50 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 >>> 16:33:50 1 0.00 0.00 0.00 0.00 0.00 17.58 0.00 0.00 82.42 >>> >>> 16:33:50 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >>> 16:34:00 all 0.00 0.00 0.05 0.00 0.00 59.44 0.00 0.00 40.51 >>> 16:34:00 0 0.00 0.00 0.10 0.00 0.00 99.90 0.00 0.00 0.00 >>> 16:34:00 1 0.00 0.00 0.00 0.00 0.00 18.98 0.00 0.00 81.02 >>> >>> 16:34:00 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >>> 16:34:10 all 0.00 0.00 0.00 0.00 0.00 59.59 0.00 0.00 40.41 >>> 16:34:10 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 >>> 16:34:10 1 0.00 0.00 0.00 0.00 0.00 19.18 0.00 0.00 80.82 >>> >>> Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >>> Average: all 0.00 0.00 0.02 0.00 0.00 59.27 0.00 0.00 40.71 >>> Average: 0 0.00 0.00 0.03 0.00 0.00 99.97 0.00 0.00 0.00 >>> Average: 1 0.00 0.00 0.00 0.00 0.00 18.58 0.00 0.00 81.42 >>> >>> >>> 2) ethtool -K eth0 gro off + iperf running (941 Mb/s) >>> root@OpenWrt:/# mpstat -P ALL 10 3 >>> Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU) >>> >>> 16:34:39 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >>> 16:34:49 all 0.00 0.00 0.05 0.00 0.00 86.91 0.00 0.00 13.04 >>> 16:34:49 0 0.00 0.00 0.10 0.00 0.00 78.22 0.00 0.00 21.68 >>> 16:34:49 1 0.00 0.00 0.00 0.00 0.00 95.60 0.00 0.00 4.40 >>> >>> 16:34:49 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >>> 16:34:59 all 0.00 0.00 0.10 0.00 0.00 87.06 0.00 0.00 12.84 >>> 16:34:59 0 0.00 0.00 0.20 0.00 0.00 79.72 0.00 0.00 20.08 >>> 16:34:59 1 0.00 0.00 0.00 0.00 0.00 94.41 0.00 0.00 5.59 >>> >>> 16:34:59 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >>> 16:35:09 all 0.00 0.00 0.05 0.00 0.00 85.71 0.00 0.00 14.24 >>> 16:35:09 0 0.00 0.00 0.10 0.00 0.00 79.42 0.00 0.00 20.48 >>> 16:35:09 1 0.00 0.00 0.00 0.00 0.00 92.01 0.00 0.00 7.99 >>> >>> Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >>> Average: all 0.00 0.00 0.07 0.00 0.00 86.56 0.00 0.00 13.37 >>> Average: 0 0.00 0.00 0.13 0.00 0.00 79.12 0.00 0.00 20.75 >>> Average: 1 0.00 0.00 0.00 0.00 0.00 94.01 0.00 0.00 5.99 >>> >>> >>> 3) System idle (no iperf) >>> root@OpenWrt:/# mpstat -P ALL 10 1 >>> Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU) >>> >>> 16:35:31 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >>> 16:35:41 all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 >>> 16:35:41 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 >>> 16:35:41 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 >>> >>> Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >>> Average: all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 >>> Average: 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 >>> Average: 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 >>> >>> >>>> If CPU is 100%, perf may help us analyze your problem. If it's >>>> available, try running below while testing: >>>> # perf record -a -g -- sleep 5 >>>> >>>> And then run this after testing: >>>> # perf report --no-child >>> >>> I can see my CPU 0 is fully loaded when using "gro on". I'll try perf now. >> >> I guess its GRO + csum_partial() to be blamed for this performance drop. >> >> Maybe csum_partial() is very fast on your powerful machine and few extra calls >> don't make a difference? I can imagine it affecting much slower home router with >> ARM cores. > Most high performance Ethernet devices implement hardware checksum > offload, which completely gets rid of this overhead. > Unfortunately, the BCM53xx/47xx Ethernet MAC doesn't have this, which is > why you're getting such crappy performance. Hmm... now I disabled rx checksum and tried the test again, and indeed I see csum_partial from GRO path. But I also see csum_partial even without GRO from nf_conntrack_in -> tcp_packet -> __skb_checksum_complete. Probably Rafał disabled nf_conntrack_checksum sysctl knob? But anyway even with disabling rx csum offload my machine has better performance with GRO. I'm sure in some cases GRO should be disabled, but I guess it's difficult to determine whether we should disable GRO or not automatically when csum offload is not available. -- Toshiaki Makita ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: NAT performance regression caused by vlan GRO support 2019-04-05 7:58 ` Toshiaki Makita @ 2019-04-05 8:12 ` Rafał Miłecki 2019-04-05 8:24 ` Rafał Miłecki 2019-04-05 10:18 ` Toke Høiland-Jørgensen 1 sibling, 1 reply; 16+ messages in thread From: Rafał Miłecki @ 2019-04-05 8:12 UTC (permalink / raw) To: Toshiaki Makita, Felix Fietkau Cc: Toshiaki Makita, netdev, David S. Miller, Stefano Brivio, Sabrina Dubroca, David Ahern, Jo-Philipp Wich, Koen Vandeputte On 05.04.2019 09:58, Toshiaki Makita wrote: > On 2019/04/05 16:14, Felix Fietkau wrote: >> On 2019-04-05 09:11, Rafał Miłecki wrote: >>> I guess its GRO + csum_partial() to be blamed for this performance drop. >>> >>> Maybe csum_partial() is very fast on your powerful machine and few extra calls >>> don't make a difference? I can imagine it affecting much slower home router with >>> ARM cores. >> Most high performance Ethernet devices implement hardware checksum >> offload, which completely gets rid of this overhead. >> Unfortunately, the BCM53xx/47xx Ethernet MAC doesn't have this, which is >> why you're getting such crappy performance. > > Hmm... now I disabled rx checksum and tried the test again, and indeed I > see csum_partial from GRO path. But I also see csum_partial even without > GRO from nf_conntrack_in -> tcp_packet -> __skb_checksum_complete. > Probably Rafał disabled nf_conntrack_checksum sysctl knob? > > But anyway even with disabling rx csum offload my machine has better > performance with GRO. I'm sure in some cases GRO should be disabled, but > I guess it's difficult to determine whether we should disable GRO or not > automatically when csum offload is not available. Few testing results: 1) ethtool -K eth0 gro off; echo 0 > /proc/sys/net/netfilter/nf_conntrack_checksum [ 6] 0.0-60.0 sec 6.57 GBytes 940 Mbits/sec 2) ethtool -K eth0 gro off; echo 1 > /proc/sys/net/netfilter/nf_conntrack_checksum [ 6] 0.0-60.0 sec 4.65 GBytes 666 Mbits/sec 3) ethtool -K eth0 gro on; echo 0 > /proc/sys/net/netfilter/nf_conntrack_checksum [ 6] 0.0-60.0 sec 4.02 GBytes 575 Mbits/sec 4) ethtool -K eth0 gro on; echo 1 > /proc/sys/net/netfilter/nf_conntrack_checksum [ 6] 0.0-60.0 sec 4.04 GBytes 579 Mbits/sec ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: NAT performance regression caused by vlan GRO support 2019-04-05 8:12 ` Rafał Miłecki @ 2019-04-05 8:24 ` Rafał Miłecki 0 siblings, 0 replies; 16+ messages in thread From: Rafał Miłecki @ 2019-04-05 8:24 UTC (permalink / raw) To: Toshiaki Makita, Felix Fietkau Cc: Toshiaki Makita, netdev, David S. Miller, Stefano Brivio, Sabrina Dubroca, David Ahern, Jo-Philipp Wich, Koen Vandeputte On 05.04.2019 10:12, Rafał Miłecki wrote: > On 05.04.2019 09:58, Toshiaki Makita wrote: >> On 2019/04/05 16:14, Felix Fietkau wrote: >>> On 2019-04-05 09:11, Rafał Miłecki wrote: >>>> I guess its GRO + csum_partial() to be blamed for this performance drop. >>>> >>>> Maybe csum_partial() is very fast on your powerful machine and few extra calls >>>> don't make a difference? I can imagine it affecting much slower home router with >>>> ARM cores. >>> Most high performance Ethernet devices implement hardware checksum >>> offload, which completely gets rid of this overhead. >>> Unfortunately, the BCM53xx/47xx Ethernet MAC doesn't have this, which is >>> why you're getting such crappy performance. >> >> Hmm... now I disabled rx checksum and tried the test again, and indeed I >> see csum_partial from GRO path. But I also see csum_partial even without >> GRO from nf_conntrack_in -> tcp_packet -> __skb_checksum_complete. >> Probably Rafał disabled nf_conntrack_checksum sysctl knob? >> >> But anyway even with disabling rx csum offload my machine has better >> performance with GRO. I'm sure in some cases GRO should be disabled, but >> I guess it's difficult to determine whether we should disable GRO or not >> automatically when csum offload is not available. > > Few testing results: > > 1) ethtool -K eth0 gro off; echo 0 > /proc/sys/net/netfilter/nf_conntrack_checksum > [ 6] 0.0-60.0 sec 6.57 GBytes 940 Mbits/sec > > 2) ethtool -K eth0 gro off; echo 1 > /proc/sys/net/netfilter/nf_conntrack_checksum > [ 6] 0.0-60.0 sec 4.65 GBytes 666 Mbits/sec For this case (GRO off and nf_conntrack_checksum enabled) I can confirm I see csum_partial() in the perf output. It's taking 13,14% instead of 25,46% (as when using GRO) though. Samples: 38K of event 'cycles', Event count (approx.): 12209908413 Overhead Command Shared Object Symbol + 13,14% ksoftirqd/1 [kernel.kallsyms] [k] csum_partial + 10,16% swapper [kernel.kallsyms] [k] v7_dma_inv_range + 6,36% swapper [kernel.kallsyms] [k] l2c210_inv_range + 4,89% swapper [kernel.kallsyms] [k] __irqentry_text_end + 4,12% ksoftirqd/1 [kernel.kallsyms] [k] v7_dma_clean_range + 3,78% swapper [kernel.kallsyms] [k] bcma_host_soc_read32 + 2,76% swapper [kernel.kallsyms] [k] arch_cpu_idle + 2,45% ksoftirqd/1 [kernel.kallsyms] [k] __netif_receive_skb_core + 2,37% ksoftirqd/1 [kernel.kallsyms] [k] l2c210_clean_range + 1,76% ksoftirqd/1 [kernel.kallsyms] [k] bgmac_start_xmit + 1,66% swapper [kernel.kallsyms] [k] bgmac_poll + 1,55% ksoftirqd/1 [kernel.kallsyms] [k] __dev_queue_xmit + 1,11% ksoftirqd/1 [kernel.kallsyms] [k] skb_vlan_untag > 3) ethtool -K eth0 gro on; echo 0 > /proc/sys/net/netfilter/nf_conntrack_checksum > [ 6] 0.0-60.0 sec 4.02 GBytes 575 Mbits/sec > > 4) ethtool -K eth0 gro on; echo 1 > /proc/sys/net/netfilter/nf_conntrack_checksum > [ 6] 0.0-60.0 sec 4.04 GBytes 579 Mbits/sec ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: NAT performance regression caused by vlan GRO support 2019-04-05 7:58 ` Toshiaki Makita 2019-04-05 8:12 ` Rafał Miłecki @ 2019-04-05 10:18 ` Toke Høiland-Jørgensen 2019-04-05 10:51 ` Florian Westphal 1 sibling, 1 reply; 16+ messages in thread From: Toke Høiland-Jørgensen @ 2019-04-05 10:18 UTC (permalink / raw) To: Toshiaki Makita, Felix Fietkau, Rafał Miłecki Cc: Toshiaki Makita, netdev, David S. Miller, Stefano Brivio, Sabrina Dubroca, David Ahern, Jo-Philipp Wich, Koen Vandeputte Toshiaki Makita <makita.toshiaki@lab.ntt.co.jp> writes: > On 2019/04/05 16:14, Felix Fietkau wrote: >> On 2019-04-05 09:11, Rafał Miłecki wrote: >>> On 05.04.2019 07:48, Rafał Miłecki wrote: >>>> On 05.04.2019 06:26, Toshiaki Makita wrote: >>>>> My test results: >>>>> >>>>> Receiving packets from eth0.10, forwarding them to eth0.20 and applying >>>>> MASQUERADE on eth0.20, using i40e 25G NIC on kernel 4.20.13. >>>>> Disabled rxvlan by ethtool -K to exercise vlan_gro_receive(). >>>>> Measured TCP throughput by netperf. >>>>> >>>>> GRO on : 17 Gbps >>>>> GRO off: 5 Gbps >>>>> >>>>> So I failed to reproduce your problem. >>>> >>>> :( Thanks for trying & checking that! >>>> >>>> >>>>> Would you check the CPU usage by "mpstat -P ALL" or similar (like "sar >>>>> -u ALL -P ALL") to check if the traffic is able to consume 100% CPU on >>>>> your machine? >>>> >>>> 1) ethtool -K eth0 gro on + iperf running (577 Mb/s) >>>> root@OpenWrt:/# mpstat -P ALL 10 3 >>>> Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU) >>>> >>>> 16:33:40 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >>>> 16:33:50 all 0.00 0.00 0.00 0.00 0.00 58.79 0.00 0.00 41.21 >>>> 16:33:50 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 >>>> 16:33:50 1 0.00 0.00 0.00 0.00 0.00 17.58 0.00 0.00 82.42 >>>> >>>> 16:33:50 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >>>> 16:34:00 all 0.00 0.00 0.05 0.00 0.00 59.44 0.00 0.00 40.51 >>>> 16:34:00 0 0.00 0.00 0.10 0.00 0.00 99.90 0.00 0.00 0.00 >>>> 16:34:00 1 0.00 0.00 0.00 0.00 0.00 18.98 0.00 0.00 81.02 >>>> >>>> 16:34:00 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >>>> 16:34:10 all 0.00 0.00 0.00 0.00 0.00 59.59 0.00 0.00 40.41 >>>> 16:34:10 0 0.00 0.00 0.00 0.00 0.00 100.00 0.00 0.00 0.00 >>>> 16:34:10 1 0.00 0.00 0.00 0.00 0.00 19.18 0.00 0.00 80.82 >>>> >>>> Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >>>> Average: all 0.00 0.00 0.02 0.00 0.00 59.27 0.00 0.00 40.71 >>>> Average: 0 0.00 0.00 0.03 0.00 0.00 99.97 0.00 0.00 0.00 >>>> Average: 1 0.00 0.00 0.00 0.00 0.00 18.58 0.00 0.00 81.42 >>>> >>>> >>>> 2) ethtool -K eth0 gro off + iperf running (941 Mb/s) >>>> root@OpenWrt:/# mpstat -P ALL 10 3 >>>> Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU) >>>> >>>> 16:34:39 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >>>> 16:34:49 all 0.00 0.00 0.05 0.00 0.00 86.91 0.00 0.00 13.04 >>>> 16:34:49 0 0.00 0.00 0.10 0.00 0.00 78.22 0.00 0.00 21.68 >>>> 16:34:49 1 0.00 0.00 0.00 0.00 0.00 95.60 0.00 0.00 4.40 >>>> >>>> 16:34:49 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >>>> 16:34:59 all 0.00 0.00 0.10 0.00 0.00 87.06 0.00 0.00 12.84 >>>> 16:34:59 0 0.00 0.00 0.20 0.00 0.00 79.72 0.00 0.00 20.08 >>>> 16:34:59 1 0.00 0.00 0.00 0.00 0.00 94.41 0.00 0.00 5.59 >>>> >>>> 16:34:59 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >>>> 16:35:09 all 0.00 0.00 0.05 0.00 0.00 85.71 0.00 0.00 14.24 >>>> 16:35:09 0 0.00 0.00 0.10 0.00 0.00 79.42 0.00 0.00 20.48 >>>> 16:35:09 1 0.00 0.00 0.00 0.00 0.00 92.01 0.00 0.00 7.99 >>>> >>>> Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >>>> Average: all 0.00 0.00 0.07 0.00 0.00 86.56 0.00 0.00 13.37 >>>> Average: 0 0.00 0.00 0.13 0.00 0.00 79.12 0.00 0.00 20.75 >>>> Average: 1 0.00 0.00 0.00 0.00 0.00 94.01 0.00 0.00 5.99 >>>> >>>> >>>> 3) System idle (no iperf) >>>> root@OpenWrt:/# mpstat -P ALL 10 1 >>>> Linux 5.1.0-rc3+ (OpenWrt) 03/27/19 _armv7l_ (2 CPU) >>>> >>>> 16:35:31 CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >>>> 16:35:41 all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 >>>> 16:35:41 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 >>>> 16:35:41 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 >>>> >>>> Average: CPU %usr %nice %sys %iowait %irq %soft %steal %guest %idle >>>> Average: all 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 >>>> Average: 0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 >>>> Average: 1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 100.00 >>>> >>>> >>>>> If CPU is 100%, perf may help us analyze your problem. If it's >>>>> available, try running below while testing: >>>>> # perf record -a -g -- sleep 5 >>>>> >>>>> And then run this after testing: >>>>> # perf report --no-child >>>> >>>> I can see my CPU 0 is fully loaded when using "gro on". I'll try perf now. >>> >>> I guess its GRO + csum_partial() to be blamed for this performance drop. >>> >>> Maybe csum_partial() is very fast on your powerful machine and few extra calls >>> don't make a difference? I can imagine it affecting much slower home router with >>> ARM cores. >> Most high performance Ethernet devices implement hardware checksum >> offload, which completely gets rid of this overhead. >> Unfortunately, the BCM53xx/47xx Ethernet MAC doesn't have this, which is >> why you're getting such crappy performance. > > Hmm... now I disabled rx checksum and tried the test again, and indeed I > see csum_partial from GRO path. But I also see csum_partial even without > GRO from nf_conntrack_in -> tcp_packet -> __skb_checksum_complete. > Probably Rafał disabled nf_conntrack_checksum sysctl knob? > > But anyway even with disabling rx csum offload my machine has better > performance with GRO. But you're also running at way higher speeds, where the benefit of GRO is higher. > I'm sure in some cases GRO should be disabled, but I guess it's > difficult to determine whether we should disable GRO or not > automatically when csum offload is not available. As a first approximation, maybe just: if (!has_hardware_cksum_offload(netdev) && link_rate(netdev) <= 1Gbps) disable_gro(); We used 1Gbps as the threshold for when to split GRO packets by default in sck_cake as well... -Toke ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: NAT performance regression caused by vlan GRO support 2019-04-05 10:18 ` Toke Høiland-Jørgensen @ 2019-04-05 10:51 ` Florian Westphal 2019-04-05 11:00 ` Eric Dumazet 0 siblings, 1 reply; 16+ messages in thread From: Florian Westphal @ 2019-04-05 10:51 UTC (permalink / raw) To: Toke Høiland-Jørgensen Cc: Toshiaki Makita, Felix Fietkau, Rafał Miłecki, Toshiaki Makita, netdev, David S. Miller, Stefano Brivio, Sabrina Dubroca, David Ahern, Jo-Philipp Wich, Koen Vandeputte Toke Høiland-Jørgensen <toke@redhat.com> wrote: > As a first approximation, maybe just: > > if (!has_hardware_cksum_offload(netdev) && link_rate(netdev) <= 1Gbps) > disable_gro(); I don't think its a good idea. For local delivery case, there is no way to avoid the checksum cost, so might as well have GRO enabled. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: NAT performance regression caused by vlan GRO support 2019-04-05 10:51 ` Florian Westphal @ 2019-04-05 11:00 ` Eric Dumazet 0 siblings, 0 replies; 16+ messages in thread From: Eric Dumazet @ 2019-04-05 11:00 UTC (permalink / raw) To: Florian Westphal, Toke Høiland-Jørgensen Cc: Toshiaki Makita, Felix Fietkau, Rafał Miłecki, Toshiaki Makita, netdev, David S. Miller, Stefano Brivio, Sabrina Dubroca, David Ahern, Jo-Philipp Wich, Koen Vandeputte On 04/05/2019 03:51 AM, Florian Westphal wrote: > Toke Høiland-Jørgensen <toke@redhat.com> wrote: >> As a first approximation, maybe just: >> >> if (!has_hardware_cksum_offload(netdev) && link_rate(netdev) <= 1Gbps) >> disable_gro(); > > I don't think its a good idea. For local delivery case, there is no > way to avoid the checksum cost, so might as well have GRO enabled. > We might add a sysctl or a way to tell GRO layer : Do not attempt checksumming if forwarding is enabled on the host. Basically GRO if NIC has provided checksum offload. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: NAT performance regression caused by vlan GRO support 2019-04-04 12:57 NAT performance regression caused by vlan GRO support Rafał Miłecki 2019-04-04 15:17 ` Toshiaki Makita @ 2019-04-07 11:53 ` Rafał Miłecki 2019-04-07 11:54 ` Rafał Miłecki 1 sibling, 1 reply; 16+ messages in thread From: Rafał Miłecki @ 2019-04-07 11:53 UTC (permalink / raw) To: netdev, David S. Miller, Toshiaki Makita, Toke Høiland-Jørgensen, Florian Westphal, Eric Dumazet Cc: Stefano Brivio, Sabrina Dubroca, David Ahern, Felix Fietkau, Jo-Philipp Wich, Koen Vandeputte On 04.04.2019 14:57, Rafał Miłecki wrote: > Long story short, starting with the commit 66e5133f19e9 ("vlan: Add GRO support > for non hardware accelerated vlan") - which first hit kernel 4.2 - NAT > performance of my router dropped by 30% - 40%. I'll try to provide some summary for this issue. I'll focus on TCP traffic as that's what I happened to test. Basically all slowdowns are related to the csum_partial(). Calculating checksum has a significant impact on NAT performance on less CPU powerful devices. ********** GRO disabled Without GRO a csum_partial() is used only when validating TCP packets in the nf_conntrack_tcp_packet() (known as tcp_packet() in kernels older than 5.1). Simplified forward trace for that case: nf_conntrack_in nf_conntrack_tcp_packet tcp_error if (state->net->ct.sysctl_checksum) nf_checksum nf_ip_checksum __skb_checksum_complete That validation can be disabled using nf_conntrack_checksum sysfs and it bumps NAT speed for me from 666 Mb/s to 940 Mb/s (+41%). ********** GRO enabled First of all GRO also includes TCP validation that requires calculating a checksum. Simplified forward trace for that case: vlan_gro_receive call_gro_receive inet_gro_receive indirect_call_gro_receive tcp4_gro_receive skb_gro_checksum_validate tcp_gro_receive *If* we had a way to disable that validation it *would* result in bumping NAT speed for me from 577 Mb/s to 825 Mb/s (+43%). Secondly using GRO means we need to calculate a checksum before transmitting packets (applies to devices without HW checksum offloading). I think it's related to packets merging in the skb_gro_receive() and then setting CHECKSUM_PARTIAL: vlan_gro_complete inet_gro_complete tcp4_gro_complete tcp_gro_complete skb->ip_summed = CHECKSUM_PARTIAL; That results in bgmac calculating a checksum from the scratch, take a look at the bgmac_dma_tx_add() which does: if (skb->ip_summed == CHECKSUM_PARTIAL) skb_checksum_help(skb); Performing that whole checksum calculation will always result in GRO slowing down NAT for me when using BCM47094 SoC with that not-so-powerful ARM CPUs. ^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: NAT performance regression caused by vlan GRO support 2019-04-07 11:53 ` Rafał Miłecki @ 2019-04-07 11:54 ` Rafał Miłecki 2019-04-08 13:31 ` David Laight 0 siblings, 1 reply; 16+ messages in thread From: Rafał Miłecki @ 2019-04-07 11:54 UTC (permalink / raw) To: netdev, David S. Miller, Toshiaki Makita, Toke Høiland-Jørgensen, Florian Westphal, Eric Dumazet Cc: Stefano Brivio, Sabrina Dubroca, David Ahern, Felix Fietkau, Jo-Philipp Wich, Koen Vandeputte Now I have some questions regarding possible optimizations. Note I'm too familiar with the net subsystem so maybe I got wrong ideas. On 07.04.2019 13:53, Rafał Miłecki wrote: > On 04.04.2019 14:57, Rafał Miłecki wrote: >> Long story short, starting with the commit 66e5133f19e9 ("vlan: Add GRO support >> for non hardware accelerated vlan") - which first hit kernel 4.2 - NAT >> performance of my router dropped by 30% - 40%. > > I'll try to provide some summary for this issue. I'll focus on TCP traffic as > that's what I happened to test. > > Basically all slowdowns are related to the csum_partial(). Calculating checksum > has a significant impact on NAT performance on less CPU powerful devices. > > ********** > > GRO disabled > > Without GRO a csum_partial() is used only when validating TCP packets in the > nf_conntrack_tcp_packet() (known as tcp_packet() in kernels older than 5.1). > > Simplified forward trace for that case: > nf_conntrack_in > nf_conntrack_tcp_packet > tcp_error > if (state->net->ct.sysctl_checksum) > nf_checksum > nf_ip_checksum > __skb_checksum_complete > > That validation can be disabled using nf_conntrack_checksum sysfs and it bumps > NAT speed for me from 666 Mb/s to 940 Mb/s (+41%). > > ********** > > GRO enabled > > First of all GRO also includes TCP validation that requires calculating a > checksum. > > Simplified forward trace for that case: > vlan_gro_receive > call_gro_receive > inet_gro_receive > indirect_call_gro_receive > tcp4_gro_receive > skb_gro_checksum_validate > tcp_gro_receive > > *If* we had a way to disable that validation it *would* result in bumping NAT > speed for me from 577 Mb/s to 825 Mb/s (+43%). Could we have tcp4_gro_receive() behave similarly to the tcp_error() and make it respect the nf_conntrack_checksum sysfs value? Could we simply add something like: if (dev_net(skb->dev)->ct.sysctl_checksum) to it (to additionally protect a skb_gro_checksum_validate() call)? > Secondly using GRO means we need to calculate a checksum before transmitting > packets (applies to devices without HW checksum offloading). I think it's > related to packets merging in the skb_gro_receive() and then setting > CHECKSUM_PARTIAL: > > vlan_gro_complete > inet_gro_complete > tcp4_gro_complete > tcp_gro_complete > skb->ip_summed = CHECKSUM_PARTIAL; > > That results in bgmac calculating a checksum from the scratch, take a look at > the bgmac_dma_tx_add() which does: > > if (skb->ip_summed == CHECKSUM_PARTIAL) > skb_checksum_help(skb); > > Performing that whole checksum calculation will always result in GRO slowing > down NAT for me when using BCM47094 SoC with that not-so-powerful ARM CPUs. Is this possible to avoid CHECKSUM_PARTIAL & skb_checksum_help() which has to calculate a whole checksum? It's definitely possible to *update* checksum after simple packet changes (e.g. amending an IP or port). Would that be possible to use similar method when dealing with packets with GRO enabled? If not, maybe w really need to think about some good & clever condition for disabling GRO by default on hw without checksum offloading. ^ permalink raw reply [flat|nested] 16+ messages in thread
* RE: NAT performance regression caused by vlan GRO support 2019-04-07 11:54 ` Rafał Miłecki @ 2019-04-08 13:31 ` David Laight 0 siblings, 0 replies; 16+ messages in thread From: David Laight @ 2019-04-08 13:31 UTC (permalink / raw) To: 'Rafał Miłecki', netdev, David S. Miller, Toshiaki Makita, Toke Høiland-Jørgensen, Florian Westphal, Eric Dumazet Cc: Stefano Brivio, Sabrina Dubroca, David Ahern, Felix Fietkau, Jo-Philipp Wich, Koen Vandeputte From: Rafal Milecki > Sent: 07 April 2019 12:55 ... > If not, maybe w really need to think about some good & clever condition for > disabling GRO by default on hw without checksum offloading. Maybe GRO could assume the checksums are valid so the checksum would only be verified when the packet is delivered locally. If the packet is forwarded then, provided the same packet boundaries are used, the original checksums (maybe modified by NAT) can be used. No idea how easy this might be :-) David - Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK Registration No: 1397386 (Wales) ^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2019-04-08 13:30 UTC | newest] Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2019-04-04 12:57 NAT performance regression caused by vlan GRO support Rafał Miłecki 2019-04-04 15:17 ` Toshiaki Makita 2019-04-04 20:22 ` Rafał Miłecki 2019-04-05 4:26 ` Toshiaki Makita 2019-04-05 5:48 ` Rafał Miłecki 2019-04-05 7:11 ` Rafał Miłecki 2019-04-05 7:14 ` Felix Fietkau 2019-04-05 7:58 ` Toshiaki Makita 2019-04-05 8:12 ` Rafał Miłecki 2019-04-05 8:24 ` Rafał Miłecki 2019-04-05 10:18 ` Toke Høiland-Jørgensen 2019-04-05 10:51 ` Florian Westphal 2019-04-05 11:00 ` Eric Dumazet 2019-04-07 11:53 ` Rafał Miłecki 2019-04-07 11:54 ` Rafał Miłecki 2019-04-08 13:31 ` David Laight
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.