From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1EC5CC4360F for ; Fri, 5 Apr 2019 05:05:54 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id ED86A21855 for ; Fri, 5 Apr 2019 05:05:53 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727608AbfDEFFw (ORCPT ); Fri, 5 Apr 2019 01:05:52 -0400 Received: from tama500.ecl.ntt.co.jp ([129.60.39.148]:47013 "EHLO tama500.ecl.ntt.co.jp" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1725955AbfDEFFw (ORCPT ); Fri, 5 Apr 2019 01:05:52 -0400 X-Greylist: delayed 2268 seconds by postgrey-1.27 at vger.kernel.org; Fri, 05 Apr 2019 01:05:52 EDT Received: from vc2.ecl.ntt.co.jp (vc2.ecl.ntt.co.jp [129.60.86.154]) by tama500.ecl.ntt.co.jp (8.13.8/8.13.8) with ESMTP id x354RfqE022131; Fri, 5 Apr 2019 13:27:41 +0900 Received: from vc2.ecl.ntt.co.jp (localhost [127.0.0.1]) by vc2.ecl.ntt.co.jp (Postfix) with ESMTP id C4215638ABE; Fri, 5 Apr 2019 13:27:41 +0900 (JST) Received: from jcms-pop21.ecl.ntt.co.jp (jcms-pop21.ecl.ntt.co.jp [129.60.87.134]) by vc2.ecl.ntt.co.jp (Postfix) with ESMTP id AE6116389CB; Fri, 5 Apr 2019 13:27:41 +0900 (JST) Received: from [IPv6:::1] (eb8460w-makita.sic.ecl.ntt.co.jp [129.60.241.47]) by jcms-pop21.ecl.ntt.co.jp (Postfix) with ESMTPSA id A0D52400802; Fri, 5 Apr 2019 13:27:41 +0900 (JST) Subject: Re: NAT performance regression caused by vlan GRO support References: <73223229-6bc0-2647-6952-975961811866@gmail.com> <75961408-fd62-0f12-bd4b-79008b27576c@gmail.com> <53588a9f-8cc8-0ee5-0947-8ab2b2e56f15@gmail.com> From: Toshiaki Makita Message-ID: Date: Fri, 5 Apr 2019 13:26:31 +0900 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <53588a9f-8cc8-0ee5-0947-8ab2b2e56f15@gmail.com> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-CC-Mail-RelayStamp: 1 To: =?UTF-8?B?UmFmYcWCIE1pxYJlY2tp?= Cc: Toshiaki Makita , netdev@vger.kernel.org, "David S. Miller" , Stefano Brivio , Sabrina Dubroca , David Ahern , Felix Fietkau , Jo-Philipp Wich , Koen Vandeputte X-TM-AS-MML: disable Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On 2019/04/05 5:22, Rafał Miłecki wrote: > On 04.04.2019 17:17, Toshiaki Makita wrote: >> On 19/04/04 (木) 21:57:15, Rafał Miłecki wrote: >>> I'd like to report a regression that goes back to the 2015. I know >>> it's damn >>> late, but the good thing is, the regression is still easy to >>> reproduce, verify & >>> revert. >>> >>> Long story short, starting with the commit 66e5133f19e9 ("vlan: Add >>> GRO support >>> for non hardware accelerated vlan") - which first hit kernel 4.2 - NAT >>> performance of my router dropped by 30% - 40%. >>> >>> My hardware is BCM47094 SoC (dual core ARM) with integrated network >>> controller >>> and external BCM53012 switch. >>> >>> Relevant setup: >>> * SoC network controller is wired to the hardware switch >>> * Switch passes 802.1q frames with VID 1 to four LAN ports >>> * Switch passes 802.1q frames with VID 2 to WAN port >>> * Linux does NAT for LAN (eth0.1) to WAN (eth0.2) >>> * Linux uses pfifo and "echo 2 > rps_cpus" >>> * Ryzen 5 PRO 2500U (x86_64) laptop connected to a LAN port >>> * Intel i7-2670QM laptop connected to a WAN port >>> * Speed of LAN to WAN measured using iperf & TCP over 10 minutes >>> >>> 1) 5.1.0-rc3 >>> [  6]  0.0-600.0 sec  39.9 GBytes   572 Mbits/sec >>> >>> 2) 5.1.0-rc3 + rtcache patch >>> [  6]  0.0-600.0 sec  40.0 GBytes   572 Mbits/sec >>> >>> 3) 5.1.0-rc3 + disable GRO support >>> [  6]  0.0-300.4 sec  27.5 GBytes   786 Mbits/sec >>> >>> 4) 5.1.0-rc3 + rtcache patch + disable GRO support >>> [  6]  0.0-600.0 sec  65.6 GBytes   939 Mbits/sec >> >> Did you test it with disabling GRO by ethtool -K? > > Oh, I didn't know about such possibility! I just tested: > 1) Kernel with GRO support left in place (no local patch disabling it) > 2) ethtool -K eth0 gro off > and it bumped my NAT performance from 576 Mb/s to 939 Mb/s. I can reliably > break/fix NAT performance by just calling ethtool -K eth0 gro on/off. > > >> Is this the result with your reverting patch? > > Previous results were coming from kernel with patched > vlan_offload_init() - see > diff at the end of my first e-mail. > > >> It's late night in Japan so I think I will try to reproduce it tomorrow. My test results: Receiving packets from eth0.10, forwarding them to eth0.20 and applying MASQUERADE on eth0.20, using i40e 25G NIC on kernel 4.20.13. Disabled rxvlan by ethtool -K to exercise vlan_gro_receive(). Measured TCP throughput by netperf. GRO on : 17 Gbps GRO off: 5 Gbps So I failed to reproduce your problem. Would you check the CPU usage by "mpstat -P ALL" or similar (like "sar -u ALL -P ALL") to check if the traffic is able to consume 100% CPU on your machine? If CPU is 100%, perf may help us analyze your problem. If it's available, try running below while testing: # perf record -a -g -- sleep 5 And then run this after testing: # perf report --no-child -- Toshiaki Makita