From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A6603C4360F for ; Fri, 5 Apr 2019 07:16:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 42E3A217D4 for ; Fri, 5 Apr 2019 07:16:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=nbd.name header.i=@nbd.name header.b="sZWnbLRF" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1729792AbfDEHOY (ORCPT ); Fri, 5 Apr 2019 03:14:24 -0400 Received: from nbd.name ([46.4.11.11]:35904 "EHLO nbd.name" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726726AbfDEHOY (ORCPT ); Fri, 5 Apr 2019 03:14:24 -0400 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=nbd.name; s=20160729; h=Content-Transfer-Encoding:Content-Type:In-Reply-To: MIME-Version:Date:Message-ID:From:References:Cc:To:Subject:Sender:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:List-Id:List-Help:List-Unsubscribe: List-Subscribe:List-Post:List-Owner:List-Archive; bh=SS9zFrQ+2F+s/wQ90VCbxe3pU73VfYZuzTWJf2KgQY4=; b=sZWnbLRFsuGIwgt676+oI2pkJ1 iYsGk89rIo0avXnjgto9davmu6ELfHCYLQ7SvtZGRpP5HqOnka3onB9pzkAGagp6IoYxVm8VMvaS/ IHKm0zxD1uEyADuGbhReixHJxc+ILeoIUymOWbXXc2hwf3Su4ZhXCXGtxrf4vjWBETd0=; Received: from p4ff13bac.dip0.t-ipconnect.de ([79.241.59.172] helo=nf.local) by ds12 with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.89) (envelope-from ) id 1hCJ3E-0002JU-F4; Fri, 05 Apr 2019 09:14:13 +0200 Subject: Re: NAT performance regression caused by vlan GRO support To: =?UTF-8?B?UmFmYcWCIE1pxYJlY2tp?= , Toshiaki Makita Cc: Toshiaki Makita , netdev@vger.kernel.org, "David S. Miller" , Stefano Brivio , Sabrina Dubroca , David Ahern , Jo-Philipp Wich , Koen Vandeputte References: <73223229-6bc0-2647-6952-975961811866@gmail.com> <75961408-fd62-0f12-bd4b-79008b27576c@gmail.com> <53588a9f-8cc8-0ee5-0947-8ab2b2e56f15@gmail.com> From: Felix Fietkau Openpgp: preference=signencrypt Autocrypt: addr=nbd@nbd.name; prefer-encrypt=mutual; keydata= mQGiBEah5CcRBADIY7pu4LIv3jBlyQ/2u87iIZGe6f0f8pyB4UjzfJNXhJb8JylYYRzIOSxh ExKsdLCnJqsG1PY1mqTtoG8sONpwsHr2oJ4itjcGHfn5NJSUGTbtbbxLro13tHkGFCoCr4Z5 Pv+XRgiANSpYlIigiMbOkide6wbggQK32tC20QxUIwCg4k6dtV/4kwEeiOUfErq00TVqIiEE AKcUi4taOuh/PQWx/Ujjl/P1LfJXqLKRPa8PwD4j2yjoc9l+7LptSxJThL9KSu6gtXQjcoR2 vCK0OeYJhgO4kYMI78h1TSaxmtImEAnjFPYJYVsxrhay92jisYc7z5R/76AaELfF6RCjjGeP wdalulG+erWju710Bif7E1yjYVWeA/9Wd1lsOmx6uwwYgNqoFtcAunDaMKi9xVQW18FsUusM TdRvTZLBpoUAy+MajAL+R73TwLq3LnKpIcCwftyQXK5pEDKq57OhxJVv1Q8XkA9Dn1SBOjNB l25vJDFAT9ntp9THeDD2fv15yk4EKpWhu4H00/YX8KkhFsrtUs69+vZQwbQcRmVsaXggRmll dGthdSA8bmJkQG5iZC5uYW1lPohgBBMRAgAgBQJGoeQnAhsjBgsJCAcDAgQVAggDBBYCAwEC HgECF4AACgkQ130UHQKnbvXsvgCgjsAIIOsY7xZ8VcSm7NABpi91yTMAniMMmH7FRenEAYMa VrwYTIThkTlQuQINBEah5FQQCACMIep/hTzgPZ9HbCTKm9xN4bZX0JjrqjFem1Nxf3MBM5vN CYGBn8F4sGIzPmLhl4xFeq3k5irVg/YvxSDbQN6NJv8o+tP6zsMeWX2JjtV0P4aDIN1pK2/w VxcicArw0VYdv2ZCarccFBgH2a6GjswqlCqVM3gNIMI8ikzenKcso8YErGGiKYeMEZLwHaxE Y7mTPuOTrWL8uWWRL5mVjhZEVvDez6em/OYvzBwbkhImrryF29e3Po2cfY2n7EKjjr3/141K DHBBdgXlPNfDwROnA5ugjjEBjwkwBQqPpDA7AYPvpHh5vLbZnVGu5CwG7NAsrb2isRmjYoqk wu++3117AAMFB/9S0Sj7qFFQcD4laADVsabTpNNpaV4wAgVTRHKV/kC9luItzwDnUcsZUPdQ f3MueRJ3jIHU0UmRBG3uQftqbZJj3ikhnfvyLmkCNe+/hXhPu9sGvXyi2D4vszICvc1KL4RD aLSrOsROx22eZ26KqcW4ny7+va2FnvjsZgI8h4sDmaLzKczVRIiLITiMpLFEU/VoSv0m1F4B FtRgoiyjFzigWG0MsTdAN6FJzGh4mWWGIlE7o5JraNhnTd+yTUIPtw3ym6l8P+gbvfoZida0 TspgwBWLnXQvP5EDvlZnNaKa/3oBes6z0QdaSOwZCRA3QSLHBwtgUsrT6RxRSweLrcabiEkE GBECAAkFAkah5FQCGwwACgkQ130UHQKnbvW2GgCfTKx80VvCR/PvsUlrvdOLsIgeRGAAn1ee RjMaxwtSdaCKMw3j33ZbsWS4 Message-ID: <45b6fe37-ba1a-91c2-1d4a-2d045793babd@nbd.name> Date: Fri, 5 Apr 2019 09:14:11 +0200 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On 2019-04-05 09:11, Rafał Miłecki wrote: > On 05.04.2019 07:48, Rafał Miłecki wrote: >> On 05.04.2019 06:26, Toshiaki Makita wrote: >>> My test results: >>> >>> Receiving packets from eth0.10, forwarding them to eth0.20 and applying >>> MASQUERADE on eth0.20, using i40e 25G NIC on kernel 4.20.13. >>> Disabled rxvlan by ethtool -K to exercise vlan_gro_receive(). >>> Measured TCP throughput by netperf. >>> >>> GRO on : 17 Gbps >>> GRO off:  5 Gbps >>> >>> So I failed to reproduce your problem. >> >> :( Thanks for trying & checking that! >> >> >>> Would you check the CPU usage by "mpstat -P ALL" or similar (like "sar >>> -u ALL -P ALL") to check if the traffic is able to consume 100% CPU on >>> your machine? >> >> 1) ethtool -K eth0 gro on + iperf running (577 Mb/s) >> root@OpenWrt:/# mpstat -P ALL 10 3 >> Linux 5.1.0-rc3+ (OpenWrt)      03/27/19        _armv7l_        (2 CPU) >> >> 16:33:40     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle >> 16:33:50     all    0.00    0.00    0.00    0.00    0.00   58.79    0.00    0.00   41.21 >> 16:33:50       0    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00 >> 16:33:50       1    0.00    0.00    0.00    0.00    0.00   17.58    0.00    0.00   82.42 >> >> 16:33:50     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle >> 16:34:00     all    0.00    0.00    0.05    0.00    0.00   59.44    0.00    0.00   40.51 >> 16:34:00       0    0.00    0.00    0.10    0.00    0.00   99.90    0.00    0.00    0.00 >> 16:34:00       1    0.00    0.00    0.00    0.00    0.00   18.98    0.00    0.00   81.02 >> >> 16:34:00     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle >> 16:34:10     all    0.00    0.00    0.00    0.00    0.00   59.59    0.00    0.00   40.41 >> 16:34:10       0    0.00    0.00    0.00    0.00    0.00  100.00    0.00    0.00    0.00 >> 16:34:10       1    0.00    0.00    0.00    0.00    0.00   19.18    0.00    0.00   80.82 >> >> Average:     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle >> Average:     all    0.00    0.00    0.02    0.00    0.00   59.27    0.00    0.00   40.71 >> Average:       0    0.00    0.00    0.03    0.00    0.00   99.97    0.00    0.00    0.00 >> Average:       1    0.00    0.00    0.00    0.00    0.00   18.58    0.00    0.00   81.42 >> >> >> 2) ethtool -K eth0 gro off + iperf running (941 Mb/s) >> root@OpenWrt:/# mpstat -P ALL 10 3 >> Linux 5.1.0-rc3+ (OpenWrt)      03/27/19        _armv7l_        (2 CPU) >> >> 16:34:39     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle >> 16:34:49     all    0.00    0.00    0.05    0.00    0.00   86.91    0.00    0.00   13.04 >> 16:34:49       0    0.00    0.00    0.10    0.00    0.00   78.22    0.00    0.00   21.68 >> 16:34:49       1    0.00    0.00    0.00    0.00    0.00   95.60    0.00    0.00    4.40 >> >> 16:34:49     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle >> 16:34:59     all    0.00    0.00    0.10    0.00    0.00   87.06    0.00    0.00   12.84 >> 16:34:59       0    0.00    0.00    0.20    0.00    0.00   79.72    0.00    0.00   20.08 >> 16:34:59       1    0.00    0.00    0.00    0.00    0.00   94.41    0.00    0.00    5.59 >> >> 16:34:59     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle >> 16:35:09     all    0.00    0.00    0.05    0.00    0.00   85.71    0.00    0.00   14.24 >> 16:35:09       0    0.00    0.00    0.10    0.00    0.00   79.42    0.00    0.00   20.48 >> 16:35:09       1    0.00    0.00    0.00    0.00    0.00   92.01    0.00    0.00    7.99 >> >> Average:     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle >> Average:     all    0.00    0.00    0.07    0.00    0.00   86.56    0.00    0.00   13.37 >> Average:       0    0.00    0.00    0.13    0.00    0.00   79.12    0.00    0.00   20.75 >> Average:       1    0.00    0.00    0.00    0.00    0.00   94.01    0.00    0.00    5.99 >> >> >> 3) System idle (no iperf) >> root@OpenWrt:/# mpstat -P ALL 10 1 >> Linux 5.1.0-rc3+ (OpenWrt)      03/27/19        _armv7l_        (2 CPU) >> >> 16:35:31     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle >> 16:35:41     all    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00 >> 16:35:41       0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00 >> 16:35:41       1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00 >> >> Average:     CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle >> Average:     all    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00 >> Average:       0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00 >> Average:       1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00 >> >> >>> If CPU is 100%, perf may help us analyze your problem. If it's >>> available, try running below while testing: >>> # perf record -a -g -- sleep 5 >>> >>> And then run this after testing: >>> # perf report --no-child >> >> I can see my CPU 0 is fully loaded when using "gro on". I'll try perf now. > > I guess its GRO + csum_partial() to be blamed for this performance drop. > > Maybe csum_partial() is very fast on your powerful machine and few extra calls > don't make a difference? I can imagine it affecting much slower home router with > ARM cores. Most high performance Ethernet devices implement hardware checksum offload, which completely gets rid of this overhead. Unfortunately, the BCM53xx/47xx Ethernet MAC doesn't have this, which is why you're getting such crappy performance. - Felix