From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.7 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,FROM_EXCESS_BASE64, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5E6CEC10F00 for ; Fri, 5 Apr 2019 08:12:46 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 2E40A2186A for ; Fri, 5 Apr 2019 08:12:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="ESeJkn4z" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1730447AbfDEIMm (ORCPT ); Fri, 5 Apr 2019 04:12:42 -0400 Received: from mail-ed1-f68.google.com ([209.85.208.68]:38450 "EHLO mail-ed1-f68.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726594AbfDEIMk (ORCPT ); Fri, 5 Apr 2019 04:12:40 -0400 Received: by mail-ed1-f68.google.com with SMTP id q14so4690490edr.5 for ; Fri, 05 Apr 2019 01:12:39 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=FE9mrJMZgJK5GNDQ/v3FPESx29YB9gsQCZ4fxfDi6wk=; b=ESeJkn4zu4DR0u5p3K+Lbd4Huh14MpS4udAVY6CxTFQwB5QEnAxJ4lJsVQIIuhSETq JAgRz6US/sCj0AICELuhaxtdooUrh2zCpJwbNbrB5BTQuZ+zCX/2MVqr+++H00575bTo qnLWpqn+BI8SY3+cKuiyB0kHiEo0uJqi1Ko6eUM6uD/adPNBT1pXnZ/aLEXQYtTKynVi wmUDvC7rcsmWAuoBftCGfI6kZVvMaibm/KyhW2N5l5SaVjDGnT0gQqkiOE9K8Vwfu3Lm v97hZAlCW8ztcIk32M854Ie3J2ofcLBaO50Fx8ODWsbseLmfECRjNOOy4UkTTnMusQ3b Ga8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=FE9mrJMZgJK5GNDQ/v3FPESx29YB9gsQCZ4fxfDi6wk=; b=VLu+xAt8jDDcrDtNu/eyJd9mVC+PRKotUWR9tNxDKU0kfn8vDIsgpKJVNPhaEU6fK9 rh0Wm+d+UjhFyPKgG/5R+E8uaOZW8L6/gTGMr+4B25tQ1q/DLMrPl+5844Gy1pZjCpIb uvlCOJzU8GOd7xmVw/pZuHdvCfMsjhUUxb6hxl9CdGjo5lhpD1Ai4BiHTUDILub6gl8v hiMjRwngXvULbzgDI8TMWbR6HuXdu6AD8UJb0mHyYZO/4uxtYXodNIUeddG7Dp7Fg+H+ D9cQuas7/WAmeLsRTfKBGF4ZXurW9X/kXQHr4ly5ZdLrxchDlny3HJKYSUh1zvw2PXpY Jl1g== X-Gm-Message-State: APjAAAU84nCgFWO6CHH0/UtfaIb+jJjqsJTXg/KgKGOkbO/uqKGETXEA CMshQYDIRqxZIbG8sUzeehI= X-Google-Smtp-Source: APXvYqwbT/mgj7dceH7XcObxc43sywvyD7UU698xTlQoqnR7copv+VZ+BRzAbl7r+/xLfYtOi3B+8w== X-Received: by 2002:aa7:c442:: with SMTP id n2mr2126953edr.55.1554451958843; Fri, 05 Apr 2019 01:12:38 -0700 (PDT) Received: from elitebook.lan (ip-194-187-74-233.konfederacka.maverick.com.pl. [194.187.74.233]) by smtp.googlemail.com with ESMTPSA id z35sm6299306edd.81.2019.04.05.01.12.36 (version=TLS1_3 cipher=AEAD-AES128-GCM-SHA256 bits=128/128); Fri, 05 Apr 2019 01:12:37 -0700 (PDT) Subject: Re: NAT performance regression caused by vlan GRO support To: Toshiaki Makita , Felix Fietkau Cc: Toshiaki Makita , netdev@vger.kernel.org, "David S. Miller" , Stefano Brivio , Sabrina Dubroca , David Ahern , Jo-Philipp Wich , Koen Vandeputte References: <73223229-6bc0-2647-6952-975961811866@gmail.com> <75961408-fd62-0f12-bd4b-79008b27576c@gmail.com> <53588a9f-8cc8-0ee5-0947-8ab2b2e56f15@gmail.com> <45b6fe37-ba1a-91c2-1d4a-2d045793babd@nbd.name> <67d634cd-cf16-df21-7b8a-5d865d95e4e6@lab.ntt.co.jp> From: =?UTF-8?B?UmFmYcWCIE1pxYJlY2tp?= Message-ID: <31acd23f-6973-1912-7fcc-575a5d4e00e7@gmail.com> Date: Fri, 5 Apr 2019 10:12:35 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.5.2 MIME-Version: 1.0 In-Reply-To: <67d634cd-cf16-df21-7b8a-5d865d95e4e6@lab.ntt.co.jp> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On 05.04.2019 09:58, Toshiaki Makita wrote: > On 2019/04/05 16:14, Felix Fietkau wrote: >> On 2019-04-05 09:11, Rafał Miłecki wrote: >>> I guess its GRO + csum_partial() to be blamed for this performance drop. >>> >>> Maybe csum_partial() is very fast on your powerful machine and few extra calls >>> don't make a difference? I can imagine it affecting much slower home router with >>> ARM cores. >> Most high performance Ethernet devices implement hardware checksum >> offload, which completely gets rid of this overhead. >> Unfortunately, the BCM53xx/47xx Ethernet MAC doesn't have this, which is >> why you're getting such crappy performance. > > Hmm... now I disabled rx checksum and tried the test again, and indeed I > see csum_partial from GRO path. But I also see csum_partial even without > GRO from nf_conntrack_in -> tcp_packet -> __skb_checksum_complete. > Probably Rafał disabled nf_conntrack_checksum sysctl knob? > > But anyway even with disabling rx csum offload my machine has better > performance with GRO. I'm sure in some cases GRO should be disabled, but > I guess it's difficult to determine whether we should disable GRO or not > automatically when csum offload is not available. Few testing results: 1) ethtool -K eth0 gro off; echo 0 > /proc/sys/net/netfilter/nf_conntrack_checksum [ 6] 0.0-60.0 sec 6.57 GBytes 940 Mbits/sec 2) ethtool -K eth0 gro off; echo 1 > /proc/sys/net/netfilter/nf_conntrack_checksum [ 6] 0.0-60.0 sec 4.65 GBytes 666 Mbits/sec 3) ethtool -K eth0 gro on; echo 0 > /proc/sys/net/netfilter/nf_conntrack_checksum [ 6] 0.0-60.0 sec 4.02 GBytes 575 Mbits/sec 4) ethtool -K eth0 gro on; echo 1 > /proc/sys/net/netfilter/nf_conntrack_checksum [ 6] 0.0-60.0 sec 4.04 GBytes 579 Mbits/sec