From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.7 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,FROM_EXCESS_BASE64, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D3804C10F0E for ; Sun, 7 Apr 2019 11:55:05 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 975E720880 for ; Sun, 7 Apr 2019 11:55:05 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lzvFd53q" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726531AbfDGLzE (ORCPT ); Sun, 7 Apr 2019 07:55:04 -0400 Received: from mail-ed1-f67.google.com ([209.85.208.67]:36220 "EHLO mail-ed1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726263AbfDGLzE (ORCPT ); Sun, 7 Apr 2019 07:55:04 -0400 Received: by mail-ed1-f67.google.com with SMTP id s16so9196165edr.3 for ; Sun, 07 Apr 2019 04:55:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:references:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=zuz4cL8KQGircLIuaI/dzos/6jHAUrJTxYchkhi2KA4=; b=lzvFd53qO/GYAia1BjxvgvBVL1sskWyH6j3KiTkb3QQXeH7c6rfH9zFd+SoHHuigMr HqB1jdrY4YqAxUiiBcWI4iD06DCwYNIDjqTSRTyWj62O1rtG1Zu+o0U6jTNkmNEnMif8 5mXNpH4p3EfQWBGcYdt4m8G91IXF4WwbGfAjTknHGgG9giMzh2H8c/Qt9hmlZp0Q+Kwx 45MvLlnI+0D7m2aWj30mMNsJ06M6Ob7Esocm2gvyM9jC3TZ7pNZe7bO0G8BrEiOoNEHO pB7GJefDsD36NZT8J0PCh/++aoDShF5Ik3qUodK/z/WEp38ZF5Pt9NxPkW7kmex3Ph1s Q5mA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:references:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=zuz4cL8KQGircLIuaI/dzos/6jHAUrJTxYchkhi2KA4=; b=PGNcpv1INEuW7AwZP955YXPmUXadLB+dl5TMsXLhBNxoUf5X2EPvheWCXcw1/h6fXX ONKNRhN5aZjLH8wbITURi1jHzfrHcqrQNqq+onQJrb3sfkOS15IIeO2SpKR0k33j58Sc BoPMKcvoQJS75pdUzymb21B5a32JIMPSFBd9CP4xKBNuO5ZCswUtiGam6x9a5WH5A/yf 5l+1P4NiYFcV0rPBIpiAnDnmRk3QOCg0E27i38wm5EetEC7ENAEBafYvPhufBh9b1wq/ 7n8svNwRONwa6e9UohduCv43BuIazM2BHhio7RtsXO9vhkRzRv2biQbSsi6UzIwibMNt t2CA== X-Gm-Message-State: APjAAAW9dG2pzlbAstLI3vRRa03UAbV6RKOPuI9dYFrbcMlwLbBMQiPD Mh5P6D7yPA2j6boGkIjbwf4= X-Google-Smtp-Source: APXvYqz8niDbgQXycNCaLsIvVNqlXpZi0u/uMC4X0a71fNlfhUV4k7t/BC/VkVhKl5IYke/Uw/QYRw== X-Received: by 2002:a50:c40f:: with SMTP id v15mr14726542edf.236.1554638101794; Sun, 07 Apr 2019 04:55:01 -0700 (PDT) Received: from elitebook.lan (ip-194-187-74-233.konfederacka.maverick.com.pl. [194.187.74.233]) by smtp.googlemail.com with ESMTPSA id m20sm3999393ejz.72.2019.04.07.04.55.00 (version=TLS1_3 cipher=AEAD-AES128-GCM-SHA256 bits=128/128); Sun, 07 Apr 2019 04:55:01 -0700 (PDT) Subject: Re: NAT performance regression caused by vlan GRO support From: =?UTF-8?B?UmFmYcWCIE1pxYJlY2tp?= To: netdev@vger.kernel.org, "David S. Miller" , Toshiaki Makita , =?UTF-8?Q?Toke_H=c3=b8iland-J=c3=b8rgensen?= , Florian Westphal , Eric Dumazet Cc: Stefano Brivio , Sabrina Dubroca , David Ahern , Felix Fietkau , Jo-Philipp Wich , Koen Vandeputte References: <73223229-6bc0-2647-6952-975961811866@gmail.com> Message-ID: <2149862a-b12e-4025-c51d-6857d26b9a77@gmail.com> Date: Sun, 7 Apr 2019 13:54:59 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.5.2 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org Now I have some questions regarding possible optimizations. Note I'm too familiar with the net subsystem so maybe I got wrong ideas. On 07.04.2019 13:53, Rafał Miłecki wrote: > On 04.04.2019 14:57, Rafał Miłecki wrote: >> Long story short, starting with the commit 66e5133f19e9 ("vlan: Add GRO support >> for non hardware accelerated vlan") - which first hit kernel 4.2 - NAT >> performance of my router dropped by 30% - 40%. > > I'll try to provide some summary for this issue. I'll focus on TCP traffic as > that's what I happened to test. > > Basically all slowdowns are related to the csum_partial(). Calculating checksum > has a significant impact on NAT performance on less CPU powerful devices. > > ********** > > GRO disabled > > Without GRO a csum_partial() is used only when validating TCP packets in the > nf_conntrack_tcp_packet() (known as tcp_packet() in kernels older than 5.1). > > Simplified forward trace for that case: > nf_conntrack_in >     nf_conntrack_tcp_packet >         tcp_error >             if (state->net->ct.sysctl_checksum) >                 nf_checksum >                     nf_ip_checksum >                         __skb_checksum_complete > > That validation can be disabled using nf_conntrack_checksum sysfs and it bumps > NAT speed for me from 666 Mb/s to 940 Mb/s (+41%). > > ********** > > GRO enabled > > First of all GRO also includes TCP validation that requires calculating a > checksum. > > Simplified forward trace for that case: > vlan_gro_receive >     call_gro_receive >         inet_gro_receive >             indirect_call_gro_receive >                 tcp4_gro_receive >                     skb_gro_checksum_validate >                     tcp_gro_receive > > *If* we had a way to disable that validation it *would* result in bumping NAT > speed for me from 577 Mb/s to 825 Mb/s (+43%). Could we have tcp4_gro_receive() behave similarly to the tcp_error() and make it respect the nf_conntrack_checksum sysfs value? Could we simply add something like: if (dev_net(skb->dev)->ct.sysctl_checksum) to it (to additionally protect a skb_gro_checksum_validate() call)? > Secondly using GRO means we need to calculate a checksum before transmitting > packets (applies to devices without HW checksum offloading). I think it's > related to packets merging in the skb_gro_receive() and then setting > CHECKSUM_PARTIAL: > > vlan_gro_complete >     inet_gro_complete >         tcp4_gro_complete >             tcp_gro_complete >                 skb->ip_summed = CHECKSUM_PARTIAL; > > That results in bgmac calculating a checksum from the scratch, take a look at > the bgmac_dma_tx_add() which does: > > if (skb->ip_summed == CHECKSUM_PARTIAL) >     skb_checksum_help(skb); > > Performing that whole checksum calculation will always result in GRO slowing > down NAT for me when using BCM47094 SoC with that not-so-powerful ARM CPUs. Is this possible to avoid CHECKSUM_PARTIAL & skb_checksum_help() which has to calculate a whole checksum? It's definitely possible to *update* checksum after simple packet changes (e.g. amending an IP or port). Would that be possible to use similar method when dealing with packets with GRO enabled? If not, maybe w really need to think about some good & clever condition for disabling GRO by default on hw without checksum offloading.