From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.7 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM,FROM_EXCESS_BASE64, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C7F7C10F0E for ; Sun, 7 Apr 2019 11:53:26 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 1274E20880 for ; Sun, 7 Apr 2019 11:53:26 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="Wdw1QloE" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1726397AbfDGLxY (ORCPT ); Sun, 7 Apr 2019 07:53:24 -0400 Received: from mail-ed1-f67.google.com ([209.85.208.67]:44288 "EHLO mail-ed1-f67.google.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1726263AbfDGLxY (ORCPT ); Sun, 7 Apr 2019 07:53:24 -0400 Received: by mail-ed1-f67.google.com with SMTP id d11so9167087edp.11 for ; Sun, 07 Apr 2019 04:53:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:references:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=5+BeyeTOTHng9HGvPbiKHTVe+WhPoQ0dj7I9wPeCP1I=; b=Wdw1QloEWbvaCJPVfCABLeptFU8v6to3wI5FNzwJdGIidXTU2TrBOuWrjFxG/17Wq6 VARAtGb4AubRj1mAgeycDnMZlk10zLmouRba63XYEKfiebbdZCpLGeOxSpFu+l1wP9dR MnsYpT2Y+FL3IeoRXWsRVjY6pJzmLG5C6/FfyjURHFUptiBYQfKHyYdLFkv96l5BzoxJ Q+WNJiXrabcWU3KAaxBm6+ngXKeHrtOwXSqWTBCQF4B3IStPNOl7Rdja3NzNSisoO4HF 3+1gBg7ICcsQNn5b/pzhMbZqE77Ru/m8CrbayJjR0hr05mOUpzWvJvc9NfT7H7tlywqQ BT5g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:references:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=5+BeyeTOTHng9HGvPbiKHTVe+WhPoQ0dj7I9wPeCP1I=; b=hbeF9aI+wvX93HhWy3QIK2M6GEo0sNNkvcRe8THnoHrc4ucoAJw8EjP8ov9rEHMT3O 45tadQvXyfdeaA7lXLxrGIFhwr0lSVNNOKBMB/dY+wABKblGwoP5LoHyoLI3XHyM+ahz QIQLdyY5rbJJsWgLWUBeJuOtRgNduSKRMnUAdBiNzqi8j7BOD5zeZ/OcdUZKnC6IE7wf BaUdnDSmbRLMsPU1br6oOKdCSh4yhkVclzOorT4/NTKd4VEHz1eDqPQiQXnkp8WLC5NX FzL+gSRaMzHFLrWREd6ySLCD0SDP4FHpS8B15wu9xUOGOD+t0VUCnRrJnBBSz8g2rzSg zH+g== X-Gm-Message-State: APjAAAXgdLHtH3nQ/bXJz55J1nMiDi7E8QQVGq5EheoepfiscGb9a/IE EUhTK+jzznmZCmaMSeQVbEU= X-Google-Smtp-Source: APXvYqx2DYb55TCYnk8nruKIGtkVnIEcRC+BY1/WOURLbxBKOhZfIllTcLL/W+bq19qsSU8eDVNRUA== X-Received: by 2002:a50:ee11:: with SMTP id g17mr7440557eds.102.1554638002951; Sun, 07 Apr 2019 04:53:22 -0700 (PDT) Received: from elitebook.lan (ip-194-187-74-233.konfederacka.maverick.com.pl. [194.187.74.233]) by smtp.googlemail.com with ESMTPSA id k8sm7915981edr.85.2019.04.07.04.53.21 (version=TLS1_3 cipher=AEAD-AES128-GCM-SHA256 bits=128/128); Sun, 07 Apr 2019 04:53:21 -0700 (PDT) Subject: Re: NAT performance regression caused by vlan GRO support From: =?UTF-8?B?UmFmYcWCIE1pxYJlY2tp?= To: netdev@vger.kernel.org, "David S. Miller" , Toshiaki Makita , =?UTF-8?Q?Toke_H=c3=b8iland-J=c3=b8rgensen?= , Florian Westphal , Eric Dumazet Cc: Stefano Brivio , Sabrina Dubroca , David Ahern , Felix Fietkau , Jo-Philipp Wich , Koen Vandeputte References: <73223229-6bc0-2647-6952-975961811866@gmail.com> Message-ID: Date: Sun, 7 Apr 2019 13:53:19 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.5.2 MIME-Version: 1.0 In-Reply-To: <73223229-6bc0-2647-6952-975961811866@gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 8bit Sender: netdev-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org On 04.04.2019 14:57, Rafał Miłecki wrote: > Long story short, starting with the commit 66e5133f19e9 ("vlan: Add GRO support > for non hardware accelerated vlan") - which first hit kernel 4.2 - NAT > performance of my router dropped by 30% - 40%. I'll try to provide some summary for this issue. I'll focus on TCP traffic as that's what I happened to test. Basically all slowdowns are related to the csum_partial(). Calculating checksum has a significant impact on NAT performance on less CPU powerful devices. ********** GRO disabled Without GRO a csum_partial() is used only when validating TCP packets in the nf_conntrack_tcp_packet() (known as tcp_packet() in kernels older than 5.1). Simplified forward trace for that case: nf_conntrack_in nf_conntrack_tcp_packet tcp_error if (state->net->ct.sysctl_checksum) nf_checksum nf_ip_checksum __skb_checksum_complete That validation can be disabled using nf_conntrack_checksum sysfs and it bumps NAT speed for me from 666 Mb/s to 940 Mb/s (+41%). ********** GRO enabled First of all GRO also includes TCP validation that requires calculating a checksum. Simplified forward trace for that case: vlan_gro_receive call_gro_receive inet_gro_receive indirect_call_gro_receive tcp4_gro_receive skb_gro_checksum_validate tcp_gro_receive *If* we had a way to disable that validation it *would* result in bumping NAT speed for me from 577 Mb/s to 825 Mb/s (+43%). Secondly using GRO means we need to calculate a checksum before transmitting packets (applies to devices without HW checksum offloading). I think it's related to packets merging in the skb_gro_receive() and then setting CHECKSUM_PARTIAL: vlan_gro_complete inet_gro_complete tcp4_gro_complete tcp_gro_complete skb->ip_summed = CHECKSUM_PARTIAL; That results in bgmac calculating a checksum from the scratch, take a look at the bgmac_dma_tx_add() which does: if (skb->ip_summed == CHECKSUM_PARTIAL) skb_checksum_help(skb); Performing that whole checksum calculation will always result in GRO slowing down NAT for me when using BCM47094 SoC with that not-so-powerful ARM CPUs.