From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id AF338C4708E for ; Thu, 5 Jan 2023 19:56:41 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S235140AbjAET4h (ORCPT ); Thu, 5 Jan 2023 14:56:37 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:44968 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232270AbjAET4f (ORCPT ); Thu, 5 Jan 2023 14:56:35 -0500 Received: from mail-yb1-xb2f.google.com (mail-yb1-xb2f.google.com [IPv6:2607:f8b0:4864:20::b2f]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id C7179F00C for ; Thu, 5 Jan 2023 11:56:34 -0800 (PST) Received: by mail-yb1-xb2f.google.com with SMTP id p188so5954847yba.5 for ; Thu, 05 Jan 2023 11:56:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gooddata.com; s=google; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=Pb4JzoqwO6Y7xcnirhQ6av4NijRSytITGxbvvvwNLyk=; b=HYEbK6iYzsRavkIoiY9kI2B8zpjbMzzXT3Yb5Y2nJBfgxN/Q9+t4mwnVQD6HC6Lrfs vm/DJ4sUpjLePsmHmn/LLBI4WY9WXTwu4zRaxUrO2T9AFJVKzdsQuYUXNh7q13je5g3K 3GdJsHpeaKKfXe77LXtgpDuY4lp949pTVTzV0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Pb4JzoqwO6Y7xcnirhQ6av4NijRSytITGxbvvvwNLyk=; b=8I7tAMlLrlayZS8fdyWpMfa52fhLxUk4bnvKZB4mvfdwA0ruAvVuZJo+Ft7scJ6eMZ Mg4D8LMXn8eIoDx5EyvyMKgsPYpjopsxun4n9mu1EKFHy0u73yxhYAF0QexAdIfAVyh6 F88xpIcc6AOChEi7KLgnECq5tF3yjM+QVUjwDj2J/0+tyE6Ip1ng5joeO8oL9wavkDh9 JJCHxG3/tBszfJkncTN02lkQgkSlI76U1zV58tLDHKA4C6Wq38vR2Y2Kg+knu1vm7ZMp 5TKuE8hD5NvWJ2esCl2sRV3hbcMOoCXJySO49iHG9+qBZS1nKOKIra6wASXH5C/0HwuO 5v+Q== X-Gm-Message-State: AFqh2krzxFYt47JFsOwJcx+Co0Eu41yR0d0yv4Pc8K7xa2104wVp2fkD XQFnnoyHtWFJIM/OzbwEWyaB2xyZI3iuZnEwVZe2Kg== X-Google-Smtp-Source: AMrXdXv7aIMY3mYffkOneZWpGuT7EpAPBbHhrTEm/5MmoCWyicdgGvLU60gEKU27fAzrZBKY9oGj4C35bq3unjWsS88= X-Received: by 2002:a25:73d3:0:b0:700:1dd4:bbfa with SMTP id o202-20020a2573d3000000b007001dd4bbfamr7139222ybc.455.1672948593905; Thu, 05 Jan 2023 11:56:33 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Jaroslav Pulchart Date: Thu, 5 Jan 2023 20:56:08 +0100 Message-ID: Subject: Re: Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack) To: Eric Dumazet Cc: netdev@vger.kernel.org, lixiaoyan@google.com, pabeni@redhat.com, davem@davemloft.net, Igor Raits Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: netdev@vger.kernel.org I can and I did. Your "Random guess" patch fix the problem and speed is like with 6.0.y: # git clone ... ... Receiving objects: 100% (571306/571306), 350.58 MiB | 67.51 MiB/s, done. Thanks, JP =C4=8Dt 5. 1. 2023 v 19:49 odes=C3=ADlatel Eric Dumazet napsal: > > On Thu, Jan 5, 2023 at 6:54 PM Jaroslav Pulchart > wrote: > > > > It is at KVM based VMs "CentOS 9 Stream" and "CentOS 8 Stream" using > > upstream kernel 6.1.y. Hosted on Dell PowerEdge 7525 servers (2x AMD > > 74F3) and OS CentOS 9 Stream again with upstream kernel 6.1.y or > > 6.0.y. > > > > # ethtool -k eth0 > > Features for eth0: > > rx-checksumming: on [fixed] > > tx-checksumming: on > > tx-checksum-ipv4: off [fixed] > > tx-checksum-ip-generic: on > > tx-checksum-ipv6: off [fixed] > > tx-checksum-fcoe-crc: off [fixed] > > tx-checksum-sctp: off [fixed] > > scatter-gather: on > > tx-scatter-gather: on > > tx-scatter-gather-fraglist: off [fixed] > > tcp-segmentation-offload: on > > tx-tcp-segmentation: on > > tx-tcp-ecn-segmentation: on > > tx-tcp-mangleid-segmentation: off > > tx-tcp6-segmentation: on > > generic-segmentation-offload: on > > generic-receive-offload: on > > large-receive-offload: off [fixed] > > rx-vlan-offload: off [fixed] > > tx-vlan-offload: off [fixed] > > ntuple-filters: off [fixed] > > receive-hashing: off [fixed] > > highdma: on [fixed] > > rx-vlan-filter: on [fixed] > > vlan-challenged: off [fixed] > > tx-lockless: off [fixed] > > netns-local: off [fixed] > > tx-gso-robust: on [fixed] > > tx-fcoe-segmentation: off [fixed] > > tx-gre-segmentation: off [fixed] > > tx-gre-csum-segmentation: off [fixed] > > tx-ipxip4-segmentation: off [fixed] > > tx-ipxip6-segmentation: off [fixed] > > tx-udp_tnl-segmentation: off [fixed] > > tx-udp_tnl-csum-segmentation: off [fixed] > > tx-gso-partial: off [fixed] > > tx-tunnel-remcsum-segmentation: off [fixed] > > tx-sctp-segmentation: off [fixed] > > tx-esp-segmentation: off [fixed] > > tx-udp-segmentation: off [fixed] > > tx-gso-list: off [fixed] > > fcoe-mtu: off [fixed] > > tx-nocache-copy: off > > loopback: off [fixed] > > rx-fcs: off [fixed] > > rx-all: off [fixed] > > tx-vlan-stag-hw-insert: off [fixed] > > rx-vlan-stag-hw-parse: off [fixed] > > rx-vlan-stag-filter: off [fixed] > > l2-fwd-offload: off [fixed] > > hw-tc-offload: off [fixed] > > esp-hw-offload: off [fixed] > > esp-tx-csum-hw-offload: off [fixed] > > rx-udp_tunnel-port-offload: off [fixed] > > tls-hw-tx-offload: off [fixed] > > tls-hw-rx-offload: off [fixed] > > rx-gro-hw: on > > tls-hw-record: off [fixed] > > rx-gro-list: off > > macsec-hw-offload: off [fixed] > > rx-udp-gro-forwarding: off > > hsr-tag-ins-offload: off [fixed] > > hsr-tag-rm-offload: off [fixed] > > hsr-fwd-offload: off [fixed] > > hsr-dup-offload: off [fixed] > > > > # ethtool -i eth0 > > driver: virtio_net > > version: 1.0.0 > > firmware-version: > > expansion-rom-version: > > bus-info: 0000:03:00.0 > > supports-statistics: yes > > supports-test: no > > supports-eeprom-access: no > > supports-register-dump: no > > supports-priv-flags: no > > Random guess, can you try: > > diff --git a/net/core/gro.c b/net/core/gro.c > index fd8c6a7e8d3e2e6b439109d0089f44a547c7347e..f162674e7ae1bdf96bcbf7e1e= d7326729d862f9a > 100644 > --- a/net/core/gro.c > +++ b/net/core/gro.c > @@ -500,7 +500,8 @@ static enum gro_result dev_gro_receive(struct > napi_struct *napi, struct sk_buff > BUILD_BUG_ON(!IS_ALIGNED(offsetof(struct napi_gro_cb, zeroed), > sizeof(u32))); /* Avoid slow > unaligned acc */ > *(u32 *)&NAPI_GRO_CB(skb)->zeroed =3D 0; > - NAPI_GRO_CB(skb)->flush =3D skb_has_frag_list(skb); > + NAPI_GRO_CB(skb)->flush =3D skb_has_frag_list(skb) || > + (skb_shinfo(skb)->gso_type & SKB_GSO_DO= DGY); > NAPI_GRO_CB(skb)->is_atomic =3D 1; > NAPI_GRO_CB(skb)->count =3D 1; > if (unlikely(skb_is_gso(skb))) { > > > > > > > =C4=8Dt 5. 1. 2023 v 18:43 odes=C3=ADlatel Eric Dumazet napsal: > > > > > > On Thu, Jan 5, 2023 at 6:37 PM Jaroslav Pulchart > > > wrote: > > > > > > > > Hello, > > > > > > > > I would like to report a 6.1,y regression in a network performance > > > > observed when using "git clone". > > > > > > > > BAD: "git clone" speed with kernel 6.1,y: > > > > # git clone git@github.com:..../.....git > > > > ... > > > > Receiving objects: 8% (47797/571306), 20.69 MiB | 3.27 MiB/s > > > > > > > > GOOD: "git clone" speed with kernel 6.0,y: > > > > # git clone git@github.com:..../.....git > > > > ... > > > > Receiving objects: 72% (411341/571306), 181.05 MiB | 60.27 MiB/= s > > > > > > > > I bisected the issue to a commit > > > > 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)g= ro > > > > packets to gro stack". Reverting it from 6.1.y branch makes the git > > > > clone fast like with 6.0.y. > > > > > > > > > > Hmm, please provide more information. > > > > > > NIC used ? (ethtool -i eth0) > > > > > > ethtool -k eth0 # replace by your netdev name > > > > > > And packet captures would be nice (with and without the patch) > > > > > > Thanks. > > > > > > > > -- > > Jaroslav Pulchart > > Sr. Principal SW Engineer > > GoodData --=20 Jaroslav Pulchart Sr. Principal SW Engineer GoodData