Re: Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack)

[Eric Dumazet <edumazet@google.com>]

On Thu, Jan 5, 2023 at 6:54 PM Jaroslav Pulchart
<jaroslav.pulchart@gooddata.com> wrote:
>
> It is at KVM based VMs "CentOS 9 Stream" and "CentOS 8 Stream" using
> upstream kernel 6.1.y. Hosted on Dell PowerEdge 7525 servers (2x AMD
> 74F3) and OS CentOS 9 Stream again with upstream kernel 6.1.y or
> 6.0.y.
>
> # ethtool -k eth0
> Features for eth0:
> rx-checksumming: on [fixed]
> tx-checksumming: on
> tx-checksum-ipv4: off [fixed]
> tx-checksum-ip-generic: on
> tx-checksum-ipv6: off [fixed]
> tx-checksum-fcoe-crc: off [fixed]
> tx-checksum-sctp: off [fixed]
> scatter-gather: on
> tx-scatter-gather: on
> tx-scatter-gather-fraglist: off [fixed]
> tcp-segmentation-offload: on
> tx-tcp-segmentation: on
> tx-tcp-ecn-segmentation: on
> tx-tcp-mangleid-segmentation: off
> tx-tcp6-segmentation: on
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off [fixed]
> rx-vlan-offload: off [fixed]
> tx-vlan-offload: off [fixed]
> ntuple-filters: off [fixed]
> receive-hashing: off [fixed]
> highdma: on [fixed]
> rx-vlan-filter: on [fixed]
> vlan-challenged: off [fixed]
> tx-lockless: off [fixed]
> netns-local: off [fixed]
> tx-gso-robust: on [fixed]
> tx-fcoe-segmentation: off [fixed]
> tx-gre-segmentation: off [fixed]
> tx-gre-csum-segmentation: off [fixed]
> tx-ipxip4-segmentation: off [fixed]
> tx-ipxip6-segmentation: off [fixed]
> tx-udp_tnl-segmentation: off [fixed]
> tx-udp_tnl-csum-segmentation: off [fixed]
> tx-gso-partial: off [fixed]
> tx-tunnel-remcsum-segmentation: off [fixed]
> tx-sctp-segmentation: off [fixed]
> tx-esp-segmentation: off [fixed]
> tx-udp-segmentation: off [fixed]
> tx-gso-list: off [fixed]
> fcoe-mtu: off [fixed]
> tx-nocache-copy: off
> loopback: off [fixed]
> rx-fcs: off [fixed]
> rx-all: off [fixed]
> tx-vlan-stag-hw-insert: off [fixed]
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: off [fixed]
> l2-fwd-offload: off [fixed]
> hw-tc-offload: off [fixed]
> esp-hw-offload: off [fixed]
> esp-tx-csum-hw-offload: off [fixed]
> rx-udp_tunnel-port-offload: off [fixed]
> tls-hw-tx-offload: off [fixed]
> tls-hw-rx-offload: off [fixed]
> rx-gro-hw: on
> tls-hw-record: off [fixed]
> rx-gro-list: off
> macsec-hw-offload: off [fixed]
> rx-udp-gro-forwarding: off
> hsr-tag-ins-offload: off [fixed]
> hsr-tag-rm-offload: off [fixed]
> hsr-fwd-offload: off [fixed]
> hsr-dup-offload: off [fixed]
>
> # ethtool -i eth0
> driver: virtio_net
> version: 1.0.0
> firmware-version:
> expansion-rom-version:
> bus-info: 0000:03:00.0
> supports-statistics: yes
> supports-test: no
> supports-eeprom-access: no
> supports-register-dump: no
> supports-priv-flags: no

Random guess, can you try:

diff --git a/net/core/gro.c b/net/core/gro.c
index fd8c6a7e8d3e2e6b439109d0089f44a547c7347e..f162674e7ae1bdf96bcbf7e1ed7326729d862f9a
100644
--- a/net/core/gro.c
+++ b/net/core/gro.c
@@ -500,7 +500,8 @@ static enum gro_result dev_gro_receive(struct
napi_struct *napi, struct sk_buff
        BUILD_BUG_ON(!IS_ALIGNED(offsetof(struct napi_gro_cb, zeroed),
                                        sizeof(u32))); /* Avoid slow
unaligned acc */
        *(u32 *)&NAPI_GRO_CB(skb)->zeroed = 0;
-       NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb);
+       NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb) ||
+                                 (skb_shinfo(skb)->gso_type & SKB_GSO_DODGY);
        NAPI_GRO_CB(skb)->is_atomic = 1;
        NAPI_GRO_CB(skb)->count = 1;
        if (unlikely(skb_is_gso(skb))) {



>
> čt 5. 1. 2023 v 18:43 odesílatel Eric Dumazet <edumazet@google.com> napsal:
> >
> > On Thu, Jan 5, 2023 at 6:37 PM Jaroslav Pulchart
> > <jaroslav.pulchart@gooddata.com> wrote:
> > >
> > > Hello,
> > >
> > > I would like to report a 6.1,y regression in a network performance
> > > observed when using "git clone".
> > >
> > > BAD: "git clone" speed with kernel 6.1,y:
> > >    # git clone git@github.com:..../.....git
> > >    ...
> > >    Receiving objects:   8% (47797/571306), 20.69 MiB | 3.27 MiB/s
> > >
> > > GOOD: "git clone" speed with kernel 6.0,y:
> > >    # git clone git@github.com:..../.....git
> > >    ...
> > >    Receiving objects:  72% (411341/571306), 181.05 MiB | 60.27 MiB/s
> > >
> > > I bisected the issue to a commit
> > > 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro
> > > packets to gro stack". Reverting it from 6.1.y branch makes the git
> > > clone fast like with 6.0.y.
> > >
> >
> > Hmm, please provide more information.
> >
> > NIC used ? (ethtool -i eth0)
> >
> > ethtool -k eth0  # replace by your netdev name
> >
> > And packet captures would be nice (with and without the patch)
> >
> > Thanks.
>
>
>
> --
> Jaroslav Pulchart
> Sr. Principal SW Engineer
> GoodData