Re: Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack)

[Eric Dumazet <edumazet@google.com>]

On Thu, Jan 5, 2023 at 8:56 PM Jaroslav Pulchart
<jaroslav.pulchart@gooddata.com> wrote:
>
> I can and I did. Your "Random guess" patch fix the problem and speed
> is like with 6.0.y:
>
>   # git clone ...
>   ...
>   Receiving objects: 100% (571306/571306), 350.58 MiB | 67.51 MiB/s, done.
>
> Thanks,

Excellent :)

I will move the new test out of the fast path though, in a section we
are sure gso_size is != 0.

Something like this.

diff --git a/net/core/gro.c b/net/core/gro.c
index fd8c6a7e8d3e2e6b439109d0089f44a547c7347e..506f83d715f873c9bc3727e28ace71e00fa79d2f
100644
--- a/net/core/gro.c
+++ b/net/core/gro.c
@@ -505,8 +505,9 @@ static enum gro_result dev_gro_receive(struct
napi_struct *napi, struct sk_buff
        NAPI_GRO_CB(skb)->count = 1;
        if (unlikely(skb_is_gso(skb))) {
                NAPI_GRO_CB(skb)->count = skb_shinfo(skb)->gso_segs;
-               /* Only support TCP at the moment. */
-               if (!skb_is_gso_tcp(skb))
+               /* Only support TCP and non DODGY users. */
+               if (!skb_is_gso_tcp(skb) ||
+                   (skb_shinfo(skb)->gso_type & SKB_GSO_DODGY))
                        NAPI_GRO_CB(skb)->flush = 1;
        }



> JP
>
> čt 5. 1. 2023 v 19:49 odesílatel Eric Dumazet <edumazet@google.com> napsal:
> >
> > On Thu, Jan 5, 2023 at 6:54 PM Jaroslav Pulchart
> > <jaroslav.pulchart@gooddata.com> wrote:
> > >
> > > It is at KVM based VMs "CentOS 9 Stream" and "CentOS 8 Stream" using
> > > upstream kernel 6.1.y. Hosted on Dell PowerEdge 7525 servers (2x AMD
> > > 74F3) and OS CentOS 9 Stream again with upstream kernel 6.1.y or
> > > 6.0.y.
> > >
> > > # ethtool -k eth0
> > > Features for eth0:
> > > rx-checksumming: on [fixed]
> > > tx-checksumming: on
> > > tx-checksum-ipv4: off [fixed]
> > > tx-checksum-ip-generic: on
> > > tx-checksum-ipv6: off [fixed]
> > > tx-checksum-fcoe-crc: off [fixed]
> > > tx-checksum-sctp: off [fixed]
> > > scatter-gather: on
> > > tx-scatter-gather: on
> > > tx-scatter-gather-fraglist: off [fixed]
> > > tcp-segmentation-offload: on
> > > tx-tcp-segmentation: on
> > > tx-tcp-ecn-segmentation: on
> > > tx-tcp-mangleid-segmentation: off
> > > tx-tcp6-segmentation: on
> > > generic-segmentation-offload: on
> > > generic-receive-offload: on
> > > large-receive-offload: off [fixed]
> > > rx-vlan-offload: off [fixed]
> > > tx-vlan-offload: off [fixed]
> > > ntuple-filters: off [fixed]
> > > receive-hashing: off [fixed]
> > > highdma: on [fixed]
> > > rx-vlan-filter: on [fixed]
> > > vlan-challenged: off [fixed]
> > > tx-lockless: off [fixed]
> > > netns-local: off [fixed]
> > > tx-gso-robust: on [fixed]
> > > tx-fcoe-segmentation: off [fixed]
> > > tx-gre-segmentation: off [fixed]
> > > tx-gre-csum-segmentation: off [fixed]
> > > tx-ipxip4-segmentation: off [fixed]
> > > tx-ipxip6-segmentation: off [fixed]
> > > tx-udp_tnl-segmentation: off [fixed]
> > > tx-udp_tnl-csum-segmentation: off [fixed]
> > > tx-gso-partial: off [fixed]
> > > tx-tunnel-remcsum-segmentation: off [fixed]
> > > tx-sctp-segmentation: off [fixed]
> > > tx-esp-segmentation: off [fixed]
> > > tx-udp-segmentation: off [fixed]
> > > tx-gso-list: off [fixed]
> > > fcoe-mtu: off [fixed]
> > > tx-nocache-copy: off
> > > loopback: off [fixed]
> > > rx-fcs: off [fixed]
> > > rx-all: off [fixed]
> > > tx-vlan-stag-hw-insert: off [fixed]
> > > rx-vlan-stag-hw-parse: off [fixed]
> > > rx-vlan-stag-filter: off [fixed]
> > > l2-fwd-offload: off [fixed]
> > > hw-tc-offload: off [fixed]
> > > esp-hw-offload: off [fixed]
> > > esp-tx-csum-hw-offload: off [fixed]
> > > rx-udp_tunnel-port-offload: off [fixed]
> > > tls-hw-tx-offload: off [fixed]
> > > tls-hw-rx-offload: off [fixed]
> > > rx-gro-hw: on
> > > tls-hw-record: off [fixed]
> > > rx-gro-list: off
> > > macsec-hw-offload: off [fixed]
> > > rx-udp-gro-forwarding: off
> > > hsr-tag-ins-offload: off [fixed]
> > > hsr-tag-rm-offload: off [fixed]
> > > hsr-fwd-offload: off [fixed]
> > > hsr-dup-offload: off [fixed]
> > >
> > > # ethtool -i eth0
> > > driver: virtio_net
> > > version: 1.0.0
> > > firmware-version:
> > > expansion-rom-version:
> > > bus-info: 0000:03:00.0
> > > supports-statistics: yes
> > > supports-test: no
> > > supports-eeprom-access: no
> > > supports-register-dump: no
> > > supports-priv-flags: no
> >
> > Random guess, can you try:
> >
> > diff --git a/net/core/gro.c b/net/core/gro.c
> > index fd8c6a7e8d3e2e6b439109d0089f44a547c7347e..f162674e7ae1bdf96bcbf7e1ed7326729d862f9a
> > 100644
> > --- a/net/core/gro.c
> > +++ b/net/core/gro.c
> > @@ -500,7 +500,8 @@ static enum gro_result dev_gro_receive(struct
> > napi_struct *napi, struct sk_buff
> >         BUILD_BUG_ON(!IS_ALIGNED(offsetof(struct napi_gro_cb, zeroed),
> >                                         sizeof(u32))); /* Avoid slow
> > unaligned acc */
> >         *(u32 *)&NAPI_GRO_CB(skb)->zeroed = 0;
> > -       NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb);
> > +       NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb) ||
> > +                                 (skb_shinfo(skb)->gso_type & SKB_GSO_DODGY);
> >         NAPI_GRO_CB(skb)->is_atomic = 1;
> >         NAPI_GRO_CB(skb)->count = 1;
> >         if (unlikely(skb_is_gso(skb))) {
> >
> >
> >
> > >
> > > čt 5. 1. 2023 v 18:43 odesílatel Eric Dumazet <edumazet@google.com> napsal:
> > > >
> > > > On Thu, Jan 5, 2023 at 6:37 PM Jaroslav Pulchart
> > > > <jaroslav.pulchart@gooddata.com> wrote:
> > > > >
> > > > > Hello,
> > > > >
> > > > > I would like to report a 6.1,y regression in a network performance
> > > > > observed when using "git clone".
> > > > >
> > > > > BAD: "git clone" speed with kernel 6.1,y:
> > > > >    # git clone git@github.com:..../.....git
> > > > >    ...
> > > > >    Receiving objects:   8% (47797/571306), 20.69 MiB | 3.27 MiB/s
> > > > >
> > > > > GOOD: "git clone" speed with kernel 6.0,y:
> > > > >    # git clone git@github.com:..../.....git
> > > > >    ...
> > > > >    Receiving objects:  72% (411341/571306), 181.05 MiB | 60.27 MiB/s
> > > > >
> > > > > I bisected the issue to a commit
> > > > > 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro
> > > > > packets to gro stack". Reverting it from 6.1.y branch makes the git
> > > > > clone fast like with 6.0.y.
> > > > >
> > > >
> > > > Hmm, please provide more information.
> > > >
> > > > NIC used ? (ethtool -i eth0)
> > > >
> > > > ethtool -k eth0  # replace by your netdev name
> > > >
> > > > And packet captures would be nice (with and without the patch)
> > > >
> > > > Thanks.
> > >
> > >
> > >
> > > --
> > > Jaroslav Pulchart
> > > Sr. Principal SW Engineer
> > > GoodData
>
>
>
> --
> Jaroslav Pulchart
> Sr. Principal SW Engineer
> GoodData