Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack)

Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack)

[Jaroslav Pulchart]

Hello,

I would like to report a 6.1,y regression in a network performance
observed when using "git clone".

BAD: "git clone" speed with kernel 6.1,y:
   # git clone git@github.com:..../.....git
   ...
   Receiving objects:   8% (47797/571306), 20.69 MiB | 3.27 MiB/s

GOOD: "git clone" speed with kernel 6.0,y:
   # git clone git@github.com:..../.....git
   ...
   Receiving objects:  72% (411341/571306), 181.05 MiB | 60.27 MiB/s

I bisected the issue to a commit
5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro
packets to gro stack". Reverting it from 6.1.y branch makes the git
clone fast like with 6.0.y.

Best regards,
--
Jaroslav Pulchart
Sr. Principal SW Engineer
GoodData

Re: Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack)

[Eric Dumazet]

On Thu, Jan 5, 2023 at 6:37 PM Jaroslav Pulchart
<jaroslav.pulchart@gooddata.com> wrote:
>
> Hello,
>
> I would like to report a 6.1,y regression in a network performance
> observed when using "git clone".
>
> BAD: "git clone" speed with kernel 6.1,y:
>    # git clone git@github.com:..../.....git
>    ...
>    Receiving objects:   8% (47797/571306), 20.69 MiB | 3.27 MiB/s
>
> GOOD: "git clone" speed with kernel 6.0,y:
>    # git clone git@github.com:..../.....git
>    ...
>    Receiving objects:  72% (411341/571306), 181.05 MiB | 60.27 MiB/s
>
> I bisected the issue to a commit
> 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro
> packets to gro stack". Reverting it from 6.1.y branch makes the git
> clone fast like with 6.0.y.
>

Hmm, please provide more information.

NIC used ? (ethtool -i eth0)

ethtool -k eth0  # replace by your netdev name

And packet captures would be nice (with and without the patch)

Thanks.

Re: Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack)

[Jaroslav Pulchart]

It is at KVM based VMs "CentOS 9 Stream" and "CentOS 8 Stream" using
upstream kernel 6.1.y. Hosted on Dell PowerEdge 7525 servers (2x AMD
74F3) and OS CentOS 9 Stream again with upstream kernel 6.1.y or
6.0.y.

# ethtool -k eth0
Features for eth0:
rx-checksumming: on [fixed]
tx-checksumming: on
tx-checksum-ipv4: off [fixed]
tx-checksum-ip-generic: on
tx-checksum-ipv6: off [fixed]
tx-checksum-fcoe-crc: off [fixed]
tx-checksum-sctp: off [fixed]
scatter-gather: on
tx-scatter-gather: on
tx-scatter-gather-fraglist: off [fixed]
tcp-segmentation-offload: on
tx-tcp-segmentation: on
tx-tcp-ecn-segmentation: on
tx-tcp-mangleid-segmentation: off
tx-tcp6-segmentation: on
generic-segmentation-offload: on
generic-receive-offload: on
large-receive-offload: off [fixed]
rx-vlan-offload: off [fixed]
tx-vlan-offload: off [fixed]
ntuple-filters: off [fixed]
receive-hashing: off [fixed]
highdma: on [fixed]
rx-vlan-filter: on [fixed]
vlan-challenged: off [fixed]
tx-lockless: off [fixed]
netns-local: off [fixed]
tx-gso-robust: on [fixed]
tx-fcoe-segmentation: off [fixed]
tx-gre-segmentation: off [fixed]
tx-gre-csum-segmentation: off [fixed]
tx-ipxip4-segmentation: off [fixed]
tx-ipxip6-segmentation: off [fixed]
tx-udp_tnl-segmentation: off [fixed]
tx-udp_tnl-csum-segmentation: off [fixed]
tx-gso-partial: off [fixed]
tx-tunnel-remcsum-segmentation: off [fixed]
tx-sctp-segmentation: off [fixed]
tx-esp-segmentation: off [fixed]
tx-udp-segmentation: off [fixed]
tx-gso-list: off [fixed]
fcoe-mtu: off [fixed]
tx-nocache-copy: off
loopback: off [fixed]
rx-fcs: off [fixed]
rx-all: off [fixed]
tx-vlan-stag-hw-insert: off [fixed]
rx-vlan-stag-hw-parse: off [fixed]
rx-vlan-stag-filter: off [fixed]
l2-fwd-offload: off [fixed]
hw-tc-offload: off [fixed]
esp-hw-offload: off [fixed]
esp-tx-csum-hw-offload: off [fixed]
rx-udp_tunnel-port-offload: off [fixed]
tls-hw-tx-offload: off [fixed]
tls-hw-rx-offload: off [fixed]
rx-gro-hw: on
tls-hw-record: off [fixed]
rx-gro-list: off
macsec-hw-offload: off [fixed]
rx-udp-gro-forwarding: off
hsr-tag-ins-offload: off [fixed]
hsr-tag-rm-offload: off [fixed]
hsr-fwd-offload: off [fixed]
hsr-dup-offload: off [fixed]

# ethtool -i eth0
driver: virtio_net
version: 1.0.0
firmware-version:
expansion-rom-version:
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

čt 5. 1. 2023 v 18:43 odesílatel Eric Dumazet <edumazet@google.com> napsal:
>
> On Thu, Jan 5, 2023 at 6:37 PM Jaroslav Pulchart
> <jaroslav.pulchart@gooddata.com> wrote:
> >
> > Hello,
> >
> > I would like to report a 6.1,y regression in a network performance
> > observed when using "git clone".
> >
> > BAD: "git clone" speed with kernel 6.1,y:
> >    # git clone git@github.com:..../.....git
> >    ...
> >    Receiving objects:   8% (47797/571306), 20.69 MiB | 3.27 MiB/s
> >
> > GOOD: "git clone" speed with kernel 6.0,y:
> >    # git clone git@github.com:..../.....git
> >    ...
> >    Receiving objects:  72% (411341/571306), 181.05 MiB | 60.27 MiB/s
> >
> > I bisected the issue to a commit
> > 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro
> > packets to gro stack". Reverting it from 6.1.y branch makes the git
> > clone fast like with 6.0.y.
> >
>
> Hmm, please provide more information.
>
> NIC used ? (ethtool -i eth0)
>
> ethtool -k eth0  # replace by your netdev name
>
> And packet captures would be nice (with and without the patch)
>
> Thanks.



-- 
Jaroslav Pulchart
Sr. Principal SW Engineer
GoodData

Re: Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack)

[Eric Dumazet]

On Thu, Jan 5, 2023 at 6:54 PM Jaroslav Pulchart
<jaroslav.pulchart@gooddata.com> wrote:
>
> It is at KVM based VMs "CentOS 9 Stream" and "CentOS 8 Stream" using
> upstream kernel 6.1.y. Hosted on Dell PowerEdge 7525 servers (2x AMD
> 74F3) and OS CentOS 9 Stream again with upstream kernel 6.1.y or
> 6.0.y.
>
> # ethtool -k eth0
> Features for eth0:
> rx-checksumming: on [fixed]
> tx-checksumming: on
> tx-checksum-ipv4: off [fixed]
> tx-checksum-ip-generic: on
> tx-checksum-ipv6: off [fixed]
> tx-checksum-fcoe-crc: off [fixed]
> tx-checksum-sctp: off [fixed]
> scatter-gather: on
> tx-scatter-gather: on
> tx-scatter-gather-fraglist: off [fixed]
> tcp-segmentation-offload: on
> tx-tcp-segmentation: on
> tx-tcp-ecn-segmentation: on
> tx-tcp-mangleid-segmentation: off
> tx-tcp6-segmentation: on
> generic-segmentation-offload: on
> generic-receive-offload: on
> large-receive-offload: off [fixed]
> rx-vlan-offload: off [fixed]
> tx-vlan-offload: off [fixed]
> ntuple-filters: off [fixed]
> receive-hashing: off [fixed]
> highdma: on [fixed]
> rx-vlan-filter: on [fixed]
> vlan-challenged: off [fixed]
> tx-lockless: off [fixed]
> netns-local: off [fixed]
> tx-gso-robust: on [fixed]
> tx-fcoe-segmentation: off [fixed]
> tx-gre-segmentation: off [fixed]
> tx-gre-csum-segmentation: off [fixed]
> tx-ipxip4-segmentation: off [fixed]
> tx-ipxip6-segmentation: off [fixed]
> tx-udp_tnl-segmentation: off [fixed]
> tx-udp_tnl-csum-segmentation: off [fixed]
> tx-gso-partial: off [fixed]
> tx-tunnel-remcsum-segmentation: off [fixed]
> tx-sctp-segmentation: off [fixed]
> tx-esp-segmentation: off [fixed]
> tx-udp-segmentation: off [fixed]
> tx-gso-list: off [fixed]
> fcoe-mtu: off [fixed]
> tx-nocache-copy: off
> loopback: off [fixed]
> rx-fcs: off [fixed]
> rx-all: off [fixed]
> tx-vlan-stag-hw-insert: off [fixed]
> rx-vlan-stag-hw-parse: off [fixed]
> rx-vlan-stag-filter: off [fixed]
> l2-fwd-offload: off [fixed]
> hw-tc-offload: off [fixed]
> esp-hw-offload: off [fixed]
> esp-tx-csum-hw-offload: off [fixed]
> rx-udp_tunnel-port-offload: off [fixed]
> tls-hw-tx-offload: off [fixed]
> tls-hw-rx-offload: off [fixed]
> rx-gro-hw: on
> tls-hw-record: off [fixed]
> rx-gro-list: off
> macsec-hw-offload: off [fixed]
> rx-udp-gro-forwarding: off
> hsr-tag-ins-offload: off [fixed]
> hsr-tag-rm-offload: off [fixed]
> hsr-fwd-offload: off [fixed]
> hsr-dup-offload: off [fixed]
>
> # ethtool -i eth0
> driver: virtio_net
> version: 1.0.0
> firmware-version:
> expansion-rom-version:
> bus-info: 0000:03:00.0
> supports-statistics: yes
> supports-test: no
> supports-eeprom-access: no
> supports-register-dump: no
> supports-priv-flags: no

Random guess, can you try:

diff --git a/net/core/gro.c b/net/core/gro.c
index fd8c6a7e8d3e2e6b439109d0089f44a547c7347e..f162674e7ae1bdf96bcbf7e1ed7326729d862f9a
100644
--- a/net/core/gro.c
+++ b/net/core/gro.c
@@ -500,7 +500,8 @@ static enum gro_result dev_gro_receive(struct
napi_struct *napi, struct sk_buff
        BUILD_BUG_ON(!IS_ALIGNED(offsetof(struct napi_gro_cb, zeroed),
                                        sizeof(u32))); /* Avoid slow
unaligned acc */
        *(u32 *)&NAPI_GRO_CB(skb)->zeroed = 0;
-       NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb);
+       NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb) ||
+                                 (skb_shinfo(skb)->gso_type & SKB_GSO_DODGY);
        NAPI_GRO_CB(skb)->is_atomic = 1;
        NAPI_GRO_CB(skb)->count = 1;
        if (unlikely(skb_is_gso(skb))) {



>
> čt 5. 1. 2023 v 18:43 odesílatel Eric Dumazet <edumazet@google.com> napsal:
> >
> > On Thu, Jan 5, 2023 at 6:37 PM Jaroslav Pulchart
> > <jaroslav.pulchart@gooddata.com> wrote:
> > >
> > > Hello,
> > >
> > > I would like to report a 6.1,y regression in a network performance
> > > observed when using "git clone".
> > >
> > > BAD: "git clone" speed with kernel 6.1,y:
> > >    # git clone git@github.com:..../.....git
> > >    ...
> > >    Receiving objects:   8% (47797/571306), 20.69 MiB | 3.27 MiB/s
> > >
> > > GOOD: "git clone" speed with kernel 6.0,y:
> > >    # git clone git@github.com:..../.....git
> > >    ...
> > >    Receiving objects:  72% (411341/571306), 181.05 MiB | 60.27 MiB/s
> > >
> > > I bisected the issue to a commit
> > > 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro
> > > packets to gro stack". Reverting it from 6.1.y branch makes the git
> > > clone fast like with 6.0.y.
> > >
> >
> > Hmm, please provide more information.
> >
> > NIC used ? (ethtool -i eth0)
> >
> > ethtool -k eth0  # replace by your netdev name
> >
> > And packet captures would be nice (with and without the patch)
> >
> > Thanks.
>
>
>
> --
> Jaroslav Pulchart
> Sr. Principal SW Engineer
> GoodData

Re: Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack)

[Jaroslav Pulchart]

I can and I did. Your "Random guess" patch fix the problem and speed
is like with 6.0.y:

  # git clone ...
  ...
  Receiving objects: 100% (571306/571306), 350.58 MiB | 67.51 MiB/s, done.

Thanks,
JP

čt 5. 1. 2023 v 19:49 odesílatel Eric Dumazet <edumazet@google.com> napsal:
>
> On Thu, Jan 5, 2023 at 6:54 PM Jaroslav Pulchart
> <jaroslav.pulchart@gooddata.com> wrote:
> >
> > It is at KVM based VMs "CentOS 9 Stream" and "CentOS 8 Stream" using
> > upstream kernel 6.1.y. Hosted on Dell PowerEdge 7525 servers (2x AMD
> > 74F3) and OS CentOS 9 Stream again with upstream kernel 6.1.y or
> > 6.0.y.
> >
> > # ethtool -k eth0
> > Features for eth0:
> > rx-checksumming: on [fixed]
> > tx-checksumming: on
> > tx-checksum-ipv4: off [fixed]
> > tx-checksum-ip-generic: on
> > tx-checksum-ipv6: off [fixed]
> > tx-checksum-fcoe-crc: off [fixed]
> > tx-checksum-sctp: off [fixed]
> > scatter-gather: on
> > tx-scatter-gather: on
> > tx-scatter-gather-fraglist: off [fixed]
> > tcp-segmentation-offload: on
> > tx-tcp-segmentation: on
> > tx-tcp-ecn-segmentation: on
> > tx-tcp-mangleid-segmentation: off
> > tx-tcp6-segmentation: on
> > generic-segmentation-offload: on
> > generic-receive-offload: on
> > large-receive-offload: off [fixed]
> > rx-vlan-offload: off [fixed]
> > tx-vlan-offload: off [fixed]
> > ntuple-filters: off [fixed]
> > receive-hashing: off [fixed]
> > highdma: on [fixed]
> > rx-vlan-filter: on [fixed]
> > vlan-challenged: off [fixed]
> > tx-lockless: off [fixed]
> > netns-local: off [fixed]
> > tx-gso-robust: on [fixed]
> > tx-fcoe-segmentation: off [fixed]
> > tx-gre-segmentation: off [fixed]
> > tx-gre-csum-segmentation: off [fixed]
> > tx-ipxip4-segmentation: off [fixed]
> > tx-ipxip6-segmentation: off [fixed]
> > tx-udp_tnl-segmentation: off [fixed]
> > tx-udp_tnl-csum-segmentation: off [fixed]
> > tx-gso-partial: off [fixed]
> > tx-tunnel-remcsum-segmentation: off [fixed]
> > tx-sctp-segmentation: off [fixed]
> > tx-esp-segmentation: off [fixed]
> > tx-udp-segmentation: off [fixed]
> > tx-gso-list: off [fixed]
> > fcoe-mtu: off [fixed]
> > tx-nocache-copy: off
> > loopback: off [fixed]
> > rx-fcs: off [fixed]
> > rx-all: off [fixed]
> > tx-vlan-stag-hw-insert: off [fixed]
> > rx-vlan-stag-hw-parse: off [fixed]
> > rx-vlan-stag-filter: off [fixed]
> > l2-fwd-offload: off [fixed]
> > hw-tc-offload: off [fixed]
> > esp-hw-offload: off [fixed]
> > esp-tx-csum-hw-offload: off [fixed]
> > rx-udp_tunnel-port-offload: off [fixed]
> > tls-hw-tx-offload: off [fixed]
> > tls-hw-rx-offload: off [fixed]
> > rx-gro-hw: on
> > tls-hw-record: off [fixed]
> > rx-gro-list: off
> > macsec-hw-offload: off [fixed]
> > rx-udp-gro-forwarding: off
> > hsr-tag-ins-offload: off [fixed]
> > hsr-tag-rm-offload: off [fixed]
> > hsr-fwd-offload: off [fixed]
> > hsr-dup-offload: off [fixed]
> >
> > # ethtool -i eth0
> > driver: virtio_net
> > version: 1.0.0
> > firmware-version:
> > expansion-rom-version:
> > bus-info: 0000:03:00.0
> > supports-statistics: yes
> > supports-test: no
> > supports-eeprom-access: no
> > supports-register-dump: no
> > supports-priv-flags: no
>
> Random guess, can you try:
>
> diff --git a/net/core/gro.c b/net/core/gro.c
> index fd8c6a7e8d3e2e6b439109d0089f44a547c7347e..f162674e7ae1bdf96bcbf7e1ed7326729d862f9a
> 100644
> --- a/net/core/gro.c
> +++ b/net/core/gro.c
> @@ -500,7 +500,8 @@ static enum gro_result dev_gro_receive(struct
> napi_struct *napi, struct sk_buff
>         BUILD_BUG_ON(!IS_ALIGNED(offsetof(struct napi_gro_cb, zeroed),
>                                         sizeof(u32))); /* Avoid slow
> unaligned acc */
>         *(u32 *)&NAPI_GRO_CB(skb)->zeroed = 0;
> -       NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb);
> +       NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb) ||
> +                                 (skb_shinfo(skb)->gso_type & SKB_GSO_DODGY);
>         NAPI_GRO_CB(skb)->is_atomic = 1;
>         NAPI_GRO_CB(skb)->count = 1;
>         if (unlikely(skb_is_gso(skb))) {
>
>
>
> >
> > čt 5. 1. 2023 v 18:43 odesílatel Eric Dumazet <edumazet@google.com> napsal:
> > >
> > > On Thu, Jan 5, 2023 at 6:37 PM Jaroslav Pulchart
> > > <jaroslav.pulchart@gooddata.com> wrote:
> > > >
> > > > Hello,
> > > >
> > > > I would like to report a 6.1,y regression in a network performance
> > > > observed when using "git clone".
> > > >
> > > > BAD: "git clone" speed with kernel 6.1,y:
> > > >    # git clone git@github.com:..../.....git
> > > >    ...
> > > >    Receiving objects:   8% (47797/571306), 20.69 MiB | 3.27 MiB/s
> > > >
> > > > GOOD: "git clone" speed with kernel 6.0,y:
> > > >    # git clone git@github.com:..../.....git
> > > >    ...
> > > >    Receiving objects:  72% (411341/571306), 181.05 MiB | 60.27 MiB/s
> > > >
> > > > I bisected the issue to a commit
> > > > 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro
> > > > packets to gro stack". Reverting it from 6.1.y branch makes the git
> > > > clone fast like with 6.0.y.
> > > >
> > >
> > > Hmm, please provide more information.
> > >
> > > NIC used ? (ethtool -i eth0)
> > >
> > > ethtool -k eth0  # replace by your netdev name
> > >
> > > And packet captures would be nice (with and without the patch)
> > >
> > > Thanks.
> >
> >
> >
> > --
> > Jaroslav Pulchart
> > Sr. Principal SW Engineer
> > GoodData



-- 
Jaroslav Pulchart
Sr. Principal SW Engineer
GoodData

Re: Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack)

[Linux kernel regression tracking (#adding)]

[TLDR: I'm adding this report to the list of tracked Linux kernel
regressions; all text you find below is based on a few templates
paragraphs you might have encountered already already in similar form.
See link in footer if these mails annoy you.]

[CCing the regression list, as it should be in the loop for regressions:
https://docs.kernel.org/admin-guide/reporting-regressions.html]

On 05.01.23 18:37, Jaroslav Pulchart wrote:
> Hello,
> 
> I would like to report a 6.1,y regression in a network performance
> observed when using "git clone".
> 
> BAD: "git clone" speed with kernel 6.1,y:
>    # git clone git@github.com:..../.....git
>    ...
>    Receiving objects:   8% (47797/571306), 20.69 MiB | 3.27 MiB/s
> 
> GOOD: "git clone" speed with kernel 6.0,y:
>    # git clone git@github.com:..../.....git
>    ...
>    Receiving objects:  72% (411341/571306), 181.05 MiB | 60.27 MiB/s
> 
> I bisected the issue to a commit
> 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro
> packets to gro stack". Reverting it from 6.1.y branch makes the git
> clone fast like with 6.0.y.

Thanks for the report. To be sure the issue doesn't fall through the
cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression
tracking bot:

#regzbot ^introduced 5eddb24901ee49eee23c0bfce6af2e83fd5679bd
#regzbot title net: gro: Network performance regressed
#regzbot ignore-activity

This isn't a regression? This issue or a fix for it are already
discussed somewhere else? It was fixed already? You want to clarify when
the regression started to happen? Or point out I got the title or
something else totally wrong? Then just reply and tell me -- ideally
while also telling regzbot about it, as explained by the page listed in
the footer of this mail.

Reminder for developers: When fixing the issue, add 'Link:' tags
pointing to the report (see page linked in footer for details).

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

Re: Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack)

[Eric Dumazet]

On Thu, Jan 5, 2023 at 8:56 PM Jaroslav Pulchart
<jaroslav.pulchart@gooddata.com> wrote:
>
> I can and I did. Your "Random guess" patch fix the problem and speed
> is like with 6.0.y:
>
>   # git clone ...
>   ...
>   Receiving objects: 100% (571306/571306), 350.58 MiB | 67.51 MiB/s, done.
>
> Thanks,

Excellent :)

I will move the new test out of the fast path though, in a section we
are sure gso_size is != 0.

Something like this.

diff --git a/net/core/gro.c b/net/core/gro.c
index fd8c6a7e8d3e2e6b439109d0089f44a547c7347e..506f83d715f873c9bc3727e28ace71e00fa79d2f
100644
--- a/net/core/gro.c
+++ b/net/core/gro.c
@@ -505,8 +505,9 @@ static enum gro_result dev_gro_receive(struct
napi_struct *napi, struct sk_buff
        NAPI_GRO_CB(skb)->count = 1;
        if (unlikely(skb_is_gso(skb))) {
                NAPI_GRO_CB(skb)->count = skb_shinfo(skb)->gso_segs;
-               /* Only support TCP at the moment. */
-               if (!skb_is_gso_tcp(skb))
+               /* Only support TCP and non DODGY users. */
+               if (!skb_is_gso_tcp(skb) ||
+                   (skb_shinfo(skb)->gso_type & SKB_GSO_DODGY))
                        NAPI_GRO_CB(skb)->flush = 1;
        }



> JP
>
> čt 5. 1. 2023 v 19:49 odesílatel Eric Dumazet <edumazet@google.com> napsal:
> >
> > On Thu, Jan 5, 2023 at 6:54 PM Jaroslav Pulchart
> > <jaroslav.pulchart@gooddata.com> wrote:
> > >
> > > It is at KVM based VMs "CentOS 9 Stream" and "CentOS 8 Stream" using
> > > upstream kernel 6.1.y. Hosted on Dell PowerEdge 7525 servers (2x AMD
> > > 74F3) and OS CentOS 9 Stream again with upstream kernel 6.1.y or
> > > 6.0.y.
> > >
> > > # ethtool -k eth0
> > > Features for eth0:
> > > rx-checksumming: on [fixed]
> > > tx-checksumming: on
> > > tx-checksum-ipv4: off [fixed]
> > > tx-checksum-ip-generic: on
> > > tx-checksum-ipv6: off [fixed]
> > > tx-checksum-fcoe-crc: off [fixed]
> > > tx-checksum-sctp: off [fixed]
> > > scatter-gather: on
> > > tx-scatter-gather: on
> > > tx-scatter-gather-fraglist: off [fixed]
> > > tcp-segmentation-offload: on
> > > tx-tcp-segmentation: on
> > > tx-tcp-ecn-segmentation: on
> > > tx-tcp-mangleid-segmentation: off
> > > tx-tcp6-segmentation: on
> > > generic-segmentation-offload: on
> > > generic-receive-offload: on
> > > large-receive-offload: off [fixed]
> > > rx-vlan-offload: off [fixed]
> > > tx-vlan-offload: off [fixed]
> > > ntuple-filters: off [fixed]
> > > receive-hashing: off [fixed]
> > > highdma: on [fixed]
> > > rx-vlan-filter: on [fixed]
> > > vlan-challenged: off [fixed]
> > > tx-lockless: off [fixed]
> > > netns-local: off [fixed]
> > > tx-gso-robust: on [fixed]
> > > tx-fcoe-segmentation: off [fixed]
> > > tx-gre-segmentation: off [fixed]
> > > tx-gre-csum-segmentation: off [fixed]
> > > tx-ipxip4-segmentation: off [fixed]
> > > tx-ipxip6-segmentation: off [fixed]
> > > tx-udp_tnl-segmentation: off [fixed]
> > > tx-udp_tnl-csum-segmentation: off [fixed]
> > > tx-gso-partial: off [fixed]
> > > tx-tunnel-remcsum-segmentation: off [fixed]
> > > tx-sctp-segmentation: off [fixed]
> > > tx-esp-segmentation: off [fixed]
> > > tx-udp-segmentation: off [fixed]
> > > tx-gso-list: off [fixed]
> > > fcoe-mtu: off [fixed]
> > > tx-nocache-copy: off
> > > loopback: off [fixed]
> > > rx-fcs: off [fixed]
> > > rx-all: off [fixed]
> > > tx-vlan-stag-hw-insert: off [fixed]
> > > rx-vlan-stag-hw-parse: off [fixed]
> > > rx-vlan-stag-filter: off [fixed]
> > > l2-fwd-offload: off [fixed]
> > > hw-tc-offload: off [fixed]
> > > esp-hw-offload: off [fixed]
> > > esp-tx-csum-hw-offload: off [fixed]
> > > rx-udp_tunnel-port-offload: off [fixed]
> > > tls-hw-tx-offload: off [fixed]
> > > tls-hw-rx-offload: off [fixed]
> > > rx-gro-hw: on
> > > tls-hw-record: off [fixed]
> > > rx-gro-list: off
> > > macsec-hw-offload: off [fixed]
> > > rx-udp-gro-forwarding: off
> > > hsr-tag-ins-offload: off [fixed]
> > > hsr-tag-rm-offload: off [fixed]
> > > hsr-fwd-offload: off [fixed]
> > > hsr-dup-offload: off [fixed]
> > >
> > > # ethtool -i eth0
> > > driver: virtio_net
> > > version: 1.0.0
> > > firmware-version:
> > > expansion-rom-version:
> > > bus-info: 0000:03:00.0
> > > supports-statistics: yes
> > > supports-test: no
> > > supports-eeprom-access: no
> > > supports-register-dump: no
> > > supports-priv-flags: no
> >
> > Random guess, can you try:
> >
> > diff --git a/net/core/gro.c b/net/core/gro.c
> > index fd8c6a7e8d3e2e6b439109d0089f44a547c7347e..f162674e7ae1bdf96bcbf7e1ed7326729d862f9a
> > 100644
> > --- a/net/core/gro.c
> > +++ b/net/core/gro.c
> > @@ -500,7 +500,8 @@ static enum gro_result dev_gro_receive(struct
> > napi_struct *napi, struct sk_buff
> >         BUILD_BUG_ON(!IS_ALIGNED(offsetof(struct napi_gro_cb, zeroed),
> >                                         sizeof(u32))); /* Avoid slow
> > unaligned acc */
> >         *(u32 *)&NAPI_GRO_CB(skb)->zeroed = 0;
> > -       NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb);
> > +       NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb) ||
> > +                                 (skb_shinfo(skb)->gso_type & SKB_GSO_DODGY);
> >         NAPI_GRO_CB(skb)->is_atomic = 1;
> >         NAPI_GRO_CB(skb)->count = 1;
> >         if (unlikely(skb_is_gso(skb))) {
> >
> >
> >
> > >
> > > čt 5. 1. 2023 v 18:43 odesílatel Eric Dumazet <edumazet@google.com> napsal:
> > > >
> > > > On Thu, Jan 5, 2023 at 6:37 PM Jaroslav Pulchart
> > > > <jaroslav.pulchart@gooddata.com> wrote:
> > > > >
> > > > > Hello,
> > > > >
> > > > > I would like to report a 6.1,y regression in a network performance
> > > > > observed when using "git clone".
> > > > >
> > > > > BAD: "git clone" speed with kernel 6.1,y:
> > > > >    # git clone git@github.com:..../.....git
> > > > >    ...
> > > > >    Receiving objects:   8% (47797/571306), 20.69 MiB | 3.27 MiB/s
> > > > >
> > > > > GOOD: "git clone" speed with kernel 6.0,y:
> > > > >    # git clone git@github.com:..../.....git
> > > > >    ...
> > > > >    Receiving objects:  72% (411341/571306), 181.05 MiB | 60.27 MiB/s
> > > > >
> > > > > I bisected the issue to a commit
> > > > > 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro
> > > > > packets to gro stack". Reverting it from 6.1.y branch makes the git
> > > > > clone fast like with 6.0.y.
> > > > >
> > > >
> > > > Hmm, please provide more information.
> > > >
> > > > NIC used ? (ethtool -i eth0)
> > > >
> > > > ethtool -k eth0  # replace by your netdev name
> > > >
> > > > And packet captures would be nice (with and without the patch)
> > > >
> > > > Thanks.
> > >
> > >
> > >
> > > --
> > > Jaroslav Pulchart
> > > Sr. Principal SW Engineer
> > > GoodData
>
>
>
> --
> Jaroslav Pulchart
> Sr. Principal SW Engineer
> GoodData

Re: Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack)

[Jaroslav Pulchart]

OK, I re-tested it with a new version of patch and it still works.



pá 6. 1. 2023 v 15:04 odesílatel Eric Dumazet <edumazet@google.com> napsal:
>
> On Thu, Jan 5, 2023 at 8:56 PM Jaroslav Pulchart
> <jaroslav.pulchart@gooddata.com> wrote:
> >
> > I can and I did. Your "Random guess" patch fix the problem and speed
> > is like with 6.0.y:
> >
> >   # git clone ...
> >   ...
> >   Receiving objects: 100% (571306/571306), 350.58 MiB | 67.51 MiB/s, done.
> >
> > Thanks,
>
> Excellent :)
>
> I will move the new test out of the fast path though, in a section we
> are sure gso_size is != 0.
>
> Something like this.
>
> diff --git a/net/core/gro.c b/net/core/gro.c
> index fd8c6a7e8d3e2e6b439109d0089f44a547c7347e..506f83d715f873c9bc3727e28ace71e00fa79d2f
> 100644
> --- a/net/core/gro.c
> +++ b/net/core/gro.c
> @@ -505,8 +505,9 @@ static enum gro_result dev_gro_receive(struct
> napi_struct *napi, struct sk_buff
>         NAPI_GRO_CB(skb)->count = 1;
>         if (unlikely(skb_is_gso(skb))) {
>                 NAPI_GRO_CB(skb)->count = skb_shinfo(skb)->gso_segs;
> -               /* Only support TCP at the moment. */
> -               if (!skb_is_gso_tcp(skb))
> +               /* Only support TCP and non DODGY users. */
> +               if (!skb_is_gso_tcp(skb) ||
> +                   (skb_shinfo(skb)->gso_type & SKB_GSO_DODGY))
>                         NAPI_GRO_CB(skb)->flush = 1;
>         }
>
>
>
> > JP
> >
> > čt 5. 1. 2023 v 19:49 odesílatel Eric Dumazet <edumazet@google.com> napsal:
> > >
> > > On Thu, Jan 5, 2023 at 6:54 PM Jaroslav Pulchart
> > > <jaroslav.pulchart@gooddata.com> wrote:
> > > >
> > > > It is at KVM based VMs "CentOS 9 Stream" and "CentOS 8 Stream" using
> > > > upstream kernel 6.1.y. Hosted on Dell PowerEdge 7525 servers (2x AMD
> > > > 74F3) and OS CentOS 9 Stream again with upstream kernel 6.1.y or
> > > > 6.0.y.
> > > >
> > > > # ethtool -k eth0
> > > > Features for eth0:
> > > > rx-checksumming: on [fixed]
> > > > tx-checksumming: on
> > > > tx-checksum-ipv4: off [fixed]
> > > > tx-checksum-ip-generic: on
> > > > tx-checksum-ipv6: off [fixed]
> > > > tx-checksum-fcoe-crc: off [fixed]
> > > > tx-checksum-sctp: off [fixed]
> > > > scatter-gather: on
> > > > tx-scatter-gather: on
> > > > tx-scatter-gather-fraglist: off [fixed]
> > > > tcp-segmentation-offload: on
> > > > tx-tcp-segmentation: on
> > > > tx-tcp-ecn-segmentation: on
> > > > tx-tcp-mangleid-segmentation: off
> > > > tx-tcp6-segmentation: on
> > > > generic-segmentation-offload: on
> > > > generic-receive-offload: on
> > > > large-receive-offload: off [fixed]
> > > > rx-vlan-offload: off [fixed]
> > > > tx-vlan-offload: off [fixed]
> > > > ntuple-filters: off [fixed]
> > > > receive-hashing: off [fixed]
> > > > highdma: on [fixed]
> > > > rx-vlan-filter: on [fixed]
> > > > vlan-challenged: off [fixed]
> > > > tx-lockless: off [fixed]
> > > > netns-local: off [fixed]
> > > > tx-gso-robust: on [fixed]
> > > > tx-fcoe-segmentation: off [fixed]
> > > > tx-gre-segmentation: off [fixed]
> > > > tx-gre-csum-segmentation: off [fixed]
> > > > tx-ipxip4-segmentation: off [fixed]
> > > > tx-ipxip6-segmentation: off [fixed]
> > > > tx-udp_tnl-segmentation: off [fixed]
> > > > tx-udp_tnl-csum-segmentation: off [fixed]
> > > > tx-gso-partial: off [fixed]
> > > > tx-tunnel-remcsum-segmentation: off [fixed]
> > > > tx-sctp-segmentation: off [fixed]
> > > > tx-esp-segmentation: off [fixed]
> > > > tx-udp-segmentation: off [fixed]
> > > > tx-gso-list: off [fixed]
> > > > fcoe-mtu: off [fixed]
> > > > tx-nocache-copy: off
> > > > loopback: off [fixed]
> > > > rx-fcs: off [fixed]
> > > > rx-all: off [fixed]
> > > > tx-vlan-stag-hw-insert: off [fixed]
> > > > rx-vlan-stag-hw-parse: off [fixed]
> > > > rx-vlan-stag-filter: off [fixed]
> > > > l2-fwd-offload: off [fixed]
> > > > hw-tc-offload: off [fixed]
> > > > esp-hw-offload: off [fixed]
> > > > esp-tx-csum-hw-offload: off [fixed]
> > > > rx-udp_tunnel-port-offload: off [fixed]
> > > > tls-hw-tx-offload: off [fixed]
> > > > tls-hw-rx-offload: off [fixed]
> > > > rx-gro-hw: on
> > > > tls-hw-record: off [fixed]
> > > > rx-gro-list: off
> > > > macsec-hw-offload: off [fixed]
> > > > rx-udp-gro-forwarding: off
> > > > hsr-tag-ins-offload: off [fixed]
> > > > hsr-tag-rm-offload: off [fixed]
> > > > hsr-fwd-offload: off [fixed]
> > > > hsr-dup-offload: off [fixed]
> > > >
> > > > # ethtool -i eth0
> > > > driver: virtio_net
> > > > version: 1.0.0
> > > > firmware-version:
> > > > expansion-rom-version:
> > > > bus-info: 0000:03:00.0
> > > > supports-statistics: yes
> > > > supports-test: no
> > > > supports-eeprom-access: no
> > > > supports-register-dump: no
> > > > supports-priv-flags: no
> > >
> > > Random guess, can you try:
> > >
> > > diff --git a/net/core/gro.c b/net/core/gro.c
> > > index fd8c6a7e8d3e2e6b439109d0089f44a547c7347e..f162674e7ae1bdf96bcbf7e1ed7326729d862f9a
> > > 100644
> > > --- a/net/core/gro.c
> > > +++ b/net/core/gro.c
> > > @@ -500,7 +500,8 @@ static enum gro_result dev_gro_receive(struct
> > > napi_struct *napi, struct sk_buff
> > >         BUILD_BUG_ON(!IS_ALIGNED(offsetof(struct napi_gro_cb, zeroed),
> > >                                         sizeof(u32))); /* Avoid slow
> > > unaligned acc */
> > >         *(u32 *)&NAPI_GRO_CB(skb)->zeroed = 0;
> > > -       NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb);
> > > +       NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb) ||
> > > +                                 (skb_shinfo(skb)->gso_type & SKB_GSO_DODGY);
> > >         NAPI_GRO_CB(skb)->is_atomic = 1;
> > >         NAPI_GRO_CB(skb)->count = 1;
> > >         if (unlikely(skb_is_gso(skb))) {
> > >
> > >
> > >
> > > >
> > > > čt 5. 1. 2023 v 18:43 odesílatel Eric Dumazet <edumazet@google.com> napsal:
> > > > >
> > > > > On Thu, Jan 5, 2023 at 6:37 PM Jaroslav Pulchart
> > > > > <jaroslav.pulchart@gooddata.com> wrote:
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > I would like to report a 6.1,y regression in a network performance
> > > > > > observed when using "git clone".
> > > > > >
> > > > > > BAD: "git clone" speed with kernel 6.1,y:
> > > > > >    # git clone git@github.com:..../.....git
> > > > > >    ...
> > > > > >    Receiving objects:   8% (47797/571306), 20.69 MiB | 3.27 MiB/s
> > > > > >
> > > > > > GOOD: "git clone" speed with kernel 6.0,y:
> > > > > >    # git clone git@github.com:..../.....git
> > > > > >    ...
> > > > > >    Receiving objects:  72% (411341/571306), 181.05 MiB | 60.27 MiB/s
> > > > > >
> > > > > > I bisected the issue to a commit
> > > > > > 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro
> > > > > > packets to gro stack". Reverting it from 6.1.y branch makes the git
> > > > > > clone fast like with 6.0.y.
> > > > > >
> > > > >
> > > > > Hmm, please provide more information.
> > > > >
> > > > > NIC used ? (ethtool -i eth0)
> > > > >
> > > > > ethtool -k eth0  # replace by your netdev name
> > > > >
> > > > > And packet captures would be nice (with and without the patch)
> > > > >
> > > > > Thanks.
> > > >
> > > >
> > > >
> > > > --
> > > > Jaroslav Pulchart
> > > > Sr. Principal SW Engineer
> > > > GoodData
> >
> >
> >
> > --
> > Jaroslav Pulchart
> > Sr. Principal SW Engineer
> > GoodData



-- 
Jaroslav Pulchart
Sr. Principal SW Engineer
GoodData

Re: Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack)

[Linux kernel regression tracking (Thorsten Leemhuis)]

[TLDR: This mail in primarily relevant for Linux kernel regression
tracking. See link in footer if these mails annoy you.]

On 06.01.23 09:36, Linux kernel regression tracking (#adding) wrote:

> On 05.01.23 18:37, Jaroslav Pulchart wrote:
>> Hello,
>>
>> I would like to report a 6.1,y regression in a network performance
>> observed when using "git clone".
>>
>> BAD: "git clone" speed with kernel 6.1,y:
>>    # git clone git@github.com:..../.....git
>>    ...
>>    Receiving objects:   8% (47797/571306), 20.69 MiB | 3.27 MiB/s
>>
>> GOOD: "git clone" speed with kernel 6.0,y:
>>    # git clone git@github.com:..../.....git
>>    ...
>>    Receiving objects:  72% (411341/571306), 181.05 MiB | 60.27 MiB/s
>>
>> I bisected the issue to a commit
>> 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro
>> packets to gro stack". Reverting it from 6.1.y branch makes the git
>> clone fast like with 6.0.y.
> 
> Thanks for the report. To be sure the issue doesn't fall through the
> cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression
> tracking bot:
> 
> #regzbot ^introduced 5eddb24901ee49eee23c0bfce6af2e83fd5679bd
> #regzbot title net: gro: Network performance regressed
> #regzbot ignore-activity

#regzbot fix: 7871f54e3dee

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

#regzbot ignore-activity