* Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack) @ 2023-01-05 17:37 Jaroslav Pulchart 2023-01-05 17:43 ` Eric Dumazet 2023-01-06 8:36 ` Linux kernel regression tracking (#adding) 0 siblings, 2 replies; 9+ messages in thread From: Jaroslav Pulchart @ 2023-01-05 17:37 UTC (permalink / raw) To: edumazet, netdev, lixiaoyan, pabeni, davem; +Cc: Igor Raits Hello, I would like to report a 6.1,y regression in a network performance observed when using "git clone". BAD: "git clone" speed with kernel 6.1,y: # git clone git@github.com:..../.....git ... Receiving objects: 8% (47797/571306), 20.69 MiB | 3.27 MiB/s GOOD: "git clone" speed with kernel 6.0,y: # git clone git@github.com:..../.....git ... Receiving objects: 72% (411341/571306), 181.05 MiB | 60.27 MiB/s I bisected the issue to a commit 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro packets to gro stack". Reverting it from 6.1.y branch makes the git clone fast like with 6.0.y. Best regards, -- Jaroslav Pulchart Sr. Principal SW Engineer GoodData ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack) 2023-01-05 17:37 Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack) Jaroslav Pulchart @ 2023-01-05 17:43 ` Eric Dumazet 2023-01-05 17:54 ` Jaroslav Pulchart 2023-01-06 8:36 ` Linux kernel regression tracking (#adding) 1 sibling, 1 reply; 9+ messages in thread From: Eric Dumazet @ 2023-01-05 17:43 UTC (permalink / raw) To: Jaroslav Pulchart; +Cc: netdev, lixiaoyan, pabeni, davem, Igor Raits On Thu, Jan 5, 2023 at 6:37 PM Jaroslav Pulchart <jaroslav.pulchart@gooddata.com> wrote: > > Hello, > > I would like to report a 6.1,y regression in a network performance > observed when using "git clone". > > BAD: "git clone" speed with kernel 6.1,y: > # git clone git@github.com:..../.....git > ... > Receiving objects: 8% (47797/571306), 20.69 MiB | 3.27 MiB/s > > GOOD: "git clone" speed with kernel 6.0,y: > # git clone git@github.com:..../.....git > ... > Receiving objects: 72% (411341/571306), 181.05 MiB | 60.27 MiB/s > > I bisected the issue to a commit > 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro > packets to gro stack". Reverting it from 6.1.y branch makes the git > clone fast like with 6.0.y. > Hmm, please provide more information. NIC used ? (ethtool -i eth0) ethtool -k eth0 # replace by your netdev name And packet captures would be nice (with and without the patch) Thanks. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack) 2023-01-05 17:43 ` Eric Dumazet @ 2023-01-05 17:54 ` Jaroslav Pulchart 2023-01-05 18:49 ` Eric Dumazet 0 siblings, 1 reply; 9+ messages in thread From: Jaroslav Pulchart @ 2023-01-05 17:54 UTC (permalink / raw) To: Eric Dumazet; +Cc: netdev, lixiaoyan, pabeni, davem, Igor Raits It is at KVM based VMs "CentOS 9 Stream" and "CentOS 8 Stream" using upstream kernel 6.1.y. Hosted on Dell PowerEdge 7525 servers (2x AMD 74F3) and OS CentOS 9 Stream again with upstream kernel 6.1.y or 6.0.y. # ethtool -k eth0 Features for eth0: rx-checksumming: on [fixed] tx-checksumming: on tx-checksum-ipv4: off [fixed] tx-checksum-ip-generic: on tx-checksum-ipv6: off [fixed] tx-checksum-fcoe-crc: off [fixed] tx-checksum-sctp: off [fixed] scatter-gather: on tx-scatter-gather: on tx-scatter-gather-fraglist: off [fixed] tcp-segmentation-offload: on tx-tcp-segmentation: on tx-tcp-ecn-segmentation: on tx-tcp-mangleid-segmentation: off tx-tcp6-segmentation: on generic-segmentation-offload: on generic-receive-offload: on large-receive-offload: off [fixed] rx-vlan-offload: off [fixed] tx-vlan-offload: off [fixed] ntuple-filters: off [fixed] receive-hashing: off [fixed] highdma: on [fixed] rx-vlan-filter: on [fixed] vlan-challenged: off [fixed] tx-lockless: off [fixed] netns-local: off [fixed] tx-gso-robust: on [fixed] tx-fcoe-segmentation: off [fixed] tx-gre-segmentation: off [fixed] tx-gre-csum-segmentation: off [fixed] tx-ipxip4-segmentation: off [fixed] tx-ipxip6-segmentation: off [fixed] tx-udp_tnl-segmentation: off [fixed] tx-udp_tnl-csum-segmentation: off [fixed] tx-gso-partial: off [fixed] tx-tunnel-remcsum-segmentation: off [fixed] tx-sctp-segmentation: off [fixed] tx-esp-segmentation: off [fixed] tx-udp-segmentation: off [fixed] tx-gso-list: off [fixed] fcoe-mtu: off [fixed] tx-nocache-copy: off loopback: off [fixed] rx-fcs: off [fixed] rx-all: off [fixed] tx-vlan-stag-hw-insert: off [fixed] rx-vlan-stag-hw-parse: off [fixed] rx-vlan-stag-filter: off [fixed] l2-fwd-offload: off [fixed] hw-tc-offload: off [fixed] esp-hw-offload: off [fixed] esp-tx-csum-hw-offload: off [fixed] rx-udp_tunnel-port-offload: off [fixed] tls-hw-tx-offload: off [fixed] tls-hw-rx-offload: off [fixed] rx-gro-hw: on tls-hw-record: off [fixed] rx-gro-list: off macsec-hw-offload: off [fixed] rx-udp-gro-forwarding: off hsr-tag-ins-offload: off [fixed] hsr-tag-rm-offload: off [fixed] hsr-fwd-offload: off [fixed] hsr-dup-offload: off [fixed] # ethtool -i eth0 driver: virtio_net version: 1.0.0 firmware-version: expansion-rom-version: bus-info: 0000:03:00.0 supports-statistics: yes supports-test: no supports-eeprom-access: no supports-register-dump: no supports-priv-flags: no čt 5. 1. 2023 v 18:43 odesílatel Eric Dumazet <edumazet@google.com> napsal: > > On Thu, Jan 5, 2023 at 6:37 PM Jaroslav Pulchart > <jaroslav.pulchart@gooddata.com> wrote: > > > > Hello, > > > > I would like to report a 6.1,y regression in a network performance > > observed when using "git clone". > > > > BAD: "git clone" speed with kernel 6.1,y: > > # git clone git@github.com:..../.....git > > ... > > Receiving objects: 8% (47797/571306), 20.69 MiB | 3.27 MiB/s > > > > GOOD: "git clone" speed with kernel 6.0,y: > > # git clone git@github.com:..../.....git > > ... > > Receiving objects: 72% (411341/571306), 181.05 MiB | 60.27 MiB/s > > > > I bisected the issue to a commit > > 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro > > packets to gro stack". Reverting it from 6.1.y branch makes the git > > clone fast like with 6.0.y. > > > > Hmm, please provide more information. > > NIC used ? (ethtool -i eth0) > > ethtool -k eth0 # replace by your netdev name > > And packet captures would be nice (with and without the patch) > > Thanks. -- Jaroslav Pulchart Sr. Principal SW Engineer GoodData ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack) 2023-01-05 17:54 ` Jaroslav Pulchart @ 2023-01-05 18:49 ` Eric Dumazet 2023-01-05 19:56 ` Jaroslav Pulchart 0 siblings, 1 reply; 9+ messages in thread From: Eric Dumazet @ 2023-01-05 18:49 UTC (permalink / raw) To: Jaroslav Pulchart; +Cc: netdev, lixiaoyan, pabeni, davem, Igor Raits On Thu, Jan 5, 2023 at 6:54 PM Jaroslav Pulchart <jaroslav.pulchart@gooddata.com> wrote: > > It is at KVM based VMs "CentOS 9 Stream" and "CentOS 8 Stream" using > upstream kernel 6.1.y. Hosted on Dell PowerEdge 7525 servers (2x AMD > 74F3) and OS CentOS 9 Stream again with upstream kernel 6.1.y or > 6.0.y. > > # ethtool -k eth0 > Features for eth0: > rx-checksumming: on [fixed] > tx-checksumming: on > tx-checksum-ipv4: off [fixed] > tx-checksum-ip-generic: on > tx-checksum-ipv6: off [fixed] > tx-checksum-fcoe-crc: off [fixed] > tx-checksum-sctp: off [fixed] > scatter-gather: on > tx-scatter-gather: on > tx-scatter-gather-fraglist: off [fixed] > tcp-segmentation-offload: on > tx-tcp-segmentation: on > tx-tcp-ecn-segmentation: on > tx-tcp-mangleid-segmentation: off > tx-tcp6-segmentation: on > generic-segmentation-offload: on > generic-receive-offload: on > large-receive-offload: off [fixed] > rx-vlan-offload: off [fixed] > tx-vlan-offload: off [fixed] > ntuple-filters: off [fixed] > receive-hashing: off [fixed] > highdma: on [fixed] > rx-vlan-filter: on [fixed] > vlan-challenged: off [fixed] > tx-lockless: off [fixed] > netns-local: off [fixed] > tx-gso-robust: on [fixed] > tx-fcoe-segmentation: off [fixed] > tx-gre-segmentation: off [fixed] > tx-gre-csum-segmentation: off [fixed] > tx-ipxip4-segmentation: off [fixed] > tx-ipxip6-segmentation: off [fixed] > tx-udp_tnl-segmentation: off [fixed] > tx-udp_tnl-csum-segmentation: off [fixed] > tx-gso-partial: off [fixed] > tx-tunnel-remcsum-segmentation: off [fixed] > tx-sctp-segmentation: off [fixed] > tx-esp-segmentation: off [fixed] > tx-udp-segmentation: off [fixed] > tx-gso-list: off [fixed] > fcoe-mtu: off [fixed] > tx-nocache-copy: off > loopback: off [fixed] > rx-fcs: off [fixed] > rx-all: off [fixed] > tx-vlan-stag-hw-insert: off [fixed] > rx-vlan-stag-hw-parse: off [fixed] > rx-vlan-stag-filter: off [fixed] > l2-fwd-offload: off [fixed] > hw-tc-offload: off [fixed] > esp-hw-offload: off [fixed] > esp-tx-csum-hw-offload: off [fixed] > rx-udp_tunnel-port-offload: off [fixed] > tls-hw-tx-offload: off [fixed] > tls-hw-rx-offload: off [fixed] > rx-gro-hw: on > tls-hw-record: off [fixed] > rx-gro-list: off > macsec-hw-offload: off [fixed] > rx-udp-gro-forwarding: off > hsr-tag-ins-offload: off [fixed] > hsr-tag-rm-offload: off [fixed] > hsr-fwd-offload: off [fixed] > hsr-dup-offload: off [fixed] > > # ethtool -i eth0 > driver: virtio_net > version: 1.0.0 > firmware-version: > expansion-rom-version: > bus-info: 0000:03:00.0 > supports-statistics: yes > supports-test: no > supports-eeprom-access: no > supports-register-dump: no > supports-priv-flags: no Random guess, can you try: diff --git a/net/core/gro.c b/net/core/gro.c index fd8c6a7e8d3e2e6b439109d0089f44a547c7347e..f162674e7ae1bdf96bcbf7e1ed7326729d862f9a 100644 --- a/net/core/gro.c +++ b/net/core/gro.c @@ -500,7 +500,8 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff BUILD_BUG_ON(!IS_ALIGNED(offsetof(struct napi_gro_cb, zeroed), sizeof(u32))); /* Avoid slow unaligned acc */ *(u32 *)&NAPI_GRO_CB(skb)->zeroed = 0; - NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb); + NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb) || + (skb_shinfo(skb)->gso_type & SKB_GSO_DODGY); NAPI_GRO_CB(skb)->is_atomic = 1; NAPI_GRO_CB(skb)->count = 1; if (unlikely(skb_is_gso(skb))) { > > čt 5. 1. 2023 v 18:43 odesílatel Eric Dumazet <edumazet@google.com> napsal: > > > > On Thu, Jan 5, 2023 at 6:37 PM Jaroslav Pulchart > > <jaroslav.pulchart@gooddata.com> wrote: > > > > > > Hello, > > > > > > I would like to report a 6.1,y regression in a network performance > > > observed when using "git clone". > > > > > > BAD: "git clone" speed with kernel 6.1,y: > > > # git clone git@github.com:..../.....git > > > ... > > > Receiving objects: 8% (47797/571306), 20.69 MiB | 3.27 MiB/s > > > > > > GOOD: "git clone" speed with kernel 6.0,y: > > > # git clone git@github.com:..../.....git > > > ... > > > Receiving objects: 72% (411341/571306), 181.05 MiB | 60.27 MiB/s > > > > > > I bisected the issue to a commit > > > 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro > > > packets to gro stack". Reverting it from 6.1.y branch makes the git > > > clone fast like with 6.0.y. > > > > > > > Hmm, please provide more information. > > > > NIC used ? (ethtool -i eth0) > > > > ethtool -k eth0 # replace by your netdev name > > > > And packet captures would be nice (with and without the patch) > > > > Thanks. > > > > -- > Jaroslav Pulchart > Sr. Principal SW Engineer > GoodData ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack) 2023-01-05 18:49 ` Eric Dumazet @ 2023-01-05 19:56 ` Jaroslav Pulchart 2023-01-06 14:04 ` Eric Dumazet 0 siblings, 1 reply; 9+ messages in thread From: Jaroslav Pulchart @ 2023-01-05 19:56 UTC (permalink / raw) To: Eric Dumazet; +Cc: netdev, lixiaoyan, pabeni, davem, Igor Raits I can and I did. Your "Random guess" patch fix the problem and speed is like with 6.0.y: # git clone ... ... Receiving objects: 100% (571306/571306), 350.58 MiB | 67.51 MiB/s, done. Thanks, JP čt 5. 1. 2023 v 19:49 odesílatel Eric Dumazet <edumazet@google.com> napsal: > > On Thu, Jan 5, 2023 at 6:54 PM Jaroslav Pulchart > <jaroslav.pulchart@gooddata.com> wrote: > > > > It is at KVM based VMs "CentOS 9 Stream" and "CentOS 8 Stream" using > > upstream kernel 6.1.y. Hosted on Dell PowerEdge 7525 servers (2x AMD > > 74F3) and OS CentOS 9 Stream again with upstream kernel 6.1.y or > > 6.0.y. > > > > # ethtool -k eth0 > > Features for eth0: > > rx-checksumming: on [fixed] > > tx-checksumming: on > > tx-checksum-ipv4: off [fixed] > > tx-checksum-ip-generic: on > > tx-checksum-ipv6: off [fixed] > > tx-checksum-fcoe-crc: off [fixed] > > tx-checksum-sctp: off [fixed] > > scatter-gather: on > > tx-scatter-gather: on > > tx-scatter-gather-fraglist: off [fixed] > > tcp-segmentation-offload: on > > tx-tcp-segmentation: on > > tx-tcp-ecn-segmentation: on > > tx-tcp-mangleid-segmentation: off > > tx-tcp6-segmentation: on > > generic-segmentation-offload: on > > generic-receive-offload: on > > large-receive-offload: off [fixed] > > rx-vlan-offload: off [fixed] > > tx-vlan-offload: off [fixed] > > ntuple-filters: off [fixed] > > receive-hashing: off [fixed] > > highdma: on [fixed] > > rx-vlan-filter: on [fixed] > > vlan-challenged: off [fixed] > > tx-lockless: off [fixed] > > netns-local: off [fixed] > > tx-gso-robust: on [fixed] > > tx-fcoe-segmentation: off [fixed] > > tx-gre-segmentation: off [fixed] > > tx-gre-csum-segmentation: off [fixed] > > tx-ipxip4-segmentation: off [fixed] > > tx-ipxip6-segmentation: off [fixed] > > tx-udp_tnl-segmentation: off [fixed] > > tx-udp_tnl-csum-segmentation: off [fixed] > > tx-gso-partial: off [fixed] > > tx-tunnel-remcsum-segmentation: off [fixed] > > tx-sctp-segmentation: off [fixed] > > tx-esp-segmentation: off [fixed] > > tx-udp-segmentation: off [fixed] > > tx-gso-list: off [fixed] > > fcoe-mtu: off [fixed] > > tx-nocache-copy: off > > loopback: off [fixed] > > rx-fcs: off [fixed] > > rx-all: off [fixed] > > tx-vlan-stag-hw-insert: off [fixed] > > rx-vlan-stag-hw-parse: off [fixed] > > rx-vlan-stag-filter: off [fixed] > > l2-fwd-offload: off [fixed] > > hw-tc-offload: off [fixed] > > esp-hw-offload: off [fixed] > > esp-tx-csum-hw-offload: off [fixed] > > rx-udp_tunnel-port-offload: off [fixed] > > tls-hw-tx-offload: off [fixed] > > tls-hw-rx-offload: off [fixed] > > rx-gro-hw: on > > tls-hw-record: off [fixed] > > rx-gro-list: off > > macsec-hw-offload: off [fixed] > > rx-udp-gro-forwarding: off > > hsr-tag-ins-offload: off [fixed] > > hsr-tag-rm-offload: off [fixed] > > hsr-fwd-offload: off [fixed] > > hsr-dup-offload: off [fixed] > > > > # ethtool -i eth0 > > driver: virtio_net > > version: 1.0.0 > > firmware-version: > > expansion-rom-version: > > bus-info: 0000:03:00.0 > > supports-statistics: yes > > supports-test: no > > supports-eeprom-access: no > > supports-register-dump: no > > supports-priv-flags: no > > Random guess, can you try: > > diff --git a/net/core/gro.c b/net/core/gro.c > index fd8c6a7e8d3e2e6b439109d0089f44a547c7347e..f162674e7ae1bdf96bcbf7e1ed7326729d862f9a > 100644 > --- a/net/core/gro.c > +++ b/net/core/gro.c > @@ -500,7 +500,8 @@ static enum gro_result dev_gro_receive(struct > napi_struct *napi, struct sk_buff > BUILD_BUG_ON(!IS_ALIGNED(offsetof(struct napi_gro_cb, zeroed), > sizeof(u32))); /* Avoid slow > unaligned acc */ > *(u32 *)&NAPI_GRO_CB(skb)->zeroed = 0; > - NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb); > + NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb) || > + (skb_shinfo(skb)->gso_type & SKB_GSO_DODGY); > NAPI_GRO_CB(skb)->is_atomic = 1; > NAPI_GRO_CB(skb)->count = 1; > if (unlikely(skb_is_gso(skb))) { > > > > > > > čt 5. 1. 2023 v 18:43 odesílatel Eric Dumazet <edumazet@google.com> napsal: > > > > > > On Thu, Jan 5, 2023 at 6:37 PM Jaroslav Pulchart > > > <jaroslav.pulchart@gooddata.com> wrote: > > > > > > > > Hello, > > > > > > > > I would like to report a 6.1,y regression in a network performance > > > > observed when using "git clone". > > > > > > > > BAD: "git clone" speed with kernel 6.1,y: > > > > # git clone git@github.com:..../.....git > > > > ... > > > > Receiving objects: 8% (47797/571306), 20.69 MiB | 3.27 MiB/s > > > > > > > > GOOD: "git clone" speed with kernel 6.0,y: > > > > # git clone git@github.com:..../.....git > > > > ... > > > > Receiving objects: 72% (411341/571306), 181.05 MiB | 60.27 MiB/s > > > > > > > > I bisected the issue to a commit > > > > 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro > > > > packets to gro stack". Reverting it from 6.1.y branch makes the git > > > > clone fast like with 6.0.y. > > > > > > > > > > Hmm, please provide more information. > > > > > > NIC used ? (ethtool -i eth0) > > > > > > ethtool -k eth0 # replace by your netdev name > > > > > > And packet captures would be nice (with and without the patch) > > > > > > Thanks. > > > > > > > > -- > > Jaroslav Pulchart > > Sr. Principal SW Engineer > > GoodData -- Jaroslav Pulchart Sr. Principal SW Engineer GoodData ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack) 2023-01-05 19:56 ` Jaroslav Pulchart @ 2023-01-06 14:04 ` Eric Dumazet 2023-01-06 15:04 ` Jaroslav Pulchart 0 siblings, 1 reply; 9+ messages in thread From: Eric Dumazet @ 2023-01-06 14:04 UTC (permalink / raw) To: Jaroslav Pulchart; +Cc: netdev, lixiaoyan, pabeni, davem, Igor Raits On Thu, Jan 5, 2023 at 8:56 PM Jaroslav Pulchart <jaroslav.pulchart@gooddata.com> wrote: > > I can and I did. Your "Random guess" patch fix the problem and speed > is like with 6.0.y: > > # git clone ... > ... > Receiving objects: 100% (571306/571306), 350.58 MiB | 67.51 MiB/s, done. > > Thanks, Excellent :) I will move the new test out of the fast path though, in a section we are sure gso_size is != 0. Something like this. diff --git a/net/core/gro.c b/net/core/gro.c index fd8c6a7e8d3e2e6b439109d0089f44a547c7347e..506f83d715f873c9bc3727e28ace71e00fa79d2f 100644 --- a/net/core/gro.c +++ b/net/core/gro.c @@ -505,8 +505,9 @@ static enum gro_result dev_gro_receive(struct napi_struct *napi, struct sk_buff NAPI_GRO_CB(skb)->count = 1; if (unlikely(skb_is_gso(skb))) { NAPI_GRO_CB(skb)->count = skb_shinfo(skb)->gso_segs; - /* Only support TCP at the moment. */ - if (!skb_is_gso_tcp(skb)) + /* Only support TCP and non DODGY users. */ + if (!skb_is_gso_tcp(skb) || + (skb_shinfo(skb)->gso_type & SKB_GSO_DODGY)) NAPI_GRO_CB(skb)->flush = 1; } > JP > > čt 5. 1. 2023 v 19:49 odesílatel Eric Dumazet <edumazet@google.com> napsal: > > > > On Thu, Jan 5, 2023 at 6:54 PM Jaroslav Pulchart > > <jaroslav.pulchart@gooddata.com> wrote: > > > > > > It is at KVM based VMs "CentOS 9 Stream" and "CentOS 8 Stream" using > > > upstream kernel 6.1.y. Hosted on Dell PowerEdge 7525 servers (2x AMD > > > 74F3) and OS CentOS 9 Stream again with upstream kernel 6.1.y or > > > 6.0.y. > > > > > > # ethtool -k eth0 > > > Features for eth0: > > > rx-checksumming: on [fixed] > > > tx-checksumming: on > > > tx-checksum-ipv4: off [fixed] > > > tx-checksum-ip-generic: on > > > tx-checksum-ipv6: off [fixed] > > > tx-checksum-fcoe-crc: off [fixed] > > > tx-checksum-sctp: off [fixed] > > > scatter-gather: on > > > tx-scatter-gather: on > > > tx-scatter-gather-fraglist: off [fixed] > > > tcp-segmentation-offload: on > > > tx-tcp-segmentation: on > > > tx-tcp-ecn-segmentation: on > > > tx-tcp-mangleid-segmentation: off > > > tx-tcp6-segmentation: on > > > generic-segmentation-offload: on > > > generic-receive-offload: on > > > large-receive-offload: off [fixed] > > > rx-vlan-offload: off [fixed] > > > tx-vlan-offload: off [fixed] > > > ntuple-filters: off [fixed] > > > receive-hashing: off [fixed] > > > highdma: on [fixed] > > > rx-vlan-filter: on [fixed] > > > vlan-challenged: off [fixed] > > > tx-lockless: off [fixed] > > > netns-local: off [fixed] > > > tx-gso-robust: on [fixed] > > > tx-fcoe-segmentation: off [fixed] > > > tx-gre-segmentation: off [fixed] > > > tx-gre-csum-segmentation: off [fixed] > > > tx-ipxip4-segmentation: off [fixed] > > > tx-ipxip6-segmentation: off [fixed] > > > tx-udp_tnl-segmentation: off [fixed] > > > tx-udp_tnl-csum-segmentation: off [fixed] > > > tx-gso-partial: off [fixed] > > > tx-tunnel-remcsum-segmentation: off [fixed] > > > tx-sctp-segmentation: off [fixed] > > > tx-esp-segmentation: off [fixed] > > > tx-udp-segmentation: off [fixed] > > > tx-gso-list: off [fixed] > > > fcoe-mtu: off [fixed] > > > tx-nocache-copy: off > > > loopback: off [fixed] > > > rx-fcs: off [fixed] > > > rx-all: off [fixed] > > > tx-vlan-stag-hw-insert: off [fixed] > > > rx-vlan-stag-hw-parse: off [fixed] > > > rx-vlan-stag-filter: off [fixed] > > > l2-fwd-offload: off [fixed] > > > hw-tc-offload: off [fixed] > > > esp-hw-offload: off [fixed] > > > esp-tx-csum-hw-offload: off [fixed] > > > rx-udp_tunnel-port-offload: off [fixed] > > > tls-hw-tx-offload: off [fixed] > > > tls-hw-rx-offload: off [fixed] > > > rx-gro-hw: on > > > tls-hw-record: off [fixed] > > > rx-gro-list: off > > > macsec-hw-offload: off [fixed] > > > rx-udp-gro-forwarding: off > > > hsr-tag-ins-offload: off [fixed] > > > hsr-tag-rm-offload: off [fixed] > > > hsr-fwd-offload: off [fixed] > > > hsr-dup-offload: off [fixed] > > > > > > # ethtool -i eth0 > > > driver: virtio_net > > > version: 1.0.0 > > > firmware-version: > > > expansion-rom-version: > > > bus-info: 0000:03:00.0 > > > supports-statistics: yes > > > supports-test: no > > > supports-eeprom-access: no > > > supports-register-dump: no > > > supports-priv-flags: no > > > > Random guess, can you try: > > > > diff --git a/net/core/gro.c b/net/core/gro.c > > index fd8c6a7e8d3e2e6b439109d0089f44a547c7347e..f162674e7ae1bdf96bcbf7e1ed7326729d862f9a > > 100644 > > --- a/net/core/gro.c > > +++ b/net/core/gro.c > > @@ -500,7 +500,8 @@ static enum gro_result dev_gro_receive(struct > > napi_struct *napi, struct sk_buff > > BUILD_BUG_ON(!IS_ALIGNED(offsetof(struct napi_gro_cb, zeroed), > > sizeof(u32))); /* Avoid slow > > unaligned acc */ > > *(u32 *)&NAPI_GRO_CB(skb)->zeroed = 0; > > - NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb); > > + NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb) || > > + (skb_shinfo(skb)->gso_type & SKB_GSO_DODGY); > > NAPI_GRO_CB(skb)->is_atomic = 1; > > NAPI_GRO_CB(skb)->count = 1; > > if (unlikely(skb_is_gso(skb))) { > > > > > > > > > > > > čt 5. 1. 2023 v 18:43 odesílatel Eric Dumazet <edumazet@google.com> napsal: > > > > > > > > On Thu, Jan 5, 2023 at 6:37 PM Jaroslav Pulchart > > > > <jaroslav.pulchart@gooddata.com> wrote: > > > > > > > > > > Hello, > > > > > > > > > > I would like to report a 6.1,y regression in a network performance > > > > > observed when using "git clone". > > > > > > > > > > BAD: "git clone" speed with kernel 6.1,y: > > > > > # git clone git@github.com:..../.....git > > > > > ... > > > > > Receiving objects: 8% (47797/571306), 20.69 MiB | 3.27 MiB/s > > > > > > > > > > GOOD: "git clone" speed with kernel 6.0,y: > > > > > # git clone git@github.com:..../.....git > > > > > ... > > > > > Receiving objects: 72% (411341/571306), 181.05 MiB | 60.27 MiB/s > > > > > > > > > > I bisected the issue to a commit > > > > > 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro > > > > > packets to gro stack". Reverting it from 6.1.y branch makes the git > > > > > clone fast like with 6.0.y. > > > > > > > > > > > > > Hmm, please provide more information. > > > > > > > > NIC used ? (ethtool -i eth0) > > > > > > > > ethtool -k eth0 # replace by your netdev name > > > > > > > > And packet captures would be nice (with and without the patch) > > > > > > > > Thanks. > > > > > > > > > > > > -- > > > Jaroslav Pulchart > > > Sr. Principal SW Engineer > > > GoodData > > > > -- > Jaroslav Pulchart > Sr. Principal SW Engineer > GoodData ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack) 2023-01-06 14:04 ` Eric Dumazet @ 2023-01-06 15:04 ` Jaroslav Pulchart 0 siblings, 0 replies; 9+ messages in thread From: Jaroslav Pulchart @ 2023-01-06 15:04 UTC (permalink / raw) To: Eric Dumazet; +Cc: netdev, lixiaoyan, pabeni, davem, Igor Raits OK, I re-tested it with a new version of patch and it still works. pá 6. 1. 2023 v 15:04 odesílatel Eric Dumazet <edumazet@google.com> napsal: > > On Thu, Jan 5, 2023 at 8:56 PM Jaroslav Pulchart > <jaroslav.pulchart@gooddata.com> wrote: > > > > I can and I did. Your "Random guess" patch fix the problem and speed > > is like with 6.0.y: > > > > # git clone ... > > ... > > Receiving objects: 100% (571306/571306), 350.58 MiB | 67.51 MiB/s, done. > > > > Thanks, > > Excellent :) > > I will move the new test out of the fast path though, in a section we > are sure gso_size is != 0. > > Something like this. > > diff --git a/net/core/gro.c b/net/core/gro.c > index fd8c6a7e8d3e2e6b439109d0089f44a547c7347e..506f83d715f873c9bc3727e28ace71e00fa79d2f > 100644 > --- a/net/core/gro.c > +++ b/net/core/gro.c > @@ -505,8 +505,9 @@ static enum gro_result dev_gro_receive(struct > napi_struct *napi, struct sk_buff > NAPI_GRO_CB(skb)->count = 1; > if (unlikely(skb_is_gso(skb))) { > NAPI_GRO_CB(skb)->count = skb_shinfo(skb)->gso_segs; > - /* Only support TCP at the moment. */ > - if (!skb_is_gso_tcp(skb)) > + /* Only support TCP and non DODGY users. */ > + if (!skb_is_gso_tcp(skb) || > + (skb_shinfo(skb)->gso_type & SKB_GSO_DODGY)) > NAPI_GRO_CB(skb)->flush = 1; > } > > > > > JP > > > > čt 5. 1. 2023 v 19:49 odesílatel Eric Dumazet <edumazet@google.com> napsal: > > > > > > On Thu, Jan 5, 2023 at 6:54 PM Jaroslav Pulchart > > > <jaroslav.pulchart@gooddata.com> wrote: > > > > > > > > It is at KVM based VMs "CentOS 9 Stream" and "CentOS 8 Stream" using > > > > upstream kernel 6.1.y. Hosted on Dell PowerEdge 7525 servers (2x AMD > > > > 74F3) and OS CentOS 9 Stream again with upstream kernel 6.1.y or > > > > 6.0.y. > > > > > > > > # ethtool -k eth0 > > > > Features for eth0: > > > > rx-checksumming: on [fixed] > > > > tx-checksumming: on > > > > tx-checksum-ipv4: off [fixed] > > > > tx-checksum-ip-generic: on > > > > tx-checksum-ipv6: off [fixed] > > > > tx-checksum-fcoe-crc: off [fixed] > > > > tx-checksum-sctp: off [fixed] > > > > scatter-gather: on > > > > tx-scatter-gather: on > > > > tx-scatter-gather-fraglist: off [fixed] > > > > tcp-segmentation-offload: on > > > > tx-tcp-segmentation: on > > > > tx-tcp-ecn-segmentation: on > > > > tx-tcp-mangleid-segmentation: off > > > > tx-tcp6-segmentation: on > > > > generic-segmentation-offload: on > > > > generic-receive-offload: on > > > > large-receive-offload: off [fixed] > > > > rx-vlan-offload: off [fixed] > > > > tx-vlan-offload: off [fixed] > > > > ntuple-filters: off [fixed] > > > > receive-hashing: off [fixed] > > > > highdma: on [fixed] > > > > rx-vlan-filter: on [fixed] > > > > vlan-challenged: off [fixed] > > > > tx-lockless: off [fixed] > > > > netns-local: off [fixed] > > > > tx-gso-robust: on [fixed] > > > > tx-fcoe-segmentation: off [fixed] > > > > tx-gre-segmentation: off [fixed] > > > > tx-gre-csum-segmentation: off [fixed] > > > > tx-ipxip4-segmentation: off [fixed] > > > > tx-ipxip6-segmentation: off [fixed] > > > > tx-udp_tnl-segmentation: off [fixed] > > > > tx-udp_tnl-csum-segmentation: off [fixed] > > > > tx-gso-partial: off [fixed] > > > > tx-tunnel-remcsum-segmentation: off [fixed] > > > > tx-sctp-segmentation: off [fixed] > > > > tx-esp-segmentation: off [fixed] > > > > tx-udp-segmentation: off [fixed] > > > > tx-gso-list: off [fixed] > > > > fcoe-mtu: off [fixed] > > > > tx-nocache-copy: off > > > > loopback: off [fixed] > > > > rx-fcs: off [fixed] > > > > rx-all: off [fixed] > > > > tx-vlan-stag-hw-insert: off [fixed] > > > > rx-vlan-stag-hw-parse: off [fixed] > > > > rx-vlan-stag-filter: off [fixed] > > > > l2-fwd-offload: off [fixed] > > > > hw-tc-offload: off [fixed] > > > > esp-hw-offload: off [fixed] > > > > esp-tx-csum-hw-offload: off [fixed] > > > > rx-udp_tunnel-port-offload: off [fixed] > > > > tls-hw-tx-offload: off [fixed] > > > > tls-hw-rx-offload: off [fixed] > > > > rx-gro-hw: on > > > > tls-hw-record: off [fixed] > > > > rx-gro-list: off > > > > macsec-hw-offload: off [fixed] > > > > rx-udp-gro-forwarding: off > > > > hsr-tag-ins-offload: off [fixed] > > > > hsr-tag-rm-offload: off [fixed] > > > > hsr-fwd-offload: off [fixed] > > > > hsr-dup-offload: off [fixed] > > > > > > > > # ethtool -i eth0 > > > > driver: virtio_net > > > > version: 1.0.0 > > > > firmware-version: > > > > expansion-rom-version: > > > > bus-info: 0000:03:00.0 > > > > supports-statistics: yes > > > > supports-test: no > > > > supports-eeprom-access: no > > > > supports-register-dump: no > > > > supports-priv-flags: no > > > > > > Random guess, can you try: > > > > > > diff --git a/net/core/gro.c b/net/core/gro.c > > > index fd8c6a7e8d3e2e6b439109d0089f44a547c7347e..f162674e7ae1bdf96bcbf7e1ed7326729d862f9a > > > 100644 > > > --- a/net/core/gro.c > > > +++ b/net/core/gro.c > > > @@ -500,7 +500,8 @@ static enum gro_result dev_gro_receive(struct > > > napi_struct *napi, struct sk_buff > > > BUILD_BUG_ON(!IS_ALIGNED(offsetof(struct napi_gro_cb, zeroed), > > > sizeof(u32))); /* Avoid slow > > > unaligned acc */ > > > *(u32 *)&NAPI_GRO_CB(skb)->zeroed = 0; > > > - NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb); > > > + NAPI_GRO_CB(skb)->flush = skb_has_frag_list(skb) || > > > + (skb_shinfo(skb)->gso_type & SKB_GSO_DODGY); > > > NAPI_GRO_CB(skb)->is_atomic = 1; > > > NAPI_GRO_CB(skb)->count = 1; > > > if (unlikely(skb_is_gso(skb))) { > > > > > > > > > > > > > > > > > čt 5. 1. 2023 v 18:43 odesílatel Eric Dumazet <edumazet@google.com> napsal: > > > > > > > > > > On Thu, Jan 5, 2023 at 6:37 PM Jaroslav Pulchart > > > > > <jaroslav.pulchart@gooddata.com> wrote: > > > > > > > > > > > > Hello, > > > > > > > > > > > > I would like to report a 6.1,y regression in a network performance > > > > > > observed when using "git clone". > > > > > > > > > > > > BAD: "git clone" speed with kernel 6.1,y: > > > > > > # git clone git@github.com:..../.....git > > > > > > ... > > > > > > Receiving objects: 8% (47797/571306), 20.69 MiB | 3.27 MiB/s > > > > > > > > > > > > GOOD: "git clone" speed with kernel 6.0,y: > > > > > > # git clone git@github.com:..../.....git > > > > > > ... > > > > > > Receiving objects: 72% (411341/571306), 181.05 MiB | 60.27 MiB/s > > > > > > > > > > > > I bisected the issue to a commit > > > > > > 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro > > > > > > packets to gro stack". Reverting it from 6.1.y branch makes the git > > > > > > clone fast like with 6.0.y. > > > > > > > > > > > > > > > > Hmm, please provide more information. > > > > > > > > > > NIC used ? (ethtool -i eth0) > > > > > > > > > > ethtool -k eth0 # replace by your netdev name > > > > > > > > > > And packet captures would be nice (with and without the patch) > > > > > > > > > > Thanks. > > > > > > > > > > > > > > > > -- > > > > Jaroslav Pulchart > > > > Sr. Principal SW Engineer > > > > GoodData > > > > > > > > -- > > Jaroslav Pulchart > > Sr. Principal SW Engineer > > GoodData -- Jaroslav Pulchart Sr. Principal SW Engineer GoodData ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack) 2023-01-05 17:37 Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack) Jaroslav Pulchart 2023-01-05 17:43 ` Eric Dumazet @ 2023-01-06 8:36 ` Linux kernel regression tracking (#adding) 2023-01-30 12:55 ` Linux kernel regression tracking (Thorsten Leemhuis) 1 sibling, 1 reply; 9+ messages in thread From: Linux kernel regression tracking (#adding) @ 2023-01-06 8:36 UTC (permalink / raw) To: Jaroslav Pulchart, edumazet, netdev, lixiaoyan, pabeni, davem, regressions Cc: Igor Raits [TLDR: I'm adding this report to the list of tracked Linux kernel regressions; all text you find below is based on a few templates paragraphs you might have encountered already already in similar form. See link in footer if these mails annoy you.] [CCing the regression list, as it should be in the loop for regressions: https://docs.kernel.org/admin-guide/reporting-regressions.html] On 05.01.23 18:37, Jaroslav Pulchart wrote: > Hello, > > I would like to report a 6.1,y regression in a network performance > observed when using "git clone". > > BAD: "git clone" speed with kernel 6.1,y: > # git clone git@github.com:..../.....git > ... > Receiving objects: 8% (47797/571306), 20.69 MiB | 3.27 MiB/s > > GOOD: "git clone" speed with kernel 6.0,y: > # git clone git@github.com:..../.....git > ... > Receiving objects: 72% (411341/571306), 181.05 MiB | 60.27 MiB/s > > I bisected the issue to a commit > 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro > packets to gro stack". Reverting it from 6.1.y branch makes the git > clone fast like with 6.0.y. Thanks for the report. To be sure the issue doesn't fall through the cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression tracking bot: #regzbot ^introduced 5eddb24901ee49eee23c0bfce6af2e83fd5679bd #regzbot title net: gro: Network performance regressed #regzbot ignore-activity This isn't a regression? This issue or a fix for it are already discussed somewhere else? It was fixed already? You want to clarify when the regression started to happen? Or point out I got the title or something else totally wrong? Then just reply and tell me -- ideally while also telling regzbot about it, as explained by the page listed in the footer of this mail. Reminder for developers: When fixing the issue, add 'Link:' tags pointing to the report (see page linked in footer for details). Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) -- Everything you wanna know about Linux kernel regression tracking: https://linux-regtracking.leemhuis.info/about/#tldr That page also explains what to do if mails like this annoy you. ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack) 2023-01-06 8:36 ` Linux kernel regression tracking (#adding) @ 2023-01-30 12:55 ` Linux kernel regression tracking (Thorsten Leemhuis) 0 siblings, 0 replies; 9+ messages in thread From: Linux kernel regression tracking (Thorsten Leemhuis) @ 2023-01-30 12:55 UTC (permalink / raw) To: Jaroslav Pulchart, edumazet, netdev, lixiaoyan, pabeni, davem, regressions Cc: Igor Raits [TLDR: This mail in primarily relevant for Linux kernel regression tracking. See link in footer if these mails annoy you.] On 06.01.23 09:36, Linux kernel regression tracking (#adding) wrote: > On 05.01.23 18:37, Jaroslav Pulchart wrote: >> Hello, >> >> I would like to report a 6.1,y regression in a network performance >> observed when using "git clone". >> >> BAD: "git clone" speed with kernel 6.1,y: >> # git clone git@github.com:..../.....git >> ... >> Receiving objects: 8% (47797/571306), 20.69 MiB | 3.27 MiB/s >> >> GOOD: "git clone" speed with kernel 6.0,y: >> # git clone git@github.com:..../.....git >> ... >> Receiving objects: 72% (411341/571306), 181.05 MiB | 60.27 MiB/s >> >> I bisected the issue to a commit >> 5eddb24901ee49eee23c0bfce6af2e83fd5679bd "gro: add support of (hw)gro >> packets to gro stack". Reverting it from 6.1.y branch makes the git >> clone fast like with 6.0.y. > > Thanks for the report. To be sure the issue doesn't fall through the > cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression > tracking bot: > > #regzbot ^introduced 5eddb24901ee49eee23c0bfce6af2e83fd5679bd > #regzbot title net: gro: Network performance regressed > #regzbot ignore-activity #regzbot fix: 7871f54e3dee Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat) -- Everything you wanna know about Linux kernel regression tracking: https://linux-regtracking.leemhuis.info/about/#tldr That page also explains what to do if mails like this annoy you. #regzbot ignore-activity ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2023-01-30 12:56 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-01-05 17:37 Network performance regression with linux 6.1.y. Issue bisected to "5eddb24901ee49eee23c0bfce6af2e83fd5679bd" (gro: add support of (hw)gro packets to gro stack) Jaroslav Pulchart 2023-01-05 17:43 ` Eric Dumazet 2023-01-05 17:54 ` Jaroslav Pulchart 2023-01-05 18:49 ` Eric Dumazet 2023-01-05 19:56 ` Jaroslav Pulchart 2023-01-06 14:04 ` Eric Dumazet 2023-01-06 15:04 ` Jaroslav Pulchart 2023-01-06 8:36 ` Linux kernel regression tracking (#adding) 2023-01-30 12:55 ` Linux kernel regression tracking (Thorsten Leemhuis)
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).