* [RFT net-next] net: ti: add pp skb recycling support
@ 2021-06-09 12:01 Lorenzo Bianconi
2021-06-09 12:20 ` Matteo Croce
0 siblings, 1 reply; 8+ messages in thread
From: Lorenzo Bianconi @ 2021-06-09 12:01 UTC (permalink / raw)
To: netdev
Cc: lorenzo.bianconi, davem, kuba, grygorii.strashko, mcroce,
ilias.apalodimas, brouer
As already done for mvneta and mvpp2, enable skb recycling for ti
ethernet drivers
Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
This patch has just compile-tested
---
drivers/net/ethernet/ti/cpsw.c | 4 ++--
drivers/net/ethernet/ti/cpsw_new.c | 4 ++--
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/drivers/net/ethernet/ti/cpsw.c b/drivers/net/ethernet/ti/cpsw.c
index c0cd7de88316..049508667a6d 100644
--- a/drivers/net/ethernet/ti/cpsw.c
+++ b/drivers/net/ethernet/ti/cpsw.c
@@ -430,8 +430,8 @@ static void cpsw_rx_handler(void *token, int len, int status)
cpts_rx_timestamp(cpsw->cpts, skb);
skb->protocol = eth_type_trans(skb, ndev);
- /* unmap page as no netstack skb page recycling */
- page_pool_release_page(pool, page);
+ /* mark skb for recycling */
+ skb_mark_for_recycle(skb, page, pool);
netif_receive_skb(skb);
ndev->stats.rx_bytes += len;
diff --git a/drivers/net/ethernet/ti/cpsw_new.c b/drivers/net/ethernet/ti/cpsw_new.c
index 69b7a4e0220a..11f536138495 100644
--- a/drivers/net/ethernet/ti/cpsw_new.c
+++ b/drivers/net/ethernet/ti/cpsw_new.c
@@ -373,8 +373,8 @@ static void cpsw_rx_handler(void *token, int len, int status)
cpts_rx_timestamp(cpsw->cpts, skb);
skb->protocol = eth_type_trans(skb, ndev);
- /* unmap page as no netstack skb page recycling */
- page_pool_release_page(pool, page);
+ /* mark skb for recycling */
+ skb_mark_for_recycle(skb, page, pool);
netif_receive_skb(skb);
ndev->stats.rx_bytes += len;
--
2.31.1
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [RFT net-next] net: ti: add pp skb recycling support
2021-06-09 12:01 [RFT net-next] net: ti: add pp skb recycling support Lorenzo Bianconi
@ 2021-06-09 12:20 ` Matteo Croce
2021-06-09 15:02 ` Grygorii Strashko
2021-06-09 17:03 ` Grygorii Strashko
0 siblings, 2 replies; 8+ messages in thread
From: Matteo Croce @ 2021-06-09 12:20 UTC (permalink / raw)
To: Lorenzo Bianconi
Cc: netdev, Lorenzo Bianconi, David Miller, Jakub Kicinski,
grygorii.strashko, Ilias Apalodimas, Jesper Dangaard Brouer
On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi <lorenzo@kernel.org> wrote:
>
> As already done for mvneta and mvpp2, enable skb recycling for ti
> ethernet drivers
>
> Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
Looks good! If someone with the HW could provide a with and without
the patch, that would be nice!
--
per aspera ad upstream
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFT net-next] net: ti: add pp skb recycling support
2021-06-09 12:20 ` Matteo Croce
@ 2021-06-09 15:02 ` Grygorii Strashko
2021-06-09 15:12 ` Lorenzo Bianconi
2021-06-09 15:43 ` Matteo Croce
2021-06-09 17:03 ` Grygorii Strashko
1 sibling, 2 replies; 8+ messages in thread
From: Grygorii Strashko @ 2021-06-09 15:02 UTC (permalink / raw)
To: Matteo Croce, Lorenzo Bianconi
Cc: netdev, Lorenzo Bianconi, David Miller, Jakub Kicinski,
Ilias Apalodimas, Jesper Dangaard Brouer
hi
On 09/06/2021 15:20, Matteo Croce wrote:
> On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi <lorenzo@kernel.org> wrote:
>>
>> As already done for mvneta and mvpp2, enable skb recycling for ti
>> ethernet drivers
>>
>> Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
>
> Looks good! If someone with the HW could provide a with and without
> the patch, that would be nice!
>
What test would you recommend to run?
--
Best regards,
grygorii
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFT net-next] net: ti: add pp skb recycling support
2021-06-09 15:02 ` Grygorii Strashko
@ 2021-06-09 15:12 ` Lorenzo Bianconi
2021-06-09 15:43 ` Matteo Croce
1 sibling, 0 replies; 8+ messages in thread
From: Lorenzo Bianconi @ 2021-06-09 15:12 UTC (permalink / raw)
To: Grygorii Strashko
Cc: Matteo Croce, Lorenzo Bianconi, netdev, David Miller,
Jakub Kicinski, Ilias Apalodimas, Jesper Dangaard Brouer
[-- Attachment #1: Type: text/plain, Size: 640 bytes --]
> hi
>
> On 09/06/2021 15:20, Matteo Croce wrote:
> > On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi <lorenzo@kernel.org> wrote:
> > >
> > > As already done for mvneta and mvpp2, enable skb recycling for ti
> > > ethernet drivers
> > >
> > > Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
> >
> > Looks good! If someone with the HW could provide a with and without
> > the patch, that would be nice!
> >
>
> What test would you recommend to run?
Hi Grygorii,
Just pass the skb to the stack, so I think regular XDP_PASS use case is enough.
Regards,
Lorenzo
>
> --
> Best regards,
> grygorii
>
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFT net-next] net: ti: add pp skb recycling support
2021-06-09 15:02 ` Grygorii Strashko
2021-06-09 15:12 ` Lorenzo Bianconi
@ 2021-06-09 15:43 ` Matteo Croce
2021-06-09 15:55 ` Jesper Dangaard Brouer
1 sibling, 1 reply; 8+ messages in thread
From: Matteo Croce @ 2021-06-09 15:43 UTC (permalink / raw)
To: Grygorii Strashko
Cc: Lorenzo Bianconi, netdev, Lorenzo Bianconi, David Miller,
Jakub Kicinski, Ilias Apalodimas, Jesper Dangaard Brouer
On Wed, Jun 9, 2021 at 5:03 PM Grygorii Strashko
<grygorii.strashko@ti.com> wrote:
>
> hi
>
> On 09/06/2021 15:20, Matteo Croce wrote:
> > On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi <lorenzo@kernel.org> wrote:
> >>
> >> As already done for mvneta and mvpp2, enable skb recycling for ti
> >> ethernet drivers
> >>
> >> Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
> >
> > Looks good! If someone with the HW could provide a with and without
> > the patch, that would be nice!
> >
>
> What test would you recommend to run?
>
> --
> Best regards,
> grygorii
Hi Grygorii,
A test which benefits most from this kind of change is one in which
the frames are freed early.
One option would be to use mausezahn which by default sends frames
with an invalid ethertype, that are dropped very early from the stack
(I think in __netif_receive_skb_core() or near there).
Then, on the device I just watch the device statistics to count
packets per second:
mausezahn eth0 -c 0 -b $board_mac_address
This test should be precise enough for a gigabit link, on faster links
I usually use a DPDK or AF_XDP based one.
Regards,
--
per aspera ad upstream
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFT net-next] net: ti: add pp skb recycling support
2021-06-09 15:43 ` Matteo Croce
@ 2021-06-09 15:55 ` Jesper Dangaard Brouer
2021-06-09 16:58 ` Eric Dumazet
0 siblings, 1 reply; 8+ messages in thread
From: Jesper Dangaard Brouer @ 2021-06-09 15:55 UTC (permalink / raw)
To: Matteo Croce
Cc: Grygorii Strashko, Lorenzo Bianconi, netdev, Lorenzo Bianconi,
David Miller, Jakub Kicinski, Ilias Apalodimas, brouer,
Willem de Bruijn
On Wed, 9 Jun 2021 17:43:57 +0200
Matteo Croce <mcroce@linux.microsoft.com> wrote:
> On Wed, Jun 9, 2021 at 5:03 PM Grygorii Strashko
> <grygorii.strashko@ti.com> wrote:
> >
> > On 09/06/2021 15:20, Matteo Croce wrote:
> > > On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi <lorenzo@kernel.org> wrote:
> > >>
> > >> As already done for mvneta and mvpp2, enable skb recycling for ti
> > >> ethernet drivers
> > >>
> > >> Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
> > >
> > > Looks good! If someone with the HW could provide a with and without
> > > the patch, that would be nice!
> > >
> >
> > What test would you recommend to run?
> >
[...]
>
> A test which benefits most from this kind of change is one in which
> the frames are freed early.
I would also recommend running an XDP_PASS program, and then running
something that let the packets travel as deep as possible into netstack.
Not to test performance, but to make sure we didn't break something!
I've hacked up bnxt driver (it's not as straight forward as this driver
to convert) and is running some TCP tests. Not problems so-far :-0
I wanted to ask if someone knows howto setup the zero-copy TCP stuff
that google did? (but is that TX only?)
--
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
LinkedIn: http://www.linkedin.com/in/brouer
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFT net-next] net: ti: add pp skb recycling support
2021-06-09 15:55 ` Jesper Dangaard Brouer
@ 2021-06-09 16:58 ` Eric Dumazet
0 siblings, 0 replies; 8+ messages in thread
From: Eric Dumazet @ 2021-06-09 16:58 UTC (permalink / raw)
To: Jesper Dangaard Brouer, Matteo Croce
Cc: Grygorii Strashko, Lorenzo Bianconi, netdev, Lorenzo Bianconi,
David Miller, Jakub Kicinski, Ilias Apalodimas, Willem de Bruijn
On 6/9/21 5:55 PM, Jesper Dangaard Brouer wrote:
> On Wed, 9 Jun 2021 17:43:57 +0200
> Matteo Croce <mcroce@linux.microsoft.com> wrote:
>
>> On Wed, Jun 9, 2021 at 5:03 PM Grygorii Strashko
>> <grygorii.strashko@ti.com> wrote:
>>>
>>> On 09/06/2021 15:20, Matteo Croce wrote:
>>>> On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi <lorenzo@kernel.org> wrote:
>>>>>
>>>>> As already done for mvneta and mvpp2, enable skb recycling for ti
>>>>> ethernet drivers
>>>>>
>>>>> Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
>>>>
>>>> Looks good! If someone with the HW could provide a with and without
>>>> the patch, that would be nice!
>>>>
>>>
>>> What test would you recommend to run?
>>>
> [...]
>>
>> A test which benefits most from this kind of change is one in which
>> the frames are freed early.
>
> I would also recommend running an XDP_PASS program, and then running
> something that let the packets travel as deep as possible into netstack.
> Not to test performance, but to make sure we didn't break something!
>
> I've hacked up bnxt driver (it's not as straight forward as this driver
> to convert) and is running some TCP tests. Not problems so-far :-0
>
> I wanted to ask if someone knows howto setup the zero-copy TCP stuff
> that google did? (but is that TX only?)
>
TX does not need anything fancy really.
For RX part, you can look at https://netdevconf.info/0x14/pub/slides/62/Implementing%20TCP%20RX%20zero%20copy.pdf
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [RFT net-next] net: ti: add pp skb recycling support
2021-06-09 12:20 ` Matteo Croce
2021-06-09 15:02 ` Grygorii Strashko
@ 2021-06-09 17:03 ` Grygorii Strashko
1 sibling, 0 replies; 8+ messages in thread
From: Grygorii Strashko @ 2021-06-09 17:03 UTC (permalink / raw)
To: Matteo Croce, Lorenzo Bianconi
Cc: netdev, Lorenzo Bianconi, David Miller, Jakub Kicinski,
Ilias Apalodimas, Jesper Dangaard Brouer
On 09/06/2021 15:20, Matteo Croce wrote:
> On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi <lorenzo@kernel.org> wrote:
>>
>> As already done for mvneta and mvpp2, enable skb recycling for ti
>> ethernet drivers
>>
>> Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
>
> Looks good! If someone with the HW could provide a with and without
> the patch, that would be nice!
>
Not sure to which mail to answer, so answering here - thanks all
1) I've simulated packet drop using iperf
Host:
- arp -s <some IP> <unknown MAC>
- iperf -c <some IP> -u -l60 -b700M -t60 -i1
DUP:
- place interface in promisc mode
- check rx_packets stats
I see big improvement ~47Kpps vs ~64Kpps
2) I've run iperf3 tests - see no regressions, but also not too much improvements
3) I've applied 2 patches
- this one
- and [1]
Results are below, Thank you
Tested-by: Grygorii Strashko <grygorii.strashko@ti.com>
Reviewed-by: Grygorii Strashko <grygorii.strashko@ti.com>
[1] https://patchwork.kernel.org/project/netdevbpf/patch/20210609103326.278782-18-toke@redhat.com/
=========== Before:
[perf top]
47.15% [kernel] [k] _raw_spin_unlock_irqrestore
11.77% [kernel] [k] __cpdma_chan_free
3.16% [kernel] [k] ___bpf_prog_run
2.52% [kernel] [k] cpsw_rx_vlan_encap
2.34% [kernel] [k] __netif_receive_skb_core
2.27% [kernel] [k] free_unref_page
2.26% [kernel] [k] kmem_cache_free
2.24% [kernel] [k] kmem_cache_alloc
1.69% [kernel] [k] __softirqentry_text_start
1.61% [kernel] [k] cpsw_rx_handler
1.19% [kernel] [k] page_pool_release_page
1.19% [kernel] [k] clear_bits_ll
1.15% [kernel] [k] page_frag_free
1.06% [kernel] [k] __dma_page_dev_to_cpu
0.99% [kernel] [k] memset
0.94% [kernel] [k] __alloc_pages_bulk
0.92% [kernel] [k] kfree_skb
0.85% [kernel] [k] packet_rcv
0.78% [kernel] [k] page_address
0.75% [kernel] [k] v7_dma_inv_range
0.71% [kernel] [k] __lock_text_start
[rx packets - with packet drop]
rxdiff 48004
rxdiff 47630
rxdiff 47538Š
…\x03
[iperf3 TCP]
iperf3 -c 192.168.1.1 -i1
[ 5] 0.00-10.00 sec 873 MBytes 732 Mbits/sec 0 sender
[ 5] 0.00-10.01 sec 866 MBytes 726 Mbits/sec receiver
=========== After:
[perf top - with packet drop]
40.58% [kernel] [k] _raw_spin_unlock_irqrestore
16.18% [kernel] [k] __softirqentry_text_start
10.33% [kernel] [k] __cpdma_chan_free
2.62% [kernel] [k] ___bpf_prog_run
2.05% [kernel] [k] cpsw_rx_vlan_encap
2.00% [kernel] [k] kmem_cache_alloc
1.86% [kernel] [k] __netif_receive_skb_core
1.80% [kernel] [k] kmem_cache_free
1.63% [kernel] [k] cpsw_rx_handler
1.12% [kernel] [k] cpsw_rx_mq_poll
1.11% [kernel] [k] page_pool_put_page
1.04% [kernel] [k] _raw_spin_unlock
0.97% [kernel] [k] clear_bits_ll
0.90% [kernel] [k] packet_rcv
0.88% [kernel] [k] __dma_page_dev_to_cpu
0.85% [kernel] [k] kfree_skb
0.80% [kernel] [k] memset
0.71% [kernel] [k] __lock_text_start
0.66% [kernel] [k] v7_dma_inv_range
0.64% [kernel] [k] gen_pool_free_owner
0.58% [kernel] [k] __rcu_read_unlock
[rx packets - with packet drop]
rxdiff 65843
rxdiff 66722
rxdiff 65264
[iperf3 TCP]
iperf3 -c 192.168.1.1 -i1
[ 5] 0.00-10.00 sec 884 MBytes 742 Mbits/sec 0 sender
[ 5] 0.00-10.01 sec 878 MBytes 735 Mbits/sec receiver
--
Best regards,
grygorii
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2021-06-09 17:03 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-09 12:01 [RFT net-next] net: ti: add pp skb recycling support Lorenzo Bianconi
2021-06-09 12:20 ` Matteo Croce
2021-06-09 15:02 ` Grygorii Strashko
2021-06-09 15:12 ` Lorenzo Bianconi
2021-06-09 15:43 ` Matteo Croce
2021-06-09 15:55 ` Jesper Dangaard Brouer
2021-06-09 16:58 ` Eric Dumazet
2021-06-09 17:03 ` Grygorii Strashko
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.