All of lore.kernel.org
 help / color / mirror / Atom feed
* [RFT net-next] net: ti: add pp skb recycling support
@ 2021-06-09 12:01 Lorenzo Bianconi
  2021-06-09 12:20 ` Matteo Croce
  0 siblings, 1 reply; 8+ messages in thread
From: Lorenzo Bianconi @ 2021-06-09 12:01 UTC (permalink / raw)
  To: netdev
  Cc: lorenzo.bianconi, davem, kuba, grygorii.strashko, mcroce,
	ilias.apalodimas, brouer

As already done for mvneta and mvpp2, enable skb recycling for ti
ethernet drivers

Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
---
This patch has just compile-tested
---
 drivers/net/ethernet/ti/cpsw.c     | 4 ++--
 drivers/net/ethernet/ti/cpsw_new.c | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/drivers/net/ethernet/ti/cpsw.c b/drivers/net/ethernet/ti/cpsw.c
index c0cd7de88316..049508667a6d 100644
--- a/drivers/net/ethernet/ti/cpsw.c
+++ b/drivers/net/ethernet/ti/cpsw.c
@@ -430,8 +430,8 @@ static void cpsw_rx_handler(void *token, int len, int status)
 		cpts_rx_timestamp(cpsw->cpts, skb);
 	skb->protocol = eth_type_trans(skb, ndev);
 
-	/* unmap page as no netstack skb page recycling */
-	page_pool_release_page(pool, page);
+	/* mark skb for recycling */
+	skb_mark_for_recycle(skb, page, pool);
 	netif_receive_skb(skb);
 
 	ndev->stats.rx_bytes += len;
diff --git a/drivers/net/ethernet/ti/cpsw_new.c b/drivers/net/ethernet/ti/cpsw_new.c
index 69b7a4e0220a..11f536138495 100644
--- a/drivers/net/ethernet/ti/cpsw_new.c
+++ b/drivers/net/ethernet/ti/cpsw_new.c
@@ -373,8 +373,8 @@ static void cpsw_rx_handler(void *token, int len, int status)
 		cpts_rx_timestamp(cpsw->cpts, skb);
 	skb->protocol = eth_type_trans(skb, ndev);
 
-	/* unmap page as no netstack skb page recycling */
-	page_pool_release_page(pool, page);
+	/* mark skb for recycling */
+	skb_mark_for_recycle(skb, page, pool);
 	netif_receive_skb(skb);
 
 	ndev->stats.rx_bytes += len;
-- 
2.31.1


^ permalink raw reply related	[flat|nested] 8+ messages in thread
* Re: [RFT net-next] net: ti: add pp skb recycling support
  2021-06-09 12:01 [RFT net-next] net: ti: add pp skb recycling support Lorenzo Bianconi
@ 2021-06-09 12:20 ` Matteo Croce
  2021-06-09 15:02   ` Grygorii Strashko
  2021-06-09 17:03   ` Grygorii Strashko
  0 siblings, 2 replies; 8+ messages in thread
From: Matteo Croce @ 2021-06-09 12:20 UTC (permalink / raw)
  To: Lorenzo Bianconi
  Cc: netdev, Lorenzo Bianconi, David Miller, Jakub Kicinski,
	grygorii.strashko, Ilias Apalodimas, Jesper Dangaard Brouer

On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi <lorenzo@kernel.org> wrote:
>
> As already done for mvneta and mvpp2, enable skb recycling for ti
> ethernet drivers
>
> Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>

Looks good! If someone with the HW could provide a with and without
the patch, that would be nice!

-- 
per aspera ad upstream

^ permalink raw reply	[flat|nested] 8+ messages in thread
* Re: [RFT net-next] net: ti: add pp skb recycling support
  2021-06-09 12:20 ` Matteo Croce
@ 2021-06-09 15:02   ` Grygorii Strashko
  2021-06-09 15:12     ` Lorenzo Bianconi
  2021-06-09 15:43     ` Matteo Croce
  2021-06-09 17:03   ` Grygorii Strashko
  1 sibling, 2 replies; 8+ messages in thread
From: Grygorii Strashko @ 2021-06-09 15:02 UTC (permalink / raw)
  To: Matteo Croce, Lorenzo Bianconi
  Cc: netdev, Lorenzo Bianconi, David Miller, Jakub Kicinski,
	Ilias Apalodimas, Jesper Dangaard Brouer

hi

On 09/06/2021 15:20, Matteo Croce wrote:
> On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi <lorenzo@kernel.org> wrote:
>>
>> As already done for mvneta and mvpp2, enable skb recycling for ti
>> ethernet drivers
>>
>> Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
> 
> Looks good! If someone with the HW could provide a with and without
> the patch, that would be nice!
> 

What test would you recommend to run?

-- 
Best regards,
grygorii

^ permalink raw reply	[flat|nested] 8+ messages in thread
* Re: [RFT net-next] net: ti: add pp skb recycling support
  2021-06-09 15:02   ` Grygorii Strashko
@ 2021-06-09 15:12     ` Lorenzo Bianconi
  2021-06-09 15:43     ` Matteo Croce
  1 sibling, 0 replies; 8+ messages in thread
From: Lorenzo Bianconi @ 2021-06-09 15:12 UTC (permalink / raw)
  To: Grygorii Strashko
  Cc: Matteo Croce, Lorenzo Bianconi, netdev, David Miller,
	Jakub Kicinski, Ilias Apalodimas, Jesper Dangaard Brouer

[-- Attachment #1: Type: text/plain, Size: 640 bytes --]

> hi
> 
> On 09/06/2021 15:20, Matteo Croce wrote:
> > On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi <lorenzo@kernel.org> wrote:
> > > 
> > > As already done for mvneta and mvpp2, enable skb recycling for ti
> > > ethernet drivers
> > > 
> > > Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
> > 
> > Looks good! If someone with the HW could provide a with and without
> > the patch, that would be nice!
> > 
> 
> What test would you recommend to run?

Hi Grygorii,

Just pass the skb to the stack, so I think regular XDP_PASS use case is enough.

Regards,
Lorenzo

> 
> -- 
> Best regards,
> grygorii
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

^ permalink raw reply	[flat|nested] 8+ messages in thread
* Re: [RFT net-next] net: ti: add pp skb recycling support
  2021-06-09 15:02   ` Grygorii Strashko
  2021-06-09 15:12     ` Lorenzo Bianconi
@ 2021-06-09 15:43     ` Matteo Croce
  2021-06-09 15:55       ` Jesper Dangaard Brouer
  1 sibling, 1 reply; 8+ messages in thread
From: Matteo Croce @ 2021-06-09 15:43 UTC (permalink / raw)
  To: Grygorii Strashko
  Cc: Lorenzo Bianconi, netdev, Lorenzo Bianconi, David Miller,
	Jakub Kicinski, Ilias Apalodimas, Jesper Dangaard Brouer

On Wed, Jun 9, 2021 at 5:03 PM Grygorii Strashko
<grygorii.strashko@ti.com> wrote:
>
> hi
>
> On 09/06/2021 15:20, Matteo Croce wrote:
> > On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi <lorenzo@kernel.org> wrote:
> >>
> >> As already done for mvneta and mvpp2, enable skb recycling for ti
> >> ethernet drivers
> >>
> >> Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
> >
> > Looks good! If someone with the HW could provide a with and without
> > the patch, that would be nice!
> >
>
> What test would you recommend to run?
>
> --
> Best regards,
> grygorii

Hi Grygorii,

A test which benefits most from this kind of change is one in which
the frames are freed early.
One option would be to use mausezahn which by default sends frames
with an invalid ethertype, that are dropped very early from the stack
(I think in __netif_receive_skb_core() or near there).
Then, on the device I just watch the device statistics to count
packets per second:

mausezahn eth0 -c 0 -b $board_mac_address

This test should be precise enough for a gigabit link, on faster links
I usually use a DPDK or AF_XDP based one.

Regards,
-- 
per aspera ad upstream

^ permalink raw reply	[flat|nested] 8+ messages in thread
* Re: [RFT net-next] net: ti: add pp skb recycling support
  2021-06-09 15:43     ` Matteo Croce
@ 2021-06-09 15:55       ` Jesper Dangaard Brouer
  2021-06-09 16:58         ` Eric Dumazet
  0 siblings, 1 reply; 8+ messages in thread
From: Jesper Dangaard Brouer @ 2021-06-09 15:55 UTC (permalink / raw)
  To: Matteo Croce
  Cc: Grygorii Strashko, Lorenzo Bianconi, netdev, Lorenzo Bianconi,
	David Miller, Jakub Kicinski, Ilias Apalodimas, brouer,
	Willem de Bruijn

On Wed, 9 Jun 2021 17:43:57 +0200
Matteo Croce <mcroce@linux.microsoft.com> wrote:

> On Wed, Jun 9, 2021 at 5:03 PM Grygorii Strashko
> <grygorii.strashko@ti.com> wrote:
> >
> > On 09/06/2021 15:20, Matteo Croce wrote:  
> > > On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi <lorenzo@kernel.org> wrote:  
> > >>
> > >> As already done for mvneta and mvpp2, enable skb recycling for ti
> > >> ethernet drivers
> > >>
> > >> Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>  
> > >
> > > Looks good! If someone with the HW could provide a with and without
> > > the patch, that would be nice!
> > >  
> >
> > What test would you recommend to run?
> >
[...]
> 
> A test which benefits most from this kind of change is one in which
> the frames are freed early.

I would also recommend running an XDP_PASS program, and then running
something that let the packets travel as deep as possible into netstack.
Not to test performance, but to make sure we didn't break something!

I've hacked up bnxt driver (it's not as straight forward as this driver
to convert) and is running some TCP tests.  Not problems so-far :-0

I wanted to ask if someone knows howto setup the zero-copy TCP stuff
that google did? (but is that TX only?)

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer


^ permalink raw reply	[flat|nested] 8+ messages in thread
* Re: [RFT net-next] net: ti: add pp skb recycling support
  2021-06-09 15:55       ` Jesper Dangaard Brouer
@ 2021-06-09 16:58         ` Eric Dumazet
  0 siblings, 0 replies; 8+ messages in thread
From: Eric Dumazet @ 2021-06-09 16:58 UTC (permalink / raw)
  To: Jesper Dangaard Brouer, Matteo Croce
  Cc: Grygorii Strashko, Lorenzo Bianconi, netdev, Lorenzo Bianconi,
	David Miller, Jakub Kicinski, Ilias Apalodimas, Willem de Bruijn



On 6/9/21 5:55 PM, Jesper Dangaard Brouer wrote:
> On Wed, 9 Jun 2021 17:43:57 +0200
> Matteo Croce <mcroce@linux.microsoft.com> wrote:
> 
>> On Wed, Jun 9, 2021 at 5:03 PM Grygorii Strashko
>> <grygorii.strashko@ti.com> wrote:
>>>
>>> On 09/06/2021 15:20, Matteo Croce wrote:  
>>>> On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi <lorenzo@kernel.org> wrote:  
>>>>>
>>>>> As already done for mvneta and mvpp2, enable skb recycling for ti
>>>>> ethernet drivers
>>>>>
>>>>> Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>  
>>>>
>>>> Looks good! If someone with the HW could provide a with and without
>>>> the patch, that would be nice!
>>>>  
>>>
>>> What test would you recommend to run?
>>>
> [...]
>>
>> A test which benefits most from this kind of change is one in which
>> the frames are freed early.
> 
> I would also recommend running an XDP_PASS program, and then running
> something that let the packets travel as deep as possible into netstack.
> Not to test performance, but to make sure we didn't break something!
> 
> I've hacked up bnxt driver (it's not as straight forward as this driver
> to convert) and is running some TCP tests.  Not problems so-far :-0
> 
> I wanted to ask if someone knows howto setup the zero-copy TCP stuff
> that google did? (but is that TX only?)
> 

TX does not need anything fancy really.

For RX part, you can look at https://netdevconf.info/0x14/pub/slides/62/Implementing%20TCP%20RX%20zero%20copy.pdf

^ permalink raw reply	[flat|nested] 8+ messages in thread
* Re: [RFT net-next] net: ti: add pp skb recycling support
  2021-06-09 12:20 ` Matteo Croce
  2021-06-09 15:02   ` Grygorii Strashko
@ 2021-06-09 17:03   ` Grygorii Strashko
  1 sibling, 0 replies; 8+ messages in thread
From: Grygorii Strashko @ 2021-06-09 17:03 UTC (permalink / raw)
  To: Matteo Croce, Lorenzo Bianconi
  Cc: netdev, Lorenzo Bianconi, David Miller, Jakub Kicinski,
	Ilias Apalodimas, Jesper Dangaard Brouer



On 09/06/2021 15:20, Matteo Croce wrote:
> On Wed, Jun 9, 2021 at 2:01 PM Lorenzo Bianconi <lorenzo@kernel.org> wrote:
>>
>> As already done for mvneta and mvpp2, enable skb recycling for ti
>> ethernet drivers
>>
>> Signed-off-by: Lorenzo Bianconi <lorenzo@kernel.org>
> 
> Looks good! If someone with the HW could provide a with and without
> the patch, that would be nice!
> 

Not sure to which mail to answer, so answering here - thanks all

1) I've simulated packet drop using iperf
Host:
- arp -s <some IP> <unknown MAC>
- iperf -c <some IP> -u -l60 -b700M -t60 -i1

DUP:
- place interface in promisc mode
- check rx_packets stats

I see big improvement ~47Kpps vs ~64Kpps

2) I've run iperf3 tests - see no regressions, but also not too much improvements

3) I've applied 2 patches
- this one
- and [1]

Results are below, Thank you

Tested-by: Grygorii Strashko <grygorii.strashko@ti.com>
Reviewed-by: Grygorii Strashko <grygorii.strashko@ti.com>

[1] https://patchwork.kernel.org/project/netdevbpf/patch/20210609103326.278782-18-toke@redhat.com/

=========== Before:
[perf top]
  47.15%  [kernel]                   [k] _raw_spin_unlock_irqrestore
   11.77%  [kernel]                   [k] __cpdma_chan_free
    3.16%  [kernel]                   [k] ___bpf_prog_run
    2.52%  [kernel]                   [k] cpsw_rx_vlan_encap
    2.34%  [kernel]                   [k] __netif_receive_skb_core
    2.27%  [kernel]                   [k] free_unref_page
    2.26%  [kernel]                   [k] kmem_cache_free
    2.24%  [kernel]                   [k] kmem_cache_alloc
    1.69%  [kernel]                   [k] __softirqentry_text_start
    1.61%  [kernel]                   [k] cpsw_rx_handler
    1.19%  [kernel]                   [k] page_pool_release_page
    1.19%  [kernel]                   [k] clear_bits_ll
    1.15%  [kernel]                   [k] page_frag_free
    1.06%  [kernel]                   [k] __dma_page_dev_to_cpu
    0.99%  [kernel]                   [k] memset
    0.94%  [kernel]                   [k] __alloc_pages_bulk
    0.92%  [kernel]                   [k] kfree_skb
    0.85%  [kernel]                   [k] packet_rcv
    0.78%  [kernel]                   [k] page_address
    0.75%  [kernel]                   [k] v7_dma_inv_range
    0.71%  [kernel]                   [k] __lock_text_start

[rx packets - with packet drop]
rxdiff 48004
rxdiff 47630
rxdiff 47538Š
…\x03
[iperf3 TCP]
iperf3 -c 192.168.1.1 -i1
[  5]   0.00-10.00  sec   873 MBytes   732 Mbits/sec    0             sender
[  5]   0.00-10.01  sec   866 MBytes   726 Mbits/sec                  receiver

=========== After:
[perf top - with packet drop]
  40.58%  [kernel]                   [k] _raw_spin_unlock_irqrestore
   16.18%  [kernel]                   [k] __softirqentry_text_start
   10.33%  [kernel]                   [k] __cpdma_chan_free
    2.62%  [kernel]                   [k] ___bpf_prog_run
    2.05%  [kernel]                   [k] cpsw_rx_vlan_encap
    2.00%  [kernel]                   [k] kmem_cache_alloc
    1.86%  [kernel]                   [k] __netif_receive_skb_core
    1.80%  [kernel]                   [k] kmem_cache_free
    1.63%  [kernel]                   [k] cpsw_rx_handler
    1.12%  [kernel]                   [k] cpsw_rx_mq_poll
    1.11%  [kernel]                   [k] page_pool_put_page
    1.04%  [kernel]                   [k] _raw_spin_unlock
    0.97%  [kernel]                   [k] clear_bits_ll
    0.90%  [kernel]                   [k] packet_rcv
    0.88%  [kernel]                   [k] __dma_page_dev_to_cpu
    0.85%  [kernel]                   [k] kfree_skb
    0.80%  [kernel]                   [k] memset
    0.71%  [kernel]                   [k] __lock_text_start
    0.66%  [kernel]                   [k] v7_dma_inv_range
    0.64%  [kernel]                   [k] gen_pool_free_owner
    0.58%  [kernel]                   [k] __rcu_read_unlock

[rx packets - with packet drop]
rxdiff 65843
rxdiff 66722
rxdiff 65264

[iperf3 TCP]
iperf3 -c 192.168.1.1 -i1
[  5]   0.00-10.00  sec   884 MBytes   742 Mbits/sec    0             sender
[  5]   0.00-10.01  sec   878 MBytes   735 Mbits/sec                  receiver


-- 
Best regards,
grygorii

^ permalink raw reply	[flat|nested] 8+ messages in thread
end of thread, other threads:[~2021-06-09 17:03 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-06-09 12:01 [RFT net-next] net: ti: add pp skb recycling support Lorenzo Bianconi
2021-06-09 12:20 ` Matteo Croce
2021-06-09 15:02   ` Grygorii Strashko
2021-06-09 15:12     ` Lorenzo Bianconi
2021-06-09 15:43     ` Matteo Croce
2021-06-09 15:55       ` Jesper Dangaard Brouer
2021-06-09 16:58         ` Eric Dumazet
2021-06-09 17:03   ` Grygorii Strashko

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.