Re: [net-next, RFC, 4/8] net: core: add recycle capabilities on skbs via page_pool API

From: Jesper Dangaard Brouer <brouer@redhat.com>
To: Florian Westphal <fw@strlen.de>
Cc: netdev@vger.kernel.org, "David S. Miller" <davem@davemloft.net>,
	"Toke Høiland-Jørgensen" <toke@toke.dk>,
	ard.biesheuvel@linaro.org, "Jason Wang" <jasowang@redhat.com>,
	ilias.apalodimas@linaro.org, BjörnTöpel <bjorn.topel@intel.com>,
	w@1wt.eu, "Saeed Mahameed" <saeedm@mellanox.com>,
	mykyta.iziumtsev@gmail.com,
	"Daniel Borkmann" <borkmann@iogearbox.net>,
	"Alexei Starovoitov" <alexei.starovoitov@gmail.com>,
	"Tariq Toukan" <tariqt@mellanox.com>,
	brouer@redhat.com
Subject: Re: [net-next, RFC, 4/8] net: core: add recycle capabilities on skbs via page_pool API
Date: Sat, 8 Dec 2018 12:36:10 +0100	[thread overview]
Message-ID: <20181208123610.7990e85d@redhat.com> (raw)
In-Reply-To: <20181208095758.GA32028@strlen.de>

On Sat, 8 Dec 2018 10:57:58 +0100
Florian Westphal <fw@strlen.de> wrote:

> Jesper Dangaard Brouer <brouer@redhat.com> wrote:
> > From: Ilias Apalodimas <ilias.apalodimas@linaro.org>
> > 
> > This patch is changing struct sk_buff, and is thus per-definition
> > controversial.
> > 
> > Place a new member 'mem_info' of type struct xdp_mem_info, just after
> > members (flags) head_frag and pfmemalloc, And not in between
> > headers_start/end to ensure skb_copy() and pskb_copy() work as-is.
> > Copying mem_info during skb_clone() is required.  This makes sure that
> > pages are correctly freed or recycled during the altered
> > skb_free_head() invocation.  
> 
> I read this to mean that this 'info' isn't accessed/needed until skb
> is freed.  Any reason its not added at the end?
>
> This would avoid moving other fields that are probably accessed
> more frequently during processing.

It is placed here because it is close to skb->head_frag, which is used
also used when SKB is freed.  This is the reason, both because I think
we can move the skb->head_frag bit into mem_info, and due to cacheline
accesses (e.g. skb->cloned and destructor pointer are also read, which
is on that cache-line)

The annoying part is actually that depending on the kernel config
options CONFIG_XFRM, CONFIG_NF_CONNTRACK and CONFIG_BRIDGE_NETFILTER,
whether there is a cache-line split, where mem_info gets moved into the
next cacheline.

I do like the idea of moving it to the end, iif we also move
skb->head_frag into mem_info, as skb->head (on end-cacheline) is also
accessed during free, but given we still need to read the cache-line
containing skb->{cloned,destructor}, when I don't think it will be a
win.  I think the skb->cloned value is read during lifetime of the SKB.

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer