Re: [net-next, RFC, 4/8] net: core: add recycle capabilities on skbs via page_pool API

From: Ilias Apalodimas <ilias.apalodimas@linaro.org>
To: David Miller <davem@davemloft.net>
Cc: eric.dumazet@gmail.com, fw@strlen.de, brouer@redhat.com,
	netdev@vger.kernel.org, toke@toke.dk, ard.biesheuvel@linaro.org,
	jasowang@redhat.com, bjorn.topel@intel.com, w@1wt.eu,
	saeedm@mellanox.com, mykyta.iziumtsev@gmail.com,
	borkmann@iogearbox.net, alexei.starovoitov@gmail.com,
	tariqt@mellanox.com
Subject: Re: [net-next, RFC, 4/8] net: core: add recycle capabilities on skbs via page_pool API
Date: Sat, 8 Dec 2018 22:29:54 +0200	[thread overview]
Message-ID: <20181208202954.GA18340@apalos> (raw)
In-Reply-To: <20181208.122110.2167626522000213499.davem@davemloft.net>

On Sat, Dec 08, 2018 at 12:21:10PM -0800, David Miller wrote:
> From: Ilias Apalodimas <ilias.apalodimas@linaro.org>
> Date: Sat, 8 Dec 2018 16:57:28 +0200
> 
> > The patchset speeds up the mvneta driver on the default network
> > stack. The only change that was needed was to adapt the driver to
> > using the page_pool API. The speed improvements we are seeing on
> > specific workloads (i.e 256b < packet < 400b) are almost 3x.
> > 
> > Lots of high speed drivers are doing similar recycling tricks themselves (and
> > there's no common code, everyone is doing something similar though). All we are
> > trying to do is provide a unified API to make that easier for the rest. Another
> > advantage is that if the some drivers switch to the API, adding XDP
> > functionality on them is pretty trivial.
> 
> Yeah this is a very important point moving forward.
> 
> Jesse Brandeberg brought the following up to me at LPC and I'd like to
> develop it further.
> 
> Right now we tell driver authors to write a new driver as SKB based,
> and once they've done all of that work we tell them to basically
> shoe-horn XDP support into that somewhat different framework.
> 
> Instead, the model should be the other way around, because with a raw
> meta-data free set of data buffers we can always construct an SKB or
> pass it to XDP.

Yeah exactly and it gets even worst. If the driver writer doesn't go through the
'proper' path, i.e allocate buffers and use build_skb, you end up having to
rewrite dma/memory management for the nornal stack. So it's more than 
'shoe-horning' XDP, it's re-writing and re-testing the whole thing.
The API also offers dma mapping capabilities (configurable). So you remove 
potential nasty bugs there as well.

> 
> So drivers should be targetting some raw data buffer kind of interface
> which takes care of all of this stuff.  If the buffers get wrapped
> into an SKB and get pushed into the traditional networking stack, the
> driver shouldn't know or care.  Likewise if it ends up being processed
> with XDP, it should not need to know or care.
> 
> All of those details should be behind a common layer.  Then we can
> control:
> 
> 1) Buffer handling, recycling, "fast paths"
> 
> 2) Statistics
> 
> 3) XDP feature sets
> 
> We can consolidate behavior and semantics across all of the drivers
> if we do this.  No more talk about "supporting all XDP features",
> and the inconsistencies we have because of that.
> 
> The whole common statistics discussion could be resolved with this
> common layer as well.
> 
> We'd be able to control and properly optimize everything.

/Ilias