Re: [EXT] Re: [PATCH v3] aquantia: Remove the build_skb path

From: Igor Russkikh <irusskikh@marvell.com>
To: Maciej Fijalkowski <maciej.fijalkowski@intel.com>,
	"Ramsay, Lincoln" <Lincoln.Ramsay@digi.com>
Cc: Florian Westphal <fw@strlen.de>,
	"David S. Miller" <davem@davemloft.net>,
	Jakub Kicinski <kuba@kernel.org>, <netdev@vger.kernel.org>,
	"Dmitry Bogdanov [C]" <dbogdanov@marvell.com>
Subject: Re: [EXT] Re: [PATCH v3] aquantia: Remove the build_skb path
Date: Fri, 20 Nov 2020 11:18:34 +0300	[thread overview]
Message-ID: <2fbb195a-a1b5-cec0-1ba1-bf45efc0ad24@marvell.com> (raw)
In-Reply-To: <20201119224916.GA24569@ranger.igk.intel.com>

On 20/11/2020 1:49 am, Maciej Fijalkowski wrote:
> External Email
> 
> ----------------------------------------------------------------------
> On Thu, Nov 19, 2020 at 10:34:48PM +0000, Ramsay, Lincoln wrote:
>> When performing IPv6 forwarding, there is an expectation that SKBs
>> will have some headroom. When forwarding a packet from the aquantia
>> driver, this does not always happen, triggering a kernel warning.
>>
>> The build_skb path fails to allow for an SKB header, but the hardware
>> buffer it is built around won't allow for this anyway. Just always use
> the
>> slower codepath that copies memory into an allocated SKB.
>>
>> Signed-off-by: Lincoln Ramsay <lincoln.ramsay@opengear.com>
>> ---
> 
> (Next time please include in the subject the tree that you're targetting
> the patch)
> 
> I feel like it's only a workaround, not a real solution. On previous
> thread Igor says:
> 
> "The limitation here is we can't tell HW on granularity less than 1K."
> 
> Are you saying that the minimum headroom that we could provide is 1k?

We can tell HW to place packets with 4 bytes granularity addresses, but the
problem is the length granularity of this buffer - 1K.

This means we can do as Ramsay initially suggested - just offset the packet
placement. But then we have to guarantee that 1K after this offset is
available to HW.

Since normal layout is 1400 packets - we do use 2K (half page) for each packet.
This way we reuse each allocated page for at least two packets (and putting
skb_shared into the remaining 512b).

Obviously we may allocate 4K page for a single packet, and tell HW that it can
use 3K for data. This'll give 1K headroom. Quite an overload - assuming IMIX
is of 0.5K - 1.4K..

Of course that depends on a usecase. If you know all your traffic is 16K
jumbos - putting 1K headroom is very small overhead on memory usage.

> Maybe put more pressure on memory side and pull in order-1 pages, provide
> this big headroom and tailroom for skb_shared_info and use build_skb by
> default? With standard 1500 byte MTU.
I know many customers do consider AQC chips in near embedded environments
(routers, etc). They really do care about memories. So that could be risky.

> This issue would pop up again if this driver would like to support XDP
> where 256 byte headroom will have to be provided.

Actually it already popped. Thats one of the reasons I'm delaying with xdp
patch series for this driver.

I think the best tradeoff here would be allocating order 1 or 2 pages (i.e. 8K
or 16K), and reuse the page for multiple placements of 2K XDP packets:

(256+2048)*3 = 6912 (1K overhead for each 3 packets)

(256+2048)*7 = 16128 (200b overhead over 7 packets)

Regards,
  Igor