Re: [PATCH net-next v2 05/12] iavf: always use a full order-0 page

From: Alexander Lobakin <aleksander.lobakin@intel.com>
To: David Laight <David.Laight@ACULAB.COM>
Cc: "David S. Miller" <davem@davemloft.net>,
	Eric Dumazet <edumazet@google.com>,
	Jakub Kicinski <kuba@kernel.org>, Paolo Abeni <pabeni@redhat.com>,
	Maciej Fijalkowski <maciej.fijalkowski@intel.com>,
	Magnus Karlsson <magnus.karlsson@intel.com>,
	Michal Kubiak <michal.kubiak@intel.com>,
	Larysa Zaremba <larysa.zaremba@intel.com>,
	"Jesper Dangaard Brouer" <hawk@kernel.org>,
	Ilias Apalodimas <ilias.apalodimas@linaro.org>,
	Christoph Hellwig <hch@lst.de>,
	Paul Menzel <pmenzel@molgen.mpg.de>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"intel-wired-lan@lists.osuosl.org"
	<intel-wired-lan@lists.osuosl.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH net-next v2 05/12] iavf: always use a full order-0 page
Date: Fri, 26 May 2023 14:52:57 +0200	[thread overview]
Message-ID: <88c4f8c4-085c-1934-536b-0fc8a38205c7@intel.com> (raw)
In-Reply-To: <9acb1863f53542b6bd247ad641b8c0fa@AcuMS.aculab.com>

From: David Laight <David.Laight@ACULAB.COM>
Date: Fri, 26 May 2023 08:57:31 +0000

> From: Alexander Lobakin
>> Sent: 25 May 2023 13:58

[...]

>> 4096 page
>> 64 head, 320 tail
>> 3712 HW buffer size
>> 3686 max MTU w/o frags
> 
> I'd have thought it was important to pack multiple buffers for
> MTU 1500 into a single page.
> 512 bytes split between head and tail room really ought to
> be enough for most cases.
> 
> Is much tailroom ever used for received packets?

You don't have any tailroom at all when you split 4k page into two
halves on x86_64. From those 512 bytes, 320+ are used for
skb_shared_info. And then you're left with 192 bytes or even less (with
increased MAX_SKB_FRAGS, which becomes a new trend thanks to Eric and
Big TCP). XDP requires 256 bytes of headroom -- and you already don't
have them even with the default number of frags.

> It is used to append data to packets being sent - but that isn't
> really relevant here.
> 
> While the unused memory is moderate for 4k pages, it is horrid
> for anything with large pages - think 64k and above.

But hey, there's always unused space and it's arbitrary whether to treat
it "horrid". For example, imagine a machine mostly handling 64-byte
traffic. From that point of view, even splitting pages into 2 halves
still is "horrid" -- 2048 bytes of truesize only to receive 64 bytes :>

> IIRC large pages are common on big PPC and maybe some arm cpus.

Even in that case, the percentage of 4k-page machines running those NICs
are much higher than > 4k. I thought we usually optimize things for the
most common usecases. Adding a couple hundred lines, branches on
hotpaths, limitations etc. only to serve a particular architecture... Dunno.
I have 16k pages on my private development machines and I have no issues
with using 1 page per frame in their NIC drivers :s

> 
> 	David
> 
> -
> Registered Address Lakeside, Bramley Road, Mount Farm, Milton Keynes, MK1 1PT, UK
> Registration No: 1397386 (Wales)
> 

Thanks,
Olek