Re: [PATCH 00/11] XDP unaligned chunk placement support

From: "Laatz, Kevin" <kevin.laatz@intel.com>
To: Jonathan Lemon <jonathan.lemon@gmail.com>
Cc: Jakub Kicinski <jakub.kicinski@netronome.com>,
	netdev@vger.kernel.org, ast@kernel.org, daniel@iogearbox.net,
	bjorn.topel@intel.com, magnus.karlsson@intel.com,
	bpf@vger.kernel.org, intel-wired-lan@lists.osuosl.org,
	bruce.richardson@intel.com, ciara.loftus@intel.com
Subject: Re: [PATCH 00/11] XDP unaligned chunk placement support
Date: Mon, 1 Jul 2019 15:58:05 +0100	[thread overview]
Message-ID: <c6c48475-c4c8-599f-8217-4bc0f4d26a90@intel.com> (raw)
In-Reply-To: <BAE24CBF-416D-4665-B2C9-CE1F5EAE28FF@gmail.com>

On 28/06/2019 21:29, Jonathan Lemon wrote:
> On 28 Jun 2019, at 9:19, Laatz, Kevin wrote:
>> On 27/06/2019 22:25, Jakub Kicinski wrote:
>>> I think that's very limiting.  What is the challenge in providing
>>> aligned addresses, exactly?
>> The challenges are two-fold:
>> 1) it prevents using arbitrary buffer sizes, which will be an issue 
>> supporting e.g. jumbo frames in future.
>> 2) higher level user-space frameworks which may want to use AF_XDP, 
>> such as DPDK, do not currently support having buffers with 'fixed' 
>> alignment.
>>     The reason that DPDK uses arbitrary placement is that:
>>         - it would stop things working on certain NICs which need the 
>> actual writable space specified in units of 1k - therefore we need 2k 
>> + metadata space.
>>         - we place padding between buffers to avoid constantly 
>> hitting the same memory channels when accessing memory.
>>         - it allows the application to choose the actual buffer size 
>> it wants to use.
>>     We make use of the above to allow us to speed up processing 
>> significantly and also reduce the packet buffer memory size.
>>
>>     Not having arbitrary buffer alignment also means an AF_XDP driver 
>> for DPDK cannot be a drop-in replacement for existing drivers in 
>> those frameworks. Even with a new capability to allow an arbitrary 
>> buffer alignment, existing apps will need to be modified to use that 
>> new capability.
>
> Since all buffers in the umem are the same chunk size, the original 
> buffer
> address can be recalculated with some multiply/shift math. However, 
> this is
> more expensive than just a mask operation.

Yes, we can do this.

Another option we have is to add a socket option for querying the 
metadata length from the driver (assuming it doesn't vary per packet). 
We can use that information to get back the original address using 
subtraction.

Alternatively, we can change the Rx descriptor format to include the 
metadata length. We could do this in a couple of ways, for example, 
rather than returning the address at the start of the packet, instead 
return the buffer address that was passed in, and adding another 16-bit 
field to specify the start of the packet offset with that buffer. Id 
using 16-bits of descriptor space is not desirable, an alternative could 
be to limit umem sizes to e.g. 2^48 bits (256 terabytes should be 
enough, right :-) ) and use the remaining 16 bits of the address as a 
packet offset. Other variations on these approaches are obviously 
possible too.