AF_XDP and dma

AF_XDP and dma

[ratheesh kannoth]

Hi List,

Could you help with the below query ?

if userspace allocates UMEM memory in non contiguous physical memory
 (contiguous in virual address space),  how can i configure DMA ?

https://github.com/xdp-project/xdp-tutorial  AF_XDP example uses
posix_memalign(). will it ensure each buffer chunk is in contiguous
physical address space ?

-Ratheesh

Re: AF_XDP and dma

[Magnus Karlsson]

On Fri, Apr 9, 2021 at 11:36 AM ratheesh kannoth <ratheesh.ksz@gmail.com> wrote:
>
> Hi List,
>
> Could you help with the below query ?
>
> if userspace allocates UMEM memory in non contiguous physical memory
>  (contiguous in virual address space),  how can i configure DMA ?
>
> https://github.com/xdp-project/xdp-tutorial  AF_XDP example uses
> posix_memalign(). will it ensure each buffer chunk is in contiguous
> physical address space ?

The short answer is that you do not have to worry about that at all in
user space. DMA is taken care of by the kernel. The only requirement
is that the umem is located in a virtually contiguous address space
since you can only register one area per umem. Physically, it can be
located scattered around the whole memory, but you do not have to
worry about that. posix_memalign() only aligns the allocated area in
virtual memory. It does not guarantee anything else.

/Magnus

> -Ratheesh

Re: AF_XDP and dma

[ratheesh kannoth]

On Fri, Apr 9, 2021 at 3:13 PM Magnus Karlsson
<magnus.karlsson@gmail.com> wrote:
> The short answer is that you do not have to worry about that at all in
> user space.
i am interested in kernel space, where i have to configure these
memory address (physical) in HW

>>>DMA is taken care of by the kernel.
Agreed. For AF_XDP in driver mode (for zero copy mode), I need to
configure the DMA address.

> /Magnus

Re: AF_XDP and dma

[Magnus Karlsson]

On Fri, Apr 9, 2021 at 12:01 PM ratheesh kannoth <ratheesh.ksz@gmail.com> wrote:
>
> On Fri, Apr 9, 2021 at 3:13 PM Magnus Karlsson
> <magnus.karlsson@gmail.com> wrote:
> > The short answer is that you do not have to worry about that at all in
> > user space.
> i am interested in kernel space, where i have to configure these
> memory address (physical) in HW
>
> >>>DMA is taken care of by the kernel.
> Agreed. For AF_XDP in driver mode (for zero copy mode), I need to
> configure the DMA address.

Only if you are writing your own kernel driver for a NIC. Are you doing that?

> > /Magnus

Re: AF_XDP and dma

[ratheesh kannoth]

On Fri, Apr 9, 2021 at 3:37 PM Magnus Karlsson
<magnus.karlsson@gmail.com> wrote:
> Only if you are writing your own kernel driver for a NIC. Are you doing that?
Yes.

> > > /Magnus

Re: AF_XDP and dma

[Magnus Karlsson]

On Fri, Apr 9, 2021 at 12:12 PM ratheesh kannoth <ratheesh.ksz@gmail.com> wrote:
>
> On Fri, Apr 9, 2021 at 3:37 PM Magnus Karlsson
> <magnus.karlsson@gmail.com> wrote:
> > Only if you are writing your own kernel driver for a NIC. Are you doing that?
> Yes.

Then looking at user-space code will not help you :-). Take a look at
the code in driver/net/ethernet/intel/i40e/i40e_xsk.c in the Linux
kernel for an example on how to DMA map the umem.

> > > > /Magnus

Re: AF_XDP and dma

[ratheesh kannoth]

On Fri, Apr 9, 2021 at 3:50 PM Magnus Karlsson
<magnus.karlsson@gmail.com> wrote:
> Then looking at user-space code will not help you :-). Take a look at
> the code in driver/net/ethernet/intel/i40e/i40e_xsk.c in the Linux
> kernel for an example on how to DMA map the umem.

Thanks, i could see that it is pinning the pages and mapping.

Using Kernel DMA api, we can map the virtual memory. But how the NIC
can write/read to the memory if it is not physically contiguous. NIC
card only understand physical addresses. Correct me.

> > > > > /Magnus

Re: AF_XDP and dma

[Magnus Karlsson]

On Fri, Apr 9, 2021 at 12:50 PM ratheesh kannoth <ratheesh.ksz@gmail.com> wrote:
>
> On Fri, Apr 9, 2021 at 3:50 PM Magnus Karlsson
> <magnus.karlsson@gmail.com> wrote:
> > Then looking at user-space code will not help you :-). Take a look at
> > the code in driver/net/ethernet/intel/i40e/i40e_xsk.c in the Linux
> > kernel for an example on how to DMA map the umem.
>
> Thanks, i could see that it is pinning the pages and mapping.
>
> Using Kernel DMA api, we can map the virtual memory. But how the NIC
> can write/read to the memory if it is not physically contiguous. NIC
> card only understand physical addresses. Correct me.

In XDP/AF_XDP mode, the NIC only uses a single 4K page (on x86-64) per
packet so it will be physically contiguous by definition. Every single
page will be dma mapped separately and can reside anywhere. There is
no scatter-gather in XDP/AF_XDP mode.

> > > > > > /Magnus

Re: AF_XDP and dma

[ratheesh kannoth]

On Fri, Apr 9, 2021 at 4:36 PM Magnus Karlsson
<magnus.karlsson@gmail.com> wrote:

> In XDP/AF_XDP mode, the NIC only uses a single 4K page (on x86-64) per
> packet so it will be physically contiguous by definition.
How ? if userspace allocates memory, it would be only contiguous in
virtual address space.
 >Every single
> page will be dma mapped separately and can reside anywhere.
I agree.  My concern is not on total memory. it is about each chunk.

>There is
> no scatter-gather in XDP/AF_XDP mode.
i agree.  i am not talking about Scatter gather (nr_frags or frag_list)

> > > > > > > /Magnus

Re: AF_XDP and dma

[Magnus Karlsson]

On Fri, Apr 9, 2021 at 1:23 PM ratheesh kannoth <ratheesh.ksz@gmail.com> wrote:
>
> On Fri, Apr 9, 2021 at 4:36 PM Magnus Karlsson
> <magnus.karlsson@gmail.com> wrote:
>
> > In XDP/AF_XDP mode, the NIC only uses a single 4K page (on x86-64) per
> > packet so it will be physically contiguous by definition.
> How ? if userspace allocates memory, it would be only contiguous in
> virtual address space.

Physically it only has to be contiguous in a 4K page, since we are
only using a single 4K page for each packet. And a single page is
always contiguous in both virtual and physical memory by definition.
The umem is broken up into pages and these are mapped separately. The
whole umem does not have to be physically contiguous. The NIC only
cares about every single frame (and only one frame per packet is
allowed with XDP for now) being in a physically contiguous region and
that we satisfy by limiting the size of a packet to one page.

>  >Every single
> > page will be dma mapped separately and can reside anywhere.
> I agree.  My concern is not on total memory. it is about each chunk.
>
> >There is
> > no scatter-gather in XDP/AF_XDP mode.
> i agree.  i am not talking about Scatter gather (nr_frags or frag_list)
>
> > > > > > > > /Magnus