AF_XDP integration with FDio VPP? (Was: Questions about XDP)

AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Jesper Dangaard Brouer]

Bringing these questions to the xdp-newbies list, where they belong.
Answers inlined below.

On Tue, 20 Aug 2019 21:17:57 +0200 Július Milan <Julius.Milan@pantheon.tech>
> 
> I am writing AF_XDP driver for FDio VPP. I have 2 questions.
>

That sounds excellent.  I was hoping someone would do this for FDio VPP.
Do notice that DPDK now also got AF_XDP support.  IHMO it makes a lot
of sense to implement AF_XDP for FDio, and avoid the DPDK dependency.
(AFAIK FDio already got other back-ends than DPDK).
 

> 1 - I created a simple driver according to sample in kernel. I load my XDP
> program and pin the maps.
> 
>   Then in user application I create a socket, mmap the memory and
> push it to xskmap in program. All fine yet.
> 
>   Then I start another instance of user application and do the
> same, create socket, mmap the memory and trying to
> 
>   push it somewhere else into the map. But I got  errno: 16
> "Device or resource busy" when trying to bind.
> 
>   I guess the memory can’t be mmaped 2 times, but should be
> shared, is that correct?

I'm cc'ing the AF_XDP experts, as I'm not sure myself.  I mostly deal
with the in-kernel XDP path.  (AF_XDP is essentially kernel bypass :-O)

 
>   If so, I am wondering how to solve this nicely.
> 
>   Can I store the value of first socket (that created the mmaped
> memory) in some special map in my XDP program to avoid complicated
> inter-process communication?
> 
>   And what happens if this first socket is closed while any other
> sockets are still alive (using its shared mmaped memory)?
> 
>   What would you recommend? Maybe you have some sample.

We just added a sample (by Eelco Cc'ed) into XDP-tutorial:
 https://github.com/xdp-project/xdp-tutorial/tree/master/advanced03-AF_XDP  

At-least read the README.org file... to get over the common gotchas.

AFAIK the sample doesn't cover your use-case.  I guess, we/someone
should extend the sample, to illustrate how how multiple interfaces can
share the same UMEM.

The official documentation is:
 https://www.kernel.org/doc/html/latest/networking/af_xdp.html


>   Can I do also atomic operations? (I want it just for such rare
> cases as initialization of next socket, to check if there already is
> one, that mmaped the memory)
> 
> 
> 
> 2 – We want to do also some decap/encap on XDP layer, before
> redirecting it to the socket.
> 

Decap on XDP layer is an excellent use-case, that demonstrate
cooperation between XDP and AF_XDP kernel-bypass facility.


>   On RX way it is easy, I do what I want and redirect it to the
> socket, but can I achieve the same also on TX?
> 

(Yes, RX case is easy)

We don't have an XDP TX hook yet... but so many people have requested
this, that we should add this.

>   Can I catch the packet while TX in XDP and do something with it
> (encapsulate it) before sending it out?

Usually, we recommend people use the TC egress BPF hook to do the encap
in TX.  For the AF_XDP use-case, the TC hook isn't there... so that is
not an option.  Again an argument for an XDP-TX hook.  You, could
of-cause add the encap header in your AF_XDP userspace program, but I
do understand it would make architectural sense that in-kernel XDP
would act as a decap/encap layer.

 
>   If so what about performance?
> 

For AF_XDP RX-side is really really fast, even in copy-mode.

For AF_XDP TX-side in copy-mode, it is rather slow, as it allocates
SKBs etc.  We could optimize this further but we have not.  When
enabling AF_XDP zero-copy mode, the TX-side is also super fast.

Another hint, for AF_XDP TX-side, remember to "produce" several packets
before doing the sendmsg system call.  Thus, effectively doing bulking
on the TX-ring.


>
> By the way, great job with XDP ;)

Thanks! 

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Marek Zavodsky]

Hi Jesper,

Thanks for your reply.
I apologize, I'm new to kernel dev, so I may be missing some background.

Let's bring some more light into this. We are using kernel 5.0.0 and
used samples/bpf/xdpsock as start point.
I checked master, and example evolved (e.g. by adding cleanup
mechanisms), but in terms what I need of it, it looks equal (and even
more complicated, because now XDP attaching to interface is
interleaved with XSK allocation).
I built latest kernel, but it refused to boot, so I haven't had chance
yet to tray the latest.

I took the _user part and split it into two:
"loader" -  Executed once to setup environment and once to cleanup,
loads _kern.o, attaches it to interface and pin maps under
/sys/fs/bpf.
and
"worker" - Executed as many as required. Every instance loads maps
from /sys/fs/bpf, create one AF_XDP sock, update xsks record and start
listen/process packets from AF_XDP (in test scenario we are using
l2fwd because of write-back). I had to add missing cleanups there(
close(fd), munmap()). This should be vpp in final solution.
So far so good.
I'm unable to start more than one worker due to previously mentioned
error. First instance works properly, every other fails on bind
(lineno may not match due to local changes):
xdpsock_user.c:xsk_configure:595: Assertion failed: bind(sfd, (struct
sockaddr *)&sxdp, sizeof(sxdp)) == 0: errno: 16/"Device or resource
busy"

I modified it to allocate multiple sockets within one process, and I
was successful with shared umem:
num_socks = 0;
xsks[num_socks++] = xsk_configure(NULL);
for (; num_socks < opt_alloc; num_socks++)
        xsks[num_socks] = xsk_configure(xsks[0]->umem);


but got same behavior (first ok, second failed on bind) when tried non-shared:
num_socks = 0;
for (; num_socks < opt_alloc; num_socks++)
      xsks[num_socks] = xsk_configure(NULL);



And the TX processing... as a workaround we moved VLAN pop/push to
"worker" and XDP does only xsk-map redirects based on vlan-id, but it
violates the purpose. It there any estimate when we could expect
something on XDP TX front?

I guess changing opt_ifindex to xsk->fd in
bpf_set_link_xdp_fd(opt_ifindex, prog_fd, opt_xdp_flags); won't help,
as they are 2 different things, right? :)

One side question. I noticed that bpf_trace_printk creates sparse
entries in /sys/kernel/debug/tracing/trace.
When I run sample of 100 packets I may get 0 to many entries there.
It's a bit annoying to run "load test" just to verify I hit the
correct code path. Is it doing sampling? Can I tweak it somehow?
Any trick how to use tail -f on /sys/kernel/debug/tracing/trace?

Thanks,
marek


On Fri, Aug 23, 2019 at 12:43 PM Jesper Dangaard Brouer
<brouer@redhat.com> wrote:
>
>
> Bringing these questions to the xdp-newbies list, where they belong.
> Answers inlined below.
>
> On Tue, 20 Aug 2019 21:17:57 +0200 Július Milan <Julius.Milan@pantheon.tech>
> >
> > I am writing AF_XDP driver for FDio VPP. I have 2 questions.
> >
>
> That sounds excellent.  I was hoping someone would do this for FDio VPP.
> Do notice that DPDK now also got AF_XDP support.  IHMO it makes a lot
> of sense to implement AF_XDP for FDio, and avoid the DPDK dependency.
> (AFAIK FDio already got other back-ends than DPDK).
>
>
> > 1 - I created a simple driver according to sample in kernel. I load my XDP
> > program and pin the maps.
> >
> >   Then in user application I create a socket, mmap the memory and
> > push it to xskmap in program. All fine yet.
> >
> >   Then I start another instance of user application and do the
> > same, create socket, mmap the memory and trying to
> >
> >   push it somewhere else into the map. But I got  errno: 16
> > "Device or resource busy" when trying to bind.
> >
> >   I guess the memory can’t be mmaped 2 times, but should be
> > shared, is that correct?
>
> I'm cc'ing the AF_XDP experts, as I'm not sure myself.  I mostly deal
> with the in-kernel XDP path.  (AF_XDP is essentially kernel bypass :-O)
>
>
> >   If so, I am wondering how to solve this nicely.
> >
> >   Can I store the value of first socket (that created the mmaped
> > memory) in some special map in my XDP program to avoid complicated
> > inter-process communication?
> >
> >   And what happens if this first socket is closed while any other
> > sockets are still alive (using its shared mmaped memory)?
> >
> >   What would you recommend? Maybe you have some sample.
>
> We just added a sample (by Eelco Cc'ed) into XDP-tutorial:
>  https://github.com/xdp-project/xdp-tutorial/tree/master/advanced03-AF_XDP
>
> At-least read the README.org file... to get over the common gotchas.
>
> AFAIK the sample doesn't cover your use-case.  I guess, we/someone
> should extend the sample, to illustrate how how multiple interfaces can
> share the same UMEM.
>
> The official documentation is:
>  https://www.kernel.org/doc/html/latest/networking/af_xdp.html
>
>
> >   Can I do also atomic operations? (I want it just for such rare
> > cases as initialization of next socket, to check if there already is
> > one, that mmaped the memory)
> >
> >
> >
> > 2 – We want to do also some decap/encap on XDP layer, before
> > redirecting it to the socket.
> >
>
> Decap on XDP layer is an excellent use-case, that demonstrate
> cooperation between XDP and AF_XDP kernel-bypass facility.
>
>
> >   On RX way it is easy, I do what I want and redirect it to the
> > socket, but can I achieve the same also on TX?
> >
>
> (Yes, RX case is easy)
>
> We don't have an XDP TX hook yet... but so many people have requested
> this, that we should add this.
>
> >   Can I catch the packet while TX in XDP and do something with it
> > (encapsulate it) before sending it out?
>
> Usually, we recommend people use the TC egress BPF hook to do the encap
> in TX.  For the AF_XDP use-case, the TC hook isn't there... so that is
> not an option.  Again an argument for an XDP-TX hook.  You, could
> of-cause add the encap header in your AF_XDP userspace program, but I
> do understand it would make architectural sense that in-kernel XDP
> would act as a decap/encap layer.
>
>
> >   If so what about performance?
> >
>
> For AF_XDP RX-side is really really fast, even in copy-mode.
>
> For AF_XDP TX-side in copy-mode, it is rather slow, as it allocates
> SKBs etc.  We could optimize this further but we have not.  When
> enabling AF_XDP zero-copy mode, the TX-side is also super fast.
>
> Another hint, for AF_XDP TX-side, remember to "produce" several packets
> before doing the sendmsg system call.  Thus, effectively doing bulking
> on the TX-ring.
>
>
> >
> > By the way, great job with XDP ;)
>
> Thanks!
>
> --
> Best regards,
>   Jesper Dangaard Brouer
>   MSc.CS, Principal Kernel Engineer at Red Hat
>   LinkedIn: http://www.linkedin.com/in/brouer

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[William Tu]

Hi Marek,

Answer some of your questions below, I leave the rest for others.

On Fri, Aug 23, 2019 at 3:38 AM Marek Závodský
<marek.zavodsky@pantheon.tech> wrote:
>
> Hi Jasper,
>
>
> Thanks for your reply.
>
> I apologize, I'm new to kernel dev, so I may be missing some background.
>
>
> Let's bring some more light into this. We are using kernel 5.0.0 and samples/bpf/xdpsock as an example.

Do you want to consider using AF_XDP API from libbpf?

The samples/bpf/xdpsock_user.c in 5.0.0 still not uses libbpf
https://elixir.bootlin.com/linux/v5.0/source/samples/bpf/xdpsock_user.c

kernel 5.1 xdpsock uses libbpf
https://elixir.bootlin.com/linux/v5.1/source/samples/bpf/xdpsock_user.c

>
> I checked master, and example evolved (e.g. by adding cleanup mechanisms), but in terms what I need of it, it looks equal (and even more complicated, because now XDP attaching to interface is interleaved with XSK allocation).
>
> I built latest kernel, but it refused to boot, so I haven't had chance yet to tray the latest.

Recently there are some fixes, I would suggest using the latest one.

>
>
> I took the _user part and split it into two:
>
> "loader" -  Executed once to setup environment and once to cleanup, loads _kern.o, attaches it to interface and pin maps under /sys/fs/bpf.
>
> and
>
> "worker" - Executed as many as required. Every instance loads maps from /sys/fs/bpf, create one AF_XDP sock, update xsks record and start listen/process packets from AF_XDP (in test scenario we are using l2fwd because of write-back). I had to add missing cleanups there( close(fd), munmap()). This should be vpp in final solution.
>
> So far so good.
>
> I'm unable to start more than one worker due to previously mentioned error. First instance works properly, every other fails on bind (lineno may not match due to local changes):
>
> xdpsock_user.c:xsk_configure:595: Assertion failed: bind(sfd, (struct sockaddr *)&sxdp, sizeof(sxdp)) == 0: errno: 16/"Device or resource busy"
>
>
I don't think you can have multiple threads binding one XSK, see
xsk_bind() in kernel source.
For AF_XDP in OVS, we create multiple XSKs, non-shared umem and each
has its thread.

> I modified it to allocate multiple sockets within one process, and I was successful with shared umem:
>
> num_socks = 0;
>
> xsks[num_socks++] = xsk_configure(NULL);
> for (; num_socks < opt_alloc; num_socks++)
>         xsks[num_socks] = xsk_configure(xsks[0]->umem);
>
>
> but got same behavior (first ok, second failed on bind) when tried non-shared:
>
> num_socks = 0;
>
> for (; num_socks < opt_alloc; num_socks++)
>
>       xsks[num_socks] = xsk_configure(NULL);
>
I never try shared-umem, I would suggest start with non-shared case.

Regards,
William
>
> And the TX processing... as a workaround we moved VLAN pop/push to "worker" and XDP does only xsk-map redirects based on vlan-id, but it violates the design. It there any estimate when we could expect something on XDP TX front? I can try BPF TC TX meantime.
>
>
> I guess changing opt_ifindex to xsk->fd in bpf_set_link_xdp_fd(opt_ifindex, prog_fd, opt_xdp_flags);
>
> won't help :)
>
>
> One side question. I noticed that bpf_trace_printk creates sparse entries in /sys/kernel/debug/tracing/trace.
>
> When I run sample of 100 packets I may get 0 to many entries there. I't a bit annoying to run "load test" just to verify I hit the correct code path. Is it doing sampling? Can I tweak it somehow?
>
>
> Thanks,
>
> marek
>
> ________________________________
> From: Jesper Dangaard Brouer <brouer@redhat.com>
> Sent: Friday, August 23, 2019 10:22:24 AM
> To: brouer@redhat.com; Július Milan; Marek Závodský
> Cc: xdp-newbies@vger.kernel.org; Karlsson, Magnus; Björn Töpel; Eelco Chaudron; Thomas F Herbert; William Tu
> Subject: AF_XDP integration with FDio VPP? (Was: Questions about XDP)
>
>
> Bringing these questions to the xdp-newbies list, where they belong.
> Answers inlined below.
>
> On Tue, 20 Aug 2019 21:17:57 +0200 Július Milan <Julius.Milan@pantheon.tech>
> >
> > I am writing AF_XDP driver for FDio VPP. I have 2 questions.
> >
>
> That sounds excellent.  I was hoping someone would do this for FDio VPP.
> Do notice that DPDK now also got AF_XDP support.  IHMO it makes a lot
> of sense to implement AF_XDP for FDio, and avoid the DPDK dependency.
> (AFAIK FDio already got other back-ends than DPDK).
>
>
> > 1 - I created a simple driver according to sample in kernel. I load my XDP
> > program and pin the maps.
> >
> >   Then in user application I create a socket, mmap the memory and
> > push it to xskmap in program. All fine yet.
> >
> >   Then I start another instance of user application and do the
> > same, create socket, mmap the memory and trying to
> >
> >   push it somewhere else into the map. But I got  errno: 16
> > "Device or resource busy" when trying to bind.
> >
> >   I guess the memory can’t be mmaped 2 times, but should be
> > shared, is that correct?
>
> I'm cc'ing the AF_XDP experts, as I'm not sure myself.  I mostly deal
> with the in-kernel XDP path.  (AF_XDP is essentially kernel bypass :-O)
>
>
> >   If so, I am wondering how to solve this nicely.
> >
> >   Can I store the value of first socket (that created the mmaped
> > memory) in some special map in my XDP program to avoid complicated
> > inter-process communication?
> >
> >   And what happens if this first socket is closed while any other
> > sockets are still alive (using its shared mmaped memory)?
> >
> >   What would you recommend? Maybe you have some sample.
>
> We just added a sample (by Eelco Cc'ed) into XDP-tutorial:
>  https://github.com/xdp-project/xdp-tutorial/tree/master/advanced03-AF_XDP
>
> At-least read the README.org file... to get over the common gotchas.
>
> AFAIK the sample doesn't cover your use-case.  I guess, we/someone
> should extend the sample, to illustrate how how multiple interfaces can
> share the same UMEM.
>
> The official documentation is:
>  https://www.kernel.org/doc/html/latest/networking/af_xdp.html
>
>
> >   Can I do also atomic operations? (I want it just for such rare
> > cases as initialization of next socket, to check if there already is
> > one, that mmaped the memory)
> >
> >
> >
> > 2 – We want to do also some decap/encap on XDP layer, before
> > redirecting it to the socket.
> >
>
> Decap on XDP layer is an excellent use-case, that demonstrate
> cooperation between XDP and AF_XDP kernel-bypass facility.
>
>
> >   On RX way it is easy, I do what I want and redirect it to the
> > socket, but can I achieve the same also on TX?
> >
>
> (Yes, RX case is easy)
>
> We don't have an XDP TX hook yet... but so many people have requested
> this, that we should add this.
>
> >   Can I catch the packet while TX in XDP and do something with it
> > (encapsulate it) before sending it out?
>
> Usually, we recommend people use the TC egress BPF hook to do the encap
> in TX.  For the AF_XDP use-case, the TC hook isn't there... so that is
> not an option.  Again an argument for an XDP-TX hook.  You, could
> of-cause add the encap header in your AF_XDP userspace program, but I
> do understand it would make architectural sense that in-kernel XDP
> would act as a decap/encap layer.
>
>
> >   If so what about performance?
> >
>
> For AF_XDP RX-side is really really fast, even in copy-mode.
>
> For AF_XDP TX-side in copy-mode, it is rather slow, as it allocates
> SKBs etc.  We could optimize this further but we have not.  When
> enabling AF_XDP zero-copy mode, the TX-side is also super fast.
>
> Another hint, for AF_XDP TX-side, remember to "produce" several packets
> before doing the sendmsg system call.  Thus, effectively doing bulking
> on the TX-ring.
>
>
> >
> > By the way, great job with XDP ;)
>
> Thanks!
>
> --
> Best regards,
>   Jesper Dangaard Brouer
>   MSc.CS, Principal Kernel Engineer at Red Hat
>   LinkedIn: http://www.linkedin.com/in/brouer

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[William Tu]

On Fri, Aug 23, 2019 at 1:22 AM Jesper Dangaard Brouer
<brouer@redhat.com> wrote:
>
>
> Bringing these questions to the xdp-newbies list, where they belong.
> Answers inlined below.
>
> On Tue, 20 Aug 2019 21:17:57 +0200 Július Milan <Julius.Milan@pantheon.tech>
> >
> > I am writing AF_XDP driver for FDio VPP. I have 2 questions.
> >
>
> That sounds excellent.  I was hoping someone would do this for FDio VPP.
> Do notice that DPDK now also got AF_XDP support.  IHMO it makes a lot
> of sense to implement AF_XDP for FDio, and avoid the DPDK dependency.
> (AFAIK FDio already got other back-ends than DPDK).
>
Hi Marek,

But isn't most of the VPP deployment use DPDK as back-ends?
If that's the case, why not use DPDK's AF_XDP implementation?

Regards,
William

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Björn Töpel]

On 2019-08-23 15:15, William Tu wrote:
> But isn't most of the VPP deployment use DPDK as back-ends?
> If that's the case, why not use DPDK's AF_XDP implementation?

I'd say the other way around, but I might be wrong. (In fact, VPP comes
with their own userspace drivers as well.)

VPP and DPDK has many overlapping components (mempools, drivers), and 
AFAIK (last time I checked) VPP pretty much only used the PMDs from DPDK.

So, from my perspective, a standalone AF_XDP implementation makes much
sense!


Cheers,
Björn

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Björn Töpel]

On 2019-08-23 15:07, William Tu wrote:
> Hi Marek,
> 
> Answer some of your questions below, I leave the rest for others.
> 
> On Fri, Aug 23, 2019 at 3:38 AM Marek Závodský
> <marek.zavodsky@pantheon.tech> wrote:
>>
>> Hi Jasper,
>>
>>
>> Thanks for your reply.
>>
>> I apologize, I'm new to kernel dev, so I may be missing some background.
>>
>>
>> Let's bring some more light into this. We are using kernel 5.0.0 and samples/bpf/xdpsock as an example.
> 
> Do you want to consider using AF_XDP API from libbpf?
> 
> The samples/bpf/xdpsock_user.c in 5.0.0 still not uses libbpf
> https://elixir.bootlin.com/linux/v5.0/source/samples/bpf/xdpsock_user.c
> 
> kernel 5.1 xdpsock uses libbpf
> https://elixir.bootlin.com/linux/v5.1/source/samples/bpf/xdpsock_user.c
> 
>>
>> I checked master, and example evolved (e.g. by adding cleanup mechanisms), but in terms what I need of it, it looks equal (and even more complicated, because now XDP attaching to interface is interleaved with XSK allocation).
>>
>> I built latest kernel, but it refused to boot, so I haven't had chance yet to tray the latest.
> 
> Recently there are some fixes, I would suggest using the latest one.
> 
>>
>>
>> I took the _user part and split it into two:
>>
>> "loader" -  Executed once to setup environment and once to cleanup, loads _kern.o, attaches it to interface and pin maps under /sys/fs/bpf.
>>
>> and
>>
>> "worker" - Executed as many as required. Every instance loads maps from /sys/fs/bpf, create one AF_XDP sock, update xsks record and start listen/process packets from AF_XDP (in test scenario we are using l2fwd because of write-back). I had to add missing cleanups there( close(fd), munmap()). This should be vpp in final solution.
>>
>> So far so good.
>>
>> I'm unable to start more than one worker due to previously mentioned error. First instance works properly, every other fails on bind (lineno may not match due to local changes):
>>
>> xdpsock_user.c:xsk_configure:595: Assertion failed: bind(sfd, (struct sockaddr *)&sxdp, sizeof(sxdp)) == 0: errno: 16/"Device or resource busy"
>>
>>
> I don't think you can have multiple threads binding one XSK, see
> xsk_bind() in kernel source.
> For AF_XDP in OVS, we create multiple XSKs, non-shared umem and each
> has its thread.
> 
>> I modified it to allocate multiple sockets within one process, and I was successful with shared umem:
>>
>> num_socks = 0;
>>
>> xsks[num_socks++] = xsk_configure(NULL);
>> for (; num_socks < opt_alloc; num_socks++)
>>          xsks[num_socks] = xsk_configure(xsks[0]->umem);
>>
>>
>> but got same behavior (first ok, second failed on bind) when tried non-shared:
>>
>> num_socks = 0;
>>
>> for (; num_socks < opt_alloc; num_socks++)
>>
>>        xsks[num_socks] = xsk_configure(NULL);
>>
> I never try shared-umem, I would suggest start with non-shared case.
>

William did a much better job, than I would do, at answering the
questions. Thank you! +1 to all replies.


Cheers,
Björn


> Regards,
> William
>>
>> And the TX processing... as a workaround we moved VLAN pop/push to "worker" and XDP does only xsk-map redirects based on vlan-id, but it violates the design. It there any estimate when we could expect something on XDP TX front? I can try BPF TC TX meantime.
>>
>>
>> I guess changing opt_ifindex to xsk->fd in bpf_set_link_xdp_fd(opt_ifindex, prog_fd, opt_xdp_flags);
>>
>> won't help :)
>>
>>
>> One side question. I noticed that bpf_trace_printk creates sparse entries in /sys/kernel/debug/tracing/trace.
>>
>> When I run sample of 100 packets I may get 0 to many entries there. I't a bit annoying to run "load test" just to verify I hit the correct code path. Is it doing sampling? Can I tweak it somehow?
>>
>>
>> Thanks,
>>
>> marek
>>
>> ________________________________
>> From: Jesper Dangaard Brouer <brouer@redhat.com>
>> Sent: Friday, August 23, 2019 10:22:24 AM
>> To: brouer@redhat.com; Július Milan; Marek Závodský
>> Cc: xdp-newbies@vger.kernel.org; Karlsson, Magnus; Björn Töpel; Eelco Chaudron; Thomas F Herbert; William Tu
>> Subject: AF_XDP integration with FDio VPP? (Was: Questions about XDP)
>>
>>
>> Bringing these questions to the xdp-newbies list, where they belong.
>> Answers inlined below.
>>
>> On Tue, 20 Aug 2019 21:17:57 +0200 Július Milan <Julius.Milan@pantheon.tech>
>>>
>>> I am writing AF_XDP driver for FDio VPP. I have 2 questions.
>>>
>>
>> That sounds excellent.  I was hoping someone would do this for FDio VPP.
>> Do notice that DPDK now also got AF_XDP support.  IHMO it makes a lot
>> of sense to implement AF_XDP for FDio, and avoid the DPDK dependency.
>> (AFAIK FDio already got other back-ends than DPDK).
>>
>>
>>> 1 - I created a simple driver according to sample in kernel. I load my XDP
>>> program and pin the maps.
>>>
>>>    Then in user application I create a socket, mmap the memory and
>>> push it to xskmap in program. All fine yet.
>>>
>>>    Then I start another instance of user application and do the
>>> same, create socket, mmap the memory and trying to
>>>
>>>    push it somewhere else into the map. But I got  errno: 16
>>> "Device or resource busy" when trying to bind.
>>>
>>>    I guess the memory can’t be mmaped 2 times, but should be
>>> shared, is that correct?
>>
>> I'm cc'ing the AF_XDP experts, as I'm not sure myself.  I mostly deal
>> with the in-kernel XDP path.  (AF_XDP is essentially kernel bypass :-O)
>>
>>
>>>    If so, I am wondering how to solve this nicely.
>>>
>>>    Can I store the value of first socket (that created the mmaped
>>> memory) in some special map in my XDP program to avoid complicated
>>> inter-process communication?
>>>
>>>    And what happens if this first socket is closed while any other
>>> sockets are still alive (using its shared mmaped memory)?
>>>
>>>    What would you recommend? Maybe you have some sample.
>>
>> We just added a sample (by Eelco Cc'ed) into XDP-tutorial:
>>   https://github.com/xdp-project/xdp-tutorial/tree/master/advanced03-AF_XDP
>>
>> At-least read the README.org file... to get over the common gotchas.
>>
>> AFAIK the sample doesn't cover your use-case.  I guess, we/someone
>> should extend the sample, to illustrate how how multiple interfaces can
>> share the same UMEM.
>>
>> The official documentation is:
>>   https://www.kernel.org/doc/html/latest/networking/af_xdp.html
>>
>>
>>>    Can I do also atomic operations? (I want it just for such rare
>>> cases as initialization of next socket, to check if there already is
>>> one, that mmaped the memory)
>>>
>>>
>>>
>>> 2 – We want to do also some decap/encap on XDP layer, before
>>> redirecting it to the socket.
>>>
>>
>> Decap on XDP layer is an excellent use-case, that demonstrate
>> cooperation between XDP and AF_XDP kernel-bypass facility.
>>
>>
>>>    On RX way it is easy, I do what I want and redirect it to the
>>> socket, but can I achieve the same also on TX?
>>>
>>
>> (Yes, RX case is easy)
>>
>> We don't have an XDP TX hook yet... but so many people have requested
>> this, that we should add this.
>>
>>>    Can I catch the packet while TX in XDP and do something with it
>>> (encapsulate it) before sending it out?
>>
>> Usually, we recommend people use the TC egress BPF hook to do the encap
>> in TX.  For the AF_XDP use-case, the TC hook isn't there... so that is
>> not an option.  Again an argument for an XDP-TX hook.  You, could
>> of-cause add the encap header in your AF_XDP userspace program, but I
>> do understand it would make architectural sense that in-kernel XDP
>> would act as a decap/encap layer.
>>
>>
>>>    If so what about performance?
>>>
>>
>> For AF_XDP RX-side is really really fast, even in copy-mode.
>>
>> For AF_XDP TX-side in copy-mode, it is rather slow, as it allocates
>> SKBs etc.  We could optimize this further but we have not.  When
>> enabling AF_XDP zero-copy mode, the TX-side is also super fast.
>>
>> Another hint, for AF_XDP TX-side, remember to "produce" several packets
>> before doing the sendmsg system call.  Thus, effectively doing bulking
>> on the TX-ring.
>>
>>
>>>
>>> By the way, great job with XDP ;)
>>
>> Thanks!
>>
>> --
>> Best regards,
>>    Jesper Dangaard Brouer
>>    MSc.CS, Principal Kernel Engineer at Red Hat
>>    LinkedIn: http://www.linkedin.com/in/brouer

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Marek Zavodsky]

Hi Guys,

Thanks four your insight.

@William, I'll definitely take a look at
https://github.com/openvswitch/ovs/blob/05629ed271a67c737227a92f35bcff199648d604/lib/netdev-afxdp.c

I'll give a try to latest implementation using libbpf.
Shared umem is not preferred case, but it was only option where I was
able to go past one successful bind so far.

@VPP vs DPDK
I do not knw details of original project, but there were multiple
layest of VPP running on single node and the performance wasn't as
expected. Therefore we are exploring XDP as lightweigt alternative.

marek

Regards,
marek

On Fri, Aug 23, 2019 at 2:23 PM Marek Zavodsky <marek.zavodsky@gmail.com> wrote:
>
> Hi Jesper,
>
> Thanks for your reply.
> I apologize, I'm new to kernel dev, so I may be missing some background.
>
> Let's bring some more light into this. We are using kernel 5.0.0 and
> used samples/bpf/xdpsock as start point.
> I checked master, and example evolved (e.g. by adding cleanup
> mechanisms), but in terms what I need of it, it looks equal (and even
> more complicated, because now XDP attaching to interface is
> interleaved with XSK allocation).
> I built latest kernel, but it refused to boot, so I haven't had chance
> yet to tray the latest.
>
> I took the _user part and split it into two:
> "loader" -  Executed once to setup environment and once to cleanup,
> loads _kern.o, attaches it to interface and pin maps under
> /sys/fs/bpf.
> and
> "worker" - Executed as many as required. Every instance loads maps
> from /sys/fs/bpf, create one AF_XDP sock, update xsks record and start
> listen/process packets from AF_XDP (in test scenario we are using
> l2fwd because of write-back). I had to add missing cleanups there(
> close(fd), munmap()). This should be vpp in final solution.
> So far so good.
> I'm unable to start more than one worker due to previously mentioned
> error. First instance works properly, every other fails on bind
> (lineno may not match due to local changes):
> xdpsock_user.c:xsk_configure:595: Assertion failed: bind(sfd, (struct
> sockaddr *)&sxdp, sizeof(sxdp)) == 0: errno: 16/"Device or resource
> busy"
>
> I modified it to allocate multiple sockets within one process, and I
> was successful with shared umem:
> num_socks = 0;
> xsks[num_socks++] = xsk_configure(NULL);
> for (; num_socks < opt_alloc; num_socks++)
>         xsks[num_socks] = xsk_configure(xsks[0]->umem);
>
>
> but got same behavior (first ok, second failed on bind) when tried non-shared:
> num_socks = 0;
> for (; num_socks < opt_alloc; num_socks++)
>       xsks[num_socks] = xsk_configure(NULL);
>
>
>
> And the TX processing... as a workaround we moved VLAN pop/push to
> "worker" and XDP does only xsk-map redirects based on vlan-id, but it
> violates the purpose. It there any estimate when we could expect
> something on XDP TX front?
>
> I guess changing opt_ifindex to xsk->fd in
> bpf_set_link_xdp_fd(opt_ifindex, prog_fd, opt_xdp_flags); won't help,
> as they are 2 different things, right? :)
>
> One side question. I noticed that bpf_trace_printk creates sparse
> entries in /sys/kernel/debug/tracing/trace.
> When I run sample of 100 packets I may get 0 to many entries there.
> It's a bit annoying to run "load test" just to verify I hit the
> correct code path. Is it doing sampling? Can I tweak it somehow?
> Any trick how to use tail -f on /sys/kernel/debug/tracing/trace?
>
> Thanks,
> marek
>
>
> On Fri, Aug 23, 2019 at 12:43 PM Jesper Dangaard Brouer
> <brouer@redhat.com> wrote:
> >
> >
> > Bringing these questions to the xdp-newbies list, where they belong.
> > Answers inlined below.
> >
> > On Tue, 20 Aug 2019 21:17:57 +0200 Július Milan <Julius.Milan@pantheon.tech>
> > >
> > > I am writing AF_XDP driver for FDio VPP. I have 2 questions.
> > >
> >
> > That sounds excellent.  I was hoping someone would do this for FDio VPP.
> > Do notice that DPDK now also got AF_XDP support.  IHMO it makes a lot
> > of sense to implement AF_XDP for FDio, and avoid the DPDK dependency.
> > (AFAIK FDio already got other back-ends than DPDK).
> >
> >
> > > 1 - I created a simple driver according to sample in kernel. I load my XDP
> > > program and pin the maps.
> > >
> > >   Then in user application I create a socket, mmap the memory and
> > > push it to xskmap in program. All fine yet.
> > >
> > >   Then I start another instance of user application and do the
> > > same, create socket, mmap the memory and trying to
> > >
> > >   push it somewhere else into the map. But I got  errno: 16
> > > "Device or resource busy" when trying to bind.
> > >
> > >   I guess the memory can’t be mmaped 2 times, but should be
> > > shared, is that correct?
> >
> > I'm cc'ing the AF_XDP experts, as I'm not sure myself.  I mostly deal
> > with the in-kernel XDP path.  (AF_XDP is essentially kernel bypass :-O)
> >
> >
> > >   If so, I am wondering how to solve this nicely.
> > >
> > >   Can I store the value of first socket (that created the mmaped
> > > memory) in some special map in my XDP program to avoid complicated
> > > inter-process communication?
> > >
> > >   And what happens if this first socket is closed while any other
> > > sockets are still alive (using its shared mmaped memory)?
> > >
> > >   What would you recommend? Maybe you have some sample.
> >
> > We just added a sample (by Eelco Cc'ed) into XDP-tutorial:
> >  https://github.com/xdp-project/xdp-tutorial/tree/master/advanced03-AF_XDP
> >
> > At-least read the README.org file... to get over the common gotchas.
> >
> > AFAIK the sample doesn't cover your use-case.  I guess, we/someone
> > should extend the sample, to illustrate how how multiple interfaces can
> > share the same UMEM.
> >
> > The official documentation is:
> >  https://www.kernel.org/doc/html/latest/networking/af_xdp.html
> >
> >
> > >   Can I do also atomic operations? (I want it just for such rare
> > > cases as initialization of next socket, to check if there already is
> > > one, that mmaped the memory)
> > >
> > >
> > >
> > > 2 – We want to do also some decap/encap on XDP layer, before
> > > redirecting it to the socket.
> > >
> >
> > Decap on XDP layer is an excellent use-case, that demonstrate
> > cooperation between XDP and AF_XDP kernel-bypass facility.
> >
> >
> > >   On RX way it is easy, I do what I want and redirect it to the
> > > socket, but can I achieve the same also on TX?
> > >
> >
> > (Yes, RX case is easy)
> >
> > We don't have an XDP TX hook yet... but so many people have requested
> > this, that we should add this.
> >
> > >   Can I catch the packet while TX in XDP and do something with it
> > > (encapsulate it) before sending it out?
> >
> > Usually, we recommend people use the TC egress BPF hook to do the encap
> > in TX.  For the AF_XDP use-case, the TC hook isn't there... so that is
> > not an option.  Again an argument for an XDP-TX hook.  You, could
> > of-cause add the encap header in your AF_XDP userspace program, but I
> > do understand it would make architectural sense that in-kernel XDP
> > would act as a decap/encap layer.
> >
> >
> > >   If so what about performance?
> > >
> >
> > For AF_XDP RX-side is really really fast, even in copy-mode.
> >
> > For AF_XDP TX-side in copy-mode, it is rather slow, as it allocates
> > SKBs etc.  We could optimize this further but we have not.  When
> > enabling AF_XDP zero-copy mode, the TX-side is also super fast.
> >
> > Another hint, for AF_XDP TX-side, remember to "produce" several packets
> > before doing the sendmsg system call.  Thus, effectively doing bulking
> > on the TX-ring.
> >
> >
> > >
> > > By the way, great job with XDP ;)
> >
> > Thanks!
> >
> > --
> > Best regards,
> >   Jesper Dangaard Brouer
> >   MSc.CS, Principal Kernel Engineer at Red Hat
> >   LinkedIn: http://www.linkedin.com/in/brouer

RE: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Július Milan]

Many thanks guys, very appretiated.

Going to take a look at OVS implementation, but I would like to ensure something before.

>> I took the _user part and split it into two:
>>
>> "loader" -  Executed once to setup environment and once to cleanup, loads _kern.o, attaches it to interface and pin maps under /sys/fs/bpf.
>>
>> and
>>
>> "worker" - Executed as many as required. Every instance loads maps from /sys/fs/bpf, create one AF_XDP sock, update xsks record and start listen/process packets from AF_XDP (in test scenario we are using l2fwd because of write-back). I had to add missing cleanups there( close(fd), munmap()). This should be vpp in final solution.
>>
>> So far so good.
>>
>> I'm unable to start more than one worker due to previously mentioned error. First instance works properly, every other fails on bind (lineno may not match due to local changes):
>>
>> xdpsock_user.c:xsk_configure:595: Assertion failed: bind(sfd, (struct sockaddr *)&sxdp, sizeof(sxdp)) == 0: errno: 16/"Device or resource busy"
>>
>>
> I don't think you can have multiple threads binding one XSK, see
> xsk_bind() in kernel source.
> For AF_XDP in OVS, we create multiple XSKs, non-shared umem and each 
> has its thread.

In OVS, can you bind two sockets with non-shared umem to the same interface?
Our goal is to have 2 or more processes (VPPs) listening on the same interface via XDP socket,
while XDP program decides where to redirect the packets at the moment.

Best Regards
Julius

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[William Tu]

On Fri, Aug 23, 2019 at 7:56 AM Július Milan <Julius.Milan@pantheon.tech> wrote:
>
> Many thanks guys, very appretiated.
>
> Going to take a look at OVS implementation, but I would like to ensure something before.
>
> >> I took the _user part and split it into two:
> >>
> >> "loader" -  Executed once to setup environment and once to cleanup, loads _kern.o, attaches it to interface and pin maps under /sys/fs/bpf.
> >>
> >> and
> >>
> >> "worker" - Executed as many as required. Every instance loads maps from /sys/fs/bpf, create one AF_XDP sock, update xsks record and start listen/process packets from AF_XDP (in test scenario we are using l2fwd because of write-back). I had to add missing cleanups there( close(fd), munmap()). This should be vpp in final solution.
> >>
> >> So far so good.
> >>
> >> I'm unable to start more than one worker due to previously mentioned error. First instance works properly, every other fails on bind (lineno may not match due to local changes):
> >>
> >> xdpsock_user.c:xsk_configure:595: Assertion failed: bind(sfd, (struct sockaddr *)&sxdp, sizeof(sxdp)) == 0: errno: 16/"Device or resource busy"
> >>
> >>
> > I don't think you can have multiple threads binding one XSK, see
> > xsk_bind() in kernel source.
> > For AF_XDP in OVS, we create multiple XSKs, non-shared umem and each
> > has its thread.
>
> In OVS, can you bind two sockets with non-shared umem to the same interface?

Yes, but to the different queue id on the same interface.
So each xsk with non-shared umem bind to distinct queue id of that interface.

> Our goal is to have 2 or more processes (VPPs) listening on the same interface via XDP socket,
> while XDP program decides where to redirect the packets at the moment.
Make sense.

Regards,
William

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Pavel Popa]

Hi all,

I'd like to join the questions too, specifically more oriented towards
AF_XDP devs.
I understand why the choice of SPSC ring queue design, however what
would be the cost of a MPSC design (Multiple Producer Single
Consumer)? In order to have a single UMEM available to the entire net
device (i.e. bound to all net device's hardware queues), imho this
would have a huge impact because now, among other things, one wouldn't
have to bother with ethtool. And do we really want to question the
convenience of "UMEM per hardware queue" vs "UMEM per net device"?
Btw, congrats for the great work guys.

Il giorno sab 24 ago 2019 alle ore 01:29 William Tu
<u9012063@gmail.com> ha scritto:
>
> On Fri, Aug 23, 2019 at 7:56 AM Július Milan <Julius.Milan@pantheon.tech> wrote:
> >
> > Many thanks guys, very appretiated.
> >
> > Going to take a look at OVS implementation, but I would like to ensure something before.
> >
> > >> I took the _user part and split it into two:
> > >>
> > >> "loader" -  Executed once to setup environment and once to cleanup, loads _kern.o, attaches it to interface and pin maps under /sys/fs/bpf.
> > >>
> > >> and
> > >>
> > >> "worker" - Executed as many as required. Every instance loads maps from /sys/fs/bpf, create one AF_XDP sock, update xsks record and start listen/process packets from AF_XDP (in test scenario we are using l2fwd because of write-back). I had to add missing cleanups there( close(fd), munmap()). This should be vpp in final solution.
> > >>
> > >> So far so good.
> > >>
> > >> I'm unable to start more than one worker due to previously mentioned error. First instance works properly, every other fails on bind (lineno may not match due to local changes):
> > >>
> > >> xdpsock_user.c:xsk_configure:595: Assertion failed: bind(sfd, (struct sockaddr *)&sxdp, sizeof(sxdp)) == 0: errno: 16/"Device or resource busy"
> > >>
> > >>
> > > I don't think you can have multiple threads binding one XSK, see
> > > xsk_bind() in kernel source.
> > > For AF_XDP in OVS, we create multiple XSKs, non-shared umem and each
> > > has its thread.
> >
> > In OVS, can you bind two sockets with non-shared umem to the same interface?
>
> Yes, but to the different queue id on the same interface.
> So each xsk with non-shared umem bind to distinct queue id of that interface.
>
> > Our goal is to have 2 or more processes (VPPs) listening on the same interface via XDP socket,
> > while XDP program decides where to redirect the packets at the moment.
> Make sense.
>
> Regards,
> William

RE: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Július Milan]

Hi all

Regarding 4K frame size constraint of AF_XDP, what does AF_XDP when receives frame bigger than 4K? Drops it or cut it or split it between more frames?
Thinking about what to do with them on TX side.

If you are interested in mentioned AF_XDP driver for vpp, here you can find it: https://gerrit.fd.io/r/c/vpp/+/21606
It is still work in progress, I submitted it for review just yesterday and rebased it on recently found code in review of Damjan.

To run it, you need to have compatible XDP program loaded on interface you want to attach to.
This separation is here because we want user to be able to create custom XDP programs independently.
We also consider automatic loading of some default XDP program (that just redirects) if none is loaded in time of interface creation, but that is of minor priority for us yet.
For loading you can use attached loader (it will be added for review soon as well). It just loads the program and pins its maps.
As XDP program, we use yet xdpsock_kern.c from kernel v5.0.0 sources (attached).
XDP program compatibility requirements (from vpp point of view) are:
having xskmap called "xsks_map" and pinned at /sys/fs/bpf/<ifname>/

I added support for multiple queues into my patch for vpp, as we discussed (1 NIC, n sockets on different queues), but I didn't test it yet. Other things should work fine.
Tested on kernel 5.0.0.

Július

> Many thanks guys, very appretiated.
>
> Going to take a look at OVS implementation, but I would like to ensure something before.
>
>>> I took the _user part and split it into two:
>>>
>>> "loader" -  Executed once to setup environment and once to cleanup, loads _kern.o, attaches it to interface and pin maps under /sys/fs/bpf.
>>>
>>> and
>>>
>>> "worker" - Executed as many as required. Every instance loads maps from /sys/fs/bpf, create one AF_XDP sock, update xsks record and start listen/process packets from AF_XDP (in test scenario we are using l2fwd because of write-back). I had to add missing cleanups there( close(fd), munmap()). This should be vpp in final solution.
>>>
>>> So far so good.
>>>
>>> I'm unable to start more than one worker due to previously mentioned error. First instance works properly, every other fails on bind (lineno may not match due to local changes):
>>>
>>> xdpsock_user.c:xsk_configure:595: Assertion failed: bind(sfd, (struct sockaddr *)&sxdp, sizeof(sxdp)) == 0: errno: 16/"Device or resource busy"
>>>
>>>
>> I don't think you can have multiple threads binding one XSK, see
>> xsk_bind() in kernel source.
>> For AF_XDP in OVS, we create multiple XSKs, non-shared umem and each 
>> has its thread.
>
> In OVS, can you bind two sockets with non-shared umem to the same interface?
> Our goal is to have 2 or more processes (VPPs) listening on the same interface via XDP socket, while XDP program decides where to redirect the packets at the moment.
>
> Best Regards
> Julius

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Björn Töpel]

On 2019-08-24 09:36, Pavel Popa wrote:
> I'd like to join the questions too, specifically more oriented towards
> AF_XDP devs.
> I understand why the choice of SPSC ring queue design, however what
> would be the cost of a MPSC design (Multiple Producer Single
> Consumer)? In order to have a single UMEM available to the entire net
> device (i.e. bound to all net device's hardware queues), imho this
> would have a huge impact because now, among other things, one wouldn't
> have to bother with ethtool. And do we really want to question the
> convenience of "UMEM per hardware queue" vs "UMEM per net device"?
> Btw, congrats for the great work guys.

Late on the ball...

A MPSC will definitely have a negative performance impact (let me see if
I can find some number from back when we started), but also open up
other issues like shared/multiple address spaces. We opted in for a
simple, performant model where filtering should be done in HW (zero-copy
mode).

What you're describing (SW based switching) might be a better fit for a
XDP + (say) tap with AF_XDP.

Most, if not all, users looking into AF_XDP do it for the zero-copy
performance, and going to the model you're describing will impact
performance too much.



Cheers,
Björn

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Björn Töpel]

On 2019-08-30 18:37, Július Milan wrote:
> Hi all
> 
> Regarding 4K frame size constraint of AF_XDP, what does AF_XDP when receives frame bigger than 4K? Drops it or cut it or split it between more frames?
> Thinking about what to do with them on TX side.
>

AF_XDP stands on XDP's shoulders, and currently XDP does not support
frames larger than a page (4k). This will most likely change for AF_XDP.


> If you are interested in mentioned AF_XDP driver for vpp, here you can find it: https://gerrit.fd.io/r/c/vpp/+/21606
> It is still work in progress, I submitted it for review just yesterday and rebased it on recently found code in review of Damjan.
>

Very cool! Will check it out!


Cheers,
Björn

> To run it, you need to have compatible XDP program loaded on interface you want to attach to.
> This separation is here because we want user to be able to create custom XDP programs independently.
> We also consider automatic loading of some default XDP program (that just redirects) if none is loaded in time of interface creation, but that is of minor priority for us yet.
> For loading you can use attached loader (it will be added for review soon as well). It just loads the program and pins its maps.
> As XDP program, we use yet xdpsock_kern.c from kernel v5.0.0 sources (attached).
> XDP program compatibility requirements (from vpp point of view) are:
> having xskmap called "xsks_map" and pinned at /sys/fs/bpf/<ifname>/
> 
> I added support for multiple queues into my patch for vpp, as we discussed (1 NIC, n sockets on different queues), but I didn't test it yet. Other things should work fine.
> Tested on kernel 5.0.0.
> 
> Július
> 
>> Many thanks guys, very appretiated.
>>
>> Going to take a look at OVS implementation, but I would like to ensure something before.
>>
>>>> I took the _user part and split it into two:
>>>>
>>>> "loader" -  Executed once to setup environment and once to cleanup, loads _kern.o, attaches it to interface and pin maps under /sys/fs/bpf.
>>>>
>>>> and
>>>>
>>>> "worker" - Executed as many as required. Every instance loads maps from /sys/fs/bpf, create one AF_XDP sock, update xsks record and start listen/process packets from AF_XDP (in test scenario we are using l2fwd because of write-back). I had to add missing cleanups there( close(fd), munmap()). This should be vpp in final solution.
>>>>
>>>> So far so good.
>>>>
>>>> I'm unable to start more than one worker due to previously mentioned error. First instance works properly, every other fails on bind (lineno may not match due to local changes):
>>>>
>>>> xdpsock_user.c:xsk_configure:595: Assertion failed: bind(sfd, (struct sockaddr *)&sxdp, sizeof(sxdp)) == 0: errno: 16/"Device or resource busy"
>>>>
>>>>
>>> I don't think you can have multiple threads binding one XSK, see
>>> xsk_bind() in kernel source.
>>> For AF_XDP in OVS, we create multiple XSKs, non-shared umem and each
>>> has its thread.
>>
>> In OVS, can you bind two sockets with non-shared umem to the same interface?
>> Our goal is to have 2 or more processes (VPPs) listening on the same interface via XDP socket, while XDP program decides where to redirect the packets at the moment.
>>
>> Best Regards
>> Julius
> 

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[William Tu]

On Fri, Aug 30, 2019 at 9:37 AM Július Milan <Julius.Milan@pantheon.tech> wrote:
>
> Hi all
>
> Regarding 4K frame size constraint of AF_XDP, what does AF_XDP when receives frame bigger than 4K? Drops it or cut it or split it between more frames?
> Thinking about what to do with them on TX side.
>
> If you are interested in mentioned AF_XDP driver for vpp, here you can find it: https://gerrit.fd.io/r/c/vpp/+/21606

Hi Július,

I took a look at the patch, I can passed compile but I haven't run it.
In the src/plugins/af_xdp/device.c and src/plugins/af_xdp/xsk_defs.h,
since you already link libbpf, you can remove a lot of existing code by
using AF_XDP APIs in libbpf (see xsk_ring_*, xsk_umem_*)

If you want to load your custom XDP program, you can enable
XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD and load XDP program

Regards,
William

RE: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Július Milan]

Hi

> On Fri, Aug 30, 2019 at 9:37 AM Július Milan <Julius.Milan@pantheon.tech> wrote:
>>
>> Hi all
>>
>> Regarding 4K frame size constraint of AF_XDP, what does AF_XDP when receives frame bigger than 4K? Drops it or cut it or split it between more frames?
>> Thinking about what to do with them on TX side.
>>
>> If you are interested in mentioned AF_XDP driver for vpp, here you can 
>> find it: https://gerrit.fd.io/r/c/vpp/+/21606
>
> Hi Július,
>
> I took a look at the patch, I can passed compile but I haven't run it.
> In the src/plugins/af_xdp/device.c and src/plugins/af_xdp/xsk_defs.h, since you already link libbpf, you can remove a lot of existing code by using AF_XDP APIs in libbpf (see xsk_ring_*, xsk_umem_*)
>
> If you want to load your custom XDP program, you can enable XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD and load XDP program

Thank you William, I updated it, but its still work in progress, I inspired by OVS implementation as well.

>
> Regards,
> William

I have next 2 questions:

1] When I use xsk_ring_prod__reserve and successive xsk_ring_prod__submit. Is it correct to submit also less than I reserved?
    In some cases I can't exactly determine how much to reserve in advance, since vpp buffers have different size than xdp frames.

2] Can I use hugepage backed memory for umem? If not, is it planned for future?
    Yet it does copy pakets from rx rings to vpp buffers, but speculating about straight zerocopy way.

Many thanks
Július

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Magnus Karlsson]

On Wed, Sep 18, 2019 at 8:14 PM Július Milan <Julius.Milan@pantheon.tech> wrote:
>
> Hi
>
> > On Fri, Aug 30, 2019 at 9:37 AM Július Milan <Julius.Milan@pantheon.tech> wrote:
> >>
> >> Hi all
> >>
> >> Regarding 4K frame size constraint of AF_XDP, what does AF_XDP when receives frame bigger than 4K? Drops it or cut it or split it between more frames?
> >> Thinking about what to do with them on TX side.
> >>
> >> If you are interested in mentioned AF_XDP driver for vpp, here you can
> >> find it: https://gerrit.fd.io/r/c/vpp/+/21606
> >
> > Hi Július,
> >
> > I took a look at the patch, I can passed compile but I haven't run it.
> > In the src/plugins/af_xdp/device.c and src/plugins/af_xdp/xsk_defs.h, since you already link libbpf, you can remove a lot of existing code by using AF_XDP APIs in libbpf (see xsk_ring_*, xsk_umem_*)
> >
> > If you want to load your custom XDP program, you can enable XSK_LIBBPF_FLAGS__INHIBIT_PROG_LOAD and load XDP program
>
> Thank you William, I updated it, but its still work in progress, I inspired by OVS implementation as well.
>
> >
> > Regards,
> > William
>
> I have next 2 questions:
>
> 1] When I use xsk_ring_prod__reserve and successive xsk_ring_prod__submit. Is it correct to submit also less than I reserved?
>     In some cases I can't exactly determine how much to reserve in advance, since vpp buffers have different size than xdp frames.

Let me see so I understand this correctly. Ponder you reserve 10 slots
and later submit 4. This means you have reserved 6 more than you need.
Do you want to "unreserve" these and give them back to the ring? This
is not supported by the interface today. Another way of solving this
(if this is your problem and I am understanding it correctly, that is)
is that you in the next iteration only reserve 10 - 6 = 4 slots
because you already have 6 slots available from the last iteration.
You could still submit 10 after this. But adding something like an
unreserve option would be easy as long as we made sure it only
affected local ring state. The global state seen in the shared
variables between user space and kernel would not be touched, as this
would affect performance negatively. Please let me know what you
think.

> 2] Can I use hugepage backed memory for umem? If not, is it planned for future?
>     Yet it does copy pakets from rx rings to vpp buffers, but speculating about straight zerocopy way.

Yes you can use huge pages today, but the internal AF_XDP code has not
been optimized to use huge pages, so you will not get the full benefit
from them today. Kevin Laatz, added to this mail, is working on
optimizing the AF_XDP code for huge pages. If you want to know more or
have some requirements, do not hesitate to contact him.

/Magnus

> Many thanks
> Július

RE: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Július Milan]

Many Thanks Magnus

>> I have next 2 questions:
>>
>> 1] When I use xsk_ring_prod__reserve and successive xsk_ring_prod__submit. Is it correct to submit also less than I reserved?
>>     In some cases I can't exactly determine how much to reserve in advance, since vpp buffers have different size than xdp frames.
>
> Let me see so I understand this correctly. Ponder you reserve 10 slots and later submit 4. This means you have reserved 6 more than you need.
> Do you want to "unreserve" these and give them back to the ring? This is not supported by the interface today. Another way of solving this (if this is your problem and I am understanding it correctly, that is) is that you in the next iteration only reserve 10 - 6 = 4 slots because you already have 6 slots available from the last iteration.
> You could still submit 10 after this. But adding something like an unreserve option would be easy as long as we made sure it only affected local ring state. The global state seen in the shared variables between user space and kernel would not be touched, as this would affect performance negatively. Please let me know what you think.
>
Yes, You understand it correctly, I implemented it the way you suggested, i.e. by marking index and count of reserved slots (not committed yet, but works well), thanks again.

>> 2] Can I use hugepage backed memory for umem? If not, is it planned for future?
>>     Yet it does copy pakets from rx rings to vpp buffers, but speculating about straight zerocopy way.
>
> Yes you can use huge pages today, but the internal AF_XDP code has not been optimized to use huge pages, so you will not get the full benefit from them today. Kevin Laatz, added to this mail, is working on optimizing the AF_XDP code for huge pages. If you want to know more or have some requirements, do not hesitate to contact him.
>
Kevin will the API for using hugepages change while optimization process significantly or can I already start to rewrite my vpp driver to use hugepages backed memory?
Also please let me know, when you consider AF_XDP code optimized to use huge pages.

William, if I may ask next question.
Does OVS implementation of af_xdp driver copy paket data from af_xdp mmaped ring buffers into OVS "buffers" (some structure to represent the packet in OVS) or is it zerocopy
in this manner, i.e. OVS "buffers" mempool is directly mmaped as ring and so no copy on RX is needed. Since in 2nd case it would be very valuable for me as inspiration.

> /Magnus
>

Thanks a lot,

Július

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Eelco Chaudron]

On 23 Sep 2019, at 11:00, Július Milan wrote:

> Many Thanks Magnus
>
>>> I have next 2 questions:
>>>
>>> 1] When I use xsk_ring_prod__reserve and successive 
>>> xsk_ring_prod__submit. Is it correct to submit also less than I 
>>> reserved?
>>>     In some cases I can't exactly determine how much to reserve in 
>>> advance, since vpp buffers have different size than xdp frames.
>>
>> Let me see so I understand this correctly. Ponder you reserve 10 
>> slots and later submit 4. This means you have reserved 6 more than 
>> you need.
>> Do you want to "unreserve" these and give them back to the ring? This 
>> is not supported by the interface today. Another way of solving this 
>> (if this is your problem and I am understanding it correctly, that 
>> is) is that you in the next iteration only reserve 10 - 6 = 4 slots 
>> because you already have 6 slots available from the last iteration.
>> You could still submit 10 after this. But adding something like an 
>> unreserve option would be easy as long as we made sure it only 
>> affected local ring state. The global state seen in the shared 
>> variables between user space and kernel would not be touched, as this 
>> would affect performance negatively. Please let me know what you 
>> think.
>>
> Yes, You understand it correctly, I implemented it the way you 
> suggested, i.e. by marking index and count of reserved slots (not 
> committed yet, but works well), thanks again.
>
>>> 2] Can I use hugepage backed memory for umem? If not, is it planned 
>>> for future?
>>>     Yet it does copy pakets from rx rings to vpp buffers, but 
>>> speculating about straight zerocopy way.
>>
>> Yes you can use huge pages today, but the internal AF_XDP code has 
>> not been optimized to use huge pages, so you will not get the full 
>> benefit from them today. Kevin Laatz, added to this mail, is working 
>> on optimizing the AF_XDP code for huge pages. If you want to know 
>> more or have some requirements, do not hesitate to contact him.
>>
> Kevin will the API for using hugepages change while optimization 
> process significantly or can I already start to rewrite my vpp driver 
> to use hugepages backed memory?
> Also please let me know, when you consider AF_XDP code optimized to 
> use huge pages.
>
> William, if I may ask next question.
> Does OVS implementation of af_xdp driver copy paket data from af_xdp 
> mmaped ring buffers into OVS "buffers" (some structure to represent 
> the packet in OVS) or is it zerocopy
> in this manner, i.e. OVS "buffers" mempool is directly mmaped as ring 
> and so no copy on RX is needed. Since in 2nd case it would be very 
> valuable for me as inspiration.

Currently, OVS uses the mmaped memory directly, however on egress, it is 
copying the memory to the egress interface it’s mmaped memory. 
Currently, OVS uses an AF_XDP memory pool per interface, so a further 
optimization could be to use a global memory pool so this extra copy is 
not needed.

>
>> /Magnus
>>
>
> Thanks a lot,
>
> Július

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[William Tu]

On Mon, Sep 23, 2019 at 6:01 AM Eelco Chaudron <echaudro@redhat.com> wrote:
>
>
>
> On 23 Sep 2019, at 11:00, Július Milan wrote:
>
> > Many Thanks Magnus
> >
> >>> I have next 2 questions:
> >>>
> >>> 1] When I use xsk_ring_prod__reserve and successive
> >>> xsk_ring_prod__submit. Is it correct to submit also less than I
> >>> reserved?
> >>>     In some cases I can't exactly determine how much to reserve in
> >>> advance, since vpp buffers have different size than xdp frames.
> >>
> >> Let me see so I understand this correctly. Ponder you reserve 10
> >> slots and later submit 4. This means you have reserved 6 more than
> >> you need.
> >> Do you want to "unreserve" these and give them back to the ring? This
> >> is not supported by the interface today. Another way of solving this
> >> (if this is your problem and I am understanding it correctly, that
> >> is) is that you in the next iteration only reserve 10 - 6 = 4 slots
> >> because you already have 6 slots available from the last iteration.
> >> You could still submit 10 after this. But adding something like an
> >> unreserve option would be easy as long as we made sure it only
> >> affected local ring state. The global state seen in the shared
> >> variables between user space and kernel would not be touched, as this
> >> would affect performance negatively. Please let me know what you
> >> think.
> >>
> > Yes, You understand it correctly, I implemented it the way you
> > suggested, i.e. by marking index and count of reserved slots (not
> > committed yet, but works well), thanks again.
> >
> >>> 2] Can I use hugepage backed memory for umem? If not, is it planned
> >>> for future?
> >>>     Yet it does copy pakets from rx rings to vpp buffers, but
> >>> speculating about straight zerocopy way.
> >>
> >> Yes you can use huge pages today, but the internal AF_XDP code has
> >> not been optimized to use huge pages, so you will not get the full
> >> benefit from them today. Kevin Laatz, added to this mail, is working
> >> on optimizing the AF_XDP code for huge pages. If you want to know
> >> more or have some requirements, do not hesitate to contact him.
> >>
> > Kevin will the API for using hugepages change while optimization
> > process significantly or can I already start to rewrite my vpp driver
> > to use hugepages backed memory?
> > Also please let me know, when you consider AF_XDP code optimized to
> > use huge pages.
> >
> > William, if I may ask next question.
> > Does OVS implementation of af_xdp driver copy paket data from af_xdp
> > mmaped ring buffers into OVS "buffers" (some structure to represent
> > the packet in OVS) or is it zerocopy
> > in this manner, i.e. OVS "buffers" mempool is directly mmaped as ring
> > and so no copy on RX is needed. Since in 2nd case it would be very
> > valuable for me as inspiration.
>
> Currently, OVS uses the mmaped memory directly, however on egress, it is
> copying the memory to the egress interface it’s mmaped memory.
> Currently, OVS uses an AF_XDP memory pool per interface, so a further
> optimization could be to use a global memory pool so this extra copy is
> not needed.
>
That's right, thanks Eelco!
For OVS, it is zerocopy in the RX.

William

RE: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Július Milan]

Hi Eelco

> Currently, OVS uses the mmaped memory directly, however on egress, it is copying the memory to the egress interface it’s mmaped memory. 
Great, thanks for making this clear to me.

> Currently, OVS uses an AF_XDP memory pool per interface, so a further optimization could be to use a global memory pool so this extra copy is not needed.
Is it even possible to make this further optimization? Since every interface has it's own non-shared umem, so from my point of view, at least one
copy for case as you described above (when RX interface is different then TX interface) is necessery. Or am I missing something?

Július

-----Original Message-----
From: Eelco Chaudron [mailto:echaudro@redhat.com] 
Sent: Monday, September 23, 2019 3:02 PM
To: Július Milan <Julius.Milan@pantheon.tech>
Cc: Magnus Karlsson <magnus.karlsson@gmail.com>; William Tu <u9012063@gmail.com>; Björn Töpel <bjorn.topel@intel.com>; Marek Závodský <marek.zavodsky@pantheon.tech>; Jesper Dangaard Brouer <brouer@redhat.com>; xdp-newbies@vger.kernel.org; Karlsson, Magnus <magnus.karlsson@intel.com>; Thomas F Herbert <therbert@redhat.com>; Kevin Laatz <kevin.laatz@intel.com>
Subject: Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)



On 23 Sep 2019, at 11:00, Július Milan wrote:

> Many Thanks Magnus
>
>>> I have next 2 questions:
>>>
>>> 1] When I use xsk_ring_prod__reserve and successive 
>>> xsk_ring_prod__submit. Is it correct to submit also less than I 
>>> reserved?
>>>     In some cases I can't exactly determine how much to reserve in 
>>> advance, since vpp buffers have different size than xdp frames.
>>
>> Let me see so I understand this correctly. Ponder you reserve 10 
>> slots and later submit 4. This means you have reserved 6 more than 
>> you need.
>> Do you want to "unreserve" these and give them back to the ring? This 
>> is not supported by the interface today. Another way of solving this 
>> (if this is your problem and I am understanding it correctly, that
>> is) is that you in the next iteration only reserve 10 - 6 = 4 slots 
>> because you already have 6 slots available from the last iteration.
>> You could still submit 10 after this. But adding something like an 
>> unreserve option would be easy as long as we made sure it only 
>> affected local ring state. The global state seen in the shared 
>> variables between user space and kernel would not be touched, as this 
>> would affect performance negatively. Please let me know what you 
>> think.
>>
> Yes, You understand it correctly, I implemented it the way you 
> suggested, i.e. by marking index and count of reserved slots (not 
> committed yet, but works well), thanks again.
>
>>> 2] Can I use hugepage backed memory for umem? If not, is it planned 
>>> for future?
>>>     Yet it does copy pakets from rx rings to vpp buffers, but 
>>> speculating about straight zerocopy way.
>>
>> Yes you can use huge pages today, but the internal AF_XDP code has 
>> not been optimized to use huge pages, so you will not get the full 
>> benefit from them today. Kevin Laatz, added to this mail, is working 
>> on optimizing the AF_XDP code for huge pages. If you want to know 
>> more or have some requirements, do not hesitate to contact him.
>>
> Kevin will the API for using hugepages change while optimization 
> process significantly or can I already start to rewrite my vpp driver 
> to use hugepages backed memory?
> Also please let me know, when you consider AF_XDP code optimized to 
> use huge pages.
>
> William, if I may ask next question.
> Does OVS implementation of af_xdp driver copy paket data from af_xdp 
> mmaped ring buffers into OVS "buffers" (some structure to represent 
> the packet in OVS) or is it zerocopy in this manner, i.e. OVS 
> "buffers" mempool is directly mmaped as ring and so no copy on RX is 
> needed. Since in 2nd case it would be very valuable for me as 
> inspiration.

Currently, OVS uses the mmaped memory directly, however on egress, it is copying the memory to the egress interface it’s mmaped memory. 
Currently, OVS uses an AF_XDP memory pool per interface, so a further optimization could be to use a global memory pool so this extra copy is not needed.

>
>> /Magnus
>>
>
> Thanks a lot,
>
> Július

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Eelco Chaudron]

On 25 Sep 2019, at 8:46, Július Milan wrote:

> Hi Eelco
>
>> Currently, OVS uses the mmaped memory directly, however on egress, it 
>> is copying the memory to the egress interface it’s mmaped memory.
> Great, thanks for making this clear to me.
>
>> Currently, OVS uses an AF_XDP memory pool per interface, so a further 
>> optimization could be to use a global memory pool so this extra copy 
>> is not needed.
> Is it even possible to make this further optimization? Since every 
> interface has it's own non-shared umem, so from my point of view, at 
> least one
> copy for case as you described above (when RX interface is different 
> then TX interface) is necessery. Or am I missing something?

Some one @Intel told me it would be possible to have one huge mempool 
that can be shared between interfaces. However I have not 
researched/tried it.
Maybe Magnus can confirm?


> Július
>
> -----Original Message-----
> From: Eelco Chaudron [mailto:echaudro@redhat.com]
> Sent: Monday, September 23, 2019 3:02 PM
> To: Július Milan <Julius.Milan@pantheon.tech>
> Cc: Magnus Karlsson <magnus.karlsson@gmail.com>; William Tu 
> <u9012063@gmail.com>; Björn Töpel <bjorn.topel@intel.com>; Marek 
> Závodský <marek.zavodsky@pantheon.tech>; Jesper Dangaard Brouer 
> <brouer@redhat.com>; xdp-newbies@vger.kernel.org; Karlsson, Magnus 
> <magnus.karlsson@intel.com>; Thomas F Herbert <therbert@redhat.com>; 
> Kevin Laatz <kevin.laatz@intel.com>
> Subject: Re: AF_XDP integration with FDio VPP? (Was: Questions about 
> XDP)
>
>
>
> On 23 Sep 2019, at 11:00, Július Milan wrote:
>
>> Many Thanks Magnus
>>
>>>> I have next 2 questions:
>>>>
>>>> 1] When I use xsk_ring_prod__reserve and successive
>>>> xsk_ring_prod__submit. Is it correct to submit also less than I
>>>> reserved?
>>>>     In some cases I can't exactly determine how much to reserve in
>>>> advance, since vpp buffers have different size than xdp frames.
>>>
>>> Let me see so I understand this correctly. Ponder you reserve 10
>>> slots and later submit 4. This means you have reserved 6 more than
>>> you need.
>>> Do you want to "unreserve" these and give them back to the ring? 
>>> This
>>> is not supported by the interface today. Another way of solving this
>>> (if this is your problem and I am understanding it correctly, that
>>> is) is that you in the next iteration only reserve 10 - 6 = 4 slots
>>> because you already have 6 slots available from the last iteration.
>>> You could still submit 10 after this. But adding something like an
>>> unreserve option would be easy as long as we made sure it only
>>> affected local ring state. The global state seen in the shared
>>> variables between user space and kernel would not be touched, as 
>>> this
>>> would affect performance negatively. Please let me know what you
>>> think.
>>>
>> Yes, You understand it correctly, I implemented it the way you
>> suggested, i.e. by marking index and count of reserved slots (not
>> committed yet, but works well), thanks again.
>>
>>>> 2] Can I use hugepage backed memory for umem? If not, is it planned
>>>> for future?
>>>>     Yet it does copy pakets from rx rings to vpp buffers, but
>>>> speculating about straight zerocopy way.
>>>
>>> Yes you can use huge pages today, but the internal AF_XDP code has
>>> not been optimized to use huge pages, so you will not get the full
>>> benefit from them today. Kevin Laatz, added to this mail, is working
>>> on optimizing the AF_XDP code for huge pages. If you want to know
>>> more or have some requirements, do not hesitate to contact him.
>>>
>> Kevin will the API for using hugepages change while optimization
>> process significantly or can I already start to rewrite my vpp driver
>> to use hugepages backed memory?
>> Also please let me know, when you consider AF_XDP code optimized to
>> use huge pages.
>>
>> William, if I may ask next question.
>> Does OVS implementation of af_xdp driver copy paket data from af_xdp
>> mmaped ring buffers into OVS "buffers" (some structure to represent
>> the packet in OVS) or is it zerocopy in this manner, i.e. OVS
>> "buffers" mempool is directly mmaped as ring and so no copy on RX is
>> needed. Since in 2nd case it would be very valuable for me as
>> inspiration.
>
> Currently, OVS uses the mmaped memory directly, however on egress, it 
> is copying the memory to the egress interface it’s mmaped memory.
> Currently, OVS uses an AF_XDP memory pool per interface, so a further 
> optimization could be to use a global memory pool so this extra copy 
> is not needed.
>
>>
>>> /Magnus
>>>
>>
>> Thanks a lot,
>>
>> Július

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[William Tu]

On Wed, Sep 25, 2019 at 12:48 AM Eelco Chaudron <echaudro@redhat.com> wrote:
>
>
>
> On 25 Sep 2019, at 8:46, Július Milan wrote:
>
> > Hi Eelco
> >
> >> Currently, OVS uses the mmaped memory directly, however on egress, it
> >> is copying the memory to the egress interface it’s mmaped memory.
> > Great, thanks for making this clear to me.
> >
> >> Currently, OVS uses an AF_XDP memory pool per interface, so a further
> >> optimization could be to use a global memory pool so this extra copy
> >> is not needed.
> > Is it even possible to make this further optimization? Since every
> > interface has it's own non-shared umem, so from my point of view, at
> > least one
> > copy for case as you described above (when RX interface is different
> > then TX interface) is necessery. Or am I missing something?
>
> Some one @Intel told me it would be possible to have one huge mempool
> that can be shared between interfaces. However I have not
> researched/tried it.

I thought about it before, but the problem is cq and fq are per-umem.
So when having only one umem shared with many queues or devices,
each one has to acquire a lock, then they can access cq or fq. I think
that might become much slower.

> Maybe Magnus can confirm?
>
>
> > Július
> >
> > -----Original Message-----
> > From: Eelco Chaudron [mailto:echaudro@redhat.com]
> > Sent: Monday, September 23, 2019 3:02 PM
> > To: Július Milan <Julius.Milan@pantheon.tech>
> > Cc: Magnus Karlsson <magnus.karlsson@gmail.com>; William Tu
> > <u9012063@gmail.com>; Björn Töpel <bjorn.topel@intel.com>; Marek
> > Závodský <marek.zavodsky@pantheon.tech>; Jesper Dangaard Brouer
> > <brouer@redhat.com>; xdp-newbies@vger.kernel.org; Karlsson, Magnus
> > <magnus.karlsson@intel.com>; Thomas F Herbert <therbert@redhat.com>;
> > Kevin Laatz <kevin.laatz@intel.com>
> > Subject: Re: AF_XDP integration with FDio VPP? (Was: Questions about
> > XDP)
> >
> >
> >
> > On 23 Sep 2019, at 11:00, Július Milan wrote:
> >
> >> Many Thanks Magnus
> >>
> >>>> I have next 2 questions:
> >>>>
> >>>> 1] When I use xsk_ring_prod__reserve and successive
> >>>> xsk_ring_prod__submit. Is it correct to submit also less than I
> >>>> reserved?
> >>>>     In some cases I can't exactly determine how much to reserve in
> >>>> advance, since vpp buffers have different size than xdp frames.
> >>>
> >>> Let me see so I understand this correctly. Ponder you reserve 10
> >>> slots and later submit 4. This means you have reserved 6 more than
> >>> you need.
> >>> Do you want to "unreserve" these and give them back to the ring?
> >>> This
> >>> is not supported by the interface today. Another way of solving this
> >>> (if this is your problem and I am understanding it correctly, that
> >>> is) is that you in the next iteration only reserve 10 - 6 = 4 slots
> >>> because you already have 6 slots available from the last iteration.
> >>> You could still submit 10 after this. But adding something like an
> >>> unreserve option would be easy as long as we made sure it only
> >>> affected local ring state. The global state seen in the shared
> >>> variables between user space and kernel would not be touched, as
> >>> this
> >>> would affect performance negatively. Please let me know what you
> >>> think.
> >>>
> >> Yes, You understand it correctly, I implemented it the way you
> >> suggested, i.e. by marking index and count of reserved slots (not
> >> committed yet, but works well), thanks again.
> >>
> >>>> 2] Can I use hugepage backed memory for umem? If not, is it planned
> >>>> for future?
> >>>>     Yet it does copy pakets from rx rings to vpp buffers, but
> >>>> speculating about straight zerocopy way.
> >>>
> >>> Yes you can use huge pages today, but the internal AF_XDP code has
> >>> not been optimized to use huge pages, so you will not get the full
> >>> benefit from them today. Kevin Laatz, added to this mail, is working
> >>> on optimizing the AF_XDP code for huge pages. If you want to know
> >>> more or have some requirements, do not hesitate to contact him.
> >>>
> >> Kevin will the API for using hugepages change while optimization
> >> process significantly or can I already start to rewrite my vpp driver
> >> to use hugepages backed memory?
> >> Also please let me know, when you consider AF_XDP code optimized to
> >> use huge pages.
> >>
> >> William, if I may ask next question.
> >> Does OVS implementation of af_xdp driver copy paket data from af_xdp
> >> mmaped ring buffers into OVS "buffers" (some structure to represent
> >> the packet in OVS) or is it zerocopy in this manner, i.e. OVS
> >> "buffers" mempool is directly mmaped as ring and so no copy on RX is
> >> needed. Since in 2nd case it would be very valuable for me as
> >> inspiration.
> >
> > Currently, OVS uses the mmaped memory directly, however on egress, it
> > is copying the memory to the egress interface it’s mmaped memory.
> > Currently, OVS uses an AF_XDP memory pool per interface, so a further
> > optimization could be to use a global memory pool so this extra copy
> > is not needed.
> >
> >>
> >>> /Magnus
> >>>
> >>
> >> Thanks a lot,
> >>
> >> Július

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Magnus Karlsson]

On Thu, Sep 26, 2019 at 1:34 AM William Tu <u9012063@gmail.com> wrote:
>
> On Wed, Sep 25, 2019 at 12:48 AM Eelco Chaudron <echaudro@redhat.com> wrote:
> >
> >
> >
> > On 25 Sep 2019, at 8:46, Július Milan wrote:
> >
> > > Hi Eelco
> > >
> > >> Currently, OVS uses the mmaped memory directly, however on egress, it
> > >> is copying the memory to the egress interface it’s mmaped memory.
> > > Great, thanks for making this clear to me.
> > >
> > >> Currently, OVS uses an AF_XDP memory pool per interface, so a further
> > >> optimization could be to use a global memory pool so this extra copy
> > >> is not needed.
> > > Is it even possible to make this further optimization? Since every
> > > interface has it's own non-shared umem, so from my point of view, at
> > > least one
> > > copy for case as you described above (when RX interface is different
> > > then TX interface) is necessery. Or am I missing something?
> >
> > Some one @Intel told me it would be possible to have one huge mempool
> > that can be shared between interfaces. However I have not
> > researched/tried it.
>
> I thought about it before, but the problem is cq and fq are per-umem.
> So when having only one umem shared with many queues or devices,
> each one has to acquire a lock, then they can access cq or fq. I think
> that might become much slower.

You basically have to implement a mempool that can be used by multiple
processes. Unfortunately, there is no lean and mean standalone
implementation of a mempool. There is a good one in DPDK, but then you
get the whole DPDK package into your application which is likely what
you wanted to avoid in the first place. Anyone for writing libmempool?

/Magnus

> > Maybe Magnus can confirm?
> >
> >
> > > Július
> > >
> > > -----Original Message-----
> > > From: Eelco Chaudron [mailto:echaudro@redhat.com]
> > > Sent: Monday, September 23, 2019 3:02 PM
> > > To: Július Milan <Julius.Milan@pantheon.tech>
> > > Cc: Magnus Karlsson <magnus.karlsson@gmail.com>; William Tu
> > > <u9012063@gmail.com>; Björn Töpel <bjorn.topel@intel.com>; Marek
> > > Závodský <marek.zavodsky@pantheon.tech>; Jesper Dangaard Brouer
> > > <brouer@redhat.com>; xdp-newbies@vger.kernel.org; Karlsson, Magnus
> > > <magnus.karlsson@intel.com>; Thomas F Herbert <therbert@redhat.com>;
> > > Kevin Laatz <kevin.laatz@intel.com>
> > > Subject: Re: AF_XDP integration with FDio VPP? (Was: Questions about
> > > XDP)
> > >
> > >
> > >
> > > On 23 Sep 2019, at 11:00, Július Milan wrote:
> > >
> > >> Many Thanks Magnus
> > >>
> > >>>> I have next 2 questions:
> > >>>>
> > >>>> 1] When I use xsk_ring_prod__reserve and successive
> > >>>> xsk_ring_prod__submit. Is it correct to submit also less than I
> > >>>> reserved?
> > >>>>     In some cases I can't exactly determine how much to reserve in
> > >>>> advance, since vpp buffers have different size than xdp frames.
> > >>>
> > >>> Let me see so I understand this correctly. Ponder you reserve 10
> > >>> slots and later submit 4. This means you have reserved 6 more than
> > >>> you need.
> > >>> Do you want to "unreserve" these and give them back to the ring?
> > >>> This
> > >>> is not supported by the interface today. Another way of solving this
> > >>> (if this is your problem and I am understanding it correctly, that
> > >>> is) is that you in the next iteration only reserve 10 - 6 = 4 slots
> > >>> because you already have 6 slots available from the last iteration.
> > >>> You could still submit 10 after this. But adding something like an
> > >>> unreserve option would be easy as long as we made sure it only
> > >>> affected local ring state. The global state seen in the shared
> > >>> variables between user space and kernel would not be touched, as
> > >>> this
> > >>> would affect performance negatively. Please let me know what you
> > >>> think.
> > >>>
> > >> Yes, You understand it correctly, I implemented it the way you
> > >> suggested, i.e. by marking index and count of reserved slots (not
> > >> committed yet, but works well), thanks again.
> > >>
> > >>>> 2] Can I use hugepage backed memory for umem? If not, is it planned
> > >>>> for future?
> > >>>>     Yet it does copy pakets from rx rings to vpp buffers, but
> > >>>> speculating about straight zerocopy way.
> > >>>
> > >>> Yes you can use huge pages today, but the internal AF_XDP code has
> > >>> not been optimized to use huge pages, so you will not get the full
> > >>> benefit from them today. Kevin Laatz, added to this mail, is working
> > >>> on optimizing the AF_XDP code for huge pages. If you want to know
> > >>> more or have some requirements, do not hesitate to contact him.
> > >>>
> > >> Kevin will the API for using hugepages change while optimization
> > >> process significantly or can I already start to rewrite my vpp driver
> > >> to use hugepages backed memory?
> > >> Also please let me know, when you consider AF_XDP code optimized to
> > >> use huge pages.
> > >>
> > >> William, if I may ask next question.
> > >> Does OVS implementation of af_xdp driver copy paket data from af_xdp
> > >> mmaped ring buffers into OVS "buffers" (some structure to represent
> > >> the packet in OVS) or is it zerocopy in this manner, i.e. OVS
> > >> "buffers" mempool is directly mmaped as ring and so no copy on RX is
> > >> needed. Since in 2nd case it would be very valuable for me as
> > >> inspiration.
> > >
> > > Currently, OVS uses the mmaped memory directly, however on egress, it
> > > is copying the memory to the egress interface it’s mmaped memory.
> > > Currently, OVS uses an AF_XDP memory pool per interface, so a further
> > > optimization could be to use a global memory pool so this extra copy
> > > is not needed.
> > >
> > >>
> > >>> /Magnus
> > >>>
> > >>
> > >> Thanks a lot,
> > >>
> > >> Július

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[William Tu]

On Fri, Sep 27, 2019 at 12:02 AM Magnus Karlsson
<magnus.karlsson@gmail.com> wrote:
>
> On Thu, Sep 26, 2019 at 1:34 AM William Tu <u9012063@gmail.com> wrote:
> >
> > On Wed, Sep 25, 2019 at 12:48 AM Eelco Chaudron <echaudro@redhat.com> wrote:
> > >
> > >
> > >
> > > On 25 Sep 2019, at 8:46, Július Milan wrote:
> > >
> > > > Hi Eelco
> > > >
> > > >> Currently, OVS uses the mmaped memory directly, however on egress, it
> > > >> is copying the memory to the egress interface it’s mmaped memory.
> > > > Great, thanks for making this clear to me.
> > > >
> > > >> Currently, OVS uses an AF_XDP memory pool per interface, so a further
> > > >> optimization could be to use a global memory pool so this extra copy
> > > >> is not needed.
> > > > Is it even possible to make this further optimization? Since every
> > > > interface has it's own non-shared umem, so from my point of view, at
> > > > least one
> > > > copy for case as you described above (when RX interface is different
> > > > then TX interface) is necessery. Or am I missing something?
> > >
> > > Some one @Intel told me it would be possible to have one huge mempool
> > > that can be shared between interfaces. However I have not
> > > researched/tried it.
> >
> > I thought about it before, but the problem is cq and fq are per-umem.
> > So when having only one umem shared with many queues or devices,
> > each one has to acquire a lock, then they can access cq or fq. I think
> > that might become much slower.
>
> You basically have to implement a mempool that can be used by multiple
> processes. Unfortunately, there is no lean and mean standalone
> implementation of a mempool. There is a good one in DPDK, but then you
> get the whole DPDK package into your application which is likely what
> you wanted to avoid in the first place. Anyone for writing libmempool?
>
> /Magnus
>

That's interesting.
Do you mean the DPDK's rte_mempool which supports multiple-producer?

If I create a shared umem for queue1  and queue2, then each queue has its
own tx/rx ring so they can process in parallel. But for handling the per-umem
cq/fq, I can create a dedicated thread to process cq/fq.
So for example:
Thread 1 for handling cq/fq
Thread 2 for processing queue1 tx/rx queue
Thread 3 for processing queue2 tx/rx queue
and the mempool should allow multiple producer and consumer.

Does this sound correct?

Thanks
Wiliam

> > > Maybe Magnus can confirm?
> > >
> > >
> > > > Július
> > > >
> > > > -----Original Message-----
> > > > From: Eelco Chaudron [mailto:echaudro@redhat.com]
> > > > Sent: Monday, September 23, 2019 3:02 PM
> > > > To: Július Milan <Julius.Milan@pantheon.tech>
> > > > Cc: Magnus Karlsson <magnus.karlsson@gmail.com>; William Tu
> > > > <u9012063@gmail.com>; Björn Töpel <bjorn.topel@intel.com>; Marek
> > > > Závodský <marek.zavodsky@pantheon.tech>; Jesper Dangaard Brouer
> > > > <brouer@redhat.com>; xdp-newbies@vger.kernel.org; Karlsson, Magnus
> > > > <magnus.karlsson@intel.com>; Thomas F Herbert <therbert@redhat.com>;
> > > > Kevin Laatz <kevin.laatz@intel.com>
> > > > Subject: Re: AF_XDP integration with FDio VPP? (Was: Questions about
> > > > XDP)
> > > >
> > > >
> > > >
> > > > On 23 Sep 2019, at 11:00, Július Milan wrote:
> > > >
> > > >> Many Thanks Magnus
> > > >>
> > > >>>> I have next 2 questions:
> > > >>>>
> > > >>>> 1] When I use xsk_ring_prod__reserve and successive
> > > >>>> xsk_ring_prod__submit. Is it correct to submit also less than I
> > > >>>> reserved?
> > > >>>>     In some cases I can't exactly determine how much to reserve in
> > > >>>> advance, since vpp buffers have different size than xdp frames.
> > > >>>
> > > >>> Let me see so I understand this correctly. Ponder you reserve 10
> > > >>> slots and later submit 4. This means you have reserved 6 more than
> > > >>> you need.
> > > >>> Do you want to "unreserve" these and give them back to the ring?
> > > >>> This
> > > >>> is not supported by the interface today. Another way of solving this
> > > >>> (if this is your problem and I am understanding it correctly, that
> > > >>> is) is that you in the next iteration only reserve 10 - 6 = 4 slots
> > > >>> because you already have 6 slots available from the last iteration.
> > > >>> You could still submit 10 after this. But adding something like an
> > > >>> unreserve option would be easy as long as we made sure it only
> > > >>> affected local ring state. The global state seen in the shared
> > > >>> variables between user space and kernel would not be touched, as
> > > >>> this
> > > >>> would affect performance negatively. Please let me know what you
> > > >>> think.
> > > >>>
> > > >> Yes, You understand it correctly, I implemented it the way you
> > > >> suggested, i.e. by marking index and count of reserved slots (not
> > > >> committed yet, but works well), thanks again.
> > > >>
> > > >>>> 2] Can I use hugepage backed memory for umem? If not, is it planned
> > > >>>> for future?
> > > >>>>     Yet it does copy pakets from rx rings to vpp buffers, but
> > > >>>> speculating about straight zerocopy way.
> > > >>>
> > > >>> Yes you can use huge pages today, but the internal AF_XDP code has
> > > >>> not been optimized to use huge pages, so you will not get the full
> > > >>> benefit from them today. Kevin Laatz, added to this mail, is working
> > > >>> on optimizing the AF_XDP code for huge pages. If you want to know
> > > >>> more or have some requirements, do not hesitate to contact him.
> > > >>>
> > > >> Kevin will the API for using hugepages change while optimization
> > > >> process significantly or can I already start to rewrite my vpp driver
> > > >> to use hugepages backed memory?
> > > >> Also please let me know, when you consider AF_XDP code optimized to
> > > >> use huge pages.
> > > >>
> > > >> William, if I may ask next question.
> > > >> Does OVS implementation of af_xdp driver copy paket data from af_xdp
> > > >> mmaped ring buffers into OVS "buffers" (some structure to represent
> > > >> the packet in OVS) or is it zerocopy in this manner, i.e. OVS
> > > >> "buffers" mempool is directly mmaped as ring and so no copy on RX is
> > > >> needed. Since in 2nd case it would be very valuable for me as
> > > >> inspiration.
> > > >
> > > > Currently, OVS uses the mmaped memory directly, however on egress, it
> > > > is copying the memory to the egress interface it’s mmaped memory.
> > > > Currently, OVS uses an AF_XDP memory pool per interface, so a further
> > > > optimization could be to use a global memory pool so this extra copy
> > > > is not needed.
> > > >
> > > >>
> > > >>> /Magnus
> > > >>>
> > > >>
> > > >> Thanks a lot,
> > > >>
> > > >> Július

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Magnus Karlsson]

On Fri, Sep 27, 2019 at 8:09 PM William Tu <u9012063@gmail.com> wrote:
>
> On Fri, Sep 27, 2019 at 12:02 AM Magnus Karlsson
> <magnus.karlsson@gmail.com> wrote:
> >
> > On Thu, Sep 26, 2019 at 1:34 AM William Tu <u9012063@gmail.com> wrote:
> > >
> > > On Wed, Sep 25, 2019 at 12:48 AM Eelco Chaudron <echaudro@redhat.com> wrote:
> > > >
> > > >
> > > >
> > > > On 25 Sep 2019, at 8:46, Július Milan wrote:
> > > >
> > > > > Hi Eelco
> > > > >
> > > > >> Currently, OVS uses the mmaped memory directly, however on egress, it
> > > > >> is copying the memory to the egress interface it’s mmaped memory.
> > > > > Great, thanks for making this clear to me.
> > > > >
> > > > >> Currently, OVS uses an AF_XDP memory pool per interface, so a further
> > > > >> optimization could be to use a global memory pool so this extra copy
> > > > >> is not needed.
> > > > > Is it even possible to make this further optimization? Since every
> > > > > interface has it's own non-shared umem, so from my point of view, at
> > > > > least one
> > > > > copy for case as you described above (when RX interface is different
> > > > > then TX interface) is necessery. Or am I missing something?
> > > >
> > > > Some one @Intel told me it would be possible to have one huge mempool
> > > > that can be shared between interfaces. However I have not
> > > > researched/tried it.
> > >
> > > I thought about it before, but the problem is cq and fq are per-umem.
> > > So when having only one umem shared with many queues or devices,
> > > each one has to acquire a lock, then they can access cq or fq. I think
> > > that might become much slower.
> >
> > You basically have to implement a mempool that can be used by multiple
> > processes. Unfortunately, there is no lean and mean standalone
> > implementation of a mempool. There is a good one in DPDK, but then you
> > get the whole DPDK package into your application which is likely what
> > you wanted to avoid in the first place. Anyone for writing libmempool?
> >
> > /Magnus
> >
>
> That's interesting.
> Do you mean the DPDK's rte_mempool which supports multiple-producer?

Yes.

> If I create a shared umem for queue1  and queue2, then each queue has its
> own tx/rx ring so they can process in parallel. But for handling the per-umem
> cq/fq, I can create a dedicated thread to process cq/fq.
> So for example:
> Thread 1 for handling cq/fq
> Thread 2 for processing queue1 tx/rx queue
> Thread 3 for processing queue2 tx/rx queue
> and the mempool should allow multiple producer and consumer.
>
> Does this sound correct?

You do not need a dedicated process. Just something in the mempool
code that enforces mutual exclusion (a mutex or whatever) between
thread 2 and 3 when they are performing operations on the mempool.
Going with a dedicated process sounds complicated.

/Magnus

> Thanks
> Wiliam
>
> > > > Maybe Magnus can confirm?
> > > >
> > > >
> > > > > Július
> > > > >
> > > > > -----Original Message-----
> > > > > From: Eelco Chaudron [mailto:echaudro@redhat.com]
> > > > > Sent: Monday, September 23, 2019 3:02 PM
> > > > > To: Július Milan <Julius.Milan@pantheon.tech>
> > > > > Cc: Magnus Karlsson <magnus.karlsson@gmail.com>; William Tu
> > > > > <u9012063@gmail.com>; Björn Töpel <bjorn.topel@intel.com>; Marek
> > > > > Závodský <marek.zavodsky@pantheon.tech>; Jesper Dangaard Brouer
> > > > > <brouer@redhat.com>; xdp-newbies@vger.kernel.org; Karlsson, Magnus
> > > > > <magnus.karlsson@intel.com>; Thomas F Herbert <therbert@redhat.com>;
> > > > > Kevin Laatz <kevin.laatz@intel.com>
> > > > > Subject: Re: AF_XDP integration with FDio VPP? (Was: Questions about
> > > > > XDP)
> > > > >
> > > > >
> > > > >
> > > > > On 23 Sep 2019, at 11:00, Július Milan wrote:
> > > > >
> > > > >> Many Thanks Magnus
> > > > >>
> > > > >>>> I have next 2 questions:
> > > > >>>>
> > > > >>>> 1] When I use xsk_ring_prod__reserve and successive
> > > > >>>> xsk_ring_prod__submit. Is it correct to submit also less than I
> > > > >>>> reserved?
> > > > >>>>     In some cases I can't exactly determine how much to reserve in
> > > > >>>> advance, since vpp buffers have different size than xdp frames.
> > > > >>>
> > > > >>> Let me see so I understand this correctly. Ponder you reserve 10
> > > > >>> slots and later submit 4. This means you have reserved 6 more than
> > > > >>> you need.
> > > > >>> Do you want to "unreserve" these and give them back to the ring?
> > > > >>> This
> > > > >>> is not supported by the interface today. Another way of solving this
> > > > >>> (if this is your problem and I am understanding it correctly, that
> > > > >>> is) is that you in the next iteration only reserve 10 - 6 = 4 slots
> > > > >>> because you already have 6 slots available from the last iteration.
> > > > >>> You could still submit 10 after this. But adding something like an
> > > > >>> unreserve option would be easy as long as we made sure it only
> > > > >>> affected local ring state. The global state seen in the shared
> > > > >>> variables between user space and kernel would not be touched, as
> > > > >>> this
> > > > >>> would affect performance negatively. Please let me know what you
> > > > >>> think.
> > > > >>>
> > > > >> Yes, You understand it correctly, I implemented it the way you
> > > > >> suggested, i.e. by marking index and count of reserved slots (not
> > > > >> committed yet, but works well), thanks again.
> > > > >>
> > > > >>>> 2] Can I use hugepage backed memory for umem? If not, is it planned
> > > > >>>> for future?
> > > > >>>>     Yet it does copy pakets from rx rings to vpp buffers, but
> > > > >>>> speculating about straight zerocopy way.
> > > > >>>
> > > > >>> Yes you can use huge pages today, but the internal AF_XDP code has
> > > > >>> not been optimized to use huge pages, so you will not get the full
> > > > >>> benefit from them today. Kevin Laatz, added to this mail, is working
> > > > >>> on optimizing the AF_XDP code for huge pages. If you want to know
> > > > >>> more or have some requirements, do not hesitate to contact him.
> > > > >>>
> > > > >> Kevin will the API for using hugepages change while optimization
> > > > >> process significantly or can I already start to rewrite my vpp driver
> > > > >> to use hugepages backed memory?
> > > > >> Also please let me know, when you consider AF_XDP code optimized to
> > > > >> use huge pages.
> > > > >>
> > > > >> William, if I may ask next question.
> > > > >> Does OVS implementation of af_xdp driver copy paket data from af_xdp
> > > > >> mmaped ring buffers into OVS "buffers" (some structure to represent
> > > > >> the packet in OVS) or is it zerocopy in this manner, i.e. OVS
> > > > >> "buffers" mempool is directly mmaped as ring and so no copy on RX is
> > > > >> needed. Since in 2nd case it would be very valuable for me as
> > > > >> inspiration.
> > > > >
> > > > > Currently, OVS uses the mmaped memory directly, however on egress, it
> > > > > is copying the memory to the egress interface it’s mmaped memory.
> > > > > Currently, OVS uses an AF_XDP memory pool per interface, so a further
> > > > > optimization could be to use a global memory pool so this extra copy
> > > > > is not needed.
> > > > >
> > > > >>
> > > > >>> /Magnus
> > > > >>>
> > > > >>
> > > > >> Thanks a lot,
> > > > >>
> > > > >> Július

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Eelco Chaudron]

On 30 Sep 2019, at 8:51, Magnus Karlsson wrote:

> On Fri, Sep 27, 2019 at 8:09 PM William Tu <u9012063@gmail.com> wrote:
>>
>> On Fri, Sep 27, 2019 at 12:02 AM Magnus Karlsson
>> <magnus.karlsson@gmail.com> wrote:
>>>
>>> On Thu, Sep 26, 2019 at 1:34 AM William Tu <u9012063@gmail.com> 
>>> wrote:
>>>>
>>>> On Wed, Sep 25, 2019 at 12:48 AM Eelco Chaudron 
>>>> <echaudro@redhat.com> wrote:
>>>>>
>>>>>
>>>>>
>>>>> On 25 Sep 2019, at 8:46, Július Milan wrote:
>>>>>
>>>>>> Hi Eelco
>>>>>>
>>>>>>> Currently, OVS uses the mmaped memory directly, however on 
>>>>>>> egress, it
>>>>>>> is copying the memory to the egress interface it’s mmaped 
>>>>>>> memory.
>>>>>> Great, thanks for making this clear to me.
>>>>>>
>>>>>>> Currently, OVS uses an AF_XDP memory pool per interface, so a 
>>>>>>> further
>>>>>>> optimization could be to use a global memory pool so this extra 
>>>>>>> copy
>>>>>>> is not needed.
>>>>>> Is it even possible to make this further optimization? Since 
>>>>>> every
>>>>>> interface has it's own non-shared umem, so from my point of view, 
>>>>>> at
>>>>>> least one
>>>>>> copy for case as you described above (when RX interface is 
>>>>>> different
>>>>>> then TX interface) is necessery. Or am I missing something?
>>>>>
>>>>> Some one @Intel told me it would be possible to have one huge 
>>>>> mempool
>>>>> that can be shared between interfaces. However I have not
>>>>> researched/tried it.
>>>>
>>>> I thought about it before, but the problem is cq and fq are 
>>>> per-umem.
>>>> So when having only one umem shared with many queues or devices,
>>>> each one has to acquire a lock, then they can access cq or fq. I 
>>>> think
>>>> that might become much slower.
>>>
>>> You basically have to implement a mempool that can be used by 
>>> multiple
>>> processes. Unfortunately, there is no lean and mean standalone
>>> implementation of a mempool. There is a good one in DPDK, but then 
>>> you
>>> get the whole DPDK package into your application which is likely 
>>> what
>>> you wanted to avoid in the first place. Anyone for writing 
>>> libmempool?
>>>
>>> /Magnus
>>>
>>
>> That's interesting.
>> Do you mean the DPDK's rte_mempool which supports multiple-producer?
>
> Yes.
>
>> If I create a shared umem for queue1  and queue2, then each queue has 
>> its
>> own tx/rx ring so they can process in parallel. But for handling the 
>> per-umem
>> cq/fq, I can create a dedicated thread to process cq/fq.
>> So for example:
>> Thread 1 for handling cq/fq
>> Thread 2 for processing queue1 tx/rx queue
>> Thread 3 for processing queue2 tx/rx queue
>> and the mempool should allow multiple producer and consumer.
>>
>> Does this sound correct?
>
> You do not need a dedicated process. Just something in the mempool
> code that enforces mutual exclusion (a mutex or whatever) between
> thread 2 and 3 when they are performing operations on the mempool.
> Going with a dedicated process sounds complicated.

I was trying to see how to experiment with this using libbpf, but looks 
like it’s not yet supported?

Is see the following in xsk_socket__create():

475         if (umem->refcount) {
476                 pr_warning("Error: shared umems not supported by 
libbpf.\n");
477                 return -EBUSY;
478         }

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Magnus Karlsson]

On Mon, Sep 30, 2019 at 11:28 AM Eelco Chaudron <echaudro@redhat.com> wrote:
>
>
>
> On 30 Sep 2019, at 8:51, Magnus Karlsson wrote:
>
> > On Fri, Sep 27, 2019 at 8:09 PM William Tu <u9012063@gmail.com> wrote:
> >>
> >> On Fri, Sep 27, 2019 at 12:02 AM Magnus Karlsson
> >> <magnus.karlsson@gmail.com> wrote:
> >>>
> >>> On Thu, Sep 26, 2019 at 1:34 AM William Tu <u9012063@gmail.com>
> >>> wrote:
> >>>>
> >>>> On Wed, Sep 25, 2019 at 12:48 AM Eelco Chaudron
> >>>> <echaudro@redhat.com> wrote:
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 25 Sep 2019, at 8:46, Július Milan wrote:
> >>>>>
> >>>>>> Hi Eelco
> >>>>>>
> >>>>>>> Currently, OVS uses the mmaped memory directly, however on
> >>>>>>> egress, it
> >>>>>>> is copying the memory to the egress interface it’s mmaped
> >>>>>>> memory.
> >>>>>> Great, thanks for making this clear to me.
> >>>>>>
> >>>>>>> Currently, OVS uses an AF_XDP memory pool per interface, so a
> >>>>>>> further
> >>>>>>> optimization could be to use a global memory pool so this extra
> >>>>>>> copy
> >>>>>>> is not needed.
> >>>>>> Is it even possible to make this further optimization? Since
> >>>>>> every
> >>>>>> interface has it's own non-shared umem, so from my point of view,
> >>>>>> at
> >>>>>> least one
> >>>>>> copy for case as you described above (when RX interface is
> >>>>>> different
> >>>>>> then TX interface) is necessery. Or am I missing something?
> >>>>>
> >>>>> Some one @Intel told me it would be possible to have one huge
> >>>>> mempool
> >>>>> that can be shared between interfaces. However I have not
> >>>>> researched/tried it.
> >>>>
> >>>> I thought about it before, but the problem is cq and fq are
> >>>> per-umem.
> >>>> So when having only one umem shared with many queues or devices,
> >>>> each one has to acquire a lock, then they can access cq or fq. I
> >>>> think
> >>>> that might become much slower.
> >>>
> >>> You basically have to implement a mempool that can be used by
> >>> multiple
> >>> processes. Unfortunately, there is no lean and mean standalone
> >>> implementation of a mempool. There is a good one in DPDK, but then
> >>> you
> >>> get the whole DPDK package into your application which is likely
> >>> what
> >>> you wanted to avoid in the first place. Anyone for writing
> >>> libmempool?
> >>>
> >>> /Magnus
> >>>
> >>
> >> That's interesting.
> >> Do you mean the DPDK's rte_mempool which supports multiple-producer?
> >
> > Yes.
> >
> >> If I create a shared umem for queue1  and queue2, then each queue has
> >> its
> >> own tx/rx ring so they can process in parallel. But for handling the
> >> per-umem
> >> cq/fq, I can create a dedicated thread to process cq/fq.
> >> So for example:
> >> Thread 1 for handling cq/fq
> >> Thread 2 for processing queue1 tx/rx queue
> >> Thread 3 for processing queue2 tx/rx queue
> >> and the mempool should allow multiple producer and consumer.
> >>
> >> Does this sound correct?
> >
> > You do not need a dedicated process. Just something in the mempool
> > code that enforces mutual exclusion (a mutex or whatever) between
> > thread 2 and 3 when they are performing operations on the mempool.
> > Going with a dedicated process sounds complicated.
>
> I was trying to see how to experiment with this using libbpf, but looks
> like it’s not yet supported?
>
> Is see the following in xsk_socket__create():
>
> 475         if (umem->refcount) {
> 476                 pr_warning("Error: shared umems not supported by
> libbpf.\n");
> 477                 return -EBUSY;
> 478         }
>

Using the XDP_SHARED_UMEM option is not supported in libbpf at this
point in time. In this mode you share a single umem with a single
completion queue and a single fill queue among many xsk sockets tied
to the same queue id. But note that you can register the same umem
area multiple times (creating multiple umem handles and multiple fqs
and cqs) to be able to support xsk sockets that have different queue
ids, but the same umem area. In both cases you need a mempool that can
handle multiple threads.

The old xdpsock application prior to libbpf had support for the
XDP_SHARED_UMEM option. Take a look at that one if you would like to
experiment with it.

/Magnus

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Eelco Chaudron]

On 30 Sep 2019, at 13:02, Magnus Karlsson wrote:

> On Mon, Sep 30, 2019 at 11:28 AM Eelco Chaudron <echaudro@redhat.com> 
> wrote:
>>
>>
>>
>> On 30 Sep 2019, at 8:51, Magnus Karlsson wrote:
>>
>>> On Fri, Sep 27, 2019 at 8:09 PM William Tu <u9012063@gmail.com> 
>>> wrote:
>>>>
>>>> On Fri, Sep 27, 2019 at 12:02 AM Magnus Karlsson
>>>> <magnus.karlsson@gmail.com> wrote:
>>>>>
>>>>> On Thu, Sep 26, 2019 at 1:34 AM William Tu <u9012063@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> On Wed, Sep 25, 2019 at 12:48 AM Eelco Chaudron
>>>>>> <echaudro@redhat.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 25 Sep 2019, at 8:46, Július Milan wrote:
>>>>>>>
>>>>>>>> Hi Eelco
>>>>>>>>
>>>>>>>>> Currently, OVS uses the mmaped memory directly, however on
>>>>>>>>> egress, it
>>>>>>>>> is copying the memory to the egress interface it’s mmaped
>>>>>>>>> memory.
>>>>>>>> Great, thanks for making this clear to me.
>>>>>>>>
>>>>>>>>> Currently, OVS uses an AF_XDP memory pool per interface, so a
>>>>>>>>> further
>>>>>>>>> optimization could be to use a global memory pool so this 
>>>>>>>>> extra
>>>>>>>>> copy
>>>>>>>>> is not needed.
>>>>>>>> Is it even possible to make this further optimization? Since
>>>>>>>> every
>>>>>>>> interface has it's own non-shared umem, so from my point of 
>>>>>>>> view,
>>>>>>>> at
>>>>>>>> least one
>>>>>>>> copy for case as you described above (when RX interface is
>>>>>>>> different
>>>>>>>> then TX interface) is necessery. Or am I missing something?
>>>>>>>
>>>>>>> Some one @Intel told me it would be possible to have one huge
>>>>>>> mempool
>>>>>>> that can be shared between interfaces. However I have not
>>>>>>> researched/tried it.
>>>>>>
>>>>>> I thought about it before, but the problem is cq and fq are
>>>>>> per-umem.
>>>>>> So when having only one umem shared with many queues or devices,
>>>>>> each one has to acquire a lock, then they can access cq or fq. I
>>>>>> think
>>>>>> that might become much slower.
>>>>>
>>>>> You basically have to implement a mempool that can be used by
>>>>> multiple
>>>>> processes. Unfortunately, there is no lean and mean standalone
>>>>> implementation of a mempool. There is a good one in DPDK, but then
>>>>> you
>>>>> get the whole DPDK package into your application which is likely
>>>>> what
>>>>> you wanted to avoid in the first place. Anyone for writing
>>>>> libmempool?
>>>>>
>>>>> /Magnus
>>>>>
>>>>
>>>> That's interesting.
>>>> Do you mean the DPDK's rte_mempool which supports 
>>>> multiple-producer?
>>>
>>> Yes.
>>>
>>>> If I create a shared umem for queue1  and queue2, then each queue 
>>>> has
>>>> its
>>>> own tx/rx ring so they can process in parallel. But for handling 
>>>> the
>>>> per-umem
>>>> cq/fq, I can create a dedicated thread to process cq/fq.
>>>> So for example:
>>>> Thread 1 for handling cq/fq
>>>> Thread 2 for processing queue1 tx/rx queue
>>>> Thread 3 for processing queue2 tx/rx queue
>>>> and the mempool should allow multiple producer and consumer.
>>>>
>>>> Does this sound correct?
>>>
>>> You do not need a dedicated process. Just something in the mempool
>>> code that enforces mutual exclusion (a mutex or whatever) between
>>> thread 2 and 3 when they are performing operations on the mempool.
>>> Going with a dedicated process sounds complicated.
>>
>> I was trying to see how to experiment with this using libbpf, but 
>> looks
>> like it’s not yet supported?
>>
>> Is see the following in xsk_socket__create():
>>
>> 475         if (umem->refcount) {
>> 476                 pr_warning("Error: shared umems not supported by
>> libbpf.\n");
>> 477                 return -EBUSY;
>> 478         }
>>
>
> Using the XDP_SHARED_UMEM option is not supported in libbpf at this
> point in time. In this mode you share a single umem with a single
> completion queue and a single fill queue among many xsk sockets tied
> to the same queue id. But note that you can register the same umem
> area multiple times (creating multiple umem handles and multiple fqs
> and cqs) to be able to support xsk sockets that have different queue
> ids, but the same umem area. In both cases you need a mempool that can
> handle multiple threads.

Cool, this was not clear, and is what would fit better than the shared 
fqs/cqs.

William this would be an interesting option for OVS to support zero 
memcpy on tx.

> The old xdpsock application prior to libbpf had support for the
> XDP_SHARED_UMEM option. Take a look at that one if you would like to
> experiment with it.
>
> /Magnus

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[William Tu]

On Mon, Sep 30, 2019 at 5:17 AM Eelco Chaudron <echaudro@redhat.com> wrote:
>
>
>
> On 30 Sep 2019, at 13:02, Magnus Karlsson wrote:
>
> > On Mon, Sep 30, 2019 at 11:28 AM Eelco Chaudron <echaudro@redhat.com>
> > wrote:
> >>
> >>
> >>
> >> On 30 Sep 2019, at 8:51, Magnus Karlsson wrote:
> >>
> >>> On Fri, Sep 27, 2019 at 8:09 PM William Tu <u9012063@gmail.com>
> >>> wrote:
> >>>>
> >>>> On Fri, Sep 27, 2019 at 12:02 AM Magnus Karlsson
> >>>> <magnus.karlsson@gmail.com> wrote:
> >>>>>
> >>>>> On Thu, Sep 26, 2019 at 1:34 AM William Tu <u9012063@gmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> On Wed, Sep 25, 2019 at 12:48 AM Eelco Chaudron
> >>>>>> <echaudro@redhat.com> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On 25 Sep 2019, at 8:46, Július Milan wrote:
> >>>>>>>
> >>>>>>>> Hi Eelco
> >>>>>>>>
> >>>>>>>>> Currently, OVS uses the mmaped memory directly, however on
> >>>>>>>>> egress, it
> >>>>>>>>> is copying the memory to the egress interface it’s mmaped
> >>>>>>>>> memory.
> >>>>>>>> Great, thanks for making this clear to me.
> >>>>>>>>
> >>>>>>>>> Currently, OVS uses an AF_XDP memory pool per interface, so a
> >>>>>>>>> further
> >>>>>>>>> optimization could be to use a global memory pool so this
> >>>>>>>>> extra
> >>>>>>>>> copy
> >>>>>>>>> is not needed.
> >>>>>>>> Is it even possible to make this further optimization? Since
> >>>>>>>> every
> >>>>>>>> interface has it's own non-shared umem, so from my point of
> >>>>>>>> view,
> >>>>>>>> at
> >>>>>>>> least one
> >>>>>>>> copy for case as you described above (when RX interface is
> >>>>>>>> different
> >>>>>>>> then TX interface) is necessery. Or am I missing something?
> >>>>>>>
> >>>>>>> Some one @Intel told me it would be possible to have one huge
> >>>>>>> mempool
> >>>>>>> that can be shared between interfaces. However I have not
> >>>>>>> researched/tried it.
> >>>>>>
> >>>>>> I thought about it before, but the problem is cq and fq are
> >>>>>> per-umem.
> >>>>>> So when having only one umem shared with many queues or devices,
> >>>>>> each one has to acquire a lock, then they can access cq or fq. I
> >>>>>> think
> >>>>>> that might become much slower.
> >>>>>
> >>>>> You basically have to implement a mempool that can be used by
> >>>>> multiple
> >>>>> processes. Unfortunately, there is no lean and mean standalone
> >>>>> implementation of a mempool. There is a good one in DPDK, but then
> >>>>> you
> >>>>> get the whole DPDK package into your application which is likely
> >>>>> what
> >>>>> you wanted to avoid in the first place. Anyone for writing
> >>>>> libmempool?
> >>>>>
> >>>>> /Magnus
> >>>>>
> >>>>
> >>>> That's interesting.
> >>>> Do you mean the DPDK's rte_mempool which supports
> >>>> multiple-producer?
> >>>
> >>> Yes.
> >>>
> >>>> If I create a shared umem for queue1  and queue2, then each queue
> >>>> has
> >>>> its
> >>>> own tx/rx ring so they can process in parallel. But for handling
> >>>> the
> >>>> per-umem
> >>>> cq/fq, I can create a dedicated thread to process cq/fq.
> >>>> So for example:
> >>>> Thread 1 for handling cq/fq
> >>>> Thread 2 for processing queue1 tx/rx queue
> >>>> Thread 3 for processing queue2 tx/rx queue
> >>>> and the mempool should allow multiple producer and consumer.
> >>>>
> >>>> Does this sound correct?
> >>>
> >>> You do not need a dedicated process. Just something in the mempool
> >>> code that enforces mutual exclusion (a mutex or whatever) between
> >>> thread 2 and 3 when they are performing operations on the mempool.
> >>> Going with a dedicated process sounds complicated.
> >>
> >> I was trying to see how to experiment with this using libbpf, but
> >> looks
> >> like it’s not yet supported?
> >>
> >> Is see the following in xsk_socket__create():
> >>
> >> 475         if (umem->refcount) {
> >> 476                 pr_warning("Error: shared umems not supported by
> >> libbpf.\n");
> >> 477                 return -EBUSY;
> >> 478         }
> >>
> >
> > Using the XDP_SHARED_UMEM option is not supported in libbpf at this
> > point in time. In this mode you share a single umem with a single
> > completion queue and a single fill queue among many xsk sockets tied
> > to the same queue id. But note that you can register the same umem
> > area multiple times (creating multiple umem handles and multiple fqs
> > and cqs) to be able to support xsk sockets that have different queue
> > ids, but the same umem area. In both cases you need a mempool that can
> > handle multiple threads.
>
> Cool, this was not clear, and is what would fit better than the shared
> fqs/cqs.
>
> William this would be an interesting option for OVS to support zero
> memcpy on tx.

Great, much clear to me now. I will take a look!

William

RE: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Július Milan]

On 30 Sep 2019, at 13:02, Magnus Karlsson wrote:

> On Mon, Sep 30, 2019 at 11:28 AM Eelco Chaudron <echaudro@redhat.com>
> wrote:
>>
>>
>>
>> On 30 Sep 2019, at 8:51, Magnus Karlsson wrote:
>>
>>> On Fri, Sep 27, 2019 at 8:09 PM William Tu <u9012063@gmail.com>
>>> wrote:
>>>>
>>>> On Fri, Sep 27, 2019 at 12:02 AM Magnus Karlsson 
>>>> <magnus.karlsson@gmail.com> wrote:
>>>>>
>>>>> On Thu, Sep 26, 2019 at 1:34 AM William Tu <u9012063@gmail.com>
>>>>> wrote:
>>>>>>
>>>>>> On Wed, Sep 25, 2019 at 12:48 AM Eelco Chaudron 
>>>>>> <echaudro@redhat.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 25 Sep 2019, at 8:46, Július Milan wrote:
>>>>>>>
>>>>>>>> Hi Eelco
>>>>>>>>
>>>>>>>>> Currently, OVS uses the mmaped memory directly, however on 
>>>>>>>>> egress, it is copying the memory to the egress interface it’s 
>>>>>>>>> mmaped memory.
>>>>>>>> Great, thanks for making this clear to me.
>>>>>>>>
>>>>>>>>> Currently, OVS uses an AF_XDP memory pool per interface, so a 
>>>>>>>>> further optimization could be to use a global memory pool so 
>>>>>>>>> this extra copy is not needed.
>>>>>>>> Is it even possible to make this further optimization? Since 
>>>>>>>> every interface has it's own non-shared umem, so from my point 
>>>>>>>> of view, at least one copy for case as you described above 
>>>>>>>> (when RX interface is different then TX interface) is 
>>>>>>>> necessery. Or am I missing something?
>>>>>>>
>>>>>>> Some one @Intel told me it would be possible to have one huge 
>>>>>>> mempool that can be shared between interfaces. However I have 
>>>>>>> not researched/tried it.
>>>>>>
>>>>>> I thought about it before, but the problem is cq and fq are 
>>>>>> per-umem.
>>>>>> So when having only one umem shared with many queues or devices, 
>>>>>> each one has to acquire a lock, then they can access cq or fq. I 
>>>>>> think that might become much slower.
>>>>>
>>>>> You basically have to implement a mempool that can be used by 
>>>>> multiple processes. Unfortunately, there is no lean and mean 
>>>>> standalone implementation of a mempool. There is a good one in 
>>>>> DPDK, but then you get the whole DPDK package into your 
>>>>> application which is likely what you wanted to avoid in the first 
>>>>> place. Anyone for writing libmempool?
>>>>>
>>>>> /Magnus
>>>>>
>>>>
>>>> That's interesting.
>>>> Do you mean the DPDK's rte_mempool which supports 
>>>> multiple-producer?
>>>
>>> Yes.
>>>
>>>> If I create a shared umem for queue1  and queue2, then each queue 
>>>> has its own tx/rx ring so they can process in parallel. But for 
>>>> handling the per-umem cq/fq, I can create a dedicated thread to 
>>>> process cq/fq.
>>>> So for example:
>>>> Thread 1 for handling cq/fq
>>>> Thread 2 for processing queue1 tx/rx queue Thread 3 for processing 
>>>> queue2 tx/rx queue and the mempool should allow multiple producer 
>>>> and consumer.
>>>>
>>>> Does this sound correct?
>>>
>>> You do not need a dedicated process. Just something in the mempool 
>>> code that enforces mutual exclusion (a mutex or whatever) between 
>>> thread 2 and 3 when they are performing operations on the mempool.
>>> Going with a dedicated process sounds complicated.
>>
>> I was trying to see how to experiment with this using libbpf, but 
>> looks like it’s not yet supported?
>>
>> Is see the following in xsk_socket__create():
>>
>> 475         if (umem->refcount) {
>> 476                 pr_warning("Error: shared umems not supported by
>> libbpf.\n");
>> 477                 return -EBUSY;
>> 478         }
>>
>
> Using the XDP_SHARED_UMEM option is not supported in libbpf at this 
> point in time. In this mode you share a single umem with a single 
> completion queue and a single fill queue among many xsk sockets tied 
> to the same queue id. But note that you can register the same umem 
> area multiple times (creating multiple umem handles and multiple fqs 
> and cqs) to be able to support xsk sockets that have different queue 
> ids, but the same umem area. In both cases you need a mempool that can 
> handle multiple threads.

Thinking about libmempool with umem shared among various independent processes, that would be great.
So that multiple processes could share same NIC or even queue if registered with all necessary locking in libmempool.
But what if one process crashes? Wondering how to achieve proper cleanup and if it is even possible with architecture I mentioned.
Maybe with some monitoring thread, but that's complicated. Any ideas?

> The old xdpsock application prior to libbpf had support for the 
> XDP_SHARED_UMEM option. Take a look at that one if you would like to 
> experiment with it.
>
> /Magnus

Július

Re: AF_XDP integration with FDio VPP? (Was: Questions about XDP)

[Magnus Karlsson]

On Thu, Oct 10, 2019 at 11:30 AM Július Milan
<Julius.Milan@pantheon.tech> wrote:
>
> On 30 Sep 2019, at 13:02, Magnus Karlsson wrote:
>
> > On Mon, Sep 30, 2019 at 11:28 AM Eelco Chaudron <echaudro@redhat.com>
> > wrote:
> >>
> >>
> >>
> >> On 30 Sep 2019, at 8:51, Magnus Karlsson wrote:
> >>
> >>> On Fri, Sep 27, 2019 at 8:09 PM William Tu <u9012063@gmail.com>
> >>> wrote:
> >>>>
> >>>> On Fri, Sep 27, 2019 at 12:02 AM Magnus Karlsson
> >>>> <magnus.karlsson@gmail.com> wrote:
> >>>>>
> >>>>> On Thu, Sep 26, 2019 at 1:34 AM William Tu <u9012063@gmail.com>
> >>>>> wrote:
> >>>>>>
> >>>>>> On Wed, Sep 25, 2019 at 12:48 AM Eelco Chaudron
> >>>>>> <echaudro@redhat.com> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On 25 Sep 2019, at 8:46, Július Milan wrote:
> >>>>>>>
> >>>>>>>> Hi Eelco
> >>>>>>>>
> >>>>>>>>> Currently, OVS uses the mmaped memory directly, however on
> >>>>>>>>> egress, it is copying the memory to the egress interface it’s
> >>>>>>>>> mmaped memory.
> >>>>>>>> Great, thanks for making this clear to me.
> >>>>>>>>
> >>>>>>>>> Currently, OVS uses an AF_XDP memory pool per interface, so a
> >>>>>>>>> further optimization could be to use a global memory pool so
> >>>>>>>>> this extra copy is not needed.
> >>>>>>>> Is it even possible to make this further optimization? Since
> >>>>>>>> every interface has it's own non-shared umem, so from my point
> >>>>>>>> of view, at least one copy for case as you described above
> >>>>>>>> (when RX interface is different then TX interface) is
> >>>>>>>> necessery. Or am I missing something?
> >>>>>>>
> >>>>>>> Some one @Intel told me it would be possible to have one huge
> >>>>>>> mempool that can be shared between interfaces. However I have
> >>>>>>> not researched/tried it.
> >>>>>>
> >>>>>> I thought about it before, but the problem is cq and fq are
> >>>>>> per-umem.
> >>>>>> So when having only one umem shared with many queues or devices,
> >>>>>> each one has to acquire a lock, then they can access cq or fq. I
> >>>>>> think that might become much slower.
> >>>>>
> >>>>> You basically have to implement a mempool that can be used by
> >>>>> multiple processes. Unfortunately, there is no lean and mean
> >>>>> standalone implementation of a mempool. There is a good one in
> >>>>> DPDK, but then you get the whole DPDK package into your
> >>>>> application which is likely what you wanted to avoid in the first
> >>>>> place. Anyone for writing libmempool?
> >>>>>
> >>>>> /Magnus
> >>>>>
> >>>>
> >>>> That's interesting.
> >>>> Do you mean the DPDK's rte_mempool which supports
> >>>> multiple-producer?
> >>>
> >>> Yes.
> >>>
> >>>> If I create a shared umem for queue1  and queue2, then each queue
> >>>> has its own tx/rx ring so they can process in parallel. But for
> >>>> handling the per-umem cq/fq, I can create a dedicated thread to
> >>>> process cq/fq.
> >>>> So for example:
> >>>> Thread 1 for handling cq/fq
> >>>> Thread 2 for processing queue1 tx/rx queue Thread 3 for processing
> >>>> queue2 tx/rx queue and the mempool should allow multiple producer
> >>>> and consumer.
> >>>>
> >>>> Does this sound correct?
> >>>
> >>> You do not need a dedicated process. Just something in the mempool
> >>> code that enforces mutual exclusion (a mutex or whatever) between
> >>> thread 2 and 3 when they are performing operations on the mempool.
> >>> Going with a dedicated process sounds complicated.
> >>
> >> I was trying to see how to experiment with this using libbpf, but
> >> looks like it’s not yet supported?
> >>
> >> Is see the following in xsk_socket__create():
> >>
> >> 475         if (umem->refcount) {
> >> 476                 pr_warning("Error: shared umems not supported by
> >> libbpf.\n");
> >> 477                 return -EBUSY;
> >> 478         }
> >>
> >
> > Using the XDP_SHARED_UMEM option is not supported in libbpf at this
> > point in time. In this mode you share a single umem with a single
> > completion queue and a single fill queue among many xsk sockets tied
> > to the same queue id. But note that you can register the same umem
> > area multiple times (creating multiple umem handles and multiple fqs
> > and cqs) to be able to support xsk sockets that have different queue
> > ids, but the same umem area. In both cases you need a mempool that can
> > handle multiple threads.
>
> Thinking about libmempool with umem shared among various independent processes, that would be great.
> So that multiple processes could share same NIC or even queue if registered with all necessary locking in libmempool.
> But what if one process crashes? Wondering how to achieve proper cleanup and if it is even possible with architecture I mentioned.
> Maybe with some monitoring thread, but that's complicated. Any ideas?

That is a correct observation. Dealing with failures when processes
share memory is hard. Much easier with a private memory model, but
that usually has negative performance implications. For some
inspiration, you can check "man pthread_mutexattr_getrobust" and how
robust mutexes can be use when a process holding the mutex dies. Makes
the problem a little bit more tractable.

> > The old xdpsock application prior to libbpf had support for the
> > XDP_SHARED_UMEM option. Take a look at that one if you would like to
> > experiment with it.
> >
> > /Magnus
>
> Július