Why choosing AF_XDP (busy polling) over DPDK

Why choosing AF_XDP (busy polling) over DPDK

[Federico Parola]

Hello everyone,
in a context in which all traffic is sent to user space, and the main 
concern is performance (so AF_XDP busy polling is the best candidate), 
is there a reason to choose AF_XDP instead of DPDK for packet I/O, given 
that the latter is still much faster?

Best regards,
Federico Parola

Re: Why choosing AF_XDP (busy polling) over DPDK

[Magnus Karlsson]

On Wed, Dec 14, 2022 at 9:16 AM Federico Parola
<federico.parola@polito.it> wrote:
>
> Hello everyone,
> in a context in which all traffic is sent to user space, and the main
> concern is performance (so AF_XDP busy polling is the best candidate),
> is there a reason to choose AF_XDP instead of DPDK for packet I/O, given
> that the latter is still much faster?

The choice is not AF_XDP vs DPDK, since DPDK can run on top of AF_XDP.
The choice is do you want to use user-space drivers or kernel space
drivers. There are many pros and cons with both approaches. The two
main advantages of user-space drivers in DPDK are that they are faster
than their kernel counterparts and you get access to many NIC features
that are not available right now from user-space if you are using a
kernel driver. Kernel drivers, on the other hand, enables you to
deploy your application on any Linux system (from a certain version)
and does not force you into the strict execution model of DPDK (which
is what most people use with user-space drivers). Everything in Linux
is still usable and it is easy to have a system that mixes data-plane
processing with other types of workloads. If the only thing you care
about is max performance of a pure packet processing workload on a
system you have complete control over, the choice is easy: go with a
user-space driver and DPDK. If this is not your ultimate goal, then
XDP and AF_XDP might be something to consider.

There are likely some good write-ups on the Internet about this.

/Magnus

> Best regards,
> Federico Parola

Re: Why choosing AF_XDP (busy polling) over DPDK

[Federico Parola]

On 12/14/22 10:00, Magnus Karlsson wrote:
> On Wed, Dec 14, 2022 at 9:16 AM Federico Parola
> <federico.parola@polito.it> wrote:
>>
>> Hello everyone,
>> in a context in which all traffic is sent to user space, and the main
>> concern is performance (so AF_XDP busy polling is the best candidate),
>> is there a reason to choose AF_XDP instead of DPDK for packet I/O, given
>> that the latter is still much faster?
> 
> The choice is not AF_XDP vs DPDK, since DPDK can run on top of AF_XDP.
> The choice is do you want to use user-space drivers or kernel space
> drivers. There are many pros and cons with both approaches. The two
> main advantages of user-space drivers in DPDK are that they are faster
> than their kernel counterparts and you get access to many NIC features
> that are not available right now from user-space if you are using a
> kernel driver. Kernel drivers, on the other hand, enables you to
> deploy your application on any Linux system (from a certain version)
> and does not force you into the strict execution model of DPDK (which
> is what most people use with user-space drivers). Everything in Linux
> is still usable and it is easy to have a system that mixes data-plane
> processing with other types of workloads. If the only thing you care
> about is max performance of a pure packet processing workload on a
> system you have complete control over, the choice is easy: go with a
> user-space driver and DPDK. If this is not your ultimate goal, then
> XDP and AF_XDP might be something to consider.
Thanks for your answer Magnus.

- Concerning deploying an application on any Linux system, what could be 
the limitations of DPDK userspace drivers that AF_XDP doesn't have? The 
use of hugepages?

- Concerning a system that mixes data-plane processing with other types 
of workloads, do you refer only to the capability of XDP to steer 
traffic either towards userspace with AF_XDP or let it proceed to the 
kernel? Or are there other features of DPDK userspace drivers that would 
impact this kind of system? Maybe the use of busy polling is too 
aggressive to share the CPU with other workloads?

> 
> There are likely some good write-ups on the Internet about this.
> 
> /Magnus
> 
>> Best regards,
>> Federico Parola

Re: Why choosing AF_XDP (busy polling) over DPDK

[Magnus Karlsson]

On Wed, Dec 14, 2022 at 4:16 PM Federico Parola
<federico.parola@polito.it> wrote:
>
> On 12/14/22 10:00, Magnus Karlsson wrote:
> > On Wed, Dec 14, 2022 at 9:16 AM Federico Parola
> > <federico.parola@polito.it> wrote:
> >>
> >> Hello everyone,
> >> in a context in which all traffic is sent to user space, and the main
> >> concern is performance (so AF_XDP busy polling is the best candidate),
> >> is there a reason to choose AF_XDP instead of DPDK for packet I/O, given
> >> that the latter is still much faster?
> >
> > The choice is not AF_XDP vs DPDK, since DPDK can run on top of AF_XDP.
> > The choice is do you want to use user-space drivers or kernel space
> > drivers. There are many pros and cons with both approaches. The two
> > main advantages of user-space drivers in DPDK are that they are faster
> > than their kernel counterparts and you get access to many NIC features
> > that are not available right now from user-space if you are using a
> > kernel driver. Kernel drivers, on the other hand, enables you to
> > deploy your application on any Linux system (from a certain version)
> > and does not force you into the strict execution model of DPDK (which
> > is what most people use with user-space drivers). Everything in Linux
> > is still usable and it is easy to have a system that mixes data-plane
> > processing with other types of workloads. If the only thing you care
> > about is max performance of a pure packet processing workload on a
> > system you have complete control over, the choice is easy: go with a
> > user-space driver and DPDK. If this is not your ultimate goal, then
> > XDP and AF_XDP might be something to consider.
> Thanks for your answer Magnus.
>
> - Concerning deploying an application on any Linux system, what could be
> the limitations of DPDK userspace drivers that AF_XDP doesn't have? The
> use of hugepages?

That would be one. The requirement of shared memory, forced unity
mmap() calls that might fail, lack of binary forward compatibility,
etc.

> - Concerning a system that mixes data-plane processing with other types
> of workloads, do you refer only to the capability of XDP to steer
> traffic either towards userspace with AF_XDP or let it proceed to the
> kernel? Or are there other features of DPDK userspace drivers that would
> impact this kind of system? Maybe the use of busy polling is too
> aggressive to share the CPU with other workloads?

Yes, busy-polling and sharing a core is not a good idea in any system.
But there is a good solution to this in DPDK that uses capabilities in
newer CPUs. Basically, if you use DPDK you should adhere to its strict
model of one process per core, otherwise you will likely get into
trouble. Either by performance dropping significantly because you
enable things like interrupts or 4K pages in DPDK, or because you do
things like trying to run to DPDK polling threads on the same core,
which might result in deadlock. If I am not mistaken, there is a
concept in DPDK called service cores in which you can share a core
between a normal process and a DPDK data plane thread. But it requires
you to program specifically for it.

If you want to steer part of the traffic to the kernel, you are better
off using XDP. Bouncing things through user-space is expensive, DPDK
or not.

> >
> > There are likely some good write-ups on the Internet about this.
> >
> > /Magnus
> >
> >> Best regards,
> >> Federico Parola

Re: Why choosing AF_XDP (busy polling) over DPDK

[Andrey Slastenov]

Hello everyone

Check out this short article about why XDP was chosen over DPDK. The
reasoning for this decision is also explained.

https://gcore.com/blog/how-we-use-regular-expressions-in-xdp-for-packet-filtering/


BR, Andrey


ср, 14 дек. 2022 г. в 16:20, Federico Parola <federico.parola@polito.it>:
>
> On 12/14/22 10:00, Magnus Karlsson wrote:
> > On Wed, Dec 14, 2022 at 9:16 AM Federico Parola
> > <federico.parola@polito.it> wrote:
> >>
> >> Hello everyone,
> >> in a context in which all traffic is sent to user space, and the main
> >> concern is performance (so AF_XDP busy polling is the best candidate),
> >> is there a reason to choose AF_XDP instead of DPDK for packet I/O, given
> >> that the latter is still much faster?
> >
> > The choice is not AF_XDP vs DPDK, since DPDK can run on top of AF_XDP.
> > The choice is do you want to use user-space drivers or kernel space
> > drivers. There are many pros and cons with both approaches. The two
> > main advantages of user-space drivers in DPDK are that they are faster
> > than their kernel counterparts and you get access to many NIC features
> > that are not available right now from user-space if you are using a
> > kernel driver. Kernel drivers, on the other hand, enables you to
> > deploy your application on any Linux system (from a certain version)
> > and does not force you into the strict execution model of DPDK (which
> > is what most people use with user-space drivers). Everything in Linux
> > is still usable and it is easy to have a system that mixes data-plane
> > processing with other types of workloads. If the only thing you care
> > about is max performance of a pure packet processing workload on a
> > system you have complete control over, the choice is easy: go with a
> > user-space driver and DPDK. If this is not your ultimate goal, then
> > XDP and AF_XDP might be something to consider.
> Thanks for your answer Magnus.
>
> - Concerning deploying an application on any Linux system, what could be
> the limitations of DPDK userspace drivers that AF_XDP doesn't have? The
> use of hugepages?
>
> - Concerning a system that mixes data-plane processing with other types
> of workloads, do you refer only to the capability of XDP to steer
> traffic either towards userspace with AF_XDP or let it proceed to the
> kernel? Or are there other features of DPDK userspace drivers that would
> impact this kind of system? Maybe the use of busy polling is too
> aggressive to share the CPU with other workloads?
>
> >
> > There are likely some good write-ups on the Internet about this.
> >
> > /Magnus
> >
> >> Best regards,
> >> Federico Parola


ср, 14 дек. 2022 г. в 16:20, Federico Parola <federico.parola@polito.it>:
>
> On 12/14/22 10:00, Magnus Karlsson wrote:
> > On Wed, Dec 14, 2022 at 9:16 AM Federico Parola
> > <federico.parola@polito.it> wrote:
> >>
> >> Hello everyone,
> >> in a context in which all traffic is sent to user space, and the main
> >> concern is performance (so AF_XDP busy polling is the best candidate),
> >> is there a reason to choose AF_XDP instead of DPDK for packet I/O, given
> >> that the latter is still much faster?
> >
> > The choice is not AF_XDP vs DPDK, since DPDK can run on top of AF_XDP.
> > The choice is do you want to use user-space drivers or kernel space
> > drivers. There are many pros and cons with both approaches. The two
> > main advantages of user-space drivers in DPDK are that they are faster
> > than their kernel counterparts and you get access to many NIC features
> > that are not available right now from user-space if you are using a
> > kernel driver. Kernel drivers, on the other hand, enables you to
> > deploy your application on any Linux system (from a certain version)
> > and does not force you into the strict execution model of DPDK (which
> > is what most people use with user-space drivers). Everything in Linux
> > is still usable and it is easy to have a system that mixes data-plane
> > processing with other types of workloads. If the only thing you care
> > about is max performance of a pure packet processing workload on a
> > system you have complete control over, the choice is easy: go with a
> > user-space driver and DPDK. If this is not your ultimate goal, then
> > XDP and AF_XDP might be something to consider.
> Thanks for your answer Magnus.
>
> - Concerning deploying an application on any Linux system, what could be
> the limitations of DPDK userspace drivers that AF_XDP doesn't have? The
> use of hugepages?
>
> - Concerning a system that mixes data-plane processing with other types
> of workloads, do you refer only to the capability of XDP to steer
> traffic either towards userspace with AF_XDP or let it proceed to the
> kernel? Or are there other features of DPDK userspace drivers that would
> impact this kind of system? Maybe the use of busy polling is too
> aggressive to share the CPU with other workloads?
>
> >
> > There are likely some good write-ups on the Internet about this.
> >
> > /Magnus
> >
> >> Best regards,
> >> Federico Parola



-- 


Andrey Slastenov
Network Expert
CCIE #19983
Email:a.slastenov@gmail.com

Re: Why choosing AF_XDP (busy polling) over DPDK

[Federico Parola]

On 12/14/22 16:31, Magnus Karlsson wrote:
> On Wed, Dec 14, 2022 at 4:16 PM Federico Parola
> <federico.parola@polito.it> wrote:
>>
>> On 12/14/22 10:00, Magnus Karlsson wrote:
>>> On Wed, Dec 14, 2022 at 9:16 AM Federico Parola
>>> <federico.parola@polito.it> wrote:
>>>>
>>>> Hello everyone,
>>>> in a context in which all traffic is sent to user space, and the main
>>>> concern is performance (so AF_XDP busy polling is the best candidate),
>>>> is there a reason to choose AF_XDP instead of DPDK for packet I/O, given
>>>> that the latter is still much faster?
>>>
>>> The choice is not AF_XDP vs DPDK, since DPDK can run on top of AF_XDP.
>>> The choice is do you want to use user-space drivers or kernel space
>>> drivers. There are many pros and cons with both approaches. The two
>>> main advantages of user-space drivers in DPDK are that they are faster
>>> than their kernel counterparts and you get access to many NIC features
>>> that are not available right now from user-space if you are using a
>>> kernel driver. Kernel drivers, on the other hand, enables you to
>>> deploy your application on any Linux system (from a certain version)
>>> and does not force you into the strict execution model of DPDK (which
>>> is what most people use with user-space drivers). Everything in Linux
>>> is still usable and it is easy to have a system that mixes data-plane
>>> processing with other types of workloads. If the only thing you care
>>> about is max performance of a pure packet processing workload on a
>>> system you have complete control over, the choice is easy: go with a
>>> user-space driver and DPDK. If this is not your ultimate goal, then
>>> XDP and AF_XDP might be something to consider.
>> Thanks for your answer Magnus.
>>
>> - Concerning deploying an application on any Linux system, what could be
>> the limitations of DPDK userspace drivers that AF_XDP doesn't have? The
>> use of hugepages?
> 
> That would be one. The requirement of shared memory, forced unity
> mmap() calls that might fail, lack of binary forward compatibility,
> etc.
> 
>> - Concerning a system that mixes data-plane processing with other types
>> of workloads, do you refer only to the capability of XDP to steer
>> traffic either towards userspace with AF_XDP or let it proceed to the
>> kernel? Or are there other features of DPDK userspace drivers that would
>> impact this kind of system? Maybe the use of busy polling is too
>> aggressive to share the CPU with other workloads?
> 
> Yes, busy-polling and sharing a core is not a good idea in any system.
> But there is a good solution to this in DPDK that uses capabilities in
> newer CPUs. Basically, if you use DPDK you should adhere to its strict
> model of one process per core, otherwise you will likely get into
> trouble. Either by performance dropping significantly because you
> enable things like interrupts or 4K pages in DPDK, or because you do
> things like trying to run to DPDK polling threads on the same core,
> which might result in deadlock. If I am not mistaken, there is a
> concept in DPDK called service cores in which you can share a core
> between a normal process and a DPDK data plane thread. But it requires
> you to program specifically for it.
> 
> If you want to steer part of the traffic to the kernel, you are better
> off using XDP. Bouncing things through user-space is expensive, DPDK
> or not.
> 
>>>
>>> There are likely some good write-ups on the Internet about this.
>>>
>>> /Magnus
>>>
>>>> Best regards,
>>>> Federico Parola

Thank you very much for this information Magnus.

On 12/14/22 20:46, Andrey Slastenov wrote:> Hello everyone
 >
 > Check out this short article about why XDP was chosen over DPDK. The
 > reasoning for this decision is also explained.
 >
 > 
https://gcore.com/blog/how-we-use-regular-expressions-in-xdp-for-packet-filtering/
 >
 >
 > BR, Andrey

Thanks for sharing this article Andrey.

Best regards,
Federico