AF_XDP umem and jumbo frames?

AF_XDP umem and jumbo frames?

[Alex Forster]

What happens in the case where umem's chunk size is smaller than the
NIC frame size, like when jumbo frames are enabled? I.e, a NIC using
MTU ~9000 on an x86-64 machine with a page size of 4096.

Alex Forster

Re: AF_XDP umem and jumbo frames?

[Justin Azoff]

I don't know if this affects AF_XDP as well, but XDP does not support
jumbo frames at all:

https://prototype-kernel.readthedocs.io/en/latest/networking/XDP/design/requirements.html#page-per-packet

On my test box running 4.18 if XDP is in use the MTU can not be set
higher than 3050.

On Mon, Sep 24, 2018 at 4:53 PM Alex Forster <aforster@cloudflare.com> wrote:
>
> What happens in the case where umem's chunk size is smaller than the
> NIC frame size, like when jumbo frames are enabled? I.e, a NIC using
> MTU ~9000 on an x86-64 machine with a page size of 4096.
>
> Alex Forster



-- 
- Justin

Re: AF_XDP umem and jumbo frames?

[Alex Forster]

> On my test box running 4.18 if XDP is in use the MTU can not be set higher than 3050.

Ah, that answers a few questions for me. Thanks!

Alex Forster

Re: AF_XDP umem and jumbo frames?

[Rob Sherwood]

Thanks for the reference and the page-per-packet point makes sense.
At the same time, not supporting jumbo frames seems like a non-trivial
limitation.  Are there a subset of drivers that do support jumbo
frames (or LRO or the other features that require multiple pages per
packet)?

Thanks in advance,

- Rob
.
On Tue, Sep 25, 2018 at 9:44 AM Alex Forster <aforster@cloudflare.com> wrote:
>
> > On my test box running 4.18 if XDP is in use the MTU can not be set higher than 3050.
>
> Ah, that answers a few questions for me. Thanks!
>
> Alex Forster

Re: AF_XDP umem and jumbo frames?

[Björn Töpel]

Den tors 27 sep. 2018 kl 02:56 skrev Rob Sherwood <rob.sherwood@gmail.com>:
>
> Thanks for the reference and the page-per-packet point makes sense.
> At the same time, not supporting jumbo frames seems like a non-trivial
> limitation.  Are there a subset of drivers that do support jumbo
> frames (or LRO or the other features that require multiple pages per
> packet)?
>

No, not at the moment. XDP has a strict "one frame cannot exceed a
page" constraint. Everything that applies to XDP in terms of
constraints, applies to AF_XDP as well.

Just to clarify, XDP supports jumbo frames -- i.e. larger than 1500B
payload, just not the maximum 9000B size. My personal observation is
that many deployments that "require jumbo frames", are usually OK with
an of MTU ~3000B. Jumbo frames, yes. Full jumbo frames, no. :-)


Cheers,
Björn


> Thanks in advance,
>
> - Rob
> .
> On Tue, Sep 25, 2018 at 9:44 AM Alex Forster <aforster@cloudflare.com> wrote:
> >
> > > On my test box running 4.18 if XDP is in use the MTU can not be set higher than 3050.
> >
> > Ah, that answers a few questions for me. Thanks!
> >
> > Alex Forster

Re: AF_XDP umem and jumbo frames?

[Jesper Dangaard Brouer]

On Thu, 4 Oct 2018 08:44:27 +0200
Björn Töpel <bjorn.topel@gmail.com> wrote:

> Den tors 27 sep. 2018 kl 02:56 skrev Rob Sherwood <rob.sherwood@gmail.com>:
> >
> > Thanks for the reference and the page-per-packet point makes sense.
> > At the same time, not supporting jumbo frames seems like a non-trivial
> > limitation.  Are there a subset of drivers that do support jumbo
> > frames (or LRO or the other features that require multiple pages per
> > packet)?
> >  
> 
> No, not at the moment. XDP has a strict "one frame cannot exceed a
> page" constraint. Everything that applies to XDP in terms of
> constraints, applies to AF_XDP as well.
> 
> Just to clarify, XDP supports jumbo frames -- i.e. larger than 1500B
> payload, just not the maximum 9000B size. My personal observation is
> that many deployments that "require jumbo frames", are usually OK with
> an of MTU ~3000B. Jumbo frames, yes. Full jumbo frames, no. :-)

Thank you for clarifying that Bjørn.

Can Alex or Rob explain:
 
(1) What is your use-case for wanting jumbo-frames?
 
And (2) will an MTU of ~3000Bytes be sufficient? (which XDP does support)


> > On Tue, Sep 25, 2018 at 9:44 AM Alex Forster <aforster@cloudflare.com> wrote:  
> > >  
> > > > On my test box running 4.18 if XDP is in use the MTU can not be
> > > > set higher than 3050.  
> > >
> > > Ah, that answers a few questions for me. Thanks!
> > >
> > > Alex Forster  

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Re: AF_XDP umem and jumbo frames?

[Justin Azoff]

On Thu, Oct 4, 2018 at 3:52 AM Jesper Dangaard Brouer <brouer@redhat.com> wrote:
> Can Alex or Rob explain:
>
> (1) What is your use-case for wanting jumbo-frames?
>
> And (2) will an MTU of ~3000Bytes be sufficient? (which XDP does support)

Not Alex or Rob, but lack of jumbo frames is currently a blocker for using XDP.

I'm working on https://github.com/JustinAzoff/libflowbypass which is similar to
the bypass stuff suricata can do, but implemented as a standalone library.

We have a number of high performance networks that use jumbo frames and the XDP
NICs sit on fiber taps.  So it's not so much that I want to use jumbo frames
but that some of the traffic I need to analyze is using them.  For
this same reason 3000
bytes isn't enough since I see 9000 byte frames.

If XDP could implicitly XDP_PASS jumbo frames instead of truncating them at
3000 that would at least allow the use of XDP to inspect and drop the
majority of  < 3000
byte frames.

-- 
- Justin

Re: AF_XDP umem and jumbo frames?

[Alex Forster]

> Just to clarify, XDP supports jumbo frames -- i.e. larger than 1500B payload, just not the maximum 9000B size. My personal observation is that many deployments that "require jumbo frames", are usually OK with an of MTU ~3000B. Jumbo frames, yes. Full jumbo frames, no. :-)

Right, that's where my confusion was. My understanding was limited to
"XDP supports jumbo frames" until I got here[1] in the umem
implementation ("computer says no") and started questioning myself.

> Can Alex or Rob explain:
>
> (1) What is your use-case for wanting jumbo-frames?
>
> And (2) will an MTU of ~3000Bytes be sufficient? (which XDP does support)

I don't personally have a need for MTU9K right now. ~3K is totally
sufficient for the usecases I have/know of.

Food for thought, though: I think MTU9K is considered a hard
requirement in the storage space.

Alex Forster

[1]: https://github.com/torvalds/linux/blob/v4.19-rc4/net/xdp/xdp_umem.c#L278

On Thu, Oct 4, 2018 at 2:52 AM Jesper Dangaard Brouer <brouer@redhat.com> wrote:
>
> On Thu, 4 Oct 2018 08:44:27 +0200
> Björn Töpel <bjorn.topel@gmail.com> wrote:
>
> > Den tors 27 sep. 2018 kl 02:56 skrev Rob Sherwood <rob.sherwood@gmail.com>:
> > >
> > > Thanks for the reference and the page-per-packet point makes sense.
> > > At the same time, not supporting jumbo frames seems like a non-trivial
> > > limitation.  Are there a subset of drivers that do support jumbo
> > > frames (or LRO or the other features that require multiple pages per
> > > packet)?
> > >
> >
> > No, not at the moment. XDP has a strict "one frame cannot exceed a
> > page" constraint. Everything that applies to XDP in terms of
> > constraints, applies to AF_XDP as well.
> >
> > Just to clarify, XDP supports jumbo frames -- i.e. larger than 1500B
> > payload, just not the maximum 9000B size. My personal observation is
> > that many deployments that "require jumbo frames", are usually OK with
> > an of MTU ~3000B. Jumbo frames, yes. Full jumbo frames, no. :-)
>
> Thank you for clarifying that Bjørn.
>
> Can Alex or Rob explain:
>
> (1) What is your use-case for wanting jumbo-frames?
>
> And (2) will an MTU of ~3000Bytes be sufficient? (which XDP does support)
>
>
> > > On Tue, Sep 25, 2018 at 9:44 AM Alex Forster <aforster@cloudflare.com> wrote:
> > > >
> > > > > On my test box running 4.18 if XDP is in use the MTU can not be
> > > > > set higher than 3050.
> > > >
> > > > Ah, that answers a few questions for me. Thanks!
> > > >
> > > > Alex Forster
>
> --
> Best regards,
>   Jesper Dangaard Brouer
>   MSc.CS, Principal Kernel Engineer at Red Hat
>   LinkedIn: http://www.linkedin.com/in/brouer

Re: AF_XDP umem and jumbo frames?

[Rob Sherwood]

[not speaking for my current employer, but just from past experience ]

Certainly a lot of the 'hard' requirements (hard meaning - "without
this it won't work")  I've seen could be served with a ~3k non-full
jumbo frame.  But at least what I've seen in the past was that because
many of the host-side operations are per-packet limited (e.g., because
of CPU or RAM, but ultimately turns into a max pps per host), a
trivial way to increase application performance/reduce CPU for
networking was to run at as large a frame size as possible.  For
example, if your application/host is really pps limited, then getting
the frame size to increase from 3k to 9k means either 3x more
bandwidth for the same cpu usage (assuming the application is
bandwidth limited) or 1/3x the CPU usage for the same bandwidth (if
the application is not bandwidth limited).  Either way, IMHO it's a
pretty big win.

- Rob
.
On Thu, Oct 4, 2018 at 12:52 AM Jesper Dangaard Brouer
<brouer@redhat.com> wrote:
>
> On Thu, 4 Oct 2018 08:44:27 +0200
> Björn Töpel <bjorn.topel@gmail.com> wrote:
>
> > Den tors 27 sep. 2018 kl 02:56 skrev Rob Sherwood <rob.sherwood@gmail.com>:
> > >
> > > Thanks for the reference and the page-per-packet point makes sense.
> > > At the same time, not supporting jumbo frames seems like a non-trivial
> > > limitation.  Are there a subset of drivers that do support jumbo
> > > frames (or LRO or the other features that require multiple pages per
> > > packet)?
> > >
> >
> > No, not at the moment. XDP has a strict "one frame cannot exceed a
> > page" constraint. Everything that applies to XDP in terms of
> > constraints, applies to AF_XDP as well.
> >
> > Just to clarify, XDP supports jumbo frames -- i.e. larger than 1500B
> > payload, just not the maximum 9000B size. My personal observation is
> > that many deployments that "require jumbo frames", are usually OK with
> > an of MTU ~3000B. Jumbo frames, yes. Full jumbo frames, no. :-)
>
> Thank you for clarifying that Bjørn.
>
> Can Alex or Rob explain:
>
> (1) What is your use-case for wanting jumbo-frames?
>
> And (2) will an MTU of ~3000Bytes be sufficient? (which XDP does support)
>
>
> > > On Tue, Sep 25, 2018 at 9:44 AM Alex Forster <aforster@cloudflare.com> wrote:
> > > >
> > > > > On my test box running 4.18 if XDP is in use the MTU can not be
> > > > > set higher than 3050.
> > > >
> > > > Ah, that answers a few questions for me. Thanks!
> > > >
> > > > Alex Forster
>
> --
> Best regards,
>   Jesper Dangaard Brouer
>   MSc.CS, Principal Kernel Engineer at Red Hat
>   LinkedIn: http://www.linkedin.com/in/brouer

Re: AF_XDP umem and jumbo frames?

[Jesper Dangaard Brouer]

On Thu, 4 Oct 2018 08:47:45 -0700
Rob Sherwood <rob.sherwood@gmail.com> wrote:

> [not speaking for my current employer, but just from past experience ]
> 
> Certainly a lot of the 'hard' requirements (hard meaning - "without
> this it won't work")  I've seen could be served with a ~3k non-full
> jumbo frame. 

Glad to hear that _most_ use-cases can be solved with a ~3k non-full
jumbo-frame.

> But at least what I've seen in the past was that because
> many of the host-side operations are per-packet limited (e.g., because
> of CPU or RAM, but ultimately turns into a max pps per host), a
> trivial way to increase application performance/reduce CPU for
> networking was to run at as large a frame size as possible.  For
> example, if your application/host is really pps limited, then getting
> the frame size to increase from 3k to 9k means either 3x more
> bandwidth for the same cpu usage (assuming the application is
> bandwidth limited) or 1/3x the CPU usage for the same bandwidth (if
> the application is not bandwidth limited).  Either way, IMHO it's a
> pretty big win.

With XDP we have basically solved the issue of being PPS (packets per
sec) limited.  And we can avoid these workarounds of using jumbo frames.
That is why it is a bit provoking to ask for jumbo-frames ;-)


People on this list might not realize that there is a significant
overhead in supporting larger that 4K frames for XDP, that is larger
than one memory-page. So let me explain...

It is actually trivially easy for XDP to support jumbo frames, if the
NIC hardware supports storing RX frames into higher order pages (aka
compound pages, more 4K pages physically after each-other) which most
HW does. (Page order0 = 4KB, order1=8KB, order2=16KB, order3=32KB).
As then XDP will work out-of-the-box, as the requirement is really that
packet-payload is layout as phys continuous memory.

Kernel page allocator can give us high-order pages, sure, but is cost
more, see slide 12 of [1].  The large jump to order-1, is because
order-0 have a Per-Cpu-Pages (PCP) cache.  From order-1 and above, the
page allocator goes through a central (per NUMA) lock, which makes
thing even worse, as this does not scale to multiple CPUs.  And there
is also the point of wasting memory when processing 64Byte packets.
So, it is not 100% of the picture, that we could support jumbo-frames
for XDP.  Mostly because we can workaround this cost/issue, by having
recycle caches for these pages, which we even do for order-0 pages.
Hint, I actually left this door open, as you can specify page-order
when setting up the page_pool API in the driver...

[1] http://people.netfilter.org/hawk/presentations/MM-summit2017/MM-summit2017-JesperBrouer.pdf

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer




> On Thu, Oct 4, 2018 at 12:52 AM Jesper Dangaard Brouer
> <brouer@redhat.com> wrote:
> >
> > On Thu, 4 Oct 2018 08:44:27 +0200
> > Björn Töpel <bjorn.topel@gmail.com> wrote:
> >  
> > > Den tors 27 sep. 2018 kl 02:56 skrev Rob Sherwood <rob.sherwood@gmail.com>:  
> > > >
> > > > Thanks for the reference and the page-per-packet point makes sense.
> > > > At the same time, not supporting jumbo frames seems like a non-trivial
> > > > limitation.  Are there a subset of drivers that do support jumbo
> > > > frames (or LRO or the other features that require multiple pages per
> > > > packet)?
> > > >  
> > >
> > > No, not at the moment. XDP has a strict "one frame cannot exceed a
> > > page" constraint. Everything that applies to XDP in terms of
> > > constraints, applies to AF_XDP as well.
> > >
> > > Just to clarify, XDP supports jumbo frames -- i.e. larger than 1500B
> > > payload, just not the maximum 9000B size. My personal observation is
> > > that many deployments that "require jumbo frames", are usually OK with
> > > an of MTU ~3000B. Jumbo frames, yes. Full jumbo frames, no. :-)  
> >
> > Thank you for clarifying that Bjørn.
> >
> > Can Alex or Rob explain:
> >
> > (1) What is your use-case for wanting jumbo-frames?
> >
> > And (2) will an MTU of ~3000Bytes be sufficient? (which XDP does support)
> >
> >  
> > > > On Tue, Sep 25, 2018 at 9:44 AM Alex Forster <aforster@cloudflare.com> wrote:  
> > > > >  
> > > > > > On my test box running 4.18 if XDP is in use the MTU can not be
> > > > > > set higher than 3050.  
> > > > >
> > > > > Ah, that answers a few questions for me. Thanks!
> > > > >
> > > > > Alex Forster  
> >
> > --
> > Best regards,
> >   Jesper Dangaard Brouer
> >   MSc.CS, Principal Kernel Engineer at Red Hat
> >   LinkedIn: http://www.linkedin.com/in/brouer  

Re: AF_XDP umem and jumbo frames?

[Zvi Effron]

If the requirement is just for contiguous memory, could this be
resolved with allowing the driver to request multiple contiguous 4KB
pages instead of one higher order page? Does that reduce the cost? (Or
does it actually increase it?)

--Zvi
On Thu, Oct 4, 2018 at 12:45 PM Jesper Dangaard Brouer
<brouer@redhat.com> wrote:
>
> On Thu, 4 Oct 2018 08:47:45 -0700
> Rob Sherwood <rob.sherwood@gmail.com> wrote:
>
> > [not speaking for my current employer, but just from past experience ]
> >
> > Certainly a lot of the 'hard' requirements (hard meaning - "without
> > this it won't work")  I've seen could be served with a ~3k non-full
> > jumbo frame.
>
> Glad to hear that _most_ use-cases can be solved with a ~3k non-full
> jumbo-frame.
>
> > But at least what I've seen in the past was that because
> > many of the host-side operations are per-packet limited (e.g., because
> > of CPU or RAM, but ultimately turns into a max pps per host), a
> > trivial way to increase application performance/reduce CPU for
> > networking was to run at as large a frame size as possible.  For
> > example, if your application/host is really pps limited, then getting
> > the frame size to increase from 3k to 9k means either 3x more
> > bandwidth for the same cpu usage (assuming the application is
> > bandwidth limited) or 1/3x the CPU usage for the same bandwidth (if
> > the application is not bandwidth limited).  Either way, IMHO it's a
> > pretty big win.
>
> With XDP we have basically solved the issue of being PPS (packets per
> sec) limited.  And we can avoid these workarounds of using jumbo frames.
> That is why it is a bit provoking to ask for jumbo-frames ;-)
>
>
> People on this list might not realize that there is a significant
> overhead in supporting larger that 4K frames for XDP, that is larger
> than one memory-page. So let me explain...
>
> It is actually trivially easy for XDP to support jumbo frames, if the
> NIC hardware supports storing RX frames into higher order pages (aka
> compound pages, more 4K pages physically after each-other) which most
> HW does. (Page order0 = 4KB, order1=8KB, order2=16KB, order3=32KB).
> As then XDP will work out-of-the-box, as the requirement is really that
> packet-payload is layout as phys continuous memory.
>
> Kernel page allocator can give us high-order pages, sure, but is cost
> more, see slide 12 of [1].  The large jump to order-1, is because
> order-0 have a Per-Cpu-Pages (PCP) cache.  From order-1 and above, the
> page allocator goes through a central (per NUMA) lock, which makes
> thing even worse, as this does not scale to multiple CPUs.  And there
> is also the point of wasting memory when processing 64Byte packets.
> So, it is not 100% of the picture, that we could support jumbo-frames
> for XDP.  Mostly because we can workaround this cost/issue, by having
> recycle caches for these pages, which we even do for order-0 pages.
> Hint, I actually left this door open, as you can specify page-order
> when setting up the page_pool API in the driver...
>
> [1] http://people.netfilter.org/hawk/presentations/MM-summit2017/MM-summit2017-JesperBrouer.pdf
>
> --
> Best regards,
>   Jesper Dangaard Brouer
>   MSc.CS, Principal Kernel Engineer at Red Hat
>   LinkedIn: http://www.linkedin.com/in/brouer
>
>
>
>
> > On Thu, Oct 4, 2018 at 12:52 AM Jesper Dangaard Brouer
> > <brouer@redhat.com> wrote:
> > >
> > > On Thu, 4 Oct 2018 08:44:27 +0200
> > > Björn Töpel <bjorn.topel@gmail.com> wrote:
> > >
> > > > Den tors 27 sep. 2018 kl 02:56 skrev Rob Sherwood <rob.sherwood@gmail.com>:
> > > > >
> > > > > Thanks for the reference and the page-per-packet point makes sense.
> > > > > At the same time, not supporting jumbo frames seems like a non-trivial
> > > > > limitation.  Are there a subset of drivers that do support jumbo
> > > > > frames (or LRO or the other features that require multiple pages per
> > > > > packet)?
> > > > >
> > > >
> > > > No, not at the moment. XDP has a strict "one frame cannot exceed a
> > > > page" constraint. Everything that applies to XDP in terms of
> > > > constraints, applies to AF_XDP as well.
> > > >
> > > > Just to clarify, XDP supports jumbo frames -- i.e. larger than 1500B
> > > > payload, just not the maximum 9000B size. My personal observation is
> > > > that many deployments that "require jumbo frames", are usually OK with
> > > > an of MTU ~3000B. Jumbo frames, yes. Full jumbo frames, no. :-)
> > >
> > > Thank you for clarifying that Bjørn.
> > >
> > > Can Alex or Rob explain:
> > >
> > > (1) What is your use-case for wanting jumbo-frames?
> > >
> > > And (2) will an MTU of ~3000Bytes be sufficient? (which XDP does support)
> > >
> > >
> > > > > On Tue, Sep 25, 2018 at 9:44 AM Alex Forster <aforster@cloudflare.com> wrote:
> > > > > >
> > > > > > > On my test box running 4.18 if XDP is in use the MTU can not be
> > > > > > > set higher than 3050.
> > > > > >
> > > > > > Ah, that answers a few questions for me. Thanks!
> > > > > >
> > > > > > Alex Forster
> > >
> > > --
> > > Best regards,
> > >   Jesper Dangaard Brouer
> > >   MSc.CS, Principal Kernel Engineer at Red Hat
> > >   LinkedIn: http://www.linkedin.com/in/brouer
>
>

Re: AF_XDP umem and jumbo frames?

[Justin Azoff]

> People on this list might not realize that there is a significant
> overhead in supporting larger that 4K frames for XDP, that is larger
> than one memory-page. So let me explain...
>
> It is actually trivially easy for XDP to support jumbo frames, if the
> NIC hardware supports storing RX frames into higher order pages (aka
> compound pages, more 4K pages physically after each-other) which most
> HW does. (Page order0 = 4KB, order1=8KB, order2=16KB, order3=32KB).
> As then XDP will work out-of-the-box, as the requirement is really that
> packet-payload is layout as phys continuous memory.
>

For the use cases of XDP_DROP or XDP_PASS, could XDP send as much of
the packet that fits in a single page up to the ebpf program and allow
decisions based on that?

For the flow bypass, ddos drop stuff, you only need the l3 header to
make the PASS/DROP decision, not the entire packet.

I suppose this would be a bit more complicated for modifying headers
and using XDP_TX.

-- 
- Justin

Re: AF_XDP umem and jumbo frames?

[Rob Sherwood]

Thanks Jesper for the detailed and thoughtful answer!

I definitely understand that XDP-based applications significantly
reduce per packet overhead and thus XDP-based applications don't need
jumbo frames.  I guess I was thinking more about the performance of
non-XDP applications that still benefit from jumbo frames.  A lot of
the (at least to me) interesting applications of XDP are as
infrastructure to other applications, e.g., XDP-based NATs, firewalls,
load-balancers, etc. that pass packets generated by non-XDP traffic
from web servers, streaming video, etc.  But if those other
applications benefit from jumbo frames to work efficiently, then if we
want to use an XDP-based application, then they can't use jumbo frames
either.  Does that make sense?

I admit that part of my thinking (implicit assumption) is that I've
never seen IP fragmentation work efficiently (either on the
fragmentation or defragmentation side), but I guess it's in theory
possible that an application could source a jumbo framed packet and
something else could fragment it and then an XDP-app could manage it.
Is this more along the lines of what you were thinking?

Thanks again for the detailed reply!

- Rob
.
On Thu, Oct 4, 2018 at 12:44 PM Jesper Dangaard Brouer
<brouer@redhat.com> wrote:
>
> On Thu, 4 Oct 2018 08:47:45 -0700
> Rob Sherwood <rob.sherwood@gmail.com> wrote:
>
> > [not speaking for my current employer, but just from past experience ]
> >
> > Certainly a lot of the 'hard' requirements (hard meaning - "without
> > this it won't work")  I've seen could be served with a ~3k non-full
> > jumbo frame.
>
> Glad to hear that _most_ use-cases can be solved with a ~3k non-full
> jumbo-frame.
>
> > But at least what I've seen in the past was that because
> > many of the host-side operations are per-packet limited (e.g., because
> > of CPU or RAM, but ultimately turns into a max pps per host), a
> > trivial way to increase application performance/reduce CPU for
> > networking was to run at as large a frame size as possible.  For
> > example, if your application/host is really pps limited, then getting
> > the frame size to increase from 3k to 9k means either 3x more
> > bandwidth for the same cpu usage (assuming the application is
> > bandwidth limited) or 1/3x the CPU usage for the same bandwidth (if
> > the application is not bandwidth limited).  Either way, IMHO it's a
> > pretty big win.
>
> With XDP we have basically solved the issue of being PPS (packets per
> sec) limited.  And we can avoid these workarounds of using jumbo frames.
> That is why it is a bit provoking to ask for jumbo-frames ;-)
>
>
> People on this list might not realize that there is a significant
> overhead in supporting larger that 4K frames for XDP, that is larger
> than one memory-page. So let me explain...
>
> It is actually trivially easy for XDP to support jumbo frames, if the
> NIC hardware supports storing RX frames into higher order pages (aka
> compound pages, more 4K pages physically after each-other) which most
> HW does. (Page order0 = 4KB, order1=8KB, order2=16KB, order3=32KB).
> As then XDP will work out-of-the-box, as the requirement is really that
> packet-payload is layout as phys continuous memory.
>
> Kernel page allocator can give us high-order pages, sure, but is cost
> more, see slide 12 of [1].  The large jump to order-1, is because
> order-0 have a Per-Cpu-Pages (PCP) cache.  From order-1 and above, the
> page allocator goes through a central (per NUMA) lock, which makes
> thing even worse, as this does not scale to multiple CPUs.  And there
> is also the point of wasting memory when processing 64Byte packets.
> So, it is not 100% of the picture, that we could support jumbo-frames
> for XDP.  Mostly because we can workaround this cost/issue, by having
> recycle caches for these pages, which we even do for order-0 pages.
> Hint, I actually left this door open, as you can specify page-order
> when setting up the page_pool API in the driver...
>
> [1] http://people.netfilter.org/hawk/presentations/MM-summit2017/MM-summit2017-JesperBrouer.pdf
>
> --
> Best regards,
>   Jesper Dangaard Brouer
>   MSc.CS, Principal Kernel Engineer at Red Hat
>   LinkedIn: http://www.linkedin.com/in/brouer
>
>
>
>
> > On Thu, Oct 4, 2018 at 12:52 AM Jesper Dangaard Brouer
> > <brouer@redhat.com> wrote:
> > >
> > > On Thu, 4 Oct 2018 08:44:27 +0200
> > > Björn Töpel <bjorn.topel@gmail.com> wrote:
> > >
> > > > Den tors 27 sep. 2018 kl 02:56 skrev Rob Sherwood <rob.sherwood@gmail.com>:
> > > > >
> > > > > Thanks for the reference and the page-per-packet point makes sense.
> > > > > At the same time, not supporting jumbo frames seems like a non-trivial
> > > > > limitation.  Are there a subset of drivers that do support jumbo
> > > > > frames (or LRO or the other features that require multiple pages per
> > > > > packet)?
> > > > >
> > > >
> > > > No, not at the moment. XDP has a strict "one frame cannot exceed a
> > > > page" constraint. Everything that applies to XDP in terms of
> > > > constraints, applies to AF_XDP as well.
> > > >
> > > > Just to clarify, XDP supports jumbo frames -- i.e. larger than 1500B
> > > > payload, just not the maximum 9000B size. My personal observation is
> > > > that many deployments that "require jumbo frames", are usually OK with
> > > > an of MTU ~3000B. Jumbo frames, yes. Full jumbo frames, no. :-)
> > >
> > > Thank you for clarifying that Bjørn.
> > >
> > > Can Alex or Rob explain:
> > >
> > > (1) What is your use-case for wanting jumbo-frames?
> > >
> > > And (2) will an MTU of ~3000Bytes be sufficient? (which XDP does support)
> > >
> > >
> > > > > On Tue, Sep 25, 2018 at 9:44 AM Alex Forster <aforster@cloudflare.com> wrote:
> > > > > >
> > > > > > > On my test box running 4.18 if XDP is in use the MTU can not be
> > > > > > > set higher than 3050.
> > > > > >
> > > > > > Ah, that answers a few questions for me. Thanks!
> > > > > >
> > > > > > Alex Forster
> > >
> > > --
> > > Best regards,
> > >   Jesper Dangaard Brouer
> > >   MSc.CS, Principal Kernel Engineer at Red Hat
> > >   LinkedIn: http://www.linkedin.com/in/brouer
>
>

Re: AF_XDP umem and jumbo frames?

[Jesper Dangaard Brouer]

On Fri, 5 Oct 2018 11:47:25 -0700 Zvi Effron <zeffron@riotgames.com> wrote:

> If the requirement is just for contiguous memory, could this be
> resolved with allowing the driver to request multiple contiguous 4KB
> pages instead of one higher order page? Does that reduce the cost? (Or
> does it actually increase it?)

Sorry, but your question does not make sense. A higher order page _is_
multiple contiguous 4KB pages. Thus, the answer is that it is the same.

--Jesper

> On Thu, Oct 4, 2018 at 12:45 PM Jesper Dangaard Brouer
> <brouer@redhat.com> wrote:
> >
> > On Thu, 4 Oct 2018 08:47:45 -0700
> > Rob Sherwood <rob.sherwood@gmail.com> wrote:
> >  
> > > [not speaking for my current employer, but just from past experience ]
> > >
> > > Certainly a lot of the 'hard' requirements (hard meaning - "without
> > > this it won't work")  I've seen could be served with a ~3k non-full
> > > jumbo frame.  
> >
> > Glad to hear that _most_ use-cases can be solved with a ~3k non-full
> > jumbo-frame.
> >  
> > > But at least what I've seen in the past was that because
> > > many of the host-side operations are per-packet limited (e.g., because
> > > of CPU or RAM, but ultimately turns into a max pps per host), a
> > > trivial way to increase application performance/reduce CPU for
> > > networking was to run at as large a frame size as possible.  For
> > > example, if your application/host is really pps limited, then getting
> > > the frame size to increase from 3k to 9k means either 3x more
> > > bandwidth for the same cpu usage (assuming the application is
> > > bandwidth limited) or 1/3x the CPU usage for the same bandwidth (if
> > > the application is not bandwidth limited).  Either way, IMHO it's a
> > > pretty big win.  
> >
> > With XDP we have basically solved the issue of being PPS (packets per
> > sec) limited.  And we can avoid these workarounds of using jumbo frames.
> > That is why it is a bit provoking to ask for jumbo-frames ;-)
> >
> >
> > People on this list might not realize that there is a significant
> > overhead in supporting larger that 4K frames for XDP, that is larger
> > than one memory-page. So let me explain...
> >
> > It is actually trivially easy for XDP to support jumbo frames, if the
> > NIC hardware supports storing RX frames into higher order pages (aka
> > compound pages, more 4K pages physically after each-other) which most
> > HW does. (Page order0 = 4KB, order1=8KB, order2=16KB, order3=32KB).
> > As then XDP will work out-of-the-box, as the requirement is really that
> > packet-payload is layout as phys continuous memory.
> >
> > Kernel page allocator can give us high-order pages, sure, but is cost
> > more, see slide 12 of [1].  The large jump to order-1, is because
> > order-0 have a Per-Cpu-Pages (PCP) cache.  From order-1 and above, the
> > page allocator goes through a central (per NUMA) lock, which makes
> > thing even worse, as this does not scale to multiple CPUs.  And there
> > is also the point of wasting memory when processing 64Byte packets.
> > So, it is not 100% of the picture, that we could support jumbo-frames
> > for XDP.  Mostly because we can workaround this cost/issue, by having
> > recycle caches for these pages, which we even do for order-0 pages.
> > Hint, I actually left this door open, as you can specify page-order
> > when setting up the page_pool API in the driver...
> >
> > [1] http://people.netfilter.org/hawk/presentations/MM-summit2017/MM-summit2017-JesperBrouer.pdf
> >
> > --
> > Best regards,
> >   Jesper Dangaard Brouer
> >   MSc.CS, Principal Kernel Engineer at Red Hat
> >   LinkedIn: http://www.linkedin.com/in/brouer
> >
> >
> >
> >  
> > > On Thu, Oct 4, 2018 at 12:52 AM Jesper Dangaard Brouer
> > > <brouer@redhat.com> wrote:  
> > > >
> > > > On Thu, 4 Oct 2018 08:44:27 +0200
> > > > Björn Töpel <bjorn.topel@gmail.com> wrote:
> > > >  
> > > > > Den tors 27 sep. 2018 kl 02:56 skrev Rob Sherwood <rob.sherwood@gmail.com>:  
> > > > > >
> > > > > > Thanks for the reference and the page-per-packet point makes sense.
> > > > > > At the same time, not supporting jumbo frames seems like a non-trivial
> > > > > > limitation.  Are there a subset of drivers that do support jumbo
> > > > > > frames (or LRO or the other features that require multiple pages per
> > > > > > packet)?
> > > > > >  
> > > > >
> > > > > No, not at the moment. XDP has a strict "one frame cannot exceed a
> > > > > page" constraint. Everything that applies to XDP in terms of
> > > > > constraints, applies to AF_XDP as well.
> > > > >
> > > > > Just to clarify, XDP supports jumbo frames -- i.e. larger than 1500B
> > > > > payload, just not the maximum 9000B size. My personal observation is
> > > > > that many deployments that "require jumbo frames", are usually OK with
> > > > > an of MTU ~3000B. Jumbo frames, yes. Full jumbo frames, no. :-)  
> > > >
> > > > Thank you for clarifying that Bjørn.
> > > >
> > > > Can Alex or Rob explain:
> > > >
> > > > (1) What is your use-case for wanting jumbo-frames?
> > > >
> > > > And (2) will an MTU of ~3000Bytes be sufficient? (which XDP does support)
> > > >
> > > >  
> > > > > > On Tue, Sep 25, 2018 at 9:44 AM Alex Forster <aforster@cloudflare.com> wrote:  
> > > > > > >  
> > > > > > > > On my test box running 4.18 if XDP is in use the MTU can not be
> > > > > > > > set higher than 3050.  
> > > > > > >
> > > > > > > Ah, that answers a few questions for me. Thanks!
> > > > > > >
> > > > > > > Alex Forster  
> > > >
> > > > --
> > > > Best regards,
> > > >   Jesper Dangaard Brouer
> > > >   MSc.CS, Principal Kernel Engineer at Red Hat
> > > >   LinkedIn: http://www.linkedin.com/in/brouer  
> >
> >  



-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Re: AF_XDP umem and jumbo frames?

[Jesper Dangaard Brouer]

On Fri, 5 Oct 2018 15:56:31 -0400
Justin Azoff <justin.azoff@gmail.com> wrote:

> > People on this list might not realize that there is a significant
> > overhead in supporting larger that 4K frames for XDP, that is larger
> > than one memory-page. So let me explain...
> >
> > It is actually trivially easy for XDP to support jumbo frames, if the
> > NIC hardware supports storing RX frames into higher order pages (aka
> > compound pages, more 4K pages physically after each-other) which most
> > HW does. (Page order0 = 4KB, order1=8KB, order2=16KB, order3=32KB).
> > As then XDP will work out-of-the-box, as the requirement is really that
> > packet-payload is layout as phys continuous memory.
> >  
> 
> For the use cases of XDP_DROP or XDP_PASS, could XDP send as much of
> the packet that fits in a single page up to the ebpf program and allow
> decisions based on that?
> 
> For the flow bypass, ddos drop stuff, you only need the l3 header to
> make the PASS/DROP decision, not the entire packet.
> 
> I suppose this would be a bit more complicated for modifying headers
> and using XDP_TX.

The key in your question is just "bit more complicated", then we can
support feature "X".  For XDP is designed for performance where every
nanosec counts.  Feature creep will slowly but surely kill this
performance edge.

I'll try to explain the overhead of jumbo-frame again, with another
angle.  XDP have gained performance up-front by saying we don't support
jumbo-frames. As instead of (per RX packet) allocating 3x 4KB pages, we
only need to alloc a single 4KB page.  That in itself is a huge
performance win.  Are you saying that you want a feature, that is used
in 1-5% use-cases, that in general is going to slowdown the baseline
performance of XDP?

One thing I realize is that people on this list, are perhaps not
familiar how NIC RX (via DMA) works.  On RX, we cannot know the RX
packet size up-front.  Thus, when filling the NIC RX-ring memory slots,
then we have to allocated room for the "worse-case", e.g. 9000Bytes is
minimum 3x4K=12K, and due to page-alloc limits min 4x4K=16K.  Thus,
regardless of packet length the alloc size is the same.  (I will not go
into detail on how different drivers tries to reduce this mem-overhead,
but only say that those tricks costs CPU cycles).

A last word of adding features to XDP: When adding features, I look
long and hard for ways that the features checks can be pushed to setup
time, rather than runtime.

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Re: AF_XDP umem and jumbo frames?

[Eric Leblond]

Hello Jesper,

On Sun, 2018-10-07 at 18:13 +0200, Jesper Dangaard Brouer wrote:
> On Fri, 5 Oct 2018 15:56:31 -0400
> Justin Azoff <justin.azoff@gmail.com> wrote:
> 
> > > People on this list might not realize that there is a significant
> > > overhead in supporting larger that 4K frames for XDP, that is
> > > larger
> > > than one memory-page. So let me explain...

...

> One thing I realize is that people on this list, are perhaps not
> familiar how NIC RX (via DMA) works.  On RX, we cannot know the RX
> packet size up-front.  Thus, when filling the NIC RX-ring memory
> slots,
> then we have to allocated room for the "worse-case", e.g. 9000Bytes
> is
> minimum 3x4K=12K, and due to page-alloc limits min 4x4K=16K.  Thus,
> regardless of packet length the alloc size is the same.  (I will not
> go
> into detail on how different drivers tries to reduce this mem-
> overhead,
> but only say that those tricks costs CPU cycles).
> 
> A last word of adding features to XDP: When adding features, I look
> long and hard for ways that the features checks can be pushed to
> setup
> time, rather than runtime.

I get your point about performance and really understand why they
matter but we have kind of an issue for pure packet sniffing here.

For this type of usage we get a copy of the organization network
traffic via methods such as TAP devices or port mirroring. The
consequences is that we receive the jumbo frames of other devices and
can not set their length. As we need a pure copy of packets to be able
to analyze the traffic correctly we have to be able to send the full
jumbo frame to userspace for analysis (or feature like file extraction
will not work).

Is there a possible workaround even if lower performance (kind of slow
path) ?

Best regards,
-- 
Eric Leblond <eric@regit.org>

Re: AF_XDP umem and jumbo frames?

[Johannes Berg]

On Sun, 2018-10-07 at 18:13 +0200, Jesper Dangaard Brouer wrote:

> One thing I realize is that people on this list, are perhaps not
> familiar how NIC RX (via DMA) works.  On RX, we cannot know the RX
> packet size up-front.  Thus, when filling the NIC RX-ring memory slots,
> then we have to allocated room for the "worse-case", e.g. 9000Bytes is
> minimum 3x4K=12K, and due to page-alloc limits min 4x4K=16K.  Thus,
> regardless of packet length the alloc size is the same.  (I will not go
> into detail on how different drivers tries to reduce this mem-overhead,
> but only say that those tricks costs CPU cycles).

Isn't all that generally only true if the NIC has no capability for
split packets? Some (wireless) NICs will - afaict - simply split the
larger packets into multiple single DMA descriptors (pages).

Obviously then that isn't contiguous memory, but it seems that part
could be solved by "mumble mumble something like rewriting eBPF accesses
at skb->data+4k to virtual memory" - which yes, would be *tremendously*
expensive once your program actually does that, but shouldn't actually
cause the general case to be worse off?

(Yes, I also realize that there's a question of being able to prove that
the access is >=4k, but in general I'd assume you can prove that it's
significantly less than 4k and thus not punt to such a slow-path)

johannes