[RFC] xdp, multiple rss objects

[RFC] xdp, multiple rss objects

[Jonathan Lemon]

Background:

With AF_XDP, packets are placed on queues which are serviced directly from
an application running in user space, which promises greatly increased
performance.  However, not all incoming packets should be delivered to the
application; there may be management traffic, or packets which should be
handled by normal Linux networking applications.

Having the AF_XDP application handle the packets and return them to the
kernel for processing is not ideal, nor is adding a bpf filter at the kernel
level which performs the redirection.  It would be nicer if the redirection
was done in the NIC itself.  Fortunately, there is a facility for this: the
RX network flow classification.

Unfortunately, the target of the classification is currently a single queue.

Traffic arriving at a NIC which is not steered to a specific queue can be
distributed across a set of queues through the RX flow hash redirection
table (RSS steering).  But being able to target a separate RSS object would
allow arbitrary distribution of previously classified traffic.

Ask:

It would be nice to have more than one RSS table, which would be the target
of a classifier or default packet stream.

Queues would be attached to an RSS object, and flow classifiers could target
the RSS object for further distribution.  For example, something like:

  ethtool -X eth0 rss 0 queue 0-13
  ethtool -X eth0 rss 1 queue 14-55
  ethtool -N eth0 flow-type udp6 dst-port 4242 rss 1

This would set up a default RSS object (rss 0), which sprays traffic to 14
queues; this would be equivalent to "ethtool -X eth0 equal 14".  At the same
time, a second object (rss 1) is set up, distributing traffic across another
set of queues.

Is there interest in adding this type of feature?  Having the ability to
express this concept (not necessarily in ethtool) should facilitate having
HW vendors add this to their products.  I believe that some vendors have
hardware which may be capable of this already.
-- 
Jonathan

Re: [RFC] xdp, multiple rss objects

[Jakub Kicinski]

On Fri, 29 Mar 2019 16:14:57 +0000, Jonathan Lemon wrote:
> Background:
> 
> With AF_XDP, packets are placed on queues which are serviced directly from
> an application running in user space, which promises greatly increased
> performance.  However, not all incoming packets should be delivered to the
> application; there may be management traffic, or packets which should be
> handled by normal Linux networking applications.
> 
> Having the AF_XDP application handle the packets and return them to the
> kernel for processing is not ideal, nor is adding a bpf filter at the kernel
> level which performs the redirection.  It would be nicer if the redirection
> was done in the NIC itself.  Fortunately, there is a facility for this: the
> RX network flow classification.
> 
> Unfortunately, the target of the classification is currently a single queue.
> 
> Traffic arriving at a NIC which is not steered to a specific queue can be
> distributed across a set of queues through the RX flow hash redirection
> table (RSS steering).  But being able to target a separate RSS object would
> allow arbitrary distribution of previously classified traffic.
> 
> Ask:
> 
> It would be nice to have more than one RSS table, which would be the target
> of a classifier or default packet stream.
> 
> Queues would be attached to an RSS object, and flow classifiers could target
> the RSS object for further distribution.  For example, something like:
> 
>   ethtool -X eth0 rss 0 queue 0-13
>   ethtool -X eth0 rss 1 queue 14-55
>   ethtool -N eth0 flow-type udp6 dst-port 4242 rss 1
> 
> This would set up a default RSS object (rss 0), which sprays traffic to 14
> queues; this would be equivalent to "ethtool -X eth0 equal 14".  At the same
> time, a second object (rss 1) is set up, distributing traffic across another
> set of queues.
> 
> Is there interest in adding this type of feature?  Having the ability to
> express this concept (not necessarily in ethtool) should facilitate having
> HW vendors add this to their products.  I believe that some vendors have
> hardware which may be capable of this already.

I thought Ed Cree added something like that to ethtool - multiple RSS
contexts that is: commit 84a1d9c48200 ("net: ethtool: extend RXNFC API
to support RSS spreading of filter matches").

With Netronome NICs you could also program your RSS with offloaded BPF.

Unfortunately only Intel supports AF_XDP right now so that probably
doesn't help you :(

Re: [RFC] xdp, multiple rss objects

[Jonathan Lemon]

On 29 Mar 2019, at 14:24, Jakub Kicinski wrote:

> On Fri, 29 Mar 2019 16:14:57 +0000, Jonathan Lemon wrote:
>> Background:
>>
>> With AF_XDP, packets are placed on queues which are serviced directly 
>> from
>> an application running in user space, which promises greatly 
>> increased
>> performance.  However, not all incoming packets should be delivered 
>> to the
>> application; there may be management traffic, or packets which should 
>> be
>> handled by normal Linux networking applications.
>>
>> Having the AF_XDP application handle the packets and return them to 
>> the
>> kernel for processing is not ideal, nor is adding a bpf filter at the 
>> kernel
>> level which performs the redirection.  It would be nicer if the 
>> redirection
>> was done in the NIC itself.  Fortunately, there is a facility for 
>> this: the
>> RX network flow classification.
>>
>> Unfortunately, the target of the classification is currently a single 
>> queue.
>>
>> Traffic arriving at a NIC which is not steered to a specific queue 
>> can be
>> distributed across a set of queues through the RX flow hash 
>> redirection
>> table (RSS steering).  But being able to target a separate RSS object 
>> would
>> allow arbitrary distribution of previously classified traffic.
>>
>> Ask:
>>
>> It would be nice to have more than one RSS table, which would be the 
>> target
>> of a classifier or default packet stream.
>>
>> Queues would be attached to an RSS object, and flow classifiers could 
>> target
>> the RSS object for further distribution.  For example, something 
>> like:
>>
>>   ethtool -X eth0 rss 0 queue 0-13
>>   ethtool -X eth0 rss 1 queue 14-55
>>   ethtool -N eth0 flow-type udp6 dst-port 4242 rss 1
>>
>> This would set up a default RSS object (rss 0), which sprays traffic 
>> to 14
>> queues; this would be equivalent to "ethtool -X eth0 equal 14".  At 
>> the same
>> time, a second object (rss 1) is set up, distributing traffic across 
>> another
>> set of queues.
>>
>> Is there interest in adding this type of feature?  Having the ability 
>> to
>> express this concept (not necessarily in ethtool) should facilitate 
>> having
>> HW vendors add this to their products.  I believe that some vendors 
>> have
>> hardware which may be capable of this already.
>
> I thought Ed Cree added something like that to ethtool - multiple RSS
> contexts that is: commit 84a1d9c48200 ("net: ethtool: extend RXNFC API
> to support RSS spreading of filter matches").
>
> With Netronome NICs you could also program your RSS with offloaded 
> BPF.
>
> Unfortunately only Intel supports AF_XDP right now so that probably
> doesn't help you :(

Thanks - I didn't know about the ethtool changes.  This does look fairly
close to what I'm looking for, but I'm not seeing the ability to specify
a distribution of (start, length).
-- 
Jonathan

Re: [RFC] xdp, multiple rss objects

[William Tu]

Hi Jonathan,

Sorry I don't have good suggestions but I have a couple of questions.

On Fri, Mar 29, 2019 at 9:17 AM Jonathan Lemon <bsd@fb.com> wrote:
>
> Background:
>
> With AF_XDP, packets are placed on queues which are serviced directly from
> an application running in user space, which promises greatly increased
> performance.  However, not all incoming packets should be delivered to the
> application; there may be management traffic, or packets which should be
> handled by normal Linux networking applications.

I thought we can put a XDP program to filter for AF_XDP.
So pass the management traffic to normal linux networking stack, and
other traffic to AF_XDP socket. Does this help?

>
> Having the AF_XDP application handle the packets and return them to the
> kernel for processing is not ideal, nor is adding a bpf filter at the kernel
> level which performs the redirection.  It would be nicer if the redirection
> was done in the NIC itself.  Fortunately, there is a facility for this: the
> RX network flow classification.
>
May I know your use case for packets handled in AF_XDP, and then
return/re-inject to kernel?
Our case is that when packets received from AF_XDP netdev, we want
to forward to a vm (tun/tap) or container (veth), so have to reinject to kernel.

Regards,
William

> Unfortunately, the target of the classification is currently a single queue.
>
> Traffic arriving at a NIC which is not steered to a specific queue can be
> distributed across a set of queues through the RX flow hash redirection
> table (RSS steering).  But being able to target a separate RSS object would
> allow arbitrary distribution of previously classified traffic.
>
> Ask:
>
> It would be nice to have more than one RSS table, which would be the target
> of a classifier or default packet stream.
>
> Queues would be attached to an RSS object, and flow classifiers could target
> the RSS object for further distribution.  For example, something like:
>
>   ethtool -X eth0 rss 0 queue 0-13
>   ethtool -X eth0 rss 1 queue 14-55
>   ethtool -N eth0 flow-type udp6 dst-port 4242 rss 1
>
> This would set up a default RSS object (rss 0), which sprays traffic to 14
> queues; this would be equivalent to "ethtool -X eth0 equal 14".  At the same
> time, a second object (rss 1) is set up, distributing traffic across another
> set of queues.
>
> Is there interest in adding this type of feature?  Having the ability to
> express this concept (not necessarily in ethtool) should facilitate having
> HW vendors add this to their products.  I believe that some vendors have
> hardware which may be capable of this already.
> --
> Jonathan

Re: [RFC] xdp, multiple rss objects

[Jonathan Lemon]

On 30 Mar 2019, at 8:52, William Tu wrote:

> Hi Jonathan,
>
> Sorry I don't have good suggestions but I have a couple of questions.
>
> On Fri, Mar 29, 2019 at 9:17 AM Jonathan Lemon <bsd@fb.com> wrote:
>>
>> Background:
>>
>> With AF_XDP, packets are placed on queues which are serviced directly 
>> from
>> an application running in user space, which promises greatly 
>> increased
>> performance.  However, not all incoming packets should be delivered 
>> to the
>> application; there may be management traffic, or packets which should 
>> be
>> handled by normal Linux networking applications.
>
> I thought we can put a XDP program to filter for AF_XDP.
> So pass the management traffic to normal linux networking stack, and
> other traffic to AF_XDP socket. Does this help?

It's not an ideal situation - the CPU still has to handle the redirected
traffic.


>> Having the AF_XDP application handle the packets and return them to 
>> the
>> kernel for processing is not ideal, nor is adding a bpf filter at the 
>> kernel
>> level which performs the redirection.  It would be nicer if the 
>> redirection
>> was done in the NIC itself.  Fortunately, there is a facility for 
>> this: the
>> RX network flow classification.
>>
> May I know your use case for packets handled in AF_XDP, and then
> return/re-inject to kernel?
> Our case is that when packets received from AF_XDP netdev, we want
> to forward to a vm (tun/tap) or container (veth), so have to reinject 
> to kernel.

One of the cases I'm targeting is QUIC, another is a custom application. 
  In
both cases these AF_XDP applications are not the only thing running on 
the box.
Packets for these applications would be received, and then sent out 
directly,
while packets for other apps would be handled via the kernel normally.
-- 
Jonathan