Offloading Priority Tables for queue classification

Offloading Priority Tables for queue classification

[Maxime Chevallier]

Hello everyone,

I'm starting this thread before any kind of series submission to
discuss on the manner with which we could deal with Queue
Classification in NICs, and more specifically how we could handle
internal classification tables like DSCP, VLAN Prio and any other
tables that we can populate in modern NICs to quickly assign a priority
to both egress and ingress traffic, and use that priority to select a
queue in which the packet will be enqueued/dequeued from.

The use-case we have in mind is to offload all of these classification
steps into a switch , where traffic would be classified internally
into queues, that we could configure for Frame-Preemption or any other
QoS techniques. I know that Frame Preemption itself and the way to
configure it is still under discussion, with debate on where to
configure the queue preemptability (hence why some of you ended-up CC'd
to that thread) :

https://lore.kernel.org/netdev/20210626003314.3159402-1-vinicius.gomes@intel.com/
  
There are already ways to do this classification, but from what
I've gathered, it looks like it's scattered across several places :

 - In TC, we can of course use TC flower for that. We can neatly decide
   which flows goes where, match on any of the fields that we can use
   to determine the priority of the packet. This however scales poorly
   when the underlying hardware uses tables dedicated only to matching
   specific fields, to assign each DSCP or VLAN a priority.

   TC flower works well when we want to use a full-featured
   classifier, using a TCAM of some sort combined with complete
   classification rules. Using TC flower to configure such tables would
   mean entering one rule per entry in our tables, which could work for
   VLAN prio, but not that much for DSCP tables for example.

 - TC skbedit with the priority offloading is exactly what we want to
   achieve, that is to emulate the skb->priority behaviour that we can
   configure with various ways, and map this priority to queues with
   mqpriofor example. tc-skbedit priority when offloaded handles that
   notion of "packet priority" that is used internally in a switch.

 - TC mqprio and TC taprio can be used for the actual prio->queue
   mapping, even though there's the "traffic class" layer sitting in
   the middle.

 - It looks like DCB could be a way to go to configure the DSCP/VLAN
   prio/any other QoS tables, since we can configure all of these tables
   with the "dcb app" command, which then calls hooks into the driver to
   configure offloading of these tables. Using DCB for this is perfect,
   since the traffic to prio assignment really is independant to the
   mqprio layer.

 - Finally for the last part of the chain, we can setup the queues for
   PFC or Frame-Preemption, possibly using ethtool as suggested in the
   above-mentionned thread.

So in the end, my question would be : Is it OK to mix all of these
together ? Using the dcb layer to configure the internal mapping of
traffic -> priority, then using mqprio to configure the priority ->
queue mapping, and then either TC again or ethtool do configure the
behaviour of the queues themselves ? Or is there some other way that
we've missed ?

Thanks in advance,

Best regards,

Maxime

Re: Offloading Priority Tables for queue classification

[Vladimir Oltean]

Hi Maxime!

On Fri, Apr 15, 2022 at 05:37:18PM +0200, Maxime Chevallier wrote:
> Hello everyone,
> 
> I'm starting this thread before any kind of series submission to
> discuss on the manner with which we could deal with Queue
> Classification in NICs, and more specifically how we could handle
> internal classification tables like DSCP, VLAN Prio and any other
> tables that we can populate in modern NICs to quickly assign a priority
> to both egress and ingress traffic, and use that priority to select a
> queue in which the packet will be enqueued/dequeued from.
> 
> The use-case we have in mind is to offload all of these classification
> steps into a switch , where traffic would be classified internally
> into queues, that we could configure for Frame-Preemption or any other
> QoS techniques. I know that Frame Preemption itself and the way to
> configure it is still under discussion, with debate on where to
> configure the queue preemptability (hence why some of you ended-up CC'd
> to that thread) :
> 
> https://lore.kernel.org/netdev/20210626003314.3159402-1-vinicius.gomes@intel.com/
>   
> There are already ways to do this classification, but from what
> I've gathered, it looks like it's scattered across several places :
> 
>  - In TC, we can of course use TC flower for that. We can neatly decide
>    which flows goes where, match on any of the fields that we can use
>    to determine the priority of the packet. This however scales poorly
>    when the underlying hardware uses tables dedicated only to matching
>    specific fields, to assign each DSCP or VLAN a priority.
> 
>    TC flower works well when we want to use a full-featured
>    classifier, using a TCAM of some sort combined with complete
>    classification rules. Using TC flower to configure such tables would
>    mean entering one rule per entry in our tables, which could work for
>    VLAN prio, but not that much for DSCP tables for example.
> 
>  - TC skbedit with the priority offloading is exactly what we want to
>    achieve, that is to emulate the skb->priority behaviour that we can
>    configure with various ways, and map this priority to queues with
>    mqpriofor example. tc-skbedit priority when offloaded handles that
>    notion of "packet priority" that is used internally in a switch.
> 
>  - TC mqprio and TC taprio can be used for the actual prio->queue
>    mapping, even though there's the "traffic class" layer sitting in
>    the middle.
> 
>  - It looks like DCB could be a way to go to configure the DSCP/VLAN
>    prio/any other QoS tables, since we can configure all of these tables
>    with the "dcb app" command, which then calls hooks into the driver to
>    configure offloading of these tables. Using DCB for this is perfect,
>    since the traffic to prio assignment really is independant to the
>    mqprio layer.
> 
>  - Finally for the last part of the chain, we can setup the queues for
>    PFC or Frame-Preemption, possibly using ethtool as suggested in the
>    above-mentionned thread.
> 
> So in the end, my question would be : Is it OK to mix all of these
> together ? Using the dcb layer to configure the internal mapping of
> traffic -> priority, then using mqprio to configure the priority ->
> queue mapping, and then either TC again or ethtool do configure the
> behaviour of the queues themselves ? Or is there some other way that
> we've missed ?

I think it's ok to mix all of those together. At least the ocelot/felix
DSA switches do support both advance QoS classification using tc-flower
+ skbedit priority action, and basic QoS classification (port-based or
IP DSCP based) using the dcbnl application table. In short, at the end
of the QoS classification process, a traffic class for the packet is
selected. Then, the frame preemption would operate on the packet's
traffic class.

Do you have any particular concerns?

Re: Offloading Priority Tables for queue classification

[Maxime Chevallier]

Hi Vladimir !

On Fri, 15 Apr 2022 15:42:29 +0000
Vladimir Oltean <vladimir.oltean@nxp.com> wrote:

> Hi Maxime!
> 
> On Fri, Apr 15, 2022 at 05:37:18PM +0200, Maxime Chevallier wrote:
> > So in the end, my question would be : Is it OK to mix all of these
> > together ? Using the dcb layer to configure the internal mapping of
> > traffic -> priority, then using mqprio to configure the priority ->
> > queue mapping, and then either TC again or ethtool do configure the
> > behaviour of the queues themselves ? Or is there some other way that
> > we've missed ?  
> 
> I think it's ok to mix all of those together. At least the
> ocelot/felix DSA switches do support both advance QoS classification
> using tc-flower
> + skbedit priority action, and basic QoS classification (port-based or
> IP DSCP based) using the dcbnl application table. In short, at the end
> of the QoS classification process, a traffic class for the packet is
> selected. Then, the frame preemption would operate on the packet's
> traffic class.

Thank you very much for that answer, it helps a lot. TBH when digging
into classification, especially with TC, it's a bit overwhelming and it
gets difficult pretty quickly to get what's the right approach.

> Do you have any particular concerns?

My concerns were mainly about not reinventing the wheel, but also
making sure that I have the correct understanding on these
classification steps. I'll make sure to CC you when I have a first
series implementing that.

Best regards,

Maxime