RE: [PATCH v3 0/7] introduce per-queue limit watermark and host shaper

[Spike Du <spiked@nvidia.com>]

> -----Original Message-----
> From: Morten Brørup <mb@smartsharesystems.com>
> Sent: Wednesday, May 25, 2022 3:00 AM
> To: NBU-Contact-Thomas Monjalon (EXTERNAL) <thomas@monjalon.net>;
> Spike Du <spiked@nvidia.com>
> Cc: Matan Azrad <matan@nvidia.com>; Slava Ovsiienko
> <viacheslavo@nvidia.com>; Ori Kam <orika@nvidia.com>; dev@dpdk.org;
> Raslan Darawsheh <rasland@nvidia.com>; stephen@networkplumber.org;
> andrew.rybchenko@oktetlabs.ru; ferruh.yigit@amd.com;
> david.marchand@redhat.com
> Subject: RE: [PATCH v3 0/7] introduce per-queue limit watermark and host
> shaper
> 
> External email: Use caution opening links or attachments
> 
> 
> > From: Thomas Monjalon [mailto:thomas@monjalon.net]
> > Sent: Tuesday, 24 May 2022 17.59
> >
> > +Cc people involved in previous versions
> >
> > 24/05/2022 17:20, Spike Du:
> > > LWM(limit watermark) is per RX queue attribute, when RX queue
> > fullness reach the LWM limit, HW sends an event to dpdk application.
> > > Host shaper can configure shaper rate and lwm-triggered for a host
> > port.
> 
> Please ignore this comment, it is not important, but I had to get it out of my
> system: I assume that the "LWM" name is from the NIC datasheet; otherwise
> I would probably prefer something with "threshold"... LWM is easily
> confused with "low water mark", which is the opposite of what the LWM
> does. Names are always open for discussion, so I won't object to it.
> 
> > > The shaper limits the rate of traffic from host port to wire port.
> 
> From host to wire? It is RX, so you must mean from wire to host.

The host shaper is quite private to Nvidia's BlueField 2 NIC. The NIC is inserted
In a server which we call it host-system, and the NIC has an embedded Arm-system
Which does the forwarding.
The traffic flows from host-system to wire like this:
Host-system generates traffic, send it to Arm-system, Arm sends it to physical/wire port.
So the RX happens between host-system and Arm-system, and the traffic is host to wire.
The shaper also works in a special way: you configure it on Arm-system, but it takes effect
On host-sysmem's TX side. 

> 
> > > If lwm-triggered is enabled, a 100Mbps shaper is enabled
> > automatically when one of the host port's Rx queues receives LWM event.
> > >
> > > These two features can combine to control traffic from host port to
> > wire port.
> 
> Again, you mean from wire to host?

Pls see above.

> 
> > > The work flow is configure LWM to RX queue and enable lwm-triggered
> > flag in host shaper, after receiving LWM event, delay a while until RX
> > queue is empty , then disable the shaper. We recycle this work flow to
> > reduce RX queue drops.
> 
> You delay while RX queue gets drained by some other threads, I assume.

The PMD thread drains the Rx queue, the PMD receiving  as normal, as the PMD
Implementation uses rte interrupt thread to handle LWM event.

> 
> Surely, the excess packets must be dropped somewhere, e.g. by the shaper?
> 
> > >
> > > Add new libethdev API to set LWM, add rte event
> > RTE_ETH_EVENT_RXQ_LIMIT_REACHED to handle LWM event.
> 
> Makes sense to make it public; could be usable for other purposes, similar to
> interrupt coalescing, as mentioned by Stephen.
> 
> > > For host shaper,
> > because it doesn't align to existing DPDK framework and is specific to
> > Nvidia NIC, use PMD private API.
> 
> Makes sense to keep it private.
> 
> > >
> > > For integration with testpmd, put the private cmdline function and
> > LWM event handler in mlx5 PMD directory by adding a new file
> > mlx5_test.c. Only add minimal code in testpmd to invoke interfaces
> > from mlx5_test.c.
> > >
> > > Spike Du (7):
> > >   net/mlx5: add LWM support for Rxq
> > >   common/mlx5: share interrupt management
> > >   ethdev: introduce Rx queue based limit watermark
> > >   net/mlx5: add LWM event handling support
> > >   net/mlx5: support Rx queue based limit watermark
> > >   net/mlx5: add private API to config host port shaper
> > >   app/testpmd: add LWM and Host Shaper command
> > >