AF_XDP RX processing with NO rx-ring

AF_XDP RX processing with NO rx-ring

[Srivats P]

Hi,

I want to clarify my AF_XDP understanding for a particular scenario.

Consider a single UMEM with 1 Fill Ring and 1 Completion Ring and
single XDP socket  bound to queue 0 with 1 Tx Ring but NO Rx Ring.
Assume the NIC has only a single queue (to keep things simple for
explaining this scenario).

There is a XDP program attached to Queue 0 which does either a
XDP_DROP or XDP_PASS for all Rx packets.

We are running in Driver mode.

What happens to RX packets received on Queue 0?

Here's my understanding -

 * Even though there is no AF_XDP Rx Ring, there will be a NIC Rx Ring
for queue 0
 * The NIC Rx Ring for queue 0 is populated by the driver with UMEM
buffers taken from the Fill ring
 * A Rx packet will be received in the NIC Rx Ring for Queue 0 first
 * The driver will run the XDP program on the Rx packet buffer (a UMEM buffer)
 * If the program results in XDP_DROP, the driver will "free" the Umem
buffer by putting it on the Completion Ring
 * If the program results in XDP_PASS, the driver will allocate a
standard Linux kernel SKB, copy the packet buffer contents into the
SKB and queue it up for standard netdev processing; It will then
"free" the RX Umem buffer by putting it on the Completion Ring (since
we have already copied packet into the skb)

Is this understanding correct or am I mistaken anywhere?

Thanks in advance,
Srivats

Re: AF_XDP RX processing with NO rx-ring

[Magnus Karlsson]

On Thu, 5 Oct 2023 at 16:31, Srivats P <pstavirs@gmail.com> wrote:
>
> Hi,
>
> I want to clarify my AF_XDP understanding for a particular scenario.
>
> Consider a single UMEM with 1 Fill Ring and 1 Completion Ring and
> single XDP socket  bound to queue 0 with 1 Tx Ring but NO Rx Ring.
> Assume the NIC has only a single queue (to keep things simple for
> explaining this scenario).
>
> There is a XDP program attached to Queue 0 which does either a
> XDP_DROP or XDP_PASS for all Rx packets.
>
> We are running in Driver mode.
>
> What happens to RX packets received on Queue 0?
>
> Here's my understanding -
>
>  * Even though there is no AF_XDP Rx Ring, there will be a NIC Rx Ring
> for queue 0
>  * The NIC Rx Ring for queue 0 is populated by the driver with UMEM
> buffers taken from the Fill ring

In driver mode (i.e. not zero-copy mode) this will not happen. The
ring will be populated by kernel buffers.

>  * A Rx packet will be received in the NIC Rx Ring for Queue 0 first
>  * The driver will run the XDP program on the Rx packet buffer (a UMEM buffer)
>  * If the program results in XDP_DROP, the driver will "free" the Umem
> buffer by putting it on the Completion Ring
>  * If the program results in XDP_PASS, the driver will allocate a
> standard Linux kernel SKB, copy the packet buffer contents into the
> SKB and queue it up for standard netdev processing; It will then
> "free" the RX Umem buffer by putting it on the Completion Ring (since
> we have already copied packet into the skb)

As the buffers are kernel buffers, user-space will not be notified.
The completion ring is solely for the Tx path, saying that user space
can have the buffer back.

The rest is correct.

/Magnus

> Is this understanding correct or am I mistaken anywhere?
>
> Thanks in advance,
> Srivats

Re: AF_XDP RX processing with NO rx-ring

[Srivats P]

On Thu, Oct 5, 2023 at 8:40 PM Magnus Karlsson
<magnus.karlsson@gmail.com> wrote:
>
> On Thu, 5 Oct 2023 at 16:31, Srivats P <pstavirs@gmail.com> wrote:
> >
> > Hi,
> >
> > I want to clarify my AF_XDP understanding for a particular scenario.
> >
> > Consider a single UMEM with 1 Fill Ring and 1 Completion Ring and
> > single XDP socket  bound to queue 0 with 1 Tx Ring but NO Rx Ring.
> > Assume the NIC has only a single queue (to keep things simple for
> > explaining this scenario).
> >
> > There is a XDP program attached to Queue 0 which does either a
> > XDP_DROP or XDP_PASS for all Rx packets.
> >
> > We are running in Driver mode.
> >
> > What happens to RX packets received on Queue 0?
> >
> > Here's my understanding -
> >
> >  * Even though there is no AF_XDP Rx Ring, there will be a NIC Rx Ring
> > for queue 0
> >  * The NIC Rx Ring for queue 0 is populated by the driver with UMEM
> > buffers taken from the Fill ring
>
> In driver mode (i.e. not zero-copy mode) this will not happen. The
> ring will be populated by kernel buffers.

What is the behaviour in zero-copy mode?

>
> >  * A Rx packet will be received in the NIC Rx Ring for Queue 0 first
> >  * The driver will run the XDP program on the Rx packet buffer (a UMEM buffer)
> >  * If the program results in XDP_DROP, the driver will "free" the Umem
> > buffer by putting it on the Completion Ring
> >  * If the program results in XDP_PASS, the driver will allocate a
> > standard Linux kernel SKB, copy the packet buffer contents into the
> > SKB and queue it up for standard netdev processing; It will then
> > "free" the RX Umem buffer by putting it on the Completion Ring (since
> > we have already copied packet into the skb)
>
> As the buffers are kernel buffers, user-space will not be notified.
> The completion ring is solely for the Tx path, saying that user space
> can have the buffer back.
>
> The rest is correct.
>
> /Magnus
>
> > Is this understanding correct or am I mistaken anywhere?
> >
> > Thanks in advance,
> > Srivats

Re: AF_XDP RX processing with NO rx-ring

[Magnus Karlsson]

On Thu, 5 Oct 2023 at 17:18, Srivats P <pstavirs@gmail.com> wrote:
>
> On Thu, Oct 5, 2023 at 8:40 PM Magnus Karlsson
> <magnus.karlsson@gmail.com> wrote:
> >
> > On Thu, 5 Oct 2023 at 16:31, Srivats P <pstavirs@gmail.com> wrote:
> > >
> > > Hi,
> > >
> > > I want to clarify my AF_XDP understanding for a particular scenario.
> > >
> > > Consider a single UMEM with 1 Fill Ring and 1 Completion Ring and
> > > single XDP socket  bound to queue 0 with 1 Tx Ring but NO Rx Ring.
> > > Assume the NIC has only a single queue (to keep things simple for
> > > explaining this scenario).
> > >
> > > There is a XDP program attached to Queue 0 which does either a
> > > XDP_DROP or XDP_PASS for all Rx packets.
> > >
> > > We are running in Driver mode.
> > >
> > > What happens to RX packets received on Queue 0?
> > >
> > > Here's my understanding -
> > >
> > >  * Even though there is no AF_XDP Rx Ring, there will be a NIC Rx Ring
> > > for queue 0
> > >  * The NIC Rx Ring for queue 0 is populated by the driver with UMEM
> > > buffers taken from the Fill ring
> >
> > In driver mode (i.e. not zero-copy mode) this will not happen. The
> > ring will be populated by kernel buffers.
>
> What is the behaviour in zero-copy mode?

Then the behaviour is according to what you wrote above. Just note
that nothing will be returned in the fill ring as you are not sending
anything to the AF_XDP socket.

> >
> > >  * A Rx packet will be received in the NIC Rx Ring for Queue 0 first
> > >  * The driver will run the XDP program on the Rx packet buffer (a UMEM buffer)
> > >  * If the program results in XDP_DROP, the driver will "free" the Umem
> > > buffer by putting it on the Completion Ring
> > >  * If the program results in XDP_PASS, the driver will allocate a
> > > standard Linux kernel SKB, copy the packet buffer contents into the
> > > SKB and queue it up for standard netdev processing; It will then
> > > "free" the RX Umem buffer by putting it on the Completion Ring (since
> > > we have already copied packet into the skb)
> >
> > As the buffers are kernel buffers, user-space will not be notified.
> > The completion ring is solely for the Tx path, saying that user space
> > can have the buffer back.
> >
> > The rest is correct.
> >
> > /Magnus
> >
> > > Is this understanding correct or am I mistaken anywhere?
> > >
> > > Thanks in advance,
> > > Srivats

Re: AF_XDP RX processing with NO rx-ring

[Srivats P]

On Thu, Oct 5, 2023 at 9:03 PM Magnus Karlsson
<magnus.karlsson@gmail.com> wrote:
>
> On Thu, 5 Oct 2023 at 17:18, Srivats P <pstavirs@gmail.com> wrote:
> >
> > On Thu, Oct 5, 2023 at 8:40 PM Magnus Karlsson
> > <magnus.karlsson@gmail.com> wrote:
> > >
> > > On Thu, 5 Oct 2023 at 16:31, Srivats P <pstavirs@gmail.com> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I want to clarify my AF_XDP understanding for a particular scenario.
> > > >
> > > > Consider a single UMEM with 1 Fill Ring and 1 Completion Ring and
> > > > single XDP socket  bound to queue 0 with 1 Tx Ring but NO Rx Ring.
> > > > Assume the NIC has only a single queue (to keep things simple for
> > > > explaining this scenario).
> > > >
> > > > There is a XDP program attached to Queue 0 which does either a
> > > > XDP_DROP or XDP_PASS for all Rx packets.
> > > >
> > > > We are running in Driver mode.
> > > >
> > > > What happens to RX packets received on Queue 0?
> > > >
> > > > Here's my understanding -
> > > >
> > > >  * Even though there is no AF_XDP Rx Ring, there will be a NIC Rx Ring
> > > > for queue 0
> > > >  * The NIC Rx Ring for queue 0 is populated by the driver with UMEM
> > > > buffers taken from the Fill ring
> > >
> > > In driver mode (i.e. not zero-copy mode) this will not happen. The
> > > ring will be populated by kernel buffers.
> >
> > What is the behaviour in zero-copy mode?
>
> Then the behaviour is according to what you wrote above. Just note
> that nothing will be returned in the fill ring as you are not sending
> anything to the AF_XDP socket.

I assume you meant "Rx Ring" and not "Fill ring" above?

>
> > >
> > > >  * A Rx packet will be received in the NIC Rx Ring for Queue 0 first
> > > >  * The driver will run the XDP program on the Rx packet buffer (a UMEM buffer)
> > > >  * If the program results in XDP_DROP, the driver will "free" the Umem
> > > > buffer by putting it on the Completion Ring
> > > >  * If the program results in XDP_PASS, the driver will allocate a
> > > > standard Linux kernel SKB, copy the packet buffer contents into the
> > > > SKB and queue it up for standard netdev processing; It will then
> > > > "free" the RX Umem buffer by putting it on the Completion Ring (since
> > > > we have already copied packet into the skb)
> > >
> > > As the buffers are kernel buffers, user-space will not be notified.
> > > The completion ring is solely for the Tx path, saying that user space
> > > can have the buffer back.
> > >
> > > The rest is correct.
> > >
> > > /Magnus
> > >
> > > > Is this understanding correct or am I mistaken anywhere?
> > > >
> > > > Thanks in advance,
> > > > Srivats

Re: AF_XDP RX processing with NO rx-ring

[Magnus Karlsson]

On Fri, 6 Oct 2023 at 13:41, Srivats P <pstavirs@gmail.com> wrote:
>
> On Thu, Oct 5, 2023 at 9:03 PM Magnus Karlsson
> <magnus.karlsson@gmail.com> wrote:
> >
> > On Thu, 5 Oct 2023 at 17:18, Srivats P <pstavirs@gmail.com> wrote:
> > >
> > > On Thu, Oct 5, 2023 at 8:40 PM Magnus Karlsson
> > > <magnus.karlsson@gmail.com> wrote:
> > > >
> > > > On Thu, 5 Oct 2023 at 16:31, Srivats P <pstavirs@gmail.com> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > I want to clarify my AF_XDP understanding for a particular scenario.
> > > > >
> > > > > Consider a single UMEM with 1 Fill Ring and 1 Completion Ring and
> > > > > single XDP socket  bound to queue 0 with 1 Tx Ring but NO Rx Ring.
> > > > > Assume the NIC has only a single queue (to keep things simple for
> > > > > explaining this scenario).
> > > > >
> > > > > There is a XDP program attached to Queue 0 which does either a
> > > > > XDP_DROP or XDP_PASS for all Rx packets.
> > > > >
> > > > > We are running in Driver mode.
> > > > >
> > > > > What happens to RX packets received on Queue 0?
> > > > >
> > > > > Here's my understanding -
> > > > >
> > > > >  * Even though there is no AF_XDP Rx Ring, there will be a NIC Rx Ring
> > > > > for queue 0
> > > > >  * The NIC Rx Ring for queue 0 is populated by the driver with UMEM
> > > > > buffers taken from the Fill ring
> > > >
> > > > In driver mode (i.e. not zero-copy mode) this will not happen. The
> > > > ring will be populated by kernel buffers.
> > >
> > > What is the behaviour in zero-copy mode?
> >
> > Then the behaviour is according to what you wrote above. Just note
> > that nothing will be returned in the fill ring as you are not sending
> > anything to the AF_XDP socket.
>
> I assume you meant "Rx Ring" and not "Fill ring" above?

Yes, sorry. Nothing will arrive in the Rx ring, and the Fill ring will
not be used apart from the initial burst of buffers that will be
grabbed by the system. Note though that you have to provide some
buffers in the fill ring as these are the ones that are used by the
NIC in zero-copy mode. They are just never returned to user-space
through the Rx ring. They are just recycled in the driver so the
system never needs to grab more in your scenario.

> >
> > > >
> > > > >  * A Rx packet will be received in the NIC Rx Ring for Queue 0 first
> > > > >  * The driver will run the XDP program on the Rx packet buffer (a UMEM buffer)
> > > > >  * If the program results in XDP_DROP, the driver will "free" the Umem
> > > > > buffer by putting it on the Completion Ring
> > > > >  * If the program results in XDP_PASS, the driver will allocate a
> > > > > standard Linux kernel SKB, copy the packet buffer contents into the
> > > > > SKB and queue it up for standard netdev processing; It will then
> > > > > "free" the RX Umem buffer by putting it on the Completion Ring (since
> > > > > we have already copied packet into the skb)
> > > >
> > > > As the buffers are kernel buffers, user-space will not be notified.
> > > > The completion ring is solely for the Tx path, saying that user space
> > > > can have the buffer back.
> > > >
> > > > The rest is correct.
> > > >
> > > > /Magnus
> > > >
> > > > > Is this understanding correct or am I mistaken anywhere?
> > > > >
> > > > > Thanks in advance,
> > > > > Srivats

Re: AF_XDP RX processing with NO rx-ring

[Srivats P]

On Fri, Oct 6, 2023 at 5:23 PM Magnus Karlsson
<magnus.karlsson@gmail.com> wrote:
>
> On Fri, 6 Oct 2023 at 13:41, Srivats P <pstavirs@gmail.com> wrote:
> >
> > On Thu, Oct 5, 2023 at 9:03 PM Magnus Karlsson
> > <magnus.karlsson@gmail.com> wrote:
> > >
> > > On Thu, 5 Oct 2023 at 17:18, Srivats P <pstavirs@gmail.com> wrote:
> > > >
> > > > On Thu, Oct 5, 2023 at 8:40 PM Magnus Karlsson
> > > > <magnus.karlsson@gmail.com> wrote:
> > > > >
> > > > > On Thu, 5 Oct 2023 at 16:31, Srivats P <pstavirs@gmail.com> wrote:
> > > > > >
> > > > > > Hi,
> > > > > >
> > > > > > I want to clarify my AF_XDP understanding for a particular scenario.
> > > > > >
> > > > > > Consider a single UMEM with 1 Fill Ring and 1 Completion Ring and
> > > > > > single XDP socket  bound to queue 0 with 1 Tx Ring but NO Rx Ring.
> > > > > > Assume the NIC has only a single queue (to keep things simple for
> > > > > > explaining this scenario).
> > > > > >
> > > > > > There is a XDP program attached to Queue 0 which does either a
> > > > > > XDP_DROP or XDP_PASS for all Rx packets.
> > > > > >
> > > > > > We are running in Driver mode.
> > > > > >
> > > > > > What happens to RX packets received on Queue 0?
> > > > > >
> > > > > > Here's my understanding -
> > > > > >
> > > > > >  * Even though there is no AF_XDP Rx Ring, there will be a NIC Rx Ring
> > > > > > for queue 0
> > > > > >  * The NIC Rx Ring for queue 0 is populated by the driver with UMEM
> > > > > > buffers taken from the Fill ring
> > > > >
> > > > > In driver mode (i.e. not zero-copy mode) this will not happen. The
> > > > > ring will be populated by kernel buffers.
> > > >
> > > > What is the behaviour in zero-copy mode?
> > >
> > > Then the behaviour is according to what you wrote above. Just note
> > > that nothing will be returned in the fill ring as you are not sending
> > > anything to the AF_XDP socket.
> >
> > I assume you meant "Rx Ring" and not "Fill ring" above?
>
> Yes, sorry. Nothing will arrive in the Rx ring, and the Fill ring will
> not be used apart from the initial burst of buffers that will be
> grabbed by the system. Note though that you have to provide some
> buffers in the fill ring as these are the ones that are used by the
> NIC in zero-copy mode. They are just never returned to user-space
> through the Rx ring. They are just recycled in the driver so the
> system never needs to grab more in your scenario.
>

After a XDP_DROP or XDP_PASS at RX, will the driver recycle the umem
buffer back into the NIC's RX-ring or will it put it on the CQ? In
case of the latter, the FQ will eventually run out of buffers.

Is this behaviour driver specific (recycle to NIC RX-Ring or CQ) or
same for all drivers in ZC mode?


> > >
> > > > >
> > > > > >  * A Rx packet will be received in the NIC Rx Ring for Queue 0 first
> > > > > >  * The driver will run the XDP program on the Rx packet buffer (a UMEM buffer)
> > > > > >  * If the program results in XDP_DROP, the driver will "free" the Umem
> > > > > > buffer by putting it on the Completion Ring
> > > > > >  * If the program results in XDP_PASS, the driver will allocate a
> > > > > > standard Linux kernel SKB, copy the packet buffer contents into the
> > > > > > SKB and queue it up for standard netdev processing; It will then
> > > > > > "free" the RX Umem buffer by putting it on the Completion Ring (since
> > > > > > we have already copied packet into the skb)
> > > > >
> > > > > As the buffers are kernel buffers, user-space will not be notified.
> > > > > The completion ring is solely for the Tx path, saying that user space
> > > > > can have the buffer back.
> > > > >
> > > > > The rest is correct.
> > > > >
> > > > > /Magnus
> > > > >
> > > > > > Is this understanding correct or am I mistaken anywhere?
> > > > > >
> > > > > > Thanks in advance,
> > > > > > Srivats

Re: AF_XDP RX processing with NO rx-ring

[Magnus Karlsson]

On Fri, 6 Oct 2023 at 14:36, Srivats P <pstavirs@gmail.com> wrote:
>
> On Fri, Oct 6, 2023 at 5:23 PM Magnus Karlsson
> <magnus.karlsson@gmail.com> wrote:
> >
> > On Fri, 6 Oct 2023 at 13:41, Srivats P <pstavirs@gmail.com> wrote:
> > >
> > > On Thu, Oct 5, 2023 at 9:03 PM Magnus Karlsson
> > > <magnus.karlsson@gmail.com> wrote:
> > > >
> > > > On Thu, 5 Oct 2023 at 17:18, Srivats P <pstavirs@gmail.com> wrote:
> > > > >
> > > > > On Thu, Oct 5, 2023 at 8:40 PM Magnus Karlsson
> > > > > <magnus.karlsson@gmail.com> wrote:
> > > > > >
> > > > > > On Thu, 5 Oct 2023 at 16:31, Srivats P <pstavirs@gmail.com> wrote:
> > > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > I want to clarify my AF_XDP understanding for a particular scenario.
> > > > > > >
> > > > > > > Consider a single UMEM with 1 Fill Ring and 1 Completion Ring and
> > > > > > > single XDP socket  bound to queue 0 with 1 Tx Ring but NO Rx Ring.
> > > > > > > Assume the NIC has only a single queue (to keep things simple for
> > > > > > > explaining this scenario).
> > > > > > >
> > > > > > > There is a XDP program attached to Queue 0 which does either a
> > > > > > > XDP_DROP or XDP_PASS for all Rx packets.
> > > > > > >
> > > > > > > We are running in Driver mode.
> > > > > > >
> > > > > > > What happens to RX packets received on Queue 0?
> > > > > > >
> > > > > > > Here's my understanding -
> > > > > > >
> > > > > > >  * Even though there is no AF_XDP Rx Ring, there will be a NIC Rx Ring
> > > > > > > for queue 0
> > > > > > >  * The NIC Rx Ring for queue 0 is populated by the driver with UMEM
> > > > > > > buffers taken from the Fill ring
> > > > > >
> > > > > > In driver mode (i.e. not zero-copy mode) this will not happen. The
> > > > > > ring will be populated by kernel buffers.
> > > > >
> > > > > What is the behaviour in zero-copy mode?
> > > >
> > > > Then the behaviour is according to what you wrote above. Just note
> > > > that nothing will be returned in the fill ring as you are not sending
> > > > anything to the AF_XDP socket.
> > >
> > > I assume you meant "Rx Ring" and not "Fill ring" above?
> >
> > Yes, sorry. Nothing will arrive in the Rx ring, and the Fill ring will
> > not be used apart from the initial burst of buffers that will be
> > grabbed by the system. Note though that you have to provide some
> > buffers in the fill ring as these are the ones that are used by the
> > NIC in zero-copy mode. They are just never returned to user-space
> > through the Rx ring. They are just recycled in the driver so the
> > system never needs to grab more in your scenario.
> >
>
> After a XDP_DROP or XDP_PASS at RX, will the driver recycle the umem
> buffer back into the NIC's RX-ring or will it put it on the CQ? In
> case of the latter, the FQ will eventually run out of buffers.

It will recycle it back to the Rx ring of the NIC. The kernel will
never take a buffer used for Rx and reuse it for Tx, or the opposite.
A buffer put on the fill ring can only return on the Rx ring and a
buffer put on the Tx ring can only return on the Completion ring.

> Is this behaviour driver specific (recycle to NIC RX-Ring or CQ) or
> same for all drivers in ZC mode?

This is core AF_XDP code, so all drivers will do the same thing.

>
> > > >
> > > > > >
> > > > > > >  * A Rx packet will be received in the NIC Rx Ring for Queue 0 first
> > > > > > >  * The driver will run the XDP program on the Rx packet buffer (a UMEM buffer)
> > > > > > >  * If the program results in XDP_DROP, the driver will "free" the Umem
> > > > > > > buffer by putting it on the Completion Ring
> > > > > > >  * If the program results in XDP_PASS, the driver will allocate a
> > > > > > > standard Linux kernel SKB, copy the packet buffer contents into the
> > > > > > > SKB and queue it up for standard netdev processing; It will then
> > > > > > > "free" the RX Umem buffer by putting it on the Completion Ring (since
> > > > > > > we have already copied packet into the skb)
> > > > > >
> > > > > > As the buffers are kernel buffers, user-space will not be notified.
> > > > > > The completion ring is solely for the Tx path, saying that user space
> > > > > > can have the buffer back.
> > > > > >
> > > > > > The rest is correct.
> > > > > >
> > > > > > /Magnus
> > > > > >
> > > > > > > Is this understanding correct or am I mistaken anywhere?
> > > > > > >
> > > > > > > Thanks in advance,
> > > > > > > Srivats

Re: AF_XDP RX processing with NO rx-ring

[Srivats P]

On Fri, Oct 6, 2023 at 6:34 PM Magnus Karlsson
<magnus.karlsson@gmail.com> wrote:
>
> On Fri, 6 Oct 2023 at 14:36, Srivats P <pstavirs@gmail.com> wrote:
> >
> > On Fri, Oct 6, 2023 at 5:23 PM Magnus Karlsson
> > <magnus.karlsson@gmail.com> wrote:
> > >
> > > On Fri, 6 Oct 2023 at 13:41, Srivats P <pstavirs@gmail.com> wrote:
> > > >
> > > > On Thu, Oct 5, 2023 at 9:03 PM Magnus Karlsson
> > > > <magnus.karlsson@gmail.com> wrote:
> > > > >
> > > > > On Thu, 5 Oct 2023 at 17:18, Srivats P <pstavirs@gmail.com> wrote:
> > > > > >
> > > > > > On Thu, Oct 5, 2023 at 8:40 PM Magnus Karlsson
> > > > > > <magnus.karlsson@gmail.com> wrote:
> > > > > > >
> > > > > > > On Thu, 5 Oct 2023 at 16:31, Srivats P <pstavirs@gmail.com> wrote:
> > > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > I want to clarify my AF_XDP understanding for a particular scenario.
> > > > > > > >
> > > > > > > > Consider a single UMEM with 1 Fill Ring and 1 Completion Ring and
> > > > > > > > single XDP socket  bound to queue 0 with 1 Tx Ring but NO Rx Ring.
> > > > > > > > Assume the NIC has only a single queue (to keep things simple for
> > > > > > > > explaining this scenario).
> > > > > > > >
> > > > > > > > There is a XDP program attached to Queue 0 which does either a
> > > > > > > > XDP_DROP or XDP_PASS for all Rx packets.
> > > > > > > >
> > > > > > > > We are running in Driver mode.
> > > > > > > >
> > > > > > > > What happens to RX packets received on Queue 0?
> > > > > > > >
> > > > > > > > Here's my understanding -
> > > > > > > >
> > > > > > > >  * Even though there is no AF_XDP Rx Ring, there will be a NIC Rx Ring
> > > > > > > > for queue 0
> > > > > > > >  * The NIC Rx Ring for queue 0 is populated by the driver with UMEM
> > > > > > > > buffers taken from the Fill ring
> > > > > > >
> > > > > > > In driver mode (i.e. not zero-copy mode) this will not happen. The
> > > > > > > ring will be populated by kernel buffers.
> > > > > >
> > > > > > What is the behaviour in zero-copy mode?
> > > > >
> > > > > Then the behaviour is according to what you wrote above. Just note
> > > > > that nothing will be returned in the fill ring as you are not sending
> > > > > anything to the AF_XDP socket.
> > > >
> > > > I assume you meant "Rx Ring" and not "Fill ring" above?
> > >
> > > Yes, sorry. Nothing will arrive in the Rx ring, and the Fill ring will
> > > not be used apart from the initial burst of buffers that will be
> > > grabbed by the system. Note though that you have to provide some
> > > buffers in the fill ring as these are the ones that are used by the
> > > NIC in zero-copy mode. They are just never returned to user-space
> > > through the Rx ring. They are just recycled in the driver so the
> > > system never needs to grab more in your scenario.
> > >
> >
> > After a XDP_DROP or XDP_PASS at RX, will the driver recycle the umem
> > buffer back into the NIC's RX-ring or will it put it on the CQ? In
> > case of the latter, the FQ will eventually run out of buffers.
>
> It will recycle it back to the Rx ring of the NIC. The kernel will
> never take a buffer used for Rx and reuse it for Tx, or the opposite.
> A buffer put on the fill ring can only return on the Rx ring and a
> buffer put on the Tx ring can only return on the Completion ring.

Ah! So what I wrote in my original email was incorrect - a Rx buffer
will never be put on the Completion Ring.

I'm debugging a problem where after sending 'n' packets on the TX
ring, the app expects 'n' packets to be freed from the completion ring
- but sometimes it gets less than n (in a busy loop) or worse even
more than n packets from the completion ring - how is this possible?
Any pointers or leads are appreciated.

This is ixgbe. Not able to reproduce on an i40e.

Please note that the app may put the same umem buffer into multiple Tx
descriptors, so the same umem buffer could be in multiple descriptors
of Tx ring and completion ring. The packet buffers are prebuilt before
transmit begins and read only during transmit, so I presume this
should not cause any problem, but would like to confirm anyway.

> > Is this behaviour driver specific (recycle to NIC RX-Ring or CQ) or
> > same for all drivers in ZC mode?
>
> This is core AF_XDP code, so all drivers will do the same thing.
>
> >
> > > > >
> > > > > > >
> > > > > > > >  * A Rx packet will be received in the NIC Rx Ring for Queue 0 first
> > > > > > > >  * The driver will run the XDP program on the Rx packet buffer (a UMEM buffer)
> > > > > > > >  * If the program results in XDP_DROP, the driver will "free" the Umem
> > > > > > > > buffer by putting it on the Completion Ring
> > > > > > > >  * If the program results in XDP_PASS, the driver will allocate a
> > > > > > > > standard Linux kernel SKB, copy the packet buffer contents into the
> > > > > > > > SKB and queue it up for standard netdev processing; It will then
> > > > > > > > "free" the RX Umem buffer by putting it on the Completion Ring (since
> > > > > > > > we have already copied packet into the skb)
> > > > > > >
> > > > > > > As the buffers are kernel buffers, user-space will not be notified.
> > > > > > > The completion ring is solely for the Tx path, saying that user space
> > > > > > > can have the buffer back.
> > > > > > >
> > > > > > > The rest is correct.
> > > > > > >
> > > > > > > /Magnus
> > > > > > >
> > > > > > > > Is this understanding correct or am I mistaken anywhere?
> > > > > > > >
> > > > > > > > Thanks in advance,
> > > > > > > > Srivats

Re: AF_XDP RX processing with NO rx-ring

[Magnus Karlsson]

On Sat, 7 Oct 2023 at 01:07, Srivats P <pstavirs@gmail.com> wrote:
>
> On Fri, Oct 6, 2023 at 6:34 PM Magnus Karlsson
> <magnus.karlsson@gmail.com> wrote:
> >
> > On Fri, 6 Oct 2023 at 14:36, Srivats P <pstavirs@gmail.com> wrote:
> > >
> > > On Fri, Oct 6, 2023 at 5:23 PM Magnus Karlsson
> > > <magnus.karlsson@gmail.com> wrote:
> > > >
> > > > On Fri, 6 Oct 2023 at 13:41, Srivats P <pstavirs@gmail.com> wrote:
> > > > >
> > > > > On Thu, Oct 5, 2023 at 9:03 PM Magnus Karlsson
> > > > > <magnus.karlsson@gmail.com> wrote:
> > > > > >
> > > > > > On Thu, 5 Oct 2023 at 17:18, Srivats P <pstavirs@gmail.com> wrote:
> > > > > > >
> > > > > > > On Thu, Oct 5, 2023 at 8:40 PM Magnus Karlsson
> > > > > > > <magnus.karlsson@gmail.com> wrote:
> > > > > > > >
> > > > > > > > On Thu, 5 Oct 2023 at 16:31, Srivats P <pstavirs@gmail.com> wrote:
> > > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > I want to clarify my AF_XDP understanding for a particular scenario.
> > > > > > > > >
> > > > > > > > > Consider a single UMEM with 1 Fill Ring and 1 Completion Ring and
> > > > > > > > > single XDP socket  bound to queue 0 with 1 Tx Ring but NO Rx Ring.
> > > > > > > > > Assume the NIC has only a single queue (to keep things simple for
> > > > > > > > > explaining this scenario).
> > > > > > > > >
> > > > > > > > > There is a XDP program attached to Queue 0 which does either a
> > > > > > > > > XDP_DROP or XDP_PASS for all Rx packets.
> > > > > > > > >
> > > > > > > > > We are running in Driver mode.
> > > > > > > > >
> > > > > > > > > What happens to RX packets received on Queue 0?
> > > > > > > > >
> > > > > > > > > Here's my understanding -
> > > > > > > > >
> > > > > > > > >  * Even though there is no AF_XDP Rx Ring, there will be a NIC Rx Ring
> > > > > > > > > for queue 0
> > > > > > > > >  * The NIC Rx Ring for queue 0 is populated by the driver with UMEM
> > > > > > > > > buffers taken from the Fill ring
> > > > > > > >
> > > > > > > > In driver mode (i.e. not zero-copy mode) this will not happen. The
> > > > > > > > ring will be populated by kernel buffers.
> > > > > > >
> > > > > > > What is the behaviour in zero-copy mode?
> > > > > >
> > > > > > Then the behaviour is according to what you wrote above. Just note
> > > > > > that nothing will be returned in the fill ring as you are not sending
> > > > > > anything to the AF_XDP socket.
> > > > >
> > > > > I assume you meant "Rx Ring" and not "Fill ring" above?
> > > >
> > > > Yes, sorry. Nothing will arrive in the Rx ring, and the Fill ring will
> > > > not be used apart from the initial burst of buffers that will be
> > > > grabbed by the system. Note though that you have to provide some
> > > > buffers in the fill ring as these are the ones that are used by the
> > > > NIC in zero-copy mode. They are just never returned to user-space
> > > > through the Rx ring. They are just recycled in the driver so the
> > > > system never needs to grab more in your scenario.
> > > >
> > >
> > > After a XDP_DROP or XDP_PASS at RX, will the driver recycle the umem
> > > buffer back into the NIC's RX-ring or will it put it on the CQ? In
> > > case of the latter, the FQ will eventually run out of buffers.
> >
> > It will recycle it back to the Rx ring of the NIC. The kernel will
> > never take a buffer used for Rx and reuse it for Tx, or the opposite.
> > A buffer put on the fill ring can only return on the Rx ring and a
> > buffer put on the Tx ring can only return on the Completion ring.
>
> Ah! So what I wrote in my original email was incorrect - a Rx buffer
> will never be put on the Completion Ring.
>
> I'm debugging a problem where after sending 'n' packets on the TX
> ring, the app expects 'n' packets to be freed from the completion ring
> - but sometimes it gets less than n (in a busy loop) or worse even
> more than n packets from the completion ring - how is this possible?
> Any pointers or leads are appreciated.
>
> This is ixgbe. Not able to reproduce on an i40e.

Sounds like a bug. Maybe in the driver as you cannot reproduce it on i40e.

> Please note that the app may put the same umem buffer into multiple Tx
> descriptors, so the same umem buffer could be in multiple descriptors
> of Tx ring and completion ring. The packet buffers are prebuilt before
> transmit begins and read only during transmit, so I presume this
> should not cause any problem, but would like to confirm anyway.

That should be fine.

> > > Is this behaviour driver specific (recycle to NIC RX-Ring or CQ) or
> > > same for all drivers in ZC mode?
> >
> > This is core AF_XDP code, so all drivers will do the same thing.
> >
> > >
> > > > > >
> > > > > > > >
> > > > > > > > >  * A Rx packet will be received in the NIC Rx Ring for Queue 0 first
> > > > > > > > >  * The driver will run the XDP program on the Rx packet buffer (a UMEM buffer)
> > > > > > > > >  * If the program results in XDP_DROP, the driver will "free" the Umem
> > > > > > > > > buffer by putting it on the Completion Ring
> > > > > > > > >  * If the program results in XDP_PASS, the driver will allocate a
> > > > > > > > > standard Linux kernel SKB, copy the packet buffer contents into the
> > > > > > > > > SKB and queue it up for standard netdev processing; It will then
> > > > > > > > > "free" the RX Umem buffer by putting it on the Completion Ring (since
> > > > > > > > > we have already copied packet into the skb)
> > > > > > > >
> > > > > > > > As the buffers are kernel buffers, user-space will not be notified.
> > > > > > > > The completion ring is solely for the Tx path, saying that user space
> > > > > > > > can have the buffer back.
> > > > > > > >
> > > > > > > > The rest is correct.
> > > > > > > >
> > > > > > > > /Magnus
> > > > > > > >
> > > > > > > > > Is this understanding correct or am I mistaken anywhere?
> > > > > > > > >
> > > > > > > > > Thanks in advance,
> > > > > > > > > Srivats