AF_XDP socket TX path interact with TC?

AF_XDP socket TX path interact with TC?

[Jun Hu]

Hi,
I want to put a timestamp in the egress packets, and I want to use
AF_XDP socket for both sending and receiving packets; since a XDP
kernel program only work in ingress direction, I wonder if I could use
a tc kernel program to change a packet sent out via an AF_XDP socket?
I searched around, but couldn't find any direct answer...

HJ

Re: AF_XDP socket TX path interact with TC?

[Toke Høiland-Jørgensen]

Jun Hu <hujun.work@gmail.com> writes:

> Hi,
> I want to put a timestamp in the egress packets, and I want to use
> AF_XDP socket for both sending and receiving packets; since a XDP
> kernel program only work in ingress direction, I wonder if I could use
> a tc kernel program to change a packet sent out via an AF_XDP socket?
> I searched around, but couldn't find any direct answer...

Nope, AF_XDP bypasses the stack completely, so TC won't see the packet
on egress...

-Toke

Re: AF_XDP socket TX path interact with TC?

[Jun Hu]

On Mon, Oct 25, 2021 at 3:48 AM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>
> Jun Hu <hujun.work@gmail.com> writes:
>
> > Hi,
> > I want to put a timestamp in the egress packets, and I want to use
> > AF_XDP socket for both sending and receiving packets; since a XDP
> > kernel program only work in ingress direction, I wonder if I could use
> > a tc kernel program to change a packet sent out via an AF_XDP socket?
> > I searched around, but couldn't find any direct answer...
>
> Nope, AF_XDP bypasses the stack completely, so TC won't see the packet
> on egress...
>
> -Toke
Ok, thanks for the clarification, I guess I have to wait for the XDP
egress support ...

Re: AF_XDP socket TX path interact with TC?

[Konrad Zemek]

On Monday, October 25th, 2021 at 8:26 PM, Jun Hu <hujun.work@gmail.com> wrote:

> On Mon, Oct 25, 2021 at 3:48 AM Toke Høiland-Jørgensen toke@redhat.com wrote:
>
> > Jun Hu hujun.work@gmail.com writes:
> >
> > > Hi,
> > >
> > > I want to put a timestamp in the egress packets, and I want to use
> > >
> > > AF_XDP socket for both sending and receiving packets; since a XDP
> > >
> > > kernel program only work in ingress direction, I wonder if I could use
> > >
> > > a tc kernel program to change a packet sent out via an AF_XDP socket?
> > >
> > > I searched around, but couldn't find any direct answer...
> >
> > Nope, AF_XDP bypasses the stack completely, so TC won't see the packet
> >
> > on egress...
> >
> > -Toke
>
> Ok, thanks for the clarification, I guess I have to wait for the XDP
>
> egress support ...

You can also directly modify the packet that you send out through an AF_XDP socket, before putting it on the outgoing queue. Is there anything specific that you would want to do in an egress-attached BPF program that you would not be able to do in the userspace program that manages your AF_XDP socket?

Konrad

Re: AF_XDP socket TX path interact with TC?

[Jun Hu]

On Tue, Oct 26, 2021 at 4:08 AM Konrad Zemek <konrad@zemek.io> wrote:
>
> On Monday, October 25th, 2021 at 8:26 PM, Jun Hu <hujun.work@gmail.com> wrote:
>
> > On Mon, Oct 25, 2021 at 3:48 AM Toke Høiland-Jørgensen toke@redhat.com wrote:
> >
> > > Jun Hu hujun.work@gmail.com writes:
> > >
> > > > Hi,
> > > >
> > > > I want to put a timestamp in the egress packets, and I want to use
> > > >
> > > > AF_XDP socket for both sending and receiving packets; since a XDP
> > > >
> > > > kernel program only work in ingress direction, I wonder if I could use
> > > >
> > > > a tc kernel program to change a packet sent out via an AF_XDP socket?
> > > >
> > > > I searched around, but couldn't find any direct answer...
> > >
> > > Nope, AF_XDP bypasses the stack completely, so TC won't see the packet
> > >
> > > on egress...
> > >
> > > -Toke
> >
> > Ok, thanks for the clarification, I guess I have to wait for the XDP
> >
> > egress support ...
>
> You can also directly modify the packet that you send out through an AF_XDP socket, before putting it on the outgoing queue. Is there anything specific that you would want to do in an egress-attached BPF program that you would not be able to do in the userspace program that manages your AF_XDP socket?
>
> Konrad

Yes, that's what I am doing right now, however ideally I want to add a
timestamp to the packet as close as it egress the NIC; since AF_XDP
doesn't have access to hardware timestamp, that's why I was searching
for an alternative solution.

Re: AF_XDP socket TX path interact with TC?

[Toke Høiland-Jørgensen]

Jun Hu <hujun.work@gmail.com> writes:

> On Tue, Oct 26, 2021 at 4:08 AM Konrad Zemek <konrad@zemek.io> wrote:
>>
>> On Monday, October 25th, 2021 at 8:26 PM, Jun Hu <hujun.work@gmail.com> wrote:
>>
>> > On Mon, Oct 25, 2021 at 3:48 AM Toke Høiland-Jørgensen toke@redhat.com wrote:
>> >
>> > > Jun Hu hujun.work@gmail.com writes:
>> > >
>> > > > Hi,
>> > > >
>> > > > I want to put a timestamp in the egress packets, and I want to use
>> > > >
>> > > > AF_XDP socket for both sending and receiving packets; since a XDP
>> > > >
>> > > > kernel program only work in ingress direction, I wonder if I could use
>> > > >
>> > > > a tc kernel program to change a packet sent out via an AF_XDP socket?
>> > > >
>> > > > I searched around, but couldn't find any direct answer...
>> > >
>> > > Nope, AF_XDP bypasses the stack completely, so TC won't see the packet
>> > >
>> > > on egress...
>> > >
>> > > -Toke
>> >
>> > Ok, thanks for the clarification, I guess I have to wait for the XDP
>> >
>> > egress support ...
>>
>> You can also directly modify the packet that you send out through an
>> AF_XDP socket, before putting it on the outgoing queue. Is there
>> anything specific that you would want to do in an egress-attached BPF
>> program that you would not be able to do in the userspace program
>> that manages your AF_XDP socket?
>>
>> Konrad
>
> Yes, that's what I am doing right now, however ideally I want to add a
> timestamp to the packet as close as it egress the NIC; since AF_XDP
> doesn't have access to hardware timestamp, that's why I was searching
> for an alternative solution.

Cc. Jesper, as I believe timestamps is one of his initial use cases for
xdp-hints (hardware metadata).

-Toke

Re: AF_XDP socket TX path interact with TC?

[Jesper Dangaard Brouer]

On 27/10/2021 13.35, Toke Høiland-Jørgensen wrote:
> Jun Hu <hujun.work@gmail.com> writes:
> 
>> On Tue, Oct 26, 2021 at 4:08 AM Konrad Zemek <konrad@zemek.io> 
>> wrote:
>>> 
>>> On Monday, October 25th, 2021 at 8:26 PM, Jun Hu 
>>> <hujun.work@gmail.com> wrote:
>>> 
>>>> On Mon, Oct 25, 2021 at 3:48 AM Toke Høiland-Jørgensen 
>>>> toke@redhat.com wrote:
>>>> 
>>>>> Jun Hu hujun.work@gmail.com writes:
>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> I want to put a timestamp in the egress packets, and I
>>>>>> want to use
>>>>>> 
>>>>>> AF_XDP socket for both sending and receiving packets;
>>>>>> since a XDP
>>>>>> 
>>>>>> kernel program only work in ingress direction, I wonder if 
>>>>>> I could use
>>>>>> 
>>>>>> a tc kernel program to change a packet sent out via an 
>>>>>> AF_XDP socket?
>>>>>> 
>>>>>> I searched around, but couldn't find any direct answer...
>>>>> 
>>>>> Nope, AF_XDP bypasses the stack completely, so TC won't see 
>>>>> the packet
>>>>> 
>>>>> on egress...
>>>>> 
>>>>> -Toke
>>>> 
>>>> Ok, thanks for the clarification, I guess I have to wait for 
>>>> the XDP
>>>> 
>>>> egress support ...
>>> 
>>> You can also directly modify the packet that you send out
>>> through an AF_XDP socket, before putting it on the outgoing
>>> queue. Is there anything specific that you would want to do in
>>> an egress-attached BPF program that you would not be able to do
>>> in the userspace program that manages your AF_XDP socket?
>>> 
>>> Konrad
>> 
>> Yes, that's what I am doing right now, however ideally I want to 
>> add a timestamp to the packet as close as it egress the NIC; since 
>> AF_XDP doesn't have access to hardware timestamp, that's why I was 
>> searching for an alternative solution.
> 
> Cc. Jesper, as I believe timestamps is one of his initial use cases 
> for xdp-hints (hardware metadata).

Correct, Toke.  I have a use-case/customer that want timestamps working
with AF_XDP.
And yes the plan is to leverage XDP-hints (via XDP metadata area) to
expose/export these different packet timestamps.

I have two use-case for timestamps:
  (1) RX-timestamping.
  (2) TX-launch-time.

This week, I have promised to implement a software version of (1)
RX-timestamping, where XDP will add ktime into the metadata area and
AF_XDP userspace program will consume this.  It is rather trivial to do,
but I want extend it and use it to build a flexible BTF-info variant
on-top (which doesn't requires any kernel changes).
The goal is to get the hardware RX-timestamp, via changing the NIC
driver to populate the XDP-hints in metadata areas (from
HW-RX-descriptor).  Again using BTF to describe XDP-hints contents, then
AF_XDP userspace prog can hopefully use the same code to find and
extract members related to the HW
RX-timestamp.

For (2) TX-launch-time, as you mention XDP is missing an egress hook or
queueing layer.  Thus, there are some kernel work a-head.  Toke and a 
PhD stud is looking into this area.
As discussed/described before on xdp-hints mailing list[1] (cc'ed), in
this step it is the AF_XDP program that updates the XDP-hints area
before sending it, and the (non-existing XDP egress) driver hook moves 
requested LaunchTime into the HW-TX-descriptor.


  [1] 
https://lists.xdp-project.net/postorius/lists/xdp-hints.xdp-project.net/

What is your use-case for adding a TX-timestamp to the packet?

--Jesper

Re: AF_XDP socket TX path interact with TC?

[Jun Hu]

> What is your use-case for adding a TX-timestamp to the packet?
>
> --Jesper
my use case is rather simple, I want to use AF_XDP for traffic tester
type of application, essentially, this application generates testing
traffic, egress via NIC-1, goes into a DUT, and DUT forward the
traffic back into NIC-2, the application need to measure the DUT's
forwarding latency for each packet, so it needs to know the timestamp
of test traffic TX on NIC-1 and RX on NIC-2

                  +-----+
|---------------- | DUT |----------------|
|                 +-----+                |
|                                        |
|          +-------------------+         |
|  <-----  |                   |  <----  |
+----------|NIC-1        NIC-2 |---------+
           |                   |
           +-------------------+