Re: AF_XDP socket TX path interact with TC?

[Jesper Dangaard Brouer <jbrouer@redhat.com>]

On 27/10/2021 13.35, Toke Høiland-Jørgensen wrote:
> Jun Hu <hujun.work@gmail.com> writes:
> 
>> On Tue, Oct 26, 2021 at 4:08 AM Konrad Zemek <konrad@zemek.io> 
>> wrote:
>>> 
>>> On Monday, October 25th, 2021 at 8:26 PM, Jun Hu 
>>> <hujun.work@gmail.com> wrote:
>>> 
>>>> On Mon, Oct 25, 2021 at 3:48 AM Toke Høiland-Jørgensen 
>>>> toke@redhat.com wrote:
>>>> 
>>>>> Jun Hu hujun.work@gmail.com writes:
>>>>> 
>>>>>> Hi,
>>>>>> 
>>>>>> I want to put a timestamp in the egress packets, and I
>>>>>> want to use
>>>>>> 
>>>>>> AF_XDP socket for both sending and receiving packets;
>>>>>> since a XDP
>>>>>> 
>>>>>> kernel program only work in ingress direction, I wonder if 
>>>>>> I could use
>>>>>> 
>>>>>> a tc kernel program to change a packet sent out via an 
>>>>>> AF_XDP socket?
>>>>>> 
>>>>>> I searched around, but couldn't find any direct answer...
>>>>> 
>>>>> Nope, AF_XDP bypasses the stack completely, so TC won't see 
>>>>> the packet
>>>>> 
>>>>> on egress...
>>>>> 
>>>>> -Toke
>>>> 
>>>> Ok, thanks for the clarification, I guess I have to wait for 
>>>> the XDP
>>>> 
>>>> egress support ...
>>> 
>>> You can also directly modify the packet that you send out
>>> through an AF_XDP socket, before putting it on the outgoing
>>> queue. Is there anything specific that you would want to do in
>>> an egress-attached BPF program that you would not be able to do
>>> in the userspace program that manages your AF_XDP socket?
>>> 
>>> Konrad
>> 
>> Yes, that's what I am doing right now, however ideally I want to 
>> add a timestamp to the packet as close as it egress the NIC; since 
>> AF_XDP doesn't have access to hardware timestamp, that's why I was 
>> searching for an alternative solution.
> 
> Cc. Jesper, as I believe timestamps is one of his initial use cases 
> for xdp-hints (hardware metadata).

Correct, Toke.  I have a use-case/customer that want timestamps working
with AF_XDP.
And yes the plan is to leverage XDP-hints (via XDP metadata area) to
expose/export these different packet timestamps.

I have two use-case for timestamps:
  (1) RX-timestamping.
  (2) TX-launch-time.

This week, I have promised to implement a software version of (1)
RX-timestamping, where XDP will add ktime into the metadata area and
AF_XDP userspace program will consume this.  It is rather trivial to do,
but I want extend it and use it to build a flexible BTF-info variant
on-top (which doesn't requires any kernel changes).
The goal is to get the hardware RX-timestamp, via changing the NIC
driver to populate the XDP-hints in metadata areas (from
HW-RX-descriptor).  Again using BTF to describe XDP-hints contents, then
AF_XDP userspace prog can hopefully use the same code to find and
extract members related to the HW
RX-timestamp.

For (2) TX-launch-time, as you mention XDP is missing an egress hook or
queueing layer.  Thus, there are some kernel work a-head.  Toke and a 
PhD stud is looking into this area.
As discussed/described before on xdp-hints mailing list[1] (cc'ed), in
this step it is the AF_XDP program that updates the XDP-hints area
before sending it, and the (non-existing XDP egress) driver hook moves 
requested LaunchTime into the HW-TX-descriptor.


  [1] 
https://lists.xdp-project.net/postorius/lists/xdp-hints.xdp-project.net/

What is your use-case for adding a TX-timestamp to the packet?

--Jesper