Hardware time stamping support for AF_XDP applications

Hardware time stamping support for AF_XDP applications

[Patel, Vedang]

[Sorry if you got the email twice. Resending because it was rejected by netdev for containing HTML]

Hi Saeed/Jesper, 
 
I am working in the Time Sensitive Networking team at Intel. We work on implementing and upstreaming support for TSN related features for intel based NICs. Recently we have been adding support for XDP in i225. One of the features which we want to add support for is passing the hardware timestamp information to the userspace application running AF_XDP sockets (for both Tx and Rx). I came across the XDP Workshop[1] conducted in July 2020 and there you stated that you are already working on adding support for BTF based metadata to pass hardware hints for XDP Programs. My understanding (along with a few questions) of the current state is: 
* This feature is currently being maintained out of tree. I found that an RFC Series[2] was posted in June 2018. Are you planning to post an updated version to be merged in the mainline anytime soon? 
* I am guessing hardware timestamp is one of the metadata fields which will be eventually supported? [3]
* The Metadata support will be extended to pass on the hardware hints to AF_XDP sockets. Are there any rough plans on what metadata will be transferred?
* The current plan for Tx side only includes passing data from the application to the driver. Are there any plans to support passing information (like HW TX timestamp) from driver to the Application?
 
Finally, is there any way I can help in expediting the development and upstreaming of this feature? I have been working on studying how XDP works and can work on implementing some part of this feature if you would like.
 
Thanks,
Vedang Patel
Software Engineer
Intel Corporation
 
[1] - https://netdevconf.info/0x14/session.html?workshop-XDP
[2] - https://patchwork.ozlabs.org/project/netdev/cover/20180627024615.17856-1-saeedm@mellanox.com/
[3] - https://xdp-project.net/#outline-container-Important-medium-term-tasks

Re: Hardware time stamping support for AF_XDP applications

[Saeed Mahameed]

On Tue, 2020-11-10 at 22:44 +0000, Patel, Vedang wrote:
> [Sorry if you got the email twice. Resending because it was rejected
> by netdev for containing HTML]
> 
> Hi Saeed/Jesper, 
>  
> I am working in the Time Sensitive Networking team at Intel. We work
> on implementing and upstreaming support for TSN related features for
> intel based NICs. Recently we have been adding support for XDP in
> i225. One of the features which we want to add support for is passing
> the hardware timestamp information to the userspace application
> running AF_XDP sockets (for both Tx and Rx). I came across the XDP
> Workshop[1] conducted in July 2020 and there you stated that you are
> already working on adding support for BTF based metadata to pass
> hardware hints for XDP Programs. My understanding (along with a few
> questions) of the current state is: 

Hi Patel,

> * This feature is currently being maintained out of tree. I found
> that an RFC Series[2] was posted in June 2018. Are you planning to
> post an updated version to be merged in the mainline anytime soon? 

Yes hopefully in the coming couple of weeks.

> * I am guessing hardware timestamp is one of the metadata fields
> which will be eventually supported? [3]

With BTF formatted metadata it is up to the driver to advertise
whatever it can/want :)
so yes.

> * The Metadata support will be extended to pass on the hardware hints
> to AF_XDP sockets. Are there any rough plans on what metadata will be
> transferred?

AF_XDP is not part of my series, but supporting AF_XDP with metadata
offlaod is up to the driver to implement, should be straight forward
and identical to XDP.

what meta data to pass is up to the driver.


> * The current plan for Tx side only includes passing data from the
> application to the driver. Are there any plans to support passing
> information (like HW TX timestamp) from driver to the Application?
>  

you mean for AF_XDP ? i actually haven't thought about this, 
but we could use TX umem packet buffer headroom to pass TX completion
metadata to AF_XDP app, or extend the completion queue entries to host
metadata, i am sure that the 1st approach is preferred, but i am not
planing to support this in my initial series. 

> Finally, is there any way I can help in expediting the development
> and upstreaming of this feature? I have been working on studying how
> XDP works and can work on implementing some part of this feature if
> you would like.
>  

Sure,
Please feel free to clone and test the following branch if you add
support to  your driver and implement offloads for AF_XDP that would be
awesome, and i will append your patches to my series before submission.

it is always great to send new features with multiple use cases and
multi vendor support, this will differently expedite submission and
acceptance

My Latest work can be found at:

https://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git/log/?h=topic/xdp_metadata3

Please feel free to send me any questions about the code in private or
public.

Thanks,
Saeed.

> Thanks,
> Vedang Patel
> Software Engineer
> Intel Corporation
>  
> [1] - https://netdevconf.info/0x14/session.html?workshop-XDP
> [2] - 
> https://patchwork.ozlabs.org/project/netdev/cover/20180627024615.17856-1-saeedm@mellanox.com/
> [3] - 
> https://xdp-project.net/#outline-container-Important-medium-term-tasks
> 
> 

Re: Hardware time stamping support for AF_XDP applications

[Patel, Vedang]

Hi Saeed, 

> On Nov 10, 2020, at 3:32 PM, Saeed Mahameed <saeed@kernel.org> wrote:
> 
> On Tue, 2020-11-10 at 22:44 +0000, Patel, Vedang wrote:
>> [Sorry if you got the email twice. Resending because it was rejected
>> by netdev for containing HTML]
>> 
>> Hi Saeed/Jesper, 
>> 
>> I am working in the Time Sensitive Networking team at Intel. We work
>> on implementing and upstreaming support for TSN related features for
>> intel based NICs. Recently we have been adding support for XDP in
>> i225. One of the features which we want to add support for is passing
>> the hardware timestamp information to the userspace application
>> running AF_XDP sockets (for both Tx and Rx). I came across the XDP
>> Workshop[1] conducted in July 2020 and there you stated that you are
>> already working on adding support for BTF based metadata to pass
>> hardware hints for XDP Programs. My understanding (along with a few
>> questions) of the current state is: 
> 
> Hi Patel,
> 
>> * This feature is currently being maintained out of tree. I found
>> that an RFC Series[2] was posted in June 2018. Are you planning to
>> post an updated version to be merged in the mainline anytime soon? 
> 
> Yes hopefully in the coming couple of weeks.
> 
Sure! I will start testing/developing on top of your branch mentioned below for now.
>> * I am guessing hardware timestamp is one of the metadata fields
>> which will be eventually supported? [3]
> 
> With BTF formatted metadata it is up to the driver to advertise
> whatever it can/want :)
> so yes.
I have a very basic question here. From what I understand about BTF, I can generate a header file (using bpftool?) containing the BTF data format provided by the driver. If so, how can I design an application which can work with multiple NICs drivers without recompilation? I am guessing there is some sort of “master list” of HW hints the drivers will agree upon?
> 
>> * The Metadata support will be extended to pass on the hardware hints
>> to AF_XDP sockets. Are there any rough plans on what metadata will be
>> transferred?
> 
> AF_XDP is not part of my series, but supporting AF_XDP with metadata
> offlaod is up to the driver to implement, should be straight forward
> and identical to XDP.
> 
> what meta data to pass is up to the driver.
Alright, let me take a closer look at your latest code. I will come back will questions if I have any.
> 
> 
>> * The current plan for Tx side only includes passing data from the
>> application to the driver. Are there any plans to support passing
>> information (like HW TX timestamp) from driver to the Application?
>> 
> 
> you mean for AF_XDP ? i actually haven't thought about this, 
> but we could use TX umem packet buffer headroom to pass TX completion
> metadata to AF_XDP app, or extend the completion queue entries to host
> metadata, i am sure that the 1st approach is preferred, but i am not
> planing to support this in my initial series. 
> 
Yeah, I was thinking of using approach 1 as well for this. Let me first work on the Rx side. Then we can scope this one out.
>> Finally, is there any way I can help in expediting the development
>> and upstreaming of this feature? I have been working on studying how
>> XDP works and can work on implementing some part of this feature if
>> you would like.
>> 
> 
> Sure,
> Please feel free to clone and test the following branch if you add
> support to  your driver and implement offloads for AF_XDP that would be
> awesome, and i will append your patches to my series before submission.
> 
> it is always great to send new features with multiple use cases and
> multi vendor support, this will differently expedite submission and
> acceptance
> 
> My Latest work can be found at:
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git/log/?h=topic/xdp_metadata3
> 
> Please feel free to send me any questions about the code in private or
> public.
Thanks Saeed for all the information! This is really helpful. :)
> 
> Thanks,
> Saeed.
> 
>> Thanks,
>> Vedang Patel
>> Software Engineer
>> Intel Corporation
>> 
>> [1] - https://netdevconf.info/0x14/session.html?workshop-XDP
>> [2] - 
>> https://patchwork.ozlabs.org/project/netdev/cover/20180627024615.17856-1-saeedm@mellanox.com/
>> [3] - 
>> https://xdp-project.net/#outline-container-Important-medium-term-tasks

Re: Hardware time stamping support for AF_XDP applications

[Jesper Dangaard Brouer]

On Tue, 10 Nov 2020 23:53:41 +0000
"Patel, Vedang" <vedang.patel@intel.com> wrote:

> Hi Saeed, 
> 
> > On Nov 10, 2020, at 3:32 PM, Saeed Mahameed <saeed@kernel.org> wrote:
> > 
> > On Tue, 2020-11-10 at 22:44 +0000, Patel, Vedang wrote:  
> >> [Sorry if you got the email twice. Resending because it was rejected
> >> by netdev for containing HTML]
> >> 
> >> Hi Saeed/Jesper, 
> >> 
> >> I am working in the Time Sensitive Networking team at Intel. We work
> >> on implementing and upstreaming support for TSN related features for
> >> intel based NICs. Recently we have been adding support for XDP in
> >> i225. One of the features which we want to add support for is passing
> >> the hardware timestamp information to the userspace application
> >> running AF_XDP sockets (for both Tx and Rx). I came across the XDP
> >> Workshop[1] conducted in July 2020 and there you stated that you are
> >> already working on adding support for BTF based metadata to pass
> >> hardware hints for XDP Programs. My understanding (along with a few
> >> questions) of the current state is:  

Have the i225 XDP support been upstreamed?

Can I buy a i255 NIC for my server, or is this embedded NICs?

Ilias have played with PoC for TSN (on ARM) here:
 https://github.com/xdp-project/xdp-project/blob/master/areas/arm64/xdp_for_tsn.org

> > Hi Patel,
> >   
> >> * This feature is currently being maintained out of tree. I found
> >> that an RFC Series[2] was posted in June 2018. Are you planning to
> >> post an updated version to be merged in the mainline anytime soon?   
> > 
> > Yes hopefully in the coming couple of weeks.
> >   
>
> Sure! I will start testing/developing on top of your branch mentioned
> below for now.

I've also signed up for helping out on this effort.  Notice Andrii (cc)
have already pointed out something that can be improved, and even made
easier.


> >> * I am guessing hardware timestamp is one of the metadata fields
> >> which will be eventually supported? [3]  
> > 
> > With BTF formatted metadata it is up to the driver to advertise
> > whatever it can/want :)
> > so yes.  
>
> I have a very basic question here. From what I understand about BTF,
> I can generate a header file (using bpftool?) containing the BTF data
> format provided by the driver. If so, how can I design an application
> which can work with multiple NICs drivers without recompilation? I am
> guessing there is some sort of “master list” of HW hints the drivers
> will agree upon?

I recommend that you read Andrii's blogpost:
 https://facebookmicrosites.github.io/bpf/blog/2020/02/19/bpf-portability-and-co-re.html

I need to learn more about BTF myself, but the way I understand this:
We need to agree on the meaning of struct member names (e.g. rxhash32).
Then you compile BPF with a BTF struct that have this rxhash32 member
name, and at BPF load-time the kernel CO-RE infra will remap the offset
to the rxhash32 offset used by the specific driver.

> >   
> >> * The Metadata support will be extended to pass on the hardware hints
> >> to AF_XDP sockets. Are there any rough plans on what metadata will be
> >> transferred?  
> > 
> > AF_XDP is not part of my series, but supporting AF_XDP with metadata
> > offlaod is up to the driver to implement, should be straight forward
> > and identical to XDP.

The XDP data_meta area is also transferred into the AF_XDP frame, also
in the copy-mode version of AF_XDP.


> > what meta data to pass is up to the driver.  
>
> Alright, let me take a closer look at your latest code. I will come
> back will questions if I have any.
> > 
> >   
> >> * The current plan for Tx side only includes passing data from the
> >> application to the driver. Are there any plans to support passing
> >> information (like HW TX timestamp) from driver to the Application?
> >>   
> > 
> > you mean for AF_XDP ? i actually haven't thought about this, 
> > but we could use TX umem packet buffer headroom to pass TX completion
> > metadata to AF_XDP app, or extend the completion queue entries to host
> > metadata, i am sure that the 1st approach is preferred, but i am not
> > planing to support this in my initial series. 
> >   
> Yeah, I was thinking of using approach 1 as well for this. Let me
> first work on the Rx side. Then we can scope this one out.
>
> >> Finally, is there any way I can help in expediting the development
> >> and upstreaming of this feature? I have been working on studying how
> >> XDP works and can work on implementing some part of this feature if
> >> you would like.
> >>   
> > 
> > Sure,
> > Please feel free to clone and test the following branch if you add
> > support to  your driver and implement offloads for AF_XDP that would be
> > awesome, and i will append your patches to my series before submission.
> > 
> > it is always great to send new features with multiple use cases and
> > multi vendor support, this will differently expedite submission and
> > acceptance
> > 
> > My Latest work can be found at:
> > 
> > https://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git/log/?h=topic/xdp_metadata3
> > 
> > Please feel free to send me any questions about the code in private or
> > public.
>
> Thanks Saeed for all the information! This is really helpful. :)
> > 
> > Thanks,
> > Saeed.
> >   
> >> Thanks,
> >> Vedang Patel
> >> Software Engineer
> >> Intel Corporation
> >> 
> >> [1] - https://netdevconf.info/0x14/session.html?workshop-XDP
> >> [2] - 
> >> https://patchwork.ozlabs.org/project/netdev/cover/20180627024615.17856-1-saeedm@mellanox.com/
> >> [3] - 
> >> https://xdp-project.net/#outline-container-Important-medium-term-tasks  
> 



-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Re: Hardware time stamping support for AF_XDP applications

[Patel, Vedang]

[Looks like format settings for my mail client were not persistent and I sent out HTML message again by mistake. Hopefully it’s fixed now. Sorry for the spam. :( ]

Hi Jesper, 

Thanks for all the inputs. My Responses inline:

> On Nov 11, 2020, at 6:53 AM, Jesper Dangaard Brouer <brouer@redhat.com> wrote:
> 
> On Tue, 10 Nov 2020 23:53:41 +0000
> "Patel, Vedang" <vedang.patel@intel.com> wrote:
> 
>> Hi Saeed, 
>> 
>>> On Nov 10, 2020, at 3:32 PM, Saeed Mahameed <saeed@kernel.org> wrote:
>>> 
>>> On Tue, 2020-11-10 at 22:44 +0000, Patel, Vedang wrote:  
>>>> [Sorry if you got the email twice. Resending because it was rejected
>>>> by netdev for containing HTML]
>>>> 
>>>> Hi Saeed/Jesper, 
>>>> 
>>>> I am working in the Time Sensitive Networking team at Intel. We work
>>>> on implementing and upstreaming support for TSN related features for
>>>> intel based NICs. Recently we have been adding support for XDP in
>>>> i225. One of the features which we want to add support for is passing
>>>> the hardware timestamp information to the userspace application
>>>> running AF_XDP sockets (for both Tx and Rx). I came across the XDP
>>>> Workshop[1] conducted in July 2020 and there you stated that you are
>>>> already working on adding support for BTF based metadata to pass
>>>> hardware hints for XDP Programs. My understanding (along with a few
>>>> questions) of the current state is:  
> 
> Have the i225 XDP support been upstreamed?
Andre (CC’d here) is working on adding support. The first series [1] just got merged into dev queue. He is working on adding Zero Copy and UMEM support.

[1] - https://patchwork.ozlabs.org/project/intel-wired-lan/cover/20201104031210.27772-1-andre.guedes@intel.com/
> 
> Can I buy a i255 NIC for my server, or is this embedded NICs?
Yes, i225 NICs are available in discrete form factor. One of the links where it’s publicly available is:
https://shop.qnap.com/index.php?route=product/product&product_id=388

> 
> Ilias have played with PoC for TSN (on ARM) here:
> https://github.com/xdp-project/xdp-project/blob/master/areas/arm64/xdp_for_tsn.org
Our aim is to use the approach described in the last paragraph to gather end-to-end latency.
> 
>>> Hi Patel,
>>> 
>>>> * This feature is currently being maintained out of tree. I found
>>>> that an RFC Series[2] was posted in June 2018. Are you planning to
>>>> post an updated version to be merged in the mainline anytime soon?   
>>> 
>>> Yes hopefully in the coming couple of weeks.
>>> 
>> 
>> Sure! I will start testing/developing on top of your branch mentioned
>> below for now.
> 
> I've also signed up for helping out on this effort.  Notice Andrii (cc)
> have already pointed out something that can be improved, and even made
> easier.
That’s great! Can you point me to the improvement Andrii mentioned?
> 
> 
>>>> * I am guessing hardware timestamp is one of the metadata fields
>>>> which will be eventually supported? [3]  
>>> 
>>> With BTF formatted metadata it is up to the driver to advertise
>>> whatever it can/want :)
>>> so yes.  
>> 
>> I have a very basic question here. From what I understand about BTF,
>> I can generate a header file (using bpftool?) containing the BTF data
>> format provided by the driver. If so, how can I design an application
>> which can work with multiple NICs drivers without recompilation? I am
>> guessing there is some sort of “master list” of HW hints the drivers
>> will agree upon?
> 
> I recommend that you read Andrii's blogpost:
> https://facebookmicrosites.github.io/bpf/blog/2020/02/19/bpf-portability-and-co-re.html
This is quite insightful!
> 
> I need to learn more about BTF myself, but the way I understand this:
> We need to agree on the meaning of struct member names (e.g. rxhash32).
> Then you compile BPF with a BTF struct that have this rxhash32 member
> name, and at BPF load-time the kernel CO-RE infra will remap the offset
> to the rxhash32 offset used by the specific driver.
> 
>>> 
>>>> * The Metadata support will be extended to pass on the hardware hints
>>>> to AF_XDP sockets. Are there any rough plans on what metadata will be
>>>> transferred?  
>>> 
>>> AF_XDP is not part of my series, but supporting AF_XDP with metadata
>>> offlaod is up to the driver to implement, should be straight forward
>>> and identical to XDP.
> 
> The XDP data_meta area is also transferred into the AF_XDP frame, also
> in the copy-mode version of AF_XDP.
> 
> 
>>> what meta data to pass is up to the driver.  
>> 
>> Alright, let me take a closer look at your latest code. I will come
>> back will questions if I have any.
>>> 
>>> 
>>>> * The current plan for Tx side only includes passing data from the
>>>> application to the driver. Are there any plans to support passing
>>>> information (like HW TX timestamp) from driver to the Application?
>>>> 
>>> 
>>> you mean for AF_XDP ? i actually haven't thought about this, 
>>> but we could use TX umem packet buffer headroom to pass TX completion
>>> metadata to AF_XDP app, or extend the completion queue entries to host
>>> metadata, i am sure that the 1st approach is preferred, but i am not
>>> planing to support this in my initial series. 
>>> 
>> Yeah, I was thinking of using approach 1 as well for this. Let me
>> first work on the Rx side. Then we can scope this one out.
>> 
>>>> Finally, is there any way I can help in expediting the development
>>>> and upstreaming of this feature? I have been working on studying how
>>>> XDP works and can work on implementing some part of this feature if
>>>> you would like.
>>>> 
>>> 
>>> Sure,
>>> Please feel free to clone and test the following branch if you add
>>> support to  your driver and implement offloads for AF_XDP that would be
>>> awesome, and i will append your patches to my series before submission.
>>> 
>>> it is always great to send new features with multiple use cases and
>>> multi vendor support, this will differently expedite submission and
>>> acceptance
>>> 
>>> My Latest work can be found at:
>>> 
>>> https://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux.git/log/?h=topic/xdp_metadata3
>>> 
>>> Please feel free to send me any questions about the code in private or
>>> public.
>> 
>> Thanks Saeed for all the information! This is really helpful. :)
>>> 
>>> Thanks,
>>> Saeed.
>>> 
>>>> Thanks,
>>>> Vedang Patel
>>>> Software Engineer
>>>> Intel Corporation
>>>> 
>>>> [1] - https://netdevconf.info/0x14/session.html?workshop-XDP
>>>> [2] - 
>>>> https://patchwork.ozlabs.org/project/netdev/cover/20180627024615.17856-1-saeedm@mellanox.com/
>>>> [3] - 
>>>> https://xdp-project.net/#outline-container-Important-medium-term-tasks  
>> 
> 
> 
> 
> -- 
> Best regards,
>  Jesper Dangaard Brouer
>  MSc.CS, Principal Kernel Engineer at Red Hat
>  LinkedIn: http://www.linkedin.com/in/brouer

Re: Hardware time stamping support for AF_XDP applications

[Saeed Mahameed]

On Tue, 2020-11-10 at 23:53 +0000, Patel, Vedang wrote:
> > With BTF formatted metadata it is up to the driver to advertise
> > whatever it can/want :)
> > so yes.
> 
> I have a very basic question here. From what I understand about BTF,
> I can generate a header file (using bpftool?) containing the BTF data
> format provided by the driver. If so, how can I design an application
> which can work with multiple NICs drivers without recompilation? I am
> guessing there is some sort of “master list” of HW hints the drivers
> will agree upon?

Hi Patel, as Jesper mentioned, some hints will be well defined in BTF
format, by name, size and type, e.g.:

   u32 hash32;
   u16 vlan_tci;
   u64 timestamp;

etc.. 

if the driver reports only well known hints, a program compiled with
these can work in theory on any NIC that supports them. the BPF program
loader/verifier in the kernel can check compatibility before loading a
program on a NIC.

now the question remains, What if different NICs/Drivers re-arrange
those fields differently? 
this also can be solved by the BPF XDP program loader in the kernel at
rung time, it can re-arrange the meta data offsets according to the
current NIC directly in the byte code, but this is going to be a future
work.

Re: Hardware time stamping support for AF_XDP applications

[Patel, Vedang]

Hi Saeed,

> On Nov 12, 2020, at 11:31 AM, Saeed Mahameed <saeed@kernel.org> wrote:
> 
> On Tue, 2020-11-10 at 23:53 +0000, Patel, Vedang wrote:
>>> With BTF formatted metadata it is up to the driver to advertise
>>> whatever it can/want :)
>>> so yes.
>> 
>> I have a very basic question here. From what I understand about BTF,
>> I can generate a header file (using bpftool?) containing the BTF data
>> format provided by the driver. If so, how can I design an application
>> which can work with multiple NICs drivers without recompilation? I am
>> guessing there is some sort of “master list” of HW hints the drivers
>> will agree upon?
> 
> Hi Patel, as Jesper mentioned, some hints will be well defined in BTF
> format, by name, size and type, e.g.:
> 
>   u32 hash32;
>   u16 vlan_tci;
>   u64 timestamp;
> 
> etc.. 
> 
> if the driver reports only well known hints, a program compiled with
> these can work in theory on any NIC that supports them. the BPF program
> loader/verifier in the kernel can check compatibility before loading a
> program on a NIC.
> 
> now the question remains, What if different NICs/Drivers re-arrange
> those fields differently? 
> this also can be solved by the BPF XDP program loader in the kernel at
> rung time, it can re-arrange the meta data offsets according to the
> current NIC directly in the byte code, but this is going to be a future
> work.
> 
Thanks for more info!

I have pulled in your changes and started modifying the igc driver. I will report back on how it goes.

Thanks,
Vedang 

Re: Hardware time stamping support for AF_XDP applications

[Patel, Vedang]

Hi Saeed,

> On Nov 13, 2020, at 10:02 AM, Patel, Vedang <vedang.patel@intel.com> wrote:
> 
> Hi Saeed,
> 
>> On Nov 12, 2020, at 11:31 AM, Saeed Mahameed <saeed@kernel.org> wrote:
>> 
>> On Tue, 2020-11-10 at 23:53 +0000, Patel, Vedang wrote:
>>>> With BTF formatted metadata it is up to the driver to advertise
>>>> whatever it can/want :)
>>>> so yes.
>>> 
>>> I have a very basic question here. From what I understand about BTF,
>>> I can generate a header file (using bpftool?) containing the BTF data
>>> format provided by the driver. If so, how can I design an application
>>> which can work with multiple NICs drivers without recompilation? I am
>>> guessing there is some sort of “master list” of HW hints the drivers
>>> will agree upon?
>> 
>> Hi Patel, as Jesper mentioned, some hints will be well defined in BTF
>> format, by name, size and type, e.g.:
>> 
>>  u32 hash32;
>>  u16 vlan_tci;
>>  u64 timestamp;
>> 
>> etc.. 
>> 
>> if the driver reports only well known hints, a program compiled with
>> these can work in theory on any NIC that supports them. the BPF program
>> loader/verifier in the kernel can check compatibility before loading a
>> program on a NIC.
>> 
>> now the question remains, What if different NICs/Drivers re-arrange
>> those fields differently? 
>> this also can be solved by the BPF XDP program loader in the kernel at
>> rung time, it can re-arrange the meta data offsets according to the
>> current NIC directly in the byte code, but this is going to be a future
>> work.
>> 
> Thanks for more info!
> 
> I have pulled in your changes and started modifying the igc driver. I will report back on how it goes.
> 
I was able to add support for igc and now I have a few more opens about the feature: 
- i225 contains multiple PHC clocks which can report timestamps for a packet. So, along with the timestamp, we also need to return clock id which was used to timestamp the packet. I was wondering if there are other NICs which have similar functionalities so that we can align on the interface used to present the timestamp to AF_XDP sockets. 
- I am also planning to add capability to set timestamp using AF_XDP socket. I am just planning to add another socket option (like what is done in AF_PACKET). Let me know if there are other ideas.
- From what I understand metadata size will be accounted for in the XDP_PACKET_HEADROOM. Is that correct? If so, what will happen in the scenario when the metadata exceeds the amount of space available (XDP_PACKET_HEADROOM - sizeof(struct xdp_frame))?
- From looking at the code, it looks like BTF metadata will be enabled at NIC level. So, all AF_XDP sockets will receive metadata whether they want it or not. Do we need a knob for AF_XDP sockets to enable metadata? I think applications which expect larger frame size will not want metadata in order to save space.

Thanks,
Vedang
> Thanks,
> Vedang

Re: Hardware time stamping support for AF_XDP applications

[Patel, Vedang]

> On Nov 18, 2020, at 4:57 PM, Patel, Vedang <vedang.patel@intel.com> wrote:
> 
> Hi Saeed,
> 
>> On Nov 13, 2020, at 10:02 AM, Patel, Vedang <vedang.patel@intel.com> wrote:
>> 
>> Hi Saeed,
>> 
>>> On Nov 12, 2020, at 11:31 AM, Saeed Mahameed <saeed@kernel.org> wrote:
>>> 
>>> On Tue, 2020-11-10 at 23:53 +0000, Patel, Vedang wrote:
>>>>> With BTF formatted metadata it is up to the driver to advertise
>>>>> whatever it can/want :)
>>>>> so yes.
>>>> 
>>>> I have a very basic question here. From what I understand about BTF,
>>>> I can generate a header file (using bpftool?) containing the BTF data
>>>> format provided by the driver. If so, how can I design an application
>>>> which can work with multiple NICs drivers without recompilation? I am
>>>> guessing there is some sort of “master list” of HW hints the drivers
>>>> will agree upon?
>>> 
>>> Hi Patel, as Jesper mentioned, some hints will be well defined in BTF
>>> format, by name, size and type, e.g.:
>>> 
>>> u32 hash32;
>>> u16 vlan_tci;
>>> u64 timestamp;
>>> 
>>> etc.. 
>>> 
>>> if the driver reports only well known hints, a program compiled with
>>> these can work in theory on any NIC that supports them. the BPF program
>>> loader/verifier in the kernel can check compatibility before loading a
>>> program on a NIC.
>>> 
>>> now the question remains, What if different NICs/Drivers re-arrange
>>> those fields differently? 
>>> this also can be solved by the BPF XDP program loader in the kernel at
>>> rung time, it can re-arrange the meta data offsets according to the
>>> current NIC directly in the byte code, but this is going to be a future
>>> work.
>>> 
>> Thanks for more info!
>> 
>> I have pulled in your changes and started modifying the igc driver. I will report back on how it goes.
>> 
> I was able to add support for igc and now I have a few more opens about the feature: 
> - i225 contains multiple PHC clocks which can report timestamps for a packet. So, along with the timestamp, we also need to return clock id which was used to timestamp the packet. I was wondering if there are other NICs which have similar functionalities so that we can align on the interface used to present the timestamp to AF_XDP sockets. 
> - I am also planning to add capability to set timestamp using AF_XDP socket. I am just planning to add another socket option (like what is done in AF_PACKET). Let me know if there are other ideas.
Looking a bit more into the code, the only think which is required to enable timestamping to issue ioctl() call to the driver which is already implemented for i225. So, there is no change required in the XDP infrastructure for this.
> - From what I understand metadata size will be accounted for in the XDP_PACKET_HEADROOM. Is that correct? If so, what will happen in the scenario when the metadata exceeds the amount of space available (XDP_PACKET_HEADROOM - sizeof(struct xdp_frame))?
> - From looking at the code, it looks like BTF metadata will be enabled at NIC level. So, all AF_XDP sockets will receive metadata whether they want it or not. Do we need a knob for AF_XDP sockets to enable metadata? I think applications which expect larger frame size will not want metadata in order to save space.
> 
> Thanks,
> Vedang
>> Thanks,
>> Vedang