Questions regarding AF_XDP (ETF, TX queue selection)

Questions regarding AF_XDP (ETF, TX queue selection)

[Deric, Nemanja]

Dear XDP-newbies,

I am Nemanja Deric, and I am working as a Researcher for Industrial 
Networking Department of Siemens AG (Munich, Germany). Recently, 
we have started looking into AF_XDP/XDP and its applicability for some 
of our industrial use cases. Thus, we have a few questions about the 
technology which we hope that this mailing list can help us clarify.

If you believe that these questions are better suited for some other 
mailing list or direct email contact, please let me know so that I can 
contact the correct person/mailing list.

ETF, Timestamping:
In comparison to AF_PACKET sockets AF_XDP sockets do not support 
timestamping (SOL_TXTIME) yet. There seems no way to pass such 
request or meta-information to the kernel / NIC. Is that correct? We 
already heard about "xdp hints" but we have failed to get further 
information like plans / roadmap / implementation state / ideas. 
Any pointers where to look at?

TX queue selection:
AF_XDP sockets are always bound to a specific RX queue. Instead, 
TX queues are "automatically" chosen by the kernel - TX is more or 
less handled by NAPI. Correct? Has someone already thought about 
TX queue selection? Any plans / roadmap / details according 
implementation state available?

In advance, thank you for your help.

Regards,
Nemanja

Re: Questions regarding AF_XDP (ETF, TX queue selection)

[Toke Høiland-Jørgensen]

"Deric, Nemanja" <nemanja.deric@siemens.com> writes:

> Dear XDP-newbies,
>
> I am Nemanja Deric, and I am working as a Researcher for Industrial 
> Networking Department of Siemens AG (Munich, Germany). Recently, 
> we have started looking into AF_XDP/XDP and its applicability for some 
> of our industrial use cases. Thus, we have a few questions about the 
> technology which we hope that this mailing list can help us clarify.
>
> If you believe that these questions are better suited for some other 
> mailing list or direct email contact, please let me know so that I can 
> contact the correct person/mailing list.
>
> ETF, Timestamping:
> In comparison to AF_PACKET sockets AF_XDP sockets do not support 
> timestamping (SOL_TXTIME) yet. There seems no way to pass such 
> request or meta-information to the kernel / NIC. Is that correct? We 
> already heard about "xdp hints" but we have failed to get further 
> information like plans / roadmap / implementation state / ideas. 
> Any pointers where to look at?

Your understanding is basically correct: there's currently no support
for that kind of metadata in XDP at all (which extends to AF_XDP). The
effort to fix that is xdp-hints, and timestamps are explicitly part of
that. That's still work in progress; there's a mailing list dedicated
to this that you may want to subscribe to (and feel free to send a
message there indicating your interest!):

https://lists.xdp-project.net/postorius/lists/xdp-hints.xdp-project.net/

The closest you can get to a "progress report" is probably looking
through the archive of that list.

> TX queue selection:
> AF_XDP sockets are always bound to a specific RX queue. Instead, 
> TX queues are "automatically" chosen by the kernel - TX is more or 
> less handled by NAPI. Correct? Has someone already thought about 
> TX queue selection? Any plans / roadmap / details according 
> implementation state available?

+Magnus who may have an idea about this.

-Toke

Re: Questions regarding AF_XDP (ETF, TX queue selection)

[Magnus Karlsson]

On Sat, Apr 9, 2022 at 3:00 PM Toke Høiland-Jørgensen <toke@redhat.com> wrote:
>
> "Deric, Nemanja" <nemanja.deric@siemens.com> writes:
>
> > Dear XDP-newbies,
> >
> > I am Nemanja Deric, and I am working as a Researcher for Industrial
> > Networking Department of Siemens AG (Munich, Germany). Recently,
> > we have started looking into AF_XDP/XDP and its applicability for some
> > of our industrial use cases. Thus, we have a few questions about the
> > technology which we hope that this mailing list can help us clarify.
> >
> > If you believe that these questions are better suited for some other
> > mailing list or direct email contact, please let me know so that I can
> > contact the correct person/mailing list.
> >
> > ETF, Timestamping:
> > In comparison to AF_PACKET sockets AF_XDP sockets do not support
> > timestamping (SOL_TXTIME) yet. There seems no way to pass such
> > request or meta-information to the kernel / NIC. Is that correct? We
> > already heard about "xdp hints" but we have failed to get further
> > information like plans / roadmap / implementation state / ideas.
> > Any pointers where to look at?
>
> Your understanding is basically correct: there's currently no support
> for that kind of metadata in XDP at all (which extends to AF_XDP). The
> effort to fix that is xdp-hints, and timestamps are explicitly part of
> that. That's still work in progress; there's a mailing list dedicated
> to this that you may want to subscribe to (and feel free to send a
> message there indicating your interest!):
>
> https://lists.xdp-project.net/postorius/lists/xdp-hints.xdp-project.net/
>
> The closest you can get to a "progress report" is probably looking
> through the archive of that list.
>
> > TX queue selection:
> > AF_XDP sockets are always bound to a specific RX queue. Instead,
> > TX queues are "automatically" chosen by the kernel - TX is more or
> > less handled by NAPI. Correct? Has someone already thought about
> > TX queue selection? Any plans / roadmap / details according
> > implementation state available?
>
> +Magnus who may have an idea about this.

Tx queue id = Rx queue id if you create an AF_XDP socket with both Rx
and Tx. So if you bind to queue 2, you will receive any traffic that
is bound to that queue id on that socket and any packets sent on the
same socket will be transferred using queue id 2 (not any other
queue). This is true for both SKB mode (copy mode) and zero-copy mode.
Will this suffice for you? If not, please let me know of your
requirements.

The answer to the NAPI question depends on what mode you are running
in. In SKB mode the sending is done in process context and Tx
completion is performed in NAPI, just as it is with any regular
socket. In zero-copy mode, both the sending and completion are
performed in NAPI context. Note, that since SKB mode uses the regular
send path of the kernel, packet loss is possible. Though zero-copy
mode is guaranteed to have no packet loss, at least on all Intel
drivers.

/Magnus

> -Toke
>