Re: AF_XDP metadata/hints

From: Jesper Dangaard Brouer <brouer@redhat.com>
To: John Fastabend <john.fastabend@gmail.com>
Cc: "Toke Høiland-Jørgensen" <toke@redhat.com>,
	"Alexander Lobakin" <alexandr.lobakin@intel.com>,
	"Saeed Mahameed" <saeed@kernel.org>,
	"Raczynski, Piotr" <piotr.raczynski@intel.com>,
	"Zhang, Jessica" <jessica.zhang@intel.com>,
	"Kubiak, Marcin" <marcin.kubiak@intel.com>,
	"Joseph, Jithu" <jithu.joseph@intel.com>,
	"kurt@linutronix.de" <kurt@linutronix.de>,
	"Maloor, Kishen" <kishen.maloor@intel.com>,
	"Gomes, Vinicius" <vinicius.gomes@intel.com>,
	"Brandeburg, Jesse" <jesse.brandeburg@intel.com>,
	"Swiatkowski, Michal" <michal.swiatkowski@intel.com>,
	"Plantykow, Marta A" <marta.a.plantykow@intel.com>,
	"Ong, Boon Leong" <boon.leong.ong@intel.com>,
	"Desouza, Ederson" <ederson.desouza@intel.com>,
	"Song, Yoong Siang" <yoong.siang.song@intel.com>,
	"Czapnik, Lukasz" <lukasz.czapnik@intel.com>,
	bpf@vger.kernel.org, brouer@redhat.com
Subject: Re: AF_XDP metadata/hints
Date: Wed, 26 May 2021 19:38:18 +0200	[thread overview]
Message-ID: <20210526193818.2fda7dba@carbon> (raw)
In-Reply-To: <60ae6ad5a2e04_18bf20819@john-XPS-13-9370.notmuch>

On Wed, 26 May 2021 08:35:49 -0700
John Fastabend <john.fastabend@gmail.com> wrote:

> I'll still have a basic question though. I've never invested much time
> into the hints because its still not clear to me what the use case is?
> What would we put in the hints and do we have any data to show it would be
> a performance win.

I've documented and measured[1] the performance overhead of the missing
checksum for UDP packets when XDP-redirecting into veth (that does
XDP_PASS).  Full delivery into a socket we can save 8% (54.28 ns /
+109Kpps).  Lorenzo is working patches outside XDP-hints for this, but
it only handle CHECKSUM_UNNECESSARY, and if we need CHECKSUM_COMPLETE
then we also need XDP-hints/metadata (for storing skb->csum).

[1] https://github.com/xdp-project/xdp-project/blob/master/areas/core/xdp_frame01_checksum.org

> If its a simple hash of the headers then how would we use it?

Even with a simple/smaller hash you can tune the RSS-hash on parts of
the packet you like, see[2]. That would be valuable for doing lookups
in BPF-maps.

[2] https://github.com/stackpath/rxtxcpu/tree/master/Documentation/case-studies

For cpumap redirect I would like to spread packets with this RX-hash,
as I can avoid parsing packets headers on RX-CPU.

The mlx5 NIC support 64-bit unique flow hash, that you could use as a
lookup key in your (Cilium) conntrack table, or a container/sockmap
redirect-tracking table.

> The map_lookup/updates use IP addrs for keys in Cilium. So I think the
> suggestion is to offload the jhash operation? But that requires some
> program changes to work. Could someone convince me?

Is my explanations enough, or are you still not convinced? 

> Maybe packet timestamp?

I have a concrete use-case that needs packet timestamps for AF_XDP. It
is the control system inside a wind-turbine that use a time-triggered
Real-Time protocol.  I actually both need hardware RX-timestamps and
TX-timestamps.  A lot it lacking on the TX-side to allow AF_XDP to send
down a transmission timestamp, but I have a real-use-case that needs
this (before end-of year).  I have hardware i210 and i225 chips in my
testlab so I can get this working.

Thanks you John for engaging and challenging us in our design of
XDP-hints, I truly appreciate your feedback! :-)
-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer