Re: [RFC bpf-next 2/6] net: xdp: RX meta data infrastructure

From: "Waskiewicz Jr, Peter" <peter.waskiewicz.jr@intel.com>
To: "alexei.starovoitov@gmail.com" <alexei.starovoitov@gmail.com>,
	"jakub.kicinski@netronome.com" <jakub.kicinski@netronome.com>
Cc: "Duyck, Alexander H" <alexander.h.duyck@intel.com>,
	"daniel@iogearbox.net" <daniel@iogearbox.net>,
	"saeedm@mellanox.com" <saeedm@mellanox.com>,
	"brouer@redhat.com" <brouer@redhat.com>,
	"borkmann@iogearbox.net" <borkmann@iogearbox.net>,
	"tariqt@mellanox.com" <tariqt@mellanox.com>,
	"john.fastabend@gmail.com" <john.fastabend@gmail.com>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"saeedm@dev.mellanox.co.il" <saeedm@dev.mellanox.co.il>
Subject: Re: [RFC bpf-next 2/6] net: xdp: RX meta data infrastructure
Date: Fri, 6 Jul 2018 20:44:24 +0000	[thread overview]
Message-ID: <1530909863.4291.12.camel@intel.com> (raw)
In-Reply-To: <20180706163041.xstyfednmgho23m3@ast-mbp.dhcp.thefacebook.com>

On Fri, 2018-07-06 at 09:30 -0700, Alexei Starovoitov wrote:
> On Thu, Jul 05, 2018 at 10:18:23AM -0700, Jakub Kicinski wrote:
> > 
> > I'm also not 100% on board with the argument that "future" FW can
> > reshuffle things whatever way it wants to.  Is the assumption that
> > future ASICs/FW will be designed to always use the "blessed" BTF
> > format?  Or will it be reconfigurable at runtime?
> 
> let's table configuration of metadata aside for a second.

I agree that this should/could be NIC-specific and shouldn't weigh on
the metadata interface between the drivers and XDP layer.

> Describing metedata layout in BTF allows NICs to disclose everything
> NIC has to users in a standard and generic way.
> Whether firmware is reconfigurable on the fly or has to reflashed
> and hw powercycled to have new md layout (and corresponding BTF
> description)
> is a separate discussion.
> Saeed's proposal introduces the concept of 'offset' inside 'struct
> xdp_md_info'
> to reach 'hash' value in metadata.
> Essentially it's a run-time way to access 'hash' instead of build-
> time.
> So bpf program would need two loads to read csum or hash field
> instead of one.
> With BTF the layout of metadata is known to the program at build-
> time.
> 
> To reiterate the proposal:
> - driver+firmware keep layout of the metadata in BTF format (either
> in the driver
>   or driver can read it from firmware)
> - 'bpftool read-metadata-desc eth0 > md_desc.h' command will query
> the driver and
>   generate normal C header file based on BTF in the given NIC
> - user does #include "md_desc.h" and bpf program can access md->csum
> or md->hash
>   with direct single load out of metadata area in front of the packet

This piece is where I'd like to discuss more.  When we discussed this
in Seoul, the initial proposal was a static struct that we'd try to
hammer out a common layout between the interested parties.  That
obviously wasn't going to scale, and we wanted to pursue something more
dynamic.  But I thought the goal was the XDP/eBPF program wouldn't want
to care what the underlying device is, and could just ask for metadata
that it's interested in.  With this approach, your eBPF program is now
bound/tied to the NIC/driver, and if you switched to a differen
NIC/driver combo, then you'd have to rewrite part of your eBPF program
to comprehend that.  I thought we were trying to avoid that.

Our proposed approach (still working on something ready to RFC) is to
provide a method for the eBPF program to send a struct of requested
hints down to the driver on load.  If the driver can provide the hints,
then that'd be how they'd be laid out in the metadata.  If it can't
provide them, we'd probably reject the program loading, or discuss
providing a software fallback (I know this is an area of contention).

I suppose we could get there with the rewriting mechanism described
below, but that'd be a tough sell to set a bit of ABI for metadata,
then change it to be potentially dynamic at runtime in the future.

-PJ