From: Magnus Karlsson <magnus.karlsson@gmail.com>
To: Stanislav Fomichev <sdf@google.com>
Cc: "Toke Høiland-Jørgensen" <toke@kernel.org>,
bpf@vger.kernel.org, ast@kernel.org, daniel@iogearbox.net,
andrii@kernel.org, martin.lau@linux.dev, song@kernel.org,
yhs@fb.com, john.fastabend@gmail.com, kpsingh@kernel.org,
haoluo@google.com, jolsa@kernel.org, willemb@google.com,
dsahern@kernel.org, magnus.karlsson@intel.com, bjorn@kernel.org,
maciej.fijalkowski@intel.com, netdev@vger.kernel.org
Subject: Re: [RFC bpf-next 0/7] bpf: netdev TX metadata
Date: Mon, 19 Jun 2023 09:15:27 +0200 [thread overview]
Message-ID: <CAJ8uoz2zOHpBRfKhN97eR0VWipBTxnh=R9G57Z2UUujX4JzneQ@mail.gmail.com> (raw)
In-Reply-To: <ZIzr5ffeHsUqppaS@google.com>
On Sat, 17 Jun 2023 at 01:10, Stanislav Fomichev <sdf@google.com> wrote:
>
> On 06/16, Stanislav Fomichev wrote:
> > On Fri, Jun 16, 2023 at 1:13 AM Magnus Karlsson
> > <magnus.karlsson@gmail.com> wrote:
> > >
> > > On Fri, 16 Jun 2023 at 02:09, Stanislav Fomichev <sdf@google.com> wrote:
> > > >
> > > > On Mon, Jun 12, 2023 at 2:01 PM Toke Høiland-Jørgensen <toke@kernel.org> wrote:
> > > > >
> > > > > Some immediate thoughts after glancing through this:
> > > > >
> > > > > > --- Use cases ---
> > > > > >
> > > > > > The goal of this series is to add two new standard-ish places
> > > > > > in the transmit path:
> > > > > >
> > > > > > 1. Right before the packet is transmitted (with access to TX
> > > > > > descriptors)
> > > > > > 2. Right after the packet is actually transmitted and we've received the
> > > > > > completion (again, with access to TX completion descriptors)
> > > > > >
> > > > > > Accessing TX descriptors unlocks the following use-cases:
> > > > > >
> > > > > > - Setting device hints at TX: XDP/AF_XDP might use these new hooks to
> > > > > > use device offloads. The existing case implements TX timestamp.
> > > > > > - Observability: global per-netdev hooks can be used for tracing
> > > > > > the packets and exploring completion descriptors for all sorts of
> > > > > > device errors.
> > > > > >
> > > > > > Accessing TX descriptors also means that the hooks have to be called
> > > > > > from the drivers.
> > > > > >
> > > > > > The hooks are a light-weight alternative to XDP at egress and currently
> > > > > > don't provide any packet modification abilities. However, eventually,
> > > > > > can expose new kfuncs to operate on the packet (or, rather, the actual
> > > > > > descriptors; for performance sake).
> > > > >
> > > > > dynptr?
> > > > >
> > > > > > --- UAPI ---
> > > > > >
> > > > > > The hooks are implemented in a HID-BPF style. Meaning they don't
> > > > > > expose any UAPI and are implemented as tracing programs that call
> > > > > > a bunch of kfuncs. The attach/detach operation happen via BPF syscall
> > > > > > programs. The series expands device-bound infrastructure to tracing
> > > > > > programs.
> > > > >
> > > > > Not a fan of the "attach from BPF syscall program" thing. These are part
> > > > > of the XDP data path API, and I think we should expose them as proper
> > > > > bpf_link attachments from userspace with introspection etc. But I guess
> > > > > the bpf_mprog thing will give us that?
> > > > >
> > > > > > --- skb vs xdp ---
> > > > > >
> > > > > > The hooks operate on a new light-weight devtx_frame which contains:
> > > > > > - data
> > > > > > - len
> > > > > > - sinfo
> > > > > >
> > > > > > This should allow us to have a unified (from BPF POW) place at TX
> > > > > > and not be super-taxing (we need to copy 2 pointers + len to the stack
> > > > > > for each invocation).
> > > > >
> > > > > Not sure what I think about this one. At the very least I think we
> > > > > should expose xdp->data_meta as well. I'm not sure what the use case for
> > > > > accessing skbs is? If that *is* indeed useful, probably there will also
> > > > > end up being a use case for accessing the full skb?
> > > >
> > > > I spent some time looking at data_meta story on AF_XDP TX and it
> > > > doesn't look like it's supported (at least in a general way).
> > > > You obviously get some data_meta when you do XDP_TX, but if you want
> > > > to pass something to the bpf prog when doing TX via the AF_XDP ring,
> > > > it gets complicated.
> > >
> > > When we designed this some 5 - 6 years ago, we thought that there
> > > would be an XDP for egress action in the "nearish" future that could
> > > be used to interpret the metadata field in front of the packet.
> > > Basically, the user would load an XDP egress program that would define
> > > the metadata layout by the operations it would perform on the metadata
> > > area. But since XDP on egress has not happened, you are right, there
> > > is definitely something missing to be able to use metadata on Tx. Or
> > > could your proposed hook points be used for something like this?
> >
> > Thanks for the context!
> > Yes, the proposal is to use these new tx hooks to read out af_xdp
> > metadata and apply it to the packet via a bunch of tbd kfuncs.
> > AF_XDP and BPF programs would have to have a contract about the
> > metadata layout (same as we have on rx).
> >
> > > > In zerocopy mode, we can probably use XDP_UMEM_UNALIGNED_CHUNK_FLAG
> > > > and pass something in the headroom.
> > >
> > > This feature is mainly used to allow for multiple packets on the same
> > > chunk (to save space) and also to be able to have packets spanning two
> > > chunks. Even in aligned mode, you can start a packet at an arbitrary
> > > address in the chunk as long as the whole packet fits into the chunk.
> > > So no problem having headroom in any of the modes.
> >
> > But if I put it into the headroom it will only be passed down to the
> > driver in zero-copy mode, right?
> > If I do tx_desc->addr = packet_start, no medata (that goes prior to
> > packet_start) gets copied into skb in the copy mode (it seems).
> > Or do you suggest that the interface should be tx_desc->addr =
> > metadata_start and the bpf program should call the equivalent of
> > bpf_xdp_adjust_head to consume this metadata?
>
> For copy-mode, here is what I've prototyped. That seems to work.
> For zero-copy, I don't think we need anything extra (besides exposing
> xsk->tx_meta_len at the hook point, tbd). Does the patch below make
> sense?
Was just going to suggest adding a setsockopt, so this makes perfect
sense to me. Thanks!
> diff --git a/include/net/xdp_sock.h b/include/net/xdp_sock.h
> index e96a1151ec75..30018b3b862d 100644
> --- a/include/net/xdp_sock.h
> +++ b/include/net/xdp_sock.h
> @@ -51,6 +51,7 @@ struct xdp_sock {
> struct list_head flush_node;
> struct xsk_buff_pool *pool;
> u16 queue_id;
> + u8 tx_metadata_len;
> bool zc;
> enum {
> XSK_READY = 0,
> diff --git a/include/uapi/linux/if_xdp.h b/include/uapi/linux/if_xdp.h
> index a78a8096f4ce..2374eafff7db 100644
> --- a/include/uapi/linux/if_xdp.h
> +++ b/include/uapi/linux/if_xdp.h
> @@ -63,6 +63,7 @@ struct xdp_mmap_offsets {
> #define XDP_UMEM_COMPLETION_RING 6
> #define XDP_STATISTICS 7
> #define XDP_OPTIONS 8
> +#define XDP_TX_METADATA_LEN 9
>
> struct xdp_umem_reg {
> __u64 addr; /* Start of packet data area */
> diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c
> index cc1e7f15fa73..a95872712547 100644
> --- a/net/xdp/xsk.c
> +++ b/net/xdp/xsk.c
> @@ -493,14 +493,21 @@ static struct sk_buff *xsk_build_skb(struct xdp_sock *xs,
> return ERR_PTR(err);
>
> skb_reserve(skb, hr);
> - skb_put(skb, len);
> + skb_put(skb, len + xs->tx_metadata_len);
>
> buffer = xsk_buff_raw_get_data(xs->pool, desc->addr);
> + buffer -= xs->tx_metadata_len;
> +
> err = skb_store_bits(skb, 0, buffer, len);
> if (unlikely(err)) {
> kfree_skb(skb);
> return ERR_PTR(err);
> }
> +
> + if (xs->tx_metadata_len) {
> + skb_metadata_set(skb, xs->tx_metadata_len);
> + __skb_pull(skb, xs->tx_metadata_len);
> + }
> }
>
> skb->dev = dev;
> @@ -1137,6 +1144,27 @@ static int xsk_setsockopt(struct socket *sock, int level, int optname,
> mutex_unlock(&xs->mutex);
> return err;
> }
> + case XDP_TX_METADATA_LEN:
> + {
> + int val;
> +
> + if (optlen < sizeof(val))
> + return -EINVAL;
> + if (copy_from_sockptr(&val, optval, sizeof(val)))
> + return -EFAULT;
> +
> + if (val >= 256)
> + return -EINVAL;
> +
> + mutex_lock(&xs->mutex);
> + if (xs->state != XSK_READY) {
> + mutex_unlock(&xs->mutex);
> + return -EBUSY;
> + }
> + xs->tx_metadata_len = val;
> + mutex_unlock(&xs->mutex);
> + return err;
> + }
> default:
> break;
> }
next prev parent reply other threads:[~2023-06-19 7:15 UTC|newest]
Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-06-12 17:23 [RFC bpf-next 0/7] bpf: netdev TX metadata Stanislav Fomichev
2023-06-12 17:23 ` [RFC bpf-next 1/7] bpf: rename some xdp-metadata functions into dev-bound Stanislav Fomichev
2023-06-12 17:23 ` [RFC bpf-next 2/7] bpf: resolve single typedef when walking structs Stanislav Fomichev
2023-06-12 17:23 ` [RFC bpf-next 3/7] bpf: implement devtx hook points Stanislav Fomichev
2023-06-13 14:54 ` Willem de Bruijn
2023-06-13 19:00 ` Stanislav Fomichev
2023-06-13 19:29 ` Willem de Bruijn
2023-06-13 15:08 ` Simon Horman
2023-06-13 19:00 ` Stanislav Fomichev
2023-06-14 7:02 ` Simon Horman
2023-06-14 17:18 ` Stanislav Fomichev
2023-06-16 5:46 ` Kui-Feng Lee
2023-06-16 17:32 ` Stanislav Fomichev
2023-06-12 17:23 ` [RFC bpf-next 4/7] bpf: implement devtx timestamp kfunc Stanislav Fomichev
2023-06-13 15:14 ` Simon Horman
2023-06-13 18:39 ` Stanislav Fomichev
2023-06-12 17:23 ` [RFC bpf-next 5/7] net: veth: implement devtx timestamp kfuncs Stanislav Fomichev
2023-06-12 17:23 ` [RFC bpf-next 6/7] selftests/bpf: extend xdp_metadata with devtx kfuncs Stanislav Fomichev
2023-06-13 14:47 ` Willem de Bruijn
2023-06-13 19:00 ` Stanislav Fomichev
2023-06-12 17:23 ` [RFC bpf-next 7/7] selftests/bpf: extend xdp_hw_metadata " Stanislav Fomichev
2023-06-13 15:03 ` Willem de Bruijn
2023-06-13 19:00 ` Stanislav Fomichev
2023-06-12 21:00 ` [RFC bpf-next 0/7] bpf: netdev TX metadata Toke Høiland-Jørgensen
2023-06-13 16:32 ` Stanislav Fomichev
2023-06-13 17:18 ` Toke Høiland-Jørgensen
2023-06-13 18:39 ` Stanislav Fomichev
2023-06-13 19:10 ` Toke Høiland-Jørgensen
2023-06-13 21:17 ` Stanislav Fomichev
2023-06-13 22:32 ` Alexei Starovoitov
2023-06-13 23:16 ` Stanislav Fomichev
2023-06-14 4:19 ` Alexei Starovoitov
2023-06-14 11:59 ` Toke Høiland-Jørgensen
2023-06-14 16:27 ` Alexei Starovoitov
2023-06-15 12:36 ` Toke Høiland-Jørgensen
2023-06-15 16:10 ` Alexei Starovoitov
2023-06-15 16:31 ` Stanislav Fomichev
2023-06-16 1:50 ` Jakub Kicinski
2023-06-16 0:09 ` Stanislav Fomichev
2023-06-16 8:12 ` Magnus Karlsson
2023-06-16 17:32 ` Stanislav Fomichev
2023-06-16 23:10 ` Stanislav Fomichev
2023-06-19 7:15 ` Magnus Karlsson [this message]
2023-06-14 3:31 ` Jakub Kicinski
2023-06-14 3:54 ` David Ahern
2023-06-14 5:05 ` Jakub Kicinski
2023-06-14 17:17 ` Stanislav Fomichev
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='CAJ8uoz2zOHpBRfKhN97eR0VWipBTxnh=R9G57Z2UUujX4JzneQ@mail.gmail.com' \
--to=magnus.karlsson@gmail.com \
--cc=andrii@kernel.org \
--cc=ast@kernel.org \
--cc=bjorn@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=daniel@iogearbox.net \
--cc=dsahern@kernel.org \
--cc=haoluo@google.com \
--cc=john.fastabend@gmail.com \
--cc=jolsa@kernel.org \
--cc=kpsingh@kernel.org \
--cc=maciej.fijalkowski@intel.com \
--cc=magnus.karlsson@intel.com \
--cc=martin.lau@linux.dev \
--cc=netdev@vger.kernel.org \
--cc=sdf@google.com \
--cc=song@kernel.org \
--cc=toke@kernel.org \
--cc=willemb@google.com \
--cc=yhs@fb.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).