bpf.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: "Richardson, Bruce" <bruce.richardson@intel.com>
To: Jonathan Lemon <jonathan.lemon@gmail.com>,
	"Laatz, Kevin" <kevin.laatz@intel.com>
Cc: "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"ast@kernel.org" <ast@kernel.org>,
	"daniel@iogearbox.net" <daniel@iogearbox.net>,
	"Topel, Bjorn" <bjorn.topel@intel.com>,
	"Karlsson, Magnus" <magnus.karlsson@intel.com>,
	"jakub.kicinski@netronome.com" <jakub.kicinski@netronome.com>,
	"saeedm@mellanox.com" <saeedm@mellanox.com>,
	"maximmi@mellanox.com" <maximmi@mellanox.com>,
	"stephen@networkplumber.org" <stephen@networkplumber.org>,
	"Loftus, Ciara" <ciara.loftus@intel.com>,
	"bpf@vger.kernel.org" <bpf@vger.kernel.org>,
	"intel-wired-lan@lists.osuosl.org"
	<intel-wired-lan@lists.osuosl.org>
Subject: RE: [PATCH bpf-next v3 00/11] XDP unaligned chunk placement support
Date: Thu, 25 Jul 2019 15:56:15 +0000	[thread overview]
Message-ID: <59AF69C657FD0841A61C55336867B5B07EDB5C3F@IRSMSX103.ger.corp.intel.com> (raw)
In-Reply-To: <94EAD717-F632-499F-8BBD-FFF5A5333CBF@gmail.com>



> -----Original Message-----
> From: Jonathan Lemon [mailto:jonathan.lemon@gmail.com]
> Sent: Thursday, July 25, 2019 4:39 PM
> To: Laatz, Kevin <kevin.laatz@intel.com>
> Cc: netdev@vger.kernel.org; ast@kernel.org; daniel@iogearbox.net; Topel,
> Bjorn <bjorn.topel@intel.com>; Karlsson, Magnus
> <magnus.karlsson@intel.com>; jakub.kicinski@netronome.com;
> saeedm@mellanox.com; maximmi@mellanox.com; stephen@networkplumber.org;
> Richardson, Bruce <bruce.richardson@intel.com>; Loftus, Ciara
> <ciara.loftus@intel.com>; bpf@vger.kernel.org; intel-wired-
> lan@lists.osuosl.org
> Subject: Re: [PATCH bpf-next v3 00/11] XDP unaligned chunk placement
> support
> 
> 
> 
> On 23 Jul 2019, at 22:10, Kevin Laatz wrote:
> 
> > This patch set adds the ability to use unaligned chunks in the XDP umem.
> >
> > Currently, all chunk addresses passed to the umem are masked to be
> > chunk size aligned (max is PAGE_SIZE). This limits where we can place
> > chunks within the umem as well as limiting the packet sizes that are
> supported.
> >
> > The changes in this patch set removes these restrictions, allowing XDP
> > to be more flexible in where it can place a chunk within a umem. By
> > relaxing where the chunks can be placed, it allows us to use an
> > arbitrary buffer size and place that wherever we have a free address
> > in the umem. These changes add the ability to support arbitrary frame
> > sizes up to 4k
> > (PAGE_SIZE) and make it easy to integrate with other existing
> > frameworks that have their own memory management systems, such as DPDK.
> > In DPDK, for example, there is already support for AF_XDP with zero-
> copy.
> > However, with this patch set the integration will be much more seamless.
> > You can find the DPDK AF_XDP driver at:
> > https://git.dpdk.org/dpdk/tree/drivers/net/af_xdp
> >
> > Since we are now dealing with arbitrary frame sizes, we need also need
> > to update how we pass around addresses. Currently, the addresses can
> > simply be masked to 2k to get back to the original address. This
> > becomes less trivial when using frame sizes that are not a 'power of
> > 2' size. This patch set modifies the Rx/Tx descriptor format to use
> > the upper 16-bits of the addr field for an offset value, leaving the
> > lower 48-bits for the address (this leaves us with 256 Terabytes,
> > which should be enough!). We only need to use the upper 16-bits to store
> the offset when running in unaligned mode.
> > Rather than adding the offset (headroom etc) to the address, we will
> > store it in the upper 16-bits of the address field. This way, we can
> > easily add the offset to the address where we need it, using some bit
> > manipulation and addition, and we can also easily get the original
> > address wherever we need it (for example in i40e_zca_fr-- ee) by
> > simply masking to get the lower 48-bits of the address field.
> 
> I wonder if it would be better to break backwards compatibility here and
> say that a handle is going to change from [addr] to [base | offset], or
> even [index | offset], where address = (index * chunk size) + offset, and
> then use accessor macros to manipulate the queue entries.
> 
> This way, the XDP hotpath can adjust the handle with simple arithmetic,
> bypassing the "if (unaligned)", check, as it changes the offset directly.
> 
> Using a chunk index instead of a base address is safer, otherwise it is
> too easy to corrupt things.
> --

The trouble with using a chunk index is that it assumes that all chunks are
contiguous, which is not always going to be the case. For example, for
userspace apps the easiest way to get memory that is IOVA/physically 
contiguous is to use hugepages, but even then we still need to skip space
when crossing a 2MB barrier.

Specifically in this example case, with a 3k buffer size and 2MB hugepages,
we'd get 666 buffers on a single page, but then waste a few KB before
starting the 667th buffer at byte 0 on the second 2MB page.
This is the setup used in DPDK, for instance, when we allocate memory for
use in buffer pools. 

Therefore, I think it's better to just have the kernel sanity checking the
request for safety and leave userspace the freedom to decide where in its
memory area it wants to place the buffers.

Regards,
/Bruce

  reply	other threads:[~2019-07-25 15:56 UTC|newest]

Thread overview: 136+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-06-20  9:09 [PATCH 00/11] XDP unaligned chunk placement support Kevin Laatz
2019-06-20  9:09 ` [PATCH 01/11] i40e: simplify Rx buffer recycle Kevin Laatz
2019-06-24 14:29   ` Björn Töpel
2019-06-20  9:09 ` [PATCH 02/11] ixgbe: " Kevin Laatz
2019-06-24 14:30   ` Björn Töpel
2019-06-20  9:09 ` [PATCH 03/11] xdp: add offset param to zero_copy_allocator Kevin Laatz
2019-06-24 14:31   ` Björn Töpel
2019-06-24 19:23   ` Jakub Kicinski
2019-06-25 13:14     ` Laatz, Kevin
2019-06-20  9:09 ` [PATCH 04/11] i40e: add offset to zca_free Kevin Laatz
2019-06-24 14:32   ` Björn Töpel
2019-06-20  9:09 ` [PATCH 05/11] ixgbe: " Kevin Laatz
2019-06-24 14:32   ` Björn Töpel
2019-06-20  9:09 ` [PATCH 06/11] xsk: add support to allow unaligned chunk placement Kevin Laatz
2019-06-24 15:29   ` Björn Töpel
2019-06-20  9:09 ` [PATCH 07/11] libbpf: add flags to umem config Kevin Laatz
2019-06-24 15:30   ` Björn Töpel
2019-06-20  9:09 ` [PATCH 08/11] samples/bpf: add unaligned chunks mode support to xdpsock Kevin Laatz
2019-06-24 15:31   ` Björn Töpel
2019-06-20  9:09 ` [PATCH 09/11] samples/bpf: add buffer recycling for unaligned chunks " Kevin Laatz
2019-06-24 15:35   ` Björn Töpel
2019-06-20  9:09 ` [PATCH 10/11] samples/bpf: use hugepages in xdpsock app Kevin Laatz
2019-06-24 15:36   ` Björn Töpel
2019-06-20  9:09 ` [PATCH 11/11] doc/af_xdp: include unaligned chunk case Kevin Laatz
2019-06-24 15:34   ` Björn Töpel
2019-07-16  3:06 ` [PATCH v2 00/10] XDP unaligned chunk placement support Kevin Laatz
2019-07-16  3:06   ` [PATCH v2 01/10] i40e: simplify Rx buffer recycle Kevin Laatz
2019-07-19 17:19     ` [Intel-wired-lan] " Bowers, AndrewX
2019-07-16  3:06   ` [PATCH v2 02/10] ixgbe: " Kevin Laatz
2019-07-19 17:20     ` [Intel-wired-lan] " Bowers, AndrewX
2019-07-16  3:06   ` [PATCH v2 03/10] xsk: add support to allow unaligned chunk placement Kevin Laatz
2019-07-19 17:21     ` [Intel-wired-lan] " Bowers, AndrewX
2019-07-16  3:06   ` [PATCH v2 04/10] i40e: modify driver for handling offsets Kevin Laatz
2019-07-19 17:22     ` [Intel-wired-lan] " Bowers, AndrewX
2019-07-16  3:06   ` [PATCH v2 05/10] ixgbe: " Kevin Laatz
2019-07-19 17:22     ` [Intel-wired-lan] " Bowers, AndrewX
2019-07-16  3:06   ` [PATCH v2 06/10] libbpf: add flags to umem config Kevin Laatz
2019-07-16  3:06   ` [PATCH v2 07/10] samples/bpf: add unaligned chunks mode support to xdpsock Kevin Laatz
2019-07-16  3:06   ` [PATCH v2 08/10] samples/bpf: add buffer recycling for unaligned chunks " Kevin Laatz
2019-07-16  3:06   ` [PATCH v2 09/10] samples/bpf: use hugepages in xdpsock app Kevin Laatz
2019-07-16  3:06   ` [PATCH v2 10/10] doc/af_xdp: include unaligned chunk case Kevin Laatz
2019-07-23 21:08   ` [PATCH v2 00/10] XDP unaligned chunk placement support Alexei Starovoitov
2019-07-24 13:25     ` [Intel-wired-lan] " Magnus Karlsson
2019-07-25 15:43       ` Jonathan Lemon
2019-07-24  5:10   ` [PATCH bpf-next v3 00/11] " Kevin Laatz
2019-07-24  5:10     ` [PATCH bpf-next v3 01/11] i40e: simplify Rx buffer recycle Kevin Laatz
2019-07-24  5:10     ` [PATCH bpf-next v3 02/11] ixgbe: " Kevin Laatz
2019-07-24  5:10     ` [PATCH bpf-next v3 03/11] xsk: add support to allow unaligned chunk placement Kevin Laatz
2019-07-25  2:22       ` Jakub Kicinski
2019-07-25 17:01         ` Laatz, Kevin
2019-07-25  9:27       ` Maxim Mikityanskiy
2019-07-25 17:00         ` Laatz, Kevin
2019-07-25 10:08       ` Maxim Mikityanskiy
2019-07-25 15:39       ` Jonathan Lemon
2019-07-24  5:10     ` [PATCH bpf-next v3 04/11] i40e: modify driver for handling offsets Kevin Laatz
2019-07-24  5:10     ` [PATCH bpf-next v3 05/11] ixgbe: " Kevin Laatz
2019-07-24  5:10     ` [PATCH bpf-next v3 06/11] mlx5e: " Kevin Laatz
2019-07-25 10:15       ` Maxim Mikityanskiy
2019-07-25 17:00         ` Laatz, Kevin
2019-07-24  5:10     ` [PATCH bpf-next v3 07/11] libbpf: add flags to umem config Kevin Laatz
2019-07-24  5:10     ` [PATCH bpf-next v3 08/11] samples/bpf: add unaligned chunks mode support to xdpsock Kevin Laatz
2019-07-25  9:43       ` Maxim Mikityanskiy
2019-07-25 17:00         ` Laatz, Kevin
2019-07-24  5:10     ` [PATCH bpf-next v3 09/11] samples/bpf: add buffer recycling for unaligned chunks " Kevin Laatz
2019-07-24  5:10     ` [PATCH bpf-next v3 10/11] samples/bpf: use hugepages in xdpsock app Kevin Laatz
2019-07-24  5:10     ` [PATCH bpf-next v3 11/11] doc/af_xdp: include unaligned chunk case Kevin Laatz
2019-07-25 15:39     ` [PATCH bpf-next v3 00/11] XDP unaligned chunk placement support Jonathan Lemon
2019-07-25 15:56       ` Richardson, Bruce [this message]
2019-07-25 17:30         ` Jonathan Lemon
2019-07-26  8:41           ` Bruce Richardson
2019-07-30  8:53     ` [PATCH bpf-next v4 " Kevin Laatz
2019-07-30  8:53       ` [PATCH bpf-next v4 01/11] i40e: simplify Rx buffer recycle Kevin Laatz
2019-07-30  8:53       ` [PATCH bpf-next v4 02/11] ixgbe: " Kevin Laatz
2019-07-30  8:53       ` [PATCH bpf-next v4 03/11] libbpf: add flags to umem config Kevin Laatz
2019-07-31 12:45         ` [Intel-wired-lan] " Björn Töpel
2019-07-31 14:25         ` Björn Töpel
2019-08-01  6:59           ` Andrii Nakryiko
2019-08-01  7:34             ` Björn Töpel
2019-08-02  7:19               ` Andrii Nakryiko
2019-08-02  7:26                 ` Björn Töpel
2019-07-30  8:53       ` [PATCH bpf-next v4 04/11] xsk: add support to allow unaligned chunk placement Kevin Laatz
2019-07-31 18:11         ` Jonathan Lemon
2019-07-30  8:53       ` [PATCH bpf-next v4 05/11] i40e: modify driver for handling offsets Kevin Laatz
2019-07-30  8:53       ` [PATCH bpf-next v4 06/11] ixgbe: " Kevin Laatz
2019-07-30  8:53       ` [PATCH bpf-next v4 07/11] mlx5e: " Kevin Laatz
2019-07-31 18:10         ` Jonathan Lemon
2019-08-01 10:05         ` Maxim Mikityanskiy
2019-08-19 14:36           ` Maxim Mikityanskiy
2019-08-19 14:47             ` Laatz, Kevin
2019-07-30  8:53       ` [PATCH bpf-next v4 08/11] samples/bpf: add unaligned chunks mode support to xdpsock Kevin Laatz
2019-07-30  8:53       ` [PATCH bpf-next v4 09/11] samples/bpf: add buffer recycling for unaligned chunks " Kevin Laatz
2019-07-31 18:26         ` Jonathan Lemon
2019-07-30  8:53       ` [PATCH bpf-next v4 10/11] samples/bpf: use hugepages in xdpsock app Kevin Laatz
2019-07-30  8:54       ` [PATCH bpf-next v4 11/11] doc/af_xdp: include unaligned chunk case Kevin Laatz
2019-08-22  1:44       ` [PATCH bpf-next v5 00/11] XDP unaligned chunk placement support Kevin Laatz
2019-08-22  1:44         ` [PATCH bpf-next v5 01/11] i40e: simplify Rx buffer recycle Kevin Laatz
2019-08-22  1:44         ` [PATCH bpf-next v5 02/11] ixgbe: " Kevin Laatz
2019-08-22  1:44         ` [PATCH bpf-next v5 03/11] xsk: add support to allow unaligned chunk placement Kevin Laatz
2019-08-22 18:43           ` Jonathan Lemon
2019-08-23 13:35             ` Laatz, Kevin
2019-08-27  7:36           ` Maxim Mikityanskiy
2019-08-22  1:44         ` [PATCH bpf-next v5 04/11] i40e: modify driver for handling offsets Kevin Laatz
2019-08-22  1:44         ` [PATCH bpf-next v5 05/11] ixgbe: " Kevin Laatz
2019-08-22  1:44         ` [PATCH bpf-next v5 06/11] mlx5e: " Kevin Laatz
2019-08-22  1:44         ` [PATCH bpf-next v5 07/11] libbpf: add flags to umem config Kevin Laatz
2019-08-22  1:44         ` [PATCH bpf-next v5 08/11] samples/bpf: add unaligned chunks mode support to xdpsock Kevin Laatz
2019-08-22  1:44         ` [PATCH bpf-next v5 09/11] samples/bpf: add buffer recycling for unaligned chunks " Kevin Laatz
2019-08-22  1:44         ` [PATCH bpf-next v5 10/11] samples/bpf: use hugepages in xdpsock app Kevin Laatz
2019-08-22  1:44         ` [PATCH bpf-next v5 11/11] doc/af_xdp: include unaligned chunk case Kevin Laatz
2019-08-27  2:25         ` [PATCH bpf-next v6 00/12] XDP unaligned chunk placement support Kevin Laatz
2019-08-27  2:25           ` [PATCH bpf-next v6 01/12] i40e: simplify Rx buffer recycle Kevin Laatz
2019-08-30 15:37             ` Jonathan Lemon
2019-08-27  2:25           ` [PATCH bpf-next v6 02/12] ixgbe: " Kevin Laatz
2019-08-30 15:39             ` Jonathan Lemon
2019-08-27  2:25           ` [PATCH bpf-next v6 03/12] xsk: add support to allow unaligned chunk placement Kevin Laatz
2019-08-30 15:41             ` Jonathan Lemon
2019-08-27  2:25           ` [PATCH bpf-next v6 04/12] i40e: modify driver for handling offsets Kevin Laatz
2019-08-30 15:42             ` Jonathan Lemon
2019-08-27  2:25           ` [PATCH bpf-next v6 05/12] ixgbe: " Kevin Laatz
2019-08-30 15:42             ` Jonathan Lemon
2019-08-27  2:25           ` [PATCH bpf-next v6 06/12] mlx5e: " Kevin Laatz
2019-08-30 15:43             ` Jonathan Lemon
2019-08-27  2:25           ` [PATCH bpf-next v6 07/12] net/mlx5e: Allow XSK frames smaller than a page Kevin Laatz
2019-08-30 15:45             ` Jonathan Lemon
2019-08-27  2:25           ` [PATCH bpf-next v6 08/12] libbpf: add flags to umem config Kevin Laatz
2019-08-30 15:46             ` Jonathan Lemon
2019-08-27  2:25           ` [PATCH bpf-next v6 09/12] samples/bpf: add unaligned chunks mode support to xdpsock Kevin Laatz
2019-08-30 15:47             ` Jonathan Lemon
2019-08-27  2:25           ` [PATCH bpf-next v6 10/12] samples/bpf: add buffer recycling for unaligned chunks " Kevin Laatz
2019-08-30 15:49             ` Jonathan Lemon
2019-08-27  2:25           ` [PATCH bpf-next v6 11/12] samples/bpf: use hugepages in xdpsock app Kevin Laatz
2019-08-30 15:51             ` Jonathan Lemon
2019-08-27  2:25           ` [PATCH bpf-next v6 12/12] doc/af_xdp: include unaligned chunk case Kevin Laatz
2019-08-30 15:51             ` Jonathan Lemon
2019-08-30 15:52           ` [PATCH bpf-next v6 00/12] XDP unaligned chunk placement support Jonathan Lemon
2019-08-30 23:29           ` Daniel Borkmann

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=59AF69C657FD0841A61C55336867B5B07EDB5C3F@IRSMSX103.ger.corp.intel.com \
    --to=bruce.richardson@intel.com \
    --cc=ast@kernel.org \
    --cc=bjorn.topel@intel.com \
    --cc=bpf@vger.kernel.org \
    --cc=ciara.loftus@intel.com \
    --cc=daniel@iogearbox.net \
    --cc=intel-wired-lan@lists.osuosl.org \
    --cc=jakub.kicinski@netronome.com \
    --cc=jonathan.lemon@gmail.com \
    --cc=kevin.laatz@intel.com \
    --cc=magnus.karlsson@intel.com \
    --cc=maximmi@mellanox.com \
    --cc=netdev@vger.kernel.org \
    --cc=saeedm@mellanox.com \
    --cc=stephen@networkplumber.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).