From: Jakub Kicinski <jakub.kicinski@netronome.com> To: "Laatz, Kevin" <kevin.laatz@intel.com> Cc: Jonathan Lemon <jonathan.lemon@gmail.com>, netdev@vger.kernel.org, ast@kernel.org, daniel@iogearbox.net, bjorn.topel@intel.com, magnus.karlsson@intel.com, bpf@vger.kernel.org, intel-wired-lan@lists.osuosl.org, bruce.richardson@intel.com, ciara.loftus@intel.com Subject: Re: [PATCH 00/11] XDP unaligned chunk placement support Date: Fri, 28 Jun 2019 13:25:16 -0700 [thread overview] Message-ID: <20190628132516.723ef517@cakuba.netronome.com> (raw) In-Reply-To: <f0ca817a-02b4-df22-d01b-7bc07171a4dc@intel.com> On Fri, 28 Jun 2019 17:19:09 +0100, Laatz, Kevin wrote: > On 27/06/2019 22:25, Jakub Kicinski wrote: > > On Thu, 27 Jun 2019 12:14:50 +0100, Laatz, Kevin wrote: > >> On the application side (xdpsock), we don't have to worry about the user > >> defined headroom, since it is 0, so we only need to account for the > >> XDP_PACKET_HEADROOM when computing the original address (in the default > >> scenario). > > That assumes specific layout for the data inside the buffer. Some NICs > > will prepend information like timestamp to the packet, meaning the > > packet would start at offset XDP_PACKET_HEADROOM + metadata len.. > > Yes, if NICs prepend extra data to the packet that would be a problem for > using this feature in isolation. However, if we also add in support for > in-order RX and TX rings, that would no longer be an issue. Can you shed more light on in-order rings? Do you mean that RX frames come in order buffers were placed in the fill queue? That wouldn't make practical sense, no? Even if the application does no reordering there is also XDP_DROP and XDP_TX. Please explain :) > However, even for NICs which do prepend data, this patchset should > not break anything that is currently working. My understanding from the beginnings of AF_XDP was that we were searching for a format flexible enough to support most if not all NICs. Creating an ABI which will preclude vendors from supporting DPDK via AF_XDP would seriously undermine the neutrality aspect. > > I think that's very limiting. What is the challenge in providing > > aligned addresses, exactly? > The challenges are two-fold: > 1) it prevents using arbitrary buffer sizes, which will be an issue > supporting e.g. jumbo frames in future. Presumably support for jumbos would require a multi-buffer setup, and therefore extensions to the ring format. Should we perhaps look into implementing unaligned chunks by extending ring format as well? > 2) higher level user-space frameworks which may want to use AF_XDP, such > as DPDK, do not currently support having buffers with 'fixed' alignment. > The reason that DPDK uses arbitrary placement is that: > - it would stop things working on certain NICs which need the > actual writable space specified in units of 1k - therefore we need 2k + > metadata space. > - we place padding between buffers to avoid constantly hitting > the same memory channels when accessing memory. > - it allows the application to choose the actual buffer size it > wants to use. > We make use of the above to allow us to speed up processing > significantly and also reduce the packet buffer memory size. > > Not having arbitrary buffer alignment also means an AF_XDP driver > for DPDK cannot be a drop-in replacement for existing drivers in those > frameworks. Even with a new capability to allow an arbitrary buffer > alignment, existing apps will need to be modified to use that new > capability.
WARNING: multiple messages have this Message-ID (diff)
From: Jakub Kicinski <jakub.kicinski@netronome.com> To: intel-wired-lan@osuosl.org Subject: [Intel-wired-lan] [PATCH 00/11] XDP unaligned chunk placement support Date: Fri, 28 Jun 2019 13:25:16 -0700 [thread overview] Message-ID: <20190628132516.723ef517@cakuba.netronome.com> (raw) In-Reply-To: <f0ca817a-02b4-df22-d01b-7bc07171a4dc@intel.com> On Fri, 28 Jun 2019 17:19:09 +0100, Laatz, Kevin wrote: > On 27/06/2019 22:25, Jakub Kicinski wrote: > > On Thu, 27 Jun 2019 12:14:50 +0100, Laatz, Kevin wrote: > >> On the application side (xdpsock), we don't have to worry about the user > >> defined headroom, since it is 0, so we only need to account for the > >> XDP_PACKET_HEADROOM when computing the original address (in the default > >> scenario). > > That assumes specific layout for the data inside the buffer. Some NICs > > will prepend information like timestamp to the packet, meaning the > > packet would start at offset XDP_PACKET_HEADROOM + metadata len.. > > Yes, if NICs prepend extra data to the packet that would be a problem for > using this feature in isolation. However, if we also add in support for > in-order RX and TX rings, that would no longer be an issue. Can you shed more light on in-order rings? Do you mean that RX frames come in order buffers were placed in the fill queue? That wouldn't make practical sense, no? Even if the application does no reordering there is also XDP_DROP and XDP_TX. Please explain :) > However, even for NICs which do prepend data, this patchset should > not break anything that is currently working. My understanding from the beginnings of AF_XDP was that we were searching for a format flexible enough to support most if not all NICs. Creating an ABI which will preclude vendors from supporting DPDK via AF_XDP would seriously undermine the neutrality aspect. > > I think that's very limiting. What is the challenge in providing > > aligned addresses, exactly? > The challenges are two-fold: > 1) it prevents using arbitrary buffer sizes, which will be an issue > supporting e.g. jumbo frames in future. Presumably support for jumbos would require a multi-buffer setup, and therefore extensions to the ring format. Should we perhaps look into implementing unaligned chunks by extending ring format as well? > 2) higher level user-space frameworks which may want to use AF_XDP, such > as DPDK, do not currently support having buffers with 'fixed' alignment. > ??? The reason that DPDK uses arbitrary placement is that: > ??? ??? - it would stop things working on certain NICs which need the > actual writable space specified in units of 1k - therefore we need 2k + > metadata space. > ??? ??? - we place padding between buffers to avoid constantly hitting > the same memory channels when accessing memory. > ??? ??? - it allows the application to choose the actual buffer size it > wants to use. > ??? We make use of the above to allow us to speed up processing > significantly and also reduce the packet buffer memory size. > > ??? Not having arbitrary buffer alignment also means an AF_XDP driver > for DPDK cannot be a drop-in replacement for existing drivers in those > frameworks. Even with a new capability to allow an arbitrary buffer > alignment, existing apps will need to be modified to use that new > capability.
next prev parent reply other threads:[~2019-06-28 20:25 UTC|newest] Thread overview: 54+ messages / expand[flat|nested] mbox.gz Atom feed top 2019-06-20 8:39 [PATCH 00/11] XDP unaligned chunk placement support Kevin Laatz 2019-06-20 8:39 ` [Intel-wired-lan] " Kevin Laatz 2019-06-20 8:39 ` [PATCH 01/11] i40e: simplify Rx buffer recycle Kevin Laatz 2019-06-20 8:39 ` [Intel-wired-lan] " Kevin Laatz 2019-06-20 8:39 ` [PATCH 02/11] ixgbe: " Kevin Laatz 2019-06-20 8:39 ` [Intel-wired-lan] " Kevin Laatz 2019-06-20 8:39 ` [PATCH 03/11] xdp: add offset param to zero_copy_allocator Kevin Laatz 2019-06-20 8:39 ` [Intel-wired-lan] " Kevin Laatz 2019-06-20 8:39 ` [PATCH 04/11] i40e: add offset to zca_free Kevin Laatz 2019-06-20 8:39 ` [Intel-wired-lan] " Kevin Laatz 2019-06-20 8:39 ` [PATCH 05/11] ixgbe: " Kevin Laatz 2019-06-20 8:39 ` [Intel-wired-lan] " Kevin Laatz 2019-06-20 8:39 ` [PATCH 06/11] xsk: add support to allow unaligned chunk placement Kevin Laatz 2019-06-20 8:39 ` [Intel-wired-lan] " Kevin Laatz 2019-06-20 8:39 ` [PATCH 07/11] libbpf: add flags to umem config Kevin Laatz 2019-06-20 8:39 ` [Intel-wired-lan] " Kevin Laatz 2019-06-20 8:39 ` [PATCH 08/11] samples/bpf: add unaligned chunks mode support to xdpsock Kevin Laatz 2019-06-20 8:39 ` [Intel-wired-lan] " Kevin Laatz 2019-06-20 8:39 ` [PATCH 09/11] samples/bpf: add buffer recycling for unaligned chunks " Kevin Laatz 2019-06-20 8:39 ` [Intel-wired-lan] " Kevin Laatz 2019-06-20 8:39 ` [PATCH 10/11] samples/bpf: use hugepages in xdpsock app Kevin Laatz 2019-06-20 8:39 ` [Intel-wired-lan] " Kevin Laatz 2019-06-20 8:39 ` [PATCH 11/11] doc/af_xdp: include unaligned chunk case Kevin Laatz 2019-06-20 8:39 ` [Intel-wired-lan] " Kevin Laatz 2019-06-24 15:38 ` [PATCH 00/11] XDP unaligned chunk placement support Björn Töpel 2019-06-24 15:38 ` [Intel-wired-lan] " =?unknown-8bit?q?Bj=C3=B6rn_T=C3=B6pel?= 2019-06-25 13:12 ` Laatz, Kevin 2019-06-25 13:12 ` [Intel-wired-lan] " Laatz, Kevin 2019-06-25 18:44 ` Jonathan Lemon 2019-06-25 18:44 ` [Intel-wired-lan] " Jonathan Lemon 2019-06-27 11:14 ` Laatz, Kevin 2019-06-27 11:14 ` [Intel-wired-lan] " Laatz, Kevin 2019-06-27 21:25 ` Jakub Kicinski 2019-06-27 21:25 ` [Intel-wired-lan] " Jakub Kicinski 2019-06-28 16:19 ` Laatz, Kevin 2019-06-28 16:19 ` [Intel-wired-lan] " Laatz, Kevin 2019-06-28 16:51 ` Björn Töpel 2019-06-28 16:51 ` [Intel-wired-lan] " =?unknown-8bit?q?Bj=C3=B6rn_T=C3=B6pel?= 2019-06-28 20:08 ` Jakub Kicinski 2019-06-28 20:08 ` [Intel-wired-lan] " Jakub Kicinski 2019-06-28 20:25 ` Jakub Kicinski [this message] 2019-06-28 20:25 ` Jakub Kicinski 2019-06-28 20:29 ` Jonathan Lemon 2019-06-28 20:29 ` [Intel-wired-lan] " Jonathan Lemon 2019-07-01 14:44 ` Laatz, Kevin 2019-07-01 21:20 ` Jakub Kicinski 2019-07-01 21:20 ` [Intel-wired-lan] " Jakub Kicinski 2019-07-02 9:27 ` Richardson, Bruce 2019-07-02 9:27 ` [Intel-wired-lan] " Richardson, Bruce 2019-07-02 16:33 ` Jonathan Lemon 2019-07-02 16:33 ` [Intel-wired-lan] " Jonathan Lemon 2019-07-01 14:58 ` Laatz, Kevin 2019-07-01 14:58 ` [Intel-wired-lan] " Laatz, Kevin 2019-06-20 9:09 Kevin Laatz
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20190628132516.723ef517@cakuba.netronome.com \ --to=jakub.kicinski@netronome.com \ --cc=ast@kernel.org \ --cc=bjorn.topel@intel.com \ --cc=bpf@vger.kernel.org \ --cc=bruce.richardson@intel.com \ --cc=ciara.loftus@intel.com \ --cc=daniel@iogearbox.net \ --cc=intel-wired-lan@lists.osuosl.org \ --cc=jonathan.lemon@gmail.com \ --cc=kevin.laatz@intel.com \ --cc=magnus.karlsson@intel.com \ --cc=netdev@vger.kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.