From: Mina Almasry <almasrymina@google.com>
To: netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-doc@vger.kernel.org, linux-alpha@vger.kernel.org,
linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org,
sparclinux@vger.kernel.org, linux-trace-kernel@vger.kernel.org,
linux-arch@vger.kernel.org, bpf@vger.kernel.org,
linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org,
dri-devel@lists.freedesktop.org
Cc: "Mina Almasry" <almasrymina@google.com>,
"David S. Miller" <davem@davemloft.net>,
"Eric Dumazet" <edumazet@google.com>,
"Jakub Kicinski" <kuba@kernel.org>,
"Paolo Abeni" <pabeni@redhat.com>,
"Jonathan Corbet" <corbet@lwn.net>,
"Richard Henderson" <richard.henderson@linaro.org>,
"Ivan Kokshaysky" <ink@jurassic.park.msu.ru>,
"Matt Turner" <mattst88@gmail.com>,
"Thomas Bogendoerfer" <tsbogend@alpha.franken.de>,
"James E.J. Bottomley" <James.Bottomley@HansenPartnership.com>,
"Helge Deller" <deller@gmx.de>,
"Andreas Larsson" <andreas@gaisler.com>,
"Jesper Dangaard Brouer" <hawk@kernel.org>,
"Ilias Apalodimas" <ilias.apalodimas@linaro.org>,
"Steven Rostedt" <rostedt@goodmis.org>,
"Masami Hiramatsu" <mhiramat@kernel.org>,
"Mathieu Desnoyers" <mathieu.desnoyers@efficios.com>,
"Arnd Bergmann" <arnd@arndb.de>,
"Alexei Starovoitov" <ast@kernel.org>,
"Daniel Borkmann" <daniel@iogearbox.net>,
"Andrii Nakryiko" <andrii@kernel.org>,
"Martin KaFai Lau" <martin.lau@linux.dev>,
"Eduard Zingerman" <eddyz87@gmail.com>,
"Song Liu" <song@kernel.org>,
"Yonghong Song" <yonghong.song@linux.dev>,
"John Fastabend" <john.fastabend@gmail.com>,
"KP Singh" <kpsingh@kernel.org>,
"Stanislav Fomichev" <sdf@google.com>,
"Hao Luo" <haoluo@google.com>, "Jiri Olsa" <jolsa@kernel.org>,
"David Ahern" <dsahern@kernel.org>,
"Willem de Bruijn" <willemdebruijn.kernel@gmail.com>,
"Shuah Khan" <shuah@kernel.org>,
"Sumit Semwal" <sumit.semwal@linaro.org>,
"Christian König" <christian.koenig@amd.com>,
"Pavel Begunkov" <asml.silence@gmail.com>,
"David Wei" <dw@davidwei.uk>, "Jason Gunthorpe" <jgg@ziepe.ca>,
"Yunsheng Lin" <linyunsheng@huawei.com>,
"Shailend Chand" <shailend@google.com>,
"Harshitha Ramamurthy" <hramamurthy@google.com>,
"Shakeel Butt" <shakeelb@google.com>,
"Jeroen de Borst" <jeroendb@google.com>,
"Praveen Kaligineedi" <pkaligineedi@google.com>,
"Willem de Bruijn" <willemb@google.com>,
"Kaiyuan Zhang" <kaiyuanz@google.com>
Subject: [RFC PATCH net-next v6 09/15] memory-provider: dmabuf devmem memory provider
Date: Mon, 4 Mar 2024 18:01:44 -0800 [thread overview]
Message-ID: <20240305020153.2787423-10-almasrymina@google.com> (raw)
In-Reply-To: <20240305020153.2787423-1-almasrymina@google.com>
Implement a memory provider that allocates dmabuf devmem in the form of
net_iov.
The provider receives a reference to the struct netdev_dmabuf_binding
via the pool->mp_priv pointer. The driver needs to set this pointer for
the provider in the net_iov.
The provider obtains a reference on the netdev_dmabuf_binding which
guarantees the binding and the underlying mapping remains alive until
the provider is destroyed.
Usage of PP_FLAG_DMA_MAP is required for this memory provide such that
the page_pool can provide the driver with the dma-addrs of the devmem.
Support for PP_FLAG_DMA_SYNC_DEV is omitted for simplicity & p.order !=
0.
Signed-off-by: Willem de Bruijn <willemb@google.com>
Signed-off-by: Kaiyuan Zhang <kaiyuanz@google.com>
Signed-off-by: Mina Almasry <almasrymina@google.com>
---
v6:
- refactor new memory provider functions into net/core/devmem.c (Pavel)
v2:
- Disable devmem for p.order != 0
v1:
- static_branch check in page_is_page_pool_iov() (Willem & Paolo).
- PP_DEVMEM -> PP_IOV (David).
- Require PP_FLAG_DMA_MAP (Jakub).
---
include/net/netmem.h | 14 ++++++
include/net/page_pool/helpers.h | 21 +++++++++
include/net/page_pool/types.h | 2 +
net/core/devmem.c | 82 +++++++++++++++++++++++++++++++++
net/core/page_pool.c | 35 ++++++--------
5 files changed, 132 insertions(+), 22 deletions(-)
diff --git a/include/net/netmem.h b/include/net/netmem.h
index 8699788d587d..a2de9411025d 100644
--- a/include/net/netmem.h
+++ b/include/net/netmem.h
@@ -127,6 +127,20 @@ static inline struct page *netmem_to_page(netmem_ref netmem)
return (__force struct page *)netmem;
}
+static inline struct net_iov *netmem_to_net_iov(netmem_ref netmem)
+{
+ if (netmem_is_net_iov(netmem))
+ return (struct net_iov *)((__force unsigned long)netmem & ~NET_IOV);
+
+ DEBUG_NET_WARN_ON_ONCE(true);
+ return NULL;
+}
+
+static inline netmem_ref net_iov_to_netmem(struct net_iov *niov)
+{
+ return (__force netmem_ref)((unsigned long)niov | NET_IOV);
+}
+
static inline netmem_ref page_to_netmem(struct page *page)
{
return (__force netmem_ref)page;
diff --git a/include/net/page_pool/helpers.h b/include/net/page_pool/helpers.h
index c6a55eddefae..00682b4de6e8 100644
--- a/include/net/page_pool/helpers.h
+++ b/include/net/page_pool/helpers.h
@@ -453,4 +453,25 @@ static inline void page_pool_nid_changed(struct page_pool *pool, int new_nid)
page_pool_update_nid(pool, new_nid);
}
+static inline void page_pool_set_pp_info(struct page_pool *pool, netmem_ref netmem)
+{
+ netmem_set_pp(netmem, pool);
+ netmem_or_pp_magic(netmem, PP_SIGNATURE);
+
+ /* Ensuring all pages have been split into one fragment initially:
+ * page_pool_set_pp_info() is only called once for every page when it
+ * is allocated from the page allocator and page_pool_fragment_page()
+ * is dirtying the same cache line as the page->pp_magic above, so
+ * the overhead is negligible.
+ */
+ page_pool_fragment_netmem(netmem, 1);
+ if (pool->has_init_callback)
+ pool->slow.init_callback(netmem, pool->slow.init_arg);
+}
+
+static inline void page_pool_clear_pp_info(netmem_ref netmem)
+{
+ netmem_clear_pp_magic(netmem);
+ netmem_set_pp(netmem, NULL);
+}
#endif /* _NET_PAGE_POOL_HELPERS_H */
diff --git a/include/net/page_pool/types.h b/include/net/page_pool/types.h
index e29e77f7934e..096cd2455b2c 100644
--- a/include/net/page_pool/types.h
+++ b/include/net/page_pool/types.h
@@ -136,6 +136,8 @@ struct memory_provider_ops {
bool (*release_page)(struct page_pool *pool, netmem_ref netmem);
};
+extern const struct memory_provider_ops dmabuf_devmem_ops;
+
struct page_pool {
struct page_pool_params_fast p;
diff --git a/net/core/devmem.c b/net/core/devmem.c
index 57d3a1f223ef..3ced312f7860 100644
--- a/net/core/devmem.c
+++ b/net/core/devmem.c
@@ -329,3 +329,85 @@ int netdev_bind_dmabuf(struct net_device *dev, unsigned int dmabuf_fd,
return err;
}
#endif
+
+/*** "Dmabuf devmem memory provider" ***/
+
+static int mp_dmabuf_devmem_init(struct page_pool *pool)
+{
+ struct netdev_dmabuf_binding *binding = pool->mp_priv;
+
+ if (!binding)
+ return -EINVAL;
+
+ if (!(pool->p.flags & PP_FLAG_DMA_MAP))
+ return -EOPNOTSUPP;
+
+ if (pool->p.flags & PP_FLAG_DMA_SYNC_DEV)
+ return -EOPNOTSUPP;
+
+ if (pool->p.order != 0)
+ return -E2BIG;
+
+ netdev_dmabuf_binding_get(binding);
+ return 0;
+}
+
+static netmem_ref mp_dmabuf_devmem_alloc_pages(struct page_pool *pool,
+ gfp_t gfp)
+{
+ struct netdev_dmabuf_binding *binding = pool->mp_priv;
+ netmem_ref netmem;
+ struct net_iov *niov;
+ dma_addr_t dma_addr;
+
+ niov = netdev_alloc_dmabuf(binding);
+ if (!niov)
+ return 0;
+
+ dma_addr = net_iov_dma_addr(niov);
+
+ netmem = net_iov_to_netmem(niov);
+
+ page_pool_set_pp_info(pool, netmem);
+
+ if (page_pool_set_dma_addr_netmem(netmem, dma_addr))
+ goto err_free;
+
+ pool->pages_state_hold_cnt++;
+ trace_page_pool_state_hold(pool, netmem, pool->pages_state_hold_cnt);
+ return netmem;
+
+err_free:
+ netdev_free_dmabuf(niov);
+ return 0;
+}
+
+static void mp_dmabuf_devmem_destroy(struct page_pool *pool)
+{
+ struct netdev_dmabuf_binding *binding = pool->mp_priv;
+
+ netdev_dmabuf_binding_put(binding);
+}
+
+static bool mp_dmabuf_devmem_release_page(struct page_pool *pool,
+ netmem_ref netmem)
+{
+ WARN_ON_ONCE(!netmem_is_net_iov(netmem));
+ WARN_ON_ONCE(atomic_long_read(netmem_get_pp_ref_count_ref(netmem))
+ != 1);
+
+ page_pool_clear_pp_info(netmem);
+
+ netdev_free_dmabuf(netmem_to_net_iov(netmem));
+
+ /* We don't want the page pool put_page()ing our net_iovs. */
+ return false;
+}
+
+const struct memory_provider_ops dmabuf_devmem_ops = {
+ .init = mp_dmabuf_devmem_init,
+ .destroy = mp_dmabuf_devmem_destroy,
+ .alloc_pages = mp_dmabuf_devmem_alloc_pages,
+ .release_page = mp_dmabuf_devmem_release_page,
+};
+EXPORT_SYMBOL(dmabuf_devmem_ops);
diff --git a/net/core/page_pool.c b/net/core/page_pool.c
index 22e3d439da18..2cee7d9f6ca6 100644
--- a/net/core/page_pool.c
+++ b/net/core/page_pool.c
@@ -12,6 +12,7 @@
#include <net/page_pool/helpers.h>
#include <net/xdp.h>
+#include <net/netdev_rx_queue.h>
#include <linux/dma-direction.h>
#include <linux/dma-mapping.h>
@@ -20,12 +21,15 @@
#include <linux/poison.h>
#include <linux/ethtool.h>
#include <linux/netdevice.h>
+#include <linux/genalloc.h>
+#include <net/devmem.h>
#include <trace/events/page_pool.h>
#include "page_pool_priv.h"
DEFINE_STATIC_KEY_FALSE(page_pool_mem_providers);
+EXPORT_SYMBOL(page_pool_mem_providers);
#define DEFER_TIME (msecs_to_jiffies(1000))
#define DEFER_WARN_INTERVAL (60 * HZ)
@@ -178,6 +182,7 @@ static int page_pool_init(struct page_pool *pool,
const struct page_pool_params *params,
int cpuid)
{
+ struct netdev_dmabuf_binding *binding = NULL;
unsigned int ring_qsize = 1024; /* Default */
int err;
@@ -251,6 +256,14 @@ static int page_pool_init(struct page_pool *pool,
/* Driver calling page_pool_create() also call page_pool_destroy() */
refcount_set(&pool->user_cnt, 1);
+ if (pool->p.queue)
+ binding = READ_ONCE(pool->p.queue->binding);
+
+ if (binding) {
+ pool->mp_ops = &dmabuf_devmem_ops;
+ pool->mp_priv = binding;
+ }
+
if (pool->mp_ops) {
err = pool->mp_ops->init(pool);
if (err) {
@@ -444,28 +457,6 @@ static bool page_pool_dma_map(struct page_pool *pool, netmem_ref netmem)
return false;
}
-static void page_pool_set_pp_info(struct page_pool *pool, netmem_ref netmem)
-{
- netmem_set_pp(netmem, pool);
- netmem_or_pp_magic(netmem, PP_SIGNATURE);
-
- /* Ensuring all pages have been split into one fragment initially:
- * page_pool_set_pp_info() is only called once for every page when it
- * is allocated from the page allocator and page_pool_fragment_page()
- * is dirtying the same cache line as the page->pp_magic above, so
- * the overhead is negligible.
- */
- page_pool_fragment_netmem(netmem, 1);
- if (pool->has_init_callback)
- pool->slow.init_callback(netmem, pool->slow.init_arg);
-}
-
-static void page_pool_clear_pp_info(netmem_ref netmem)
-{
- netmem_clear_pp_magic(netmem);
- netmem_set_pp(netmem, NULL);
-}
-
static struct page *__page_pool_alloc_page_order(struct page_pool *pool,
gfp_t gfp)
{
--
2.44.0.rc1.240.g4c46232300-goog
next prev parent reply other threads:[~2024-03-05 2:02 UTC|newest]
Thread overview: 73+ messages / expand[flat|nested] mbox.gz Atom feed top
2024-03-05 2:01 [RFC PATCH net-next v6 00/15] Device Memory TCP Mina Almasry
2024-03-05 2:01 ` [RFC PATCH net-next v6 01/15] queue_api: define queue api Mina Almasry
2024-03-08 1:30 ` Jakub Kicinski
2024-03-08 2:08 ` Mina Almasry
2024-03-08 3:36 ` Jakub Kicinski
2024-03-08 23:47 ` David Wei
2024-03-09 0:27 ` Mina Almasry
2024-03-11 1:12 ` David Ahern
2024-03-05 2:01 ` [RFC PATCH net-next v6 02/15] net: page_pool: create hooks for custom page providers Mina Almasry
2024-03-05 21:54 ` David Wei
2024-03-05 22:36 ` Mina Almasry
2024-03-06 14:29 ` Pavel Begunkov
2024-03-06 17:04 ` Mina Almasry
2024-03-06 19:12 ` Pavel Begunkov
2024-03-06 21:59 ` Mina Almasry
2024-03-07 14:25 ` Pavel Begunkov
2024-03-08 4:57 ` David Wei
2024-03-08 19:53 ` Mina Almasry
2024-03-18 2:02 ` Christoph Hellwig
2024-03-18 2:49 ` David Wei
2024-03-18 23:22 ` Christoph Hellwig
2024-03-22 17:40 ` Mina Almasry
2024-03-22 23:19 ` Jakub Kicinski
2024-03-24 23:35 ` Christoph Hellwig
2024-03-24 23:35 ` Christoph Hellwig
2024-03-22 17:54 ` Mina Almasry
2024-03-24 23:37 ` Christoph Hellwig
2024-03-26 20:19 ` Mina Almasry
2024-03-28 7:31 ` Christoph Hellwig
2024-04-01 19:22 ` Mina Almasry
2024-04-08 15:34 ` Cong Wang
2024-03-05 2:01 ` [RFC PATCH net-next v6 03/15] net: page_pool: factor out page_pool recycle check Mina Almasry
2024-03-05 12:55 ` Yunsheng Lin
2024-03-05 2:01 ` [RFC PATCH net-next v6 04/15] net: netdev netlink api to bind dma-buf to a net device Mina Almasry
2024-03-05 2:01 ` [RFC PATCH net-next v6 05/15] netdev: support binding dma-buf to netdevice Mina Almasry
2024-03-05 9:04 ` Arnd Bergmann
2024-03-05 20:00 ` Mina Almasry
2024-03-05 21:42 ` Arnd Bergmann
2024-03-05 12:55 ` Yunsheng Lin
2024-03-05 21:17 ` Mina Almasry
2024-03-06 12:38 ` Yunsheng Lin
2024-03-06 22:10 ` Mina Almasry
2024-03-07 12:15 ` Yunsheng Lin
2024-03-08 3:58 ` Jakub Kicinski
2024-03-05 2:01 ` [RFC PATCH net-next v6 06/15] netdev: netdevice devmem allocator Mina Almasry
2024-03-05 2:01 ` [RFC PATCH net-next v6 07/15] page_pool: convert to use netmem Mina Almasry
2024-03-05 21:30 ` Mina Almasry
2024-03-05 2:01 ` [RFC PATCH net-next v6 08/15] page_pool: devmem support Mina Almasry
2024-03-05 21:42 ` Mina Almasry
2024-03-05 2:01 ` Mina Almasry [this message]
2024-03-06 2:28 ` [RFC PATCH net-next v6 09/15] memory-provider: dmabuf devmem memory provider David Wei
2024-03-06 2:42 ` Mina Almasry
2024-03-06 2:46 ` David Wei
2024-03-06 2:54 ` Mina Almasry
2024-03-06 14:58 ` Pavel Begunkov
2024-03-06 16:51 ` Mina Almasry
2024-03-05 2:01 ` [RFC PATCH net-next v6 10/15] net: support non paged skb frags Mina Almasry
2024-03-05 2:01 ` [RFC PATCH net-next v6 11/15] net: add support for skbs with unreadable frags Mina Almasry
2024-03-05 2:01 ` [RFC PATCH net-next v6 12/15] tcp: RX path for devmem TCP Mina Almasry
2024-03-05 8:41 ` Arnd Bergmann
2024-03-05 19:22 ` Mina Almasry
2024-03-05 19:39 ` Arnd Bergmann
2024-03-05 2:01 ` [RFC PATCH net-next v6 13/15] net: add SO_DEVMEM_DONTNEED setsockopt to release RX frags Mina Almasry
2024-03-05 2:01 ` [RFC PATCH net-next v6 14/15] net: add devmem TCP documentation Mina Almasry
2024-03-08 1:52 ` Jakub Kicinski
2024-03-05 2:01 ` [RFC PATCH net-next v6 15/15] selftests: add ncdevmem, netcat for devmem TCP Mina Almasry
2024-03-05 7:16 ` [RFC PATCH net-next v6 07/15] page_pool: convert to use netmem David Howells
2024-03-05 12:54 ` [RFC PATCH net-next v6 00/15] Device Memory TCP Yunsheng Lin
2024-03-05 19:38 ` Mina Almasry
2024-03-06 12:37 ` Yunsheng Lin
2024-03-26 0:28 ` Mina Almasry
2024-03-26 12:47 ` Yunsheng Lin
2024-03-26 20:14 ` Mina Almasry
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20240305020153.2787423-10-almasrymina@google.com \
--to=almasrymina@google.com \
--cc=James.Bottomley@HansenPartnership.com \
--cc=andreas@gaisler.com \
--cc=andrii@kernel.org \
--cc=arnd@arndb.de \
--cc=asml.silence@gmail.com \
--cc=ast@kernel.org \
--cc=bpf@vger.kernel.org \
--cc=christian.koenig@amd.com \
--cc=corbet@lwn.net \
--cc=daniel@iogearbox.net \
--cc=davem@davemloft.net \
--cc=deller@gmx.de \
--cc=dri-devel@lists.freedesktop.org \
--cc=dsahern@kernel.org \
--cc=dw@davidwei.uk \
--cc=eddyz87@gmail.com \
--cc=edumazet@google.com \
--cc=haoluo@google.com \
--cc=hawk@kernel.org \
--cc=hramamurthy@google.com \
--cc=ilias.apalodimas@linaro.org \
--cc=ink@jurassic.park.msu.ru \
--cc=jeroendb@google.com \
--cc=jgg@ziepe.ca \
--cc=john.fastabend@gmail.com \
--cc=jolsa@kernel.org \
--cc=kaiyuanz@google.com \
--cc=kpsingh@kernel.org \
--cc=kuba@kernel.org \
--cc=linux-alpha@vger.kernel.org \
--cc=linux-arch@vger.kernel.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-media@vger.kernel.org \
--cc=linux-mips@vger.kernel.org \
--cc=linux-parisc@vger.kernel.org \
--cc=linux-trace-kernel@vger.kernel.org \
--cc=linyunsheng@huawei.com \
--cc=martin.lau@linux.dev \
--cc=mathieu.desnoyers@efficios.com \
--cc=mattst88@gmail.com \
--cc=mhiramat@kernel.org \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=pkaligineedi@google.com \
--cc=richard.henderson@linaro.org \
--cc=rostedt@goodmis.org \
--cc=sdf@google.com \
--cc=shailend@google.com \
--cc=shakeelb@google.com \
--cc=shuah@kernel.org \
--cc=song@kernel.org \
--cc=sparclinux@vger.kernel.org \
--cc=sumit.semwal@linaro.org \
--cc=tsbogend@alpha.franken.de \
--cc=willemb@google.com \
--cc=willemdebruijn.kernel@gmail.com \
--cc=yonghong.song@linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).