From: Yunsheng Lin <linyunsheng@huawei.com>
To: Mina Almasry <almasrymina@google.com>
Cc: "Shailend Chand" <shailend@google.com>,
netdev@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-doc@vger.kernel.org, linux-arch@vger.kernel.org,
linux-kselftest@vger.kernel.org, bpf@vger.kernel.org,
linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org,
"David S. Miller" <davem@davemloft.net>,
"Eric Dumazet" <edumazet@google.com>,
"Jakub Kicinski" <kuba@kernel.org>,
"Paolo Abeni" <pabeni@redhat.com>,
"Jonathan Corbet" <corbet@lwn.net>,
"Jeroen de Borst" <jeroendb@google.com>,
"Praveen Kaligineedi" <pkaligineedi@google.com>,
"Jesper Dangaard Brouer" <hawk@kernel.org>,
"Ilias Apalodimas" <ilias.apalodimas@linaro.org>,
"Arnd Bergmann" <arnd@arndb.de>,
"David Ahern" <dsahern@kernel.org>,
"Willem de Bruijn" <willemdebruijn.kernel@gmail.com>,
"Shuah Khan" <shuah@kernel.org>,
"Sumit Semwal" <sumit.semwal@linaro.org>,
"Christian König" <christian.koenig@amd.com>,
"Harshitha Ramamurthy" <hramamurthy@google.com>,
"Shakeel Butt" <shakeelb@google.com>
Subject: Re: [net-next v1 09/16] page_pool: device memory support
Date: Tue, 12 Dec 2023 19:17:21 +0800 [thread overview]
Message-ID: <2cdf173c-95e4-2141-56f7-0761705cd737@huawei.com> (raw)
In-Reply-To: <CAHS8izMdpo0D7GYzMkOtg1ueCODAVNxtwSP_qPseSYXNMhPGCw@mail.gmail.com>
On 2023/12/12 2:14, Mina Almasry wrote:
> On Mon, Dec 11, 2023 at 3:51 AM Yunsheng Lin <linyunsheng@huawei.com> wrote:
>>
>> On 2023/12/11 12:04, Mina Almasry wrote:
>>> On Sun, Dec 10, 2023 at 6:26 PM Mina Almasry <almasrymina@google.com> wrote:
>>>>
>>>> On Sun, Dec 10, 2023 at 6:04 PM Yunsheng Lin <linyunsheng@huawei.com> wrote:
>>>>>
>>>>> On 2023/12/9 0:05, Mina Almasry wrote:
>>>>>> On Fri, Dec 8, 2023 at 1:30 AM Yunsheng Lin <linyunsheng@huawei.com> wrote:
>>>>>>>
>>>>>>>
>>>>>>> As mentioned before, it seems we need to have the above checking every
>>>>>>> time we need to do some per-page handling in page_pool core, is there
>>>>>>> a plan in your mind how to remove those kind of checking in the future?
>>>>>>>
>>>>>>
>>>>>> I see 2 ways to remove the checking, both infeasible:
>>>>>>
>>>>>> 1. Allocate a wrapper struct that pulls out all the fields the page pool needs:
>>>>>>
>>>>>> struct netmem {
>>>>>> /* common fields */
>>>>>> refcount_t refcount;
>>>>>> bool is_pfmemalloc;
>>>>>> int nid;
>>>>>> ...
>>>>>> union {
>>>>>> struct dmabuf_genpool_chunk_owner *owner;
>>>>>> struct page * page;
>>>>>> };
>>>>>> };
>>>>>>
>>>>>> The page pool can then not care if the underlying memory is iov or
>>>>>> page. However this introduces significant memory bloat as this struct
>>>>>> needs to be allocated for each page or ppiov, which I imagine is not
>>>>>> acceptable for the upside of removing a few static_branch'd if
>>>>>> statements with no performance cost.
>>>>>>
>>>>>> 2. Create a unified struct for page and dmabuf memory, which the mm
>>>>>> folks have repeatedly nacked, and I imagine will repeatedly nack in
>>>>>> the future.
>>>>>>
>>>>>> So I imagine the special handling of ppiov in some form is critical
>>>>>> and the checking may not be removable.
>>>>>
>>>>> If the above is true, perhaps devmem is not really supposed to be intergated
>>>>> into page_pool.
>>>>>
>>>>> Adding a checking for every per-page handling in page_pool core is just too
>>>>> hacky to be really considerred a longterm solution.
>>>>>
>>>>
>>>> The only other option is to implement another page_pool for ppiov and
>>>> have the driver create page_pool or ppiov_pool depending on the state
>>>> of the netdev_rx_queue (or some helper in the net stack to do that for
>>>> the driver). This introduces some code duplication. The ppiov_pool &
>>>> page_pool would look similar in implementation.
>>
>> I think there is a design pattern already to deal with this kind of problem,
>> refactoring common code used by both page_pool and ppiov into a library to
>> aovid code duplication if most of them have similar implementation.
>>
>
> Code can be refactored if it's identical, not if it is similar. I
Similarity indicates an opportunity to the refactor out the common
code, like the page_frag case below:
https://patchwork.kernel.org/project/netdevbpf/cover/20231205113444.63015-1-linyunsheng@huawei.com/
But untill we do a proof of concept implemention, it is hard to tell if
it is feasiable or not.
> suspect the page_pools will be only similar, and if you're not willing
> to take devmem handling into the page pool then refactoring page_pool
> code into helpers that do devmem handling may also not be an option.
>
>>>>
>>>> But this was all discussed in detail in RFC v2 and the last response I
>>>> heard from Jesper was in favor if this approach, if I understand
>>>> correctly:
>>>>
>>>> https://lore.kernel.org/netdev/7aedc5d5-0daf-63be-21bc-3b724cc1cab9@redhat.com/
>>>>
>>>> Would love to have the maintainer weigh in here.
>>>>
>>>
>>> I should note we may be able to remove some of the checking, but maybe not all.
>>>
>>> - Checks that disable page fragging for ppiov can be removed once
>>> ppiov has frag support (in this series or follow up).
>>>
>>> - If we use page->pp_frag_count (or page->pp_ref_count) for
>>> refcounting ppiov, we can remove the if checking in the refcounting.
>>>
>
> I'm not sure this is actually possible in the short term. The
> page_pool uses both page->_refcount and page->pp_frag_count for
> refcounting, and I will not be able to remove the special handling
> around page->_refcount as i'm not allowed to call page_ref_*() APIs on
> a non-struct page.
the page_ref_*() API may be avoided using the below patch:
https://patchwork.kernel.org/project/netdevbpf/patch/20231113130041.58124-7-linyunsheng@huawei.com/
But I am not sure how to do that for tx part if devmem for tx is not
intergating into page_pool, that is why I suggest having a tx implementation
for the next version, so that we can have a whole picture of devmem.
>
>>> - We may be able to store the dma_addr of the ppiov in page->dma_addr,
>>> but I'm unsure if that actually works, because the dma_buf dmaddr is
>>> dma_addr_t (u32 or u64), but page->dma_addr is unsigned long (4 bytes
>>> I think). But if it works for pages I may be able to make it work for
>>> ppiov as well.
>>>
>>> - Checks that obtain the page->pp can work with ppiov if we align the
>>> offset of page->pp and ppiov->pp.
>>>
>>> - Checks around page->pp_magic can be removed if we also have offset
>>> aligned ppiov->pp_magic.
>>>
>>> Sadly I don't see us removing the checking for these other cases:
>>>
>>> - page_is_pfmemalloc(): I'm not allowed to pass a non-struct page into
>>> that helper.
>>
>> We can do similar trick like above as bit 1 of page->pp_magic is used to
>> indicate that if it is a pfmemalloc page.
>>
>
> Likely yes.
>
>>>
>>> - page_to_nid(): I'm not allowed to pass a non-struct page into that helper.
>>
>> Yes, this one need special case.
>>
>>>
>>> - page_pool_free_va(): ppiov have no va.
>>
>> Doesn't the skb_frags_readable() checking will protect the page_pool_free_va()
>> from being called on devmem?
>>
>
> This function seems to be only called from veth which doesn't support
> devmem. I can remove the handling there.
>
>>>
>>> - page_pool_sync_for_dev/page_pool_dma_map: ppiov backed by dma-buf
>>> fundamentally can't get mapped again.
>>
>> Can we just fail the page_pool creation with PP_FLAG_DMA_MAP and
>> DMA_ATTR_SKIP_CPU_SYNC flags for devmem provider?
>>
>
> Jakub says PP_FLAG_DMA_MAP must be enabled for devmem, such that the
> page_pool handles the dma mapping of the devmem and the driver doesn't
> use it on its own.
I am not sure what benefit does it bring by enabling the DMA_MAP for devmem,
as devmem seems to call dma_buf_map_attachment() in netdev_bind_dmabuf(), it
does not really need enabling PP_FLAG_DMA_MAP to get the dma addr for the
devmem chunk.
next prev parent reply other threads:[~2023-12-12 11:17 UTC|newest]
Thread overview: 75+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-12-08 0:52 [net-next v1 00/16] Device Memory TCP Mina Almasry
2023-12-08 0:52 ` [net-next v1 01/16] net: page_pool: factor out releasing DMA from releasing the page Mina Almasry
2023-12-10 3:49 ` Shakeel Butt
2023-12-12 8:11 ` Ilias Apalodimas
2023-12-08 0:52 ` [net-next v1 02/16] net: page_pool: create hooks for custom page providers Mina Almasry
2023-12-12 8:07 ` Ilias Apalodimas
2023-12-12 14:47 ` Mina Almasry
2023-12-08 0:52 ` [net-next v1 03/16] queue_api: define queue api Mina Almasry
2023-12-14 1:15 ` Jakub Kicinski
2023-12-08 0:52 ` [net-next v1 04/16] gve: implement " Mina Almasry
2024-03-05 11:45 ` Arnd Bergmann
2023-12-08 0:52 ` [net-next v1 05/16] net: netdev netlink api to bind dma-buf to a net device Mina Almasry
2023-12-14 1:17 ` Jakub Kicinski
2023-12-08 0:52 ` [net-next v1 06/16] netdev: support binding dma-buf to netdevice Mina Almasry
2023-12-08 15:40 ` kernel test robot
2023-12-08 16:02 ` kernel test robot
2023-12-08 17:48 ` David Ahern
2023-12-08 19:22 ` Mina Almasry
2023-12-08 20:32 ` Mina Almasry
2023-12-09 23:29 ` David Ahern
2023-12-11 2:19 ` Mina Almasry
2023-12-08 0:52 ` [net-next v1 07/16] netdev: netdevice devmem allocator Mina Almasry
2023-12-08 17:56 ` David Ahern
2023-12-08 19:27 ` Mina Almasry
2023-12-08 0:52 ` [net-next v1 08/16] memory-provider: dmabuf devmem memory provider Mina Almasry
2023-12-08 22:48 ` Pavel Begunkov
2023-12-08 23:25 ` Mina Almasry
2023-12-10 3:03 ` Pavel Begunkov
2023-12-11 2:30 ` Mina Almasry
2023-12-11 20:35 ` Pavel Begunkov
2023-12-14 20:03 ` Mina Almasry
2023-12-19 23:55 ` Pavel Begunkov
2023-12-08 23:05 ` Pavel Begunkov
2023-12-12 12:25 ` Jason Gunthorpe
2023-12-12 13:07 ` Christoph Hellwig
2023-12-12 14:26 ` Mina Almasry
2023-12-12 14:39 ` Jason Gunthorpe
2023-12-12 14:58 ` Mina Almasry
2023-12-12 15:08 ` Jason Gunthorpe
2023-12-13 1:09 ` Mina Almasry
2023-12-13 2:19 ` David Ahern
2023-12-13 7:49 ` Yinjun Zhang
2023-12-08 0:52 ` [net-next v1 09/16] page_pool: device memory support Mina Almasry
2023-12-08 9:30 ` Yunsheng Lin
2023-12-08 16:05 ` Mina Almasry
2023-12-11 2:04 ` Yunsheng Lin
2023-12-11 2:26 ` Mina Almasry
2023-12-11 4:04 ` Mina Almasry
2023-12-11 11:51 ` Yunsheng Lin
2023-12-11 18:14 ` Mina Almasry
2023-12-12 11:17 ` Yunsheng Lin [this message]
2023-12-12 14:28 ` Mina Almasry
2023-12-13 11:48 ` Yunsheng Lin
2023-12-13 7:52 ` Mina Almasry
2023-12-08 0:52 ` [net-next v1 10/16] page_pool: don't release iov on elevanted refcount Mina Almasry
2023-12-08 0:52 ` [net-next v1 11/16] net: support non paged skb frags Mina Almasry
2023-12-08 0:52 ` [net-next v1 12/16] net: add support for skbs with unreadable frags Mina Almasry
2023-12-08 0:52 ` [net-next v1 13/16] tcp: RX path for devmem TCP Mina Almasry
2023-12-08 15:40 ` kernel test robot
2023-12-08 17:55 ` David Ahern
2023-12-08 19:23 ` Mina Almasry
2023-12-08 0:52 ` [net-next v1 14/16] net: add SO_DEVMEM_DONTNEED setsockopt to release RX frags Mina Almasry
2023-12-12 19:08 ` Simon Horman
2023-12-08 0:52 ` [net-next v1 15/16] net: add devmem TCP documentation Mina Almasry
2023-12-12 19:14 ` Simon Horman
2023-12-08 0:52 ` [net-next v1 16/16] selftests: add ncdevmem, netcat for devmem TCP Mina Almasry
2023-12-08 1:47 ` [net-next v1 00/16] Device Memory TCP Mina Almasry
2023-12-08 17:57 ` David Ahern
2023-12-08 19:31 ` Mina Almasry
2023-12-10 3:48 ` Shakeel Butt
2023-12-12 5:58 ` Christoph Hellwig
2023-12-14 6:20 ` patchwork-bot+netdevbpf
2023-12-14 6:48 ` Christoph Hellwig
2023-12-14 6:51 ` Mina Almasry
2023-12-14 6:59 ` Christoph Hellwig
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=2cdf173c-95e4-2141-56f7-0761705cd737@huawei.com \
--to=linyunsheng@huawei.com \
--cc=almasrymina@google.com \
--cc=arnd@arndb.de \
--cc=bpf@vger.kernel.org \
--cc=christian.koenig@amd.com \
--cc=corbet@lwn.net \
--cc=davem@davemloft.net \
--cc=dri-devel@lists.freedesktop.org \
--cc=dsahern@kernel.org \
--cc=edumazet@google.com \
--cc=hawk@kernel.org \
--cc=hramamurthy@google.com \
--cc=ilias.apalodimas@linaro.org \
--cc=jeroendb@google.com \
--cc=kuba@kernel.org \
--cc=linux-arch@vger.kernel.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=linux-media@vger.kernel.org \
--cc=netdev@vger.kernel.org \
--cc=pabeni@redhat.com \
--cc=pkaligineedi@google.com \
--cc=shailend@google.com \
--cc=shakeelb@google.com \
--cc=shuah@kernel.org \
--cc=sumit.semwal@linaro.org \
--cc=willemdebruijn.kernel@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).