From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id DC29CC001DC for ; Tue, 11 Jul 2023 17:25:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231509AbjGKRZ1 (ORCPT ); Tue, 11 Jul 2023 13:25:27 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39052 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232932AbjGKRYq (ORCPT ); Tue, 11 Jul 2023 13:24:46 -0400 Received: from mail-ua1-x931.google.com (mail-ua1-x931.google.com [IPv6:2607:f8b0:4864:20::931]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 7BC651730 for ; Tue, 11 Jul 2023 10:24:24 -0700 (PDT) Received: by mail-ua1-x931.google.com with SMTP id a1e0cc1a2514c-7948540a736so2018970241.1 for ; Tue, 11 Jul 2023 10:24:24 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20221208; t=1689096263; x=1691688263; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=vxEt0Z2eexi8Ljxbl6AxsjACGcp0PhbeI1Zbkig5Ea8=; b=2MNo9qcQE70ijvOMfhfeCr5b57NMCaHmxyfLaple5c0ON72I6Fl6KkCpEF2ZUAP6Ja V0z2dG+x+IsgZmudM/8wwO3HeFqajJZZHGPKbEt6SALfCu+Pjnxdii+e3bsqiFj2E3my Y/ErVdBmKyVEkX5L9MQap9ZPz0HWB43sGOmSZ/vOEk0haEGCaJ4BXm0i68AOESbjIK// yHeVhjhPwmEJgs7OzM7NHltcgWysktGwYwz4yjqmTJOBKT8q9CDeeMZvsmkc/oZsH0tp 3B2iJT8PUjxWK3Udzzlf3cKSa9yNqS4nY+xFqcPU5ISJors0n4DWQZ0X9dxbx9KCcKzN SXWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689096263; x=1691688263; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=vxEt0Z2eexi8Ljxbl6AxsjACGcp0PhbeI1Zbkig5Ea8=; b=TnK79iiiH6QeHX+sWxBuy6pyJQ35TESEcMnSDNcqAmQDZ7wAZt3tblQFg1blF0pd6M wg2vzrGednZabPQPaFsxKJP8c7JtXZOM0/N2a598c2B5EfL0oPMveMazE5B6X4x4Tp2O lTNXb8TLi3jLM1FBlZZq0rcEE8Dxg6DLnEBc2bRn65tVmEZ/k18E4QrYsJH6jjJq6/FD 73wdclc2huaXAQ/vI7yk5KEfjqbHU2gldyuhoBRr5Sn11cQr+ShycUV5ktaG0v8gJxjn 1qnYCH0O54qnvBPNdEf9s8SoOdj+7mQ67YlH3g8nERe+8ol+YsAzpfCIlIPaVFry8KTN pMPg== X-Gm-Message-State: ABy/qLYrBe8FizNkvT4gv38vA4aNsD8hh+i7022g6H2KxSIeFnRS2Jgr Ki+8PFALsfcCGJer+f8biwjOJ/CAvCUgLKXZz9WdCg== X-Google-Smtp-Source: APBJJlEbNbeKglx24tA12e4BmXAPi9rr0ZNOak9iV8KnYQWaC46zN/l1EJsSvydC+DrZ4BU4kBJ0Te5mDCNhUBfgEcw= X-Received: by 2002:a67:fd59:0:b0:440:cfc5:f91a with SMTP id g25-20020a67fd59000000b00440cfc5f91amr8676055vsr.29.1689096263370; Tue, 11 Jul 2023 10:24:23 -0700 (PDT) MIME-Version: 1.0 References: <20230619110705.106ec599@kernel.org> <5e0ac5bb-2cfa-3b58-9503-1e161f3c9bd5@kernel.org> In-Reply-To: From: Mina Almasry Date: Tue, 11 Jul 2023 10:24:11 -0700 Message-ID: Subject: Re: Memory providers multiplexing (Was: [PATCH net-next v4 4/5] page_pool: remove PP_FLAG_PAGE_FRAG flag) To: Jason Gunthorpe , Bjorn Helgaas , logang@deltatee.com Cc: Christoph Hellwig , John Hubbard , Dan Williams , David Ahern , Jakub Kicinski , Jesper Dangaard Brouer , brouer@redhat.com, Alexander Duyck , Yunsheng Lin , davem@davemloft.net, pabeni@redhat.com, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Lorenzo Bianconi , Yisen Zhuang , Salil Mehta , Eric Dumazet , Sunil Goutham , Geetha sowjanya , Subbaraya Sundeep , hariprasad , Saeed Mahameed , Leon Romanovsky , Felix Fietkau , Ryder Lee , Shayne Chen , Sean Wang , Kalle Valo , Matthias Brugger , AngeloGioacchino Del Regno , Jesper Dangaard Brouer , Ilias Apalodimas , linux-rdma@vger.kernel.org, linux-wireless@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org, Jonathan Lemon Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Jul 11, 2023 at 6:11=E2=80=AFAM Jason Gunthorpe wrot= e: > > On Mon, Jul 10, 2023 at 05:45:05PM -0700, Mina Almasry wrote: > > > > At least from my position I want to see MEMORY_DEVICE_PCI_P2PDMA used > > > to represent P2P memory. > > > > Would using p2pdma API instead of dmabuf be an acceptable direction? > > "p2pdma API" is really just using MEMORY_DEVICE_PCI_P2PDMA and > teaching the pagepool how to work with ZONE_DEVICE pages. > Yes, that's what I have in mind. Roughly something along the lines where the device memory provider (GPU or what not) does something like a pci_p2pdma_add_resource(), and the NIC client driver allocates the pages from the resource, and if is_pci_p2pdma_page() use pci_p2pdma_map_sg() instead of dma_map_sg() and whatnot. > I suspect this will clash badly with Matthew's work here: > > https://lore.kernel.org/all/20230111042214.907030-1-willy@infradead.org/ > > As from a mm side we haven't ever considered that ZONE_DEVICE and > "netmem" can be composed together. The entire point of netmem like > stuff is that the allocator hands over the majority of struct page to > the allocatee, and ZONE_DEVICE can't work like that. > > However, assuming that can be solved in some agreeable way then it > would be OK to go down this path. > I think there is potential to solve this in an agreeable way. We may be able to get netmem ZONE_DEVICE pages using the memdesc idea you proposed, or something like the xarray meta data you mention below. If not that, I think Jakub already said that he is considering coming up with a 'page pool like' API that drivers can use, with an implementation that is compatible with ZONE_DEVICE pages. > But, I feel like this is just overall too hard a direction from the mm > perspective. > > I don't know anything about page pool, but the main sticking point is > its reliance on struct page. If it can find another way to locate its > meta data (eg an xarray), at least for some cases, it would make > things alot easier. > --=20 Thanks, Mina