From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5B23CEB64D9 for ; Wed, 12 Jul 2023 13:02:01 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231893AbjGLNCA (ORCPT ); Wed, 12 Jul 2023 09:02:00 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:38218 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233018AbjGLNB4 (ORCPT ); Wed, 12 Jul 2023 09:01:56 -0400 Received: from mail-qk1-x72d.google.com (mail-qk1-x72d.google.com [IPv6:2607:f8b0:4864:20::72d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 89BC619BA for ; Wed, 12 Jul 2023 06:01:52 -0700 (PDT) Received: by mail-qk1-x72d.google.com with SMTP id af79cd13be357-765ae938b1bso652008085a.0 for ; Wed, 12 Jul 2023 06:01:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; t=1689166911; x=1691758911; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=YWpuSX94flQxvT/7GFNnyt8FcFburvzifDepq9lHVso=; b=iYD+caT5C5DZuj4SVL8bXKmm54Fife2Oz9uLMzl6trcTZ3BhImO/CJ2diP8cel4MIq XvWzM2pXRBQEjLiVo3Hgt06eJWjI2g2knQDKpA2bpSVnKpPcB5kCx3F2DSBXcUoh1RcW evRk6qNJjMFTRHZraP9CHksMsuP51beQli1rjtg5cRvCFZIiXfjjqzmq7912pS2ynyNH /SqCiWdXv8u7bRbNQhIjI9Y4D6U/0F8mt1bkXpyu60M2tzC25xcrXZUnG7XOxux/chyn iUj28ba/siqynf7x2SgrK9XpRda5v7gelAg/mnDloWN4XKIrIZHL0ttb9xLfMYOzQPpS n/hQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1689166911; x=1691758911; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=YWpuSX94flQxvT/7GFNnyt8FcFburvzifDepq9lHVso=; b=gedT4GHFbSmMs/c/EOJ52rLfqPcP9h15NIrTfSvQRnvC23P9vSQbIDZqFgnFW2SBcB daqzsFuzWxrYP4uhxWkE5ZiQahEac0/xGoc1nN5nnpjQFb3zbpl2HKVWBen8QbAKA4f+ UgN4+5h7jnZi/O22Xkzp6YWuGQ7HIHQNHiJOQHAJprmKYQ4Qj1Y1k0l3XFmRjrabS1OU auSUJaID7iAddzFHBa1YukyFwsXZTobqsSTYwfpDyC1f8fkaV5NX6g3ALjmVlDZR532O CIZnLTnR/ShdqyntF+qBsfL+WjRi012la9UIOCbpmYk9nfic9Y6v0ARfYDAjtBVI3ZzG L2Vg== X-Gm-Message-State: ABy/qLYK+VfwZVNFgNnSSi/Tz8Weyfy82rwgveiX/ypD9NCcPCauG2HT IPi7E67tlWqaLzkronn50dZzcdgwN3/ZUPvgbNc= X-Google-Smtp-Source: APBJJlFY1e3i66W1OtOcIX3n2XlytjAfsqmXfgyyy6DjtCl2reodlKdOdi1miiqTfvLyhJirNei5sQ== X-Received: by 2002:ae9:dfc5:0:b0:767:d847:278a with SMTP id t188-20020ae9dfc5000000b00767d847278amr6940605qkf.74.1689166911567; Wed, 12 Jul 2023 06:01:51 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-68-25-194.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.68.25.194]) by smtp.gmail.com with ESMTPSA id u17-20020a05620a121100b007673f8803c3sm2105604qkj.96.2023.07.12.06.01.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 12 Jul 2023 06:01:47 -0700 (PDT) Received: from jgg by wakko with local (Exim 4.95) (envelope-from ) id 1qJZTU-000OJR-6S; Wed, 12 Jul 2023 10:01:44 -0300 Date: Wed, 12 Jul 2023 10:01:44 -0300 From: Jason Gunthorpe To: Mina Almasry Cc: David Ahern , Samiullah Khawaja , Willem de Bruijn , Jakub Kicinski , Christoph Hellwig , John Hubbard , Dan Williams , Jesper Dangaard Brouer , brouer@redhat.com, Alexander Duyck , Yunsheng Lin , davem@davemloft.net, pabeni@redhat.com, netdev@vger.kernel.org, linux-kernel@vger.kernel.org, Lorenzo Bianconi , Yisen Zhuang , Salil Mehta , Eric Dumazet , Sunil Goutham , Geetha sowjanya , Subbaraya Sundeep , hariprasad , Saeed Mahameed , Leon Romanovsky , Felix Fietkau , Ryder Lee , Shayne Chen , Sean Wang , Kalle Valo , Matthias Brugger , AngeloGioacchino Del Regno , Jesper Dangaard Brouer , Ilias Apalodimas , linux-rdma@vger.kernel.org, linux-wireless@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mediatek@lists.infradead.org, Jonathan Lemon , Christian =?utf-8?B?S8O2bmln?= , logang@deltatee.com, Bjorn Helgaas Subject: Re: Memory providers multiplexing (Was: [PATCH net-next v4 4/5] page_pool: remove PP_FLAG_PAGE_FRAG flag) Message-ID: References: <20230710215906.49514550@kernel.org> <20230711050445.GA19323@lst.de> <20230711090047.37d7fe06@kernel.org> <04187826-8dad-d17b-2469-2837bafd3cd5@kernel.org> <20230711093224.1bf30ed5@kernel.org> <20230711133915.03482fdc@kernel.org> <2263ae79-690e-8a4d-fca2-31aacc5c9bc6@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Jul 11, 2023 at 08:42:24PM -0700, Mina Almasry wrote: > 1. The device memory driver would be the p2pdma provider. It would > expose a user API which allocates a device memory region, calls > pci_p2pdma_add_resource() and pci_p2pmem_publish() on it, and returns > a reference to it to the userspace. This is not quite right, if you convert any of the GPU drivers to use P2PDMA you are going to need to restructure the p2pmem stuff to seperate the genalloc. The GPU driver must continue to be the owner and allocator of the MMIO memory it already controls, we can't have two allocators working in parallel. The genalloc stuff supports the special NVMe use case, I don't know of anything else that would like to work that way. > 2. The NIC driver would be the p2pdma client and orchestrator. It > would expose a user API which binds an rxq to a pci device. Prior to > the bind the user API would check that the pci device has published > p2p memory (pci_has_p2pmem()), and check the the p2p mem is accessible > to the driver (pci_p2pdma_distance() I think), etc. This doesn't fit the programming model for GPUs at all. You don't want to get packets landing in random GPU memory that a kernel side allocator selects, you want packets landing in GPU memory owned by a specific process that owns the TCP connection. This is why DMABUF is used here as it gives a handle to the GPU memory. What you want is to get the P2P pages either directly from the DMABUF or via pin_user_pages() on the DMABUF's mmap. > AFAICT, all the concerns brought up in this thread are sidestepped by > using p2pdma. I need not allocate struct pages in the core dma-buf > code anymore (or anywhere), and I need not allocate pgmaps. I would > just re-use the p2pdma support. Well, as I said it is going to be a big ask to P2P enable any of the DRM drivers. And you still have the netmem vs zone_device struct page conflict to figure out But it is alot closer to reasonable than this RFC. Jason