From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.5 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS, URIBL_BLOCKED autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C6E98C2BA83 for ; Fri, 7 Feb 2020 19:46:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8E4D022314 for ; Fri, 7 Feb 2020 19:46:29 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="YM+PmivT" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8E4D022314 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 394E56B0003; Fri, 7 Feb 2020 14:46:29 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3449B6B0006; Fri, 7 Feb 2020 14:46:29 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 20C016B0007; Fri, 7 Feb 2020 14:46:29 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0070.hostedemail.com [216.40.44.70]) by kanga.kvack.org (Postfix) with ESMTP id 028A56B0003 for ; Fri, 7 Feb 2020 14:46:28 -0500 (EST) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id AE07A40F4 for ; Fri, 7 Feb 2020 19:46:28 +0000 (UTC) X-FDA: 76464362856.22.thumb46_72b2aad7cc458 X-HE-Tag: thumb46_72b2aad7cc458 X-Filterd-Recvd-Size: 5313 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf26.hostedemail.com (Postfix) with ESMTP for ; Fri, 7 Feb 2020 19:46:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=In-Reply-To:Content-Transfer-Encoding :Content-Type:MIME-Version:References:Message-ID:Subject:Cc:To:From:Date: Sender:Reply-To:Content-ID:Content-Description; bh=MRJAZfyI+DPUYAJafF7nxGyfRX9uOty2eCox/3nz/no=; b=YM+PmivThUdMJ5jmolg8uNgZJp +2xOME1RexCGlKsXWUBk9tp92oU+hPi4rEezZmdU1xJa/1+t9v1O142GCRcelacVAFIiwR/FgP/Nx p8yJKbyGvgWK6wWBlBGr7ITAk/eYLvbWRdjlzezrfhvRG140VaUkSvqEI9FV4Eq0jq57m2ou20c77 F4p5WAXepFXcWeI0Y+U/K6BaoOqjckjTw1RzuTK0e0Twp8GePH2f/j7c6+KW4HE5Rm04toDnlp10L 6uKXTdNVZr2iENiI6gqiQGWVRnGaQlZqR1fN5fV5iLlvVAKnH5ELzIesccDh0jZUm7lsexnZKwpjf DHepZ6qw==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1j09a0-0006Xs-Ny; Fri, 07 Feb 2020 19:46:20 +0000 Date: Fri, 7 Feb 2020 11:46:20 -0800 From: Matthew Wilcox To: Jason Gunthorpe Cc: lsf-pc@lists.linux-foundation.org, linux-mm@kvack.org, linux-pci@vger.kernel.org, linux-rdma@vger.kernel.org, Christian =?iso-8859-1?Q?K=F6nig?= , Daniel Vetter , Logan Gunthorpe , Stephen Bates , =?iso-8859-1?B?Suly9G1l?= Glisse , Ira Weiny , Christoph Hellwig , John Hubbard , Ralph Campbell , Dan Williams , Don Dutile , Thomas =?iso-8859-1?Q?Hellstr=F6m_=28VMware=29?= , Joao Martins Subject: Re: [LSF/MM TOPIC] get_user_pages() for PCI BAR Memory Message-ID: <20200207194620.GG8731@bombadil.infradead.org> References: <20200207182457.GM23346@mellanox.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline In-Reply-To: <20200207182457.GM23346@mellanox.com> Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Feb 07, 2020 at 02:24:57PM -0400, Jason Gunthorpe wrote: > Many systems can now support direct DMA between two PCI devices, for > instance between a RDMA NIC and a NVMe CMB, or a RDMA NIC and GPU > graphics memory. In many system architectures this peer-to-peer PCI-E > DMA transfer is critical to achieving performance as there is simply > not enough system memory/PCI-E bandwidth for data traffic to go > through the CPU socket. >=20 > For many years various out of tree solutions have existed to serve > this need. Recently some components have been accpeted into mainline, > such as the p2pdma system, which allows co-operating drivers to setup > P2P DMA transfers at the PCI level. This has allowed some kernel P2P > DMA transfers related to NVMe CMB and RDMA to become supported. >=20 > A major next step is to enable P2P transfers under userspace > control. This is a very broad topic, but for this session I propose to > focus on initial cases of supporting drivers can setup a P2P transfer > from a PCI BAR page mmap'd to userspace. This is the basic starting > point for future discussions on how to adapt get_user_pages() IO paths > (ie O_DIRECT, net zero copy TX, RDMA, etc) to support PCI BAR memory. >=20 > As all current drivers doing DMA from user space must go through > get_user_pages() (or its new sibling hmm_range_fault()), some > extension of the get_user_pages() API is needed to allow drivers > supporting P2P to see the pages. >=20 > get_user_pages() will require some 'struct page' and 'struct > vm_area_struct' representation of the BAR memory beyond what today's > io_remap_pfn_range()/etc produces. >=20 > This topic has been discussed in small groups in various conferences > over the last year, (plumbers, ALPSS, LSF/MM 2019, etc). Having a > larger group together would be productive, especially as the direction > has a notable impact on the general mm. >=20 > For patch sets, we've seen a number of attempts so far, but little has > been merged yet. Common elements of past discussions have been: > - Building struct page for BAR memory > - Stuffing BAR memory into scatter/gather lists, bios and skbs > - DMA mapping BAR memory > - Referencing BAR memory without a struct page > - Managing lifetime of BAR memory across multiple drivers >=20 > Based on past work, the people in the CC list would be recommended > participants: >=20 > Christian K=F6nig > Daniel Vetter > Logan Gunthorpe > Stephen Bates > J=E9r=F4me Glisse > Ira Weiny > Christoph Hellwig > John Hubbard > Ralph Campbell > Dan Williams > Don Dutile That's a long list, and you're missing=20 "Thomas Hellstr=F6m (VMware)" Joao Martins both of whom have been working on related projects (for PFNs without page= s). Hey, you missed me too! ;-)