From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-7.3 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1697BC64E7B for ; Wed, 2 Dec 2020 18:01:30 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 5DECB22201 for ; Wed, 2 Dec 2020 18:01:29 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 5DECB22201 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=deltatee.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 557D86B0036; Wed, 2 Dec 2020 13:01:28 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 4E0506B005C; Wed, 2 Dec 2020 13:01:28 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 380ED6B005D; Wed, 2 Dec 2020 13:01:28 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0029.hostedemail.com [216.40.44.29]) by kanga.kvack.org (Postfix) with ESMTP id 1F46B6B0036 for ; Wed, 2 Dec 2020 13:01:28 -0500 (EST) Received: from smtpin23.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id BA2A82465 for ; Wed, 2 Dec 2020 18:01:27 +0000 (UTC) X-FDA: 77549109414.23.sound30_2110da2273b5 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin23.hostedemail.com (Postfix) with ESMTP id 9E68B37609 for ; Wed, 2 Dec 2020 18:01:27 +0000 (UTC) X-HE-Tag: sound30_2110da2273b5 X-Filterd-Recvd-Size: 5260 Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf05.hostedemail.com (Postfix) with ESMTP for ; Wed, 2 Dec 2020 18:01:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:In-Reply-To:MIME-Version:Date: Message-ID:From:References:Cc:To:content-disposition; bh=PNgBCQ8XfFmn9D3DM+iYXdqG+oFaCFQKqsKGuaozXnM=; b=I/z1+NEbS5rhAEcJyNoJdYZjKX GF2BGYAtWOziMnVigqzE1eUHMglwIZi+OmM1AakjZs+Hsd+bzWcf+PlXB/NIrTjj2GlCUoWYRDZcR zPrHxLOH3/uzMbPUwi8F4aVw+xFx1S2W8X9T3V8IPnUoy56PN4haxXE8+yqphc/TCQmKcb8g8ZBDB dFrTJvFFcAG2SkWwRdgs6sezejN5wBuhZoRuXG/ncxXX65UP6Ef8E3zQkSlkTF4IEVKC8dcAdHVlK +xQADBoG7nV0lmFg7pnNxU24eHo65/Fy8AJIsfGoESJY8P6Q+68xlKurQ45l25yeVq2/CtZ28awXH iDej8pIw==; Received: from s01060023bee90a7d.cg.shawcable.net ([24.64.145.4] helo=[192.168.0.10]) by ale.deltatee.com with esmtpsa (TLS1.3:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.92) (envelope-from ) id 1kkWRD-0003mr-EY; Wed, 02 Dec 2020 11:01:12 -0700 To: Christoph Hellwig , Ralph Campbell Cc: linux-mm@kvack.org, nouveau@lists.freedesktop.org, linux-kselftest@vger.kernel.org, linux-kernel@vger.kernel.org, Jerome Glisse , John Hubbard , Alistair Popple , Jason Gunthorpe , Bharata B Rao , Zi Yan , "Kirill A . Shutemov" , Yang Shi , Ben Skeggs , Shuah Khan , Andrew Morton , Dan Williams , linux-nvdimm@lists.01.org, linux-fsdevel@vger.kernel.org References: <20201106005147.20113-1-rcampbell@nvidia.com> <20201106005147.20113-4-rcampbell@nvidia.com> <20201106080322.GE31341@lst.de> <20201109091415.GC28918@lst.de> <20201202101426.GC7597@lst.de> From: Logan Gunthorpe Message-ID: <7229bb21-7bf7-4989-e7cf-210834190693@deltatee.com> Date: Wed, 2 Dec 2020 11:01:06 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.5.0 MIME-Version: 1.0 In-Reply-To: <20201202101426.GC7597@lst.de> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 24.64.145.4 X-SA-Exim-Rcpt-To: linux-fsdevel@vger.kernel.org, linux-nvdimm@lists.01.org, dan.j.williams@intel.com, akpm@linux-foundation.org, shuah@kernel.org, bskeggs@redhat.com, yang.shi@linux.alibaba.com, kirill.shutemov@linux.intel.com, ziy@nvidia.com, bharata@linux.ibm.com, jgg@nvidia.com, apopple@nvidia.com, jhubbard@nvidia.com, jglisse@redhat.com, linux-kernel@vger.kernel.org, linux-kselftest@vger.kernel.org, nouveau@lists.freedesktop.org, linux-mm@kvack.org, rcampbell@nvidia.com, hch@lst.de X-SA-Exim-Mail-From: logang@deltatee.com Subject: Re: [PATCH v3 3/6] mm: support THP migration to device private memory X-SA-Exim-Version: 4.2.1 (built Wed, 08 May 2019 21:11:16 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 2020-12-02 3:14 a.m., Christoph Hellwig wrote:>> MEMORY_DEVICE_PCI_P2PDMA: >> Struct pages are created in pci_p2pdma_add_resource() and represent device >> memory accessible by PCIe bar address space. Memory is allocated with >> pci_alloc_p2pmem() based on a byte length but the gen_pool_alloc_owner() >> call will allocate memory in a minimum of PAGE_SIZE units. >> Reference counting is +1 per *allocation* on the pgmap->ref reference count. >> Note that this is not +1 per page which is what put_page() expects. So >> currently, a get_page()/put_page() works OK because the page reference count >> only goes 1->2 and 2->1. If it went to zero, the pgmap->ref reference count >> would be incorrect if the allocation size was greater than one page. >> >> I see pci_alloc_p2pmem() is called by nvme_alloc_sq_cmds() and >> pci_p2pmem_alloc_sgl() to create a command queue and a struct scatterlist *. >> Looks like sg_page(sg) returns the ZONE_DEVICE struct page of the scatterlist. >> There are a huge number of places sg_page() is called so it is hard to tell >> whether or not get_page()/put_page() is ever called on MEMORY_DEVICE_PCI_P2PDMA >> pages. > > Nothing should call get_page/put_page on them, as they are not treated > as refcountable memory. More importantly nothing is allowed to keep > a reference longer than the time of the I/O. Yes, right now this is safe, as Christoph notes there are no places where these should be got/put. But eventually we'll need to change how pci_alloc_p2pmem() works to take references on the actual pages and allow freeing individual pages, similar to what you suggest. This is one of the issues Jason pointed out in my last RFC to try to pass these pages through GUP. Logan