From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5D394C04A95 for ; Wed, 28 Sep 2022 21:38:56 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9E2986B0074; Wed, 28 Sep 2022 17:38:55 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 991D46B0075; Wed, 28 Sep 2022 17:38:55 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 85A736B0078; Wed, 28 Sep 2022 17:38:55 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 751B36B0074 for ; Wed, 28 Sep 2022 17:38:55 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 34C2FC0F0A for ; Wed, 28 Sep 2022 21:38:55 +0000 (UTC) X-FDA: 79962809430.07.F78FA3C Received: from ale.deltatee.com (ale.deltatee.com [204.191.154.188]) by imf12.hostedemail.com (Postfix) with ESMTP id A18EB4000E for ; Wed, 28 Sep 2022 21:38:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=deltatee.com; s=20200525; h=Subject:In-Reply-To:From:References:Cc:To: MIME-Version:Date:Message-ID:content-disposition; bh=VOh9QcFEDE1YfzcxWIGp+1k7f+qP9ixR08UnNbWnKpc=; b=mOBFIK5m8mLy4ywi3FVTK4mL6V oj75zw9aDB26C9Y6E3q3Zh+2OVl0EQqV4erRsyOn6/kj0Tt8zleEZbtbluaojd3dKCXarw1p0vw2N Mjm+cJscVTFtRI6ZPBK1NMDtmJ37SkFDwrsyizQWmLDPIffvRJnH6j1msXmitMEuUo/Vnmp2R1D4p 3eeslAUPK9oaVUW/PBgxclT7nvMxXVf6RSS/t7hKvCFdGuk4mom65QCMpDVM/oB2p4534U7GPMsT2 IrYHvtiO2s/gVPnZi67xksWEgfLmwrPbCcD3YWEry/HYu/+x3cwqtiiEiJwyJj2kiqcaFXQ9u2+sd n5FHDhcw==; Received: from s0106ac1f6bb1ecac.cg.shawcable.net ([70.73.163.230] helo=[192.168.11.155]) by ale.deltatee.com with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 (Exim 4.94.2) (envelope-from ) id 1odelP-00D1s7-Da; Wed, 28 Sep 2022 15:38:44 -0600 Message-ID: Date: Wed, 28 Sep 2022 15:38:36 -0600 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:91.0) Gecko/20100101 Thunderbird/91.13.0 Content-Language: en-US To: Jason Gunthorpe Cc: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , Greg Kroah-Hartman , Dan Williams , =?UTF-8?Q?Christian_K=c3=b6nig?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Stephen Bates References: <64f8da81-7803-4db4-73da-a158295cbc9c@deltatee.com> <2327d393-af5c-3f4c-b9b9-6852b9d72f90@deltatee.com> <3840c1c6-3a5c-2286-e577-949f0d4ea7a6@deltatee.com> <980899e1-532a-772b-2f6d-6fb017def50b@deltatee.com> From: Logan Gunthorpe In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 70.73.163.230 X-SA-Exim-Rcpt-To: jgg@ziepe.ca, linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, hch@lst.de, gregkh@linuxfoundation.org, dan.j.williams@intel.com, christian.koenig@amd.com, jhubbard@nvidia.com, ddutile@redhat.com, willy@infradead.org, daniel.vetter@ffwll.ch, dave.b.minturn@intel.com, jason@jlekstrand.net, dave.hansen@linux.intel.com, jianxin.xiong@intel.com, helgaas@kernel.org, ira.weiny@intel.com, robin.murphy@arm.com, martin.oliveira@eideticom.com, ckulkarnilinux@gmail.com, rcampbell@nvidia.com, sbates@raithlin.com X-SA-Exim-Mail-From: logang@deltatee.com Subject: Re: [PATCH v10 1/8] mm: introduce FOLL_PCI_P2PDMA to gate getting PCI P2PDMA pages X-SA-Exim-Version: 4.2.1 (built Sat, 13 Feb 2021 17:57:42 +0000) X-SA-Exim-Scanned: Yes (on ale.deltatee.com) ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1664401134; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VOh9QcFEDE1YfzcxWIGp+1k7f+qP9ixR08UnNbWnKpc=; b=d7O4C4RcIwsNCu6m+XSAhUSuoUtVfpoTGHDF6TK7JRl8ojm4JmfjrU1of2xu1bTOqeErwv Ntq2q54a6mxwTiy77yduSLsDCEP20JIVvLA/g1I7gy4yErgMBM4gAzmo16hqZCSHe9QYZv 9mJ6yz/mIO/UvqMWa6Ax1y+eBvaXMAw= ARC-Authentication-Results: i=1; imf12.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=mOBFIK5m; spf=pass (imf12.hostedemail.com: domain of logang@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=logang@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1664401134; a=rsa-sha256; cv=none; b=XmD8CznLUkfFKGNvLM0kC1uUZvnN2StlMGj04BsXtlUlD6ge+Dzwz5LwL6WCeEMC9sOswi kJ4X1SwLfhGQU6tW/IMGqRZURpM26BGXY//uAbtBo3Ia9WSQ92G9VwHbuofKzs0qRHJQmW DiZnV+k294O7dqQletgqstrNpdl/jNQ= Authentication-Results: imf12.hostedemail.com; dkim=pass header.d=deltatee.com header.s=20200525 header.b=mOBFIK5m; spf=pass (imf12.hostedemail.com: domain of logang@deltatee.com designates 204.191.154.188 as permitted sender) smtp.mailfrom=logang@deltatee.com; dmarc=pass (policy=none) header.from=deltatee.com X-Rspam-User: X-Stat-Signature: yz7wyt9fzdx6p8zxyu1jpr1a7g8fro8i X-Rspamd-Queue-Id: A18EB4000E X-Rspamd-Server: rspam08 X-HE-Tag: 1664401131-294343 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 2022-09-26 16:57, Jason Gunthorpe wrote: > On Fri, Sep 23, 2022 at 05:51:49PM -0600, Logan Gunthorpe wrote: >> Userspace code that's written for purpose can look at the EREMOTEIO error >> and tell the user something useful, if we return the correct error. >> If we return ENOMEM in this case, that is not possible because >> lots of things might have caused that error. > > That is reasonable, but I'd still prefer to see it done more > centrally. > > I mean the way the code is structured is at the top of the call chain > the PIN/GET/0 is decided and then the callchain is run. All the > callsites of try_grab_page() must be safe to call under FOLL_PIN > because their caller is making the decision what flag to use. Ok, so I've done some auditing here. I've convinced myself it's safe to access the page before incrementing the reference: * In the try_grab_page() case it must be safe as all call sites do seem to be called under the appropriate ptl or mmap_lock (though this is hard to audit). It's also true that it touches the page struct in the sense of the reference. * In the try_grab_folio() case there already is already a similar FOLL_LONGTERM check in that function *before* getting the reference and the page should be stable due to the existing gup fast guarantees. So we don't need to do the check after we have the reference and release it when it fails. This simplifies things. Moving the check into try_grab_x() should be possible with some cleanup. For try_grab_page(), there are a few call sites that WARN_ON if it fails, assuming it cannot fail seeing the page is stable. try_grab_page() already has a WARN_ON on failure so it appears fine to remove the second WARN_ON and add a new failure path that doesn't WARN. For try_grab_folio() there's one call site in follow_hugetlb_page() that assumes success and warns on failure; but this call site only applies to hugetlb pages which should never be P2PDMA pages (nor non-longterm pages which is another existing failure path). So I've added a note in the comment with a couple other conditions that should not be possible. I expect this work is way too late for the merge window now so I'll send v11 after the window. In the meantime, if you want to do a quick review on the first two patches, it would speed things up if there are obvious changes. You can see these patches on this git branch: https://github.com/sbates130272/linux-p2pmem/ p2pdma_user_cmb_v11pre Thanks, Logan