From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3860FC04A95 for ; Fri, 23 Sep 2022 19:53:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 60CE78000D; Fri, 23 Sep 2022 15:53:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5BC7E80007; Fri, 23 Sep 2022 15:53:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 45CE88000D; Fri, 23 Sep 2022 15:53:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 3719180007 for ; Fri, 23 Sep 2022 15:53:30 -0400 (EDT) Received: from smtpin16.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id F0F6A4093A for ; Fri, 23 Sep 2022 19:53:29 +0000 (UTC) X-FDA: 79944399738.16.9F1C427 Received: from mail-qk1-f181.google.com (mail-qk1-f181.google.com [209.85.222.181]) by imf04.hostedemail.com (Postfix) with ESMTP id 90FE940007 for ; Fri, 23 Sep 2022 19:53:29 +0000 (UTC) Received: by mail-qk1-f181.google.com with SMTP id c19so778809qkm.7 for ; Fri, 23 Sep 2022 12:53:29 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date; bh=Xu8gZjkVZRWUz/+3Yxaqqu61sLGoB8IQaIP0sQAkChQ=; b=Cki3lJod5MEkJT5PN6PT1v5YUMO4lOq8Qgdcx15IKRvBbMdbxNdRgk41r5xfQQ/1HG 2ASULy9J6ltZ9W3t4wV3uaujA3tMiHfytMnnyRwJ7MTaF+BowtUE0Pkfd7y5ZTm6LRU4 iL+guWh7lj4qja4HLmTeHePH8D5hMfjQ+ltTRZ7mm/+RJczYgFcvcJpCtbEcZmnsPPWf fegxJ8YTeFW3wcgHYmVZhupjegar2Xf7IUWRnxVeo3IyP9whokDd3qJmBEVrcoSU6U9L 70gBpR7otxqq7USMmzoekqnIsRytd0JMACVCWp7tqC4bTgF0wMjR+GhMk1KK2fvW1DQu lumg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date; bh=Xu8gZjkVZRWUz/+3Yxaqqu61sLGoB8IQaIP0sQAkChQ=; b=8FDu95YpZdF7MPo+VvvdZQt4f+JgvdtoL4FIgYlfP6yo0i6HgkFcAl5RZ9tj43UHEf p5kOGoZ0PsNADHhGw//SMXd3ewG7F4pcT4+ZM1ow/sbS3F/Vw7tQrr7Ffsz+0oU6i/rN zRwpjoEnvDab3cEBPPJHkYuviYDs0WazeubCp73deCm7kTXsjDx2Biti+wQ8C9saFg44 UJJ2RoJaMXfdFshFPBbFLkIAbEoshFKbA/SEtDqNLN1ETHJExr94DkTegzFa079BA7FM S5epBdgdsB+XlJMU7ufW9xzzrOLxxjMzZLML9c9Jigb6GOrGLk0tmBxVLOzyHd1DO5nI qyNQ== X-Gm-Message-State: ACrzQf3YcW2NoP/HKVgGUnGz5yWbVov6NUNzxGtyVAmH0kuH5dKkE9oL XG9x3vTFB1FLhfeCGpmoX9zjEA== X-Google-Smtp-Source: AMsMyM4n7SyKJcwiqURLSex/T5WPj5xfwJObpr4myLNC80+tgwD+8gKZ6p/kJJeI2S3cibPwERKwrw== X-Received: by 2002:a05:620a:24d6:b0:6cd:f96a:35b with SMTP id m22-20020a05620a24d600b006cdf96a035bmr6784230qkn.471.1663962808669; Fri, 23 Sep 2022 12:53:28 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-162-113-129.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.113.129]) by smtp.gmail.com with ESMTPSA id d21-20020ac84e35000000b00359961365f1sm6178104qtw.68.2022.09.23.12.53.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 23 Sep 2022 12:53:28 -0700 (PDT) Received: from jgg by wakko with local (Exim 4.95) (envelope-from ) id 1obojn-002y80-Ez; Fri, 23 Sep 2022 16:53:27 -0300 Date: Fri, 23 Sep 2022 16:53:27 -0300 From: Jason Gunthorpe To: Logan Gunthorpe Cc: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, Christoph Hellwig , Greg Kroah-Hartman , Dan Williams , Christian =?utf-8?B?S8O2bmln?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Stephen Bates Subject: Re: [PATCH v10 1/8] mm: introduce FOLL_PCI_P2PDMA to gate getting PCI P2PDMA pages Message-ID: References: <20220922163926.7077-1-logang@deltatee.com> <20220922163926.7077-2-logang@deltatee.com> <64f8da81-7803-4db4-73da-a158295cbc9c@deltatee.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <64f8da81-7803-4db4-73da-a158295cbc9c@deltatee.com> ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=Cki3lJod; spf=pass (imf04.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.222.181 as permitted sender) smtp.mailfrom=jgg@ziepe.ca; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1663962809; a=rsa-sha256; cv=none; b=Di/ZiapSKBbZ3UMeU8zlW5DEE9Zemr1INh/4QFXWg2GOxQxI9szhQWLE8ehpk6GbZ6zbu+ 5E+jfOzGUOlnV1zdz6FIT3LfnTNRbcGYMZBdzqsSUv0nMgUst2oB25G+FJ5L3PN7Uwfznb TrXM7OROeI9TQbrlTJ3G2XcIE9mMAlw= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1663962809; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Xu8gZjkVZRWUz/+3Yxaqqu61sLGoB8IQaIP0sQAkChQ=; b=SS/b9LzEfq0GHhLswFgZcKsjql15l/PVnR6tLka565S34N2TTaBD2V3YhS6zXAHFvjRHbn ZAUDUdZO/1mxuEaWn/ab4mKFXok7qv9zviw3ndq09xemQSlBch8LMzEa5kDCFzD4LLULza BSb2i+SMJiHd1o99VjDdsNQH3pwNu3Q= Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=ziepe.ca header.s=google header.b=Cki3lJod; spf=pass (imf04.hostedemail.com: domain of jgg@ziepe.ca designates 209.85.222.181 as permitted sender) smtp.mailfrom=jgg@ziepe.ca; dmarc=none X-Stat-Signature: ox4fd8ef498meusojzqxjr8i4bigm5dt X-Rspamd-Queue-Id: 90FE940007 X-Rspamd-Server: rspam04 X-Rspam-User: X-HE-Tag: 1663962809-500586 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri, Sep 23, 2022 at 01:08:31PM -0600, Logan Gunthorpe wrote: > > > On 2022-09-23 12:13, Jason Gunthorpe wrote: > > On Thu, Sep 22, 2022 at 10:39:19AM -0600, Logan Gunthorpe wrote: > >> GUP Callers that expect PCI P2PDMA pages can now set FOLL_PCI_P2PDMA to > >> allow obtaining P2PDMA pages. If GUP is called without the flag and a > >> P2PDMA page is found, it will return an error. > >> > >> FOLL_PCI_P2PDMA cannot be set if FOLL_LONGTERM is set. > > > > What is causing this? It is really troublesome, I would like to fix > > it. eg I would like to have P2PDMA pages in VFIO iommu page tables and > > in RDMA MR's - both require longterm. > > You had said it was required if we were relying on unmap_mapping_range()... Ah.. Ok. Dan and I have been talking about this a lot, and it turns out the DAX approach of unmap_mapping_range() still has problems, really the same problem as FOLL_LONGTERM: https://lore.kernel.org/all/Yy2pC%2FupZNEkVmc5@nvidia.com/ ie nothing actually waits for the page refs to go to zero during memunmap_pages(). (indeed they are not actually zero because currently they are instantly reset to 1 if they become zero) The current design requires that the pgmap user hold the pgmap_ref in a way that it remains elevated until page_free() is called for every page that was ever used. I'm encouraging Dan to work on better infrastructure in pgmap core because every pgmap implementation has this issue currently. For that reason it is probably not so relavent to this series. Perhaps just clarify in the commit message that the FOLL_LONGTERM restriction is to copy DAX until the pgmap page refcounts are fixed. > > Is it just because ZONE_DEVICE was created for DAX and carried that > > revocable assumption over? Does anything in your series require > > revocable? > > We still rely on unmap_mapping_range() indirectly in the unbind > path. So I expect if something takes a LONGERM mapping that would > block until whatever process holds the pin releases it. That's less > than ideal and I'm not sure what can be done about it. We could improve the blocking with some kind of FOLL_LONGTERM notifier thingy eg after the unmap_mapping_rage() broadcast that a range of PFNs is going away and FOLL_LONGTERM users can do a revoke if they support it. It is a rare enough we don't necessarily need to optimize this alot, and blocking unbind until some FDs close is annoying not critical.. (eg you already can't unmount a filesystem to unbind the device on the nvme while FS FDs are open) Jason