From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 822F2C433EF for ; Thu, 2 Jun 2022 00:00:47 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232617AbiFBAAp (ORCPT ); Wed, 1 Jun 2022 20:00:45 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:55006 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232606AbiFBAAm (ORCPT ); Wed, 1 Jun 2022 20:00:42 -0400 Received: from mail-qt1-x836.google.com (mail-qt1-x836.google.com [IPv6:2607:f8b0:4864:20::836]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 30A752941D5 for ; Wed, 1 Jun 2022 17:00:41 -0700 (PDT) Received: by mail-qt1-x836.google.com with SMTP id v29so2367232qtc.12 for ; Wed, 01 Jun 2022 17:00:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=XHclFl6TiOnpsdbRTbpf8I5ef93NwLHxOT7ZftAP5OQ=; b=PNI9gR5OxCMLK2wesmPP7beTza1LVUIA9KonYJGbDAFowaWR1+gOMSPFQyILlCbdK9 gUfn5ltQQND4Igpr1PcoJgKaxcD5dBAwzrvhneuGmR5cB6Xf3NpyEwE9pIIgcm5WsDfj OMBTPHZn69Csi4sdfYc+o9dQdNtlWVyz5OxRR+X8nyDPYF6K2ahO4BRlxiavJkdgkk5b vBCAgX831wn8MMIkjC+ZBM62WCGcevuCk6vOX7Pr2o1vHeySgvRlg6SFXHlD9BjBBgRR tgzeldcoyTnuS2Xyrkl/LEo9UbgD40oyCdtA/wA5nQPC/hTnaLXrjoV5QjIEjd2lSEr7 qhiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=XHclFl6TiOnpsdbRTbpf8I5ef93NwLHxOT7ZftAP5OQ=; b=d9P1HiPtMEj8y41Ib4WvcJSdaRYa6cs3dve/XFt4e0FlCQK/OgCFBjkX0Q/u8Rt//f Vygn8huAnmsQJ9o9hRg+RirDUUWDKBHr5z/bQouONzrXPfo7pZxyiswQ39vmvjMnNhLe KKaj6xpF/xA2esD5rsKh4wDE5ERTdGeVKr/dcO4XQ7pd/YboiTQBmfyzI2Kz9L1slsdd oljLmX/BOGGz0HgZU73xeomrxdH7RtncyHH6Cunqq0fmVmBrBdasJEIc2uRAWMDyDYMZ ENMcNMX3FiUEeZMsqefvYPdEecvWoDCDRhtpOXaYswWEEF+l8vsFHFIXE1iZdJfFc4Ed AcOg== X-Gm-Message-State: AOAM532sg9VydXoJFv13+aEaWE7irOoVO80VGF7Lu+YGRs8BYMS8riVM WIf6icZjPwnYpqqryVTeBQzE+w== X-Google-Smtp-Source: ABdhPJyvPa7jsnbpw9EbFCPQVbuxZnWhMX7Sx5D9T0fweaGvqMgcaf7CabbpUfVwopaFMLH+hIknjw== X-Received: by 2002:a05:622a:313:b0:2f3:b9a1:dd4a with SMTP id q19-20020a05622a031300b002f3b9a1dd4amr1870963qtw.536.1654128040009; Wed, 01 Jun 2022 17:00:40 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-162-113-129.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.113.129]) by smtp.gmail.com with ESMTPSA id b16-20020a05620a04f000b006a38debe62csm1867208qkh.89.2022.06.01.17.00.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Jun 2022 17:00:39 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1nwYGU-00GCY4-Af; Wed, 01 Jun 2022 21:00:38 -0300 Date: Wed, 1 Jun 2022 21:00:38 -0300 From: Jason Gunthorpe To: Logan Gunthorpe Cc: linux-kernel@vger.kernel.org, linux-nvme@lists.infradead.org, linux-block@vger.kernel.org, linux-pci@vger.kernel.org, linux-mm@kvack.org, iommu@lists.linux-foundation.org, Stephen Bates , Christoph Hellwig , Dan Williams , Christian =?utf-8?B?S8O2bmln?= , John Hubbard , Don Dutile , Matthew Wilcox , Daniel Vetter , Jakowski Andrzej , Minturn Dave B , Jason Ekstrand , Dave Hansen , Xiong Jianxin , Bjorn Helgaas , Ira Weiny , Robin Murphy , Martin Oliveira , Chaitanya Kulkarni , Ralph Campbell , Bjorn Helgaas Subject: Re: [PATCH v6 20/21] PCI/P2PDMA: Introduce pci_mmap_p2pmem() Message-ID: <20220602000038.GK2960187@ziepe.ca> References: <20220407154717.7695-1-logang@deltatee.com> <20220407154717.7695-21-logang@deltatee.com> <20220527125501.GD2960187@ziepe.ca> <20220527190307.GG2960187@ziepe.ca> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org On Fri, May 27, 2022 at 04:41:08PM -0600, Logan Gunthorpe wrote: > > > > IIRC this is the last part: > > > > https://lore.kernel.org/linux-mm/20220524190632.3304-1-alex.sierra@amd.com/ > > > > And the earlier bit with Christoph's pieces looks like it might get > > merged to v5.19.. > > > > The general idea is once pte_devmap is not set then all the > > refcounting works the way it should. This is what all new ZONE_DEVICE > > users should do.. > > Ok, I don't actually follow how those patches relate to this. > > Based on your description I guess I don't need to set PFN_DEV and Yes > perhaps not use vmf_insert_mixed()? And then just use vm_normal_page()? I'm not sure ATM the best function to use, but yes, a function that doesn't set PFN_DEV is needed here. > But the refcounting of the pages seemed like it was already sane to me, > unless you mean that the code no longer has to synchronize_rcu() before > returning the pages... Right. It also doesn't need to call unmap range or keep track of the inode, or do any of that stuff unless it really needs mmap revokation semantics (which I doubt this use case does) unmap range was only necessary because the refcounting is wrong - since the pte's don't hold a ref on the page in PFN_DEV mode it is necessary to wipe all the PTE explicitly before going ahead to decrement the refcount on this path. Just stuff the pages into the mmap, and your driver unprobe will automatically block until all the mmaps are closed - no different than having an open file descriptor or something. Jason From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from smtp4.osuosl.org (smtp4.osuosl.org [140.211.166.137]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6D96EC43334 for ; Thu, 2 Jun 2022 00:00:47 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp4.osuosl.org (Postfix) with ESMTP id D7FEE417BB; Thu, 2 Jun 2022 00:00:46 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from smtp4.osuosl.org ([127.0.0.1]) by localhost (smtp4.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id Ex1IKhWeSFts; Thu, 2 Jun 2022 00:00:45 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by smtp4.osuosl.org (Postfix) with ESMTPS id 7025D4174F; Thu, 2 Jun 2022 00:00:45 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 4E299C0039; Thu, 2 Jun 2022 00:00:45 +0000 (UTC) Received: from smtp2.osuosl.org (smtp2.osuosl.org [IPv6:2605:bc80:3010::133]) by lists.linuxfoundation.org (Postfix) with ESMTP id BB418C002D for ; Thu, 2 Jun 2022 00:00:42 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp2.osuosl.org (Postfix) with ESMTP id A22CE403EA for ; Thu, 2 Jun 2022 00:00:42 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Authentication-Results: smtp2.osuosl.org (amavisd-new); dkim=pass (2048-bit key) header.d=ziepe.ca Received: from smtp2.osuosl.org ([127.0.0.1]) by localhost (smtp2.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tqmUYjCqejx0 for ; Thu, 2 Jun 2022 00:00:41 +0000 (UTC) X-Greylist: whitelisted by SQLgrey-1.8.0 Received: from mail-qt1-x834.google.com (mail-qt1-x834.google.com [IPv6:2607:f8b0:4864:20::834]) by smtp2.osuosl.org (Postfix) with ESMTPS id 7C146401FB for ; Thu, 2 Jun 2022 00:00:41 +0000 (UTC) Received: by mail-qt1-x834.google.com with SMTP id hh4so2371719qtb.10 for ; Wed, 01 Jun 2022 17:00:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=ziepe.ca; s=google; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=XHclFl6TiOnpsdbRTbpf8I5ef93NwLHxOT7ZftAP5OQ=; b=PNI9gR5OxCMLK2wesmPP7beTza1LVUIA9KonYJGbDAFowaWR1+gOMSPFQyILlCbdK9 gUfn5ltQQND4Igpr1PcoJgKaxcD5dBAwzrvhneuGmR5cB6Xf3NpyEwE9pIIgcm5WsDfj OMBTPHZn69Csi4sdfYc+o9dQdNtlWVyz5OxRR+X8nyDPYF6K2ahO4BRlxiavJkdgkk5b vBCAgX831wn8MMIkjC+ZBM62WCGcevuCk6vOX7Pr2o1vHeySgvRlg6SFXHlD9BjBBgRR tgzeldcoyTnuS2Xyrkl/LEo9UbgD40oyCdtA/wA5nQPC/hTnaLXrjoV5QjIEjd2lSEr7 qhiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=XHclFl6TiOnpsdbRTbpf8I5ef93NwLHxOT7ZftAP5OQ=; b=inSC5rbRoQ+FZBdJXYHCdDqGLVXTMFOshNsDScnlNeT6yvbhSJRZPSXFJuhT8car8Z qedFyC3KK/ml0wfjQQG/AuvoWZvxHh+CsBrgz4wmFHIuF9PELMh2Iw9P/metrvIZkgeK f4sx4cfP00mby44+dEQKBR2GYTILAkf2SEXqxEhcqbtFCsV36RcJCDYs90fm5q8VE4uC MxprosB1hIDcxa1KS6n2B45TgoXS0y5lV01e0CFvmLyUhtnj47vGStYH//+FjMnJTDZP KWVj6r9z6ErLekwJHUAsQJu6RNqLdVucBsmBQwWVMfeO8il/g16dtyA5w35TMW1o8544 HkrQ== X-Gm-Message-State: AOAM533jEhHL7Q9lT/JTIPYN3JQTOUAvxKZU61oy0BD5eEpZepSWlgQB beA91Mws4eak5CB2cnSgLr7ZEg== X-Google-Smtp-Source: ABdhPJyvPa7jsnbpw9EbFCPQVbuxZnWhMX7Sx5D9T0fweaGvqMgcaf7CabbpUfVwopaFMLH+hIknjw== X-Received: by 2002:a05:622a:313:b0:2f3:b9a1:dd4a with SMTP id q19-20020a05622a031300b002f3b9a1dd4amr1870963qtw.536.1654128040009; Wed, 01 Jun 2022 17:00:40 -0700 (PDT) Received: from ziepe.ca (hlfxns017vw-142-162-113-129.dhcp-dynamic.fibreop.ns.bellaliant.net. [142.162.113.129]) by smtp.gmail.com with ESMTPSA id b16-20020a05620a04f000b006a38debe62csm1867208qkh.89.2022.06.01.17.00.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 01 Jun 2022 17:00:39 -0700 (PDT) Received: from jgg by mlx with local (Exim 4.94) (envelope-from ) id 1nwYGU-00GCY4-Af; Wed, 01 Jun 2022 21:00:38 -0300 Date: Wed, 1 Jun 2022 21:00:38 -0300 From: Jason Gunthorpe To: Logan Gunthorpe Subject: Re: [PATCH v6 20/21] PCI/P2PDMA: Introduce pci_mmap_p2pmem() Message-ID: <20220602000038.GK2960187@ziepe.ca> References: <20220407154717.7695-1-logang@deltatee.com> <20220407154717.7695-21-logang@deltatee.com> <20220527125501.GD2960187@ziepe.ca> <20220527190307.GG2960187@ziepe.ca> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Cc: linux-pci@vger.kernel.org, Dave Hansen , linux-nvme@lists.infradead.org, Stephen Bates , linux-mm@kvack.org, Jason Ekstrand , Ira Weiny , Christoph Hellwig , Minturn Dave B , Martin Oliveira , Matthew Wilcox , Chaitanya Kulkarni , Bjorn Helgaas , Daniel Vetter , Ralph Campbell , John Hubbard , linux-block@vger.kernel.org, Bjorn Helgaas , Dan Williams , Jakowski Andrzej , Xiong Jianxin , linux-kernel@vger.kernel.org, iommu@lists.linux-foundation.org, Robin Murphy , Christian =?utf-8?B?S8O2bmln?= X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" On Fri, May 27, 2022 at 04:41:08PM -0600, Logan Gunthorpe wrote: > > > > IIRC this is the last part: > > > > https://lore.kernel.org/linux-mm/20220524190632.3304-1-alex.sierra@amd.com/ > > > > And the earlier bit with Christoph's pieces looks like it might get > > merged to v5.19.. > > > > The general idea is once pte_devmap is not set then all the > > refcounting works the way it should. This is what all new ZONE_DEVICE > > users should do.. > > Ok, I don't actually follow how those patches relate to this. > > Based on your description I guess I don't need to set PFN_DEV and Yes > perhaps not use vmf_insert_mixed()? And then just use vm_normal_page()? I'm not sure ATM the best function to use, but yes, a function that doesn't set PFN_DEV is needed here. > But the refcounting of the pages seemed like it was already sane to me, > unless you mean that the code no longer has to synchronize_rcu() before > returning the pages... Right. It also doesn't need to call unmap range or keep track of the inode, or do any of that stuff unless it really needs mmap revokation semantics (which I doubt this use case does) unmap range was only necessary because the refcounting is wrong - since the pte's don't hold a ref on the page in PFN_DEV mode it is necessary to wipe all the PTE explicitly before going ahead to decrement the refcount on this path. Just stuff the pages into the mmap, and your driver unprobe will automatically block until all the mmaps are closed - no different than having an open file descriptor or something. Jason _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu