From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1F9B7C77B6E for ; Wed, 12 Apr 2023 12:28:17 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A903B900003; Wed, 12 Apr 2023 08:28:16 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id A3ED3900002; Wed, 12 Apr 2023 08:28:16 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 90894900003; Wed, 12 Apr 2023 08:28:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 82999900002 for ; Wed, 12 Apr 2023 08:28:16 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 55E221A0117 for ; Wed, 12 Apr 2023 12:28:16 +0000 (UTC) X-FDA: 80672666592.07.8841369 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf02.hostedemail.com (Postfix) with ESMTP id 6DA7C80004 for ; Wed, 12 Apr 2023 12:28:13 +0000 (UTC) Authentication-Results: imf02.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Z5Iteany; spf=pass (imf02.hostedemail.com: domain of maz@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=maz@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1681302493; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mlxm1AajJpAnXSJdhFgSvMQ+1gIA4yVtwMmFnWpqwwg=; b=TdIem/vGf+7iipWF6qNIXGC5q1lhv5Pae85s2WBjQ7D4q2YdnsAVXhDCn0K0CEW7IXdZ57 XuhATF9OIxAMPSGe3PIRtlnV1MroZgUFfi/4O2QZgU3FSeNt/EnR66pPLR3NGAOh/ljU3z JUwgPYS/LzaL6JDMSGIRgniCpfp/68k= ARC-Authentication-Results: i=1; imf02.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=Z5Iteany; spf=pass (imf02.hostedemail.com: domain of maz@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=maz@kernel.org; dmarc=pass (policy=none) header.from=kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1681302493; a=rsa-sha256; cv=none; b=uhipXDyl9TBa115lTNRnsfuIstIRKtzVjYt4Du7vkw1QmFI85lu63/RP8G3tm47v5dgPCb NkRTmyhIRts6Mav7mOpSJ39UYsHUcrRgTX8iInex7RXiDbmP3RUstsTqEamEwfCXvRqsON SEExHjHb5yRc1X6Cj4pUlgvelu19lKc= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3A4F662950; Wed, 12 Apr 2023 12:28:12 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 9AFD2C4339B; Wed, 12 Apr 2023 12:28:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1681302491; bh=6FoH81ZoljOnwwsVDKczs8ze1AD1h4N7YdQq49DUJCI=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=Z5IteanyktRgqXMBi2ojlqQXm8aO8qY7wuKa+kxHoEdUI1Pv9v2nMdYQmddGty2oP +kzfgwhsEDdamBBntfgPzpUuvQetCM6y5bYI5SBFZRf2EOuvI5y6dOPFm3Q8EbbcPi UoVKdcc08m7tw6v0HicScjoBxoZ8lMFZTyiE5t4o2vNgk4S2SaFRkDji8JZ/Jx75eN rOSw5e+BYDUBg2JSz+lv+Jcb0vFkeICU9BKzr7dOzNrQgRpLyBMNfGrq7FdLYIwamI AOyrFmM9wzyKCo4TKDfpuAi8AtM8tDq7/x48T7k02VPExTTHCbndcSrjI2FnTEQ9Cg aIfjWNIYYuC3Q== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1pmZa5-007pOX-7J; Wed, 12 Apr 2023 13:28:09 +0100 Date: Wed, 12 Apr 2023 13:28:08 +0100 Message-ID: <86sfd5l1yf.wl-maz@kernel.org> From: Marc Zyngier To: Cc: , , , , , , , , , , , , , , , , Subject: Re: [PATCH v3 0/6] Expose GPU memory as coherently CPU accessible In-Reply-To: <20230405180134.16932-1-ankita@nvidia.com> References: <20230405180134.16932-1-ankita@nvidia.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/28.2 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: ankita@nvidia.com, jgg@nvidia.com, alex.williamson@redhat.com, naoya.horiguchi@nec.com, oliver.upton@linux.dev, aniketa@nvidia.com, cjia@nvidia.com, kwankhede@nvidia.com, targupta@nvidia.com, vsethi@nvidia.com, acurrid@nvidia.com, apopple@nvidia.com, jhubbard@nvidia.com, danw@nvidia.com, kvm@vger.kernel.org, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 6DA7C80004 X-Stat-Signature: 4t1m5gnx5bsrc5agajuzaccoeidp3677 X-Rspam-User: X-HE-Tag: 1681302493-540115 X-HE-Meta: U2FsdGVkX1/jyyR/DpWE6pDZk5Uup8B3ROXfoZXnI8ZNU+cI5pSEleP3thU1RDh7Pc11euKmQ08gFTjbHj0ULGGE6IVva1gt9hLi7tTSobMYrnFNOSxX7NJm9rpg1/bPG8e3HjZq3xmRBxJ6b+3Ae4YhuupjQ3Hx0ze6+n485H20WJc8bTA/PS+nbDJ6fd85KZZFjm6K/ajtDaaruTfJh03FVOGWfnkxuakqfc+F1mjO05YkQygZeR9PRe8qCniKhCl/Gif5IazNshds3Hlpf/wOQMFgtxd8Qvr8A5wGRkSQDWO0ehPCD9D0Qe1esd7llPsNGL9EzrDFlr8yRDZbXKgLI+UlGNzvxP8CRTpoOo5X5xKuMWIUOjUe1QW3xv2HPrw5e/oGIIOlybYw7ObkDSrtHl5t9SpEerm803Ry0rMKatgEhX9T3qDxSemPT0Srral3t33xEgLMviewypw9Ym1GCbi5/XYoxNIEIIUxrCILwSJfTF4MVyRqOBUT8DqCWjxc0rmqOQtvltTj/VKXPLL54Ie6kk8lYHXniuJU5qdlMT3mFl+UqOS91XCaRcK5OX5YLL2sk+WyGwLKvjqQ0OZ1nNI8QJ5HbzBl0CQ79srD+pyMCxh3qB0R0rHNhmj0zDR6SAh1uyIWFTanwmePtk5X2fuslbAU2mjgGuyZruwgJ9VtDXVAyX6qzRrl5sL5OSfXtUGb4Uvfi774rSB9ERzEDn5JcYwnd46B4mSpdJ98MWaZByitOvHhgZeO/WfqqZU+cnnUrs9BPIPWQpw1t4KHBw7lZ024xRdanW8b/IR5jeDLu9Xm01ObiPW8MWd3AOVc6NBGqj6Nb9+693atNsro1MXFK+FrJBIQzfAVdlI9txqw0zvQJMsNSZjKihWxZCtZIRMWLO5yg+/9MUeTucuiMe57WOn85/59K1Bd864AWJPPAH+/4LxloyZ/Bedk7yAvsTGSBs59hUDgL2R gcY+ImfG BcmM+xN3j4PITT3WokA8wUNLTYGhoBpbPNFiDo8PaNSlqaLwE3prK/yLpFuVase1nlkOa9Jh1CbxS1H0cdCvlxwxVCL9Sg2cJ+AFcc06eM1XE1dYtTY+5Vvpah7evEa3BZYTZRdEpgLtJURmS8w7RHBLV5YZo3oVZSB9TrsN50938DvG0r63ccQKr2hXiJjT/w5Kpwrd276ZyZMxVb9eQuzYonX5YiLEEUoQPlfjfe4Ltm+AtZ3YGi2Xv5wDsjukgmzp1SdvB7A+pjyWDd362Ef5WmFnWv3oeQO23Gyk+TXXlSsouNgqmM9SOuDsRjm3Kx/HfVMLw46My+eQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, 05 Apr 2023 19:01:28 +0100, wrote: > > From: Ankit Agrawal > > NVIDIA's upcoming Grace Hopper Superchip provides a PCI-like device > for the on-chip GPU that is the logical OS representation of the > internal propritary cache coherent interconnect. > > This representation has a number of limitations compared to a real PCI > device, in particular, it does not model the coherent GPU memory > aperture as a PCI config space BAR, and PCI doesn't know anything > about cacheable memory types. > > Provide a VFIO PCI variant driver that adapts the unique PCI > representation into a more standard PCI representation facing > userspace. The GPU memory aperture is obtained from ACPI, according to > the FW specification, and exported to userspace as the VFIO_REGION > that covers the first PCI BAR. qemu will naturally generate a PCI > device in the VM where the cacheable aperture is reported in BAR1. > > Since this memory region is actually cache coherent with the CPU, the > VFIO variant driver will mmap it into VMA using a cacheable mapping. > > As this is the first time an ARM environment has placed cacheable > non-struct page backed memory (eg from remap_pfn_range) into a KVM > page table, fix a bug in ARM KVM where it does not copy the cacheable > memory attributes from non-struct page backed PTEs to ensure the guest > also gets a cacheable mapping. This is not a bug, but a conscious design decision. As you pointed out above, nothing needed this until now, and a device mapping is the only safe thing to do as we know exactly *nothing* about the memory that gets mapped. M. -- Without deviation from the norm, progress is not possible.