From: Will Deacon <will@kernel.org> To: Lorenzo Pieralisi <lpieralisi@kernel.org> Cc: Catalin Marinas <catalin.marinas@arm.com>, Jason Gunthorpe <jgg@nvidia.com>, ankita@nvidia.com, maz@kernel.org, oliver.upton@linux.dev, aniketa@nvidia.com, cjia@nvidia.com, kwankhede@nvidia.com, targupta@nvidia.com, vsethi@nvidia.com, acurrid@nvidia.com, apopple@nvidia.com, jhubbard@nvidia.com, danw@nvidia.com, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org Subject: Re: [PATCH v1 2/2] KVM: arm64: allow the VM to select DEVICE_* and NORMAL_NC for IO memory Date: Thu, 12 Oct 2023 13:35:41 +0100 [thread overview] Message-ID: <20231012123541.GB11824@willie-the-truck> (raw) In-Reply-To: <ZR6IZwcFNw55asW0@lpieralisi> On Thu, Oct 05, 2023 at 11:56:55AM +0200, Lorenzo Pieralisi wrote: > On Tue, Sep 26, 2023 at 02:52:13PM +0100, Catalin Marinas wrote: > > [...] > > > Anyway, the text looks fine to me. Thanks for putting it together > > Lorenzo. > > Thanks ! > > > One thing not mentioned here is that pci-vfio still maps such memory as > > Device-nGnRnE in user space and relaxing this potentially creates an > > alias. But such alias is only relevant of both the VMM and the VM try to > > access the same device which I doubt is a realistic scenario. > > A revised log, FWIW: Thanks for putting this together, Lorenzo. Just one thing below: > --- > Currently, KVM for ARM64 maps at stage 2 memory that is > considered device (ie it is not RAM) with DEVICE_nGnRE > memory attributes; this setting overrides (as per the ARM > architecture [1]) any device MMIO mapping present at stage > 1, resulting in a set-up whereby a guest operating system > can't determine device MMIO mapping memory attributes on its > own but it is always overriden by the KVM stage 2 default. > > This set-up does not allow guest operating systems to select > device memory attributes on a page by page basis independently > from KVM stage-2 mappings (refer to [1], "Combining stage 1 and stage > 2 memory type attributes"), which turns out to be an issue in that > guest operating systems (eg Linux) may request to map > devices MMIO regions with memory attributes that guarantee > better performance (eg gathering attribute - that for some > devices can generate larger PCIe memory writes TLPs) > and specific operations (eg unaligned transactions) such as > the NormalNC memory type. > > The default device stage 2 mapping was chosen in KVM > for ARM64 since it was considered safer (ie it would > not allow guests to trigger uncontained failures > ultimately crashing the machine) but this turned out > to be imprecise. > > Failures containability is a property of the platform > and is independent from the memory type used for MMIO > device memory mappings (ie DEVICE_nGnRE memory type is > even more problematic than NormalNC in terms of containability > since eg aborts triggered on loads cannot be made synchronous, > which make them harder to contain); this means that, > regardless of the combined stage1+stage2 mappings a > platform is safe if and only if device transactions cannot trigger > uncontained failures; reworded, the default KVM device > stage 2 memory attributes play no role in making device > assignment safer for a given platform and therefore can > be relaxed. > > For all these reasons, relax the KVM stage 2 device > memory attributes from DEVICE_nGnRE to NormalNC. The reasoning above suggests to me that this should probably just be Normal cacheable, as that is what actually allows the guest to control the attributes. So what is the rationale behind stopping at Normal-NC? Will
WARNING: multiple messages have this Message-ID (diff)
From: Will Deacon <will@kernel.org> To: Lorenzo Pieralisi <lpieralisi@kernel.org> Cc: Catalin Marinas <catalin.marinas@arm.com>, Jason Gunthorpe <jgg@nvidia.com>, ankita@nvidia.com, maz@kernel.org, oliver.upton@linux.dev, aniketa@nvidia.com, cjia@nvidia.com, kwankhede@nvidia.com, targupta@nvidia.com, vsethi@nvidia.com, acurrid@nvidia.com, apopple@nvidia.com, jhubbard@nvidia.com, danw@nvidia.com, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, linux-kernel@vger.kernel.org Subject: Re: [PATCH v1 2/2] KVM: arm64: allow the VM to select DEVICE_* and NORMAL_NC for IO memory Date: Thu, 12 Oct 2023 13:35:41 +0100 [thread overview] Message-ID: <20231012123541.GB11824@willie-the-truck> (raw) In-Reply-To: <ZR6IZwcFNw55asW0@lpieralisi> On Thu, Oct 05, 2023 at 11:56:55AM +0200, Lorenzo Pieralisi wrote: > On Tue, Sep 26, 2023 at 02:52:13PM +0100, Catalin Marinas wrote: > > [...] > > > Anyway, the text looks fine to me. Thanks for putting it together > > Lorenzo. > > Thanks ! > > > One thing not mentioned here is that pci-vfio still maps such memory as > > Device-nGnRnE in user space and relaxing this potentially creates an > > alias. But such alias is only relevant of both the VMM and the VM try to > > access the same device which I doubt is a realistic scenario. > > A revised log, FWIW: Thanks for putting this together, Lorenzo. Just one thing below: > --- > Currently, KVM for ARM64 maps at stage 2 memory that is > considered device (ie it is not RAM) with DEVICE_nGnRE > memory attributes; this setting overrides (as per the ARM > architecture [1]) any device MMIO mapping present at stage > 1, resulting in a set-up whereby a guest operating system > can't determine device MMIO mapping memory attributes on its > own but it is always overriden by the KVM stage 2 default. > > This set-up does not allow guest operating systems to select > device memory attributes on a page by page basis independently > from KVM stage-2 mappings (refer to [1], "Combining stage 1 and stage > 2 memory type attributes"), which turns out to be an issue in that > guest operating systems (eg Linux) may request to map > devices MMIO regions with memory attributes that guarantee > better performance (eg gathering attribute - that for some > devices can generate larger PCIe memory writes TLPs) > and specific operations (eg unaligned transactions) such as > the NormalNC memory type. > > The default device stage 2 mapping was chosen in KVM > for ARM64 since it was considered safer (ie it would > not allow guests to trigger uncontained failures > ultimately crashing the machine) but this turned out > to be imprecise. > > Failures containability is a property of the platform > and is independent from the memory type used for MMIO > device memory mappings (ie DEVICE_nGnRE memory type is > even more problematic than NormalNC in terms of containability > since eg aborts triggered on loads cannot be made synchronous, > which make them harder to contain); this means that, > regardless of the combined stage1+stage2 mappings a > platform is safe if and only if device transactions cannot trigger > uncontained failures; reworded, the default KVM device > stage 2 memory attributes play no role in making device > assignment safer for a given platform and therefore can > be relaxed. > > For all these reasons, relax the KVM stage 2 device > memory attributes from DEVICE_nGnRE to NormalNC. The reasoning above suggests to me that this should probably just be Normal cacheable, as that is what actually allows the guest to control the attributes. So what is the rationale behind stopping at Normal-NC? Will _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2023-10-12 12:35 UTC|newest] Thread overview: 110+ messages / expand[flat|nested] mbox.gz Atom feed top 2023-09-07 18:14 [PATCH v1 0/2] KVM: arm64: support write combining and cachable IO memory in VMs ankita 2023-09-07 18:14 ` ankita 2023-09-07 18:14 ` [PATCH v1 1/2] KVM: arm64: determine memory type from VMA ankita 2023-09-07 18:14 ` ankita 2023-09-07 19:12 ` Jason Gunthorpe 2023-09-07 19:12 ` Jason Gunthorpe 2023-10-05 16:15 ` Catalin Marinas 2023-10-05 16:15 ` Catalin Marinas 2023-10-05 16:54 ` Jason Gunthorpe 2023-10-05 16:54 ` Jason Gunthorpe 2023-10-10 14:25 ` Catalin Marinas 2023-10-10 14:25 ` Catalin Marinas 2023-10-10 15:05 ` Jason Gunthorpe 2023-10-10 15:05 ` Jason Gunthorpe 2023-10-10 17:19 ` Catalin Marinas 2023-10-10 17:19 ` Catalin Marinas 2023-10-10 18:23 ` Jason Gunthorpe 2023-10-10 18:23 ` Jason Gunthorpe 2023-10-11 17:45 ` Catalin Marinas 2023-10-11 17:45 ` Catalin Marinas 2023-10-11 18:38 ` Jason Gunthorpe 2023-10-11 18:38 ` Jason Gunthorpe 2023-10-12 16:16 ` Catalin Marinas 2023-10-12 16:16 ` Catalin Marinas 2024-03-10 3:49 ` Ankit Agrawal 2024-03-10 3:49 ` Ankit Agrawal 2024-03-19 13:38 ` Jason Gunthorpe 2024-03-19 13:38 ` Jason Gunthorpe 2023-10-23 13:20 ` Shameerali Kolothum Thodi 2023-10-23 13:20 ` Shameerali Kolothum Thodi 2023-09-07 18:14 ` [PATCH v1 2/2] KVM: arm64: allow the VM to select DEVICE_* and NORMAL_NC for IO memory ankita 2023-09-07 18:14 ` ankita 2023-09-08 16:40 ` Catalin Marinas 2023-09-08 16:40 ` Catalin Marinas 2023-09-11 14:57 ` Lorenzo Pieralisi 2023-09-11 14:57 ` Lorenzo Pieralisi 2023-09-11 17:20 ` Jason Gunthorpe 2023-09-11 17:20 ` Jason Gunthorpe 2023-09-13 15:26 ` Lorenzo Pieralisi 2023-09-13 15:26 ` Lorenzo Pieralisi 2023-09-13 18:54 ` Jason Gunthorpe 2023-09-13 18:54 ` Jason Gunthorpe 2023-09-26 8:31 ` Lorenzo Pieralisi 2023-09-26 8:31 ` Lorenzo Pieralisi 2023-09-26 12:25 ` Jason Gunthorpe 2023-09-26 12:25 ` Jason Gunthorpe 2023-09-26 13:52 ` Catalin Marinas 2023-09-26 13:52 ` Catalin Marinas 2023-09-26 16:12 ` Lorenzo Pieralisi 2023-09-26 16:12 ` Lorenzo Pieralisi 2023-10-05 9:56 ` Lorenzo Pieralisi 2023-10-05 9:56 ` Lorenzo Pieralisi 2023-10-05 11:56 ` Jason Gunthorpe 2023-10-05 11:56 ` Jason Gunthorpe 2023-10-05 14:08 ` Lorenzo Pieralisi 2023-10-05 14:08 ` Lorenzo Pieralisi 2023-10-12 12:35 ` Will Deacon [this message] 2023-10-12 12:35 ` Will Deacon 2023-10-12 13:20 ` Jason Gunthorpe 2023-10-12 13:20 ` Jason Gunthorpe 2023-10-12 14:29 ` Lorenzo Pieralisi 2023-10-12 14:29 ` Lorenzo Pieralisi 2023-10-12 13:53 ` Catalin Marinas 2023-10-12 13:53 ` Catalin Marinas 2023-10-12 14:48 ` Will Deacon 2023-10-12 14:48 ` Will Deacon 2023-10-12 15:44 ` Jason Gunthorpe 2023-10-12 15:44 ` Jason Gunthorpe 2023-10-12 16:39 ` Will Deacon 2023-10-12 16:39 ` Will Deacon 2023-10-12 18:36 ` Jason Gunthorpe 2023-10-12 18:36 ` Jason Gunthorpe 2023-10-13 9:29 ` Will Deacon 2023-10-13 9:29 ` Will Deacon 2023-10-12 17:26 ` Catalin Marinas 2023-10-12 17:26 ` Catalin Marinas 2023-10-13 9:29 ` Will Deacon 2023-10-13 9:29 ` Will Deacon 2023-10-13 13:08 ` Catalin Marinas 2023-10-13 13:08 ` Catalin Marinas 2023-10-13 13:45 ` Jason Gunthorpe 2023-10-13 13:45 ` Jason Gunthorpe 2023-10-19 11:07 ` Catalin Marinas 2023-10-19 11:07 ` Catalin Marinas 2023-10-19 11:51 ` Jason Gunthorpe 2023-10-19 11:51 ` Jason Gunthorpe 2023-10-20 11:21 ` Catalin Marinas 2023-10-20 11:21 ` Catalin Marinas 2023-10-20 11:47 ` Jason Gunthorpe 2023-10-20 11:47 ` Jason Gunthorpe 2023-10-20 14:03 ` Lorenzo Pieralisi 2023-10-20 14:03 ` Lorenzo Pieralisi 2023-10-20 14:28 ` Jason Gunthorpe 2023-10-20 14:28 ` Jason Gunthorpe 2023-10-19 13:35 ` Lorenzo Pieralisi 2023-10-19 13:35 ` Lorenzo Pieralisi 2023-10-13 15:28 ` Lorenzo Pieralisi 2023-10-13 15:28 ` Lorenzo Pieralisi 2023-10-19 11:12 ` Catalin Marinas 2023-10-19 11:12 ` Catalin Marinas 2023-11-09 15:34 ` Lorenzo Pieralisi 2023-11-09 15:34 ` Lorenzo Pieralisi 2023-11-10 14:26 ` Jason Gunthorpe 2023-11-10 14:26 ` Jason Gunthorpe 2023-11-13 0:42 ` Lorenzo Pieralisi 2023-11-13 0:42 ` Lorenzo Pieralisi 2023-11-13 17:41 ` Catalin Marinas 2023-11-13 17:41 ` Catalin Marinas 2023-10-12 12:27 ` Will Deacon 2023-10-12 12:27 ` Will Deacon
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20231012123541.GB11824@willie-the-truck \ --to=will@kernel.org \ --cc=acurrid@nvidia.com \ --cc=aniketa@nvidia.com \ --cc=ankita@nvidia.com \ --cc=apopple@nvidia.com \ --cc=catalin.marinas@arm.com \ --cc=cjia@nvidia.com \ --cc=danw@nvidia.com \ --cc=jgg@nvidia.com \ --cc=jhubbard@nvidia.com \ --cc=kvmarm@lists.linux.dev \ --cc=kwankhede@nvidia.com \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=lpieralisi@kernel.org \ --cc=maz@kernel.org \ --cc=oliver.upton@linux.dev \ --cc=targupta@nvidia.com \ --cc=vsethi@nvidia.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.