From: <ankita@nvidia.com> To: <ankita@nvidia.com>, <jgg@nvidia.com>, <maz@kernel.org>, <oliver.upton@linux.dev>, <james.morse@arm.com>, <suzuki.poulose@arm.com>, <yuzenghui@huawei.com>, <reinette.chatre@intel.com>, <surenb@google.com>, <stefanha@redhat.com>, <brauner@kernel.org>, <catalin.marinas@arm.com>, <will@kernel.org>, <mark.rutland@arm.com>, <alex.williamson@redhat.com>, <kevin.tian@intel.com>, <yi.l.liu@intel.com>, <ardb@kernel.org>, <akpm@linux-foundation.org>, <andreyknvl@gmail.com>, <wangjinchao@xfusion.com>, <gshan@redhat.com>, <ricarkol@google.com>, <linux-mm@kvack.org>, <lpieralisi@kernel.org>, <rananta@google.com>, <ryan.roberts@arm.com> Cc: <aniketa@nvidia.com>, <cjia@nvidia.com>, <kwankhede@nvidia.com>, <targupta@nvidia.com>, <vsethi@nvidia.com>, <acurrid@nvidia.com>, <apopple@nvidia.com>, <jhubbard@nvidia.com>, <danw@nvidia.com>, <kvmarm@lists.linux.dev>, <mochs@nvidia.com>, <zhiw@nvidia.com>, <kvm@vger.kernel.org>, <linux-kernel@vger.kernel.org>, <linux-arm-kernel@lists.infradead.org> Subject: [PATCH v6 0/4] kvm: arm64: allow the VM to select DEVICE_* and NORMAL_NC for IO memory Date: Thu, 8 Feb 2024 02:16:48 +0530 [thread overview] Message-ID: <20240207204652.22954-1-ankita@nvidia.com> (raw) From: Ankit Agrawal <ankita@nvidia.com> Currently, KVM for ARM64 maps at stage 2 memory that is considered device with DEVICE_nGnRE memory attributes; this setting overrides (per ARM architecture [1]) any device MMIO mapping present at stage 1, resulting in a set-up whereby a guest operating system cannot determine device MMIO mapping memory attributes on its own but it is always overridden by the KVM stage 2 default. This set-up does not allow guest operating systems to select device memory attributes independently from KVM stage-2 mappings (refer to [1], "Combining stage 1 and stage 2 memory type attributes"), which turns out to be an issue in that guest operating systems (e.g. Linux) may request to map devices MMIO regions with memory attributes that guarantee better performance (e.g. gathering attribute - that for some devices can generate larger PCIe memory writes TLPs) and specific operations (e.g. unaligned transactions) such as the NormalNC memory type. The default device stage 2 mapping was chosen in KVM for ARM64 since it was considered safer (i.e. it would not allow guests to trigger uncontained failures ultimately crashing the machine) but this turned out to be asynchronous (SError) defeating the purpose. For these reasons, relax the KVM stage 2 device memory attributes from DEVICE_nGnRE to Normal-NC. Generalizing to other devices may be problematic, however. E.g. GICv2 VCPU interface, which is effectively a shared peripheral, can allow a guest to affect another guest's interrupt distribution. Hence limit the change to VFIO PCI as caution. This is achieved by making the VFIO PCI core module set a flag that is tested by KVM to activate the code. This could be extended to other devices in the future once that is deemed safe. [1] section D8.5 - DDI0487J_a_a-profile_architecture_reference_manual.pdf Applied over v6.8-rc2. History ======= v5 -> v6 - Rebased to v6.8-rc2 v4 -> v5 - Moved the cover letter description text to patch 1/4. - Cleaned up stage2_set_prot_attr() based on Marc Zyngier suggestions. - Moved the mm header file changes to a separate patch. - Rebased to v6.7-rc3. v3 -> v4 - Moved the vfio-pci change to use the VM_VFIO_ALLOW_WC into separate patch. - Added check to warn on the case NORMAL_NC and DEVICE are set simultaneously. - Fixed miscellaneous nitpicks suggested in v3. v2 -> v3 - Added a new patch (and converted to patch series) suggested by Catalin Marinas to ensure the code changes are restricted to VFIO PCI devices. - Introduced VM_VFIO_ALLOW_WC flag for VFIO PCI to communicate with VMM. - Reverted GIC mapping to DEVICE. v1 -> v2 - Updated commit log to the one posted by Lorenzo Pieralisi <lpieralisi@kernel.org> (Thanks!) - Added new flag to represent the NORMAL_NC setting. Updated stage2_set_prot_attr() to handle new flag. v5 Link: https://lore.kernel.org/all/20231221154002.32622-1-ankita@nvidia.com/ Signed-off-by: Ankit Agrawal <ankita@nvidia.com> Suggested-by: Jason Gunthorpe <jgg@nvidia.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Ankit Agrawal (4): kvm: arm64: introduce new flag for non-cacheable IO memory mm: introduce new flag to indicate wc safe kvm: arm64: set io memory s2 pte as normalnc for vfio pci device vfio: convey kvm that the vfio-pci device is wc safe arch/arm64/include/asm/kvm_pgtable.h | 2 ++ arch/arm64/include/asm/memory.h | 2 ++ arch/arm64/kvm/hyp/pgtable.c | 23 ++++++++++++++++++----- arch/arm64/kvm/mmu.c | 18 ++++++++++++++---- drivers/vfio/pci/vfio_pci_core.c | 3 ++- include/linux/mm.h | 14 ++++++++++++++ 6 files changed, 52 insertions(+), 10 deletions(-) -- 2.34.1
WARNING: multiple messages have this Message-ID (diff)
From: <ankita@nvidia.com> To: <ankita@nvidia.com>, <jgg@nvidia.com>, <maz@kernel.org>, <oliver.upton@linux.dev>, <james.morse@arm.com>, <suzuki.poulose@arm.com>, <yuzenghui@huawei.com>, <reinette.chatre@intel.com>, <surenb@google.com>, <stefanha@redhat.com>, <brauner@kernel.org>, <catalin.marinas@arm.com>, <will@kernel.org>, <mark.rutland@arm.com>, <alex.williamson@redhat.com>, <kevin.tian@intel.com>, <yi.l.liu@intel.com>, <ardb@kernel.org>, <akpm@linux-foundation.org>, <andreyknvl@gmail.com>, <wangjinchao@xfusion.com>, <gshan@redhat.com>, <ricarkol@google.com>, <linux-mm@kvack.org>, <lpieralisi@kernel.org>, <rananta@google.com>, <ryan.roberts@arm.com> Cc: <aniketa@nvidia.com>, <cjia@nvidia.com>, <kwankhede@nvidia.com>, <targupta@nvidia.com>, <vsethi@nvidia.com>, <acurrid@nvidia.com>, <apopple@nvidia.com>, <jhubbard@nvidia.com>, <danw@nvidia.com>, <kvmarm@lists.linux.dev>, <mochs@nvidia.com>, <zhiw@nvidia.com>, <kvm@vger.kernel.org>, <linux-kernel@vger.kernel.org>, <linux-arm-kernel@lists.infradead.org> Subject: [PATCH v6 0/4] kvm: arm64: allow the VM to select DEVICE_* and NORMAL_NC for IO memory Date: Thu, 8 Feb 2024 02:16:48 +0530 [thread overview] Message-ID: <20240207204652.22954-1-ankita@nvidia.com> (raw) From: Ankit Agrawal <ankita@nvidia.com> Currently, KVM for ARM64 maps at stage 2 memory that is considered device with DEVICE_nGnRE memory attributes; this setting overrides (per ARM architecture [1]) any device MMIO mapping present at stage 1, resulting in a set-up whereby a guest operating system cannot determine device MMIO mapping memory attributes on its own but it is always overridden by the KVM stage 2 default. This set-up does not allow guest operating systems to select device memory attributes independently from KVM stage-2 mappings (refer to [1], "Combining stage 1 and stage 2 memory type attributes"), which turns out to be an issue in that guest operating systems (e.g. Linux) may request to map devices MMIO regions with memory attributes that guarantee better performance (e.g. gathering attribute - that for some devices can generate larger PCIe memory writes TLPs) and specific operations (e.g. unaligned transactions) such as the NormalNC memory type. The default device stage 2 mapping was chosen in KVM for ARM64 since it was considered safer (i.e. it would not allow guests to trigger uncontained failures ultimately crashing the machine) but this turned out to be asynchronous (SError) defeating the purpose. For these reasons, relax the KVM stage 2 device memory attributes from DEVICE_nGnRE to Normal-NC. Generalizing to other devices may be problematic, however. E.g. GICv2 VCPU interface, which is effectively a shared peripheral, can allow a guest to affect another guest's interrupt distribution. Hence limit the change to VFIO PCI as caution. This is achieved by making the VFIO PCI core module set a flag that is tested by KVM to activate the code. This could be extended to other devices in the future once that is deemed safe. [1] section D8.5 - DDI0487J_a_a-profile_architecture_reference_manual.pdf Applied over v6.8-rc2. History ======= v5 -> v6 - Rebased to v6.8-rc2 v4 -> v5 - Moved the cover letter description text to patch 1/4. - Cleaned up stage2_set_prot_attr() based on Marc Zyngier suggestions. - Moved the mm header file changes to a separate patch. - Rebased to v6.7-rc3. v3 -> v4 - Moved the vfio-pci change to use the VM_VFIO_ALLOW_WC into separate patch. - Added check to warn on the case NORMAL_NC and DEVICE are set simultaneously. - Fixed miscellaneous nitpicks suggested in v3. v2 -> v3 - Added a new patch (and converted to patch series) suggested by Catalin Marinas to ensure the code changes are restricted to VFIO PCI devices. - Introduced VM_VFIO_ALLOW_WC flag for VFIO PCI to communicate with VMM. - Reverted GIC mapping to DEVICE. v1 -> v2 - Updated commit log to the one posted by Lorenzo Pieralisi <lpieralisi@kernel.org> (Thanks!) - Added new flag to represent the NORMAL_NC setting. Updated stage2_set_prot_attr() to handle new flag. v5 Link: https://lore.kernel.org/all/20231221154002.32622-1-ankita@nvidia.com/ Signed-off-by: Ankit Agrawal <ankita@nvidia.com> Suggested-by: Jason Gunthorpe <jgg@nvidia.com> Acked-by: Catalin Marinas <catalin.marinas@arm.com> Ankit Agrawal (4): kvm: arm64: introduce new flag for non-cacheable IO memory mm: introduce new flag to indicate wc safe kvm: arm64: set io memory s2 pte as normalnc for vfio pci device vfio: convey kvm that the vfio-pci device is wc safe arch/arm64/include/asm/kvm_pgtable.h | 2 ++ arch/arm64/include/asm/memory.h | 2 ++ arch/arm64/kvm/hyp/pgtable.c | 23 ++++++++++++++++++----- arch/arm64/kvm/mmu.c | 18 ++++++++++++++---- drivers/vfio/pci/vfio_pci_core.c | 3 ++- include/linux/mm.h | 14 ++++++++++++++ 6 files changed, 52 insertions(+), 10 deletions(-) -- 2.34.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next reply other threads:[~2024-02-07 20:47 UTC|newest] Thread overview: 38+ messages / expand[flat|nested] mbox.gz Atom feed top 2024-02-07 20:46 ankita [this message] 2024-02-07 20:46 ` [PATCH v6 0/4] kvm: arm64: allow the VM to select DEVICE_* and NORMAL_NC for IO memory ankita 2024-02-07 20:46 ` [PATCH v6 1/4] kvm: arm64: introduce new flag for non-cacheable " ankita 2024-02-07 20:46 ` ankita 2024-02-08 13:00 ` Catalin Marinas 2024-02-08 13:00 ` Catalin Marinas 2024-02-08 13:24 ` Oliver Upton 2024-02-08 13:24 ` Oliver Upton 2024-02-09 14:10 ` Ankit Agrawal 2024-02-09 14:10 ` Ankit Agrawal 2024-02-08 13:19 ` Will Deacon 2024-02-08 13:19 ` Will Deacon 2024-02-09 14:12 ` Ankit Agrawal 2024-02-09 14:12 ` Ankit Agrawal 2024-02-07 20:46 ` [PATCH v6 2/4] mm: introduce new flag to indicate wc safe ankita 2024-02-07 20:46 ` ankita 2024-02-08 13:03 ` Catalin Marinas 2024-02-08 13:03 ` Catalin Marinas 2024-02-08 14:34 ` Jason Gunthorpe 2024-02-08 14:34 ` Jason Gunthorpe 2024-02-07 20:46 ` [PATCH v6 3/4] kvm: arm64: set io memory s2 pte as normalnc for vfio pci device ankita 2024-02-07 20:46 ` ankita 2024-02-08 13:26 ` Oliver Upton 2024-02-08 13:26 ` Oliver Upton 2024-02-08 14:51 ` Catalin Marinas 2024-02-08 14:51 ` Catalin Marinas 2024-02-09 14:05 ` Ankit Agrawal 2024-02-09 14:05 ` Ankit Agrawal 2024-02-07 20:46 ` [PATCH v6 4/4] vfio: convey kvm that the vfio-pci device is wc safe ankita 2024-02-07 20:46 ` ankita 2024-02-08 14:53 ` Catalin Marinas 2024-02-08 14:53 ` Catalin Marinas 2024-02-08 17:30 ` Alex Williamson 2024-02-08 17:30 ` Alex Williamson 2024-02-08 17:54 ` Jason Gunthorpe 2024-02-08 17:54 ` Jason Gunthorpe 2024-02-09 14:02 ` Ankit Agrawal 2024-02-09 14:02 ` Ankit Agrawal
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20240207204652.22954-1-ankita@nvidia.com \ --to=ankita@nvidia.com \ --cc=acurrid@nvidia.com \ --cc=akpm@linux-foundation.org \ --cc=alex.williamson@redhat.com \ --cc=andreyknvl@gmail.com \ --cc=aniketa@nvidia.com \ --cc=apopple@nvidia.com \ --cc=ardb@kernel.org \ --cc=brauner@kernel.org \ --cc=catalin.marinas@arm.com \ --cc=cjia@nvidia.com \ --cc=danw@nvidia.com \ --cc=gshan@redhat.com \ --cc=james.morse@arm.com \ --cc=jgg@nvidia.com \ --cc=jhubbard@nvidia.com \ --cc=kevin.tian@intel.com \ --cc=kvm@vger.kernel.org \ --cc=kvmarm@lists.linux.dev \ --cc=kwankhede@nvidia.com \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=lpieralisi@kernel.org \ --cc=mark.rutland@arm.com \ --cc=maz@kernel.org \ --cc=mochs@nvidia.com \ --cc=oliver.upton@linux.dev \ --cc=rananta@google.com \ --cc=reinette.chatre@intel.com \ --cc=ricarkol@google.com \ --cc=ryan.roberts@arm.com \ --cc=stefanha@redhat.com \ --cc=surenb@google.com \ --cc=suzuki.poulose@arm.com \ --cc=targupta@nvidia.com \ --cc=vsethi@nvidia.com \ --cc=wangjinchao@xfusion.com \ --cc=will@kernel.org \ --cc=yi.l.liu@intel.com \ --cc=yuzenghui@huawei.com \ --cc=zhiw@nvidia.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.