* [PATCH v12 00/11] Enable Sub-Page Write Protection Support
@ 2020-05-16 12:54 Yang Weijiang
2020-05-16 12:54 ` [PATCH v12 01/11] Documentation: Add EPT based Subpage Protection and related APIs Yang Weijiang
` (10 more replies)
0 siblings, 11 replies; 12+ messages in thread
From: Yang Weijiang @ 2020-05-16 12:54 UTC (permalink / raw)
To: kvm, linux-kernel, pbonzini, jmattson, sean.j.christopherson
Cc: yu.c.zhang, alazar, edwin.zhai, ssicleru, Yang Weijiang
EPT-Based Sub-Page write Protection(SPP) allows Virtual Machine Monitor(VMM)
specify write-permission for guest physical memory at a sub-page(128 byte)
granularity. When SPP works, HW enforces write-access check for sub-pages
within a protected 4KB page.
The feature targets to provide fine-grained memory protection for usages such
as memory guard and VM introspection etc.
SPP is active when the "sub-page write protection" (bit 23) is set in Secondary
VM-Execution Controls. The feature is backed with a Sub-Page Permission Table(SPPT).
The subpage permission vector is stored in the leaf entry of SPPT. The root page
is referenced via a Sub-Page Permission Table Pointer (SPPTP) in VMCS.
To enable SPP for guest memory, the guest page should be first mapped in a 4KB EPT
entry, then set SPP bit 61 of the corresponding entry. While HW walks EPT, it traverses
SPPT with the gpa to look up the sub-page permission vector within SPPT leaf entry.
If the corresponding bit is set, write to sub-page is permitted, otherwise, SPP induced
EPT violation is generated.
This patch serial passed SPP function test and selftest on Ice-Lake platform.
Please refer to the SPP introduction document in this patch set and
Intel SDM for details:
Intel SDM:
https://software.intel.com/sites/default/files/managed/39/c5/325462-sdm-vol-1-2abcd-3abcd.pdf
Patch 1: Documentation for SPP and related API.
Patch 2: Put MMU/SSP shared definitions to a new mmu_internal.h file.
Patch 3: SPPT setup functions
Patch 4: Functions to {get|set}_subpage permission
Patch 5: Introduce user-space SPP IOCTLs
Patch 6: Handle SPP induced vmexit and EPT violation
Patch 7: Enable Lazy mode SPP protection
Patch 8: Re-enable SPP if EPT mapping changes
Patch 9: Enable SPP in instruction emulation
Patch 10: Initialize SPP related data structures.
Patch 11: selftest for SPP.
Change logs:
v12:
1. Put MMU/SPP shared definitions/prototypes into a new mmu_internal.h per
maintainers' comments.
2. Changed fast_page_fault()'s return type from bool to RET_PF_* per Paolo's
suggestion.
3. Re-allocate SPPT root page if the root is allocated in an early spp_init()
but is freed accompanied with EPT root page release.
The issue is reported by Stefan Sicleru <ssicleru@bitdefender.com>.
4. Other refactor per above changes.
5. Rebased patches to 5.7-rc5.
6. Fixed a virtual address mapping issue of selftest.
v11:
1. Refactored patches Per Sean's review feedback.
2. Added HW/KVM capabilities check before initializes SPP.
3. Combined a few functions having similar usages.
4. Removed unecessary functions in kvm_x86_ops.
5. Other code fix according to testing.
v10:
1. Cleared SPP active flag on VM resetting.
2. Added trancepoints on subpage setup and SPP induced vmexits.
3. Fixed a few code issues reported by Intel test robot.
v9:
1. Added SPP protection check in pte prefetch case.
2. Flushed EPT rmap to remove existing mappings of the target gfns.
3. Modified documentation to reflect recent changes.
4. Other minor code refactor.
v8:
1. Changed ioctl interface definition per Paolo's comments.
2. Replaced SPP_INIT ioctl funciton with KVM_ENABLE_CAP.
3. Removed SPP bit from X86 feature word.
4. Returned instruction length to user-space when SPP induced EPT
violation happens, this is to provide flexibility to use SPP,
revert write or track write.
5. Modified selftest application and added into this serial.
6. Simplified SPP permission vector check.
7. Moved spp.c and spp.h to kvm/mmu folder.
8. Other code fix according to Paolo's feedback and testing.
v7:
1. Configured all available protected pages once SPP induced vmexit
happens since there's no PRESENT bit in SPPT leaf entry.
2. Changed SPP protection check flow in tdp_page_fault().
3. Code refactor and minior fixes.
v6:
1. Added SPP protection patch for emulation cases per Jim's review.
2. Modified documentation and added API description per Jim's review.
3. Other minior changes suggested by Jim.
v5:
1. Enable SPP support for Hugepage(1GB/2MB) to extend application.
2. Make SPP miss vm-exit handler as the unified place to set up SPPT.
3. If SPP protected pages are access-tracked or dirty-page-tracked,
store SPP flag in reserved address bit, restore it in
fast_page_fault() handler.
4. Move SPP specific functions to vmx/spp.c and vmx/spp.h
5. Rebased code to kernel v5.3
6. Other change suggested by KVM community.
v4:
1. Modified documentation to make it consistent with patches.
2. Allocated SPPT root page in init_spp() instead of vmx_set_cr3() to
avoid SPPT miss error.
3. Added back co-developers and sign-offs.
v3:
1. Rebased patches to kernel 5.1 release
2. Deferred SPPT setup to EPT fault handler if the page is not
available while set_subpage() is being called.
3. Added init IOCTL to reduce extra cost if SPP is not used.
4. Refactored patch structure, cleaned up cross referenced functions.
5. Added code to deal with memory swapping/migration/shrinker cases.
v1:
1. Rebased to 4.20-rc1
2. Move VMCS change to a separated patch.
3. Code refine and Bug fix
Yang Weijiang (11):
Documentation: Add EPT based Subpage Protection and related APIs
mmu: spp: Add a new header file to put definitions shared by MMU and
SPP
mmu: spp: Implement SPPT setup functions
mmu: spp: Implement functions to {get|set}_subpage permission
x86: spp: Introduce user-space SPP IOCTLs
vmx: spp: Handle SPP induced vmexit and EPT violation
mmu: spp: Enable Lazy mode SPP protection
mmu: spp: Re-enable SPP protection when EPT mapping changes
x86: spp: Add SPP protection check in instruction emulation
vmx: spp: Initialize SPP bitmap and SPP protection
kvm: selftests: selftest for Sub-Page protection
Documentation/virt/kvm/api.rst | 38 ++
Documentation/virtual/kvm/spp_kvm.txt | 179 +++++
arch/x86/include/asm/kvm_host.h | 11 +-
arch/x86/include/asm/vmx.h | 10 +
arch/x86/include/asm/vmxfeatures.h | 1 +
arch/x86/include/uapi/asm/vmx.h | 2 +
arch/x86/kvm/Makefile | 2 +-
arch/x86/kvm/mmu.h | 9 +-
arch/x86/kvm/mmu/mmu.c | 287 ++++----
arch/x86/kvm/mmu/spp.c | 621 ++++++++++++++++++
arch/x86/kvm/mmu/spp.h | 39 ++
arch/x86/kvm/mmu_internal.h | 147 +++++
arch/x86/kvm/mmutrace.h | 10 +-
arch/x86/kvm/trace.h | 66 ++
arch/x86/kvm/vmx/capabilities.h | 5 +
arch/x86/kvm/vmx/vmx.c | 110 ++++
arch/x86/kvm/x86.c | 135 ++++
include/uapi/linux/kvm.h | 17 +
tools/testing/selftests/kvm/Makefile | 1 +
tools/testing/selftests/kvm/lib/kvm_util.c | 1 +
tools/testing/selftests/kvm/x86_64/spp_test.c | 235 +++++++
21 files changed, 1793 insertions(+), 133 deletions(-)
create mode 100644 Documentation/virtual/kvm/spp_kvm.txt
create mode 100644 arch/x86/kvm/mmu/spp.c
create mode 100644 arch/x86/kvm/mmu/spp.h
create mode 100644 arch/x86/kvm/mmu_internal.h
create mode 100644 tools/testing/selftests/kvm/x86_64/spp_test.c
--
2.17.2
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH v12 01/11] Documentation: Add EPT based Subpage Protection and related APIs
2020-05-16 12:54 [PATCH v12 00/11] Enable Sub-Page Write Protection Support Yang Weijiang
@ 2020-05-16 12:54 ` Yang Weijiang
2020-05-16 12:54 ` [PATCH v12 02/11] mmu: spp: Add a new header file to put definitions shared by MMU and SPP Yang Weijiang
` (9 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Yang Weijiang @ 2020-05-16 12:54 UTC (permalink / raw)
To: kvm, linux-kernel, pbonzini, jmattson, sean.j.christopherson
Cc: yu.c.zhang, alazar, edwin.zhai, ssicleru, Yang Weijiang
Co-developed-by: yi.z.zhang@linux.intel.com
Signed-off-by: yi.z.zhang@linux.intel.com
Signed-off-by: Yang Weijiang <weijiang.yang@intel.com>
---
Documentation/virt/kvm/api.rst | 38 ++++++
Documentation/virtual/kvm/spp_kvm.txt | 179 ++++++++++++++++++++++++++
2 files changed, 217 insertions(+)
create mode 100644 Documentation/virtual/kvm/spp_kvm.txt
diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index efbbe570aa9b..b441280e1218 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -4690,6 +4690,44 @@ KVM_PV_VM_VERIFY
Verify the integrity of the unpacked image. Only if this succeeds,
KVM is allowed to start protected VCPUs.
+4.126 KVM_SUBPAGES_GET_ACCESS
+
+Architectures: x86
+Type: vm ioctl
+Parameters: struct kvm_subpage_info (in/out)
+Returns: 0 on success, < 0 on error
+
+#define KVM_SUBPAGE_MAX_PAGES 512
+struct kvm_subpage {
+ __u64 gfn_base; /* the first page gfn of the contiguous pages */
+ __u32 npages; /* number of 4K pages */
+ __u32 flags; /* reserved to 0 now */
+ __u32 access_map[0]; /* start place of bitmap array */
+};
+
+This ioctl fetches subpage permission from contiguous pages starting with
+gfn. npages is the number of contiguous pages to fetch. access_map contains permission
+vectors fetched for all the pages.
+
+4.127 KVM_SUBPAGES_SET_ACCESS
+
+Architectures: x86
+Type: vm ioctl
+Parameters: struct kvm_subpage_info (in/out)
+Returns: 0 on success, < 0 on error
+
+#define KVM_SUBPAGE_MAX_PAGES 512
+struct kvm_subpage {
+ __u64 gfn_base; /* the first page gfn of the contiguous pages */
+ __u32 npages; /* number of 4K pages */
+ __u32 flags; /* reserved to 0 now */
+ __u32 access_map[0]; /* start place of bitmap array */
+};
+
+This ioctl sets subpage permission for contiguous pages starting with gfn. npages is
+the number of contiguous pages to set. access_map contains permission vectors for all the
+pages. Since during execution of the ioctl, it holds mmu_lock, so limits the MAX pages
+to 512 to reduce the impact to EPT.
5. The kvm_run structure
========================
diff --git a/Documentation/virtual/kvm/spp_kvm.txt b/Documentation/virtual/kvm/spp_kvm.txt
new file mode 100644
index 000000000000..1b41125e0cb1
--- /dev/null
+++ b/Documentation/virtual/kvm/spp_kvm.txt
@@ -0,0 +1,179 @@
+EPT-Based Sub-Page Protection (SPP) for KVM
+====================================================
+
+1.Overview
+ EPT-based Sub-Page Protection(SPP) allows VMM to specify
+ fine-grained(128byte per sub-page) write-protection for guest physical
+ memory. When it's enabled, the CPU enforces write-access permission
+ for the sub-pages within a 4KB page, if corresponding bit is set in
+ permission vector, write to sub-page region is allowed, otherwise,
+ it's prevented with a EPT violation.
+
+ *Note*: In current implementation, SPP is exclusive with nested flag,
+ if it's on, SPP feature won't work.
+
+2.SPP Operation
+ Sub-Page Protection Table (SPPT) is introduced to manage sub-page
+ write-access permission.
+
+ It is active when:
+ a) nested flag is turned off.
+ b) "sub-page write protection" VM-execution control is 1.
+ c) SPP is initialized with KVM_ENABLE_CAP ioctl and sub-class KVM_CAP_X86_SPP.
+ d) Sub-page permissions are set with KVM_SUBPAGES_SET_ACCESS ioctl.
+ see below sections for details.
+
+ __________________________________________________________________________
+
+ How SPP hardware works:
+ __________________________________________________________________________
+
+ Guest write access --> GPA --> Walk EPT --> EPT leaf entry -----|
+ |---------------------------------------------------------------|
+ |-> if VMexec_control.spp && ept_leaf_entry.spp_bit (bit 61)
+ |
+ |-> <false> --> EPT legacy behavior
+ |
+ |
+ |-> <true> --> if ept_leaf_entry.writable
+ |
+ |-> <true> --> Ignore SPP
+ |
+ |-> <false> --> GPA --> Walk SPP 4-level table--|
+ |
+ |------------<----------get-the-SPPT-point-from-VMCS-field-----<------|
+ |
+ Walk SPP L4E table
+ |
+ |---> if-entry-misconfiguration ------------>-------|-------<---------|
+ | | |
+ else | |
+ | | |
+ | |------------------SPP VMexit<-----------------| |
+ | | |
+ | |-> exit_qualification & sppt_misconfig --> sppt misconfig |
+ | | |
+ | |-> exit_qualification & sppt_miss --> sppt miss |
+ |---| |
+ | |
+ walk SPPT L3E--|--> if-entry-misconfiguration------------>------------|
+ | |
+ else |
+ | |
+ | |
+ walk SPPT L2E --|--> if-entry-misconfiguration-------->-------|
+ | |
+ else |
+ | |
+ | |
+ walk SPPT L1E --|-> if-entry-misconfiguration--->----|
+ |
+ else
+ |
+ |-> if sub-page writable
+ |-> <true> allow, write access
+ |-> <false> disallow, EPT violation
+ ______________________________________________________________________________
+
+3.IOCTL Interfaces
+
+ KVM_ENABLE_CAP(capability: KVM_CAP_X86_SPP):
+ Allocate storage for sub-page permission vectors and SPPT root page.
+
+ KVM_SUBPAGES_GET_ACCESS:
+ Get sub-page write permission vectors for given contiguous guest pages.
+
+ KVM_SUBPAGES_SET_ACCESS
+ Set SPP bit in EPT leaf entries for given contiguous guest pages. The
+ actual SPPT setup is triggered when SPP miss vm-exit is handled.
+
+ struct kvm_subpage{
+ __u64 gfn_base; /* the first page gfn of the contiguous pages */
+ __u32 npages; /* number of 4K pages */
+ __u32 flags; /* reserved to 0 now */
+ __u32 access_map[0]; /* start place of bitmap array */
+ };
+
+ #define KVM_SUBPAGES_GET_ACCESS _IOR(KVMIO, 0x49, __u64)
+ #define KVM_SUBPAGES_SET_ACCESS _IOW(KVMIO, 0x4a, __u64)
+
+4.Set Sub-Page Permission
+
+ * To enable SPP protection, KVM user-space application sets sub-page permission
+ via KVM_SUBPAGES_SET_ACCESS ioctl:
+ (1) It first stores the access permissions in bitmap array.
+
+ (2) Then, if the target 4KB pages are mapped as PT_PAGE_TABLE_LEVEL entry in EPT,
+ it sets SPP bit of the corresponding entry to mark sub-page protection.
+ If the 4KB pages are mapped within PT_DIRECTORY_LEVEL or PT_PDPE_LEVEL entry,
+ it first zaps the hugepage entries so as to let following memory access to trigger
+ EPT violation, there the gfn is check against SPP permission bitmap and
+ proper level is selected to set up EPT entry.
+
+
+ The SPPT paging structure format is as below:
+
+ Format of the SPPT L4E, L3E, L2E:
+ | Bit | Contents |
+ | :----- | :------------------------------------------------------------------------|
+ | 0 | Valid entry when set; indicates whether the entry is present |
+ | 11:1 | Reserved (0) |
+ | N-1:12 | Physical address of 4KB aligned SPPT LX-1 Table referenced by this entry |
+ | 51:N | Reserved (0) |
+ | 63:52 | Reserved (0) |
+ Note: N is the physical address width supported by the processor. X is the page level
+
+ Format of the SPPT L1E:
+ | Bit | Contents |
+ | :---- | :---------------------------------------------------------------- |
+ | 0+2i | Write permission for i-th 128 byte sub-page region. |
+ | 1+2i | Reserved (0). |
+ Note: 0<=i<=31
+
+5.SPPT-induced VM exit
+
+ * SPPT miss and misconfiguration induced VM exit
+
+ A SPPT missing VM exit occurs when walk the SPPT, there is no SPPT
+ misconfiguration but a paging-structure entry is not
+ present in any of L4E/L3E/L2E entries.
+
+ A SPPT misconfiguration VM exit occurs when reserved bits or unsupported values
+ are set in SPPT entry.
+
+ *NOTE* SPPT miss and SPPT misconfigurations can occur only due to
+ "eligible" memory write, this excludes, e.g., guest paging structure,
+ please refer to SDM 28.2 for details of "non-eligible" cases.
+
+ * SPP permission induced VM exit
+ SPP sub-page permission induced violation is reported as EPT violation
+ therefore causes VM exit.
+
+6.SPPT-induced VM exit handling
+
+ #define EXIT_REASON_SPP 66
+
+ static int (*const kvm_vmx_exit_handlers[])(struct kvm_vcpu *vcpu) = {
+ ...
+ [EXIT_REASON_SPP] = handle_spp,
+ ...
+ };
+
+ New exit qualification for SPPT-induced vmexits.
+
+ | Bit | Contents |
+ | :---- | :---------------------------------------------------------------- |
+ | 10:0 | Reserved (0). |
+ | 11 | SPPT VM exit type. Set for SPPT Miss, cleared for SPPT Misconfig. |
+ | 12 | NMI unblocking due to IRET |
+ | 63:13 | Reserved (0) |
+
+ * SPPT miss induced VM exit
+ Set up SPPT entries correctly.
+
+ * SPPT misconfiguration induced VM exit
+ This is left to user-space application to handle.
+
+ * SPP permission induced VM exit
+ This is left to user-space application to handle, e.g.,
+ retry the fault instruction or skip it.
--
2.17.2
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH v12 02/11] mmu: spp: Add a new header file to put definitions shared by MMU and SPP
2020-05-16 12:54 [PATCH v12 00/11] Enable Sub-Page Write Protection Support Yang Weijiang
2020-05-16 12:54 ` [PATCH v12 01/11] Documentation: Add EPT based Subpage Protection and related APIs Yang Weijiang
@ 2020-05-16 12:54 ` Yang Weijiang
2020-05-16 12:54 ` [PATCH v12 03/11] mmu: spp: Implement SPPT setup functions Yang Weijiang
` (8 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Yang Weijiang @ 2020-05-16 12:54 UTC (permalink / raw)
To: kvm, linux-kernel, pbonzini, jmattson, sean.j.christopherson
Cc: yu.c.zhang, alazar, edwin.zhai, ssicleru, Yang Weijiang
SPP re-uses many MMU data structures, macros and functions etc. To separate spp.c
as a new compilation unit, put these shared parts into a new mmu_internal.h file.
Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Yang Weijiang <weijiang.yang@intel.com>
---
arch/x86/kvm/mmu.h | 7 --
arch/x86/kvm/mmu/mmu.c | 140 ++++++++++------------------------
arch/x86/kvm/mmu_internal.h | 145 ++++++++++++++++++++++++++++++++++++
3 files changed, 185 insertions(+), 107 deletions(-)
create mode 100644 arch/x86/kvm/mmu_internal.h
diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h
index 8a3b1bce722a..da199f0a69db 100644
--- a/arch/x86/kvm/mmu.h
+++ b/arch/x86/kvm/mmu.h
@@ -22,8 +22,6 @@
#define PT_ACCESSED_MASK (1ULL << PT_ACCESSED_SHIFT)
#define PT_DIRTY_SHIFT 6
#define PT_DIRTY_MASK (1ULL << PT_DIRTY_SHIFT)
-#define PT_PAGE_SIZE_SHIFT 7
-#define PT_PAGE_SIZE_MASK (1ULL << PT_PAGE_SIZE_SHIFT)
#define PT_PAT_MASK (1ULL << 7)
#define PT_GLOBAL_MASK (1ULL << 8)
#define PT64_NX_SHIFT 63
@@ -38,11 +36,6 @@
#define PT32_DIR_PSE36_MASK \
(((1ULL << PT32_DIR_PSE36_SIZE) - 1) << PT32_DIR_PSE36_SHIFT)
-#define PT64_ROOT_5LEVEL 5
-#define PT64_ROOT_4LEVEL 4
-#define PT32_ROOT_LEVEL 2
-#define PT32E_ROOT_LEVEL 3
-
static inline u64 rsvd_bits(int s, int e)
{
if (e < s)
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 8071952e9cf2..615effaf5814 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -16,6 +16,7 @@
*/
#include "irq.h"
+#include "mmu_internal.h"
#include "mmu.h"
#include "x86.h"
#include "kvm_cache_regs.h"
@@ -116,7 +117,6 @@ module_param(dbg, bool, 0644);
#define PTE_PREFETCH_NUM 8
#define PT_FIRST_AVAIL_BITS_SHIFT 10
-#define PT64_SECOND_AVAIL_BITS_SHIFT 54
/*
* The mask used to denote special SPTEs, which can be either MMIO SPTEs or
@@ -128,15 +128,6 @@ module_param(dbg, bool, 0644);
#define SPTE_AD_WRPROT_ONLY_MASK (2ULL << 52)
#define SPTE_MMIO_MASK (3ULL << 52)
-#define PT64_LEVEL_BITS 9
-
-#define PT64_LEVEL_SHIFT(level) \
- (PAGE_SHIFT + (level - 1) * PT64_LEVEL_BITS)
-
-#define PT64_INDEX(address, level)\
- (((address) >> PT64_LEVEL_SHIFT(level)) & ((1 << PT64_LEVEL_BITS) - 1))
-
-
#define PT32_LEVEL_BITS 10
#define PT32_LEVEL_SHIFT(level) \
@@ -149,7 +140,6 @@ module_param(dbg, bool, 0644);
#define PT32_INDEX(address, level)\
(((address) >> PT32_LEVEL_SHIFT(level)) & ((1 << PT32_LEVEL_BITS) - 1))
-
#ifdef CONFIG_DYNAMIC_PHYSICAL_MASK
#define PT64_BASE_ADDR_MASK (physical_mask & ~(u64)(PAGE_SIZE-1))
#else
@@ -186,11 +176,6 @@ module_param(dbg, bool, 0644);
#define SPTE_HOST_WRITEABLE (1ULL << PT_FIRST_AVAIL_BITS_SHIFT)
#define SPTE_MMU_WRITEABLE (1ULL << (PT_FIRST_AVAIL_BITS_SHIFT + 1))
-#define SHADOW_PT_INDEX(addr, level) PT64_INDEX(addr, level)
-
-/* make pte_list_desc fit well in cache line */
-#define PTE_LIST_EXT 3
-
/*
* Return values of handle_mmio_page_fault and mmu.page_fault:
* RET_PF_RETRY: let CPU fault again on the address.
@@ -205,30 +190,6 @@ enum {
RET_PF_INVALID = 2,
};
-struct pte_list_desc {
- u64 *sptes[PTE_LIST_EXT];
- struct pte_list_desc *more;
-};
-
-struct kvm_shadow_walk_iterator {
- u64 addr;
- hpa_t shadow_addr;
- u64 *sptep;
- int level;
- unsigned index;
-};
-
-#define for_each_shadow_entry_using_root(_vcpu, _root, _addr, _walker) \
- for (shadow_walk_init_using_root(&(_walker), (_vcpu), \
- (_root), (_addr)); \
- shadow_walk_okay(&(_walker)); \
- shadow_walk_next(&(_walker)))
-
-#define for_each_shadow_entry(_vcpu, _addr, _walker) \
- for (shadow_walk_init(&(_walker), _vcpu, _addr); \
- shadow_walk_okay(&(_walker)); \
- shadow_walk_next(&(_walker)))
-
#define for_each_shadow_entry_lockless(_vcpu, _addr, _walker, spte) \
for (shadow_walk_init(&(_walker), _vcpu, _addr); \
shadow_walk_okay(&(_walker)) && \
@@ -237,15 +198,15 @@ struct kvm_shadow_walk_iterator {
static struct kmem_cache *pte_list_desc_cache;
static struct kmem_cache *mmu_page_header_cache;
-static struct percpu_counter kvm_total_used_mmu_pages;
+struct percpu_counter kvm_total_used_mmu_pages;
static u64 __read_mostly shadow_nx_mask;
static u64 __read_mostly shadow_x_mask; /* mutual exclusive with nx_mask */
static u64 __read_mostly shadow_user_mask;
static u64 __read_mostly shadow_accessed_mask;
static u64 __read_mostly shadow_dirty_mask;
-static u64 __read_mostly shadow_mmio_mask;
-static u64 __read_mostly shadow_mmio_value;
+u64 __read_mostly shadow_mmio_mask;
+u64 __read_mostly shadow_mmio_value;
static u64 __read_mostly shadow_mmio_access_mask;
static u64 __read_mostly shadow_present_mask;
static u64 __read_mostly shadow_me_mask;
@@ -294,7 +255,7 @@ static u64 __read_mostly shadow_nonpresent_or_rsvd_lower_gfn_mask;
*/
static u8 __read_mostly shadow_phys_bits;
-static void mmu_spte_set(u64 *sptep, u64 spte);
+void mmu_spte_set(u64 *sptep, u64 spte);
static bool is_executable_pte(u64 spte);
static union kvm_mmu_page_role
kvm_mmu_calc_root_page_role(struct kvm_vcpu *vcpu);
@@ -341,7 +302,7 @@ void kvm_mmu_set_mmio_spte_mask(u64 mmio_mask, u64 mmio_value, u64 access_mask)
}
EXPORT_SYMBOL_GPL(kvm_mmu_set_mmio_spte_mask);
-static bool is_mmio_spte(u64 spte)
+bool is_mmio_spte(u64 spte)
{
return (spte & shadow_mmio_mask) == shadow_mmio_value;
}
@@ -608,17 +569,17 @@ static int is_nx(struct kvm_vcpu *vcpu)
return vcpu->arch.efer & EFER_NX;
}
-static int is_shadow_present_pte(u64 pte)
+int is_shadow_present_pte(u64 pte)
{
return (pte != 0) && !is_mmio_spte(pte);
}
-static int is_large_pte(u64 pte)
+int is_large_pte(u64 pte)
{
return pte & PT_PAGE_SIZE_MASK;
}
-static int is_last_spte(u64 pte, int level)
+int is_last_spte(u64 pte, int level)
{
if (level == PT_PAGE_TABLE_LEVEL)
return 1;
@@ -645,7 +606,7 @@ static gfn_t pse36_gfn_delta(u32 gpte)
}
#ifdef CONFIG_X86_64
-static void __set_spte(u64 *sptep, u64 spte)
+void __set_spte(u64 *sptep, u64 spte)
{
WRITE_ONCE(*sptep, spte);
}
@@ -660,7 +621,7 @@ static u64 __update_clear_spte_slow(u64 *sptep, u64 spte)
return xchg(sptep, spte);
}
-static u64 __get_spte_lockless(u64 *sptep)
+u64 __get_spte_lockless(u64 *sptep)
{
return READ_ONCE(*sptep);
}
@@ -685,7 +646,7 @@ static void count_spte_clear(u64 *sptep, u64 spte)
sp->clear_spte_count++;
}
-static void __set_spte(u64 *sptep, u64 spte)
+void __set_spte(u64 *sptep, u64 spte)
{
union split_spte *ssptep, sspte;
@@ -757,7 +718,7 @@ static u64 __update_clear_spte_slow(u64 *sptep, u64 spte)
* present->non-present updates: if it changed while reading the spte,
* we might have hit the race. This is done using clear_spte_count.
*/
-static u64 __get_spte_lockless(u64 *sptep)
+u64 __get_spte_lockless(u64 *sptep)
{
struct kvm_mmu_page *sp = page_header(__pa(sptep));
union split_spte spte, *orig = (union split_spte *)sptep;
@@ -832,7 +793,7 @@ static bool is_dirty_spte(u64 spte)
* or in a state where the hardware will not attempt to update
* the spte.
*/
-static void mmu_spte_set(u64 *sptep, u64 new_spte)
+void mmu_spte_set(u64 *sptep, u64 new_spte)
{
WARN_ON(is_shadow_present_pte(*sptep));
__set_spte(sptep, new_spte);
@@ -842,7 +803,7 @@ static void mmu_spte_set(u64 *sptep, u64 new_spte)
* Update the SPTE (excluding the PFN), but do not track changes in its
* accessed/dirty status.
*/
-static u64 mmu_spte_update_no_track(u64 *sptep, u64 new_spte)
+u64 mmu_spte_update_no_track(u64 *sptep, u64 new_spte)
{
u64 old_spte = *sptep;
@@ -874,7 +835,7 @@ static u64 mmu_spte_update_no_track(u64 *sptep, u64 new_spte)
*
* Returns true if the TLB needs to be flushed
*/
-static bool mmu_spte_update(u64 *sptep, u64 new_spte)
+bool mmu_spte_update(u64 *sptep, u64 new_spte)
{
bool flush = false;
u64 old_spte = mmu_spte_update_no_track(sptep, new_spte);
@@ -915,7 +876,7 @@ static bool mmu_spte_update(u64 *sptep, u64 new_spte)
* state bits, it is used to clear the last level sptep.
* Returns non-zero if the PTE was previously valid.
*/
-static int mmu_spte_clear_track_bits(u64 *sptep)
+int mmu_spte_clear_track_bits(u64 *sptep)
{
kvm_pfn_t pfn;
u64 old_spte = *sptep;
@@ -956,7 +917,7 @@ static void mmu_spte_clear_no_track(u64 *sptep)
__update_clear_spte_fast(sptep, 0ull);
}
-static u64 mmu_spte_get_lockless(u64 *sptep)
+u64 mmu_spte_get_lockless(u64 *sptep)
{
return __get_spte_lockless(sptep);
}
@@ -1109,7 +1070,7 @@ static void mmu_free_memory_cache_page(struct kvm_mmu_memory_cache *mc)
free_page((unsigned long)mc->objects[--mc->nobjs]);
}
-static int mmu_topup_memory_caches(struct kvm_vcpu *vcpu)
+int mmu_topup_memory_caches(struct kvm_vcpu *vcpu)
{
int r;
@@ -1135,7 +1096,7 @@ static void mmu_free_memory_caches(struct kvm_vcpu *vcpu)
mmu_page_header_cache);
}
-static void *mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc)
+void *mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc)
{
void *p;
@@ -1144,7 +1105,7 @@ static void *mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc)
return p;
}
-static struct pte_list_desc *mmu_alloc_pte_list_desc(struct kvm_vcpu *vcpu)
+struct pte_list_desc *mmu_alloc_pte_list_desc(struct kvm_vcpu *vcpu)
{
return mmu_memory_cache_alloc(&vcpu->arch.mmu_pte_list_desc_cache);
}
@@ -1293,7 +1254,7 @@ gfn_to_memslot_dirty_bitmap(struct kvm_vcpu *vcpu, gfn_t gfn,
/*
* Returns the number of pointers in the rmap chain, not counting the new one.
*/
-static int pte_list_add(struct kvm_vcpu *vcpu, u64 *spte,
+int pte_list_add(struct kvm_vcpu *vcpu, u64 *spte,
struct kvm_rmap_head *rmap_head)
{
struct pte_list_desc *desc;
@@ -1350,7 +1311,7 @@ pte_list_desc_remove_entry(struct kvm_rmap_head *rmap_head,
mmu_free_pte_list_desc(desc);
}
-static void __pte_list_remove(u64 *spte, struct kvm_rmap_head *rmap_head)
+void __pte_list_remove(u64 *spte, struct kvm_rmap_head *rmap_head)
{
struct pte_list_desc *desc;
struct pte_list_desc *prev_desc;
@@ -1386,13 +1347,13 @@ static void __pte_list_remove(u64 *spte, struct kvm_rmap_head *rmap_head)
}
}
-static void pte_list_remove(struct kvm_rmap_head *rmap_head, u64 *sptep)
+void pte_list_remove(struct kvm_rmap_head *rmap_head, u64 *sptep)
{
mmu_spte_clear_track_bits(sptep);
__pte_list_remove(sptep, rmap_head);
}
-static struct kvm_rmap_head *__gfn_to_rmap(gfn_t gfn, int level,
+struct kvm_rmap_head *__gfn_to_rmap(gfn_t gfn, int level,
struct kvm_memory_slot *slot)
{
unsigned long idx;
@@ -1431,7 +1392,7 @@ static int rmap_add(struct kvm_vcpu *vcpu, u64 *spte, gfn_t gfn)
return pte_list_add(vcpu, spte, rmap_head);
}
-static void rmap_remove(struct kvm *kvm, u64 *spte)
+void rmap_remove(struct kvm *kvm, u64 *spte)
{
struct kvm_mmu_page *sp;
gfn_t gfn;
@@ -1443,16 +1404,6 @@ static void rmap_remove(struct kvm *kvm, u64 *spte)
__pte_list_remove(spte, rmap_head);
}
-/*
- * Used by the following functions to iterate through the sptes linked by a
- * rmap. All fields are private and not assumed to be used outside.
- */
-struct rmap_iterator {
- /* private fields */
- struct pte_list_desc *desc; /* holds the sptep if not NULL */
- int pos; /* index of the sptep */
-};
-
/*
* Iteration must be started by this function. This should also be used after
* removing/dropping sptes from the rmap link because in such cases the
@@ -1460,7 +1411,7 @@ struct rmap_iterator {
*
* Returns sptep if found, NULL otherwise.
*/
-static u64 *rmap_get_first(struct kvm_rmap_head *rmap_head,
+u64 *rmap_get_first(struct kvm_rmap_head *rmap_head,
struct rmap_iterator *iter)
{
u64 *sptep;
@@ -1487,7 +1438,7 @@ static u64 *rmap_get_first(struct kvm_rmap_head *rmap_head,
*
* Returns sptep if found, NULL otherwise.
*/
-static u64 *rmap_get_next(struct rmap_iterator *iter)
+u64 *rmap_get_next(struct rmap_iterator *iter)
{
u64 *sptep;
@@ -1515,11 +1466,7 @@ static u64 *rmap_get_next(struct rmap_iterator *iter)
return sptep;
}
-#define for_each_rmap_spte(_rmap_head_, _iter_, _spte_) \
- for (_spte_ = rmap_get_first(_rmap_head_, _iter_); \
- _spte_; _spte_ = rmap_get_next(_iter_))
-
-static void drop_spte(struct kvm *kvm, u64 *sptep)
+void drop_spte(struct kvm *kvm, u64 *sptep)
{
if (mmu_spte_clear_track_bits(sptep))
rmap_remove(kvm, sptep);
@@ -1562,7 +1509,7 @@ static void drop_large_spte(struct kvm_vcpu *vcpu, u64 *sptep)
*
* Return true if tlb need be flushed.
*/
-static bool spte_write_protect(u64 *sptep, bool pt_protect)
+bool spte_write_protect(u64 *sptep, bool pt_protect)
{
u64 spte = *sptep;
@@ -2057,7 +2004,7 @@ static int is_empty_shadow_page(u64 *spt)
* aggregate version in order to make the slab shrinker
* faster
*/
-static inline void kvm_mod_used_mmu_pages(struct kvm *kvm, unsigned long nr)
+inline void kvm_mod_used_mmu_pages(struct kvm *kvm, unsigned long nr)
{
kvm->arch.n_used_mmu_pages += nr;
percpu_counter_add(&kvm_total_used_mmu_pages, nr);
@@ -2074,12 +2021,12 @@ static void kvm_mmu_free_page(struct kvm_mmu_page *sp)
kmem_cache_free(mmu_page_header_cache, sp);
}
-static unsigned kvm_page_table_hashfn(gfn_t gfn)
+unsigned kvm_page_table_hashfn(gfn_t gfn)
{
return hash_64(gfn, KVM_MMU_HASH_SHIFT);
}
-static void mmu_page_add_parent_pte(struct kvm_vcpu *vcpu,
+void mmu_page_add_parent_pte(struct kvm_vcpu *vcpu,
struct kvm_mmu_page *sp, u64 *parent_pte)
{
if (!parent_pte)
@@ -2101,7 +2048,7 @@ static void drop_parent_pte(struct kvm_mmu_page *sp,
mmu_spte_clear_no_track(parent_pte);
}
-static struct kvm_mmu_page *kvm_mmu_alloc_page(struct kvm_vcpu *vcpu, int direct)
+struct kvm_mmu_page *kvm_mmu_alloc_page(struct kvm_vcpu *vcpu, int direct)
{
struct kvm_mmu_page *sp;
@@ -2262,13 +2209,6 @@ static bool kvm_mmu_prepare_zap_page(struct kvm *kvm, struct kvm_mmu_page *sp,
static void kvm_mmu_commit_zap_page(struct kvm *kvm,
struct list_head *invalid_list);
-
-#define for_each_valid_sp(_kvm, _sp, _gfn) \
- hlist_for_each_entry(_sp, \
- &(_kvm)->arch.mmu_page_hash[kvm_page_table_hashfn(_gfn)], hash_link) \
- if (is_obsolete_sp((_kvm), (_sp))) { \
- } else
-
#define for_each_gfn_indirect_valid_sp(_kvm, _sp, _gfn) \
for_each_valid_sp(_kvm, _sp, _gfn) \
if ((_sp)->gfn != (_gfn) || (_sp)->role.direct) {} else
@@ -2323,7 +2263,7 @@ static void kvm_mmu_audit(struct kvm_vcpu *vcpu, int point) { }
static void mmu_audit_disable(void) { }
#endif
-static bool is_obsolete_sp(struct kvm *kvm, struct kvm_mmu_page *sp)
+bool is_obsolete_sp(struct kvm *kvm, struct kvm_mmu_page *sp)
{
return sp->role.invalid ||
unlikely(sp->mmu_valid_gen != kvm->arch.mmu_valid_gen);
@@ -2563,7 +2503,7 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu,
return sp;
}
-static void shadow_walk_init_using_root(struct kvm_shadow_walk_iterator *iterator,
+void shadow_walk_init_using_root(struct kvm_shadow_walk_iterator *iterator,
struct kvm_vcpu *vcpu, hpa_t root,
u64 addr)
{
@@ -2592,14 +2532,14 @@ static void shadow_walk_init_using_root(struct kvm_shadow_walk_iterator *iterato
}
}
-static void shadow_walk_init(struct kvm_shadow_walk_iterator *iterator,
+void shadow_walk_init(struct kvm_shadow_walk_iterator *iterator,
struct kvm_vcpu *vcpu, u64 addr)
{
shadow_walk_init_using_root(iterator, vcpu, vcpu->arch.mmu->root_hpa,
addr);
}
-static bool shadow_walk_okay(struct kvm_shadow_walk_iterator *iterator)
+bool shadow_walk_okay(struct kvm_shadow_walk_iterator *iterator)
{
if (iterator->level < PT_PAGE_TABLE_LEVEL)
return false;
@@ -2609,7 +2549,7 @@ static bool shadow_walk_okay(struct kvm_shadow_walk_iterator *iterator)
return true;
}
-static void __shadow_walk_next(struct kvm_shadow_walk_iterator *iterator,
+void __shadow_walk_next(struct kvm_shadow_walk_iterator *iterator,
u64 spte)
{
if (is_last_spte(spte, iterator->level)) {
@@ -2621,7 +2561,7 @@ static void __shadow_walk_next(struct kvm_shadow_walk_iterator *iterator,
--iterator->level;
}
-static void shadow_walk_next(struct kvm_shadow_walk_iterator *iterator)
+void shadow_walk_next(struct kvm_shadow_walk_iterator *iterator)
{
__shadow_walk_next(iterator, *iterator->sptep);
}
diff --git a/arch/x86/kvm/mmu_internal.h b/arch/x86/kvm/mmu_internal.h
new file mode 100644
index 000000000000..68e8179e7642
--- /dev/null
+++ b/arch/x86/kvm/mmu_internal.h
@@ -0,0 +1,145 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __KVM_X86_MMU_INTERNAL_H
+#define __KVM_X86_MMU_INTERNAL_H
+
+/* make pte_list_desc fit well in cache line */
+#define PTE_LIST_EXT 3
+#define PT64_SECOND_AVAIL_BITS_SHIFT 54
+#define PT64_ROOT_5LEVEL 5
+#define PT64_ROOT_4LEVEL 4
+#define PT32_ROOT_LEVEL 2
+#define PT32E_ROOT_LEVEL 3
+#define PT64_LEVEL_BITS 9
+#define PT_PAGE_SIZE_SHIFT 7
+#define PT_PAGE_SIZE_MASK (1ULL << PT_PAGE_SIZE_SHIFT)
+
+#define PT64_LEVEL_SHIFT(level) \
+ (PAGE_SHIFT + (level - 1) * PT64_LEVEL_BITS)
+
+#define PT64_INDEX(address, level)\
+ (((address) >> PT64_LEVEL_SHIFT(level)) & ((1 << PT64_LEVEL_BITS) - 1))
+
+#define SHADOW_PT_INDEX(addr, level) PT64_INDEX(addr, level)
+
+#ifdef CONFIG_DYNAMIC_PHYSICAL_MASK
+#define PT64_BASE_ADDR_MASK (physical_mask & ~(u64)(PAGE_SIZE-1))
+#else
+#define PT64_BASE_ADDR_MASK (((1ULL << 52) - 1) & ~(u64)(PAGE_SIZE-1))
+#endif
+#define PT64_LVL_ADDR_MASK(level) \
+ (PT64_BASE_ADDR_MASK & ~((1ULL << (PAGE_SHIFT + (((level) - 1) \
+ * PT64_LEVEL_BITS))) - 1))
+#define PT64_SPP_SAVED_BIT (1ULL << (PT64_SECOND_AVAIL_BITS_SHIFT + 1))
+/*
+ * Used by the following functions to iterate through the sptes linked by a
+ * rmap. All fields are private and not assumed to be used outside.
+ */
+struct rmap_iterator {
+ /* private fields */
+ struct pte_list_desc *desc; /* holds the sptep if not NULL */
+ int pos; /* index of the sptep */
+};
+
+struct pte_list_desc {
+ u64 *sptes[PTE_LIST_EXT];
+ struct pte_list_desc *more;
+};
+
+struct kvm_shadow_walk_iterator {
+ u64 addr;
+ hpa_t shadow_addr;
+ u64 *sptep;
+ int level;
+ unsigned index;
+};
+
+int is_large_pte(u64 pte);
+int is_last_spte(u64 pte, int level);
+
+u64 mmu_spte_update_no_track(u64 *sptep, u64 new_spte);
+
+void shadow_walk_init_using_root(struct kvm_shadow_walk_iterator *iterator,
+ struct kvm_vcpu *vcpu, hpa_t root,
+ u64 addr);
+void shadow_walk_init(struct kvm_shadow_walk_iterator *iterator,
+ struct kvm_vcpu *vcpu, u64 addr);
+bool shadow_walk_okay(struct kvm_shadow_walk_iterator *iterator);
+
+void __shadow_walk_next(struct kvm_shadow_walk_iterator *iterator,
+ u64 spte);
+void shadow_walk_next(struct kvm_shadow_walk_iterator *iterator);
+
+u64 *rmap_get_first(struct kvm_rmap_head *rmap_head,
+ struct rmap_iterator *iter);
+
+bool spte_write_protect(u64 *sptep, bool pt_protect);
+
+u64 *rmap_get_next(struct rmap_iterator *iter);
+
+bool is_obsolete_sp(struct kvm *kvm, struct kvm_mmu_page *sp);
+
+#define for_each_valid_sp(_kvm, _sp, _gfn) \
+ hlist_for_each_entry(_sp, \
+ &(_kvm)->arch.mmu_page_hash[kvm_page_table_hashfn(_gfn)], hash_link) \
+ if (is_obsolete_sp((_kvm), (_sp))) { \
+ } else
+
+#define for_each_rmap_spte(_rmap_head_, _iter_, _spte_) \
+ for (_spte_ = rmap_get_first(_rmap_head_, _iter_); \
+ _spte_; _spte_ = rmap_get_next(_iter_))
+
+#define for_each_shadow_entry_using_root(_vcpu, _root, _addr, _walker) \
+ for (shadow_walk_init_using_root(&(_walker), (_vcpu), \
+ (_root), (_addr)); \
+ shadow_walk_okay(&(_walker)); \
+ shadow_walk_next(&(_walker)))
+
+#define for_each_shadow_entry(_vcpu, _addr, _walker) \
+ for (shadow_walk_init(&(_walker), _vcpu, _addr); \
+ shadow_walk_okay(&(_walker)); \
+ shadow_walk_next(&(_walker)))
+
+bool is_mmio_spte(u64 spte);
+
+int is_shadow_present_pte(u64 pte);
+
+void __set_spte(u64 *sptep, u64 spte);
+
+int mmu_topup_memory_caches(struct kvm_vcpu *vcpu);
+
+void drop_spte(struct kvm *kvm, u64 *sptep);
+
+int mmu_spte_clear_track_bits(u64 *sptep);
+
+void rmap_remove(struct kvm *kvm, u64 *spte);
+
+bool mmu_spte_update(u64 *sptep, u64 new_spte);
+
+struct kvm_rmap_head *__gfn_to_rmap(gfn_t gfn, int level,
+ struct kvm_memory_slot *slot);
+
+void __pte_list_remove(u64 *spte, struct kvm_rmap_head *rmap_head);
+
+void pte_list_remove(struct kvm_rmap_head *rmap_head, u64 *sptep);
+
+u64 __get_spte_lockless(u64 *sptep);
+
+u64 mmu_spte_get_lockless(u64 *sptep);
+
+unsigned kvm_page_table_hashfn(gfn_t gfn);
+
+void mmu_spte_set(u64 *sptep, u64 new_spte);
+
+void *mmu_memory_cache_alloc(struct kvm_mmu_memory_cache *mc);
+
+struct pte_list_desc *mmu_alloc_pte_list_desc(struct kvm_vcpu *vcpu);
+
+int pte_list_add(struct kvm_vcpu *vcpu, u64 *spte,
+ struct kvm_rmap_head *rmap_head);
+
+void mmu_page_add_parent_pte(struct kvm_vcpu *vcpu,
+ struct kvm_mmu_page *sp, u64 *parent_pte);
+void kvm_mod_used_mmu_pages(struct kvm *kvm, unsigned long nr);
+
+struct kvm_mmu_page *kvm_mmu_alloc_page(struct kvm_vcpu *vcpu, int direct);
+#endif
--
2.17.2
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH v12 03/11] mmu: spp: Implement SPPT setup functions
2020-05-16 12:54 [PATCH v12 00/11] Enable Sub-Page Write Protection Support Yang Weijiang
2020-05-16 12:54 ` [PATCH v12 01/11] Documentation: Add EPT based Subpage Protection and related APIs Yang Weijiang
2020-05-16 12:54 ` [PATCH v12 02/11] mmu: spp: Add a new header file to put definitions shared by MMU and SPP Yang Weijiang
@ 2020-05-16 12:54 ` Yang Weijiang
2020-05-16 12:55 ` [PATCH v12 04/11] mmu: spp: Implement functions to {get|set}_subpage permission Yang Weijiang
` (7 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Yang Weijiang @ 2020-05-16 12:54 UTC (permalink / raw)
To: kvm, linux-kernel, pbonzini, jmattson, sean.j.christopherson
Cc: yu.c.zhang, alazar, edwin.zhai, ssicleru, Yang Weijiang
SPPT(Sub-Page Permission Table) is a 4-level structure similar
to EPT. If SPP is enabled in secondary execution control, WP and
SPP bit(61) are set in a 4KB EPT leaf entry, then SPPT is traversed
with the gfn, the leaf entry of SPPT contains the permission vector,
every subpage within the 4KB page owns one bit.
SPPT setup is similar to that of EPT therefore a lot of EPT helpers
are re-used in spp.c. To make least change to mmu.c and keep the patch
clean meanwhile, spp.c is embedded at the end of mmu.c.
Specific to SPPT:
1)The leaf entry contains a 64-bit permission vector. Subpage is
128B each so 4KB/128B = 32bits are required for write permission.
The even bits(2*i) corrrespond to write-permission to subpage(i),
if it's 1, subpage(i) is writable, otherwise, it's write-protected.
The odd bits are reserved and must be 0.
2)When permission vectors are updated, it first flushes the corresponding
entry at SPPT L2E by making the entry invalid so that following SPPT
walk can trigger SPP-miss handling, there the permission vectors are
rebuilt.
spp.c is built as a separate object file now per previous patch.
Co-developed-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: He Chen <he.chen@linux.intel.com>
Co-developed-by: Zhang Yi <yi.z.zhang@linux.intel.com>
Signed-off-by: Zhang Yi <yi.z.zhang@linux.intel.com>
Signed-off-by: Yang Weijiang <weijiang.yang@intel.com>
---
arch/x86/include/asm/kvm_host.h | 5 +-
arch/x86/kvm/Makefile | 2 +-
arch/x86/kvm/mmu/spp.c | 129 ++++++++++++++++++++++++++++++++
arch/x86/kvm/mmu/spp.h | 5 ++
4 files changed, 139 insertions(+), 2 deletions(-)
create mode 100644 arch/x86/kvm/mmu/spp.c
create mode 100644 arch/x86/kvm/mmu/spp.h
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 0dea9f122bb9..ee4721bd8703 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -274,7 +274,8 @@ union kvm_mmu_page_role {
unsigned smap_andnot_wp:1;
unsigned ad_disabled:1;
unsigned guest_mode:1;
- unsigned :6;
+ unsigned spp:1;
+ unsigned reserved:5;
/*
* This is left at the top of the word so that
@@ -982,6 +983,8 @@ struct kvm_arch {
struct kvm_pmu_event_filter *pmu_event_filter;
struct task_struct *nx_lpage_recovery_thread;
+
+ hpa_t sppt_root;
};
struct kvm_vm_stat {
diff --git a/arch/x86/kvm/Makefile b/arch/x86/kvm/Makefile
index 4a3081e9f4b5..1561543d443a 100644
--- a/arch/x86/kvm/Makefile
+++ b/arch/x86/kvm/Makefile
@@ -15,7 +15,7 @@ kvm-$(CONFIG_KVM_ASYNC_PF) += $(KVM)/async_pf.o
kvm-y += x86.o emulate.o i8259.o irq.o lapic.o \
i8254.o ioapic.o irq_comm.o cpuid.o pmu.o mtrr.o \
- hyperv.o debugfs.o mmu/mmu.o mmu/page_track.o
+ hyperv.o debugfs.o mmu/mmu.o mmu/page_track.o mmu/spp.o
kvm-intel-y += vmx/vmx.o vmx/vmenter.o vmx/pmu_intel.o vmx/vmcs12.o vmx/evmcs.o vmx/nested.o
kvm-amd-y += svm/svm.o svm/vmenter.o svm/pmu.o svm/nested.o svm/avic.o svm/sev.o
diff --git a/arch/x86/kvm/mmu/spp.c b/arch/x86/kvm/mmu/spp.c
new file mode 100644
index 000000000000..8924096df390
--- /dev/null
+++ b/arch/x86/kvm/mmu/spp.c
@@ -0,0 +1,129 @@
+// SPDX-License-Identifier: GPL-2.0
+#include <linux/kvm_host.h>
+#include "mmu_internal.h"
+#include "mmu.h"
+#include "spp.h"
+
+#define for_each_shadow_spp_entry(_vcpu, _addr, _walker) \
+ for (shadow_spp_walk_init(&(_walker), _vcpu, _addr); \
+ shadow_walk_okay(&(_walker)); \
+ shadow_walk_next(&(_walker)))
+
+static void shadow_spp_walk_init(struct kvm_shadow_walk_iterator *iterator,
+ struct kvm_vcpu *vcpu, u64 addr)
+{
+ iterator->addr = addr;
+ iterator->shadow_addr = vcpu->kvm->arch.sppt_root;
+
+ /* SPP Table is a 4-level paging structure */
+ iterator->level = PT64_ROOT_4LEVEL;
+}
+
+struct kvm_mmu_page *kvm_spp_get_page(struct kvm_vcpu *vcpu,
+ gfn_t gfn,
+ unsigned int level)
+{
+ struct kvm_mmu_page *sp;
+ union kvm_mmu_page_role role;
+
+ role = vcpu->arch.mmu->mmu_role.base;
+ role.level = level;
+ role.direct = true;
+ role.spp = true;
+
+ for_each_valid_sp(vcpu->kvm, sp, gfn) {
+ if (sp->gfn != gfn)
+ continue;
+ if (sp->role.word != role.word)
+ continue;
+ if (sp->role.spp && sp->role.level == level)
+ goto out;
+ }
+
+ sp = kvm_mmu_alloc_page(vcpu, true);
+ sp->gfn = gfn;
+ sp->role = role;
+ hlist_add_head(&sp->hash_link,
+ &vcpu->kvm->arch.mmu_page_hash
+ [kvm_page_table_hashfn(gfn)]);
+ clear_page(sp->spt);
+out:
+ return sp;
+}
+
+static void link_spp_shadow_page(struct kvm_vcpu *vcpu, u64 *sptep,
+ struct kvm_mmu_page *sp)
+{
+ u64 spte;
+
+ spte = __pa(sp->spt) | PT_PRESENT_MASK;
+
+ mmu_spte_set(sptep, spte);
+
+ mmu_page_add_parent_pte(vcpu, sp, sptep);
+}
+
+static u64 format_spp_spte(u32 spp_wp_bitmap)
+{
+ u64 new_spte = 0;
+ int i = 0;
+
+ /*
+ * One 4K-page contains 32 sub-pages, they're flagged in even bits in
+ * SPPT L4E, the odd bits are reserved now, so convert 4-byte write
+ * permission bitmap to 8-byte SPP L4E format.
+ */
+ for (i = 0; i < 32; i++)
+ new_spte |= (spp_wp_bitmap & BIT_ULL(i)) << i;
+
+ return new_spte;
+}
+
+static void spp_spte_set(u64 *sptep, u64 new_spte)
+{
+ __set_spte(sptep, new_spte);
+}
+
+int kvm_spp_setup_structure(struct kvm_vcpu *vcpu,
+ u32 access_map, gfn_t gfn)
+{
+ struct kvm_shadow_walk_iterator iter;
+ struct kvm_mmu_page *sp;
+ gfn_t pseudo_gfn;
+ u64 old_spte, spp_spte;
+ int ret = -EFAULT;
+
+ if (!VALID_PAGE(vcpu->kvm->arch.sppt_root))
+ return -EFAULT;
+
+ for_each_shadow_spp_entry(vcpu, (u64)gfn << PAGE_SHIFT, iter) {
+ if (iter.level == PT_PAGE_TABLE_LEVEL) {
+ spp_spte = format_spp_spte(access_map);
+ old_spte = mmu_spte_get_lockless(iter.sptep);
+ if (old_spte != spp_spte)
+ spp_spte_set(iter.sptep, spp_spte);
+ ret = 0;
+ break;
+ }
+
+ if (!is_shadow_present_pte(*iter.sptep)) {
+ u64 base_addr = iter.addr;
+
+ base_addr &= PT64_LVL_ADDR_MASK(iter.level);
+ pseudo_gfn = base_addr >> PAGE_SHIFT;
+ sp = kvm_spp_get_page(vcpu, pseudo_gfn,
+ iter.level - 1);
+ link_spp_shadow_page(vcpu, iter.sptep, sp);
+ } else if (iter.level == PT_DIRECTORY_LEVEL) {
+ spp_spte = mmu_spte_get_lockless(iter.sptep);
+ if (!(spp_spte & PT_PRESENT_MASK) &&
+ (spp_spte & PT64_BASE_ADDR_MASK)) {
+ spp_spte |= PT_PRESENT_MASK;
+ spp_spte_set(iter.sptep, spp_spte);
+ }
+ }
+ }
+
+ kvm_flush_remote_tlbs(vcpu->kvm);
+ return ret;
+}
diff --git a/arch/x86/kvm/mmu/spp.h b/arch/x86/kvm/mmu/spp.h
new file mode 100644
index 000000000000..03e4dfad595a
--- /dev/null
+++ b/arch/x86/kvm/mmu/spp.h
@@ -0,0 +1,5 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef __KVM_X86_VMX_SPP_H
+#define __KVM_X86_VMX_SPP_H
+
+#endif /* __KVM_X86_VMX_SPP_H */
--
2.17.2
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH v12 04/11] mmu: spp: Implement functions to {get|set}_subpage permission
2020-05-16 12:54 [PATCH v12 00/11] Enable Sub-Page Write Protection Support Yang Weijiang
` (2 preceding siblings ...)
2020-05-16 12:54 ` [PATCH v12 03/11] mmu: spp: Implement SPPT setup functions Yang Weijiang
@ 2020-05-16 12:55 ` Yang Weijiang
2020-05-16 12:55 ` [PATCH v12 05/11] x86: spp: Introduce user-space SPP IOCTLs Yang Weijiang
` (6 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Yang Weijiang @ 2020-05-16 12:55 UTC (permalink / raw)
To: kvm, linux-kernel, pbonzini, jmattson, sean.j.christopherson
Cc: yu.c.zhang, alazar, edwin.zhai, ssicleru, Yang Weijiang
Userspace application can {set|get} subpage permission via
IOCTLs if SPP has been initialized.
Steps for set_permission:
1)Store the permission vectors to SPP bitmap buffer.
2)Flush existing hugepage mapping in rmap so as to avoid
stale mapping.
3)Walk EPT to check if gfn->pfn 4KB mapping is there,
mark the existing entry as WP and SPP protected.
4)Zap the entry if gfn->pfn is hugepage mapping so that
following memory access can trigger EPT page_fault() to
set up SPP protection.
Co-developed-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: He Chen <he.chen@linux.intel.com>
Co-developed-by: Zhang Yi <yi.z.zhang@linux.intel.com>
Signed-off-by: Zhang Yi <yi.z.zhang@linux.intel.com>
Signed-off-by: Yang Weijiang <weijiang.yang@intel.com>
---
arch/x86/include/asm/kvm_host.h | 3 +
arch/x86/kvm/mmu.h | 2 +
arch/x86/kvm/mmu/spp.c | 244 ++++++++++++++++++++++++++++++++
arch/x86/kvm/mmu/spp.h | 12 ++
arch/x86/kvm/mmu_internal.h | 2 +
arch/x86/kvm/trace.h | 22 +++
arch/x86/kvm/x86.c | 1 +
include/uapi/linux/kvm.h | 8 ++
8 files changed, 294 insertions(+)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index ee4721bd8703..c8fa8a5ebf4b 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -723,6 +723,7 @@ struct kvm_vcpu_arch {
unsigned nmi_pending; /* NMI queued after currently running handler */
bool nmi_injected; /* Trying to inject an NMI this entry */
bool smi_pending; /* SMI queued after currently running handler */
+ bool spp_pending; /* SPP has been requested, need to update VMCS */
struct kvm_mtrr mtrr_state;
u64 pat;
@@ -829,6 +830,7 @@ struct kvm_lpage_info {
struct kvm_arch_memory_slot {
struct kvm_rmap_head *rmap[KVM_NR_PAGE_SIZES];
+ u32 *subpage_wp_info;
struct kvm_lpage_info *lpage_info[KVM_NR_PAGE_SIZES - 1];
unsigned short *gfn_track[KVM_PAGE_TRACK_MAX];
};
@@ -985,6 +987,7 @@ struct kvm_arch {
struct task_struct *nx_lpage_recovery_thread;
hpa_t sppt_root;
+ bool spp_active;
};
struct kvm_vm_stat {
diff --git a/arch/x86/kvm/mmu.h b/arch/x86/kvm/mmu.h
index da199f0a69db..10cf86b3c60a 100644
--- a/arch/x86/kvm/mmu.h
+++ b/arch/x86/kvm/mmu.h
@@ -26,6 +26,8 @@
#define PT_GLOBAL_MASK (1ULL << 8)
#define PT64_NX_SHIFT 63
#define PT64_NX_MASK (1ULL << PT64_NX_SHIFT)
+#define PT_SPP_SHIFT 61
+#define PT_SPP_MASK (1ULL << PT_SPP_SHIFT)
#define PT_PAT_SHIFT 7
#define PT_DIR_PAT_SHIFT 12
diff --git a/arch/x86/kvm/mmu/spp.c b/arch/x86/kvm/mmu/spp.c
index 8924096df390..df5d79b17ef3 100644
--- a/arch/x86/kvm/mmu/spp.c
+++ b/arch/x86/kvm/mmu/spp.c
@@ -2,6 +2,7 @@
#include <linux/kvm_host.h>
#include "mmu_internal.h"
#include "mmu.h"
+#include "trace.h"
#include "spp.h"
#define for_each_shadow_spp_entry(_vcpu, _addr, _walker) \
@@ -19,6 +20,68 @@ static void shadow_spp_walk_init(struct kvm_shadow_walk_iterator *iterator,
iterator->level = PT64_ROOT_4LEVEL;
}
+u32 *gfn_to_subpage_wp_info(struct kvm_memory_slot *slot, gfn_t gfn)
+{
+ unsigned long idx;
+
+ if (!slot->arch.subpage_wp_info)
+ return NULL;
+
+ idx = gfn_to_index(gfn, slot->base_gfn, PT_PAGE_TABLE_LEVEL);
+ if (idx > slot->npages - 1)
+ return NULL;
+
+ return &slot->arch.subpage_wp_info[idx];
+}
+
+static bool __rmap_update_subpage_bit(struct kvm *kvm,
+ struct kvm_rmap_head *rmap_head,
+ bool setbit)
+{
+ struct rmap_iterator iter;
+ bool flush = false;
+ u64 *sptep;
+ u64 spte;
+
+ for_each_rmap_spte(rmap_head, &iter, sptep) {
+ /*
+ * SPP works only when the page is write-protected
+ * and SPP bit is set in EPT leaf entry.
+ */
+ flush |= spte_write_protect(sptep, false);
+ spte = setbit ? (*sptep | PT_SPP_MASK) :
+ (*sptep & ~PT_SPP_MASK);
+ flush |= mmu_spte_update(sptep, spte);
+ }
+
+ return flush;
+}
+
+static int kvm_spp_update_write_protect(struct kvm *kvm,
+ struct kvm_memory_slot *slot,
+ gfn_t gfn,
+ bool enable)
+{
+ struct kvm_rmap_head *rmap_head;
+ bool flush = false;
+
+ /*
+ * SPP is only supported with 4KB level1 memory page, check
+ * if the page is mapped in EPT leaf entry.
+ */
+ rmap_head = __gfn_to_rmap(gfn, PT_PAGE_TABLE_LEVEL, slot);
+
+ if (!rmap_head->val)
+ return -EFAULT;
+
+ flush |= __rmap_update_subpage_bit(kvm, rmap_head, enable);
+
+ if (flush)
+ kvm_flush_remote_tlbs(kvm);
+
+ return 0;
+}
+
struct kvm_mmu_page *kvm_spp_get_page(struct kvm_vcpu *vcpu,
gfn_t gfn,
unsigned int level)
@@ -84,6 +147,20 @@ static void spp_spte_set(u64 *sptep, u64 new_spte)
__set_spte(sptep, new_spte);
}
+static int kvm_spp_level_pages(gfn_t gfn_lower, gfn_t gfn_upper, int level)
+{
+ int page_num = KVM_PAGES_PER_HPAGE(level);
+ gfn_t gfn_max = (gfn_lower & ~(page_num - 1)) + page_num - 1;
+ int ret;
+
+ if (gfn_upper <= gfn_max)
+ ret = gfn_upper - gfn_lower + 1;
+ else
+ ret = gfn_max - gfn_lower + 1;
+
+ return ret;
+}
+
int kvm_spp_setup_structure(struct kvm_vcpu *vcpu,
u32 access_map, gfn_t gfn)
{
@@ -127,3 +204,170 @@ int kvm_spp_setup_structure(struct kvm_vcpu *vcpu,
kvm_flush_remote_tlbs(vcpu->kvm);
return ret;
}
+
+int kvm_spp_get_permission(struct kvm *kvm, u64 gfn, u32 npages,
+ u32 *access_map)
+{
+ u32 *access;
+ struct kvm_memory_slot *slot;
+ int i;
+
+ if (!kvm->arch.spp_active)
+ return -ENODEV;
+
+ for (i = 0; i < npages; i++, gfn++) {
+ slot = gfn_to_memslot(kvm, gfn);
+ if (!slot)
+ return -EFAULT;
+ access = gfn_to_subpage_wp_info(slot, gfn);
+ if (!access)
+ return -EFAULT;
+ access_map[i] = *access;
+ }
+
+ return i;
+}
+
+static void kvm_spp_zap_pte(struct kvm *kvm, u64 *spte, int level)
+{
+ u64 pte;
+
+ pte = *spte;
+ if (is_shadow_present_pte(pte) && is_last_spte(pte, level)) {
+ drop_spte(kvm, spte);
+ if (is_large_pte(pte))
+ --kvm->stat.lpages;
+ }
+}
+
+static bool kvm_spp_flush_rmap(struct kvm *kvm, u64 gfn_min, u64 gfn_max)
+{
+ u64 *sptep;
+ struct rmap_iterator iter;
+ struct kvm_rmap_head *rmap_head;
+ int level;
+ struct kvm_memory_slot *slot;
+ bool flush = false;
+
+ slot = gfn_to_memslot(kvm, gfn_min);
+ if (!slot)
+ return false;
+
+ for (; gfn_min <= gfn_max; gfn_min++) {
+ for (level = PT_PAGE_TABLE_LEVEL;
+ level <= PT_DIRECTORY_LEVEL; level++) {
+ rmap_head = __gfn_to_rmap(gfn_min, level, slot);
+ for_each_rmap_spte(rmap_head, &iter, sptep) {
+ pte_list_remove(rmap_head, sptep);
+ flush = true;
+ }
+ }
+ }
+
+ return flush;
+}
+
+int kvm_spp_set_permission(struct kvm *kvm, u64 gfn, u32 npages,
+ u32 *access_map)
+{
+ gfn_t old_gfn = gfn;
+ u32 *access;
+ struct kvm_memory_slot *slot;
+ struct kvm_shadow_walk_iterator iterator;
+ struct kvm_vcpu *vcpu;
+ gfn_t gfn_end;
+ int i, count, level;
+ bool flush = false;
+
+ if (!kvm->arch.spp_active)
+ return -ENODEV;
+
+ vcpu = kvm_get_vcpu(kvm, 0);
+ if (!VALID_PAGE(vcpu->kvm->arch.sppt_root))
+ return -EFAULT;
+
+ for (i = 0; i < npages; i++, gfn++) {
+ slot = gfn_to_memslot(kvm, gfn);
+ if (!slot)
+ return -EFAULT;
+
+ access = gfn_to_subpage_wp_info(slot, gfn);
+ if (!access)
+ return -EFAULT;
+ *access = access_map[i];
+ trace_kvm_spp_set_subpages(vcpu, gfn, *access);
+ }
+
+ gfn = old_gfn;
+ gfn_end = gfn + npages - 1;
+ vcpu = kvm_get_vcpu(kvm, 0);
+
+ if (!vcpu || (vcpu && !VALID_PAGE(vcpu->arch.mmu->root_hpa)))
+ goto out;
+
+ /* Flush any existing stale mappings in EPT before set up SPP */
+ flush = kvm_spp_flush_rmap(kvm, gfn, gfn_end);
+
+ for (i = 0; gfn <= gfn_end; i++, gfn++) {
+ for_each_shadow_entry(vcpu, (u64)gfn << PAGE_SHIFT, iterator) {
+ if (!is_shadow_present_pte(*iterator.sptep))
+ break;
+
+ if (iterator.level == PT_PAGE_TABLE_LEVEL) {
+ if (kvm_spp_mark_protection(kvm,
+ gfn,
+ access_map[i]) < 0)
+ return -EFAULT;
+ break;
+ } else if (is_large_pte(*iterator.sptep)) {
+ level = iterator.level;
+ if (access_map[i] == FULL_SPP_ACCESS)
+ break;
+ count = kvm_spp_level_pages(gfn,
+ gfn_end,
+ level);
+ /*
+ * Zap existing hugepage entry so that eligible
+ * 4KB mappings can be rebuilt in page_fault.
+ */
+ kvm_spp_zap_pte(kvm, iterator.sptep, level);
+ flush = true;
+ if (count >= npages)
+ goto out;
+ gfn += count - 1;
+ }
+ }
+ }
+out:
+ if (flush)
+ kvm_flush_remote_tlbs(kvm);
+ return npages;
+}
+
+int kvm_spp_mark_protection(struct kvm *kvm, u64 gfn, u32 access)
+{
+ struct kvm_memory_slot *slot;
+ struct kvm_rmap_head *rmap_head;
+ int ret = 0;
+ bool enable;
+
+ if (!kvm->arch.spp_active)
+ return -ENODEV;
+
+ slot = gfn_to_memslot(kvm, gfn);
+ if (!slot)
+ return -EFAULT;
+
+ /*
+ * check whether the target 4KB page exists in EPT leaf
+ * entry.If it's there, just flag SPP bit of the entry,
+ * defer the setup to SPPT miss induced vm-exit handler.
+ */
+ rmap_head = __gfn_to_rmap(gfn, PT_PAGE_TABLE_LEVEL, slot);
+
+ if (rmap_head->val) {
+ enable = access != FULL_SPP_ACCESS;
+ ret = kvm_spp_update_write_protect(kvm, slot, gfn, enable);
+ }
+ return ret;
+}
diff --git a/arch/x86/kvm/mmu/spp.h b/arch/x86/kvm/mmu/spp.h
index 03e4dfad595a..9171e682be1f 100644
--- a/arch/x86/kvm/mmu/spp.h
+++ b/arch/x86/kvm/mmu/spp.h
@@ -2,4 +2,16 @@
#ifndef __KVM_X86_VMX_SPP_H
#define __KVM_X86_VMX_SPP_H
+#define FULL_SPP_ACCESS (u32)(BIT_ULL(32) - 1)
+
+int kvm_spp_get_permission(struct kvm *kvm, u64 gfn, u32 npages,
+ u32 *access_map);
+int kvm_spp_set_permission(struct kvm *kvm, u64 gfn, u32 npages,
+ u32 *access_map);
+int kvm_spp_mark_protection(struct kvm *kvm, u64 gfn, u32 access);
+
+int kvm_spp_setup_structure(struct kvm_vcpu *vcpu,
+ u32 access_map, gfn_t gfn);
+u32 *gfn_to_subpage_wp_info(struct kvm_memory_slot *slot, gfn_t gfn);
+
#endif /* __KVM_X86_VMX_SPP_H */
diff --git a/arch/x86/kvm/mmu_internal.h b/arch/x86/kvm/mmu_internal.h
index 68e8179e7642..e54594941377 100644
--- a/arch/x86/kvm/mmu_internal.h
+++ b/arch/x86/kvm/mmu_internal.h
@@ -124,6 +124,8 @@ void pte_list_remove(struct kvm_rmap_head *rmap_head, u64 *sptep);
u64 __get_spte_lockless(u64 *sptep);
+void pte_list_remove(struct kvm_rmap_head *rmap_head, u64 *sptep);
+
u64 mmu_spte_get_lockless(u64 *sptep);
unsigned kvm_page_table_hashfn(gfn_t gfn);
diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h
index 249062f24b94..035767345763 100644
--- a/arch/x86/kvm/trace.h
+++ b/arch/x86/kvm/trace.h
@@ -1539,6 +1539,28 @@ TRACE_EVENT(kvm_nested_vmenter_failed,
__print_symbolic(__entry->err, VMX_VMENTER_INSTRUCTION_ERRORS))
);
+TRACE_EVENT(kvm_spp_set_subpages,
+ TP_PROTO(struct kvm_vcpu *vcpu, gfn_t gfn, u32 access),
+ TP_ARGS(vcpu, gfn, access),
+
+ TP_STRUCT__entry(
+ __field(int, vcpu_id)
+ __field(gfn_t, gfn)
+ __field(u32, access)
+ ),
+
+ TP_fast_assign(
+ __entry->vcpu_id = vcpu->vcpu_id;
+ __entry->gfn = gfn;
+ __entry->access = access;
+ ),
+
+ TP_printk("vcpu %d gfn %llx access %x",
+ __entry->vcpu_id,
+ __entry->gfn,
+ __entry->access)
+);
+
#endif /* _TRACE_KVM_H */
#undef TRACE_INCLUDE_PATH
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index d786c7d27ce5..35e4b57dbabf 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -10601,3 +10601,4 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_avic_unaccelerated_access);
EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_avic_incomplete_ipi);
EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_avic_ga_log);
EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_apicv_update_request);
+EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_spp_set_subpages);
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 428c7dde6b4b..63a477720a17 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -102,6 +102,14 @@ struct kvm_userspace_memory_region {
__u64 userspace_addr; /* start of the userspace allocated memory */
};
+/* for KVM_SUBPAGES_GET_ACCESS and KVM_SUBPAGES_SET_ACCESS */
+struct kvm_subpage {
+ __u64 gfn_base; /* the first page gfn of the contiguous pages */
+ __u32 npages; /* number of 4K pages */
+ __u32 flags; /* reserved to 0 now */
+ __u32 access_map[0]; /* start place of bitmap array */
+};
+
/*
* The bit 0 ~ bit 15 of kvm_memory_region::flags are visible for userspace,
* other bits are reserved for kvm internal use which are defined in
--
2.17.2
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH v12 05/11] x86: spp: Introduce user-space SPP IOCTLs
2020-05-16 12:54 [PATCH v12 00/11] Enable Sub-Page Write Protection Support Yang Weijiang
` (3 preceding siblings ...)
2020-05-16 12:55 ` [PATCH v12 04/11] mmu: spp: Implement functions to {get|set}_subpage permission Yang Weijiang
@ 2020-05-16 12:55 ` Yang Weijiang
2020-05-16 12:55 ` [PATCH v12 06/11] vmx: spp: Handle SPP induced vmexit and EPT violation Yang Weijiang
` (5 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Yang Weijiang @ 2020-05-16 12:55 UTC (permalink / raw)
To: kvm, linux-kernel, pbonzini, jmattson, sean.j.christopherson
Cc: yu.c.zhang, alazar, edwin.zhai, ssicleru, Yang Weijiang
Userspace application, e.g., QEMU or VMI, must initialize SPP
before {gets|sets} SPP subpages.
Co-developed-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: He Chen <he.chen@linux.intel.com>
Co-developed-by: Zhang Yi <yi.z.zhang@linux.intel.com>
Signed-off-by: Zhang Yi <yi.z.zhang@linux.intel.com>
Signed-off-by: Yang Weijiang <weijiang.yang@intel.com>
---
arch/x86/kvm/mmu/spp.c | 79 ++++++++++++++++++++++++++++++++++++
arch/x86/kvm/mmu/spp.h | 11 ++++-
arch/x86/kvm/x86.c | 88 ++++++++++++++++++++++++++++++++++++++++
include/uapi/linux/kvm.h | 2 +
4 files changed, 179 insertions(+), 1 deletion(-)
diff --git a/arch/x86/kvm/mmu/spp.c b/arch/x86/kvm/mmu/spp.c
index df5d79b17ef3..25207f1ac9f8 100644
--- a/arch/x86/kvm/mmu/spp.c
+++ b/arch/x86/kvm/mmu/spp.c
@@ -205,6 +205,46 @@ int kvm_spp_setup_structure(struct kvm_vcpu *vcpu,
return ret;
}
+int spp_flush_sppt(struct kvm *kvm, u64 gfn_base, u32 npages)
+{
+ struct kvm_shadow_walk_iterator iter;
+ struct kvm_vcpu *vcpu;
+ gfn_t gfn = gfn_base;
+ gfn_t gfn_end = gfn_base + npages - 1;
+ u64 spde;
+ int count;
+ bool flush = false;
+
+ vcpu = kvm_get_vcpu(kvm, 0);
+ if (!VALID_PAGE(vcpu->kvm->arch.sppt_root))
+ return -EFAULT;
+
+ for (; gfn <= gfn_end; gfn++) {
+ for_each_shadow_spp_entry(vcpu, (u64)gfn << PAGE_SHIFT, iter) {
+ if (!is_shadow_present_pte(*iter.sptep))
+ break;
+
+ if (iter.level != PT_DIRECTORY_LEVEL)
+ continue;
+
+ spde = *iter.sptep;
+ spde &= ~PT_PRESENT_MASK;
+ spp_spte_set(iter.sptep, spde);
+ count = kvm_spp_level_pages(gfn, gfn_end,
+ PT_DIRECTORY_LEVEL);
+ flush = true;
+ if (count >= npages)
+ goto out;
+ gfn += count;
+ break;
+ }
+ }
+out:
+ if (flush)
+ kvm_flush_remote_tlbs(kvm);
+ return 0;
+}
+
int kvm_spp_get_permission(struct kvm *kvm, u64 gfn, u32 npages,
u32 *access_map)
{
@@ -371,3 +411,42 @@ int kvm_spp_mark_protection(struct kvm *kvm, u64 gfn, u32 access)
}
return ret;
}
+
+int kvm_vm_ioctl_get_subpages(struct kvm *kvm,
+ u64 gfn,
+ u32 npages,
+ u32 *access_map)
+{
+ int ret;
+
+ mutex_lock(&kvm->slots_lock);
+ ret = kvm_spp_get_permission(kvm, gfn, npages, access_map);
+ mutex_unlock(&kvm->slots_lock);
+
+ return ret;
+}
+
+int kvm_vm_ioctl_set_subpages(struct kvm *kvm,
+ u64 gfn,
+ u32 npages,
+ u32 *access_map)
+{
+ int ret;
+
+ spin_lock(&kvm->mmu_lock);
+ ret = spp_flush_sppt(kvm, gfn, npages);
+ spin_unlock(&kvm->mmu_lock);
+
+ if (ret < 0)
+ return ret;
+
+ mutex_lock(&kvm->slots_lock);
+ spin_lock(&kvm->mmu_lock);
+
+ ret = kvm_spp_set_permission(kvm, gfn, npages, access_map);
+
+ spin_unlock(&kvm->mmu_lock);
+ mutex_unlock(&kvm->slots_lock);
+
+ return ret;
+}
diff --git a/arch/x86/kvm/mmu/spp.h b/arch/x86/kvm/mmu/spp.h
index 9171e682be1f..75d4bfd64dbd 100644
--- a/arch/x86/kvm/mmu/spp.h
+++ b/arch/x86/kvm/mmu/spp.h
@@ -3,15 +3,24 @@
#define __KVM_X86_VMX_SPP_H
#define FULL_SPP_ACCESS (u32)(BIT_ULL(32) - 1)
+#define KVM_SUBPAGE_MAX_PAGES 512
int kvm_spp_get_permission(struct kvm *kvm, u64 gfn, u32 npages,
u32 *access_map);
int kvm_spp_set_permission(struct kvm *kvm, u64 gfn, u32 npages,
u32 *access_map);
int kvm_spp_mark_protection(struct kvm *kvm, u64 gfn, u32 access);
-
+int kvm_vm_ioctl_get_subpages(struct kvm *kvm,
+ u64 gfn,
+ u32 npages,
+ u32 *access_map);
+int kvm_vm_ioctl_set_subpages(struct kvm *kvm,
+ u64 gfn,
+ u32 npages,
+ u32 *access_map);
int kvm_spp_setup_structure(struct kvm_vcpu *vcpu,
u32 access_map, gfn_t gfn);
u32 *gfn_to_subpage_wp_info(struct kvm_memory_slot *slot, gfn_t gfn);
+int spp_flush_sppt(struct kvm *kvm, u64 gfn_base, u32 npages);
#endif /* __KVM_X86_VMX_SPP_H */
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 35e4b57dbabf..c0b09d8be31b 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -28,6 +28,7 @@
#include "pmu.h"
#include "hyperv.h"
#include "lapic.h"
+#include "mmu/spp.h"
#include <linux/clocksource.h>
#include <linux/interrupt.h>
@@ -5196,6 +5197,93 @@ long kvm_arch_vm_ioctl(struct file *filp,
case KVM_SET_PMU_EVENT_FILTER:
r = kvm_vm_ioctl_set_pmu_event_filter(kvm, argp);
break;
+ case KVM_SUBPAGES_GET_ACCESS: {
+ struct kvm_subpage spp_info, *pinfo;
+ u32 total;
+
+ r = -ENODEV;
+ if (!kvm->arch.spp_active)
+ goto out;
+
+ r = -EFAULT;
+ if (copy_from_user(&spp_info, argp, sizeof(spp_info)))
+ goto out;
+
+ r = -EINVAL;
+ if (spp_info.flags != 0 ||
+ spp_info.npages > KVM_SUBPAGE_MAX_PAGES)
+ goto out;
+ r = 0;
+ if (!spp_info.npages)
+ goto out;
+
+ total = sizeof(spp_info) +
+ sizeof(spp_info.access_map[0]) * spp_info.npages;
+ pinfo = kvzalloc(total, GFP_KERNEL_ACCOUNT);
+
+ r = -ENOMEM;
+ if (!pinfo)
+ goto out;
+
+ r = -EFAULT;
+ if (copy_from_user(pinfo, argp, total))
+ goto out;
+
+ r = kvm_vm_ioctl_get_subpages(kvm,
+ pinfo->gfn_base,
+ pinfo->npages,
+ pinfo->access_map);
+ if (r != pinfo->npages)
+ goto out;
+
+ r = -EFAULT;
+ if (copy_to_user(argp, pinfo, total))
+ goto out;
+
+ r = pinfo->npages;
+ kfree(pinfo);
+ break;
+ }
+ case KVM_SUBPAGES_SET_ACCESS: {
+ struct kvm_subpage spp_info, *pinfo;
+ u32 total;
+
+ r = -ENODEV;
+ if (!kvm->arch.spp_active)
+ goto out;
+
+ r = -EFAULT;
+ if (copy_from_user(&spp_info, argp, sizeof(spp_info)))
+ goto out;
+
+ r = -EINVAL;
+ if (spp_info.flags != 0 ||
+ spp_info.npages > KVM_SUBPAGE_MAX_PAGES)
+ goto out;
+
+ r = 0;
+ if (!spp_info.npages)
+ goto out;
+
+ total = sizeof(spp_info) +
+ sizeof(spp_info.access_map[0]) * spp_info.npages;
+ pinfo = kvzalloc(total, GFP_KERNEL_ACCOUNT);
+
+ r = -ENOMEM;
+ if (!pinfo)
+ goto out;
+
+ r = -EFAULT;
+ if (copy_from_user(pinfo, argp, total))
+ goto out;
+
+ r = kvm_vm_ioctl_set_subpages(kvm,
+ pinfo->gfn_base,
+ pinfo->npages,
+ pinfo->access_map);
+ kfree(pinfo);
+ break;
+ }
default:
r = -ENOTTY;
}
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 63a477720a17..2b70cf0d402a 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1280,6 +1280,8 @@ struct kvm_vfio_spapr_tce {
struct kvm_userspace_memory_region)
#define KVM_SET_TSS_ADDR _IO(KVMIO, 0x47)
#define KVM_SET_IDENTITY_MAP_ADDR _IOW(KVMIO, 0x48, __u64)
+#define KVM_SUBPAGES_GET_ACCESS _IOR(KVMIO, 0x49, __u64)
+#define KVM_SUBPAGES_SET_ACCESS _IOW(KVMIO, 0x4a, __u64)
/* enable ucontrol for s390 */
struct kvm_s390_ucas_mapping {
--
2.17.2
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH v12 06/11] vmx: spp: Handle SPP induced vmexit and EPT violation
2020-05-16 12:54 [PATCH v12 00/11] Enable Sub-Page Write Protection Support Yang Weijiang
` (4 preceding siblings ...)
2020-05-16 12:55 ` [PATCH v12 05/11] x86: spp: Introduce user-space SPP IOCTLs Yang Weijiang
@ 2020-05-16 12:55 ` Yang Weijiang
2020-05-16 12:55 ` [PATCH v12 07/11] mmu: spp: Enable Lazy mode SPP protection Yang Weijiang
` (4 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Yang Weijiang @ 2020-05-16 12:55 UTC (permalink / raw)
To: kvm, linux-kernel, pbonzini, jmattson, sean.j.christopherson
Cc: yu.c.zhang, alazar, edwin.zhai, ssicleru, Yang Weijiang
If write to subpage is prohibited, EPT violation is generated
and handled in fast_page_fault().
In current implementation, SPPT setup is handled in handle_spp()
handler, it's triggered when SPP is enabled in EPT leaf entry
while SPPT entry is invalid.
There could be two kinds of SPP usages, one is for write-protection,
the other is for access-tracking, the differece is the former keeps
memory unchange while the latter just records the memory access and
may let the write take effect. To fit these two cases, when SPP induced
vmexit to userspace, the fault instruction length is returned, the
application may take action according to the specific use-case, re-do
write operation or discard it.
To make SPP operatable with dirty-logging, introduce a free bit in
EPT entry to store SPP bit, after dirty-logging happened, it restores
SPP bit and make entry SPP protected again so that a retry write will
trigger a normal SPP induced vmexit.
Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Co-developed-by: He Chen <he.chen@linux.intel.com>
Signed-off-by: He Chen <he.chen@linux.intel.com>
Co-developed-by: Zhang Yi <yi.z.zhang@linux.intel.com>
Signed-off-by: Zhang Yi <yi.z.zhang@linux.intel.com>
Signed-off-by: Yang Weijiang <weijiang.yang@intel.com>
---
arch/x86/include/asm/kvm_host.h | 2 +
arch/x86/include/asm/vmx.h | 9 ++++
arch/x86/include/uapi/asm/vmx.h | 2 +
arch/x86/kvm/mmu/mmu.c | 92 ++++++++++++++++++++++++++-------
arch/x86/kvm/mmu/spp.c | 27 ++++++++++
arch/x86/kvm/mmu/spp.h | 5 ++
arch/x86/kvm/mmutrace.h | 10 ++--
arch/x86/kvm/trace.h | 44 ++++++++++++++++
arch/x86/kvm/vmx/vmx.c | 74 ++++++++++++++++++++++++++
arch/x86/kvm/x86.c | 2 +
include/uapi/linux/kvm.h | 6 +++
11 files changed, 248 insertions(+), 25 deletions(-)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index c8fa8a5ebf4b..d34c0b91d427 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1260,6 +1260,8 @@ struct kvm_x86_ops {
bool (*apic_init_signal_blocked)(struct kvm_vcpu *vcpu);
int (*enable_direct_tlbflush)(struct kvm_vcpu *vcpu);
+
+ int (*get_insn_len)(struct kvm_vcpu *vcpu);
};
struct kvm_x86_init_ops {
diff --git a/arch/x86/include/asm/vmx.h b/arch/x86/include/asm/vmx.h
index 5e090d1f03f8..93000497ddd9 100644
--- a/arch/x86/include/asm/vmx.h
+++ b/arch/x86/include/asm/vmx.h
@@ -217,6 +217,8 @@ enum vmcs_field {
XSS_EXIT_BITMAP_HIGH = 0x0000202D,
ENCLS_EXITING_BITMAP = 0x0000202E,
ENCLS_EXITING_BITMAP_HIGH = 0x0000202F,
+ SPPT_POINTER = 0x00002030,
+ SPPT_POINTER_HIGH = 0x00002031,
TSC_MULTIPLIER = 0x00002032,
TSC_MULTIPLIER_HIGH = 0x00002033,
GUEST_PHYSICAL_ADDRESS = 0x00002400,
@@ -550,6 +552,13 @@ struct vmx_msr_entry {
#define EPT_VIOLATION_EXECUTABLE (1 << EPT_VIOLATION_EXECUTABLE_BIT)
#define EPT_VIOLATION_GVA_TRANSLATED (1 << EPT_VIOLATION_GVA_TRANSLATED_BIT)
+/*
+ * Exit Qualifications for SPPT-Induced vmexits
+ */
+#define SPPT_INDUCED_EXIT_TYPE_BIT 11
+#define SPPT_INDUCED_EXIT_TYPE (1 << SPPT_INDUCED_EXIT_TYPE_BIT)
+#define SPPT_INTR_INFO_UNBLOCK_NMI INTR_INFO_UNBLOCK_NMI
+
/*
* VM-instruction error numbers
*/
diff --git a/arch/x86/include/uapi/asm/vmx.h b/arch/x86/include/uapi/asm/vmx.h
index e95b72ec19bc..ab58c58e0b1f 100644
--- a/arch/x86/include/uapi/asm/vmx.h
+++ b/arch/x86/include/uapi/asm/vmx.h
@@ -86,6 +86,7 @@
#define EXIT_REASON_PML_FULL 62
#define EXIT_REASON_XSAVES 63
#define EXIT_REASON_XRSTORS 64
+#define EXIT_REASON_SPP 66
#define EXIT_REASON_UMWAIT 67
#define EXIT_REASON_TPAUSE 68
@@ -145,6 +146,7 @@
{ EXIT_REASON_ENCLS, "ENCLS" }, \
{ EXIT_REASON_RDSEED, "RDSEED" }, \
{ EXIT_REASON_PML_FULL, "PML_FULL" }, \
+ { EXIT_REASON_SPP, "SPP" }, \
{ EXIT_REASON_XSAVES, "XSAVES" }, \
{ EXIT_REASON_XRSTORS, "XRSTORS" }, \
{ EXIT_REASON_UMWAIT, "UMWAIT" }, \
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 615effaf5814..270dd567272e 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -22,6 +22,7 @@
#include "kvm_cache_regs.h"
#include "kvm_emulate.h"
#include "cpuid.h"
+#include "spp.h"
#include <linux/kvm_host.h>
#include <linux/types.h>
@@ -170,6 +171,7 @@ module_param(dbg, bool, 0644);
/* The mask for the R/X bits in EPT PTEs */
#define PT64_EPT_READABLE_MASK 0x1ull
#define PT64_EPT_EXECUTABLE_MASK 0x4ull
+#define PT64_SPP_SAVED_BIT (1ULL << (PT64_SECOND_AVAIL_BITS_SHIFT + 1))
#include <trace/events/kvm.h>
@@ -188,6 +190,7 @@ enum {
RET_PF_RETRY = 0,
RET_PF_EMULATE = 1,
RET_PF_INVALID = 2,
+ RET_PF_USERSPACE = 3,
};
#define for_each_shadow_entry_lockless(_vcpu, _addr, _walker, spte) \
@@ -947,6 +950,9 @@ static u64 mark_spte_for_access_track(u64 spte)
shadow_acc_track_saved_bits_shift;
spte &= ~shadow_acc_track_mask;
+ if (spte & PT_SPP_MASK)
+ save_spp_bit(&spte);
+
return spte;
}
@@ -1512,9 +1518,16 @@ static void drop_large_spte(struct kvm_vcpu *vcpu, u64 *sptep)
bool spte_write_protect(u64 *sptep, bool pt_protect)
{
u64 spte = *sptep;
+ bool spp_protected = false;
+
+ if (spte & PT_SPP_MASK) {
+ save_spp_bit(&spte);
+ spp_protected = true;
+ }
if (!is_writable_pte(spte) &&
- !(pt_protect && spte_can_locklessly_be_made_writable(spte)))
+ !(pt_protect && spte_can_locklessly_be_made_writable(spte)) &&
+ !spp_protected)
return false;
rmap_printk("rmap_write_protect: spte %p %llx\n", sptep, *sptep);
@@ -1555,9 +1568,15 @@ static bool spte_wrprot_for_clear_dirty(u64 *sptep)
{
bool was_writable = test_and_clear_bit(PT_WRITABLE_SHIFT,
(unsigned long *)sptep);
+ bool was_spp_armed = test_and_clear_bit(PT_SPP_SHIFT,
+ (unsigned long *)sptep);
+
if (was_writable && !spte_ad_enabled(*sptep))
kvm_set_pfn_dirty(spte_to_pfn(*sptep));
+ if (was_spp_armed)
+ *sptep |= PT64_SPP_SAVED_BIT;
+
return was_writable;
}
@@ -3399,7 +3418,8 @@ static bool page_fault_can_be_fast(u32 error_code)
*/
static bool
fast_pf_fix_direct_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
- u64 *sptep, u64 old_spte, u64 new_spte)
+ u64 *sptep, u64 old_spte, u64 new_spte,
+ bool spp_protected)
{
gfn_t gfn;
@@ -3420,7 +3440,8 @@ fast_pf_fix_direct_spte(struct kvm_vcpu *vcpu, struct kvm_mmu_page *sp,
if (cmpxchg64(sptep, old_spte, new_spte) != old_spte)
return false;
- if (is_writable_pte(new_spte) && !is_writable_pte(old_spte)) {
+ if ((is_writable_pte(new_spte) && !is_writable_pte(old_spte)) ||
+ spp_protected) {
/*
* The gfn of direct spte is stable since it is
* calculated by sp->gfn.
@@ -3446,15 +3467,17 @@ static bool is_access_allowed(u32 fault_err_code, u64 spte)
/*
* Return value:
- * - true: let the vcpu to access on the same address again.
- * - false: let the real page fault path to fix it.
+ * - RET_PF_INVALID: let the real page fault path to fix it.
+ * - RET_PF_RETRY: the fault has been fixed here.
+ * - RET_PF_USERSPACE: exit to user space to further handle it.
*/
-static bool fast_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
+static int fast_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
u32 error_code)
{
struct kvm_shadow_walk_iterator iterator;
struct kvm_mmu_page *sp;
- bool fault_handled = false;
+ int ret = RET_PF_INVALID;
+ bool spp_protected = false;
u64 spte = 0ull;
uint retry_count = 0;
@@ -3485,7 +3508,7 @@ static bool fast_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
* they are always ACC_ALL.
*/
if (is_access_allowed(error_code, spte)) {
- fault_handled = true;
+ ret = RET_PF_RETRY;
break;
}
@@ -3501,7 +3524,30 @@ static bool fast_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
*/
if ((error_code & PFERR_WRITE_MASK) &&
spte_can_locklessly_be_made_writable(spte)) {
- new_spte |= PT_WRITABLE_MASK;
+ /*
+ * Record write protect fault caused by
+ * Sub-page Protection, let VMI decide
+ * the next step.
+ */
+ if (spte & PT_SPP_MASK) {
+ int len = kvm_x86_ops.get_insn_len(vcpu);
+
+ ret = RET_PF_USERSPACE;
+ vcpu->run->exit_reason = KVM_EXIT_SPP;
+ vcpu->run->spp.addr = cr2_or_gpa;
+ vcpu->run->spp.insn_len = len;
+ trace_kvm_spp_induced_page_fault(vcpu,
+ cr2_or_gpa,
+ len);
+ break;
+ }
+
+ if (was_spp_armed(new_spte)) {
+ restore_spp_bit(&new_spte);
+ spp_protected = true;
+ } else {
+ new_spte |= PT_WRITABLE_MASK;
+ }
/*
* Do not fix write-permission on the large spte. Since
@@ -3520,7 +3566,8 @@ static bool fast_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
/* Verify that the fault can be handled in the fast path */
if (new_spte == spte ||
- !is_access_allowed(error_code, new_spte))
+ (!is_access_allowed(error_code, new_spte) &&
+ !spp_protected))
break;
/*
@@ -3528,11 +3575,12 @@ static bool fast_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
* since the gfn is not stable for indirect shadow page. See
* Documentation/virt/kvm/locking.txt to get more detail.
*/
- fault_handled = fast_pf_fix_direct_spte(vcpu, sp,
- iterator.sptep, spte,
- new_spte);
- if (fault_handled)
- break;
+ if (fast_pf_fix_direct_spte(vcpu, sp, iterator.sptep, spte,
+ new_spte,
+ spp_protected))
+ ret = RET_PF_RETRY;
+ else
+ ret = RET_PF_INVALID;
if (++retry_count > 4) {
printk_once(KERN_WARNING
@@ -3543,10 +3591,10 @@ static bool fast_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
} while (true);
trace_fast_page_fault(vcpu, cr2_or_gpa, error_code, iterator.sptep,
- spte, fault_handled);
+ spte, ret);
walk_shadow_page_lockless_end(vcpu);
- return fault_handled;
+ return ret;
}
static void mmu_free_root_page(struct kvm *kvm, hpa_t *root_hpa,
@@ -4077,8 +4125,10 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,
if (lpage_disallowed)
max_level = PT_PAGE_TABLE_LEVEL;
- if (fast_page_fault(vcpu, gpa, error_code))
- return RET_PF_RETRY;
+ r = fast_page_fault(vcpu, gpa, error_code);
+
+ if (r != RET_PF_INVALID)
+ return r;
mmu_seq = vcpu->kvm->mmu_notifier_seq;
smp_rmb();
@@ -5077,7 +5127,6 @@ void kvm_init_mmu(struct kvm_vcpu *vcpu, bool reset_roots)
uint i;
vcpu->arch.mmu->root_hpa = INVALID_PAGE;
-
for (i = 0; i < KVM_MMU_NUM_PREV_ROOTS; i++)
vcpu->arch.mmu->prev_roots[i] = KVM_MMU_ROOT_INFO_INVALID;
}
@@ -5385,6 +5434,7 @@ int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 error_code,
if (r == RET_PF_INVALID) {
r = kvm_mmu_do_page_fault(vcpu, cr2_or_gpa,
lower_32_bits(error_code), false);
+
WARN_ON(r == RET_PF_INVALID);
}
@@ -5393,6 +5443,8 @@ int kvm_mmu_page_fault(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u64 error_code,
if (r < 0)
return r;
+ if (r == RET_PF_USERSPACE)
+ return 0;
/*
* Before emulating the instruction, check if the error code
* was due to a RO violation while translating the guest page.
diff --git a/arch/x86/kvm/mmu/spp.c b/arch/x86/kvm/mmu/spp.c
index 25207f1ac9f8..bb8f298d72da 100644
--- a/arch/x86/kvm/mmu/spp.c
+++ b/arch/x86/kvm/mmu/spp.c
@@ -20,6 +20,25 @@ static void shadow_spp_walk_init(struct kvm_shadow_walk_iterator *iterator,
iterator->level = PT64_ROOT_4LEVEL;
}
+/* Save reserved bit for SPP armed PTE */
+void save_spp_bit(u64 *spte)
+{
+ *spte |= PT64_SPP_SAVED_BIT;
+ *spte &= ~PT_SPP_MASK;
+}
+
+/* Restore reserved bit for SPP armed PTE */
+void restore_spp_bit(u64 *spte)
+{
+ *spte &= ~PT64_SPP_SAVED_BIT;
+ *spte |= PT_SPP_MASK;
+}
+
+bool was_spp_armed(u64 spte)
+{
+ return !!(spte & PT64_SPP_SAVED_BIT);
+}
+
u32 *gfn_to_subpage_wp_info(struct kvm_memory_slot *slot, gfn_t gfn)
{
unsigned long idx;
@@ -33,6 +52,7 @@ u32 *gfn_to_subpage_wp_info(struct kvm_memory_slot *slot, gfn_t gfn)
return &slot->arch.subpage_wp_info[idx];
}
+EXPORT_SYMBOL_GPL(gfn_to_subpage_wp_info);
static bool __rmap_update_subpage_bit(struct kvm *kvm,
struct kvm_rmap_head *rmap_head,
@@ -204,6 +224,7 @@ int kvm_spp_setup_structure(struct kvm_vcpu *vcpu,
kvm_flush_remote_tlbs(vcpu->kvm);
return ret;
}
+EXPORT_SYMBOL_GPL(kvm_spp_setup_structure);
int spp_flush_sppt(struct kvm *kvm, u64 gfn_base, u32 npages)
{
@@ -450,3 +471,9 @@ int kvm_vm_ioctl_set_subpages(struct kvm *kvm,
return ret;
}
+
+inline u64 construct_spptp(unsigned long root_hpa)
+{
+ return root_hpa & PAGE_MASK;
+}
+EXPORT_SYMBOL_GPL(construct_spptp);
diff --git a/arch/x86/kvm/mmu/spp.h b/arch/x86/kvm/mmu/spp.h
index 75d4bfd64dbd..c3588c20be52 100644
--- a/arch/x86/kvm/mmu/spp.h
+++ b/arch/x86/kvm/mmu/spp.h
@@ -4,6 +4,7 @@
#define FULL_SPP_ACCESS (u32)(BIT_ULL(32) - 1)
#define KVM_SUBPAGE_MAX_PAGES 512
+#define MAX_ENTRIES_PER_MMUPAGE BIT(9)
int kvm_spp_get_permission(struct kvm *kvm, u64 gfn, u32 npages,
u32 *access_map);
@@ -22,5 +23,9 @@ int kvm_spp_setup_structure(struct kvm_vcpu *vcpu,
u32 access_map, gfn_t gfn);
u32 *gfn_to_subpage_wp_info(struct kvm_memory_slot *slot, gfn_t gfn);
int spp_flush_sppt(struct kvm *kvm, u64 gfn_base, u32 npages);
+void save_spp_bit(u64 *spte);
+void restore_spp_bit(u64 *spte);
+bool was_spp_armed(u64 spte);
+u64 construct_spptp(unsigned long root_hpa);
#endif /* __KVM_X86_VMX_SPP_H */
diff --git a/arch/x86/kvm/mmutrace.h b/arch/x86/kvm/mmutrace.h
index ffcd96fc02d0..76942390bcc2 100644
--- a/arch/x86/kvm/mmutrace.h
+++ b/arch/x86/kvm/mmutrace.h
@@ -245,13 +245,13 @@ TRACE_EVENT(
);
#define __spte_satisfied(__spte) \
- (__entry->retry && is_writable_pte(__entry->__spte))
+ (__entry->ret == RET_PF_RETRY && is_writable_pte(__entry->__spte))
TRACE_EVENT(
fast_page_fault,
TP_PROTO(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa, u32 error_code,
- u64 *sptep, u64 old_spte, bool retry),
- TP_ARGS(vcpu, cr2_or_gpa, error_code, sptep, old_spte, retry),
+ u64 *sptep, u64 old_spte, int ret),
+ TP_ARGS(vcpu, cr2_or_gpa, error_code, sptep, old_spte, ret),
TP_STRUCT__entry(
__field(int, vcpu_id)
@@ -260,7 +260,7 @@ TRACE_EVENT(
__field(u64 *, sptep)
__field(u64, old_spte)
__field(u64, new_spte)
- __field(bool, retry)
+ __field(int, ret)
),
TP_fast_assign(
@@ -270,7 +270,7 @@ TRACE_EVENT(
__entry->sptep = sptep;
__entry->old_spte = old_spte;
__entry->new_spte = *sptep;
- __entry->retry = retry;
+ __entry->ret = ret;
),
TP_printk("vcpu %d gva %llx error_code %s sptep %p old %#llx"
diff --git a/arch/x86/kvm/trace.h b/arch/x86/kvm/trace.h
index 035767345763..24bf0c18b84a 100644
--- a/arch/x86/kvm/trace.h
+++ b/arch/x86/kvm/trace.h
@@ -1561,6 +1561,50 @@ TRACE_EVENT(kvm_spp_set_subpages,
__entry->access)
);
+TRACE_EVENT(kvm_spp_induced_exit,
+ TP_PROTO(struct kvm_vcpu *vcpu, gpa_t gpa, u32 qualification),
+ TP_ARGS(vcpu, gpa, qualification),
+
+ TP_STRUCT__entry(
+ __field(int, vcpu_id)
+ __field(gpa_t, gpa)
+ __field(u32, qualification)
+ ),
+
+ TP_fast_assign(
+ __entry->vcpu_id = vcpu->vcpu_id;
+ __entry->gpa = gpa;
+ __entry->qualification = qualification;
+ ),
+
+ TP_printk("vcpu %d gpa %llx qualificaiton %x",
+ __entry->vcpu_id,
+ __entry->gpa,
+ __entry->qualification)
+);
+
+TRACE_EVENT(kvm_spp_induced_page_fault,
+ TP_PROTO(struct kvm_vcpu *vcpu, gpa_t gpa, int insn_len),
+ TP_ARGS(vcpu, gpa, insn_len),
+
+ TP_STRUCT__entry(
+ __field(int, vcpu_id)
+ __field(gpa_t, gpa)
+ __field(int, insn_len)
+ ),
+
+ TP_fast_assign(
+ __entry->vcpu_id = vcpu->vcpu_id;
+ __entry->gpa = gpa;
+ __entry->insn_len = insn_len;
+ ),
+
+ TP_printk("vcpu %d gpa %llx insn_len %d",
+ __entry->vcpu_id,
+ __entry->gpa,
+ __entry->insn_len)
+);
+
#endif /* _TRACE_KVM_H */
#undef TRACE_INCLUDE_PATH
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index c2c6335a998c..452c93c296a0 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -62,6 +62,7 @@
#include "vmcs12.h"
#include "vmx.h"
#include "x86.h"
+#include "mmu/spp.h"
MODULE_AUTHOR("Qumranet");
MODULE_LICENSE("GPL");
@@ -1402,6 +1403,13 @@ static bool emulation_required(struct kvm_vcpu *vcpu)
return emulate_invalid_guest_state && !guest_state_valid(vcpu);
}
+static int vmx_get_insn_len(struct kvm_vcpu *vcpu)
+{
+ return vmcs_read32(VM_EXIT_INSTRUCTION_LEN);
+}
+
+static void vmx_decache_cr0_guest_bits(struct kvm_vcpu *vcpu);
+
unsigned long vmx_get_rflags(struct kvm_vcpu *vcpu)
{
struct vcpu_vmx *vmx = to_vmx(vcpu);
@@ -5382,6 +5390,69 @@ static int handle_monitor_trap(struct kvm_vcpu *vcpu)
return 1;
}
+static int handle_spp(struct kvm_vcpu *vcpu)
+{
+ unsigned long exit_qualification;
+ struct kvm_memory_slot *slot;
+ gfn_t gfn, gfn_end;
+ u32 *access;
+ gpa_t gpa;
+
+ exit_qualification = vmcs_readl(EXIT_QUALIFICATION);
+
+ /*
+ * SPP VM exit happened while executing iret from NMI,
+ * "blocked by NMI" bit has to be set before next VM entry.
+ * There are errata that may cause this bit to not be set:
+ * AAK134, BY25.
+ */
+ if (!(to_vmx(vcpu)->idt_vectoring_info & VECTORING_INFO_VALID_MASK) &&
+ (exit_qualification & SPPT_INTR_INFO_UNBLOCK_NMI))
+ vmcs_set_bits(GUEST_INTERRUPTIBILITY_INFO,
+ GUEST_INTR_STATE_NMI);
+
+ vcpu->arch.exit_qualification = exit_qualification;
+ if (WARN_ON(!(exit_qualification & SPPT_INDUCED_EXIT_TYPE)))
+ goto out_err;
+ /*
+ * SPPT missing
+ * We don't set SPP write access for the corresponding
+ * GPA, if we haven't setup, we need to construct
+ * SPP table here.
+ */
+ gpa = vmcs_read64(GUEST_PHYSICAL_ADDRESS);
+ gfn = gpa_to_gfn(gpa);
+ trace_kvm_spp_induced_exit(vcpu, gpa, exit_qualification);
+ /*
+ * In level 1 of SPPT, there's no PRESENT bit, all data is
+ * regarded as permission vector, so need to check from
+ * level 2 to set up the vector if target page is protected.
+ */
+ spin_lock(&vcpu->kvm->mmu_lock);
+ gfn &= ~(MAX_ENTRIES_PER_MMUPAGE - 1);
+ gfn_end = gfn + MAX_ENTRIES_PER_MMUPAGE;
+ for (; gfn < gfn_end; gfn++) {
+ slot = gfn_to_memslot(vcpu->kvm, gfn);
+ if (!slot)
+ continue;
+ access = gfn_to_subpage_wp_info(slot, gfn);
+ if (access && *access != FULL_SPP_ACCESS)
+ kvm_spp_setup_structure(vcpu, *access, gfn);
+ }
+ spin_unlock(&vcpu->kvm->mmu_lock);
+ return 1;
+out_err:
+ /*
+ * SPPT Misconfig
+ * This is probably caused by some mis-configuration in SPPT
+ * entries, cannot handle it here, escalate the fault to
+ * emulator.
+ */
+ vcpu->run->exit_reason = KVM_EXIT_UNKNOWN;
+ vcpu->run->hw.hardware_exit_reason = EXIT_REASON_SPP;
+ return 0;
+}
+
static int handle_monitor(struct kvm_vcpu *vcpu)
{
printk_once(KERN_WARNING "kvm: MONITOR instruction emulated as NOP!\n");
@@ -5596,6 +5667,7 @@ static int (*kvm_vmx_exit_handlers[])(struct kvm_vcpu *vcpu) = {
[EXIT_REASON_INVVPID] = handle_vmx_instruction,
[EXIT_REASON_RDRAND] = handle_invalid_op,
[EXIT_REASON_RDSEED] = handle_invalid_op,
+ [EXIT_REASON_SPP] = handle_spp,
[EXIT_REASON_PML_FULL] = handle_pml_full,
[EXIT_REASON_INVPCID] = handle_invpcid,
[EXIT_REASON_VMFUNC] = handle_vmx_instruction,
@@ -7838,6 +7910,8 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = {
.nested_get_evmcs_version = NULL,
.need_emulation_on_page_fault = vmx_need_emulation_on_page_fault,
.apic_init_signal_blocked = vmx_apic_init_signal_blocked,
+
+ .get_insn_len = vmx_get_insn_len,
};
static __init int hardware_setup(void)
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index c0b09d8be31b..4b033a39d6c3 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -10690,3 +10690,5 @@ EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_avic_incomplete_ipi);
EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_avic_ga_log);
EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_apicv_update_request);
EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_spp_set_subpages);
+EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_spp_induced_exit);
+EXPORT_TRACEPOINT_SYMBOL_GPL(kvm_spp_induced_page_fault);
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index 2b70cf0d402a..b81094e1e1c7 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -244,6 +244,7 @@ struct kvm_hyperv_exit {
#define KVM_EXIT_IOAPIC_EOI 26
#define KVM_EXIT_HYPERV 27
#define KVM_EXIT_ARM_NISV 28
+#define KVM_EXIT_SPP 29
/* For KVM_EXIT_INTERNAL_ERROR */
/* Emulate instruction failed. */
@@ -401,6 +402,11 @@ struct kvm_run {
struct {
__u8 vector;
} eoi;
+ /* KVM_EXIT_SPP */
+ struct {
+ __u64 addr;
+ __u8 insn_len;
+ } spp;
/* KVM_EXIT_HYPERV */
struct kvm_hyperv_exit hyperv;
/* KVM_EXIT_ARM_NISV */
--
2.17.2
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH v12 07/11] mmu: spp: Enable Lazy mode SPP protection
2020-05-16 12:54 [PATCH v12 00/11] Enable Sub-Page Write Protection Support Yang Weijiang
` (5 preceding siblings ...)
2020-05-16 12:55 ` [PATCH v12 06/11] vmx: spp: Handle SPP induced vmexit and EPT violation Yang Weijiang
@ 2020-05-16 12:55 ` Yang Weijiang
2020-05-16 12:55 ` [PATCH v12 08/11] mmu: spp: Re-enable SPP protection when EPT mapping changes Yang Weijiang
` (3 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Yang Weijiang @ 2020-05-16 12:55 UTC (permalink / raw)
To: kvm, linux-kernel, pbonzini, jmattson, sean.j.christopherson
Cc: yu.c.zhang, alazar, edwin.zhai, ssicleru, Yang Weijiang
When SPP protection is set but the gfn->pfn mapping isn't there,
we need to check and mark SPP protection in EPT while
gfn->pfn mapping is being built, but the setup for SPPT is deferred
to handle_spp() handler.
From HW's capability, SPP only works for 4KB mappings, to apply
SPP protection for hugepage(2MB,1GB) cases, hugepage entries need
to be zapped before SPP set up. In tdp_page_fault(), there's check
for SPP protection before sets 4KB, 2MB or 1GB mapping, the target
is introducing least impact to hugepage setup, i.e., falls back to
most possible hugepage mapping.
Suggested-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Yang Weijiang <weijiang.yang@intel.com>
---
arch/x86/kvm/mmu/mmu.c | 19 +++++++++++++++++++
arch/x86/kvm/mmu/spp.c | 43 ++++++++++++++++++++++++++++++++++++++++++
arch/x86/kvm/mmu/spp.h | 2 ++
3 files changed, 64 insertions(+)
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 270dd567272e..9d5a0eb3d24e 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -3131,6 +3131,7 @@ static int direct_pte_prefetch_many(struct kvm_vcpu *vcpu,
unsigned int access = sp->role.access;
int i, ret;
gfn_t gfn;
+ u32 *wp_bitmap;
gfn = kvm_mmu_page_get_gfn(sp, start - sp->spt);
slot = gfn_to_memslot_dirty_bitmap(vcpu, gfn, access & ACC_WRITE_MASK);
@@ -3144,6 +3145,13 @@ static int direct_pte_prefetch_many(struct kvm_vcpu *vcpu,
for (i = 0; i < ret; i++, gfn++, start++) {
mmu_set_spte(vcpu, start, access, 0, sp->role.level, gfn,
page_to_pfn(pages[i]), true, true);
+ if (vcpu->kvm->arch.spp_active) {
+ wp_bitmap = gfn_to_subpage_wp_info(slot, gfn);
+ if (wp_bitmap && *wp_bitmap != FULL_SPP_ACCESS)
+ kvm_spp_mark_protection(vcpu->kvm,
+ gfn,
+ *wp_bitmap);
+ }
put_page(pages[i]);
}
@@ -3336,6 +3344,15 @@ static int __direct_map(struct kvm_vcpu *vcpu, gpa_t gpa, int write,
map_writable);
direct_pte_prefetch(vcpu, it.sptep);
++vcpu->stat.pf_fixed;
+ if (level == PT_PAGE_TABLE_LEVEL) {
+ int ret;
+ u32 access;
+
+ ret = kvm_spp_get_permission(vcpu->kvm, gfn, 1, &access);
+ if (ret == 1 && access != FULL_SPP_ACCESS)
+ kvm_spp_mark_protection(vcpu->kvm, gfn, access);
+ }
+
return ret;
}
@@ -4125,6 +4142,8 @@ static int direct_page_fault(struct kvm_vcpu *vcpu, gpa_t gpa, u32 error_code,
if (lpage_disallowed)
max_level = PT_PAGE_TABLE_LEVEL;
+ check_spp_protection(vcpu, gfn, &max_level);
+
r = fast_page_fault(vcpu, gpa, error_code);
if (r != RET_PF_INVALID)
diff --git a/arch/x86/kvm/mmu/spp.c b/arch/x86/kvm/mmu/spp.c
index bb8f298d72da..8f89df57da7c 100644
--- a/arch/x86/kvm/mmu/spp.c
+++ b/arch/x86/kvm/mmu/spp.c
@@ -433,6 +433,49 @@ int kvm_spp_mark_protection(struct kvm *kvm, u64 gfn, u32 access)
return ret;
}
+static bool is_spp_protected(struct kvm_memory_slot *slot, gfn_t gfn, int level)
+{
+ int page_num = KVM_PAGES_PER_HPAGE(level);
+ u32 *access;
+ gfn_t gfn_max;
+
+ gfn &= ~(page_num - 1);
+ gfn_max = gfn + page_num - 1;
+ for (; gfn <= gfn_max; gfn++) {
+ access = gfn_to_subpage_wp_info(slot, gfn);
+ if (access && *access != FULL_SPP_ACCESS)
+ return true;
+ }
+ return false;
+}
+
+void check_spp_protection(struct kvm_vcpu *vcpu, gfn_t gfn,
+ int *max_level)
+{
+ struct kvm *kvm = vcpu->kvm;
+ struct kvm_memory_slot *slot;
+ bool protected;
+
+ if (!kvm->arch.spp_active)
+ return;
+
+ slot = gfn_to_memslot(kvm, gfn);
+
+ if (!slot)
+ return;
+
+ protected = is_spp_protected(slot, gfn, PT_DIRECTORY_LEVEL);
+
+ if (protected) {
+ *max_level = PT_PAGE_TABLE_LEVEL;
+ } else if (*max_level == PT_PDPE_LEVEL &&
+ is_spp_protected(slot, gfn, PT_PDPE_LEVEL)) {
+ *max_level = PT_DIRECTORY_LEVEL;
+ }
+
+ return;
+}
+
int kvm_vm_ioctl_get_subpages(struct kvm *kvm,
u64 gfn,
u32 npages,
diff --git a/arch/x86/kvm/mmu/spp.h b/arch/x86/kvm/mmu/spp.h
index c3588c20be52..e0541901ec41 100644
--- a/arch/x86/kvm/mmu/spp.h
+++ b/arch/x86/kvm/mmu/spp.h
@@ -11,6 +11,8 @@ int kvm_spp_get_permission(struct kvm *kvm, u64 gfn, u32 npages,
int kvm_spp_set_permission(struct kvm *kvm, u64 gfn, u32 npages,
u32 *access_map);
int kvm_spp_mark_protection(struct kvm *kvm, u64 gfn, u32 access);
+void check_spp_protection(struct kvm_vcpu *vcpu, gfn_t gfn,
+ int *max_level);
int kvm_vm_ioctl_get_subpages(struct kvm *kvm,
u64 gfn,
u32 npages,
--
2.17.2
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH v12 08/11] mmu: spp: Re-enable SPP protection when EPT mapping changes
2020-05-16 12:54 [PATCH v12 00/11] Enable Sub-Page Write Protection Support Yang Weijiang
` (6 preceding siblings ...)
2020-05-16 12:55 ` [PATCH v12 07/11] mmu: spp: Enable Lazy mode SPP protection Yang Weijiang
@ 2020-05-16 12:55 ` Yang Weijiang
2020-05-16 12:55 ` [PATCH v12 09/11] x86: spp: Add SPP protection check in instruction emulation Yang Weijiang
` (2 subsequent siblings)
10 siblings, 0 replies; 12+ messages in thread
From: Yang Weijiang @ 2020-05-16 12:55 UTC (permalink / raw)
To: kvm, linux-kernel, pbonzini, jmattson, sean.j.christopherson
Cc: yu.c.zhang, alazar, edwin.zhai, ssicleru, Yang Weijiang
Host page swapping/migration may change the translation in
EPT leaf entry, if the target page is SPP protected,
re-enable SPP protection. When SPPT mmu-page is reclaimed,
no need to clear rmap as no memory-mapping is in SPPT L4E.
Signed-off-by: Yang Weijiang <weijiang.yang@intel.com>
---
arch/x86/kvm/mmu/mmu.c | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 9d5a0eb3d24e..1cf25752e37c 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -1796,6 +1796,19 @@ static int kvm_set_pte_rmapp(struct kvm *kvm, struct kvm_rmap_head *rmap_head,
new_spte &= ~PT_WRITABLE_MASK;
new_spte &= ~SPTE_HOST_WRITEABLE;
+ /*
+ * if it's EPT leaf entry and the physical page is
+ * SPP protected, then re-enable SPP protection for
+ * the page.
+ */
+ if (kvm->arch.spp_active &&
+ level == PT_PAGE_TABLE_LEVEL) {
+ u32 *access = gfn_to_subpage_wp_info(slot, gfn);
+
+ if (access && *access != FULL_SPP_ACCESS)
+ new_spte |= PT_SPP_MASK;
+ }
+
new_spte = mark_spte_for_access_track(new_spte);
mmu_spte_clear_track_bits(sptep);
@@ -2639,6 +2652,10 @@ static bool mmu_page_zap_pte(struct kvm *kvm, struct kvm_mmu_page *sp,
pte = *spte;
if (is_shadow_present_pte(pte)) {
if (is_last_spte(pte, sp->role.level)) {
+ /* SPPT leaf entries don't have rmaps*/
+ if (sp->role.spp && sp->role.level ==
+ PT_PAGE_TABLE_LEVEL)
+ return true;
drop_spte(kvm, spte);
if (is_large_pte(pte))
--kvm->stat.lpages;
--
2.17.2
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH v12 09/11] x86: spp: Add SPP protection check in instruction emulation
2020-05-16 12:54 [PATCH v12 00/11] Enable Sub-Page Write Protection Support Yang Weijiang
` (7 preceding siblings ...)
2020-05-16 12:55 ` [PATCH v12 08/11] mmu: spp: Re-enable SPP protection when EPT mapping changes Yang Weijiang
@ 2020-05-16 12:55 ` Yang Weijiang
2020-05-16 12:55 ` [PATCH v12 10/11] vmx: spp: Initialize SPP bitmap and SPP protection Yang Weijiang
2020-05-16 12:55 ` [PATCH v12 11/11] kvm: selftests: selftest for Sub-Page protection Yang Weijiang
10 siblings, 0 replies; 12+ messages in thread
From: Yang Weijiang @ 2020-05-16 12:55 UTC (permalink / raw)
To: kvm, linux-kernel, pbonzini, jmattson, sean.j.christopherson
Cc: yu.c.zhang, alazar, edwin.zhai, ssicleru, Yang Weijiang
In instruction/mmio emulation cases, if the target write memroy
is SPP protected, "fake" an vmexit to userspace to let application
handle it.
Signed-off-by: Yang Weijiang <weijiang.yang@intel.com>
---
arch/x86/kvm/x86.c | 37 +++++++++++++++++++++++++++++++++++++
1 file changed, 37 insertions(+)
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 4b033a39d6c3..e3999a3ab911 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -5788,6 +5788,37 @@ static const struct read_write_emulator_ops write_emultor = {
.write = true,
};
+static bool is_emulator_spp_protected(struct kvm_vcpu *vcpu,
+ gpa_t gpa,
+ unsigned int bytes)
+{
+ gfn_t gfn, gfn_start, gfn_end;
+ struct kvm *kvm = vcpu->kvm;
+ struct kvm_memory_slot *slot;
+ u32 *access;
+
+ if (!kvm->arch.spp_active)
+ return false;
+
+ gfn_start = gpa_to_gfn(gpa);
+ gfn_end = gpa_to_gfn(gpa + bytes);
+ for (gfn = gfn_start; gfn <= gfn_end; gfn++) {
+ slot = gfn_to_memslot(kvm, gfn);
+ if (slot) {
+ access = gfn_to_subpage_wp_info(slot, gfn);
+ if (access && *access != FULL_SPP_ACCESS) {
+ vcpu->run->exit_reason = KVM_EXIT_SPP;
+ vcpu->run->spp.addr = gfn;
+ vcpu->run->spp.insn_len =
+ kvm_x86_ops.get_insn_len(vcpu);
+ return true;
+ }
+ }
+ }
+
+ return false;
+}
+
static int emulator_read_write_onepage(unsigned long addr, void *val,
unsigned int bytes,
struct x86_exception *exception,
@@ -5817,6 +5848,9 @@ static int emulator_read_write_onepage(unsigned long addr, void *val,
return X86EMUL_PROPAGATE_FAULT;
}
+ if (write && is_emulator_spp_protected(vcpu, gpa, bytes))
+ return X86EMUL_UNHANDLEABLE;
+
if (!ret && ops->read_write_emulate(vcpu, gpa, val, bytes))
return X86EMUL_CONTINUE;
@@ -6963,6 +6997,9 @@ int x86_emulate_instruction(struct kvm_vcpu *vcpu, gpa_t cr2_or_gpa,
return 1;
if (r == EMULATION_FAILED) {
+ if (vcpu->run->exit_reason == KVM_EXIT_SPP)
+ return 0;
+
if (reexecute_instruction(vcpu, cr2_or_gpa, write_fault_to_spt,
emulation_type))
return 1;
--
2.17.2
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH v12 10/11] vmx: spp: Initialize SPP bitmap and SPP protection
2020-05-16 12:54 [PATCH v12 00/11] Enable Sub-Page Write Protection Support Yang Weijiang
` (8 preceding siblings ...)
2020-05-16 12:55 ` [PATCH v12 09/11] x86: spp: Add SPP protection check in instruction emulation Yang Weijiang
@ 2020-05-16 12:55 ` Yang Weijiang
2020-05-16 12:55 ` [PATCH v12 11/11] kvm: selftests: selftest for Sub-Page protection Yang Weijiang
10 siblings, 0 replies; 12+ messages in thread
From: Yang Weijiang @ 2020-05-16 12:55 UTC (permalink / raw)
To: kvm, linux-kernel, pbonzini, jmattson, sean.j.christopherson
Cc: yu.c.zhang, alazar, edwin.zhai, ssicleru, Yang Weijiang
For each memoryslot, there's a SPP bitmap buffer allocated,
every 4Byte corresponds to subpages within a 4KB page. The
original default value for each 4byte is all 1s, meaning
the whole 4KB page is not SPP protected, this eases following
SPP protection check.
To support SPP enablement on-demand, SPP initialization can be
done via KVM_ENABLE_CAP with subcode KVM_CAP_X86_SPP.
KVM_SUBPAGE_MAX_PAGES is set to 512 to reduce the impact to EPT
page_fault() handling because when SPP protection is configured,
mmu-lock is held.
All vcpus share the same SPPT, a KVM_REQ_LOAD_CR3 request is issued
to each vcpu after SPP is initialized, in handling of the request,
SPPTP and VMX SPP execution control bit are configured in VMCS.
Signed-off-by: Yang Weijiang <weijiang.yang@intel.com>
---
arch/x86/include/asm/kvm_host.h | 1 +
arch/x86/include/asm/vmx.h | 1 +
arch/x86/include/asm/vmxfeatures.h | 1 +
arch/x86/kvm/mmu/mmu.c | 21 +++++++
arch/x86/kvm/mmu/spp.c | 99 ++++++++++++++++++++++++++++++
arch/x86/kvm/mmu/spp.h | 8 ++-
arch/x86/kvm/vmx/capabilities.h | 5 ++
arch/x86/kvm/vmx/vmx.c | 36 +++++++++++
arch/x86/kvm/x86.c | 7 +++
include/uapi/linux/kvm.h | 1 +
10 files changed, 179 insertions(+), 1 deletion(-)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index d34c0b91d427..a6730cc409b3 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1262,6 +1262,7 @@ struct kvm_x86_ops {
int (*enable_direct_tlbflush)(struct kvm_vcpu *vcpu);
int (*get_insn_len)(struct kvm_vcpu *vcpu);
+ u32 (*get_spp_status)(struct kvm_vcpu *vcpu);
};
struct kvm_x86_init_ops {
diff --git a/arch/x86/include/asm/vmx.h b/arch/x86/include/asm/vmx.h
index 93000497ddd9..0eb18750ff6e 100644
--- a/arch/x86/include/asm/vmx.h
+++ b/arch/x86/include/asm/vmx.h
@@ -70,6 +70,7 @@
#define SECONDARY_EXEC_PT_CONCEAL_VMX VMCS_CONTROL_BIT(PT_CONCEAL_VMX)
#define SECONDARY_EXEC_XSAVES VMCS_CONTROL_BIT(XSAVES)
#define SECONDARY_EXEC_MODE_BASED_EPT_EXEC VMCS_CONTROL_BIT(MODE_BASED_EPT_EXEC)
+#define SECONDARY_EXEC_SPP VMCS_CONTROL_BIT(SPP)
#define SECONDARY_EXEC_PT_USE_GPA VMCS_CONTROL_BIT(PT_USE_GPA)
#define SECONDARY_EXEC_TSC_SCALING VMCS_CONTROL_BIT(TSC_SCALING)
#define SECONDARY_EXEC_ENABLE_USR_WAIT_PAUSE VMCS_CONTROL_BIT(USR_WAIT_PAUSE)
diff --git a/arch/x86/include/asm/vmxfeatures.h b/arch/x86/include/asm/vmxfeatures.h
index 9915990fd8cf..8895ddb5766b 100644
--- a/arch/x86/include/asm/vmxfeatures.h
+++ b/arch/x86/include/asm/vmxfeatures.h
@@ -79,6 +79,7 @@
#define VMX_FEATURE_PT_CONCEAL_VMX ( 2*32+ 19) /* "" Suppress VMX indicators in Processor Trace */
#define VMX_FEATURE_XSAVES ( 2*32+ 20) /* "" Enable XSAVES and XRSTORS in guest */
#define VMX_FEATURE_MODE_BASED_EPT_EXEC ( 2*32+ 22) /* "ept_mode_based_exec" Enable separate EPT EXEC bits for supervisor vs. user */
+#define VMX_FEATURE_SPP ( 2*32+ 23) /* "spp" Enable Sub-page Permission feature */
#define VMX_FEATURE_PT_USE_GPA ( 2*32+ 24) /* "" Processor Trace logs GPAs */
#define VMX_FEATURE_TSC_SCALING ( 2*32+ 25) /* Scale hardware TSC when read in guest */
#define VMX_FEATURE_USR_WAIT_PAUSE ( 2*32+ 26) /* Enable TPAUSE, UMONITOR, UMWAIT in guest */
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 1cf25752e37c..941c164f8312 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -3680,6 +3680,11 @@ void kvm_mmu_free_roots(struct kvm_vcpu *vcpu, struct kvm_mmu *mmu,
(mmu->root_level >= PT64_ROOT_4LEVEL || mmu->direct_map)) {
mmu_free_root_page(vcpu->kvm, &mmu->root_hpa,
&invalid_list);
+ if (VALID_PAGE(vcpu->kvm->arch.sppt_root)) {
+ mmu_free_root_page(vcpu->kvm,
+ &vcpu->kvm->arch.sppt_root,
+ &invalid_list);
+ }
} else {
for (i = 0; i < 4; ++i)
if (mmu->pae_root[i] != 0)
@@ -3747,6 +3752,19 @@ static int mmu_alloc_direct_roots(struct kvm_vcpu *vcpu)
/* root_cr3 is ignored for direct MMUs. */
vcpu->arch.mmu->root_cr3 = 0;
+ /*
+ * If spp_init() is called too early, in some cases, e.g., CR0 paging
+ * bit changes from disable to enable, this causes mmu root page to
+ * be freed together with sppt root page, then here need to re-allocate
+ * it.
+ */
+ if (vcpu->kvm->arch.spp_active &&
+ !VALID_PAGE(vcpu->kvm->arch.sppt_root)) {
+ sp = kvm_spp_get_page(vcpu, 0,
+ vcpu->arch.mmu->shadow_root_level);
+ vcpu->kvm->arch.sppt_root = __pa(sp->spt);
+ ++sp->root_count;
+ }
return 0;
}
@@ -5163,6 +5181,9 @@ void kvm_init_mmu(struct kvm_vcpu *vcpu, bool reset_roots)
uint i;
vcpu->arch.mmu->root_hpa = INVALID_PAGE;
+ if (!vcpu->kvm->arch.spp_active)
+ vcpu->kvm->arch.sppt_root = INVALID_PAGE;
+
for (i = 0; i < KVM_MMU_NUM_PREV_ROOTS; i++)
vcpu->arch.mmu->prev_roots[i] = KVM_MMU_ROOT_INFO_INVALID;
}
diff --git a/arch/x86/kvm/mmu/spp.c b/arch/x86/kvm/mmu/spp.c
index 8f89df57da7c..5e6e603bc27f 100644
--- a/arch/x86/kvm/mmu/spp.c
+++ b/arch/x86/kvm/mmu/spp.c
@@ -266,6 +266,105 @@ int spp_flush_sppt(struct kvm *kvm, u64 gfn_base, u32 npages)
return 0;
}
+static int kvm_spp_create_bitmaps(struct kvm *kvm)
+{
+ struct kvm_memslots *slots;
+ struct kvm_memory_slot *memslot;
+ int i, j, ret;
+ u32 *buff;
+
+ for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) {
+ slots = __kvm_memslots(kvm, i);
+ kvm_for_each_memslot(memslot, slots) {
+ buff = kvzalloc(memslot->npages *
+ sizeof(*memslot->arch.subpage_wp_info),
+ GFP_KERNEL);
+
+ if (!buff) {
+ ret = -ENOMEM;
+ goto out_free;
+ }
+ memslot->arch.subpage_wp_info = buff;
+
+ for (j = 0; j < memslot->npages; j++)
+ buff[j] = FULL_SPP_ACCESS;
+ }
+ }
+
+ return 0;
+out_free:
+ for (i = 0; i < KVM_ADDRESS_SPACE_NUM; i++) {
+ slots = __kvm_memslots(kvm, i);
+ kvm_for_each_memslot(memslot, slots) {
+ if (memslot->arch.subpage_wp_info) {
+ kvfree(memslot->arch.subpage_wp_info);
+ memslot->arch.subpage_wp_info = NULL;
+ }
+ }
+ }
+
+ return ret;
+}
+
+void kvm_spp_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot)
+{
+ struct kvm_vcpu *vcpu;
+
+ if (slot && slot->arch.subpage_wp_info) {
+ kvfree(slot->arch.subpage_wp_info);
+ slot->arch.subpage_wp_info = NULL;
+ vcpu = kvm_get_vcpu(kvm, 0);
+ if (vcpu)
+ vcpu->kvm->arch.spp_active = false;
+ }
+}
+
+int spp_init(struct kvm *kvm)
+{
+ bool first_root = true;
+ int i, ret;
+ int root_level;
+ u32 status;
+ struct kvm_vcpu *vcpu;
+ struct kvm_mmu_page *ssp_sp;
+
+ /* SPP feature is exclusive with nested VM.*/
+ if (kvm_x86_ops.get_nested_state)
+ return -EPERM;
+
+ vcpu = kvm_get_vcpu(kvm, 0);
+ status = kvm_x86_ops.get_spp_status(vcpu);
+
+ if ((status & (SPP_STATUS_VMX_SUPPORT | SPP_STATUS_EPT_SUPPORT)) !=
+ (SPP_STATUS_VMX_SUPPORT | SPP_STATUS_EPT_SUPPORT))
+ return -ENODEV;
+
+ if (kvm->arch.spp_active)
+ return 0;
+
+ ret = kvm_spp_create_bitmaps(kvm);
+
+ if (ret)
+ return ret;
+
+ kvm_for_each_vcpu(i, vcpu, kvm) {
+ if (first_root) {
+ /* prepare caches for SPP setup.*/
+ mmu_topup_memory_caches(vcpu);
+ root_level = vcpu->arch.mmu->shadow_root_level;
+ ssp_sp = kvm_spp_get_page(vcpu, 0, root_level);
+ first_root = false;
+ vcpu->kvm->arch.sppt_root = __pa(ssp_sp->spt);
+ }
+ vcpu->arch.spp_pending = true;
+ ++ssp_sp->root_count;
+ kvm_make_request(KVM_REQ_LOAD_MMU_PGD, vcpu);
+ }
+
+ kvm->arch.spp_active = true;
+ return 0;
+}
+
int kvm_spp_get_permission(struct kvm *kvm, u64 gfn, u32 npages,
u32 *access_map)
{
diff --git a/arch/x86/kvm/mmu/spp.h b/arch/x86/kvm/mmu/spp.h
index e0541901ec41..e9b5fd44c215 100644
--- a/arch/x86/kvm/mmu/spp.h
+++ b/arch/x86/kvm/mmu/spp.h
@@ -5,7 +5,11 @@
#define FULL_SPP_ACCESS (u32)(BIT_ULL(32) - 1)
#define KVM_SUBPAGE_MAX_PAGES 512
#define MAX_ENTRIES_PER_MMUPAGE BIT(9)
+#define SPP_STATUS_VMX_SUPPORT 0x1
+#define SPP_STATUS_EPT_SUPPORT 0x2
+int spp_init(struct kvm *kvm);
+void kvm_spp_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot);
int kvm_spp_get_permission(struct kvm *kvm, u64 gfn, u32 npages,
u32 *access_map);
int kvm_spp_set_permission(struct kvm *kvm, u64 gfn, u32 npages,
@@ -29,5 +33,7 @@ void save_spp_bit(u64 *spte);
void restore_spp_bit(u64 *spte);
bool was_spp_armed(u64 spte);
u64 construct_spptp(unsigned long root_hpa);
-
+struct kvm_mmu_page *kvm_spp_get_page(struct kvm_vcpu *vcpu,
+ gfn_t gfn,
+ unsigned int level);
#endif /* __KVM_X86_VMX_SPP_H */
diff --git a/arch/x86/kvm/vmx/capabilities.h b/arch/x86/kvm/vmx/capabilities.h
index 8903475f751e..e5ccc1f11d87 100644
--- a/arch/x86/kvm/vmx/capabilities.h
+++ b/arch/x86/kvm/vmx/capabilities.h
@@ -248,6 +248,11 @@ static inline bool vmx_xsaves_supported(void)
SECONDARY_EXEC_XSAVES;
}
+static inline bool cpu_has_vmx_ept_spp(void)
+{
+ return vmcs_config.cpu_based_2nd_exec_ctrl & SECONDARY_EXEC_SPP;
+}
+
static inline bool vmx_waitpkg_supported(void)
{
return vmcs_config.cpu_based_2nd_exec_ctrl &
diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c
index 452c93c296a0..89045a7f9d4b 100644
--- a/arch/x86/kvm/vmx/vmx.c
+++ b/arch/x86/kvm/vmx/vmx.c
@@ -117,6 +117,9 @@ module_param_named(pml, enable_pml, bool, S_IRUGO);
static bool __read_mostly dump_invalid_vmcs = 0;
module_param(dump_invalid_vmcs, bool, 0644);
+/* SPP is disabled by default unless it's enabled via KVM_ENABLE_CAP. */
+static bool __read_mostly enable_spp = 0;
+
#define MSR_BITMAP_MODE_X2APIC 1
#define MSR_BITMAP_MODE_X2APIC_APICV 2
@@ -1408,6 +1411,17 @@ static int vmx_get_insn_len(struct kvm_vcpu *vcpu)
return vmcs_read32(VM_EXIT_INSTRUCTION_LEN);
}
+static u32 vmx_get_spp_status(struct kvm_vcpu *vcpu)
+{
+ u32 status = 0;
+
+ if (cpu_has_vmx_ept_spp())
+ status |= SPP_STATUS_VMX_SUPPORT;
+ if (enable_ept)
+ status |= SPP_STATUS_EPT_SUPPORT;
+ return status;
+}
+
static void vmx_decache_cr0_guest_bits(struct kvm_vcpu *vcpu);
unsigned long vmx_get_rflags(struct kvm_vcpu *vcpu)
@@ -2394,6 +2408,7 @@ static __init int setup_vmcs_config(struct vmcs_config *vmcs_conf,
SECONDARY_EXEC_RDSEED_EXITING |
SECONDARY_EXEC_RDRAND_EXITING |
SECONDARY_EXEC_ENABLE_PML |
+ SECONDARY_EXEC_SPP |
SECONDARY_EXEC_TSC_SCALING |
SECONDARY_EXEC_ENABLE_USR_WAIT_PAUSE |
SECONDARY_EXEC_PT_USE_GPA |
@@ -2998,6 +3013,7 @@ void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, unsigned long cr3)
bool update_guest_cr3 = true;
unsigned long guest_cr3;
u64 eptp;
+ u64 spptp;
guest_cr3 = cr3;
if (enable_ept) {
@@ -3026,6 +3042,20 @@ void vmx_load_mmu_pgd(struct kvm_vcpu *vcpu, unsigned long cr3)
if (update_guest_cr3)
vmcs_writel(GUEST_CR3, guest_cr3);
+
+ if (kvm->arch.spp_active && VALID_PAGE(vcpu->kvm->arch.sppt_root)) {
+ spptp = construct_spptp(vcpu->kvm->arch.sppt_root);
+ vmcs_write64(SPPT_POINTER, spptp);
+
+ if (vcpu->arch.spp_pending && cpu_has_secondary_exec_ctrls()) {
+ struct vcpu_vmx *vmx = to_vmx(vcpu);
+
+ secondary_exec_controls_setbit(vmx,
+ SECONDARY_EXEC_SPP);
+ enable_spp = 1;
+ vcpu->arch.spp_pending = false;
+ }
+ }
}
int vmx_set_cr4(struct kvm_vcpu *vcpu, unsigned long cr4)
@@ -4042,6 +4072,9 @@ static void vmx_compute_secondary_exec_control(struct vcpu_vmx *vmx)
if (!enable_pml)
exec_control &= ~SECONDARY_EXEC_ENABLE_PML;
+ if (!enable_spp)
+ exec_control &= ~SECONDARY_EXEC_SPP;
+
if (vmx_xsaves_supported()) {
/* Exposing XSAVES only when XSAVE is exposed */
bool xsaves_enabled =
@@ -5900,6 +5933,8 @@ void dump_vmcs(void)
pr_err("PostedIntrVec = 0x%02x\n", vmcs_read16(POSTED_INTR_NV));
if ((secondary_exec_control & SECONDARY_EXEC_ENABLE_EPT))
pr_err("EPT pointer = 0x%016llx\n", vmcs_read64(EPT_POINTER));
+ if ((secondary_exec_control & SECONDARY_EXEC_SPP))
+ pr_err("SPPT pointer = 0x%016llx\n", vmcs_read64(SPPT_POINTER));
n = vmcs_read32(CR3_TARGET_COUNT);
for (i = 0; i + 1 < n; i += 4)
pr_err("CR3 target%u=%016lx target%u=%016lx\n",
@@ -7912,6 +7947,7 @@ static struct kvm_x86_ops vmx_x86_ops __initdata = {
.apic_init_signal_blocked = vmx_apic_init_signal_blocked,
.get_insn_len = vmx_get_insn_len,
+ .get_spp_status = vmx_get_spp_status,
};
static __init int hardware_setup(void)
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index e3999a3ab911..a96463f9f33b 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -3436,6 +3436,9 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
case KVM_CAP_HYPERV_ENLIGHTENED_VMCS:
r = kvm_x86_ops.nested_enable_evmcs != NULL;
break;
+ case KVM_CAP_X86_SPP:
+ r = KVM_SUBPAGE_MAX_PAGES;
+ break;
default:
break;
}
@@ -4884,6 +4887,9 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
kvm->arch.exception_payload_enabled = cap->args[0];
r = 0;
break;
+ case KVM_CAP_X86_SPP:
+ r = spp_init(kvm);
+ break;
default:
r = -EINVAL;
break;
@@ -10047,6 +10053,7 @@ void kvm_arch_free_memslot(struct kvm *kvm, struct kvm_memory_slot *slot)
}
kvm_page_track_free_memslot(slot);
+ kvm_spp_free_memslot(kvm, slot);
}
static int kvm_alloc_memslot_metadata(struct kvm_memory_slot *slot,
diff --git a/include/uapi/linux/kvm.h b/include/uapi/linux/kvm.h
index b81094e1e1c7..7a2a2eaa5704 100644
--- a/include/uapi/linux/kvm.h
+++ b/include/uapi/linux/kvm.h
@@ -1031,6 +1031,7 @@ struct kvm_ppc_resize_hpt {
#define KVM_CAP_S390_VCPU_RESETS 179
#define KVM_CAP_S390_PROTECTED 180
#define KVM_CAP_PPC_SECURE_GUEST 181
+#define KVM_CAP_X86_SPP 182
#ifdef KVM_CAP_IRQ_ROUTING
--
2.17.2
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH v12 11/11] kvm: selftests: selftest for Sub-Page protection
2020-05-16 12:54 [PATCH v12 00/11] Enable Sub-Page Write Protection Support Yang Weijiang
` (9 preceding siblings ...)
2020-05-16 12:55 ` [PATCH v12 10/11] vmx: spp: Initialize SPP bitmap and SPP protection Yang Weijiang
@ 2020-05-16 12:55 ` Yang Weijiang
10 siblings, 0 replies; 12+ messages in thread
From: Yang Weijiang @ 2020-05-16 12:55 UTC (permalink / raw)
To: kvm, linux-kernel, pbonzini, jmattson, sean.j.christopherson
Cc: yu.c.zhang, alazar, edwin.zhai, ssicleru, Yang Weijiang
Sub-Page Permission(SPP) is to protect finer granularity subpages
(128Byte each) within a 4KB page. It's not enabled in KVM by default,
the test first initializes the SPP runtime environment with
KVM_ENABLE_CAP ioctl, then sets protection with KVM_SUBPAGES_SET_ACCESS
for the target guest page, check permissions with KVM_SUBPAGES_GET_ACCESS
to make sure they are set as expected.
Two steps in guest code to very whether SPP is working:
1) protect all 128byte subpages, write data to each subpage
to see if SPP induced EPT violation happening. 2)unprotect all
subpages, again write data to each subpage to see if SPP still
works or not.
Signed-off-by: Yang Weijiang <weijiang.yang@intel.com>
---
tools/testing/selftests/kvm/Makefile | 1 +
tools/testing/selftests/kvm/lib/kvm_util.c | 1 +
tools/testing/selftests/kvm/x86_64/spp_test.c | 235 ++++++++++++++++++
3 files changed, 237 insertions(+)
create mode 100644 tools/testing/selftests/kvm/x86_64/spp_test.c
diff --git a/tools/testing/selftests/kvm/Makefile b/tools/testing/selftests/kvm/Makefile
index b728c0a0f9b2..91815f5bba5b 100644
--- a/tools/testing/selftests/kvm/Makefile
+++ b/tools/testing/selftests/kvm/Makefile
@@ -60,6 +60,7 @@ TEST_GEN_PROGS_x86_64 += dirty_log_test
TEST_GEN_PROGS_x86_64 += kvm_create_max_vcpus
TEST_GEN_PROGS_x86_64 += steal_time
+TEST_GEN_PROGS_x86_64 += x86_64/spp_test
TEST_GEN_PROGS_aarch64 += clear_dirty_log_test
TEST_GEN_PROGS_aarch64 += demand_paging_test
TEST_GEN_PROGS_aarch64 += dirty_log_test
diff --git a/tools/testing/selftests/kvm/lib/kvm_util.c b/tools/testing/selftests/kvm/lib/kvm_util.c
index 8a3523d4434f..3bcb5e4dc3e6 100644
--- a/tools/testing/selftests/kvm/lib/kvm_util.c
+++ b/tools/testing/selftests/kvm/lib/kvm_util.c
@@ -1552,6 +1552,7 @@ static struct exit_reason {
{KVM_EXIT_UNKNOWN, "UNKNOWN"},
{KVM_EXIT_EXCEPTION, "EXCEPTION"},
{KVM_EXIT_IO, "IO"},
+ {KVM_EXIT_SPP, "SPP"},
{KVM_EXIT_HYPERCALL, "HYPERCALL"},
{KVM_EXIT_DEBUG, "DEBUG"},
{KVM_EXIT_HLT, "HLT"},
diff --git a/tools/testing/selftests/kvm/x86_64/spp_test.c b/tools/testing/selftests/kvm/x86_64/spp_test.c
new file mode 100644
index 000000000000..73a5719135a5
--- /dev/null
+++ b/tools/testing/selftests/kvm/x86_64/spp_test.c
@@ -0,0 +1,235 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Sub-Page Permission test
+ *
+ * Copyright (C) 2019, Intel Corp.
+ *
+ */
+
+#include <fcntl.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/ioctl.h>
+
+#include "test_util.h"
+#include "kvm_util.h"
+#include "processor.h"
+#include "../../lib/kvm_util_internal.h"
+#include "linux/kvm.h"
+
+#define VCPU_ID 1
+#define PAGE_SIZE (4096)
+#define SPP_GUARD_SIZE (16 * PAGE_SIZE)
+#define SPP_GUARD_MEMSLOT (1)
+#define SPP_GUARD_PAGES (SPP_GUARD_SIZE / PAGE_SIZE)
+#define SPP_GUARD_GPA 0x10000000
+
+#define SUBPAGE_ACCESS_DEFAULT (0x0)
+#define SUBPAGE_ACCESS_FULL (0xFFFFFFFF)
+#define START_SPP_VM_ADDR (0x70000000)
+#define SUBPAGE_SIZE (128)
+
+vm_vaddr_t vspp_start;
+vm_paddr_t pspp_start;
+
+void guest_code(void)
+{
+ uint8_t *iterator = (uint8_t *)vspp_start;
+ int count;
+
+ GUEST_SYNC(1);
+ /*
+ * expect EPT violation induced by SPP in each interation since
+ * the full page is protected by SPP.
+ */
+ for (count = 0; count < PAGE_SIZE / SUBPAGE_SIZE; count++) {
+ *(uint32_t *)(iterator) = 0x99;
+ iterator += SUBPAGE_SIZE;
+ }
+ GUEST_SYNC(2);
+ iterator = (uint8_t *)vspp_start;
+
+ /*
+ * don't expect EPT violation happen since SPP is disabled
+ * for the page
+ */
+ for (count = 0; count < PAGE_SIZE / SUBPAGE_SIZE; count++) {
+ *(uint32_t *)(iterator) = 0x99;
+ iterator += SUBPAGE_SIZE;
+ }
+}
+
+void prepare_test(struct kvm_vm **g_vm, struct kvm_run **g_run)
+{
+ void *spp_hva;
+ struct kvm_vm *vm;
+ struct kvm_run *run;
+ /* Create VM, SPP is only valid for 4KB page mode */
+ *g_vm = vm_create_default(VCPU_ID, 0, guest_code);
+ vm = *g_vm;
+
+ *g_run = vcpu_state(vm, VCPU_ID);
+ run = *g_run;
+
+ vm_userspace_mem_region_add(vm, VM_MEM_SRC_ANONYMOUS, SPP_GUARD_GPA,
+ SPP_GUARD_MEMSLOT, SPP_GUARD_PAGES, 0);
+
+ pspp_start = vm_phy_pages_alloc(vm, 1, SPP_GUARD_GPA,
+ SPP_GUARD_MEMSLOT);
+
+ memset(addr_gpa2hva(vm, SPP_GUARD_GPA), 0x0, PAGE_SIZE);
+
+ virt_map(vm, START_SPP_VM_ADDR, SPP_GUARD_GPA, PAGE_SIZE, 0);
+
+ vspp_start = vm_vaddr_alloc(vm, PAGE_SIZE, START_SPP_VM_ADDR,
+ SPP_GUARD_MEMSLOT, 0);
+
+ spp_hva = addr_gva2hva(vm, vspp_start);
+
+ pspp_start = addr_hva2gpa(vm, spp_hva);
+
+ printf("SPP protected zone: size = %d, gva = 0x%lx, gpa = 0x%lx, "
+ "hva = %p\n", PAGE_SIZE, vspp_start, pspp_start, spp_hva);
+
+ /* make sure the virtual address is visible to VM. */
+ sync_global_to_guest(vm, vspp_start);
+
+ vcpu_run(vm, VCPU_ID);
+
+ TEST_ASSERT(run->exit_reason == KVM_EXIT_IO,
+ "exit reason: %u (%s),\n", run->exit_reason,
+ exit_reason_str(run->exit_reason));
+}
+
+void setup_spp(struct kvm_vm *vm)
+{
+ struct kvm_enable_cap cap;
+ int ret = 0;
+ struct kvm_subpage *sp;
+ int len;
+
+ memset(&cap, 0, sizeof(cap));
+ cap.cap = KVM_CAP_X86_SPP;
+ cap.flags = 0;
+
+ /* initialize the SPP runtime environment.*/
+ ret = ioctl(vm->fd, KVM_ENABLE_CAP, &cap);
+ TEST_ASSERT(ret == 0, "KVM_CAP_X86_SPP failed.");
+ len = sizeof(*sp) + sizeof(__u32);
+ printf("SPP initialized successfully.\n");
+
+ sp = malloc(len);
+ TEST_ASSERT(sp > 0, "Low memory 1!");
+ memset(sp, 0, len);
+ /* set up SPP protection for the page. */
+ sp->npages = 1;
+ sp->gfn_base = pspp_start >> 12;
+ sp->access_map[0] = SUBPAGE_ACCESS_DEFAULT;
+ ret = ioctl(vm->fd, KVM_SUBPAGES_SET_ACCESS, sp);
+
+ TEST_ASSERT(ret == 1, "KVM_SUBPAGES_SET_ACCESS failed. ret = 0x%x, "
+ "gfn_base = 0x%llx\n", ret, sp->gfn_base);
+ printf("set spp protection info: gfn = 0x%llx, access = 0x%x, "
+ "npages = %d\n", sp->gfn_base, sp->access_map[0],
+ sp->npages);
+
+ memset(sp, 0, len);
+ /* make sure the SPP permission bits are actully set as expected. */
+ sp->npages = 1;
+ sp->gfn_base = pspp_start >> 12;
+
+ ret = ioctl(vm->fd, KVM_SUBPAGES_GET_ACCESS, sp);
+
+ TEST_ASSERT(ret == 1, "KVM_SUBPAGES_GET_ACCESS failed.");
+
+ TEST_ASSERT(sp->access_map[0] == SUBPAGE_ACCESS_DEFAULT,
+ "subpage access didn't match.");
+ printf("get spp protection info: gfn = 0x%llx, access = 0x%x, "
+ "npages = %d\n", sp->gfn_base,
+ sp->access_map[0], sp->npages);
+
+ free(sp);
+ printf("got matched subpage permission vector.\n");
+ printf("expect VM exits caused by SPP below.\n");
+}
+
+void unset_spp(struct kvm_vm *vm)
+{
+ struct kvm_subpage *sp;
+ int len;
+
+ len = sizeof(*sp) + sizeof(__u32);
+ sp = malloc(len);
+ TEST_ASSERT(sp > 0, "Low memory 2!");
+ memset(sp, 0, len);
+
+ /* now unprotect the SPP to the page.*/
+ sp->npages = 1;
+ sp->gfn_base = pspp_start >> 12;
+ sp->access_map[0] = SUBPAGE_ACCESS_FULL;
+ ioctl(vm->fd, KVM_SUBPAGES_SET_ACCESS, sp);
+
+ printf("unset SPP protection at gfn: 0x%llx\n", sp->gfn_base);
+ printf("expect NO VM exits caused by SPP below.\n");
+ free(sp);
+}
+
+#define TEST_SYNC_FIELDS KVM_SYNC_X86_REGS
+
+void run_test(struct kvm_vm *vm, struct kvm_run *run)
+{
+ int loop;
+ int ept_fault = 0;
+ struct kvm_regs regs;
+
+ run->kvm_valid_regs = TEST_SYNC_FIELDS;
+ vcpu_run(vm, VCPU_ID);
+
+ for (loop = 0; loop < PAGE_SIZE / SUBPAGE_SIZE; loop++) {
+ /*
+ * if everything goes correctly, should get VM exit
+ * with KVM_EXIT_SPP.
+ */
+ TEST_ASSERT(run->exit_reason == KVM_EXIT_SPP,
+ "exit reason: %u (%s),\n", run->exit_reason,
+ exit_reason_str(run->exit_reason));
+ printf("%d - exit reason: %s\n", loop + 1,
+ exit_reason_str(run->exit_reason));
+ ept_fault++;
+
+ vcpu_regs_get(vm, VCPU_ID, ®s);
+
+ run->s.regs.regs.rip += run->spp.insn_len;
+
+ run->kvm_valid_regs = TEST_SYNC_FIELDS;
+ run->kvm_dirty_regs = KVM_SYNC_X86_REGS;
+
+ vcpu_run(vm, VCPU_ID);
+ }
+
+ printf("total EPT violation count: %d\n", ept_fault);
+}
+
+int main(int argc, char *argv[])
+{
+ struct kvm_vm *vm;
+ struct kvm_run *run;
+
+ prepare_test(&vm, &run);
+
+ setup_spp(vm);
+
+ run_test(vm, run);
+
+ unset_spp(vm);
+
+ vcpu_run(vm, VCPU_ID);
+
+ printf("completed SPP test successfully!\n");
+
+ kvm_vm_free(vm);
+
+ return 0;
+}
+
--
2.17.2
^ permalink raw reply related [flat|nested] 12+ messages in thread
end of thread, other threads:[~2020-05-16 12:54 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-05-16 12:54 [PATCH v12 00/11] Enable Sub-Page Write Protection Support Yang Weijiang
2020-05-16 12:54 ` [PATCH v12 01/11] Documentation: Add EPT based Subpage Protection and related APIs Yang Weijiang
2020-05-16 12:54 ` [PATCH v12 02/11] mmu: spp: Add a new header file to put definitions shared by MMU and SPP Yang Weijiang
2020-05-16 12:54 ` [PATCH v12 03/11] mmu: spp: Implement SPPT setup functions Yang Weijiang
2020-05-16 12:55 ` [PATCH v12 04/11] mmu: spp: Implement functions to {get|set}_subpage permission Yang Weijiang
2020-05-16 12:55 ` [PATCH v12 05/11] x86: spp: Introduce user-space SPP IOCTLs Yang Weijiang
2020-05-16 12:55 ` [PATCH v12 06/11] vmx: spp: Handle SPP induced vmexit and EPT violation Yang Weijiang
2020-05-16 12:55 ` [PATCH v12 07/11] mmu: spp: Enable Lazy mode SPP protection Yang Weijiang
2020-05-16 12:55 ` [PATCH v12 08/11] mmu: spp: Re-enable SPP protection when EPT mapping changes Yang Weijiang
2020-05-16 12:55 ` [PATCH v12 09/11] x86: spp: Add SPP protection check in instruction emulation Yang Weijiang
2020-05-16 12:55 ` [PATCH v12 10/11] vmx: spp: Initialize SPP bitmap and SPP protection Yang Weijiang
2020-05-16 12:55 ` [PATCH v12 11/11] kvm: selftests: selftest for Sub-Page protection Yang Weijiang
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.