* [PATCH 0/2] KVM: x86/mmu: Optimizations for kvm_get_mmu_page()
@ 2020-06-23 19:40 Sean Christopherson
2020-06-23 19:40 ` [PATCH 1/2] KVM: x86/mmu: Avoid multiple hash lookups in kvm_get_mmu_page() Sean Christopherson
` (2 more replies)
0 siblings, 3 replies; 6+ messages in thread
From: Sean Christopherson @ 2020-06-23 19:40 UTC (permalink / raw)
To: Paolo Bonzini
Cc: Sean Christopherson, Vitaly Kuznetsov, Wanpeng Li, Jim Mattson,
Joerg Roedel, kvm, linux-kernel, Peter Feiner, Jon Cargille
Avoid multiple hash lookups in kvm_get_mmu_page(), and tweak the cache
loop to optimize it for TDP.
Sean Christopherson (2):
KVM: x86/mmu: Avoid multiple hash lookups in kvm_get_mmu_page()
KVM: x86/mmu: Optimize MMU page cache lookup for fully direct MMUs
arch/x86/kvm/mmu/mmu.c | 26 ++++++++++++++++----------
1 file changed, 16 insertions(+), 10 deletions(-)
--
2.26.0
^ permalink raw reply [flat|nested] 6+ messages in thread
* [PATCH 1/2] KVM: x86/mmu: Avoid multiple hash lookups in kvm_get_mmu_page()
2020-06-23 19:40 [PATCH 0/2] KVM: x86/mmu: Optimizations for kvm_get_mmu_page() Sean Christopherson
@ 2020-06-23 19:40 ` Sean Christopherson
2020-06-23 20:27 ` Jon Cargille
2020-06-23 19:40 ` [PATCH 2/2] KVM: x86/mmu: Optimize MMU page cache lookup for fully direct MMUs Sean Christopherson
2020-07-03 17:17 ` [PATCH 0/2] KVM: x86/mmu: Optimizations for kvm_get_mmu_page() Paolo Bonzini
2 siblings, 1 reply; 6+ messages in thread
From: Sean Christopherson @ 2020-06-23 19:40 UTC (permalink / raw)
To: Paolo Bonzini
Cc: Sean Christopherson, Vitaly Kuznetsov, Wanpeng Li, Jim Mattson,
Joerg Roedel, kvm, linux-kernel, Peter Feiner, Jon Cargille
Refactor for_each_valid_sp() to take the list of shadow pages instead of
retrieving it from a gfn to avoid doing the gfn->list hash and lookup
multiple times during kvm_get_mmu_page().
Cc: Peter Feiner <pfeiner@google.com>
Cc: Jon Cargille <jcargill@google.com>
Cc: Jim Mattson <jmattson@google.com>
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
---
arch/x86/kvm/mmu/mmu.c | 17 +++++++++--------
1 file changed, 9 insertions(+), 8 deletions(-)
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 3dd0af7e7515..67f8f82e9783 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -2258,15 +2258,14 @@ static bool kvm_mmu_prepare_zap_page(struct kvm *kvm, struct kvm_mmu_page *sp,
static void kvm_mmu_commit_zap_page(struct kvm *kvm,
struct list_head *invalid_list);
-
-#define for_each_valid_sp(_kvm, _sp, _gfn) \
- hlist_for_each_entry(_sp, \
- &(_kvm)->arch.mmu_page_hash[kvm_page_table_hashfn(_gfn)], hash_link) \
+#define for_each_valid_sp(_kvm, _sp, _list) \
+ hlist_for_each_entry(_sp, _list, hash_link) \
if (is_obsolete_sp((_kvm), (_sp))) { \
} else
#define for_each_gfn_indirect_valid_sp(_kvm, _sp, _gfn) \
- for_each_valid_sp(_kvm, _sp, _gfn) \
+ for_each_valid_sp(_kvm, _sp, \
+ &(_kvm)->arch.mmu_page_hash[kvm_page_table_hashfn(_gfn)]) \
if ((_sp)->gfn != (_gfn) || (_sp)->role.direct) {} else
static inline bool is_ept_sp(struct kvm_mmu_page *sp)
@@ -2477,6 +2476,7 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu,
unsigned int access)
{
union kvm_mmu_page_role role;
+ struct hlist_head *sp_list;
unsigned quadrant;
struct kvm_mmu_page *sp;
bool need_sync = false;
@@ -2496,7 +2496,9 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu,
quadrant &= (1 << ((PT32_PT_BITS - PT64_PT_BITS) * level)) - 1;
role.quadrant = quadrant;
}
- for_each_valid_sp(vcpu->kvm, sp, gfn) {
+
+ sp_list = &vcpu->kvm->arch.mmu_page_hash[kvm_page_table_hashfn(gfn)];
+ for_each_valid_sp(vcpu->kvm, sp, sp_list) {
if (sp->gfn != gfn) {
collisions++;
continue;
@@ -2533,8 +2535,7 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu,
sp->gfn = gfn;
sp->role = role;
- hlist_add_head(&sp->hash_link,
- &vcpu->kvm->arch.mmu_page_hash[kvm_page_table_hashfn(gfn)]);
+ hlist_add_head(&sp->hash_link, sp_list);
if (!direct) {
/*
* we should do write protection before syncing pages
--
2.26.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* [PATCH 2/2] KVM: x86/mmu: Optimize MMU page cache lookup for fully direct MMUs
2020-06-23 19:40 [PATCH 0/2] KVM: x86/mmu: Optimizations for kvm_get_mmu_page() Sean Christopherson
2020-06-23 19:40 ` [PATCH 1/2] KVM: x86/mmu: Avoid multiple hash lookups in kvm_get_mmu_page() Sean Christopherson
@ 2020-06-23 19:40 ` Sean Christopherson
2020-06-23 20:28 ` Jon Cargille
2020-07-03 17:17 ` [PATCH 0/2] KVM: x86/mmu: Optimizations for kvm_get_mmu_page() Paolo Bonzini
2 siblings, 1 reply; 6+ messages in thread
From: Sean Christopherson @ 2020-06-23 19:40 UTC (permalink / raw)
To: Paolo Bonzini
Cc: Sean Christopherson, Vitaly Kuznetsov, Wanpeng Li, Jim Mattson,
Joerg Roedel, kvm, linux-kernel, Peter Feiner, Jon Cargille
Skip the unsync checks and the write flooding clearing for fully direct
MMUs, which are guaranteed to not have unsync'd or indirect pages (write
flooding detection only applies to indirect pages). For TDP, this
avoids unnecessary memory reads and writes, and for the write flooding
count will also avoid dirtying a cache line (unsync_child_bitmap itself
consumes a cache line, i.e. write_flooding_count is guaranteed to be in
a different cache line than parent_ptes).
Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
---
arch/x86/kvm/mmu/mmu.c | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
index 67f8f82e9783..c568a5c55276 100644
--- a/arch/x86/kvm/mmu/mmu.c
+++ b/arch/x86/kvm/mmu/mmu.c
@@ -2475,6 +2475,7 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu,
int direct,
unsigned int access)
{
+ bool direct_mmu = vcpu->arch.mmu->direct_map;
union kvm_mmu_page_role role;
struct hlist_head *sp_list;
unsigned quadrant;
@@ -2490,8 +2491,7 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu,
if (role.direct)
role.gpte_is_8_bytes = true;
role.access = access;
- if (!vcpu->arch.mmu->direct_map
- && vcpu->arch.mmu->root_level <= PT32_ROOT_LEVEL) {
+ if (!direct_mmu && vcpu->arch.mmu->root_level <= PT32_ROOT_LEVEL) {
quadrant = gaddr >> (PAGE_SHIFT + (PT64_PT_BITS * level));
quadrant &= (1 << ((PT32_PT_BITS - PT64_PT_BITS) * level)) - 1;
role.quadrant = quadrant;
@@ -2510,6 +2510,9 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu,
if (sp->role.word != role.word)
continue;
+ if (direct_mmu)
+ goto trace_get_page;
+
if (sp->unsync) {
/* The page is good, but __kvm_sync_page might still end
* up zapping it. If so, break in order to rebuild it.
@@ -2525,6 +2528,8 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu,
kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
__clear_sp_write_flooding_count(sp);
+
+trace_get_page:
trace_kvm_mmu_get_page(sp, false);
goto out;
}
--
2.26.0
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: [PATCH 1/2] KVM: x86/mmu: Avoid multiple hash lookups in kvm_get_mmu_page()
2020-06-23 19:40 ` [PATCH 1/2] KVM: x86/mmu: Avoid multiple hash lookups in kvm_get_mmu_page() Sean Christopherson
@ 2020-06-23 20:27 ` Jon Cargille
0 siblings, 0 replies; 6+ messages in thread
From: Jon Cargille @ 2020-06-23 20:27 UTC (permalink / raw)
To: Sean Christopherson
Cc: Paolo Bonzini, Vitaly Kuznetsov, Wanpeng Li, Jim Mattson,
Joerg Roedel, kvm, linux-kernel, Peter Feiner
LGTM.
Reviewed-By: Jon Cargille <jcargill@google.com>
On Tue, Jun 23, 2020 at 12:40 PM Sean Christopherson
<sean.j.christopherson@intel.com> wrote:
>
> Refactor for_each_valid_sp() to take the list of shadow pages instead of
> retrieving it from a gfn to avoid doing the gfn->list hash and lookup
> multiple times during kvm_get_mmu_page().
>
> Cc: Peter Feiner <pfeiner@google.com>
> Cc: Jon Cargille <jcargill@google.com>
> Cc: Jim Mattson <jmattson@google.com>
> Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
> ---
> arch/x86/kvm/mmu/mmu.c | 17 +++++++++--------
> 1 file changed, 9 insertions(+), 8 deletions(-)
>
> diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
> index 3dd0af7e7515..67f8f82e9783 100644
> --- a/arch/x86/kvm/mmu/mmu.c
> +++ b/arch/x86/kvm/mmu/mmu.c
> @@ -2258,15 +2258,14 @@ static bool kvm_mmu_prepare_zap_page(struct kvm *kvm, struct kvm_mmu_page *sp,
> static void kvm_mmu_commit_zap_page(struct kvm *kvm,
> struct list_head *invalid_list);
>
> -
> -#define for_each_valid_sp(_kvm, _sp, _gfn) \
> - hlist_for_each_entry(_sp, \
> - &(_kvm)->arch.mmu_page_hash[kvm_page_table_hashfn(_gfn)], hash_link) \
> +#define for_each_valid_sp(_kvm, _sp, _list) \
> + hlist_for_each_entry(_sp, _list, hash_link) \
> if (is_obsolete_sp((_kvm), (_sp))) { \
> } else
>
> #define for_each_gfn_indirect_valid_sp(_kvm, _sp, _gfn) \
> - for_each_valid_sp(_kvm, _sp, _gfn) \
> + for_each_valid_sp(_kvm, _sp, \
> + &(_kvm)->arch.mmu_page_hash[kvm_page_table_hashfn(_gfn)]) \
> if ((_sp)->gfn != (_gfn) || (_sp)->role.direct) {} else
>
> static inline bool is_ept_sp(struct kvm_mmu_page *sp)
> @@ -2477,6 +2476,7 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu,
> unsigned int access)
> {
> union kvm_mmu_page_role role;
> + struct hlist_head *sp_list;
> unsigned quadrant;
> struct kvm_mmu_page *sp;
> bool need_sync = false;
> @@ -2496,7 +2496,9 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu,
> quadrant &= (1 << ((PT32_PT_BITS - PT64_PT_BITS) * level)) - 1;
> role.quadrant = quadrant;
> }
> - for_each_valid_sp(vcpu->kvm, sp, gfn) {
> +
> + sp_list = &vcpu->kvm->arch.mmu_page_hash[kvm_page_table_hashfn(gfn)];
> + for_each_valid_sp(vcpu->kvm, sp, sp_list) {
> if (sp->gfn != gfn) {
> collisions++;
> continue;
> @@ -2533,8 +2535,7 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu,
>
> sp->gfn = gfn;
> sp->role = role;
> - hlist_add_head(&sp->hash_link,
> - &vcpu->kvm->arch.mmu_page_hash[kvm_page_table_hashfn(gfn)]);
> + hlist_add_head(&sp->hash_link, sp_list);
> if (!direct) {
> /*
> * we should do write protection before syncing pages
> --
> 2.26.0
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 2/2] KVM: x86/mmu: Optimize MMU page cache lookup for fully direct MMUs
2020-06-23 19:40 ` [PATCH 2/2] KVM: x86/mmu: Optimize MMU page cache lookup for fully direct MMUs Sean Christopherson
@ 2020-06-23 20:28 ` Jon Cargille
0 siblings, 0 replies; 6+ messages in thread
From: Jon Cargille @ 2020-06-23 20:28 UTC (permalink / raw)
To: Sean Christopherson
Cc: Paolo Bonzini, Vitaly Kuznetsov, Wanpeng Li, Jim Mattson,
Joerg Roedel, kvm, linux-kernel, Peter Feiner
LGTM.
Reviewed-By: Jon Cargille <jcargill@google.com>
On Tue, Jun 23, 2020 at 12:40 PM Sean Christopherson
<sean.j.christopherson@intel.com> wrote:
>
> Skip the unsync checks and the write flooding clearing for fully direct
> MMUs, which are guaranteed to not have unsync'd or indirect pages (write
> flooding detection only applies to indirect pages). For TDP, this
> avoids unnecessary memory reads and writes, and for the write flooding
> count will also avoid dirtying a cache line (unsync_child_bitmap itself
> consumes a cache line, i.e. write_flooding_count is guaranteed to be in
> a different cache line than parent_ptes).
>
> Signed-off-by: Sean Christopherson <sean.j.christopherson@intel.com>
> ---
> arch/x86/kvm/mmu/mmu.c | 9 +++++++--
> 1 file changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
> index 67f8f82e9783..c568a5c55276 100644
> --- a/arch/x86/kvm/mmu/mmu.c
> +++ b/arch/x86/kvm/mmu/mmu.c
> @@ -2475,6 +2475,7 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu,
> int direct,
> unsigned int access)
> {
> + bool direct_mmu = vcpu->arch.mmu->direct_map;
> union kvm_mmu_page_role role;
> struct hlist_head *sp_list;
> unsigned quadrant;
> @@ -2490,8 +2491,7 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu,
> if (role.direct)
> role.gpte_is_8_bytes = true;
> role.access = access;
> - if (!vcpu->arch.mmu->direct_map
> - && vcpu->arch.mmu->root_level <= PT32_ROOT_LEVEL) {
> + if (!direct_mmu && vcpu->arch.mmu->root_level <= PT32_ROOT_LEVEL) {
> quadrant = gaddr >> (PAGE_SHIFT + (PT64_PT_BITS * level));
> quadrant &= (1 << ((PT32_PT_BITS - PT64_PT_BITS) * level)) - 1;
> role.quadrant = quadrant;
> @@ -2510,6 +2510,9 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu,
> if (sp->role.word != role.word)
> continue;
>
> + if (direct_mmu)
> + goto trace_get_page;
> +
> if (sp->unsync) {
> /* The page is good, but __kvm_sync_page might still end
> * up zapping it. If so, break in order to rebuild it.
> @@ -2525,6 +2528,8 @@ static struct kvm_mmu_page *kvm_mmu_get_page(struct kvm_vcpu *vcpu,
> kvm_make_request(KVM_REQ_TLB_FLUSH_CURRENT, vcpu);
>
> __clear_sp_write_flooding_count(sp);
> +
> +trace_get_page:
> trace_kvm_mmu_get_page(sp, false);
> goto out;
> }
> --
> 2.26.0
>
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH 0/2] KVM: x86/mmu: Optimizations for kvm_get_mmu_page()
2020-06-23 19:40 [PATCH 0/2] KVM: x86/mmu: Optimizations for kvm_get_mmu_page() Sean Christopherson
2020-06-23 19:40 ` [PATCH 1/2] KVM: x86/mmu: Avoid multiple hash lookups in kvm_get_mmu_page() Sean Christopherson
2020-06-23 19:40 ` [PATCH 2/2] KVM: x86/mmu: Optimize MMU page cache lookup for fully direct MMUs Sean Christopherson
@ 2020-07-03 17:17 ` Paolo Bonzini
2 siblings, 0 replies; 6+ messages in thread
From: Paolo Bonzini @ 2020-07-03 17:17 UTC (permalink / raw)
To: Sean Christopherson
Cc: Vitaly Kuznetsov, Wanpeng Li, Jim Mattson, Joerg Roedel, kvm,
linux-kernel, Peter Feiner, Jon Cargille
On 23/06/20 21:40, Sean Christopherson wrote:
> Avoid multiple hash lookups in kvm_get_mmu_page(), and tweak the cache
> loop to optimize it for TDP.
>
> Sean Christopherson (2):
> KVM: x86/mmu: Avoid multiple hash lookups in kvm_get_mmu_page()
> KVM: x86/mmu: Optimize MMU page cache lookup for fully direct MMUs
>
> arch/x86/kvm/mmu/mmu.c | 26 ++++++++++++++++----------
> 1 file changed, 16 insertions(+), 10 deletions(-)
>
Queued, thanks.
Paolo
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2020-07-03 17:17 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-06-23 19:40 [PATCH 0/2] KVM: x86/mmu: Optimizations for kvm_get_mmu_page() Sean Christopherson
2020-06-23 19:40 ` [PATCH 1/2] KVM: x86/mmu: Avoid multiple hash lookups in kvm_get_mmu_page() Sean Christopherson
2020-06-23 20:27 ` Jon Cargille
2020-06-23 19:40 ` [PATCH 2/2] KVM: x86/mmu: Optimize MMU page cache lookup for fully direct MMUs Sean Christopherson
2020-06-23 20:28 ` Jon Cargille
2020-07-03 17:17 ` [PATCH 0/2] KVM: x86/mmu: Optimizations for kvm_get_mmu_page() Paolo Bonzini
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).