From: Vipin Sharma <vipinsh@google.com>
To: seanjc@google.com, pbonzini@redhat.com, bgardon@google.com,
dmatlack@google.com
Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
Vipin Sharma <vipinsh@google.com>
Subject: [Patch v3 2/7] KVM: x86/mmu: Atomically clear SPTE dirty state in the clear-dirty-log flow
Date: Fri, 10 Feb 2023 17:46:21 -0800 [thread overview]
Message-ID: <20230211014626.3659152-3-vipinsh@google.com> (raw)
In-Reply-To: <20230211014626.3659152-1-vipinsh@google.com>
Do atomic-AND to clear the dirty state of SPTEs. Optimize clear-dirty-log
flow by avoiding to go through __handle_changed_spte() and directly call
kvm_set_pfn_dirty() instead.
Atomic-AND allows to fetch the latest value in SPTE, clear only its
dirty state and set the new SPTE value. This optimization avoids
executing unnecessary checks by not calling __handle_changed_spte().
With the removal of tdp_mmu_set_spte_no_dirty_log(), "record_dirty_log"
parameter in __tdp_mmu_set_spte() is now obsolete. It will always be set
to true by its caller. This dead code will be cleaned up in future
commits.
Tested on a VM (160 vCPUs, 160 GB memory) and found that performance of clear
dirty log stage improved by ~40% in dirty_log_perf_test
Before optimization:
--------------------
Iteration 1 clear dirty log time: 3.638543593s
Iteration 2 clear dirty log time: 3.145032742s
Iteration 3 clear dirty log time: 3.142340358s
Clear dirty log over 3 iterations took 9.925916693s. (Avg 3.308638897s/iteration)
After optimization:
-------------------
Iteration 1 clear dirty log time: 2.318988110s
Iteration 2 clear dirty log time: 1.794470164s
Iteration 3 clear dirty log time: 1.791668628s
Clear dirty log over 3 iterations took 5.905126902s. (Avg 1.968375634s/iteration)
Signed-off-by: Vipin Sharma <vipinsh@google.com>
---
arch/x86/kvm/mmu/tdp_iter.h | 14 ++++++++++++++
arch/x86/kvm/mmu/tdp_mmu.c | 35 +++++++++++++++--------------------
2 files changed, 29 insertions(+), 20 deletions(-)
diff --git a/arch/x86/kvm/mmu/tdp_iter.h b/arch/x86/kvm/mmu/tdp_iter.h
index c11c5d00b2c1..fae559559a80 100644
--- a/arch/x86/kvm/mmu/tdp_iter.h
+++ b/arch/x86/kvm/mmu/tdp_iter.h
@@ -58,6 +58,20 @@ static inline u64 kvm_tdp_mmu_write_spte(tdp_ptep_t sptep, u64 old_spte,
return old_spte;
}
+static inline u64 tdp_mmu_clear_spte_bits(tdp_ptep_t sptep, u64 old_spte,
+ u64 mask, int level)
+{
+ atomic64_t *sptep_atomic;
+
+ if (kvm_tdp_mmu_spte_need_atomic_write(old_spte, level)) {
+ sptep_atomic = (atomic64_t *)rcu_dereference(sptep);
+ return (u64)atomic64_fetch_and(~mask, sptep_atomic);
+ }
+
+ __kvm_tdp_mmu_write_spte(sptep, old_spte & ~mask);
+ return old_spte;
+}
+
/*
* A TDP iterator performs a pre-order walk over a TDP paging structure.
*/
diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c
index bba33aea0fb0..66ccbeb9d845 100644
--- a/arch/x86/kvm/mmu/tdp_mmu.c
+++ b/arch/x86/kvm/mmu/tdp_mmu.c
@@ -771,13 +771,6 @@ static inline void tdp_mmu_set_spte_no_acc_track(struct kvm *kvm,
_tdp_mmu_set_spte(kvm, iter, new_spte, false, true);
}
-static inline void tdp_mmu_set_spte_no_dirty_log(struct kvm *kvm,
- struct tdp_iter *iter,
- u64 new_spte)
-{
- _tdp_mmu_set_spte(kvm, iter, new_spte, true, false);
-}
-
#define tdp_root_for_each_pte(_iter, _root, _start, _end) \
for_each_tdp_pte(_iter, _root, _start, _end)
@@ -1677,8 +1670,13 @@ bool kvm_tdp_mmu_clear_dirty_slot(struct kvm *kvm,
static void clear_dirty_pt_masked(struct kvm *kvm, struct kvm_mmu_page *root,
gfn_t gfn, unsigned long mask, bool wrprot)
{
+ /*
+ * Either all SPTEs in TDP MMU will need write protection or none. This
+ * contract will not be modified for TDP MMU pages.
+ */
+ u64 clear_bit = (wrprot || !kvm_ad_enabled()) ? PT_WRITABLE_MASK :
+ shadow_dirty_mask;
struct tdp_iter iter;
- u64 new_spte;
rcu_read_lock();
@@ -1693,19 +1691,16 @@ static void clear_dirty_pt_masked(struct kvm *kvm, struct kvm_mmu_page *root,
mask &= ~(1UL << (iter.gfn - gfn));
- if (wrprot || spte_ad_need_write_protect(iter.old_spte)) {
- if (is_writable_pte(iter.old_spte))
- new_spte = iter.old_spte & ~PT_WRITABLE_MASK;
- else
- continue;
- } else {
- if (iter.old_spte & shadow_dirty_mask)
- new_spte = iter.old_spte & ~shadow_dirty_mask;
- else
- continue;
- }
+ if (!(iter.old_spte & clear_bit))
+ continue;
- tdp_mmu_set_spte_no_dirty_log(kvm, &iter, new_spte);
+ iter.old_spte = tdp_mmu_clear_spte_bits(iter.sptep,
+ iter.old_spte,
+ clear_bit, iter.level);
+ trace_kvm_tdp_mmu_spte_changed(iter.as_id, iter.gfn, iter.level,
+ iter.old_spte,
+ iter.old_spte & ~clear_bit);
+ kvm_set_pfn_dirty(spte_to_pfn(iter.old_spte));
}
rcu_read_unlock();
--
2.39.1.581.gbfd45094c4-goog
next prev parent reply other threads:[~2023-02-11 1:46 UTC|newest]
Thread overview: 26+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-02-11 1:46 [Patch v3 0/7] Optimize clear dirty log Vipin Sharma
2023-02-11 1:46 ` [Patch v3 1/7] KVM: x86/mmu: Add a helper function to check if an SPTE needs atomic write Vipin Sharma
2023-02-11 1:46 ` Vipin Sharma [this message]
2023-02-15 21:12 ` [Patch v3 2/7] KVM: x86/mmu: Atomically clear SPTE dirty state in the clear-dirty-log flow David Matlack
2023-03-17 22:59 ` Sean Christopherson
2023-03-17 23:50 ` Vipin Sharma
2023-02-11 1:46 ` [Patch v3 3/7] KVM: x86/mmu: Remove "record_dirty_log" in __tdp_mmu_set_spte() Vipin Sharma
2023-02-15 21:10 ` David Matlack
2023-02-11 1:46 ` [Patch v3 4/7] KVM: x86/mmu: Optimize SPTE change for aging gfn range Vipin Sharma
2023-02-15 21:15 ` David Matlack
2023-03-21 0:51 ` Sean Christopherson
2023-02-11 1:46 ` [Patch v3 5/7] KVM: x86/mmu: Remove "record_acc_track" in __tdp_mmu_set_spte() Vipin Sharma
2023-02-22 19:31 ` David Matlack
2023-02-11 1:46 ` [Patch v3 6/7] KVM: x86/mmu: Remove handle_changed_spte_dirty_log() Vipin Sharma
2023-02-22 19:36 ` David Matlack
2023-02-11 1:46 ` [Patch v3 7/7] KVM: x86/mmu: Merge all handle_changed_pte* functions Vipin Sharma
2023-02-22 19:42 ` David Matlack
2023-03-17 22:51 ` Sean Christopherson
2023-03-17 23:48 ` Vipin Sharma
2023-03-17 22:57 ` [Patch v3 0/7] Optimize clear dirty log Sean Christopherson
2023-03-17 23:51 ` Vipin Sharma
2023-03-21 0:41 ` Sean Christopherson
2023-03-21 18:11 ` Vipin Sharma
2023-03-21 19:57 ` Sean Christopherson
2023-03-21 20:40 ` Sean Christopherson
2023-03-21 21:38 ` Sean Christopherson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230211014626.3659152-3-vipinsh@google.com \
--to=vipinsh@google.com \
--cc=bgardon@google.com \
--cc=dmatlack@google.com \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=pbonzini@redhat.com \
--cc=seanjc@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).