From: David Matlack <dmatlack@google.com>
To: Paolo Bonzini <pbonzini@redhat.com>
Cc: kvm@vger.kernel.org, Ben Gardon <bgardon@google.com>,
Joerg Roedel <joro@8bytes.org>, Jim Mattson <jmattson@google.com>,
Wanpeng Li <wanpengli@tencent.com>,
Vitaly Kuznetsov <vkuznets@redhat.com>,
Sean Christopherson <seanjc@google.com>,
Janis Schoetterl-Glausch <scgl@linux.vnet.ibm.com>,
Junaid Shahid <junaids@google.com>,
Oliver Upton <oupton@google.com>,
Harish Barathvajasankar <hbarath@google.com>,
Peter Xu <peterx@redhat.com>, Peter Shier <pshier@google.com>,
David Matlack <dmatlack@google.com>
Subject: [RFC PATCH 03/15] KVM: x86/mmu: Automatically update iter->old_spte if cmpxchg fails
Date: Fri, 19 Nov 2021 23:57:47 +0000 [thread overview]
Message-ID: <20211119235759.1304274-4-dmatlack@google.com> (raw)
In-Reply-To: <20211119235759.1304274-1-dmatlack@google.com>
Consolidate a bunch of code that was manually re-reading the spte if the
cmpxchg fails. There is no extra cost of doing this because we already
have the spte value as a result of the cmpxchg (and in fact this
eliminates re-reading the spte), and none of the call sites depend on
iter->old_spte retaining the stale spte value.
Signed-off-by: David Matlack <dmatlack@google.com>
---
arch/x86/kvm/mmu/tdp_mmu.c | 56 ++++++++++++--------------------------
1 file changed, 18 insertions(+), 38 deletions(-)
diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c
index 377a96718a2e..cc9fe33c9b36 100644
--- a/arch/x86/kvm/mmu/tdp_mmu.c
+++ b/arch/x86/kvm/mmu/tdp_mmu.c
@@ -492,16 +492,22 @@ static void handle_changed_spte(struct kvm *kvm, int as_id, gfn_t gfn,
* and handle the associated bookkeeping. Do not mark the page dirty
* in KVM's dirty bitmaps.
*
+ * If setting the SPTE fails because it has changed, iter->old_spte will be
+ * updated with the updated value of the spte.
+ *
* @kvm: kvm instance
* @iter: a tdp_iter instance currently on the SPTE that should be set
* @new_spte: The value the SPTE should be set to
* Returns: true if the SPTE was set, false if it was not. If false is returned,
- * this function will have no side-effects.
+ * this function will have no side-effects other than updating
+ * iter->old_spte to the latest value of spte.
*/
static inline bool tdp_mmu_set_spte_atomic(struct kvm *kvm,
struct tdp_iter *iter,
u64 new_spte)
{
+ u64 old_spte;
+
lockdep_assert_held_read(&kvm->mmu_lock);
/*
@@ -515,9 +521,11 @@ static inline bool tdp_mmu_set_spte_atomic(struct kvm *kvm,
* Note, fast_pf_fix_direct_spte() can also modify TDP MMU SPTEs and
* does not hold the mmu_lock.
*/
- if (cmpxchg64(rcu_dereference(iter->sptep), iter->old_spte,
- new_spte) != iter->old_spte)
+ old_spte = cmpxchg64(rcu_dereference(iter->sptep), iter->old_spte, new_spte);
+ if (old_spte != iter->old_spte) {
+ iter->old_spte = old_spte;
return false;
+ }
__handle_changed_spte(kvm, iter->as_id, iter->gfn, iter->old_spte,
new_spte, iter->level, true);
@@ -747,14 +755,8 @@ static bool zap_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,
if (!shared) {
tdp_mmu_set_spte(kvm, &iter, 0);
flush = true;
- } else if (!tdp_mmu_zap_spte_atomic(kvm, &iter)) {
- /*
- * The iter must explicitly re-read the SPTE because
- * the atomic cmpxchg failed.
- */
- iter.old_spte = READ_ONCE(*rcu_dereference(iter.sptep));
+ } else if (!tdp_mmu_zap_spte_atomic(kvm, &iter))
goto retry;
- }
}
rcu_read_unlock();
@@ -978,13 +980,6 @@ int kvm_tdp_mmu_map(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
is_large_pte(iter.old_spte)) {
if (!tdp_mmu_zap_spte_atomic(vcpu->kvm, &iter))
break;
-
- /*
- * The iter must explicitly re-read the spte here
- * because the new value informs the !present
- * path below.
- */
- iter.old_spte = READ_ONCE(*rcu_dereference(iter.sptep));
}
if (!is_shadow_present_pte(iter.old_spte)) {
@@ -1190,14 +1185,9 @@ static bool wrprot_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,
new_spte = iter.old_spte & ~PT_WRITABLE_MASK;
- if (!tdp_mmu_set_spte_atomic(kvm, &iter, new_spte)) {
- /*
- * The iter must explicitly re-read the SPTE because
- * the atomic cmpxchg failed.
- */
- iter.old_spte = READ_ONCE(*rcu_dereference(iter.sptep));
+ if (!tdp_mmu_set_spte_atomic(kvm, &iter, new_spte))
goto retry;
- }
+
spte_set = true;
}
@@ -1258,14 +1248,9 @@ static bool clear_dirty_gfn_range(struct kvm *kvm, struct kvm_mmu_page *root,
continue;
}
- if (!tdp_mmu_set_spte_atomic(kvm, &iter, new_spte)) {
- /*
- * The iter must explicitly re-read the SPTE because
- * the atomic cmpxchg failed.
- */
- iter.old_spte = READ_ONCE(*rcu_dereference(iter.sptep));
+ if (!tdp_mmu_set_spte_atomic(kvm, &iter, new_spte))
goto retry;
- }
+
spte_set = true;
}
@@ -1391,14 +1376,9 @@ static bool zap_collapsible_spte_range(struct kvm *kvm,
pfn, PG_LEVEL_NUM))
continue;
- if (!tdp_mmu_zap_spte_atomic(kvm, &iter)) {
- /*
- * The iter must explicitly re-read the SPTE because
- * the atomic cmpxchg failed.
- */
- iter.old_spte = READ_ONCE(*rcu_dereference(iter.sptep));
+ if (!tdp_mmu_zap_spte_atomic(kvm, &iter))
goto retry;
- }
+
flush = true;
}
--
2.34.0.rc2.393.gf8c9666880-goog
next prev parent reply other threads:[~2021-11-19 23:58 UTC|newest]
Thread overview: 77+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-11-19 23:57 [RFC PATCH 00/15] KVM: x86/mmu: Eager Page Splitting for the TDP MMU David Matlack
2021-11-19 23:57 ` [RFC PATCH 01/15] KVM: x86/mmu: Rename rmap_write_protect to kvm_vcpu_write_protect_gfn David Matlack
2021-11-22 18:52 ` Ben Gardon
2021-11-26 12:18 ` Peter Xu
2021-11-19 23:57 ` [RFC PATCH 02/15] KVM: x86/mmu: Rename __rmap_write_protect to rmap_write_protect David Matlack
2021-11-22 18:52 ` Ben Gardon
2021-11-26 12:18 ` Peter Xu
2021-11-19 23:57 ` David Matlack [this message]
2021-11-22 18:52 ` [RFC PATCH 03/15] KVM: x86/mmu: Automatically update iter->old_spte if cmpxchg fails Ben Gardon
2021-11-30 23:25 ` David Matlack
2021-11-19 23:57 ` [RFC PATCH 04/15] KVM: x86/mmu: Factor out logic to atomically install a new page table David Matlack
2021-11-22 18:52 ` Ben Gardon
2021-11-30 23:27 ` David Matlack
2021-12-01 19:13 ` Sean Christopherson
2021-12-01 21:52 ` David Matlack
2021-11-19 23:57 ` [RFC PATCH 05/15] KVM: x86/mmu: Abstract mmu caches out to a separate struct David Matlack
2021-11-22 18:55 ` Ben Gardon
2021-11-22 18:55 ` Ben Gardon
2021-11-30 23:28 ` David Matlack
2021-11-19 23:57 ` [RFC PATCH 06/15] KVM: x86/mmu: Derive page role from parent David Matlack
2021-11-20 12:53 ` Paolo Bonzini
2021-11-27 2:07 ` Lai Jiangshan
2021-11-27 10:26 ` Paolo Bonzini
2021-11-30 23:31 ` David Matlack
2021-12-01 0:45 ` Sean Christopherson
2021-12-01 21:56 ` David Matlack
2021-11-19 23:57 ` [RFC PATCH 07/15] KVM: x86/mmu: Pass in vcpu->arch.mmu_caches instead of vcpu David Matlack
2021-11-22 18:56 ` Ben Gardon
2021-11-19 23:57 ` [RFC PATCH 08/15] KVM: x86/mmu: Helper method to check for large and present sptes David Matlack
2021-11-22 18:56 ` Ben Gardon
2021-12-01 18:34 ` Sean Christopherson
2021-12-01 21:13 ` David Matlack
2021-11-19 23:57 ` [RFC PATCH 09/15] KVM: x86/mmu: Move restore_acc_track_spte to spte.c David Matlack
2021-11-22 18:56 ` Ben Gardon
2021-11-19 23:57 ` [RFC PATCH 10/15] KVM: x86/mmu: Abstract need_resched logic from tdp_mmu_iter_cond_resched David Matlack
2021-11-22 18:56 ` Ben Gardon
2021-11-19 23:57 ` [RFC PATCH 11/15] KVM: x86/mmu: Refactor tdp_mmu iterators to take kvm_mmu_page root David Matlack
2021-11-22 18:56 ` Ben Gardon
2021-11-19 23:57 ` [RFC PATCH 12/15] KVM: x86/mmu: Split large pages when dirty logging is enabled David Matlack
2021-11-22 5:05 ` Nikunj A. Dadhania
2021-11-30 23:33 ` David Matlack
2021-11-22 19:30 ` Ben Gardon
2021-11-30 23:44 ` David Matlack
2021-11-26 12:01 ` Peter Xu
2021-11-30 23:56 ` David Matlack
2021-12-01 1:00 ` Sean Christopherson
2021-12-01 1:29 ` David Matlack
2021-12-01 2:29 ` Peter Xu
2021-12-01 18:29 ` Sean Christopherson
2021-12-01 21:36 ` David Matlack
2021-12-01 23:37 ` Sean Christopherson
2021-12-02 17:41 ` David Matlack
2021-12-02 18:42 ` Sean Christopherson
2021-12-03 0:00 ` David Matlack
2021-12-03 1:07 ` Sean Christopherson
2021-12-03 17:22 ` David Matlack
2021-11-19 23:57 ` [RFC PATCH 13/15] KVM: x86/mmu: Split large pages during CLEAR_DIRTY_LOG David Matlack
2021-11-26 12:17 ` Peter Xu
2021-12-01 0:16 ` David Matlack
2021-12-01 0:17 ` David Matlack
2021-12-01 4:03 ` Peter Xu
2021-12-01 22:14 ` David Matlack
2021-12-03 4:57 ` Peter Xu
2021-12-01 19:22 ` Sean Christopherson
2021-12-01 19:49 ` Ben Gardon
2021-12-01 20:16 ` Sean Christopherson
2021-12-01 22:11 ` Ben Gardon
2021-12-01 22:17 ` David Matlack
2021-11-19 23:57 ` [RFC PATCH 14/15] KVM: x86/mmu: Add tracepoint for splitting large pages David Matlack
2021-11-19 23:57 ` [RFC PATCH 15/15] KVM: x86/mmu: Update page stats when " David Matlack
2021-12-01 19:36 ` Sean Christopherson
2021-12-01 21:11 ` David Matlack
2021-11-26 14:13 ` [RFC PATCH 00/15] KVM: x86/mmu: Eager Page Splitting for the TDP MMU Peter Xu
2021-11-30 23:22 ` David Matlack
2021-12-01 4:10 ` Peter Xu
2021-12-01 4:19 ` Peter Xu
2021-12-01 21:46 ` David Matlack
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20211119235759.1304274-4-dmatlack@google.com \
--to=dmatlack@google.com \
--cc=bgardon@google.com \
--cc=hbarath@google.com \
--cc=jmattson@google.com \
--cc=joro@8bytes.org \
--cc=junaids@google.com \
--cc=kvm@vger.kernel.org \
--cc=oupton@google.com \
--cc=pbonzini@redhat.com \
--cc=peterx@redhat.com \
--cc=pshier@google.com \
--cc=scgl@linux.vnet.ibm.com \
--cc=seanjc@google.com \
--cc=vkuznets@redhat.com \
--cc=wanpengli@tencent.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).