From: Yu Zhao <yuzhao@google.com>
To: Andrew Morton <akpm@linux-foundation.org>,
Paolo Bonzini <pbonzini@redhat.com>
Cc: "Jason A. Donenfeld" <Jason@zx2c4.com>,
x86@kernel.org, Gavin Shan <gshan@redhat.com>,
kvm@vger.kernel.org, linux-doc@vger.kernel.org,
Catalin Marinas <catalin.marinas@arm.com>,
Dave Hansen <dave.hansen@linux.intel.com>,
Peter Xu <peterx@redhat.com>,
linux-mm@kvack.org, Ben Gardon <bgardon@google.com>,
Chao Peng <chao.p.peng@linux.intel.com>,
Will Deacon <will@kernel.org>,
Gaosheng Cui <cuigaosheng1@huawei.com>,
Marc Zyngier <maz@kernel.org>, "H. Peter Anvin" <hpa@zytor.com>,
Yu Zhao <yuzhao@google.com>, Jonathan Corbet <corbet@lwn.net>,
Alistair Popple <apopple@nvidia.com>,
Jason Gunthorpe <jgg@ziepe.ca>, Ingo Molnar <mingo@redhat.com>,
Zenghui Yu <yuzenghui@huawei.com>,
linux-trace-kernel@vger.kernel.org, linux-mm@google.com,
Thomas Huth <thuth@redhat.com>,
Suzuki K Poulose <suzuki.poulose@arm.com>,
Nicholas Piggin <npiggin@gmail.com>,
Borislav Petkov <bp@alien8.de>,
Steven Rostedt <rostedt@goodmis.org>,
kvmarm@lists.linux.dev, Thomas Gleixner <tglx@linutronix.de>,
linux-arm-ke rnel@lists.infradead.org,
Fabiano Rosas <farosas@linux.ibm.com>,
Michael Larabel <michael@michaellarabel.com>,
Sean Christopherson <seanjc@google.com>,
linux-kernel@vger.kernel.org,
Oliver Upton <oliver.upton@linux.dev>,
James Morse <james.morse@arm.com>,
Masami Hiramatsu <mhiramat@kernel.org>,
Anup Patel <anup@brainfault.org>,
linuxppc-dev@lists.ozlabs.org, Mike Rapoport <rppt@kernel.org>
Subject: [PATCH mm-unstable v2 09/10] kvm/x86: add kvm_arch_test_clear_young()
Date: Fri, 26 May 2023 17:44:34 -0600 [thread overview]
Message-ID: <20230526234435.662652-10-yuzhao@google.com> (raw)
In-Reply-To: <20230526234435.662652-1-yuzhao@google.com>
Implement kvm_arch_test_clear_young() to support the fast path in
mmu_notifier_ops->test_clear_young().
It focuses on a simple case, i.e., TDP MMU sets the accessed bit in
KVM PTEs and VMs are not nested, where it can rely on RCU and
clear_bit() to safely clear the accessed bit without taking
kvm->mmu_lock. Complex cases fall back to the existing slow path
where kvm->mmu_lock is then taken.
Signed-off-by: Yu Zhao <yuzhao@google.com>
---
arch/x86/include/asm/kvm_host.h | 7 +++++++
arch/x86/kvm/mmu/tdp_mmu.c | 34 +++++++++++++++++++++++++++++++++
2 files changed, 41 insertions(+)
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 753c67072c47..d6dfdebe3d94 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -2223,4 +2223,11 @@ int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages);
*/
#define KVM_EXIT_HYPERCALL_MBZ GENMASK_ULL(31, 1)
+#define kvm_arch_has_test_clear_young kvm_arch_has_test_clear_young
+static inline bool kvm_arch_has_test_clear_young(void)
+{
+ return IS_ENABLED(CONFIG_X86_64) &&
+ (!IS_REACHABLE(CONFIG_KVM) || (tdp_mmu_enabled && shadow_accessed_mask));
+}
+
#endif /* _ASM_X86_KVM_HOST_H */
diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c
index 08340219c35a..6875a819e007 100644
--- a/arch/x86/kvm/mmu/tdp_mmu.c
+++ b/arch/x86/kvm/mmu/tdp_mmu.c
@@ -1232,6 +1232,40 @@ bool kvm_tdp_mmu_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range)
return kvm_tdp_mmu_handle_gfn(kvm, range, test_age_gfn);
}
+bool kvm_arch_test_clear_young(struct kvm *kvm, struct kvm_gfn_range *range)
+{
+ struct kvm_mmu_page *root;
+ int offset = ffs(shadow_accessed_mask) - 1;
+
+ if (kvm_shadow_root_allocated(kvm))
+ return true;
+
+ rcu_read_lock();
+
+ list_for_each_entry_rcu(root, &kvm->arch.tdp_mmu_roots, link) {
+ struct tdp_iter iter;
+
+ if (kvm_mmu_page_as_id(root) != range->slot->as_id)
+ continue;
+
+ tdp_root_for_each_leaf_pte(iter, root, range->start, range->end) {
+ u64 *sptep = rcu_dereference(iter.sptep);
+
+ VM_WARN_ON_ONCE(!page_count(virt_to_page(sptep)));
+
+ if (!(iter.old_spte & shadow_accessed_mask))
+ continue;
+
+ if (kvm_should_clear_young(range, iter.gfn))
+ clear_bit(offset, (unsigned long *)sptep);
+ }
+ }
+
+ rcu_read_unlock();
+
+ return false;
+}
+
static bool set_spte_gfn(struct kvm *kvm, struct tdp_iter *iter,
struct kvm_gfn_range *range)
{
--
2.41.0.rc0.172.g3f132b7071-goog
next prev parent reply other threads:[~2023-05-26 23:53 UTC|newest]
Thread overview: 41+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-05-26 23:44 [PATCH mm-unstable v2 00/10] mm/kvm: locklessly clear the accessed bit Yu Zhao
2023-05-26 23:44 ` [PATCH mm-unstable v2 01/10] mm/kvm: add mmu_notifier_ops->test_clear_young() Yu Zhao
2023-06-06 8:34 ` Tzung-Bi Shih
2023-06-09 1:00 ` Yu Zhao
[not found] ` <ZHedMX470b7EMwbe@ziepe.ca>
2023-06-09 9:04 ` Paolo Bonzini
2023-06-15 17:42 ` Sean Christopherson
2023-06-20 7:30 ` Nicholas Piggin
2023-05-26 23:44 ` [PATCH mm-unstable v2 02/10] mm/kvm: use mmu_notifier_ops->test_clear_young() Yu Zhao
2023-05-26 23:44 ` [PATCH mm-unstable v2 03/10] kvm/arm64: export stage2_try_set_pte() and macros Yu Zhao
2023-05-26 23:44 ` [PATCH mm-unstable v2 04/10] kvm/arm64: make stage2 page tables RCU safe Yu Zhao
2023-05-27 18:08 ` Oliver Upton
2023-05-27 20:13 ` Yu Zhao
2023-05-30 19:37 ` Oliver Upton
2023-05-30 20:06 ` Yu Zhao
[not found] ` <ZHef0VsZvZ1Vnz0u@linux.dev>
2023-05-31 23:10 ` Yu Zhao
2023-05-31 23:22 ` Oliver Upton
2023-05-31 23:41 ` Yu Zhao
2023-05-26 23:44 ` [PATCH mm-unstable v2 05/10] kvm/arm64: add kvm_arch_test_clear_young() Yu Zhao
[not found] ` <ZHemUc3DiSbxQbxJ@linux.dev>
2023-05-31 21:12 ` Yu Zhao
2023-05-26 23:44 ` [PATCH mm-unstable v2 06/10] kvm/powerpc: make radix page tables RCU safe Yu Zhao
2023-06-20 6:32 ` Nicholas Piggin
2023-06-20 8:00 ` Yu Zhao
2023-06-20 10:49 ` Nicholas Piggin
2023-05-26 23:44 ` [PATCH mm-unstable v2 07/10] kvm/powerpc: add kvm_arch_test_clear_young() Yu Zhao
2023-06-20 7:47 ` Nicholas Piggin
2023-06-21 0:38 ` Yu Zhao
2023-06-21 2:51 ` Nicholas Piggin
2023-05-26 23:44 ` [PATCH mm-unstable v2 08/10] kvm/x86: move tdp_mmu_enabled and shadow_accessed_mask Yu Zhao
2023-06-15 16:59 ` Sean Christopherson
2023-05-26 23:44 ` Yu Zhao [this message]
2023-06-09 9:06 ` [PATCH mm-unstable v2 09/10] kvm/x86: add kvm_arch_test_clear_young() Paolo Bonzini
2023-06-15 18:26 ` Sean Christopherson
2023-05-26 23:44 ` [PATCH mm-unstable v2 10/10] mm: multi-gen LRU: use mmu_notifier_test_clear_young() Yu Zhao
2023-06-09 0:59 ` kvm/arm64: Spark benchmark Yu Zhao
2023-06-09 13:04 ` Marc Zyngier
2023-06-18 20:11 ` Yu Zhao
2023-06-09 0:59 ` kvm/powerpc: memcached benchmark Yu Zhao
2023-06-09 0:59 ` kvm/x86: multichase benchmark Yu Zhao
2023-06-18 19:19 ` Yu Zhao
2023-06-09 9:07 ` [PATCH mm-unstable v2 00/10] mm/kvm: locklessly clear the accessed bit Paolo Bonzini
2023-06-20 2:19 ` Yu Zhao
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20230526234435.662652-10-yuzhao@google.com \
--to=yuzhao@google.com \
--cc=Jason@zx2c4.com \
--cc=akpm@linux-foundation.org \
--cc=apopple@nvidia.com \
--cc=bgardon@google.com \
--cc=bp@alien8.de \
--cc=catalin.marinas@arm.com \
--cc=chao.p.peng@linux.intel.com \
--cc=corbet@lwn.net \
--cc=cuigaosheng1@huawei.com \
--cc=dave.hansen@linux.intel.com \
--cc=gshan@redhat.com \
--cc=hpa@zytor.com \
--cc=jgg@ziepe.ca \
--cc=kvm@vger.kernel.org \
--cc=kvmarm@lists.linux.dev \
--cc=linux-doc@vger.kernel.org \
--cc=linux-mm@google.com \
--cc=linux-mm@kvack.org \
--cc=linux-trace-kernel@vger.kernel.org \
--cc=maz@kernel.org \
--cc=mingo@redhat.com \
--cc=npiggin@gmail.com \
--cc=pbonzini@redhat.com \
--cc=peterx@redhat.com \
--cc=rostedt@goodmis.org \
--cc=suzuki.poulose@arm.com \
--cc=tglx@linutronix.de \
--cc=thuth@redhat.com \
--cc=will@kernel.org \
--cc=x86@kernel.org \
--cc=yuzenghui@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).