From: James Houghton <jthoughton@google.com> To: Andrew Morton <akpm@linux-foundation.org>, Paolo Bonzini <pbonzini@redhat.com> Cc: Yu Zhao <yuzhao@google.com>, David Matlack <dmatlack@google.com>, Marc Zyngier <maz@kernel.org>, Oliver Upton <oliver.upton@linux.dev>, Sean Christopherson <seanjc@google.com>, Jonathan Corbet <corbet@lwn.net>, James Morse <james.morse@arm.com>, Suzuki K Poulose <suzuki.poulose@arm.com>, Zenghui Yu <yuzenghui@huawei.com>, Catalin Marinas <catalin.marinas@arm.com>, Will Deacon <will@kernel.org>, Thomas Gleixner <tglx@linutronix.de>, Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>, Dave Hansen <dave.hansen@linux.intel.com>, "H. Peter Anvin" <hpa@zytor.com>, Steven Rostedt <rostedt@goodmis.org>, Masami Hiramatsu <mhiramat@kernel.org>, Mathieu Desnoyers <mathieu.desnoyers@efficios.com>, Shaoqin Huang <shahuang@redhat.com>, Gavin Shan <gshan@redhat.com>, Ricardo Koller <ricarkol@google.com>, Raghavendra Rao Ananta <rananta@google.com>, Ryan Roberts <ryan.roberts@arm.com>, David Rientjes <rientjes@google.com>, Axel Rasmussen <axelrasmussen@google.com>, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, James Houghton <jthoughton@google.com> Subject: [PATCH v3 5/7] KVM: x86: Participate in bitmap-based PTE aging Date: Mon, 1 Apr 2024 23:29:44 +0000 [thread overview] Message-ID: <20240401232946.1837665-6-jthoughton@google.com> (raw) In-Reply-To: <20240401232946.1837665-1-jthoughton@google.com> Only handle the TDP MMU case for now. In other cases, if a bitmap was not provided, fallback to the slowpath that takes mmu_lock, or, if a bitmap was provided, inform the caller that the bitmap is unreliable. Suggested-by: Yu Zhao <yuzhao@google.com> Signed-off-by: James Houghton <jthoughton@google.com> --- arch/x86/include/asm/kvm_host.h | 14 ++++++++++++++ arch/x86/kvm/mmu/mmu.c | 16 ++++++++++++++-- arch/x86/kvm/mmu/tdp_mmu.c | 10 +++++++++- 3 files changed, 37 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 3b58e2306621..c30918d0887e 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2324,4 +2324,18 @@ int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages); */ #define KVM_EXIT_HYPERCALL_MBZ GENMASK_ULL(31, 1) +#define kvm_arch_prepare_bitmap_age kvm_arch_prepare_bitmap_age +static inline bool kvm_arch_prepare_bitmap_age(struct mmu_notifier *mn) +{ + /* + * Indicate that we support bitmap-based aging when using the TDP MMU + * and the accessed bit is available in the TDP page tables. + * + * We have no other preparatory work to do here, so we do not need to + * redefine kvm_arch_finish_bitmap_age(). + */ + return IS_ENABLED(CONFIG_X86_64) && tdp_mmu_enabled + && shadow_accessed_mask; +} + #endif /* _ASM_X86_KVM_HOST_H */ diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 992e651540e8..fae1a75750bb 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1674,8 +1674,14 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { bool young = false; - if (kvm_memslots_have_rmaps(kvm)) + if (kvm_memslots_have_rmaps(kvm)) { + if (range->lockless) { + kvm_age_set_unreliable(range); + return false; + } + young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); + } if (tdp_mmu_enabled) young |= kvm_tdp_mmu_age_gfn_range(kvm, range); @@ -1687,8 +1693,14 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { bool young = false; - if (kvm_memslots_have_rmaps(kvm)) + if (kvm_memslots_have_rmaps(kvm)) { + if (range->lockless) { + kvm_age_set_unreliable(range); + return false; + } + young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap); + } if (tdp_mmu_enabled) young |= kvm_tdp_mmu_test_age_gfn(kvm, range); diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index d078157e62aa..edea01bc145f 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1217,6 +1217,9 @@ static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter, if (!is_accessed_spte(iter->old_spte)) return false; + if (!kvm_gfn_should_age(range, iter->gfn)) + return false; + if (spte_ad_enabled(iter->old_spte)) { iter->old_spte = tdp_mmu_clear_spte_bits(iter->sptep, iter->old_spte, @@ -1250,7 +1253,12 @@ bool kvm_tdp_mmu_age_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) static bool test_age_gfn(struct kvm *kvm, struct tdp_iter *iter, struct kvm_gfn_range *range) { - return is_accessed_spte(iter->old_spte); + bool young = is_accessed_spte(iter->old_spte); + + if (young) + kvm_gfn_record_young(range, iter->gfn); + + return young; } bool kvm_tdp_mmu_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) -- 2.44.0.478.gd926399ef9-goog
WARNING: multiple messages have this Message-ID (diff)
From: James Houghton <jthoughton@google.com> To: Andrew Morton <akpm@linux-foundation.org>, Paolo Bonzini <pbonzini@redhat.com> Cc: Yu Zhao <yuzhao@google.com>, David Matlack <dmatlack@google.com>, Marc Zyngier <maz@kernel.org>, Oliver Upton <oliver.upton@linux.dev>, Sean Christopherson <seanjc@google.com>, Jonathan Corbet <corbet@lwn.net>, James Morse <james.morse@arm.com>, Suzuki K Poulose <suzuki.poulose@arm.com>, Zenghui Yu <yuzenghui@huawei.com>, Catalin Marinas <catalin.marinas@arm.com>, Will Deacon <will@kernel.org>, Thomas Gleixner <tglx@linutronix.de>, Ingo Molnar <mingo@redhat.com>, Borislav Petkov <bp@alien8.de>, Dave Hansen <dave.hansen@linux.intel.com>, "H. Peter Anvin" <hpa@zytor.com>, Steven Rostedt <rostedt@goodmis.org>, Masami Hiramatsu <mhiramat@kernel.org>, Mathieu Desnoyers <mathieu.desnoyers@efficios.com>, Shaoqin Huang <shahuang@redhat.com>, Gavin Shan <gshan@redhat.com>, Ricardo Koller <ricarkol@google.com>, Raghavendra Rao Ananta <rananta@google.com>, Ryan Roberts <ryan.roberts@arm.com>, David Rientjes <rientjes@google.com>, Axel Rasmussen <axelrasmussen@google.com>, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, kvmarm@lists.linux.dev, kvm@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, James Houghton <jthoughton@google.com> Subject: [PATCH v3 5/7] KVM: x86: Participate in bitmap-based PTE aging Date: Mon, 1 Apr 2024 23:29:44 +0000 [thread overview] Message-ID: <20240401232946.1837665-6-jthoughton@google.com> (raw) In-Reply-To: <20240401232946.1837665-1-jthoughton@google.com> Only handle the TDP MMU case for now. In other cases, if a bitmap was not provided, fallback to the slowpath that takes mmu_lock, or, if a bitmap was provided, inform the caller that the bitmap is unreliable. Suggested-by: Yu Zhao <yuzhao@google.com> Signed-off-by: James Houghton <jthoughton@google.com> --- arch/x86/include/asm/kvm_host.h | 14 ++++++++++++++ arch/x86/kvm/mmu/mmu.c | 16 ++++++++++++++-- arch/x86/kvm/mmu/tdp_mmu.c | 10 +++++++++- 3 files changed, 37 insertions(+), 3 deletions(-) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 3b58e2306621..c30918d0887e 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -2324,4 +2324,18 @@ int memslot_rmap_alloc(struct kvm_memory_slot *slot, unsigned long npages); */ #define KVM_EXIT_HYPERCALL_MBZ GENMASK_ULL(31, 1) +#define kvm_arch_prepare_bitmap_age kvm_arch_prepare_bitmap_age +static inline bool kvm_arch_prepare_bitmap_age(struct mmu_notifier *mn) +{ + /* + * Indicate that we support bitmap-based aging when using the TDP MMU + * and the accessed bit is available in the TDP page tables. + * + * We have no other preparatory work to do here, so we do not need to + * redefine kvm_arch_finish_bitmap_age(). + */ + return IS_ENABLED(CONFIG_X86_64) && tdp_mmu_enabled + && shadow_accessed_mask; +} + #endif /* _ASM_X86_KVM_HOST_H */ diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 992e651540e8..fae1a75750bb 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -1674,8 +1674,14 @@ bool kvm_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { bool young = false; - if (kvm_memslots_have_rmaps(kvm)) + if (kvm_memslots_have_rmaps(kvm)) { + if (range->lockless) { + kvm_age_set_unreliable(range); + return false; + } + young = kvm_handle_gfn_range(kvm, range, kvm_age_rmap); + } if (tdp_mmu_enabled) young |= kvm_tdp_mmu_age_gfn_range(kvm, range); @@ -1687,8 +1693,14 @@ bool kvm_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) { bool young = false; - if (kvm_memslots_have_rmaps(kvm)) + if (kvm_memslots_have_rmaps(kvm)) { + if (range->lockless) { + kvm_age_set_unreliable(range); + return false; + } + young = kvm_handle_gfn_range(kvm, range, kvm_test_age_rmap); + } if (tdp_mmu_enabled) young |= kvm_tdp_mmu_test_age_gfn(kvm, range); diff --git a/arch/x86/kvm/mmu/tdp_mmu.c b/arch/x86/kvm/mmu/tdp_mmu.c index d078157e62aa..edea01bc145f 100644 --- a/arch/x86/kvm/mmu/tdp_mmu.c +++ b/arch/x86/kvm/mmu/tdp_mmu.c @@ -1217,6 +1217,9 @@ static bool age_gfn_range(struct kvm *kvm, struct tdp_iter *iter, if (!is_accessed_spte(iter->old_spte)) return false; + if (!kvm_gfn_should_age(range, iter->gfn)) + return false; + if (spte_ad_enabled(iter->old_spte)) { iter->old_spte = tdp_mmu_clear_spte_bits(iter->sptep, iter->old_spte, @@ -1250,7 +1253,12 @@ bool kvm_tdp_mmu_age_gfn_range(struct kvm *kvm, struct kvm_gfn_range *range) static bool test_age_gfn(struct kvm *kvm, struct tdp_iter *iter, struct kvm_gfn_range *range) { - return is_accessed_spte(iter->old_spte); + bool young = is_accessed_spte(iter->old_spte); + + if (young) + kvm_gfn_record_young(range, iter->gfn); + + return young; } bool kvm_tdp_mmu_test_age_gfn(struct kvm *kvm, struct kvm_gfn_range *range) -- 2.44.0.478.gd926399ef9-goog _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2024-04-01 23:30 UTC|newest] Thread overview: 64+ messages / expand[flat|nested] mbox.gz Atom feed top 2024-04-01 23:29 [PATCH v3 0/7] mm/kvm: Improve parallelism for access bit harvesting James Houghton 2024-04-01 23:29 ` James Houghton 2024-04-01 23:29 ` [PATCH v3 1/7] mm: Add a bitmap into mmu_notifier_{clear,test}_young James Houghton 2024-04-01 23:29 ` James Houghton 2024-04-04 18:52 ` David Hildenbrand 2024-04-04 18:52 ` David Hildenbrand 2024-04-09 18:31 ` James Houghton 2024-04-09 18:31 ` James Houghton 2024-04-09 19:35 ` David Hildenbrand 2024-04-09 19:35 ` David Hildenbrand 2024-04-11 0:35 ` James Houghton 2024-04-11 0:35 ` James Houghton 2024-04-12 18:45 ` David Matlack 2024-04-12 18:45 ` David Matlack 2024-04-19 20:34 ` James Houghton 2024-04-19 20:34 ` James Houghton 2024-04-01 23:29 ` [PATCH v3 2/7] KVM: Move MMU notifier function declarations James Houghton 2024-04-01 23:29 ` James Houghton 2024-04-01 23:29 ` [PATCH v3 3/7] KVM: Add basic bitmap support into kvm_mmu_notifier_test/clear_young James Houghton 2024-04-01 23:29 ` James Houghton 2024-04-12 20:28 ` David Matlack 2024-04-12 20:28 ` David Matlack 2024-04-19 20:41 ` James Houghton 2024-04-19 20:41 ` James Houghton 2024-04-01 23:29 ` [PATCH v3 4/7] KVM: x86: Move tdp_mmu_enabled and shadow_accessed_mask James Houghton 2024-04-01 23:29 ` James Houghton 2024-04-01 23:29 ` James Houghton [this message] 2024-04-01 23:29 ` [PATCH v3 5/7] KVM: x86: Participate in bitmap-based PTE aging James Houghton 2024-04-11 17:08 ` David Matlack 2024-04-11 17:08 ` David Matlack 2024-04-11 17:28 ` David Matlack 2024-04-11 17:28 ` David Matlack 2024-04-11 18:00 ` David Matlack 2024-04-11 18:00 ` David Matlack 2024-04-11 18:07 ` David Matlack 2024-04-11 18:07 ` David Matlack 2024-04-19 20:47 ` James Houghton 2024-04-19 20:47 ` James Houghton 2024-04-19 21:06 ` David Matlack 2024-04-19 21:06 ` David Matlack 2024-04-19 21:48 ` James Houghton 2024-04-19 21:48 ` James Houghton 2024-04-21 0:19 ` Yu Zhao 2024-04-21 0:19 ` Yu Zhao 2024-04-12 20:44 ` David Matlack 2024-04-12 20:44 ` David Matlack 2024-04-19 20:54 ` James Houghton 2024-04-19 20:54 ` James Houghton 2024-04-01 23:29 ` [PATCH v3 6/7] KVM: arm64: " James Houghton 2024-04-01 23:29 ` James Houghton 2024-04-02 4:06 ` Yu Zhao 2024-04-02 4:06 ` Yu Zhao 2024-04-02 7:00 ` Oliver Upton 2024-04-02 7:00 ` Oliver Upton 2024-04-02 7:33 ` Marc Zyngier 2024-04-02 7:33 ` Marc Zyngier 2024-04-01 23:29 ` [PATCH v3 7/7] mm: multi-gen LRU: use mmu_notifier_test_clear_young() James Houghton 2024-04-01 23:29 ` James Houghton 2024-04-12 18:41 ` [PATCH v3 0/7] mm/kvm: Improve parallelism for access bit harvesting David Matlack 2024-04-12 18:41 ` David Matlack 2024-04-19 20:57 ` James Houghton 2024-04-19 20:57 ` James Houghton 2024-04-19 22:23 ` Oliver Upton 2024-04-19 22:23 ` Oliver Upton
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20240401232946.1837665-6-jthoughton@google.com \ --to=jthoughton@google.com \ --cc=akpm@linux-foundation.org \ --cc=axelrasmussen@google.com \ --cc=bp@alien8.de \ --cc=catalin.marinas@arm.com \ --cc=corbet@lwn.net \ --cc=dave.hansen@linux.intel.com \ --cc=dmatlack@google.com \ --cc=gshan@redhat.com \ --cc=hpa@zytor.com \ --cc=james.morse@arm.com \ --cc=kvm@vger.kernel.org \ --cc=kvmarm@lists.linux.dev \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=linux-trace-kernel@vger.kernel.org \ --cc=mathieu.desnoyers@efficios.com \ --cc=maz@kernel.org \ --cc=mhiramat@kernel.org \ --cc=mingo@redhat.com \ --cc=oliver.upton@linux.dev \ --cc=pbonzini@redhat.com \ --cc=rananta@google.com \ --cc=ricarkol@google.com \ --cc=rientjes@google.com \ --cc=rostedt@goodmis.org \ --cc=ryan.roberts@arm.com \ --cc=seanjc@google.com \ --cc=shahuang@redhat.com \ --cc=suzuki.poulose@arm.com \ --cc=tglx@linutronix.de \ --cc=will@kernel.org \ --cc=yuzenghui@huawei.com \ --cc=yuzhao@google.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.