From: "Suthikulpanit, Suravee" <suravee.suthikulpanit@amd.com>
To: Sean Christopherson <seanjc@google.com>,
Paolo Bonzini <pbonzini@redhat.com>
Cc: kvm@vger.kernel.org, linux-kernel@vger.kernel.org,
Maxim Levitsky <mlevitsk@redhat.com>,
Li RongQing <lirongqing@baidu.com>
Subject: Re: [PATCH v2 04/23] KVM: x86: Inhibit AVIC SPTEs if any vCPU enables x2APIC
Date: Tue, 13 Sep 2022 14:52:40 -0500 [thread overview]
Message-ID: <b6fcb487-56fc-12ea-6f67-b14b0b156ee0@amd.com> (raw)
In-Reply-To: <20220903002254.2411750-5-seanjc@google.com>
Sean,
This patch inhibits VM running in x2APIC mode on system w/ x2AVIC support.
On 9/2/2022 7:22 PM, Sean Christopherson wrote:
> Reintroduce APICV_INHIBIT_REASON_X2APIC as a "partial" inhibit for AMD
> to fix a bug where the APIC access page is visible to vCPUs that have
> x2APIC enabled, i.e. shouldn't be able to "see" the xAPIC MMIO region.
>
> On AMD, due to its "hybrid" mode where AVIC is enabled when x2APIC is
> enabled even without x2AVIC support, the bug occurs any time AVIC is
> enabled as x2APIC is fully emulated by KVM. I.e. hardware isn't aware
> that the guest is operating in x2APIC mode.
>
> Opportunistically drop the "can" while updating avic_activate_vmcb()'s
> comment, i.e. to state that KVM _does_ support the hybrid mode. Move
> the "Note:" down a line to conform to preferred kernel/KVM multi-line
> comment style.
>
> Leave Intel as-is for now to avoid a subtle performance regression, even
> though Intel likely suffers from the same bug. On Intel, in theory the
> bug rears its head only when vCPUs share host page tables (extremely
> likely) and x2APIC enabling is not consistent within the guest, i.e. if
> some vCPUs have x2APIC enabled and other does do not (unlikely to occur
> except in certain situations, e.g. bringing up APs).
>
> Fixes: 0e311d33bfbe ("KVM: SVM: Introduce hybrid-AVIC mode")
> Cc: stable@vger.kernel.org
> Suggested-by: Maxim Levitsky <mlevitsk@redhat.com>
> Signed-off-by: Sean Christopherson <seanjc@google.com>
> ---
> arch/x86/include/asm/kvm_host.h | 10 ++++++++++
> arch/x86/kvm/lapic.c | 4 +++-
> arch/x86/kvm/mmu/mmu.c | 2 +-
> arch/x86/kvm/svm/avic.c | 15 +++++++-------
> arch/x86/kvm/x86.c | 35 +++++++++++++++++++++++++++++----
> 5 files changed, 53 insertions(+), 13 deletions(-)
>
> diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
> index 2c96c43c313a..1fd1b66ceeb6 100644
> --- a/arch/x86/include/asm/kvm_host.h
> +++ b/arch/x86/include/asm/kvm_host.h
> @@ -1132,6 +1132,15 @@ enum kvm_apicv_inhibit {
> * AVIC is disabled because SEV doesn't support it.
> */
> APICV_INHIBIT_REASON_SEV,
> +
> + /*
> + * Due to sharing page tables across vCPUs, the xAPIC memslot must be
> + * inhibited if any vCPU has x2APIC enabled. Note, this is a "partial"
> + * inhibit; APICv can still be activated, but KVM mustn't retain/create
> + * SPTEs for the APIC access page. Like the APIC ID and APIC base
> + * inhibits, this is sticky for simplicity.
> + */
> + APICV_INHIBIT_REASON_X2APIC,
Actually, shouldn't the APICV_INHIBIT_REASON_X2APIC is set only when
vCPU has x2APIC enabled on the system with _NO x2AVIC support_ ? For
example, .....
> };
>
> struct kvm_arch {
> @@ -1903,6 +1912,7 @@ gpa_t kvm_mmu_gva_to_gpa_write(struct kvm_vcpu *vcpu, gva_t gva,
> gpa_t kvm_mmu_gva_to_gpa_system(struct kvm_vcpu *vcpu, gva_t gva,
> struct x86_exception *exception);
>
> +bool kvm_apicv_memslot_activated(struct kvm *kvm);
> bool kvm_apicv_activated(struct kvm *kvm);
> bool kvm_vcpu_apicv_activated(struct kvm_vcpu *vcpu);
> void kvm_vcpu_update_apicv(struct kvm_vcpu *vcpu);
> diff --git a/arch/x86/kvm/lapic.c b/arch/x86/kvm/lapic.c
> index 38e9b8e5278c..d956cd37908e 100644
> --- a/arch/x86/kvm/lapic.c
> +++ b/arch/x86/kvm/lapic.c
> @@ -2394,8 +2394,10 @@ void kvm_lapic_set_base(struct kvm_vcpu *vcpu, u64 value)
> }
> }
>
> - if (((old_value ^ value) & X2APIC_ENABLE) && (value & X2APIC_ENABLE))
> + if (((old_value ^ value) & X2APIC_ENABLE) && (value & X2APIC_ENABLE)) {
> kvm_apic_set_x2apic_id(apic, vcpu->vcpu_id);
> + kvm_set_apicv_inhibit(vcpu->kvm, APICV_INHIBIT_REASON_X2APIC);
> + }
.... Here, since we do not want to inhibit APICV/AVIC on system that can
support x2AVIC, this should be set in the vendor-specific call-back
function, where appropriate checks can be made.
>
> if ((old_value ^ value) & (MSR_IA32_APICBASE_ENABLE | X2APIC_ENABLE)) {
> kvm_vcpu_update_apicv(vcpu);
> diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c
> index e418ef3ecfcb..cea25552869f 100644
> --- a/arch/x86/kvm/mmu/mmu.c
> +++ b/arch/x86/kvm/mmu/mmu.c
> @@ -4150,7 +4150,7 @@ static int kvm_faultin_pfn(struct kvm_vcpu *vcpu, struct kvm_page_fault *fault)
> * when the AVIC is re-enabled.
> */
> if (slot && slot->id == APIC_ACCESS_PAGE_PRIVATE_MEMSLOT &&
> - !kvm_apicv_activated(vcpu->kvm))
> + !kvm_apicv_memslot_activated(vcpu->kvm))
> return RET_PF_EMULATE;
> }
>
> ....
> > diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> index d7374d768296..6ab9088c2531 100644
> --- a/arch/x86/kvm/x86.c
> +++ b/arch/x86/kvm/x86.c
> @@ -9379,15 +9379,29 @@ static void kvm_pv_kick_cpu_op(struct kvm *kvm, int apicid)
> kvm_irq_delivery_to_apic(kvm, NULL, &lapic_irq, NULL);
> }
>
> -bool kvm_apicv_activated(struct kvm *kvm)
> +bool kvm_apicv_memslot_activated(struct kvm *kvm)
> {
> return (READ_ONCE(kvm->arch.apicv_inhibit_reasons) == 0);
> }
> +
> +static unsigned long kvm_apicv_get_inhibit_reasons(struct kvm *kvm)
> +{
> + /*
> + * x2APIC only needs to "inhibit" the MMIO region, all other aspects of
> + * APICv can continue to be utilized.
> + */
> + return READ_ONCE(kvm->arch.apicv_inhibit_reasons) & ~APICV_INHIBIT_REASON_X2APIC;
> +}
> +
> +bool kvm_apicv_activated(struct kvm *kvm)
> +{
> + return !kvm_apicv_get_inhibit_reasons(kvm);
> +}
> EXPORT_SYMBOL_GPL(kvm_apicv_activated);
>
> bool kvm_vcpu_apicv_activated(struct kvm_vcpu *vcpu)
> {
> - ulong vm_reasons = READ_ONCE(vcpu->kvm->arch.apicv_inhibit_reasons);
> + ulong vm_reasons = kvm_apicv_get_inhibit_reasons(vcpu->kvm);
> ulong vcpu_reasons = static_call(kvm_x86_vcpu_get_apicv_inhibit_reasons)(vcpu);
>
> return (vm_reasons | vcpu_reasons) == 0;
> @@ -10122,7 +10136,15 @@ void __kvm_set_or_clear_apicv_inhibit(struct kvm *kvm,
>
> set_or_clear_apicv_inhibit(&new, reason, set);
>
> - if (!!old != !!new) {
> + /*
> + * If the overall "is APICv activated" status is unchanged, simply add
> + * or remove the inihbit from the pile. x2APIC is an exception, as it
> + * is a partial inhibit (only blocks SPTEs for the APIC access page).
> + * If x2APIC is the only inhibit in either the old or the new set, then
> + * vCPUs need to be kicked to transition between partially-inhibited
> + * and fully-inhibited.
> + */
> + if ((!!old != !!new) || old == X2APIC_ENABLE || new == X2APIC_ENABLE) {
Why are we comparing APICV inhibit reasons (old, new) with X2APIC_ENABLE
here? Do you mean to compare with APICV_INHIBIT_REASON_X2APIC?
Thanks,
Suravee
> /*
> * Kick all vCPUs before setting apicv_inhibit_reasons to avoid
> * false positives in the sanity check WARN in svm_vcpu_run().
> @@ -10137,7 +10159,12 @@ void __kvm_set_or_clear_apicv_inhibit(struct kvm *kvm,
> */
> kvm_make_all_cpus_request(kvm, KVM_REQ_APICV_UPDATE);
> kvm->arch.apicv_inhibit_reasons = new;
> - if (new) {
> +
> + /*
> + * Zap SPTEs for the APIC access page if APICv is newly
> + * inhibited (partially or fully).
> + */
> + if (new && !old) {
> unsigned long gfn = gpa_to_gfn(APIC_DEFAULT_PHYS_BASE);
> kvm_zap_gfn_range(kvm, gfn, gfn+1);
> }
next prev parent reply other threads:[~2022-09-13 19:52 UTC|newest]
Thread overview: 42+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-09-03 0:22 [PATCH v2 00/23] KVM: x86: AVIC and local APIC fixes+cleanups Sean Christopherson
2022-09-03 0:22 ` [PATCH v2 01/23] KVM: x86: Purge "highest ISR" cache when updating APICv state Sean Christopherson
2022-09-05 21:58 ` Paolo Bonzini
2022-09-03 0:22 ` [PATCH v2 02/23] KVM: SVM: Flush the "current" TLB when activating AVIC Sean Christopherson
2022-09-05 21:58 ` Paolo Bonzini
2022-09-03 0:22 ` [PATCH v2 03/23] KVM: SVM: Process ICR on AVIC IPI delivery failure due to invalid target Sean Christopherson
2022-09-05 21:59 ` Paolo Bonzini
2022-09-03 0:22 ` [PATCH v2 04/23] KVM: x86: Inhibit AVIC SPTEs if any vCPU enables x2APIC Sean Christopherson
2022-09-05 22:02 ` Paolo Bonzini
2022-09-13 19:52 ` Suthikulpanit, Suravee [this message]
2022-09-14 7:39 ` Sean Christopherson
2022-09-14 17:41 ` Suthikulpanit, Suravee
2022-09-16 17:47 ` Sean Christopherson
2022-09-16 19:10 ` Sean Christopherson
2022-09-20 13:07 ` Maxim Levitsky
2022-09-20 15:46 ` Sean Christopherson
2022-09-20 16:50 ` Sean Christopherson
2022-09-23 10:18 ` Maxim Levitsky
2022-09-23 10:19 ` Maxim Levitsky
2022-09-03 0:22 ` [PATCH v2 05/23] KVM: SVM: Don't put/load AVIC when setting virtual APIC mode Sean Christopherson
2022-09-03 0:22 ` [PATCH v2 06/23] KVM: SVM: Replace "avic_mode" enum with "x2avic_enabled" boolean Sean Christopherson
2022-09-03 0:22 ` [PATCH v2 07/23] KVM: SVM: Compute dest based on sender's x2APIC status for AVIC kick Sean Christopherson
2022-09-03 0:22 ` [PATCH v2 08/23] KVM: SVM: Fix x2APIC Logical ID calculation for avic_kick_target_vcpus_fast Sean Christopherson
2022-09-03 0:22 ` [PATCH v2 09/23] Revert "KVM: SVM: Use target APIC ID to complete x2AVIC IRQs when possible" Sean Christopherson
2022-09-03 0:22 ` [PATCH v2 10/23] KVM: SVM: Document that vCPU ID == APIC ID in AVIC kick fastpatch Sean Christopherson
2022-09-03 0:22 ` [PATCH v2 11/23] KVM: SVM: Add helper to perform final AVIC "kick" of single vCPU Sean Christopherson
2022-09-03 0:22 ` [PATCH v2 12/23] KVM: x86: Disable APIC logical map if logical ID covers multiple MDAs Sean Christopherson
2022-09-03 0:22 ` [PATCH v2 13/23] KVM: x86: Disable APIC logical map if vCPUs are aliased in logical mode Sean Christopherson
2022-09-13 23:32 ` Suthikulpanit, Suravee
2022-09-14 7:42 ` Sean Christopherson
2022-09-15 2:11 ` Suthikulpanit, Suravee
2022-09-16 18:52 ` Sean Christopherson
2022-09-03 0:22 ` [PATCH v2 14/23] KVM: x86: Honor architectural behavior for aliased 8-bit APIC IDs Sean Christopherson
2022-09-03 0:22 ` [PATCH v2 15/23] KVM: x86: Explicitly skip adding vCPU to optimized logical map if LDR==0 Sean Christopherson
2022-09-03 0:22 ` [PATCH v2 16/23] KVM: x86: Explicitly track all possibilities for APIC map's logical modes Sean Christopherson
2022-09-03 0:22 ` [PATCH v2 17/23] KVM: SVM: Inhibit AVIC if vCPUs are aliased in logical mode Sean Christopherson
2022-09-03 0:22 ` [PATCH v2 18/23] KVM: SVM: Always update local APIC on writes to logical dest register Sean Christopherson
2022-09-03 0:22 ` [PATCH v2 19/23] KVM: SVM: Update svm->ldr_reg cache even if LDR is "bad" Sean Christopherson
2022-09-03 0:22 ` [PATCH v2 20/23] KVM: SVM: Require logical ID to be power-of-2 for AVIC entry Sean Christopherson
2022-09-03 0:22 ` [PATCH v2 21/23] KVM: SVM: Handle multiple logical targets in AVIC kick fastpath Sean Christopherson
2022-09-03 0:22 ` [PATCH v2 22/23] KVM: SVM: Ignore writes to Remote Read Data on AVIC write traps Sean Christopherson
2022-09-03 0:22 ` [PATCH v2 23/23] Revert "KVM: SVM: Do not throw warning when calling avic_vcpu_load on a running vcpu" Sean Christopherson
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=b6fcb487-56fc-12ea-6f67-b14b0b156ee0@amd.com \
--to=suravee.suthikulpanit@amd.com \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=lirongqing@baidu.com \
--cc=mlevitsk@redhat.com \
--cc=pbonzini@redhat.com \
--cc=seanjc@google.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).