From: Maxim Levitsky <mlevitsk@redhat.com>
To: Sean Christopherson <seanjc@google.com>
Cc: Marc Zyngier <maz@kernel.org>,
Huacai Chen <chenhuacai@kernel.org>,
Aleksandar Markovic <aleksandar.qemu.devel@gmail.com>,
Paul Mackerras <paulus@ozlabs.org>,
Anup Patel <anup.patel@wdc.com>,
Paul Walmsley <paul.walmsley@sifive.com>,
Palmer Dabbelt <palmer@dabbelt.com>,
Albert Ou <aou@eecs.berkeley.edu>,
Christian Borntraeger <borntraeger@de.ibm.com>,
Janosch Frank <frankja@linux.ibm.com>,
Paolo Bonzini <pbonzini@redhat.com>,
James Morse <james.morse@arm.com>,
Alexandru Elisei <alexandru.elisei@arm.com>,
Suzuki K Poulose <suzuki.poulose@arm.com>,
Atish Patra <atish.patra@wdc.com>,
David Hildenbrand <david@redhat.com>,
Cornelia Huck <cohuck@redhat.com>,
Claudio Imbrenda <imbrenda@linux.ibm.com>,
Vitaly Kuznetsov <vkuznets@redhat.com>,
Wanpeng Li <wanpengli@tencent.com>,
Jim Mattson <jmattson@google.com>, Joerg Roedel <joro@8bytes.org>,
linux-arm-kernel@lists.infradead.org,
kvmarm@lists.cs.columbia.edu, linux-mips@vger.kernel.org,
kvm@vger.kernel.org, kvm-ppc@vger.kernel.org,
kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org,
linux-kernel@vger.kernel.org, David Matlack <dmatlack@google.com>,
Oliver Upton <oupton@google.com>,
Jing Zhang <jingzhangos@google.com>
Subject: Re: [PATCH v2 28/43] KVM: VMX: Remove vCPU from PI wakeup list before updating PID.NV
Date: Mon, 01 Nov 2021 00:52:55 +0200 [thread overview]
Message-ID: <b03c61ae0fa7e9531db8115569ef1c201c1aecef.camel@redhat.com> (raw)
In-Reply-To: <YXrbpvHG7YD0MNO2@google.com>
On Thu, 2021-10-28 at 17:19 +0000, Sean Christopherson wrote:
> On Thu, Oct 28, 2021, Maxim Levitsky wrote:
> > On Fri, 2021-10-08 at 19:12 -0700, Sean Christopherson wrote:
> > > Remove the vCPU from the wakeup list before updating the notification
> > > vector in the posted interrupt post-block helper. There is no need to
> > > wake the current vCPU as it is by definition not blocking. Practically
> > > speaking this is a nop as it only shaves a few meager cycles in the
> > > unlikely case that the vCPU was migrated and the previous pCPU gets a
> > > wakeup IRQ right before PID.NV is updated. The real motivation is to
> > > allow for more readable code in the future, when post-block is merged
> > > with vmx_vcpu_pi_load(), at which point removal from the list will be
> > > conditional on the old notification vector.
> > >
> > > Opportunistically add comments to document why KVM has a per-CPU spinlock
> > > that, at first glance, appears to be taken only on the owning CPU.
> > > Explicitly call out that the spinlock must be taken with IRQs disabled, a
> > > detail that was "lost" when KVM switched from spin_lock_irqsave() to
> > > spin_lock(), with IRQs disabled for the entirety of the relevant path.
> > >
> > > Signed-off-by: Sean Christopherson <seanjc@google.com>
> > > ---
> > > arch/x86/kvm/vmx/posted_intr.c | 49 +++++++++++++++++++++++-----------
> > > 1 file changed, 33 insertions(+), 16 deletions(-)
> > >
> > > diff --git a/arch/x86/kvm/vmx/posted_intr.c b/arch/x86/kvm/vmx/posted_intr.c
> > > index 2b2206339174..901b7a5f7777 100644
> > > --- a/arch/x86/kvm/vmx/posted_intr.c
> > > +++ b/arch/x86/kvm/vmx/posted_intr.c
> > > @@ -10,10 +10,22 @@
> > > #include "vmx.h"
> > >
> > > /*
> > > - * We maintain a per-CPU linked-list of vCPU, so in wakeup_handler() we
> > > - * can find which vCPU should be waken up.
> > > + * Maintain a per-CPU list of vCPUs that need to be awakened by wakeup_handler()
> > Nit: While at it, it would be nice to rename this to pi_wakeup_hanlder() so
> > that it can be more easilly found.
>
> Ah, good catch.
>
> > > + * when a WAKEUP_VECTOR interrupted is posted. vCPUs are added to the list when
> > > + * the vCPU is scheduled out and is blocking (e.g. in HLT) with IRQs enabled.
> > s/interrupted/interrupt ?
> >
> > Isn't that comment incorrect? As I see, the PI hardware is setup to use the WAKEUP_VECTOR
> > when vcpu blocks (in pi_pre_block) and then that vcpu is added to the list.
> > The pi_wakeup_hanlder just goes over the list and wakes up all vcpus on the lsit.
>
> Doh, yes. This patch is predicting the future. The comment becomes correct as of
>
> KVM: VMX: Handle PI wakeup shenanigans during vcpu_put/load
>
> but as of this patch the "scheduled out" piece doesn't hold true.
>
> > > + * The vCPUs posted interrupt descriptor is updated at the same time to set its
> > > + * notification vector to WAKEUP_VECTOR, so that posted interrupt from devices
> > > + * wake the target vCPUs. vCPUs are removed from the list and the notification
> > > + * vector is reset when the vCPU is scheduled in.
> > > */
> > > static DEFINE_PER_CPU(struct list_head, blocked_vcpu_on_cpu);
> > Also while at it, why not to rename this to 'blocked_vcpu_list'?
> > to explain that this is list of blocked vcpus. Its a per-cpu variable
> > so 'on_cpu' suffix isn't needed IMHO.
>
> As you noted, addressed in a future patch.
>
> > > +/*
> > > + * Protect the per-CPU list with a per-CPU spinlock to handle task migration.
> > > + * When a blocking vCPU is awakened _and_ migrated to a different pCPU, the
> > > + * ->sched_in() path will need to take the vCPU off the list of the _previous_
> > > + * CPU. IRQs must be disabled when taking this lock, otherwise deadlock will
> > > + * occur if a wakeup IRQ arrives and attempts to acquire the lock.
> > > + */
> > > static DEFINE_PER_CPU(spinlock_t, blocked_vcpu_on_cpu_lock);
> > >
> > > static inline struct pi_desc *vcpu_to_pi_desc(struct kvm_vcpu *vcpu)
> > > @@ -101,23 +113,28 @@ static void __pi_post_block(struct kvm_vcpu *vcpu)
> > > WARN(pi_desc->nv != POSTED_INTR_WAKEUP_VECTOR,
> > > "Wakeup handler not enabled while the vCPU was blocking");
> > >
> > > - dest = cpu_physical_id(vcpu->cpu);
> > > - if (!x2apic_mode)
> > > - dest = (dest << 8) & 0xFF00;
> > > -
> > > - do {
> > > - old.control = new.control = READ_ONCE(pi_desc->control);
> > > -
> > > - new.ndst = dest;
> > > -
> > > - /* set 'NV' to 'notification vector' */
> > > - new.nv = POSTED_INTR_VECTOR;
> > > - } while (cmpxchg64(&pi_desc->control, old.control,
> > > - new.control) != old.control);
> > > -
> > > + /*
> > > + * Remove the vCPU from the wakeup list of the _previous_ pCPU, which
> > > + * will not be the same as the current pCPU if the task was migrated.
> > > + */
> > > spin_lock(&per_cpu(blocked_vcpu_on_cpu_lock, vcpu->pre_pcpu));
> > > list_del(&vcpu->blocked_vcpu_list);
> > > spin_unlock(&per_cpu(blocked_vcpu_on_cpu_lock, vcpu->pre_pcpu));
> > > +
> > > + dest = cpu_physical_id(vcpu->cpu);
> > > + if (!x2apic_mode)
> > > + dest = (dest << 8) & 0xFF00;
> > It would be nice to have a function for this, this appears in this file twice.
> > Maybe there is a function already somewhere?
>
> The second instance does go away by the aforementioned:
Then no need for a helper.
>
> KVM: VMX: Handle PI wakeup shenanigans during vcpu_put/load
>
> I'm inclined to say we don't want a helper because there should only ever be one
> path that changes PI.ndst. But a comment would definitely help to explain the
> difference between xAPIC and x2APIC IDs.
>
Makes sense!
Best regards,
Maxim Levitsky
next prev parent reply other threads:[~2021-10-31 22:53 UTC|newest]
Thread overview: 141+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-10-09 2:11 [PATCH v2 00/43] KVM: Halt-polling and x86 APICv overhaul Sean Christopherson
2021-10-09 2:11 ` [PATCH v2 01/43] KVM: VMX: Don't unblock vCPU w/ Posted IRQ if IRQs are disabled in guest Sean Christopherson
2021-10-09 2:11 ` [PATCH v2 02/43] KVM: SVM: Ensure target pCPU is read once when signalling AVIC doorbell Sean Christopherson
2021-10-25 13:21 ` Paolo Bonzini
2021-10-27 9:50 ` Maxim Levitsky
2021-10-09 2:11 ` [PATCH v2 03/43] KVM: s390: Ensure kvm_arch_no_poll() is read once when blocking vCPU Sean Christopherson
2021-10-09 2:11 ` [PATCH v2 04/43] KVM: Force PPC to define its own rcuwait object Sean Christopherson
2021-10-09 2:11 ` [PATCH v2 05/43] KVM: Update halt-polling stats if and only if halt-polling was attempted Sean Christopherson
2021-10-27 10:18 ` Maxim Levitsky
2021-10-09 2:11 ` [PATCH v2 06/43] KVM: Refactor and document halt-polling stats update helper Sean Christopherson
2021-10-27 10:56 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 07/43] KVM: Reconcile discrepancies in halt-polling stats Sean Christopherson
2021-10-27 11:29 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 08/43] KVM: s390: Clear valid_wakeup in kvm_s390_handle_wait(), not in arch hook Sean Christopherson
2021-10-09 2:12 ` [PATCH v2 09/43] KVM: Drop obsolete kvm_arch_vcpu_block_finish() Sean Christopherson
2021-10-22 15:02 ` Anup Patel
2021-10-09 2:12 ` [PATCH v2 10/43] KVM: arm64: Move vGIC v4 handling for WFI out arch callback hook Sean Christopherson
2021-10-25 13:31 ` Paolo Bonzini
2021-10-26 15:41 ` Marc Zyngier
2021-10-26 16:12 ` Paolo Bonzini
2021-11-30 11:39 ` Paolo Bonzini
2021-11-30 12:04 ` Marc Zyngier
2021-11-30 16:07 ` Paolo Bonzini
2021-10-09 2:12 ` [PATCH v2 11/43] KVM: Don't block+unblock when halt-polling is successful Sean Christopherson
2021-10-27 13:40 ` Maxim Levitsky
2021-11-28 22:16 ` Maxim Levitsky
2021-11-29 17:25 ` Sean Christopherson
2021-11-29 17:53 ` Paolo Bonzini
2021-11-29 18:55 ` Sean Christopherson
2021-11-29 19:18 ` Paolo Bonzini
2021-11-29 22:53 ` Maxim Levitsky
2021-12-02 0:20 ` Maxim Levitsky
2021-12-02 2:00 ` Sean Christopherson
2021-12-02 10:31 ` Paolo Bonzini
2021-11-29 17:55 ` Paolo Bonzini
2021-11-29 22:55 ` Maxim Levitsky
2021-12-02 10:20 ` Maxim Levitsky
2021-12-02 10:47 ` Maxim Levitsky
2021-12-02 12:02 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 12/43] KVM: x86: Tweak halt emulation helper names to free up kvm_vcpu_halt() Sean Christopherson
2021-10-27 14:10 ` Maxim Levitsky
2021-10-27 14:18 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 13/43] KVM: Rename kvm_vcpu_block() => kvm_vcpu_halt() Sean Christopherson
2021-10-22 15:04 ` Anup Patel
2021-10-09 2:12 ` [PATCH v2 14/43] KVM: Split out a kvm_vcpu_block() helper from kvm_vcpu_halt() Sean Christopherson
2021-10-09 2:12 ` [PATCH v2 15/43] KVM: stats: Add stat to detect if vcpu is currently blocking Sean Christopherson
2021-10-09 2:12 ` [PATCH v2 16/43] KVM: Don't redo ktime_get() when calculating halt-polling stop/deadline Sean Christopherson
2021-10-25 14:26 ` Paolo Bonzini
2021-10-27 14:35 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 17/43] KVM: x86: Directly block (instead of "halting") UNINITIALIZED vCPUs Sean Christopherson
2021-10-27 14:46 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 18/43] KVM: x86: Invoke kvm_vcpu_block() directly for non-HALTED wait states Sean Christopherson
2021-10-27 15:18 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 19/43] KVM: Add helpers to wake/query blocking vCPU Sean Christopherson
2021-10-25 14:06 ` Paolo Bonzini
2021-10-27 19:27 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 20/43] KVM: VMX: Skip Posted Interrupt updates if APICv is hard disabled Sean Christopherson
2021-10-25 13:48 ` Paolo Bonzini
2021-10-28 9:12 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 21/43] KVM: VMX: Clean up PI pre/post-block WARNs Sean Christopherson
2021-10-28 10:20 ` Maxim Levitsky
2021-10-28 15:34 ` Sean Christopherson
2021-10-09 2:12 ` [PATCH v2 22/43] KVM: VMX: Drop unnecessary PI logic to handle impossible conditions Sean Christopherson
2021-10-25 13:53 ` Paolo Bonzini
2021-10-28 14:36 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 23/43] KVM: VMX: Use boolean returns for Posted Interrupt "test" helpers Sean Christopherson
2021-10-28 6:05 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 24/43] KVM: VMX: Drop pointless PI.NDST update when blocking Sean Christopherson
2021-10-25 14:01 ` Paolo Bonzini
2021-10-27 14:26 ` Sean Christopherson
2021-10-28 10:53 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 25/43] KVM: VMX: Save/restore IRQs (instead of CLI/STI) during PI pre/post block Sean Christopherson
2021-10-28 10:54 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 26/43] KVM: VMX: Read Posted Interrupt "control" exactly once per loop iteration Sean Christopherson
2021-10-28 10:58 ` Maxim Levitsky
2021-10-28 15:55 ` Sean Christopherson
2021-10-31 22:48 ` Maxim Levitsky
2021-11-01 17:41 ` Sean Christopherson
2021-10-09 2:12 ` [PATCH v2 27/43] KVM: VMX: Move Posted Interrupt ndst computation out of write loop Sean Christopherson
2021-10-28 11:28 ` Maxim Levitsky
2021-10-28 16:09 ` Maxim Levitsky
2021-10-28 16:12 ` Sean Christopherson
2021-10-31 22:51 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 28/43] KVM: VMX: Remove vCPU from PI wakeup list before updating PID.NV Sean Christopherson
2021-10-28 12:53 ` Maxim Levitsky
2021-10-28 17:19 ` Sean Christopherson
2021-10-31 22:52 ` Maxim Levitsky [this message]
2021-10-09 2:12 ` [PATCH v2 29/43] KVM: VMX: Handle PI wakeup shenanigans during vcpu_put/load Sean Christopherson
2021-10-28 15:14 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 30/43] KVM: Drop unused kvm_vcpu.pre_pcpu field Sean Christopherson
2021-10-28 15:16 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 31/43] KVM: Move x86 VMX's posted interrupt list_head to vcpu_vmx Sean Christopherson
2021-10-28 15:33 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 32/43] KVM: VMX: Move preemption timer <=> hrtimer dance to common x86 Sean Christopherson
2021-10-28 15:45 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 33/43] KVM: x86: Unexport LAPIC's switch_to_{hv,sw}_timer() helpers Sean Christopherson
2021-10-28 15:45 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 34/43] KVM: x86: Remove defunct pre_block/post_block kvm_x86_ops hooks Sean Christopherson
2021-10-28 15:46 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 35/43] KVM: SVM: Signal AVIC doorbell iff vCPU is in guest mode Sean Christopherson
2021-10-25 14:26 ` Paolo Bonzini
2021-10-27 15:06 ` Sean Christopherson
2021-10-27 15:36 ` Paolo Bonzini
2021-10-27 16:08 ` Sean Christopherson
2021-10-27 16:14 ` Paolo Bonzini
2021-10-28 16:12 ` Maxim Levitsky
2021-10-28 17:06 ` Sean Christopherson
2021-10-09 2:12 ` [PATCH v2 36/43] KVM: SVM: Don't bother checking for "running" AVIC when kicking for IPIs Sean Christopherson
2021-10-31 14:50 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 37/43] KVM: SVM: Unconditionally mark AVIC as running on vCPU load (with APICv) Sean Christopherson
2021-10-25 14:22 ` Paolo Bonzini
2021-10-25 15:48 ` Sean Christopherson
2021-10-25 15:57 ` Paolo Bonzini
2021-10-25 16:00 ` Sean Christopherson
2021-10-31 16:34 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 38/43] KVM: Drop defunct kvm_arch_vcpu_(un)blocking() hooks Sean Christopherson
2021-10-31 16:35 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 39/43] KVM: VMX: Don't do full kick when triggering posted interrupt "fails" Sean Christopherson
2021-10-25 14:34 ` Paolo Bonzini
2021-10-27 16:04 ` Sean Christopherson
2021-10-27 22:09 ` Paolo Bonzini
2021-10-31 22:15 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 40/43] KVM: VMX: Wake vCPU when delivering posted IRQ even if vCPU == this vCPU Sean Christopherson
2021-10-25 15:05 ` Paolo Bonzini
2021-10-27 15:30 ` Sean Christopherson
2021-10-31 22:19 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 41/43] KVM: VMX: Pass desired vector instead of bool for triggering posted IRQ Sean Christopherson
2021-10-31 22:25 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 42/43] KVM: VMX: Fold fallback path into triggering posted IRQ helper Sean Christopherson
2021-10-31 22:30 ` Maxim Levitsky
2021-10-09 2:12 ` [PATCH v2 43/43] KVM: VMX: Don't do full kick when handling posted interrupt wakeup Sean Christopherson
2021-10-25 14:16 ` Paolo Bonzini
2021-10-31 22:33 ` Maxim Levitsky
2021-10-25 14:13 ` [PATCH v2 00/43] KVM: Halt-polling and x86 APICv overhaul Paolo Bonzini
2021-10-27 14:41 ` Sean Christopherson
2021-10-27 14:57 ` Paolo Bonzini
2021-10-27 15:28 ` Sean Christopherson
2021-10-27 15:37 ` Paolo Bonzini
2021-10-26 7:20 ` Christian Borntraeger
2021-10-26 14:48 ` Sean Christopherson
2021-10-26 18:29 ` Christian Borntraeger
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=b03c61ae0fa7e9531db8115569ef1c201c1aecef.camel@redhat.com \
--to=mlevitsk@redhat.com \
--cc=aleksandar.qemu.devel@gmail.com \
--cc=alexandru.elisei@arm.com \
--cc=anup.patel@wdc.com \
--cc=aou@eecs.berkeley.edu \
--cc=atish.patra@wdc.com \
--cc=borntraeger@de.ibm.com \
--cc=chenhuacai@kernel.org \
--cc=cohuck@redhat.com \
--cc=david@redhat.com \
--cc=dmatlack@google.com \
--cc=frankja@linux.ibm.com \
--cc=imbrenda@linux.ibm.com \
--cc=james.morse@arm.com \
--cc=jingzhangos@google.com \
--cc=jmattson@google.com \
--cc=joro@8bytes.org \
--cc=kvm-ppc@vger.kernel.org \
--cc=kvm-riscv@lists.infradead.org \
--cc=kvm@vger.kernel.org \
--cc=kvmarm@lists.cs.columbia.edu \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mips@vger.kernel.org \
--cc=linux-riscv@lists.infradead.org \
--cc=maz@kernel.org \
--cc=oupton@google.com \
--cc=palmer@dabbelt.com \
--cc=paul.walmsley@sifive.com \
--cc=paulus@ozlabs.org \
--cc=pbonzini@redhat.com \
--cc=seanjc@google.com \
--cc=suzuki.poulose@arm.com \
--cc=vkuznets@redhat.com \
--cc=wanpengli@tencent.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).