From: Marcelo Tosatti <mtosatti@redhat.com>
To: "Wu, Feng" <feng.wu@intel.com>, hpa@zytor.com
Cc: "tglx@linutronix.de" <tglx@linutronix.de>,
"mingo@redhat.com" <mingo@redhat.com>,
"hpa@zytor.com" <hpa@zytor.com>,
"x86@kernel.org" <x86@kernel.org>,
"gleb@kernel.org" <gleb@kernel.org>,
"pbonzini@redhat.com" <pbonzini@redhat.com>,
"dwmw2@infradead.org" <dwmw2@infradead.org>,
"joro@8bytes.org" <joro@8bytes.org>,
"alex.williamson@redhat.com" <alex.williamson@redhat.com>,
"jiang.liu@linux.intel.com" <jiang.liu@linux.intel.com>,
"eric.auger@linaro.org" <eric.auger@linaro.org>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
"iommu@lists.linux-foundation.org"
<iommu@lists.linux-foundation.org>,
"kvm@vger.kernel.org" <kvm@vger.kernel.org>
Subject: Re: [v3 24/26] KVM: Update Posted-Interrupts Descriptor when vCPU is blocked
Date: Wed, 25 Mar 2015 20:17:38 -0300 [thread overview]
Message-ID: <20150325231738.GA7333@amt.cnet> (raw)
In-Reply-To: <E959C4978C3B6342920538CF579893F00243A819@SHSMSX104.ccr.corp.intel.com>
On Mon, Mar 16, 2015 at 11:42:06AM +0000, Wu, Feng wrote:
> > Do you have any reason why having the code at vcpu_put/vcpu_load is
> > better than the proposal to have the code at kvm_vcpu_block?
>
> I think your proposal is good, I just want to better understand your idea here.:)
Reduce the overhead of vcpu sched in / vcpu sched out, basically.
> One thing, even we put the code to kvm_vcpu_block, we still need to add code
> at vcpu_put/vcpu_load for the preemption case like what I did now.
>
> >
> > >
> > > Global vector is a limited resources in the system, and this involves
> > > common x86 interrupt code changes. I am not sure we can allocate
> > > so many dedicated global vector for KVM usage.
> >
> > Why not? Have KVM use all free vectors (so if vectors are necessary for
> > other purposes, people should shrink the KVM vector pool).
>
> If we want to allocate more global vector for this usage, we need hpa's
> input about it. Peter, what is your opinion?
Peter?
> > BTW the Intel docs talk about that ("one vector per vCPU").
> Yes, the Spec talks about this, but it is more complex using one vector per vCPU.
>
> >
> > > > > > It seems there is a bunch free:
> > > > > >
> > > > > > commit 52aec3308db85f4e9f5c8b9f5dc4fbd0138c6fa4
> > > > > > Author: Alex Shi <alex.shi@intel.com>
> > > > > > Date: Thu Jun 28 09:02:23 2012 +0800
> > > > > >
> > > > > > x86/tlb: replace INVALIDATE_TLB_VECTOR by
> > > > CALL_FUNCTION_VECTOR
> > > > > >
> > > > > > Can you add only vcpus which have posted IRTEs that point to this pCPU
> > > > > > to the HLT'ed vcpu lists? (so for example, vcpus without assigned
> > > > > > devices are not part of the list).
> > > > >
> > > > > Is it easy to find whether a vCPU (or the associated domain) has assigned
> > > > devices?
> > > > > If so, we can only add those vCPUs with assigned devices.
> > > >
> > > > When configuring IRTE, at kvm_arch_vfio_update_pi_irte?
> > >
> > > Yes.
> > >
> > > >
> > > > > > > +
> > > > > > > static int __init vmx_init(void)
> > > > > > > {
> > > > > > > int r, i, msr;
> > > > > > > @@ -9429,6 +9523,8 @@ static int __init vmx_init(void)
> > > > > > >
> > > > > > > update_ple_window_actual_max();
> > > > > > >
> > > > > > > + wakeup_handler_callback = wakeup_handler;
> > > > > > > +
> > > > > > > return 0;
> > > > > > >
> > > > > > > out7:
> > > > > > > diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
> > > > > > > index 0033df3..1551a46 100644
> > > > > > > --- a/arch/x86/kvm/x86.c
> > > > > > > +++ b/arch/x86/kvm/x86.c
> > > > > > > @@ -6152,6 +6152,21 @@ static int vcpu_enter_guest(struct
> > kvm_vcpu
> > > > > > *vcpu)
> > > > > > > kvm_vcpu_reload_apic_access_page(vcpu);
> > > > > > > }
> > > > > > >
> > > > > > > + /*
> > > > > > > + * Since posted-interrupts can be set by VT-d HW now, in this
> > > > > > > + * case, KVM_REQ_EVENT is not set. We move the following
> > > > > > > + * operations out of the if statement.
> > > > > > > + */
> > > > > > > + if (kvm_lapic_enabled(vcpu)) {
> > > > > > > + /*
> > > > > > > + * Update architecture specific hints for APIC
> > > > > > > + * virtual interrupt delivery.
> > > > > > > + */
> > > > > > > + if (kvm_x86_ops->hwapic_irr_update)
> > > > > > > + kvm_x86_ops->hwapic_irr_update(vcpu,
> > > > > > > + kvm_lapic_find_highest_irr(vcpu));
> > > > > > > + }
> > > > > > > +
> > > > > >
> > > > > > This is a hot fast path. You can set KVM_REQ_EVENT from
> > wakeup_handler.
> > > > >
> > > > > I am afraid Setting KVM_REQ_EVENT from wakeup_handler doesn't help
> > > > much,
> > > > > if vCPU is running in ROOT mode, and VT-d hardware issues an notification
> > > > event,
> > > > > POSTED_INTR_VECTOR interrupt handler will be called.
> > > >
> > > > If vCPU is in root mode, remapping HW will find IRTE configured with
> > > > vector == POSTED_INTR_WAKEUP_VECTOR, use that vector, which will
> > > > VM-exit, and execute the interrupt handler wakeup_handler. Right?
> > >
> > > There are two cases:
> > > Case 1: vCPU is blocked, so it is in root mode, this is what you described
> > above.
> > > Case 2, vCPU is running in root mode, such as, handling vm-exits, in this case,
> > > the notification vector is 'POSTED_INTR_VECTOR', and if external interrupts
> > > from assigned devices happen, the handled of 'POSTED_INTR_VECTOR' will
> > > be called ( it is 'smp_kvm_posted_intr_ipi' in fact), this routine doesn't need
> > > do real things, since the pending interrupts in PIR will be synced to vIRR
> > before
> > > VM-Entry (this code have already been there when enabling CPU-side
> > > posted-interrupt along with APICv). Like what I said before, it is a little hard
> > to
> > > get vCPU related information in it, even if we get, it is not accurate and may
> > harm
> > > the performance.(need search)
> > >
> > > So only setting KVM_REQ_EVENT in wakeup_handler cannot cover the
> > notification
> > > event for 'POSTED_INTR_VECTOR'.
> > >
> > > >
> > > > The point of this comment is that you can keep the
> > > >
> > > > "if (kvm_x86_ops->hwapic_irr_update)
> > > > kvm_x86_ops->hwapic_irr_update(vcpu,
> > > > kvm_lapic_find_highest_irr(vcpu));
> > > > "
> > > >
> > > > Code inside KVM_REQ_EVENT handling section of vcpu_run, as long as
> > > > wakeup_handler sets KVM_REQ_EVENT.
> > >
> > > Please see above.
> >
> > OK can you set KVM_REQ_EVENT in case the ON bit is set,
> > after disabling interrupts ?
> >
> Currently, the following code is executed before local_irq_disable() is called,
> so do you mean 1)moving local_irq_disable() to the place before it. 2) after interrupt
> is disabled, set KVM_REQ_EVENT in case the ON bit is set?
2) after interrupt is disabled, set KVM_REQ_EVENT in case the ON bit
is set.
>
> "if (kvm_x86_ops->hwapic_irr_update)
> kvm_x86_ops->hwapic_irr_update(vcpu,
> kvm_lapic_find_highest_irr(vcpu));
>
> > kvm_lapic_find_highest_irr(vcpu) eats some cache
> > (4 cachelines) versus 1 cacheline for reading ON bit.
> >
> > > > > > Please remove blocked and wakeup_cpu, they should not be necessary.
> > > > >
> > > > > Why do you think wakeup_cpu is not needed, when vCPU is blocked,
> > > > > wakeup_cpu saves the cpu which the vCPU is blocked on, after vCPU
> > > > > is woken up, it can run on a different cpu, so we need wakeup_cpu to
> > > > > find the right list to wake up the vCPU.
> > > >
> > > > If the vCPU was moved it should have updated IRTE destination field
> > > > to the pCPU which it has moved to?
> > >
> > > Every time a vCPU is scheduled to a new pCPU, the IRTE destination filed
> > > would be updated accordingly.
> > >
> > > When vCPU is blocked. To wake up the blocked vCPU, we need to find which
> > > list the vCPU is blocked on, and this is what wakeup_cpu used for?
> >
> > Right, perhaps prev_vcpu is a better name.
>
> Do you mean "prev_pcpu"?
Yes.
next prev parent reply other threads:[~2015-03-25 23:17 UTC|newest]
Thread overview: 140+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-12-12 15:14 [v3 00/26] Add VT-d Posted-Interrupts support Feng Wu
2014-12-12 15:14 ` [v3 01/26] genirq: Introduce irq_set_vcpu_affinity() to target an interrupt to a VCPU Feng Wu
2014-12-12 15:14 ` [v3 02/26] iommu: Add new member capability to struct irq_remap_ops Feng Wu
2015-01-28 15:22 ` David Woodhouse
2015-01-29 8:34 ` Wu, Feng
2014-12-12 15:14 ` [v3 03/26] iommu, x86: Define new irte structure for VT-d Posted-Interrupts Feng Wu
2015-01-28 15:26 ` David Woodhouse
2014-12-12 15:14 ` [v3 04/26] iommu, x86: Implement irq_set_vcpu_affinity for intel_ir_chip Feng Wu
2015-01-28 15:26 ` David Woodhouse
2015-01-29 7:55 ` Wu, Feng
2014-12-12 15:14 ` [v3 05/26] x86, irq: Implement irq_set_vcpu_affinity for pci_msi_ir_controller Feng Wu
2014-12-12 15:14 ` [v3 06/26] iommu, x86: No need to migrating irq for VT-d Posted-Interrupts Feng Wu
2014-12-18 14:26 ` Zhang, Yang Z
2014-12-19 1:40 ` Wu, Feng
2014-12-19 1:46 ` Zhang, Yang Z
2014-12-19 11:59 ` Paolo Bonzini
2014-12-23 0:37 ` Zhang, Yang Z
2014-12-23 8:47 ` Paolo Bonzini
2014-12-23 9:07 ` Wu, Feng
2014-12-23 9:34 ` Paolo Bonzini
2014-12-24 1:38 ` Zhang, Yang Z
2014-12-24 2:12 ` Jiang Liu
2014-12-24 2:32 ` Zhang, Yang Z
2014-12-24 3:08 ` Wu, Feng
2014-12-24 4:04 ` Zhang, Yang Z
2014-12-24 4:54 ` Jiang Liu
2015-01-28 15:29 ` David Woodhouse
2014-12-12 15:14 ` [v3 07/26] iommu, x86: Add cap_pi_support() to detect VT-d PI capability Feng Wu
2015-01-28 15:32 ` David Woodhouse
2014-12-12 15:14 ` [v3 08/26] iommu, x86: Add intel_irq_remapping_capability() for Intel Feng Wu
2015-01-28 15:37 ` David Woodhouse
2015-01-29 8:57 ` Wu, Feng
2014-12-12 15:14 ` [v3 09/26] iommu, x86: define irq_remapping_cap() Feng Wu
2014-12-12 15:14 ` [v3 10/26] KVM: change struct pi_desc for VT-d Posted-Interrupts Feng Wu
2014-12-12 15:14 ` [v3 11/26] KVM: Add some helper functions for Posted-Interrupts Feng Wu
2014-12-12 15:14 ` [v3 12/26] KVM: Initialize VT-d Posted-Interrupts Descriptor Feng Wu
2014-12-18 15:19 ` Zhang, Yang Z
2014-12-12 15:14 ` [v3 13/26] KVM: Define a new interface kvm_find_dest_vcpu() for VT-d PI Feng Wu
2014-12-18 14:49 ` Zhang, Yang Z
2014-12-18 16:58 ` Paolo Bonzini
2014-12-19 1:13 ` Zhang, Yang Z
2014-12-19 1:30 ` Wu, Feng
2014-12-19 1:30 ` Wu, Feng
2014-12-19 1:47 ` Zhang, Yang Z
2014-12-19 11:59 ` Paolo Bonzini
2014-12-19 23:48 ` Wu, Feng
2014-12-20 13:16 ` Paolo Bonzini
2014-12-22 4:48 ` Wu, Feng
2014-12-22 9:27 ` Paolo Bonzini
2014-12-22 11:04 ` Wu, Feng
2014-12-22 11:06 ` Paolo Bonzini
2014-12-22 11:17 ` Wu, Feng
2014-12-22 11:23 ` Paolo Bonzini
2014-12-22 14:13 ` Yong Wang
2015-01-09 14:54 ` Radim Krčmář
2015-01-09 14:56 ` Paolo Bonzini
2015-01-09 15:12 ` Radim Krčmář
2015-01-09 15:18 ` Paolo Bonzini
2015-01-09 15:47 ` Radim Krčmář
2015-01-13 0:27 ` Wu, Feng
2015-01-13 16:17 ` Radim Kr?má?
2015-01-14 1:27 ` Wu, Feng
2015-01-14 13:02 ` Paolo Bonzini
2015-01-14 16:59 ` Radim Kr?má?
2015-01-20 21:04 ` Nadav Amit
2015-01-21 21:16 ` Radim Kr?má?
2014-12-12 15:14 ` [v3 14/26] KVM: Get Posted-Interrupts descriptor address from struct kvm_vcpu Feng Wu
2014-12-12 15:14 ` [v3 15/26] KVM: add interfaces to control PI outside vmx Feng Wu
2014-12-12 15:14 ` [v3 16/26] KVM: Make struct kvm_irq_routing_table accessible Feng Wu
2014-12-17 16:17 ` Paolo Bonzini
2014-12-19 2:19 ` Wu, Feng
2014-12-19 11:59 ` Paolo Bonzini
2014-12-19 23:39 ` Wu, Feng
2014-12-12 15:14 ` [v3 17/26] KVM: make kvm_set_msi_irq() public Feng Wu
2014-12-17 17:32 ` Paolo Bonzini
2014-12-12 15:14 ` [v3 18/26] KVM: kvm-vfio: User API for VT-d Posted-Interrupts Feng Wu
2014-12-12 15:14 ` [v3 19/26] KVM: kvm-vfio: implement the VFIO skeleton " Feng Wu
2014-12-12 15:14 ` [v3 20/26] KVM: x86: kvm-vfio: VT-d posted-interrupts setup Feng Wu
2014-12-12 15:14 ` [v3 21/26] x86, irq: Define a global vector for VT-d Posted-Interrupts Feng Wu
2014-12-18 14:54 ` Zhang, Yang Z
2014-12-19 0:52 ` Wu, Feng
2015-01-30 18:18 ` H. Peter Anvin
2015-02-02 1:06 ` Wu, Feng
2015-02-23 22:04 ` Marcelo Tosatti
2014-12-12 15:14 ` [v3 22/26] KVM: Define a wakeup worker thread for vCPU Feng Wu
2014-12-12 15:14 ` [v3 23/26] KVM: Update Posted-Interrupts Descriptor when vCPU is preempted Feng Wu
2014-12-17 17:11 ` Paolo Bonzini
2014-12-18 3:15 ` Wu, Feng
2014-12-18 8:32 ` Paolo Bonzini
2014-12-19 2:09 ` Wu, Feng
2015-02-23 22:21 ` Marcelo Tosatti
2015-03-02 9:12 ` Wu, Feng
2014-12-12 15:14 ` [v3 24/26] KVM: Update Posted-Interrupts Descriptor when vCPU is blocked Feng Wu
2014-12-17 17:09 ` Paolo Bonzini
2014-12-18 3:16 ` Wu, Feng
2014-12-18 8:37 ` Paolo Bonzini
2014-12-19 2:51 ` Wu, Feng
2015-02-25 21:50 ` Marcelo Tosatti
2015-02-26 8:08 ` Wu, Feng
2015-02-26 23:41 ` Marcelo Tosatti
2015-02-26 23:40 ` Marcelo Tosatti
2015-03-02 13:36 ` Wu, Feng
2015-03-04 12:06 ` Marcelo Tosatti
2015-03-06 6:51 ` Wu, Feng
2015-03-12 1:15 ` Marcelo Tosatti
2015-03-16 11:42 ` Wu, Feng
2015-03-25 23:17 ` Marcelo Tosatti [this message]
2015-03-27 6:34 ` Wu, Feng
2015-03-27 19:30 ` Marcelo Tosatti
2015-03-30 4:46 ` Wu, Feng
2015-03-30 23:55 ` Marcelo Tosatti
2015-03-31 1:13 ` Wu, Feng
2015-04-14 7:37 ` Wu, Feng
2015-06-05 21:59 ` Marcelo Tosatti
2015-06-08 1:43 ` Wu, Feng
2014-12-12 15:14 ` [v3 25/26] KVM: Suppress posted-interrupt when 'SN' is set Feng Wu
2014-12-17 17:42 ` Paolo Bonzini
2014-12-18 3:14 ` Wu, Feng
2014-12-18 8:38 ` Paolo Bonzini
2014-12-18 15:09 ` Zhang, Yang Z
2014-12-19 2:58 ` Wu, Feng
2014-12-19 3:32 ` Zhang, Yang Z
2014-12-19 4:34 ` Wu, Feng
2014-12-19 4:44 ` Zhang, Yang Z
2014-12-19 4:49 ` Wu, Feng
2014-12-19 5:25 ` Zhang, Yang Z
2014-12-19 5:46 ` Wu, Feng
2014-12-19 7:04 ` Zhang, Yang Z
2014-12-19 12:00 ` Paolo Bonzini
2014-12-19 23:34 ` Wu, Feng
2014-12-12 15:15 ` [v3 26/26] iommu/vt-d: Add a command line parameter for VT-d posted-interrupts Feng Wu
2015-01-28 15:39 ` David Woodhouse
2014-12-16 9:04 ` [v3 00/26] Add VT-d Posted-Interrupts support Wu, Feng
2015-01-06 1:10 ` Wu, Feng
2015-01-09 12:46 ` joro
2015-01-09 13:58 ` Wu, Feng
2015-01-21 2:25 ` Wu, Feng
2015-01-28 3:01 ` Wu, Feng
2015-01-28 3:44 ` Alex Williamson
2015-01-28 4:44 ` Wu, Feng
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20150325231738.GA7333@amt.cnet \
--to=mtosatti@redhat.com \
--cc=alex.williamson@redhat.com \
--cc=dwmw2@infradead.org \
--cc=eric.auger@linaro.org \
--cc=feng.wu@intel.com \
--cc=gleb@kernel.org \
--cc=hpa@zytor.com \
--cc=iommu@lists.linux-foundation.org \
--cc=jiang.liu@linux.intel.com \
--cc=joro@8bytes.org \
--cc=kvm@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=pbonzini@redhat.com \
--cc=tglx@linutronix.de \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).