Re: vmx: VT-d posted-interrupt core logic handling

From: Dario Faggioli <dario.faggioli@citrix.com>
To: George Dunlap <george.dunlap@citrix.com>,
	Jan Beulich <JBeulich@suse.com>,
	Kevin Tian <kevin.tian@intel.com>
Cc: Lars Kurth <lars.kurth@citrix.com>, Feng Wu <feng.wu@intel.com>,
	George Dunlap <George.Dunlap@eu.citrix.com>,
	Andrew Cooper <andrew.cooper3@citrix.com>,
	Ian Jackson <Ian.Jackson@eu.citrix.com>,
	"xen-devel@lists.xen.org" <xen-devel@lists.xen.org>,
	David Vrabel <david.vrabel@citrix.com>
Subject: Re: vmx: VT-d posted-interrupt core logic handling
Date: Thu, 10 Mar 2016 12:21:59 +0100	[thread overview]
Message-ID: <1457608919.3102.496.camel@citrix.com> (raw)
In-Reply-To: <56E153DD.3080309@citrix.com>

[-- Attachment #1.1: Type: text/plain, Size: 2380 bytes --]

On Thu, 2016-03-10 at 11:00 +0000, George Dunlap wrote:
> > > > > On 10.03.16 at 11:05, <kevin.tian@intel.com> wrote:
> > > >  From: Tian, Kevin
> > > > Sent: Thursday, March 10, 2016 5:20 PM
> > > > 
> > > Actually after more thinking, there is no hard requirement that
> > > the vcpu must block on the pcpu which is configured in 'NDST'
> > > of that vcpu's PI descriptor. What really matters, is that the
> > > vcpu is added to the linked list of the very pcpu, then when PI
> > > notification comes we can always find out the vcpu struct from
> > > that pcpu's linked list. Of course one drawback of such placement
> > > is additional IPI incurred in wake up path.
> > > 
> > > 
> Re "an IPI is sent to PCPU1", all that should be transparent to the
> PI
> code -- it already calls vcpu_unblock(), which will call vcpu_wake(),
> which calls the scheduling wake code, which will DTRT.
> 
Exactly. In fact, whether there will be any IPI involved is under
control of the scheduler, rather than of PI code, even right now.

In fact, no matter in what pCPU's blocked list a vCPU is, it is the
'tickling' logic (for all Credit, Credit2 and RTDS) that really decides
on which pCPU the vCPU should wakeup, and send the IPI, if that is not
the pCPU we're running on.

It can be argued that having a vCPU in the blocked list of the pCPU
where it was running when it blocked could be a good thing, because it
may then be able to restart running there when waking, which may have
positive cache effects, etc.
But that is not at all guaranteed. In fact, it could well be the case
in fairly idle systems, but under high load and/or if hard and soft
affinity are in use (which may well be the case, .e.g, on large NUMA
servers), that isn't necessarily true, and we should not base on such
assumption.

> FWIW I have much less objection to this sort of solution if it were
> confined to the PI arch_block() callback, rather than something that
> required changes to the schedulers.
> 
Same here, and I think this can well be done in such a way... Worth a
shot, IMO.

Regards,
Dario
-- 
<<This happens because I choose it to happen!>> (Raistlin Majere)
-----------------------------------------------------------------
Dario Faggioli, Ph.D, http://about.me/dario.faggioli
Senior Software Engineer, Citrix Systems R&D Ltd., Cambridge (UK)

[-- Attachment #1.2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 181 bytes --]

[-- Attachment #2: Type: text/plain, Size: 126 bytes --]

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel