From mboxrd@z Thu Jan 1 00:00:00 1970 From: Stefano Stabellini Subject: Re: [PATCH] xen/arm: introduce vwfi parameter Date: Mon, 20 Feb 2017 16:38:55 -0800 (PST) Message-ID: References: <1487286292-29502-1-git-send-email-sstabellini@kernel.org> <484f2a39-1322-f40e-de01-97746e5de280@arm.com> <1487618436.6732.230.camel@citrix.com> <1845466e-c6a3-bcb9-0813-80ecdf267f03@arm.com> <1487631215.6732.266.camel@citrix.com> Mime-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="8323329-99290668-1487637536=:29974" Return-path: Received: from mail6.bemta5.messagelabs.com ([195.245.231.135]) by lists.xenproject.org with esmtp (Exim 4.84_2) (envelope-from ) id 1cfyTu-0005J1-4J for xen-devel@lists.xenproject.org; Tue, 21 Feb 2017 00:39:02 +0000 In-Reply-To: <1487631215.6732.266.camel@citrix.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Errors-To: xen-devel-bounces@lists.xen.org Sender: "Xen-devel" To: Dario Faggioli Cc: edgar.iglesias@xilinx.com, Stefano Stabellini , george.dunlap@eu.citrix.com, Punit Agrawal , Julien Grall , xen-devel@lists.xenproject.org, nd@arm.com List-Id: xen-devel@lists.xenproject.org This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --8323329-99290668-1487637536=:29974 Content-Type: TEXT/PLAIN; charset=UTF-8 Content-Transfer-Encoding: 8BIT On Mon, 20 Feb 2017, Dario Faggioli wrote: > On Mon, 2017-02-20 at 19:38 +0000, Julien Grall wrote: > > On 20/02/17 19:20, Dario Faggioli wrote: > > > E.g., if vCPU x of domain A wants to go idle with a WFI/WFE, but > > > the > > > host is overbooked and currently really busy, Xen wants to run some > > > other vCPU (of either the same of another domain). > > > > > > That's actually the whole point of virtualization, and the reason > > > why > > > overbooking an host with more vCPUs (from multiple guests) than it > > > has > > > pCPUs works at all. If we start letting guests put the host's pCPUs > > > to > > > sleep, not only the scheduler, but many things would break, IMO! > > > > I am not speaking about general case but when you get 1 vCPU pinned > > to 1  > > pCPU (I think this is Stefano use case). No other vCPU will run on > > this  > > pCPU. So it would be fine to let the guest do the WFI. > > > Mmm... ok, yes, in that case, it may make sense and work, from a, let's > say, purely functional perspective. But still I struggle to place this > in a bigger picture. I feel the same way as you, Dario. That said, if we could make it work without breaking too many assumptions in Xen, it would be a great improvement for this use-case. > For instance, as you say, executing a WFI from a guest directly on > hardware, only makes sense if we have 1:1 static pinning. Which means > it can't just be done by default, or with a boot parameter, because we > need to check and enforce that there's only 1:1 pinning around. That's right, but we don't have a way to recognize or enforce 1:1 static pinning at the moment, do we? But maybe the nop scheduler we discussed could be a step in the right direction. > Is it possible to decide whether to trap and emulate WFI, or just > execute it, online, and change such decision dynamically? And even if > yes, how would the whole thing work? When the direct execution is > enabled for a domain we automatically enforce 1:1 pinning for that > domain, and kick all the other domain out of its pcpus? What if they > have their own pinning, what if they also have 'direct WFI' behavior > enabled? Right, I asked myself those questions as well. That is why I wrote "it breaks the scheduler" in the previous email. I don't think it can work today, but it could work one day, building on top of the nop scheduler. > If it is not possible to change all this online and on a per-domain > basis, what do we do? When dooted with the 'direct WFI' flag, we only > accept 1:1 pinning? Who should enforce that, the setvcpuaffinity > hypercall? > > These are just examples, my point being that in theory, if we consider > a very specific usecase or set of usecase, there's a lot we can do. But > when you say "why don't you let the guest directly execute WFI", in > response to a patch and a discussion like this, people may think that > you are actually proposing doing it as a solution, which is not > possible without figuring out all the open questions above (actually, > probably, more) and without introducing a lot of cross-subsystem > policing inside Xen, which is often something we don't want. +1 > But, if you let me say this again, it looks to me we are trying to > solve too many problem all at once in this thread, should we try > slowing down/refocusing? :-) Indeed. I think this patch can improve some use-cases with little maintenance cost. It's pretty straightforward to me. > > But in  > > that case, using WFI in the guest may not have been the right things > > to do. > > > But if the guest is, let's say, Linux, does it use WFI or not? And is > it the right thing or not? > > Again, the fact you're saying this probably means there's something I > am either missing or ignoring about ARM. Linux uses WFI > > I have heard use case where people wants to disable the scheduler > > (e.g a  > > nop scheduler) because they know only 1 vCPU will ever run on the > > pCPU.  > > This is exactly the use case I am thinking about. > > > Sure! Except that, in Xen, we don't know whether we have, and always > will, 1 vCPU ever run on each pCPU. Nor we have a way to enforce that, > neither in toolstack nor in the hypervisor. :-P Exactly! I should have written it more clearly from the beginning. --8323329-99290668-1487637536=:29974 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: base64 Content-Disposition: inline X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KWGVuLWRldmVs IG1haWxpbmcgbGlzdApYZW4tZGV2ZWxAbGlzdHMueGVuLm9yZwpodHRwczovL2xpc3RzLnhlbi5v cmcveGVuLWRldmVsCg== --8323329-99290668-1487637536=:29974--