Re: [RFC 0/5] xen/arm: support big.little SoC

From: Peng Fan <van.freenix@gmail.com>
To: George Dunlap <george.dunlap@citrix.com>
Cc: J??rgen Gro?? <jgross@suse.com>, Peng Fan <peng.fan@nxp.com>,
	Stefano Stabellini <sstabellini@kernel.org>,
	George Dunlap <George.Dunlap@eu.citrix.com>,
	Andrew Cooper <andrew.cooper3@citrix.com>,
	Dario Faggioli <dario.faggioli@citrix.com>,
	"xen-devel@lists.xen.org" <xen-devel@lists.xen.org>,
	Julien Grall <julien.grall@arm.com>,
	Jan Beulich <jbeulich@suse.com>
Subject: Re: [RFC 0/5] xen/arm: support big.little SoC
Date: Mon, 19 Sep 2016 21:33:01 +0800	[thread overview]
Message-ID: <20160919133259.GC7407@linux-u7w5.ap.freescale.net> (raw)
In-Reply-To: <170e2787-a410-37c5-a675-6fc7cf31ad6f@citrix.com>

On Mon, Sep 19, 2016 at 11:33:58AM +0100, George Dunlap wrote:
>On 19/09/16 11:06, Julien Grall wrote:
>> Hi George,
>> 
>> On 19/09/2016 11:45, George Dunlap wrote:
>>> On Mon, Sep 19, 2016 at 9:53 AM, Julien Grall <julien.grall@arm.com>
>>> wrote:
>>>>>> As mentioned in the mail you pointed above, this series is not
>>>>>> enough to
>>>>>> make
>>>>>> big.LITTLE working on then. Xen is always using the boot CPU to detect
>>>>>> the
>>>>>> list of features. With big.LITTLE features may not be the same.
>>>>>>
>>>>>> And I would prefer to see Xen supporting big.LITTLE correctly before
>>>>>> beginning to think to expose big.LITTLE to the userspace (via cpupool)
>>>>>> automatically.
>>>>>
>>>>>
>>>>> Do you mean vcpus be scheduled between big and little cpus freely?
>>>>
>>>>
>>>> By supporting big.LITTLE correctly I meant Xen thinks that all the
>>>> cores has
>>>> the same set of features. So the feature detection is only done the boot
>>>> CPU. See processor_setup for instance...
>>>>
>>>> Moving vCPUs between big and little cores would be a hard task (cache
>>>> line
>>>> issue, and possibly feature) and I don't expect to ever cross this in
>>>> Xen.
>>>> However, I am expecting to see big.LITTLE exposed to the guest (i.e
>>>> having
>>>> big and little vCPUs).
>>>
>>> So it sounds like the big and LITTLE cores are architecturally
>>> different enough that software must be aware of which one it's running
>>> on?
>> 
>> That's correct. Each big and LITTLE cores may have different errata,
>> different features...
>> 
>> It has also the advantage to let the guest dealing itself with its own
>> power efficiency without introducing a specific Xen interface.
>
>Well in theory there would be advantages either way -- either to
>allowing Xen to automatically add power-saving "smarts" to guests which
>weren't programmed for them, or to exposing the power-saving abilities
>to guests which were.  But it sounds like automatically migrating
>between them isn't really an option (or would be a lot more trouble than
>it's worth).
>
>>>> I care about having a design allowing an easy use of big.LITTLE on
>>>> Xen. Your
>>>> solution requires the administrator to know the underlying platform and
>>>> create the pool.
>>>>
>>>> In the solution I suggested, the pools would be created by Xen (and
>>>> the info
>>>> exposed to the userspace for the admin).
>>>
>>> FWIW another approach could be the one taken by "xl
>>> cpupool-numa-split": you could have "xl cpupool-bigLITTLE-split" or
>>> something that would automatically set up the pools.
>>>
>>> But expanding the schedulers to know about different classes of cpus,
>>> and having vcpus specified as running only on specific types of pcpus,
>>> seems like a more flexible approach.
>> 
>> So, if I understand correctly, you would not recommend to extend the
>> number of CPU pool per domain, correct?
>
>Well imagine trying to set the scheduling parameters, such as weight,
>which in the past have been per-domain.  Now you have to specify
>parameters for a domain in each of the cpupools that its' in.
>
>No, I think it would be a lot simpler to just teach the scheduler about
>different classes of cpus.  credit1 would probably need to be modified
>so that its credit algorithm would be per-class rather than pool-wide;
>but credit2 shouldn't need much modification at all, other than to make
>sure that a given runqueue doesn't include more than one class; and to
>do load-balancing only with runqueues of the same class.

I try to follow.
 - scheduler needs to be aware of different classes of cpus. ARM big.Little cpus.
 - scheduler schedules vcpus on different physical cpus in one cpupool.
 - different cpu classes needs to be in different runqueue.

Then for implementation.
 - When create a guest, specific physical cpus that the guest will be run on.
 - If the physical cpus are different cpus, indicate the guest would like to be a big.little guest.
   And have big vcpus and little vcpus.
 - If no physical cpus specificed, then the guest may runs on big cpus or on little cpus. But not both.
   How to decide runs on big or little physical cpus?
 - For Dom0, I am still not sure,default big.little or else?

If use scheduler to handle the different classes cpu, we do not need to use cpupool
to block vcpus be scheduled onto different physical cpus. And using scheudler to handle this
gives an opportunity to support big.little guest.

Thanks,
Peng.

>
> -George

-- 

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel