Re: [RFC 0/5] xen/arm: support big.little SoC

[George Dunlap <george.dunlap@citrix.com>]

On 22/09/16 10:27, Peng Fan wrote:
> On Thu, Sep 22, 2016 at 10:50:23AM +0200, Dario Faggioli wrote:
>> On Thu, 2016-09-22 at 14:49 +0800, Peng Fan wrote:
>>> On Wed, Sep 21, 2016 at 08:11:43PM +0100, Julien Grall wrote:
>>>>
>>>> Hi Stefano,
>>>>
>>>> On 21/09/2016 19:13, Stefano Stabellini wrote:
>>>>>
>>>>> On Wed, 21 Sep 2016, Julien Grall wrote:
>>>>>>
>>>>>> (CC a couple of ARM folks)
>>>>>>
>>>>>> On 21/09/16 11:22, George Dunlap wrote:
>>>>>>>
>>>>>>> On 21/09/16 11:09, Julien Grall wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On 20/09/16 21:17, Stefano Stabellini wrote:
>>>>>>>>>
>>>>>>>>> On Tue, 20 Sep 2016, Julien Grall wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Stefano,
>>>>>>>>>>
>>>>>>>>>> On 20/09/2016 20:09, Stefano Stabellini wrote:
>>>>>>>>>>>
>>>>>>>>>>> On Tue, 20 Sep 2016, Julien Grall wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> On 20/09/2016 12:27, George Dunlap wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Sep 20, 2016 at 11:03 AM, Peng Fan
>>>>>>>>>>>>> <van.freenix@gmail.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Sep 20, 2016 at 02:54:06AM +0200, Dario
>>>>>>>>>>>>>> Faggioli
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Mon, 2016-09-19 at 17:01 -0700, Stefano
>>>>>>>>>>>>>>> Stabellini wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, 20 Sep 2016, Dario Faggioli wrote:
>>>>>>>>>>>>>> I'd like to add a computing capability in
>>>>>>>>>>>>>> xen/arm, like this:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> struct compute_capatiliby
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>> ?? char *core_name;
>>>>>>>>>>>>>> ?? uint32_t rank;
>>>>>>>>>>>>>> ?? uint32_t cpu_partnum;
>>>>>>>>>>>>>> };
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> struct compute_capatiliby cc=
>>>>>>>>>>>>>> {
>>>>>>>>>>>>>> ??{"A72", 4, 0xd08},
>>>>>>>>>>>>>> ??{"A57", 3, 0xxxx},
>>>>>>>>>>>>>> ??{"A53", 2, 0xd03},
>>>>>>>>>>>>>> ??{"A35", 1, ...},
>>>>>>>>>>>>>> }
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Then when identify cpu, we decide which cpu is
>>>>>>>>>>>>>> big and which
>>>>>>>>>>>>>> cpu is
>>>>>>>>>>>>>> little
>>>>>>>>>>>>>> according to the computing rank.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Any comments?
>>>>>>>>>>>>>
>>>>>>>>>>>>> I think we definitely need to have Xen have some
>>>>>>>>>>>>> kind of idea
>>>>>>>>>>>>> the
>>>>>>>>>>>>> order between processors, so that the user
>>>>>>>>>>>>> doesn't need to
>>>>>>>>>>>>> figure out
>>>>>>>>>>>>> which class / pool is big and which pool is
>>>>>>>>>>>>> LITTLE.????Whether
>>>>>>>>>>>>> this
>>>>>>>>>>>>> sort
>>>>>>>>>>>>> of enumeration is the best way to do that I'll
>>>>>>>>>>>>> let Julien and
>>>>>>>>>>>>> Stefano
>>>>>>>>>>>>> give their opinion.
>>>>>>>>>>>>
>>>>>>>>>>>> I don't think an hardcoded list of processor in Xen
>>>>>>>>>>>> is the right
>>>>>>>>>>>> solution.
>>>>>>>>>>>> There are many existing processors and combinations
>>>>>>>>>>>> for big.LITTLE
>>>>>>>>>>>> so it
>>>>>>>>>>>> will
>>>>>>>>>>>> nearly be impossible to keep updated.
>>>>>>>>>>>>
>>>>>>>>>>>> I would expect the firmware table (device tree,
>>>>>>>>>>>> ACPI) to provide
>>>>>>>>>>>> relevant
>>>>>>>>>>>> data
>>>>>>>>>>>> for each processor and differentiate big from
>>>>>>>>>>>> LITTLE core.
>>>>>>>>>>>> Note that I haven't looked at it for now. A good
>>>>>>>>>>>> place to start is
>>>>>>>>>>>> looking
>>>>>>>>>>>> at
>>>>>>>>>>>> how Linux does.
>>>>>>>>>>>
>>>>>>>>>>> That's right, see
>>>>>>>>>>> Documentation/devicetree/bindings/arm/cpus.txt. It
>>>>>>>>>>> is
>>>>>>>>>>> trivial to identify the two different CPU classes and
>>>>>>>>>>> which cores
>>>>>>>>>>> belong
>>>>>>>>>>> to which class.t, as
>>>>>>>>>>
>>>>>>>>>> The class of the CPU can be found from the MIDR, there
>>>>>>>>>> is no need to
>>>>>>>>>> use the
>>>>>>>>>> device tree/acpi for that. Note that I don't think
>>>>>>>>>> there is an easy
>>>>>>>>>> way in
>>>>>>>>>> ACPI (i.e not in AML) to find out the class.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> It is harder to figure out which one is supposed to
>>>>>>>>>>> be
>>>>>>>>>>> big and which one LITTLE. Regardless, we could
>>>>>>>>>>> default to using the
>>>>>>>>>>> first cluster (usually big), which is also the
>>>>>>>>>>> cluster of the boot
>>>>>>>>>>> cpu,
>>>>>>>>>>> and utilize the second cluster only when the user
>>>>>>>>>>> demands it.
>>>>>>>>>>
>>>>>>>>>> Why do you think the boot CPU will usually be a big
>>>>>>>>>> one? In the case
>>>>>>>>>> of Juno
>>>>>>>>>> platform it is configurable, and the boot CPU is a
>>>>>>>>>> little core on r2
>>>>>>>>>> by
>>>>>>>>>> default.
>>>>>>>>>>
>>>>>>>>>> In any case, what we care about is differentiate
>>>>>>>>>> between two set of
>>>>>>>>>> CPUs. I
>>>>>>>>>> don't think Xen should care about migrating a guest
>>>>>>>>>> vCPU between big
>>>>>>>>>> and
>>>>>>>>>> LITTLE cpus. So I am not sure why we would want to know
>>>>>>>>>> that.
>>>>>>>>>
>>>>>>>>> No, it is not about migrating (at least yet). It is about
>>>>>>>>> giving useful
>>>>>>>>> information to the user. It would be nice if the user had
>>>>>>>>> to choose
>>>>>>>>> between "big" and "LITTLE" rather than "class 0x1" and
>>>>>>>>> "class 0x100", or
>>>>>>>>> even "A7" or "A15".
>>>>>>>>
>>>>>>>> I don't think it is wise to assume that we may have only 2
>>>>>>>> kind of CPUs
>>>>>>>> on the platform. We may have more in the future, if so how
>>>>>>>> would you
>>>>>>>> name them?
>>>>>>>
>>>>>>> I would suggest that internally Xen recognize an arbitrary
>>>>>>> number of
>>>>>>> processor "classes", and order them according to more
>>>>>>> powerful -> less
>>>>>>> powerful.????Then if at some point someone makes a platform
>>>>>>> with three
>>>>>>> processors, you can say "class 0", "class 1" or "class
>>>>>>> 2".????"big" would
>>>>>>> be an alias for "class 0" and "little" would be an alias for
>>>>>>> "class 1".
>>>>>>
>>>>>> As mentioned earlier, there is no upstreamed yet device tree
>>>>>> bindings to know
>>>>>> the "power" of a CPU (see [1]
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> And in my suggestion, we allow a richer set of labels, so
>>>>>>> that the user
>>>>>>> could also be more specific -- e.g., asking for "A15"
>>>>>>> specifically, for
>>>>>>> example, and failing to build if there are no A15 cores
>>>>>>> present, while
>>>>>>> allowing users to simply write "big" or "little" if they want
>>>>>>> simplicity
>>>>>>> / things which work across different platforms.
>>>>>>
>>>>>> Well, before trying to do something clever like that (i.e
>>>>>> naming "big" and
>>>>>> "little"), we need to have upstreamed bindings available to
>>>>>> acknowledge the
>>>>>> difference. AFAICT, it is not yet upstreamed for Device Tree
>>>>>> (see [1]) and I
>>>>>> don't know any static ACPI tables providing the similar
>>>>>> information.
>>>>>
>>>>> I like George's idea that "big" and "little" could be just
>>>>> convenience
>>>>> aliases. Of course they are predicated on the necessary device
>>>>> tree
>>>>> bindings being upstream. We don't need [1] to be upstream in
>>>>> Linux, just
>>>>> the binding:
>>>>>
>>>>> http://marc.info/?l=linux-arm-kernel&m=147308556729426&w=2
>>>>>
>>>>> which has already been acked by the relevant maintainers.
>>>>
>>>> This is device tree only. What about ACPI?
>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> I had few discussions and????more thought about big.LITTLE
>>>>>> support in Xen. The
>>>>>> main goal of big.LITTLE is power efficiency by moving task
>>>>>> around and been
>>>>>> able to idle one cluster. All the solutions suggested
>>>>>> (including mine) so far,
>>>>>> can be replicated by hand (except the VPIDR) so they are mostly
>>>>>> an automatic
>>>>>> way. This will also remove the real benefits of big.LITTLE
>>>>>> because Xen will
>>>>>> not be able to migrate vCPU across cluster for power
>>>>>> efficiency.
>>>>>
>>>>> The goal of the architects of big.LITTLE might have been power
>>>>> efficiency, but of course we are free to use any features that
>>>>> the
>>>>> hardware provides in the best way for Xen and the Xen community.
>>>>
>>>> This is very dependent on how the big.LITTLE has been implemented
>>>> by the
>>>> hardware. Some platform can not run both big and LITTLE cores at
>>>> the same
>>>> time. You need a proper switch in the firmware/hypervisor.
>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> If we care about power efficiency, we would have to handle
>>>>>> seamlessly
>>>>>> big.LITTLE in Xen (i.e a guess would only see a kind of CPU).
>>>>>> This arise quite
>>>>>> few problem, nothing insurmountable, similar to migration
>>>>>> across two platforms
>>>>>> with different micro-architecture (e.g processors): errata,
>>>>>> features
>>>>>> supported... The guest would have to know the union of all the
>>>>>> errata (this is
>>>>>> done so far via the MIDR, so we would a PV way to do it), and
>>>>>> only the
>>>>>> intersection of features would be exposed to the guest. This
>>>>>> also means the
>>>>>> scheduler would have to be modified to handle power efficiency
>>>>>> (not strictly
>>>>>> necessary at the beginning).
>>>>>>
>>>>>> I agree that a such solution would require some work to
>>>>>> implement, although
>>>>>> Xen will have a better control of the energy consumption of the
>>>>>> platform.
>>>>>>
>>>>>> So the question here, is what do we want to achieve with
>>>>>> big.LITTLE?
>>>>>
>>>>> I don't think that handling seamlessly big.LITTLE in Xen is the
>>>>> best way
>>>>> to do it in the scenarios where Xen on ARM is being used today. I
>>>>> understand the principles behind it, but I don't think that it
>>>>> will lead
>>>>> to good results in a virtualized environment, where there is more
>>>>> activity and more vcpus than pcpus.
>>>>
>>>> Can you detail why you don't think it will give good results?
>>>>
>>>>>
>>>>>
>>>>> What we discussed in this thread so far is actionable, and gives
>>>>> us
>>>>> big.LITTLE support in a short time frame. It is a good fit for
>>>>> Xen on
>>>>> ARM use cases and still leads to lower power consumption with an
>>>>> wise
>>>>> allocation of big and LITTLE vcpus and pcpus to guests.
>>>>
>>>> How this would lead to lower power consumption? If there is nothing
>>>> running
>>>> of the processor we would have a wfi loop which will never put the
>>>> physical
>>>> CPU in deep sleep. The main advantage of big.LITTLE is too be able
>>>> to switch
>>>> off a cluster/cpu when it is not used.
>>>>
>>>> Without any knowledge in Xen (such as CPU freq), I am afraid the
>>>> the power
>>>> consumption will still be the same.
>>>>
>>>>>
>>>>>
>>>>> I would start from this approach, then if somebody comes along
>>>>> with a
>>>>> plan to implement a big.LITTLE switcher in Xen, I welcome her to
>>>>> do it
>>>>> and I would be happy to accept the code in Xen. We'll just make
>>>>> it
>>>>> optional.
>>>>
>>>> I think we are discussing here a simple design for big.LITTLE. I
>>>> never asked
>>>> Peng to do all the work. I am worry that if we start to expose the
>>>> big.LITTLE
>>>> to the userspace it will be hard in the future to step back from
>>>> it.
>>>
>>> Hello Julien,
>>>
>>>
>>> I prefer the simple doable method, and As Stefano said, actionable.
>>>
>> Yep, this is a very good starting point, IMO.
>>
>>> ??- introduce a hypercall interface to let xl can get the different
>>> classes cpu info.
>>>
>> +1
>>
>>> ??- Use vcpuclass in xl cfg file to let user create different vcpus
>>>
>> Yep.
>>
>>> ??- extract cpu computing cap from dts to differentiate cpus. As you
>>> said, bindings
>>> ??????not upstreamed. But this is not the hardpoint, we could change the
>>> info, whether
>>> ??????dmips or cap in future.
>>>
>> Yes (or I should say, "whatever", as I know nothing about all
>> this! :-P)
>>
>>> ??- use cpu hard affinity to block vcpu scheduling bwtween little and
>>> big pcpu.
>>>
>> "to block vcpu scheduling within the specified classes, for each vcpu"
>>
>> But, again, yes.
>>
>>> ??- block user setting vcpu hard affinity bwtween big and LITTLE.
>>>
>> "between the specified class"
>>
>> Indeed.
>>
>>> ??- only hard affinity seems enough, no need soft affinity.
>>>
>> Correct. Just don't care at all and don't touch soft affinity for now.
>>
>>> Anyway, for vcpu scheduling bwtween big and LITTLE, if this is the
>>> right
>>> direction and you have an idea on how to implement, I could follow
>>> you on
>>> enabling this feature with you leading the work. I do not have much
>>> idea on this.
>>>
>> This can come later, either as an enhancement of this affinity based
>> solution, or being implemented on top of it.
>>
>> In any case, let's start with the affinity based solution for now. It's
>> a good level support already, and a nice first step toward future
>> improvements.
>>
>> Oh, btw, please don't throw away this cpupool patch series either!
> 
> Ok -:)
> 
>>
>> A feature like `xl cpupool-biglittle-split' can still be interesting,
> 
> "cpupool-cluster-split" maybe a better name?

I think we should name this however we name the different types of cpus.
 i.e., if we're going to call these "cpu classes", then we should call
this "cpupool-cpuclass-split" or something.

 -George


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel