Re: [RFC 0/5] xen/arm: support big.little SoC

[Julien Grall <julien.grall@arm.com>]

(CC a couple of ARM folks)

On 21/09/16 11:22, George Dunlap wrote:
> On 21/09/16 11:09, Julien Grall wrote:
>>
>>
>> On 20/09/16 21:17, Stefano Stabellini wrote:
>>> On Tue, 20 Sep 2016, Julien Grall wrote:
>>>> Hi Stefano,
>>>>
>>>> On 20/09/2016 20:09, Stefano Stabellini wrote:
>>>>> On Tue, 20 Sep 2016, Julien Grall wrote:
>>>>>> Hi,
>>>>>>
>>>>>> On 20/09/2016 12:27, George Dunlap wrote:
>>>>>>> On Tue, Sep 20, 2016 at 11:03 AM, Peng Fan <van.freenix@gmail.com>
>>>>>>> wrote:
>>>>>>>> On Tue, Sep 20, 2016 at 02:54:06AM +0200, Dario Faggioli wrote:
>>>>>>>>> On Mon, 2016-09-19 at 17:01 -0700, Stefano Stabellini wrote:
>>>>>>>>>> On Tue, 20 Sep 2016, Dario Faggioli wrote:
>>>>>>>> I'd like to add a computing capability in xen/arm, like this:
>>>>>>>>
>>>>>>>> struct compute_capatiliby
>>>>>>>> {
>>>>>>>>    char *core_name;
>>>>>>>>    uint32_t rank;
>>>>>>>>    uint32_t cpu_partnum;
>>>>>>>> };
>>>>>>>>
>>>>>>>> struct compute_capatiliby cc=
>>>>>>>> {
>>>>>>>>   {"A72", 4, 0xd08},
>>>>>>>>   {"A57", 3, 0xxxx},
>>>>>>>>   {"A53", 2, 0xd03},
>>>>>>>>   {"A35", 1, ...},
>>>>>>>> }
>>>>>>>>
>>>>>>>> Then when identify cpu, we decide which cpu is big and which cpu is
>>>>>>>> little
>>>>>>>> according to the computing rank.
>>>>>>>>
>>>>>>>> Any comments?
>>>>>>>
>>>>>>> I think we definitely need to have Xen have some kind of idea the
>>>>>>> order between processors, so that the user doesn't need to figure out
>>>>>>> which class / pool is big and which pool is LITTLE.  Whether this
>>>>>>> sort
>>>>>>> of enumeration is the best way to do that I'll let Julien and Stefano
>>>>>>> give their opinion.
>>>>>>
>>>>>> I don't think an hardcoded list of processor in Xen is the right
>>>>>> solution.
>>>>>> There are many existing processors and combinations for big.LITTLE
>>>>>> so it
>>>>>> will
>>>>>> nearly be impossible to keep updated.
>>>>>>
>>>>>> I would expect the firmware table (device tree, ACPI) to provide
>>>>>> relevant
>>>>>> data
>>>>>> for each processor and differentiate big from LITTLE core.
>>>>>> Note that I haven't looked at it for now. A good place to start is
>>>>>> looking
>>>>>> at
>>>>>> how Linux does.
>>>>>
>>>>> That's right, see Documentation/devicetree/bindings/arm/cpus.txt. It is
>>>>> trivial to identify the two different CPU classes and which cores
>>>>> belong
>>>>> to which class.t, as
>>>>
>>>> The class of the CPU can be found from the MIDR, there is no need to
>>>> use the
>>>> device tree/acpi for that. Note that I don't think there is an easy
>>>> way in
>>>> ACPI (i.e not in AML) to find out the class.
>>>>
>>>>> It is harder to figure out which one is supposed to be
>>>>> big and which one LITTLE. Regardless, we could default to using the
>>>>> first cluster (usually big), which is also the cluster of the boot cpu,
>>>>> and utilize the second cluster only when the user demands it.
>>>>
>>>> Why do you think the boot CPU will usually be a big one? In the case
>>>> of Juno
>>>> platform it is configurable, and the boot CPU is a little core on r2 by
>>>> default.
>>>>
>>>> In any case, what we care about is differentiate between two set of
>>>> CPUs. I
>>>> don't think Xen should care about migrating a guest vCPU between big and
>>>> LITTLE cpus. So I am not sure why we would want to know that.
>>>
>>> No, it is not about migrating (at least yet). It is about giving useful
>>> information to the user. It would be nice if the user had to choose
>>> between "big" and "LITTLE" rather than "class 0x1" and "class 0x100", or
>>> even "A7" or "A15".
>>
>> I don't think it is wise to assume that we may have only 2 kind of CPUs
>> on the platform. We may have more in the future, if so how would you
>> name them?
>
> I would suggest that internally Xen recognize an arbitrary number of
> processor "classes", and order them according to more powerful -> less
> powerful.  Then if at some point someone makes a platform with three
> processors, you can say "class 0", "class 1" or "class 2".  "big" would
> be an alias for "class 0" and "little" would be an alias for "class 1".

As mentioned earlier, there is no upstreamed yet device tree bindings to 
know the "power" of a CPU (see [1]

>
> And in my suggestion, we allow a richer set of labels, so that the user
> could also be more specific -- e.g., asking for "A15" specifically, for
> example, and failing to build if there are no A15 cores present, while
> allowing users to simply write "big" or "little" if they want simplicity
> / things which work across different platforms.

Well, before trying to do something clever like that (i.e naming "big" 
and "little"), we need to have upstreamed bindings available to 
acknowledge the difference. AFAICT, it is not yet upstreamed for Device 
Tree (see [1]) and I don't know any static ACPI tables providing the 
similar information.

I had few discussions and  more thought about big.LITTLE support in Xen. 
The main goal of big.LITTLE is power efficiency by moving task around 
and been able to idle one cluster. All the solutions suggested 
(including mine) so far, can be replicated by hand (except the VPIDR) so 
they are mostly an automatic way. This will also remove the real 
benefits of big.LITTLE because Xen will not be able to migrate vCPU 
across cluster for power efficiency.

If we care about power efficiency, we would have to handle seamlessly 
big.LITTLE in Xen (i.e a guess would only see a kind of CPU). This arise 
quite few problem, nothing insurmountable, similar to migration across 
two platforms with different micro-architecture (e.g processors): 
errata, features supported... The guest would have to know the union of 
all the errata (this is done so far via the MIDR, so we would a PV way 
to do it), and only the intersection of features would be exposed to the 
guest. This also means the scheduler would have to be modified to handle 
power efficiency (not strictly necessary at the beginning).

I agree that a such solution would require some work to implement, 
although Xen will have a better control of the energy consumption of the 
platform.

So the question here, is what do we want to achieve with big.LITTLE?

Regards,

[1] https://lwn.net/Articles/699569/

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel