Re: [RFC 0/5] xen/arm: support big.little SoC

[Julien Grall <julien.grall@arm.com>]

On 21/09/2016 20:11, Julien Grall wrote:
> Hi Stefano,
>
> On 21/09/2016 19:13, Stefano Stabellini wrote:
>> On Wed, 21 Sep 2016, Julien Grall wrote:
>>> (CC a couple of ARM folks)
>>>
>>> On 21/09/16 11:22, George Dunlap wrote:
>>>> On 21/09/16 11:09, Julien Grall wrote:
>>>>>
>>>>>
>>>>> On 20/09/16 21:17, Stefano Stabellini wrote:
>>>>>> On Tue, 20 Sep 2016, Julien Grall wrote:
>>>>>>> Hi Stefano,
>>>>>>>
>>>>>>> On 20/09/2016 20:09, Stefano Stabellini wrote:
>>>>>>>> On Tue, 20 Sep 2016, Julien Grall wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> On 20/09/2016 12:27, George Dunlap wrote:
>>>>>>>>>> On Tue, Sep 20, 2016 at 11:03 AM, Peng Fan
>>>>>>>>>> <van.freenix@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>> On Tue, Sep 20, 2016 at 02:54:06AM +0200, Dario Faggioli
>>>>>>>>>>> wrote:
>>>>>>>>>>>> On Mon, 2016-09-19 at 17:01 -0700, Stefano Stabellini wrote:
>>>>>>>>>>>>> On Tue, 20 Sep 2016, Dario Faggioli wrote:
>>>>>>>>>>> I'd like to add a computing capability in xen/arm, like this:
>>>>>>>>>>>
>>>>>>>>>>> struct compute_capatiliby
>>>>>>>>>>> {
>>>>>>>>>>>    char *core_name;
>>>>>>>>>>>    uint32_t rank;
>>>>>>>>>>>    uint32_t cpu_partnum;
>>>>>>>>>>> };
>>>>>>>>>>>
>>>>>>>>>>> struct compute_capatiliby cc=
>>>>>>>>>>> {
>>>>>>>>>>>   {"A72", 4, 0xd08},
>>>>>>>>>>>   {"A57", 3, 0xxxx},
>>>>>>>>>>>   {"A53", 2, 0xd03},
>>>>>>>>>>>   {"A35", 1, ...},
>>>>>>>>>>> }
>>>>>>>>>>>
>>>>>>>>>>> Then when identify cpu, we decide which cpu is big and which
>>>>>>>>>>> cpu is
>>>>>>>>>>> little
>>>>>>>>>>> according to the computing rank.
>>>>>>>>>>>
>>>>>>>>>>> Any comments?
>>>>>>>>>>
>>>>>>>>>> I think we definitely need to have Xen have some kind of idea
>>>>>>>>>> the
>>>>>>>>>> order between processors, so that the user doesn't need to
>>>>>>>>>> figure out
>>>>>>>>>> which class / pool is big and which pool is LITTLE.  Whether
>>>>>>>>>> this
>>>>>>>>>> sort
>>>>>>>>>> of enumeration is the best way to do that I'll let Julien and
>>>>>>>>>> Stefano
>>>>>>>>>> give their opinion.
>>>>>>>>>
>>>>>>>>> I don't think an hardcoded list of processor in Xen is the right
>>>>>>>>> solution.
>>>>>>>>> There are many existing processors and combinations for big.LITTLE
>>>>>>>>> so it
>>>>>>>>> will
>>>>>>>>> nearly be impossible to keep updated.
>>>>>>>>>
>>>>>>>>> I would expect the firmware table (device tree, ACPI) to provide
>>>>>>>>> relevant
>>>>>>>>> data
>>>>>>>>> for each processor and differentiate big from LITTLE core.
>>>>>>>>> Note that I haven't looked at it for now. A good place to start is
>>>>>>>>> looking
>>>>>>>>> at
>>>>>>>>> how Linux does.
>>>>>>>>
>>>>>>>> That's right, see
>>>>>>>> Documentation/devicetree/bindings/arm/cpus.txt. It
>>>>>>>> is
>>>>>>>> trivial to identify the two different CPU classes and which cores
>>>>>>>> belong
>>>>>>>> to which class.t, as
>>>>>>>
>>>>>>> The class of the CPU can be found from the MIDR, there is no need to
>>>>>>> use the
>>>>>>> device tree/acpi for that. Note that I don't think there is an easy
>>>>>>> way in
>>>>>>> ACPI (i.e not in AML) to find out the class.
>>>>>>>
>>>>>>>> It is harder to figure out which one is supposed to be
>>>>>>>> big and which one LITTLE. Regardless, we could default to using the
>>>>>>>> first cluster (usually big), which is also the cluster of the boot
>>>>>>>> cpu,
>>>>>>>> and utilize the second cluster only when the user demands it.
>>>>>>>
>>>>>>> Why do you think the boot CPU will usually be a big one? In the case
>>>>>>> of Juno
>>>>>>> platform it is configurable, and the boot CPU is a little core on r2
>>>>>>> by
>>>>>>> default.
>>>>>>>
>>>>>>> In any case, what we care about is differentiate between two set of
>>>>>>> CPUs. I
>>>>>>> don't think Xen should care about migrating a guest vCPU between big
>>>>>>> and
>>>>>>> LITTLE cpus. So I am not sure why we would want to know that.
>>>>>>
>>>>>> No, it is not about migrating (at least yet). It is about giving
>>>>>> useful
>>>>>> information to the user. It would be nice if the user had to choose
>>>>>> between "big" and "LITTLE" rather than "class 0x1" and "class
>>>>>> 0x100", or
>>>>>> even "A7" or "A15".
>>>>>
>>>>> I don't think it is wise to assume that we may have only 2 kind of
>>>>> CPUs
>>>>> on the platform. We may have more in the future, if so how would you
>>>>> name them?
>>>>
>>>> I would suggest that internally Xen recognize an arbitrary number of
>>>> processor "classes", and order them according to more powerful -> less
>>>> powerful.  Then if at some point someone makes a platform with three
>>>> processors, you can say "class 0", "class 1" or "class 2".  "big" would
>>>> be an alias for "class 0" and "little" would be an alias for "class 1".
>>>
>>> As mentioned earlier, there is no upstreamed yet device tree bindings
>>> to know
>>> the "power" of a CPU (see [1]
>>>
>>>>
>>>> And in my suggestion, we allow a richer set of labels, so that the user
>>>> could also be more specific -- e.g., asking for "A15" specifically, for
>>>> example, and failing to build if there are no A15 cores present, while
>>>> allowing users to simply write "big" or "little" if they want
>>>> simplicity
>>>> / things which work across different platforms.
>>>
>>> Well, before trying to do something clever like that (i.e naming
>>> "big" and
>>> "little"), we need to have upstreamed bindings available to
>>> acknowledge the
>>> difference. AFAICT, it is not yet upstreamed for Device Tree (see
>>> [1]) and I
>>> don't know any static ACPI tables providing the similar information.
>>
>> I like George's idea that "big" and "little" could be just convenience
>> aliases. Of course they are predicated on the necessary device tree
>> bindings being upstream. We don't need [1] to be upstream in Linux, just
>> the binding:
>>
>> http://marc.info/?l=linux-arm-kernel&m=147308556729426&w=2
>>
>> which has already been acked by the relevant maintainers.
>
> This is device tree only. What about ACPI?
>
>>
>>
>>> I had few discussions and  more thought about big.LITTLE support in
>>> Xen. The
>>> main goal of big.LITTLE is power efficiency by moving task around and
>>> been
>>> able to idle one cluster. All the solutions suggested (including
>>> mine) so far,
>>> can be replicated by hand (except the VPIDR) so they are mostly an
>>> automatic
>>> way. This will also remove the real benefits of big.LITTLE because
>>> Xen will
>>> not be able to migrate vCPU across cluster for power efficiency.
>>
>> The goal of the architects of big.LITTLE might have been power
>> efficiency, but of course we are free to use any features that the
>> hardware provides in the best way for Xen and the Xen community.
>
> This is very dependent on how the big.LITTLE has been implemented by the
> hardware. Some platform can not run both big and LITTLE cores at the
> same time. You need a proper switch in the firmware/hypervisor.
>
>>
>>> If we care about power efficiency, we would have to handle seamlessly
>>> big.LITTLE in Xen (i.e a guess would only see a kind of CPU). This
>>> arise quite
>>> few problem, nothing insurmountable, similar to migration across two
>>> platforms
>>> with different micro-architecture (e.g processors): errata, features
>>> supported... The guest would have to know the union of all the errata
>>> (this is
>>> done so far via the MIDR, so we would a PV way to do it), and only the
>>> intersection of features would be exposed to the guest. This also
>>> means the
>>> scheduler would have to be modified to handle power efficiency (not
>>> strictly
>>> necessary at the beginning).
>>>
>>> I agree that a such solution would require some work to implement,
>>> although
>>> Xen will have a better control of the energy consumption of the
>>> platform.
>>>
>>> So the question here, is what do we want to achieve with big.LITTLE?
>>
>> I don't think that handling seamlessly big.LITTLE in Xen is the best way
>> to do it in the scenarios where Xen on ARM is being used today. I
>> understand the principles behind it, but I don't think that it will lead
>> to good results in a virtualized environment, where there is more
>> activity and more vcpus than pcpus.
>
> Can you detail why you don't think it will give good results?
>
>>
>> What we discussed in this thread so far is actionable, and gives us
>> big.LITTLE support in a short time frame. It is a good fit for Xen on
>> ARM use cases and still leads to lower power consumption with an wise
>> allocation of big and LITTLE vcpus and pcpus to guests.
>
> How this would lead to lower power consumption? If there is nothing
> running of the processor we would have a wfi loop which will never put
> the physical CPU in deep sleep. The main advantage of big.LITTLE is too
> be able to switch off a cluster/cpu when it is not used.
>
> Without any knowledge in Xen (such as CPU freq), I am afraid the the
> power consumption will still be the same.
>
>>
>> I would start from this approach, then if somebody comes along with a
>> plan to implement a big.LITTLE switcher in Xen, I welcome her to do it
>> and I would be happy to accept the code in Xen. We'll just make it
>> optional.
>
> I think we are discussing here a simple design for big.LITTLE. I never
> asked Peng to do all the work. I am worry that if we start to expose the
> big.LITTLE to the userspace it will be hard in the future to step back
> from it.

Thinking a bit more to this, after looking at Dario's mail [1], his 
suggestion could be seen as a high level use of soft/hard affinity.

I will answer to this.

>
> Regards,
>

[1] https://lists.xen.org/archives/html/xen-devel/2016-09/msg02293.html

-- 
Julien Grall

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel