Re: [RFC 0/5] xen/arm: support big.little SoC

[George Dunlap <george.dunlap@citrix.com>]

On 21/09/16 09:38, Peng Fan wrote:
> On Tue, Sep 20, 2016 at 01:17:04PM -0700, Stefano Stabellini wrote:
>> On Tue, 20 Sep 2016, Julien Grall wrote:
>>> Hi Stefano,
>>>
>>> On 20/09/2016 20:09, Stefano Stabellini wrote:
>>>> On Tue, 20 Sep 2016, Julien Grall wrote:
>>>>> Hi,
>>>>>
>>>>> On 20/09/2016 12:27, George Dunlap wrote:
>>>>>> On Tue, Sep 20, 2016 at 11:03 AM, Peng Fan <van.freenix@gmail.com>
>>>>>> wrote:
>>>>>>> On Tue, Sep 20, 2016 at 02:54:06AM +0200, Dario Faggioli wrote:
>>>>>>>> On Mon, 2016-09-19 at 17:01 -0700, Stefano Stabellini wrote:
>>>>>>>>> On Tue, 20 Sep 2016, Dario Faggioli wrote:
>>>>>>> I'd like to add a computing capability in xen/arm, like this:
>>>>>>>
>>>>>>> struct compute_capatiliby
>>>>>>> {
>>>>>>>    char *core_name;
>>>>>>>    uint32_t rank;
>>>>>>>    uint32_t cpu_partnum;
>>>>>>> };
>>>>>>>
>>>>>>> struct compute_capatiliby cc=
>>>>>>> {
>>>>>>>   {"A72", 4, 0xd08},
>>>>>>>   {"A57", 3, 0xxxx},
>>>>>>>   {"A53", 2, 0xd03},
>>>>>>>   {"A35", 1, ...},
>>>>>>> }
>>>>>>>
>>>>>>> Then when identify cpu, we decide which cpu is big and which cpu is
>>>>>>> little
>>>>>>> according to the computing rank.
>>>>>>>
>>>>>>> Any comments?
>>>>>>
>>>>>> I think we definitely need to have Xen have some kind of idea the
>>>>>> order between processors, so that the user doesn't need to figure out
>>>>>> which class / pool is big and which pool is LITTLE.  Whether this sort
>>>>>> of enumeration is the best way to do that I'll let Julien and Stefano
>>>>>> give their opinion.
>>>>>
>>>>> I don't think an hardcoded list of processor in Xen is the right solution.
>>>>> There are many existing processors and combinations for big.LITTLE so it
>>>>> will
>>>>> nearly be impossible to keep updated.
>>>>>
>>>>> I would expect the firmware table (device tree, ACPI) to provide relevant
>>>>> data
>>>>> for each processor and differentiate big from LITTLE core.
>>>>> Note that I haven't looked at it for now. A good place to start is looking
>>>>> at
>>>>> how Linux does.
>>>>
>>>> That's right, see Documentation/devicetree/bindings/arm/cpus.txt. It is
>>>> trivial to identify the two different CPU classes and which cores belong
>>>> to which class.t, as
>>>
>>> The class of the CPU can be found from the MIDR, there is no need to use the
>>> device tree/acpi for that. Note that I don't think there is an easy way in
>>> ACPI (i.e not in AML) to find out the class.
>>>
>>>> It is harder to figure out which one is supposed to be
>>>> big and which one LITTLE. Regardless, we could default to using the
>>>> first cluster (usually big), which is also the cluster of the boot cpu,
>>>> and utilize the second cluster only when the user demands it.
>>>
>>> Why do you think the boot CPU will usually be a big one? In the case of Juno
>>> platform it is configurable, and the boot CPU is a little core on r2 by
>>> default.
>>>
>>> In any case, what we care about is differentiate between two set of CPUs. I
>>> don't think Xen should care about migrating a guest vCPU between big and
>>> LITTLE cpus. So I am not sure why we would want to know that.
>>
>> No, it is not about migrating (at least yet). It is about giving useful
>> information to the user. It would be nice if the user had to choose
>> between "big" and "LITTLE" rather than "class 0x1" and "class 0x100", or
>> even "A7" or "A15".
> 
> As Dario mentioned in previous email,
> for dom0 provide like this:
> 
> dom0_vcpus_big = 4
> dom0_vcpus_little = 2
> 
> to dom0.
> 
> If these two no provided, we could let dom0 runs on big pcpus or big.little.
> Anyway this is not the important point for dom0 only big or big.little.
> 
> For domU, provide "vcpus.big" and "vcpus.little" in xl configuration file.
> Such as:
> 
> vcpus.big = 2
> vcpus.litle = 4

FWIW, from a UI perspective, it would be nice if we designed the
interface such that it *can* be used simply (i.e., just "big" or
"little"), but can also be used more flexibly; for instance, specifying
"A15" or "A7" instead.

So maybe have a 'classifier' string; this could start by having just
"big" and "little", but could then be extended to allow fuller ways of
specifying specific kinds of cores.

To keep the illusion of python syntax, what about something like this:

vcpuclass=["big=2","little=4"]

Or would it be better to have a mapping of vcpu to class?

vcpuclass=["0-1:big","2-5:little"]


> According to George's comments,
> Then, I think we could use affinity to restrict little vcpus be scheduled on little vcpus,
> and restrict big vcpus on big vcpus. Seems no need to consider soft affinity, use hard
> affinity is to handle this.
> 
> We may need to provide some interface to let xl can get the information such as
> big.little or smp. if it is big.little, which is big and which is little.

If it's possible for Xen to order the cpus by class, then there could be
a hypercall listing the different classes starting with the largest
class.  On typical big.LITTLE systems, class 0 would be "big" and class
1 would be "little".


> User may change the hard affinity of a vcpu, so we also need to block a little
> vcpu be scheduled to a big physical cpu. Add some checking code in xen,
> when chaning the hard affnity, check whether the cap of a vcpu is compatible
> with the cap of the physical cpus.

Dario, what do we do with vNUMA / soft affinity?

 -George

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel