Re: [RFC PATCH 0/2] Boot time cpupools

From: Luca Fancellu <luca.fancellu@arm.com>
To: Juergen Gross <jgross@suse.com>, Julien Grall <julien@xen.org>,
	Stefano Stabellini <sstabellini@kernel.org>,
	Oleksandr Tyshchenko <oleksandr_tyshchenko@epam.com>
Cc: Bertrand Marquis <Bertrand.Marquis@arm.com>,
	Xen-devel <xen-devel@lists.xenproject.org>,
	Wei Chen <Wei.Chen@arm.com>,
	Volodymyr Babchuk <Volodymyr_Babchuk@epam.com>,
	George Dunlap <george.dunlap@citrix.com>,
	Dario Faggioli <dfaggioli@suse.com>
Subject: Re: [RFC PATCH 0/2] Boot time cpupools
Date: Tue, 7 Dec 2021 09:27:43 +0000	[thread overview]
Message-ID: <662C34A2-CCCE-49CC-BA5B-8CC5E24E5DF3@arm.com> (raw)
In-Reply-To: <59e14393-a1fc-5b82-2f6e-5567f218cb3a@suse.com>

Hi all,

Thank you for all your feedbacks, sorry for the late response. Given the amount of suggestions I’ve been working
on a proposal for the boot time cpupools that I hope could be good for everyone.

The feature will be enabled by CONFIG_BOOT_TIME_CPUPOOLS, so without it everything is behaving as now.

When the feature is enabled, the code will check the device tree and use informations from there to create the cpupools:

a72_1: cpu@0 {
        compatible = "arm,cortex-a72";
        reg = <0x0 0x0>;
        device_type = "cpu";
        [...]
};

a72_2: cpu@1 {
        compatible = "arm,cortex-a72";
        reg = <0x0 0x1>;
        device_type = "cpu";
        [...]
};

cpu@2 {
        compatible = "arm,cortex-a72";
        reg = <0x0 0x2>;
        device_type = "cpu";
        [...]
};

a53_1: cpu@100 {
        compatible = "arm,cortex-a53";
        reg = <0x0 0x100>;
        device_type = "cpu";
        [...]
};

a53_2: cpu@101 {
        compatible = "arm,cortex-a53";
        reg = <0x0 0x101>;
        device_type = "cpu";
        [...]
};

chosen {

    cpupool_a {
        compatible = "xen,cpupool";
        xen,cpupool-id = <0>;
        xen,cpupool-cpus = <&a72_1 &a72_2>;     
    };
    cpupool_b {
        compatible = "xen,cpupool";
        xen,cpupool-id = <1>;
        xen,cpupool-cpus = <&a53_1 &a53_2>;
        xen,cpupool-sched = "credit2";
    };

   […]

};

So for every node under chosen with the compatible “xen,cpupool”, a cpupool is created (if it doesn’t exists).

Mandatory properties of that node are: 
 - “xen,cpupool-id” which identifies the id of the pool
 - “xen,cpupool-cpus” which lists the handle of the cpus

Optional property is “xen,cpupool-sched” which is a string that identifies the scheduler. A cpupool with identifier
0 (zero) can’t have that property, it will get the default scheduler from Xen.

A set of rules are applied:

  1) The cpupool with id 0 is always created, being it listed or not in the DT
  2) The cpupool with id 0 must have at least one cpu, if it doesn’t the system will panic.
  3) Every cpu that is not assigned to any cpupool will be automatically assigned to the cpupool with id 0 
      (only cpu that are brought up by Xen)
  4) When a cpu is assigned to a cpupool in the DT, but the cpu is not up, the system will panic.

So, given this explanation, the above example will create a system with two cpupool:

 - cpupool with id 0 containing 3 cpu a72 (two are explicitly listed, one was not assigned to any other cpupool)
 - cpupool with id 1 containing 2 cpu a53 (cpus explicitly listed)

Clearly the above example works only if Xen is started using the hmp-unsafe=1 parameter, otherwise some cpus
won’t be started.

Given the above example, we might be able to have an option like this (“xen,domain-cpupool-id”) to assign
dom0less guests to cpupools:

chosen {

    cpupool_a {
        compatible = "xen,cpupool";
        xen,cpupool-id = <0>;
        xen,cpupool-cpus = <&a72_1 &a72_2>;     
    };
    cpupool_b {
        compatible = "xen,cpupool";
        xen,cpupool-id = <1>;
        xen,cpupool-cpus = <&a53_1 &a53_2>;
        xen,cpupool-sched = "credit2";
    };

    domU1 {
        #size-cells = <0x1>;
        #address-cells = <0x1>;
        compatible = "xen,domain";
        cpus = <0x1>;
        memory = <0x0 0xc0000>;
        xen,domain-cpupool-id = <1>;            /* Optional */
        vpl011;

        module@0 {
            compatible = "multiboot,kernel", "multiboot,module";
            […]
        };
    };

};

Any thoughts on this?

Cheers,
Luca