Re: [RFC PATCH v2 1/4] vl.c: Add -smp, clusters=* command line support for ARM cpu

From: Andrew Jones <drjones@redhat.com>
To: Yanan Wang <wangyanan55@huawei.com>
Cc: Barry Song <song.bao.hua@hisilicon.com>,
	Peter Maydell <peter.maydell@linaro.org>,
	"Michael S . Tsirkin" <mst@redhat.com>,
	wanghaibin.wang@huawei.com, qemu-devel@nongnu.org,
	yangyicong@huawei.com, Shannon Zhao <shannon.zhaosl@gmail.com>,
	qemu-arm@nongnu.org, Alistair Francis <alistair.francis@wdc.com>,
	prime.zeng@hisilicon.com, Igor Mammedov <imammedo@redhat.com>,
	yuzenghui@huawei.com, Paolo Bonzini <pbonzini@redhat.com>,
	zhukeqian1@huawei.com, Jiajie Li <lijiajie11@huawei.com>,
	David Gibson <david@gibson.dropbear.id.au>
Subject: Re: [RFC PATCH v2 1/4] vl.c: Add -smp, clusters=* command line support for ARM cpu
Date: Wed, 28 Apr 2021 12:23:53 +0200	[thread overview]
Message-ID: <20210428102353.cfjvzss56c4zwb4y@gator> (raw)
In-Reply-To: <20210413083147.34236-2-wangyanan55@huawei.com>

On Tue, Apr 13, 2021 at 04:31:44PM +0800, Yanan Wang wrote:
> A cluster means a group of cores that share some resources (e.g. cache)
> among them under the LLC. For example, ARM64 server chip Kunpeng 920 has
> 6 or 8 clusters in each NUMA, and each cluster has 4 cores. All clusters
> share L3 cache data while cores within each cluster share the L2 cache.
> 
> The cache affinity of cluster has been proved to improve the Linux kernel
> scheduling performance and a patchset has been posted, in which a general
> sched_domain for clusters was added and a cluster level was added in the
> arch-neutral cpu topology struct like below.
> struct cpu_topology {
>     int thread_id;
>     int core_id;
>     int cluster_id;
>     int package_id;
>     int llc_id;
>     cpumask_t thread_sibling;
>     cpumask_t core_sibling;
>     cpumask_t cluster_sibling;
>     cpumask_t llc_sibling;
> }
> 
> Also the Kernel Doc: Documentation/devicetree/bindings/cpu/cpu-topology.txt
> defines a four-level CPU topology hierarchy like socket/cluster/core/thread.
> According to the context, a socket node's child nodes must be one or more
> cluster nodes and a cluster node's child nodes must be one or more cluster
> nodes/one or more core nodes.
> 
> So let's add the -smp, clusters=* command line support for ARM cpu, so that
> future guest os could make use of cluster cpu topology for better scheduling
> performance. For ARM machines, a four-level cpu hierarchy can be defined and
> it will be sockets/clusters/cores/threads. Because we only support clusters
> for ARM cpu currently, a new member "unsigned smp_clusters" is added to the
> VirtMachineState structure.
> 
> Signed-off-by: Yanan Wang <wangyanan55@huawei.com>
> ---
>  include/hw/arm/virt.h |  1 +
>  qemu-options.hx       | 26 +++++++++++++++-----------
>  softmmu/vl.c          |  3 +++
>  3 files changed, 19 insertions(+), 11 deletions(-)
> 
> diff --git a/include/hw/arm/virt.h b/include/hw/arm/virt.h
> index 4a4b98e4a7..5d5d156924 100644
> --- a/include/hw/arm/virt.h
> +++ b/include/hw/arm/virt.h
> @@ -155,6 +155,7 @@ struct VirtMachineState {
>      char *pciehb_nodename;
>      const int *irqmap;
>      int fdt_size;
> +    unsigned smp_clusters;
>      uint32_t clock_phandle;
>      uint32_t gic_phandle;
>      uint32_t msi_phandle;
> diff --git a/qemu-options.hx b/qemu-options.hx
> index fd21002bd6..65343ea23c 100644
> --- a/qemu-options.hx
> +++ b/qemu-options.hx
> @@ -184,25 +184,29 @@ SRST
>  ERST
>  
>  DEF("smp", HAS_ARG, QEMU_OPTION_smp,
> -    "-smp [cpus=]n[,maxcpus=cpus][,cores=cores][,threads=threads][,dies=dies][,sockets=sockets]\n"
> +    "-smp [cpus=]n[,maxcpus=cpus][,clusters=clusters][,cores=cores][,threads=threads][,dies=dies][,sockets=sockets]\n"

Please put clusters directly in front of dies in the above list, like it
is in the list description below

>      "                set the number of CPUs to 'n' [default=1]\n"
>      "                maxcpus= maximum number of total cpus, including\n"
>      "                offline CPUs for hotplug, etc\n"
> -    "                cores= number of CPU cores on one socket (for PC, it's on one die)\n"
> +    "                cores= number of CPU cores on one socket\n"
> +    "                (it's on one die for PC, and on one cluster for ARM)\n"
>      "                threads= number of threads on one CPU core\n"
> +    "                clusters= number of CPU clusters on one socket (for ARM only)\n"
>      "                dies= number of CPU dies on one socket (for PC only)\n"
>      "                sockets= number of discrete sockets in the system\n",
>          QEMU_ARCH_ALL)
>  SRST
> -``-smp [cpus=]n[,cores=cores][,threads=threads][,dies=dies][,sockets=sockets][,maxcpus=maxcpus]``
> -    Simulate an SMP system with n CPUs. On the PC target, up to 255 CPUs
> -    are supported. On Sparc32 target, Linux limits the number of usable
> -    CPUs to 4. For the PC target, the number of cores per die, the
> -    number of threads per cores, the number of dies per packages and the
> -    total number of sockets can be specified. Missing values will be
> -    computed. If any on the three values is given, the total number of
> -    CPUs n can be omitted. maxcpus specifies the maximum number of
> -    hotpluggable CPUs.
> +``-smp [cpus=]n[,maxcpus=cpus][,clusters=clusters][,cores=cores][,threads=threads][,dies=dies][,sockets=sockets]``

Also move clusters in this list over in front of dies to match the
suggested change above.

> +    Simulate an SMP system with n CPUs. On the PC target, up to 255
> +    CPUs are supported. On the Sparc32 target, Linux limits the number
> +    of usable CPUs to 4. For the PC target, the number of threads per
> +    core, the number of cores per die, the number of dies per package
> +    and the total number of sockets can be specified. For the ARM target,
> +    the number of threads per core, the number of cores per cluster, the
> +    number of clusters per socket and the total number of sockets can be
> +    specified. And missing values will be computed. If any of the five
> +    values is given, the total number of CPUs n can be omitted. Maxcpus
> +    specifies the maximum number of hotpluggable CPUs.
>  ERST
>  
>  DEF("numa", HAS_ARG, QEMU_OPTION_numa,
> diff --git a/softmmu/vl.c b/softmmu/vl.c
> index aadb526138..46f5b6b575 100644
> --- a/softmmu/vl.c
> +++ b/softmmu/vl.c
> @@ -720,6 +720,9 @@ static QemuOptsList qemu_smp_opts = {
>          }, {
>              .name = "dies",
>              .type = QEMU_OPT_NUMBER,
> +        }, {
> +            .name = "clusters",
> +            .type = QEMU_OPT_NUMBER,
>          }, {
>              .name = "cores",
>              .type = QEMU_OPT_NUMBER,
> -- 
> 2.19.1
>

Thanks,
drew