From: Wei Chen <Wei.Chen@arm.com>
To: Stefano Stabellini <sstabellini@kernel.org>
Cc: "xen-devel@lists.xenproject.org" <xen-devel@lists.xenproject.org>,
"julien@xen.org" <julien@xen.org>,
Bertrand Marquis <Bertrand.Marquis@arm.com>
Subject: RE: [PATCH 23/37] xen/arm: implement node distance helpers for Arm
Date: Fri, 24 Sep 2021 04:41:39 +0000 [thread overview]
Message-ID: <DB9PR08MB68571BAA4C8B07BB2061C8579EA49@DB9PR08MB6857.eurprd08.prod.outlook.com> (raw)
In-Reply-To: <alpine.DEB.2.21.2109231843530.17979@sstabellini-ThinkPad-T480s>
Hi Stefano,
> -----Original Message-----
> From: Stefano Stabellini <sstabellini@kernel.org>
> Sent: 2021年9月24日 9:47
> To: Wei Chen <Wei.Chen@arm.com>
> Cc: xen-devel@lists.xenproject.org; sstabellini@kernel.org; julien@xen.org;
> Bertrand Marquis <Bertrand.Marquis@arm.com>
> Subject: Re: [PATCH 23/37] xen/arm: implement node distance helpers for
> Arm
>
> On Thu, 23 Sep 2021, Wei Chen wrote:
> > We will parse NUMA nodes distances from device tree or ACPI
> > table. So we need a matrix to record the distances between
> > any two nodes we parsed. Accordingly, we provide this
> > node_set_distance API for device tree or ACPI table parsers
> > to set the distance for any two nodes in this patch.
> > When NUMA initialization failed, __node_distance will return
> > NUMA_REMOTE_DISTANCE, this will help us avoid doing rollback
> > for distance maxtrix when NUMA initialization failed.
> >
> > Signed-off-by: Wei Chen <wei.chen@arm.com>
> > ---
> > xen/arch/arm/Makefile | 1 +
> > xen/arch/arm/numa.c | 69 ++++++++++++++++++++++++++++++++++++++
> > xen/include/asm-arm/numa.h | 13 +++++++
> > 3 files changed, 83 insertions(+)
> > create mode 100644 xen/arch/arm/numa.c
> >
> > diff --git a/xen/arch/arm/Makefile b/xen/arch/arm/Makefile
> > index ae4efbf76e..41ca311b6b 100644
> > --- a/xen/arch/arm/Makefile
> > +++ b/xen/arch/arm/Makefile
> > @@ -35,6 +35,7 @@ obj-$(CONFIG_LIVEPATCH) += livepatch.o
> > obj-y += mem_access.o
> > obj-y += mm.o
> > obj-y += monitor.o
> > +obj-$(CONFIG_NUMA) += numa.o
> > obj-y += p2m.o
> > obj-y += percpu.o
> > obj-y += platform.o
> > diff --git a/xen/arch/arm/numa.c b/xen/arch/arm/numa.c
> > new file mode 100644
> > index 0000000000..3f08870d69
> > --- /dev/null
> > +++ b/xen/arch/arm/numa.c
> > @@ -0,0 +1,69 @@
> > +// SPDX-License-Identifier: GPL-2.0
> > +/*
> > + * Arm Architecture support layer for NUMA.
> > + *
> > + * Copyright (C) 2021 Arm Ltd
> > + *
> > + * This program is free software; you can redistribute it and/or modify
> > + * it under the terms of the GNU General Public License version 2 as
> > + * published by the Free Software Foundation.
> > + *
> > + * This program is distributed in the hope that it will be useful,
> > + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> > + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > + * GNU General Public License for more details.
> > + *
> > + * You should have received a copy of the GNU General Public License
> > + * along with this program. If not, see <http://www.gnu.org/licenses/>.
> > + *
> > + */
> > +#include <xen/init.h>
> > +#include <xen/numa.h>
> > +
> > +static uint8_t __read_mostly
> > +node_distance_map[MAX_NUMNODES][MAX_NUMNODES] = {
> > + { 0 }
> > +};
> > +
> > +void __init numa_set_distance(nodeid_t from, nodeid_t to, uint32_t
> distance)
> > +{
> > + if ( from >= MAX_NUMNODES || to >= MAX_NUMNODES )
> > + {
> > + printk(KERN_WARNING
> > + "NUMA: invalid nodes: from=%"PRIu8" to=%"PRIu8"
> MAX=%"PRIu8"\n",
> > + from, to, MAX_NUMNODES);
> > + return;
> > + }
> > +
> > + /* NUMA defines 0xff as an unreachable node and 0-9 are undefined
> */
> > + if ( distance >= NUMA_NO_DISTANCE ||
> > + (distance >= NUMA_DISTANCE_UDF_MIN &&
> > + distance <= NUMA_DISTANCE_UDF_MAX) ||
> > + (from == to && distance != NUMA_LOCAL_DISTANCE) )
> > + {
> > + printk(KERN_WARNING
> > + "NUMA: invalid distance: from=%"PRIu8" to=%"PRIu8"
> distance=%"PRIu32"\n",
> > + from, to, distance);
> > + return;
> > + }
> > +
> > + node_distance_map[from][to] = distance;
> > +}
> > +
> > +uint8_t __node_distance(nodeid_t from, nodeid_t to)
> > +{
> > + /* When NUMA is off, any distance will be treated as remote. */
> > + if ( srat_disabled() )
>
> Given that this is ARM specific code and specific to ACPI, I don't think
> we should have any call to something called "srat_disabled".
>
> I suggest to either rename srat_disabled to numa_distance_disabled.
>
> Other than that, this patch looks OK to me.
>
srat stands for static resource affinity table, I think dtb also can be
treated as a static resource affinity table. So I keep SRAT in this patch
and other patches. I have seen your comment in patch#25. Before x86 maintainers
give any feedback, can we still keep srat here?
>
> > + return NUMA_REMOTE_DISTANCE;
> > +
> > + /*
> > + * Check whether the nodes are in the matrix range.
> > + * When any node is out of range, except from and to nodes are the
> > + * same, we treat them as unreachable (return 0xFF)
> > + */
> > + if ( from >= MAX_NUMNODES || to >= MAX_NUMNODES )
> > + return from == to ? NUMA_LOCAL_DISTANCE : NUMA_NO_DISTANCE;
> > +
> > + return node_distance_map[from][to];
> > +}
> > +EXPORT_SYMBOL(__node_distance);
> > diff --git a/xen/include/asm-arm/numa.h b/xen/include/asm-arm/numa.h
> > index 21569e634b..758eafeb05 100644
> > --- a/xen/include/asm-arm/numa.h
> > +++ b/xen/include/asm-arm/numa.h
> > @@ -9,8 +9,21 @@ typedef u8 nodeid_t;
> >
> > #ifdef CONFIG_NUMA
> >
> > +/*
> > + * In ACPI spec, 0-9 are the reserved values for node distance,
> > + * 10 indicates local node distance, 20 indicates remote node
> > + * distance. Set node distance map in device tree will follow
> > + * the ACPI's definition.
> > + */
> > +#define NUMA_DISTANCE_UDF_MIN 0
> > +#define NUMA_DISTANCE_UDF_MAX 9
> > +#define NUMA_LOCAL_DISTANCE 10
> > +#define NUMA_REMOTE_DISTANCE 20
> > +
> > #define NR_NODE_MEMBLKS NR_MEM_BANKS
> >
> > +extern void numa_set_distance(nodeid_t from, nodeid_t to, uint32_t
> distance);
> > +
> > #else
> >
> > /* Fake one node for now. See also node_online_map. */
> > --
> > 2.25.1
> >
next prev parent reply other threads:[~2021-09-24 4:42 UTC|newest]
Thread overview: 192+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-09-23 12:01 [PATCH 00/37] Add device tree based NUMA support to Arm Wei Chen
2021-09-23 12:02 ` [PATCH 01/37] xen/arm: Print a 64-bit number in hex from early uart Wei Chen
2021-09-23 12:02 ` [PATCH 02/37] xen: introduce a Kconfig option to configure NUMA nodes number Wei Chen
2021-09-23 23:45 ` Stefano Stabellini
2021-09-24 1:24 ` Wei Chen
2021-09-24 8:55 ` Jan Beulich
2021-09-24 10:33 ` Wei Chen
2021-09-24 10:47 ` Jan Beulich
2021-09-23 12:02 ` [PATCH 03/37] xen/x86: Initialize memnodemapsize while faking NUMA node Wei Chen
2021-09-24 8:57 ` Jan Beulich
2021-09-24 10:34 ` Wei Chen
2021-09-23 12:02 ` [PATCH 04/37] xen: introduce an arch helper for default dma zone status Wei Chen
2021-09-23 23:55 ` Stefano Stabellini
2021-09-24 1:50 ` Wei Chen
2022-01-17 16:10 ` Jan Beulich
2022-01-18 7:51 ` Wei Chen
2022-01-18 8:16 ` Jan Beulich
2022-01-18 9:20 ` Wei Chen
2022-01-18 14:16 ` Jan Beulich
2022-01-19 2:49 ` Wei Chen
2022-01-19 7:50 ` Jan Beulich
2022-01-19 8:33 ` Wei Chen
2021-09-23 12:02 ` [PATCH 05/37] xen: decouple NUMA from ACPI in Kconfig Wei Chen
2021-09-23 12:02 ` [PATCH 06/37] xen/arm: use !CONFIG_NUMA to keep fake NUMA API Wei Chen
2021-09-24 0:05 ` Stefano Stabellini
2021-09-24 10:21 ` Wei Chen
2021-09-23 12:02 ` [PATCH 07/37] xen/x86: use paddr_t for addresses in NUMA node structure Wei Chen
2021-09-24 0:11 ` Stefano Stabellini
2021-09-24 0:13 ` Stefano Stabellini
2021-09-24 3:00 ` Wei Chen
2022-01-18 15:22 ` Jan Beulich
2022-01-19 6:33 ` Wei Chen
2022-01-19 7:55 ` Jan Beulich
2022-01-19 8:36 ` Wei Chen
2021-09-23 12:02 ` [PATCH 08/37] xen/x86: add detection of discontinous node memory range Wei Chen
2021-09-24 0:25 ` Stefano Stabellini
2021-09-24 4:28 ` Wei Chen
2021-09-24 19:52 ` Stefano Stabellini
2021-09-26 10:11 ` Wei Chen
2021-09-27 3:13 ` Stefano Stabellini
2021-09-27 5:05 ` Stefano Stabellini
2021-09-27 9:50 ` Wei Chen
2021-09-27 17:19 ` Stefano Stabellini
2021-09-28 4:41 ` Wei Chen
2021-09-28 4:59 ` Stefano Stabellini
2022-01-18 16:13 ` Jan Beulich
2022-01-19 7:33 ` Wei Chen
2022-01-19 8:01 ` Jan Beulich
2022-01-19 8:24 ` Wei Chen
2021-09-23 12:02 ` [PATCH 09/37] xen/x86: introduce two helpers to access memory hotplug end Wei Chen
2021-09-24 0:29 ` Stefano Stabellini
2021-09-24 4:21 ` Wei Chen
2022-01-24 16:24 ` Jan Beulich
2022-01-26 7:53 ` Wei Chen
2021-09-23 12:02 ` [PATCH 10/37] xen/x86: use helpers to access/update mem_hotplug Wei Chen
2021-09-24 0:31 ` Stefano Stabellini
2021-09-24 4:29 ` Wei Chen
2022-01-24 16:29 ` Jan Beulich
2022-01-26 7:58 ` Wei Chen
2021-09-23 12:02 ` [PATCH 11/37] xen/x86: abstract neutral code from acpi_numa_memory_affinity_init Wei Chen
2021-09-24 0:38 ` Stefano Stabellini
2022-01-24 16:50 ` Jan Beulich
2022-01-26 10:39 ` Wei Chen
2021-09-23 12:02 ` [PATCH 12/37] xen/x86: decouple nodes_cover_memory from E820 map Wei Chen
2021-09-24 0:39 ` Stefano Stabellini
2022-01-24 16:59 ` Jan Beulich
2022-01-27 8:03 ` Wei Chen
2022-01-27 8:08 ` Jan Beulich
2022-01-27 9:03 ` Wei Chen
2022-01-27 9:22 ` Jan Beulich
2022-01-27 9:27 ` Wei Chen
2021-09-23 12:02 ` [PATCH 13/37] xen/x86: decouple processor_nodes_parsed from acpi numa functions Wei Chen
2021-09-24 0:40 ` Stefano Stabellini
2022-01-25 9:49 ` Jan Beulich
2022-01-27 8:06 ` Wei Chen
2021-09-23 12:02 ` [PATCH 14/37] xen/x86: use name fw_numa to replace acpi_numa Wei Chen
2021-09-24 0:40 ` Stefano Stabellini
2022-01-25 10:12 ` Jan Beulich
2022-01-27 8:09 ` Wei Chen
2021-09-23 12:02 ` [PATCH 15/37] xen/x86: rename acpi_scan_nodes to numa_scan_nodes Wei Chen
2021-09-24 0:40 ` Stefano Stabellini
2022-01-25 10:17 ` Jan Beulich
2022-01-27 8:14 ` Wei Chen
2021-09-23 12:02 ` [PATCH 16/37] xen/x86: export srat_bad to external Wei Chen
2021-09-24 0:41 ` Stefano Stabellini
2022-01-25 10:22 ` Jan Beulich
2022-01-27 8:35 ` Wei Chen
2022-01-27 8:37 ` Jan Beulich
2022-01-27 8:47 ` Wei Chen
2021-09-23 12:02 ` [PATCH 17/37] xen/x86: use CONFIG_NUMA to gate numa_scan_nodes Wei Chen
2021-09-24 0:41 ` Stefano Stabellini
2022-01-25 10:26 ` Jan Beulich
2022-01-27 8:37 ` Wei Chen
2021-09-23 12:02 ` [PATCH 18/37] xen: move NUMA common code from x86 to common Wei Chen
2021-09-23 12:02 ` [PATCH 19/37] xen/x86: promote VIRTUAL_BUG_ON to ASSERT in Wei Chen
2022-01-17 16:21 ` Jan Beulich
2022-01-18 7:52 ` Wei Chen
2021-09-23 12:02 ` [PATCH 20/37] xen: introduce CONFIG_EFI to stub API for non-EFI architecture Wei Chen
2021-09-24 1:15 ` Stefano Stabellini
2021-09-24 4:34 ` Wei Chen
2021-09-24 7:58 ` Jan Beulich
2021-09-24 10:31 ` Wei Chen
2021-09-24 10:49 ` Jan Beulich
2021-09-26 10:25 ` Wei Chen
2021-09-27 10:28 ` Wei Chen
2021-09-28 0:59 ` Stefano Stabellini
2021-09-28 4:16 ` Wei Chen
2021-09-28 5:01 ` Stefano Stabellini
2021-09-28 8:02 ` Jan Beulich
2021-10-03 23:28 ` Wei Chen
2022-01-25 10:34 ` Jan Beulich
2022-01-27 8:44 ` Wei Chen
2022-01-27 8:51 ` Wei Chen
2022-01-27 9:00 ` Jan Beulich
2022-01-27 9:09 ` Wei Chen
2022-01-27 9:16 ` Jan Beulich
2022-01-27 9:25 ` Wei Chen
2022-01-27 9:27 ` Jan Beulich
2022-01-27 10:00 ` Julien Grall
2022-01-28 4:35 ` Wei Chen
2021-09-23 12:02 ` [PATCH 21/37] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI Wei Chen
2021-09-24 1:23 ` Stefano Stabellini
2021-09-24 4:36 ` Wei Chen
2022-01-25 10:38 ` Jan Beulich
2022-01-27 8:45 ` Wei Chen
2021-09-23 12:02 ` [PATCH 22/37] xen/arm: use NR_MEM_BANKS to override default NR_NODE_MEMBLKS Wei Chen
2021-09-24 1:34 ` Stefano Stabellini
2021-09-26 13:13 ` Wei Chen
2021-09-27 3:25 ` Stefano Stabellini
2021-09-27 4:18 ` Wei Chen
2021-09-27 4:59 ` Stefano Stabellini
2021-09-27 6:25 ` Julien Grall
2021-09-27 6:46 ` Wei Chen
2021-09-27 6:53 ` Wei Chen
2021-09-27 7:35 ` Julien Grall
2021-09-27 10:21 ` Wei Chen
2021-09-27 10:39 ` Julien Grall
2021-09-27 16:58 ` Stefano Stabellini
2021-09-28 2:57 ` Wei Chen
2021-09-23 12:02 ` [PATCH 23/37] xen/arm: implement node distance helpers for Arm Wei Chen
2021-09-24 1:46 ` Stefano Stabellini
2021-09-24 4:41 ` Wei Chen [this message]
2021-09-24 19:36 ` Stefano Stabellini
2021-09-26 10:15 ` Wei Chen
2021-09-23 12:02 ` [PATCH 24/37] xen/arm: implement two arch helpers to get memory map info Wei Chen
2021-09-24 2:06 ` Stefano Stabellini
2021-09-24 4:42 ` Wei Chen
2021-09-23 12:02 ` [PATCH 25/37] xen/arm: implement bad_srat for Arm NUMA initialization Wei Chen
2021-09-24 2:09 ` Stefano Stabellini
2021-09-24 4:45 ` Wei Chen
2021-09-24 8:07 ` Jan Beulich
2021-09-24 19:33 ` Stefano Stabellini
2021-09-23 12:02 ` [PATCH 26/37] xen/arm: build NUMA cpu_to_node map in dt_smp_init_cpus Wei Chen
2021-09-24 2:26 ` Stefano Stabellini
2021-09-24 4:25 ` Wei Chen
2021-09-23 12:02 ` [PATCH 27/37] xen/arm: Add boot and secondary CPU to NUMA system Wei Chen
2021-09-23 12:02 ` [PATCH 28/37] xen/arm: stub memory hotplug access helpers for Arm Wei Chen
2021-09-24 2:33 ` Stefano Stabellini
2021-09-24 4:26 ` Wei Chen
2021-09-23 12:02 ` [PATCH 29/37] xen/arm: introduce a helper to parse device tree processor node Wei Chen
2021-09-24 2:44 ` Stefano Stabellini
2021-09-24 4:46 ` Wei Chen
2021-09-23 12:02 ` [PATCH 30/37] xen/arm: introduce a helper to parse device tree memory node Wei Chen
2021-09-24 3:05 ` Stefano Stabellini
2021-09-24 7:54 ` Wei Chen
2021-09-23 12:02 ` [PATCH 31/37] xen/arm: introduce a helper to parse device tree NUMA distance map Wei Chen
2021-09-24 3:05 ` Stefano Stabellini
2021-09-24 5:23 ` Wei Chen
2021-09-23 12:02 ` [PATCH 32/37] xen/arm: unified entry to parse all NUMA data from device tree Wei Chen
2021-09-24 3:16 ` Stefano Stabellini
2021-09-24 7:58 ` Wei Chen
2021-09-24 19:42 ` Stefano Stabellini
2021-09-23 12:02 ` [PATCH 33/37] xen/arm: keep guest still be NUMA unware Wei Chen
2021-09-24 3:19 ` Stefano Stabellini
2021-09-24 10:23 ` Wei Chen
2021-09-23 12:02 ` [PATCH 34/37] xen/arm: enable device tree based NUMA in system init Wei Chen
2021-09-24 3:28 ` Stefano Stabellini
2021-09-24 9:52 ` Wei Chen
2021-09-23 12:02 ` [PATCH 35/37] xen/arm: use CONFIG_NUMA to gate node_online_map in smpboot Wei Chen
2021-09-23 12:02 ` [PATCH 36/37] xen/arm: Provide Kconfig options for Arm to enable NUMA Wei Chen
2021-09-24 3:31 ` Stefano Stabellini
2021-09-24 10:13 ` Wei Chen
2021-09-24 19:39 ` Stefano Stabellini
2021-09-27 8:33 ` Jan Beulich
2021-09-27 8:45 ` Julien Grall
2021-09-27 9:17 ` Jan Beulich
2021-09-27 17:17 ` Stefano Stabellini
2021-09-28 2:59 ` Wei Chen
2021-09-28 3:30 ` Stefano Stabellini
2021-09-24 10:25 ` Jan Beulich
2021-09-24 10:37 ` Wei Chen
2021-09-23 12:02 ` [PATCH 37/37] docs: update numa command line to support Arm Wei Chen
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=DB9PR08MB68571BAA4C8B07BB2061C8579EA49@DB9PR08MB6857.eurprd08.prod.outlook.com \
--to=wei.chen@arm.com \
--cc=Bertrand.Marquis@arm.com \
--cc=julien@xen.org \
--cc=sstabellini@kernel.org \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).