* [PATCH v2 0/2] Specify empty NUMA node
@ 2021-09-27 6:41 Gavin Shan
2021-09-27 6:41 ` [PATCH v2 1/2] Documentation, dt, numa: Add note to " Gavin Shan
2021-09-27 6:41 ` [PATCH v2 2/2] of, numa: Fetch empty NUMA node ID from distance map Gavin Shan
0 siblings, 2 replies; 9+ messages in thread
From: Gavin Shan @ 2021-09-27 6:41 UTC (permalink / raw)
To: devicetree
Cc: linux-kernel, linux-efi, linux-arm-kernel, kvmarm, robh, maz,
rdunlap, catalin.marinas, will, shan.gavin
There are multiple empty nodes, where no memory resides in, are allowed
in virtual machine (VM) backed by QEMU. I don't find anywhere to document
how the corresponding device nodes are populated. This series adds one
section in Documentation/devicetree/bindings/numa.txt to document how
the empty NUMA nodes are handled: no device nodes are existing for these
empty NUMA nodes, but the NUMA IDs and distance map are still valid in
"numa-distance-map-v1" compatible device node.
PATCH[1] adds one section in Documentation/devicetree/bindings/numa.txt
to specify how device nodes are populated for these empty NUMA
nodes.
PATCH[2] fetches NUMA IDs and distance maps from "numa-distance-map-v1"
compatible device node.
Changelog
=========
* No device nodes for empty NUMA nodes (Rob)
* Add patch to fetch NUMA IDs and distance map from the
"numa-distance-map-v1" compatible device node (Rob)
Gavin Shan (2):
Documentation, dt, numa: Add note to empty NUMA node
of, numa: Fetch empty NUMA node ID from distance map
Documentation/devicetree/bindings/numa.txt | 46 +++++++++++++++++++++-
drivers/of/of_numa.c | 2 +
2 files changed, 47 insertions(+), 1 deletion(-)
--
2.23.0
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH v2 1/2] Documentation, dt, numa: Add note to empty NUMA node 2021-09-27 6:41 [PATCH v2 0/2] Specify empty NUMA node Gavin Shan @ 2021-09-27 6:41 ` Gavin Shan 2021-10-04 18:13 ` Rob Herring 2021-09-27 6:41 ` [PATCH v2 2/2] of, numa: Fetch empty NUMA node ID from distance map Gavin Shan 1 sibling, 1 reply; 9+ messages in thread From: Gavin Shan @ 2021-09-27 6:41 UTC (permalink / raw) To: devicetree Cc: linux-kernel, linux-efi, linux-arm-kernel, kvmarm, robh, maz, rdunlap, catalin.marinas, will, shan.gavin The empty memory nodes, where no memory resides in, are allowed. The NUMA node IDs are still valid and parsed, but memory may be added to them through hotplug afterwards. Currently, QEMU fails to boot when multiple empty memory nodes are specified. It's caused by device-tree population failure and duplicated memory node names. The device-tree specification doesn't provide how empty NUMA nodes are handled. Besides, I finds difficulty to get where this case is documented. So lets add a section for empty memory nodes to cover it in NUMA binding document. Signed-off-by: Gavin Shan <gshan@redhat.com> --- Documentation/devicetree/bindings/numa.txt | 46 +++++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/Documentation/devicetree/bindings/numa.txt b/Documentation/devicetree/bindings/numa.txt index 21b35053ca5a..42f282c2f3cc 100644 --- a/Documentation/devicetree/bindings/numa.txt +++ b/Documentation/devicetree/bindings/numa.txt @@ -103,7 +103,51 @@ Example: }; ============================================================================== -4 - Example dts +4 - Empty memory nodes +============================================================================== + +Empty memory nodes, which no memory resides in, are allowed. There are no +device nodes for these empty memory nodes. However, the NUMA node IDs and +distance maps are still valid and memory may be added into them through +hotplug afterwards. + +Example: + + memory@0 { + device_type = "memory"; + reg = <0x0 0x0 0x0 0x80000000>; + numa-node-id = <0>; + }; + + memory@80000000 { + device_type = "memory"; + reg = <0x0 0x80000000 0x0 0x80000000>; + numa-node-id = <1>; + }; + + /* Empty memory node 2 and 3 */ + distance-map { + compatible = "numa-distance-map-v1"; + distance-matrix = <0 0 10>, + <0 1 20>, + <0 2 40>, + <0 3 20>, + <1 0 20>, + <1 1 10>, + <1 2 20>, + <1 3 40>, + <2 0 40>, + <2 1 20>, + <2 2 10>, + <2 3 20>, + <3 0 20>, + <3 1 40>, + <3 2 20>, + <3 3 10>; + }; + +============================================================================== +5 - Example dts ============================================================================== Dual socket system consists of 2 boards connected through ccn bus and -- 2.23.0 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH v2 1/2] Documentation, dt, numa: Add note to empty NUMA node 2021-09-27 6:41 ` [PATCH v2 1/2] Documentation, dt, numa: Add note to " Gavin Shan @ 2021-10-04 18:13 ` Rob Herring 0 siblings, 0 replies; 9+ messages in thread From: Rob Herring @ 2021-10-04 18:13 UTC (permalink / raw) To: Gavin Shan Cc: devicetree, catalin.marinas, shan.gavin, will, rdunlap, maz, linux-efi, kvmarm, linux-arm-kernel, linux-kernel On Mon, 27 Sep 2021 14:41:18 +0800, Gavin Shan wrote: > The empty memory nodes, where no memory resides in, are allowed. > The NUMA node IDs are still valid and parsed, but memory may be > added to them through hotplug afterwards. Currently, QEMU fails > to boot when multiple empty memory nodes are specified. It's > caused by device-tree population failure and duplicated memory > node names. > > The device-tree specification doesn't provide how empty NUMA > nodes are handled. Besides, I finds difficulty to get where > this case is documented. So lets add a section for empty memory > nodes to cover it in NUMA binding document. > > Signed-off-by: Gavin Shan <gshan@redhat.com> > --- > Documentation/devicetree/bindings/numa.txt | 46 +++++++++++++++++++++- > 1 file changed, 45 insertions(+), 1 deletion(-) > Applied, thanks! ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH v2 2/2] of, numa: Fetch empty NUMA node ID from distance map 2021-09-27 6:41 [PATCH v2 0/2] Specify empty NUMA node Gavin Shan 2021-09-27 6:41 ` [PATCH v2 1/2] Documentation, dt, numa: Add note to " Gavin Shan @ 2021-09-27 6:41 ` Gavin Shan 2021-09-27 14:49 ` Rob Herring 2021-10-04 18:13 ` Rob Herring 1 sibling, 2 replies; 9+ messages in thread From: Gavin Shan @ 2021-09-27 6:41 UTC (permalink / raw) To: devicetree Cc: linux-kernel, linux-efi, linux-arm-kernel, kvmarm, robh, maz, rdunlap, catalin.marinas, will, shan.gavin There is no device node for the empty NUMA node. However, the corresponding NUMA node ID and distance map is still valid in "numa-distance-map-v1" compatible device node. This fetches the NUMA node ID and distance map for these empty NUMA node from "numa-distance-map-v1" compatible device node. Signed-off-by: Gavin Shan <gshan@redhat.com> --- drivers/of/of_numa.c | 2 ++ 1 file changed, 2 insertions(+) diff --git a/drivers/of/of_numa.c b/drivers/of/of_numa.c index fe6b13608e51..5949829a1b00 100644 --- a/drivers/of/of_numa.c +++ b/drivers/of/of_numa.c @@ -111,6 +111,8 @@ static int __init of_numa_parse_distance_map_v1(struct device_node *map) return -EINVAL; } + node_set(nodea, numa_nodes_parsed); + numa_set_distance(nodea, nodeb, distance); /* Set default distance of node B->A same as A->B */ -- 2.23.0 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH v2 2/2] of, numa: Fetch empty NUMA node ID from distance map 2021-09-27 6:41 ` [PATCH v2 2/2] of, numa: Fetch empty NUMA node ID from distance map Gavin Shan @ 2021-09-27 14:49 ` Rob Herring 2021-09-27 23:59 ` Gavin Shan 2021-10-04 18:13 ` Rob Herring 1 sibling, 1 reply; 9+ messages in thread From: Rob Herring @ 2021-09-27 14:49 UTC (permalink / raw) To: Gavin Shan Cc: devicetree, linux-kernel, linux-efi, linux-arm-kernel, open list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64), Marc Zyngier, Randy Dunlap, Catalin Marinas, Will Deacon, shan.gavin On Mon, Sep 27, 2021 at 1:42 AM Gavin Shan <gshan@redhat.com> wrote: > > There is no device node for the empty NUMA node. However, the > corresponding NUMA node ID and distance map is still valid in > "numa-distance-map-v1" compatible device node. > > This fetches the NUMA node ID and distance map for these empty > NUMA node from "numa-distance-map-v1" compatible device node. This is much nicer. > Signed-off-by: Gavin Shan <gshan@redhat.com> > --- > drivers/of/of_numa.c | 2 ++ > 1 file changed, 2 insertions(+) > > diff --git a/drivers/of/of_numa.c b/drivers/of/of_numa.c > index fe6b13608e51..5949829a1b00 100644 > --- a/drivers/of/of_numa.c > +++ b/drivers/of/of_numa.c > @@ -111,6 +111,8 @@ static int __init of_numa_parse_distance_map_v1(struct device_node *map) > return -EINVAL; > } > > + node_set(nodea, numa_nodes_parsed); > + With this, couldn't we remove of_numa_parse_cpu_nodes() as the only thing it does is node_set()? > numa_set_distance(nodea, nodeb, distance); > > /* Set default distance of node B->A same as A->B */ > -- > 2.23.0 > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v2 2/2] of, numa: Fetch empty NUMA node ID from distance map 2021-09-27 14:49 ` Rob Herring @ 2021-09-27 23:59 ` Gavin Shan 2021-09-28 17:22 ` Rob Herring 0 siblings, 1 reply; 9+ messages in thread From: Gavin Shan @ 2021-09-27 23:59 UTC (permalink / raw) To: Rob Herring Cc: devicetree, linux-kernel, linux-efi, linux-arm-kernel, open list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64), Marc Zyngier, Randy Dunlap, Catalin Marinas, Will Deacon, shan.gavin Hi Rob, On 9/28/21 12:49 AM, Rob Herring wrote: > On Mon, Sep 27, 2021 at 1:42 AM Gavin Shan <gshan@redhat.com> wrote: >> >> There is no device node for the empty NUMA node. However, the >> corresponding NUMA node ID and distance map is still valid in >> "numa-distance-map-v1" compatible device node. >> >> This fetches the NUMA node ID and distance map for these empty >> NUMA node from "numa-distance-map-v1" compatible device node. > > This is much nicer. > Indeed, thanks for your suggestions :) >> Signed-off-by: Gavin Shan <gshan@redhat.com> >> --- >> drivers/of/of_numa.c | 2 ++ >> 1 file changed, 2 insertions(+) >> >> diff --git a/drivers/of/of_numa.c b/drivers/of/of_numa.c >> index fe6b13608e51..5949829a1b00 100644 >> --- a/drivers/of/of_numa.c >> +++ b/drivers/of/of_numa.c >> @@ -111,6 +111,8 @@ static int __init of_numa_parse_distance_map_v1(struct device_node *map) >> return -EINVAL; >> } >> >> + node_set(nodea, numa_nodes_parsed); >> + > > With this, couldn't we remove of_numa_parse_cpu_nodes() as the only > thing it does is node_set()? > I don't think so for couple of reasons: (1) With problematic device-tree, the distance map node might be missed or incomplete. In this case, of_numa_parse_cpu_nodes() still helps. (2) @numa_nodes_parsed is also updated when the memory nodes are iterated in of_numa_parse_memory_nodes() and numa_add_memblk(). So @numa_nodes_parsed, which is synchronized to @node_possible_map afterwards, is the gathering output of CPU nodes, memory nodes and distance map node. >> numa_set_distance(nodea, nodeb, distance); >> >> /* Set default distance of node B->A same as A->B */ Thanks, Gavin ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v2 2/2] of, numa: Fetch empty NUMA node ID from distance map 2021-09-27 23:59 ` Gavin Shan @ 2021-09-28 17:22 ` Rob Herring 2021-09-29 2:00 ` Gavin Shan 0 siblings, 1 reply; 9+ messages in thread From: Rob Herring @ 2021-09-28 17:22 UTC (permalink / raw) To: Gavin Shan Cc: devicetree, linux-kernel, linux-efi, linux-arm-kernel, open list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64), Marc Zyngier, Randy Dunlap, Catalin Marinas, Will Deacon, shan.gavin On Mon, Sep 27, 2021 at 6:59 PM Gavin Shan <gshan@redhat.com> wrote: > > Hi Rob, > > On 9/28/21 12:49 AM, Rob Herring wrote: > > On Mon, Sep 27, 2021 at 1:42 AM Gavin Shan <gshan@redhat.com> wrote: > >> > >> There is no device node for the empty NUMA node. However, the > >> corresponding NUMA node ID and distance map is still valid in > >> "numa-distance-map-v1" compatible device node. > >> > >> This fetches the NUMA node ID and distance map for these empty > >> NUMA node from "numa-distance-map-v1" compatible device node. > > > > This is much nicer. > > > > Indeed, thanks for your suggestions :) > > >> Signed-off-by: Gavin Shan <gshan@redhat.com> > >> --- > >> drivers/of/of_numa.c | 2 ++ > >> 1 file changed, 2 insertions(+) > >> > >> diff --git a/drivers/of/of_numa.c b/drivers/of/of_numa.c > >> index fe6b13608e51..5949829a1b00 100644 > >> --- a/drivers/of/of_numa.c > >> +++ b/drivers/of/of_numa.c > >> @@ -111,6 +111,8 @@ static int __init of_numa_parse_distance_map_v1(struct device_node *map) > >> return -EINVAL; > >> } > >> > >> + node_set(nodea, numa_nodes_parsed); > >> + > > > > With this, couldn't we remove of_numa_parse_cpu_nodes() as the only > > thing it does is node_set()? > > > > I don't think so for couple of reasons: > > (1) With problematic device-tree, the distance map node might be missed > or incomplete. In this case, of_numa_parse_cpu_nodes() still helps. It's not the kernel's job to validate the DT (if it was, it is doing a terrible job). I would suggest writing some checks for dtc if we're worried about correctness. (The schemas don't work too well for cross node checks.) > (2) @numa_nodes_parsed is also updated when the memory nodes are iterated > in of_numa_parse_memory_nodes() and numa_add_memblk(). > > So @numa_nodes_parsed, which is synchronized to @node_possible_map afterwards, > is the gathering output of CPU nodes, memory nodes and distance map node. Is it valid to have node id's that are not in the distance map? Rob ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v2 2/2] of, numa: Fetch empty NUMA node ID from distance map 2021-09-28 17:22 ` Rob Herring @ 2021-09-29 2:00 ` Gavin Shan 0 siblings, 0 replies; 9+ messages in thread From: Gavin Shan @ 2021-09-29 2:00 UTC (permalink / raw) To: Rob Herring Cc: devicetree, linux-kernel, linux-efi, linux-arm-kernel, open list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64), Marc Zyngier, Randy Dunlap, Catalin Marinas, Will Deacon, shan.gavin On 9/29/21 3:22 AM, Rob Herring wrote: > On Mon, Sep 27, 2021 at 6:59 PM Gavin Shan <gshan@redhat.com> wrote: >> On 9/28/21 12:49 AM, Rob Herring wrote: >>> On Mon, Sep 27, 2021 at 1:42 AM Gavin Shan <gshan@redhat.com> wrote: >>>> >>>> There is no device node for the empty NUMA node. However, the >>>> corresponding NUMA node ID and distance map is still valid in >>>> "numa-distance-map-v1" compatible device node. >>>> >>>> This fetches the NUMA node ID and distance map for these empty >>>> NUMA node from "numa-distance-map-v1" compatible device node. >>> >>> This is much nicer. >>> >> >> Indeed, thanks for your suggestions :) >> >>>> Signed-off-by: Gavin Shan <gshan@redhat.com> >>>> --- >>>> drivers/of/of_numa.c | 2 ++ >>>> 1 file changed, 2 insertions(+) >>>> >>>> diff --git a/drivers/of/of_numa.c b/drivers/of/of_numa.c >>>> index fe6b13608e51..5949829a1b00 100644 >>>> --- a/drivers/of/of_numa.c >>>> +++ b/drivers/of/of_numa.c >>>> @@ -111,6 +111,8 @@ static int __init of_numa_parse_distance_map_v1(struct device_node *map) >>>> return -EINVAL; >>>> } >>>> >>>> + node_set(nodea, numa_nodes_parsed); >>>> + >>> >>> With this, couldn't we remove of_numa_parse_cpu_nodes() as the only >>> thing it does is node_set()? >>> >> >> I don't think so for couple of reasons: >> >> (1) With problematic device-tree, the distance map node might be missed >> or incomplete. In this case, of_numa_parse_cpu_nodes() still helps. > > It's not the kernel's job to validate the DT (if it was, it is doing a > terrible job). I would suggest writing some checks for dtc if we're > worried about correctness. (The schemas don't work too well for cross > node checks.) > I didn't look into dtc's code and not sure if dtc has this sort of validation. Besides, dtc is out of scope when QEMU is involved. The device-tree blob isn't produced by dtc in QEMU. >> (2) @numa_nodes_parsed is also updated when the memory nodes are iterated >> in of_numa_parse_memory_nodes() and numa_add_memblk(). >> >> So @numa_nodes_parsed, which is synchronized to @node_possible_map afterwards, >> is the gathering output of CPU nodes, memory nodes and distance map node. > > Is it valid to have node id's that are not in the distance map? > Yes, it's valid from the kernel's perspective. The default distance matrix, where the local and remote distances are 10 and 20, is applied if the distance map is missed in device-tree. The code can be found from drivers/base/arch_numa.c::numa_alloc_distance() Besides, it's possible that the distance map isn't populated by QEMU. However, I'm going to improve the situation so the distance map will be populated unconditionally. The following option is supported by QEMU, to specify the distance between two NUMA nodes. However, it's not mandatory. The distance map in device-tree won't be populated if the option is missed. -numa dist,a=<src_numa_node>,b=<dst_numa_node>,val=<distance> Thanks, Gavin ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH v2 2/2] of, numa: Fetch empty NUMA node ID from distance map 2021-09-27 6:41 ` [PATCH v2 2/2] of, numa: Fetch empty NUMA node ID from distance map Gavin Shan 2021-09-27 14:49 ` Rob Herring @ 2021-10-04 18:13 ` Rob Herring 1 sibling, 0 replies; 9+ messages in thread From: Rob Herring @ 2021-10-04 18:13 UTC (permalink / raw) To: Gavin Shan Cc: devicetree, maz, linux-efi, linux-kernel, shan.gavin, catalin.marinas, rdunlap, kvmarm, will, linux-arm-kernel On Mon, 27 Sep 2021 14:41:19 +0800, Gavin Shan wrote: > There is no device node for the empty NUMA node. However, the > corresponding NUMA node ID and distance map is still valid in > "numa-distance-map-v1" compatible device node. > > This fetches the NUMA node ID and distance map for these empty > NUMA node from "numa-distance-map-v1" compatible device node. > > Signed-off-by: Gavin Shan <gshan@redhat.com> > --- > drivers/of/of_numa.c | 2 ++ > 1 file changed, 2 insertions(+) > Applied, thanks! ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2021-10-04 18:13 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-09-27 6:41 [PATCH v2 0/2] Specify empty NUMA node Gavin Shan 2021-09-27 6:41 ` [PATCH v2 1/2] Documentation, dt, numa: Add note to " Gavin Shan 2021-10-04 18:13 ` Rob Herring 2021-09-27 6:41 ` [PATCH v2 2/2] of, numa: Fetch empty NUMA node ID from distance map Gavin Shan 2021-09-27 14:49 ` Rob Herring 2021-09-27 23:59 ` Gavin Shan 2021-09-28 17:22 ` Rob Herring 2021-09-29 2:00 ` Gavin Shan 2021-10-04 18:13 ` Rob Herring
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).