linux-efi.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v2 0/2] Specify empty NUMA node
@ 2021-09-27  6:41 Gavin Shan
  2021-09-27  6:41 ` [PATCH v2 1/2] Documentation, dt, numa: Add note to " Gavin Shan
  2021-09-27  6:41 ` [PATCH v2 2/2] of, numa: Fetch empty NUMA node ID from distance map Gavin Shan
  0 siblings, 2 replies; 9+ messages in thread
From: Gavin Shan @ 2021-09-27  6:41 UTC (permalink / raw)
  To: devicetree
  Cc: linux-kernel, linux-efi, linux-arm-kernel, kvmarm, robh, maz,
	rdunlap, catalin.marinas, will, shan.gavin

There are multiple empty nodes, where no memory resides in, are allowed
in virtual machine (VM) backed by QEMU. I don't find anywhere to document
how the corresponding device nodes are populated. This series adds one
section in Documentation/devicetree/bindings/numa.txt to document how
the empty NUMA nodes are handled: no device nodes are existing for these
empty NUMA nodes, but the NUMA IDs and distance map are still valid in
"numa-distance-map-v1" compatible device node.

PATCH[1] adds one section in Documentation/devicetree/bindings/numa.txt
         to specify how device nodes are populated for these empty NUMA
         nodes.
PATCH[2] fetches NUMA IDs and distance maps from "numa-distance-map-v1"
         compatible device node.

Changelog
=========
   * No device nodes for empty NUMA nodes                     (Rob)
   * Add patch to fetch NUMA IDs and distance map from the
     "numa-distance-map-v1" compatible device node            (Rob)

Gavin Shan (2):
  Documentation, dt, numa: Add note to empty NUMA node
  of, numa: Fetch empty NUMA node ID from distance map

 Documentation/devicetree/bindings/numa.txt | 46 +++++++++++++++++++++-
 drivers/of/of_numa.c                       |  2 +
 2 files changed, 47 insertions(+), 1 deletion(-)

-- 
2.23.0


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v2 1/2] Documentation, dt, numa: Add note to empty NUMA node
  2021-09-27  6:41 [PATCH v2 0/2] Specify empty NUMA node Gavin Shan
@ 2021-09-27  6:41 ` Gavin Shan
  2021-10-04 18:13   ` Rob Herring
  2021-09-27  6:41 ` [PATCH v2 2/2] of, numa: Fetch empty NUMA node ID from distance map Gavin Shan
  1 sibling, 1 reply; 9+ messages in thread
From: Gavin Shan @ 2021-09-27  6:41 UTC (permalink / raw)
  To: devicetree
  Cc: linux-kernel, linux-efi, linux-arm-kernel, kvmarm, robh, maz,
	rdunlap, catalin.marinas, will, shan.gavin

The empty memory nodes, where no memory resides in, are allowed.
The NUMA node IDs are still valid and parsed, but memory may be
added to them through hotplug afterwards. Currently, QEMU fails
to boot when multiple empty memory nodes are specified. It's
caused by device-tree population failure and duplicated memory
node names.

The device-tree specification doesn't provide how empty NUMA
nodes are handled. Besides, I finds difficulty to get where
this case is documented. So lets add a section for empty memory
nodes to cover it in NUMA binding document.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 Documentation/devicetree/bindings/numa.txt | 46 +++++++++++++++++++++-
 1 file changed, 45 insertions(+), 1 deletion(-)

diff --git a/Documentation/devicetree/bindings/numa.txt b/Documentation/devicetree/bindings/numa.txt
index 21b35053ca5a..42f282c2f3cc 100644
--- a/Documentation/devicetree/bindings/numa.txt
+++ b/Documentation/devicetree/bindings/numa.txt
@@ -103,7 +103,51 @@ Example:
 		};
 
 ==============================================================================
-4 - Example dts
+4 - Empty memory nodes
+==============================================================================
+
+Empty memory nodes, which no memory resides in, are allowed. There are no
+device nodes for these empty memory nodes. However, the NUMA node IDs and
+distance maps are still valid and memory may be added into them through
+hotplug afterwards.
+
+Example:
+
+	memory@0 {
+		device_type = "memory";
+		reg = <0x0 0x0 0x0 0x80000000>;
+		numa-node-id = <0>;
+	};
+
+	memory@80000000 {
+		device_type = "memory";
+		reg = <0x0 0x80000000 0x0 0x80000000>;
+		numa-node-id = <1>;
+	};
+
+	/* Empty memory node 2 and 3 */
+	distance-map {
+		compatible = "numa-distance-map-v1";
+		distance-matrix = <0 0  10>,
+				  <0 1  20>,
+				  <0 2  40>,
+				  <0 3  20>,
+				  <1 0  20>,
+				  <1 1  10>,
+				  <1 2  20>,
+				  <1 3  40>,
+				  <2 0  40>,
+				  <2 1  20>,
+				  <2 2  10>,
+				  <2 3  20>,
+				  <3 0  20>,
+				  <3 1  40>,
+				  <3 2  20>,
+				  <3 3  10>;
+	};
+
+==============================================================================
+5 - Example dts
 ==============================================================================
 
 Dual socket system consists of 2 boards connected through ccn bus and
-- 
2.23.0


^ permalink raw reply	[flat|nested] 9+ messages in thread

* [PATCH v2 2/2] of, numa: Fetch empty NUMA node ID from distance map
  2021-09-27  6:41 [PATCH v2 0/2] Specify empty NUMA node Gavin Shan
  2021-09-27  6:41 ` [PATCH v2 1/2] Documentation, dt, numa: Add note to " Gavin Shan
@ 2021-09-27  6:41 ` Gavin Shan
  2021-09-27 14:49   ` Rob Herring
  2021-10-04 18:13   ` Rob Herring
  1 sibling, 2 replies; 9+ messages in thread
From: Gavin Shan @ 2021-09-27  6:41 UTC (permalink / raw)
  To: devicetree
  Cc: linux-kernel, linux-efi, linux-arm-kernel, kvmarm, robh, maz,
	rdunlap, catalin.marinas, will, shan.gavin

There is no device node for the empty NUMA node. However, the
corresponding NUMA node ID and distance map is still valid in
"numa-distance-map-v1" compatible device node.

This fetches the NUMA node ID and distance map for these empty
NUMA node from "numa-distance-map-v1" compatible device node.

Signed-off-by: Gavin Shan <gshan@redhat.com>
---
 drivers/of/of_numa.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/of/of_numa.c b/drivers/of/of_numa.c
index fe6b13608e51..5949829a1b00 100644
--- a/drivers/of/of_numa.c
+++ b/drivers/of/of_numa.c
@@ -111,6 +111,8 @@ static int __init of_numa_parse_distance_map_v1(struct device_node *map)
 			return -EINVAL;
 		}
 
+		node_set(nodea, numa_nodes_parsed);
+
 		numa_set_distance(nodea, nodeb, distance);
 
 		/* Set default distance of node B->A same as A->B */
-- 
2.23.0


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2 2/2] of, numa: Fetch empty NUMA node ID from distance map
  2021-09-27  6:41 ` [PATCH v2 2/2] of, numa: Fetch empty NUMA node ID from distance map Gavin Shan
@ 2021-09-27 14:49   ` Rob Herring
  2021-09-27 23:59     ` Gavin Shan
  2021-10-04 18:13   ` Rob Herring
  1 sibling, 1 reply; 9+ messages in thread
From: Rob Herring @ 2021-09-27 14:49 UTC (permalink / raw)
  To: Gavin Shan
  Cc: devicetree, linux-kernel, linux-efi, linux-arm-kernel,
	open list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64),
	Marc Zyngier, Randy Dunlap, Catalin Marinas, Will Deacon,
	shan.gavin

On Mon, Sep 27, 2021 at 1:42 AM Gavin Shan <gshan@redhat.com> wrote:
>
> There is no device node for the empty NUMA node. However, the
> corresponding NUMA node ID and distance map is still valid in
> "numa-distance-map-v1" compatible device node.
>
> This fetches the NUMA node ID and distance map for these empty
> NUMA node from "numa-distance-map-v1" compatible device node.

This is much nicer.

> Signed-off-by: Gavin Shan <gshan@redhat.com>
> ---
>  drivers/of/of_numa.c | 2 ++
>  1 file changed, 2 insertions(+)
>
> diff --git a/drivers/of/of_numa.c b/drivers/of/of_numa.c
> index fe6b13608e51..5949829a1b00 100644
> --- a/drivers/of/of_numa.c
> +++ b/drivers/of/of_numa.c
> @@ -111,6 +111,8 @@ static int __init of_numa_parse_distance_map_v1(struct device_node *map)
>                         return -EINVAL;
>                 }
>
> +               node_set(nodea, numa_nodes_parsed);
> +

With this, couldn't we remove of_numa_parse_cpu_nodes() as the only
thing it does is node_set()?

>                 numa_set_distance(nodea, nodeb, distance);
>
>                 /* Set default distance of node B->A same as A->B */
> --
> 2.23.0
>

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2 2/2] of, numa: Fetch empty NUMA node ID from distance map
  2021-09-27 14:49   ` Rob Herring
@ 2021-09-27 23:59     ` Gavin Shan
  2021-09-28 17:22       ` Rob Herring
  0 siblings, 1 reply; 9+ messages in thread
From: Gavin Shan @ 2021-09-27 23:59 UTC (permalink / raw)
  To: Rob Herring
  Cc: devicetree, linux-kernel, linux-efi, linux-arm-kernel,
	open list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64),
	Marc Zyngier, Randy Dunlap, Catalin Marinas, Will Deacon,
	shan.gavin

Hi Rob,

On 9/28/21 12:49 AM, Rob Herring wrote:
> On Mon, Sep 27, 2021 at 1:42 AM Gavin Shan <gshan@redhat.com> wrote:
>>
>> There is no device node for the empty NUMA node. However, the
>> corresponding NUMA node ID and distance map is still valid in
>> "numa-distance-map-v1" compatible device node.
>>
>> This fetches the NUMA node ID and distance map for these empty
>> NUMA node from "numa-distance-map-v1" compatible device node.
> 
> This is much nicer.
> 

Indeed, thanks for your suggestions :)

>> Signed-off-by: Gavin Shan <gshan@redhat.com>
>> ---
>>   drivers/of/of_numa.c | 2 ++
>>   1 file changed, 2 insertions(+)
>>
>> diff --git a/drivers/of/of_numa.c b/drivers/of/of_numa.c
>> index fe6b13608e51..5949829a1b00 100644
>> --- a/drivers/of/of_numa.c
>> +++ b/drivers/of/of_numa.c
>> @@ -111,6 +111,8 @@ static int __init of_numa_parse_distance_map_v1(struct device_node *map)
>>                          return -EINVAL;
>>                  }
>>
>> +               node_set(nodea, numa_nodes_parsed);
>> +
> 
> With this, couldn't we remove of_numa_parse_cpu_nodes() as the only
> thing it does is node_set()?
> 

I don't think so for couple of reasons:

(1) With problematic device-tree, the distance map node might be missed
     or incomplete. In this case, of_numa_parse_cpu_nodes() still helps.

(2) @numa_nodes_parsed is also updated when the memory nodes are iterated
     in of_numa_parse_memory_nodes() and numa_add_memblk().

So @numa_nodes_parsed, which is synchronized to @node_possible_map afterwards,
is the gathering output of CPU nodes, memory nodes and distance map node.

>>                  numa_set_distance(nodea, nodeb, distance);
>>
>>                  /* Set default distance of node B->A same as A->B */

Thanks,
Gavin


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2 2/2] of, numa: Fetch empty NUMA node ID from distance map
  2021-09-27 23:59     ` Gavin Shan
@ 2021-09-28 17:22       ` Rob Herring
  2021-09-29  2:00         ` Gavin Shan
  0 siblings, 1 reply; 9+ messages in thread
From: Rob Herring @ 2021-09-28 17:22 UTC (permalink / raw)
  To: Gavin Shan
  Cc: devicetree, linux-kernel, linux-efi, linux-arm-kernel,
	open list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64),
	Marc Zyngier, Randy Dunlap, Catalin Marinas, Will Deacon,
	shan.gavin

On Mon, Sep 27, 2021 at 6:59 PM Gavin Shan <gshan@redhat.com> wrote:
>
> Hi Rob,
>
> On 9/28/21 12:49 AM, Rob Herring wrote:
> > On Mon, Sep 27, 2021 at 1:42 AM Gavin Shan <gshan@redhat.com> wrote:
> >>
> >> There is no device node for the empty NUMA node. However, the
> >> corresponding NUMA node ID and distance map is still valid in
> >> "numa-distance-map-v1" compatible device node.
> >>
> >> This fetches the NUMA node ID and distance map for these empty
> >> NUMA node from "numa-distance-map-v1" compatible device node.
> >
> > This is much nicer.
> >
>
> Indeed, thanks for your suggestions :)
>
> >> Signed-off-by: Gavin Shan <gshan@redhat.com>
> >> ---
> >>   drivers/of/of_numa.c | 2 ++
> >>   1 file changed, 2 insertions(+)
> >>
> >> diff --git a/drivers/of/of_numa.c b/drivers/of/of_numa.c
> >> index fe6b13608e51..5949829a1b00 100644
> >> --- a/drivers/of/of_numa.c
> >> +++ b/drivers/of/of_numa.c
> >> @@ -111,6 +111,8 @@ static int __init of_numa_parse_distance_map_v1(struct device_node *map)
> >>                          return -EINVAL;
> >>                  }
> >>
> >> +               node_set(nodea, numa_nodes_parsed);
> >> +
> >
> > With this, couldn't we remove of_numa_parse_cpu_nodes() as the only
> > thing it does is node_set()?
> >
>
> I don't think so for couple of reasons:
>
> (1) With problematic device-tree, the distance map node might be missed
>      or incomplete. In this case, of_numa_parse_cpu_nodes() still helps.

It's not the kernel's job to validate the DT (if it was, it is doing a
terrible job). I would suggest writing some checks for dtc if we're
worried about correctness. (The schemas don't work too well for cross
node checks.)

> (2) @numa_nodes_parsed is also updated when the memory nodes are iterated
>      in of_numa_parse_memory_nodes() and numa_add_memblk().
>
> So @numa_nodes_parsed, which is synchronized to @node_possible_map afterwards,
> is the gathering output of CPU nodes, memory nodes and distance map node.

Is it valid to have node id's that are not in the distance map?

Rob

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2 2/2] of, numa: Fetch empty NUMA node ID from distance map
  2021-09-28 17:22       ` Rob Herring
@ 2021-09-29  2:00         ` Gavin Shan
  0 siblings, 0 replies; 9+ messages in thread
From: Gavin Shan @ 2021-09-29  2:00 UTC (permalink / raw)
  To: Rob Herring
  Cc: devicetree, linux-kernel, linux-efi, linux-arm-kernel,
	open list:KERNEL VIRTUAL MACHINE FOR ARM64 (KVM/arm64),
	Marc Zyngier, Randy Dunlap, Catalin Marinas, Will Deacon,
	shan.gavin

On 9/29/21 3:22 AM, Rob Herring wrote:
> On Mon, Sep 27, 2021 at 6:59 PM Gavin Shan <gshan@redhat.com> wrote:
>> On 9/28/21 12:49 AM, Rob Herring wrote:
>>> On Mon, Sep 27, 2021 at 1:42 AM Gavin Shan <gshan@redhat.com> wrote:
>>>>
>>>> There is no device node for the empty NUMA node. However, the
>>>> corresponding NUMA node ID and distance map is still valid in
>>>> "numa-distance-map-v1" compatible device node.
>>>>
>>>> This fetches the NUMA node ID and distance map for these empty
>>>> NUMA node from "numa-distance-map-v1" compatible device node.
>>>
>>> This is much nicer.
>>>
>>
>> Indeed, thanks for your suggestions :)
>>
>>>> Signed-off-by: Gavin Shan <gshan@redhat.com>
>>>> ---
>>>>    drivers/of/of_numa.c | 2 ++
>>>>    1 file changed, 2 insertions(+)
>>>>
>>>> diff --git a/drivers/of/of_numa.c b/drivers/of/of_numa.c
>>>> index fe6b13608e51..5949829a1b00 100644
>>>> --- a/drivers/of/of_numa.c
>>>> +++ b/drivers/of/of_numa.c
>>>> @@ -111,6 +111,8 @@ static int __init of_numa_parse_distance_map_v1(struct device_node *map)
>>>>                           return -EINVAL;
>>>>                   }
>>>>
>>>> +               node_set(nodea, numa_nodes_parsed);
>>>> +
>>>
>>> With this, couldn't we remove of_numa_parse_cpu_nodes() as the only
>>> thing it does is node_set()?
>>>
>>
>> I don't think so for couple of reasons:
>>
>> (1) With problematic device-tree, the distance map node might be missed
>>       or incomplete. In this case, of_numa_parse_cpu_nodes() still helps.
> 
> It's not the kernel's job to validate the DT (if it was, it is doing a
> terrible job). I would suggest writing some checks for dtc if we're
> worried about correctness. (The schemas don't work too well for cross
> node checks.)
> 

I didn't look into dtc's code and not sure if dtc has this sort of validation.
Besides, dtc is out of scope when QEMU is involved. The device-tree blob isn't
produced by dtc in QEMU.


>> (2) @numa_nodes_parsed is also updated when the memory nodes are iterated
>>       in of_numa_parse_memory_nodes() and numa_add_memblk().
>>
>> So @numa_nodes_parsed, which is synchronized to @node_possible_map afterwards,
>> is the gathering output of CPU nodes, memory nodes and distance map node.
> 
> Is it valid to have node id's that are not in the distance map?
> 

Yes, it's valid from the kernel's perspective. The default distance
matrix, where the local and remote distances are 10 and 20, is applied
if the distance map is missed in device-tree. The code can be found from
drivers/base/arch_numa.c::numa_alloc_distance()

Besides, it's possible that the distance map isn't populated by QEMU.
However, I'm going to improve the situation so the distance map will
be populated unconditionally.

The following option is supported by QEMU, to specify the distance
between two NUMA nodes. However, it's not mandatory. The distance
map in device-tree won't be populated if the option is missed.

     -numa dist,a=<src_numa_node>,b=<dst_numa_node>,val=<distance>

Thanks,
Gavin


^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2 1/2] Documentation, dt, numa: Add note to empty NUMA node
  2021-09-27  6:41 ` [PATCH v2 1/2] Documentation, dt, numa: Add note to " Gavin Shan
@ 2021-10-04 18:13   ` Rob Herring
  0 siblings, 0 replies; 9+ messages in thread
From: Rob Herring @ 2021-10-04 18:13 UTC (permalink / raw)
  To: Gavin Shan
  Cc: devicetree, catalin.marinas, shan.gavin, will, rdunlap, maz,
	linux-efi, kvmarm, linux-arm-kernel, linux-kernel

On Mon, 27 Sep 2021 14:41:18 +0800, Gavin Shan wrote:
> The empty memory nodes, where no memory resides in, are allowed.
> The NUMA node IDs are still valid and parsed, but memory may be
> added to them through hotplug afterwards. Currently, QEMU fails
> to boot when multiple empty memory nodes are specified. It's
> caused by device-tree population failure and duplicated memory
> node names.
> 
> The device-tree specification doesn't provide how empty NUMA
> nodes are handled. Besides, I finds difficulty to get where
> this case is documented. So lets add a section for empty memory
> nodes to cover it in NUMA binding document.
> 
> Signed-off-by: Gavin Shan <gshan@redhat.com>
> ---
>  Documentation/devicetree/bindings/numa.txt | 46 +++++++++++++++++++++-
>  1 file changed, 45 insertions(+), 1 deletion(-)
> 

Applied, thanks!

^ permalink raw reply	[flat|nested] 9+ messages in thread

* Re: [PATCH v2 2/2] of, numa: Fetch empty NUMA node ID from distance map
  2021-09-27  6:41 ` [PATCH v2 2/2] of, numa: Fetch empty NUMA node ID from distance map Gavin Shan
  2021-09-27 14:49   ` Rob Herring
@ 2021-10-04 18:13   ` Rob Herring
  1 sibling, 0 replies; 9+ messages in thread
From: Rob Herring @ 2021-10-04 18:13 UTC (permalink / raw)
  To: Gavin Shan
  Cc: devicetree, maz, linux-efi, linux-kernel, shan.gavin,
	catalin.marinas, rdunlap, kvmarm, will, linux-arm-kernel

On Mon, 27 Sep 2021 14:41:19 +0800, Gavin Shan wrote:
> There is no device node for the empty NUMA node. However, the
> corresponding NUMA node ID and distance map is still valid in
> "numa-distance-map-v1" compatible device node.
> 
> This fetches the NUMA node ID and distance map for these empty
> NUMA node from "numa-distance-map-v1" compatible device node.
> 
> Signed-off-by: Gavin Shan <gshan@redhat.com>
> ---
>  drivers/of/of_numa.c | 2 ++
>  1 file changed, 2 insertions(+)
> 

Applied, thanks!

^ permalink raw reply	[flat|nested] 9+ messages in thread

end of thread, other threads:[~2021-10-04 18:13 UTC | newest]

Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-09-27  6:41 [PATCH v2 0/2] Specify empty NUMA node Gavin Shan
2021-09-27  6:41 ` [PATCH v2 1/2] Documentation, dt, numa: Add note to " Gavin Shan
2021-10-04 18:13   ` Rob Herring
2021-09-27  6:41 ` [PATCH v2 2/2] of, numa: Fetch empty NUMA node ID from distance map Gavin Shan
2021-09-27 14:49   ` Rob Herring
2021-09-27 23:59     ` Gavin Shan
2021-09-28 17:22       ` Rob Herring
2021-09-29  2:00         ` Gavin Shan
2021-10-04 18:13   ` Rob Herring

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).