Question about NUMA distance calculation in powerpc/mm/numa.c

Question about NUMA distance calculation in powerpc/mm/numa.c

[Daniel Henrique Barboza]

Hello,


I didn't find an explanation about the 'double the distance' logic in
'git log' or anywhere in the kernel docs:


(arch/powerpc/mm/numa.c, __node_distance()):

for (i = 0; i < distance_ref_points_depth; i++) {
	if (distance_lookup_table[a][i] == distance_lookup_table[b][i])
		break;

	/* Double the distance for each NUMA level */
	distance *= 2;
}

For reference, the commit that added it:


commit 41eab6f88f24124df89e38067b3766b7bef06ddb
Author: Anton Blanchard <anton@samba.org>
Date:   Sun May 16 20:22:31 2010 +0000

     powerpc/numa: Use form 1 affinity to setup node distance
  

Is there a technical reason for the distance being calculated as the double
for each NUMA level?

The reason I'm asking is because of the QEMU/Libvirt capability to define NUMA
node distances in the VMs. For x86, an user is capable of setting any distance
values to the NUMA topology due to how ACPI SLIT works.

The user, of course, wants the pseries guest to behave the same way. The best
we can do for now is document why this will not happen. I'll document the
limitations imposed by the design itself (how ibm,associativity-reference-points
is capped to MAX_DISTANCE_REF_POINTS and so on). I also would like to document
that the pseries kernel will double the distance for each NUMA level, and for
that it would be nice to provide an actual reason for that to happen, if
there is any.


Thanks,


Daniel

Re: Question about NUMA distance calculation in powerpc/mm/numa.c

[Michael Ellerman]

Daniel Henrique Barboza <danielhb413@gmail.com> writes:
> Hello,
>
>
> I didn't find an explanation about the 'double the distance' logic in
> 'git log' or anywhere in the kernel docs:
>
>
> (arch/powerpc/mm/numa.c, __node_distance()):

Adding more context:

  int distance = LOCAL_DISTANCE;
  ...

> for (i = 0; i < distance_ref_points_depth; i++) {
> 	if (distance_lookup_table[a][i] == distance_lookup_table[b][i])
> 		break;
>
> 	/* Double the distance for each NUMA level */
> 	distance *= 2;
> }

And:

#define LOCAL_DISTANCE		10
#define REMOTE_DISTANCE		20


So AFAICS the doubling is just a way to ensure we go from LOCAL_DISTANCE
to REMOTE_DISTANCE at the first level, and then after that it's fairly
arbitrary.

cheers