All of lore.kernel.org
 help / color / mirror / Atom feed
* [XEN RFC PATCH 00/40] Add device tree based NUMA support to Arm64
@ 2021-08-11 10:23 Wei Chen
  2021-08-11 10:23 ` [XEN RFC PATCH 01/40] tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf Wei Chen
                   ` (42 more replies)
  0 siblings, 43 replies; 196+ messages in thread
From: Wei Chen @ 2021-08-11 10:23 UTC (permalink / raw)
  To: wei.chen, xen-devel, sstabellini, julien, jbeulich; +Cc: Bertrand.Marquis

Xen memory allocation and scheduler modules are NUMA aware.
But actually, on x86 has implemented the architecture APIs
to support NUMA. Arm was providing a set of fake architecture
APIs to make it compatible with NUMA awared memory allocation
and scheduler.

Arm system was working well as a single node NUMA system with
these fake APIs, because we didn't have multiple nodes NUMA
system on Arm. But in recent years, more and more Arm devices
support multiple nodes NUMA system. Like TX2, some Hisilicon
chips and the Ampere Altra.

So now we have a new problem. When Xen is running on these Arm
devices, Xen still treat them as single node SMP systems. The
NUMA affinity capability of Xen memory allocation and scheduler
becomes meaningless. Because they rely on input data that does
not reflect real NUMA layout.

Xen still think the access time for all of the memory is the
same for all CPUs. However, Xen may allocate memory to a VM
from different NUMA nodes with different access speeds. This
difference can be amplified in workloads inside VM, causing
performance instability and timeouts. 

So in this patch series, we implement a set of NUMA API to use
device tree to describe the NUMA layout. We reuse most of the
code of x86 NUMA to create and maintain the mapping between
memory and CPU, create the matrix between any two NUMA nodes.
Except ACPI and some x86 specified code, we have moved other
code to common. In next stage, when we implement ACPI based
NUMA for Arm64, we may move the ACPI NUMA code to common too,
but in current stage, we keep it as x86 only.

This patch serires has been tested and booted well on one
Arm64 NUMA machine and one HPE x86 NUMA machine.

Hongda Deng (2):
  xen/arm: return default DMA bit width when platform is not set
  xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0

Wei Chen (38):
  tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf
  xen/arm: Print a 64-bit number in hex from early uart
  xen/x86: Initialize memnodemapsize while faking NUMA node
  xen: decouple NUMA from ACPI in Kconfig
  xen/arm: use !CONFIG_NUMA to keep fake NUMA API
  xen/x86: Move NUMA memory node map functions to common
  xen/x86: Move numa_add_cpu_node to common
  xen/x86: Move NR_NODE_MEMBLKS macro to common
  xen/x86: Move NUMA nodes and memory block ranges to common
  xen/x86: Move numa_initmem_init to common
  xen/arm: introduce numa_set_node for Arm
  xen/arm: set NUMA nodes max number to 64 by default
  xen/x86: move NUMA API from x86 header to common header
  xen/arm: Create a fake NUMA node to use common code
  xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64
  xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI
  xen: fdt: Introduce a helper to check fdt node type
  xen/arm: implement node distance helpers for Arm64
  xen/arm: introduce device_tree_numa as a switch for device tree NUMA
  xen/arm: introduce a helper to parse device tree processor node
  xen/arm: introduce a helper to parse device tree memory node
  xen/arm: introduce a helper to parse device tree NUMA distance map
  xen/arm: unified entry to parse all NUMA data from device tree
  xen/arm: Add boot and secondary CPU to NUMA system
  xen/arm: build CPU NUMA node map while creating cpu_logical_map
  xen/x86: decouple nodes_cover_memory with E820 map
  xen/arm: implement Arm arch helpers Arm to get memory map info
  xen: move NUMA memory and CPU parsed nodemasks to common
  xen/x86: move nodes_cover_memory to common
  xen/x86: make acpi_scan_nodes to be neutral
  xen: export bad_srat and srat_disabled to extern
  xen: move numa_scan_nodes from x86 to common
  xen: enable numa_scan_nodes for device tree based NUMA
  xen/arm: keep guest still be NUMA unware
  xen: introduce an arch helper to do NUMA init failed fallback
  xen/arm: enable device tree based NUMA in system init
  xen/x86: move numa_setup to common to support NUMA switch in command
    line
  xen/x86: move dump_numa info hotkey to common

 tools/libs/util/libxlu_pci.c    |   3 +-
 xen/arch/arm/Kconfig            |  10 +
 xen/arch/arm/Makefile           |   2 +
 xen/arch/arm/arm64/head.S       |   9 +-
 xen/arch/arm/bootfdt.c          |   8 +-
 xen/arch/arm/domain_build.c     |  17 +-
 xen/arch/arm/efi/efi-boot.h     |  25 --
 xen/arch/arm/numa.c             | 162 +++++++++
 xen/arch/arm/numa_device_tree.c | 292 ++++++++++++++++
 xen/arch/arm/platform.c         |   4 +-
 xen/arch/arm/setup.c            |  14 +
 xen/arch/arm/smpboot.c          |  37 +-
 xen/arch/x86/Kconfig            |   2 +-
 xen/arch/x86/numa.c             | 421 +----------------------
 xen/arch/x86/srat.c             | 147 +-------
 xen/common/Kconfig              |   3 +
 xen/common/Makefile             |   1 +
 xen/common/libfdt/fdt_ro.c      |  15 +
 xen/common/numa.c               | 588 ++++++++++++++++++++++++++++++++
 xen/common/page_alloc.c         |   2 +-
 xen/drivers/acpi/Kconfig        |   3 +-
 xen/drivers/acpi/Makefile       |   2 +-
 xen/include/asm-arm/numa.h      |  33 ++
 xen/include/asm-arm/setup.h     |   6 +
 xen/include/asm-x86/acpi.h      |   4 -
 xen/include/asm-x86/config.h    |   1 -
 xen/include/asm-x86/numa.h      |  65 +---
 xen/include/asm-x86/setup.h     |   1 -
 xen/include/xen/libfdt/libfdt.h |  25 ++
 xen/include/xen/nodemask.h      |   2 +
 xen/include/xen/numa.h          |  80 +++++
 31 files changed, 1325 insertions(+), 659 deletions(-)
 create mode 100644 xen/arch/arm/numa.c
 create mode 100644 xen/arch/arm/numa_device_tree.c
 create mode 100644 xen/common/numa.c

-- 
2.25.1



^ permalink raw reply	[flat|nested] 196+ messages in thread

end of thread, other threads:[~2021-09-09  3:54 UTC | newest]

Thread overview: 196+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-08-11 10:23 [XEN RFC PATCH 00/40] Add device tree based NUMA support to Arm64 Wei Chen
2021-08-11 10:23 ` [XEN RFC PATCH 01/40] tools: Fix -Werror=maybe-uninitialized for xlu_pci_parse_bdf Wei Chen
2021-08-11 10:49   ` Jan Beulich
2021-08-13  6:28     ` Wei Chen
2021-08-11 10:23 ` [XEN RFC PATCH 02/40] xen/arm: Print a 64-bit number in hex from early uart Wei Chen
2021-08-19 13:05   ` Julien Grall
2021-08-20  1:13     ` Wei Chen
2021-08-11 10:23 ` [XEN RFC PATCH 03/40] xen/x86: Initialize memnodemapsize while faking NUMA node Wei Chen
2021-08-12 15:32   ` Jan Beulich
2021-08-13  7:26     ` Wei Chen
2021-08-13  8:29       ` Jan Beulich
2021-08-11 10:23 ` [XEN RFC PATCH 04/40] xen/arm: return default DMA bit width when platform is not set Wei Chen
2021-08-11 10:54   ` Jan Beulich
2021-08-13  6:54     ` Wei Chen
2021-08-13  6:56       ` Jan Beulich
2021-08-19 13:28   ` Julien Grall
2021-08-20  2:04     ` Wei Chen
2021-08-20  8:20       ` Julien Grall
2021-08-20  9:37         ` Wei Chen
2021-08-20 11:18           ` Julien Grall
2021-08-20 11:58             ` Wei Chen
2021-08-11 10:23 ` [XEN RFC PATCH 05/40] xen/arm: Fix lowmem_bitsize when arch_get_dma_bitsize return 0 Wei Chen
2021-08-19 13:32   ` Julien Grall
2021-08-20  2:05     ` Wei Chen
2021-08-11 10:23 ` [XEN RFC PATCH 06/40] xen: decouple NUMA from ACPI in Kconfig Wei Chen
2021-08-12 15:36   ` Jan Beulich
2021-08-13  7:27     ` Wei Chen
2021-08-12 16:54   ` Julien Grall
2021-08-13  7:28     ` Wei Chen
2021-08-11 10:23 ` [XEN RFC PATCH 07/40] xen/arm: use !CONFIG_NUMA to keep fake NUMA API Wei Chen
2021-08-19 13:34   ` Julien Grall
2021-08-20  2:08     ` Wei Chen
2021-08-20  8:23       ` Julien Grall
2021-08-20 10:24         ` Wei Chen
2021-08-20 11:24           ` Julien Grall
2021-08-20 12:23             ` Wei Chen
2021-08-20 14:41               ` Julien Grall
2021-08-11 10:23 ` [XEN RFC PATCH 08/40] xen/x86: Move NUMA memory node map functions to common Wei Chen
2021-08-23 17:47   ` Julien Grall
2021-08-24  4:07     ` Wei Chen
2021-08-11 10:23 ` [XEN RFC PATCH 09/40] xen/x86: Move numa_add_cpu_node " Wei Chen
2021-08-23 17:54   ` Julien Grall
2021-08-24  4:18     ` Wei Chen
2021-08-11 10:23 ` [XEN RFC PATCH 10/40] xen/x86: Move NR_NODE_MEMBLKS macro " Wei Chen
2021-08-23 17:58   ` Julien Grall
2021-08-11 10:23 ` [XEN RFC PATCH 11/40] xen/x86: Move NUMA nodes and memory block ranges " Wei Chen
2021-08-24 17:40   ` Julien Grall
2021-08-25  0:57     ` Wei Chen
2021-08-11 10:23 ` [XEN RFC PATCH 12/40] xen/x86: Move numa_initmem_init " Wei Chen
2021-08-25 10:21   ` Julien Grall
2021-08-25 11:15     ` Wei Chen
2021-08-25 13:26       ` Julien Grall
2021-08-11 10:23 ` [XEN RFC PATCH 13/40] xen/arm: introduce numa_set_node for Arm Wei Chen
2021-08-25 10:36   ` Julien Grall
2021-08-25 12:07     ` Wei Chen
2021-08-25 13:24       ` Julien Grall
2021-08-26  5:13         ` Wei Chen
2021-08-11 10:23 ` [XEN RFC PATCH 14/40] xen/arm: set NUMA nodes max number to 64 by default Wei Chen
2021-08-25 13:28   ` Julien Grall
2021-08-25 13:36     ` Jan Beulich
2021-08-26  2:26       ` Wei Chen
2021-08-11 10:23 ` [XEN RFC PATCH 15/40] xen/x86: move NUMA API from x86 header to common header Wei Chen
2021-08-11 10:23 ` [XEN RFC PATCH 16/40] xen/arm: Create a fake NUMA node to use common code Wei Chen
2021-08-26 23:10   ` Stefano Stabellini
2021-08-27  1:15     ` Wei Chen
2021-08-27  6:18     ` Jan Beulich
2021-08-27  9:32       ` Wei Chen
2021-08-11 10:24 ` [XEN RFC PATCH 17/40] xen/arm: Introduce DEVICE_TREE_NUMA Kconfig for arm64 Wei Chen
2021-08-19 13:38   ` Julien Grall
2021-08-20  2:30     ` Wei Chen
2021-08-20  8:41       ` Julien Grall
2021-08-20 10:49         ` Wei Chen
2021-08-20 11:28           ` Julien Grall
2021-08-20 12:25             ` Wei Chen
2021-08-11 10:24 ` [XEN RFC PATCH 18/40] xen/arm: Keep memory nodes in dtb for NUMA when boot from EFI Wei Chen
2021-08-19 17:35   ` Julien Grall
2021-08-20  2:18     ` Wei Chen
2021-08-26 23:24   ` Stefano Stabellini
2021-08-27  7:41     ` Julien Grall
2021-08-27 23:10       ` Stefano Stabellini
2021-08-27  9:23     ` Wei Chen
2021-08-11 10:24 ` [XEN RFC PATCH 19/40] xen: fdt: Introduce a helper to check fdt node type Wei Chen
2021-08-25 13:39   ` Julien Grall
2021-08-26  6:00     ` Wei Chen
2021-08-11 10:24 ` [XEN RFC PATCH 20/40] xen/arm: implement node distance helpers for Arm64 Wei Chen
2021-08-26 23:52   ` Stefano Stabellini
2021-08-27  9:30     ` Wei Chen
2021-08-11 10:24 ` [XEN RFC PATCH 21/40] xen/arm: introduce device_tree_numa as a switch for device tree NUMA Wei Chen
2021-08-19 17:45   ` Julien Grall
2021-08-20  2:21     ` Wei Chen
2021-08-11 10:24 ` [XEN RFC PATCH 22/40] xen/arm: introduce a helper to parse device tree processor node Wei Chen
2021-08-19 18:09   ` Julien Grall
2021-08-23  8:42     ` Wei Chen
2021-08-19 18:10   ` Julien Grall
2021-08-23  8:47     ` Wei Chen
2021-08-23 10:59       ` Julien Grall
2021-08-24  4:09         ` Wei Chen
2021-08-19 18:13   ` Julien Grall
2021-08-20  2:23     ` Wei Chen
2021-08-20  8:44       ` Julien Grall
2021-08-20 11:53         ` Wei Chen
2021-08-27  0:06   ` Stefano Stabellini
2021-08-27  9:31     ` Wei Chen
2021-08-11 10:24 ` [XEN RFC PATCH 23/40] xen/arm: introduce a helper to parse device tree memory node Wei Chen
2021-08-25 13:48   ` Julien Grall
2021-08-26  6:35     ` Wei Chen
2021-08-26  8:21       ` Julien Grall
2021-08-26 11:54         ` Wei Chen
2021-08-28  1:06   ` Stefano Stabellini
2021-08-28  3:56     ` Wei Chen
2021-08-28 10:33       ` Julien Grall
2021-08-28 13:58         ` Wei Chen
2021-09-08  7:34           ` Wei Chen
2021-09-08 22:31             ` Stefano Stabellini
2021-09-09  3:54               ` Wei Chen
2021-08-11 10:24 ` [XEN RFC PATCH 24/40] xen/arm: introduce a helper to parse device tree NUMA distance map Wei Chen
2021-08-25 13:56   ` Julien Grall
2021-08-26  7:01     ` Wei Chen
2021-08-31  0:48   ` Stefano Stabellini
2021-08-31 10:17     ` Wei Chen
2021-08-31 21:36       ` Stefano Stabellini
2021-09-01 11:04         ` Wei Chen
2021-09-01 16:21           ` Stefano Stabellini
2021-09-02  2:30             ` Wei Chen
2021-09-02 15:19               ` Stefano Stabellini
2021-09-02  6:00             ` Jan Beulich
2021-09-02 14:14               ` Wei Chen
2021-08-11 10:24 ` [XEN RFC PATCH 25/40] xen/arm: unified entry to parse all NUMA data from device tree Wei Chen
2021-08-31  0:54   ` Stefano Stabellini
2021-08-31 17:47     ` Julien Grall
2021-09-01 18:30       ` Stefano Stabellini
2021-09-02  2:48         ` Wei Chen
2021-08-11 10:24 ` [XEN RFC PATCH 26/40] xen/arm: Add boot and secondary CPU to NUMA system Wei Chen
2021-08-25 16:58   ` Julien Grall
2021-08-26  7:24     ` Wei Chen
2021-08-26  8:49       ` Julien Grall
2021-08-26  9:39         ` Jan Beulich
2021-08-26 12:08           ` Wei Chen
2021-08-26 12:26             ` Jan Beulich
2021-08-11 10:24 ` [XEN RFC PATCH 27/40] xen/arm: build CPU NUMA node map while creating cpu_logical_map Wei Chen
2021-08-25 17:06   ` Julien Grall
2021-08-26  7:26     ` Wei Chen
2021-08-11 10:24 ` [XEN RFC PATCH 28/40] xen/x86: decouple nodes_cover_memory with E820 map Wei Chen
2021-08-31  1:07   ` Stefano Stabellini
2021-08-31 10:19     ` Wei Chen
2021-08-11 10:24 ` [XEN RFC PATCH 29/40] xen/arm: implement Arm arch helpers Arm to get memory map info Wei Chen
2021-08-25 17:09   ` Julien Grall
2021-08-26  7:27     ` Wei Chen
2021-08-11 10:24 ` [XEN RFC PATCH 30/40] xen: move NUMA memory and CPU parsed nodemasks to common Wei Chen
2021-08-25 17:16   ` Julien Grall
2021-08-26  7:29     ` Wei Chen
2021-08-11 10:24 ` [XEN RFC PATCH 31/40] xen/x86: move nodes_cover_memory " Wei Chen
2021-08-31  1:16   ` Stefano Stabellini
2021-08-31 13:43     ` Wei Chen
2021-08-11 10:24 ` [XEN RFC PATCH 32/40] xen/x86: make acpi_scan_nodes to be neutral Wei Chen
2021-08-27 14:08   ` Julien Grall
2021-08-28  2:11     ` Wei Chen
2021-08-11 10:24 ` [XEN RFC PATCH 33/40] xen: export bad_srat and srat_disabled to extern Wei Chen
2021-08-11 10:24 ` [XEN RFC PATCH 34/40] xen: move numa_scan_nodes from x86 to common Wei Chen
2021-08-27 14:14   ` Julien Grall
2021-08-28  2:12     ` Wei Chen
2021-08-31  1:26   ` Stefano Stabellini
2021-08-31 13:43     ` Wei Chen
2021-08-11 10:24 ` [XEN RFC PATCH 35/40] xen: enable numa_scan_nodes for device tree based NUMA Wei Chen
2021-08-27 14:19   ` Julien Grall
2021-08-28  2:13     ` Wei Chen
2021-08-11 10:24 ` [XEN RFC PATCH 36/40] xen/arm: keep guest still be NUMA unware Wei Chen
2021-08-27 14:28   ` Julien Grall
2021-08-28  2:19     ` Wei Chen
2021-08-11 10:24 ` [XEN RFC PATCH 37/40] xen: introduce an arch helper to do NUMA init failed fallback Wei Chen
2021-08-27 14:30   ` Julien Grall
2021-08-28  3:09     ` Wei Chen
2021-08-28  3:45       ` Wei Chen
2021-08-30  9:52         ` Jan Beulich
2021-08-30 10:38           ` Wei Chen
2021-08-11 10:24 ` [XEN RFC PATCH 38/40] xen/arm: enable device tree based NUMA in system init Wei Chen
2021-08-27 14:32   ` Julien Grall
2021-08-28  3:17     ` Wei Chen
2021-08-28 10:45       ` Julien Grall
2021-08-28 14:02         ` Wei Chen
2021-08-31  1:50   ` Stefano Stabellini
2021-08-31 13:43     ` Wei Chen
2021-08-11 10:24 ` [XEN RFC PATCH 39/40] xen/x86: move numa_setup to common to support NUMA switch in command line Wei Chen
2021-08-27 14:37   ` Julien Grall
2021-08-28  3:22     ` Wei Chen
2021-08-31  1:53   ` Stefano Stabellini
2021-08-31 13:44     ` Wei Chen
2021-08-11 10:24 ` [XEN RFC PATCH 40/40] xen/x86: move dump_numa info hotkey to common Wei Chen
2021-08-11 10:41 ` [XEN RFC PATCH 00/40] Add device tree based NUMA support to Arm64 Jan Beulich
2021-08-13  2:33   ` Wei Chen
2021-08-13  6:53     ` Jan Beulich
2021-08-19 13:42 ` Julien Grall
2021-08-19 14:05   ` Bertrand Marquis
2021-08-19 17:11     ` Julien Grall
2021-08-26  0:09 ` Stefano Stabellini
2021-08-26  7:31   ` Wei Chen

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.