From mboxrd@z Thu Jan 1 00:00:00 1970 From: Wei Liu Subject: [PATCH v5 00/24] Virtual NUMA for PV and HVM Date: Thu, 12 Feb 2015 19:44:30 +0000 Message-ID: <1423770294-9779-1-git-send-email-wei.liu2@citrix.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: xen-devel@lists.xen.org Cc: Wei Liu , ian.campbell@citrix.com, andrew.cooper3@citrix.com, dario.faggioli@citrix.com, ian.jackson@eu.citrix.com, JBeulich@suse.com, ufimtseva@gmail.com List-Id: xen-devel@lists.xenproject.org Hi all This is version 5 of this series rebased on top of master. This patch series implements virtual NUMA support for both PV and HVM guest. That is, admin can configure via libxl what virtual NUMA topology the guest sees. This is the stage 1 (basic vNUMA support) and part of stage 2 (vNUMA-ware ballooning, hypervisor side) described in my previous email to xen-devel [0]. This series is broken into several parts: 1. xen patches: vNUMA debug output and vNUMA-aware memory hypercall support. 2. libxc/libxl support for PV vNUMA. 3. libxc/libxl/hypervisor support for HVM vNUMA. 4. xl vNUMA configuration documentation and parser. One significant difference from Elena's work is that this patch series makes use of multiple vmemranges should there be a memory hole, instead of shrinking ram. This matches the behaviour of real hardware. The vNUMA auto placement algorithm is missing at the moment and Dario is working on it. This series can be found at: git://xenbits.xen.org/people/liuw/xen.git wip.vnuma-v5 With this series, the following configuration can be used to enabled virtual NUMA support, and it works for both PV and HVM guests. vnuma = [ [ "pnode=0","size=3000","vcpus=0-3","vdistances=10,20" ], [ "pnode=0","size=3000","vcpus=4-7","vdistances=20,10" ], ] For example output of guest NUMA information, please look at [1]. In terms of libxl / libxc internal, things are broken into several parts: 1. libxl interface Users of libxl can only specify how many vnodes a guest can have, but currently they have no control over the actual memory layout. Note that it's fairly easy to export the interface to control memory layout in the future. 2. libxl internal It generates some internal vNUMA configurations when building domain, then transform them into libxc representations. It also validates vNUMA configuration along the line. 3. libxc internal Libxc does what it's told to do. It doesn't do anything smart (in fact, I delibrately didn't put any smart logic inside it). Libxc will also report back some information in HVM case to libxl but that's it. Wei. [0] <20141111173606.GC21312@zion.uk.xensource.com> [1] <1416582421-10789-1-git-send-email-wei.liu2@citrix.com> Changes in v5: 1. Rewrite PV memory allocation functions, take vmemranges into account. 2. Address Ian J's comments with regard to libxlu. 3. Address Jan and Andrew's comments with regard to hypervisor patches. 4. New syntax for vNUMA xl configuration. Changes in v4: 1. Address comments from many people. 2. Break down the libxlu patch into three. 3. Use dedicate patches for non-functional changes. Changes in v3: 1. Address comments made by Jan. 2. More commit messages and comments. 3. Shorten some error messages. Changes in v2: 1. Make vnuma_vdistances mandatory. 2. Use nested list to specify distances among nodes. 3. Hvmloader uses hypercall to retrieve vNUMA information. 4. Fix some problems spotted by Jan. Wei Liu (24): xen: dump vNUMA information with debug key "u" xen: make two memory hypercalls vNUMA-aware libxc: duplicate snippet to allocate p2m_host array libxc: add p2m_size to xc_dom_image libxc: allocate memory with vNUMA information for PV guest libxl: introduce vNUMA types libxl: add vmemrange to libxl__domain_build_state libxl: introduce libxl__vnuma_config_check libxl: x86: factor out e820_host_sanitize libxl: functions to build vmemranges for PV guest libxl: build, check and pass vNUMA info to Xen for PV guest hvmloader: retrieve vNUMA information from hypervisor hvmloader: construct SRAT hvmloader: construct SLIT libxc: indentation change to xc_hvm_build_x86.c libxc: allocate memory with vNUMA information for HVM guest libxl: build, check and pass vNUMA info to Xen for HVM guest libxl: disallow memory relocation when vNUMA is enabled libxl: define LIBXL_HAVE_VNUMA libxlu: rework internal representation of setting libxlu: nested list support libxlu: introduce new APIs xl: introduce xcalloc xl: vNUMA support docs/man/xl.cfg.pod.5 | 54 +++++++ tools/firmware/hvmloader/Makefile | 2 +- tools/firmware/hvmloader/acpi/acpi2_0.h | 61 ++++++++ tools/firmware/hvmloader/acpi/build.c | 110 +++++++++++++++ tools/firmware/hvmloader/hvmloader.c | 3 + tools/firmware/hvmloader/vnuma.c | 84 +++++++++++ tools/firmware/hvmloader/vnuma.h | 52 +++++++ tools/libxc/include/xc_dom.h | 7 + tools/libxc/include/xenguest.h | 11 ++ tools/libxc/xc_dom_arm.c | 1 + tools/libxc/xc_dom_core.c | 8 +- tools/libxc/xc_dom_x86.c | 132 ++++++++++++++---- tools/libxc/xc_hvm_build_x86.c | 240 +++++++++++++++++++++----------- tools/libxc/xc_private.h | 2 + tools/libxl/Makefile | 2 +- tools/libxl/libxl.h | 6 + tools/libxl/libxl_arch.h | 6 + tools/libxl/libxl_arm.c | 8 ++ tools/libxl/libxl_create.c | 9 ++ tools/libxl/libxl_dm.c | 6 +- tools/libxl/libxl_dom.c | 114 +++++++++++++++ tools/libxl/libxl_internal.h | 23 +++ tools/libxl/libxl_types.idl | 10 ++ tools/libxl/libxl_vnuma.c | 228 ++++++++++++++++++++++++++++++ tools/libxl/libxl_x86.c | 105 ++++++++++++-- tools/libxl/libxlu_cfg.c | 199 +++++++++++++++++++------- tools/libxl/libxlu_cfg_i.h | 13 +- tools/libxl/libxlu_cfg_y.c | 46 +++--- tools/libxl/libxlu_cfg_y.h | 2 +- tools/libxl/libxlu_cfg_y.y | 14 +- tools/libxl/libxlu_internal.h | 23 ++- tools/libxl/libxlutil.h | 13 ++ tools/libxl/xl_cmdimpl.c | 151 +++++++++++++++++++- xen/arch/x86/numa.c | 71 +++++++++- xen/common/kernel.c | 2 +- xen/common/memory.c | 51 ++++++- xen/include/public/features.h | 3 + xen/include/public/memory.h | 2 + 38 files changed, 1655 insertions(+), 219 deletions(-) create mode 100644 tools/firmware/hvmloader/vnuma.c create mode 100644 tools/firmware/hvmloader/vnuma.h create mode 100644 tools/libxl/libxl_vnuma.c -- 1.9.1