From: Marco Solieri <marco.solieri@minervasys.tech>
To: xen-devel@lists.xenproject.org
Cc: Marco Solieri <marco.solieri@minervasys.tech>,
Andrew Cooper <andrew.cooper3@citrix.com>,
George Dunlap <george.dunlap@citrix.com>,
Jan Beulich <jbeulich@suse.com>, Julien Grall <julien@xen.org>,
Stefano Stabellini <sstabellini@kernel.org>, Wei Liu <wl@xen.org>,
Marco Solieri <marco.solieri@unimore.it>,
Andrea Bastoni <andrea.bastoni@minervasys.tech>
Subject: [PATCH 00/36] Arm cache coloring
Date: Fri, 4 Mar 2022 18:46:25 +0100 [thread overview]
Message-ID: <20220304174701.1453977-1-marco.solieri@minervasys.tech> (raw)
Shared caches in multi-core CPU architectures represent a problem for
predictability of memory access latency. This jeopardizes applicability
of many Arm platform in real-time critical and mixed-criticality
scenarios. We introduce support for cache partitioning with page
coloring, a transparent software technique that enables isolation
between domains and Xen, and thus avoids cache interference.
When creating a domain, a simple syntax (e.g. `0-3` or `4-11`) allows
the user to define assignments of cache partitions ids, called colors,
where assigning different colors guarantees no mutual eviction on cache
will ever happen. This instructs the Xen memory allocator to provide
the i-th color assignee only with pages that maps to color i, i.e. that
are indexed in the i-th cache partition.
The proposed implementation supports the dom0less experimental feature.
The solution has been tested in several scenarios, including Xilinx Zynq
MPSoCs.
Overview of implementation and commits structure
------------------------------------------------
- Coloring support is added for dom0 and domU by defining the core
logic, as well as the hardware inspection functionalities used for
getting needed coloring information [4-17].
- A new memory page allocator that implement the cache coloring
mechanism is introduced. The allocation algorithm follows the given
coloring scheme specified for each domain, and maximizes contiguity in
the page selection [18-21].
- Coloring support is added to Xen .text region [22-29], as well as to
dom0less domains [30].
- Extensive documentation details the technique and gently explains
usage [33-36].
Known limitations
-----------------
- We need to bring back [1-3] the relocation feature in order to move
Xen memory to a colored space where the hypervisor could be isolated
from VMs interference.
- When cache coloring is used, static memory assignment is disabled to
avoid incompatibility. [31]
- Due to assert failure [32], the number of supported colors is
currently limited at 64, which should be satisfactory for most chips.
In particular, the problem lies in the cache coloring configuration
data structure that belongs to each domain. We are aware that this is
not a clean solution but we hope that this could be discussed and
solved within this pull request.
Acknowledgements
----------------
This work is sponsored by Xilinx Inc., and supported by University of
Modena and Reggio Emilia and Minerva Systems.
***
Luca Miccio (36):
Revert "xen/arm: setup: Add Xen as boot module before printing all
boot modules"
Revert "xen/arm: mm: Initialize page-tables earlier"
xen/arm: restore xen_paddr argument in setup_pagetables
xen/arm: add parsing function for cache coloring configuration
xen/arm: compute LLC way size by hardware inspection
xen/arm: add coloring basic initialization
xen/arm: add coloring data to domains
xen/arm: add colored flag to page struct
xen/arch: add default colors selection function
xen/arch: check color selection function
xen/include: define hypercall parameter for coloring
xen/arm: initialize cache coloring data for Dom0/U
xen/arm: A domain is not direct mapped when coloring is enabled
xen/arch: add dump coloring info for domains
tools: add support for cache coloring configuration
xen/color alloc: implement color_from_page for ARM64
xen/arm: add get_max_color function
Alloc: introduce page_list_for_each_reverse
xen/arch: introduce cache-coloring allocator
xen/common: introduce buddy required reservation
xen/common: add colored allocator initialization
xen/arch: init cache coloring conf for Xen
xen/arch: coloring: manually calculate Xen physical addresses
xen/arm: enable consider_modules for coloring
xen/arm: bring back get_xen_paddr
xen/arm: add argument to remove_early_mappings
xen/arch: add coloring support for Xen
xen/arm: introduce xen_map_text_rw
xen/arm: add dump function for coloring info
xen/arm: add coloring support to dom0less
Disable coloring if static memory support is selected
xen/arm: reduce the number of supported colors
doc, xen-command-line: introduce coloring options
doc, xl.cfg: introduce coloring configuration option
doc, device-tree: introduce 'colors' property
doc, arm: add usage documentation for cache coloring support
docs/man/xl.cfg.5.pod.in | 14 +
docs/misc/arm/cache_coloring.rst | 191 +++++++++++
docs/misc/arm/device-tree/booting.txt | 3 +
docs/misc/xen-command-line.pandoc | 51 ++-
tools/libs/light/libxl_arm.c | 11 +
tools/libs/light/libxl_types.idl | 1 +
tools/xl/xl_parse.c | 59 +++-
xen/arch/arm/Kconfig | 6 +
xen/arch/arm/Makefile | 2 +-
xen/arch/arm/alternative.c | 8 +-
xen/arch/arm/coloring.c | 469 ++++++++++++++++++++++++++
xen/arch/arm/domain.c | 56 +++
xen/arch/arm/domain_build.c | 42 ++-
xen/arch/arm/include/asm/coloring.h | 98 ++++++
xen/arch/arm/include/asm/mm.h | 18 +-
xen/arch/arm/mm.c | 245 +++++++++++++-
xen/arch/arm/psci.c | 4 +-
xen/arch/arm/setup.c | 94 +++++-
xen/arch/arm/smpboot.c | 19 +-
xen/common/page_alloc.c | 321 +++++++++++++++++-
xen/common/vmap.c | 4 +-
xen/include/public/arch-arm.h | 8 +
xen/include/xen/mm.h | 7 +
xen/include/xen/sched.h | 4 +
xen/include/xen/vmap.h | 2 +
25 files changed, 1689 insertions(+), 48 deletions(-)
create mode 100644 docs/misc/arm/cache_coloring.rst
create mode 100644 xen/arch/arm/coloring.c
create mode 100644 xen/arch/arm/include/asm/coloring.h
--
2.30.2
next reply other threads:[~2022-03-04 18:17 UTC|newest]
Thread overview: 79+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-03-04 17:46 Marco Solieri [this message]
2022-03-04 17:46 ` [PATCH 01/36] Revert "xen/arm: setup: Add Xen as boot module before printing all boot modules" Marco Solieri
2022-03-04 18:50 ` Julien Grall
2022-03-04 17:46 ` [PATCH 02/36] Revert "xen/arm: mm: Initialize page-tables earlier" Marco Solieri
2022-03-04 17:46 ` [PATCH 03/36] xen/arm: restore xen_paddr argument in setup_pagetables Marco Solieri
2022-03-04 17:46 ` [PATCH 04/36] xen/arm: add parsing function for cache coloring configuration Marco Solieri
2022-03-09 19:09 ` Julien Grall
2022-03-22 9:17 ` Luca Miccio
2022-03-23 19:02 ` Julien Grall
2022-05-13 14:22 ` Carlo Nonato
2022-05-13 17:41 ` Julien Grall
2022-03-04 17:46 ` [PATCH 05/36] xen/arm: compute LLC way size by hardware inspection Marco Solieri
2022-03-09 20:12 ` Julien Grall
2022-05-13 14:34 ` Carlo Nonato
2022-05-13 19:08 ` Julien Grall
2022-03-04 17:46 ` [PATCH 06/36] xen/arm: add coloring basic initialization Marco Solieri
2022-03-04 17:46 ` [PATCH 07/36] xen/arm: add coloring data to domains Marco Solieri
2022-03-07 7:22 ` Jan Beulich
2022-03-04 17:46 ` [PATCH 08/36] xen/arm: add colored flag to page struct Marco Solieri
2022-03-04 20:13 ` Julien Grall
2022-03-04 17:46 ` [PATCH 09/36] xen/arch: add default colors selection function Marco Solieri
2022-03-07 7:28 ` Jan Beulich
2022-03-04 17:46 ` [PATCH 10/36] xen/arch: check color " Marco Solieri
2022-03-09 20:17 ` Julien Grall
2022-03-14 6:06 ` Henry Wang
2022-03-04 17:46 ` [PATCH 11/36] xen/include: define hypercall parameter for coloring Marco Solieri
2022-03-07 7:31 ` Jan Beulich
2022-03-09 20:29 ` Julien Grall
2022-03-04 17:46 ` [PATCH 12/36] xen/arm: initialize cache coloring data for Dom0/U Marco Solieri
2022-03-11 19:05 ` Julien Grall
2022-03-04 17:46 ` [PATCH 13/36] xen/arm: A domain is not direct mapped when coloring is enabled Marco Solieri
2022-03-09 20:34 ` Julien Grall
2022-03-04 17:46 ` [PATCH 14/36] xen/arch: add dump coloring info for domains Marco Solieri
2022-03-04 17:46 ` [PATCH 15/36] tools: add support for cache coloring configuration Marco Solieri
2022-03-04 17:46 ` [PATCH 16/36] xen/color alloc: implement color_from_page for ARM64 Marco Solieri
2022-03-04 20:54 ` Julien Grall
2022-03-11 17:39 ` Marco Solieri
2022-03-11 17:57 ` Julien Grall
2022-03-04 17:46 ` [PATCH 17/36] xen/arm: add get_max_color function Marco Solieri
2022-03-11 19:09 ` Julien Grall
2022-03-04 17:46 ` [PATCH 18/36] Alloc: introduce page_list_for_each_reverse Marco Solieri
2022-03-07 7:35 ` Jan Beulich
2022-03-04 17:46 ` [PATCH 19/36] xen/arch: introduce cache-coloring allocator Marco Solieri
2022-03-09 14:35 ` Jan Beulich
2022-03-04 17:46 ` [PATCH 20/36] xen/common: introduce buddy required reservation Marco Solieri
2022-03-09 14:45 ` Jan Beulich
2022-03-09 14:47 ` Jan Beulich
2022-03-04 17:46 ` [PATCH 21/36] xen/common: add colored allocator initialization Marco Solieri
2022-03-09 14:58 ` Jan Beulich
2022-03-04 17:46 ` [PATCH 22/36] xen/arch: init cache coloring conf for Xen Marco Solieri
2022-03-14 18:59 ` Julien Grall
2022-03-04 17:46 ` [PATCH 23/36] xen/arch: coloring: manually calculate Xen physical addresses Marco Solieri
2022-03-14 19:23 ` Julien Grall
2022-03-04 17:46 ` [PATCH 24/36] xen/arm: enable consider_modules for coloring Marco Solieri
2022-03-14 19:24 ` Julien Grall
2022-03-04 17:46 ` [PATCH 25/36] xen/arm: bring back get_xen_paddr Marco Solieri
2022-03-04 17:46 ` [PATCH 26/36] xen/arm: add argument to remove_early_mappings Marco Solieri
2022-03-14 19:59 ` Julien Grall
2022-03-04 17:46 ` [PATCH 27/36] xen/arch: add coloring support for Xen Marco Solieri
2022-03-04 19:47 ` Julien Grall
2022-03-09 11:28 ` Julien Grall
2022-03-14 3:47 ` Henry Wang
2022-03-14 21:58 ` Julien Grall
2022-03-04 17:46 ` [PATCH 28/36] xen/arm: introduce xen_map_text_rw Marco Solieri
2022-03-07 7:39 ` Jan Beulich
2022-03-11 22:28 ` Julien Grall
2022-03-04 17:46 ` [PATCH 29/36] xen/arm: add dump function for coloring info Marco Solieri
2022-03-04 17:46 ` [PATCH 30/36] xen/arm: add coloring support to dom0less Marco Solieri
2022-03-04 17:46 ` [PATCH 31/36] Disable coloring if static memory support is selected Marco Solieri
2022-03-14 20:04 ` Julien Grall
2022-03-04 17:46 ` [PATCH 32/36] xen/arm: reduce the number of supported colors Marco Solieri
2022-03-04 17:46 ` [PATCH 33/36] doc, xen-command-line: introduce coloring options Marco Solieri
2022-03-07 7:42 ` Jan Beulich
2022-03-14 22:07 ` Julien Grall
2022-03-04 17:46 ` [PATCH 34/36] doc, xl.cfg: introduce coloring configuration option Marco Solieri
2022-03-04 17:47 ` [PATCH 35/36] doc, device-tree: introduce 'colors' property Marco Solieri
2022-03-14 22:17 ` Julien Grall
2022-03-04 17:47 ` [PATCH 36/36] doc, arm: add usage documentation for cache coloring support Marco Solieri
2022-03-15 19:23 ` Julien Grall
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220304174701.1453977-1-marco.solieri@minervasys.tech \
--to=marco.solieri@minervasys.tech \
--cc=andrea.bastoni@minervasys.tech \
--cc=andrew.cooper3@citrix.com \
--cc=george.dunlap@citrix.com \
--cc=jbeulich@suse.com \
--cc=julien@xen.org \
--cc=marco.solieri@unimore.it \
--cc=sstabellini@kernel.org \
--cc=wl@xen.org \
--cc=xen-devel@lists.xenproject.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.