* [PATCH v3 00/10] Independent per-CPU data section for nVHE
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Suzuki K Poulose, Catalin Marinas, linux-kernel,
James Morse, linux-arm-kernel, Marc Zyngier, Tejun Heo,
Dennis Zhou, Christoph Lameter, David Brazdil, Will Deacon,
Julien Thierry
Introduce '.hyp.data..percpu' as part of ongoing effort to make nVHE
hyp code self-contained and independent of the rest of the kernel.
Main benefits:
* independent nVHE per-CPU data section that can be unmapped from host,
* more robust linking of nVHE hyp code,
* no need for hyp-specific macros to access per-CPU variables.
The series is structured as follows:
- patch 1: Improve existing hyp build rules. This could be sent and merged
independently of per-CPU but this series builds on it.
- patches 2-3: Minor cleanups.
- patches 4-5: Replace hyp helpers for accessing per-CPU variables
with common helpers modified to work correctly in hyp. Per-CPU
variables can now be accessed with one API anywhere.
- patches 6-8: Where VHE and nVHE use per-CPU variables defined in
kernel proper, move their definitions to hyp/ where they are
duplicated and owned by VHE/nVHE, respectively. Non-VHE hyp code
now refers only to per-CPU variables defined in its source files.
Helpers are added so that kernel proper can continue to access
nVHE hyp variables, same way as it does with other nVHE symbols.
- patches 9-11: Introduce '.hyp.data..percpu' ELF section and allocate
memory for every CPU core during KVM init. All nVHE per-CPU state
is now grouped together in ELF and in memory. Introducing a new
per-CPU variable does not require adding new memory mappings any
more. nVHE hyp code cannot accidentally refer to kernel-proper
per-CPU data as it only has the pointer to its own per-CPU memory.
Patches are rebased on v5.9-rc5 and available in branch 'topic/percpu-v3' at:
https://android-kvm.googlesource.com/linux
For maintainers: In case of interest, there are patches that remove the need
for redefining macros under DEBUG_PREEMPT available at the same repo, branch
'topic/percpu-v3-debug-preempt'. Since they are non-trivial, I am not going
to post them here so late in the 5.10 window. I plan to post them for 5.11
when they will also be useful for other patches.
Changes v2 -> v3:
* Use PERCPU_INPUT in hyp.ld instead of modifying PERCPU_SECTION
* Only pass linker script once to LD (fix error message)
* Renamed '.hyp.o' to '.nvhe.o'
* Use __KVM_VHE_HYPERVISOR__ to select TPIDR_EL2 instead of alternatives
* Move all prefixing-related macros to hyp_image.h
Changes v1 -> v2:
* partially link hyp code, add linker script
David Brazdil (11):
kvm: arm64: Partially link nVHE hyp code, simplify HYPCOPY
kvm: arm64: Move nVHE hyp namespace macros to hyp_image.h
kvm: arm64: Only define __kvm_ex_table for CONFIG_KVM
kvm: arm64: Remove __hyp_this_cpu_read
kvm: arm64: Remove hyp_adr/ldr_this_cpu
kvm: arm64: Add helpers for accessing nVHE hyp per-cpu vars
kvm: arm64: Duplicate arm64_ssbd_callback_required for nVHE hyp
kvm: arm64: Create separate instances of kvm_host_data for VHE/nVHE
kvm: arm64: Mark hyp stack pages reserved
kvm: arm64: Set up hyp percpu data for nVHE
kvm: arm64: Remove unnecessary hyp mappings
arch/arm64/include/asm/assembler.h | 27 ++++--
arch/arm64/include/asm/hyp_image.h | 36 +++++++
arch/arm64/include/asm/kvm_asm.h | 82 ++++++++--------
arch/arm64/include/asm/kvm_host.h | 2 +-
arch/arm64/include/asm/kvm_mmu.h | 19 ++--
arch/arm64/include/asm/percpu.h | 33 ++++++-
arch/arm64/include/asm/sections.h | 1 +
arch/arm64/kernel/image-vars.h | 4 -
arch/arm64/kernel/vmlinux.lds.S | 13 +++
arch/arm64/kvm/arm.c | 109 +++++++++++++++++++---
arch/arm64/kvm/hyp/hyp-entry.S | 2 +-
arch/arm64/kvm/hyp/include/hyp/debug-sr.h | 4 +-
arch/arm64/kvm/hyp/include/hyp/switch.h | 8 +-
arch/arm64/kvm/hyp/nvhe/Makefile | 60 ++++++------
arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 19 ++++
arch/arm64/kvm/hyp/nvhe/switch.c | 8 +-
arch/arm64/kvm/hyp/vhe/switch.c | 5 +-
arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 4 +-
arch/arm64/kvm/pmu.c | 13 ++-
19 files changed, 320 insertions(+), 129 deletions(-)
create mode 100644 arch/arm64/include/asm/hyp_image.h
create mode 100644 arch/arm64/kvm/hyp/nvhe/hyp.lds.S
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 87+ messages in thread* [PATCH v3 00/10] Independent per-CPU data section for nVHE
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Catalin Marinas, linux-kernel, linux-arm-kernel,
Marc Zyngier, Tejun Heo, Dennis Zhou, Christoph Lameter,
Will Deacon
Introduce '.hyp.data..percpu' as part of ongoing effort to make nVHE
hyp code self-contained and independent of the rest of the kernel.
Main benefits:
* independent nVHE per-CPU data section that can be unmapped from host,
* more robust linking of nVHE hyp code,
* no need for hyp-specific macros to access per-CPU variables.
The series is structured as follows:
- patch 1: Improve existing hyp build rules. This could be sent and merged
independently of per-CPU but this series builds on it.
- patches 2-3: Minor cleanups.
- patches 4-5: Replace hyp helpers for accessing per-CPU variables
with common helpers modified to work correctly in hyp. Per-CPU
variables can now be accessed with one API anywhere.
- patches 6-8: Where VHE and nVHE use per-CPU variables defined in
kernel proper, move their definitions to hyp/ where they are
duplicated and owned by VHE/nVHE, respectively. Non-VHE hyp code
now refers only to per-CPU variables defined in its source files.
Helpers are added so that kernel proper can continue to access
nVHE hyp variables, same way as it does with other nVHE symbols.
- patches 9-11: Introduce '.hyp.data..percpu' ELF section and allocate
memory for every CPU core during KVM init. All nVHE per-CPU state
is now grouped together in ELF and in memory. Introducing a new
per-CPU variable does not require adding new memory mappings any
more. nVHE hyp code cannot accidentally refer to kernel-proper
per-CPU data as it only has the pointer to its own per-CPU memory.
Patches are rebased on v5.9-rc5 and available in branch 'topic/percpu-v3' at:
https://android-kvm.googlesource.com/linux
For maintainers: In case of interest, there are patches that remove the need
for redefining macros under DEBUG_PREEMPT available at the same repo, branch
'topic/percpu-v3-debug-preempt'. Since they are non-trivial, I am not going
to post them here so late in the 5.10 window. I plan to post them for 5.11
when they will also be useful for other patches.
Changes v2 -> v3:
* Use PERCPU_INPUT in hyp.ld instead of modifying PERCPU_SECTION
* Only pass linker script once to LD (fix error message)
* Renamed '.hyp.o' to '.nvhe.o'
* Use __KVM_VHE_HYPERVISOR__ to select TPIDR_EL2 instead of alternatives
* Move all prefixing-related macros to hyp_image.h
Changes v1 -> v2:
* partially link hyp code, add linker script
David Brazdil (11):
kvm: arm64: Partially link nVHE hyp code, simplify HYPCOPY
kvm: arm64: Move nVHE hyp namespace macros to hyp_image.h
kvm: arm64: Only define __kvm_ex_table for CONFIG_KVM
kvm: arm64: Remove __hyp_this_cpu_read
kvm: arm64: Remove hyp_adr/ldr_this_cpu
kvm: arm64: Add helpers for accessing nVHE hyp per-cpu vars
kvm: arm64: Duplicate arm64_ssbd_callback_required for nVHE hyp
kvm: arm64: Create separate instances of kvm_host_data for VHE/nVHE
kvm: arm64: Mark hyp stack pages reserved
kvm: arm64: Set up hyp percpu data for nVHE
kvm: arm64: Remove unnecessary hyp mappings
arch/arm64/include/asm/assembler.h | 27 ++++--
arch/arm64/include/asm/hyp_image.h | 36 +++++++
arch/arm64/include/asm/kvm_asm.h | 82 ++++++++--------
arch/arm64/include/asm/kvm_host.h | 2 +-
arch/arm64/include/asm/kvm_mmu.h | 19 ++--
arch/arm64/include/asm/percpu.h | 33 ++++++-
arch/arm64/include/asm/sections.h | 1 +
arch/arm64/kernel/image-vars.h | 4 -
arch/arm64/kernel/vmlinux.lds.S | 13 +++
arch/arm64/kvm/arm.c | 109 +++++++++++++++++++---
arch/arm64/kvm/hyp/hyp-entry.S | 2 +-
arch/arm64/kvm/hyp/include/hyp/debug-sr.h | 4 +-
arch/arm64/kvm/hyp/include/hyp/switch.h | 8 +-
arch/arm64/kvm/hyp/nvhe/Makefile | 60 ++++++------
arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 19 ++++
arch/arm64/kvm/hyp/nvhe/switch.c | 8 +-
arch/arm64/kvm/hyp/vhe/switch.c | 5 +-
arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 4 +-
arch/arm64/kvm/pmu.c | 13 ++-
19 files changed, 320 insertions(+), 129 deletions(-)
create mode 100644 arch/arm64/include/asm/hyp_image.h
create mode 100644 arch/arm64/kvm/hyp/nvhe/hyp.lds.S
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply [flat|nested] 87+ messages in thread* [PATCH v3 01/11] kvm: arm64: Partially link nVHE hyp code, simplify HYPCOPY
2020-09-16 17:34 ` David Brazdil
(?)
@ 2020-09-16 17:34 ` David Brazdil
-1 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: Catalin Marinas, Will Deacon, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team,
David Brazdil
Relying on objcopy to prefix the ELF section names of the nVHE hyp code
is brittle and prevents us from using wildcards to match specific
section names.
Improve the build rules by partially linking all '.nvhe.o' files and
prefixing their ELF section names using a linker script. Continue using
objcopy for prefixing ELF symbol names.
One immediate advantage of this approach is that all subsections
matching a pattern can be merged into a single prefixed section, eg.
.text and .text.* can be linked into a single '.hyp.text'. This removes
the need for -fno-reorder-functions on GCC and will be useful in the
future too: LTO builds use .text subsections, compilers routinely
generate .rodata subsections, etc.
Partially linking all hyp code into a single object file also makes it
easier to analyze.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/hyp_image.h | 24 ++++++++++++
arch/arm64/kvm/hyp/nvhe/Makefile | 60 ++++++++++++++++--------------
arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 13 +++++++
3 files changed, 70 insertions(+), 27 deletions(-)
create mode 100644 arch/arm64/include/asm/hyp_image.h
create mode 100644 arch/arm64/kvm/hyp/nvhe/hyp.lds.S
diff --git a/arch/arm64/include/asm/hyp_image.h b/arch/arm64/include/asm/hyp_image.h
new file mode 100644
index 000000000000..5b1e3b9ef376
--- /dev/null
+++ b/arch/arm64/include/asm/hyp_image.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Google LLC.
+ * Written by David Brazdil <dbrazdil@google.com>
+ */
+
+#ifndef __ARM64_HYP_IMAGE_H__
+#define __ARM64_HYP_IMAGE_H__
+
+#ifdef LINKER_SCRIPT
+
+/*
+ * KVM nVHE ELF section names are prefixed with .hyp, to separate them
+ * from the kernel proper.
+ */
+#define HYP_SECTION_NAME(NAME) .hyp##NAME
+
+/* Defines an ELF hyp section from input section @NAME and its subsections. */
+#define HYP_SECTION(NAME) \
+ HYP_SECTION_NAME(NAME) : { *(NAME NAME##.[0-9a-zA-Z_]*) }
+
+#endif /* LINKER_SCRIPT */
+
+#endif /* __ARM64_HYP_IMAGE_H__ */
diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile
index aef76487edc2..2b27b60182f9 100644
--- a/arch/arm64/kvm/hyp/nvhe/Makefile
+++ b/arch/arm64/kvm/hyp/nvhe/Makefile
@@ -10,40 +10,46 @@ obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o
obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \
../fpsimd.o ../hyp-entry.o
-obj-y := $(patsubst %.o,%.hyp.o,$(obj-y))
-extra-y := $(patsubst %.hyp.o,%.hyp.tmp.o,$(obj-y))
+##
+## Build rules for compiling nVHE hyp code
+## Output of this folder is `kvm_nvhe.o`, a partially linked object
+## file containing all nVHE hyp code and data.
+##
-$(obj)/%.hyp.tmp.o: $(src)/%.c FORCE
+hyp-obj := $(patsubst %.o,%.nvhe.o,$(obj-y))
+obj-y := kvm_nvhe.o
+extra-y := $(hyp-obj) kvm_nvhe.tmp.o hyp.lds
+
+# 1) Compile all source files to `.nvhe.o` object files. The file extension
+# avoids file name clashes for files shared with VHE.
+$(obj)/%.nvhe.o: $(src)/%.c FORCE
$(call if_changed_rule,cc_o_c)
-$(obj)/%.hyp.tmp.o: $(src)/%.S FORCE
+$(obj)/%.nvhe.o: $(src)/%.S FORCE
$(call if_changed_rule,as_o_S)
-$(obj)/%.hyp.o: $(obj)/%.hyp.tmp.o FORCE
- $(call if_changed,hypcopy)
-# Disable reordering functions by GCC (enabled at -O2).
-# This pass puts functions into '.text.*' sections to aid the linker
-# in optimizing ELF layout. See HYPCOPY comment below for more info.
-ccflags-y += $(call cc-option,-fno-reorder-functions)
+# 2) Compile linker script.
+$(obj)/hyp.lds: $(src)/hyp.lds.S FORCE
+ $(call if_changed_dep,cpp_lds_S)
+
+# 3) Partially link all '.nvhe.o' files and apply the linker script.
+# Prefixes names of ELF sections with '.hyp', eg. '.hyp.text'.
+# Note: The following rule assumes that the 'ld' rule puts LDFLAGS before
+# the list of dependencies to form '-T $(obj)/hyp.lds'. This is to
+# keep the dependency on the target while avoiding an error from
+# GNU ld if the linker script is passed to it twice.
+LDFLAGS_kvm_nvhe.tmp.o := -r -T
+$(obj)/kvm_nvhe.tmp.o: $(obj)/hyp.lds $(addprefix $(obj)/,$(hyp-obj)) FORCE
+ $(call if_changed,ld)
+
+# 4) Produce the final 'kvm_nvhe.o', ready to be linked into 'vmlinux'.
+# Prefixes names of ELF symbols with '__kvm_nvhe_'.
+$(obj)/kvm_nvhe.o: $(obj)/kvm_nvhe.tmp.o FORCE
+ $(call if_changed,hypcopy)
# The HYPCOPY command uses `objcopy` to prefix all ELF symbol names
-# and relevant ELF section names to avoid clashes with VHE code/data.
-#
-# Hyp code is assumed to be in the '.text' section of the input object
-# files (with the exception of specialized sections such as
-# '.hyp.idmap.text'). This assumption may be broken by a compiler that
-# divides code into sections like '.text.unlikely' so as to optimize
-# ELF layout. HYPCOPY checks that no such sections exist in the input
-# using `objdump`, otherwise they would be linked together with other
-# kernel code and not memory-mapped correctly at runtime.
+# to avoid clashes with VHE code/data.
quiet_cmd_hypcopy = HYPCOPY $@
- cmd_hypcopy = \
- if $(OBJDUMP) -h $< | grep -F '.text.'; then \
- echo "$@: function reordering not supported in nVHE hyp code" >&2; \
- /bin/false; \
- fi; \
- $(OBJCOPY) --prefix-symbols=__kvm_nvhe_ \
- --rename-section=.text=.hyp.text \
- $< $@
+ cmd_hypcopy = $(OBJCOPY) --prefix-symbols=__kvm_nvhe_ $< $@
# Remove ftrace and Shadow Call Stack CFLAGS.
# This is equivalent to the 'notrace' and '__noscs' annotations.
diff --git a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S
new file mode 100644
index 000000000000..3b13d1c7cd1a
--- /dev/null
+++ b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Google LLC.
+ * Written by David Brazdil <dbrazdil@google.com>
+ *
+ * Linker script used for partial linking of nVHE EL2 object files.
+ */
+
+#include <asm/hyp_image.h>
+
+SECTIONS {
+ HYP_SECTION(.text)
+}
--
2.28.0.618.gf4bc123cb7-goog
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v3 01/11] kvm: arm64: Partially link nVHE hyp code, simplify HYPCOPY
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Suzuki K Poulose, Catalin Marinas, linux-kernel,
James Morse, linux-arm-kernel, Marc Zyngier, Tejun Heo,
Dennis Zhou, Christoph Lameter, David Brazdil, Will Deacon,
Julien Thierry
Relying on objcopy to prefix the ELF section names of the nVHE hyp code
is brittle and prevents us from using wildcards to match specific
section names.
Improve the build rules by partially linking all '.nvhe.o' files and
prefixing their ELF section names using a linker script. Continue using
objcopy for prefixing ELF symbol names.
One immediate advantage of this approach is that all subsections
matching a pattern can be merged into a single prefixed section, eg.
.text and .text.* can be linked into a single '.hyp.text'. This removes
the need for -fno-reorder-functions on GCC and will be useful in the
future too: LTO builds use .text subsections, compilers routinely
generate .rodata subsections, etc.
Partially linking all hyp code into a single object file also makes it
easier to analyze.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/hyp_image.h | 24 ++++++++++++
arch/arm64/kvm/hyp/nvhe/Makefile | 60 ++++++++++++++++--------------
arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 13 +++++++
3 files changed, 70 insertions(+), 27 deletions(-)
create mode 100644 arch/arm64/include/asm/hyp_image.h
create mode 100644 arch/arm64/kvm/hyp/nvhe/hyp.lds.S
diff --git a/arch/arm64/include/asm/hyp_image.h b/arch/arm64/include/asm/hyp_image.h
new file mode 100644
index 000000000000..5b1e3b9ef376
--- /dev/null
+++ b/arch/arm64/include/asm/hyp_image.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Google LLC.
+ * Written by David Brazdil <dbrazdil@google.com>
+ */
+
+#ifndef __ARM64_HYP_IMAGE_H__
+#define __ARM64_HYP_IMAGE_H__
+
+#ifdef LINKER_SCRIPT
+
+/*
+ * KVM nVHE ELF section names are prefixed with .hyp, to separate them
+ * from the kernel proper.
+ */
+#define HYP_SECTION_NAME(NAME) .hyp##NAME
+
+/* Defines an ELF hyp section from input section @NAME and its subsections. */
+#define HYP_SECTION(NAME) \
+ HYP_SECTION_NAME(NAME) : { *(NAME NAME##.[0-9a-zA-Z_]*) }
+
+#endif /* LINKER_SCRIPT */
+
+#endif /* __ARM64_HYP_IMAGE_H__ */
diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile
index aef76487edc2..2b27b60182f9 100644
--- a/arch/arm64/kvm/hyp/nvhe/Makefile
+++ b/arch/arm64/kvm/hyp/nvhe/Makefile
@@ -10,40 +10,46 @@ obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o
obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \
../fpsimd.o ../hyp-entry.o
-obj-y := $(patsubst %.o,%.hyp.o,$(obj-y))
-extra-y := $(patsubst %.hyp.o,%.hyp.tmp.o,$(obj-y))
+##
+## Build rules for compiling nVHE hyp code
+## Output of this folder is `kvm_nvhe.o`, a partially linked object
+## file containing all nVHE hyp code and data.
+##
-$(obj)/%.hyp.tmp.o: $(src)/%.c FORCE
+hyp-obj := $(patsubst %.o,%.nvhe.o,$(obj-y))
+obj-y := kvm_nvhe.o
+extra-y := $(hyp-obj) kvm_nvhe.tmp.o hyp.lds
+
+# 1) Compile all source files to `.nvhe.o` object files. The file extension
+# avoids file name clashes for files shared with VHE.
+$(obj)/%.nvhe.o: $(src)/%.c FORCE
$(call if_changed_rule,cc_o_c)
-$(obj)/%.hyp.tmp.o: $(src)/%.S FORCE
+$(obj)/%.nvhe.o: $(src)/%.S FORCE
$(call if_changed_rule,as_o_S)
-$(obj)/%.hyp.o: $(obj)/%.hyp.tmp.o FORCE
- $(call if_changed,hypcopy)
-# Disable reordering functions by GCC (enabled at -O2).
-# This pass puts functions into '.text.*' sections to aid the linker
-# in optimizing ELF layout. See HYPCOPY comment below for more info.
-ccflags-y += $(call cc-option,-fno-reorder-functions)
+# 2) Compile linker script.
+$(obj)/hyp.lds: $(src)/hyp.lds.S FORCE
+ $(call if_changed_dep,cpp_lds_S)
+
+# 3) Partially link all '.nvhe.o' files and apply the linker script.
+# Prefixes names of ELF sections with '.hyp', eg. '.hyp.text'.
+# Note: The following rule assumes that the 'ld' rule puts LDFLAGS before
+# the list of dependencies to form '-T $(obj)/hyp.lds'. This is to
+# keep the dependency on the target while avoiding an error from
+# GNU ld if the linker script is passed to it twice.
+LDFLAGS_kvm_nvhe.tmp.o := -r -T
+$(obj)/kvm_nvhe.tmp.o: $(obj)/hyp.lds $(addprefix $(obj)/,$(hyp-obj)) FORCE
+ $(call if_changed,ld)
+
+# 4) Produce the final 'kvm_nvhe.o', ready to be linked into 'vmlinux'.
+# Prefixes names of ELF symbols with '__kvm_nvhe_'.
+$(obj)/kvm_nvhe.o: $(obj)/kvm_nvhe.tmp.o FORCE
+ $(call if_changed,hypcopy)
# The HYPCOPY command uses `objcopy` to prefix all ELF symbol names
-# and relevant ELF section names to avoid clashes with VHE code/data.
-#
-# Hyp code is assumed to be in the '.text' section of the input object
-# files (with the exception of specialized sections such as
-# '.hyp.idmap.text'). This assumption may be broken by a compiler that
-# divides code into sections like '.text.unlikely' so as to optimize
-# ELF layout. HYPCOPY checks that no such sections exist in the input
-# using `objdump`, otherwise they would be linked together with other
-# kernel code and not memory-mapped correctly at runtime.
+# to avoid clashes with VHE code/data.
quiet_cmd_hypcopy = HYPCOPY $@
- cmd_hypcopy = \
- if $(OBJDUMP) -h $< | grep -F '.text.'; then \
- echo "$@: function reordering not supported in nVHE hyp code" >&2; \
- /bin/false; \
- fi; \
- $(OBJCOPY) --prefix-symbols=__kvm_nvhe_ \
- --rename-section=.text=.hyp.text \
- $< $@
+ cmd_hypcopy = $(OBJCOPY) --prefix-symbols=__kvm_nvhe_ $< $@
# Remove ftrace and Shadow Call Stack CFLAGS.
# This is equivalent to the 'notrace' and '__noscs' annotations.
diff --git a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S
new file mode 100644
index 000000000000..3b13d1c7cd1a
--- /dev/null
+++ b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Google LLC.
+ * Written by David Brazdil <dbrazdil@google.com>
+ *
+ * Linker script used for partial linking of nVHE EL2 object files.
+ */
+
+#include <asm/hyp_image.h>
+
+SECTIONS {
+ HYP_SECTION(.text)
+}
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v3 01/11] kvm: arm64: Partially link nVHE hyp code, simplify HYPCOPY
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Catalin Marinas, linux-kernel, linux-arm-kernel,
Marc Zyngier, Tejun Heo, Dennis Zhou, Christoph Lameter,
Will Deacon
Relying on objcopy to prefix the ELF section names of the nVHE hyp code
is brittle and prevents us from using wildcards to match specific
section names.
Improve the build rules by partially linking all '.nvhe.o' files and
prefixing their ELF section names using a linker script. Continue using
objcopy for prefixing ELF symbol names.
One immediate advantage of this approach is that all subsections
matching a pattern can be merged into a single prefixed section, eg.
.text and .text.* can be linked into a single '.hyp.text'. This removes
the need for -fno-reorder-functions on GCC and will be useful in the
future too: LTO builds use .text subsections, compilers routinely
generate .rodata subsections, etc.
Partially linking all hyp code into a single object file also makes it
easier to analyze.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/hyp_image.h | 24 ++++++++++++
arch/arm64/kvm/hyp/nvhe/Makefile | 60 ++++++++++++++++--------------
arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 13 +++++++
3 files changed, 70 insertions(+), 27 deletions(-)
create mode 100644 arch/arm64/include/asm/hyp_image.h
create mode 100644 arch/arm64/kvm/hyp/nvhe/hyp.lds.S
diff --git a/arch/arm64/include/asm/hyp_image.h b/arch/arm64/include/asm/hyp_image.h
new file mode 100644
index 000000000000..5b1e3b9ef376
--- /dev/null
+++ b/arch/arm64/include/asm/hyp_image.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Google LLC.
+ * Written by David Brazdil <dbrazdil@google.com>
+ */
+
+#ifndef __ARM64_HYP_IMAGE_H__
+#define __ARM64_HYP_IMAGE_H__
+
+#ifdef LINKER_SCRIPT
+
+/*
+ * KVM nVHE ELF section names are prefixed with .hyp, to separate them
+ * from the kernel proper.
+ */
+#define HYP_SECTION_NAME(NAME) .hyp##NAME
+
+/* Defines an ELF hyp section from input section @NAME and its subsections. */
+#define HYP_SECTION(NAME) \
+ HYP_SECTION_NAME(NAME) : { *(NAME NAME##.[0-9a-zA-Z_]*) }
+
+#endif /* LINKER_SCRIPT */
+
+#endif /* __ARM64_HYP_IMAGE_H__ */
diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile
index aef76487edc2..2b27b60182f9 100644
--- a/arch/arm64/kvm/hyp/nvhe/Makefile
+++ b/arch/arm64/kvm/hyp/nvhe/Makefile
@@ -10,40 +10,46 @@ obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o
obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \
../fpsimd.o ../hyp-entry.o
-obj-y := $(patsubst %.o,%.hyp.o,$(obj-y))
-extra-y := $(patsubst %.hyp.o,%.hyp.tmp.o,$(obj-y))
+##
+## Build rules for compiling nVHE hyp code
+## Output of this folder is `kvm_nvhe.o`, a partially linked object
+## file containing all nVHE hyp code and data.
+##
-$(obj)/%.hyp.tmp.o: $(src)/%.c FORCE
+hyp-obj := $(patsubst %.o,%.nvhe.o,$(obj-y))
+obj-y := kvm_nvhe.o
+extra-y := $(hyp-obj) kvm_nvhe.tmp.o hyp.lds
+
+# 1) Compile all source files to `.nvhe.o` object files. The file extension
+# avoids file name clashes for files shared with VHE.
+$(obj)/%.nvhe.o: $(src)/%.c FORCE
$(call if_changed_rule,cc_o_c)
-$(obj)/%.hyp.tmp.o: $(src)/%.S FORCE
+$(obj)/%.nvhe.o: $(src)/%.S FORCE
$(call if_changed_rule,as_o_S)
-$(obj)/%.hyp.o: $(obj)/%.hyp.tmp.o FORCE
- $(call if_changed,hypcopy)
-# Disable reordering functions by GCC (enabled at -O2).
-# This pass puts functions into '.text.*' sections to aid the linker
-# in optimizing ELF layout. See HYPCOPY comment below for more info.
-ccflags-y += $(call cc-option,-fno-reorder-functions)
+# 2) Compile linker script.
+$(obj)/hyp.lds: $(src)/hyp.lds.S FORCE
+ $(call if_changed_dep,cpp_lds_S)
+
+# 3) Partially link all '.nvhe.o' files and apply the linker script.
+# Prefixes names of ELF sections with '.hyp', eg. '.hyp.text'.
+# Note: The following rule assumes that the 'ld' rule puts LDFLAGS before
+# the list of dependencies to form '-T $(obj)/hyp.lds'. This is to
+# keep the dependency on the target while avoiding an error from
+# GNU ld if the linker script is passed to it twice.
+LDFLAGS_kvm_nvhe.tmp.o := -r -T
+$(obj)/kvm_nvhe.tmp.o: $(obj)/hyp.lds $(addprefix $(obj)/,$(hyp-obj)) FORCE
+ $(call if_changed,ld)
+
+# 4) Produce the final 'kvm_nvhe.o', ready to be linked into 'vmlinux'.
+# Prefixes names of ELF symbols with '__kvm_nvhe_'.
+$(obj)/kvm_nvhe.o: $(obj)/kvm_nvhe.tmp.o FORCE
+ $(call if_changed,hypcopy)
# The HYPCOPY command uses `objcopy` to prefix all ELF symbol names
-# and relevant ELF section names to avoid clashes with VHE code/data.
-#
-# Hyp code is assumed to be in the '.text' section of the input object
-# files (with the exception of specialized sections such as
-# '.hyp.idmap.text'). This assumption may be broken by a compiler that
-# divides code into sections like '.text.unlikely' so as to optimize
-# ELF layout. HYPCOPY checks that no such sections exist in the input
-# using `objdump`, otherwise they would be linked together with other
-# kernel code and not memory-mapped correctly at runtime.
+# to avoid clashes with VHE code/data.
quiet_cmd_hypcopy = HYPCOPY $@
- cmd_hypcopy = \
- if $(OBJDUMP) -h $< | grep -F '.text.'; then \
- echo "$@: function reordering not supported in nVHE hyp code" >&2; \
- /bin/false; \
- fi; \
- $(OBJCOPY) --prefix-symbols=__kvm_nvhe_ \
- --rename-section=.text=.hyp.text \
- $< $@
+ cmd_hypcopy = $(OBJCOPY) --prefix-symbols=__kvm_nvhe_ $< $@
# Remove ftrace and Shadow Call Stack CFLAGS.
# This is equivalent to the 'notrace' and '__noscs' annotations.
diff --git a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S
new file mode 100644
index 000000000000..3b13d1c7cd1a
--- /dev/null
+++ b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S
@@ -0,0 +1,13 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Google LLC.
+ * Written by David Brazdil <dbrazdil@google.com>
+ *
+ * Linker script used for partial linking of nVHE EL2 object files.
+ */
+
+#include <asm/hyp_image.h>
+
+SECTIONS {
+ HYP_SECTION(.text)
+}
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v3 01/11] kvm: arm64: Partially link nVHE hyp code, simplify HYPCOPY
2020-09-16 17:34 ` David Brazdil
(?)
@ 2020-09-18 8:51 ` Will Deacon
-1 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 8:51 UTC (permalink / raw)
To: David Brazdil
Cc: kvmarm, Catalin Marinas, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team
On Wed, Sep 16, 2020 at 06:34:29PM +0100, David Brazdil wrote:
> Relying on objcopy to prefix the ELF section names of the nVHE hyp code
> is brittle and prevents us from using wildcards to match specific
> section names.
>
> Improve the build rules by partially linking all '.nvhe.o' files and
> prefixing their ELF section names using a linker script. Continue using
> objcopy for prefixing ELF symbol names.
>
> One immediate advantage of this approach is that all subsections
> matching a pattern can be merged into a single prefixed section, eg.
> .text and .text.* can be linked into a single '.hyp.text'. This removes
> the need for -fno-reorder-functions on GCC and will be useful in the
> future too: LTO builds use .text subsections, compilers routinely
> generate .rodata subsections, etc.
>
> Partially linking all hyp code into a single object file also makes it
> easier to analyze.
>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/hyp_image.h | 24 ++++++++++++
> arch/arm64/kvm/hyp/nvhe/Makefile | 60 ++++++++++++++++--------------
> arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 13 +++++++
> 3 files changed, 70 insertions(+), 27 deletions(-)
> create mode 100644 arch/arm64/include/asm/hyp_image.h
> create mode 100644 arch/arm64/kvm/hyp/nvhe/hyp.lds.S
>
> diff --git a/arch/arm64/include/asm/hyp_image.h b/arch/arm64/include/asm/hyp_image.h
> new file mode 100644
> index 000000000000..5b1e3b9ef376
> --- /dev/null
> +++ b/arch/arm64/include/asm/hyp_image.h
> @@ -0,0 +1,24 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Copyright (C) 2020 Google LLC.
> + * Written by David Brazdil <dbrazdil@google.com>
> + */
> +
> +#ifndef __ARM64_HYP_IMAGE_H__
> +#define __ARM64_HYP_IMAGE_H__
> +
> +#ifdef LINKER_SCRIPT
> +
> +/*
> + * KVM nVHE ELF section names are prefixed with .hyp, to separate them
> + * from the kernel proper.
> + */
> +#define HYP_SECTION_NAME(NAME) .hyp##NAME
> +
> +/* Defines an ELF hyp section from input section @NAME and its subsections. */
> +#define HYP_SECTION(NAME) \
> + HYP_SECTION_NAME(NAME) : { *(NAME NAME##.[0-9a-zA-Z_]*) }
I still don't get why we can't just use NAME ## .* for the regex here. That
matches what we do elsewhere for linker script wildcarding, e.g. .rodata.*,
.init.text.* ... in asm-generic/vmlinux.lds.h. Why is it different for
these sections?
> +
> +#endif /* LINKER_SCRIPT */
> +
> +#endif /* __ARM64_HYP_IMAGE_H__ */
> diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile
> index aef76487edc2..2b27b60182f9 100644
> --- a/arch/arm64/kvm/hyp/nvhe/Makefile
> +++ b/arch/arm64/kvm/hyp/nvhe/Makefile
> @@ -10,40 +10,46 @@ obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o
> obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \
> ../fpsimd.o ../hyp-entry.o
>
> -obj-y := $(patsubst %.o,%.hyp.o,$(obj-y))
> -extra-y := $(patsubst %.hyp.o,%.hyp.tmp.o,$(obj-y))
> +##
> +## Build rules for compiling nVHE hyp code
> +## Output of this folder is `kvm_nvhe.o`, a partially linked object
> +## file containing all nVHE hyp code and data.
> +##
>
> -$(obj)/%.hyp.tmp.o: $(src)/%.c FORCE
> +hyp-obj := $(patsubst %.o,%.nvhe.o,$(obj-y))
> +obj-y := kvm_nvhe.o
> +extra-y := $(hyp-obj) kvm_nvhe.tmp.o hyp.lds
> +
> +# 1) Compile all source files to `.nvhe.o` object files. The file extension
> +# avoids file name clashes for files shared with VHE.
> +$(obj)/%.nvhe.o: $(src)/%.c FORCE
> $(call if_changed_rule,cc_o_c)
> -$(obj)/%.hyp.tmp.o: $(src)/%.S FORCE
> +$(obj)/%.nvhe.o: $(src)/%.S FORCE
> $(call if_changed_rule,as_o_S)
> -$(obj)/%.hyp.o: $(obj)/%.hyp.tmp.o FORCE
> - $(call if_changed,hypcopy)
>
> -# Disable reordering functions by GCC (enabled at -O2).
> -# This pass puts functions into '.text.*' sections to aid the linker
> -# in optimizing ELF layout. See HYPCOPY comment below for more info.
> -ccflags-y += $(call cc-option,-fno-reorder-functions)
> +# 2) Compile linker script.
> +$(obj)/hyp.lds: $(src)/hyp.lds.S FORCE
> + $(call if_changed_dep,cpp_lds_S)
You need a .gitignore file listing hyp.lds, otherwise some idiot will end
up committing it. I definitely didn't do that when playing around with this
series. Nope. Not at all.
With that, and the regex resolved:
Acked-by: Will Deacon <will@kernel.org>
Will
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v3 01/11] kvm: arm64: Partially link nVHE hyp code, simplify HYPCOPY
@ 2020-09-18 8:51 ` Will Deacon
0 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 8:51 UTC (permalink / raw)
To: David Brazdil
Cc: Suzuki K Poulose, Catalin Marinas, linux-kernel, James Morse,
linux-arm-kernel, Marc Zyngier, Tejun Heo, Dennis Zhou,
Christoph Lameter, kernel-team, kvmarm, Julien Thierry
On Wed, Sep 16, 2020 at 06:34:29PM +0100, David Brazdil wrote:
> Relying on objcopy to prefix the ELF section names of the nVHE hyp code
> is brittle and prevents us from using wildcards to match specific
> section names.
>
> Improve the build rules by partially linking all '.nvhe.o' files and
> prefixing their ELF section names using a linker script. Continue using
> objcopy for prefixing ELF symbol names.
>
> One immediate advantage of this approach is that all subsections
> matching a pattern can be merged into a single prefixed section, eg.
> .text and .text.* can be linked into a single '.hyp.text'. This removes
> the need for -fno-reorder-functions on GCC and will be useful in the
> future too: LTO builds use .text subsections, compilers routinely
> generate .rodata subsections, etc.
>
> Partially linking all hyp code into a single object file also makes it
> easier to analyze.
>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/hyp_image.h | 24 ++++++++++++
> arch/arm64/kvm/hyp/nvhe/Makefile | 60 ++++++++++++++++--------------
> arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 13 +++++++
> 3 files changed, 70 insertions(+), 27 deletions(-)
> create mode 100644 arch/arm64/include/asm/hyp_image.h
> create mode 100644 arch/arm64/kvm/hyp/nvhe/hyp.lds.S
>
> diff --git a/arch/arm64/include/asm/hyp_image.h b/arch/arm64/include/asm/hyp_image.h
> new file mode 100644
> index 000000000000..5b1e3b9ef376
> --- /dev/null
> +++ b/arch/arm64/include/asm/hyp_image.h
> @@ -0,0 +1,24 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Copyright (C) 2020 Google LLC.
> + * Written by David Brazdil <dbrazdil@google.com>
> + */
> +
> +#ifndef __ARM64_HYP_IMAGE_H__
> +#define __ARM64_HYP_IMAGE_H__
> +
> +#ifdef LINKER_SCRIPT
> +
> +/*
> + * KVM nVHE ELF section names are prefixed with .hyp, to separate them
> + * from the kernel proper.
> + */
> +#define HYP_SECTION_NAME(NAME) .hyp##NAME
> +
> +/* Defines an ELF hyp section from input section @NAME and its subsections. */
> +#define HYP_SECTION(NAME) \
> + HYP_SECTION_NAME(NAME) : { *(NAME NAME##.[0-9a-zA-Z_]*) }
I still don't get why we can't just use NAME ## .* for the regex here. That
matches what we do elsewhere for linker script wildcarding, e.g. .rodata.*,
.init.text.* ... in asm-generic/vmlinux.lds.h. Why is it different for
these sections?
> +
> +#endif /* LINKER_SCRIPT */
> +
> +#endif /* __ARM64_HYP_IMAGE_H__ */
> diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile
> index aef76487edc2..2b27b60182f9 100644
> --- a/arch/arm64/kvm/hyp/nvhe/Makefile
> +++ b/arch/arm64/kvm/hyp/nvhe/Makefile
> @@ -10,40 +10,46 @@ obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o
> obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \
> ../fpsimd.o ../hyp-entry.o
>
> -obj-y := $(patsubst %.o,%.hyp.o,$(obj-y))
> -extra-y := $(patsubst %.hyp.o,%.hyp.tmp.o,$(obj-y))
> +##
> +## Build rules for compiling nVHE hyp code
> +## Output of this folder is `kvm_nvhe.o`, a partially linked object
> +## file containing all nVHE hyp code and data.
> +##
>
> -$(obj)/%.hyp.tmp.o: $(src)/%.c FORCE
> +hyp-obj := $(patsubst %.o,%.nvhe.o,$(obj-y))
> +obj-y := kvm_nvhe.o
> +extra-y := $(hyp-obj) kvm_nvhe.tmp.o hyp.lds
> +
> +# 1) Compile all source files to `.nvhe.o` object files. The file extension
> +# avoids file name clashes for files shared with VHE.
> +$(obj)/%.nvhe.o: $(src)/%.c FORCE
> $(call if_changed_rule,cc_o_c)
> -$(obj)/%.hyp.tmp.o: $(src)/%.S FORCE
> +$(obj)/%.nvhe.o: $(src)/%.S FORCE
> $(call if_changed_rule,as_o_S)
> -$(obj)/%.hyp.o: $(obj)/%.hyp.tmp.o FORCE
> - $(call if_changed,hypcopy)
>
> -# Disable reordering functions by GCC (enabled at -O2).
> -# This pass puts functions into '.text.*' sections to aid the linker
> -# in optimizing ELF layout. See HYPCOPY comment below for more info.
> -ccflags-y += $(call cc-option,-fno-reorder-functions)
> +# 2) Compile linker script.
> +$(obj)/hyp.lds: $(src)/hyp.lds.S FORCE
> + $(call if_changed_dep,cpp_lds_S)
You need a .gitignore file listing hyp.lds, otherwise some idiot will end
up committing it. I definitely didn't do that when playing around with this
series. Nope. Not at all.
With that, and the regex resolved:
Acked-by: Will Deacon <will@kernel.org>
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v3 01/11] kvm: arm64: Partially link nVHE hyp code, simplify HYPCOPY
@ 2020-09-18 8:51 ` Will Deacon
0 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 8:51 UTC (permalink / raw)
To: David Brazdil
Cc: Catalin Marinas, linux-kernel, linux-arm-kernel, Marc Zyngier,
Tejun Heo, Dennis Zhou, Christoph Lameter, kernel-team, kvmarm
On Wed, Sep 16, 2020 at 06:34:29PM +0100, David Brazdil wrote:
> Relying on objcopy to prefix the ELF section names of the nVHE hyp code
> is brittle and prevents us from using wildcards to match specific
> section names.
>
> Improve the build rules by partially linking all '.nvhe.o' files and
> prefixing their ELF section names using a linker script. Continue using
> objcopy for prefixing ELF symbol names.
>
> One immediate advantage of this approach is that all subsections
> matching a pattern can be merged into a single prefixed section, eg.
> .text and .text.* can be linked into a single '.hyp.text'. This removes
> the need for -fno-reorder-functions on GCC and will be useful in the
> future too: LTO builds use .text subsections, compilers routinely
> generate .rodata subsections, etc.
>
> Partially linking all hyp code into a single object file also makes it
> easier to analyze.
>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/hyp_image.h | 24 ++++++++++++
> arch/arm64/kvm/hyp/nvhe/Makefile | 60 ++++++++++++++++--------------
> arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 13 +++++++
> 3 files changed, 70 insertions(+), 27 deletions(-)
> create mode 100644 arch/arm64/include/asm/hyp_image.h
> create mode 100644 arch/arm64/kvm/hyp/nvhe/hyp.lds.S
>
> diff --git a/arch/arm64/include/asm/hyp_image.h b/arch/arm64/include/asm/hyp_image.h
> new file mode 100644
> index 000000000000..5b1e3b9ef376
> --- /dev/null
> +++ b/arch/arm64/include/asm/hyp_image.h
> @@ -0,0 +1,24 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * Copyright (C) 2020 Google LLC.
> + * Written by David Brazdil <dbrazdil@google.com>
> + */
> +
> +#ifndef __ARM64_HYP_IMAGE_H__
> +#define __ARM64_HYP_IMAGE_H__
> +
> +#ifdef LINKER_SCRIPT
> +
> +/*
> + * KVM nVHE ELF section names are prefixed with .hyp, to separate them
> + * from the kernel proper.
> + */
> +#define HYP_SECTION_NAME(NAME) .hyp##NAME
> +
> +/* Defines an ELF hyp section from input section @NAME and its subsections. */
> +#define HYP_SECTION(NAME) \
> + HYP_SECTION_NAME(NAME) : { *(NAME NAME##.[0-9a-zA-Z_]*) }
I still don't get why we can't just use NAME ## .* for the regex here. That
matches what we do elsewhere for linker script wildcarding, e.g. .rodata.*,
.init.text.* ... in asm-generic/vmlinux.lds.h. Why is it different for
these sections?
> +
> +#endif /* LINKER_SCRIPT */
> +
> +#endif /* __ARM64_HYP_IMAGE_H__ */
> diff --git a/arch/arm64/kvm/hyp/nvhe/Makefile b/arch/arm64/kvm/hyp/nvhe/Makefile
> index aef76487edc2..2b27b60182f9 100644
> --- a/arch/arm64/kvm/hyp/nvhe/Makefile
> +++ b/arch/arm64/kvm/hyp/nvhe/Makefile
> @@ -10,40 +10,46 @@ obj-y := timer-sr.o sysreg-sr.o debug-sr.o switch.o tlb.o hyp-init.o
> obj-y += ../vgic-v3-sr.o ../aarch32.o ../vgic-v2-cpuif-proxy.o ../entry.o \
> ../fpsimd.o ../hyp-entry.o
>
> -obj-y := $(patsubst %.o,%.hyp.o,$(obj-y))
> -extra-y := $(patsubst %.hyp.o,%.hyp.tmp.o,$(obj-y))
> +##
> +## Build rules for compiling nVHE hyp code
> +## Output of this folder is `kvm_nvhe.o`, a partially linked object
> +## file containing all nVHE hyp code and data.
> +##
>
> -$(obj)/%.hyp.tmp.o: $(src)/%.c FORCE
> +hyp-obj := $(patsubst %.o,%.nvhe.o,$(obj-y))
> +obj-y := kvm_nvhe.o
> +extra-y := $(hyp-obj) kvm_nvhe.tmp.o hyp.lds
> +
> +# 1) Compile all source files to `.nvhe.o` object files. The file extension
> +# avoids file name clashes for files shared with VHE.
> +$(obj)/%.nvhe.o: $(src)/%.c FORCE
> $(call if_changed_rule,cc_o_c)
> -$(obj)/%.hyp.tmp.o: $(src)/%.S FORCE
> +$(obj)/%.nvhe.o: $(src)/%.S FORCE
> $(call if_changed_rule,as_o_S)
> -$(obj)/%.hyp.o: $(obj)/%.hyp.tmp.o FORCE
> - $(call if_changed,hypcopy)
>
> -# Disable reordering functions by GCC (enabled at -O2).
> -# This pass puts functions into '.text.*' sections to aid the linker
> -# in optimizing ELF layout. See HYPCOPY comment below for more info.
> -ccflags-y += $(call cc-option,-fno-reorder-functions)
> +# 2) Compile linker script.
> +$(obj)/hyp.lds: $(src)/hyp.lds.S FORCE
> + $(call if_changed_dep,cpp_lds_S)
You need a .gitignore file listing hyp.lds, otherwise some idiot will end
up committing it. I definitely didn't do that when playing around with this
series. Nope. Not at all.
With that, and the regex resolved:
Acked-by: Will Deacon <will@kernel.org>
Will
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v3 01/11] kvm: arm64: Partially link nVHE hyp code, simplify HYPCOPY
2020-09-18 8:51 ` Will Deacon
(?)
@ 2020-09-22 17:55 ` David Brazdil
-1 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-22 17:55 UTC (permalink / raw)
To: Will Deacon
Cc: kvmarm, Catalin Marinas, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team
> > +#define HYP_SECTION_NAME(NAME) .hyp##NAME
> > +
> > +/* Defines an ELF hyp section from input section @NAME and its subsections. */
> > +#define HYP_SECTION(NAME) \
> > + HYP_SECTION_NAME(NAME) : { *(NAME NAME##.[0-9a-zA-Z_]*) }
>
> I still don't get why we can't just use NAME ## .* for the regex here. That
> matches what we do elsewhere for linker script wildcarding, e.g. .rodata.*,
> .init.text.* ... in asm-generic/vmlinux.lds.h. Why is it different for
> these sections?
Shouldn't matter. I copied it from a script in the Clang LTO branch. Will align
it with what's in the main arm64 script.
> > +# 2) Compile linker script.
> > +$(obj)/hyp.lds: $(src)/hyp.lds.S FORCE
> > + $(call if_changed_dep,cpp_lds_S)
>
> You need a .gitignore file listing hyp.lds, otherwise some idiot will end
> up committing it. I definitely didn't do that when playing around with this
> series. Nope. Not at all.
Thanks for noticing, I never compile in-tree. Added
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v3 01/11] kvm: arm64: Partially link nVHE hyp code, simplify HYPCOPY
@ 2020-09-22 17:55 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-22 17:55 UTC (permalink / raw)
To: Will Deacon
Cc: Suzuki K Poulose, Catalin Marinas, linux-kernel, James Morse,
linux-arm-kernel, Marc Zyngier, Tejun Heo, Dennis Zhou,
Christoph Lameter, kernel-team, kvmarm, Julien Thierry
> > +#define HYP_SECTION_NAME(NAME) .hyp##NAME
> > +
> > +/* Defines an ELF hyp section from input section @NAME and its subsections. */
> > +#define HYP_SECTION(NAME) \
> > + HYP_SECTION_NAME(NAME) : { *(NAME NAME##.[0-9a-zA-Z_]*) }
>
> I still don't get why we can't just use NAME ## .* for the regex here. That
> matches what we do elsewhere for linker script wildcarding, e.g. .rodata.*,
> .init.text.* ... in asm-generic/vmlinux.lds.h. Why is it different for
> these sections?
Shouldn't matter. I copied it from a script in the Clang LTO branch. Will align
it with what's in the main arm64 script.
> > +# 2) Compile linker script.
> > +$(obj)/hyp.lds: $(src)/hyp.lds.S FORCE
> > + $(call if_changed_dep,cpp_lds_S)
>
> You need a .gitignore file listing hyp.lds, otherwise some idiot will end
> up committing it. I definitely didn't do that when playing around with this
> series. Nope. Not at all.
Thanks for noticing, I never compile in-tree. Added
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v3 01/11] kvm: arm64: Partially link nVHE hyp code, simplify HYPCOPY
@ 2020-09-22 17:55 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-22 17:55 UTC (permalink / raw)
To: Will Deacon
Cc: Catalin Marinas, linux-kernel, linux-arm-kernel, Marc Zyngier,
Tejun Heo, Dennis Zhou, Christoph Lameter, kernel-team, kvmarm
> > +#define HYP_SECTION_NAME(NAME) .hyp##NAME
> > +
> > +/* Defines an ELF hyp section from input section @NAME and its subsections. */
> > +#define HYP_SECTION(NAME) \
> > + HYP_SECTION_NAME(NAME) : { *(NAME NAME##.[0-9a-zA-Z_]*) }
>
> I still don't get why we can't just use NAME ## .* for the regex here. That
> matches what we do elsewhere for linker script wildcarding, e.g. .rodata.*,
> .init.text.* ... in asm-generic/vmlinux.lds.h. Why is it different for
> these sections?
Shouldn't matter. I copied it from a script in the Clang LTO branch. Will align
it with what's in the main arm64 script.
> > +# 2) Compile linker script.
> > +$(obj)/hyp.lds: $(src)/hyp.lds.S FORCE
> > + $(call if_changed_dep,cpp_lds_S)
>
> You need a .gitignore file listing hyp.lds, otherwise some idiot will end
> up committing it. I definitely didn't do that when playing around with this
> series. Nope. Not at all.
Thanks for noticing, I never compile in-tree. Added
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v3 02/11] kvm: arm64: Move nVHE hyp namespace macros to hyp_image.h
2020-09-16 17:34 ` David Brazdil
(?)
@ 2020-09-16 17:34 ` David Brazdil
-1 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: Catalin Marinas, Will Deacon, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team,
David Brazdil
Minor cleanup to move all macros related to prefixing nVHE hyp section
and symbol names into one place: hyp_image.h.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/hyp_image.h | 12 ++++++++++++
arch/arm64/include/asm/kvm_asm.h | 8 +-------
arch/arm64/kernel/image-vars.h | 2 --
3 files changed, 13 insertions(+), 9 deletions(-)
diff --git a/arch/arm64/include/asm/hyp_image.h b/arch/arm64/include/asm/hyp_image.h
index 5b1e3b9ef376..9848de588238 100644
--- a/arch/arm64/include/asm/hyp_image.h
+++ b/arch/arm64/include/asm/hyp_image.h
@@ -7,6 +7,12 @@
#ifndef __ARM64_HYP_IMAGE_H__
#define __ARM64_HYP_IMAGE_H__
+/*
+ * KVM nVHE code has its own symbol namespace prefixed with __kvm_nvhe_,
+ * to separate it from the kernel proper.
+ */
+#define kvm_nvhe_sym(sym) __kvm_nvhe_##sym
+
#ifdef LINKER_SCRIPT
/*
@@ -19,6 +25,12 @@
#define HYP_SECTION(NAME) \
HYP_SECTION_NAME(NAME) : { *(NAME NAME##.[0-9a-zA-Z_]*) }
+/*
+ * Defines a linker script alias of a kernel-proper symbol referenced by
+ * KVM nVHE hyp code.
+ */
+#define KVM_NVHE_ALIAS(sym) kvm_nvhe_sym(sym) = sym;
+
#endif /* LINKER_SCRIPT */
#endif /* __ARM64_HYP_IMAGE_H__ */
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index 6f98fbd0ac81..c085032e2e3e 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -7,6 +7,7 @@
#ifndef __ARM_KVM_ASM_H__
#define __ARM_KVM_ASM_H__
+#include <asm/hyp_image.h>
#include <asm/virt.h>
#define VCPU_WORKAROUND_2_FLAG_SHIFT 0
@@ -42,13 +43,6 @@
#include <linux/mm.h>
-/*
- * Translate name of a symbol defined in nVHE hyp to the name seen
- * by kernel proper. All nVHE symbols are prefixed by the build system
- * to avoid clashes with the VHE variants.
- */
-#define kvm_nvhe_sym(sym) __kvm_nvhe_##sym
-
#define DECLARE_KVM_VHE_SYM(sym) extern char sym[]
#define DECLARE_KVM_NVHE_SYM(sym) extern char kvm_nvhe_sym(sym)[]
diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h
index 8982b68289b7..76da2ad1010c 100644
--- a/arch/arm64/kernel/image-vars.h
+++ b/arch/arm64/kernel/image-vars.h
@@ -61,8 +61,6 @@ __efistub__ctype = _ctype;
* memory mappings.
*/
-#define KVM_NVHE_ALIAS(sym) __kvm_nvhe_##sym = sym;
-
/* Alternative callbacks for init-time patching of nVHE hyp code. */
KVM_NVHE_ALIAS(arm64_enable_wa2_handling);
KVM_NVHE_ALIAS(kvm_patch_vector_branch);
--
2.28.0.618.gf4bc123cb7-goog
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v3 02/11] kvm: arm64: Move nVHE hyp namespace macros to hyp_image.h
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Suzuki K Poulose, Catalin Marinas, linux-kernel,
James Morse, linux-arm-kernel, Marc Zyngier, Tejun Heo,
Dennis Zhou, Christoph Lameter, David Brazdil, Will Deacon,
Julien Thierry
Minor cleanup to move all macros related to prefixing nVHE hyp section
and symbol names into one place: hyp_image.h.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/hyp_image.h | 12 ++++++++++++
arch/arm64/include/asm/kvm_asm.h | 8 +-------
arch/arm64/kernel/image-vars.h | 2 --
3 files changed, 13 insertions(+), 9 deletions(-)
diff --git a/arch/arm64/include/asm/hyp_image.h b/arch/arm64/include/asm/hyp_image.h
index 5b1e3b9ef376..9848de588238 100644
--- a/arch/arm64/include/asm/hyp_image.h
+++ b/arch/arm64/include/asm/hyp_image.h
@@ -7,6 +7,12 @@
#ifndef __ARM64_HYP_IMAGE_H__
#define __ARM64_HYP_IMAGE_H__
+/*
+ * KVM nVHE code has its own symbol namespace prefixed with __kvm_nvhe_,
+ * to separate it from the kernel proper.
+ */
+#define kvm_nvhe_sym(sym) __kvm_nvhe_##sym
+
#ifdef LINKER_SCRIPT
/*
@@ -19,6 +25,12 @@
#define HYP_SECTION(NAME) \
HYP_SECTION_NAME(NAME) : { *(NAME NAME##.[0-9a-zA-Z_]*) }
+/*
+ * Defines a linker script alias of a kernel-proper symbol referenced by
+ * KVM nVHE hyp code.
+ */
+#define KVM_NVHE_ALIAS(sym) kvm_nvhe_sym(sym) = sym;
+
#endif /* LINKER_SCRIPT */
#endif /* __ARM64_HYP_IMAGE_H__ */
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index 6f98fbd0ac81..c085032e2e3e 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -7,6 +7,7 @@
#ifndef __ARM_KVM_ASM_H__
#define __ARM_KVM_ASM_H__
+#include <asm/hyp_image.h>
#include <asm/virt.h>
#define VCPU_WORKAROUND_2_FLAG_SHIFT 0
@@ -42,13 +43,6 @@
#include <linux/mm.h>
-/*
- * Translate name of a symbol defined in nVHE hyp to the name seen
- * by kernel proper. All nVHE symbols are prefixed by the build system
- * to avoid clashes with the VHE variants.
- */
-#define kvm_nvhe_sym(sym) __kvm_nvhe_##sym
-
#define DECLARE_KVM_VHE_SYM(sym) extern char sym[]
#define DECLARE_KVM_NVHE_SYM(sym) extern char kvm_nvhe_sym(sym)[]
diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h
index 8982b68289b7..76da2ad1010c 100644
--- a/arch/arm64/kernel/image-vars.h
+++ b/arch/arm64/kernel/image-vars.h
@@ -61,8 +61,6 @@ __efistub__ctype = _ctype;
* memory mappings.
*/
-#define KVM_NVHE_ALIAS(sym) __kvm_nvhe_##sym = sym;
-
/* Alternative callbacks for init-time patching of nVHE hyp code. */
KVM_NVHE_ALIAS(arm64_enable_wa2_handling);
KVM_NVHE_ALIAS(kvm_patch_vector_branch);
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v3 02/11] kvm: arm64: Move nVHE hyp namespace macros to hyp_image.h
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Catalin Marinas, linux-kernel, linux-arm-kernel,
Marc Zyngier, Tejun Heo, Dennis Zhou, Christoph Lameter,
Will Deacon
Minor cleanup to move all macros related to prefixing nVHE hyp section
and symbol names into one place: hyp_image.h.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/hyp_image.h | 12 ++++++++++++
arch/arm64/include/asm/kvm_asm.h | 8 +-------
arch/arm64/kernel/image-vars.h | 2 --
3 files changed, 13 insertions(+), 9 deletions(-)
diff --git a/arch/arm64/include/asm/hyp_image.h b/arch/arm64/include/asm/hyp_image.h
index 5b1e3b9ef376..9848de588238 100644
--- a/arch/arm64/include/asm/hyp_image.h
+++ b/arch/arm64/include/asm/hyp_image.h
@@ -7,6 +7,12 @@
#ifndef __ARM64_HYP_IMAGE_H__
#define __ARM64_HYP_IMAGE_H__
+/*
+ * KVM nVHE code has its own symbol namespace prefixed with __kvm_nvhe_,
+ * to separate it from the kernel proper.
+ */
+#define kvm_nvhe_sym(sym) __kvm_nvhe_##sym
+
#ifdef LINKER_SCRIPT
/*
@@ -19,6 +25,12 @@
#define HYP_SECTION(NAME) \
HYP_SECTION_NAME(NAME) : { *(NAME NAME##.[0-9a-zA-Z_]*) }
+/*
+ * Defines a linker script alias of a kernel-proper symbol referenced by
+ * KVM nVHE hyp code.
+ */
+#define KVM_NVHE_ALIAS(sym) kvm_nvhe_sym(sym) = sym;
+
#endif /* LINKER_SCRIPT */
#endif /* __ARM64_HYP_IMAGE_H__ */
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index 6f98fbd0ac81..c085032e2e3e 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -7,6 +7,7 @@
#ifndef __ARM_KVM_ASM_H__
#define __ARM_KVM_ASM_H__
+#include <asm/hyp_image.h>
#include <asm/virt.h>
#define VCPU_WORKAROUND_2_FLAG_SHIFT 0
@@ -42,13 +43,6 @@
#include <linux/mm.h>
-/*
- * Translate name of a symbol defined in nVHE hyp to the name seen
- * by kernel proper. All nVHE symbols are prefixed by the build system
- * to avoid clashes with the VHE variants.
- */
-#define kvm_nvhe_sym(sym) __kvm_nvhe_##sym
-
#define DECLARE_KVM_VHE_SYM(sym) extern char sym[]
#define DECLARE_KVM_NVHE_SYM(sym) extern char kvm_nvhe_sym(sym)[]
diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h
index 8982b68289b7..76da2ad1010c 100644
--- a/arch/arm64/kernel/image-vars.h
+++ b/arch/arm64/kernel/image-vars.h
@@ -61,8 +61,6 @@ __efistub__ctype = _ctype;
* memory mappings.
*/
-#define KVM_NVHE_ALIAS(sym) __kvm_nvhe_##sym = sym;
-
/* Alternative callbacks for init-time patching of nVHE hyp code. */
KVM_NVHE_ALIAS(arm64_enable_wa2_handling);
KVM_NVHE_ALIAS(kvm_patch_vector_branch);
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v3 02/11] kvm: arm64: Move nVHE hyp namespace macros to hyp_image.h
2020-09-16 17:34 ` David Brazdil
(?)
@ 2020-09-18 8:52 ` Will Deacon
-1 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 8:52 UTC (permalink / raw)
To: David Brazdil
Cc: kvmarm, Catalin Marinas, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team
On Wed, Sep 16, 2020 at 06:34:30PM +0100, David Brazdil wrote:
> Minor cleanup to move all macros related to prefixing nVHE hyp section
> and symbol names into one place: hyp_image.h.
>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/hyp_image.h | 12 ++++++++++++
> arch/arm64/include/asm/kvm_asm.h | 8 +-------
> arch/arm64/kernel/image-vars.h | 2 --
> 3 files changed, 13 insertions(+), 9 deletions(-)
Acked-by: Will Deacon <will@kernel.org>
Will
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3 02/11] kvm: arm64: Move nVHE hyp namespace macros to hyp_image.h
@ 2020-09-18 8:52 ` Will Deacon
0 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 8:52 UTC (permalink / raw)
To: David Brazdil
Cc: Suzuki K Poulose, Catalin Marinas, linux-kernel, James Morse,
linux-arm-kernel, Marc Zyngier, Tejun Heo, Dennis Zhou,
Christoph Lameter, kernel-team, kvmarm, Julien Thierry
On Wed, Sep 16, 2020 at 06:34:30PM +0100, David Brazdil wrote:
> Minor cleanup to move all macros related to prefixing nVHE hyp section
> and symbol names into one place: hyp_image.h.
>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/hyp_image.h | 12 ++++++++++++
> arch/arm64/include/asm/kvm_asm.h | 8 +-------
> arch/arm64/kernel/image-vars.h | 2 --
> 3 files changed, 13 insertions(+), 9 deletions(-)
Acked-by: Will Deacon <will@kernel.org>
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3 02/11] kvm: arm64: Move nVHE hyp namespace macros to hyp_image.h
@ 2020-09-18 8:52 ` Will Deacon
0 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 8:52 UTC (permalink / raw)
To: David Brazdil
Cc: Catalin Marinas, linux-kernel, linux-arm-kernel, Marc Zyngier,
Tejun Heo, Dennis Zhou, Christoph Lameter, kernel-team, kvmarm
On Wed, Sep 16, 2020 at 06:34:30PM +0100, David Brazdil wrote:
> Minor cleanup to move all macros related to prefixing nVHE hyp section
> and symbol names into one place: hyp_image.h.
>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/hyp_image.h | 12 ++++++++++++
> arch/arm64/include/asm/kvm_asm.h | 8 +-------
> arch/arm64/kernel/image-vars.h | 2 --
> 3 files changed, 13 insertions(+), 9 deletions(-)
Acked-by: Will Deacon <will@kernel.org>
Will
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v3 03/11] kvm: arm64: Only define __kvm_ex_table for CONFIG_KVM
2020-09-16 17:34 ` David Brazdil
(?)
@ 2020-09-16 17:34 ` David Brazdil
-1 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: Catalin Marinas, Will Deacon, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team,
David Brazdil
Minor cleanup that only creates __kvm_ex_table ELF section and
related symbols if CONFIG_KVM is enabled.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/kernel/vmlinux.lds.S | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
index 7cba7623fcec..0d543570e811 100644
--- a/arch/arm64/kernel/vmlinux.lds.S
+++ b/arch/arm64/kernel/vmlinux.lds.S
@@ -21,11 +21,15 @@ ENTRY(_text)
jiffies = jiffies_64;
+#ifdef CONFIG_KVM
#define HYPERVISOR_EXTABLE \
. = ALIGN(SZ_8); \
__start___kvm_ex_table = .; \
*(__kvm_ex_table) \
__stop___kvm_ex_table = .;
+#else /* CONFIG_KVM */
+#define HYPERVISOR_EXTABLE
+#endif
#define HYPERVISOR_TEXT \
/* \
--
2.28.0.618.gf4bc123cb7-goog
^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH v3 03/11] kvm: arm64: Only define __kvm_ex_table for CONFIG_KVM
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Suzuki K Poulose, Catalin Marinas, linux-kernel,
James Morse, linux-arm-kernel, Marc Zyngier, Tejun Heo,
Dennis Zhou, Christoph Lameter, David Brazdil, Will Deacon,
Julien Thierry
Minor cleanup that only creates __kvm_ex_table ELF section and
related symbols if CONFIG_KVM is enabled.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/kernel/vmlinux.lds.S | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
index 7cba7623fcec..0d543570e811 100644
--- a/arch/arm64/kernel/vmlinux.lds.S
+++ b/arch/arm64/kernel/vmlinux.lds.S
@@ -21,11 +21,15 @@ ENTRY(_text)
jiffies = jiffies_64;
+#ifdef CONFIG_KVM
#define HYPERVISOR_EXTABLE \
. = ALIGN(SZ_8); \
__start___kvm_ex_table = .; \
*(__kvm_ex_table) \
__stop___kvm_ex_table = .;
+#else /* CONFIG_KVM */
+#define HYPERVISOR_EXTABLE
+#endif
#define HYPERVISOR_TEXT \
/* \
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH v3 03/11] kvm: arm64: Only define __kvm_ex_table for CONFIG_KVM
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Catalin Marinas, linux-kernel, linux-arm-kernel,
Marc Zyngier, Tejun Heo, Dennis Zhou, Christoph Lameter,
Will Deacon
Minor cleanup that only creates __kvm_ex_table ELF section and
related symbols if CONFIG_KVM is enabled.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/kernel/vmlinux.lds.S | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
index 7cba7623fcec..0d543570e811 100644
--- a/arch/arm64/kernel/vmlinux.lds.S
+++ b/arch/arm64/kernel/vmlinux.lds.S
@@ -21,11 +21,15 @@ ENTRY(_text)
jiffies = jiffies_64;
+#ifdef CONFIG_KVM
#define HYPERVISOR_EXTABLE \
. = ALIGN(SZ_8); \
__start___kvm_ex_table = .; \
*(__kvm_ex_table) \
__stop___kvm_ex_table = .;
+#else /* CONFIG_KVM */
+#define HYPERVISOR_EXTABLE
+#endif
#define HYPERVISOR_TEXT \
/* \
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply related [flat|nested] 87+ messages in thread
* Re: [PATCH v3 03/11] kvm: arm64: Only define __kvm_ex_table for CONFIG_KVM
2020-09-16 17:34 ` David Brazdil
(?)
@ 2020-09-18 9:00 ` Will Deacon
-1 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 9:00 UTC (permalink / raw)
To: David Brazdil
Cc: kvmarm, Catalin Marinas, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team
On Wed, Sep 16, 2020 at 06:34:31PM +0100, David Brazdil wrote:
> Minor cleanup that only creates __kvm_ex_table ELF section and
> related symbols if CONFIG_KVM is enabled.
>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/kernel/vmlinux.lds.S | 4 ++++
> 1 file changed, 4 insertions(+)
It's also useful because we're about to add another entry here, so:
Acked-by: Will Deacon <will@kernel.org>
Will
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3 03/11] kvm: arm64: Only define __kvm_ex_table for CONFIG_KVM
@ 2020-09-18 9:00 ` Will Deacon
0 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 9:00 UTC (permalink / raw)
To: David Brazdil
Cc: Suzuki K Poulose, Catalin Marinas, linux-kernel, James Morse,
linux-arm-kernel, Marc Zyngier, Tejun Heo, Dennis Zhou,
Christoph Lameter, kernel-team, kvmarm, Julien Thierry
On Wed, Sep 16, 2020 at 06:34:31PM +0100, David Brazdil wrote:
> Minor cleanup that only creates __kvm_ex_table ELF section and
> related symbols if CONFIG_KVM is enabled.
>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/kernel/vmlinux.lds.S | 4 ++++
> 1 file changed, 4 insertions(+)
It's also useful because we're about to add another entry here, so:
Acked-by: Will Deacon <will@kernel.org>
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3 03/11] kvm: arm64: Only define __kvm_ex_table for CONFIG_KVM
@ 2020-09-18 9:00 ` Will Deacon
0 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 9:00 UTC (permalink / raw)
To: David Brazdil
Cc: Catalin Marinas, linux-kernel, linux-arm-kernel, Marc Zyngier,
Tejun Heo, Dennis Zhou, Christoph Lameter, kernel-team, kvmarm
On Wed, Sep 16, 2020 at 06:34:31PM +0100, David Brazdil wrote:
> Minor cleanup that only creates __kvm_ex_table ELF section and
> related symbols if CONFIG_KVM is enabled.
>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/kernel/vmlinux.lds.S | 4 ++++
> 1 file changed, 4 insertions(+)
It's also useful because we're about to add another entry here, so:
Acked-by: Will Deacon <will@kernel.org>
Will
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v3 04/11] kvm: arm64: Remove __hyp_this_cpu_read
2020-09-16 17:34 ` David Brazdil
(?)
@ 2020-09-16 17:34 ` David Brazdil
-1 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: Catalin Marinas, Will Deacon, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team,
David Brazdil, Andrew Scull
this_cpu_ptr is meant for use in kernel proper because it selects between
TPIDR_EL1/2 based on nVHE/VHE. __hyp_this_cpu_ptr was used in hyp to always
select TPIDR_EL2. Unify all users behind this_cpu_ptr and friends by
selecting _EL2 register under __KVM_VHE/NVHE_HYPERVISOR__.
Under CONFIG_DEBUG_PREEMPT, the kernel helpers perform a preemption check
which is omitted by the hyp helpers. Preserve the behavior for nVHE by
overriding the corresponding macros under __KVM_NVHE_HYPERVISOR__. Extend
the checks into VHE hyp code.
Acked-by: Andrew Scull <ascull@google.com>
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/kvm_asm.h | 20 --------------
arch/arm64/include/asm/percpu.h | 33 +++++++++++++++++++++--
arch/arm64/kvm/hyp/include/hyp/debug-sr.h | 4 +--
arch/arm64/kvm/hyp/include/hyp/switch.h | 8 +++---
arch/arm64/kvm/hyp/nvhe/switch.c | 2 +-
arch/arm64/kvm/hyp/vhe/switch.c | 2 +-
arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 4 +--
7 files changed, 41 insertions(+), 32 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index c085032e2e3e..c196eec25498 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -143,26 +143,6 @@ extern char __smccc_workaround_1_smc[__SMCCC_WORKAROUND_1_SMC_SZ];
addr; \
})
-/*
- * Home-grown __this_cpu_{ptr,read} variants that always work at HYP,
- * provided that sym is really a *symbol* and not a pointer obtained from
- * a data structure. As for SHIFT_PERCPU_PTR(), the creative casting keeps
- * sparse quiet.
- */
-#define __hyp_this_cpu_ptr(sym) \
- ({ \
- void *__ptr; \
- __verify_pcpu_ptr(&sym); \
- __ptr = hyp_symbol_addr(sym); \
- __ptr += read_sysreg(tpidr_el2); \
- (typeof(sym) __kernel __force *)__ptr; \
- })
-
-#define __hyp_this_cpu_read(sym) \
- ({ \
- *__hyp_this_cpu_ptr(sym); \
- })
-
#define __KVM_EXTABLE(from, to) \
" .pushsection __kvm_ex_table, \"a\"\n" \
" .align 3\n" \
diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index 0b6409b89e5e..36f304401c38 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -19,7 +19,21 @@ static inline void set_my_cpu_offset(unsigned long off)
:: "r" (off) : "memory");
}
-static inline unsigned long __my_cpu_offset(void)
+static inline unsigned long __hyp_my_cpu_offset(void)
+{
+ unsigned long off;
+
+ /*
+ * We want to allow caching the value, so avoid using volatile and
+ * instead use a fake stack read to hazard against barrier().
+ */
+ asm("mrs %0, tpidr_el2" : "=r" (off) :
+ "Q" (*(const unsigned long *)current_stack_pointer));
+
+ return off;
+}
+
+static inline unsigned long __kern_my_cpu_offset(void)
{
unsigned long off;
@@ -35,7 +49,12 @@ static inline unsigned long __my_cpu_offset(void)
return off;
}
-#define __my_cpu_offset __my_cpu_offset()
+
+#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
+#define __my_cpu_offset __hyp_my_cpu_offset()
+#else
+#define __my_cpu_offset __kern_my_cpu_offset()
+#endif
#define PERCPU_RW_OPS(sz) \
static inline unsigned long __percpu_read_##sz(void *ptr) \
@@ -227,4 +246,14 @@ PERCPU_RET_OP(add, add, ldadd)
#include <asm-generic/percpu.h>
+/* Redefine macros for nVHE hyp under DEBUG_PREEMPT to avoid its dependencies. */
+#if defined(__KVM_NVHE_HYPERVISOR__) && defined(CONFIG_DEBUG_PREEMPT)
+#undef this_cpu_ptr
+#define this_cpu_ptr raw_cpu_ptr
+#undef __this_cpu_read
+#define __this_cpu_read raw_cpu_read
+#undef __this_cpu_write
+#define __this_cpu_write raw_cpu_write
+#endif
+
#endif /* __ASM_PERCPU_H */
diff --git a/arch/arm64/kvm/hyp/include/hyp/debug-sr.h b/arch/arm64/kvm/hyp/include/hyp/debug-sr.h
index 5e28ea6aa097..4ebe9f558f3a 100644
--- a/arch/arm64/kvm/hyp/include/hyp/debug-sr.h
+++ b/arch/arm64/kvm/hyp/include/hyp/debug-sr.h
@@ -135,7 +135,7 @@ static inline void __debug_switch_to_guest_common(struct kvm_vcpu *vcpu)
if (!(vcpu->arch.flags & KVM_ARM64_DEBUG_DIRTY))
return;
- host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
guest_ctxt = &vcpu->arch.ctxt;
host_dbg = &vcpu->arch.host_debug_state.regs;
guest_dbg = kern_hyp_va(vcpu->arch.debug_ptr);
@@ -154,7 +154,7 @@ static inline void __debug_switch_to_host_common(struct kvm_vcpu *vcpu)
if (!(vcpu->arch.flags & KVM_ARM64_DEBUG_DIRTY))
return;
- host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
guest_ctxt = &vcpu->arch.ctxt;
host_dbg = &vcpu->arch.host_debug_state.regs;
guest_dbg = kern_hyp_va(vcpu->arch.debug_ptr);
diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
index 5b6b8fa00f0a..f150407fa798 100644
--- a/arch/arm64/kvm/hyp/include/hyp/switch.h
+++ b/arch/arm64/kvm/hyp/include/hyp/switch.h
@@ -386,7 +386,7 @@ static inline bool __hyp_handle_ptrauth(struct kvm_vcpu *vcpu)
!esr_is_ptrauth_trap(kvm_vcpu_get_esr(vcpu)))
return false;
- ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
__ptrauth_save_key(ctxt, APIA);
__ptrauth_save_key(ctxt, APIB);
__ptrauth_save_key(ctxt, APDA);
@@ -495,7 +495,7 @@ static inline void __set_guest_arch_workaround_state(struct kvm_vcpu *vcpu)
* guest wants it disabled, so be it...
*/
if (__needs_ssbd_off(vcpu) &&
- __hyp_this_cpu_read(arm64_ssbd_callback_required))
+ __this_cpu_read(arm64_ssbd_callback_required))
arm_smccc_1_1_smc(ARM_SMCCC_ARCH_WORKAROUND_2, 0, NULL);
#endif
}
@@ -507,7 +507,7 @@ static inline void __set_host_arch_workaround_state(struct kvm_vcpu *vcpu)
* If the guest has disabled the workaround, bring it back on.
*/
if (__needs_ssbd_off(vcpu) &&
- __hyp_this_cpu_read(arm64_ssbd_callback_required))
+ __this_cpu_read(arm64_ssbd_callback_required))
arm_smccc_1_1_smc(ARM_SMCCC_ARCH_WORKAROUND_2, 1, NULL);
#endif
}
@@ -521,7 +521,7 @@ static inline void __kvm_unexpected_el2_exception(void)
entry = hyp_symbol_addr(__start___kvm_ex_table);
end = hyp_symbol_addr(__stop___kvm_ex_table);
- host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
while (entry < end) {
addr = (unsigned long)&entry->insn + entry->insn;
diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c
index 0970442d2dbc..cc4f8e790fb3 100644
--- a/arch/arm64/kvm/hyp/nvhe/switch.c
+++ b/arch/arm64/kvm/hyp/nvhe/switch.c
@@ -175,7 +175,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
vcpu = kern_hyp_va(vcpu);
- host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
host_ctxt->__hyp_running_vcpu = vcpu;
guest_ctxt = &vcpu->arch.ctxt;
diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c
index c1da4f86ccac..575e8054f116 100644
--- a/arch/arm64/kvm/hyp/vhe/switch.c
+++ b/arch/arm64/kvm/hyp/vhe/switch.c
@@ -108,7 +108,7 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
struct kvm_cpu_context *guest_ctxt;
u64 exit_code;
- host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
host_ctxt->__hyp_running_vcpu = vcpu;
guest_ctxt = &vcpu->arch.ctxt;
diff --git a/arch/arm64/kvm/hyp/vhe/sysreg-sr.c b/arch/arm64/kvm/hyp/vhe/sysreg-sr.c
index 996471e4c138..2a0b8c88d74f 100644
--- a/arch/arm64/kvm/hyp/vhe/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/vhe/sysreg-sr.c
@@ -66,7 +66,7 @@ void kvm_vcpu_load_sysregs_vhe(struct kvm_vcpu *vcpu)
struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt;
struct kvm_cpu_context *host_ctxt;
- host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
__sysreg_save_user_state(host_ctxt);
/*
@@ -100,7 +100,7 @@ void kvm_vcpu_put_sysregs_vhe(struct kvm_vcpu *vcpu)
struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt;
struct kvm_cpu_context *host_ctxt;
- host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
deactivate_traps_vhe_put();
__sysreg_save_el1_state(guest_ctxt);
--
2.28.0.618.gf4bc123cb7-goog
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v3 04/11] kvm: arm64: Remove __hyp_this_cpu_read
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Suzuki K Poulose, Catalin Marinas, linux-kernel,
James Morse, linux-arm-kernel, Marc Zyngier, Tejun Heo,
Dennis Zhou, Christoph Lameter, David Brazdil, Will Deacon,
Julien Thierry, Andrew Scull
this_cpu_ptr is meant for use in kernel proper because it selects between
TPIDR_EL1/2 based on nVHE/VHE. __hyp_this_cpu_ptr was used in hyp to always
select TPIDR_EL2. Unify all users behind this_cpu_ptr and friends by
selecting _EL2 register under __KVM_VHE/NVHE_HYPERVISOR__.
Under CONFIG_DEBUG_PREEMPT, the kernel helpers perform a preemption check
which is omitted by the hyp helpers. Preserve the behavior for nVHE by
overriding the corresponding macros under __KVM_NVHE_HYPERVISOR__. Extend
the checks into VHE hyp code.
Acked-by: Andrew Scull <ascull@google.com>
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/kvm_asm.h | 20 --------------
arch/arm64/include/asm/percpu.h | 33 +++++++++++++++++++++--
arch/arm64/kvm/hyp/include/hyp/debug-sr.h | 4 +--
arch/arm64/kvm/hyp/include/hyp/switch.h | 8 +++---
arch/arm64/kvm/hyp/nvhe/switch.c | 2 +-
arch/arm64/kvm/hyp/vhe/switch.c | 2 +-
arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 4 +--
7 files changed, 41 insertions(+), 32 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index c085032e2e3e..c196eec25498 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -143,26 +143,6 @@ extern char __smccc_workaround_1_smc[__SMCCC_WORKAROUND_1_SMC_SZ];
addr; \
})
-/*
- * Home-grown __this_cpu_{ptr,read} variants that always work at HYP,
- * provided that sym is really a *symbol* and not a pointer obtained from
- * a data structure. As for SHIFT_PERCPU_PTR(), the creative casting keeps
- * sparse quiet.
- */
-#define __hyp_this_cpu_ptr(sym) \
- ({ \
- void *__ptr; \
- __verify_pcpu_ptr(&sym); \
- __ptr = hyp_symbol_addr(sym); \
- __ptr += read_sysreg(tpidr_el2); \
- (typeof(sym) __kernel __force *)__ptr; \
- })
-
-#define __hyp_this_cpu_read(sym) \
- ({ \
- *__hyp_this_cpu_ptr(sym); \
- })
-
#define __KVM_EXTABLE(from, to) \
" .pushsection __kvm_ex_table, \"a\"\n" \
" .align 3\n" \
diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index 0b6409b89e5e..36f304401c38 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -19,7 +19,21 @@ static inline void set_my_cpu_offset(unsigned long off)
:: "r" (off) : "memory");
}
-static inline unsigned long __my_cpu_offset(void)
+static inline unsigned long __hyp_my_cpu_offset(void)
+{
+ unsigned long off;
+
+ /*
+ * We want to allow caching the value, so avoid using volatile and
+ * instead use a fake stack read to hazard against barrier().
+ */
+ asm("mrs %0, tpidr_el2" : "=r" (off) :
+ "Q" (*(const unsigned long *)current_stack_pointer));
+
+ return off;
+}
+
+static inline unsigned long __kern_my_cpu_offset(void)
{
unsigned long off;
@@ -35,7 +49,12 @@ static inline unsigned long __my_cpu_offset(void)
return off;
}
-#define __my_cpu_offset __my_cpu_offset()
+
+#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
+#define __my_cpu_offset __hyp_my_cpu_offset()
+#else
+#define __my_cpu_offset __kern_my_cpu_offset()
+#endif
#define PERCPU_RW_OPS(sz) \
static inline unsigned long __percpu_read_##sz(void *ptr) \
@@ -227,4 +246,14 @@ PERCPU_RET_OP(add, add, ldadd)
#include <asm-generic/percpu.h>
+/* Redefine macros for nVHE hyp under DEBUG_PREEMPT to avoid its dependencies. */
+#if defined(__KVM_NVHE_HYPERVISOR__) && defined(CONFIG_DEBUG_PREEMPT)
+#undef this_cpu_ptr
+#define this_cpu_ptr raw_cpu_ptr
+#undef __this_cpu_read
+#define __this_cpu_read raw_cpu_read
+#undef __this_cpu_write
+#define __this_cpu_write raw_cpu_write
+#endif
+
#endif /* __ASM_PERCPU_H */
diff --git a/arch/arm64/kvm/hyp/include/hyp/debug-sr.h b/arch/arm64/kvm/hyp/include/hyp/debug-sr.h
index 5e28ea6aa097..4ebe9f558f3a 100644
--- a/arch/arm64/kvm/hyp/include/hyp/debug-sr.h
+++ b/arch/arm64/kvm/hyp/include/hyp/debug-sr.h
@@ -135,7 +135,7 @@ static inline void __debug_switch_to_guest_common(struct kvm_vcpu *vcpu)
if (!(vcpu->arch.flags & KVM_ARM64_DEBUG_DIRTY))
return;
- host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
guest_ctxt = &vcpu->arch.ctxt;
host_dbg = &vcpu->arch.host_debug_state.regs;
guest_dbg = kern_hyp_va(vcpu->arch.debug_ptr);
@@ -154,7 +154,7 @@ static inline void __debug_switch_to_host_common(struct kvm_vcpu *vcpu)
if (!(vcpu->arch.flags & KVM_ARM64_DEBUG_DIRTY))
return;
- host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
guest_ctxt = &vcpu->arch.ctxt;
host_dbg = &vcpu->arch.host_debug_state.regs;
guest_dbg = kern_hyp_va(vcpu->arch.debug_ptr);
diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
index 5b6b8fa00f0a..f150407fa798 100644
--- a/arch/arm64/kvm/hyp/include/hyp/switch.h
+++ b/arch/arm64/kvm/hyp/include/hyp/switch.h
@@ -386,7 +386,7 @@ static inline bool __hyp_handle_ptrauth(struct kvm_vcpu *vcpu)
!esr_is_ptrauth_trap(kvm_vcpu_get_esr(vcpu)))
return false;
- ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
__ptrauth_save_key(ctxt, APIA);
__ptrauth_save_key(ctxt, APIB);
__ptrauth_save_key(ctxt, APDA);
@@ -495,7 +495,7 @@ static inline void __set_guest_arch_workaround_state(struct kvm_vcpu *vcpu)
* guest wants it disabled, so be it...
*/
if (__needs_ssbd_off(vcpu) &&
- __hyp_this_cpu_read(arm64_ssbd_callback_required))
+ __this_cpu_read(arm64_ssbd_callback_required))
arm_smccc_1_1_smc(ARM_SMCCC_ARCH_WORKAROUND_2, 0, NULL);
#endif
}
@@ -507,7 +507,7 @@ static inline void __set_host_arch_workaround_state(struct kvm_vcpu *vcpu)
* If the guest has disabled the workaround, bring it back on.
*/
if (__needs_ssbd_off(vcpu) &&
- __hyp_this_cpu_read(arm64_ssbd_callback_required))
+ __this_cpu_read(arm64_ssbd_callback_required))
arm_smccc_1_1_smc(ARM_SMCCC_ARCH_WORKAROUND_2, 1, NULL);
#endif
}
@@ -521,7 +521,7 @@ static inline void __kvm_unexpected_el2_exception(void)
entry = hyp_symbol_addr(__start___kvm_ex_table);
end = hyp_symbol_addr(__stop___kvm_ex_table);
- host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
while (entry < end) {
addr = (unsigned long)&entry->insn + entry->insn;
diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c
index 0970442d2dbc..cc4f8e790fb3 100644
--- a/arch/arm64/kvm/hyp/nvhe/switch.c
+++ b/arch/arm64/kvm/hyp/nvhe/switch.c
@@ -175,7 +175,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
vcpu = kern_hyp_va(vcpu);
- host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
host_ctxt->__hyp_running_vcpu = vcpu;
guest_ctxt = &vcpu->arch.ctxt;
diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c
index c1da4f86ccac..575e8054f116 100644
--- a/arch/arm64/kvm/hyp/vhe/switch.c
+++ b/arch/arm64/kvm/hyp/vhe/switch.c
@@ -108,7 +108,7 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
struct kvm_cpu_context *guest_ctxt;
u64 exit_code;
- host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
host_ctxt->__hyp_running_vcpu = vcpu;
guest_ctxt = &vcpu->arch.ctxt;
diff --git a/arch/arm64/kvm/hyp/vhe/sysreg-sr.c b/arch/arm64/kvm/hyp/vhe/sysreg-sr.c
index 996471e4c138..2a0b8c88d74f 100644
--- a/arch/arm64/kvm/hyp/vhe/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/vhe/sysreg-sr.c
@@ -66,7 +66,7 @@ void kvm_vcpu_load_sysregs_vhe(struct kvm_vcpu *vcpu)
struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt;
struct kvm_cpu_context *host_ctxt;
- host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
__sysreg_save_user_state(host_ctxt);
/*
@@ -100,7 +100,7 @@ void kvm_vcpu_put_sysregs_vhe(struct kvm_vcpu *vcpu)
struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt;
struct kvm_cpu_context *host_ctxt;
- host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
deactivate_traps_vhe_put();
__sysreg_save_el1_state(guest_ctxt);
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v3 04/11] kvm: arm64: Remove __hyp_this_cpu_read
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Catalin Marinas, linux-kernel, linux-arm-kernel,
Marc Zyngier, Tejun Heo, Dennis Zhou, Christoph Lameter,
Will Deacon
this_cpu_ptr is meant for use in kernel proper because it selects between
TPIDR_EL1/2 based on nVHE/VHE. __hyp_this_cpu_ptr was used in hyp to always
select TPIDR_EL2. Unify all users behind this_cpu_ptr and friends by
selecting _EL2 register under __KVM_VHE/NVHE_HYPERVISOR__.
Under CONFIG_DEBUG_PREEMPT, the kernel helpers perform a preemption check
which is omitted by the hyp helpers. Preserve the behavior for nVHE by
overriding the corresponding macros under __KVM_NVHE_HYPERVISOR__. Extend
the checks into VHE hyp code.
Acked-by: Andrew Scull <ascull@google.com>
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/kvm_asm.h | 20 --------------
arch/arm64/include/asm/percpu.h | 33 +++++++++++++++++++++--
arch/arm64/kvm/hyp/include/hyp/debug-sr.h | 4 +--
arch/arm64/kvm/hyp/include/hyp/switch.h | 8 +++---
arch/arm64/kvm/hyp/nvhe/switch.c | 2 +-
arch/arm64/kvm/hyp/vhe/switch.c | 2 +-
arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 4 +--
7 files changed, 41 insertions(+), 32 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index c085032e2e3e..c196eec25498 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -143,26 +143,6 @@ extern char __smccc_workaround_1_smc[__SMCCC_WORKAROUND_1_SMC_SZ];
addr; \
})
-/*
- * Home-grown __this_cpu_{ptr,read} variants that always work at HYP,
- * provided that sym is really a *symbol* and not a pointer obtained from
- * a data structure. As for SHIFT_PERCPU_PTR(), the creative casting keeps
- * sparse quiet.
- */
-#define __hyp_this_cpu_ptr(sym) \
- ({ \
- void *__ptr; \
- __verify_pcpu_ptr(&sym); \
- __ptr = hyp_symbol_addr(sym); \
- __ptr += read_sysreg(tpidr_el2); \
- (typeof(sym) __kernel __force *)__ptr; \
- })
-
-#define __hyp_this_cpu_read(sym) \
- ({ \
- *__hyp_this_cpu_ptr(sym); \
- })
-
#define __KVM_EXTABLE(from, to) \
" .pushsection __kvm_ex_table, \"a\"\n" \
" .align 3\n" \
diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
index 0b6409b89e5e..36f304401c38 100644
--- a/arch/arm64/include/asm/percpu.h
+++ b/arch/arm64/include/asm/percpu.h
@@ -19,7 +19,21 @@ static inline void set_my_cpu_offset(unsigned long off)
:: "r" (off) : "memory");
}
-static inline unsigned long __my_cpu_offset(void)
+static inline unsigned long __hyp_my_cpu_offset(void)
+{
+ unsigned long off;
+
+ /*
+ * We want to allow caching the value, so avoid using volatile and
+ * instead use a fake stack read to hazard against barrier().
+ */
+ asm("mrs %0, tpidr_el2" : "=r" (off) :
+ "Q" (*(const unsigned long *)current_stack_pointer));
+
+ return off;
+}
+
+static inline unsigned long __kern_my_cpu_offset(void)
{
unsigned long off;
@@ -35,7 +49,12 @@ static inline unsigned long __my_cpu_offset(void)
return off;
}
-#define __my_cpu_offset __my_cpu_offset()
+
+#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
+#define __my_cpu_offset __hyp_my_cpu_offset()
+#else
+#define __my_cpu_offset __kern_my_cpu_offset()
+#endif
#define PERCPU_RW_OPS(sz) \
static inline unsigned long __percpu_read_##sz(void *ptr) \
@@ -227,4 +246,14 @@ PERCPU_RET_OP(add, add, ldadd)
#include <asm-generic/percpu.h>
+/* Redefine macros for nVHE hyp under DEBUG_PREEMPT to avoid its dependencies. */
+#if defined(__KVM_NVHE_HYPERVISOR__) && defined(CONFIG_DEBUG_PREEMPT)
+#undef this_cpu_ptr
+#define this_cpu_ptr raw_cpu_ptr
+#undef __this_cpu_read
+#define __this_cpu_read raw_cpu_read
+#undef __this_cpu_write
+#define __this_cpu_write raw_cpu_write
+#endif
+
#endif /* __ASM_PERCPU_H */
diff --git a/arch/arm64/kvm/hyp/include/hyp/debug-sr.h b/arch/arm64/kvm/hyp/include/hyp/debug-sr.h
index 5e28ea6aa097..4ebe9f558f3a 100644
--- a/arch/arm64/kvm/hyp/include/hyp/debug-sr.h
+++ b/arch/arm64/kvm/hyp/include/hyp/debug-sr.h
@@ -135,7 +135,7 @@ static inline void __debug_switch_to_guest_common(struct kvm_vcpu *vcpu)
if (!(vcpu->arch.flags & KVM_ARM64_DEBUG_DIRTY))
return;
- host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
guest_ctxt = &vcpu->arch.ctxt;
host_dbg = &vcpu->arch.host_debug_state.regs;
guest_dbg = kern_hyp_va(vcpu->arch.debug_ptr);
@@ -154,7 +154,7 @@ static inline void __debug_switch_to_host_common(struct kvm_vcpu *vcpu)
if (!(vcpu->arch.flags & KVM_ARM64_DEBUG_DIRTY))
return;
- host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
guest_ctxt = &vcpu->arch.ctxt;
host_dbg = &vcpu->arch.host_debug_state.regs;
guest_dbg = kern_hyp_va(vcpu->arch.debug_ptr);
diff --git a/arch/arm64/kvm/hyp/include/hyp/switch.h b/arch/arm64/kvm/hyp/include/hyp/switch.h
index 5b6b8fa00f0a..f150407fa798 100644
--- a/arch/arm64/kvm/hyp/include/hyp/switch.h
+++ b/arch/arm64/kvm/hyp/include/hyp/switch.h
@@ -386,7 +386,7 @@ static inline bool __hyp_handle_ptrauth(struct kvm_vcpu *vcpu)
!esr_is_ptrauth_trap(kvm_vcpu_get_esr(vcpu)))
return false;
- ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
__ptrauth_save_key(ctxt, APIA);
__ptrauth_save_key(ctxt, APIB);
__ptrauth_save_key(ctxt, APDA);
@@ -495,7 +495,7 @@ static inline void __set_guest_arch_workaround_state(struct kvm_vcpu *vcpu)
* guest wants it disabled, so be it...
*/
if (__needs_ssbd_off(vcpu) &&
- __hyp_this_cpu_read(arm64_ssbd_callback_required))
+ __this_cpu_read(arm64_ssbd_callback_required))
arm_smccc_1_1_smc(ARM_SMCCC_ARCH_WORKAROUND_2, 0, NULL);
#endif
}
@@ -507,7 +507,7 @@ static inline void __set_host_arch_workaround_state(struct kvm_vcpu *vcpu)
* If the guest has disabled the workaround, bring it back on.
*/
if (__needs_ssbd_off(vcpu) &&
- __hyp_this_cpu_read(arm64_ssbd_callback_required))
+ __this_cpu_read(arm64_ssbd_callback_required))
arm_smccc_1_1_smc(ARM_SMCCC_ARCH_WORKAROUND_2, 1, NULL);
#endif
}
@@ -521,7 +521,7 @@ static inline void __kvm_unexpected_el2_exception(void)
entry = hyp_symbol_addr(__start___kvm_ex_table);
end = hyp_symbol_addr(__stop___kvm_ex_table);
- host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
while (entry < end) {
addr = (unsigned long)&entry->insn + entry->insn;
diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c
index 0970442d2dbc..cc4f8e790fb3 100644
--- a/arch/arm64/kvm/hyp/nvhe/switch.c
+++ b/arch/arm64/kvm/hyp/nvhe/switch.c
@@ -175,7 +175,7 @@ int __kvm_vcpu_run(struct kvm_vcpu *vcpu)
vcpu = kern_hyp_va(vcpu);
- host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
host_ctxt->__hyp_running_vcpu = vcpu;
guest_ctxt = &vcpu->arch.ctxt;
diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c
index c1da4f86ccac..575e8054f116 100644
--- a/arch/arm64/kvm/hyp/vhe/switch.c
+++ b/arch/arm64/kvm/hyp/vhe/switch.c
@@ -108,7 +108,7 @@ static int __kvm_vcpu_run_vhe(struct kvm_vcpu *vcpu)
struct kvm_cpu_context *guest_ctxt;
u64 exit_code;
- host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
host_ctxt->__hyp_running_vcpu = vcpu;
guest_ctxt = &vcpu->arch.ctxt;
diff --git a/arch/arm64/kvm/hyp/vhe/sysreg-sr.c b/arch/arm64/kvm/hyp/vhe/sysreg-sr.c
index 996471e4c138..2a0b8c88d74f 100644
--- a/arch/arm64/kvm/hyp/vhe/sysreg-sr.c
+++ b/arch/arm64/kvm/hyp/vhe/sysreg-sr.c
@@ -66,7 +66,7 @@ void kvm_vcpu_load_sysregs_vhe(struct kvm_vcpu *vcpu)
struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt;
struct kvm_cpu_context *host_ctxt;
- host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
__sysreg_save_user_state(host_ctxt);
/*
@@ -100,7 +100,7 @@ void kvm_vcpu_put_sysregs_vhe(struct kvm_vcpu *vcpu)
struct kvm_cpu_context *guest_ctxt = &vcpu->arch.ctxt;
struct kvm_cpu_context *host_ctxt;
- host_ctxt = &__hyp_this_cpu_ptr(kvm_host_data)->host_ctxt;
+ host_ctxt = &this_cpu_ptr(&kvm_host_data)->host_ctxt;
deactivate_traps_vhe_put();
__sysreg_save_el1_state(guest_ctxt);
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v3 04/11] kvm: arm64: Remove __hyp_this_cpu_read
2020-09-16 17:34 ` David Brazdil
(?)
@ 2020-09-18 9:00 ` Will Deacon
-1 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 9:00 UTC (permalink / raw)
To: David Brazdil
Cc: kvmarm, Catalin Marinas, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team,
Andrew Scull
On Wed, Sep 16, 2020 at 06:34:32PM +0100, David Brazdil wrote:
> this_cpu_ptr is meant for use in kernel proper because it selects between
> TPIDR_EL1/2 based on nVHE/VHE. __hyp_this_cpu_ptr was used in hyp to always
> select TPIDR_EL2. Unify all users behind this_cpu_ptr and friends by
> selecting _EL2 register under __KVM_VHE/NVHE_HYPERVISOR__.
>
> Under CONFIG_DEBUG_PREEMPT, the kernel helpers perform a preemption check
> which is omitted by the hyp helpers. Preserve the behavior for nVHE by
> overriding the corresponding macros under __KVM_NVHE_HYPERVISOR__. Extend
> the checks into VHE hyp code.
>
> Acked-by: Andrew Scull <ascull@google.com>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/kvm_asm.h | 20 --------------
> arch/arm64/include/asm/percpu.h | 33 +++++++++++++++++++++--
> arch/arm64/kvm/hyp/include/hyp/debug-sr.h | 4 +--
> arch/arm64/kvm/hyp/include/hyp/switch.h | 8 +++---
> arch/arm64/kvm/hyp/nvhe/switch.c | 2 +-
> arch/arm64/kvm/hyp/vhe/switch.c | 2 +-
> arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 4 +--
> 7 files changed, 41 insertions(+), 32 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
> index c085032e2e3e..c196eec25498 100644
> --- a/arch/arm64/include/asm/kvm_asm.h
> +++ b/arch/arm64/include/asm/kvm_asm.h
> @@ -143,26 +143,6 @@ extern char __smccc_workaround_1_smc[__SMCCC_WORKAROUND_1_SMC_SZ];
> addr; \
> })
>
> -/*
> - * Home-grown __this_cpu_{ptr,read} variants that always work at HYP,
> - * provided that sym is really a *symbol* and not a pointer obtained from
> - * a data structure. As for SHIFT_PERCPU_PTR(), the creative casting keeps
> - * sparse quiet.
> - */
> -#define __hyp_this_cpu_ptr(sym) \
> - ({ \
> - void *__ptr; \
> - __verify_pcpu_ptr(&sym); \
> - __ptr = hyp_symbol_addr(sym); \
> - __ptr += read_sysreg(tpidr_el2); \
> - (typeof(sym) __kernel __force *)__ptr; \
> - })
> -
> -#define __hyp_this_cpu_read(sym) \
> - ({ \
> - *__hyp_this_cpu_ptr(sym); \
> - })
> -
> #define __KVM_EXTABLE(from, to) \
> " .pushsection __kvm_ex_table, \"a\"\n" \
> " .align 3\n" \
> diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
> index 0b6409b89e5e..36f304401c38 100644
> --- a/arch/arm64/include/asm/percpu.h
> +++ b/arch/arm64/include/asm/percpu.h
> @@ -19,7 +19,21 @@ static inline void set_my_cpu_offset(unsigned long off)
> :: "r" (off) : "memory");
> }
>
> -static inline unsigned long __my_cpu_offset(void)
> +static inline unsigned long __hyp_my_cpu_offset(void)
> +{
> + unsigned long off;
> +
> + /*
> + * We want to allow caching the value, so avoid using volatile and
> + * instead use a fake stack read to hazard against barrier().
> + */
I don't think we need to copy/paste the comment...
> + asm("mrs %0, tpidr_el2" : "=r" (off) :
> + "Q" (*(const unsigned long *)current_stack_pointer));
... especially given that we're not preemptible at EL2 with nVHE, maybe
we don't need to play this trick at all because we're always going to be
on the same CPU. So we could actually just do:
return read_sysreg(tpidr_el2);
which is much better, and the comment should say something to that effect.
> +
> + return off;
> +}
> +
> +static inline unsigned long __kern_my_cpu_offset(void)
> {
> unsigned long off;
>
> @@ -35,7 +49,12 @@ static inline unsigned long __my_cpu_offset(void)
>
> return off;
> }
> -#define __my_cpu_offset __my_cpu_offset()
> +
> +#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
> +#define __my_cpu_offset __hyp_my_cpu_offset()
Why would VHE code need to use this? Especially in light of my preemption
comments above, shouldn't it now be using __kern_my_cpu_offset()?
Will
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v3 04/11] kvm: arm64: Remove __hyp_this_cpu_read
@ 2020-09-18 9:00 ` Will Deacon
0 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 9:00 UTC (permalink / raw)
To: David Brazdil
Cc: Suzuki K Poulose, Catalin Marinas, linux-kernel, James Morse,
linux-arm-kernel, Marc Zyngier, Tejun Heo, Dennis Zhou,
Christoph Lameter, kernel-team, kvmarm, Julien Thierry,
Andrew Scull
On Wed, Sep 16, 2020 at 06:34:32PM +0100, David Brazdil wrote:
> this_cpu_ptr is meant for use in kernel proper because it selects between
> TPIDR_EL1/2 based on nVHE/VHE. __hyp_this_cpu_ptr was used in hyp to always
> select TPIDR_EL2. Unify all users behind this_cpu_ptr and friends by
> selecting _EL2 register under __KVM_VHE/NVHE_HYPERVISOR__.
>
> Under CONFIG_DEBUG_PREEMPT, the kernel helpers perform a preemption check
> which is omitted by the hyp helpers. Preserve the behavior for nVHE by
> overriding the corresponding macros under __KVM_NVHE_HYPERVISOR__. Extend
> the checks into VHE hyp code.
>
> Acked-by: Andrew Scull <ascull@google.com>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/kvm_asm.h | 20 --------------
> arch/arm64/include/asm/percpu.h | 33 +++++++++++++++++++++--
> arch/arm64/kvm/hyp/include/hyp/debug-sr.h | 4 +--
> arch/arm64/kvm/hyp/include/hyp/switch.h | 8 +++---
> arch/arm64/kvm/hyp/nvhe/switch.c | 2 +-
> arch/arm64/kvm/hyp/vhe/switch.c | 2 +-
> arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 4 +--
> 7 files changed, 41 insertions(+), 32 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
> index c085032e2e3e..c196eec25498 100644
> --- a/arch/arm64/include/asm/kvm_asm.h
> +++ b/arch/arm64/include/asm/kvm_asm.h
> @@ -143,26 +143,6 @@ extern char __smccc_workaround_1_smc[__SMCCC_WORKAROUND_1_SMC_SZ];
> addr; \
> })
>
> -/*
> - * Home-grown __this_cpu_{ptr,read} variants that always work at HYP,
> - * provided that sym is really a *symbol* and not a pointer obtained from
> - * a data structure. As for SHIFT_PERCPU_PTR(), the creative casting keeps
> - * sparse quiet.
> - */
> -#define __hyp_this_cpu_ptr(sym) \
> - ({ \
> - void *__ptr; \
> - __verify_pcpu_ptr(&sym); \
> - __ptr = hyp_symbol_addr(sym); \
> - __ptr += read_sysreg(tpidr_el2); \
> - (typeof(sym) __kernel __force *)__ptr; \
> - })
> -
> -#define __hyp_this_cpu_read(sym) \
> - ({ \
> - *__hyp_this_cpu_ptr(sym); \
> - })
> -
> #define __KVM_EXTABLE(from, to) \
> " .pushsection __kvm_ex_table, \"a\"\n" \
> " .align 3\n" \
> diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
> index 0b6409b89e5e..36f304401c38 100644
> --- a/arch/arm64/include/asm/percpu.h
> +++ b/arch/arm64/include/asm/percpu.h
> @@ -19,7 +19,21 @@ static inline void set_my_cpu_offset(unsigned long off)
> :: "r" (off) : "memory");
> }
>
> -static inline unsigned long __my_cpu_offset(void)
> +static inline unsigned long __hyp_my_cpu_offset(void)
> +{
> + unsigned long off;
> +
> + /*
> + * We want to allow caching the value, so avoid using volatile and
> + * instead use a fake stack read to hazard against barrier().
> + */
I don't think we need to copy/paste the comment...
> + asm("mrs %0, tpidr_el2" : "=r" (off) :
> + "Q" (*(const unsigned long *)current_stack_pointer));
... especially given that we're not preemptible at EL2 with nVHE, maybe
we don't need to play this trick at all because we're always going to be
on the same CPU. So we could actually just do:
return read_sysreg(tpidr_el2);
which is much better, and the comment should say something to that effect.
> +
> + return off;
> +}
> +
> +static inline unsigned long __kern_my_cpu_offset(void)
> {
> unsigned long off;
>
> @@ -35,7 +49,12 @@ static inline unsigned long __my_cpu_offset(void)
>
> return off;
> }
> -#define __my_cpu_offset __my_cpu_offset()
> +
> +#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
> +#define __my_cpu_offset __hyp_my_cpu_offset()
Why would VHE code need to use this? Especially in light of my preemption
comments above, shouldn't it now be using __kern_my_cpu_offset()?
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v3 04/11] kvm: arm64: Remove __hyp_this_cpu_read
@ 2020-09-18 9:00 ` Will Deacon
0 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 9:00 UTC (permalink / raw)
To: David Brazdil
Cc: Catalin Marinas, linux-kernel, linux-arm-kernel, Marc Zyngier,
Tejun Heo, Dennis Zhou, Christoph Lameter, kernel-team, kvmarm
On Wed, Sep 16, 2020 at 06:34:32PM +0100, David Brazdil wrote:
> this_cpu_ptr is meant for use in kernel proper because it selects between
> TPIDR_EL1/2 based on nVHE/VHE. __hyp_this_cpu_ptr was used in hyp to always
> select TPIDR_EL2. Unify all users behind this_cpu_ptr and friends by
> selecting _EL2 register under __KVM_VHE/NVHE_HYPERVISOR__.
>
> Under CONFIG_DEBUG_PREEMPT, the kernel helpers perform a preemption check
> which is omitted by the hyp helpers. Preserve the behavior for nVHE by
> overriding the corresponding macros under __KVM_NVHE_HYPERVISOR__. Extend
> the checks into VHE hyp code.
>
> Acked-by: Andrew Scull <ascull@google.com>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/kvm_asm.h | 20 --------------
> arch/arm64/include/asm/percpu.h | 33 +++++++++++++++++++++--
> arch/arm64/kvm/hyp/include/hyp/debug-sr.h | 4 +--
> arch/arm64/kvm/hyp/include/hyp/switch.h | 8 +++---
> arch/arm64/kvm/hyp/nvhe/switch.c | 2 +-
> arch/arm64/kvm/hyp/vhe/switch.c | 2 +-
> arch/arm64/kvm/hyp/vhe/sysreg-sr.c | 4 +--
> 7 files changed, 41 insertions(+), 32 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
> index c085032e2e3e..c196eec25498 100644
> --- a/arch/arm64/include/asm/kvm_asm.h
> +++ b/arch/arm64/include/asm/kvm_asm.h
> @@ -143,26 +143,6 @@ extern char __smccc_workaround_1_smc[__SMCCC_WORKAROUND_1_SMC_SZ];
> addr; \
> })
>
> -/*
> - * Home-grown __this_cpu_{ptr,read} variants that always work at HYP,
> - * provided that sym is really a *symbol* and not a pointer obtained from
> - * a data structure. As for SHIFT_PERCPU_PTR(), the creative casting keeps
> - * sparse quiet.
> - */
> -#define __hyp_this_cpu_ptr(sym) \
> - ({ \
> - void *__ptr; \
> - __verify_pcpu_ptr(&sym); \
> - __ptr = hyp_symbol_addr(sym); \
> - __ptr += read_sysreg(tpidr_el2); \
> - (typeof(sym) __kernel __force *)__ptr; \
> - })
> -
> -#define __hyp_this_cpu_read(sym) \
> - ({ \
> - *__hyp_this_cpu_ptr(sym); \
> - })
> -
> #define __KVM_EXTABLE(from, to) \
> " .pushsection __kvm_ex_table, \"a\"\n" \
> " .align 3\n" \
> diff --git a/arch/arm64/include/asm/percpu.h b/arch/arm64/include/asm/percpu.h
> index 0b6409b89e5e..36f304401c38 100644
> --- a/arch/arm64/include/asm/percpu.h
> +++ b/arch/arm64/include/asm/percpu.h
> @@ -19,7 +19,21 @@ static inline void set_my_cpu_offset(unsigned long off)
> :: "r" (off) : "memory");
> }
>
> -static inline unsigned long __my_cpu_offset(void)
> +static inline unsigned long __hyp_my_cpu_offset(void)
> +{
> + unsigned long off;
> +
> + /*
> + * We want to allow caching the value, so avoid using volatile and
> + * instead use a fake stack read to hazard against barrier().
> + */
I don't think we need to copy/paste the comment...
> + asm("mrs %0, tpidr_el2" : "=r" (off) :
> + "Q" (*(const unsigned long *)current_stack_pointer));
... especially given that we're not preemptible at EL2 with nVHE, maybe
we don't need to play this trick at all because we're always going to be
on the same CPU. So we could actually just do:
return read_sysreg(tpidr_el2);
which is much better, and the comment should say something to that effect.
> +
> + return off;
> +}
> +
> +static inline unsigned long __kern_my_cpu_offset(void)
> {
> unsigned long off;
>
> @@ -35,7 +49,12 @@ static inline unsigned long __my_cpu_offset(void)
>
> return off;
> }
> -#define __my_cpu_offset __my_cpu_offset()
> +
> +#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
> +#define __my_cpu_offset __hyp_my_cpu_offset()
Why would VHE code need to use this? Especially in light of my preemption
comments above, shouldn't it now be using __kern_my_cpu_offset()?
Will
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v3 04/11] kvm: arm64: Remove __hyp_this_cpu_read
2020-09-18 9:00 ` Will Deacon
(?)
@ 2020-09-21 13:43 ` David Brazdil
-1 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-21 13:43 UTC (permalink / raw)
To: Will Deacon
Cc: kvmarm, Catalin Marinas, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team,
Andrew Scull
Hi Will,
> > +static inline unsigned long __hyp_my_cpu_offset(void)
> > +{
> > + unsigned long off;
> > +
> > + /*
> > + * We want to allow caching the value, so avoid using volatile and
> > + * instead use a fake stack read to hazard against barrier().
> > + */
>
> I don't think we need to copy/paste the comment...
>
> > + asm("mrs %0, tpidr_el2" : "=r" (off) :
> > + "Q" (*(const unsigned long *)current_stack_pointer));
>
> ... especially given that we're not preemptible at EL2 with nVHE, maybe
> we don't need to play this trick at all because we're always going to be
> on the same CPU. So we could actually just do:
>
> return read_sysreg(tpidr_el2);
>
> which is much better, and the comment should say something to that effect.
I must be misinterpreting the comment. I understood that it enables the compiler
optimizing multiple reads of TPIDR by avoiding 'asm volatile' (signaling that
the value does not change between reads). So what exactly does it do?
read_sysreg expands to 'asm volatile' but I have no problem with priotizing
readability over a micro-optimization.
> > +#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
> > +#define __my_cpu_offset __hyp_my_cpu_offset()
>
> Why would VHE code need to use this? Especially in light of my preemption
> comments above, shouldn't it now be using __kern_my_cpu_offset()?
During v2 review Andrew Scull pointed out we can avoid alternatives on VHE code
by using __hyp_my_cpu_offset for it as well. Obviously if __hyp_my_cpu_offset
becomes nVHE-specific, we can always move VHE back to __kern. This was just
about saving a few cycles during boot.
David
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v3 04/11] kvm: arm64: Remove __hyp_this_cpu_read
@ 2020-09-21 13:43 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-21 13:43 UTC (permalink / raw)
To: Will Deacon
Cc: Suzuki K Poulose, Catalin Marinas, linux-kernel, James Morse,
linux-arm-kernel, Marc Zyngier, Tejun Heo, Dennis Zhou,
Christoph Lameter, kernel-team, kvmarm, Julien Thierry,
Andrew Scull
Hi Will,
> > +static inline unsigned long __hyp_my_cpu_offset(void)
> > +{
> > + unsigned long off;
> > +
> > + /*
> > + * We want to allow caching the value, so avoid using volatile and
> > + * instead use a fake stack read to hazard against barrier().
> > + */
>
> I don't think we need to copy/paste the comment...
>
> > + asm("mrs %0, tpidr_el2" : "=r" (off) :
> > + "Q" (*(const unsigned long *)current_stack_pointer));
>
> ... especially given that we're not preemptible at EL2 with nVHE, maybe
> we don't need to play this trick at all because we're always going to be
> on the same CPU. So we could actually just do:
>
> return read_sysreg(tpidr_el2);
>
> which is much better, and the comment should say something to that effect.
I must be misinterpreting the comment. I understood that it enables the compiler
optimizing multiple reads of TPIDR by avoiding 'asm volatile' (signaling that
the value does not change between reads). So what exactly does it do?
read_sysreg expands to 'asm volatile' but I have no problem with priotizing
readability over a micro-optimization.
> > +#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
> > +#define __my_cpu_offset __hyp_my_cpu_offset()
>
> Why would VHE code need to use this? Especially in light of my preemption
> comments above, shouldn't it now be using __kern_my_cpu_offset()?
During v2 review Andrew Scull pointed out we can avoid alternatives on VHE code
by using __hyp_my_cpu_offset for it as well. Obviously if __hyp_my_cpu_offset
becomes nVHE-specific, we can always move VHE back to __kern. This was just
about saving a few cycles during boot.
David
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v3 04/11] kvm: arm64: Remove __hyp_this_cpu_read
@ 2020-09-21 13:43 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-21 13:43 UTC (permalink / raw)
To: Will Deacon
Cc: Catalin Marinas, linux-kernel, linux-arm-kernel, Marc Zyngier,
Tejun Heo, Dennis Zhou, Christoph Lameter, kernel-team, kvmarm
Hi Will,
> > +static inline unsigned long __hyp_my_cpu_offset(void)
> > +{
> > + unsigned long off;
> > +
> > + /*
> > + * We want to allow caching the value, so avoid using volatile and
> > + * instead use a fake stack read to hazard against barrier().
> > + */
>
> I don't think we need to copy/paste the comment...
>
> > + asm("mrs %0, tpidr_el2" : "=r" (off) :
> > + "Q" (*(const unsigned long *)current_stack_pointer));
>
> ... especially given that we're not preemptible at EL2 with nVHE, maybe
> we don't need to play this trick at all because we're always going to be
> on the same CPU. So we could actually just do:
>
> return read_sysreg(tpidr_el2);
>
> which is much better, and the comment should say something to that effect.
I must be misinterpreting the comment. I understood that it enables the compiler
optimizing multiple reads of TPIDR by avoiding 'asm volatile' (signaling that
the value does not change between reads). So what exactly does it do?
read_sysreg expands to 'asm volatile' but I have no problem with priotizing
readability over a micro-optimization.
> > +#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
> > +#define __my_cpu_offset __hyp_my_cpu_offset()
>
> Why would VHE code need to use this? Especially in light of my preemption
> comments above, shouldn't it now be using __kern_my_cpu_offset()?
During v2 review Andrew Scull pointed out we can avoid alternatives on VHE code
by using __hyp_my_cpu_offset for it as well. Obviously if __hyp_my_cpu_offset
becomes nVHE-specific, we can always move VHE back to __kern. This was just
about saving a few cycles during boot.
David
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v3 05/11] kvm: arm64: Remove hyp_adr/ldr_this_cpu
2020-09-16 17:34 ` David Brazdil
(?)
@ 2020-09-16 17:34 ` David Brazdil
-1 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: Catalin Marinas, Will Deacon, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team,
David Brazdil, Andrew Scull
The hyp_adr/ldr_this_cpu helpers were introduced for use in hyp code
because they always needed to use TPIDR_EL2 for base, while
adr/ldr_this_cpu from kernel proper would select between TPIDR_EL2 and
_EL1 based on VHE/nVHE.
Simplify this now that the hyp mode case can be handled using the
__KVM_VHE/NVHE_HYPERVISOR__ macros.
Acked-by: Andrew Scull <ascull@google.com>
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/assembler.h | 27 +++++++++++++++++----------
arch/arm64/include/asm/kvm_asm.h | 14 +-------------
arch/arm64/kvm/hyp/hyp-entry.S | 2 +-
3 files changed, 19 insertions(+), 24 deletions(-)
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index 54d181177656..f79231a0f949 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -218,6 +218,21 @@ lr .req x30 // link register
str \src, [\tmp, :lo12:\sym]
.endm
+ /*
+ * @dst: destination register (32 or 64 bit wide)
+ */
+ .macro this_cpu_offset, dst
+#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
+ mrs \dst, tpidr_el2
+#else
+alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
+ mrs \dst, tpidr_el1
+alternative_else
+ mrs \dst, tpidr_el2
+alternative_endif
+#endif
+ .endm
+
/*
* @dst: Result of per_cpu(sym, smp_processor_id()) (can be SP)
* @sym: The name of the per-cpu variable
@@ -226,11 +241,7 @@ lr .req x30 // link register
.macro adr_this_cpu, dst, sym, tmp
adrp \tmp, \sym
add \dst, \tmp, #:lo12:\sym
-alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
- mrs \tmp, tpidr_el1
-alternative_else
- mrs \tmp, tpidr_el2
-alternative_endif
+ this_cpu_offset \tmp
add \dst, \dst, \tmp
.endm
@@ -241,11 +252,7 @@ alternative_endif
*/
.macro ldr_this_cpu dst, sym, tmp
adr_l \dst, \sym
-alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
- mrs \tmp, tpidr_el1
-alternative_else
- mrs \tmp, tpidr_el2
-alternative_endif
+ this_cpu_offset \tmp
ldr \dst, [\dst, \tmp]
.endm
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index c196eec25498..cf9456663289 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -173,20 +173,8 @@ extern char __smccc_workaround_1_smc[__SMCCC_WORKAROUND_1_SMC_SZ];
#else /* __ASSEMBLY__ */
-.macro hyp_adr_this_cpu reg, sym, tmp
- adr_l \reg, \sym
- mrs \tmp, tpidr_el2
- add \reg, \reg, \tmp
-.endm
-
-.macro hyp_ldr_this_cpu reg, sym, tmp
- adr_l \reg, \sym
- mrs \tmp, tpidr_el2
- ldr \reg, [\reg, \tmp]
-.endm
-
.macro get_host_ctxt reg, tmp
- hyp_adr_this_cpu \reg, kvm_host_data, \tmp
+ adr_this_cpu \reg, kvm_host_data, \tmp
add \reg, \reg, #HOST_DATA_CONTEXT
.endm
diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S
index 46b4dab933d0..fba91c2ab410 100644
--- a/arch/arm64/kvm/hyp/hyp-entry.S
+++ b/arch/arm64/kvm/hyp/hyp-entry.S
@@ -132,7 +132,7 @@ alternative_cb_end
str x0, [x2, #VCPU_WORKAROUND_FLAGS]
/* Check that we actually need to perform the call */
- hyp_ldr_this_cpu x0, arm64_ssbd_callback_required, x2
+ ldr_this_cpu x0, arm64_ssbd_callback_required, x2
cbz x0, wa2_end
mov w0, #ARM_SMCCC_ARCH_WORKAROUND_2
--
2.28.0.618.gf4bc123cb7-goog
^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH v3 05/11] kvm: arm64: Remove hyp_adr/ldr_this_cpu
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Suzuki K Poulose, Catalin Marinas, linux-kernel,
James Morse, linux-arm-kernel, Marc Zyngier, Tejun Heo,
Dennis Zhou, Christoph Lameter, David Brazdil, Will Deacon,
Julien Thierry, Andrew Scull
The hyp_adr/ldr_this_cpu helpers were introduced for use in hyp code
because they always needed to use TPIDR_EL2 for base, while
adr/ldr_this_cpu from kernel proper would select between TPIDR_EL2 and
_EL1 based on VHE/nVHE.
Simplify this now that the hyp mode case can be handled using the
__KVM_VHE/NVHE_HYPERVISOR__ macros.
Acked-by: Andrew Scull <ascull@google.com>
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/assembler.h | 27 +++++++++++++++++----------
arch/arm64/include/asm/kvm_asm.h | 14 +-------------
arch/arm64/kvm/hyp/hyp-entry.S | 2 +-
3 files changed, 19 insertions(+), 24 deletions(-)
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index 54d181177656..f79231a0f949 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -218,6 +218,21 @@ lr .req x30 // link register
str \src, [\tmp, :lo12:\sym]
.endm
+ /*
+ * @dst: destination register (32 or 64 bit wide)
+ */
+ .macro this_cpu_offset, dst
+#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
+ mrs \dst, tpidr_el2
+#else
+alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
+ mrs \dst, tpidr_el1
+alternative_else
+ mrs \dst, tpidr_el2
+alternative_endif
+#endif
+ .endm
+
/*
* @dst: Result of per_cpu(sym, smp_processor_id()) (can be SP)
* @sym: The name of the per-cpu variable
@@ -226,11 +241,7 @@ lr .req x30 // link register
.macro adr_this_cpu, dst, sym, tmp
adrp \tmp, \sym
add \dst, \tmp, #:lo12:\sym
-alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
- mrs \tmp, tpidr_el1
-alternative_else
- mrs \tmp, tpidr_el2
-alternative_endif
+ this_cpu_offset \tmp
add \dst, \dst, \tmp
.endm
@@ -241,11 +252,7 @@ alternative_endif
*/
.macro ldr_this_cpu dst, sym, tmp
adr_l \dst, \sym
-alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
- mrs \tmp, tpidr_el1
-alternative_else
- mrs \tmp, tpidr_el2
-alternative_endif
+ this_cpu_offset \tmp
ldr \dst, [\dst, \tmp]
.endm
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index c196eec25498..cf9456663289 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -173,20 +173,8 @@ extern char __smccc_workaround_1_smc[__SMCCC_WORKAROUND_1_SMC_SZ];
#else /* __ASSEMBLY__ */
-.macro hyp_adr_this_cpu reg, sym, tmp
- adr_l \reg, \sym
- mrs \tmp, tpidr_el2
- add \reg, \reg, \tmp
-.endm
-
-.macro hyp_ldr_this_cpu reg, sym, tmp
- adr_l \reg, \sym
- mrs \tmp, tpidr_el2
- ldr \reg, [\reg, \tmp]
-.endm
-
.macro get_host_ctxt reg, tmp
- hyp_adr_this_cpu \reg, kvm_host_data, \tmp
+ adr_this_cpu \reg, kvm_host_data, \tmp
add \reg, \reg, #HOST_DATA_CONTEXT
.endm
diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S
index 46b4dab933d0..fba91c2ab410 100644
--- a/arch/arm64/kvm/hyp/hyp-entry.S
+++ b/arch/arm64/kvm/hyp/hyp-entry.S
@@ -132,7 +132,7 @@ alternative_cb_end
str x0, [x2, #VCPU_WORKAROUND_FLAGS]
/* Check that we actually need to perform the call */
- hyp_ldr_this_cpu x0, arm64_ssbd_callback_required, x2
+ ldr_this_cpu x0, arm64_ssbd_callback_required, x2
cbz x0, wa2_end
mov w0, #ARM_SMCCC_ARCH_WORKAROUND_2
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH v3 05/11] kvm: arm64: Remove hyp_adr/ldr_this_cpu
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Catalin Marinas, linux-kernel, linux-arm-kernel,
Marc Zyngier, Tejun Heo, Dennis Zhou, Christoph Lameter,
Will Deacon
The hyp_adr/ldr_this_cpu helpers were introduced for use in hyp code
because they always needed to use TPIDR_EL2 for base, while
adr/ldr_this_cpu from kernel proper would select between TPIDR_EL2 and
_EL1 based on VHE/nVHE.
Simplify this now that the hyp mode case can be handled using the
__KVM_VHE/NVHE_HYPERVISOR__ macros.
Acked-by: Andrew Scull <ascull@google.com>
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/assembler.h | 27 +++++++++++++++++----------
arch/arm64/include/asm/kvm_asm.h | 14 +-------------
arch/arm64/kvm/hyp/hyp-entry.S | 2 +-
3 files changed, 19 insertions(+), 24 deletions(-)
diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
index 54d181177656..f79231a0f949 100644
--- a/arch/arm64/include/asm/assembler.h
+++ b/arch/arm64/include/asm/assembler.h
@@ -218,6 +218,21 @@ lr .req x30 // link register
str \src, [\tmp, :lo12:\sym]
.endm
+ /*
+ * @dst: destination register (32 or 64 bit wide)
+ */
+ .macro this_cpu_offset, dst
+#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
+ mrs \dst, tpidr_el2
+#else
+alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
+ mrs \dst, tpidr_el1
+alternative_else
+ mrs \dst, tpidr_el2
+alternative_endif
+#endif
+ .endm
+
/*
* @dst: Result of per_cpu(sym, smp_processor_id()) (can be SP)
* @sym: The name of the per-cpu variable
@@ -226,11 +241,7 @@ lr .req x30 // link register
.macro adr_this_cpu, dst, sym, tmp
adrp \tmp, \sym
add \dst, \tmp, #:lo12:\sym
-alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
- mrs \tmp, tpidr_el1
-alternative_else
- mrs \tmp, tpidr_el2
-alternative_endif
+ this_cpu_offset \tmp
add \dst, \dst, \tmp
.endm
@@ -241,11 +252,7 @@ alternative_endif
*/
.macro ldr_this_cpu dst, sym, tmp
adr_l \dst, \sym
-alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
- mrs \tmp, tpidr_el1
-alternative_else
- mrs \tmp, tpidr_el2
-alternative_endif
+ this_cpu_offset \tmp
ldr \dst, [\dst, \tmp]
.endm
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index c196eec25498..cf9456663289 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -173,20 +173,8 @@ extern char __smccc_workaround_1_smc[__SMCCC_WORKAROUND_1_SMC_SZ];
#else /* __ASSEMBLY__ */
-.macro hyp_adr_this_cpu reg, sym, tmp
- adr_l \reg, \sym
- mrs \tmp, tpidr_el2
- add \reg, \reg, \tmp
-.endm
-
-.macro hyp_ldr_this_cpu reg, sym, tmp
- adr_l \reg, \sym
- mrs \tmp, tpidr_el2
- ldr \reg, [\reg, \tmp]
-.endm
-
.macro get_host_ctxt reg, tmp
- hyp_adr_this_cpu \reg, kvm_host_data, \tmp
+ adr_this_cpu \reg, kvm_host_data, \tmp
add \reg, \reg, #HOST_DATA_CONTEXT
.endm
diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S
index 46b4dab933d0..fba91c2ab410 100644
--- a/arch/arm64/kvm/hyp/hyp-entry.S
+++ b/arch/arm64/kvm/hyp/hyp-entry.S
@@ -132,7 +132,7 @@ alternative_cb_end
str x0, [x2, #VCPU_WORKAROUND_FLAGS]
/* Check that we actually need to perform the call */
- hyp_ldr_this_cpu x0, arm64_ssbd_callback_required, x2
+ ldr_this_cpu x0, arm64_ssbd_callback_required, x2
cbz x0, wa2_end
mov w0, #ARM_SMCCC_ARCH_WORKAROUND_2
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply related [flat|nested] 87+ messages in thread
* Re: [PATCH v3 05/11] kvm: arm64: Remove hyp_adr/ldr_this_cpu
2020-09-16 17:34 ` David Brazdil
(?)
@ 2020-09-18 9:05 ` Will Deacon
-1 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 9:05 UTC (permalink / raw)
To: David Brazdil
Cc: kvmarm, Catalin Marinas, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team,
Andrew Scull
On Wed, Sep 16, 2020 at 06:34:33PM +0100, David Brazdil wrote:
> The hyp_adr/ldr_this_cpu helpers were introduced for use in hyp code
> because they always needed to use TPIDR_EL2 for base, while
> adr/ldr_this_cpu from kernel proper would select between TPIDR_EL2 and
> _EL1 based on VHE/nVHE.
>
> Simplify this now that the hyp mode case can be handled using the
> __KVM_VHE/NVHE_HYPERVISOR__ macros.
>
> Acked-by: Andrew Scull <ascull@google.com>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/assembler.h | 27 +++++++++++++++++----------
> arch/arm64/include/asm/kvm_asm.h | 14 +-------------
> arch/arm64/kvm/hyp/hyp-entry.S | 2 +-
> 3 files changed, 19 insertions(+), 24 deletions(-)
>
> diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
> index 54d181177656..f79231a0f949 100644
> --- a/arch/arm64/include/asm/assembler.h
> +++ b/arch/arm64/include/asm/assembler.h
> @@ -218,6 +218,21 @@ lr .req x30 // link register
> str \src, [\tmp, :lo12:\sym]
> .endm
>
> + /*
> + * @dst: destination register (32 or 64 bit wide)
> + */
> + .macro this_cpu_offset, dst
> +#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
> + mrs \dst, tpidr_el2
> +#else
> +alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
> + mrs \dst, tpidr_el1
> +alternative_else
> + mrs \dst, tpidr_el2
> +alternative_endif
> +#endif
Cosmetic, but I think it would be cleaner just to define two variants of the
macro here:
#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
.macro this_cpu_offset, dst
mrs \dst, tpidr_el2
.endm
#else
.macro this_cpu_offset, dst
alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
mrs \dst, tpidr_el1
alternative_else
mrs \dst, tpidr_el2
alternative_endif
.endm
#endif
(and should we have a shorthand __HYPERVISOR define to avoid the NVHE || VHE
logic?)
With that:
Acked-by: Will Deacon <will@kernel.org>
Will
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3 05/11] kvm: arm64: Remove hyp_adr/ldr_this_cpu
@ 2020-09-18 9:05 ` Will Deacon
0 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 9:05 UTC (permalink / raw)
To: David Brazdil
Cc: Suzuki K Poulose, Catalin Marinas, linux-kernel, James Morse,
linux-arm-kernel, Marc Zyngier, Tejun Heo, Dennis Zhou,
Christoph Lameter, kernel-team, kvmarm, Julien Thierry,
Andrew Scull
On Wed, Sep 16, 2020 at 06:34:33PM +0100, David Brazdil wrote:
> The hyp_adr/ldr_this_cpu helpers were introduced for use in hyp code
> because they always needed to use TPIDR_EL2 for base, while
> adr/ldr_this_cpu from kernel proper would select between TPIDR_EL2 and
> _EL1 based on VHE/nVHE.
>
> Simplify this now that the hyp mode case can be handled using the
> __KVM_VHE/NVHE_HYPERVISOR__ macros.
>
> Acked-by: Andrew Scull <ascull@google.com>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/assembler.h | 27 +++++++++++++++++----------
> arch/arm64/include/asm/kvm_asm.h | 14 +-------------
> arch/arm64/kvm/hyp/hyp-entry.S | 2 +-
> 3 files changed, 19 insertions(+), 24 deletions(-)
>
> diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
> index 54d181177656..f79231a0f949 100644
> --- a/arch/arm64/include/asm/assembler.h
> +++ b/arch/arm64/include/asm/assembler.h
> @@ -218,6 +218,21 @@ lr .req x30 // link register
> str \src, [\tmp, :lo12:\sym]
> .endm
>
> + /*
> + * @dst: destination register (32 or 64 bit wide)
> + */
> + .macro this_cpu_offset, dst
> +#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
> + mrs \dst, tpidr_el2
> +#else
> +alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
> + mrs \dst, tpidr_el1
> +alternative_else
> + mrs \dst, tpidr_el2
> +alternative_endif
> +#endif
Cosmetic, but I think it would be cleaner just to define two variants of the
macro here:
#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
.macro this_cpu_offset, dst
mrs \dst, tpidr_el2
.endm
#else
.macro this_cpu_offset, dst
alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
mrs \dst, tpidr_el1
alternative_else
mrs \dst, tpidr_el2
alternative_endif
.endm
#endif
(and should we have a shorthand __HYPERVISOR define to avoid the NVHE || VHE
logic?)
With that:
Acked-by: Will Deacon <will@kernel.org>
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3 05/11] kvm: arm64: Remove hyp_adr/ldr_this_cpu
@ 2020-09-18 9:05 ` Will Deacon
0 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 9:05 UTC (permalink / raw)
To: David Brazdil
Cc: Catalin Marinas, linux-kernel, linux-arm-kernel, Marc Zyngier,
Tejun Heo, Dennis Zhou, Christoph Lameter, kernel-team, kvmarm
On Wed, Sep 16, 2020 at 06:34:33PM +0100, David Brazdil wrote:
> The hyp_adr/ldr_this_cpu helpers were introduced for use in hyp code
> because they always needed to use TPIDR_EL2 for base, while
> adr/ldr_this_cpu from kernel proper would select between TPIDR_EL2 and
> _EL1 based on VHE/nVHE.
>
> Simplify this now that the hyp mode case can be handled using the
> __KVM_VHE/NVHE_HYPERVISOR__ macros.
>
> Acked-by: Andrew Scull <ascull@google.com>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/assembler.h | 27 +++++++++++++++++----------
> arch/arm64/include/asm/kvm_asm.h | 14 +-------------
> arch/arm64/kvm/hyp/hyp-entry.S | 2 +-
> 3 files changed, 19 insertions(+), 24 deletions(-)
>
> diff --git a/arch/arm64/include/asm/assembler.h b/arch/arm64/include/asm/assembler.h
> index 54d181177656..f79231a0f949 100644
> --- a/arch/arm64/include/asm/assembler.h
> +++ b/arch/arm64/include/asm/assembler.h
> @@ -218,6 +218,21 @@ lr .req x30 // link register
> str \src, [\tmp, :lo12:\sym]
> .endm
>
> + /*
> + * @dst: destination register (32 or 64 bit wide)
> + */
> + .macro this_cpu_offset, dst
> +#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
> + mrs \dst, tpidr_el2
> +#else
> +alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
> + mrs \dst, tpidr_el1
> +alternative_else
> + mrs \dst, tpidr_el2
> +alternative_endif
> +#endif
Cosmetic, but I think it would be cleaner just to define two variants of the
macro here:
#if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
.macro this_cpu_offset, dst
mrs \dst, tpidr_el2
.endm
#else
.macro this_cpu_offset, dst
alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
mrs \dst, tpidr_el1
alternative_else
mrs \dst, tpidr_el2
alternative_endif
.endm
#endif
(and should we have a shorthand __HYPERVISOR define to avoid the NVHE || VHE
logic?)
With that:
Acked-by: Will Deacon <will@kernel.org>
Will
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3 05/11] kvm: arm64: Remove hyp_adr/ldr_this_cpu
2020-09-18 9:05 ` Will Deacon
(?)
@ 2020-09-21 14:53 ` David Brazdil
-1 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-21 14:53 UTC (permalink / raw)
To: Will Deacon
Cc: kvmarm, Catalin Marinas, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team,
Andrew Scull
Hi Will,
> Cosmetic, but I think it would be cleaner just to define two variants of the
> macro here:
>
> #if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
> .macro this_cpu_offset, dst
> mrs \dst, tpidr_el2
> .endm
> #else
> .macro this_cpu_offset, dst
> alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
> mrs \dst, tpidr_el1
> alternative_else
> mrs \dst, tpidr_el2
> alternative_endif
> .endm
> #endif
Sure.
> (and should we have a shorthand __HYPERVISOR define to avoid the NVHE || VHE
> logic?)
Happy to add this but let's agree on the details.
* name: just __HYPERVISOR or __KVM_HYPERVISOR__?
* defined where? I'm wary of defining it in a header file because then sombody
will forget to add it and 'ifdef HYP' will be skipped. So I'd put this as
another '-D__HYPERVISOR' in the build rules. Do you agree?
David
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3 05/11] kvm: arm64: Remove hyp_adr/ldr_this_cpu
@ 2020-09-21 14:53 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-21 14:53 UTC (permalink / raw)
To: Will Deacon
Cc: Suzuki K Poulose, Catalin Marinas, linux-kernel, James Morse,
linux-arm-kernel, Marc Zyngier, Tejun Heo, Dennis Zhou,
Christoph Lameter, kernel-team, kvmarm, Julien Thierry,
Andrew Scull
Hi Will,
> Cosmetic, but I think it would be cleaner just to define two variants of the
> macro here:
>
> #if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
> .macro this_cpu_offset, dst
> mrs \dst, tpidr_el2
> .endm
> #else
> .macro this_cpu_offset, dst
> alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
> mrs \dst, tpidr_el1
> alternative_else
> mrs \dst, tpidr_el2
> alternative_endif
> .endm
> #endif
Sure.
> (and should we have a shorthand __HYPERVISOR define to avoid the NVHE || VHE
> logic?)
Happy to add this but let's agree on the details.
* name: just __HYPERVISOR or __KVM_HYPERVISOR__?
* defined where? I'm wary of defining it in a header file because then sombody
will forget to add it and 'ifdef HYP' will be skipped. So I'd put this as
another '-D__HYPERVISOR' in the build rules. Do you agree?
David
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3 05/11] kvm: arm64: Remove hyp_adr/ldr_this_cpu
@ 2020-09-21 14:53 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-21 14:53 UTC (permalink / raw)
To: Will Deacon
Cc: Catalin Marinas, linux-kernel, linux-arm-kernel, Marc Zyngier,
Tejun Heo, Dennis Zhou, Christoph Lameter, kernel-team, kvmarm
Hi Will,
> Cosmetic, but I think it would be cleaner just to define two variants of the
> macro here:
>
> #if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
> .macro this_cpu_offset, dst
> mrs \dst, tpidr_el2
> .endm
> #else
> .macro this_cpu_offset, dst
> alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
> mrs \dst, tpidr_el1
> alternative_else
> mrs \dst, tpidr_el2
> alternative_endif
> .endm
> #endif
Sure.
> (and should we have a shorthand __HYPERVISOR define to avoid the NVHE || VHE
> logic?)
Happy to add this but let's agree on the details.
* name: just __HYPERVISOR or __KVM_HYPERVISOR__?
* defined where? I'm wary of defining it in a header file because then sombody
will forget to add it and 'ifdef HYP' will be skipped. So I'd put this as
another '-D__HYPERVISOR' in the build rules. Do you agree?
David
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3 05/11] kvm: arm64: Remove hyp_adr/ldr_this_cpu
2020-09-21 14:53 ` David Brazdil
(?)
@ 2020-09-21 17:15 ` Will Deacon
-1 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-21 17:15 UTC (permalink / raw)
To: David Brazdil
Cc: kvmarm, Catalin Marinas, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team,
Andrew Scull
On Mon, Sep 21, 2020 at 03:53:20PM +0100, David Brazdil wrote:
> > Cosmetic, but I think it would be cleaner just to define two variants of the
> > macro here:
> >
> > #if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
> > .macro this_cpu_offset, dst
> > mrs \dst, tpidr_el2
> > .endm
> > #else
> > .macro this_cpu_offset, dst
> > alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
> > mrs \dst, tpidr_el1
> > alternative_else
> > mrs \dst, tpidr_el2
> > alternative_endif
> > .endm
> > #endif
>
> Sure.
Ta.
> > (and should we have a shorthand __HYPERVISOR define to avoid the NVHE || VHE
> > logic?)
>
> Happy to add this but let's agree on the details.
> * name: just __HYPERVISOR or __KVM_HYPERVISOR__?
Hey, I leave the hard decisions to others!
> * defined where? I'm wary of defining it in a header file because then sombody
> will forget to add it and 'ifdef HYP' will be skipped. So I'd put this as
> another '-D__HYPERVISOR' in the build rules. Do you agree?
Hmm, that's a good point. Leave that part as-is for now then and we can
tackle it separately if it comes up again.
Will
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3 05/11] kvm: arm64: Remove hyp_adr/ldr_this_cpu
@ 2020-09-21 17:15 ` Will Deacon
0 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-21 17:15 UTC (permalink / raw)
To: David Brazdil
Cc: Suzuki K Poulose, Catalin Marinas, linux-kernel, James Morse,
linux-arm-kernel, Marc Zyngier, Tejun Heo, Dennis Zhou,
Christoph Lameter, kernel-team, kvmarm, Julien Thierry,
Andrew Scull
On Mon, Sep 21, 2020 at 03:53:20PM +0100, David Brazdil wrote:
> > Cosmetic, but I think it would be cleaner just to define two variants of the
> > macro here:
> >
> > #if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
> > .macro this_cpu_offset, dst
> > mrs \dst, tpidr_el2
> > .endm
> > #else
> > .macro this_cpu_offset, dst
> > alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
> > mrs \dst, tpidr_el1
> > alternative_else
> > mrs \dst, tpidr_el2
> > alternative_endif
> > .endm
> > #endif
>
> Sure.
Ta.
> > (and should we have a shorthand __HYPERVISOR define to avoid the NVHE || VHE
> > logic?)
>
> Happy to add this but let's agree on the details.
> * name: just __HYPERVISOR or __KVM_HYPERVISOR__?
Hey, I leave the hard decisions to others!
> * defined where? I'm wary of defining it in a header file because then sombody
> will forget to add it and 'ifdef HYP' will be skipped. So I'd put this as
> another '-D__HYPERVISOR' in the build rules. Do you agree?
Hmm, that's a good point. Leave that part as-is for now then and we can
tackle it separately if it comes up again.
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3 05/11] kvm: arm64: Remove hyp_adr/ldr_this_cpu
@ 2020-09-21 17:15 ` Will Deacon
0 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-21 17:15 UTC (permalink / raw)
To: David Brazdil
Cc: Catalin Marinas, linux-kernel, linux-arm-kernel, Marc Zyngier,
Tejun Heo, Dennis Zhou, Christoph Lameter, kernel-team, kvmarm
On Mon, Sep 21, 2020 at 03:53:20PM +0100, David Brazdil wrote:
> > Cosmetic, but I think it would be cleaner just to define two variants of the
> > macro here:
> >
> > #if defined(__KVM_NVHE_HYPERVISOR__) || defined(__KVM_VHE_HYPERVISOR__)
> > .macro this_cpu_offset, dst
> > mrs \dst, tpidr_el2
> > .endm
> > #else
> > .macro this_cpu_offset, dst
> > alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
> > mrs \dst, tpidr_el1
> > alternative_else
> > mrs \dst, tpidr_el2
> > alternative_endif
> > .endm
> > #endif
>
> Sure.
Ta.
> > (and should we have a shorthand __HYPERVISOR define to avoid the NVHE || VHE
> > logic?)
>
> Happy to add this but let's agree on the details.
> * name: just __HYPERVISOR or __KVM_HYPERVISOR__?
Hey, I leave the hard decisions to others!
> * defined where? I'm wary of defining it in a header file because then sombody
> will forget to add it and 'ifdef HYP' will be skipped. So I'd put this as
> another '-D__HYPERVISOR' in the build rules. Do you agree?
Hmm, that's a good point. Leave that part as-is for now then and we can
tackle it separately if it comes up again.
Will
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v3 06/11] kvm: arm64: Add helpers for accessing nVHE hyp per-cpu vars
2020-09-16 17:34 ` David Brazdil
(?)
@ 2020-09-16 17:34 ` David Brazdil
-1 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: Catalin Marinas, Will Deacon, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team,
David Brazdil
Defining a per-CPU variable in hyp/nvhe will result in its name being
prefixed with __kvm_nvhe_. Add helpers for declaring these variables
in kernel proper and accessing them with this_cpu_ptr and per_cpu_ptr.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/kvm_asm.h | 25 +++++++++++++++++++++++--
1 file changed, 23 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index cf9456663289..abc03f386b40 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -54,9 +54,21 @@
DECLARE_KVM_VHE_SYM(sym); \
DECLARE_KVM_NVHE_SYM(sym)
+#define DECLARE_KVM_VHE_PER_CPU(type, sym) \
+ DECLARE_PER_CPU(type, sym)
+#define DECLARE_KVM_NVHE_PER_CPU(type, sym) \
+ DECLARE_PER_CPU(type, kvm_nvhe_sym(sym))
+
+#define DECLARE_KVM_HYP_PER_CPU(type, sym) \
+ DECLARE_KVM_VHE_PER_CPU(type, sym); \
+ DECLARE_KVM_NVHE_PER_CPU(type, sym)
+
#define CHOOSE_VHE_SYM(sym) sym
#define CHOOSE_NVHE_SYM(sym) kvm_nvhe_sym(sym)
+#define this_cpu_ptr_nvhe(sym) this_cpu_ptr(&kvm_nvhe_sym(sym))
+#define per_cpu_ptr_nvhe(sym, cpu) per_cpu_ptr(&kvm_nvhe_sym(sym), cpu)
+
#ifndef __KVM_NVHE_HYPERVISOR__
/*
* BIG FAT WARNINGS:
@@ -69,12 +81,21 @@
* - Don't let the nVHE hypervisor have access to this, as it will
* pick the *wrong* symbol (yes, it runs at EL2...).
*/
-#define CHOOSE_HYP_SYM(sym) (is_kernel_in_hyp_mode() ? CHOOSE_VHE_SYM(sym) \
+#define CHOOSE_HYP_SYM(sym) (is_kernel_in_hyp_mode() \
+ ? CHOOSE_VHE_SYM(sym) \
: CHOOSE_NVHE_SYM(sym))
+#define this_cpu_ptr_hyp(sym) (is_kernel_in_hyp_mode() \
+ ? this_cpu_ptr(&sym) \
+ : this_cpu_ptr_nvhe(sym))
+#define per_cpu_ptr_hyp(sym, cpu) (is_kernel_in_hyp_mode() \
+ ? per_cpu_ptr(&sym, cpu) \
+ : per_cpu_ptr_nvhe(sym, cpu))
#else
/* The nVHE hypervisor shouldn't even try to access anything */
extern void *__nvhe_undefined_symbol;
-#define CHOOSE_HYP_SYM(sym) __nvhe_undefined_symbol
+#define CHOOSE_HYP_SYM(sym) __nvhe_undefined_symbol
+#define this_cpu_ptr_hyp(sym) (&__nvhe_undefined_symbol)
+#define per_cpu_ptr_hyp(sym, cpu) (&__nvhe_undefined_symbol)
#endif
/* Translate a kernel address @ptr into its equivalent linear mapping */
--
2.28.0.618.gf4bc123cb7-goog
^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH v3 06/11] kvm: arm64: Add helpers for accessing nVHE hyp per-cpu vars
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Suzuki K Poulose, Catalin Marinas, linux-kernel,
James Morse, linux-arm-kernel, Marc Zyngier, Tejun Heo,
Dennis Zhou, Christoph Lameter, David Brazdil, Will Deacon,
Julien Thierry
Defining a per-CPU variable in hyp/nvhe will result in its name being
prefixed with __kvm_nvhe_. Add helpers for declaring these variables
in kernel proper and accessing them with this_cpu_ptr and per_cpu_ptr.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/kvm_asm.h | 25 +++++++++++++++++++++++--
1 file changed, 23 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index cf9456663289..abc03f386b40 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -54,9 +54,21 @@
DECLARE_KVM_VHE_SYM(sym); \
DECLARE_KVM_NVHE_SYM(sym)
+#define DECLARE_KVM_VHE_PER_CPU(type, sym) \
+ DECLARE_PER_CPU(type, sym)
+#define DECLARE_KVM_NVHE_PER_CPU(type, sym) \
+ DECLARE_PER_CPU(type, kvm_nvhe_sym(sym))
+
+#define DECLARE_KVM_HYP_PER_CPU(type, sym) \
+ DECLARE_KVM_VHE_PER_CPU(type, sym); \
+ DECLARE_KVM_NVHE_PER_CPU(type, sym)
+
#define CHOOSE_VHE_SYM(sym) sym
#define CHOOSE_NVHE_SYM(sym) kvm_nvhe_sym(sym)
+#define this_cpu_ptr_nvhe(sym) this_cpu_ptr(&kvm_nvhe_sym(sym))
+#define per_cpu_ptr_nvhe(sym, cpu) per_cpu_ptr(&kvm_nvhe_sym(sym), cpu)
+
#ifndef __KVM_NVHE_HYPERVISOR__
/*
* BIG FAT WARNINGS:
@@ -69,12 +81,21 @@
* - Don't let the nVHE hypervisor have access to this, as it will
* pick the *wrong* symbol (yes, it runs at EL2...).
*/
-#define CHOOSE_HYP_SYM(sym) (is_kernel_in_hyp_mode() ? CHOOSE_VHE_SYM(sym) \
+#define CHOOSE_HYP_SYM(sym) (is_kernel_in_hyp_mode() \
+ ? CHOOSE_VHE_SYM(sym) \
: CHOOSE_NVHE_SYM(sym))
+#define this_cpu_ptr_hyp(sym) (is_kernel_in_hyp_mode() \
+ ? this_cpu_ptr(&sym) \
+ : this_cpu_ptr_nvhe(sym))
+#define per_cpu_ptr_hyp(sym, cpu) (is_kernel_in_hyp_mode() \
+ ? per_cpu_ptr(&sym, cpu) \
+ : per_cpu_ptr_nvhe(sym, cpu))
#else
/* The nVHE hypervisor shouldn't even try to access anything */
extern void *__nvhe_undefined_symbol;
-#define CHOOSE_HYP_SYM(sym) __nvhe_undefined_symbol
+#define CHOOSE_HYP_SYM(sym) __nvhe_undefined_symbol
+#define this_cpu_ptr_hyp(sym) (&__nvhe_undefined_symbol)
+#define per_cpu_ptr_hyp(sym, cpu) (&__nvhe_undefined_symbol)
#endif
/* Translate a kernel address @ptr into its equivalent linear mapping */
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 87+ messages in thread
* [PATCH v3 06/11] kvm: arm64: Add helpers for accessing nVHE hyp per-cpu vars
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Catalin Marinas, linux-kernel, linux-arm-kernel,
Marc Zyngier, Tejun Heo, Dennis Zhou, Christoph Lameter,
Will Deacon
Defining a per-CPU variable in hyp/nvhe will result in its name being
prefixed with __kvm_nvhe_. Add helpers for declaring these variables
in kernel proper and accessing them with this_cpu_ptr and per_cpu_ptr.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/kvm_asm.h | 25 +++++++++++++++++++++++--
1 file changed, 23 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index cf9456663289..abc03f386b40 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -54,9 +54,21 @@
DECLARE_KVM_VHE_SYM(sym); \
DECLARE_KVM_NVHE_SYM(sym)
+#define DECLARE_KVM_VHE_PER_CPU(type, sym) \
+ DECLARE_PER_CPU(type, sym)
+#define DECLARE_KVM_NVHE_PER_CPU(type, sym) \
+ DECLARE_PER_CPU(type, kvm_nvhe_sym(sym))
+
+#define DECLARE_KVM_HYP_PER_CPU(type, sym) \
+ DECLARE_KVM_VHE_PER_CPU(type, sym); \
+ DECLARE_KVM_NVHE_PER_CPU(type, sym)
+
#define CHOOSE_VHE_SYM(sym) sym
#define CHOOSE_NVHE_SYM(sym) kvm_nvhe_sym(sym)
+#define this_cpu_ptr_nvhe(sym) this_cpu_ptr(&kvm_nvhe_sym(sym))
+#define per_cpu_ptr_nvhe(sym, cpu) per_cpu_ptr(&kvm_nvhe_sym(sym), cpu)
+
#ifndef __KVM_NVHE_HYPERVISOR__
/*
* BIG FAT WARNINGS:
@@ -69,12 +81,21 @@
* - Don't let the nVHE hypervisor have access to this, as it will
* pick the *wrong* symbol (yes, it runs at EL2...).
*/
-#define CHOOSE_HYP_SYM(sym) (is_kernel_in_hyp_mode() ? CHOOSE_VHE_SYM(sym) \
+#define CHOOSE_HYP_SYM(sym) (is_kernel_in_hyp_mode() \
+ ? CHOOSE_VHE_SYM(sym) \
: CHOOSE_NVHE_SYM(sym))
+#define this_cpu_ptr_hyp(sym) (is_kernel_in_hyp_mode() \
+ ? this_cpu_ptr(&sym) \
+ : this_cpu_ptr_nvhe(sym))
+#define per_cpu_ptr_hyp(sym, cpu) (is_kernel_in_hyp_mode() \
+ ? per_cpu_ptr(&sym, cpu) \
+ : per_cpu_ptr_nvhe(sym, cpu))
#else
/* The nVHE hypervisor shouldn't even try to access anything */
extern void *__nvhe_undefined_symbol;
-#define CHOOSE_HYP_SYM(sym) __nvhe_undefined_symbol
+#define CHOOSE_HYP_SYM(sym) __nvhe_undefined_symbol
+#define this_cpu_ptr_hyp(sym) (&__nvhe_undefined_symbol)
+#define per_cpu_ptr_hyp(sym, cpu) (&__nvhe_undefined_symbol)
#endif
/* Translate a kernel address @ptr into its equivalent linear mapping */
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply related [flat|nested] 87+ messages in thread
* Re: [PATCH v3 06/11] kvm: arm64: Add helpers for accessing nVHE hyp per-cpu vars
2020-09-16 17:34 ` David Brazdil
(?)
@ 2020-09-18 9:24 ` Will Deacon
-1 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 9:24 UTC (permalink / raw)
To: David Brazdil
Cc: kvmarm, Catalin Marinas, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team
On Wed, Sep 16, 2020 at 06:34:34PM +0100, David Brazdil wrote:
> Defining a per-CPU variable in hyp/nvhe will result in its name being
> prefixed with __kvm_nvhe_. Add helpers for declaring these variables
> in kernel proper and accessing them with this_cpu_ptr and per_cpu_ptr.
>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/kvm_asm.h | 25 +++++++++++++++++++++++--
> 1 file changed, 23 insertions(+), 2 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
> index cf9456663289..abc03f386b40 100644
> --- a/arch/arm64/include/asm/kvm_asm.h
> +++ b/arch/arm64/include/asm/kvm_asm.h
> @@ -54,9 +54,21 @@
> DECLARE_KVM_VHE_SYM(sym); \
> DECLARE_KVM_NVHE_SYM(sym)
>
> +#define DECLARE_KVM_VHE_PER_CPU(type, sym) \
> + DECLARE_PER_CPU(type, sym)
> +#define DECLARE_KVM_NVHE_PER_CPU(type, sym) \
> + DECLARE_PER_CPU(type, kvm_nvhe_sym(sym))
> +
> +#define DECLARE_KVM_HYP_PER_CPU(type, sym) \
> + DECLARE_KVM_VHE_PER_CPU(type, sym); \
> + DECLARE_KVM_NVHE_PER_CPU(type, sym)
> +
> #define CHOOSE_VHE_SYM(sym) sym
> #define CHOOSE_NVHE_SYM(sym) kvm_nvhe_sym(sym)
>
> +#define this_cpu_ptr_nvhe(sym) this_cpu_ptr(&kvm_nvhe_sym(sym))
> +#define per_cpu_ptr_nvhe(sym, cpu) per_cpu_ptr(&kvm_nvhe_sym(sym), cpu)
nit: I'd probably stick a _sym suffix on these macros, to make it clear
that they're just munging the symbol name rather than doing some completely
different pcpu implementation.
THat said, do you expect these to be used outside of the pcpu
implementation? If not, I suggest some underscores as a prefix as well.
> #ifndef __KVM_NVHE_HYPERVISOR__
> /*
> * BIG FAT WARNINGS:
> @@ -69,12 +81,21 @@
> * - Don't let the nVHE hypervisor have access to this, as it will
> * pick the *wrong* symbol (yes, it runs at EL2...).
> */
> -#define CHOOSE_HYP_SYM(sym) (is_kernel_in_hyp_mode() ? CHOOSE_VHE_SYM(sym) \
> +#define CHOOSE_HYP_SYM(sym) (is_kernel_in_hyp_mode() \
> + ? CHOOSE_VHE_SYM(sym) \
> : CHOOSE_NVHE_SYM(sym))
> +#define this_cpu_ptr_hyp(sym) (is_kernel_in_hyp_mode() \
> + ? this_cpu_ptr(&sym) \
> + : this_cpu_ptr_nvhe(sym))
> +#define per_cpu_ptr_hyp(sym, cpu) (is_kernel_in_hyp_mode() \
> + ? per_cpu_ptr(&sym, cpu) \
> + : per_cpu_ptr_nvhe(sym, cpu))
is_kernel_in_hyp_mode() reads a system register to determine the current
exception level, so this doesn't seem like something we should be doing
everytime here. Perhaps is_kernel_in_hyp_mode() should avoid read_sysreg()
and instead use a non-volatile asm to allow the result to be cached by
the compiler. Hmm.
But I think that can be tackled as a future patch, so with the naming nits
resolved:
Acked-by: Will Deacon <will@kernel.org>
Will
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3 06/11] kvm: arm64: Add helpers for accessing nVHE hyp per-cpu vars
@ 2020-09-18 9:24 ` Will Deacon
0 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 9:24 UTC (permalink / raw)
To: David Brazdil
Cc: Suzuki K Poulose, Catalin Marinas, linux-kernel, James Morse,
linux-arm-kernel, Marc Zyngier, Tejun Heo, Dennis Zhou,
Christoph Lameter, kernel-team, kvmarm, Julien Thierry
On Wed, Sep 16, 2020 at 06:34:34PM +0100, David Brazdil wrote:
> Defining a per-CPU variable in hyp/nvhe will result in its name being
> prefixed with __kvm_nvhe_. Add helpers for declaring these variables
> in kernel proper and accessing them with this_cpu_ptr and per_cpu_ptr.
>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/kvm_asm.h | 25 +++++++++++++++++++++++--
> 1 file changed, 23 insertions(+), 2 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
> index cf9456663289..abc03f386b40 100644
> --- a/arch/arm64/include/asm/kvm_asm.h
> +++ b/arch/arm64/include/asm/kvm_asm.h
> @@ -54,9 +54,21 @@
> DECLARE_KVM_VHE_SYM(sym); \
> DECLARE_KVM_NVHE_SYM(sym)
>
> +#define DECLARE_KVM_VHE_PER_CPU(type, sym) \
> + DECLARE_PER_CPU(type, sym)
> +#define DECLARE_KVM_NVHE_PER_CPU(type, sym) \
> + DECLARE_PER_CPU(type, kvm_nvhe_sym(sym))
> +
> +#define DECLARE_KVM_HYP_PER_CPU(type, sym) \
> + DECLARE_KVM_VHE_PER_CPU(type, sym); \
> + DECLARE_KVM_NVHE_PER_CPU(type, sym)
> +
> #define CHOOSE_VHE_SYM(sym) sym
> #define CHOOSE_NVHE_SYM(sym) kvm_nvhe_sym(sym)
>
> +#define this_cpu_ptr_nvhe(sym) this_cpu_ptr(&kvm_nvhe_sym(sym))
> +#define per_cpu_ptr_nvhe(sym, cpu) per_cpu_ptr(&kvm_nvhe_sym(sym), cpu)
nit: I'd probably stick a _sym suffix on these macros, to make it clear
that they're just munging the symbol name rather than doing some completely
different pcpu implementation.
THat said, do you expect these to be used outside of the pcpu
implementation? If not, I suggest some underscores as a prefix as well.
> #ifndef __KVM_NVHE_HYPERVISOR__
> /*
> * BIG FAT WARNINGS:
> @@ -69,12 +81,21 @@
> * - Don't let the nVHE hypervisor have access to this, as it will
> * pick the *wrong* symbol (yes, it runs at EL2...).
> */
> -#define CHOOSE_HYP_SYM(sym) (is_kernel_in_hyp_mode() ? CHOOSE_VHE_SYM(sym) \
> +#define CHOOSE_HYP_SYM(sym) (is_kernel_in_hyp_mode() \
> + ? CHOOSE_VHE_SYM(sym) \
> : CHOOSE_NVHE_SYM(sym))
> +#define this_cpu_ptr_hyp(sym) (is_kernel_in_hyp_mode() \
> + ? this_cpu_ptr(&sym) \
> + : this_cpu_ptr_nvhe(sym))
> +#define per_cpu_ptr_hyp(sym, cpu) (is_kernel_in_hyp_mode() \
> + ? per_cpu_ptr(&sym, cpu) \
> + : per_cpu_ptr_nvhe(sym, cpu))
is_kernel_in_hyp_mode() reads a system register to determine the current
exception level, so this doesn't seem like something we should be doing
everytime here. Perhaps is_kernel_in_hyp_mode() should avoid read_sysreg()
and instead use a non-volatile asm to allow the result to be cached by
the compiler. Hmm.
But I think that can be tackled as a future patch, so with the naming nits
resolved:
Acked-by: Will Deacon <will@kernel.org>
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3 06/11] kvm: arm64: Add helpers for accessing nVHE hyp per-cpu vars
@ 2020-09-18 9:24 ` Will Deacon
0 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 9:24 UTC (permalink / raw)
To: David Brazdil
Cc: Catalin Marinas, linux-kernel, linux-arm-kernel, Marc Zyngier,
Tejun Heo, Dennis Zhou, Christoph Lameter, kernel-team, kvmarm
On Wed, Sep 16, 2020 at 06:34:34PM +0100, David Brazdil wrote:
> Defining a per-CPU variable in hyp/nvhe will result in its name being
> prefixed with __kvm_nvhe_. Add helpers for declaring these variables
> in kernel proper and accessing them with this_cpu_ptr and per_cpu_ptr.
>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/kvm_asm.h | 25 +++++++++++++++++++++++--
> 1 file changed, 23 insertions(+), 2 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
> index cf9456663289..abc03f386b40 100644
> --- a/arch/arm64/include/asm/kvm_asm.h
> +++ b/arch/arm64/include/asm/kvm_asm.h
> @@ -54,9 +54,21 @@
> DECLARE_KVM_VHE_SYM(sym); \
> DECLARE_KVM_NVHE_SYM(sym)
>
> +#define DECLARE_KVM_VHE_PER_CPU(type, sym) \
> + DECLARE_PER_CPU(type, sym)
> +#define DECLARE_KVM_NVHE_PER_CPU(type, sym) \
> + DECLARE_PER_CPU(type, kvm_nvhe_sym(sym))
> +
> +#define DECLARE_KVM_HYP_PER_CPU(type, sym) \
> + DECLARE_KVM_VHE_PER_CPU(type, sym); \
> + DECLARE_KVM_NVHE_PER_CPU(type, sym)
> +
> #define CHOOSE_VHE_SYM(sym) sym
> #define CHOOSE_NVHE_SYM(sym) kvm_nvhe_sym(sym)
>
> +#define this_cpu_ptr_nvhe(sym) this_cpu_ptr(&kvm_nvhe_sym(sym))
> +#define per_cpu_ptr_nvhe(sym, cpu) per_cpu_ptr(&kvm_nvhe_sym(sym), cpu)
nit: I'd probably stick a _sym suffix on these macros, to make it clear
that they're just munging the symbol name rather than doing some completely
different pcpu implementation.
THat said, do you expect these to be used outside of the pcpu
implementation? If not, I suggest some underscores as a prefix as well.
> #ifndef __KVM_NVHE_HYPERVISOR__
> /*
> * BIG FAT WARNINGS:
> @@ -69,12 +81,21 @@
> * - Don't let the nVHE hypervisor have access to this, as it will
> * pick the *wrong* symbol (yes, it runs at EL2...).
> */
> -#define CHOOSE_HYP_SYM(sym) (is_kernel_in_hyp_mode() ? CHOOSE_VHE_SYM(sym) \
> +#define CHOOSE_HYP_SYM(sym) (is_kernel_in_hyp_mode() \
> + ? CHOOSE_VHE_SYM(sym) \
> : CHOOSE_NVHE_SYM(sym))
> +#define this_cpu_ptr_hyp(sym) (is_kernel_in_hyp_mode() \
> + ? this_cpu_ptr(&sym) \
> + : this_cpu_ptr_nvhe(sym))
> +#define per_cpu_ptr_hyp(sym, cpu) (is_kernel_in_hyp_mode() \
> + ? per_cpu_ptr(&sym, cpu) \
> + : per_cpu_ptr_nvhe(sym, cpu))
is_kernel_in_hyp_mode() reads a system register to determine the current
exception level, so this doesn't seem like something we should be doing
everytime here. Perhaps is_kernel_in_hyp_mode() should avoid read_sysreg()
and instead use a non-volatile asm to allow the result to be cached by
the compiler. Hmm.
But I think that can be tackled as a future patch, so with the naming nits
resolved:
Acked-by: Will Deacon <will@kernel.org>
Will
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v3 07/11] kvm: arm64: Duplicate arm64_ssbd_callback_required for nVHE hyp
2020-09-16 17:34 ` David Brazdil
(?)
@ 2020-09-16 17:34 ` David Brazdil
-1 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: Catalin Marinas, Will Deacon, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team,
David Brazdil
Hyp keeps track of which cores require SSBD callback by accessing a
kernel-proper global variable. Create an nVHE symbol of the same name
and copy the value from kernel proper to nVHE at KVM init time.
Done in preparation for separating percpu memory owned by kernel
proper and nVHE.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/kvm_mmu.h | 10 +++++++---
arch/arm64/kernel/image-vars.h | 1 -
arch/arm64/kvm/arm.c | 2 +-
arch/arm64/kvm/hyp/nvhe/switch.c | 3 +++
4 files changed, 11 insertions(+), 5 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 189839c3706a..9db93da35606 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -529,23 +529,27 @@ static inline int kvm_map_vectors(void)
#ifdef CONFIG_ARM64_SSBD
DECLARE_PER_CPU_READ_MOSTLY(u64, arm64_ssbd_callback_required);
+DECLARE_KVM_NVHE_PER_CPU(u64, arm64_ssbd_callback_required);
-static inline int hyp_map_aux_data(void)
+static inline int hyp_init_aux_data(void)
{
int cpu, err;
for_each_possible_cpu(cpu) {
u64 *ptr;
- ptr = per_cpu_ptr(&arm64_ssbd_callback_required, cpu);
+ ptr = per_cpu_ptr_nvhe(arm64_ssbd_callback_required, cpu);
err = create_hyp_mappings(ptr, ptr + 1, PAGE_HYP);
if (err)
return err;
+
+ /* Copy value from kernel to hyp. */
+ *ptr = per_cpu(arm64_ssbd_callback_required, cpu);
}
return 0;
}
#else
-static inline int hyp_map_aux_data(void)
+static inline int hyp_init_aux_data(void)
{
return 0;
}
diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h
index 76da2ad1010c..59d12a0b4622 100644
--- a/arch/arm64/kernel/image-vars.h
+++ b/arch/arm64/kernel/image-vars.h
@@ -67,7 +67,6 @@ KVM_NVHE_ALIAS(kvm_patch_vector_branch);
KVM_NVHE_ALIAS(kvm_update_va_mask);
/* Global kernel state accessed by nVHE hyp code. */
-KVM_NVHE_ALIAS(arm64_ssbd_callback_required);
KVM_NVHE_ALIAS(kvm_host_data);
KVM_NVHE_ALIAS(kvm_vgic_global_state);
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index b588c3b5c2f0..3bdc2661d276 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -1552,7 +1552,7 @@ static int init_hyp_mode(void)
}
}
- err = hyp_map_aux_data();
+ err = hyp_init_aux_data();
if (err)
kvm_err("Cannot map host auxiliary data: %d\n", err);
diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c
index cc4f8e790fb3..4662df6330d7 100644
--- a/arch/arm64/kvm/hyp/nvhe/switch.c
+++ b/arch/arm64/kvm/hyp/nvhe/switch.c
@@ -27,6 +27,9 @@
#include <asm/processor.h>
#include <asm/thread_info.h>
+/* Non-VHE copy of the kernel symbol. */
+DEFINE_PER_CPU_READ_MOSTLY(u64, arm64_ssbd_callback_required);
+
static void __activate_traps(struct kvm_vcpu *vcpu)
{
u64 val;
--
2.28.0.618.gf4bc123cb7-goog
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v3 07/11] kvm: arm64: Duplicate arm64_ssbd_callback_required for nVHE hyp
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Suzuki K Poulose, Catalin Marinas, linux-kernel,
James Morse, linux-arm-kernel, Marc Zyngier, Tejun Heo,
Dennis Zhou, Christoph Lameter, David Brazdil, Will Deacon,
Julien Thierry
Hyp keeps track of which cores require SSBD callback by accessing a
kernel-proper global variable. Create an nVHE symbol of the same name
and copy the value from kernel proper to nVHE at KVM init time.
Done in preparation for separating percpu memory owned by kernel
proper and nVHE.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/kvm_mmu.h | 10 +++++++---
arch/arm64/kernel/image-vars.h | 1 -
arch/arm64/kvm/arm.c | 2 +-
arch/arm64/kvm/hyp/nvhe/switch.c | 3 +++
4 files changed, 11 insertions(+), 5 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 189839c3706a..9db93da35606 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -529,23 +529,27 @@ static inline int kvm_map_vectors(void)
#ifdef CONFIG_ARM64_SSBD
DECLARE_PER_CPU_READ_MOSTLY(u64, arm64_ssbd_callback_required);
+DECLARE_KVM_NVHE_PER_CPU(u64, arm64_ssbd_callback_required);
-static inline int hyp_map_aux_data(void)
+static inline int hyp_init_aux_data(void)
{
int cpu, err;
for_each_possible_cpu(cpu) {
u64 *ptr;
- ptr = per_cpu_ptr(&arm64_ssbd_callback_required, cpu);
+ ptr = per_cpu_ptr_nvhe(arm64_ssbd_callback_required, cpu);
err = create_hyp_mappings(ptr, ptr + 1, PAGE_HYP);
if (err)
return err;
+
+ /* Copy value from kernel to hyp. */
+ *ptr = per_cpu(arm64_ssbd_callback_required, cpu);
}
return 0;
}
#else
-static inline int hyp_map_aux_data(void)
+static inline int hyp_init_aux_data(void)
{
return 0;
}
diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h
index 76da2ad1010c..59d12a0b4622 100644
--- a/arch/arm64/kernel/image-vars.h
+++ b/arch/arm64/kernel/image-vars.h
@@ -67,7 +67,6 @@ KVM_NVHE_ALIAS(kvm_patch_vector_branch);
KVM_NVHE_ALIAS(kvm_update_va_mask);
/* Global kernel state accessed by nVHE hyp code. */
-KVM_NVHE_ALIAS(arm64_ssbd_callback_required);
KVM_NVHE_ALIAS(kvm_host_data);
KVM_NVHE_ALIAS(kvm_vgic_global_state);
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index b588c3b5c2f0..3bdc2661d276 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -1552,7 +1552,7 @@ static int init_hyp_mode(void)
}
}
- err = hyp_map_aux_data();
+ err = hyp_init_aux_data();
if (err)
kvm_err("Cannot map host auxiliary data: %d\n", err);
diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c
index cc4f8e790fb3..4662df6330d7 100644
--- a/arch/arm64/kvm/hyp/nvhe/switch.c
+++ b/arch/arm64/kvm/hyp/nvhe/switch.c
@@ -27,6 +27,9 @@
#include <asm/processor.h>
#include <asm/thread_info.h>
+/* Non-VHE copy of the kernel symbol. */
+DEFINE_PER_CPU_READ_MOSTLY(u64, arm64_ssbd_callback_required);
+
static void __activate_traps(struct kvm_vcpu *vcpu)
{
u64 val;
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v3 07/11] kvm: arm64: Duplicate arm64_ssbd_callback_required for nVHE hyp
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Catalin Marinas, linux-kernel, linux-arm-kernel,
Marc Zyngier, Tejun Heo, Dennis Zhou, Christoph Lameter,
Will Deacon
Hyp keeps track of which cores require SSBD callback by accessing a
kernel-proper global variable. Create an nVHE symbol of the same name
and copy the value from kernel proper to nVHE at KVM init time.
Done in preparation for separating percpu memory owned by kernel
proper and nVHE.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/kvm_mmu.h | 10 +++++++---
arch/arm64/kernel/image-vars.h | 1 -
arch/arm64/kvm/arm.c | 2 +-
arch/arm64/kvm/hyp/nvhe/switch.c | 3 +++
4 files changed, 11 insertions(+), 5 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 189839c3706a..9db93da35606 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -529,23 +529,27 @@ static inline int kvm_map_vectors(void)
#ifdef CONFIG_ARM64_SSBD
DECLARE_PER_CPU_READ_MOSTLY(u64, arm64_ssbd_callback_required);
+DECLARE_KVM_NVHE_PER_CPU(u64, arm64_ssbd_callback_required);
-static inline int hyp_map_aux_data(void)
+static inline int hyp_init_aux_data(void)
{
int cpu, err;
for_each_possible_cpu(cpu) {
u64 *ptr;
- ptr = per_cpu_ptr(&arm64_ssbd_callback_required, cpu);
+ ptr = per_cpu_ptr_nvhe(arm64_ssbd_callback_required, cpu);
err = create_hyp_mappings(ptr, ptr + 1, PAGE_HYP);
if (err)
return err;
+
+ /* Copy value from kernel to hyp. */
+ *ptr = per_cpu(arm64_ssbd_callback_required, cpu);
}
return 0;
}
#else
-static inline int hyp_map_aux_data(void)
+static inline int hyp_init_aux_data(void)
{
return 0;
}
diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h
index 76da2ad1010c..59d12a0b4622 100644
--- a/arch/arm64/kernel/image-vars.h
+++ b/arch/arm64/kernel/image-vars.h
@@ -67,7 +67,6 @@ KVM_NVHE_ALIAS(kvm_patch_vector_branch);
KVM_NVHE_ALIAS(kvm_update_va_mask);
/* Global kernel state accessed by nVHE hyp code. */
-KVM_NVHE_ALIAS(arm64_ssbd_callback_required);
KVM_NVHE_ALIAS(kvm_host_data);
KVM_NVHE_ALIAS(kvm_vgic_global_state);
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index b588c3b5c2f0..3bdc2661d276 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -1552,7 +1552,7 @@ static int init_hyp_mode(void)
}
}
- err = hyp_map_aux_data();
+ err = hyp_init_aux_data();
if (err)
kvm_err("Cannot map host auxiliary data: %d\n", err);
diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c
index cc4f8e790fb3..4662df6330d7 100644
--- a/arch/arm64/kvm/hyp/nvhe/switch.c
+++ b/arch/arm64/kvm/hyp/nvhe/switch.c
@@ -27,6 +27,9 @@
#include <asm/processor.h>
#include <asm/thread_info.h>
+/* Non-VHE copy of the kernel symbol. */
+DEFINE_PER_CPU_READ_MOSTLY(u64, arm64_ssbd_callback_required);
+
static void __activate_traps(struct kvm_vcpu *vcpu)
{
u64 val;
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v3 07/11] kvm: arm64: Duplicate arm64_ssbd_callback_required for nVHE hyp
2020-09-16 17:34 ` David Brazdil
(?)
@ 2020-09-18 11:59 ` Will Deacon
-1 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 11:59 UTC (permalink / raw)
To: David Brazdil
Cc: kvmarm, Catalin Marinas, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team
On Wed, Sep 16, 2020 at 06:34:35PM +0100, David Brazdil wrote:
> Hyp keeps track of which cores require SSBD callback by accessing a
> kernel-proper global variable. Create an nVHE symbol of the same name
> and copy the value from kernel proper to nVHE at KVM init time.
>
> Done in preparation for separating percpu memory owned by kernel
> proper and nVHE.
>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/kvm_mmu.h | 10 +++++++---
> arch/arm64/kernel/image-vars.h | 1 -
> arch/arm64/kvm/arm.c | 2 +-
> arch/arm64/kvm/hyp/nvhe/switch.c | 3 +++
> 4 files changed, 11 insertions(+), 5 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
> index 189839c3706a..9db93da35606 100644
> --- a/arch/arm64/include/asm/kvm_mmu.h
> +++ b/arch/arm64/include/asm/kvm_mmu.h
> @@ -529,23 +529,27 @@ static inline int kvm_map_vectors(void)
>
> #ifdef CONFIG_ARM64_SSBD
> DECLARE_PER_CPU_READ_MOSTLY(u64, arm64_ssbd_callback_required);
> +DECLARE_KVM_NVHE_PER_CPU(u64, arm64_ssbd_callback_required);
>
> -static inline int hyp_map_aux_data(void)
> +static inline int hyp_init_aux_data(void)
> {
> int cpu, err;
>
> for_each_possible_cpu(cpu) {
> u64 *ptr;
>
> - ptr = per_cpu_ptr(&arm64_ssbd_callback_required, cpu);
> + ptr = per_cpu_ptr_nvhe(arm64_ssbd_callback_required, cpu);
> err = create_hyp_mappings(ptr, ptr + 1, PAGE_HYP);
> if (err)
> return err;
> +
> + /* Copy value from kernel to hyp. */
> + *ptr = per_cpu(arm64_ssbd_callback_required, cpu);
Hmm. Is this correct for late arriving CPUs, where we don't know whether
a callback is required at the point we do the copy?
That sounds fiddly to resolve, but this _might_ all be moot because I'm
about to post a series that allows us to remove the hyp mapping of this
variable entirely. So leave this for now, but maybe stick a comment in
that it doesn't work for late cpus.
Will
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v3 07/11] kvm: arm64: Duplicate arm64_ssbd_callback_required for nVHE hyp
@ 2020-09-18 11:59 ` Will Deacon
0 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 11:59 UTC (permalink / raw)
To: David Brazdil
Cc: Suzuki K Poulose, Catalin Marinas, linux-kernel, James Morse,
linux-arm-kernel, Marc Zyngier, Tejun Heo, Dennis Zhou,
Christoph Lameter, kernel-team, kvmarm, Julien Thierry
On Wed, Sep 16, 2020 at 06:34:35PM +0100, David Brazdil wrote:
> Hyp keeps track of which cores require SSBD callback by accessing a
> kernel-proper global variable. Create an nVHE symbol of the same name
> and copy the value from kernel proper to nVHE at KVM init time.
>
> Done in preparation for separating percpu memory owned by kernel
> proper and nVHE.
>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/kvm_mmu.h | 10 +++++++---
> arch/arm64/kernel/image-vars.h | 1 -
> arch/arm64/kvm/arm.c | 2 +-
> arch/arm64/kvm/hyp/nvhe/switch.c | 3 +++
> 4 files changed, 11 insertions(+), 5 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
> index 189839c3706a..9db93da35606 100644
> --- a/arch/arm64/include/asm/kvm_mmu.h
> +++ b/arch/arm64/include/asm/kvm_mmu.h
> @@ -529,23 +529,27 @@ static inline int kvm_map_vectors(void)
>
> #ifdef CONFIG_ARM64_SSBD
> DECLARE_PER_CPU_READ_MOSTLY(u64, arm64_ssbd_callback_required);
> +DECLARE_KVM_NVHE_PER_CPU(u64, arm64_ssbd_callback_required);
>
> -static inline int hyp_map_aux_data(void)
> +static inline int hyp_init_aux_data(void)
> {
> int cpu, err;
>
> for_each_possible_cpu(cpu) {
> u64 *ptr;
>
> - ptr = per_cpu_ptr(&arm64_ssbd_callback_required, cpu);
> + ptr = per_cpu_ptr_nvhe(arm64_ssbd_callback_required, cpu);
> err = create_hyp_mappings(ptr, ptr + 1, PAGE_HYP);
> if (err)
> return err;
> +
> + /* Copy value from kernel to hyp. */
> + *ptr = per_cpu(arm64_ssbd_callback_required, cpu);
Hmm. Is this correct for late arriving CPUs, where we don't know whether
a callback is required at the point we do the copy?
That sounds fiddly to resolve, but this _might_ all be moot because I'm
about to post a series that allows us to remove the hyp mapping of this
variable entirely. So leave this for now, but maybe stick a comment in
that it doesn't work for late cpus.
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v3 07/11] kvm: arm64: Duplicate arm64_ssbd_callback_required for nVHE hyp
@ 2020-09-18 11:59 ` Will Deacon
0 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 11:59 UTC (permalink / raw)
To: David Brazdil
Cc: Catalin Marinas, linux-kernel, linux-arm-kernel, Marc Zyngier,
Tejun Heo, Dennis Zhou, Christoph Lameter, kernel-team, kvmarm
On Wed, Sep 16, 2020 at 06:34:35PM +0100, David Brazdil wrote:
> Hyp keeps track of which cores require SSBD callback by accessing a
> kernel-proper global variable. Create an nVHE symbol of the same name
> and copy the value from kernel proper to nVHE at KVM init time.
>
> Done in preparation for separating percpu memory owned by kernel
> proper and nVHE.
>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/kvm_mmu.h | 10 +++++++---
> arch/arm64/kernel/image-vars.h | 1 -
> arch/arm64/kvm/arm.c | 2 +-
> arch/arm64/kvm/hyp/nvhe/switch.c | 3 +++
> 4 files changed, 11 insertions(+), 5 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
> index 189839c3706a..9db93da35606 100644
> --- a/arch/arm64/include/asm/kvm_mmu.h
> +++ b/arch/arm64/include/asm/kvm_mmu.h
> @@ -529,23 +529,27 @@ static inline int kvm_map_vectors(void)
>
> #ifdef CONFIG_ARM64_SSBD
> DECLARE_PER_CPU_READ_MOSTLY(u64, arm64_ssbd_callback_required);
> +DECLARE_KVM_NVHE_PER_CPU(u64, arm64_ssbd_callback_required);
>
> -static inline int hyp_map_aux_data(void)
> +static inline int hyp_init_aux_data(void)
> {
> int cpu, err;
>
> for_each_possible_cpu(cpu) {
> u64 *ptr;
>
> - ptr = per_cpu_ptr(&arm64_ssbd_callback_required, cpu);
> + ptr = per_cpu_ptr_nvhe(arm64_ssbd_callback_required, cpu);
> err = create_hyp_mappings(ptr, ptr + 1, PAGE_HYP);
> if (err)
> return err;
> +
> + /* Copy value from kernel to hyp. */
> + *ptr = per_cpu(arm64_ssbd_callback_required, cpu);
Hmm. Is this correct for late arriving CPUs, where we don't know whether
a callback is required at the point we do the copy?
That sounds fiddly to resolve, but this _might_ all be moot because I'm
about to post a series that allows us to remove the hyp mapping of this
variable entirely. So leave this for now, but maybe stick a comment in
that it doesn't work for late cpus.
Will
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v3 07/11] kvm: arm64: Duplicate arm64_ssbd_callback_required for nVHE hyp
2020-09-18 11:59 ` Will Deacon
(?)
@ 2020-09-22 18:07 ` David Brazdil
-1 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-22 18:07 UTC (permalink / raw)
To: Will Deacon
Cc: kvmarm, Catalin Marinas, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team
> > u64 *ptr;
> >
> > - ptr = per_cpu_ptr(&arm64_ssbd_callback_required, cpu);
> > + ptr = per_cpu_ptr_nvhe(arm64_ssbd_callback_required, cpu);
> > err = create_hyp_mappings(ptr, ptr + 1, PAGE_HYP);
> > if (err)
> > return err;
> > +
> > + /* Copy value from kernel to hyp. */
> > + *ptr = per_cpu(arm64_ssbd_callback_required, cpu);
>
> Hmm. Is this correct for late arriving CPUs, where we don't know whether
> a callback is required at the point we do the copy?
>
> That sounds fiddly to resolve, but this _might_ all be moot because I'm
> about to post a series that allows us to remove the hyp mapping of this
> variable entirely. So leave this for now, but maybe stick a comment in
> that it doesn't work for late cpus.
Ah, good point. I'll move the value copy at the end of cpu_init_hyp_mode().
It must be known at that point. And if your series gets rid of this completely,
even better.
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3 07/11] kvm: arm64: Duplicate arm64_ssbd_callback_required for nVHE hyp
@ 2020-09-22 18:07 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-22 18:07 UTC (permalink / raw)
To: Will Deacon
Cc: Suzuki K Poulose, Catalin Marinas, linux-kernel, James Morse,
linux-arm-kernel, Marc Zyngier, Tejun Heo, Dennis Zhou,
Christoph Lameter, kernel-team, kvmarm, Julien Thierry
> > u64 *ptr;
> >
> > - ptr = per_cpu_ptr(&arm64_ssbd_callback_required, cpu);
> > + ptr = per_cpu_ptr_nvhe(arm64_ssbd_callback_required, cpu);
> > err = create_hyp_mappings(ptr, ptr + 1, PAGE_HYP);
> > if (err)
> > return err;
> > +
> > + /* Copy value from kernel to hyp. */
> > + *ptr = per_cpu(arm64_ssbd_callback_required, cpu);
>
> Hmm. Is this correct for late arriving CPUs, where we don't know whether
> a callback is required at the point we do the copy?
>
> That sounds fiddly to resolve, but this _might_ all be moot because I'm
> about to post a series that allows us to remove the hyp mapping of this
> variable entirely. So leave this for now, but maybe stick a comment in
> that it doesn't work for late cpus.
Ah, good point. I'll move the value copy at the end of cpu_init_hyp_mode().
It must be known at that point. And if your series gets rid of this completely,
even better.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3 07/11] kvm: arm64: Duplicate arm64_ssbd_callback_required for nVHE hyp
@ 2020-09-22 18:07 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-22 18:07 UTC (permalink / raw)
To: Will Deacon
Cc: Catalin Marinas, linux-kernel, linux-arm-kernel, Marc Zyngier,
Tejun Heo, Dennis Zhou, Christoph Lameter, kernel-team, kvmarm
> > u64 *ptr;
> >
> > - ptr = per_cpu_ptr(&arm64_ssbd_callback_required, cpu);
> > + ptr = per_cpu_ptr_nvhe(arm64_ssbd_callback_required, cpu);
> > err = create_hyp_mappings(ptr, ptr + 1, PAGE_HYP);
> > if (err)
> > return err;
> > +
> > + /* Copy value from kernel to hyp. */
> > + *ptr = per_cpu(arm64_ssbd_callback_required, cpu);
>
> Hmm. Is this correct for late arriving CPUs, where we don't know whether
> a callback is required at the point we do the copy?
>
> That sounds fiddly to resolve, but this _might_ all be moot because I'm
> about to post a series that allows us to remove the hyp mapping of this
> variable entirely. So leave this for now, but maybe stick a comment in
> that it doesn't work for late cpus.
Ah, good point. I'll move the value copy at the end of cpu_init_hyp_mode().
It must be known at that point. And if your series gets rid of this completely,
even better.
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v3 08/11] kvm: arm64: Create separate instances of kvm_host_data for VHE/nVHE
2020-09-16 17:34 ` David Brazdil
(?)
@ 2020-09-16 17:34 ` David Brazdil
-1 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: Catalin Marinas, Will Deacon, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team,
David Brazdil
Host CPU context is stored in a global per-cpu variable `kvm_host_data`.
In preparation for introducing independent per-CPU region for nVHE hyp,
create two separate instances of `kvm_host_data`, one for VHE and one
for nVHE.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/kvm_host.h | 2 +-
arch/arm64/kernel/image-vars.h | 1 -
arch/arm64/kvm/arm.c | 5 ++---
arch/arm64/kvm/hyp/nvhe/switch.c | 3 +++
arch/arm64/kvm/hyp/vhe/switch.c | 3 +++
arch/arm64/kvm/pmu.c | 8 ++++----
6 files changed, 13 insertions(+), 9 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 905c2b87e05a..5d8c63f5e97e 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -565,7 +565,7 @@ void kvm_set_sei_esr(struct kvm_vcpu *vcpu, u64 syndrome);
struct kvm_vcpu *kvm_mpidr_to_vcpu(struct kvm *kvm, unsigned long mpidr);
-DECLARE_PER_CPU(kvm_host_data_t, kvm_host_data);
+DECLARE_KVM_HYP_PER_CPU(kvm_host_data_t, kvm_host_data);
static inline void kvm_init_host_cpu_context(struct kvm_cpu_context *cpu_ctxt)
{
diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h
index 59d12a0b4622..80da861b8180 100644
--- a/arch/arm64/kernel/image-vars.h
+++ b/arch/arm64/kernel/image-vars.h
@@ -67,7 +67,6 @@ KVM_NVHE_ALIAS(kvm_patch_vector_branch);
KVM_NVHE_ALIAS(kvm_update_va_mask);
/* Global kernel state accessed by nVHE hyp code. */
-KVM_NVHE_ALIAS(kvm_host_data);
KVM_NVHE_ALIAS(kvm_vgic_global_state);
/* Kernel constant needed to compute idmap addresses. */
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 3bdc2661d276..7af9809fa193 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -46,7 +46,6 @@
__asm__(".arch_extension virt");
#endif
-DEFINE_PER_CPU(kvm_host_data_t, kvm_host_data);
static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page);
/* The VMID used in the VTTBR */
@@ -1308,7 +1307,7 @@ static void cpu_hyp_reset(void)
static void cpu_hyp_reinit(void)
{
- kvm_init_host_cpu_context(&this_cpu_ptr(&kvm_host_data)->host_ctxt);
+ kvm_init_host_cpu_context(&this_cpu_ptr_hyp(kvm_host_data)->host_ctxt);
cpu_hyp_reset();
@@ -1543,7 +1542,7 @@ static int init_hyp_mode(void)
for_each_possible_cpu(cpu) {
kvm_host_data_t *cpu_data;
- cpu_data = per_cpu_ptr(&kvm_host_data, cpu);
+ cpu_data = per_cpu_ptr_hyp(kvm_host_data, cpu);
err = create_hyp_mappings(cpu_data, cpu_data + 1, PAGE_HYP);
if (err) {
diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c
index 4662df6330d7..a7e9b03bd9d1 100644
--- a/arch/arm64/kvm/hyp/nvhe/switch.c
+++ b/arch/arm64/kvm/hyp/nvhe/switch.c
@@ -30,6 +30,9 @@
/* Non-VHE copy of the kernel symbol. */
DEFINE_PER_CPU_READ_MOSTLY(u64, arm64_ssbd_callback_required);
+/* Non-VHE instance of kvm_host_data. */
+DEFINE_PER_CPU(kvm_host_data_t, kvm_host_data);
+
static void __activate_traps(struct kvm_vcpu *vcpu)
{
u64 val;
diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c
index 575e8054f116..0949fc97bf03 100644
--- a/arch/arm64/kvm/hyp/vhe/switch.c
+++ b/arch/arm64/kvm/hyp/vhe/switch.c
@@ -28,6 +28,9 @@
const char __hyp_panic_string[] = "HYP panic:\nPS:%08llx PC:%016llx ESR:%08llx\nFAR:%016llx HPFAR:%016llx PAR:%016llx\nVCPU:%p\n";
+/* VHE instance of kvm_host_data. */
+DEFINE_PER_CPU(kvm_host_data_t, kvm_host_data);
+
static void __activate_traps(struct kvm_vcpu *vcpu)
{
u64 val;
diff --git a/arch/arm64/kvm/pmu.c b/arch/arm64/kvm/pmu.c
index 3c224162b3dd..6d80ffe1ebfc 100644
--- a/arch/arm64/kvm/pmu.c
+++ b/arch/arm64/kvm/pmu.c
@@ -31,7 +31,7 @@ static bool kvm_pmu_switch_needed(struct perf_event_attr *attr)
*/
void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr)
{
- struct kvm_host_data *ctx = this_cpu_ptr(&kvm_host_data);
+ struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data);
if (!kvm_pmu_switch_needed(attr))
return;
@@ -47,7 +47,7 @@ void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr)
*/
void kvm_clr_pmu_events(u32 clr)
{
- struct kvm_host_data *ctx = this_cpu_ptr(&kvm_host_data);
+ struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data);
ctx->pmu_events.events_host &= ~clr;
ctx->pmu_events.events_guest &= ~clr;
@@ -173,7 +173,7 @@ void kvm_vcpu_pmu_restore_guest(struct kvm_vcpu *vcpu)
return;
preempt_disable();
- host = this_cpu_ptr(&kvm_host_data);
+ host = this_cpu_ptr_hyp(kvm_host_data);
events_guest = host->pmu_events.events_guest;
events_host = host->pmu_events.events_host;
@@ -193,7 +193,7 @@ void kvm_vcpu_pmu_restore_host(struct kvm_vcpu *vcpu)
if (!has_vhe())
return;
- host = this_cpu_ptr(&kvm_host_data);
+ host = this_cpu_ptr_hyp(kvm_host_data);
events_guest = host->pmu_events.events_guest;
events_host = host->pmu_events.events_host;
--
2.28.0.618.gf4bc123cb7-goog
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v3 08/11] kvm: arm64: Create separate instances of kvm_host_data for VHE/nVHE
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Suzuki K Poulose, Catalin Marinas, linux-kernel,
James Morse, linux-arm-kernel, Marc Zyngier, Tejun Heo,
Dennis Zhou, Christoph Lameter, David Brazdil, Will Deacon,
Julien Thierry
Host CPU context is stored in a global per-cpu variable `kvm_host_data`.
In preparation for introducing independent per-CPU region for nVHE hyp,
create two separate instances of `kvm_host_data`, one for VHE and one
for nVHE.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/kvm_host.h | 2 +-
arch/arm64/kernel/image-vars.h | 1 -
arch/arm64/kvm/arm.c | 5 ++---
arch/arm64/kvm/hyp/nvhe/switch.c | 3 +++
arch/arm64/kvm/hyp/vhe/switch.c | 3 +++
arch/arm64/kvm/pmu.c | 8 ++++----
6 files changed, 13 insertions(+), 9 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 905c2b87e05a..5d8c63f5e97e 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -565,7 +565,7 @@ void kvm_set_sei_esr(struct kvm_vcpu *vcpu, u64 syndrome);
struct kvm_vcpu *kvm_mpidr_to_vcpu(struct kvm *kvm, unsigned long mpidr);
-DECLARE_PER_CPU(kvm_host_data_t, kvm_host_data);
+DECLARE_KVM_HYP_PER_CPU(kvm_host_data_t, kvm_host_data);
static inline void kvm_init_host_cpu_context(struct kvm_cpu_context *cpu_ctxt)
{
diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h
index 59d12a0b4622..80da861b8180 100644
--- a/arch/arm64/kernel/image-vars.h
+++ b/arch/arm64/kernel/image-vars.h
@@ -67,7 +67,6 @@ KVM_NVHE_ALIAS(kvm_patch_vector_branch);
KVM_NVHE_ALIAS(kvm_update_va_mask);
/* Global kernel state accessed by nVHE hyp code. */
-KVM_NVHE_ALIAS(kvm_host_data);
KVM_NVHE_ALIAS(kvm_vgic_global_state);
/* Kernel constant needed to compute idmap addresses. */
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 3bdc2661d276..7af9809fa193 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -46,7 +46,6 @@
__asm__(".arch_extension virt");
#endif
-DEFINE_PER_CPU(kvm_host_data_t, kvm_host_data);
static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page);
/* The VMID used in the VTTBR */
@@ -1308,7 +1307,7 @@ static void cpu_hyp_reset(void)
static void cpu_hyp_reinit(void)
{
- kvm_init_host_cpu_context(&this_cpu_ptr(&kvm_host_data)->host_ctxt);
+ kvm_init_host_cpu_context(&this_cpu_ptr_hyp(kvm_host_data)->host_ctxt);
cpu_hyp_reset();
@@ -1543,7 +1542,7 @@ static int init_hyp_mode(void)
for_each_possible_cpu(cpu) {
kvm_host_data_t *cpu_data;
- cpu_data = per_cpu_ptr(&kvm_host_data, cpu);
+ cpu_data = per_cpu_ptr_hyp(kvm_host_data, cpu);
err = create_hyp_mappings(cpu_data, cpu_data + 1, PAGE_HYP);
if (err) {
diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c
index 4662df6330d7..a7e9b03bd9d1 100644
--- a/arch/arm64/kvm/hyp/nvhe/switch.c
+++ b/arch/arm64/kvm/hyp/nvhe/switch.c
@@ -30,6 +30,9 @@
/* Non-VHE copy of the kernel symbol. */
DEFINE_PER_CPU_READ_MOSTLY(u64, arm64_ssbd_callback_required);
+/* Non-VHE instance of kvm_host_data. */
+DEFINE_PER_CPU(kvm_host_data_t, kvm_host_data);
+
static void __activate_traps(struct kvm_vcpu *vcpu)
{
u64 val;
diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c
index 575e8054f116..0949fc97bf03 100644
--- a/arch/arm64/kvm/hyp/vhe/switch.c
+++ b/arch/arm64/kvm/hyp/vhe/switch.c
@@ -28,6 +28,9 @@
const char __hyp_panic_string[] = "HYP panic:\nPS:%08llx PC:%016llx ESR:%08llx\nFAR:%016llx HPFAR:%016llx PAR:%016llx\nVCPU:%p\n";
+/* VHE instance of kvm_host_data. */
+DEFINE_PER_CPU(kvm_host_data_t, kvm_host_data);
+
static void __activate_traps(struct kvm_vcpu *vcpu)
{
u64 val;
diff --git a/arch/arm64/kvm/pmu.c b/arch/arm64/kvm/pmu.c
index 3c224162b3dd..6d80ffe1ebfc 100644
--- a/arch/arm64/kvm/pmu.c
+++ b/arch/arm64/kvm/pmu.c
@@ -31,7 +31,7 @@ static bool kvm_pmu_switch_needed(struct perf_event_attr *attr)
*/
void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr)
{
- struct kvm_host_data *ctx = this_cpu_ptr(&kvm_host_data);
+ struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data);
if (!kvm_pmu_switch_needed(attr))
return;
@@ -47,7 +47,7 @@ void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr)
*/
void kvm_clr_pmu_events(u32 clr)
{
- struct kvm_host_data *ctx = this_cpu_ptr(&kvm_host_data);
+ struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data);
ctx->pmu_events.events_host &= ~clr;
ctx->pmu_events.events_guest &= ~clr;
@@ -173,7 +173,7 @@ void kvm_vcpu_pmu_restore_guest(struct kvm_vcpu *vcpu)
return;
preempt_disable();
- host = this_cpu_ptr(&kvm_host_data);
+ host = this_cpu_ptr_hyp(kvm_host_data);
events_guest = host->pmu_events.events_guest;
events_host = host->pmu_events.events_host;
@@ -193,7 +193,7 @@ void kvm_vcpu_pmu_restore_host(struct kvm_vcpu *vcpu)
if (!has_vhe())
return;
- host = this_cpu_ptr(&kvm_host_data);
+ host = this_cpu_ptr_hyp(kvm_host_data);
events_guest = host->pmu_events.events_guest;
events_host = host->pmu_events.events_host;
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v3 08/11] kvm: arm64: Create separate instances of kvm_host_data for VHE/nVHE
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Catalin Marinas, linux-kernel, linux-arm-kernel,
Marc Zyngier, Tejun Heo, Dennis Zhou, Christoph Lameter,
Will Deacon
Host CPU context is stored in a global per-cpu variable `kvm_host_data`.
In preparation for introducing independent per-CPU region for nVHE hyp,
create two separate instances of `kvm_host_data`, one for VHE and one
for nVHE.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/kvm_host.h | 2 +-
arch/arm64/kernel/image-vars.h | 1 -
arch/arm64/kvm/arm.c | 5 ++---
arch/arm64/kvm/hyp/nvhe/switch.c | 3 +++
arch/arm64/kvm/hyp/vhe/switch.c | 3 +++
arch/arm64/kvm/pmu.c | 8 ++++----
6 files changed, 13 insertions(+), 9 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 905c2b87e05a..5d8c63f5e97e 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -565,7 +565,7 @@ void kvm_set_sei_esr(struct kvm_vcpu *vcpu, u64 syndrome);
struct kvm_vcpu *kvm_mpidr_to_vcpu(struct kvm *kvm, unsigned long mpidr);
-DECLARE_PER_CPU(kvm_host_data_t, kvm_host_data);
+DECLARE_KVM_HYP_PER_CPU(kvm_host_data_t, kvm_host_data);
static inline void kvm_init_host_cpu_context(struct kvm_cpu_context *cpu_ctxt)
{
diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h
index 59d12a0b4622..80da861b8180 100644
--- a/arch/arm64/kernel/image-vars.h
+++ b/arch/arm64/kernel/image-vars.h
@@ -67,7 +67,6 @@ KVM_NVHE_ALIAS(kvm_patch_vector_branch);
KVM_NVHE_ALIAS(kvm_update_va_mask);
/* Global kernel state accessed by nVHE hyp code. */
-KVM_NVHE_ALIAS(kvm_host_data);
KVM_NVHE_ALIAS(kvm_vgic_global_state);
/* Kernel constant needed to compute idmap addresses. */
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 3bdc2661d276..7af9809fa193 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -46,7 +46,6 @@
__asm__(".arch_extension virt");
#endif
-DEFINE_PER_CPU(kvm_host_data_t, kvm_host_data);
static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page);
/* The VMID used in the VTTBR */
@@ -1308,7 +1307,7 @@ static void cpu_hyp_reset(void)
static void cpu_hyp_reinit(void)
{
- kvm_init_host_cpu_context(&this_cpu_ptr(&kvm_host_data)->host_ctxt);
+ kvm_init_host_cpu_context(&this_cpu_ptr_hyp(kvm_host_data)->host_ctxt);
cpu_hyp_reset();
@@ -1543,7 +1542,7 @@ static int init_hyp_mode(void)
for_each_possible_cpu(cpu) {
kvm_host_data_t *cpu_data;
- cpu_data = per_cpu_ptr(&kvm_host_data, cpu);
+ cpu_data = per_cpu_ptr_hyp(kvm_host_data, cpu);
err = create_hyp_mappings(cpu_data, cpu_data + 1, PAGE_HYP);
if (err) {
diff --git a/arch/arm64/kvm/hyp/nvhe/switch.c b/arch/arm64/kvm/hyp/nvhe/switch.c
index 4662df6330d7..a7e9b03bd9d1 100644
--- a/arch/arm64/kvm/hyp/nvhe/switch.c
+++ b/arch/arm64/kvm/hyp/nvhe/switch.c
@@ -30,6 +30,9 @@
/* Non-VHE copy of the kernel symbol. */
DEFINE_PER_CPU_READ_MOSTLY(u64, arm64_ssbd_callback_required);
+/* Non-VHE instance of kvm_host_data. */
+DEFINE_PER_CPU(kvm_host_data_t, kvm_host_data);
+
static void __activate_traps(struct kvm_vcpu *vcpu)
{
u64 val;
diff --git a/arch/arm64/kvm/hyp/vhe/switch.c b/arch/arm64/kvm/hyp/vhe/switch.c
index 575e8054f116..0949fc97bf03 100644
--- a/arch/arm64/kvm/hyp/vhe/switch.c
+++ b/arch/arm64/kvm/hyp/vhe/switch.c
@@ -28,6 +28,9 @@
const char __hyp_panic_string[] = "HYP panic:\nPS:%08llx PC:%016llx ESR:%08llx\nFAR:%016llx HPFAR:%016llx PAR:%016llx\nVCPU:%p\n";
+/* VHE instance of kvm_host_data. */
+DEFINE_PER_CPU(kvm_host_data_t, kvm_host_data);
+
static void __activate_traps(struct kvm_vcpu *vcpu)
{
u64 val;
diff --git a/arch/arm64/kvm/pmu.c b/arch/arm64/kvm/pmu.c
index 3c224162b3dd..6d80ffe1ebfc 100644
--- a/arch/arm64/kvm/pmu.c
+++ b/arch/arm64/kvm/pmu.c
@@ -31,7 +31,7 @@ static bool kvm_pmu_switch_needed(struct perf_event_attr *attr)
*/
void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr)
{
- struct kvm_host_data *ctx = this_cpu_ptr(&kvm_host_data);
+ struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data);
if (!kvm_pmu_switch_needed(attr))
return;
@@ -47,7 +47,7 @@ void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr)
*/
void kvm_clr_pmu_events(u32 clr)
{
- struct kvm_host_data *ctx = this_cpu_ptr(&kvm_host_data);
+ struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data);
ctx->pmu_events.events_host &= ~clr;
ctx->pmu_events.events_guest &= ~clr;
@@ -173,7 +173,7 @@ void kvm_vcpu_pmu_restore_guest(struct kvm_vcpu *vcpu)
return;
preempt_disable();
- host = this_cpu_ptr(&kvm_host_data);
+ host = this_cpu_ptr_hyp(kvm_host_data);
events_guest = host->pmu_events.events_guest;
events_host = host->pmu_events.events_host;
@@ -193,7 +193,7 @@ void kvm_vcpu_pmu_restore_host(struct kvm_vcpu *vcpu)
if (!has_vhe())
return;
- host = this_cpu_ptr(&kvm_host_data);
+ host = this_cpu_ptr_hyp(kvm_host_data);
events_guest = host->pmu_events.events_guest;
events_host = host->pmu_events.events_host;
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v3 08/11] kvm: arm64: Create separate instances of kvm_host_data for VHE/nVHE
2020-09-16 17:34 ` David Brazdil
(?)
@ 2020-09-18 11:58 ` Will Deacon
-1 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 11:58 UTC (permalink / raw)
To: David Brazdil
Cc: kvmarm, Catalin Marinas, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team
On Wed, Sep 16, 2020 at 06:34:36PM +0100, David Brazdil wrote:
> Host CPU context is stored in a global per-cpu variable `kvm_host_data`.
> In preparation for introducing independent per-CPU region for nVHE hyp,
> create two separate instances of `kvm_host_data`, one for VHE and one
> for nVHE.
>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/kvm_host.h | 2 +-
> arch/arm64/kernel/image-vars.h | 1 -
> arch/arm64/kvm/arm.c | 5 ++---
> arch/arm64/kvm/hyp/nvhe/switch.c | 3 +++
> arch/arm64/kvm/hyp/vhe/switch.c | 3 +++
> arch/arm64/kvm/pmu.c | 8 ++++----
> 6 files changed, 13 insertions(+), 9 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index 905c2b87e05a..5d8c63f5e97e 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -565,7 +565,7 @@ void kvm_set_sei_esr(struct kvm_vcpu *vcpu, u64 syndrome);
>
> struct kvm_vcpu *kvm_mpidr_to_vcpu(struct kvm *kvm, unsigned long mpidr);
>
> -DECLARE_PER_CPU(kvm_host_data_t, kvm_host_data);
> +DECLARE_KVM_HYP_PER_CPU(kvm_host_data_t, kvm_host_data);
>
> static inline void kvm_init_host_cpu_context(struct kvm_cpu_context *cpu_ctxt)
> {
> diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h
> index 59d12a0b4622..80da861b8180 100644
> --- a/arch/arm64/kernel/image-vars.h
> +++ b/arch/arm64/kernel/image-vars.h
> @@ -67,7 +67,6 @@ KVM_NVHE_ALIAS(kvm_patch_vector_branch);
> KVM_NVHE_ALIAS(kvm_update_va_mask);
>
> /* Global kernel state accessed by nVHE hyp code. */
> -KVM_NVHE_ALIAS(kvm_host_data);
> KVM_NVHE_ALIAS(kvm_vgic_global_state);
>
> /* Kernel constant needed to compute idmap addresses. */
> diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> index 3bdc2661d276..7af9809fa193 100644
> --- a/arch/arm64/kvm/arm.c
> +++ b/arch/arm64/kvm/arm.c
> @@ -46,7 +46,6 @@
> __asm__(".arch_extension virt");
> #endif
>
> -DEFINE_PER_CPU(kvm_host_data_t, kvm_host_data);
> static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page);
>
> /* The VMID used in the VTTBR */
> @@ -1308,7 +1307,7 @@ static void cpu_hyp_reset(void)
>
> static void cpu_hyp_reinit(void)
> {
> - kvm_init_host_cpu_context(&this_cpu_ptr(&kvm_host_data)->host_ctxt);
> + kvm_init_host_cpu_context(&this_cpu_ptr_hyp(kvm_host_data)->host_ctxt);
>
> cpu_hyp_reset();
>
> @@ -1543,7 +1542,7 @@ static int init_hyp_mode(void)
> for_each_possible_cpu(cpu) {
> kvm_host_data_t *cpu_data;
>
> - cpu_data = per_cpu_ptr(&kvm_host_data, cpu);
> + cpu_data = per_cpu_ptr_hyp(kvm_host_data, cpu);
I stand by my earlier comment to add _sym here, given that the ampersand
gets dropped from the argument.
So assuming you do that in the earlier patch:
Acked-by: Will Deacon <will@kernel.org>
Will
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v3 08/11] kvm: arm64: Create separate instances of kvm_host_data for VHE/nVHE
@ 2020-09-18 11:58 ` Will Deacon
0 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 11:58 UTC (permalink / raw)
To: David Brazdil
Cc: Suzuki K Poulose, Catalin Marinas, linux-kernel, James Morse,
linux-arm-kernel, Marc Zyngier, Tejun Heo, Dennis Zhou,
Christoph Lameter, kernel-team, kvmarm, Julien Thierry
On Wed, Sep 16, 2020 at 06:34:36PM +0100, David Brazdil wrote:
> Host CPU context is stored in a global per-cpu variable `kvm_host_data`.
> In preparation for introducing independent per-CPU region for nVHE hyp,
> create two separate instances of `kvm_host_data`, one for VHE and one
> for nVHE.
>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/kvm_host.h | 2 +-
> arch/arm64/kernel/image-vars.h | 1 -
> arch/arm64/kvm/arm.c | 5 ++---
> arch/arm64/kvm/hyp/nvhe/switch.c | 3 +++
> arch/arm64/kvm/hyp/vhe/switch.c | 3 +++
> arch/arm64/kvm/pmu.c | 8 ++++----
> 6 files changed, 13 insertions(+), 9 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index 905c2b87e05a..5d8c63f5e97e 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -565,7 +565,7 @@ void kvm_set_sei_esr(struct kvm_vcpu *vcpu, u64 syndrome);
>
> struct kvm_vcpu *kvm_mpidr_to_vcpu(struct kvm *kvm, unsigned long mpidr);
>
> -DECLARE_PER_CPU(kvm_host_data_t, kvm_host_data);
> +DECLARE_KVM_HYP_PER_CPU(kvm_host_data_t, kvm_host_data);
>
> static inline void kvm_init_host_cpu_context(struct kvm_cpu_context *cpu_ctxt)
> {
> diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h
> index 59d12a0b4622..80da861b8180 100644
> --- a/arch/arm64/kernel/image-vars.h
> +++ b/arch/arm64/kernel/image-vars.h
> @@ -67,7 +67,6 @@ KVM_NVHE_ALIAS(kvm_patch_vector_branch);
> KVM_NVHE_ALIAS(kvm_update_va_mask);
>
> /* Global kernel state accessed by nVHE hyp code. */
> -KVM_NVHE_ALIAS(kvm_host_data);
> KVM_NVHE_ALIAS(kvm_vgic_global_state);
>
> /* Kernel constant needed to compute idmap addresses. */
> diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> index 3bdc2661d276..7af9809fa193 100644
> --- a/arch/arm64/kvm/arm.c
> +++ b/arch/arm64/kvm/arm.c
> @@ -46,7 +46,6 @@
> __asm__(".arch_extension virt");
> #endif
>
> -DEFINE_PER_CPU(kvm_host_data_t, kvm_host_data);
> static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page);
>
> /* The VMID used in the VTTBR */
> @@ -1308,7 +1307,7 @@ static void cpu_hyp_reset(void)
>
> static void cpu_hyp_reinit(void)
> {
> - kvm_init_host_cpu_context(&this_cpu_ptr(&kvm_host_data)->host_ctxt);
> + kvm_init_host_cpu_context(&this_cpu_ptr_hyp(kvm_host_data)->host_ctxt);
>
> cpu_hyp_reset();
>
> @@ -1543,7 +1542,7 @@ static int init_hyp_mode(void)
> for_each_possible_cpu(cpu) {
> kvm_host_data_t *cpu_data;
>
> - cpu_data = per_cpu_ptr(&kvm_host_data, cpu);
> + cpu_data = per_cpu_ptr_hyp(kvm_host_data, cpu);
I stand by my earlier comment to add _sym here, given that the ampersand
gets dropped from the argument.
So assuming you do that in the earlier patch:
Acked-by: Will Deacon <will@kernel.org>
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v3 08/11] kvm: arm64: Create separate instances of kvm_host_data for VHE/nVHE
@ 2020-09-18 11:58 ` Will Deacon
0 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 11:58 UTC (permalink / raw)
To: David Brazdil
Cc: Catalin Marinas, linux-kernel, linux-arm-kernel, Marc Zyngier,
Tejun Heo, Dennis Zhou, Christoph Lameter, kernel-team, kvmarm
On Wed, Sep 16, 2020 at 06:34:36PM +0100, David Brazdil wrote:
> Host CPU context is stored in a global per-cpu variable `kvm_host_data`.
> In preparation for introducing independent per-CPU region for nVHE hyp,
> create two separate instances of `kvm_host_data`, one for VHE and one
> for nVHE.
>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/kvm_host.h | 2 +-
> arch/arm64/kernel/image-vars.h | 1 -
> arch/arm64/kvm/arm.c | 5 ++---
> arch/arm64/kvm/hyp/nvhe/switch.c | 3 +++
> arch/arm64/kvm/hyp/vhe/switch.c | 3 +++
> arch/arm64/kvm/pmu.c | 8 ++++----
> 6 files changed, 13 insertions(+), 9 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index 905c2b87e05a..5d8c63f5e97e 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -565,7 +565,7 @@ void kvm_set_sei_esr(struct kvm_vcpu *vcpu, u64 syndrome);
>
> struct kvm_vcpu *kvm_mpidr_to_vcpu(struct kvm *kvm, unsigned long mpidr);
>
> -DECLARE_PER_CPU(kvm_host_data_t, kvm_host_data);
> +DECLARE_KVM_HYP_PER_CPU(kvm_host_data_t, kvm_host_data);
>
> static inline void kvm_init_host_cpu_context(struct kvm_cpu_context *cpu_ctxt)
> {
> diff --git a/arch/arm64/kernel/image-vars.h b/arch/arm64/kernel/image-vars.h
> index 59d12a0b4622..80da861b8180 100644
> --- a/arch/arm64/kernel/image-vars.h
> +++ b/arch/arm64/kernel/image-vars.h
> @@ -67,7 +67,6 @@ KVM_NVHE_ALIAS(kvm_patch_vector_branch);
> KVM_NVHE_ALIAS(kvm_update_va_mask);
>
> /* Global kernel state accessed by nVHE hyp code. */
> -KVM_NVHE_ALIAS(kvm_host_data);
> KVM_NVHE_ALIAS(kvm_vgic_global_state);
>
> /* Kernel constant needed to compute idmap addresses. */
> diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> index 3bdc2661d276..7af9809fa193 100644
> --- a/arch/arm64/kvm/arm.c
> +++ b/arch/arm64/kvm/arm.c
> @@ -46,7 +46,6 @@
> __asm__(".arch_extension virt");
> #endif
>
> -DEFINE_PER_CPU(kvm_host_data_t, kvm_host_data);
> static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page);
>
> /* The VMID used in the VTTBR */
> @@ -1308,7 +1307,7 @@ static void cpu_hyp_reset(void)
>
> static void cpu_hyp_reinit(void)
> {
> - kvm_init_host_cpu_context(&this_cpu_ptr(&kvm_host_data)->host_ctxt);
> + kvm_init_host_cpu_context(&this_cpu_ptr_hyp(kvm_host_data)->host_ctxt);
>
> cpu_hyp_reset();
>
> @@ -1543,7 +1542,7 @@ static int init_hyp_mode(void)
> for_each_possible_cpu(cpu) {
> kvm_host_data_t *cpu_data;
>
> - cpu_data = per_cpu_ptr(&kvm_host_data, cpu);
> + cpu_data = per_cpu_ptr_hyp(kvm_host_data, cpu);
I stand by my earlier comment to add _sym here, given that the ampersand
gets dropped from the argument.
So assuming you do that in the earlier patch:
Acked-by: Will Deacon <will@kernel.org>
Will
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v3 09/11] kvm: arm64: Mark hyp stack pages reserved
2020-09-16 17:34 ` David Brazdil
(?)
@ 2020-09-16 17:34 ` David Brazdil
-1 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: Catalin Marinas, Will Deacon, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team,
David Brazdil
In preparation for unmapping hyp pages from host stage-2, allocate/free hyp
stack using new helpers which automatically mark the pages reserved.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/kvm/arm.c | 49 ++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 47 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 7af9809fa193..58d7d614519b 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -1456,13 +1456,58 @@ static int init_subsystems(void)
return err;
}
+/*
+ * Alloc pages and mark them reserved so the kernel never tries to
+ * take them away from the hypervisor.
+ */
+static unsigned long alloc_hyp_pages(gfp_t flags, unsigned int order)
+{
+ struct page *page;
+ unsigned long i;
+
+ page = alloc_pages(flags, order);
+ if (!page)
+ return 0;
+
+ for (i = 0; i < (1ul << order); ++i)
+ mark_page_reserved(page + i);
+
+ return (unsigned long)page_address(page);
+}
+
+static unsigned long alloc_hyp_page(gfp_t flags)
+{
+ return alloc_hyp_pages(flags, 0);
+}
+
+/*
+ * Free pages which were previously marked reserved for the hypervisor.
+ */
+static void free_hyp_pages(unsigned long addr, unsigned int order)
+{
+ unsigned long i;
+ struct page *page;
+
+ if (!addr)
+ return;
+
+ page = virt_to_page(addr);
+ for (i = 0; i < (1ul << order); ++i)
+ free_reserved_page(page + i);
+}
+
+static void free_hyp_page(unsigned long addr)
+{
+ return free_hyp_pages(addr, 0);
+}
+
static void teardown_hyp_mode(void)
{
int cpu;
free_hyp_pgds();
for_each_possible_cpu(cpu)
- free_page(per_cpu(kvm_arm_hyp_stack_page, cpu));
+ free_hyp_page(per_cpu(kvm_arm_hyp_stack_page, cpu));
}
/**
@@ -1486,7 +1531,7 @@ static int init_hyp_mode(void)
for_each_possible_cpu(cpu) {
unsigned long stack_page;
- stack_page = __get_free_page(GFP_KERNEL);
+ stack_page = alloc_hyp_page(GFP_KERNEL);
if (!stack_page) {
err = -ENOMEM;
goto out_err;
--
2.28.0.618.gf4bc123cb7-goog
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v3 09/11] kvm: arm64: Mark hyp stack pages reserved
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Suzuki K Poulose, Catalin Marinas, linux-kernel,
James Morse, linux-arm-kernel, Marc Zyngier, Tejun Heo,
Dennis Zhou, Christoph Lameter, David Brazdil, Will Deacon,
Julien Thierry
In preparation for unmapping hyp pages from host stage-2, allocate/free hyp
stack using new helpers which automatically mark the pages reserved.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/kvm/arm.c | 49 ++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 47 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 7af9809fa193..58d7d614519b 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -1456,13 +1456,58 @@ static int init_subsystems(void)
return err;
}
+/*
+ * Alloc pages and mark them reserved so the kernel never tries to
+ * take them away from the hypervisor.
+ */
+static unsigned long alloc_hyp_pages(gfp_t flags, unsigned int order)
+{
+ struct page *page;
+ unsigned long i;
+
+ page = alloc_pages(flags, order);
+ if (!page)
+ return 0;
+
+ for (i = 0; i < (1ul << order); ++i)
+ mark_page_reserved(page + i);
+
+ return (unsigned long)page_address(page);
+}
+
+static unsigned long alloc_hyp_page(gfp_t flags)
+{
+ return alloc_hyp_pages(flags, 0);
+}
+
+/*
+ * Free pages which were previously marked reserved for the hypervisor.
+ */
+static void free_hyp_pages(unsigned long addr, unsigned int order)
+{
+ unsigned long i;
+ struct page *page;
+
+ if (!addr)
+ return;
+
+ page = virt_to_page(addr);
+ for (i = 0; i < (1ul << order); ++i)
+ free_reserved_page(page + i);
+}
+
+static void free_hyp_page(unsigned long addr)
+{
+ return free_hyp_pages(addr, 0);
+}
+
static void teardown_hyp_mode(void)
{
int cpu;
free_hyp_pgds();
for_each_possible_cpu(cpu)
- free_page(per_cpu(kvm_arm_hyp_stack_page, cpu));
+ free_hyp_page(per_cpu(kvm_arm_hyp_stack_page, cpu));
}
/**
@@ -1486,7 +1531,7 @@ static int init_hyp_mode(void)
for_each_possible_cpu(cpu) {
unsigned long stack_page;
- stack_page = __get_free_page(GFP_KERNEL);
+ stack_page = alloc_hyp_page(GFP_KERNEL);
if (!stack_page) {
err = -ENOMEM;
goto out_err;
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v3 09/11] kvm: arm64: Mark hyp stack pages reserved
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Catalin Marinas, linux-kernel, linux-arm-kernel,
Marc Zyngier, Tejun Heo, Dennis Zhou, Christoph Lameter,
Will Deacon
In preparation for unmapping hyp pages from host stage-2, allocate/free hyp
stack using new helpers which automatically mark the pages reserved.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/kvm/arm.c | 49 ++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 47 insertions(+), 2 deletions(-)
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 7af9809fa193..58d7d614519b 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -1456,13 +1456,58 @@ static int init_subsystems(void)
return err;
}
+/*
+ * Alloc pages and mark them reserved so the kernel never tries to
+ * take them away from the hypervisor.
+ */
+static unsigned long alloc_hyp_pages(gfp_t flags, unsigned int order)
+{
+ struct page *page;
+ unsigned long i;
+
+ page = alloc_pages(flags, order);
+ if (!page)
+ return 0;
+
+ for (i = 0; i < (1ul << order); ++i)
+ mark_page_reserved(page + i);
+
+ return (unsigned long)page_address(page);
+}
+
+static unsigned long alloc_hyp_page(gfp_t flags)
+{
+ return alloc_hyp_pages(flags, 0);
+}
+
+/*
+ * Free pages which were previously marked reserved for the hypervisor.
+ */
+static void free_hyp_pages(unsigned long addr, unsigned int order)
+{
+ unsigned long i;
+ struct page *page;
+
+ if (!addr)
+ return;
+
+ page = virt_to_page(addr);
+ for (i = 0; i < (1ul << order); ++i)
+ free_reserved_page(page + i);
+}
+
+static void free_hyp_page(unsigned long addr)
+{
+ return free_hyp_pages(addr, 0);
+}
+
static void teardown_hyp_mode(void)
{
int cpu;
free_hyp_pgds();
for_each_possible_cpu(cpu)
- free_page(per_cpu(kvm_arm_hyp_stack_page, cpu));
+ free_hyp_page(per_cpu(kvm_arm_hyp_stack_page, cpu));
}
/**
@@ -1486,7 +1531,7 @@ static int init_hyp_mode(void)
for_each_possible_cpu(cpu) {
unsigned long stack_page;
- stack_page = __get_free_page(GFP_KERNEL);
+ stack_page = alloc_hyp_page(GFP_KERNEL);
if (!stack_page) {
err = -ENOMEM;
goto out_err;
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v3 09/11] kvm: arm64: Mark hyp stack pages reserved
2020-09-16 17:34 ` David Brazdil
(?)
@ 2020-09-18 12:00 ` Will Deacon
-1 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 12:00 UTC (permalink / raw)
To: David Brazdil
Cc: kvmarm, Catalin Marinas, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team
On Wed, Sep 16, 2020 at 06:34:37PM +0100, David Brazdil wrote:
> In preparation for unmapping hyp pages from host stage-2, allocate/free hyp
> stack using new helpers which automatically mark the pages reserved.
Given that this series doesn't get us that the point of having a stage-2 for
the host, I don't see why we need this part yet.
Will
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3 09/11] kvm: arm64: Mark hyp stack pages reserved
@ 2020-09-18 12:00 ` Will Deacon
0 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 12:00 UTC (permalink / raw)
To: David Brazdil
Cc: Suzuki K Poulose, Catalin Marinas, linux-kernel, James Morse,
linux-arm-kernel, Marc Zyngier, Tejun Heo, Dennis Zhou,
Christoph Lameter, kernel-team, kvmarm, Julien Thierry
On Wed, Sep 16, 2020 at 06:34:37PM +0100, David Brazdil wrote:
> In preparation for unmapping hyp pages from host stage-2, allocate/free hyp
> stack using new helpers which automatically mark the pages reserved.
Given that this series doesn't get us that the point of having a stage-2 for
the host, I don't see why we need this part yet.
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3 09/11] kvm: arm64: Mark hyp stack pages reserved
@ 2020-09-18 12:00 ` Will Deacon
0 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 12:00 UTC (permalink / raw)
To: David Brazdil
Cc: Catalin Marinas, linux-kernel, linux-arm-kernel, Marc Zyngier,
Tejun Heo, Dennis Zhou, Christoph Lameter, kernel-team, kvmarm
On Wed, Sep 16, 2020 at 06:34:37PM +0100, David Brazdil wrote:
> In preparation for unmapping hyp pages from host stage-2, allocate/free hyp
> stack using new helpers which automatically mark the pages reserved.
Given that this series doesn't get us that the point of having a stage-2 for
the host, I don't see why we need this part yet.
Will
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3 09/11] kvm: arm64: Mark hyp stack pages reserved
2020-09-18 12:00 ` Will Deacon
(?)
@ 2020-09-22 18:08 ` David Brazdil
-1 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-22 18:08 UTC (permalink / raw)
To: Will Deacon
Cc: kvmarm, Catalin Marinas, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team
On Fri, Sep 18, 2020 at 01:00:44PM +0100, Will Deacon wrote:
> On Wed, Sep 16, 2020 at 06:34:37PM +0100, David Brazdil wrote:
> > In preparation for unmapping hyp pages from host stage-2, allocate/free hyp
> > stack using new helpers which automatically mark the pages reserved.
>
> Given that this series doesn't get us that the point of having a stage-2 for
> the host, I don't see why we need this part yet.
>
> Will
Dropped
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3 09/11] kvm: arm64: Mark hyp stack pages reserved
@ 2020-09-22 18:08 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-22 18:08 UTC (permalink / raw)
To: Will Deacon
Cc: Suzuki K Poulose, Catalin Marinas, linux-kernel, James Morse,
linux-arm-kernel, Marc Zyngier, Tejun Heo, Dennis Zhou,
Christoph Lameter, kernel-team, kvmarm, Julien Thierry
On Fri, Sep 18, 2020 at 01:00:44PM +0100, Will Deacon wrote:
> On Wed, Sep 16, 2020 at 06:34:37PM +0100, David Brazdil wrote:
> > In preparation for unmapping hyp pages from host stage-2, allocate/free hyp
> > stack using new helpers which automatically mark the pages reserved.
>
> Given that this series doesn't get us that the point of having a stage-2 for
> the host, I don't see why we need this part yet.
>
> Will
Dropped
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 87+ messages in thread
* Re: [PATCH v3 09/11] kvm: arm64: Mark hyp stack pages reserved
@ 2020-09-22 18:08 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-22 18:08 UTC (permalink / raw)
To: Will Deacon
Cc: Catalin Marinas, linux-kernel, linux-arm-kernel, Marc Zyngier,
Tejun Heo, Dennis Zhou, Christoph Lameter, kernel-team, kvmarm
On Fri, Sep 18, 2020 at 01:00:44PM +0100, Will Deacon wrote:
> On Wed, Sep 16, 2020 at 06:34:37PM +0100, David Brazdil wrote:
> > In preparation for unmapping hyp pages from host stage-2, allocate/free hyp
> > stack using new helpers which automatically mark the pages reserved.
>
> Given that this series doesn't get us that the point of having a stage-2 for
> the host, I don't see why we need this part yet.
>
> Will
Dropped
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v3 10/11] kvm: arm64: Set up hyp percpu data for nVHE
2020-09-16 17:34 ` David Brazdil
(?)
@ 2020-09-16 17:34 ` David Brazdil
-1 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: Catalin Marinas, Will Deacon, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team,
David Brazdil
Add hyp percpu section to linker script and rename the corresponding ELF
sections of hyp/nvhe object files. This moves all nVHE-specific percpu
variables to the new hyp percpu section.
Allocate sufficient amount of memory for all percpu hyp regions at global KVM
init time and create corresponding hyp mappings.
The base addresses of hyp percpu regions are kept in a dynamically allocated
array in the kernel.
Add NULL checks in PMU event-reset code as it may run before KVM memory is
initialized.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/kvm_asm.h | 19 +++++++++--
arch/arm64/include/asm/sections.h | 1 +
arch/arm64/kernel/vmlinux.lds.S | 9 +++++
arch/arm64/kvm/arm.c | 56 +++++++++++++++++++++++++++++--
arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 6 ++++
arch/arm64/kvm/pmu.c | 5 ++-
6 files changed, 90 insertions(+), 6 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index abc03f386b40..e0e1e404f6eb 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -66,8 +66,23 @@
#define CHOOSE_VHE_SYM(sym) sym
#define CHOOSE_NVHE_SYM(sym) kvm_nvhe_sym(sym)
-#define this_cpu_ptr_nvhe(sym) this_cpu_ptr(&kvm_nvhe_sym(sym))
-#define per_cpu_ptr_nvhe(sym, cpu) per_cpu_ptr(&kvm_nvhe_sym(sym), cpu)
+/* Array of percpu base addresses. Length of the array is nr_cpu_ids. */
+extern unsigned long *kvm_arm_hyp_percpu_base;
+
+/*
+ * Compute pointer to a symbol defined in nVHE percpu region.
+ * Returns NULL if percpu memory has not been allocated yet.
+ */
+#define this_cpu_ptr_nvhe(sym) per_cpu_ptr_nvhe(sym, smp_processor_id())
+#define per_cpu_ptr_nvhe(sym, cpu) \
+ ({ \
+ unsigned long base, off; \
+ base = kvm_arm_hyp_percpu_base \
+ ? kvm_arm_hyp_percpu_base[cpu] : 0; \
+ off = (unsigned long)&kvm_nvhe_sym(sym) - \
+ (unsigned long)&kvm_nvhe_sym(__per_cpu_start); \
+ base ? (typeof(kvm_nvhe_sym(sym))*)(base + off) : NULL; \
+ })
#ifndef __KVM_NVHE_HYPERVISOR__
/*
diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h
index 3994169985ef..5062553a6847 100644
--- a/arch/arm64/include/asm/sections.h
+++ b/arch/arm64/include/asm/sections.h
@@ -18,5 +18,6 @@ extern char __exittext_begin[], __exittext_end[];
extern char __irqentry_text_start[], __irqentry_text_end[];
extern char __mmuoff_data_start[], __mmuoff_data_end[];
extern char __entry_tramp_text_start[], __entry_tramp_text_end[];
+extern char __kvm_nvhe___per_cpu_start[], __kvm_nvhe___per_cpu_end[];
#endif /* __ASM_SECTIONS_H */
diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
index 0d543570e811..d52e6b5dbfd3 100644
--- a/arch/arm64/kernel/vmlinux.lds.S
+++ b/arch/arm64/kernel/vmlinux.lds.S
@@ -9,6 +9,7 @@
#include <asm-generic/vmlinux.lds.h>
#include <asm/cache.h>
+#include <asm/hyp_image.h>
#include <asm/kernel-pgtable.h>
#include <asm/memory.h>
#include <asm/page.h>
@@ -27,8 +28,15 @@ jiffies = jiffies_64;
__start___kvm_ex_table = .; \
*(__kvm_ex_table) \
__stop___kvm_ex_table = .;
+
+#define HYPERVISOR_PERCPU_SECTION \
+ . = ALIGN(PAGE_SIZE); \
+ HYP_SECTION_NAME(.data..percpu) : { \
+ *(HYP_SECTION_NAME(.data..percpu)) \
+ }
#else /* CONFIG_KVM */
#define HYPERVISOR_EXTABLE
+#define HYPERVISOR_PERCPU_SECTION
#endif
#define HYPERVISOR_TEXT \
@@ -194,6 +202,7 @@ SECTIONS
}
PERCPU_SECTION(L1_CACHE_BYTES)
+ HYPERVISOR_PERCPU_SECTION
.rela.dyn : ALIGN(8) {
*(.rela .rela*)
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 58d7d614519b..8293319a32e7 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -47,6 +47,7 @@ __asm__(".arch_extension virt");
#endif
static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page);
+unsigned long *kvm_arm_hyp_percpu_base;
/* The VMID used in the VTTBR */
static atomic64_t kvm_vmid_gen = ATOMIC64_INIT(1);
@@ -1258,6 +1259,15 @@ long kvm_arch_vm_ioctl(struct file *filp,
}
}
+#define kvm_hyp_percpu_base(cpu) ((unsigned long)per_cpu_ptr_nvhe(__per_cpu_start, cpu))
+#define kvm_hyp_percpu_array_size (nr_cpu_ids * sizeof(*kvm_arm_hyp_percpu_base))
+#define kvm_hyp_percpu_array_order (get_order(kvm_hyp_percpu_array_size))
+#define kvm_hyp_percpu_start (CHOOSE_NVHE_SYM(__per_cpu_start))
+#define kvm_hyp_percpu_end (CHOOSE_NVHE_SYM(__per_cpu_end))
+#define kvm_hyp_percpu_size ((unsigned long)kvm_hyp_percpu_end - \
+ (unsigned long)kvm_hyp_percpu_start)
+#define kvm_hyp_percpu_order (kvm_hyp_percpu_size ? get_order(kvm_hyp_percpu_size) : 0)
+
static void cpu_init_hyp_mode(void)
{
phys_addr_t pgd_ptr;
@@ -1273,8 +1283,8 @@ static void cpu_init_hyp_mode(void)
* kernel's mapping to the linear mapping, and store it in tpidr_el2
* so that we can use adr_l to access per-cpu variables in EL2.
*/
- tpidr_el2 = ((unsigned long)this_cpu_ptr(&kvm_host_data) -
- (unsigned long)kvm_ksym_ref(&kvm_host_data));
+ tpidr_el2 = (unsigned long)this_cpu_ptr_nvhe(__per_cpu_start) -
+ (unsigned long)kvm_ksym_ref(kvm_hyp_percpu_start);
pgd_ptr = kvm_mmu_get_httbr();
hyp_stack_ptr = __this_cpu_read(kvm_arm_hyp_stack_page) + PAGE_SIZE;
@@ -1506,8 +1516,11 @@ static void teardown_hyp_mode(void)
int cpu;
free_hyp_pgds();
- for_each_possible_cpu(cpu)
+ for_each_possible_cpu(cpu) {
free_hyp_page(per_cpu(kvm_arm_hyp_stack_page, cpu));
+ free_hyp_pages(kvm_hyp_percpu_base(cpu), kvm_hyp_percpu_order);
+ }
+ free_hyp_pages((unsigned long)kvm_arm_hyp_percpu_base, kvm_hyp_percpu_array_order);
}
/**
@@ -1540,6 +1553,28 @@ static int init_hyp_mode(void)
per_cpu(kvm_arm_hyp_stack_page, cpu) = stack_page;
}
+ /*
+ * Allocate and initialize pages for Hypervisor-mode percpu regions.
+ */
+ kvm_arm_hyp_percpu_base = (unsigned long *)alloc_hyp_pages(
+ GFP_KERNEL | __GFP_ZERO, kvm_hyp_percpu_array_order);
+ if (!kvm_arm_hyp_percpu_base) {
+ err = -ENOMEM;
+ goto out_err;
+ }
+ for_each_possible_cpu(cpu) {
+ unsigned long percpu_base;
+
+ percpu_base = alloc_hyp_pages(GFP_KERNEL, kvm_hyp_percpu_order);
+ if (!percpu_base) {
+ err = -ENOMEM;
+ goto out_err;
+ }
+
+ memcpy((void *)percpu_base, kvm_hyp_percpu_start, kvm_hyp_percpu_size);
+ kvm_arm_hyp_percpu_base[cpu] = percpu_base;
+ }
+
/*
* Map the Hyp-code called directly from the host
*/
@@ -1584,6 +1619,21 @@ static int init_hyp_mode(void)
}
}
+ /*
+ * Map Hyp percpu pages
+ */
+ for_each_possible_cpu(cpu) {
+ char *percpu_begin = (char *)kvm_arm_hyp_percpu_base[cpu];
+ char *percpu_end = percpu_begin + PAGE_ALIGN(kvm_hyp_percpu_size);
+
+ err = create_hyp_mappings(percpu_begin, percpu_end, PAGE_HYP);
+
+ if (err) {
+ kvm_err("Cannot map hyp percpu region\n");
+ goto out_err;
+ }
+ }
+
for_each_possible_cpu(cpu) {
kvm_host_data_t *cpu_data;
diff --git a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S
index 3b13d1c7cd1a..bb2d986ff696 100644
--- a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S
+++ b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S
@@ -7,7 +7,13 @@
*/
#include <asm/hyp_image.h>
+#include <asm-generic/vmlinux.lds.h>
+#include <asm/cache.h>
+#include <asm/memory.h>
SECTIONS {
HYP_SECTION(.text)
+ HYP_SECTION_NAME(.data..percpu) : {
+ PERCPU_INPUT(L1_CACHE_BYTES)
+ }
}
diff --git a/arch/arm64/kvm/pmu.c b/arch/arm64/kvm/pmu.c
index 6d80ffe1ebfc..cd653091f213 100644
--- a/arch/arm64/kvm/pmu.c
+++ b/arch/arm64/kvm/pmu.c
@@ -33,7 +33,7 @@ void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr)
{
struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data);
- if (!kvm_pmu_switch_needed(attr))
+ if (!ctx || !kvm_pmu_switch_needed(attr))
return;
if (!attr->exclude_host)
@@ -49,6 +49,9 @@ void kvm_clr_pmu_events(u32 clr)
{
struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data);
+ if (!ctx)
+ return;
+
ctx->pmu_events.events_host &= ~clr;
ctx->pmu_events.events_guest &= ~clr;
}
--
2.28.0.618.gf4bc123cb7-goog
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v3 10/11] kvm: arm64: Set up hyp percpu data for nVHE
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Suzuki K Poulose, Catalin Marinas, linux-kernel,
James Morse, linux-arm-kernel, Marc Zyngier, Tejun Heo,
Dennis Zhou, Christoph Lameter, David Brazdil, Will Deacon,
Julien Thierry
Add hyp percpu section to linker script and rename the corresponding ELF
sections of hyp/nvhe object files. This moves all nVHE-specific percpu
variables to the new hyp percpu section.
Allocate sufficient amount of memory for all percpu hyp regions at global KVM
init time and create corresponding hyp mappings.
The base addresses of hyp percpu regions are kept in a dynamically allocated
array in the kernel.
Add NULL checks in PMU event-reset code as it may run before KVM memory is
initialized.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/kvm_asm.h | 19 +++++++++--
arch/arm64/include/asm/sections.h | 1 +
arch/arm64/kernel/vmlinux.lds.S | 9 +++++
arch/arm64/kvm/arm.c | 56 +++++++++++++++++++++++++++++--
arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 6 ++++
arch/arm64/kvm/pmu.c | 5 ++-
6 files changed, 90 insertions(+), 6 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index abc03f386b40..e0e1e404f6eb 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -66,8 +66,23 @@
#define CHOOSE_VHE_SYM(sym) sym
#define CHOOSE_NVHE_SYM(sym) kvm_nvhe_sym(sym)
-#define this_cpu_ptr_nvhe(sym) this_cpu_ptr(&kvm_nvhe_sym(sym))
-#define per_cpu_ptr_nvhe(sym, cpu) per_cpu_ptr(&kvm_nvhe_sym(sym), cpu)
+/* Array of percpu base addresses. Length of the array is nr_cpu_ids. */
+extern unsigned long *kvm_arm_hyp_percpu_base;
+
+/*
+ * Compute pointer to a symbol defined in nVHE percpu region.
+ * Returns NULL if percpu memory has not been allocated yet.
+ */
+#define this_cpu_ptr_nvhe(sym) per_cpu_ptr_nvhe(sym, smp_processor_id())
+#define per_cpu_ptr_nvhe(sym, cpu) \
+ ({ \
+ unsigned long base, off; \
+ base = kvm_arm_hyp_percpu_base \
+ ? kvm_arm_hyp_percpu_base[cpu] : 0; \
+ off = (unsigned long)&kvm_nvhe_sym(sym) - \
+ (unsigned long)&kvm_nvhe_sym(__per_cpu_start); \
+ base ? (typeof(kvm_nvhe_sym(sym))*)(base + off) : NULL; \
+ })
#ifndef __KVM_NVHE_HYPERVISOR__
/*
diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h
index 3994169985ef..5062553a6847 100644
--- a/arch/arm64/include/asm/sections.h
+++ b/arch/arm64/include/asm/sections.h
@@ -18,5 +18,6 @@ extern char __exittext_begin[], __exittext_end[];
extern char __irqentry_text_start[], __irqentry_text_end[];
extern char __mmuoff_data_start[], __mmuoff_data_end[];
extern char __entry_tramp_text_start[], __entry_tramp_text_end[];
+extern char __kvm_nvhe___per_cpu_start[], __kvm_nvhe___per_cpu_end[];
#endif /* __ASM_SECTIONS_H */
diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
index 0d543570e811..d52e6b5dbfd3 100644
--- a/arch/arm64/kernel/vmlinux.lds.S
+++ b/arch/arm64/kernel/vmlinux.lds.S
@@ -9,6 +9,7 @@
#include <asm-generic/vmlinux.lds.h>
#include <asm/cache.h>
+#include <asm/hyp_image.h>
#include <asm/kernel-pgtable.h>
#include <asm/memory.h>
#include <asm/page.h>
@@ -27,8 +28,15 @@ jiffies = jiffies_64;
__start___kvm_ex_table = .; \
*(__kvm_ex_table) \
__stop___kvm_ex_table = .;
+
+#define HYPERVISOR_PERCPU_SECTION \
+ . = ALIGN(PAGE_SIZE); \
+ HYP_SECTION_NAME(.data..percpu) : { \
+ *(HYP_SECTION_NAME(.data..percpu)) \
+ }
#else /* CONFIG_KVM */
#define HYPERVISOR_EXTABLE
+#define HYPERVISOR_PERCPU_SECTION
#endif
#define HYPERVISOR_TEXT \
@@ -194,6 +202,7 @@ SECTIONS
}
PERCPU_SECTION(L1_CACHE_BYTES)
+ HYPERVISOR_PERCPU_SECTION
.rela.dyn : ALIGN(8) {
*(.rela .rela*)
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 58d7d614519b..8293319a32e7 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -47,6 +47,7 @@ __asm__(".arch_extension virt");
#endif
static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page);
+unsigned long *kvm_arm_hyp_percpu_base;
/* The VMID used in the VTTBR */
static atomic64_t kvm_vmid_gen = ATOMIC64_INIT(1);
@@ -1258,6 +1259,15 @@ long kvm_arch_vm_ioctl(struct file *filp,
}
}
+#define kvm_hyp_percpu_base(cpu) ((unsigned long)per_cpu_ptr_nvhe(__per_cpu_start, cpu))
+#define kvm_hyp_percpu_array_size (nr_cpu_ids * sizeof(*kvm_arm_hyp_percpu_base))
+#define kvm_hyp_percpu_array_order (get_order(kvm_hyp_percpu_array_size))
+#define kvm_hyp_percpu_start (CHOOSE_NVHE_SYM(__per_cpu_start))
+#define kvm_hyp_percpu_end (CHOOSE_NVHE_SYM(__per_cpu_end))
+#define kvm_hyp_percpu_size ((unsigned long)kvm_hyp_percpu_end - \
+ (unsigned long)kvm_hyp_percpu_start)
+#define kvm_hyp_percpu_order (kvm_hyp_percpu_size ? get_order(kvm_hyp_percpu_size) : 0)
+
static void cpu_init_hyp_mode(void)
{
phys_addr_t pgd_ptr;
@@ -1273,8 +1283,8 @@ static void cpu_init_hyp_mode(void)
* kernel's mapping to the linear mapping, and store it in tpidr_el2
* so that we can use adr_l to access per-cpu variables in EL2.
*/
- tpidr_el2 = ((unsigned long)this_cpu_ptr(&kvm_host_data) -
- (unsigned long)kvm_ksym_ref(&kvm_host_data));
+ tpidr_el2 = (unsigned long)this_cpu_ptr_nvhe(__per_cpu_start) -
+ (unsigned long)kvm_ksym_ref(kvm_hyp_percpu_start);
pgd_ptr = kvm_mmu_get_httbr();
hyp_stack_ptr = __this_cpu_read(kvm_arm_hyp_stack_page) + PAGE_SIZE;
@@ -1506,8 +1516,11 @@ static void teardown_hyp_mode(void)
int cpu;
free_hyp_pgds();
- for_each_possible_cpu(cpu)
+ for_each_possible_cpu(cpu) {
free_hyp_page(per_cpu(kvm_arm_hyp_stack_page, cpu));
+ free_hyp_pages(kvm_hyp_percpu_base(cpu), kvm_hyp_percpu_order);
+ }
+ free_hyp_pages((unsigned long)kvm_arm_hyp_percpu_base, kvm_hyp_percpu_array_order);
}
/**
@@ -1540,6 +1553,28 @@ static int init_hyp_mode(void)
per_cpu(kvm_arm_hyp_stack_page, cpu) = stack_page;
}
+ /*
+ * Allocate and initialize pages for Hypervisor-mode percpu regions.
+ */
+ kvm_arm_hyp_percpu_base = (unsigned long *)alloc_hyp_pages(
+ GFP_KERNEL | __GFP_ZERO, kvm_hyp_percpu_array_order);
+ if (!kvm_arm_hyp_percpu_base) {
+ err = -ENOMEM;
+ goto out_err;
+ }
+ for_each_possible_cpu(cpu) {
+ unsigned long percpu_base;
+
+ percpu_base = alloc_hyp_pages(GFP_KERNEL, kvm_hyp_percpu_order);
+ if (!percpu_base) {
+ err = -ENOMEM;
+ goto out_err;
+ }
+
+ memcpy((void *)percpu_base, kvm_hyp_percpu_start, kvm_hyp_percpu_size);
+ kvm_arm_hyp_percpu_base[cpu] = percpu_base;
+ }
+
/*
* Map the Hyp-code called directly from the host
*/
@@ -1584,6 +1619,21 @@ static int init_hyp_mode(void)
}
}
+ /*
+ * Map Hyp percpu pages
+ */
+ for_each_possible_cpu(cpu) {
+ char *percpu_begin = (char *)kvm_arm_hyp_percpu_base[cpu];
+ char *percpu_end = percpu_begin + PAGE_ALIGN(kvm_hyp_percpu_size);
+
+ err = create_hyp_mappings(percpu_begin, percpu_end, PAGE_HYP);
+
+ if (err) {
+ kvm_err("Cannot map hyp percpu region\n");
+ goto out_err;
+ }
+ }
+
for_each_possible_cpu(cpu) {
kvm_host_data_t *cpu_data;
diff --git a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S
index 3b13d1c7cd1a..bb2d986ff696 100644
--- a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S
+++ b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S
@@ -7,7 +7,13 @@
*/
#include <asm/hyp_image.h>
+#include <asm-generic/vmlinux.lds.h>
+#include <asm/cache.h>
+#include <asm/memory.h>
SECTIONS {
HYP_SECTION(.text)
+ HYP_SECTION_NAME(.data..percpu) : {
+ PERCPU_INPUT(L1_CACHE_BYTES)
+ }
}
diff --git a/arch/arm64/kvm/pmu.c b/arch/arm64/kvm/pmu.c
index 6d80ffe1ebfc..cd653091f213 100644
--- a/arch/arm64/kvm/pmu.c
+++ b/arch/arm64/kvm/pmu.c
@@ -33,7 +33,7 @@ void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr)
{
struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data);
- if (!kvm_pmu_switch_needed(attr))
+ if (!ctx || !kvm_pmu_switch_needed(attr))
return;
if (!attr->exclude_host)
@@ -49,6 +49,9 @@ void kvm_clr_pmu_events(u32 clr)
{
struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data);
+ if (!ctx)
+ return;
+
ctx->pmu_events.events_host &= ~clr;
ctx->pmu_events.events_guest &= ~clr;
}
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v3 10/11] kvm: arm64: Set up hyp percpu data for nVHE
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Catalin Marinas, linux-kernel, linux-arm-kernel,
Marc Zyngier, Tejun Heo, Dennis Zhou, Christoph Lameter,
Will Deacon
Add hyp percpu section to linker script and rename the corresponding ELF
sections of hyp/nvhe object files. This moves all nVHE-specific percpu
variables to the new hyp percpu section.
Allocate sufficient amount of memory for all percpu hyp regions at global KVM
init time and create corresponding hyp mappings.
The base addresses of hyp percpu regions are kept in a dynamically allocated
array in the kernel.
Add NULL checks in PMU event-reset code as it may run before KVM memory is
initialized.
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/kvm_asm.h | 19 +++++++++--
arch/arm64/include/asm/sections.h | 1 +
arch/arm64/kernel/vmlinux.lds.S | 9 +++++
arch/arm64/kvm/arm.c | 56 +++++++++++++++++++++++++++++--
arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 6 ++++
arch/arm64/kvm/pmu.c | 5 ++-
6 files changed, 90 insertions(+), 6 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
index abc03f386b40..e0e1e404f6eb 100644
--- a/arch/arm64/include/asm/kvm_asm.h
+++ b/arch/arm64/include/asm/kvm_asm.h
@@ -66,8 +66,23 @@
#define CHOOSE_VHE_SYM(sym) sym
#define CHOOSE_NVHE_SYM(sym) kvm_nvhe_sym(sym)
-#define this_cpu_ptr_nvhe(sym) this_cpu_ptr(&kvm_nvhe_sym(sym))
-#define per_cpu_ptr_nvhe(sym, cpu) per_cpu_ptr(&kvm_nvhe_sym(sym), cpu)
+/* Array of percpu base addresses. Length of the array is nr_cpu_ids. */
+extern unsigned long *kvm_arm_hyp_percpu_base;
+
+/*
+ * Compute pointer to a symbol defined in nVHE percpu region.
+ * Returns NULL if percpu memory has not been allocated yet.
+ */
+#define this_cpu_ptr_nvhe(sym) per_cpu_ptr_nvhe(sym, smp_processor_id())
+#define per_cpu_ptr_nvhe(sym, cpu) \
+ ({ \
+ unsigned long base, off; \
+ base = kvm_arm_hyp_percpu_base \
+ ? kvm_arm_hyp_percpu_base[cpu] : 0; \
+ off = (unsigned long)&kvm_nvhe_sym(sym) - \
+ (unsigned long)&kvm_nvhe_sym(__per_cpu_start); \
+ base ? (typeof(kvm_nvhe_sym(sym))*)(base + off) : NULL; \
+ })
#ifndef __KVM_NVHE_HYPERVISOR__
/*
diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h
index 3994169985ef..5062553a6847 100644
--- a/arch/arm64/include/asm/sections.h
+++ b/arch/arm64/include/asm/sections.h
@@ -18,5 +18,6 @@ extern char __exittext_begin[], __exittext_end[];
extern char __irqentry_text_start[], __irqentry_text_end[];
extern char __mmuoff_data_start[], __mmuoff_data_end[];
extern char __entry_tramp_text_start[], __entry_tramp_text_end[];
+extern char __kvm_nvhe___per_cpu_start[], __kvm_nvhe___per_cpu_end[];
#endif /* __ASM_SECTIONS_H */
diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
index 0d543570e811..d52e6b5dbfd3 100644
--- a/arch/arm64/kernel/vmlinux.lds.S
+++ b/arch/arm64/kernel/vmlinux.lds.S
@@ -9,6 +9,7 @@
#include <asm-generic/vmlinux.lds.h>
#include <asm/cache.h>
+#include <asm/hyp_image.h>
#include <asm/kernel-pgtable.h>
#include <asm/memory.h>
#include <asm/page.h>
@@ -27,8 +28,15 @@ jiffies = jiffies_64;
__start___kvm_ex_table = .; \
*(__kvm_ex_table) \
__stop___kvm_ex_table = .;
+
+#define HYPERVISOR_PERCPU_SECTION \
+ . = ALIGN(PAGE_SIZE); \
+ HYP_SECTION_NAME(.data..percpu) : { \
+ *(HYP_SECTION_NAME(.data..percpu)) \
+ }
#else /* CONFIG_KVM */
#define HYPERVISOR_EXTABLE
+#define HYPERVISOR_PERCPU_SECTION
#endif
#define HYPERVISOR_TEXT \
@@ -194,6 +202,7 @@ SECTIONS
}
PERCPU_SECTION(L1_CACHE_BYTES)
+ HYPERVISOR_PERCPU_SECTION
.rela.dyn : ALIGN(8) {
*(.rela .rela*)
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 58d7d614519b..8293319a32e7 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -47,6 +47,7 @@ __asm__(".arch_extension virt");
#endif
static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page);
+unsigned long *kvm_arm_hyp_percpu_base;
/* The VMID used in the VTTBR */
static atomic64_t kvm_vmid_gen = ATOMIC64_INIT(1);
@@ -1258,6 +1259,15 @@ long kvm_arch_vm_ioctl(struct file *filp,
}
}
+#define kvm_hyp_percpu_base(cpu) ((unsigned long)per_cpu_ptr_nvhe(__per_cpu_start, cpu))
+#define kvm_hyp_percpu_array_size (nr_cpu_ids * sizeof(*kvm_arm_hyp_percpu_base))
+#define kvm_hyp_percpu_array_order (get_order(kvm_hyp_percpu_array_size))
+#define kvm_hyp_percpu_start (CHOOSE_NVHE_SYM(__per_cpu_start))
+#define kvm_hyp_percpu_end (CHOOSE_NVHE_SYM(__per_cpu_end))
+#define kvm_hyp_percpu_size ((unsigned long)kvm_hyp_percpu_end - \
+ (unsigned long)kvm_hyp_percpu_start)
+#define kvm_hyp_percpu_order (kvm_hyp_percpu_size ? get_order(kvm_hyp_percpu_size) : 0)
+
static void cpu_init_hyp_mode(void)
{
phys_addr_t pgd_ptr;
@@ -1273,8 +1283,8 @@ static void cpu_init_hyp_mode(void)
* kernel's mapping to the linear mapping, and store it in tpidr_el2
* so that we can use adr_l to access per-cpu variables in EL2.
*/
- tpidr_el2 = ((unsigned long)this_cpu_ptr(&kvm_host_data) -
- (unsigned long)kvm_ksym_ref(&kvm_host_data));
+ tpidr_el2 = (unsigned long)this_cpu_ptr_nvhe(__per_cpu_start) -
+ (unsigned long)kvm_ksym_ref(kvm_hyp_percpu_start);
pgd_ptr = kvm_mmu_get_httbr();
hyp_stack_ptr = __this_cpu_read(kvm_arm_hyp_stack_page) + PAGE_SIZE;
@@ -1506,8 +1516,11 @@ static void teardown_hyp_mode(void)
int cpu;
free_hyp_pgds();
- for_each_possible_cpu(cpu)
+ for_each_possible_cpu(cpu) {
free_hyp_page(per_cpu(kvm_arm_hyp_stack_page, cpu));
+ free_hyp_pages(kvm_hyp_percpu_base(cpu), kvm_hyp_percpu_order);
+ }
+ free_hyp_pages((unsigned long)kvm_arm_hyp_percpu_base, kvm_hyp_percpu_array_order);
}
/**
@@ -1540,6 +1553,28 @@ static int init_hyp_mode(void)
per_cpu(kvm_arm_hyp_stack_page, cpu) = stack_page;
}
+ /*
+ * Allocate and initialize pages for Hypervisor-mode percpu regions.
+ */
+ kvm_arm_hyp_percpu_base = (unsigned long *)alloc_hyp_pages(
+ GFP_KERNEL | __GFP_ZERO, kvm_hyp_percpu_array_order);
+ if (!kvm_arm_hyp_percpu_base) {
+ err = -ENOMEM;
+ goto out_err;
+ }
+ for_each_possible_cpu(cpu) {
+ unsigned long percpu_base;
+
+ percpu_base = alloc_hyp_pages(GFP_KERNEL, kvm_hyp_percpu_order);
+ if (!percpu_base) {
+ err = -ENOMEM;
+ goto out_err;
+ }
+
+ memcpy((void *)percpu_base, kvm_hyp_percpu_start, kvm_hyp_percpu_size);
+ kvm_arm_hyp_percpu_base[cpu] = percpu_base;
+ }
+
/*
* Map the Hyp-code called directly from the host
*/
@@ -1584,6 +1619,21 @@ static int init_hyp_mode(void)
}
}
+ /*
+ * Map Hyp percpu pages
+ */
+ for_each_possible_cpu(cpu) {
+ char *percpu_begin = (char *)kvm_arm_hyp_percpu_base[cpu];
+ char *percpu_end = percpu_begin + PAGE_ALIGN(kvm_hyp_percpu_size);
+
+ err = create_hyp_mappings(percpu_begin, percpu_end, PAGE_HYP);
+
+ if (err) {
+ kvm_err("Cannot map hyp percpu region\n");
+ goto out_err;
+ }
+ }
+
for_each_possible_cpu(cpu) {
kvm_host_data_t *cpu_data;
diff --git a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S
index 3b13d1c7cd1a..bb2d986ff696 100644
--- a/arch/arm64/kvm/hyp/nvhe/hyp.lds.S
+++ b/arch/arm64/kvm/hyp/nvhe/hyp.lds.S
@@ -7,7 +7,13 @@
*/
#include <asm/hyp_image.h>
+#include <asm-generic/vmlinux.lds.h>
+#include <asm/cache.h>
+#include <asm/memory.h>
SECTIONS {
HYP_SECTION(.text)
+ HYP_SECTION_NAME(.data..percpu) : {
+ PERCPU_INPUT(L1_CACHE_BYTES)
+ }
}
diff --git a/arch/arm64/kvm/pmu.c b/arch/arm64/kvm/pmu.c
index 6d80ffe1ebfc..cd653091f213 100644
--- a/arch/arm64/kvm/pmu.c
+++ b/arch/arm64/kvm/pmu.c
@@ -33,7 +33,7 @@ void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr)
{
struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data);
- if (!kvm_pmu_switch_needed(attr))
+ if (!ctx || !kvm_pmu_switch_needed(attr))
return;
if (!attr->exclude_host)
@@ -49,6 +49,9 @@ void kvm_clr_pmu_events(u32 clr)
{
struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data);
+ if (!ctx)
+ return;
+
ctx->pmu_events.events_host &= ~clr;
ctx->pmu_events.events_guest &= ~clr;
}
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply related [flat|nested] 87+ messages in thread* Re: [PATCH v3 10/11] kvm: arm64: Set up hyp percpu data for nVHE
2020-09-16 17:34 ` David Brazdil
(?)
@ 2020-09-18 12:22 ` Will Deacon
-1 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 12:22 UTC (permalink / raw)
To: David Brazdil
Cc: kvmarm, Catalin Marinas, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team
On Wed, Sep 16, 2020 at 06:34:38PM +0100, David Brazdil wrote:
> Add hyp percpu section to linker script and rename the corresponding ELF
> sections of hyp/nvhe object files. This moves all nVHE-specific percpu
> variables to the new hyp percpu section.
>
> Allocate sufficient amount of memory for all percpu hyp regions at global KVM
> init time and create corresponding hyp mappings.
>
> The base addresses of hyp percpu regions are kept in a dynamically allocated
> array in the kernel.
>
> Add NULL checks in PMU event-reset code as it may run before KVM memory is
> initialized.
>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/kvm_asm.h | 19 +++++++++--
> arch/arm64/include/asm/sections.h | 1 +
> arch/arm64/kernel/vmlinux.lds.S | 9 +++++
> arch/arm64/kvm/arm.c | 56 +++++++++++++++++++++++++++++--
> arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 6 ++++
> arch/arm64/kvm/pmu.c | 5 ++-
> 6 files changed, 90 insertions(+), 6 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
> index abc03f386b40..e0e1e404f6eb 100644
> --- a/arch/arm64/include/asm/kvm_asm.h
> +++ b/arch/arm64/include/asm/kvm_asm.h
> @@ -66,8 +66,23 @@
> #define CHOOSE_VHE_SYM(sym) sym
> #define CHOOSE_NVHE_SYM(sym) kvm_nvhe_sym(sym)
>
> -#define this_cpu_ptr_nvhe(sym) this_cpu_ptr(&kvm_nvhe_sym(sym))
> -#define per_cpu_ptr_nvhe(sym, cpu) per_cpu_ptr(&kvm_nvhe_sym(sym), cpu)
> +/* Array of percpu base addresses. Length of the array is nr_cpu_ids. */
> +extern unsigned long *kvm_arm_hyp_percpu_base;
> +
> +/*
> + * Compute pointer to a symbol defined in nVHE percpu region.
> + * Returns NULL if percpu memory has not been allocated yet.
> + */
> +#define this_cpu_ptr_nvhe(sym) per_cpu_ptr_nvhe(sym, smp_processor_id())
Don't you run into similar problems here with the pcpu accessors when
CONFIG_DEBUG_PREEMPT=y? I'm worried we can end up with an unresolved
reference to debug_smp_processor_id().
> +#define per_cpu_ptr_nvhe(sym, cpu) \
> + ({ \
> + unsigned long base, off; \
> + base = kvm_arm_hyp_percpu_base \
> + ? kvm_arm_hyp_percpu_base[cpu] : 0; \
> + off = (unsigned long)&kvm_nvhe_sym(sym) - \
> + (unsigned long)&kvm_nvhe_sym(__per_cpu_start); \
> + base ? (typeof(kvm_nvhe_sym(sym))*)(base + off) : NULL; \
> + })
>
> #ifndef __KVM_NVHE_HYPERVISOR__
> /*
> diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h
> index 3994169985ef..5062553a6847 100644
> --- a/arch/arm64/include/asm/sections.h
> +++ b/arch/arm64/include/asm/sections.h
> @@ -18,5 +18,6 @@ extern char __exittext_begin[], __exittext_end[];
> extern char __irqentry_text_start[], __irqentry_text_end[];
> extern char __mmuoff_data_start[], __mmuoff_data_end[];
> extern char __entry_tramp_text_start[], __entry_tramp_text_end[];
> +extern char __kvm_nvhe___per_cpu_start[], __kvm_nvhe___per_cpu_end[];
>
> #endif /* __ASM_SECTIONS_H */
> diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
> index 0d543570e811..d52e6b5dbfd3 100644
> --- a/arch/arm64/kernel/vmlinux.lds.S
> +++ b/arch/arm64/kernel/vmlinux.lds.S
> @@ -9,6 +9,7 @@
>
> #include <asm-generic/vmlinux.lds.h>
> #include <asm/cache.h>
> +#include <asm/hyp_image.h>
> #include <asm/kernel-pgtable.h>
> #include <asm/memory.h>
> #include <asm/page.h>
> @@ -27,8 +28,15 @@ jiffies = jiffies_64;
> __start___kvm_ex_table = .; \
> *(__kvm_ex_table) \
> __stop___kvm_ex_table = .;
> +
> +#define HYPERVISOR_PERCPU_SECTION \
> + . = ALIGN(PAGE_SIZE); \
> + HYP_SECTION_NAME(.data..percpu) : { \
> + *(HYP_SECTION_NAME(.data..percpu)) \
> + }
> #else /* CONFIG_KVM */
> #define HYPERVISOR_EXTABLE
> +#define HYPERVISOR_PERCPU_SECTION
> #endif
>
> #define HYPERVISOR_TEXT \
> @@ -194,6 +202,7 @@ SECTIONS
> }
>
> PERCPU_SECTION(L1_CACHE_BYTES)
> + HYPERVISOR_PERCPU_SECTION
>
> .rela.dyn : ALIGN(8) {
> *(.rela .rela*)
> diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> index 58d7d614519b..8293319a32e7 100644
> --- a/arch/arm64/kvm/arm.c
> +++ b/arch/arm64/kvm/arm.c
> @@ -47,6 +47,7 @@ __asm__(".arch_extension virt");
> #endif
>
> static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page);
> +unsigned long *kvm_arm_hyp_percpu_base;
>
> /* The VMID used in the VTTBR */
> static atomic64_t kvm_vmid_gen = ATOMIC64_INIT(1);
> @@ -1258,6 +1259,15 @@ long kvm_arch_vm_ioctl(struct file *filp,
> }
> }
>
> +#define kvm_hyp_percpu_base(cpu) ((unsigned long)per_cpu_ptr_nvhe(__per_cpu_start, cpu))
Having both kvm_arm_hyp_percpu_base and kvm_hyp_percpu_base be so
completely different is a recipe for disaster! Please can you rename
one/both of them to make it clearer what they represent?
> diff --git a/arch/arm64/kvm/pmu.c b/arch/arm64/kvm/pmu.c
> index 6d80ffe1ebfc..cd653091f213 100644
> --- a/arch/arm64/kvm/pmu.c
> +++ b/arch/arm64/kvm/pmu.c
> @@ -33,7 +33,7 @@ void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr)
> {
> struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data);
>
> - if (!kvm_pmu_switch_needed(attr))
> + if (!ctx || !kvm_pmu_switch_needed(attr))
> return;
>
> if (!attr->exclude_host)
> @@ -49,6 +49,9 @@ void kvm_clr_pmu_events(u32 clr)
> {
> struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data);
>
> + if (!ctx)
> + return;
I guess this should only happen if KVM failed to initialise or something,
right? (e.g. if we were booted at EL1). If so, I'm wondering whether it
would be better to do something like:
if (!is_hyp_mode_available())
return;
WARN_ON_ONCE(!ctx);
so that any unexpected NULL pointer there screams loudly, rather than causes
the register switch to be silently ignored. What do you think?
Will
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v3 10/11] kvm: arm64: Set up hyp percpu data for nVHE
@ 2020-09-18 12:22 ` Will Deacon
0 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 12:22 UTC (permalink / raw)
To: David Brazdil
Cc: Suzuki K Poulose, Catalin Marinas, linux-kernel, James Morse,
linux-arm-kernel, Marc Zyngier, Tejun Heo, Dennis Zhou,
Christoph Lameter, kernel-team, kvmarm, Julien Thierry
On Wed, Sep 16, 2020 at 06:34:38PM +0100, David Brazdil wrote:
> Add hyp percpu section to linker script and rename the corresponding ELF
> sections of hyp/nvhe object files. This moves all nVHE-specific percpu
> variables to the new hyp percpu section.
>
> Allocate sufficient amount of memory for all percpu hyp regions at global KVM
> init time and create corresponding hyp mappings.
>
> The base addresses of hyp percpu regions are kept in a dynamically allocated
> array in the kernel.
>
> Add NULL checks in PMU event-reset code as it may run before KVM memory is
> initialized.
>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/kvm_asm.h | 19 +++++++++--
> arch/arm64/include/asm/sections.h | 1 +
> arch/arm64/kernel/vmlinux.lds.S | 9 +++++
> arch/arm64/kvm/arm.c | 56 +++++++++++++++++++++++++++++--
> arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 6 ++++
> arch/arm64/kvm/pmu.c | 5 ++-
> 6 files changed, 90 insertions(+), 6 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
> index abc03f386b40..e0e1e404f6eb 100644
> --- a/arch/arm64/include/asm/kvm_asm.h
> +++ b/arch/arm64/include/asm/kvm_asm.h
> @@ -66,8 +66,23 @@
> #define CHOOSE_VHE_SYM(sym) sym
> #define CHOOSE_NVHE_SYM(sym) kvm_nvhe_sym(sym)
>
> -#define this_cpu_ptr_nvhe(sym) this_cpu_ptr(&kvm_nvhe_sym(sym))
> -#define per_cpu_ptr_nvhe(sym, cpu) per_cpu_ptr(&kvm_nvhe_sym(sym), cpu)
> +/* Array of percpu base addresses. Length of the array is nr_cpu_ids. */
> +extern unsigned long *kvm_arm_hyp_percpu_base;
> +
> +/*
> + * Compute pointer to a symbol defined in nVHE percpu region.
> + * Returns NULL if percpu memory has not been allocated yet.
> + */
> +#define this_cpu_ptr_nvhe(sym) per_cpu_ptr_nvhe(sym, smp_processor_id())
Don't you run into similar problems here with the pcpu accessors when
CONFIG_DEBUG_PREEMPT=y? I'm worried we can end up with an unresolved
reference to debug_smp_processor_id().
> +#define per_cpu_ptr_nvhe(sym, cpu) \
> + ({ \
> + unsigned long base, off; \
> + base = kvm_arm_hyp_percpu_base \
> + ? kvm_arm_hyp_percpu_base[cpu] : 0; \
> + off = (unsigned long)&kvm_nvhe_sym(sym) - \
> + (unsigned long)&kvm_nvhe_sym(__per_cpu_start); \
> + base ? (typeof(kvm_nvhe_sym(sym))*)(base + off) : NULL; \
> + })
>
> #ifndef __KVM_NVHE_HYPERVISOR__
> /*
> diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h
> index 3994169985ef..5062553a6847 100644
> --- a/arch/arm64/include/asm/sections.h
> +++ b/arch/arm64/include/asm/sections.h
> @@ -18,5 +18,6 @@ extern char __exittext_begin[], __exittext_end[];
> extern char __irqentry_text_start[], __irqentry_text_end[];
> extern char __mmuoff_data_start[], __mmuoff_data_end[];
> extern char __entry_tramp_text_start[], __entry_tramp_text_end[];
> +extern char __kvm_nvhe___per_cpu_start[], __kvm_nvhe___per_cpu_end[];
>
> #endif /* __ASM_SECTIONS_H */
> diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
> index 0d543570e811..d52e6b5dbfd3 100644
> --- a/arch/arm64/kernel/vmlinux.lds.S
> +++ b/arch/arm64/kernel/vmlinux.lds.S
> @@ -9,6 +9,7 @@
>
> #include <asm-generic/vmlinux.lds.h>
> #include <asm/cache.h>
> +#include <asm/hyp_image.h>
> #include <asm/kernel-pgtable.h>
> #include <asm/memory.h>
> #include <asm/page.h>
> @@ -27,8 +28,15 @@ jiffies = jiffies_64;
> __start___kvm_ex_table = .; \
> *(__kvm_ex_table) \
> __stop___kvm_ex_table = .;
> +
> +#define HYPERVISOR_PERCPU_SECTION \
> + . = ALIGN(PAGE_SIZE); \
> + HYP_SECTION_NAME(.data..percpu) : { \
> + *(HYP_SECTION_NAME(.data..percpu)) \
> + }
> #else /* CONFIG_KVM */
> #define HYPERVISOR_EXTABLE
> +#define HYPERVISOR_PERCPU_SECTION
> #endif
>
> #define HYPERVISOR_TEXT \
> @@ -194,6 +202,7 @@ SECTIONS
> }
>
> PERCPU_SECTION(L1_CACHE_BYTES)
> + HYPERVISOR_PERCPU_SECTION
>
> .rela.dyn : ALIGN(8) {
> *(.rela .rela*)
> diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> index 58d7d614519b..8293319a32e7 100644
> --- a/arch/arm64/kvm/arm.c
> +++ b/arch/arm64/kvm/arm.c
> @@ -47,6 +47,7 @@ __asm__(".arch_extension virt");
> #endif
>
> static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page);
> +unsigned long *kvm_arm_hyp_percpu_base;
>
> /* The VMID used in the VTTBR */
> static atomic64_t kvm_vmid_gen = ATOMIC64_INIT(1);
> @@ -1258,6 +1259,15 @@ long kvm_arch_vm_ioctl(struct file *filp,
> }
> }
>
> +#define kvm_hyp_percpu_base(cpu) ((unsigned long)per_cpu_ptr_nvhe(__per_cpu_start, cpu))
Having both kvm_arm_hyp_percpu_base and kvm_hyp_percpu_base be so
completely different is a recipe for disaster! Please can you rename
one/both of them to make it clearer what they represent?
> diff --git a/arch/arm64/kvm/pmu.c b/arch/arm64/kvm/pmu.c
> index 6d80ffe1ebfc..cd653091f213 100644
> --- a/arch/arm64/kvm/pmu.c
> +++ b/arch/arm64/kvm/pmu.c
> @@ -33,7 +33,7 @@ void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr)
> {
> struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data);
>
> - if (!kvm_pmu_switch_needed(attr))
> + if (!ctx || !kvm_pmu_switch_needed(attr))
> return;
>
> if (!attr->exclude_host)
> @@ -49,6 +49,9 @@ void kvm_clr_pmu_events(u32 clr)
> {
> struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data);
>
> + if (!ctx)
> + return;
I guess this should only happen if KVM failed to initialise or something,
right? (e.g. if we were booted at EL1). If so, I'm wondering whether it
would be better to do something like:
if (!is_hyp_mode_available())
return;
WARN_ON_ONCE(!ctx);
so that any unexpected NULL pointer there screams loudly, rather than causes
the register switch to be silently ignored. What do you think?
Will
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v3 10/11] kvm: arm64: Set up hyp percpu data for nVHE
@ 2020-09-18 12:22 ` Will Deacon
0 siblings, 0 replies; 87+ messages in thread
From: Will Deacon @ 2020-09-18 12:22 UTC (permalink / raw)
To: David Brazdil
Cc: Catalin Marinas, linux-kernel, linux-arm-kernel, Marc Zyngier,
Tejun Heo, Dennis Zhou, Christoph Lameter, kernel-team, kvmarm
On Wed, Sep 16, 2020 at 06:34:38PM +0100, David Brazdil wrote:
> Add hyp percpu section to linker script and rename the corresponding ELF
> sections of hyp/nvhe object files. This moves all nVHE-specific percpu
> variables to the new hyp percpu section.
>
> Allocate sufficient amount of memory for all percpu hyp regions at global KVM
> init time and create corresponding hyp mappings.
>
> The base addresses of hyp percpu regions are kept in a dynamically allocated
> array in the kernel.
>
> Add NULL checks in PMU event-reset code as it may run before KVM memory is
> initialized.
>
> Signed-off-by: David Brazdil <dbrazdil@google.com>
> ---
> arch/arm64/include/asm/kvm_asm.h | 19 +++++++++--
> arch/arm64/include/asm/sections.h | 1 +
> arch/arm64/kernel/vmlinux.lds.S | 9 +++++
> arch/arm64/kvm/arm.c | 56 +++++++++++++++++++++++++++++--
> arch/arm64/kvm/hyp/nvhe/hyp.lds.S | 6 ++++
> arch/arm64/kvm/pmu.c | 5 ++-
> 6 files changed, 90 insertions(+), 6 deletions(-)
>
> diff --git a/arch/arm64/include/asm/kvm_asm.h b/arch/arm64/include/asm/kvm_asm.h
> index abc03f386b40..e0e1e404f6eb 100644
> --- a/arch/arm64/include/asm/kvm_asm.h
> +++ b/arch/arm64/include/asm/kvm_asm.h
> @@ -66,8 +66,23 @@
> #define CHOOSE_VHE_SYM(sym) sym
> #define CHOOSE_NVHE_SYM(sym) kvm_nvhe_sym(sym)
>
> -#define this_cpu_ptr_nvhe(sym) this_cpu_ptr(&kvm_nvhe_sym(sym))
> -#define per_cpu_ptr_nvhe(sym, cpu) per_cpu_ptr(&kvm_nvhe_sym(sym), cpu)
> +/* Array of percpu base addresses. Length of the array is nr_cpu_ids. */
> +extern unsigned long *kvm_arm_hyp_percpu_base;
> +
> +/*
> + * Compute pointer to a symbol defined in nVHE percpu region.
> + * Returns NULL if percpu memory has not been allocated yet.
> + */
> +#define this_cpu_ptr_nvhe(sym) per_cpu_ptr_nvhe(sym, smp_processor_id())
Don't you run into similar problems here with the pcpu accessors when
CONFIG_DEBUG_PREEMPT=y? I'm worried we can end up with an unresolved
reference to debug_smp_processor_id().
> +#define per_cpu_ptr_nvhe(sym, cpu) \
> + ({ \
> + unsigned long base, off; \
> + base = kvm_arm_hyp_percpu_base \
> + ? kvm_arm_hyp_percpu_base[cpu] : 0; \
> + off = (unsigned long)&kvm_nvhe_sym(sym) - \
> + (unsigned long)&kvm_nvhe_sym(__per_cpu_start); \
> + base ? (typeof(kvm_nvhe_sym(sym))*)(base + off) : NULL; \
> + })
>
> #ifndef __KVM_NVHE_HYPERVISOR__
> /*
> diff --git a/arch/arm64/include/asm/sections.h b/arch/arm64/include/asm/sections.h
> index 3994169985ef..5062553a6847 100644
> --- a/arch/arm64/include/asm/sections.h
> +++ b/arch/arm64/include/asm/sections.h
> @@ -18,5 +18,6 @@ extern char __exittext_begin[], __exittext_end[];
> extern char __irqentry_text_start[], __irqentry_text_end[];
> extern char __mmuoff_data_start[], __mmuoff_data_end[];
> extern char __entry_tramp_text_start[], __entry_tramp_text_end[];
> +extern char __kvm_nvhe___per_cpu_start[], __kvm_nvhe___per_cpu_end[];
>
> #endif /* __ASM_SECTIONS_H */
> diff --git a/arch/arm64/kernel/vmlinux.lds.S b/arch/arm64/kernel/vmlinux.lds.S
> index 0d543570e811..d52e6b5dbfd3 100644
> --- a/arch/arm64/kernel/vmlinux.lds.S
> +++ b/arch/arm64/kernel/vmlinux.lds.S
> @@ -9,6 +9,7 @@
>
> #include <asm-generic/vmlinux.lds.h>
> #include <asm/cache.h>
> +#include <asm/hyp_image.h>
> #include <asm/kernel-pgtable.h>
> #include <asm/memory.h>
> #include <asm/page.h>
> @@ -27,8 +28,15 @@ jiffies = jiffies_64;
> __start___kvm_ex_table = .; \
> *(__kvm_ex_table) \
> __stop___kvm_ex_table = .;
> +
> +#define HYPERVISOR_PERCPU_SECTION \
> + . = ALIGN(PAGE_SIZE); \
> + HYP_SECTION_NAME(.data..percpu) : { \
> + *(HYP_SECTION_NAME(.data..percpu)) \
> + }
> #else /* CONFIG_KVM */
> #define HYPERVISOR_EXTABLE
> +#define HYPERVISOR_PERCPU_SECTION
> #endif
>
> #define HYPERVISOR_TEXT \
> @@ -194,6 +202,7 @@ SECTIONS
> }
>
> PERCPU_SECTION(L1_CACHE_BYTES)
> + HYPERVISOR_PERCPU_SECTION
>
> .rela.dyn : ALIGN(8) {
> *(.rela .rela*)
> diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> index 58d7d614519b..8293319a32e7 100644
> --- a/arch/arm64/kvm/arm.c
> +++ b/arch/arm64/kvm/arm.c
> @@ -47,6 +47,7 @@ __asm__(".arch_extension virt");
> #endif
>
> static DEFINE_PER_CPU(unsigned long, kvm_arm_hyp_stack_page);
> +unsigned long *kvm_arm_hyp_percpu_base;
>
> /* The VMID used in the VTTBR */
> static atomic64_t kvm_vmid_gen = ATOMIC64_INIT(1);
> @@ -1258,6 +1259,15 @@ long kvm_arch_vm_ioctl(struct file *filp,
> }
> }
>
> +#define kvm_hyp_percpu_base(cpu) ((unsigned long)per_cpu_ptr_nvhe(__per_cpu_start, cpu))
Having both kvm_arm_hyp_percpu_base and kvm_hyp_percpu_base be so
completely different is a recipe for disaster! Please can you rename
one/both of them to make it clearer what they represent?
> diff --git a/arch/arm64/kvm/pmu.c b/arch/arm64/kvm/pmu.c
> index 6d80ffe1ebfc..cd653091f213 100644
> --- a/arch/arm64/kvm/pmu.c
> +++ b/arch/arm64/kvm/pmu.c
> @@ -33,7 +33,7 @@ void kvm_set_pmu_events(u32 set, struct perf_event_attr *attr)
> {
> struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data);
>
> - if (!kvm_pmu_switch_needed(attr))
> + if (!ctx || !kvm_pmu_switch_needed(attr))
> return;
>
> if (!attr->exclude_host)
> @@ -49,6 +49,9 @@ void kvm_clr_pmu_events(u32 clr)
> {
> struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data);
>
> + if (!ctx)
> + return;
I guess this should only happen if KVM failed to initialise or something,
right? (e.g. if we were booted at EL1). If so, I'm wondering whether it
would be better to do something like:
if (!is_hyp_mode_available())
return;
WARN_ON_ONCE(!ctx);
so that any unexpected NULL pointer there screams loudly, rather than causes
the register switch to be silently ignored. What do you think?
Will
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v3 10/11] kvm: arm64: Set up hyp percpu data for nVHE
2020-09-18 12:22 ` Will Deacon
(?)
@ 2020-09-22 18:34 ` David Brazdil
-1 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-22 18:34 UTC (permalink / raw)
To: Will Deacon
Cc: kvmarm, Catalin Marinas, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team
> > -#define this_cpu_ptr_nvhe(sym) this_cpu_ptr(&kvm_nvhe_sym(sym))
> > -#define per_cpu_ptr_nvhe(sym, cpu) per_cpu_ptr(&kvm_nvhe_sym(sym), cpu)
> > +/* Array of percpu base addresses. Length of the array is nr_cpu_ids. */
> > +extern unsigned long *kvm_arm_hyp_percpu_base;
> > +
> > +/*
> > + * Compute pointer to a symbol defined in nVHE percpu region.
> > + * Returns NULL if percpu memory has not been allocated yet.
> > + */
> > +#define this_cpu_ptr_nvhe(sym) per_cpu_ptr_nvhe(sym, smp_processor_id())
>
> Don't you run into similar problems here with the pcpu accessors when
> CONFIG_DEBUG_PREEMPT=y? I'm worried we can end up with an unresolved
> reference to debug_smp_processor_id().
Fortunately not. This now doesn't use the generic macros at all.
> > /* The VMID used in the VTTBR */
> > static atomic64_t kvm_vmid_gen = ATOMIC64_INIT(1);
> > @@ -1258,6 +1259,15 @@ long kvm_arch_vm_ioctl(struct file *filp,
> > }
> > }
> >
> > +#define kvm_hyp_percpu_base(cpu) ((unsigned long)per_cpu_ptr_nvhe(__per_cpu_start, cpu))
>
> Having both kvm_arm_hyp_percpu_base and kvm_hyp_percpu_base be so
> completely different is a recipe for disaster! Please can you rename
> one/both of them to make it clearer what they represent?
I am heavily simplifying this code in v4. Got rid of this macro completely, so
hopefully there will be no confusion.
> > - if (!kvm_pmu_switch_needed(attr))
> > + if (!ctx || !kvm_pmu_switch_needed(attr))
> > return;
> >
> > if (!attr->exclude_host)
> > @@ -49,6 +49,9 @@ void kvm_clr_pmu_events(u32 clr)
> > {
> > struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data);
> >
> > + if (!ctx)
> > + return;
>
> I guess this should only happen if KVM failed to initialise or something,
> right? (e.g. if we were booted at EL1). If so, I'm wondering whether it
> would be better to do something like:
>
> if (!is_hyp_mode_available())
> return;
>
> WARN_ON_ONCE(!ctx);
>
> so that any unexpected NULL pointer there screams loudly, rather than causes
> the register switch to be silently ignored. What do you think?
Unfortunately, this happens on every boot. I don't fully understand how the
boot order is determined, so please correct me if this makes no sense, but
kvm_clr_pmu_events is called as part of CPUHP_AP_PERF_ARM_STARTING. The first
time that happens is before KVM initialized (tested from inserting
BUG_ON(!ctx)). That's not a problem, the per-CPU memory is there and it's all
zeroes. It becomes a problem with this patch because the per-CPU memory is not
there *yet*.
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v3 10/11] kvm: arm64: Set up hyp percpu data for nVHE
@ 2020-09-22 18:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-22 18:34 UTC (permalink / raw)
To: Will Deacon
Cc: Suzuki K Poulose, Catalin Marinas, linux-kernel, James Morse,
linux-arm-kernel, Marc Zyngier, Tejun Heo, Dennis Zhou,
Christoph Lameter, kernel-team, kvmarm, Julien Thierry
> > -#define this_cpu_ptr_nvhe(sym) this_cpu_ptr(&kvm_nvhe_sym(sym))
> > -#define per_cpu_ptr_nvhe(sym, cpu) per_cpu_ptr(&kvm_nvhe_sym(sym), cpu)
> > +/* Array of percpu base addresses. Length of the array is nr_cpu_ids. */
> > +extern unsigned long *kvm_arm_hyp_percpu_base;
> > +
> > +/*
> > + * Compute pointer to a symbol defined in nVHE percpu region.
> > + * Returns NULL if percpu memory has not been allocated yet.
> > + */
> > +#define this_cpu_ptr_nvhe(sym) per_cpu_ptr_nvhe(sym, smp_processor_id())
>
> Don't you run into similar problems here with the pcpu accessors when
> CONFIG_DEBUG_PREEMPT=y? I'm worried we can end up with an unresolved
> reference to debug_smp_processor_id().
Fortunately not. This now doesn't use the generic macros at all.
> > /* The VMID used in the VTTBR */
> > static atomic64_t kvm_vmid_gen = ATOMIC64_INIT(1);
> > @@ -1258,6 +1259,15 @@ long kvm_arch_vm_ioctl(struct file *filp,
> > }
> > }
> >
> > +#define kvm_hyp_percpu_base(cpu) ((unsigned long)per_cpu_ptr_nvhe(__per_cpu_start, cpu))
>
> Having both kvm_arm_hyp_percpu_base and kvm_hyp_percpu_base be so
> completely different is a recipe for disaster! Please can you rename
> one/both of them to make it clearer what they represent?
I am heavily simplifying this code in v4. Got rid of this macro completely, so
hopefully there will be no confusion.
> > - if (!kvm_pmu_switch_needed(attr))
> > + if (!ctx || !kvm_pmu_switch_needed(attr))
> > return;
> >
> > if (!attr->exclude_host)
> > @@ -49,6 +49,9 @@ void kvm_clr_pmu_events(u32 clr)
> > {
> > struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data);
> >
> > + if (!ctx)
> > + return;
>
> I guess this should only happen if KVM failed to initialise or something,
> right? (e.g. if we were booted at EL1). If so, I'm wondering whether it
> would be better to do something like:
>
> if (!is_hyp_mode_available())
> return;
>
> WARN_ON_ONCE(!ctx);
>
> so that any unexpected NULL pointer there screams loudly, rather than causes
> the register switch to be silently ignored. What do you think?
Unfortunately, this happens on every boot. I don't fully understand how the
boot order is determined, so please correct me if this makes no sense, but
kvm_clr_pmu_events is called as part of CPUHP_AP_PERF_ARM_STARTING. The first
time that happens is before KVM initialized (tested from inserting
BUG_ON(!ctx)). That's not a problem, the per-CPU memory is there and it's all
zeroes. It becomes a problem with this patch because the per-CPU memory is not
there *yet*.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply [flat|nested] 87+ messages in thread* Re: [PATCH v3 10/11] kvm: arm64: Set up hyp percpu data for nVHE
@ 2020-09-22 18:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-22 18:34 UTC (permalink / raw)
To: Will Deacon
Cc: Catalin Marinas, linux-kernel, linux-arm-kernel, Marc Zyngier,
Tejun Heo, Dennis Zhou, Christoph Lameter, kernel-team, kvmarm
> > -#define this_cpu_ptr_nvhe(sym) this_cpu_ptr(&kvm_nvhe_sym(sym))
> > -#define per_cpu_ptr_nvhe(sym, cpu) per_cpu_ptr(&kvm_nvhe_sym(sym), cpu)
> > +/* Array of percpu base addresses. Length of the array is nr_cpu_ids. */
> > +extern unsigned long *kvm_arm_hyp_percpu_base;
> > +
> > +/*
> > + * Compute pointer to a symbol defined in nVHE percpu region.
> > + * Returns NULL if percpu memory has not been allocated yet.
> > + */
> > +#define this_cpu_ptr_nvhe(sym) per_cpu_ptr_nvhe(sym, smp_processor_id())
>
> Don't you run into similar problems here with the pcpu accessors when
> CONFIG_DEBUG_PREEMPT=y? I'm worried we can end up with an unresolved
> reference to debug_smp_processor_id().
Fortunately not. This now doesn't use the generic macros at all.
> > /* The VMID used in the VTTBR */
> > static atomic64_t kvm_vmid_gen = ATOMIC64_INIT(1);
> > @@ -1258,6 +1259,15 @@ long kvm_arch_vm_ioctl(struct file *filp,
> > }
> > }
> >
> > +#define kvm_hyp_percpu_base(cpu) ((unsigned long)per_cpu_ptr_nvhe(__per_cpu_start, cpu))
>
> Having both kvm_arm_hyp_percpu_base and kvm_hyp_percpu_base be so
> completely different is a recipe for disaster! Please can you rename
> one/both of them to make it clearer what they represent?
I am heavily simplifying this code in v4. Got rid of this macro completely, so
hopefully there will be no confusion.
> > - if (!kvm_pmu_switch_needed(attr))
> > + if (!ctx || !kvm_pmu_switch_needed(attr))
> > return;
> >
> > if (!attr->exclude_host)
> > @@ -49,6 +49,9 @@ void kvm_clr_pmu_events(u32 clr)
> > {
> > struct kvm_host_data *ctx = this_cpu_ptr_hyp(kvm_host_data);
> >
> > + if (!ctx)
> > + return;
>
> I guess this should only happen if KVM failed to initialise or something,
> right? (e.g. if we were booted at EL1). If so, I'm wondering whether it
> would be better to do something like:
>
> if (!is_hyp_mode_available())
> return;
>
> WARN_ON_ONCE(!ctx);
>
> so that any unexpected NULL pointer there screams loudly, rather than causes
> the register switch to be silently ignored. What do you think?
Unfortunately, this happens on every boot. I don't fully understand how the
boot order is determined, so please correct me if this makes no sense, but
kvm_clr_pmu_events is called as part of CPUHP_AP_PERF_ARM_STARTING. The first
time that happens is before KVM initialized (tested from inserting
BUG_ON(!ctx)). That's not a problem, the per-CPU memory is there and it's all
zeroes. It becomes a problem with this patch because the per-CPU memory is not
there *yet*.
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply [flat|nested] 87+ messages in thread
* [PATCH v3 11/11] kvm: arm64: Remove unnecessary hyp mappings
2020-09-16 17:34 ` David Brazdil
(?)
@ 2020-09-16 17:34 ` David Brazdil
-1 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: Catalin Marinas, Will Deacon, Marc Zyngier, James Morse,
Julien Thierry, Suzuki K Poulose, Dennis Zhou, Tejun Heo,
Christoph Lameter, linux-arm-kernel, linux-kernel, kernel-team,
David Brazdil, Andrew Scull
With all nVHE per-CPU variables being part of the hyp per-CPU region,
mapping them individual is not necessary any longer. They are mapped to hyp
as part of the overall per-CPU region.
Acked-by: Andrew Scull<ascull@google.com>
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/kvm_mmu.h | 19 +++++--------------
arch/arm64/kvm/arm.c | 17 +----------------
2 files changed, 6 insertions(+), 30 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 9db93da35606..7d1581bc8b97 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -531,28 +531,19 @@ static inline int kvm_map_vectors(void)
DECLARE_PER_CPU_READ_MOSTLY(u64, arm64_ssbd_callback_required);
DECLARE_KVM_NVHE_PER_CPU(u64, arm64_ssbd_callback_required);
-static inline int hyp_init_aux_data(void)
+static inline void hyp_init_aux_data(void)
{
- int cpu, err;
+ int cpu;
+ /* Copy arm64_ssbd_callback_required values from kernel to hyp. */
for_each_possible_cpu(cpu) {
- u64 *ptr;
+ u64 *ptr = per_cpu_ptr_nvhe(arm64_ssbd_callback_required, cpu);
- ptr = per_cpu_ptr_nvhe(arm64_ssbd_callback_required, cpu);
- err = create_hyp_mappings(ptr, ptr + 1, PAGE_HYP);
- if (err)
- return err;
-
- /* Copy value from kernel to hyp. */
*ptr = per_cpu(arm64_ssbd_callback_required, cpu);
}
- return 0;
}
#else
-static inline int hyp_init_aux_data(void)
-{
- return 0;
-}
+static inline void hyp_init_aux_data(void) {}
#endif
#define kvm_phys_to_vttbr(addr) phys_to_ttbr(addr)
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 8293319a32e7..4483b6aca5d7 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -1634,22 +1634,7 @@ static int init_hyp_mode(void)
}
}
- for_each_possible_cpu(cpu) {
- kvm_host_data_t *cpu_data;
-
- cpu_data = per_cpu_ptr_hyp(kvm_host_data, cpu);
- err = create_hyp_mappings(cpu_data, cpu_data + 1, PAGE_HYP);
-
- if (err) {
- kvm_err("Cannot map host CPU state: %d\n", err);
- goto out_err;
- }
- }
-
- err = hyp_init_aux_data();
- if (err)
- kvm_err("Cannot map host auxiliary data: %d\n", err);
-
+ hyp_init_aux_data();
return 0;
out_err:
--
2.28.0.618.gf4bc123cb7-goog
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v3 11/11] kvm: arm64: Remove unnecessary hyp mappings
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Suzuki K Poulose, Catalin Marinas, linux-kernel,
James Morse, linux-arm-kernel, Marc Zyngier, Tejun Heo,
Dennis Zhou, Christoph Lameter, David Brazdil, Will Deacon,
Julien Thierry, Andrew Scull
With all nVHE per-CPU variables being part of the hyp per-CPU region,
mapping them individual is not necessary any longer. They are mapped to hyp
as part of the overall per-CPU region.
Acked-by: Andrew Scull<ascull@google.com>
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/kvm_mmu.h | 19 +++++--------------
arch/arm64/kvm/arm.c | 17 +----------------
2 files changed, 6 insertions(+), 30 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 9db93da35606..7d1581bc8b97 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -531,28 +531,19 @@ static inline int kvm_map_vectors(void)
DECLARE_PER_CPU_READ_MOSTLY(u64, arm64_ssbd_callback_required);
DECLARE_KVM_NVHE_PER_CPU(u64, arm64_ssbd_callback_required);
-static inline int hyp_init_aux_data(void)
+static inline void hyp_init_aux_data(void)
{
- int cpu, err;
+ int cpu;
+ /* Copy arm64_ssbd_callback_required values from kernel to hyp. */
for_each_possible_cpu(cpu) {
- u64 *ptr;
+ u64 *ptr = per_cpu_ptr_nvhe(arm64_ssbd_callback_required, cpu);
- ptr = per_cpu_ptr_nvhe(arm64_ssbd_callback_required, cpu);
- err = create_hyp_mappings(ptr, ptr + 1, PAGE_HYP);
- if (err)
- return err;
-
- /* Copy value from kernel to hyp. */
*ptr = per_cpu(arm64_ssbd_callback_required, cpu);
}
- return 0;
}
#else
-static inline int hyp_init_aux_data(void)
-{
- return 0;
-}
+static inline void hyp_init_aux_data(void) {}
#endif
#define kvm_phys_to_vttbr(addr) phys_to_ttbr(addr)
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 8293319a32e7..4483b6aca5d7 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -1634,22 +1634,7 @@ static int init_hyp_mode(void)
}
}
- for_each_possible_cpu(cpu) {
- kvm_host_data_t *cpu_data;
-
- cpu_data = per_cpu_ptr_hyp(kvm_host_data, cpu);
- err = create_hyp_mappings(cpu_data, cpu_data + 1, PAGE_HYP);
-
- if (err) {
- kvm_err("Cannot map host CPU state: %d\n", err);
- goto out_err;
- }
- }
-
- err = hyp_init_aux_data();
- if (err)
- kvm_err("Cannot map host auxiliary data: %d\n", err);
-
+ hyp_init_aux_data();
return 0;
out_err:
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
^ permalink raw reply related [flat|nested] 87+ messages in thread* [PATCH v3 11/11] kvm: arm64: Remove unnecessary hyp mappings
@ 2020-09-16 17:34 ` David Brazdil
0 siblings, 0 replies; 87+ messages in thread
From: David Brazdil @ 2020-09-16 17:34 UTC (permalink / raw)
To: kvmarm
Cc: kernel-team, Catalin Marinas, linux-kernel, linux-arm-kernel,
Marc Zyngier, Tejun Heo, Dennis Zhou, Christoph Lameter,
Will Deacon
With all nVHE per-CPU variables being part of the hyp per-CPU region,
mapping them individual is not necessary any longer. They are mapped to hyp
as part of the overall per-CPU region.
Acked-by: Andrew Scull<ascull@google.com>
Signed-off-by: David Brazdil <dbrazdil@google.com>
---
arch/arm64/include/asm/kvm_mmu.h | 19 +++++--------------
arch/arm64/kvm/arm.c | 17 +----------------
2 files changed, 6 insertions(+), 30 deletions(-)
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 9db93da35606..7d1581bc8b97 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -531,28 +531,19 @@ static inline int kvm_map_vectors(void)
DECLARE_PER_CPU_READ_MOSTLY(u64, arm64_ssbd_callback_required);
DECLARE_KVM_NVHE_PER_CPU(u64, arm64_ssbd_callback_required);
-static inline int hyp_init_aux_data(void)
+static inline void hyp_init_aux_data(void)
{
- int cpu, err;
+ int cpu;
+ /* Copy arm64_ssbd_callback_required values from kernel to hyp. */
for_each_possible_cpu(cpu) {
- u64 *ptr;
+ u64 *ptr = per_cpu_ptr_nvhe(arm64_ssbd_callback_required, cpu);
- ptr = per_cpu_ptr_nvhe(arm64_ssbd_callback_required, cpu);
- err = create_hyp_mappings(ptr, ptr + 1, PAGE_HYP);
- if (err)
- return err;
-
- /* Copy value from kernel to hyp. */
*ptr = per_cpu(arm64_ssbd_callback_required, cpu);
}
- return 0;
}
#else
-static inline int hyp_init_aux_data(void)
-{
- return 0;
-}
+static inline void hyp_init_aux_data(void) {}
#endif
#define kvm_phys_to_vttbr(addr) phys_to_ttbr(addr)
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index 8293319a32e7..4483b6aca5d7 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -1634,22 +1634,7 @@ static int init_hyp_mode(void)
}
}
- for_each_possible_cpu(cpu) {
- kvm_host_data_t *cpu_data;
-
- cpu_data = per_cpu_ptr_hyp(kvm_host_data, cpu);
- err = create_hyp_mappings(cpu_data, cpu_data + 1, PAGE_HYP);
-
- if (err) {
- kvm_err("Cannot map host CPU state: %d\n", err);
- goto out_err;
- }
- }
-
- err = hyp_init_aux_data();
- if (err)
- kvm_err("Cannot map host auxiliary data: %d\n", err);
-
+ hyp_init_aux_data();
return 0;
out_err:
--
2.28.0.618.gf4bc123cb7-goog
_______________________________________________
kvmarm mailing list
kvmarm@lists.cs.columbia.edu
https://lists.cs.columbia.edu/mailman/listinfo/kvmarm
^ permalink raw reply related [flat|nested] 87+ messages in thread