All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v6 00/26] KVM/arm64: Randomise EL2 mappings (variant 3a mitigation)
@ 2018-03-14 16:50 ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Catalin Marinas, Will Deacon, Kristina Martsenko

Whilst KVM benefits from the kernel randomisation via KASLR, there is
no additional randomisation when the kernel is running at EL1, as we
directly use a fixed offset from the linear mapping. This is not
necessarily a problem, but we could do a bit better by independently
randomizing the HYP placement.

This series proposes to randomise the offset by inserting a few random
bits between the MSB of the RAM linear mapping and the top of the HYP
VA (VA_BITS - 2). That's not a lot of random bits (on my Mustang, I
get 13 bits), but that's better than nothing.

In order to achieve this, we need to be able to patch dynamic values
in the kernel text. This results in a bunch of changes to the
alternative framework, the insn library, and a few more hacks in KVM
itself (we get a new way to map the GIC at EL2).

Another (and more recent) goal of this series is to work around what
has been described as "variant 3a", which covers speculative reads of
privileged system registers. Randomizing the location of the
hypervisor would be pointless if one could simply obtain VBAR_EL2. In
order to work around this, we place the vectors at a fairly static
location (next to the idmap), independently of the hypervisor's own
mappings. This ensures that we can leak VBAR_EL2 without disclosing
much about HYP itself (and is similar to what the rest of the kernel
does with KPTI). This is only enabled at runtime for Cortex-A57 and
Cortex-A72.

This has been tested on the FVP model, Seattle (both 39 and 48bit VA),
Mustang and Thunder-X. I've also done a sanity check on 32bit (which
is only impacted by the HYP IO VA stuff).

Thanks,

	M.

* From v5:
  - Countless fixes, thanks to James excellent review
  - Additional patch to address 52bit PA HYP idmap unmap
  - Additional patch for HYP idmap page alignment/size
  - Additional patch to move the BP hardening slots into .hyp.text
  - New and improved far branch into the vectors (shorter epilogue)
  - Some extra tidying up thanks to Drew's review
  - Collected AB/RB tags

* From v4:
  - Added some more patches to work around speculative reads of
    VBAR_EL2
  - Bunch of cleanups and clarifications thanks to Christoffer's review
  - Dropped the initial asm-offsets rework on which this series
    didn't really rely anymore

* From v3:
  - Reworked the alternative code to leave the actual patching to
    the callback function. This should allow for more flexibility
    should someone or something require it
  - Now detects underflows in the IOVA allocator
  - Moved the VA patching code to va_layout.c

* From v2:
  - Fixed a crapload of bugs in the immediate generation patch
    I now have a test harness for it, making sure it generates the
    same thing as GAS...
  - Fixed a bug in the asm-offsets.h exclusion patch
  - Reworked the alternative_cb code to be nicer and avoid generating
    pointless nops

* From v1:
  - Now works correctly with KASLR
  - Dropped the callback field from alt_instr, and reuse one of the
    existing fields to store an offset to the callback
  - Fix HYP teardown path (depends on fixes previously posted)
  - Dropped the VA offset macros


Marc Zyngier (26):
  arm64: alternatives: Add dynamic patching feature
  arm64: insn: Add N immediate encoding
  arm64: insn: Add encoder for bitwise operations using literals
  arm64: KVM: Dynamically patch the kernel/hyp VA mask
  arm64: cpufeatures: Drop the ARM64_HYP_OFFSET_LOW feature flag
  KVM: arm/arm64: Do not use kern_hyp_va() with kvm_vgic_global_state
  KVM: arm/arm64: Demote HYP VA range display to being a debug feature
  KVM: arm/arm64: Move ioremap calls to create_hyp_io_mappings
  KVM: arm/arm64: Keep GICv2 HYP VAs in kvm_vgic_global_state
  KVM: arm/arm64: Fix idmap size and alignment
  KVM: arm64: Fix HYP idmap unmap when using 52bit PA
  KVM: arm/arm64: Move HYP IO VAs to the "idmap" range
  arm64; insn: Add encoder for the EXTR instruction
  arm64: insn: Allow ADD/SUB (immediate) with LSL #12
  arm64: KVM: Dynamically compute the HYP VA mask
  arm64: KVM: Introduce EL2 VA randomisation
  arm64: Update the KVM memory map documentation
  arm64: KVM: Move vector offsetting from hyp-init.S to
    kvm_get_hyp_vector
  arm64: KVM: Move stashing of x0/x1 into the vector code itself
  arm64: KVM: Move BP hardening vectors into .hyp.text section
  arm64: KVM: Reserve 4 additional instructions in the BPI template
  arm64: KVM: Allow far branches from vector slots to the main vectors
  arm/arm64: KVM: Introduce EL2-specific executable mappings
  arm64: Make BP hardening slot counter available
  arm64: KVM: Allow mapping of vectors outside of the RAM region
  arm64: Enable ARM64_HARDEN_EL2_VECTORS on Cortex-A57 and A72

 Documentation/arm64/memory.txt       |   9 +-
 arch/arm/include/asm/kvm_mmu.h       |  16 ++-
 arch/arm64/Kconfig                   |  16 +++
 arch/arm64/include/asm/alternative.h |  41 ++++++-
 arch/arm64/include/asm/cpucaps.h     |   2 +-
 arch/arm64/include/asm/insn.h        |  16 +++
 arch/arm64/include/asm/kvm_mmu.h     | 163 ++++++++++++++++++-------
 arch/arm64/include/asm/mmu.h         |   8 +-
 arch/arm64/kernel/Makefile           |   4 +-
 arch/arm64/kernel/alternative.c      |  43 +++++--
 arch/arm64/kernel/bpi.S              |  67 +++++++----
 arch/arm64/kernel/cpu_errata.c       |  21 +++-
 arch/arm64/kernel/cpufeature.c       |  19 ---
 arch/arm64/kernel/insn.c             | 190 ++++++++++++++++++++++++++++-
 arch/arm64/kvm/Kconfig               |   3 +
 arch/arm64/kvm/Makefile              |   2 +-
 arch/arm64/kvm/hyp-init.S            |   1 -
 arch/arm64/kvm/hyp/hyp-entry.S       |  58 +++++----
 arch/arm64/kvm/va_layout.c           | 227 +++++++++++++++++++++++++++++++++++
 include/kvm/arm_vgic.h               |  12 +-
 virt/kvm/arm/hyp/vgic-v2-sr.c        |  12 +-
 virt/kvm/arm/mmu.c                   | 175 ++++++++++++++++++++++-----
 virt/kvm/arm/vgic/vgic-init.c        |   6 -
 virt/kvm/arm/vgic/vgic-v2.c          |  40 ++----
 24 files changed, 929 insertions(+), 222 deletions(-)
 create mode 100644 arch/arm64/kvm/va_layout.c

-- 
2.14.2

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 00/26] KVM/arm64: Randomise EL2 mappings (variant 3a mitigation)
@ 2018-03-14 16:50 ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

Whilst KVM benefits from the kernel randomisation via KASLR, there is
no additional randomisation when the kernel is running at EL1, as we
directly use a fixed offset from the linear mapping. This is not
necessarily a problem, but we could do a bit better by independently
randomizing the HYP placement.

This series proposes to randomise the offset by inserting a few random
bits between the MSB of the RAM linear mapping and the top of the HYP
VA (VA_BITS - 2). That's not a lot of random bits (on my Mustang, I
get 13 bits), but that's better than nothing.

In order to achieve this, we need to be able to patch dynamic values
in the kernel text. This results in a bunch of changes to the
alternative framework, the insn library, and a few more hacks in KVM
itself (we get a new way to map the GIC at EL2).

Another (and more recent) goal of this series is to work around what
has been described as "variant 3a", which covers speculative reads of
privileged system registers. Randomizing the location of the
hypervisor would be pointless if one could simply obtain VBAR_EL2. In
order to work around this, we place the vectors at a fairly static
location (next to the idmap), independently of the hypervisor's own
mappings. This ensures that we can leak VBAR_EL2 without disclosing
much about HYP itself (and is similar to what the rest of the kernel
does with KPTI). This is only enabled at runtime for Cortex-A57 and
Cortex-A72.

This has been tested on the FVP model, Seattle (both 39 and 48bit VA),
Mustang and Thunder-X. I've also done a sanity check on 32bit (which
is only impacted by the HYP IO VA stuff).

Thanks,

	M.

* From v5:
  - Countless fixes, thanks to James excellent review
  - Additional patch to address 52bit PA HYP idmap unmap
  - Additional patch for HYP idmap page alignment/size
  - Additional patch to move the BP hardening slots into .hyp.text
  - New and improved far branch into the vectors (shorter epilogue)
  - Some extra tidying up thanks to Drew's review
  - Collected AB/RB tags

* From v4:
  - Added some more patches to work around speculative reads of
    VBAR_EL2
  - Bunch of cleanups and clarifications thanks to Christoffer's review
  - Dropped the initial asm-offsets rework on which this series
    didn't really rely anymore

* From v3:
  - Reworked the alternative code to leave the actual patching to
    the callback function. This should allow for more flexibility
    should someone or something require it
  - Now detects underflows in the IOVA allocator
  - Moved the VA patching code to va_layout.c

* From v2:
  - Fixed a crapload of bugs in the immediate generation patch
    I now have a test harness for it, making sure it generates the
    same thing as GAS...
  - Fixed a bug in the asm-offsets.h exclusion patch
  - Reworked the alternative_cb code to be nicer and avoid generating
    pointless nops

* From v1:
  - Now works correctly with KASLR
  - Dropped the callback field from alt_instr, and reuse one of the
    existing fields to store an offset to the callback
  - Fix HYP teardown path (depends on fixes previously posted)
  - Dropped the VA offset macros


Marc Zyngier (26):
  arm64: alternatives: Add dynamic patching feature
  arm64: insn: Add N immediate encoding
  arm64: insn: Add encoder for bitwise operations using literals
  arm64: KVM: Dynamically patch the kernel/hyp VA mask
  arm64: cpufeatures: Drop the ARM64_HYP_OFFSET_LOW feature flag
  KVM: arm/arm64: Do not use kern_hyp_va() with kvm_vgic_global_state
  KVM: arm/arm64: Demote HYP VA range display to being a debug feature
  KVM: arm/arm64: Move ioremap calls to create_hyp_io_mappings
  KVM: arm/arm64: Keep GICv2 HYP VAs in kvm_vgic_global_state
  KVM: arm/arm64: Fix idmap size and alignment
  KVM: arm64: Fix HYP idmap unmap when using 52bit PA
  KVM: arm/arm64: Move HYP IO VAs to the "idmap" range
  arm64; insn: Add encoder for the EXTR instruction
  arm64: insn: Allow ADD/SUB (immediate) with LSL #12
  arm64: KVM: Dynamically compute the HYP VA mask
  arm64: KVM: Introduce EL2 VA randomisation
  arm64: Update the KVM memory map documentation
  arm64: KVM: Move vector offsetting from hyp-init.S to
    kvm_get_hyp_vector
  arm64: KVM: Move stashing of x0/x1 into the vector code itself
  arm64: KVM: Move BP hardening vectors into .hyp.text section
  arm64: KVM: Reserve 4 additional instructions in the BPI template
  arm64: KVM: Allow far branches from vector slots to the main vectors
  arm/arm64: KVM: Introduce EL2-specific executable mappings
  arm64: Make BP hardening slot counter available
  arm64: KVM: Allow mapping of vectors outside of the RAM region
  arm64: Enable ARM64_HARDEN_EL2_VECTORS on Cortex-A57 and A72

 Documentation/arm64/memory.txt       |   9 +-
 arch/arm/include/asm/kvm_mmu.h       |  16 ++-
 arch/arm64/Kconfig                   |  16 +++
 arch/arm64/include/asm/alternative.h |  41 ++++++-
 arch/arm64/include/asm/cpucaps.h     |   2 +-
 arch/arm64/include/asm/insn.h        |  16 +++
 arch/arm64/include/asm/kvm_mmu.h     | 163 ++++++++++++++++++-------
 arch/arm64/include/asm/mmu.h         |   8 +-
 arch/arm64/kernel/Makefile           |   4 +-
 arch/arm64/kernel/alternative.c      |  43 +++++--
 arch/arm64/kernel/bpi.S              |  67 +++++++----
 arch/arm64/kernel/cpu_errata.c       |  21 +++-
 arch/arm64/kernel/cpufeature.c       |  19 ---
 arch/arm64/kernel/insn.c             | 190 ++++++++++++++++++++++++++++-
 arch/arm64/kvm/Kconfig               |   3 +
 arch/arm64/kvm/Makefile              |   2 +-
 arch/arm64/kvm/hyp-init.S            |   1 -
 arch/arm64/kvm/hyp/hyp-entry.S       |  58 +++++----
 arch/arm64/kvm/va_layout.c           | 227 +++++++++++++++++++++++++++++++++++
 include/kvm/arm_vgic.h               |  12 +-
 virt/kvm/arm/hyp/vgic-v2-sr.c        |  12 +-
 virt/kvm/arm/mmu.c                   | 175 ++++++++++++++++++++++-----
 virt/kvm/arm/vgic/vgic-init.c        |   6 -
 virt/kvm/arm/vgic/vgic-v2.c          |  40 ++----
 24 files changed, 929 insertions(+), 222 deletions(-)
 create mode 100644 arch/arm64/kvm/va_layout.c

-- 
2.14.2

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 01/26] arm64: alternatives: Add dynamic patching feature
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Catalin Marinas, Will Deacon, Kristina Martsenko

We've so far relied on a patching infrastructure that only gave us
a single alternative, without any way to provide a range of potential
replacement instructions. For a single feature, this is an all or
nothing thing.

It would be interesting to have a more flexible grained way of patching
the kernel though, where we could dynamically tune the code that gets
injected.

In order to achive this, let's introduce a new form of dynamic patching,
assiciating a callback to a patching site. This callback gets source and
target locations of the patching request, as well as the number of
instructions to be patched.

Dynamic patching is declared with the new ALTERNATIVE_CB and alternative_cb
directives:

	asm volatile(ALTERNATIVE_CB("mov %0, #0\n", callback)
		     : "r" (v));
or
	alternative_cb callback
		mov	x0, #0
	alternative_cb_end

where callback is the C function computing the alternative.

Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/include/asm/alternative.h | 41 ++++++++++++++++++++++++++++++----
 arch/arm64/kernel/alternative.c      | 43 +++++++++++++++++++++++++++---------
 2 files changed, 69 insertions(+), 15 deletions(-)

diff --git a/arch/arm64/include/asm/alternative.h b/arch/arm64/include/asm/alternative.h
index 669028172fd6..a91933b1e2e6 100644
--- a/arch/arm64/include/asm/alternative.h
+++ b/arch/arm64/include/asm/alternative.h
@@ -5,6 +5,8 @@
 #include <asm/cpucaps.h>
 #include <asm/insn.h>
 
+#define ARM64_CB_PATCH ARM64_NCAPS
+
 #ifndef __ASSEMBLY__
 
 #include <linux/init.h>
@@ -22,12 +24,19 @@ struct alt_instr {
 	u8  alt_len;		/* size of new instruction(s), <= orig_len */
 };
 
+typedef void (*alternative_cb_t)(struct alt_instr *alt,
+				 __le32 *origptr, __le32 *updptr, int nr_inst);
+
 void __init apply_alternatives_all(void);
 void apply_alternatives(void *start, size_t length);
 
-#define ALTINSTR_ENTRY(feature)						      \
+#define ALTINSTR_ENTRY(feature,cb)					      \
 	" .word 661b - .\n"				/* label           */ \
+	" .if " __stringify(cb) " == 0\n"				      \
 	" .word 663f - .\n"				/* new instruction */ \
+	" .else\n"							      \
+	" .word " __stringify(cb) "- .\n"		/* callback */	      \
+	" .endif\n"							      \
 	" .hword " __stringify(feature) "\n"		/* feature bit     */ \
 	" .byte 662b-661b\n"				/* source len      */ \
 	" .byte 664f-663f\n"				/* replacement len */
@@ -45,15 +54,18 @@ void apply_alternatives(void *start, size_t length);
  * but most assemblers die if insn1 or insn2 have a .inst. This should
  * be fixed in a binutils release posterior to 2.25.51.0.2 (anything
  * containing commit 4e4d08cf7399b606 or c1baaddf8861).
+ *
+ * Alternatives with callbacks do not generate replacement instructions.
  */
-#define __ALTERNATIVE_CFG(oldinstr, newinstr, feature, cfg_enabled)	\
+#define __ALTERNATIVE_CFG(oldinstr, newinstr, feature, cfg_enabled, cb)	\
 	".if "__stringify(cfg_enabled)" == 1\n"				\
 	"661:\n\t"							\
 	oldinstr "\n"							\
 	"662:\n"							\
 	".pushsection .altinstructions,\"a\"\n"				\
-	ALTINSTR_ENTRY(feature)						\
+	ALTINSTR_ENTRY(feature,cb)					\
 	".popsection\n"							\
+	" .if " __stringify(cb) " == 0\n"				\
 	".pushsection .altinstr_replacement, \"a\"\n"			\
 	"663:\n\t"							\
 	newinstr "\n"							\
@@ -61,11 +73,17 @@ void apply_alternatives(void *start, size_t length);
 	".popsection\n\t"						\
 	".org	. - (664b-663b) + (662b-661b)\n\t"			\
 	".org	. - (662b-661b) + (664b-663b)\n"			\
+	".else\n\t"							\
+	"663:\n\t"							\
+	"664:\n\t"							\
+	".endif\n"							\
 	".endif\n"
 
 #define _ALTERNATIVE_CFG(oldinstr, newinstr, feature, cfg, ...)	\
-	__ALTERNATIVE_CFG(oldinstr, newinstr, feature, IS_ENABLED(cfg))
+	__ALTERNATIVE_CFG(oldinstr, newinstr, feature, IS_ENABLED(cfg), 0)
 
+#define ALTERNATIVE_CB(oldinstr, cb) \
+	__ALTERNATIVE_CFG(oldinstr, "NOT_AN_INSTRUCTION", ARM64_CB_PATCH, 1, cb)
 #else
 
 #include <asm/assembler.h>
@@ -132,6 +150,14 @@ void apply_alternatives(void *start, size_t length);
 661:
 .endm
 
+.macro alternative_cb cb
+	.set .Lasm_alt_mode, 0
+	.pushsection .altinstructions, "a"
+	altinstruction_entry 661f, \cb, ARM64_CB_PATCH, 662f-661f, 0
+	.popsection
+661:
+.endm
+
 /*
  * Provide the other half of the alternative code sequence.
  */
@@ -157,6 +183,13 @@ void apply_alternatives(void *start, size_t length);
 	.org	. - (662b-661b) + (664b-663b)
 .endm
 
+/*
+ * Callback-based alternative epilogue
+ */
+.macro alternative_cb_end
+662:
+.endm
+
 /*
  * Provides a trivial alternative or default sequence consisting solely
  * of NOPs. The number of NOPs is chosen automatically to match the
diff --git a/arch/arm64/kernel/alternative.c b/arch/arm64/kernel/alternative.c
index 414288a558c8..5c4bce4ac381 100644
--- a/arch/arm64/kernel/alternative.c
+++ b/arch/arm64/kernel/alternative.c
@@ -107,32 +107,53 @@ static u32 get_alt_insn(struct alt_instr *alt, __le32 *insnptr, __le32 *altinsnp
 	return insn;
 }
 
+static void patch_alternative(struct alt_instr *alt,
+			      __le32 *origptr, __le32 *updptr, int nr_inst)
+{
+	__le32 *replptr;
+	int i;
+
+	replptr = ALT_REPL_PTR(alt);
+	for (i = 0; i < nr_inst; i++) {
+		u32 insn;
+
+		insn = get_alt_insn(alt, origptr + i, replptr + i);
+		updptr[i] = cpu_to_le32(insn);
+	}
+}
+
 static void __apply_alternatives(void *alt_region, bool use_linear_alias)
 {
 	struct alt_instr *alt;
 	struct alt_region *region = alt_region;
-	__le32 *origptr, *replptr, *updptr;
+	__le32 *origptr, *updptr;
+	alternative_cb_t alt_cb;
 
 	for (alt = region->begin; alt < region->end; alt++) {
-		u32 insn;
-		int i, nr_inst;
+		int nr_inst;
 
-		if (!cpus_have_cap(alt->cpufeature))
+		/* Use ARM64_CB_PATCH as an unconditional patch */
+		if (alt->cpufeature < ARM64_CB_PATCH &&
+		    !cpus_have_cap(alt->cpufeature))
 			continue;
 
-		BUG_ON(alt->alt_len != alt->orig_len);
+		if (alt->cpufeature == ARM64_CB_PATCH)
+			BUG_ON(alt->alt_len != 0);
+		else
+			BUG_ON(alt->alt_len != alt->orig_len);
 
 		pr_info_once("patching kernel code\n");
 
 		origptr = ALT_ORIG_PTR(alt);
-		replptr = ALT_REPL_PTR(alt);
 		updptr = use_linear_alias ? lm_alias(origptr) : origptr;
-		nr_inst = alt->alt_len / sizeof(insn);
+		nr_inst = alt->orig_len / AARCH64_INSN_SIZE;
 
-		for (i = 0; i < nr_inst; i++) {
-			insn = get_alt_insn(alt, origptr + i, replptr + i);
-			updptr[i] = cpu_to_le32(insn);
-		}
+		if (alt->cpufeature < ARM64_CB_PATCH)
+			alt_cb = patch_alternative;
+		else
+			alt_cb  = ALT_REPL_PTR(alt);
+
+		alt_cb(alt, origptr, updptr, nr_inst);
 
 		flush_icache_range((uintptr_t)origptr,
 				   (uintptr_t)(origptr + nr_inst));
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 01/26] arm64: alternatives: Add dynamic patching feature
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

We've so far relied on a patching infrastructure that only gave us
a single alternative, without any way to provide a range of potential
replacement instructions. For a single feature, this is an all or
nothing thing.

It would be interesting to have a more flexible grained way of patching
the kernel though, where we could dynamically tune the code that gets
injected.

In order to achive this, let's introduce a new form of dynamic patching,
assiciating a callback to a patching site. This callback gets source and
target locations of the patching request, as well as the number of
instructions to be patched.

Dynamic patching is declared with the new ALTERNATIVE_CB and alternative_cb
directives:

	asm volatile(ALTERNATIVE_CB("mov %0, #0\n", callback)
		     : "r" (v));
or
	alternative_cb callback
		mov	x0, #0
	alternative_cb_end

where callback is the C function computing the alternative.

Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/include/asm/alternative.h | 41 ++++++++++++++++++++++++++++++----
 arch/arm64/kernel/alternative.c      | 43 +++++++++++++++++++++++++++---------
 2 files changed, 69 insertions(+), 15 deletions(-)

diff --git a/arch/arm64/include/asm/alternative.h b/arch/arm64/include/asm/alternative.h
index 669028172fd6..a91933b1e2e6 100644
--- a/arch/arm64/include/asm/alternative.h
+++ b/arch/arm64/include/asm/alternative.h
@@ -5,6 +5,8 @@
 #include <asm/cpucaps.h>
 #include <asm/insn.h>
 
+#define ARM64_CB_PATCH ARM64_NCAPS
+
 #ifndef __ASSEMBLY__
 
 #include <linux/init.h>
@@ -22,12 +24,19 @@ struct alt_instr {
 	u8  alt_len;		/* size of new instruction(s), <= orig_len */
 };
 
+typedef void (*alternative_cb_t)(struct alt_instr *alt,
+				 __le32 *origptr, __le32 *updptr, int nr_inst);
+
 void __init apply_alternatives_all(void);
 void apply_alternatives(void *start, size_t length);
 
-#define ALTINSTR_ENTRY(feature)						      \
+#define ALTINSTR_ENTRY(feature,cb)					      \
 	" .word 661b - .\n"				/* label           */ \
+	" .if " __stringify(cb) " == 0\n"				      \
 	" .word 663f - .\n"				/* new instruction */ \
+	" .else\n"							      \
+	" .word " __stringify(cb) "- .\n"		/* callback */	      \
+	" .endif\n"							      \
 	" .hword " __stringify(feature) "\n"		/* feature bit     */ \
 	" .byte 662b-661b\n"				/* source len      */ \
 	" .byte 664f-663f\n"				/* replacement len */
@@ -45,15 +54,18 @@ void apply_alternatives(void *start, size_t length);
  * but most assemblers die if insn1 or insn2 have a .inst. This should
  * be fixed in a binutils release posterior to 2.25.51.0.2 (anything
  * containing commit 4e4d08cf7399b606 or c1baaddf8861).
+ *
+ * Alternatives with callbacks do not generate replacement instructions.
  */
-#define __ALTERNATIVE_CFG(oldinstr, newinstr, feature, cfg_enabled)	\
+#define __ALTERNATIVE_CFG(oldinstr, newinstr, feature, cfg_enabled, cb)	\
 	".if "__stringify(cfg_enabled)" == 1\n"				\
 	"661:\n\t"							\
 	oldinstr "\n"							\
 	"662:\n"							\
 	".pushsection .altinstructions,\"a\"\n"				\
-	ALTINSTR_ENTRY(feature)						\
+	ALTINSTR_ENTRY(feature,cb)					\
 	".popsection\n"							\
+	" .if " __stringify(cb) " == 0\n"				\
 	".pushsection .altinstr_replacement, \"a\"\n"			\
 	"663:\n\t"							\
 	newinstr "\n"							\
@@ -61,11 +73,17 @@ void apply_alternatives(void *start, size_t length);
 	".popsection\n\t"						\
 	".org	. - (664b-663b) + (662b-661b)\n\t"			\
 	".org	. - (662b-661b) + (664b-663b)\n"			\
+	".else\n\t"							\
+	"663:\n\t"							\
+	"664:\n\t"							\
+	".endif\n"							\
 	".endif\n"
 
 #define _ALTERNATIVE_CFG(oldinstr, newinstr, feature, cfg, ...)	\
-	__ALTERNATIVE_CFG(oldinstr, newinstr, feature, IS_ENABLED(cfg))
+	__ALTERNATIVE_CFG(oldinstr, newinstr, feature, IS_ENABLED(cfg), 0)
 
+#define ALTERNATIVE_CB(oldinstr, cb) \
+	__ALTERNATIVE_CFG(oldinstr, "NOT_AN_INSTRUCTION", ARM64_CB_PATCH, 1, cb)
 #else
 
 #include <asm/assembler.h>
@@ -132,6 +150,14 @@ void apply_alternatives(void *start, size_t length);
 661:
 .endm
 
+.macro alternative_cb cb
+	.set .Lasm_alt_mode, 0
+	.pushsection .altinstructions, "a"
+	altinstruction_entry 661f, \cb, ARM64_CB_PATCH, 662f-661f, 0
+	.popsection
+661:
+.endm
+
 /*
  * Provide the other half of the alternative code sequence.
  */
@@ -157,6 +183,13 @@ void apply_alternatives(void *start, size_t length);
 	.org	. - (662b-661b) + (664b-663b)
 .endm
 
+/*
+ * Callback-based alternative epilogue
+ */
+.macro alternative_cb_end
+662:
+.endm
+
 /*
  * Provides a trivial alternative or default sequence consisting solely
  * of NOPs. The number of NOPs is chosen automatically to match the
diff --git a/arch/arm64/kernel/alternative.c b/arch/arm64/kernel/alternative.c
index 414288a558c8..5c4bce4ac381 100644
--- a/arch/arm64/kernel/alternative.c
+++ b/arch/arm64/kernel/alternative.c
@@ -107,32 +107,53 @@ static u32 get_alt_insn(struct alt_instr *alt, __le32 *insnptr, __le32 *altinsnp
 	return insn;
 }
 
+static void patch_alternative(struct alt_instr *alt,
+			      __le32 *origptr, __le32 *updptr, int nr_inst)
+{
+	__le32 *replptr;
+	int i;
+
+	replptr = ALT_REPL_PTR(alt);
+	for (i = 0; i < nr_inst; i++) {
+		u32 insn;
+
+		insn = get_alt_insn(alt, origptr + i, replptr + i);
+		updptr[i] = cpu_to_le32(insn);
+	}
+}
+
 static void __apply_alternatives(void *alt_region, bool use_linear_alias)
 {
 	struct alt_instr *alt;
 	struct alt_region *region = alt_region;
-	__le32 *origptr, *replptr, *updptr;
+	__le32 *origptr, *updptr;
+	alternative_cb_t alt_cb;
 
 	for (alt = region->begin; alt < region->end; alt++) {
-		u32 insn;
-		int i, nr_inst;
+		int nr_inst;
 
-		if (!cpus_have_cap(alt->cpufeature))
+		/* Use ARM64_CB_PATCH as an unconditional patch */
+		if (alt->cpufeature < ARM64_CB_PATCH &&
+		    !cpus_have_cap(alt->cpufeature))
 			continue;
 
-		BUG_ON(alt->alt_len != alt->orig_len);
+		if (alt->cpufeature == ARM64_CB_PATCH)
+			BUG_ON(alt->alt_len != 0);
+		else
+			BUG_ON(alt->alt_len != alt->orig_len);
 
 		pr_info_once("patching kernel code\n");
 
 		origptr = ALT_ORIG_PTR(alt);
-		replptr = ALT_REPL_PTR(alt);
 		updptr = use_linear_alias ? lm_alias(origptr) : origptr;
-		nr_inst = alt->alt_len / sizeof(insn);
+		nr_inst = alt->orig_len / AARCH64_INSN_SIZE;
 
-		for (i = 0; i < nr_inst; i++) {
-			insn = get_alt_insn(alt, origptr + i, replptr + i);
-			updptr[i] = cpu_to_le32(insn);
-		}
+		if (alt->cpufeature < ARM64_CB_PATCH)
+			alt_cb = patch_alternative;
+		else
+			alt_cb  = ALT_REPL_PTR(alt);
+
+		alt_cb(alt, origptr, updptr, nr_inst);
 
 		flush_icache_range((uintptr_t)origptr,
 				   (uintptr_t)(origptr + nr_inst));
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 02/26] arm64: insn: Add N immediate encoding
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Mark Rutland, Peter Maydell, Andrew Jones, Christoffer Dall,
	Steve Capper, Catalin Marinas, Will Deacon, Kristina Martsenko,
	James Morse

We're missing the a way to generate the encoding of the N immediate,
which is only a single bit used in a number of instruction that take
an immediate.

Acked-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/include/asm/insn.h | 1 +
 arch/arm64/kernel/insn.c      | 4 ++++
 2 files changed, 5 insertions(+)

diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index 4214c38d016b..21fffdd290a3 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -70,6 +70,7 @@ enum aarch64_insn_imm_type {
 	AARCH64_INSN_IMM_6,
 	AARCH64_INSN_IMM_S,
 	AARCH64_INSN_IMM_R,
+	AARCH64_INSN_IMM_N,
 	AARCH64_INSN_IMM_MAX
 };
 
diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index 2718a77da165..7e432662d454 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -343,6 +343,10 @@ static int __kprobes aarch64_get_imm_shift_mask(enum aarch64_insn_imm_type type,
 		mask = BIT(6) - 1;
 		shift = 16;
 		break;
+	case AARCH64_INSN_IMM_N:
+		mask = 1;
+		shift = 22;
+		break;
 	default:
 		return -EINVAL;
 	}
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 02/26] arm64: insn: Add N immediate encoding
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

We're missing the a way to generate the encoding of the N immediate,
which is only a single bit used in a number of instruction that take
an immediate.

Acked-by: Christoffer Dall <christoffer.dall@linaro.org>
Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/include/asm/insn.h | 1 +
 arch/arm64/kernel/insn.c      | 4 ++++
 2 files changed, 5 insertions(+)

diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index 4214c38d016b..21fffdd290a3 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -70,6 +70,7 @@ enum aarch64_insn_imm_type {
 	AARCH64_INSN_IMM_6,
 	AARCH64_INSN_IMM_S,
 	AARCH64_INSN_IMM_R,
+	AARCH64_INSN_IMM_N,
 	AARCH64_INSN_IMM_MAX
 };
 
diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index 2718a77da165..7e432662d454 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -343,6 +343,10 @@ static int __kprobes aarch64_get_imm_shift_mask(enum aarch64_insn_imm_type type,
 		mask = BIT(6) - 1;
 		shift = 16;
 		break;
+	case AARCH64_INSN_IMM_N:
+		mask = 1;
+		shift = 22;
+		break;
 	default:
 		return -EINVAL;
 	}
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 03/26] arm64: insn: Add encoder for bitwise operations using literals
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Mark Rutland, Peter Maydell, Andrew Jones, Christoffer Dall,
	Steve Capper, Catalin Marinas, Will Deacon, Kristina Martsenko,
	James Morse

We lack a way to encode operations such as AND, ORR, EOR that take
an immediate value. Doing so is quite involved, and is all about
reverse engineering the decoding algorithm described in the
pseudocode function DecodeBitMasks().

This has been tested by feeding it all the possible literal values
and comparing the output with that of GAS.

Acked-by: Christoffer Dall <christoffer.dall@linaro.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/include/asm/insn.h |   9 +++
 arch/arm64/kernel/insn.c      | 136 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 145 insertions(+)

diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index 21fffdd290a3..815b35bc53ed 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -315,6 +315,10 @@ __AARCH64_INSN_FUNCS(eor,	0x7F200000, 0x4A000000)
 __AARCH64_INSN_FUNCS(eon,	0x7F200000, 0x4A200000)
 __AARCH64_INSN_FUNCS(ands,	0x7F200000, 0x6A000000)
 __AARCH64_INSN_FUNCS(bics,	0x7F200000, 0x6A200000)
+__AARCH64_INSN_FUNCS(and_imm,	0x7F800000, 0x12000000)
+__AARCH64_INSN_FUNCS(orr_imm,	0x7F800000, 0x32000000)
+__AARCH64_INSN_FUNCS(eor_imm,	0x7F800000, 0x52000000)
+__AARCH64_INSN_FUNCS(ands_imm,	0x7F800000, 0x72000000)
 __AARCH64_INSN_FUNCS(b,		0xFC000000, 0x14000000)
 __AARCH64_INSN_FUNCS(bl,	0xFC000000, 0x94000000)
 __AARCH64_INSN_FUNCS(cbz,	0x7F000000, 0x34000000)
@@ -424,6 +428,11 @@ u32 aarch64_insn_gen_logical_shifted_reg(enum aarch64_insn_register dst,
 					 int shift,
 					 enum aarch64_insn_variant variant,
 					 enum aarch64_insn_logic_type type);
+u32 aarch64_insn_gen_logical_immediate(enum aarch64_insn_logic_type type,
+				       enum aarch64_insn_variant variant,
+				       enum aarch64_insn_register Rn,
+				       enum aarch64_insn_register Rd,
+				       u64 imm);
 u32 aarch64_insn_gen_prefetch(enum aarch64_insn_register base,
 			      enum aarch64_insn_prfm_type type,
 			      enum aarch64_insn_prfm_target target,
diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index 7e432662d454..e87d6dcd7c82 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -1485,3 +1485,139 @@ pstate_check_t * const aarch32_opcode_cond_checks[16] = {
 	__check_hi, __check_ls, __check_ge, __check_lt,
 	__check_gt, __check_le, __check_al, __check_al
 };
+
+static bool range_of_ones(u64 val)
+{
+	/* Doesn't handle full ones or full zeroes */
+	u64 sval = val >> __ffs64(val);
+
+	/* One of Sean Eron Anderson's bithack tricks */
+	return ((sval + 1) & (sval)) == 0;
+}
+
+static u32 aarch64_encode_immediate(u64 imm,
+				    enum aarch64_insn_variant variant,
+				    u32 insn)
+{
+	unsigned int immr, imms, n, ones, ror, esz, tmp;
+	u64 mask = ~0UL;
+
+	/* Can't encode full zeroes or full ones */
+	if (!imm || !~imm)
+		return AARCH64_BREAK_FAULT;
+
+	switch (variant) {
+	case AARCH64_INSN_VARIANT_32BIT:
+		if (upper_32_bits(imm))
+			return AARCH64_BREAK_FAULT;
+		esz = 32;
+		break;
+	case AARCH64_INSN_VARIANT_64BIT:
+		insn |= AARCH64_INSN_SF_BIT;
+		esz = 64;
+		break;
+	default:
+		pr_err("%s: unknown variant encoding %d\n", __func__, variant);
+		return AARCH64_BREAK_FAULT;
+	}
+
+	/*
+	 * Inverse of Replicate(). Try to spot a repeating pattern
+	 * with a pow2 stride.
+	 */
+	for (tmp = esz / 2; tmp >= 2; tmp /= 2) {
+		u64 emask = BIT(tmp) - 1;
+
+		if ((imm & emask) != ((imm >> tmp) & emask))
+			break;
+
+		esz = tmp;
+		mask = emask;
+	}
+
+	/* N is only set if we're encoding a 64bit value */
+	n = esz == 64;
+
+	/* Trim imm to the element size */
+	imm &= mask;
+
+	/* That's how many ones we need to encode */
+	ones = hweight64(imm);
+
+	/*
+	 * imms is set to (ones - 1), prefixed with a string of ones
+	 * and a zero if they fit. Cap it to 6 bits.
+	 */
+	imms  = ones - 1;
+	imms |= 0xf << ffs(esz);
+	imms &= BIT(6) - 1;
+
+	/* Compute the rotation */
+	if (range_of_ones(imm)) {
+		/*
+		 * Pattern: 0..01..10..0
+		 *
+		 * Compute how many rotate we need to align it right
+		 */
+		ror = __ffs64(imm);
+	} else {
+		/*
+		 * Pattern: 0..01..10..01..1
+		 *
+		 * Fill the unused top bits with ones, and check if
+		 * the result is a valid immediate (all ones with a
+		 * contiguous ranges of zeroes).
+		 */
+		imm |= ~mask;
+		if (!range_of_ones(~imm))
+			return AARCH64_BREAK_FAULT;
+
+		/*
+		 * Compute the rotation to get a continuous set of
+		 * ones, with the first bit set at position 0
+		 */
+		ror = fls(~imm);
+	}
+
+	/*
+	 * immr is the number of bits we need to rotate back to the
+	 * original set of ones. Note that this is relative to the
+	 * element size...
+	 */
+	immr = (esz - ror) % esz;
+
+	insn = aarch64_insn_encode_immediate(AARCH64_INSN_IMM_N, insn, n);
+	insn = aarch64_insn_encode_immediate(AARCH64_INSN_IMM_R, insn, immr);
+	return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_S, insn, imms);
+}
+
+u32 aarch64_insn_gen_logical_immediate(enum aarch64_insn_logic_type type,
+				       enum aarch64_insn_variant variant,
+				       enum aarch64_insn_register Rn,
+				       enum aarch64_insn_register Rd,
+				       u64 imm)
+{
+	u32 insn;
+
+	switch (type) {
+	case AARCH64_INSN_LOGIC_AND:
+		insn = aarch64_insn_get_and_imm_value();
+		break;
+	case AARCH64_INSN_LOGIC_ORR:
+		insn = aarch64_insn_get_orr_imm_value();
+		break;
+	case AARCH64_INSN_LOGIC_EOR:
+		insn = aarch64_insn_get_eor_imm_value();
+		break;
+	case AARCH64_INSN_LOGIC_AND_SETFLAGS:
+		insn = aarch64_insn_get_ands_imm_value();
+		break;
+	default:
+		pr_err("%s: unknown logical encoding %d\n", __func__, type);
+		return AARCH64_BREAK_FAULT;
+	}
+
+	insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RD, insn, Rd);
+	insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn, Rn);
+	return aarch64_encode_immediate(imm, variant, insn);
+}
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 03/26] arm64: insn: Add encoder for bitwise operations using literals
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

We lack a way to encode operations such as AND, ORR, EOR that take
an immediate value. Doing so is quite involved, and is all about
reverse engineering the decoding algorithm described in the
pseudocode function DecodeBitMasks().

This has been tested by feeding it all the possible literal values
and comparing the output with that of GAS.

Acked-by: Christoffer Dall <christoffer.dall@linaro.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/include/asm/insn.h |   9 +++
 arch/arm64/kernel/insn.c      | 136 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 145 insertions(+)

diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index 21fffdd290a3..815b35bc53ed 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -315,6 +315,10 @@ __AARCH64_INSN_FUNCS(eor,	0x7F200000, 0x4A000000)
 __AARCH64_INSN_FUNCS(eon,	0x7F200000, 0x4A200000)
 __AARCH64_INSN_FUNCS(ands,	0x7F200000, 0x6A000000)
 __AARCH64_INSN_FUNCS(bics,	0x7F200000, 0x6A200000)
+__AARCH64_INSN_FUNCS(and_imm,	0x7F800000, 0x12000000)
+__AARCH64_INSN_FUNCS(orr_imm,	0x7F800000, 0x32000000)
+__AARCH64_INSN_FUNCS(eor_imm,	0x7F800000, 0x52000000)
+__AARCH64_INSN_FUNCS(ands_imm,	0x7F800000, 0x72000000)
 __AARCH64_INSN_FUNCS(b,		0xFC000000, 0x14000000)
 __AARCH64_INSN_FUNCS(bl,	0xFC000000, 0x94000000)
 __AARCH64_INSN_FUNCS(cbz,	0x7F000000, 0x34000000)
@@ -424,6 +428,11 @@ u32 aarch64_insn_gen_logical_shifted_reg(enum aarch64_insn_register dst,
 					 int shift,
 					 enum aarch64_insn_variant variant,
 					 enum aarch64_insn_logic_type type);
+u32 aarch64_insn_gen_logical_immediate(enum aarch64_insn_logic_type type,
+				       enum aarch64_insn_variant variant,
+				       enum aarch64_insn_register Rn,
+				       enum aarch64_insn_register Rd,
+				       u64 imm);
 u32 aarch64_insn_gen_prefetch(enum aarch64_insn_register base,
 			      enum aarch64_insn_prfm_type type,
 			      enum aarch64_insn_prfm_target target,
diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index 7e432662d454..e87d6dcd7c82 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -1485,3 +1485,139 @@ pstate_check_t * const aarch32_opcode_cond_checks[16] = {
 	__check_hi, __check_ls, __check_ge, __check_lt,
 	__check_gt, __check_le, __check_al, __check_al
 };
+
+static bool range_of_ones(u64 val)
+{
+	/* Doesn't handle full ones or full zeroes */
+	u64 sval = val >> __ffs64(val);
+
+	/* One of Sean Eron Anderson's bithack tricks */
+	return ((sval + 1) & (sval)) == 0;
+}
+
+static u32 aarch64_encode_immediate(u64 imm,
+				    enum aarch64_insn_variant variant,
+				    u32 insn)
+{
+	unsigned int immr, imms, n, ones, ror, esz, tmp;
+	u64 mask = ~0UL;
+
+	/* Can't encode full zeroes or full ones */
+	if (!imm || !~imm)
+		return AARCH64_BREAK_FAULT;
+
+	switch (variant) {
+	case AARCH64_INSN_VARIANT_32BIT:
+		if (upper_32_bits(imm))
+			return AARCH64_BREAK_FAULT;
+		esz = 32;
+		break;
+	case AARCH64_INSN_VARIANT_64BIT:
+		insn |= AARCH64_INSN_SF_BIT;
+		esz = 64;
+		break;
+	default:
+		pr_err("%s: unknown variant encoding %d\n", __func__, variant);
+		return AARCH64_BREAK_FAULT;
+	}
+
+	/*
+	 * Inverse of Replicate(). Try to spot a repeating pattern
+	 * with a pow2 stride.
+	 */
+	for (tmp = esz / 2; tmp >= 2; tmp /= 2) {
+		u64 emask = BIT(tmp) - 1;
+
+		if ((imm & emask) != ((imm >> tmp) & emask))
+			break;
+
+		esz = tmp;
+		mask = emask;
+	}
+
+	/* N is only set if we're encoding a 64bit value */
+	n = esz == 64;
+
+	/* Trim imm to the element size */
+	imm &= mask;
+
+	/* That's how many ones we need to encode */
+	ones = hweight64(imm);
+
+	/*
+	 * imms is set to (ones - 1), prefixed with a string of ones
+	 * and a zero if they fit. Cap it to 6 bits.
+	 */
+	imms  = ones - 1;
+	imms |= 0xf << ffs(esz);
+	imms &= BIT(6) - 1;
+
+	/* Compute the rotation */
+	if (range_of_ones(imm)) {
+		/*
+		 * Pattern: 0..01..10..0
+		 *
+		 * Compute how many rotate we need to align it right
+		 */
+		ror = __ffs64(imm);
+	} else {
+		/*
+		 * Pattern: 0..01..10..01..1
+		 *
+		 * Fill the unused top bits with ones, and check if
+		 * the result is a valid immediate (all ones with a
+		 * contiguous ranges of zeroes).
+		 */
+		imm |= ~mask;
+		if (!range_of_ones(~imm))
+			return AARCH64_BREAK_FAULT;
+
+		/*
+		 * Compute the rotation to get a continuous set of
+		 * ones, with the first bit set@position 0
+		 */
+		ror = fls(~imm);
+	}
+
+	/*
+	 * immr is the number of bits we need to rotate back to the
+	 * original set of ones. Note that this is relative to the
+	 * element size...
+	 */
+	immr = (esz - ror) % esz;
+
+	insn = aarch64_insn_encode_immediate(AARCH64_INSN_IMM_N, insn, n);
+	insn = aarch64_insn_encode_immediate(AARCH64_INSN_IMM_R, insn, immr);
+	return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_S, insn, imms);
+}
+
+u32 aarch64_insn_gen_logical_immediate(enum aarch64_insn_logic_type type,
+				       enum aarch64_insn_variant variant,
+				       enum aarch64_insn_register Rn,
+				       enum aarch64_insn_register Rd,
+				       u64 imm)
+{
+	u32 insn;
+
+	switch (type) {
+	case AARCH64_INSN_LOGIC_AND:
+		insn = aarch64_insn_get_and_imm_value();
+		break;
+	case AARCH64_INSN_LOGIC_ORR:
+		insn = aarch64_insn_get_orr_imm_value();
+		break;
+	case AARCH64_INSN_LOGIC_EOR:
+		insn = aarch64_insn_get_eor_imm_value();
+		break;
+	case AARCH64_INSN_LOGIC_AND_SETFLAGS:
+		insn = aarch64_insn_get_ands_imm_value();
+		break;
+	default:
+		pr_err("%s: unknown logical encoding %d\n", __func__, type);
+		return AARCH64_BREAK_FAULT;
+	}
+
+	insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RD, insn, Rd);
+	insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn, Rn);
+	return aarch64_encode_immediate(imm, variant, insn);
+}
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 04/26] arm64: KVM: Dynamically patch the kernel/hyp VA mask
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Mark Rutland, Peter Maydell, Andrew Jones, Christoffer Dall,
	Steve Capper, Catalin Marinas, Will Deacon, Kristina Martsenko,
	James Morse

So far, we're using a complicated sequence of alternatives to
patch the kernel/hyp VA mask on non-VHE, and NOP out the
masking altogether when on VHE.

The newly introduced dynamic patching gives us the opportunity
to simplify that code by patching a single instruction with
the correct mask (instead of the mind bending cummulative masking
we have at the moment) or even a single NOP on VHE. This also
adds some initial code that will allow the patching callback
to switch to a more complex patching.

Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/include/asm/kvm_mmu.h | 46 ++++++--------------
 arch/arm64/kvm/Makefile          |  2 +-
 arch/arm64/kvm/va_layout.c       | 91 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 105 insertions(+), 34 deletions(-)
 create mode 100644 arch/arm64/kvm/va_layout.c

diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 7faed6e48b46..e3bc1d0a5e93 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -69,9 +69,6 @@
  * mappings, and none of this applies in that case.
  */
 
-#define HYP_PAGE_OFFSET_HIGH_MASK	((UL(1) << VA_BITS) - 1)
-#define HYP_PAGE_OFFSET_LOW_MASK	((UL(1) << (VA_BITS - 1)) - 1)
-
 #ifdef __ASSEMBLY__
 
 #include <asm/alternative.h>
@@ -81,28 +78,15 @@
  * Convert a kernel VA into a HYP VA.
  * reg: VA to be converted.
  *
- * This generates the following sequences:
- * - High mask:
- *		and x0, x0, #HYP_PAGE_OFFSET_HIGH_MASK
- *		nop
- * - Low mask:
- *		and x0, x0, #HYP_PAGE_OFFSET_HIGH_MASK
- *		and x0, x0, #HYP_PAGE_OFFSET_LOW_MASK
- * - VHE:
- *		nop
- *		nop
- *
- * The "low mask" version works because the mask is a strict subset of
- * the "high mask", hence performing the first mask for nothing.
- * Should be completely invisible on any viable CPU.
+ * The actual code generation takes place in kvm_update_va_mask, and
+ * the instructions below are only there to reserve the space and
+ * perform the register allocation (kvm_update_va_mask uses the
+ * specific registers encoded in the instructions).
  */
 .macro kern_hyp_va	reg
-alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
-	and     \reg, \reg, #HYP_PAGE_OFFSET_HIGH_MASK
-alternative_else_nop_endif
-alternative_if ARM64_HYP_OFFSET_LOW
-	and     \reg, \reg, #HYP_PAGE_OFFSET_LOW_MASK
-alternative_else_nop_endif
+alternative_cb kvm_update_va_mask
+	and     \reg, \reg, #1
+alternative_cb_end
 .endm
 
 #else
@@ -113,18 +97,14 @@ alternative_else_nop_endif
 #include <asm/mmu_context.h>
 #include <asm/pgtable.h>
 
+void kvm_update_va_mask(struct alt_instr *alt,
+			__le32 *origptr, __le32 *updptr, int nr_inst);
+
 static inline unsigned long __kern_hyp_va(unsigned long v)
 {
-	asm volatile(ALTERNATIVE("and %0, %0, %1",
-				 "nop",
-				 ARM64_HAS_VIRT_HOST_EXTN)
-		     : "+r" (v)
-		     : "i" (HYP_PAGE_OFFSET_HIGH_MASK));
-	asm volatile(ALTERNATIVE("nop",
-				 "and %0, %0, %1",
-				 ARM64_HYP_OFFSET_LOW)
-		     : "+r" (v)
-		     : "i" (HYP_PAGE_OFFSET_LOW_MASK));
+	asm volatile(ALTERNATIVE_CB("and %0, %0, #1\n",
+				    kvm_update_va_mask)
+		     : "+r" (v));
 	return v;
 }
 
diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
index 87c4f7ae24de..93afff91cb7c 100644
--- a/arch/arm64/kvm/Makefile
+++ b/arch/arm64/kvm/Makefile
@@ -16,7 +16,7 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/e
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/arm.o $(KVM)/arm/mmu.o $(KVM)/arm/mmio.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/psci.o $(KVM)/arm/perf.o
 
-kvm-$(CONFIG_KVM_ARM_HOST) += inject_fault.o regmap.o
+kvm-$(CONFIG_KVM_ARM_HOST) += inject_fault.o regmap.o va_layout.o
 kvm-$(CONFIG_KVM_ARM_HOST) += hyp.o hyp-init.o handle_exit.o
 kvm-$(CONFIG_KVM_ARM_HOST) += guest.o debug.o reset.o sys_regs.o sys_regs_generic_v8.o
 kvm-$(CONFIG_KVM_ARM_HOST) += vgic-sys-reg-v3.o
diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c
new file mode 100644
index 000000000000..45e7802328d4
--- /dev/null
+++ b/arch/arm64/kvm/va_layout.c
@@ -0,0 +1,91 @@
+/*
+ * Copyright (C) 2017 ARM Ltd.
+ * Author: Marc Zyngier <marc.zyngier@arm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/kvm_host.h>
+#include <asm/alternative.h>
+#include <asm/debug-monitors.h>
+#include <asm/insn.h>
+#include <asm/kvm_mmu.h>
+
+#define HYP_PAGE_OFFSET_HIGH_MASK	((UL(1) << VA_BITS) - 1)
+#define HYP_PAGE_OFFSET_LOW_MASK	((UL(1) << (VA_BITS - 1)) - 1)
+
+static u64 va_mask;
+
+static void compute_layout(void)
+{
+	phys_addr_t idmap_addr = __pa_symbol(__hyp_idmap_text_start);
+	unsigned long mask = HYP_PAGE_OFFSET_HIGH_MASK;
+
+	/*
+	 * Activate the lower HYP offset only if the idmap doesn't
+	 * clash with it,
+	 */
+	if (idmap_addr > HYP_PAGE_OFFSET_LOW_MASK)
+		mask = HYP_PAGE_OFFSET_LOW_MASK;
+
+	va_mask = mask;
+}
+
+static u32 compute_instruction(int n, u32 rd, u32 rn)
+{
+	u32 insn = AARCH64_BREAK_FAULT;
+
+	switch (n) {
+	case 0:
+		insn = aarch64_insn_gen_logical_immediate(AARCH64_INSN_LOGIC_AND,
+							  AARCH64_INSN_VARIANT_64BIT,
+							  rn, rd, va_mask);
+		break;
+	}
+
+	return insn;
+}
+
+void __init kvm_update_va_mask(struct alt_instr *alt,
+			       __le32 *origptr, __le32 *updptr, int nr_inst)
+{
+	int i;
+
+	/* We only expect a single instruction in the alternative sequence */
+	BUG_ON(nr_inst != 1);
+
+	if (!has_vhe() && !va_mask)
+		compute_layout();
+
+	for (i = 0; i < nr_inst; i++) {
+		u32 rd, rn, insn, oinsn;
+
+		/*
+		 * VHE doesn't need any address translation, let's NOP
+		 * everything.
+		 */
+		if (has_vhe()) {
+			updptr[i] = aarch64_insn_gen_nop();
+			continue;
+		}
+
+		oinsn = le32_to_cpu(origptr[i]);
+		rd = aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RD, oinsn);
+		rn = aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RN, oinsn);
+
+		insn = compute_instruction(i, rd, rn);
+		BUG_ON(insn == AARCH64_BREAK_FAULT);
+
+		updptr[i] = cpu_to_le32(insn);
+	}
+}
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 04/26] arm64: KVM: Dynamically patch the kernel/hyp VA mask
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

So far, we're using a complicated sequence of alternatives to
patch the kernel/hyp VA mask on non-VHE, and NOP out the
masking altogether when on VHE.

The newly introduced dynamic patching gives us the opportunity
to simplify that code by patching a single instruction with
the correct mask (instead of the mind bending cummulative masking
we have at the moment) or even a single NOP on VHE. This also
adds some initial code that will allow the patching callback
to switch to a more complex patching.

Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/include/asm/kvm_mmu.h | 46 ++++++--------------
 arch/arm64/kvm/Makefile          |  2 +-
 arch/arm64/kvm/va_layout.c       | 91 ++++++++++++++++++++++++++++++++++++++++
 3 files changed, 105 insertions(+), 34 deletions(-)
 create mode 100644 arch/arm64/kvm/va_layout.c

diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 7faed6e48b46..e3bc1d0a5e93 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -69,9 +69,6 @@
  * mappings, and none of this applies in that case.
  */
 
-#define HYP_PAGE_OFFSET_HIGH_MASK	((UL(1) << VA_BITS) - 1)
-#define HYP_PAGE_OFFSET_LOW_MASK	((UL(1) << (VA_BITS - 1)) - 1)
-
 #ifdef __ASSEMBLY__
 
 #include <asm/alternative.h>
@@ -81,28 +78,15 @@
  * Convert a kernel VA into a HYP VA.
  * reg: VA to be converted.
  *
- * This generates the following sequences:
- * - High mask:
- *		and x0, x0, #HYP_PAGE_OFFSET_HIGH_MASK
- *		nop
- * - Low mask:
- *		and x0, x0, #HYP_PAGE_OFFSET_HIGH_MASK
- *		and x0, x0, #HYP_PAGE_OFFSET_LOW_MASK
- * - VHE:
- *		nop
- *		nop
- *
- * The "low mask" version works because the mask is a strict subset of
- * the "high mask", hence performing the first mask for nothing.
- * Should be completely invisible on any viable CPU.
+ * The actual code generation takes place in kvm_update_va_mask, and
+ * the instructions below are only there to reserve the space and
+ * perform the register allocation (kvm_update_va_mask uses the
+ * specific registers encoded in the instructions).
  */
 .macro kern_hyp_va	reg
-alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
-	and     \reg, \reg, #HYP_PAGE_OFFSET_HIGH_MASK
-alternative_else_nop_endif
-alternative_if ARM64_HYP_OFFSET_LOW
-	and     \reg, \reg, #HYP_PAGE_OFFSET_LOW_MASK
-alternative_else_nop_endif
+alternative_cb kvm_update_va_mask
+	and     \reg, \reg, #1
+alternative_cb_end
 .endm
 
 #else
@@ -113,18 +97,14 @@ alternative_else_nop_endif
 #include <asm/mmu_context.h>
 #include <asm/pgtable.h>
 
+void kvm_update_va_mask(struct alt_instr *alt,
+			__le32 *origptr, __le32 *updptr, int nr_inst);
+
 static inline unsigned long __kern_hyp_va(unsigned long v)
 {
-	asm volatile(ALTERNATIVE("and %0, %0, %1",
-				 "nop",
-				 ARM64_HAS_VIRT_HOST_EXTN)
-		     : "+r" (v)
-		     : "i" (HYP_PAGE_OFFSET_HIGH_MASK));
-	asm volatile(ALTERNATIVE("nop",
-				 "and %0, %0, %1",
-				 ARM64_HYP_OFFSET_LOW)
-		     : "+r" (v)
-		     : "i" (HYP_PAGE_OFFSET_LOW_MASK));
+	asm volatile(ALTERNATIVE_CB("and %0, %0, #1\n",
+				    kvm_update_va_mask)
+		     : "+r" (v));
 	return v;
 }
 
diff --git a/arch/arm64/kvm/Makefile b/arch/arm64/kvm/Makefile
index 87c4f7ae24de..93afff91cb7c 100644
--- a/arch/arm64/kvm/Makefile
+++ b/arch/arm64/kvm/Makefile
@@ -16,7 +16,7 @@ kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/kvm_main.o $(KVM)/coalesced_mmio.o $(KVM)/e
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/arm.o $(KVM)/arm/mmu.o $(KVM)/arm/mmio.o
 kvm-$(CONFIG_KVM_ARM_HOST) += $(KVM)/arm/psci.o $(KVM)/arm/perf.o
 
-kvm-$(CONFIG_KVM_ARM_HOST) += inject_fault.o regmap.o
+kvm-$(CONFIG_KVM_ARM_HOST) += inject_fault.o regmap.o va_layout.o
 kvm-$(CONFIG_KVM_ARM_HOST) += hyp.o hyp-init.o handle_exit.o
 kvm-$(CONFIG_KVM_ARM_HOST) += guest.o debug.o reset.o sys_regs.o sys_regs_generic_v8.o
 kvm-$(CONFIG_KVM_ARM_HOST) += vgic-sys-reg-v3.o
diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c
new file mode 100644
index 000000000000..45e7802328d4
--- /dev/null
+++ b/arch/arm64/kvm/va_layout.c
@@ -0,0 +1,91 @@
+/*
+ * Copyright (C) 2017 ARM Ltd.
+ * Author: Marc Zyngier <marc.zyngier@arm.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License
+ * along with this program.  If not, see <http://www.gnu.org/licenses/>.
+ */
+
+#include <linux/kvm_host.h>
+#include <asm/alternative.h>
+#include <asm/debug-monitors.h>
+#include <asm/insn.h>
+#include <asm/kvm_mmu.h>
+
+#define HYP_PAGE_OFFSET_HIGH_MASK	((UL(1) << VA_BITS) - 1)
+#define HYP_PAGE_OFFSET_LOW_MASK	((UL(1) << (VA_BITS - 1)) - 1)
+
+static u64 va_mask;
+
+static void compute_layout(void)
+{
+	phys_addr_t idmap_addr = __pa_symbol(__hyp_idmap_text_start);
+	unsigned long mask = HYP_PAGE_OFFSET_HIGH_MASK;
+
+	/*
+	 * Activate the lower HYP offset only if the idmap doesn't
+	 * clash with it,
+	 */
+	if (idmap_addr > HYP_PAGE_OFFSET_LOW_MASK)
+		mask = HYP_PAGE_OFFSET_LOW_MASK;
+
+	va_mask = mask;
+}
+
+static u32 compute_instruction(int n, u32 rd, u32 rn)
+{
+	u32 insn = AARCH64_BREAK_FAULT;
+
+	switch (n) {
+	case 0:
+		insn = aarch64_insn_gen_logical_immediate(AARCH64_INSN_LOGIC_AND,
+							  AARCH64_INSN_VARIANT_64BIT,
+							  rn, rd, va_mask);
+		break;
+	}
+
+	return insn;
+}
+
+void __init kvm_update_va_mask(struct alt_instr *alt,
+			       __le32 *origptr, __le32 *updptr, int nr_inst)
+{
+	int i;
+
+	/* We only expect a single instruction in the alternative sequence */
+	BUG_ON(nr_inst != 1);
+
+	if (!has_vhe() && !va_mask)
+		compute_layout();
+
+	for (i = 0; i < nr_inst; i++) {
+		u32 rd, rn, insn, oinsn;
+
+		/*
+		 * VHE doesn't need any address translation, let's NOP
+		 * everything.
+		 */
+		if (has_vhe()) {
+			updptr[i] = aarch64_insn_gen_nop();
+			continue;
+		}
+
+		oinsn = le32_to_cpu(origptr[i]);
+		rd = aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RD, oinsn);
+		rn = aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RN, oinsn);
+
+		insn = compute_instruction(i, rd, rn);
+		BUG_ON(insn == AARCH64_BREAK_FAULT);
+
+		updptr[i] = cpu_to_le32(insn);
+	}
+}
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 05/26] arm64: cpufeatures: Drop the ARM64_HYP_OFFSET_LOW feature flag
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Catalin Marinas, Will Deacon, Kristina Martsenko

Now that we can dynamically compute the kernek/hyp VA mask, there
is no need for a feature flag to trigger the alternative patching.
Let's drop the flag and everything that depends on it.

Acked-by: Christoffer Dall <christoffer.dall@linaro.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/include/asm/cpucaps.h |  2 +-
 arch/arm64/kernel/cpufeature.c   | 19 -------------------
 2 files changed, 1 insertion(+), 20 deletions(-)

diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h
index bb263820de13..76a43a17449a 100644
--- a/arch/arm64/include/asm/cpucaps.h
+++ b/arch/arm64/include/asm/cpucaps.h
@@ -32,7 +32,7 @@
 #define ARM64_HAS_VIRT_HOST_EXTN		11
 #define ARM64_WORKAROUND_CAVIUM_27456		12
 #define ARM64_HAS_32BIT_EL0			13
-#define ARM64_HYP_OFFSET_LOW			14
+/* #define ARM64_UNALLOCATED_ENTRY			14 */
 #define ARM64_MISMATCHED_CACHE_LINE_SIZE	15
 #define ARM64_HAS_NO_FPSIMD			16
 #define ARM64_WORKAROUND_REPEAT_TLBI		17
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 2985a067fc13..5b25d56bccfd 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -831,19 +831,6 @@ static bool runs_at_el2(const struct arm64_cpu_capabilities *entry, int __unused
 	return is_kernel_in_hyp_mode();
 }
 
-static bool hyp_offset_low(const struct arm64_cpu_capabilities *entry,
-			   int __unused)
-{
-	phys_addr_t idmap_addr = __pa_symbol(__hyp_idmap_text_start);
-
-	/*
-	 * Activate the lower HYP offset only if:
-	 * - the idmap doesn't clash with it,
-	 * - the kernel is not running at EL2.
-	 */
-	return idmap_addr > GENMASK(VA_BITS - 2, 0) && !is_kernel_in_hyp_mode();
-}
-
 static bool has_no_fpsimd(const struct arm64_cpu_capabilities *entry, int __unused)
 {
 	u64 pfr0 = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
@@ -1029,12 +1016,6 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.field_pos = ID_AA64PFR0_EL0_SHIFT,
 		.min_field_value = ID_AA64PFR0_EL0_32BIT_64BIT,
 	},
-	{
-		.desc = "Reduced HYP mapping offset",
-		.capability = ARM64_HYP_OFFSET_LOW,
-		.def_scope = SCOPE_SYSTEM,
-		.matches = hyp_offset_low,
-	},
 #ifdef CONFIG_UNMAP_KERNEL_AT_EL0
 	{
 		.desc = "Kernel page table isolation (KPTI)",
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 05/26] arm64: cpufeatures: Drop the ARM64_HYP_OFFSET_LOW feature flag
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

Now that we can dynamically compute the kernek/hyp VA mask, there
is no need for a feature flag to trigger the alternative patching.
Let's drop the flag and everything that depends on it.

Acked-by: Christoffer Dall <christoffer.dall@linaro.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/include/asm/cpucaps.h |  2 +-
 arch/arm64/kernel/cpufeature.c   | 19 -------------------
 2 files changed, 1 insertion(+), 20 deletions(-)

diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h
index bb263820de13..76a43a17449a 100644
--- a/arch/arm64/include/asm/cpucaps.h
+++ b/arch/arm64/include/asm/cpucaps.h
@@ -32,7 +32,7 @@
 #define ARM64_HAS_VIRT_HOST_EXTN		11
 #define ARM64_WORKAROUND_CAVIUM_27456		12
 #define ARM64_HAS_32BIT_EL0			13
-#define ARM64_HYP_OFFSET_LOW			14
+/* #define ARM64_UNALLOCATED_ENTRY			14 */
 #define ARM64_MISMATCHED_CACHE_LINE_SIZE	15
 #define ARM64_HAS_NO_FPSIMD			16
 #define ARM64_WORKAROUND_REPEAT_TLBI		17
diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c
index 2985a067fc13..5b25d56bccfd 100644
--- a/arch/arm64/kernel/cpufeature.c
+++ b/arch/arm64/kernel/cpufeature.c
@@ -831,19 +831,6 @@ static bool runs_at_el2(const struct arm64_cpu_capabilities *entry, int __unused
 	return is_kernel_in_hyp_mode();
 }
 
-static bool hyp_offset_low(const struct arm64_cpu_capabilities *entry,
-			   int __unused)
-{
-	phys_addr_t idmap_addr = __pa_symbol(__hyp_idmap_text_start);
-
-	/*
-	 * Activate the lower HYP offset only if:
-	 * - the idmap doesn't clash with it,
-	 * - the kernel is not running at EL2.
-	 */
-	return idmap_addr > GENMASK(VA_BITS - 2, 0) && !is_kernel_in_hyp_mode();
-}
-
 static bool has_no_fpsimd(const struct arm64_cpu_capabilities *entry, int __unused)
 {
 	u64 pfr0 = read_sanitised_ftr_reg(SYS_ID_AA64PFR0_EL1);
@@ -1029,12 +1016,6 @@ static const struct arm64_cpu_capabilities arm64_features[] = {
 		.field_pos = ID_AA64PFR0_EL0_SHIFT,
 		.min_field_value = ID_AA64PFR0_EL0_32BIT_64BIT,
 	},
-	{
-		.desc = "Reduced HYP mapping offset",
-		.capability = ARM64_HYP_OFFSET_LOW,
-		.def_scope = SCOPE_SYSTEM,
-		.matches = hyp_offset_low,
-	},
 #ifdef CONFIG_UNMAP_KERNEL_AT_EL0
 	{
 		.desc = "Kernel page table isolation (KPTI)",
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 06/26] KVM: arm/arm64: Do not use kern_hyp_va() with kvm_vgic_global_state
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Catalin Marinas, Will Deacon, Kristina Martsenko

kvm_vgic_global_state is part of the read-only section, and is
usually accessed using a PC-relative address generation (adrp + add).

It is thus useless to use kern_hyp_va() on it, and actively problematic
if kern_hyp_va() becomes non-idempotent. On the other hand, there is
no way that the compiler is going to guarantee that such access is
always PC relative.

So let's bite the bullet and provide our own accessor.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm/include/asm/kvm_mmu.h   |  7 +++++++
 arch/arm64/include/asm/kvm_mmu.h | 20 ++++++++++++++++++++
 virt/kvm/arm/hyp/vgic-v2-sr.c    |  4 ++--
 3 files changed, 29 insertions(+), 2 deletions(-)

diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
index de1b919404e4..a6808d2869f5 100644
--- a/arch/arm/include/asm/kvm_mmu.h
+++ b/arch/arm/include/asm/kvm_mmu.h
@@ -28,6 +28,13 @@
  */
 #define kern_hyp_va(kva)	(kva)
 
+/* Resolving symbol addresses in a PC-relative way is easy... */
+#define hyp_symbol_addr(s)						\
+	({								\
+		typeof(s) *addr = &(s);					\
+		addr;							\
+	})
+
 /*
  * KVM_MMU_CACHE_MIN_PAGES is the number of stage2 page table translation levels.
  */
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index e3bc1d0a5e93..7120bf3f22c7 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -110,6 +110,26 @@ static inline unsigned long __kern_hyp_va(unsigned long v)
 
 #define kern_hyp_va(v) 	((typeof(v))(__kern_hyp_va((unsigned long)(v))))
 
+/*
+ * Obtain the PC-relative address of a kernel symbol
+ * s: symbol
+ *
+ * The goal of this macro is to return a symbol's address based on a
+ * PC-relative computation, as opposed to a loading the VA from a
+ * constant pool or something similar. This works well for HYP, as an
+ * absolute VA is guaranteed to be wrong. Only use this if trying to
+ * obtain the address of a symbol (i.e. not something you obtained by
+ * following a pointer).
+ */
+#define hyp_symbol_addr(s)						\
+	({								\
+		typeof(s) *addr;					\
+		asm volatile("adrp	%0, %1\n"			\
+			     "add	%0, %0, :lo12:%1\n"		\
+			     : "=r" (addr) : "S" (&s));			\
+		addr;							\
+	})
+
 /*
  * We currently only support a 40bit IPA.
  */
diff --git a/virt/kvm/arm/hyp/vgic-v2-sr.c b/virt/kvm/arm/hyp/vgic-v2-sr.c
index 4fe6e797e8b3..a6ca049d9651 100644
--- a/virt/kvm/arm/hyp/vgic-v2-sr.c
+++ b/virt/kvm/arm/hyp/vgic-v2-sr.c
@@ -26,7 +26,7 @@
 static void __hyp_text save_elrsr(struct kvm_vcpu *vcpu, void __iomem *base)
 {
 	struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
-	int nr_lr = (kern_hyp_va(&kvm_vgic_global_state))->nr_lr;
+	int nr_lr = hyp_symbol_addr(kvm_vgic_global_state)->nr_lr;
 	u32 elrsr0, elrsr1;
 
 	elrsr0 = readl_relaxed(base + GICH_ELRSR0);
@@ -140,7 +140,7 @@ int __hyp_text __vgic_v2_perform_cpuif_access(struct kvm_vcpu *vcpu)
 		return -1;
 
 	rd = kvm_vcpu_dabt_get_rd(vcpu);
-	addr  = kern_hyp_va((kern_hyp_va(&kvm_vgic_global_state))->vcpu_base_va);
+	addr  = kern_hyp_va(hyp_symbol_addr(kvm_vgic_global_state)->vcpu_base_va);
 	addr += fault_ipa - vgic->vgic_cpu_base;
 
 	if (kvm_vcpu_dabt_iswrite(vcpu)) {
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 06/26] KVM: arm/arm64: Do not use kern_hyp_va() with kvm_vgic_global_state
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

kvm_vgic_global_state is part of the read-only section, and is
usually accessed using a PC-relative address generation (adrp + add).

It is thus useless to use kern_hyp_va() on it, and actively problematic
if kern_hyp_va() becomes non-idempotent. On the other hand, there is
no way that the compiler is going to guarantee that such access is
always PC relative.

So let's bite the bullet and provide our own accessor.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm/include/asm/kvm_mmu.h   |  7 +++++++
 arch/arm64/include/asm/kvm_mmu.h | 20 ++++++++++++++++++++
 virt/kvm/arm/hyp/vgic-v2-sr.c    |  4 ++--
 3 files changed, 29 insertions(+), 2 deletions(-)

diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
index de1b919404e4..a6808d2869f5 100644
--- a/arch/arm/include/asm/kvm_mmu.h
+++ b/arch/arm/include/asm/kvm_mmu.h
@@ -28,6 +28,13 @@
  */
 #define kern_hyp_va(kva)	(kva)
 
+/* Resolving symbol addresses in a PC-relative way is easy... */
+#define hyp_symbol_addr(s)						\
+	({								\
+		typeof(s) *addr = &(s);					\
+		addr;							\
+	})
+
 /*
  * KVM_MMU_CACHE_MIN_PAGES is the number of stage2 page table translation levels.
  */
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index e3bc1d0a5e93..7120bf3f22c7 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -110,6 +110,26 @@ static inline unsigned long __kern_hyp_va(unsigned long v)
 
 #define kern_hyp_va(v) 	((typeof(v))(__kern_hyp_va((unsigned long)(v))))
 
+/*
+ * Obtain the PC-relative address of a kernel symbol
+ * s: symbol
+ *
+ * The goal of this macro is to return a symbol's address based on a
+ * PC-relative computation, as opposed to a loading the VA from a
+ * constant pool or something similar. This works well for HYP, as an
+ * absolute VA is guaranteed to be wrong. Only use this if trying to
+ * obtain the address of a symbol (i.e. not something you obtained by
+ * following a pointer).
+ */
+#define hyp_symbol_addr(s)						\
+	({								\
+		typeof(s) *addr;					\
+		asm volatile("adrp	%0, %1\n"			\
+			     "add	%0, %0, :lo12:%1\n"		\
+			     : "=r" (addr) : "S" (&s));			\
+		addr;							\
+	})
+
 /*
  * We currently only support a 40bit IPA.
  */
diff --git a/virt/kvm/arm/hyp/vgic-v2-sr.c b/virt/kvm/arm/hyp/vgic-v2-sr.c
index 4fe6e797e8b3..a6ca049d9651 100644
--- a/virt/kvm/arm/hyp/vgic-v2-sr.c
+++ b/virt/kvm/arm/hyp/vgic-v2-sr.c
@@ -26,7 +26,7 @@
 static void __hyp_text save_elrsr(struct kvm_vcpu *vcpu, void __iomem *base)
 {
 	struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
-	int nr_lr = (kern_hyp_va(&kvm_vgic_global_state))->nr_lr;
+	int nr_lr = hyp_symbol_addr(kvm_vgic_global_state)->nr_lr;
 	u32 elrsr0, elrsr1;
 
 	elrsr0 = readl_relaxed(base + GICH_ELRSR0);
@@ -140,7 +140,7 @@ int __hyp_text __vgic_v2_perform_cpuif_access(struct kvm_vcpu *vcpu)
 		return -1;
 
 	rd = kvm_vcpu_dabt_get_rd(vcpu);
-	addr  = kern_hyp_va((kern_hyp_va(&kvm_vgic_global_state))->vcpu_base_va);
+	addr  = kern_hyp_va(hyp_symbol_addr(kvm_vgic_global_state)->vcpu_base_va);
 	addr += fault_ipa - vgic->vgic_cpu_base;
 
 	if (kvm_vcpu_dabt_iswrite(vcpu)) {
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 07/26] KVM: arm/arm64: Demote HYP VA range display to being a debug feature
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Mark Rutland, Peter Maydell, Andrew Jones, Christoffer Dall,
	Steve Capper, Catalin Marinas, Will Deacon, Kristina Martsenko,
	James Morse

Displaying the HYP VA information is slightly counterproductive when
using VA randomization. Turn it into a debug feature only, and adjust
the last displayed value to reflect the top of RAM instead of ~0.

Acked-by: Christoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 virt/kvm/arm/mmu.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index ec62d1cccab7..711473cd1097 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -1810,9 +1810,10 @@ int kvm_mmu_init(void)
 	 */
 	BUG_ON((hyp_idmap_start ^ (hyp_idmap_end - 1)) & PAGE_MASK);
 
-	kvm_info("IDMAP page: %lx\n", hyp_idmap_start);
-	kvm_info("HYP VA range: %lx:%lx\n",
-		 kern_hyp_va(PAGE_OFFSET), kern_hyp_va(~0UL));
+	kvm_debug("IDMAP page: %lx\n", hyp_idmap_start);
+	kvm_debug("HYP VA range: %lx:%lx\n",
+		  kern_hyp_va(PAGE_OFFSET),
+		  kern_hyp_va((unsigned long)high_memory - 1));
 
 	if (hyp_idmap_start >= kern_hyp_va(PAGE_OFFSET) &&
 	    hyp_idmap_start <  kern_hyp_va(~0UL) &&
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 07/26] KVM: arm/arm64: Demote HYP VA range display to being a debug feature
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

Displaying the HYP VA information is slightly counterproductive when
using VA randomization. Turn it into a debug feature only, and adjust
the last displayed value to reflect the top of RAM instead of ~0.

Acked-by: Christoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 virt/kvm/arm/mmu.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index ec62d1cccab7..711473cd1097 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -1810,9 +1810,10 @@ int kvm_mmu_init(void)
 	 */
 	BUG_ON((hyp_idmap_start ^ (hyp_idmap_end - 1)) & PAGE_MASK);
 
-	kvm_info("IDMAP page: %lx\n", hyp_idmap_start);
-	kvm_info("HYP VA range: %lx:%lx\n",
-		 kern_hyp_va(PAGE_OFFSET), kern_hyp_va(~0UL));
+	kvm_debug("IDMAP page: %lx\n", hyp_idmap_start);
+	kvm_debug("HYP VA range: %lx:%lx\n",
+		  kern_hyp_va(PAGE_OFFSET),
+		  kern_hyp_va((unsigned long)high_memory - 1));
 
 	if (hyp_idmap_start >= kern_hyp_va(PAGE_OFFSET) &&
 	    hyp_idmap_start <  kern_hyp_va(~0UL) &&
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 08/26] KVM: arm/arm64: Move ioremap calls to create_hyp_io_mappings
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Catalin Marinas, Will Deacon, Kristina Martsenko

Both HYP io mappings call ioremap, followed by create_hyp_io_mappings.
Let's move the ioremap call into create_hyp_io_mappings itself, which
simplifies the code a bit and allows for further refactoring.

Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm/include/asm/kvm_mmu.h   |  3 ++-
 arch/arm64/include/asm/kvm_mmu.h |  3 ++-
 virt/kvm/arm/mmu.c               | 24 ++++++++++++++----------
 virt/kvm/arm/vgic/vgic-v2.c      | 31 ++++++++-----------------------
 4 files changed, 26 insertions(+), 35 deletions(-)

diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
index a6808d2869f5..477c5ed426ef 100644
--- a/arch/arm/include/asm/kvm_mmu.h
+++ b/arch/arm/include/asm/kvm_mmu.h
@@ -50,7 +50,8 @@
 #include <asm/stage2_pgtable.h>
 
 int create_hyp_mappings(void *from, void *to, pgprot_t prot);
-int create_hyp_io_mappings(void *from, void *to, phys_addr_t);
+int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
+			   void __iomem **kaddr);
 void free_hyp_pgds(void);
 
 void stage2_unmap_vm(struct kvm *kvm);
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 7120bf3f22c7..5885564f9863 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -140,7 +140,8 @@ static inline unsigned long __kern_hyp_va(unsigned long v)
 #include <asm/stage2_pgtable.h>
 
 int create_hyp_mappings(void *from, void *to, pgprot_t prot);
-int create_hyp_io_mappings(void *from, void *to, phys_addr_t);
+int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
+			   void __iomem **kaddr);
 void free_hyp_pgds(void);
 
 void stage2_unmap_vm(struct kvm *kvm);
diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index 711473cd1097..d58388d8550e 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -709,26 +709,30 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot)
 }
 
 /**
- * create_hyp_io_mappings - duplicate a kernel IO mapping into Hyp mode
- * @from:	The kernel start VA of the range
- * @to:		The kernel end VA of the range (exclusive)
+ * create_hyp_io_mappings - Map IO into both kernel and HYP
  * @phys_addr:	The physical start address which gets mapped
+ * @size:	Size of the region being mapped
+ * @kaddr:	Kernel VA for this mapping
  *
  * The resulting HYP VA is the same as the kernel VA, modulo
  * HYP_PAGE_OFFSET.
  */
-int create_hyp_io_mappings(void *from, void *to, phys_addr_t phys_addr)
+int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
+			   void __iomem **kaddr)
 {
-	unsigned long start = kern_hyp_va((unsigned long)from);
-	unsigned long end = kern_hyp_va((unsigned long)to);
+	unsigned long start, end;
 
-	if (is_kernel_in_hyp_mode())
+	*kaddr = ioremap(phys_addr, size);
+	if (!*kaddr)
+		return -ENOMEM;
+
+	if (is_kernel_in_hyp_mode()) {
 		return 0;
+	}
 
-	/* Check for a valid kernel IO mapping */
-	if (!is_vmalloc_addr(from) || !is_vmalloc_addr(to - 1))
-		return -EINVAL;
 
+	start = kern_hyp_va((unsigned long)*kaddr);
+	end = kern_hyp_va((unsigned long)*kaddr + size);
 	return __create_hyp_mappings(hyp_pgd, PTRS_PER_PGD, start, end,
 				     __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
 }
diff --git a/virt/kvm/arm/vgic/vgic-v2.c b/virt/kvm/arm/vgic/vgic-v2.c
index c32d7b93ffd1..21f963f9780e 100644
--- a/virt/kvm/arm/vgic/vgic-v2.c
+++ b/virt/kvm/arm/vgic/vgic-v2.c
@@ -361,16 +361,10 @@ int vgic_v2_probe(const struct gic_kvm_info *info)
 	if (!PAGE_ALIGNED(info->vcpu.start) ||
 	    !PAGE_ALIGNED(resource_size(&info->vcpu))) {
 		kvm_info("GICV region size/alignment is unsafe, using trapping (reduced performance)\n");
-		kvm_vgic_global_state.vcpu_base_va = ioremap(info->vcpu.start,
-							     resource_size(&info->vcpu));
-		if (!kvm_vgic_global_state.vcpu_base_va) {
-			kvm_err("Cannot ioremap GICV\n");
-			return -ENOMEM;
-		}
 
-		ret = create_hyp_io_mappings(kvm_vgic_global_state.vcpu_base_va,
-					     kvm_vgic_global_state.vcpu_base_va + resource_size(&info->vcpu),
-					     info->vcpu.start);
+		ret = create_hyp_io_mappings(info->vcpu.start,
+					     resource_size(&info->vcpu),
+					     &kvm_vgic_global_state.vcpu_base_va);
 		if (ret) {
 			kvm_err("Cannot map GICV into hyp\n");
 			goto out;
@@ -379,26 +373,17 @@ int vgic_v2_probe(const struct gic_kvm_info *info)
 		static_branch_enable(&vgic_v2_cpuif_trap);
 	}
 
-	kvm_vgic_global_state.vctrl_base = ioremap(info->vctrl.start,
-						   resource_size(&info->vctrl));
-	if (!kvm_vgic_global_state.vctrl_base) {
-		kvm_err("Cannot ioremap GICH\n");
-		ret = -ENOMEM;
+	ret = create_hyp_io_mappings(info->vctrl.start,
+				     resource_size(&info->vctrl),
+				     &kvm_vgic_global_state.vctrl_base);
+	if (ret) {
+		kvm_err("Cannot map VCTRL into hyp\n");
 		goto out;
 	}
 
 	vtr = readl_relaxed(kvm_vgic_global_state.vctrl_base + GICH_VTR);
 	kvm_vgic_global_state.nr_lr = (vtr & 0x3f) + 1;
 
-	ret = create_hyp_io_mappings(kvm_vgic_global_state.vctrl_base,
-				     kvm_vgic_global_state.vctrl_base +
-					 resource_size(&info->vctrl),
-				     info->vctrl.start);
-	if (ret) {
-		kvm_err("Cannot map VCTRL into hyp\n");
-		goto out;
-	}
-
 	ret = kvm_register_vgic_device(KVM_DEV_TYPE_ARM_VGIC_V2);
 	if (ret) {
 		kvm_err("Cannot register GICv2 KVM device\n");
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 08/26] KVM: arm/arm64: Move ioremap calls to create_hyp_io_mappings
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

Both HYP io mappings call ioremap, followed by create_hyp_io_mappings.
Let's move the ioremap call into create_hyp_io_mappings itself, which
simplifies the code a bit and allows for further refactoring.

Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm/include/asm/kvm_mmu.h   |  3 ++-
 arch/arm64/include/asm/kvm_mmu.h |  3 ++-
 virt/kvm/arm/mmu.c               | 24 ++++++++++++++----------
 virt/kvm/arm/vgic/vgic-v2.c      | 31 ++++++++-----------------------
 4 files changed, 26 insertions(+), 35 deletions(-)

diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
index a6808d2869f5..477c5ed426ef 100644
--- a/arch/arm/include/asm/kvm_mmu.h
+++ b/arch/arm/include/asm/kvm_mmu.h
@@ -50,7 +50,8 @@
 #include <asm/stage2_pgtable.h>
 
 int create_hyp_mappings(void *from, void *to, pgprot_t prot);
-int create_hyp_io_mappings(void *from, void *to, phys_addr_t);
+int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
+			   void __iomem **kaddr);
 void free_hyp_pgds(void);
 
 void stage2_unmap_vm(struct kvm *kvm);
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 7120bf3f22c7..5885564f9863 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -140,7 +140,8 @@ static inline unsigned long __kern_hyp_va(unsigned long v)
 #include <asm/stage2_pgtable.h>
 
 int create_hyp_mappings(void *from, void *to, pgprot_t prot);
-int create_hyp_io_mappings(void *from, void *to, phys_addr_t);
+int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
+			   void __iomem **kaddr);
 void free_hyp_pgds(void);
 
 void stage2_unmap_vm(struct kvm *kvm);
diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index 711473cd1097..d58388d8550e 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -709,26 +709,30 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot)
 }
 
 /**
- * create_hyp_io_mappings - duplicate a kernel IO mapping into Hyp mode
- * @from:	The kernel start VA of the range
- * @to:		The kernel end VA of the range (exclusive)
+ * create_hyp_io_mappings - Map IO into both kernel and HYP
  * @phys_addr:	The physical start address which gets mapped
+ * @size:	Size of the region being mapped
+ * @kaddr:	Kernel VA for this mapping
  *
  * The resulting HYP VA is the same as the kernel VA, modulo
  * HYP_PAGE_OFFSET.
  */
-int create_hyp_io_mappings(void *from, void *to, phys_addr_t phys_addr)
+int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
+			   void __iomem **kaddr)
 {
-	unsigned long start = kern_hyp_va((unsigned long)from);
-	unsigned long end = kern_hyp_va((unsigned long)to);
+	unsigned long start, end;
 
-	if (is_kernel_in_hyp_mode())
+	*kaddr = ioremap(phys_addr, size);
+	if (!*kaddr)
+		return -ENOMEM;
+
+	if (is_kernel_in_hyp_mode()) {
 		return 0;
+	}
 
-	/* Check for a valid kernel IO mapping */
-	if (!is_vmalloc_addr(from) || !is_vmalloc_addr(to - 1))
-		return -EINVAL;
 
+	start = kern_hyp_va((unsigned long)*kaddr);
+	end = kern_hyp_va((unsigned long)*kaddr + size);
 	return __create_hyp_mappings(hyp_pgd, PTRS_PER_PGD, start, end,
 				     __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
 }
diff --git a/virt/kvm/arm/vgic/vgic-v2.c b/virt/kvm/arm/vgic/vgic-v2.c
index c32d7b93ffd1..21f963f9780e 100644
--- a/virt/kvm/arm/vgic/vgic-v2.c
+++ b/virt/kvm/arm/vgic/vgic-v2.c
@@ -361,16 +361,10 @@ int vgic_v2_probe(const struct gic_kvm_info *info)
 	if (!PAGE_ALIGNED(info->vcpu.start) ||
 	    !PAGE_ALIGNED(resource_size(&info->vcpu))) {
 		kvm_info("GICV region size/alignment is unsafe, using trapping (reduced performance)\n");
-		kvm_vgic_global_state.vcpu_base_va = ioremap(info->vcpu.start,
-							     resource_size(&info->vcpu));
-		if (!kvm_vgic_global_state.vcpu_base_va) {
-			kvm_err("Cannot ioremap GICV\n");
-			return -ENOMEM;
-		}
 
-		ret = create_hyp_io_mappings(kvm_vgic_global_state.vcpu_base_va,
-					     kvm_vgic_global_state.vcpu_base_va + resource_size(&info->vcpu),
-					     info->vcpu.start);
+		ret = create_hyp_io_mappings(info->vcpu.start,
+					     resource_size(&info->vcpu),
+					     &kvm_vgic_global_state.vcpu_base_va);
 		if (ret) {
 			kvm_err("Cannot map GICV into hyp\n");
 			goto out;
@@ -379,26 +373,17 @@ int vgic_v2_probe(const struct gic_kvm_info *info)
 		static_branch_enable(&vgic_v2_cpuif_trap);
 	}
 
-	kvm_vgic_global_state.vctrl_base = ioremap(info->vctrl.start,
-						   resource_size(&info->vctrl));
-	if (!kvm_vgic_global_state.vctrl_base) {
-		kvm_err("Cannot ioremap GICH\n");
-		ret = -ENOMEM;
+	ret = create_hyp_io_mappings(info->vctrl.start,
+				     resource_size(&info->vctrl),
+				     &kvm_vgic_global_state.vctrl_base);
+	if (ret) {
+		kvm_err("Cannot map VCTRL into hyp\n");
 		goto out;
 	}
 
 	vtr = readl_relaxed(kvm_vgic_global_state.vctrl_base + GICH_VTR);
 	kvm_vgic_global_state.nr_lr = (vtr & 0x3f) + 1;
 
-	ret = create_hyp_io_mappings(kvm_vgic_global_state.vctrl_base,
-				     kvm_vgic_global_state.vctrl_base +
-					 resource_size(&info->vctrl),
-				     info->vctrl.start);
-	if (ret) {
-		kvm_err("Cannot map VCTRL into hyp\n");
-		goto out;
-	}
-
 	ret = kvm_register_vgic_device(KVM_DEV_TYPE_ARM_VGIC_V2);
 	if (ret) {
 		kvm_err("Cannot register GICv2 KVM device\n");
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 09/26] KVM: arm/arm64: Keep GICv2 HYP VAs in kvm_vgic_global_state
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Catalin Marinas, Will Deacon, Kristina Martsenko

As we're about to change the way we map devices at HYP, we need
to move away from kern_hyp_va on an IO address.

One way of achieving this is to store the VAs in kvm_vgic_global_state,
and use that directly from the HYP code. This requires a small change
to create_hyp_io_mappings so that it can also return a HYP VA.

We take this opportunity to nuke the vctrl_base field in the emulated
distributor, as it is not used anymore.

Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm/include/asm/kvm_mmu.h   |  3 ++-
 arch/arm64/include/asm/kvm_mmu.h |  3 ++-
 include/kvm/arm_vgic.h           | 12 ++++++------
 virt/kvm/arm/hyp/vgic-v2-sr.c    | 10 +++-------
 virt/kvm/arm/mmu.c               | 20 +++++++++++++++-----
 virt/kvm/arm/vgic/vgic-init.c    |  6 ------
 virt/kvm/arm/vgic/vgic-v2.c      | 13 +++++++------
 7 files changed, 35 insertions(+), 32 deletions(-)

diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
index 477c5ed426ef..5b12a3d2755e 100644
--- a/arch/arm/include/asm/kvm_mmu.h
+++ b/arch/arm/include/asm/kvm_mmu.h
@@ -51,7 +51,8 @@
 
 int create_hyp_mappings(void *from, void *to, pgprot_t prot);
 int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
-			   void __iomem **kaddr);
+			   void __iomem **kaddr,
+			   void __iomem **haddr);
 void free_hyp_pgds(void);
 
 void stage2_unmap_vm(struct kvm *kvm);
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 5885564f9863..c4934470403b 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -141,7 +141,8 @@ static inline unsigned long __kern_hyp_va(unsigned long v)
 
 int create_hyp_mappings(void *from, void *to, pgprot_t prot);
 int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
-			   void __iomem **kaddr);
+			   void __iomem **kaddr,
+			   void __iomem **haddr);
 void free_hyp_pgds(void);
 
 void stage2_unmap_vm(struct kvm *kvm);
diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
index cdbd142ca7f2..b3fcb6cd62f7 100644
--- a/include/kvm/arm_vgic.h
+++ b/include/kvm/arm_vgic.h
@@ -57,11 +57,15 @@ struct vgic_global {
 	/* Physical address of vgic virtual cpu interface */
 	phys_addr_t		vcpu_base;
 
-	/* GICV mapping */
+	/* GICV mapping, kernel VA */
 	void __iomem		*vcpu_base_va;
+	/* GICV mapping, HYP VA */
+	void __iomem		*vcpu_hyp_va;
 
-	/* virtual control interface mapping */
+	/* virtual control interface mapping, kernel VA */
 	void __iomem		*vctrl_base;
+	/* virtual control interface mapping, HYP VA */
+	void __iomem		*vctrl_hyp;
 
 	/* Number of implemented list registers */
 	int			nr_lr;
@@ -209,10 +213,6 @@ struct vgic_dist {
 
 	int			nr_spis;
 
-	/* TODO: Consider moving to global state */
-	/* Virtual control interface mapping */
-	void __iomem		*vctrl_base;
-
 	/* base addresses in guest physical address space: */
 	gpa_t			vgic_dist_base;		/* distributor */
 	union {
diff --git a/virt/kvm/arm/hyp/vgic-v2-sr.c b/virt/kvm/arm/hyp/vgic-v2-sr.c
index a6ca049d9651..7a14b3bf6f5e 100644
--- a/virt/kvm/arm/hyp/vgic-v2-sr.c
+++ b/virt/kvm/arm/hyp/vgic-v2-sr.c
@@ -57,10 +57,8 @@ static void __hyp_text save_lrs(struct kvm_vcpu *vcpu, void __iomem *base)
 /* vcpu is already in the HYP VA space */
 void __hyp_text __vgic_v2_save_state(struct kvm_vcpu *vcpu)
 {
-	struct kvm *kvm = kern_hyp_va(vcpu->kvm);
 	struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
-	struct vgic_dist *vgic = &kvm->arch.vgic;
-	void __iomem *base = kern_hyp_va(vgic->vctrl_base);
+	void __iomem *base = hyp_symbol_addr(kvm_vgic_global_state)->vctrl_hyp;
 	u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
 
 	if (!base)
@@ -82,10 +80,8 @@ void __hyp_text __vgic_v2_save_state(struct kvm_vcpu *vcpu)
 /* vcpu is already in the HYP VA space */
 void __hyp_text __vgic_v2_restore_state(struct kvm_vcpu *vcpu)
 {
-	struct kvm *kvm = kern_hyp_va(vcpu->kvm);
 	struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
-	struct vgic_dist *vgic = &kvm->arch.vgic;
-	void __iomem *base = kern_hyp_va(vgic->vctrl_base);
+	void __iomem *base = hyp_symbol_addr(kvm_vgic_global_state)->vctrl_hyp;
 	int i;
 	u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
 
@@ -140,7 +136,7 @@ int __hyp_text __vgic_v2_perform_cpuif_access(struct kvm_vcpu *vcpu)
 		return -1;
 
 	rd = kvm_vcpu_dabt_get_rd(vcpu);
-	addr  = kern_hyp_va(hyp_symbol_addr(kvm_vgic_global_state)->vcpu_base_va);
+	addr  = hyp_symbol_addr(kvm_vgic_global_state)->vcpu_hyp_va;
 	addr += fault_ipa - vgic->vgic_cpu_base;
 
 	if (kvm_vcpu_dabt_iswrite(vcpu)) {
diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index d58388d8550e..0e5cfffb4c21 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -713,28 +713,38 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot)
  * @phys_addr:	The physical start address which gets mapped
  * @size:	Size of the region being mapped
  * @kaddr:	Kernel VA for this mapping
- *
- * The resulting HYP VA is the same as the kernel VA, modulo
- * HYP_PAGE_OFFSET.
+ * @haddr:	HYP VA for this mapping
  */
 int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
-			   void __iomem **kaddr)
+			   void __iomem **kaddr,
+			   void __iomem **haddr)
 {
 	unsigned long start, end;
+	int ret;
 
 	*kaddr = ioremap(phys_addr, size);
 	if (!*kaddr)
 		return -ENOMEM;
 
 	if (is_kernel_in_hyp_mode()) {
+		*haddr = *kaddr;
 		return 0;
 	}
 
 
 	start = kern_hyp_va((unsigned long)*kaddr);
 	end = kern_hyp_va((unsigned long)*kaddr + size);
-	return __create_hyp_mappings(hyp_pgd, PTRS_PER_PGD, start, end,
+	ret = __create_hyp_mappings(hyp_pgd, PTRS_PER_PGD, start, end,
 				     __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
+
+	if (ret) {
+		iounmap(*kaddr);
+		*kaddr = NULL;
+		return ret;
+	}
+
+	*haddr = (void __iomem *)start;
+	return 0;
 }
 
 /**
diff --git a/virt/kvm/arm/vgic/vgic-init.c b/virt/kvm/arm/vgic/vgic-init.c
index 743ca5cb05ef..b673a6c72ef8 100644
--- a/virt/kvm/arm/vgic/vgic-init.c
+++ b/virt/kvm/arm/vgic/vgic-init.c
@@ -166,12 +166,6 @@ int kvm_vgic_create(struct kvm *kvm, u32 type)
 	kvm->arch.vgic.in_kernel = true;
 	kvm->arch.vgic.vgic_model = type;
 
-	/*
-	 * kvm_vgic_global_state.vctrl_base is set on vgic probe (kvm_arch_init)
-	 * it is stored in distributor struct for asm save/restore purpose
-	 */
-	kvm->arch.vgic.vctrl_base = kvm_vgic_global_state.vctrl_base;
-
 	kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
 	kvm->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF;
 	kvm->arch.vgic.vgic_redist_base = VGIC_ADDR_UNDEF;
diff --git a/virt/kvm/arm/vgic/vgic-v2.c b/virt/kvm/arm/vgic/vgic-v2.c
index 21f963f9780e..38406825e663 100644
--- a/virt/kvm/arm/vgic/vgic-v2.c
+++ b/virt/kvm/arm/vgic/vgic-v2.c
@@ -364,7 +364,8 @@ int vgic_v2_probe(const struct gic_kvm_info *info)
 
 		ret = create_hyp_io_mappings(info->vcpu.start,
 					     resource_size(&info->vcpu),
-					     &kvm_vgic_global_state.vcpu_base_va);
+					     &kvm_vgic_global_state.vcpu_base_va,
+					     &kvm_vgic_global_state.vcpu_hyp_va);
 		if (ret) {
 			kvm_err("Cannot map GICV into hyp\n");
 			goto out;
@@ -375,7 +376,8 @@ int vgic_v2_probe(const struct gic_kvm_info *info)
 
 	ret = create_hyp_io_mappings(info->vctrl.start,
 				     resource_size(&info->vctrl),
-				     &kvm_vgic_global_state.vctrl_base);
+				     &kvm_vgic_global_state.vctrl_base,
+				     &kvm_vgic_global_state.vctrl_hyp);
 	if (ret) {
 		kvm_err("Cannot map VCTRL into hyp\n");
 		goto out;
@@ -410,15 +412,14 @@ int vgic_v2_probe(const struct gic_kvm_info *info)
 void vgic_v2_load(struct kvm_vcpu *vcpu)
 {
 	struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
-	struct vgic_dist *vgic = &vcpu->kvm->arch.vgic;
 
-	writel_relaxed(cpu_if->vgic_vmcr, vgic->vctrl_base + GICH_VMCR);
+	writel_relaxed(cpu_if->vgic_vmcr,
+		       kvm_vgic_global_state.vctrl_base + GICH_VMCR);
 }
 
 void vgic_v2_put(struct kvm_vcpu *vcpu)
 {
 	struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
-	struct vgic_dist *vgic = &vcpu->kvm->arch.vgic;
 
-	cpu_if->vgic_vmcr = readl_relaxed(vgic->vctrl_base + GICH_VMCR);
+	cpu_if->vgic_vmcr = readl_relaxed(kvm_vgic_global_state.vctrl_base + GICH_VMCR);
 }
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 09/26] KVM: arm/arm64: Keep GICv2 HYP VAs in kvm_vgic_global_state
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

As we're about to change the way we map devices at HYP, we need
to move away from kern_hyp_va on an IO address.

One way of achieving this is to store the VAs in kvm_vgic_global_state,
and use that directly from the HYP code. This requires a small change
to create_hyp_io_mappings so that it can also return a HYP VA.

We take this opportunity to nuke the vctrl_base field in the emulated
distributor, as it is not used anymore.

Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm/include/asm/kvm_mmu.h   |  3 ++-
 arch/arm64/include/asm/kvm_mmu.h |  3 ++-
 include/kvm/arm_vgic.h           | 12 ++++++------
 virt/kvm/arm/hyp/vgic-v2-sr.c    | 10 +++-------
 virt/kvm/arm/mmu.c               | 20 +++++++++++++++-----
 virt/kvm/arm/vgic/vgic-init.c    |  6 ------
 virt/kvm/arm/vgic/vgic-v2.c      | 13 +++++++------
 7 files changed, 35 insertions(+), 32 deletions(-)

diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
index 477c5ed426ef..5b12a3d2755e 100644
--- a/arch/arm/include/asm/kvm_mmu.h
+++ b/arch/arm/include/asm/kvm_mmu.h
@@ -51,7 +51,8 @@
 
 int create_hyp_mappings(void *from, void *to, pgprot_t prot);
 int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
-			   void __iomem **kaddr);
+			   void __iomem **kaddr,
+			   void __iomem **haddr);
 void free_hyp_pgds(void);
 
 void stage2_unmap_vm(struct kvm *kvm);
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 5885564f9863..c4934470403b 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -141,7 +141,8 @@ static inline unsigned long __kern_hyp_va(unsigned long v)
 
 int create_hyp_mappings(void *from, void *to, pgprot_t prot);
 int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
-			   void __iomem **kaddr);
+			   void __iomem **kaddr,
+			   void __iomem **haddr);
 void free_hyp_pgds(void);
 
 void stage2_unmap_vm(struct kvm *kvm);
diff --git a/include/kvm/arm_vgic.h b/include/kvm/arm_vgic.h
index cdbd142ca7f2..b3fcb6cd62f7 100644
--- a/include/kvm/arm_vgic.h
+++ b/include/kvm/arm_vgic.h
@@ -57,11 +57,15 @@ struct vgic_global {
 	/* Physical address of vgic virtual cpu interface */
 	phys_addr_t		vcpu_base;
 
-	/* GICV mapping */
+	/* GICV mapping, kernel VA */
 	void __iomem		*vcpu_base_va;
+	/* GICV mapping, HYP VA */
+	void __iomem		*vcpu_hyp_va;
 
-	/* virtual control interface mapping */
+	/* virtual control interface mapping, kernel VA */
 	void __iomem		*vctrl_base;
+	/* virtual control interface mapping, HYP VA */
+	void __iomem		*vctrl_hyp;
 
 	/* Number of implemented list registers */
 	int			nr_lr;
@@ -209,10 +213,6 @@ struct vgic_dist {
 
 	int			nr_spis;
 
-	/* TODO: Consider moving to global state */
-	/* Virtual control interface mapping */
-	void __iomem		*vctrl_base;
-
 	/* base addresses in guest physical address space: */
 	gpa_t			vgic_dist_base;		/* distributor */
 	union {
diff --git a/virt/kvm/arm/hyp/vgic-v2-sr.c b/virt/kvm/arm/hyp/vgic-v2-sr.c
index a6ca049d9651..7a14b3bf6f5e 100644
--- a/virt/kvm/arm/hyp/vgic-v2-sr.c
+++ b/virt/kvm/arm/hyp/vgic-v2-sr.c
@@ -57,10 +57,8 @@ static void __hyp_text save_lrs(struct kvm_vcpu *vcpu, void __iomem *base)
 /* vcpu is already in the HYP VA space */
 void __hyp_text __vgic_v2_save_state(struct kvm_vcpu *vcpu)
 {
-	struct kvm *kvm = kern_hyp_va(vcpu->kvm);
 	struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
-	struct vgic_dist *vgic = &kvm->arch.vgic;
-	void __iomem *base = kern_hyp_va(vgic->vctrl_base);
+	void __iomem *base = hyp_symbol_addr(kvm_vgic_global_state)->vctrl_hyp;
 	u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
 
 	if (!base)
@@ -82,10 +80,8 @@ void __hyp_text __vgic_v2_save_state(struct kvm_vcpu *vcpu)
 /* vcpu is already in the HYP VA space */
 void __hyp_text __vgic_v2_restore_state(struct kvm_vcpu *vcpu)
 {
-	struct kvm *kvm = kern_hyp_va(vcpu->kvm);
 	struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
-	struct vgic_dist *vgic = &kvm->arch.vgic;
-	void __iomem *base = kern_hyp_va(vgic->vctrl_base);
+	void __iomem *base = hyp_symbol_addr(kvm_vgic_global_state)->vctrl_hyp;
 	int i;
 	u64 used_lrs = vcpu->arch.vgic_cpu.used_lrs;
 
@@ -140,7 +136,7 @@ int __hyp_text __vgic_v2_perform_cpuif_access(struct kvm_vcpu *vcpu)
 		return -1;
 
 	rd = kvm_vcpu_dabt_get_rd(vcpu);
-	addr  = kern_hyp_va(hyp_symbol_addr(kvm_vgic_global_state)->vcpu_base_va);
+	addr  = hyp_symbol_addr(kvm_vgic_global_state)->vcpu_hyp_va;
 	addr += fault_ipa - vgic->vgic_cpu_base;
 
 	if (kvm_vcpu_dabt_iswrite(vcpu)) {
diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index d58388d8550e..0e5cfffb4c21 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -713,28 +713,38 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot)
  * @phys_addr:	The physical start address which gets mapped
  * @size:	Size of the region being mapped
  * @kaddr:	Kernel VA for this mapping
- *
- * The resulting HYP VA is the same as the kernel VA, modulo
- * HYP_PAGE_OFFSET.
+ * @haddr:	HYP VA for this mapping
  */
 int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
-			   void __iomem **kaddr)
+			   void __iomem **kaddr,
+			   void __iomem **haddr)
 {
 	unsigned long start, end;
+	int ret;
 
 	*kaddr = ioremap(phys_addr, size);
 	if (!*kaddr)
 		return -ENOMEM;
 
 	if (is_kernel_in_hyp_mode()) {
+		*haddr = *kaddr;
 		return 0;
 	}
 
 
 	start = kern_hyp_va((unsigned long)*kaddr);
 	end = kern_hyp_va((unsigned long)*kaddr + size);
-	return __create_hyp_mappings(hyp_pgd, PTRS_PER_PGD, start, end,
+	ret = __create_hyp_mappings(hyp_pgd, PTRS_PER_PGD, start, end,
 				     __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
+
+	if (ret) {
+		iounmap(*kaddr);
+		*kaddr = NULL;
+		return ret;
+	}
+
+	*haddr = (void __iomem *)start;
+	return 0;
 }
 
 /**
diff --git a/virt/kvm/arm/vgic/vgic-init.c b/virt/kvm/arm/vgic/vgic-init.c
index 743ca5cb05ef..b673a6c72ef8 100644
--- a/virt/kvm/arm/vgic/vgic-init.c
+++ b/virt/kvm/arm/vgic/vgic-init.c
@@ -166,12 +166,6 @@ int kvm_vgic_create(struct kvm *kvm, u32 type)
 	kvm->arch.vgic.in_kernel = true;
 	kvm->arch.vgic.vgic_model = type;
 
-	/*
-	 * kvm_vgic_global_state.vctrl_base is set on vgic probe (kvm_arch_init)
-	 * it is stored in distributor struct for asm save/restore purpose
-	 */
-	kvm->arch.vgic.vctrl_base = kvm_vgic_global_state.vctrl_base;
-
 	kvm->arch.vgic.vgic_dist_base = VGIC_ADDR_UNDEF;
 	kvm->arch.vgic.vgic_cpu_base = VGIC_ADDR_UNDEF;
 	kvm->arch.vgic.vgic_redist_base = VGIC_ADDR_UNDEF;
diff --git a/virt/kvm/arm/vgic/vgic-v2.c b/virt/kvm/arm/vgic/vgic-v2.c
index 21f963f9780e..38406825e663 100644
--- a/virt/kvm/arm/vgic/vgic-v2.c
+++ b/virt/kvm/arm/vgic/vgic-v2.c
@@ -364,7 +364,8 @@ int vgic_v2_probe(const struct gic_kvm_info *info)
 
 		ret = create_hyp_io_mappings(info->vcpu.start,
 					     resource_size(&info->vcpu),
-					     &kvm_vgic_global_state.vcpu_base_va);
+					     &kvm_vgic_global_state.vcpu_base_va,
+					     &kvm_vgic_global_state.vcpu_hyp_va);
 		if (ret) {
 			kvm_err("Cannot map GICV into hyp\n");
 			goto out;
@@ -375,7 +376,8 @@ int vgic_v2_probe(const struct gic_kvm_info *info)
 
 	ret = create_hyp_io_mappings(info->vctrl.start,
 				     resource_size(&info->vctrl),
-				     &kvm_vgic_global_state.vctrl_base);
+				     &kvm_vgic_global_state.vctrl_base,
+				     &kvm_vgic_global_state.vctrl_hyp);
 	if (ret) {
 		kvm_err("Cannot map VCTRL into hyp\n");
 		goto out;
@@ -410,15 +412,14 @@ int vgic_v2_probe(const struct gic_kvm_info *info)
 void vgic_v2_load(struct kvm_vcpu *vcpu)
 {
 	struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
-	struct vgic_dist *vgic = &vcpu->kvm->arch.vgic;
 
-	writel_relaxed(cpu_if->vgic_vmcr, vgic->vctrl_base + GICH_VMCR);
+	writel_relaxed(cpu_if->vgic_vmcr,
+		       kvm_vgic_global_state.vctrl_base + GICH_VMCR);
 }
 
 void vgic_v2_put(struct kvm_vcpu *vcpu)
 {
 	struct vgic_v2_cpu_if *cpu_if = &vcpu->arch.vgic_cpu.vgic_v2;
-	struct vgic_dist *vgic = &vcpu->kvm->arch.vgic;
 
-	cpu_if->vgic_vmcr = readl_relaxed(vgic->vctrl_base + GICH_VMCR);
+	cpu_if->vgic_vmcr = readl_relaxed(kvm_vgic_global_state.vctrl_base + GICH_VMCR);
 }
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 10/26] KVM: arm/arm64: Fix idmap size and alignment
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Catalin Marinas, Will Deacon, Kristina Martsenko

Although the idmap section of KVM can only be at most 4kB and
must be aligned on a 4kB boundary, the rest of the code expects
it to be page aligned. Things get messy when tearing down the
HYP page tables when PAGE_SIZE is 64K, and the idmap section isn't
64K aligned.

Let's fix this by computing aligned boundaries that the HYP code
will use.

Reported-by: James Morse <james.morse@arm.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 virt/kvm/arm/mmu.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index 0e5cfffb4c21..a9e0513868e9 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -1815,7 +1815,9 @@ int kvm_mmu_init(void)
 	int err;
 
 	hyp_idmap_start = kvm_virt_to_phys(__hyp_idmap_text_start);
+	hyp_idmap_start = ALIGN_DOWN(hyp_idmap_start, PAGE_SIZE);
 	hyp_idmap_end = kvm_virt_to_phys(__hyp_idmap_text_end);
+	hyp_idmap_end = ALIGN(hyp_idmap_end, PAGE_SIZE);
 	hyp_idmap_vector = kvm_virt_to_phys(__kvm_hyp_init);
 
 	/*
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 10/26] KVM: arm/arm64: Fix idmap size and alignment
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

Although the idmap section of KVM can only be at most 4kB and
must be aligned on a 4kB boundary, the rest of the code expects
it to be page aligned. Things get messy when tearing down the
HYP page tables when PAGE_SIZE is 64K, and the idmap section isn't
64K aligned.

Let's fix this by computing aligned boundaries that the HYP code
will use.

Reported-by: James Morse <james.morse@arm.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 virt/kvm/arm/mmu.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index 0e5cfffb4c21..a9e0513868e9 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -1815,7 +1815,9 @@ int kvm_mmu_init(void)
 	int err;
 
 	hyp_idmap_start = kvm_virt_to_phys(__hyp_idmap_text_start);
+	hyp_idmap_start = ALIGN_DOWN(hyp_idmap_start, PAGE_SIZE);
 	hyp_idmap_end = kvm_virt_to_phys(__hyp_idmap_text_end);
+	hyp_idmap_end = ALIGN(hyp_idmap_end, PAGE_SIZE);
 	hyp_idmap_vector = kvm_virt_to_phys(__kvm_hyp_init);
 
 	/*
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 11/26] KVM: arm64: Fix HYP idmap unmap when using 52bit PA
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Catalin Marinas, Will Deacon, Kristina Martsenko

Unmapping the idmap range using 52bit PA is quite broken, as we
don't take into account the right number of PGD entries, and rely
on PTRS_PER_PGD. The result is that pgd_index() truncates the
address, and we end-up in the weed.

Let's introduce a new unmap_hyp_idmap_range() that knows about this,
together with a kvm_pgd_index() helper, which hides a bit of the
complexity of the issue.

Fixes: 98732d1b189b ("KVM: arm/arm64: fix HYP ID map extension to 52 bits")
Reported-by: James Morse <james.morse@arm.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 virt/kvm/arm/mmu.c | 26 +++++++++++++++++++++-----
 1 file changed, 21 insertions(+), 5 deletions(-)

diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index a9e0513868e9..9946f8a38b26 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -479,7 +479,13 @@ static void unmap_hyp_puds(pgd_t *pgd, phys_addr_t addr, phys_addr_t end)
 		clear_hyp_pgd_entry(pgd);
 }
 
-static void unmap_hyp_range(pgd_t *pgdp, phys_addr_t start, u64 size)
+static unsigned int kvm_pgd_index(unsigned long addr, unsigned int ptrs_per_pgd)
+{
+	return (addr >> PGDIR_SHIFT) & (ptrs_per_pgd - 1);
+}
+
+static void __unmap_hyp_range(pgd_t *pgdp, unsigned long ptrs_per_pgd,
+			      phys_addr_t start, u64 size)
 {
 	pgd_t *pgd;
 	phys_addr_t addr = start, end = start + size;
@@ -489,7 +495,7 @@ static void unmap_hyp_range(pgd_t *pgdp, phys_addr_t start, u64 size)
 	 * We don't unmap anything from HYP, except at the hyp tear down.
 	 * Hence, we don't have to invalidate the TLBs here.
 	 */
-	pgd = pgdp + pgd_index(addr);
+	pgd = pgdp + kvm_pgd_index(addr, ptrs_per_pgd);
 	do {
 		next = pgd_addr_end(addr, end);
 		if (!pgd_none(*pgd))
@@ -497,6 +503,16 @@ static void unmap_hyp_range(pgd_t *pgdp, phys_addr_t start, u64 size)
 	} while (pgd++, addr = next, addr != end);
 }
 
+static void unmap_hyp_range(pgd_t *pgdp, phys_addr_t start, u64 size)
+{
+	__unmap_hyp_range(pgdp, PTRS_PER_PGD, start, size);
+}
+
+static void unmap_hyp_idmap_range(pgd_t *pgdp, phys_addr_t start, u64 size)
+{
+	__unmap_hyp_range(pgdp, __kvm_idmap_ptrs_per_pgd(), start, size);
+}
+
 /**
  * free_hyp_pgds - free Hyp-mode page tables
  *
@@ -512,13 +528,13 @@ void free_hyp_pgds(void)
 	mutex_lock(&kvm_hyp_pgd_mutex);
 
 	if (boot_hyp_pgd) {
-		unmap_hyp_range(boot_hyp_pgd, hyp_idmap_start, PAGE_SIZE);
+		unmap_hyp_idmap_range(boot_hyp_pgd, hyp_idmap_start, PAGE_SIZE);
 		free_pages((unsigned long)boot_hyp_pgd, hyp_pgd_order);
 		boot_hyp_pgd = NULL;
 	}
 
 	if (hyp_pgd) {
-		unmap_hyp_range(hyp_pgd, hyp_idmap_start, PAGE_SIZE);
+		unmap_hyp_idmap_range(hyp_pgd, hyp_idmap_start, PAGE_SIZE);
 		unmap_hyp_range(hyp_pgd, kern_hyp_va(PAGE_OFFSET),
 				(uintptr_t)high_memory - PAGE_OFFSET);
 		unmap_hyp_range(hyp_pgd, kern_hyp_va(VMALLOC_START),
@@ -634,7 +650,7 @@ static int __create_hyp_mappings(pgd_t *pgdp, unsigned long ptrs_per_pgd,
 	addr = start & PAGE_MASK;
 	end = PAGE_ALIGN(end);
 	do {
-		pgd = pgdp + ((addr >> PGDIR_SHIFT) & (ptrs_per_pgd - 1));
+		pgd = pgdp + kvm_pgd_index(addr, ptrs_per_pgd);
 
 		if (pgd_none(*pgd)) {
 			pud = pud_alloc_one(NULL, addr);
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 11/26] KVM: arm64: Fix HYP idmap unmap when using 52bit PA
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

Unmapping the idmap range using 52bit PA is quite broken, as we
don't take into account the right number of PGD entries, and rely
on PTRS_PER_PGD. The result is that pgd_index() truncates the
address, and we end-up in the weed.

Let's introduce a new unmap_hyp_idmap_range() that knows about this,
together with a kvm_pgd_index() helper, which hides a bit of the
complexity of the issue.

Fixes: 98732d1b189b ("KVM: arm/arm64: fix HYP ID map extension to 52 bits")
Reported-by: James Morse <james.morse@arm.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 virt/kvm/arm/mmu.c | 26 +++++++++++++++++++++-----
 1 file changed, 21 insertions(+), 5 deletions(-)

diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index a9e0513868e9..9946f8a38b26 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -479,7 +479,13 @@ static void unmap_hyp_puds(pgd_t *pgd, phys_addr_t addr, phys_addr_t end)
 		clear_hyp_pgd_entry(pgd);
 }
 
-static void unmap_hyp_range(pgd_t *pgdp, phys_addr_t start, u64 size)
+static unsigned int kvm_pgd_index(unsigned long addr, unsigned int ptrs_per_pgd)
+{
+	return (addr >> PGDIR_SHIFT) & (ptrs_per_pgd - 1);
+}
+
+static void __unmap_hyp_range(pgd_t *pgdp, unsigned long ptrs_per_pgd,
+			      phys_addr_t start, u64 size)
 {
 	pgd_t *pgd;
 	phys_addr_t addr = start, end = start + size;
@@ -489,7 +495,7 @@ static void unmap_hyp_range(pgd_t *pgdp, phys_addr_t start, u64 size)
 	 * We don't unmap anything from HYP, except at the hyp tear down.
 	 * Hence, we don't have to invalidate the TLBs here.
 	 */
-	pgd = pgdp + pgd_index(addr);
+	pgd = pgdp + kvm_pgd_index(addr, ptrs_per_pgd);
 	do {
 		next = pgd_addr_end(addr, end);
 		if (!pgd_none(*pgd))
@@ -497,6 +503,16 @@ static void unmap_hyp_range(pgd_t *pgdp, phys_addr_t start, u64 size)
 	} while (pgd++, addr = next, addr != end);
 }
 
+static void unmap_hyp_range(pgd_t *pgdp, phys_addr_t start, u64 size)
+{
+	__unmap_hyp_range(pgdp, PTRS_PER_PGD, start, size);
+}
+
+static void unmap_hyp_idmap_range(pgd_t *pgdp, phys_addr_t start, u64 size)
+{
+	__unmap_hyp_range(pgdp, __kvm_idmap_ptrs_per_pgd(), start, size);
+}
+
 /**
  * free_hyp_pgds - free Hyp-mode page tables
  *
@@ -512,13 +528,13 @@ void free_hyp_pgds(void)
 	mutex_lock(&kvm_hyp_pgd_mutex);
 
 	if (boot_hyp_pgd) {
-		unmap_hyp_range(boot_hyp_pgd, hyp_idmap_start, PAGE_SIZE);
+		unmap_hyp_idmap_range(boot_hyp_pgd, hyp_idmap_start, PAGE_SIZE);
 		free_pages((unsigned long)boot_hyp_pgd, hyp_pgd_order);
 		boot_hyp_pgd = NULL;
 	}
 
 	if (hyp_pgd) {
-		unmap_hyp_range(hyp_pgd, hyp_idmap_start, PAGE_SIZE);
+		unmap_hyp_idmap_range(hyp_pgd, hyp_idmap_start, PAGE_SIZE);
 		unmap_hyp_range(hyp_pgd, kern_hyp_va(PAGE_OFFSET),
 				(uintptr_t)high_memory - PAGE_OFFSET);
 		unmap_hyp_range(hyp_pgd, kern_hyp_va(VMALLOC_START),
@@ -634,7 +650,7 @@ static int __create_hyp_mappings(pgd_t *pgdp, unsigned long ptrs_per_pgd,
 	addr = start & PAGE_MASK;
 	end = PAGE_ALIGN(end);
 	do {
-		pgd = pgdp + ((addr >> PGDIR_SHIFT) & (ptrs_per_pgd - 1));
+		pgd = pgdp + kvm_pgd_index(addr, ptrs_per_pgd);
 
 		if (pgd_none(*pgd)) {
 			pud = pud_alloc_one(NULL, addr);
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 12/26] KVM: arm/arm64: Move HYP IO VAs to the "idmap" range
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Catalin Marinas, Will Deacon, Kristina Martsenko

We so far mapped our HYP IO (which is essentially the GICv2 control
registers) using the same method as for memory. It recently appeared
that is a bit unsafe:

We compute the HYP VA using the kern_hyp_va helper, but that helper
is only designed to deal with kernel VAs coming from the linear map,
and not from the vmalloc region... This could in turn cause some bad
aliasing between the two, amplified by the upcoming VA randomisation.

A solution is to come up with our very own basic VA allocator for
MMIO. Since half of the HYP address space only contains a single
page (the idmap), we have plenty to borrow from. Let's use the idmap
as a base, and allocate downwards from it. GICv2 now lives on the
other side of the great VA barrier.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm/include/asm/kvm_mmu.h |  3 ++
 virt/kvm/arm/mmu.c             | 72 ++++++++++++++++++++++++++++++++++--------
 2 files changed, 62 insertions(+), 13 deletions(-)

diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
index 5b12a3d2755e..26eb6b1cec9b 100644
--- a/arch/arm/include/asm/kvm_mmu.h
+++ b/arch/arm/include/asm/kvm_mmu.h
@@ -49,6 +49,9 @@
 #include <asm/pgalloc.h>
 #include <asm/stage2_pgtable.h>
 
+/* Ensure compatibility with arm64 */
+#define VA_BITS			32
+
 int create_hyp_mappings(void *from, void *to, pgprot_t prot);
 int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
 			   void __iomem **kaddr,
diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index 9946f8a38b26..a9577ecc3e6a 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -43,6 +43,9 @@ static unsigned long hyp_idmap_start;
 static unsigned long hyp_idmap_end;
 static phys_addr_t hyp_idmap_vector;
 
+static DEFINE_MUTEX(io_map_lock);
+static unsigned long io_map_base;
+
 #define S2_PGD_SIZE	(PTRS_PER_S2_PGD * sizeof(pgd_t))
 #define hyp_pgd_order get_order(PTRS_PER_PGD * sizeof(pgd_t))
 
@@ -518,27 +521,37 @@ static void unmap_hyp_idmap_range(pgd_t *pgdp, phys_addr_t start, u64 size)
  *
  * Assumes hyp_pgd is a page table used strictly in Hyp-mode and
  * therefore contains either mappings in the kernel memory area (above
- * PAGE_OFFSET), or device mappings in the vmalloc range (from
- * VMALLOC_START to VMALLOC_END).
+ * PAGE_OFFSET), or device mappings in the idmap range.
  *
- * boot_hyp_pgd should only map two pages for the init code.
+ * boot_hyp_pgd should only map the idmap range, and is only used in
+ * the extended idmap case.
  */
 void free_hyp_pgds(void)
 {
+	pgd_t *id_pgd;
+
 	mutex_lock(&kvm_hyp_pgd_mutex);
 
+	id_pgd = boot_hyp_pgd ? boot_hyp_pgd : hyp_pgd;
+
+	if (id_pgd) {
+		mutex_lock(&io_map_lock);
+		/* In case we never called hyp_mmu_init() */
+		if (!io_map_base)
+			io_map_base = hyp_idmap_start;
+		unmap_hyp_idmap_range(id_pgd, io_map_base,
+				      hyp_idmap_start + PAGE_SIZE - io_map_base);
+		mutex_unlock(&io_map_lock);
+	}
+
 	if (boot_hyp_pgd) {
-		unmap_hyp_idmap_range(boot_hyp_pgd, hyp_idmap_start, PAGE_SIZE);
 		free_pages((unsigned long)boot_hyp_pgd, hyp_pgd_order);
 		boot_hyp_pgd = NULL;
 	}
 
 	if (hyp_pgd) {
-		unmap_hyp_idmap_range(hyp_pgd, hyp_idmap_start, PAGE_SIZE);
 		unmap_hyp_range(hyp_pgd, kern_hyp_va(PAGE_OFFSET),
 				(uintptr_t)high_memory - PAGE_OFFSET);
-		unmap_hyp_range(hyp_pgd, kern_hyp_va(VMALLOC_START),
-				VMALLOC_END - VMALLOC_START);
 
 		free_pages((unsigned long)hyp_pgd, hyp_pgd_order);
 		hyp_pgd = NULL;
@@ -735,7 +748,8 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
 			   void __iomem **kaddr,
 			   void __iomem **haddr)
 {
-	unsigned long start, end;
+	pgd_t *pgd = hyp_pgd;
+	unsigned long base;
 	int ret;
 
 	*kaddr = ioremap(phys_addr, size);
@@ -747,11 +761,43 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
 		return 0;
 	}
 
+	mutex_lock(&io_map_lock);
 
-	start = kern_hyp_va((unsigned long)*kaddr);
-	end = kern_hyp_va((unsigned long)*kaddr + size);
-	ret = __create_hyp_mappings(hyp_pgd, PTRS_PER_PGD, start, end,
-				     __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
+	/*
+	 * This assumes that we we have enough space below the idmap
+	 * page to allocate our VAs. If not, the check below will
+	 * kick. A potential alternative would be to detect that
+	 * overflow and switch to an allocation above the idmap.
+	 *
+	 * The allocated size is always a multiple of PAGE_SIZE.
+	 */
+	size = PAGE_ALIGN(size + offset_in_page(phys_addr));
+	base = io_map_base - size;
+
+	/*
+	 * Verify that BIT(VA_BITS - 1) hasn't been flipped by
+	 * allocating the new area, as it would indicate we've
+	 * overflowed the idmap/IO address range.
+	 */
+	if ((base ^ io_map_base) & BIT(VA_BITS - 1)) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	if (__kvm_cpu_uses_extended_idmap())
+		pgd = boot_hyp_pgd;
+
+	ret = __create_hyp_mappings(pgd, __kvm_idmap_ptrs_per_pgd(),
+				    base, base + size,
+				    __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
+	if (ret)
+		goto out;
+
+	*haddr = (void __iomem *)base + offset_in_page(phys_addr);
+	io_map_base = base;
+
+out:
+	mutex_unlock(&io_map_lock);
 
 	if (ret) {
 		iounmap(*kaddr);
@@ -759,7 +805,6 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
 		return ret;
 	}
 
-	*haddr = (void __iomem *)start;
 	return 0;
 }
 
@@ -1892,6 +1937,7 @@ int kvm_mmu_init(void)
 			goto out;
 	}
 
+	io_map_base = hyp_idmap_start;
 	return 0;
 out:
 	free_hyp_pgds();
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 12/26] KVM: arm/arm64: Move HYP IO VAs to the "idmap" range
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

We so far mapped our HYP IO (which is essentially the GICv2 control
registers) using the same method as for memory. It recently appeared
that is a bit unsafe:

We compute the HYP VA using the kern_hyp_va helper, but that helper
is only designed to deal with kernel VAs coming from the linear map,
and not from the vmalloc region... This could in turn cause some bad
aliasing between the two, amplified by the upcoming VA randomisation.

A solution is to come up with our very own basic VA allocator for
MMIO. Since half of the HYP address space only contains a single
page (the idmap), we have plenty to borrow from. Let's use the idmap
as a base, and allocate downwards from it. GICv2 now lives on the
other side of the great VA barrier.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm/include/asm/kvm_mmu.h |  3 ++
 virt/kvm/arm/mmu.c             | 72 ++++++++++++++++++++++++++++++++++--------
 2 files changed, 62 insertions(+), 13 deletions(-)

diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
index 5b12a3d2755e..26eb6b1cec9b 100644
--- a/arch/arm/include/asm/kvm_mmu.h
+++ b/arch/arm/include/asm/kvm_mmu.h
@@ -49,6 +49,9 @@
 #include <asm/pgalloc.h>
 #include <asm/stage2_pgtable.h>
 
+/* Ensure compatibility with arm64 */
+#define VA_BITS			32
+
 int create_hyp_mappings(void *from, void *to, pgprot_t prot);
 int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
 			   void __iomem **kaddr,
diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index 9946f8a38b26..a9577ecc3e6a 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -43,6 +43,9 @@ static unsigned long hyp_idmap_start;
 static unsigned long hyp_idmap_end;
 static phys_addr_t hyp_idmap_vector;
 
+static DEFINE_MUTEX(io_map_lock);
+static unsigned long io_map_base;
+
 #define S2_PGD_SIZE	(PTRS_PER_S2_PGD * sizeof(pgd_t))
 #define hyp_pgd_order get_order(PTRS_PER_PGD * sizeof(pgd_t))
 
@@ -518,27 +521,37 @@ static void unmap_hyp_idmap_range(pgd_t *pgdp, phys_addr_t start, u64 size)
  *
  * Assumes hyp_pgd is a page table used strictly in Hyp-mode and
  * therefore contains either mappings in the kernel memory area (above
- * PAGE_OFFSET), or device mappings in the vmalloc range (from
- * VMALLOC_START to VMALLOC_END).
+ * PAGE_OFFSET), or device mappings in the idmap range.
  *
- * boot_hyp_pgd should only map two pages for the init code.
+ * boot_hyp_pgd should only map the idmap range, and is only used in
+ * the extended idmap case.
  */
 void free_hyp_pgds(void)
 {
+	pgd_t *id_pgd;
+
 	mutex_lock(&kvm_hyp_pgd_mutex);
 
+	id_pgd = boot_hyp_pgd ? boot_hyp_pgd : hyp_pgd;
+
+	if (id_pgd) {
+		mutex_lock(&io_map_lock);
+		/* In case we never called hyp_mmu_init() */
+		if (!io_map_base)
+			io_map_base = hyp_idmap_start;
+		unmap_hyp_idmap_range(id_pgd, io_map_base,
+				      hyp_idmap_start + PAGE_SIZE - io_map_base);
+		mutex_unlock(&io_map_lock);
+	}
+
 	if (boot_hyp_pgd) {
-		unmap_hyp_idmap_range(boot_hyp_pgd, hyp_idmap_start, PAGE_SIZE);
 		free_pages((unsigned long)boot_hyp_pgd, hyp_pgd_order);
 		boot_hyp_pgd = NULL;
 	}
 
 	if (hyp_pgd) {
-		unmap_hyp_idmap_range(hyp_pgd, hyp_idmap_start, PAGE_SIZE);
 		unmap_hyp_range(hyp_pgd, kern_hyp_va(PAGE_OFFSET),
 				(uintptr_t)high_memory - PAGE_OFFSET);
-		unmap_hyp_range(hyp_pgd, kern_hyp_va(VMALLOC_START),
-				VMALLOC_END - VMALLOC_START);
 
 		free_pages((unsigned long)hyp_pgd, hyp_pgd_order);
 		hyp_pgd = NULL;
@@ -735,7 +748,8 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
 			   void __iomem **kaddr,
 			   void __iomem **haddr)
 {
-	unsigned long start, end;
+	pgd_t *pgd = hyp_pgd;
+	unsigned long base;
 	int ret;
 
 	*kaddr = ioremap(phys_addr, size);
@@ -747,11 +761,43 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
 		return 0;
 	}
 
+	mutex_lock(&io_map_lock);
 
-	start = kern_hyp_va((unsigned long)*kaddr);
-	end = kern_hyp_va((unsigned long)*kaddr + size);
-	ret = __create_hyp_mappings(hyp_pgd, PTRS_PER_PGD, start, end,
-				     __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
+	/*
+	 * This assumes that we we have enough space below the idmap
+	 * page to allocate our VAs. If not, the check below will
+	 * kick. A potential alternative would be to detect that
+	 * overflow and switch to an allocation above the idmap.
+	 *
+	 * The allocated size is always a multiple of PAGE_SIZE.
+	 */
+	size = PAGE_ALIGN(size + offset_in_page(phys_addr));
+	base = io_map_base - size;
+
+	/*
+	 * Verify that BIT(VA_BITS - 1) hasn't been flipped by
+	 * allocating the new area, as it would indicate we've
+	 * overflowed the idmap/IO address range.
+	 */
+	if ((base ^ io_map_base) & BIT(VA_BITS - 1)) {
+		ret = -ENOMEM;
+		goto out;
+	}
+
+	if (__kvm_cpu_uses_extended_idmap())
+		pgd = boot_hyp_pgd;
+
+	ret = __create_hyp_mappings(pgd, __kvm_idmap_ptrs_per_pgd(),
+				    base, base + size,
+				    __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
+	if (ret)
+		goto out;
+
+	*haddr = (void __iomem *)base + offset_in_page(phys_addr);
+	io_map_base = base;
+
+out:
+	mutex_unlock(&io_map_lock);
 
 	if (ret) {
 		iounmap(*kaddr);
@@ -759,7 +805,6 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
 		return ret;
 	}
 
-	*haddr = (void __iomem *)start;
 	return 0;
 }
 
@@ -1892,6 +1937,7 @@ int kvm_mmu_init(void)
 			goto out;
 	}
 
+	io_map_base = hyp_idmap_start;
 	return 0;
 out:
 	free_hyp_pgds();
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 13/26] arm64; insn: Add encoder for the EXTR instruction
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Catalin Marinas, Will Deacon, Kristina Martsenko

Add an encoder for the EXTR instruction, which also implements the ROR
variant (where Rn == Rm).

Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/include/asm/insn.h |  6 ++++++
 arch/arm64/kernel/insn.c      | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index 815b35bc53ed..f62c56b1793f 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -319,6 +319,7 @@ __AARCH64_INSN_FUNCS(and_imm,	0x7F800000, 0x12000000)
 __AARCH64_INSN_FUNCS(orr_imm,	0x7F800000, 0x32000000)
 __AARCH64_INSN_FUNCS(eor_imm,	0x7F800000, 0x52000000)
 __AARCH64_INSN_FUNCS(ands_imm,	0x7F800000, 0x72000000)
+__AARCH64_INSN_FUNCS(extr,	0x7FA00000, 0x13800000)
 __AARCH64_INSN_FUNCS(b,		0xFC000000, 0x14000000)
 __AARCH64_INSN_FUNCS(bl,	0xFC000000, 0x94000000)
 __AARCH64_INSN_FUNCS(cbz,	0x7F000000, 0x34000000)
@@ -433,6 +434,11 @@ u32 aarch64_insn_gen_logical_immediate(enum aarch64_insn_logic_type type,
 				       enum aarch64_insn_register Rn,
 				       enum aarch64_insn_register Rd,
 				       u64 imm);
+u32 aarch64_insn_gen_extr(enum aarch64_insn_variant variant,
+			  enum aarch64_insn_register Rm,
+			  enum aarch64_insn_register Rn,
+			  enum aarch64_insn_register Rd,
+			  u8 lsb);
 u32 aarch64_insn_gen_prefetch(enum aarch64_insn_register base,
 			      enum aarch64_insn_prfm_type type,
 			      enum aarch64_insn_prfm_target target,
diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index e87d6dcd7c82..2929adaad587 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -1621,3 +1621,35 @@ u32 aarch64_insn_gen_logical_immediate(enum aarch64_insn_logic_type type,
 	insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn, Rn);
 	return aarch64_encode_immediate(imm, variant, insn);
 }
+
+u32 aarch64_insn_gen_extr(enum aarch64_insn_variant variant,
+			  enum aarch64_insn_register Rm,
+			  enum aarch64_insn_register Rn,
+			  enum aarch64_insn_register Rd,
+			  u8 lsb)
+{
+	u32 insn;
+
+	insn = aarch64_insn_get_extr_value();
+
+	switch (variant) {
+	case AARCH64_INSN_VARIANT_32BIT:
+		if (lsb > 31)
+			return AARCH64_BREAK_FAULT;
+		break;
+	case AARCH64_INSN_VARIANT_64BIT:
+		if (lsb > 63)
+			return AARCH64_BREAK_FAULT;
+		insn |= AARCH64_INSN_SF_BIT;
+		insn = aarch64_insn_encode_immediate(AARCH64_INSN_IMM_N, insn, 1);
+		break;
+	default:
+		pr_err("%s: unknown variant encoding %d\n", __func__, variant);
+		return AARCH64_BREAK_FAULT;
+	}
+
+	insn = aarch64_insn_encode_immediate(AARCH64_INSN_IMM_S, insn, lsb);
+	insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RD, insn, Rd);
+	insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn, Rn);
+	return aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RM, insn, Rm);
+}
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 13/26] arm64; insn: Add encoder for the EXTR instruction
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

Add an encoder for the EXTR instruction, which also implements the ROR
variant (where Rn == Rm).

Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/include/asm/insn.h |  6 ++++++
 arch/arm64/kernel/insn.c      | 32 ++++++++++++++++++++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/arch/arm64/include/asm/insn.h b/arch/arm64/include/asm/insn.h
index 815b35bc53ed..f62c56b1793f 100644
--- a/arch/arm64/include/asm/insn.h
+++ b/arch/arm64/include/asm/insn.h
@@ -319,6 +319,7 @@ __AARCH64_INSN_FUNCS(and_imm,	0x7F800000, 0x12000000)
 __AARCH64_INSN_FUNCS(orr_imm,	0x7F800000, 0x32000000)
 __AARCH64_INSN_FUNCS(eor_imm,	0x7F800000, 0x52000000)
 __AARCH64_INSN_FUNCS(ands_imm,	0x7F800000, 0x72000000)
+__AARCH64_INSN_FUNCS(extr,	0x7FA00000, 0x13800000)
 __AARCH64_INSN_FUNCS(b,		0xFC000000, 0x14000000)
 __AARCH64_INSN_FUNCS(bl,	0xFC000000, 0x94000000)
 __AARCH64_INSN_FUNCS(cbz,	0x7F000000, 0x34000000)
@@ -433,6 +434,11 @@ u32 aarch64_insn_gen_logical_immediate(enum aarch64_insn_logic_type type,
 				       enum aarch64_insn_register Rn,
 				       enum aarch64_insn_register Rd,
 				       u64 imm);
+u32 aarch64_insn_gen_extr(enum aarch64_insn_variant variant,
+			  enum aarch64_insn_register Rm,
+			  enum aarch64_insn_register Rn,
+			  enum aarch64_insn_register Rd,
+			  u8 lsb);
 u32 aarch64_insn_gen_prefetch(enum aarch64_insn_register base,
 			      enum aarch64_insn_prfm_type type,
 			      enum aarch64_insn_prfm_target target,
diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index e87d6dcd7c82..2929adaad587 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -1621,3 +1621,35 @@ u32 aarch64_insn_gen_logical_immediate(enum aarch64_insn_logic_type type,
 	insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn, Rn);
 	return aarch64_encode_immediate(imm, variant, insn);
 }
+
+u32 aarch64_insn_gen_extr(enum aarch64_insn_variant variant,
+			  enum aarch64_insn_register Rm,
+			  enum aarch64_insn_register Rn,
+			  enum aarch64_insn_register Rd,
+			  u8 lsb)
+{
+	u32 insn;
+
+	insn = aarch64_insn_get_extr_value();
+
+	switch (variant) {
+	case AARCH64_INSN_VARIANT_32BIT:
+		if (lsb > 31)
+			return AARCH64_BREAK_FAULT;
+		break;
+	case AARCH64_INSN_VARIANT_64BIT:
+		if (lsb > 63)
+			return AARCH64_BREAK_FAULT;
+		insn |= AARCH64_INSN_SF_BIT;
+		insn = aarch64_insn_encode_immediate(AARCH64_INSN_IMM_N, insn, 1);
+		break;
+	default:
+		pr_err("%s: unknown variant encoding %d\n", __func__, variant);
+		return AARCH64_BREAK_FAULT;
+	}
+
+	insn = aarch64_insn_encode_immediate(AARCH64_INSN_IMM_S, insn, lsb);
+	insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RD, insn, Rd);
+	insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn, Rn);
+	return aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RM, insn, Rm);
+}
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 14/26] arm64: insn: Allow ADD/SUB (immediate) with LSL #12
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Mark Rutland, Peter Maydell, Andrew Jones, Christoffer Dall,
	Steve Capper, Catalin Marinas, Will Deacon, Kristina Martsenko,
	James Morse

The encoder for ADD/SUB (immediate) can only cope with 12bit
immediates, while there is an encoding for a 12bit immediate shifted
by 12 bits to the left.

Let's fix this small oversight by allowing the LSL_12 bit to be set.

Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/kernel/insn.c | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index 2929adaad587..816d03c4c913 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -35,6 +35,7 @@
 
 #define AARCH64_INSN_SF_BIT	BIT(31)
 #define AARCH64_INSN_N_BIT	BIT(22)
+#define AARCH64_INSN_LSL_12	BIT(22)
 
 static int aarch64_insn_encoding_class[] = {
 	AARCH64_INSN_CLS_UNKNOWN,
@@ -903,9 +904,18 @@ u32 aarch64_insn_gen_add_sub_imm(enum aarch64_insn_register dst,
 		return AARCH64_BREAK_FAULT;
 	}
 
+	/* We can't encode more than a 24bit value (12bit + 12bit shift) */
+	if (imm & ~(BIT(24) - 1))
+		goto out;
+
+	/* If we have something in the top 12 bits... */
 	if (imm & ~(SZ_4K - 1)) {
-		pr_err("%s: invalid immediate encoding %d\n", __func__, imm);
-		return AARCH64_BREAK_FAULT;
+		/* ... and in the low 12 bits -> error */
+		if (imm & (SZ_4K - 1))
+			goto out;
+
+		imm >>= 12;
+		insn |= AARCH64_INSN_LSL_12;
 	}
 
 	insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RD, insn, dst);
@@ -913,6 +923,10 @@ u32 aarch64_insn_gen_add_sub_imm(enum aarch64_insn_register dst,
 	insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn, src);
 
 	return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_12, insn, imm);
+
+out:
+	pr_err("%s: invalid immediate encoding %d\n", __func__, imm);
+	return AARCH64_BREAK_FAULT;
 }
 
 u32 aarch64_insn_gen_bitfield(enum aarch64_insn_register dst,
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 14/26] arm64: insn: Allow ADD/SUB (immediate) with LSL #12
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

The encoder for ADD/SUB (immediate) can only cope with 12bit
immediates, while there is an encoding for a 12bit immediate shifted
by 12 bits to the left.

Let's fix this small oversight by allowing the LSL_12 bit to be set.

Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>
Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/kernel/insn.c | 18 ++++++++++++++++--
 1 file changed, 16 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/kernel/insn.c b/arch/arm64/kernel/insn.c
index 2929adaad587..816d03c4c913 100644
--- a/arch/arm64/kernel/insn.c
+++ b/arch/arm64/kernel/insn.c
@@ -35,6 +35,7 @@
 
 #define AARCH64_INSN_SF_BIT	BIT(31)
 #define AARCH64_INSN_N_BIT	BIT(22)
+#define AARCH64_INSN_LSL_12	BIT(22)
 
 static int aarch64_insn_encoding_class[] = {
 	AARCH64_INSN_CLS_UNKNOWN,
@@ -903,9 +904,18 @@ u32 aarch64_insn_gen_add_sub_imm(enum aarch64_insn_register dst,
 		return AARCH64_BREAK_FAULT;
 	}
 
+	/* We can't encode more than a 24bit value (12bit + 12bit shift) */
+	if (imm & ~(BIT(24) - 1))
+		goto out;
+
+	/* If we have something in the top 12 bits... */
 	if (imm & ~(SZ_4K - 1)) {
-		pr_err("%s: invalid immediate encoding %d\n", __func__, imm);
-		return AARCH64_BREAK_FAULT;
+		/* ... and in the low 12 bits -> error */
+		if (imm & (SZ_4K - 1))
+			goto out;
+
+		imm >>= 12;
+		insn |= AARCH64_INSN_LSL_12;
 	}
 
 	insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RD, insn, dst);
@@ -913,6 +923,10 @@ u32 aarch64_insn_gen_add_sub_imm(enum aarch64_insn_register dst,
 	insn = aarch64_insn_encode_register(AARCH64_INSN_REGTYPE_RN, insn, src);
 
 	return aarch64_insn_encode_immediate(AARCH64_INSN_IMM_12, insn, imm);
+
+out:
+	pr_err("%s: invalid immediate encoding %d\n", __func__, imm);
+	return AARCH64_BREAK_FAULT;
 }
 
 u32 aarch64_insn_gen_bitfield(enum aarch64_insn_register dst,
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 15/26] arm64: KVM: Dynamically compute the HYP VA mask
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Catalin Marinas, Will Deacon, Kristina Martsenko

As we're moving towards a much more dynamic way to compute our
HYP VA, let's express the mask in a slightly different way.

Instead of comparing the idmap position to the "low" VA mask,
we directly compute the mask by taking into account the idmap's
(VA_BIT-1) bit.

No functionnal change.

Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/kvm/va_layout.c | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)

diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c
index 45e7802328d4..2bcbc1d33b64 100644
--- a/arch/arm64/kvm/va_layout.c
+++ b/arch/arm64/kvm/va_layout.c
@@ -21,24 +21,19 @@
 #include <asm/insn.h>
 #include <asm/kvm_mmu.h>
 
-#define HYP_PAGE_OFFSET_HIGH_MASK	((UL(1) << VA_BITS) - 1)
-#define HYP_PAGE_OFFSET_LOW_MASK	((UL(1) << (VA_BITS - 1)) - 1)
-
 static u64 va_mask;
 
 static void compute_layout(void)
 {
 	phys_addr_t idmap_addr = __pa_symbol(__hyp_idmap_text_start);
-	unsigned long mask = HYP_PAGE_OFFSET_HIGH_MASK;
+	u64 hyp_va_msb;
 
-	/*
-	 * Activate the lower HYP offset only if the idmap doesn't
-	 * clash with it,
-	 */
-	if (idmap_addr > HYP_PAGE_OFFSET_LOW_MASK)
-		mask = HYP_PAGE_OFFSET_LOW_MASK;
+	/* Where is my RAM region? */
+	hyp_va_msb  = idmap_addr & BIT(VA_BITS - 1);
+	hyp_va_msb ^= BIT(VA_BITS - 1);
 
-	va_mask = mask;
+	va_mask  = GENMASK_ULL(VA_BITS - 2, 0);
+	va_mask |= hyp_va_msb;
 }
 
 static u32 compute_instruction(int n, u32 rd, u32 rn)
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 15/26] arm64: KVM: Dynamically compute the HYP VA mask
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

As we're moving towards a much more dynamic way to compute our
HYP VA, let's express the mask in a slightly different way.

Instead of comparing the idmap position to the "low" VA mask,
we directly compute the mask by taking into account the idmap's
(VA_BIT-1) bit.

No functionnal change.

Reviewed-by: Christoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/kvm/va_layout.c | 17 ++++++-----------
 1 file changed, 6 insertions(+), 11 deletions(-)

diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c
index 45e7802328d4..2bcbc1d33b64 100644
--- a/arch/arm64/kvm/va_layout.c
+++ b/arch/arm64/kvm/va_layout.c
@@ -21,24 +21,19 @@
 #include <asm/insn.h>
 #include <asm/kvm_mmu.h>
 
-#define HYP_PAGE_OFFSET_HIGH_MASK	((UL(1) << VA_BITS) - 1)
-#define HYP_PAGE_OFFSET_LOW_MASK	((UL(1) << (VA_BITS - 1)) - 1)
-
 static u64 va_mask;
 
 static void compute_layout(void)
 {
 	phys_addr_t idmap_addr = __pa_symbol(__hyp_idmap_text_start);
-	unsigned long mask = HYP_PAGE_OFFSET_HIGH_MASK;
+	u64 hyp_va_msb;
 
-	/*
-	 * Activate the lower HYP offset only if the idmap doesn't
-	 * clash with it,
-	 */
-	if (idmap_addr > HYP_PAGE_OFFSET_LOW_MASK)
-		mask = HYP_PAGE_OFFSET_LOW_MASK;
+	/* Where is my RAM region? */
+	hyp_va_msb  = idmap_addr & BIT(VA_BITS - 1);
+	hyp_va_msb ^= BIT(VA_BITS - 1);
 
-	va_mask = mask;
+	va_mask  = GENMASK_ULL(VA_BITS - 2, 0);
+	va_mask |= hyp_va_msb;
 }
 
 static u32 compute_instruction(int n, u32 rd, u32 rn)
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 16/26] arm64: KVM: Introduce EL2 VA randomisation
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Catalin Marinas, Will Deacon, Kristina Martsenko

The main idea behind randomising the EL2 VA is that we usually have
a few spare bits between the most significant bit of the VA mask
and the most significant bit of the linear mapping.

Those bits could be a bunch of zeroes, and could be useful
to move things around a bit. Of course, the more memory you have,
the less randomisation you get...

Alternatively, these bits could be the result of KASLR, in which
case they are already random. But it would be nice to have a
*different* randomization, just to make the job of a potential
attacker a bit more difficult.

Inserting these random bits is a bit involved. We don't have a spare
register (short of rewriting all the kern_hyp_va call sites), and
the immediate we want to insert is too random to be used with the
ORR instruction. The best option I could come up with is the following
sequence:

	and x0, x0, #va_mask
	ror x0, x0, #first_random_bit
	add x0, x0, #(random & 0xfff)
	add x0, x0, #(random >> 12), lsl #12
	ror x0, x0, #(63 - first_random_bit)

making it a fairly long sequence, but one that a decent CPU should
be able to execute without breaking a sweat. It is of course NOPed
out on VHE. The last 4 instructions can also be turned into NOPs
if it appears that there is no free bits to use.

Reviewed-by: James Morse <james.morse@arm.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/include/asm/kvm_mmu.h | 12 +++++--
 arch/arm64/kvm/va_layout.c       | 76 +++++++++++++++++++++++++++++++++++++---
 virt/kvm/arm/mmu.c               |  2 +-
 3 files changed, 82 insertions(+), 8 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index c4934470403b..a5e0fc900de8 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -85,7 +85,11 @@
  */
 .macro kern_hyp_va	reg
 alternative_cb kvm_update_va_mask
-	and     \reg, \reg, #1
+	and     \reg, \reg, #1		/* mask with va_mask */
+	ror	\reg, \reg, #1		/* rotate to the first tag bit */
+	add	\reg, \reg, #0		/* insert the low 12 bits of the tag */
+	add	\reg, \reg, #0, lsl 12	/* insert the top 12 bits of the tag */
+	ror	\reg, \reg, #63		/* rotate back */
 alternative_cb_end
 .endm
 
@@ -102,7 +106,11 @@ void kvm_update_va_mask(struct alt_instr *alt,
 
 static inline unsigned long __kern_hyp_va(unsigned long v)
 {
-	asm volatile(ALTERNATIVE_CB("and %0, %0, #1\n",
+	asm volatile(ALTERNATIVE_CB("and %0, %0, #1\n"
+				    "ror %0, %0, #1\n"
+				    "add %0, %0, #0\n"
+				    "add %0, %0, #0, lsl 12\n"
+				    "ror %0, %0, #63\n",
 				    kvm_update_va_mask)
 		     : "+r" (v));
 	return v;
diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c
index 2bcbc1d33b64..8b23f6cdb0d6 100644
--- a/arch/arm64/kvm/va_layout.c
+++ b/arch/arm64/kvm/va_layout.c
@@ -16,24 +16,60 @@
  */
 
 #include <linux/kvm_host.h>
+#include <linux/random.h>
+#include <linux/memblock.h>
 #include <asm/alternative.h>
 #include <asm/debug-monitors.h>
 #include <asm/insn.h>
 #include <asm/kvm_mmu.h>
 
+/*
+ * The LSB of the random hyp VA tag or 0 if no randomization is used.
+ */
+static u8 tag_lsb;
+/*
+ * The random hyp VA tag value with the region bit if hyp randomization is used
+ */
+static u64 tag_val;
 static u64 va_mask;
 
 static void compute_layout(void)
 {
 	phys_addr_t idmap_addr = __pa_symbol(__hyp_idmap_text_start);
 	u64 hyp_va_msb;
+	int kva_msb;
 
 	/* Where is my RAM region? */
 	hyp_va_msb  = idmap_addr & BIT(VA_BITS - 1);
 	hyp_va_msb ^= BIT(VA_BITS - 1);
 
-	va_mask  = GENMASK_ULL(VA_BITS - 2, 0);
-	va_mask |= hyp_va_msb;
+	kva_msb = fls64((u64)phys_to_virt(memblock_start_of_DRAM()) ^
+			(u64)(high_memory - 1));
+
+	if (kva_msb == (VA_BITS - 1)) {
+		/*
+		 * No space in the address, let's compute the mask so
+		 * that it covers (VA_BITS - 1) bits, and the region
+		 * bit. The tag stays set to zero.
+		 */
+		va_mask  = BIT(VA_BITS - 1) - 1;
+		va_mask |= hyp_va_msb;
+	} else {
+		/*
+		 * We do have some free bits to insert a random tag.
+		 * Hyp VAs are now created from kernel linear map VAs
+		 * using the following formula (with V == VA_BITS):
+		 *
+		 *  63 ... V |     V-1    | V-2 .. tag_lsb | tag_lsb - 1 .. 0
+		 *  ---------------------------------------------------------
+		 * | 0000000 | hyp_va_msb |    random tag  |  kern linear VA |
+		 */
+		tag_lsb = kva_msb;
+		va_mask = GENMASK_ULL(tag_lsb - 1, 0);
+		tag_val = get_random_long() & GENMASK_ULL(VA_BITS - 2, tag_lsb);
+		tag_val |= hyp_va_msb;
+		tag_val >>= tag_lsb;
+	}
 }
 
 static u32 compute_instruction(int n, u32 rd, u32 rn)
@@ -46,6 +82,33 @@ static u32 compute_instruction(int n, u32 rd, u32 rn)
 							  AARCH64_INSN_VARIANT_64BIT,
 							  rn, rd, va_mask);
 		break;
+
+	case 1:
+		/* ROR is a variant of EXTR with Rm = Rn */
+		insn = aarch64_insn_gen_extr(AARCH64_INSN_VARIANT_64BIT,
+					     rn, rn, rd,
+					     tag_lsb);
+		break;
+
+	case 2:
+		insn = aarch64_insn_gen_add_sub_imm(rd, rn,
+						    tag_val & GENMASK(11, 0),
+						    AARCH64_INSN_VARIANT_64BIT,
+						    AARCH64_INSN_ADSB_ADD);
+		break;
+
+	case 3:
+		insn = aarch64_insn_gen_add_sub_imm(rd, rn,
+						    tag_val & GENMASK(23, 12),
+						    AARCH64_INSN_VARIANT_64BIT,
+						    AARCH64_INSN_ADSB_ADD);
+		break;
+
+	case 4:
+		/* ROR is a variant of EXTR with Rm = Rn */
+		insn = aarch64_insn_gen_extr(AARCH64_INSN_VARIANT_64BIT,
+					     rn, rn, rd, 64 - tag_lsb);
+		break;
 	}
 
 	return insn;
@@ -56,8 +119,7 @@ void __init kvm_update_va_mask(struct alt_instr *alt,
 {
 	int i;
 
-	/* We only expect a single instruction in the alternative sequence */
-	BUG_ON(nr_inst != 1);
+	BUG_ON(nr_inst != 5);
 
 	if (!has_vhe() && !va_mask)
 		compute_layout();
@@ -68,8 +130,12 @@ void __init kvm_update_va_mask(struct alt_instr *alt,
 		/*
 		 * VHE doesn't need any address translation, let's NOP
 		 * everything.
+		 *
+		 * Alternatively, if we don't have any spare bits in
+		 * the address, NOP everything after masking that
+		 * kernel VA.
 		 */
-		if (has_vhe()) {
+		if (has_vhe() || (!tag_lsb && i > 0)) {
 			updptr[i] = aarch64_insn_gen_nop();
 			continue;
 		}
diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index a9577ecc3e6a..b4d1948c8dd6 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -1893,7 +1893,7 @@ int kvm_mmu_init(void)
 		  kern_hyp_va((unsigned long)high_memory - 1));
 
 	if (hyp_idmap_start >= kern_hyp_va(PAGE_OFFSET) &&
-	    hyp_idmap_start <  kern_hyp_va(~0UL) &&
+	    hyp_idmap_start <  kern_hyp_va((unsigned long)high_memory - 1) &&
 	    hyp_idmap_start != (unsigned long)__hyp_idmap_text_start) {
 		/*
 		 * The idmap page is intersecting with the VA space,
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 16/26] arm64: KVM: Introduce EL2 VA randomisation
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

The main idea behind randomising the EL2 VA is that we usually have
a few spare bits between the most significant bit of the VA mask
and the most significant bit of the linear mapping.

Those bits could be a bunch of zeroes, and could be useful
to move things around a bit. Of course, the more memory you have,
the less randomisation you get...

Alternatively, these bits could be the result of KASLR, in which
case they are already random. But it would be nice to have a
*different* randomization, just to make the job of a potential
attacker a bit more difficult.

Inserting these random bits is a bit involved. We don't have a spare
register (short of rewriting all the kern_hyp_va call sites), and
the immediate we want to insert is too random to be used with the
ORR instruction. The best option I could come up with is the following
sequence:

	and x0, x0, #va_mask
	ror x0, x0, #first_random_bit
	add x0, x0, #(random & 0xfff)
	add x0, x0, #(random >> 12), lsl #12
	ror x0, x0, #(63 - first_random_bit)

making it a fairly long sequence, but one that a decent CPU should
be able to execute without breaking a sweat. It is of course NOPed
out on VHE. The last 4 instructions can also be turned into NOPs
if it appears that there is no free bits to use.

Reviewed-by: James Morse <james.morse@arm.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/include/asm/kvm_mmu.h | 12 +++++--
 arch/arm64/kvm/va_layout.c       | 76 +++++++++++++++++++++++++++++++++++++---
 virt/kvm/arm/mmu.c               |  2 +-
 3 files changed, 82 insertions(+), 8 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index c4934470403b..a5e0fc900de8 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -85,7 +85,11 @@
  */
 .macro kern_hyp_va	reg
 alternative_cb kvm_update_va_mask
-	and     \reg, \reg, #1
+	and     \reg, \reg, #1		/* mask with va_mask */
+	ror	\reg, \reg, #1		/* rotate to the first tag bit */
+	add	\reg, \reg, #0		/* insert the low 12 bits of the tag */
+	add	\reg, \reg, #0, lsl 12	/* insert the top 12 bits of the tag */
+	ror	\reg, \reg, #63		/* rotate back */
 alternative_cb_end
 .endm
 
@@ -102,7 +106,11 @@ void kvm_update_va_mask(struct alt_instr *alt,
 
 static inline unsigned long __kern_hyp_va(unsigned long v)
 {
-	asm volatile(ALTERNATIVE_CB("and %0, %0, #1\n",
+	asm volatile(ALTERNATIVE_CB("and %0, %0, #1\n"
+				    "ror %0, %0, #1\n"
+				    "add %0, %0, #0\n"
+				    "add %0, %0, #0, lsl 12\n"
+				    "ror %0, %0, #63\n",
 				    kvm_update_va_mask)
 		     : "+r" (v));
 	return v;
diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c
index 2bcbc1d33b64..8b23f6cdb0d6 100644
--- a/arch/arm64/kvm/va_layout.c
+++ b/arch/arm64/kvm/va_layout.c
@@ -16,24 +16,60 @@
  */
 
 #include <linux/kvm_host.h>
+#include <linux/random.h>
+#include <linux/memblock.h>
 #include <asm/alternative.h>
 #include <asm/debug-monitors.h>
 #include <asm/insn.h>
 #include <asm/kvm_mmu.h>
 
+/*
+ * The LSB of the random hyp VA tag or 0 if no randomization is used.
+ */
+static u8 tag_lsb;
+/*
+ * The random hyp VA tag value with the region bit if hyp randomization is used
+ */
+static u64 tag_val;
 static u64 va_mask;
 
 static void compute_layout(void)
 {
 	phys_addr_t idmap_addr = __pa_symbol(__hyp_idmap_text_start);
 	u64 hyp_va_msb;
+	int kva_msb;
 
 	/* Where is my RAM region? */
 	hyp_va_msb  = idmap_addr & BIT(VA_BITS - 1);
 	hyp_va_msb ^= BIT(VA_BITS - 1);
 
-	va_mask  = GENMASK_ULL(VA_BITS - 2, 0);
-	va_mask |= hyp_va_msb;
+	kva_msb = fls64((u64)phys_to_virt(memblock_start_of_DRAM()) ^
+			(u64)(high_memory - 1));
+
+	if (kva_msb == (VA_BITS - 1)) {
+		/*
+		 * No space in the address, let's compute the mask so
+		 * that it covers (VA_BITS - 1) bits, and the region
+		 * bit. The tag stays set to zero.
+		 */
+		va_mask  = BIT(VA_BITS - 1) - 1;
+		va_mask |= hyp_va_msb;
+	} else {
+		/*
+		 * We do have some free bits to insert a random tag.
+		 * Hyp VAs are now created from kernel linear map VAs
+		 * using the following formula (with V == VA_BITS):
+		 *
+		 *  63 ... V |     V-1    | V-2 .. tag_lsb | tag_lsb - 1 .. 0
+		 *  ---------------------------------------------------------
+		 * | 0000000 | hyp_va_msb |    random tag  |  kern linear VA |
+		 */
+		tag_lsb = kva_msb;
+		va_mask = GENMASK_ULL(tag_lsb - 1, 0);
+		tag_val = get_random_long() & GENMASK_ULL(VA_BITS - 2, tag_lsb);
+		tag_val |= hyp_va_msb;
+		tag_val >>= tag_lsb;
+	}
 }
 
 static u32 compute_instruction(int n, u32 rd, u32 rn)
@@ -46,6 +82,33 @@ static u32 compute_instruction(int n, u32 rd, u32 rn)
 							  AARCH64_INSN_VARIANT_64BIT,
 							  rn, rd, va_mask);
 		break;
+
+	case 1:
+		/* ROR is a variant of EXTR with Rm = Rn */
+		insn = aarch64_insn_gen_extr(AARCH64_INSN_VARIANT_64BIT,
+					     rn, rn, rd,
+					     tag_lsb);
+		break;
+
+	case 2:
+		insn = aarch64_insn_gen_add_sub_imm(rd, rn,
+						    tag_val & GENMASK(11, 0),
+						    AARCH64_INSN_VARIANT_64BIT,
+						    AARCH64_INSN_ADSB_ADD);
+		break;
+
+	case 3:
+		insn = aarch64_insn_gen_add_sub_imm(rd, rn,
+						    tag_val & GENMASK(23, 12),
+						    AARCH64_INSN_VARIANT_64BIT,
+						    AARCH64_INSN_ADSB_ADD);
+		break;
+
+	case 4:
+		/* ROR is a variant of EXTR with Rm = Rn */
+		insn = aarch64_insn_gen_extr(AARCH64_INSN_VARIANT_64BIT,
+					     rn, rn, rd, 64 - tag_lsb);
+		break;
 	}
 
 	return insn;
@@ -56,8 +119,7 @@ void __init kvm_update_va_mask(struct alt_instr *alt,
 {
 	int i;
 
-	/* We only expect a single instruction in the alternative sequence */
-	BUG_ON(nr_inst != 1);
+	BUG_ON(nr_inst != 5);
 
 	if (!has_vhe() && !va_mask)
 		compute_layout();
@@ -68,8 +130,12 @@ void __init kvm_update_va_mask(struct alt_instr *alt,
 		/*
 		 * VHE doesn't need any address translation, let's NOP
 		 * everything.
+		 *
+		 * Alternatively, if we don't have any spare bits in
+		 * the address, NOP everything after masking that
+		 * kernel VA.
 		 */
-		if (has_vhe()) {
+		if (has_vhe() || (!tag_lsb && i > 0)) {
 			updptr[i] = aarch64_insn_gen_nop();
 			continue;
 		}
diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index a9577ecc3e6a..b4d1948c8dd6 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -1893,7 +1893,7 @@ int kvm_mmu_init(void)
 		  kern_hyp_va((unsigned long)high_memory - 1));
 
 	if (hyp_idmap_start >= kern_hyp_va(PAGE_OFFSET) &&
-	    hyp_idmap_start <  kern_hyp_va(~0UL) &&
+	    hyp_idmap_start <  kern_hyp_va((unsigned long)high_memory - 1) &&
 	    hyp_idmap_start != (unsigned long)__hyp_idmap_text_start) {
 		/*
 		 * The idmap page is intersecting with the VA space,
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 17/26] arm64: Update the KVM memory map documentation
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Catalin Marinas, Will Deacon, Kristina Martsenko

Update the documentation to reflect the new tricks we play on the
EL2 mappings...

Acked-by: Christoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 Documentation/arm64/memory.txt | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/Documentation/arm64/memory.txt b/Documentation/arm64/memory.txt
index 671bc0639262..c58cc5dbe667 100644
--- a/Documentation/arm64/memory.txt
+++ b/Documentation/arm64/memory.txt
@@ -86,9 +86,11 @@ Translation table lookup with 64KB pages:
  +-------------------------------------------------> [63] TTBR0/1
 
 
-When using KVM without the Virtualization Host Extensions, the hypervisor
-maps kernel pages in EL2 at a fixed offset from the kernel VA. See the
-kern_hyp_va macro for more details.
+When using KVM without the Virtualization Host Extensions, the
+hypervisor maps kernel pages in EL2 at a fixed (and potentially
+random) offset from the linear mapping. See the kern_hyp_va macro and
+kvm_update_va_mask function for more details. MMIO devices such as
+GICv2 gets mapped next to the HYP idmap page.
 
 When using KVM with the Virtualization Host Extensions, no additional
 mappings are created, since the host kernel runs directly in EL2.
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 17/26] arm64: Update the KVM memory map documentation
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

Update the documentation to reflect the new tricks we play on the
EL2 mappings...

Acked-by: Christoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 Documentation/arm64/memory.txt | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/Documentation/arm64/memory.txt b/Documentation/arm64/memory.txt
index 671bc0639262..c58cc5dbe667 100644
--- a/Documentation/arm64/memory.txt
+++ b/Documentation/arm64/memory.txt
@@ -86,9 +86,11 @@ Translation table lookup with 64KB pages:
  +-------------------------------------------------> [63] TTBR0/1
 
 
-When using KVM without the Virtualization Host Extensions, the hypervisor
-maps kernel pages in EL2 at a fixed offset from the kernel VA. See the
-kern_hyp_va macro for more details.
+When using KVM without the Virtualization Host Extensions, the
+hypervisor maps kernel pages in EL2 at a fixed (and potentially
+random) offset from the linear mapping. See the kern_hyp_va macro and
+kvm_update_va_mask function for more details. MMIO devices such as
+GICv2 gets mapped next to the HYP idmap page.
 
 When using KVM with the Virtualization Host Extensions, no additional
 mappings are created, since the host kernel runs directly in EL2.
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 18/26] arm64: KVM: Move vector offsetting from hyp-init.S to kvm_get_hyp_vector
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Catalin Marinas, Will Deacon, Kristina Martsenko

We currently provide the hyp-init code with a kernel VA, and expect
it to turn it into a HYP va by itself. As we're about to provide
the hypervisor with mappings that are not necessarily in the memory
range, let's move the kern_hyp_va macro to kvm_get_hyp_vector.

No functionnal change.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/include/asm/kvm_mmu.h | 3 ++-
 arch/arm64/kvm/hyp-init.S        | 1 -
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index a5e0fc900de8..bfb81e73da06 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -374,6 +374,7 @@ static inline void *kvm_get_hyp_vector(void)
 			vect = lm_alias(vect);
 	}
 
+	vect = kern_hyp_va(vect);
 	return vect;
 }
 
@@ -387,7 +388,7 @@ static inline int kvm_map_vectors(void)
 #else
 static inline void *kvm_get_hyp_vector(void)
 {
-	return kvm_ksym_ref(__kvm_hyp_vector);
+	return kern_hyp_va(kvm_ksym_ref(__kvm_hyp_vector));
 }
 
 static inline int kvm_map_vectors(void)
diff --git a/arch/arm64/kvm/hyp-init.S b/arch/arm64/kvm/hyp-init.S
index 5aa9ccf6db99..6fd91b31a131 100644
--- a/arch/arm64/kvm/hyp-init.S
+++ b/arch/arm64/kvm/hyp-init.S
@@ -117,7 +117,6 @@ CPU_BE(	orr	x4, x4, #SCTLR_ELx_EE)
 	/* Set the stack and new vectors */
 	kern_hyp_va	x1
 	mov	sp, x1
-	kern_hyp_va	x2
 	msr	vbar_el2, x2
 
 	/* copy tpidr_el1 into tpidr_el2 for use by HYP */
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 18/26] arm64: KVM: Move vector offsetting from hyp-init.S to kvm_get_hyp_vector
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

We currently provide the hyp-init code with a kernel VA, and expect
it to turn it into a HYP va by itself. As we're about to provide
the hypervisor with mappings that are not necessarily in the memory
range, let's move the kern_hyp_va macro to kvm_get_hyp_vector.

No functionnal change.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/include/asm/kvm_mmu.h | 3 ++-
 arch/arm64/kvm/hyp-init.S        | 1 -
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index a5e0fc900de8..bfb81e73da06 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -374,6 +374,7 @@ static inline void *kvm_get_hyp_vector(void)
 			vect = lm_alias(vect);
 	}
 
+	vect = kern_hyp_va(vect);
 	return vect;
 }
 
@@ -387,7 +388,7 @@ static inline int kvm_map_vectors(void)
 #else
 static inline void *kvm_get_hyp_vector(void)
 {
-	return kvm_ksym_ref(__kvm_hyp_vector);
+	return kern_hyp_va(kvm_ksym_ref(__kvm_hyp_vector));
 }
 
 static inline int kvm_map_vectors(void)
diff --git a/arch/arm64/kvm/hyp-init.S b/arch/arm64/kvm/hyp-init.S
index 5aa9ccf6db99..6fd91b31a131 100644
--- a/arch/arm64/kvm/hyp-init.S
+++ b/arch/arm64/kvm/hyp-init.S
@@ -117,7 +117,6 @@ CPU_BE(	orr	x4, x4, #SCTLR_ELx_EE)
 	/* Set the stack and new vectors */
 	kern_hyp_va	x1
 	mov	sp, x1
-	kern_hyp_va	x2
 	msr	vbar_el2, x2
 
 	/* copy tpidr_el1 into tpidr_el2 for use by HYP */
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 19/26] arm64: KVM: Move stashing of x0/x1 into the vector code itself
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Catalin Marinas, Will Deacon, Kristina Martsenko

All our useful entry points into the hypervisor are starting by
saving x0 and x1 on the stack. Let's move those into the vectors
by introducing macros that annotate whether a vector is valid or
not, thus indicating whether we want to stash registers or not.

The only drawback is that we now also stash registers for el2_error,
but this should never happen, and we pop them back right at the
start of the handling sequence.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/kvm/hyp/hyp-entry.S | 56 ++++++++++++++++++++++++------------------
 1 file changed, 32 insertions(+), 24 deletions(-)

diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S
index f36464bd57c5..0f62b5f76aa5 100644
--- a/arch/arm64/kvm/hyp/hyp-entry.S
+++ b/arch/arm64/kvm/hyp/hyp-entry.S
@@ -55,7 +55,6 @@ ENTRY(__vhe_hyp_call)
 ENDPROC(__vhe_hyp_call)
 
 el1_sync:				// Guest trapped into EL2
-	stp	x0, x1, [sp, #-16]!
 
 alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
 	mrs	x1, esr_el2
@@ -137,18 +136,18 @@ alternative_else_nop_endif
 	b	__guest_exit
 
 el1_irq:
-	stp     x0, x1, [sp, #-16]!
 	ldr	x1, [sp, #16 + 8]
 	mov	x0, #ARM_EXCEPTION_IRQ
 	b	__guest_exit
 
 el1_error:
-	stp     x0, x1, [sp, #-16]!
 	ldr	x1, [sp, #16 + 8]
 	mov	x0, #ARM_EXCEPTION_EL1_SERROR
 	b	__guest_exit
 
 el2_error:
+	ldp	x0, x1, [sp], #16
+
 	/*
 	 * Only two possibilities:
 	 * 1) Either we come from the exit path, having just unmasked
@@ -206,32 +205,41 @@ ENDPROC(\label)
 	invalid_vector	el2h_sync_invalid
 	invalid_vector	el2h_irq_invalid
 	invalid_vector	el2h_fiq_invalid
-	invalid_vector	el1_sync_invalid
-	invalid_vector	el1_irq_invalid
 	invalid_vector	el1_fiq_invalid
 
 	.ltorg
 
 	.align 11
 
+.macro valid_vect target
+	.align 7
+	stp	x0, x1, [sp, #-16]!
+	b	\target
+.endm
+
+.macro invalid_vect target
+	.align 7
+	b	\target
+.endm
+
 ENTRY(__kvm_hyp_vector)
-	ventry	el2t_sync_invalid		// Synchronous EL2t
-	ventry	el2t_irq_invalid		// IRQ EL2t
-	ventry	el2t_fiq_invalid		// FIQ EL2t
-	ventry	el2t_error_invalid		// Error EL2t
-
-	ventry	el2h_sync_invalid		// Synchronous EL2h
-	ventry	el2h_irq_invalid		// IRQ EL2h
-	ventry	el2h_fiq_invalid		// FIQ EL2h
-	ventry	el2_error			// Error EL2h
-
-	ventry	el1_sync			// Synchronous 64-bit EL1
-	ventry	el1_irq				// IRQ 64-bit EL1
-	ventry	el1_fiq_invalid			// FIQ 64-bit EL1
-	ventry	el1_error			// Error 64-bit EL1
-
-	ventry	el1_sync			// Synchronous 32-bit EL1
-	ventry	el1_irq				// IRQ 32-bit EL1
-	ventry	el1_fiq_invalid			// FIQ 32-bit EL1
-	ventry	el1_error			// Error 32-bit EL1
+	invalid_vect	el2t_sync_invalid	// Synchronous EL2t
+	invalid_vect	el2t_irq_invalid	// IRQ EL2t
+	invalid_vect	el2t_fiq_invalid	// FIQ EL2t
+	invalid_vect	el2t_error_invalid	// Error EL2t
+
+	invalid_vect	el2h_sync_invalid	// Synchronous EL2h
+	invalid_vect	el2h_irq_invalid	// IRQ EL2h
+	invalid_vect	el2h_fiq_invalid	// FIQ EL2h
+	valid_vect	el2_error		// Error EL2h
+
+	valid_vect	el1_sync		// Synchronous 64-bit EL1
+	valid_vect	el1_irq			// IRQ 64-bit EL1
+	invalid_vect	el1_fiq_invalid		// FIQ 64-bit EL1
+	valid_vect	el1_error		// Error 64-bit EL1
+
+	valid_vect	el1_sync		// Synchronous 32-bit EL1
+	valid_vect	el1_irq			// IRQ 32-bit EL1
+	invalid_vect	el1_fiq_invalid		// FIQ 32-bit EL1
+	valid_vect	el1_error		// Error 32-bit EL1
 ENDPROC(__kvm_hyp_vector)
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 19/26] arm64: KVM: Move stashing of x0/x1 into the vector code itself
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

All our useful entry points into the hypervisor are starting by
saving x0 and x1 on the stack. Let's move those into the vectors
by introducing macros that annotate whether a vector is valid or
not, thus indicating whether we want to stash registers or not.

The only drawback is that we now also stash registers for el2_error,
but this should never happen, and we pop them back right at the
start of the handling sequence.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/kvm/hyp/hyp-entry.S | 56 ++++++++++++++++++++++++------------------
 1 file changed, 32 insertions(+), 24 deletions(-)

diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S
index f36464bd57c5..0f62b5f76aa5 100644
--- a/arch/arm64/kvm/hyp/hyp-entry.S
+++ b/arch/arm64/kvm/hyp/hyp-entry.S
@@ -55,7 +55,6 @@ ENTRY(__vhe_hyp_call)
 ENDPROC(__vhe_hyp_call)
 
 el1_sync:				// Guest trapped into EL2
-	stp	x0, x1, [sp, #-16]!
 
 alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
 	mrs	x1, esr_el2
@@ -137,18 +136,18 @@ alternative_else_nop_endif
 	b	__guest_exit
 
 el1_irq:
-	stp     x0, x1, [sp, #-16]!
 	ldr	x1, [sp, #16 + 8]
 	mov	x0, #ARM_EXCEPTION_IRQ
 	b	__guest_exit
 
 el1_error:
-	stp     x0, x1, [sp, #-16]!
 	ldr	x1, [sp, #16 + 8]
 	mov	x0, #ARM_EXCEPTION_EL1_SERROR
 	b	__guest_exit
 
 el2_error:
+	ldp	x0, x1, [sp], #16
+
 	/*
 	 * Only two possibilities:
 	 * 1) Either we come from the exit path, having just unmasked
@@ -206,32 +205,41 @@ ENDPROC(\label)
 	invalid_vector	el2h_sync_invalid
 	invalid_vector	el2h_irq_invalid
 	invalid_vector	el2h_fiq_invalid
-	invalid_vector	el1_sync_invalid
-	invalid_vector	el1_irq_invalid
 	invalid_vector	el1_fiq_invalid
 
 	.ltorg
 
 	.align 11
 
+.macro valid_vect target
+	.align 7
+	stp	x0, x1, [sp, #-16]!
+	b	\target
+.endm
+
+.macro invalid_vect target
+	.align 7
+	b	\target
+.endm
+
 ENTRY(__kvm_hyp_vector)
-	ventry	el2t_sync_invalid		// Synchronous EL2t
-	ventry	el2t_irq_invalid		// IRQ EL2t
-	ventry	el2t_fiq_invalid		// FIQ EL2t
-	ventry	el2t_error_invalid		// Error EL2t
-
-	ventry	el2h_sync_invalid		// Synchronous EL2h
-	ventry	el2h_irq_invalid		// IRQ EL2h
-	ventry	el2h_fiq_invalid		// FIQ EL2h
-	ventry	el2_error			// Error EL2h
-
-	ventry	el1_sync			// Synchronous 64-bit EL1
-	ventry	el1_irq				// IRQ 64-bit EL1
-	ventry	el1_fiq_invalid			// FIQ 64-bit EL1
-	ventry	el1_error			// Error 64-bit EL1
-
-	ventry	el1_sync			// Synchronous 32-bit EL1
-	ventry	el1_irq				// IRQ 32-bit EL1
-	ventry	el1_fiq_invalid			// FIQ 32-bit EL1
-	ventry	el1_error			// Error 32-bit EL1
+	invalid_vect	el2t_sync_invalid	// Synchronous EL2t
+	invalid_vect	el2t_irq_invalid	// IRQ EL2t
+	invalid_vect	el2t_fiq_invalid	// FIQ EL2t
+	invalid_vect	el2t_error_invalid	// Error EL2t
+
+	invalid_vect	el2h_sync_invalid	// Synchronous EL2h
+	invalid_vect	el2h_irq_invalid	// IRQ EL2h
+	invalid_vect	el2h_fiq_invalid	// FIQ EL2h
+	valid_vect	el2_error		// Error EL2h
+
+	valid_vect	el1_sync		// Synchronous 64-bit EL1
+	valid_vect	el1_irq			// IRQ 64-bit EL1
+	invalid_vect	el1_fiq_invalid		// FIQ 64-bit EL1
+	valid_vect	el1_error		// Error 64-bit EL1
+
+	valid_vect	el1_sync		// Synchronous 32-bit EL1
+	valid_vect	el1_irq			// IRQ 32-bit EL1
+	invalid_vect	el1_fiq_invalid		// FIQ 32-bit EL1
+	valid_vect	el1_error		// Error 32-bit EL1
 ENDPROC(__kvm_hyp_vector)
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 20/26] arm64: KVM: Move BP hardening vectors into .hyp.text section
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Catalin Marinas, Will Deacon, Kristina Martsenko

There is no reason why the BP hardening vectors shouldn't be part
of the HYP text at compile time, rather than being mapped at runtime.

Also introduce a new config symbol that controls the compilation
of bpi.S.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/include/asm/kvm_mmu.h | 4 +---
 arch/arm64/kernel/Makefile       | 4 +---
 arch/arm64/kernel/bpi.S          | 6 ++++++
 arch/arm64/kvm/Kconfig           | 3 +++
 4 files changed, 11 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index bfb81e73da06..c2beb2d25170 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -380,9 +380,7 @@ static inline void *kvm_get_hyp_vector(void)
 
 static inline int kvm_map_vectors(void)
 {
-	return create_hyp_mappings(kvm_ksym_ref(__bp_harden_hyp_vecs_start),
-				   kvm_ksym_ref(__bp_harden_hyp_vecs_end),
-				   PAGE_HYP_EXEC);
+	return 0;
 }
 
 #else
diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile
index b87541360f43..93bce17109a6 100644
--- a/arch/arm64/kernel/Makefile
+++ b/arch/arm64/kernel/Makefile
@@ -54,9 +54,7 @@ arm64-reloc-test-y := reloc_test_core.o reloc_test_syms.o
 arm64-obj-$(CONFIG_CRASH_DUMP)		+= crash_dump.o
 arm64-obj-$(CONFIG_ARM_SDE_INTERFACE)	+= sdei.o
 
-ifeq ($(CONFIG_KVM),y)
-arm64-obj-$(CONFIG_HARDEN_BRANCH_PREDICTOR)	+= bpi.o
-endif
+arm64-obj-$(CONFIG_KVM_INDIRECT_VECTORS)+= bpi.o
 
 obj-y					+= $(arm64-obj-y) vdso/ probes/
 obj-m					+= $(arm64-obj-m)
diff --git a/arch/arm64/kernel/bpi.S b/arch/arm64/kernel/bpi.S
index e5de33513b5d..447188e2a664 100644
--- a/arch/arm64/kernel/bpi.S
+++ b/arch/arm64/kernel/bpi.S
@@ -48,6 +48,10 @@
 	ventry \target + 0x780
 .endm
 
+
+	.text
+	.pushsection	.hyp.text, "ax"
+
 	.align	11
 ENTRY(__bp_harden_hyp_vecs_start)
 	.rept 4
@@ -55,6 +59,8 @@ ENTRY(__bp_harden_hyp_vecs_start)
 	.endr
 ENTRY(__bp_harden_hyp_vecs_end)
 
+	.popsection
+
 ENTRY(__qcom_hyp_sanitize_link_stack_start)
 	stp     x29, x30, [sp, #-16]!
 	.rept	16
diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
index 2257dfcc44cc..bd8cc03d7522 100644
--- a/arch/arm64/kvm/Kconfig
+++ b/arch/arm64/kvm/Kconfig
@@ -57,6 +57,9 @@ config KVM_ARM_PMU
 	  Adds support for a virtual Performance Monitoring Unit (PMU) in
 	  virtual machines.
 
+config KVM_INDIRECT_VECTORS
+       def_bool KVM && HARDEN_BRANCH_PREDICTOR
+
 source drivers/vhost/Kconfig
 
 endif # VIRTUALIZATION
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 20/26] arm64: KVM: Move BP hardening vectors into .hyp.text section
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

There is no reason why the BP hardening vectors shouldn't be part
of the HYP text at compile time, rather than being mapped at runtime.

Also introduce a new config symbol that controls the compilation
of bpi.S.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/include/asm/kvm_mmu.h | 4 +---
 arch/arm64/kernel/Makefile       | 4 +---
 arch/arm64/kernel/bpi.S          | 6 ++++++
 arch/arm64/kvm/Kconfig           | 3 +++
 4 files changed, 11 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index bfb81e73da06..c2beb2d25170 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -380,9 +380,7 @@ static inline void *kvm_get_hyp_vector(void)
 
 static inline int kvm_map_vectors(void)
 {
-	return create_hyp_mappings(kvm_ksym_ref(__bp_harden_hyp_vecs_start),
-				   kvm_ksym_ref(__bp_harden_hyp_vecs_end),
-				   PAGE_HYP_EXEC);
+	return 0;
 }
 
 #else
diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile
index b87541360f43..93bce17109a6 100644
--- a/arch/arm64/kernel/Makefile
+++ b/arch/arm64/kernel/Makefile
@@ -54,9 +54,7 @@ arm64-reloc-test-y := reloc_test_core.o reloc_test_syms.o
 arm64-obj-$(CONFIG_CRASH_DUMP)		+= crash_dump.o
 arm64-obj-$(CONFIG_ARM_SDE_INTERFACE)	+= sdei.o
 
-ifeq ($(CONFIG_KVM),y)
-arm64-obj-$(CONFIG_HARDEN_BRANCH_PREDICTOR)	+= bpi.o
-endif
+arm64-obj-$(CONFIG_KVM_INDIRECT_VECTORS)+= bpi.o
 
 obj-y					+= $(arm64-obj-y) vdso/ probes/
 obj-m					+= $(arm64-obj-m)
diff --git a/arch/arm64/kernel/bpi.S b/arch/arm64/kernel/bpi.S
index e5de33513b5d..447188e2a664 100644
--- a/arch/arm64/kernel/bpi.S
+++ b/arch/arm64/kernel/bpi.S
@@ -48,6 +48,10 @@
 	ventry \target + 0x780
 .endm
 
+
+	.text
+	.pushsection	.hyp.text, "ax"
+
 	.align	11
 ENTRY(__bp_harden_hyp_vecs_start)
 	.rept 4
@@ -55,6 +59,8 @@ ENTRY(__bp_harden_hyp_vecs_start)
 	.endr
 ENTRY(__bp_harden_hyp_vecs_end)
 
+	.popsection
+
 ENTRY(__qcom_hyp_sanitize_link_stack_start)
 	stp     x29, x30, [sp, #-16]!
 	.rept	16
diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
index 2257dfcc44cc..bd8cc03d7522 100644
--- a/arch/arm64/kvm/Kconfig
+++ b/arch/arm64/kvm/Kconfig
@@ -57,6 +57,9 @@ config KVM_ARM_PMU
 	  Adds support for a virtual Performance Monitoring Unit (PMU) in
 	  virtual machines.
 
+config KVM_INDIRECT_VECTORS
+       def_bool KVM && HARDEN_BRANCH_PREDICTOR
+
 source drivers/vhost/Kconfig
 
 endif # VIRTUALIZATION
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 21/26] arm64: KVM: Reserve 4 additional instructions in the BPI template
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Catalin Marinas, Will Deacon, Kristina Martsenko

So far, we only reserve a single instruction in the BPI template in
order to branch to the vectors. As we're going to stuff a few more
instructions there, let's reserve a total of 5 instructions, which
we're going to patch later on as required.

We also introduce a small refactor of the vectors themselves, so that
we stop carrying the target branch around.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/kernel/bpi.S | 39 +++++++++++++++------------------------
 1 file changed, 15 insertions(+), 24 deletions(-)

diff --git a/arch/arm64/kernel/bpi.S b/arch/arm64/kernel/bpi.S
index 447188e2a664..ce1cfe3b24e6 100644
--- a/arch/arm64/kernel/bpi.S
+++ b/arch/arm64/kernel/bpi.S
@@ -19,33 +19,24 @@
 #include <linux/linkage.h>
 #include <linux/arm-smccc.h>
 
-.macro ventry target
-	.rept 31
+.macro hyp_ventry
+	.align 7
+1:	.rept 27
 	nop
 	.endr
-	b	\target
+	b	__kvm_hyp_vector + (1b - 0b)
+	nop
+	nop
+	nop
+	nop
 .endm
 
-.macro vectors target
-	ventry \target + 0x000
-	ventry \target + 0x080
-	ventry \target + 0x100
-	ventry \target + 0x180
-
-	ventry \target + 0x200
-	ventry \target + 0x280
-	ventry \target + 0x300
-	ventry \target + 0x380
-
-	ventry \target + 0x400
-	ventry \target + 0x480
-	ventry \target + 0x500
-	ventry \target + 0x580
-
-	ventry \target + 0x600
-	ventry \target + 0x680
-	ventry \target + 0x700
-	ventry \target + 0x780
+.macro generate_vectors
+0:
+	.rept 16
+	hyp_ventry
+	.endr
+	.org 0b + SZ_2K		// Safety measure
 .endm
 
 
@@ -55,7 +46,7 @@
 	.align	11
 ENTRY(__bp_harden_hyp_vecs_start)
 	.rept 4
-	vectors __kvm_hyp_vector
+	generate_vectors
 	.endr
 ENTRY(__bp_harden_hyp_vecs_end)
 
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 21/26] arm64: KVM: Reserve 4 additional instructions in the BPI template
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

So far, we only reserve a single instruction in the BPI template in
order to branch to the vectors. As we're going to stuff a few more
instructions there, let's reserve a total of 5 instructions, which
we're going to patch later on as required.

We also introduce a small refactor of the vectors themselves, so that
we stop carrying the target branch around.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/kernel/bpi.S | 39 +++++++++++++++------------------------
 1 file changed, 15 insertions(+), 24 deletions(-)

diff --git a/arch/arm64/kernel/bpi.S b/arch/arm64/kernel/bpi.S
index 447188e2a664..ce1cfe3b24e6 100644
--- a/arch/arm64/kernel/bpi.S
+++ b/arch/arm64/kernel/bpi.S
@@ -19,33 +19,24 @@
 #include <linux/linkage.h>
 #include <linux/arm-smccc.h>
 
-.macro ventry target
-	.rept 31
+.macro hyp_ventry
+	.align 7
+1:	.rept 27
 	nop
 	.endr
-	b	\target
+	b	__kvm_hyp_vector + (1b - 0b)
+	nop
+	nop
+	nop
+	nop
 .endm
 
-.macro vectors target
-	ventry \target + 0x000
-	ventry \target + 0x080
-	ventry \target + 0x100
-	ventry \target + 0x180
-
-	ventry \target + 0x200
-	ventry \target + 0x280
-	ventry \target + 0x300
-	ventry \target + 0x380
-
-	ventry \target + 0x400
-	ventry \target + 0x480
-	ventry \target + 0x500
-	ventry \target + 0x580
-
-	ventry \target + 0x600
-	ventry \target + 0x680
-	ventry \target + 0x700
-	ventry \target + 0x780
+.macro generate_vectors
+0:
+	.rept 16
+	hyp_ventry
+	.endr
+	.org 0b + SZ_2K		// Safety measure
 .endm
 
 
@@ -55,7 +46,7 @@
 	.align	11
 ENTRY(__bp_harden_hyp_vecs_start)
 	.rept 4
-	vectors __kvm_hyp_vector
+	generate_vectors
 	.endr
 ENTRY(__bp_harden_hyp_vecs_end)
 
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 22/26] arm64: KVM: Allow far branches from vector slots to the main vectors
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Catalin Marinas, Will Deacon, Kristina Martsenko

So far, the branch from the vector slots to the main vectors can at
most be 4GB from the main vectors (the reach of ADRP), and this
distance is known at compile time. If we were to remap the slots
to an unrelated VA, things would break badly.

A way to achieve VA independence would be to load the absolute
address of the vectors (__kvm_hyp_vector), either using a constant
pool or a series of movs, followed by an indirect branch.

This patches implements the latter solution, using another instance
of a patching callback. Note that since we have to save a register
pair on the stack, we branch to the *second* instruction in the
vectors in order to compensate for it. This also results in having
to adjust this balance in the invalid vector entry point.

Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/include/asm/cpucaps.h |  2 +-
 arch/arm64/kernel/bpi.S          | 21 ++++++++++++
 arch/arm64/kvm/hyp/hyp-entry.S   |  2 ++
 arch/arm64/kvm/va_layout.c       | 72 ++++++++++++++++++++++++++++++++++++++++
 4 files changed, 96 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h
index 76a43a17449a..d4cc54ed0656 100644
--- a/arch/arm64/include/asm/cpucaps.h
+++ b/arch/arm64/include/asm/cpucaps.h
@@ -32,7 +32,7 @@
 #define ARM64_HAS_VIRT_HOST_EXTN		11
 #define ARM64_WORKAROUND_CAVIUM_27456		12
 #define ARM64_HAS_32BIT_EL0			13
-/* #define ARM64_UNALLOCATED_ENTRY			14 */
+#define ARM64_HARDEN_EL2_VECTORS		14
 #define ARM64_MISMATCHED_CACHE_LINE_SIZE	15
 #define ARM64_HAS_NO_FPSIMD			16
 #define ARM64_WORKAROUND_REPEAT_TLBI		17
diff --git a/arch/arm64/kernel/bpi.S b/arch/arm64/kernel/bpi.S
index ce1cfe3b24e6..dc51ef2ce98a 100644
--- a/arch/arm64/kernel/bpi.S
+++ b/arch/arm64/kernel/bpi.S
@@ -19,16 +19,37 @@
 #include <linux/linkage.h>
 #include <linux/arm-smccc.h>
 
+#include <asm/alternative.h>
+
 .macro hyp_ventry
 	.align 7
 1:	.rept 27
 	nop
 	.endr
+/*
+ * The default sequence is to directly branch to the KVM vectors,
+ * using the computed offset. This applies for VHE as well as
+ * !ARM64_HARDEN_EL2_VECTORS.
+ *
+ * For ARM64_HARDEN_EL2_VECTORS configurations, this gets replaced
+ * with:
+ *
+ * stp	x0, x1, [sp, #-16]!
+ * movz	x0, #(addr & 0xffff)
+ * movk	x0, #((addr >> 16) & 0xffff), lsl #16
+ * movk	x0, #((addr >> 32) & 0xffff), lsl #32
+ * br	x0
+ *
+ * Where addr = kern_hyp_va(__kvm_hyp_vector) + vector-offset + 4.
+ * See kvm_patch_vector_branch for details.
+ */
+alternative_cb	kvm_patch_vector_branch
 	b	__kvm_hyp_vector + (1b - 0b)
 	nop
 	nop
 	nop
 	nop
+alternative_cb_end
 .endm
 
 .macro generate_vectors
diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S
index 0f62b5f76aa5..fc6a1006cc08 100644
--- a/arch/arm64/kvm/hyp/hyp-entry.S
+++ b/arch/arm64/kvm/hyp/hyp-entry.S
@@ -220,6 +220,8 @@ ENDPROC(\label)
 .macro invalid_vect target
 	.align 7
 	b	\target
+	ldp	x0, x1, [sp], #16
+	b	\target
 .endm
 
 ENTRY(__kvm_hyp_vector)
diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c
index 8b23f6cdb0d6..9d35c17016ed 100644
--- a/arch/arm64/kvm/va_layout.c
+++ b/arch/arm64/kvm/va_layout.c
@@ -150,3 +150,75 @@ void __init kvm_update_va_mask(struct alt_instr *alt,
 		updptr[i] = cpu_to_le32(insn);
 	}
 }
+
+void kvm_patch_vector_branch(struct alt_instr *alt,
+			     __le32 *origptr, __le32 *updptr, int nr_inst)
+{
+	u64 addr;
+	u32 insn;
+
+	BUG_ON(nr_inst != 5);
+
+	if (has_vhe() || !cpus_have_const_cap(ARM64_HARDEN_EL2_VECTORS)) {
+		WARN_ON_ONCE(cpus_have_const_cap(ARM64_HARDEN_EL2_VECTORS));
+		return;
+	}
+
+	if (!va_mask)
+		compute_layout();
+
+	/*
+	 * Compute HYP VA by using the same computation as kern_hyp_va()
+	 */
+	addr = (uintptr_t)kvm_ksym_ref(__kvm_hyp_vector);
+	addr &= va_mask;
+	addr |= tag_val << tag_lsb;
+
+	/* Use PC[10:7] to branch to the same vector in KVM */
+	addr |= ((u64)origptr & GENMASK_ULL(10, 7));
+
+	/*
+	 * Branch to the second instruction in the vectors in order to
+	 * avoid the initial store on the stack (which we already
+	 * perform in the hardening vectors).
+	 */
+	addr += AARCH64_INSN_SIZE;
+
+	/* stp x0, x1, [sp, #-16]! */
+	insn = aarch64_insn_gen_load_store_pair(AARCH64_INSN_REG_0,
+						AARCH64_INSN_REG_1,
+						AARCH64_INSN_REG_SP,
+						-16,
+						AARCH64_INSN_VARIANT_64BIT,
+						AARCH64_INSN_LDST_STORE_PAIR_PRE_INDEX);
+	*updptr++ = cpu_to_le32(insn);
+
+	/* movz x0, #(addr & 0xffff) */
+	insn = aarch64_insn_gen_movewide(AARCH64_INSN_REG_0,
+					 (u16)addr,
+					 0,
+					 AARCH64_INSN_VARIANT_64BIT,
+					 AARCH64_INSN_MOVEWIDE_ZERO);
+	*updptr++ = cpu_to_le32(insn);
+
+	/* movk x0, #((addr >> 16) & 0xffff), lsl #16 */
+	insn = aarch64_insn_gen_movewide(AARCH64_INSN_REG_0,
+					 (u16)(addr >> 16),
+					 16,
+					 AARCH64_INSN_VARIANT_64BIT,
+					 AARCH64_INSN_MOVEWIDE_KEEP);
+	*updptr++ = cpu_to_le32(insn);
+
+	/* movk x0, #((addr >> 32) & 0xffff), lsl #32 */
+	insn = aarch64_insn_gen_movewide(AARCH64_INSN_REG_0,
+					 (u16)(addr >> 32),
+					 32,
+					 AARCH64_INSN_VARIANT_64BIT,
+					 AARCH64_INSN_MOVEWIDE_KEEP);
+	*updptr++ = cpu_to_le32(insn);
+
+	/* br x0 */
+	insn = aarch64_insn_gen_branch_reg(AARCH64_INSN_REG_0,
+					   AARCH64_INSN_BRANCH_NOLINK);
+	*updptr++ = cpu_to_le32(insn);
+}
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 22/26] arm64: KVM: Allow far branches from vector slots to the main vectors
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

So far, the branch from the vector slots to the main vectors can at
most be 4GB from the main vectors (the reach of ADRP), and this
distance is known at compile time. If we were to remap the slots
to an unrelated VA, things would break badly.

A way to achieve VA independence would be to load the absolute
address of the vectors (__kvm_hyp_vector), either using a constant
pool or a series of movs, followed by an indirect branch.

This patches implements the latter solution, using another instance
of a patching callback. Note that since we have to save a register
pair on the stack, we branch to the *second* instruction in the
vectors in order to compensate for it. This also results in having
to adjust this balance in the invalid vector entry point.

Acked-by: Catalin Marinas <catalin.marinas@arm.com>
Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/include/asm/cpucaps.h |  2 +-
 arch/arm64/kernel/bpi.S          | 21 ++++++++++++
 arch/arm64/kvm/hyp/hyp-entry.S   |  2 ++
 arch/arm64/kvm/va_layout.c       | 72 ++++++++++++++++++++++++++++++++++++++++
 4 files changed, 96 insertions(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/cpucaps.h b/arch/arm64/include/asm/cpucaps.h
index 76a43a17449a..d4cc54ed0656 100644
--- a/arch/arm64/include/asm/cpucaps.h
+++ b/arch/arm64/include/asm/cpucaps.h
@@ -32,7 +32,7 @@
 #define ARM64_HAS_VIRT_HOST_EXTN		11
 #define ARM64_WORKAROUND_CAVIUM_27456		12
 #define ARM64_HAS_32BIT_EL0			13
-/* #define ARM64_UNALLOCATED_ENTRY			14 */
+#define ARM64_HARDEN_EL2_VECTORS		14
 #define ARM64_MISMATCHED_CACHE_LINE_SIZE	15
 #define ARM64_HAS_NO_FPSIMD			16
 #define ARM64_WORKAROUND_REPEAT_TLBI		17
diff --git a/arch/arm64/kernel/bpi.S b/arch/arm64/kernel/bpi.S
index ce1cfe3b24e6..dc51ef2ce98a 100644
--- a/arch/arm64/kernel/bpi.S
+++ b/arch/arm64/kernel/bpi.S
@@ -19,16 +19,37 @@
 #include <linux/linkage.h>
 #include <linux/arm-smccc.h>
 
+#include <asm/alternative.h>
+
 .macro hyp_ventry
 	.align 7
 1:	.rept 27
 	nop
 	.endr
+/*
+ * The default sequence is to directly branch to the KVM vectors,
+ * using the computed offset. This applies for VHE as well as
+ * !ARM64_HARDEN_EL2_VECTORS.
+ *
+ * For ARM64_HARDEN_EL2_VECTORS configurations, this gets replaced
+ * with:
+ *
+ * stp	x0, x1, [sp, #-16]!
+ * movz	x0, #(addr & 0xffff)
+ * movk	x0, #((addr >> 16) & 0xffff), lsl #16
+ * movk	x0, #((addr >> 32) & 0xffff), lsl #32
+ * br	x0
+ *
+ * Where addr = kern_hyp_va(__kvm_hyp_vector) + vector-offset + 4.
+ * See kvm_patch_vector_branch for details.
+ */
+alternative_cb	kvm_patch_vector_branch
 	b	__kvm_hyp_vector + (1b - 0b)
 	nop
 	nop
 	nop
 	nop
+alternative_cb_end
 .endm
 
 .macro generate_vectors
diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S
index 0f62b5f76aa5..fc6a1006cc08 100644
--- a/arch/arm64/kvm/hyp/hyp-entry.S
+++ b/arch/arm64/kvm/hyp/hyp-entry.S
@@ -220,6 +220,8 @@ ENDPROC(\label)
 .macro invalid_vect target
 	.align 7
 	b	\target
+	ldp	x0, x1, [sp], #16
+	b	\target
 .endm
 
 ENTRY(__kvm_hyp_vector)
diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c
index 8b23f6cdb0d6..9d35c17016ed 100644
--- a/arch/arm64/kvm/va_layout.c
+++ b/arch/arm64/kvm/va_layout.c
@@ -150,3 +150,75 @@ void __init kvm_update_va_mask(struct alt_instr *alt,
 		updptr[i] = cpu_to_le32(insn);
 	}
 }
+
+void kvm_patch_vector_branch(struct alt_instr *alt,
+			     __le32 *origptr, __le32 *updptr, int nr_inst)
+{
+	u64 addr;
+	u32 insn;
+
+	BUG_ON(nr_inst != 5);
+
+	if (has_vhe() || !cpus_have_const_cap(ARM64_HARDEN_EL2_VECTORS)) {
+		WARN_ON_ONCE(cpus_have_const_cap(ARM64_HARDEN_EL2_VECTORS));
+		return;
+	}
+
+	if (!va_mask)
+		compute_layout();
+
+	/*
+	 * Compute HYP VA by using the same computation as kern_hyp_va()
+	 */
+	addr = (uintptr_t)kvm_ksym_ref(__kvm_hyp_vector);
+	addr &= va_mask;
+	addr |= tag_val << tag_lsb;
+
+	/* Use PC[10:7] to branch to the same vector in KVM */
+	addr |= ((u64)origptr & GENMASK_ULL(10, 7));
+
+	/*
+	 * Branch to the second instruction in the vectors in order to
+	 * avoid the initial store on the stack (which we already
+	 * perform in the hardening vectors).
+	 */
+	addr += AARCH64_INSN_SIZE;
+
+	/* stp x0, x1, [sp, #-16]! */
+	insn = aarch64_insn_gen_load_store_pair(AARCH64_INSN_REG_0,
+						AARCH64_INSN_REG_1,
+						AARCH64_INSN_REG_SP,
+						-16,
+						AARCH64_INSN_VARIANT_64BIT,
+						AARCH64_INSN_LDST_STORE_PAIR_PRE_INDEX);
+	*updptr++ = cpu_to_le32(insn);
+
+	/* movz x0, #(addr & 0xffff) */
+	insn = aarch64_insn_gen_movewide(AARCH64_INSN_REG_0,
+					 (u16)addr,
+					 0,
+					 AARCH64_INSN_VARIANT_64BIT,
+					 AARCH64_INSN_MOVEWIDE_ZERO);
+	*updptr++ = cpu_to_le32(insn);
+
+	/* movk x0, #((addr >> 16) & 0xffff), lsl #16 */
+	insn = aarch64_insn_gen_movewide(AARCH64_INSN_REG_0,
+					 (u16)(addr >> 16),
+					 16,
+					 AARCH64_INSN_VARIANT_64BIT,
+					 AARCH64_INSN_MOVEWIDE_KEEP);
+	*updptr++ = cpu_to_le32(insn);
+
+	/* movk x0, #((addr >> 32) & 0xffff), lsl #32 */
+	insn = aarch64_insn_gen_movewide(AARCH64_INSN_REG_0,
+					 (u16)(addr >> 32),
+					 32,
+					 AARCH64_INSN_VARIANT_64BIT,
+					 AARCH64_INSN_MOVEWIDE_KEEP);
+	*updptr++ = cpu_to_le32(insn);
+
+	/* br x0 */
+	insn = aarch64_insn_gen_branch_reg(AARCH64_INSN_REG_0,
+					   AARCH64_INSN_BRANCH_NOLINK);
+	*updptr++ = cpu_to_le32(insn);
+}
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 23/26] arm/arm64: KVM: Introduce EL2-specific executable mappings
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Mark Rutland, Peter Maydell, Andrew Jones, Christoffer Dall,
	Steve Capper, Catalin Marinas, Will Deacon, Kristina Martsenko,
	James Morse

Until now, all EL2 executable mappings were derived from their
EL1 VA. Since we want to decouple the vectors mapping from
the rest of the hypervisor, we need to be able to map some
text somewhere else.

The "idmap" region (for lack of a better name) is ideally suited
for this, as we have a huge range that hardly has anything in it.

Let's extend the IO allocator to also deal with executable mappings,
thus providing the required feature.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm/include/asm/kvm_mmu.h   |  2 +
 arch/arm64/include/asm/kvm_mmu.h |  2 +
 virt/kvm/arm/mmu.c               | 80 +++++++++++++++++++++++++++++-----------
 3 files changed, 63 insertions(+), 21 deletions(-)

diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
index 26eb6b1cec9b..1b7f592eae57 100644
--- a/arch/arm/include/asm/kvm_mmu.h
+++ b/arch/arm/include/asm/kvm_mmu.h
@@ -56,6 +56,8 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot);
 int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
 			   void __iomem **kaddr,
 			   void __iomem **haddr);
+int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
+			     void **haddr);
 void free_hyp_pgds(void);
 
 void stage2_unmap_vm(struct kvm *kvm);
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index c2beb2d25170..97af11065bbd 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -151,6 +151,8 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot);
 int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
 			   void __iomem **kaddr,
 			   void __iomem **haddr);
+int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
+			     void **haddr);
 void free_hyp_pgds(void);
 
 void stage2_unmap_vm(struct kvm *kvm);
diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index b4d1948c8dd6..554ad5493e7d 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -737,30 +737,13 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot)
 	return 0;
 }
 
-/**
- * create_hyp_io_mappings - Map IO into both kernel and HYP
- * @phys_addr:	The physical start address which gets mapped
- * @size:	Size of the region being mapped
- * @kaddr:	Kernel VA for this mapping
- * @haddr:	HYP VA for this mapping
- */
-int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
-			   void __iomem **kaddr,
-			   void __iomem **haddr)
+static int __create_hyp_private_mapping(phys_addr_t phys_addr, size_t size,
+					unsigned long *haddr, pgprot_t prot)
 {
 	pgd_t *pgd = hyp_pgd;
 	unsigned long base;
 	int ret;
 
-	*kaddr = ioremap(phys_addr, size);
-	if (!*kaddr)
-		return -ENOMEM;
-
-	if (is_kernel_in_hyp_mode()) {
-		*haddr = *kaddr;
-		return 0;
-	}
-
 	mutex_lock(&io_map_lock);
 
 	/*
@@ -789,22 +772,77 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
 
 	ret = __create_hyp_mappings(pgd, __kvm_idmap_ptrs_per_pgd(),
 				    base, base + size,
-				    __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
+				    __phys_to_pfn(phys_addr), prot);
 	if (ret)
 		goto out;
 
-	*haddr = (void __iomem *)base + offset_in_page(phys_addr);
+	*haddr = base + offset_in_page(phys_addr);
 	io_map_base = base;
 
 out:
 	mutex_unlock(&io_map_lock);
 
+	return ret;
+}
+
+/**
+ * create_hyp_io_mappings - Map IO into both kernel and HYP
+ * @phys_addr:	The physical start address which gets mapped
+ * @size:	Size of the region being mapped
+ * @kaddr:	Kernel VA for this mapping
+ * @haddr:	HYP VA for this mapping
+ */
+int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
+			   void __iomem **kaddr,
+			   void __iomem **haddr)
+{
+	unsigned long addr;
+	int ret;
+
+	*kaddr = ioremap(phys_addr, size);
+	if (!*kaddr)
+		return -ENOMEM;
+
+	if (is_kernel_in_hyp_mode()) {
+		*haddr = *kaddr;
+		return 0;
+	}
+
+	ret = __create_hyp_private_mapping(phys_addr, size,
+					   &addr, PAGE_HYP_DEVICE);
 	if (ret) {
 		iounmap(*kaddr);
 		*kaddr = NULL;
+		*haddr = NULL;
+		return ret;
+	}
+
+	*haddr = (void __iomem *)addr;
+	return 0;
+}
+
+/**
+ * create_hyp_exec_mappings - Map an executable range into into HYP
+ * @phys_addr:	The physical start address which gets mapped
+ * @size:	Size of the region being mapped
+ * @haddr:	HYP VA for this mapping
+ */
+int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
+			     void **haddr)
+{
+	unsigned long addr;
+	int ret;
+
+	BUG_ON(is_kernel_in_hyp_mode());
+
+	ret = __create_hyp_private_mapping(phys_addr, size,
+					   &addr, PAGE_HYP_EXEC);
+	if (ret) {
+		*haddr = NULL;
 		return ret;
 	}
 
+	*haddr = (void *)addr;
 	return 0;
 }
 
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 23/26] arm/arm64: KVM: Introduce EL2-specific executable mappings
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

Until now, all EL2 executable mappings were derived from their
EL1 VA. Since we want to decouple the vectors mapping from
the rest of the hypervisor, we need to be able to map some
text somewhere else.

The "idmap" region (for lack of a better name) is ideally suited
for this, as we have a huge range that hardly has anything in it.

Let's extend the IO allocator to also deal with executable mappings,
thus providing the required feature.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm/include/asm/kvm_mmu.h   |  2 +
 arch/arm64/include/asm/kvm_mmu.h |  2 +
 virt/kvm/arm/mmu.c               | 80 +++++++++++++++++++++++++++++-----------
 3 files changed, 63 insertions(+), 21 deletions(-)

diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
index 26eb6b1cec9b..1b7f592eae57 100644
--- a/arch/arm/include/asm/kvm_mmu.h
+++ b/arch/arm/include/asm/kvm_mmu.h
@@ -56,6 +56,8 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot);
 int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
 			   void __iomem **kaddr,
 			   void __iomem **haddr);
+int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
+			     void **haddr);
 void free_hyp_pgds(void);
 
 void stage2_unmap_vm(struct kvm *kvm);
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index c2beb2d25170..97af11065bbd 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -151,6 +151,8 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot);
 int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
 			   void __iomem **kaddr,
 			   void __iomem **haddr);
+int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
+			     void **haddr);
 void free_hyp_pgds(void);
 
 void stage2_unmap_vm(struct kvm *kvm);
diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index b4d1948c8dd6..554ad5493e7d 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -737,30 +737,13 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot)
 	return 0;
 }
 
-/**
- * create_hyp_io_mappings - Map IO into both kernel and HYP
- * @phys_addr:	The physical start address which gets mapped
- * @size:	Size of the region being mapped
- * @kaddr:	Kernel VA for this mapping
- * @haddr:	HYP VA for this mapping
- */
-int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
-			   void __iomem **kaddr,
-			   void __iomem **haddr)
+static int __create_hyp_private_mapping(phys_addr_t phys_addr, size_t size,
+					unsigned long *haddr, pgprot_t prot)
 {
 	pgd_t *pgd = hyp_pgd;
 	unsigned long base;
 	int ret;
 
-	*kaddr = ioremap(phys_addr, size);
-	if (!*kaddr)
-		return -ENOMEM;
-
-	if (is_kernel_in_hyp_mode()) {
-		*haddr = *kaddr;
-		return 0;
-	}
-
 	mutex_lock(&io_map_lock);
 
 	/*
@@ -789,22 +772,77 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
 
 	ret = __create_hyp_mappings(pgd, __kvm_idmap_ptrs_per_pgd(),
 				    base, base + size,
-				    __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
+				    __phys_to_pfn(phys_addr), prot);
 	if (ret)
 		goto out;
 
-	*haddr = (void __iomem *)base + offset_in_page(phys_addr);
+	*haddr = base + offset_in_page(phys_addr);
 	io_map_base = base;
 
 out:
 	mutex_unlock(&io_map_lock);
 
+	return ret;
+}
+
+/**
+ * create_hyp_io_mappings - Map IO into both kernel and HYP
+ * @phys_addr:	The physical start address which gets mapped
+ * @size:	Size of the region being mapped
+ * @kaddr:	Kernel VA for this mapping
+ * @haddr:	HYP VA for this mapping
+ */
+int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
+			   void __iomem **kaddr,
+			   void __iomem **haddr)
+{
+	unsigned long addr;
+	int ret;
+
+	*kaddr = ioremap(phys_addr, size);
+	if (!*kaddr)
+		return -ENOMEM;
+
+	if (is_kernel_in_hyp_mode()) {
+		*haddr = *kaddr;
+		return 0;
+	}
+
+	ret = __create_hyp_private_mapping(phys_addr, size,
+					   &addr, PAGE_HYP_DEVICE);
 	if (ret) {
 		iounmap(*kaddr);
 		*kaddr = NULL;
+		*haddr = NULL;
+		return ret;
+	}
+
+	*haddr = (void __iomem *)addr;
+	return 0;
+}
+
+/**
+ * create_hyp_exec_mappings - Map an executable range into into HYP
+ * @phys_addr:	The physical start address which gets mapped
+ * @size:	Size of the region being mapped
+ * @haddr:	HYP VA for this mapping
+ */
+int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
+			     void **haddr)
+{
+	unsigned long addr;
+	int ret;
+
+	BUG_ON(is_kernel_in_hyp_mode());
+
+	ret = __create_hyp_private_mapping(phys_addr, size,
+					   &addr, PAGE_HYP_EXEC);
+	if (ret) {
+		*haddr = NULL;
 		return ret;
 	}
 
+	*haddr = (void *)addr;
 	return 0;
 }
 
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 24/26] arm64: Make BP hardening slot counter available
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Catalin Marinas, Will Deacon, Kristina Martsenko

We're about to need to allocate hardening slots from other parts
of the kernel (in order to support ARM64_HARDEN_EL2_VECTORS).

Turn the counter into an atomic_t and make it available to the
rest of the kernel. Also add BP_HARDEN_EL2_SLOTS as the number of
slots instead of the hardcoded 4...

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/include/asm/mmu.h   | 3 +++
 arch/arm64/kernel/bpi.S        | 3 ++-
 arch/arm64/kernel/cpu_errata.c | 9 ++++-----
 3 files changed, 9 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h
index a050d4f3615d..3baf010fe883 100644
--- a/arch/arm64/include/asm/mmu.h
+++ b/arch/arm64/include/asm/mmu.h
@@ -21,6 +21,8 @@
 #define USER_ASID_FLAG	(UL(1) << USER_ASID_BIT)
 #define TTBR_ASID_MASK	(UL(0xffff) << 48)
 
+#define BP_HARDEN_EL2_SLOTS 4
+
 #ifndef __ASSEMBLY__
 
 typedef struct {
@@ -51,6 +53,7 @@ struct bp_hardening_data {
 
 #ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
 extern char __bp_harden_hyp_vecs_start[], __bp_harden_hyp_vecs_end[];
+extern atomic_t arm64_el2_vector_last_slot;
 
 DECLARE_PER_CPU_READ_MOSTLY(struct bp_hardening_data, bp_hardening_data);
 
diff --git a/arch/arm64/kernel/bpi.S b/arch/arm64/kernel/bpi.S
index dc51ef2ce98a..bb0b67722e86 100644
--- a/arch/arm64/kernel/bpi.S
+++ b/arch/arm64/kernel/bpi.S
@@ -20,6 +20,7 @@
 #include <linux/arm-smccc.h>
 
 #include <asm/alternative.h>
+#include <asm/mmu.h>
 
 .macro hyp_ventry
 	.align 7
@@ -66,7 +67,7 @@ alternative_cb_end
 
 	.align	11
 ENTRY(__bp_harden_hyp_vecs_start)
-	.rept 4
+	.rept BP_HARDEN_EL2_SLOTS
 	generate_vectors
 	.endr
 ENTRY(__bp_harden_hyp_vecs_end)
diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c
index 52f15cd896e1..8cf6b60a085a 100644
--- a/arch/arm64/kernel/cpu_errata.c
+++ b/arch/arm64/kernel/cpu_errata.c
@@ -60,6 +60,8 @@ static int cpu_enable_trap_ctr_access(void *__unused)
 	return 0;
 }
 
+atomic_t arm64_el2_vector_last_slot = ATOMIC_INIT(-1);
+
 #ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
 #include <asm/mmu_context.h>
 #include <asm/cacheflush.h>
@@ -90,7 +92,6 @@ static void __install_bp_hardening_cb(bp_hardening_cb_t fn,
 				      const char *hyp_vecs_start,
 				      const char *hyp_vecs_end)
 {
-	static int last_slot = -1;
 	static DEFINE_SPINLOCK(bp_lock);
 	int cpu, slot = -1;
 
@@ -103,10 +104,8 @@ static void __install_bp_hardening_cb(bp_hardening_cb_t fn,
 	}
 
 	if (slot == -1) {
-		last_slot++;
-		BUG_ON(((__bp_harden_hyp_vecs_end - __bp_harden_hyp_vecs_start)
-			/ SZ_2K) <= last_slot);
-		slot = last_slot;
+		slot = atomic_inc_return(&arm64_el2_vector_last_slot);
+		BUG_ON(slot >= BP_HARDEN_EL2_SLOTS);
 		__copy_hyp_vect_bpi(slot, hyp_vecs_start, hyp_vecs_end);
 	}
 
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 24/26] arm64: Make BP hardening slot counter available
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

We're about to need to allocate hardening slots from other parts
of the kernel (in order to support ARM64_HARDEN_EL2_VECTORS).

Turn the counter into an atomic_t and make it available to the
rest of the kernel. Also add BP_HARDEN_EL2_SLOTS as the number of
slots instead of the hardcoded 4...

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/include/asm/mmu.h   | 3 +++
 arch/arm64/kernel/bpi.S        | 3 ++-
 arch/arm64/kernel/cpu_errata.c | 9 ++++-----
 3 files changed, 9 insertions(+), 6 deletions(-)

diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h
index a050d4f3615d..3baf010fe883 100644
--- a/arch/arm64/include/asm/mmu.h
+++ b/arch/arm64/include/asm/mmu.h
@@ -21,6 +21,8 @@
 #define USER_ASID_FLAG	(UL(1) << USER_ASID_BIT)
 #define TTBR_ASID_MASK	(UL(0xffff) << 48)
 
+#define BP_HARDEN_EL2_SLOTS 4
+
 #ifndef __ASSEMBLY__
 
 typedef struct {
@@ -51,6 +53,7 @@ struct bp_hardening_data {
 
 #ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
 extern char __bp_harden_hyp_vecs_start[], __bp_harden_hyp_vecs_end[];
+extern atomic_t arm64_el2_vector_last_slot;
 
 DECLARE_PER_CPU_READ_MOSTLY(struct bp_hardening_data, bp_hardening_data);
 
diff --git a/arch/arm64/kernel/bpi.S b/arch/arm64/kernel/bpi.S
index dc51ef2ce98a..bb0b67722e86 100644
--- a/arch/arm64/kernel/bpi.S
+++ b/arch/arm64/kernel/bpi.S
@@ -20,6 +20,7 @@
 #include <linux/arm-smccc.h>
 
 #include <asm/alternative.h>
+#include <asm/mmu.h>
 
 .macro hyp_ventry
 	.align 7
@@ -66,7 +67,7 @@ alternative_cb_end
 
 	.align	11
 ENTRY(__bp_harden_hyp_vecs_start)
-	.rept 4
+	.rept BP_HARDEN_EL2_SLOTS
 	generate_vectors
 	.endr
 ENTRY(__bp_harden_hyp_vecs_end)
diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c
index 52f15cd896e1..8cf6b60a085a 100644
--- a/arch/arm64/kernel/cpu_errata.c
+++ b/arch/arm64/kernel/cpu_errata.c
@@ -60,6 +60,8 @@ static int cpu_enable_trap_ctr_access(void *__unused)
 	return 0;
 }
 
+atomic_t arm64_el2_vector_last_slot = ATOMIC_INIT(-1);
+
 #ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
 #include <asm/mmu_context.h>
 #include <asm/cacheflush.h>
@@ -90,7 +92,6 @@ static void __install_bp_hardening_cb(bp_hardening_cb_t fn,
 				      const char *hyp_vecs_start,
 				      const char *hyp_vecs_end)
 {
-	static int last_slot = -1;
 	static DEFINE_SPINLOCK(bp_lock);
 	int cpu, slot = -1;
 
@@ -103,10 +104,8 @@ static void __install_bp_hardening_cb(bp_hardening_cb_t fn,
 	}
 
 	if (slot == -1) {
-		last_slot++;
-		BUG_ON(((__bp_harden_hyp_vecs_end - __bp_harden_hyp_vecs_start)
-			/ SZ_2K) <= last_slot);
-		slot = last_slot;
+		slot = atomic_inc_return(&arm64_el2_vector_last_slot);
+		BUG_ON(slot >= BP_HARDEN_EL2_SLOTS);
 		__copy_hyp_vect_bpi(slot, hyp_vecs_start, hyp_vecs_end);
 	}
 
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 25/26] arm64: KVM: Allow mapping of vectors outside of the RAM region
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Catalin Marinas, Will Deacon, Kristina Martsenko

We're now ready to map our vectors in weird and wonderful locations.
On enabling ARM64_HARDEN_EL2_VECTORS, a vector slots gets allocated
if this hasn't been already done via ARM64_HARDEN_BRANCH_PREDICTOR
and gets mapped outside of the normal RAM region, next to the
idmap.

That way, being able to obtain VBAR_EL2 doesn't reveal the mapping
of the rest of the hypervisor code.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 Documentation/arm64/memory.txt   |  3 +-
 arch/arm64/Kconfig               | 16 +++++++++
 arch/arm64/include/asm/kvm_mmu.h | 78 +++++++++++++++++++++++++++++++++++-----
 arch/arm64/include/asm/mmu.h     |  5 ++-
 arch/arm64/kvm/Kconfig           |  2 +-
 arch/arm64/kvm/va_layout.c       |  3 ++
 6 files changed, 95 insertions(+), 12 deletions(-)

diff --git a/Documentation/arm64/memory.txt b/Documentation/arm64/memory.txt
index c58cc5dbe667..c5dab30d3389 100644
--- a/Documentation/arm64/memory.txt
+++ b/Documentation/arm64/memory.txt
@@ -90,7 +90,8 @@ When using KVM without the Virtualization Host Extensions, the
 hypervisor maps kernel pages in EL2 at a fixed (and potentially
 random) offset from the linear mapping. See the kern_hyp_va macro and
 kvm_update_va_mask function for more details. MMIO devices such as
-GICv2 gets mapped next to the HYP idmap page.
+GICv2 gets mapped next to the HYP idmap page, as do vectors when
+ARM64_HARDEN_EL2_VECTORS is selected for particular CPUs.
 
 When using KVM with the Virtualization Host Extensions, no additional
 mappings are created, since the host kernel runs directly in EL2.
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 7381eeb7ef8e..e6be4393aaad 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -904,6 +904,22 @@ config HARDEN_BRANCH_PREDICTOR
 
 	  If unsure, say Y.
 
+config HARDEN_EL2_VECTORS
+	bool "Harden EL2 vector mapping against system register leak" if EXPERT
+	default y
+	help
+	  Speculation attacks against some high-performance processors can
+	  be used to leak privileged information such as the vector base
+	  register, resulting in a potential defeat of the EL2 layout
+	  randomization.
+
+	  This config option will map the vectors to a fixed location,
+	  independent of the EL2 code mapping, so that revealing VBAR_EL2
+	  to an attacker does no give away any extra information. This
+	  only gets enabled on affected CPUs.
+
+	  If unsure, say Y.
+
 menuconfig ARMV8_DEPRECATED
 	bool "Emulate deprecated/obsolete ARMv8 instructions"
 	depends on COMPAT
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 97af11065bbd..f936d0928661 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -360,31 +360,91 @@ static inline unsigned int kvm_get_vmid_bits(void)
 	return (cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR1_VMIDBITS_SHIFT) == 2) ? 16 : 8;
 }
 
-#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+#ifdef CONFIG_KVM_INDIRECT_VECTORS
+/*
+ * EL2 vectors can be mapped and rerouted in a number of ways,
+ * depending on the kernel configuration and CPU present:
+ *
+ * - If the CPU has the ARM64_HARDEN_BRANCH_PREDICTOR cap, the
+ *   hardening sequence is placed in one of the vector slots, which is
+ *   executed before jumping to the real vectors.
+ *
+ * - If the CPU has both the ARM64_HARDEN_EL2_VECTORS and BP
+ *   hardening, the slot containing the hardening sequence is mapped
+ *   next to the idmap page, and executed before jumping to the real
+ *   vectors.
+ *
+ * - If the CPU only has ARM64_HARDEN_EL2_VECTORS, then an empty slot
+ *   is selected, mapped next to the idmap page, and executed before
+ *   jumping to the real vectors.
+ *
+ * Note that ARM64_HARDEN_EL2_VECTORS is somewhat incompatible with
+ * VHE, as we don't have hypervisor-specific mappings. If the system
+ * is VHE and yet selects this capability, it will be ignored.
+ */
 #include <asm/mmu.h>
 
+extern void *__kvm_bp_vect_base;
+extern int __kvm_harden_el2_vector_slot;
+
 static inline void *kvm_get_hyp_vector(void)
 {
 	struct bp_hardening_data *data = arm64_get_bp_hardening_data();
-	void *vect = kvm_ksym_ref(__kvm_hyp_vector);
+	void *vect = kern_hyp_va(kvm_ksym_ref(__kvm_hyp_vector));
+	int slot = -1;
 
-	if (data->fn) {
-		vect = __bp_harden_hyp_vecs_start +
-		       data->hyp_vectors_slot * SZ_2K;
+	if (cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR) && data->fn) {
+		vect = __bp_harden_hyp_vecs_start;
+		slot = data->hyp_vectors_slot;
+	}
 
-		if (!has_vhe())
-			vect = lm_alias(vect);
+	if (this_cpu_has_cap(ARM64_HARDEN_EL2_VECTORS) && !has_vhe()) {
+		vect = __kvm_bp_vect_base;
+		if (slot == -1)
+			slot = __kvm_harden_el2_vector_slot;
 	}
 
-	vect = kern_hyp_va(vect);
+	if (slot != -1)
+		vect += slot * SZ_2K;
+
 	return vect;
 }
 
+/*  This is only called on a !VHE system */
 static inline int kvm_map_vectors(void)
 {
+	/*
+	 * HBP  = ARM64_HARDEN_BRANCH_PREDICTOR
+	 * HLE2 = ARM64_HARDEN_EL2_VECTORS
+	 *
+	 * !HBP + !HEL2 -> use direct vectors
+	 *  HBP + !HEL2 -> use hardenned vectors in place
+	 * !HBP +  HEL2 -> allocate one vector slot and use exec mapping
+	 *  HBP +  HEL2 -> use hardenned vertors and use exec mapping
+	 */
+	if (cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR)) {
+		__kvm_bp_vect_base = kvm_ksym_ref(__bp_harden_hyp_vecs_start);
+		__kvm_bp_vect_base = kern_hyp_va(__kvm_bp_vect_base);
+	}
+
+	if (cpus_have_const_cap(ARM64_HARDEN_EL2_VECTORS)) {
+		phys_addr_t vect_pa = __pa_symbol(__bp_harden_hyp_vecs_start);
+		unsigned long size = (__bp_harden_hyp_vecs_end -
+				      __bp_harden_hyp_vecs_start);
+
+		/*
+		 * Always allocate a spare vector slot, as we don't
+		 * know yet which CPUs have a BP hardening slot that
+		 * we can reuse.
+		 */
+		__kvm_harden_el2_vector_slot = atomic_inc_return(&arm64_el2_vector_last_slot);
+		BUG_ON(__kvm_harden_el2_vector_slot >= BP_HARDEN_EL2_SLOTS);
+		return create_hyp_exec_mappings(vect_pa, size,
+						&__kvm_bp_vect_base);
+	}
+
 	return 0;
 }
-
 #else
 static inline void *kvm_get_hyp_vector(void)
 {
diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h
index 3baf010fe883..0b0cc69031c1 100644
--- a/arch/arm64/include/asm/mmu.h
+++ b/arch/arm64/include/asm/mmu.h
@@ -51,10 +51,12 @@ struct bp_hardening_data {
 	bp_hardening_cb_t	fn;
 };
 
-#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+#if (defined(CONFIG_HARDEN_BRANCH_PREDICTOR) ||	\
+     defined(CONFIG_HARDEN_EL2_VECTORS))
 extern char __bp_harden_hyp_vecs_start[], __bp_harden_hyp_vecs_end[];
 extern atomic_t arm64_el2_vector_last_slot;
 
+#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
 DECLARE_PER_CPU_READ_MOSTLY(struct bp_hardening_data, bp_hardening_data);
 
 static inline struct bp_hardening_data *arm64_get_bp_hardening_data(void)
@@ -81,6 +83,7 @@ static inline struct bp_hardening_data *arm64_get_bp_hardening_data(void)
 
 static inline void arm64_apply_bp_hardening(void)	{ }
 #endif	/* CONFIG_HARDEN_BRANCH_PREDICTOR */
+#endif  /* CONFIG_HARDEN_BRANCH_PREDICTOR || CONFIG_HARDEN_EL2_VECTORS */
 
 extern void paging_init(void);
 extern void bootmem_init(void);
diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
index bd8cc03d7522..a2e3a5af1113 100644
--- a/arch/arm64/kvm/Kconfig
+++ b/arch/arm64/kvm/Kconfig
@@ -58,7 +58,7 @@ config KVM_ARM_PMU
 	  virtual machines.
 
 config KVM_INDIRECT_VECTORS
-       def_bool KVM && HARDEN_BRANCH_PREDICTOR
+       def_bool KVM && (HARDEN_BRANCH_PREDICTOR || HARDEN_EL2_VECTORS)
 
 source drivers/vhost/Kconfig
 
diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c
index 9d35c17016ed..fd2d658a11b5 100644
--- a/arch/arm64/kvm/va_layout.c
+++ b/arch/arm64/kvm/va_layout.c
@@ -151,6 +151,9 @@ void __init kvm_update_va_mask(struct alt_instr *alt,
 	}
 }
 
+void *__kvm_bp_vect_base;
+int __kvm_harden_el2_vector_slot;
+
 void kvm_patch_vector_branch(struct alt_instr *alt,
 			     __le32 *origptr, __le32 *updptr, int nr_inst)
 {
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 25/26] arm64: KVM: Allow mapping of vectors outside of the RAM region
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

We're now ready to map our vectors in weird and wonderful locations.
On enabling ARM64_HARDEN_EL2_VECTORS, a vector slots gets allocated
if this hasn't been already done via ARM64_HARDEN_BRANCH_PREDICTOR
and gets mapped outside of the normal RAM region, next to the
idmap.

That way, being able to obtain VBAR_EL2 doesn't reveal the mapping
of the rest of the hypervisor code.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 Documentation/arm64/memory.txt   |  3 +-
 arch/arm64/Kconfig               | 16 +++++++++
 arch/arm64/include/asm/kvm_mmu.h | 78 +++++++++++++++++++++++++++++++++++-----
 arch/arm64/include/asm/mmu.h     |  5 ++-
 arch/arm64/kvm/Kconfig           |  2 +-
 arch/arm64/kvm/va_layout.c       |  3 ++
 6 files changed, 95 insertions(+), 12 deletions(-)

diff --git a/Documentation/arm64/memory.txt b/Documentation/arm64/memory.txt
index c58cc5dbe667..c5dab30d3389 100644
--- a/Documentation/arm64/memory.txt
+++ b/Documentation/arm64/memory.txt
@@ -90,7 +90,8 @@ When using KVM without the Virtualization Host Extensions, the
 hypervisor maps kernel pages in EL2 at a fixed (and potentially
 random) offset from the linear mapping. See the kern_hyp_va macro and
 kvm_update_va_mask function for more details. MMIO devices such as
-GICv2 gets mapped next to the HYP idmap page.
+GICv2 gets mapped next to the HYP idmap page, as do vectors when
+ARM64_HARDEN_EL2_VECTORS is selected for particular CPUs.
 
 When using KVM with the Virtualization Host Extensions, no additional
 mappings are created, since the host kernel runs directly in EL2.
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 7381eeb7ef8e..e6be4393aaad 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -904,6 +904,22 @@ config HARDEN_BRANCH_PREDICTOR
 
 	  If unsure, say Y.
 
+config HARDEN_EL2_VECTORS
+	bool "Harden EL2 vector mapping against system register leak" if EXPERT
+	default y
+	help
+	  Speculation attacks against some high-performance processors can
+	  be used to leak privileged information such as the vector base
+	  register, resulting in a potential defeat of the EL2 layout
+	  randomization.
+
+	  This config option will map the vectors to a fixed location,
+	  independent of the EL2 code mapping, so that revealing VBAR_EL2
+	  to an attacker does no give away any extra information. This
+	  only gets enabled on affected CPUs.
+
+	  If unsure, say Y.
+
 menuconfig ARMV8_DEPRECATED
 	bool "Emulate deprecated/obsolete ARMv8 instructions"
 	depends on COMPAT
diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
index 97af11065bbd..f936d0928661 100644
--- a/arch/arm64/include/asm/kvm_mmu.h
+++ b/arch/arm64/include/asm/kvm_mmu.h
@@ -360,31 +360,91 @@ static inline unsigned int kvm_get_vmid_bits(void)
 	return (cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR1_VMIDBITS_SHIFT) == 2) ? 16 : 8;
 }
 
-#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+#ifdef CONFIG_KVM_INDIRECT_VECTORS
+/*
+ * EL2 vectors can be mapped and rerouted in a number of ways,
+ * depending on the kernel configuration and CPU present:
+ *
+ * - If the CPU has the ARM64_HARDEN_BRANCH_PREDICTOR cap, the
+ *   hardening sequence is placed in one of the vector slots, which is
+ *   executed before jumping to the real vectors.
+ *
+ * - If the CPU has both the ARM64_HARDEN_EL2_VECTORS and BP
+ *   hardening, the slot containing the hardening sequence is mapped
+ *   next to the idmap page, and executed before jumping to the real
+ *   vectors.
+ *
+ * - If the CPU only has ARM64_HARDEN_EL2_VECTORS, then an empty slot
+ *   is selected, mapped next to the idmap page, and executed before
+ *   jumping to the real vectors.
+ *
+ * Note that ARM64_HARDEN_EL2_VECTORS is somewhat incompatible with
+ * VHE, as we don't have hypervisor-specific mappings. If the system
+ * is VHE and yet selects this capability, it will be ignored.
+ */
 #include <asm/mmu.h>
 
+extern void *__kvm_bp_vect_base;
+extern int __kvm_harden_el2_vector_slot;
+
 static inline void *kvm_get_hyp_vector(void)
 {
 	struct bp_hardening_data *data = arm64_get_bp_hardening_data();
-	void *vect = kvm_ksym_ref(__kvm_hyp_vector);
+	void *vect = kern_hyp_va(kvm_ksym_ref(__kvm_hyp_vector));
+	int slot = -1;
 
-	if (data->fn) {
-		vect = __bp_harden_hyp_vecs_start +
-		       data->hyp_vectors_slot * SZ_2K;
+	if (cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR) && data->fn) {
+		vect = __bp_harden_hyp_vecs_start;
+		slot = data->hyp_vectors_slot;
+	}
 
-		if (!has_vhe())
-			vect = lm_alias(vect);
+	if (this_cpu_has_cap(ARM64_HARDEN_EL2_VECTORS) && !has_vhe()) {
+		vect = __kvm_bp_vect_base;
+		if (slot == -1)
+			slot = __kvm_harden_el2_vector_slot;
 	}
 
-	vect = kern_hyp_va(vect);
+	if (slot != -1)
+		vect += slot * SZ_2K;
+
 	return vect;
 }
 
+/*  This is only called on a !VHE system */
 static inline int kvm_map_vectors(void)
 {
+	/*
+	 * HBP  = ARM64_HARDEN_BRANCH_PREDICTOR
+	 * HLE2 = ARM64_HARDEN_EL2_VECTORS
+	 *
+	 * !HBP + !HEL2 -> use direct vectors
+	 *  HBP + !HEL2 -> use hardenned vectors in place
+	 * !HBP +  HEL2 -> allocate one vector slot and use exec mapping
+	 *  HBP +  HEL2 -> use hardenned vertors and use exec mapping
+	 */
+	if (cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR)) {
+		__kvm_bp_vect_base = kvm_ksym_ref(__bp_harden_hyp_vecs_start);
+		__kvm_bp_vect_base = kern_hyp_va(__kvm_bp_vect_base);
+	}
+
+	if (cpus_have_const_cap(ARM64_HARDEN_EL2_VECTORS)) {
+		phys_addr_t vect_pa = __pa_symbol(__bp_harden_hyp_vecs_start);
+		unsigned long size = (__bp_harden_hyp_vecs_end -
+				      __bp_harden_hyp_vecs_start);
+
+		/*
+		 * Always allocate a spare vector slot, as we don't
+		 * know yet which CPUs have a BP hardening slot that
+		 * we can reuse.
+		 */
+		__kvm_harden_el2_vector_slot = atomic_inc_return(&arm64_el2_vector_last_slot);
+		BUG_ON(__kvm_harden_el2_vector_slot >= BP_HARDEN_EL2_SLOTS);
+		return create_hyp_exec_mappings(vect_pa, size,
+						&__kvm_bp_vect_base);
+	}
+
 	return 0;
 }
-
 #else
 static inline void *kvm_get_hyp_vector(void)
 {
diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h
index 3baf010fe883..0b0cc69031c1 100644
--- a/arch/arm64/include/asm/mmu.h
+++ b/arch/arm64/include/asm/mmu.h
@@ -51,10 +51,12 @@ struct bp_hardening_data {
 	bp_hardening_cb_t	fn;
 };
 
-#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
+#if (defined(CONFIG_HARDEN_BRANCH_PREDICTOR) ||	\
+     defined(CONFIG_HARDEN_EL2_VECTORS))
 extern char __bp_harden_hyp_vecs_start[], __bp_harden_hyp_vecs_end[];
 extern atomic_t arm64_el2_vector_last_slot;
 
+#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
 DECLARE_PER_CPU_READ_MOSTLY(struct bp_hardening_data, bp_hardening_data);
 
 static inline struct bp_hardening_data *arm64_get_bp_hardening_data(void)
@@ -81,6 +83,7 @@ static inline struct bp_hardening_data *arm64_get_bp_hardening_data(void)
 
 static inline void arm64_apply_bp_hardening(void)	{ }
 #endif	/* CONFIG_HARDEN_BRANCH_PREDICTOR */
+#endif  /* CONFIG_HARDEN_BRANCH_PREDICTOR || CONFIG_HARDEN_EL2_VECTORS */
 
 extern void paging_init(void);
 extern void bootmem_init(void);
diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
index bd8cc03d7522..a2e3a5af1113 100644
--- a/arch/arm64/kvm/Kconfig
+++ b/arch/arm64/kvm/Kconfig
@@ -58,7 +58,7 @@ config KVM_ARM_PMU
 	  virtual machines.
 
 config KVM_INDIRECT_VECTORS
-       def_bool KVM && HARDEN_BRANCH_PREDICTOR
+       def_bool KVM && (HARDEN_BRANCH_PREDICTOR || HARDEN_EL2_VECTORS)
 
 source drivers/vhost/Kconfig
 
diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c
index 9d35c17016ed..fd2d658a11b5 100644
--- a/arch/arm64/kvm/va_layout.c
+++ b/arch/arm64/kvm/va_layout.c
@@ -151,6 +151,9 @@ void __init kvm_update_va_mask(struct alt_instr *alt,
 	}
 }
 
+void *__kvm_bp_vect_base;
+int __kvm_harden_el2_vector_slot;
+
 void kvm_patch_vector_branch(struct alt_instr *alt,
 			     __le32 *origptr, __le32 *updptr, int nr_inst)
 {
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 26/26] arm64: Enable ARM64_HARDEN_EL2_VECTORS on Cortex-A57 and A72
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-14 16:50   ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel, kvm, kvmarm
  Cc: Catalin Marinas, Will Deacon, Kristina Martsenko

Cortex-A57 and A72 are vulnerable to the so-called "variant 3a" of
Meltdown, where an attacker can speculatively obtain the value
of a privileged system register.

By enabling ARM64_HARDEN_EL2_VECTORS on these CPUs, obtaining
VBAR_EL2 is not disclosing the hypervisor mappings anymore.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/kernel/cpu_errata.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c
index 8cf6b60a085a..aacdc118c4c9 100644
--- a/arch/arm64/kernel/cpu_errata.c
+++ b/arch/arm64/kernel/cpu_errata.c
@@ -424,6 +424,18 @@ const struct arm64_cpu_capabilities arm64_errata[] = {
 		MIDR_ALL_VERSIONS(MIDR_CAVIUM_THUNDERX2),
 		.enable = enable_smccc_arch_workaround_1,
 	},
+#endif
+#ifdef CONFIG_HARDEN_EL2_VECTORS
+	{
+		.desc = "Cortex-A57 EL2 vector hardening",
+		.capability = ARM64_HARDEN_EL2_VECTORS,
+		MIDR_ALL_VERSIONS(MIDR_CORTEX_A57),
+	},
+	{
+		.desc = "Cortex-A72 EL2 vector hardening",
+		.capability = ARM64_HARDEN_EL2_VECTORS,
+		MIDR_ALL_VERSIONS(MIDR_CORTEX_A72),
+	},
 #endif
 	{
 	}
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 26/26] arm64: Enable ARM64_HARDEN_EL2_VECTORS on Cortex-A57 and A72
@ 2018-03-14 16:50   ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-14 16:50 UTC (permalink / raw)
  To: linux-arm-kernel

Cortex-A57 and A72 are vulnerable to the so-called "variant 3a" of
Meltdown, where an attacker can speculatively obtain the value
of a privileged system register.

By enabling ARM64_HARDEN_EL2_VECTORS on these CPUs, obtaining
VBAR_EL2 is not disclosing the hypervisor mappings anymore.

Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
---
 arch/arm64/kernel/cpu_errata.c | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c
index 8cf6b60a085a..aacdc118c4c9 100644
--- a/arch/arm64/kernel/cpu_errata.c
+++ b/arch/arm64/kernel/cpu_errata.c
@@ -424,6 +424,18 @@ const struct arm64_cpu_capabilities arm64_errata[] = {
 		MIDR_ALL_VERSIONS(MIDR_CAVIUM_THUNDERX2),
 		.enable = enable_smccc_arch_workaround_1,
 	},
+#endif
+#ifdef CONFIG_HARDEN_EL2_VECTORS
+	{
+		.desc = "Cortex-A57 EL2 vector hardening",
+		.capability = ARM64_HARDEN_EL2_VECTORS,
+		MIDR_ALL_VERSIONS(MIDR_CORTEX_A57),
+	},
+	{
+		.desc = "Cortex-A72 EL2 vector hardening",
+		.capability = ARM64_HARDEN_EL2_VECTORS,
+		MIDR_ALL_VERSIONS(MIDR_CORTEX_A72),
+	},
 #endif
 	{
 	}
-- 
2.14.2

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 19/26] arm64: KVM: Move stashing of x0/x1 into the vector code itself
  2018-03-14 16:50   ` Marc Zyngier
@ 2018-03-15 14:39     ` Andrew Jones
  -1 siblings, 0 replies; 124+ messages in thread
From: Andrew Jones @ 2018-03-15 14:39 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: Mark Rutland, Peter Maydell, Christoffer Dall, kvm, Steve Capper,
	Catalin Marinas, Will Deacon, Kristina Martsenko, James Morse,
	kvmarm, linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:42PM +0000, Marc Zyngier wrote:
> All our useful entry points into the hypervisor are starting by
> saving x0 and x1 on the stack. Let's move those into the vectors
> by introducing macros that annotate whether a vector is valid or
> not, thus indicating whether we want to stash registers or not.
> 
> The only drawback is that we now also stash registers for el2_error,
> but this should never happen, and we pop them back right at the
> start of the handling sequence.
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> ---
>  arch/arm64/kvm/hyp/hyp-entry.S | 56 ++++++++++++++++++++++++------------------
>  1 file changed, 32 insertions(+), 24 deletions(-)
>

Reviewed-by: Andrew Jones <drjones@redhat.com>

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 19/26] arm64: KVM: Move stashing of x0/x1 into the vector code itself
@ 2018-03-15 14:39     ` Andrew Jones
  0 siblings, 0 replies; 124+ messages in thread
From: Andrew Jones @ 2018-03-15 14:39 UTC (permalink / raw)
  To: linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:42PM +0000, Marc Zyngier wrote:
> All our useful entry points into the hypervisor are starting by
> saving x0 and x1 on the stack. Let's move those into the vectors
> by introducing macros that annotate whether a vector is valid or
> not, thus indicating whether we want to stash registers or not.
> 
> The only drawback is that we now also stash registers for el2_error,
> but this should never happen, and we pop them back right at the
> start of the handling sequence.
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> ---
>  arch/arm64/kvm/hyp/hyp-entry.S | 56 ++++++++++++++++++++++++------------------
>  1 file changed, 32 insertions(+), 24 deletions(-)
>

Reviewed-by: Andrew Jones <drjones@redhat.com>

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 20/26] arm64: KVM: Move BP hardening vectors into .hyp.text section
  2018-03-14 16:50   ` Marc Zyngier
@ 2018-03-15 14:42     ` Andrew Jones
  -1 siblings, 0 replies; 124+ messages in thread
From: Andrew Jones @ 2018-03-15 14:42 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: Mark Rutland, Peter Maydell, Christoffer Dall, kvm, Steve Capper,
	Catalin Marinas, Will Deacon, Kristina Martsenko, James Morse,
	kvmarm, linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:43PM +0000, Marc Zyngier wrote:
> There is no reason why the BP hardening vectors shouldn't be part
> of the HYP text at compile time, rather than being mapped at runtime.
> 
> Also introduce a new config symbol that controls the compilation
> of bpi.S.
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> ---
>  arch/arm64/include/asm/kvm_mmu.h | 4 +---
>  arch/arm64/kernel/Makefile       | 4 +---
>  arch/arm64/kernel/bpi.S          | 6 ++++++
>  arch/arm64/kvm/Kconfig           | 3 +++
>  4 files changed, 11 insertions(+), 6 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
> index bfb81e73da06..c2beb2d25170 100644
> --- a/arch/arm64/include/asm/kvm_mmu.h
> +++ b/arch/arm64/include/asm/kvm_mmu.h
> @@ -380,9 +380,7 @@ static inline void *kvm_get_hyp_vector(void)
>  
>  static inline int kvm_map_vectors(void)
>  {
> -	return create_hyp_mappings(kvm_ksym_ref(__bp_harden_hyp_vecs_start),
> -				   kvm_ksym_ref(__bp_harden_hyp_vecs_end),
> -				   PAGE_HYP_EXEC);
> +	return 0;
>  }
>  
>  #else
> diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile
> index b87541360f43..93bce17109a6 100644
> --- a/arch/arm64/kernel/Makefile
> +++ b/arch/arm64/kernel/Makefile
> @@ -54,9 +54,7 @@ arm64-reloc-test-y := reloc_test_core.o reloc_test_syms.o
>  arm64-obj-$(CONFIG_CRASH_DUMP)		+= crash_dump.o
>  arm64-obj-$(CONFIG_ARM_SDE_INTERFACE)	+= sdei.o
>  
> -ifeq ($(CONFIG_KVM),y)
> -arm64-obj-$(CONFIG_HARDEN_BRANCH_PREDICTOR)	+= bpi.o
> -endif
> +arm64-obj-$(CONFIG_KVM_INDIRECT_VECTORS)+= bpi.o

I was about to say a space before the += would be nice, but I see the
operator actually lines up with the others this way.

>  
>  obj-y					+= $(arm64-obj-y) vdso/ probes/
>  obj-m					+= $(arm64-obj-m)
> diff --git a/arch/arm64/kernel/bpi.S b/arch/arm64/kernel/bpi.S
> index e5de33513b5d..447188e2a664 100644
> --- a/arch/arm64/kernel/bpi.S
> +++ b/arch/arm64/kernel/bpi.S
> @@ -48,6 +48,10 @@
>  	ventry \target + 0x780
>  .endm
>  
> +
> +	.text
> +	.pushsection	.hyp.text, "ax"
> +
>  	.align	11
>  ENTRY(__bp_harden_hyp_vecs_start)
>  	.rept 4
> @@ -55,6 +59,8 @@ ENTRY(__bp_harden_hyp_vecs_start)
>  	.endr
>  ENTRY(__bp_harden_hyp_vecs_end)
>  
> +	.popsection
> +
>  ENTRY(__qcom_hyp_sanitize_link_stack_start)
>  	stp     x29, x30, [sp, #-16]!
>  	.rept	16
> diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
> index 2257dfcc44cc..bd8cc03d7522 100644
> --- a/arch/arm64/kvm/Kconfig
> +++ b/arch/arm64/kvm/Kconfig
> @@ -57,6 +57,9 @@ config KVM_ARM_PMU
>  	  Adds support for a virtual Performance Monitoring Unit (PMU) in
>  	  virtual machines.
>  
> +config KVM_INDIRECT_VECTORS
> +       def_bool KVM && HARDEN_BRANCH_PREDICTOR
> +
>  source drivers/vhost/Kconfig
>  
>  endif # VIRTUALIZATION
> -- 
> 2.14.2
> 

Reviewed-by: Andrew Jones <drjones@redhat.com>

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 20/26] arm64: KVM: Move BP hardening vectors into .hyp.text section
@ 2018-03-15 14:42     ` Andrew Jones
  0 siblings, 0 replies; 124+ messages in thread
From: Andrew Jones @ 2018-03-15 14:42 UTC (permalink / raw)
  To: linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:43PM +0000, Marc Zyngier wrote:
> There is no reason why the BP hardening vectors shouldn't be part
> of the HYP text at compile time, rather than being mapped at runtime.
> 
> Also introduce a new config symbol that controls the compilation
> of bpi.S.
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> ---
>  arch/arm64/include/asm/kvm_mmu.h | 4 +---
>  arch/arm64/kernel/Makefile       | 4 +---
>  arch/arm64/kernel/bpi.S          | 6 ++++++
>  arch/arm64/kvm/Kconfig           | 3 +++
>  4 files changed, 11 insertions(+), 6 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
> index bfb81e73da06..c2beb2d25170 100644
> --- a/arch/arm64/include/asm/kvm_mmu.h
> +++ b/arch/arm64/include/asm/kvm_mmu.h
> @@ -380,9 +380,7 @@ static inline void *kvm_get_hyp_vector(void)
>  
>  static inline int kvm_map_vectors(void)
>  {
> -	return create_hyp_mappings(kvm_ksym_ref(__bp_harden_hyp_vecs_start),
> -				   kvm_ksym_ref(__bp_harden_hyp_vecs_end),
> -				   PAGE_HYP_EXEC);
> +	return 0;
>  }
>  
>  #else
> diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile
> index b87541360f43..93bce17109a6 100644
> --- a/arch/arm64/kernel/Makefile
> +++ b/arch/arm64/kernel/Makefile
> @@ -54,9 +54,7 @@ arm64-reloc-test-y := reloc_test_core.o reloc_test_syms.o
>  arm64-obj-$(CONFIG_CRASH_DUMP)		+= crash_dump.o
>  arm64-obj-$(CONFIG_ARM_SDE_INTERFACE)	+= sdei.o
>  
> -ifeq ($(CONFIG_KVM),y)
> -arm64-obj-$(CONFIG_HARDEN_BRANCH_PREDICTOR)	+= bpi.o
> -endif
> +arm64-obj-$(CONFIG_KVM_INDIRECT_VECTORS)+= bpi.o

I was about to say a space before the += would be nice, but I see the
operator actually lines up with the others this way.

>  
>  obj-y					+= $(arm64-obj-y) vdso/ probes/
>  obj-m					+= $(arm64-obj-m)
> diff --git a/arch/arm64/kernel/bpi.S b/arch/arm64/kernel/bpi.S
> index e5de33513b5d..447188e2a664 100644
> --- a/arch/arm64/kernel/bpi.S
> +++ b/arch/arm64/kernel/bpi.S
> @@ -48,6 +48,10 @@
>  	ventry \target + 0x780
>  .endm
>  
> +
> +	.text
> +	.pushsection	.hyp.text, "ax"
> +
>  	.align	11
>  ENTRY(__bp_harden_hyp_vecs_start)
>  	.rept 4
> @@ -55,6 +59,8 @@ ENTRY(__bp_harden_hyp_vecs_start)
>  	.endr
>  ENTRY(__bp_harden_hyp_vecs_end)
>  
> +	.popsection
> +
>  ENTRY(__qcom_hyp_sanitize_link_stack_start)
>  	stp     x29, x30, [sp, #-16]!
>  	.rept	16
> diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
> index 2257dfcc44cc..bd8cc03d7522 100644
> --- a/arch/arm64/kvm/Kconfig
> +++ b/arch/arm64/kvm/Kconfig
> @@ -57,6 +57,9 @@ config KVM_ARM_PMU
>  	  Adds support for a virtual Performance Monitoring Unit (PMU) in
>  	  virtual machines.
>  
> +config KVM_INDIRECT_VECTORS
> +       def_bool KVM && HARDEN_BRANCH_PREDICTOR
> +
>  source drivers/vhost/Kconfig
>  
>  endif # VIRTUALIZATION
> -- 
> 2.14.2
> 

Reviewed-by: Andrew Jones <drjones@redhat.com>

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 21/26] arm64: KVM: Reserve 4 additional instructions in the BPI template
  2018-03-14 16:50   ` Marc Zyngier
@ 2018-03-15 14:46     ` Andrew Jones
  -1 siblings, 0 replies; 124+ messages in thread
From: Andrew Jones @ 2018-03-15 14:46 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: kvm, Catalin Marinas, Will Deacon, Kristina Martsenko, kvmarm,
	linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:44PM +0000, Marc Zyngier wrote:
> So far, we only reserve a single instruction in the BPI template in
> order to branch to the vectors. As we're going to stuff a few more
> instructions there, let's reserve a total of 5 instructions, which
> we're going to patch later on as required.
> 
> We also introduce a small refactor of the vectors themselves, so that
> we stop carrying the target branch around.
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> ---
>  arch/arm64/kernel/bpi.S | 39 +++++++++++++++------------------------
>  1 file changed, 15 insertions(+), 24 deletions(-)
>

Reviewed-by: Andrew Jones <drjones@redhat.com>

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 21/26] arm64: KVM: Reserve 4 additional instructions in the BPI template
@ 2018-03-15 14:46     ` Andrew Jones
  0 siblings, 0 replies; 124+ messages in thread
From: Andrew Jones @ 2018-03-15 14:46 UTC (permalink / raw)
  To: linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:44PM +0000, Marc Zyngier wrote:
> So far, we only reserve a single instruction in the BPI template in
> order to branch to the vectors. As we're going to stuff a few more
> instructions there, let's reserve a total of 5 instructions, which
> we're going to patch later on as required.
> 
> We also introduce a small refactor of the vectors themselves, so that
> we stop carrying the target branch around.
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> ---
>  arch/arm64/kernel/bpi.S | 39 +++++++++++++++------------------------
>  1 file changed, 15 insertions(+), 24 deletions(-)
>

Reviewed-by: Andrew Jones <drjones@redhat.com>

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 23/26] arm/arm64: KVM: Introduce EL2-specific executable mappings
  2018-03-14 16:50   ` Marc Zyngier
@ 2018-03-15 15:03     ` Andrew Jones
  -1 siblings, 0 replies; 124+ messages in thread
From: Andrew Jones @ 2018-03-15 15:03 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: kvm, Catalin Marinas, Will Deacon, Kristina Martsenko, kvmarm,
	linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:46PM +0000, Marc Zyngier wrote:
> Until now, all EL2 executable mappings were derived from their
> EL1 VA. Since we want to decouple the vectors mapping from
> the rest of the hypervisor, we need to be able to map some
> text somewhere else.
> 
> The "idmap" region (for lack of a better name) is ideally suited
> for this, as we have a huge range that hardly has anything in it.
> 
> Let's extend the IO allocator to also deal with executable mappings,
> thus providing the required feature.
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> ---
>  arch/arm/include/asm/kvm_mmu.h   |  2 +
>  arch/arm64/include/asm/kvm_mmu.h |  2 +
>  virt/kvm/arm/mmu.c               | 80 +++++++++++++++++++++++++++++-----------
>  3 files changed, 63 insertions(+), 21 deletions(-)
> 
> diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
> index 26eb6b1cec9b..1b7f592eae57 100644
> --- a/arch/arm/include/asm/kvm_mmu.h
> +++ b/arch/arm/include/asm/kvm_mmu.h
> @@ -56,6 +56,8 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot);
>  int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
>  			   void __iomem **kaddr,
>  			   void __iomem **haddr);
> +int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
> +			     void **haddr);
>  void free_hyp_pgds(void);
>  
>  void stage2_unmap_vm(struct kvm *kvm);
> diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
> index c2beb2d25170..97af11065bbd 100644
> --- a/arch/arm64/include/asm/kvm_mmu.h
> +++ b/arch/arm64/include/asm/kvm_mmu.h
> @@ -151,6 +151,8 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot);
>  int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
>  			   void __iomem **kaddr,
>  			   void __iomem **haddr);
> +int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
> +			     void **haddr);
>  void free_hyp_pgds(void);
>  
>  void stage2_unmap_vm(struct kvm *kvm);
> diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
> index b4d1948c8dd6..554ad5493e7d 100644
> --- a/virt/kvm/arm/mmu.c
> +++ b/virt/kvm/arm/mmu.c
> @@ -737,30 +737,13 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot)
>  	return 0;
>  }
>  
> -/**
> - * create_hyp_io_mappings - Map IO into both kernel and HYP
> - * @phys_addr:	The physical start address which gets mapped
> - * @size:	Size of the region being mapped
> - * @kaddr:	Kernel VA for this mapping
> - * @haddr:	HYP VA for this mapping
> - */
> -int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
> -			   void __iomem **kaddr,
> -			   void __iomem **haddr)
> +static int __create_hyp_private_mapping(phys_addr_t phys_addr, size_t size,
> +					unsigned long *haddr, pgprot_t prot)
>  {
>  	pgd_t *pgd = hyp_pgd;
>  	unsigned long base;
>  	int ret;
>  
> -	*kaddr = ioremap(phys_addr, size);
> -	if (!*kaddr)
> -		return -ENOMEM;
> -
> -	if (is_kernel_in_hyp_mode()) {
> -		*haddr = *kaddr;
> -		return 0;
> -	}
> -
>  	mutex_lock(&io_map_lock);
>  
>  	/*
> @@ -789,22 +772,77 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
>  
>  	ret = __create_hyp_mappings(pgd, __kvm_idmap_ptrs_per_pgd(),
>  				    base, base + size,
> -				    __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
> +				    __phys_to_pfn(phys_addr), prot);
>  	if (ret)
>  		goto out;
>  
> -	*haddr = (void __iomem *)base + offset_in_page(phys_addr);
> +	*haddr = base + offset_in_page(phys_addr);
>  	io_map_base = base;
>  
>  out:
>  	mutex_unlock(&io_map_lock);
>  
> +	return ret;
> +}
> +
> +/**
> + * create_hyp_io_mappings - Map IO into both kernel and HYP
> + * @phys_addr:	The physical start address which gets mapped
> + * @size:	Size of the region being mapped
> + * @kaddr:	Kernel VA for this mapping
> + * @haddr:	HYP VA for this mapping
> + */
> +int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
> +			   void __iomem **kaddr,
> +			   void __iomem **haddr)
> +{
> +	unsigned long addr;
> +	int ret;
> +
> +	*kaddr = ioremap(phys_addr, size);
> +	if (!*kaddr)
> +		return -ENOMEM;
> +
> +	if (is_kernel_in_hyp_mode()) {
> +		*haddr = *kaddr;
> +		return 0;
> +	}
> +
> +	ret = __create_hyp_private_mapping(phys_addr, size,
> +					   &addr, PAGE_HYP_DEVICE);
>  	if (ret) {
>  		iounmap(*kaddr);
>  		*kaddr = NULL;
> +		*haddr = NULL;
> +		return ret;
> +	}
> +
> +	*haddr = (void __iomem *)addr;
> +	return 0;
> +}
> +
> +/**
> + * create_hyp_exec_mappings - Map an executable range into into HYP

s/into into/into/

> + * @phys_addr:	The physical start address which gets mapped
> + * @size:	Size of the region being mapped
> + * @haddr:	HYP VA for this mapping
> + */
> +int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
> +			     void **haddr)
> +{
> +	unsigned long addr;
> +	int ret;
> +
> +	BUG_ON(is_kernel_in_hyp_mode());

Why BUG_ON instead of just giving the caller the same address, similar
to what create_hyp_io_mappings() does?

> +
> +	ret = __create_hyp_private_mapping(phys_addr, size,
> +					   &addr, PAGE_HYP_EXEC);
> +	if (ret) {
> +		*haddr = NULL;
>  		return ret;
>  	}
>  
> +	*haddr = (void *)addr;
>  	return 0;
>  }
>  
> -- 
> 2.14.2
>

Otherwise

Reviewed-by: Andrew Jones <drjones@redhat.com>

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 23/26] arm/arm64: KVM: Introduce EL2-specific executable mappings
@ 2018-03-15 15:03     ` Andrew Jones
  0 siblings, 0 replies; 124+ messages in thread
From: Andrew Jones @ 2018-03-15 15:03 UTC (permalink / raw)
  To: linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:46PM +0000, Marc Zyngier wrote:
> Until now, all EL2 executable mappings were derived from their
> EL1 VA. Since we want to decouple the vectors mapping from
> the rest of the hypervisor, we need to be able to map some
> text somewhere else.
> 
> The "idmap" region (for lack of a better name) is ideally suited
> for this, as we have a huge range that hardly has anything in it.
> 
> Let's extend the IO allocator to also deal with executable mappings,
> thus providing the required feature.
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> ---
>  arch/arm/include/asm/kvm_mmu.h   |  2 +
>  arch/arm64/include/asm/kvm_mmu.h |  2 +
>  virt/kvm/arm/mmu.c               | 80 +++++++++++++++++++++++++++++-----------
>  3 files changed, 63 insertions(+), 21 deletions(-)
> 
> diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
> index 26eb6b1cec9b..1b7f592eae57 100644
> --- a/arch/arm/include/asm/kvm_mmu.h
> +++ b/arch/arm/include/asm/kvm_mmu.h
> @@ -56,6 +56,8 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot);
>  int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
>  			   void __iomem **kaddr,
>  			   void __iomem **haddr);
> +int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
> +			     void **haddr);
>  void free_hyp_pgds(void);
>  
>  void stage2_unmap_vm(struct kvm *kvm);
> diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
> index c2beb2d25170..97af11065bbd 100644
> --- a/arch/arm64/include/asm/kvm_mmu.h
> +++ b/arch/arm64/include/asm/kvm_mmu.h
> @@ -151,6 +151,8 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot);
>  int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
>  			   void __iomem **kaddr,
>  			   void __iomem **haddr);
> +int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
> +			     void **haddr);
>  void free_hyp_pgds(void);
>  
>  void stage2_unmap_vm(struct kvm *kvm);
> diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
> index b4d1948c8dd6..554ad5493e7d 100644
> --- a/virt/kvm/arm/mmu.c
> +++ b/virt/kvm/arm/mmu.c
> @@ -737,30 +737,13 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot)
>  	return 0;
>  }
>  
> -/**
> - * create_hyp_io_mappings - Map IO into both kernel and HYP
> - * @phys_addr:	The physical start address which gets mapped
> - * @size:	Size of the region being mapped
> - * @kaddr:	Kernel VA for this mapping
> - * @haddr:	HYP VA for this mapping
> - */
> -int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
> -			   void __iomem **kaddr,
> -			   void __iomem **haddr)
> +static int __create_hyp_private_mapping(phys_addr_t phys_addr, size_t size,
> +					unsigned long *haddr, pgprot_t prot)
>  {
>  	pgd_t *pgd = hyp_pgd;
>  	unsigned long base;
>  	int ret;
>  
> -	*kaddr = ioremap(phys_addr, size);
> -	if (!*kaddr)
> -		return -ENOMEM;
> -
> -	if (is_kernel_in_hyp_mode()) {
> -		*haddr = *kaddr;
> -		return 0;
> -	}
> -
>  	mutex_lock(&io_map_lock);
>  
>  	/*
> @@ -789,22 +772,77 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
>  
>  	ret = __create_hyp_mappings(pgd, __kvm_idmap_ptrs_per_pgd(),
>  				    base, base + size,
> -				    __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
> +				    __phys_to_pfn(phys_addr), prot);
>  	if (ret)
>  		goto out;
>  
> -	*haddr = (void __iomem *)base + offset_in_page(phys_addr);
> +	*haddr = base + offset_in_page(phys_addr);
>  	io_map_base = base;
>  
>  out:
>  	mutex_unlock(&io_map_lock);
>  
> +	return ret;
> +}
> +
> +/**
> + * create_hyp_io_mappings - Map IO into both kernel and HYP
> + * @phys_addr:	The physical start address which gets mapped
> + * @size:	Size of the region being mapped
> + * @kaddr:	Kernel VA for this mapping
> + * @haddr:	HYP VA for this mapping
> + */
> +int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
> +			   void __iomem **kaddr,
> +			   void __iomem **haddr)
> +{
> +	unsigned long addr;
> +	int ret;
> +
> +	*kaddr = ioremap(phys_addr, size);
> +	if (!*kaddr)
> +		return -ENOMEM;
> +
> +	if (is_kernel_in_hyp_mode()) {
> +		*haddr = *kaddr;
> +		return 0;
> +	}
> +
> +	ret = __create_hyp_private_mapping(phys_addr, size,
> +					   &addr, PAGE_HYP_DEVICE);
>  	if (ret) {
>  		iounmap(*kaddr);
>  		*kaddr = NULL;
> +		*haddr = NULL;
> +		return ret;
> +	}
> +
> +	*haddr = (void __iomem *)addr;
> +	return 0;
> +}
> +
> +/**
> + * create_hyp_exec_mappings - Map an executable range into into HYP

s/into into/into/

> + * @phys_addr:	The physical start address which gets mapped
> + * @size:	Size of the region being mapped
> + * @haddr:	HYP VA for this mapping
> + */
> +int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
> +			     void **haddr)
> +{
> +	unsigned long addr;
> +	int ret;
> +
> +	BUG_ON(is_kernel_in_hyp_mode());

Why BUG_ON instead of just giving the caller the same address, similar
to what create_hyp_io_mappings() does?

> +
> +	ret = __create_hyp_private_mapping(phys_addr, size,
> +					   &addr, PAGE_HYP_EXEC);
> +	if (ret) {
> +		*haddr = NULL;
>  		return ret;
>  	}
>  
> +	*haddr = (void *)addr;
>  	return 0;
>  }
>  
> -- 
> 2.14.2
>

Otherwise

Reviewed-by: Andrew Jones <drjones@redhat.com>

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 24/26] arm64: Make BP hardening slot counter available
  2018-03-14 16:50   ` Marc Zyngier
@ 2018-03-15 15:05     ` Andrew Jones
  -1 siblings, 0 replies; 124+ messages in thread
From: Andrew Jones @ 2018-03-15 15:05 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: Mark Rutland, Peter Maydell, Christoffer Dall, kvm, Steve Capper,
	Catalin Marinas, Will Deacon, Kristina Martsenko, James Morse,
	kvmarm, linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:47PM +0000, Marc Zyngier wrote:
> We're about to need to allocate hardening slots from other parts
> of the kernel (in order to support ARM64_HARDEN_EL2_VECTORS).
> 
> Turn the counter into an atomic_t and make it available to the
> rest of the kernel. Also add BP_HARDEN_EL2_SLOTS as the number of
> slots instead of the hardcoded 4...
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> ---
>  arch/arm64/include/asm/mmu.h   | 3 +++
>  arch/arm64/kernel/bpi.S        | 3 ++-
>  arch/arm64/kernel/cpu_errata.c | 9 ++++-----
>  3 files changed, 9 insertions(+), 6 deletions(-)
>

Reviewed-by: Andrew Jones <drjones@redhat.com>

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 24/26] arm64: Make BP hardening slot counter available
@ 2018-03-15 15:05     ` Andrew Jones
  0 siblings, 0 replies; 124+ messages in thread
From: Andrew Jones @ 2018-03-15 15:05 UTC (permalink / raw)
  To: linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:47PM +0000, Marc Zyngier wrote:
> We're about to need to allocate hardening slots from other parts
> of the kernel (in order to support ARM64_HARDEN_EL2_VECTORS).
> 
> Turn the counter into an atomic_t and make it available to the
> rest of the kernel. Also add BP_HARDEN_EL2_SLOTS as the number of
> slots instead of the hardcoded 4...
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> ---
>  arch/arm64/include/asm/mmu.h   | 3 +++
>  arch/arm64/kernel/bpi.S        | 3 ++-
>  arch/arm64/kernel/cpu_errata.c | 9 ++++-----
>  3 files changed, 9 insertions(+), 6 deletions(-)
>

Reviewed-by: Andrew Jones <drjones@redhat.com>

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 23/26] arm/arm64: KVM: Introduce EL2-specific executable mappings
  2018-03-15 15:03     ` Andrew Jones
@ 2018-03-15 15:53       ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-15 15:53 UTC (permalink / raw)
  To: Andrew Jones
  Cc: kvm, Catalin Marinas, Will Deacon, Kristina Martsenko, kvmarm,
	linux-arm-kernel

On 15/03/18 15:03, Andrew Jones wrote:
> On Wed, Mar 14, 2018 at 04:50:46PM +0000, Marc Zyngier wrote:
>> Until now, all EL2 executable mappings were derived from their
>> EL1 VA. Since we want to decouple the vectors mapping from
>> the rest of the hypervisor, we need to be able to map some
>> text somewhere else.
>>
>> The "idmap" region (for lack of a better name) is ideally suited
>> for this, as we have a huge range that hardly has anything in it.
>>
>> Let's extend the IO allocator to also deal with executable mappings,
>> thus providing the required feature.
>>
>> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
>> ---
>>  arch/arm/include/asm/kvm_mmu.h   |  2 +
>>  arch/arm64/include/asm/kvm_mmu.h |  2 +
>>  virt/kvm/arm/mmu.c               | 80 +++++++++++++++++++++++++++++-----------
>>  3 files changed, 63 insertions(+), 21 deletions(-)
>>
>> diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
>> index 26eb6b1cec9b..1b7f592eae57 100644
>> --- a/arch/arm/include/asm/kvm_mmu.h
>> +++ b/arch/arm/include/asm/kvm_mmu.h
>> @@ -56,6 +56,8 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot);
>>  int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
>>  			   void __iomem **kaddr,
>>  			   void __iomem **haddr);
>> +int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
>> +			     void **haddr);
>>  void free_hyp_pgds(void);
>>  
>>  void stage2_unmap_vm(struct kvm *kvm);
>> diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
>> index c2beb2d25170..97af11065bbd 100644
>> --- a/arch/arm64/include/asm/kvm_mmu.h
>> +++ b/arch/arm64/include/asm/kvm_mmu.h
>> @@ -151,6 +151,8 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot);
>>  int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
>>  			   void __iomem **kaddr,
>>  			   void __iomem **haddr);
>> +int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
>> +			     void **haddr);
>>  void free_hyp_pgds(void);
>>  
>>  void stage2_unmap_vm(struct kvm *kvm);
>> diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
>> index b4d1948c8dd6..554ad5493e7d 100644
>> --- a/virt/kvm/arm/mmu.c
>> +++ b/virt/kvm/arm/mmu.c
>> @@ -737,30 +737,13 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot)
>>  	return 0;
>>  }
>>  
>> -/**
>> - * create_hyp_io_mappings - Map IO into both kernel and HYP
>> - * @phys_addr:	The physical start address which gets mapped
>> - * @size:	Size of the region being mapped
>> - * @kaddr:	Kernel VA for this mapping
>> - * @haddr:	HYP VA for this mapping
>> - */
>> -int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
>> -			   void __iomem **kaddr,
>> -			   void __iomem **haddr)
>> +static int __create_hyp_private_mapping(phys_addr_t phys_addr, size_t size,
>> +					unsigned long *haddr, pgprot_t prot)
>>  {
>>  	pgd_t *pgd = hyp_pgd;
>>  	unsigned long base;
>>  	int ret;
>>  
>> -	*kaddr = ioremap(phys_addr, size);
>> -	if (!*kaddr)
>> -		return -ENOMEM;
>> -
>> -	if (is_kernel_in_hyp_mode()) {
>> -		*haddr = *kaddr;
>> -		return 0;
>> -	}
>> -
>>  	mutex_lock(&io_map_lock);
>>  
>>  	/*
>> @@ -789,22 +772,77 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
>>  
>>  	ret = __create_hyp_mappings(pgd, __kvm_idmap_ptrs_per_pgd(),
>>  				    base, base + size,
>> -				    __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
>> +				    __phys_to_pfn(phys_addr), prot);
>>  	if (ret)
>>  		goto out;
>>  
>> -	*haddr = (void __iomem *)base + offset_in_page(phys_addr);
>> +	*haddr = base + offset_in_page(phys_addr);
>>  	io_map_base = base;
>>  
>>  out:
>>  	mutex_unlock(&io_map_lock);
>>  
>> +	return ret;
>> +}
>> +
>> +/**
>> + * create_hyp_io_mappings - Map IO into both kernel and HYP
>> + * @phys_addr:	The physical start address which gets mapped
>> + * @size:	Size of the region being mapped
>> + * @kaddr:	Kernel VA for this mapping
>> + * @haddr:	HYP VA for this mapping
>> + */
>> +int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
>> +			   void __iomem **kaddr,
>> +			   void __iomem **haddr)
>> +{
>> +	unsigned long addr;
>> +	int ret;
>> +
>> +	*kaddr = ioremap(phys_addr, size);
>> +	if (!*kaddr)
>> +		return -ENOMEM;
>> +
>> +	if (is_kernel_in_hyp_mode()) {
>> +		*haddr = *kaddr;
>> +		return 0;
>> +	}
>> +
>> +	ret = __create_hyp_private_mapping(phys_addr, size,
>> +					   &addr, PAGE_HYP_DEVICE);
>>  	if (ret) {
>>  		iounmap(*kaddr);
>>  		*kaddr = NULL;
>> +		*haddr = NULL;
>> +		return ret;
>> +	}
>> +
>> +	*haddr = (void __iomem *)addr;
>> +	return 0;
>> +}
>> +
>> +/**
>> + * create_hyp_exec_mappings - Map an executable range into into HYP
> 
> s/into into/into/
> 
>> + * @phys_addr:	The physical start address which gets mapped
>> + * @size:	Size of the region being mapped
>> + * @haddr:	HYP VA for this mapping
>> + */
>> +int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
>> +			     void **haddr)
>> +{
>> +	unsigned long addr;
>> +	int ret;
>> +
>> +	BUG_ON(is_kernel_in_hyp_mode());
> 
> Why BUG_ON instead of just giving the caller the same address, similar
> to what create_hyp_io_mappings() does?

Because we just don't have that information on VHE:

- create_hyp_io_mapping does an ioremap, and return the same address for
both
- create_hyp_exec_mapping doesn't have a mapping at all to return. It
just has a PA. We could convert it to a VA, but then which one? Linear
mapping, or kernel mapping?

The real problem is that this makes no sense on VHE, because we don't
have an alternative address space similar to !VHE. And if you call this
on VHE, you're doing something very wrong.

In all honesty, the real issue is create_hyp_io_mapping(), because it
gives you a false sense of security and hides the distinction between
VHE and !VHE.

> 
>> +
>> +	ret = __create_hyp_private_mapping(phys_addr, size,
>> +					   &addr, PAGE_HYP_EXEC);
>> +	if (ret) {
>> +		*haddr = NULL;
>>  		return ret;
>>  	}
>>  
>> +	*haddr = (void *)addr;
>>  	return 0;
>>  }
>>  
>> -- 
>> 2.14.2
>>
> 
> Otherwise
> 
> Reviewed-by: Andrew Jones <drjones@redhat.com>

Thanks!

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 23/26] arm/arm64: KVM: Introduce EL2-specific executable mappings
@ 2018-03-15 15:53       ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-15 15:53 UTC (permalink / raw)
  To: linux-arm-kernel

On 15/03/18 15:03, Andrew Jones wrote:
> On Wed, Mar 14, 2018 at 04:50:46PM +0000, Marc Zyngier wrote:
>> Until now, all EL2 executable mappings were derived from their
>> EL1 VA. Since we want to decouple the vectors mapping from
>> the rest of the hypervisor, we need to be able to map some
>> text somewhere else.
>>
>> The "idmap" region (for lack of a better name) is ideally suited
>> for this, as we have a huge range that hardly has anything in it.
>>
>> Let's extend the IO allocator to also deal with executable mappings,
>> thus providing the required feature.
>>
>> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
>> ---
>>  arch/arm/include/asm/kvm_mmu.h   |  2 +
>>  arch/arm64/include/asm/kvm_mmu.h |  2 +
>>  virt/kvm/arm/mmu.c               | 80 +++++++++++++++++++++++++++++-----------
>>  3 files changed, 63 insertions(+), 21 deletions(-)
>>
>> diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
>> index 26eb6b1cec9b..1b7f592eae57 100644
>> --- a/arch/arm/include/asm/kvm_mmu.h
>> +++ b/arch/arm/include/asm/kvm_mmu.h
>> @@ -56,6 +56,8 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot);
>>  int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
>>  			   void __iomem **kaddr,
>>  			   void __iomem **haddr);
>> +int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
>> +			     void **haddr);
>>  void free_hyp_pgds(void);
>>  
>>  void stage2_unmap_vm(struct kvm *kvm);
>> diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
>> index c2beb2d25170..97af11065bbd 100644
>> --- a/arch/arm64/include/asm/kvm_mmu.h
>> +++ b/arch/arm64/include/asm/kvm_mmu.h
>> @@ -151,6 +151,8 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot);
>>  int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
>>  			   void __iomem **kaddr,
>>  			   void __iomem **haddr);
>> +int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
>> +			     void **haddr);
>>  void free_hyp_pgds(void);
>>  
>>  void stage2_unmap_vm(struct kvm *kvm);
>> diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
>> index b4d1948c8dd6..554ad5493e7d 100644
>> --- a/virt/kvm/arm/mmu.c
>> +++ b/virt/kvm/arm/mmu.c
>> @@ -737,30 +737,13 @@ int create_hyp_mappings(void *from, void *to, pgprot_t prot)
>>  	return 0;
>>  }
>>  
>> -/**
>> - * create_hyp_io_mappings - Map IO into both kernel and HYP
>> - * @phys_addr:	The physical start address which gets mapped
>> - * @size:	Size of the region being mapped
>> - * @kaddr:	Kernel VA for this mapping
>> - * @haddr:	HYP VA for this mapping
>> - */
>> -int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
>> -			   void __iomem **kaddr,
>> -			   void __iomem **haddr)
>> +static int __create_hyp_private_mapping(phys_addr_t phys_addr, size_t size,
>> +					unsigned long *haddr, pgprot_t prot)
>>  {
>>  	pgd_t *pgd = hyp_pgd;
>>  	unsigned long base;
>>  	int ret;
>>  
>> -	*kaddr = ioremap(phys_addr, size);
>> -	if (!*kaddr)
>> -		return -ENOMEM;
>> -
>> -	if (is_kernel_in_hyp_mode()) {
>> -		*haddr = *kaddr;
>> -		return 0;
>> -	}
>> -
>>  	mutex_lock(&io_map_lock);
>>  
>>  	/*
>> @@ -789,22 +772,77 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
>>  
>>  	ret = __create_hyp_mappings(pgd, __kvm_idmap_ptrs_per_pgd(),
>>  				    base, base + size,
>> -				    __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
>> +				    __phys_to_pfn(phys_addr), prot);
>>  	if (ret)
>>  		goto out;
>>  
>> -	*haddr = (void __iomem *)base + offset_in_page(phys_addr);
>> +	*haddr = base + offset_in_page(phys_addr);
>>  	io_map_base = base;
>>  
>>  out:
>>  	mutex_unlock(&io_map_lock);
>>  
>> +	return ret;
>> +}
>> +
>> +/**
>> + * create_hyp_io_mappings - Map IO into both kernel and HYP
>> + * @phys_addr:	The physical start address which gets mapped
>> + * @size:	Size of the region being mapped
>> + * @kaddr:	Kernel VA for this mapping
>> + * @haddr:	HYP VA for this mapping
>> + */
>> +int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
>> +			   void __iomem **kaddr,
>> +			   void __iomem **haddr)
>> +{
>> +	unsigned long addr;
>> +	int ret;
>> +
>> +	*kaddr = ioremap(phys_addr, size);
>> +	if (!*kaddr)
>> +		return -ENOMEM;
>> +
>> +	if (is_kernel_in_hyp_mode()) {
>> +		*haddr = *kaddr;
>> +		return 0;
>> +	}
>> +
>> +	ret = __create_hyp_private_mapping(phys_addr, size,
>> +					   &addr, PAGE_HYP_DEVICE);
>>  	if (ret) {
>>  		iounmap(*kaddr);
>>  		*kaddr = NULL;
>> +		*haddr = NULL;
>> +		return ret;
>> +	}
>> +
>> +	*haddr = (void __iomem *)addr;
>> +	return 0;
>> +}
>> +
>> +/**
>> + * create_hyp_exec_mappings - Map an executable range into into HYP
> 
> s/into into/into/
> 
>> + * @phys_addr:	The physical start address which gets mapped
>> + * @size:	Size of the region being mapped
>> + * @haddr:	HYP VA for this mapping
>> + */
>> +int create_hyp_exec_mappings(phys_addr_t phys_addr, size_t size,
>> +			     void **haddr)
>> +{
>> +	unsigned long addr;
>> +	int ret;
>> +
>> +	BUG_ON(is_kernel_in_hyp_mode());
> 
> Why BUG_ON instead of just giving the caller the same address, similar
> to what create_hyp_io_mappings() does?

Because we just don't have that information on VHE:

- create_hyp_io_mapping does an ioremap, and return the same address for
both
- create_hyp_exec_mapping doesn't have a mapping at all to return. It
just has a PA. We could convert it to a VA, but then which one? Linear
mapping, or kernel mapping?

The real problem is that this makes no sense on VHE, because we don't
have an alternative address space similar to !VHE. And if you call this
on VHE, you're doing something very wrong.

In all honesty, the real issue is create_hyp_io_mapping(), because it
gives you a false sense of security and hides the distinction between
VHE and !VHE.

> 
>> +
>> +	ret = __create_hyp_private_mapping(phys_addr, size,
>> +					   &addr, PAGE_HYP_EXEC);
>> +	if (ret) {
>> +		*haddr = NULL;
>>  		return ret;
>>  	}
>>  
>> +	*haddr = (void *)addr;
>>  	return 0;
>>  }
>>  
>> -- 
>> 2.14.2
>>
> 
> Otherwise
> 
> Reviewed-by: Andrew Jones <drjones@redhat.com>

Thanks!

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 25/26] arm64: KVM: Allow mapping of vectors outside of the RAM region
  2018-03-14 16:50   ` Marc Zyngier
@ 2018-03-15 15:54     ` Andrew Jones
  -1 siblings, 0 replies; 124+ messages in thread
From: Andrew Jones @ 2018-03-15 15:54 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: kvm, Catalin Marinas, Will Deacon, Kristina Martsenko, kvmarm,
	linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:48PM +0000, Marc Zyngier wrote:
> We're now ready to map our vectors in weird and wonderful locations.
> On enabling ARM64_HARDEN_EL2_VECTORS, a vector slots gets allocated

s/slots/slot/

> if this hasn't been already done via ARM64_HARDEN_BRANCH_PREDICTOR
> and gets mapped outside of the normal RAM region, next to the
> idmap.
> 
> That way, being able to obtain VBAR_EL2 doesn't reveal the mapping
> of the rest of the hypervisor code.
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> ---
>  Documentation/arm64/memory.txt   |  3 +-
>  arch/arm64/Kconfig               | 16 +++++++++
>  arch/arm64/include/asm/kvm_mmu.h | 78 +++++++++++++++++++++++++++++++++++-----
>  arch/arm64/include/asm/mmu.h     |  5 ++-
>  arch/arm64/kvm/Kconfig           |  2 +-
>  arch/arm64/kvm/va_layout.c       |  3 ++
>  6 files changed, 95 insertions(+), 12 deletions(-)
> 
> diff --git a/Documentation/arm64/memory.txt b/Documentation/arm64/memory.txt
> index c58cc5dbe667..c5dab30d3389 100644
> --- a/Documentation/arm64/memory.txt
> +++ b/Documentation/arm64/memory.txt
> @@ -90,7 +90,8 @@ When using KVM without the Virtualization Host Extensions, the
>  hypervisor maps kernel pages in EL2 at a fixed (and potentially
>  random) offset from the linear mapping. See the kern_hyp_va macro and
>  kvm_update_va_mask function for more details. MMIO devices such as
> -GICv2 gets mapped next to the HYP idmap page.
> +GICv2 gets mapped next to the HYP idmap page, as do vectors when
> +ARM64_HARDEN_EL2_VECTORS is selected for particular CPUs.
>  
>  When using KVM with the Virtualization Host Extensions, no additional
>  mappings are created, since the host kernel runs directly in EL2.
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index 7381eeb7ef8e..e6be4393aaad 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -904,6 +904,22 @@ config HARDEN_BRANCH_PREDICTOR
>  
>  	  If unsure, say Y.
>  
> +config HARDEN_EL2_VECTORS
> +	bool "Harden EL2 vector mapping against system register leak" if EXPERT
> +	default y
> +	help
> +	  Speculation attacks against some high-performance processors can
> +	  be used to leak privileged information such as the vector base
> +	  register, resulting in a potential defeat of the EL2 layout
> +	  randomization.
> +
> +	  This config option will map the vectors to a fixed location,
> +	  independent of the EL2 code mapping, so that revealing VBAR_EL2

s/VBAR_EL2/the vector base register/ ?

> +	  to an attacker does no give away any extra information. This

s/no/not/

> +	  only gets enabled on affected CPUs.
> +
> +	  If unsure, say Y.
> +
>  menuconfig ARMV8_DEPRECATED
>  	bool "Emulate deprecated/obsolete ARMv8 instructions"
>  	depends on COMPAT
> diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
> index 97af11065bbd..f936d0928661 100644
> --- a/arch/arm64/include/asm/kvm_mmu.h
> +++ b/arch/arm64/include/asm/kvm_mmu.h
> @@ -360,31 +360,91 @@ static inline unsigned int kvm_get_vmid_bits(void)
>  	return (cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR1_VMIDBITS_SHIFT) == 2) ? 16 : 8;
>  }
>  
> -#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
> +#ifdef CONFIG_KVM_INDIRECT_VECTORS
> +/*
> + * EL2 vectors can be mapped and rerouted in a number of ways,
> + * depending on the kernel configuration and CPU present:
> + *
> + * - If the CPU has the ARM64_HARDEN_BRANCH_PREDICTOR cap, the
> + *   hardening sequence is placed in one of the vector slots, which is
> + *   executed before jumping to the real vectors.
> + *
> + * - If the CPU has both the ARM64_HARDEN_EL2_VECTORS and BP
                                                        ^cap

Maybe s/BP hardening/the ARM64_HARDEN_BRANCH_PREDICTOR cap/ ?

> + *   hardening, the slot containing the hardening sequence is mapped
> + *   next to the idmap page, and executed before jumping to the real
> + *   vectors.
> + *
> + * - If the CPU only has ARM64_HARDEN_EL2_VECTORS, then an empty slot

...has the ARM64_HARDEN_EL2_VECTORS cap

Or maybe change the above ones from 'has the ... cap' to just 'has ...',
like this one.

> + *   is selected, mapped next to the idmap page, and executed before
> + *   jumping to the real vectors.
> + *
> + * Note that ARM64_HARDEN_EL2_VECTORS is somewhat incompatible with
> + * VHE, as we don't have hypervisor-specific mappings. If the system
> + * is VHE and yet selects this capability, it will be ignored.
> + */
>  #include <asm/mmu.h>
>  
> +extern void *__kvm_bp_vect_base;
> +extern int __kvm_harden_el2_vector_slot;
> +
>  static inline void *kvm_get_hyp_vector(void)
>  {
>  	struct bp_hardening_data *data = arm64_get_bp_hardening_data();
> -	void *vect = kvm_ksym_ref(__kvm_hyp_vector);
> +	void *vect = kern_hyp_va(kvm_ksym_ref(__kvm_hyp_vector));
> +	int slot = -1;
>  
> -	if (data->fn) {
> -		vect = __bp_harden_hyp_vecs_start +
> -		       data->hyp_vectors_slot * SZ_2K;
> +	if (cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR) && data->fn) {

Now that I'm trying to consider heterogeneous systems, I'm wondering why
harden BP isn't checked with this_cpu_has_cap() like harden EL2 vectors.
I'm also wondering if it's even possible for CPUs to come up without all
of them having harden EL2 vectors if the boot CPU has it. Won't
verify_local_cpu_errata_workarounds() require it? I'm probably just
getting lost in all the capability stuff...

> +		vect = __bp_harden_hyp_vecs_start;
> +		slot = data->hyp_vectors_slot;
> +	}
>  
> -		if (!has_vhe())
> -			vect = lm_alias(vect);
> +	if (this_cpu_has_cap(ARM64_HARDEN_EL2_VECTORS) && !has_vhe()) {
> +		vect = __kvm_bp_vect_base;
> +		if (slot == -1)
> +			slot = __kvm_harden_el2_vector_slot;
>  	}
>  
> -	vect = kern_hyp_va(vect);
> +	if (slot != -1)
> +		vect += slot * SZ_2K;
> +
>  	return vect;
>  }
>  
> +/*  This is only called on a !VHE system */
>  static inline int kvm_map_vectors(void)
>  {
> +	/*
> +	 * HBP  = ARM64_HARDEN_BRANCH_PREDICTOR
> +	 * HLE2 = ARM64_HARDEN_EL2_VECTORS

s/HLE2/HEL2/

> +	 *
> +	 * !HBP + !HEL2 -> use direct vectors
> +	 *  HBP + !HEL2 -> use hardenned vectors in place

hardened 

> +	 * !HBP +  HEL2 -> allocate one vector slot and use exec mapping
> +	 *  HBP +  HEL2 -> use hardenned vertors and use exec mapping

hardened vectors

> +	 */
> +	if (cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR)) {
> +		__kvm_bp_vect_base = kvm_ksym_ref(__bp_harden_hyp_vecs_start);
> +		__kvm_bp_vect_base = kern_hyp_va(__kvm_bp_vect_base);
> +	}
> +
> +	if (cpus_have_const_cap(ARM64_HARDEN_EL2_VECTORS)) {

What happened to this_cpu_has_cap()? Did we switch because this is only
running on the boot CPU?

> +		phys_addr_t vect_pa = __pa_symbol(__bp_harden_hyp_vecs_start);
> +		unsigned long size = (__bp_harden_hyp_vecs_end -
> +				      __bp_harden_hyp_vecs_start);
> +
> +		/*
> +		 * Always allocate a spare vector slot, as we don't
> +		 * know yet which CPUs have a BP hardening slot that
> +		 * we can reuse.
> +		 */
> +		__kvm_harden_el2_vector_slot = atomic_inc_return(&arm64_el2_vector_last_slot);
> +		BUG_ON(__kvm_harden_el2_vector_slot >= BP_HARDEN_EL2_SLOTS);
> +		return create_hyp_exec_mappings(vect_pa, size,
> +						&__kvm_bp_vect_base);
> +	}
> +
>  	return 0;
>  }
> -
>  #else
>  static inline void *kvm_get_hyp_vector(void)
>  {
> diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h
> index 3baf010fe883..0b0cc69031c1 100644
> --- a/arch/arm64/include/asm/mmu.h
> +++ b/arch/arm64/include/asm/mmu.h
> @@ -51,10 +51,12 @@ struct bp_hardening_data {
>  	bp_hardening_cb_t	fn;
>  };
>  
> -#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
> +#if (defined(CONFIG_HARDEN_BRANCH_PREDICTOR) ||	\
> +     defined(CONFIG_HARDEN_EL2_VECTORS))
>  extern char __bp_harden_hyp_vecs_start[], __bp_harden_hyp_vecs_end[];
>  extern atomic_t arm64_el2_vector_last_slot;
>  
> +#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
>  DECLARE_PER_CPU_READ_MOSTLY(struct bp_hardening_data, bp_hardening_data);
>  
>  static inline struct bp_hardening_data *arm64_get_bp_hardening_data(void)
> @@ -81,6 +83,7 @@ static inline struct bp_hardening_data *arm64_get_bp_hardening_data(void)
>  
>  static inline void arm64_apply_bp_hardening(void)	{ }
>  #endif	/* CONFIG_HARDEN_BRANCH_PREDICTOR */
> +#endif  /* CONFIG_HARDEN_BRANCH_PREDICTOR || CONFIG_HARDEN_EL2_VECTORS */
>  
>  extern void paging_init(void);
>  extern void bootmem_init(void);
> diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
> index bd8cc03d7522..a2e3a5af1113 100644
> --- a/arch/arm64/kvm/Kconfig
> +++ b/arch/arm64/kvm/Kconfig
> @@ -58,7 +58,7 @@ config KVM_ARM_PMU
>  	  virtual machines.
>  
>  config KVM_INDIRECT_VECTORS
> -       def_bool KVM && HARDEN_BRANCH_PREDICTOR
> +       def_bool KVM && (HARDEN_BRANCH_PREDICTOR || HARDEN_EL2_VECTORS)
>  
>  source drivers/vhost/Kconfig
>  
> diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c
> index 9d35c17016ed..fd2d658a11b5 100644
> --- a/arch/arm64/kvm/va_layout.c
> +++ b/arch/arm64/kvm/va_layout.c
> @@ -151,6 +151,9 @@ void __init kvm_update_va_mask(struct alt_instr *alt,
>  	}
>  }
>  
> +void *__kvm_bp_vect_base;
> +int __kvm_harden_el2_vector_slot;
> +
>  void kvm_patch_vector_branch(struct alt_instr *alt,
>  			     __le32 *origptr, __le32 *updptr, int nr_inst)
>  {
> -- 
> 2.14.2
>

Thanks,
drew 

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 25/26] arm64: KVM: Allow mapping of vectors outside of the RAM region
@ 2018-03-15 15:54     ` Andrew Jones
  0 siblings, 0 replies; 124+ messages in thread
From: Andrew Jones @ 2018-03-15 15:54 UTC (permalink / raw)
  To: linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:48PM +0000, Marc Zyngier wrote:
> We're now ready to map our vectors in weird and wonderful locations.
> On enabling ARM64_HARDEN_EL2_VECTORS, a vector slots gets allocated

s/slots/slot/

> if this hasn't been already done via ARM64_HARDEN_BRANCH_PREDICTOR
> and gets mapped outside of the normal RAM region, next to the
> idmap.
> 
> That way, being able to obtain VBAR_EL2 doesn't reveal the mapping
> of the rest of the hypervisor code.
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> ---
>  Documentation/arm64/memory.txt   |  3 +-
>  arch/arm64/Kconfig               | 16 +++++++++
>  arch/arm64/include/asm/kvm_mmu.h | 78 +++++++++++++++++++++++++++++++++++-----
>  arch/arm64/include/asm/mmu.h     |  5 ++-
>  arch/arm64/kvm/Kconfig           |  2 +-
>  arch/arm64/kvm/va_layout.c       |  3 ++
>  6 files changed, 95 insertions(+), 12 deletions(-)
> 
> diff --git a/Documentation/arm64/memory.txt b/Documentation/arm64/memory.txt
> index c58cc5dbe667..c5dab30d3389 100644
> --- a/Documentation/arm64/memory.txt
> +++ b/Documentation/arm64/memory.txt
> @@ -90,7 +90,8 @@ When using KVM without the Virtualization Host Extensions, the
>  hypervisor maps kernel pages in EL2 at a fixed (and potentially
>  random) offset from the linear mapping. See the kern_hyp_va macro and
>  kvm_update_va_mask function for more details. MMIO devices such as
> -GICv2 gets mapped next to the HYP idmap page.
> +GICv2 gets mapped next to the HYP idmap page, as do vectors when
> +ARM64_HARDEN_EL2_VECTORS is selected for particular CPUs.
>  
>  When using KVM with the Virtualization Host Extensions, no additional
>  mappings are created, since the host kernel runs directly in EL2.
> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
> index 7381eeb7ef8e..e6be4393aaad 100644
> --- a/arch/arm64/Kconfig
> +++ b/arch/arm64/Kconfig
> @@ -904,6 +904,22 @@ config HARDEN_BRANCH_PREDICTOR
>  
>  	  If unsure, say Y.
>  
> +config HARDEN_EL2_VECTORS
> +	bool "Harden EL2 vector mapping against system register leak" if EXPERT
> +	default y
> +	help
> +	  Speculation attacks against some high-performance processors can
> +	  be used to leak privileged information such as the vector base
> +	  register, resulting in a potential defeat of the EL2 layout
> +	  randomization.
> +
> +	  This config option will map the vectors to a fixed location,
> +	  independent of the EL2 code mapping, so that revealing VBAR_EL2

s/VBAR_EL2/the vector base register/ ?

> +	  to an attacker does no give away any extra information. This

s/no/not/

> +	  only gets enabled on affected CPUs.
> +
> +	  If unsure, say Y.
> +
>  menuconfig ARMV8_DEPRECATED
>  	bool "Emulate deprecated/obsolete ARMv8 instructions"
>  	depends on COMPAT
> diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
> index 97af11065bbd..f936d0928661 100644
> --- a/arch/arm64/include/asm/kvm_mmu.h
> +++ b/arch/arm64/include/asm/kvm_mmu.h
> @@ -360,31 +360,91 @@ static inline unsigned int kvm_get_vmid_bits(void)
>  	return (cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR1_VMIDBITS_SHIFT) == 2) ? 16 : 8;
>  }
>  
> -#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
> +#ifdef CONFIG_KVM_INDIRECT_VECTORS
> +/*
> + * EL2 vectors can be mapped and rerouted in a number of ways,
> + * depending on the kernel configuration and CPU present:
> + *
> + * - If the CPU has the ARM64_HARDEN_BRANCH_PREDICTOR cap, the
> + *   hardening sequence is placed in one of the vector slots, which is
> + *   executed before jumping to the real vectors.
> + *
> + * - If the CPU has both the ARM64_HARDEN_EL2_VECTORS and BP
                                                        ^cap

Maybe s/BP hardening/the ARM64_HARDEN_BRANCH_PREDICTOR cap/ ?

> + *   hardening, the slot containing the hardening sequence is mapped
> + *   next to the idmap page, and executed before jumping to the real
> + *   vectors.
> + *
> + * - If the CPU only has ARM64_HARDEN_EL2_VECTORS, then an empty slot

...has the ARM64_HARDEN_EL2_VECTORS cap

Or maybe change the above ones from 'has the ... cap' to just 'has ...',
like this one.

> + *   is selected, mapped next to the idmap page, and executed before
> + *   jumping to the real vectors.
> + *
> + * Note that ARM64_HARDEN_EL2_VECTORS is somewhat incompatible with
> + * VHE, as we don't have hypervisor-specific mappings. If the system
> + * is VHE and yet selects this capability, it will be ignored.
> + */
>  #include <asm/mmu.h>
>  
> +extern void *__kvm_bp_vect_base;
> +extern int __kvm_harden_el2_vector_slot;
> +
>  static inline void *kvm_get_hyp_vector(void)
>  {
>  	struct bp_hardening_data *data = arm64_get_bp_hardening_data();
> -	void *vect = kvm_ksym_ref(__kvm_hyp_vector);
> +	void *vect = kern_hyp_va(kvm_ksym_ref(__kvm_hyp_vector));
> +	int slot = -1;
>  
> -	if (data->fn) {
> -		vect = __bp_harden_hyp_vecs_start +
> -		       data->hyp_vectors_slot * SZ_2K;
> +	if (cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR) && data->fn) {

Now that I'm trying to consider heterogeneous systems, I'm wondering why
harden BP isn't checked with this_cpu_has_cap() like harden EL2 vectors.
I'm also wondering if it's even possible for CPUs to come up without all
of them having harden EL2 vectors if the boot CPU has it. Won't
verify_local_cpu_errata_workarounds() require it? I'm probably just
getting lost in all the capability stuff...

> +		vect = __bp_harden_hyp_vecs_start;
> +		slot = data->hyp_vectors_slot;
> +	}
>  
> -		if (!has_vhe())
> -			vect = lm_alias(vect);
> +	if (this_cpu_has_cap(ARM64_HARDEN_EL2_VECTORS) && !has_vhe()) {
> +		vect = __kvm_bp_vect_base;
> +		if (slot == -1)
> +			slot = __kvm_harden_el2_vector_slot;
>  	}
>  
> -	vect = kern_hyp_va(vect);
> +	if (slot != -1)
> +		vect += slot * SZ_2K;
> +
>  	return vect;
>  }
>  
> +/*  This is only called on a !VHE system */
>  static inline int kvm_map_vectors(void)
>  {
> +	/*
> +	 * HBP  = ARM64_HARDEN_BRANCH_PREDICTOR
> +	 * HLE2 = ARM64_HARDEN_EL2_VECTORS

s/HLE2/HEL2/

> +	 *
> +	 * !HBP + !HEL2 -> use direct vectors
> +	 *  HBP + !HEL2 -> use hardenned vectors in place

hardened 

> +	 * !HBP +  HEL2 -> allocate one vector slot and use exec mapping
> +	 *  HBP +  HEL2 -> use hardenned vertors and use exec mapping

hardened vectors

> +	 */
> +	if (cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR)) {
> +		__kvm_bp_vect_base = kvm_ksym_ref(__bp_harden_hyp_vecs_start);
> +		__kvm_bp_vect_base = kern_hyp_va(__kvm_bp_vect_base);
> +	}
> +
> +	if (cpus_have_const_cap(ARM64_HARDEN_EL2_VECTORS)) {

What happened to this_cpu_has_cap()? Did we switch because this is only
running on the boot CPU?

> +		phys_addr_t vect_pa = __pa_symbol(__bp_harden_hyp_vecs_start);
> +		unsigned long size = (__bp_harden_hyp_vecs_end -
> +				      __bp_harden_hyp_vecs_start);
> +
> +		/*
> +		 * Always allocate a spare vector slot, as we don't
> +		 * know yet which CPUs have a BP hardening slot that
> +		 * we can reuse.
> +		 */
> +		__kvm_harden_el2_vector_slot = atomic_inc_return(&arm64_el2_vector_last_slot);
> +		BUG_ON(__kvm_harden_el2_vector_slot >= BP_HARDEN_EL2_SLOTS);
> +		return create_hyp_exec_mappings(vect_pa, size,
> +						&__kvm_bp_vect_base);
> +	}
> +
>  	return 0;
>  }
> -
>  #else
>  static inline void *kvm_get_hyp_vector(void)
>  {
> diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h
> index 3baf010fe883..0b0cc69031c1 100644
> --- a/arch/arm64/include/asm/mmu.h
> +++ b/arch/arm64/include/asm/mmu.h
> @@ -51,10 +51,12 @@ struct bp_hardening_data {
>  	bp_hardening_cb_t	fn;
>  };
>  
> -#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
> +#if (defined(CONFIG_HARDEN_BRANCH_PREDICTOR) ||	\
> +     defined(CONFIG_HARDEN_EL2_VECTORS))
>  extern char __bp_harden_hyp_vecs_start[], __bp_harden_hyp_vecs_end[];
>  extern atomic_t arm64_el2_vector_last_slot;
>  
> +#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
>  DECLARE_PER_CPU_READ_MOSTLY(struct bp_hardening_data, bp_hardening_data);
>  
>  static inline struct bp_hardening_data *arm64_get_bp_hardening_data(void)
> @@ -81,6 +83,7 @@ static inline struct bp_hardening_data *arm64_get_bp_hardening_data(void)
>  
>  static inline void arm64_apply_bp_hardening(void)	{ }
>  #endif	/* CONFIG_HARDEN_BRANCH_PREDICTOR */
> +#endif  /* CONFIG_HARDEN_BRANCH_PREDICTOR || CONFIG_HARDEN_EL2_VECTORS */
>  
>  extern void paging_init(void);
>  extern void bootmem_init(void);
> diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
> index bd8cc03d7522..a2e3a5af1113 100644
> --- a/arch/arm64/kvm/Kconfig
> +++ b/arch/arm64/kvm/Kconfig
> @@ -58,7 +58,7 @@ config KVM_ARM_PMU
>  	  virtual machines.
>  
>  config KVM_INDIRECT_VECTORS
> -       def_bool KVM && HARDEN_BRANCH_PREDICTOR
> +       def_bool KVM && (HARDEN_BRANCH_PREDICTOR || HARDEN_EL2_VECTORS)
>  
>  source drivers/vhost/Kconfig
>  
> diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c
> index 9d35c17016ed..fd2d658a11b5 100644
> --- a/arch/arm64/kvm/va_layout.c
> +++ b/arch/arm64/kvm/va_layout.c
> @@ -151,6 +151,9 @@ void __init kvm_update_va_mask(struct alt_instr *alt,
>  	}
>  }
>  
> +void *__kvm_bp_vect_base;
> +int __kvm_harden_el2_vector_slot;
> +
>  void kvm_patch_vector_branch(struct alt_instr *alt,
>  			     __le32 *origptr, __le32 *updptr, int nr_inst)
>  {
> -- 
> 2.14.2
>

Thanks,
drew 

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 00/26] KVM/arm64: Randomise EL2 mappings (variant 3a mitigation)
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-15 15:57   ` Andrew Jones
  -1 siblings, 0 replies; 124+ messages in thread
From: Andrew Jones @ 2018-03-15 15:57 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: kvm, Catalin Marinas, Will Deacon, Kristina Martsenko, kvmarm,
	linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:23PM +0000, Marc Zyngier wrote:
> Whilst KVM benefits from the kernel randomisation via KASLR, there is
> no additional randomisation when the kernel is running at EL1, as we
> directly use a fixed offset from the linear mapping. This is not
> necessarily a problem, but we could do a bit better by independently
> randomizing the HYP placement.
> 
> This series proposes to randomise the offset by inserting a few random
> bits between the MSB of the RAM linear mapping and the top of the HYP
> VA (VA_BITS - 2). That's not a lot of random bits (on my Mustang, I
> get 13 bits), but that's better than nothing.
> 
> In order to achieve this, we need to be able to patch dynamic values
> in the kernel text. This results in a bunch of changes to the
> alternative framework, the insn library, and a few more hacks in KVM
> itself (we get a new way to map the GIC at EL2).
> 
> Another (and more recent) goal of this series is to work around what
> has been described as "variant 3a", which covers speculative reads of
> privileged system registers. Randomizing the location of the
> hypervisor would be pointless if one could simply obtain VBAR_EL2. In
> order to work around this, we place the vectors at a fairly static
> location (next to the idmap), independently of the hypervisor's own
> mappings. This ensures that we can leak VBAR_EL2 without disclosing
> much about HYP itself (and is similar to what the rest of the kernel
> does with KPTI). This is only enabled at runtime for Cortex-A57 and
> Cortex-A72.
> 
> This has been tested on the FVP model, Seattle (both 39 and 48bit VA),
> Mustang and Thunder-X. I've also done a sanity check on 32bit (which
> is only impacted by the HYP IO VA stuff).
>

I've smoke tested this series on a seattle with several busy VMs running
simultaneously. My host kernel configures 64K pages. I didn't see any
problems.

Thanks,
drew

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 00/26] KVM/arm64: Randomise EL2 mappings (variant 3a mitigation)
@ 2018-03-15 15:57   ` Andrew Jones
  0 siblings, 0 replies; 124+ messages in thread
From: Andrew Jones @ 2018-03-15 15:57 UTC (permalink / raw)
  To: linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:23PM +0000, Marc Zyngier wrote:
> Whilst KVM benefits from the kernel randomisation via KASLR, there is
> no additional randomisation when the kernel is running at EL1, as we
> directly use a fixed offset from the linear mapping. This is not
> necessarily a problem, but we could do a bit better by independently
> randomizing the HYP placement.
> 
> This series proposes to randomise the offset by inserting a few random
> bits between the MSB of the RAM linear mapping and the top of the HYP
> VA (VA_BITS - 2). That's not a lot of random bits (on my Mustang, I
> get 13 bits), but that's better than nothing.
> 
> In order to achieve this, we need to be able to patch dynamic values
> in the kernel text. This results in a bunch of changes to the
> alternative framework, the insn library, and a few more hacks in KVM
> itself (we get a new way to map the GIC at EL2).
> 
> Another (and more recent) goal of this series is to work around what
> has been described as "variant 3a", which covers speculative reads of
> privileged system registers. Randomizing the location of the
> hypervisor would be pointless if one could simply obtain VBAR_EL2. In
> order to work around this, we place the vectors at a fairly static
> location (next to the idmap), independently of the hypervisor's own
> mappings. This ensures that we can leak VBAR_EL2 without disclosing
> much about HYP itself (and is similar to what the rest of the kernel
> does with KPTI). This is only enabled at runtime for Cortex-A57 and
> Cortex-A72.
> 
> This has been tested on the FVP model, Seattle (both 39 and 48bit VA),
> Mustang and Thunder-X. I've also done a sanity check on 32bit (which
> is only impacted by the HYP IO VA stuff).
>

I've smoke tested this series on a seattle with several busy VMs running
simultaneously. My host kernel configures 64K pages. I didn't see any
problems.

Thanks,
drew

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 25/26] arm64: KVM: Allow mapping of vectors outside of the RAM region
  2018-03-15 15:54     ` Andrew Jones
@ 2018-03-15 16:17       ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-15 16:17 UTC (permalink / raw)
  To: Andrew Jones
  Cc: Mark Rutland, Peter Maydell, Christoffer Dall, kvm, Steve Capper,
	Catalin Marinas, Will Deacon, Kristina Martsenko, James Morse,
	kvmarm, linux-arm-kernel

On 15/03/18 15:54, Andrew Jones wrote:
> On Wed, Mar 14, 2018 at 04:50:48PM +0000, Marc Zyngier wrote:
>> We're now ready to map our vectors in weird and wonderful locations.
>> On enabling ARM64_HARDEN_EL2_VECTORS, a vector slots gets allocated
> 
> s/slots/slot/
> 
>> if this hasn't been already done via ARM64_HARDEN_BRANCH_PREDICTOR
>> and gets mapped outside of the normal RAM region, next to the
>> idmap.
>>
>> That way, being able to obtain VBAR_EL2 doesn't reveal the mapping
>> of the rest of the hypervisor code.
>>
>> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
>> ---
>>  Documentation/arm64/memory.txt   |  3 +-
>>  arch/arm64/Kconfig               | 16 +++++++++
>>  arch/arm64/include/asm/kvm_mmu.h | 78 +++++++++++++++++++++++++++++++++++-----
>>  arch/arm64/include/asm/mmu.h     |  5 ++-
>>  arch/arm64/kvm/Kconfig           |  2 +-
>>  arch/arm64/kvm/va_layout.c       |  3 ++
>>  6 files changed, 95 insertions(+), 12 deletions(-)
>>
>> diff --git a/Documentation/arm64/memory.txt b/Documentation/arm64/memory.txt
>> index c58cc5dbe667..c5dab30d3389 100644
>> --- a/Documentation/arm64/memory.txt
>> +++ b/Documentation/arm64/memory.txt
>> @@ -90,7 +90,8 @@ When using KVM without the Virtualization Host Extensions, the
>>  hypervisor maps kernel pages in EL2 at a fixed (and potentially
>>  random) offset from the linear mapping. See the kern_hyp_va macro and
>>  kvm_update_va_mask function for more details. MMIO devices such as
>> -GICv2 gets mapped next to the HYP idmap page.
>> +GICv2 gets mapped next to the HYP idmap page, as do vectors when
>> +ARM64_HARDEN_EL2_VECTORS is selected for particular CPUs.
>>  
>>  When using KVM with the Virtualization Host Extensions, no additional
>>  mappings are created, since the host kernel runs directly in EL2.
>> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
>> index 7381eeb7ef8e..e6be4393aaad 100644
>> --- a/arch/arm64/Kconfig
>> +++ b/arch/arm64/Kconfig
>> @@ -904,6 +904,22 @@ config HARDEN_BRANCH_PREDICTOR
>>  
>>  	  If unsure, say Y.
>>  
>> +config HARDEN_EL2_VECTORS
>> +	bool "Harden EL2 vector mapping against system register leak" if EXPERT
>> +	default y
>> +	help
>> +	  Speculation attacks against some high-performance processors can
>> +	  be used to leak privileged information such as the vector base
>> +	  register, resulting in a potential defeat of the EL2 layout
>> +	  randomization.
>> +
>> +	  This config option will map the vectors to a fixed location,
>> +	  independent of the EL2 code mapping, so that revealing VBAR_EL2
> 
> s/VBAR_EL2/the vector base register/ ?

Which vector base register? VBAR_EL1 is under control of the guest, so
it can readily find that one. We're really trying to prevent that
particular register from being used to disclose anything useful.

> 
>> +	  to an attacker does no give away any extra information. This
> 
> s/no/not/
> 
>> +	  only gets enabled on affected CPUs.
>> +
>> +	  If unsure, say Y.
>> +
>>  menuconfig ARMV8_DEPRECATED
>>  	bool "Emulate deprecated/obsolete ARMv8 instructions"
>>  	depends on COMPAT
>> diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
>> index 97af11065bbd..f936d0928661 100644
>> --- a/arch/arm64/include/asm/kvm_mmu.h
>> +++ b/arch/arm64/include/asm/kvm_mmu.h
>> @@ -360,31 +360,91 @@ static inline unsigned int kvm_get_vmid_bits(void)
>>  	return (cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR1_VMIDBITS_SHIFT) == 2) ? 16 : 8;
>>  }
>>  
>> -#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
>> +#ifdef CONFIG_KVM_INDIRECT_VECTORS
>> +/*
>> + * EL2 vectors can be mapped and rerouted in a number of ways,
>> + * depending on the kernel configuration and CPU present:
>> + *
>> + * - If the CPU has the ARM64_HARDEN_BRANCH_PREDICTOR cap, the
>> + *   hardening sequence is placed in one of the vector slots, which is
>> + *   executed before jumping to the real vectors.
>> + *
>> + * - If the CPU has both the ARM64_HARDEN_EL2_VECTORS and BP
>                                                         ^cap
> 
> Maybe s/BP hardening/the ARM64_HARDEN_BRANCH_PREDICTOR cap/ ?
> 
>> + *   hardening, the slot containing the hardening sequence is mapped
>> + *   next to the idmap page, and executed before jumping to the real
>> + *   vectors.
>> + *
>> + * - If the CPU only has ARM64_HARDEN_EL2_VECTORS, then an empty slot
> 
> ...has the ARM64_HARDEN_EL2_VECTORS cap
> 
> Or maybe change the above ones from 'has the ... cap' to just 'has ...',
> like this one.
> 
>> + *   is selected, mapped next to the idmap page, and executed before
>> + *   jumping to the real vectors.
>> + *
>> + * Note that ARM64_HARDEN_EL2_VECTORS is somewhat incompatible with
>> + * VHE, as we don't have hypervisor-specific mappings. If the system
>> + * is VHE and yet selects this capability, it will be ignored.
>> + */
>>  #include <asm/mmu.h>
>>  
>> +extern void *__kvm_bp_vect_base;
>> +extern int __kvm_harden_el2_vector_slot;
>> +
>>  static inline void *kvm_get_hyp_vector(void)
>>  {
>>  	struct bp_hardening_data *data = arm64_get_bp_hardening_data();
>> -	void *vect = kvm_ksym_ref(__kvm_hyp_vector);
>> +	void *vect = kern_hyp_va(kvm_ksym_ref(__kvm_hyp_vector));
>> +	int slot = -1;
>>  
>> -	if (data->fn) {
>> -		vect = __bp_harden_hyp_vecs_start +
>> -		       data->hyp_vectors_slot * SZ_2K;
>> +	if (cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR) && data->fn) {
> 
> Now that I'm trying to consider heterogeneous systems, I'm wondering why
> harden BP isn't checked with this_cpu_has_cap() like harden EL2 vectors.
> I'm also wondering if it's even possible for CPUs to come up without all
> of them having harden EL2 vectors if the boot CPU has it. Won't
> verify_local_cpu_errata_workarounds() require it? I'm probably just
> getting lost in all the capability stuff...

Checking harden BP on this particular CPU will only tell you if this CPU
is affected, but won't give you any additional information (i.e. you'd
still need to check obtain the stuff pointed to by data).

Checking for *all* CPUs in one go has some advantages: it is a static
key, which means that unaffected platforms will fly (this is a hot path
on VHE), and you can check data if you're affected.

> 
>> +		vect = __bp_harden_hyp_vecs_start;
>> +		slot = data->hyp_vectors_slot;
>> +	}
>>  
>> -		if (!has_vhe())
>> -			vect = lm_alias(vect);
>> +	if (this_cpu_has_cap(ARM64_HARDEN_EL2_VECTORS) && !has_vhe()) {
>> +		vect = __kvm_bp_vect_base;
>> +		if (slot == -1)
>> +			slot = __kvm_harden_el2_vector_slot;
>>  	}
>>  
>> -	vect = kern_hyp_va(vect);
>> +	if (slot != -1)
>> +		vect += slot * SZ_2K;
>> +
>>  	return vect;
>>  }
>>  
>> +/*  This is only called on a !VHE system */
>>  static inline int kvm_map_vectors(void)
>>  {
>> +	/*
>> +	 * HBP  = ARM64_HARDEN_BRANCH_PREDICTOR
>> +	 * HLE2 = ARM64_HARDEN_EL2_VECTORS
> 
> s/HLE2/HEL2/
> 
>> +	 *
>> +	 * !HBP + !HEL2 -> use direct vectors
>> +	 *  HBP + !HEL2 -> use hardenned vectors in place
> 
> hardened 
> 
>> +	 * !HBP +  HEL2 -> allocate one vector slot and use exec mapping
>> +	 *  HBP +  HEL2 -> use hardenned vertors and use exec mapping
> 
> hardened vectors
> 
>> +	 */
>> +	if (cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR)) {
>> +		__kvm_bp_vect_base = kvm_ksym_ref(__bp_harden_hyp_vecs_start);
>> +		__kvm_bp_vect_base = kern_hyp_va(__kvm_bp_vect_base);
>> +	}
>> +
>> +	if (cpus_have_const_cap(ARM64_HARDEN_EL2_VECTORS)) {
> 
> What happened to this_cpu_has_cap()? Did we switch because this is only
> running on the boot CPU?

Only one CPU performs all the mappings (see init_hyp_mode()).

> 
>> +		phys_addr_t vect_pa = __pa_symbol(__bp_harden_hyp_vecs_start);
>> +		unsigned long size = (__bp_harden_hyp_vecs_end -
>> +				      __bp_harden_hyp_vecs_start);
>> +
>> +		/*
>> +		 * Always allocate a spare vector slot, as we don't
>> +		 * know yet which CPUs have a BP hardening slot that
>> +		 * we can reuse.
>> +		 */
>> +		__kvm_harden_el2_vector_slot = atomic_inc_return(&arm64_el2_vector_last_slot);
>> +		BUG_ON(__kvm_harden_el2_vector_slot >= BP_HARDEN_EL2_SLOTS);
>> +		return create_hyp_exec_mappings(vect_pa, size,
>> +						&__kvm_bp_vect_base);
>> +	}
>> +
>>  	return 0;
>>  }
>> -
>>  #else
>>  static inline void *kvm_get_hyp_vector(void)
>>  {
>> diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h
>> index 3baf010fe883..0b0cc69031c1 100644
>> --- a/arch/arm64/include/asm/mmu.h
>> +++ b/arch/arm64/include/asm/mmu.h
>> @@ -51,10 +51,12 @@ struct bp_hardening_data {
>>  	bp_hardening_cb_t	fn;
>>  };
>>  
>> -#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
>> +#if (defined(CONFIG_HARDEN_BRANCH_PREDICTOR) ||	\
>> +     defined(CONFIG_HARDEN_EL2_VECTORS))
>>  extern char __bp_harden_hyp_vecs_start[], __bp_harden_hyp_vecs_end[];
>>  extern atomic_t arm64_el2_vector_last_slot;
>>  
>> +#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
>>  DECLARE_PER_CPU_READ_MOSTLY(struct bp_hardening_data, bp_hardening_data);
>>  
>>  static inline struct bp_hardening_data *arm64_get_bp_hardening_data(void)
>> @@ -81,6 +83,7 @@ static inline struct bp_hardening_data *arm64_get_bp_hardening_data(void)
>>  
>>  static inline void arm64_apply_bp_hardening(void)	{ }
>>  #endif	/* CONFIG_HARDEN_BRANCH_PREDICTOR */
>> +#endif  /* CONFIG_HARDEN_BRANCH_PREDICTOR || CONFIG_HARDEN_EL2_VECTORS */
>>  
>>  extern void paging_init(void);
>>  extern void bootmem_init(void);
>> diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
>> index bd8cc03d7522..a2e3a5af1113 100644
>> --- a/arch/arm64/kvm/Kconfig
>> +++ b/arch/arm64/kvm/Kconfig
>> @@ -58,7 +58,7 @@ config KVM_ARM_PMU
>>  	  virtual machines.
>>  
>>  config KVM_INDIRECT_VECTORS
>> -       def_bool KVM && HARDEN_BRANCH_PREDICTOR
>> +       def_bool KVM && (HARDEN_BRANCH_PREDICTOR || HARDEN_EL2_VECTORS)
>>  
>>  source drivers/vhost/Kconfig
>>  
>> diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c
>> index 9d35c17016ed..fd2d658a11b5 100644
>> --- a/arch/arm64/kvm/va_layout.c
>> +++ b/arch/arm64/kvm/va_layout.c
>> @@ -151,6 +151,9 @@ void __init kvm_update_va_mask(struct alt_instr *alt,
>>  	}
>>  }
>>  
>> +void *__kvm_bp_vect_base;
>> +int __kvm_harden_el2_vector_slot;
>> +
>>  void kvm_patch_vector_branch(struct alt_instr *alt,
>>  			     __le32 *origptr, __le32 *updptr, int nr_inst)
>>  {
>> -- 
>> 2.14.2
>>

I'll fix all the spelling and stylistic remarks.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 25/26] arm64: KVM: Allow mapping of vectors outside of the RAM region
@ 2018-03-15 16:17       ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-15 16:17 UTC (permalink / raw)
  To: linux-arm-kernel

On 15/03/18 15:54, Andrew Jones wrote:
> On Wed, Mar 14, 2018 at 04:50:48PM +0000, Marc Zyngier wrote:
>> We're now ready to map our vectors in weird and wonderful locations.
>> On enabling ARM64_HARDEN_EL2_VECTORS, a vector slots gets allocated
> 
> s/slots/slot/
> 
>> if this hasn't been already done via ARM64_HARDEN_BRANCH_PREDICTOR
>> and gets mapped outside of the normal RAM region, next to the
>> idmap.
>>
>> That way, being able to obtain VBAR_EL2 doesn't reveal the mapping
>> of the rest of the hypervisor code.
>>
>> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
>> ---
>>  Documentation/arm64/memory.txt   |  3 +-
>>  arch/arm64/Kconfig               | 16 +++++++++
>>  arch/arm64/include/asm/kvm_mmu.h | 78 +++++++++++++++++++++++++++++++++++-----
>>  arch/arm64/include/asm/mmu.h     |  5 ++-
>>  arch/arm64/kvm/Kconfig           |  2 +-
>>  arch/arm64/kvm/va_layout.c       |  3 ++
>>  6 files changed, 95 insertions(+), 12 deletions(-)
>>
>> diff --git a/Documentation/arm64/memory.txt b/Documentation/arm64/memory.txt
>> index c58cc5dbe667..c5dab30d3389 100644
>> --- a/Documentation/arm64/memory.txt
>> +++ b/Documentation/arm64/memory.txt
>> @@ -90,7 +90,8 @@ When using KVM without the Virtualization Host Extensions, the
>>  hypervisor maps kernel pages in EL2 at a fixed (and potentially
>>  random) offset from the linear mapping. See the kern_hyp_va macro and
>>  kvm_update_va_mask function for more details. MMIO devices such as
>> -GICv2 gets mapped next to the HYP idmap page.
>> +GICv2 gets mapped next to the HYP idmap page, as do vectors when
>> +ARM64_HARDEN_EL2_VECTORS is selected for particular CPUs.
>>  
>>  When using KVM with the Virtualization Host Extensions, no additional
>>  mappings are created, since the host kernel runs directly in EL2.
>> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
>> index 7381eeb7ef8e..e6be4393aaad 100644
>> --- a/arch/arm64/Kconfig
>> +++ b/arch/arm64/Kconfig
>> @@ -904,6 +904,22 @@ config HARDEN_BRANCH_PREDICTOR
>>  
>>  	  If unsure, say Y.
>>  
>> +config HARDEN_EL2_VECTORS
>> +	bool "Harden EL2 vector mapping against system register leak" if EXPERT
>> +	default y
>> +	help
>> +	  Speculation attacks against some high-performance processors can
>> +	  be used to leak privileged information such as the vector base
>> +	  register, resulting in a potential defeat of the EL2 layout
>> +	  randomization.
>> +
>> +	  This config option will map the vectors to a fixed location,
>> +	  independent of the EL2 code mapping, so that revealing VBAR_EL2
> 
> s/VBAR_EL2/the vector base register/ ?

Which vector base register? VBAR_EL1 is under control of the guest, so
it can readily find that one. We're really trying to prevent that
particular register from being used to disclose anything useful.

> 
>> +	  to an attacker does no give away any extra information. This
> 
> s/no/not/
> 
>> +	  only gets enabled on affected CPUs.
>> +
>> +	  If unsure, say Y.
>> +
>>  menuconfig ARMV8_DEPRECATED
>>  	bool "Emulate deprecated/obsolete ARMv8 instructions"
>>  	depends on COMPAT
>> diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
>> index 97af11065bbd..f936d0928661 100644
>> --- a/arch/arm64/include/asm/kvm_mmu.h
>> +++ b/arch/arm64/include/asm/kvm_mmu.h
>> @@ -360,31 +360,91 @@ static inline unsigned int kvm_get_vmid_bits(void)
>>  	return (cpuid_feature_extract_unsigned_field(reg, ID_AA64MMFR1_VMIDBITS_SHIFT) == 2) ? 16 : 8;
>>  }
>>  
>> -#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
>> +#ifdef CONFIG_KVM_INDIRECT_VECTORS
>> +/*
>> + * EL2 vectors can be mapped and rerouted in a number of ways,
>> + * depending on the kernel configuration and CPU present:
>> + *
>> + * - If the CPU has the ARM64_HARDEN_BRANCH_PREDICTOR cap, the
>> + *   hardening sequence is placed in one of the vector slots, which is
>> + *   executed before jumping to the real vectors.
>> + *
>> + * - If the CPU has both the ARM64_HARDEN_EL2_VECTORS and BP
>                                                         ^cap
> 
> Maybe s/BP hardening/the ARM64_HARDEN_BRANCH_PREDICTOR cap/ ?
> 
>> + *   hardening, the slot containing the hardening sequence is mapped
>> + *   next to the idmap page, and executed before jumping to the real
>> + *   vectors.
>> + *
>> + * - If the CPU only has ARM64_HARDEN_EL2_VECTORS, then an empty slot
> 
> ...has the ARM64_HARDEN_EL2_VECTORS cap
> 
> Or maybe change the above ones from 'has the ... cap' to just 'has ...',
> like this one.
> 
>> + *   is selected, mapped next to the idmap page, and executed before
>> + *   jumping to the real vectors.
>> + *
>> + * Note that ARM64_HARDEN_EL2_VECTORS is somewhat incompatible with
>> + * VHE, as we don't have hypervisor-specific mappings. If the system
>> + * is VHE and yet selects this capability, it will be ignored.
>> + */
>>  #include <asm/mmu.h>
>>  
>> +extern void *__kvm_bp_vect_base;
>> +extern int __kvm_harden_el2_vector_slot;
>> +
>>  static inline void *kvm_get_hyp_vector(void)
>>  {
>>  	struct bp_hardening_data *data = arm64_get_bp_hardening_data();
>> -	void *vect = kvm_ksym_ref(__kvm_hyp_vector);
>> +	void *vect = kern_hyp_va(kvm_ksym_ref(__kvm_hyp_vector));
>> +	int slot = -1;
>>  
>> -	if (data->fn) {
>> -		vect = __bp_harden_hyp_vecs_start +
>> -		       data->hyp_vectors_slot * SZ_2K;
>> +	if (cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR) && data->fn) {
> 
> Now that I'm trying to consider heterogeneous systems, I'm wondering why
> harden BP isn't checked with this_cpu_has_cap() like harden EL2 vectors.
> I'm also wondering if it's even possible for CPUs to come up without all
> of them having harden EL2 vectors if the boot CPU has it. Won't
> verify_local_cpu_errata_workarounds() require it? I'm probably just
> getting lost in all the capability stuff...

Checking harden BP on this particular CPU will only tell you if this CPU
is affected, but won't give you any additional information (i.e. you'd
still need to check obtain the stuff pointed to by data).

Checking for *all* CPUs in one go has some advantages: it is a static
key, which means that unaffected platforms will fly (this is a hot path
on VHE), and you can check data if you're affected.

> 
>> +		vect = __bp_harden_hyp_vecs_start;
>> +		slot = data->hyp_vectors_slot;
>> +	}
>>  
>> -		if (!has_vhe())
>> -			vect = lm_alias(vect);
>> +	if (this_cpu_has_cap(ARM64_HARDEN_EL2_VECTORS) && !has_vhe()) {
>> +		vect = __kvm_bp_vect_base;
>> +		if (slot == -1)
>> +			slot = __kvm_harden_el2_vector_slot;
>>  	}
>>  
>> -	vect = kern_hyp_va(vect);
>> +	if (slot != -1)
>> +		vect += slot * SZ_2K;
>> +
>>  	return vect;
>>  }
>>  
>> +/*  This is only called on a !VHE system */
>>  static inline int kvm_map_vectors(void)
>>  {
>> +	/*
>> +	 * HBP  = ARM64_HARDEN_BRANCH_PREDICTOR
>> +	 * HLE2 = ARM64_HARDEN_EL2_VECTORS
> 
> s/HLE2/HEL2/
> 
>> +	 *
>> +	 * !HBP + !HEL2 -> use direct vectors
>> +	 *  HBP + !HEL2 -> use hardenned vectors in place
> 
> hardened 
> 
>> +	 * !HBP +  HEL2 -> allocate one vector slot and use exec mapping
>> +	 *  HBP +  HEL2 -> use hardenned vertors and use exec mapping
> 
> hardened vectors
> 
>> +	 */
>> +	if (cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR)) {
>> +		__kvm_bp_vect_base = kvm_ksym_ref(__bp_harden_hyp_vecs_start);
>> +		__kvm_bp_vect_base = kern_hyp_va(__kvm_bp_vect_base);
>> +	}
>> +
>> +	if (cpus_have_const_cap(ARM64_HARDEN_EL2_VECTORS)) {
> 
> What happened to this_cpu_has_cap()? Did we switch because this is only
> running on the boot CPU?

Only one CPU performs all the mappings (see init_hyp_mode()).

> 
>> +		phys_addr_t vect_pa = __pa_symbol(__bp_harden_hyp_vecs_start);
>> +		unsigned long size = (__bp_harden_hyp_vecs_end -
>> +				      __bp_harden_hyp_vecs_start);
>> +
>> +		/*
>> +		 * Always allocate a spare vector slot, as we don't
>> +		 * know yet which CPUs have a BP hardening slot that
>> +		 * we can reuse.
>> +		 */
>> +		__kvm_harden_el2_vector_slot = atomic_inc_return(&arm64_el2_vector_last_slot);
>> +		BUG_ON(__kvm_harden_el2_vector_slot >= BP_HARDEN_EL2_SLOTS);
>> +		return create_hyp_exec_mappings(vect_pa, size,
>> +						&__kvm_bp_vect_base);
>> +	}
>> +
>>  	return 0;
>>  }
>> -
>>  #else
>>  static inline void *kvm_get_hyp_vector(void)
>>  {
>> diff --git a/arch/arm64/include/asm/mmu.h b/arch/arm64/include/asm/mmu.h
>> index 3baf010fe883..0b0cc69031c1 100644
>> --- a/arch/arm64/include/asm/mmu.h
>> +++ b/arch/arm64/include/asm/mmu.h
>> @@ -51,10 +51,12 @@ struct bp_hardening_data {
>>  	bp_hardening_cb_t	fn;
>>  };
>>  
>> -#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
>> +#if (defined(CONFIG_HARDEN_BRANCH_PREDICTOR) ||	\
>> +     defined(CONFIG_HARDEN_EL2_VECTORS))
>>  extern char __bp_harden_hyp_vecs_start[], __bp_harden_hyp_vecs_end[];
>>  extern atomic_t arm64_el2_vector_last_slot;
>>  
>> +#ifdef CONFIG_HARDEN_BRANCH_PREDICTOR
>>  DECLARE_PER_CPU_READ_MOSTLY(struct bp_hardening_data, bp_hardening_data);
>>  
>>  static inline struct bp_hardening_data *arm64_get_bp_hardening_data(void)
>> @@ -81,6 +83,7 @@ static inline struct bp_hardening_data *arm64_get_bp_hardening_data(void)
>>  
>>  static inline void arm64_apply_bp_hardening(void)	{ }
>>  #endif	/* CONFIG_HARDEN_BRANCH_PREDICTOR */
>> +#endif  /* CONFIG_HARDEN_BRANCH_PREDICTOR || CONFIG_HARDEN_EL2_VECTORS */
>>  
>>  extern void paging_init(void);
>>  extern void bootmem_init(void);
>> diff --git a/arch/arm64/kvm/Kconfig b/arch/arm64/kvm/Kconfig
>> index bd8cc03d7522..a2e3a5af1113 100644
>> --- a/arch/arm64/kvm/Kconfig
>> +++ b/arch/arm64/kvm/Kconfig
>> @@ -58,7 +58,7 @@ config KVM_ARM_PMU
>>  	  virtual machines.
>>  
>>  config KVM_INDIRECT_VECTORS
>> -       def_bool KVM && HARDEN_BRANCH_PREDICTOR
>> +       def_bool KVM && (HARDEN_BRANCH_PREDICTOR || HARDEN_EL2_VECTORS)
>>  
>>  source drivers/vhost/Kconfig
>>  
>> diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c
>> index 9d35c17016ed..fd2d658a11b5 100644
>> --- a/arch/arm64/kvm/va_layout.c
>> +++ b/arch/arm64/kvm/va_layout.c
>> @@ -151,6 +151,9 @@ void __init kvm_update_va_mask(struct alt_instr *alt,
>>  	}
>>  }
>>  
>> +void *__kvm_bp_vect_base;
>> +int __kvm_harden_el2_vector_slot;
>> +
>>  void kvm_patch_vector_branch(struct alt_instr *alt,
>>  			     __le32 *origptr, __le32 *updptr, int nr_inst)
>>  {
>> -- 
>> 2.14.2
>>

I'll fix all the spelling and stylistic remarks.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 00/26] KVM/arm64: Randomise EL2 mappings (variant 3a mitigation)
  2018-03-15 15:57   ` Andrew Jones
@ 2018-03-15 16:19     ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-15 16:19 UTC (permalink / raw)
  To: Andrew Jones
  Cc: Mark Rutland, Peter Maydell, Christoffer Dall, kvm, Steve Capper,
	Catalin Marinas, Will Deacon, Kristina Martsenko, James Morse,
	kvmarm, linux-arm-kernel

On 15/03/18 15:57, Andrew Jones wrote:
> On Wed, Mar 14, 2018 at 04:50:23PM +0000, Marc Zyngier wrote:
>> Whilst KVM benefits from the kernel randomisation via KASLR, there is
>> no additional randomisation when the kernel is running at EL1, as we
>> directly use a fixed offset from the linear mapping. This is not
>> necessarily a problem, but we could do a bit better by independently
>> randomizing the HYP placement.
>>
>> This series proposes to randomise the offset by inserting a few random
>> bits between the MSB of the RAM linear mapping and the top of the HYP
>> VA (VA_BITS - 2). That's not a lot of random bits (on my Mustang, I
>> get 13 bits), but that's better than nothing.
>>
>> In order to achieve this, we need to be able to patch dynamic values
>> in the kernel text. This results in a bunch of changes to the
>> alternative framework, the insn library, and a few more hacks in KVM
>> itself (we get a new way to map the GIC at EL2).
>>
>> Another (and more recent) goal of this series is to work around what
>> has been described as "variant 3a", which covers speculative reads of
>> privileged system registers. Randomizing the location of the
>> hypervisor would be pointless if one could simply obtain VBAR_EL2. In
>> order to work around this, we place the vectors at a fairly static
>> location (next to the idmap), independently of the hypervisor's own
>> mappings. This ensures that we can leak VBAR_EL2 without disclosing
>> much about HYP itself (and is similar to what the rest of the kernel
>> does with KPTI). This is only enabled at runtime for Cortex-A57 and
>> Cortex-A72.
>>
>> This has been tested on the FVP model, Seattle (both 39 and 48bit VA),
>> Mustang and Thunder-X. I've also done a sanity check on 32bit (which
>> is only impacted by the HYP IO VA stuff).
>>
> 
> I've smoke tested this series on a seattle with several busy VMs running
> simultaneously. My host kernel configures 64K pages. I didn't see any
> problems.

Excellent, thanks for testing. Out of curiosity: do you have a firmware
that implements the SMCCC_ARCH_WORKAROUND_1 call or not?

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 00/26] KVM/arm64: Randomise EL2 mappings (variant 3a mitigation)
@ 2018-03-15 16:19     ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-15 16:19 UTC (permalink / raw)
  To: linux-arm-kernel

On 15/03/18 15:57, Andrew Jones wrote:
> On Wed, Mar 14, 2018 at 04:50:23PM +0000, Marc Zyngier wrote:
>> Whilst KVM benefits from the kernel randomisation via KASLR, there is
>> no additional randomisation when the kernel is running at EL1, as we
>> directly use a fixed offset from the linear mapping. This is not
>> necessarily a problem, but we could do a bit better by independently
>> randomizing the HYP placement.
>>
>> This series proposes to randomise the offset by inserting a few random
>> bits between the MSB of the RAM linear mapping and the top of the HYP
>> VA (VA_BITS - 2). That's not a lot of random bits (on my Mustang, I
>> get 13 bits), but that's better than nothing.
>>
>> In order to achieve this, we need to be able to patch dynamic values
>> in the kernel text. This results in a bunch of changes to the
>> alternative framework, the insn library, and a few more hacks in KVM
>> itself (we get a new way to map the GIC at EL2).
>>
>> Another (and more recent) goal of this series is to work around what
>> has been described as "variant 3a", which covers speculative reads of
>> privileged system registers. Randomizing the location of the
>> hypervisor would be pointless if one could simply obtain VBAR_EL2. In
>> order to work around this, we place the vectors at a fairly static
>> location (next to the idmap), independently of the hypervisor's own
>> mappings. This ensures that we can leak VBAR_EL2 without disclosing
>> much about HYP itself (and is similar to what the rest of the kernel
>> does with KPTI). This is only enabled at runtime for Cortex-A57 and
>> Cortex-A72.
>>
>> This has been tested on the FVP model, Seattle (both 39 and 48bit VA),
>> Mustang and Thunder-X. I've also done a sanity check on 32bit (which
>> is only impacted by the HYP IO VA stuff).
>>
> 
> I've smoke tested this series on a seattle with several busy VMs running
> simultaneously. My host kernel configures 64K pages. I didn't see any
> problems.

Excellent, thanks for testing. Out of curiosity: do you have a firmware
that implements the SMCCC_ARCH_WORKAROUND_1 call or not?

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 00/26] KVM/arm64: Randomise EL2 mappings (variant 3a mitigation)
  2018-03-15 16:19     ` Marc Zyngier
@ 2018-03-15 16:40       ` Andrew Jones
  -1 siblings, 0 replies; 124+ messages in thread
From: Andrew Jones @ 2018-03-15 16:40 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: Mark Rutland, Peter Maydell, Christoffer Dall, kvm, Steve Capper,
	Catalin Marinas, Will Deacon, Kristina Martsenko, James Morse,
	kvmarm, linux-arm-kernel

On Thu, Mar 15, 2018 at 04:19:56PM +0000, Marc Zyngier wrote:
> On 15/03/18 15:57, Andrew Jones wrote:
> > On Wed, Mar 14, 2018 at 04:50:23PM +0000, Marc Zyngier wrote:
> >> Whilst KVM benefits from the kernel randomisation via KASLR, there is
> >> no additional randomisation when the kernel is running at EL1, as we
> >> directly use a fixed offset from the linear mapping. This is not
> >> necessarily a problem, but we could do a bit better by independently
> >> randomizing the HYP placement.
> >>
> >> This series proposes to randomise the offset by inserting a few random
> >> bits between the MSB of the RAM linear mapping and the top of the HYP
> >> VA (VA_BITS - 2). That's not a lot of random bits (on my Mustang, I
> >> get 13 bits), but that's better than nothing.
> >>
> >> In order to achieve this, we need to be able to patch dynamic values
> >> in the kernel text. This results in a bunch of changes to the
> >> alternative framework, the insn library, and a few more hacks in KVM
> >> itself (we get a new way to map the GIC at EL2).
> >>
> >> Another (and more recent) goal of this series is to work around what
> >> has been described as "variant 3a", which covers speculative reads of
> >> privileged system registers. Randomizing the location of the
> >> hypervisor would be pointless if one could simply obtain VBAR_EL2. In
> >> order to work around this, we place the vectors at a fairly static
> >> location (next to the idmap), independently of the hypervisor's own
> >> mappings. This ensures that we can leak VBAR_EL2 without disclosing
> >> much about HYP itself (and is similar to what the rest of the kernel
> >> does with KPTI). This is only enabled at runtime for Cortex-A57 and
> >> Cortex-A72.
> >>
> >> This has been tested on the FVP model, Seattle (both 39 and 48bit VA),
> >> Mustang and Thunder-X. I've also done a sanity check on 32bit (which
> >> is only impacted by the HYP IO VA stuff).
> >>
> > 
> > I've smoke tested this series on a seattle with several busy VMs running
> > simultaneously. My host kernel configures 64K pages. I didn't see any
> > problems.
> 
> Excellent, thanks for testing. Out of curiosity: do you have a firmware
> that implements the SMCCC_ARCH_WORKAROUND_1 call or not?

Unfortunately not, so I guess that means I was testing the
__kvm_bp_vect_base base with the __kvm_harden_el2_vector_slot slot.

Thanks,
drew

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 00/26] KVM/arm64: Randomise EL2 mappings (variant 3a mitigation)
@ 2018-03-15 16:40       ` Andrew Jones
  0 siblings, 0 replies; 124+ messages in thread
From: Andrew Jones @ 2018-03-15 16:40 UTC (permalink / raw)
  To: linux-arm-kernel

On Thu, Mar 15, 2018 at 04:19:56PM +0000, Marc Zyngier wrote:
> On 15/03/18 15:57, Andrew Jones wrote:
> > On Wed, Mar 14, 2018 at 04:50:23PM +0000, Marc Zyngier wrote:
> >> Whilst KVM benefits from the kernel randomisation via KASLR, there is
> >> no additional randomisation when the kernel is running at EL1, as we
> >> directly use a fixed offset from the linear mapping. This is not
> >> necessarily a problem, but we could do a bit better by independently
> >> randomizing the HYP placement.
> >>
> >> This series proposes to randomise the offset by inserting a few random
> >> bits between the MSB of the RAM linear mapping and the top of the HYP
> >> VA (VA_BITS - 2). That's not a lot of random bits (on my Mustang, I
> >> get 13 bits), but that's better than nothing.
> >>
> >> In order to achieve this, we need to be able to patch dynamic values
> >> in the kernel text. This results in a bunch of changes to the
> >> alternative framework, the insn library, and a few more hacks in KVM
> >> itself (we get a new way to map the GIC at EL2).
> >>
> >> Another (and more recent) goal of this series is to work around what
> >> has been described as "variant 3a", which covers speculative reads of
> >> privileged system registers. Randomizing the location of the
> >> hypervisor would be pointless if one could simply obtain VBAR_EL2. In
> >> order to work around this, we place the vectors at a fairly static
> >> location (next to the idmap), independently of the hypervisor's own
> >> mappings. This ensures that we can leak VBAR_EL2 without disclosing
> >> much about HYP itself (and is similar to what the rest of the kernel
> >> does with KPTI). This is only enabled at runtime for Cortex-A57 and
> >> Cortex-A72.
> >>
> >> This has been tested on the FVP model, Seattle (both 39 and 48bit VA),
> >> Mustang and Thunder-X. I've also done a sanity check on 32bit (which
> >> is only impacted by the HYP IO VA stuff).
> >>
> > 
> > I've smoke tested this series on a seattle with several busy VMs running
> > simultaneously. My host kernel configures 64K pages. I didn't see any
> > problems.
> 
> Excellent, thanks for testing. Out of curiosity: do you have a firmware
> that implements the SMCCC_ARCH_WORKAROUND_1 call or not?

Unfortunately not, so I guess that means I was testing the
__kvm_bp_vect_base base with the __kvm_harden_el2_vector_slot slot.

Thanks,
drew

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 00/26] KVM/arm64: Randomise EL2 mappings (variant 3a mitigation)
  2018-03-15 16:40       ` Andrew Jones
@ 2018-03-15 16:52         ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-15 16:52 UTC (permalink / raw)
  To: Andrew Jones
  Cc: kvm, Catalin Marinas, Will Deacon, Kristina Martsenko, kvmarm,
	linux-arm-kernel

On 15/03/18 16:40, Andrew Jones wrote:
> On Thu, Mar 15, 2018 at 04:19:56PM +0000, Marc Zyngier wrote:
>> On 15/03/18 15:57, Andrew Jones wrote:
>>> On Wed, Mar 14, 2018 at 04:50:23PM +0000, Marc Zyngier wrote:
>>>> Whilst KVM benefits from the kernel randomisation via KASLR, there is
>>>> no additional randomisation when the kernel is running at EL1, as we
>>>> directly use a fixed offset from the linear mapping. This is not
>>>> necessarily a problem, but we could do a bit better by independently
>>>> randomizing the HYP placement.
>>>>
>>>> This series proposes to randomise the offset by inserting a few random
>>>> bits between the MSB of the RAM linear mapping and the top of the HYP
>>>> VA (VA_BITS - 2). That's not a lot of random bits (on my Mustang, I
>>>> get 13 bits), but that's better than nothing.
>>>>
>>>> In order to achieve this, we need to be able to patch dynamic values
>>>> in the kernel text. This results in a bunch of changes to the
>>>> alternative framework, the insn library, and a few more hacks in KVM
>>>> itself (we get a new way to map the GIC at EL2).
>>>>
>>>> Another (and more recent) goal of this series is to work around what
>>>> has been described as "variant 3a", which covers speculative reads of
>>>> privileged system registers. Randomizing the location of the
>>>> hypervisor would be pointless if one could simply obtain VBAR_EL2. In
>>>> order to work around this, we place the vectors at a fairly static
>>>> location (next to the idmap), independently of the hypervisor's own
>>>> mappings. This ensures that we can leak VBAR_EL2 without disclosing
>>>> much about HYP itself (and is similar to what the rest of the kernel
>>>> does with KPTI). This is only enabled at runtime for Cortex-A57 and
>>>> Cortex-A72.
>>>>
>>>> This has been tested on the FVP model, Seattle (both 39 and 48bit VA),
>>>> Mustang and Thunder-X. I've also done a sanity check on 32bit (which
>>>> is only impacted by the HYP IO VA stuff).
>>>>
>>>
>>> I've smoke tested this series on a seattle with several busy VMs running
>>> simultaneously. My host kernel configures 64K pages. I didn't see any
>>> problems.
>>
>> Excellent, thanks for testing. Out of curiosity: do you have a firmware
>> that implements the SMCCC_ARCH_WORKAROUND_1 call or not?
> 
> Unfortunately not, so I guess that means I was testing the
> __kvm_bp_vect_base base with the __kvm_harden_el2_vector_slot slot.

Indeed. That's a good data point nonetheless, and a good complement to
my own setup where the workaround is implemented. Getting full coverage
is pretty mind-boggling...

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 00/26] KVM/arm64: Randomise EL2 mappings (variant 3a mitigation)
@ 2018-03-15 16:52         ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-15 16:52 UTC (permalink / raw)
  To: linux-arm-kernel

On 15/03/18 16:40, Andrew Jones wrote:
> On Thu, Mar 15, 2018 at 04:19:56PM +0000, Marc Zyngier wrote:
>> On 15/03/18 15:57, Andrew Jones wrote:
>>> On Wed, Mar 14, 2018 at 04:50:23PM +0000, Marc Zyngier wrote:
>>>> Whilst KVM benefits from the kernel randomisation via KASLR, there is
>>>> no additional randomisation when the kernel is running at EL1, as we
>>>> directly use a fixed offset from the linear mapping. This is not
>>>> necessarily a problem, but we could do a bit better by independently
>>>> randomizing the HYP placement.
>>>>
>>>> This series proposes to randomise the offset by inserting a few random
>>>> bits between the MSB of the RAM linear mapping and the top of the HYP
>>>> VA (VA_BITS - 2). That's not a lot of random bits (on my Mustang, I
>>>> get 13 bits), but that's better than nothing.
>>>>
>>>> In order to achieve this, we need to be able to patch dynamic values
>>>> in the kernel text. This results in a bunch of changes to the
>>>> alternative framework, the insn library, and a few more hacks in KVM
>>>> itself (we get a new way to map the GIC at EL2).
>>>>
>>>> Another (and more recent) goal of this series is to work around what
>>>> has been described as "variant 3a", which covers speculative reads of
>>>> privileged system registers. Randomizing the location of the
>>>> hypervisor would be pointless if one could simply obtain VBAR_EL2. In
>>>> order to work around this, we place the vectors at a fairly static
>>>> location (next to the idmap), independently of the hypervisor's own
>>>> mappings. This ensures that we can leak VBAR_EL2 without disclosing
>>>> much about HYP itself (and is similar to what the rest of the kernel
>>>> does with KPTI). This is only enabled at runtime for Cortex-A57 and
>>>> Cortex-A72.
>>>>
>>>> This has been tested on the FVP model, Seattle (both 39 and 48bit VA),
>>>> Mustang and Thunder-X. I've also done a sanity check on 32bit (which
>>>> is only impacted by the HYP IO VA stuff).
>>>>
>>>
>>> I've smoke tested this series on a seattle with several busy VMs running
>>> simultaneously. My host kernel configures 64K pages. I didn't see any
>>> problems.
>>
>> Excellent, thanks for testing. Out of curiosity: do you have a firmware
>> that implements the SMCCC_ARCH_WORKAROUND_1 call or not?
> 
> Unfortunately not, so I guess that means I was testing the
> __kvm_bp_vect_base base with the __kvm_harden_el2_vector_slot slot.

Indeed. That's a good data point nonetheless, and a good complement to
my own setup where the workaround is implemented. Getting full coverage
is pretty mind-boggling...

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 25/26] arm64: KVM: Allow mapping of vectors outside of the RAM region
  2018-03-15 16:17       ` Marc Zyngier
@ 2018-03-15 17:08         ` Andrew Jones
  -1 siblings, 0 replies; 124+ messages in thread
From: Andrew Jones @ 2018-03-15 17:08 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: kvm, Catalin Marinas, Will Deacon, Kristina Martsenko, kvmarm,
	linux-arm-kernel

On Thu, Mar 15, 2018 at 04:17:53PM +0000, Marc Zyngier wrote:
> On 15/03/18 15:54, Andrew Jones wrote:
> > On Wed, Mar 14, 2018 at 04:50:48PM +0000, Marc Zyngier wrote:
> >>  static inline void *kvm_get_hyp_vector(void)
> >>  {
> >>  	struct bp_hardening_data *data = arm64_get_bp_hardening_data();
> >> -	void *vect = kvm_ksym_ref(__kvm_hyp_vector);
> >> +	void *vect = kern_hyp_va(kvm_ksym_ref(__kvm_hyp_vector));
> >> +	int slot = -1;
> >>  
> >> -	if (data->fn) {
> >> -		vect = __bp_harden_hyp_vecs_start +
> >> -		       data->hyp_vectors_slot * SZ_2K;
> >> +	if (cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR) && data->fn) {
> > 
> > Now that I'm trying to consider heterogeneous systems, I'm wondering why
> > harden BP isn't checked with this_cpu_has_cap() like harden EL2 vectors.
> > I'm also wondering if it's even possible for CPUs to come up without all
> > of them having harden EL2 vectors if the boot CPU has it. Won't
> > verify_local_cpu_errata_workarounds() require it? I'm probably just
> > getting lost in all the capability stuff...
> 
> Checking harden BP on this particular CPU will only tell you if this CPU
> is affected, but won't give you any additional information (i.e. you'd
> still need to check obtain the stuff pointed to by data).
> 
> Checking for *all* CPUs in one go has some advantages: it is a static
> key, which means that unaffected platforms will fly (this is a hot path
> on VHE), and you can check data if you're affected.

Is there a problem with using the static key to check
ARM64_HARDEN_EL2_VECTORS below? (Sorry, I'm still confused why we're not
using the same functions for both.) If we can't use the static key
below, then should we swap the check with !has_vhe() to ensure the
compiler will avoid emitting the this_cpu_has_cap() call on VHE's fast
path?

I also noticed that calling this_cpu_has_cap() puts a constraint on
kvm_get_hyp_vector() that it must never be called without preemption
disabled. It's not, but it's one more thing to think about. If there's
no reason not to use the static key with cpus_have_const_cap() then
maybe we should?

> 
> > 
> >> +		vect = __bp_harden_hyp_vecs_start;
> >> +		slot = data->hyp_vectors_slot;
> >> +	}
> >>  
> >> -		if (!has_vhe())
> >> -			vect = lm_alias(vect);
> >> +	if (this_cpu_has_cap(ARM64_HARDEN_EL2_VECTORS) && !has_vhe()) {
> >> +		vect = __kvm_bp_vect_base;
> >> +		if (slot == -1)
> >> +			slot = __kvm_harden_el2_vector_slot;
> >>  	}

Thanks,
drew

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 25/26] arm64: KVM: Allow mapping of vectors outside of the RAM region
@ 2018-03-15 17:08         ` Andrew Jones
  0 siblings, 0 replies; 124+ messages in thread
From: Andrew Jones @ 2018-03-15 17:08 UTC (permalink / raw)
  To: linux-arm-kernel

On Thu, Mar 15, 2018 at 04:17:53PM +0000, Marc Zyngier wrote:
> On 15/03/18 15:54, Andrew Jones wrote:
> > On Wed, Mar 14, 2018 at 04:50:48PM +0000, Marc Zyngier wrote:
> >>  static inline void *kvm_get_hyp_vector(void)
> >>  {
> >>  	struct bp_hardening_data *data = arm64_get_bp_hardening_data();
> >> -	void *vect = kvm_ksym_ref(__kvm_hyp_vector);
> >> +	void *vect = kern_hyp_va(kvm_ksym_ref(__kvm_hyp_vector));
> >> +	int slot = -1;
> >>  
> >> -	if (data->fn) {
> >> -		vect = __bp_harden_hyp_vecs_start +
> >> -		       data->hyp_vectors_slot * SZ_2K;
> >> +	if (cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR) && data->fn) {
> > 
> > Now that I'm trying to consider heterogeneous systems, I'm wondering why
> > harden BP isn't checked with this_cpu_has_cap() like harden EL2 vectors.
> > I'm also wondering if it's even possible for CPUs to come up without all
> > of them having harden EL2 vectors if the boot CPU has it. Won't
> > verify_local_cpu_errata_workarounds() require it? I'm probably just
> > getting lost in all the capability stuff...
> 
> Checking harden BP on this particular CPU will only tell you if this CPU
> is affected, but won't give you any additional information (i.e. you'd
> still need to check obtain the stuff pointed to by data).
> 
> Checking for *all* CPUs in one go has some advantages: it is a static
> key, which means that unaffected platforms will fly (this is a hot path
> on VHE), and you can check data if you're affected.

Is there a problem with using the static key to check
ARM64_HARDEN_EL2_VECTORS below? (Sorry, I'm still confused why we're not
using the same functions for both.) If we can't use the static key
below, then should we swap the check with !has_vhe() to ensure the
compiler will avoid emitting the this_cpu_has_cap() call on VHE's fast
path?

I also noticed that calling this_cpu_has_cap() puts a constraint on
kvm_get_hyp_vector() that it must never be called without preemption
disabled. It's not, but it's one more thing to think about. If there's
no reason not to use the static key with cpus_have_const_cap() then
maybe we should?

> 
> > 
> >> +		vect = __bp_harden_hyp_vecs_start;
> >> +		slot = data->hyp_vectors_slot;
> >> +	}
> >>  
> >> -		if (!has_vhe())
> >> -			vect = lm_alias(vect);
> >> +	if (this_cpu_has_cap(ARM64_HARDEN_EL2_VECTORS) && !has_vhe()) {
> >> +		vect = __kvm_bp_vect_base;
> >> +		if (slot == -1)
> >> +			slot = __kvm_harden_el2_vector_slot;
> >>  	}

Thanks,
drew

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 25/26] arm64: KVM: Allow mapping of vectors outside of the RAM region
  2018-03-15 17:08         ` Andrew Jones
@ 2018-03-15 18:47           ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-15 18:47 UTC (permalink / raw)
  To: Andrew Jones
  Cc: Mark Rutland, Peter Maydell, Christoffer Dall, kvm, Steve Capper,
	Catalin Marinas, Will Deacon, Kristina Martsenko, James Morse,
	kvmarm, linux-arm-kernel

On 15/03/18 17:08, Andrew Jones wrote:
> On Thu, Mar 15, 2018 at 04:17:53PM +0000, Marc Zyngier wrote:
>> On 15/03/18 15:54, Andrew Jones wrote:
>>> On Wed, Mar 14, 2018 at 04:50:48PM +0000, Marc Zyngier wrote:
>>>>  static inline void *kvm_get_hyp_vector(void)
>>>>  {
>>>>  	struct bp_hardening_data *data = arm64_get_bp_hardening_data();
>>>> -	void *vect = kvm_ksym_ref(__kvm_hyp_vector);
>>>> +	void *vect = kern_hyp_va(kvm_ksym_ref(__kvm_hyp_vector));
>>>> +	int slot = -1;
>>>>  
>>>> -	if (data->fn) {
>>>> -		vect = __bp_harden_hyp_vecs_start +
>>>> -		       data->hyp_vectors_slot * SZ_2K;
>>>> +	if (cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR) && data->fn) {
>>>
>>> Now that I'm trying to consider heterogeneous systems, I'm wondering why
>>> harden BP isn't checked with this_cpu_has_cap() like harden EL2 vectors.
>>> I'm also wondering if it's even possible for CPUs to come up without all
>>> of them having harden EL2 vectors if the boot CPU has it. Won't
>>> verify_local_cpu_errata_workarounds() require it? I'm probably just
>>> getting lost in all the capability stuff...
>>
>> Checking harden BP on this particular CPU will only tell you if this CPU
>> is affected, but won't give you any additional information (i.e. you'd
>> still need to check obtain the stuff pointed to by data).
>>
>> Checking for *all* CPUs in one go has some advantages: it is a static
>> key, which means that unaffected platforms will fly (this is a hot path
>> on VHE), and you can check data if you're affected.
> 
> Is there a problem with using the static key to check
> ARM64_HARDEN_EL2_VECTORS below? (Sorry, I'm still confused why we're not
> using the same functions for both.) 

Consider the following case: heterogeneous system with CPU0 that has
HBP, and CPU1 that has HEL. CPU0 is using its own vector slot, and CPU1
should be using an extra one (it doesn't have a dedicated slot).

When CPU0 claims its vectors, it will get the "normal" version of the
mapping. And then, if we're using cpus_have_cap, we turn that into an
out-of-line mapping. That's not completely wrong (the system still runs
fine), but that's not the expected behaviour. If we restrict the second
phase to this_cpu_has_cap(), we get the expected CPU0 using the normal
mapping, and CPU1 does the exact opposite (which we also want).

> If we can't use the static key
> below, then should we swap the check with !has_vhe() to ensure the
> compiler will avoid emitting the this_cpu_has_cap() call on VHE's fast
> path?

Fair enough, that's a good point.

> I also noticed that calling this_cpu_has_cap() puts a constraint on
> kvm_get_hyp_vector() that it must never be called without preemption
> disabled. It's not, but it's one more thing to think about. If there's
> no reason not to use the static key with cpus_have_const_cap() then
> maybe we should?

We're definitely in a non-preemptible section at this stage. On !VHE,
we're in IPI context (cpu_init_hyp_mode), and on VHE we are in the
switch code, having disabled interrupts a long time ago.

> 
>>
>>>
>>>> +		vect = __bp_harden_hyp_vecs_start;

Nit: massive buglet here, which I thought I had squashed in the series,
and obviously didn't (still have it as a fixup). It should read:

vect = kern_hyp_va(kvm_ksym_ref(__bp_harden_hyp_vecs_start));

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 25/26] arm64: KVM: Allow mapping of vectors outside of the RAM region
@ 2018-03-15 18:47           ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-15 18:47 UTC (permalink / raw)
  To: linux-arm-kernel

On 15/03/18 17:08, Andrew Jones wrote:
> On Thu, Mar 15, 2018 at 04:17:53PM +0000, Marc Zyngier wrote:
>> On 15/03/18 15:54, Andrew Jones wrote:
>>> On Wed, Mar 14, 2018 at 04:50:48PM +0000, Marc Zyngier wrote:
>>>>  static inline void *kvm_get_hyp_vector(void)
>>>>  {
>>>>  	struct bp_hardening_data *data = arm64_get_bp_hardening_data();
>>>> -	void *vect = kvm_ksym_ref(__kvm_hyp_vector);
>>>> +	void *vect = kern_hyp_va(kvm_ksym_ref(__kvm_hyp_vector));
>>>> +	int slot = -1;
>>>>  
>>>> -	if (data->fn) {
>>>> -		vect = __bp_harden_hyp_vecs_start +
>>>> -		       data->hyp_vectors_slot * SZ_2K;
>>>> +	if (cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR) && data->fn) {
>>>
>>> Now that I'm trying to consider heterogeneous systems, I'm wondering why
>>> harden BP isn't checked with this_cpu_has_cap() like harden EL2 vectors.
>>> I'm also wondering if it's even possible for CPUs to come up without all
>>> of them having harden EL2 vectors if the boot CPU has it. Won't
>>> verify_local_cpu_errata_workarounds() require it? I'm probably just
>>> getting lost in all the capability stuff...
>>
>> Checking harden BP on this particular CPU will only tell you if this CPU
>> is affected, but won't give you any additional information (i.e. you'd
>> still need to check obtain the stuff pointed to by data).
>>
>> Checking for *all* CPUs in one go has some advantages: it is a static
>> key, which means that unaffected platforms will fly (this is a hot path
>> on VHE), and you can check data if you're affected.
> 
> Is there a problem with using the static key to check
> ARM64_HARDEN_EL2_VECTORS below? (Sorry, I'm still confused why we're not
> using the same functions for both.) 

Consider the following case: heterogeneous system with CPU0 that has
HBP, and CPU1 that has HEL. CPU0 is using its own vector slot, and CPU1
should be using an extra one (it doesn't have a dedicated slot).

When CPU0 claims its vectors, it will get the "normal" version of the
mapping. And then, if we're using cpus_have_cap, we turn that into an
out-of-line mapping. That's not completely wrong (the system still runs
fine), but that's not the expected behaviour. If we restrict the second
phase to this_cpu_has_cap(), we get the expected CPU0 using the normal
mapping, and CPU1 does the exact opposite (which we also want).

> If we can't use the static key
> below, then should we swap the check with !has_vhe() to ensure the
> compiler will avoid emitting the this_cpu_has_cap() call on VHE's fast
> path?

Fair enough, that's a good point.

> I also noticed that calling this_cpu_has_cap() puts a constraint on
> kvm_get_hyp_vector() that it must never be called without preemption
> disabled. It's not, but it's one more thing to think about. If there's
> no reason not to use the static key with cpus_have_const_cap() then
> maybe we should?

We're definitely in a non-preemptible section at this stage. On !VHE,
we're in IPI context (cpu_init_hyp_mode), and on VHE we are in the
switch code, having disabled interrupts a long time ago.

> 
>>
>>>
>>>> +		vect = __bp_harden_hyp_vecs_start;

Nit: massive buglet here, which I thought I had squashed in the series,
and obviously didn't (still have it as a fixup). It should read:

vect = kern_hyp_va(kvm_ksym_ref(__bp_harden_hyp_vecs_start));

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 12/26] KVM: arm/arm64: Move HYP IO VAs to the "idmap" range
  2018-03-14 16:50   ` Marc Zyngier
@ 2018-03-15 19:09     ` James Morse
  -1 siblings, 0 replies; 124+ messages in thread
From: James Morse @ 2018-03-15 19:09 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: Mark Rutland, Peter Maydell, Andrew Jones, Christoffer Dall, kvm,
	Steve Capper, Catalin Marinas, Will Deacon, Kristina Martsenko,
	kvmarm, linux-arm-kernel

Hi Marc,

On 14/03/18 16:50, Marc Zyngier wrote:
> We so far mapped our HYP IO (which is essentially the GICv2 control
> registers) using the same method as for memory. It recently appeared
> that is a bit unsafe:
> 
> We compute the HYP VA using the kern_hyp_va helper, but that helper
> is only designed to deal with kernel VAs coming from the linear map,
> and not from the vmalloc region... This could in turn cause some bad
> aliasing between the two, amplified by the upcoming VA randomisation.
> 
> A solution is to come up with our very own basic VA allocator for
> MMIO. Since half of the HYP address space only contains a single
> page (the idmap), we have plenty to borrow from. Let's use the idmap
> as a base, and allocate downwards from it. GICv2 now lives on the
> other side of the great VA barrier.

> diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
> index 9946f8a38b26..a9577ecc3e6a 100644
> --- a/virt/kvm/arm/mmu.c
> +++ b/virt/kvm/arm/mmu.c
> @@ -43,6 +43,9 @@ static unsigned long hyp_idmap_start;
>  static unsigned long hyp_idmap_end;
>  static phys_addr_t hyp_idmap_vector;
>  
> +static DEFINE_MUTEX(io_map_lock);
> +static unsigned long io_map_base;
> +
>  #define S2_PGD_SIZE	(PTRS_PER_S2_PGD * sizeof(pgd_t))
>  #define hyp_pgd_order get_order(PTRS_PER_PGD * sizeof(pgd_t))
>  
> @@ -518,27 +521,37 @@ static void unmap_hyp_idmap_range(pgd_t *pgdp, phys_addr_t start, u64 size)
>   *
>   * Assumes hyp_pgd is a page table used strictly in Hyp-mode and
>   * therefore contains either mappings in the kernel memory area (above
> - * PAGE_OFFSET), or device mappings in the vmalloc range (from
> - * VMALLOC_START to VMALLOC_END).
> + * PAGE_OFFSET), or device mappings in the idmap range.
>   *
> - * boot_hyp_pgd should only map two pages for the init code.
> + * boot_hyp_pgd should only map the idmap range, and is only used in
> + * the extended idmap case.
>   */
>  void free_hyp_pgds(void)
>  {
> +	pgd_t *id_pgd;
> +
>  	mutex_lock(&kvm_hyp_pgd_mutex);
>  
> +	id_pgd = boot_hyp_pgd ? boot_hyp_pgd : hyp_pgd;
> +
> +	if (id_pgd) {
> +		mutex_lock(&io_map_lock);

This takes kvm_hyp_pgd_mutex then io_map_lock ...


> +		/* In case we never called hyp_mmu_init() */
> +		if (!io_map_base)
> +			io_map_base = hyp_idmap_start;
> +		unmap_hyp_idmap_range(id_pgd, io_map_base,
> +				      hyp_idmap_start + PAGE_SIZE - io_map_base);
> +		mutex_unlock(&io_map_lock);
> +	}
> +
>  	if (boot_hyp_pgd) {
> -		unmap_hyp_idmap_range(boot_hyp_pgd, hyp_idmap_start, PAGE_SIZE);
>  		free_pages((unsigned long)boot_hyp_pgd, hyp_pgd_order);
>  		boot_hyp_pgd = NULL;
>  	}
>  
>  	if (hyp_pgd) {
> -		unmap_hyp_idmap_range(hyp_pgd, hyp_idmap_start, PAGE_SIZE);
>  		unmap_hyp_range(hyp_pgd, kern_hyp_va(PAGE_OFFSET),
>  				(uintptr_t)high_memory - PAGE_OFFSET);
> -		unmap_hyp_range(hyp_pgd, kern_hyp_va(VMALLOC_START),
> -				VMALLOC_END - VMALLOC_START);
>  
>  		free_pages((unsigned long)hyp_pgd, hyp_pgd_order);
>  		hyp_pgd = NULL;

> @@ -747,11 +761,43 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
>  		return 0;
>  	}

> +	mutex_lock(&io_map_lock);

> -	start = kern_hyp_va((unsigned long)*kaddr);
> -	end = kern_hyp_va((unsigned long)*kaddr + size);
> -	ret = __create_hyp_mappings(hyp_pgd, PTRS_PER_PGD, start, end,
> -				     __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
> +	/*
> +	 * This assumes that we we have enough space below the idmap
> +	 * page to allocate our VAs. If not, the check below will
> +	 * kick. A potential alternative would be to detect that
> +	 * overflow and switch to an allocation above the idmap.
> +	 *
> +	 * The allocated size is always a multiple of PAGE_SIZE.
> +	 */
> +	size = PAGE_ALIGN(size + offset_in_page(phys_addr));
> +	base = io_map_base - size;
> +
> +	/*
> +	 * Verify that BIT(VA_BITS - 1) hasn't been flipped by
> +	 * allocating the new area, as it would indicate we've
> +	 * overflowed the idmap/IO address range.
> +	 */
> +	if ((base ^ io_map_base) & BIT(VA_BITS - 1)) {
> +		ret = -ENOMEM;
> +		goto out;
> +	}
> +
> +	if (__kvm_cpu_uses_extended_idmap())
> +		pgd = boot_hyp_pgd;
> +
> +	ret = __create_hyp_mappings(pgd, __kvm_idmap_ptrs_per_pgd(),
> +				    base, base + size,
> +				    __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);

And here we have io_map_lock, but __create_hyp_mappings takes kvm_hyp_pgd_mutex.

There is another example of this in create_hyp_exec_mappings, I think making
this the required order makes sense: if you are mapping to the KVM-IO region,
you take the io_map_lock before taking the pgd lock to write in your allocated
location. (free_hyp_pgds() is always going to need to take both).

Never going to happen, but it generates a lockdep splat[1].
Fixup diff[0] below fixes it for me.


> +	if (ret)
> +		goto out;
> +
> +	*haddr = (void __iomem *)base + offset_in_page(phys_addr);
> +	io_map_base = base;
> +
> +out:
> +	mutex_unlock(&io_map_lock);
>  
>  	if (ret) {
>  		iounmap(*kaddr);



Thanks,

James


[0]
--------------------------%<--------------------------
diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index 554ad5493e7d..f7642c96fc56 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -43,6 +43,7 @@ static unsigned long hyp_idmap_start;
 static unsigned long hyp_idmap_end;
 static phys_addr_t hyp_idmap_vector;

+/* Take io_map_lock before kvm_hyp_pgd_mutex */
 static DEFINE_MUTEX(io_map_lock);
 static unsigned long io_map_base;

@@ -530,18 +531,17 @@ void free_hyp_pgds(void)
 {
        pgd_t *id_pgd;

+       mutex_lock(&io_map_lock);
        mutex_lock(&kvm_hyp_pgd_mutex);

        id_pgd = boot_hyp_pgd ? boot_hyp_pgd : hyp_pgd;

        if (id_pgd) {
-               mutex_lock(&io_map_lock);
                /* In case we never called hyp_mmu_init() */
                if (!io_map_base)
                        io_map_base = hyp_idmap_start;
                unmap_hyp_idmap_range(id_pgd, io_map_base,
                                      hyp_idmap_start + PAGE_SIZE - io_map_base);
-               mutex_unlock(&io_map_lock);
        }

        if (boot_hyp_pgd) {
@@ -563,6 +563,7 @@ void free_hyp_pgds(void)
        }

        mutex_unlock(&kvm_hyp_pgd_mutex);
+       mutex_unlock(&io_map_lock);
 }

 static void create_hyp_pte_mappings(pmd_t *pmd, unsigned long start,
--------------------------%<--------------------------
[1]
[    2.795711] kvm [1]: 8-bit VMID
[    2.800456]
[    2.801944] ======================================================
[    2.808086] WARNING: possible circular locking dependency detected
[    2.814230] 4.16.0-rc5-00027-gca4a12e92d2d-dirty #9471 Not tainted
[    2.820371] ------------------------------------------------------
[    2.826513] swapper/0/1 is trying to acquire lock:
[    2.831274]  (io_map_lock){+.+.}, at: [<00000000cf644f15>] free_hyp_pgds+0x44
/0x148
[    2.838914]
[    2.838914] but task is already holding lock:
[    2.844713]  (kvm_hyp_pgd_mutex){+.+.}, at: [<00000000e786face>] free_hyp_pgd
s+0x20/0x148
[    2.852863]
[    2.852863] which lock already depends on the new lock.
[    2.852863]
[    2.860995]
[    2.860995] the existing dependency chain (in reverse order) is:
[    2.868433]
[    2.868433] -> #1 (kvm_hyp_pgd_mutex){+.+.}:
[    2.874174]        __mutex_lock+0x70/0x8d0
[    2.878249]        mutex_lock_nested+0x1c/0x28
[    2.882671]        __create_hyp_mappings+0x48/0x3e0
[    2.887523]        __create_hyp_private_mapping+0xa4/0xf8
[    2.892893]        create_hyp_exec_mappings+0x28/0x58
[    2.897918]        kvm_arch_init+0x524/0x6e8
[    2.902167]        kvm_init+0x20/0x2d8
[    2.905897]        arm_init+0x1c/0x28
[    2.909542]        do_one_initcall+0x38/0x128
[    2.913884]        kernel_init_freeable+0x1e0/0x284
[    2.918737]        kernel_init+0x10/0x100
[    2.922726]        ret_from_fork+0x10/0x18
[    2.926797]
[    2.926797] -> #0 (io_map_lock){+.+.}:
[    2.932020]        lock_acquire+0x6c/0xb0
[    2.936008]        __mutex_lock+0x70/0x8d0
[    2.940082]        mutex_lock_nested+0x1c/0x28
[    2.944502]        free_hyp_pgds+0x44/0x148
[    2.948663]        teardown_hyp_mode+0x34/0x90
[    2.953084]        kvm_arch_init+0x3b8/0x6e8
[    2.957332]        kvm_init+0x20/0x2d8
[    2.961061]        arm_init+0x1c/0x28
[    2.964704]        do_one_initcall+0x38/0x128
[    2.969039]        kernel_init_freeable+0x1e0/0x284
[    2.973891]        kernel_init+0x10/0x100
[    2.977880]        ret_from_fork+0x10/0x18
[    2.981950]
[    2.981950] other info that might help us debug this:
[    2.981950]
[    2.989910]  Possible unsafe locking scenario:
[    2.989910]
[    2.995796]        CPU0                    CPU1
[    3.000297]        ----                    ----
[    3.004798]   lock(kvm_hyp_pgd_mutex);
[    3.008533]                                lock(io_map_lock);
[    3.014252]                                lock(kvm_hyp_pgd_mutex);
[    3.020488]   lock(io_map_lock);
[    3.023705]
[    3.023705]  *** DEADLOCK ***
[    3.023705]
[    3.029597] 1 lock held by swapper/0/1:
[    3.033409]  #0:  (kvm_hyp_pgd_mutex){+.+.}, at: [<00000000e786face>] free_hy
p_pgds+0x20/0x148
[    3.041993]
[    3.041993] stack backtrace:
[    3.046331] CPU: 4 PID: 1 Comm: swapper/0 Not tainted 4.16.0-rc5-00027-gca4a1
2e92d2d-dirty #9471
[    3.055062] Hardware name: ARM Juno development board (r1) (DT)
[    3.060945] Call trace:
[    3.063383]  dump_backtrace+0x0/0x190
[    3.067027]  show_stack+0x14/0x20
[    3.070326]  dump_stack+0xac/0xe8
[    3.073626]  print_circular_bug.isra.19+0x1d4/0x2e0
[    3.078476]  __lock_acquire+0x19f4/0x1a50
[    3.082464]  lock_acquire+0x6c/0xb0
[    3.085934]  __mutex_lock+0x70/0x8d0
[    3.089491]  mutex_lock_nested+0x1c/0x28
[    3.093394]  free_hyp_pgds+0x44/0x148
[    3.097037]  teardown_hyp_mode+0x34/0x90
[    3.100940]  kvm_arch_init+0x3b8/0x6e8
[    3.104670]  kvm_init+0x20/0x2d8
[    3.107882]  arm_init+0x1c/0x28
[    3.111007]  do_one_initcall+0x38/0x128
[    3.114823]  kernel_init_freeable+0x1e0/0x284
[    3.119158]  kernel_init+0x10/0x100
[    3.122628]  ret_from_fork+0x10/0x18

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* [PATCH v6 12/26] KVM: arm/arm64: Move HYP IO VAs to the "idmap" range
@ 2018-03-15 19:09     ` James Morse
  0 siblings, 0 replies; 124+ messages in thread
From: James Morse @ 2018-03-15 19:09 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Marc,

On 14/03/18 16:50, Marc Zyngier wrote:
> We so far mapped our HYP IO (which is essentially the GICv2 control
> registers) using the same method as for memory. It recently appeared
> that is a bit unsafe:
> 
> We compute the HYP VA using the kern_hyp_va helper, but that helper
> is only designed to deal with kernel VAs coming from the linear map,
> and not from the vmalloc region... This could in turn cause some bad
> aliasing between the two, amplified by the upcoming VA randomisation.
> 
> A solution is to come up with our very own basic VA allocator for
> MMIO. Since half of the HYP address space only contains a single
> page (the idmap), we have plenty to borrow from. Let's use the idmap
> as a base, and allocate downwards from it. GICv2 now lives on the
> other side of the great VA barrier.

> diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
> index 9946f8a38b26..a9577ecc3e6a 100644
> --- a/virt/kvm/arm/mmu.c
> +++ b/virt/kvm/arm/mmu.c
> @@ -43,6 +43,9 @@ static unsigned long hyp_idmap_start;
>  static unsigned long hyp_idmap_end;
>  static phys_addr_t hyp_idmap_vector;
>  
> +static DEFINE_MUTEX(io_map_lock);
> +static unsigned long io_map_base;
> +
>  #define S2_PGD_SIZE	(PTRS_PER_S2_PGD * sizeof(pgd_t))
>  #define hyp_pgd_order get_order(PTRS_PER_PGD * sizeof(pgd_t))
>  
> @@ -518,27 +521,37 @@ static void unmap_hyp_idmap_range(pgd_t *pgdp, phys_addr_t start, u64 size)
>   *
>   * Assumes hyp_pgd is a page table used strictly in Hyp-mode and
>   * therefore contains either mappings in the kernel memory area (above
> - * PAGE_OFFSET), or device mappings in the vmalloc range (from
> - * VMALLOC_START to VMALLOC_END).
> + * PAGE_OFFSET), or device mappings in the idmap range.
>   *
> - * boot_hyp_pgd should only map two pages for the init code.
> + * boot_hyp_pgd should only map the idmap range, and is only used in
> + * the extended idmap case.
>   */
>  void free_hyp_pgds(void)
>  {
> +	pgd_t *id_pgd;
> +
>  	mutex_lock(&kvm_hyp_pgd_mutex);
>  
> +	id_pgd = boot_hyp_pgd ? boot_hyp_pgd : hyp_pgd;
> +
> +	if (id_pgd) {
> +		mutex_lock(&io_map_lock);

This takes kvm_hyp_pgd_mutex then io_map_lock ...


> +		/* In case we never called hyp_mmu_init() */
> +		if (!io_map_base)
> +			io_map_base = hyp_idmap_start;
> +		unmap_hyp_idmap_range(id_pgd, io_map_base,
> +				      hyp_idmap_start + PAGE_SIZE - io_map_base);
> +		mutex_unlock(&io_map_lock);
> +	}
> +
>  	if (boot_hyp_pgd) {
> -		unmap_hyp_idmap_range(boot_hyp_pgd, hyp_idmap_start, PAGE_SIZE);
>  		free_pages((unsigned long)boot_hyp_pgd, hyp_pgd_order);
>  		boot_hyp_pgd = NULL;
>  	}
>  
>  	if (hyp_pgd) {
> -		unmap_hyp_idmap_range(hyp_pgd, hyp_idmap_start, PAGE_SIZE);
>  		unmap_hyp_range(hyp_pgd, kern_hyp_va(PAGE_OFFSET),
>  				(uintptr_t)high_memory - PAGE_OFFSET);
> -		unmap_hyp_range(hyp_pgd, kern_hyp_va(VMALLOC_START),
> -				VMALLOC_END - VMALLOC_START);
>  
>  		free_pages((unsigned long)hyp_pgd, hyp_pgd_order);
>  		hyp_pgd = NULL;

> @@ -747,11 +761,43 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
>  		return 0;
>  	}

> +	mutex_lock(&io_map_lock);

> -	start = kern_hyp_va((unsigned long)*kaddr);
> -	end = kern_hyp_va((unsigned long)*kaddr + size);
> -	ret = __create_hyp_mappings(hyp_pgd, PTRS_PER_PGD, start, end,
> -				     __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
> +	/*
> +	 * This assumes that we we have enough space below the idmap
> +	 * page to allocate our VAs. If not, the check below will
> +	 * kick. A potential alternative would be to detect that
> +	 * overflow and switch to an allocation above the idmap.
> +	 *
> +	 * The allocated size is always a multiple of PAGE_SIZE.
> +	 */
> +	size = PAGE_ALIGN(size + offset_in_page(phys_addr));
> +	base = io_map_base - size;
> +
> +	/*
> +	 * Verify that BIT(VA_BITS - 1) hasn't been flipped by
> +	 * allocating the new area, as it would indicate we've
> +	 * overflowed the idmap/IO address range.
> +	 */
> +	if ((base ^ io_map_base) & BIT(VA_BITS - 1)) {
> +		ret = -ENOMEM;
> +		goto out;
> +	}
> +
> +	if (__kvm_cpu_uses_extended_idmap())
> +		pgd = boot_hyp_pgd;
> +
> +	ret = __create_hyp_mappings(pgd, __kvm_idmap_ptrs_per_pgd(),
> +				    base, base + size,
> +				    __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);

And here we have io_map_lock, but __create_hyp_mappings takes kvm_hyp_pgd_mutex.

There is another example of this in create_hyp_exec_mappings, I think making
this the required order makes sense: if you are mapping to the KVM-IO region,
you take the io_map_lock before taking the pgd lock to write in your allocated
location. (free_hyp_pgds() is always going to need to take both).

Never going to happen, but it generates a lockdep splat[1].
Fixup diff[0] below fixes it for me.


> +	if (ret)
> +		goto out;
> +
> +	*haddr = (void __iomem *)base + offset_in_page(phys_addr);
> +	io_map_base = base;
> +
> +out:
> +	mutex_unlock(&io_map_lock);
>  
>  	if (ret) {
>  		iounmap(*kaddr);



Thanks,

James


[0]
--------------------------%<--------------------------
diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
index 554ad5493e7d..f7642c96fc56 100644
--- a/virt/kvm/arm/mmu.c
+++ b/virt/kvm/arm/mmu.c
@@ -43,6 +43,7 @@ static unsigned long hyp_idmap_start;
 static unsigned long hyp_idmap_end;
 static phys_addr_t hyp_idmap_vector;

+/* Take io_map_lock before kvm_hyp_pgd_mutex */
 static DEFINE_MUTEX(io_map_lock);
 static unsigned long io_map_base;

@@ -530,18 +531,17 @@ void free_hyp_pgds(void)
 {
        pgd_t *id_pgd;

+       mutex_lock(&io_map_lock);
        mutex_lock(&kvm_hyp_pgd_mutex);

        id_pgd = boot_hyp_pgd ? boot_hyp_pgd : hyp_pgd;

        if (id_pgd) {
-               mutex_lock(&io_map_lock);
                /* In case we never called hyp_mmu_init() */
                if (!io_map_base)
                        io_map_base = hyp_idmap_start;
                unmap_hyp_idmap_range(id_pgd, io_map_base,
                                      hyp_idmap_start + PAGE_SIZE - io_map_base);
-               mutex_unlock(&io_map_lock);
        }

        if (boot_hyp_pgd) {
@@ -563,6 +563,7 @@ void free_hyp_pgds(void)
        }

        mutex_unlock(&kvm_hyp_pgd_mutex);
+       mutex_unlock(&io_map_lock);
 }

 static void create_hyp_pte_mappings(pmd_t *pmd, unsigned long start,
--------------------------%<--------------------------
[1]
[    2.795711] kvm [1]: 8-bit VMID
[    2.800456]
[    2.801944] ======================================================
[    2.808086] WARNING: possible circular locking dependency detected
[    2.814230] 4.16.0-rc5-00027-gca4a12e92d2d-dirty #9471 Not tainted
[    2.820371] ------------------------------------------------------
[    2.826513] swapper/0/1 is trying to acquire lock:
[    2.831274]  (io_map_lock){+.+.}, at: [<00000000cf644f15>] free_hyp_pgds+0x44
/0x148
[    2.838914]
[    2.838914] but task is already holding lock:
[    2.844713]  (kvm_hyp_pgd_mutex){+.+.}, at: [<00000000e786face>] free_hyp_pgd
s+0x20/0x148
[    2.852863]
[    2.852863] which lock already depends on the new lock.
[    2.852863]
[    2.860995]
[    2.860995] the existing dependency chain (in reverse order) is:
[    2.868433]
[    2.868433] -> #1 (kvm_hyp_pgd_mutex){+.+.}:
[    2.874174]        __mutex_lock+0x70/0x8d0
[    2.878249]        mutex_lock_nested+0x1c/0x28
[    2.882671]        __create_hyp_mappings+0x48/0x3e0
[    2.887523]        __create_hyp_private_mapping+0xa4/0xf8
[    2.892893]        create_hyp_exec_mappings+0x28/0x58
[    2.897918]        kvm_arch_init+0x524/0x6e8
[    2.902167]        kvm_init+0x20/0x2d8
[    2.905897]        arm_init+0x1c/0x28
[    2.909542]        do_one_initcall+0x38/0x128
[    2.913884]        kernel_init_freeable+0x1e0/0x284
[    2.918737]        kernel_init+0x10/0x100
[    2.922726]        ret_from_fork+0x10/0x18
[    2.926797]
[    2.926797] -> #0 (io_map_lock){+.+.}:
[    2.932020]        lock_acquire+0x6c/0xb0
[    2.936008]        __mutex_lock+0x70/0x8d0
[    2.940082]        mutex_lock_nested+0x1c/0x28
[    2.944502]        free_hyp_pgds+0x44/0x148
[    2.948663]        teardown_hyp_mode+0x34/0x90
[    2.953084]        kvm_arch_init+0x3b8/0x6e8
[    2.957332]        kvm_init+0x20/0x2d8
[    2.961061]        arm_init+0x1c/0x28
[    2.964704]        do_one_initcall+0x38/0x128
[    2.969039]        kernel_init_freeable+0x1e0/0x284
[    2.973891]        kernel_init+0x10/0x100
[    2.977880]        ret_from_fork+0x10/0x18
[    2.981950]
[    2.981950] other info that might help us debug this:
[    2.981950]
[    2.989910]  Possible unsafe locking scenario:
[    2.989910]
[    2.995796]        CPU0                    CPU1
[    3.000297]        ----                    ----
[    3.004798]   lock(kvm_hyp_pgd_mutex);
[    3.008533]                                lock(io_map_lock);
[    3.014252]                                lock(kvm_hyp_pgd_mutex);
[    3.020488]   lock(io_map_lock);
[    3.023705]
[    3.023705]  *** DEADLOCK ***
[    3.023705]
[    3.029597] 1 lock held by swapper/0/1:
[    3.033409]  #0:  (kvm_hyp_pgd_mutex){+.+.}, at: [<00000000e786face>] free_hy
p_pgds+0x20/0x148
[    3.041993]
[    3.041993] stack backtrace:
[    3.046331] CPU: 4 PID: 1 Comm: swapper/0 Not tainted 4.16.0-rc5-00027-gca4a1
2e92d2d-dirty #9471
[    3.055062] Hardware name: ARM Juno development board (r1) (DT)
[    3.060945] Call trace:
[    3.063383]  dump_backtrace+0x0/0x190
[    3.067027]  show_stack+0x14/0x20
[    3.070326]  dump_stack+0xac/0xe8
[    3.073626]  print_circular_bug.isra.19+0x1d4/0x2e0
[    3.078476]  __lock_acquire+0x19f4/0x1a50
[    3.082464]  lock_acquire+0x6c/0xb0
[    3.085934]  __mutex_lock+0x70/0x8d0
[    3.089491]  mutex_lock_nested+0x1c/0x28
[    3.093394]  free_hyp_pgds+0x44/0x148
[    3.097037]  teardown_hyp_mode+0x34/0x90
[    3.100940]  kvm_arch_init+0x3b8/0x6e8
[    3.104670]  kvm_init+0x20/0x2d8
[    3.107882]  arm_init+0x1c/0x28
[    3.111007]  do_one_initcall+0x38/0x128
[    3.114823]  kernel_init_freeable+0x1e0/0x284
[    3.119158]  kernel_init+0x10/0x100
[    3.122628]  ret_from_fork+0x10/0x18

^ permalink raw reply related	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 04/26] arm64: KVM: Dynamically patch the kernel/hyp VA mask
  2018-03-14 16:50   ` Marc Zyngier
@ 2018-03-15 19:15     ` James Morse
  -1 siblings, 0 replies; 124+ messages in thread
From: James Morse @ 2018-03-15 19:15 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: Mark Rutland, Peter Maydell, Andrew Jones, Christoffer Dall, kvm,
	Steve Capper, Catalin Marinas, Will Deacon, Kristina Martsenko,
	kvmarm, linux-arm-kernel

Hi Marc,

On 14/03/18 16:50, Marc Zyngier wrote:
> So far, we're using a complicated sequence of alternatives to
> patch the kernel/hyp VA mask on non-VHE, and NOP out the
> masking altogether when on VHE.
> 
> The newly introduced dynamic patching gives us the opportunity
> to simplify that code by patching a single instruction with
> the correct mask (instead of the mind bending cummulative masking

(Nit: cumulative)

(so this series removes mind bending code?)

> we have at the moment) or even a single NOP on VHE. This also
> adds some initial code that will allow the patching callback
> to switch to a more complex patching.

> diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c
> new file mode 100644
> index 000000000000..45e7802328d4
> --- /dev/null
> +++ b/arch/arm64/kvm/va_layout.c

> +void __init kvm_update_va_mask(struct alt_instr *alt,
> +			       __le32 *origptr, __le32 *updptr, int nr_inst)
> +{
> +	int i;
> +
> +	/* We only expect a single instruction in the alternative sequence */
> +	BUG_ON(nr_inst != 1);
> +
> +	if (!has_vhe() && !va_mask)
> +		compute_layout();
> +
> +	for (i = 0; i < nr_inst; i++) {
> +		u32 rd, rn, insn, oinsn;
> +
> +		/*
> +		 * VHE doesn't need any address translation, let's NOP
> +		 * everything.
> +		 */
> +		if (has_vhe()) {
> +			updptr[i] = aarch64_insn_gen_nop();

cpu_to_le32()? (I'm not going to try an boot a BE VHE model...)

aarch64_insn_gen_nop() returns:
| aarch64_insn_get_hint_value() | AARCH64_INSN_HINT_NOP;

It doesn't look like these aarch64_insn_get_XXX_value() helpers are forcing a
particular endianness. ftrace uses this, via ftrace_modify_code() ->
aarch64_insn_patch_text_nosync() -> aarch64_insn_write(), which does:
| return  __aarch64_insn_write(addr, cpu_to_le32(insn));

So it looks like the conversion is required. Patch 16 looks fine for this.

(and, I ran the teardown code on Juno big-endian...)


> +			continue;
> +		}
> +
> +		oinsn = le32_to_cpu(origptr[i]);
> +		rd = aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RD, oinsn);
> +		rn = aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RN, oinsn);
> +
> +		insn = compute_instruction(i, rd, rn);
> +		BUG_ON(insn == AARCH64_BREAK_FAULT);
> +
> +		updptr[i] = cpu_to_le32(insn);
> +	}
> +}

With that,

Reviewed-by: James Morse <james.morse@arm.com>


Thanks,

James

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 04/26] arm64: KVM: Dynamically patch the kernel/hyp VA mask
@ 2018-03-15 19:15     ` James Morse
  0 siblings, 0 replies; 124+ messages in thread
From: James Morse @ 2018-03-15 19:15 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Marc,

On 14/03/18 16:50, Marc Zyngier wrote:
> So far, we're using a complicated sequence of alternatives to
> patch the kernel/hyp VA mask on non-VHE, and NOP out the
> masking altogether when on VHE.
> 
> The newly introduced dynamic patching gives us the opportunity
> to simplify that code by patching a single instruction with
> the correct mask (instead of the mind bending cummulative masking

(Nit: cumulative)

(so this series removes mind bending code?)

> we have at the moment) or even a single NOP on VHE. This also
> adds some initial code that will allow the patching callback
> to switch to a more complex patching.

> diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c
> new file mode 100644
> index 000000000000..45e7802328d4
> --- /dev/null
> +++ b/arch/arm64/kvm/va_layout.c

> +void __init kvm_update_va_mask(struct alt_instr *alt,
> +			       __le32 *origptr, __le32 *updptr, int nr_inst)
> +{
> +	int i;
> +
> +	/* We only expect a single instruction in the alternative sequence */
> +	BUG_ON(nr_inst != 1);
> +
> +	if (!has_vhe() && !va_mask)
> +		compute_layout();
> +
> +	for (i = 0; i < nr_inst; i++) {
> +		u32 rd, rn, insn, oinsn;
> +
> +		/*
> +		 * VHE doesn't need any address translation, let's NOP
> +		 * everything.
> +		 */
> +		if (has_vhe()) {
> +			updptr[i] = aarch64_insn_gen_nop();

cpu_to_le32()? (I'm not going to try an boot a BE VHE model...)

aarch64_insn_gen_nop() returns:
| aarch64_insn_get_hint_value() | AARCH64_INSN_HINT_NOP;

It doesn't look like these aarch64_insn_get_XXX_value() helpers are forcing a
particular endianness. ftrace uses this, via ftrace_modify_code() ->
aarch64_insn_patch_text_nosync() -> aarch64_insn_write(), which does:
| return  __aarch64_insn_write(addr, cpu_to_le32(insn));

So it looks like the conversion is required. Patch 16 looks fine for this.

(and, I ran the teardown code on Juno big-endian...)


> +			continue;
> +		}
> +
> +		oinsn = le32_to_cpu(origptr[i]);
> +		rd = aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RD, oinsn);
> +		rn = aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RN, oinsn);
> +
> +		insn = compute_instruction(i, rd, rn);
> +		BUG_ON(insn == AARCH64_BREAK_FAULT);
> +
> +		updptr[i] = cpu_to_le32(insn);
> +	}
> +}

With that,

Reviewed-by: James Morse <james.morse@arm.com>


Thanks,

James

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 10/26] KVM: arm/arm64: Fix idmap size and alignment
  2018-03-14 16:50   ` Marc Zyngier
@ 2018-03-15 19:15     ` James Morse
  -1 siblings, 0 replies; 124+ messages in thread
From: James Morse @ 2018-03-15 19:15 UTC (permalink / raw)
  To: Marc Zyngier, kvmarm
  Cc: Mark Rutland, Peter Maydell, Andrew Jones, Christoffer Dall, kvm,
	Steve Capper, Catalin Marinas, Will Deacon, Kristina Martsenko,
	linux-arm-kernel

On 14/03/18 16:50, Marc Zyngier wrote:
> Although the idmap section of KVM can only be at most 4kB and
> must be aligned on a 4kB boundary, the rest of the code expects
> it to be page aligned. Things get messy when tearing down the
> HYP page tables when PAGE_SIZE is 64K, and the idmap section isn't
> 64K aligned.
> 
> Let's fix this by computing aligned boundaries that the HYP code
> will use.

> diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
> index 0e5cfffb4c21..a9e0513868e9 100644
> --- a/virt/kvm/arm/mmu.c
> +++ b/virt/kvm/arm/mmu.c
> @@ -1815,7 +1815,9 @@ int kvm_mmu_init(void)
>  	int err;
>  
>  	hyp_idmap_start = kvm_virt_to_phys(__hyp_idmap_text_start);
> +	hyp_idmap_start = ALIGN_DOWN(hyp_idmap_start, PAGE_SIZE);
>  	hyp_idmap_end = kvm_virt_to_phys(__hyp_idmap_text_end);
> +	hyp_idmap_end = ALIGN(hyp_idmap_end, PAGE_SIZE);
>  	hyp_idmap_vector = kvm_virt_to_phys(__kvm_hyp_init);
>  
>  	/*


This makes the:
| hyp_idmap_start != (unsigned long)__hyp_idmap_text_start

check below look funny, but that must be for 32bit which only has 4K pages (I
think), so its behaviour hasn't changed.

Reviewed-by: James Morse <james.morse@arm.com>


Thanks,

James

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 10/26] KVM: arm/arm64: Fix idmap size and alignment
@ 2018-03-15 19:15     ` James Morse
  0 siblings, 0 replies; 124+ messages in thread
From: James Morse @ 2018-03-15 19:15 UTC (permalink / raw)
  To: linux-arm-kernel

On 14/03/18 16:50, Marc Zyngier wrote:
> Although the idmap section of KVM can only be at most 4kB and
> must be aligned on a 4kB boundary, the rest of the code expects
> it to be page aligned. Things get messy when tearing down the
> HYP page tables when PAGE_SIZE is 64K, and the idmap section isn't
> 64K aligned.
> 
> Let's fix this by computing aligned boundaries that the HYP code
> will use.

> diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
> index 0e5cfffb4c21..a9e0513868e9 100644
> --- a/virt/kvm/arm/mmu.c
> +++ b/virt/kvm/arm/mmu.c
> @@ -1815,7 +1815,9 @@ int kvm_mmu_init(void)
>  	int err;
>  
>  	hyp_idmap_start = kvm_virt_to_phys(__hyp_idmap_text_start);
> +	hyp_idmap_start = ALIGN_DOWN(hyp_idmap_start, PAGE_SIZE);
>  	hyp_idmap_end = kvm_virt_to_phys(__hyp_idmap_text_end);
> +	hyp_idmap_end = ALIGN(hyp_idmap_end, PAGE_SIZE);
>  	hyp_idmap_vector = kvm_virt_to_phys(__kvm_hyp_init);
>  
>  	/*


This makes the:
| hyp_idmap_start != (unsigned long)__hyp_idmap_text_start

check below look funny, but that must be for 32bit which only has 4K pages (I
think), so its behaviour hasn't changed.

Reviewed-by: James Morse <james.morse@arm.com>


Thanks,

James

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 06/26] KVM: arm/arm64: Do not use kern_hyp_va() with kvm_vgic_global_state
  2018-03-14 16:50   ` Marc Zyngier
@ 2018-03-15 19:16     ` James Morse
  -1 siblings, 0 replies; 124+ messages in thread
From: James Morse @ 2018-03-15 19:16 UTC (permalink / raw)
  To: Marc Zyngier, kvmarm
  Cc: Mark Rutland, Peter Maydell, Andrew Jones, Christoffer Dall, kvm,
	Steve Capper, Catalin Marinas, Will Deacon, Kristina Martsenko,
	linux-arm-kernel

Hi Marc,

On 14/03/18 16:50, Marc Zyngier wrote:
> kvm_vgic_global_state is part of the read-only section, and is
> usually accessed using a PC-relative address generation (adrp + add).
> 
> It is thus useless to use kern_hyp_va() on it, and actively problematic
> if kern_hyp_va() becomes non-idempotent. On the other hand, there is
> no way that the compiler is going to guarantee that such access is
> always PC relative.
> 
> So let's bite the bullet and provide our own accessor.


> diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
> index de1b919404e4..a6808d2869f5 100644
> --- a/arch/arm/include/asm/kvm_mmu.h
> +++ b/arch/arm/include/asm/kvm_mmu.h
> @@ -28,6 +28,13 @@
>   */
>  #define kern_hyp_va(kva)	(kva)
>  
> +/* Resolving symbol addresses in a PC-relative way is easy... */

(So this is guaranteed on 32bit? I thought this was because 32bit uses the
kernel-VA's at HYP, so any way the compiler generates the address will work.)


> +#define hyp_symbol_addr(s)						\
> +	({								\
> +		typeof(s) *addr = &(s);					\
> +		addr;							\
> +	})
> +
>  /*
>   * KVM_MMU_CACHE_MIN_PAGES is the number of stage2 page table translation levels.
>   */

> diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
> index e3bc1d0a5e93..7120bf3f22c7 100644
> --- a/arch/arm64/include/asm/kvm_mmu.h
> +++ b/arch/arm64/include/asm/kvm_mmu.h
> @@ -110,6 +110,26 @@ static inline unsigned long __kern_hyp_va(unsigned long v)
>  
>  #define kern_hyp_va(v) 	((typeof(v))(__kern_hyp_va((unsigned long)(v))))
>  
> +/*
> + * Obtain the PC-relative address of a kernel symbol
> + * s: symbol
> + *
> + * The goal of this macro is to return a symbol's address based on a
> + * PC-relative computation, as opposed to a loading the VA from a
> + * constant pool or something similar. This works well for HYP, as an
> + * absolute VA is guaranteed to be wrong. Only use this if trying to
> + * obtain the address of a symbol (i.e. not something you obtained by
> + * following a pointer).
> + */
> +#define hyp_symbol_addr(s)						\
> +	({								\
> +		typeof(s) *addr;					\
> +		asm volatile("adrp	%0, %1\n"			\
> +			     "add	%0, %0, :lo12:%1\n"		\
> +			     : "=r" (addr) : "S" (&s));			\
> +		addr;							\
> +	})

Why does this need to be volatile? I see gcc v4.9 generate this sequence twice
in __vgic_v2_save_state(). Can't it cache and reorder this sequence as it likes?

Either-way:
Reviewed-by: James Morse <james.morse@arm.com>


(I had a go at using 'Ush', to let the compiler schedule the adrp, but couldn't
get it to work.)

Thanks,

James

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 06/26] KVM: arm/arm64: Do not use kern_hyp_va() with kvm_vgic_global_state
@ 2018-03-15 19:16     ` James Morse
  0 siblings, 0 replies; 124+ messages in thread
From: James Morse @ 2018-03-15 19:16 UTC (permalink / raw)
  To: linux-arm-kernel

Hi Marc,

On 14/03/18 16:50, Marc Zyngier wrote:
> kvm_vgic_global_state is part of the read-only section, and is
> usually accessed using a PC-relative address generation (adrp + add).
> 
> It is thus useless to use kern_hyp_va() on it, and actively problematic
> if kern_hyp_va() becomes non-idempotent. On the other hand, there is
> no way that the compiler is going to guarantee that such access is
> always PC relative.
> 
> So let's bite the bullet and provide our own accessor.


> diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
> index de1b919404e4..a6808d2869f5 100644
> --- a/arch/arm/include/asm/kvm_mmu.h
> +++ b/arch/arm/include/asm/kvm_mmu.h
> @@ -28,6 +28,13 @@
>   */
>  #define kern_hyp_va(kva)	(kva)
>  
> +/* Resolving symbol addresses in a PC-relative way is easy... */

(So this is guaranteed on 32bit? I thought this was because 32bit uses the
kernel-VA's at HYP, so any way the compiler generates the address will work.)


> +#define hyp_symbol_addr(s)						\
> +	({								\
> +		typeof(s) *addr = &(s);					\
> +		addr;							\
> +	})
> +
>  /*
>   * KVM_MMU_CACHE_MIN_PAGES is the number of stage2 page table translation levels.
>   */

> diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
> index e3bc1d0a5e93..7120bf3f22c7 100644
> --- a/arch/arm64/include/asm/kvm_mmu.h
> +++ b/arch/arm64/include/asm/kvm_mmu.h
> @@ -110,6 +110,26 @@ static inline unsigned long __kern_hyp_va(unsigned long v)
>  
>  #define kern_hyp_va(v) 	((typeof(v))(__kern_hyp_va((unsigned long)(v))))
>  
> +/*
> + * Obtain the PC-relative address of a kernel symbol
> + * s: symbol
> + *
> + * The goal of this macro is to return a symbol's address based on a
> + * PC-relative computation, as opposed to a loading the VA from a
> + * constant pool or something similar. This works well for HYP, as an
> + * absolute VA is guaranteed to be wrong. Only use this if trying to
> + * obtain the address of a symbol (i.e. not something you obtained by
> + * following a pointer).
> + */
> +#define hyp_symbol_addr(s)						\
> +	({								\
> +		typeof(s) *addr;					\
> +		asm volatile("adrp	%0, %1\n"			\
> +			     "add	%0, %0, :lo12:%1\n"		\
> +			     : "=r" (addr) : "S" (&s));			\
> +		addr;							\
> +	})

Why does this need to be volatile? I see gcc v4.9 generate this sequence twice
in __vgic_v2_save_state(). Can't it cache and reorder this sequence as it likes?

Either-way:
Reviewed-by: James Morse <james.morse@arm.com>


(I had a go at using 'Ush', to let the compiler schedule the adrp, but couldn't
get it to work.)

Thanks,

James

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 12/26] KVM: arm/arm64: Move HYP IO VAs to the "idmap" range
  2018-03-15 19:09     ` James Morse
@ 2018-03-16  8:44       ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-16  8:44 UTC (permalink / raw)
  To: James Morse
  Cc: Mark Rutland, Peter Maydell, Andrew Jones, Christoffer Dall, kvm,
	Steve Capper, Catalin Marinas, Will Deacon, Kristina Martsenko,
	kvmarm, linux-arm-kernel

Hi James,

On 15/03/18 19:09, James Morse wrote:
> Hi Marc,
> 
> On 14/03/18 16:50, Marc Zyngier wrote:
>> We so far mapped our HYP IO (which is essentially the GICv2 control
>> registers) using the same method as for memory. It recently appeared
>> that is a bit unsafe:
>>
>> We compute the HYP VA using the kern_hyp_va helper, but that helper
>> is only designed to deal with kernel VAs coming from the linear map,
>> and not from the vmalloc region... This could in turn cause some bad
>> aliasing between the two, amplified by the upcoming VA randomisation.
>>
>> A solution is to come up with our very own basic VA allocator for
>> MMIO. Since half of the HYP address space only contains a single
>> page (the idmap), we have plenty to borrow from. Let's use the idmap
>> as a base, and allocate downwards from it. GICv2 now lives on the
>> other side of the great VA barrier.
> 
>> diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
>> index 9946f8a38b26..a9577ecc3e6a 100644
>> --- a/virt/kvm/arm/mmu.c
>> +++ b/virt/kvm/arm/mmu.c
>> @@ -43,6 +43,9 @@ static unsigned long hyp_idmap_start;
>>  static unsigned long hyp_idmap_end;
>>  static phys_addr_t hyp_idmap_vector;
>>  
>> +static DEFINE_MUTEX(io_map_lock);
>> +static unsigned long io_map_base;
>> +
>>  #define S2_PGD_SIZE	(PTRS_PER_S2_PGD * sizeof(pgd_t))
>>  #define hyp_pgd_order get_order(PTRS_PER_PGD * sizeof(pgd_t))
>>  
>> @@ -518,27 +521,37 @@ static void unmap_hyp_idmap_range(pgd_t *pgdp, phys_addr_t start, u64 size)
>>   *
>>   * Assumes hyp_pgd is a page table used strictly in Hyp-mode and
>>   * therefore contains either mappings in the kernel memory area (above
>> - * PAGE_OFFSET), or device mappings in the vmalloc range (from
>> - * VMALLOC_START to VMALLOC_END).
>> + * PAGE_OFFSET), or device mappings in the idmap range.
>>   *
>> - * boot_hyp_pgd should only map two pages for the init code.
>> + * boot_hyp_pgd should only map the idmap range, and is only used in
>> + * the extended idmap case.
>>   */
>>  void free_hyp_pgds(void)
>>  {
>> +	pgd_t *id_pgd;
>> +
>>  	mutex_lock(&kvm_hyp_pgd_mutex);
>>  
>> +	id_pgd = boot_hyp_pgd ? boot_hyp_pgd : hyp_pgd;
>> +
>> +	if (id_pgd) {
>> +		mutex_lock(&io_map_lock);
> 
> This takes kvm_hyp_pgd_mutex then io_map_lock ...
> 
> 
>> +		/* In case we never called hyp_mmu_init() */
>> +		if (!io_map_base)
>> +			io_map_base = hyp_idmap_start;
>> +		unmap_hyp_idmap_range(id_pgd, io_map_base,
>> +				      hyp_idmap_start + PAGE_SIZE - io_map_base);
>> +		mutex_unlock(&io_map_lock);
>> +	}
>> +
>>  	if (boot_hyp_pgd) {
>> -		unmap_hyp_idmap_range(boot_hyp_pgd, hyp_idmap_start, PAGE_SIZE);
>>  		free_pages((unsigned long)boot_hyp_pgd, hyp_pgd_order);
>>  		boot_hyp_pgd = NULL;
>>  	}
>>  
>>  	if (hyp_pgd) {
>> -		unmap_hyp_idmap_range(hyp_pgd, hyp_idmap_start, PAGE_SIZE);
>>  		unmap_hyp_range(hyp_pgd, kern_hyp_va(PAGE_OFFSET),
>>  				(uintptr_t)high_memory - PAGE_OFFSET);
>> -		unmap_hyp_range(hyp_pgd, kern_hyp_va(VMALLOC_START),
>> -				VMALLOC_END - VMALLOC_START);
>>  
>>  		free_pages((unsigned long)hyp_pgd, hyp_pgd_order);
>>  		hyp_pgd = NULL;
> 
>> @@ -747,11 +761,43 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
>>  		return 0;
>>  	}
> 
>> +	mutex_lock(&io_map_lock);
> 
>> -	start = kern_hyp_va((unsigned long)*kaddr);
>> -	end = kern_hyp_va((unsigned long)*kaddr + size);
>> -	ret = __create_hyp_mappings(hyp_pgd, PTRS_PER_PGD, start, end,
>> -				     __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
>> +	/*
>> +	 * This assumes that we we have enough space below the idmap
>> +	 * page to allocate our VAs. If not, the check below will
>> +	 * kick. A potential alternative would be to detect that
>> +	 * overflow and switch to an allocation above the idmap.
>> +	 *
>> +	 * The allocated size is always a multiple of PAGE_SIZE.
>> +	 */
>> +	size = PAGE_ALIGN(size + offset_in_page(phys_addr));
>> +	base = io_map_base - size;
>> +
>> +	/*
>> +	 * Verify that BIT(VA_BITS - 1) hasn't been flipped by
>> +	 * allocating the new area, as it would indicate we've
>> +	 * overflowed the idmap/IO address range.
>> +	 */
>> +	if ((base ^ io_map_base) & BIT(VA_BITS - 1)) {
>> +		ret = -ENOMEM;
>> +		goto out;
>> +	}
>> +
>> +	if (__kvm_cpu_uses_extended_idmap())
>> +		pgd = boot_hyp_pgd;
>> +
>> +	ret = __create_hyp_mappings(pgd, __kvm_idmap_ptrs_per_pgd(),
>> +				    base, base + size,
>> +				    __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
> 
> And here we have io_map_lock, but __create_hyp_mappings takes kvm_hyp_pgd_mutex.
> 
> There is another example of this in create_hyp_exec_mappings, I think making
> this the required order makes sense: if you are mapping to the KVM-IO region,
> you take the io_map_lock before taking the pgd lock to write in your allocated
> location. (free_hyp_pgds() is always going to need to take both).
> 
> Never going to happen, but it generates a lockdep splat[1].
> Fixup diff[0] below fixes it for me.

Thanks for noticing this.

In the end, I wonder if there is an actual need for this io_map_lock at
all. We can perfectly take the the pgd lock to allocate the IOVA area,
commit the allocation, drop the lock and call __create_hyp_mapping,
which is going to take it again.

What if the mapping fails? We end-up having committed the IOVA
allocation, and having dropped the lock makes it nigh impossibly to roll
it back. But failing a HYP mapping that early is pretty fatal anyway,
and we're going to tear down the whole of HYP, never mind the IOVA
allocator. So I'm more than tempted to do away with that extra lock, and
rely on the existing one. This isn't exactly a fast path anyway...

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 12/26] KVM: arm/arm64: Move HYP IO VAs to the "idmap" range
@ 2018-03-16  8:44       ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-16  8:44 UTC (permalink / raw)
  To: linux-arm-kernel

Hi James,

On 15/03/18 19:09, James Morse wrote:
> Hi Marc,
> 
> On 14/03/18 16:50, Marc Zyngier wrote:
>> We so far mapped our HYP IO (which is essentially the GICv2 control
>> registers) using the same method as for memory. It recently appeared
>> that is a bit unsafe:
>>
>> We compute the HYP VA using the kern_hyp_va helper, but that helper
>> is only designed to deal with kernel VAs coming from the linear map,
>> and not from the vmalloc region... This could in turn cause some bad
>> aliasing between the two, amplified by the upcoming VA randomisation.
>>
>> A solution is to come up with our very own basic VA allocator for
>> MMIO. Since half of the HYP address space only contains a single
>> page (the idmap), we have plenty to borrow from. Let's use the idmap
>> as a base, and allocate downwards from it. GICv2 now lives on the
>> other side of the great VA barrier.
> 
>> diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
>> index 9946f8a38b26..a9577ecc3e6a 100644
>> --- a/virt/kvm/arm/mmu.c
>> +++ b/virt/kvm/arm/mmu.c
>> @@ -43,6 +43,9 @@ static unsigned long hyp_idmap_start;
>>  static unsigned long hyp_idmap_end;
>>  static phys_addr_t hyp_idmap_vector;
>>  
>> +static DEFINE_MUTEX(io_map_lock);
>> +static unsigned long io_map_base;
>> +
>>  #define S2_PGD_SIZE	(PTRS_PER_S2_PGD * sizeof(pgd_t))
>>  #define hyp_pgd_order get_order(PTRS_PER_PGD * sizeof(pgd_t))
>>  
>> @@ -518,27 +521,37 @@ static void unmap_hyp_idmap_range(pgd_t *pgdp, phys_addr_t start, u64 size)
>>   *
>>   * Assumes hyp_pgd is a page table used strictly in Hyp-mode and
>>   * therefore contains either mappings in the kernel memory area (above
>> - * PAGE_OFFSET), or device mappings in the vmalloc range (from
>> - * VMALLOC_START to VMALLOC_END).
>> + * PAGE_OFFSET), or device mappings in the idmap range.
>>   *
>> - * boot_hyp_pgd should only map two pages for the init code.
>> + * boot_hyp_pgd should only map the idmap range, and is only used in
>> + * the extended idmap case.
>>   */
>>  void free_hyp_pgds(void)
>>  {
>> +	pgd_t *id_pgd;
>> +
>>  	mutex_lock(&kvm_hyp_pgd_mutex);
>>  
>> +	id_pgd = boot_hyp_pgd ? boot_hyp_pgd : hyp_pgd;
>> +
>> +	if (id_pgd) {
>> +		mutex_lock(&io_map_lock);
> 
> This takes kvm_hyp_pgd_mutex then io_map_lock ...
> 
> 
>> +		/* In case we never called hyp_mmu_init() */
>> +		if (!io_map_base)
>> +			io_map_base = hyp_idmap_start;
>> +		unmap_hyp_idmap_range(id_pgd, io_map_base,
>> +				      hyp_idmap_start + PAGE_SIZE - io_map_base);
>> +		mutex_unlock(&io_map_lock);
>> +	}
>> +
>>  	if (boot_hyp_pgd) {
>> -		unmap_hyp_idmap_range(boot_hyp_pgd, hyp_idmap_start, PAGE_SIZE);
>>  		free_pages((unsigned long)boot_hyp_pgd, hyp_pgd_order);
>>  		boot_hyp_pgd = NULL;
>>  	}
>>  
>>  	if (hyp_pgd) {
>> -		unmap_hyp_idmap_range(hyp_pgd, hyp_idmap_start, PAGE_SIZE);
>>  		unmap_hyp_range(hyp_pgd, kern_hyp_va(PAGE_OFFSET),
>>  				(uintptr_t)high_memory - PAGE_OFFSET);
>> -		unmap_hyp_range(hyp_pgd, kern_hyp_va(VMALLOC_START),
>> -				VMALLOC_END - VMALLOC_START);
>>  
>>  		free_pages((unsigned long)hyp_pgd, hyp_pgd_order);
>>  		hyp_pgd = NULL;
> 
>> @@ -747,11 +761,43 @@ int create_hyp_io_mappings(phys_addr_t phys_addr, size_t size,
>>  		return 0;
>>  	}
> 
>> +	mutex_lock(&io_map_lock);
> 
>> -	start = kern_hyp_va((unsigned long)*kaddr);
>> -	end = kern_hyp_va((unsigned long)*kaddr + size);
>> -	ret = __create_hyp_mappings(hyp_pgd, PTRS_PER_PGD, start, end,
>> -				     __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
>> +	/*
>> +	 * This assumes that we we have enough space below the idmap
>> +	 * page to allocate our VAs. If not, the check below will
>> +	 * kick. A potential alternative would be to detect that
>> +	 * overflow and switch to an allocation above the idmap.
>> +	 *
>> +	 * The allocated size is always a multiple of PAGE_SIZE.
>> +	 */
>> +	size = PAGE_ALIGN(size + offset_in_page(phys_addr));
>> +	base = io_map_base - size;
>> +
>> +	/*
>> +	 * Verify that BIT(VA_BITS - 1) hasn't been flipped by
>> +	 * allocating the new area, as it would indicate we've
>> +	 * overflowed the idmap/IO address range.
>> +	 */
>> +	if ((base ^ io_map_base) & BIT(VA_BITS - 1)) {
>> +		ret = -ENOMEM;
>> +		goto out;
>> +	}
>> +
>> +	if (__kvm_cpu_uses_extended_idmap())
>> +		pgd = boot_hyp_pgd;
>> +
>> +	ret = __create_hyp_mappings(pgd, __kvm_idmap_ptrs_per_pgd(),
>> +				    base, base + size,
>> +				    __phys_to_pfn(phys_addr), PAGE_HYP_DEVICE);
> 
> And here we have io_map_lock, but __create_hyp_mappings takes kvm_hyp_pgd_mutex.
> 
> There is another example of this in create_hyp_exec_mappings, I think making
> this the required order makes sense: if you are mapping to the KVM-IO region,
> you take the io_map_lock before taking the pgd lock to write in your allocated
> location. (free_hyp_pgds() is always going to need to take both).
> 
> Never going to happen, but it generates a lockdep splat[1].
> Fixup diff[0] below fixes it for me.

Thanks for noticing this.

In the end, I wonder if there is an actual need for this io_map_lock at
all. We can perfectly take the the pgd lock to allocate the IOVA area,
commit the allocation, drop the lock and call __create_hyp_mapping,
which is going to take it again.

What if the mapping fails? We end-up having committed the IOVA
allocation, and having dropped the lock makes it nigh impossibly to roll
it back. But failing a HYP mapping that early is pretty fatal anyway,
and we're going to tear down the whole of HYP, never mind the IOVA
allocator. So I'm more than tempted to do away with that extra lock, and
rely on the existing one. This isn't exactly a fast path anyway...

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 04/26] arm64: KVM: Dynamically patch the kernel/hyp VA mask
  2018-03-15 19:15     ` James Morse
@ 2018-03-16  8:52       ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-16  8:52 UTC (permalink / raw)
  To: James Morse
  Cc: Mark Rutland, Peter Maydell, Andrew Jones, Christoffer Dall, kvm,
	Steve Capper, Catalin Marinas, Will Deacon, Kristina Martsenko,
	kvmarm, linux-arm-kernel

On 15/03/18 19:15, James Morse wrote:
> Hi Marc,
> 
> On 14/03/18 16:50, Marc Zyngier wrote:
>> So far, we're using a complicated sequence of alternatives to
>> patch the kernel/hyp VA mask on non-VHE, and NOP out the
>> masking altogether when on VHE.
>>
>> The newly introduced dynamic patching gives us the opportunity
>> to simplify that code by patching a single instruction with
>> the correct mask (instead of the mind bending cummulative masking
> 
> (Nit: cumulative)
> 
> (so this series removes mind bending code?)

Absolutely. And replaces it with... erm... (/me shuts up).

> 
>> we have at the moment) or even a single NOP on VHE. This also
>> adds some initial code that will allow the patching callback
>> to switch to a more complex patching.
> 
>> diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c
>> new file mode 100644
>> index 000000000000..45e7802328d4
>> --- /dev/null
>> +++ b/arch/arm64/kvm/va_layout.c
> 
>> +void __init kvm_update_va_mask(struct alt_instr *alt,
>> +			       __le32 *origptr, __le32 *updptr, int nr_inst)
>> +{
>> +	int i;
>> +
>> +	/* We only expect a single instruction in the alternative sequence */
>> +	BUG_ON(nr_inst != 1);
>> +
>> +	if (!has_vhe() && !va_mask)
>> +		compute_layout();
>> +
>> +	for (i = 0; i < nr_inst; i++) {
>> +		u32 rd, rn, insn, oinsn;
>> +
>> +		/*
>> +		 * VHE doesn't need any address translation, let's NOP
>> +		 * everything.
>> +		 */
>> +		if (has_vhe()) {
>> +			updptr[i] = aarch64_insn_gen_nop();
> 
> cpu_to_le32()? (I'm not going to try an boot a BE VHE model...)

You're missing out. Let's make BE big again.

> 
> aarch64_insn_gen_nop() returns:
> | aarch64_insn_get_hint_value() | AARCH64_INSN_HINT_NOP;
> 
> It doesn't look like these aarch64_insn_get_XXX_value() helpers are forcing a
> particular endianness. ftrace uses this, via ftrace_modify_code() ->
> aarch64_insn_patch_text_nosync() -> aarch64_insn_write(), which does:
> | return  __aarch64_insn_write(addr, cpu_to_le32(insn));
> 
> So it looks like the conversion is required. Patch 16 looks fine for this.

Absolutely correct, I'm missing the byte swap. Now fixed.

> (and, I ran the teardown code on Juno big-endian...)

Wow. You *the* user!!!! ;-)

> 
> 
>> +			continue;
>> +		}
>> +
>> +		oinsn = le32_to_cpu(origptr[i]);
>> +		rd = aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RD, oinsn);
>> +		rn = aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RN, oinsn);
>> +
>> +		insn = compute_instruction(i, rd, rn);
>> +		BUG_ON(insn == AARCH64_BREAK_FAULT);
>> +
>> +		updptr[i] = cpu_to_le32(insn);
>> +	}
>> +}
> 
> With that,
> 
> Reviewed-by: James Morse <james.morse@arm.com>

Thanks a lot,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 04/26] arm64: KVM: Dynamically patch the kernel/hyp VA mask
@ 2018-03-16  8:52       ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-16  8:52 UTC (permalink / raw)
  To: linux-arm-kernel

On 15/03/18 19:15, James Morse wrote:
> Hi Marc,
> 
> On 14/03/18 16:50, Marc Zyngier wrote:
>> So far, we're using a complicated sequence of alternatives to
>> patch the kernel/hyp VA mask on non-VHE, and NOP out the
>> masking altogether when on VHE.
>>
>> The newly introduced dynamic patching gives us the opportunity
>> to simplify that code by patching a single instruction with
>> the correct mask (instead of the mind bending cummulative masking
> 
> (Nit: cumulative)
> 
> (so this series removes mind bending code?)

Absolutely. And replaces it with... erm... (/me shuts up).

> 
>> we have at the moment) or even a single NOP on VHE. This also
>> adds some initial code that will allow the patching callback
>> to switch to a more complex patching.
> 
>> diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c
>> new file mode 100644
>> index 000000000000..45e7802328d4
>> --- /dev/null
>> +++ b/arch/arm64/kvm/va_layout.c
> 
>> +void __init kvm_update_va_mask(struct alt_instr *alt,
>> +			       __le32 *origptr, __le32 *updptr, int nr_inst)
>> +{
>> +	int i;
>> +
>> +	/* We only expect a single instruction in the alternative sequence */
>> +	BUG_ON(nr_inst != 1);
>> +
>> +	if (!has_vhe() && !va_mask)
>> +		compute_layout();
>> +
>> +	for (i = 0; i < nr_inst; i++) {
>> +		u32 rd, rn, insn, oinsn;
>> +
>> +		/*
>> +		 * VHE doesn't need any address translation, let's NOP
>> +		 * everything.
>> +		 */
>> +		if (has_vhe()) {
>> +			updptr[i] = aarch64_insn_gen_nop();
> 
> cpu_to_le32()? (I'm not going to try an boot a BE VHE model...)

You're missing out. Let's make BE big again.

> 
> aarch64_insn_gen_nop() returns:
> | aarch64_insn_get_hint_value() | AARCH64_INSN_HINT_NOP;
> 
> It doesn't look like these aarch64_insn_get_XXX_value() helpers are forcing a
> particular endianness. ftrace uses this, via ftrace_modify_code() ->
> aarch64_insn_patch_text_nosync() -> aarch64_insn_write(), which does:
> | return  __aarch64_insn_write(addr, cpu_to_le32(insn));
> 
> So it looks like the conversion is required. Patch 16 looks fine for this.

Absolutely correct, I'm missing the byte swap. Now fixed.

> (and, I ran the teardown code on Juno big-endian...)

Wow. You *the* user!!!! ;-)

> 
> 
>> +			continue;
>> +		}
>> +
>> +		oinsn = le32_to_cpu(origptr[i]);
>> +		rd = aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RD, oinsn);
>> +		rn = aarch64_insn_decode_register(AARCH64_INSN_REGTYPE_RN, oinsn);
>> +
>> +		insn = compute_instruction(i, rd, rn);
>> +		BUG_ON(insn == AARCH64_BREAK_FAULT);
>> +
>> +		updptr[i] = cpu_to_le32(insn);
>> +	}
>> +}
> 
> With that,
> 
> Reviewed-by: James Morse <james.morse@arm.com>

Thanks a lot,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 10/26] KVM: arm/arm64: Fix idmap size and alignment
  2018-03-15 19:15     ` James Morse
@ 2018-03-16  8:55       ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-16  8:55 UTC (permalink / raw)
  To: James Morse, kvmarm
  Cc: Mark Rutland, Peter Maydell, Andrew Jones, Christoffer Dall, kvm,
	Steve Capper, Catalin Marinas, Will Deacon, Kristina Martsenko,
	linux-arm-kernel

On 15/03/18 19:15, James Morse wrote:
> On 14/03/18 16:50, Marc Zyngier wrote:
>> Although the idmap section of KVM can only be at most 4kB and
>> must be aligned on a 4kB boundary, the rest of the code expects
>> it to be page aligned. Things get messy when tearing down the
>> HYP page tables when PAGE_SIZE is 64K, and the idmap section isn't
>> 64K aligned.
>>
>> Let's fix this by computing aligned boundaries that the HYP code
>> will use.
> 
>> diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
>> index 0e5cfffb4c21..a9e0513868e9 100644
>> --- a/virt/kvm/arm/mmu.c
>> +++ b/virt/kvm/arm/mmu.c
>> @@ -1815,7 +1815,9 @@ int kvm_mmu_init(void)
>>  	int err;
>>  
>>  	hyp_idmap_start = kvm_virt_to_phys(__hyp_idmap_text_start);
>> +	hyp_idmap_start = ALIGN_DOWN(hyp_idmap_start, PAGE_SIZE);
>>  	hyp_idmap_end = kvm_virt_to_phys(__hyp_idmap_text_end);
>> +	hyp_idmap_end = ALIGN(hyp_idmap_end, PAGE_SIZE);
>>  	hyp_idmap_vector = kvm_virt_to_phys(__kvm_hyp_init);
>>  
>>  	/*
> 
> 
> This makes the:
> | hyp_idmap_start != (unsigned long)__hyp_idmap_text_start
> 
> check below look funny, but that must be for 32bit which only has 4K pages (I
> think), so its behaviour hasn't changed.

Yes, that's to handle the 2/2 split on 32bit where some funnies may
happen, see d2896d4b55b2 ("arm: KVM: Fix idmap overlap detection when
the kernel is idmap'ed").

> Reviewed-by: James Morse <james.morse@arm.com>

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 10/26] KVM: arm/arm64: Fix idmap size and alignment
@ 2018-03-16  8:55       ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-16  8:55 UTC (permalink / raw)
  To: linux-arm-kernel

On 15/03/18 19:15, James Morse wrote:
> On 14/03/18 16:50, Marc Zyngier wrote:
>> Although the idmap section of KVM can only be at most 4kB and
>> must be aligned on a 4kB boundary, the rest of the code expects
>> it to be page aligned. Things get messy when tearing down the
>> HYP page tables when PAGE_SIZE is 64K, and the idmap section isn't
>> 64K aligned.
>>
>> Let's fix this by computing aligned boundaries that the HYP code
>> will use.
> 
>> diff --git a/virt/kvm/arm/mmu.c b/virt/kvm/arm/mmu.c
>> index 0e5cfffb4c21..a9e0513868e9 100644
>> --- a/virt/kvm/arm/mmu.c
>> +++ b/virt/kvm/arm/mmu.c
>> @@ -1815,7 +1815,9 @@ int kvm_mmu_init(void)
>>  	int err;
>>  
>>  	hyp_idmap_start = kvm_virt_to_phys(__hyp_idmap_text_start);
>> +	hyp_idmap_start = ALIGN_DOWN(hyp_idmap_start, PAGE_SIZE);
>>  	hyp_idmap_end = kvm_virt_to_phys(__hyp_idmap_text_end);
>> +	hyp_idmap_end = ALIGN(hyp_idmap_end, PAGE_SIZE);
>>  	hyp_idmap_vector = kvm_virt_to_phys(__kvm_hyp_init);
>>  
>>  	/*
> 
> 
> This makes the:
> | hyp_idmap_start != (unsigned long)__hyp_idmap_text_start
> 
> check below look funny, but that must be for 32bit which only has 4K pages (I
> think), so its behaviour hasn't changed.

Yes, that's to handle the 2/2 split on 32bit where some funnies may
happen, see d2896d4b55b2 ("arm: KVM: Fix idmap overlap detection when
the kernel is idmap'ed").

> Reviewed-by: James Morse <james.morse@arm.com>

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 06/26] KVM: arm/arm64: Do not use kern_hyp_va() with kvm_vgic_global_state
  2018-03-15 19:16     ` James Morse
@ 2018-03-16  9:31       ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-16  9:31 UTC (permalink / raw)
  To: James Morse, kvmarm
  Cc: Mark Rutland, Peter Maydell, Andrew Jones, Christoffer Dall, kvm,
	Steve Capper, Catalin Marinas, Will Deacon, Kristina Martsenko,
	linux-arm-kernel

On 15/03/18 19:16, James Morse wrote:
> Hi Marc,
> 
> On 14/03/18 16:50, Marc Zyngier wrote:
>> kvm_vgic_global_state is part of the read-only section, and is
>> usually accessed using a PC-relative address generation (adrp + add).
>>
>> It is thus useless to use kern_hyp_va() on it, and actively problematic
>> if kern_hyp_va() becomes non-idempotent. On the other hand, there is
>> no way that the compiler is going to guarantee that such access is
>> always PC relative.
>>
>> So let's bite the bullet and provide our own accessor.
> 
> 
>> diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
>> index de1b919404e4..a6808d2869f5 100644
>> --- a/arch/arm/include/asm/kvm_mmu.h
>> +++ b/arch/arm/include/asm/kvm_mmu.h
>> @@ -28,6 +28,13 @@
>>   */
>>  #define kern_hyp_va(kva)	(kva)
>>  
>> +/* Resolving symbol addresses in a PC-relative way is easy... */
> 
> (So this is guaranteed on 32bit? I thought this was because 32bit uses the
> kernel-VA's at HYP, so any way the compiler generates the address will work.)

You're right, this comment is slightly idiotic. What I meant to convey
is that we don't need to provide PC-relative addresses on 32bit. I've
replaced it with:

/*
 * Contrary to arm64, there is no need to generate a PC-relative address
 */

> 
> 
>> +#define hyp_symbol_addr(s)						\
>> +	({								\
>> +		typeof(s) *addr = &(s);					\
>> +		addr;							\
>> +	})
>> +
>>  /*
>>   * KVM_MMU_CACHE_MIN_PAGES is the number of stage2 page table translation levels.
>>   */
> 
>> diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
>> index e3bc1d0a5e93..7120bf3f22c7 100644
>> --- a/arch/arm64/include/asm/kvm_mmu.h
>> +++ b/arch/arm64/include/asm/kvm_mmu.h
>> @@ -110,6 +110,26 @@ static inline unsigned long __kern_hyp_va(unsigned long v)
>>  
>>  #define kern_hyp_va(v) 	((typeof(v))(__kern_hyp_va((unsigned long)(v))))
>>  
>> +/*
>> + * Obtain the PC-relative address of a kernel symbol
>> + * s: symbol
>> + *
>> + * The goal of this macro is to return a symbol's address based on a
>> + * PC-relative computation, as opposed to a loading the VA from a
>> + * constant pool or something similar. This works well for HYP, as an
>> + * absolute VA is guaranteed to be wrong. Only use this if trying to
>> + * obtain the address of a symbol (i.e. not something you obtained by
>> + * following a pointer).
>> + */
>> +#define hyp_symbol_addr(s)						\
>> +	({								\
>> +		typeof(s) *addr;					\
>> +		asm volatile("adrp	%0, %1\n"			\
>> +			     "add	%0, %0, :lo12:%1\n"		\
>> +			     : "=r" (addr) : "S" (&s));			\
>> +		addr;							\
>> +	})
> 
> Why does this need to be volatile? I see gcc v4.9 generate this sequence twice
> in __vgic_v2_save_state(). Can't it cache and reorder this sequence as it likes?

I think you're right. I tend to use "volatile" to prevent the compiler
from optimizing it away, but the fact that we rely on the output of the
sequence makes it impossible.

> Either-way:
> Reviewed-by: James Morse <james.morse@arm.com>
> 
> 
> (I had a go at using 'Ush', to let the compiler schedule the adrp, but couldn't
> get it to work.)

I tried that as well at some point, but couldn't see how to use it. The
compiler was never happy with my use of the constraints, so I gave up
and did it my way...

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 06/26] KVM: arm/arm64: Do not use kern_hyp_va() with kvm_vgic_global_state
@ 2018-03-16  9:31       ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-16  9:31 UTC (permalink / raw)
  To: linux-arm-kernel

On 15/03/18 19:16, James Morse wrote:
> Hi Marc,
> 
> On 14/03/18 16:50, Marc Zyngier wrote:
>> kvm_vgic_global_state is part of the read-only section, and is
>> usually accessed using a PC-relative address generation (adrp + add).
>>
>> It is thus useless to use kern_hyp_va() on it, and actively problematic
>> if kern_hyp_va() becomes non-idempotent. On the other hand, there is
>> no way that the compiler is going to guarantee that such access is
>> always PC relative.
>>
>> So let's bite the bullet and provide our own accessor.
> 
> 
>> diff --git a/arch/arm/include/asm/kvm_mmu.h b/arch/arm/include/asm/kvm_mmu.h
>> index de1b919404e4..a6808d2869f5 100644
>> --- a/arch/arm/include/asm/kvm_mmu.h
>> +++ b/arch/arm/include/asm/kvm_mmu.h
>> @@ -28,6 +28,13 @@
>>   */
>>  #define kern_hyp_va(kva)	(kva)
>>  
>> +/* Resolving symbol addresses in a PC-relative way is easy... */
> 
> (So this is guaranteed on 32bit? I thought this was because 32bit uses the
> kernel-VA's at HYP, so any way the compiler generates the address will work.)

You're right, this comment is slightly idiotic. What I meant to convey
is that we don't need to provide PC-relative addresses on 32bit. I've
replaced it with:

/*
 * Contrary to arm64, there is no need to generate a PC-relative address
 */

> 
> 
>> +#define hyp_symbol_addr(s)						\
>> +	({								\
>> +		typeof(s) *addr = &(s);					\
>> +		addr;							\
>> +	})
>> +
>>  /*
>>   * KVM_MMU_CACHE_MIN_PAGES is the number of stage2 page table translation levels.
>>   */
> 
>> diff --git a/arch/arm64/include/asm/kvm_mmu.h b/arch/arm64/include/asm/kvm_mmu.h
>> index e3bc1d0a5e93..7120bf3f22c7 100644
>> --- a/arch/arm64/include/asm/kvm_mmu.h
>> +++ b/arch/arm64/include/asm/kvm_mmu.h
>> @@ -110,6 +110,26 @@ static inline unsigned long __kern_hyp_va(unsigned long v)
>>  
>>  #define kern_hyp_va(v) 	((typeof(v))(__kern_hyp_va((unsigned long)(v))))
>>  
>> +/*
>> + * Obtain the PC-relative address of a kernel symbol
>> + * s: symbol
>> + *
>> + * The goal of this macro is to return a symbol's address based on a
>> + * PC-relative computation, as opposed to a loading the VA from a
>> + * constant pool or something similar. This works well for HYP, as an
>> + * absolute VA is guaranteed to be wrong. Only use this if trying to
>> + * obtain the address of a symbol (i.e. not something you obtained by
>> + * following a pointer).
>> + */
>> +#define hyp_symbol_addr(s)						\
>> +	({								\
>> +		typeof(s) *addr;					\
>> +		asm volatile("adrp	%0, %1\n"			\
>> +			     "add	%0, %0, :lo12:%1\n"		\
>> +			     : "=r" (addr) : "S" (&s));			\
>> +		addr;							\
>> +	})
> 
> Why does this need to be volatile? I see gcc v4.9 generate this sequence twice
> in __vgic_v2_save_state(). Can't it cache and reorder this sequence as it likes?

I think you're right. I tend to use "volatile" to prevent the compiler
from optimizing it away, but the fact that we rely on the output of the
sequence makes it impossible.

> Either-way:
> Reviewed-by: James Morse <james.morse@arm.com>
> 
> 
> (I had a go at using 'Ush', to let the compiler schedule the adrp, but couldn't
> get it to work.)

I tried that as well at some point, but couldn't see how to use it. The
compiler was never happy with my use of the constraints, so I gave up
and did it my way...

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 06/26] KVM: arm/arm64: Do not use kern_hyp_va() with kvm_vgic_global_state
  2018-03-16  9:31       ` Marc Zyngier
@ 2018-03-16 11:35         ` Andrew Jones
  -1 siblings, 0 replies; 124+ messages in thread
From: Andrew Jones @ 2018-03-16 11:35 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: Mark Rutland, Peter Maydell, Christoffer Dall, kvm, Steve Capper,
	Catalin Marinas, Will Deacon, Kristina Martsenko, James Morse,
	kvmarm, linux-arm-kernel

On Fri, Mar 16, 2018 at 09:31:57AM +0000, Marc Zyngier wrote:
> On 15/03/18 19:16, James Morse wrote:
> > 
> > (I had a go at using 'Ush', to let the compiler schedule the adrp, but couldn't
> > get it to work.)
> 
> I tried that as well at some point, but couldn't see how to use it. The
> compiler was never happy with my use of the constraints, so I gave up
> and did it my way...
>

What's 'Ush'? I tried to search for it, but came up short. I'm wondering
what things I can try (and fail) to use it on too :-)

Thanks,
drew

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 06/26] KVM: arm/arm64: Do not use kern_hyp_va() with kvm_vgic_global_state
@ 2018-03-16 11:35         ` Andrew Jones
  0 siblings, 0 replies; 124+ messages in thread
From: Andrew Jones @ 2018-03-16 11:35 UTC (permalink / raw)
  To: linux-arm-kernel

On Fri, Mar 16, 2018 at 09:31:57AM +0000, Marc Zyngier wrote:
> On 15/03/18 19:16, James Morse wrote:
> > 
> > (I had a go at using 'Ush', to let the compiler schedule the adrp, but couldn't
> > get it to work.)
> 
> I tried that as well at some point, but couldn't see how to use it. The
> compiler was never happy with my use of the constraints, so I gave up
> and did it my way...
>

What's 'Ush'? I tried to search for it, but came up short. I'm wondering
what things I can try (and fail) to use it on too :-)

Thanks,
drew

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 06/26] KVM: arm/arm64: Do not use kern_hyp_va() with kvm_vgic_global_state
  2018-03-16 11:35         ` Andrew Jones
@ 2018-03-16 11:38           ` Ard Biesheuvel
  -1 siblings, 0 replies; 124+ messages in thread
From: Ard Biesheuvel @ 2018-03-16 11:38 UTC (permalink / raw)
  To: Andrew Jones
  Cc: Mark Rutland, Peter Maydell, Christoffer Dall,
	KVM devel mailing list, Steve Capper, Marc Zyngier,
	Catalin Marinas, Will Deacon, Kristina Martsenko, James Morse,
	kvmarm, linux-arm-kernel

On 16 March 2018 at 11:35, Andrew Jones <drjones@redhat.com> wrote:
> On Fri, Mar 16, 2018 at 09:31:57AM +0000, Marc Zyngier wrote:
>> On 15/03/18 19:16, James Morse wrote:
>> >
>> > (I had a go at using 'Ush', to let the compiler schedule the adrp, but couldn't
>> > get it to work.)
>>
>> I tried that as well at some point, but couldn't see how to use it. The
>> compiler was never happy with my use of the constraints, so I gave up
>> and did it my way...
>>
>
> What's 'Ush'? I tried to search for it, but came up short. I'm wondering
> what things I can try (and fail) to use it on too :-)
>

https://gcc.gnu.org/onlinedocs/gccint/Machine-Constraints.html

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 06/26] KVM: arm/arm64: Do not use kern_hyp_va() with kvm_vgic_global_state
@ 2018-03-16 11:38           ` Ard Biesheuvel
  0 siblings, 0 replies; 124+ messages in thread
From: Ard Biesheuvel @ 2018-03-16 11:38 UTC (permalink / raw)
  To: linux-arm-kernel

On 16 March 2018 at 11:35, Andrew Jones <drjones@redhat.com> wrote:
> On Fri, Mar 16, 2018 at 09:31:57AM +0000, Marc Zyngier wrote:
>> On 15/03/18 19:16, James Morse wrote:
>> >
>> > (I had a go at using 'Ush', to let the compiler schedule the adrp, but couldn't
>> > get it to work.)
>>
>> I tried that as well at some point, but couldn't see how to use it. The
>> compiler was never happy with my use of the constraints, so I gave up
>> and did it my way...
>>
>
> What's 'Ush'? I tried to search for it, but came up short. I'm wondering
> what things I can try (and fail) to use it on too :-)
>

https://gcc.gnu.org/onlinedocs/gccint/Machine-Constraints.html

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 06/26] KVM: arm/arm64: Do not use kern_hyp_va() with kvm_vgic_global_state
  2018-03-16 11:38           ` Ard Biesheuvel
@ 2018-03-16 11:51             ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-16 11:51 UTC (permalink / raw)
  To: Ard Biesheuvel, Andrew Jones
  Cc: Mark Rutland, Peter Maydell, Christoffer Dall,
	KVM devel mailing list, Steve Capper, Catalin Marinas,
	Will Deacon, Kristina Martsenko, James Morse, kvmarm,
	linux-arm-kernel

On 16/03/18 11:38, Ard Biesheuvel wrote:
> On 16 March 2018 at 11:35, Andrew Jones <drjones@redhat.com> wrote:
>> On Fri, Mar 16, 2018 at 09:31:57AM +0000, Marc Zyngier wrote:
>>> On 15/03/18 19:16, James Morse wrote:
>>>>
>>>> (I had a go at using 'Ush', to let the compiler schedule the adrp, but couldn't
>>>> get it to work.)
>>>
>>> I tried that as well at some point, but couldn't see how to use it. The
>>> compiler was never happy with my use of the constraints, so I gave up
>>> and did it my way...
>>>
>>
>> What's 'Ush'? I tried to search for it, but came up short. I'm wondering
>> what things I can try (and fail) to use it on too :-)
>>
> 
> https://gcc.gnu.org/onlinedocs/gccint/Machine-Constraints.html

I was hoping that something like:

	asm("add %0, %1, :lo12:%2" : "=r" (addr) : "Ush" (&s), "S" (&s))

would do the right thing (generating an ADRP), but all I managed was to
get the compiler shouting at me (in a rather unhelpful manner).

I guess I could have looked into the GCC source code to find out how to
use this thing, but life is short.

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 06/26] KVM: arm/arm64: Do not use kern_hyp_va() with kvm_vgic_global_state
@ 2018-03-16 11:51             ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-16 11:51 UTC (permalink / raw)
  To: linux-arm-kernel

On 16/03/18 11:38, Ard Biesheuvel wrote:
> On 16 March 2018 at 11:35, Andrew Jones <drjones@redhat.com> wrote:
>> On Fri, Mar 16, 2018 at 09:31:57AM +0000, Marc Zyngier wrote:
>>> On 15/03/18 19:16, James Morse wrote:
>>>>
>>>> (I had a go at using 'Ush', to let the compiler schedule the adrp, but couldn't
>>>> get it to work.)
>>>
>>> I tried that as well at some point, but couldn't see how to use it. The
>>> compiler was never happy with my use of the constraints, so I gave up
>>> and did it my way...
>>>
>>
>> What's 'Ush'? I tried to search for it, but came up short. I'm wondering
>> what things I can try (and fail) to use it on too :-)
>>
> 
> https://gcc.gnu.org/onlinedocs/gccint/Machine-Constraints.html

I was hoping that something like:

	asm("add %0, %1, :lo12:%2" : "=r" (addr) : "Ush" (&s), "S" (&s))

would do the right thing (generating an ADRP), but all I managed was to
get the compiler shouting at me (in a rather unhelpful manner).

I guess I could have looked into the GCC source code to find out how to
use this thing, but life is short.

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 25/26] arm64: KVM: Allow mapping of vectors outside of the RAM region
  2018-03-15 18:47           ` Marc Zyngier
@ 2018-03-16 12:33             ` Andrew Jones
  -1 siblings, 0 replies; 124+ messages in thread
From: Andrew Jones @ 2018-03-16 12:33 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: Mark Rutland, Peter Maydell, Christoffer Dall, kvm, Steve Capper,
	Catalin Marinas, Will Deacon, Kristina Martsenko, James Morse,
	kvmarm, linux-arm-kernel

On Thu, Mar 15, 2018 at 06:47:22PM +0000, Marc Zyngier wrote:
> On 15/03/18 17:08, Andrew Jones wrote:
> > On Thu, Mar 15, 2018 at 04:17:53PM +0000, Marc Zyngier wrote:
> >> On 15/03/18 15:54, Andrew Jones wrote:
> >>> On Wed, Mar 14, 2018 at 04:50:48PM +0000, Marc Zyngier wrote:
> >>>>  static inline void *kvm_get_hyp_vector(void)
> >>>>  {
> >>>>  	struct bp_hardening_data *data = arm64_get_bp_hardening_data();
> >>>> -	void *vect = kvm_ksym_ref(__kvm_hyp_vector);
> >>>> +	void *vect = kern_hyp_va(kvm_ksym_ref(__kvm_hyp_vector));
> >>>> +	int slot = -1;
> >>>>  
> >>>> -	if (data->fn) {
> >>>> -		vect = __bp_harden_hyp_vecs_start +
> >>>> -		       data->hyp_vectors_slot * SZ_2K;
> >>>> +	if (cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR) && data->fn) {
> >>>
> >>> Now that I'm trying to consider heterogeneous systems, I'm wondering why
> >>> harden BP isn't checked with this_cpu_has_cap() like harden EL2 vectors.
> >>> I'm also wondering if it's even possible for CPUs to come up without all
> >>> of them having harden EL2 vectors if the boot CPU has it. Won't
> >>> verify_local_cpu_errata_workarounds() require it? I'm probably just
> >>> getting lost in all the capability stuff...
> >>
> >> Checking harden BP on this particular CPU will only tell you if this CPU
> >> is affected, but won't give you any additional information (i.e. you'd
> >> still need to check obtain the stuff pointed to by data).
> >>
> >> Checking for *all* CPUs in one go has some advantages: it is a static
> >> key, which means that unaffected platforms will fly (this is a hot path
> >> on VHE), and you can check data if you're affected.
> > 
> > Is there a problem with using the static key to check
> > ARM64_HARDEN_EL2_VECTORS below? (Sorry, I'm still confused why we're not
> > using the same functions for both.) 
> 
> Consider the following case: heterogeneous system with CPU0 that has
> HBP, and CPU1 that has HEL. CPU0 is using its own vector slot, and CPU1
> should be using an extra one (it doesn't have a dedicated slot).
> 
> When CPU0 claims its vectors, it will get the "normal" version of the
> mapping. And then, if we're using cpus_have_cap, we turn that into an
> out-of-line mapping. That's not completely wrong (the system still runs
> fine), but that's not the expected behaviour. If we restrict the second
> phase to this_cpu_has_cap(), we get the expected CPU0 using the normal
> mapping, and CPU1 does the exact opposite (which we also want).

Thanks for the additional explanation. I've finally got my head wrapped
around this and it looks good to me. And, I see now that my comment about
verify_local_cpu_errata_workarounds() only applies to CPUs plugged after
boot. The comment in check_local_cpu_capabilities() states clearly each
CPU present at boot gets a chance to update cpu_hwcaps - which also
clarifies why kvm_map_vectors() *must* use cpus_have_const_cap(), else
risk the el2 vector slot not being allocated.

Thanks,
drew

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 25/26] arm64: KVM: Allow mapping of vectors outside of the RAM region
@ 2018-03-16 12:33             ` Andrew Jones
  0 siblings, 0 replies; 124+ messages in thread
From: Andrew Jones @ 2018-03-16 12:33 UTC (permalink / raw)
  To: linux-arm-kernel

On Thu, Mar 15, 2018 at 06:47:22PM +0000, Marc Zyngier wrote:
> On 15/03/18 17:08, Andrew Jones wrote:
> > On Thu, Mar 15, 2018 at 04:17:53PM +0000, Marc Zyngier wrote:
> >> On 15/03/18 15:54, Andrew Jones wrote:
> >>> On Wed, Mar 14, 2018 at 04:50:48PM +0000, Marc Zyngier wrote:
> >>>>  static inline void *kvm_get_hyp_vector(void)
> >>>>  {
> >>>>  	struct bp_hardening_data *data = arm64_get_bp_hardening_data();
> >>>> -	void *vect = kvm_ksym_ref(__kvm_hyp_vector);
> >>>> +	void *vect = kern_hyp_va(kvm_ksym_ref(__kvm_hyp_vector));
> >>>> +	int slot = -1;
> >>>>  
> >>>> -	if (data->fn) {
> >>>> -		vect = __bp_harden_hyp_vecs_start +
> >>>> -		       data->hyp_vectors_slot * SZ_2K;
> >>>> +	if (cpus_have_const_cap(ARM64_HARDEN_BRANCH_PREDICTOR) && data->fn) {
> >>>
> >>> Now that I'm trying to consider heterogeneous systems, I'm wondering why
> >>> harden BP isn't checked with this_cpu_has_cap() like harden EL2 vectors.
> >>> I'm also wondering if it's even possible for CPUs to come up without all
> >>> of them having harden EL2 vectors if the boot CPU has it. Won't
> >>> verify_local_cpu_errata_workarounds() require it? I'm probably just
> >>> getting lost in all the capability stuff...
> >>
> >> Checking harden BP on this particular CPU will only tell you if this CPU
> >> is affected, but won't give you any additional information (i.e. you'd
> >> still need to check obtain the stuff pointed to by data).
> >>
> >> Checking for *all* CPUs in one go has some advantages: it is a static
> >> key, which means that unaffected platforms will fly (this is a hot path
> >> on VHE), and you can check data if you're affected.
> > 
> > Is there a problem with using the static key to check
> > ARM64_HARDEN_EL2_VECTORS below? (Sorry, I'm still confused why we're not
> > using the same functions for both.) 
> 
> Consider the following case: heterogeneous system with CPU0 that has
> HBP, and CPU1 that has HEL. CPU0 is using its own vector slot, and CPU1
> should be using an extra one (it doesn't have a dedicated slot).
> 
> When CPU0 claims its vectors, it will get the "normal" version of the
> mapping. And then, if we're using cpus_have_cap, we turn that into an
> out-of-line mapping. That's not completely wrong (the system still runs
> fine), but that's not the expected behaviour. If we restrict the second
> phase to this_cpu_has_cap(), we get the expected CPU0 using the normal
> mapping, and CPU1 does the exact opposite (which we also want).

Thanks for the additional explanation. I've finally got my head wrapped
around this and it looks good to me. And, I see now that my comment about
verify_local_cpu_errata_workarounds() only applies to CPUs plugged after
boot. The comment in check_local_cpu_capabilities() states clearly each
CPU present at boot gets a chance to update cpu_hwcaps - which also
clarifies why kvm_map_vectors() *must* use cpus_have_const_cap(), else
risk the el2 vector slot not being allocated.

Thanks,
drew

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 11/26] KVM: arm64: Fix HYP idmap unmap when using 52bit PA
  2018-03-14 16:50   ` Marc Zyngier
@ 2018-03-16 16:07     ` Catalin Marinas
  -1 siblings, 0 replies; 124+ messages in thread
From: Catalin Marinas @ 2018-03-16 16:07 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: kvm, Will Deacon, Kristina Martsenko, kvmarm, linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:34PM +0000, Marc Zyngier wrote:
> Unmapping the idmap range using 52bit PA is quite broken, as we
> don't take into account the right number of PGD entries, and rely
> on PTRS_PER_PGD. The result is that pgd_index() truncates the
> address, and we end-up in the weed.
> 
> Let's introduce a new unmap_hyp_idmap_range() that knows about this,
> together with a kvm_pgd_index() helper, which hides a bit of the
> complexity of the issue.
> 
> Fixes: 98732d1b189b ("KVM: arm/arm64: fix HYP ID map extension to 52 bits")
> Reported-by: James Morse <james.morse@arm.com>
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>

It looks fine to me:

Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 11/26] KVM: arm64: Fix HYP idmap unmap when using 52bit PA
@ 2018-03-16 16:07     ` Catalin Marinas
  0 siblings, 0 replies; 124+ messages in thread
From: Catalin Marinas @ 2018-03-16 16:07 UTC (permalink / raw)
  To: linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:34PM +0000, Marc Zyngier wrote:
> Unmapping the idmap range using 52bit PA is quite broken, as we
> don't take into account the right number of PGD entries, and rely
> on PTRS_PER_PGD. The result is that pgd_index() truncates the
> address, and we end-up in the weed.
> 
> Let's introduce a new unmap_hyp_idmap_range() that knows about this,
> together with a kvm_pgd_index() helper, which hides a bit of the
> complexity of the issue.
> 
> Fixes: 98732d1b189b ("KVM: arm/arm64: fix HYP ID map extension to 52 bits")
> Reported-by: James Morse <james.morse@arm.com>
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>

It looks fine to me:

Reviewed-by: Catalin Marinas <catalin.marinas@arm.com>

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 19/26] arm64: KVM: Move stashing of x0/x1 into the vector code itself
  2018-03-14 16:50   ` Marc Zyngier
@ 2018-03-16 16:22     ` Catalin Marinas
  -1 siblings, 0 replies; 124+ messages in thread
From: Catalin Marinas @ 2018-03-16 16:22 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: kvm, Will Deacon, Kristina Martsenko, kvmarm, linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:42PM +0000, Marc Zyngier wrote:
> All our useful entry points into the hypervisor are starting by
> saving x0 and x1 on the stack. Let's move those into the vectors
> by introducing macros that annotate whether a vector is valid or
> not, thus indicating whether we want to stash registers or not.
> 
> The only drawback is that we now also stash registers for el2_error,
> but this should never happen, and we pop them back right at the
> start of the handling sequence.
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> ---
>  arch/arm64/kvm/hyp/hyp-entry.S | 56 ++++++++++++++++++++++++------------------
>  1 file changed, 32 insertions(+), 24 deletions(-)
> 
> diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S
> index f36464bd57c5..0f62b5f76aa5 100644
> --- a/arch/arm64/kvm/hyp/hyp-entry.S
> +++ b/arch/arm64/kvm/hyp/hyp-entry.S
> @@ -55,7 +55,6 @@ ENTRY(__vhe_hyp_call)
>  ENDPROC(__vhe_hyp_call)
>  
>  el1_sync:				// Guest trapped into EL2
> -	stp	x0, x1, [sp, #-16]!
>  
>  alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
>  	mrs	x1, esr_el2
> @@ -137,18 +136,18 @@ alternative_else_nop_endif
>  	b	__guest_exit
>  
>  el1_irq:
> -	stp     x0, x1, [sp, #-16]!
>  	ldr	x1, [sp, #16 + 8]
>  	mov	x0, #ARM_EXCEPTION_IRQ
>  	b	__guest_exit
>  
>  el1_error:
> -	stp     x0, x1, [sp, #-16]!
>  	ldr	x1, [sp, #16 + 8]
>  	mov	x0, #ARM_EXCEPTION_EL1_SERROR
>  	b	__guest_exit
>  
>  el2_error:
> +	ldp	x0, x1, [sp], #16
> +

Nitpick: you don't need a memory access here, just:

	add	sp, sp, #16

(unless el2_error has changed somewhere before this patch)

-- 
Catalin

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 19/26] arm64: KVM: Move stashing of x0/x1 into the vector code itself
@ 2018-03-16 16:22     ` Catalin Marinas
  0 siblings, 0 replies; 124+ messages in thread
From: Catalin Marinas @ 2018-03-16 16:22 UTC (permalink / raw)
  To: linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:42PM +0000, Marc Zyngier wrote:
> All our useful entry points into the hypervisor are starting by
> saving x0 and x1 on the stack. Let's move those into the vectors
> by introducing macros that annotate whether a vector is valid or
> not, thus indicating whether we want to stash registers or not.
> 
> The only drawback is that we now also stash registers for el2_error,
> but this should never happen, and we pop them back right at the
> start of the handling sequence.
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> ---
>  arch/arm64/kvm/hyp/hyp-entry.S | 56 ++++++++++++++++++++++++------------------
>  1 file changed, 32 insertions(+), 24 deletions(-)
> 
> diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S
> index f36464bd57c5..0f62b5f76aa5 100644
> --- a/arch/arm64/kvm/hyp/hyp-entry.S
> +++ b/arch/arm64/kvm/hyp/hyp-entry.S
> @@ -55,7 +55,6 @@ ENTRY(__vhe_hyp_call)
>  ENDPROC(__vhe_hyp_call)
>  
>  el1_sync:				// Guest trapped into EL2
> -	stp	x0, x1, [sp, #-16]!
>  
>  alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
>  	mrs	x1, esr_el2
> @@ -137,18 +136,18 @@ alternative_else_nop_endif
>  	b	__guest_exit
>  
>  el1_irq:
> -	stp     x0, x1, [sp, #-16]!
>  	ldr	x1, [sp, #16 + 8]
>  	mov	x0, #ARM_EXCEPTION_IRQ
>  	b	__guest_exit
>  
>  el1_error:
> -	stp     x0, x1, [sp, #-16]!
>  	ldr	x1, [sp, #16 + 8]
>  	mov	x0, #ARM_EXCEPTION_EL1_SERROR
>  	b	__guest_exit
>  
>  el2_error:
> +	ldp	x0, x1, [sp], #16
> +

Nitpick: you don't need a memory access here, just:

	add	sp, sp, #16

(unless el2_error has changed somewhere before this patch)

-- 
Catalin

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 20/26] arm64: KVM: Move BP hardening vectors into .hyp.text section
  2018-03-14 16:50   ` Marc Zyngier
@ 2018-03-16 16:24     ` Catalin Marinas
  -1 siblings, 0 replies; 124+ messages in thread
From: Catalin Marinas @ 2018-03-16 16:24 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: kvm, Will Deacon, Kristina Martsenko, kvmarm, linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:43PM +0000, Marc Zyngier wrote:
> There is no reason why the BP hardening vectors shouldn't be part
> of the HYP text at compile time, rather than being mapped at runtime.
> 
> Also introduce a new config symbol that controls the compilation
> of bpi.S.
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>

Acked-by: Catalin Marinas <catalin.marinas@arm.com>

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 20/26] arm64: KVM: Move BP hardening vectors into .hyp.text section
@ 2018-03-16 16:24     ` Catalin Marinas
  0 siblings, 0 replies; 124+ messages in thread
From: Catalin Marinas @ 2018-03-16 16:24 UTC (permalink / raw)
  To: linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:43PM +0000, Marc Zyngier wrote:
> There is no reason why the BP hardening vectors shouldn't be part
> of the HYP text at compile time, rather than being mapped at runtime.
> 
> Also introduce a new config symbol that controls the compilation
> of bpi.S.
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>

Acked-by: Catalin Marinas <catalin.marinas@arm.com>

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 21/26] arm64: KVM: Reserve 4 additional instructions in the BPI template
  2018-03-14 16:50   ` Marc Zyngier
@ 2018-03-16 16:30     ` Catalin Marinas
  -1 siblings, 0 replies; 124+ messages in thread
From: Catalin Marinas @ 2018-03-16 16:30 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: kvm, Will Deacon, Kristina Martsenko, kvmarm, linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:44PM +0000, Marc Zyngier wrote:
> So far, we only reserve a single instruction in the BPI template in
> order to branch to the vectors. As we're going to stuff a few more
> instructions there, let's reserve a total of 5 instructions, which
> we're going to patch later on as required.
> 
> We also introduce a small refactor of the vectors themselves, so that
> we stop carrying the target branch around.
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>

Acked-by: Catalin Marinas <catalin.marinas@arm.com>

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 21/26] arm64: KVM: Reserve 4 additional instructions in the BPI template
@ 2018-03-16 16:30     ` Catalin Marinas
  0 siblings, 0 replies; 124+ messages in thread
From: Catalin Marinas @ 2018-03-16 16:30 UTC (permalink / raw)
  To: linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:44PM +0000, Marc Zyngier wrote:
> So far, we only reserve a single instruction in the BPI template in
> order to branch to the vectors. As we're going to stuff a few more
> instructions there, let's reserve a total of 5 instructions, which
> we're going to patch later on as required.
> 
> We also introduce a small refactor of the vectors themselves, so that
> we stop carrying the target branch around.
> 
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>

Acked-by: Catalin Marinas <catalin.marinas@arm.com>

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 19/26] arm64: KVM: Move stashing of x0/x1 into the vector code itself
  2018-03-16 16:22     ` Catalin Marinas
@ 2018-03-16 16:37       ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-16 16:37 UTC (permalink / raw)
  To: Catalin Marinas
  Cc: kvm, Will Deacon, Kristina Martsenko, kvmarm, linux-arm-kernel

On 16/03/18 16:22, Catalin Marinas wrote:
> On Wed, Mar 14, 2018 at 04:50:42PM +0000, Marc Zyngier wrote:
>> All our useful entry points into the hypervisor are starting by
>> saving x0 and x1 on the stack. Let's move those into the vectors
>> by introducing macros that annotate whether a vector is valid or
>> not, thus indicating whether we want to stash registers or not.
>>
>> The only drawback is that we now also stash registers for el2_error,
>> but this should never happen, and we pop them back right at the
>> start of the handling sequence.
>>
>> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
>> ---
>>  arch/arm64/kvm/hyp/hyp-entry.S | 56 ++++++++++++++++++++++++------------------
>>  1 file changed, 32 insertions(+), 24 deletions(-)
>>
>> diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S
>> index f36464bd57c5..0f62b5f76aa5 100644
>> --- a/arch/arm64/kvm/hyp/hyp-entry.S
>> +++ b/arch/arm64/kvm/hyp/hyp-entry.S
>> @@ -55,7 +55,6 @@ ENTRY(__vhe_hyp_call)
>>  ENDPROC(__vhe_hyp_call)
>>  
>>  el1_sync:				// Guest trapped into EL2
>> -	stp	x0, x1, [sp, #-16]!
>>  
>>  alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
>>  	mrs	x1, esr_el2
>> @@ -137,18 +136,18 @@ alternative_else_nop_endif
>>  	b	__guest_exit
>>  
>>  el1_irq:
>> -	stp     x0, x1, [sp, #-16]!
>>  	ldr	x1, [sp, #16 + 8]
>>  	mov	x0, #ARM_EXCEPTION_IRQ
>>  	b	__guest_exit
>>  
>>  el1_error:
>> -	stp     x0, x1, [sp, #-16]!
>>  	ldr	x1, [sp, #16 + 8]
>>  	mov	x0, #ARM_EXCEPTION_EL1_SERROR
>>  	b	__guest_exit
>>  
>>  el2_error:
>> +	ldp	x0, x1, [sp], #16
>> +
> 
> Nitpick: you don't need a memory access here, just:
> 
> 	add	sp, sp, #16
> 
> (unless el2_error has changed somewhere before this patch)

At this point in the series, I agree. But starting with patch 22, we
start messing with x0 if HARDEN_EL2_VECTORS is valid, meaning we really
need to restore it in order to preserve the guest state.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 19/26] arm64: KVM: Move stashing of x0/x1 into the vector code itself
@ 2018-03-16 16:37       ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-16 16:37 UTC (permalink / raw)
  To: linux-arm-kernel

On 16/03/18 16:22, Catalin Marinas wrote:
> On Wed, Mar 14, 2018 at 04:50:42PM +0000, Marc Zyngier wrote:
>> All our useful entry points into the hypervisor are starting by
>> saving x0 and x1 on the stack. Let's move those into the vectors
>> by introducing macros that annotate whether a vector is valid or
>> not, thus indicating whether we want to stash registers or not.
>>
>> The only drawback is that we now also stash registers for el2_error,
>> but this should never happen, and we pop them back right at the
>> start of the handling sequence.
>>
>> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
>> ---
>>  arch/arm64/kvm/hyp/hyp-entry.S | 56 ++++++++++++++++++++++++------------------
>>  1 file changed, 32 insertions(+), 24 deletions(-)
>>
>> diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S
>> index f36464bd57c5..0f62b5f76aa5 100644
>> --- a/arch/arm64/kvm/hyp/hyp-entry.S
>> +++ b/arch/arm64/kvm/hyp/hyp-entry.S
>> @@ -55,7 +55,6 @@ ENTRY(__vhe_hyp_call)
>>  ENDPROC(__vhe_hyp_call)
>>  
>>  el1_sync:				// Guest trapped into EL2
>> -	stp	x0, x1, [sp, #-16]!
>>  
>>  alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
>>  	mrs	x1, esr_el2
>> @@ -137,18 +136,18 @@ alternative_else_nop_endif
>>  	b	__guest_exit
>>  
>>  el1_irq:
>> -	stp     x0, x1, [sp, #-16]!
>>  	ldr	x1, [sp, #16 + 8]
>>  	mov	x0, #ARM_EXCEPTION_IRQ
>>  	b	__guest_exit
>>  
>>  el1_error:
>> -	stp     x0, x1, [sp, #-16]!
>>  	ldr	x1, [sp, #16 + 8]
>>  	mov	x0, #ARM_EXCEPTION_EL1_SERROR
>>  	b	__guest_exit
>>  
>>  el2_error:
>> +	ldp	x0, x1, [sp], #16
>> +
> 
> Nitpick: you don't need a memory access here, just:
> 
> 	add	sp, sp, #16
> 
> (unless el2_error has changed somewhere before this patch)

At this point in the series, I agree. But starting with patch 22, we
start messing with x0 if HARDEN_EL2_VECTORS is valid, meaning we really
need to restore it in order to preserve the guest state.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 19/26] arm64: KVM: Move stashing of x0/x1 into the vector code itself
  2018-03-16 16:22     ` Catalin Marinas
@ 2018-03-16 16:38       ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-16 16:38 UTC (permalink / raw)
  To: Catalin Marinas
  Cc: kvm, Will Deacon, Kristina Martsenko, kvmarm, linux-arm-kernel

On 16/03/18 16:22, Catalin Marinas wrote:
> On Wed, Mar 14, 2018 at 04:50:42PM +0000, Marc Zyngier wrote:
>> All our useful entry points into the hypervisor are starting by
>> saving x0 and x1 on the stack. Let's move those into the vectors
>> by introducing macros that annotate whether a vector is valid or
>> not, thus indicating whether we want to stash registers or not.
>>
>> The only drawback is that we now also stash registers for el2_error,
>> but this should never happen, and we pop them back right at the
>> start of the handling sequence.
>>
>> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
>> ---
>>  arch/arm64/kvm/hyp/hyp-entry.S | 56 ++++++++++++++++++++++++------------------
>>  1 file changed, 32 insertions(+), 24 deletions(-)
>>
>> diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S
>> index f36464bd57c5..0f62b5f76aa5 100644
>> --- a/arch/arm64/kvm/hyp/hyp-entry.S
>> +++ b/arch/arm64/kvm/hyp/hyp-entry.S
>> @@ -55,7 +55,6 @@ ENTRY(__vhe_hyp_call)
>>  ENDPROC(__vhe_hyp_call)
>>  
>>  el1_sync:				// Guest trapped into EL2
>> -	stp	x0, x1, [sp, #-16]!
>>  
>>  alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
>>  	mrs	x1, esr_el2
>> @@ -137,18 +136,18 @@ alternative_else_nop_endif
>>  	b	__guest_exit
>>  
>>  el1_irq:
>> -	stp     x0, x1, [sp, #-16]!
>>  	ldr	x1, [sp, #16 + 8]
>>  	mov	x0, #ARM_EXCEPTION_IRQ
>>  	b	__guest_exit
>>  
>>  el1_error:
>> -	stp     x0, x1, [sp, #-16]!
>>  	ldr	x1, [sp, #16 + 8]
>>  	mov	x0, #ARM_EXCEPTION_EL1_SERROR
>>  	b	__guest_exit
>>  
>>  el2_error:
>> +	ldp	x0, x1, [sp], #16
>> +
> 
> Nitpick: you don't need a memory access here, just:
> 
> 	add	sp, sp, #16
> 
> (unless el2_error has changed somewhere before this patch)

At this point in the series, I agree. But starting with patch 22, we
start messing with x0 if HARDEN_EL2_VECTORS is valid, meaning we really
need to restore it in order to preserve the guest state.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 19/26] arm64: KVM: Move stashing of x0/x1 into the vector code itself
@ 2018-03-16 16:38       ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-16 16:38 UTC (permalink / raw)
  To: linux-arm-kernel

On 16/03/18 16:22, Catalin Marinas wrote:
> On Wed, Mar 14, 2018 at 04:50:42PM +0000, Marc Zyngier wrote:
>> All our useful entry points into the hypervisor are starting by
>> saving x0 and x1 on the stack. Let's move those into the vectors
>> by introducing macros that annotate whether a vector is valid or
>> not, thus indicating whether we want to stash registers or not.
>>
>> The only drawback is that we now also stash registers for el2_error,
>> but this should never happen, and we pop them back right at the
>> start of the handling sequence.
>>
>> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
>> ---
>>  arch/arm64/kvm/hyp/hyp-entry.S | 56 ++++++++++++++++++++++++------------------
>>  1 file changed, 32 insertions(+), 24 deletions(-)
>>
>> diff --git a/arch/arm64/kvm/hyp/hyp-entry.S b/arch/arm64/kvm/hyp/hyp-entry.S
>> index f36464bd57c5..0f62b5f76aa5 100644
>> --- a/arch/arm64/kvm/hyp/hyp-entry.S
>> +++ b/arch/arm64/kvm/hyp/hyp-entry.S
>> @@ -55,7 +55,6 @@ ENTRY(__vhe_hyp_call)
>>  ENDPROC(__vhe_hyp_call)
>>  
>>  el1_sync:				// Guest trapped into EL2
>> -	stp	x0, x1, [sp, #-16]!
>>  
>>  alternative_if_not ARM64_HAS_VIRT_HOST_EXTN
>>  	mrs	x1, esr_el2
>> @@ -137,18 +136,18 @@ alternative_else_nop_endif
>>  	b	__guest_exit
>>  
>>  el1_irq:
>> -	stp     x0, x1, [sp, #-16]!
>>  	ldr	x1, [sp, #16 + 8]
>>  	mov	x0, #ARM_EXCEPTION_IRQ
>>  	b	__guest_exit
>>  
>>  el1_error:
>> -	stp     x0, x1, [sp, #-16]!
>>  	ldr	x1, [sp, #16 + 8]
>>  	mov	x0, #ARM_EXCEPTION_EL1_SERROR
>>  	b	__guest_exit
>>  
>>  el2_error:
>> +	ldp	x0, x1, [sp], #16
>> +
> 
> Nitpick: you don't need a memory access here, just:
> 
> 	add	sp, sp, #16
> 
> (unless el2_error has changed somewhere before this patch)

At this point in the series, I agree. But starting with patch 22, we
start messing with x0 if HARDEN_EL2_VECTORS is valid, meaning we really
need to restore it in order to preserve the guest state.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 11/26] KVM: arm64: Fix HYP idmap unmap when using 52bit PA
  2018-03-14 16:50   ` Marc Zyngier
@ 2018-03-16 16:47     ` Suzuki K Poulose
  -1 siblings, 0 replies; 124+ messages in thread
From: Suzuki K Poulose @ 2018-03-16 16:47 UTC (permalink / raw)
  To: Marc Zyngier, linux-arm-kernel, kvm, kvmarm
  Cc: Catalin Marinas, Will Deacon, Kristina Martsenko

On 14/03/18 16:50, Marc Zyngier wrote:
> Unmapping the idmap range using 52bit PA is quite broken, as we
> don't take into account the right number of PGD entries, and rely
> on PTRS_PER_PGD. The result is that pgd_index() truncates the
> address, and we end-up in the weed.
> 
> Let's introduce a new unmap_hyp_idmap_range() that knows about this,
> together with a kvm_pgd_index() helper, which hides a bit of the
> complexity of the issue.
> 
> Fixes: 98732d1b189b ("KVM: arm/arm64: fix HYP ID map extension to 52 bits")
> Reported-by: James Morse <james.morse@arm.com>
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> ---

Looks good to me, FWIW:

Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 11/26] KVM: arm64: Fix HYP idmap unmap when using 52bit PA
@ 2018-03-16 16:47     ` Suzuki K Poulose
  0 siblings, 0 replies; 124+ messages in thread
From: Suzuki K Poulose @ 2018-03-16 16:47 UTC (permalink / raw)
  To: linux-arm-kernel

On 14/03/18 16:50, Marc Zyngier wrote:
> Unmapping the idmap range using 52bit PA is quite broken, as we
> don't take into account the right number of PGD entries, and rely
> on PTRS_PER_PGD. The result is that pgd_index() truncates the
> address, and we end-up in the weed.
> 
> Let's introduce a new unmap_hyp_idmap_range() that knows about this,
> together with a kvm_pgd_index() helper, which hides a bit of the
> complexity of the issue.
> 
> Fixes: 98732d1b189b ("KVM: arm/arm64: fix HYP ID map extension to 52 bits")
> Reported-by: James Morse <james.morse@arm.com>
> Signed-off-by: Marc Zyngier <marc.zyngier@arm.com>
> ---

Looks good to me, FWIW:

Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 00/26] KVM/arm64: Randomise EL2 mappings (variant 3a mitigation)
  2018-03-14 16:50 ` Marc Zyngier
@ 2018-03-16 17:46   ` Catalin Marinas
  -1 siblings, 0 replies; 124+ messages in thread
From: Catalin Marinas @ 2018-03-16 17:46 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: kvm, Will Deacon, Kristina Martsenko, kvmarm, linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:23PM +0000, Marc Zyngier wrote:
> Whilst KVM benefits from the kernel randomisation via KASLR, there is
> no additional randomisation when the kernel is running at EL1, as we
> directly use a fixed offset from the linear mapping. This is not
> necessarily a problem, but we could do a bit better by independently
> randomizing the HYP placement.

For the rest of the patches in this series:

Acked-by: Catalin Marinas <catalin.marinas@arm.com>

Some points for a future series:

- in bpi.S, drop the __smccc_workaround_1_hvc_start/end as HVC PSCI
  conduit for hyp doesn't make much sense (could pass NULL instead to
  __install_bp_hardening_cb())

- maybe move bpi.S under arch/arm64/kvm/ if it doesn't cause other
  dependency issues in cpu_errata.c

Thanks.

-- 
Catalin

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 00/26] KVM/arm64: Randomise EL2 mappings (variant 3a mitigation)
@ 2018-03-16 17:46   ` Catalin Marinas
  0 siblings, 0 replies; 124+ messages in thread
From: Catalin Marinas @ 2018-03-16 17:46 UTC (permalink / raw)
  To: linux-arm-kernel

On Wed, Mar 14, 2018 at 04:50:23PM +0000, Marc Zyngier wrote:
> Whilst KVM benefits from the kernel randomisation via KASLR, there is
> no additional randomisation when the kernel is running at EL1, as we
> directly use a fixed offset from the linear mapping. This is not
> necessarily a problem, but we could do a bit better by independently
> randomizing the HYP placement.

For the rest of the patches in this series:

Acked-by: Catalin Marinas <catalin.marinas@arm.com>

Some points for a future series:

- in bpi.S, drop the __smccc_workaround_1_hvc_start/end as HVC PSCI
  conduit for hyp doesn't make much sense (could pass NULL instead to
  __install_bp_hardening_cb())

- maybe move bpi.S under arch/arm64/kvm/ if it doesn't cause other
  dependency issues in cpu_errata.c

Thanks.

-- 
Catalin

^ permalink raw reply	[flat|nested] 124+ messages in thread

* Re: [PATCH v6 00/26] KVM/arm64: Randomise EL2 mappings (variant 3a mitigation)
  2018-03-16 17:46   ` Catalin Marinas
@ 2018-03-16 18:05     ` Marc Zyngier
  -1 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-16 18:05 UTC (permalink / raw)
  To: Catalin Marinas
  Cc: kvm, Will Deacon, Kristina Martsenko, kvmarm, linux-arm-kernel

On 16/03/18 17:46, Catalin Marinas wrote:
> On Wed, Mar 14, 2018 at 04:50:23PM +0000, Marc Zyngier wrote:
>> Whilst KVM benefits from the kernel randomisation via KASLR, there is
>> no additional randomisation when the kernel is running at EL1, as we
>> directly use a fixed offset from the linear mapping. This is not
>> necessarily a problem, but we could do a bit better by independently
>> randomizing the HYP placement.
> 
> For the rest of the patches in this series:
> 
> Acked-by: Catalin Marinas <catalin.marinas@arm.com>

Thanks for that.

> Some points for a future series:
> 
> - in bpi.S, drop the __smccc_workaround_1_hvc_start/end as HVC PSCI
>   conduit for hyp doesn't make much sense (could pass NULL instead to
>   __install_bp_hardening_cb())
> 
> - maybe move bpi.S under arch/arm64/kvm/ if it doesn't cause other
>   dependency issues in cpu_errata.c

Sure. I'll have a look at a bit of a post-paranoia cleanup next week.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

* [PATCH v6 00/26] KVM/arm64: Randomise EL2 mappings (variant 3a mitigation)
@ 2018-03-16 18:05     ` Marc Zyngier
  0 siblings, 0 replies; 124+ messages in thread
From: Marc Zyngier @ 2018-03-16 18:05 UTC (permalink / raw)
  To: linux-arm-kernel

On 16/03/18 17:46, Catalin Marinas wrote:
> On Wed, Mar 14, 2018 at 04:50:23PM +0000, Marc Zyngier wrote:
>> Whilst KVM benefits from the kernel randomisation via KASLR, there is
>> no additional randomisation when the kernel is running at EL1, as we
>> directly use a fixed offset from the linear mapping. This is not
>> necessarily a problem, but we could do a bit better by independently
>> randomizing the HYP placement.
> 
> For the rest of the patches in this series:
> 
> Acked-by: Catalin Marinas <catalin.marinas@arm.com>

Thanks for that.

> Some points for a future series:
> 
> - in bpi.S, drop the __smccc_workaround_1_hvc_start/end as HVC PSCI
>   conduit for hyp doesn't make much sense (could pass NULL instead to
>   __install_bp_hardening_cb())
> 
> - maybe move bpi.S under arch/arm64/kvm/ if it doesn't cause other
>   dependency issues in cpu_errata.c

Sure. I'll have a look at a bit of a post-paranoia cleanup next week.

Thanks,

	M.
-- 
Jazz is not dead. It just smells funny...

^ permalink raw reply	[flat|nested] 124+ messages in thread

end of thread, other threads:[~2018-03-16 18:05 UTC | newest]

Thread overview: 124+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-03-14 16:50 [PATCH v6 00/26] KVM/arm64: Randomise EL2 mappings (variant 3a mitigation) Marc Zyngier
2018-03-14 16:50 ` Marc Zyngier
2018-03-14 16:50 ` [PATCH v6 01/26] arm64: alternatives: Add dynamic patching feature Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-14 16:50 ` [PATCH v6 02/26] arm64: insn: Add N immediate encoding Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-14 16:50 ` [PATCH v6 03/26] arm64: insn: Add encoder for bitwise operations using literals Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-14 16:50 ` [PATCH v6 04/26] arm64: KVM: Dynamically patch the kernel/hyp VA mask Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-15 19:15   ` James Morse
2018-03-15 19:15     ` James Morse
2018-03-16  8:52     ` Marc Zyngier
2018-03-16  8:52       ` Marc Zyngier
2018-03-14 16:50 ` [PATCH v6 05/26] arm64: cpufeatures: Drop the ARM64_HYP_OFFSET_LOW feature flag Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-14 16:50 ` [PATCH v6 06/26] KVM: arm/arm64: Do not use kern_hyp_va() with kvm_vgic_global_state Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-15 19:16   ` James Morse
2018-03-15 19:16     ` James Morse
2018-03-16  9:31     ` Marc Zyngier
2018-03-16  9:31       ` Marc Zyngier
2018-03-16 11:35       ` Andrew Jones
2018-03-16 11:35         ` Andrew Jones
2018-03-16 11:38         ` Ard Biesheuvel
2018-03-16 11:38           ` Ard Biesheuvel
2018-03-16 11:51           ` Marc Zyngier
2018-03-16 11:51             ` Marc Zyngier
2018-03-14 16:50 ` [PATCH v6 07/26] KVM: arm/arm64: Demote HYP VA range display to being a debug feature Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-14 16:50 ` [PATCH v6 08/26] KVM: arm/arm64: Move ioremap calls to create_hyp_io_mappings Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-14 16:50 ` [PATCH v6 09/26] KVM: arm/arm64: Keep GICv2 HYP VAs in kvm_vgic_global_state Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-14 16:50 ` [PATCH v6 10/26] KVM: arm/arm64: Fix idmap size and alignment Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-15 19:15   ` James Morse
2018-03-15 19:15     ` James Morse
2018-03-16  8:55     ` Marc Zyngier
2018-03-16  8:55       ` Marc Zyngier
2018-03-14 16:50 ` [PATCH v6 11/26] KVM: arm64: Fix HYP idmap unmap when using 52bit PA Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-16 16:07   ` Catalin Marinas
2018-03-16 16:07     ` Catalin Marinas
2018-03-16 16:47   ` Suzuki K Poulose
2018-03-16 16:47     ` Suzuki K Poulose
2018-03-14 16:50 ` [PATCH v6 12/26] KVM: arm/arm64: Move HYP IO VAs to the "idmap" range Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-15 19:09   ` James Morse
2018-03-15 19:09     ` James Morse
2018-03-16  8:44     ` Marc Zyngier
2018-03-16  8:44       ` Marc Zyngier
2018-03-14 16:50 ` [PATCH v6 13/26] arm64; insn: Add encoder for the EXTR instruction Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-14 16:50 ` [PATCH v6 14/26] arm64: insn: Allow ADD/SUB (immediate) with LSL #12 Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-14 16:50 ` [PATCH v6 15/26] arm64: KVM: Dynamically compute the HYP VA mask Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-14 16:50 ` [PATCH v6 16/26] arm64: KVM: Introduce EL2 VA randomisation Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-14 16:50 ` [PATCH v6 17/26] arm64: Update the KVM memory map documentation Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-14 16:50 ` [PATCH v6 18/26] arm64: KVM: Move vector offsetting from hyp-init.S to kvm_get_hyp_vector Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-14 16:50 ` [PATCH v6 19/26] arm64: KVM: Move stashing of x0/x1 into the vector code itself Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-15 14:39   ` Andrew Jones
2018-03-15 14:39     ` Andrew Jones
2018-03-16 16:22   ` Catalin Marinas
2018-03-16 16:22     ` Catalin Marinas
2018-03-16 16:37     ` Marc Zyngier
2018-03-16 16:37       ` Marc Zyngier
2018-03-16 16:38     ` Marc Zyngier
2018-03-16 16:38       ` Marc Zyngier
2018-03-14 16:50 ` [PATCH v6 20/26] arm64: KVM: Move BP hardening vectors into .hyp.text section Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-15 14:42   ` Andrew Jones
2018-03-15 14:42     ` Andrew Jones
2018-03-16 16:24   ` Catalin Marinas
2018-03-16 16:24     ` Catalin Marinas
2018-03-14 16:50 ` [PATCH v6 21/26] arm64: KVM: Reserve 4 additional instructions in the BPI template Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-15 14:46   ` Andrew Jones
2018-03-15 14:46     ` Andrew Jones
2018-03-16 16:30   ` Catalin Marinas
2018-03-16 16:30     ` Catalin Marinas
2018-03-14 16:50 ` [PATCH v6 22/26] arm64: KVM: Allow far branches from vector slots to the main vectors Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-14 16:50 ` [PATCH v6 23/26] arm/arm64: KVM: Introduce EL2-specific executable mappings Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-15 15:03   ` Andrew Jones
2018-03-15 15:03     ` Andrew Jones
2018-03-15 15:53     ` Marc Zyngier
2018-03-15 15:53       ` Marc Zyngier
2018-03-14 16:50 ` [PATCH v6 24/26] arm64: Make BP hardening slot counter available Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-15 15:05   ` Andrew Jones
2018-03-15 15:05     ` Andrew Jones
2018-03-14 16:50 ` [PATCH v6 25/26] arm64: KVM: Allow mapping of vectors outside of the RAM region Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-15 15:54   ` Andrew Jones
2018-03-15 15:54     ` Andrew Jones
2018-03-15 16:17     ` Marc Zyngier
2018-03-15 16:17       ` Marc Zyngier
2018-03-15 17:08       ` Andrew Jones
2018-03-15 17:08         ` Andrew Jones
2018-03-15 18:47         ` Marc Zyngier
2018-03-15 18:47           ` Marc Zyngier
2018-03-16 12:33           ` Andrew Jones
2018-03-16 12:33             ` Andrew Jones
2018-03-14 16:50 ` [PATCH v6 26/26] arm64: Enable ARM64_HARDEN_EL2_VECTORS on Cortex-A57 and A72 Marc Zyngier
2018-03-14 16:50   ` Marc Zyngier
2018-03-15 15:57 ` [PATCH v6 00/26] KVM/arm64: Randomise EL2 mappings (variant 3a mitigation) Andrew Jones
2018-03-15 15:57   ` Andrew Jones
2018-03-15 16:19   ` Marc Zyngier
2018-03-15 16:19     ` Marc Zyngier
2018-03-15 16:40     ` Andrew Jones
2018-03-15 16:40       ` Andrew Jones
2018-03-15 16:52       ` Marc Zyngier
2018-03-15 16:52         ` Marc Zyngier
2018-03-16 17:46 ` Catalin Marinas
2018-03-16 17:46   ` Catalin Marinas
2018-03-16 18:05   ` Marc Zyngier
2018-03-16 18:05     ` Marc Zyngier

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.