[PATCH] arm64: kvm: fix IDMAP overlap with HYP VA

[PATCH] arm64: kvm: fix IDMAP overlap with HYP VA

[Russell King]

Booting 5.4 on LX2160A reveals that KVM is non-functional:

kvm: Limiting the IPA size due to kernel Virtual Address limit
kvm [1]: IPA Size Limit: 43bits
kvm [1]: IDMAP intersecting with HYP VA, unable to continue
kvm [1]: error initializing Hyp mode: -22

Debugging shows:

kvm [1]: IDMAP page: 81a26000
kvm [1]: HYP VA range: 0:22ffffffff

as RAM is located at:

80000000-fbdfffff : System RAM
2080000000-237fffffff : System RAM

Comparing this with the same kernel on Armada 8040 shows:

kvm: Limiting the IPA size due to kernel Virtual Address limit
kvm [1]: IPA Size Limit: 43bits
kvm [1]: IDMAP page: 2a26000
kvm [1]: HYP VA range: 4800000000:493fffffff
...
kvm [1]: Hyp mode initialized successfully

which indicates that hyp_va_msb is set, and is always set to the
opposite value of the idmap page to avoid the overlap. This does not
happen with the LX2160A.

Further debugging shows vabits_actual = 39, kva_msb = 38 on LX2160A and
kva_msb = 33 on Armada 8040. Looking at the bit layout of the HYP VA,
there is still one bit available for hyp_va_msb. Set this bit
appropriately. This allows kvm to be functional on the LX2160A, but
without any HYP VA randomisation:

kvm: Limiting the IPA size due to kernel Virtual Address limit
kvm [1]: IPA Size Limit: 43bits
kvm [1]: IDMAP page: 81a24000
kvm [1]: HYP VA range: 4000000000:62ffffffff
...
kvm [1]: Hyp mode initialized successfully

Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
---
 arch/arm64/kvm/va_layout.c | 22 +++++++++++++++-------
 1 file changed, 15 insertions(+), 7 deletions(-)

diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c
index 2cf7d4b606c3..83f8b3f51cf4 100644
--- a/arch/arm64/kvm/va_layout.c
+++ b/arch/arm64/kvm/va_layout.c
@@ -22,6 +22,17 @@ static u8 tag_lsb;
 static u64 tag_val;
 static u64 va_mask;
 
+/*
+ * We want to generate a hyp VA with the following format:
+ *
+ *  63 ... V |     V-1    | V-2 .. tag_lsb | tag_lsb - 1 .. 0
+ *  ---------------------------------------------------------
+ * | 0000000 | hyp_va_msb |    random tag  |  kern linear VA |
+ *
+ * which does not conflict with the idmap regions. This means that hyp_va_msb
+ * must always be present. Luckily, when kva_msb == (vabits_actual - 1) we
+ * still have one bit for this, but no bits for the random tag.
+ */
 static void compute_layout(void)
 {
 	phys_addr_t idmap_addr = __pa_symbol(__hyp_idmap_text_start);
@@ -39,19 +50,16 @@ static void compute_layout(void)
 		/*
 		 * No space in the address, let's compute the mask so
 		 * that it covers (vabits_actual - 1) bits, and the region
-		 * bit. The tag stays set to zero.
+		 * bit.
 		 */
-		va_mask  = BIT(vabits_actual - 1) - 1;
-		va_mask |= hyp_va_msb;
+		tag_lsb = kva_msb;
+		va_mask = BIT(vabits_actual - 1) - 1;
+		tag_val = hyp_va_msb >> tag_lsb;
 	} else {
 		/*
 		 * We do have some free bits to insert a random tag.
 		 * Hyp VAs are now created from kernel linear map VAs
 		 * using the following formula (with V == vabits_actual):
-		 *
-		 *  63 ... V |     V-1    | V-2 .. tag_lsb | tag_lsb - 1 .. 0
-		 *  ---------------------------------------------------------
-		 * | 0000000 | hyp_va_msb |    random tag  |  kern linear VA |
 		 */
 		tag_lsb = kva_msb;
 		va_mask = GENMASK_ULL(tag_lsb - 1, 0);
-- 
2.20.1


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH] arm64: kvm: fix IDMAP overlap with HYP VA

[Russell King - ARM Linux admin]

On Fri, Dec 27, 2019 at 11:47:35AM +0000, Russell King wrote:
> Booting 5.4 on LX2160A reveals that KVM is non-functional:
> 
> kvm: Limiting the IPA size due to kernel Virtual Address limit
> kvm [1]: IPA Size Limit: 43bits
> kvm [1]: IDMAP intersecting with HYP VA, unable to continue
> kvm [1]: error initializing Hyp mode: -22
> 
> Debugging shows:
> 
> kvm [1]: IDMAP page: 81a26000
> kvm [1]: HYP VA range: 0:22ffffffff
> 
> as RAM is located at:
> 
> 80000000-fbdfffff : System RAM
> 2080000000-237fffffff : System RAM
> 
> Comparing this with the same kernel on Armada 8040 shows:
> 
> kvm: Limiting the IPA size due to kernel Virtual Address limit
> kvm [1]: IPA Size Limit: 43bits
> kvm [1]: IDMAP page: 2a26000
> kvm [1]: HYP VA range: 4800000000:493fffffff
> ...
> kvm [1]: Hyp mode initialized successfully
> 
> which indicates that hyp_va_msb is set, and is always set to the
> opposite value of the idmap page to avoid the overlap. This does not
> happen with the LX2160A.
> 
> Further debugging shows vabits_actual = 39, kva_msb = 38 on LX2160A and
> kva_msb = 33 on Armada 8040. Looking at the bit layout of the HYP VA,
> there is still one bit available for hyp_va_msb. Set this bit
> appropriately. This allows kvm to be functional on the LX2160A, but
> without any HYP VA randomisation:
> 
> kvm: Limiting the IPA size due to kernel Virtual Address limit
> kvm [1]: IPA Size Limit: 43bits
> kvm [1]: IDMAP page: 81a24000
> kvm [1]: HYP VA range: 4000000000:62ffffffff
> ...
> kvm [1]: Hyp mode initialized successfully
> 
> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
> ---

I assume it's not possible for kva_msb >= vabits_actual ?

>  arch/arm64/kvm/va_layout.c | 22 +++++++++++++++-------
>  1 file changed, 15 insertions(+), 7 deletions(-)
> 
> diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c
> index 2cf7d4b606c3..83f8b3f51cf4 100644
> --- a/arch/arm64/kvm/va_layout.c
> +++ b/arch/arm64/kvm/va_layout.c
> @@ -22,6 +22,17 @@ static u8 tag_lsb;
>  static u64 tag_val;
>  static u64 va_mask;
>  
> +/*
> + * We want to generate a hyp VA with the following format:
> + *
> + *  63 ... V |     V-1    | V-2 .. tag_lsb | tag_lsb - 1 .. 0
> + *  ---------------------------------------------------------
> + * | 0000000 | hyp_va_msb |    random tag  |  kern linear VA |
> + *
> + * which does not conflict with the idmap regions. This means that hyp_va_msb
> + * must always be present. Luckily, when kva_msb == (vabits_actual - 1) we
> + * still have one bit for this, but no bits for the random tag.
> + */
>  static void compute_layout(void)
>  {
>  	phys_addr_t idmap_addr = __pa_symbol(__hyp_idmap_text_start);
> @@ -39,19 +50,16 @@ static void compute_layout(void)
>  		/*
>  		 * No space in the address, let's compute the mask so
>  		 * that it covers (vabits_actual - 1) bits, and the region
> -		 * bit. The tag stays set to zero.
> +		 * bit.
>  		 */
> -		va_mask  = BIT(vabits_actual - 1) - 1;
> -		va_mask |= hyp_va_msb;
> +		tag_lsb = kva_msb;
> +		va_mask = BIT(vabits_actual - 1) - 1;
> +		tag_val = hyp_va_msb >> tag_lsb;
>  	} else {
>  		/*
>  		 * We do have some free bits to insert a random tag.
>  		 * Hyp VAs are now created from kernel linear map VAs
>  		 * using the following formula (with V == vabits_actual):
> -		 *
> -		 *  63 ... V |     V-1    | V-2 .. tag_lsb | tag_lsb - 1 .. 0
> -		 *  ---------------------------------------------------------
> -		 * | 0000000 | hyp_va_msb |    random tag  |  kern linear VA |
>  		 */
>  		tag_lsb = kva_msb;
>  		va_mask = GENMASK_ULL(tag_lsb - 1, 0);
> -- 
> 2.20.1
> 
> 

-- 
RMK's Patch system: https://www.armlinux.org.uk/developer/patches/
FTTC broadband for 0.8mile line in suburbia: sync at 12.1Mbps down 622kbps up
According to speedtest.net: 11.9Mbps down 500kbps up

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH] arm64: kvm: fix IDMAP overlap with HYP VA

[Marc Zyngier]

Hi Russell,

Thanks for this.

On Fri, 27 Dec 2019 11:47:35 +0000,
Russell King <rmk+kernel@armlinux.org.uk> wrote:
> 
> Booting 5.4 on LX2160A reveals that KVM is non-functional:
> 
> kvm: Limiting the IPA size due to kernel Virtual Address limit
> kvm [1]: IPA Size Limit: 43bits
> kvm [1]: IDMAP intersecting with HYP VA, unable to continue
> kvm [1]: error initializing Hyp mode: -22
> 
> Debugging shows:
> 
> kvm [1]: IDMAP page: 81a26000
> kvm [1]: HYP VA range: 0:22ffffffff
> 
> as RAM is located at:
> 
> 80000000-fbdfffff : System RAM
> 2080000000-237fffffff : System RAM

Ouch. This looks like a terrible choice for a memory map.

> 
> Comparing this with the same kernel on Armada 8040 shows:
> 
> kvm: Limiting the IPA size due to kernel Virtual Address limit
> kvm [1]: IPA Size Limit: 43bits
> kvm [1]: IDMAP page: 2a26000
> kvm [1]: HYP VA range: 4800000000:493fffffff
> ...
> kvm [1]: Hyp mode initialized successfully
> 
> which indicates that hyp_va_msb is set, and is always set to the
> opposite value of the idmap page to avoid the overlap. This does not
> happen with the LX2160A.
> 
> Further debugging shows vabits_actual = 39, kva_msb = 38 on LX2160A and
> kva_msb = 33 on Armada 8040. Looking at the bit layout of the HYP VA,
> there is still one bit available for hyp_va_msb. Set this bit
> appropriately. This allows kvm to be functional on the LX2160A, but
> without any HYP VA randomisation:
> 
> kvm: Limiting the IPA size due to kernel Virtual Address limit
> kvm [1]: IPA Size Limit: 43bits
> kvm [1]: IDMAP page: 81a24000
> kvm [1]: HYP VA range: 4000000000:62ffffffff
> ...
> kvm [1]: Hyp mode initialized successfully

Nice bit of debugging. I guess part of the confusion is due to the
fact that the hyp_va_msb is part of the tag (as you found out), but
that the documentation doesn't really make that clear at all (and only
mentions the random part of the tag).

> 
> Signed-off-by: Russell King <rmk+kernel@armlinux.org.uk>
> ---
>  arch/arm64/kvm/va_layout.c | 22 +++++++++++++++-------
>  1 file changed, 15 insertions(+), 7 deletions(-)
> 
> diff --git a/arch/arm64/kvm/va_layout.c b/arch/arm64/kvm/va_layout.c
> index 2cf7d4b606c3..83f8b3f51cf4 100644
> --- a/arch/arm64/kvm/va_layout.c
> +++ b/arch/arm64/kvm/va_layout.c
> @@ -22,6 +22,17 @@ static u8 tag_lsb;
>  static u64 tag_val;
>  static u64 va_mask;
>  
> +/*
> + * We want to generate a hyp VA with the following format:
> + *
> + *  63 ... V |     V-1    | V-2 .. tag_lsb | tag_lsb - 1 .. 0
> + *  ---------------------------------------------------------
> + * | 0000000 | hyp_va_msb |    random tag  |  kern linear VA |
                \---------- tag -------------/

How about the above, to make it clearer that the tag must include the
hyp_va_msb bit to avoid clashing with the IDMAP?

> + *
> + * which does not conflict with the idmap regions. This means that hyp_va_msb
> + * must always be present. Luckily, when kva_msb == (vabits_actual - 1) we

I'm not sure the "Luckily" part is appropriate here. Given the way
kva_msb is computed, I can't see how it can see how this can
fail.

This stems from the fact that the vabits space for kernel mappings is
split in two: vabits-1 for the linear map, and vabits-1 for the rest
(kernel text and co). Given that we define kva_msb as the highest
order bit that can change within the linear map, its value is at most
vabits-1.

> + * still have one bit for this, but no bits for the random tag.
> + */
>  static void compute_layout(void)
>  {
>  	phys_addr_t idmap_addr = __pa_symbol(__hyp_idmap_text_start);
> @@ -39,19 +50,16 @@ static void compute_layout(void)
>  		/*
>  		 * No space in the address, let's compute the mask so
>  		 * that it covers (vabits_actual - 1) bits, and the region
> -		 * bit. The tag stays set to zero.
> +		 * bit.
>  		 */
> -		va_mask  = BIT(vabits_actual - 1) - 1;
> -		va_mask |= hyp_va_msb;
> +		tag_lsb = kva_msb;
> +		va_mask = BIT(vabits_actual - 1) - 1;
> +		tag_val = hyp_va_msb >> tag_lsb;
>  	} else {
>  		/*
>  		 * We do have some free bits to insert a random tag.
>  		 * Hyp VAs are now created from kernel linear map VAs
>  		 * using the following formula (with V == vabits_actual):
> -		 *
> -		 *  63 ... V |     V-1    | V-2 .. tag_lsb | tag_lsb - 1 .. 0
> -		 *  ---------------------------------------------------------
> -		 * | 0000000 | hyp_va_msb |    random tag  |  kern linear VA |
>  		 */
>  		tag_lsb = kva_msb;
>  		va_mask = GENMASK_ULL(tag_lsb - 1, 0);

In the light of this, it'd be great to rework this code to simplify it
(getting rid of kva_msb, which really is tag_lsb) and make some of the
computing common to both branches.

Thanks,

	M.

-- 
Jazz is not dead, it just smells funny.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel