linux-arm-kernel.lists.infradead.org archive mirror
 help / color / mirror / Atom feed
* [PATCH] KVM: arm64: Cap default IPA size to the host's own size
@ 2021-03-08 17:46 Marc Zyngier
  2021-03-09 11:09 ` Suzuki Poulose
                   ` (2 more replies)
  0 siblings, 3 replies; 10+ messages in thread
From: Marc Zyngier @ 2021-03-08 17:46 UTC (permalink / raw)
  To: kvmarm, linux-arm-kernel, kvm
  Cc: James Morse, Julien Thierry, Suzuki K Poulose, Alexandru Elisei,
	kernel-team

KVM/arm64 has forever used a 40bit default IPA space, partially
due to its 32bit heritage (where the only choice is 40bit).

However, there are implementations in the wild that have a *cough*
much smaller *cough* IPA space, which leads to a misprogramming of
VTCR_EL2, and a guest that is stuck on its first memory access
if userspace dares to ask for the default IPA setting (which most
VMMs do).

Instead, cap the default IPA size to what the host can actually
do, and spit out a one-off message on the console. The boot warning
is turned into a more meaningfull message, and the new behaviour
is also documented.

Although this is a userspace ABI change, it doesn't really change
much for userspace:

- the guest couldn't run before this change, while it now has
  a chance to if the memory range fits the reduced IPA space

- a memory slot that was accepted because it did fit the default
  IPA space but didn't fit the HW constraints is now properly
  rejected

The other thing that's left doing is to convince userspace to
actually use the IPA space setting instead of relying on the
antiquated default.

Signed-off-by: Marc Zyngier <maz@kernel.org>
---
 Documentation/virt/kvm/api.rst | 13 +++++++------
 arch/arm64/kvm/reset.c         | 12 ++++++++----
 2 files changed, 15 insertions(+), 10 deletions(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index aed52b0fc16e..80c710035f31 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -158,12 +158,13 @@ flag KVM_VM_MIPS_VZ.
 
 
 On arm64, the physical address size for a VM (IPA Size limit) is limited
-to 40bits by default. The limit can be configured if the host supports the
-extension KVM_CAP_ARM_VM_IPA_SIZE. When supported, use
+to 40bits by default, though capped to the host's limit. The VM's own
+limit can be configured if the host supports the extension
+KVM_CAP_ARM_VM_IPA_SIZE. When supported, use
 KVM_VM_TYPE_ARM_IPA_SIZE(IPA_Bits) to set the size in the machine type
-identifier, where IPA_Bits is the maximum width of any physical
-address used by the VM. The IPA_Bits is encoded in bits[7-0] of the
-machine type identifier.
+identifier, where IPA_Bits is the maximum width of any physical address
+used by the VM. The IPA_Bits is encoded in bits[7-0] of the machine type
+identifier.
 
 e.g, to configure a guest to use 48bit physical address size::
 
@@ -172,7 +173,7 @@ e.g, to configure a guest to use 48bit physical address size::
 The requested size (IPA_Bits) must be:
 
  ==   =========================================================
-  0   Implies default size, 40bits (for backward compatibility)
+  0   Implies default size, 40bits or less (for backward compatibility)
   N   Implies N bits, where N is a positive integer such that,
       32 <= N <= Host_IPA_Limit
  ==   =========================================================
diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c
index 47f3f035f3ea..1f22b36a0eff 100644
--- a/arch/arm64/kvm/reset.c
+++ b/arch/arm64/kvm/reset.c
@@ -324,10 +324,9 @@ int kvm_set_ipa_limit(void)
 	}
 
 	kvm_ipa_limit = id_aa64mmfr0_parange_to_phys_shift(parange);
-	WARN(kvm_ipa_limit < KVM_PHYS_SHIFT,
-	     "KVM IPA Size Limit (%d bits) is smaller than default size\n",
-	     kvm_ipa_limit);
-	kvm_info("IPA Size Limit: %d bits\n", kvm_ipa_limit);
+	kvm_info("IPA Size Limit: %d bits%s\n", kvm_ipa_limit,
+		 ((kvm_ipa_limit < KVM_PHYS_SHIFT) ?
+		  " (Reduced IPA size, limited VM compatibility)" : ""));
 
 	return 0;
 }
@@ -356,6 +355,11 @@ int kvm_arm_setup_stage2(struct kvm *kvm, unsigned long type)
 			return -EINVAL;
 	} else {
 		phys_shift = KVM_PHYS_SHIFT;
+		if (phys_shift > kvm_ipa_limit) {
+			pr_warn_once("Userspace using unsupported default IPA limit, capping to %d bits\n",
+				     kvm_ipa_limit);
+			phys_shift = kvm_ipa_limit;
+		}
 	}
 
 	mmfr0 = read_sanitised_ftr_reg(SYS_ID_AA64MMFR0_EL1);
-- 
2.30.0


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* Re: [PATCH] KVM: arm64: Cap default IPA size to the host's own size
  2021-03-08 17:46 [PATCH] KVM: arm64: Cap default IPA size to the host's own size Marc Zyngier
@ 2021-03-09 11:09 ` Suzuki Poulose
  2021-03-09 14:25   ` Marc Zyngier
  2021-03-09 11:26 ` Will Deacon
  2021-03-09 13:20 ` Andrew Jones
  2 siblings, 1 reply; 10+ messages in thread
From: Suzuki Poulose @ 2021-03-09 11:09 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: kvmarm, linux-arm-kernel, kvm, James Morse, Julien Thierry,
	Alexandru Elisei, kernel-team



> On 8 Mar 2021, at 17:46, Marc Zyngier <maz@kernel.org> wrote:
> 
> KVM/arm64 has forever used a 40bit default IPA space, partially
> due to its 32bit heritage (where the only choice is 40bit).
> 
> However, there are implementations in the wild that have a *cough*
> much smaller *cough* IPA space, which leads to a misprogramming of
> VTCR_EL2, and a guest that is stuck on its first memory access
> if userspace dares to ask for the default IPA setting (which most
> VMMs do).
> 
> Instead, cap the default IPA size to what the host can actually
> do, and spit out a one-off message on the console. The boot warning
> is turned into a more meaningfull message, and the new behaviour
> is also documented.
> 
> Although this is a userspace ABI change, it doesn't really change
> much for userspace:
> 
> - the guest couldn't run before this change, while it now has
>  a chance to if the memory range fits the reduced IPA space
> 
> - a memory slot that was accepted because it did fit the default
>  IPA space but didn't fit the HW constraints is now properly
>  rejected
> 
> The other thing that's left doing is to convince userspace to
> actually use the IPA space setting instead of relying on the
> antiquated default.
> 
> Signed-off-by: Marc Zyngier <maz@kernel.org>
> ---

Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>



_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] KVM: arm64: Cap default IPA size to the host's own size
  2021-03-08 17:46 [PATCH] KVM: arm64: Cap default IPA size to the host's own size Marc Zyngier
  2021-03-09 11:09 ` Suzuki Poulose
@ 2021-03-09 11:26 ` Will Deacon
  2021-03-09 11:35   ` Marc Zyngier
  2021-03-09 13:20 ` Andrew Jones
  2 siblings, 1 reply; 10+ messages in thread
From: Will Deacon @ 2021-03-09 11:26 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: kvmarm, linux-arm-kernel, kvm, James Morse, Julien Thierry,
	Suzuki K Poulose, Alexandru Elisei, kernel-team

On Mon, Mar 08, 2021 at 05:46:43PM +0000, Marc Zyngier wrote:
> KVM/arm64 has forever used a 40bit default IPA space, partially
> due to its 32bit heritage (where the only choice is 40bit).
> 
> However, there are implementations in the wild that have a *cough*
> much smaller *cough* IPA space, which leads to a misprogramming of
> VTCR_EL2, and a guest that is stuck on its first memory access
> if userspace dares to ask for the default IPA setting (which most
> VMMs do).
> 
> Instead, cap the default IPA size to what the host can actually
> do, and spit out a one-off message on the console. The boot warning
> is turned into a more meaningfull message, and the new behaviour
> is also documented.
> 
> Although this is a userspace ABI change, it doesn't really change
> much for userspace:
> 
> - the guest couldn't run before this change, while it now has
>   a chance to if the memory range fits the reduced IPA space
> 
> - a memory slot that was accepted because it did fit the default
>   IPA space but didn't fit the HW constraints is now properly
>   rejected
> 
> The other thing that's left doing is to convince userspace to
> actually use the IPA space setting instead of relying on the
> antiquated default.

Is there a way for userspace to discover the default IPA size, or does
it have to try setting values until it finds one that sticks?

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] KVM: arm64: Cap default IPA size to the host's own size
  2021-03-09 11:26 ` Will Deacon
@ 2021-03-09 11:35   ` Marc Zyngier
  2021-03-09 11:40     ` Will Deacon
  0 siblings, 1 reply; 10+ messages in thread
From: Marc Zyngier @ 2021-03-09 11:35 UTC (permalink / raw)
  To: Will Deacon
  Cc: kvmarm, linux-arm-kernel, kvm, James Morse, Julien Thierry,
	Suzuki K Poulose, Alexandru Elisei, kernel-team

On Tue, 09 Mar 2021 11:26:59 +0000,
Will Deacon <will@kernel.org> wrote:
> 
> On Mon, Mar 08, 2021 at 05:46:43PM +0000, Marc Zyngier wrote:
> > KVM/arm64 has forever used a 40bit default IPA space, partially
> > due to its 32bit heritage (where the only choice is 40bit).
> > 
> > However, there are implementations in the wild that have a *cough*
> > much smaller *cough* IPA space, which leads to a misprogramming of
> > VTCR_EL2, and a guest that is stuck on its first memory access
> > if userspace dares to ask for the default IPA setting (which most
> > VMMs do).
> > 
> > Instead, cap the default IPA size to what the host can actually
> > do, and spit out a one-off message on the console. The boot warning
> > is turned into a more meaningfull message, and the new behaviour
> > is also documented.
> > 
> > Although this is a userspace ABI change, it doesn't really change
> > much for userspace:
> > 
> > - the guest couldn't run before this change, while it now has
> >   a chance to if the memory range fits the reduced IPA space
> > 
> > - a memory slot that was accepted because it did fit the default
> >   IPA space but didn't fit the HW constraints is now properly
> >   rejected
> > 
> > The other thing that's left doing is to convince userspace to
> > actually use the IPA space setting instead of relying on the
> > antiquated default.
> 
> Is there a way for userspace to discover the default IPA size, or does
> it have to try setting values until it finds one that sticks?

Yes, since 233a7cb23531 ("kvm: arm64: Allow tuning the physical
address size for VM").

The VMM can issue a KVM_CAP_ARM_VM_IPA_SIZE ioctl(), and get in return
the maximum IPA size (I have a patch for kvmtool that does this).

	M.

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] KVM: arm64: Cap default IPA size to the host's own size
  2021-03-09 11:35   ` Marc Zyngier
@ 2021-03-09 11:40     ` Will Deacon
  0 siblings, 0 replies; 10+ messages in thread
From: Will Deacon @ 2021-03-09 11:40 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: kvmarm, linux-arm-kernel, kvm, James Morse, Julien Thierry,
	Suzuki K Poulose, Alexandru Elisei, kernel-team

On Tue, Mar 09, 2021 at 11:35:54AM +0000, Marc Zyngier wrote:
> On Tue, 09 Mar 2021 11:26:59 +0000,
> Will Deacon <will@kernel.org> wrote:
> > 
> > On Mon, Mar 08, 2021 at 05:46:43PM +0000, Marc Zyngier wrote:
> > > KVM/arm64 has forever used a 40bit default IPA space, partially
> > > due to its 32bit heritage (where the only choice is 40bit).
> > > 
> > > However, there are implementations in the wild that have a *cough*
> > > much smaller *cough* IPA space, which leads to a misprogramming of
> > > VTCR_EL2, and a guest that is stuck on its first memory access
> > > if userspace dares to ask for the default IPA setting (which most
> > > VMMs do).
> > > 
> > > Instead, cap the default IPA size to what the host can actually
> > > do, and spit out a one-off message on the console. The boot warning
> > > is turned into a more meaningfull message, and the new behaviour
> > > is also documented.
> > > 
> > > Although this is a userspace ABI change, it doesn't really change
> > > much for userspace:
> > > 
> > > - the guest couldn't run before this change, while it now has
> > >   a chance to if the memory range fits the reduced IPA space
> > > 
> > > - a memory slot that was accepted because it did fit the default
> > >   IPA space but didn't fit the HW constraints is now properly
> > >   rejected
> > > 
> > > The other thing that's left doing is to convince userspace to
> > > actually use the IPA space setting instead of relying on the
> > > antiquated default.
> > 
> > Is there a way for userspace to discover the default IPA size, or does
> > it have to try setting values until it finds one that sticks?
> 
> Yes, since 233a7cb23531 ("kvm: arm64: Allow tuning the physical
> address size for VM").
> 
> The VMM can issue a KVM_CAP_ARM_VM_IPA_SIZE ioctl(), and get in return
> the maximum IPA size (I have a patch for kvmtool that does this).

Great, thanks -- that's exactly what I was thinking about when I asked the
question!

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] KVM: arm64: Cap default IPA size to the host's own size
  2021-03-08 17:46 [PATCH] KVM: arm64: Cap default IPA size to the host's own size Marc Zyngier
  2021-03-09 11:09 ` Suzuki Poulose
  2021-03-09 11:26 ` Will Deacon
@ 2021-03-09 13:20 ` Andrew Jones
  2021-03-09 13:43   ` Marc Zyngier
  2 siblings, 1 reply; 10+ messages in thread
From: Andrew Jones @ 2021-03-09 13:20 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: kvmarm, linux-arm-kernel, kvm, James Morse, Julien Thierry,
	Suzuki K Poulose, Alexandru Elisei, kernel-team

Hi Marc,

On Mon, Mar 08, 2021 at 05:46:43PM +0000, Marc Zyngier wrote:
> KVM/arm64 has forever used a 40bit default IPA space, partially
> due to its 32bit heritage (where the only choice is 40bit).
> 
> However, there are implementations in the wild that have a *cough*
> much smaller *cough* IPA space, which leads to a misprogramming of
> VTCR_EL2, and a guest that is stuck on its first memory access
> if userspace dares to ask for the default IPA setting (which most
> VMMs do).
> 
> Instead, cap the default IPA size to what the host can actually
> do, and spit out a one-off message on the console. The boot warning
> is turned into a more meaningfull message, and the new behaviour
> is also documented.
> 
> Although this is a userspace ABI change, it doesn't really change
> much for userspace:
> 
> - the guest couldn't run before this change, while it now has
>   a chance to if the memory range fits the reduced IPA space
> 
> - a memory slot that was accepted because it did fit the default
>   IPA space but didn't fit the HW constraints is now properly
>   rejected

I'm not sure deferring the misconfiguration error until memslot
request time is better than just failing to create a VM. If
userspace doesn't use KVM_CAP_ARM_VM_IPA_SIZE to determine the
limit (which it hasn't been obliged to do) and it is able to
successfully create a VM, then it will assume up to 40-bit IPAs
are supported. Later, when it tries to add memslots and fails
it may be confused, especially if that later is much, much later
with memory hotplug.

> 
> The other thing that's left doing is to convince userspace to
> actually use the IPA space setting instead of relying on the
> antiquated default.

Failing to create any VM which hasn't selected a valid IPA limit
should be pretty convincing :-)

Thanks,
drew


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] KVM: arm64: Cap default IPA size to the host's own size
  2021-03-09 13:20 ` Andrew Jones
@ 2021-03-09 13:43   ` Marc Zyngier
  2021-03-09 14:29     ` Andrew Jones
  0 siblings, 1 reply; 10+ messages in thread
From: Marc Zyngier @ 2021-03-09 13:43 UTC (permalink / raw)
  To: Andrew Jones
  Cc: kvmarm, linux-arm-kernel, kvm, James Morse, Julien Thierry,
	Suzuki K Poulose, Alexandru Elisei, kernel-team

Hi Andrew,

On Tue, 09 Mar 2021 13:20:21 +0000,
Andrew Jones <drjones@redhat.com> wrote:
> 
> Hi Marc,
> 
> On Mon, Mar 08, 2021 at 05:46:43PM +0000, Marc Zyngier wrote:
> > KVM/arm64 has forever used a 40bit default IPA space, partially
> > due to its 32bit heritage (where the only choice is 40bit).
> > 
> > However, there are implementations in the wild that have a *cough*
> > much smaller *cough* IPA space, which leads to a misprogramming of
> > VTCR_EL2, and a guest that is stuck on its first memory access
> > if userspace dares to ask for the default IPA setting (which most
> > VMMs do).
> > 
> > Instead, cap the default IPA size to what the host can actually
> > do, and spit out a one-off message on the console. The boot warning
> > is turned into a more meaningfull message, and the new behaviour
> > is also documented.
> > 
> > Although this is a userspace ABI change, it doesn't really change
> > much for userspace:
> > 
> > - the guest couldn't run before this change, while it now has
> >   a chance to if the memory range fits the reduced IPA space
> > 
> > - a memory slot that was accepted because it did fit the default
> >   IPA space but didn't fit the HW constraints is now properly
> >   rejected
> 
> I'm not sure deferring the misconfiguration error until memslot
> request time is better than just failing to create a VM. If
> userspace doesn't use KVM_CAP_ARM_VM_IPA_SIZE to determine the
> limit (which it hasn't been obliged to do) and it is able to
> successfully create a VM, then it will assume up to 40-bit IPAs
> are supported. Later, when it tries to add memslots and fails
> it may be confused, especially if that later is much, much later
> with memory hotplug.

That's a fair point. However, no existing userspace will work on these
systems. Is that what we want to do? I don't care much, but having
non-usable defaults feel a bit... odd. I do spit out a warning, but I
agree this isn't great either.

> > The other thing that's left doing is to convince userspace to
> > actually use the IPA space setting instead of relying on the
> > antiquated default.
> 
> Failing to create any VM which hasn't selected a valid IPA limit
> should be pretty convincing :-)

I'll make sure to redirect the reports your way! :D

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] KVM: arm64: Cap default IPA size to the host's own size
  2021-03-09 11:09 ` Suzuki Poulose
@ 2021-03-09 14:25   ` Marc Zyngier
  0 siblings, 0 replies; 10+ messages in thread
From: Marc Zyngier @ 2021-03-09 14:25 UTC (permalink / raw)
  To: Suzuki Poulose
  Cc: kvmarm, linux-arm-kernel, kvm, James Morse, Julien Thierry,
	Alexandru Elisei, kernel-team

Hi Suzuki,

On Tue, 09 Mar 2021 11:09:48 +0000,
Suzuki Poulose <suzuki.poulose@arm.com> wrote:
> 
> 
> 
> > On 8 Mar 2021, at 17:46, Marc Zyngier <maz@kernel.org> wrote:
> > 
> > KVM/arm64 has forever used a 40bit default IPA space, partially
> > due to its 32bit heritage (where the only choice is 40bit).
> > 
> > However, there are implementations in the wild that have a *cough*
> > much smaller *cough* IPA space, which leads to a misprogramming of
> > VTCR_EL2, and a guest that is stuck on its first memory access
> > if userspace dares to ask for the default IPA setting (which most
> > VMMs do).
> > 
> > Instead, cap the default IPA size to what the host can actually
> > do, and spit out a one-off message on the console. The boot warning
> > is turned into a more meaningfull message, and the new behaviour
> > is also documented.
> > 
> > Although this is a userspace ABI change, it doesn't really change
> > much for userspace:
> > 
> > - the guest couldn't run before this change, while it now has
> >  a chance to if the memory range fits the reduced IPA space
> > 
> > - a memory slot that was accepted because it did fit the default
> >  IPA space but didn't fit the HW constraints is now properly
> >  rejected
> > 
> > The other thing that's left doing is to convince userspace to
> > actually use the IPA space setting instead of relying on the
> > antiquated default.
> > 
> > Signed-off-by: Marc Zyngier <maz@kernel.org>
> > ---
> 
> Reviewed-by: Suzuki K Poulose <suzuki.poulose@arm.com>

Thanks for that. Whilst I have your attention and given that you are
responsible for most of the variable IPA stuff... ;-)

I think we have another issue around the handling of our IPA
size. Let's say I create a VM with a 32bit IPA space. If I register a
2GB memslot at 0x8000000, I'm getting an error, which I think is
bogus.

I came to the conclusion that kvm_arch_prepare_memory_region() is a
bit overzealous when rejecting the memslot, and I used the following
patchlet to address it.

Does this seem sensible to you?

	M.

diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c
index 77cb2d28f2a4..8711894db8c2 100644
--- a/arch/arm64/kvm/mmu.c
+++ b/arch/arm64/kvm/mmu.c
@@ -1312,8 +1312,7 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm,
 	 * Prevent userspace from creating a memory region outside of the IPA
 	 * space addressable by the KVM guest IPA space.
 	 */
-	if (memslot->base_gfn + memslot->npages >=
-	    (kvm_phys_size(kvm) >> PAGE_SHIFT))
+	if ((memslot->base_gfn + memslot->npages) > (kvm_phys_size(kvm) >> PAGE_SHIFT))
 		return -EFAULT;
 
 	mmap_read_lock(current->mm);

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply related	[flat|nested] 10+ messages in thread

* Re: [PATCH] KVM: arm64: Cap default IPA size to the host's own size
  2021-03-09 13:43   ` Marc Zyngier
@ 2021-03-09 14:29     ` Andrew Jones
  2021-03-09 14:46       ` Marc Zyngier
  0 siblings, 1 reply; 10+ messages in thread
From: Andrew Jones @ 2021-03-09 14:29 UTC (permalink / raw)
  To: Marc Zyngier
  Cc: kvmarm, linux-arm-kernel, kvm, James Morse, Julien Thierry,
	Suzuki K Poulose, Alexandru Elisei, kernel-team

On Tue, Mar 09, 2021 at 01:43:40PM +0000, Marc Zyngier wrote:
> Hi Andrew,
> 
> On Tue, 09 Mar 2021 13:20:21 +0000,
> Andrew Jones <drjones@redhat.com> wrote:
> > 
> > Hi Marc,
> > 
> > On Mon, Mar 08, 2021 at 05:46:43PM +0000, Marc Zyngier wrote:
> > > KVM/arm64 has forever used a 40bit default IPA space, partially
> > > due to its 32bit heritage (where the only choice is 40bit).
> > > 
> > > However, there are implementations in the wild that have a *cough*
> > > much smaller *cough* IPA space, which leads to a misprogramming of
> > > VTCR_EL2, and a guest that is stuck on its first memory access
> > > if userspace dares to ask for the default IPA setting (which most
> > > VMMs do).
> > > 
> > > Instead, cap the default IPA size to what the host can actually
> > > do, and spit out a one-off message on the console. The boot warning
> > > is turned into a more meaningfull message, and the new behaviour
> > > is also documented.
> > > 
> > > Although this is a userspace ABI change, it doesn't really change
> > > much for userspace:
> > > 
> > > - the guest couldn't run before this change, while it now has
> > >   a chance to if the memory range fits the reduced IPA space
> > > 
> > > - a memory slot that was accepted because it did fit the default
> > >   IPA space but didn't fit the HW constraints is now properly
> > >   rejected
> > 
> > I'm not sure deferring the misconfiguration error until memslot
> > request time is better than just failing to create a VM. If
> > userspace doesn't use KVM_CAP_ARM_VM_IPA_SIZE to determine the
> > limit (which it hasn't been obliged to do) and it is able to
> > successfully create a VM, then it will assume up to 40-bit IPAs
> > are supported. Later, when it tries to add memslots and fails
> > it may be confused, especially if that later is much, much later
> > with memory hotplug.
> 
> That's a fair point. However, no existing userspace will work on these
> systems. Is that what we want to do? I don't care much, but having
> non-usable defaults feel a bit... odd. I do spit out a warning, but I
> agree this isn't great either.

I can send patches for QEMU, KVM selftests, and maybe even rust-vmm.
Can you point me to something about these systems I can reference
in my postings? Or I can just reference this mail thread.

> 
> > > The other thing that's left doing is to convince userspace to
> > > actually use the IPA space setting instead of relying on the
> > > antiquated default.
> > 
> > Failing to create any VM which hasn't selected a valid IPA limit
> > should be pretty convincing :-)
> 
> I'll make sure to redirect the reports your way! :D

What's the current error message when this occurs? Is it good enough, or
should we improve it to help provide people hints? Please don't change
it to "Invalid IPA limit, please mail Andrew Jones" :-)

Thanks,
drew


_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 10+ messages in thread

* Re: [PATCH] KVM: arm64: Cap default IPA size to the host's own size
  2021-03-09 14:29     ` Andrew Jones
@ 2021-03-09 14:46       ` Marc Zyngier
  0 siblings, 0 replies; 10+ messages in thread
From: Marc Zyngier @ 2021-03-09 14:46 UTC (permalink / raw)
  To: Andrew Jones
  Cc: kvmarm, linux-arm-kernel, kvm, James Morse, Julien Thierry,
	Suzuki K Poulose, Alexandru Elisei, kernel-team

On Tue, 09 Mar 2021 14:29:10 +0000,
Andrew Jones <drjones@redhat.com> wrote:
> 
> On Tue, Mar 09, 2021 at 01:43:40PM +0000, Marc Zyngier wrote:
> > Hi Andrew,
> > 
> > On Tue, 09 Mar 2021 13:20:21 +0000,
> > Andrew Jones <drjones@redhat.com> wrote:
> > > 
> > > Hi Marc,
> > > 
> > > On Mon, Mar 08, 2021 at 05:46:43PM +0000, Marc Zyngier wrote:
> > > > KVM/arm64 has forever used a 40bit default IPA space, partially
> > > > due to its 32bit heritage (where the only choice is 40bit).
> > > > 
> > > > However, there are implementations in the wild that have a *cough*
> > > > much smaller *cough* IPA space, which leads to a misprogramming of
> > > > VTCR_EL2, and a guest that is stuck on its first memory access
> > > > if userspace dares to ask for the default IPA setting (which most
> > > > VMMs do).
> > > > 
> > > > Instead, cap the default IPA size to what the host can actually
> > > > do, and spit out a one-off message on the console. The boot warning
> > > > is turned into a more meaningfull message, and the new behaviour
> > > > is also documented.
> > > > 
> > > > Although this is a userspace ABI change, it doesn't really change
> > > > much for userspace:
> > > > 
> > > > - the guest couldn't run before this change, while it now has
> > > >   a chance to if the memory range fits the reduced IPA space
> > > > 
> > > > - a memory slot that was accepted because it did fit the default
> > > >   IPA space but didn't fit the HW constraints is now properly
> > > >   rejected
> > > 
> > > I'm not sure deferring the misconfiguration error until memslot
> > > request time is better than just failing to create a VM. If
> > > userspace doesn't use KVM_CAP_ARM_VM_IPA_SIZE to determine the
> > > limit (which it hasn't been obliged to do) and it is able to
> > > successfully create a VM, then it will assume up to 40-bit IPAs
> > > are supported. Later, when it tries to add memslots and fails
> > > it may be confused, especially if that later is much, much later
> > > with memory hotplug.
> > 
> > That's a fair point. However, no existing userspace will work on these
> > systems. Is that what we want to do? I don't care much, but having
> > non-usable defaults feel a bit... odd. I do spit out a warning, but I
> > agree this isn't great either.
> 
> I can send patches for QEMU, KVM selftests, and maybe even rust-vmm.
> Can you point me to something about these systems I can reference
> in my postings? Or I can just reference this mail thread.

The system of choice to see this is an Apple M1 box. Not supported in
mainline yet, but things are progressing pretty quickly.

> 
> > 
> > > > The other thing that's left doing is to convince userspace to
> > > > actually use the IPA space setting instead of relying on the
> > > > antiquated default.
> > > 
> > > Failing to create any VM which hasn't selected a valid IPA limit
> > > should be pretty convincing :-)
> > 
> > I'll make sure to redirect the reports your way! :D
> 
> What's the current error message when this occurs? Is it good enough, or
> should we improve it to help provide people hints? Please don't change
> it to "Invalid IPA limit, please mail Andrew Jones" :-)

Well, that's part of the problem. Currently, you don't get a message,
and the guest faults on its first memory access forever (level 0
translation fault), as the VTCR_EL2.T0SZ value is bogus.

I can change this patch to reject 40bit IPA when requested as a
default with something saying "Userspace using unsupported default IPA
limit, upgrade your VMM".

Now, there is another nit[1] which I just found with my kvmtool setup
that computes the optimal IPA space for a given VM. And that one is
even more problematic...

Thanks,

	M.

[1] https://lore.kernel.org/r/87lfawxv40.wl-maz@kernel.org

-- 
Without deviation from the norm, progress is not possible.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

^ permalink raw reply	[flat|nested] 10+ messages in thread

end of thread, other threads:[~2021-03-09 14:48 UTC | newest]

Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-03-08 17:46 [PATCH] KVM: arm64: Cap default IPA size to the host's own size Marc Zyngier
2021-03-09 11:09 ` Suzuki Poulose
2021-03-09 14:25   ` Marc Zyngier
2021-03-09 11:26 ` Will Deacon
2021-03-09 11:35   ` Marc Zyngier
2021-03-09 11:40     ` Will Deacon
2021-03-09 13:20 ` Andrew Jones
2021-03-09 13:43   ` Marc Zyngier
2021-03-09 14:29     ` Andrew Jones
2021-03-09 14:46       ` Marc Zyngier

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).