[PATCH v2 1/2] KVM: arm64: mixed-width check should be skipped for uninitialized vCPUs

[PATCH v2 1/2] KVM: arm64: mixed-width check should be skipped for uninitialized vCPUs

[Reiji Watanabe]

KVM allows userspace to configure either all 32bit or 64bit vCPUs
for a guest.  At vCPU reset, vcpu_allowed_register_width() checks
if the vcpu's register width is consistent with all other vCPUs'.
Since the checking is done even against vCPUs that are not initialized
(KVM_ARM_VCPU_INIT has not been done) yet, the uninitialized vCPUs
are erroneously treated as 64bit vCPU, which causes the function to
incorrectly detect a mixed-width VM.

Introduce a new flag (el1_reg_width) in kvm_arch to indicates that
the guest needs to be configured with all 32bit or 64bit vCPUs,
and initialize it at the first KVM_ARM_VCPU_INIT for the guest.
Check vcpu's register width against the flag at the vcpu's
KVM_ARM_VCPU_INIT (instead of against other vCPUs' register width).

Fixes: 66e94d5cafd4 ("KVM: arm64: Prevent mixed-width VM creation")
Signed-off-by: Reiji Watanabe <reijiw@google.com>
---
 arch/arm64/include/asm/kvm_host.h | 13 +++++++++++++
 arch/arm64/kvm/arm.c              | 30 ++++++++++++++++++++++++++++++
 arch/arm64/kvm/reset.c            |  8 --------
 3 files changed, 43 insertions(+), 8 deletions(-)

diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
index 2a5f7f38006f..c02b7caf2c82 100644
--- a/arch/arm64/include/asm/kvm_host.h
+++ b/arch/arm64/include/asm/kvm_host.h
@@ -102,6 +102,12 @@ struct kvm_s2_mmu {
 struct kvm_arch_memory_slot {
 };
 
+enum kvm_el1_reg_width {
+	EL1_WIDTH_UNINITIALIZED = 0,
+	EL1_32BIT,
+	EL1_64BIT,
+};
+
 struct kvm_arch {
 	struct kvm_s2_mmu mmu;
 
@@ -137,6 +143,13 @@ struct kvm_arch {
 
 	/* Memory Tagging Extension enabled for the guest */
 	bool mte_enabled;
+
+	/*
+	 * EL1 register width for the guest.
+	 * This is set at the first KVM_ARM_VCPU_INIT for the guest based
+	 * on whether the vcpu has KVM_ARM_VCPU_EL1_32BIT or not.
+	 */
+	enum kvm_el1_reg_width el1_reg_width;
 };
 
 struct kvm_vcpu_fault_info {
diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
index e4727dc771bf..54ae8bf9d187 100644
--- a/arch/arm64/kvm/arm.c
+++ b/arch/arm64/kvm/arm.c
@@ -1058,6 +1058,32 @@ int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_level,
 	return -EINVAL;
 }
 
+/*
+ * A guest can have either all 32bit or 64bit vcpus only.
+ * Either one the guest has is indicated in kvm->arch.el1_reg_width.
+ * Check if the vcpu's register width is consistent with
+ * kvm->arch.el1_reg_width.  If kvm->arch.el1_reg_width is not set yet,
+ * set it based on the vcpu's KVM_ARM_VCPU_EL1_32BIT configuration.
+ */
+static int kvm_register_width_check_or_init(struct kvm_vcpu *vcpu)
+{
+	bool is32bit;
+	bool allowed = true;
+	struct kvm *kvm = vcpu->kvm;
+
+	is32bit = vcpu_has_feature(vcpu, KVM_ARM_VCPU_EL1_32BIT);
+
+	mutex_lock(&kvm->lock);
+
+	if (kvm->arch.el1_reg_width == EL1_WIDTH_UNINITIALIZED)
+		kvm->arch.el1_reg_width = is32bit ? EL1_32BIT : EL1_64BIT;
+	else
+		allowed = (is32bit == (kvm->arch.el1_reg_width == EL1_32BIT));
+
+	mutex_unlock(&kvm->lock);
+	return allowed ? 0 : -EINVAL;
+}
+
 static int kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
 			       const struct kvm_vcpu_init *init)
 {
@@ -1097,6 +1123,10 @@ static int kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
 
 	/* Now we know what it is, we can reset it. */
 	ret = kvm_reset_vcpu(vcpu);
+
+	if (!ret)
+		ret = kvm_register_width_check_or_init(vcpu);
+
 	if (ret) {
 		vcpu->arch.target = -1;
 		bitmap_zero(vcpu->arch.features, KVM_VCPU_MAX_FEATURES);
diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c
index 426bd7fbc3fd..dbf2939a6a96 100644
--- a/arch/arm64/kvm/reset.c
+++ b/arch/arm64/kvm/reset.c
@@ -168,9 +168,7 @@ static int kvm_vcpu_enable_ptrauth(struct kvm_vcpu *vcpu)
 
 static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu)
 {
-	struct kvm_vcpu *tmp;
 	bool is32bit;
-	int i;
 
 	is32bit = vcpu_has_feature(vcpu, KVM_ARM_VCPU_EL1_32BIT);
 	if (!cpus_have_const_cap(ARM64_HAS_32BIT_EL1) && is32bit)
@@ -180,12 +178,6 @@ static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu)
 	if (kvm_has_mte(vcpu->kvm) && is32bit)
 		return false;
 
-	/* Check that the vcpus are either all 32bit or all 64bit */
-	kvm_for_each_vcpu(i, tmp, vcpu->kvm) {
-		if (vcpu_has_feature(tmp, KVM_ARM_VCPU_EL1_32BIT) != is32bit)
-			return false;
-	}
-
 	return true;
 }
 

base-commit: 37144b2c855f9311c72f292125061d4a52d02856
-- 
2.34.1.703.g22d0c6ccf7-goog

[PATCH v2 2/2] KVM: arm64: selftests: Introduce vcpu_width_config

[Reiji Watanabe]

Introduce a test for aarch64 that ensures non-mixed-width vCPUs
(all 64bit vCPUs or all 32bit vcPUs) can be configured, and
mixed-width vCPUs cannot be configured.

Reviewed-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Reiji Watanabe <reijiw@google.com>
---
 tools/testing/selftests/kvm/.gitignore        |   1 +
 tools/testing/selftests/kvm/Makefile          |   1 +
 .../selftests/kvm/aarch64/vcpu_width_config.c | 125 ++++++++++++++++++
 3 files changed, 127 insertions(+)
 create mode 100644 tools/testing/selftests/kvm/aarch64/vcpu_width_config.c

diff --git a/tools/testing/selftests/kvm/.gitignore b/tools/testing/selftests/kvm/.gitignore
index 3cb5ac5da087..8795a83cc382 100644
--- a/tools/testing/selftests/kvm/.gitignore
+++ b/tools/testing/selftests/kvm/.gitignore
@@ -3,6 +3,7 @@
 /aarch64/debug-exceptions
 /aarch64/get-reg-list
 /aarch64/psci_cpu_on_test
+/aarch64/vcpu_width_config
 /aarch64/vgic_init
 /s390x/memop
 /s390x/resets
diff --git a/tools/testing/selftests/kvm/Makefile b/tools/testing/selftests/kvm/Makefile
index 17342b575e85..259e01d0735a 100644
--- a/tools/testing/selftests/kvm/Makefile
+++ b/tools/testing/selftests/kvm/Makefile
@@ -95,6 +95,7 @@ TEST_GEN_PROGS_aarch64 += aarch64/arch_timer
 TEST_GEN_PROGS_aarch64 += aarch64/debug-exceptions
 TEST_GEN_PROGS_aarch64 += aarch64/get-reg-list
 TEST_GEN_PROGS_aarch64 += aarch64/psci_cpu_on_test
+TEST_GEN_PROGS_aarch64 += aarch64/vcpu_width_config
 TEST_GEN_PROGS_aarch64 += aarch64/vgic_init
 TEST_GEN_PROGS_aarch64 += demand_paging_test
 TEST_GEN_PROGS_aarch64 += dirty_log_test
diff --git a/tools/testing/selftests/kvm/aarch64/vcpu_width_config.c b/tools/testing/selftests/kvm/aarch64/vcpu_width_config.c
new file mode 100644
index 000000000000..cd238e068236
--- /dev/null
+++ b/tools/testing/selftests/kvm/aarch64/vcpu_width_config.c
@@ -0,0 +1,125 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * vcpu_width_config - Test KVM_ARM_VCPU_INIT() with KVM_ARM_VCPU_EL1_32BIT.
+ *
+ * Copyright (c) 2022 Google LLC.
+ *
+ * This is a test that ensures that non-mixed-width vCPUs (all 64bit vCPUs
+ * or all 32bit vcPUs) can be configured and mixed-width vCPUs cannot be
+ * configured.
+ */
+
+#define _GNU_SOURCE
+
+#include "kvm_util.h"
+#include "processor.h"
+#include "test_util.h"
+
+
+/*
+ * Add a vCPU, run KVM_ARM_VCPU_INIT with @init1, and then
+ * add another vCPU, and run KVM_ARM_VCPU_INIT with @init2.
+ */
+int add_init_2vcpus(struct kvm_vcpu_init *init1,
+			 struct kvm_vcpu_init *init2)
+{
+	struct kvm_vm *vm;
+	int ret;
+
+	vm = vm_create(VM_MODE_DEFAULT, DEFAULT_GUEST_PHY_PAGES, O_RDWR);
+
+	vm_vcpu_add(vm, 0);
+	ret = _vcpu_ioctl(vm, 0, KVM_ARM_VCPU_INIT, init1);
+	if (ret)
+		goto free_exit;
+
+	vm_vcpu_add(vm, 1);
+	ret = _vcpu_ioctl(vm, 1, KVM_ARM_VCPU_INIT, init2);
+
+free_exit:
+	kvm_vm_free(vm);
+	return ret;
+}
+
+/*
+ * Add two vCPUs, then run KVM_ARM_VCPU_INIT for one vCPU with @init1,
+ * and run KVM_ARM_VCPU_INIT for another vCPU with @init2.
+ */
+int add_2vcpus_init_2vcpus(struct kvm_vcpu_init *init1,
+				struct kvm_vcpu_init *init2)
+{
+	struct kvm_vm *vm;
+	int ret;
+
+	vm = vm_create(VM_MODE_DEFAULT, DEFAULT_GUEST_PHY_PAGES, O_RDWR);
+
+	vm_vcpu_add(vm, 0);
+	vm_vcpu_add(vm, 1);
+
+	ret = _vcpu_ioctl(vm, 0, KVM_ARM_VCPU_INIT, init1);
+	if (ret)
+		goto free_exit;
+
+	ret = _vcpu_ioctl(vm, 1, KVM_ARM_VCPU_INIT, init2);
+
+free_exit:
+	kvm_vm_free(vm);
+	return ret;
+}
+
+/*
+ * Tests that two 64bit vCPUs can be configured, two 32bit vCPUs can be
+ * configured, and two mixed-witgh vCPUs cannot be configured.
+ * Each of those three cases, configure vCPUs in two different orders.
+ * The one is running KVM_CREATE_VCPU for 2 vCPUs, and then running
+ * KVM_ARM_VCPU_INIT for them.
+ * The other is running KVM_CREATE_VCPU and KVM_ARM_VCPU_INIT for a vCPU,
+ * and then run those commands for another vCPU.
+ */
+int main(void)
+{
+	struct kvm_vcpu_init init1, init2;
+	struct kvm_vm *vm;
+	int ret;
+
+	if (kvm_check_cap(KVM_CAP_ARM_EL1_32BIT) <= 0) {
+		print_skip("KVM_CAP_ARM_EL1_32BIT is not supported");
+		exit(KSFT_SKIP);
+	}
+
+	/* Get the preferred target type and copy that to init2 */
+	vm = vm_create(VM_MODE_DEFAULT, DEFAULT_GUEST_PHY_PAGES, O_RDWR);
+	vm_ioctl(vm, KVM_ARM_PREFERRED_TARGET, &init1);
+	kvm_vm_free(vm);
+	memcpy(&init2, &init1, sizeof(init2));
+
+	/* Test with 64bit vCPUs */
+	ret = add_init_2vcpus(&init1, &init2);
+	TEST_ASSERT(ret == 0,
+		    "Configuring 64bit EL1 vCPUs failed unexpectedly");
+	ret = add_2vcpus_init_2vcpus(&init1, &init2);
+	TEST_ASSERT(ret == 0,
+		    "Configuring 64bit EL1 vCPUs failed unexpectedly");
+
+	/* Test with 32bit vCPUs */
+	init1.features[0] = (1 << KVM_ARM_VCPU_EL1_32BIT);
+	init2.features[0] = (1 << KVM_ARM_VCPU_EL1_32BIT);
+	ret = add_init_2vcpus(&init1, &init2);
+	TEST_ASSERT(ret == 0,
+		    "Configuring 32bit EL1 vCPUs failed unexpectedly");
+	ret = add_2vcpus_init_2vcpus(&init1, &init2);
+	TEST_ASSERT(ret == 0,
+		    "Configuring 32bit EL1 vCPUs failed unexpectedly");
+
+	/* Test with mixed-width vCPUs  */
+	init1.features[0] = 0;
+	init2.features[0] = (1 << KVM_ARM_VCPU_EL1_32BIT);
+	ret = add_init_2vcpus(&init1, &init2);
+	TEST_ASSERT(ret != 0,
+		    "Configuring mixed-width vCPUs worked unexpectedly");
+	ret = add_2vcpus_init_2vcpus(&init1, &init2);
+	TEST_ASSERT(ret != 0,
+		    "Configuring mixed-width vCPUs worked unexpectedly");
+
+	return 0;
+}
-- 
2.34.1.703.g22d0c6ccf7-goog

Re: [PATCH v2 1/2] KVM: arm64: mixed-width check should be skipped for uninitialized vCPUs

[Marc Zyngier]

On Tue, 18 Jan 2022 04:19:22 +0000,
Reiji Watanabe <reijiw@google.com> wrote:
> 
> KVM allows userspace to configure either all 32bit or 64bit vCPUs
> for a guest.  At vCPU reset, vcpu_allowed_register_width() checks
> if the vcpu's register width is consistent with all other vCPUs'.
> Since the checking is done even against vCPUs that are not initialized
> (KVM_ARM_VCPU_INIT has not been done) yet, the uninitialized vCPUs
> are erroneously treated as 64bit vCPU, which causes the function to
> incorrectly detect a mixed-width VM.
> 
> Introduce a new flag (el1_reg_width) in kvm_arch to indicates that
> the guest needs to be configured with all 32bit or 64bit vCPUs,
> and initialize it at the first KVM_ARM_VCPU_INIT for the guest.
> Check vcpu's register width against the flag at the vcpu's
> KVM_ARM_VCPU_INIT (instead of against other vCPUs' register width).
> 
> Fixes: 66e94d5cafd4 ("KVM: arm64: Prevent mixed-width VM creation")
> Signed-off-by: Reiji Watanabe <reijiw@google.com>
> ---
>  arch/arm64/include/asm/kvm_host.h | 13 +++++++++++++
>  arch/arm64/kvm/arm.c              | 30 ++++++++++++++++++++++++++++++
>  arch/arm64/kvm/reset.c            |  8 --------
>  3 files changed, 43 insertions(+), 8 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> index 2a5f7f38006f..c02b7caf2c82 100644
> --- a/arch/arm64/include/asm/kvm_host.h
> +++ b/arch/arm64/include/asm/kvm_host.h
> @@ -102,6 +102,12 @@ struct kvm_s2_mmu {
>  struct kvm_arch_memory_slot {
>  };
>  
> +enum kvm_el1_reg_width {
> +	EL1_WIDTH_UNINITIALIZED = 0,
> +	EL1_32BIT,
> +	EL1_64BIT,
> +};
> +
>  struct kvm_arch {
>  	struct kvm_s2_mmu mmu;
>  
> @@ -137,6 +143,13 @@ struct kvm_arch {
>  
>  	/* Memory Tagging Extension enabled for the guest */
>  	bool mte_enabled;
> +
> +	/*
> +	 * EL1 register width for the guest.
> +	 * This is set at the first KVM_ARM_VCPU_INIT for the guest based
> +	 * on whether the vcpu has KVM_ARM_VCPU_EL1_32BIT or not.
> +	 */
> +	enum kvm_el1_reg_width el1_reg_width;

I really don't like that we need to keep track of yet another bit of
state on top of the existing one. Duplicating state is a source of
bugs, because you always end up checking the wrong one at the wrong
time (and I have scars to prove it).

>  };
>  
>  struct kvm_vcpu_fault_info {
> diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> index e4727dc771bf..54ae8bf9d187 100644
> --- a/arch/arm64/kvm/arm.c
> +++ b/arch/arm64/kvm/arm.c
> @@ -1058,6 +1058,32 @@ int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_level,
>  	return -EINVAL;
>  }
>  
> +/*
> + * A guest can have either all 32bit or 64bit vcpus only.

That's not strictly true. All we are enforcing is that EL1 is either
32 or 64bit.

> + * Either one the guest has is indicated in kvm->arch.el1_reg_width.
> + * Check if the vcpu's register width is consistent with
> + * kvm->arch.el1_reg_width.  If kvm->arch.el1_reg_width is not set yet,
> + * set it based on the vcpu's KVM_ARM_VCPU_EL1_32BIT configuration.
> + */
> +static int kvm_register_width_check_or_init(struct kvm_vcpu *vcpu)
> +{
> +	bool is32bit;
> +	bool allowed = true;
> +	struct kvm *kvm = vcpu->kvm;
> +
> +	is32bit = vcpu_has_feature(vcpu, KVM_ARM_VCPU_EL1_32BIT);
> +
> +	mutex_lock(&kvm->lock);
> +
> +	if (kvm->arch.el1_reg_width == EL1_WIDTH_UNINITIALIZED)
> +		kvm->arch.el1_reg_width = is32bit ? EL1_32BIT : EL1_64BIT;
> +	else
> +		allowed = (is32bit == (kvm->arch.el1_reg_width == EL1_32BIT));
> +
> +	mutex_unlock(&kvm->lock);
> +	return allowed ? 0 : -EINVAL;
> +}
> +
>  static int kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
>  			       const struct kvm_vcpu_init *init)
>  {
> @@ -1097,6 +1123,10 @@ static int kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
>  
>  	/* Now we know what it is, we can reset it. */
>  	ret = kvm_reset_vcpu(vcpu);
> +
> +	if (!ret)
> +		ret = kvm_register_width_check_or_init(vcpu);
> +
>  	if (ret) {
>  		vcpu->arch.target = -1;
>  		bitmap_zero(vcpu->arch.features, KVM_VCPU_MAX_FEATURES);
> diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c
> index 426bd7fbc3fd..dbf2939a6a96 100644
> --- a/arch/arm64/kvm/reset.c
> +++ b/arch/arm64/kvm/reset.c
> @@ -168,9 +168,7 @@ static int kvm_vcpu_enable_ptrauth(struct kvm_vcpu *vcpu)
>  
>  static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu)
>  {
> -	struct kvm_vcpu *tmp;
>  	bool is32bit;
> -	int i;
>  
>  	is32bit = vcpu_has_feature(vcpu, KVM_ARM_VCPU_EL1_32BIT);
>  	if (!cpus_have_const_cap(ARM64_HAS_32BIT_EL1) && is32bit)
> @@ -180,12 +178,6 @@ static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu)
>  	if (kvm_has_mte(vcpu->kvm) && is32bit)
>  		return false;
>  
> -	/* Check that the vcpus are either all 32bit or all 64bit */
> -	kvm_for_each_vcpu(i, tmp, vcpu->kvm) {
> -		if (vcpu_has_feature(tmp, KVM_ARM_VCPU_EL1_32BIT) != is32bit)
> -			return false;
> -	}
> -

In [1], I suggested another approach that didn't require extra state,
and moved the existing checks under the kvm lock. What was wrong with
that approach?

Thanks,

	M.

[1] https://lore.kernel.org/r/875yqqtn5q.wl-maz@kernel.org

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH v2 1/2] KVM: arm64: mixed-width check should be skipped for uninitialized vCPUs

[Reiji Watanabe]

Hi Marc,

On Tue, Feb 8, 2022 at 6:41 AM Marc Zyngier <maz@kernel.org> wrote:
>
> On Tue, 18 Jan 2022 04:19:22 +0000,
> Reiji Watanabe <reijiw@google.com> wrote:
> >
> > KVM allows userspace to configure either all 32bit or 64bit vCPUs
> > for a guest.  At vCPU reset, vcpu_allowed_register_width() checks
> > if the vcpu's register width is consistent with all other vCPUs'.
> > Since the checking is done even against vCPUs that are not initialized
> > (KVM_ARM_VCPU_INIT has not been done) yet, the uninitialized vCPUs
> > are erroneously treated as 64bit vCPU, which causes the function to
> > incorrectly detect a mixed-width VM.
> >
> > Introduce a new flag (el1_reg_width) in kvm_arch to indicates that
> > the guest needs to be configured with all 32bit or 64bit vCPUs,
> > and initialize it at the first KVM_ARM_VCPU_INIT for the guest.
> > Check vcpu's register width against the flag at the vcpu's
> > KVM_ARM_VCPU_INIT (instead of against other vCPUs' register width).
> >
> > Fixes: 66e94d5cafd4 ("KVM: arm64: Prevent mixed-width VM creation")
> > Signed-off-by: Reiji Watanabe <reijiw@google.com>
> > ---
> >  arch/arm64/include/asm/kvm_host.h | 13 +++++++++++++
> >  arch/arm64/kvm/arm.c              | 30 ++++++++++++++++++++++++++++++
> >  arch/arm64/kvm/reset.c            |  8 --------
> >  3 files changed, 43 insertions(+), 8 deletions(-)
> >
> > diff --git a/arch/arm64/include/asm/kvm_host.h b/arch/arm64/include/asm/kvm_host.h
> > index 2a5f7f38006f..c02b7caf2c82 100644
> > --- a/arch/arm64/include/asm/kvm_host.h
> > +++ b/arch/arm64/include/asm/kvm_host.h
> > @@ -102,6 +102,12 @@ struct kvm_s2_mmu {
> >  struct kvm_arch_memory_slot {
> >  };
> >
> > +enum kvm_el1_reg_width {
> > +     EL1_WIDTH_UNINITIALIZED = 0,
> > +     EL1_32BIT,
> > +     EL1_64BIT,
> > +};
> > +
> >  struct kvm_arch {
> >       struct kvm_s2_mmu mmu;
> >
> > @@ -137,6 +143,13 @@ struct kvm_arch {
> >
> >       /* Memory Tagging Extension enabled for the guest */
> >       bool mte_enabled;
> > +
> > +     /*
> > +      * EL1 register width for the guest.
> > +      * This is set at the first KVM_ARM_VCPU_INIT for the guest based
> > +      * on whether the vcpu has KVM_ARM_VCPU_EL1_32BIT or not.
> > +      */
> > +     enum kvm_el1_reg_width el1_reg_width;
>
> I really don't like that we need to keep track of yet another bit of
> state on top of the existing one. Duplicating state is a source of
> bugs, because you always end up checking the wrong one at the wrong
> time (and I have scars to prove it).
>
> >  };
> >
> >  struct kvm_vcpu_fault_info {
> > diff --git a/arch/arm64/kvm/arm.c b/arch/arm64/kvm/arm.c
> > index e4727dc771bf..54ae8bf9d187 100644
> > --- a/arch/arm64/kvm/arm.c
> > +++ b/arch/arm64/kvm/arm.c
> > @@ -1058,6 +1058,32 @@ int kvm_vm_ioctl_irq_line(struct kvm *kvm, struct kvm_irq_level *irq_level,
> >       return -EINVAL;
> >  }
> >
> > +/*
> > + * A guest can have either all 32bit or 64bit vcpus only.
>
> That's not strictly true. All we are enforcing is that EL1 is either
> 32 or 64bit.

I will fix the comment.


>
> > + * Either one the guest has is indicated in kvm->arch.el1_reg_width.
> > + * Check if the vcpu's register width is consistent with
> > + * kvm->arch.el1_reg_width.  If kvm->arch.el1_reg_width is not set yet,
> > + * set it based on the vcpu's KVM_ARM_VCPU_EL1_32BIT configuration.
> > + */
> > +static int kvm_register_width_check_or_init(struct kvm_vcpu *vcpu)
> > +{
> > +     bool is32bit;
> > +     bool allowed = true;
> > +     struct kvm *kvm = vcpu->kvm;
> > +
> > +     is32bit = vcpu_has_feature(vcpu, KVM_ARM_VCPU_EL1_32BIT);
> > +
> > +     mutex_lock(&kvm->lock);
> > +
> > +     if (kvm->arch.el1_reg_width == EL1_WIDTH_UNINITIALIZED)
> > +             kvm->arch.el1_reg_width = is32bit ? EL1_32BIT : EL1_64BIT;
> > +     else
> > +             allowed = (is32bit == (kvm->arch.el1_reg_width == EL1_32BIT));
> > +
> > +     mutex_unlock(&kvm->lock);
> > +     return allowed ? 0 : -EINVAL;
> > +}
> > +
> >  static int kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
> >                              const struct kvm_vcpu_init *init)
> >  {
> > @@ -1097,6 +1123,10 @@ static int kvm_vcpu_set_target(struct kvm_vcpu *vcpu,
> >
> >       /* Now we know what it is, we can reset it. */
> >       ret = kvm_reset_vcpu(vcpu);
> > +
> > +     if (!ret)
> > +             ret = kvm_register_width_check_or_init(vcpu);
> > +
> >       if (ret) {
> >               vcpu->arch.target = -1;
> >               bitmap_zero(vcpu->arch.features, KVM_VCPU_MAX_FEATURES);
> > diff --git a/arch/arm64/kvm/reset.c b/arch/arm64/kvm/reset.c
> > index 426bd7fbc3fd..dbf2939a6a96 100644
> > --- a/arch/arm64/kvm/reset.c
> > +++ b/arch/arm64/kvm/reset.c
> > @@ -168,9 +168,7 @@ static int kvm_vcpu_enable_ptrauth(struct kvm_vcpu *vcpu)
> >
> >  static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu)
> >  {
> > -     struct kvm_vcpu *tmp;
> >       bool is32bit;
> > -     int i;
> >
> >       is32bit = vcpu_has_feature(vcpu, KVM_ARM_VCPU_EL1_32BIT);
> >       if (!cpus_have_const_cap(ARM64_HAS_32BIT_EL1) && is32bit)
> > @@ -180,12 +178,6 @@ static bool vcpu_allowed_register_width(struct kvm_vcpu *vcpu)
> >       if (kvm_has_mte(vcpu->kvm) && is32bit)
> >               return false;
> >
> > -     /* Check that the vcpus are either all 32bit or all 64bit */
> > -     kvm_for_each_vcpu(i, tmp, vcpu->kvm) {
> > -             if (vcpu_has_feature(tmp, KVM_ARM_VCPU_EL1_32BIT) != is32bit)
> > -                     return false;
> > -     }
> > -
>
> In [1], I suggested another approach that didn't require extra state,
> and moved the existing checks under the kvm lock. What was wrong with
> that approach?

With that approach, even for a vcpu that has a broken set of features,
which leads kvm_reset_vcpu() to fail for the vcpu, the vcpu->arch.features
are checked by other vCPUs' vcpu_allowed_register_width() until the
vcpu->arch.target is set to -1.
Due to this, I would think some or possibly all vCPUs' kvm_reset_vcpu()
may or may not fail (e.g. if userspace tries to configure vCPU#0 with
32bit EL1, and vCPU#1 and #2 with 64 bit EL1, KVM_ARM_VCPU_INIT
for either vCPU#0, or both vCPU#1 and #2 should fail.  But, with that
approach, it doesn't always work that way.  Instead, KVM_ARM_VCPU_INIT
for all vCPUs could fail or KVM_ARM_VCPU_INIT for vCPU#0 and #1 could
fail while the one for CPU#2 works).
Also, even after the first KVM_RUN for vCPUs are already done,
(the first) KVM_ARM_VCPU_INIT for another vCPU could cause the
kvm_reset_vcpu() for those vCPUs to fail.

I would think those behaviors are odd, and I wanted to avoid them.

Thanks,
Reiji




>
> Thanks,
>
>         M.
>
> [1] https://lore.kernel.org/r/875yqqtn5q.wl-maz@kernel.org
>
> --
> Without deviation from the norm, progress is not possible.

Re: [PATCH v2 1/2] KVM: arm64: mixed-width check should be skipped for uninitialized vCPUs

[Marc Zyngier]

Hi Reiji,

On Wed, 09 Feb 2022 05:32:36 +0000,
Reiji Watanabe <reijiw@google.com> wrote:
> 
> Hi Marc,
> 
> On Tue, Feb 8, 2022 at 6:41 AM Marc Zyngier <maz@kernel.org> wrote:
> >
> > In [1], I suggested another approach that didn't require extra state,
> > and moved the existing checks under the kvm lock. What was wrong with
> > that approach?
> 
> With that approach, even for a vcpu that has a broken set of features,
> which leads kvm_reset_vcpu() to fail for the vcpu, the vcpu->arch.features
> are checked by other vCPUs' vcpu_allowed_register_width() until the
> vcpu->arch.target is set to -1.
> Due to this, I would think some or possibly all vCPUs' kvm_reset_vcpu()
> may or may not fail (e.g. if userspace tries to configure vCPU#0 with
> 32bit EL1, and vCPU#1 and #2 with 64 bit EL1, KVM_ARM_VCPU_INIT
> for either vCPU#0, or both vCPU#1 and #2 should fail.  But, with that
> approach, it doesn't always work that way.  Instead, KVM_ARM_VCPU_INIT
> for all vCPUs could fail or KVM_ARM_VCPU_INIT for vCPU#0 and #1 could
> fail while the one for CPU#2 works).
> Also, even after the first KVM_RUN for vCPUs are already done,
> (the first) KVM_ARM_VCPU_INIT for another vCPU could cause the
> kvm_reset_vcpu() for those vCPUs to fail.
> 
> I would think those behaviors are odd, and I wanted to avoid them.

OK, fair enough. But then you need to remove most of the uses of
KVM_ARM_VCPU_EL1_32BIT so that it is only used as a userspace
interface and maybe not carried as part of the vcpu feature flag
anymore.

Also, we really should turn all these various bits in the kvm struct
into a set of flags. I have a patch posted there[1] for this, feel
free to pick it up.

Thanks,

	M.

[1] https://lore.kernel.org/r/20211004174849.2831548-2-maz@kernel.org

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH v2 1/2] KVM: arm64: mixed-width check should be skipped for uninitialized vCPUs

[Reiji Watanabe]

Hi Marc,

On Wed, Feb 9, 2022 at 4:04 AM Marc Zyngier <maz@kernel.org> wrote:
>
> Hi Reiji,
>
> On Wed, 09 Feb 2022 05:32:36 +0000,
> Reiji Watanabe <reijiw@google.com> wrote:
> >
> > Hi Marc,
> >
> > On Tue, Feb 8, 2022 at 6:41 AM Marc Zyngier <maz@kernel.org> wrote:
> > >
> > > In [1], I suggested another approach that didn't require extra state,
> > > and moved the existing checks under the kvm lock. What was wrong with
> > > that approach?
> >
> > With that approach, even for a vcpu that has a broken set of features,
> > which leads kvm_reset_vcpu() to fail for the vcpu, the vcpu->arch.features
> > are checked by other vCPUs' vcpu_allowed_register_width() until the
> > vcpu->arch.target is set to -1.
> > Due to this, I would think some or possibly all vCPUs' kvm_reset_vcpu()
> > may or may not fail (e.g. if userspace tries to configure vCPU#0 with
> > 32bit EL1, and vCPU#1 and #2 with 64 bit EL1, KVM_ARM_VCPU_INIT
> > for either vCPU#0, or both vCPU#1 and #2 should fail.  But, with that
> > approach, it doesn't always work that way.  Instead, KVM_ARM_VCPU_INIT
> > for all vCPUs could fail or KVM_ARM_VCPU_INIT for vCPU#0 and #1 could
> > fail while the one for CPU#2 works).
> > Also, even after the first KVM_RUN for vCPUs are already done,
> > (the first) KVM_ARM_VCPU_INIT for another vCPU could cause the
> > kvm_reset_vcpu() for those vCPUs to fail.
> >
> > I would think those behaviors are odd, and I wanted to avoid them.
>
> OK, fair enough. But then you need to remove most of the uses of
> KVM_ARM_VCPU_EL1_32BIT so that it is only used as a userspace
> interface and

Yes, I will.

> maybe not carried as part of the vcpu feature flag anymore.

At the first call of kvm_reset_vcpu() for the guest, the new kvm
flag is not set yet. So, KVM_ARM_VCPU_EL1_32BIT will be needed
by the function (unless we pass the flag as an argument for the
function or by any other way).

> Also, we really should turn all these various bits in the kvm struct
> into a set of flags. I have a patch posted there[1] for this, feel
> free to pick it up.

Thank you for the suggestion. But, kvm->arch.el1_reg_width is not
a binary because it needs to indicate an uninitialized state.  So, it
won't fit perfectly with kvm->arch.flags, which is introduced by [1]
as it is. Of course it's feasible by using 2 bits of the flags though...

Thanks,
Reiji

>
> Thanks,
>
>         M.
>
> [1] https://lore.kernel.org/r/20211004174849.2831548-2-maz@kernel.org
>
> --
> Without deviation from the norm, progress is not possible.

Re: [PATCH v2 1/2] KVM: arm64: mixed-width check should be skipped for uninitialized vCPUs

[Marc Zyngier]

On Thu, 10 Feb 2022 05:31:49 +0000,
Reiji Watanabe <reijiw@google.com> wrote:
> 
> Hi Marc,
> 
> On Wed, Feb 9, 2022 at 4:04 AM Marc Zyngier <maz@kernel.org> wrote:
> >
> > Hi Reiji,
> >
> > On Wed, 09 Feb 2022 05:32:36 +0000,
> > Reiji Watanabe <reijiw@google.com> wrote:
> > >
> > > Hi Marc,
> > >
> > > On Tue, Feb 8, 2022 at 6:41 AM Marc Zyngier <maz@kernel.org> wrote:
> > > >
> > > > In [1], I suggested another approach that didn't require extra state,
> > > > and moved the existing checks under the kvm lock. What was wrong with
> > > > that approach?
> > >
> > > With that approach, even for a vcpu that has a broken set of features,
> > > which leads kvm_reset_vcpu() to fail for the vcpu, the vcpu->arch.features
> > > are checked by other vCPUs' vcpu_allowed_register_width() until the
> > > vcpu->arch.target is set to -1.
> > > Due to this, I would think some or possibly all vCPUs' kvm_reset_vcpu()
> > > may or may not fail (e.g. if userspace tries to configure vCPU#0 with
> > > 32bit EL1, and vCPU#1 and #2 with 64 bit EL1, KVM_ARM_VCPU_INIT
> > > for either vCPU#0, or both vCPU#1 and #2 should fail.  But, with that
> > > approach, it doesn't always work that way.  Instead, KVM_ARM_VCPU_INIT
> > > for all vCPUs could fail or KVM_ARM_VCPU_INIT for vCPU#0 and #1 could
> > > fail while the one for CPU#2 works).
> > > Also, even after the first KVM_RUN for vCPUs are already done,
> > > (the first) KVM_ARM_VCPU_INIT for another vCPU could cause the
> > > kvm_reset_vcpu() for those vCPUs to fail.
> > >
> > > I would think those behaviors are odd, and I wanted to avoid them.
> >
> > OK, fair enough. But then you need to remove most of the uses of
> > KVM_ARM_VCPU_EL1_32BIT so that it is only used as a userspace
> > interface and
> 
> Yes, I will.
> 
> > maybe not carried as part of the vcpu feature flag anymore.
> 
> At the first call of kvm_reset_vcpu() for the guest, the new kvm
> flag is not set yet. So, KVM_ARM_VCPU_EL1_32BIT will be needed
> by the function (unless we pass the flag as an argument for the
> function or by any other way).

Which is why I said 'maybe'. It's not a big deal if the flags stays,
but I don't want it evaluated further down the line. It is also pretty
similar to HCR_EL2.RW, which we already test with vcpu_el1_is_32bit().

Overall, we need to reduce that state to be as simple as possible.

> 
> > Also, we really should turn all these various bits in the kvm struct
> > into a set of flags. I have a patch posted there[1] for this, feel
> > free to pick it up.
> 
> Thank you for the suggestion. But, kvm->arch.el1_reg_width is not
> a binary because it needs to indicate an uninitialized state.  So, it
> won't fit perfectly with kvm->arch.flags, which is introduced by [1]
> as it is. Of course it's feasible by using 2 bits of the flags though...

2 bits is what I had in mind (one bit to indicate that it has already
been initialised, another to carry the actual width).

Thanks,

	M.

-- 
Without deviation from the norm, progress is not possible.

Re: [PATCH v2 1/2] KVM: arm64: mixed-width check should be skipped for uninitialized vCPUs

[Reiji Watanabe]

Hi Marc,

> > > Also, we really should turn all these various bits in the kvm struct
> > > into a set of flags. I have a patch posted there[1] for this, feel
> > > free to pick it up.
> >
> > Thank you for the suggestion. But, kvm->arch.el1_reg_width is not
> > a binary because it needs to indicate an uninitialized state.  So, it
> > won't fit perfectly with kvm->arch.flags, which is introduced by [1]
> > as it is. Of course it's feasible by using 2 bits of the flags though...
>
> 2 bits is what I had in mind (one bit to indicate that it has already
> been initialised, another to carry the actual width).

Understood. Then, I will take the patch and will work on v3.
Thank you for all the comments!

Regards,
Reiji