[PATCH v9 8/9] KVM: x86: Allow userspace set maximum VCPU id for VM

[PATCH v9 8/9] KVM: x86: Allow userspace set maximum VCPU id for VM

[Zeng Guang]

Introduce new max_vcpu_ids in KVM for x86 architecture. Userspace
can assign maximum possible vcpu id for current VM session using
KVM_CAP_MAX_VCPU_ID of KVM_ENABLE_CAP ioctl().

This is done for x86 only because the sole use case is to guide
memory allocation for PID-pointer table, a structure needed to
enable VMX IPI.

By default, max_vcpu_ids set as KVM_MAX_VCPU_IDS.

Suggested-by: Sean Christopherson <seanjc@google.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Signed-off-by: Zeng Guang <guang.zeng@intel.com>
---
 Documentation/virt/kvm/api.rst  | 18 ++++++++++++++++++
 arch/x86/include/asm/kvm_host.h |  6 ++++++
 arch/x86/kvm/x86.c              | 25 ++++++++++++++++++++++++-
 3 files changed, 48 insertions(+), 1 deletion(-)

diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
index d13fa6600467..0c6ad2d8bea0 100644
--- a/Documentation/virt/kvm/api.rst
+++ b/Documentation/virt/kvm/api.rst
@@ -7136,6 +7136,24 @@ The valid bits in cap.args[0] are:
                                     IA32_MISC_ENABLE[bit 18] is cleared.
 =================================== ============================================
 
+7.32 KVM_CAP_MAX_VCPU_ID
+------------------------
+
+:Architectures: x86
+:Target: VM
+:Parameters: args[0] - maximum APIC ID value set for current VM
+:Returns: 0 on success, -EINVAL if args[0] is beyond KVM_MAX_VCPU_IDS
+          supported in KVM or if it has been settled.
+
+Userspace is able to calculate the limit to APIC ID values from designated CPU
+topology. This capability allows userspace to specify maximum possible APIC ID
+assigned for current VM session prior to the creation of vCPUs. By design, it
+can set only once and doesn't accept change any more. KVM will manage memory
+allocation of VM-scope structures which depends on the value of APIC ID.
+
+Calling KVM_CHECK_EXTENSION for this capability returns the value of maximum APIC
+ID that KVM supports at runtime. It sets as KVM_MAX_VCPU_IDS by default.
+
 8. Other capabilities.
 ======================
 
diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index d23e80a56eb8..cdd14033988d 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -1238,6 +1238,12 @@ struct kvm_arch {
 	hpa_t	hv_root_tdp;
 	spinlock_t hv_root_tdp_lock;
 #endif
+	/*
+	 * VM-scope maximum vCPU ID. Used to determine the size of structures
+	 * that increase along with the maximum vCPU ID, in which case, using
+	 * the global KVM_MAX_VCPU_IDS may lead to significant memory waste.
+	 */
+	u32 max_vcpu_ids;
 };
 
 struct kvm_vm_stat {
diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
index 277a0da8c290..744e88a71b63 100644
--- a/arch/x86/kvm/x86.c
+++ b/arch/x86/kvm/x86.c
@@ -4320,7 +4320,10 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
 		r = KVM_MAX_VCPUS;
 		break;
 	case KVM_CAP_MAX_VCPU_ID:
-		r = KVM_MAX_VCPU_IDS;
+		if (!kvm->arch.max_vcpu_ids)
+			r = KVM_MAX_VCPU_IDS;
+		else
+			r = kvm->arch.max_vcpu_ids;
 		break;
 	case KVM_CAP_PV_MMU:	/* obsolete */
 		r = 0;
@@ -6064,6 +6067,20 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
 		}
 		mutex_unlock(&kvm->lock);
 		break;
+	case KVM_CAP_MAX_VCPU_ID:
+		r = -EINVAL;
+		if (cap->args[0] > KVM_MAX_VCPU_IDS)
+			break;
+
+		mutex_lock(&kvm->lock);
+		if (kvm->arch.max_vcpu_ids == cap->args[0]) {
+			r = 0;
+		} else if (!kvm->arch.max_vcpu_ids) {
+			kvm->arch.max_vcpu_ids = cap->args[0];
+			r = 0;
+		}
+		mutex_unlock(&kvm->lock);
+		break;
 	default:
 		r = -EINVAL;
 		break;
@@ -11172,6 +11189,12 @@ int kvm_arch_vcpu_precreate(struct kvm *kvm, unsigned int id)
 		pr_warn_once("kvm: SMP vm created on host with unstable TSC; "
 			     "guest TSC will not be reliable\n");
 
+	if (!kvm->arch.max_vcpu_ids)
+		kvm->arch.max_vcpu_ids = KVM_MAX_VCPU_IDS;
+
+	if (id >= kvm->arch.max_vcpu_ids)
+		return -EINVAL;
+
 	return 0;
 }
 
-- 
2.27.0

Re: [PATCH v9 8/9] KVM: x86: Allow userspace set maximum VCPU id for VM

[Chao Gao]

On Tue, Apr 19, 2022 at 11:44:44PM +0800, Zeng Guang wrote:
>Introduce new max_vcpu_ids in KVM for x86 architecture. Userspace
>can assign maximum possible vcpu id for current VM session using
>KVM_CAP_MAX_VCPU_ID of KVM_ENABLE_CAP ioctl().
>
>This is done for x86 only because the sole use case is to guide
>memory allocation for PID-pointer table, a structure needed to
>enable VMX IPI.
>
>By default, max_vcpu_ids set as KVM_MAX_VCPU_IDS.
>
>Suggested-by: Sean Christopherson <seanjc@google.com>
>Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
>Signed-off-by: Zeng Guang <guang.zeng@intel.com>
>---
> Documentation/virt/kvm/api.rst  | 18 ++++++++++++++++++
> arch/x86/include/asm/kvm_host.h |  6 ++++++
> arch/x86/kvm/x86.c              | 25 ++++++++++++++++++++++++-
> 3 files changed, 48 insertions(+), 1 deletion(-)
>
>diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst
>index d13fa6600467..0c6ad2d8bea0 100644
>--- a/Documentation/virt/kvm/api.rst
>+++ b/Documentation/virt/kvm/api.rst
>@@ -7136,6 +7136,24 @@ The valid bits in cap.args[0] are:
>                                     IA32_MISC_ENABLE[bit 18] is cleared.
> =================================== ============================================
> 
>+7.32 KVM_CAP_MAX_VCPU_ID
>+------------------------
>+
>+:Architectures: x86
>+:Target: VM
>+:Parameters: args[0] - maximum APIC ID value set for current VM
>+:Returns: 0 on success, -EINVAL if args[0] is beyond KVM_MAX_VCPU_IDS
>+          supported in KVM or if it has been settled.
>+
>+Userspace is able to calculate the limit to APIC ID values from designated CPU
>+topology. This capability allows userspace to specify maximum possible APIC ID
>+assigned for current VM session prior to the creation of vCPUs. By design, it
>+can set only once and doesn't accept change any more. KVM will manage memory
>+allocation of VM-scope structures which depends on the value of APIC ID.
>+
>+Calling KVM_CHECK_EXTENSION for this capability returns the value of maximum APIC
>+ID that KVM supports at runtime. It sets as KVM_MAX_VCPU_IDS by default.
>+
> 8. Other capabilities.
> ======================
> 
>diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
>index d23e80a56eb8..cdd14033988d 100644
>--- a/arch/x86/include/asm/kvm_host.h
>+++ b/arch/x86/include/asm/kvm_host.h
>@@ -1238,6 +1238,12 @@ struct kvm_arch {
> 	hpa_t	hv_root_tdp;
> 	spinlock_t hv_root_tdp_lock;
> #endif
>+	/*
>+	 * VM-scope maximum vCPU ID. Used to determine the size of structures
>+	 * that increase along with the maximum vCPU ID, in which case, using
>+	 * the global KVM_MAX_VCPU_IDS may lead to significant memory waste.
>+	 */
>+	u32 max_vcpu_ids;
> };
> 
> struct kvm_vm_stat {
>diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c
>index 277a0da8c290..744e88a71b63 100644
>--- a/arch/x86/kvm/x86.c
>+++ b/arch/x86/kvm/x86.c
>@@ -4320,7 +4320,10 @@ int kvm_vm_ioctl_check_extension(struct kvm *kvm, long ext)
> 		r = KVM_MAX_VCPUS;
> 		break;
> 	case KVM_CAP_MAX_VCPU_ID:
>-		r = KVM_MAX_VCPU_IDS;
>+		if (!kvm->arch.max_vcpu_ids)
>+			r = KVM_MAX_VCPU_IDS;
>+		else
>+			r = kvm->arch.max_vcpu_ids;
> 		break;
> 	case KVM_CAP_PV_MMU:	/* obsolete */
> 		r = 0;
>@@ -6064,6 +6067,20 @@ int kvm_vm_ioctl_enable_cap(struct kvm *kvm,
> 		}
> 		mutex_unlock(&kvm->lock);
> 		break;
>+	case KVM_CAP_MAX_VCPU_ID:
>+		r = -EINVAL;
>+		if (cap->args[0] > KVM_MAX_VCPU_IDS)
>+			break;
>+
>+		mutex_lock(&kvm->lock);
>+		if (kvm->arch.max_vcpu_ids == cap->args[0]) {
>+			r = 0;
>+		} else if (!kvm->arch.max_vcpu_ids) {
>+			kvm->arch.max_vcpu_ids = cap->args[0];
>+			r = 0;
>+		}
>+		mutex_unlock(&kvm->lock);
>+		break;

It would be better to have a kselftest to exercise this capability.
For example,
1. launch a VM.
2. set the max vCPU ID via KVM_CAP_MAX_VCPU_ID
3. read the max vCPU ID to check if the value written is returned.
4. create a vCPU which has apic id larger than the maximum.
5. try to change the max vCPU ID after set once.
...

This test can be the last patch of this series or posted separately.

Re: [PATCH v9 8/9] KVM: x86: Allow userspace set maximum VCPU id for VM

[Paolo Bonzini]

On 4/19/22 17:44, Zeng Guang wrote:
> +Userspace is able to calculate the limit to APIC ID values from designated CPU
> +topology. This capability allows userspace to specify maximum possible APIC ID
> +assigned for current VM session prior to the creation of vCPUs. By design, it
> +can set only once and doesn't accept change any more. KVM will manage memory
> +allocation of VM-scope structures which depends on the value of APIC ID.
> +
> +Calling KVM_CHECK_EXTENSION for this capability returns the value of maximum APIC
> +ID that KVM supports at runtime. It sets as KVM_MAX_VCPU_IDS by default.

Better:

This capability allows userspace to specify maximum possible APIC ID
assigned for current VM session prior to the creation of vCPUs, saving
memory for data structures indexed by the APIC ID.  Userspace is able
to calculate the limit to APIC ID values from designated
CPU topology.

The value can be changed only until KVM_ENABLE_CAP is set to a nonzero
value or until a vCPU is created.  Upon creation of the first vCPU,
if the value was set to zero or KVM_ENABLE_CAP was not invoked, KVM
uses the return value of KVM_CHECK_EXTENSION(KVM_CAP_MAX_VCPU_ID) as
the maximum APIC ID.

>   	case KVM_CAP_MAX_VCPU_ID:
> -		r = KVM_MAX_VCPU_IDS;
> +		if (!kvm->arch.max_vcpu_ids)
> +			r = KVM_MAX_VCPU_IDS;
> +		else
> +			r = kvm->arch.max_vcpu_ids;

I think returning the constant KVM_CAP_MAX_VCPU_IDS is better.

Paolo

Re: [PATCH v9 8/9] KVM: x86: Allow userspace set maximum VCPU id for VM

[Zeng Guang]

On 5/3/2022 12:07 AM, Paolo Bonzini wrote:
> On 4/19/22 17:44, Zeng Guang wrote:
>> +Userspace is able to calculate the limit to APIC ID values from designated CPU
>> +topology. This capability allows userspace to specify maximum possible APIC ID
>> +assigned for current VM session prior to the creation of vCPUs. By design, it
>> +can set only once and doesn't accept change any more. KVM will manage memory
>> +allocation of VM-scope structures which depends on the value of APIC ID.
>> +
>> +Calling KVM_CHECK_EXTENSION for this capability returns the value of maximum APIC
>> +ID that KVM supports at runtime. It sets as KVM_MAX_VCPU_IDS by default.
> Better:
>
> This capability allows userspace to specify maximum possible APIC ID
> assigned for current VM session prior to the creation of vCPUs, saving
> memory for data structures indexed by the APIC ID.  Userspace is able
> to calculate the limit to APIC ID values from designated
> CPU topology.
>
> The value can be changed only until KVM_ENABLE_CAP is set to a nonzero
> value or until a vCPU is created.  Upon creation of the first vCPU,
> if the value was set to zero or KVM_ENABLE_CAP was not invoked, KVM
> uses the return value of KVM_CHECK_EXTENSION(KVM_CAP_MAX_VCPU_ID) as
> the maximum APIC ID.
>
>>    	case KVM_CAP_MAX_VCPU_ID:
>> -		r = KVM_MAX_VCPU_IDS;
>> +		if (!kvm->arch.max_vcpu_ids)
>> +			r = KVM_MAX_VCPU_IDS;
>> +		else
>> +			r = kvm->arch.max_vcpu_ids;
> I think returning the constant KVM_CAP_MAX_VCPU_IDS is better.
>
> Paolo
Thanks. I will change as you suggested.