[PATCH] KVM: x86: Increase KVM_MAX_VCPUS to 2048

[PATCH] KVM: x86: Increase KVM_MAX_VCPUS to 2048

[Kyle Meyer]

Increase KVM_MAX_VCPUS to 2048 so we can run larger virtual machines.

Notable changes:

* KVM_CAP_MAX_VCPUS will return 2048.
* KVM_MAX_VCPU_IDS will increase from 4096 to 8192.
* KVM_HV_MAX_SPARSE_VCPU_SET_BITS will increase from 16 to 32.

* CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (00x40000005)].EAX will now be 2048.

* struct kvm will increase from 40336 B to 40464 B.
* struct kvm_arch will increase from 34488 B to 34616 B.
* struct kvm_ioapic will increase from 5240 B to 9848 B.

* vcpu_mask in kvm_hv_flush_tlb will increase from 128 B to 256 B.
* vcpu_mask in kvm_hv_send_ipi will increase from 128 B to 256 B.
* vcpu_bitmap in ioapic_write_indirect will increase from 128 B to 256 B.
* vp_bitmap in sparse_set_to_vcpu_mask will increase from 128 B to 256 B.
* sparse_banks in kvm_hv_flush_tlb will increase from 128 B to 256 B.
* sparse_banks in kvm_hv_send_ipi will increase from 128 B to 256 B.

Signed-off-by: Kyle Meyer <kyle.meyer@hpe.com>
---
 arch/x86/include/asm/kvm_host.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 3a240a64ac68..58653c63899f 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -38,7 +38,7 @@
 
 #define __KVM_HAVE_ARCH_VCPU_DEBUGFS
 
-#define KVM_MAX_VCPUS 1024
+#define KVM_MAX_VCPUS 2048
 
 /*
  * In x86, the VCPU ID corresponds to the APIC ID, and APIC IDs
-- 
2.26.2

Re: [PATCH] KVM: x86: Increase KVM_MAX_VCPUS to 2048

[Vitaly Kuznetsov]

Kyle Meyer <kyle.meyer@hpe.com> writes:

> Increase KVM_MAX_VCPUS to 2048 so we can run larger virtual machines.
>
> Notable changes:
>
> * KVM_CAP_MAX_VCPUS will return 2048.
> * KVM_MAX_VCPU_IDS will increase from 4096 to 8192.
> * KVM_HV_MAX_SPARSE_VCPU_SET_BITS will increase from 16 to 32.
>
> * CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (00x40000005)].EAX will now be 2048.
>
> * struct kvm will increase from 40336 B to 40464 B.
> * struct kvm_arch will increase from 34488 B to 34616 B.
> * struct kvm_ioapic will increase from 5240 B to 9848 B.
>
> * vcpu_mask in kvm_hv_flush_tlb will increase from 128 B to 256 B.
> * vcpu_mask in kvm_hv_send_ipi will increase from 128 B to 256 B.

FWIW, this one will go away when
https://lore.kernel.org/kvm/20220606083655.2014609-11-vkuznets@redhat.com/
lands.

> * vcpu_bitmap in ioapic_write_indirect will increase from 128 B to 256 B.
> * vp_bitmap in sparse_set_to_vcpu_mask will increase from 128 B to 256 B.

...

> * sparse_banks in kvm_hv_flush_tlb will increase from 128 B to 256 B.
> * sparse_banks in kvm_hv_send_ipi will increase from 128 B to 256 B.

and these two are going away with
https://lore.kernel.org/kvm/20220606083655.2014609-13-vkuznets@redhat.com/
(at the cost of growing 'struct kvm_vcpu_hv')

>
> Signed-off-by: Kyle Meyer <kyle.meyer@hpe.com>
> ---
>  arch/x86/include/asm/kvm_host.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
> index 3a240a64ac68..58653c63899f 100644
> --- a/arch/x86/include/asm/kvm_host.h
> +++ b/arch/x86/include/asm/kvm_host.h
> @@ -38,7 +38,7 @@
>  
>  #define __KVM_HAVE_ARCH_VCPU_DEBUGFS
>  
> -#define KVM_MAX_VCPUS 1024
> +#define KVM_MAX_VCPUS 2048
>  
>  /*
>   * In x86, the VCPU ID corresponds to the APIC ID, and APIC IDs

-- 
Vitaly

Re: [PATCH] KVM: x86: Increase KVM_MAX_VCPUS to 2048

[David Matlack]

On Mon, Jun 13, 2022 at 11:35 AM Kyle Meyer <kyle.meyer@hpe.com> wrote:
>
> Increase KVM_MAX_VCPUS to 2048 so we can run larger virtual machines.

Does the host machine have 2048 CPUs (or more) as well in your usecase?

I'm wondering if it makes sense to start configuring KVM_MAX_VCPUS
based on NR_CPUS. That way KVM can scale up on large machines without
using more memory on small machines.

e.g.

/* Provide backwards compatibility. */
#if NR_CPUS < 1024
  #define KVM_MAX_VCPUS 1024
#else
  #define KVM_MAX_VCPUS NR_CPUS
#endif

The only downside I can see for this approach is if you are trying to
kick the tires a new large VM on a smaller host because the new "large
host" hardware hasn't landed yet.

>
> Notable changes:
>
> * KVM_CAP_MAX_VCPUS will return 2048.
> * KVM_MAX_VCPU_IDS will increase from 4096 to 8192.
> * KVM_HV_MAX_SPARSE_VCPU_SET_BITS will increase from 16 to 32.
>
> * CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (00x40000005)].EAX will now be 2048.
>
> * struct kvm will increase from 40336 B to 40464 B.
> * struct kvm_arch will increase from 34488 B to 34616 B.
> * struct kvm_ioapic will increase from 5240 B to 9848 B.
>
> * vcpu_mask in kvm_hv_flush_tlb will increase from 128 B to 256 B.
> * vcpu_mask in kvm_hv_send_ipi will increase from 128 B to 256 B.
> * vcpu_bitmap in ioapic_write_indirect will increase from 128 B to 256 B.
> * vp_bitmap in sparse_set_to_vcpu_mask will increase from 128 B to 256 B.
> * sparse_banks in kvm_hv_flush_tlb will increase from 128 B to 256 B.
> * sparse_banks in kvm_hv_send_ipi will increase from 128 B to 256 B.
>
> Signed-off-by: Kyle Meyer <kyle.meyer@hpe.com>
> ---
>  arch/x86/include/asm/kvm_host.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
> index 3a240a64ac68..58653c63899f 100644
> --- a/arch/x86/include/asm/kvm_host.h
> +++ b/arch/x86/include/asm/kvm_host.h
> @@ -38,7 +38,7 @@
>
>  #define __KVM_HAVE_ARCH_VCPU_DEBUGFS
>
> -#define KVM_MAX_VCPUS 1024
> +#define KVM_MAX_VCPUS 2048
>
>  /*
>   * In x86, the VCPU ID corresponds to the APIC ID, and APIC IDs
> --
> 2.26.2
>

Re: [PATCH] KVM: x86: Increase KVM_MAX_VCPUS to 2048

[Vitaly Kuznetsov]

David Matlack <dmatlack@google.com> writes:

> On Mon, Jun 13, 2022 at 11:35 AM Kyle Meyer <kyle.meyer@hpe.com> wrote:
>>
>> Increase KVM_MAX_VCPUS to 2048 so we can run larger virtual machines.
>
> Does the host machine have 2048 CPUs (or more) as well in your usecase?
>
> I'm wondering if it makes sense to start configuring KVM_MAX_VCPUS
> based on NR_CPUS. That way KVM can scale up on large machines without
> using more memory on small machines.
>
> e.g.
>
> /* Provide backwards compatibility. */
> #if NR_CPUS < 1024
>   #define KVM_MAX_VCPUS 1024
> #else
>   #define KVM_MAX_VCPUS NR_CPUS
> #endif
>
> The only downside I can see for this approach is if you are trying to
> kick the tires a new large VM on a smaller host because the new "large
> host" hardware hasn't landed yet.

FWIW, while I don't think there's anything wrong with such approach, it
won't help much distro kernels which are not recompiled to meet the
needs of a particular host. According to Kyle's numbers, the biggest
growth is observed with 'struct kvm_ioapic' and that's only because of
'struct rtc_status' embedded in it. Maybe it's possible to use something
different from a KVM_MAX_VCPU_IDS-bound flat bitmask there? I'm not sure
how important this is as it's just another 4K per-VM and when guest's
memory is taken into account it's probably not much.

The growth in 'struct kvm'/'struct kvm_arch' seems to be insignificant
and on-stack allocations are probably OK.

-- 
Vitaly

Re: [PATCH] KVM: x86: Increase KVM_MAX_VCPUS to 2048

[David Matlack]

On Tue, Jun 14, 2022 at 1:28 AM Vitaly Kuznetsov <vkuznets@redhat.com> wrote:
>
> David Matlack <dmatlack@google.com> writes:
>
> > On Mon, Jun 13, 2022 at 11:35 AM Kyle Meyer <kyle.meyer@hpe.com> wrote:
> >>
> >> Increase KVM_MAX_VCPUS to 2048 so we can run larger virtual machines.
> >
> > Does the host machine have 2048 CPUs (or more) as well in your usecase?
> >
> > I'm wondering if it makes sense to start configuring KVM_MAX_VCPUS
> > based on NR_CPUS. That way KVM can scale up on large machines without
> > using more memory on small machines.
> >
> > e.g.
> >
> > /* Provide backwards compatibility. */
> > #if NR_CPUS < 1024
> >   #define KVM_MAX_VCPUS 1024
> > #else
> >   #define KVM_MAX_VCPUS NR_CPUS
> > #endif
> >
> > The only downside I can see for this approach is if you are trying to
> > kick the tires a new large VM on a smaller host because the new "large
> > host" hardware hasn't landed yet.

Heh. My point here doesn't make sense. The actual number of CPUs in
the host machine wouldn't matter, just the host kernel's NR_CPUS.

>
> FWIW, while I don't think there's anything wrong with such approach, it
> won't help much distro kernels which are not recompiled to meet the
> needs of a particular host.

But is there a use-case for running a VM with more vCPUs than the
kernel's NR_CPUS?


> According to Kyle's numbers, the biggest
> growth is observed with 'struct kvm_ioapic' and that's only because of
> 'struct rtc_status' embedded in it. Maybe it's possible to use something
> different from a KVM_MAX_VCPU_IDS-bound flat bitmask there? I'm not sure
> how important this is as it's just another 4K per-VM and when guest's
> memory is taken into account it's probably not much.
>
> The growth in 'struct kvm'/'struct kvm_arch' seems to be insignificant
> and on-stack allocations are probably OK.
>
> --
> Vitaly
>

Re: [PATCH] KVM: x86: Increase KVM_MAX_VCPUS to 2048

[Sean Christopherson]

On Thu, Jun 16, 2022, David Matlack wrote:
> On Tue, Jun 14, 2022 at 1:28 AM Vitaly Kuznetsov <vkuznets@redhat.com> wrote:
> >
> > David Matlack <dmatlack@google.com> writes:
> >
> > > On Mon, Jun 13, 2022 at 11:35 AM Kyle Meyer <kyle.meyer@hpe.com> wrote:
> > >>
> > >> Increase KVM_MAX_VCPUS to 2048 so we can run larger virtual machines.
> > >
> > > Does the host machine have 2048 CPUs (or more) as well in your usecase?
> > >
> > > I'm wondering if it makes sense to start configuring KVM_MAX_VCPUS
> > > based on NR_CPUS. That way KVM can scale up on large machines without
> > > using more memory on small machines.
> > >
> > > e.g.
> > >
> > > /* Provide backwards compatibility. */
> > > #if NR_CPUS < 1024
> > >   #define KVM_MAX_VCPUS 1024
> > > #else
> > >   #define KVM_MAX_VCPUS NR_CPUS
> > > #endif
> > >
> > > The only downside I can see for this approach is if you are trying to
> > > kick the tires a new large VM on a smaller host because the new "large
> > > host" hardware hasn't landed yet.
> 
> Heh. My point here doesn't make sense. The actual number of CPUs in
> the host machine wouldn't matter, just the host kernel's NR_CPUS.
> 
> >
> > FWIW, while I don't think there's anything wrong with such approach, it
> > won't help much distro kernels which are not recompiled to meet the
> > needs of a particular host.
> 
> But is there a use-case for running a VM with more vCPUs than the
> kernel's NR_CPUS?

With custom configs yes, e.g. my host kernel config for my anemic old system has
CONFIG_NR_CPUS=12 and I occasionally run VMs with >12 vCPUs for testing purposes.

But that obviously doesn't apply if KVM_MAX_VCPUS has a lower bound of 1024, so
I agree that making KVM_MAX_VCPUS=max(1024, NR_CPUS) should do just fine.

[PATCH v2] KVM: x86: Increase KVM_MAX_VCPUS to 4096

[Kyle Meyer]

Increase KVM_MAX_VCPUS to 4096 when the maximum number of supported CPUs
(NR_CPUS) is greater than 1024.

The Hyper-V TLFS doesn't allow more than 64 sparse banks, which allows a
maximum of 4096 virtual CPUs. Limit KVM's maximum number of virtual CPUs
to 4096 to avoid exceeding that limit.

Notable changes:

* KVM_MAX_VCPU_IDS will increase from 4096 to 16384.
* KVM_HV_MAX_SPARSE_VCPU_SET_BITS will increase from 16 to 64.

* CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (00x40000005)].EAX will now be 4096.

* struct kvm will increase from 40336 B to 40720 B.
* struct kvm_ioapic will increase from 5240 B to 19064 B.

* vcpu_mask in kvm_hv_flush_tlb will increase from 128 B to 512 B.
* vcpu_bitmap in ioapic_write_indirect will increase from 128 B to 512 B.
* vp_bitmap in sparse_set_to_vcpu_mask will increase from 128 B to 512 B.

Signed-off-by: Kyle Meyer <kyle.meyer@hpe.com>
---
 arch/x86/include/asm/kvm_host.h | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
index 9217bd6cf0d1..867a945f0152 100644
--- a/arch/x86/include/asm/kvm_host.h
+++ b/arch/x86/include/asm/kvm_host.h
@@ -38,7 +38,11 @@
 
 #define __KVM_HAVE_ARCH_VCPU_DEBUGFS
 
+#if NR_CPUS < 1024
 #define KVM_MAX_VCPUS 1024
+#else
+#define KVM_MAX_VCPUS 4096
+#endif
 
 /*
  * In x86, the VCPU ID corresponds to the APIC ID, and APIC IDs
-- 
2.26.2

Re: [PATCH v2] KVM: x86: Increase KVM_MAX_VCPUS to 4096

[Vitaly Kuznetsov]

Kyle Meyer <kyle.meyer@hpe.com> writes:

> Increase KVM_MAX_VCPUS to 4096 when the maximum number of supported CPUs
> (NR_CPUS) is greater than 1024.
>
> The Hyper-V TLFS doesn't allow more than 64 sparse banks, which allows a
> maximum of 4096 virtual CPUs. Limit KVM's maximum number of virtual CPUs
> to 4096 to avoid exceeding that limit.

In theory, it's just TLB flush and IPI hypercalls which have this
limitation. Strictly speaking, guest can have more than 4096 vCPUs,
it'll just have to do IPIs/TLB flush in a different way.

>
> Notable changes:
>
> * KVM_MAX_VCPU_IDS will increase from 4096 to 16384.
> * KVM_HV_MAX_SPARSE_VCPU_SET_BITS will increase from 16 to 64.
>
> * CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (00x40000005)].EAX will now be 4096.

	redundant leading zero		 ^^

>
> * struct kvm will increase from 40336 B to 40720 B.
> * struct kvm_ioapic will increase from 5240 B to 19064 B.
>
> * vcpu_mask in kvm_hv_flush_tlb will increase from 128 B to 512 B.
> * vcpu_bitmap in ioapic_write_indirect will increase from 128 B to 512 B.
> * vp_bitmap in sparse_set_to_vcpu_mask will increase from 128 B to 512 B.
>
> Signed-off-by: Kyle Meyer <kyle.meyer@hpe.com>
> ---
>  arch/x86/include/asm/kvm_host.h | 4 ++++
>  1 file changed, 4 insertions(+)
>
> diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
> index 9217bd6cf0d1..867a945f0152 100644
> --- a/arch/x86/include/asm/kvm_host.h
> +++ b/arch/x86/include/asm/kvm_host.h
> @@ -38,7 +38,11 @@
>  
>  #define __KVM_HAVE_ARCH_VCPU_DEBUGFS
>  
> +#if NR_CPUS < 1024
>  #define KVM_MAX_VCPUS 1024
> +#else
> +#define KVM_MAX_VCPUS 4096
> +#endif
>  
>  /*
>   * In x86, the VCPU ID corresponds to the APIC ID, and APIC IDs

-- 
Vitaly

Re: [PATCH v2] KVM: x86: Increase KVM_MAX_VCPUS to 4096

[Sean Christopherson]

On Thu, Jun 30, 2022, Vitaly Kuznetsov wrote:
> Kyle Meyer <kyle.meyer@hpe.com> writes:
> 
> > Increase KVM_MAX_VCPUS to 4096 when the maximum number of supported CPUs
> > (NR_CPUS) is greater than 1024.

Changelog says "greater than 1024", code says "at least 1024". 

> > The Hyper-V TLFS doesn't allow more than 64 sparse banks, which allows a
> > maximum of 4096 virtual CPUs. Limit KVM's maximum number of virtual CPUs
> > to 4096 to avoid exceeding that limit.
> 
> In theory, it's just TLB flush and IPI hypercalls which have this
> limitation. Strictly speaking, guest can have more than 4096 vCPUs,
> it'll just have to do IPIs/TLB flush in a different way.

Yeah, I don't see any reason to arbitrarily limit this to 4096.  And conversely,
I don't see any reason to force it to 4096 if NR_CPUS < 4096, it seems highly
unlikely that there's a use case for oversubscribing vCPUs in a single VM when
there are more than 1024 pCPUs.

Re: [PATCH] KVM: x86: Increase KVM_MAX_VCPUS to 2048

[Meyer, Kyle]

> > > Increase KVM_MAX_VCPUS to 2048 so we can run larger virtual machines.
> >
> > Does the host machine have 2048 CPUs (or more) as well in your usecase?

No, the largest system that I've used had 1792 CPUs.

> > I'm wondering if it makes sense to start configuring KVM_MAX_VCPUS
> > based on NR_CPUS. That way KVM can scale up on large machines without
> > using more memory on small machines.
> >
> > e.g.
> >
> > /* Provide backwards compatibility. */
> > #if NR_CPUS < 1024
> >   #define KVM_MAX_VCPUS 1024
> > #else
> >   #define KVM_MAX_VCPUS NR_CPUS
> > #endif
> >
> > The only downside I can see for this approach is if you are trying to
> > kick the tires a new large VM on a smaller host because the new "large
> > host" hardware hasn't landed yet.
>
> FWIW, while I don't think there's anything wrong with such approach, it
> won't help much distro kernels which are not recompiled to meet the
> needs of a particular host. According to Kyle's numbers, the biggest
> growth is observed with 'struct kvm_ioapic' and that's only because of
> 'struct rtc_status' embedded in it. Maybe it's possible to use something
> different from a KVM_MAX_VCPU_IDS-bound flat bitmask there? I'm not sure
> how important this is as it's just another 4K per-VM and when guest's
> memory is taken into account it's probably not much.
>
> The growth in 'struct kvm'/'struct kvm_arch' seems to be insignificant
> and on-stack allocations are probably OK.

If NR_CPUS is used, KVM_MAX_VCPUS will be 8192 and KVM_MAX_VCPU_IDS will be
32768 when MAXSMP is set.

Thanks,
Kyle Meyer

________________________________________
From: Vitaly Kuznetsov <vkuznets@redhat.com>
Sent: Tuesday, June 14, 2022 3:27 AM
To: David Matlack; Meyer, Kyle
Cc: kvm list; X86 ML; Borislav Petkov; Dave Hansen; Ingo Molnar; Thomas Gleixner; Anderson, Russ; Payton, Brian; H. Peter Anvin; Jim Mattson; Joerg Roedel; Sean Christopherson; Wanpeng Li
Subject: Re: [PATCH] KVM: x86: Increase KVM_MAX_VCPUS to 2048

David Matlack <dmatlack@google.com> writes:

> On Mon, Jun 13, 2022 at 11:35 AM Kyle Meyer <kyle.meyer@hpe.com> wrote:
>>
>> Increase KVM_MAX_VCPUS to 2048 so we can run larger virtual machines.
>
> Does the host machine have 2048 CPUs (or more) as well in your usecase?
>
> I'm wondering if it makes sense to start configuring KVM_MAX_VCPUS
> based on NR_CPUS. That way KVM can scale up on large machines without
> using more memory on small machines.
>
> e.g.
>
> /* Provide backwards compatibility. */
> #if NR_CPUS < 1024
>   #define KVM_MAX_VCPUS 1024
> #else
>   #define KVM_MAX_VCPUS NR_CPUS
> #endif
>
> The only downside I can see for this approach is if you are trying to
> kick the tires a new large VM on a smaller host because the new "large
> host" hardware hasn't landed yet.

FWIW, while I don't think there's anything wrong with such approach, it
won't help much distro kernels which are not recompiled to meet the
needs of a particular host. According to Kyle's numbers, the biggest
growth is observed with 'struct kvm_ioapic' and that's only because of
'struct rtc_status' embedded in it. Maybe it's possible to use something
different from a KVM_MAX_VCPU_IDS-bound flat bitmask there? I'm not sure
how important this is as it's just another 4K per-VM and when guest's
memory is taken into account it's probably not much.

The growth in 'struct kvm'/'struct kvm_arch' seems to be insignificant
and on-stack allocations are probably OK.

--
Vitaly