Re: [PATCH] KVM: x86: Increase KVM_MAX_VCPUS to 2048

From: David Matlack <dmatlack@google.com>
To: Kyle Meyer <kyle.meyer@hpe.com>
Cc: kvm list <kvm@vger.kernel.org>, X86 ML <x86@kernel.org>,
	Borislav Petkov <bp@alien8.de>,
	Dave Hansen <dave.hansen@linux.intel.com>,
	Ingo Molnar <mingo@redhat.com>,
	Thomas Gleixner <tglx@linutronix.de>,
	russ.anderson@hpe.com, payton@hpe.com,
	"H. Peter Anvin" <hpa@zytor.com>,
	Jim Mattson <jmattson@google.com>, Joerg Roedel <joro@8bytes.org>,
	Sean Christopherson <seanjc@google.com>,
	Vitaly Kuznetsov <vkuznets@redhat.com>,
	Wanpeng Li <wanpengli@tencent.com>
Subject: Re: [PATCH] KVM: x86: Increase KVM_MAX_VCPUS to 2048
Date: Mon, 13 Jun 2022 13:24:55 -0700	[thread overview]
Message-ID: <CALzav=eWPiii4_zmYifdi_pSS6nUvMEchwQcvD+W2CfOR+-s8Q@mail.gmail.com> (raw)
In-Reply-To: <20220613145022.183105-1-kyle.meyer@hpe.com>

On Mon, Jun 13, 2022 at 11:35 AM Kyle Meyer <kyle.meyer@hpe.com> wrote:
>
> Increase KVM_MAX_VCPUS to 2048 so we can run larger virtual machines.

Does the host machine have 2048 CPUs (or more) as well in your usecase?

I'm wondering if it makes sense to start configuring KVM_MAX_VCPUS
based on NR_CPUS. That way KVM can scale up on large machines without
using more memory on small machines.

e.g.

/* Provide backwards compatibility. */
#if NR_CPUS < 1024
  #define KVM_MAX_VCPUS 1024
#else
  #define KVM_MAX_VCPUS NR_CPUS
#endif

The only downside I can see for this approach is if you are trying to
kick the tires a new large VM on a smaller host because the new "large
host" hardware hasn't landed yet.

>
> Notable changes:
>
> * KVM_CAP_MAX_VCPUS will return 2048.
> * KVM_MAX_VCPU_IDS will increase from 4096 to 8192.
> * KVM_HV_MAX_SPARSE_VCPU_SET_BITS will increase from 16 to 32.
>
> * CPUID[HYPERV_CPUID_IMPLEMENT_LIMITS (00x40000005)].EAX will now be 2048.
>
> * struct kvm will increase from 40336 B to 40464 B.
> * struct kvm_arch will increase from 34488 B to 34616 B.
> * struct kvm_ioapic will increase from 5240 B to 9848 B.
>
> * vcpu_mask in kvm_hv_flush_tlb will increase from 128 B to 256 B.
> * vcpu_mask in kvm_hv_send_ipi will increase from 128 B to 256 B.
> * vcpu_bitmap in ioapic_write_indirect will increase from 128 B to 256 B.
> * vp_bitmap in sparse_set_to_vcpu_mask will increase from 128 B to 256 B.
> * sparse_banks in kvm_hv_flush_tlb will increase from 128 B to 256 B.
> * sparse_banks in kvm_hv_send_ipi will increase from 128 B to 256 B.
>
> Signed-off-by: Kyle Meyer <kyle.meyer@hpe.com>
> ---
>  arch/x86/include/asm/kvm_host.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
>
> diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h
> index 3a240a64ac68..58653c63899f 100644
> --- a/arch/x86/include/asm/kvm_host.h
> +++ b/arch/x86/include/asm/kvm_host.h
> @@ -38,7 +38,7 @@
>
>  #define __KVM_HAVE_ARCH_VCPU_DEBUGFS
>
> -#define KVM_MAX_VCPUS 1024
> +#define KVM_MAX_VCPUS 2048
>
>  /*
>   * In x86, the VCPU ID corresponds to the APIC ID, and APIC IDs
> --
> 2.26.2
>