[PATCH v2] pc: q35: Bump max_cpus to 1024

[PATCH v2] pc: q35: Bump max_cpus to 1024

[Suravee Suthikulpanit]

Since KVM_MAX_VCPUS is currently defined to 1024 for x86 as shown in
arch/x86/include/asm/kvm_host.h, update QEMU limits to the same number.

In case KVM could not support the specified number of vcpus, QEMU would
return the following error message:

  qemu-system-x86_64: kvm_init_vcpu: kvm_get_vcpu failed (xxx): Invalid argument

Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---

Changes from V1:
(https://lore.kernel.org/all/YnkDGsIi1vFvXmiP@redhat.com/T/)
 * Bump from 512 to KVM_MAX_VCPUS (per Igor's suggestion)

Note:
 From the last discussion, Daniel mentioned that SMBIO 2.1 tables might
 cause overflow at approx 720 CPUs, and it might require using the
 SMBIO 3.0 entry point. Also, we might need to change the default for
 the x86 machine type to SMBIO 3.0. However, I do not know the status
 of this.

Thank you,
Suravee

 hw/i386/pc_q35.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
index f02919d92c..e7dc226bd5 100644
--- a/hw/i386/pc_q35.c
+++ b/hw/i386/pc_q35.c
@@ -370,7 +370,7 @@ static void pc_q35_machine_options(MachineClass *m)
     machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
     machine_class_allow_dynamic_sysbus_dev(m, TYPE_RAMFB_DEVICE);
     machine_class_allow_dynamic_sysbus_dev(m, TYPE_VMBUS_BRIDGE);
-    m->max_cpus = 288;
+    m->max_cpus = 1024;
 }
 
 static void pc_q35_8_1_machine_options(MachineClass *m)
-- 
2.34.1

Re: [PATCH v2] pc: q35: Bump max_cpus to 1024

[Joao Martins]

On 31/05/2023 23:51, Suravee Suthikulpanit wrote:
> Since KVM_MAX_VCPUS is currently defined to 1024 for x86 as shown in
> arch/x86/include/asm/kvm_host.h, update QEMU limits to the same number.
> 
> In case KVM could not support the specified number of vcpus, QEMU would
> return the following error message:
> 
>   qemu-system-x86_64: kvm_init_vcpu: kvm_get_vcpu failed (xxx): Invalid argument
> 
> Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
> ---
> 
> Changes from V1:
> (https://lore.kernel.org/all/YnkDGsIi1vFvXmiP@redhat.com/T/)
>  * Bump from 512 to KVM_MAX_VCPUS (per Igor's suggestion)
> 
> Note:
>  From the last discussion, Daniel mentioned that SMBIO 2.1 tables might
>  cause overflow at approx 720 CPUs, and it might require using the
>  SMBIO 3.0 entry point. Also, we might need to change the default for
>  the x86 machine type to SMBIO 3.0. However, I do not know the status
>  of this.
> 
I suspect smbios 3.0 (64-bit entry point) is already supported.

With current qemu and all the smbios fixes in the last cycle, perhaps this is
mainly just setting smbios_entry_point_type to SMBIOS_ENTRY_POINT_TYPE_64 if
MachineState::smp::max_cpus is bigger than 720 (e.g. in pc_q35_init()?)

> Thank you,
> Suravee
> 
>  hw/i386/pc_q35.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
> index f02919d92c..e7dc226bd5 100644
> --- a/hw/i386/pc_q35.c
> +++ b/hw/i386/pc_q35.c
> @@ -370,7 +370,7 @@ static void pc_q35_machine_options(MachineClass *m)
>      machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
>      machine_class_allow_dynamic_sysbus_dev(m, TYPE_RAMFB_DEVICE);
>      machine_class_allow_dynamic_sysbus_dev(m, TYPE_VMBUS_BRIDGE);
> -    m->max_cpus = 288;
> +    m->max_cpus = 1024;
>  }
>  
>  static void pc_q35_8_1_machine_options(MachineClass *m)

Re: [PATCH v2] pc: q35: Bump max_cpus to 1024

[Daniel P. Berrangé]

On Thu, Jun 01, 2023 at 11:09:45AM +0100, Joao Martins wrote:
> 
> 
> On 31/05/2023 23:51, Suravee Suthikulpanit wrote:
> > Since KVM_MAX_VCPUS is currently defined to 1024 for x86 as shown in
> > arch/x86/include/asm/kvm_host.h, update QEMU limits to the same number.
> > 
> > In case KVM could not support the specified number of vcpus, QEMU would
> > return the following error message:
> > 
> >   qemu-system-x86_64: kvm_init_vcpu: kvm_get_vcpu failed (xxx): Invalid argument
> > 
> > Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
> > ---
> > 
> > Changes from V1:
> > (https://lore.kernel.org/all/YnkDGsIi1vFvXmiP@redhat.com/T/)
> >  * Bump from 512 to KVM_MAX_VCPUS (per Igor's suggestion)
> > 
> > Note:
> >  From the last discussion, Daniel mentioned that SMBIO 2.1 tables might
> >  cause overflow at approx 720 CPUs, and it might require using the
> >  SMBIO 3.0 entry point. Also, we might need to change the default for
> >  the x86 machine type to SMBIO 3.0. However, I do not know the status
> >  of this.
> > 
> I suspect smbios 3.0 (64-bit entry point) is already supported.
> 
> With current qemu and all the smbios fixes in the last cycle, perhaps this is
> mainly just setting smbios_entry_point_type to SMBIOS_ENTRY_POINT_TYPE_64 if
> MachineState::smp::max_cpus is bigger than 720 (e.g. in pc_q35_init()?)

The need for the 64-bit entry point depends on a combination of RAM config
and CPU count. IMHO we need to unconditionally switch the latest machine
types to use the 64-bit entry point by default, rather than trying to infer
some special condition to dynamically change on the fly.

Before doing that we need confirmation that our firmware builds in qemu.git
(SeaBIOS and EDK2) will be happy with the 64-bit EP. I suspect they should
be fine by now, but I've CC'd Gerd to confirm.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2] pc: q35: Bump max_cpus to 1024

[Gerd Hoffmann]

  Hi,

> > I suspect smbios 3.0 (64-bit entry point) is already supported.
> > 
> > With current qemu and all the smbios fixes in the last cycle, perhaps this is
> > mainly just setting smbios_entry_point_type to SMBIOS_ENTRY_POINT_TYPE_64 if
> > MachineState::smp::max_cpus is bigger than 720 (e.g. in pc_q35_init()?)
> 
> The need for the 64-bit entry point depends on a combination of RAM config
> and CPU count. IMHO we need to unconditionally switch the latest machine
> types to use the 64-bit entry point by default, rather than trying to infer
> some special condition to dynamically change on the fly.
> 
> Before doing that we need confirmation that our firmware builds in qemu.git
> (SeaBIOS and EDK2) will be happy with the 64-bit EP. I suspect they should
> be fine by now, but I've CC'd Gerd to confirm.

edk2 is ready for quite a while (years).

seabios was lagging behind for a long time, but smbios 3.0 support
finally landed early last year, in release 1.16.0 (qemu 7.0).

So, yes, firmware-wise we are ready to flip the default now.

take care,
  Gerd

Re: [PATCH v2] pc: q35: Bump max_cpus to 1024

[Michael S. Tsirkin]

On Thu, Jun 01, 2023 at 11:17:30AM +0100, Daniel P. Berrangé wrote:
> On Thu, Jun 01, 2023 at 11:09:45AM +0100, Joao Martins wrote:
> > 
> > 
> > On 31/05/2023 23:51, Suravee Suthikulpanit wrote:
> > > Since KVM_MAX_VCPUS is currently defined to 1024 for x86 as shown in
> > > arch/x86/include/asm/kvm_host.h, update QEMU limits to the same number.
> > > 
> > > In case KVM could not support the specified number of vcpus, QEMU would
> > > return the following error message:
> > > 
> > >   qemu-system-x86_64: kvm_init_vcpu: kvm_get_vcpu failed (xxx): Invalid argument
> > > 
> > > Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
> > > ---
> > > 
> > > Changes from V1:
> > > (https://lore.kernel.org/all/YnkDGsIi1vFvXmiP@redhat.com/T/)
> > >  * Bump from 512 to KVM_MAX_VCPUS (per Igor's suggestion)
> > > 
> > > Note:
> > >  From the last discussion, Daniel mentioned that SMBIO 2.1 tables might
> > >  cause overflow at approx 720 CPUs, and it might require using the
> > >  SMBIO 3.0 entry point. Also, we might need to change the default for
> > >  the x86 machine type to SMBIO 3.0. However, I do not know the status
> > >  of this.
> > > 
> > I suspect smbios 3.0 (64-bit entry point) is already supported.
> > 
> > With current qemu and all the smbios fixes in the last cycle, perhaps this is
> > mainly just setting smbios_entry_point_type to SMBIOS_ENTRY_POINT_TYPE_64 if
> > MachineState::smp::max_cpus is bigger than 720 (e.g. in pc_q35_init()?)
> 
> The need for the 64-bit entry point depends on a combination of RAM config
> and CPU count. IMHO we need to unconditionally switch the latest machine
> types to use the 64-bit entry point by default, rather than trying to infer
> some special condition to dynamically change on the fly.

Yes, makes sense.

> Before doing that we need confirmation that our firmware builds in qemu.git
> (SeaBIOS and EDK2) will be happy with the 64-bit EP. I suspect they should
> be fine by now, but I've CC'd Gerd to confirm.
> 
> With regards,
> Daniel

Re: [PATCH v2] pc: q35: Bump max_cpus to 1024

[Suthikulpanit, Suravee]

On 6/1/2023 6:09 PM, Michael S. Tsirkin wrote:
> On Thu, Jun 01, 2023 at 11:17:30AM +0100, Daniel P. Berrangé wrote:
>> On Thu, Jun 01, 2023 at 11:09:45AM +0100, Joao Martins wrote:
>>>
>>> On 31/05/2023 23:51, Suravee Suthikulpanit wrote:
>>>> Since KVM_MAX_VCPUS is currently defined to 1024 for x86 as shown in
>>>> arch/x86/include/asm/kvm_host.h, update QEMU limits to the same number.
>>>>
>>>> In case KVM could not support the specified number of vcpus, QEMU would
>>>> return the following error message:
>>>>
>>>>    qemu-system-x86_64: kvm_init_vcpu: kvm_get_vcpu failed (xxx): Invalid argument
>>>>
>>>> Signed-off-by: Suravee Suthikulpanit<suravee.suthikulpanit@amd.com>
>>>> ---
>>>>
>>>> Changes from V1:
>>>> (https://lore.kernel.org/all/YnkDGsIi1vFvXmiP@redhat.com/T/)
>>>>   * Bump from 512 to KVM_MAX_VCPUS (per Igor's suggestion)
>>>>
>>>> Note:
>>>>   From the last discussion, Daniel mentioned that SMBIO 2.1 tables might
>>>>   cause overflow at approx 720 CPUs, and it might require using the
>>>>   SMBIO 3.0 entry point. Also, we might need to change the default for
>>>>   the x86 machine type to SMBIO 3.0. However, I do not know the status
>>>>   of this.
>>>>
>>> I suspect smbios 3.0 (64-bit entry point) is already supported.
>>>
>>> With current qemu and all the smbios fixes in the last cycle, perhaps this is
>>> mainly just setting smbios_entry_point_type to SMBIOS_ENTRY_POINT_TYPE_64 if
>>> MachineState::smp::max_cpus is bigger than 720 (e.g. in pc_q35_init()?)
>> >> The need for the 64-bit entry point depends on a combination of RAM 
config
>> and CPU count. IMHO we need to unconditionally switch the latest machine
>> types to use the 64-bit entry point by default, rather than trying to infer
>> some special condition to dynamically change on the fly.
> Yes, makes sense.

Thanks all for the feedback. So, IIUC, here is how the SMBIOS entry 
point types would be affected by the QEMU options:

- pc-q35-8.1 and later, default to SMBIOS EP type 64.
- pc-q35-8.0 and older, default to SMBIOS EP type 32.
- User can override the type w/ QEMU option "-M .., 
smbios-entry-point-type=[32|64]"

Please let me know if I am missing anything. If this is accurate, I'll 
send out v3 with this change.

Thanks,
Suravee

Re: [PATCH v2] pc: q35: Bump max_cpus to 1024

[Daniel P. Berrangé]

On Thu, Jun 01, 2023 at 02:54:54PM -0700, Suthikulpanit, Suravee wrote:
> 
> 
> On 6/1/2023 6:09 PM, Michael S. Tsirkin wrote:
> > On Thu, Jun 01, 2023 at 11:17:30AM +0100, Daniel P. Berrangé wrote:
> > > On Thu, Jun 01, 2023 at 11:09:45AM +0100, Joao Martins wrote:
> > > > 
> > > > On 31/05/2023 23:51, Suravee Suthikulpanit wrote:
> > > > > Since KVM_MAX_VCPUS is currently defined to 1024 for x86 as shown in
> > > > > arch/x86/include/asm/kvm_host.h, update QEMU limits to the same number.
> > > > > 
> > > > > In case KVM could not support the specified number of vcpus, QEMU would
> > > > > return the following error message:
> > > > > 
> > > > >    qemu-system-x86_64: kvm_init_vcpu: kvm_get_vcpu failed (xxx): Invalid argument
> > > > > 
> > > > > Signed-off-by: Suravee Suthikulpanit<suravee.suthikulpanit@amd.com>
> > > > > ---
> > > > > 
> > > > > Changes from V1:
> > > > > (https://lore.kernel.org/all/YnkDGsIi1vFvXmiP@redhat.com/T/)
> > > > >   * Bump from 512 to KVM_MAX_VCPUS (per Igor's suggestion)
> > > > > 
> > > > > Note:
> > > > >   From the last discussion, Daniel mentioned that SMBIO 2.1 tables might
> > > > >   cause overflow at approx 720 CPUs, and it might require using the
> > > > >   SMBIO 3.0 entry point. Also, we might need to change the default for
> > > > >   the x86 machine type to SMBIO 3.0. However, I do not know the status
> > > > >   of this.
> > > > > 
> > > > I suspect smbios 3.0 (64-bit entry point) is already supported.
> > > > 
> > > > With current qemu and all the smbios fixes in the last cycle, perhaps this is
> > > > mainly just setting smbios_entry_point_type to SMBIOS_ENTRY_POINT_TYPE_64 if
> > > > MachineState::smp::max_cpus is bigger than 720 (e.g. in pc_q35_init()?)
> > > >> The need for the 64-bit entry point depends on a combination of
> > > RAM
> config
> > > and CPU count. IMHO we need to unconditionally switch the latest machine
> > > types to use the 64-bit entry point by default, rather than trying to infer
> > > some special condition to dynamically change on the fly.
> > Yes, makes sense.
> 
> Thanks all for the feedback. So, IIUC, here is how the SMBIOS entry point
> types would be affected by the QEMU options:
> 
> - pc-q35-8.1 and later, default to SMBIOS EP type 64.
> - pc-q35-8.0 and older, default to SMBIOS EP type 32.
> - User can override the type w/ QEMU option "-M ..,
> smbios-entry-point-type=[32|64]"
> 
> Please let me know if I am missing anything. If this is accurate, I'll send
> out v3 with this change.

Yes, and for pc-i440fx* too


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v2] pc: q35: Bump max_cpus to 1024

[Gerd Hoffmann]

On Fri, Nov 18, 2022 at 09:55:35AM +0100, Dario Faggioli wrote:
> Keep the old limit of 288 for machine versions 7.2 and earlier.
> 
> Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
> ---
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Richard Henderson <richard.henderson@linaro.org>
> Cc: Eduardo Habkost <eduardo@habkost.net>
> Cc: "Michael S. Tsirkin" <mst@redhat.com>
> Cc: Marcel Apfelbaum <marcel.apfelbaum@gmail.com>
> ---
> Changes from v1:
> - fix actually keeping the old max value for the 7.2 machine type,
>   which was the original goal, but was done wrongly
> 
> ---
> This is related to:
> 
> https://lore.kernel.org/qemu-devel/c705d0d8d6ed1a520b1ff92cb2f83fef19522d30.camel@suse.com/
> 
> With this applied to QEMU, I've been able to start a VM with as high as
> 980 vCPUs (even if I was on an host with 384 pCPUs, so everything was
> super slow!). After that, I started to see messages like this:
> 
> "SMBIOS 2.1 table length 66822 exceeds 65535"

We finally have smbios 3.0 support in both seabios and edk2, so we can
use that to avoid problems like this.  So when raising the limits for
new machine types we should also flip the smbios default to 3.0.

take care,
  Gerd

Re: [PATCH v2] pc: q35: Bump max_cpus to 1024

[Daniel P. Berrangé]

On Fri, Nov 18, 2022 at 09:55:35AM +0100, Dario Faggioli wrote:
> Keep the old limit of 288 for machine versions 7.2 and earlier.
> 
> Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
> ---
> Cc: Paolo Bonzini <pbonzini@redhat.com>
> Cc: Richard Henderson <richard.henderson@linaro.org>
> Cc: Eduardo Habkost <eduardo@habkost.net>
> Cc: "Michael S. Tsirkin" <mst@redhat.com>
> Cc: Marcel Apfelbaum <marcel.apfelbaum@gmail.com>
> ---
> Changes from v1:
> - fix actually keeping the old max value for the 7.2 machine type,
>   which was the original goal, but was done wrongly
> 
> ---
> This is related to:
> 
> https://lore.kernel.org/qemu-devel/c705d0d8d6ed1a520b1ff92cb2f83fef19522d30.camel@suse.com/
> 
> With this applied to QEMU, I've been able to start a VM with as high as
> 980 vCPUs (even if I was on an host with 384 pCPUs, so everything was
> super slow!). After that, I started to see messages like this:
> 
> "SMBIOS 2.1 table length 66822 exceeds 65535"

I'm suprised you got as high as 980 vCPUs. In testing real world
configs in RHEL, we hit problems in the early 700 vCPU level, as
such large vCPU counts are also going to have very large RAM sizes
and likely NUMA topology too, which IIUC expand SMBIOS usage.

> Basing on the discussion happening in that thread, I'm going straight
> to 1024, as it seems to me that it's going to be working well soon
> (especially considering that this is meant for next release, not for
> 7.2)
>
> Thanks and Regards
> ---
>  hw/i386/pc_q35.c |    3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
> index a496bd6e74..54804337e9 100644
> --- a/hw/i386/pc_q35.c
> +++ b/hw/i386/pc_q35.c
> @@ -367,7 +367,7 @@ static void pc_q35_machine_options(MachineClass *m)
>      machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
>      machine_class_allow_dynamic_sysbus_dev(m, TYPE_RAMFB_DEVICE);
>      machine_class_allow_dynamic_sysbus_dev(m, TYPE_VMBUS_BRIDGE);
> -    m->max_cpus = 288;
> +    m->max_cpus = 1024;
>  }
>  
>  static void pc_q35_7_2_machine_options(MachineClass *m)
> @@ -375,6 +375,7 @@ static void pc_q35_7_2_machine_options(MachineClass *m)
>      PCMachineClass *pcmc = PC_MACHINE_CLASS(m);
>      pc_q35_machine_options(m);
>      m->alias = "q35";
> +    m->max_cpus = 288;
>      pcmc->default_cpu_version = 1;
>  }
>  
> 
> 
> 

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

[PATCH v2] pc: q35: Bump max_cpus to 1024

[Dario Faggioli]

Keep the old limit of 288 for machine versions 7.2 and earlier.

Signed-off-by: Dario Faggioli <dfaggioli@suse.com>
---
Cc: Paolo Bonzini <pbonzini@redhat.com>
Cc: Richard Henderson <richard.henderson@linaro.org>
Cc: Eduardo Habkost <eduardo@habkost.net>
Cc: "Michael S. Tsirkin" <mst@redhat.com>
Cc: Marcel Apfelbaum <marcel.apfelbaum@gmail.com>
---
Changes from v1:
- fix actually keeping the old max value for the 7.2 machine type,
  which was the original goal, but was done wrongly

---
This is related to:

https://lore.kernel.org/qemu-devel/c705d0d8d6ed1a520b1ff92cb2f83fef19522d30.camel@suse.com/

With this applied to QEMU, I've been able to start a VM with as high as
980 vCPUs (even if I was on an host with 384 pCPUs, so everything was
super slow!). After that, I started to see messages like this:

"SMBIOS 2.1 table length 66822 exceeds 65535"

Basing on the discussion happening in that thread, I'm going straight
to 1024, as it seems to me that it's going to be working well soon
(especially considering that this is meant for next release, not for
7.2)

Thanks and Regards
---
 hw/i386/pc_q35.c |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
index a496bd6e74..54804337e9 100644
--- a/hw/i386/pc_q35.c
+++ b/hw/i386/pc_q35.c
@@ -367,7 +367,7 @@ static void pc_q35_machine_options(MachineClass *m)
     machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
     machine_class_allow_dynamic_sysbus_dev(m, TYPE_RAMFB_DEVICE);
     machine_class_allow_dynamic_sysbus_dev(m, TYPE_VMBUS_BRIDGE);
-    m->max_cpus = 288;
+    m->max_cpus = 1024;
 }
 
 static void pc_q35_7_2_machine_options(MachineClass *m)
@@ -375,6 +375,7 @@ static void pc_q35_7_2_machine_options(MachineClass *m)
     PCMachineClass *pcmc = PC_MACHINE_CLASS(m);
     pc_q35_machine_options(m);
     m->alias = "q35";
+    m->max_cpus = 288;
     pcmc->default_cpu_version = 1;
 }