[PATCH] pc: q35: Bump max_cpus to 512

[PATCH] pc: q35: Bump max_cpus to 512

[Suravee Suthikulpanit]

This is the maximum number of vCPU supported by
the AMD x2APIC virtualization.

Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
---
 hw/i386/pc_q35.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
index 302288342a..e82b1c690d 100644
--- a/hw/i386/pc_q35.c
+++ b/hw/i386/pc_q35.c
@@ -357,7 +357,7 @@ static void pc_q35_machine_options(MachineClass *m)
     machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
     machine_class_allow_dynamic_sysbus_dev(m, TYPE_RAMFB_DEVICE);
     machine_class_allow_dynamic_sysbus_dev(m, TYPE_VMBUS_BRIDGE);
-    m->max_cpus = 288;
+    m->max_cpus = 512;
 }
 
 static void pc_q35_7_1_machine_options(MachineClass *m)
-- 
2.25.1

Re: [PATCH] pc: q35: Bump max_cpus to 512

[Igor Mammedov]

On Wed, 4 May 2022 08:16:39 -0500
Suravee Suthikulpanit <suravee.suthikulpanit@amd.com> wrote:

> This is the maximum number of vCPU supported by
> the AMD x2APIC virtualization.
> 
> Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
> ---
>  hw/i386/pc_q35.c | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
> index 302288342a..e82b1c690d 100644
> --- a/hw/i386/pc_q35.c
> +++ b/hw/i386/pc_q35.c
> @@ -357,7 +357,7 @@ static void pc_q35_machine_options(MachineClass *m)
>      machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
>      machine_class_allow_dynamic_sysbus_dev(m, TYPE_RAMFB_DEVICE);
>      machine_class_allow_dynamic_sysbus_dev(m, TYPE_VMBUS_BRIDGE);
> -    m->max_cpus = 288;
> +    m->max_cpus = 512;

Maybe we should bump it to KVM VCPU maximum,
and make sure we error out if asked for combination of
hardware/irqchip is not usable.


>  }
>  
>  static void pc_q35_7_1_machine_options(MachineClass *m)

Re: [PATCH] pc: q35: Bump max_cpus to 512

[Suthikulpanit, Suravee]

Igor,

On 5/9/2022 2:12 PM, Igor Mammedov wrote:
> On Wed, 4 May 2022 08:16:39 -0500
> Suravee Suthikulpanit<suravee.suthikulpanit@amd.com>  wrote:
> 
>> This is the maximum number of vCPU supported by
>> the AMD x2APIC virtualization.
>>
>> Signed-off-by: Suravee Suthikulpanit<suravee.suthikulpanit@amd.com>
>> ---
>>   hw/i386/pc_q35.c | 2 +-
>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>
>> diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
>> index 302288342a..e82b1c690d 100644
>> --- a/hw/i386/pc_q35.c
>> +++ b/hw/i386/pc_q35.c
>> @@ -357,7 +357,7 @@ static void pc_q35_machine_options(MachineClass *m)
>>       machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
>>       machine_class_allow_dynamic_sysbus_dev(m, TYPE_RAMFB_DEVICE);
>>       machine_class_allow_dynamic_sysbus_dev(m, TYPE_VMBUS_BRIDGE);
>> -    m->max_cpus = 288;
>> +    m->max_cpus = 512;
> Maybe we should bump it to KVM VCPU maximum,

Grepping through the Linux kernel source, the KVM_MAX_VCPUS macro is defined per architecture.
AFAICT, the absolute maximum is for x86, which is 1024. Does that sound about right?

> and make sure we error out if asked for combination of
> hardware/irqchip is not usable.

Could you please elaborate on this part?

Thanks,
Suravee

Re: [PATCH] pc: q35: Bump max_cpus to 512

[Daniel P. Berrangé]

On Mon, May 09, 2022 at 09:12:49AM +0200, Igor Mammedov wrote:
> On Wed, 4 May 2022 08:16:39 -0500
> Suravee Suthikulpanit <suravee.suthikulpanit@amd.com> wrote:
> 
> > This is the maximum number of vCPU supported by
> > the AMD x2APIC virtualization.
> > 
> > Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
> > ---
> >  hw/i386/pc_q35.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
> > index 302288342a..e82b1c690d 100644
> > --- a/hw/i386/pc_q35.c
> > +++ b/hw/i386/pc_q35.c
> > @@ -357,7 +357,7 @@ static void pc_q35_machine_options(MachineClass *m)
> >      machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
> >      machine_class_allow_dynamic_sysbus_dev(m, TYPE_RAMFB_DEVICE);
> >      machine_class_allow_dynamic_sysbus_dev(m, TYPE_VMBUS_BRIDGE);
> > -    m->max_cpus = 288;
> > +    m->max_cpus = 512;
> 
> Maybe we should bump it to KVM VCPU maximum,
> and make sure we error out if asked for combination of
> hardware/irqchip is not usable.

In RHEL downstream we currently bump this to 710 CPUs, because you
overflow the SMBIOS 2.1 tables at approx 720 CPUs (give/take a little
depending on other config options).

Going beyond 710 CPUs value requires using the SMBIOS 3 entry point.

AFAIK, the x86 machine types still default to SMBIOS 2.1, so that
would need changing too.

With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH] pc: q35: Bump max_cpus to 512

[Igor Mammedov]

On Mon, 9 May 2022 13:03:38 +0100
Daniel P. Berrangé <berrange@redhat.com> wrote:

> On Mon, May 09, 2022 at 09:12:49AM +0200, Igor Mammedov wrote:
> > On Wed, 4 May 2022 08:16:39 -0500
> > Suravee Suthikulpanit <suravee.suthikulpanit@amd.com> wrote:
> >   
> > > This is the maximum number of vCPU supported by
> > > the AMD x2APIC virtualization.
> > > 
> > > Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
> > > ---
> > >  hw/i386/pc_q35.c | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
> > > index 302288342a..e82b1c690d 100644
> > > --- a/hw/i386/pc_q35.c
> > > +++ b/hw/i386/pc_q35.c
> > > @@ -357,7 +357,7 @@ static void pc_q35_machine_options(MachineClass *m)
> > >      machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
> > >      machine_class_allow_dynamic_sysbus_dev(m, TYPE_RAMFB_DEVICE);
> > >      machine_class_allow_dynamic_sysbus_dev(m, TYPE_VMBUS_BRIDGE);
> > > -    m->max_cpus = 288;
> > > +    m->max_cpus = 512;  
> > 
> > Maybe we should bump it to KVM VCPU maximum,
> > and make sure we error out if asked for combination of
> > hardware/irqchip is not usable.  
> 
> In RHEL downstream we currently bump this to 710 CPUs, because you
> overflow the SMBIOS 2.1 tables at approx 720 CPUs (give/take a little
> depending on other config options).

Also there were some testing done with 1024,
but my main reason for matching KVM's limit is unblock upstream
testing so it would be easier to push limits for others.
Downstream can clamp that value down to whatever it deems as supported.

> Going beyond 710 CPUs value requires using the SMBIOS 3 entry point.
> 
> AFAIK, the x86 machine types still default to SMBIOS 2.1, so that
> would need changing too.

Yep, we can change default to SMBIOS 3 starting with new machine type (7.1?)
or conditionally depending on requested number of CPUs,
though I'd prefer machine type approach.

As for SMBIOS 3, we still have to update CPU structures to support more than
255 vcpus (Julia was working on it). It's long standing bug, but that doesn't
seem to be critical, as guest boots fine with old structures.

Other than that I don't recall any outright bugs related to large number of VCPUs.

> With regards,
> Daniel

Re: [PATCH] pc: q35: Bump max_cpus to 512

[Igor Mammedov]

On Mon, 9 May 2022 18:41:17 +0700
"Suthikulpanit, Suravee" <suravee.suthikulpanit@amd.com> wrote:

> Igor,
> 
> On 5/9/2022 2:12 PM, Igor Mammedov wrote:
> > On Wed, 4 May 2022 08:16:39 -0500
> > Suravee Suthikulpanit<suravee.suthikulpanit@amd.com>  wrote:
> >   
> >> This is the maximum number of vCPU supported by
> >> the AMD x2APIC virtualization.
> >>
> >> Signed-off-by: Suravee Suthikulpanit<suravee.suthikulpanit@amd.com>
> >> ---
> >>   hw/i386/pc_q35.c | 2 +-
> >>   1 file changed, 1 insertion(+), 1 deletion(-)
> >>
> >> diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
> >> index 302288342a..e82b1c690d 100644
> >> --- a/hw/i386/pc_q35.c
> >> +++ b/hw/i386/pc_q35.c
> >> @@ -357,7 +357,7 @@ static void pc_q35_machine_options(MachineClass *m)
> >>       machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
> >>       machine_class_allow_dynamic_sysbus_dev(m, TYPE_RAMFB_DEVICE);
> >>       machine_class_allow_dynamic_sysbus_dev(m, TYPE_VMBUS_BRIDGE);
> >> -    m->max_cpus = 288;
> >> +    m->max_cpus = 512;  
> > Maybe we should bump it to KVM VCPU maximum,  
> 
> Grepping through the Linux kernel source, the KVM_MAX_VCPUS macro is defined per architecture.
> AFAICT, the absolute maximum is for x86, which is 1024. Does that sound about right?

yep

> > and make sure we error out if asked for combination of
> > hardware/irqchip is not usable.  
> 
> Could you please elaborate on this part?

I meant that is you start with large VCPU number but accelerator/irqchip
can't create correct (usable) VM, QEMU should exit with error.
 
> Thanks,
> Suravee
> 

Re: [PATCH] pc: q35: Bump max_cpus to 512

[Daniel P. Berrangé]

On Tue, May 10, 2022 at 09:03:25AM +0200, Igor Mammedov wrote:
> On Mon, 9 May 2022 13:03:38 +0100
> Daniel P. Berrangé <berrange@redhat.com> wrote:
> 
> > On Mon, May 09, 2022 at 09:12:49AM +0200, Igor Mammedov wrote:
> > > On Wed, 4 May 2022 08:16:39 -0500
> > > Suravee Suthikulpanit <suravee.suthikulpanit@amd.com> wrote:
> > >   
> > > > This is the maximum number of vCPU supported by
> > > > the AMD x2APIC virtualization.
> > > > 
> > > > Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
> > > > ---
> > > >  hw/i386/pc_q35.c | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > 
> > > > diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
> > > > index 302288342a..e82b1c690d 100644
> > > > --- a/hw/i386/pc_q35.c
> > > > +++ b/hw/i386/pc_q35.c
> > > > @@ -357,7 +357,7 @@ static void pc_q35_machine_options(MachineClass *m)
> > > >      machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
> > > >      machine_class_allow_dynamic_sysbus_dev(m, TYPE_RAMFB_DEVICE);
> > > >      machine_class_allow_dynamic_sysbus_dev(m, TYPE_VMBUS_BRIDGE);
> > > > -    m->max_cpus = 288;
> > > > +    m->max_cpus = 512;  
> > > 
> > > Maybe we should bump it to KVM VCPU maximum,
> > > and make sure we error out if asked for combination of
> > > hardware/irqchip is not usable.  
> > 
> > In RHEL downstream we currently bump this to 710 CPUs, because you
> > overflow the SMBIOS 2.1 tables at approx 720 CPUs (give/take a little
> > depending on other config options).
> 
> Also there were some testing done with 1024,
> but my main reason for matching KVM's limit is unblock upstream
> testing so it would be easier to push limits for others.
> Downstream can clamp that value down to whatever it deems as supported.
> 
> > Going beyond 710 CPUs value requires using the SMBIOS 3 entry point.
> > 
> > AFAIK, the x86 machine types still default to SMBIOS 2.1, so that
> > would need changing too.
> 
> Yep, we can change default to SMBIOS 3 starting with new machine type (7.1?)
> or conditionally depending on requested number of CPUs,
> though I'd prefer machine type approach.

Agree, machine type is better IMHO, a it gives us a consistent guest
ABI regardless of CPU count.

> As for SMBIOS 3, we still have to update CPU structures to support more than
> 255 vcpus (Julia was working on it). It's long standing bug, but that doesn't
> seem to be critical, as guest boots fine with old structures.

What's the impact of SMBIOS 3 being limited to 255 ?  That's lower than
the current max CPUs of 288 in upstream / 710 in downstream.


With regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH] pc: q35: Bump max_cpus to 512

[Igor Mammedov]

On Tue, 10 May 2022 09:14:19 +0100
Daniel P. Berrangé <berrange@redhat.com> wrote:

> On Tue, May 10, 2022 at 09:03:25AM +0200, Igor Mammedov wrote:
> > On Mon, 9 May 2022 13:03:38 +0100
> > Daniel P. Berrangé <berrange@redhat.com> wrote:
> >   
> > > On Mon, May 09, 2022 at 09:12:49AM +0200, Igor Mammedov wrote:  
> > > > On Wed, 4 May 2022 08:16:39 -0500
> > > > Suravee Suthikulpanit <suravee.suthikulpanit@amd.com> wrote:
> > > >     
> > > > > This is the maximum number of vCPU supported by
> > > > > the AMD x2APIC virtualization.
> > > > > 
> > > > > Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
> > > > > ---
> > > > >  hw/i386/pc_q35.c | 2 +-
> > > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > > 
> > > > > diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
> > > > > index 302288342a..e82b1c690d 100644
> > > > > --- a/hw/i386/pc_q35.c
> > > > > +++ b/hw/i386/pc_q35.c
> > > > > @@ -357,7 +357,7 @@ static void pc_q35_machine_options(MachineClass *m)
> > > > >      machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
> > > > >      machine_class_allow_dynamic_sysbus_dev(m, TYPE_RAMFB_DEVICE);
> > > > >      machine_class_allow_dynamic_sysbus_dev(m, TYPE_VMBUS_BRIDGE);
> > > > > -    m->max_cpus = 288;
> > > > > +    m->max_cpus = 512;    
> > > > 
> > > > Maybe we should bump it to KVM VCPU maximum,
> > > > and make sure we error out if asked for combination of
> > > > hardware/irqchip is not usable.    
> > > 
> > > In RHEL downstream we currently bump this to 710 CPUs, because you
> > > overflow the SMBIOS 2.1 tables at approx 720 CPUs (give/take a little
> > > depending on other config options).  
> > 
> > Also there were some testing done with 1024,
> > but my main reason for matching KVM's limit is unblock upstream
> > testing so it would be easier to push limits for others.
> > Downstream can clamp that value down to whatever it deems as supported.
> >   
> > > Going beyond 710 CPUs value requires using the SMBIOS 3 entry point.
> > > 
> > > AFAIK, the x86 machine types still default to SMBIOS 2.1, so that
> > > would need changing too.  
> > 
> > Yep, we can change default to SMBIOS 3 starting with new machine type (7.1?)
> > or conditionally depending on requested number of CPUs,
> > though I'd prefer machine type approach.  
> 
> Agree, machine type is better IMHO, a it gives us a consistent guest
> ABI regardless of CPU count.
> 
> > As for SMBIOS 3, we still have to update CPU structures to support more than
> > 255 vcpus (Julia was working on it). It's long standing bug, but that doesn't
> > seem to be critical, as guest boots fine with old structures.  
> 
> What's the impact of SMBIOS 3 being limited to 255 ?  That's lower than
> the current max CPUs of 288 in upstream / 710 in downstream.

possibly users that look into SMBIOS for licensing purposes and/or inventory
Julia told me that dmidecode somehow figures out correct number of total vcpus.

CCing Julia for patches ETA.

> 
> 
> With regards,
> Daniel

Re: [PATCH] pc: q35: Bump max_cpus to 512

[Michael S. Tsirkin]

On Mon, May 09, 2022 at 09:12:49AM +0200, Igor Mammedov wrote:
> On Wed, 4 May 2022 08:16:39 -0500
> Suravee Suthikulpanit <suravee.suthikulpanit@amd.com> wrote:
> 
> > This is the maximum number of vCPU supported by
> > the AMD x2APIC virtualization.
> > 
> > Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
> > ---
> >  hw/i386/pc_q35.c | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
> > index 302288342a..e82b1c690d 100644
> > --- a/hw/i386/pc_q35.c
> > +++ b/hw/i386/pc_q35.c
> > @@ -357,7 +357,7 @@ static void pc_q35_machine_options(MachineClass *m)
> >      machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
> >      machine_class_allow_dynamic_sysbus_dev(m, TYPE_RAMFB_DEVICE);
> >      machine_class_allow_dynamic_sysbus_dev(m, TYPE_VMBUS_BRIDGE);
> > -    m->max_cpus = 288;
> > +    m->max_cpus = 512;
> 
> Maybe we should bump it to KVM VCPU maximum,
> and make sure we error out if asked for combination of
> hardware/irqchip is not usable.

So what happens if one does 710 and then tries to use AMD x2APIC?
We'd like that to error out, right?

> 
> >  }
> >  
> >  static void pc_q35_7_1_machine_options(MachineClass *m)

Re: [PATCH] pc: q35: Bump max_cpus to 512

[Suravee Suthikulpanit]

On 5/13/22 6:23 PM, Michael S. Tsirkin wrote:
> On Mon, May 09, 2022 at 09:12:49AM +0200, Igor Mammedov wrote:
>> On Wed, 4 May 2022 08:16:39 -0500
>> Suravee Suthikulpanit<suravee.suthikulpanit@amd.com>  wrote:
>>
>>> This is the maximum number of vCPU supported by
>>> the AMD x2APIC virtualization.
>>>
>>> Signed-off-by: Suravee Suthikulpanit<suravee.suthikulpanit@amd.com>
>>> ---
>>>   hw/i386/pc_q35.c | 2 +-
>>>   1 file changed, 1 insertion(+), 1 deletion(-)
>>>
>>> diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
>>> index 302288342a..e82b1c690d 100644
>>> --- a/hw/i386/pc_q35.c
>>> +++ b/hw/i386/pc_q35.c
>>> @@ -357,7 +357,7 @@ static void pc_q35_machine_options(MachineClass *m)
>>>       machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
>>>       machine_class_allow_dynamic_sysbus_dev(m, TYPE_RAMFB_DEVICE);
>>>       machine_class_allow_dynamic_sysbus_dev(m, TYPE_VMBUS_BRIDGE);
>>> -    m->max_cpus = 288;
>>> +    m->max_cpus = 512;
>> Maybe we should bump it to KVM VCPU maximum,
>> and make sure we error out if asked for combination of
>> hardware/irqchip is not usable.
>
> So what happens if one does 710 and then tries to use AMD x2APIC?
> We'd like that to error out, right?
> 

Since the AMD SVM x2AVIC can support only upto 512 vCPUs,
KVM would return error when QEMU tries to create more than 512 vcpus
in the following call-path:

   kvm_vm_ioctl_create_vcpu
     kvm_arch_vcpu_create
       svm_vcpu_create
         avic_init_vcpu

Also, I tried to find a way to get the KVM VCPU max value via KVM_CAP_MAX_VCPUS ioctl,
but the accel/kvm/kvm-all.c:kvm_init() is called later than hw/core/machine-smp.c:
machine_parse_smp_config(), where it checks the specified QEMU -smp option.

So, if there is no objection, I will send out a patch to change m->max_cpus = 1024
to match current KVM limit.

Best Regards,
Suravee

Re: [PATCH] pc: q35: Bump max_cpus to 512

[Julia Suvorova]

On Tue, May 10, 2022 at 1:38 PM Igor Mammedov <imammedo@redhat.com> wrote:
>
> On Tue, 10 May 2022 09:14:19 +0100
> Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> > On Tue, May 10, 2022 at 09:03:25AM +0200, Igor Mammedov wrote:
> > > On Mon, 9 May 2022 13:03:38 +0100
> > > Daniel P. Berrangé <berrange@redhat.com> wrote:
> > >
> > > > On Mon, May 09, 2022 at 09:12:49AM +0200, Igor Mammedov wrote:
> > > > > On Wed, 4 May 2022 08:16:39 -0500
> > > > > Suravee Suthikulpanit <suravee.suthikulpanit@amd.com> wrote:
> > > > >
> > > > > > This is the maximum number of vCPU supported by
> > > > > > the AMD x2APIC virtualization.
> > > > > >
> > > > > > Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
> > > > > > ---
> > > > > >  hw/i386/pc_q35.c | 2 +-
> > > > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > > >
> > > > > > diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
> > > > > > index 302288342a..e82b1c690d 100644
> > > > > > --- a/hw/i386/pc_q35.c
> > > > > > +++ b/hw/i386/pc_q35.c
> > > > > > @@ -357,7 +357,7 @@ static void pc_q35_machine_options(MachineClass *m)
> > > > > >      machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
> > > > > >      machine_class_allow_dynamic_sysbus_dev(m, TYPE_RAMFB_DEVICE);
> > > > > >      machine_class_allow_dynamic_sysbus_dev(m, TYPE_VMBUS_BRIDGE);
> > > > > > -    m->max_cpus = 288;
> > > > > > +    m->max_cpus = 512;
> > > > >
> > > > > Maybe we should bump it to KVM VCPU maximum,
> > > > > and make sure we error out if asked for combination of
> > > > > hardware/irqchip is not usable.
> > > >
> > > > In RHEL downstream we currently bump this to 710 CPUs, because you
> > > > overflow the SMBIOS 2.1 tables at approx 720 CPUs (give/take a little
> > > > depending on other config options).
> > >
> > > Also there were some testing done with 1024,
> > > but my main reason for matching KVM's limit is unblock upstream
> > > testing so it would be easier to push limits for others.
> > > Downstream can clamp that value down to whatever it deems as supported.
> > >
> > > > Going beyond 710 CPUs value requires using the SMBIOS 3 entry point.
> > > >
> > > > AFAIK, the x86 machine types still default to SMBIOS 2.1, so that
> > > > would need changing too.
> > >
> > > Yep, we can change default to SMBIOS 3 starting with new machine type (7.1?)
> > > or conditionally depending on requested number of CPUs,
> > > though I'd prefer machine type approach.
> >
> > Agree, machine type is better IMHO, a it gives us a consistent guest
> > ABI regardless of CPU count.
> >
> > > As for SMBIOS 3, we still have to update CPU structures to support more than
> > > 255 vcpus (Julia was working on it). It's long standing bug, but that doesn't
> > > seem to be critical, as guest boots fine with old structures.
> >
> > What's the impact of SMBIOS 3 being limited to 255 ?  That's lower than
> > the current max CPUs of 288 in upstream / 710 in downstream.
>
> possibly users that look into SMBIOS for licensing purposes and/or inventory
> Julia told me that dmidecode somehow figures out correct number of total vcpus.
>
> CCing Julia for patches ETA.

Should be this week.

Best regards, Julia Suvorova.

>
> >
> >
> > With regards,
> > Daniel
>

Re: [PATCH] pc: q35: Bump max_cpus to 512

[Igor Mammedov]

On Thu, 19 May 2022 13:53:49 +0700
Suravee Suthikulpanit <suravee.suthikulpanit@amd.com> wrote:

> On 5/13/22 6:23 PM, Michael S. Tsirkin wrote:
> > On Mon, May 09, 2022 at 09:12:49AM +0200, Igor Mammedov wrote:  
> >> On Wed, 4 May 2022 08:16:39 -0500
> >> Suravee Suthikulpanit<suravee.suthikulpanit@amd.com>  wrote:
> >>  
> >>> This is the maximum number of vCPU supported by
> >>> the AMD x2APIC virtualization.
> >>>
> >>> Signed-off-by: Suravee Suthikulpanit<suravee.suthikulpanit@amd.com>
> >>> ---
> >>>   hw/i386/pc_q35.c | 2 +-
> >>>   1 file changed, 1 insertion(+), 1 deletion(-)
> >>>
> >>> diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
> >>> index 302288342a..e82b1c690d 100644
> >>> --- a/hw/i386/pc_q35.c
> >>> +++ b/hw/i386/pc_q35.c
> >>> @@ -357,7 +357,7 @@ static void pc_q35_machine_options(MachineClass *m)
> >>>       machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
> >>>       machine_class_allow_dynamic_sysbus_dev(m, TYPE_RAMFB_DEVICE);
> >>>       machine_class_allow_dynamic_sysbus_dev(m, TYPE_VMBUS_BRIDGE);
> >>> -    m->max_cpus = 288;
> >>> +    m->max_cpus = 512;  
> >> Maybe we should bump it to KVM VCPU maximum,
> >> and make sure we error out if asked for combination of
> >> hardware/irqchip is not usable.  
> >
> > So what happens if one does 710 and then tries to use AMD x2APIC?
> > We'd like that to error out, right?
> >   
> 
> Since the AMD SVM x2AVIC can support only upto 512 vCPUs,
> KVM would return error when QEMU tries to create more than 512 vcpus
> in the following call-path:
> 
>    kvm_vm_ioctl_create_vcpu
>      kvm_arch_vcpu_create
>        svm_vcpu_create
>          avic_init_vcpu

what is actual error message that end user will get from that failure?

 
> Also, I tried to find a way to get the KVM VCPU max value via KVM_CAP_MAX_VCPUS ioctl,
> but the accel/kvm/kvm-all.c:kvm_init() is called later than hw/core/machine-smp.c:
> machine_parse_smp_config(), where it checks the specified QEMU -smp option.
> 
> So, if there is no objection, I will send out a patch to change m->max_cpus = 1024
> to match current KVM limit.
> 
> Best Regards,
> Suravee
> 

Re: [PATCH] pc: q35: Bump max_cpus to 512

[Suravee Suthikulpanit]

On 5/19/22 7:37 PM, Igor Mammedov wrote:
> On Thu, 19 May 2022 13:53:49 +0700
> Suravee Suthikulpanit<suravee.suthikulpanit@amd.com>  wrote:
> 
>> On 5/13/22 6:23 PM, Michael S. Tsirkin wrote:
>>> On Mon, May 09, 2022 at 09:12:49AM +0200, Igor Mammedov wrote:
>>>> On Wed, 4 May 2022 08:16:39 -0500
>>>> Suravee Suthikulpanit<suravee.suthikulpanit@amd.com>   wrote:
>>>>   
>>>>> This is the maximum number of vCPU supported by
>>>>> the AMD x2APIC virtualization.
>>>>>
>>>>> Signed-off-by: Suravee Suthikulpanit<suravee.suthikulpanit@amd.com>
>>>>> ---
>>>>>    hw/i386/pc_q35.c | 2 +-
>>>>>    1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>
>>>>> diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
>>>>> index 302288342a..e82b1c690d 100644
>>>>> --- a/hw/i386/pc_q35.c
>>>>> +++ b/hw/i386/pc_q35.c
>>>>> @@ -357,7 +357,7 @@ static void pc_q35_machine_options(MachineClass *m)
>>>>>        machine_class_allow_dynamic_sysbus_dev(m, TYPE_INTEL_IOMMU_DEVICE);
>>>>>        machine_class_allow_dynamic_sysbus_dev(m, TYPE_RAMFB_DEVICE);
>>>>>        machine_class_allow_dynamic_sysbus_dev(m, TYPE_VMBUS_BRIDGE);
>>>>> -    m->max_cpus = 288;
>>>>> +    m->max_cpus = 512;
>>>> Maybe we should bump it to KVM VCPU maximum,
>>>> and make sure we error out if asked for combination of
>>>> hardware/irqchip is not usable.
>>> So what happens if one does 710 and then tries to use AMD x2APIC?
>>> We'd like that to error out, right?
>>>    
>> Since the AMD SVM x2AVIC can support only upto 512 vCPUs,
>> KVM would return error when QEMU tries to create more than 512 vcpus
>> in the following call-path:
>>
>>     kvm_vm_ioctl_create_vcpu
>>       kvm_arch_vcpu_create
>>         svm_vcpu_create
>>           avic_init_vcpu
> what is actual error message that end user will get from that failure?

qemu-system-x86_64: kvm_init_vcpu: kvm_get_vcpu failed (512): Invalid argument

Best Regards,
Suravee