[Qemu-devel] CPU hotplug

[Qemu-devel] CPU hotplug

[Stefan Priebe]

Hello list,

what is the status of CPU hotplug support?

I tried latest 1.2rc1 kvm-qemu with vanilla kernel v3.5.2 but the VM 
just crashes when sending cpu_set X online through qm monitor.

Greets,
Stefan

Re: [Qemu-devel] CPU hotplug

[Igor Mammedov]

On Thu, 30 Aug 2012 11:06:21 +0200
Stefan Priebe <s.priebe@profihost.ag> wrote:

> Hello list,
> 
> what is the status of CPU hotplug support?
> 
> I tried latest 1.2rc1 kvm-qemu with vanilla kernel v3.5.2 but the VM 
> just crashes when sending cpu_set X online through qm monitor.
>
it's work in progress and not implemented upstream yet.
You could try RHEl6.3 if you'd like to play with tech-preview feature.
 
> Greets,
> Stefan
> 

I'll create cpu hotplug page on qemu wiki to track status and todo list if no
one objects to it.
-- 
Regards,
  Igor

Re: [Qemu-devel] CPU hotplug

[Andreas Färber]

Hello,

Am 30.08.2012 11:06, schrieb Stefan Priebe:
> I tried latest 1.2rc1 kvm-qemu with vanilla kernel v3.5.2 but the VM
> just crashes when sending cpu_set X online through qm monitor.

For SLES we're carrying a patch by Kamalesh Babulal that prevents this
(BNC#747339).

Don't know why that patch is not in upstream - Bo Yang?

Regards,
Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] CPU hotplug

[Michael Tokarev]

30.08.2012 19:41, Andreas Färber wrote:
> Hello,
> 
> Am 30.08.2012 11:06, schrieb Stefan Priebe:
>> I tried latest 1.2rc1 kvm-qemu with vanilla kernel v3.5.2 but the VM
>> just crashes when sending cpu_set X online through qm monitor.
> 
> For SLES we're carrying a patch by Kamalesh Babulal that prevents this
> (BNC#747339).

The same is for debian/ubuntu: http://bugs.debian.org/680551

/mjt

Re: [Qemu-devel] CPU hotplug

[Stefan Priebe]

Am 30.08.2012 um 17:41 schrieb Andreas Färber <afaerber@suse.de>:

> Hello,
> 
> Am 30.08.2012 11:06, schrieb Stefan Priebe:
>> I tried latest 1.2rc1 kvm-qemu with vanilla kernel v3.5.2 but the VM
>> just crashes when sending cpu_set X online through qm monitor.
> 
> For SLES we're carrying a patch by Kamalesh Babulal that prevents this
> (BNC#747339).
> 
> Don't know why that patch is not in upstream - Bo Yang?

But this just disables CPU Hotelzug and does Not fix it?

Stefan

Re: [Qemu-devel] CPU hotplug

[Andreas Färber]

Am 30.08.2012 18:35, schrieb Stefan Priebe:
> Am 30.08.2012 um 17:41 schrieb Andreas Färber <afaerber@suse.de>:
> 
>> Am 30.08.2012 11:06, schrieb Stefan Priebe:
>>> I tried latest 1.2rc1 kvm-qemu with vanilla kernel v3.5.2 but the VM
>>> just crashes when sending cpu_set X online through qm monitor.
>>
>> For SLES we're carrying a patch by Kamalesh Babulal that prevents this
>> (BNC#747339).
> 
> But this just disables CPU [hotplug] and does Not fix it?

It fixes the crash. Hotplug needs to be implemented first, and this has
been taking several months already (for x86, to be specific).

Andreas

-- 
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg, Germany
GF: Jeff Hawn, Jennifer Guild, Felix Imendörffer; HRB 16746 AG Nürnberg

Re: [Qemu-devel] CPU hotplug

[Stefan Priebe]

Am 30.08.2012 18:43, schrieb Andreas Färber:
> Am 30.08.2012 18:35, schrieb Stefan Priebe:
>> Am 30.08.2012 um 17:41 schrieb Andreas Färber <afaerber@suse.de>:
>>
>>> Am 30.08.2012 11:06, schrieb Stefan Priebe:
>>>> I tried latest 1.2rc1 kvm-qemu with vanilla kernel v3.5.2 but the VM
>>>> just crashes when sending cpu_set X online through qm monitor.
>>>
>>> For SLES we're carrying a patch by Kamalesh Babulal that prevents this
>>> (BNC#747339).
>>
>> But this just disables CPU [hotplug] and does Not fix it?
>
> It fixes the crash. Hotplug needs to be implemented first, and this has
> been taking several months already (for x86, to be specific).

Mhm RHEL 6.3 claims to support this?
Dynamic virtual CPU allocation
=>
https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/6.3_Release_Notes/virtualization.html 


Greets,
Stefan

Re: [Qemu-devel] CPU hotplug

[Igor Mammedov]

On Thu, 30 Aug 2012 19:23:14 +0200
Stefan Priebe <s.priebe@profihost.ag> wrote:

> Am 30.08.2012 18:43, schrieb Andreas Färber:
> > Am 30.08.2012 18:35, schrieb Stefan Priebe:
> >> Am 30.08.2012 um 17:41 schrieb Andreas Färber <afaerber@suse.de>:
> >>
> >>> Am 30.08.2012 11:06, schrieb Stefan Priebe:
> >>>> I tried latest 1.2rc1 kvm-qemu with vanilla kernel v3.5.2 but the VM
> >>>> just crashes when sending cpu_set X online through qm monitor.
> >>>
> >>> For SLES we're carrying a patch by Kamalesh Babulal that prevents this
> >>> (BNC#747339).
> >>
> >> But this just disables CPU [hotplug] and does Not fix it?
> >
> > It fixes the crash. Hotplug needs to be implemented first, and this has
> > been taking several months already (for x86, to be specific).
> 
> Mhm RHEL 6.3 claims to support this?
it's not officially supported, it's just tech-preview. That allows to play with
hotplug and uncover possible guest issues early.

> Dynamic virtual CPU allocation
> =>
> https://access.redhat.com/knowledge/docs/en-US/Red_Hat_Enterprise_Linux/6/html/6.3_Release_Notes/virtualization.html 
> 
> 
> Greets,
> Stefan


-- 
Regards,
  Igor

Re: [Qemu-devel] CPU hotplug

[Stefan Priebe]

Am 30.08.2012 20:40, schrieb Igor Mammedov:
>>>> Am 30.08.2012 um 17:41 schrieb Andreas Färber <afaerber@suse.de>:
>>>>
>>>>> Am 30.08.2012 11:06, schrieb Stefan Priebe:
>>>>>> I tried latest 1.2rc1 kvm-qemu with vanilla kernel v3.5.2 but the VM
>>>>>> just crashes when sending cpu_set X online through qm monitor.
>>>>>
>>>>> For SLES we're carrying a patch by Kamalesh Babulal that prevents this
>>>>> (BNC#747339).
>>>>
>>>> But this just disables CPU [hotplug] and does Not fix it?
>>>
>>> It fixes the crash. Hotplug needs to be implemented first, and this has
>>> been taking several months already (for x86, to be specific).
>>
>> Mhm RHEL 6.3 claims to support this?
> it's not officially supported, it's just tech-preview. That allows to play with
> hotplug and uncover possible guest issues early.

Yes, but does this mean that is doesn't work at RHEL too? i wasn't able 
to get it working any guest or host at all.

Greets,
Stefan

Re: [Qemu-devel] CPU hotplug

[Igor Mammedov]

On Thu, 30 Aug 2012 20:45:10 +0200
Stefan Priebe <s.priebe@profihost.ag> wrote:

> Am 30.08.2012 20:40, schrieb Igor Mammedov:
> >>>> Am 30.08.2012 um 17:41 schrieb Andreas Färber <afaerber@suse.de>:
> >>>>
> >>>>> Am 30.08.2012 11:06, schrieb Stefan Priebe:
> >>>>>> I tried latest 1.2rc1 kvm-qemu with vanilla kernel v3.5.2 but the VM
> >>>>>> just crashes when sending cpu_set X online through qm monitor.
> >>>>>
> >>>>> For SLES we're carrying a patch by Kamalesh Babulal that prevents this
> >>>>> (BNC#747339).
> >>>>
> >>>> But this just disables CPU [hotplug] and does Not fix it?
> >>>
> >>> It fixes the crash. Hotplug needs to be implemented first, and this has
> >>> been taking several months already (for x86, to be specific).
> >>
> >> Mhm RHEL 6.3 claims to support this?
> > it's not officially supported, it's just tech-preview. That allows to play with
> > hotplug and uncover possible guest issues early.
> 
> Yes, but does this mean that is doesn't work at RHEL too? i wasn't able 
> to get it working any guest or host at all.
It works with RHEL 6.3 host/guest combo. Extra testing is greatly appreciated. 

> Greets,
> Stefan
> 


-- 
Regards,
  Igor

Re: [Qemu-devel] CPU hotplug

[Stefan Priebe]

Am 30.08.2012 20:56, schrieb Igor Mammedov:
> On Thu, 30 Aug 2012 20:45:10 +0200
> Stefan Priebe <s.priebe@profihost.ag> wrote:
>
>> Am 30.08.2012 20:40, schrieb Igor Mammedov:
>>>>>> Am 30.08.2012 um 17:41 schrieb Andreas Färber <afaerber@suse.de>:
>>>>>>
>>>>>>> Am 30.08.2012 11:06, schrieb Stefan Priebe:
>>>>>>>> I tried latest 1.2rc1 kvm-qemu with vanilla kernel v3.5.2 but the VM
>>>>>>>> just crashes when sending cpu_set X online through qm monitor.
>>>>>>>
>>>>>>> For SLES we're carrying a patch by Kamalesh Babulal that prevents this
>>>>>>> (BNC#747339).
>>>>>>
>>>>>> But this just disables CPU [hotplug] and does Not fix it?
>>>>>
>>>>> It fixes the crash. Hotplug needs to be implemented first, and this has
>>>>> been taking several months already (for x86, to be specific).
>>>>
>>>> Mhm RHEL 6.3 claims to support this?
>>> it's not officially supported, it's just tech-preview. That allows to play with
>>> hotplug and uncover possible guest issues early.
>>
>> Yes, but does this mean that is doesn't work at RHEL too? i wasn't able
>> to get it working any guest or host at all.
> It works with RHEL 6.3 host/guest combo. Extra testing is greatly appreciated.

mhm OK. I'm using Debian Squeeze with qemu-kvm 1.2rc1 and vanilla kernel 
3.5.2 as guest AND host.

This just results in the known crash.

Stefan

Re: [Qemu-devel] CPU hotplug

[Eduardo Habkost]

On Wed, Feb 03, 2016 at 12:50:02PM +1100, David Gibson wrote:
> On Tue, Feb 02, 2016 at 04:33:27PM -0200, Eduardo Habkost wrote:
> > On Mon, Feb 01, 2016 at 04:35:17PM +1100, David Gibson wrote:
> > > Hi,
> > > 
> > > It seems to me we're getting rather bogged down in how to proceed with
> > > an improved CPU hotplug (and hot unplug) interface, both generically
> > > and for ppc in particular.
> > > 
> > > So here's a somewhat more concrete suggestion of a way forward, to see
> > > if we can get some consensus.
> > > 
> > > The biggest difficulty I think we're grappling with is that device-add
> > > is actually *not* a great interface to cpu hotplug.  Or rather, it's
> > > not great as the _only_ interface: in order to represent the many
> > > different constraints on how cpus can be plugged on various platforms,
> > > it's natural to use a heirarchy of cpu core / socket / package types
> > > specific to the specific platform or real-world cpu package being
> > > modeled.  However, for the normal case of a regular homogenous (and at
> > > least slightly para-virtualized) server, that interface is nasty for
> > > management layers because they have to know the right type to
> > > instantiate.
> > > 
> > > To address this, I'm proposing this two layer interface:
> > > 
> > > Layer 1: Low-level, device-add based
> > > 
> > >     * a new, generic cpu-package QOM type represents a group of 1 or
> > >       more cpu threads which can be hotplugged as a unit
> > >     * cpu-package is abstract and can't be instantiated directly
> > >     * archs and/or individual platforms have specific subtypes of
> > >       cpu-package which can be instantiated
> > >     * for platforms attempting to be faithful representations of real
> > >       hardware these subtypes would match the specific characteristics
> > >       of the real hardware devices.  In addition to the cpu threads,
> > >       they may have other on chip devices as sub-objects.
> > >     * for platforms which are paravirtual - or which have existing
> > >       firmware abstractions for cpu cores/sockets/packages/whatever -
> > >       these could be more abstract, but would still be tied to that
> > >       platform's constraints
> > >     * Depending on the platform the cpu-package object could have
> > >       further internal structure (e.g. a package object representing a
> > >       socket contains package objects representing each core, which in
> > >       turn contain cpu objects for each thread)
> > >         * Some crazy platform that has multiple daughterboards each with
> > >           several multi-chip-modules each with several chips, each
> > > 	  with several cores each with several threads could represent
> > > 	  that too.
> > 
> > What exactly in this approach makes it depend on device-add? We
> > could have something very similar based on creation of QOM
> > objects, for example.
> 
> Uh.. I guess it doesn't.  device_add just seemed the obvious thing to
> me.
> 
> > > What would be common to all the cpu-package subtypes is:
> > >     * A boolean "present" attribute ("realized" might already be
> > >       suitable, but I'm not certain)
> > 
> > "realized" might be suitable, but I am not even sure we want
> > cpu-package to be a TYPE_DEVICE subclass. It could be a simple
> > QOM class or even a QOM interface (machines could choose to
> > implement it as TYPE_DEVICE, or not).
> 
> Yeah, I think doing it as a QOM interface makes sense.
> 
> > >     * A generic means of determining the number of cpu threads in the
> > >       package, and enumerating those
> > 
> > This could be based on QOM links.
> 
> Yes, that makes sense.
> 
> > >     * A generic means of determining if the package is hotpluggable or
> > >       not
> > 
> > Isn't this a machine attribute, instead of a package attribute?
> 
> Not necessarily.  I was thinking of cases where for architectural
> reasons you can't hotplug chip/cpu/module 0 but can plug or unplug all
> the rest.

Right, that makes sense too. But you can't query the cpu-package
object unless it was already created. I guess that's another
reason to start with the Layer 2 solution (where there's no need
for on-the-fly creation of cpu-package objects).

> 
> > >     * They'd get listed in a standard place in the QOM tree
> > 
> > If we allow CPU thread enumeration and package enumeration be
> > based in QOM links, we can let machines implement those
> > interfaces without introducing QOM hierarchy requirements.
> > 
> > We have one example where we would need to make this flexible
> > enough about QOM hierarchy, below (thread-based hotplug in x86).
> 
> Makes sense.
> 
> > 
> > > 
> > > This interface is suitable if you want complete control over
> > > constructing the system, including weird cases like heterogeneous
> > > machines (either totally different cpu types, or just different
> > > numbers of threads in different packages).
> > > 
> > > The intention is that these objects would never look at the global cpu
> > > type or sockets/cores/threads numbers.  The next level up would
> > > instead configure the packages to match those for the common case.
> > > 
> > > Layer 2: Higher-level
> > > 
> > >     * not all machine types need support this model, but I'd expect
> > >       all future versions of machine types designed for production use
> > >       to do so
> > >     * machine types don't construct cpu objects directly
> > >     * instead they create enough cpu-package objects - of a subtype
> > >       suitable for this machine - to provide maxcpus threads
> > >     * the machine type would set the "present" bit on enough of the
> > >       cpu packages to provide the base number of cpu threads
> > 
> > Sounds interesting, and very simple for management code. What I
> > don't see is: what exactly makes it easier to implement just
> > Layer 1 and not Layer 2?
> 
> 
> > Implementing Layer 1 looks more difficult to me, because it
> > requires supporting creation of cpu-package objects on the fly,
> > using device_add (or whatever mechanism we choose for cpu-package
> > creation). Layer 2 lets the implementation choose freely when/how
> > exactly the other objects will be created and how exactly they
> > will appear in the device tree. They just need to do the right
> > thing when the "present" property is flipped.
> 
> Hmm.. good point.  Ok, how about this revised plan:
> 
>   1. Implement the QOM backend structures for cpu packages, but don't
>      allow them to be user instantiated
>   2. Implement Layer 2 in terms of (1)
>   3. When/if we need it, add the extra stuff necessary to allow direct
>      instantiation of the cpu packages

Makes sense to me. With generic mechanisms to let management
enumerate the socket/core/thread IDs contained/available in each
cpu-package slot, this should be good enough for all
architectures and still give management enough information to
decide how to plug/unplug specific CPUs.

I have another suggestion: if we are going to expose these
objects where new CPUs could be plugged by just flipping a
property, what about naming them "cpu-slot"? Because they
wouldn't represent an actual CPU package, but just a slot where a
CPU package could be plugged in. The initial (simpler) solution
could involve just flipping a property in the cpu-slot.  Later,
we may allow more complex scenarios where complex QOM objects are
created before linking them to the cpu-slot.

> 
> > > Management layers can then manage hotplug without knowing platform
> > > specifics by using qmp to toggle the "present" bit on packages.
> > > Platforms that allow thread-level pluggability can expose a package
> > > for every thread, those that allow core-level expose a package per
> > > core, those that have even less granularity expose a package at
> > > whatever grouping they can do hotplug on.
> > > 
> > > Examples:
> > > 
> > > For use with pc (or q35 or whatever) machine type, I'd expect a
> > > cpu-package subtype called, say "acpi-thread" which represents a
> > > single thread in the ACPI sense.  Toggling those would trigger ACPI
> > > hotplug events as cpu_add does now.
> > 
> > You have a good point here: I remember seeing suggestions of
> > making CPU hotplug tied to the socket/core/thread hierarchy
> > somehow. But this won't change the fact that x86 allows hotplug
> > of individual CPU threads.
> 
> Right, this seems to be where we're bogged down - we seem to be going
> back and forth betweeh core level, socket level, thread level
> proposals without really looking at the big picture to come up with a
> scheme that works for all platforms.
> 
> > In other words, if we make a /machine/socket/core/thread QOM
> > hierarchy, the cpu-packages for x86 won't necessarily represent
> > CPU sockets (but in other architectures, they might).  The
> > interface needs to be generic enough to not assume anything about
> > the CPU topology level where CPU hotplug happens.
> 
> Exactly.  That's what I see as the key advantage of this proposal over
> earlier ones.

Right.

> 
> > > For use with pseries, I'd expect a "papr-core" cpu-package subtype,
> > > which represents a single (paravirtual) core.  Toggling present on
> > > this would trigger the PAPR hotplug events.  A property would control
> > > the number of threads in the core (only settable before enabling
> > > present).
> > > 
> > > For use with the powernv machine type (once ready for merge) I'd
> > > expect "POWER8-package" type which represents a POWER8 chip / module
> > > as close to the real hardware as we can get.  It would have a fixed
> > > number of cores and threads within it as per the real hardware, and
> > > would also include xscoms and other per-module logic.
> > > 
> > > From here to there:
> > > 
> > > A suggested order of implementation to get there without too much risk
> > > of breaking things.
> > > 
> > >   1. Fix bugs with creation / removal of CPU objects (Bharata's cpu
> > >      hotplug series already has this)
> > >   2. Split creation and realization of CPU objects, so machine types
> > >      must explicitly perform both steps (Bharata's series has this
> > >      too)
> > >   3. Add the abstract cpu-package type, and define the generic
> > >      interfaces it needs (Bharata's series has something that could be
> > >      changed to this fairly easily)
> > >   4. For each machine type we care to convert:
> > >       4.1. Add platform suitable cpu-package subtypes
> > >       4.2. Convert the (latest version) machine type to instantiate packages instead of
> > >            cpu threads directly
> > 
> > Machines could even have the freedom to instantiate CPU threads
> > directly and then set up package objects for them. Reusing
> > generic code is useful, but it doesn't even need to be mandatory,
> > as long as the objects are available at the right place in the
> > QOM hierarchy.
> 
> Ah, yes, I guess so.
> 
> > 
> > >       4.3. Add any necessary backwards compat goo
> > >   5. Teach libvirt how to toggle cpu-packages
> > 
> > This is different from the very flexible QOM object
> > building/linking approach Andreas was talking about in last KVM
> > Forum.
> 
> So, I know this stuff was discussed at KVM Forum, but unfortunately I
> never got a clear picture of what the outcome was.

I didn't see a clear conclusion after the KVM Forum discussions,
except for some ideas to use QOM object creation/linking to
implement something flexible enough for all architectures. But I
don't think we were moving towards something generic enough that
would allow management to implement CPU hotplug without
arch-specific code to setup CPU
sockets/packages/cores/threads/etc.

> > But while I would love to have the ability to build
> > arbitrary QOM hierarchies with complex links between CPUs
> > sockets, cores, threads, etc, I believe we need an interface that
> > is: 1) generic enough for multiple architectures and machines to
> > implement them; 2) simple enough so that libvirt can use it
> > easily without requiring more arch-specific code.
> > 
> > Also, your approach doesn't prevent the simple cpu-package
> > interface from having a complex QOM hierarchy hidden behind it.
> 
> Exactly.

-- 
Eduardo

Re: [Qemu-devel] CPU hotplug

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 5408 bytes --]

On Wed, Feb 03, 2016 at 10:33:48AM +0530, Bharata B Rao wrote:
> On Mon, Feb 01, 2016 at 04:35:17PM +1100, David Gibson wrote:
> > Hi,
> > 
> > It seems to me we're getting rather bogged down in how to proceed with
> > an improved CPU hotplug (and hot unplug) interface, both generically
> > and for ppc in particular.
> > 
> > So here's a somewhat more concrete suggestion of a way forward, to see
> > if we can get some consensus.
> > 
> > The biggest difficulty I think we're grappling with is that device-add
> > is actually *not* a great interface to cpu hotplug.  Or rather, it's
> > not great as the _only_ interface: in order to represent the many
> > different constraints on how cpus can be plugged on various platforms,
> > it's natural to use a heirarchy of cpu core / socket / package types
> > specific to the specific platform or real-world cpu package being
> > modeled.  However, for the normal case of a regular homogenous (and at
> > least slightly para-virtualized) server, that interface is nasty for
> > management layers because they have to know the right type to
> > instantiate.
> > 
> > To address this, I'm proposing this two layer interface:
> > 
> > Layer 1: Low-level, device-add based
> > 
> >     * a new, generic cpu-package QOM type represents a group of 1 or
> >       more cpu threads which can be hotplugged as a unit
> >     * cpu-package is abstract and can't be instantiated directly
> >     * archs and/or individual platforms have specific subtypes of
> >       cpu-package which can be instantiated
> >     * for platforms attempting to be faithful representations of real
> >       hardware these subtypes would match the specific characteristics
> >       of the real hardware devices.  In addition to the cpu threads,
> >       they may have other on chip devices as sub-objects.
> >     * for platforms which are paravirtual - or which have existing
> >       firmware abstractions for cpu cores/sockets/packages/whatever -
> >       these could be more abstract, but would still be tied to that
> >       platform's constraints
> >     * Depending on the platform the cpu-package object could have
> >       further internal structure (e.g. a package object representing a
> >       socket contains package objects representing each core, which in
> >       turn contain cpu objects for each thread)
> >         * Some crazy platform that has multiple daughterboards each with
> >           several multi-chip-modules each with several chips, each
> > 	  with several cores each with several threads could represent
> > 	  that too.
> > 
> > What would be common to all the cpu-package subtypes is:
> >     * A boolean "present" attribute ("realized" might already be
> >       suitable, but I'm not certain)
> >     * A generic means of determining the number of cpu threads in the
> >       package, and enumerating those
> >     * A generic means of determining if the package is hotpluggable or
> >       not
> >     * They'd get listed in a standard place in the QOM tree
> > 
> > This interface is suitable if you want complete control over
> > constructing the system, including weird cases like heterogeneous
> > machines (either totally different cpu types, or just different
> > numbers of threads in different packages).
> > 
> > The intention is that these objects would never look at the global cpu
> > type or sockets/cores/threads numbers.  The next level up would
> > instead configure the packages to match those for the common case.
> > 
> > Layer 2: Higher-level
> > 
> >     * not all machine types need support this model, but I'd expect
> >       all future versions of machine types designed for production use
> >       to do so
> >     * machine types don't construct cpu objects directly
> >     * instead they create enough cpu-package objects - of a subtype
> >       suitable for this machine - to provide maxcpus threads
> >     * the machine type would set the "present" bit on enough of the
> >       cpu packages to provide the base number of cpu threads
> 
> In the generic cpu-core RFC that I posted last year
> (https://lists.gnu.org/archive/html/qemu-devel/2015-12/msg01526.html),
> I did have backend objects (which I called them sockets) into which
> the generic cpu-core device would fit it and I used the QOM links to
> bring out the notion of cpu-core device populating the socket.
> 
> I had the sockets as backend objects and created as many of them as needed
> upfront to fit the max_cpus. These objects weren't exposed them to the user,
> but instead the cpu-core device was exposed to the user.

Right, as I mentioned on IRC this is based partly on your earlier
proposal.

The big difference, as I see it, is that in this proposal the cpu
package objects aren't linked directly to the socket/core/thread
heirarchy - different platforms can place them differently based on
what works for them.

> However, I like the current proposal where Layer 2 interface is exposed to the
> user and letting archs build up the CPU topology underneath in the manner
> that they deem fit for the arch.
> 
> Regards,
> Bharata.
> 

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] CPU hotplug

[Bharata B Rao]

On Mon, Feb 01, 2016 at 04:35:17PM +1100, David Gibson wrote:
> Hi,
> 
> It seems to me we're getting rather bogged down in how to proceed with
> an improved CPU hotplug (and hot unplug) interface, both generically
> and for ppc in particular.
> 
> So here's a somewhat more concrete suggestion of a way forward, to see
> if we can get some consensus.
> 
> The biggest difficulty I think we're grappling with is that device-add
> is actually *not* a great interface to cpu hotplug.  Or rather, it's
> not great as the _only_ interface: in order to represent the many
> different constraints on how cpus can be plugged on various platforms,
> it's natural to use a heirarchy of cpu core / socket / package types
> specific to the specific platform or real-world cpu package being
> modeled.  However, for the normal case of a regular homogenous (and at
> least slightly para-virtualized) server, that interface is nasty for
> management layers because they have to know the right type to
> instantiate.
> 
> To address this, I'm proposing this two layer interface:
> 
> Layer 1: Low-level, device-add based
> 
>     * a new, generic cpu-package QOM type represents a group of 1 or
>       more cpu threads which can be hotplugged as a unit
>     * cpu-package is abstract and can't be instantiated directly
>     * archs and/or individual platforms have specific subtypes of
>       cpu-package which can be instantiated
>     * for platforms attempting to be faithful representations of real
>       hardware these subtypes would match the specific characteristics
>       of the real hardware devices.  In addition to the cpu threads,
>       they may have other on chip devices as sub-objects.
>     * for platforms which are paravirtual - or which have existing
>       firmware abstractions for cpu cores/sockets/packages/whatever -
>       these could be more abstract, but would still be tied to that
>       platform's constraints
>     * Depending on the platform the cpu-package object could have
>       further internal structure (e.g. a package object representing a
>       socket contains package objects representing each core, which in
>       turn contain cpu objects for each thread)
>         * Some crazy platform that has multiple daughterboards each with
>           several multi-chip-modules each with several chips, each
> 	  with several cores each with several threads could represent
> 	  that too.
> 
> What would be common to all the cpu-package subtypes is:
>     * A boolean "present" attribute ("realized" might already be
>       suitable, but I'm not certain)
>     * A generic means of determining the number of cpu threads in the
>       package, and enumerating those
>     * A generic means of determining if the package is hotpluggable or
>       not
>     * They'd get listed in a standard place in the QOM tree
> 
> This interface is suitable if you want complete control over
> constructing the system, including weird cases like heterogeneous
> machines (either totally different cpu types, or just different
> numbers of threads in different packages).
> 
> The intention is that these objects would never look at the global cpu
> type or sockets/cores/threads numbers.  The next level up would
> instead configure the packages to match those for the common case.
> 
> Layer 2: Higher-level
> 
>     * not all machine types need support this model, but I'd expect
>       all future versions of machine types designed for production use
>       to do so
>     * machine types don't construct cpu objects directly
>     * instead they create enough cpu-package objects - of a subtype
>       suitable for this machine - to provide maxcpus threads
>     * the machine type would set the "present" bit on enough of the
>       cpu packages to provide the base number of cpu threads

In the generic cpu-core RFC that I posted last year
(https://lists.gnu.org/archive/html/qemu-devel/2015-12/msg01526.html),
I did have backend objects (which I called them sockets) into which
the generic cpu-core device would fit it and I used the QOM links to
bring out the notion of cpu-core device populating the socket.

I had the sockets as backend objects and created as many of them as needed
upfront to fit the max_cpus. These objects weren't exposed them to the user,
but instead the cpu-core device was exposed to the user.

However, I like the current proposal where Layer 2 interface is exposed to the
user and letting archs build up the CPU topology underneath in the manner
that they deem fit for the arch.

Regards,
Bharata.

Re: [Qemu-devel] CPU hotplug

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 10694 bytes --]

On Tue, Feb 02, 2016 at 04:33:27PM -0200, Eduardo Habkost wrote:
> On Mon, Feb 01, 2016 at 04:35:17PM +1100, David Gibson wrote:
> > Hi,
> > 
> > It seems to me we're getting rather bogged down in how to proceed with
> > an improved CPU hotplug (and hot unplug) interface, both generically
> > and for ppc in particular.
> > 
> > So here's a somewhat more concrete suggestion of a way forward, to see
> > if we can get some consensus.
> > 
> > The biggest difficulty I think we're grappling with is that device-add
> > is actually *not* a great interface to cpu hotplug.  Or rather, it's
> > not great as the _only_ interface: in order to represent the many
> > different constraints on how cpus can be plugged on various platforms,
> > it's natural to use a heirarchy of cpu core / socket / package types
> > specific to the specific platform or real-world cpu package being
> > modeled.  However, for the normal case of a regular homogenous (and at
> > least slightly para-virtualized) server, that interface is nasty for
> > management layers because they have to know the right type to
> > instantiate.
> > 
> > To address this, I'm proposing this two layer interface:
> > 
> > Layer 1: Low-level, device-add based
> > 
> >     * a new, generic cpu-package QOM type represents a group of 1 or
> >       more cpu threads which can be hotplugged as a unit
> >     * cpu-package is abstract and can't be instantiated directly
> >     * archs and/or individual platforms have specific subtypes of
> >       cpu-package which can be instantiated
> >     * for platforms attempting to be faithful representations of real
> >       hardware these subtypes would match the specific characteristics
> >       of the real hardware devices.  In addition to the cpu threads,
> >       they may have other on chip devices as sub-objects.
> >     * for platforms which are paravirtual - or which have existing
> >       firmware abstractions for cpu cores/sockets/packages/whatever -
> >       these could be more abstract, but would still be tied to that
> >       platform's constraints
> >     * Depending on the platform the cpu-package object could have
> >       further internal structure (e.g. a package object representing a
> >       socket contains package objects representing each core, which in
> >       turn contain cpu objects for each thread)
> >         * Some crazy platform that has multiple daughterboards each with
> >           several multi-chip-modules each with several chips, each
> > 	  with several cores each with several threads could represent
> > 	  that too.
> 
> What exactly in this approach makes it depend on device-add? We
> could have something very similar based on creation of QOM
> objects, for example.

Uh.. I guess it doesn't.  device_add just seemed the obvious thing to
me.

> > What would be common to all the cpu-package subtypes is:
> >     * A boolean "present" attribute ("realized" might already be
> >       suitable, but I'm not certain)
> 
> "realized" might be suitable, but I am not even sure we want
> cpu-package to be a TYPE_DEVICE subclass. It could be a simple
> QOM class or even a QOM interface (machines could choose to
> implement it as TYPE_DEVICE, or not).

Yeah, I think doing it as a QOM interface makes sense.

> >     * A generic means of determining the number of cpu threads in the
> >       package, and enumerating those
> 
> This could be based on QOM links.

Yes, that makes sense.

> >     * A generic means of determining if the package is hotpluggable or
> >       not
> 
> Isn't this a machine attribute, instead of a package attribute?

Not necessarily.  I was thinking of cases where for architectural
reasons you can't hotplug chip/cpu/module 0 but can plug or unplug all
the rest.

> >     * They'd get listed in a standard place in the QOM tree
> 
> If we allow CPU thread enumeration and package enumeration be
> based in QOM links, we can let machines implement those
> interfaces without introducing QOM hierarchy requirements.
> 
> We have one example where we would need to make this flexible
> enough about QOM hierarchy, below (thread-based hotplug in x86).

Makes sense.

> 
> > 
> > This interface is suitable if you want complete control over
> > constructing the system, including weird cases like heterogeneous
> > machines (either totally different cpu types, or just different
> > numbers of threads in different packages).
> > 
> > The intention is that these objects would never look at the global cpu
> > type or sockets/cores/threads numbers.  The next level up would
> > instead configure the packages to match those for the common case.
> > 
> > Layer 2: Higher-level
> > 
> >     * not all machine types need support this model, but I'd expect
> >       all future versions of machine types designed for production use
> >       to do so
> >     * machine types don't construct cpu objects directly
> >     * instead they create enough cpu-package objects - of a subtype
> >       suitable for this machine - to provide maxcpus threads
> >     * the machine type would set the "present" bit on enough of the
> >       cpu packages to provide the base number of cpu threads
> 
> Sounds interesting, and very simple for management code. What I
> don't see is: what exactly makes it easier to implement just
> Layer 1 and not Layer 2?


> Implementing Layer 1 looks more difficult to me, because it
> requires supporting creation of cpu-package objects on the fly,
> using device_add (or whatever mechanism we choose for cpu-package
> creation). Layer 2 lets the implementation choose freely when/how
> exactly the other objects will be created and how exactly they
> will appear in the device tree. They just need to do the right
> thing when the "present" property is flipped.

Hmm.. good point.  Ok, how about this revised plan:

  1. Implement the QOM backend structures for cpu packages, but don't
     allow them to be user instantiated
  2. Implement Layer 2 in terms of (1)
  3. When/if we need it, add the extra stuff necessary to allow direct
     instantiation of the cpu packages

> > Management layers can then manage hotplug without knowing platform
> > specifics by using qmp to toggle the "present" bit on packages.
> > Platforms that allow thread-level pluggability can expose a package
> > for every thread, those that allow core-level expose a package per
> > core, those that have even less granularity expose a package at
> > whatever grouping they can do hotplug on.
> > 
> > Examples:
> > 
> > For use with pc (or q35 or whatever) machine type, I'd expect a
> > cpu-package subtype called, say "acpi-thread" which represents a
> > single thread in the ACPI sense.  Toggling those would trigger ACPI
> > hotplug events as cpu_add does now.
> 
> You have a good point here: I remember seeing suggestions of
> making CPU hotplug tied to the socket/core/thread hierarchy
> somehow. But this won't change the fact that x86 allows hotplug
> of individual CPU threads.

Right, this seems to be where we're bogged down - we seem to be going
back and forth betweeh core level, socket level, thread level
proposals without really looking at the big picture to come up with a
scheme that works for all platforms.

> In other words, if we make a /machine/socket/core/thread QOM
> hierarchy, the cpu-packages for x86 won't necessarily represent
> CPU sockets (but in other architectures, they might).  The
> interface needs to be generic enough to not assume anything about
> the CPU topology level where CPU hotplug happens.

Exactly.  That's what I see as the key advantage of this proposal over
earlier ones.

> > For use with pseries, I'd expect a "papr-core" cpu-package subtype,
> > which represents a single (paravirtual) core.  Toggling present on
> > this would trigger the PAPR hotplug events.  A property would control
> > the number of threads in the core (only settable before enabling
> > present).
> > 
> > For use with the powernv machine type (once ready for merge) I'd
> > expect "POWER8-package" type which represents a POWER8 chip / module
> > as close to the real hardware as we can get.  It would have a fixed
> > number of cores and threads within it as per the real hardware, and
> > would also include xscoms and other per-module logic.
> > 
> > From here to there:
> > 
> > A suggested order of implementation to get there without too much risk
> > of breaking things.
> > 
> >   1. Fix bugs with creation / removal of CPU objects (Bharata's cpu
> >      hotplug series already has this)
> >   2. Split creation and realization of CPU objects, so machine types
> >      must explicitly perform both steps (Bharata's series has this
> >      too)
> >   3. Add the abstract cpu-package type, and define the generic
> >      interfaces it needs (Bharata's series has something that could be
> >      changed to this fairly easily)
> >   4. For each machine type we care to convert:
> >       4.1. Add platform suitable cpu-package subtypes
> >       4.2. Convert the (latest version) machine type to instantiate packages instead of
> >            cpu threads directly
> 
> Machines could even have the freedom to instantiate CPU threads
> directly and then set up package objects for them. Reusing
> generic code is useful, but it doesn't even need to be mandatory,
> as long as the objects are available at the right place in the
> QOM hierarchy.

Ah, yes, I guess so.

> 
> >       4.3. Add any necessary backwards compat goo
> >   5. Teach libvirt how to toggle cpu-packages
> 
> This is different from the very flexible QOM object
> building/linking approach Andreas was talking about in last KVM
> Forum.

So, I know this stuff was discussed at KVM Forum, but unfortunately I
never got a clear picture of what the outcome was.

> But while I would love to have the ability to build
> arbitrary QOM hierarchies with complex links between CPUs
> sockets, cores, threads, etc, I believe we need an interface that
> is: 1) generic enough for multiple architectures and machines to
> implement them; 2) simple enough so that libvirt can use it
> easily without requiring more arch-specific code.
> 
> Also, your approach doesn't prevent the simple cpu-package
> interface from having a complex QOM hierarchy hidden behind it.

Exactly.


-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [Qemu-devel] CPU hotplug

[Eduardo Habkost]

On Mon, Feb 01, 2016 at 04:35:17PM +1100, David Gibson wrote:
> Hi,
> 
> It seems to me we're getting rather bogged down in how to proceed with
> an improved CPU hotplug (and hot unplug) interface, both generically
> and for ppc in particular.
> 
> So here's a somewhat more concrete suggestion of a way forward, to see
> if we can get some consensus.
> 
> The biggest difficulty I think we're grappling with is that device-add
> is actually *not* a great interface to cpu hotplug.  Or rather, it's
> not great as the _only_ interface: in order to represent the many
> different constraints on how cpus can be plugged on various platforms,
> it's natural to use a heirarchy of cpu core / socket / package types
> specific to the specific platform or real-world cpu package being
> modeled.  However, for the normal case of a regular homogenous (and at
> least slightly para-virtualized) server, that interface is nasty for
> management layers because they have to know the right type to
> instantiate.
> 
> To address this, I'm proposing this two layer interface:
> 
> Layer 1: Low-level, device-add based
> 
>     * a new, generic cpu-package QOM type represents a group of 1 or
>       more cpu threads which can be hotplugged as a unit
>     * cpu-package is abstract and can't be instantiated directly
>     * archs and/or individual platforms have specific subtypes of
>       cpu-package which can be instantiated
>     * for platforms attempting to be faithful representations of real
>       hardware these subtypes would match the specific characteristics
>       of the real hardware devices.  In addition to the cpu threads,
>       they may have other on chip devices as sub-objects.
>     * for platforms which are paravirtual - or which have existing
>       firmware abstractions for cpu cores/sockets/packages/whatever -
>       these could be more abstract, but would still be tied to that
>       platform's constraints
>     * Depending on the platform the cpu-package object could have
>       further internal structure (e.g. a package object representing a
>       socket contains package objects representing each core, which in
>       turn contain cpu objects for each thread)
>         * Some crazy platform that has multiple daughterboards each with
>           several multi-chip-modules each with several chips, each
> 	  with several cores each with several threads could represent
> 	  that too.

What exactly in this approach makes it depend on device-add? We
could have something very similar based on creation of QOM
objects, for example.

> 
> What would be common to all the cpu-package subtypes is:
>     * A boolean "present" attribute ("realized" might already be
>       suitable, but I'm not certain)

"realized" might be suitable, but I am not even sure we want
cpu-package to be a TYPE_DEVICE subclass. It could be a simple
QOM class or even a QOM interface (machines could choose to
implement it as TYPE_DEVICE, or not).

>     * A generic means of determining the number of cpu threads in the
>       package, and enumerating those

This could be based on QOM links.

>     * A generic means of determining if the package is hotpluggable or
>       not

Isn't this a machine attribute, instead of a package attribute?

>     * They'd get listed in a standard place in the QOM tree

If we allow CPU thread enumeration and package enumeration be
based in QOM links, we can let machines implement those
interfaces without introducing QOM hierarchy requirements.

We have one example where we would need to make this flexible
enough about QOM hierarchy, below (thread-based hotplug in x86).

> 
> This interface is suitable if you want complete control over
> constructing the system, including weird cases like heterogeneous
> machines (either totally different cpu types, or just different
> numbers of threads in different packages).
> 
> The intention is that these objects would never look at the global cpu
> type or sockets/cores/threads numbers.  The next level up would
> instead configure the packages to match those for the common case.
> 
> Layer 2: Higher-level
> 
>     * not all machine types need support this model, but I'd expect
>       all future versions of machine types designed for production use
>       to do so
>     * machine types don't construct cpu objects directly
>     * instead they create enough cpu-package objects - of a subtype
>       suitable for this machine - to provide maxcpus threads
>     * the machine type would set the "present" bit on enough of the
>       cpu packages to provide the base number of cpu threads

Sounds interesting, and very simple for management code. What I
don't see is: what exactly makes it easier to implement just
Layer 1 and not Layer 2?

Implementing Layer 1 looks more difficult to me, because it
requires supporting creation of cpu-package objects on the fly,
using device_add (or whatever mechanism we choose for cpu-package
creation). Layer 2 lets the implementation choose freely when/how
exactly the other objects will be created and how exactly they
will appear in the device tree. They just need to do the right
thing when the "present" property is flipped.

> 
> Management layers can then manage hotplug without knowing platform
> specifics by using qmp to toggle the "present" bit on packages.
> Platforms that allow thread-level pluggability can expose a package
> for every thread, those that allow core-level expose a package per
> core, those that have even less granularity expose a package at
> whatever grouping they can do hotplug on.
> 
> Examples:
> 
> For use with pc (or q35 or whatever) machine type, I'd expect a
> cpu-package subtype called, say "acpi-thread" which represents a
> single thread in the ACPI sense.  Toggling those would trigger ACPI
> hotplug events as cpu_add does now.

You have a good point here: I remember seeing suggestions of
making CPU hotplug tied to the socket/core/thread hierarchy
somehow. But this won't change the fact that x86 allows hotplug
of individual CPU threads.

In other words, if we make a /machine/socket/core/thread QOM
hierarchy, the cpu-packages for x86 won't necessarily represent
CPU sockets (but in other architectures, they might).  The
interface needs to be generic enough to not assume anything about
the CPU topology level where CPU hotplug happens.

> 
> For use with pseries, I'd expect a "papr-core" cpu-package subtype,
> which represents a single (paravirtual) core.  Toggling present on
> this would trigger the PAPR hotplug events.  A property would control
> the number of threads in the core (only settable before enabling
> present).
> 
> For use with the powernv machine type (once ready for merge) I'd
> expect "POWER8-package" type which represents a POWER8 chip / module
> as close to the real hardware as we can get.  It would have a fixed
> number of cores and threads within it as per the real hardware, and
> would also include xscoms and other per-module logic.
> 
> From here to there:
> 
> A suggested order of implementation to get there without too much risk
> of breaking things.
> 
>   1. Fix bugs with creation / removal of CPU objects (Bharata's cpu
>      hotplug series already has this)
>   2. Split creation and realization of CPU objects, so machine types
>      must explicitly perform both steps (Bharata's series has this
>      too)
>   3. Add the abstract cpu-package type, and define the generic
>      interfaces it needs (Bharata's series has something that could be
>      changed to this fairly easily)
>   4. For each machine type we care to convert:
>       4.1. Add platform suitable cpu-package subtypes
>       4.2. Convert the (latest version) machine type to instantiate packages instead of
>            cpu threads directly

Machines could even have the freedom to instantiate CPU threads
directly and then set up package objects for them. Reusing
generic code is useful, but it doesn't even need to be mandatory,
as long as the objects are available at the right place in the
QOM hierarchy.

>       4.3. Add any necessary backwards compat goo
>   5. Teach libvirt how to toggle cpu-packages

This is different from the very flexible QOM object
building/linking approach Andreas was talking about in last KVM
Forum. But while I would love to have the ability to build
arbitrary QOM hierarchies with complex links between CPUs
sockets, cores, threads, etc, I believe we need an interface that
is: 1) generic enough for multiple architectures and machines to
implement them; 2) simple enough so that libvirt can use it
easily without requiring more arch-specific code.

Also, your approach doesn't prevent the simple cpu-package
interface from having a complex QOM hierarchy hidden behind it.

-- 
Eduardo

Re: [Qemu-devel] CPU hotplug

[Christian Borntraeger]

On 02/01/2016 06:35 AM, David Gibson wrote:
> Hi,
> 
> It seems to me we're getting rather bogged down in how to proceed with
> an improved CPU hotplug (and hot unplug) interface, both generically
> and for ppc in particular.

Yes, s390 also needs this. 
Can you add Matthew in any cpu hotplug discussion?


> 
> So here's a somewhat more concrete suggestion of a way forward, to see
> if we can get some consensus.
> 
> The biggest difficulty I think we're grappling with is that device-add
> is actually *not* a great interface to cpu hotplug.  Or rather, it's
> not great as the _only_ interface: in order to represent the many
> different constraints on how cpus can be plugged on various platforms,
> it's natural to use a heirarchy of cpu core / socket / package types
> specific to the specific platform or real-world cpu package being
> modeled.  However, for the normal case of a regular homogenous (and at
> least slightly para-virtualized) server, that interface is nasty for
> management layers because they have to know the right type to
> instantiate.
> 
> To address this, I'm proposing this two layer interface:
> 
> Layer 1: Low-level, device-add based
> 
>     * a new, generic cpu-package QOM type represents a group of 1 or
>       more cpu threads which can be hotplugged as a unit
>     * cpu-package is abstract and can't be instantiated directly
>     * archs and/or individual platforms have specific subtypes of
>       cpu-package which can be instantiated
>     * for platforms attempting to be faithful representations of real
>       hardware these subtypes would match the specific characteristics
>       of the real hardware devices.  In addition to the cpu threads,
>       they may have other on chip devices as sub-objects.
>     * for platforms which are paravirtual - or which have existing
>       firmware abstractions for cpu cores/sockets/packages/whatever -
>       these could be more abstract, but would still be tied to that
>       platform's constraints
>     * Depending on the platform the cpu-package object could have
>       further internal structure (e.g. a package object representing a
>       socket contains package objects representing each core, which in
>       turn contain cpu objects for each thread)
>         * Some crazy platform that has multiple daughterboards each with
>           several multi-chip-modules each with several chips, each
> 	  with several cores each with several threads could represent
> 	  that too.
> 
> What would be common to all the cpu-package subtypes is:
>     * A boolean "present" attribute ("realized" might already be
>       suitable, but I'm not certain)
>     * A generic means of determining the number of cpu threads in the
>       package, and enumerating those
>     * A generic means of determining if the package is hotpluggable or
>       not
>     * They'd get listed in a standard place in the QOM tree
> 
> This interface is suitable if you want complete control over
> constructing the system, including weird cases like heterogeneous
> machines (either totally different cpu types, or just different
> numbers of threads in different packages).
> 
> The intention is that these objects would never look at the global cpu
> type or sockets/cores/threads numbers.  The next level up would
> instead configure the packages to match those for the common case.
> 
> Layer 2: Higher-level
> 
>     * not all machine types need support this model, but I'd expect
>       all future versions of machine types designed for production use
>       to do so
>     * machine types don't construct cpu objects directly
>     * instead they create enough cpu-package objects - of a subtype
>       suitable for this machine - to provide maxcpus threads
>     * the machine type would set the "present" bit on enough of the
>       cpu packages to provide the base number of cpu threads
> 
> Management layers can then manage hotplug without knowing platform
> specifics by using qmp to toggle the "present" bit on packages.
> Platforms that allow thread-level pluggability can expose a package
> for every thread, those that allow core-level expose a package per
> core, those that have even less granularity expose a package at
> whatever grouping they can do hotplug on.
> 
> Examples:
> 
> For use with pc (or q35 or whatever) machine type, I'd expect a
> cpu-package subtype called, say "acpi-thread" which represents a
> single thread in the ACPI sense.  Toggling those would trigger ACPI
> hotplug events as cpu_add does now.
> 
> For use with pseries, I'd expect a "papr-core" cpu-package subtype,
> which represents a single (paravirtual) core.  Toggling present on
> this would trigger the PAPR hotplug events.  A property would control
> the number of threads in the core (only settable before enabling
> present).
> 
> For use with the powernv machine type (once ready for merge) I'd
> expect "POWER8-package" type which represents a POWER8 chip / module
> as close to the real hardware as we can get.  It would have a fixed
> number of cores and threads within it as per the real hardware, and
> would also include xscoms and other per-module logic.
> 
> From here to there:
> 
> A suggested order of implementation to get there without too much risk
> of breaking things.
> 
>   1. Fix bugs with creation / removal of CPU objects (Bharata's cpu
>      hotplug series already has this)
>   2. Split creation and realization of CPU objects, so machine types
>      must explicitly perform both steps (Bharata's series has this
>      too)
>   3. Add the abstract cpu-package type, and define the generic
>      interfaces it needs (Bharata's series has something that could be
>      changed to this fairly easily)
>   4. For each machine type we care to convert:
>       4.1. Add platform suitable cpu-package subtypes
>       4.2. Convert the (latest version) machine type to instantiate packages instead of
>            cpu threads directly
>       4.3. Add any necessary backwards compat goo
>   5. Teach libvirt how to toggle cpu-packages
> 

[Qemu-devel] CPU hotplug

[David Gibson]

[-- Attachment #1: Type: text/plain, Size: 6007 bytes --]

Hi,

It seems to me we're getting rather bogged down in how to proceed with
an improved CPU hotplug (and hot unplug) interface, both generically
and for ppc in particular.

So here's a somewhat more concrete suggestion of a way forward, to see
if we can get some consensus.

The biggest difficulty I think we're grappling with is that device-add
is actually *not* a great interface to cpu hotplug.  Or rather, it's
not great as the _only_ interface: in order to represent the many
different constraints on how cpus can be plugged on various platforms,
it's natural to use a heirarchy of cpu core / socket / package types
specific to the specific platform or real-world cpu package being
modeled.  However, for the normal case of a regular homogenous (and at
least slightly para-virtualized) server, that interface is nasty for
management layers because they have to know the right type to
instantiate.

To address this, I'm proposing this two layer interface:

Layer 1: Low-level, device-add based

    * a new, generic cpu-package QOM type represents a group of 1 or
      more cpu threads which can be hotplugged as a unit
    * cpu-package is abstract and can't be instantiated directly
    * archs and/or individual platforms have specific subtypes of
      cpu-package which can be instantiated
    * for platforms attempting to be faithful representations of real
      hardware these subtypes would match the specific characteristics
      of the real hardware devices.  In addition to the cpu threads,
      they may have other on chip devices as sub-objects.
    * for platforms which are paravirtual - or which have existing
      firmware abstractions for cpu cores/sockets/packages/whatever -
      these could be more abstract, but would still be tied to that
      platform's constraints
    * Depending on the platform the cpu-package object could have
      further internal structure (e.g. a package object representing a
      socket contains package objects representing each core, which in
      turn contain cpu objects for each thread)
        * Some crazy platform that has multiple daughterboards each with
          several multi-chip-modules each with several chips, each
	  with several cores each with several threads could represent
	  that too.

What would be common to all the cpu-package subtypes is:
    * A boolean "present" attribute ("realized" might already be
      suitable, but I'm not certain)
    * A generic means of determining the number of cpu threads in the
      package, and enumerating those
    * A generic means of determining if the package is hotpluggable or
      not
    * They'd get listed in a standard place in the QOM tree

This interface is suitable if you want complete control over
constructing the system, including weird cases like heterogeneous
machines (either totally different cpu types, or just different
numbers of threads in different packages).

The intention is that these objects would never look at the global cpu
type or sockets/cores/threads numbers.  The next level up would
instead configure the packages to match those for the common case.

Layer 2: Higher-level

    * not all machine types need support this model, but I'd expect
      all future versions of machine types designed for production use
      to do so
    * machine types don't construct cpu objects directly
    * instead they create enough cpu-package objects - of a subtype
      suitable for this machine - to provide maxcpus threads
    * the machine type would set the "present" bit on enough of the
      cpu packages to provide the base number of cpu threads

Management layers can then manage hotplug without knowing platform
specifics by using qmp to toggle the "present" bit on packages.
Platforms that allow thread-level pluggability can expose a package
for every thread, those that allow core-level expose a package per
core, those that have even less granularity expose a package at
whatever grouping they can do hotplug on.

Examples:

For use with pc (or q35 or whatever) machine type, I'd expect a
cpu-package subtype called, say "acpi-thread" which represents a
single thread in the ACPI sense.  Toggling those would trigger ACPI
hotplug events as cpu_add does now.

For use with pseries, I'd expect a "papr-core" cpu-package subtype,
which represents a single (paravirtual) core.  Toggling present on
this would trigger the PAPR hotplug events.  A property would control
the number of threads in the core (only settable before enabling
present).

For use with the powernv machine type (once ready for merge) I'd
expect "POWER8-package" type which represents a POWER8 chip / module
as close to the real hardware as we can get.  It would have a fixed
number of cores and threads within it as per the real hardware, and
would also include xscoms and other per-module logic.

From here to there:

A suggested order of implementation to get there without too much risk
of breaking things.

  1. Fix bugs with creation / removal of CPU objects (Bharata's cpu
     hotplug series already has this)
  2. Split creation and realization of CPU objects, so machine types
     must explicitly perform both steps (Bharata's series has this
     too)
  3. Add the abstract cpu-package type, and define the generic
     interfaces it needs (Bharata's series has something that could be
     changed to this fairly easily)
  4. For each machine type we care to convert:
      4.1. Add platform suitable cpu-package subtypes
      4.2. Convert the (latest version) machine type to instantiate packages instead of
           cpu threads directly
      4.3. Add any necessary backwards compat goo
  5. Teach libvirt how to toggle cpu-packages

-- 
David Gibson			| I'll have my music baroque, and my code
david AT gibson.dropbear.id.au	| minimalist, thank you.  NOT _the_ _other_
				| _way_ _around_!
http://www.ozlabs.org/~dgibson

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]