[Qemu-devel] CPU model versioning separate from machine type versioning ?

[Qemu-devel] CPU model versioning separate from machine type versioning ?

[Daniel P. Berrangé]

This post is to raise question about helping use of named CPU models with
KVM ie any case not using -cpu host.

In the old days (ie before 2018), the world was innocent and we had a nice
set of named CPU models that corresponded to different Intel/AMD physical
CPU families/generations (lets temporarily ignore the -noTSX fiasco).

An application could query libvirt to determine what the host CPU model
was/is and use that model name in the guest XML and be fairly happy. If
they wanted to, they could explicitly include the extra features listed
by capabilities XML, or just rely on the host-model.

Then Spectre happened, and QEMU took the decision to almost double the
number of x86 models, adding in -IBRS / -IBPB variants for most CPU model,
so that applications could get the spec_ctrl / ibpb flags set without
having to manually list them.

In retrospect this was somewhat pointless, at least at the QEMU level,
because there is little difference in complexity between the two approaches:

   -cpu Westmere,+spec-ctrl
   -cpu Westmere-IBRS

At a higher level the extra named CPU models were slightly useful in so much
as many application developers had taken a lazy approach and not provided
users any way to explicitly turn on extra flags. This affected oVirt,
OpenStack and virt-manager, and probably more. Though OpenStack since added
ability to turn on arbitrary flags in response to the Spectre flaw, others
have not.

Then a recently along came the Speculative Store Bypass hardware vulnerability
requiring addition of yet another CPU flag to guest configs. This required use
of 'ssbd' on Intel and 'virt-ssbd' on AMD. While QEMU could have now added yet
more CPU models, eg Westmere-SSBD, this does not feel like a winning strategy
long term. Looking at the models how would a user have any clue whether the
-IBRS or -SSBD or -NEXT-FLAW or -YET-ANOTHER-FLAW suffix is "better" ? So QEMU
and libvirt took the joint decision to stop adding new named CPU models when
CPU vulnerabilities are discovered from this point forwards. Applications /
users would be expected to turn on CPU features explicitly as needed and are
considered broken if they don't provide this functionality.

As briefly mentioned above though, even before Spectre we had the pain of
dealing with the -noTSX CPU models working around brokenness in the Intel TSX
impl where they had to delete a CPU feature during microcode updates. This was
rather painful to roll out at the time.

An alternative to adding CPU models is to change meaning of existing CPU
models. QEMU has a way todo this by tieing the change to machine types, and
it has in fact been used to correct mistakes in the specification of CPU
models in the past, when those mistakes have not had dependancies on microcode
changes. This is not a particularly attractice way to deal with the errata.
Short life distros tend to stick with upstream QEMU machine types and won't
want to diverge by adding their own machine types. This gates them on having
upstream define the extra machine types which is tricky under embargo. Long
life distros do typically take on the burden of defining custom machine types,
but usually only add them when doing major updates.

The pain point with machine types is that the testing matrix grows at O(n^2)
Using machine types for CPU security errata would significant increase the
number of machine types and thus the testing matrix. eg if a security fix
is needed in rhel-7.3, 7.4, 7.5 we can't just add a pc-rhel-7.5.1 machine
with the fix, as it would not be possible to implement that in 7.3. So we
would need would need pc-rhel-7.3.1,  pc-rhel-7.4.1,  pc-rhel-7.5.1, machine
types, with 7.5 gaining all three. Finally CPU model changes have host
hardware dependancies and machine types need to be independant of the host,
since they are decided statically are build time. The only nice thing about
machine type is that it is reasonably obvious what the "best" machine type
is as they include a version number in the name, and users automatically get
the best if they use an unversioned name.


What if we can borrow the concept of versioning from machine types and apply
it to CPU models directly. For example, considering the history of "Haswell"
in QEMU, if we had versioned things, we would by now have:

     Haswell-1.3.0 - first version (37507094f350b75c62dc059f998e7185de3ab60a)
     Haswell-2.2.0 - added 'rdrand' (78a611f1936b3eac8ed78a2be2146a742a85212c_
     Haswell-2.3.0 - removed 'hle' & 'rtm' (a356850b80b3d13b2ef737dad2acb05e6da03753)
     Haswell-2.5.0 - added 'abm' (becb66673ec30cb604926d247ab9449a60ad8b11
     Haswell-2.12.0 - added 'spec-ctrl' (ac96c41354b7e4c70b756342d9b686e31ab87458)
     Haswell-3.0.0  - added 'ssbd' (never done)

If we followed the machine type approach, then a bare "Haswell" would
statically resolve at build time to the most recent Haswell-X.X.X version
associated with the QEMU release. This is unhelpful as we have a direct
dependancy on the host hardware features. Better would be for a bare
"Haswell" to be dynamically resolved at runtime, picking the most recent
version that is capable of launching given the current hardware, KVM/TCG impl
and QEMU version.

  ie -cpu  Haswell

should use Haswell-2.5.0  if on silicon with the TSX errata applied,
but use Haswell-2.12.0 if the Spectre errata is applied in microcode,
and use Haswell-3.0.0 once Intel finally releases SSBD microcode errata.

Versioning of CPU models as opposed to using arbitrary string suffixes
(-noTSX, -IBRS) has a number of usability improvements that we would
gain with versioned machine types, while avoiding exploding the machine
type matrix. With versioned CPU models we can

 - Automatically tailor the best model based on hardware support

 - Users always get the best model if they use the bare CPU name

 - It is obvious to users which is the "best" / "newest" CPU model

 - Avoid combinatorial expansion of machines since same CPU model
   version can be added to all releases without adding machine types.

 - Users can still force a specific downgraded model by using the
   fully versioned name.

Such versioning of CPU models would largely "just work" with existing
libvirt versions, but to libvirt would really want to expand the bare
CPU name to a versioned CPU name when recording new guest XML, so the
ABI is preserved long term.

An application like virt-manager which wants a simple UI can forever be
happy simply giving users a list of bare CPU model names, and allowing
libvirt / QEMU to automatically expand to the best versioned model for
their host.

An application like oVirt/OpenStack which wants direct control can allow
the admin to choice if a bare name, or explicitly picking a versioned name
if they need to cope with possibility of outdated hosts.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] CPU model versioning separate from machine type versioning ?

[Dr. David Alan Gilbert]

* Daniel P. Berrangé (berrange@redhat.com) wrote:
> This post is to raise question about helping use of named CPU models with
> KVM ie any case not using -cpu host.
> 
> In the old days (ie before 2018), the world was innocent and we had a nice
> set of named CPU models that corresponded to different Intel/AMD physical
> CPU families/generations (lets temporarily ignore the -noTSX fiasco).
> 
> An application could query libvirt to determine what the host CPU model
> was/is and use that model name in the guest XML and be fairly happy. If
> they wanted to, they could explicitly include the extra features listed
> by capabilities XML, or just rely on the host-model.
> 
> Then Spectre happened, and QEMU took the decision to almost double the
> number of x86 models, adding in -IBRS / -IBPB variants for most CPU model,
> so that applications could get the spec_ctrl / ibpb flags set without
> having to manually list them.
> 
> In retrospect this was somewhat pointless, at least at the QEMU level,
> because there is little difference in complexity between the two approaches:
> 
>    -cpu Westmere,+spec-ctrl
>    -cpu Westmere-IBRS
> 
> At a higher level the extra named CPU models were slightly useful in so much
> as many application developers had taken a lazy approach and not provided
> users any way to explicitly turn on extra flags. This affected oVirt,
> OpenStack and virt-manager, and probably more. Though OpenStack since added
> ability to turn on arbitrary flags in response to the Spectre flaw, others
> have not.
> 
> Then a recently along came the Speculative Store Bypass hardware vulnerability
> requiring addition of yet another CPU flag to guest configs. This required use
> of 'ssbd' on Intel and 'virt-ssbd' on AMD. While QEMU could have now added yet
> more CPU models, eg Westmere-SSBD, this does not feel like a winning strategy
> long term. Looking at the models how would a user have any clue whether the
> -IBRS or -SSBD or -NEXT-FLAW or -YET-ANOTHER-FLAW suffix is "better" ? So QEMU
> and libvirt took the joint decision to stop adding new named CPU models when
> CPU vulnerabilities are discovered from this point forwards. Applications /
> users would be expected to turn on CPU features explicitly as needed and are
> considered broken if they don't provide this functionality.
> 
> As briefly mentioned above though, even before Spectre we had the pain of
> dealing with the -noTSX CPU models working around brokenness in the Intel TSX
> impl where they had to delete a CPU feature during microcode updates. This was
> rather painful to roll out at the time.
> 
> An alternative to adding CPU models is to change meaning of existing CPU
> models. QEMU has a way todo this by tieing the change to machine types, and
> it has in fact been used to correct mistakes in the specification of CPU
> models in the past, when those mistakes have not had dependancies on microcode
> changes. This is not a particularly attractice way to deal with the errata.
> Short life distros tend to stick with upstream QEMU machine types and won't
> want to diverge by adding their own machine types. This gates them on having
> upstream define the extra machine types which is tricky under embargo. Long
> life distros do typically take on the burden of defining custom machine types,
> but usually only add them when doing major updates.
> 
> The pain point with machine types is that the testing matrix grows at O(n^2)
> Using machine types for CPU security errata would significant increase the
> number of machine types and thus the testing matrix. eg if a security fix
> is needed in rhel-7.3, 7.4, 7.5 we can't just add a pc-rhel-7.5.1 machine
> with the fix, as it would not be possible to implement that in 7.3. So we
> would need would need pc-rhel-7.3.1,  pc-rhel-7.4.1,  pc-rhel-7.5.1, machine
> types, with 7.5 gaining all three. Finally CPU model changes have host
> hardware dependancies and machine types need to be independant of the host,
> since they are decided statically are build time. The only nice thing about
> machine type is that it is reasonably obvious what the "best" machine type
> is as they include a version number in the name, and users automatically get
> the best if they use an unversioned name.
> 
> 
> What if we can borrow the concept of versioning from machine types and apply
> it to CPU models directly. For example, considering the history of "Haswell"
> in QEMU, if we had versioned things, we would by now have:
> 
>      Haswell-1.3.0 - first version (37507094f350b75c62dc059f998e7185de3ab60a)
>      Haswell-2.2.0 - added 'rdrand' (78a611f1936b3eac8ed78a2be2146a742a85212c_
>      Haswell-2.3.0 - removed 'hle' & 'rtm' (a356850b80b3d13b2ef737dad2acb05e6da03753)
>      Haswell-2.5.0 - added 'abm' (becb66673ec30cb604926d247ab9449a60ad8b11
>      Haswell-2.12.0 - added 'spec-ctrl' (ac96c41354b7e4c70b756342d9b686e31ab87458)
>      Haswell-3.0.0  - added 'ssbd' (never done)

OK.
Note that this isn't that different to what happens on some real
hardware where you have different 'steppings'

> If we followed the machine type approach, then a bare "Haswell" would
> statically resolve at build time to the most recent Haswell-X.X.X version
> associated with the QEMU release. This is unhelpful as we have a direct
> dependancy on the host hardware features. Better would be for a bare
> "Haswell" to be dynamically resolved at runtime, picking the most recent
> version that is capable of launching given the current hardware, KVM/TCG impl
> and QEMU version.
> 
>   ie -cpu  Haswell
> 
> should use Haswell-2.5.0  if on silicon with the TSX errata applied,
> but use Haswell-2.12.0 if the Spectre errata is applied in microcode,
> and use Haswell-3.0.0 once Intel finally releases SSBD microcode errata.
> 
> Versioning of CPU models as opposed to using arbitrary string suffixes
> (-noTSX, -IBRS) has a number of usability improvements that we would
> gain with versioned machine types, while avoiding exploding the machine
> type matrix. With versioned CPU models we can
> 
>  - Automatically tailor the best model based on hardware support
> 
>  - Users always get the best model if they use the bare CPU name
> 
>  - It is obvious to users which is the "best" / "newest" CPU model
> 
>  - Avoid combinatorial expansion of machines since same CPU model
>    version can be added to all releases without adding machine types.
> 
>  - Users can still force a specific downgraded model by using the
>    fully versioned name.
> 
> Such versioning of CPU models would largely "just work" with existing
> libvirt versions, but to libvirt would really want to expand the bare
> CPU name to a versioned CPU name when recording new guest XML, so the
> ABI is preserved long term.
> 
> An application like virt-manager which wants a simple UI can forever be
> happy simply giving users a list of bare CPU model names, and allowing
> libvirt / QEMU to automatically expand to the best versioned model for
> their host.
> 
> An application like oVirt/OpenStack which wants direct control can allow
> the admin to choice if a bare name, or explicitly picking a versioned name
> if they need to cope with possibility of outdated hosts.

I fear people are going to find this out the hard way, when they add
a new system into their cluster, a little bit later it gets a VM started
on it, and then they try and migrate it to one of the older machines.

Now if there was something that could take the CPU defintions from all
the machines in the cluster and tell it which to use/which problems
they had then that might make sense.  It would be best for each
higher level not to reinvent that.

Would you restrict the combinations to cut down the test matrix - e.g.
not allow Haswell-3.0.0 on anything prior to a 2.12 machine type?

Dave

> Regards,
> Daniel
> -- 
> |: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
> |: https://libvirt.org         -o-            https://fstop138.berrange.com :|
> |: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] CPU model versioning separate from machine type versioning ?

[Eduardo Habkost]

On Thu, Jun 28, 2018 at 07:59:38PM +0100, Dr. David Alan Gilbert wrote:
[...]
> > An application like virt-manager which wants a simple UI can forever be
> > happy simply giving users a list of bare CPU model names, and allowing
> > libvirt / QEMU to automatically expand to the best versioned model for
> > their host.
> > 
> > An application like oVirt/OpenStack which wants direct control can allow
> > the admin to choice if a bare name, or explicitly picking a versioned name
> > if they need to cope with possibility of outdated hosts.
> 
> I fear people are going to find this out the hard way, when they add
> a new system into their cluster, a little bit later it gets a VM started
> on it, and then they try and migrate it to one of the older machines.
> 
> Now if there was something that could take the CPU defintions from all
> the machines in the cluster and tell it which to use/which problems
> they had then that might make sense.  It would be best for each
> higher level not to reinvent that.

I think QEMU already provides enough info to allow that to be
implemented.  I'm not sure sure if the libvirt API already
provides all the info needed for this (I think it does).


> 
> Would you restrict the combinations to cut down the test matrix - e.g.
> not allow Haswell-3.0.0 on anything prior to a 2.12 machine type?

Not sure if it would be worth the extra complexity: we would need
an interface to tell libvirt which CPU models are usable on which
machine-types.

But a generic mechanism to tell libvirt which devices are allowed
on each machine-type would very interesting to have, anyway.

-- 
Eduardo

Re: [Qemu-devel] CPU model versioning separate from machine type versioning ?

[Eduardo Habkost]

On Thu, Jun 28, 2018 at 04:45:02PM +0100, Daniel P. Berrangé wrote:
[...]
> What if we can borrow the concept of versioning from machine types and apply
> it to CPU models directly. For example, considering the history of "Haswell"
> in QEMU, if we had versioned things, we would by now have:
> 
>      Haswell-1.3.0 - first version (37507094f350b75c62dc059f998e7185de3ab60a)
>      Haswell-2.2.0 - added 'rdrand' (78a611f1936b3eac8ed78a2be2146a742a85212c_
>      Haswell-2.3.0 - removed 'hle' & 'rtm' (a356850b80b3d13b2ef737dad2acb05e6da03753)
>      Haswell-2.5.0 - added 'abm' (becb66673ec30cb604926d247ab9449a60ad8b11
>      Haswell-2.12.0 - added 'spec-ctrl' (ac96c41354b7e4c70b756342d9b686e31ab87458)
>      Haswell-3.0.0  - added 'ssbd' (never done)
> 
> If we followed the machine type approach, then a bare "Haswell" would
> statically resolve at build time to the most recent Haswell-X.X.X version
> associated with the QEMU release. This is unhelpful as we have a direct
> dependancy on the host hardware features. Better would be for a bare
> "Haswell" to be dynamically resolved at runtime, picking the most recent
> version that is capable of launching given the current hardware, KVM/TCG impl
> and QEMU version.
> 
>   ie -cpu  Haswell
> 
> should use Haswell-2.5.0  if on silicon with the TSX errata applied,
> but use Haswell-2.12.0 if the Spectre errata is applied in microcode,
> and use Haswell-3.0.0 once Intel finally releases SSBD microcode errata.

Doing this unconditionally would make
"-machine pc-q35-3.1 -cpu Haswell" unsafe for live migration, and
break existing usage.  But this behavior could be enabled
explicitly somehow.

> 
> Versioning of CPU models as opposed to using arbitrary string suffixes
> (-noTSX, -IBRS) has a number of usability improvements that we would
> gain with versioned machine types, while avoiding exploding the machine
> type matrix. With versioned CPU models we can
> 
>  - Automatically tailor the best model based on hardware support
> 
>  - Users always get the best model if they use the bare CPU name
> 
>  - It is obvious to users which is the "best" / "newest" CPU model
> 
>  - Avoid combinatorial expansion of machines since same CPU model
>    version can be added to all releases without adding machine types.
> 
>  - Users can still force a specific downgraded model by using the
>    fully versioned name.
> 
> Such versioning of CPU models would largely "just work" with existing
> libvirt versions, but to libvirt would really want to expand the bare
> CPU name to a versioned CPU name when recording new guest XML, so the
> ABI is preserved long term.
> 
> An application like virt-manager which wants a simple UI can forever be
> happy simply giving users a list of bare CPU model names, and allowing
> libvirt / QEMU to automatically expand to the best versioned model for
> their host.
> 
> An application like oVirt/OpenStack which wants direct control can allow
> the admin to choice if a bare name, or explicitly picking a versioned name
> if they need to cope with possibility of outdated hosts.
> 

The proposal makes sense, and I think most of it can be already
implemented on top of existing query-cpu-model-* commands.
query-cpu-model-expansion type=static can expand to a versioned
CPU model.

We will probably need to make query-cpu-model-expansion accept a
machine-type name as input, and/or add a new flag meaning "please
give me the best CPU version you have, not the one defined by the
current machine-type".

I'm not sure what would be the best way to encode two types of
information, though:

* Fallback/alternatives info, e.g.: "It makes sense to use
  Haswell-{3.0,2.12,2.5,...} if Haswell-3.1 is not runnable and the
  user asked for Haswell".

* Ordering/preference info, e.g.: "Haswell-3.1 is better than
  Haswell-3.0, prefer the latter"

-- 
Eduardo

Re: [Qemu-devel] [libvirt] CPU model versioning separate from machine type versioning ?

[Jiri Denemark]

On Thu, Jun 28, 2018 at 16:23:53 -0300, Eduardo Habkost wrote:
> On Thu, Jun 28, 2018 at 07:59:38PM +0100, Dr. David Alan Gilbert wrote:
> [...]
> > > An application like virt-manager which wants a simple UI can forever be
> > > happy simply giving users a list of bare CPU model names, and allowing
> > > libvirt / QEMU to automatically expand to the best versioned model for
> > > their host.
> > > 
> > > An application like oVirt/OpenStack which wants direct control can allow
> > > the admin to choice if a bare name, or explicitly picking a versioned name
> > > if they need to cope with possibility of outdated hosts.
> > 
> > I fear people are going to find this out the hard way, when they add
> > a new system into their cluster, a little bit later it gets a VM started
> > on it, and then they try and migrate it to one of the older machines.
> > 
> > Now if there was something that could take the CPU defintions from all
> > the machines in the cluster and tell it which to use/which problems
> > they had then that might make sense.  It would be best for each
> > higher level not to reinvent that.
> 
> I think QEMU already provides enough info to allow that to be
> implemented.  I'm not sure sure if the libvirt API already
> provides all the info needed for this (I think it does).

Right, libvirt provides virConnectBaselineHypervisorCPU API which
accepts a list of host CPU models from several hosts and returns the
best CPU definition runnable on these hosts. However, OpenStack
clusters are too dynamic for this to be practical so the admin would
need to take care of this by setting an appropriate model statically.

> > Would you restrict the combinations to cut down the test matrix - e.g.
> > not allow Haswell-3.0.0 on anything prior to a 2.12 machine type?
> 
> Not sure if it would be worth the extra complexity: we would need
> an interface to tell libvirt which CPU models are usable on which
> machine-types.

In case we do this, libvirt should already by ready for it on the API
level for both reporting capabilities and CPU comparison/baseline. All
these APIs already accept machine type as an optional parameter so that
different results can be provided depending on machine type.

Jirka

Re: [Qemu-devel] CPU model versioning separate from machine type versioning ?

[Dr. David Alan Gilbert]

* Eduardo Habkost (ehabkost@redhat.com) wrote:
> On Thu, Jun 28, 2018 at 04:45:02PM +0100, Daniel P. Berrangé wrote:
> [...]
> > What if we can borrow the concept of versioning from machine types and apply
> > it to CPU models directly. For example, considering the history of "Haswell"
> > in QEMU, if we had versioned things, we would by now have:
> > 
> >      Haswell-1.3.0 - first version (37507094f350b75c62dc059f998e7185de3ab60a)
> >      Haswell-2.2.0 - added 'rdrand' (78a611f1936b3eac8ed78a2be2146a742a85212c_
> >      Haswell-2.3.0 - removed 'hle' & 'rtm' (a356850b80b3d13b2ef737dad2acb05e6da03753)
> >      Haswell-2.5.0 - added 'abm' (becb66673ec30cb604926d247ab9449a60ad8b11
> >      Haswell-2.12.0 - added 'spec-ctrl' (ac96c41354b7e4c70b756342d9b686e31ab87458)
> >      Haswell-3.0.0  - added 'ssbd' (never done)
> > 
> > If we followed the machine type approach, then a bare "Haswell" would
> > statically resolve at build time to the most recent Haswell-X.X.X version
> > associated with the QEMU release. This is unhelpful as we have a direct
> > dependancy on the host hardware features. Better would be for a bare
> > "Haswell" to be dynamically resolved at runtime, picking the most recent
> > version that is capable of launching given the current hardware, KVM/TCG impl
> > and QEMU version.
> > 
> >   ie -cpu  Haswell
> > 
> > should use Haswell-2.5.0  if on silicon with the TSX errata applied,
> > but use Haswell-2.12.0 if the Spectre errata is applied in microcode,
> > and use Haswell-3.0.0 once Intel finally releases SSBD microcode errata.
> 
> Doing this unconditionally would make
> "-machine pc-q35-3.1 -cpu Haswell" unsafe for live migration, and
> break existing usage.  But this behavior could be enabled
> explicitly somehow.
> 
> > 
> > Versioning of CPU models as opposed to using arbitrary string suffixes
> > (-noTSX, -IBRS) has a number of usability improvements that we would
> > gain with versioned machine types, while avoiding exploding the machine
> > type matrix. With versioned CPU models we can
> > 
> >  - Automatically tailor the best model based on hardware support
> > 
> >  - Users always get the best model if they use the bare CPU name
> > 
> >  - It is obvious to users which is the "best" / "newest" CPU model
> > 
> >  - Avoid combinatorial expansion of machines since same CPU model
> >    version can be added to all releases without adding machine types.
> > 
> >  - Users can still force a specific downgraded model by using the
> >    fully versioned name.
> > 
> > Such versioning of CPU models would largely "just work" with existing
> > libvirt versions, but to libvirt would really want to expand the bare
> > CPU name to a versioned CPU name when recording new guest XML, so the
> > ABI is preserved long term.
> > 
> > An application like virt-manager which wants a simple UI can forever be
> > happy simply giving users a list of bare CPU model names, and allowing
> > libvirt / QEMU to automatically expand to the best versioned model for
> > their host.
> > 
> > An application like oVirt/OpenStack which wants direct control can allow
> > the admin to choice if a bare name, or explicitly picking a versioned name
> > if they need to cope with possibility of outdated hosts.
> > 
> 
> The proposal makes sense, and I think most of it can be already
> implemented on top of existing query-cpu-model-* commands.
> query-cpu-model-expansion type=static can expand to a versioned
> CPU model.
> 
> We will probably need to make query-cpu-model-expansion accept a
> machine-type name as input, and/or add a new flag meaning "please
> give me the best CPU version you have, not the one defined by the
> current machine-type".
> 
> I'm not sure what would be the best way to encode two types of
> information, though:
> 

Both of those are solved with the numbering scheme

> * Fallback/alternatives info, e.g.: "It makes sense to use
>   Haswell-{3.0,2.12,2.5,...} if Haswell-3.1 is not runnable and the
>   user asked for Haswell".

Use the highest that works.

> * Ordering/preference info, e.g.: "Haswell-3.1 is better than
>   Haswell-3.0, prefer the latter"

Higher is better.

The only thing that worries me about a numbering scheme is that
it's now more difficult for a user to know whether they've got
the type with a fix for a particular vulnerability.
We're going to have to say something like:
  'For the new XYZ vulnerability make sure you're using
  Haswell-3.2 or later, SkyLake-2.6 or later, Westmere-4.8 or later
  .....'

which all gets a bit confusing.

Dave


> -- 
> Eduardo
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] CPU model versioning separate from machine type versioning ?

[Daniel P. Berrangé]

On Thu, Jun 28, 2018 at 07:59:38PM +0100, Dr. David Alan Gilbert wrote:
> * Daniel P. Berrangé (berrange@redhat.com) wrote:
> > This post is to raise question about helping use of named CPU models with
> > KVM ie any case not using -cpu host.
> > 
> > In the old days (ie before 2018), the world was innocent and we had a nice
> > set of named CPU models that corresponded to different Intel/AMD physical
> > CPU families/generations (lets temporarily ignore the -noTSX fiasco).
> > 
> > An application could query libvirt to determine what the host CPU model
> > was/is and use that model name in the guest XML and be fairly happy. If
> > they wanted to, they could explicitly include the extra features listed
> > by capabilities XML, or just rely on the host-model.
> > 
> > Then Spectre happened, and QEMU took the decision to almost double the
> > number of x86 models, adding in -IBRS / -IBPB variants for most CPU model,
> > so that applications could get the spec_ctrl / ibpb flags set without
> > having to manually list them.
> > 
> > In retrospect this was somewhat pointless, at least at the QEMU level,
> > because there is little difference in complexity between the two approaches:
> > 
> >    -cpu Westmere,+spec-ctrl
> >    -cpu Westmere-IBRS
> > 
> > At a higher level the extra named CPU models were slightly useful in so much
> > as many application developers had taken a lazy approach and not provided
> > users any way to explicitly turn on extra flags. This affected oVirt,
> > OpenStack and virt-manager, and probably more. Though OpenStack since added
> > ability to turn on arbitrary flags in response to the Spectre flaw, others
> > have not.
> > 
> > Then a recently along came the Speculative Store Bypass hardware vulnerability
> > requiring addition of yet another CPU flag to guest configs. This required use
> > of 'ssbd' on Intel and 'virt-ssbd' on AMD. While QEMU could have now added yet
> > more CPU models, eg Westmere-SSBD, this does not feel like a winning strategy
> > long term. Looking at the models how would a user have any clue whether the
> > -IBRS or -SSBD or -NEXT-FLAW or -YET-ANOTHER-FLAW suffix is "better" ? So QEMU
> > and libvirt took the joint decision to stop adding new named CPU models when
> > CPU vulnerabilities are discovered from this point forwards. Applications /
> > users would be expected to turn on CPU features explicitly as needed and are
> > considered broken if they don't provide this functionality.
> > 
> > As briefly mentioned above though, even before Spectre we had the pain of
> > dealing with the -noTSX CPU models working around brokenness in the Intel TSX
> > impl where they had to delete a CPU feature during microcode updates. This was
> > rather painful to roll out at the time.
> > 
> > An alternative to adding CPU models is to change meaning of existing CPU
> > models. QEMU has a way todo this by tieing the change to machine types, and
> > it has in fact been used to correct mistakes in the specification of CPU
> > models in the past, when those mistakes have not had dependancies on microcode
> > changes. This is not a particularly attractice way to deal with the errata.
> > Short life distros tend to stick with upstream QEMU machine types and won't
> > want to diverge by adding their own machine types. This gates them on having
> > upstream define the extra machine types which is tricky under embargo. Long
> > life distros do typically take on the burden of defining custom machine types,
> > but usually only add them when doing major updates.
> > 
> > The pain point with machine types is that the testing matrix grows at O(n^2)
> > Using machine types for CPU security errata would significant increase the
> > number of machine types and thus the testing matrix. eg if a security fix
> > is needed in rhel-7.3, 7.4, 7.5 we can't just add a pc-rhel-7.5.1 machine
> > with the fix, as it would not be possible to implement that in 7.3. So we
> > would need would need pc-rhel-7.3.1,  pc-rhel-7.4.1,  pc-rhel-7.5.1, machine
> > types, with 7.5 gaining all three. Finally CPU model changes have host
> > hardware dependancies and machine types need to be independant of the host,
> > since they are decided statically are build time. The only nice thing about
> > machine type is that it is reasonably obvious what the "best" machine type
> > is as they include a version number in the name, and users automatically get
> > the best if they use an unversioned name.
> > 
> > 
> > What if we can borrow the concept of versioning from machine types and apply
> > it to CPU models directly. For example, considering the history of "Haswell"
> > in QEMU, if we had versioned things, we would by now have:
> > 
> >      Haswell-1.3.0 - first version (37507094f350b75c62dc059f998e7185de3ab60a)
> >      Haswell-2.2.0 - added 'rdrand' (78a611f1936b3eac8ed78a2be2146a742a85212c_
> >      Haswell-2.3.0 - removed 'hle' & 'rtm' (a356850b80b3d13b2ef737dad2acb05e6da03753)
> >      Haswell-2.5.0 - added 'abm' (becb66673ec30cb604926d247ab9449a60ad8b11
> >      Haswell-2.12.0 - added 'spec-ctrl' (ac96c41354b7e4c70b756342d9b686e31ab87458)
> >      Haswell-3.0.0  - added 'ssbd' (never done)
> 
> OK.
> Note that this isn't that different to what happens on some real
> hardware where you have different 'steppings'
> 
> > If we followed the machine type approach, then a bare "Haswell" would
> > statically resolve at build time to the most recent Haswell-X.X.X version
> > associated with the QEMU release. This is unhelpful as we have a direct
> > dependancy on the host hardware features. Better would be for a bare
> > "Haswell" to be dynamically resolved at runtime, picking the most recent
> > version that is capable of launching given the current hardware, KVM/TCG impl
> > and QEMU version.
> > 
> >   ie -cpu  Haswell
> > 
> > should use Haswell-2.5.0  if on silicon with the TSX errata applied,
> > but use Haswell-2.12.0 if the Spectre errata is applied in microcode,
> > and use Haswell-3.0.0 once Intel finally releases SSBD microcode errata.
> > 
> > Versioning of CPU models as opposed to using arbitrary string suffixes
> > (-noTSX, -IBRS) has a number of usability improvements that we would
> > gain with versioned machine types, while avoiding exploding the machine
> > type matrix. With versioned CPU models we can
> > 
> >  - Automatically tailor the best model based on hardware support
> > 
> >  - Users always get the best model if they use the bare CPU name
> > 
> >  - It is obvious to users which is the "best" / "newest" CPU model
> > 
> >  - Avoid combinatorial expansion of machines since same CPU model
> >    version can be added to all releases without adding machine types.
> > 
> >  - Users can still force a specific downgraded model by using the
> >    fully versioned name.
> > 
> > Such versioning of CPU models would largely "just work" with existing
> > libvirt versions, but to libvirt would really want to expand the bare
> > CPU name to a versioned CPU name when recording new guest XML, so the
> > ABI is preserved long term.
> > 
> > An application like virt-manager which wants a simple UI can forever be
> > happy simply giving users a list of bare CPU model names, and allowing
> > libvirt / QEMU to automatically expand to the best versioned model for
> > their host.
> > 
> > An application like oVirt/OpenStack which wants direct control can allow
> > the admin to choice if a bare name, or explicitly picking a versioned name
> > if they need to cope with possibility of outdated hosts.
> 
> I fear people are going to find this out the hard way, when they add
> a new system into their cluster, a little bit later it gets a VM started
> on it, and then they try and migrate it to one of the older machines.
> 
> Now if there was something that could take the CPU defintions from all
> the machines in the cluster and tell it which to use/which problems
> they had then that might make sense.  It would be best for each
> higher level not to reinvent that.

Libvirt / QEMU have ability to let apps do that. Nova could choose to
do that, but it does not wish to take advantage of that right now. The
nova admin already has make an explicit config to set the maximum
Nova database object schema version, and maximum machine type version
across the cluster. The max CPU model version fits in with what they
are already doing quite well. Essentially the admin knows what versions
of QEMU they have across their cluster - generally it is jus N, and N+1
during course of a version upgrade. So it is quite easy for the admin
to know what the max version should be. In fact the admin doens't
really need to be involved in it at all - the cluster provisioning
tool already knows the old + new host versions, so can set it automatically
at the right times during upgrade process.

> Would you restrict the combinations to cut down the test matrix - e.g.
> not allow Haswell-3.0.0 on anything prior to a 2.12 machine type?

The key point of having CPU versions separate from machine type
versions, is to allow arbitrary mixing of versions. For example when
we issued Spectre updates for RHEL, we had to add new CPU model to
something like 12 different QEMU versions, and we needed the same
CPU model changes to work with all the different machine types we
had across those versions. So restricting CPU versions to only
certain machine types would defeat the puporse of having CPU versions.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] CPU model versioning separate from machine type versioning ?

[Daniel P. Berrangé]

On Thu, Jun 28, 2018 at 04:52:27PM -0300, Eduardo Habkost wrote:
> On Thu, Jun 28, 2018 at 04:45:02PM +0100, Daniel P. Berrangé wrote:
> [...]
> > What if we can borrow the concept of versioning from machine types and apply
> > it to CPU models directly. For example, considering the history of "Haswell"
> > in QEMU, if we had versioned things, we would by now have:
> > 
> >      Haswell-1.3.0 - first version (37507094f350b75c62dc059f998e7185de3ab60a)
> >      Haswell-2.2.0 - added 'rdrand' (78a611f1936b3eac8ed78a2be2146a742a85212c_
> >      Haswell-2.3.0 - removed 'hle' & 'rtm' (a356850b80b3d13b2ef737dad2acb05e6da03753)
> >      Haswell-2.5.0 - added 'abm' (becb66673ec30cb604926d247ab9449a60ad8b11
> >      Haswell-2.12.0 - added 'spec-ctrl' (ac96c41354b7e4c70b756342d9b686e31ab87458)
> >      Haswell-3.0.0  - added 'ssbd' (never done)
> > 
> > If we followed the machine type approach, then a bare "Haswell" would
> > statically resolve at build time to the most recent Haswell-X.X.X version
> > associated with the QEMU release. This is unhelpful as we have a direct
> > dependancy on the host hardware features. Better would be for a bare
> > "Haswell" to be dynamically resolved at runtime, picking the most recent
> > version that is capable of launching given the current hardware, KVM/TCG impl
> > and QEMU version.
> > 
> >   ie -cpu  Haswell
> > 
> > should use Haswell-2.5.0  if on silicon with the TSX errata applied,
> > but use Haswell-2.12.0 if the Spectre errata is applied in microcode,
> > and use Haswell-3.0.0 once Intel finally releases SSBD microcode errata.
> 
> Doing this unconditionally would make
> "-machine pc-q35-3.1 -cpu Haswell" unsafe for live migration, and
> break existing usage.  But this behavior could be enabled
> explicitly somehow.

True, for full back compat with existing libvirt we would probably
want to opt-in to it.

eg  -cpu Haswell could pick a fixed Haswell--XXX version according
to the machine type.  -cpu Haswell,best=on  could pick best version
for the host with the caveat about migration between heterogenous
hosts.

> > Versioning of CPU models as opposed to using arbitrary string suffixes
> > (-noTSX, -IBRS) has a number of usability improvements that we would
> > gain with versioned machine types, while avoiding exploding the machine
> > type matrix. With versioned CPU models we can
> > 
> >  - Automatically tailor the best model based on hardware support
> > 
> >  - Users always get the best model if they use the bare CPU name
> > 
> >  - It is obvious to users which is the "best" / "newest" CPU model
> > 
> >  - Avoid combinatorial expansion of machines since same CPU model
> >    version can be added to all releases without adding machine types.
> > 
> >  - Users can still force a specific downgraded model by using the
> >    fully versioned name.
> > 
> > Such versioning of CPU models would largely "just work" with existing
> > libvirt versions, but to libvirt would really want to expand the bare
> > CPU name to a versioned CPU name when recording new guest XML, so the
> > ABI is preserved long term.
> > 
> > An application like virt-manager which wants a simple UI can forever be
> > happy simply giving users a list of bare CPU model names, and allowing
> > libvirt / QEMU to automatically expand to the best versioned model for
> > their host.
> > 
> > An application like oVirt/OpenStack which wants direct control can allow
> > the admin to choice if a bare name, or explicitly picking a versioned name
> > if they need to cope with possibility of outdated hosts.
> > 
> 
> The proposal makes sense, and I think most of it can be already
> implemented on top of existing query-cpu-model-* commands.
> query-cpu-model-expansion type=static can expand to a versioned
> CPU model.
> 
> We will probably need to make query-cpu-model-expansion accept a
> machine-type name as input, and/or add a new flag meaning "please
> give me the best CPU version you have, not the one defined by the
> current machine-type".
> 
> I'm not sure what would be the best way to encode two types of
> information, though:
> 
> * Fallback/alternatives info, e.g.: "It makes sense to use
>   Haswell-{3.0,2.12,2.5,...} if Haswell-3.1 is not runnable and the
>   user asked for Haswell".
> 
> * Ordering/preference info, e.g.: "Haswell-3.1 is better than
>   Haswell-3.0, prefer the latter"

The version number of course gives an ordering, but we generally
tell people not to assume version is numeric.  We could report
an explicit "priority" in some manner against each.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] CPU model versioning separate from machine type versioning ?

[Daniel P. Berrangé]

On Fri, Jun 29, 2018 at 09:53:53AM +0100, Dr. David Alan Gilbert wrote:
> * Eduardo Habkost (ehabkost@redhat.com) wrote:
> > On Thu, Jun 28, 2018 at 04:45:02PM +0100, Daniel P. Berrangé wrote:
> > [...]
> > > What if we can borrow the concept of versioning from machine types and apply
> > > it to CPU models directly. For example, considering the history of "Haswell"
> > > in QEMU, if we had versioned things, we would by now have:
> > > 
> > >      Haswell-1.3.0 - first version (37507094f350b75c62dc059f998e7185de3ab60a)
> > >      Haswell-2.2.0 - added 'rdrand' (78a611f1936b3eac8ed78a2be2146a742a85212c_
> > >      Haswell-2.3.0 - removed 'hle' & 'rtm' (a356850b80b3d13b2ef737dad2acb05e6da03753)
> > >      Haswell-2.5.0 - added 'abm' (becb66673ec30cb604926d247ab9449a60ad8b11
> > >      Haswell-2.12.0 - added 'spec-ctrl' (ac96c41354b7e4c70b756342d9b686e31ab87458)
> > >      Haswell-3.0.0  - added 'ssbd' (never done)
> > > 
> > > If we followed the machine type approach, then a bare "Haswell" would
> > > statically resolve at build time to the most recent Haswell-X.X.X version
> > > associated with the QEMU release. This is unhelpful as we have a direct
> > > dependancy on the host hardware features. Better would be for a bare
> > > "Haswell" to be dynamically resolved at runtime, picking the most recent
> > > version that is capable of launching given the current hardware, KVM/TCG impl
> > > and QEMU version.
> > > 
> > >   ie -cpu  Haswell
> > > 
> > > should use Haswell-2.5.0  if on silicon with the TSX errata applied,
> > > but use Haswell-2.12.0 if the Spectre errata is applied in microcode,
> > > and use Haswell-3.0.0 once Intel finally releases SSBD microcode errata.
> > 
> > Doing this unconditionally would make
> > "-machine pc-q35-3.1 -cpu Haswell" unsafe for live migration, and
> > break existing usage.  But this behavior could be enabled
> > explicitly somehow.
> > 
> > > 
> > > Versioning of CPU models as opposed to using arbitrary string suffixes
> > > (-noTSX, -IBRS) has a number of usability improvements that we would
> > > gain with versioned machine types, while avoiding exploding the machine
> > > type matrix. With versioned CPU models we can
> > > 
> > >  - Automatically tailor the best model based on hardware support
> > > 
> > >  - Users always get the best model if they use the bare CPU name
> > > 
> > >  - It is obvious to users which is the "best" / "newest" CPU model
> > > 
> > >  - Avoid combinatorial expansion of machines since same CPU model
> > >    version can be added to all releases without adding machine types.
> > > 
> > >  - Users can still force a specific downgraded model by using the
> > >    fully versioned name.
> > > 
> > > Such versioning of CPU models would largely "just work" with existing
> > > libvirt versions, but to libvirt would really want to expand the bare
> > > CPU name to a versioned CPU name when recording new guest XML, so the
> > > ABI is preserved long term.
> > > 
> > > An application like virt-manager which wants a simple UI can forever be
> > > happy simply giving users a list of bare CPU model names, and allowing
> > > libvirt / QEMU to automatically expand to the best versioned model for
> > > their host.
> > > 
> > > An application like oVirt/OpenStack which wants direct control can allow
> > > the admin to choice if a bare name, or explicitly picking a versioned name
> > > if they need to cope with possibility of outdated hosts.
> > > 
> > 
> > The proposal makes sense, and I think most of it can be already
> > implemented on top of existing query-cpu-model-* commands.
> > query-cpu-model-expansion type=static can expand to a versioned
> > CPU model.
> > 
> > We will probably need to make query-cpu-model-expansion accept a
> > machine-type name as input, and/or add a new flag meaning "please
> > give me the best CPU version you have, not the one defined by the
> > current machine-type".
> > 
> > I'm not sure what would be the best way to encode two types of
> > information, though:
> > 
> 
> Both of those are solved with the numbering scheme
> 
> > * Fallback/alternatives info, e.g.: "It makes sense to use
> >   Haswell-{3.0,2.12,2.5,...} if Haswell-3.1 is not runnable and the
> >   user asked for Haswell".
> 
> Use the highest that works.
> 
> > * Ordering/preference info, e.g.: "Haswell-3.1 is better than
> >   Haswell-3.0, prefer the latter"
> 
> Higher is better.
> 
> The only thing that worries me about a numbering scheme is that
> it's now more difficult for a user to know whether they've got
> the type with a fix for a particular vulnerability.

True, but if more vulns arrive we have the same problem with named
suffixes too. eg if we added  -SSBD variants, users would ask whether
-SSBD includes the -IBRS fix or vica-verca, as a year down the line
they're not going to remember which or SSBD/IBRS came out first.

> We're going to have to say something like:
>   'For the new XYZ vulnerability make sure you're using
>   Haswell-3.2 or later, SkyLake-2.6 or later, Westmere-4.8 or later
>   .....'
> 
> which all gets a bit confusing.

The kernel has a /sys/devices/system/cpu/vulnerabilities dir
that lists status of various flaws.

I have been thinking about whether libvirt should create a
'virt-guest-validate' command that looks at guest XML and
reports whether any of the config settings are vulnerable
or otherwise diverging from best practice in some way.

QEMU itself would perhaps have a 'query-vulnerabilities'
monitor command to report whether the current config is
satisfactory or not.

Ultimately though, getting a fixed guest involves host
kernel, microcode, qemu, and guest kernel. So to get a
true picture of your safety people should really look
straight to the guest kernels' /sys/devices/system/cpu/vulnerabilities
directory. They only need to look at host/microcode/qemu if the
guest is reporting something is wrong.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] CPU model versioning separate from machine type versioning ?

[Jiri Denemark]

On Fri, Jun 29, 2018 at 11:14:17 +0100, Daniel P. Berrangé wrote:
> On Thu, Jun 28, 2018 at 04:52:27PM -0300, Eduardo Habkost wrote:
> > On Thu, Jun 28, 2018 at 04:45:02PM +0100, Daniel P. Berrangé wrote:
> > [...]
> > > What if we can borrow the concept of versioning from machine types and apply
> > > it to CPU models directly. For example, considering the history of "Haswell"
> > > in QEMU, if we had versioned things, we would by now have:
> > > 
> > >      Haswell-1.3.0 - first version (37507094f350b75c62dc059f998e7185de3ab60a)
> > >      Haswell-2.2.0 - added 'rdrand' (78a611f1936b3eac8ed78a2be2146a742a85212c_
> > >      Haswell-2.3.0 - removed 'hle' & 'rtm' (a356850b80b3d13b2ef737dad2acb05e6da03753)
> > >      Haswell-2.5.0 - added 'abm' (becb66673ec30cb604926d247ab9449a60ad8b11
> > >      Haswell-2.12.0 - added 'spec-ctrl' (ac96c41354b7e4c70b756342d9b686e31ab87458)
> > >      Haswell-3.0.0  - added 'ssbd' (never done)
> > > 
> > > If we followed the machine type approach, then a bare "Haswell" would
> > > statically resolve at build time to the most recent Haswell-X.X.X version
> > > associated with the QEMU release. This is unhelpful as we have a direct
> > > dependancy on the host hardware features. Better would be for a bare
> > > "Haswell" to be dynamically resolved at runtime, picking the most recent
> > > version that is capable of launching given the current hardware, KVM/TCG impl
> > > and QEMU version.
> > > 
> > >   ie -cpu  Haswell
> > > 
> > > should use Haswell-2.5.0  if on silicon with the TSX errata applied,
> > > but use Haswell-2.12.0 if the Spectre errata is applied in microcode,
> > > and use Haswell-3.0.0 once Intel finally releases SSBD microcode errata.
> > 
> > Doing this unconditionally would make
> > "-machine pc-q35-3.1 -cpu Haswell" unsafe for live migration, and
> > break existing usage.  But this behavior could be enabled
> > explicitly somehow.
> 
> True, for full back compat with existing libvirt we would probably
> want to opt-in to it.
> 
> eg  -cpu Haswell could pick a fixed Haswell--XXX version according
> to the machine type.  -cpu Haswell,best=on  could pick best version
> for the host with the caveat about migration between heterogenous
> hosts.

I was thinking we could even separate the CPU model version from the
name itself:

    -cpu Haswell                    (the old, compatible way)
    -cpu Haswell,version=best
    -cpu Haswell,version=2.12.0

It would be slightly more work for the upper management layers, but IMHO
it would make more sense.

In any case, we have to think about keeping guest ABI stable.

I hope the automatic version selection would not cause any problems for
subsequent cold starts (such as Windows activation issues). It should be
very similar to updating CPU microcode which the guest OS is already
supposed to deal with in real hardware. However, in the past QEMU
changed CPU signature (family, model, stepping) for new machine types
and it is likely to happen with separately versioned CPU models too. I
believe CPU microcode updates do not touch these values. On the other
hand, it's similar to host-model and the user can always specify exact
version to avoid this slight change should it be a problem.

Once the domain starts, we need to keep stable ABI across migrations,
save/restores, or snapshots. Libvirt already does so by talking to QEMU
before starting vCPUs and checking for disabled/enabled features. Then
we store this information in the active domain XML to make sure we can
enforce the same CPU later. This concept would need to be enhanced to
include the CPU model version which QEMU would need to be able to
report.

A significantly more fun would result from letting libvirt use the
versioned CPU model stuff by default without an explicit knob in the
XML. But I guess you don't want to go that direction, do you?

Jirka

Re: [Qemu-devel] CPU model versioning separate from machine type versioning ?

[Daniel P. Berrangé]

On Fri, Jun 29, 2018 at 02:12:51PM +0200, Jiri Denemark wrote:
> On Fri, Jun 29, 2018 at 11:14:17 +0100, Daniel P. Berrangé wrote:
> > On Thu, Jun 28, 2018 at 04:52:27PM -0300, Eduardo Habkost wrote:
> > > On Thu, Jun 28, 2018 at 04:45:02PM +0100, Daniel P. Berrangé wrote:
> > > [...]
> > > > What if we can borrow the concept of versioning from machine types and apply
> > > > it to CPU models directly. For example, considering the history of "Haswell"
> > > > in QEMU, if we had versioned things, we would by now have:
> > > > 
> > > >      Haswell-1.3.0 - first version (37507094f350b75c62dc059f998e7185de3ab60a)
> > > >      Haswell-2.2.0 - added 'rdrand' (78a611f1936b3eac8ed78a2be2146a742a85212c_
> > > >      Haswell-2.3.0 - removed 'hle' & 'rtm' (a356850b80b3d13b2ef737dad2acb05e6da03753)
> > > >      Haswell-2.5.0 - added 'abm' (becb66673ec30cb604926d247ab9449a60ad8b11
> > > >      Haswell-2.12.0 - added 'spec-ctrl' (ac96c41354b7e4c70b756342d9b686e31ab87458)
> > > >      Haswell-3.0.0  - added 'ssbd' (never done)
> > > > 
> > > > If we followed the machine type approach, then a bare "Haswell" would
> > > > statically resolve at build time to the most recent Haswell-X.X.X version
> > > > associated with the QEMU release. This is unhelpful as we have a direct
> > > > dependancy on the host hardware features. Better would be for a bare
> > > > "Haswell" to be dynamically resolved at runtime, picking the most recent
> > > > version that is capable of launching given the current hardware, KVM/TCG impl
> > > > and QEMU version.
> > > > 
> > > >   ie -cpu  Haswell
> > > > 
> > > > should use Haswell-2.5.0  if on silicon with the TSX errata applied,
> > > > but use Haswell-2.12.0 if the Spectre errata is applied in microcode,
> > > > and use Haswell-3.0.0 once Intel finally releases SSBD microcode errata.
> > > 
> > > Doing this unconditionally would make
> > > "-machine pc-q35-3.1 -cpu Haswell" unsafe for live migration, and
> > > break existing usage.  But this behavior could be enabled
> > > explicitly somehow.
> > 
> > True, for full back compat with existing libvirt we would probably
> > want to opt-in to it.
> > 
> > eg  -cpu Haswell could pick a fixed Haswell--XXX version according
> > to the machine type.  -cpu Haswell,best=on  could pick best version
> > for the host with the caveat about migration between heterogenous
> > hosts.
> 
> I was thinking we could even separate the CPU model version from the
> name itself:
> 
>     -cpu Haswell                    (the old, compatible way)
>     -cpu Haswell,version=best
>     -cpu Haswell,version=2.12.0
> 
> It would be slightly more work for the upper management layers, but IMHO
> it would make more sense.
> 
> In any case, we have to think about keeping guest ABI stable.
> 
> I hope the automatic version selection would not cause any problems for
> subsequent cold starts (such as Windows activation issues). It should be
> very similar to updating CPU microcode which the guest OS is already
> supposed to deal with in real hardware. However, in the past QEMU
> changed CPU signature (family, model, stepping) for new machine types
> and it is likely to happen with separately versioned CPU models too. I
> believe CPU microcode updates do not touch these values. On the other
> hand, it's similar to host-model and the user can always specify exact
> version to avoid this slight change should it be a problem.
> 
> Once the domain starts, we need to keep stable ABI across migrations,
> save/restores, or snapshots. Libvirt already does so by talking to QEMU
> before starting vCPUs and checking for disabled/enabled features. Then
> we store this information in the active domain XML to make sure we can
> enforce the same CPU later. This concept would need to be enhanced to
> include the CPU model version which QEMU would need to be able to
> report.
> 
> A significantly more fun would result from letting libvirt use the
> versioned CPU model stuff by default without an explicit knob in the
> XML. But I guess you don't want to go that direction, do you?

>From libvirt POV I think we have to always fully expand the version
at time of define, in the same way we expand the machine type. Nothing
else can guarantee ABI.

So once you do have a guest and you want to update it to use a newer
version, you'd have to explicitly delete the version from the XML and
let libvirt update it again, similar to how you update existing guest
to new machine type.

We could perhaps make this easier - eg  VIR_DOMAIN_XML_UPDATE_CPU
flag for virDomainDefine/virDomainCreateXML  or similar.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [Qemu-devel] [libvirt] CPU model versioning separate from machine type versioning ?

[Eduardo Habkost]

On Fri, Jun 29, 2018 at 08:06:04AM +0200, Jiri Denemark wrote:
> On Thu, Jun 28, 2018 at 16:23:53 -0300, Eduardo Habkost wrote:
> > On Thu, Jun 28, 2018 at 07:59:38PM +0100, Dr. David Alan Gilbert wrote:
[...]
> > > Would you restrict the combinations to cut down the test matrix - e.g.
> > > not allow Haswell-3.0.0 on anything prior to a 2.12 machine type?
> > 
> > Not sure if it would be worth the extra complexity: we would need
> > an interface to tell libvirt which CPU models are usable on which
> > machine-types.
> 
> In case we do this, libvirt should already by ready for it on the API
> level for both reporting capabilities and CPU comparison/baseline. All
> these APIs already accept machine type as an optional parameter so that
> different results can be provided depending on machine type.

Does it have an abstraction that can already represent "CPU model
$C can/can't be used with machine-type $M"?

-- 
Eduardo

Re: [Qemu-devel] CPU model versioning separate from machine type versioning ?

[Eduardo Habkost]

On Fri, Jun 29, 2018 at 02:12:51PM +0200, Jiri Denemark wrote:
> On Fri, Jun 29, 2018 at 11:14:17 +0100, Daniel P. Berrangé wrote:
> > On Thu, Jun 28, 2018 at 04:52:27PM -0300, Eduardo Habkost wrote:
> > > On Thu, Jun 28, 2018 at 04:45:02PM +0100, Daniel P. Berrangé wrote:
> > > [...]
> > > > What if we can borrow the concept of versioning from machine types and apply
> > > > it to CPU models directly. For example, considering the history of "Haswell"
> > > > in QEMU, if we had versioned things, we would by now have:
> > > > 
> > > >      Haswell-1.3.0 - first version (37507094f350b75c62dc059f998e7185de3ab60a)
> > > >      Haswell-2.2.0 - added 'rdrand' (78a611f1936b3eac8ed78a2be2146a742a85212c_
> > > >      Haswell-2.3.0 - removed 'hle' & 'rtm' (a356850b80b3d13b2ef737dad2acb05e6da03753)
> > > >      Haswell-2.5.0 - added 'abm' (becb66673ec30cb604926d247ab9449a60ad8b11
> > > >      Haswell-2.12.0 - added 'spec-ctrl' (ac96c41354b7e4c70b756342d9b686e31ab87458)
> > > >      Haswell-3.0.0  - added 'ssbd' (never done)
> > > > 
> > > > If we followed the machine type approach, then a bare "Haswell" would
> > > > statically resolve at build time to the most recent Haswell-X.X.X version
> > > > associated with the QEMU release. This is unhelpful as we have a direct
> > > > dependancy on the host hardware features. Better would be for a bare
> > > > "Haswell" to be dynamically resolved at runtime, picking the most recent
> > > > version that is capable of launching given the current hardware, KVM/TCG impl
> > > > and QEMU version.
> > > > 
> > > >   ie -cpu  Haswell
> > > > 
> > > > should use Haswell-2.5.0  if on silicon with the TSX errata applied,
> > > > but use Haswell-2.12.0 if the Spectre errata is applied in microcode,
> > > > and use Haswell-3.0.0 once Intel finally releases SSBD microcode errata.
> > > 
> > > Doing this unconditionally would make
> > > "-machine pc-q35-3.1 -cpu Haswell" unsafe for live migration, and
> > > break existing usage.  But this behavior could be enabled
> > > explicitly somehow.
> > 
> > True, for full back compat with existing libvirt we would probably
> > want to opt-in to it.
> > 
> > eg  -cpu Haswell could pick a fixed Haswell--XXX version according
> > to the machine type.  -cpu Haswell,best=on  could pick best version
> > for the host with the caveat about migration between heterogenous
> > hosts.
> 
> I was thinking we could even separate the CPU model version from the
> name itself:
> 
>     -cpu Haswell                    (the old, compatible way)
>     -cpu Haswell,version=best
>     -cpu Haswell,version=2.12.0

That's a nice idea.  The only problem I see is that this:

  -> query-cpu-model-expansion type=static model=Haswell
  <- { model: { name: "Haswell-2.12.0" } }

is returning a static CPU model ("Haswell-2.12.0") on
`model.name`, which matches the documentation for type=static.
But this:

  -> query-cpu-model-expansion type=static model=Haswell
  <- { model: { name: "Haswell", version="2.12.0" } }

is returning a non-static CPU model name ("Haswell") on
`model.name`, which breaks the existing documentation of
type=static ("Expand to a static CPU model, a combination of a
static base model name and property delta changes").

Maybe this would work:

  -> query-cpu-model-expansion type=static model=Haswell
  <- { model: { name: "Haswell-base", version="2.12.0" } }

"Haswell-base" would be a static CPU model.  "Haswell" would be a
non-static but migration-safe CPU model (which is already the
case today).

Having a "Haswell-2.12.0" alias (that looks like a regular CPU
model) for legacy management management software would be
possible too.

-- 
Eduardo

Re: [Qemu-devel] CPU model versioning separate from machine type versioning ?

[Eduardo Habkost]

On Fri, Jun 29, 2018 at 11:19:17AM +0100, Daniel P. Berrangé wrote:
> On Fri, Jun 29, 2018 at 09:53:53AM +0100, Dr. David Alan Gilbert wrote:
[...]
> > We're going to have to say something like:
> >   'For the new XYZ vulnerability make sure you're using
> >   Haswell-3.2 or later, SkyLake-2.6 or later, Westmere-4.8 or later
> >   .....'
> > 
> > which all gets a bit confusing.
> 
> The kernel has a /sys/devices/system/cpu/vulnerabilities dir
> that lists status of various flaws.
> 
> I have been thinking about whether libvirt should create a
> 'virt-guest-validate' command that looks at guest XML and
> reports whether any of the config settings are vulnerable
> or otherwise diverging from best practice in some way.
> 
> QEMU itself would perhaps have a 'query-vulnerabilities'
> monitor command to report whether the current config is
> satisfactory or not.

Makes sense to me.  I wanted to make QEMU emit warnings on
obviously insecure configurations.  Adding a
query-vulnerabilities command would be the QMP counterpart of
that.

-- 
Eduardo

Re: [Qemu-devel] CPU model versioning separate from machine type versioning ?

[Eduardo Habkost]

On Fri, Jun 29, 2018 at 11:14:17AM +0100, Daniel P. Berrangé wrote:
> On Thu, Jun 28, 2018 at 04:52:27PM -0300, Eduardo Habkost wrote:
[...]
> > I'm not sure what would be the best way to encode two types of
> > information, though:
> > 
> > * Fallback/alternatives info, e.g.: "It makes sense to use
> >   Haswell-{3.0,2.12,2.5,...} if Haswell-3.1 is not runnable and the
> >   user asked for Haswell".
> > 
> > * Ordering/preference info, e.g.: "Haswell-3.1 is better than
> >   Haswell-3.0, prefer the latter"
> 
> The version number of course gives an ordering, but we generally
> tell people not to assume version is numeric.  We could report
> an explicit "priority" in some manner against each.

Makes sense.  "priority" could be included on
query-cpu-definitions to help software choose the best
alternative, and "version" could be just an opaque string that
libvirt needs to save after expanding a CPU model.

-- 
Eduardo