[PATCH v3 0/4] target/i386/cpu: introduce new CPU models for x86-64 ABI levels

[PATCH v3 0/4] target/i386/cpu: introduce new CPU models for x86-64 ABI levels

[Daniel P. Berrangé]

This series is motivated by this blog that describes how RHEL-9
will recommend use of the x86-64-v2 microarchitectural ABI level:

  https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level/

The implication of compiling code with -march=x86-64-v2 is that
this code will no longer be guaranteed to be runnable on a
number of the CPU models exposed by the x86_64 target emulator,
most notably qemu64 which is the default.

This series is not proposing to change the QEMU default CPU model
for x86_64 target. I show how this is can trivially be done in
patch 3, but not suggesting that we actually do that, as upstream
is quite conservative in dropping support for old host hardware.

New CPU models
==============

It is instead exploring the possibilities of defining new CPU
models in QEMU to closely match the x86-64 uarch ABI levels in
a relatively vendor agnostic manner. This could be used by
downstream vendors who wish to target specific uarch ABI levels
in custom machine types.

One of the nice things about "qemu64" is that its naming presents
it as effectively being a vendor-neutral model (if we ignore that
vendor=AMD is in fact reported in CPUID).

If we look at the feature set fo x86-64-v2 ABI, we see that the
QEMU "Nehalem" model is the closest match. This is also happens
to be runnable on AMD Opteron G4/G5 and EPYC hosts. None the less,
the use of an Intel specific CPU model name on an AMD host feels
uncomfortable.

Vendor neutral naming
=====================

The idea behind this series is thus to introduce new CPU model
names with vendor neutral naming, to more directly correlate
with defined x86-64 uarch ABI levels. We don't want to just
invent CPUs with a completely arbitrary set of CPU features as
history has shown that brings its own problems. eg a guest
uses features A and B, but only does CPUID checks for existence
of feature B, assuming that B implies A.

The specification for x86-64 ABI levels uses x86-64-vNN naming
but this clashes with QEMU's use of "vNN" for versioning. I
felt it would be confusing to end up with CPU model names
like  "x86-64-v1-v1". Thus I've used an "-abiNNN" suffix
instead. Also note I have an "-abi1" suffix here for the
baseline. Arguably we could just drop the suffix entirely for
the baseline.

A further note is that we could have used "qemu64" as the
naming prefix, eg qemu64-abi2, qemu64-abi3, etc. Alot of
people hold negative opinions of the qemu64 model in general
though, so I felt a clean break with the past might be
desirable, even though the x86-64-abi1 CPU  model is effectively
identical to qemu64.

Runnability of new models
=========================

The goal of the x86-64-abiNNN CPU models is that they should be
runnable on any physical host which supports the CPUIDs features
for that uarch ABI level. It is hard to figure out what exact
set of CPUID features we should report. The uarch ABI document
only specifies the minimum expectation, but we can't define a
CPU in QEMU using only the minimum set as that would omit
countless important features.

Thus to determine the feature set to use in x86-64-abiNNN CPU
models, this series used a script (see patch 4) which did the
following:

  * Find list of current CPU models that satisfy ABI NN
  * From this list, calculate the lowest common denominator (LCD)
    of CPUID features
  * From the LCD, find the existing CPU model that has the fewest
    extra features over the LCD.
  * Use that closest match, minus the extra features
    to define x86-64-abiNN

This approach works well for the baseline ABI, and level 2 ABI

For ABI level 3 there is a bit of a larger divergance in feature
sets, because the LCD between Intel and EPYC models is relatively
large. We omit aes pcid erms invpcid tsc-deadline x2apic pclmulqdq

For ABI level 4 the divergance is small. I believe this is a
deceptive situation that arises because currently only Intel
CPUs in QEMU are able to satisfy ABI level 4 currently. None
of the EPYC CPUs defined in QEMU are able to satisfy it.

I'm concerned that if future EPYC CPU generations are added to
QEMU which *do* indeed satisfy x86-64 v4, then QEMU's x86-64-abi4
CPUID may prove to be too greedy and thus be unable to run on
the future CPUs. In theory we can address this through use of
CPU versioning, but ideally we would not have to deal with that.

Nested virt caveats
===================

A further caveat is that none of these generic models define
any VMX feature set. At the very least I believe this means
that nested virt can't live migrate, but maybe this breaks
ability to use nested virt entirely. I'm unclear what the
best approach is to deal with this, that would let the user
do both "-cpu x86-64-abi2,svm=on" and "-cpu x86-64-abi2,vmx=on"

Side channel mitigation caveats
===============================

Finally none of the x86-64-abiNN models include any of the
features related to CPU side channel mitigations. We can't
add them without sacrificing the ability to run on some
physical hardware that would otherwise satisfy the uarch
ABI level.

This makes me a little uneasy. One of the main downsides
of using "qemu64" is that it leaves guests vulnerable to the
side channel attacks. Introducing new general purpose CPUs
that still have this flaw feels undesirable. I don't know
whethe to prioritize safety, while sacrificing runability.

Crypto accelerator caveats
==========================

Similarly I'm not a huge fan of leaving out the "aes"
instruction for accelerated crypto, as missing "aes" is
also one of the key factors in making qemu64 a bad choice.

If we include 'aes' in x86-64-abi2, then we loose support
for Nehalem hosts.

If we include 'aes' in x86-64-abi3 then we further loose
support for Dhyana hosts (an EPYC derived CPU).

Wrap up
=======

Overall this series shows some CPU models for QEMU that
can map to each x86-64 ABI level, but I'm unsure whether
it is wise to actually go forward with this idea given
the various caveats.

I think they'll mostly only be useful as a built-in default
for machine types. If anyone is going to explicitly specify
a CPU model they'll almost always be better off picking a
vendor specific model or host passthrough. There can be
cases where people have a mix of AMD and Intel bare metal
machines they need portability across but this feel rare.

Whatever else comes of this series, at least patch 1 should
be useful, giving people an idea of what ABI levels each
QEMU CPU can satisfy in the documentation.

Changed in v3:

 - Rebased to resolve conflicts with git master

Changed in v2:

 - Don't document any of the CPU aliases, since they vary per
   machine type, only the concrete types.
 - Formally add the script for re-generating the docs

Daniel P. Berrangé (4):
  docs: add a table showing x86-64 ABI compatibility levels
  target/i386: define CPU models to model x86-64 ABI levels
  scripts: helper to generate x86_64 CPU ABI compat info
  NOT FOR MERGE target/i386: use x86-64-abi1 CPU model as default on
    x86_64

 MAINTAINERS                        |   2 +-
 docs/system/cpu-models-x86-abi.csv |  75 +++++++++++
 docs/system/cpu-models-x86.rst.inc |  22 ++++
 hw/i386/pc_piix.c                  |   3 +
 hw/i386/pc_q35.c                   |   3 +
 scripts/cpu-x86-uarch-abi.py       | 194 +++++++++++++++++++++++++++++
 target/i386/cpu.c                  | 156 +++++++++++++++++++++++
 target/i386/cpu.h                  |   2 +-
 8 files changed, 455 insertions(+), 2 deletions(-)
 create mode 100644 docs/system/cpu-models-x86-abi.csv
 create mode 100644 scripts/cpu-x86-uarch-abi.py

-- 
2.31.1

[PATCH v3 1/4] docs: add a table showing x86-64 ABI compatibility levels

[Daniel P. Berrangé]

It is useful to know which CPUs satisfy each x86-64 ABI
compatibility level, when dealing with guest OS that require
something newer than the baseline ABI.

These ABI levels are defined in:

  https://gitlab.com/x86-psABIs/x86-64-ABI/

and supported by GCC, Clang, glibc and more.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 MAINTAINERS                        |  2 +-
 docs/system/cpu-models-x86-abi.csv | 67 ++++++++++++++++++++++++++++++
 docs/system/cpu-models-x86.rst.inc | 22 ++++++++++
 3 files changed, 90 insertions(+), 1 deletion(-)
 create mode 100644 docs/system/cpu-models-x86-abi.csv

diff --git a/MAINTAINERS b/MAINTAINERS
index 7d9cd29042..56888121ec 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -328,7 +328,7 @@ F: tests/tcg/i386/
 F: tests/tcg/x86_64/
 F: hw/i386/
 F: disas/i386.c
-F: docs/system/cpu-models-x86.rst.inc
+F: docs/system/cpu-models-x86*
 T: git https://gitlab.com/ehabkost/qemu.git x86-next
 
 Xtensa TCG CPUs
diff --git a/docs/system/cpu-models-x86-abi.csv b/docs/system/cpu-models-x86-abi.csv
new file mode 100644
index 0000000000..f3f3b60be1
--- /dev/null
+++ b/docs/system/cpu-models-x86-abi.csv
@@ -0,0 +1,67 @@
+Model,baseline,v2,v3,v4
+486-v1,,,,
+Broadwell-v1,✅,✅,✅,
+Broadwell-v2,✅,✅,✅,
+Broadwell-v3,✅,✅,✅,
+Broadwell-v4,✅,✅,✅,
+Cascadelake-Server-v1,✅,✅,✅,✅
+Cascadelake-Server-v2,✅,✅,✅,✅
+Cascadelake-Server-v3,✅,✅,✅,✅
+Cascadelake-Server-v4,✅,✅,✅,✅
+Conroe-v1,✅,,,
+Cooperlake-v1,✅,✅,✅,✅
+Denverton-v1,✅,✅,,
+Denverton-v2,✅,✅,,
+Dhyana-v1,✅,✅,✅,
+EPYC-Milan-v1,✅,✅,✅,
+EPYC-Rome-v1,✅,✅,✅,
+EPYC-Rome-v2,✅,✅,✅,
+EPYC-v1,✅,✅,✅,
+EPYC-v2,✅,✅,✅,
+EPYC-v3,✅,✅,✅,
+Haswell-v1,✅,✅,✅,
+Haswell-v2,✅,✅,✅,
+Haswell-v3,✅,✅,✅,
+Haswell-v4,✅,✅,✅,
+Icelake-Client-v1,✅,✅,✅,
+Icelake-Client-v2,✅,✅,✅,
+Icelake-Server-v1,✅,✅,✅,✅
+Icelake-Server-v2,✅,✅,✅,✅
+Icelake-Server-v3,✅,✅,✅,✅
+Icelake-Server-v4,✅,✅,✅,✅
+IvyBridge-v1,✅,✅,,
+IvyBridge-v2,✅,✅,,
+KnightsMill-v1,✅,✅,✅,
+Nehalem-v1,✅,✅,,
+Nehalem-v2,✅,✅,,
+Opteron_G1-v1,✅,,,
+Opteron_G2-v1,✅,,,
+Opteron_G3-v1,✅,,,
+Opteron_G4-v1,✅,✅,,
+Opteron_G5-v1,✅,✅,,
+Penryn-v1,✅,,,
+SandyBridge-v1,✅,✅,,
+SandyBridge-v2,✅,✅,,
+Skylake-Client-v1,✅,✅,✅,
+Skylake-Client-v2,✅,✅,✅,
+Skylake-Client-v3,✅,✅,✅,
+Skylake-Server-v1,✅,✅,✅,✅
+Skylake-Server-v2,✅,✅,✅,✅
+Skylake-Server-v3,✅,✅,✅,✅
+Skylake-Server-v4,✅,✅,✅,✅
+Snowridge-v1,✅,✅,,
+Snowridge-v2,✅,✅,,
+Westmere-v1,✅,✅,,
+Westmere-v2,✅,✅,,
+athlon-v1,,,,
+core2duo-v1,✅,,,
+coreduo-v1,,,,
+kvm32-v1,,,,
+kvm64-v1,✅,,,
+n270-v1,,,,
+pentium-v1,,,,
+pentium2-v1,,,,
+pentium3-v1,,,,
+phenom-v1,✅,,,
+qemu32-v1,,,,
+qemu64-v1,✅,,,
diff --git a/docs/system/cpu-models-x86.rst.inc b/docs/system/cpu-models-x86.rst.inc
index 867c8216b5..f40ee03ecc 100644
--- a/docs/system/cpu-models-x86.rst.inc
+++ b/docs/system/cpu-models-x86.rst.inc
@@ -39,6 +39,28 @@ CPU, as they would with "Host passthrough", but gives much of the
 benefit of passthrough, while making live migration safe.
 
 
+ABI compatibility levels for CPU models
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The x86_64 architecture has a number of `ABI compatibility levels`_
+defined. Traditionally most operating systems and toolchains would
+only target the original baseline ABI. It is expected that in
+future OS and toolchains are likely to target newer ABIs. The
+table that follows illustrates which ABI compatibility levels
+can be satisfied by the QEMU CPU models. Note that the table only
+lists the long term stable CPU model versions (eg Haswell-v4).
+In addition to whats listed, there are also many CPU model
+aliases which resolve to a different CPU model version,
+depending on the machine type is in use.
+
+.. _ABI compatibility levels: https://gitlab.com/x86-psABIs/x86-64-ABI/
+
+.. csv-table:: x86-64 ABI compatibility levels
+   :file: cpu-models-x86-abi.csv
+   :widths: 40,15,15,15,15
+   :header-rows: 2
+
+
 Preferred CPU models for Intel x86 hosts
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-- 
2.31.1

[PATCH v3 2/4] target/i386: define CPU models to model x86-64 ABI levels

[Daniel P. Berrangé]

To paraphrase:

  https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level/

In 2020, AMD, Intel, Red Hat, and SUSE worked together to define
three microarchitecture levels on top of the historical x86-64
baseline:

  * x86-64:    original x86_64 baseline instruction set
  * x86-64-v2: vector instructions up to Streaming SIMD
               Extensions 4.2 (SSE4.2)  and Supplemental
	       Streaming SIMD Extensions 3 (SSSE3), the
	       POPCNT instruction, and CMPXCHG16B
  * x86-64-v3: vector instructions up to AVX2, MOVBE,
               and additional bit-manipulation instructions.
  * x86-64-v4: vector instructions from some of the
               AVX-512 variants.

This list of features is defined in the doc at:

  https://gitlab.com/x86-psABIs/x86-64-ABI/

QEMU has historically defaulted to the "qemu64" CPU model on
x86_64 targets, which is approximately the x86-64 baseline
ABI, with 'SVM' added.

It is thought it might be desirable if QEMU could provide CPU models
that closely correspond to the ABI levels, while offering portability
across the maximum number of physical CPUs.

Historically we've found that defining CPU models with an arbitrary
combination of CPU features can be problematic, as some guest OS
will not check all features they use, and instead assume that if
they see feature "XX", then "YY" will always exist. This is reasonable
in bare metal, but subject to breakage in virtualization.

Thus in defining the CPU models for the ABI levels, this patch attempted
to base them off an existing CPU model.

While each x86-64-abiNNN has a designated vendor, they are designed
to be vendor agnostic models. ie they are capable of running on any
physical x86_64 CPU model that satisfies the ABI level. eg although
the x86-64-abi2 model is based on Nehalem, it should be able to run
guests on an Opteron_G4/G5/EPYC host, since those CPUs support the
features required by the x86-64 v2 ABI.

More precisely the models were defined as:

 * x86-64-abi1: close match for Opteron_G1, minus
                vme
 * x86-64-abi2: perfect match for Nehalem
 * x86-64-abi3: close match of Haswell-noTSX, minus
                aes pcid erms invpcid tsc-deadline
		x2apic pclmulqdq
 * x86-64-abi4: close match of Skylake-Server-noTSX-IBRS, minus
                spec-ctrl

None of the CPU models declare any VMX/SVM capability features.
IOW, even if a "vmx"/"svm" flag is added, it will still be unsafe
to attempt to live migrate the L1 guest if there are any L2
guests running with hardware virtualization.

Given their vendor agnostic design, these CPU models are primarily
thought tobe  useful as the default CPU model for machine types.
QEMU upstream is quite conservative in mandating new hardware features,
but a downstream vendor may choose to mandate a newer x86-64 ABI level
for downstream only machine types.

Note that TCG is not capable of supporting the 2 newest ABI levels
currently due to missing features:

* x86-64-abi3:

  CPUID.01H:ECX.fma [bit 12]
  CPUID.01H:ECX.avx [bit 28]
  CPUID.01H:ECX.f16c [bit 29]
  CPUID.07H:EBX.avx2 [bit 5]

* x86-64-abi4:

  CPUID.01H:ECX.pcid [bit 17]
  CPUID.01H:ECX.x2apic [bit 21]
  CPUID.01H:ECX.tsc-deadline [bit 24]
  CPUID.07H:EBX.invpcid [bit 10]
  CPUID.07H:EBX.avx512f [bit 16]
  CPUID.07H:EBX.avx512dq [bit 17]
  CPUID.07H:EBX.rdseed [bit 18]
  CPUID.07H:EBX.avx512cd [bit 28]
  CPUID.07H:EBX.avx512bw [bit 30]
  CPUID.07H:EBX.avx512vl [bit 31]
  CPUID.80000001H:ECX.3dnowprefetch [bit 8]
  CPUID.0DH:EAX.xsavec [bit 1]

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 docs/system/cpu-models-x86-abi.csv |   8 ++
 target/i386/cpu.c                  | 156 +++++++++++++++++++++++++++++
 2 files changed, 164 insertions(+)

diff --git a/docs/system/cpu-models-x86-abi.csv b/docs/system/cpu-models-x86-abi.csv
index f3f3b60be1..3392c39753 100644
--- a/docs/system/cpu-models-x86-abi.csv
+++ b/docs/system/cpu-models-x86-abi.csv
@@ -65,3 +65,11 @@ pentium3-v1,,,,
 phenom-v1,✅,,,
 qemu32-v1,,,,
 qemu64-v1,✅,,,
+x86-64-abi1,✅,,,
+x86-64-abi1-v1,✅,,,
+x86-64-abi2,✅,✅,,
+x86-64-abi2-v1,✅,✅,,
+x86-64-abi3,✅,✅,✅,
+x86-64-abi3-v1,✅,✅,✅,
+x86-64-abi4,✅,✅,✅,✅
+x86-64-abi4-v1,✅,✅,✅,✅
diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index a9fe1662d3..e3b8a6f6f1 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -1711,6 +1711,162 @@ static const CPUCaches epyc_milan_cache_info = {
  */
 
 static const X86CPUDefinition builtin_x86_defs[] = {
+    /*
+     * These first few CPU models are designed to satisfy the
+     * x86_64 ABI levels defined in:
+     *
+     *   https://gitlab.com/x86-psABIs/x86-64-ABI/
+     *
+     * They were constructed as follows:
+     *
+     *   - Find all the CPU models which can satisfy the ABI
+     *   - Calculate the lowest common denominator (LCD) of these
+     *     models' features
+     *   - Find the named model most closely matching the LCD
+     *   - Strip its features back to the LCD
+     *
+     * The above spec uses the "x86-64-vNN" naming convention.
+     * This clashes with the "vNN" suffix QEMU uses for versioning.
+     * Thus we use "abiNNN" as a suffix.
+     */
+    {
+        /*
+         * Derived from Opteron_G1, minus
+         *   vme
+         */
+        .name = "x86-64-abi1",
+        .level = 5,
+        .vendor = CPUID_VENDOR_AMD,
+        .family = 15,
+        .model = 6,
+        .stepping = 1,
+        .features[FEAT_1_EDX] =
+            CPUID_SSE2 | CPUID_SSE | CPUID_FXSR | CPUID_MMX |
+            CPUID_CLFLUSH | CPUID_PSE36 | CPUID_PAT | CPUID_CMOV | CPUID_MCA |
+            CPUID_PGE | CPUID_MTRR | CPUID_SEP | CPUID_APIC | CPUID_CX8 |
+            CPUID_MCE | CPUID_PAE | CPUID_MSR | CPUID_TSC | CPUID_PSE |
+            CPUID_DE | CPUID_FP87,
+        .features[FEAT_1_ECX] =
+            CPUID_EXT_SSE3,
+        .features[FEAT_8000_0001_EDX] =
+            CPUID_EXT2_LM | CPUID_EXT2_NX | CPUID_EXT2_SYSCALL,
+        .xlevel = 0x80000008,
+        .model_id = "QEMU x86-64 baseline ABI",
+    },
+    {
+        /* Derived from Nehalem */
+        .name = "x86-64-abi2",
+        .level = 11,
+        .vendor = CPUID_VENDOR_INTEL,
+        .family = 6,
+        .model = 26,
+        .stepping = 3,
+        .features[FEAT_1_EDX] =
+            CPUID_VME | CPUID_SSE2 | CPUID_SSE | CPUID_FXSR | CPUID_MMX |
+            CPUID_CLFLUSH | CPUID_PSE36 | CPUID_PAT | CPUID_CMOV | CPUID_MCA |
+            CPUID_PGE | CPUID_MTRR | CPUID_SEP | CPUID_APIC | CPUID_CX8 |
+            CPUID_MCE | CPUID_PAE | CPUID_MSR | CPUID_TSC | CPUID_PSE |
+            CPUID_DE | CPUID_FP87,
+        .features[FEAT_1_ECX] =
+            CPUID_EXT_POPCNT | CPUID_EXT_SSE42 | CPUID_EXT_SSE41 |
+            CPUID_EXT_CX16 | CPUID_EXT_SSSE3 | CPUID_EXT_SSE3,
+        .features[FEAT_8000_0001_EDX] =
+            CPUID_EXT2_LM | CPUID_EXT2_SYSCALL | CPUID_EXT2_NX,
+        .features[FEAT_8000_0001_ECX] =
+            CPUID_EXT3_LAHF_LM,
+        .xlevel = 0x80000008,
+        .model_id = "QEMU x86-64-v2 ABI",
+    },
+    {
+        /*
+         * Derived from Haswell-noTSX, minus
+         *   aes pcid erms invpcid tsc-deadline x2apic pclmulqdq
+         */
+        .name = "x86-64-abi3",
+        .level = 0xd,
+        .vendor = CPUID_VENDOR_INTEL,
+        .family = 6,
+        .model = 60,
+        .stepping = 1,
+        .features[FEAT_1_EDX] =
+            CPUID_VME | CPUID_SSE2 | CPUID_SSE | CPUID_FXSR | CPUID_MMX |
+            CPUID_CLFLUSH | CPUID_PSE36 | CPUID_PAT | CPUID_CMOV | CPUID_MCA |
+            CPUID_PGE | CPUID_MTRR | CPUID_SEP | CPUID_APIC | CPUID_CX8 |
+            CPUID_MCE | CPUID_PAE | CPUID_MSR | CPUID_TSC | CPUID_PSE |
+            CPUID_DE | CPUID_FP87,
+        .features[FEAT_1_ECX] =
+            CPUID_EXT_AVX | CPUID_EXT_XSAVE |
+            CPUID_EXT_POPCNT | CPUID_EXT_SSE42 |
+            CPUID_EXT_SSE41 | CPUID_EXT_CX16 | CPUID_EXT_SSSE3 |
+            CPUID_EXT_SSE3 |
+            CPUID_EXT_FMA | CPUID_EXT_MOVBE |
+            CPUID_EXT_F16C | CPUID_EXT_RDRAND,
+        .features[FEAT_8000_0001_EDX] =
+            CPUID_EXT2_LM | CPUID_EXT2_RDTSCP | CPUID_EXT2_NX |
+            CPUID_EXT2_SYSCALL,
+        .features[FEAT_8000_0001_ECX] =
+            CPUID_EXT3_ABM | CPUID_EXT3_LAHF_LM,
+        .features[FEAT_7_0_EBX] =
+            CPUID_7_0_EBX_FSGSBASE | CPUID_7_0_EBX_BMI1 |
+            CPUID_7_0_EBX_AVX2 | CPUID_7_0_EBX_SMEP |
+            CPUID_7_0_EBX_BMI2,
+        .features[FEAT_XSAVE] =
+            CPUID_XSAVE_XSAVEOPT,
+        .features[FEAT_6_EAX] =
+            CPUID_6_EAX_ARAT,
+        .xlevel = 0x80000008,
+        .model_id = "QEMU x86-64-v3 ABI",
+    },
+
+    {
+        /*
+         * Derived from Skylake-Server-noTSX-IBRS, minus:
+         *  spec-ctrl
+         */
+        .name = "x86-64-abi4",
+        .level = 0xd,
+        .vendor = CPUID_VENDOR_INTEL,
+        .family = 6,
+        .model = 85,
+        .stepping = 4,
+        .features[FEAT_1_EDX] =
+            CPUID_VME | CPUID_SSE2 | CPUID_SSE | CPUID_FXSR | CPUID_MMX |
+            CPUID_CLFLUSH | CPUID_PSE36 | CPUID_PAT | CPUID_CMOV | CPUID_MCA |
+            CPUID_PGE | CPUID_MTRR | CPUID_SEP | CPUID_APIC | CPUID_CX8 |
+            CPUID_MCE | CPUID_PAE | CPUID_MSR | CPUID_TSC | CPUID_PSE |
+            CPUID_DE | CPUID_FP87,
+        .features[FEAT_1_ECX] =
+            CPUID_EXT_AVX | CPUID_EXT_XSAVE | CPUID_EXT_AES |
+            CPUID_EXT_POPCNT | CPUID_EXT_X2APIC | CPUID_EXT_SSE42 |
+            CPUID_EXT_SSE41 | CPUID_EXT_CX16 | CPUID_EXT_SSSE3 |
+            CPUID_EXT_PCLMULQDQ | CPUID_EXT_SSE3 |
+            CPUID_EXT_TSC_DEADLINE_TIMER | CPUID_EXT_FMA | CPUID_EXT_MOVBE |
+            CPUID_EXT_PCID | CPUID_EXT_F16C | CPUID_EXT_RDRAND,
+        .features[FEAT_8000_0001_EDX] =
+            CPUID_EXT2_LM | CPUID_EXT2_PDPE1GB | CPUID_EXT2_RDTSCP |
+            CPUID_EXT2_NX | CPUID_EXT2_SYSCALL,
+        .features[FEAT_8000_0001_ECX] =
+            CPUID_EXT3_ABM | CPUID_EXT3_LAHF_LM | CPUID_EXT3_3DNOWPREFETCH,
+        .features[FEAT_7_0_EBX] =
+            CPUID_7_0_EBX_FSGSBASE | CPUID_7_0_EBX_BMI1 |
+            CPUID_7_0_EBX_AVX2 | CPUID_7_0_EBX_SMEP |
+            CPUID_7_0_EBX_BMI2 | CPUID_7_0_EBX_ERMS | CPUID_7_0_EBX_INVPCID |
+            CPUID_7_0_EBX_RDSEED | CPUID_7_0_EBX_ADX |
+            CPUID_7_0_EBX_SMAP | CPUID_7_0_EBX_CLWB |
+            CPUID_7_0_EBX_AVX512F | CPUID_7_0_EBX_AVX512DQ |
+            CPUID_7_0_EBX_AVX512BW | CPUID_7_0_EBX_AVX512CD |
+            CPUID_7_0_EBX_AVX512VL,
+        .features[FEAT_7_0_ECX] =
+            CPUID_7_0_ECX_PKU,
+        .features[FEAT_XSAVE] =
+            CPUID_XSAVE_XSAVEOPT | CPUID_XSAVE_XSAVEC |
+            CPUID_XSAVE_XGETBV1,
+        .features[FEAT_6_EAX] =
+            CPUID_6_EAX_ARAT,
+        .xlevel = 0x80000008,
+        .model_id = "QEMU x86-64-v4 ABI",
+    },
+
     {
         .name = "qemu64",
         .level = 0xd,
-- 
2.31.1

[PATCH v3 3/4] scripts: helper to generate x86_64 CPU ABI compat info

[Daniel P. Berrangé]

This script is what is used to generate the docs data table in:

  docs/system/cpu-models-x86-abi.csv

It can be useful to run if adding new CPU models / versions and
the csv needs updating.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 scripts/cpu-x86-uarch-abi.py | 194 +++++++++++++++++++++++++++++++++++
 1 file changed, 194 insertions(+)
 create mode 100644 scripts/cpu-x86-uarch-abi.py

diff --git a/scripts/cpu-x86-uarch-abi.py b/scripts/cpu-x86-uarch-abi.py
new file mode 100644
index 0000000000..08acc52a81
--- /dev/null
+++ b/scripts/cpu-x86-uarch-abi.py
@@ -0,0 +1,194 @@
+#!/usr/bin/python3
+#
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# A script to generate a CSV file showing the x86_64 ABI
+# compatibility levels for each CPU model.
+#
+
+from qemu import qmp
+import sys
+
+if len(sys.argv) != 1:
+    print("syntax: %s QMP-SOCK\n\n" % __file__ +
+          "Where QMP-SOCK points to a QEMU process such as\n\n" +
+          " # qemu-system-x86_64 -qmp unix:/tmp/qmp,server,nowait " +
+          "-display none -accel kvm", file=sys.stderr)
+    sys.exit(1)
+
+# Mandatory CPUID features for each microarch ABI level
+levels = [
+    [ # x86-64 baseline
+        "cmov",
+        "cx8",
+        "fpu",
+        "fxsr",
+        "mmx",
+        "syscall",
+        "sse",
+        "sse2",
+    ],
+    [ # x86-64-v2
+        "cx16",
+        "lahf-lm",
+        "popcnt",
+        "pni",
+        "sse4.1",
+        "sse4.2",
+        "ssse3",
+    ],
+    [ # x86-64-v3
+        "avx",
+        "avx2",
+        "bmi1",
+        "bmi2",
+        "f16c",
+        "fma",
+        "abm",
+        "movbe",
+    ],
+    [ # x86-64-v4
+        "avx512f",
+        "avx512bw",
+        "avx512cd",
+        "avx512dq",
+        "avx512vl",
+    ],
+]
+
+# Assumes externally launched process such as
+#
+#   qemu-system-x86_64 -qmp unix:/tmp/qmp,server,nowait -display none -accel kvm
+#
+# Note different results will be obtained with TCG, as
+# TCG masks out certain features otherwise present in
+# the CPU model definitions, as does KVM.
+
+
+sock = sys.argv[1]
+cmd = sys.argv[2]
+shell = qmp.QEMUMonitorProtocol(sock)
+shell.connect()
+
+models = shell.cmd("query-cpu-definitions")
+
+# These QMP props don't correspond to CPUID fatures
+# so ignore them
+skip = [
+    "family",
+    "min-level",
+    "min-xlevel",
+    "vendor",
+    "model",
+    "model-id",
+    "stepping",
+]
+
+names = []
+
+for model in models["return"]:
+    if "alias-of" in model:
+        continue
+    names.append(model["name"])
+
+models = {}
+
+for name in sorted(names):
+    cpu = shell.cmd("query-cpu-model-expansion",
+                     { "type": "static",
+                       "model": { "name": name }})
+
+    got = {}
+    for (feature, present) in cpu["return"]["model"]["props"].items():
+        if present and feature not in skip:
+            got[feature] = True
+
+    if name in ["host", "max", "base"]:
+        continue
+
+    models[name] = {
+        # Dict of all present features in this CPU model
+        "features": got,
+
+        # Whether each x86-64 ABI level is satisfied
+        "levels": [False, False, False, False],
+
+        # Number of extra CPUID features compared to the x86-64 ABI level
+        "distance":[-1, -1, -1, -1],
+
+        # CPUID features present in model, but not in ABI level
+        "delta":[[], [], [], []],
+
+        # CPUID features in ABI level but not present in model
+        "missing": [[], [], [], []],
+    }
+
+
+# Calculate whether the CPU models satisfy each ABI level
+for name in models.keys():
+    for level in range(len(levels)):
+        got = set(models[name]["features"])
+        want = set(levels[level])
+        missing = want - got
+        match = True
+        if len(missing) > 0:
+            match = False
+        models[name]["levels"][level] = match
+        models[name]["missing"][level] = missing
+
+# Cache list of CPU models satisfying each ABI level
+abi_models = [
+    [],
+    [],
+    [],
+    [],
+]
+
+for name in models.keys():
+    for level in range(len(levels)):
+        if models[name]["levels"][level]:
+            abi_models[level].append(name)
+
+
+for level in range(len(abi_models)):
+    # Find the union of features in all CPU models satisfying this ABI
+    allfeatures = {}
+    for name in abi_models[level]:
+        for feat in models[name]["features"]:
+            allfeatures[feat] = True
+
+    # Find the intersection of features in all CPU models satisfying this ABI
+    commonfeatures = []
+    for feat in allfeatures:
+        present = True
+        for name in models.keys():
+            if not models[name]["levels"][level]:
+                continue
+            if feat not in models[name]["features"]:
+                present = False
+        if present:
+            commonfeatures.append(feat)
+
+    # Determine how many extra features are present compared to the lowest
+    # common denominator
+    for name in models.keys():
+        if not models[name]["levels"][level]:
+            continue
+
+        delta = set(models[name]["features"].keys()) - set(commonfeatures)
+        models[name]["distance"][level] = len(delta)
+        models[name]["delta"][level] = delta
+
+def print_uarch_abi_csv():
+    print("# Automatically generated from '%s'" % __file__)
+    print("Model,baseline,v2,v3,v4")
+    for name in models.keys():
+        print(name, end="")
+        for level in range(len(levels)):
+            if models[name]["levels"][level]:
+                print(",✅", end="")
+            else:
+                print(",", end="")
+        print()
+
+print_uarch_abi_csv()
-- 
2.31.1

[PATCH v3 4/4] NOT FOR MERGE target/i386: use x86-64-abi1 CPU model as default on x86_64

[Daniel P. Berrangé]

The only differences between x86-64-abi1 and qemu64 is that the former
does not have the 'vme' or 'svm' flags.

In practice I don't think we should make this change, because it doesn't
especially add any value as-is. The only possible case is around 'svm'
because KVM already masks that feature, but TCG allows it. Thus using
x86-64-abi1 would mean that KVM and TCG expose a more consistent feature
set.

Also note that while libvirt can cope with default CPUs changing now,
it can't with the default CPU being a model name that it does not
already know about.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 hw/i386/pc_piix.c | 3 +++
 hw/i386/pc_q35.c  | 3 +++
 target/i386/cpu.h | 2 +-
 3 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
index 30b8bd6ea9..8f23bad647 100644
--- a/hw/i386/pc_piix.c
+++ b/hw/i386/pc_piix.c
@@ -444,6 +444,9 @@ static void pc_i440fx_5_2_machine_options(MachineClass *m)
     m->is_default = false;
     compat_props_add(m->compat_props, hw_compat_5_2, hw_compat_5_2_len);
     compat_props_add(m->compat_props, pc_compat_5_2, pc_compat_5_2_len);
+#ifdef TARGET_X86_64
+    m->default_cpu_type = X86_CPU_TYPE_NAME("qemu64");
+#endif
 }
 
 DEFINE_I440FX_MACHINE(v5_2, "pc-i440fx-5.2", NULL,
diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
index 46a0f196f4..a684c1db3b 100644
--- a/hw/i386/pc_q35.c
+++ b/hw/i386/pc_q35.c
@@ -372,6 +372,9 @@ static void pc_q35_5_2_machine_options(MachineClass *m)
     m->alias = NULL;
     compat_props_add(m->compat_props, hw_compat_5_2, hw_compat_5_2_len);
     compat_props_add(m->compat_props, pc_compat_5_2, pc_compat_5_2_len);
+#ifdef TARGET_X86_64
+    m->default_cpu_type = X86_CPU_TYPE_NAME("qemu64");
+#endif
 }
 
 DEFINE_Q35_MACHINE(v5_2, "pc-q35-5.2", NULL,
diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index ac3abea97c..9b895a63c7 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -1999,7 +1999,7 @@ uint64_t cpu_get_tsc(CPUX86State *env);
 #define CPU_RESOLVING_TYPE TYPE_X86_CPU
 
 #ifdef TARGET_X86_64
-#define TARGET_DEFAULT_CPU_TYPE X86_CPU_TYPE_NAME("qemu64")
+#define TARGET_DEFAULT_CPU_TYPE X86_CPU_TYPE_NAME("x86-64-abi1")
 #else
 #define TARGET_DEFAULT_CPU_TYPE X86_CPU_TYPE_NAME("qemu32")
 #endif
-- 
2.31.1

Re: [PATCH v3 3/4] scripts: helper to generate x86_64 CPU ABI compat info

[Eduardo Habkost]

On Mon, Jun 07, 2021 at 02:58:42PM +0100, Daniel P. Berrangé wrote:
[...]
> +# Assumes externally launched process such as
> +#
> +#   qemu-system-x86_64 -qmp unix:/tmp/qmp,server,nowait -display none -accel kvm
> +#
> +# Note different results will be obtained with TCG, as
> +# TCG masks out certain features otherwise present in
> +# the CPU model definitions, as does KVM.
> +
> +
> +sock = sys.argv[1]
> +cmd = sys.argv[2]
> +shell = qmp.QEMUMonitorProtocol(sock)
> +shell.connect()
> +
> +models = shell.cmd("query-cpu-definitions")

I would make the script launch QEMU itself, using the QEMUMachine
class.

But life is too short for polishing a quick utility script, so:

Acked-by: Eduardo Habkost <ehabkost@redhat.com>

I plan to queue this once I review the rest of the series.

-- 
Eduardo

Re: [PATCH v3 1/4] docs: add a table showing x86-64 ABI compatibility levels

[Eduardo Habkost]

On Mon, Jun 07, 2021 at 02:58:40PM +0100, Daniel P. Berrangé wrote:
> It is useful to know which CPUs satisfy each x86-64 ABI
> compatibility level, when dealing with guest OS that require
> something newer than the baseline ABI.
> 
> These ABI levels are defined in:
> 
>   https://gitlab.com/x86-psABIs/x86-64-ABI/
> 
> and supported by GCC, Clang, glibc and more.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>

Queued, thanks!

-- 
Eduardo

Re: [PATCH v3 3/4] scripts: helper to generate x86_64 CPU ABI compat info

[Eduardo Habkost]

On Mon, Jun 07, 2021 at 02:11:44PM -0300, Eduardo Habkost wrote:
> On Mon, Jun 07, 2021 at 02:58:42PM +0100, Daniel P. Berrangé wrote:
> [...]
> > +# Assumes externally launched process such as
> > +#
> > +#   qemu-system-x86_64 -qmp unix:/tmp/qmp,server,nowait -display none -accel kvm
> > +#
> > +# Note different results will be obtained with TCG, as
> > +# TCG masks out certain features otherwise present in
> > +# the CPU model definitions, as does KVM.
> > +
> > +
> > +sock = sys.argv[1]
> > +cmd = sys.argv[2]
> > +shell = qmp.QEMUMonitorProtocol(sock)
> > +shell.connect()
> > +
> > +models = shell.cmd("query-cpu-definitions")
> 
> I would make the script launch QEMU itself, using the QEMUMachine
> class.
> 
> But life is too short for polishing a quick utility script, so:
> 
> Acked-by: Eduardo Habkost <ehabkost@redhat.com>
> 
> I plan to queue this once I review the rest of the series.

I'm queueing patch 1/4 and this one.

I'm not sure about the lowest common denominator approach for
generating the CPU models in patch 2/4, but I need to read your
the cover letter more carefully, to be able to comment.

-- 
Eduardo

Re: [PATCH v3 0/4] target/i386/cpu: introduce new CPU models for x86-64 ABI levels

[Eduardo Habkost]

On Mon, Jun 07, 2021 at 02:58:39PM +0100, Daniel P. Berrangé wrote:
> This series is motivated by this blog that describes how RHEL-9
> will recommend use of the x86-64-v2 microarchitectural ABI level:
> 
>   https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level/
> 
> The implication of compiling code with -march=x86-64-v2 is that
> this code will no longer be guaranteed to be runnable on a
> number of the CPU models exposed by the x86_64 target emulator,
> most notably qemu64 which is the default.
> 
> This series is not proposing to change the QEMU default CPU model
> for x86_64 target. I show how this is can trivially be done in
> patch 3, but not suggesting that we actually do that, as upstream
> is quite conservative in dropping support for old host hardware.
> 
> New CPU models
> ==============
> 
> It is instead exploring the possibilities of defining new CPU
> models in QEMU to closely match the x86-64 uarch ABI levels in
> a relatively vendor agnostic manner. This could be used by
> downstream vendors who wish to target specific uarch ABI levels
> in custom machine types.
> 
> One of the nice things about "qemu64" is that its naming presents
> it as effectively being a vendor-neutral model (if we ignore that
> vendor=AMD is in fact reported in CPUID).
> 
> If we look at the feature set fo x86-64-v2 ABI, we see that the
> QEMU "Nehalem" model is the closest match. This is also happens
> to be runnable on AMD Opteron G4/G5 and EPYC hosts. None the less,
> the use of an Intel specific CPU model name on an AMD host feels
> uncomfortable.
> 
> Vendor neutral naming
> =====================
> 
> The idea behind this series is thus to introduce new CPU model
> names with vendor neutral naming, to more directly correlate
> with defined x86-64 uarch ABI levels. We don't want to just
> invent CPUs with a completely arbitrary set of CPU features as
> history has shown that brings its own problems. eg a guest
> uses features A and B, but only does CPUID checks for existence
> of feature B, assuming that B implies A.
> 
> The specification for x86-64 ABI levels uses x86-64-vNN naming
> but this clashes with QEMU's use of "vNN" for versioning. I
> felt it would be confusing to end up with CPU model names
> like  "x86-64-v1-v1". Thus I've used an "-abiNNN" suffix
> instead. Also note I have an "-abi1" suffix here for the
> baseline. Arguably we could just drop the suffix entirely for
> the baseline.
> 
> A further note is that we could have used "qemu64" as the
> naming prefix, eg qemu64-abi2, qemu64-abi3, etc. Alot of
> people hold negative opinions of the qemu64 model in general
> though, so I felt a clean break with the past might be
> desirable, even though the x86-64-abi1 CPU  model is effectively
> identical to qemu64.
> 
> Runnability of new models
> =========================
> 
> The goal of the x86-64-abiNNN CPU models is that they should be
> runnable on any physical host which supports the CPUIDs features
> for that uarch ABI level. It is hard to figure out what exact
> set of CPUID features we should report. The uarch ABI document
> only specifies the minimum expectation, but we can't define a
> CPU in QEMU using only the minimum set as that would omit
> countless important features.
> 

Do you have a list of features that were not in the ABI document
but were included in the CPU models you've added?  What exactly
make them important enough for us, but not important enough for
the ABI level specification writers?


In patch 2/3 you wrote:

| Historically we've found that defining CPU models with an arbitrary
| combination of CPU features can be problematic, as some guest OS
| will not check all features they use, and instead assume that if
| they see feature "XX", then "YY" will always exist. This is reasonable
| in bare metal, but subject to breakage in virtualization.

Do we know how likely that is?  Any examples where this has
happened?

What if we simply used the raw list of features from the
specification, add those as CPU models experimentally, and let
people report issues if they find them?  Too risky?  Not useful
in practice?

Maybe we could provide both a "x86-64-abiNN" and a
"x86-64-abiNN-minimal" CPU model.  We could even use the -minimal
CPU model definition to write a test case that ensure the
algorithm you describe below is always being followed.  The name
should make it clear that the CPU model is more conservative but
less likely to be useful.


> Thus to determine the feature set to use in x86-64-abiNNN CPU
> models, this series used a script (see patch 4) which did the
> following:
> 
>   * Find list of current CPU models that satisfy ABI NN
>   * From this list, calculate the lowest common denominator (LCD)
>     of CPUID features
>   * From the LCD, find the existing CPU model that has the fewest
>     extra features over the LCD.
>   * Use that closest match, minus the extra features
>     to define x86-64-abiNN
> 
> This approach works well for the baseline ABI, and level 2 ABI
> 
> For ABI level 3 there is a bit of a larger divergance in feature
> sets, because the LCD between Intel and EPYC models is relatively
> large. We omit aes pcid erms invpcid tsc-deadline x2apic pclmulqdq
> 
> For ABI level 4 the divergance is small. I believe this is a
> deceptive situation that arises because currently only Intel
> CPUs in QEMU are able to satisfy ABI level 4 currently. None
> of the EPYC CPUs defined in QEMU are able to satisfy it.
> 
> I'm concerned that if future EPYC CPU generations are added to
> QEMU which *do* indeed satisfy x86-64 v4, then QEMU's x86-64-abi4
> CPUID may prove to be too greedy and thus be unable to run on
> the future CPUs. In theory we can address this through use of
> CPU versioning, but ideally we would not have to deal with that.

If no AMD CPU model satisfy v4 yet, maybe we should avoid adding
any feature that is not present in any known AMD CPUs, unless
they are already required by the ABI level specification.
Otherwise, the list is likely to contain Intel features that AMD
has no plans to implement at all.

Or maybe we should just refrain from adding a x86-64-abi4 CPU
model until we have a CPU model from AMD that satisfies v4.

> 
> Nested virt caveats
> ===================
> 
> A further caveat is that none of these generic models define
> any VMX feature set. At the very least I believe this means
> that nested virt can't live migrate, but maybe this breaks
> ability to use nested virt entirely. I'm unclear what the
> best approach is to deal with this, that would let the user
> do both "-cpu x86-64-abi2,svm=on" and "-cpu x86-64-abi2,vmx=on"

I don't get this part.  Why can't the CPU model include a VMX
feature set that is the LCD of the VMX-capable CPUs that satisfy
abiNN?  The VMX feature set is supposed to be just a default when
using vmx=on.

> 
> Side channel mitigation caveats
> ===============================
> 
> Finally none of the x86-64-abiNN models include any of the
> features related to CPU side channel mitigations. We can't
> add them without sacrificing the ability to run on some
> physical hardware that would otherwise satisfy the uarch
> ABI level.
> 
> This makes me a little uneasy. One of the main downsides
> of using "qemu64" is that it leaves guests vulnerable to the
> side channel attacks. Introducing new general purpose CPUs
> that still have this flaw feels undesirable. I don't know
> whethe to prioritize safety, while sacrificing runability.

That's the biggest caveat for me.

I believe the default should prioritize safety.  I believe you
implicitly established that runnability is not an absolute
requirement when you added features that are required by the ABI
level specification.


> 
> Crypto accelerator caveats
> ==========================
> 
> Similarly I'm not a huge fan of leaving out the "aes"
> instruction for accelerated crypto, as missing "aes" is
> also one of the key factors in making qemu64 a bad choice.
> 
> If we include 'aes' in x86-64-abi2, then we loose support
> for Nehalem hosts.
> 
> If we include 'aes' in x86-64-abi3 then we further loose
> support for Dhyana hosts (an EPYC derived CPU).
> 
> Wrap up
> =======
> 
> Overall this series shows some CPU models for QEMU that
> can map to each x86-64 ABI level, but I'm unsure whether
> it is wise to actually go forward with this idea given
> the various caveats.
> 
> I think they'll mostly only be useful as a built-in default
> for machine types. If anyone is going to explicitly specify
> a CPU model they'll almost always be better off picking a
> vendor specific model or host passthrough. There can be
> cases where people have a mix of AMD and Intel bare metal
> machines they need portability across but this feel rare.

I would prefer to address the problem of poor defaults the
following way:

- We should start requiring management software to pass a explicit
  live-migration-policy argument if live migration needs to work
  (after documenting that and waiting for the deprecation period,
  of course).
- If live migration is not required, we can default to host
  passthrough.
- If live migration is required, we should stop providing a
  default CPU model, and requiring it to be configured
  explicitly.

QEMU doesn't have enough information to choose a good CPU model
default when live migration is required.  Even if we add the
x86-64-abiNN CPU models in this series, we have no way to decide
if abi1, abi2, abi3, or abi4 is a better default, do we?

That doesn't mean the CPU models you are adding aren't useful.
They may still be useful for management software or for
downstream vendors to choose their own defaults.

> 
> Whatever else comes of this series, at least patch 1 should
> be useful, giving people an idea of what ABI levels each
> QEMU CPU can satisfy in the documentation.
> 
> Changed in v3:
> 
>  - Rebased to resolve conflicts with git master
> 
> Changed in v2:
> 
>  - Don't document any of the CPU aliases, since they vary per
>    machine type, only the concrete types.
>  - Formally add the script for re-generating the docs
> 
> Daniel P. Berrangé (4):
>   docs: add a table showing x86-64 ABI compatibility levels
>   target/i386: define CPU models to model x86-64 ABI levels
>   scripts: helper to generate x86_64 CPU ABI compat info
>   NOT FOR MERGE target/i386: use x86-64-abi1 CPU model as default on
>     x86_64
> 
>  MAINTAINERS                        |   2 +-
>  docs/system/cpu-models-x86-abi.csv |  75 +++++++++++
>  docs/system/cpu-models-x86.rst.inc |  22 ++++
>  hw/i386/pc_piix.c                  |   3 +
>  hw/i386/pc_q35.c                   |   3 +
>  scripts/cpu-x86-uarch-abi.py       | 194 +++++++++++++++++++++++++++++
>  target/i386/cpu.c                  | 156 +++++++++++++++++++++++
>  target/i386/cpu.h                  |   2 +-
>  8 files changed, 455 insertions(+), 2 deletions(-)
>  create mode 100644 docs/system/cpu-models-x86-abi.csv
>  create mode 100644 scripts/cpu-x86-uarch-abi.py
> 
> -- 
> 2.31.1
> 
> 

-- 
Eduardo

Re: [PATCH v3 0/4] target/i386/cpu: introduce new CPU models for x86-64 ABI levels

[Daniel P. Berrangé]

On Mon, Jun 07, 2021 at 06:33:10PM -0300, Eduardo Habkost wrote:
> On Mon, Jun 07, 2021 at 02:58:39PM +0100, Daniel P. Berrangé wrote:
> > This series is motivated by this blog that describes how RHEL-9
> > will recommend use of the x86-64-v2 microarchitectural ABI level:
> > 
> >   https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level/
> > 
> > The implication of compiling code with -march=x86-64-v2 is that
> > this code will no longer be guaranteed to be runnable on a
> > number of the CPU models exposed by the x86_64 target emulator,
> > most notably qemu64 which is the default.
> > 
> > This series is not proposing to change the QEMU default CPU model
> > for x86_64 target. I show how this is can trivially be done in
> > patch 3, but not suggesting that we actually do that, as upstream
> > is quite conservative in dropping support for old host hardware.
> > 
> > New CPU models
> > ==============
> > 
> > It is instead exploring the possibilities of defining new CPU
> > models in QEMU to closely match the x86-64 uarch ABI levels in
> > a relatively vendor agnostic manner. This could be used by
> > downstream vendors who wish to target specific uarch ABI levels
> > in custom machine types.
> > 
> > One of the nice things about "qemu64" is that its naming presents
> > it as effectively being a vendor-neutral model (if we ignore that
> > vendor=AMD is in fact reported in CPUID).
> > 
> > If we look at the feature set fo x86-64-v2 ABI, we see that the
> > QEMU "Nehalem" model is the closest match. This is also happens
> > to be runnable on AMD Opteron G4/G5 and EPYC hosts. None the less,
> > the use of an Intel specific CPU model name on an AMD host feels
> > uncomfortable.
> > 
> > Vendor neutral naming
> > =====================
> > 
> > The idea behind this series is thus to introduce new CPU model
> > names with vendor neutral naming, to more directly correlate
> > with defined x86-64 uarch ABI levels. We don't want to just
> > invent CPUs with a completely arbitrary set of CPU features as
> > history has shown that brings its own problems. eg a guest
> > uses features A and B, but only does CPUID checks for existence
> > of feature B, assuming that B implies A.
> > 
> > The specification for x86-64 ABI levels uses x86-64-vNN naming
> > but this clashes with QEMU's use of "vNN" for versioning. I
> > felt it would be confusing to end up with CPU model names
> > like  "x86-64-v1-v1". Thus I've used an "-abiNNN" suffix
> > instead. Also note I have an "-abi1" suffix here for the
> > baseline. Arguably we could just drop the suffix entirely for
> > the baseline.
> > 
> > A further note is that we could have used "qemu64" as the
> > naming prefix, eg qemu64-abi2, qemu64-abi3, etc. Alot of
> > people hold negative opinions of the qemu64 model in general
> > though, so I felt a clean break with the past might be
> > desirable, even though the x86-64-abi1 CPU  model is effectively
> > identical to qemu64.
> > 
> > Runnability of new models
> > =========================
> > 
> > The goal of the x86-64-abiNNN CPU models is that they should be
> > runnable on any physical host which supports the CPUIDs features
> > for that uarch ABI level. It is hard to figure out what exact
> > set of CPUID features we should report. The uarch ABI document
> > only specifies the minimum expectation, but we can't define a
> > CPU in QEMU using only the minimum set as that would omit
> > countless important features.
> > 
> 
> Do you have a list of features that were not in the ABI document
> but were included in the CPU models you've added?  What exactly
> make them important enough for us, but not important enough for
> the ABI level specification writers?

The ABI specification isn't trying to define a real CPU, so it
doesn't define a full set of features. It is merely setting a
minimum bar, against which vendor specific CPUs are evaluated
for compliance. IOW, there are multiple AMD and Intel CPUs that
satisfy x86-64-abi1. They will all have have a distinct set of
features, but share a common core. The ABI spec is fairly
focused on the various SIMD matrix instructions in particular.

In this QEMU patch meanwhile, we are attempting to define a
real CPU, so need to spec more than just the core featureset.

> In patch 2/3 you wrote:
> 
> | Historically we've found that defining CPU models with an arbitrary
> | combination of CPU features can be problematic, as some guest OS
> | will not check all features they use, and instead assume that if
> | they see feature "XX", then "YY" will always exist. This is reasonable
> | in bare metal, but subject to breakage in virtualization.
> 
> Do we know how likely that is?  Any examples where this has
> happened?

I don't have specific bugs off hand. I just know that the traditional
way libvirt implemented host-model by taking a base model and turning
on/off arbitrary features has been a source of bugs.

> What if we simply used the raw list of features from the
> specification, add those as CPU models experimentally, and let
> people report issues if they find them?  Too risky?  Not useful
> in practice?

Taking ABI level 1.

The combination of level 0 and level 1 ABI specifies
these features:

{'popcnt', 'sse4.1', 'sse', 'mmx', 'ssse3', 'pni', 'cx16', 'fpu', 'sse2', 'sse4.2', 'cx8', 'syscall', 'fxsr', 'cmov', 'lahf-lm'}

The Nehalem CPU which I used as a basis for this level provides 

{'hypervisor', 'pge', 'mmx', 'ssse3', 'kvm-nopiodelay', 'pse', 'de', 'sep', 'tsc', 'nx', 'popcnt', 'mtrr', 'kvm-steal-time', 'apic', 'sse2', 'clflush', 'fxsr', 'sse4.1', 'sse', 'cx16', 'syscall', 'x2apic', 'kvmclock-stable-bit', 'pni', 'kvm-pv-eoi', 'kvmclock', 'kvm-asyncpf', 'vme', 'mce', 'msr', 'mca', 'sse4.2', 'fpu', 'cx8', 'lm', 'pse36', 'pat', 'cmov', 'lahf-lm', 'pae'}

So if we removed all features except those explicitly documented
by the ABI spec we loose:

{'hypervisor', 'pge', 'pse', 'kvm-nopiodelay', 'de', 'sep', 'tsc', 'nx', 'mtrr', 'kvm-steal-time', 'apic', 'clflush', 'x2apic', 'kvmclock-stable-bit', 'kvm-pv-eoi', 'kvmclock', 'kvm-asyncpf', 'vme', 'mce', 'msr', 'mca', 'pse36', 'lm', 'pat', 'pae'}

Many (most) of these are pretty important features IMHO. I don't
think you can define a useful CPU without them.


> Maybe we could provide both a "x86-64-abiNN" and a
> "x86-64-abiNN-minimal" CPU model.  We could even use the -minimal
> CPU model definition to write a test case that ensure the
> algorithm you describe below is always being followed.  The name
> should make it clear that the CPU model is more conservative but
> less likely to be useful.

I don't think the minimal is usable in practice.

> > Thus to determine the feature set to use in x86-64-abiNNN CPU
> > models, this series used a script (see patch 4) which did the
> > following:
> > 
> >   * Find list of current CPU models that satisfy ABI NN
> >   * From this list, calculate the lowest common denominator (LCD)
> >     of CPUID features
> >   * From the LCD, find the existing CPU model that has the fewest
> >     extra features over the LCD.
> >   * Use that closest match, minus the extra features
> >     to define x86-64-abiNN
> > 
> > This approach works well for the baseline ABI, and level 2 ABI
> > 
> > For ABI level 3 there is a bit of a larger divergance in feature
> > sets, because the LCD between Intel and EPYC models is relatively
> > large. We omit aes pcid erms invpcid tsc-deadline x2apic pclmulqdq
> > 
> > For ABI level 4 the divergance is small. I believe this is a
> > deceptive situation that arises because currently only Intel
> > CPUs in QEMU are able to satisfy ABI level 4 currently. None
> > of the EPYC CPUs defined in QEMU are able to satisfy it.
> > 
> > I'm concerned that if future EPYC CPU generations are added to
> > QEMU which *do* indeed satisfy x86-64 v4, then QEMU's x86-64-abi4
> > CPUID may prove to be too greedy and thus be unable to run on
> > the future CPUs. In theory we can address this through use of
> > CPU versioning, but ideally we would not have to deal with that.
> 
> If no AMD CPU model satisfy v4 yet, maybe we should avoid adding
> any feature that is not present in any known AMD CPUs, unless
> they are already required by the ABI level specification.
> Otherwise, the list is likely to contain Intel features that AMD
> has no plans to implement at all.
> 
> Or maybe we should just refrain from adding a x86-64-abi4 CPU
> model until we have a CPU model from AMD that satisfies v4.

I'd probably go for the latter option and just ignore v4.


> > Nested virt caveats
> > ===================
> > 
> > A further caveat is that none of these generic models define
> > any VMX feature set. At the very least I believe this means
> > that nested virt can't live migrate, but maybe this breaks
> > ability to use nested virt entirely. I'm unclear what the
> > best approach is to deal with this, that would let the user
> > do both "-cpu x86-64-abi2,svm=on" and "-cpu x86-64-abi2,vmx=on"
> 
> I don't get this part.  Why can't the CPU model include a VMX
> feature set that is the LCD of the VMX-capable CPUs that satisfy
> abiNN?  The VMX feature set is supposed to be just a default when
> using vmx=on.

Maybe I'm mis-interpreting things. I didn't think it made sense
to define all the VMX capabilities, if the CPU was later intended
to be used with svm=on, but it sounds like you're saying this is
harmless.

> > 
> > Side channel mitigation caveats
> > ===============================
> > 
> > Finally none of the x86-64-abiNN models include any of the
> > features related to CPU side channel mitigations. We can't
> > add them without sacrificing the ability to run on some
> > physical hardware that would otherwise satisfy the uarch
> > ABI level.
> > 
> > This makes me a little uneasy. One of the main downsides
> > of using "qemu64" is that it leaves guests vulnerable to the
> > side channel attacks. Introducing new general purpose CPUs
> > that still have this flaw feels undesirable. I don't know
> > whethe to prioritize safety, while sacrificing runability.
> 
> That's the biggest caveat for me.
> 
> I believe the default should prioritize safety.  I believe you
> implicitly established that runnability is not an absolute
> requirement when you added features that are required by the ABI
> level specification.


The challenge with the side channel mitigations is that the flags
to be exposed are rather vendor specific, so I don't see a way
to define a single CPU that works for both Intel and AMD hosts
and includes the side channel mods.

> > Crypto accelerator caveats
> > ==========================
> > 
> > Similarly I'm not a huge fan of leaving out the "aes"
> > instruction for accelerated crypto, as missing "aes" is
> > also one of the key factors in making qemu64 a bad choice.
> > 
> > If we include 'aes' in x86-64-abi2, then we loose support
> > for Nehalem hosts.
> > 
> > If we include 'aes' in x86-64-abi3 then we further loose
> > support for Dhyana hosts (an EPYC derived CPU).
> > 
> > Wrap up
> > =======
> > 
> > Overall this series shows some CPU models for QEMU that
> > can map to each x86-64 ABI level, but I'm unsure whether
> > it is wise to actually go forward with this idea given
> > the various caveats.
> > 
> > I think they'll mostly only be useful as a built-in default
> > for machine types. If anyone is going to explicitly specify
> > a CPU model they'll almost always be better off picking a
> > vendor specific model or host passthrough. There can be
> > cases where people have a mix of AMD and Intel bare metal
> > machines they need portability across but this feel rare.
> 
> I would prefer to address the problem of poor defaults the
> following way:
> 
> - We should start requiring management software to pass a explicit
>   live-migration-policy argument if live migration needs to work
>   (after documenting that and waiting for the deprecation period,
>   of course).
> - If live migration is not required, we can default to host
>   passthrough.
> - If live migration is required, we should stop providing a
>   default CPU model, and requiring it to be configured
>   explicitly.

I don't think this will make a significant difference, because
libvirt will end up saying live migration is required for
everything it starts, unless libvirt has in turn been told by
a mgmt app that it doesn't need live migration. The biggest
mgmt apps I know (oVirt, OpenStack, KubeVirt) all want live
migration in general. There may be some cases where live
mig is not desired, but I expect it would be 10% or less
of deployments.

> QEMU doesn't have enough information to choose a good CPU model
> default when live migration is required.  Even if we add the
> x86-64-abiNN CPU models in this series, we have no way to decide
> if abi1, abi2, abi3, or abi4 is a better default, do we?

Nope, QEMU doesn't have enough info to choose. The only way I
can see is if the OS vendor decides what their baseline is
going to be. eg Fedora explicitly supports all CPUs right back
to first generation x86_64 chips, so to provide runnability on
all possible Fedora hosts, we need qemu64.  In RHEL-9, it was
decided to compile everything with march=x86_64-abi1, so we
can guarantee that x86_64-abi1 is always runnable on any RHEL9
host, and thus can use that instead of qemu64. 


FWIW, I'm totally fine with saying that this concept of defining
CPUs for x86_64 ABI levels is a failed experiment.

To me the real killers are the lack of side channel mitigations
and lack of AES accelerator instructions. Without those, it just
doesn't feel like we're offering a big enough step up over the
qemu64 CPU. We can't add the side channels as it feels too
vendor specific, and we can't add the AES accel, because there
are physical CPUs satisfying abi1, which don't have AES.

At least the documentation of ABI compatibiltiy levels is useful
though.

Given that my x86_64-abi1 CPU is a copy of Nehalem, if RHEL9
did want a newer CPU default, it could just pick Nehalem as
that.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v3 3/4] scripts: helper to generate x86_64 CPU ABI compat info

[Daniel P. Berrangé]

On Mon, Jun 07, 2021 at 02:58:42PM +0100, Daniel P. Berrangé wrote:
> This script is what is used to generate the docs data table in:
> 
>   docs/system/cpu-models-x86-abi.csv
> 
> It can be useful to run if adding new CPU models / versions and
> the csv needs updating.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  scripts/cpu-x86-uarch-abi.py | 194 +++++++++++++++++++++++++++++++++++
>  1 file changed, 194 insertions(+)
>  create mode 100644 scripts/cpu-x86-uarch-abi.py

Sorry I messed up just before sending this when I deleted some
code and incorrectly fixed up argv handling. Since you mentioned
you've queued it, it needs two changes

> 
> diff --git a/scripts/cpu-x86-uarch-abi.py b/scripts/cpu-x86-uarch-abi.py
> new file mode 100644
> index 0000000000..08acc52a81
> --- /dev/null
> +++ b/scripts/cpu-x86-uarch-abi.py
> @@ -0,0 +1,194 @@
> +#!/usr/bin/python3
> +#
> +# SPDX-License-Identifier: GPL-2.0-or-later
> +#
> +# A script to generate a CSV file showing the x86_64 ABI
> +# compatibility levels for each CPU model.
> +#
> +
> +from qemu import qmp
> +import sys
> +
> +if len(sys.argv) != 1:

s/1/2/

> +    print("syntax: %s QMP-SOCK\n\n" % __file__ +
> +          "Where QMP-SOCK points to a QEMU process such as\n\n" +
> +          " # qemu-system-x86_64 -qmp unix:/tmp/qmp,server,nowait " +
> +          "-display none -accel kvm", file=sys.stderr)
> +    sys.exit(1)
> +
> +# Mandatory CPUID features for each microarch ABI level
> +levels = [
> +    [ # x86-64 baseline
> +        "cmov",
> +        "cx8",
> +        "fpu",
> +        "fxsr",
> +        "mmx",
> +        "syscall",
> +        "sse",
> +        "sse2",
> +    ],
> +    [ # x86-64-v2
> +        "cx16",
> +        "lahf-lm",
> +        "popcnt",
> +        "pni",
> +        "sse4.1",
> +        "sse4.2",
> +        "ssse3",
> +    ],
> +    [ # x86-64-v3
> +        "avx",
> +        "avx2",
> +        "bmi1",
> +        "bmi2",
> +        "f16c",
> +        "fma",
> +        "abm",
> +        "movbe",
> +    ],
> +    [ # x86-64-v4
> +        "avx512f",
> +        "avx512bw",
> +        "avx512cd",
> +        "avx512dq",
> +        "avx512vl",
> +    ],
> +]
> +
> +# Assumes externally launched process such as
> +#
> +#   qemu-system-x86_64 -qmp unix:/tmp/qmp,server,nowait -display none -accel kvm
> +#
> +# Note different results will be obtained with TCG, as
> +# TCG masks out certain features otherwise present in
> +# the CPU model definitions, as does KVM.
> +
> +
> +sock = sys.argv[1]
> +cmd = sys.argv[2]

Delete this line since sys.argv[2] is not required

> +shell = qmp.QEMUMonitorProtocol(sock)
> +shell.connect()
> +
> +models = shell.cmd("query-cpu-definitions")


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|