[PATCH RFC 0/4] target/i386/cpu: introduce new CPU models for x86-64 ABI levels

[PATCH RFC 0/4] target/i386/cpu: introduce new CPU models for x86-64 ABI levels

[Daniel P. Berrangé]

This series is motivated by this blog that describes how RHEL-9
will recommend use of the x86-64-v2 microarchitectural ABI level:

  https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-l=
inux-9-for-the-x86-64-v2-microarchitecture-level/

The implication of compiling code with -march=3Dx86-64-v2 is that
this code will no longer be guaranteed to be runnable on a
number of the CPU models exposed by the x86_64 target emulator,
most notably qemu64 which is the default.

This series is not proposing to change the QEMU default CPU model
for x86_64 target. I show how this is can trivially be done in
patch 3, but not suggesting that we actually do that, as upstream
is quite conservative in dropping support for old host hardware.

New CPU models
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

It is instead exploring the possibilities of defining new CPU
models in QEMU to closely match the x86-64 uarch ABI levels in
a relatively vendor agnostic manner. This could be used by
downstream vendors who wish to target specific uarch ABI levels
in custom machine types.

One of the nice things about "qemu64" is that its naming presents
it as effectively being a vendor-neutral model (if we ignore that
vendor=3DAMD is in fact reported in CPUID).

If we look at the feature set fo x86-64-v2 ABI, we see that the
QEMU "Nehalem" model is the closest match. This is also happens
to be runnable on AMD Opteron G4/G5 and EPYC hosts. None the less,
the use of an Intel specific CPU model name on an AMD host feels
uncomfortable.

Vendor neutral naming
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

The idea behind this series is thus to introduce new CPU model
names with vendor neutral naming, to more directly correlate
with defined x86-64 uarch ABI levels. We don't want to just
invent CPUs with a completely arbitrary set of CPU features as
history has shown that brings its own problems. eg a guest
uses features A and B, but only does CPUID checks for existence
of feature B, assuming that B implies A.

The specification for x86-64 ABI levels uses x86-64-vNN naming
but this clashes with QEMU's use of "vNN" for versioning. I
felt it would be confusing to end up with CPU model names
like  "x86-64-v1-v1". Thus I've used an "-abiNNN" suffix
instead. Also note I have an "-abi1" suffix here for the
baseline. Arguably we could just drop the suffix entirely for
the baseline.

A further note is that we could have used "qemu64" as the
naming prefix, eg qemu64-abi2, qemu64-abi3, etc. Alot of
people hold negative opinions of the qemu64 model in general
though, so I felt a clean break with the past might be
desirable, even though the x86-64-abi1 CPU  model is effectively
identical to qemu64.

Runnability of new models
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

The goal of the x86-64-abiNNN CPU models is that they should be
runnable on any physical host which supports the CPUIDs features
for that uarch ABI level. It is hard to figure out what exact
set of CPUID features we should report. The uarch ABI document
only specifies the minimum expectation, but we can't define a
CPU in QEMU using only the minimum set as that would omit
countless important features.

Thus to determine the feature set to use in x86-64-abiNNN CPU
models, this series used a script (see patch 4) which did the
following:

  * Find list of current CPU models that satisfy ABI NN
  * From this list, calculate the lowest common denominator (LCD)
    of CPUID features
  * From the LCD, find the existing CPU model that has the fewest
    extra features over the LCD.
  * Use that closest match, minus the extra features
    to define x86-64-abiNN

This approach works well for the baseline ABI, and level 2 ABI

For ABI level 3 there is a bit of a larger divergance in feature
sets, because the LCD between Intel and EPYC models is relatively
large. We omit aes pcid erms invpcid tsc-deadline x2apic pclmulqdq

For ABI level 4 the divergance is small. I believe this is a
deceptive situation that arises because currently only Intel
CPUs in QEMU are able to satisfy ABI level 4 currently. None
of the EPYC CPUs defined in QEMU are able to satisfy it.

I'm concerned that if future EPYC CPU generations are added to
QEMU which *do* indeed satisfy x86-64 v4, then QEMU's x86-64-abi4
CPUID may prove to be too greedy and thus be unable to run on
the future CPUs. In theory we can address this through use of
CPU versioning, but ideally we would not have to deal with that.

Nested virt caveats
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

A further caveat is that none of these generic models define
any VMX feature set. At the very least I believe this means
that nested virt can't live migrate, but maybe this breaks
ability to use nested virt entirely. I'm unclear what the
best approach is to deal with this, that would let the user
do both "-cpu x86-64-abi2,svm=3Don" and "-cpu x86-64-abi2,vmx=3Don"

Side channel mitigation caveats
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D

Finally none of the x86-64-abiNN models include any of the
features related to CPU side channel mitigations. We can't
add them without sacrificing the ability to run on some
physical hardware that would otherwise satisfy the uarch
ABI level.

This makes me a little uneasy. One of the main downsides
of using "qemu64" is that it leaves guests vulnerable to the
side channel attacks. Introducing new general purpose CPUs
that still have this flaw feels undesirable. I don't know
whethe to prioritize safety, while sacrificing runability.

Crypto accelerator caveats
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

Similarly I'm not a huge fan of leaving out the "aes"
instruction for accelerated crypto, as missing "aes" is
also one of the key factors in making qemu64 a bad choice.

If we include 'aes' in x86-64-abi2, then we loose support
for Nehalem hosts.

If we include 'aes' in x86-64-abi3 then we further loose
support for Dhyana hosts (an EPYC derived CPU).

Wrap up
=3D=3D=3D=3D=3D=3D=3D

Overall this series shows some CPU models for QEMU that
can map to each x86-64 ABI level, but I'm unsure whether
it is wise to actually go forward with this idea given
the various caveats above.

I think they'll mostly only be useful as a built-in default
for machine types. If anyone is going to explicitly specify
a CPU model they'll almost always be better off picking a
vendor specific model or host passthrough. There can be
cases where people have a mix of AMD and Intel bare metal
machines they need portability across but this feel rare.

Whatever else comes of this series, at least patch 1 should
be useful, giving people an idea of what ABI levels each
QEMU CPU can satisfy in the documentation.

Daniel P. Berrang=C3=A9 (4):
  docs: add a table showing x86-64 ABI compatibility levels
  target/i386: define CPU models to model x86-64 ABI levels
  NOT FOR MERGE target/i386: use x86-64-abi1 CPU model as default on
    x86_64
  NOT FOR MERGE: script for CPU model stuff related to x86-64 ABI levels

 MAINTAINERS                        |   2 +-
 docs/system/cpu-models-x86-abi.csv | 129 +++++++++++++++++++
 docs/system/cpu-models-x86.rst.inc |  18 +++
 hw/i386/pc_piix.c                  |   3 +
 hw/i386/pc_q35.c                   |   3 +
 scripts/cpu-x86-uarch-abi.py       | 194 +++++++++++++++++++++++++++++
 target/i386/cpu.c                  | 156 +++++++++++++++++++++++
 target/i386/cpu.h                  |   2 +-
 8 files changed, 505 insertions(+), 2 deletions(-)
 create mode 100644 docs/system/cpu-models-x86-abi.csv
 create mode 100644 scripts/cpu-x86-uarch-abi.py

--=20
2.29.2

[PATCH RFC 1/4] docs: add a table showing x86-64 ABI compatibility levels

[Daniel P. Berrangé]

It is useful to know which CPUs satisfy each x86-64 ABI
compatibility level, when dealing with guest OS that require
something newer than the baseline ABI.

These ABI levels are defined in:

  https://gitlab.com/x86-psABIs/x86-64-ABI/

and supported by GCC, CLang, GLibC and more.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 MAINTAINERS                        |   2 +-
 docs/system/cpu-models-x86-abi.csv | 121 +++++++++++++++++++++++++++++
 docs/system/cpu-models-x86.rst.inc |  18 +++++
 3 files changed, 140 insertions(+), 1 deletion(-)
 create mode 100644 docs/system/cpu-models-x86-abi.csv

diff --git a/MAINTAINERS b/MAINTAINERS
index fbb228ef2b..bb8d60c458 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -344,7 +344,7 @@ F: tests/tcg/i386/
 F: tests/tcg/x86_64/
 F: hw/i386/
 F: disas/i386.c
-F: docs/system/cpu-models-x86.rst.inc
+F: docs/system/cpu-models-x86*
 T: git https://gitlab.com/ehabkost/qemu.git x86-next
 
 Xtensa TCG CPUs
diff --git a/docs/system/cpu-models-x86-abi.csv b/docs/system/cpu-models-x86-abi.csv
new file mode 100644
index 0000000000..4565e6a535
--- /dev/null
+++ b/docs/system/cpu-models-x86-abi.csv
@@ -0,0 +1,121 @@
+Model,baseline,v2,v3,v4
+486,,,,
+486-v1,,,,
+Broadwell,✅,✅,✅,
+Broadwell-IBRS,✅,✅,✅,
+Broadwell-noTSX,✅,✅,✅,
+Broadwell-noTSX-IBRS,✅,✅,✅,
+Broadwell-v1,✅,✅,✅,
+Broadwell-v2,✅,✅,✅,
+Broadwell-v3,✅,✅,✅,
+Broadwell-v4,✅,✅,✅,
+Cascadelake-Server,✅,✅,✅,✅
+Cascadelake-Server-noTSX,✅,✅,✅,✅
+Cascadelake-Server-v1,✅,✅,✅,✅
+Cascadelake-Server-v2,✅,✅,✅,✅
+Cascadelake-Server-v3,✅,✅,✅,✅
+Cascadelake-Server-v4,✅,✅,✅,✅
+Conroe,✅,,,
+Conroe-v1,✅,,,
+Cooperlake,✅,✅,✅,✅
+Cooperlake-v1,✅,✅,✅,✅
+Denverton,✅,✅,,
+Denverton-v1,✅,✅,,
+Denverton-v2,✅,✅,,
+Dhyana,✅,✅,✅,
+Dhyana-v1,✅,✅,✅,
+EPYC,✅,✅,✅,
+EPYC-IBPB,✅,✅,✅,
+EPYC-Rome,✅,✅,✅,
+EPYC-Rome-v1,✅,✅,✅,
+EPYC-v1,✅,✅,✅,
+EPYC-v2,✅,✅,✅,
+EPYC-v3,✅,✅,✅,
+Haswell,✅,✅,✅,
+Haswell-IBRS,✅,✅,✅,
+Haswell-noTSX,✅,✅,✅,
+Haswell-noTSX-IBRS,✅,✅,✅,
+Haswell-v1,✅,✅,✅,
+Haswell-v2,✅,✅,✅,
+Haswell-v3,✅,✅,✅,
+Haswell-v4,✅,✅,✅,
+Icelake-Client,✅,✅,✅,
+Icelake-Client-noTSX,✅,✅,✅,
+Icelake-Client-v1,✅,✅,✅,
+Icelake-Client-v2,✅,✅,✅,
+Icelake-Server,✅,✅,✅,✅
+Icelake-Server-noTSX,✅,✅,✅,✅
+Icelake-Server-v1,✅,✅,✅,✅
+Icelake-Server-v2,✅,✅,✅,✅
+Icelake-Server-v3,✅,✅,✅,✅
+Icelake-Server-v4,✅,✅,✅,✅
+IvyBridge,✅,✅,,
+IvyBridge-IBRS,✅,✅,,
+IvyBridge-v1,✅,✅,,
+IvyBridge-v2,✅,✅,,
+KnightsMill,✅,✅,✅,
+KnightsMill-v1,✅,✅,✅,
+Nehalem,✅,✅,,
+Nehalem-IBRS,✅,✅,,
+Nehalem-v1,✅,✅,,
+Nehalem-v2,✅,✅,,
+Opteron_G1,✅,,,
+Opteron_G1-v1,✅,,,
+Opteron_G2,✅,,,
+Opteron_G2-v1,✅,,,
+Opteron_G3,✅,,,
+Opteron_G3-v1,✅,,,
+Opteron_G4,✅,✅,,
+Opteron_G4-v1,✅,✅,,
+Opteron_G5,✅,✅,,
+Opteron_G5-v1,✅,✅,,
+Penryn,✅,,,
+Penryn-v1,✅,,,
+SandyBridge,✅,✅,,
+SandyBridge-IBRS,✅,✅,,
+SandyBridge-v1,✅,✅,,
+SandyBridge-v2,✅,✅,,
+Skylake-Client,✅,✅,✅,
+Skylake-Client-IBRS,✅,✅,✅,
+Skylake-Client-noTSX-IBRS,✅,✅,✅,
+Skylake-Client-v1,✅,✅,✅,
+Skylake-Client-v2,✅,✅,✅,
+Skylake-Client-v3,✅,✅,✅,
+Skylake-Server,✅,✅,✅,✅
+Skylake-Server-IBRS,✅,✅,✅,✅
+Skylake-Server-noTSX-IBRS,✅,✅,✅,✅
+Skylake-Server-v1,✅,✅,✅,✅
+Skylake-Server-v2,✅,✅,✅,✅
+Skylake-Server-v3,✅,✅,✅,✅
+Skylake-Server-v4,✅,✅,✅,✅
+Snowridge,✅,✅,,
+Snowridge-v1,✅,✅,,
+Snowridge-v2,✅,✅,,
+Westmere,✅,✅,,
+Westmere-IBRS,✅,✅,,
+Westmere-v1,✅,✅,,
+Westmere-v2,✅,✅,,
+athlon,,,,
+athlon-v1,,,,
+core2duo,✅,,,
+core2duo-v1,✅,,,
+coreduo,,,,
+coreduo-v1,,,,
+kvm32,,,,
+kvm32-v1,,,,
+kvm64,✅,,,
+kvm64-v1,✅,,,
+n270,,,,
+n270-v1,,,,
+pentium,,,,
+pentium-v1,,,,
+pentium2,,,,
+pentium2-v1,,,,
+pentium3,,,,
+pentium3-v1,,,,
+phenom,✅,,,
+phenom-v1,✅,,,
+qemu32,,,,
+qemu32-v1,,,,
+qemu64,✅,,,
+qemu64-v1,✅,,,
diff --git a/docs/system/cpu-models-x86.rst.inc b/docs/system/cpu-models-x86.rst.inc
index 9a2327828e..b964b29c78 100644
--- a/docs/system/cpu-models-x86.rst.inc
+++ b/docs/system/cpu-models-x86.rst.inc
@@ -39,6 +39,24 @@ CPU, as they would with "Host passthrough", but gives much of the
 benefit of passthrough, while making live migration safe.
 
 
+ABI compatibility levels for CPU models
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The x86_64 architecture has a number of `ABI compatibility levels`_
+defined. Traditionally most operating systems and toolchains would
+only target the original baseline ABI. It is expected that in
+future OS and toolchains are likely to target newer ABIs. The
+following table illustrates which ABI compatibility levels can be
+satisfied by the QEMU CPU models
+
+.. _ABI compatibility levels: https://gitlab.com/x86-psABIs/x86-64-ABI/
+
+.. csv-table:: x86-64 ABI compatibility levels
+   :file: cpu-models-x86-abi.csv
+   :widths: 40,15,15,15,15
+   :header-rows: 1
+
+
 Preferred CPU models for Intel x86 hosts
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-- 
2.29.2

[PATCH RFC 2/4] target/i386: define CPU models to model x86-64 ABI levels

[Daniel P. Berrangé]

To paraphrase:

  https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level/

In 2020, AMD, Intel, Red Hat, and SUSE worked together to define
three microarchitecture levels on top of the historical x86-64
baseline:

  * x86-64:    original x86_64 baseline instruction set
  * x86-64-v2: vector instructions up to Streaming SIMD
               Extensions 4.2 (SSE4.2)  and Supplemental
	       Streaming SIMD Extensions 3 (SSSE3), the
	       POPCNT instruction, and CMPXCHG16B
  * x86-64-v3: vector instructions up to AVX2, MOVBE,
               and additional bit-manipulation instructions.
  * x86-64-v4: vector instructions from some of the
               AVX-512 variants.

This list of features is defined in the doc at:

  https://gitlab.com/x86-psABIs/x86-64-ABI/

QEMU has historically defaulted to the "qemu64" CPU model on
x86_64 targets, which is approximately the x86-64 baseline
ABI, with 'SVM' added.

It is thought it might be desirable if QEMU could provide CPU models
that closely correspond to the ABI levels, while offering portability
across the maximum number of physical CPUs.

Historically we've found that defining CPU models with an arbitrary
combination of CPU features can be problematic, as some guest OS
will not check all features they use, and instead assume that if
they see feature "XX", then "YY" will always exist. This is reasonable
in bare metal, but subject to breakage in virtualization.

Thus in defining the CPI models for the ABI levels, this patch attempted
to base them off an existing CPU model.

While each x86-64-abiNNN has a designated vendor, they are designed
to be vendor agnostic models. ie they are capable of running on any
AMD or Intel physical CPU model that satisfies the ABI level. eg
althgouh the x86-64-abi2 model is based on Nehalem, it should be
able to run guests on an Opteron_G4/G5/EPYC host, since those CPUs
support the features required by the x86-64 v2 ABI.

More precisely the models were defined as:

 * x86-64-abi1: close match for Opteron_G1, minus
                vme
 * x86-64-abi2: perfect match for Nehalem
 * x86-64-abi3: close match of Haswell-noTSX, minus
                aes pcid erms invpcid tsc-deadline
		x2apic pclmulqdq
 * x86-64-abi4: close match of Skylake-Server-noTSX-IBRS, minus
                spec-ctrl

None of the CPU models declare any VMX/SVM features. This would
make them unable to support nested virtualization with live
migration.

Given their vendor agnostic design, these CPU models are primarily
though to useful as the default CPU model for machine types. QEMU
upstream is quite conservative in mandating new hardware features,
but a downstream vendor may choose to mandate a newer x86-64 ABI
level for downstream only machine types.

Note that TCG is not capable of supporting the 2 newest ABI levels
currently:

* x86-64-abi3:

  CPUID.01H:ECX.fma [bit 12]
  CPUID.01H:ECX.avx [bit 28]
  CPUID.01H:ECX.f16c [bit 29]
  CPUID.07H:EBX.avx2 [bit 5]

* x86-64-abi4:

  CPUID.01H:ECX.pcid [bit 17]
  CPUID.01H:ECX.x2apic [bit 21]
  CPUID.01H:ECX.tsc-deadline [bit 24]
  CPUID.07H:EBX.invpcid [bit 10]
  CPUID.07H:EBX.avx512f [bit 16]
  CPUID.07H:EBX.avx512dq [bit 17]
  CPUID.07H:EBX.rdseed [bit 18]
  CPUID.07H:EBX.avx512cd [bit 28]
  CPUID.07H:EBX.avx512bw [bit 30]
  CPUID.07H:EBX.avx512vl [bit 31]
  CPUID.80000001H:ECX.3dnowprefetch [bit 8]
  CPUID.0DH:EAX.xsavec [bit 1]

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 docs/system/cpu-models-x86-abi.csv |   8 ++
 target/i386/cpu.c                  | 156 +++++++++++++++++++++++++++++
 2 files changed, 164 insertions(+)

diff --git a/docs/system/cpu-models-x86-abi.csv b/docs/system/cpu-models-x86-abi.csv
index 4565e6a535..d34d95d485 100644
--- a/docs/system/cpu-models-x86-abi.csv
+++ b/docs/system/cpu-models-x86-abi.csv
@@ -119,3 +119,11 @@ qemu32,,,,
 qemu32-v1,,,,
 qemu64,✅,,,
 qemu64-v1,✅,,,
+x86-64-abi1,✅,,,
+x86-64-abi1-v1,✅,,,
+x86-64-abi2,✅,✅,,
+x86-64-abi2-v1,✅,✅,,
+x86-64-abi3,✅,✅,✅,
+x86-64-abi3-v1,✅,✅,✅,
+x86-64-abi4,✅,✅,✅,✅
+x86-64-abi4-v1,✅,✅,✅,✅
diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index ae89024d36..87a775a5eb 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -1827,6 +1827,162 @@ static CPUCaches epyc_rome_cache_info = {
  */
 
 static X86CPUDefinition builtin_x86_defs[] = {
+    /*
+     * These first few CPU models are designed to satisfy the
+     * x86_64 ABI levels defined in:
+     *
+     *   https://gitlab.com/x86-psABIs/x86-64-ABI/
+     *
+     * They were constructed as follows:
+     *
+     *   - Find all the CPU models which can satisfy the ABI
+     *   - Calculate the lowest common denominator (LCD) of these
+     *     models' features
+     *   - Find the named model most closely matching the LCD
+     *   - Strip its features back to the LCD
+     *
+     * The above spec uses the "x86-64-vNN" naming convention.
+     * This clashes with the "vNN" suffix QEMU uses for versioning.
+     * Thus we use "abiNNN" as a suffix.
+     */
+    {
+        /*
+         * Derived from Opteron_G1, minus
+         *   vme
+         */
+        .name = "x86-64-abi1",
+        .level = 5,
+        .vendor = CPUID_VENDOR_AMD,
+        .family = 15,
+        .model = 6,
+        .stepping = 1,
+        .features[FEAT_1_EDX] =
+            CPUID_SSE2 | CPUID_SSE | CPUID_FXSR | CPUID_MMX |
+            CPUID_CLFLUSH | CPUID_PSE36 | CPUID_PAT | CPUID_CMOV | CPUID_MCA |
+            CPUID_PGE | CPUID_MTRR | CPUID_SEP | CPUID_APIC | CPUID_CX8 |
+            CPUID_MCE | CPUID_PAE | CPUID_MSR | CPUID_TSC | CPUID_PSE |
+            CPUID_DE | CPUID_FP87,
+        .features[FEAT_1_ECX] =
+            CPUID_EXT_SSE3,
+        .features[FEAT_8000_0001_EDX] =
+            CPUID_EXT2_LM | CPUID_EXT2_NX | CPUID_EXT2_SYSCALL,
+        .xlevel = 0x80000008,
+        .model_id = "QEMU x86-64 baseline ABI",
+    },
+    {
+        /* Derived from Nehalem */
+        .name = "x86-64-abi2",
+        .level = 11,
+        .vendor = CPUID_VENDOR_INTEL,
+        .family = 6,
+        .model = 26,
+        .stepping = 3,
+        .features[FEAT_1_EDX] =
+            CPUID_VME | CPUID_SSE2 | CPUID_SSE | CPUID_FXSR | CPUID_MMX |
+            CPUID_CLFLUSH | CPUID_PSE36 | CPUID_PAT | CPUID_CMOV | CPUID_MCA |
+            CPUID_PGE | CPUID_MTRR | CPUID_SEP | CPUID_APIC | CPUID_CX8 |
+            CPUID_MCE | CPUID_PAE | CPUID_MSR | CPUID_TSC | CPUID_PSE |
+            CPUID_DE | CPUID_FP87,
+        .features[FEAT_1_ECX] =
+            CPUID_EXT_POPCNT | CPUID_EXT_SSE42 | CPUID_EXT_SSE41 |
+            CPUID_EXT_CX16 | CPUID_EXT_SSSE3 | CPUID_EXT_SSE3,
+        .features[FEAT_8000_0001_EDX] =
+            CPUID_EXT2_LM | CPUID_EXT2_SYSCALL | CPUID_EXT2_NX,
+        .features[FEAT_8000_0001_ECX] =
+            CPUID_EXT3_LAHF_LM,
+        .xlevel = 0x80000008,
+        .model_id = "QEMU x86-64-v2 ABI",
+    },
+    {
+        /*
+         * Derived from Haswell-noTSX, minus
+         *   aes pcid erms invpcid tsc-deadline x2apic pclmulqdq
+         */
+        .name = "x86-64-abi3",
+        .level = 0xd,
+        .vendor = CPUID_VENDOR_INTEL,
+        .family = 6,
+        .model = 60,
+        .stepping = 1,
+        .features[FEAT_1_EDX] =
+            CPUID_VME | CPUID_SSE2 | CPUID_SSE | CPUID_FXSR | CPUID_MMX |
+            CPUID_CLFLUSH | CPUID_PSE36 | CPUID_PAT | CPUID_CMOV | CPUID_MCA |
+            CPUID_PGE | CPUID_MTRR | CPUID_SEP | CPUID_APIC | CPUID_CX8 |
+            CPUID_MCE | CPUID_PAE | CPUID_MSR | CPUID_TSC | CPUID_PSE |
+            CPUID_DE | CPUID_FP87,
+        .features[FEAT_1_ECX] =
+            CPUID_EXT_AVX | CPUID_EXT_XSAVE |
+            CPUID_EXT_POPCNT | CPUID_EXT_SSE42 |
+            CPUID_EXT_SSE41 | CPUID_EXT_CX16 | CPUID_EXT_SSSE3 |
+            CPUID_EXT_SSE3 |
+            CPUID_EXT_FMA | CPUID_EXT_MOVBE |
+            CPUID_EXT_F16C | CPUID_EXT_RDRAND,
+        .features[FEAT_8000_0001_EDX] =
+            CPUID_EXT2_LM | CPUID_EXT2_RDTSCP | CPUID_EXT2_NX |
+            CPUID_EXT2_SYSCALL,
+        .features[FEAT_8000_0001_ECX] =
+            CPUID_EXT3_ABM | CPUID_EXT3_LAHF_LM,
+        .features[FEAT_7_0_EBX] =
+            CPUID_7_0_EBX_FSGSBASE | CPUID_7_0_EBX_BMI1 |
+            CPUID_7_0_EBX_AVX2 | CPUID_7_0_EBX_SMEP |
+            CPUID_7_0_EBX_BMI2,
+        .features[FEAT_XSAVE] =
+            CPUID_XSAVE_XSAVEOPT,
+        .features[FEAT_6_EAX] =
+            CPUID_6_EAX_ARAT,
+        .xlevel = 0x80000008,
+        .model_id = "QEMU x86-64-v3 ABI",
+    },
+
+    {
+        /*
+         * Derived from Skylake-Server-noTSX-IBRS, minus:
+         *  spec-ctrl
+         */
+        .name = "x86-64-abi4",
+        .level = 0xd,
+        .vendor = CPUID_VENDOR_INTEL,
+        .family = 6,
+        .model = 85,
+        .stepping = 4,
+        .features[FEAT_1_EDX] =
+            CPUID_VME | CPUID_SSE2 | CPUID_SSE | CPUID_FXSR | CPUID_MMX |
+            CPUID_CLFLUSH | CPUID_PSE36 | CPUID_PAT | CPUID_CMOV | CPUID_MCA |
+            CPUID_PGE | CPUID_MTRR | CPUID_SEP | CPUID_APIC | CPUID_CX8 |
+            CPUID_MCE | CPUID_PAE | CPUID_MSR | CPUID_TSC | CPUID_PSE |
+            CPUID_DE | CPUID_FP87,
+        .features[FEAT_1_ECX] =
+            CPUID_EXT_AVX | CPUID_EXT_XSAVE | CPUID_EXT_AES |
+            CPUID_EXT_POPCNT | CPUID_EXT_X2APIC | CPUID_EXT_SSE42 |
+            CPUID_EXT_SSE41 | CPUID_EXT_CX16 | CPUID_EXT_SSSE3 |
+            CPUID_EXT_PCLMULQDQ | CPUID_EXT_SSE3 |
+            CPUID_EXT_TSC_DEADLINE_TIMER | CPUID_EXT_FMA | CPUID_EXT_MOVBE |
+            CPUID_EXT_PCID | CPUID_EXT_F16C | CPUID_EXT_RDRAND,
+        .features[FEAT_8000_0001_EDX] =
+            CPUID_EXT2_LM | CPUID_EXT2_PDPE1GB | CPUID_EXT2_RDTSCP |
+            CPUID_EXT2_NX | CPUID_EXT2_SYSCALL,
+        .features[FEAT_8000_0001_ECX] =
+            CPUID_EXT3_ABM | CPUID_EXT3_LAHF_LM | CPUID_EXT3_3DNOWPREFETCH,
+        .features[FEAT_7_0_EBX] =
+            CPUID_7_0_EBX_FSGSBASE | CPUID_7_0_EBX_BMI1 |
+            CPUID_7_0_EBX_AVX2 | CPUID_7_0_EBX_SMEP |
+            CPUID_7_0_EBX_BMI2 | CPUID_7_0_EBX_ERMS | CPUID_7_0_EBX_INVPCID |
+            CPUID_7_0_EBX_RDSEED | CPUID_7_0_EBX_ADX |
+            CPUID_7_0_EBX_SMAP | CPUID_7_0_EBX_CLWB |
+            CPUID_7_0_EBX_AVX512F | CPUID_7_0_EBX_AVX512DQ |
+            CPUID_7_0_EBX_AVX512BW | CPUID_7_0_EBX_AVX512CD |
+            CPUID_7_0_EBX_AVX512VL,
+        .features[FEAT_7_0_ECX] =
+            CPUID_7_0_ECX_PKU,
+        .features[FEAT_XSAVE] =
+            CPUID_XSAVE_XSAVEOPT | CPUID_XSAVE_XSAVEC |
+            CPUID_XSAVE_XGETBV1,
+        .features[FEAT_6_EAX] =
+            CPUID_6_EAX_ARAT,
+        .xlevel = 0x80000008,
+        .model_id = "QEMU x86-64-v4 ABI",
+    },
+
     {
         .name = "qemu64",
         .level = 0xd,
-- 
2.29.2

[PATCH RFC 3/4] NOT FOR MERGE target/i386: use x86-64-abi1 CPU model as default on x86_64

[Daniel P. Berrangé]

The only differences between x86-64-abi1 and qemu64 is that the former
does not have the 'vme' or 'svm' flags.

In practice I don't think we should make this change, because it doesn't
especially add any value as-is. The only possible case is around 'svm'
because KVM already masks that feature, but TCG allows it. Thus using
x86-64-abi1 would mean that KVM and TCG expose a more consistent feature
set.

Also note that while libvirt can cope with default CPUs changing now,
it can't with the default CPU being a model name that it does not
already know about.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 hw/i386/pc_piix.c | 3 +++
 hw/i386/pc_q35.c  | 3 +++
 target/i386/cpu.h | 2 +-
 3 files changed, 7 insertions(+), 1 deletion(-)

diff --git a/hw/i386/pc_piix.c b/hw/i386/pc_piix.c
index 6188c3e97e..c4c003599f 100644
--- a/hw/i386/pc_piix.c
+++ b/hw/i386/pc_piix.c
@@ -445,6 +445,9 @@ static void pc_i440fx_5_2_machine_options(MachineClass *m)
     m->is_default = false;
     compat_props_add(m->compat_props, hw_compat_5_2, hw_compat_5_2_len);
     compat_props_add(m->compat_props, pc_compat_5_2, pc_compat_5_2_len);
+#ifdef TARGET_X86_64
+    m->default_cpu_type = X86_CPU_TYPE_NAME("qemu64");
+#endif
 }
 
 DEFINE_I440FX_MACHINE(v5_2, "pc-i440fx-5.2", NULL,
diff --git a/hw/i386/pc_q35.c b/hw/i386/pc_q35.c
index 0a212443aa..606ac4f1f4 100644
--- a/hw/i386/pc_q35.c
+++ b/hw/i386/pc_q35.c
@@ -361,6 +361,9 @@ static void pc_q35_5_2_machine_options(MachineClass *m)
     m->alias = NULL;
     compat_props_add(m->compat_props, hw_compat_5_2, hw_compat_5_2_len);
     compat_props_add(m->compat_props, pc_compat_5_2, pc_compat_5_2_len);
+#ifdef TARGET_X86_64
+    m->default_cpu_type = X86_CPU_TYPE_NAME("qemu64");
+#endif
 }
 
 DEFINE_Q35_MACHINE(v5_2, "pc-q35-5.2", NULL,
diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index d23a5b340a..0a436b575f 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -1956,7 +1956,7 @@ uint64_t cpu_get_tsc(CPUX86State *env);
 #define CPU_RESOLVING_TYPE TYPE_X86_CPU
 
 #ifdef TARGET_X86_64
-#define TARGET_DEFAULT_CPU_TYPE X86_CPU_TYPE_NAME("qemu64")
+#define TARGET_DEFAULT_CPU_TYPE X86_CPU_TYPE_NAME("x86-64-abi1")
 #else
 #define TARGET_DEFAULT_CPU_TYPE X86_CPU_TYPE_NAME("qemu32")
 #endif
-- 
2.29.2

[PATCH RFC 4/4] NOT FOR MERGE: script for CPU model stuff related to x86-64 ABI levels

[Daniel P. Berrangé]

This script is something that I wrote in order to help with creation of
the first two patches. Since those patches are essentially static data
once created, I don't intend for this script to be called repeatedly
in future. I've just included here as a reference to show how I came
up with content.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
---
 scripts/cpu-x86-uarch-abi.py | 194 +++++++++++++++++++++++++++++++++++
 1 file changed, 194 insertions(+)
 create mode 100644 scripts/cpu-x86-uarch-abi.py

diff --git a/scripts/cpu-x86-uarch-abi.py b/scripts/cpu-x86-uarch-abi.py
new file mode 100644
index 0000000000..8ef6d3ea08
--- /dev/null
+++ b/scripts/cpu-x86-uarch-abi.py
@@ -0,0 +1,194 @@
+#!/usr/bin/python3
+
+from qemu import qmp
+import sys
+
+# Mandatory CPUID features for each microarch ABI level
+levels = [
+    [ # x86-64 baseline
+        "cmov",
+        "cx8",
+        "fpu",
+        "fxsr",
+        "mmx",
+        "syscall",
+        "sse",
+        "sse2",
+    ],
+    [ # x86-64-v2
+        "cx16",
+        "lahf-lm",
+        "popcnt",
+        "pni",
+        "sse4.1",
+        "sse4.2",
+        "ssse3",
+    ],
+    [ # x86-64-v3
+        "avx",
+        "avx2",
+        "bmi1",
+        "bmi2",
+        "f16c",
+        "fma",
+        "abm",
+        "movbe",
+    ],
+    [ # x86-64-v4
+        "avx512f",
+        "avx512bw",
+        "avx512cd",
+        "avx512dq",
+        "avx512vl",
+    ],
+]
+
+# Assumes externally launched process such as
+#
+#   qemu-system-x86_64 -qmp unix:/tmp/qmp,server,nowait -display none -accel kvm
+#
+# Note different results will be obtained with TCG, as
+# TCG masks out certain features otherwise present in
+# the CPU model definitions, as does KVM.
+
+
+sock = sys.argv[1]
+cmd = sys.argv[2]
+shell = qmp.QEMUMonitorProtocol(sock)
+shell.connect()
+
+models = shell.cmd("query-cpu-definitions")
+
+# These QMP props don't correspond to CPUID fatures
+# so ignore them
+skip = [
+    "family",
+    "min-level",
+    "min-xlevel",
+    "vendor",
+    "model",
+    "model-id",
+    "stepping",
+]
+
+names = [model["name"] for model in models["return"]]
+
+models = {}
+
+for name in sorted(names):
+    cpu = shell.cmd("query-cpu-model-expansion",
+                     { "type": "static",
+                       "model": { "name": name }})
+
+    got = {}
+    for (feature, present) in cpu["return"]["model"]["props"].items():
+        if present and feature not in skip:
+            got[feature] = True
+
+    if name in ["host", "max", "base"]:
+        continue
+
+    models[name] = {
+        # Dict of all present features in this CPU model
+        "features": got,
+
+        # Whether each x86-64 ABI level is satisfied
+        "levels": [False, False, False, False],
+
+        # Number of extra CPUID features compared to the x86-64 ABI level
+        "distance":[-1, -1, -1, -1],
+
+        # CPUID features present in model, but not in ABI level
+        "delta":[[], [], [], []],
+    }
+
+
+# Calculate whether the CPU models satisfy each ABI level
+for name in models.keys():
+    for level in range(len(levels)):
+        match = True
+        for feat in levels[level]:
+            if feat not in models[name]["features"]:
+                match = False
+        models[name]["levels"][level] = match
+
+# Cache list of CPU models satisfying each ABI level
+abi_models = [
+    [],
+    [],
+    [],
+    [],
+]
+
+for name in models.keys():
+    for level in range(len(levels)):
+        if models[name]["levels"][level]:
+            abi_models[level].append(name)
+
+
+for level in range(len(abi_models)):
+    # Find the union of features in all CPU models satisfying this ABI
+    allfeatures = {}
+    for name in abi_models[level]:
+        for feat in models[name]["features"]:
+            allfeatures[feat] = True
+
+    # Find the intersection of features in all CPU models satisfying this ABI
+    commonfeatures = []
+    for feat in allfeatures:
+        present = True
+        for name in models.keys():
+            if not models[name]["levels"][level]:
+                continue
+            if feat not in models[name]["features"]:
+                present = False
+        if present:
+            commonfeatures.append(feat)
+
+    # Determine how many extra features are present compared to the lowest
+    # common denominator
+    for name in models.keys():
+        if not models[name]["levels"][level]:
+            continue
+
+        delta = set(models[name]["features"].keys()) - set(commonfeatures)
+        models[name]["distance"][level] = len(delta)
+        models[name]["delta"][level] = delta
+
+def print_uarch_abi_csv():
+    print("Model,baseline,v2,v3,v4")
+    for name in models.keys():
+        print(name, end="")
+        for level in range(len(levels)):
+            if models[name]["levels"][level]:
+                print(",✅", end="")
+            else:
+                print(",", end="")
+        print()
+
+def find_closest_abi_models():
+    for level in range(len(abi_models)):
+        # Determine which CPU model(s) are the "best" match for the lowest
+        # common denominator. One of these will be used as the basis for
+        # defining the generic CPU model for this ABI level
+        distance = -1
+        for name in models.keys():
+            if not models[name]["levels"][level]:
+                continue
+
+            this = models[name]["distance"][level]
+            if distance == -1 or this < distance:
+                distance = this
+                best = name
+
+        for name in models:
+            if models[name]["distance"][level] == distance:
+                print("  >> Level %d match %s (distance %d)" % (
+                    level, name, models[name]["distance"][level]))
+                print("     Remove features: %s" %
+                      " ".join(models[name]["delta"][level]))
+
+if cmd == "abi-table":
+    print_uarch_abi_csv()
+elif cmd == "abi-model":
+    find_closest_abi_models()
-- 
2.29.2

Re: [PATCH RFC 1/4] docs: add a table showing x86-64 ABI compatibility levels

[Florian Weimer]

* Daniel P. Berrangé:

> and supported by GCC, CLang, GLibC and more.

Clang and glibc are the official spellings, I think.

> diff --git a/docs/system/cpu-models-x86-abi.csv b/docs/system/cpu-models-x86-abi.csv
> new file mode 100644
> index 0000000000..4565e6a535
> --- /dev/null
> +++ b/docs/system/cpu-models-x86-abi.csv

> +Icelake-Client,✅,✅,✅,
> +Icelake-Client-noTSX,✅,✅,✅,
> +Icelake-Client-v1,✅,✅,✅,
> +Icelake-Client-v2,✅,✅,✅,

Icelake Client supports x86-64-v4 according to Intel ARK and a quick
test on a reference system.  Have you defined it differently in QEMU?

> +KnightsMill,✅,✅,✅,
> +KnightsMill-v1,✅,✅,✅,

This one is correct.  Even though Knights Mill supports AVX-512, it does
not cover the variants that are considered definitive for x86-64-v4.

> +Skylake-Server,✅,✅,✅,✅
> +Skylake-Server-IBRS,✅,✅,✅,✅
> +Skylake-Server-noTSX-IBRS,✅,✅,✅,✅
> +Skylake-Server-v1,✅,✅,✅,✅
> +Skylake-Server-v2,✅,✅,✅,✅
> +Skylake-Server-v3,✅,✅,✅,✅
> +Skylake-Server-v4,✅,✅,✅,✅

This one is a little bit odd.  Skylake Xeons which are not Xeon Scalable
Processors exist, and they do not support x86-64-v4.  Is this again a
matter of different naming in QEMU?

Thanks,
Florian
-- 
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

Re: [PATCH RFC 1/4] docs: add a table showing x86-64 ABI compatibility levels

[Peter Maydell]

On Mon, 1 Feb 2021 at 15:39, Daniel P. Berrangé <berrange@redhat.com> wrote:
>
> It is useful to know which CPUs satisfy each x86-64 ABI
> compatibility level, when dealing with guest OS that require
> something newer than the baseline ABI.
>
> These ABI levels are defined in:
>
>   https://gitlab.com/x86-psABIs/x86-64-ABI/
>
> and supported by GCC, CLang, GLibC and more.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> +ABI compatibility levels for CPU models
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +The x86_64 architecture has a number of `ABI compatibility levels`_
> +defined. Traditionally most operating systems and toolchains would
> +only target the original baseline ABI. It is expected that in
> +future OS and toolchains are likely to target newer ABIs. The
> +following table illustrates which ABI compatibility levels can be
> +satisfied by the QEMU CPU models
> +
> +.. _ABI compatibility levels: https://gitlab.com/x86-psABIs/x86-64-ABI/
> +
> +.. csv-table:: x86-64 ABI compatibility levels
> +   :file: cpu-models-x86-abi.csv
> +   :widths: 40,15,15,15,15
> +   :header-rows: 1

Apart from the QEMU/KVM specific CPU models, why is this something
we should be documenting rather than, say, the people specifying
what the ABI compatiblity levels are ?

thanks
-- PMM

Re: [PATCH RFC 1/4] docs: add a table showing x86-64 ABI compatibility levels

[Daniel P. Berrangé]

On Mon, Feb 01, 2021 at 05:33:53PM +0100, Florian Weimer wrote:
> * Daniel P. Berrangé:
> 
> > and supported by GCC, CLang, GLibC and more.
> 
> Clang and glibc are the official spellings, I think.

Ok.

> > diff --git a/docs/system/cpu-models-x86-abi.csv b/docs/system/cpu-models-x86-abi.csv
> > new file mode 100644
> > index 0000000000..4565e6a535
> > --- /dev/null
> > +++ b/docs/system/cpu-models-x86-abi.csv
> 
> > +Icelake-Client,✅,✅,✅,
> > +Icelake-Client-noTSX,✅,✅,✅,
> > +Icelake-Client-v1,✅,✅,✅,
> > +Icelake-Client-v2,✅,✅,✅,
> 
> Icelake Client supports x86-64-v4 according to Intel ARK and a quick
> test on a reference system.  Have you defined it differently in QEMU?

QEMU's Icelake-Client CPU models appear to be missing most of the AVX-512
CPUIDs bits:

  https://gitlab.com/qemu-project/qemu/-/blob/master/target/i386/cpu.c#L3291

Compared to Icelake-Server which does have them:

  https://gitlab.com/qemu-project/qemu/-/blob/master/target/i386/cpu.c#L3409

I don't know why it is specified this way in QEMU. It could easily
be a bug in QEMU's definitions. Alternatively there might be a subset
of Icelake-Client SKUs which genuinely do lack these features, and
this influenced the decision to omit them from QEMU models.


> > +KnightsMill,✅,✅,✅,
> > +KnightsMill-v1,✅,✅,✅,
> 
> This one is correct.  Even though Knights Mill supports AVX-512, it does
> not cover the variants that are considered definitive for x86-64-v4.
> 
> > +Skylake-Server,✅,✅,✅,✅
> > +Skylake-Server-IBRS,✅,✅,✅,✅
> > +Skylake-Server-noTSX-IBRS,✅,✅,✅,✅
> > +Skylake-Server-v1,✅,✅,✅,✅
> > +Skylake-Server-v2,✅,✅,✅,✅
> > +Skylake-Server-v3,✅,✅,✅,✅
> > +Skylake-Server-v4,✅,✅,✅,✅
> 
> This one is a little bit odd.  Skylake Xeons which are not Xeon Scalable
> Processors exist, and they do not support x86-64-v4.  Is this again a
> matter of different naming in QEMU?

Most likely this is just a case of the QEMU Skylake-Server model being
written in terms of the most common SKUs, and ignoring the inconvenience
of certain SKUs lacking the features.


In general there are waaaay too many different variants of Intel CPUs for
QEMU to provide a named model to cope with every scenario. So the QEMU
models are always an approximation of what exists in the silicon.

If there are places where we've made bad mistake, we do now have the
ability to do CPU versioning. So in theory we could introduce a new
Skylake-Server-v5 which removes the AVX512 stuff if there's a genuine
problem with some variants of silicon not supporting it. Alternatively
people with such hosts can just use an older named model like
Skylake-Client.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH RFC 1/4] docs: add a table showing x86-64 ABI compatibility levels

[Daniel P. Berrangé]

On Mon, Feb 01, 2021 at 04:53:03PM +0000, Peter Maydell wrote:
> On Mon, 1 Feb 2021 at 15:39, Daniel P. Berrangé <berrange@redhat.com> wrote:
> >
> > It is useful to know which CPUs satisfy each x86-64 ABI
> > compatibility level, when dealing with guest OS that require
> > something newer than the baseline ABI.
> >
> > These ABI levels are defined in:
> >
> >   https://gitlab.com/x86-psABIs/x86-64-ABI/
> >
> > and supported by GCC, CLang, GLibC and more.
> >
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > +ABI compatibility levels for CPU models
> > +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> > +
> > +The x86_64 architecture has a number of `ABI compatibility levels`_
> > +defined. Traditionally most operating systems and toolchains would
> > +only target the original baseline ABI. It is expected that in
> > +future OS and toolchains are likely to target newer ABIs. The
> > +following table illustrates which ABI compatibility levels can be
> > +satisfied by the QEMU CPU models
> > +
> > +.. _ABI compatibility levels: https://gitlab.com/x86-psABIs/x86-64-ABI/
> > +
> > +.. csv-table:: x86-64 ABI compatibility levels
> > +   :file: cpu-models-x86-abi.csv
> > +   :widths: 40,15,15,15,15
> > +   :header-rows: 1
> 
> Apart from the QEMU/KVM specific CPU models, why is this something
> we should be documenting rather than, say, the people specifying
> what the ABI compatiblity levels are ?

QEMU's named CPU models are not a perfect match for features in the
real world silicon. So even if someone has a Skylake Server CPU with
feature X, this doesn't mean QEMU's definition of the Skylake-Server
CPU model is guaranteed to have feature X.  So we want to document
the compatibility of the exact CPU models that QEMU has defined.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH RFC 1/4] docs: add a table showing x86-64 ABI compatibility levels

[Eduardo Habkost]

On Mon, Feb 01, 2021 at 03:36:03PM +0000, Daniel P. Berrangé wrote:
> It is useful to know which CPUs satisfy each x86-64 ABI
> compatibility level, when dealing with guest OS that require
> something newer than the baseline ABI.
> 
> These ABI levels are defined in:
> 
>   https://gitlab.com/x86-psABIs/x86-64-ABI/
> 
> and supported by GCC, CLang, GLibC and more.
> 
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
[...]
> diff --git a/docs/system/cpu-models-x86-abi.csv b/docs/system/cpu-models-x86-abi.csv
> new file mode 100644
> index 0000000000..4565e6a535
> --- /dev/null
> +++ b/docs/system/cpu-models-x86-abi.csv
> @@ -0,0 +1,121 @@
> +Model,baseline,v2,v3,v4
> +486,,,,
> +486-v1,,,,
> +Broadwell,✅,✅,✅,
> +Broadwell-IBRS,✅,✅,✅,
> +Broadwell-noTSX,✅,✅,✅,
> +Broadwell-noTSX-IBRS,✅,✅,✅,
> +Broadwell-v1,✅,✅,✅,
> +Broadwell-v2,✅,✅,✅,
> +Broadwell-v3,✅,✅,✅,
> +Broadwell-v4,✅,✅,✅,

Unversioned names like "Broadwell" are machine-type-dependent
aliases.  I don't think they should be present in the table.

Models with suffixes like -IBRS, -noTSX, etc. are also aliases to
specific versions.  Maybe they could appear in the table for
completeness, but I'm not sure.

-- 
Eduardo

Re: [PATCH RFC 1/4] docs: add a table showing x86-64 ABI compatibility levels

[Florian Weimer]

* Peter Maydell:

> Apart from the QEMU/KVM specific CPU models, why is this something
> we should be documenting rather than, say, the people specifying
> what the ABI compatiblity levels are ?

The psABI only cares about userspace, and the specification there
deliberately excludes CPU features used in cryptography blocks, and
features that seem unstable and likely to be disabled in firmware or
future microcode updates.  This seemed a good trade-off for the psABI
because in userspace, run-time detection is still available, to access
additional CPU features not available as part of the target x86-64 level
at build time.

This doesn't really work for virtualization, which also has to decide
what to do with CPU features not relevant to userspace.  And a decision
to exclude features is final in the sense that guests cannot use the
feature at all because run-time detection shows it is as unavailable.

This is why a 1:1 copy of the userspace levels to virtualized machine
models is not possible, I think.

Thanks,
Florian
-- 
Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

Re: [PATCH RFC 1/4] docs: add a table showing x86-64 ABI compatibility levels

[David Edmondson]

On Monday, 2021-02-01 at 15:36:03 GMT, Daniel P. Berrangé wrote:

> It is useful to know which CPUs satisfy each x86-64 ABI
> compatibility level, when dealing with guest OS that require
> something newer than the baseline ABI.
>
> These ABI levels are defined in:
>
>   https://gitlab.com/x86-psABIs/x86-64-ABI/
>
> and supported by GCC, CLang, GLibC and more.
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  MAINTAINERS                        |   2 +-
>  docs/system/cpu-models-x86-abi.csv | 121 +++++++++++++++++++++++++++++
>  docs/system/cpu-models-x86.rst.inc |  18 +++++
>  3 files changed, 140 insertions(+), 1 deletion(-)
>  create mode 100644 docs/system/cpu-models-x86-abi.csv
>
> diff --git a/MAINTAINERS b/MAINTAINERS
> index fbb228ef2b..bb8d60c458 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -344,7 +344,7 @@ F: tests/tcg/i386/
>  F: tests/tcg/x86_64/
>  F: hw/i386/
>  F: disas/i386.c
> -F: docs/system/cpu-models-x86.rst.inc
> +F: docs/system/cpu-models-x86*
>  T: git https://gitlab.com/ehabkost/qemu.git x86-next
>  
>  Xtensa TCG CPUs
> diff --git a/docs/system/cpu-models-x86-abi.csv b/docs/system/cpu-models-x86-abi.csv
> new file mode 100644
> index 0000000000..4565e6a535
> --- /dev/null
> +++ b/docs/system/cpu-models-x86-abi.csv
> @@ -0,0 +1,121 @@
> +Model,baseline,v2,v3,v4
> +486,,,,
> +486-v1,,,,
> +Broadwell,✅,✅,✅,
> +Broadwell-IBRS,✅,✅,✅,
> +Broadwell-noTSX,✅,✅,✅,
> +Broadwell-noTSX-IBRS,✅,✅,✅,

Would it be useful to add an explicit negative mark (✘) in the slots
where the CPU does not satisfy the requirement? It makes reading the
table a little easier (my opinion, of course).

> +Broadwell-v1,✅,✅,✅,
> +Broadwell-v2,✅,✅,✅,
> +Broadwell-v3,✅,✅,✅,
> +Broadwell-v4,✅,✅,✅,
> +Cascadelake-Server,✅,✅,✅,✅
> +Cascadelake-Server-noTSX,✅,✅,✅,✅
> +Cascadelake-Server-v1,✅,✅,✅,✅
> +Cascadelake-Server-v2,✅,✅,✅,✅
> +Cascadelake-Server-v3,✅,✅,✅,✅
> +Cascadelake-Server-v4,✅,✅,✅,✅
> +Conroe,✅,,,
> +Conroe-v1,✅,,,
> +Cooperlake,✅,✅,✅,✅
> +Cooperlake-v1,✅,✅,✅,✅
> +Denverton,✅,✅,,
> +Denverton-v1,✅,✅,,
> +Denverton-v2,✅,✅,,
> +Dhyana,✅,✅,✅,
> +Dhyana-v1,✅,✅,✅,
> +EPYC,✅,✅,✅,
> +EPYC-IBPB,✅,✅,✅,
> +EPYC-Rome,✅,✅,✅,
> +EPYC-Rome-v1,✅,✅,✅,
> +EPYC-v1,✅,✅,✅,
> +EPYC-v2,✅,✅,✅,
> +EPYC-v3,✅,✅,✅,
> +Haswell,✅,✅,✅,
> +Haswell-IBRS,✅,✅,✅,
> +Haswell-noTSX,✅,✅,✅,
> +Haswell-noTSX-IBRS,✅,✅,✅,
> +Haswell-v1,✅,✅,✅,
> +Haswell-v2,✅,✅,✅,
> +Haswell-v3,✅,✅,✅,
> +Haswell-v4,✅,✅,✅,
> +Icelake-Client,✅,✅,✅,
> +Icelake-Client-noTSX,✅,✅,✅,
> +Icelake-Client-v1,✅,✅,✅,
> +Icelake-Client-v2,✅,✅,✅,
> +Icelake-Server,✅,✅,✅,✅
> +Icelake-Server-noTSX,✅,✅,✅,✅
> +Icelake-Server-v1,✅,✅,✅,✅
> +Icelake-Server-v2,✅,✅,✅,✅
> +Icelake-Server-v3,✅,✅,✅,✅
> +Icelake-Server-v4,✅,✅,✅,✅
> +IvyBridge,✅,✅,,
> +IvyBridge-IBRS,✅,✅,,
> +IvyBridge-v1,✅,✅,,
> +IvyBridge-v2,✅,✅,,
> +KnightsMill,✅,✅,✅,
> +KnightsMill-v1,✅,✅,✅,
> +Nehalem,✅,✅,,
> +Nehalem-IBRS,✅,✅,,
> +Nehalem-v1,✅,✅,,
> +Nehalem-v2,✅,✅,,
> +Opteron_G1,✅,,,
> +Opteron_G1-v1,✅,,,
> +Opteron_G2,✅,,,
> +Opteron_G2-v1,✅,,,
> +Opteron_G3,✅,,,
> +Opteron_G3-v1,✅,,,
> +Opteron_G4,✅,✅,,
> +Opteron_G4-v1,✅,✅,,
> +Opteron_G5,✅,✅,,
> +Opteron_G5-v1,✅,✅,,
> +Penryn,✅,,,
> +Penryn-v1,✅,,,
> +SandyBridge,✅,✅,,
> +SandyBridge-IBRS,✅,✅,,
> +SandyBridge-v1,✅,✅,,
> +SandyBridge-v2,✅,✅,,
> +Skylake-Client,✅,✅,✅,
> +Skylake-Client-IBRS,✅,✅,✅,
> +Skylake-Client-noTSX-IBRS,✅,✅,✅,
> +Skylake-Client-v1,✅,✅,✅,
> +Skylake-Client-v2,✅,✅,✅,
> +Skylake-Client-v3,✅,✅,✅,
> +Skylake-Server,✅,✅,✅,✅
> +Skylake-Server-IBRS,✅,✅,✅,✅
> +Skylake-Server-noTSX-IBRS,✅,✅,✅,✅
> +Skylake-Server-v1,✅,✅,✅,✅
> +Skylake-Server-v2,✅,✅,✅,✅
> +Skylake-Server-v3,✅,✅,✅,✅
> +Skylake-Server-v4,✅,✅,✅,✅
> +Snowridge,✅,✅,,
> +Snowridge-v1,✅,✅,,
> +Snowridge-v2,✅,✅,,
> +Westmere,✅,✅,,
> +Westmere-IBRS,✅,✅,,
> +Westmere-v1,✅,✅,,
> +Westmere-v2,✅,✅,,
> +athlon,,,,
> +athlon-v1,,,,
> +core2duo,✅,,,
> +core2duo-v1,✅,,,
> +coreduo,,,,
> +coreduo-v1,,,,
> +kvm32,,,,
> +kvm32-v1,,,,
> +kvm64,✅,,,
> +kvm64-v1,✅,,,
> +n270,,,,
> +n270-v1,,,,
> +pentium,,,,
> +pentium-v1,,,,
> +pentium2,,,,
> +pentium2-v1,,,,
> +pentium3,,,,
> +pentium3-v1,,,,
> +phenom,✅,,,
> +phenom-v1,✅,,,
> +qemu32,,,,
> +qemu32-v1,,,,
> +qemu64,✅,,,
> +qemu64-v1,✅,,,
> diff --git a/docs/system/cpu-models-x86.rst.inc b/docs/system/cpu-models-x86.rst.inc
> index 9a2327828e..b964b29c78 100644
> --- a/docs/system/cpu-models-x86.rst.inc
> +++ b/docs/system/cpu-models-x86.rst.inc
> @@ -39,6 +39,24 @@ CPU, as they would with "Host passthrough", but gives much of the
>  benefit of passthrough, while making live migration safe.
>  
>  
> +ABI compatibility levels for CPU models
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +The x86_64 architecture has a number of `ABI compatibility levels`_
> +defined. Traditionally most operating systems and toolchains would
> +only target the original baseline ABI. It is expected that in
> +future OS and toolchains are likely to target newer ABIs. The
> +following table illustrates which ABI compatibility levels can be
> +satisfied by the QEMU CPU models

Missing full stop (or colon?).

> +
> +.. _ABI compatibility levels: https://gitlab.com/x86-psABIs/x86-64-ABI/
> +
> +.. csv-table:: x86-64 ABI compatibility levels
> +   :file: cpu-models-x86-abi.csv
> +   :widths: 40,15,15,15,15
> +   :header-rows: 1
> +
> +
>  Preferred CPU models for Intel x86 hosts
>  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>  
> -- 
> 2.29.2

dme.
-- 
All of us, we're going out tonight. We're gonna walk all over your cars.

Re: [PATCH RFC 2/4] target/i386: define CPU models to model x86-64 ABI levels

[David Edmondson]

On Monday, 2021-02-01 at 15:36:04 GMT, Daniel P. Berrangé wrote:

> To paraphrase:
>
>   https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level/
>
> In 2020, AMD, Intel, Red Hat, and SUSE worked together to define
> three microarchitecture levels on top of the historical x86-64
> baseline:
>
>   * x86-64:    original x86_64 baseline instruction set
>   * x86-64-v2: vector instructions up to Streaming SIMD
>                Extensions 4.2 (SSE4.2)  and Supplemental
> 	       Streaming SIMD Extensions 3 (SSSE3), the
> 	       POPCNT instruction, and CMPXCHG16B
>   * x86-64-v3: vector instructions up to AVX2, MOVBE,
>                and additional bit-manipulation instructions.
>   * x86-64-v4: vector instructions from some of the
>                AVX-512 variants.
>
> This list of features is defined in the doc at:
>
>   https://gitlab.com/x86-psABIs/x86-64-ABI/
>
> QEMU has historically defaulted to the "qemu64" CPU model on
> x86_64 targets, which is approximately the x86-64 baseline
> ABI, with 'SVM' added.
>
> It is thought it might be desirable if QEMU could provide CPU models
> that closely correspond to the ABI levels, while offering portability
> across the maximum number of physical CPUs.
>
> Historically we've found that defining CPU models with an arbitrary
> combination of CPU features can be problematic, as some guest OS
> will not check all features they use, and instead assume that if
> they see feature "XX", then "YY" will always exist. This is reasonable
> in bare metal, but subject to breakage in virtualization.
>
> Thus in defining the CPI models for the ABI levels, this patch attempted

s/CPI/CPU/

> to base them off an existing CPU model.
>
> While each x86-64-abiNNN has a designated vendor, they are designed
> to be vendor agnostic models. ie they are capable of running on any
> AMD or Intel physical CPU model that satisfies the ABI level. eg

Only AMD or Intel? (You mention the Hugon chips elsewhere.)

> althgouh the x86-64-abi2 model is based on Nehalem, it should be
> able to run guests on an Opteron_G4/G5/EPYC host, since those CPUs
> support the features required by the x86-64 v2 ABI.
>
> More precisely the models were defined as:
>
>  * x86-64-abi1: close match for Opteron_G1, minus
>                 vme
>  * x86-64-abi2: perfect match for Nehalem
>  * x86-64-abi3: close match of Haswell-noTSX, minus
>                 aes pcid erms invpcid tsc-deadline
> 		x2apic pclmulqdq
>  * x86-64-abi4: close match of Skylake-Server-noTSX-IBRS, minus
>                 spec-ctrl
>
> None of the CPU models declare any VMX/SVM features. This would
> make them unable to support nested virtualization with live
> migration.

How about "Unable to support hardware accelerated nested
virtualization." ?

Is live migration relevant?

> Given their vendor agnostic design, these CPU models are primarily
> though to useful as the default CPU model for machine types. QEMU
> upstream is quite conservative in mandating new hardware features,
> but a downstream vendor may choose to mandate a newer x86-64 ABI
> level for downstream only machine types.
>
> Note that TCG is not capable of supporting the 2 newest ABI levels
> currently:
>
> * x86-64-abi3:
>
>   CPUID.01H:ECX.fma [bit 12]
>   CPUID.01H:ECX.avx [bit 28]
>   CPUID.01H:ECX.f16c [bit 29]
>   CPUID.07H:EBX.avx2 [bit 5]
>
> * x86-64-abi4:
>
>   CPUID.01H:ECX.pcid [bit 17]
>   CPUID.01H:ECX.x2apic [bit 21]
>   CPUID.01H:ECX.tsc-deadline [bit 24]
>   CPUID.07H:EBX.invpcid [bit 10]
>   CPUID.07H:EBX.avx512f [bit 16]
>   CPUID.07H:EBX.avx512dq [bit 17]
>   CPUID.07H:EBX.rdseed [bit 18]
>   CPUID.07H:EBX.avx512cd [bit 28]
>   CPUID.07H:EBX.avx512bw [bit 30]
>   CPUID.07H:EBX.avx512vl [bit 31]
>   CPUID.80000001H:ECX.3dnowprefetch [bit 8]
>   CPUID.0DH:EAX.xsavec [bit 1]
>
> Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> ---
>  docs/system/cpu-models-x86-abi.csv |   8 ++
>  target/i386/cpu.c                  | 156 +++++++++++++++++++++++++++++
>  2 files changed, 164 insertions(+)
>
> diff --git a/docs/system/cpu-models-x86-abi.csv b/docs/system/cpu-models-x86-abi.csv
> index 4565e6a535..d34d95d485 100644
> --- a/docs/system/cpu-models-x86-abi.csv
> +++ b/docs/system/cpu-models-x86-abi.csv
> @@ -119,3 +119,11 @@ qemu32,,,,
>  qemu32-v1,,,,
>  qemu64,✅,,,
>  qemu64-v1,✅,,,
> +x86-64-abi1,✅,,,
> +x86-64-abi1-v1,✅,,,
> +x86-64-abi2,✅,✅,,
> +x86-64-abi2-v1,✅,✅,,
> +x86-64-abi3,✅,✅,✅,
> +x86-64-abi3-v1,✅,✅,✅,
> +x86-64-abi4,✅,✅,✅,✅
> +x86-64-abi4-v1,✅,✅,✅,✅
> diff --git a/target/i386/cpu.c b/target/i386/cpu.c
> index ae89024d36..87a775a5eb 100644
> --- a/target/i386/cpu.c
> +++ b/target/i386/cpu.c
> @@ -1827,6 +1827,162 @@ static CPUCaches epyc_rome_cache_info = {
>   */
>  
>  static X86CPUDefinition builtin_x86_defs[] = {
> +    /*
> +     * These first few CPU models are designed to satisfy the
> +     * x86_64 ABI levels defined in:
> +     *
> +     *   https://gitlab.com/x86-psABIs/x86-64-ABI/
> +     *
> +     * They were constructed as follows:
> +     *
> +     *   - Find all the CPU models which can satisfy the ABI
> +     *   - Calculate the lowest common denominator (LCD) of these
> +     *     models' features
> +     *   - Find the named model most closely matching the LCD
> +     *   - Strip its features back to the LCD
> +     *
> +     * The above spec uses the "x86-64-vNN" naming convention.
> +     * This clashes with the "vNN" suffix QEMU uses for versioning.
> +     * Thus we use "abiNNN" as a suffix.
> +     */
> +    {
> +        /*
> +         * Derived from Opteron_G1, minus
> +         *   vme
> +         */
> +        .name = "x86-64-abi1",
> +        .level = 5,
> +        .vendor = CPUID_VENDOR_AMD,
> +        .family = 15,
> +        .model = 6,
> +        .stepping = 1,
> +        .features[FEAT_1_EDX] =
> +            CPUID_SSE2 | CPUID_SSE | CPUID_FXSR | CPUID_MMX |
> +            CPUID_CLFLUSH | CPUID_PSE36 | CPUID_PAT | CPUID_CMOV | CPUID_MCA |
> +            CPUID_PGE | CPUID_MTRR | CPUID_SEP | CPUID_APIC | CPUID_CX8 |
> +            CPUID_MCE | CPUID_PAE | CPUID_MSR | CPUID_TSC | CPUID_PSE |
> +            CPUID_DE | CPUID_FP87,
> +        .features[FEAT_1_ECX] =
> +            CPUID_EXT_SSE3,
> +        .features[FEAT_8000_0001_EDX] =
> +            CPUID_EXT2_LM | CPUID_EXT2_NX | CPUID_EXT2_SYSCALL,
> +        .xlevel = 0x80000008,
> +        .model_id = "QEMU x86-64 baseline ABI",
> +    },
> +    {
> +        /* Derived from Nehalem */
> +        .name = "x86-64-abi2",
> +        .level = 11,
> +        .vendor = CPUID_VENDOR_INTEL,
> +        .family = 6,
> +        .model = 26,
> +        .stepping = 3,
> +        .features[FEAT_1_EDX] =
> +            CPUID_VME | CPUID_SSE2 | CPUID_SSE | CPUID_FXSR | CPUID_MMX |
> +            CPUID_CLFLUSH | CPUID_PSE36 | CPUID_PAT | CPUID_CMOV | CPUID_MCA |
> +            CPUID_PGE | CPUID_MTRR | CPUID_SEP | CPUID_APIC | CPUID_CX8 |
> +            CPUID_MCE | CPUID_PAE | CPUID_MSR | CPUID_TSC | CPUID_PSE |
> +            CPUID_DE | CPUID_FP87,
> +        .features[FEAT_1_ECX] =
> +            CPUID_EXT_POPCNT | CPUID_EXT_SSE42 | CPUID_EXT_SSE41 |
> +            CPUID_EXT_CX16 | CPUID_EXT_SSSE3 | CPUID_EXT_SSE3,
> +        .features[FEAT_8000_0001_EDX] =
> +            CPUID_EXT2_LM | CPUID_EXT2_SYSCALL | CPUID_EXT2_NX,
> +        .features[FEAT_8000_0001_ECX] =
> +            CPUID_EXT3_LAHF_LM,
> +        .xlevel = 0x80000008,
> +        .model_id = "QEMU x86-64-v2 ABI",
> +    },
> +    {
> +        /*
> +         * Derived from Haswell-noTSX, minus
> +         *   aes pcid erms invpcid tsc-deadline x2apic pclmulqdq
> +         */
> +        .name = "x86-64-abi3",
> +        .level = 0xd,
> +        .vendor = CPUID_VENDOR_INTEL,
> +        .family = 6,
> +        .model = 60,
> +        .stepping = 1,
> +        .features[FEAT_1_EDX] =
> +            CPUID_VME | CPUID_SSE2 | CPUID_SSE | CPUID_FXSR | CPUID_MMX |
> +            CPUID_CLFLUSH | CPUID_PSE36 | CPUID_PAT | CPUID_CMOV | CPUID_MCA |
> +            CPUID_PGE | CPUID_MTRR | CPUID_SEP | CPUID_APIC | CPUID_CX8 |
> +            CPUID_MCE | CPUID_PAE | CPUID_MSR | CPUID_TSC | CPUID_PSE |
> +            CPUID_DE | CPUID_FP87,
> +        .features[FEAT_1_ECX] =
> +            CPUID_EXT_AVX | CPUID_EXT_XSAVE |
> +            CPUID_EXT_POPCNT | CPUID_EXT_SSE42 |
> +            CPUID_EXT_SSE41 | CPUID_EXT_CX16 | CPUID_EXT_SSSE3 |
> +            CPUID_EXT_SSE3 |
> +            CPUID_EXT_FMA | CPUID_EXT_MOVBE |
> +            CPUID_EXT_F16C | CPUID_EXT_RDRAND,
> +        .features[FEAT_8000_0001_EDX] =
> +            CPUID_EXT2_LM | CPUID_EXT2_RDTSCP | CPUID_EXT2_NX |
> +            CPUID_EXT2_SYSCALL,
> +        .features[FEAT_8000_0001_ECX] =
> +            CPUID_EXT3_ABM | CPUID_EXT3_LAHF_LM,
> +        .features[FEAT_7_0_EBX] =
> +            CPUID_7_0_EBX_FSGSBASE | CPUID_7_0_EBX_BMI1 |
> +            CPUID_7_0_EBX_AVX2 | CPUID_7_0_EBX_SMEP |
> +            CPUID_7_0_EBX_BMI2,
> +        .features[FEAT_XSAVE] =
> +            CPUID_XSAVE_XSAVEOPT,
> +        .features[FEAT_6_EAX] =
> +            CPUID_6_EAX_ARAT,
> +        .xlevel = 0x80000008,
> +        .model_id = "QEMU x86-64-v3 ABI",
> +    },
> +
> +    {
> +        /*
> +         * Derived from Skylake-Server-noTSX-IBRS, minus:
> +         *  spec-ctrl
> +         */
> +        .name = "x86-64-abi4",
> +        .level = 0xd,
> +        .vendor = CPUID_VENDOR_INTEL,
> +        .family = 6,
> +        .model = 85,
> +        .stepping = 4,
> +        .features[FEAT_1_EDX] =
> +            CPUID_VME | CPUID_SSE2 | CPUID_SSE | CPUID_FXSR | CPUID_MMX |
> +            CPUID_CLFLUSH | CPUID_PSE36 | CPUID_PAT | CPUID_CMOV | CPUID_MCA |
> +            CPUID_PGE | CPUID_MTRR | CPUID_SEP | CPUID_APIC | CPUID_CX8 |
> +            CPUID_MCE | CPUID_PAE | CPUID_MSR | CPUID_TSC | CPUID_PSE |
> +            CPUID_DE | CPUID_FP87,
> +        .features[FEAT_1_ECX] =
> +            CPUID_EXT_AVX | CPUID_EXT_XSAVE | CPUID_EXT_AES |
> +            CPUID_EXT_POPCNT | CPUID_EXT_X2APIC | CPUID_EXT_SSE42 |
> +            CPUID_EXT_SSE41 | CPUID_EXT_CX16 | CPUID_EXT_SSSE3 |
> +            CPUID_EXT_PCLMULQDQ | CPUID_EXT_SSE3 |
> +            CPUID_EXT_TSC_DEADLINE_TIMER | CPUID_EXT_FMA | CPUID_EXT_MOVBE |
> +            CPUID_EXT_PCID | CPUID_EXT_F16C | CPUID_EXT_RDRAND,
> +        .features[FEAT_8000_0001_EDX] =
> +            CPUID_EXT2_LM | CPUID_EXT2_PDPE1GB | CPUID_EXT2_RDTSCP |
> +            CPUID_EXT2_NX | CPUID_EXT2_SYSCALL,
> +        .features[FEAT_8000_0001_ECX] =
> +            CPUID_EXT3_ABM | CPUID_EXT3_LAHF_LM | CPUID_EXT3_3DNOWPREFETCH,
> +        .features[FEAT_7_0_EBX] =
> +            CPUID_7_0_EBX_FSGSBASE | CPUID_7_0_EBX_BMI1 |
> +            CPUID_7_0_EBX_AVX2 | CPUID_7_0_EBX_SMEP |
> +            CPUID_7_0_EBX_BMI2 | CPUID_7_0_EBX_ERMS | CPUID_7_0_EBX_INVPCID |
> +            CPUID_7_0_EBX_RDSEED | CPUID_7_0_EBX_ADX |
> +            CPUID_7_0_EBX_SMAP | CPUID_7_0_EBX_CLWB |
> +            CPUID_7_0_EBX_AVX512F | CPUID_7_0_EBX_AVX512DQ |
> +            CPUID_7_0_EBX_AVX512BW | CPUID_7_0_EBX_AVX512CD |
> +            CPUID_7_0_EBX_AVX512VL,
> +        .features[FEAT_7_0_ECX] =
> +            CPUID_7_0_ECX_PKU,
> +        .features[FEAT_XSAVE] =
> +            CPUID_XSAVE_XSAVEOPT | CPUID_XSAVE_XSAVEC |
> +            CPUID_XSAVE_XGETBV1,
> +        .features[FEAT_6_EAX] =
> +            CPUID_6_EAX_ARAT,
> +        .xlevel = 0x80000008,
> +        .model_id = "QEMU x86-64-v4 ABI",
> +    },
> +
>      {
>          .name = "qemu64",
>          .level = 0xd,
> -- 
> 2.29.2

dme.
-- 
All those lines and circles, to me, a mystery.

Re: [PATCH RFC 1/4] docs: add a table showing x86-64 ABI compatibility levels

[Daniel P. Berrangé]

On Tue, Feb 02, 2021 at 09:41:15AM +0000, David Edmondson wrote:
> On Monday, 2021-02-01 at 15:36:03 GMT, Daniel P. Berrangé wrote:
> 
> > It is useful to know which CPUs satisfy each x86-64 ABI
> > compatibility level, when dealing with guest OS that require
> > something newer than the baseline ABI.
> >
> > These ABI levels are defined in:
> >
> >   https://gitlab.com/x86-psABIs/x86-64-ABI/
> >
> > and supported by GCC, CLang, GLibC and more.
> >
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> > ---
> >  MAINTAINERS                        |   2 +-
> >  docs/system/cpu-models-x86-abi.csv | 121 +++++++++++++++++++++++++++++
> >  docs/system/cpu-models-x86.rst.inc |  18 +++++
> >  3 files changed, 140 insertions(+), 1 deletion(-)
> >  create mode 100644 docs/system/cpu-models-x86-abi.csv
> >
> > diff --git a/MAINTAINERS b/MAINTAINERS
> > index fbb228ef2b..bb8d60c458 100644
> > --- a/MAINTAINERS
> > +++ b/MAINTAINERS
> > @@ -344,7 +344,7 @@ F: tests/tcg/i386/
> >  F: tests/tcg/x86_64/
> >  F: hw/i386/
> >  F: disas/i386.c
> > -F: docs/system/cpu-models-x86.rst.inc
> > +F: docs/system/cpu-models-x86*
> >  T: git https://gitlab.com/ehabkost/qemu.git x86-next
> >  
> >  Xtensa TCG CPUs
> > diff --git a/docs/system/cpu-models-x86-abi.csv b/docs/system/cpu-models-x86-abi.csv
> > new file mode 100644
> > index 0000000000..4565e6a535
> > --- /dev/null
> > +++ b/docs/system/cpu-models-x86-abi.csv
> > @@ -0,0 +1,121 @@
> > +Model,baseline,v2,v3,v4
> > +486,,,,
> > +486-v1,,,,
> > +Broadwell,✅,✅,✅,
> > +Broadwell-IBRS,✅,✅,✅,
> > +Broadwell-noTSX,✅,✅,✅,
> > +Broadwell-noTSX-IBRS,✅,✅,✅,
> 
> Would it be useful to add an explicit negative mark (✘) in the slots
> where the CPU does not satisfy the requirement? It makes reading the
> table a little easier (my opinion, of course).

I felt it was clearer to only show the positive case. Since the
ABI levels are additive, you can count the ticks at a glance to see
the ABI level achieved. Also this CSV file isn't really meant to
be seen by users directly. It is just data input that gets rendered
into an HTML table that looks like this:

  https://berrange.gitlab.io/-/qemu/-/jobs/1001700036/artifacts/public/system/target-i386.html#recommendations-for-kvm-cpu-model-configuration-on-x86-hosts

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH RFC 1/4] docs: add a table showing x86-64 ABI compatibility levels

[Daniel P. Berrangé]

On Mon, Feb 01, 2021 at 01:28:51PM -0500, Eduardo Habkost wrote:
> On Mon, Feb 01, 2021 at 03:36:03PM +0000, Daniel P. Berrangé wrote:
> > It is useful to know which CPUs satisfy each x86-64 ABI
> > compatibility level, when dealing with guest OS that require
> > something newer than the baseline ABI.
> > 
> > These ABI levels are defined in:
> > 
> >   https://gitlab.com/x86-psABIs/x86-64-ABI/
> > 
> > and supported by GCC, CLang, GLibC and more.
> > 
> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
> [...]
> > diff --git a/docs/system/cpu-models-x86-abi.csv b/docs/system/cpu-models-x86-abi.csv
> > new file mode 100644
> > index 0000000000..4565e6a535
> > --- /dev/null
> > +++ b/docs/system/cpu-models-x86-abi.csv
> > @@ -0,0 +1,121 @@
> > +Model,baseline,v2,v3,v4
> > +486,,,,
> > +486-v1,,,,
> > +Broadwell,✅,✅,✅,
> > +Broadwell-IBRS,✅,✅,✅,
> > +Broadwell-noTSX,✅,✅,✅,
> > +Broadwell-noTSX-IBRS,✅,✅,✅,
> > +Broadwell-v1,✅,✅,✅,
> > +Broadwell-v2,✅,✅,✅,
> > +Broadwell-v3,✅,✅,✅,
> > +Broadwell-v4,✅,✅,✅,
> 
> Unversioned names like "Broadwell" are machine-type-dependent
> aliases.  I don't think they should be present in the table.
> 
> Models with suffixes like -IBRS, -noTSX, etc. are also aliases to
> specific versions.  Maybe they could appear in the table for
> completeness, but I'm not sure.

I guess just skip the CPUs with "alias-of" reported is easiest


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH RFC 2/4] target/i386: define CPU models to model x86-64 ABI levels

[Daniel P. Berrangé]

On Tue, Feb 02, 2021 at 09:46:55AM +0000, David Edmondson wrote:
> On Monday, 2021-02-01 at 15:36:04 GMT, Daniel P. Berrangé wrote:
> 
> > To paraphrase:
> >
> >   https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level/
> >
> > In 2020, AMD, Intel, Red Hat, and SUSE worked together to define
> > three microarchitecture levels on top of the historical x86-64
> > baseline:
> >
> >   * x86-64:    original x86_64 baseline instruction set
> >   * x86-64-v2: vector instructions up to Streaming SIMD
> >                Extensions 4.2 (SSE4.2)  and Supplemental
> > 	       Streaming SIMD Extensions 3 (SSSE3), the
> > 	       POPCNT instruction, and CMPXCHG16B
> >   * x86-64-v3: vector instructions up to AVX2, MOVBE,
> >                and additional bit-manipulation instructions.
> >   * x86-64-v4: vector instructions from some of the
> >                AVX-512 variants.
> >
> > This list of features is defined in the doc at:
> >
> >   https://gitlab.com/x86-psABIs/x86-64-ABI/
> >
> > QEMU has historically defaulted to the "qemu64" CPU model on
> > x86_64 targets, which is approximately the x86-64 baseline
> > ABI, with 'SVM' added.
> >
> > It is thought it might be desirable if QEMU could provide CPU models
> > that closely correspond to the ABI levels, while offering portability
> > across the maximum number of physical CPUs.
> >
> > Historically we've found that defining CPU models with an arbitrary
> > combination of CPU features can be problematic, as some guest OS
> > will not check all features they use, and instead assume that if
> > they see feature "XX", then "YY" will always exist. This is reasonable
> > in bare metal, but subject to breakage in virtualization.
> >
> > Thus in defining the CPI models for the ABI levels, this patch attempted
> 
> s/CPI/CPU/
> 
> > to base them off an existing CPU model.
> >
> > While each x86-64-abiNNN has a designated vendor, they are designed
> > to be vendor agnostic models. ie they are capable of running on any
> > AMD or Intel physical CPU model that satisfies the ABI level. eg
> 
> Only AMD or Intel? (You mention the Hugon chips elsewhere.)

In theory any x86 CPU that meets the ABI level, regardless of vendor
but if any vendor's set of CPUID features diverges too far from other
CPUs of similar level we might have problems.

For example, I had to specially avoid including "aes" in the
x86-64-abi3 because of the Hugon chips lacking it. There might
be other cases like this, since I've only compared CPUID sets
for named CPUs that QEMU includes.

> > None of the CPU models declare any VMX/SVM features. This would
> > make them unable to support nested virtualization with live
> > migration.
> 
> How about "Unable to support hardware accelerated nested
> virtualization." ?
> 
> Is live migration relevant?

Choice of CPU models is a critical decision in your future ability
to live migrate, so I wanted to call that out specifically.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH RFC 1/4] docs: add a table showing x86-64 ABI compatibility levels

[David Edmondson]

On Tuesday, 2021-02-02 at 12:23:42 GMT, Daniel P. Berrangé wrote:

> On Tue, Feb 02, 2021 at 09:41:15AM +0000, David Edmondson wrote:
>> On Monday, 2021-02-01 at 15:36:03 GMT, Daniel P. Berrangé wrote:
>> 
>> > It is useful to know which CPUs satisfy each x86-64 ABI
>> > compatibility level, when dealing with guest OS that require
>> > something newer than the baseline ABI.
>> >
>> > These ABI levels are defined in:
>> >
>> >   https://gitlab.com/x86-psABIs/x86-64-ABI/
>> >
>> > and supported by GCC, CLang, GLibC and more.
>> >
>> > Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
>> > ---
>> >  MAINTAINERS                        |   2 +-
>> >  docs/system/cpu-models-x86-abi.csv | 121 +++++++++++++++++++++++++++++
>> >  docs/system/cpu-models-x86.rst.inc |  18 +++++
>> >  3 files changed, 140 insertions(+), 1 deletion(-)
>> >  create mode 100644 docs/system/cpu-models-x86-abi.csv
>> >
>> > diff --git a/MAINTAINERS b/MAINTAINERS
>> > index fbb228ef2b..bb8d60c458 100644
>> > --- a/MAINTAINERS
>> > +++ b/MAINTAINERS
>> > @@ -344,7 +344,7 @@ F: tests/tcg/i386/
>> >  F: tests/tcg/x86_64/
>> >  F: hw/i386/
>> >  F: disas/i386.c
>> > -F: docs/system/cpu-models-x86.rst.inc
>> > +F: docs/system/cpu-models-x86*
>> >  T: git https://gitlab.com/ehabkost/qemu.git x86-next
>> >  
>> >  Xtensa TCG CPUs
>> > diff --git a/docs/system/cpu-models-x86-abi.csv b/docs/system/cpu-models-x86-abi.csv
>> > new file mode 100644
>> > index 0000000000..4565e6a535
>> > --- /dev/null
>> > +++ b/docs/system/cpu-models-x86-abi.csv
>> > @@ -0,0 +1,121 @@
>> > +Model,baseline,v2,v3,v4
>> > +486,,,,
>> > +486-v1,,,,
>> > +Broadwell,✅,✅,✅,
>> > +Broadwell-IBRS,✅,✅,✅,
>> > +Broadwell-noTSX,✅,✅,✅,
>> > +Broadwell-noTSX-IBRS,✅,✅,✅,
>> 
>> Would it be useful to add an explicit negative mark (✘) in the slots
>> where the CPU does not satisfy the requirement? It makes reading the
>> table a little easier (my opinion, of course).
>
> I felt it was clearer to only show the positive case. Since the
> ABI levels are additive, you can count the ticks at a glance to see
> the ABI level achieved. Also this CSV file isn't really meant to
> be seen by users directly. It is just data input that gets rendered
> into an HTML table that looks like this:
>
>   https://berrange.gitlab.io/-/qemu/-/jobs/1001700036/artifacts/public/system/target-i386.html#recommendations-for-kvm-cpu-model-configuration-on-x86-hosts

Fine with me.

dme.
-- 
Another lonely day, no one here but me-o.

Re: [PATCH RFC 2/4] target/i386: define CPU models to model x86-64 ABI levels

[David Edmondson]

On Tuesday, 2021-02-02 at 12:32:39 GMT, Daniel P. Berrangé wrote:

> On Tue, Feb 02, 2021 at 09:46:55AM +0000, David Edmondson wrote:
>> On Monday, 2021-02-01 at 15:36:04 GMT, Daniel P. Berrangé wrote:
>> 
>> > To paraphrase:
>> >
>> >   https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level/
>> >
>> > In 2020, AMD, Intel, Red Hat, and SUSE worked together to define
>> > three microarchitecture levels on top of the historical x86-64
>> > baseline:
>> >
>> >   * x86-64:    original x86_64 baseline instruction set
>> >   * x86-64-v2: vector instructions up to Streaming SIMD
>> >                Extensions 4.2 (SSE4.2)  and Supplemental
>> > 	       Streaming SIMD Extensions 3 (SSSE3), the
>> > 	       POPCNT instruction, and CMPXCHG16B
>> >   * x86-64-v3: vector instructions up to AVX2, MOVBE,
>> >                and additional bit-manipulation instructions.
>> >   * x86-64-v4: vector instructions from some of the
>> >                AVX-512 variants.
>> >
>> > This list of features is defined in the doc at:
>> >
>> >   https://gitlab.com/x86-psABIs/x86-64-ABI/
>> >
>> > QEMU has historically defaulted to the "qemu64" CPU model on
>> > x86_64 targets, which is approximately the x86-64 baseline
>> > ABI, with 'SVM' added.
>> >
>> > It is thought it might be desirable if QEMU could provide CPU models
>> > that closely correspond to the ABI levels, while offering portability
>> > across the maximum number of physical CPUs.
>> >
>> > Historically we've found that defining CPU models with an arbitrary
>> > combination of CPU features can be problematic, as some guest OS
>> > will not check all features they use, and instead assume that if
>> > they see feature "XX", then "YY" will always exist. This is reasonable
>> > in bare metal, but subject to breakage in virtualization.
>> >
>> > Thus in defining the CPI models for the ABI levels, this patch attempted
>> 
>> s/CPI/CPU/
>> 
>> > to base them off an existing CPU model.
>> >
>> > While each x86-64-abiNNN has a designated vendor, they are designed
>> > to be vendor agnostic models. ie they are capable of running on any
>> > AMD or Intel physical CPU model that satisfies the ABI level. eg
>> 
>> Only AMD or Intel? (You mention the Hugon chips elsewhere.)
>
> In theory any x86 CPU that meets the ABI level, regardless of vendor
> but if any vendor's set of CPUID features diverges too far from other
> CPUs of similar level we might have problems.

Understood - so why say "AMD or Intel"?

> For example, I had to specially avoid including "aes" in the
> x86-64-abi3 because of the Hugon chips lacking it. There might
> be other cases like this, since I've only compared CPUID sets
> for named CPUs that QEMU includes.
>
>> > None of the CPU models declare any VMX/SVM features. This would
>> > make them unable to support nested virtualization with live
>> > migration.
>> 
>> How about "Unable to support hardware accelerated nested
>> virtualization." ?
>> 
>> Is live migration relevant?
>
> Choice of CPU models is a critical decision in your future ability
> to live migrate, so I wanted to call that out specifically.

But the restriction, I believe, is not that you won't be able to live
migrate with nested VMs, it's that you don't get support for nested VMs
(well, hardware accelerated - you could still run a fully emulated
nested VM). Live migration with nested VMs is irrelevant if I don't
*get* nested VMs.

dme.
-- 
I don't care 'bout your other girls, just be good to me.

Re: [PATCH RFC 2/4] target/i386: define CPU models to model x86-64 ABI levels

[Daniel P. Berrangé]

On Tue, Feb 02, 2021 at 12:50:53PM +0000, David Edmondson wrote:
> On Tuesday, 2021-02-02 at 12:32:39 GMT, Daniel P. Berrangé wrote:
> 
> > On Tue, Feb 02, 2021 at 09:46:55AM +0000, David Edmondson wrote:
> >> On Monday, 2021-02-01 at 15:36:04 GMT, Daniel P. Berrangé wrote:
> >> 
> >> > To paraphrase:
> >> >
> >> >   https://developers.redhat.com/blog/2021/01/05/building-red-hat-enterprise-linux-9-for-the-x86-64-v2-microarchitecture-level/
> >> >
> >> > In 2020, AMD, Intel, Red Hat, and SUSE worked together to define
> >> > three microarchitecture levels on top of the historical x86-64
> >> > baseline:
> >> >
> >> >   * x86-64:    original x86_64 baseline instruction set
> >> >   * x86-64-v2: vector instructions up to Streaming SIMD
> >> >                Extensions 4.2 (SSE4.2)  and Supplemental
> >> > 	       Streaming SIMD Extensions 3 (SSSE3), the
> >> > 	       POPCNT instruction, and CMPXCHG16B
> >> >   * x86-64-v3: vector instructions up to AVX2, MOVBE,
> >> >                and additional bit-manipulation instructions.
> >> >   * x86-64-v4: vector instructions from some of the
> >> >                AVX-512 variants.
> >> >
> >> > This list of features is defined in the doc at:
> >> >
> >> >   https://gitlab.com/x86-psABIs/x86-64-ABI/
> >> >
> >> > QEMU has historically defaulted to the "qemu64" CPU model on
> >> > x86_64 targets, which is approximately the x86-64 baseline
> >> > ABI, with 'SVM' added.
> >> >
> >> > It is thought it might be desirable if QEMU could provide CPU models
> >> > that closely correspond to the ABI levels, while offering portability
> >> > across the maximum number of physical CPUs.
> >> >
> >> > Historically we've found that defining CPU models with an arbitrary
> >> > combination of CPU features can be problematic, as some guest OS
> >> > will not check all features they use, and instead assume that if
> >> > they see feature "XX", then "YY" will always exist. This is reasonable
> >> > in bare metal, but subject to breakage in virtualization.
> >> >
> >> > Thus in defining the CPI models for the ABI levels, this patch attempted
> >> 
> >> s/CPI/CPU/
> >> 
> >> > to base them off an existing CPU model.
> >> >
> >> > While each x86-64-abiNNN has a designated vendor, they are designed
> >> > to be vendor agnostic models. ie they are capable of running on any
> >> > AMD or Intel physical CPU model that satisfies the ABI level. eg
> >> 
> >> Only AMD or Intel? (You mention the Hugon chips elsewhere.)
> >
> > In theory any x86 CPU that meets the ABI level, regardless of vendor
> > but if any vendor's set of CPUID features diverges too far from other
> > CPUs of similar level we might have problems.
> 
> Understood - so why say "AMD or Intel"?

This text is just giving an example - it doesn't need to be an
exhaustive list of vendors.  Running AMD CPUs models on Intel
host and vica-verca are the two most common scenaroos.

> 
> > For example, I had to specially avoid including "aes" in the
> > x86-64-abi3 because of the Hugon chips lacking it. There might
> > be other cases like this, since I've only compared CPUID sets
> > for named CPUs that QEMU includes.
> >
> >> > None of the CPU models declare any VMX/SVM features. This would
> >> > make them unable to support nested virtualization with live
> >> > migration.
> >> 
> >> How about "Unable to support hardware accelerated nested
> >> virtualization." ?
> >> 
> >> Is live migration relevant?
> >
> > Choice of CPU models is a critical decision in your future ability
> > to live migrate, so I wanted to call that out specifically.
> 
> But the restriction, I believe, is not that you won't be able to live
> migrate with nested VMs, it's that you don't get support for nested VMs
> (well, hardware accelerated - you could still run a fully emulated
> nested VM). Live migration with nested VMs is irrelevant if I don't
> *get* nested VMs.

What I mean is that if you use  "-cpu x86-64-abi2,+vmx" and KVM will
enable nested virt, but I believe live migration will fail because
we've not declared any VMX capabilities in the CPU model. I'll have
to defer to Paolo on the actual failure scenario details.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|