[PATCH v6 00/19] i386: KVM: expand Hyper-V features early

[PATCH v6 00/19] i386: KVM: expand Hyper-V features early

[Vitaly Kuznetsov]

Changes since v5:
- Temporary drop 'hv-default' feature as it is causing some controversy.
 The rest of the patchset is valuable on its own.
- Add 3 additiona fixes:
 i386: kill off hv_cpuid_check_and_set() [Igor's suggestion]
 i386: HV_HYPERCALL_AVAILABLE privilege bit is always needed
 i386: Hyper-V SynIC requires POST_MESSAGES/SIGNAL_EVENTS priviliges

The last two patches are inspired by 'Fine-grained access check to Hyper-V
hypercalls and MSRs' work for KVM:
https://lore.kernel.org/kvm/20210419160127.192712-1-vkuznets@redhat.com/

Original description:

Upper layer tools like libvirt want to figure out which Hyper-V features are
supported by the underlying stack (QEMU/KVM) but currently they are unable to
do so. We have a nice 'hv_passthrough' CPU flag supported by QEMU but it has
no effect on e.g. QMP's 

query-cpu-model-expansion type=full model={"name":"host","props":{"hv-passthrough":true}}

command as we parse Hyper-V features after creating KVM vCPUs and not at
feature expansion time. To support the use-case we first need to make 
KVM_GET_SUPPORTED_HV_CPUID ioctl a system-wide ioctl as the existing
vCPU version can't be used that early. This is what KVM part does. With
that done, we can make early Hyper-V feature expansion (this series).

Vitaly Kuznetsov (19):
  i386: keep hyperv_vendor string up-to-date
  i386: invert hyperv_spinlock_attempts setting logic with
    hv_passthrough
  i386: always fill Hyper-V CPUID feature leaves from X86CPU data
  i386: stop using env->features[] for filling Hyper-V CPUIDs
  i386: introduce hyperv_feature_supported()
  i386: introduce hv_cpuid_get_host()
  i386: drop FEAT_HYPERV feature leaves
  i386: introduce hv_cpuid_cache
  i386: split hyperv_handle_properties() into
    hyperv_expand_features()/hyperv_fill_cpuids()
  i386: move eVMCS enablement to hyperv_init_vcpu()
  i386: switch hyperv_expand_features() to using error_setg()
  i386: adjust the expected KVM_GET_SUPPORTED_HV_CPUID array size
  i386: prefer system KVM_GET_SUPPORTED_HV_CPUID ioctl over vCPU's one
  i386: use global kvm_state in hyperv_enabled() check
  i386: expand Hyper-V features during CPU feature expansion time
  i386: kill off hv_cpuid_check_and_set()
  i386: HV_HYPERCALL_AVAILABLE privilege bit is always needed
  i386: Hyper-V SynIC requires POST_MESSAGES/SIGNAL_EVENTS priviliges
  qtest/hyperv: Introduce a simple hyper-v test

 MAINTAINERS                    |   1 +
 target/i386/cpu.c              | 113 +-------
 target/i386/cpu.h              |   6 +-
 target/i386/kvm/hyperv-proto.h |   6 +
 target/i386/kvm/kvm-stub.c     |   5 +
 target/i386/kvm/kvm.c          | 514 +++++++++++++++++----------------
 target/i386/kvm/kvm_i386.h     |   1 +
 tests/qtest/hyperv-test.c      | 225 +++++++++++++++
 tests/qtest/meson.build        |   3 +-
 9 files changed, 526 insertions(+), 348 deletions(-)
 create mode 100644 tests/qtest/hyperv-test.c

-- 
2.30.2

[PATCH v6 01/19] i386: keep hyperv_vendor string up-to-date

[Vitaly Kuznetsov]

When cpu->hyperv_vendor is not set manually we default to "Microsoft Hv"
and in 'hv_passthrough' mode we get the information from the host. This
information is stored in cpu->hyperv_vendor_id[] array but we don't update
cpu->hyperv_vendor string so e.g. QMP's query-cpu-model-expansion output
is incorrect.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 target/i386/cpu.c     | 19 +++++++++----------
 target/i386/kvm/kvm.c |  5 +++++
 2 files changed, 14 insertions(+), 10 deletions(-)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index ad99cad0e7ce..2d05df232329 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -6665,17 +6665,16 @@ static void x86_cpu_hyperv_realize(X86CPU *cpu)
 
     /* Hyper-V vendor id */
     if (!cpu->hyperv_vendor) {
-        memcpy(cpu->hyperv_vendor_id, "Microsoft Hv", 12);
-    } else {
-        len = strlen(cpu->hyperv_vendor);
-
-        if (len > 12) {
-            warn_report("hv-vendor-id truncated to 12 characters");
-            len = 12;
-        }
-        memset(cpu->hyperv_vendor_id, 0, 12);
-        memcpy(cpu->hyperv_vendor_id, cpu->hyperv_vendor, len);
+        object_property_set_str(OBJECT(cpu), "hv-vendor-id", "Microsoft Hv",
+                                &error_abort);
+    }
+    len = strlen(cpu->hyperv_vendor);
+    if (len > 12) {
+        warn_report("hv-vendor-id truncated to 12 characters");
+        len = 12;
     }
+    memset(cpu->hyperv_vendor_id, 0, 12);
+    memcpy(cpu->hyperv_vendor_id, cpu->hyperv_vendor, len);
 
     /* 'Hv#1' interface identification*/
     cpu->hyperv_interface_id[0] = 0x31237648;
diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index 7fe9f527103c..ab073a5e9c44 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -1215,6 +1215,11 @@ static int hyperv_handle_properties(CPUState *cs,
             cpu->hyperv_vendor_id[0] = c->ebx;
             cpu->hyperv_vendor_id[1] = c->ecx;
             cpu->hyperv_vendor_id[2] = c->edx;
+            cpu->hyperv_vendor = g_realloc(cpu->hyperv_vendor,
+                                           sizeof(cpu->hyperv_vendor_id) + 1);
+            memcpy(cpu->hyperv_vendor, cpu->hyperv_vendor_id,
+                   sizeof(cpu->hyperv_vendor_id));
+            cpu->hyperv_vendor[sizeof(cpu->hyperv_vendor_id)] = 0;
         }
 
         c = cpuid_find_entry(cpuid, HV_CPUID_INTERFACE, 0);
-- 
2.30.2

[PATCH v6 02/19] i386: invert hyperv_spinlock_attempts setting logic with hv_passthrough

[Vitaly Kuznetsov]

There is no need to have this special case: like all other Hyper-V
enlightenments we can just use kernel's supplied value in hv_passthrough
mode.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 target/i386/kvm/kvm.c | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index ab073a5e9c44..afd173514da1 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -1256,11 +1256,7 @@ static int hyperv_handle_properties(CPUState *cs,
         c = cpuid_find_entry(cpuid, HV_CPUID_ENLIGHTMENT_INFO, 0);
         if (c) {
             env->features[FEAT_HV_RECOMM_EAX] = c->eax;
-
-            /* hv-spinlocks may have been overriden */
-            if (cpu->hyperv_spinlock_attempts != HYPERV_SPINLOCK_NEVER_NOTIFY) {
-                c->ebx = cpu->hyperv_spinlock_attempts;
-            }
+            cpu->hyperv_spinlock_attempts = c->ebx;
         }
         c = cpuid_find_entry(cpuid, HV_CPUID_NESTED_FEATURES, 0);
         if (c) {
-- 
2.30.2

[PATCH v6 03/19] i386: always fill Hyper-V CPUID feature leaves from X86CPU data

[Vitaly Kuznetsov]

We have all the required data in X86CPU already and as we are about to
split hyperv_handle_properties() into hyperv_expand_features()/
hyperv_fill_cpuids() we can remove the blind copy. The functional change
is that QEMU won't pass CPUID leaves it doesn't currently know about
to the guest but arguably this is a good change.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 target/i386/kvm/kvm.c | 9 ---------
 1 file changed, 9 deletions(-)

diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index afd173514da1..7c751185491f 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -1207,9 +1207,6 @@ static int hyperv_handle_properties(CPUState *cs,
     }
 
     if (cpu->hyperv_passthrough) {
-        memcpy(cpuid_ent, &cpuid->entries[0],
-               cpuid->nent * sizeof(cpuid->entries[0]));
-
         c = cpuid_find_entry(cpuid, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, 0);
         if (c) {
             cpu->hyperv_vendor_id[0] = c->ebx;
@@ -1309,12 +1306,6 @@ static int hyperv_handle_properties(CPUState *cs,
         goto free;
     }
 
-    if (cpu->hyperv_passthrough) {
-        /* We already copied all feature words from KVM as is */
-        r = cpuid->nent;
-        goto free;
-    }
-
     c = &cpuid_ent[cpuid_i++];
     c->function = HV_CPUID_VENDOR_AND_MAX_FUNCTIONS;
     c->eax = hyperv_feat_enabled(cpu, HYPERV_FEAT_EVMCS) ?
-- 
2.30.2

[PATCH v6 04/19] i386: stop using env->features[] for filling Hyper-V CPUIDs

[Vitaly Kuznetsov]

As a preparatory patch to dropping Hyper-V CPUID leaves from
feature_word_info[] stop using env->features[] as a temporary
storage of Hyper-V CPUIDs, just build Hyper-V CPUID leaves directly
from kvm_hyperv_properties[] data.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 target/i386/cpu.h     |  1 +
 target/i386/kvm/kvm.c | 80 +++++++++++++++++++++++--------------------
 2 files changed, 43 insertions(+), 38 deletions(-)

diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index 570f916878f9..c8295aa2a1e7 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -1684,6 +1684,7 @@ struct X86CPU {
     uint32_t hyperv_interface_id[4];
     uint32_t hyperv_version_id[4];
     uint32_t hyperv_limits[3];
+    uint32_t hyperv_nested[4];
 
     bool check_cpuid;
     bool enforce_cpuid;
diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index 7c751185491f..f791baa29acf 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -1111,7 +1111,6 @@ static int hv_cpuid_check_and_set(CPUState *cs, struct kvm_cpuid2 *cpuid,
                                   int feature)
 {
     X86CPU *cpu = X86_CPU(cs);
-    CPUX86State *env = &cpu->env;
     uint32_t r, fw, bits;
     uint64_t deps;
     int i, dep_feat;
@@ -1151,8 +1150,6 @@ static int hv_cpuid_check_and_set(CPUState *cs, struct kvm_cpuid2 *cpuid,
                 return 0;
             }
         }
-
-        env->features[fw] |= bits;
     }
 
     if (cpu->hyperv_passthrough) {
@@ -1162,6 +1159,29 @@ static int hv_cpuid_check_and_set(CPUState *cs, struct kvm_cpuid2 *cpuid,
     return 0;
 }
 
+static uint32_t hv_build_cpuid_leaf(CPUState *cs, uint32_t fw)
+{
+    X86CPU *cpu = X86_CPU(cs);
+    uint32_t r = 0;
+    int i, j;
+
+    for (i = 0; i < ARRAY_SIZE(kvm_hyperv_properties); i++) {
+        if (!hyperv_feat_enabled(cpu, i)) {
+            continue;
+        }
+
+        for (j = 0; j < ARRAY_SIZE(kvm_hyperv_properties[i].flags); j++) {
+            if (kvm_hyperv_properties[i].flags[j].fw != fw) {
+                continue;
+            }
+
+            r |= kvm_hyperv_properties[i].flags[j].bits;
+        }
+    }
+
+    return r;
+}
+
 /*
  * Fill in Hyper-V CPUIDs. Returns the number of entries filled in cpuid_ent in
  * case of success, errno < 0 in case of failure and 0 when no Hyper-V
@@ -1171,9 +1191,8 @@ static int hyperv_handle_properties(CPUState *cs,
                                     struct kvm_cpuid_entry2 *cpuid_ent)
 {
     X86CPU *cpu = X86_CPU(cs);
-    CPUX86State *env = &cpu->env;
     struct kvm_cpuid2 *cpuid;
-    struct kvm_cpuid_entry2 *c;
+    struct kvm_cpuid_entry2 *c, *c2;
     uint32_t cpuid_i = 0;
     int r;
 
@@ -1194,9 +1213,7 @@ static int hyperv_handle_properties(CPUState *cs,
         }
 
         if (!r) {
-            env->features[FEAT_HV_RECOMM_EAX] |=
-                HV_ENLIGHTENED_VMCS_RECOMMENDED;
-            env->features[FEAT_HV_NESTED_EAX] = evmcs_version;
+            cpu->hyperv_nested[0] = evmcs_version;
         }
     }
 
@@ -1235,13 +1252,6 @@ static int hyperv_handle_properties(CPUState *cs,
             cpu->hyperv_version_id[3] = c->edx;
         }
 
-        c = cpuid_find_entry(cpuid, HV_CPUID_FEATURES, 0);
-        if (c) {
-            env->features[FEAT_HYPERV_EAX] = c->eax;
-            env->features[FEAT_HYPERV_EBX] = c->ebx;
-            env->features[FEAT_HYPERV_EDX] = c->edx;
-        }
-
         c = cpuid_find_entry(cpuid, HV_CPUID_IMPLEMENT_LIMITS, 0);
         if (c) {
             cpu->hv_max_vps = c->eax;
@@ -1252,23 +1262,8 @@ static int hyperv_handle_properties(CPUState *cs,
 
         c = cpuid_find_entry(cpuid, HV_CPUID_ENLIGHTMENT_INFO, 0);
         if (c) {
-            env->features[FEAT_HV_RECOMM_EAX] = c->eax;
             cpu->hyperv_spinlock_attempts = c->ebx;
         }
-        c = cpuid_find_entry(cpuid, HV_CPUID_NESTED_FEATURES, 0);
-        if (c) {
-            env->features[FEAT_HV_NESTED_EAX] = c->eax;
-        }
-    }
-
-    if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_ON) {
-        env->features[FEAT_HV_RECOMM_EAX] |= HV_NO_NONARCH_CORESHARING;
-    } else if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_AUTO) {
-        c = cpuid_find_entry(cpuid, HV_CPUID_ENLIGHTMENT_INFO, 0);
-        if (c) {
-            env->features[FEAT_HV_RECOMM_EAX] |=
-                c->eax & HV_NO_NONARCH_CORESHARING;
-        }
     }
 
     /* Features */
@@ -1298,9 +1293,6 @@ static int hyperv_handle_properties(CPUState *cs,
         r |= 1;
     }
 
-    /* Not exposed by KVM but needed to make CPU hotplug in Windows work */
-    env->features[FEAT_HYPERV_EDX] |= HV_CPU_DYNAMIC_PARTITIONING_AVAILABLE;
-
     if (r) {
         r = -ENOSYS;
         goto free;
@@ -1330,15 +1322,27 @@ static int hyperv_handle_properties(CPUState *cs,
 
     c = &cpuid_ent[cpuid_i++];
     c->function = HV_CPUID_FEATURES;
-    c->eax = env->features[FEAT_HYPERV_EAX];
-    c->ebx = env->features[FEAT_HYPERV_EBX];
-    c->edx = env->features[FEAT_HYPERV_EDX];
+    c->eax = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EAX);
+    c->ebx = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EBX);
+    c->edx = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EDX);
+
+    /* Not exposed by KVM but needed to make CPU hotplug in Windows work */
+    c->edx |= HV_CPU_DYNAMIC_PARTITIONING_AVAILABLE;
 
     c = &cpuid_ent[cpuid_i++];
     c->function = HV_CPUID_ENLIGHTMENT_INFO;
-    c->eax = env->features[FEAT_HV_RECOMM_EAX];
+    c->eax = hv_build_cpuid_leaf(cs, FEAT_HV_RECOMM_EAX);
     c->ebx = cpu->hyperv_spinlock_attempts;
 
+    if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_ON) {
+        c->eax |= HV_NO_NONARCH_CORESHARING;
+    } else if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_AUTO) {
+        c2 = cpuid_find_entry(cpuid, HV_CPUID_ENLIGHTMENT_INFO, 0);
+        if (c2) {
+            c->eax |= c2->eax & HV_NO_NONARCH_CORESHARING;
+        }
+    }
+
     c = &cpuid_ent[cpuid_i++];
     c->function = HV_CPUID_IMPLEMENT_LIMITS;
     c->eax = cpu->hv_max_vps;
@@ -1358,7 +1362,7 @@ static int hyperv_handle_properties(CPUState *cs,
 
         c = &cpuid_ent[cpuid_i++];
         c->function = HV_CPUID_NESTED_FEATURES;
-        c->eax = env->features[FEAT_HV_NESTED_EAX];
+        c->eax = cpu->hyperv_nested[0];
     }
     r = cpuid_i;
 
-- 
2.30.2

[PATCH v6 05/19] i386: introduce hyperv_feature_supported()

[Vitaly Kuznetsov]

Clean up hv_cpuid_check_and_set() by separating hyperv_feature_supported()
off it. No functional change intended.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 target/i386/kvm/kvm.c | 49 ++++++++++++++++++++++++++-----------------
 1 file changed, 30 insertions(+), 19 deletions(-)

diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index f791baa29acf..ba093dba4d23 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -1107,13 +1107,33 @@ static int hv_cpuid_get_fw(struct kvm_cpuid2 *cpuid, int fw, uint32_t *r)
     return 0;
 }
 
+static bool hyperv_feature_supported(struct kvm_cpuid2 *cpuid, int feature)
+{
+    uint32_t r, fw, bits;
+    int i;
+
+    for (i = 0; i < ARRAY_SIZE(kvm_hyperv_properties[feature].flags); i++) {
+        fw = kvm_hyperv_properties[feature].flags[i].fw;
+        bits = kvm_hyperv_properties[feature].flags[i].bits;
+
+        if (!fw) {
+            continue;
+        }
+
+        if (hv_cpuid_get_fw(cpuid, fw, &r) || (r & bits) != bits) {
+            return false;
+        }
+    }
+
+    return true;
+}
+
 static int hv_cpuid_check_and_set(CPUState *cs, struct kvm_cpuid2 *cpuid,
                                   int feature)
 {
     X86CPU *cpu = X86_CPU(cs);
-    uint32_t r, fw, bits;
     uint64_t deps;
-    int i, dep_feat;
+    int dep_feat;
 
     if (!hyperv_feat_enabled(cpu, feature) && !cpu->hyperv_passthrough) {
         return 0;
@@ -1132,23 +1152,14 @@ static int hv_cpuid_check_and_set(CPUState *cs, struct kvm_cpuid2 *cpuid,
         deps &= ~(1ull << dep_feat);
     }
 
-    for (i = 0; i < ARRAY_SIZE(kvm_hyperv_properties[feature].flags); i++) {
-        fw = kvm_hyperv_properties[feature].flags[i].fw;
-        bits = kvm_hyperv_properties[feature].flags[i].bits;
-
-        if (!fw) {
-            continue;
-        }
-
-        if (hv_cpuid_get_fw(cpuid, fw, &r) || (r & bits) != bits) {
-            if (hyperv_feat_enabled(cpu, feature)) {
-                fprintf(stderr,
-                        "Hyper-V %s is not supported by kernel\n",
-                        kvm_hyperv_properties[feature].desc);
-                return 1;
-            } else {
-                return 0;
-            }
+    if (!hyperv_feature_supported(cpuid, feature)) {
+        if (hyperv_feat_enabled(cpu, feature)) {
+            fprintf(stderr,
+                    "Hyper-V %s is not supported by kernel\n",
+                    kvm_hyperv_properties[feature].desc);
+            return 1;
+        } else {
+            return 0;
         }
     }
 
-- 
2.30.2

[PATCH v6 06/19] i386: introduce hv_cpuid_get_host()

[Vitaly Kuznetsov]

As a preparation to implementing hv_cpuid_cache intro introduce
hv_cpuid_get_host(). No functional change intended.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 target/i386/kvm/kvm.c | 102 +++++++++++++++++++++++-------------------
 1 file changed, 57 insertions(+), 45 deletions(-)

diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index ba093dba4d23..7aeb704b016e 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -1107,6 +1107,19 @@ static int hv_cpuid_get_fw(struct kvm_cpuid2 *cpuid, int fw, uint32_t *r)
     return 0;
 }
 
+static uint32_t hv_cpuid_get_host(struct kvm_cpuid2 *cpuid, uint32_t func,
+                                  int reg)
+{
+    struct kvm_cpuid_entry2 *entry;
+
+    entry = cpuid_find_entry(cpuid, func, 0);
+    if (!entry) {
+        return 0;
+    }
+
+    return cpuid_entry_get_reg(entry, reg);
+}
+
 static bool hyperv_feature_supported(struct kvm_cpuid2 *cpuid, int feature)
 {
     uint32_t r, fw, bits;
@@ -1203,7 +1216,7 @@ static int hyperv_handle_properties(CPUState *cs,
 {
     X86CPU *cpu = X86_CPU(cs);
     struct kvm_cpuid2 *cpuid;
-    struct kvm_cpuid_entry2 *c, *c2;
+    struct kvm_cpuid_entry2 *c;
     uint32_t cpuid_i = 0;
     int r;
 
@@ -1235,46 +1248,47 @@ static int hyperv_handle_properties(CPUState *cs,
     }
 
     if (cpu->hyperv_passthrough) {
-        c = cpuid_find_entry(cpuid, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, 0);
-        if (c) {
-            cpu->hyperv_vendor_id[0] = c->ebx;
-            cpu->hyperv_vendor_id[1] = c->ecx;
-            cpu->hyperv_vendor_id[2] = c->edx;
-            cpu->hyperv_vendor = g_realloc(cpu->hyperv_vendor,
-                                           sizeof(cpu->hyperv_vendor_id) + 1);
-            memcpy(cpu->hyperv_vendor, cpu->hyperv_vendor_id,
-                   sizeof(cpu->hyperv_vendor_id));
-            cpu->hyperv_vendor[sizeof(cpu->hyperv_vendor_id)] = 0;
-        }
-
-        c = cpuid_find_entry(cpuid, HV_CPUID_INTERFACE, 0);
-        if (c) {
-            cpu->hyperv_interface_id[0] = c->eax;
-            cpu->hyperv_interface_id[1] = c->ebx;
-            cpu->hyperv_interface_id[2] = c->ecx;
-            cpu->hyperv_interface_id[3] = c->edx;
-        }
-
-        c = cpuid_find_entry(cpuid, HV_CPUID_VERSION, 0);
-        if (c) {
-            cpu->hyperv_version_id[0] = c->eax;
-            cpu->hyperv_version_id[1] = c->ebx;
-            cpu->hyperv_version_id[2] = c->ecx;
-            cpu->hyperv_version_id[3] = c->edx;
-        }
-
-        c = cpuid_find_entry(cpuid, HV_CPUID_IMPLEMENT_LIMITS, 0);
-        if (c) {
-            cpu->hv_max_vps = c->eax;
-            cpu->hyperv_limits[0] = c->ebx;
-            cpu->hyperv_limits[1] = c->ecx;
-            cpu->hyperv_limits[2] = c->edx;
-        }
-
-        c = cpuid_find_entry(cpuid, HV_CPUID_ENLIGHTMENT_INFO, 0);
-        if (c) {
-            cpu->hyperv_spinlock_attempts = c->ebx;
-        }
+        cpu->hyperv_vendor_id[0] =
+            hv_cpuid_get_host(cpuid, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, R_EBX);
+        cpu->hyperv_vendor_id[1] =
+            hv_cpuid_get_host(cpuid, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, R_ECX);
+        cpu->hyperv_vendor_id[2] =
+            hv_cpuid_get_host(cpuid, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, R_EDX);
+        cpu->hyperv_vendor = g_realloc(cpu->hyperv_vendor,
+                                       sizeof(cpu->hyperv_vendor_id) + 1);
+        memcpy(cpu->hyperv_vendor, cpu->hyperv_vendor_id,
+               sizeof(cpu->hyperv_vendor_id));
+        cpu->hyperv_vendor[sizeof(cpu->hyperv_vendor_id)] = 0;
+
+        cpu->hyperv_interface_id[0] =
+            hv_cpuid_get_host(cpuid, HV_CPUID_INTERFACE, R_EAX);
+        cpu->hyperv_interface_id[1] =
+            hv_cpuid_get_host(cpuid, HV_CPUID_INTERFACE, R_EBX);
+        cpu->hyperv_interface_id[2] =
+            hv_cpuid_get_host(cpuid, HV_CPUID_INTERFACE, R_ECX);
+        cpu->hyperv_interface_id[3] =
+            hv_cpuid_get_host(cpuid, HV_CPUID_INTERFACE, R_EDX);
+
+        cpu->hyperv_version_id[0] =
+            hv_cpuid_get_host(cpuid, HV_CPUID_VERSION, R_EAX);
+        cpu->hyperv_version_id[1] =
+            hv_cpuid_get_host(cpuid, HV_CPUID_VERSION, R_EBX);
+        cpu->hyperv_version_id[2] =
+            hv_cpuid_get_host(cpuid, HV_CPUID_VERSION, R_ECX);
+        cpu->hyperv_version_id[3] =
+            hv_cpuid_get_host(cpuid, HV_CPUID_VERSION, R_EDX);
+
+        cpu->hv_max_vps = hv_cpuid_get_host(cpuid, HV_CPUID_IMPLEMENT_LIMITS,
+                                            R_EAX);
+        cpu->hyperv_limits[0] =
+            hv_cpuid_get_host(cpuid, HV_CPUID_IMPLEMENT_LIMITS, R_EBX);
+        cpu->hyperv_limits[1] =
+            hv_cpuid_get_host(cpuid, HV_CPUID_IMPLEMENT_LIMITS, R_ECX);
+        cpu->hyperv_limits[2] =
+            hv_cpuid_get_host(cpuid, HV_CPUID_IMPLEMENT_LIMITS, R_EDX);
+
+        cpu->hyperv_spinlock_attempts =
+            hv_cpuid_get_host(cpuid, HV_CPUID_ENLIGHTMENT_INFO, R_EBX);
     }
 
     /* Features */
@@ -1348,10 +1362,8 @@ static int hyperv_handle_properties(CPUState *cs,
     if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_ON) {
         c->eax |= HV_NO_NONARCH_CORESHARING;
     } else if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_AUTO) {
-        c2 = cpuid_find_entry(cpuid, HV_CPUID_ENLIGHTMENT_INFO, 0);
-        if (c2) {
-            c->eax |= c2->eax & HV_NO_NONARCH_CORESHARING;
-        }
+        c->eax |= hv_cpuid_get_host(cpuid, HV_CPUID_ENLIGHTMENT_INFO, R_EAX) &
+            HV_NO_NONARCH_CORESHARING;
     }
 
     c = &cpuid_ent[cpuid_i++];
-- 
2.30.2

[PATCH v6 07/19] i386: drop FEAT_HYPERV feature leaves

[Vitaly Kuznetsov]

Hyper-V feature leaves are weird. We have some of them in
feature_word_info[] array but we don't use feature_word_info
magic to enable them. Neither do we use feature_dependencies[]
mechanism to validate the configuration as it doesn't allign
well with Hyper-V's many-to-many dependency chains. Some of
the feature leaves hold not only feature bits, but also values.
E.g. FEAT_HV_NESTED_EAX contains both features and the supported
Enlightened VMCS range.

Hyper-V features are already represented in 'struct X86CPU' with
uint64_t hyperv_features so duplicating them in env->features adds
little (or zero) benefits. THe other half of Hyper-V emulation features
is also stored with values in hyperv_vendor_id[], hyperv_limits[],...
so env->features[] is already incomplete.

Remove Hyper-V feature leaves from env->features[] completely.
kvm_hyperv_properties[] is converted to using raw CPUID func/reg
pairs for features, this allows us to get rid of hv_cpuid_get_fw()
conversion.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 target/i386/cpu.c     |  90 +----------------------------------
 target/i386/cpu.h     |   5 --
 target/i386/kvm/kvm.c | 108 ++++++++++++++----------------------------
 3 files changed, 37 insertions(+), 166 deletions(-)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index 2d05df232329..d72b8760e7a3 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -832,94 +832,6 @@ static FeatureWordInfo feature_word_info[FEATURE_WORDS] = {
          */
         .no_autoenable_flags = ~0U,
     },
-    /*
-     * .feat_names are commented out for Hyper-V enlightenments because we
-     * don't want to have two different ways for enabling them on QEMU command
-     * line. Some features (e.g. "hyperv_time", "hyperv_vapic", ...) require
-     * enabling several feature bits simultaneously, exposing these bits
-     * individually may just confuse guests.
-     */
-    [FEAT_HYPERV_EAX] = {
-        .type = CPUID_FEATURE_WORD,
-        .feat_names = {
-            NULL /* hv_msr_vp_runtime_access */, NULL /* hv_msr_time_refcount_access */,
-            NULL /* hv_msr_synic_access */, NULL /* hv_msr_stimer_access */,
-            NULL /* hv_msr_apic_access */, NULL /* hv_msr_hypercall_access */,
-            NULL /* hv_vpindex_access */, NULL /* hv_msr_reset_access */,
-            NULL /* hv_msr_stats_access */, NULL /* hv_reftsc_access */,
-            NULL /* hv_msr_idle_access */, NULL /* hv_msr_frequency_access */,
-            NULL /* hv_msr_debug_access */, NULL /* hv_msr_reenlightenment_access */,
-            NULL, NULL,
-            NULL, NULL, NULL, NULL,
-            NULL, NULL, NULL, NULL,
-            NULL, NULL, NULL, NULL,
-            NULL, NULL, NULL, NULL,
-        },
-        .cpuid = { .eax = 0x40000003, .reg = R_EAX, },
-    },
-    [FEAT_HYPERV_EBX] = {
-        .type = CPUID_FEATURE_WORD,
-        .feat_names = {
-            NULL /* hv_create_partitions */, NULL /* hv_access_partition_id */,
-            NULL /* hv_access_memory_pool */, NULL /* hv_adjust_message_buffers */,
-            NULL /* hv_post_messages */, NULL /* hv_signal_events */,
-            NULL /* hv_create_port */, NULL /* hv_connect_port */,
-            NULL /* hv_access_stats */, NULL, NULL, NULL /* hv_debugging */,
-            NULL /* hv_cpu_power_management */, NULL /* hv_configure_profiler */,
-            NULL, NULL,
-            NULL, NULL, NULL, NULL,
-            NULL, NULL, NULL, NULL,
-            NULL, NULL, NULL, NULL,
-            NULL, NULL, NULL, NULL,
-        },
-        .cpuid = { .eax = 0x40000003, .reg = R_EBX, },
-    },
-    [FEAT_HYPERV_EDX] = {
-        .type = CPUID_FEATURE_WORD,
-        .feat_names = {
-            NULL /* hv_mwait */, NULL /* hv_guest_debugging */,
-            NULL /* hv_perf_monitor */, NULL /* hv_cpu_dynamic_part */,
-            NULL /* hv_hypercall_params_xmm */, NULL /* hv_guest_idle_state */,
-            NULL, NULL,
-            NULL, NULL, NULL /* hv_guest_crash_msr */, NULL,
-            NULL, NULL, NULL, NULL,
-            NULL, NULL, NULL, NULL,
-            NULL, NULL, NULL, NULL,
-            NULL, NULL, NULL, NULL,
-            NULL, NULL, NULL, NULL,
-        },
-        .cpuid = { .eax = 0x40000003, .reg = R_EDX, },
-    },
-    [FEAT_HV_RECOMM_EAX] = {
-        .type = CPUID_FEATURE_WORD,
-        .feat_names = {
-            NULL /* hv_recommend_pv_as_switch */,
-            NULL /* hv_recommend_pv_tlbflush_local */,
-            NULL /* hv_recommend_pv_tlbflush_remote */,
-            NULL /* hv_recommend_msr_apic_access */,
-            NULL /* hv_recommend_msr_reset */,
-            NULL /* hv_recommend_relaxed_timing */,
-            NULL /* hv_recommend_dma_remapping */,
-            NULL /* hv_recommend_int_remapping */,
-            NULL /* hv_recommend_x2apic_msrs */,
-            NULL /* hv_recommend_autoeoi_deprecation */,
-            NULL /* hv_recommend_pv_ipi */,
-            NULL /* hv_recommend_ex_hypercalls */,
-            NULL /* hv_hypervisor_is_nested */,
-            NULL /* hv_recommend_int_mbec */,
-            NULL /* hv_recommend_evmcs */,
-            NULL,
-            NULL, NULL, NULL, NULL,
-            NULL, NULL, NULL, NULL,
-            NULL, NULL, NULL, NULL,
-            NULL, NULL, NULL, NULL,
-        },
-        .cpuid = { .eax = 0x40000004, .reg = R_EAX, },
-    },
-    [FEAT_HV_NESTED_EAX] = {
-        .type = CPUID_FEATURE_WORD,
-        .cpuid = { .eax = 0x4000000A, .reg = R_EAX, },
-    },
     [FEAT_SVM] = {
         .type = CPUID_FEATURE_WORD,
         .feat_names = {
@@ -7062,7 +6974,7 @@ static GuestPanicInformation *x86_cpu_get_crash_info(CPUState *cs)
     CPUX86State *env = &cpu->env;
     GuestPanicInformation *panic_info = NULL;
 
-    if (env->features[FEAT_HYPERV_EDX] & HV_GUEST_CRASH_MSR_AVAILABLE) {
+    if (hyperv_feat_enabled(cpu, HYPERV_FEAT_CRASH)) {
         panic_info = g_malloc0(sizeof(GuestPanicInformation));
 
         panic_info->type = GUEST_PANIC_INFORMATION_TYPE_HYPER_V;
diff --git a/target/i386/cpu.h b/target/i386/cpu.h
index c8295aa2a1e7..8e8da25d8d84 100644
--- a/target/i386/cpu.h
+++ b/target/i386/cpu.h
@@ -518,11 +518,6 @@ typedef enum FeatureWord {
     FEAT_C000_0001_EDX, /* CPUID[C000_0001].EDX */
     FEAT_KVM,           /* CPUID[4000_0001].EAX (KVM_CPUID_FEATURES) */
     FEAT_KVM_HINTS,     /* CPUID[4000_0001].EDX */
-    FEAT_HYPERV_EAX,    /* CPUID[4000_0003].EAX */
-    FEAT_HYPERV_EBX,    /* CPUID[4000_0003].EBX */
-    FEAT_HYPERV_EDX,    /* CPUID[4000_0003].EDX */
-    FEAT_HV_RECOMM_EAX, /* CPUID[4000_0004].EAX */
-    FEAT_HV_NESTED_EAX, /* CPUID[4000_000A].EAX */
     FEAT_SVM,           /* CPUID[8000_000A].EDX */
     FEAT_XSAVE,         /* CPUID[EAX=0xd,ECX=1].EAX */
     FEAT_6_EAX,         /* CPUID[6].EAX */
diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index 7aeb704b016e..9dc085ffc9b2 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -800,7 +800,8 @@ static bool tsc_is_stable_and_known(CPUX86State *env)
 static struct {
     const char *desc;
     struct {
-        uint32_t fw;
+        uint32_t func;
+        int reg;
         uint32_t bits;
     } flags[2];
     uint64_t dependencies;
@@ -808,25 +809,25 @@ static struct {
     [HYPERV_FEAT_RELAXED] = {
         .desc = "relaxed timing (hv-relaxed)",
         .flags = {
-            {.fw = FEAT_HYPERV_EAX,
+            {.func = HV_CPUID_FEATURES, .reg = R_EAX,
              .bits = HV_HYPERCALL_AVAILABLE},
-            {.fw = FEAT_HV_RECOMM_EAX,
+            {.func = HV_CPUID_ENLIGHTMENT_INFO, .reg = R_EAX,
              .bits = HV_RELAXED_TIMING_RECOMMENDED}
         }
     },
     [HYPERV_FEAT_VAPIC] = {
         .desc = "virtual APIC (hv-vapic)",
         .flags = {
-            {.fw = FEAT_HYPERV_EAX,
+            {.func = HV_CPUID_FEATURES, .reg = R_EAX,
              .bits = HV_HYPERCALL_AVAILABLE | HV_APIC_ACCESS_AVAILABLE},
-            {.fw = FEAT_HV_RECOMM_EAX,
+            {.func = HV_CPUID_ENLIGHTMENT_INFO, .reg = R_EAX,
              .bits = HV_APIC_ACCESS_RECOMMENDED}
         }
     },
     [HYPERV_FEAT_TIME] = {
         .desc = "clocksources (hv-time)",
         .flags = {
-            {.fw = FEAT_HYPERV_EAX,
+            {.func = HV_CPUID_FEATURES, .reg = R_EAX,
              .bits = HV_HYPERCALL_AVAILABLE | HV_TIME_REF_COUNT_AVAILABLE |
              HV_REFERENCE_TSC_AVAILABLE}
         }
@@ -834,42 +835,42 @@ static struct {
     [HYPERV_FEAT_CRASH] = {
         .desc = "crash MSRs (hv-crash)",
         .flags = {
-            {.fw = FEAT_HYPERV_EDX,
+            {.func = HV_CPUID_FEATURES, .reg = R_EDX,
              .bits = HV_GUEST_CRASH_MSR_AVAILABLE}
         }
     },
     [HYPERV_FEAT_RESET] = {
         .desc = "reset MSR (hv-reset)",
         .flags = {
-            {.fw = FEAT_HYPERV_EAX,
+            {.func = HV_CPUID_FEATURES, .reg = R_EAX,
              .bits = HV_RESET_AVAILABLE}
         }
     },
     [HYPERV_FEAT_VPINDEX] = {
         .desc = "VP_INDEX MSR (hv-vpindex)",
         .flags = {
-            {.fw = FEAT_HYPERV_EAX,
+            {.func = HV_CPUID_FEATURES, .reg = R_EAX,
              .bits = HV_VP_INDEX_AVAILABLE}
         }
     },
     [HYPERV_FEAT_RUNTIME] = {
         .desc = "VP_RUNTIME MSR (hv-runtime)",
         .flags = {
-            {.fw = FEAT_HYPERV_EAX,
+            {.func = HV_CPUID_FEATURES, .reg = R_EAX,
              .bits = HV_VP_RUNTIME_AVAILABLE}
         }
     },
     [HYPERV_FEAT_SYNIC] = {
         .desc = "synthetic interrupt controller (hv-synic)",
         .flags = {
-            {.fw = FEAT_HYPERV_EAX,
+            {.func = HV_CPUID_FEATURES, .reg = R_EAX,
              .bits = HV_SYNIC_AVAILABLE}
         }
     },
     [HYPERV_FEAT_STIMER] = {
         .desc = "synthetic timers (hv-stimer)",
         .flags = {
-            {.fw = FEAT_HYPERV_EAX,
+            {.func = HV_CPUID_FEATURES, .reg = R_EAX,
              .bits = HV_SYNTIMERS_AVAILABLE}
         },
         .dependencies = BIT(HYPERV_FEAT_SYNIC) | BIT(HYPERV_FEAT_TIME)
@@ -877,23 +878,23 @@ static struct {
     [HYPERV_FEAT_FREQUENCIES] = {
         .desc = "frequency MSRs (hv-frequencies)",
         .flags = {
-            {.fw = FEAT_HYPERV_EAX,
+            {.func = HV_CPUID_FEATURES, .reg = R_EAX,
              .bits = HV_ACCESS_FREQUENCY_MSRS},
-            {.fw = FEAT_HYPERV_EDX,
+            {.func = HV_CPUID_FEATURES, .reg = R_EDX,
              .bits = HV_FREQUENCY_MSRS_AVAILABLE}
         }
     },
     [HYPERV_FEAT_REENLIGHTENMENT] = {
         .desc = "reenlightenment MSRs (hv-reenlightenment)",
         .flags = {
-            {.fw = FEAT_HYPERV_EAX,
+            {.func = HV_CPUID_FEATURES, .reg = R_EAX,
              .bits = HV_ACCESS_REENLIGHTENMENTS_CONTROL}
         }
     },
     [HYPERV_FEAT_TLBFLUSH] = {
         .desc = "paravirtualized TLB flush (hv-tlbflush)",
         .flags = {
-            {.fw = FEAT_HV_RECOMM_EAX,
+            {.func = HV_CPUID_ENLIGHTMENT_INFO, .reg = R_EAX,
              .bits = HV_REMOTE_TLB_FLUSH_RECOMMENDED |
              HV_EX_PROCESSOR_MASKS_RECOMMENDED}
         },
@@ -902,7 +903,7 @@ static struct {
     [HYPERV_FEAT_EVMCS] = {
         .desc = "enlightened VMCS (hv-evmcs)",
         .flags = {
-            {.fw = FEAT_HV_RECOMM_EAX,
+            {.func = HV_CPUID_ENLIGHTMENT_INFO, .reg = R_EAX,
              .bits = HV_ENLIGHTENED_VMCS_RECOMMENDED}
         },
         .dependencies = BIT(HYPERV_FEAT_VAPIC)
@@ -910,7 +911,7 @@ static struct {
     [HYPERV_FEAT_IPI] = {
         .desc = "paravirtualized IPI (hv-ipi)",
         .flags = {
-            {.fw = FEAT_HV_RECOMM_EAX,
+            {.func = HV_CPUID_ENLIGHTMENT_INFO, .reg = R_EAX,
              .bits = HV_CLUSTER_IPI_RECOMMENDED |
              HV_EX_PROCESSOR_MASKS_RECOMMENDED}
         },
@@ -919,7 +920,7 @@ static struct {
     [HYPERV_FEAT_STIMER_DIRECT] = {
         .desc = "direct mode synthetic timers (hv-stimer-direct)",
         .flags = {
-            {.fw = FEAT_HYPERV_EDX,
+            {.func = HV_CPUID_FEATURES, .reg = R_EDX,
              .bits = HV_STIMER_DIRECT_MODE_AVAILABLE}
         },
         .dependencies = BIT(HYPERV_FEAT_STIMER)
@@ -1065,48 +1066,6 @@ static struct kvm_cpuid2 *get_supported_hv_cpuid_legacy(CPUState *cs)
     return cpuid;
 }
 
-static int hv_cpuid_get_fw(struct kvm_cpuid2 *cpuid, int fw, uint32_t *r)
-{
-    struct kvm_cpuid_entry2 *entry;
-    uint32_t func;
-    int reg;
-
-    switch (fw) {
-    case FEAT_HYPERV_EAX:
-        reg = R_EAX;
-        func = HV_CPUID_FEATURES;
-        break;
-    case FEAT_HYPERV_EDX:
-        reg = R_EDX;
-        func = HV_CPUID_FEATURES;
-        break;
-    case FEAT_HV_RECOMM_EAX:
-        reg = R_EAX;
-        func = HV_CPUID_ENLIGHTMENT_INFO;
-        break;
-    default:
-        return -EINVAL;
-    }
-
-    entry = cpuid_find_entry(cpuid, func, 0);
-    if (!entry) {
-        return -ENOENT;
-    }
-
-    switch (reg) {
-    case R_EAX:
-        *r = entry->eax;
-        break;
-    case R_EDX:
-        *r = entry->edx;
-        break;
-    default:
-        return -EINVAL;
-    }
-
-    return 0;
-}
-
 static uint32_t hv_cpuid_get_host(struct kvm_cpuid2 *cpuid, uint32_t func,
                                   int reg)
 {
@@ -1122,18 +1081,20 @@ static uint32_t hv_cpuid_get_host(struct kvm_cpuid2 *cpuid, uint32_t func,
 
 static bool hyperv_feature_supported(struct kvm_cpuid2 *cpuid, int feature)
 {
-    uint32_t r, fw, bits;
-    int i;
+    uint32_t func, bits;
+    int i, reg;
 
     for (i = 0; i < ARRAY_SIZE(kvm_hyperv_properties[feature].flags); i++) {
-        fw = kvm_hyperv_properties[feature].flags[i].fw;
+
+        func = kvm_hyperv_properties[feature].flags[i].func;
+        reg = kvm_hyperv_properties[feature].flags[i].reg;
         bits = kvm_hyperv_properties[feature].flags[i].bits;
 
-        if (!fw) {
+        if (!func) {
             continue;
         }
 
-        if (hv_cpuid_get_fw(cpuid, fw, &r) || (r & bits) != bits) {
+        if ((hv_cpuid_get_host(cpuid, func, reg) & bits) != bits) {
             return false;
         }
     }
@@ -1183,7 +1144,7 @@ static int hv_cpuid_check_and_set(CPUState *cs, struct kvm_cpuid2 *cpuid,
     return 0;
 }
 
-static uint32_t hv_build_cpuid_leaf(CPUState *cs, uint32_t fw)
+static uint32_t hv_build_cpuid_leaf(CPUState *cs, uint32_t func, int reg)
 {
     X86CPU *cpu = X86_CPU(cs);
     uint32_t r = 0;
@@ -1195,7 +1156,10 @@ static uint32_t hv_build_cpuid_leaf(CPUState *cs, uint32_t fw)
         }
 
         for (j = 0; j < ARRAY_SIZE(kvm_hyperv_properties[i].flags); j++) {
-            if (kvm_hyperv_properties[i].flags[j].fw != fw) {
+            if (kvm_hyperv_properties[i].flags[j].func != func) {
+                continue;
+            }
+            if (kvm_hyperv_properties[i].flags[j].reg != reg) {
                 continue;
             }
 
@@ -1347,16 +1311,16 @@ static int hyperv_handle_properties(CPUState *cs,
 
     c = &cpuid_ent[cpuid_i++];
     c->function = HV_CPUID_FEATURES;
-    c->eax = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EAX);
-    c->ebx = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EBX);
-    c->edx = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EDX);
+    c->eax = hv_build_cpuid_leaf(cs, HV_CPUID_FEATURES, R_EAX);
+    c->ebx = hv_build_cpuid_leaf(cs, HV_CPUID_FEATURES, R_EBX);
+    c->edx = hv_build_cpuid_leaf(cs, HV_CPUID_FEATURES, R_EDX);
 
     /* Not exposed by KVM but needed to make CPU hotplug in Windows work */
     c->edx |= HV_CPU_DYNAMIC_PARTITIONING_AVAILABLE;
 
     c = &cpuid_ent[cpuid_i++];
     c->function = HV_CPUID_ENLIGHTMENT_INFO;
-    c->eax = hv_build_cpuid_leaf(cs, FEAT_HV_RECOMM_EAX);
+    c->eax = hv_build_cpuid_leaf(cs, HV_CPUID_ENLIGHTMENT_INFO, R_EAX);
     c->ebx = cpu->hyperv_spinlock_attempts;
 
     if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_ON) {
-- 
2.30.2

[PATCH v6 08/19] i386: introduce hv_cpuid_cache

[Vitaly Kuznetsov]

Just like with cpuid_cache, it makes no sense to call
KVM_GET_SUPPORTED_HV_CPUID more than once and instead of (ab)using
env->features[] and/or trying to keep all the code in one place, it is
better to introduce persistent hv_cpuid_cache and hv_cpuid_get_host()
accessor to it.

Note, hv_cpuid_get_fw() is converted to using hv_cpuid_get_host()
just to be removed later with Hyper-V specific feature words.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 target/i386/kvm/kvm.c | 109 ++++++++++++++++++++++--------------------
 1 file changed, 56 insertions(+), 53 deletions(-)

diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index 9dc085ffc9b2..3694c3f596e5 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -128,6 +128,7 @@ static int has_exception_payload;
 static bool has_msr_mcg_ext_ctl;
 
 static struct kvm_cpuid2 *cpuid_cache;
+static struct kvm_cpuid2 *hv_cpuid_cache;
 static struct kvm_msr_list *kvm_feature_msrs;
 
 int kvm_has_pit_state2(void)
@@ -1066,10 +1067,25 @@ static struct kvm_cpuid2 *get_supported_hv_cpuid_legacy(CPUState *cs)
     return cpuid;
 }
 
-static uint32_t hv_cpuid_get_host(struct kvm_cpuid2 *cpuid, uint32_t func,
-                                  int reg)
+static uint32_t hv_cpuid_get_host(CPUState *cs, uint32_t func, int reg)
 {
     struct kvm_cpuid_entry2 *entry;
+    struct kvm_cpuid2 *cpuid;
+
+    if (hv_cpuid_cache) {
+        cpuid = hv_cpuid_cache;
+    } else {
+        if (kvm_check_extension(kvm_state, KVM_CAP_HYPERV_CPUID) > 0) {
+            cpuid = get_supported_hv_cpuid(cs);
+        } else {
+            cpuid = get_supported_hv_cpuid_legacy(cs);
+        }
+        hv_cpuid_cache = cpuid;
+    }
+
+    if (!cpuid) {
+        return 0;
+    }
 
     entry = cpuid_find_entry(cpuid, func, 0);
     if (!entry) {
@@ -1079,7 +1095,7 @@ static uint32_t hv_cpuid_get_host(struct kvm_cpuid2 *cpuid, uint32_t func,
     return cpuid_entry_get_reg(entry, reg);
 }
 
-static bool hyperv_feature_supported(struct kvm_cpuid2 *cpuid, int feature)
+static bool hyperv_feature_supported(CPUState *cs, int feature)
 {
     uint32_t func, bits;
     int i, reg;
@@ -1094,7 +1110,7 @@ static bool hyperv_feature_supported(struct kvm_cpuid2 *cpuid, int feature)
             continue;
         }
 
-        if ((hv_cpuid_get_host(cpuid, func, reg) & bits) != bits) {
+        if ((hv_cpuid_get_host(cs, func, reg) & bits) != bits) {
             return false;
         }
     }
@@ -1102,8 +1118,7 @@ static bool hyperv_feature_supported(struct kvm_cpuid2 *cpuid, int feature)
     return true;
 }
 
-static int hv_cpuid_check_and_set(CPUState *cs, struct kvm_cpuid2 *cpuid,
-                                  int feature)
+static int hv_cpuid_check_and_set(CPUState *cs, int feature)
 {
     X86CPU *cpu = X86_CPU(cs);
     uint64_t deps;
@@ -1126,7 +1141,7 @@ static int hv_cpuid_check_and_set(CPUState *cs, struct kvm_cpuid2 *cpuid,
         deps &= ~(1ull << dep_feat);
     }
 
-    if (!hyperv_feature_supported(cpuid, feature)) {
+    if (!hyperv_feature_supported(cs, feature)) {
         if (hyperv_feat_enabled(cpu, feature)) {
             fprintf(stderr,
                     "Hyper-V %s is not supported by kernel\n",
@@ -1179,7 +1194,6 @@ static int hyperv_handle_properties(CPUState *cs,
                                     struct kvm_cpuid_entry2 *cpuid_ent)
 {
     X86CPU *cpu = X86_CPU(cs);
-    struct kvm_cpuid2 *cpuid;
     struct kvm_cpuid_entry2 *c;
     uint32_t cpuid_i = 0;
     int r;
@@ -1205,19 +1219,13 @@ static int hyperv_handle_properties(CPUState *cs,
         }
     }
 
-    if (kvm_check_extension(cs->kvm_state, KVM_CAP_HYPERV_CPUID) > 0) {
-        cpuid = get_supported_hv_cpuid(cs);
-    } else {
-        cpuid = get_supported_hv_cpuid_legacy(cs);
-    }
-
     if (cpu->hyperv_passthrough) {
         cpu->hyperv_vendor_id[0] =
-            hv_cpuid_get_host(cpuid, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, R_EBX);
+            hv_cpuid_get_host(cs, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, R_EBX);
         cpu->hyperv_vendor_id[1] =
-            hv_cpuid_get_host(cpuid, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, R_ECX);
+            hv_cpuid_get_host(cs, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, R_ECX);
         cpu->hyperv_vendor_id[2] =
-            hv_cpuid_get_host(cpuid, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, R_EDX);
+            hv_cpuid_get_host(cs, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, R_EDX);
         cpu->hyperv_vendor = g_realloc(cpu->hyperv_vendor,
                                        sizeof(cpu->hyperv_vendor_id) + 1);
         memcpy(cpu->hyperv_vendor, cpu->hyperv_vendor_id,
@@ -1225,52 +1233,52 @@ static int hyperv_handle_properties(CPUState *cs,
         cpu->hyperv_vendor[sizeof(cpu->hyperv_vendor_id)] = 0;
 
         cpu->hyperv_interface_id[0] =
-            hv_cpuid_get_host(cpuid, HV_CPUID_INTERFACE, R_EAX);
+            hv_cpuid_get_host(cs, HV_CPUID_INTERFACE, R_EAX);
         cpu->hyperv_interface_id[1] =
-            hv_cpuid_get_host(cpuid, HV_CPUID_INTERFACE, R_EBX);
+            hv_cpuid_get_host(cs, HV_CPUID_INTERFACE, R_EBX);
         cpu->hyperv_interface_id[2] =
-            hv_cpuid_get_host(cpuid, HV_CPUID_INTERFACE, R_ECX);
+            hv_cpuid_get_host(cs, HV_CPUID_INTERFACE, R_ECX);
         cpu->hyperv_interface_id[3] =
-            hv_cpuid_get_host(cpuid, HV_CPUID_INTERFACE, R_EDX);
+            hv_cpuid_get_host(cs, HV_CPUID_INTERFACE, R_EDX);
 
         cpu->hyperv_version_id[0] =
-            hv_cpuid_get_host(cpuid, HV_CPUID_VERSION, R_EAX);
+            hv_cpuid_get_host(cs, HV_CPUID_VERSION, R_EAX);
         cpu->hyperv_version_id[1] =
-            hv_cpuid_get_host(cpuid, HV_CPUID_VERSION, R_EBX);
+            hv_cpuid_get_host(cs, HV_CPUID_VERSION, R_EBX);
         cpu->hyperv_version_id[2] =
-            hv_cpuid_get_host(cpuid, HV_CPUID_VERSION, R_ECX);
+            hv_cpuid_get_host(cs, HV_CPUID_VERSION, R_ECX);
         cpu->hyperv_version_id[3] =
-            hv_cpuid_get_host(cpuid, HV_CPUID_VERSION, R_EDX);
+            hv_cpuid_get_host(cs, HV_CPUID_VERSION, R_EDX);
 
-        cpu->hv_max_vps = hv_cpuid_get_host(cpuid, HV_CPUID_IMPLEMENT_LIMITS,
+        cpu->hv_max_vps = hv_cpuid_get_host(cs, HV_CPUID_IMPLEMENT_LIMITS,
                                             R_EAX);
         cpu->hyperv_limits[0] =
-            hv_cpuid_get_host(cpuid, HV_CPUID_IMPLEMENT_LIMITS, R_EBX);
+            hv_cpuid_get_host(cs, HV_CPUID_IMPLEMENT_LIMITS, R_EBX);
         cpu->hyperv_limits[1] =
-            hv_cpuid_get_host(cpuid, HV_CPUID_IMPLEMENT_LIMITS, R_ECX);
+            hv_cpuid_get_host(cs, HV_CPUID_IMPLEMENT_LIMITS, R_ECX);
         cpu->hyperv_limits[2] =
-            hv_cpuid_get_host(cpuid, HV_CPUID_IMPLEMENT_LIMITS, R_EDX);
+            hv_cpuid_get_host(cs, HV_CPUID_IMPLEMENT_LIMITS, R_EDX);
 
         cpu->hyperv_spinlock_attempts =
-            hv_cpuid_get_host(cpuid, HV_CPUID_ENLIGHTMENT_INFO, R_EBX);
+            hv_cpuid_get_host(cs, HV_CPUID_ENLIGHTMENT_INFO, R_EBX);
     }
 
     /* Features */
-    r = hv_cpuid_check_and_set(cs, cpuid, HYPERV_FEAT_RELAXED);
-    r |= hv_cpuid_check_and_set(cs, cpuid, HYPERV_FEAT_VAPIC);
-    r |= hv_cpuid_check_and_set(cs, cpuid, HYPERV_FEAT_TIME);
-    r |= hv_cpuid_check_and_set(cs, cpuid, HYPERV_FEAT_CRASH);
-    r |= hv_cpuid_check_and_set(cs, cpuid, HYPERV_FEAT_RESET);
-    r |= hv_cpuid_check_and_set(cs, cpuid, HYPERV_FEAT_VPINDEX);
-    r |= hv_cpuid_check_and_set(cs, cpuid, HYPERV_FEAT_RUNTIME);
-    r |= hv_cpuid_check_and_set(cs, cpuid, HYPERV_FEAT_SYNIC);
-    r |= hv_cpuid_check_and_set(cs, cpuid, HYPERV_FEAT_STIMER);
-    r |= hv_cpuid_check_and_set(cs, cpuid, HYPERV_FEAT_FREQUENCIES);
-    r |= hv_cpuid_check_and_set(cs, cpuid, HYPERV_FEAT_REENLIGHTENMENT);
-    r |= hv_cpuid_check_and_set(cs, cpuid, HYPERV_FEAT_TLBFLUSH);
-    r |= hv_cpuid_check_and_set(cs, cpuid, HYPERV_FEAT_EVMCS);
-    r |= hv_cpuid_check_and_set(cs, cpuid, HYPERV_FEAT_IPI);
-    r |= hv_cpuid_check_and_set(cs, cpuid, HYPERV_FEAT_STIMER_DIRECT);
+    r = hv_cpuid_check_and_set(cs, HYPERV_FEAT_RELAXED);
+    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_VAPIC);
+    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_TIME);
+    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_CRASH);
+    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_RESET);
+    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_VPINDEX);
+    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_RUNTIME);
+    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_SYNIC);
+    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_STIMER);
+    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_FREQUENCIES);
+    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_REENLIGHTENMENT);
+    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_TLBFLUSH);
+    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_EVMCS);
+    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_IPI);
+    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_STIMER_DIRECT);
 
     /* Additional dependencies not covered by kvm_hyperv_properties[] */
     if (hyperv_feat_enabled(cpu, HYPERV_FEAT_SYNIC) &&
@@ -1283,8 +1291,7 @@ static int hyperv_handle_properties(CPUState *cs,
     }
 
     if (r) {
-        r = -ENOSYS;
-        goto free;
+        return -ENOSYS;
     }
 
     c = &cpuid_ent[cpuid_i++];
@@ -1326,7 +1333,7 @@ static int hyperv_handle_properties(CPUState *cs,
     if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_ON) {
         c->eax |= HV_NO_NONARCH_CORESHARING;
     } else if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_AUTO) {
-        c->eax |= hv_cpuid_get_host(cpuid, HV_CPUID_ENLIGHTMENT_INFO, R_EAX) &
+        c->eax |= hv_cpuid_get_host(cs, HV_CPUID_ENLIGHTMENT_INFO, R_EAX) &
             HV_NO_NONARCH_CORESHARING;
     }
 
@@ -1351,12 +1358,8 @@ static int hyperv_handle_properties(CPUState *cs,
         c->function = HV_CPUID_NESTED_FEATURES;
         c->eax = cpu->hyperv_nested[0];
     }
-    r = cpuid_i;
 
-free:
-    g_free(cpuid);
-
-    return r;
+    return cpuid_i;
 }
 
 static Error *hv_passthrough_mig_blocker;
-- 
2.30.2

[PATCH v6 09/19] i386: split hyperv_handle_properties() into hyperv_expand_features()/hyperv_fill_cpuids()

[Vitaly Kuznetsov]

The intention is to call hyperv_expand_features() early, before vCPUs
are created and use the acquired data later when we set guest visible
CPUID data.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 target/i386/kvm/kvm.c | 34 ++++++++++++++++++++++++----------
 1 file changed, 24 insertions(+), 10 deletions(-)

diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index 3694c3f596e5..6b391db7a030 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -1186,16 +1186,15 @@ static uint32_t hv_build_cpuid_leaf(CPUState *cs, uint32_t func, int reg)
 }
 
 /*
- * Fill in Hyper-V CPUIDs. Returns the number of entries filled in cpuid_ent in
- * case of success, errno < 0 in case of failure and 0 when no Hyper-V
- * extentions are enabled.
+ * Expand Hyper-V CPU features. In partucular, check that all the requested
+ * features are supported by the host and the sanity of the configuration
+ * (that all the required dependencies are included). Also, this takes care
+ * of 'hv_passthrough' mode and fills the environment with all supported
+ * Hyper-V features.
  */
-static int hyperv_handle_properties(CPUState *cs,
-                                    struct kvm_cpuid_entry2 *cpuid_ent)
+static int hyperv_expand_features(CPUState *cs)
 {
     X86CPU *cpu = X86_CPU(cs);
-    struct kvm_cpuid_entry2 *c;
-    uint32_t cpuid_i = 0;
     int r;
 
     if (!hyperv_enabled(cpu))
@@ -1294,6 +1293,19 @@ static int hyperv_handle_properties(CPUState *cs,
         return -ENOSYS;
     }
 
+    return 0;
+}
+
+/*
+ * Fill in Hyper-V CPUIDs. Returns the number of entries filled in cpuid_ent.
+ */
+static int hyperv_fill_cpuids(CPUState *cs,
+                              struct kvm_cpuid_entry2 *cpuid_ent)
+{
+    X86CPU *cpu = X86_CPU(cs);
+    struct kvm_cpuid_entry2 *c;
+    uint32_t cpuid_i = 0;
+
     c = &cpuid_ent[cpuid_i++];
     c->function = HV_CPUID_VENDOR_AND_MAX_FUNCTIONS;
     c->eax = hyperv_feat_enabled(cpu, HYPERV_FEAT_EVMCS) ?
@@ -1501,11 +1513,13 @@ int kvm_arch_init_vcpu(CPUState *cs)
     env->apic_bus_freq = KVM_APIC_BUS_FREQUENCY;
 
     /* Paravirtualization CPUIDs */
-    r = hyperv_handle_properties(cs, cpuid_data.entries);
+    r = hyperv_expand_features(cs);
     if (r < 0) {
         return r;
-    } else if (r > 0) {
-        cpuid_i = r;
+    }
+
+    if (hyperv_enabled(cpu)) {
+        cpuid_i = hyperv_fill_cpuids(cs, cpuid_data.entries);
         kvm_base = KVM_CPUID_SIGNATURE_NEXT;
         has_msr_hv_hypercall = true;
     }
-- 
2.30.2

[PATCH v6 10/19] i386: move eVMCS enablement to hyperv_init_vcpu()

[Vitaly Kuznetsov]

hyperv_expand_features() will be called before we create vCPU so
evmcs enablement should go away. hyperv_init_vcpu() looks like the
right place.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 target/i386/kvm/kvm.c | 60 ++++++++++++++++++++++++++-----------------
 1 file changed, 37 insertions(+), 23 deletions(-)

diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index 6b391db7a030..a2ef2dc154a2 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -962,6 +962,7 @@ static struct kvm_cpuid2 *get_supported_hv_cpuid(CPUState *cs)
 {
     struct kvm_cpuid2 *cpuid;
     int max = 7; /* 0x40000000..0x40000005, 0x4000000A */
+    int i;
 
     /*
      * When the buffer is too small, KVM_GET_SUPPORTED_HV_CPUID fails with
@@ -971,6 +972,22 @@ static struct kvm_cpuid2 *get_supported_hv_cpuid(CPUState *cs)
     while ((cpuid = try_get_hv_cpuid(cs, max)) == NULL) {
         max++;
     }
+
+    /*
+     * KVM_GET_SUPPORTED_HV_CPUID does not set EVMCS CPUID bit before
+     * KVM_CAP_HYPERV_ENLIGHTENED_VMCS is enabled but we want to get the
+     * information early, just check for the capability and set the bit
+     * manually.
+     */
+    if (kvm_check_extension(cs->kvm_state,
+                            KVM_CAP_HYPERV_ENLIGHTENED_VMCS) > 0) {
+        for (i = 0; i < cpuid->nent; i++) {
+            if (cpuid->entries[i].function == HV_CPUID_ENLIGHTMENT_INFO) {
+                cpuid->entries[i].eax |= HV_ENLIGHTENED_VMCS_RECOMMENDED;
+            }
+        }
+    }
+
     return cpuid;
 }
 
@@ -1200,24 +1217,6 @@ static int hyperv_expand_features(CPUState *cs)
     if (!hyperv_enabled(cpu))
         return 0;
 
-    if (hyperv_feat_enabled(cpu, HYPERV_FEAT_EVMCS) ||
-        cpu->hyperv_passthrough) {
-        uint16_t evmcs_version;
-
-        r = kvm_vcpu_enable_cap(cs, KVM_CAP_HYPERV_ENLIGHTENED_VMCS, 0,
-                                (uintptr_t)&evmcs_version);
-
-        if (hyperv_feat_enabled(cpu, HYPERV_FEAT_EVMCS) && r) {
-            fprintf(stderr, "Hyper-V %s is not supported by kernel\n",
-                    kvm_hyperv_properties[HYPERV_FEAT_EVMCS].desc);
-            return -ENOSYS;
-        }
-
-        if (!r) {
-            cpu->hyperv_nested[0] = evmcs_version;
-        }
-    }
-
     if (cpu->hyperv_passthrough) {
         cpu->hyperv_vendor_id[0] =
             hv_cpuid_get_host(cs, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, R_EBX);
@@ -1455,6 +1454,21 @@ static int hyperv_init_vcpu(X86CPU *cpu)
         }
     }
 
+    if (hyperv_feat_enabled(cpu, HYPERV_FEAT_EVMCS)) {
+        uint16_t evmcs_version;
+
+        ret = kvm_vcpu_enable_cap(cs, KVM_CAP_HYPERV_ENLIGHTENED_VMCS, 0,
+                                  (uintptr_t)&evmcs_version);
+
+        if (ret < 0) {
+            fprintf(stderr, "Hyper-V %s is not supported by kernel\n",
+                    kvm_hyperv_properties[HYPERV_FEAT_EVMCS].desc);
+            return ret;
+        }
+
+        cpu->hyperv_nested[0] = evmcs_version;
+    }
+
     return 0;
 }
 
@@ -1519,6 +1533,11 @@ int kvm_arch_init_vcpu(CPUState *cs)
     }
 
     if (hyperv_enabled(cpu)) {
+        r = hyperv_init_vcpu(cpu);
+        if (r) {
+            return r;
+        }
+
         cpuid_i = hyperv_fill_cpuids(cs, cpuid_data.entries);
         kvm_base = KVM_CPUID_SIGNATURE_NEXT;
         has_msr_hv_hypercall = true;
@@ -1868,11 +1887,6 @@ int kvm_arch_init_vcpu(CPUState *cs)
 
     kvm_init_msrs(cpu);
 
-    r = hyperv_init_vcpu(cpu);
-    if (r) {
-        goto fail;
-    }
-
     return 0;
 
  fail:
-- 
2.30.2

[PATCH v6 11/19] i386: switch hyperv_expand_features() to using error_setg()

[Vitaly Kuznetsov]

Use standard error_setg() mechanism in hyperv_expand_features().

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 target/i386/kvm/kvm.c | 101 +++++++++++++++++++++++++-----------------
 1 file changed, 61 insertions(+), 40 deletions(-)

diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index a2ef2dc154a2..f33ba325187f 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -1135,7 +1135,7 @@ static bool hyperv_feature_supported(CPUState *cs, int feature)
     return true;
 }
 
-static int hv_cpuid_check_and_set(CPUState *cs, int feature)
+static int hv_cpuid_check_and_set(CPUState *cs, int feature, Error **errp)
 {
     X86CPU *cpu = X86_CPU(cs);
     uint64_t deps;
@@ -1149,20 +1149,18 @@ static int hv_cpuid_check_and_set(CPUState *cs, int feature)
     while (deps) {
         dep_feat = ctz64(deps);
         if (!(hyperv_feat_enabled(cpu, dep_feat))) {
-                fprintf(stderr,
-                        "Hyper-V %s requires Hyper-V %s\n",
-                        kvm_hyperv_properties[feature].desc,
-                        kvm_hyperv_properties[dep_feat].desc);
-                return 1;
+            error_setg(errp, "Hyper-V %s requires Hyper-V %s",
+                       kvm_hyperv_properties[feature].desc,
+                       kvm_hyperv_properties[dep_feat].desc);
+            return 1;
         }
         deps &= ~(1ull << dep_feat);
     }
 
     if (!hyperv_feature_supported(cs, feature)) {
         if (hyperv_feat_enabled(cpu, feature)) {
-            fprintf(stderr,
-                    "Hyper-V %s is not supported by kernel\n",
-                    kvm_hyperv_properties[feature].desc);
+            error_setg(errp, "Hyper-V %s is not supported by kernel",
+                       kvm_hyperv_properties[feature].desc);
             return 1;
         } else {
             return 0;
@@ -1209,13 +1207,12 @@ static uint32_t hv_build_cpuid_leaf(CPUState *cs, uint32_t func, int reg)
  * of 'hv_passthrough' mode and fills the environment with all supported
  * Hyper-V features.
  */
-static int hyperv_expand_features(CPUState *cs)
+static void hyperv_expand_features(CPUState *cs, Error **errp)
 {
     X86CPU *cpu = X86_CPU(cs);
-    int r;
 
     if (!hyperv_enabled(cpu))
-        return 0;
+        return;
 
     if (cpu->hyperv_passthrough) {
         cpu->hyperv_vendor_id[0] =
@@ -1262,37 +1259,60 @@ static int hyperv_expand_features(CPUState *cs)
     }
 
     /* Features */
-    r = hv_cpuid_check_and_set(cs, HYPERV_FEAT_RELAXED);
-    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_VAPIC);
-    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_TIME);
-    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_CRASH);
-    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_RESET);
-    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_VPINDEX);
-    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_RUNTIME);
-    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_SYNIC);
-    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_STIMER);
-    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_FREQUENCIES);
-    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_REENLIGHTENMENT);
-    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_TLBFLUSH);
-    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_EVMCS);
-    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_IPI);
-    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_STIMER_DIRECT);
+    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_RELAXED, errp)) {
+        return;
+    }
+    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_VAPIC, errp)) {
+        return;
+    }
+    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_TIME, errp)) {
+        return;
+    }
+    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_CRASH, errp)) {
+        return;
+    }
+    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_RESET, errp)) {
+        return;
+    }
+    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_VPINDEX, errp)) {
+        return;
+    }
+    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_RUNTIME, errp)) {
+        return;
+    }
+    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_SYNIC, errp)) {
+        return;
+    }
+    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_STIMER, errp)) {
+        return;
+    }
+    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_FREQUENCIES, errp)) {
+        return;
+    }
+    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_REENLIGHTENMENT, errp)) {
+        return;
+    }
+    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_TLBFLUSH, errp)) {
+        return;
+    }
+    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_EVMCS, errp)) {
+        return;
+    }
+    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_IPI, errp)) {
+        return;
+    }
+    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_STIMER_DIRECT, errp)) {
+        return;
+    }
 
     /* Additional dependencies not covered by kvm_hyperv_properties[] */
     if (hyperv_feat_enabled(cpu, HYPERV_FEAT_SYNIC) &&
         !cpu->hyperv_synic_kvm_only &&
         !hyperv_feat_enabled(cpu, HYPERV_FEAT_VPINDEX)) {
-        fprintf(stderr, "Hyper-V %s requires Hyper-V %s\n",
-                kvm_hyperv_properties[HYPERV_FEAT_SYNIC].desc,
-                kvm_hyperv_properties[HYPERV_FEAT_VPINDEX].desc);
-        r |= 1;
-    }
-
-    if (r) {
-        return -ENOSYS;
+        error_setg(errp, "Hyper-V %s requires Hyper-V %s",
+                   kvm_hyperv_properties[HYPERV_FEAT_SYNIC].desc,
+                   kvm_hyperv_properties[HYPERV_FEAT_VPINDEX].desc);
     }
-
-    return 0;
 }
 
 /*
@@ -1527,9 +1547,10 @@ int kvm_arch_init_vcpu(CPUState *cs)
     env->apic_bus_freq = KVM_APIC_BUS_FREQUENCY;
 
     /* Paravirtualization CPUIDs */
-    r = hyperv_expand_features(cs);
-    if (r < 0) {
-        return r;
+    hyperv_expand_features(cs, &local_err);
+    if (local_err) {
+        error_report_err(local_err);
+        return -ENOSYS;
     }
 
     if (hyperv_enabled(cpu)) {
-- 
2.30.2

[PATCH v6 12/19] i386: adjust the expected KVM_GET_SUPPORTED_HV_CPUID array size

[Vitaly Kuznetsov]

SYNDBG leaves were recently (Linux-5.8) added to KVM but we haven't
updated the expected size of KVM_GET_SUPPORTED_HV_CPUID output in
KVM so we now make serveral tries before succeeding. Update the
default.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 target/i386/kvm/kvm.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index f33ba325187f..feec9f25ba12 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -961,7 +961,8 @@ static struct kvm_cpuid2 *try_get_hv_cpuid(CPUState *cs, int max)
 static struct kvm_cpuid2 *get_supported_hv_cpuid(CPUState *cs)
 {
     struct kvm_cpuid2 *cpuid;
-    int max = 7; /* 0x40000000..0x40000005, 0x4000000A */
+    /* 0x40000000..0x40000005, 0x4000000A, 0x40000080..0x40000080 leaves */
+    int max = 10;
     int i;
 
     /*
-- 
2.30.2

[PATCH v6 13/19] i386: prefer system KVM_GET_SUPPORTED_HV_CPUID ioctl over vCPU's one

[Vitaly Kuznetsov]

KVM_GET_SUPPORTED_HV_CPUID was made a system wide ioctl which can be called
prior to creating vCPUs and we are going to use that to expand Hyper-V cpu
features early. Use it when it is supported by KVM.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 target/i386/kvm/kvm.c | 17 +++++++++++++----
 1 file changed, 13 insertions(+), 4 deletions(-)

diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index feec9f25ba12..5d08f3a39ef7 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -928,7 +928,8 @@ static struct {
     },
 };
 
-static struct kvm_cpuid2 *try_get_hv_cpuid(CPUState *cs, int max)
+static struct kvm_cpuid2 *try_get_hv_cpuid(CPUState *cs, int max,
+                                           bool do_sys_ioctl)
 {
     struct kvm_cpuid2 *cpuid;
     int r, size;
@@ -937,7 +938,11 @@ static struct kvm_cpuid2 *try_get_hv_cpuid(CPUState *cs, int max)
     cpuid = g_malloc0(size);
     cpuid->nent = max;
 
-    r = kvm_vcpu_ioctl(cs, KVM_GET_SUPPORTED_HV_CPUID, cpuid);
+    if (do_sys_ioctl) {
+        r = kvm_ioctl(kvm_state, KVM_GET_SUPPORTED_HV_CPUID, cpuid);
+    } else {
+        r = kvm_vcpu_ioctl(cs, KVM_GET_SUPPORTED_HV_CPUID, cpuid);
+    }
     if (r == 0 && cpuid->nent >= max) {
         r = -E2BIG;
     }
@@ -964,13 +969,17 @@ static struct kvm_cpuid2 *get_supported_hv_cpuid(CPUState *cs)
     /* 0x40000000..0x40000005, 0x4000000A, 0x40000080..0x40000080 leaves */
     int max = 10;
     int i;
+    bool do_sys_ioctl;
+
+    do_sys_ioctl =
+        kvm_check_extension(kvm_state, KVM_CAP_SYS_HYPERV_CPUID) > 0;
 
     /*
      * When the buffer is too small, KVM_GET_SUPPORTED_HV_CPUID fails with
      * -E2BIG, however, it doesn't report back the right size. Keep increasing
      * it and re-trying until we succeed.
      */
-    while ((cpuid = try_get_hv_cpuid(cs, max)) == NULL) {
+    while ((cpuid = try_get_hv_cpuid(cs, max, do_sys_ioctl)) == NULL) {
         max++;
     }
 
@@ -980,7 +989,7 @@ static struct kvm_cpuid2 *get_supported_hv_cpuid(CPUState *cs)
      * information early, just check for the capability and set the bit
      * manually.
      */
-    if (kvm_check_extension(cs->kvm_state,
+    if (!do_sys_ioctl && kvm_check_extension(cs->kvm_state,
                             KVM_CAP_HYPERV_ENLIGHTENED_VMCS) > 0) {
         for (i = 0; i < cpuid->nent; i++) {
             if (cpuid->entries[i].function == HV_CPUID_ENLIGHTMENT_INFO) {
-- 
2.30.2

[PATCH v6 14/19] i386: use global kvm_state in hyperv_enabled() check

[Vitaly Kuznetsov]

There is no need to use vCPU-specific kvm state in hyperv_enabled() check
and we need to do that when feature expansion happens early, before vCPU
specific KVM state is created.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 target/i386/kvm/kvm.c | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index 5d08f3a39ef7..a42263b24fca 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -715,8 +715,7 @@ unsigned long kvm_arch_vcpu_id(CPUState *cs)
 
 static bool hyperv_enabled(X86CPU *cpu)
 {
-    CPUState *cs = CPU(cpu);
-    return kvm_check_extension(cs->kvm_state, KVM_CAP_HYPERV) > 0 &&
+    return kvm_check_extension(kvm_state, KVM_CAP_HYPERV) > 0 &&
         ((cpu->hyperv_spinlock_attempts != HYPERV_SPINLOCK_NEVER_NOTIFY) ||
          cpu->hyperv_features || cpu->hyperv_passthrough);
 }
-- 
2.30.2

[PATCH v6 15/19] i386: expand Hyper-V features during CPU feature expansion time

[Vitaly Kuznetsov]

To make Hyper-V features appear in e.g. QMP query-cpu-model-expansion we
need to expand and set the corresponding CPUID leaves early. Modify
x86_cpu_get_supported_feature_word() to call newly intoduced Hyper-V
specific kvm_hv_get_supported_cpuid() instead of
kvm_arch_get_supported_cpuid(). We can't use kvm_arch_get_supported_cpuid()
as Hyper-V specific CPUID leaves intersect with KVM's.

Note, early expansion will only happen when KVM supports system wide
KVM_GET_SUPPORTED_HV_CPUID ioctl (KVM_CAP_SYS_HYPERV_CPUID).

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 target/i386/cpu.c          |  4 ++++
 target/i386/kvm/kvm-stub.c |  5 +++++
 target/i386/kvm/kvm.c      | 15 ++++++++++++---
 target/i386/kvm/kvm_i386.h |  1 +
 4 files changed, 22 insertions(+), 3 deletions(-)

diff --git a/target/i386/cpu.c b/target/i386/cpu.c
index d72b8760e7a3..18b57f3d8b9c 100644
--- a/target/i386/cpu.c
+++ b/target/i386/cpu.c
@@ -6514,6 +6514,10 @@ static void x86_cpu_expand_features(X86CPU *cpu, Error **errp)
     if (env->cpuid_xlevel2 == UINT32_MAX) {
         env->cpuid_xlevel2 = env->cpuid_min_xlevel2;
     }
+
+    if (kvm_enabled()) {
+        kvm_hyperv_expand_features(cpu, errp);
+    }
 }
 
 /*
diff --git a/target/i386/kvm/kvm-stub.c b/target/i386/kvm/kvm-stub.c
index 92f49121b8fa..7f175faa3abd 100644
--- a/target/i386/kvm/kvm-stub.c
+++ b/target/i386/kvm/kvm-stub.c
@@ -39,3 +39,8 @@ bool kvm_hv_vpindex_settable(void)
 {
     return false;
 }
+
+void kvm_hyperv_expand_features(X86CPU *cpu, Error **errp)
+{
+    return;
+}
diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index a42263b24fca..d5551c4ab5cf 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -1216,13 +1216,22 @@ static uint32_t hv_build_cpuid_leaf(CPUState *cs, uint32_t func, int reg)
  * of 'hv_passthrough' mode and fills the environment with all supported
  * Hyper-V features.
  */
-static void hyperv_expand_features(CPUState *cs, Error **errp)
+void kvm_hyperv_expand_features(X86CPU *cpu, Error **errp)
 {
-    X86CPU *cpu = X86_CPU(cs);
+    CPUState *cs = CPU(cpu);
 
     if (!hyperv_enabled(cpu))
         return;
 
+    /*
+     * When kvm_hyperv_expand_features is called at CPU feature expansion
+     * time per-CPU kvm_state is not available yet so we can only proceed
+     * when KVM_CAP_SYS_HYPERV_CPUID is supported.
+     */
+    if (!cs->kvm_state &&
+        !kvm_check_extension(kvm_state, KVM_CAP_SYS_HYPERV_CPUID))
+        return;
+
     if (cpu->hyperv_passthrough) {
         cpu->hyperv_vendor_id[0] =
             hv_cpuid_get_host(cs, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, R_EBX);
@@ -1556,7 +1565,7 @@ int kvm_arch_init_vcpu(CPUState *cs)
     env->apic_bus_freq = KVM_APIC_BUS_FREQUENCY;
 
     /* Paravirtualization CPUIDs */
-    hyperv_expand_features(cs, &local_err);
+    kvm_hyperv_expand_features(cpu, &local_err);
     if (local_err) {
         error_report_err(local_err);
         return -ENOSYS;
diff --git a/target/i386/kvm/kvm_i386.h b/target/i386/kvm/kvm_i386.h
index dc725083891c..f1176491051d 100644
--- a/target/i386/kvm/kvm_i386.h
+++ b/target/i386/kvm/kvm_i386.h
@@ -47,6 +47,7 @@ bool kvm_has_x2apic_api(void);
 bool kvm_has_waitpkg(void);
 
 bool kvm_hv_vpindex_settable(void);
+void kvm_hyperv_expand_features(X86CPU *cpu, Error **errp);
 
 uint64_t kvm_swizzle_msi_ext_dest_id(uint64_t address);
 
-- 
2.30.2

[PATCH v6 16/19] i386: kill off hv_cpuid_check_and_set()

[Vitaly Kuznetsov]

hv_cpuid_check_and_set() does too much:
- Checks if the feature is supported by KVM;
- Checks if all dependencies are enabled;
- Sets the feature bit in cpu->hyperv_features for 'passthrough' mode.

To reduce the complexity, move all the logic except for dependencies
check out of it. Also, in 'passthrough' mode we don't really need to
check dependencies because KVM is supposed to provide a consistent
set anyway.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 target/i386/kvm/kvm.c | 105 +++++++++++++++---------------------------
 1 file changed, 36 insertions(+), 69 deletions(-)

diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index d5551c4ab5cf..2c1a77f9b00f 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -1144,16 +1144,12 @@ static bool hyperv_feature_supported(CPUState *cs, int feature)
     return true;
 }
 
-static int hv_cpuid_check_and_set(CPUState *cs, int feature, Error **errp)
+/* Checks that all feature dependencies are enabled */
+static void hv_feature_check_deps(X86CPU *cpu, int feature, Error **errp)
 {
-    X86CPU *cpu = X86_CPU(cs);
     uint64_t deps;
     int dep_feat;
 
-    if (!hyperv_feat_enabled(cpu, feature) && !cpu->hyperv_passthrough) {
-        return 0;
-    }
-
     deps = kvm_hyperv_properties[feature].dependencies;
     while (deps) {
         dep_feat = ctz64(deps);
@@ -1161,26 +1157,10 @@ static int hv_cpuid_check_and_set(CPUState *cs, int feature, Error **errp)
             error_setg(errp, "Hyper-V %s requires Hyper-V %s",
                        kvm_hyperv_properties[feature].desc,
                        kvm_hyperv_properties[dep_feat].desc);
-            return 1;
+            return;
         }
         deps &= ~(1ull << dep_feat);
     }
-
-    if (!hyperv_feature_supported(cs, feature)) {
-        if (hyperv_feat_enabled(cpu, feature)) {
-            error_setg(errp, "Hyper-V %s is not supported by kernel",
-                       kvm_hyperv_properties[feature].desc);
-            return 1;
-        } else {
-            return 0;
-        }
-    }
-
-    if (cpu->hyperv_passthrough) {
-        cpu->hyperv_features |= BIT(feature);
-    }
-
-    return 0;
 }
 
 static uint32_t hv_build_cpuid_leaf(CPUState *cs, uint32_t func, int reg)
@@ -1219,6 +1199,8 @@ static uint32_t hv_build_cpuid_leaf(CPUState *cs, uint32_t func, int reg)
 void kvm_hyperv_expand_features(X86CPU *cpu, Error **errp)
 {
     CPUState *cs = CPU(cpu);
+    Error *local_err = NULL;
+    int feat;
 
     if (!hyperv_enabled(cpu))
         return;
@@ -1274,53 +1256,38 @@ void kvm_hyperv_expand_features(X86CPU *cpu, Error **errp)
 
         cpu->hyperv_spinlock_attempts =
             hv_cpuid_get_host(cs, HV_CPUID_ENLIGHTMENT_INFO, R_EBX);
-    }
 
-    /* Features */
-    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_RELAXED, errp)) {
-        return;
-    }
-    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_VAPIC, errp)) {
-        return;
-    }
-    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_TIME, errp)) {
-        return;
-    }
-    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_CRASH, errp)) {
-        return;
-    }
-    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_RESET, errp)) {
-        return;
-    }
-    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_VPINDEX, errp)) {
-        return;
-    }
-    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_RUNTIME, errp)) {
-        return;
-    }
-    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_SYNIC, errp)) {
-        return;
-    }
-    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_STIMER, errp)) {
-        return;
-    }
-    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_FREQUENCIES, errp)) {
-        return;
-    }
-    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_REENLIGHTENMENT, errp)) {
-        return;
-    }
-    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_TLBFLUSH, errp)) {
-        return;
-    }
-    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_EVMCS, errp)) {
-        return;
-    }
-    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_IPI, errp)) {
-        return;
-    }
-    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_STIMER_DIRECT, errp)) {
-        return;
+        /*
+         * Mark feature as enabled in 'cpu->hyperv_features' as
+         * hv_build_cpuid_leaf() uses this info to build guest CPUIDs.
+         */
+        for (feat = 0; feat < ARRAY_SIZE(kvm_hyperv_properties); feat++) {
+            if (hyperv_feature_supported(cs, feat)) {
+                cpu->hyperv_features |= BIT(feat);
+            }
+        }
+    } else {
+        /* Check features availability and dependencies */
+        for (feat = 0; feat < ARRAY_SIZE(kvm_hyperv_properties); feat++) {
+            /* If the feature was not requested skip it. */
+            if (!hyperv_feat_enabled(cpu, feat)) {
+                continue;
+            }
+
+            /* Check if the feature is supported by KVM */
+            if (!hyperv_feature_supported(cs, feat)) {
+                error_setg(errp, "Hyper-V %s is not supported by kernel",
+                           kvm_hyperv_properties[feat].desc);
+                return;
+            }
+
+            /* Check dependencies */
+            hv_feature_check_deps(cpu, feat, &local_err);
+            if (local_err != NULL) {
+                error_propagate(errp, local_err);
+                return;
+            }
+        }
     }
 
     /* Additional dependencies not covered by kvm_hyperv_properties[] */
-- 
2.30.2

[PATCH v6 17/19] i386: HV_HYPERCALL_AVAILABLE privilege bit is always needed

[Vitaly Kuznetsov]

According to TLFS, Hyper-V guest is supposed to check
HV_HYPERCALL_AVAILABLE privilege bit before accessing
HV_X64_MSR_GUEST_OS_ID/HV_X64_MSR_HYPERCALL MSRs but at least some
Windows versions ignore that. As KVM is very permissive and allows
accessing these MSRs unconditionally, no issue is observed. We may,
however, want to tighten the checks eventually. Conforming to the
spec is probably also a good idea.

Add HV_HYPERCALL_AVAILABLE to all 'leaf' features with no dependencies.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 target/i386/kvm/kvm.c | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index 2c1a77f9b00f..d81451276cd8 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -835,6 +835,8 @@ static struct {
     [HYPERV_FEAT_CRASH] = {
         .desc = "crash MSRs (hv-crash)",
         .flags = {
+            {.func = HV_CPUID_FEATURES, .reg = R_EAX,
+             .bits = HV_HYPERCALL_AVAILABLE},
             {.func = HV_CPUID_FEATURES, .reg = R_EDX,
              .bits = HV_GUEST_CRASH_MSR_AVAILABLE}
         }
@@ -843,28 +845,28 @@ static struct {
         .desc = "reset MSR (hv-reset)",
         .flags = {
             {.func = HV_CPUID_FEATURES, .reg = R_EAX,
-             .bits = HV_RESET_AVAILABLE}
+             .bits = HV_HYPERCALL_AVAILABLE | HV_RESET_AVAILABLE}
         }
     },
     [HYPERV_FEAT_VPINDEX] = {
         .desc = "VP_INDEX MSR (hv-vpindex)",
         .flags = {
             {.func = HV_CPUID_FEATURES, .reg = R_EAX,
-             .bits = HV_VP_INDEX_AVAILABLE}
+             .bits = HV_HYPERCALL_AVAILABLE | HV_VP_INDEX_AVAILABLE}
         }
     },
     [HYPERV_FEAT_RUNTIME] = {
         .desc = "VP_RUNTIME MSR (hv-runtime)",
         .flags = {
             {.func = HV_CPUID_FEATURES, .reg = R_EAX,
-             .bits = HV_VP_RUNTIME_AVAILABLE}
+             .bits = HV_HYPERCALL_AVAILABLE | HV_VP_RUNTIME_AVAILABLE}
         }
     },
     [HYPERV_FEAT_SYNIC] = {
         .desc = "synthetic interrupt controller (hv-synic)",
         .flags = {
             {.func = HV_CPUID_FEATURES, .reg = R_EAX,
-             .bits = HV_SYNIC_AVAILABLE}
+             .bits = HV_HYPERCALL_AVAILABLE | HV_SYNIC_AVAILABLE}
         }
     },
     [HYPERV_FEAT_STIMER] = {
@@ -879,7 +881,7 @@ static struct {
         .desc = "frequency MSRs (hv-frequencies)",
         .flags = {
             {.func = HV_CPUID_FEATURES, .reg = R_EAX,
-             .bits = HV_ACCESS_FREQUENCY_MSRS},
+             .bits = HV_HYPERCALL_AVAILABLE | HV_ACCESS_FREQUENCY_MSRS},
             {.func = HV_CPUID_FEATURES, .reg = R_EDX,
              .bits = HV_FREQUENCY_MSRS_AVAILABLE}
         }
@@ -888,7 +890,8 @@ static struct {
         .desc = "reenlightenment MSRs (hv-reenlightenment)",
         .flags = {
             {.func = HV_CPUID_FEATURES, .reg = R_EAX,
-             .bits = HV_ACCESS_REENLIGHTENMENTS_CONTROL}
+             .bits = HV_HYPERCALL_AVAILABLE |
+             HV_ACCESS_REENLIGHTENMENTS_CONTROL}
         }
     },
     [HYPERV_FEAT_TLBFLUSH] = {
-- 
2.30.2

[PATCH v6 18/19] i386: Hyper-V SynIC requires POST_MESSAGES/SIGNAL_EVENTS priviliges

[Vitaly Kuznetsov]

When Hyper-V SynIC has its QEMU part enabled (no 'x-hv-synic-kvm-only'),
we may need to allow Windows guest to make hypercalls (POST_MESSAGES/
SIGNAL_EVENTS). No issue is currently observed because KVM is very
permissive, allowing these hypercalls regarding of guest visible CPUid
bits.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 target/i386/kvm/hyperv-proto.h | 6 ++++++
 target/i386/kvm/kvm.c          | 5 +++++
 2 files changed, 11 insertions(+)

diff --git a/target/i386/kvm/hyperv-proto.h b/target/i386/kvm/hyperv-proto.h
index e30d64b4ade4..5fbb385cc136 100644
--- a/target/i386/kvm/hyperv-proto.h
+++ b/target/i386/kvm/hyperv-proto.h
@@ -38,6 +38,12 @@
 #define HV_ACCESS_FREQUENCY_MSRS     (1u << 11)
 #define HV_ACCESS_REENLIGHTENMENTS_CONTROL  (1u << 13)
 
+/*
+ * HV_CPUID_FEATURES.EBX bits
+ */
+#define HV_POST_MESSAGES             (1u << 4)
+#define HV_SIGNAL_EVENTS             (1u << 5)
+
 /*
  * HV_CPUID_FEATURES.EDX bits
  */
diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
index d81451276cd8..88c64909a2ff 100644
--- a/target/i386/kvm/kvm.c
+++ b/target/i386/kvm/kvm.c
@@ -1341,6 +1341,11 @@ static int hyperv_fill_cpuids(CPUState *cs,
     c->ebx = hv_build_cpuid_leaf(cs, HV_CPUID_FEATURES, R_EBX);
     c->edx = hv_build_cpuid_leaf(cs, HV_CPUID_FEATURES, R_EDX);
 
+    /* In-QEMU SynIC and Vmbus devices require messages/signals hypercalls */
+    if (!cpu->hyperv_synic_kvm_only) {
+        c->ebx |= HV_POST_MESSAGES | HV_SIGNAL_EVENTS;
+    }
+
     /* Not exposed by KVM but needed to make CPU hotplug in Windows work */
     c->edx |= HV_CPU_DYNAMIC_PARTITIONING_AVAILABLE;
 
-- 
2.30.2

[PATCH v6 19/19] qtest/hyperv: Introduce a simple hyper-v test

[Vitaly Kuznetsov]

For the beginning, just test 'hv-passthrough' and a couple of custom
Hyper-V  enlightenments configurations through QMP. Later, it would
be great to complement this by checking CPUID values from within the
guest.

Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
---
 MAINTAINERS               |   1 +
 tests/qtest/hyperv-test.c | 225 ++++++++++++++++++++++++++++++++++++++
 tests/qtest/meson.build   |   3 +-
 3 files changed, 228 insertions(+), 1 deletion(-)
 create mode 100644 tests/qtest/hyperv-test.c

diff --git a/MAINTAINERS b/MAINTAINERS
index 36055f14c594..86d731e86f4a 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1552,6 +1552,7 @@ F: hw/isa/apm.c
 F: include/hw/isa/apm.h
 F: tests/unit/test-x86-cpuid.c
 F: tests/qtest/test-x86-cpuid-compat.c
+F: tests/qtest/hyperv-test.c
 
 PC Chipset
 M: Michael S. Tsirkin <mst@redhat.com>
diff --git a/tests/qtest/hyperv-test.c b/tests/qtest/hyperv-test.c
new file mode 100644
index 000000000000..0be689548e18
--- /dev/null
+++ b/tests/qtest/hyperv-test.c
@@ -0,0 +1,225 @@
+/*
+ * Hyper-V emulation CPU feature test cases
+ *
+ * Copyright (c) 2021 Red Hat Inc.
+ * Authors:
+ *  Vitaly Kuznetsov <vkuznets@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+#include <linux/kvm.h>
+#include <sys/ioctl.h>
+
+#include "qemu/osdep.h"
+#include "qemu/bitops.h"
+#include "libqos/libqtest.h"
+#include "qapi/qmp/qdict.h"
+#include "qapi/qmp/qjson.h"
+
+#define MACHINE_KVM "-machine pc-q35-5.2 -accel kvm "
+#define QUERY_HEAD  "{ 'execute': 'query-cpu-model-expansion', " \
+                    "  'arguments': { 'type': 'full', "
+#define QUERY_TAIL  "}}"
+
+static bool kvm_enabled(QTestState *qts)
+{
+    QDict *resp, *qdict;
+    bool enabled;
+
+    resp = qtest_qmp(qts, "{ 'execute': 'query-kvm' }");
+    g_assert(qdict_haskey(resp, "return"));
+    qdict = qdict_get_qdict(resp, "return");
+    g_assert(qdict_haskey(qdict, "enabled"));
+    enabled = qdict_get_bool(qdict, "enabled");
+    qobject_unref(resp);
+
+    return enabled;
+}
+
+static bool kvm_has_sys_hyperv_cpuid(void)
+{
+    int fd = open("/dev/kvm", O_RDWR);
+    int ret;
+
+    g_assert(fd > 0);
+
+    ret = ioctl(fd, KVM_CHECK_EXTENSION, KVM_CAP_SYS_HYPERV_CPUID);
+
+    close(fd);
+
+    return ret > 0;
+}
+
+static QDict *do_query_no_props(QTestState *qts, const char *cpu_type)
+{
+    return qtest_qmp(qts, QUERY_HEAD "'model': { 'name': %s }"
+                          QUERY_TAIL, cpu_type);
+}
+
+static bool resp_has_props(QDict *resp)
+{
+    QDict *qdict;
+
+    g_assert(resp);
+
+    if (!qdict_haskey(resp, "return")) {
+        return false;
+    }
+    qdict = qdict_get_qdict(resp, "return");
+
+    if (!qdict_haskey(qdict, "model")) {
+        return false;
+    }
+    qdict = qdict_get_qdict(qdict, "model");
+
+    return qdict_haskey(qdict, "props");
+}
+
+static QDict *resp_get_props(QDict *resp)
+{
+    QDict *qdict;
+
+    g_assert(resp);
+    g_assert(resp_has_props(resp));
+
+    qdict = qdict_get_qdict(resp, "return");
+    qdict = qdict_get_qdict(qdict, "model");
+    qdict = qdict_get_qdict(qdict, "props");
+
+    return qdict;
+}
+
+static bool resp_get_feature(QDict *resp, const char *feature)
+{
+    QDict *props;
+
+    g_assert(resp);
+    g_assert(resp_has_props(resp));
+    props = resp_get_props(resp);
+    g_assert(qdict_get(props, feature));
+    return qdict_get_bool(props, feature);
+}
+
+#define assert_has_feature(qts, cpu_type, feature)                     \
+({                                                                     \
+    QDict *_resp = do_query_no_props(qts, cpu_type);                   \
+    g_assert(_resp);                                                   \
+    g_assert(resp_has_props(_resp));                                   \
+    g_assert(qdict_get(resp_get_props(_resp), feature));               \
+    qobject_unref(_resp);                                              \
+})
+
+#define resp_assert_feature(resp, feature, expected_value)             \
+({                                                                     \
+    QDict *_props;                                                     \
+                                                                       \
+    g_assert(_resp);                                                   \
+    g_assert(resp_has_props(_resp));                                   \
+    _props = resp_get_props(_resp);                                    \
+    g_assert(qdict_get(_props, feature));                              \
+    g_assert(qdict_get_bool(_props, feature) == (expected_value));     \
+})
+
+#define assert_feature(qts, cpu_type, feature, expected_value)         \
+({                                                                     \
+    QDict *_resp;                                                      \
+                                                                       \
+    _resp = do_query_no_props(qts, cpu_type);                          \
+    g_assert(_resp);                                                   \
+    resp_assert_feature(_resp, feature, expected_value);               \
+    qobject_unref(_resp);                                              \
+})
+
+#define assert_has_feature_enabled(qts, cpu_type, feature)             \
+    assert_feature(qts, cpu_type, feature, true)
+
+#define assert_has_feature_disabled(qts, cpu_type, feature)            \
+    assert_feature(qts, cpu_type, feature, false)
+
+static void test_assert_hyperv_all_but_evmcs(QTestState *qts)
+{
+    assert_has_feature_enabled(qts, "host", "hv-relaxed");
+    assert_has_feature_enabled(qts, "host", "hv-vapic");
+    assert_has_feature_enabled(qts, "host", "hv-vpindex");
+    assert_has_feature_enabled(qts, "host", "hv-runtime");
+    assert_has_feature_enabled(qts, "host", "hv-crash");
+    assert_has_feature_enabled(qts, "host", "hv-time");
+    assert_has_feature_enabled(qts, "host", "hv-synic");
+    assert_has_feature_enabled(qts, "host", "hv-stimer");
+    assert_has_feature_enabled(qts, "host", "hv-tlbflush");
+    assert_has_feature_enabled(qts, "host", "hv-ipi");
+    assert_has_feature_enabled(qts, "host", "hv-reset");
+    assert_has_feature_enabled(qts, "host", "hv-frequencies");
+    assert_has_feature_enabled(qts, "host", "hv-reenlightenment");
+    assert_has_feature_enabled(qts, "host", "hv-stimer-direct");
+}
+
+static void test_query_cpu_hv_all_but_evmcs(const void *data)
+{
+    QTestState *qts;
+
+    qts = qtest_init(MACHINE_KVM "-cpu host,hv-relaxed,hv-vapic,hv-vpindex,"
+                     "hv-runtime,hv-crash,hv-time,hv-synic,hv-stimer,"
+                     "hv-tlbflush,hv-ipi,hv-reset,hv-frequencies,"
+                     "hv-reenlightenment,hv-stimer-direct");
+
+    test_assert_hyperv_all_but_evmcs(qts);
+
+    qtest_quit(qts);
+}
+
+static void test_query_cpu_hv_custom(const void *data)
+{
+    QTestState *qts;
+
+    qts = qtest_init(MACHINE_KVM "-cpu host,hv-vpindex");
+
+    assert_has_feature_enabled(qts, "host", "hv-vpindex");
+    assert_has_feature_disabled(qts, "host", "hv-synic");
+
+    qtest_quit(qts);
+}
+
+static void test_query_cpu_hv_passthrough(const void *data)
+{
+    QTestState *qts;
+    QDict *resp;
+
+    qts = qtest_init(MACHINE_KVM "-cpu host,hv-passthrough");
+    if (!kvm_enabled(qts)) {
+        qtest_quit(qts);
+        return;
+    }
+
+    test_assert_hyperv_all_but_evmcs(qts);
+
+    resp = do_query_no_props(qts, "host");
+    if (resp_get_feature(resp, "vmx")) {
+        assert_has_feature_enabled(qts, "host", "hv-evmcs");
+    } else {
+        assert_has_feature_disabled(qts, "host", "hv-evmcs");
+    }
+
+    qtest_quit(qts);
+}
+
+int main(int argc, char **argv)
+{
+    const char *arch = qtest_get_arch();
+
+    g_test_init(&argc, &argv, NULL);
+
+    if (!strcmp(arch, "i386") || !strcmp(arch, "x86_64")) {
+        qtest_add_data_func("/hyperv/hv-all-but-evmcs",
+                            NULL, test_query_cpu_hv_all_but_evmcs);
+        qtest_add_data_func("/hyperv/hv-custom",
+                            NULL, test_query_cpu_hv_custom);
+        if (kvm_has_sys_hyperv_cpuid()) {
+            qtest_add_data_func("/hyperv/hv-passthrough",
+                                NULL, test_query_cpu_hv_passthrough);
+       }
+    }
+
+    return g_test_run();
+}
diff --git a/tests/qtest/meson.build b/tests/qtest/meson.build
index 0c7673892179..03da00f82ba9 100644
--- a/tests/qtest/meson.build
+++ b/tests/qtest/meson.build
@@ -83,7 +83,8 @@ qtests_i386 = \
    'vmgenid-test',
    'migration-test',
    'test-x86-cpuid-compat',
-   'numa-test']
+   'numa-test',
+   'hyperv-test']
 
 dbus_daemon = find_program('dbus-daemon', required: false)
 if dbus_daemon.found() and config_host.has_key('GDBUS_CODEGEN')
-- 
2.30.2

Re: [PATCH v6 01/19] i386: keep hyperv_vendor string up-to-date

[Eduardo Habkost]

On Thu, Apr 22, 2021 at 06:11:12PM +0200, Vitaly Kuznetsov wrote:
> When cpu->hyperv_vendor is not set manually we default to "Microsoft Hv"
> and in 'hv_passthrough' mode we get the information from the host. This
> information is stored in cpu->hyperv_vendor_id[] array but we don't update
> cpu->hyperv_vendor string so e.g. QMP's query-cpu-model-expansion output
> is incorrect.

I was confused for a while because this can't happen until patch
15/19 is applied.  Probably worth a note in the commit message
indicating that hyperv_handle_properties() will be called by
x86_cpu_expand_features() in the future.

> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
>  target/i386/cpu.c     | 19 +++++++++----------
>  target/i386/kvm/kvm.c |  5 +++++
>  2 files changed, 14 insertions(+), 10 deletions(-)
> 
> diff --git a/target/i386/cpu.c b/target/i386/cpu.c
> index ad99cad0e7ce..2d05df232329 100644
> --- a/target/i386/cpu.c
> +++ b/target/i386/cpu.c
> @@ -6665,17 +6665,16 @@ static void x86_cpu_hyperv_realize(X86CPU *cpu)
>  
>      /* Hyper-V vendor id */
>      if (!cpu->hyperv_vendor) {
> -        memcpy(cpu->hyperv_vendor_id, "Microsoft Hv", 12);
> -    } else {
> -        len = strlen(cpu->hyperv_vendor);
> -
> -        if (len > 12) {
> -            warn_report("hv-vendor-id truncated to 12 characters");
> -            len = 12;
> -        }
> -        memset(cpu->hyperv_vendor_id, 0, 12);
> -        memcpy(cpu->hyperv_vendor_id, cpu->hyperv_vendor, len);
> +        object_property_set_str(OBJECT(cpu), "hv-vendor-id", "Microsoft Hv",
> +                                &error_abort);
> +    }
> +    len = strlen(cpu->hyperv_vendor);
> +    if (len > 12) {
> +        warn_report("hv-vendor-id truncated to 12 characters");
> +        len = 12;
>      }
> +    memset(cpu->hyperv_vendor_id, 0, 12);
> +    memcpy(cpu->hyperv_vendor_id, cpu->hyperv_vendor, len);

Existing issue: hardcoded 12 as the size of hyperv_vendor_id here
(compare with the code you add below using sizeof()).  I don't
think this should hold the whole series, so it can be fixed in a
follow up patch if necessary.

>  
>      /* 'Hv#1' interface identification*/
>      cpu->hyperv_interface_id[0] = 0x31237648;
> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
> index 7fe9f527103c..ab073a5e9c44 100644
> --- a/target/i386/kvm/kvm.c
> +++ b/target/i386/kvm/kvm.c
> @@ -1215,6 +1215,11 @@ static int hyperv_handle_properties(CPUState *cs,
>              cpu->hyperv_vendor_id[0] = c->ebx;
>              cpu->hyperv_vendor_id[1] = c->ecx;
>              cpu->hyperv_vendor_id[2] = c->edx;
> +            cpu->hyperv_vendor = g_realloc(cpu->hyperv_vendor,
> +                                           sizeof(cpu->hyperv_vendor_id) + 1);
> +            memcpy(cpu->hyperv_vendor, cpu->hyperv_vendor_id,
> +                   sizeof(cpu->hyperv_vendor_id));
> +            cpu->hyperv_vendor[sizeof(cpu->hyperv_vendor_id)] = 0;

I don't like having to do manual g_realloc() + memcpy() here
(calling object_property_set_str() would be simpler), but I
believe it will be easier to clean this up after this whole
series is applied.

Reluctantly:

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

>          }
>  
>          c = cpuid_find_entry(cpuid, HV_CPUID_INTERFACE, 0);
> -- 
> 2.30.2
> 

-- 
Eduardo

Re: [PATCH v6 02/19] i386: invert hyperv_spinlock_attempts setting logic with hv_passthrough

[Eduardo Habkost]

On Thu, Apr 22, 2021 at 06:11:13PM +0200, Vitaly Kuznetsov wrote:
> There is no need to have this special case: like all other Hyper-V
> enlightenments we can just use kernel's supplied value in hv_passthrough
> mode.

Worth documenting this behaviour at docs/hyperv.txt?  I was
expecting all properties set explicitly to override
hv-passthrough.

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
>  target/i386/kvm/kvm.c | 6 +-----
>  1 file changed, 1 insertion(+), 5 deletions(-)
> 
> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
> index ab073a5e9c44..afd173514da1 100644
> --- a/target/i386/kvm/kvm.c
> +++ b/target/i386/kvm/kvm.c
> @@ -1256,11 +1256,7 @@ static int hyperv_handle_properties(CPUState *cs,
>          c = cpuid_find_entry(cpuid, HV_CPUID_ENLIGHTMENT_INFO, 0);
>          if (c) {
>              env->features[FEAT_HV_RECOMM_EAX] = c->eax;
> -
> -            /* hv-spinlocks may have been overriden */
> -            if (cpu->hyperv_spinlock_attempts != HYPERV_SPINLOCK_NEVER_NOTIFY) {
> -                c->ebx = cpu->hyperv_spinlock_attempts;
> -            }
> +            cpu->hyperv_spinlock_attempts = c->ebx;
>          }
>          c = cpuid_find_entry(cpuid, HV_CPUID_NESTED_FEATURES, 0);
>          if (c) {
> -- 
> 2.30.2
> 

-- 
Eduardo

Re: [PATCH v6 03/19] i386: always fill Hyper-V CPUID feature leaves from X86CPU data

[Eduardo Habkost]

On Thu, Apr 22, 2021 at 06:11:14PM +0200, Vitaly Kuznetsov wrote:
> We have all the required data in X86CPU already and as we are about to
> split hyperv_handle_properties() into hyperv_expand_features()/
> hyperv_fill_cpuids() we can remove the blind copy. The functional change
> is that QEMU won't pass CPUID leaves it doesn't currently know about
> to the guest but arguably this is a good change.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>

This makes hv-passthrough less useful for debugging and
development, but safer for using in production.  I assume we want
to eventually make hv-passthrough supported in production when
live migration support isn't required.

I'll trust your judgement here and assume this is really a good
change, but maybe this should be documented more explicitly in
the hv-passthrough section at docs/hyperv.txt?

Anyway:

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

> ---
>  target/i386/kvm/kvm.c | 9 ---------
>  1 file changed, 9 deletions(-)
> 
> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
> index afd173514da1..7c751185491f 100644
> --- a/target/i386/kvm/kvm.c
> +++ b/target/i386/kvm/kvm.c
> @@ -1207,9 +1207,6 @@ static int hyperv_handle_properties(CPUState *cs,
>      }
>  
>      if (cpu->hyperv_passthrough) {
> -        memcpy(cpuid_ent, &cpuid->entries[0],
> -               cpuid->nent * sizeof(cpuid->entries[0]));
> -
>          c = cpuid_find_entry(cpuid, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, 0);
>          if (c) {
>              cpu->hyperv_vendor_id[0] = c->ebx;
> @@ -1309,12 +1306,6 @@ static int hyperv_handle_properties(CPUState *cs,
>          goto free;
>      }
>  
> -    if (cpu->hyperv_passthrough) {
> -        /* We already copied all feature words from KVM as is */
> -        r = cpuid->nent;
> -        goto free;
> -    }
> -
>      c = &cpuid_ent[cpuid_i++];
>      c->function = HV_CPUID_VENDOR_AND_MAX_FUNCTIONS;
>      c->eax = hyperv_feat_enabled(cpu, HYPERV_FEAT_EVMCS) ?
> -- 
> 2.30.2
> 

-- 
Eduardo

Re: [PATCH v6 04/19] i386: stop using env->features[] for filling Hyper-V CPUIDs

[Eduardo Habkost]

On Thu, Apr 22, 2021 at 06:11:15PM +0200, Vitaly Kuznetsov wrote:
> As a preparatory patch to dropping Hyper-V CPUID leaves from
> feature_word_info[] stop using env->features[] as a temporary
> storage of Hyper-V CPUIDs, just build Hyper-V CPUID leaves directly
> from kvm_hyperv_properties[] data.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
>  target/i386/cpu.h     |  1 +
>  target/i386/kvm/kvm.c | 80 +++++++++++++++++++++++--------------------
>  2 files changed, 43 insertions(+), 38 deletions(-)
> 
> diff --git a/target/i386/cpu.h b/target/i386/cpu.h
> index 570f916878f9..c8295aa2a1e7 100644
> --- a/target/i386/cpu.h
> +++ b/target/i386/cpu.h
> @@ -1684,6 +1684,7 @@ struct X86CPU {
>      uint32_t hyperv_interface_id[4];
>      uint32_t hyperv_version_id[4];
>      uint32_t hyperv_limits[3];
> +    uint32_t hyperv_nested[4];
>  
>      bool check_cpuid;
>      bool enforce_cpuid;
> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
> index 7c751185491f..f791baa29acf 100644
> --- a/target/i386/kvm/kvm.c
> +++ b/target/i386/kvm/kvm.c
> @@ -1111,7 +1111,6 @@ static int hv_cpuid_check_and_set(CPUState *cs, struct kvm_cpuid2 *cpuid,
>                                    int feature)
>  {
>      X86CPU *cpu = X86_CPU(cs);
> -    CPUX86State *env = &cpu->env;
>      uint32_t r, fw, bits;
>      uint64_t deps;
>      int i, dep_feat;
> @@ -1151,8 +1150,6 @@ static int hv_cpuid_check_and_set(CPUState *cs, struct kvm_cpuid2 *cpuid,
>                  return 0;
>              }
>          }
> -
> -        env->features[fw] |= bits;
>      }
>  
>      if (cpu->hyperv_passthrough) {
> @@ -1162,6 +1159,29 @@ static int hv_cpuid_check_and_set(CPUState *cs, struct kvm_cpuid2 *cpuid,
>      return 0;
>  }
>  
> +static uint32_t hv_build_cpuid_leaf(CPUState *cs, uint32_t fw)
> +{
> +    X86CPU *cpu = X86_CPU(cs);
> +    uint32_t r = 0;
> +    int i, j;
> +
> +    for (i = 0; i < ARRAY_SIZE(kvm_hyperv_properties); i++) {
> +        if (!hyperv_feat_enabled(cpu, i)) {
> +            continue;
> +        }
> +
> +        for (j = 0; j < ARRAY_SIZE(kvm_hyperv_properties[i].flags); j++) {
> +            if (kvm_hyperv_properties[i].flags[j].fw != fw) {
> +                continue;
> +            }
> +
> +            r |= kvm_hyperv_properties[i].flags[j].bits;
> +        }
> +    }
> +
> +    return r;
> +}
> +
>  /*
>   * Fill in Hyper-V CPUIDs. Returns the number of entries filled in cpuid_ent in
>   * case of success, errno < 0 in case of failure and 0 when no Hyper-V
> @@ -1171,9 +1191,8 @@ static int hyperv_handle_properties(CPUState *cs,
>                                      struct kvm_cpuid_entry2 *cpuid_ent)
>  {
>      X86CPU *cpu = X86_CPU(cs);
> -    CPUX86State *env = &cpu->env;
>      struct kvm_cpuid2 *cpuid;
> -    struct kvm_cpuid_entry2 *c;
> +    struct kvm_cpuid_entry2 *c, *c2;
>      uint32_t cpuid_i = 0;
>      int r;
>  
> @@ -1194,9 +1213,7 @@ static int hyperv_handle_properties(CPUState *cs,
>          }
>  
>          if (!r) {

I think I now I understand why removing FEAT_HYPERV_* makes sense:

The rules mapping hyperv features to CPUID bits were encoded
twice in the code: in both hyperv_handle_properties() and
kvm_hyperv_properties[].  More work to maintain the rules, and
too easy to accidentally make them inconsistent.


Now, let me see if I can prove to myself that the new code works:

> -            env->features[FEAT_HV_RECOMM_EAX] |=
> -                HV_ENLIGHTENED_VMCS_RECOMMENDED;

[Line1]

The only code reading env->features[FEAT_HV_RECOMM_EAX] seems to
be [Line2] below:

  eax = env->features[FEAT_HV_RECOMM_EAX];

which is replaced with [Line3]:

  c->eax = hv_build_cpuid_leaf(cs, FEAT_HV_RECOMM_EAX);

so if hv_build_cpuid_leaf() do its job and set return
HV_ENLIGHTENED_VMCS_RECOMMENDED set at [Line2], we can safely
delete [Line1].

Will hv_build_cpuid_leaf() set HV_ENLIGHTENED_VMCS_RECOMMENDED in
[Line2] for all cases where [Line1] was being executed?

Let's check what's necessary to make hv_build_cpuid_leaf()
set HV_ENLIGHTENED_VMCS_RECOMMENDED:

There's only one entry with
FEAT_HV_RECOMM_EAX/HV_ENLIGHTENED_VMCS_RECOMMENDED at
kvm_hyperv_properties:

    [HYPERV_FEAT_EVMCS] = {
        .desc = "enlightened VMCS (hv-evmcs)",
        .flags = {
            {.fw = FEAT_HV_RECOMM_EAX,
             .bits = HV_ENLIGHTENED_VMCS_RECOMMENDED}
        },
        .dependencies = BIT(HYPERV_FEAT_VAPIC)
    },

The logic at hv_build_cpuid_leaf() will make
HV_ENLIGHTENED_VMCS_RECOMMENDED be set only if
hyperv_feat_enabled(HYPERV_FEAT_EVMCS) is true.  Which is what I
expected, because the line of code you are removing is inside a
hyperv_feat_enabled(cpu, HYPERV_FEAT_EVMCS) conditional.

For reference, hyperv_feat_enabled(cpu, feat) returns:
  !!(cpu->hyperv_features & BIT(feat))

I don't see any code _clearing_ hyperv_features, except for
properties that could change hyperv_features.  I don't expect the
"hv-evmcs" QOM property to be touched by this function, so we
should be safe: if hyperv_feat_enabled(cpu, HYPERV_FEAT_EVMCS)
returned before executing [Line1], it will return true when
executing [Line2].

This means hv_build_cpuid_leaf() will set
HV_ENLIGHTENED_VMCS_RECOMMENDED if
hyperv_feat_enabled(HYPERV_FEAT_EVMCS) is true, and
hyperv_feat_enabled(HYPERV_FEAT_EVMCS) will be true at [Line2] on
all cases when [Line1] was being executed.

We also need to be sure the HV_ENLIGHTENED_VMCS_RECOMMENDED will
be _unset_ at hv_build_cpuid_leaf() when expected, but the rules
are trickier (due to hyperv_passthrough). I'll try to prove that
later.


> -            env->features[FEAT_HV_NESTED_EAX] = evmcs_version;

[Line4]


Can we delete [Line4]?

The only code reading env->features[FEAT_HV_NESTED_EAX]
is [Line5]:
    c->eax = env->features[FEAT_HV_NESTED_EAX];
which is replaced with [Line6]:
    c->eax = cpu->hyperv_nested[0];

We are also replacing env->features[FEAT_HV_NESTED_EAX] with
cpu->hyperv_nested[0], here:

> +            cpu->hyperv_nested[0] = evmcs_version;

This will make [Line6] set c->eax to evmcs_version, unless other
code writes to cpu->hyperv_nested[0].

I don't see any other code writing to hyperv_nested in this
patch, so we are good.

>          }
>      }
>  
> @@ -1235,13 +1252,6 @@ static int hyperv_handle_properties(CPUState *cs,
>              cpu->hyperv_version_id[3] = c->edx;
>          }
>  
> -        c = cpuid_find_entry(cpuid, HV_CPUID_FEATURES, 0);
> -        if (c) {
> -            env->features[FEAT_HYPERV_EAX] = c->eax;

[Line7A]

> -            env->features[FEAT_HYPERV_EBX] = c->ebx;

[Line7B]

> -            env->features[FEAT_HYPERV_EDX] = c->edx;

[Line7D]


Can we delete [Line7*]?

The only code reading env->features[FEAT_HYPERV_*] seems to be
[Line8*]:
    c->eax = env->features[FEAT_HYPERV_EAX];
    c->ebx = env->features[FEAT_HYPERV_EBX];
    c->edx = env->features[FEAT_HYPERV_EDX];
which is replaced by [Line9*]:
    c->eax = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EAX);
    c->ebx = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EBX);
    c->edx = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EDX);

So we need to make sure hv_build_cpuid_leaf() will return the
right values at [Line9*].

This one will be trickier to evaluate: there are lots of entries in
kvm_hyperv_properties[] that affect FEAT_HYPERV_EAX.

I will pause here and continue next week.   :)

This smells like it could have been split into smaller patches,
but I'm not sure if it would be possible.  Maybe the existing
code was too tangled for splitting this refactor into smaller
patches.


> -        }
> -
>          c = cpuid_find_entry(cpuid, HV_CPUID_IMPLEMENT_LIMITS, 0);
>          if (c) {
>              cpu->hv_max_vps = c->eax;
> @@ -1252,23 +1262,8 @@ static int hyperv_handle_properties(CPUState *cs,
>  
>          c = cpuid_find_entry(cpuid, HV_CPUID_ENLIGHTMENT_INFO, 0);
>          if (c) {
> -            env->features[FEAT_HV_RECOMM_EAX] = c->eax;
>              cpu->hyperv_spinlock_attempts = c->ebx;
>          }
> -        c = cpuid_find_entry(cpuid, HV_CPUID_NESTED_FEATURES, 0);
> -        if (c) {
> -            env->features[FEAT_HV_NESTED_EAX] = c->eax;
> -        }
> -    }
> -
> -    if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_ON) {
> -        env->features[FEAT_HV_RECOMM_EAX] |= HV_NO_NONARCH_CORESHARING;
> -    } else if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_AUTO) {
> -        c = cpuid_find_entry(cpuid, HV_CPUID_ENLIGHTMENT_INFO, 0);
> -        if (c) {
> -            env->features[FEAT_HV_RECOMM_EAX] |=
> -                c->eax & HV_NO_NONARCH_CORESHARING;
> -        }
>      }
>  
>      /* Features */
> @@ -1298,9 +1293,6 @@ static int hyperv_handle_properties(CPUState *cs,
>          r |= 1;
>      }
>  
> -    /* Not exposed by KVM but needed to make CPU hotplug in Windows work */
> -    env->features[FEAT_HYPERV_EDX] |= HV_CPU_DYNAMIC_PARTITIONING_AVAILABLE;
> -
>      if (r) {
>          r = -ENOSYS;
>          goto free;
> @@ -1330,15 +1322,27 @@ static int hyperv_handle_properties(CPUState *cs,
>  
>      c = &cpuid_ent[cpuid_i++];
>      c->function = HV_CPUID_FEATURES;
> -    c->eax = env->features[FEAT_HYPERV_EAX];

[Line8A]

> -    c->ebx = env->features[FEAT_HYPERV_EBX];

[Line8B]

> -    c->edx = env->features[FEAT_HYPERV_EDX];

[Line8D]

> +    c->eax = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EAX);

[Line9A]

> +    c->ebx = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EBX);

[Line9B]

> +    c->edx = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EDX);

[Line9D]

> +
> +    /* Not exposed by KVM but needed to make CPU hotplug in Windows work */
> +    c->edx |= HV_CPU_DYNAMIC_PARTITIONING_AVAILABLE;
>  
>      c = &cpuid_ent[cpuid_i++];
>      c->function = HV_CPUID_ENLIGHTMENT_INFO;
> -    c->eax = env->features[FEAT_HV_RECOMM_EAX];

[Line2]

> +    c->eax = hv_build_cpuid_leaf(cs, FEAT_HV_RECOMM_EAX);

[Line3]

>      c->ebx = cpu->hyperv_spinlock_attempts;
>  
> +    if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_ON) {
> +        c->eax |= HV_NO_NONARCH_CORESHARING;
> +    } else if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_AUTO) {
> +        c2 = cpuid_find_entry(cpuid, HV_CPUID_ENLIGHTMENT_INFO, 0);
> +        if (c2) {
> +            c->eax |= c2->eax & HV_NO_NONARCH_CORESHARING;
> +        }
> +    }
> +
>      c = &cpuid_ent[cpuid_i++];
>      c->function = HV_CPUID_IMPLEMENT_LIMITS;
>      c->eax = cpu->hv_max_vps;
> @@ -1358,7 +1362,7 @@ static int hyperv_handle_properties(CPUState *cs,
>  
>          c = &cpuid_ent[cpuid_i++];
>          c->function = HV_CPUID_NESTED_FEATURES;
> -        c->eax = env->features[FEAT_HV_NESTED_EAX];
> +        c->eax = cpu->hyperv_nested[0];
>      }
>      r = cpuid_i;
>  
> -- 
> 2.30.2
> 

-- 
Eduardo

Re: [PATCH v6 04/19] i386: stop using env->features[] for filling Hyper-V CPUIDs

[Eduardo Habkost]

On Fri, Apr 30, 2021 at 08:34:40PM -0400, Eduardo Habkost wrote:
> On Thu, Apr 22, 2021 at 06:11:15PM +0200, Vitaly Kuznetsov wrote:
> > As a preparatory patch to dropping Hyper-V CPUID leaves from
> > feature_word_info[] stop using env->features[] as a temporary
> > storage of Hyper-V CPUIDs, just build Hyper-V CPUID leaves directly
> > from kvm_hyperv_properties[] data.
> > 
> > Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> > ---
> >  target/i386/cpu.h     |  1 +
> >  target/i386/kvm/kvm.c | 80 +++++++++++++++++++++++--------------------
> >  2 files changed, 43 insertions(+), 38 deletions(-)
> > 
> > diff --git a/target/i386/cpu.h b/target/i386/cpu.h
> > index 570f916878f9..c8295aa2a1e7 100644
> > --- a/target/i386/cpu.h
> > +++ b/target/i386/cpu.h
> > @@ -1684,6 +1684,7 @@ struct X86CPU {
> >      uint32_t hyperv_interface_id[4];
> >      uint32_t hyperv_version_id[4];
> >      uint32_t hyperv_limits[3];
> > +    uint32_t hyperv_nested[4];
> >  
> >      bool check_cpuid;
> >      bool enforce_cpuid;
> > diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
> > index 7c751185491f..f791baa29acf 100644
> > --- a/target/i386/kvm/kvm.c
> > +++ b/target/i386/kvm/kvm.c
> > @@ -1111,7 +1111,6 @@ static int hv_cpuid_check_and_set(CPUState *cs, struct kvm_cpuid2 *cpuid,
> >                                    int feature)
> >  {
> >      X86CPU *cpu = X86_CPU(cs);
> > -    CPUX86State *env = &cpu->env;
> >      uint32_t r, fw, bits;
> >      uint64_t deps;
> >      int i, dep_feat;
> > @@ -1151,8 +1150,6 @@ static int hv_cpuid_check_and_set(CPUState *cs, struct kvm_cpuid2 *cpuid,
> >                  return 0;
> >              }
> >          }
> > -
> > -        env->features[fw] |= bits;
> >      }
> >  
> >      if (cpu->hyperv_passthrough) {
> > @@ -1162,6 +1159,29 @@ static int hv_cpuid_check_and_set(CPUState *cs, struct kvm_cpuid2 *cpuid,
> >      return 0;
> >  }
> >  
> > +static uint32_t hv_build_cpuid_leaf(CPUState *cs, uint32_t fw)
> > +{
> > +    X86CPU *cpu = X86_CPU(cs);
> > +    uint32_t r = 0;
> > +    int i, j;
> > +
> > +    for (i = 0; i < ARRAY_SIZE(kvm_hyperv_properties); i++) {
> > +        if (!hyperv_feat_enabled(cpu, i)) {
> > +            continue;
> > +        }
> > +
> > +        for (j = 0; j < ARRAY_SIZE(kvm_hyperv_properties[i].flags); j++) {
> > +            if (kvm_hyperv_properties[i].flags[j].fw != fw) {
> > +                continue;
> > +            }
> > +
> > +            r |= kvm_hyperv_properties[i].flags[j].bits;
> > +        }
> > +    }
> > +
> > +    return r;
> > +}
> > +
> >  /*
> >   * Fill in Hyper-V CPUIDs. Returns the number of entries filled in cpuid_ent in
> >   * case of success, errno < 0 in case of failure and 0 when no Hyper-V
> > @@ -1171,9 +1191,8 @@ static int hyperv_handle_properties(CPUState *cs,
> >                                      struct kvm_cpuid_entry2 *cpuid_ent)
> >  {
> >      X86CPU *cpu = X86_CPU(cs);
> > -    CPUX86State *env = &cpu->env;
> >      struct kvm_cpuid2 *cpuid;
> > -    struct kvm_cpuid_entry2 *c;
> > +    struct kvm_cpuid_entry2 *c, *c2;
> >      uint32_t cpuid_i = 0;
> >      int r;
> >  
> > @@ -1194,9 +1213,7 @@ static int hyperv_handle_properties(CPUState *cs,
> >          }
> >  
> >          if (!r) {
> 
> I think I now I understand why removing FEAT_HYPERV_* makes sense:
> 
> The rules mapping hyperv features to CPUID bits were encoded
> twice in the code: in both hyperv_handle_properties() and
> kvm_hyperv_properties[].  More work to maintain the rules, and
> too easy to accidentally make them inconsistent.
> 
> 
> Now, let me see if I can prove to myself that the new code works:
> 
> > -            env->features[FEAT_HV_RECOMM_EAX] |=
> > -                HV_ENLIGHTENED_VMCS_RECOMMENDED;
> 
> [Line1]
> 
> The only code reading env->features[FEAT_HV_RECOMM_EAX] seems to
> be [Line2] below:
> 
>   eax = env->features[FEAT_HV_RECOMM_EAX];
> 
> which is replaced with [Line3]:
> 
>   c->eax = hv_build_cpuid_leaf(cs, FEAT_HV_RECOMM_EAX);
> 
> so if hv_build_cpuid_leaf() do its job and set return
> HV_ENLIGHTENED_VMCS_RECOMMENDED set at [Line2], we can safely
> delete [Line1].
> 
> Will hv_build_cpuid_leaf() set HV_ENLIGHTENED_VMCS_RECOMMENDED in
> [Line2] for all cases where [Line1] was being executed?
> 
> Let's check what's necessary to make hv_build_cpuid_leaf()
> set HV_ENLIGHTENED_VMCS_RECOMMENDED:
> 
> There's only one entry with
> FEAT_HV_RECOMM_EAX/HV_ENLIGHTENED_VMCS_RECOMMENDED at
> kvm_hyperv_properties:
> 
>     [HYPERV_FEAT_EVMCS] = {
>         .desc = "enlightened VMCS (hv-evmcs)",
>         .flags = {
>             {.fw = FEAT_HV_RECOMM_EAX,
>              .bits = HV_ENLIGHTENED_VMCS_RECOMMENDED}
>         },
>         .dependencies = BIT(HYPERV_FEAT_VAPIC)
>     },
> 
> The logic at hv_build_cpuid_leaf() will make
> HV_ENLIGHTENED_VMCS_RECOMMENDED be set only if
> hyperv_feat_enabled(HYPERV_FEAT_EVMCS) is true.  Which is what I
> expected, because the line of code you are removing is inside a
> hyperv_feat_enabled(cpu, HYPERV_FEAT_EVMCS) conditional.
> 
> For reference, hyperv_feat_enabled(cpu, feat) returns:
>   !!(cpu->hyperv_features & BIT(feat))
> 
> I don't see any code _clearing_ hyperv_features, except for
> properties that could change hyperv_features.  I don't expect the
> "hv-evmcs" QOM property to be touched by this function, so we
> should be safe: if hyperv_feat_enabled(cpu, HYPERV_FEAT_EVMCS)
> returned before executing [Line1], it will return true when
> executing [Line2].
> 
> This means hv_build_cpuid_leaf() will set
> HV_ENLIGHTENED_VMCS_RECOMMENDED if
> hyperv_feat_enabled(HYPERV_FEAT_EVMCS) is true, and
> hyperv_feat_enabled(HYPERV_FEAT_EVMCS) will be true at [Line2] on
> all cases when [Line1] was being executed.
> 
> We also need to be sure the HV_ENLIGHTENED_VMCS_RECOMMENDED will
> be _unset_ at hv_build_cpuid_leaf() when expected, but the rules
> are trickier (due to hyperv_passthrough). I'll try to prove that
> later.
> 
> 
> > -            env->features[FEAT_HV_NESTED_EAX] = evmcs_version;
> 
> [Line4]
> 
> 
> Can we delete [Line4]?
> 
> The only code reading env->features[FEAT_HV_NESTED_EAX]
> is [Line5]:
>     c->eax = env->features[FEAT_HV_NESTED_EAX];
> which is replaced with [Line6]:
>     c->eax = cpu->hyperv_nested[0];
> 
> We are also replacing env->features[FEAT_HV_NESTED_EAX] with
> cpu->hyperv_nested[0], here:
> 
> > +            cpu->hyperv_nested[0] = evmcs_version;
> 
> This will make [Line6] set c->eax to evmcs_version, unless other
> code writes to cpu->hyperv_nested[0].
> 
> I don't see any other code writing to hyperv_nested in this
> patch, so we are good.
> 
> >          }
> >      }
> >  
> > @@ -1235,13 +1252,6 @@ static int hyperv_handle_properties(CPUState *cs,
> >              cpu->hyperv_version_id[3] = c->edx;
> >          }
> >  
> > -        c = cpuid_find_entry(cpuid, HV_CPUID_FEATURES, 0);
> > -        if (c) {
> > -            env->features[FEAT_HYPERV_EAX] = c->eax;
> 
> [Line7A]
> 
> > -            env->features[FEAT_HYPERV_EBX] = c->ebx;
> 
> [Line7B]
> 
> > -            env->features[FEAT_HYPERV_EDX] = c->edx;
> 
> [Line7D]
> 
> 
> Can we delete [Line7*]?
> 
> The only code reading env->features[FEAT_HYPERV_*] seems to be
> [Line8*]:
>     c->eax = env->features[FEAT_HYPERV_EAX];
>     c->ebx = env->features[FEAT_HYPERV_EBX];
>     c->edx = env->features[FEAT_HYPERV_EDX];
> which is replaced by [Line9*]:
>     c->eax = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EAX);
>     c->ebx = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EBX);
>     c->edx = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EDX);
> 
> So we need to make sure hv_build_cpuid_leaf() will return the
> right values at [Line9*].
> 
> This one will be trickier to evaluate: there are lots of entries in
> kvm_hyperv_properties[] that affect FEAT_HYPERV_EAX.
> 
> I will pause here and continue next week.   :)
> 

Continuing:

So, [Line7*] above was for hyperv_passthrough mode only.  This
means now hv-passthrough will enable only known features (the
ones at kvm_hyperv_properties).

The same comment I sent to the previous patch applies here:

"""
This makes hv-passthrough less useful for debugging and
development, but safer for using in production.  I assume we want
to eventually make hv-passthrough supported in production when
live migration support isn't required.

I'll trust your judgement here and assume this is really a good
change, but maybe this should be documented more explicitly in
the hv-passthrough section at docs/hyperv.txt?
"""



> This smells like it could have been split into smaller patches,
> but I'm not sure if it would be possible.  Maybe the existing
> code was too tangled for splitting this refactor into smaller
> patches.
> 
> 
> > -        }
> > -
> >          c = cpuid_find_entry(cpuid, HV_CPUID_IMPLEMENT_LIMITS, 0);
> >          if (c) {
> >              cpu->hv_max_vps = c->eax;
> > @@ -1252,23 +1262,8 @@ static int hyperv_handle_properties(CPUState *cs,
> >  
> >          c = cpuid_find_entry(cpuid, HV_CPUID_ENLIGHTMENT_INFO, 0);
> >          if (c) {
> > -            env->features[FEAT_HV_RECOMM_EAX] = c->eax;

Same as above: this is hv-passthrough code, and the comment above
applies.

> >              cpu->hyperv_spinlock_attempts = c->ebx;
> >          }
> > -        c = cpuid_find_entry(cpuid, HV_CPUID_NESTED_FEATURES, 0);
> > -        if (c) {
> > -            env->features[FEAT_HV_NESTED_EAX] = c->eax;
> > -        }

Same as above: this is hv-passthrough code, and the comment above
applies.

> > -    }
> > -

Now we're outside the hv-passthrough code:

> > -    if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_ON) {
> > -        env->features[FEAT_HV_RECOMM_EAX] |= HV_NO_NONARCH_CORESHARING;
> > -    } else if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_AUTO) {
> > -        c = cpuid_find_entry(cpuid, HV_CPUID_ENLIGHTMENT_INFO, 0);
> > -        if (c) {
> > -            env->features[FEAT_HV_RECOMM_EAX] |=
> > -                c->eax & HV_NO_NONARCH_CORESHARING;
> > -        }
> >      }

The hack above is copied to [Line10] below.  Looks OK.

> >  
> >      /* Features */
> > @@ -1298,9 +1293,6 @@ static int hyperv_handle_properties(CPUState *cs,
> >          r |= 1;
> >      }
> >  
> > -    /* Not exposed by KVM but needed to make CPU hotplug in Windows work */
> > -    env->features[FEAT_HYPERV_EDX] |= HV_CPU_DYNAMIC_PARTITIONING_AVAILABLE;

The hack above is copied to [Line10] below.  Looks OK.

> > -
> >      if (r) {
> >          r = -ENOSYS;
> >          goto free;
> > @@ -1330,15 +1322,27 @@ static int hyperv_handle_properties(CPUState *cs,
> >  
> >      c = &cpuid_ent[cpuid_i++];
> >      c->function = HV_CPUID_FEATURES;
> > -    c->eax = env->features[FEAT_HYPERV_EAX];
> 
> [Line8A]
> 
> > -    c->ebx = env->features[FEAT_HYPERV_EBX];
> 
> [Line8B]
> 
> > -    c->edx = env->features[FEAT_HYPERV_EDX];
> 
> [Line8D]
> 
> > +    c->eax = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EAX);
> 
> [Line9A]
> 
> > +    c->ebx = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EBX);
> 
> [Line9B]
> 
> > +    c->edx = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EDX);
> 
> [Line9D]

Already reviewed at [Line7*] above.

> 
> > +
> > +    /* Not exposed by KVM but needed to make CPU hotplug in Windows work */
> > +    c->edx |= HV_CPU_DYNAMIC_PARTITIONING_AVAILABLE;

[Line11]

Looks OK, but I wonder if this could be encoded in
kvm_hyperv_properties somehow.


> >  
> >      c = &cpuid_ent[cpuid_i++];
> >      c->function = HV_CPUID_ENLIGHTMENT_INFO;
> > -    c->eax = env->features[FEAT_HV_RECOMM_EAX];
> 
> [Line2]
> 
> > +    c->eax = hv_build_cpuid_leaf(cs, FEAT_HV_RECOMM_EAX);
> 
> [Line3]
> 
> >      c->ebx = cpu->hyperv_spinlock_attempts;
> >  
> > +    if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_ON) {
> > +        c->eax |= HV_NO_NONARCH_CORESHARING;
> > +    } else if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_AUTO) {
> > +        c2 = cpuid_find_entry(cpuid, HV_CPUID_ENLIGHTMENT_INFO, 0);
> > +        if (c2) {
> > +            c->eax |= c2->eax & HV_NO_NONARCH_CORESHARING;
> > +        }
> > +    }

[Line10]

Matches the code above.

> > +
> >      c = &cpuid_ent[cpuid_i++];
> >      c->function = HV_CPUID_IMPLEMENT_LIMITS;
> >      c->eax = cpu->hv_max_vps;
> > @@ -1358,7 +1362,7 @@ static int hyperv_handle_properties(CPUState *cs,
> >  
> >          c = &cpuid_ent[cpuid_i++];
> >          c->function = HV_CPUID_NESTED_FEATURES;
> > -        c->eax = env->features[FEAT_HV_NESTED_EAX];
> > +        c->eax = cpu->hyperv_nested[0];
> >      }
> >      r = cpuid_i;
> >  
> > -- 
> > 2.30.2
> > 
> 

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

Sorry for the long delay in reviewing this!

-- 
Eduardo

Re: [PATCH v6 05/19] i386: introduce hyperv_feature_supported()

[Eduardo Habkost]

On Thu, Apr 22, 2021 at 06:11:16PM +0200, Vitaly Kuznetsov wrote:
> Clean up hv_cpuid_check_and_set() by separating hyperv_feature_supported()
> off it. No functional change intended.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
>  target/i386/kvm/kvm.c | 49 ++++++++++++++++++++++++++-----------------
>  1 file changed, 30 insertions(+), 19 deletions(-)
> 
> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
> index f791baa29acf..ba093dba4d23 100644
> --- a/target/i386/kvm/kvm.c
> +++ b/target/i386/kvm/kvm.c
> @@ -1107,13 +1107,33 @@ static int hv_cpuid_get_fw(struct kvm_cpuid2 *cpuid, int fw, uint32_t *r)
>      return 0;
>  }
>  
> +static bool hyperv_feature_supported(struct kvm_cpuid2 *cpuid, int feature)
> +{
> +    uint32_t r, fw, bits;
> +    int i;
> +
> +    for (i = 0; i < ARRAY_SIZE(kvm_hyperv_properties[feature].flags); i++) {
> +        fw = kvm_hyperv_properties[feature].flags[i].fw;
> +        bits = kvm_hyperv_properties[feature].flags[i].bits;
> +
> +        if (!fw) {
> +            continue;
> +        }
> +
> +        if (hv_cpuid_get_fw(cpuid, fw, &r) || (r & bits) != bits) {
> +            return false;
> +        }
> +    }
> +
> +    return true;
> +}
> +
>  static int hv_cpuid_check_and_set(CPUState *cs, struct kvm_cpuid2 *cpuid,
>                                    int feature)
>  {
>      X86CPU *cpu = X86_CPU(cs);
> -    uint32_t r, fw, bits;
>      uint64_t deps;
> -    int i, dep_feat;
> +    int dep_feat;
>  
>      if (!hyperv_feat_enabled(cpu, feature) && !cpu->hyperv_passthrough) {
>          return 0;
> @@ -1132,23 +1152,14 @@ static int hv_cpuid_check_and_set(CPUState *cs, struct kvm_cpuid2 *cpuid,
>          deps &= ~(1ull << dep_feat);
>      }
>  
> -    for (i = 0; i < ARRAY_SIZE(kvm_hyperv_properties[feature].flags); i++) {
> -        fw = kvm_hyperv_properties[feature].flags[i].fw;
> -        bits = kvm_hyperv_properties[feature].flags[i].bits;
> -
> -        if (!fw) {
> -            continue;
> -        }
> -
> -        if (hv_cpuid_get_fw(cpuid, fw, &r) || (r & bits) != bits) {
> -            if (hyperv_feat_enabled(cpu, feature)) {
> -                fprintf(stderr,
> -                        "Hyper-V %s is not supported by kernel\n",
> -                        kvm_hyperv_properties[feature].desc);
> -                return 1;
> -            } else {
> -                return 0;
> -            }
> +    if (!hyperv_feature_supported(cpuid, feature)) {
> +        if (hyperv_feat_enabled(cpu, feature)) {
> +            fprintf(stderr,
> +                    "Hyper-V %s is not supported by kernel\n",
> +                    kvm_hyperv_properties[feature].desc);
> +            return 1;
> +        } else {
> +            return 0;

The reason for returning prematurely here when
!hyperv_feat_enabled() is not clear to me, but you are keeping
the existing behavior, so:

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

>          }
>      }
>  
> -- 
> 2.30.2
> 

-- 
Eduardo

Re: [PATCH v6 06/19] i386: introduce hv_cpuid_get_host()

[Eduardo Habkost]

On Thu, Apr 22, 2021 at 06:11:17PM +0200, Vitaly Kuznetsov wrote:
> As a preparation to implementing hv_cpuid_cache intro introduce
> hv_cpuid_get_host(). No functional change intended.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
>  target/i386/kvm/kvm.c | 102 +++++++++++++++++++++++-------------------
>  1 file changed, 57 insertions(+), 45 deletions(-)
> 
> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
> index ba093dba4d23..7aeb704b016e 100644
> --- a/target/i386/kvm/kvm.c
> +++ b/target/i386/kvm/kvm.c
> @@ -1107,6 +1107,19 @@ static int hv_cpuid_get_fw(struct kvm_cpuid2 *cpuid, int fw, uint32_t *r)
>      return 0;
>  }
>  
> +static uint32_t hv_cpuid_get_host(struct kvm_cpuid2 *cpuid, uint32_t func,
> +                                  int reg)
> +{
> +    struct kvm_cpuid_entry2 *entry;
> +
> +    entry = cpuid_find_entry(cpuid, func, 0);
> +    if (!entry) {
> +        return 0;

One possible difference here is that leaves not supported by the
host will now be zeroed out.

...which I expected to be the correct behavior of hv-passthrough,
so:

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>


> +    }
> +
> +    return cpuid_entry_get_reg(entry, reg);
> +}
> +
>  static bool hyperv_feature_supported(struct kvm_cpuid2 *cpuid, int feature)
>  {
>      uint32_t r, fw, bits;
> @@ -1203,7 +1216,7 @@ static int hyperv_handle_properties(CPUState *cs,
>  {
>      X86CPU *cpu = X86_CPU(cs);
>      struct kvm_cpuid2 *cpuid;
> -    struct kvm_cpuid_entry2 *c, *c2;
> +    struct kvm_cpuid_entry2 *c;
>      uint32_t cpuid_i = 0;
>      int r;
>  
> @@ -1235,46 +1248,47 @@ static int hyperv_handle_properties(CPUState *cs,
>      }
>  
>      if (cpu->hyperv_passthrough) {
> -        c = cpuid_find_entry(cpuid, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, 0);
> -        if (c) {
> -            cpu->hyperv_vendor_id[0] = c->ebx;
> -            cpu->hyperv_vendor_id[1] = c->ecx;
> -            cpu->hyperv_vendor_id[2] = c->edx;
> -            cpu->hyperv_vendor = g_realloc(cpu->hyperv_vendor,
> -                                           sizeof(cpu->hyperv_vendor_id) + 1);
> -            memcpy(cpu->hyperv_vendor, cpu->hyperv_vendor_id,
> -                   sizeof(cpu->hyperv_vendor_id));
> -            cpu->hyperv_vendor[sizeof(cpu->hyperv_vendor_id)] = 0;
> -        }
> -
> -        c = cpuid_find_entry(cpuid, HV_CPUID_INTERFACE, 0);
> -        if (c) {
> -            cpu->hyperv_interface_id[0] = c->eax;
> -            cpu->hyperv_interface_id[1] = c->ebx;
> -            cpu->hyperv_interface_id[2] = c->ecx;
> -            cpu->hyperv_interface_id[3] = c->edx;
> -        }
> -
> -        c = cpuid_find_entry(cpuid, HV_CPUID_VERSION, 0);
> -        if (c) {
> -            cpu->hyperv_version_id[0] = c->eax;
> -            cpu->hyperv_version_id[1] = c->ebx;
> -            cpu->hyperv_version_id[2] = c->ecx;
> -            cpu->hyperv_version_id[3] = c->edx;
> -        }
> -
> -        c = cpuid_find_entry(cpuid, HV_CPUID_IMPLEMENT_LIMITS, 0);
> -        if (c) {
> -            cpu->hv_max_vps = c->eax;
> -            cpu->hyperv_limits[0] = c->ebx;
> -            cpu->hyperv_limits[1] = c->ecx;
> -            cpu->hyperv_limits[2] = c->edx;
> -        }
> -
> -        c = cpuid_find_entry(cpuid, HV_CPUID_ENLIGHTMENT_INFO, 0);
> -        if (c) {
> -            cpu->hyperv_spinlock_attempts = c->ebx;
> -        }
> +        cpu->hyperv_vendor_id[0] =
> +            hv_cpuid_get_host(cpuid, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, R_EBX);
> +        cpu->hyperv_vendor_id[1] =
> +            hv_cpuid_get_host(cpuid, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, R_ECX);
> +        cpu->hyperv_vendor_id[2] =
> +            hv_cpuid_get_host(cpuid, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, R_EDX);
> +        cpu->hyperv_vendor = g_realloc(cpu->hyperv_vendor,
> +                                       sizeof(cpu->hyperv_vendor_id) + 1);
> +        memcpy(cpu->hyperv_vendor, cpu->hyperv_vendor_id,
> +               sizeof(cpu->hyperv_vendor_id));
> +        cpu->hyperv_vendor[sizeof(cpu->hyperv_vendor_id)] = 0;
> +
> +        cpu->hyperv_interface_id[0] =
> +            hv_cpuid_get_host(cpuid, HV_CPUID_INTERFACE, R_EAX);
> +        cpu->hyperv_interface_id[1] =
> +            hv_cpuid_get_host(cpuid, HV_CPUID_INTERFACE, R_EBX);
> +        cpu->hyperv_interface_id[2] =
> +            hv_cpuid_get_host(cpuid, HV_CPUID_INTERFACE, R_ECX);
> +        cpu->hyperv_interface_id[3] =
> +            hv_cpuid_get_host(cpuid, HV_CPUID_INTERFACE, R_EDX);
> +
> +        cpu->hyperv_version_id[0] =
> +            hv_cpuid_get_host(cpuid, HV_CPUID_VERSION, R_EAX);
> +        cpu->hyperv_version_id[1] =
> +            hv_cpuid_get_host(cpuid, HV_CPUID_VERSION, R_EBX);
> +        cpu->hyperv_version_id[2] =
> +            hv_cpuid_get_host(cpuid, HV_CPUID_VERSION, R_ECX);
> +        cpu->hyperv_version_id[3] =
> +            hv_cpuid_get_host(cpuid, HV_CPUID_VERSION, R_EDX);
> +
> +        cpu->hv_max_vps = hv_cpuid_get_host(cpuid, HV_CPUID_IMPLEMENT_LIMITS,
> +                                            R_EAX);
> +        cpu->hyperv_limits[0] =
> +            hv_cpuid_get_host(cpuid, HV_CPUID_IMPLEMENT_LIMITS, R_EBX);
> +        cpu->hyperv_limits[1] =
> +            hv_cpuid_get_host(cpuid, HV_CPUID_IMPLEMENT_LIMITS, R_ECX);
> +        cpu->hyperv_limits[2] =
> +            hv_cpuid_get_host(cpuid, HV_CPUID_IMPLEMENT_LIMITS, R_EDX);
> +
> +        cpu->hyperv_spinlock_attempts =
> +            hv_cpuid_get_host(cpuid, HV_CPUID_ENLIGHTMENT_INFO, R_EBX);
>      }
>  
>      /* Features */
> @@ -1348,10 +1362,8 @@ static int hyperv_handle_properties(CPUState *cs,
>      if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_ON) {
>          c->eax |= HV_NO_NONARCH_CORESHARING;
>      } else if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_AUTO) {
> -        c2 = cpuid_find_entry(cpuid, HV_CPUID_ENLIGHTMENT_INFO, 0);
> -        if (c2) {
> -            c->eax |= c2->eax & HV_NO_NONARCH_CORESHARING;
> -        }
> +        c->eax |= hv_cpuid_get_host(cpuid, HV_CPUID_ENLIGHTMENT_INFO, R_EAX) &
> +            HV_NO_NONARCH_CORESHARING;
>      }
>  
>      c = &cpuid_ent[cpuid_i++];
> -- 
> 2.30.2
> 

-- 
Eduardo

Re: [PATCH v6 07/19] i386: drop FEAT_HYPERV feature leaves

[Eduardo Habkost]

On Thu, Apr 22, 2021 at 06:11:18PM +0200, Vitaly Kuznetsov wrote:
> Hyper-V feature leaves are weird. We have some of them in
> feature_word_info[] array but we don't use feature_word_info
> magic to enable them. Neither do we use feature_dependencies[]
> mechanism to validate the configuration as it doesn't allign
> well with Hyper-V's many-to-many dependency chains. Some of
> the feature leaves hold not only feature bits, but also values.
> E.g. FEAT_HV_NESTED_EAX contains both features and the supported
> Enlightened VMCS range.
> 
> Hyper-V features are already represented in 'struct X86CPU' with
> uint64_t hyperv_features so duplicating them in env->features adds
> little (or zero) benefits. THe other half of Hyper-V emulation features
> is also stored with values in hyperv_vendor_id[], hyperv_limits[],...
> so env->features[] is already incomplete.
> 
> Remove Hyper-V feature leaves from env->features[] completely.
> kvm_hyperv_properties[] is converted to using raw CPUID func/reg
> pairs for features, this allows us to get rid of hv_cpuid_get_fw()
> conversion.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
>  target/i386/cpu.c     |  90 +----------------------------------
>  target/i386/cpu.h     |   5 --
>  target/i386/kvm/kvm.c | 108 ++++++++++++++----------------------------
>  3 files changed, 37 insertions(+), 166 deletions(-)

Nice diffstat!

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

-- 
Eduardo

Re: [PATCH v6 08/19] i386: introduce hv_cpuid_cache

[Eduardo Habkost]

On Thu, Apr 22, 2021 at 06:11:19PM +0200, Vitaly Kuznetsov wrote:
> Just like with cpuid_cache, it makes no sense to call
> KVM_GET_SUPPORTED_HV_CPUID more than once and instead of (ab)using
> env->features[] and/or trying to keep all the code in one place, it is
> better to introduce persistent hv_cpuid_cache and hv_cpuid_get_host()
> accessor to it.
> 
> Note, hv_cpuid_get_fw() is converted to using hv_cpuid_get_host()
> just to be removed later with Hyper-V specific feature words.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

-- 
Eduardo

Re: [PATCH v6 09/19] i386: split hyperv_handle_properties() into hyperv_expand_features()/hyperv_fill_cpuids()

[Eduardo Habkost]

On Thu, Apr 22, 2021 at 06:11:20PM +0200, Vitaly Kuznetsov wrote:
> The intention is to call hyperv_expand_features() early, before vCPUs
> are created and use the acquired data later when we set guest visible
> CPUID data.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

-- 
Eduardo

Re: [PATCH v6 04/19] i386: stop using env->features[] for filling Hyper-V CPUIDs

[Vitaly Kuznetsov]

Eduardo Habkost <ehabkost@redhat.com> writes:

> On Fri, Apr 30, 2021 at 08:34:40PM -0400, Eduardo Habkost wrote:
>> On Thu, Apr 22, 2021 at 06:11:15PM +0200, Vitaly Kuznetsov wrote:
>> > As a preparatory patch to dropping Hyper-V CPUID leaves from
>> > feature_word_info[] stop using env->features[] as a temporary
>> > storage of Hyper-V CPUIDs, just build Hyper-V CPUID leaves directly
>> > from kvm_hyperv_properties[] data.
>> > 
>> > Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
>> > ---
>> >  target/i386/cpu.h     |  1 +
>> >  target/i386/kvm/kvm.c | 80 +++++++++++++++++++++++--------------------
>> >  2 files changed, 43 insertions(+), 38 deletions(-)
>> > 
>> > diff --git a/target/i386/cpu.h b/target/i386/cpu.h
>> > index 570f916878f9..c8295aa2a1e7 100644
>> > --- a/target/i386/cpu.h
>> > +++ b/target/i386/cpu.h
>> > @@ -1684,6 +1684,7 @@ struct X86CPU {
>> >      uint32_t hyperv_interface_id[4];
>> >      uint32_t hyperv_version_id[4];
>> >      uint32_t hyperv_limits[3];
>> > +    uint32_t hyperv_nested[4];
>> >  
>> >      bool check_cpuid;
>> >      bool enforce_cpuid;
>> > diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
>> > index 7c751185491f..f791baa29acf 100644
>> > --- a/target/i386/kvm/kvm.c
>> > +++ b/target/i386/kvm/kvm.c
>> > @@ -1111,7 +1111,6 @@ static int hv_cpuid_check_and_set(CPUState *cs, struct kvm_cpuid2 *cpuid,
>> >                                    int feature)
>> >  {
>> >      X86CPU *cpu = X86_CPU(cs);
>> > -    CPUX86State *env = &cpu->env;
>> >      uint32_t r, fw, bits;
>> >      uint64_t deps;
>> >      int i, dep_feat;
>> > @@ -1151,8 +1150,6 @@ static int hv_cpuid_check_and_set(CPUState *cs, struct kvm_cpuid2 *cpuid,
>> >                  return 0;
>> >              }
>> >          }
>> > -
>> > -        env->features[fw] |= bits;
>> >      }
>> >  
>> >      if (cpu->hyperv_passthrough) {
>> > @@ -1162,6 +1159,29 @@ static int hv_cpuid_check_and_set(CPUState *cs, struct kvm_cpuid2 *cpuid,
>> >      return 0;
>> >  }
>> >  
>> > +static uint32_t hv_build_cpuid_leaf(CPUState *cs, uint32_t fw)
>> > +{
>> > +    X86CPU *cpu = X86_CPU(cs);
>> > +    uint32_t r = 0;
>> > +    int i, j;
>> > +
>> > +    for (i = 0; i < ARRAY_SIZE(kvm_hyperv_properties); i++) {
>> > +        if (!hyperv_feat_enabled(cpu, i)) {
>> > +            continue;
>> > +        }
>> > +
>> > +        for (j = 0; j < ARRAY_SIZE(kvm_hyperv_properties[i].flags); j++) {
>> > +            if (kvm_hyperv_properties[i].flags[j].fw != fw) {
>> > +                continue;
>> > +            }
>> > +
>> > +            r |= kvm_hyperv_properties[i].flags[j].bits;
>> > +        }
>> > +    }
>> > +
>> > +    return r;
>> > +}
>> > +
>> >  /*
>> >   * Fill in Hyper-V CPUIDs. Returns the number of entries filled in cpuid_ent in
>> >   * case of success, errno < 0 in case of failure and 0 when no Hyper-V
>> > @@ -1171,9 +1191,8 @@ static int hyperv_handle_properties(CPUState *cs,
>> >                                      struct kvm_cpuid_entry2 *cpuid_ent)
>> >  {
>> >      X86CPU *cpu = X86_CPU(cs);
>> > -    CPUX86State *env = &cpu->env;
>> >      struct kvm_cpuid2 *cpuid;
>> > -    struct kvm_cpuid_entry2 *c;
>> > +    struct kvm_cpuid_entry2 *c, *c2;
>> >      uint32_t cpuid_i = 0;
>> >      int r;
>> >  
>> > @@ -1194,9 +1213,7 @@ static int hyperv_handle_properties(CPUState *cs,
>> >          }
>> >  
>> >          if (!r) {
>> 
>> I think I now I understand why removing FEAT_HYPERV_* makes sense:
>> 
>> The rules mapping hyperv features to CPUID bits were encoded
>> twice in the code: in both hyperv_handle_properties() and
>> kvm_hyperv_properties[].  More work to maintain the rules, and
>> too easy to accidentally make them inconsistent.
>> 
>> 
>> Now, let me see if I can prove to myself that the new code works:
>> 
>> > -            env->features[FEAT_HV_RECOMM_EAX] |=
>> > -                HV_ENLIGHTENED_VMCS_RECOMMENDED;
>> 
>> [Line1]
>> 
>> The only code reading env->features[FEAT_HV_RECOMM_EAX] seems to
>> be [Line2] below:
>> 
>>   eax = env->features[FEAT_HV_RECOMM_EAX];
>> 
>> which is replaced with [Line3]:
>> 
>>   c->eax = hv_build_cpuid_leaf(cs, FEAT_HV_RECOMM_EAX);
>> 
>> so if hv_build_cpuid_leaf() do its job and set return
>> HV_ENLIGHTENED_VMCS_RECOMMENDED set at [Line2], we can safely
>> delete [Line1].
>> 
>> Will hv_build_cpuid_leaf() set HV_ENLIGHTENED_VMCS_RECOMMENDED in
>> [Line2] for all cases where [Line1] was being executed?
>> 
>> Let's check what's necessary to make hv_build_cpuid_leaf()
>> set HV_ENLIGHTENED_VMCS_RECOMMENDED:
>> 
>> There's only one entry with
>> FEAT_HV_RECOMM_EAX/HV_ENLIGHTENED_VMCS_RECOMMENDED at
>> kvm_hyperv_properties:
>> 
>>     [HYPERV_FEAT_EVMCS] = {
>>         .desc = "enlightened VMCS (hv-evmcs)",
>>         .flags = {
>>             {.fw = FEAT_HV_RECOMM_EAX,
>>              .bits = HV_ENLIGHTENED_VMCS_RECOMMENDED}
>>         },
>>         .dependencies = BIT(HYPERV_FEAT_VAPIC)
>>     },
>> 
>> The logic at hv_build_cpuid_leaf() will make
>> HV_ENLIGHTENED_VMCS_RECOMMENDED be set only if
>> hyperv_feat_enabled(HYPERV_FEAT_EVMCS) is true.  Which is what I
>> expected, because the line of code you are removing is inside a
>> hyperv_feat_enabled(cpu, HYPERV_FEAT_EVMCS) conditional.
>> 
>> For reference, hyperv_feat_enabled(cpu, feat) returns:
>>   !!(cpu->hyperv_features & BIT(feat))
>> 
>> I don't see any code _clearing_ hyperv_features, except for
>> properties that could change hyperv_features.  I don't expect the
>> "hv-evmcs" QOM property to be touched by this function, so we
>> should be safe: if hyperv_feat_enabled(cpu, HYPERV_FEAT_EVMCS)
>> returned before executing [Line1], it will return true when
>> executing [Line2].
>> 
>> This means hv_build_cpuid_leaf() will set
>> HV_ENLIGHTENED_VMCS_RECOMMENDED if
>> hyperv_feat_enabled(HYPERV_FEAT_EVMCS) is true, and
>> hyperv_feat_enabled(HYPERV_FEAT_EVMCS) will be true at [Line2] on
>> all cases when [Line1] was being executed.
>> 
>> We also need to be sure the HV_ENLIGHTENED_VMCS_RECOMMENDED will
>> be _unset_ at hv_build_cpuid_leaf() when expected, but the rules
>> are trickier (due to hyperv_passthrough). I'll try to prove that
>> later.
>> 
>> 
>> > -            env->features[FEAT_HV_NESTED_EAX] = evmcs_version;
>> 
>> [Line4]
>> 
>> 
>> Can we delete [Line4]?
>> 
>> The only code reading env->features[FEAT_HV_NESTED_EAX]
>> is [Line5]:
>>     c->eax = env->features[FEAT_HV_NESTED_EAX];
>> which is replaced with [Line6]:
>>     c->eax = cpu->hyperv_nested[0];
>> 
>> We are also replacing env->features[FEAT_HV_NESTED_EAX] with
>> cpu->hyperv_nested[0], here:
>> 
>> > +            cpu->hyperv_nested[0] = evmcs_version;
>> 
>> This will make [Line6] set c->eax to evmcs_version, unless other
>> code writes to cpu->hyperv_nested[0].
>> 
>> I don't see any other code writing to hyperv_nested in this
>> patch, so we are good.
>> 
>> >          }
>> >      }
>> >  
>> > @@ -1235,13 +1252,6 @@ static int hyperv_handle_properties(CPUState *cs,
>> >              cpu->hyperv_version_id[3] = c->edx;
>> >          }
>> >  
>> > -        c = cpuid_find_entry(cpuid, HV_CPUID_FEATURES, 0);
>> > -        if (c) {
>> > -            env->features[FEAT_HYPERV_EAX] = c->eax;
>> 
>> [Line7A]
>> 
>> > -            env->features[FEAT_HYPERV_EBX] = c->ebx;
>> 
>> [Line7B]
>> 
>> > -            env->features[FEAT_HYPERV_EDX] = c->edx;
>> 
>> [Line7D]
>> 
>> 
>> Can we delete [Line7*]?
>> 
>> The only code reading env->features[FEAT_HYPERV_*] seems to be
>> [Line8*]:
>>     c->eax = env->features[FEAT_HYPERV_EAX];
>>     c->ebx = env->features[FEAT_HYPERV_EBX];
>>     c->edx = env->features[FEAT_HYPERV_EDX];
>> which is replaced by [Line9*]:
>>     c->eax = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EAX);
>>     c->ebx = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EBX);
>>     c->edx = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EDX);
>> 
>> So we need to make sure hv_build_cpuid_leaf() will return the
>> right values at [Line9*].
>> 
>> This one will be trickier to evaluate: there are lots of entries in
>> kvm_hyperv_properties[] that affect FEAT_HYPERV_EAX.
>> 
>> I will pause here and continue next week.   :)
>> 
>
> Continuing:
>
> So, [Line7*] above was for hyperv_passthrough mode only.  This
> means now hv-passthrough will enable only known features (the
> ones at kvm_hyperv_properties).
>
> The same comment I sent to the previous patch applies here:
>
> """
> This makes hv-passthrough less useful for debugging and
> development, but safer for using in production.  I assume we want
> to eventually make hv-passthrough supported in production when
> live migration support isn't required.
>
> I'll trust your judgement here and assume this is really a good
> change, but maybe this should be documented more explicitly in
> the hv-passthrough section at docs/hyperv.txt?
> """
>

Makes sense, will do.

>
>
>> This smells like it could have been split into smaller patches,
>> but I'm not sure if it would be possible.  Maybe the existing
>> code was too tangled for splitting this refactor into smaller
>> patches.
>> 
>> 
>> > -        }
>> > -
>> >          c = cpuid_find_entry(cpuid, HV_CPUID_IMPLEMENT_LIMITS, 0);
>> >          if (c) {
>> >              cpu->hv_max_vps = c->eax;
>> > @@ -1252,23 +1262,8 @@ static int hyperv_handle_properties(CPUState *cs,
>> >  
>> >          c = cpuid_find_entry(cpuid, HV_CPUID_ENLIGHTMENT_INFO, 0);
>> >          if (c) {
>> > -            env->features[FEAT_HV_RECOMM_EAX] = c->eax;
>
> Same as above: this is hv-passthrough code, and the comment above
> applies.
>
>> >              cpu->hyperv_spinlock_attempts = c->ebx;
>> >          }
>> > -        c = cpuid_find_entry(cpuid, HV_CPUID_NESTED_FEATURES, 0);
>> > -        if (c) {
>> > -            env->features[FEAT_HV_NESTED_EAX] = c->eax;
>> > -        }
>
> Same as above: this is hv-passthrough code, and the comment above
> applies.
>
>> > -    }
>> > -
>
> Now we're outside the hv-passthrough code:
>
>> > -    if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_ON) {
>> > -        env->features[FEAT_HV_RECOMM_EAX] |= HV_NO_NONARCH_CORESHARING;
>> > -    } else if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_AUTO) {
>> > -        c = cpuid_find_entry(cpuid, HV_CPUID_ENLIGHTMENT_INFO, 0);
>> > -        if (c) {
>> > -            env->features[FEAT_HV_RECOMM_EAX] |=
>> > -                c->eax & HV_NO_NONARCH_CORESHARING;
>> > -        }
>> >      }
>
> The hack above is copied to [Line10] below.  Looks OK.
>
>> >  
>> >      /* Features */
>> > @@ -1298,9 +1293,6 @@ static int hyperv_handle_properties(CPUState *cs,
>> >          r |= 1;
>> >      }
>> >  
>> > -    /* Not exposed by KVM but needed to make CPU hotplug in Windows work */
>> > -    env->features[FEAT_HYPERV_EDX] |= HV_CPU_DYNAMIC_PARTITIONING_AVAILABLE;
>
> The hack above is copied to [Line10] below.  Looks OK.
>
>> > -
>> >      if (r) {
>> >          r = -ENOSYS;
>> >          goto free;
>> > @@ -1330,15 +1322,27 @@ static int hyperv_handle_properties(CPUState *cs,
>> >  
>> >      c = &cpuid_ent[cpuid_i++];
>> >      c->function = HV_CPUID_FEATURES;
>> > -    c->eax = env->features[FEAT_HYPERV_EAX];
>> 
>> [Line8A]
>> 
>> > -    c->ebx = env->features[FEAT_HYPERV_EBX];
>> 
>> [Line8B]
>> 
>> > -    c->edx = env->features[FEAT_HYPERV_EDX];
>> 
>> [Line8D]
>> 
>> > +    c->eax = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EAX);
>> 
>> [Line9A]
>> 
>> > +    c->ebx = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EBX);
>> 
>> [Line9B]
>> 
>> > +    c->edx = hv_build_cpuid_leaf(cs, FEAT_HYPERV_EDX);
>> 
>> [Line9D]
>
> Already reviewed at [Line7*] above.
>
>> 
>> > +
>> > +    /* Not exposed by KVM but needed to make CPU hotplug in Windows work */
>> > +    c->edx |= HV_CPU_DYNAMIC_PARTITIONING_AVAILABLE;
>
> [Line11]
>
> Looks OK, but I wonder if this could be encoded in
> kvm_hyperv_properties somehow.
>

(The patch series changes nothing in this regard) this is not a separate
feature as we enable it unconditionally. It is similar to e.g
HV_HYPERCALL_AVAILABLE which we add unconditionally. The only reason why
we can't just move it to kvm_hyperv_properties[] is because it is not
exposed by KVM (and I'm not sure why to be honest but it is already
'legacy').

>
>> >  
>> >      c = &cpuid_ent[cpuid_i++];
>> >      c->function = HV_CPUID_ENLIGHTMENT_INFO;
>> > -    c->eax = env->features[FEAT_HV_RECOMM_EAX];
>> 
>> [Line2]
>> 
>> > +    c->eax = hv_build_cpuid_leaf(cs, FEAT_HV_RECOMM_EAX);
>> 
>> [Line3]
>> 
>> >      c->ebx = cpu->hyperv_spinlock_attempts;
>> >  
>> > +    if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_ON) {
>> > +        c->eax |= HV_NO_NONARCH_CORESHARING;
>> > +    } else if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_AUTO) {
>> > +        c2 = cpuid_find_entry(cpuid, HV_CPUID_ENLIGHTMENT_INFO, 0);
>> > +        if (c2) {
>> > +            c->eax |= c2->eax & HV_NO_NONARCH_CORESHARING;
>> > +        }
>> > +    }
>
> [Line10]
>
> Matches the code above.
>
>> > +
>> >      c = &cpuid_ent[cpuid_i++];
>> >      c->function = HV_CPUID_IMPLEMENT_LIMITS;
>> >      c->eax = cpu->hv_max_vps;
>> > @@ -1358,7 +1362,7 @@ static int hyperv_handle_properties(CPUState *cs,
>> >  
>> >          c = &cpuid_ent[cpuid_i++];
>> >          c->function = HV_CPUID_NESTED_FEATURES;
>> > -        c->eax = env->features[FEAT_HV_NESTED_EAX];
>> > +        c->eax = cpu->hyperv_nested[0];
>> >      }
>> >      r = cpuid_i;
>> >  
>> > -- 
>> > 2.30.2
>> > 
>> 
>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>

Thanks!

-- 
Vitaly

Re: [PATCH v6 05/19] i386: introduce hyperv_feature_supported()

[Vitaly Kuznetsov]

Eduardo Habkost <ehabkost@redhat.com> writes:

> On Thu, Apr 22, 2021 at 06:11:16PM +0200, Vitaly Kuznetsov wrote:
>> Clean up hv_cpuid_check_and_set() by separating hyperv_feature_supported()
>> off it. No functional change intended.
>> 
>> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
>> ---
>>  target/i386/kvm/kvm.c | 49 ++++++++++++++++++++++++++-----------------
>>  1 file changed, 30 insertions(+), 19 deletions(-)
>> 
>> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
>> index f791baa29acf..ba093dba4d23 100644
>> --- a/target/i386/kvm/kvm.c
>> +++ b/target/i386/kvm/kvm.c
>> @@ -1107,13 +1107,33 @@ static int hv_cpuid_get_fw(struct kvm_cpuid2 *cpuid, int fw, uint32_t *r)
>>      return 0;
>>  }
>>  
>> +static bool hyperv_feature_supported(struct kvm_cpuid2 *cpuid, int feature)
>> +{
>> +    uint32_t r, fw, bits;
>> +    int i;
>> +
>> +    for (i = 0; i < ARRAY_SIZE(kvm_hyperv_properties[feature].flags); i++) {
>> +        fw = kvm_hyperv_properties[feature].flags[i].fw;
>> +        bits = kvm_hyperv_properties[feature].flags[i].bits;
>> +
>> +        if (!fw) {
>> +            continue;
>> +        }
>> +
>> +        if (hv_cpuid_get_fw(cpuid, fw, &r) || (r & bits) != bits) {
>> +            return false;
>> +        }
>> +    }
>> +
>> +    return true;
>> +}
>> +
>>  static int hv_cpuid_check_and_set(CPUState *cs, struct kvm_cpuid2 *cpuid,
>>                                    int feature)
>>  {
>>      X86CPU *cpu = X86_CPU(cs);
>> -    uint32_t r, fw, bits;
>>      uint64_t deps;
>> -    int i, dep_feat;
>> +    int dep_feat;
>>  
>>      if (!hyperv_feat_enabled(cpu, feature) && !cpu->hyperv_passthrough) {
>>          return 0;
>> @@ -1132,23 +1152,14 @@ static int hv_cpuid_check_and_set(CPUState *cs, struct kvm_cpuid2 *cpuid,
>>          deps &= ~(1ull << dep_feat);
>>      }
>>  
>> -    for (i = 0; i < ARRAY_SIZE(kvm_hyperv_properties[feature].flags); i++) {
>> -        fw = kvm_hyperv_properties[feature].flags[i].fw;
>> -        bits = kvm_hyperv_properties[feature].flags[i].bits;
>> -
>> -        if (!fw) {
>> -            continue;
>> -        }
>> -
>> -        if (hv_cpuid_get_fw(cpuid, fw, &r) || (r & bits) != bits) {
>> -            if (hyperv_feat_enabled(cpu, feature)) {
>> -                fprintf(stderr,
>> -                        "Hyper-V %s is not supported by kernel\n",
>> -                        kvm_hyperv_properties[feature].desc);
>> -                return 1;
>> -            } else {
>> -                return 0;
>> -            }
>> +    if (!hyperv_feature_supported(cpuid, feature)) {
>> +        if (hyperv_feat_enabled(cpu, feature)) {
>> +            fprintf(stderr,
>> +                    "Hyper-V %s is not supported by kernel\n",
>> +                    kvm_hyperv_properties[feature].desc);
>> +            return 1;
>> +        } else {
>> +            return 0;
>
> The reason for returning prematurely here when
> !hyperv_feat_enabled() is not clear to me

This hopefully gets much better at the end of the series where
hv_cpuid_check_and_set() is split into 'check' and 'set'. The reason to
return immediately is: if the feature was not requested explicitly and
we're not in 'hv-passthrough' mode, there's no need to check whether KVM
supports it, if we have all the dependencies,... 

> but you are keeping the existing behavior, so:
>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>

Thanks!

-- 
Vitaly

Re: [PATCH v6 06/19] i386: introduce hv_cpuid_get_host()

[Vitaly Kuznetsov]

Eduardo Habkost <ehabkost@redhat.com> writes:

> On Thu, Apr 22, 2021 at 06:11:17PM +0200, Vitaly Kuznetsov wrote:
>> As a preparation to implementing hv_cpuid_cache intro introduce
>> hv_cpuid_get_host(). No functional change intended.
>> 
>> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
>> ---
>>  target/i386/kvm/kvm.c | 102 +++++++++++++++++++++++-------------------
>>  1 file changed, 57 insertions(+), 45 deletions(-)
>> 
>> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
>> index ba093dba4d23..7aeb704b016e 100644
>> --- a/target/i386/kvm/kvm.c
>> +++ b/target/i386/kvm/kvm.c
>> @@ -1107,6 +1107,19 @@ static int hv_cpuid_get_fw(struct kvm_cpuid2 *cpuid, int fw, uint32_t *r)
>>      return 0;
>>  }
>>  
>> +static uint32_t hv_cpuid_get_host(struct kvm_cpuid2 *cpuid, uint32_t func,
>> +                                  int reg)
>> +{
>> +    struct kvm_cpuid_entry2 *entry;
>> +
>> +    entry = cpuid_find_entry(cpuid, func, 0);
>> +    if (!entry) {
>> +        return 0;
>
> One possible difference here is that leaves not supported by the
> host will now be zeroed out.
>
> ...which I expected to be the correct behavior of hv-passthrough,

Yes, at least that's the intention.

> so:
>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>

Thanks!

>
>> +    }
>> +
>> +    return cpuid_entry_get_reg(entry, reg);
>> +}
>> +
>>  static bool hyperv_feature_supported(struct kvm_cpuid2 *cpuid, int feature)
>>  {
>>      uint32_t r, fw, bits;
>> @@ -1203,7 +1216,7 @@ static int hyperv_handle_properties(CPUState *cs,
>>  {
>>      X86CPU *cpu = X86_CPU(cs);
>>      struct kvm_cpuid2 *cpuid;
>> -    struct kvm_cpuid_entry2 *c, *c2;
>> +    struct kvm_cpuid_entry2 *c;
>>      uint32_t cpuid_i = 0;
>>      int r;
>>  
>> @@ -1235,46 +1248,47 @@ static int hyperv_handle_properties(CPUState *cs,
>>      }
>>  
>>      if (cpu->hyperv_passthrough) {
>> -        c = cpuid_find_entry(cpuid, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, 0);
>> -        if (c) {
>> -            cpu->hyperv_vendor_id[0] = c->ebx;
>> -            cpu->hyperv_vendor_id[1] = c->ecx;
>> -            cpu->hyperv_vendor_id[2] = c->edx;
>> -            cpu->hyperv_vendor = g_realloc(cpu->hyperv_vendor,
>> -                                           sizeof(cpu->hyperv_vendor_id) + 1);
>> -            memcpy(cpu->hyperv_vendor, cpu->hyperv_vendor_id,
>> -                   sizeof(cpu->hyperv_vendor_id));
>> -            cpu->hyperv_vendor[sizeof(cpu->hyperv_vendor_id)] = 0;
>> -        }
>> -
>> -        c = cpuid_find_entry(cpuid, HV_CPUID_INTERFACE, 0);
>> -        if (c) {
>> -            cpu->hyperv_interface_id[0] = c->eax;
>> -            cpu->hyperv_interface_id[1] = c->ebx;
>> -            cpu->hyperv_interface_id[2] = c->ecx;
>> -            cpu->hyperv_interface_id[3] = c->edx;
>> -        }
>> -
>> -        c = cpuid_find_entry(cpuid, HV_CPUID_VERSION, 0);
>> -        if (c) {
>> -            cpu->hyperv_version_id[0] = c->eax;
>> -            cpu->hyperv_version_id[1] = c->ebx;
>> -            cpu->hyperv_version_id[2] = c->ecx;
>> -            cpu->hyperv_version_id[3] = c->edx;
>> -        }
>> -
>> -        c = cpuid_find_entry(cpuid, HV_CPUID_IMPLEMENT_LIMITS, 0);
>> -        if (c) {
>> -            cpu->hv_max_vps = c->eax;
>> -            cpu->hyperv_limits[0] = c->ebx;
>> -            cpu->hyperv_limits[1] = c->ecx;
>> -            cpu->hyperv_limits[2] = c->edx;
>> -        }
>> -
>> -        c = cpuid_find_entry(cpuid, HV_CPUID_ENLIGHTMENT_INFO, 0);
>> -        if (c) {
>> -            cpu->hyperv_spinlock_attempts = c->ebx;
>> -        }
>> +        cpu->hyperv_vendor_id[0] =
>> +            hv_cpuid_get_host(cpuid, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, R_EBX);
>> +        cpu->hyperv_vendor_id[1] =
>> +            hv_cpuid_get_host(cpuid, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, R_ECX);
>> +        cpu->hyperv_vendor_id[2] =
>> +            hv_cpuid_get_host(cpuid, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, R_EDX);
>> +        cpu->hyperv_vendor = g_realloc(cpu->hyperv_vendor,
>> +                                       sizeof(cpu->hyperv_vendor_id) + 1);
>> +        memcpy(cpu->hyperv_vendor, cpu->hyperv_vendor_id,
>> +               sizeof(cpu->hyperv_vendor_id));
>> +        cpu->hyperv_vendor[sizeof(cpu->hyperv_vendor_id)] = 0;
>> +
>> +        cpu->hyperv_interface_id[0] =
>> +            hv_cpuid_get_host(cpuid, HV_CPUID_INTERFACE, R_EAX);
>> +        cpu->hyperv_interface_id[1] =
>> +            hv_cpuid_get_host(cpuid, HV_CPUID_INTERFACE, R_EBX);
>> +        cpu->hyperv_interface_id[2] =
>> +            hv_cpuid_get_host(cpuid, HV_CPUID_INTERFACE, R_ECX);
>> +        cpu->hyperv_interface_id[3] =
>> +            hv_cpuid_get_host(cpuid, HV_CPUID_INTERFACE, R_EDX);
>> +
>> +        cpu->hyperv_version_id[0] =
>> +            hv_cpuid_get_host(cpuid, HV_CPUID_VERSION, R_EAX);
>> +        cpu->hyperv_version_id[1] =
>> +            hv_cpuid_get_host(cpuid, HV_CPUID_VERSION, R_EBX);
>> +        cpu->hyperv_version_id[2] =
>> +            hv_cpuid_get_host(cpuid, HV_CPUID_VERSION, R_ECX);
>> +        cpu->hyperv_version_id[3] =
>> +            hv_cpuid_get_host(cpuid, HV_CPUID_VERSION, R_EDX);
>> +
>> +        cpu->hv_max_vps = hv_cpuid_get_host(cpuid, HV_CPUID_IMPLEMENT_LIMITS,
>> +                                            R_EAX);
>> +        cpu->hyperv_limits[0] =
>> +            hv_cpuid_get_host(cpuid, HV_CPUID_IMPLEMENT_LIMITS, R_EBX);
>> +        cpu->hyperv_limits[1] =
>> +            hv_cpuid_get_host(cpuid, HV_CPUID_IMPLEMENT_LIMITS, R_ECX);
>> +        cpu->hyperv_limits[2] =
>> +            hv_cpuid_get_host(cpuid, HV_CPUID_IMPLEMENT_LIMITS, R_EDX);
>> +
>> +        cpu->hyperv_spinlock_attempts =
>> +            hv_cpuid_get_host(cpuid, HV_CPUID_ENLIGHTMENT_INFO, R_EBX);
>>      }
>>  
>>      /* Features */
>> @@ -1348,10 +1362,8 @@ static int hyperv_handle_properties(CPUState *cs,
>>      if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_ON) {
>>          c->eax |= HV_NO_NONARCH_CORESHARING;
>>      } else if (cpu->hyperv_no_nonarch_cs == ON_OFF_AUTO_AUTO) {
>> -        c2 = cpuid_find_entry(cpuid, HV_CPUID_ENLIGHTMENT_INFO, 0);
>> -        if (c2) {
>> -            c->eax |= c2->eax & HV_NO_NONARCH_CORESHARING;
>> -        }
>> +        c->eax |= hv_cpuid_get_host(cpuid, HV_CPUID_ENLIGHTMENT_INFO, R_EAX) &
>> +            HV_NO_NONARCH_CORESHARING;
>>      }
>>  
>>      c = &cpuid_ent[cpuid_i++];
>> -- 
>> 2.30.2
>> 

-- 
Vitaly

Re: [PATCH v6 10/19] i386: move eVMCS enablement to hyperv_init_vcpu()

[Eduardo Habkost]

On Thu, Apr 22, 2021 at 06:11:21PM +0200, Vitaly Kuznetsov wrote:
> hyperv_expand_features() will be called before we create vCPU so
> evmcs enablement should go away. hyperv_init_vcpu() looks like the
> right place.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
>  target/i386/kvm/kvm.c | 60 ++++++++++++++++++++++++++-----------------
>  1 file changed, 37 insertions(+), 23 deletions(-)
> 
> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
> index 6b391db7a030..a2ef2dc154a2 100644
> --- a/target/i386/kvm/kvm.c
> +++ b/target/i386/kvm/kvm.c
> @@ -962,6 +962,7 @@ static struct kvm_cpuid2 *get_supported_hv_cpuid(CPUState *cs)
>  {
>      struct kvm_cpuid2 *cpuid;
>      int max = 7; /* 0x40000000..0x40000005, 0x4000000A */
> +    int i;
>  
>      /*
>       * When the buffer is too small, KVM_GET_SUPPORTED_HV_CPUID fails with
> @@ -971,6 +972,22 @@ static struct kvm_cpuid2 *get_supported_hv_cpuid(CPUState *cs)
>      while ((cpuid = try_get_hv_cpuid(cs, max)) == NULL) {
>          max++;
>      }
> +
> +    /*
> +     * KVM_GET_SUPPORTED_HV_CPUID does not set EVMCS CPUID bit before
> +     * KVM_CAP_HYPERV_ENLIGHTENED_VMCS is enabled but we want to get the
> +     * information early, just check for the capability and set the bit
> +     * manually.
> +     */

Should we add a comment noting that this hack won't be necessary
if using the system ioctl?  I assume we still want to support
Linux < v5.11 for a while, so simply 


> +    if (kvm_check_extension(cs->kvm_state,
> +                            KVM_CAP_HYPERV_ENLIGHTENED_VMCS) > 0) {
> +        for (i = 0; i < cpuid->nent; i++) {
> +            if (cpuid->entries[i].function == HV_CPUID_ENLIGHTMENT_INFO) {
> +                cpuid->entries[i].eax |= HV_ENLIGHTENED_VMCS_RECOMMENDED;
> +            }
> +        }
> +    }
> +
>      return cpuid;
>  }
>  
> @@ -1200,24 +1217,6 @@ static int hyperv_expand_features(CPUState *cs)
>      if (!hyperv_enabled(cpu))
>          return 0;
>  
> -    if (hyperv_feat_enabled(cpu, HYPERV_FEAT_EVMCS) ||
> -        cpu->hyperv_passthrough) {
> -        uint16_t evmcs_version;
> -
> -        r = kvm_vcpu_enable_cap(cs, KVM_CAP_HYPERV_ENLIGHTENED_VMCS, 0,
> -                                (uintptr_t)&evmcs_version);
> -
> -        if (hyperv_feat_enabled(cpu, HYPERV_FEAT_EVMCS) && r) {
> -            fprintf(stderr, "Hyper-V %s is not supported by kernel\n",
> -                    kvm_hyperv_properties[HYPERV_FEAT_EVMCS].desc);
> -            return -ENOSYS;
> -        }
> -
> -        if (!r) {
> -            cpu->hyperv_nested[0] = evmcs_version;
> -        }
> -    }
> -
>      if (cpu->hyperv_passthrough) {
>          cpu->hyperv_vendor_id[0] =
>              hv_cpuid_get_host(cs, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, R_EBX);
> @@ -1455,6 +1454,21 @@ static int hyperv_init_vcpu(X86CPU *cpu)
>          }
>      }
>  
> +    if (hyperv_feat_enabled(cpu, HYPERV_FEAT_EVMCS)) {
> +        uint16_t evmcs_version;
> +
> +        ret = kvm_vcpu_enable_cap(cs, KVM_CAP_HYPERV_ENLIGHTENED_VMCS, 0,
> +                                  (uintptr_t)&evmcs_version);
> +
> +        if (ret < 0) {
> +            fprintf(stderr, "Hyper-V %s is not supported by kernel\n",
> +                    kvm_hyperv_properties[HYPERV_FEAT_EVMCS].desc);
> +            return ret;
> +        }
> +
> +        cpu->hyperv_nested[0] = evmcs_version;

Wait, won't this break guest ABI?  Do we want to make
HYPERV_FEAT_EVMCS a migration blocker until this is fixed?


> +    }
> +
>      return 0;
>  }
>  
> @@ -1519,6 +1533,11 @@ int kvm_arch_init_vcpu(CPUState *cs)
>      }
>  
>      if (hyperv_enabled(cpu)) {
> +        r = hyperv_init_vcpu(cpu);
> +        if (r) {
> +            return r;
> +        }
> +
>          cpuid_i = hyperv_fill_cpuids(cs, cpuid_data.entries);
>          kvm_base = KVM_CPUID_SIGNATURE_NEXT;
>          has_msr_hv_hypercall = true;
> @@ -1868,11 +1887,6 @@ int kvm_arch_init_vcpu(CPUState *cs)
>  
>      kvm_init_msrs(cpu);
>  
> -    r = hyperv_init_vcpu(cpu);
> -    if (r) {
> -        goto fail;
> -    }
> -
>      return 0;

I would move the two hunks above to a separate patch, but not a
big deal.  The guest ABI issue is existing, and the comment
suggestion can be done in a follow up patch, so:

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

>  
>   fail:
> -- 
> 2.30.2
> 

-- 
Eduardo

Re: [PATCH v6 11/19] i386: switch hyperv_expand_features() to using error_setg()

[Eduardo Habkost]

On Thu, Apr 22, 2021 at 06:11:22PM +0200, Vitaly Kuznetsov wrote:
> Use standard error_setg() mechanism in hyperv_expand_features().
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>

No objections, but only suggestions below:

> ---
>  target/i386/kvm/kvm.c | 101 +++++++++++++++++++++++++-----------------
>  1 file changed, 61 insertions(+), 40 deletions(-)
> 
> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
> index a2ef2dc154a2..f33ba325187f 100644
> --- a/target/i386/kvm/kvm.c
> +++ b/target/i386/kvm/kvm.c
> @@ -1135,7 +1135,7 @@ static bool hyperv_feature_supported(CPUState *cs, int feature)
>      return true;
>  }
>  
> -static int hv_cpuid_check_and_set(CPUState *cs, int feature)
> +static int hv_cpuid_check_and_set(CPUState *cs, int feature, Error **errp)

If changing the function signature, and the function only returns 0 or 1, maybe
it's a good opportunity to change to a bool return value format?

From include/qapi/error.h:

 * - Whenever practical, also return a value that indicates success /
 *   failure.  This can make the error checking more concise, and can
 *   avoid useless error object creation and destruction.  Note that
 *   we still have many functions returning void.  We recommend
 *   • bool-valued functions return true on success / false on failure,
 *   • pointer-valued functions return non-null / null pointer, and
 *   • integer-valued functions return non-negative / negative.


>  {
>      X86CPU *cpu = X86_CPU(cs);
>      uint64_t deps;
> @@ -1149,20 +1149,18 @@ static int hv_cpuid_check_and_set(CPUState *cs, int feature)
>      while (deps) {
>          dep_feat = ctz64(deps);
>          if (!(hyperv_feat_enabled(cpu, dep_feat))) {
> -                fprintf(stderr,
> -                        "Hyper-V %s requires Hyper-V %s\n",
> -                        kvm_hyperv_properties[feature].desc,
> -                        kvm_hyperv_properties[dep_feat].desc);
> -                return 1;
> +            error_setg(errp, "Hyper-V %s requires Hyper-V %s",
> +                       kvm_hyperv_properties[feature].desc,
> +                       kvm_hyperv_properties[dep_feat].desc);
> +            return 1;
>          }
>          deps &= ~(1ull << dep_feat);
>      }
>  
>      if (!hyperv_feature_supported(cs, feature)) {
>          if (hyperv_feat_enabled(cpu, feature)) {
> -            fprintf(stderr,
> -                    "Hyper-V %s is not supported by kernel\n",
> -                    kvm_hyperv_properties[feature].desc);
> +            error_setg(errp, "Hyper-V %s is not supported by kernel",
> +                       kvm_hyperv_properties[feature].desc);
>              return 1;
>          } else {
>              return 0;
> @@ -1209,13 +1207,12 @@ static uint32_t hv_build_cpuid_leaf(CPUState *cs, uint32_t func, int reg)
>   * of 'hv_passthrough' mode and fills the environment with all supported
>   * Hyper-V features.
>   */
> -static int hyperv_expand_features(CPUState *cs)
> +static void hyperv_expand_features(CPUState *cs, Error **errp)

Same as above: returning a value to indicate error is preferred.  If you are no
longer returning an integer error code, I suggest returning bool instead.

>  {
>      X86CPU *cpu = X86_CPU(cs);
> -    int r;
>  
>      if (!hyperv_enabled(cpu))
> -        return 0;
> +        return;
>  
>      if (cpu->hyperv_passthrough) {
>          cpu->hyperv_vendor_id[0] =
> @@ -1262,37 +1259,60 @@ static int hyperv_expand_features(CPUState *cs)
>      }
>  
>      /* Features */
> -    r = hv_cpuid_check_and_set(cs, HYPERV_FEAT_RELAXED);
> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_VAPIC);
> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_TIME);
> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_CRASH);
> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_RESET);
> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_VPINDEX);
> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_RUNTIME);
> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_SYNIC);
> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_STIMER);
> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_FREQUENCIES);
> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_REENLIGHTENMENT);
> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_TLBFLUSH);
> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_EVMCS);
> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_IPI);
> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_STIMER_DIRECT);
> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_RELAXED, errp)) {
> +        return;
> +    }

What about a loop?

    for (feat = 0; feat < ARRAY_SIZE(kvm_hyperv_properties); feat++) {
        if (hv_cpuid_check_and_set(cs, feat, errp)) {
            return;
        }
    }

> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_VAPIC, errp)) {
> +        return;
> +    }
> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_TIME, errp)) {
> +        return;
> +    }
> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_CRASH, errp)) {
> +        return;
> +    }
> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_RESET, errp)) {
> +        return;
> +    }
> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_VPINDEX, errp)) {
> +        return;
> +    }
> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_RUNTIME, errp)) {
> +        return;
> +    }
> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_SYNIC, errp)) {
> +        return;
> +    }
> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_STIMER, errp)) {
> +        return;
> +    }
> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_FREQUENCIES, errp)) {
> +        return;
> +    }
> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_REENLIGHTENMENT, errp)) {
> +        return;
> +    }
> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_TLBFLUSH, errp)) {
> +        return;
> +    }
> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_EVMCS, errp)) {
> +        return;
> +    }
> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_IPI, errp)) {
> +        return;
> +    }
> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_STIMER_DIRECT, errp)) {
> +        return;
> +    }
>  
>      /* Additional dependencies not covered by kvm_hyperv_properties[] */
>      if (hyperv_feat_enabled(cpu, HYPERV_FEAT_SYNIC) &&
>          !cpu->hyperv_synic_kvm_only &&
>          !hyperv_feat_enabled(cpu, HYPERV_FEAT_VPINDEX)) {
> -        fprintf(stderr, "Hyper-V %s requires Hyper-V %s\n",
> -                kvm_hyperv_properties[HYPERV_FEAT_SYNIC].desc,
> -                kvm_hyperv_properties[HYPERV_FEAT_VPINDEX].desc);
> -        r |= 1;
> -    }
> -
> -    if (r) {
> -        return -ENOSYS;
> +        error_setg(errp, "Hyper-V %s requires Hyper-V %s",
> +                   kvm_hyperv_properties[HYPERV_FEAT_SYNIC].desc,
> +                   kvm_hyperv_properties[HYPERV_FEAT_VPINDEX].desc);
>      }
> -
> -    return 0;
>  }
>  
>  /*
> @@ -1527,9 +1547,10 @@ int kvm_arch_init_vcpu(CPUState *cs)
>      env->apic_bus_freq = KVM_APIC_BUS_FREQUENCY;
>  
>      /* Paravirtualization CPUIDs */
> -    r = hyperv_expand_features(cs);
> -    if (r < 0) {
> -        return r;
> +    hyperv_expand_features(cs, &local_err);
> +    if (local_err) {
> +        error_report_err(local_err);
> +        return -ENOSYS;
>      }
>  
>      if (hyperv_enabled(cpu)) {

I don't want to block this series because of the suggestions above, so:

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

But I still encourage you to implement those suggestions, anyway.

-- 
Eduardo

Re: [PATCH v6 12/19] i386: adjust the expected KVM_GET_SUPPORTED_HV_CPUID array size

[Eduardo Habkost]

On Thu, Apr 22, 2021 at 06:11:23PM +0200, Vitaly Kuznetsov wrote:
> SYNDBG leaves were recently (Linux-5.8) added to KVM but we haven't
> updated the expected size of KVM_GET_SUPPORTED_HV_CPUID output in
> KVM so we now make serveral tries before succeeding. Update the
> default.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

-- 
Eduardo

Re: [PATCH v6 13/19] i386: prefer system KVM_GET_SUPPORTED_HV_CPUID ioctl over vCPU's one

[Eduardo Habkost]

On Thu, Apr 22, 2021 at 06:11:24PM +0200, Vitaly Kuznetsov wrote:
> KVM_GET_SUPPORTED_HV_CPUID was made a system wide ioctl which can be called
> prior to creating vCPUs and we are going to use that to expand Hyper-V cpu
> features early. Use it when it is supported by KVM.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
>  target/i386/kvm/kvm.c | 17 +++++++++++++----
>  1 file changed, 13 insertions(+), 4 deletions(-)
> 
> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
> index feec9f25ba12..5d08f3a39ef7 100644
> --- a/target/i386/kvm/kvm.c
> +++ b/target/i386/kvm/kvm.c
> @@ -928,7 +928,8 @@ static struct {
>      },
>  };
>  
> -static struct kvm_cpuid2 *try_get_hv_cpuid(CPUState *cs, int max)
> +static struct kvm_cpuid2 *try_get_hv_cpuid(CPUState *cs, int max,
> +                                           bool do_sys_ioctl)
>  {
>      struct kvm_cpuid2 *cpuid;
>      int r, size;
> @@ -937,7 +938,11 @@ static struct kvm_cpuid2 *try_get_hv_cpuid(CPUState *cs, int max)
>      cpuid = g_malloc0(size);
>      cpuid->nent = max;
>  
> -    r = kvm_vcpu_ioctl(cs, KVM_GET_SUPPORTED_HV_CPUID, cpuid);
> +    if (do_sys_ioctl) {
> +        r = kvm_ioctl(kvm_state, KVM_GET_SUPPORTED_HV_CPUID, cpuid);
> +    } else {
> +        r = kvm_vcpu_ioctl(cs, KVM_GET_SUPPORTED_HV_CPUID, cpuid);
> +    }
>      if (r == 0 && cpuid->nent >= max) {
>          r = -E2BIG;
>      }
> @@ -964,13 +969,17 @@ static struct kvm_cpuid2 *get_supported_hv_cpuid(CPUState *cs)
>      /* 0x40000000..0x40000005, 0x4000000A, 0x40000080..0x40000080 leaves */
>      int max = 10;
>      int i;
> +    bool do_sys_ioctl;
> +
> +    do_sys_ioctl =
> +        kvm_check_extension(kvm_state, KVM_CAP_SYS_HYPERV_CPUID) > 0;
>  
>      /*
>       * When the buffer is too small, KVM_GET_SUPPORTED_HV_CPUID fails with
>       * -E2BIG, however, it doesn't report back the right size. Keep increasing
>       * it and re-trying until we succeed.
>       */
> -    while ((cpuid = try_get_hv_cpuid(cs, max)) == NULL) {
> +    while ((cpuid = try_get_hv_cpuid(cs, max, do_sys_ioctl)) == NULL) {
>          max++;
>      }
>  
> @@ -980,7 +989,7 @@ static struct kvm_cpuid2 *get_supported_hv_cpuid(CPUState *cs)
>       * information early, just check for the capability and set the bit
>       * manually.
>       */
> -    if (kvm_check_extension(cs->kvm_state,
> +    if (!do_sys_ioctl && kvm_check_extension(cs->kvm_state,
>                              KVM_CAP_HYPERV_ENLIGHTENED_VMCS) > 0) {

Oh, this conditional replaces the comment I suggested in patch
10/19.  It makes it obvious that the hack can be deleted if we
remove support for the VCPU ioctl.

So, when exactly will we be able to delete the VCPU ioctl code
and support only the system ioctl?

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

>          for (i = 0; i < cpuid->nent; i++) {
>              if (cpuid->entries[i].function == HV_CPUID_ENLIGHTMENT_INFO) {
> -- 
> 2.30.2
> 

-- 
Eduardo

Re: [PATCH v6 14/19] i386: use global kvm_state in hyperv_enabled() check

[Eduardo Habkost]

On Thu, Apr 22, 2021 at 06:11:25PM +0200, Vitaly Kuznetsov wrote:
> There is no need to use vCPU-specific kvm state in hyperv_enabled() check
> and we need to do that when feature expansion happens early, before vCPU
> specific KVM state is created.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

-- 
Eduardo

Re: [PATCH v6 15/19] i386: expand Hyper-V features during CPU feature expansion time

[Eduardo Habkost]

On Thu, Apr 22, 2021 at 06:11:26PM +0200, Vitaly Kuznetsov wrote:
> To make Hyper-V features appear in e.g. QMP query-cpu-model-expansion we
> need to expand and set the corresponding CPUID leaves early. Modify
> x86_cpu_get_supported_feature_word() to call newly intoduced Hyper-V
> specific kvm_hv_get_supported_cpuid() instead of
> kvm_arch_get_supported_cpuid(). We can't use kvm_arch_get_supported_cpuid()
> as Hyper-V specific CPUID leaves intersect with KVM's.
> 
> Note, early expansion will only happen when KVM supports system wide
> KVM_GET_SUPPORTED_HV_CPUID ioctl (KVM_CAP_SYS_HYPERV_CPUID).
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
>  target/i386/cpu.c          |  4 ++++
>  target/i386/kvm/kvm-stub.c |  5 +++++
>  target/i386/kvm/kvm.c      | 15 ++++++++++++---
>  target/i386/kvm/kvm_i386.h |  1 +
>  4 files changed, 22 insertions(+), 3 deletions(-)
> 
> diff --git a/target/i386/cpu.c b/target/i386/cpu.c
> index d72b8760e7a3..18b57f3d8b9c 100644
> --- a/target/i386/cpu.c
> +++ b/target/i386/cpu.c
> @@ -6514,6 +6514,10 @@ static void x86_cpu_expand_features(X86CPU *cpu, Error **errp)
>      if (env->cpuid_xlevel2 == UINT32_MAX) {
>          env->cpuid_xlevel2 = env->cpuid_min_xlevel2;
>      }
> +
> +    if (kvm_enabled()) {
> +        kvm_hyperv_expand_features(cpu, errp);
> +    }
>  }
>  
>  /*
> diff --git a/target/i386/kvm/kvm-stub.c b/target/i386/kvm/kvm-stub.c
> index 92f49121b8fa..7f175faa3abd 100644
> --- a/target/i386/kvm/kvm-stub.c
> +++ b/target/i386/kvm/kvm-stub.c
> @@ -39,3 +39,8 @@ bool kvm_hv_vpindex_settable(void)
>  {
>      return false;
>  }
> +
> +void kvm_hyperv_expand_features(X86CPU *cpu, Error **errp)
> +{
> +    return;

Maybe we should make it abort()?  This function should never be
called if KVM is disabled.

> +}
> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
> index a42263b24fca..d5551c4ab5cf 100644
> --- a/target/i386/kvm/kvm.c
> +++ b/target/i386/kvm/kvm.c
> @@ -1216,13 +1216,22 @@ static uint32_t hv_build_cpuid_leaf(CPUState *cs, uint32_t func, int reg)
>   * of 'hv_passthrough' mode and fills the environment with all supported
>   * Hyper-V features.
>   */
> -static void hyperv_expand_features(CPUState *cs, Error **errp)
> +void kvm_hyperv_expand_features(X86CPU *cpu, Error **errp)
>  {
> -    X86CPU *cpu = X86_CPU(cs);
> +    CPUState *cs = CPU(cpu);
>  
>      if (!hyperv_enabled(cpu))
>          return;
>  
> +    /*
> +     * When kvm_hyperv_expand_features is called at CPU feature expansion
> +     * time per-CPU kvm_state is not available yet so we can only proceed
> +     * when KVM_CAP_SYS_HYPERV_CPUID is supported.
> +     */
> +    if (!cs->kvm_state &&
> +        !kvm_check_extension(kvm_state, KVM_CAP_SYS_HYPERV_CPUID))
> +        return;
> +
>      if (cpu->hyperv_passthrough) {
>          cpu->hyperv_vendor_id[0] =
>              hv_cpuid_get_host(cs, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, R_EBX);
> @@ -1556,7 +1565,7 @@ int kvm_arch_init_vcpu(CPUState *cs)
>      env->apic_bus_freq = KVM_APIC_BUS_FREQUENCY;
>  
>      /* Paravirtualization CPUIDs */
> -    hyperv_expand_features(cs, &local_err);
> +    kvm_hyperv_expand_features(cpu, &local_err);

Do we still need to call the function again here?

If the first expansion isn't expanding everything, I'm afraid
this second call will hide bugs in query-cpu-model-expansion.


>      if (local_err) {
>          error_report_err(local_err);
>          return -ENOSYS;
> diff --git a/target/i386/kvm/kvm_i386.h b/target/i386/kvm/kvm_i386.h
> index dc725083891c..f1176491051d 100644
> --- a/target/i386/kvm/kvm_i386.h
> +++ b/target/i386/kvm/kvm_i386.h
> @@ -47,6 +47,7 @@ bool kvm_has_x2apic_api(void);
>  bool kvm_has_waitpkg(void);
>  
>  bool kvm_hv_vpindex_settable(void);
> +void kvm_hyperv_expand_features(X86CPU *cpu, Error **errp);
>  
>  uint64_t kvm_swizzle_msi_ext_dest_id(uint64_t address);
>  
> -- 
> 2.30.2
> 

-- 
Eduardo

Re: [PATCH v6 16/19] i386: kill off hv_cpuid_check_and_set()

[Eduardo Habkost]

On Thu, Apr 22, 2021 at 06:11:27PM +0200, Vitaly Kuznetsov wrote:
> hv_cpuid_check_and_set() does too much:
> - Checks if the feature is supported by KVM;
> - Checks if all dependencies are enabled;
> - Sets the feature bit in cpu->hyperv_features for 'passthrough' mode.
> 
> To reduce the complexity, move all the logic except for dependencies
> check out of it. Also, in 'passthrough' mode we don't really need to
> check dependencies because KVM is supposed to provide a consistent
> set anyway.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> ---
>  target/i386/kvm/kvm.c | 105 +++++++++++++++---------------------------
>  1 file changed, 36 insertions(+), 69 deletions(-)
> 
> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
> index d5551c4ab5cf..2c1a77f9b00f 100644
> --- a/target/i386/kvm/kvm.c
> +++ b/target/i386/kvm/kvm.c
> @@ -1144,16 +1144,12 @@ static bool hyperv_feature_supported(CPUState *cs, int feature)
>      return true;
>  }
>  
> -static int hv_cpuid_check_and_set(CPUState *cs, int feature, Error **errp)
> +/* Checks that all feature dependencies are enabled */
> +static void hv_feature_check_deps(X86CPU *cpu, int feature, Error **errp)

Same suggestion as in patch 11/19: also returning a value to
indicate error is preferred.  I would return a boolean.

(I don't think this alone should block the series, though)


>  {
> -    X86CPU *cpu = X86_CPU(cs);
>      uint64_t deps;
>      int dep_feat;
>  
> -    if (!hyperv_feat_enabled(cpu, feature) && !cpu->hyperv_passthrough) {
> -        return 0;
> -    }
> -
>      deps = kvm_hyperv_properties[feature].dependencies;
>      while (deps) {
>          dep_feat = ctz64(deps);
> @@ -1161,26 +1157,10 @@ static int hv_cpuid_check_and_set(CPUState *cs, int feature, Error **errp)
>              error_setg(errp, "Hyper-V %s requires Hyper-V %s",
>                         kvm_hyperv_properties[feature].desc,
>                         kvm_hyperv_properties[dep_feat].desc);
> -            return 1;
> +            return;
>          }
>          deps &= ~(1ull << dep_feat);
>      }
> -
> -    if (!hyperv_feature_supported(cs, feature)) {
> -        if (hyperv_feat_enabled(cpu, feature)) {
> -            error_setg(errp, "Hyper-V %s is not supported by kernel",
> -                       kvm_hyperv_properties[feature].desc);
> -            return 1;
> -        } else {
> -            return 0;
> -        }
> -    }
> -
> -    if (cpu->hyperv_passthrough) {
> -        cpu->hyperv_features |= BIT(feature);
> -    }
> -
> -    return 0;
>  }
>  
>  static uint32_t hv_build_cpuid_leaf(CPUState *cs, uint32_t func, int reg)
> @@ -1219,6 +1199,8 @@ static uint32_t hv_build_cpuid_leaf(CPUState *cs, uint32_t func, int reg)
>  void kvm_hyperv_expand_features(X86CPU *cpu, Error **errp)
>  {
>      CPUState *cs = CPU(cpu);
> +    Error *local_err = NULL;
> +    int feat;
>  
>      if (!hyperv_enabled(cpu))
>          return;
> @@ -1274,53 +1256,38 @@ void kvm_hyperv_expand_features(X86CPU *cpu, Error **errp)
>  
>          cpu->hyperv_spinlock_attempts =
>              hv_cpuid_get_host(cs, HV_CPUID_ENLIGHTMENT_INFO, R_EBX);
> -    }
>  
> -    /* Features */
> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_RELAXED, errp)) {
> -        return;
> -    }
> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_VAPIC, errp)) {
> -        return;
> -    }
> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_TIME, errp)) {
> -        return;
> -    }
> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_CRASH, errp)) {
> -        return;
> -    }
> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_RESET, errp)) {
> -        return;
> -    }
> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_VPINDEX, errp)) {
> -        return;
> -    }
> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_RUNTIME, errp)) {
> -        return;
> -    }
> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_SYNIC, errp)) {
> -        return;
> -    }
> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_STIMER, errp)) {
> -        return;
> -    }
> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_FREQUENCIES, errp)) {
> -        return;
> -    }
> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_REENLIGHTENMENT, errp)) {
> -        return;
> -    }
> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_TLBFLUSH, errp)) {
> -        return;
> -    }
> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_EVMCS, errp)) {
> -        return;
> -    }
> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_IPI, errp)) {
> -        return;
> -    }
> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_STIMER_DIRECT, errp)) {
> -        return;
> +        /*
> +         * Mark feature as enabled in 'cpu->hyperv_features' as
> +         * hv_build_cpuid_leaf() uses this info to build guest CPUIDs.
> +         */
> +        for (feat = 0; feat < ARRAY_SIZE(kvm_hyperv_properties); feat++) {
> +            if (hyperv_feature_supported(cs, feat)) {
> +                cpu->hyperv_features |= BIT(feat);
> +            }
> +        }
> +    } else {
> +        /* Check features availability and dependencies */
> +        for (feat = 0; feat < ARRAY_SIZE(kvm_hyperv_properties); feat++) {
> +            /* If the feature was not requested skip it. */
> +            if (!hyperv_feat_enabled(cpu, feat)) {
> +                continue;
> +            }

That's the loop I suggested in patch 11/19.  Nice.  :)

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

> +
> +            /* Check if the feature is supported by KVM */
> +            if (!hyperv_feature_supported(cs, feat)) {
> +                error_setg(errp, "Hyper-V %s is not supported by kernel",
> +                           kvm_hyperv_properties[feat].desc);
> +                return;
> +            }
> +
> +            /* Check dependencies */
> +            hv_feature_check_deps(cpu, feat, &local_err);
> +            if (local_err != NULL) {
> +                error_propagate(errp, local_err);
> +                return;
> +            }
> +        }
>      }
>  
>      /* Additional dependencies not covered by kvm_hyperv_properties[] */
> -- 
> 2.30.2
> 

-- 
Eduardo

Re: [PATCH v6 17/19] i386: HV_HYPERCALL_AVAILABLE privilege bit is always needed

[Eduardo Habkost]

On Thu, Apr 22, 2021 at 06:11:28PM +0200, Vitaly Kuznetsov wrote:
> According to TLFS, Hyper-V guest is supposed to check
> HV_HYPERCALL_AVAILABLE privilege bit before accessing
> HV_X64_MSR_GUEST_OS_ID/HV_X64_MSR_HYPERCALL MSRs but at least some
> Windows versions ignore that. As KVM is very permissive and allows
> accessing these MSRs unconditionally, no issue is observed. We may,
> however, want to tighten the checks eventually. Conforming to the
> spec is probably also a good idea.
> 
> Add HV_HYPERCALL_AVAILABLE to all 'leaf' features with no dependencies.
> 
> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>

Are all VMs being created with HV_HYPERCALL_AVAILABLE unset,
today?

Wouldn't it be simpler to simply add a new
HYPERV_FEAT_HYPERCALL_AVAILABLE bit to hyperv_features, and
enabling it by default?

We don't necessarily need to make it configurable by the user,
but probably it would be a good idea to keep the bit unset by
default on older machine types.  Even if guests don't mind seeing
the bit changing under their feet, it would make it easier for
automated test cases that check for unexpected changes in raw
CPUID data.


> ---
>  target/i386/kvm/kvm.c | 15 +++++++++------
>  1 file changed, 9 insertions(+), 6 deletions(-)
> 
> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
> index 2c1a77f9b00f..d81451276cd8 100644
> --- a/target/i386/kvm/kvm.c
> +++ b/target/i386/kvm/kvm.c
> @@ -835,6 +835,8 @@ static struct {
>      [HYPERV_FEAT_CRASH] = {
>          .desc = "crash MSRs (hv-crash)",
>          .flags = {
> +            {.func = HV_CPUID_FEATURES, .reg = R_EAX,
> +             .bits = HV_HYPERCALL_AVAILABLE},
>              {.func = HV_CPUID_FEATURES, .reg = R_EDX,
>               .bits = HV_GUEST_CRASH_MSR_AVAILABLE}
>          }
> @@ -843,28 +845,28 @@ static struct {
>          .desc = "reset MSR (hv-reset)",
>          .flags = {
>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
> -             .bits = HV_RESET_AVAILABLE}
> +             .bits = HV_HYPERCALL_AVAILABLE | HV_RESET_AVAILABLE}
>          }
>      },
>      [HYPERV_FEAT_VPINDEX] = {
>          .desc = "VP_INDEX MSR (hv-vpindex)",
>          .flags = {
>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
> -             .bits = HV_VP_INDEX_AVAILABLE}
> +             .bits = HV_HYPERCALL_AVAILABLE | HV_VP_INDEX_AVAILABLE}
>          }
>      },
>      [HYPERV_FEAT_RUNTIME] = {
>          .desc = "VP_RUNTIME MSR (hv-runtime)",
>          .flags = {
>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
> -             .bits = HV_VP_RUNTIME_AVAILABLE}
> +             .bits = HV_HYPERCALL_AVAILABLE | HV_VP_RUNTIME_AVAILABLE}
>          }
>      },
>      [HYPERV_FEAT_SYNIC] = {
>          .desc = "synthetic interrupt controller (hv-synic)",
>          .flags = {
>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
> -             .bits = HV_SYNIC_AVAILABLE}
> +             .bits = HV_HYPERCALL_AVAILABLE | HV_SYNIC_AVAILABLE}
>          }
>      },
>      [HYPERV_FEAT_STIMER] = {
> @@ -879,7 +881,7 @@ static struct {
>          .desc = "frequency MSRs (hv-frequencies)",
>          .flags = {
>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
> -             .bits = HV_ACCESS_FREQUENCY_MSRS},
> +             .bits = HV_HYPERCALL_AVAILABLE | HV_ACCESS_FREQUENCY_MSRS},
>              {.func = HV_CPUID_FEATURES, .reg = R_EDX,
>               .bits = HV_FREQUENCY_MSRS_AVAILABLE}
>          }
> @@ -888,7 +890,8 @@ static struct {
>          .desc = "reenlightenment MSRs (hv-reenlightenment)",
>          .flags = {
>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
> -             .bits = HV_ACCESS_REENLIGHTENMENTS_CONTROL}
> +             .bits = HV_HYPERCALL_AVAILABLE |
> +             HV_ACCESS_REENLIGHTENMENTS_CONTROL}
>          }
>      },
>      [HYPERV_FEAT_TLBFLUSH] = {
> -- 
> 2.30.2
> 

-- 
Eduardo

Re: [PATCH v6 10/19] i386: move eVMCS enablement to hyperv_init_vcpu()

[Vitaly Kuznetsov]

Eduardo Habkost <ehabkost@redhat.com> writes:

> On Thu, Apr 22, 2021 at 06:11:21PM +0200, Vitaly Kuznetsov wrote:
>> hyperv_expand_features() will be called before we create vCPU so
>> evmcs enablement should go away. hyperv_init_vcpu() looks like the
>> right place.
>> 
>> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
>> ---
>>  target/i386/kvm/kvm.c | 60 ++++++++++++++++++++++++++-----------------
>>  1 file changed, 37 insertions(+), 23 deletions(-)
>> 
>> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
>> index 6b391db7a030..a2ef2dc154a2 100644
>> --- a/target/i386/kvm/kvm.c
>> +++ b/target/i386/kvm/kvm.c
>> @@ -962,6 +962,7 @@ static struct kvm_cpuid2 *get_supported_hv_cpuid(CPUState *cs)
>>  {
>>      struct kvm_cpuid2 *cpuid;
>>      int max = 7; /* 0x40000000..0x40000005, 0x4000000A */
>> +    int i;
>>  
>>      /*
>>       * When the buffer is too small, KVM_GET_SUPPORTED_HV_CPUID fails with
>> @@ -971,6 +972,22 @@ static struct kvm_cpuid2 *get_supported_hv_cpuid(CPUState *cs)
>>      while ((cpuid = try_get_hv_cpuid(cs, max)) == NULL) {
>>          max++;
>>      }
>> +
>> +    /*
>> +     * KVM_GET_SUPPORTED_HV_CPUID does not set EVMCS CPUID bit before
>> +     * KVM_CAP_HYPERV_ENLIGHTENED_VMCS is enabled but we want to get the
>> +     * information early, just check for the capability and set the bit
>> +     * manually.
>> +     */
>
> Should we add a comment noting that this hack won't be necessary
> if using the system ioctl?  I assume we still want to support
> Linux < v5.11 for a while, so simply 

Not exactly sure what was supposed to be here but yes, the hack is not
needed with system KVM_GET_SUPPORTED_HV_CPUID ioctl but we want to
support older kernels.

>
>
>> +    if (kvm_check_extension(cs->kvm_state,
>> +                            KVM_CAP_HYPERV_ENLIGHTENED_VMCS) > 0) {
>> +        for (i = 0; i < cpuid->nent; i++) {
>> +            if (cpuid->entries[i].function == HV_CPUID_ENLIGHTMENT_INFO) {
>> +                cpuid->entries[i].eax |= HV_ENLIGHTENED_VMCS_RECOMMENDED;
>> +            }
>> +        }
>> +    }
>> +
>>      return cpuid;
>>  }
>>  
>> @@ -1200,24 +1217,6 @@ static int hyperv_expand_features(CPUState *cs)
>>      if (!hyperv_enabled(cpu))
>>          return 0;
>>  
>> -    if (hyperv_feat_enabled(cpu, HYPERV_FEAT_EVMCS) ||
>> -        cpu->hyperv_passthrough) {
>> -        uint16_t evmcs_version;
>> -
>> -        r = kvm_vcpu_enable_cap(cs, KVM_CAP_HYPERV_ENLIGHTENED_VMCS, 0,
>> -                                (uintptr_t)&evmcs_version);
>> -
>> -        if (hyperv_feat_enabled(cpu, HYPERV_FEAT_EVMCS) && r) {
>> -            fprintf(stderr, "Hyper-V %s is not supported by kernel\n",
>> -                    kvm_hyperv_properties[HYPERV_FEAT_EVMCS].desc);
>> -            return -ENOSYS;
>> -        }
>> -
>> -        if (!r) {
>> -            cpu->hyperv_nested[0] = evmcs_version;
>> -        }
>> -    }
>> -
>>      if (cpu->hyperv_passthrough) {
>>          cpu->hyperv_vendor_id[0] =
>>              hv_cpuid_get_host(cs, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, R_EBX);
>> @@ -1455,6 +1454,21 @@ static int hyperv_init_vcpu(X86CPU *cpu)
>>          }
>>      }
>>  
>> +    if (hyperv_feat_enabled(cpu, HYPERV_FEAT_EVMCS)) {
>> +        uint16_t evmcs_version;
>> +
>> +        ret = kvm_vcpu_enable_cap(cs, KVM_CAP_HYPERV_ENLIGHTENED_VMCS, 0,
>> +                                  (uintptr_t)&evmcs_version);
>> +
>> +        if (ret < 0) {
>> +            fprintf(stderr, "Hyper-V %s is not supported by kernel\n",
>> +                    kvm_hyperv_properties[HYPERV_FEAT_EVMCS].desc);
>> +            return ret;
>> +        }
>> +
>> +        cpu->hyperv_nested[0] = evmcs_version;
>
> Wait, won't this break guest ABI?  Do we want to make
> HYPERV_FEAT_EVMCS a migration blocker until this is fixed?
>

Could you please elaborate on the issue? I read the above is: when 
evmcs' feature was requested, make an attempt to enable
KVM_CAP_HYPERV_ENLIGHTENED_VMCS, and bail out if this fails. Propagate
the the acquired evmcs version to 'cpu->hyperv_nested[]' otherwise.

>
>> +    }
>> +
>>      return 0;
>>  }
>>  
>> @@ -1519,6 +1533,11 @@ int kvm_arch_init_vcpu(CPUState *cs)
>>      }
>>  
>>      if (hyperv_enabled(cpu)) {
>> +        r = hyperv_init_vcpu(cpu);
>> +        if (r) {
>> +            return r;
>> +        }
>> +
>>          cpuid_i = hyperv_fill_cpuids(cs, cpuid_data.entries);
>>          kvm_base = KVM_CPUID_SIGNATURE_NEXT;
>>          has_msr_hv_hypercall = true;
>> @@ -1868,11 +1887,6 @@ int kvm_arch_init_vcpu(CPUState *cs)
>>  
>>      kvm_init_msrs(cpu);
>>  
>> -    r = hyperv_init_vcpu(cpu);
>> -    if (r) {
>> -        goto fail;
>> -    }
>> -
>>      return 0;
>
> I would move the two hunks above to a separate patch, but not a
> big deal.  The guest ABI issue is existing, and the comment
> suggestion can be done in a follow up patch, so:
>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>

Thanks!

>>  
>>   fail:
>> -- 
>> 2.30.2
>> 

-- 
Vitaly

Re: [PATCH v6 11/19] i386: switch hyperv_expand_features() to using error_setg()

[Vitaly Kuznetsov]

Eduardo Habkost <ehabkost@redhat.com> writes:

> On Thu, Apr 22, 2021 at 06:11:22PM +0200, Vitaly Kuznetsov wrote:
>> Use standard error_setg() mechanism in hyperv_expand_features().
>> 
>> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
>
> No objections, but only suggestions below:
>
>> ---
>>  target/i386/kvm/kvm.c | 101 +++++++++++++++++++++++++-----------------
>>  1 file changed, 61 insertions(+), 40 deletions(-)
>> 
>> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
>> index a2ef2dc154a2..f33ba325187f 100644
>> --- a/target/i386/kvm/kvm.c
>> +++ b/target/i386/kvm/kvm.c
>> @@ -1135,7 +1135,7 @@ static bool hyperv_feature_supported(CPUState *cs, int feature)
>>      return true;
>>  }
>>  
>> -static int hv_cpuid_check_and_set(CPUState *cs, int feature)
>> +static int hv_cpuid_check_and_set(CPUState *cs, int feature, Error **errp)
>
> If changing the function signature, and the function only returns 0 or 1, maybe
> it's a good opportunity to change to a bool return value format?
>
> From include/qapi/error.h:
>
>  * - Whenever practical, also return a value that indicates success /
>  *   failure.  This can make the error checking more concise, and can
>  *   avoid useless error object creation and destruction.  Note that
>  *   we still have many functions returning void.  We recommend
>  *   • bool-valued functions return true on success / false on failure,
>  *   • pointer-valued functions return non-null / null pointer, and
>  *   • integer-valued functions return non-negative / negative.
>
>
>>  {
>>      X86CPU *cpu = X86_CPU(cs);
>>      uint64_t deps;
>> @@ -1149,20 +1149,18 @@ static int hv_cpuid_check_and_set(CPUState *cs, int feature)
>>      while (deps) {
>>          dep_feat = ctz64(deps);
>>          if (!(hyperv_feat_enabled(cpu, dep_feat))) {
>> -                fprintf(stderr,
>> -                        "Hyper-V %s requires Hyper-V %s\n",
>> -                        kvm_hyperv_properties[feature].desc,
>> -                        kvm_hyperv_properties[dep_feat].desc);
>> -                return 1;
>> +            error_setg(errp, "Hyper-V %s requires Hyper-V %s",
>> +                       kvm_hyperv_properties[feature].desc,
>> +                       kvm_hyperv_properties[dep_feat].desc);
>> +            return 1;
>>          }
>>          deps &= ~(1ull << dep_feat);
>>      }
>>  
>>      if (!hyperv_feature_supported(cs, feature)) {
>>          if (hyperv_feat_enabled(cpu, feature)) {
>> -            fprintf(stderr,
>> -                    "Hyper-V %s is not supported by kernel\n",
>> -                    kvm_hyperv_properties[feature].desc);
>> +            error_setg(errp, "Hyper-V %s is not supported by kernel",
>> +                       kvm_hyperv_properties[feature].desc);
>>              return 1;
>>          } else {
>>              return 0;
>> @@ -1209,13 +1207,12 @@ static uint32_t hv_build_cpuid_leaf(CPUState *cs, uint32_t func, int reg)
>>   * of 'hv_passthrough' mode and fills the environment with all supported
>>   * Hyper-V features.
>>   */
>> -static int hyperv_expand_features(CPUState *cs)
>> +static void hyperv_expand_features(CPUState *cs, Error **errp)
>
> Same as above: returning a value to indicate error is preferred.  If you are no
> longer returning an integer error code, I suggest returning bool instead.
>

hv_cpuid_check_and_set() is eliminated later in the series but
hyperv_expand_features() stays, I can make it bool.


>>  {
>>      X86CPU *cpu = X86_CPU(cs);
>> -    int r;
>>  
>>      if (!hyperv_enabled(cpu))
>> -        return 0;
>> +        return;
>>  
>>      if (cpu->hyperv_passthrough) {
>>          cpu->hyperv_vendor_id[0] =
>> @@ -1262,37 +1259,60 @@ static int hyperv_expand_features(CPUState *cs)
>>      }
>>  
>>      /* Features */
>> -    r = hv_cpuid_check_and_set(cs, HYPERV_FEAT_RELAXED);
>> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_VAPIC);
>> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_TIME);
>> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_CRASH);
>> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_RESET);
>> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_VPINDEX);
>> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_RUNTIME);
>> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_SYNIC);
>> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_STIMER);
>> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_FREQUENCIES);
>> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_REENLIGHTENMENT);
>> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_TLBFLUSH);
>> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_EVMCS);
>> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_IPI);
>> -    r |= hv_cpuid_check_and_set(cs, HYPERV_FEAT_STIMER_DIRECT);
>> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_RELAXED, errp)) {
>> +        return;
>> +    }
>
> What about a loop?
>
>     for (feat = 0; feat < ARRAY_SIZE(kvm_hyperv_properties); feat++) {
>         if (hv_cpuid_check_and_set(cs, feat, errp)) {
>             return;
>         }
>     }
>

This is done later in the series ("i386: kill off hv_cpuid_check_and_set()").

>> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_VAPIC, errp)) {
>> +        return;
>> +    }
>> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_TIME, errp)) {
>> +        return;
>> +    }
>> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_CRASH, errp)) {
>> +        return;
>> +    }
>> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_RESET, errp)) {
>> +        return;
>> +    }
>> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_VPINDEX, errp)) {
>> +        return;
>> +    }
>> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_RUNTIME, errp)) {
>> +        return;
>> +    }
>> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_SYNIC, errp)) {
>> +        return;
>> +    }
>> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_STIMER, errp)) {
>> +        return;
>> +    }
>> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_FREQUENCIES, errp)) {
>> +        return;
>> +    }
>> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_REENLIGHTENMENT, errp)) {
>> +        return;
>> +    }
>> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_TLBFLUSH, errp)) {
>> +        return;
>> +    }
>> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_EVMCS, errp)) {
>> +        return;
>> +    }
>> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_IPI, errp)) {
>> +        return;
>> +    }
>> +    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_STIMER_DIRECT, errp)) {
>> +        return;
>> +    }
>>  
>>      /* Additional dependencies not covered by kvm_hyperv_properties[] */
>>      if (hyperv_feat_enabled(cpu, HYPERV_FEAT_SYNIC) &&
>>          !cpu->hyperv_synic_kvm_only &&
>>          !hyperv_feat_enabled(cpu, HYPERV_FEAT_VPINDEX)) {
>> -        fprintf(stderr, "Hyper-V %s requires Hyper-V %s\n",
>> -                kvm_hyperv_properties[HYPERV_FEAT_SYNIC].desc,
>> -                kvm_hyperv_properties[HYPERV_FEAT_VPINDEX].desc);
>> -        r |= 1;
>> -    }
>> -
>> -    if (r) {
>> -        return -ENOSYS;
>> +        error_setg(errp, "Hyper-V %s requires Hyper-V %s",
>> +                   kvm_hyperv_properties[HYPERV_FEAT_SYNIC].desc,
>> +                   kvm_hyperv_properties[HYPERV_FEAT_VPINDEX].desc);
>>      }
>> -
>> -    return 0;
>>  }
>>  
>>  /*
>> @@ -1527,9 +1547,10 @@ int kvm_arch_init_vcpu(CPUState *cs)
>>      env->apic_bus_freq = KVM_APIC_BUS_FREQUENCY;
>>  
>>      /* Paravirtualization CPUIDs */
>> -    r = hyperv_expand_features(cs);
>> -    if (r < 0) {
>> -        return r;
>> +    hyperv_expand_features(cs, &local_err);
>> +    if (local_err) {
>> +        error_report_err(local_err);
>> +        return -ENOSYS;
>>      }
>>  
>>      if (hyperv_enabled(cpu)) {
>
> I don't want to block this series because of the suggestions above, so:
>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>
> But I still encourage you to implement those suggestions, anyway.

'Loop' idea is already implemented and hv_cpuid_check_and_set() is gone
but I'll remember to make hyperv_expand_features() bool. Thanks!

-- 
Vitaly

Re: [PATCH v6 13/19] i386: prefer system KVM_GET_SUPPORTED_HV_CPUID ioctl over vCPU's one

[Vitaly Kuznetsov]

Eduardo Habkost <ehabkost@redhat.com> writes:

> On Thu, Apr 22, 2021 at 06:11:24PM +0200, Vitaly Kuznetsov wrote:
>> KVM_GET_SUPPORTED_HV_CPUID was made a system wide ioctl which can be called
>> prior to creating vCPUs and we are going to use that to expand Hyper-V cpu
>> features early. Use it when it is supported by KVM.
>> 
>> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
>> ---
>>  target/i386/kvm/kvm.c | 17 +++++++++++++----
>>  1 file changed, 13 insertions(+), 4 deletions(-)
>> 
>> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
>> index feec9f25ba12..5d08f3a39ef7 100644
>> --- a/target/i386/kvm/kvm.c
>> +++ b/target/i386/kvm/kvm.c
>> @@ -928,7 +928,8 @@ static struct {
>>      },
>>  };
>>  
>> -static struct kvm_cpuid2 *try_get_hv_cpuid(CPUState *cs, int max)
>> +static struct kvm_cpuid2 *try_get_hv_cpuid(CPUState *cs, int max,
>> +                                           bool do_sys_ioctl)
>>  {
>>      struct kvm_cpuid2 *cpuid;
>>      int r, size;
>> @@ -937,7 +938,11 @@ static struct kvm_cpuid2 *try_get_hv_cpuid(CPUState *cs, int max)
>>      cpuid = g_malloc0(size);
>>      cpuid->nent = max;
>>  
>> -    r = kvm_vcpu_ioctl(cs, KVM_GET_SUPPORTED_HV_CPUID, cpuid);
>> +    if (do_sys_ioctl) {
>> +        r = kvm_ioctl(kvm_state, KVM_GET_SUPPORTED_HV_CPUID, cpuid);
>> +    } else {
>> +        r = kvm_vcpu_ioctl(cs, KVM_GET_SUPPORTED_HV_CPUID, cpuid);
>> +    }
>>      if (r == 0 && cpuid->nent >= max) {
>>          r = -E2BIG;
>>      }
>> @@ -964,13 +969,17 @@ static struct kvm_cpuid2 *get_supported_hv_cpuid(CPUState *cs)
>>      /* 0x40000000..0x40000005, 0x4000000A, 0x40000080..0x40000080 leaves */
>>      int max = 10;
>>      int i;
>> +    bool do_sys_ioctl;
>> +
>> +    do_sys_ioctl =
>> +        kvm_check_extension(kvm_state, KVM_CAP_SYS_HYPERV_CPUID) > 0;
>>  
>>      /*
>>       * When the buffer is too small, KVM_GET_SUPPORTED_HV_CPUID fails with
>>       * -E2BIG, however, it doesn't report back the right size. Keep increasing
>>       * it and re-trying until we succeed.
>>       */
>> -    while ((cpuid = try_get_hv_cpuid(cs, max)) == NULL) {
>> +    while ((cpuid = try_get_hv_cpuid(cs, max, do_sys_ioctl)) == NULL) {
>>          max++;
>>      }
>>  
>> @@ -980,7 +989,7 @@ static struct kvm_cpuid2 *get_supported_hv_cpuid(CPUState *cs)
>>       * information early, just check for the capability and set the bit
>>       * manually.
>>       */
>> -    if (kvm_check_extension(cs->kvm_state,
>> +    if (!do_sys_ioctl && kvm_check_extension(cs->kvm_state,
>>                              KVM_CAP_HYPERV_ENLIGHTENED_VMCS) > 0) {
>
> Oh, this conditional replaces the comment I suggested in patch
> 10/19.  It makes it obvious that the hack can be deleted if we
> remove support for the VCPU ioctl.
>
> So, when exactly will we be able to delete the VCPU ioctl code
> and support only the system ioctl?

When QEMU drops support for kernels < 5.11? Note, current RHEL8 already
supports system version so we're talking about upstream kernels/Ubuntu
LTS/... 

I remember there was a list of supported kernels for QEMU somewhere but
don't seem to be able to find it quickly, could you maybe point me in
the right direction?

>
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
>

Thanks!

>>          for (i = 0; i < cpuid->nent; i++) {
>>              if (cpuid->entries[i].function == HV_CPUID_ENLIGHTMENT_INFO) {
>> -- 
>> 2.30.2
>> 

-- 
Vitaly

Re: [PATCH v6 15/19] i386: expand Hyper-V features during CPU feature expansion time

[Vitaly Kuznetsov]

Eduardo Habkost <ehabkost@redhat.com> writes:

> On Thu, Apr 22, 2021 at 06:11:26PM +0200, Vitaly Kuznetsov wrote:
>> To make Hyper-V features appear in e.g. QMP query-cpu-model-expansion we
>> need to expand and set the corresponding CPUID leaves early. Modify
>> x86_cpu_get_supported_feature_word() to call newly intoduced Hyper-V
>> specific kvm_hv_get_supported_cpuid() instead of
>> kvm_arch_get_supported_cpuid(). We can't use kvm_arch_get_supported_cpuid()
>> as Hyper-V specific CPUID leaves intersect with KVM's.
>> 
>> Note, early expansion will only happen when KVM supports system wide
>> KVM_GET_SUPPORTED_HV_CPUID ioctl (KVM_CAP_SYS_HYPERV_CPUID).
>> 
>> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
>> ---
>>  target/i386/cpu.c          |  4 ++++
>>  target/i386/kvm/kvm-stub.c |  5 +++++
>>  target/i386/kvm/kvm.c      | 15 ++++++++++++---
>>  target/i386/kvm/kvm_i386.h |  1 +
>>  4 files changed, 22 insertions(+), 3 deletions(-)
>> 
>> diff --git a/target/i386/cpu.c b/target/i386/cpu.c
>> index d72b8760e7a3..18b57f3d8b9c 100644
>> --- a/target/i386/cpu.c
>> +++ b/target/i386/cpu.c
>> @@ -6514,6 +6514,10 @@ static void x86_cpu_expand_features(X86CPU *cpu, Error **errp)
>>      if (env->cpuid_xlevel2 == UINT32_MAX) {
>>          env->cpuid_xlevel2 = env->cpuid_min_xlevel2;
>>      }
>> +
>> +    if (kvm_enabled()) {
>> +        kvm_hyperv_expand_features(cpu, errp);
>> +    }
>>  }
>>  
>>  /*
>> diff --git a/target/i386/kvm/kvm-stub.c b/target/i386/kvm/kvm-stub.c
>> index 92f49121b8fa..7f175faa3abd 100644
>> --- a/target/i386/kvm/kvm-stub.c
>> +++ b/target/i386/kvm/kvm-stub.c
>> @@ -39,3 +39,8 @@ bool kvm_hv_vpindex_settable(void)
>>  {
>>      return false;
>>  }
>> +
>> +void kvm_hyperv_expand_features(X86CPU *cpu, Error **errp)
>> +{
>> +    return;
>
> Maybe we should make it abort()?  This function should never be
> called if KVM is disabled.
>

Sure, can do that.

>> +}
>> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
>> index a42263b24fca..d5551c4ab5cf 100644
>> --- a/target/i386/kvm/kvm.c
>> +++ b/target/i386/kvm/kvm.c
>> @@ -1216,13 +1216,22 @@ static uint32_t hv_build_cpuid_leaf(CPUState *cs, uint32_t func, int reg)
>>   * of 'hv_passthrough' mode and fills the environment with all supported
>>   * Hyper-V features.
>>   */
>> -static void hyperv_expand_features(CPUState *cs, Error **errp)
>> +void kvm_hyperv_expand_features(X86CPU *cpu, Error **errp)
>>  {
>> -    X86CPU *cpu = X86_CPU(cs);
>> +    CPUState *cs = CPU(cpu);
>>  
>>      if (!hyperv_enabled(cpu))
>>          return;
>>  
>> +    /*
>> +     * When kvm_hyperv_expand_features is called at CPU feature expansion
>> +     * time per-CPU kvm_state is not available yet so we can only proceed
>> +     * when KVM_CAP_SYS_HYPERV_CPUID is supported.
>> +     */
>> +    if (!cs->kvm_state &&
>> +        !kvm_check_extension(kvm_state, KVM_CAP_SYS_HYPERV_CPUID))
>> +        return;
>> +
>>      if (cpu->hyperv_passthrough) {
>>          cpu->hyperv_vendor_id[0] =
>>              hv_cpuid_get_host(cs, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, R_EBX);
>> @@ -1556,7 +1565,7 @@ int kvm_arch_init_vcpu(CPUState *cs)
>>      env->apic_bus_freq = KVM_APIC_BUS_FREQUENCY;
>>  
>>      /* Paravirtualization CPUIDs */
>> -    hyperv_expand_features(cs, &local_err);
>> +    kvm_hyperv_expand_features(cpu, &local_err);
>
> Do we still need to call the function again here?
>
> If the first expansion isn't expanding everything, I'm afraid
> this second call will hide bugs in query-cpu-model-expansion.
>

The first expansion will do nothing if KVM_CAP_SYS_HYPERV_CPUID is not
supported, calling it here allows us to proceed. The series makes
'query-cpu-model-expansion' output correct only with
KVM_CAP_SYS_HYPERV_CPUID, without it we don't seem to be able to do much
(unless we decide to create a 'scratch' CPU or something like that).

>
>>      if (local_err) {
>>          error_report_err(local_err);
>>          return -ENOSYS;
>> diff --git a/target/i386/kvm/kvm_i386.h b/target/i386/kvm/kvm_i386.h
>> index dc725083891c..f1176491051d 100644
>> --- a/target/i386/kvm/kvm_i386.h
>> +++ b/target/i386/kvm/kvm_i386.h
>> @@ -47,6 +47,7 @@ bool kvm_has_x2apic_api(void);
>>  bool kvm_has_waitpkg(void);
>>  
>>  bool kvm_hv_vpindex_settable(void);
>> +void kvm_hyperv_expand_features(X86CPU *cpu, Error **errp);
>>  
>>  uint64_t kvm_swizzle_msi_ext_dest_id(uint64_t address);
>>  
>> -- 
>> 2.30.2
>> 

-- 
Vitaly

Re: [PATCH v6 16/19] i386: kill off hv_cpuid_check_and_set()

[Vitaly Kuznetsov]

Eduardo Habkost <ehabkost@redhat.com> writes:

> On Thu, Apr 22, 2021 at 06:11:27PM +0200, Vitaly Kuznetsov wrote:
>> hv_cpuid_check_and_set() does too much:
>> - Checks if the feature is supported by KVM;
>> - Checks if all dependencies are enabled;
>> - Sets the feature bit in cpu->hyperv_features for 'passthrough' mode.
>> 
>> To reduce the complexity, move all the logic except for dependencies
>> check out of it. Also, in 'passthrough' mode we don't really need to
>> check dependencies because KVM is supposed to provide a consistent
>> set anyway.
>> 
>> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
>> ---
>>  target/i386/kvm/kvm.c | 105 +++++++++++++++---------------------------
>>  1 file changed, 36 insertions(+), 69 deletions(-)
>> 
>> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
>> index d5551c4ab5cf..2c1a77f9b00f 100644
>> --- a/target/i386/kvm/kvm.c
>> +++ b/target/i386/kvm/kvm.c
>> @@ -1144,16 +1144,12 @@ static bool hyperv_feature_supported(CPUState *cs, int feature)
>>      return true;
>>  }
>>  
>> -static int hv_cpuid_check_and_set(CPUState *cs, int feature, Error **errp)
>> +/* Checks that all feature dependencies are enabled */
>> +static void hv_feature_check_deps(X86CPU *cpu, int feature, Error **errp)
>
> Same suggestion as in patch 11/19: also returning a value to
> indicate error is preferred.  I would return a boolean.
>

Sure, can do.

> (I don't think this alone should block the series, though)
>
>
>>  {
>> -    X86CPU *cpu = X86_CPU(cs);
>>      uint64_t deps;
>>      int dep_feat;
>>  
>> -    if (!hyperv_feat_enabled(cpu, feature) && !cpu->hyperv_passthrough) {
>> -        return 0;
>> -    }
>> -
>>      deps = kvm_hyperv_properties[feature].dependencies;
>>      while (deps) {
>>          dep_feat = ctz64(deps);
>> @@ -1161,26 +1157,10 @@ static int hv_cpuid_check_and_set(CPUState *cs, int feature, Error **errp)
>>              error_setg(errp, "Hyper-V %s requires Hyper-V %s",
>>                         kvm_hyperv_properties[feature].desc,
>>                         kvm_hyperv_properties[dep_feat].desc);
>> -            return 1;
>> +            return;
>>          }
>>          deps &= ~(1ull << dep_feat);
>>      }
>> -
>> -    if (!hyperv_feature_supported(cs, feature)) {
>> -        if (hyperv_feat_enabled(cpu, feature)) {
>> -            error_setg(errp, "Hyper-V %s is not supported by kernel",
>> -                       kvm_hyperv_properties[feature].desc);
>> -            return 1;
>> -        } else {
>> -            return 0;
>> -        }
>> -    }
>> -
>> -    if (cpu->hyperv_passthrough) {
>> -        cpu->hyperv_features |= BIT(feature);
>> -    }
>> -
>> -    return 0;
>>  }
>>  
>>  static uint32_t hv_build_cpuid_leaf(CPUState *cs, uint32_t func, int reg)
>> @@ -1219,6 +1199,8 @@ static uint32_t hv_build_cpuid_leaf(CPUState *cs, uint32_t func, int reg)
>>  void kvm_hyperv_expand_features(X86CPU *cpu, Error **errp)
>>  {
>>      CPUState *cs = CPU(cpu);
>> +    Error *local_err = NULL;
>> +    int feat;
>>  
>>      if (!hyperv_enabled(cpu))
>>          return;
>> @@ -1274,53 +1256,38 @@ void kvm_hyperv_expand_features(X86CPU *cpu, Error **errp)
>>  
>>          cpu->hyperv_spinlock_attempts =
>>              hv_cpuid_get_host(cs, HV_CPUID_ENLIGHTMENT_INFO, R_EBX);
>> -    }
>>  
>> -    /* Features */
>> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_RELAXED, errp)) {
>> -        return;
>> -    }
>> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_VAPIC, errp)) {
>> -        return;
>> -    }
>> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_TIME, errp)) {
>> -        return;
>> -    }
>> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_CRASH, errp)) {
>> -        return;
>> -    }
>> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_RESET, errp)) {
>> -        return;
>> -    }
>> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_VPINDEX, errp)) {
>> -        return;
>> -    }
>> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_RUNTIME, errp)) {
>> -        return;
>> -    }
>> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_SYNIC, errp)) {
>> -        return;
>> -    }
>> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_STIMER, errp)) {
>> -        return;
>> -    }
>> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_FREQUENCIES, errp)) {
>> -        return;
>> -    }
>> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_REENLIGHTENMENT, errp)) {
>> -        return;
>> -    }
>> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_TLBFLUSH, errp)) {
>> -        return;
>> -    }
>> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_EVMCS, errp)) {
>> -        return;
>> -    }
>> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_IPI, errp)) {
>> -        return;
>> -    }
>> -    if (hv_cpuid_check_and_set(cs, HYPERV_FEAT_STIMER_DIRECT, errp)) {
>> -        return;
>> +        /*
>> +         * Mark feature as enabled in 'cpu->hyperv_features' as
>> +         * hv_build_cpuid_leaf() uses this info to build guest CPUIDs.
>> +         */
>> +        for (feat = 0; feat < ARRAY_SIZE(kvm_hyperv_properties); feat++) {
>> +            if (hyperv_feature_supported(cs, feat)) {
>> +                cpu->hyperv_features |= BIT(feat);
>> +            }
>> +        }
>> +    } else {
>> +        /* Check features availability and dependencies */
>> +        for (feat = 0; feat < ARRAY_SIZE(kvm_hyperv_properties); feat++) {
>> +            /* If the feature was not requested skip it. */
>> +            if (!hyperv_feat_enabled(cpu, feat)) {
>> +                continue;
>> +            }
>
> That's the loop I suggested in patch 11/19.  Nice.  :)
>

:-)

> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

Thanks!

>
>> +
>> +            /* Check if the feature is supported by KVM */
>> +            if (!hyperv_feature_supported(cs, feat)) {
>> +                error_setg(errp, "Hyper-V %s is not supported by kernel",
>> +                           kvm_hyperv_properties[feat].desc);
>> +                return;
>> +            }
>> +
>> +            /* Check dependencies */
>> +            hv_feature_check_deps(cpu, feat, &local_err);
>> +            if (local_err != NULL) {
>> +                error_propagate(errp, local_err);
>> +                return;
>> +            }
>> +        }
>>      }
>>  
>>      /* Additional dependencies not covered by kvm_hyperv_properties[] */
>> -- 
>> 2.30.2
>> 

-- 
Vitaly

Re: [PATCH v6 17/19] i386: HV_HYPERCALL_AVAILABLE privilege bit is always needed

[Vitaly Kuznetsov]

Eduardo Habkost <ehabkost@redhat.com> writes:

> On Thu, Apr 22, 2021 at 06:11:28PM +0200, Vitaly Kuznetsov wrote:
>> According to TLFS, Hyper-V guest is supposed to check
>> HV_HYPERCALL_AVAILABLE privilege bit before accessing
>> HV_X64_MSR_GUEST_OS_ID/HV_X64_MSR_HYPERCALL MSRs but at least some
>> Windows versions ignore that. As KVM is very permissive and allows
>> accessing these MSRs unconditionally, no issue is observed. We may,
>> however, want to tighten the checks eventually. Conforming to the
>> spec is probably also a good idea.
>> 
>> Add HV_HYPERCALL_AVAILABLE to all 'leaf' features with no dependencies.
>> 
>> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
>
> Are all VMs being created with HV_HYPERCALL_AVAILABLE unset,
> today?
>

No, we have HV_HYPERCALL_AVAILABLE encoded in 'hv-relaxed','hv-vapic'
and 'hv-time' features but not 


> Wouldn't it be simpler to simply add a new
> HYPERV_FEAT_HYPERCALL_AVAILABLE bit to hyperv_features, and
> enabling it by default?
>

We could do that but as I note above, we already have it for three
features.


> We don't necessarily need to make it configurable by the user,
> but probably it would be a good idea to keep the bit unset by
> default on older machine types.  Even if guests don't mind seeing
> the bit changing under their feet, it would make it easier for
> automated test cases that check for unexpected changes in raw
> CPUID data.

I see current situation as a bug. While most likely nobody runs with
a configuration like 'hv-vpindex,hv-synic' it is still valid. And if KVM
was enforcing the features (not yet), Windows would've just crashed in
early boot. Normal configurations will likely always include at least
'hv-time' which has HYPERV_FEAT_HYPERCALL_AVAILABLE enabled.

That being said, I'm not sure we need to maintain 'bug compatibility'
even for older machine types. I'm also not aware of any specific tests
for such 'crazy' configurations out there. The last patch of the series
adds a very simple test to qtest but this is about it.

>
>
>> ---
>>  target/i386/kvm/kvm.c | 15 +++++++++------
>>  1 file changed, 9 insertions(+), 6 deletions(-)
>> 
>> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
>> index 2c1a77f9b00f..d81451276cd8 100644
>> --- a/target/i386/kvm/kvm.c
>> +++ b/target/i386/kvm/kvm.c
>> @@ -835,6 +835,8 @@ static struct {
>>      [HYPERV_FEAT_CRASH] = {
>>          .desc = "crash MSRs (hv-crash)",
>>          .flags = {
>> +            {.func = HV_CPUID_FEATURES, .reg = R_EAX,
>> +             .bits = HV_HYPERCALL_AVAILABLE},
>>              {.func = HV_CPUID_FEATURES, .reg = R_EDX,
>>               .bits = HV_GUEST_CRASH_MSR_AVAILABLE}
>>          }
>> @@ -843,28 +845,28 @@ static struct {
>>          .desc = "reset MSR (hv-reset)",
>>          .flags = {
>>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
>> -             .bits = HV_RESET_AVAILABLE}
>> +             .bits = HV_HYPERCALL_AVAILABLE | HV_RESET_AVAILABLE}
>>          }
>>      },
>>      [HYPERV_FEAT_VPINDEX] = {
>>          .desc = "VP_INDEX MSR (hv-vpindex)",
>>          .flags = {
>>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
>> -             .bits = HV_VP_INDEX_AVAILABLE}
>> +             .bits = HV_HYPERCALL_AVAILABLE | HV_VP_INDEX_AVAILABLE}
>>          }
>>      },
>>      [HYPERV_FEAT_RUNTIME] = {
>>          .desc = "VP_RUNTIME MSR (hv-runtime)",
>>          .flags = {
>>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
>> -             .bits = HV_VP_RUNTIME_AVAILABLE}
>> +             .bits = HV_HYPERCALL_AVAILABLE | HV_VP_RUNTIME_AVAILABLE}
>>          }
>>      },
>>      [HYPERV_FEAT_SYNIC] = {
>>          .desc = "synthetic interrupt controller (hv-synic)",
>>          .flags = {
>>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
>> -             .bits = HV_SYNIC_AVAILABLE}
>> +             .bits = HV_HYPERCALL_AVAILABLE | HV_SYNIC_AVAILABLE}
>>          }
>>      },
>>      [HYPERV_FEAT_STIMER] = {
>> @@ -879,7 +881,7 @@ static struct {
>>          .desc = "frequency MSRs (hv-frequencies)",
>>          .flags = {
>>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
>> -             .bits = HV_ACCESS_FREQUENCY_MSRS},
>> +             .bits = HV_HYPERCALL_AVAILABLE | HV_ACCESS_FREQUENCY_MSRS},
>>              {.func = HV_CPUID_FEATURES, .reg = R_EDX,
>>               .bits = HV_FREQUENCY_MSRS_AVAILABLE}
>>          }
>> @@ -888,7 +890,8 @@ static struct {
>>          .desc = "reenlightenment MSRs (hv-reenlightenment)",
>>          .flags = {
>>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
>> -             .bits = HV_ACCESS_REENLIGHTENMENTS_CONTROL}
>> +             .bits = HV_HYPERCALL_AVAILABLE |
>> +             HV_ACCESS_REENLIGHTENMENTS_CONTROL}
>>          }
>>      },
>>      [HYPERV_FEAT_TLBFLUSH] = {
>> -- 
>> 2.30.2
>> 

-- 
Vitaly

Re: [PATCH v6 10/19] i386: move eVMCS enablement to hyperv_init_vcpu()

[Eduardo Habkost]

On Mon, May 24, 2021 at 02:00:37PM +0200, Vitaly Kuznetsov wrote:
[...]
> >> @@ -1455,6 +1454,21 @@ static int hyperv_init_vcpu(X86CPU *cpu)
> >>          }
> >>      }
> >>  
> >> +    if (hyperv_feat_enabled(cpu, HYPERV_FEAT_EVMCS)) {
> >> +        uint16_t evmcs_version;
> >> +
> >> +        ret = kvm_vcpu_enable_cap(cs, KVM_CAP_HYPERV_ENLIGHTENED_VMCS, 0,
> >> +                                  (uintptr_t)&evmcs_version);
> >> +
> >> +        if (ret < 0) {
> >> +            fprintf(stderr, "Hyper-V %s is not supported by kernel\n",
> >> +                    kvm_hyperv_properties[HYPERV_FEAT_EVMCS].desc);
> >> +            return ret;
> >> +        }
> >> +
> >> +        cpu->hyperv_nested[0] = evmcs_version;
> >
> > Wait, won't this break guest ABI?  Do we want to make
> > HYPERV_FEAT_EVMCS a migration blocker until this is fixed?
> >
> 
> Could you please elaborate on the issue? I read the above is: when 
> evmcs' feature was requested, make an attempt to enable
> KVM_CAP_HYPERV_ENLIGHTENED_VMCS, and bail out if this fails. Propagate
> the the acquired evmcs version to 'cpu->hyperv_nested[]' otherwise.

This will be visible to the guest at CPUID[0x4000000A].EAX,
correct?  You are initializing CPUID data with a value that
change depending on the host.

What is supposed to happen if live migrating to to a host with a
different evmcs_version?

-- 
Eduardo

Re: [PATCH v6 13/19] i386: prefer system KVM_GET_SUPPORTED_HV_CPUID ioctl over vCPU's one

[Eduardo Habkost]

On Mon, May 24, 2021 at 02:08:26PM +0200, Vitaly Kuznetsov wrote:
[...]
> >> @@ -980,7 +989,7 @@ static struct kvm_cpuid2 *get_supported_hv_cpuid(CPUState *cs)
> >>       * information early, just check for the capability and set the bit
> >>       * manually.
> >>       */
> >> -    if (kvm_check_extension(cs->kvm_state,
> >> +    if (!do_sys_ioctl && kvm_check_extension(cs->kvm_state,
> >>                              KVM_CAP_HYPERV_ENLIGHTENED_VMCS) > 0) {
> >
> > Oh, this conditional replaces the comment I suggested in patch
> > 10/19.  It makes it obvious that the hack can be deleted if we
> > remove support for the VCPU ioctl.
> >
> > So, when exactly will we be able to delete the VCPU ioctl code
> > and support only the system ioctl?
> 
> When QEMU drops support for kernels < 5.11? Note, current RHEL8 already
> supports system version so we're talking about upstream kernels/Ubuntu
> LTS/... 
> 
> I remember there was a list of supported kernels for QEMU somewhere but
> don't seem to be able to find it quickly, could you maybe point me in
> the right direction?

The KVM-specific kernel requirement is documented here:
https://qemu-project.gitlab.io/qemu/system/target-i386.html?highlight=kvm#os-requirements

I took a while to find it.  Maybe we should have a more visible
"runtime requirements" section in the docs, or it should be
moved to the supported build platforms section.

We have a clear policy on supported build platforms
[https://qemu-project.gitlab.io/qemu/system/build-platforms.html],
but not a clear policy for KVM kernel dependencies.

There's a table with Python and GCC versions at
[https://wiki.qemu.org/Supported_Build_Platforms].
Maybe it could include kernel version information as well.

-- 
Eduardo

Re: [PATCH v6 13/19] i386: prefer system KVM_GET_SUPPORTED_HV_CPUID ioctl over vCPU's one

[Daniel P. Berrangé]

On Wed, May 26, 2021 at 12:46:25PM -0400, Eduardo Habkost wrote:
> On Mon, May 24, 2021 at 02:08:26PM +0200, Vitaly Kuznetsov wrote:
> [...]
> > >> @@ -980,7 +989,7 @@ static struct kvm_cpuid2 *get_supported_hv_cpuid(CPUState *cs)
> > >>       * information early, just check for the capability and set the bit
> > >>       * manually.
> > >>       */
> > >> -    if (kvm_check_extension(cs->kvm_state,
> > >> +    if (!do_sys_ioctl && kvm_check_extension(cs->kvm_state,
> > >>                              KVM_CAP_HYPERV_ENLIGHTENED_VMCS) > 0) {
> > >
> > > Oh, this conditional replaces the comment I suggested in patch
> > > 10/19.  It makes it obvious that the hack can be deleted if we
> > > remove support for the VCPU ioctl.
> > >
> > > So, when exactly will we be able to delete the VCPU ioctl code
> > > and support only the system ioctl?
> > 
> > When QEMU drops support for kernels < 5.11? Note, current RHEL8 already
> > supports system version so we're talking about upstream kernels/Ubuntu
> > LTS/... 
> > 
> > I remember there was a list of supported kernels for QEMU somewhere but
> > don't seem to be able to find it quickly, could you maybe point me in
> > the right direction?
> 
> The KVM-specific kernel requirement is documented here:
> https://qemu-project.gitlab.io/qemu/system/target-i386.html?highlight=kvm#os-requirements
> 
> I took a while to find it.  Maybe we should have a more visible
> "runtime requirements" section in the docs, or it should be
> moved to the supported build platforms section.
> 
> We have a clear policy on supported build platforms
> [https://qemu-project.gitlab.io/qemu/system/build-platforms.html],
> but not a clear policy for KVM kernel dependencies.

While it says "supported build platforms", that was implicitly
assumed to also refer to "runtime platforms". We should just
rename it to "supported-platforms.html" to make it more obvious.

Thus, the minimum KVM kernel version we need follows the same rules.
Look at whatever is the oldest kernel across the distros we target.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: [PATCH v6 15/19] i386: expand Hyper-V features during CPU feature expansion time

[Eduardo Habkost]

On Mon, May 24, 2021 at 02:13:09PM +0200, Vitaly Kuznetsov wrote:
[...]
> >> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
> >> index a42263b24fca..d5551c4ab5cf 100644
> >> --- a/target/i386/kvm/kvm.c
> >> +++ b/target/i386/kvm/kvm.c
> >> @@ -1216,13 +1216,22 @@ static uint32_t hv_build_cpuid_leaf(CPUState *cs, uint32_t func, int reg)
> >>   * of 'hv_passthrough' mode and fills the environment with all supported
> >>   * Hyper-V features.
> >>   */
> >> -static void hyperv_expand_features(CPUState *cs, Error **errp)
> >> +void kvm_hyperv_expand_features(X86CPU *cpu, Error **errp)
> >>  {
> >> -    X86CPU *cpu = X86_CPU(cs);
> >> +    CPUState *cs = CPU(cpu);
> >>  
> >>      if (!hyperv_enabled(cpu))
> >>          return;
> >>  
> >> +    /*
> >> +     * When kvm_hyperv_expand_features is called at CPU feature expansion
> >> +     * time per-CPU kvm_state is not available yet so we can only proceed
> >> +     * when KVM_CAP_SYS_HYPERV_CPUID is supported.
> >> +     */
> >> +    if (!cs->kvm_state &&
> >> +        !kvm_check_extension(kvm_state, KVM_CAP_SYS_HYPERV_CPUID))
> >> +        return;
> >> +
> >>      if (cpu->hyperv_passthrough) {
> >>          cpu->hyperv_vendor_id[0] =
> >>              hv_cpuid_get_host(cs, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, R_EBX);
> >> @@ -1556,7 +1565,7 @@ int kvm_arch_init_vcpu(CPUState *cs)
> >>      env->apic_bus_freq = KVM_APIC_BUS_FREQUENCY;
> >>  
> >>      /* Paravirtualization CPUIDs */
> >> -    hyperv_expand_features(cs, &local_err);
> >> +    kvm_hyperv_expand_features(cpu, &local_err);
> >
> > Do we still need to call the function again here?
> >
> > If the first expansion isn't expanding everything, I'm afraid
> > this second call will hide bugs in query-cpu-model-expansion.
> >
> 
> The first expansion will do nothing if KVM_CAP_SYS_HYPERV_CPUID is not
> supported, calling it here allows us to proceed. The series makes
> 'query-cpu-model-expansion' output correct only with
> KVM_CAP_SYS_HYPERV_CPUID, without it we don't seem to be able to do much
> (unless we decide to create a 'scratch' CPU or something like that).

Oh, I see.  I suggest adding a comment explaining that.
Developers might be tempted to delete it and not notice it breaks
under older kernels.

-- 
Eduardo

Re: [PATCH v6 17/19] i386: HV_HYPERCALL_AVAILABLE privilege bit is always needed

[Eduardo Habkost]

On Mon, May 24, 2021 at 02:22:47PM +0200, Vitaly Kuznetsov wrote:
> Eduardo Habkost <ehabkost@redhat.com> writes:
> 
> > On Thu, Apr 22, 2021 at 06:11:28PM +0200, Vitaly Kuznetsov wrote:
> >> According to TLFS, Hyper-V guest is supposed to check
> >> HV_HYPERCALL_AVAILABLE privilege bit before accessing
> >> HV_X64_MSR_GUEST_OS_ID/HV_X64_MSR_HYPERCALL MSRs but at least some
> >> Windows versions ignore that. As KVM is very permissive and allows
> >> accessing these MSRs unconditionally, no issue is observed. We may,
> >> however, want to tighten the checks eventually. Conforming to the
> >> spec is probably also a good idea.
> >> 
> >> Add HV_HYPERCALL_AVAILABLE to all 'leaf' features with no dependencies.
> >> 
> >> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> >
> > Are all VMs being created with HV_HYPERCALL_AVAILABLE unset,
> > today?
> >
> 
> No, we have HV_HYPERCALL_AVAILABLE encoded in 'hv-relaxed','hv-vapic'
> and 'hv-time' features but not 
> 
> 
> > Wouldn't it be simpler to simply add a new
> > HYPERV_FEAT_HYPERCALL_AVAILABLE bit to hyperv_features, and
> > enabling it by default?
> >
> 
> We could do that but as I note above, we already have it for three
> features.

Do we have any cases where we do not want to enable
HV_HYPERCALL_AVAILABLE?

Would it be OK to just hardcoded it in hyperv_fill_cpuids() like
we do with HV_CPU_DYNAMIC_PARTITIONING_AVAILABLE?

> 
> 
> > We don't necessarily need to make it configurable by the user,
> > but probably it would be a good idea to keep the bit unset by
> > default on older machine types.  Even if guests don't mind seeing
> > the bit changing under their feet, it would make it easier for
> > automated test cases that check for unexpected changes in raw
> > CPUID data.
> 
> I see current situation as a bug. While most likely nobody runs with
> a configuration like 'hv-vpindexem,hv-synic' it is still valid. And if KVM
> was enforcing the features (not yet), Windows would've just crashed in
> early boot. Normal configurations will likely always include at least
> 'hv-time' which has HYPERV_FEAT_HYPERCALL_AVAILABLE enabled.
> 
> That being said, I'm not sure we need to maintain 'bug compatibility'
> even for older machine types. I'm also not aware of any specific tests
> for such 'crazy' configurations out there. The last patch of the series
> adds a very simple test to qtest but this is about it.

If you are 100% sure the CPUID change can't crash or confuse a
guest, then that's OK.  I agree that bug compatibility is not a
must if the bit is simply ignored by most guests and by KVM
emulation code.


> 
> >
> >
> >> ---
> >>  target/i386/kvm/kvm.c | 15 +++++++++------
> >>  1 file changed, 9 insertions(+), 6 deletions(-)
> >> 
> >> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
> >> index 2c1a77f9b00f..d81451276cd8 100644
> >> --- a/target/i386/kvm/kvm.c
> >> +++ b/target/i386/kvm/kvm.c
> >> @@ -835,6 +835,8 @@ static struct {
> >>      [HYPERV_FEAT_CRASH] = {
> >>          .desc = "crash MSRs (hv-crash)",
> >>          .flags = {
> >> +            {.func = HV_CPUID_FEATURES, .reg = R_EAX,
> >> +             .bits = HV_HYPERCALL_AVAILABLE},
> >>              {.func = HV_CPUID_FEATURES, .reg = R_EDX,
> >>               .bits = HV_GUEST_CRASH_MSR_AVAILABLE}
> >>          }
> >> @@ -843,28 +845,28 @@ static struct {
> >>          .desc = "reset MSR (hv-reset)",
> >>          .flags = {
> >>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
> >> -             .bits = HV_RESET_AVAILABLE}
> >> +             .bits = HV_HYPERCALL_AVAILABLE | HV_RESET_AVAILABLE}
> >>          }
> >>      },
> >>      [HYPERV_FEAT_VPINDEX] = {
> >>          .desc = "VP_INDEX MSR (hv-vpindex)",
> >>          .flags = {
> >>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
> >> -             .bits = HV_VP_INDEX_AVAILABLE}
> >> +             .bits = HV_HYPERCALL_AVAILABLE | HV_VP_INDEX_AVAILABLE}
> >>          }
> >>      },
> >>      [HYPERV_FEAT_RUNTIME] = {
> >>          .desc = "VP_RUNTIME MSR (hv-runtime)",
> >>          .flags = {
> >>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
> >> -             .bits = HV_VP_RUNTIME_AVAILABLE}
> >> +             .bits = HV_HYPERCALL_AVAILABLE | HV_VP_RUNTIME_AVAILABLE}
> >>          }
> >>      },
> >>      [HYPERV_FEAT_SYNIC] = {
> >>          .desc = "synthetic interrupt controller (hv-synic)",
> >>          .flags = {
> >>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
> >> -             .bits = HV_SYNIC_AVAILABLE}
> >> +             .bits = HV_HYPERCALL_AVAILABLE | HV_SYNIC_AVAILABLE}
> >>          }
> >>      },
> >>      [HYPERV_FEAT_STIMER] = {
> >> @@ -879,7 +881,7 @@ static struct {
> >>          .desc = "frequency MSRs (hv-frequencies)",
> >>          .flags = {
> >>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
> >> -             .bits = HV_ACCESS_FREQUENCY_MSRS},
> >> +             .bits = HV_HYPERCALL_AVAILABLE | HV_ACCESS_FREQUENCY_MSRS},
> >>              {.func = HV_CPUID_FEATURES, .reg = R_EDX,
> >>               .bits = HV_FREQUENCY_MSRS_AVAILABLE}
> >>          }
> >> @@ -888,7 +890,8 @@ static struct {
> >>          .desc = "reenlightenment MSRs (hv-reenlightenment)",
> >>          .flags = {
> >>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
> >> -             .bits = HV_ACCESS_REENLIGHTENMENTS_CONTROL}
> >> +             .bits = HV_HYPERCALL_AVAILABLE |
> >> +             HV_ACCESS_REENLIGHTENMENTS_CONTROL}
> >>          }
> >>      },
> >>      [HYPERV_FEAT_TLBFLUSH] = {
> >> -- 
> >> 2.30.2
> >> 
> 
> -- 
> Vitaly
> 

-- 
Eduardo

Re: [PATCH v6 00/19] i386: KVM: expand Hyper-V features early

[Eduardo Habkost]

On Thu, Apr 22, 2021 at 06:11:11PM +0200, Vitaly Kuznetsov wrote:
> Vitaly Kuznetsov (19):
>   i386: keep hyperv_vendor string up-to-date
>   i386: invert hyperv_spinlock_attempts setting logic with
>     hv_passthrough
>   i386: always fill Hyper-V CPUID feature leaves from X86CPU data
>   i386: stop using env->features[] for filling Hyper-V CPUIDs
>   i386: introduce hyperv_feature_supported()
>   i386: introduce hv_cpuid_get_host()
>   i386: drop FEAT_HYPERV feature leaves
>   i386: introduce hv_cpuid_cache
>   i386: split hyperv_handle_properties() into
>     hyperv_expand_features()/hyperv_fill_cpuids()
>   i386: move eVMCS enablement to hyperv_init_vcpu()
>   i386: switch hyperv_expand_features() to using error_setg()
>   i386: adjust the expected KVM_GET_SUPPORTED_HV_CPUID array size
>   i386: prefer system KVM_GET_SUPPORTED_HV_CPUID ioctl over vCPU's one
>   i386: use global kvm_state in hyperv_enabled() check

I'm queueing patches 1-14 (the ones above) on my x86-next branch.
Thanks!

>   i386: expand Hyper-V features during CPU feature expansion time
>   i386: kill off hv_cpuid_check_and_set()
>   i386: HV_HYPERCALL_AVAILABLE privilege bit is always needed
>   i386: Hyper-V SynIC requires POST_MESSAGES/SIGNAL_EVENTS priviliges
>   qtest/hyperv: Introduce a simple hyper-v test


-- 
Eduardo

Re: [PATCH v6 10/19] i386: move eVMCS enablement to hyperv_init_vcpu()

[Vitaly Kuznetsov]

Eduardo Habkost <ehabkost@redhat.com> writes:

> On Mon, May 24, 2021 at 02:00:37PM +0200, Vitaly Kuznetsov wrote:
> [...]
>> >> @@ -1455,6 +1454,21 @@ static int hyperv_init_vcpu(X86CPU *cpu)
>> >>          }
>> >>      }
>> >>  
>> >> +    if (hyperv_feat_enabled(cpu, HYPERV_FEAT_EVMCS)) {
>> >> +        uint16_t evmcs_version;
>> >> +
>> >> +        ret = kvm_vcpu_enable_cap(cs, KVM_CAP_HYPERV_ENLIGHTENED_VMCS, 0,
>> >> +                                  (uintptr_t)&evmcs_version);
>> >> +
>> >> +        if (ret < 0) {
>> >> +            fprintf(stderr, "Hyper-V %s is not supported by kernel\n",
>> >> +                    kvm_hyperv_properties[HYPERV_FEAT_EVMCS].desc);
>> >> +            return ret;
>> >> +        }
>> >> +
>> >> +        cpu->hyperv_nested[0] = evmcs_version;
>> >
>> > Wait, won't this break guest ABI?  Do we want to make
>> > HYPERV_FEAT_EVMCS a migration blocker until this is fixed?
>> >
>> 
>> Could you please elaborate on the issue? I read the above is: when 
>> evmcs' feature was requested, make an attempt to enable
>> KVM_CAP_HYPERV_ENLIGHTENED_VMCS, and bail out if this fails. Propagate
>> the the acquired evmcs version to 'cpu->hyperv_nested[]' otherwise.
>
> This will be visible to the guest at CPUID[0x4000000A].EAX,
> correct?  You are initializing CPUID data with a value that
> change depending on the host.
>
> What is supposed to happen if live migrating to to a host with a
> different evmcs_version?

(Note: 'evmcs_version' here is the 'maximum supported evmcs version',
not 'used evmcs version').

This is a purely theoretical question at this moment as there's only one
existing (and supported) eVMCS version: 1.

In future, when (and if) e.g. EVMCSv2 appears, we'll have to introduce a
different QEMU option for it most likely (or something like
'hv-evmcs=1', 'hv-evmcs=2' ... ) as how else would we prevent migration
to a host which doesn't support certain eVMCS version (e.g. EVMCSv2 ->
EVMCSv1)?

I'd be fine with hardcoding '1' and just checking that the returned
version is >= 1 for now. Migration blocker seems to be an overkill (as
there's no real problem, we're just trying to make the code future
proof). 

-- 
Vitaly

Re: [PATCH v6 15/19] i386: expand Hyper-V features during CPU feature expansion time

[Vitaly Kuznetsov]

Eduardo Habkost <ehabkost@redhat.com> writes:

> On Mon, May 24, 2021 at 02:13:09PM +0200, Vitaly Kuznetsov wrote:
> [...]
>> >> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
>> >> index a42263b24fca..d5551c4ab5cf 100644
>> >> --- a/target/i386/kvm/kvm.c
>> >> +++ b/target/i386/kvm/kvm.c
>> >> @@ -1216,13 +1216,22 @@ static uint32_t hv_build_cpuid_leaf(CPUState *cs, uint32_t func, int reg)
>> >>   * of 'hv_passthrough' mode and fills the environment with all supported
>> >>   * Hyper-V features.
>> >>   */
>> >> -static void hyperv_expand_features(CPUState *cs, Error **errp)
>> >> +void kvm_hyperv_expand_features(X86CPU *cpu, Error **errp)
>> >>  {
>> >> -    X86CPU *cpu = X86_CPU(cs);
>> >> +    CPUState *cs = CPU(cpu);
>> >>  
>> >>      if (!hyperv_enabled(cpu))
>> >>          return;
>> >>  
>> >> +    /*
>> >> +     * When kvm_hyperv_expand_features is called at CPU feature expansion
>> >> +     * time per-CPU kvm_state is not available yet so we can only proceed
>> >> +     * when KVM_CAP_SYS_HYPERV_CPUID is supported.
>> >> +     */
>> >> +    if (!cs->kvm_state &&
>> >> +        !kvm_check_extension(kvm_state, KVM_CAP_SYS_HYPERV_CPUID))
>> >> +        return;
>> >> +
>> >>      if (cpu->hyperv_passthrough) {
>> >>          cpu->hyperv_vendor_id[0] =
>> >>              hv_cpuid_get_host(cs, HV_CPUID_VENDOR_AND_MAX_FUNCTIONS, R_EBX);
>> >> @@ -1556,7 +1565,7 @@ int kvm_arch_init_vcpu(CPUState *cs)
>> >>      env->apic_bus_freq = KVM_APIC_BUS_FREQUENCY;
>> >>  
>> >>      /* Paravirtualization CPUIDs */
>> >> -    hyperv_expand_features(cs, &local_err);
>> >> +    kvm_hyperv_expand_features(cpu, &local_err);
>> >
>> > Do we still need to call the function again here?
>> >
>> > If the first expansion isn't expanding everything, I'm afraid
>> > this second call will hide bugs in query-cpu-model-expansion.
>> >
>> 
>> The first expansion will do nothing if KVM_CAP_SYS_HYPERV_CPUID is not
>> supported, calling it here allows us to proceed. The series makes
>> 'query-cpu-model-expansion' output correct only with
>> KVM_CAP_SYS_HYPERV_CPUID, without it we don't seem to be able to do much
>> (unless we decide to create a 'scratch' CPU or something like that).
>
> Oh, I see.  I suggest adding a comment explaining that.
> Developers might be tempted to delete it and not notice it breaks
> under older kernels.

Will do, thanks!

-- 
Vitaly

Re: [PATCH v6 17/19] i386: HV_HYPERCALL_AVAILABLE privilege bit is always needed

[Vitaly Kuznetsov]

Eduardo Habkost <ehabkost@redhat.com> writes:

> On Mon, May 24, 2021 at 02:22:47PM +0200, Vitaly Kuznetsov wrote:
>> Eduardo Habkost <ehabkost@redhat.com> writes:
>> 
>> > On Thu, Apr 22, 2021 at 06:11:28PM +0200, Vitaly Kuznetsov wrote:
>> >> According to TLFS, Hyper-V guest is supposed to check
>> >> HV_HYPERCALL_AVAILABLE privilege bit before accessing
>> >> HV_X64_MSR_GUEST_OS_ID/HV_X64_MSR_HYPERCALL MSRs but at least some
>> >> Windows versions ignore that. As KVM is very permissive and allows
>> >> accessing these MSRs unconditionally, no issue is observed. We may,
>> >> however, want to tighten the checks eventually. Conforming to the
>> >> spec is probably also a good idea.
>> >> 
>> >> Add HV_HYPERCALL_AVAILABLE to all 'leaf' features with no dependencies.
>> >> 
>> >> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
>> >
>> > Are all VMs being created with HV_HYPERCALL_AVAILABLE unset,
>> > today?
>> >
>> 
>> No, we have HV_HYPERCALL_AVAILABLE encoded in 'hv-relaxed','hv-vapic'
>> and 'hv-time' features but not 
>> 
>> 
>> > Wouldn't it be simpler to simply add a new
>> > HYPERV_FEAT_HYPERCALL_AVAILABLE bit to hyperv_features, and
>> > enabling it by default?
>> >
>> 
>> We could do that but as I note above, we already have it for three
>> features.
>
> Do we have any cases where we do not want to enable
> HV_HYPERCALL_AVAILABLE?
>
> Would it be OK to just hardcoded it in hyperv_fill_cpuids() like
> we do with HV_CPU_DYNAMIC_PARTITIONING_AVAILABLE?
>

struct kvm_hyperv_properties[] serves two purposes:
1) Set corresponding guest visible CPUID bits when certain features are
enabled.

2) Check, that KVM supports certain features before we expose them to the
  guest.

Whatever we hardcode in hyperv_fill_cpuids() gives us 1) but not 2). For
this particular bit it probably doesn't matter as even the oldest
supported kernel (v4.5) has it. That said, I'm OK with moving this to
hyperv_fill_cpuids().

>> 
>> 
>> > We don't necessarily need to make it configurable by the user,
>> > but probably it would be a good idea to keep the bit unset by
>> > default on older machine types.  Even if guests don't mind seeing
>> > the bit changing under their feet, it would make it easier for
>> > automated test cases that check for unexpected changes in raw
>> > CPUID data.
>> 
>> I see current situation as a bug. While most likely nobody runs with
>> a configuration like 'hv-vpindexem,hv-synic' it is still valid. And if KVM
>> was enforcing the features (not yet), Windows would've just crashed in
>> early boot. Normal configurations will likely always include at least
>> 'hv-time' which has HYPERV_FEAT_HYPERCALL_AVAILABLE enabled.
>> 
>> That being said, I'm not sure we need to maintain 'bug compatibility'
>> even for older machine types. I'm also not aware of any specific tests
>> for such 'crazy' configurations out there. The last patch of the series
>> adds a very simple test to qtest but this is about it.
>
> If you are 100% sure the CPUID change can't crash or confuse a
> guest, then that's OK.  I agree that bug compatibility is not a
> must if the bit is simply ignored by most guests and by KVM
> emulation code.

Strictly speaking, this bit has to be set or the guest can't use any of
the Hyper-V features. It was proven that at least certain Windows
versions don't even check it assuming it's always set. Moreover, we
already set it for some very basic enlightenments ('hv-time') so there's
going to be no change at all for real world configutations.

>
>
>> 
>> >
>> >
>> >> ---
>> >>  target/i386/kvm/kvm.c | 15 +++++++++------
>> >>  1 file changed, 9 insertions(+), 6 deletions(-)
>> >> 
>> >> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
>> >> index 2c1a77f9b00f..d81451276cd8 100644
>> >> --- a/target/i386/kvm/kvm.c
>> >> +++ b/target/i386/kvm/kvm.c
>> >> @@ -835,6 +835,8 @@ static struct {
>> >>      [HYPERV_FEAT_CRASH] = {
>> >>          .desc = "crash MSRs (hv-crash)",
>> >>          .flags = {
>> >> +            {.func = HV_CPUID_FEATURES, .reg = R_EAX,
>> >> +             .bits = HV_HYPERCALL_AVAILABLE},
>> >>              {.func = HV_CPUID_FEATURES, .reg = R_EDX,
>> >>               .bits = HV_GUEST_CRASH_MSR_AVAILABLE}
>> >>          }
>> >> @@ -843,28 +845,28 @@ static struct {
>> >>          .desc = "reset MSR (hv-reset)",
>> >>          .flags = {
>> >>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
>> >> -             .bits = HV_RESET_AVAILABLE}
>> >> +             .bits = HV_HYPERCALL_AVAILABLE | HV_RESET_AVAILABLE}
>> >>          }
>> >>      },
>> >>      [HYPERV_FEAT_VPINDEX] = {
>> >>          .desc = "VP_INDEX MSR (hv-vpindex)",
>> >>          .flags = {
>> >>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
>> >> -             .bits = HV_VP_INDEX_AVAILABLE}
>> >> +             .bits = HV_HYPERCALL_AVAILABLE | HV_VP_INDEX_AVAILABLE}
>> >>          }
>> >>      },
>> >>      [HYPERV_FEAT_RUNTIME] = {
>> >>          .desc = "VP_RUNTIME MSR (hv-runtime)",
>> >>          .flags = {
>> >>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
>> >> -             .bits = HV_VP_RUNTIME_AVAILABLE}
>> >> +             .bits = HV_HYPERCALL_AVAILABLE | HV_VP_RUNTIME_AVAILABLE}
>> >>          }
>> >>      },
>> >>      [HYPERV_FEAT_SYNIC] = {
>> >>          .desc = "synthetic interrupt controller (hv-synic)",
>> >>          .flags = {
>> >>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
>> >> -             .bits = HV_SYNIC_AVAILABLE}
>> >> +             .bits = HV_HYPERCALL_AVAILABLE | HV_SYNIC_AVAILABLE}
>> >>          }
>> >>      },
>> >>      [HYPERV_FEAT_STIMER] = {
>> >> @@ -879,7 +881,7 @@ static struct {
>> >>          .desc = "frequency MSRs (hv-frequencies)",
>> >>          .flags = {
>> >>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
>> >> -             .bits = HV_ACCESS_FREQUENCY_MSRS},
>> >> +             .bits = HV_HYPERCALL_AVAILABLE | HV_ACCESS_FREQUENCY_MSRS},
>> >>              {.func = HV_CPUID_FEATURES, .reg = R_EDX,
>> >>               .bits = HV_FREQUENCY_MSRS_AVAILABLE}
>> >>          }
>> >> @@ -888,7 +890,8 @@ static struct {
>> >>          .desc = "reenlightenment MSRs (hv-reenlightenment)",
>> >>          .flags = {
>> >>              {.func = HV_CPUID_FEATURES, .reg = R_EAX,
>> >> -             .bits = HV_ACCESS_REENLIGHTENMENTS_CONTROL}
>> >> +             .bits = HV_HYPERCALL_AVAILABLE |
>> >> +             HV_ACCESS_REENLIGHTENMENTS_CONTROL}
>> >>          }
>> >>      },
>> >>      [HYPERV_FEAT_TLBFLUSH] = {
>> >> -- 
>> >> 2.30.2
>> >> 
>> 
>> -- 
>> Vitaly
>> 

-- 
Vitaly

Re: [PATCH v6 00/19] i386: KVM: expand Hyper-V features early

[Vitaly Kuznetsov]

Eduardo Habkost <ehabkost@redhat.com> writes:

> On Thu, Apr 22, 2021 at 06:11:11PM +0200, Vitaly Kuznetsov wrote:
>> Vitaly Kuznetsov (19):
>>   i386: keep hyperv_vendor string up-to-date
>>   i386: invert hyperv_spinlock_attempts setting logic with
>>     hv_passthrough
>>   i386: always fill Hyper-V CPUID feature leaves from X86CPU data
>>   i386: stop using env->features[] for filling Hyper-V CPUIDs
>>   i386: introduce hyperv_feature_supported()
>>   i386: introduce hv_cpuid_get_host()
>>   i386: drop FEAT_HYPERV feature leaves
>>   i386: introduce hv_cpuid_cache
>>   i386: split hyperv_handle_properties() into
>>     hyperv_expand_features()/hyperv_fill_cpuids()
>>   i386: move eVMCS enablement to hyperv_init_vcpu()
>>   i386: switch hyperv_expand_features() to using error_setg()
>>   i386: adjust the expected KVM_GET_SUPPORTED_HV_CPUID array size
>>   i386: prefer system KVM_GET_SUPPORTED_HV_CPUID ioctl over vCPU's one
>>   i386: use global kvm_state in hyperv_enabled() check
>
> I'm queueing patches 1-14 (the ones above) on my x86-next branch.

Thank you! Is it published somewhere so I can base next version[s] on it? 

-- 
Vitaly

Re: [PATCH v6 10/19] i386: move eVMCS enablement to hyperv_init_vcpu()

[Eduardo Habkost]

On Thu, May 27, 2021 at 09:27:01AM +0200, Vitaly Kuznetsov wrote:
> Eduardo Habkost <ehabkost@redhat.com> writes:
> 
> > On Mon, May 24, 2021 at 02:00:37PM +0200, Vitaly Kuznetsov wrote:
> > [...]
> >> >> @@ -1455,6 +1454,21 @@ static int hyperv_init_vcpu(X86CPU *cpu)
> >> >>          }
> >> >>      }
> >> >>  
> >> >> +    if (hyperv_feat_enabled(cpu, HYPERV_FEAT_EVMCS)) {
> >> >> +        uint16_t evmcs_version;
> >> >> +
> >> >> +        ret = kvm_vcpu_enable_cap(cs, KVM_CAP_HYPERV_ENLIGHTENED_VMCS, 0,
> >> >> +                                  (uintptr_t)&evmcs_version);
> >> >> +
> >> >> +        if (ret < 0) {
> >> >> +            fprintf(stderr, "Hyper-V %s is not supported by kernel\n",
> >> >> +                    kvm_hyperv_properties[HYPERV_FEAT_EVMCS].desc);
> >> >> +            return ret;
> >> >> +        }
> >> >> +
> >> >> +        cpu->hyperv_nested[0] = evmcs_version;
> >> >
> >> > Wait, won't this break guest ABI?  Do we want to make
> >> > HYPERV_FEAT_EVMCS a migration blocker until this is fixed?
> >> >
> >> 
> >> Could you please elaborate on the issue? I read the above is: when 
> >> evmcs' feature was requested, make an attempt to enable
> >> KVM_CAP_HYPERV_ENLIGHTENED_VMCS, and bail out if this fails. Propagate
> >> the the acquired evmcs version to 'cpu->hyperv_nested[]' otherwise.
> >
> > This will be visible to the guest at CPUID[0x4000000A].EAX,
> > correct?  You are initializing CPUID data with a value that
> > change depending on the host.
> >
> > What is supposed to happen if live migrating to to a host with a
> > different evmcs_version?
> 
> (Note: 'evmcs_version' here is the 'maximum supported evmcs version',
> not 'used evmcs version').
> 
> This is a purely theoretical question at this moment as there's only one
> existing (and supported) eVMCS version: 1.

Good to know.  :)

> 
> In future, when (and if) e.g. EVMCSv2 appears, we'll have to introduce a
> different QEMU option for it most likely (or something like
> 'hv-evmcs=1', 'hv-evmcs=2' ... ) as how else would we prevent migration
> to a host which doesn't support certain eVMCS version (e.g. EVMCSv2 ->
> EVMCSv1)?
> 
> I'd be fine with hardcoding '1' and just checking that the returned
> version is >= 1 for now. Migration blocker seems to be an overkill (as
> there's no real problem, we're just trying to make the code future
> proof). 

Sounds good to me.  I agree a migration blocker is not the right
solution if currently all hosts have evmcs_version==1.

-- 
Eduardo

Re: [PATCH v6 17/19] i386: HV_HYPERCALL_AVAILABLE privilege bit is always needed

[Eduardo Habkost]

On Thu, May 27, 2021 at 09:37:59AM +0200, Vitaly Kuznetsov wrote:
> Eduardo Habkost <ehabkost@redhat.com> writes:
> 
> > On Mon, May 24, 2021 at 02:22:47PM +0200, Vitaly Kuznetsov wrote:
> >> Eduardo Habkost <ehabkost@redhat.com> writes:
> >> 
> >> > On Thu, Apr 22, 2021 at 06:11:28PM +0200, Vitaly Kuznetsov wrote:
> >> >> According to TLFS, Hyper-V guest is supposed to check
> >> >> HV_HYPERCALL_AVAILABLE privilege bit before accessing
> >> >> HV_X64_MSR_GUEST_OS_ID/HV_X64_MSR_HYPERCALL MSRs but at least some
> >> >> Windows versions ignore that. As KVM is very permissive and allows
> >> >> accessing these MSRs unconditionally, no issue is observed. We may,
> >> >> however, want to tighten the checks eventually. Conforming to the
> >> >> spec is probably also a good idea.
> >> >> 
> >> >> Add HV_HYPERCALL_AVAILABLE to all 'leaf' features with no dependencies.
> >> >> 
> >> >> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
> >> >
> >> > Are all VMs being created with HV_HYPERCALL_AVAILABLE unset,
> >> > today?
> >> >
> >> 
> >> No, we have HV_HYPERCALL_AVAILABLE encoded in 'hv-relaxed','hv-vapic'
> >> and 'hv-time' features but not 
> >> 
> >> 
> >> > Wouldn't it be simpler to simply add a new
> >> > HYPERV_FEAT_HYPERCALL_AVAILABLE bit to hyperv_features, and
> >> > enabling it by default?
> >> >
> >> 
> >> We could do that but as I note above, we already have it for three
> >> features.
> >
> > Do we have any cases where we do not want to enable
> > HV_HYPERCALL_AVAILABLE?
> >
> > Would it be OK to just hardcoded it in hyperv_fill_cpuids() like
> > we do with HV_CPU_DYNAMIC_PARTITIONING_AVAILABLE?
> >
> 
> struct kvm_hyperv_properties[] serves two purposes:
> 1) Set corresponding guest visible CPUID bits when certain features are
> enabled.
> 
> 2) Check, that KVM supports certain features before we expose them to the
>   guest.

Oh, you're right.

> 
> Whatever we hardcode in hyperv_fill_cpuids() gives us 1) but not 2). For
> this particular bit it probably doesn't matter as even the oldest
> supported kernel (v4.5) has it. That said, I'm OK with moving this to
> hyperv_fill_cpuids().

I'm only worried about the risk of somebody forgetting to
hardcode the HV_HYPERCALL_AVAILABLE bit in new
kvm_hyperv_expand_features[] entries in the future.

A new HYPERV_FEAT_HYPERCALL_AVAILABLE bit (hardcoded to 1 at
kvm_hyperv_expand_features()) would give us feature checking.
But if you're OK with hardcoding it at hyperv_fill_cpuids(), it's
probably the simplest solution.

> [...]

-- 
Eduardo

Re: [PATCH v6 00/19] i386: KVM: expand Hyper-V features early

[Eduardo Habkost]

On Thu, May 27, 2021 at 09:39:02AM +0200, Vitaly Kuznetsov wrote:
> Eduardo Habkost <ehabkost@redhat.com> writes:
> 
> > On Thu, Apr 22, 2021 at 06:11:11PM +0200, Vitaly Kuznetsov wrote:
> >> Vitaly Kuznetsov (19):
> >>   i386: keep hyperv_vendor string up-to-date
> >>   i386: invert hyperv_spinlock_attempts setting logic with
> >>     hv_passthrough
> >>   i386: always fill Hyper-V CPUID feature leaves from X86CPU data
> >>   i386: stop using env->features[] for filling Hyper-V CPUIDs
> >>   i386: introduce hyperv_feature_supported()
> >>   i386: introduce hv_cpuid_get_host()
> >>   i386: drop FEAT_HYPERV feature leaves
> >>   i386: introduce hv_cpuid_cache
> >>   i386: split hyperv_handle_properties() into
> >>     hyperv_expand_features()/hyperv_fill_cpuids()
> >>   i386: move eVMCS enablement to hyperv_init_vcpu()
> >>   i386: switch hyperv_expand_features() to using error_setg()
> >>   i386: adjust the expected KVM_GET_SUPPORTED_HV_CPUID array size
> >>   i386: prefer system KVM_GET_SUPPORTED_HV_CPUID ioctl over vCPU's one
> >>   i386: use global kvm_state in hyperv_enabled() check
> >
> > I'm queueing patches 1-14 (the ones above) on my x86-next branch.
> 
> Thank you! Is it published somewhere so I can base next version[s] on it? 

Yes: http://gitlab.com/ehabkost/qemu x86-next

-- 
Eduardo

Re: [PATCH v6 01/19] i386: keep hyperv_vendor string up-to-date

[Vitaly Kuznetsov]

Eduardo Habkost <ehabkost@redhat.com> writes:

> On Thu, Apr 22, 2021 at 06:11:12PM +0200, Vitaly Kuznetsov wrote:
>> When cpu->hyperv_vendor is not set manually we default to "Microsoft Hv"
>> and in 'hv_passthrough' mode we get the information from the host. This
>> information is stored in cpu->hyperv_vendor_id[] array but we don't update
>> cpu->hyperv_vendor string so e.g. QMP's query-cpu-model-expansion output
>> is incorrect.
>
> I was confused for a while because this can't happen until patch
> 15/19 is applied.  Probably worth a note in the commit message
> indicating that hyperv_handle_properties() will be called by
> x86_cpu_expand_features() in the future.
>
>> 
>> Signed-off-by: Vitaly Kuznetsov <vkuznets@redhat.com>
>> ---
>>  target/i386/cpu.c     | 19 +++++++++----------
>>  target/i386/kvm/kvm.c |  5 +++++
>>  2 files changed, 14 insertions(+), 10 deletions(-)
>> 
>> diff --git a/target/i386/cpu.c b/target/i386/cpu.c
>> index ad99cad0e7ce..2d05df232329 100644
>> --- a/target/i386/cpu.c
>> +++ b/target/i386/cpu.c
>> @@ -6665,17 +6665,16 @@ static void x86_cpu_hyperv_realize(X86CPU *cpu)
>>  
>>      /* Hyper-V vendor id */
>>      if (!cpu->hyperv_vendor) {
>> -        memcpy(cpu->hyperv_vendor_id, "Microsoft Hv", 12);
>> -    } else {
>> -        len = strlen(cpu->hyperv_vendor);
>> -
>> -        if (len > 12) {
>> -            warn_report("hv-vendor-id truncated to 12 characters");
>> -            len = 12;
>> -        }
>> -        memset(cpu->hyperv_vendor_id, 0, 12);
>> -        memcpy(cpu->hyperv_vendor_id, cpu->hyperv_vendor, len);
>> +        object_property_set_str(OBJECT(cpu), "hv-vendor-id", "Microsoft Hv",
>> +                                &error_abort);
>> +    }
>> +    len = strlen(cpu->hyperv_vendor);
>> +    if (len > 12) {
>> +        warn_report("hv-vendor-id truncated to 12 characters");
>> +        len = 12;
>>      }
>> +    memset(cpu->hyperv_vendor_id, 0, 12);
>> +    memcpy(cpu->hyperv_vendor_id, cpu->hyperv_vendor, len);
>
> Existing issue: hardcoded 12 as the size of hyperv_vendor_id here
> (compare with the code you add below using sizeof()).  I don't
> think this should hold the whole series, so it can be fixed in a
> follow up patch if necessary.
>

Agreed, adding a patch to change '12' to sizeof().

>>  
>>      /* 'Hv#1' interface identification*/
>>      cpu->hyperv_interface_id[0] = 0x31237648;
>> diff --git a/target/i386/kvm/kvm.c b/target/i386/kvm/kvm.c
>> index 7fe9f527103c..ab073a5e9c44 100644
>> --- a/target/i386/kvm/kvm.c
>> +++ b/target/i386/kvm/kvm.c
>> @@ -1215,6 +1215,11 @@ static int hyperv_handle_properties(CPUState *cs,
>>              cpu->hyperv_vendor_id[0] = c->ebx;
>>              cpu->hyperv_vendor_id[1] = c->ecx;
>>              cpu->hyperv_vendor_id[2] = c->edx;
>> +            cpu->hyperv_vendor = g_realloc(cpu->hyperv_vendor,
>> +                                           sizeof(cpu->hyperv_vendor_id) + 1);
>> +            memcpy(cpu->hyperv_vendor, cpu->hyperv_vendor_id,
>> +                   sizeof(cpu->hyperv_vendor_id));
>> +            cpu->hyperv_vendor[sizeof(cpu->hyperv_vendor_id)] = 0;
>
> I don't like having to do manual g_realloc() + memcpy() here
> (calling object_property_set_str() would be simpler), but I
> believe it will be easier to clean this up after this whole
> series is applied.

The problem here is that object_property_set_str() only works with
NULL-terminated strings and 'hyperv_vendor_id' doesn't have it (that's
why I explicitly do 'cpu->hyperv_vendor[sizeof(cpu->hyperv_vendor_id)] =
0'). We could've converted 'hyperv_vendor_id' to a char[13] array to
accomodate for '\0', but 

              cpu->hyperv_vendor_id[0] = c->ebx;
              cpu->hyperv_vendor_id[1] = c->ecx;
              cpu->hyperv_vendor_id[2] = c->edx;

assignments will have to be converted to something and this won't look
natural.

-- 
Vitaly