[Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Peter Maydell]

These patches add support to target-arm for '-cpu host'.
The general semantics are the same as for ppc and x86 (ie "whatever
the host kernel can support that looks basically like the host
CPU"), but the mechanism is a little different.

The kernel API (currently still proposed rather than implemented,
hence the RFC nature of this patchset) is that the existing
KVM_ARM_VCPU_INIT supports a new 'target' value KVM_ARM_TARGET_HOST,
which it treats as "whatever you are". On the userspace side,
we use this if the kernel supports it. If it doesn't then we
know the kernel must be an A15-on-A15 only one, and so can
safely implement '-cpu host' with KVM_ARM_TARGET_CORTEX_A15.

In order to find out details of the vCPU so we can correctly
set up our ARMCPU objects, we create a scratch VM and a single
cpu within it; we can then use the existing "read registers"
API to access the ID registers. We do this once at startup
when we register the new QOM type.

('-cpu host' is also going to be useful for AArch64 because
it gives us a path to providing KVM support without having
to implement a model of a specific v8 CPU and all its system
registers...)

Peter Maydell (2):
  target-arm: Don't hardcode KVM target CPU to be A15
  target-arm: Provide '-cpu host' when running KVM

 target-arm/helper.c |    6 ++
 target-arm/kvm.c    |  227 ++++++++++++++++++++++++++++++++++++++++++++++++++-
 2 files changed, 231 insertions(+), 2 deletions(-)

-- 
1.7.9.5

[Qemu-devel] [RFC 1/2] target-arm: Don't hardcode KVM target CPU to be A15

[Peter Maydell]

Instead of assuming that a KVM target CPU must always be a
Cortex-A15 and hardcoding this in kvm_arch_init_vcpu(), look
up the KVM_ARM_TARGET_* value based on the ARMCPU object
type. This is slightly overengineered for a single supported
CPU but provides a place to put support for future CPUs
and for "-cpu host".

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
---
 target-arm/kvm.c |   32 ++++++++++++++++++++++++++++++--
 1 file changed, 30 insertions(+), 2 deletions(-)

diff --git a/target-arm/kvm.c b/target-arm/kvm.c
index b92e00d..0e33efc 100644
--- a/target-arm/kvm.c
+++ b/target-arm/kvm.c
@@ -70,6 +70,32 @@ static int compare_u64(const void *a, const void *b)
     return *(uint64_t *)a - *(uint64_t *)b;
 }
 
+static bool kvm_arm_get_init_args(ARMCPU *cpu, struct kvm_vcpu_init *init)
+{
+    /* Fill in the kvm_vcpu_init struct appropriately for this CPU.
+     * Return true on success, false on failure (ie unsupported CPU).
+     */
+    Object *obj = OBJECT(cpu);
+    int i;
+    static const struct {
+        const char *name;
+        uint32_t target;
+    } kvm_cpus[] = {
+        { "cortex-a15-" TYPE_ARM_CPU, KVM_ARM_TARGET_CORTEX_A15 },
+    };
+
+    memset(init->features, 0, sizeof(init->features));
+
+    for (i = 0; i < ARRAY_SIZE(kvm_cpus); i++) {
+        if (object_dynamic_cast(obj, kvm_cpus[i].name)) {
+            init->target = kvm_cpus[i].target;
+            return true;
+        }
+    }
+
+    return false;
+}
+
 int kvm_arch_init_vcpu(CPUState *cs)
 {
     struct kvm_vcpu_init init;
@@ -80,8 +106,10 @@ int kvm_arch_init_vcpu(CPUState *cs)
     struct kvm_reg_list *rlp;
     ARMCPU *cpu = ARM_CPU(cs);
 
-    init.target = KVM_ARM_TARGET_CORTEX_A15;
-    memset(init.features, 0, sizeof(init.features));
+    if (!kvm_arm_get_init_args(cpu, &init)) {
+        fprintf(stderr, "KVM is not supported for this guest CPU type\n");
+        return -EINVAL;
+    }
     ret = kvm_vcpu_ioctl(cs, KVM_ARM_VCPU_INIT, &init);
     if (ret) {
         return ret;
-- 
1.7.9.5

[Qemu-devel] [RFC 2/2] target-arm: Provide '-cpu host' when running KVM

[Peter Maydell]

Implement '-cpu host' for ARM when we're using KVM, broadly
in line with other KVM-supporting architectures.

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
---
 target-arm/helper.c |    6 ++
 target-arm/kvm.c    |  195 +++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 201 insertions(+)

diff --git a/target-arm/helper.c b/target-arm/helper.c
index 4968391..6a81101 100644
--- a/target-arm/helper.c
+++ b/target-arm/helper.c
@@ -1573,6 +1573,12 @@ void arm_cpu_list(FILE *f, fprintf_function cpu_fprintf)
     (*cpu_fprintf)(f, "Available CPUs:\n");
     g_slist_foreach(list, arm_cpu_list_entry, &s);
     g_slist_free(list);
+#ifdef CONFIG_KVM
+    /* The 'host' CPU type is dynamically registered only if KVM is
+     * enabled, so we have to special-case it here:
+     */
+    (*cpu_fprintf)(f, "  host (only available in KVM mode)\n");
+#endif
 }
 
 void define_one_arm_cp_reg_with_opaque(ARMCPU *cpu,
diff --git a/target-arm/kvm.c b/target-arm/kvm.c
index 0e33efc..95573f4 100644
--- a/target-arm/kvm.c
+++ b/target-arm/kvm.c
@@ -36,12 +36,200 @@ const KVMCapabilityInfo kvm_arch_required_capabilities[] = {
     KVM_CAP_LAST_INFO
 };
 
+#define TYPE_ARM_HOST_CPU "host-" TYPE_ARM_CPU
+#define ARM_HOST_CPU_CLASS(klass) \
+    OBJECT_CLASS_CHECK(ARMHostCPUClass, (klass), TYPE_ARM_HOST_CPU)
+#define ARM_HOST_CPU_GET_CLASS(obj) \
+    OBJECT_GET_CLASS(ARMHostCPUClass, (obj), TYPE_ARM_HOST_CPU)
+
+typedef struct ARMHostCPUClass {
+    /*< private >*/
+    ARMCPUClass parent_class;
+    /*< public >*/
+
+    uint64_t features;
+    uint32_t target;
+} ARMHostCPUClass;
+
+static void kvm_arm_host_cpu_initfn(Object *obj)
+{
+    ARMHostCPUClass *ahcc = ARM_HOST_CPU_GET_CLASS(obj);
+    ARMCPU *cpu = ARM_CPU(obj);
+    CPUARMState *env = &cpu->env;
+
+    env->features = ahcc->features;
+}
+
+static inline void set_feature(uint64_t *features, int feature)
+{
+    *features |= 1ULL << feature;
+}
+
+static bool get_host_cpu_features(ARMHostCPUClass *ahcc)
+{
+    /* Identify the feature bits corresponding to the host CPU, and
+     * fill out the ARMHostCPUClass fields accordingly. To do this
+     * we have to create a scratch VM, create a single CPU inside it,
+     * and then query that CPU for the relevant ID registers.
+     */
+    int i, ret, kvmfd = -1, vmfd = -1, cpufd = -1;
+    uint32_t midr, id_pfr0, id_isar0, mvfr1;
+    uint64_t features = 0;
+    /* Any kernel old enough to not know about TARGET_HOST must be one which
+     * only supports guest A15 on host A15, so we fall back to trying A15.
+     */
+    static const uint32_t cpus_to_try[] = {
+        /* TODO add KVM_ARM_TARGET_HOST when it appears in the ABI */
+        KVM_ARM_TARGET_CORTEX_A15,
+    };
+    struct kvm_one_reg idregs[] = {
+        {
+            .id = KVM_REG_ARM | KVM_REG_SIZE_U32
+            | ENCODE_CP_REG(15, 0, 0, 0, 0, 0),
+            .addr = (uintptr_t)&midr,
+        },
+        {
+            .id = KVM_REG_ARM | KVM_REG_SIZE_U32
+            | ENCODE_CP_REG(15, 0, 0, 1, 0, 0),
+            .addr = (uintptr_t)&id_pfr0,
+        },
+        {
+            .id = KVM_REG_ARM | KVM_REG_SIZE_U32
+            | ENCODE_CP_REG(15, 0, 0, 2, 0, 0),
+            .addr = (uintptr_t)&id_isar0,
+        },
+        {
+            .id = KVM_REG_ARM | KVM_REG_SIZE_U32
+            | KVM_REG_ARM_VFP | KVM_REG_ARM_VFP_MVFR1,
+            .addr = (uintptr_t)&mvfr1,
+        },
+    };
+
+    kvmfd = qemu_open("/dev/kvm", O_RDWR);
+    if (kvmfd < 0) {
+        goto err;
+    }
+    vmfd = ioctl(kvmfd, KVM_CREATE_VM, 0);
+    if (vmfd < 0) {
+        goto err;
+    }
+    cpufd = ioctl(vmfd, KVM_CREATE_VCPU, 0);
+    if (cpufd < 0) {
+        goto err;
+    }
+
+    for (i = 0; i < ARRAY_SIZE(cpus_to_try); i++) {
+        struct kvm_vcpu_init init;
+
+        init.target = cpus_to_try[i];
+        memset(init.features, 0, sizeof(init.features));
+        ret = ioctl(cpufd, KVM_ARM_VCPU_INIT, &init);
+        if (ret >= 0) {
+            ahcc->target = init.target;
+            break;
+        }
+    }
+    if (ret < 0) {
+        goto err;
+    }
+
+    for (i = 0; i < ARRAY_SIZE(idregs); i++) {
+        ret = ioctl(cpufd, KVM_GET_ONE_REG, &idregs[i]);
+        if (ret) {
+            goto err;
+        }
+    }
+    close(cpufd);
+    close(vmfd);
+    close(kvmfd);
+
+    /* Now we've retrieved all the register information we can
+     * set the feature bits based on the ID register fields.
+     * We can assume any KVM supporting CPU is at least a v7
+     * with VFPv3, LPAE and the generic timers; this in turn implies
+     * most of the other feature bits, but a few must be tested.
+     */
+    set_feature(&features, ARM_FEATURE_V7);
+    set_feature(&features, ARM_FEATURE_VFP3);
+    set_feature(&features, ARM_FEATURE_LPAE);
+    set_feature(&features, ARM_FEATURE_GENERIC_TIMER);
+
+    switch (extract32(id_isar0, 24, 4)) {
+    case 1:
+        set_feature(&features, ARM_FEATURE_THUMB_DIV);
+        break;
+    case 2:
+        set_feature(&features, ARM_FEATURE_ARM_DIV);
+        set_feature(&features, ARM_FEATURE_THUMB_DIV);
+        break;
+    default:
+        break;
+    }
+
+    if (extract32(id_pfr0, 12, 4) == 1) {
+        set_feature(&features, ARM_FEATURE_THUMB2EE);
+    }
+    if (extract32(mvfr1, 20, 4) == 1) {
+        set_feature(&features, ARM_FEATURE_VFP_FP16);
+    }
+    if (extract32(mvfr1, 12, 4) == 1) {
+        set_feature(&features, ARM_FEATURE_NEON);
+    }
+    if (extract32(mvfr1, 28, 4) == 1) {
+        /* FMAC support implies VFPv4 */
+        set_feature(&features, ARM_FEATURE_VFP4);
+    }
+
+    ahcc->features = features;
+
+    return true;
+
+err:
+    if (cpufd >= 0) {
+        close(cpufd);
+    }
+    if (vmfd >= 0) {
+        close(vmfd);
+    }
+    if (kvmfd >= 0) {
+        close(kvmfd);
+    }
+
+    return false;
+}
+
+static void kvm_arm_host_cpu_class_init(ObjectClass *oc, void *data)
+{
+    ARMHostCPUClass *ahcc = ARM_HOST_CPU_CLASS(oc);
+
+    /* All we really need to set up for the 'host' CPU
+     * is the feature bits -- we rely on the fact that the
+     * various ID register values in ARMCPU are only used for
+     * TCG CPUs.
+     */
+    if (!get_host_cpu_features(ahcc)) {
+        fprintf(stderr, "Failed to retrieve host CPU features!\n");
+        abort();
+    }
+}
+
+static const TypeInfo host_arm_cpu_type_info = {
+    .name = TYPE_ARM_HOST_CPU,
+    .parent = TYPE_ARM_CPU,
+    .instance_init = kvm_arm_host_cpu_initfn,
+    .class_init = kvm_arm_host_cpu_class_init,
+    .class_size = sizeof(ARMHostCPUClass),
+};
+
 int kvm_arch_init(KVMState *s)
 {
     /* For ARM interrupt delivery is always asynchronous,
      * whether we are using an in-kernel VGIC or not.
      */
     kvm_async_interrupts_allowed = true;
+
+    type_register_static(&host_arm_cpu_type_info);
+
     return 0;
 }
 
@@ -86,6 +274,13 @@ static bool kvm_arm_get_init_args(ARMCPU *cpu, struct kvm_vcpu_init *init)
 
     memset(init->features, 0, sizeof(init->features));
 
+    if (object_dynamic_cast(obj, TYPE_ARM_HOST_CPU)) {
+        ARMHostCPUClass *ahcc = ARM_HOST_CPU_GET_CLASS(obj);
+
+        init->target = ahcc->target;
+        return true;
+    }
+
     for (i = 0; i < ARRAY_SIZE(kvm_cpus); i++) {
         if (object_dynamic_cast(obj, kvm_cpus[i].name)) {
             init->target = kvm_cpus[i].target;
-- 
1.7.9.5

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Alexander Graf]

On 13.08.2013, at 20:03, Peter Maydell wrote:

> These patches add support to target-arm for '-cpu host'.
> The general semantics are the same as for ppc and x86 (ie "whatever
> the host kernel can support that looks basically like the host
> CPU"), but the mechanism is a little different.
> 
> The kernel API (currently still proposed rather than implemented,
> hence the RFC nature of this patchset) is that the existing
> KVM_ARM_VCPU_INIT supports a new 'target' value KVM_ARM_TARGET_HOST,
> which it treats as "whatever you are". On the userspace side,
> we use this if the kernel supports it. If it doesn't then we
> know the kernel must be an A15-on-A15 only one, and so can
> safely implement '-cpu host' with KVM_ARM_TARGET_CORTEX_A15.

How do you know what core specific registers QEMU can expect from this particular CPU? Imagine ARM changes the MMU implementation in the next ARMv8 CPU. With -cpu host you wouldn't be able to support gdbstub anymore, because you wouldn't know whether the MMU format is the old or the new one.

I really prefer the "probe the host, match it fuzzily with our own list, pick a specific own core" way of doing -cpu host. Then it's really only ever an alias and you don't have to worry about all these odd special cases.


Alex

> 
> In order to find out details of the vCPU so we can correctly
> set up our ARMCPU objects, we create a scratch VM and a single
> cpu within it; we can then use the existing "read registers"
> API to access the ID registers. We do this once at startup
> when we register the new QOM type.
> 
> ('-cpu host' is also going to be useful for AArch64 because
> it gives us a path to providing KVM support without having
> to implement a model of a specific v8 CPU and all its system
> registers...)
> 
> Peter Maydell (2):
>  target-arm: Don't hardcode KVM target CPU to be A15
>  target-arm: Provide '-cpu host' when running KVM
> 
> target-arm/helper.c |    6 ++
> target-arm/kvm.c    |  227 ++++++++++++++++++++++++++++++++++++++++++++++++++-
> 2 files changed, 231 insertions(+), 2 deletions(-)
> 
> -- 
> 1.7.9.5
> 
> _______________________________________________
> kvmarm mailing list
> kvmarm@lists.cs.columbia.edu
> https://lists.cs.columbia.edu/cucslists/listinfo/kvmarm

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Marc Zyngier]

On 2013-08-14 07:32, Alexander Graf wrote:
> On 13.08.2013, at 20:03, Peter Maydell wrote:
>
>> These patches add support to target-arm for '-cpu host'.
>> The general semantics are the same as for ppc and x86 (ie "whatever
>> the host kernel can support that looks basically like the host
>> CPU"), but the mechanism is a little different.
>>
>> The kernel API (currently still proposed rather than implemented,
>> hence the RFC nature of this patchset) is that the existing
>> KVM_ARM_VCPU_INIT supports a new 'target' value KVM_ARM_TARGET_HOST,
>> which it treats as "whatever you are". On the userspace side,
>> we use this if the kernel supports it. If it doesn't then we
>> know the kernel must be an A15-on-A15 only one, and so can
>> safely implement '-cpu host' with KVM_ARM_TARGET_CORTEX_A15.
>
> How do you know what core specific registers QEMU can expect from
> this particular CPU? Imagine ARM changes the MMU implementation in 
> the
> next ARMv8 CPU. With -cpu host you wouldn't be able to support 
> gdbstub
> anymore, because you wouldn't know whether the MMU format is the old
> or the new one.

Well, we're already there. A15 supports both classic page tables and 
LPAE.

You can discover which one is being used by inspecting the guest, and 
I'd
fully expect future versions of the architecture to expose bits in the 
ID
registers indicating the various supported extensions.

         M.
-- 
Fast, cheap, reliable. Pick two.

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Alexander Graf]

On 14.08.2013, at 10:11, Marc Zyngier wrote:

> On 2013-08-14 07:32, Alexander Graf wrote:
>> On 13.08.2013, at 20:03, Peter Maydell wrote:
>> 
>>> These patches add support to target-arm for '-cpu host'.
>>> The general semantics are the same as for ppc and x86 (ie "whatever
>>> the host kernel can support that looks basically like the host
>>> CPU"), but the mechanism is a little different.
>>> 
>>> The kernel API (currently still proposed rather than implemented,
>>> hence the RFC nature of this patchset) is that the existing
>>> KVM_ARM_VCPU_INIT supports a new 'target' value KVM_ARM_TARGET_HOST,
>>> which it treats as "whatever you are". On the userspace side,
>>> we use this if the kernel supports it. If it doesn't then we
>>> know the kernel must be an A15-on-A15 only one, and so can
>>> safely implement '-cpu host' with KVM_ARM_TARGET_CORTEX_A15.
>> 
>> How do you know what core specific registers QEMU can expect from
>> this particular CPU? Imagine ARM changes the MMU implementation in the
>> next ARMv8 CPU. With -cpu host you wouldn't be able to support gdbstub
>> anymore, because you wouldn't know whether the MMU format is the old
>> or the new one.
> 
> Well, we're already there. A15 supports both classic page tables and LPAE.
> 
> You can discover which one is being used by inspecting the guest, and I'd
> fully expect future versions of the architecture to expose bits in the ID
> registers indicating the various supported extensions.

Can you fetch those from user space? If so, we should probably create a "host" CPU type class on the fly from the host's ID registers that adjusts itself to the correct callbacks.


Alex

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Marc Zyngier]

On 2013-08-14 09:16, Alexander Graf wrote:
> On 14.08.2013, at 10:11, Marc Zyngier wrote:
>
>> On 2013-08-14 07:32, Alexander Graf wrote:
>>> On 13.08.2013, at 20:03, Peter Maydell wrote:
>>>
>>>> These patches add support to target-arm for '-cpu host'.
>>>> The general semantics are the same as for ppc and x86 (ie 
>>>> "whatever
>>>> the host kernel can support that looks basically like the host
>>>> CPU"), but the mechanism is a little different.
>>>>
>>>> The kernel API (currently still proposed rather than implemented,
>>>> hence the RFC nature of this patchset) is that the existing
>>>> KVM_ARM_VCPU_INIT supports a new 'target' value 
>>>> KVM_ARM_TARGET_HOST,
>>>> which it treats as "whatever you are". On the userspace side,
>>>> we use this if the kernel supports it. If it doesn't then we
>>>> know the kernel must be an A15-on-A15 only one, and so can
>>>> safely implement '-cpu host' with KVM_ARM_TARGET_CORTEX_A15.
>>>
>>> How do you know what core specific registers QEMU can expect from
>>> this particular CPU? Imagine ARM changes the MMU implementation in 
>>> the
>>> next ARMv8 CPU. With -cpu host you wouldn't be able to support 
>>> gdbstub
>>> anymore, because you wouldn't know whether the MMU format is the 
>>> old
>>> or the new one.
>>
>> Well, we're already there. A15 supports both classic page tables and 
>> LPAE.
>>
>> You can discover which one is being used by inspecting the guest, 
>> and I'd
>> fully expect future versions of the architecture to expose bits in 
>> the ID
>> registers indicating the various supported extensions.
>
> Can you fetch those from user space? If so, we should probably create
> a "host" CPU type class on the fly from the host's ID registers that
> adjusts itself to the correct callbacks.

All the ID registers are available to userspace as part of the one-reg 
API. What the guest actually uses (for example classic vs LPAE) can be 
found out by looking at other registers (in this example SCTRL) that 
have to be exported anyway (at least for VM migration).

         M.
-- 
Fast, cheap, reliable. Pick two.

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Alexander Graf]

On 14.08.2013, at 10:27, Marc Zyngier wrote:

> On 2013-08-14 09:16, Alexander Graf wrote:
>> On 14.08.2013, at 10:11, Marc Zyngier wrote:
>> 
>>> On 2013-08-14 07:32, Alexander Graf wrote:
>>>> On 13.08.2013, at 20:03, Peter Maydell wrote:
>>>> 
>>>>> These patches add support to target-arm for '-cpu host'.
>>>>> The general semantics are the same as for ppc and x86 (ie "whatever
>>>>> the host kernel can support that looks basically like the host
>>>>> CPU"), but the mechanism is a little different.
>>>>> 
>>>>> The kernel API (currently still proposed rather than implemented,
>>>>> hence the RFC nature of this patchset) is that the existing
>>>>> KVM_ARM_VCPU_INIT supports a new 'target' value KVM_ARM_TARGET_HOST,
>>>>> which it treats as "whatever you are". On the userspace side,
>>>>> we use this if the kernel supports it. If it doesn't then we
>>>>> know the kernel must be an A15-on-A15 only one, and so can
>>>>> safely implement '-cpu host' with KVM_ARM_TARGET_CORTEX_A15.
>>>> 
>>>> How do you know what core specific registers QEMU can expect from
>>>> this particular CPU? Imagine ARM changes the MMU implementation in the
>>>> next ARMv8 CPU. With -cpu host you wouldn't be able to support gdbstub
>>>> anymore, because you wouldn't know whether the MMU format is the old
>>>> or the new one.
>>> 
>>> Well, we're already there. A15 supports both classic page tables and LPAE.
>>> 
>>> You can discover which one is being used by inspecting the guest, and I'd
>>> fully expect future versions of the architecture to expose bits in the ID
>>> registers indicating the various supported extensions.
>> 
>> Can you fetch those from user space? If so, we should probably create
>> a "host" CPU type class on the fly from the host's ID registers that
>> adjusts itself to the correct callbacks.
> 
> All the ID registers are available to userspace as part of the one-reg API. What the guest actually uses (for example classic vs LPAE) can be found out by looking at other registers (in this example SCTRL) that have to be exported anyway (at least for VM migration).

So can we access those even when the vcpu hasn't been init'ed yet? If so, how about the following flow of things for -cpu host:

  - QEMU fetches host target type via ioctl (how will this work for big-little?)
  - QEMU fetches ID registers, feature flags, etc via one-reg (or are they also available via auxv?), puts them into its internal "this is my cpu type" fields for the cpu object / class.
  - QEMU calls VCPU_INIT with the type it fetched via ioctl.

Or would it make sense to model it like it's done on x86 where we also filter feature flags in QEMU and KVM to make sure we only expose a compatible subset?


Alex

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Peter Maydell]

On 14 August 2013 07:32, Alexander Graf <agraf@suse.de> wrote:
>
> On 13.08.2013, at 20:03, Peter Maydell wrote:
>
>> These patches add support to target-arm for '-cpu host'.
>> The general semantics are the same as for ppc and x86 (ie "whatever
>> the host kernel can support that looks basically like the host
>> CPU"), but the mechanism is a little different.
>>
>> The kernel API (currently still proposed rather than implemented,
>> hence the RFC nature of this patchset) is that the existing
>> KVM_ARM_VCPU_INIT supports a new 'target' value KVM_ARM_TARGET_HOST,
>> which it treats as "whatever you are". On the userspace side,
>> we use this if the kernel supports it. If it doesn't then we
>> know the kernel must be an A15-on-A15 only one, and so can
>> safely implement '-cpu host' with KVM_ARM_TARGET_CORTEX_A15.
>
> How do you know what core specific registers QEMU can expect from
> this particular CPU?

For 99% of them we simply don't care...

> Imagine ARM changes the MMU implementation in the next ARMv8 CPU.
> With -cpu host you wouldn't be able to support gdbstub anymore,
> because you wouldn't know whether the MMU format is the old or
> the new one.

...and for the other 1% we can add a bit to the VCPU_INIT features
field saying "this QEMU can handle CPUs with $weird-new-thing"
and have the kernel fail CPU creation otherwise.

> I really prefer the "probe the host, match it fuzzily with our
> own list, pick a specific own core" way of doing -cpu host. Then
> it's really only ever an alias and you don't have to worry about
> all these odd special cases.

I really don't want to be trying to model five hundred different
CPUs inside QEMU half of which I probably won't have the
documentation for. That is just going to mean "KVM+QEMU won't
work on a lot of hardware when there is no reason it shouldn't".

-- PMM

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Peter Maydell]

On 14 August 2013 09:46, Alexander Graf <agraf@suse.de> wrote:
> So can we access those even when the vcpu hasn't been init'ed yet? If so, how about the following flow of things for -cpu host:
>
>   - QEMU fetches host target type via ioctl (how will this work for big-little?)
>   - QEMU fetches ID registers, feature flags, etc via one-reg (or are they also available via auxv?), puts them into its internal "this is my cpu type" fields for the cpu object / class.
>   - QEMU calls VCPU_INIT with the type it fetched via ioctl.

Curiously enough this is pretty much exactly what this patch
actually does (except we don't bother to actually stash the ID
register values because we know they'll never be used).

-- PMM

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Alexander Graf]

On 14.08.2013, at 11:07, Peter Maydell wrote:

> On 14 August 2013 09:46, Alexander Graf <agraf@suse.de> wrote:
>> So can we access those even when the vcpu hasn't been init'ed yet? If so, how about the following flow of things for -cpu host:
>> 
>>  - QEMU fetches host target type via ioctl (how will this work for big-little?)
>>  - QEMU fetches ID registers, feature flags, etc via one-reg (or are they also available via auxv?), puts them into its internal "this is my cpu type" fields for the cpu object / class.
>>  - QEMU calls VCPU_INIT with the type it fetched via ioctl.
> 
> Curiously enough this is pretty much exactly what this patch
> actually does (except we don't bother to actually stash the ID
> register values because we know they'll never be used).

Heh, you mean I should read patches after 0/x too? :)

You're right, the main difference is that KVM doesn't have any idea what a "host" style CPU is. It only knows how to report to QEMU what the current host CPU would be, so that anything from VCPU_INIT onwards is 100% identical regardless of whether the user said -cpu host or -cpu xxx.

I'm still puzzled on how this will work with BIG.little btw.


Alex

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Peter Maydell]

On 14 August 2013 10:11, Alexander Graf <agraf@suse.de> wrote:
> You're right, the main difference is that KVM doesn't have any
> idea what a "host" style CPU is. It only knows how to report to QEMU
> what the current host CPU would be, so that anything from VCPU_INIT
> onwards is 100% identical regardless of whether the user said
> -cpu host or -cpu xxx.
>
> I'm still puzzled on how this will work with BIG.little btw.

The rough idea is that for BIG.little the kernel must trap the
ID registers at least (so that the vcpu seems consistent to the
guest whether it's running on the big or the little core). For
"-cpu host" the guest would see whatever is the most low-overhead
for the kernel to provide (ie assuming the big and little CPUs
are roughly-similar you could make -cpu host provide something
that looks to the guest like the big CPU and don't have to trap
quite as much as you would for providing a vcpu that wasn't the
same as either the big or little one).

-- PMM

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Alexander Graf]

On 14.08.2013, at 11:23, Peter Maydell wrote:

> On 14 August 2013 10:11, Alexander Graf <agraf@suse.de> wrote:
>> You're right, the main difference is that KVM doesn't have any
>> idea what a "host" style CPU is. It only knows how to report to QEMU
>> what the current host CPU would be, so that anything from VCPU_INIT
>> onwards is 100% identical regardless of whether the user said
>> -cpu host or -cpu xxx.
>> 
>> I'm still puzzled on how this will work with BIG.little btw.
> 
> The rough idea is that for BIG.little the kernel must trap the
> ID registers at least (so that the vcpu seems consistent to the
> guest whether it's running on the big or the little core). For
> "-cpu host" the guest would see whatever is the most low-overhead
> for the kernel to provide (ie assuming the big and little CPUs
> are roughly-similar you could make -cpu host provide something
> that looks to the guest like the big CPU and don't have to trap
> quite as much as you would for providing a vcpu that wasn't the
> same as either the big or little one).

So -cpu host in this case wouldn't actually expose the host CPU 1:1, but instead a cortex-a15 even when it's run on an a7 BIG.little core. I see.


Alex

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Christoffer Dall]

On Wed, Aug 14, 2013 at 11:30:46AM +0200, Alexander Graf wrote:
> 
> On 14.08.2013, at 11:23, Peter Maydell wrote:
> 
> > On 14 August 2013 10:11, Alexander Graf <agraf@suse.de> wrote:
> >> You're right, the main difference is that KVM doesn't have any
> >> idea what a "host" style CPU is. It only knows how to report to QEMU
> >> what the current host CPU would be, so that anything from VCPU_INIT
> >> onwards is 100% identical regardless of whether the user said
> >> -cpu host or -cpu xxx.
> >> 
> >> I'm still puzzled on how this will work with BIG.little btw.
> > 
> > The rough idea is that for BIG.little the kernel must trap the
> > ID registers at least (so that the vcpu seems consistent to the
> > guest whether it's running on the big or the little core). For
> > "-cpu host" the guest would see whatever is the most low-overhead
> > for the kernel to provide (ie assuming the big and little CPUs
> > are roughly-similar you could make -cpu host provide something
> > that looks to the guest like the big CPU and don't have to trap
> > quite as much as you would for providing a vcpu that wasn't the
> > same as either the big or little one).
> 
> So -cpu host in this case wouldn't actually expose the host CPU 1:1, but instead a cortex-a15 even when it's run on an a7 BIG.little core. I see.
> 
Yes, from the discussion we've had the whole picture just becomes to
blurry when you start presenting multiple different CPUs to the guest
and there's really no need to that I'm aware of.

In fact the -cpu host case fits quite nicely with this state of mind;
the kernel is free to decide based on the specific hardware and config
on which it's running how to handle VMs on BL.

-Christoffer

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Alexander Graf]

On 14.08.2013, at 19:26, Christoffer Dall wrote:

> On Wed, Aug 14, 2013 at 11:30:46AM +0200, Alexander Graf wrote:
>> 
>> On 14.08.2013, at 11:23, Peter Maydell wrote:
>> 
>>> On 14 August 2013 10:11, Alexander Graf <agraf@suse.de> wrote:
>>>> You're right, the main difference is that KVM doesn't have any
>>>> idea what a "host" style CPU is. It only knows how to report to QEMU
>>>> what the current host CPU would be, so that anything from VCPU_INIT
>>>> onwards is 100% identical regardless of whether the user said
>>>> -cpu host or -cpu xxx.
>>>> 
>>>> I'm still puzzled on how this will work with BIG.little btw.
>>> 
>>> The rough idea is that for BIG.little the kernel must trap the
>>> ID registers at least (so that the vcpu seems consistent to the
>>> guest whether it's running on the big or the little core). For
>>> "-cpu host" the guest would see whatever is the most low-overhead
>>> for the kernel to provide (ie assuming the big and little CPUs
>>> are roughly-similar you could make -cpu host provide something
>>> that looks to the guest like the big CPU and don't have to trap
>>> quite as much as you would for providing a vcpu that wasn't the
>>> same as either the big or little one).
>> 
>> So -cpu host in this case wouldn't actually expose the host CPU 1:1, but instead a cortex-a15 even when it's run on an a7 BIG.little core. I see.
>> 
> Yes, from the discussion we've had the whole picture just becomes to
> blurry when you start presenting multiple different CPUs to the guest
> and there's really no need to that I'm aware of.
> 
> In fact the -cpu host case fits quite nicely with this state of mind;
> the kernel is free to decide based on the specific hardware and config
> on which it's running how to handle VMs on BL.

So why not have a vm ioctl to fetch the "best match" vcpu type? I don't like the idea of adding any awareness of a "host" type to the normal vcpu creation process.


Alex

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Christoffer Dall]

On Wed, Aug 14, 2013 at 07:31:59PM +0200, Alexander Graf wrote:
> 
> On 14.08.2013, at 19:26, Christoffer Dall wrote:
> 
> > On Wed, Aug 14, 2013 at 11:30:46AM +0200, Alexander Graf wrote:
> >> 
> >> On 14.08.2013, at 11:23, Peter Maydell wrote:
> >> 
> >>> On 14 August 2013 10:11, Alexander Graf <agraf@suse.de> wrote:
> >>>> You're right, the main difference is that KVM doesn't have any
> >>>> idea what a "host" style CPU is. It only knows how to report to QEMU
> >>>> what the current host CPU would be, so that anything from VCPU_INIT
> >>>> onwards is 100% identical regardless of whether the user said
> >>>> -cpu host or -cpu xxx.
> >>>> 
> >>>> I'm still puzzled on how this will work with BIG.little btw.
> >>> 
> >>> The rough idea is that for BIG.little the kernel must trap the
> >>> ID registers at least (so that the vcpu seems consistent to the
> >>> guest whether it's running on the big or the little core). For
> >>> "-cpu host" the guest would see whatever is the most low-overhead
> >>> for the kernel to provide (ie assuming the big and little CPUs
> >>> are roughly-similar you could make -cpu host provide something
> >>> that looks to the guest like the big CPU and don't have to trap
> >>> quite as much as you would for providing a vcpu that wasn't the
> >>> same as either the big or little one).
> >> 
> >> So -cpu host in this case wouldn't actually expose the host CPU 1:1, but instead a cortex-a15 even when it's run on an a7 BIG.little core. I see.
> >> 
> > Yes, from the discussion we've had the whole picture just becomes to
> > blurry when you start presenting multiple different CPUs to the guest
> > and there's really no need to that I'm aware of.
> > 
> > In fact the -cpu host case fits quite nicely with this state of mind;
> > the kernel is free to decide based on the specific hardware and config
> > on which it's running how to handle VMs on BL.
> 
> So why not have a vm ioctl to fetch the "best match" vcpu type? I don't like the idea of adding any awareness of a "host" type to the normal vcpu creation process.
> 
> 
That's actually what I suggested initially.  I'm not really a QEMU
expert, but I think Peter already answered this question: he doesn't
want to support hundreds of CPU models in QEMU just to be able to run
KVM when it's not necessary.

If his argument holds in that you can support -cpu host without having a
model for that specific cpu in QEMU, then indeed it is a strong
argument, and we have the problem with ARMv8 already, and this goes a
long way to solve that. No?

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Alexander Graf]

On 14.08.2013, at 19:39, Christoffer Dall wrote:

> On Wed, Aug 14, 2013 at 07:31:59PM +0200, Alexander Graf wrote:
>> 
>> On 14.08.2013, at 19:26, Christoffer Dall wrote:
>> 
>>> On Wed, Aug 14, 2013 at 11:30:46AM +0200, Alexander Graf wrote:
>>>> 
>>>> On 14.08.2013, at 11:23, Peter Maydell wrote:
>>>> 
>>>>> On 14 August 2013 10:11, Alexander Graf <agraf@suse.de> wrote:
>>>>>> You're right, the main difference is that KVM doesn't have any
>>>>>> idea what a "host" style CPU is. It only knows how to report to QEMU
>>>>>> what the current host CPU would be, so that anything from VCPU_INIT
>>>>>> onwards is 100% identical regardless of whether the user said
>>>>>> -cpu host or -cpu xxx.
>>>>>> 
>>>>>> I'm still puzzled on how this will work with BIG.little btw.
>>>>> 
>>>>> The rough idea is that for BIG.little the kernel must trap the
>>>>> ID registers at least (so that the vcpu seems consistent to the
>>>>> guest whether it's running on the big or the little core). For
>>>>> "-cpu host" the guest would see whatever is the most low-overhead
>>>>> for the kernel to provide (ie assuming the big and little CPUs
>>>>> are roughly-similar you could make -cpu host provide something
>>>>> that looks to the guest like the big CPU and don't have to trap
>>>>> quite as much as you would for providing a vcpu that wasn't the
>>>>> same as either the big or little one).
>>>> 
>>>> So -cpu host in this case wouldn't actually expose the host CPU 1:1, but instead a cortex-a15 even when it's run on an a7 BIG.little core. I see.
>>>> 
>>> Yes, from the discussion we've had the whole picture just becomes to
>>> blurry when you start presenting multiple different CPUs to the guest
>>> and there's really no need to that I'm aware of.
>>> 
>>> In fact the -cpu host case fits quite nicely with this state of mind;
>>> the kernel is free to decide based on the specific hardware and config
>>> on which it's running how to handle VMs on BL.
>> 
>> So why not have a vm ioctl to fetch the "best match" vcpu type? I don't like the idea of adding any awareness of a "host" type to the normal vcpu creation process.
>> 
>> 
> That's actually what I suggested initially.  I'm not really a QEMU
> expert, but I think Peter already answered this question: he doesn't
> want to support hundreds of CPU models in QEMU just to be able to run
> KVM when it's not necessary.
> 
> If his argument holds in that you can support -cpu host without having a
> model for that specific cpu in QEMU, then indeed it is a strong
> argument, and we have the problem with ARMv8 already, and this goes a
> long way to solve that. No?

That's up for QEMU to decide. With the "fetch and push" model we can support both flavors from user space. It also makes the kernel side more reproducible and obvious. There's simply no way to add hacks like "If I'm a -cpu host type do xxx" in KVM, because KVM never knows that it is running -cpu host.


Alex

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Peter Maydell]

On 14 August 2013 18:31, Alexander Graf <agraf@suse.de> wrote:
> So why not have a vm ioctl to fetch the "best match" vcpu type?

I don't object to that particularly, but
(a) we're not going to use the extra flexibility because
    the only thing we'll do is just bounce the answer
    back at the kernel
(b) it requires an actual new ioctl rather than just
    defining another supported value for the existing one
(c) the code is less neat if you have to do "ask kernel
    for best-match, if it has it use it otherwise fall
    back to [small list of legacy cpus]" than if you
    just have "try best-match/legacy1/legacy2/legacy3".

If the consensus is that a 'fetch best match' ioctl
is the preferred API that's fine.

-- PMM

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Alexander Graf]

On 14.08.2013, at 20:11, Peter Maydell wrote:

> On 14 August 2013 18:31, Alexander Graf <agraf@suse.de> wrote:
>> So why not have a vm ioctl to fetch the "best match" vcpu type?
> 
> I don't object to that particularly, but
> (a) we're not going to use the extra flexibility because
>    the only thing we'll do is just bounce the answer
>    back at the kernel

It makes the API more robust to abuse. I think that's a good thing.

> (b) it requires an actual new ioctl rather than just
>    defining another supported value for the existing one

Yup :).

> (c) the code is less neat if you have to do "ask kernel
>    for best-match, if it has it use it otherwise fall
>    back to [small list of legacy cpus]" than if you
>    just have "try best-match/legacy1/legacy2/legacy3".

I don't understand this point. Could you please elaborate?


Alex

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Christoffer Dall]

On Wed, Aug 14, 2013 at 07:44:25PM +0200, Alexander Graf wrote:
> 
> On 14.08.2013, at 19:39, Christoffer Dall wrote:
> 
> > On Wed, Aug 14, 2013 at 07:31:59PM +0200, Alexander Graf wrote:
> >> 
> >> On 14.08.2013, at 19:26, Christoffer Dall wrote:
> >> 
> >>> On Wed, Aug 14, 2013 at 11:30:46AM +0200, Alexander Graf wrote:
> >>>> 
> >>>> On 14.08.2013, at 11:23, Peter Maydell wrote:
> >>>> 
> >>>>> On 14 August 2013 10:11, Alexander Graf <agraf@suse.de> wrote:
> >>>>>> You're right, the main difference is that KVM doesn't have any
> >>>>>> idea what a "host" style CPU is. It only knows how to report to QEMU
> >>>>>> what the current host CPU would be, so that anything from VCPU_INIT
> >>>>>> onwards is 100% identical regardless of whether the user said
> >>>>>> -cpu host or -cpu xxx.
> >>>>>> 
> >>>>>> I'm still puzzled on how this will work with BIG.little btw.
> >>>>> 
> >>>>> The rough idea is that for BIG.little the kernel must trap the
> >>>>> ID registers at least (so that the vcpu seems consistent to the
> >>>>> guest whether it's running on the big or the little core). For
> >>>>> "-cpu host" the guest would see whatever is the most low-overhead
> >>>>> for the kernel to provide (ie assuming the big and little CPUs
> >>>>> are roughly-similar you could make -cpu host provide something
> >>>>> that looks to the guest like the big CPU and don't have to trap
> >>>>> quite as much as you would for providing a vcpu that wasn't the
> >>>>> same as either the big or little one).
> >>>> 
> >>>> So -cpu host in this case wouldn't actually expose the host CPU 1:1, but instead a cortex-a15 even when it's run on an a7 BIG.little core. I see.
> >>>> 
> >>> Yes, from the discussion we've had the whole picture just becomes to
> >>> blurry when you start presenting multiple different CPUs to the guest
> >>> and there's really no need to that I'm aware of.
> >>> 
> >>> In fact the -cpu host case fits quite nicely with this state of mind;
> >>> the kernel is free to decide based on the specific hardware and config
> >>> on which it's running how to handle VMs on BL.
> >> 
> >> So why not have a vm ioctl to fetch the "best match" vcpu type? I don't like the idea of adding any awareness of a "host" type to the normal vcpu creation process.
> >> 
> >> 
> > That's actually what I suggested initially.  I'm not really a QEMU
> > expert, but I think Peter already answered this question: he doesn't
> > want to support hundreds of CPU models in QEMU just to be able to run
> > KVM when it's not necessary.
> > 
> > If his argument holds in that you can support -cpu host without having a
> > model for that specific cpu in QEMU, then indeed it is a strong
> > argument, and we have the problem with ARMv8 already, and this goes a
> > long way to solve that. No?
> 
> That's up for QEMU to decide. With the "fetch and push" model we can support both flavors from user space. It also makes the kernel side more reproducible and obvious. There's simply no way to add hacks like "If I'm a -cpu host type do xxx" in KVM, because KVM never knows that it is running -cpu host.
> 
Do we have historical examples of this knowledge being abused inside the
kernel for other archs?  If not, can we come up with a technical
scenario where it may happen on ARM?

Also, not really sure if such code should be controlled through the user
space API; ideally we would catch bad coding behavior in the kernel
during code review.

The only reason I originally suggested the "fetch and push" model was
that I thought user space would need to know the specific CPU model for
things to work and possibly for things like debugging and migration, but
since I have been almost convinced otherwise, I don't see any real
technical arguments for not adding -cpu host support in the kernel side.

Note that this doesn't prevent us from adding an IOCTL later that gives
you the host CPU type in KVM terminology if we find it useful.  But, I
think the reduced headache with ARMv8 right now is a good argument to
proceed with Peter's RFC and kernel support for same.

-Christoffer

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Alexander Graf]

On 14.08.2013, at 20:18, Christoffer Dall wrote:

> On Wed, Aug 14, 2013 at 07:44:25PM +0200, Alexander Graf wrote:
>> 
>> On 14.08.2013, at 19:39, Christoffer Dall wrote:
>> 
>>> On Wed, Aug 14, 2013 at 07:31:59PM +0200, Alexander Graf wrote:
>>>> 
>>>> On 14.08.2013, at 19:26, Christoffer Dall wrote:
>>>> 
>>>>> On Wed, Aug 14, 2013 at 11:30:46AM +0200, Alexander Graf wrote:
>>>>>> 
>>>>>> On 14.08.2013, at 11:23, Peter Maydell wrote:
>>>>>> 
>>>>>>> On 14 August 2013 10:11, Alexander Graf <agraf@suse.de> wrote:
>>>>>>>> You're right, the main difference is that KVM doesn't have any
>>>>>>>> idea what a "host" style CPU is. It only knows how to report to QEMU
>>>>>>>> what the current host CPU would be, so that anything from VCPU_INIT
>>>>>>>> onwards is 100% identical regardless of whether the user said
>>>>>>>> -cpu host or -cpu xxx.
>>>>>>>> 
>>>>>>>> I'm still puzzled on how this will work with BIG.little btw.
>>>>>>> 
>>>>>>> The rough idea is that for BIG.little the kernel must trap the
>>>>>>> ID registers at least (so that the vcpu seems consistent to the
>>>>>>> guest whether it's running on the big or the little core). For
>>>>>>> "-cpu host" the guest would see whatever is the most low-overhead
>>>>>>> for the kernel to provide (ie assuming the big and little CPUs
>>>>>>> are roughly-similar you could make -cpu host provide something
>>>>>>> that looks to the guest like the big CPU and don't have to trap
>>>>>>> quite as much as you would for providing a vcpu that wasn't the
>>>>>>> same as either the big or little one).
>>>>>> 
>>>>>> So -cpu host in this case wouldn't actually expose the host CPU 1:1, but instead a cortex-a15 even when it's run on an a7 BIG.little core. I see.
>>>>>> 
>>>>> Yes, from the discussion we've had the whole picture just becomes to
>>>>> blurry when you start presenting multiple different CPUs to the guest
>>>>> and there's really no need to that I'm aware of.
>>>>> 
>>>>> In fact the -cpu host case fits quite nicely with this state of mind;
>>>>> the kernel is free to decide based on the specific hardware and config
>>>>> on which it's running how to handle VMs on BL.
>>>> 
>>>> So why not have a vm ioctl to fetch the "best match" vcpu type? I don't like the idea of adding any awareness of a "host" type to the normal vcpu creation process.
>>>> 
>>>> 
>>> That's actually what I suggested initially.  I'm not really a QEMU
>>> expert, but I think Peter already answered this question: he doesn't
>>> want to support hundreds of CPU models in QEMU just to be able to run
>>> KVM when it's not necessary.
>>> 
>>> If his argument holds in that you can support -cpu host without having a
>>> model for that specific cpu in QEMU, then indeed it is a strong
>>> argument, and we have the problem with ARMv8 already, and this goes a
>>> long way to solve that. No?
>> 
>> That's up for QEMU to decide. With the "fetch and push" model we can support both flavors from user space. It also makes the kernel side more reproducible and obvious. There's simply no way to add hacks like "If I'm a -cpu host type do xxx" in KVM, because KVM never knows that it is running -cpu host.
>> 
> Do we have historical examples of this knowledge being abused inside the
> kernel for other archs?  If not, can we come up with a technical
> scenario where it may happen on ARM?

if (cpu == host_cpu) {
  vgic_version = get_host_vgic_version();
}

would be an example :). Everything -cpu host does has to be reproducible without -cpu host, otherwise our compatibility layering is broken. So why not model the API like it from the beginning?

> 
> Also, not really sure if such code should be controlled through the user
> space API; ideally we would catch bad coding behavior in the kernel
> during code review.
> 
> The only reason I originally suggested the "fetch and push" model was
> that I thought user space would need to know the specific CPU model for
> things to work and possibly for things like debugging and migration, but
> since I have been almost convinced otherwise, I don't see any real
> technical arguments for not adding -cpu host support in the kernel side.
> 
> Note that this doesn't prevent us from adding an IOCTL later that gives
> you the host CPU type in KVM terminology if we find it useful.  But, I
> think the reduced headache with ARMv8 right now is a good argument to
> proceed with Peter's RFC and kernel support for same.

There's really almost no difference from the QEMU point of view if Peter choses to implement it the way he does today. He can ask the kernel for the vcpu target and pass that exact number back into the kernel.


Alex

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Peter Maydell]

On 14 August 2013 19:15, Alexander Graf <agraf@suse.de> wrote:
> On 14.08.2013, at 20:11, Peter Maydell wrote:
>> (c) the code is less neat if you have to do "ask kernel
>>    for best-match, if it has it use it otherwise fall
>>    back to [small list of legacy cpus]" than if you
>>    just have "try best-match/legacy1/legacy2/legacy3".
>
> I don't understand this point. Could you please elaborate?

See the code in the patch that basically says "try each
value in this array until one works". For v7 the only
two things we need to try are 'host' and 'a15' (because
any kernel which doesn't know about 'host' must be an
a15-only one); for v8 we have three legacy entries in
the array. If you have to ask the kernel for the best-match
then it's a special case before hand rather than just another
entry in the list.

This is frankly a pretty weak argument though; I just
liked that bit of code when I wrote it :-)

-- PMM

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Peter Maydell]

On 14 August 2013 19:21, Alexander Graf <agraf@suse.de> wrote:
> Everything -cpu host does has to be reproducible without -cpu host,

This is in general not going to be possible (the obvious case
being "QEMU doesn't know about the host CPU type at all", or
in minor variants "QEMU knows about the host CPU type but QEMU's
version has a slightly different feature set").

-- PMM

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Christoffer Dall]

On Wed, Aug 14, 2013 at 08:21:54PM +0200, Alexander Graf wrote:
> 
> On 14.08.2013, at 20:18, Christoffer Dall wrote:
> 
> > On Wed, Aug 14, 2013 at 07:44:25PM +0200, Alexander Graf wrote:
> >> 
> >> On 14.08.2013, at 19:39, Christoffer Dall wrote:
> >> 
> >>> On Wed, Aug 14, 2013 at 07:31:59PM +0200, Alexander Graf wrote:
> >>>> 
> >>>> On 14.08.2013, at 19:26, Christoffer Dall wrote:
> >>>> 
> >>>>> On Wed, Aug 14, 2013 at 11:30:46AM +0200, Alexander Graf wrote:
> >>>>>> 
> >>>>>> On 14.08.2013, at 11:23, Peter Maydell wrote:
> >>>>>> 
> >>>>>>> On 14 August 2013 10:11, Alexander Graf <agraf@suse.de> wrote:
> >>>>>>>> You're right, the main difference is that KVM doesn't have any
> >>>>>>>> idea what a "host" style CPU is. It only knows how to report to QEMU
> >>>>>>>> what the current host CPU would be, so that anything from VCPU_INIT
> >>>>>>>> onwards is 100% identical regardless of whether the user said
> >>>>>>>> -cpu host or -cpu xxx.
> >>>>>>>> 
> >>>>>>>> I'm still puzzled on how this will work with BIG.little btw.
> >>>>>>> 
> >>>>>>> The rough idea is that for BIG.little the kernel must trap the
> >>>>>>> ID registers at least (so that the vcpu seems consistent to the
> >>>>>>> guest whether it's running on the big or the little core). For
> >>>>>>> "-cpu host" the guest would see whatever is the most low-overhead
> >>>>>>> for the kernel to provide (ie assuming the big and little CPUs
> >>>>>>> are roughly-similar you could make -cpu host provide something
> >>>>>>> that looks to the guest like the big CPU and don't have to trap
> >>>>>>> quite as much as you would for providing a vcpu that wasn't the
> >>>>>>> same as either the big or little one).
> >>>>>> 
> >>>>>> So -cpu host in this case wouldn't actually expose the host CPU 1:1, but instead a cortex-a15 even when it's run on an a7 BIG.little core. I see.
> >>>>>> 
> >>>>> Yes, from the discussion we've had the whole picture just becomes to
> >>>>> blurry when you start presenting multiple different CPUs to the guest
> >>>>> and there's really no need to that I'm aware of.
> >>>>> 
> >>>>> In fact the -cpu host case fits quite nicely with this state of mind;
> >>>>> the kernel is free to decide based on the specific hardware and config
> >>>>> on which it's running how to handle VMs on BL.
> >>>> 
> >>>> So why not have a vm ioctl to fetch the "best match" vcpu type? I don't like the idea of adding any awareness of a "host" type to the normal vcpu creation process.
> >>>> 
> >>>> 
> >>> That's actually what I suggested initially.  I'm not really a QEMU
> >>> expert, but I think Peter already answered this question: he doesn't
> >>> want to support hundreds of CPU models in QEMU just to be able to run
> >>> KVM when it's not necessary.
> >>> 
> >>> If his argument holds in that you can support -cpu host without having a
> >>> model for that specific cpu in QEMU, then indeed it is a strong
> >>> argument, and we have the problem with ARMv8 already, and this goes a
> >>> long way to solve that. No?
> >> 
> >> That's up for QEMU to decide. With the "fetch and push" model we can support both flavors from user space. It also makes the kernel side more reproducible and obvious. There's simply no way to add hacks like "If I'm a -cpu host type do xxx" in KVM, because KVM never knows that it is running -cpu host.
> >> 
> > Do we have historical examples of this knowledge being abused inside the
> > kernel for other archs?  If not, can we come up with a technical
> > scenario where it may happen on ARM?
> 
> if (cpu == host_cpu) {
>   vgic_version = get_host_vgic_version();
> }
> 
> would be an example :). 

Not really, this is driven from user space, but ok.

> Everything -cpu host does has to be reproducible without -cpu host, otherwise our compatibility layering is broken. So why not model the API like it from the beginning?
> 
> > 
> > Also, not really sure if such code should be controlled through the user
> > space API; ideally we would catch bad coding behavior in the kernel
> > during code review.
> > 
> > The only reason I originally suggested the "fetch and push" model was
> > that I thought user space would need to know the specific CPU model for
> > things to work and possibly for things like debugging and migration, but
> > since I have been almost convinced otherwise, I don't see any real
> > technical arguments for not adding -cpu host support in the kernel side.
> > 
> > Note that this doesn't prevent us from adding an IOCTL later that gives
> > you the host CPU type in KVM terminology if we find it useful.  But, I
> > think the reduced headache with ARMv8 right now is a good argument to
> > proceed with Peter's RFC and kernel support for same.
> 
> There's really almost no difference from the QEMU point of view if Peter choses to implement it the way he does today. He can ask the kernel for the vcpu target and pass that exact number back into the kernel.
> 
> 
>From the kernel point of view though we have to make some informed
decision about which "best CPU target" value to return on any given new
core, where TARGET_HOST may simply work through generic handling of id
registers, traps etc. and provide better performance than say, "I don't
really know this host CPU, so I'm just going to tell you A15 and trap
everything"...

Disclaimer: I haven't really thought this through, just thinking out
loud.

-Christoffer

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Alexander Graf]

On 14.08.2013, at 20:27, Peter Maydell wrote:

> On 14 August 2013 19:21, Alexander Graf <agraf@suse.de> wrote:
>> Everything -cpu host does has to be reproducible without -cpu host,
> 
> This is in general not going to be possible (the obvious case
> being "QEMU doesn't know about the host CPU type at all", or
> in minor variants "QEMU knows about the host CPU type but QEMU's
> version has a slightly different feature set").

That's a shortcoming in that particular version of QEMU then. The KVM interface still allows to do everything you could do with -cpu host with an explicit CPU define. If user space wants, it could write away the host CPU type and try to live migrate that machine to another machine then with the explicit target vcpu id.

Please keep in mind that we're talking about the generic KVM / user space interface here, not the actual QEMU implementation.


Alex

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Alexander Graf]

On 14.08.2013, at 20:28, Christoffer Dall wrote:

> On Wed, Aug 14, 2013 at 08:21:54PM +0200, Alexander Graf wrote:
>> 
>> On 14.08.2013, at 20:18, Christoffer Dall wrote:
>> 
>>> On Wed, Aug 14, 2013 at 07:44:25PM +0200, Alexander Graf wrote:
>>>> 
>>>> On 14.08.2013, at 19:39, Christoffer Dall wrote:
>>>> 
>>>>> On Wed, Aug 14, 2013 at 07:31:59PM +0200, Alexander Graf wrote:
>>>>>> 
>>>>>> On 14.08.2013, at 19:26, Christoffer Dall wrote:
>>>>>> 
>>>>>>> On Wed, Aug 14, 2013 at 11:30:46AM +0200, Alexander Graf wrote:
>>>>>>>> 
>>>>>>>> On 14.08.2013, at 11:23, Peter Maydell wrote:
>>>>>>>> 
>>>>>>>>> On 14 August 2013 10:11, Alexander Graf <agraf@suse.de> wrote:
>>>>>>>>>> You're right, the main difference is that KVM doesn't have any
>>>>>>>>>> idea what a "host" style CPU is. It only knows how to report to QEMU
>>>>>>>>>> what the current host CPU would be, so that anything from VCPU_INIT
>>>>>>>>>> onwards is 100% identical regardless of whether the user said
>>>>>>>>>> -cpu host or -cpu xxx.
>>>>>>>>>> 
>>>>>>>>>> I'm still puzzled on how this will work with BIG.little btw.
>>>>>>>>> 
>>>>>>>>> The rough idea is that for BIG.little the kernel must trap the
>>>>>>>>> ID registers at least (so that the vcpu seems consistent to the
>>>>>>>>> guest whether it's running on the big or the little core). For
>>>>>>>>> "-cpu host" the guest would see whatever is the most low-overhead
>>>>>>>>> for the kernel to provide (ie assuming the big and little CPUs
>>>>>>>>> are roughly-similar you could make -cpu host provide something
>>>>>>>>> that looks to the guest like the big CPU and don't have to trap
>>>>>>>>> quite as much as you would for providing a vcpu that wasn't the
>>>>>>>>> same as either the big or little one).
>>>>>>>> 
>>>>>>>> So -cpu host in this case wouldn't actually expose the host CPU 1:1, but instead a cortex-a15 even when it's run on an a7 BIG.little core. I see.
>>>>>>>> 
>>>>>>> Yes, from the discussion we've had the whole picture just becomes to
>>>>>>> blurry when you start presenting multiple different CPUs to the guest
>>>>>>> and there's really no need to that I'm aware of.
>>>>>>> 
>>>>>>> In fact the -cpu host case fits quite nicely with this state of mind;
>>>>>>> the kernel is free to decide based on the specific hardware and config
>>>>>>> on which it's running how to handle VMs on BL.
>>>>>> 
>>>>>> So why not have a vm ioctl to fetch the "best match" vcpu type? I don't like the idea of adding any awareness of a "host" type to the normal vcpu creation process.
>>>>>> 
>>>>>> 
>>>>> That's actually what I suggested initially.  I'm not really a QEMU
>>>>> expert, but I think Peter already answered this question: he doesn't
>>>>> want to support hundreds of CPU models in QEMU just to be able to run
>>>>> KVM when it's not necessary.
>>>>> 
>>>>> If his argument holds in that you can support -cpu host without having a
>>>>> model for that specific cpu in QEMU, then indeed it is a strong
>>>>> argument, and we have the problem with ARMv8 already, and this goes a
>>>>> long way to solve that. No?
>>>> 
>>>> That's up for QEMU to decide. With the "fetch and push" model we can support both flavors from user space. It also makes the kernel side more reproducible and obvious. There's simply no way to add hacks like "If I'm a -cpu host type do xxx" in KVM, because KVM never knows that it is running -cpu host.
>>>> 
>>> Do we have historical examples of this knowledge being abused inside the
>>> kernel for other archs?  If not, can we come up with a technical
>>> scenario where it may happen on ARM?
>> 
>> if (cpu == host_cpu) {
>>  vgic_version = get_host_vgic_version();
>> }
>> 
>> would be an example :). 
> 
> Not really, this is driven from user space, but ok.
> 
>> Everything -cpu host does has to be reproducible without -cpu host, otherwise our compatibility layering is broken. So why not model the API like it from the beginning?
>> 
>>> 
>>> Also, not really sure if such code should be controlled through the user
>>> space API; ideally we would catch bad coding behavior in the kernel
>>> during code review.
>>> 
>>> The only reason I originally suggested the "fetch and push" model was
>>> that I thought user space would need to know the specific CPU model for
>>> things to work and possibly for things like debugging and migration, but
>>> since I have been almost convinced otherwise, I don't see any real
>>> technical arguments for not adding -cpu host support in the kernel side.
>>> 
>>> Note that this doesn't prevent us from adding an IOCTL later that gives
>>> you the host CPU type in KVM terminology if we find it useful.  But, I
>>> think the reduced headache with ARMv8 right now is a good argument to
>>> proceed with Peter's RFC and kernel support for same.
>> 
>> There's really almost no difference from the QEMU point of view if Peter choses to implement it the way he does today. He can ask the kernel for the vcpu target and pass that exact number back into the kernel.
>> 
>> 
> From the kernel point of view though we have to make some informed
> decision about which "best CPU target" value to return on any given new
> core

We have to make that decision internally anyways, because we have to choose some CPU target for the host one.

> , where TARGET_HOST may simply work through generic handling of id
> registers, traps etc. and provide better performance than say, "I don't
> really know this host CPU, so I'm just going to tell you A15 and trap
> everything"...

Yes.

target_vcpu_id = kvm_vm_ioctl(KVM_VM_GET_BEST_CPU_TARGET);

/* Old kernels only support A15 hosts */
if (target_vcpu_id < 0)
  target_vcpu_id = VCPU_CORTEX_A15;

kvm_vcpu_ioctl(vcpu_fd, KVM_INIT_VCPU, target_vcpu_id);


Alex

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Christoffer Dall]

On Wed, Aug 14, 2013 at 09:28:10PM +0200, Alexander Graf wrote:
> 
> On 14.08.2013, at 20:28, Christoffer Dall wrote:
> 
> > On Wed, Aug 14, 2013 at 08:21:54PM +0200, Alexander Graf wrote:
> >> 
> >> On 14.08.2013, at 20:18, Christoffer Dall wrote:
> >> 
> >>> On Wed, Aug 14, 2013 at 07:44:25PM +0200, Alexander Graf wrote:
> >>>> 
> >>>> On 14.08.2013, at 19:39, Christoffer Dall wrote:
> >>>> 
> >>>>> On Wed, Aug 14, 2013 at 07:31:59PM +0200, Alexander Graf wrote:
> >>>>>> 
> >>>>>> On 14.08.2013, at 19:26, Christoffer Dall wrote:
> >>>>>> 
> >>>>>>> On Wed, Aug 14, 2013 at 11:30:46AM +0200, Alexander Graf wrote:
> >>>>>>>> 
> >>>>>>>> On 14.08.2013, at 11:23, Peter Maydell wrote:
> >>>>>>>> 
> >>>>>>>>> On 14 August 2013 10:11, Alexander Graf <agraf@suse.de> wrote:
> >>>>>>>>>> You're right, the main difference is that KVM doesn't have any
> >>>>>>>>>> idea what a "host" style CPU is. It only knows how to report to QEMU
> >>>>>>>>>> what the current host CPU would be, so that anything from VCPU_INIT
> >>>>>>>>>> onwards is 100% identical regardless of whether the user said
> >>>>>>>>>> -cpu host or -cpu xxx.
> >>>>>>>>>> 
> >>>>>>>>>> I'm still puzzled on how this will work with BIG.little btw.
> >>>>>>>>> 
> >>>>>>>>> The rough idea is that for BIG.little the kernel must trap the
> >>>>>>>>> ID registers at least (so that the vcpu seems consistent to the
> >>>>>>>>> guest whether it's running on the big or the little core). For
> >>>>>>>>> "-cpu host" the guest would see whatever is the most low-overhead
> >>>>>>>>> for the kernel to provide (ie assuming the big and little CPUs
> >>>>>>>>> are roughly-similar you could make -cpu host provide something
> >>>>>>>>> that looks to the guest like the big CPU and don't have to trap
> >>>>>>>>> quite as much as you would for providing a vcpu that wasn't the
> >>>>>>>>> same as either the big or little one).
> >>>>>>>> 
> >>>>>>>> So -cpu host in this case wouldn't actually expose the host CPU 1:1, but instead a cortex-a15 even when it's run on an a7 BIG.little core. I see.
> >>>>>>>> 
> >>>>>>> Yes, from the discussion we've had the whole picture just becomes to
> >>>>>>> blurry when you start presenting multiple different CPUs to the guest
> >>>>>>> and there's really no need to that I'm aware of.
> >>>>>>> 
> >>>>>>> In fact the -cpu host case fits quite nicely with this state of mind;
> >>>>>>> the kernel is free to decide based on the specific hardware and config
> >>>>>>> on which it's running how to handle VMs on BL.
> >>>>>> 
> >>>>>> So why not have a vm ioctl to fetch the "best match" vcpu type? I don't like the idea of adding any awareness of a "host" type to the normal vcpu creation process.
> >>>>>> 
> >>>>>> 
> >>>>> That's actually what I suggested initially.  I'm not really a QEMU
> >>>>> expert, but I think Peter already answered this question: he doesn't
> >>>>> want to support hundreds of CPU models in QEMU just to be able to run
> >>>>> KVM when it's not necessary.
> >>>>> 
> >>>>> If his argument holds in that you can support -cpu host without having a
> >>>>> model for that specific cpu in QEMU, then indeed it is a strong
> >>>>> argument, and we have the problem with ARMv8 already, and this goes a
> >>>>> long way to solve that. No?
> >>>> 
> >>>> That's up for QEMU to decide. With the "fetch and push" model we can support both flavors from user space. It also makes the kernel side more reproducible and obvious. There's simply no way to add hacks like "If I'm a -cpu host type do xxx" in KVM, because KVM never knows that it is running -cpu host.
> >>>> 
> >>> Do we have historical examples of this knowledge being abused inside the
> >>> kernel for other archs?  If not, can we come up with a technical
> >>> scenario where it may happen on ARM?
> >> 
> >> if (cpu == host_cpu) {
> >>  vgic_version = get_host_vgic_version();
> >> }
> >> 
> >> would be an example :). 
> > 
> > Not really, this is driven from user space, but ok.
> > 
> >> Everything -cpu host does has to be reproducible without -cpu host, otherwise our compatibility layering is broken. So why not model the API like it from the beginning?
> >> 
> >>> 
> >>> Also, not really sure if such code should be controlled through the user
> >>> space API; ideally we would catch bad coding behavior in the kernel
> >>> during code review.
> >>> 
> >>> The only reason I originally suggested the "fetch and push" model was
> >>> that I thought user space would need to know the specific CPU model for
> >>> things to work and possibly for things like debugging and migration, but
> >>> since I have been almost convinced otherwise, I don't see any real
> >>> technical arguments for not adding -cpu host support in the kernel side.
> >>> 
> >>> Note that this doesn't prevent us from adding an IOCTL later that gives
> >>> you the host CPU type in KVM terminology if we find it useful.  But, I
> >>> think the reduced headache with ARMv8 right now is a good argument to
> >>> proceed with Peter's RFC and kernel support for same.
> >> 
> >> There's really almost no difference from the QEMU point of view if Peter choses to implement it the way he does today. He can ask the kernel for the vcpu target and pass that exact number back into the kernel.
> >> 
> >> 
> > From the kernel point of view though we have to make some informed
> > decision about which "best CPU target" value to return on any given new
> > core
> 
> We have to make that decision internally anyways, because we have to choose some CPU target for the host one.
> 

Are we sure that will always be the case?  That's how it's structured
now, yes, but maybe we can do something more intelligent (which is what
I meant with "generic handling" below).  It's a bit fuzzy for me to
think about right now, but I just want to make sure we don't shoot
ourselves in the foot with the choice of ABI.

> > , where TARGET_HOST may simply work through generic handling of id
> > registers, traps etc. and provide better performance than say, "I don't
> > really know this host CPU, so I'm just going to tell you A15 and trap
> > everything"...
> 
> Yes.
> 
> target_vcpu_id = kvm_vm_ioctl(KVM_VM_GET_BEST_CPU_TARGET);
> 
> /* Old kernels only support A15 hosts */
> if (target_vcpu_id < 0)
>   target_vcpu_id = VCPU_CORTEX_A15;
> 
> kvm_vcpu_ioctl(vcpu_fd, KVM_INIT_VCPU, target_vcpu_id);
> 

I get this part, but imagine the kernel not knowing the target_cpu id,
but just passing through whatever the hardware gives you to the guest.

I'm not saying that's necessarily going to happen or that it would be a
great thing, but do we want to prevent this from ever happening through
our choice of ABI?

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Alexander Graf]

On 14.08.2013, at 22:33, Christoffer Dall wrote:

> On Wed, Aug 14, 2013 at 09:28:10PM +0200, Alexander Graf wrote:
>> 
>> On 14.08.2013, at 20:28, Christoffer Dall wrote:
>> 
>>> On Wed, Aug 14, 2013 at 08:21:54PM +0200, Alexander Graf wrote:
>>>> 
>>>> On 14.08.2013, at 20:18, Christoffer Dall wrote:
>>>> 
>>>>> On Wed, Aug 14, 2013 at 07:44:25PM +0200, Alexander Graf wrote:
>>>>>> 
>>>>>> On 14.08.2013, at 19:39, Christoffer Dall wrote:
>>>>>> 
>>>>>>> On Wed, Aug 14, 2013 at 07:31:59PM +0200, Alexander Graf wrote:
>>>>>>>> 
>>>>>>>> On 14.08.2013, at 19:26, Christoffer Dall wrote:
>>>>>>>> 
>>>>>>>>> On Wed, Aug 14, 2013 at 11:30:46AM +0200, Alexander Graf wrote:
>>>>>>>>>> 
>>>>>>>>>> On 14.08.2013, at 11:23, Peter Maydell wrote:
>>>>>>>>>> 
>>>>>>>>>>> On 14 August 2013 10:11, Alexander Graf <agraf@suse.de> wrote:
>>>>>>>>>>>> You're right, the main difference is that KVM doesn't have any
>>>>>>>>>>>> idea what a "host" style CPU is. It only knows how to report to QEMU
>>>>>>>>>>>> what the current host CPU would be, so that anything from VCPU_INIT
>>>>>>>>>>>> onwards is 100% identical regardless of whether the user said
>>>>>>>>>>>> -cpu host or -cpu xxx.
>>>>>>>>>>>> 
>>>>>>>>>>>> I'm still puzzled on how this will work with BIG.little btw.
>>>>>>>>>>> 
>>>>>>>>>>> The rough idea is that for BIG.little the kernel must trap the
>>>>>>>>>>> ID registers at least (so that the vcpu seems consistent to the
>>>>>>>>>>> guest whether it's running on the big or the little core). For
>>>>>>>>>>> "-cpu host" the guest would see whatever is the most low-overhead
>>>>>>>>>>> for the kernel to provide (ie assuming the big and little CPUs
>>>>>>>>>>> are roughly-similar you could make -cpu host provide something
>>>>>>>>>>> that looks to the guest like the big CPU and don't have to trap
>>>>>>>>>>> quite as much as you would for providing a vcpu that wasn't the
>>>>>>>>>>> same as either the big or little one).
>>>>>>>>>> 
>>>>>>>>>> So -cpu host in this case wouldn't actually expose the host CPU 1:1, but instead a cortex-a15 even when it's run on an a7 BIG.little core. I see.
>>>>>>>>>> 
>>>>>>>>> Yes, from the discussion we've had the whole picture just becomes to
>>>>>>>>> blurry when you start presenting multiple different CPUs to the guest
>>>>>>>>> and there's really no need to that I'm aware of.
>>>>>>>>> 
>>>>>>>>> In fact the -cpu host case fits quite nicely with this state of mind;
>>>>>>>>> the kernel is free to decide based on the specific hardware and config
>>>>>>>>> on which it's running how to handle VMs on BL.
>>>>>>>> 
>>>>>>>> So why not have a vm ioctl to fetch the "best match" vcpu type? I don't like the idea of adding any awareness of a "host" type to the normal vcpu creation process.
>>>>>>>> 
>>>>>>>> 
>>>>>>> That's actually what I suggested initially.  I'm not really a QEMU
>>>>>>> expert, but I think Peter already answered this question: he doesn't
>>>>>>> want to support hundreds of CPU models in QEMU just to be able to run
>>>>>>> KVM when it's not necessary.
>>>>>>> 
>>>>>>> If his argument holds in that you can support -cpu host without having a
>>>>>>> model for that specific cpu in QEMU, then indeed it is a strong
>>>>>>> argument, and we have the problem with ARMv8 already, and this goes a
>>>>>>> long way to solve that. No?
>>>>>> 
>>>>>> That's up for QEMU to decide. With the "fetch and push" model we can support both flavors from user space. It also makes the kernel side more reproducible and obvious. There's simply no way to add hacks like "If I'm a -cpu host type do xxx" in KVM, because KVM never knows that it is running -cpu host.
>>>>>> 
>>>>> Do we have historical examples of this knowledge being abused inside the
>>>>> kernel for other archs?  If not, can we come up with a technical
>>>>> scenario where it may happen on ARM?
>>>> 
>>>> if (cpu == host_cpu) {
>>>> vgic_version = get_host_vgic_version();
>>>> }
>>>> 
>>>> would be an example :). 
>>> 
>>> Not really, this is driven from user space, but ok.
>>> 
>>>> Everything -cpu host does has to be reproducible without -cpu host, otherwise our compatibility layering is broken. So why not model the API like it from the beginning?
>>>> 
>>>>> 
>>>>> Also, not really sure if such code should be controlled through the user
>>>>> space API; ideally we would catch bad coding behavior in the kernel
>>>>> during code review.
>>>>> 
>>>>> The only reason I originally suggested the "fetch and push" model was
>>>>> that I thought user space would need to know the specific CPU model for
>>>>> things to work and possibly for things like debugging and migration, but
>>>>> since I have been almost convinced otherwise, I don't see any real
>>>>> technical arguments for not adding -cpu host support in the kernel side.
>>>>> 
>>>>> Note that this doesn't prevent us from adding an IOCTL later that gives
>>>>> you the host CPU type in KVM terminology if we find it useful.  But, I
>>>>> think the reduced headache with ARMv8 right now is a good argument to
>>>>> proceed with Peter's RFC and kernel support for same.
>>>> 
>>>> There's really almost no difference from the QEMU point of view if Peter choses to implement it the way he does today. He can ask the kernel for the vcpu target and pass that exact number back into the kernel.
>>>> 
>>>> 
>>> From the kernel point of view though we have to make some informed
>>> decision about which "best CPU target" value to return on any given new
>>> core
>> 
>> We have to make that decision internally anyways, because we have to choose some CPU target for the host one.
>> 
> 
> Are we sure that will always be the case?  That's how it's structured
> now, yes, but maybe we can do something more intelligent (which is what
> I meant with "generic handling" below).  It's a bit fuzzy for me to
> think about right now, but I just want to make sure we don't shoot
> ourselves in the foot with the choice of ABI.

Exactly that cleverness is what I'd prefer to avoid, as it breaks reproducibility with cross-chip environments.

> 
>>> , where TARGET_HOST may simply work through generic handling of id
>>> registers, traps etc. and provide better performance than say, "I don't
>>> really know this host CPU, so I'm just going to tell you A15 and trap
>>> everything"...
>> 
>> Yes.
>> 
>> target_vcpu_id = kvm_vm_ioctl(KVM_VM_GET_BEST_CPU_TARGET);
>> 
>> /* Old kernels only support A15 hosts */
>> if (target_vcpu_id < 0)
>>  target_vcpu_id = VCPU_CORTEX_A15;
>> 
>> kvm_vcpu_ioctl(vcpu_fd, KVM_INIT_VCPU, target_vcpu_id);
>> 
> 
> I get this part, but imagine the kernel not knowing the target_cpu id,
> but just passing through whatever the hardware gives you to the guest.

Don't you have to handle core specific registers anyway?

> I'm not saying that's necessarily going to happen or that it would be a
> great thing, but do we want to prevent this from ever happening through
> our choice of ABI?

I think so, yes. Can you run Linux on a core that hasn't been enabled? Why should you be able to run KVM on a core that hasn't been enabled? I'm not talking about QEMU here - that one should be happy to be ignorant. But the kernel side needs to know about the core either way, no?


Alex

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Christoffer Dall]

On Wed, Aug 14, 2013 at 10:47:08PM +0200, Alexander Graf wrote:
> 
> On 14.08.2013, at 22:33, Christoffer Dall wrote:
> 
> > On Wed, Aug 14, 2013 at 09:28:10PM +0200, Alexander Graf wrote:
> >> 
> >> On 14.08.2013, at 20:28, Christoffer Dall wrote:
> >> 
> >>> On Wed, Aug 14, 2013 at 08:21:54PM +0200, Alexander Graf wrote:
> >>>> 
> >>>> On 14.08.2013, at 20:18, Christoffer Dall wrote:
> >>>> 
> >>>>> On Wed, Aug 14, 2013 at 07:44:25PM +0200, Alexander Graf wrote:
> >>>>>> 
> >>>>>> On 14.08.2013, at 19:39, Christoffer Dall wrote:
> >>>>>> 
> >>>>>>> On Wed, Aug 14, 2013 at 07:31:59PM +0200, Alexander Graf wrote:
> >>>>>>>> 
> >>>>>>>> On 14.08.2013, at 19:26, Christoffer Dall wrote:
> >>>>>>>> 
> >>>>>>>>> On Wed, Aug 14, 2013 at 11:30:46AM +0200, Alexander Graf wrote:
> >>>>>>>>>> 
> >>>>>>>>>> On 14.08.2013, at 11:23, Peter Maydell wrote:
> >>>>>>>>>> 
> >>>>>>>>>>> On 14 August 2013 10:11, Alexander Graf <agraf@suse.de> wrote:
> >>>>>>>>>>>> You're right, the main difference is that KVM doesn't have any
> >>>>>>>>>>>> idea what a "host" style CPU is. It only knows how to report to QEMU
> >>>>>>>>>>>> what the current host CPU would be, so that anything from VCPU_INIT
> >>>>>>>>>>>> onwards is 100% identical regardless of whether the user said
> >>>>>>>>>>>> -cpu host or -cpu xxx.
> >>>>>>>>>>>> 
> >>>>>>>>>>>> I'm still puzzled on how this will work with BIG.little btw.
> >>>>>>>>>>> 
> >>>>>>>>>>> The rough idea is that for BIG.little the kernel must trap the
> >>>>>>>>>>> ID registers at least (so that the vcpu seems consistent to the
> >>>>>>>>>>> guest whether it's running on the big or the little core). For
> >>>>>>>>>>> "-cpu host" the guest would see whatever is the most low-overhead
> >>>>>>>>>>> for the kernel to provide (ie assuming the big and little CPUs
> >>>>>>>>>>> are roughly-similar you could make -cpu host provide something
> >>>>>>>>>>> that looks to the guest like the big CPU and don't have to trap
> >>>>>>>>>>> quite as much as you would for providing a vcpu that wasn't the
> >>>>>>>>>>> same as either the big or little one).
> >>>>>>>>>> 
> >>>>>>>>>> So -cpu host in this case wouldn't actually expose the host CPU 1:1, but instead a cortex-a15 even when it's run on an a7 BIG.little core. I see.
> >>>>>>>>>> 
> >>>>>>>>> Yes, from the discussion we've had the whole picture just becomes to
> >>>>>>>>> blurry when you start presenting multiple different CPUs to the guest
> >>>>>>>>> and there's really no need to that I'm aware of.
> >>>>>>>>> 
> >>>>>>>>> In fact the -cpu host case fits quite nicely with this state of mind;
> >>>>>>>>> the kernel is free to decide based on the specific hardware and config
> >>>>>>>>> on which it's running how to handle VMs on BL.
> >>>>>>>> 
> >>>>>>>> So why not have a vm ioctl to fetch the "best match" vcpu type? I don't like the idea of adding any awareness of a "host" type to the normal vcpu creation process.
> >>>>>>>> 
> >>>>>>>> 
> >>>>>>> That's actually what I suggested initially.  I'm not really a QEMU
> >>>>>>> expert, but I think Peter already answered this question: he doesn't
> >>>>>>> want to support hundreds of CPU models in QEMU just to be able to run
> >>>>>>> KVM when it's not necessary.
> >>>>>>> 
> >>>>>>> If his argument holds in that you can support -cpu host without having a
> >>>>>>> model for that specific cpu in QEMU, then indeed it is a strong
> >>>>>>> argument, and we have the problem with ARMv8 already, and this goes a
> >>>>>>> long way to solve that. No?
> >>>>>> 
> >>>>>> That's up for QEMU to decide. With the "fetch and push" model we can support both flavors from user space. It also makes the kernel side more reproducible and obvious. There's simply no way to add hacks like "If I'm a -cpu host type do xxx" in KVM, because KVM never knows that it is running -cpu host.
> >>>>>> 
> >>>>> Do we have historical examples of this knowledge being abused inside the
> >>>>> kernel for other archs?  If not, can we come up with a technical
> >>>>> scenario where it may happen on ARM?
> >>>> 
> >>>> if (cpu == host_cpu) {
> >>>> vgic_version = get_host_vgic_version();
> >>>> }
> >>>> 
> >>>> would be an example :). 
> >>> 
> >>> Not really, this is driven from user space, but ok.
> >>> 
> >>>> Everything -cpu host does has to be reproducible without -cpu host, otherwise our compatibility layering is broken. So why not model the API like it from the beginning?
> >>>> 
> >>>>> 
> >>>>> Also, not really sure if such code should be controlled through the user
> >>>>> space API; ideally we would catch bad coding behavior in the kernel
> >>>>> during code review.
> >>>>> 
> >>>>> The only reason I originally suggested the "fetch and push" model was
> >>>>> that I thought user space would need to know the specific CPU model for
> >>>>> things to work and possibly for things like debugging and migration, but
> >>>>> since I have been almost convinced otherwise, I don't see any real
> >>>>> technical arguments for not adding -cpu host support in the kernel side.
> >>>>> 
> >>>>> Note that this doesn't prevent us from adding an IOCTL later that gives
> >>>>> you the host CPU type in KVM terminology if we find it useful.  But, I
> >>>>> think the reduced headache with ARMv8 right now is a good argument to
> >>>>> proceed with Peter's RFC and kernel support for same.
> >>>> 
> >>>> There's really almost no difference from the QEMU point of view if Peter choses to implement it the way he does today. He can ask the kernel for the vcpu target and pass that exact number back into the kernel.
> >>>> 
> >>>> 
> >>> From the kernel point of view though we have to make some informed
> >>> decision about which "best CPU target" value to return on any given new
> >>> core
> >> 
> >> We have to make that decision internally anyways, because we have to choose some CPU target for the host one.
> >> 
> > 
> > Are we sure that will always be the case?  That's how it's structured
> > now, yes, but maybe we can do something more intelligent (which is what
> > I meant with "generic handling" below).  It's a bit fuzzy for me to
> > think about right now, but I just want to make sure we don't shoot
> > ourselves in the foot with the choice of ABI.
> 
> Exactly that cleverness is what I'd prefer to avoid, as it breaks reproducibility with cross-chip environments.
> 
> > 
> >>> , where TARGET_HOST may simply work through generic handling of id
> >>> registers, traps etc. and provide better performance than say, "I don't
> >>> really know this host CPU, so I'm just going to tell you A15 and trap
> >>> everything"...
> >> 
> >> Yes.
> >> 
> >> target_vcpu_id = kvm_vm_ioctl(KVM_VM_GET_BEST_CPU_TARGET);
> >> 
> >> /* Old kernels only support A15 hosts */
> >> if (target_vcpu_id < 0)
> >>  target_vcpu_id = VCPU_CORTEX_A15;
> >> 
> >> kvm_vcpu_ioctl(vcpu_fd, KVM_INIT_VCPU, target_vcpu_id);
> >> 
> > 
> > I get this part, but imagine the kernel not knowing the target_cpu id,
> > but just passing through whatever the hardware gives you to the guest.
> 
> Don't you have to handle core specific registers anyway?
> 
> > I'm not saying that's necessarily going to happen or that it would be a
> > great thing, but do we want to prevent this from ever happening through
> > our choice of ABI?
> 
> I think so, yes. Can you run Linux on a core that hasn't been enabled? Why should you be able to run KVM on a core that hasn't been enabled? I'm not talking about QEMU here - that one should be happy to be ignorant. But the kernel side needs to know about the core either way, no?
> 
> 
I guess so, maybe.  So far we haven't seen a lot of cores, so we
definietely want to know about the specific cores that we want to
support.

But, if there really are going to be hundreds, or thousands, or hundreds
of thousands of virtualization-enabled ARM cores (ok, maybe not) then
maybe we want to come up with a way to not add code in KVM for every new
core we wish to support.  But then again, if ARM achieves true world
domination and inflicts us with that many core types, we can always
add a new ABI.

So at this point, I don't really care if we do it once way or another.

-Christoffer

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Alexander Graf]

On 14.08.2013, at 22:56, Christoffer Dall wrote:

> On Wed, Aug 14, 2013 at 10:47:08PM +0200, Alexander Graf wrote:
>> 
>> On 14.08.2013, at 22:33, Christoffer Dall wrote:
>> 
>>> On Wed, Aug 14, 2013 at 09:28:10PM +0200, Alexander Graf wrote:
>>>> 
>>>> On 14.08.2013, at 20:28, Christoffer Dall wrote:
>>>> 
>>>>> On Wed, Aug 14, 2013 at 08:21:54PM +0200, Alexander Graf wrote:
>>>>>> 
>>>>>> On 14.08.2013, at 20:18, Christoffer Dall wrote:
>>>>>> 
>>>>>>> On Wed, Aug 14, 2013 at 07:44:25PM +0200, Alexander Graf wrote:
>>>>>>>> 
>>>>>>>> On 14.08.2013, at 19:39, Christoffer Dall wrote:
>>>>>>>> 
>>>>>>>>> On Wed, Aug 14, 2013 at 07:31:59PM +0200, Alexander Graf wrote:
>>>>>>>>>> 
>>>>>>>>>> On 14.08.2013, at 19:26, Christoffer Dall wrote:
>>>>>>>>>> 
>>>>>>>>>>> On Wed, Aug 14, 2013 at 11:30:46AM +0200, Alexander Graf wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>> On 14.08.2013, at 11:23, Peter Maydell wrote:
>>>>>>>>>>>> 
>>>>>>>>>>>>> On 14 August 2013 10:11, Alexander Graf <agraf@suse.de> wrote:
>>>>>>>>>>>>>> You're right, the main difference is that KVM doesn't have any
>>>>>>>>>>>>>> idea what a "host" style CPU is. It only knows how to report to QEMU
>>>>>>>>>>>>>> what the current host CPU would be, so that anything from VCPU_INIT
>>>>>>>>>>>>>> onwards is 100% identical regardless of whether the user said
>>>>>>>>>>>>>> -cpu host or -cpu xxx.
>>>>>>>>>>>>>> 
>>>>>>>>>>>>>> I'm still puzzled on how this will work with BIG.little btw.
>>>>>>>>>>>>> 
>>>>>>>>>>>>> The rough idea is that for BIG.little the kernel must trap the
>>>>>>>>>>>>> ID registers at least (so that the vcpu seems consistent to the
>>>>>>>>>>>>> guest whether it's running on the big or the little core). For
>>>>>>>>>>>>> "-cpu host" the guest would see whatever is the most low-overhead
>>>>>>>>>>>>> for the kernel to provide (ie assuming the big and little CPUs
>>>>>>>>>>>>> are roughly-similar you could make -cpu host provide something
>>>>>>>>>>>>> that looks to the guest like the big CPU and don't have to trap
>>>>>>>>>>>>> quite as much as you would for providing a vcpu that wasn't the
>>>>>>>>>>>>> same as either the big or little one).
>>>>>>>>>>>> 
>>>>>>>>>>>> So -cpu host in this case wouldn't actually expose the host CPU 1:1, but instead a cortex-a15 even when it's run on an a7 BIG.little core. I see.
>>>>>>>>>>>> 
>>>>>>>>>>> Yes, from the discussion we've had the whole picture just becomes to
>>>>>>>>>>> blurry when you start presenting multiple different CPUs to the guest
>>>>>>>>>>> and there's really no need to that I'm aware of.
>>>>>>>>>>> 
>>>>>>>>>>> In fact the -cpu host case fits quite nicely with this state of mind;
>>>>>>>>>>> the kernel is free to decide based on the specific hardware and config
>>>>>>>>>>> on which it's running how to handle VMs on BL.
>>>>>>>>>> 
>>>>>>>>>> So why not have a vm ioctl to fetch the "best match" vcpu type? I don't like the idea of adding any awareness of a "host" type to the normal vcpu creation process.
>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> That's actually what I suggested initially.  I'm not really a QEMU
>>>>>>>>> expert, but I think Peter already answered this question: he doesn't
>>>>>>>>> want to support hundreds of CPU models in QEMU just to be able to run
>>>>>>>>> KVM when it's not necessary.
>>>>>>>>> 
>>>>>>>>> If his argument holds in that you can support -cpu host without having a
>>>>>>>>> model for that specific cpu in QEMU, then indeed it is a strong
>>>>>>>>> argument, and we have the problem with ARMv8 already, and this goes a
>>>>>>>>> long way to solve that. No?
>>>>>>>> 
>>>>>>>> That's up for QEMU to decide. With the "fetch and push" model we can support both flavors from user space. It also makes the kernel side more reproducible and obvious. There's simply no way to add hacks like "If I'm a -cpu host type do xxx" in KVM, because KVM never knows that it is running -cpu host.
>>>>>>>> 
>>>>>>> Do we have historical examples of this knowledge being abused inside the
>>>>>>> kernel for other archs?  If not, can we come up with a technical
>>>>>>> scenario where it may happen on ARM?
>>>>>> 
>>>>>> if (cpu == host_cpu) {
>>>>>> vgic_version = get_host_vgic_version();
>>>>>> }
>>>>>> 
>>>>>> would be an example :). 
>>>>> 
>>>>> Not really, this is driven from user space, but ok.
>>>>> 
>>>>>> Everything -cpu host does has to be reproducible without -cpu host, otherwise our compatibility layering is broken. So why not model the API like it from the beginning?
>>>>>> 
>>>>>>> 
>>>>>>> Also, not really sure if such code should be controlled through the user
>>>>>>> space API; ideally we would catch bad coding behavior in the kernel
>>>>>>> during code review.
>>>>>>> 
>>>>>>> The only reason I originally suggested the "fetch and push" model was
>>>>>>> that I thought user space would need to know the specific CPU model for
>>>>>>> things to work and possibly for things like debugging and migration, but
>>>>>>> since I have been almost convinced otherwise, I don't see any real
>>>>>>> technical arguments for not adding -cpu host support in the kernel side.
>>>>>>> 
>>>>>>> Note that this doesn't prevent us from adding an IOCTL later that gives
>>>>>>> you the host CPU type in KVM terminology if we find it useful.  But, I
>>>>>>> think the reduced headache with ARMv8 right now is a good argument to
>>>>>>> proceed with Peter's RFC and kernel support for same.
>>>>>> 
>>>>>> There's really almost no difference from the QEMU point of view if Peter choses to implement it the way he does today. He can ask the kernel for the vcpu target and pass that exact number back into the kernel.
>>>>>> 
>>>>>> 
>>>>> From the kernel point of view though we have to make some informed
>>>>> decision about which "best CPU target" value to return on any given new
>>>>> core
>>>> 
>>>> We have to make that decision internally anyways, because we have to choose some CPU target for the host one.
>>>> 
>>> 
>>> Are we sure that will always be the case?  That's how it's structured
>>> now, yes, but maybe we can do something more intelligent (which is what
>>> I meant with "generic handling" below).  It's a bit fuzzy for me to
>>> think about right now, but I just want to make sure we don't shoot
>>> ourselves in the foot with the choice of ABI.
>> 
>> Exactly that cleverness is what I'd prefer to avoid, as it breaks reproducibility with cross-chip environments.
>> 
>>> 
>>>>> , where TARGET_HOST may simply work through generic handling of id
>>>>> registers, traps etc. and provide better performance than say, "I don't
>>>>> really know this host CPU, so I'm just going to tell you A15 and trap
>>>>> everything"...
>>>> 
>>>> Yes.
>>>> 
>>>> target_vcpu_id = kvm_vm_ioctl(KVM_VM_GET_BEST_CPU_TARGET);
>>>> 
>>>> /* Old kernels only support A15 hosts */
>>>> if (target_vcpu_id < 0)
>>>> target_vcpu_id = VCPU_CORTEX_A15;
>>>> 
>>>> kvm_vcpu_ioctl(vcpu_fd, KVM_INIT_VCPU, target_vcpu_id);
>>>> 
>>> 
>>> I get this part, but imagine the kernel not knowing the target_cpu id,
>>> but just passing through whatever the hardware gives you to the guest.
>> 
>> Don't you have to handle core specific registers anyway?
>> 
>>> I'm not saying that's necessarily going to happen or that it would be a
>>> great thing, but do we want to prevent this from ever happening through
>>> our choice of ABI?
>> 
>> I think so, yes. Can you run Linux on a core that hasn't been enabled? Why should you be able to run KVM on a core that hasn't been enabled? I'm not talking about QEMU here - that one should be happy to be ignorant. But the kernel side needs to know about the core either way, no?
>> 
>> 
> I guess so, maybe.  So far we haven't seen a lot of cores, so we
> definietely want to know about the specific cores that we want to
> support.
> 
> But, if there really are going to be hundreds, or thousands, or hundreds
> of thousands of virtualization-enabled ARM cores (ok, maybe not) then

We're talking about cores here. My take is that we're rather looking at dozens at most.

> maybe we want to come up with a way to not add code in KVM for every new
> core we wish to support.  But then again, if ARM achieves true world
> domination and inflicts us with that many core types, we can always
> add a new ABI.

If the need really emerges we can always still add a TARGET_HOST target type that the enumeration function returns. But I don't think we'll ever hit that case.

> So at this point, I don't really care if we do it once way or another.

Great :).


Alex

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Gleb Natapov]

On Wed, Aug 14, 2013 at 09:28:10PM +0200, Alexander Graf wrote:
> 
> On 14.08.2013, at 20:28, Christoffer Dall wrote:
> 
> > On Wed, Aug 14, 2013 at 08:21:54PM +0200, Alexander Graf wrote:
> >> 
> >> On 14.08.2013, at 20:18, Christoffer Dall wrote:
> >> 
> >>> On Wed, Aug 14, 2013 at 07:44:25PM +0200, Alexander Graf wrote:
> >>>> 
> >>>> On 14.08.2013, at 19:39, Christoffer Dall wrote:
> >>>> 
> >>>>> On Wed, Aug 14, 2013 at 07:31:59PM +0200, Alexander Graf wrote:
> >>>>>> 
> >>>>>> On 14.08.2013, at 19:26, Christoffer Dall wrote:
> >>>>>> 
> >>>>>>> On Wed, Aug 14, 2013 at 11:30:46AM +0200, Alexander Graf wrote:
> >>>>>>>> 
> >>>>>>>> On 14.08.2013, at 11:23, Peter Maydell wrote:
> >>>>>>>> 
> >>>>>>>>> On 14 August 2013 10:11, Alexander Graf <agraf@suse.de> wrote:
> >>>>>>>>>> You're right, the main difference is that KVM doesn't have any
> >>>>>>>>>> idea what a "host" style CPU is. It only knows how to report to QEMU
> >>>>>>>>>> what the current host CPU would be, so that anything from VCPU_INIT
> >>>>>>>>>> onwards is 100% identical regardless of whether the user said
> >>>>>>>>>> -cpu host or -cpu xxx.
> >>>>>>>>>> 
> >>>>>>>>>> I'm still puzzled on how this will work with BIG.little btw.
> >>>>>>>>> 
> >>>>>>>>> The rough idea is that for BIG.little the kernel must trap the
> >>>>>>>>> ID registers at least (so that the vcpu seems consistent to the
> >>>>>>>>> guest whether it's running on the big or the little core). For
> >>>>>>>>> "-cpu host" the guest would see whatever is the most low-overhead
> >>>>>>>>> for the kernel to provide (ie assuming the big and little CPUs
> >>>>>>>>> are roughly-similar you could make -cpu host provide something
> >>>>>>>>> that looks to the guest like the big CPU and don't have to trap
> >>>>>>>>> quite as much as you would for providing a vcpu that wasn't the
> >>>>>>>>> same as either the big or little one).
> >>>>>>>> 
> >>>>>>>> So -cpu host in this case wouldn't actually expose the host CPU 1:1, but instead a cortex-a15 even when it's run on an a7 BIG.little core. I see.
> >>>>>>>> 
> >>>>>>> Yes, from the discussion we've had the whole picture just becomes to
> >>>>>>> blurry when you start presenting multiple different CPUs to the guest
> >>>>>>> and there's really no need to that I'm aware of.
> >>>>>>> 
> >>>>>>> In fact the -cpu host case fits quite nicely with this state of mind;
> >>>>>>> the kernel is free to decide based on the specific hardware and config
> >>>>>>> on which it's running how to handle VMs on BL.
> >>>>>> 
> >>>>>> So why not have a vm ioctl to fetch the "best match" vcpu type? I don't like the idea of adding any awareness of a "host" type to the normal vcpu creation process.
> >>>>>> 
> >>>>>> 
> >>>>> That's actually what I suggested initially.  I'm not really a QEMU
> >>>>> expert, but I think Peter already answered this question: he doesn't
> >>>>> want to support hundreds of CPU models in QEMU just to be able to run
> >>>>> KVM when it's not necessary.
> >>>>> 
> >>>>> If his argument holds in that you can support -cpu host without having a
> >>>>> model for that specific cpu in QEMU, then indeed it is a strong
> >>>>> argument, and we have the problem with ARMv8 already, and this goes a
> >>>>> long way to solve that. No?
> >>>> 
> >>>> That's up for QEMU to decide. With the "fetch and push" model we can support both flavors from user space. It also makes the kernel side more reproducible and obvious. There's simply no way to add hacks like "If I'm a -cpu host type do xxx" in KVM, because KVM never knows that it is running -cpu host.
> >>>> 
> >>> Do we have historical examples of this knowledge being abused inside the
> >>> kernel for other archs?  If not, can we come up with a technical
> >>> scenario where it may happen on ARM?
> >> 
> >> if (cpu == host_cpu) {
> >>  vgic_version = get_host_vgic_version();
> >> }
> >> 
> >> would be an example :). 
> > 
> > Not really, this is driven from user space, but ok.
> > 
> >> Everything -cpu host does has to be reproducible without -cpu host, otherwise our compatibility layering is broken. So why not model the API like it from the beginning?
> >> 
> >>> 
> >>> Also, not really sure if such code should be controlled through the user
> >>> space API; ideally we would catch bad coding behavior in the kernel
> >>> during code review.
> >>> 
> >>> The only reason I originally suggested the "fetch and push" model was
> >>> that I thought user space would need to know the specific CPU model for
> >>> things to work and possibly for things like debugging and migration, but
> >>> since I have been almost convinced otherwise, I don't see any real
> >>> technical arguments for not adding -cpu host support in the kernel side.
> >>> 
> >>> Note that this doesn't prevent us from adding an IOCTL later that gives
> >>> you the host CPU type in KVM terminology if we find it useful.  But, I
> >>> think the reduced headache with ARMv8 right now is a good argument to
> >>> proceed with Peter's RFC and kernel support for same.
> >> 
> >> There's really almost no difference from the QEMU point of view if Peter choses to implement it the way he does today. He can ask the kernel for the vcpu target and pass that exact number back into the kernel.
> >> 
> >> 
> > From the kernel point of view though we have to make some informed
> > decision about which "best CPU target" value to return on any given new
> > core
> 
> We have to make that decision internally anyways, because we have to choose some CPU target for the host one.
> 
> > , where TARGET_HOST may simply work through generic handling of id
> > registers, traps etc. and provide better performance than say, "I don't
> > really know this host CPU, so I'm just going to tell you A15 and trap
> > everything"...
> 
> Yes.
> 
> target_vcpu_id = kvm_vm_ioctl(KVM_VM_GET_BEST_CPU_TARGET);
> 
> /* Old kernels only support A15 hosts */
> if (target_vcpu_id < 0)
>   target_vcpu_id = VCPU_CORTEX_A15;
> 
> kvm_vcpu_ioctl(vcpu_fd, KVM_INIT_VCPU, target_vcpu_id);
> 
Are ARM cpu features not enumerable and the only way to know what is
supported by a core is by its id? I do see feature registers in the
spec, so why wouldn't we want to be able to disable/enable features
individually?

--
			Gleb.

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Peter Maydell]

On 25 August 2013 15:42, Gleb Natapov <gleb@redhat.com> wrote:
> Are ARM cpu features not enumerable and the only way to know what is
> supported by a core is by its id? I do see feature registers in the
> spec, so why wouldn't we want to be able to disable/enable features
> individually?

It's complicated. There are feature registers which have various
bits, but only some combinations are architecturally valid. In any
case a guest might reasonably legitimately assume "this is an A15
so it will have architected timers", for example, and not bother
testing feature bits. Some feature bits really do have to match the
real hardware, like ones that say "you need to explicitly flush the
branch predictor". A lot are feature bits that are simply mandatory
for any ARMv7 or later processor (which is what you need for KVM
to work at all). And buried in there might possibly be a few features
that it might actually make sense to enable/disable separately (like
"do we have Neon?").

So what we have for now is that the INIT_VCPU ioctl has space
for feature flags (currently unused in v7; in v8 there's a "make the
VM be a 32 bit VM on a 64 bit CPU" flag), so we could wire those
up some day. (We don't have any support in QEMU for saying
"I want an ARM CPU with features X and Y but not Z" yet either.)

-- PMM

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Gleb Natapov]

On Sun, Aug 25, 2013 at 04:11:51PM +0100, Peter Maydell wrote:
> On 25 August 2013 15:42, Gleb Natapov <gleb@redhat.com> wrote:
> > Are ARM cpu features not enumerable and the only way to know what is
> > supported by a core is by its id? I do see feature registers in the
> > spec, so why wouldn't we want to be able to disable/enable features
> > individually?
> 
> It's complicated. There are feature registers which have various
> bits, but only some combinations are architecturally valid. In any
> case a guest might reasonably legitimately assume "this is an A15
> so it will have architected timers", for example, and not bother
> testing feature bits.
That's sad. If feature bit is available I would expect well behaved
guest to test it. BTW is there architectural way to tell OS that it
runs as a guest? There is dedicated CPUID bit on x86 for that which is
obviously never set on real CPUs.

>                       Some feature bits really do have to match the
> real hardware, like ones that say "you need to explicitly flush the
> branch predictor". A lot are feature bits that are simply mandatory
> for any ARMv7 or later processor (which is what you need for KVM
> to work at all).
What if you want to emulate older arm arch for a guest?

>                   And buried in there might possibly be a few features
> that it might actually make sense to enable/disable separately (like
> "do we have Neon?").
> 
> So what we have for now is that the INIT_VCPU ioctl has space
> for feature flags (currently unused in v7; in v8 there's a "make the
> VM be a 32 bit VM on a 64 bit CPU" flag), so we could wire those
> up some day. (We don't have any support in QEMU for saying
> "I want an ARM CPU with features X and Y but not Z" yet either.)
> 
Do we have a way to ask KVM if it can support those features?

--
			Gleb.

Re: [Qemu-devel] [RFC 0/2] target-arm: Provide '-cpu host' when running KVM

[Peter Maydell]

On 26 August 2013 12:18, Gleb Natapov <gleb@redhat.com> wrote:
> On Sun, Aug 25, 2013 at 04:11:51PM +0100, Peter Maydell wrote:
>> A lot are feature bits that are simply mandatory
>> for any ARMv7 or later processor (which is what you need for KVM
>> to work at all).
> What if you want to emulate older arm arch for a guest?

I don't believe that was a design goal architecturally and it's
not a design goal for KVM/ARM either. If anybody actually
cares about that use case they're welcome to submit patches.
In any case, feature bits are probably the least of your worries
(in that if the guest looks at them and uses features not strictly
present on the older CPU it's still going to run, and if it doesn't
look at them then it doesn't care what the bit says); to pick
some random examples:
 * no way to make a v7 core provide v5 unaligned-access
   behaviour
 * a v6 OS is unlikely to be able to correctly manage/clean/
   invalidate the kind of cache a v7 CPU has
 * no support for the ARM v5 legacy pagetable format
There are no doubt others.

-- PMM