[Qemu-devel] [PATCH v2 00/14] target/arm/kvm: enable SVE in guests

[Qemu-devel] [PATCH v2 00/14] target/arm/kvm: enable SVE in guests

[Andrew Jones]

Since Linux kernel v5.2-rc1 KVM has support for enabling SVE in guests.
This series provides the QEMU bits for that enablement. This is a v2
series, however it looks completely different than v1. Thank you to
all who reviewed v1. I've included all input still relevant to this
new approach. And the new approach is thanks to Igor for suggesting
it. The new approach is to use a CPU property for each vector length
and then implement the preexisting qmp_query_cpu_model_expansion
query for Arm to expose them. Here's how the series goes:

First, we select existing CPU properties representing features we
want to advertise in addition to the SVE vector lengths and prepare
them for the qmp query. Then we introduce the qmp query, applying
it immediately to those selected features. We next add a qtest for
the selected CPU features that uses the qmp query for its tests - and
we continue to add tests as we add CPU features with the following
patches. So then, once we have the support we need for CPU feature
querying and testing, we add our first SVE CPU feature property, sve,
which just allows SVE to be completely enabled/disabled. Following
that feature property, we add all 16 vector length properties along
with the input validation they need and tests to prove the validation
works. At this point the SVE features are still only for TCG, so we
provide some patches to prepare for KVM and then a patch that allows
the 'max' CPU type to enable SVE with KVM, but at first without
vector length properties. After a bit more preparation we add the
SVE vector length properties to the KVM-enabled 'max' CPU type along
with the additional input validation and tests that that needs.
Finally we allow the 'host' CPU type to also enjoy these properties
by simply sharing them with it.

Phew, I think that's everything.

Thanks!
drew

Andrew Jones (14):
  target/arm/cpu64: Ensure kvm really supports aarch64=off
  target/arm/cpu: Ensure we can use the pmu with kvm
  target/arm/monitor: Introduce qmp_query_cpu_model_expansion
  tests: arm: Introduce cpu feature tests
  target/arm/helper: zcr: Add build bug next to value range assumption
  target/arm: Allow SVE to be disabled via a CPU property
  target/arm/cpu64: max cpu: Introduce sve<vl-bits> properties
  target/arm/kvm64: Fix error returns
  target/arm/kvm64: Move the get/put of fpsimd registers out
  target/arm/kvm64: Add kvm_arch_get/put_sve
  target/arm/kvm64: max cpu: Enable SVE when available
  target/arm/kvm: scratch vcpu: Preserve input kvm_vcpu_init features
  target/arm/cpu64: max cpu: Support sve properties with KVM
  target/arm/kvm: host cpu: Add support for sve<vl-bits> properties

 qapi/target.json         |   6 +-
 target/arm/cpu.c         |  47 +++-
 target/arm/cpu.h         |  17 ++
 target/arm/cpu64.c       | 548 +++++++++++++++++++++++++++++++++++++--
 target/arm/helper.c      |  20 +-
 target/arm/kvm.c         |  34 ++-
 target/arm/kvm32.c       |   6 +-
 target/arm/kvm64.c       | 428 +++++++++++++++++++++++++-----
 target/arm/kvm_arm.h     |  73 ++++++
 target/arm/monitor.c     | 148 +++++++++++
 tests/Makefile.include   |   5 +-
 tests/arm-cpu-features.c | 509 ++++++++++++++++++++++++++++++++++++
 12 files changed, 1738 insertions(+), 103 deletions(-)
 create mode 100644 tests/arm-cpu-features.c

-- 
2.20.1

[Qemu-devel] [PATCH v2 01/14] target/arm/cpu64: Ensure kvm really supports aarch64=off

[Andrew Jones]

If -cpu <cpu>,aarch64=off is used then KVM must also be used, and it
and the host must support running the vcpu in 32-bit mode. Also, if
-cpu <cpu>,aarch64=on is used, then it doesn't matter if kvm is
enabled or not.

Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 target/arm/cpu64.c   | 12 ++++++------
 target/arm/kvm64.c   | 11 +++++++++++
 target/arm/kvm_arm.h | 14 ++++++++++++++
 3 files changed, 31 insertions(+), 6 deletions(-)

diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
index 1901997a0645..946994838d8a 100644
--- a/target/arm/cpu64.c
+++ b/target/arm/cpu64.c
@@ -407,13 +407,13 @@ static void aarch64_cpu_set_aarch64(Object *obj, bool value, Error **errp)
      * restriction allows us to avoid fixing up functionality that assumes a
      * uniform execution state like do_interrupt.
      */
-    if (!kvm_enabled()) {
-        error_setg(errp, "'aarch64' feature cannot be disabled "
-                         "unless KVM is enabled");
-        return;
-    }
-
     if (value == false) {
+        if (!kvm_enabled() || !kvm_arm_aarch32_supported(CPU(cpu))) {
+            error_setg(errp, "'aarch64' feature cannot be disabled "
+                             "unless KVM is enabled and 32-bit EL1 "
+                             "is supported");
+            return;
+        }
         unset_feature(&cpu->env, ARM_FEATURE_AARCH64);
     } else {
         set_feature(&cpu->env, ARM_FEATURE_AARCH64);
diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
index 22d19c9aec6f..45ccda589903 100644
--- a/target/arm/kvm64.c
+++ b/target/arm/kvm64.c
@@ -24,7 +24,9 @@
 #include "exec/gdbstub.h"
 #include "sysemu/sysemu.h"
 #include "sysemu/kvm.h"
+#include "sysemu/kvm_int.h"
 #include "kvm_arm.h"
+#include "hw/boards.h"
 #include "internals.h"
 
 static bool have_guest_debug;
@@ -593,6 +595,15 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
     return true;
 }
 
+bool kvm_arm_aarch32_supported(CPUState *cpu)
+{
+    KVMState *s = KVM_STATE(current_machine->accelerator);
+    int ret;
+
+    ret = kvm_check_extension(s, KVM_CAP_ARM_EL1_32BIT);
+    return ret > 0;
+}
+
 #define ARM_CPU_ID_MPIDR       3, 0, 0, 0, 5
 
 int kvm_arch_init_vcpu(CPUState *cs)
diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
index 2a07333c615f..812125f805a1 100644
--- a/target/arm/kvm_arm.h
+++ b/target/arm/kvm_arm.h
@@ -207,6 +207,15 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf);
  */
 void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu);
 
+/**
+ * kvm_arm_aarch32_supported:
+ * @cs: CPUState
+ *
+ * Returns true if the KVM VCPU can enable AArch32 mode and false
+ * otherwise.
+ */
+bool kvm_arm_aarch32_supported(CPUState *cs);
+
 /**
  * kvm_arm_get_max_vm_ipa_size - Returns the number of bits in the
  * IPA address space supported by KVM
@@ -247,6 +256,11 @@ static inline void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu)
     cpu->host_cpu_probe_failed = true;
 }
 
+static inline bool kvm_arm_aarch32_supported(CPUState *cs)
+{
+    return false;
+}
+
 static inline int kvm_arm_get_max_vm_ipa_size(MachineState *ms)
 {
     return -ENOENT;
-- 
2.20.1

Re: [Qemu-devel] [PATCH v2 01/14] target/arm/cpu64: Ensure kvm really supports aarch64=off

[Auger Eric]

Hi Drew,

On 6/21/19 6:34 PM, Andrew Jones wrote:
> If -cpu <cpu>,aarch64=off is used then KVM must also be used, and it
> and the host must support running the vcpu in 32-bit mode. Also, if
> -cpu <cpu>,aarch64=on is used, then it doesn't matter if kvm is
> enabled or not.
> 
> Signed-off-by: Andrew Jones <drjones@redhat.com>


> ---
>  target/arm/cpu64.c   | 12 ++++++------
>  target/arm/kvm64.c   | 11 +++++++++++
>  target/arm/kvm_arm.h | 14 ++++++++++++++
>  3 files changed, 31 insertions(+), 6 deletions(-)
> 
> diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
> index 1901997a0645..946994838d8a 100644
> --- a/target/arm/cpu64.c
> +++ b/target/arm/cpu64.c
> @@ -407,13 +407,13 @@ static void aarch64_cpu_set_aarch64(Object *obj, bool value, Error **errp)
>       * restriction allows us to avoid fixing up functionality that assumes a
>       * uniform execution state like do_interrupt.
>       */> -    if (!kvm_enabled()) {
> -        error_setg(errp, "'aarch64' feature cannot be disabled "
> -                         "unless KVM is enabled");
> -        return;
> -    }
> -
>      if (value == false) {
> +        if (!kvm_enabled() || !kvm_arm_aarch32_supported(CPU(cpu))) {
> +            error_setg(errp, "'aarch64' feature cannot be disabled "
> +                             "unless KVM is enabled and 32-bit EL1 "
> +                             "is supported");
> +            return;
> +        }
>          unset_feature(&cpu->env, ARM_FEATURE_AARCH64);
>      } else {
>          set_feature(&cpu->env, ARM_FEATURE_AARCH64);
> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> index 22d19c9aec6f..45ccda589903 100644
> --- a/target/arm/kvm64.c
> +++ b/target/arm/kvm64.c
> @@ -24,7 +24,9 @@
>  #include "exec/gdbstub.h"
>  #include "sysemu/sysemu.h"
>  #include "sysemu/kvm.h"
> +#include "sysemu/kvm_int.h"
>  #include "kvm_arm.h"
> +#include "hw/boards.h"
>  #include "internals.h"
>  
>  static bool have_guest_debug;
> @@ -593,6 +595,15 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
>      return true;
>  }
>  
> +bool kvm_arm_aarch32_supported(CPUState *cpu)
> +{
> +    KVMState *s = KVM_STATE(current_machine->accelerator);
> +    int ret;
> +
> +    ret = kvm_check_extension(s, KVM_CAP_ARM_EL1_32BIT);
> +    return ret > 0;
nit: return kvm_check_extension() should be sufficient
> +}
> +
>  #define ARM_CPU_ID_MPIDR       3, 0, 0, 0, 5
>  
>  int kvm_arch_init_vcpu(CPUState *cs)
> diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
> index 2a07333c615f..812125f805a1 100644
> --- a/target/arm/kvm_arm.h
> +++ b/target/arm/kvm_arm.h
> @@ -207,6 +207,15 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf);
>   */
>  void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu);
>  
> +/**
> + * kvm_arm_aarch32_supported:
> + * @cs: CPUState
use kernel-doc comment style?
> + *
> + * Returns true if the KVM VCPU can enable AArch32 mode and false
> + * otherwise.
> + */
> +bool kvm_arm_aarch32_supported(CPUState *cs);
> +
>  /**
>   * kvm_arm_get_max_vm_ipa_size - Returns the number of bits in the
>   * IPA address space supported by KVM
> @@ -247,6 +256,11 @@ static inline void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu)
>      cpu->host_cpu_probe_failed = true;
>  }
>  
> +static inline bool kvm_arm_aarch32_supported(CPUState *cs)
> +{
> +    return false;
> +}
> +
>  static inline int kvm_arm_get_max_vm_ipa_size(MachineState *ms)
>  {
>      return -ENOENT;
> 
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks

Eric

Re: [Qemu-devel] [PATCH v2 01/14] target/arm/cpu64: Ensure kvm really supports aarch64=off

[Andrew Jones]

On Tue, Jun 25, 2019 at 11:35:12AM +0200, Auger Eric wrote:
> Hi Drew,
> 
> On 6/21/19 6:34 PM, Andrew Jones wrote:
> > If -cpu <cpu>,aarch64=off is used then KVM must also be used, and it
> > and the host must support running the vcpu in 32-bit mode. Also, if
> > -cpu <cpu>,aarch64=on is used, then it doesn't matter if kvm is
> > enabled or not.
> > 
> > Signed-off-by: Andrew Jones <drjones@redhat.com>
> 
> 
> > ---
> >  target/arm/cpu64.c   | 12 ++++++------
> >  target/arm/kvm64.c   | 11 +++++++++++
> >  target/arm/kvm_arm.h | 14 ++++++++++++++
> >  3 files changed, 31 insertions(+), 6 deletions(-)
> > 
> > diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
> > index 1901997a0645..946994838d8a 100644
> > --- a/target/arm/cpu64.c
> > +++ b/target/arm/cpu64.c
> > @@ -407,13 +407,13 @@ static void aarch64_cpu_set_aarch64(Object *obj, bool value, Error **errp)
> >       * restriction allows us to avoid fixing up functionality that assumes a
> >       * uniform execution state like do_interrupt.
> >       */> -    if (!kvm_enabled()) {
> > -        error_setg(errp, "'aarch64' feature cannot be disabled "
> > -                         "unless KVM is enabled");
> > -        return;
> > -    }
> > -
> >      if (value == false) {
> > +        if (!kvm_enabled() || !kvm_arm_aarch32_supported(CPU(cpu))) {
> > +            error_setg(errp, "'aarch64' feature cannot be disabled "
> > +                             "unless KVM is enabled and 32-bit EL1 "
> > +                             "is supported");
> > +            return;
> > +        }
> >          unset_feature(&cpu->env, ARM_FEATURE_AARCH64);
> >      } else {
> >          set_feature(&cpu->env, ARM_FEATURE_AARCH64);
> > diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> > index 22d19c9aec6f..45ccda589903 100644
> > --- a/target/arm/kvm64.c
> > +++ b/target/arm/kvm64.c
> > @@ -24,7 +24,9 @@
> >  #include "exec/gdbstub.h"
> >  #include "sysemu/sysemu.h"
> >  #include "sysemu/kvm.h"
> > +#include "sysemu/kvm_int.h"
> >  #include "kvm_arm.h"
> > +#include "hw/boards.h"
> >  #include "internals.h"
> >  
> >  static bool have_guest_debug;
> > @@ -593,6 +595,15 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
> >      return true;
> >  }
> >  
> > +bool kvm_arm_aarch32_supported(CPUState *cpu)
> > +{
> > +    KVMState *s = KVM_STATE(current_machine->accelerator);
> > +    int ret;
> > +
> > +    ret = kvm_check_extension(s, KVM_CAP_ARM_EL1_32BIT);
> > +    return ret > 0;
> nit: return kvm_check_extension() should be sufficient

Ah yes, I forgot kvm_check_extension() already converts negative
error codes to zero. I'll fix that for v3.

> > +}
> > +
> >  #define ARM_CPU_ID_MPIDR       3, 0, 0, 0, 5
> >  
> >  int kvm_arch_init_vcpu(CPUState *cs)
> > diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
> > index 2a07333c615f..812125f805a1 100644
> > --- a/target/arm/kvm_arm.h
> > +++ b/target/arm/kvm_arm.h
> > @@ -207,6 +207,15 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf);
> >   */
> >  void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu);
> >  
> > +/**
> > + * kvm_arm_aarch32_supported:
> > + * @cs: CPUState
> use kernel-doc comment style?

This file (kvm_arm.h) doesn't appear to have a super consistent comment
style. I see some use @var: for the parameters and some have 'Returns:
...' lines as well. I'm happy to do whatever the maintainers prefer. For
now I was just trying to mimic whatever caught my eye.

> > + *
> > + * Returns true if the KVM VCPU can enable AArch32 mode and false
> > + * otherwise.
> > + */
> > +bool kvm_arm_aarch32_supported(CPUState *cs);
> > +
> >  /**
> >   * kvm_arm_get_max_vm_ipa_size - Returns the number of bits in the
> >   * IPA address space supported by KVM
> > @@ -247,6 +256,11 @@ static inline void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu)
> >      cpu->host_cpu_probe_failed = true;
> >  }
> >  
> > +static inline bool kvm_arm_aarch32_supported(CPUState *cs)
> > +{
> > +    return false;
> > +}
> > +
> >  static inline int kvm_arm_get_max_vm_ipa_size(MachineState *ms)
> >  {
> >      return -ENOENT;
> > 
> Reviewed-by: Eric Auger <eric.auger@redhat.com>
>

Thanks,
drew 

Re: [Qemu-devel] [PATCH v2 01/14] target/arm/cpu64: Ensure kvm really supports aarch64=off

[Auger Eric]

Hi Drew,

On 6/25/19 3:34 PM, Andrew Jones wrote:
> On Tue, Jun 25, 2019 at 11:35:12AM +0200, Auger Eric wrote:
>> Hi Drew,
>>
>> On 6/21/19 6:34 PM, Andrew Jones wrote:
>>> If -cpu <cpu>,aarch64=off is used then KVM must also be used, and it
>>> and the host must support running the vcpu in 32-bit mode. Also, if
s/and it//
>>> -cpu <cpu>,aarch64=on is used, then it doesn't matter if kvm is
>>> enabled or not.
>>>
>>> Signed-off-by: Andrew Jones <drjones@redhat.com>
>>
>>
>>> ---
>>>  target/arm/cpu64.c   | 12 ++++++------
>>>  target/arm/kvm64.c   | 11 +++++++++++
>>>  target/arm/kvm_arm.h | 14 ++++++++++++++
>>>  3 files changed, 31 insertions(+), 6 deletions(-)
>>>
>>> diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
>>> index 1901997a0645..946994838d8a 100644
>>> --- a/target/arm/cpu64.c
>>> +++ b/target/arm/cpu64.c
>>> @@ -407,13 +407,13 @@ static void aarch64_cpu_set_aarch64(Object *obj, bool value, Error **errp)
>>>       * restriction allows us to avoid fixing up functionality that assumes a
>>>       * uniform execution state like do_interrupt.
>>>       */> -    if (!kvm_enabled()) {
>>> -        error_setg(errp, "'aarch64' feature cannot be disabled "
>>> -                         "unless KVM is enabled");
>>> -        return;
>>> -    }
>>> -
>>>      if (value == false) {
>>> +        if (!kvm_enabled() || !kvm_arm_aarch32_supported(CPU(cpu))) {
>>> +            error_setg(errp, "'aarch64' feature cannot be disabled "
>>> +                             "unless KVM is enabled and 32-bit EL1 "
>>> +                             "is supported");
>>> +            return;
>>> +        }
>>>          unset_feature(&cpu->env, ARM_FEATURE_AARCH64);
>>>      } else {
>>>          set_feature(&cpu->env, ARM_FEATURE_AARCH64);
>>> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
>>> index 22d19c9aec6f..45ccda589903 100644
>>> --- a/target/arm/kvm64.c
>>> +++ b/target/arm/kvm64.c
>>> @@ -24,7 +24,9 @@
>>>  #include "exec/gdbstub.h"
>>>  #include "sysemu/sysemu.h"
>>>  #include "sysemu/kvm.h"
>>> +#include "sysemu/kvm_int.h"
>>>  #include "kvm_arm.h"
>>> +#include "hw/boards.h"
By the way those two new headers are not needed by this patch
>>>  #include "internals.h"
>>>  
>>>  static bool have_guest_debug;
>>> @@ -593,6 +595,15 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
>>>      return true;
>>>  }
>>>  
>>> +bool kvm_arm_aarch32_supported(CPUState *cpu)
>>> +{
>>> +    KVMState *s = KVM_STATE(current_machine->accelerator);
>>> +    int ret;
>>> +
>>> +    ret = kvm_check_extension(s, KVM_CAP_ARM_EL1_32BIT);
>>> +    return ret > 0;
>> nit: return kvm_check_extension() should be sufficient
> 
> Ah yes, I forgot kvm_check_extension() already converts negative
> error codes to zero. I'll fix that for v3.
> 
>>> +}
>>> +
>>>  #define ARM_CPU_ID_MPIDR       3, 0, 0, 0, 5
>>>  
>>>  int kvm_arch_init_vcpu(CPUState *cs)
>>> diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
>>> index 2a07333c615f..812125f805a1 100644
>>> --- a/target/arm/kvm_arm.h
>>> +++ b/target/arm/kvm_arm.h
>>> @@ -207,6 +207,15 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf);
>>>   */
>>>  void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu);
>>>  
>>> +/**
>>> + * kvm_arm_aarch32_supported:
>>> + * @cs: CPUState
>> use kernel-doc comment style?
> 
> This file (kvm_arm.h) doesn't appear to have a super consistent comment
> style. I see some use @var: for the parameters and some have 'Returns:
> ...' lines as well. I'm happy to do whatever the maintainers prefer. For
> now I was just trying to mimic whatever caught my eye.>
>>> + *
>>> + * Returns true if the KVM VCPU can enable AArch32 mode and false
>>> + * otherwise.
>>> + */
>>> +bool kvm_arm_aarch32_supported(CPUState *cs);
>>> +
>>>  /**
>>>   * kvm_arm_get_max_vm_ipa_size - Returns the number of bits in the
>>>   * IPA address space supported by KVM
>>> @@ -247,6 +256,11 @@ static inline void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu)
>>>      cpu->host_cpu_probe_failed = true;
>>>  }
>>>  
>>> +static inline bool kvm_arm_aarch32_supported(CPUState *cs)
>>> +{
>>> +    return false;
>>> +}
>>> +
>>>  static inline int kvm_arm_get_max_vm_ipa_size(MachineState *ms)
>>>  {
>>>      return -ENOENT;
>>>
>> Reviewed-by: Eric Auger <eric.auger@redhat.com>
Thanks

Eric
>>
> 
> Thanks,
> drew 
> 

Re: [Qemu-devel] [PATCH v2 01/14] target/arm/cpu64: Ensure kvm really supports aarch64=off

[Andrew Jones]

On Wed, Jul 24, 2019 at 02:51:15PM +0200, Auger Eric wrote:
> Hi Drew,
> 
> On 6/25/19 3:34 PM, Andrew Jones wrote:
> > On Tue, Jun 25, 2019 at 11:35:12AM +0200, Auger Eric wrote:
> >> Hi Drew,
> >>
> >> On 6/21/19 6:34 PM, Andrew Jones wrote:
> >>> If -cpu <cpu>,aarch64=off is used then KVM must also be used, and it
> >>> and the host must support running the vcpu in 32-bit mode. Also, if
> s/and it//

"and it and the host" means "and KVM and the host", as 'it' refers to the
last subject, which is KVM. I wanted to point out both the host (machine)
and KVM (version of kernel with KVM) need to support the feature.

> >>> -cpu <cpu>,aarch64=on is used, then it doesn't matter if kvm is
> >>> enabled or not.
> >>>
> >>> Signed-off-by: Andrew Jones <drjones@redhat.com>
> >>
> >>
> >>> ---
> >>>  target/arm/cpu64.c   | 12 ++++++------
> >>>  target/arm/kvm64.c   | 11 +++++++++++
> >>>  target/arm/kvm_arm.h | 14 ++++++++++++++
> >>>  3 files changed, 31 insertions(+), 6 deletions(-)
> >>>
> >>> diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
> >>> index 1901997a0645..946994838d8a 100644
> >>> --- a/target/arm/cpu64.c
> >>> +++ b/target/arm/cpu64.c
> >>> @@ -407,13 +407,13 @@ static void aarch64_cpu_set_aarch64(Object *obj, bool value, Error **errp)
> >>>       * restriction allows us to avoid fixing up functionality that assumes a
> >>>       * uniform execution state like do_interrupt.
> >>>       */> -    if (!kvm_enabled()) {
> >>> -        error_setg(errp, "'aarch64' feature cannot be disabled "
> >>> -                         "unless KVM is enabled");
> >>> -        return;
> >>> -    }
> >>> -
> >>>      if (value == false) {
> >>> +        if (!kvm_enabled() || !kvm_arm_aarch32_supported(CPU(cpu))) {
> >>> +            error_setg(errp, "'aarch64' feature cannot be disabled "
> >>> +                             "unless KVM is enabled and 32-bit EL1 "
> >>> +                             "is supported");
> >>> +            return;
> >>> +        }
> >>>          unset_feature(&cpu->env, ARM_FEATURE_AARCH64);
> >>>      } else {
> >>>          set_feature(&cpu->env, ARM_FEATURE_AARCH64);
> >>> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> >>> index 22d19c9aec6f..45ccda589903 100644
> >>> --- a/target/arm/kvm64.c
> >>> +++ b/target/arm/kvm64.c
> >>> @@ -24,7 +24,9 @@
> >>>  #include "exec/gdbstub.h"
> >>>  #include "sysemu/sysemu.h"
> >>>  #include "sysemu/kvm.h"
> >>> +#include "sysemu/kvm_int.h"
> >>>  #include "kvm_arm.h"
> >>> +#include "hw/boards.h"
> By the way those two new headers are not needed by this patch

Really?

current_machine is defined in hw/boards.h and KVM_STATE is defined
in sysemu/kvm_int.h.

> >>>  #include "internals.h"
> >>>  
> >>>  static bool have_guest_debug;
> >>> @@ -593,6 +595,15 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
> >>>      return true;
> >>>  }
> >>>  
> >>> +bool kvm_arm_aarch32_supported(CPUState *cpu)
> >>> +{
> >>> +    KVMState *s = KVM_STATE(current_machine->accelerator);
> >>> +    int ret;
> >>> +
> >>> +    ret = kvm_check_extension(s, KVM_CAP_ARM_EL1_32BIT);
> >>> +    return ret > 0;
> >> nit: return kvm_check_extension() should be sufficient
> > 
> > Ah yes, I forgot kvm_check_extension() already converts negative
> > error codes to zero. I'll fix that for v3.
> > 
> >>> +}
> >>> +
> >>>  #define ARM_CPU_ID_MPIDR       3, 0, 0, 0, 5
> >>>  
> >>>  int kvm_arch_init_vcpu(CPUState *cs)
> >>> diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
> >>> index 2a07333c615f..812125f805a1 100644
> >>> --- a/target/arm/kvm_arm.h
> >>> +++ b/target/arm/kvm_arm.h
> >>> @@ -207,6 +207,15 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf);
> >>>   */
> >>>  void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu);
> >>>  
> >>> +/**
> >>> + * kvm_arm_aarch32_supported:
> >>> + * @cs: CPUState
> >> use kernel-doc comment style?
> > 
> > This file (kvm_arm.h) doesn't appear to have a super consistent comment
> > style. I see some use @var: for the parameters and some have 'Returns:
> > ...' lines as well. I'm happy to do whatever the maintainers prefer. For
> > now I was just trying to mimic whatever caught my eye.>
> >>> + *
> >>> + * Returns true if the KVM VCPU can enable AArch32 mode and false
> >>> + * otherwise.
> >>> + */
> >>> +bool kvm_arm_aarch32_supported(CPUState *cs);
> >>> +
> >>>  /**
> >>>   * kvm_arm_get_max_vm_ipa_size - Returns the number of bits in the
> >>>   * IPA address space supported by KVM
> >>> @@ -247,6 +256,11 @@ static inline void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu)
> >>>      cpu->host_cpu_probe_failed = true;
> >>>  }
> >>>  
> >>> +static inline bool kvm_arm_aarch32_supported(CPUState *cs)
> >>> +{
> >>> +    return false;
> >>> +}
> >>> +
> >>>  static inline int kvm_arm_get_max_vm_ipa_size(MachineState *ms)
> >>>  {
> >>>      return -ENOENT;
> >>>
> >> Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks,
drew

Re: [Qemu-devel] [PATCH v2 01/14] target/arm/cpu64: Ensure kvm really supports aarch64=off

[Auger Eric]

Hi Drew,

On 7/24/19 3:52 PM, Andrew Jones wrote:
> On Wed, Jul 24, 2019 at 02:51:15PM +0200, Auger Eric wrote:
>> Hi Drew,
>>
>> On 6/25/19 3:34 PM, Andrew Jones wrote:
>>> On Tue, Jun 25, 2019 at 11:35:12AM +0200, Auger Eric wrote:
>>>> Hi Drew,
>>>>
>>>> On 6/21/19 6:34 PM, Andrew Jones wrote:
>>>>> If -cpu <cpu>,aarch64=off is used then KVM must also be used, and it
>>>>> and the host must support running the vcpu in 32-bit mode. Also, if
>> s/and it//
> 
> "and it and the host" means "and KVM and the host", as 'it' refers to the
> last subject, which is KVM. I wanted to point out both the host (machine)
> and KVM (version of kernel with KVM) need to support the feature.
hum ok
> 
>>>>> -cpu <cpu>,aarch64=on is used, then it doesn't matter if kvm is
>>>>> enabled or not.
>>>>>
>>>>> Signed-off-by: Andrew Jones <drjones@redhat.com>
>>>>
>>>>
>>>>> ---
>>>>>  target/arm/cpu64.c   | 12 ++++++------
>>>>>  target/arm/kvm64.c   | 11 +++++++++++
>>>>>  target/arm/kvm_arm.h | 14 ++++++++++++++
>>>>>  3 files changed, 31 insertions(+), 6 deletions(-)
>>>>>
>>>>> diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
>>>>> index 1901997a0645..946994838d8a 100644
>>>>> --- a/target/arm/cpu64.c
>>>>> +++ b/target/arm/cpu64.c
>>>>> @@ -407,13 +407,13 @@ static void aarch64_cpu_set_aarch64(Object *obj, bool value, Error **errp)
>>>>>       * restriction allows us to avoid fixing up functionality that assumes a
>>>>>       * uniform execution state like do_interrupt.
>>>>>       */> -    if (!kvm_enabled()) {
>>>>> -        error_setg(errp, "'aarch64' feature cannot be disabled "
>>>>> -                         "unless KVM is enabled");
>>>>> -        return;
>>>>> -    }
>>>>> -
>>>>>      if (value == false) {
>>>>> +        if (!kvm_enabled() || !kvm_arm_aarch32_supported(CPU(cpu))) {
>>>>> +            error_setg(errp, "'aarch64' feature cannot be disabled "
>>>>> +                             "unless KVM is enabled and 32-bit EL1 "
>>>>> +                             "is supported");
>>>>> +            return;
>>>>> +        }
>>>>>          unset_feature(&cpu->env, ARM_FEATURE_AARCH64);
>>>>>      } else {
>>>>>          set_feature(&cpu->env, ARM_FEATURE_AARCH64);
>>>>> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
>>>>> index 22d19c9aec6f..45ccda589903 100644
>>>>> --- a/target/arm/kvm64.c
>>>>> +++ b/target/arm/kvm64.c
>>>>> @@ -24,7 +24,9 @@
>>>>>  #include "exec/gdbstub.h"
>>>>>  #include "sysemu/sysemu.h"
>>>>>  #include "sysemu/kvm.h"
>>>>> +#include "sysemu/kvm_int.h"
>>>>>  #include "kvm_arm.h"
>>>>> +#include "hw/boards.h"
>> By the way those two new headers are not needed by this patch
> 
> Really?
> 
> current_machine is defined in hw/boards.h and KVM_STATE is defined
> in sysemu/kvm_int.h.
argh my bad.

Sorry for the noise

Eric
> 
>>>>>  #include "internals.h"
>>>>>  
>>>>>  static bool have_guest_debug;
>>>>> @@ -593,6 +595,15 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
>>>>>      return true;
>>>>>  }
>>>>>  
>>>>> +bool kvm_arm_aarch32_supported(CPUState *cpu)
>>>>> +{
>>>>> +    KVMState *s = KVM_STATE(current_machine->accelerator);
>>>>> +    int ret;
>>>>> +
>>>>> +    ret = kvm_check_extension(s, KVM_CAP_ARM_EL1_32BIT);
>>>>> +    return ret > 0;
>>>> nit: return kvm_check_extension() should be sufficient
>>>
>>> Ah yes, I forgot kvm_check_extension() already converts negative
>>> error codes to zero. I'll fix that for v3.
>>>
>>>>> +}
>>>>> +
>>>>>  #define ARM_CPU_ID_MPIDR       3, 0, 0, 0, 5
>>>>>  
>>>>>  int kvm_arch_init_vcpu(CPUState *cs)
>>>>> diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
>>>>> index 2a07333c615f..812125f805a1 100644
>>>>> --- a/target/arm/kvm_arm.h
>>>>> +++ b/target/arm/kvm_arm.h
>>>>> @@ -207,6 +207,15 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf);
>>>>>   */
>>>>>  void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu);
>>>>>  
>>>>> +/**
>>>>> + * kvm_arm_aarch32_supported:
>>>>> + * @cs: CPUState
>>>> use kernel-doc comment style?
>>>
>>> This file (kvm_arm.h) doesn't appear to have a super consistent comment
>>> style. I see some use @var: for the parameters and some have 'Returns:
>>> ...' lines as well. I'm happy to do whatever the maintainers prefer. For
>>> now I was just trying to mimic whatever caught my eye.>
>>>>> + *
>>>>> + * Returns true if the KVM VCPU can enable AArch32 mode and false
>>>>> + * otherwise.
>>>>> + */
>>>>> +bool kvm_arm_aarch32_supported(CPUState *cs);
>>>>> +
>>>>>  /**
>>>>>   * kvm_arm_get_max_vm_ipa_size - Returns the number of bits in the
>>>>>   * IPA address space supported by KVM
>>>>> @@ -247,6 +256,11 @@ static inline void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu)
>>>>>      cpu->host_cpu_probe_failed = true;
>>>>>  }
>>>>>  
>>>>> +static inline bool kvm_arm_aarch32_supported(CPUState *cs)
>>>>> +{
>>>>> +    return false;
>>>>> +}
>>>>> +
>>>>>  static inline int kvm_arm_get_max_vm_ipa_size(MachineState *ms)
>>>>>  {
>>>>>      return -ENOENT;
>>>>>
>>>> Reviewed-by: Eric Auger <eric.auger@redhat.com>
> 
> Thanks,
> drew
> 

[Qemu-devel] [PATCH v2 02/14] target/arm/cpu: Ensure we can use the pmu with kvm

[Andrew Jones]

We first convert the pmu property from a static property to one with
its own accessors. Then we use the set accessor to check if the PMU is
supported when using KVM. Indeed a 32-bit KVM host does not support
the PMU, so this check will catch an attempt to use it at property-set
time.

Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 target/arm/cpu.c     | 30 +++++++++++++++++++++++++-----
 target/arm/kvm.c     |  9 +++++++++
 target/arm/kvm_arm.h | 13 +++++++++++++
 3 files changed, 47 insertions(+), 5 deletions(-)

diff --git a/target/arm/cpu.c b/target/arm/cpu.c
index 376db154f008..858f668d226e 100644
--- a/target/arm/cpu.c
+++ b/target/arm/cpu.c
@@ -759,10 +759,6 @@ static Property arm_cpu_has_el3_property =
 static Property arm_cpu_cfgend_property =
             DEFINE_PROP_BOOL("cfgend", ARMCPU, cfgend, false);
 
-/* use property name "pmu" to match other archs and virt tools */
-static Property arm_cpu_has_pmu_property =
-            DEFINE_PROP_BOOL("pmu", ARMCPU, has_pmu, true);
-
 static Property arm_cpu_has_vfp_property =
             DEFINE_PROP_BOOL("vfp", ARMCPU, has_vfp, true);
 
@@ -785,6 +781,29 @@ static Property arm_cpu_pmsav7_dregion_property =
                                            pmsav7_dregion,
                                            qdev_prop_uint32, uint32_t);
 
+static bool arm_get_pmu(Object *obj, Error **errp)
+{
+    ARMCPU *cpu = ARM_CPU(obj);
+
+    return cpu->has_pmu;
+}
+
+static void arm_set_pmu(Object *obj, bool value, Error **errp)
+{
+    ARMCPU *cpu = ARM_CPU(obj);
+
+    if (value) {
+        if (kvm_enabled() && !kvm_arm_pmu_supported(CPU(cpu))) {
+            error_setg(errp, "'pmu' feature not supported by KVM on this host");
+            return;
+        }
+        set_feature(&cpu->env, ARM_FEATURE_PMU);
+    } else {
+        unset_feature(&cpu->env, ARM_FEATURE_PMU);
+    }
+    cpu->has_pmu = value;
+}
+
 static void arm_get_init_svtor(Object *obj, Visitor *v, const char *name,
                                void *opaque, Error **errp)
 {
@@ -859,7 +878,8 @@ void arm_cpu_post_init(Object *obj)
     }
 
     if (arm_feature(&cpu->env, ARM_FEATURE_PMU)) {
-        qdev_property_add_static(DEVICE(obj), &arm_cpu_has_pmu_property,
+        cpu->has_pmu = true;
+        object_property_add_bool(obj, "pmu", arm_get_pmu, arm_set_pmu,
                                  &error_abort);
     }
 
diff --git a/target/arm/kvm.c b/target/arm/kvm.c
index fe4f461d4ef6..69c961a4c62c 100644
--- a/target/arm/kvm.c
+++ b/target/arm/kvm.c
@@ -162,6 +162,15 @@ void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu)
     env->features = arm_host_cpu_features.features;
 }
 
+bool kvm_arm_pmu_supported(CPUState *cpu)
+{
+    KVMState *s = KVM_STATE(current_machine->accelerator);
+    int ret;
+
+    ret = kvm_check_extension(s, KVM_CAP_ARM_PMU_V3);
+    return ret > 0;
+}
+
 int kvm_arm_get_max_vm_ipa_size(MachineState *ms)
 {
     KVMState *s = KVM_STATE(ms->accelerator);
diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
index 812125f805a1..e0ded3607996 100644
--- a/target/arm/kvm_arm.h
+++ b/target/arm/kvm_arm.h
@@ -216,6 +216,14 @@ void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu);
  */
 bool kvm_arm_aarch32_supported(CPUState *cs);
 
+/**
+ * bool kvm_arm_pmu_supported:
+ * @cs: CPUState
+ *
+ * Returns true if the KVM VCPU can enable its PMU and false otherwise.
+ */
+bool kvm_arm_pmu_supported(CPUState *cs);
+
 /**
  * kvm_arm_get_max_vm_ipa_size - Returns the number of bits in the
  * IPA address space supported by KVM
@@ -261,6 +269,11 @@ static inline bool kvm_arm_aarch32_supported(CPUState *cs)
     return false;
 }
 
+static inline bool kvm_arm_pmu_supported(CPUState *cs)
+{
+    return false;
+}
+
 static inline int kvm_arm_get_max_vm_ipa_size(MachineState *ms)
 {
     return -ENOENT;
-- 
2.20.1

Re: [Qemu-devel] [PATCH v2 02/14] target/arm/cpu: Ensure we can use the pmu with kvm

[Auger Eric]

Hi Drew,

On 6/21/19 6:34 PM, Andrew Jones wrote:
> We first convert the pmu property from a static property to one with
> its own accessors. Then we use the set accessor to check if the PMU is
> supported when using KVM. Indeed a 32-bit KVM host does not support
> the PMU, so this check will catch an attempt to use it at property-set
> time.
> 
> Signed-off-by: Andrew Jones <drjones@redhat.com>
> ---
>  target/arm/cpu.c     | 30 +++++++++++++++++++++++++-----
>  target/arm/kvm.c     |  9 +++++++++
>  target/arm/kvm_arm.h | 13 +++++++++++++
>  3 files changed, 47 insertions(+), 5 deletions(-)
> 
> diff --git a/target/arm/cpu.c b/target/arm/cpu.c
> index 376db154f008..858f668d226e 100644
> --- a/target/arm/cpu.c
> +++ b/target/arm/cpu.c
> @@ -759,10 +759,6 @@ static Property arm_cpu_has_el3_property =
>  static Property arm_cpu_cfgend_property =
>              DEFINE_PROP_BOOL("cfgend", ARMCPU, cfgend, false);
>  
> -/* use property name "pmu" to match other archs and virt tools */
> -static Property arm_cpu_has_pmu_property =
> -            DEFINE_PROP_BOOL("pmu", ARMCPU, has_pmu, true);
> -
>  static Property arm_cpu_has_vfp_property =
>              DEFINE_PROP_BOOL("vfp", ARMCPU, has_vfp, true);
>  
> @@ -785,6 +781,29 @@ static Property arm_cpu_pmsav7_dregion_property =
>                                             pmsav7_dregion,
>                                             qdev_prop_uint32, uint32_t);
>  
> +static bool arm_get_pmu(Object *obj, Error **errp)
> +{
> +    ARMCPU *cpu = ARM_CPU(obj);
> +
> +    return cpu->has_pmu;
> +}
> +
> +static void arm_set_pmu(Object *obj, bool value, Error **errp)
> +{
> +    ARMCPU *cpu = ARM_CPU(obj);
> +
> +    if (value) {
> +        if (kvm_enabled() && !kvm_arm_pmu_supported(CPU(cpu))) {
> +            error_setg(errp, "'pmu' feature not supported by KVM on this host");
> +            return;
> +        }
> +        set_feature(&cpu->env, ARM_FEATURE_PMU);
> +    } else {
> +        unset_feature(&cpu->env, ARM_FEATURE_PMU);
> +    }
> +    cpu->has_pmu = value;
> +}
> +
>  static void arm_get_init_svtor(Object *obj, Visitor *v, const char *name,
>                                 void *opaque, Error **errp)
>  {
> @@ -859,7 +878,8 @@ void arm_cpu_post_init(Object *obj)
>      }
>  
>      if (arm_feature(&cpu->env, ARM_FEATURE_PMU)) {
> -        qdev_property_add_static(DEVICE(obj), &arm_cpu_has_pmu_property,
> +        cpu->has_pmu = true;
> +        object_property_add_bool(obj, "pmu", arm_get_pmu, arm_set_pmu,
>                                   &error_abort);
>      }
>  
> diff --git a/target/arm/kvm.c b/target/arm/kvm.c
> index fe4f461d4ef6..69c961a4c62c 100644
> --- a/target/arm/kvm.c
> +++ b/target/arm/kvm.c
> @@ -162,6 +162,15 @@ void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu)
>      env->features = arm_host_cpu_features.features;
>  }
>  
> +bool kvm_arm_pmu_supported(CPUState *cpu)
> +{
> +    KVMState *s = KVM_STATE(current_machine->accelerator);
> +    int ret;
> +
> +    ret = kvm_check_extension(s, KVM_CAP_ARM_PMU_V3);
> +    return ret > 0;
return kvm_check_extension
> +}
> +
>  int kvm_arm_get_max_vm_ipa_size(MachineState *ms)
>  {
>      KVMState *s = KVM_STATE(ms->accelerator);
> diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
> index 812125f805a1..e0ded3607996 100644
> --- a/target/arm/kvm_arm.h
> +++ b/target/arm/kvm_arm.h
> @@ -216,6 +216,14 @@ void kvm_arm_set_cpu_features_from_host(ARMCPU *cpu);
>   */
>  bool kvm_arm_aarch32_supported(CPUState *cs);
>  
> +/**
> + * bool kvm_arm_pmu_supported:
> + * @cs: CPUState
kernel-doc comment style?
> + *
> + * Returns true if the KVM VCPU can enable its PMU and false otherwise.
> + */
> +bool kvm_arm_pmu_supported(CPUState *cs);
> +
>  /**
>   * kvm_arm_get_max_vm_ipa_size - Returns the number of bits in the
>   * IPA address space supported by KVM
> @@ -261,6 +269,11 @@ static inline bool kvm_arm_aarch32_supported(CPUState *cs)
>      return false;
>  }
>  
> +static inline bool kvm_arm_pmu_supported(CPUState *cs)
> +{
> +    return false;
> +}
> +
>  static inline int kvm_arm_get_max_vm_ipa_size(MachineState *ms)
>  {
>      return -ENOENT;
> 
Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks

Eric

Re: [Qemu-devel] [PATCH v2 02/14] target/arm/cpu: Ensure we can use the pmu with kvm

[Richard Henderson]

On 6/21/19 6:34 PM, Andrew Jones wrote:
> We first convert the pmu property from a static property to one with
> its own accessors. Then we use the set accessor to check if the PMU is
> supported when using KVM. Indeed a 32-bit KVM host does not support
> the PMU, so this check will catch an attempt to use it at property-set
> time.
> 
> Signed-off-by: Andrew Jones <drjones@redhat.com>
> ---
>  target/arm/cpu.c     | 30 +++++++++++++++++++++++++-----
>  target/arm/kvm.c     |  9 +++++++++
>  target/arm/kvm_arm.h | 13 +++++++++++++
>  3 files changed, 47 insertions(+), 5 deletions(-)
> 
> diff --git a/target/arm/cpu.c b/target/arm/cpu.c
> index 376db154f008..858f668d226e 100644
> --- a/target/arm/cpu.c
> +++ b/target/arm/cpu.c
> @@ -759,10 +759,6 @@ static Property arm_cpu_has_el3_property =
>  static Property arm_cpu_cfgend_property =
>              DEFINE_PROP_BOOL("cfgend", ARMCPU, cfgend, false);
>  
> -/* use property name "pmu" to match other archs and virt tools */
> -static Property arm_cpu_has_pmu_property =
> -            DEFINE_PROP_BOOL("pmu", ARMCPU, has_pmu, true);
> -
>  static Property arm_cpu_has_vfp_property =
>              DEFINE_PROP_BOOL("vfp", ARMCPU, has_vfp, true);
>  
> @@ -785,6 +781,29 @@ static Property arm_cpu_pmsav7_dregion_property =
>                                             pmsav7_dregion,
>                                             qdev_prop_uint32, uint32_t);
>  
> +static bool arm_get_pmu(Object *obj, Error **errp)
> +{
> +    ARMCPU *cpu = ARM_CPU(obj);
> +
> +    return cpu->has_pmu;
> +}
> +
> +static void arm_set_pmu(Object *obj, bool value, Error **errp)
> +{
> +    ARMCPU *cpu = ARM_CPU(obj);
> +
> +    if (value) {
> +        if (kvm_enabled() && !kvm_arm_pmu_supported(CPU(cpu))) {
> +            error_setg(errp, "'pmu' feature not supported by KVM on this host");
> +            return;
> +        }
> +        set_feature(&cpu->env, ARM_FEATURE_PMU);
> +    } else {
> +        unset_feature(&cpu->env, ARM_FEATURE_PMU);
> +    }
> +    cpu->has_pmu = value;
> +}
> +
>  static void arm_get_init_svtor(Object *obj, Visitor *v, const char *name,
>                                 void *opaque, Error **errp)
>  {
> @@ -859,7 +878,8 @@ void arm_cpu_post_init(Object *obj)
>      }
>  
>      if (arm_feature(&cpu->env, ARM_FEATURE_PMU)) {
> -        qdev_property_add_static(DEVICE(obj), &arm_cpu_has_pmu_property,
> +        cpu->has_pmu = true;
> +        object_property_add_bool(obj, "pmu", arm_get_pmu, arm_set_pmu,
>                                   &error_abort);

This doesn't look right.

The static property is only enabled here if the cpu is known to support the
PMU, and thus the only useful setting is -cpu foo,pmu=off.  Which means that
the extra checking that you do in the dynamic property is unused.

It seems like you need to be checking for the PMU earlier, e.g. in
kvm_arm_get_host_cpu_features.


r~

Re: [Qemu-devel] [PATCH v2 02/14] target/arm/cpu: Ensure we can use the pmu with kvm

[Andrew Jones]

On Wed, Jun 26, 2019 at 11:49:03AM +0200, Richard Henderson wrote:
> On 6/21/19 6:34 PM, Andrew Jones wrote:
> > We first convert the pmu property from a static property to one with
> > its own accessors. Then we use the set accessor to check if the PMU is
> > supported when using KVM. Indeed a 32-bit KVM host does not support
> > the PMU, so this check will catch an attempt to use it at property-set
> > time.
> > 
> > Signed-off-by: Andrew Jones <drjones@redhat.com>
> > ---
> >  target/arm/cpu.c     | 30 +++++++++++++++++++++++++-----
> >  target/arm/kvm.c     |  9 +++++++++
> >  target/arm/kvm_arm.h | 13 +++++++++++++
> >  3 files changed, 47 insertions(+), 5 deletions(-)
> > 
> > diff --git a/target/arm/cpu.c b/target/arm/cpu.c
> > index 376db154f008..858f668d226e 100644
> > --- a/target/arm/cpu.c
> > +++ b/target/arm/cpu.c
> > @@ -759,10 +759,6 @@ static Property arm_cpu_has_el3_property =
> >  static Property arm_cpu_cfgend_property =
> >              DEFINE_PROP_BOOL("cfgend", ARMCPU, cfgend, false);
> >  
> > -/* use property name "pmu" to match other archs and virt tools */
> > -static Property arm_cpu_has_pmu_property =
> > -            DEFINE_PROP_BOOL("pmu", ARMCPU, has_pmu, true);
> > -
> >  static Property arm_cpu_has_vfp_property =
> >              DEFINE_PROP_BOOL("vfp", ARMCPU, has_vfp, true);
> >  
> > @@ -785,6 +781,29 @@ static Property arm_cpu_pmsav7_dregion_property =
> >                                             pmsav7_dregion,
> >                                             qdev_prop_uint32, uint32_t);
> >  
> > +static bool arm_get_pmu(Object *obj, Error **errp)
> > +{
> > +    ARMCPU *cpu = ARM_CPU(obj);
> > +
> > +    return cpu->has_pmu;
> > +}
> > +
> > +static void arm_set_pmu(Object *obj, bool value, Error **errp)
> > +{
> > +    ARMCPU *cpu = ARM_CPU(obj);
> > +
> > +    if (value) {
> > +        if (kvm_enabled() && !kvm_arm_pmu_supported(CPU(cpu))) {
> > +            error_setg(errp, "'pmu' feature not supported by KVM on this host");
> > +            return;
> > +        }
> > +        set_feature(&cpu->env, ARM_FEATURE_PMU);
> > +    } else {
> > +        unset_feature(&cpu->env, ARM_FEATURE_PMU);
> > +    }
> > +    cpu->has_pmu = value;
> > +}
> > +
> >  static void arm_get_init_svtor(Object *obj, Visitor *v, const char *name,
> >                                 void *opaque, Error **errp)
> >  {
> > @@ -859,7 +878,8 @@ void arm_cpu_post_init(Object *obj)
> >      }
> >  
> >      if (arm_feature(&cpu->env, ARM_FEATURE_PMU)) {
> > -        qdev_property_add_static(DEVICE(obj), &arm_cpu_has_pmu_property,
> > +        cpu->has_pmu = true;
> > +        object_property_add_bool(obj, "pmu", arm_get_pmu, arm_set_pmu,
> >                                   &error_abort);
> 
> This doesn't look right.
> 
> The static property is only enabled here if the cpu is known to support the
> PMU, and thus the only useful setting is -cpu foo,pmu=off.  Which means that
> the extra checking that you do in the dynamic property is unused.

It's used when attempting to do '-cpu foo,pmu=on' on a cpu model that
would support the PMU on TCG, and thus has the property, but won't work
when KVM is in use. I'll admit I didn't test this, because I don't have
32-bit KVM hosts available, but I'm pretty sure it should work as
expected:

 -accel tcg -cpu foo                <-- on by default
 -accel tcg -cpu foo,pmu=off        <-- off
 -accel tcg -cpu foo,pmu=on         <-- nop

 -accel kvm -cpu foo                <-- on by default if kvm+foo supports
                                        the pmu, otherwise off
 -accel kvm -cpu foo,pmu=off        <-- off
 -accel kvm -cpu foo,pmu=on         <-- nop if kvm+foo supports the pmu,
                                        otherwise error message

With the error message, which will only occur with 32-bit kvm hosts, since
64-bit kvm hosts have all supported the pmu for quite some time, being the
only new thing.

> 
> It seems like you need to be checking for the PMU earlier, e.g. in
> kvm_arm_get_host_cpu_features.

We need to push all the checks into property accessors if we want the
QMP command to work for them.

Thanks,
drew

[Qemu-devel] [PATCH v2 03/14] target/arm/monitor: Introduce qmp_query_cpu_model_expansion

[Andrew Jones]

Add support for the query-cpu-model-expansion QMP command to Arm. We
do this selectively, only exposing CPU properties which represent
optional CPU features which the user may want to enable/disable. Also,
for simplicity, we restrict the list of queryable cpu models to 'max',
'host', or the current type when KVM is in use, even though there
may exist KVM hosts where other types would also work. For example on a
seattle you could use 'host' for the current type, but then attempt to
query 'cortex-a57', which is also a valid CPU type to use with KVM on
seattle hosts, but that query will fail with our simplifications. This
shouldn't be an issue though as management layers and users have been
preferring the 'host' CPU type for use with KVM for quite some time.
Additionally, if the KVM-enabled QEMU instance running on a seattle
host is using the cortex-a57 CPU type, then querying 'cortex-a57' will
work. Finally, we only implement expansion type 'full', as Arm does not
yet have a "base" CPU type. Below are some example calls and results
(to save character clutter they're not in json, but are still json-ish
to give the idea)

 # expand the 'max' CPU model
 query-cpu-model-expansion: type:full, model:{ name:max }

 return: model:{ name:max, props:{ 'aarch64': true, 'pmu': true }}

 # attempt to expand the 'max' CPU model with pmu=off
 query-cpu-model-expansion:
   type:full, model:{ name:max, props:{ 'pmu': false }}

 return: model:{ name:max, props:{ 'aarch64': true, 'pmu': false }}

 # attempt to expand the 'max' CPU model with aarch64=off
 query-cpu-model-expansion:
   type:full, model:{ name:max, props:{ 'aarch64': false }}

 error: "'aarch64' feature cannot be disabled unless KVM is enabled
         and 32-bit EL1 is supported"

In the last example KVM was not in use so an error was returned.

Note1: It's possible for features to have dependencies on other
features. I.e. it may be possible to change one feature at a time
without error, but when attempting to change all features at once
an error could occur depending on the order they are processed. It's
also possible changing all at once doesn't generate an error, because
a feature's dependencies are satisfied with other features, but the
same feature cannot be changed independently without error. For these
reasons callers should always attempt to make their desired changes
all at once in order to ensure the collection is valid.

Note2: Certainly more features may be added to the list of
advertised features, e.g. 'vfp' and 'neon'. The only requirement
is that their property set accessors fail when invalid
configurations are detected. For vfp we would need something like

 set_vfp()
 {
   if (arm_feature(env, ARM_FEATURE_AARCH64) &&
       cpu->has_vfp != cpu->has_neon)
       error("AArch64 CPUs must have both VFP and Neon or neither")

in its set accessor, and the same for neon, rather than doing that
check at realize time, which isn't executed at qmp query time.

Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 qapi/target.json     |   6 +-
 target/arm/monitor.c | 132 +++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 135 insertions(+), 3 deletions(-)

diff --git a/qapi/target.json b/qapi/target.json
index 1d4d54b6002e..edfa2f82b916 100644
--- a/qapi/target.json
+++ b/qapi/target.json
@@ -408,7 +408,7 @@
 ##
 { 'struct': 'CpuModelExpansionInfo',
   'data': { 'model': 'CpuModelInfo' },
-  'if': 'defined(TARGET_S390X) || defined(TARGET_I386)' }
+  'if': 'defined(TARGET_S390X) || defined(TARGET_I386) || defined(TARGET_ARM)' }
 
 ##
 # @query-cpu-model-expansion:
@@ -433,7 +433,7 @@
 #   query-cpu-model-expansion while using these is not advised.
 #
 # Some architectures may not support all expansion types. s390x supports
-# "full" and "static".
+# "full" and "static". Arm only supports "full".
 #
 # Returns: a CpuModelExpansionInfo. Returns an error if expanding CPU models is
 #          not supported, if the model cannot be expanded, if the model contains
@@ -447,7 +447,7 @@
   'data': { 'type': 'CpuModelExpansionType',
             'model': 'CpuModelInfo' },
   'returns': 'CpuModelExpansionInfo',
-  'if': 'defined(TARGET_S390X) || defined(TARGET_I386)' }
+  'if': 'defined(TARGET_S390X) || defined(TARGET_I386) || defined(TARGET_ARM)' }
 
 ##
 # @CpuDefinitionInfo:
diff --git a/target/arm/monitor.c b/target/arm/monitor.c
index 41b32b94b258..19e3120eef95 100644
--- a/target/arm/monitor.c
+++ b/target/arm/monitor.c
@@ -23,7 +23,13 @@
 #include "qemu/osdep.h"
 #include "hw/boards.h"
 #include "kvm_arm.h"
+#include "qapi/error.h"
+#include "qapi/visitor.h"
+#include "qapi/qobject-input-visitor.h"
 #include "qapi/qapi-commands-target.h"
+#include "qapi/qmp/qerror.h"
+#include "qapi/qmp/qdict.h"
+#include "qom/qom-qobject.h"
 
 static GICCapability *gic_cap_new(int version)
 {
@@ -82,3 +88,129 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)
 
     return head;
 }
+
+static const char *cpu_model_advertised_features[] = {
+    "aarch64", "pmu",
+    NULL
+};
+
+CpuModelExpansionInfo *qmp_query_cpu_model_expansion(CpuModelExpansionType type,
+                                                     CpuModelInfo *model,
+                                                     Error **errp)
+{
+    CpuModelExpansionInfo *expansion_info;
+    const QDict *qdict_in = NULL;
+    QDict *qdict_out;
+    ObjectClass *oc;
+    Object *obj;
+    const char *name;
+    int i;
+
+    if (type != CPU_MODEL_EXPANSION_TYPE_FULL) {
+        error_setg(errp, "The requested expansion type is not supported.");
+        return NULL;
+    }
+
+    if (!kvm_enabled() && !strcmp(model->name, "host")) {
+        error_setg(errp, "The CPU definition '%s' requires KVM", model->name);
+        return NULL;
+    }
+
+    oc = cpu_class_by_name(TYPE_ARM_CPU, model->name);
+    if (!oc) {
+        error_setg(errp, "The CPU definition '%s' is unknown.", model->name);
+        return NULL;
+    }
+
+    if (kvm_enabled()) {
+        const char *cpu_type = current_machine->cpu_type;
+        int len = strlen(cpu_type) - strlen(ARM_CPU_TYPE_SUFFIX);
+        bool supported = false;
+
+        if (!strcmp(model->name, "host") || !strcmp(model->name, "max")) {
+            /* These are kvmarm's recommended cpu types */
+            supported = true;
+        } else if (strlen(model->name) == len &&
+                   !strncmp(model->name, cpu_type, len)) {
+            /* KVM is enabled and we're using this type, so it works. */
+            supported = true;
+        }
+        if (!supported) {
+            error_setg(errp, "The CPU definition '%s' cannot "
+                             "be used with KVM on this host", model->name);
+            return NULL;
+        }
+    }
+
+    if (model->props) {
+        qdict_in = qobject_to(QDict, model->props);
+        if (!qdict_in) {
+            error_setg(errp, QERR_INVALID_PARAMETER_TYPE, "props", "dict");
+            return NULL;
+        }
+    }
+
+    obj = object_new(object_class_get_name(oc));
+
+    if (qdict_in) {
+        Visitor *visitor;
+
+        visitor = qobject_input_visitor_new(model->props);
+        visit_start_struct(visitor, NULL, NULL, 0, errp);
+        if (*errp) {
+            object_unref(obj);
+            return NULL;
+        }
+
+        i = 0;
+        while ((name = cpu_model_advertised_features[i++]) != NULL) {
+            if (qdict_get(qdict_in, name)) {
+                object_property_set(obj, visitor, name, errp);
+                if (*errp) {
+                    break;
+                }
+            }
+        }
+
+        if (!*errp) {
+            visit_check_struct(visitor, errp);
+        }
+        visit_end_struct(visitor, NULL);
+        visit_free(visitor);
+        if (*errp) {
+            object_unref(obj);
+            return NULL;
+        }
+    }
+
+    expansion_info = g_new0(CpuModelExpansionInfo, 1);
+    expansion_info->model = g_malloc0(sizeof(*expansion_info->model));
+    expansion_info->model->name = g_strdup(model->name);
+
+    qdict_out = qdict_new();
+
+    i = 0;
+    while ((name = cpu_model_advertised_features[i++]) != NULL) {
+        ObjectProperty *prop = object_property_find(obj, name, NULL);
+        if (prop) {
+            QObject *value;
+
+            assert(prop->get);
+            value = object_property_get_qobject(obj, name, errp);
+            assert(!*errp);
+
+            qdict_put_obj(qdict_out, name, value);
+        }
+    }
+
+    if (!qdict_size(qdict_out)) {
+        qobject_unref(qdict_out);
+    } else {
+        expansion_info->model->props = QOBJECT(qdict_out);
+        expansion_info->model->has_props = true;
+    }
+
+    object_unref(obj);
+
+    return expansion_info;
+}
-- 
2.20.1

Re: [Qemu-devel] [PATCH v2 03/14] target/arm/monitor: Introduce qmp_query_cpu_model_expansion

[Auger Eric]

Hi Drew,

On 6/21/19 6:34 PM, Andrew Jones wrote:
> Add support for the query-cpu-model-expansion QMP command to Arm. We
> do this selectively, only exposing CPU properties which represent
> optional CPU features which the user may want to enable/disable. Also,
> for simplicity, we restrict the list of queryable cpu models to 'max',
> 'host', or the current type when KVM is in use, even though there
> may exist KVM hosts where other types would also work. For example on a
> seattle you could use 'host' for the current type, but then attempt to
> query 'cortex-a57', which is also a valid CPU type to use with KVM on
> seattle hosts, but that query will fail with our simplifications. This
> shouldn't be an issue though as management layers and users have been
> preferring the 'host' CPU type for use with KVM for quite some time.
> Additionally, if the KVM-enabled QEMU instance running on a seattle
> host is using the cortex-a57 CPU type, then querying 'cortex-a57' will
> work. Finally, we only implement expansion type 'full', as Arm does not
> yet have a "base" CPU type. Below are some example calls and results
> (to save character clutter they're not in json, but are still json-ish
> to give the idea)
> 
>  # expand the 'max' CPU model
>  query-cpu-model-expansion: type:full, model:{ name:max }
> 
>  return: model:{ name:max, props:{ 'aarch64': true, 'pmu': true }}
> 
>  # attempt to expand the 'max' CPU model with pmu=off
>  query-cpu-model-expansion:
>    type:full, model:{ name:max, props:{ 'pmu': false }}
> 
>  return: model:{ name:max, props:{ 'aarch64': true, 'pmu': false }}
> 
>  # attempt to expand the 'max' CPU model with aarch64=off
>  query-cpu-model-expansion:
>    type:full, model:{ name:max, props:{ 'aarch64': false }}
> 
>  error: "'aarch64' feature cannot be disabled unless KVM is enabled
>          and 32-bit EL1 is supported"
> 
> In the last example KVM was not in use so an error was returned.
> 
> Note1: It's possible for features to have dependencies on other
> features. I.e. it may be possible to change one feature at a time
> without error, but when attempting to change all features at once
> an error could occur depending on the order they are processed. It's
> also possible changing all at once doesn't generate an error, because
> a feature's dependencies are satisfied with other features, but the
> same feature cannot be changed independently without error. For these
> reasons callers should always attempt to make their desired changes
> all at once in order to ensure the collection is valid.
> 
> Note2: Certainly more features may be added to the list of
> advertised features, e.g. 'vfp' and 'neon'. The only requirement
> is that their property set accessors fail when invalid
> configurations are detected. For vfp we would need something like
> 
>  set_vfp()
>  {
>    if (arm_feature(env, ARM_FEATURE_AARCH64) &&
>        cpu->has_vfp != cpu->has_neon)
>        error("AArch64 CPUs must have both VFP and Neon or neither")
> 
> in its set accessor, and the same for neon, rather than doing that
> check at realize time, which isn't executed at qmp query time.
> 
> Signed-off-by: Andrew Jones <drjones@redhat.com>
> ---
>  qapi/target.json     |   6 +-
>  target/arm/monitor.c | 132 +++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 135 insertions(+), 3 deletions(-)
> 
> diff --git a/qapi/target.json b/qapi/target.json
> index 1d4d54b6002e..edfa2f82b916 100644
> --- a/qapi/target.json
> +++ b/qapi/target.json
> @@ -408,7 +408,7 @@
>  ##
>  { 'struct': 'CpuModelExpansionInfo',
>    'data': { 'model': 'CpuModelInfo' },
> -  'if': 'defined(TARGET_S390X) || defined(TARGET_I386)' }
> +  'if': 'defined(TARGET_S390X) || defined(TARGET_I386) || defined(TARGET_ARM)' }
>  
>  ##
>  # @query-cpu-model-expansion:
> @@ -433,7 +433,7 @@
>  #   query-cpu-model-expansion while using these is not advised.
>  #
>  # Some architectures may not support all expansion types. s390x supports
> -# "full" and "static".
> +# "full" and "static". Arm only supports "full".
>  #
>  # Returns: a CpuModelExpansionInfo. Returns an error if expanding CPU models is
>  #          not supported, if the model cannot be expanded, if the model contains
> @@ -447,7 +447,7 @@
>    'data': { 'type': 'CpuModelExpansionType',
>              'model': 'CpuModelInfo' },
>    'returns': 'CpuModelExpansionInfo',
> -  'if': 'defined(TARGET_S390X) || defined(TARGET_I386)' }
> +  'if': 'defined(TARGET_S390X) || defined(TARGET_I386) || defined(TARGET_ARM)' }
>  
>  ##
>  # @CpuDefinitionInfo:
> diff --git a/target/arm/monitor.c b/target/arm/monitor.c
> index 41b32b94b258..19e3120eef95 100644
> --- a/target/arm/monitor.c
> +++ b/target/arm/monitor.c
> @@ -23,7 +23,13 @@
>  #include "qemu/osdep.h"
>  #include "hw/boards.h"
>  #include "kvm_arm.h"
> +#include "qapi/error.h"
> +#include "qapi/visitor.h"
> +#include "qapi/qobject-input-visitor.h"
>  #include "qapi/qapi-commands-target.h"
> +#include "qapi/qmp/qerror.h"
> +#include "qapi/qmp/qdict.h"
> +#include "qom/qom-qobject.h"
>  
>  static GICCapability *gic_cap_new(int version)
>  {
> @@ -82,3 +88,129 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)
>  
>      return head;
>  }
> +
> +static const char *cpu_model_advertised_features[] = {
> +    "aarch64", "pmu",
> +    NULL
> +};
> +
> +CpuModelExpansionInfo *qmp_query_cpu_model_expansion(CpuModelExpansionType type,
> +                                                     CpuModelInfo *model,
> +                                                     Error **errp)
> +{
> +    CpuModelExpansionInfo *expansion_info;
> +    const QDict *qdict_in = NULL;
> +    QDict *qdict_out;
> +    ObjectClass *oc;
> +    Object *obj;
> +    const char *name;
> +    int i;
> +
> +    if (type != CPU_MODEL_EXPANSION_TYPE_FULL) {
> +        error_setg(errp, "The requested expansion type is not supported.");
> +        return NULL;
> +    }
> +
> +    if (!kvm_enabled() && !strcmp(model->name, "host")) {
> +        error_setg(errp, "The CPU definition '%s' requires KVM", model->name);
> +        return NULL;
> +    }
> +
> +    oc = cpu_class_by_name(TYPE_ARM_CPU, model->name);
> +    if (!oc) {
> +        error_setg(errp, "The CPU definition '%s' is unknown.", model->name);
> +        return NULL;
> +    }
> +
> +    if (kvm_enabled()) {
> +        const char *cpu_type = current_machine->cpu_type;
> +        int len = strlen(cpu_type) - strlen(ARM_CPU_TYPE_SUFFIX);
> +        bool supported = false;
> +
> +        if (!strcmp(model->name, "host") || !strcmp(model->name, "max")) {
> +            /* These are kvmarm's recommended cpu types */
> +            supported = true;
> +        } else if (strlen(model->name) == len &&
> +                   !strncmp(model->name, cpu_type, len)) {
> +            /* KVM is enabled and we're using this type, so it works. */
> +            supported = true;
> +        }
> +        if (!supported) {
> +            error_setg(errp, "The CPU definition '%s' cannot "
use model name instead of CPU definition?
> +                             "be used with KVM on this host", model->name);

According to your commit mesg doesn't it mean that we fall into the
simplification you mentionned and not necessarily that the model name
cannot be used along with KVM?

> seattle you could use 'host' for the current type, but then attempt to
> query 'cortex-a57', which is also a valid CPU type to use with KVM on
> seattle hosts, but that query will fail with our simplifications
> +            return NULL;
> +        }
> +    }
> +
> +    if (model->props) {
> +        qdict_in = qobject_to(QDict, model->props);
> +        if (!qdict_in) {
> +            error_setg(errp, QERR_INVALID_PARAMETER_TYPE, "props", "dict");
> +            return NULL;
> +        }
> +    }
> +
> +    obj = object_new(object_class_get_name(oc));
> +
> +    if (qdict_in) {
> +        Visitor *visitor;
> +
> +        visitor = qobject_input_visitor_new(model->props);
> +        visit_start_struct(visitor, NULL, NULL, 0, errp);
> +        if (*errp) {
Normally we shouldn't do that as errp can be NULL. see /include/qapi/error.h
I see the same in cpu_model_from_info() by the way (s390x/cpu_models.c)
Maybe you can guarantee that errp isn't NULL but ...
> +            object_unref(obj);
> +            return NULL;
> +        }
> +
> +        i = 0;
> +        while ((name = cpu_model_advertised_features[i++]) != NULL) {
> +            if (qdict_get(qdict_in, name)) {
> +                object_property_set(obj, visitor, name, errp);
> +                if (*errp) {> +                    break;
I don't really get why we don't continue here instead of break. I see
that later we read the props back and populate the qdict_out object
> +                }
> +            }
> +        }
> +
> +        if (!*errp) {> +            visit_check_struct(visitor, errp);
> +        }
> +        visit_end_struct(visitor, NULL);
> +        visit_free(visitor);
> +        if (*errp) {
> +            object_unref(obj);
> +            return NULL;
> +        }
> +    }
> +
> +    expansion_info = g_new0(CpuModelExpansionInfo, 1);
> +    expansion_info->model = g_malloc0(sizeof(*expansion_info->model));
> +    expansion_info->model->name = g_strdup(model->name);
> +
> +    qdict_out = qdict_new();
> +
> +    i = 0;
> +    while ((name = cpu_model_advertised_features[i++]) != NULL) {
> +        ObjectProperty *prop = object_property_find(obj, name, NULL);
> +        if (prop) {
> +            QObject *value;
> +
> +            assert(prop->get);
> +            value = object_property_get_qobject(obj, name, errp);
> +            assert(!*errp);
> +
> +            qdict_put_obj(qdict_out, name, value);
> +        }
> +    }
> +
> +    if (!qdict_size(qdict_out)) {
> +        qobject_unref(qdict_out);
> +    } else {
> +        expansion_info->model->props = QOBJECT(qdict_out);
> +        expansion_info->model->has_props = true;
> +    }> +
> +    object_unref(obj);
> +
> +    return expansion_info;
> +}
> 
Thanks

Eric

Re: [Qemu-devel] [PATCH v2 03/14] target/arm/monitor: Introduce qmp_query_cpu_model_expansion

[Andrew Jones]

On Wed, Jun 26, 2019 at 09:43:09AM +0200, Auger Eric wrote:
> Hi Drew,
> 
> On 6/21/19 6:34 PM, Andrew Jones wrote:
> > Add support for the query-cpu-model-expansion QMP command to Arm. We
> > do this selectively, only exposing CPU properties which represent
> > optional CPU features which the user may want to enable/disable. Also,
> > for simplicity, we restrict the list of queryable cpu models to 'max',
> > 'host', or the current type when KVM is in use, even though there
> > may exist KVM hosts where other types would also work. For example on a
> > seattle you could use 'host' for the current type, but then attempt to
> > query 'cortex-a57', which is also a valid CPU type to use with KVM on
> > seattle hosts, but that query will fail with our simplifications. This
> > shouldn't be an issue though as management layers and users have been
> > preferring the 'host' CPU type for use with KVM for quite some time.
> > Additionally, if the KVM-enabled QEMU instance running on a seattle
> > host is using the cortex-a57 CPU type, then querying 'cortex-a57' will
> > work. Finally, we only implement expansion type 'full', as Arm does not
> > yet have a "base" CPU type. Below are some example calls and results
> > (to save character clutter they're not in json, but are still json-ish
> > to give the idea)
> > 
> >  # expand the 'max' CPU model
> >  query-cpu-model-expansion: type:full, model:{ name:max }
> > 
> >  return: model:{ name:max, props:{ 'aarch64': true, 'pmu': true }}
> > 
> >  # attempt to expand the 'max' CPU model with pmu=off
> >  query-cpu-model-expansion:
> >    type:full, model:{ name:max, props:{ 'pmu': false }}
> > 
> >  return: model:{ name:max, props:{ 'aarch64': true, 'pmu': false }}
> > 
> >  # attempt to expand the 'max' CPU model with aarch64=off
> >  query-cpu-model-expansion:
> >    type:full, model:{ name:max, props:{ 'aarch64': false }}
> > 
> >  error: "'aarch64' feature cannot be disabled unless KVM is enabled
> >          and 32-bit EL1 is supported"
> > 
> > In the last example KVM was not in use so an error was returned.
> > 
> > Note1: It's possible for features to have dependencies on other
> > features. I.e. it may be possible to change one feature at a time
> > without error, but when attempting to change all features at once
> > an error could occur depending on the order they are processed. It's
> > also possible changing all at once doesn't generate an error, because
> > a feature's dependencies are satisfied with other features, but the
> > same feature cannot be changed independently without error. For these
> > reasons callers should always attempt to make their desired changes
> > all at once in order to ensure the collection is valid.
> > 
> > Note2: Certainly more features may be added to the list of
> > advertised features, e.g. 'vfp' and 'neon'. The only requirement
> > is that their property set accessors fail when invalid
> > configurations are detected. For vfp we would need something like
> > 
> >  set_vfp()
> >  {
> >    if (arm_feature(env, ARM_FEATURE_AARCH64) &&
> >        cpu->has_vfp != cpu->has_neon)
> >        error("AArch64 CPUs must have both VFP and Neon or neither")
> > 
> > in its set accessor, and the same for neon, rather than doing that
> > check at realize time, which isn't executed at qmp query time.
> > 
> > Signed-off-by: Andrew Jones <drjones@redhat.com>
> > ---
> >  qapi/target.json     |   6 +-
> >  target/arm/monitor.c | 132 +++++++++++++++++++++++++++++++++++++++++++
> >  2 files changed, 135 insertions(+), 3 deletions(-)
> > 
> > diff --git a/qapi/target.json b/qapi/target.json
> > index 1d4d54b6002e..edfa2f82b916 100644
> > --- a/qapi/target.json
> > +++ b/qapi/target.json
> > @@ -408,7 +408,7 @@
> >  ##
> >  { 'struct': 'CpuModelExpansionInfo',
> >    'data': { 'model': 'CpuModelInfo' },
> > -  'if': 'defined(TARGET_S390X) || defined(TARGET_I386)' }
> > +  'if': 'defined(TARGET_S390X) || defined(TARGET_I386) || defined(TARGET_ARM)' }
> >  
> >  ##
> >  # @query-cpu-model-expansion:
> > @@ -433,7 +433,7 @@
> >  #   query-cpu-model-expansion while using these is not advised.
> >  #
> >  # Some architectures may not support all expansion types. s390x supports
> > -# "full" and "static".
> > +# "full" and "static". Arm only supports "full".
> >  #
> >  # Returns: a CpuModelExpansionInfo. Returns an error if expanding CPU models is
> >  #          not supported, if the model cannot be expanded, if the model contains
> > @@ -447,7 +447,7 @@
> >    'data': { 'type': 'CpuModelExpansionType',
> >              'model': 'CpuModelInfo' },
> >    'returns': 'CpuModelExpansionInfo',
> > -  'if': 'defined(TARGET_S390X) || defined(TARGET_I386)' }
> > +  'if': 'defined(TARGET_S390X) || defined(TARGET_I386) || defined(TARGET_ARM)' }
> >  
> >  ##
> >  # @CpuDefinitionInfo:
> > diff --git a/target/arm/monitor.c b/target/arm/monitor.c
> > index 41b32b94b258..19e3120eef95 100644
> > --- a/target/arm/monitor.c
> > +++ b/target/arm/monitor.c
> > @@ -23,7 +23,13 @@
> >  #include "qemu/osdep.h"
> >  #include "hw/boards.h"
> >  #include "kvm_arm.h"
> > +#include "qapi/error.h"
> > +#include "qapi/visitor.h"
> > +#include "qapi/qobject-input-visitor.h"
> >  #include "qapi/qapi-commands-target.h"
> > +#include "qapi/qmp/qerror.h"
> > +#include "qapi/qmp/qdict.h"
> > +#include "qom/qom-qobject.h"
> >  
> >  static GICCapability *gic_cap_new(int version)
> >  {
> > @@ -82,3 +88,129 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)
> >  
> >      return head;
> >  }
> > +
> > +static const char *cpu_model_advertised_features[] = {
> > +    "aarch64", "pmu",
> > +    NULL
> > +};
> > +
> > +CpuModelExpansionInfo *qmp_query_cpu_model_expansion(CpuModelExpansionType type,
> > +                                                     CpuModelInfo *model,
> > +                                                     Error **errp)
> > +{
> > +    CpuModelExpansionInfo *expansion_info;
> > +    const QDict *qdict_in = NULL;
> > +    QDict *qdict_out;
> > +    ObjectClass *oc;
> > +    Object *obj;
> > +    const char *name;
> > +    int i;
> > +
> > +    if (type != CPU_MODEL_EXPANSION_TYPE_FULL) {
> > +        error_setg(errp, "The requested expansion type is not supported.");
> > +        return NULL;
> > +    }
> > +
> > +    if (!kvm_enabled() && !strcmp(model->name, "host")) {
> > +        error_setg(errp, "The CPU definition '%s' requires KVM", model->name);
> > +        return NULL;
> > +    }
> > +
> > +    oc = cpu_class_by_name(TYPE_ARM_CPU, model->name);
> > +    if (!oc) {
> > +        error_setg(errp, "The CPU definition '%s' is unknown.", model->name);
> > +        return NULL;
> > +    }
> > +
> > +    if (kvm_enabled()) {
> > +        const char *cpu_type = current_machine->cpu_type;
> > +        int len = strlen(cpu_type) - strlen(ARM_CPU_TYPE_SUFFIX);
> > +        bool supported = false;
> > +
> > +        if (!strcmp(model->name, "host") || !strcmp(model->name, "max")) {
> > +            /* These are kvmarm's recommended cpu types */
> > +            supported = true;
> > +        } else if (strlen(model->name) == len &&
> > +                   !strncmp(model->name, cpu_type, len)) {
> > +            /* KVM is enabled and we're using this type, so it works. */
> > +            supported = true;
> > +        }
> > +        if (!supported) {
> > +            error_setg(errp, "The CPU definition '%s' cannot "
> use model name instead of CPU definition?

I took that wording from s390x, but maybe I prefer "The CPU type..."
better. I'll change it for v3.

> > +                             "be used with KVM on this host", model->name);
> 
> According to your commit mesg doesn't it mean that we fall into the
> simplification you mentionned and not necessarily that the model name
> cannot be used along with KVM?

There's no way to know that. The simplification is meant to avoid having
to know which models will work with KVM, because most don't, but some do.
Can you suggest wording you'd prefer if you don't want to make the error
message so absolute? I think I prefer keeping it simple like this and
just saying it doesn't work.

> 
> > seattle you could use 'host' for the current type, but then attempt to
> > query 'cortex-a57', which is also a valid CPU type to use with KVM on
> > seattle hosts, but that query will fail with our simplifications
> > +            return NULL;
> > +        }
> > +    }
> > +
> > +    if (model->props) {
> > +        qdict_in = qobject_to(QDict, model->props);
> > +        if (!qdict_in) {
> > +            error_setg(errp, QERR_INVALID_PARAMETER_TYPE, "props", "dict");
> > +            return NULL;
> > +        }
> > +    }
> > +
> > +    obj = object_new(object_class_get_name(oc));
> > +
> > +    if (qdict_in) {
> > +        Visitor *visitor;
> > +
> > +        visitor = qobject_input_visitor_new(model->props);
> > +        visit_start_struct(visitor, NULL, NULL, 0, errp);
> > +        if (*errp) {
> Normally we shouldn't do that as errp can be NULL. see /include/qapi/error.h
> I see the same in cpu_model_from_info() by the way (s390x/cpu_models.c)
> Maybe you can guarantee that errp isn't NULL but ...

Yeah, I know about the errp NULL thing, which is why I use local_err
elsewhere. I decided to follow s390x here though because I'm guessing
our QMP function will never be called with a NULL errp, it just
wouldn't work that way. Would you be satisfied with an assert(errp)
at the top of the function? Or should I switch all these to local_err
and then propagate?

> > +            object_unref(obj);
> > +            return NULL;
> > +        }
> > +
> > +        i = 0;
> > +        while ((name = cpu_model_advertised_features[i++]) != NULL) {
> > +            if (qdict_get(qdict_in, name)) {
> > +                object_property_set(obj, visitor, name, errp);
> > +                if (*errp) {> +                    break;
> I don't really get why we don't continue here instead of break. I see
> that later we read the props back and populate the qdict_out object

If we get an error here we're done and want to report it. If we continued
we'd lose that error with the next object_property_set() call. See a few
lines below where we free memory and return NULL due to this error.

> > +                }
> > +            }
> > +        }
> > +
> > +        if (!*errp) {> +            visit_check_struct(visitor, errp);
> > +        }
> > +        visit_end_struct(visitor, NULL);
> > +        visit_free(visitor);
> > +        if (*errp) {
> > +            object_unref(obj);
> > +            return NULL;
> > +        }
> > +    }
> > +
> > +    expansion_info = g_new0(CpuModelExpansionInfo, 1);
> > +    expansion_info->model = g_malloc0(sizeof(*expansion_info->model));
> > +    expansion_info->model->name = g_strdup(model->name);
> > +
> > +    qdict_out = qdict_new();
> > +
> > +    i = 0;
> > +    while ((name = cpu_model_advertised_features[i++]) != NULL) {
> > +        ObjectProperty *prop = object_property_find(obj, name, NULL);
> > +        if (prop) {
> > +            QObject *value;
> > +
> > +            assert(prop->get);
> > +            value = object_property_get_qobject(obj, name, errp);
> > +            assert(!*errp);
> > +
> > +            qdict_put_obj(qdict_out, name, value);
> > +        }
> > +    }
> > +
> > +    if (!qdict_size(qdict_out)) {
> > +        qobject_unref(qdict_out);
> > +    } else {
> > +        expansion_info->model->props = QOBJECT(qdict_out);
> > +        expansion_info->model->has_props = true;
> > +    }> +
> > +    object_unref(obj);
> > +
> > +    return expansion_info;
> > +}
> > 
> Thanks
> 
> Eric
> 

Re: [Qemu-devel] [PATCH v2 03/14] target/arm/monitor: Introduce qmp_query_cpu_model_expansion

[Auger Eric]

Hi Drew,

On 6/26/19 3:26 PM, Andrew Jones wrote:
> On Wed, Jun 26, 2019 at 09:43:09AM +0200, Auger Eric wrote:
>> Hi Drew,
>>
>> On 6/21/19 6:34 PM, Andrew Jones wrote:
>>> Add support for the query-cpu-model-expansion QMP command to Arm. We
>>> do this selectively, only exposing CPU properties which represent
>>> optional CPU features which the user may want to enable/disable. Also,
>>> for simplicity, we restrict the list of queryable cpu models to 'max',
>>> 'host', or the current type when KVM is in use, even though there
>>> may exist KVM hosts where other types would also work. For example on a
>>> seattle you could use 'host' for the current type, but then attempt to
>>> query 'cortex-a57', which is also a valid CPU type to use with KVM on
>>> seattle hosts, but that query will fail with our simplifications. This
>>> shouldn't be an issue though as management layers and users have been
>>> preferring the 'host' CPU type for use with KVM for quite some time.
>>> Additionally, if the KVM-enabled QEMU instance running on a seattle
>>> host is using the cortex-a57 CPU type, then querying 'cortex-a57' will
>>> work. Finally, we only implement expansion type 'full', as Arm does not
>>> yet have a "base" CPU type. Below are some example calls and results
>>> (to save character clutter they're not in json, but are still json-ish
>>> to give the idea)
>>>
>>>  # expand the 'max' CPU model
>>>  query-cpu-model-expansion: type:full, model:{ name:max }
>>>
>>>  return: model:{ name:max, props:{ 'aarch64': true, 'pmu': true }}
>>>
>>>  # attempt to expand the 'max' CPU model with pmu=off
>>>  query-cpu-model-expansion:
>>>    type:full, model:{ name:max, props:{ 'pmu': false }}
>>>
>>>  return: model:{ name:max, props:{ 'aarch64': true, 'pmu': false }}
>>>
>>>  # attempt to expand the 'max' CPU model with aarch64=off
>>>  query-cpu-model-expansion:
>>>    type:full, model:{ name:max, props:{ 'aarch64': false }}
>>>
>>>  error: "'aarch64' feature cannot be disabled unless KVM is enabled
>>>          and 32-bit EL1 is supported"
>>>
>>> In the last example KVM was not in use so an error was returned.
>>>
>>> Note1: It's possible for features to have dependencies on other
>>> features. I.e. it may be possible to change one feature at a time
>>> without error, but when attempting to change all features at once
>>> an error could occur depending on the order they are processed. It's
>>> also possible changing all at once doesn't generate an error, because
>>> a feature's dependencies are satisfied with other features, but the
>>> same feature cannot be changed independently without error. For these
>>> reasons callers should always attempt to make their desired changes
>>> all at once in order to ensure the collection is valid.
>>>
>>> Note2: Certainly more features may be added to the list of
>>> advertised features, e.g. 'vfp' and 'neon'. The only requirement
>>> is that their property set accessors fail when invalid
>>> configurations are detected. For vfp we would need something like
>>>
>>>  set_vfp()
>>>  {
>>>    if (arm_feature(env, ARM_FEATURE_AARCH64) &&
>>>        cpu->has_vfp != cpu->has_neon)
>>>        error("AArch64 CPUs must have both VFP and Neon or neither")
>>>
>>> in its set accessor, and the same for neon, rather than doing that
>>> check at realize time, which isn't executed at qmp query time.
>>>
>>> Signed-off-by: Andrew Jones <drjones@redhat.com>
>>> ---
>>>  qapi/target.json     |   6 +-
>>>  target/arm/monitor.c | 132 +++++++++++++++++++++++++++++++++++++++++++
>>>  2 files changed, 135 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/qapi/target.json b/qapi/target.json
>>> index 1d4d54b6002e..edfa2f82b916 100644
>>> --- a/qapi/target.json
>>> +++ b/qapi/target.json
>>> @@ -408,7 +408,7 @@
>>>  ##
>>>  { 'struct': 'CpuModelExpansionInfo',
>>>    'data': { 'model': 'CpuModelInfo' },
>>> -  'if': 'defined(TARGET_S390X) || defined(TARGET_I386)' }
>>> +  'if': 'defined(TARGET_S390X) || defined(TARGET_I386) || defined(TARGET_ARM)' }
>>>  
>>>  ##
>>>  # @query-cpu-model-expansion:
>>> @@ -433,7 +433,7 @@
>>>  #   query-cpu-model-expansion while using these is not advised.
>>>  #
>>>  # Some architectures may not support all expansion types. s390x supports
>>> -# "full" and "static".
>>> +# "full" and "static". Arm only supports "full".
>>>  #
>>>  # Returns: a CpuModelExpansionInfo. Returns an error if expanding CPU models is
>>>  #          not supported, if the model cannot be expanded, if the model contains
>>> @@ -447,7 +447,7 @@
>>>    'data': { 'type': 'CpuModelExpansionType',
>>>              'model': 'CpuModelInfo' },
>>>    'returns': 'CpuModelExpansionInfo',
>>> -  'if': 'defined(TARGET_S390X) || defined(TARGET_I386)' }
>>> +  'if': 'defined(TARGET_S390X) || defined(TARGET_I386) || defined(TARGET_ARM)' }
>>>  
>>>  ##
>>>  # @CpuDefinitionInfo:
>>> diff --git a/target/arm/monitor.c b/target/arm/monitor.c
>>> index 41b32b94b258..19e3120eef95 100644
>>> --- a/target/arm/monitor.c
>>> +++ b/target/arm/monitor.c
>>> @@ -23,7 +23,13 @@
>>>  #include "qemu/osdep.h"
>>>  #include "hw/boards.h"
>>>  #include "kvm_arm.h"
>>> +#include "qapi/error.h"
>>> +#include "qapi/visitor.h"
>>> +#include "qapi/qobject-input-visitor.h"
>>>  #include "qapi/qapi-commands-target.h"
>>> +#include "qapi/qmp/qerror.h"
>>> +#include "qapi/qmp/qdict.h"
>>> +#include "qom/qom-qobject.h"
>>>  
>>>  static GICCapability *gic_cap_new(int version)
>>>  {
>>> @@ -82,3 +88,129 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)
>>>  
>>>      return head;
>>>  }
>>> +
>>> +static const char *cpu_model_advertised_features[] = {
>>> +    "aarch64", "pmu",
>>> +    NULL
>>> +};
>>> +
>>> +CpuModelExpansionInfo *qmp_query_cpu_model_expansion(CpuModelExpansionType type,
>>> +                                                     CpuModelInfo *model,
>>> +                                                     Error **errp)
>>> +{
>>> +    CpuModelExpansionInfo *expansion_info;
>>> +    const QDict *qdict_in = NULL;
>>> +    QDict *qdict_out;
>>> +    ObjectClass *oc;
>>> +    Object *obj;
>>> +    const char *name;
>>> +    int i;
>>> +
>>> +    if (type != CPU_MODEL_EXPANSION_TYPE_FULL) {
>>> +        error_setg(errp, "The requested expansion type is not supported.");
>>> +        return NULL;
>>> +    }
>>> +
>>> +    if (!kvm_enabled() && !strcmp(model->name, "host")) {
>>> +        error_setg(errp, "The CPU definition '%s' requires KVM", model->name);
>>> +        return NULL;
>>> +    }
>>> +
>>> +    oc = cpu_class_by_name(TYPE_ARM_CPU, model->name);
>>> +    if (!oc) {
>>> +        error_setg(errp, "The CPU definition '%s' is unknown.", model->name);
>>> +        return NULL;
>>> +    }
>>> +
>>> +    if (kvm_enabled()) {
>>> +        const char *cpu_type = current_machine->cpu_type;
>>> +        int len = strlen(cpu_type) - strlen(ARM_CPU_TYPE_SUFFIX);
>>> +        bool supported = false;
>>> +
>>> +        if (!strcmp(model->name, "host") || !strcmp(model->name, "max")) {
>>> +            /* These are kvmarm's recommended cpu types */
>>> +            supported = true;
>>> +        } else if (strlen(model->name) == len &&
>>> +                   !strncmp(model->name, cpu_type, len)) {
>>> +            /* KVM is enabled and we're using this type, so it works. */
>>> +            supported = true;
>>> +        }
>>> +        if (!supported) {
>>> +            error_setg(errp, "The CPU definition '%s' cannot "
>> use model name instead of CPU definition?
> 
> I took that wording from s390x, but maybe I prefer "The CPU type..."
> better. I'll change it for v3.
This CPU type is not recognized as an ARM CPU type?
> 
>>> +                             "be used with KVM on this host", model->name);
>>
>> According to your commit mesg doesn't it mean that we fall into the
>> simplification you mentionned and not necessarily that the model name
>> cannot be used along with KVM?
> 
> There's no way to know that. The simplification is meant to avoid having
> to know which models will work with KVM, because most don't, but some do.
> Can you suggest wording you'd prefer if you don't want to make the error
> message so absolute? I think I prefer keeping it simple like this and
> just saying it doesn't work.
Something like:
"We cannot guarantee the CPU type %s works with KVM on this host"
> 
>>
>>> seattle you could use 'host' for the current type, but then attempt to
>>> query 'cortex-a57', which is also a valid CPU type to use with KVM on
>>> seattle hosts, but that query will fail with our simplifications
>>> +            return NULL;
>>> +        }
>>> +    }
>>> +
>>> +    if (model->props) {
>>> +        qdict_in = qobject_to(QDict, model->props);
>>> +        if (!qdict_in) {
>>> +            error_setg(errp, QERR_INVALID_PARAMETER_TYPE, "props", "dict");
>>> +            return NULL;
>>> +        }
>>> +    }
>>> +
>>> +    obj = object_new(object_class_get_name(oc));
>>> +
>>> +    if (qdict_in) {
>>> +        Visitor *visitor;
>>> +
>>> +        visitor = qobject_input_visitor_new(model->props);
>>> +        visit_start_struct(visitor, NULL, NULL, 0, errp);
>>> +        if (*errp) {
>> Normally we shouldn't do that as errp can be NULL. see /include/qapi/error.h
>> I see the same in cpu_model_from_info() by the way (s390x/cpu_models.c)
>> Maybe you can guarantee that errp isn't NULL but ...
> 
> Yeah, I know about the errp NULL thing, which is why I use local_err
> elsewhere. I decided to follow s390x here though because I'm guessing
> our QMP function will never be called with a NULL errp, it just
> wouldn't work that way. Would you be satisfied with an assert(errp)
> at the top of the function? Or should I switch all these to local_err
> and then propagate?
well up to maintainers. If it is not that much a pain, just propagate ;-)
> 
>>> +            object_unref(obj);
>>> +            return NULL;
>>> +        }
>>> +
>>> +        i = 0;
>>> +        while ((name = cpu_model_advertised_features[i++]) != NULL) {
>>> +            if (qdict_get(qdict_in, name)) {
>>> +                object_property_set(obj, visitor, name, errp);
>>> +                if (*errp) {> +                    break;
>> I don't really get why we don't continue here instead of break. I see
>> that later we read the props back and populate the qdict_out object
> 
> If we get an error here we're done and want to report it. If we continued
> we'd lose that error with the next object_property_set() call. See a few
> lines below where we free memory and return NULL due to this error.
I get it now.

Thanks

Eric
> 
>>> +                }
>>> +            }
>>> +        }
>>> +
>>> +        if (!*errp) {> +            visit_check_struct(visitor, errp);
>>> +        }
>>> +        visit_end_struct(visitor, NULL);
>>> +        visit_free(visitor);
>>> +        if (*errp) {
>>> +            object_unref(obj);
>>> +            return NULL;
>>> +        }
>>> +    }
>>> +
>>> +    expansion_info = g_new0(CpuModelExpansionInfo, 1);
>>> +    expansion_info->model = g_malloc0(sizeof(*expansion_info->model));
>>> +    expansion_info->model->name = g_strdup(model->name);
>>> +
>>> +    qdict_out = qdict_new();
>>> +
>>> +    i = 0;
>>> +    while ((name = cpu_model_advertised_features[i++]) != NULL) {
>>> +        ObjectProperty *prop = object_property_find(obj, name, NULL);
>>> +        if (prop) {
>>> +            QObject *value;
>>> +
>>> +            assert(prop->get);
>>> +            value = object_property_get_qobject(obj, name, errp);
>>> +            assert(!*errp);
>>> +
>>> +            qdict_put_obj(qdict_out, name, value);
>>> +        }
>>> +    }
>>> +
>>> +    if (!qdict_size(qdict_out)) {
>>> +        qobject_unref(qdict_out);
>>> +    } else {
>>> +        expansion_info->model->props = QOBJECT(qdict_out);
>>> +        expansion_info->model->has_props = true;
>>> +    }> +
>>> +    object_unref(obj);
>>> +
>>> +    return expansion_info;
>>> +}
>>>
>> Thanks
>>
>> Eric
>>

Re: [Qemu-devel] [PATCH v2 03/14] target/arm/monitor: Introduce qmp_query_cpu_model_expansion

[Andrew Jones]

On Wed, Jul 24, 2019 at 02:51:08PM +0200, Auger Eric wrote:
> Hi Drew,
> 
> On 6/26/19 3:26 PM, Andrew Jones wrote:
> > On Wed, Jun 26, 2019 at 09:43:09AM +0200, Auger Eric wrote:
> >> Hi Drew,
> >>
> >> On 6/21/19 6:34 PM, Andrew Jones wrote:
> >>> Add support for the query-cpu-model-expansion QMP command to Arm. We
> >>> do this selectively, only exposing CPU properties which represent
> >>> optional CPU features which the user may want to enable/disable. Also,
> >>> for simplicity, we restrict the list of queryable cpu models to 'max',
> >>> 'host', or the current type when KVM is in use, even though there
> >>> may exist KVM hosts where other types would also work. For example on a
> >>> seattle you could use 'host' for the current type, but then attempt to
> >>> query 'cortex-a57', which is also a valid CPU type to use with KVM on
> >>> seattle hosts, but that query will fail with our simplifications. This
> >>> shouldn't be an issue though as management layers and users have been
> >>> preferring the 'host' CPU type for use with KVM for quite some time.
> >>> Additionally, if the KVM-enabled QEMU instance running on a seattle
> >>> host is using the cortex-a57 CPU type, then querying 'cortex-a57' will
> >>> work. Finally, we only implement expansion type 'full', as Arm does not
> >>> yet have a "base" CPU type. Below are some example calls and results
> >>> (to save character clutter they're not in json, but are still json-ish
> >>> to give the idea)
> >>>
> >>>  # expand the 'max' CPU model
> >>>  query-cpu-model-expansion: type:full, model:{ name:max }
> >>>
> >>>  return: model:{ name:max, props:{ 'aarch64': true, 'pmu': true }}
> >>>
> >>>  # attempt to expand the 'max' CPU model with pmu=off
> >>>  query-cpu-model-expansion:
> >>>    type:full, model:{ name:max, props:{ 'pmu': false }}
> >>>
> >>>  return: model:{ name:max, props:{ 'aarch64': true, 'pmu': false }}
> >>>
> >>>  # attempt to expand the 'max' CPU model with aarch64=off
> >>>  query-cpu-model-expansion:
> >>>    type:full, model:{ name:max, props:{ 'aarch64': false }}
> >>>
> >>>  error: "'aarch64' feature cannot be disabled unless KVM is enabled
> >>>          and 32-bit EL1 is supported"
> >>>
> >>> In the last example KVM was not in use so an error was returned.
> >>>
> >>> Note1: It's possible for features to have dependencies on other
> >>> features. I.e. it may be possible to change one feature at a time
> >>> without error, but when attempting to change all features at once
> >>> an error could occur depending on the order they are processed. It's
> >>> also possible changing all at once doesn't generate an error, because
> >>> a feature's dependencies are satisfied with other features, but the
> >>> same feature cannot be changed independently without error. For these
> >>> reasons callers should always attempt to make their desired changes
> >>> all at once in order to ensure the collection is valid.
> >>>
> >>> Note2: Certainly more features may be added to the list of
> >>> advertised features, e.g. 'vfp' and 'neon'. The only requirement
> >>> is that their property set accessors fail when invalid
> >>> configurations are detected. For vfp we would need something like
> >>>
> >>>  set_vfp()
> >>>  {
> >>>    if (arm_feature(env, ARM_FEATURE_AARCH64) &&
> >>>        cpu->has_vfp != cpu->has_neon)
> >>>        error("AArch64 CPUs must have both VFP and Neon or neither")
> >>>
> >>> in its set accessor, and the same for neon, rather than doing that
> >>> check at realize time, which isn't executed at qmp query time.
> >>>
> >>> Signed-off-by: Andrew Jones <drjones@redhat.com>
> >>> ---
> >>>  qapi/target.json     |   6 +-
> >>>  target/arm/monitor.c | 132 +++++++++++++++++++++++++++++++++++++++++++
> >>>  2 files changed, 135 insertions(+), 3 deletions(-)
> >>>
> >>> diff --git a/qapi/target.json b/qapi/target.json
> >>> index 1d4d54b6002e..edfa2f82b916 100644
> >>> --- a/qapi/target.json
> >>> +++ b/qapi/target.json
> >>> @@ -408,7 +408,7 @@
> >>>  ##
> >>>  { 'struct': 'CpuModelExpansionInfo',
> >>>    'data': { 'model': 'CpuModelInfo' },
> >>> -  'if': 'defined(TARGET_S390X) || defined(TARGET_I386)' }
> >>> +  'if': 'defined(TARGET_S390X) || defined(TARGET_I386) || defined(TARGET_ARM)' }
> >>>  
> >>>  ##
> >>>  # @query-cpu-model-expansion:
> >>> @@ -433,7 +433,7 @@
> >>>  #   query-cpu-model-expansion while using these is not advised.
> >>>  #
> >>>  # Some architectures may not support all expansion types. s390x supports
> >>> -# "full" and "static".
> >>> +# "full" and "static". Arm only supports "full".
> >>>  #
> >>>  # Returns: a CpuModelExpansionInfo. Returns an error if expanding CPU models is
> >>>  #          not supported, if the model cannot be expanded, if the model contains
> >>> @@ -447,7 +447,7 @@
> >>>    'data': { 'type': 'CpuModelExpansionType',
> >>>              'model': 'CpuModelInfo' },
> >>>    'returns': 'CpuModelExpansionInfo',
> >>> -  'if': 'defined(TARGET_S390X) || defined(TARGET_I386)' }
> >>> +  'if': 'defined(TARGET_S390X) || defined(TARGET_I386) || defined(TARGET_ARM)' }
> >>>  
> >>>  ##
> >>>  # @CpuDefinitionInfo:
> >>> diff --git a/target/arm/monitor.c b/target/arm/monitor.c
> >>> index 41b32b94b258..19e3120eef95 100644
> >>> --- a/target/arm/monitor.c
> >>> +++ b/target/arm/monitor.c
> >>> @@ -23,7 +23,13 @@
> >>>  #include "qemu/osdep.h"
> >>>  #include "hw/boards.h"
> >>>  #include "kvm_arm.h"
> >>> +#include "qapi/error.h"
> >>> +#include "qapi/visitor.h"
> >>> +#include "qapi/qobject-input-visitor.h"
> >>>  #include "qapi/qapi-commands-target.h"
> >>> +#include "qapi/qmp/qerror.h"
> >>> +#include "qapi/qmp/qdict.h"
> >>> +#include "qom/qom-qobject.h"
> >>>  
> >>>  static GICCapability *gic_cap_new(int version)
> >>>  {
> >>> @@ -82,3 +88,129 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)
> >>>  
> >>>      return head;
> >>>  }
> >>> +
> >>> +static const char *cpu_model_advertised_features[] = {
> >>> +    "aarch64", "pmu",
> >>> +    NULL
> >>> +};
> >>> +
> >>> +CpuModelExpansionInfo *qmp_query_cpu_model_expansion(CpuModelExpansionType type,
> >>> +                                                     CpuModelInfo *model,
> >>> +                                                     Error **errp)
> >>> +{
> >>> +    CpuModelExpansionInfo *expansion_info;
> >>> +    const QDict *qdict_in = NULL;
> >>> +    QDict *qdict_out;
> >>> +    ObjectClass *oc;
> >>> +    Object *obj;
> >>> +    const char *name;
> >>> +    int i;
> >>> +
> >>> +    if (type != CPU_MODEL_EXPANSION_TYPE_FULL) {
> >>> +        error_setg(errp, "The requested expansion type is not supported.");
> >>> +        return NULL;
> >>> +    }
> >>> +
> >>> +    if (!kvm_enabled() && !strcmp(model->name, "host")) {
> >>> +        error_setg(errp, "The CPU definition '%s' requires KVM", model->name);
> >>> +        return NULL;
> >>> +    }
> >>> +
> >>> +    oc = cpu_class_by_name(TYPE_ARM_CPU, model->name);
> >>> +    if (!oc) {
> >>> +        error_setg(errp, "The CPU definition '%s' is unknown.", model->name);
> >>> +        return NULL;
> >>> +    }
> >>> +
> >>> +    if (kvm_enabled()) {
> >>> +        const char *cpu_type = current_machine->cpu_type;
> >>> +        int len = strlen(cpu_type) - strlen(ARM_CPU_TYPE_SUFFIX);
> >>> +        bool supported = false;
> >>> +
> >>> +        if (!strcmp(model->name, "host") || !strcmp(model->name, "max")) {
> >>> +            /* These are kvmarm's recommended cpu types */
> >>> +            supported = true;
> >>> +        } else if (strlen(model->name) == len &&
> >>> +                   !strncmp(model->name, cpu_type, len)) {
> >>> +            /* KVM is enabled and we're using this type, so it works. */
> >>> +            supported = true;
> >>> +        }
> >>> +        if (!supported) {
> >>> +            error_setg(errp, "The CPU definition '%s' cannot "
> >> use model name instead of CPU definition?
> > 
> > I took that wording from s390x, but maybe I prefer "The CPU type..."
> > better. I'll change it for v3.
> This CPU type is not recognized as an ARM CPU type?

That's not what this error message is stating. The CPU type may well be an
ARM CPU type, but it's not one you can expect to use with KVM enabled. I
currently have

  "The CPU type '%s' cannot "
  "be used with KVM on this host", model->name)

queued up for v3.

> > 
> >>> +                             "be used with KVM on this host", model->name);
> >>
> >> According to your commit mesg doesn't it mean that we fall into the
> >> simplification you mentionned and not necessarily that the model name
> >> cannot be used along with KVM?
> > 
> > There's no way to know that. The simplification is meant to avoid having
> > to know which models will work with KVM, because most don't, but some do.
> > Can you suggest wording you'd prefer if you don't want to make the error
> > message so absolute? I think I prefer keeping it simple like this and
> > just saying it doesn't work.
> Something like:
> "We cannot guarantee the CPU type %s works with KVM on this host"

OK, I can change to this one.

> > 
> >>
> >>> seattle you could use 'host' for the current type, but then attempt to
> >>> query 'cortex-a57', which is also a valid CPU type to use with KVM on
> >>> seattle hosts, but that query will fail with our simplifications
> >>> +            return NULL;
> >>> +        }
> >>> +    }
> >>> +
> >>> +    if (model->props) {
> >>> +        qdict_in = qobject_to(QDict, model->props);
> >>> +        if (!qdict_in) {
> >>> +            error_setg(errp, QERR_INVALID_PARAMETER_TYPE, "props", "dict");
> >>> +            return NULL;
> >>> +        }
> >>> +    }
> >>> +
> >>> +    obj = object_new(object_class_get_name(oc));
> >>> +
> >>> +    if (qdict_in) {
> >>> +        Visitor *visitor;
> >>> +
> >>> +        visitor = qobject_input_visitor_new(model->props);
> >>> +        visit_start_struct(visitor, NULL, NULL, 0, errp);
> >>> +        if (*errp) {
> >> Normally we shouldn't do that as errp can be NULL. see /include/qapi/error.h
> >> I see the same in cpu_model_from_info() by the way (s390x/cpu_models.c)
> >> Maybe you can guarantee that errp isn't NULL but ...
> > 
> > Yeah, I know about the errp NULL thing, which is why I use local_err
> > elsewhere. I decided to follow s390x here though because I'm guessing
> > our QMP function will never be called with a NULL errp, it just
> > wouldn't work that way. Would you be satisfied with an assert(errp)
> > at the top of the function? Or should I switch all these to local_err
> > and then propagate?
> well up to maintainers. If it is not that much a pain, just propagate ;-)
> > 

OK, I'll just propagate.

Thanks,
drew

Re: [Qemu-devel] [PATCH v2 03/14] target/arm/monitor: Introduce qmp_query_cpu_model_expansion

[Auger Eric]

Hi Drew,
On 7/24/19 4:05 PM, Andrew Jones wrote:
> On Wed, Jul 24, 2019 at 02:51:08PM +0200, Auger Eric wrote:
>> Hi Drew,
>>
>> On 6/26/19 3:26 PM, Andrew Jones wrote:
>>> On Wed, Jun 26, 2019 at 09:43:09AM +0200, Auger Eric wrote:
>>>> Hi Drew,
>>>>
>>>> On 6/21/19 6:34 PM, Andrew Jones wrote:
>>>>> Add support for the query-cpu-model-expansion QMP command to Arm. We
>>>>> do this selectively, only exposing CPU properties which represent
>>>>> optional CPU features which the user may want to enable/disable. Also,
>>>>> for simplicity, we restrict the list of queryable cpu models to 'max',
>>>>> 'host', or the current type when KVM is in use, even though there
>>>>> may exist KVM hosts where other types would also work. For example on a
>>>>> seattle you could use 'host' for the current type, but then attempt to
>>>>> query 'cortex-a57', which is also a valid CPU type to use with KVM on
>>>>> seattle hosts, but that query will fail with our simplifications. This
>>>>> shouldn't be an issue though as management layers and users have been
>>>>> preferring the 'host' CPU type for use with KVM for quite some time.
>>>>> Additionally, if the KVM-enabled QEMU instance running on a seattle
>>>>> host is using the cortex-a57 CPU type, then querying 'cortex-a57' will
>>>>> work. Finally, we only implement expansion type 'full', as Arm does not
>>>>> yet have a "base" CPU type. Below are some example calls and results
>>>>> (to save character clutter they're not in json, but are still json-ish
>>>>> to give the idea)
>>>>>
>>>>>  # expand the 'max' CPU model
>>>>>  query-cpu-model-expansion: type:full, model:{ name:max }
>>>>>
>>>>>  return: model:{ name:max, props:{ 'aarch64': true, 'pmu': true }}
>>>>>
>>>>>  # attempt to expand the 'max' CPU model with pmu=off
>>>>>  query-cpu-model-expansion:
>>>>>    type:full, model:{ name:max, props:{ 'pmu': false }}
>>>>>
>>>>>  return: model:{ name:max, props:{ 'aarch64': true, 'pmu': false }}
>>>>>
>>>>>  # attempt to expand the 'max' CPU model with aarch64=off
>>>>>  query-cpu-model-expansion:
>>>>>    type:full, model:{ name:max, props:{ 'aarch64': false }}
>>>>>
>>>>>  error: "'aarch64' feature cannot be disabled unless KVM is enabled
>>>>>          and 32-bit EL1 is supported"
>>>>>
>>>>> In the last example KVM was not in use so an error was returned.
>>>>>
>>>>> Note1: It's possible for features to have dependencies on other
>>>>> features. I.e. it may be possible to change one feature at a time
>>>>> without error, but when attempting to change all features at once
>>>>> an error could occur depending on the order they are processed. It's
>>>>> also possible changing all at once doesn't generate an error, because
>>>>> a feature's dependencies are satisfied with other features, but the
>>>>> same feature cannot be changed independently without error. For these
>>>>> reasons callers should always attempt to make their desired changes
>>>>> all at once in order to ensure the collection is valid.
>>>>>
>>>>> Note2: Certainly more features may be added to the list of
>>>>> advertised features, e.g. 'vfp' and 'neon'. The only requirement
>>>>> is that their property set accessors fail when invalid
>>>>> configurations are detected. For vfp we would need something like
>>>>>
>>>>>  set_vfp()
>>>>>  {
>>>>>    if (arm_feature(env, ARM_FEATURE_AARCH64) &&
>>>>>        cpu->has_vfp != cpu->has_neon)
>>>>>        error("AArch64 CPUs must have both VFP and Neon or neither")
>>>>>
>>>>> in its set accessor, and the same for neon, rather than doing that
>>>>> check at realize time, which isn't executed at qmp query time.
>>>>>
>>>>> Signed-off-by: Andrew Jones <drjones@redhat.com>
>>>>> ---
>>>>>  qapi/target.json     |   6 +-
>>>>>  target/arm/monitor.c | 132 +++++++++++++++++++++++++++++++++++++++++++
>>>>>  2 files changed, 135 insertions(+), 3 deletions(-)
>>>>>
>>>>> diff --git a/qapi/target.json b/qapi/target.json
>>>>> index 1d4d54b6002e..edfa2f82b916 100644
>>>>> --- a/qapi/target.json
>>>>> +++ b/qapi/target.json
>>>>> @@ -408,7 +408,7 @@
>>>>>  ##
>>>>>  { 'struct': 'CpuModelExpansionInfo',
>>>>>    'data': { 'model': 'CpuModelInfo' },
>>>>> -  'if': 'defined(TARGET_S390X) || defined(TARGET_I386)' }
>>>>> +  'if': 'defined(TARGET_S390X) || defined(TARGET_I386) || defined(TARGET_ARM)' }
>>>>>  
>>>>>  ##
>>>>>  # @query-cpu-model-expansion:
>>>>> @@ -433,7 +433,7 @@
>>>>>  #   query-cpu-model-expansion while using these is not advised.
>>>>>  #
>>>>>  # Some architectures may not support all expansion types. s390x supports
>>>>> -# "full" and "static".
>>>>> +# "full" and "static". Arm only supports "full".
>>>>>  #
>>>>>  # Returns: a CpuModelExpansionInfo. Returns an error if expanding CPU models is
>>>>>  #          not supported, if the model cannot be expanded, if the model contains
>>>>> @@ -447,7 +447,7 @@
>>>>>    'data': { 'type': 'CpuModelExpansionType',
>>>>>              'model': 'CpuModelInfo' },
>>>>>    'returns': 'CpuModelExpansionInfo',
>>>>> -  'if': 'defined(TARGET_S390X) || defined(TARGET_I386)' }
>>>>> +  'if': 'defined(TARGET_S390X) || defined(TARGET_I386) || defined(TARGET_ARM)' }
>>>>>  
>>>>>  ##
>>>>>  # @CpuDefinitionInfo:
>>>>> diff --git a/target/arm/monitor.c b/target/arm/monitor.c
>>>>> index 41b32b94b258..19e3120eef95 100644
>>>>> --- a/target/arm/monitor.c
>>>>> +++ b/target/arm/monitor.c
>>>>> @@ -23,7 +23,13 @@
>>>>>  #include "qemu/osdep.h"
>>>>>  #include "hw/boards.h"
>>>>>  #include "kvm_arm.h"
>>>>> +#include "qapi/error.h"
>>>>> +#include "qapi/visitor.h"
>>>>> +#include "qapi/qobject-input-visitor.h"
>>>>>  #include "qapi/qapi-commands-target.h"
>>>>> +#include "qapi/qmp/qerror.h"
>>>>> +#include "qapi/qmp/qdict.h"
>>>>> +#include "qom/qom-qobject.h"
>>>>>  
>>>>>  static GICCapability *gic_cap_new(int version)
>>>>>  {
>>>>> @@ -82,3 +88,129 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)
>>>>>  
>>>>>      return head;
>>>>>  }
>>>>> +
>>>>> +static const char *cpu_model_advertised_features[] = {
>>>>> +    "aarch64", "pmu",
>>>>> +    NULL
>>>>> +};
>>>>> +
>>>>> +CpuModelExpansionInfo *qmp_query_cpu_model_expansion(CpuModelExpansionType type,
>>>>> +                                                     CpuModelInfo *model,
>>>>> +                                                     Error **errp)
>>>>> +{
>>>>> +    CpuModelExpansionInfo *expansion_info;
>>>>> +    const QDict *qdict_in = NULL;
>>>>> +    QDict *qdict_out;
>>>>> +    ObjectClass *oc;
>>>>> +    Object *obj;
>>>>> +    const char *name;
>>>>> +    int i;
>>>>> +
>>>>> +    if (type != CPU_MODEL_EXPANSION_TYPE_FULL) {
>>>>> +        error_setg(errp, "The requested expansion type is not supported.");
>>>>> +        return NULL;
>>>>> +    }
>>>>> +
>>>>> +    if (!kvm_enabled() && !strcmp(model->name, "host")) {
>>>>> +        error_setg(errp, "The CPU definition '%s' requires KVM", model->name);
>>>>> +        return NULL;
>>>>> +    }
>>>>> +
>>>>> +    oc = cpu_class_by_name(TYPE_ARM_CPU, model->name);
>>>>> +    if (!oc) {
>>>>> +        error_setg(errp, "The CPU definition '%s' is unknown.", model->name);
>>>>> +        return NULL;
>>>>> +    }
>>>>> +
>>>>> +    if (kvm_enabled()) {
>>>>> +        const char *cpu_type = current_machine->cpu_type;
>>>>> +        int len = strlen(cpu_type) - strlen(ARM_CPU_TYPE_SUFFIX);
>>>>> +        bool supported = false;
>>>>> +
>>>>> +        if (!strcmp(model->name, "host") || !strcmp(model->name, "max")) {
>>>>> +            /* These are kvmarm's recommended cpu types */
>>>>> +            supported = true;
>>>>> +        } else if (strlen(model->name) == len &&
>>>>> +                   !strncmp(model->name, cpu_type, len)) {
>>>>> +            /* KVM is enabled and we're using this type, so it works. */
>>>>> +            supported = true;
>>>>> +        }
>>>>> +        if (!supported) {
>>>>> +            error_setg(errp, "The CPU definition '%s' cannot "
>>>> use model name instead of CPU definition?
>>>
>>> I took that wording from s390x, but maybe I prefer "The CPU type..."
>>> better. I'll change it for v3.>> This CPU type is not recognized as an ARM CPU type?
> 
> That's not what this error message is stating. The CPU type may well be an
> ARM CPU type, but it's not one you can expect to use with KVM enabled. I
> currently have
> 
>   "The CPU type '%s' cannot "
>   "be used with KVM on this host", model->name)
> 
> queued up for v3.

decidedly, I meant the error message associated to:

+    oc = cpu_class_by_name(TYPE_ARM_CPU, model->name);
+    if (!oc) {
+        error_setg(errp, "The CPU definition '%s' is unknown.",
model->name);
+        return NULL;
+    }

Why am I always looking at your series when we suffer heat wave?

Thanks

Eric
> 
>>>
>>>>> +                             "be used with KVM on this host", model->name);
>>>>
>>>> According to your commit mesg doesn't it mean that we fall into the
>>>> simplification you mentionned and not necessarily that the model name
>>>> cannot be used along with KVM?
>>>
>>> There's no way to know that. The simplification is meant to avoid having
>>> to know which models will work with KVM, because most don't, but some do.
>>> Can you suggest wording you'd prefer if you don't want to make the error
>>> message so absolute? I think I prefer keeping it simple like this and
>>> just saying it doesn't work.
>> Something like:
>> "We cannot guarantee the CPU type %s works with KVM on this host"
> 
> OK, I can change to this one.
> 
>>>
>>>>
>>>>> seattle you could use 'host' for the current type, but then attempt to
>>>>> query 'cortex-a57', which is also a valid CPU type to use with KVM on
>>>>> seattle hosts, but that query will fail with our simplifications
>>>>> +            return NULL;
>>>>> +        }
>>>>> +    }
>>>>> +
>>>>> +    if (model->props) {
>>>>> +        qdict_in = qobject_to(QDict, model->props);
>>>>> +        if (!qdict_in) {
>>>>> +            error_setg(errp, QERR_INVALID_PARAMETER_TYPE, "props", "dict");
>>>>> +            return NULL;
>>>>> +        }
>>>>> +    }
>>>>> +
>>>>> +    obj = object_new(object_class_get_name(oc));
>>>>> +
>>>>> +    if (qdict_in) {
>>>>> +        Visitor *visitor;
>>>>> +
>>>>> +        visitor = qobject_input_visitor_new(model->props);
>>>>> +        visit_start_struct(visitor, NULL, NULL, 0, errp);
>>>>> +        if (*errp) {
>>>> Normally we shouldn't do that as errp can be NULL. see /include/qapi/error.h
>>>> I see the same in cpu_model_from_info() by the way (s390x/cpu_models.c)
>>>> Maybe you can guarantee that errp isn't NULL but ...
>>>
>>> Yeah, I know about the errp NULL thing, which is why I use local_err
>>> elsewhere. I decided to follow s390x here though because I'm guessing
>>> our QMP function will never be called with a NULL errp, it just
>>> wouldn't work that way. Would you be satisfied with an assert(errp)
>>> at the top of the function? Or should I switch all these to local_err
>>> and then propagate?
>> well up to maintainers. If it is not that much a pain, just propagate ;-)
>>>
> 
> OK, I'll just propagate.
> 
> Thanks,
> drew
> 

Re: [Qemu-devel] [PATCH v2 03/14] target/arm/monitor: Introduce qmp_query_cpu_model_expansion

[Andrew Jones]

On Wed, Jul 24, 2019 at 04:25:32PM +0200, Auger Eric wrote:
> >>>>> diff --git a/target/arm/monitor.c b/target/arm/monitor.c
> >>>>> index 41b32b94b258..19e3120eef95 100644
> >>>>> --- a/target/arm/monitor.c
> >>>>> +++ b/target/arm/monitor.c
> >>>>> @@ -23,7 +23,13 @@
> >>>>>  #include "qemu/osdep.h"
> >>>>>  #include "hw/boards.h"
> >>>>>  #include "kvm_arm.h"
> >>>>> +#include "qapi/error.h"
> >>>>> +#include "qapi/visitor.h"
> >>>>> +#include "qapi/qobject-input-visitor.h"
> >>>>>  #include "qapi/qapi-commands-target.h"
> >>>>> +#include "qapi/qmp/qerror.h"
> >>>>> +#include "qapi/qmp/qdict.h"
> >>>>> +#include "qom/qom-qobject.h"
> >>>>>  
> >>>>>  static GICCapability *gic_cap_new(int version)
> >>>>>  {
> >>>>> @@ -82,3 +88,129 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)
> >>>>>  
> >>>>>      return head;
> >>>>>  }
> >>>>> +
> >>>>> +static const char *cpu_model_advertised_features[] = {
> >>>>> +    "aarch64", "pmu",
> >>>>> +    NULL
> >>>>> +};
> >>>>> +
> >>>>> +CpuModelExpansionInfo *qmp_query_cpu_model_expansion(CpuModelExpansionType type,
> >>>>> +                                                     CpuModelInfo *model,
> >>>>> +                                                     Error **errp)
> >>>>> +{
> >>>>> +    CpuModelExpansionInfo *expansion_info;
> >>>>> +    const QDict *qdict_in = NULL;
> >>>>> +    QDict *qdict_out;
> >>>>> +    ObjectClass *oc;
> >>>>> +    Object *obj;
> >>>>> +    const char *name;
> >>>>> +    int i;
> >>>>> +
> >>>>> +    if (type != CPU_MODEL_EXPANSION_TYPE_FULL) {
> >>>>> +        error_setg(errp, "The requested expansion type is not supported.");
> >>>>> +        return NULL;
> >>>>> +    }
> >>>>> +
> >>>>> +    if (!kvm_enabled() && !strcmp(model->name, "host")) {
> >>>>> +        error_setg(errp, "The CPU definition '%s' requires KVM", model->name);
> >>>>> +        return NULL;
> >>>>> +    }
> >>>>> +
> >>>>> +    oc = cpu_class_by_name(TYPE_ARM_CPU, model->name);
> >>>>> +    if (!oc) {
> >>>>> +        error_setg(errp, "The CPU definition '%s' is unknown.", model->name);
> >>>>> +        return NULL;
> >>>>> +    }
> >>>>> +
> >>>>> +    if (kvm_enabled()) {
> >>>>> +        const char *cpu_type = current_machine->cpu_type;
> >>>>> +        int len = strlen(cpu_type) - strlen(ARM_CPU_TYPE_SUFFIX);
> >>>>> +        bool supported = false;
> >>>>> +
> >>>>> +        if (!strcmp(model->name, "host") || !strcmp(model->name, "max")) {
> >>>>> +            /* These are kvmarm's recommended cpu types */
> >>>>> +            supported = true;
> >>>>> +        } else if (strlen(model->name) == len &&
> >>>>> +                   !strncmp(model->name, cpu_type, len)) {
> >>>>> +            /* KVM is enabled and we're using this type, so it works. */
> >>>>> +            supported = true;
> >>>>> +        }
> >>>>> +        if (!supported) {
> >>>>> +            error_setg(errp, "The CPU definition '%s' cannot "
> >>>> use model name instead of CPU definition?
> >>>
> >>> I took that wording from s390x, but maybe I prefer "The CPU type..."
> >>> better. I'll change it for v3.>> This CPU type is not recognized as an ARM CPU type?
> > 
> > That's not what this error message is stating. The CPU type may well be an
> > ARM CPU type, but it's not one you can expect to use with KVM enabled. I
> > currently have
> > 
> >   "The CPU type '%s' cannot "
> >   "be used with KVM on this host", model->name)
> > 
> > queued up for v3.
> 
> decidedly, I meant the error message associated to:
> 
> +    oc = cpu_class_by_name(TYPE_ARM_CPU, model->name);
> +    if (!oc) {
> +        error_setg(errp, "The CPU definition '%s' is unknown.",
> model->name);
> +        return NULL;
> +    }

Ah, OK. Yeah I can change that one too. Of course if we deviate from
s390x's generic error messages for common errors, then we're assuming
the messages aren't being parsed by upper layers using code that we'd
like to easily adopt to ARM. But, I think that assumption is reasonable.

> 
> Why am I always looking at your series when we suffer heat wave?
>

Climate change generates too many heat waves. Or I generate too much
code that requires comments. Or both.

Thanks,
drew

Re: [Qemu-devel] [PATCH v2 03/14] target/arm/monitor: Introduce qmp_query_cpu_model_expansion

[Auger Eric]

Hi,
On 6/26/19 3:26 PM, Andrew Jones wrote:
> On Wed, Jun 26, 2019 at 09:43:09AM +0200, Auger Eric wrote:
>> Hi Drew,
>>
>> On 6/21/19 6:34 PM, Andrew Jones wrote:
>>> Add support for the query-cpu-model-expansion QMP command to Arm. We
>>> do this selectively, only exposing CPU properties which represent
>>> optional CPU features which the user may want to enable/disable. Also,
>>> for simplicity, we restrict the list of queryable cpu models to 'max',
>>> 'host', or the current type when KVM is in use, even though there
>>> may exist KVM hosts where other types would also work. For example on a
>>> seattle you could use 'host' for the current type, but then attempt to
>>> query 'cortex-a57', which is also a valid CPU type to use with KVM on
>>> seattle hosts, but that query will fail with our simplifications. This
>>> shouldn't be an issue though as management layers and users have been
>>> preferring the 'host' CPU type for use with KVM for quite some time.
>>> Additionally, if the KVM-enabled QEMU instance running on a seattle
>>> host is using the cortex-a57 CPU type, then querying 'cortex-a57' will
>>> work. Finally, we only implement expansion type 'full', as Arm does not
>>> yet have a "base" CPU type. Below are some example calls and results
>>> (to save character clutter they're not in json, but are still json-ish
>>> to give the idea)
>>>
>>>  # expand the 'max' CPU model
>>>  query-cpu-model-expansion: type:full, model:{ name:max }
>>>
>>>  return: model:{ name:max, props:{ 'aarch64': true, 'pmu': true }}
>>>
>>>  # attempt to expand the 'max' CPU model with pmu=off
>>>  query-cpu-model-expansion:
>>>    type:full, model:{ name:max, props:{ 'pmu': false }}
>>>
>>>  return: model:{ name:max, props:{ 'aarch64': true, 'pmu': false }}
>>>
>>>  # attempt to expand the 'max' CPU model with aarch64=off
>>>  query-cpu-model-expansion:
>>>    type:full, model:{ name:max, props:{ 'aarch64': false }}
>>>
>>>  error: "'aarch64' feature cannot be disabled unless KVM is enabled
>>>          and 32-bit EL1 is supported"
>>>
>>> In the last example KVM was not in use so an error was returned.
>>>
>>> Note1: It's possible for features to have dependencies on other
>>> features. I.e. it may be possible to change one feature at a time
>>> without error, but when attempting to change all features at once
>>> an error could occur depending on the order they are processed. It's
>>> also possible changing all at once doesn't generate an error, because
>>> a feature's dependencies are satisfied with other features, but the
>>> same feature cannot be changed independently without error. For these
>>> reasons callers should always attempt to make their desired changes
>>> all at once in order to ensure the collection is valid.
>>>
>>> Note2: Certainly more features may be added to the list of
>>> advertised features, e.g. 'vfp' and 'neon'. The only requirement
>>> is that their property set accessors fail when invalid
>>> configurations are detected. For vfp we would need something like
>>>
>>>  set_vfp()
>>>  {
>>>    if (arm_feature(env, ARM_FEATURE_AARCH64) &&
>>>        cpu->has_vfp != cpu->has_neon)
>>>        error("AArch64 CPUs must have both VFP and Neon or neither")
>>>
>>> in its set accessor, and the same for neon, rather than doing that
>>> check at realize time, which isn't executed at qmp query time.
>>>
>>> Signed-off-by: Andrew Jones <drjones@redhat.com>
>>> ---
>>>  qapi/target.json     |   6 +-
>>>  target/arm/monitor.c | 132 +++++++++++++++++++++++++++++++++++++++++++
>>>  2 files changed, 135 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/qapi/target.json b/qapi/target.json
>>> index 1d4d54b6002e..edfa2f82b916 100644
>>> --- a/qapi/target.json
>>> +++ b/qapi/target.json
>>> @@ -408,7 +408,7 @@
>>>  ##
>>>  { 'struct': 'CpuModelExpansionInfo',
>>>    'data': { 'model': 'CpuModelInfo' },
>>> -  'if': 'defined(TARGET_S390X) || defined(TARGET_I386)' }
>>> +  'if': 'defined(TARGET_S390X) || defined(TARGET_I386) || defined(TARGET_ARM)' }
>>>  
>>>  ##
>>>  # @query-cpu-model-expansion:
>>> @@ -433,7 +433,7 @@
>>>  #   query-cpu-model-expansion while using these is not advised.
>>>  #
>>>  # Some architectures may not support all expansion types. s390x supports
>>> -# "full" and "static".
>>> +# "full" and "static". Arm only supports "full".
>>>  #
>>>  # Returns: a CpuModelExpansionInfo. Returns an error if expanding CPU models is
>>>  #          not supported, if the model cannot be expanded, if the model contains
>>> @@ -447,7 +447,7 @@
>>>    'data': { 'type': 'CpuModelExpansionType',
>>>              'model': 'CpuModelInfo' },
>>>    'returns': 'CpuModelExpansionInfo',
>>> -  'if': 'defined(TARGET_S390X) || defined(TARGET_I386)' }
>>> +  'if': 'defined(TARGET_S390X) || defined(TARGET_I386) || defined(TARGET_ARM)' }
>>>  
>>>  ##
>>>  # @CpuDefinitionInfo:
>>> diff --git a/target/arm/monitor.c b/target/arm/monitor.c
>>> index 41b32b94b258..19e3120eef95 100644
>>> --- a/target/arm/monitor.c
>>> +++ b/target/arm/monitor.c
>>> @@ -23,7 +23,13 @@
>>>  #include "qemu/osdep.h"
>>>  #include "hw/boards.h"
>>>  #include "kvm_arm.h"
>>> +#include "qapi/error.h"
>>> +#include "qapi/visitor.h"
>>> +#include "qapi/qobject-input-visitor.h"
>>>  #include "qapi/qapi-commands-target.h"
>>> +#include "qapi/qmp/qerror.h"
>>> +#include "qapi/qmp/qdict.h"
>>> +#include "qom/qom-qobject.h"
>>>  
>>>  static GICCapability *gic_cap_new(int version)
>>>  {
>>> @@ -82,3 +88,129 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)
>>>  
>>>      return head;
>>>  }
>>> +
>>> +static const char *cpu_model_advertised_features[] = {
>>> +    "aarch64", "pmu",
>>> +    NULL
>>> +};
>>> +
>>> +CpuModelExpansionInfo *qmp_query_cpu_model_expansion(CpuModelExpansionType type,
>>> +                                                     CpuModelInfo *model,
>>> +                                                     Error **errp)
>>> +{
>>> +    CpuModelExpansionInfo *expansion_info;
>>> +    const QDict *qdict_in = NULL;
>>> +    QDict *qdict_out;
>>> +    ObjectClass *oc;
>>> +    Object *obj;
>>> +    const char *name;
>>> +    int i;
>>> +
>>> +    if (type != CPU_MODEL_EXPANSION_TYPE_FULL) {
>>> +        error_setg(errp, "The requested expansion type is not supported.");
>>> +        return NULL;
>>> +    }
>>> +
>>> +    if (!kvm_enabled() && !strcmp(model->name, "host")) {
>>> +        error_setg(errp, "The CPU definition '%s' requires KVM", model->name);
>>> +        return NULL;
>>> +    }
>>> +
>>> +    oc = cpu_class_by_name(TYPE_ARM_CPU, model->name);
>>> +    if (!oc) {
>>> +        error_setg(errp, "The CPU definition '%s' is unknown.", model->name);
>>> +        return NULL;
>>> +    }
>>> +
>>> +    if (kvm_enabled()) {
>>> +        const char *cpu_type = current_machine->cpu_type;
>>> +        int len = strlen(cpu_type) - strlen(ARM_CPU_TYPE_SUFFIX);
>>> +        bool supported = false;
>>> +
>>> +        if (!strcmp(model->name, "host") || !strcmp(model->name, "max")) {
>>> +            /* These are kvmarm's recommended cpu types */
>>> +            supported = true;
>>> +        } else if (strlen(model->name) == len &&
>>> +                   !strncmp(model->name, cpu_type, len)) {
>>> +            /* KVM is enabled and we're using this type, so it works. */
>>> +            supported = true;
>>> +        }
>>> +        if (!supported) {
>>> +            error_setg(errp, "The CPU definition '%s' cannot "
>> use model name instead of CPU definition?
> 
> I took that wording from s390x, but maybe I prefer "The CPU type..."
> better. I'll change it for v3.
> 
>>> +                             "be used with KVM on this host", model->name);
>>
>> According to your commit mesg doesn't it mean that we fall into the
>> simplification you mentionned and not necessarily that the model name
>> cannot be used along with KVM?
> 
> There's no way to know that. The simplification is meant to avoid having
> to know which models will work with KVM, because most don't, but some do.
> Can you suggest wording you'd prefer if you don't want to make the error
> message so absolute? I think I prefer keeping it simple like this and
> just saying it doesn't work.
> 
>>
>>> seattle you could use 'host' for the current type, but then attempt to
>>> query 'cortex-a57', which is also a valid CPU type to use with KVM on
>>> seattle hosts, but that query will fail with our simplifications
>>> +            return NULL;
>>> +        }
>>> +    }
>>> +
>>> +    if (model->props) {
>>> +        qdict_in = qobject_to(QDict, model->props);
>>> +        if (!qdict_in) {
>>> +            error_setg(errp, QERR_INVALID_PARAMETER_TYPE, "props", "dict");
>>> +            return NULL;
>>> +        }
>>> +    }
>>> +
>>> +    obj = object_new(object_class_get_name(oc));
>>> +
>>> +    if (qdict_in) {
>>> +        Visitor *visitor;
>>> +
>>> +        visitor = qobject_input_visitor_new(model->props);
>>> +        visit_start_struct(visitor, NULL, NULL, 0, errp);
>>> +        if (*errp) {
>> Normally we shouldn't do that as errp can be NULL. see /include/qapi/error.h
>> I see the same in cpu_model_from_info() by the way (s390x/cpu_models.c)
>> Maybe you can guarantee that errp isn't NULL but ...
> 
> Yeah, I know about the errp NULL thing, which is why I use local_err
> elsewhere. I decided to follow s390x here though because I'm guessing
> our QMP function will never be called with a NULL errp, it just
> wouldn't work that way. Would you be satisfied with an assert(errp)
> at the top of the function? Or should I switch all these to local_err
> and then propagate?
> 
>>> +            object_unref(obj);
>>> +            return NULL;
>>> +        }
>>> +
>>> +        i = 0;
>>> +        while ((name = cpu_model_advertised_features[i++]) != NULL) {
>>> +            if (qdict_get(qdict_in, name)) {
>>> +                object_property_set(obj, visitor, name, errp);
>>> +                if (*errp) {> +                    break;
>> I don't really get why we don't continue here instead of break. I see
>> that later we read the props back and populate the qdict_out object
> 
> If we get an error here we're done and want to report it. If we continued
> we'd lose that error with the next object_property_set() call. See a few
> lines below where we free memory and return NULL due to this error.

By the way, if you were to use local_err, you could propagate them
successively to errp and you wouldn't loose any of them. This would
allow to report several errors at a time.

Thanks

Eric
> 
>>> +                }
>>> +            }
>>> +        }
>>> +
>>> +        if (!*errp) {> +            visit_check_struct(visitor, errp);
>>> +        }
>>> +        visit_end_struct(visitor, NULL);
>>> +        visit_free(visitor);
>>> +        if (*errp) {
>>> +            object_unref(obj);
>>> +            return NULL;
>>> +        }
>>> +    }
>>> +
>>> +    expansion_info = g_new0(CpuModelExpansionInfo, 1);
>>> +    expansion_info->model = g_malloc0(sizeof(*expansion_info->model));
>>> +    expansion_info->model->name = g_strdup(model->name);
>>> +
>>> +    qdict_out = qdict_new();
>>> +
>>> +    i = 0;
>>> +    while ((name = cpu_model_advertised_features[i++]) != NULL) {
>>> +        ObjectProperty *prop = object_property_find(obj, name, NULL);
>>> +        if (prop) {
>>> +            QObject *value;
>>> +
>>> +            assert(prop->get);
>>> +            value = object_property_get_qobject(obj, name, errp);
>>> +            assert(!*errp);
>>> +
>>> +            qdict_put_obj(qdict_out, name, value);
>>> +        }
>>> +    }
>>> +
>>> +    if (!qdict_size(qdict_out)) {
>>> +        qobject_unref(qdict_out);
>>> +    } else {
>>> +        expansion_info->model->props = QOBJECT(qdict_out);
>>> +        expansion_info->model->has_props = true;
>>> +    }> +
>>> +    object_unref(obj);
>>> +
>>> +    return expansion_info;
>>> +}
>>>
>> Thanks
>>
>> Eric
>>
> 

Re: [Qemu-devel] [PATCH v2 03/14] target/arm/monitor: Introduce qmp_query_cpu_model_expansion

[Andrew Jones]

On Wed, Jul 24, 2019 at 02:55:39PM +0200, Auger Eric wrote:
> Hi,
> On 6/26/19 3:26 PM, Andrew Jones wrote:
> > On Wed, Jun 26, 2019 at 09:43:09AM +0200, Auger Eric wrote:
> >> Hi Drew,
> >>
> >> On 6/21/19 6:34 PM, Andrew Jones wrote:
> >>> Add support for the query-cpu-model-expansion QMP command to Arm. We
> >>> do this selectively, only exposing CPU properties which represent
> >>> optional CPU features which the user may want to enable/disable. Also,
> >>> for simplicity, we restrict the list of queryable cpu models to 'max',
> >>> 'host', or the current type when KVM is in use, even though there
> >>> may exist KVM hosts where other types would also work. For example on a
> >>> seattle you could use 'host' for the current type, but then attempt to
> >>> query 'cortex-a57', which is also a valid CPU type to use with KVM on
> >>> seattle hosts, but that query will fail with our simplifications. This
> >>> shouldn't be an issue though as management layers and users have been
> >>> preferring the 'host' CPU type for use with KVM for quite some time.
> >>> Additionally, if the KVM-enabled QEMU instance running on a seattle
> >>> host is using the cortex-a57 CPU type, then querying 'cortex-a57' will
> >>> work. Finally, we only implement expansion type 'full', as Arm does not
> >>> yet have a "base" CPU type. Below are some example calls and results
> >>> (to save character clutter they're not in json, but are still json-ish
> >>> to give the idea)
> >>>
> >>>  # expand the 'max' CPU model
> >>>  query-cpu-model-expansion: type:full, model:{ name:max }
> >>>
> >>>  return: model:{ name:max, props:{ 'aarch64': true, 'pmu': true }}
> >>>
> >>>  # attempt to expand the 'max' CPU model with pmu=off
> >>>  query-cpu-model-expansion:
> >>>    type:full, model:{ name:max, props:{ 'pmu': false }}
> >>>
> >>>  return: model:{ name:max, props:{ 'aarch64': true, 'pmu': false }}
> >>>
> >>>  # attempt to expand the 'max' CPU model with aarch64=off
> >>>  query-cpu-model-expansion:
> >>>    type:full, model:{ name:max, props:{ 'aarch64': false }}
> >>>
> >>>  error: "'aarch64' feature cannot be disabled unless KVM is enabled
> >>>          and 32-bit EL1 is supported"
> >>>
> >>> In the last example KVM was not in use so an error was returned.
> >>>
> >>> Note1: It's possible for features to have dependencies on other
> >>> features. I.e. it may be possible to change one feature at a time
> >>> without error, but when attempting to change all features at once
> >>> an error could occur depending on the order they are processed. It's
> >>> also possible changing all at once doesn't generate an error, because
> >>> a feature's dependencies are satisfied with other features, but the
> >>> same feature cannot be changed independently without error. For these
> >>> reasons callers should always attempt to make their desired changes
> >>> all at once in order to ensure the collection is valid.
> >>>
> >>> Note2: Certainly more features may be added to the list of
> >>> advertised features, e.g. 'vfp' and 'neon'. The only requirement
> >>> is that their property set accessors fail when invalid
> >>> configurations are detected. For vfp we would need something like
> >>>
> >>>  set_vfp()
> >>>  {
> >>>    if (arm_feature(env, ARM_FEATURE_AARCH64) &&
> >>>        cpu->has_vfp != cpu->has_neon)
> >>>        error("AArch64 CPUs must have both VFP and Neon or neither")
> >>>
> >>> in its set accessor, and the same for neon, rather than doing that
> >>> check at realize time, which isn't executed at qmp query time.
> >>>
> >>> Signed-off-by: Andrew Jones <drjones@redhat.com>
> >>> ---
> >>>  qapi/target.json     |   6 +-
> >>>  target/arm/monitor.c | 132 +++++++++++++++++++++++++++++++++++++++++++
> >>>  2 files changed, 135 insertions(+), 3 deletions(-)
> >>>
> >>> diff --git a/qapi/target.json b/qapi/target.json
> >>> index 1d4d54b6002e..edfa2f82b916 100644
> >>> --- a/qapi/target.json
> >>> +++ b/qapi/target.json
> >>> @@ -408,7 +408,7 @@
> >>>  ##
> >>>  { 'struct': 'CpuModelExpansionInfo',
> >>>    'data': { 'model': 'CpuModelInfo' },
> >>> -  'if': 'defined(TARGET_S390X) || defined(TARGET_I386)' }
> >>> +  'if': 'defined(TARGET_S390X) || defined(TARGET_I386) || defined(TARGET_ARM)' }
> >>>  
> >>>  ##
> >>>  # @query-cpu-model-expansion:
> >>> @@ -433,7 +433,7 @@
> >>>  #   query-cpu-model-expansion while using these is not advised.
> >>>  #
> >>>  # Some architectures may not support all expansion types. s390x supports
> >>> -# "full" and "static".
> >>> +# "full" and "static". Arm only supports "full".
> >>>  #
> >>>  # Returns: a CpuModelExpansionInfo. Returns an error if expanding CPU models is
> >>>  #          not supported, if the model cannot be expanded, if the model contains
> >>> @@ -447,7 +447,7 @@
> >>>    'data': { 'type': 'CpuModelExpansionType',
> >>>              'model': 'CpuModelInfo' },
> >>>    'returns': 'CpuModelExpansionInfo',
> >>> -  'if': 'defined(TARGET_S390X) || defined(TARGET_I386)' }
> >>> +  'if': 'defined(TARGET_S390X) || defined(TARGET_I386) || defined(TARGET_ARM)' }
> >>>  
> >>>  ##
> >>>  # @CpuDefinitionInfo:
> >>> diff --git a/target/arm/monitor.c b/target/arm/monitor.c
> >>> index 41b32b94b258..19e3120eef95 100644
> >>> --- a/target/arm/monitor.c
> >>> +++ b/target/arm/monitor.c
> >>> @@ -23,7 +23,13 @@
> >>>  #include "qemu/osdep.h"
> >>>  #include "hw/boards.h"
> >>>  #include "kvm_arm.h"
> >>> +#include "qapi/error.h"
> >>> +#include "qapi/visitor.h"
> >>> +#include "qapi/qobject-input-visitor.h"
> >>>  #include "qapi/qapi-commands-target.h"
> >>> +#include "qapi/qmp/qerror.h"
> >>> +#include "qapi/qmp/qdict.h"
> >>> +#include "qom/qom-qobject.h"
> >>>  
> >>>  static GICCapability *gic_cap_new(int version)
> >>>  {
> >>> @@ -82,3 +88,129 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)
> >>>  
> >>>      return head;
> >>>  }
> >>> +
> >>> +static const char *cpu_model_advertised_features[] = {
> >>> +    "aarch64", "pmu",
> >>> +    NULL
> >>> +};
> >>> +
> >>> +CpuModelExpansionInfo *qmp_query_cpu_model_expansion(CpuModelExpansionType type,
> >>> +                                                     CpuModelInfo *model,
> >>> +                                                     Error **errp)
> >>> +{
> >>> +    CpuModelExpansionInfo *expansion_info;
> >>> +    const QDict *qdict_in = NULL;
> >>> +    QDict *qdict_out;
> >>> +    ObjectClass *oc;
> >>> +    Object *obj;
> >>> +    const char *name;
> >>> +    int i;
> >>> +
> >>> +    if (type != CPU_MODEL_EXPANSION_TYPE_FULL) {
> >>> +        error_setg(errp, "The requested expansion type is not supported.");
> >>> +        return NULL;
> >>> +    }
> >>> +
> >>> +    if (!kvm_enabled() && !strcmp(model->name, "host")) {
> >>> +        error_setg(errp, "The CPU definition '%s' requires KVM", model->name);
> >>> +        return NULL;
> >>> +    }
> >>> +
> >>> +    oc = cpu_class_by_name(TYPE_ARM_CPU, model->name);
> >>> +    if (!oc) {
> >>> +        error_setg(errp, "The CPU definition '%s' is unknown.", model->name);
> >>> +        return NULL;
> >>> +    }
> >>> +
> >>> +    if (kvm_enabled()) {
> >>> +        const char *cpu_type = current_machine->cpu_type;
> >>> +        int len = strlen(cpu_type) - strlen(ARM_CPU_TYPE_SUFFIX);
> >>> +        bool supported = false;
> >>> +
> >>> +        if (!strcmp(model->name, "host") || !strcmp(model->name, "max")) {
> >>> +            /* These are kvmarm's recommended cpu types */
> >>> +            supported = true;
> >>> +        } else if (strlen(model->name) == len &&
> >>> +                   !strncmp(model->name, cpu_type, len)) {
> >>> +            /* KVM is enabled and we're using this type, so it works. */
> >>> +            supported = true;
> >>> +        }
> >>> +        if (!supported) {
> >>> +            error_setg(errp, "The CPU definition '%s' cannot "
> >> use model name instead of CPU definition?
> > 
> > I took that wording from s390x, but maybe I prefer "The CPU type..."
> > better. I'll change it for v3.
> > 
> >>> +                             "be used with KVM on this host", model->name);
> >>
> >> According to your commit mesg doesn't it mean that we fall into the
> >> simplification you mentionned and not necessarily that the model name
> >> cannot be used along with KVM?
> > 
> > There's no way to know that. The simplification is meant to avoid having
> > to know which models will work with KVM, because most don't, but some do.
> > Can you suggest wording you'd prefer if you don't want to make the error
> > message so absolute? I think I prefer keeping it simple like this and
> > just saying it doesn't work.
> > 
> >>
> >>> seattle you could use 'host' for the current type, but then attempt to
> >>> query 'cortex-a57', which is also a valid CPU type to use with KVM on
> >>> seattle hosts, but that query will fail with our simplifications
> >>> +            return NULL;
> >>> +        }
> >>> +    }
> >>> +
> >>> +    if (model->props) {
> >>> +        qdict_in = qobject_to(QDict, model->props);
> >>> +        if (!qdict_in) {
> >>> +            error_setg(errp, QERR_INVALID_PARAMETER_TYPE, "props", "dict");
> >>> +            return NULL;
> >>> +        }
> >>> +    }
> >>> +
> >>> +    obj = object_new(object_class_get_name(oc));
> >>> +
> >>> +    if (qdict_in) {
> >>> +        Visitor *visitor;
> >>> +
> >>> +        visitor = qobject_input_visitor_new(model->props);
> >>> +        visit_start_struct(visitor, NULL, NULL, 0, errp);
> >>> +        if (*errp) {
> >> Normally we shouldn't do that as errp can be NULL. see /include/qapi/error.h
> >> I see the same in cpu_model_from_info() by the way (s390x/cpu_models.c)
> >> Maybe you can guarantee that errp isn't NULL but ...
> > 
> > Yeah, I know about the errp NULL thing, which is why I use local_err
> > elsewhere. I decided to follow s390x here though because I'm guessing
> > our QMP function will never be called with a NULL errp, it just
> > wouldn't work that way. Would you be satisfied with an assert(errp)
> > at the top of the function? Or should I switch all these to local_err
> > and then propagate?
> > 
> >>> +            object_unref(obj);
> >>> +            return NULL;
> >>> +        }
> >>> +
> >>> +        i = 0;
> >>> +        while ((name = cpu_model_advertised_features[i++]) != NULL) {
> >>> +            if (qdict_get(qdict_in, name)) {
> >>> +                object_property_set(obj, visitor, name, errp);
> >>> +                if (*errp) {> +                    break;
> >> I don't really get why we don't continue here instead of break. I see
> >> that later we read the props back and populate the qdict_out object
> > 
> > If we get an error here we're done and want to report it. If we continued
> > we'd lose that error with the next object_property_set() call. See a few
> > lines below where we free memory and return NULL due to this error.
> 
> By the way, if you were to use local_err, you could propagate them
> successively to errp and you wouldn't loose any of them. This would
> allow to report several errors at a time.
>

Interesting suggestion. I'll experiment with it, but I think the QMP error
will only provide the caller the first one anyway.

Thanks,
drew

Re: [Qemu-devel] [PATCH v2 03/14] target/arm/monitor: Introduce qmp_query_cpu_model_expansion

[Auger Eric]

Hi Drew,

On 6/21/19 6:34 PM, Andrew Jones wrote:
> Add support for the query-cpu-model-expansion QMP command to Arm. We
> do this selectively, only exposing CPU properties which represent
> optional CPU features which the user may want to enable/disable. Also,
> for simplicity, we restrict the list of queryable cpu models to 'max',
> 'host', or the current type when KVM is in use, even though there
> may exist KVM hosts where other types would also work. For example on a
> seattle you could use 'host' for the current type, but then attempt to
> query 'cortex-a57', which is also a valid CPU type to use with KVM on
> seattle hosts, but that query will fail with our simplifications. This
> shouldn't be an issue though as management layers and users have been
> preferring the 'host' CPU type for use with KVM for quite some time.
> Additionally, if the KVM-enabled QEMU instance running on a seattle
> host is using the cortex-a57 CPU type, then querying 'cortex-a57' will
> work. Finally, we only implement expansion type 'full', as Arm does not
> yet have a "base" CPU type. Below are some example calls and results
> (to save character clutter they're not in json, but are still json-ish
> to give the idea)
> 
>  # expand the 'max' CPU model
>  query-cpu-model-expansion: type:full, model:{ name:max }
> 
>  return: model:{ name:max, props:{ 'aarch64': true, 'pmu': true }}
> 
>  # attempt to expand the 'max' CPU model with pmu=off
>  query-cpu-model-expansion:
>    type:full, model:{ name:max, props:{ 'pmu': false }}
> 
>  return: model:{ name:max, props:{ 'aarch64': true, 'pmu': false }}
> 
>  # attempt to expand the 'max' CPU model with aarch64=off
>  query-cpu-model-expansion:
>    type:full, model:{ name:max, props:{ 'aarch64': false }}
> 
>  error: "'aarch64' feature cannot be disabled unless KVM is enabled
>          and 32-bit EL1 is supported"

I struggled quite a lot to get the right syntax to test with qmp-shell
(which I think is the most user friendly way to run those commands). In
the commit message, you may add an example such as:

(QEMU) query-cpu-model-expansion type=full
model={"name":"host","props":{"aarch64":true,"pmu":true}}
{"return": {"model": {"name": "host", "props": {"aarch64": true, "pmu":
true}}}}

(QEMU) query-cpu-model-expansion type=full model={"name":"host"}
{"return": {"model": {"name": "host", "props": {"aarch64": true, "pmu":
true}}}}

Thanks

Eric


> 
> In the last example KVM was not in use so an error was returned.
> 
> Note1: It's possible for features to have dependencies on other
> features. I.e. it may be possible to change one feature at a time
> without error, but when attempting to change all features at once
> an error could occur depending on the order they are processed. It's
> also possible changing all at once doesn't generate an error, because
> a feature's dependencies are satisfied with other features, but the
> same feature cannot be changed independently without error. For these
> reasons callers should always attempt to make their desired changes
> all at once in order to ensure the collection is valid.
> 
> Note2: Certainly more features may be added to the list of
> advertised features, e.g. 'vfp' and 'neon'. The only requirement
> is that their property set accessors fail when invalid
> configurations are detected. For vfp we would need something like
> 
>  set_vfp()
>  {
>    if (arm_feature(env, ARM_FEATURE_AARCH64) &&
>        cpu->has_vfp != cpu->has_neon)
>        error("AArch64 CPUs must have both VFP and Neon or neither")
> 
> in its set accessor, and the same for neon, rather than doing that
> check at realize time, which isn't executed at qmp query time.
> 
> Signed-off-by: Andrew Jones <drjones@redhat.com>
> ---
>  qapi/target.json     |   6 +-
>  target/arm/monitor.c | 132 +++++++++++++++++++++++++++++++++++++++++++
>  2 files changed, 135 insertions(+), 3 deletions(-)
> 
> diff --git a/qapi/target.json b/qapi/target.json
> index 1d4d54b6002e..edfa2f82b916 100644
> --- a/qapi/target.json
> +++ b/qapi/target.json
> @@ -408,7 +408,7 @@
>  ##
>  { 'struct': 'CpuModelExpansionInfo',
>    'data': { 'model': 'CpuModelInfo' },
> -  'if': 'defined(TARGET_S390X) || defined(TARGET_I386)' }
> +  'if': 'defined(TARGET_S390X) || defined(TARGET_I386) || defined(TARGET_ARM)' }
>  
>  ##
>  # @query-cpu-model-expansion:
> @@ -433,7 +433,7 @@
>  #   query-cpu-model-expansion while using these is not advised.
>  #
>  # Some architectures may not support all expansion types. s390x supports
> -# "full" and "static".
> +# "full" and "static". Arm only supports "full".
>  #
>  # Returns: a CpuModelExpansionInfo. Returns an error if expanding CPU models is
>  #          not supported, if the model cannot be expanded, if the model contains
> @@ -447,7 +447,7 @@
>    'data': { 'type': 'CpuModelExpansionType',
>              'model': 'CpuModelInfo' },
>    'returns': 'CpuModelExpansionInfo',
> -  'if': 'defined(TARGET_S390X) || defined(TARGET_I386)' }
> +  'if': 'defined(TARGET_S390X) || defined(TARGET_I386) || defined(TARGET_ARM)' }
>  
>  ##
>  # @CpuDefinitionInfo:
> diff --git a/target/arm/monitor.c b/target/arm/monitor.c
> index 41b32b94b258..19e3120eef95 100644
> --- a/target/arm/monitor.c
> +++ b/target/arm/monitor.c
> @@ -23,7 +23,13 @@
>  #include "qemu/osdep.h"
>  #include "hw/boards.h"
>  #include "kvm_arm.h"
> +#include "qapi/error.h"
> +#include "qapi/visitor.h"
> +#include "qapi/qobject-input-visitor.h"
>  #include "qapi/qapi-commands-target.h"
> +#include "qapi/qmp/qerror.h"
> +#include "qapi/qmp/qdict.h"
> +#include "qom/qom-qobject.h"
>  
>  static GICCapability *gic_cap_new(int version)
>  {
> @@ -82,3 +88,129 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)
>  
>      return head;
>  }
> +
> +static const char *cpu_model_advertised_features[] = {
> +    "aarch64", "pmu",
> +    NULL
> +};
> +
> +CpuModelExpansionInfo *qmp_query_cpu_model_expansion(CpuModelExpansionType type,
> +                                                     CpuModelInfo *model,
> +                                                     Error **errp)
> +{
> +    CpuModelExpansionInfo *expansion_info;
> +    const QDict *qdict_in = NULL;
> +    QDict *qdict_out;
> +    ObjectClass *oc;
> +    Object *obj;
> +    const char *name;
> +    int i;
> +
> +    if (type != CPU_MODEL_EXPANSION_TYPE_FULL) {
> +        error_setg(errp, "The requested expansion type is not supported.");
> +        return NULL;
> +    }
> +
> +    if (!kvm_enabled() && !strcmp(model->name, "host")) {
> +        error_setg(errp, "The CPU definition '%s' requires KVM", model->name);
> +        return NULL;
> +    }
> +
> +    oc = cpu_class_by_name(TYPE_ARM_CPU, model->name);
> +    if (!oc) {
> +        error_setg(errp, "The CPU definition '%s' is unknown.", model->name);
> +        return NULL;
> +    }
> +
> +    if (kvm_enabled()) {
> +        const char *cpu_type = current_machine->cpu_type;
> +        int len = strlen(cpu_type) - strlen(ARM_CPU_TYPE_SUFFIX);
> +        bool supported = false;
> +
> +        if (!strcmp(model->name, "host") || !strcmp(model->name, "max")) {
> +            /* These are kvmarm's recommended cpu types */
> +            supported = true;
> +        } else if (strlen(model->name) == len &&
> +                   !strncmp(model->name, cpu_type, len)) {
> +            /* KVM is enabled and we're using this type, so it works. */
> +            supported = true;
> +        }
> +        if (!supported) {
> +            error_setg(errp, "The CPU definition '%s' cannot "
> +                             "be used with KVM on this host", model->name);
> +            return NULL;
> +        }
> +    }
> +
> +    if (model->props) {
> +        qdict_in = qobject_to(QDict, model->props);
> +        if (!qdict_in) {
> +            error_setg(errp, QERR_INVALID_PARAMETER_TYPE, "props", "dict");
> +            return NULL;
> +        }
> +    }
> +
> +    obj = object_new(object_class_get_name(oc));
> +
> +    if (qdict_in) {
> +        Visitor *visitor;
> +
> +        visitor = qobject_input_visitor_new(model->props);
> +        visit_start_struct(visitor, NULL, NULL, 0, errp);
> +        if (*errp) {
> +            object_unref(obj);
> +            return NULL;
> +        }
> +
> +        i = 0;
> +        while ((name = cpu_model_advertised_features[i++]) != NULL) {
> +            if (qdict_get(qdict_in, name)) {
> +                object_property_set(obj, visitor, name, errp);
> +                if (*errp) {
> +                    break;
> +                }
> +            }
> +        }
> +
> +        if (!*errp) {
> +            visit_check_struct(visitor, errp);
> +        }
> +        visit_end_struct(visitor, NULL);
> +        visit_free(visitor);
> +        if (*errp) {
> +            object_unref(obj);
> +            return NULL;
> +        }
> +    }
> +
> +    expansion_info = g_new0(CpuModelExpansionInfo, 1);
> +    expansion_info->model = g_malloc0(sizeof(*expansion_info->model));
> +    expansion_info->model->name = g_strdup(model->name);
> +
> +    qdict_out = qdict_new();
> +
> +    i = 0;
> +    while ((name = cpu_model_advertised_features[i++]) != NULL) {
> +        ObjectProperty *prop = object_property_find(obj, name, NULL);
> +        if (prop) {
> +            QObject *value;
> +
> +            assert(prop->get);
> +            value = object_property_get_qobject(obj, name, errp);
> +            assert(!*errp);
> +
> +            qdict_put_obj(qdict_out, name, value);
> +        }
> +    }
> +
> +    if (!qdict_size(qdict_out)) {
> +        qobject_unref(qdict_out);
> +    } else {
> +        expansion_info->model->props = QOBJECT(qdict_out);
> +        expansion_info->model->has_props = true;
> +    }
> +
> +    object_unref(obj);
> +
> +    return expansion_info;
> +}
> 

[Qemu-devel] [PATCH v2 04/14] tests: arm: Introduce cpu feature tests

[Andrew Jones]

Now that Arm CPUs have advertised features lets add tests to ensure
we maintain their expected availability with and without KVM.

Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 tests/Makefile.include   |   5 +-
 tests/arm-cpu-features.c | 221 +++++++++++++++++++++++++++++++++++++++
 2 files changed, 225 insertions(+), 1 deletion(-)
 create mode 100644 tests/arm-cpu-features.c

diff --git a/tests/Makefile.include b/tests/Makefile.include
index db750dd6d09b..d5f43fe03067 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -255,13 +255,15 @@ check-qtest-sparc64-$(CONFIG_ISA_TESTDEV) = tests/endianness-test$(EXESUF)
 check-qtest-sparc64-y += tests/prom-env-test$(EXESUF)
 check-qtest-sparc64-y += tests/boot-serial-test$(EXESUF)
 
+check-qtest-arm-y += tests/arm-cpu-features$(EXESUF)
 check-qtest-arm-y += tests/microbit-test$(EXESUF)
 check-qtest-arm-y += tests/m25p80-test$(EXESUF)
 check-qtest-arm-y += tests/test-arm-mptimer$(EXESUF)
 check-qtest-arm-y += tests/boot-serial-test$(EXESUF)
 check-qtest-arm-y += tests/hexloader-test$(EXESUF)
 
-check-qtest-aarch64-y = tests/numa-test$(EXESUF)
+check-qtest-aarch64-y += tests/arm-cpu-features$(EXESUF)
+check-qtest-aarch64-y += tests/numa-test$(EXESUF)
 check-qtest-aarch64-y += tests/boot-serial-test$(EXESUF)
 check-qtest-aarch64-y += tests/migration-test$(EXESUF)
 # TODO: once aarch64 TCG is fixed on ARM 32 bit host, make test unconditional
@@ -822,6 +824,7 @@ tests/test-qapi-util$(EXESUF): tests/test-qapi-util.o $(test-util-obj-y)
 tests/numa-test$(EXESUF): tests/numa-test.o
 tests/vmgenid-test$(EXESUF): tests/vmgenid-test.o tests/boot-sector.o tests/acpi-utils.o
 tests/cdrom-test$(EXESUF): tests/cdrom-test.o tests/boot-sector.o $(libqos-obj-y)
+tests/arm-cpu-features$(EXESUF): tests/arm-cpu-features.o
 
 tests/migration/stress$(EXESUF): tests/migration/stress.o
 	$(call quiet-command, $(LINKPROG) -static -O3 $(PTHREAD_LIB) -o $@ $< ,"LINK","$(TARGET_DIR)$@")
diff --git a/tests/arm-cpu-features.c b/tests/arm-cpu-features.c
new file mode 100644
index 000000000000..31b1c15bb979
--- /dev/null
+++ b/tests/arm-cpu-features.c
@@ -0,0 +1,221 @@
+/*
+ * Arm CPU feature test cases
+ *
+ * Copyright (c) 2019 Red Hat Inc.
+ * Authors:
+ *  Andrew Jones <drjones@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+#include "qemu/osdep.h"
+#include "libqtest.h"
+#include "qapi/qmp/qdict.h"
+#include "qapi/qmp/qjson.h"
+
+#define MACHINE    "-machine virt,gic-version=max "
+#define QUERY_HEAD "{ 'execute': 'query-cpu-model-expansion', " \
+                     "'arguments': { 'type': 'full', "
+#define QUERY_TAIL "}}"
+
+static QDict *do_query_no_props(QTestState *qts, const char *cpu_type)
+{
+    return qtest_qmp(qts, QUERY_HEAD "'model': { 'name': %s }"
+                          QUERY_TAIL, cpu_type);
+}
+
+static const char *resp_get_error(QDict *resp)
+{
+    QDict *qdict;
+
+    g_assert(resp);
+    qdict = qdict_get_qdict(resp, "error");
+    if (qdict) {
+        return qdict_get_str(qdict, "desc");
+    }
+    return NULL;
+}
+
+static char *get_error(QTestState *qts, const char *cpu_type,
+                       const char *fmt, ...)
+{
+    QDict *resp;
+    char *error;
+
+    if (fmt) {
+        QDict *args;
+        va_list ap;
+
+        va_start(ap, fmt);
+        args = qdict_from_vjsonf_nofail(fmt, ap);
+        va_end(ap);
+
+        resp = qtest_qmp(qts, QUERY_HEAD "'model': { 'name': %s, "
+                                                    "'props': %p }"
+                              QUERY_TAIL, cpu_type, args);
+    } else {
+        resp = do_query_no_props(qts, cpu_type);
+    }
+
+    g_assert(resp);
+    error = g_strdup(resp_get_error(resp));
+    qobject_unref(resp);
+
+    return error;
+}
+
+#define assert_error(qts, cpu_type, expected_error, fmt, ...)          \
+({                                                                     \
+    char *_error = get_error(qts, cpu_type, fmt, ##__VA_ARGS__);       \
+    g_assert(_error);                                                  \
+    g_assert(g_str_equal(_error, expected_error));                     \
+    g_free(_error);                                                    \
+})
+
+static QDict *resp_get_props(QDict *resp)
+{
+    QDict *qdict;
+
+    g_assert(resp);
+    g_assert(qdict_haskey(resp, "return"));
+    qdict = qdict_get_qdict(resp, "return");
+    g_assert(qdict_haskey(qdict, "model"));
+    qdict = qdict_get_qdict(qdict, "model");
+    g_assert(qdict_haskey(qdict, "props"));
+    qdict = qdict_get_qdict(qdict, "props");
+    return qdict;
+}
+
+#define assert_has_feature(qts, cpu_type, feature)                     \
+({                                                                     \
+    QDict *_resp = do_query_no_props(qts, cpu_type);                   \
+    g_assert(_resp);                                                   \
+    g_assert(qdict_get(resp_get_props(_resp), feature));               \
+    qobject_unref(_resp);                                              \
+})
+
+#define assert_has_not_feature(qts, cpu_type, feature)                 \
+({                                                                     \
+    QDict *_resp = do_query_no_props(qts, cpu_type);                   \
+    g_assert(_resp);                                                   \
+    g_assert(!qdict_get(resp_get_props(_resp), feature));              \
+    qobject_unref(_resp);                                              \
+})
+
+static void assert_type_full(QTestState *qts, const char *cpu_type)
+{
+    const char *error;
+    QDict *resp;
+
+    resp = qtest_qmp(qts, "{ 'execute': 'query-cpu-model-expansion', "
+                            "'arguments': { 'type': 'static', "
+                                           "'model': { 'name': %s }}}",
+                     cpu_type);
+    g_assert(resp);
+    error = resp_get_error(resp);
+    g_assert(error);
+    g_assert(g_str_equal(error,
+                         "The requested expansion type is not supported."));
+    qobject_unref(resp);
+}
+
+static void assert_bad_props(QTestState *qts, const char *cpu_type)
+{
+    const char *error;
+    QDict *resp;
+
+    resp = qtest_qmp(qts, "{ 'execute': 'query-cpu-model-expansion', "
+                            "'arguments': { 'type': 'full', "
+                                           "'model': { 'name': %s, "
+                                                      "'props': false }}}",
+                     cpu_type);
+    g_assert(resp);
+    error = resp_get_error(resp);
+    g_assert(error);
+    g_assert(g_str_equal(error,
+                         "Invalid parameter type for 'props', expected: dict"));
+    qobject_unref(resp);
+}
+
+static void test_query_cpu_model_expansion(const void *data)
+{
+    QTestState *qts;
+
+    qts = qtest_init(MACHINE "-cpu max");
+
+    /* Test common query-cpu-model-expansion input validation */
+    assert_type_full(qts, "foo");
+    assert_bad_props(qts, "max");
+    assert_error(qts, "foo", "The CPU definition 'foo' is unknown.", NULL);
+    assert_error(qts, "max", "Parameter 'not-a-prop' is unexpected",
+                 "{ 'not-a-prop': false }");
+    assert_error(qts, "host", "The CPU definition 'host' requires KVM", NULL);
+
+    /* Test expected feature presence/absence for some cpu types */
+    assert_has_feature(qts, "max", "pmu");
+    assert_has_feature(qts, "cortex-a15", "pmu");
+    assert_has_not_feature(qts, "cortex-a15", "aarch64");
+
+    if (g_str_equal(qtest_get_arch(), "aarch64")) {
+        assert_has_feature(qts, "max", "aarch64");
+        assert_has_feature(qts, "cortex-a57", "pmu");
+        assert_has_feature(qts, "cortex-a57", "aarch64");
+
+        /* Test that features that depend on KVM generate errors without. */
+        assert_error(qts, "max",
+                     "'aarch64' feature cannot be disabled "
+                     "unless KVM is enabled and 32-bit EL1 "
+                     "is supported",
+                     "{ 'aarch64': false }");
+    }
+
+    qtest_quit(qts);
+}
+
+static void test_query_cpu_model_expansion_kvm(const void *data)
+{
+    QTestState *qts;
+
+    qts = qtest_init(MACHINE "-accel kvm -cpu host");
+
+    assert_has_feature(qts, "host", "pmu");
+
+    if (g_str_equal(qtest_get_arch(), "aarch64")) {
+        assert_has_feature(qts, "host", "aarch64");
+
+        assert_error(qts, "cortex-a15",
+            "The CPU definition 'cortex-a15' cannot "
+            "be used with KVM on this host", NULL);
+    } else {
+        assert_error(qts, "host",
+                     "'pmu' feature not supported by KVM on this host",
+                     "{ 'pmu': true }");
+    }
+
+    qtest_quit(qts);
+}
+
+int main(int argc, char **argv)
+{
+    bool kvm_available = false;
+
+    if (!access("/dev/kvm",  R_OK | W_OK)) {
+#if defined(HOST_AARCH64)
+        kvm_available = g_str_equal(qtest_get_arch(), "aarch64");
+#elif defined(HOST_ARM)
+        kvm_available = g_str_equal(qtest_get_arch(), "arm");
+#endif
+    }
+
+    g_test_init(&argc, &argv, NULL);
+
+    qtest_add_data_func("/arm/query-cpu-model-expansion",
+                        NULL, test_query_cpu_model_expansion);
+
+    if (kvm_available) {
+        qtest_add_data_func("/arm/kvm/query-cpu-model-expansion",
+                            NULL, test_query_cpu_model_expansion_kvm);
+    }
+
+    return g_test_run();
+}
-- 
2.20.1

Re: [Qemu-devel] [PATCH v2 04/14] tests: arm: Introduce cpu feature tests

[Auger Eric]

Hi Drew,

On 6/21/19 6:34 PM, Andrew Jones wrote:
> Now that Arm CPUs have advertised features lets add tests to ensure
> we maintain their expected availability with and without KVM.
> 
> Signed-off-by: Andrew Jones <drjones@redhat.com>
> ---
>  tests/Makefile.include   |   5 +-
>  tests/arm-cpu-features.c | 221 +++++++++++++++++++++++++++++++++++++++
>  2 files changed, 225 insertions(+), 1 deletion(-)
>  create mode 100644 tests/arm-cpu-features.c
> 
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index db750dd6d09b..d5f43fe03067 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -255,13 +255,15 @@ check-qtest-sparc64-$(CONFIG_ISA_TESTDEV) = tests/endianness-test$(EXESUF)
>  check-qtest-sparc64-y += tests/prom-env-test$(EXESUF)
>  check-qtest-sparc64-y += tests/boot-serial-test$(EXESUF)
>  
> +check-qtest-arm-y += tests/arm-cpu-features$(EXESUF)
>  check-qtest-arm-y += tests/microbit-test$(EXESUF)
>  check-qtest-arm-y += tests/m25p80-test$(EXESUF)
>  check-qtest-arm-y += tests/test-arm-mptimer$(EXESUF)
>  check-qtest-arm-y += tests/boot-serial-test$(EXESUF)
>  check-qtest-arm-y += tests/hexloader-test$(EXESUF)
>  
> -check-qtest-aarch64-y = tests/numa-test$(EXESUF)
> +check-qtest-aarch64-y += tests/arm-cpu-features$(EXESUF)
> +check-qtest-aarch64-y += tests/numa-test$(EXESUF)
>  check-qtest-aarch64-y += tests/boot-serial-test$(EXESUF)
>  check-qtest-aarch64-y += tests/migration-test$(EXESUF)
>  # TODO: once aarch64 TCG is fixed on ARM 32 bit host, make test unconditional
> @@ -822,6 +824,7 @@ tests/test-qapi-util$(EXESUF): tests/test-qapi-util.o $(test-util-obj-y)
>  tests/numa-test$(EXESUF): tests/numa-test.o
>  tests/vmgenid-test$(EXESUF): tests/vmgenid-test.o tests/boot-sector.o tests/acpi-utils.o
>  tests/cdrom-test$(EXESUF): tests/cdrom-test.o tests/boot-sector.o $(libqos-obj-y)
> +tests/arm-cpu-features$(EXESUF): tests/arm-cpu-features.o
>  
>  tests/migration/stress$(EXESUF): tests/migration/stress.o
>  	$(call quiet-command, $(LINKPROG) -static -O3 $(PTHREAD_LIB) -o $@ $< ,"LINK","$(TARGET_DIR)$@")
> diff --git a/tests/arm-cpu-features.c b/tests/arm-cpu-features.c
> new file mode 100644
> index 000000000000..31b1c15bb979
> --- /dev/null
> +++ b/tests/arm-cpu-features.c
> @@ -0,0 +1,221 @@
> +/*
> + * Arm CPU feature test cases
> + *
> + * Copyright (c) 2019 Red Hat Inc.
> + * Authors:
> + *  Andrew Jones <drjones@redhat.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + */
> +#include "qemu/osdep.h"
> +#include "libqtest.h"
> +#include "qapi/qmp/qdict.h"
> +#include "qapi/qmp/qjson.h"
> +
> +#define MACHINE    "-machine virt,gic-version=max "
> +#define QUERY_HEAD "{ 'execute': 'query-cpu-model-expansion', " \
> +                     "'arguments': { 'type': 'full', "
> +#define QUERY_TAIL "}}"
> +
> +static QDict *do_query_no_props(QTestState *qts, const char *cpu_type)
> +{
> +    return qtest_qmp(qts, QUERY_HEAD "'model': { 'name': %s }"
> +                          QUERY_TAIL, cpu_type);
> +}
> +
> +static const char *resp_get_error(QDict *resp)
> +{
> +    QDict *qdict;
> +
> +    g_assert(resp);
> +    qdict = qdict_get_qdict(resp, "error");
> +    if (qdict) {
> +        return qdict_get_str(qdict, "desc");
> +    }
> +    return NULL;
> +}
> +
> +static char *get_error(QTestState *qts, const char *cpu_type,
> +                       const char *fmt, ...)
> +{
> +    QDict *resp;
> +    char *error;
> +
> +    if (fmt) {
> +        QDict *args;
> +        va_list ap;
> +
> +        va_start(ap, fmt);
> +        args = qdict_from_vjsonf_nofail(fmt, ap);
> +        va_end(ap);
> +
> +        resp = qtest_qmp(qts, QUERY_HEAD "'model': { 'name': %s, "
> +                                                    "'props': %p }"
> +                              QUERY_TAIL, cpu_type, args);
> +    } else {
> +        resp = do_query_no_props(qts, cpu_type);
> +    }
> +
> +    g_assert(resp);
> +    error = g_strdup(resp_get_error(resp));
> +    qobject_unref(resp);
> +
> +    return error;
> +}
> +
> +#define assert_error(qts, cpu_type, expected_error, fmt, ...)          \
> +({                                                                     \
> +    char *_error = get_error(qts, cpu_type, fmt, ##__VA_ARGS__);       \
> +    g_assert(_error);                                                  \
> +    g_assert(g_str_equal(_error, expected_error));                     \
> +    g_free(_error);                                                    \
> +})
> +
> +static QDict *resp_get_props(QDict *resp)
> +{
> +    QDict *qdict;
> +
> +    g_assert(resp);
> +    g_assert(qdict_haskey(resp, "return"));
> +    qdict = qdict_get_qdict(resp, "return");
> +    g_assert(qdict_haskey(qdict, "model"));
> +    qdict = qdict_get_qdict(qdict, "model");
> +    g_assert(qdict_haskey(qdict, "props"));
> +    qdict = qdict_get_qdict(qdict, "props");
> +    return qdict;
> +}
> +
> +#define assert_has_feature(qts, cpu_type, feature)                     \
> +({                                                                     \
> +    QDict *_resp = do_query_no_props(qts, cpu_type);                   \
> +    g_assert(_resp);                                                   \
> +    g_assert(qdict_get(resp_get_props(_resp), feature));               \
> +    qobject_unref(_resp);                                              \
> +})
> +
> +#define assert_has_not_feature(qts, cpu_type, feature)                 \
> +({                                                                     \
> +    QDict *_resp = do_query_no_props(qts, cpu_type);                   \
> +    g_assert(_resp);                                                   \
> +    g_assert(!qdict_get(resp_get_props(_resp), feature));              \
> +    qobject_unref(_resp);                                              \
> +})
> +
> +static void assert_type_full(QTestState *qts, const char *cpu_type)
nit: you don't really care about the cpu_type as static is not supported.
> +{
> +    const char *error;
> +    QDict *resp;
> +
> +    resp = qtest_qmp(qts, "{ 'execute': 'query-cpu-model-expansion', "
> +                            "'arguments': { 'type': 'static', "
> +                                           "'model': { 'name': %s }}}",
> +                     cpu_type);
> +    g_assert(resp);
> +    error = resp_get_error(resp);
> +    g_assert(error);
> +    g_assert(g_str_equal(error,
> +                         "The requested expansion type is not supported."));
> +    qobject_unref(resp);
> +}
> +
> +static void assert_bad_props(QTestState *qts, const char *cpu_type)
> +{
> +    const char *error;
> +    QDict *resp;
> +
> +    resp = qtest_qmp(qts, "{ 'execute': 'query-cpu-model-expansion', "
> +                            "'arguments': { 'type': 'full', "
> +                                           "'model': { 'name': %s, "
> +                                                      "'props': false }}}",
> +                     cpu_type);
> +    g_assert(resp);
> +    error = resp_get_error(resp);
> +    g_assert(error);
> +    g_assert(g_str_equal(error,
> +                         "Invalid parameter type for 'props', expected: dict"));
> +    qobject_unref(resp);
> +}
> +
> +static void test_query_cpu_model_expansion(const void *data)
> +{
> +    QTestState *qts;
> +
> +    qts = qtest_init(MACHINE "-cpu max");
> +
> +    /* Test common query-cpu-model-expansion input validation */
> +    assert_type_full(qts, "foo");
> +    assert_bad_props(qts, "max");
> +    assert_error(qts, "foo", "The CPU definition 'foo' is unknown.", NULL);
> +    assert_error(qts, "max", "Parameter 'not-a-prop' is unexpected",
> +                 "{ 'not-a-prop': false }");
> +    assert_error(qts, "host", "The CPU definition 'host' requires KVM", NULL);
> +
> +    /* Test expected feature presence/absence for some cpu types */
> +    assert_has_feature(qts, "max", "pmu");
> +    assert_has_feature(qts, "cortex-a15", "pmu");
> +    assert_has_not_feature(qts, "cortex-a15", "aarch64");
> +
> +    if (g_str_equal(qtest_get_arch(), "aarch64")) {
> +        assert_has_feature(qts, "max", "aarch64");
> +        assert_has_feature(qts, "cortex-a57", "pmu");
> +        assert_has_feature(qts, "cortex-a57", "aarch64");
> +
> +        /* Test that features that depend on KVM generate errors without. */
> +        assert_error(qts, "max",
> +                     "'aarch64' feature cannot be disabled "
> +                     "unless KVM is enabled and 32-bit EL1 "
> +                     "is supported",
> +                     "{ 'aarch64': false }");
> +    }
> +
> +    qtest_quit(qts);
> +}
> +
> +static void test_query_cpu_model_expansion_kvm(const void *data)
> +{
> +    QTestState *qts;
> +
> +    qts = qtest_init(MACHINE "-accel kvm -cpu host");
> +
> +    assert_has_feature(qts, "host", "pmu");
> +
> +    if (g_str_equal(qtest_get_arch(), "aarch64")) {
> +        assert_has_feature(qts, "host", "aarch64");
> +
> +        assert_error(qts, "cortex-a15",
> +            "The CPU definition 'cortex-a15' cannot "
> +            "be used with KVM on this host", NULL);
> +    } else {
> +        assert_error(qts, "host",
> +                     "'pmu' feature not supported by KVM on this host",
> +                     "{ 'pmu': true }");
> +    }
> +
> +    qtest_quit(qts);
> +}
> +
> +int main(int argc, char **argv)
> +{
> +    bool kvm_available = false;
> +
> +    if (!access("/dev/kvm",  R_OK | W_OK)) {
> +#if defined(HOST_AARCH64)
> +        kvm_available = g_str_equal(qtest_get_arch(), "aarch64");
> +#elif defined(HOST_ARM)
> +        kvm_available = g_str_equal(qtest_get_arch(), "arm");
> +#endif
> +    }
> +
> +    g_test_init(&argc, &argv, NULL);
> +
> +    qtest_add_data_func("/arm/query-cpu-model-expansion",
> +                        NULL, test_query_cpu_model_expansion);
> +
> +    if (kvm_available) {
> +        qtest_add_data_func("/arm/kvm/query-cpu-model-expansion",
> +                            NULL, test_query_cpu_model_expansion_kvm);
> +    }
> +
> +    return g_test_run();
> +}
> 

The code looks good to me. Assuming you updated the error strings
according to the previous discussion and test passes again, feel free to
add:

Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks

Eric

[Qemu-devel] [PATCH v2 05/14] target/arm/helper: zcr: Add build bug next to value range assumption

[Andrew Jones]

Suggested-by: Dave Martin <Dave.Martin@arm.com>
Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 target/arm/helper.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/target/arm/helper.c b/target/arm/helper.c
index df4276f5f6ca..edba94004e0b 100644
--- a/target/arm/helper.c
+++ b/target/arm/helper.c
@@ -5319,6 +5319,7 @@ static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
     int new_len;
 
     /* Bits other than [3:0] are RAZ/WI.  */
+    QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
     raw_write(env, ri, value & 0xf);
 
     /*
-- 
2.20.1

Re: [Qemu-devel] [PATCH v2 05/14] target/arm/helper: zcr: Add build bug next to value range assumption

[Dave Martin]

On Fri, Jun 21, 2019 at 05:34:13PM +0100, Andrew Jones wrote:

The purpose of this check should probably at least be described in a
comment -- i.e., what actually depends on this?

Cheers
---Dave

> Suggested-by: Dave Martin <Dave.Martin@arm.com>
> Signed-off-by: Andrew Jones <drjones@redhat.com>
> ---
>  target/arm/helper.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/target/arm/helper.c b/target/arm/helper.c
> index df4276f5f6ca..edba94004e0b 100644
> --- a/target/arm/helper.c
> +++ b/target/arm/helper.c
> @@ -5319,6 +5319,7 @@ static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
>      int new_len;
>  
>      /* Bits other than [3:0] are RAZ/WI.  */
> +    QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
>      raw_write(env, ri, value & 0xf);
>  
>      /*
> -- 
> 2.20.1
> 

Re: [Qemu-devel] [PATCH v2 05/14] target/arm/helper: zcr: Add build bug next to value range assumption

[Andrew Jones]

On Mon, Jun 24, 2019 at 12:05:07PM +0100, Dave Martin wrote:
> On Fri, Jun 21, 2019 at 05:34:13PM +0100, Andrew Jones wrote:
> 
> The purpose of this check should probably at least be described in a
> comment -- i.e., what actually depends on this?

I was thinking the already present "Bits other than [3:0] are RAZ/WI."
explained that, but how about this for an improvement?

/*
 * Only the lowest 4 bits of ZCR_ELx may be used to constrain the vector
 * length, the rest of the bits are RAZ/WI. Since the vector length of
 * 128-bits (1 in quadwords) is represented as zero in ZCR_ELx, and all
 * vector lengths are represented as their length in quadwords minus 1,
 * then four bits allow up to quadword 16 to be selected.
 */

Thanks,
drew

> 
> Cheers
> ---Dave
> 
> > Suggested-by: Dave Martin <Dave.Martin@arm.com>
> > Signed-off-by: Andrew Jones <drjones@redhat.com>
> > ---
> >  target/arm/helper.c | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/target/arm/helper.c b/target/arm/helper.c
> > index df4276f5f6ca..edba94004e0b 100644
> > --- a/target/arm/helper.c
> > +++ b/target/arm/helper.c
> > @@ -5319,6 +5319,7 @@ static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
> >      int new_len;
> >  
> >      /* Bits other than [3:0] are RAZ/WI.  */
> > +    QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
> >      raw_write(env, ri, value & 0xf);
> >  
> >      /*
> > -- 
> > 2.20.1
> > 
> 

Re: [Qemu-devel] [PATCH v2 05/14] target/arm/helper: zcr: Add build bug next to value range assumption

[Dave Martin]

On Mon, Jun 24, 2019 at 12:30:37PM +0100, Andrew Jones wrote:
> On Mon, Jun 24, 2019 at 12:05:07PM +0100, Dave Martin wrote:
> > On Fri, Jun 21, 2019 at 05:34:13PM +0100, Andrew Jones wrote:
> > 
> > The purpose of this check should probably at least be described in a
> > comment -- i.e., what actually depends on this?
> 
> I was thinking the already present "Bits other than [3:0] are RAZ/WI."
> explained that, but how about this for an improvement?
> 
> /*
>  * Only the lowest 4 bits of ZCR_ELx may be used to constrain the vector
>  * length, the rest of the bits are RAZ/WI. Since the vector length of
>  * 128-bits (1 in quadwords) is represented as zero in ZCR_ELx, and all
>  * vector lengths are represented as their length in quadwords minus 1,
>  * then four bits allow up to quadword 16 to be selected.
>  */

No, maybe the existing comment is enough.

I thought there might be more code elsewhere that assumes that checks
sve_max_vq <= ARM_MAX_VQ then then assumes that sve_max_vq <= 16.  But
if not, we probably don't need an additional comment here.

I haven't tried to understand all the code in the series beyond the
user/kernel interactions, so maybe I was just paranoid.

[...]

Cheers
---Dave

Re: [Qemu-devel] [PATCH v2 05/14] target/arm/helper: zcr: Add build bug next to value range assumption

[Andrew Jones]

On Mon, Jun 24, 2019 at 05:03:08PM +0100, Dave Martin wrote:
> On Mon, Jun 24, 2019 at 12:30:37PM +0100, Andrew Jones wrote:
> > On Mon, Jun 24, 2019 at 12:05:07PM +0100, Dave Martin wrote:
> > > On Fri, Jun 21, 2019 at 05:34:13PM +0100, Andrew Jones wrote:
> > > 
> > > The purpose of this check should probably at least be described in a
> > > comment -- i.e., what actually depends on this?
> > 
> > I was thinking the already present "Bits other than [3:0] are RAZ/WI."
> > explained that, but how about this for an improvement?
> > 
> > /*
> >  * Only the lowest 4 bits of ZCR_ELx may be used to constrain the vector
> >  * length, the rest of the bits are RAZ/WI. Since the vector length of
> >  * 128-bits (1 in quadwords) is represented as zero in ZCR_ELx, and all
> >  * vector lengths are represented as their length in quadwords minus 1,
> >  * then four bits allow up to quadword 16 to be selected.
> >  */
> 
> No, maybe the existing comment is enough.
> 
> I thought there might be more code elsewhere that assumes that checks
> sve_max_vq <= ARM_MAX_VQ then then assumes that sve_max_vq <= 16.  But
> if not, we probably don't need an additional comment here.

I suppose there is some assumption that if sve_max_vq > 0 then it is
also <= ARM_MAX_VQ elsewhere in QEMU. However here in zcr_write I don't
think that assumption is being used. Here we're simply enforcing a limit
of 16 within the emulation, without checking sve_max_vq at all. So I like
the suggestion for a build bug like the one this patch adds, because
otherwise we have 16 in two separate places; the ARM_MAX_VQ definition
and the '& 0xf'.

> 
> I haven't tried to understand all the code in the series beyond the
> user/kernel interactions, so maybe I was just paranoid.

Paranoia is good for the soul. Or something like that...

Thanks,
drew

Re: [Qemu-devel] [PATCH v2 05/14] target/arm/helper: zcr: Add build bug next to value range assumption

[Andrew Jones]

On Tue, Jun 25, 2019 at 08:11:27AM +0200, Andrew Jones wrote:
> On Mon, Jun 24, 2019 at 05:03:08PM +0100, Dave Martin wrote:
> > On Mon, Jun 24, 2019 at 12:30:37PM +0100, Andrew Jones wrote:
> > > On Mon, Jun 24, 2019 at 12:05:07PM +0100, Dave Martin wrote:
> > > > On Fri, Jun 21, 2019 at 05:34:13PM +0100, Andrew Jones wrote:
> > > > 
> > > > The purpose of this check should probably at least be described in a
> > > > comment -- i.e., what actually depends on this?
> > > 
> > > I was thinking the already present "Bits other than [3:0] are RAZ/WI."
> > > explained that, but how about this for an improvement?
> > > 
> > > /*
> > >  * Only the lowest 4 bits of ZCR_ELx may be used to constrain the vector
> > >  * length, the rest of the bits are RAZ/WI. Since the vector length of
> > >  * 128-bits (1 in quadwords) is represented as zero in ZCR_ELx, and all
> > >  * vector lengths are represented as their length in quadwords minus 1,
> > >  * then four bits allow up to quadword 16 to be selected.
> > >  */
> > 
> > No, maybe the existing comment is enough.
> > 
> > I thought there might be more code elsewhere that assumes that checks
> > sve_max_vq <= ARM_MAX_VQ then then assumes that sve_max_vq <= 16.  But
> > if not, we probably don't need an additional comment here.
> 
> I suppose there is some assumption that if sve_max_vq > 0 then it is
> also <= ARM_MAX_VQ elsewhere in QEMU. However here in zcr_write I don't
> think that assumption is being used. Here we're simply enforcing a limit
> of 16 within the emulation, without checking sve_max_vq at all. So I like
> the suggestion for a build bug like the one this patch adds, because
> otherwise we have 16 in two separate places; the ARM_MAX_VQ definition
> and the '& 0xf'.

I suppose we could also write the 0xf in terms of ARM_MAX_VQ, with a
(ARM_MAX_VQ - 1), but that's getting into emulation implementation
preferences, which I don't know anything about. So I'd leave that to
Richard and Peter.

> 
> > 
> > I haven't tried to understand all the code in the series beyond the
> > user/kernel interactions, so maybe I was just paranoid.
> 
> Paranoia is good for the soul. Or something like that...
> 
> Thanks,
> drew

Re: [Qemu-devel] [PATCH v2 05/14] target/arm/helper: zcr: Add build bug next to value range assumption

[Auger Eric]

Hi Drew,

On 6/21/19 6:34 PM, Andrew Jones wrote:
> Suggested-by: Dave Martin <Dave.Martin@arm.com>
> Signed-off-by: Andrew Jones <drjones@redhat.com>
> ---
>  target/arm/helper.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/target/arm/helper.c b/target/arm/helper.c
> index df4276f5f6ca..edba94004e0b 100644
> --- a/target/arm/helper.c
> +++ b/target/arm/helper.c
> @@ -5319,6 +5319,7 @@ static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
>      int new_len;
>  
>      /* Bits other than [3:0] are RAZ/WI.  */
> +    QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
Can you document in the commit message why this check is critical?

Thanks

Eric
>      raw_write(env, ri, value & 0xf);
>  
>      /*
> 

Re: [Qemu-devel] [PATCH v2 05/14] target/arm/helper: zcr: Add build bug next to value range assumption

[Andrew Jones]

On Wed, Jun 26, 2019 at 12:01:10PM +0200, Auger Eric wrote:
> Hi Drew,
> 
> On 6/21/19 6:34 PM, Andrew Jones wrote:
> > Suggested-by: Dave Martin <Dave.Martin@arm.com>
> > Signed-off-by: Andrew Jones <drjones@redhat.com>
> > ---
> >  target/arm/helper.c | 1 +
> >  1 file changed, 1 insertion(+)
> > 
> > diff --git a/target/arm/helper.c b/target/arm/helper.c
> > index df4276f5f6ca..edba94004e0b 100644
> > --- a/target/arm/helper.c
> > +++ b/target/arm/helper.c
> > @@ -5319,6 +5319,7 @@ static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
> >      int new_len;
> >  
> >      /* Bits other than [3:0] are RAZ/WI.  */
> > +    QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
> Can you document in the commit message why this check is critical?

Sure. I can copy+paste the email subject into the commit message :-)

drew

> 
> Thanks
> 
> Eric
> >      raw_write(env, ri, value & 0xf);
> >  
> >      /*
> > 
> 

Re: [Qemu-devel] [PATCH v2 05/14] target/arm/helper: zcr: Add build bug next to value range assumption

[Auger Eric]

Hi,

On 6/26/19 3:28 PM, Andrew Jones wrote:
> On Wed, Jun 26, 2019 at 12:01:10PM +0200, Auger Eric wrote:
>> Hi Drew,
>>
>> On 6/21/19 6:34 PM, Andrew Jones wrote:
>>> Suggested-by: Dave Martin <Dave.Martin@arm.com>
>>> Signed-off-by: Andrew Jones <drjones@redhat.com>
>>> ---
>>>  target/arm/helper.c | 1 +
>>>  1 file changed, 1 insertion(+)
>>>
>>> diff --git a/target/arm/helper.c b/target/arm/helper.c
>>> index df4276f5f6ca..edba94004e0b 100644
>>> --- a/target/arm/helper.c
>>> +++ b/target/arm/helper.c
>>> @@ -5319,6 +5319,7 @@ static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
>>>      int new_len;
>>>  
>>>      /* Bits other than [3:0] are RAZ/WI.  */
>>> +    QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
>> Can you document in the commit message why this check is critical?
> 
> Sure. I can copy+paste the email subject into the commit message :-)
Well that's not what I asked for. Are you enforcing an architectural
maximum of 2048 bits or is the limitation due to some data structs in
the existing code, ... For a non expert reviewer as I am it is not
totally obvious.

Thanks

Eric
> 
> drew
> 
>>
>> Thanks
>>
>> Eric
>>>      raw_write(env, ri, value & 0xf);
>>>  
>>>      /*
>>>
>>

Re: [Qemu-devel] [PATCH v2 05/14] target/arm/helper: zcr: Add build bug next to value range assumption

[Andrew Jones]

On Wed, Jun 26, 2019 at 03:40:11PM +0200, Auger Eric wrote:
> Hi,
> 
> On 6/26/19 3:28 PM, Andrew Jones wrote:
> > On Wed, Jun 26, 2019 at 12:01:10PM +0200, Auger Eric wrote:
> >> Hi Drew,
> >>
> >> On 6/21/19 6:34 PM, Andrew Jones wrote:
> >>> Suggested-by: Dave Martin <Dave.Martin@arm.com>
> >>> Signed-off-by: Andrew Jones <drjones@redhat.com>
> >>> ---
> >>>  target/arm/helper.c | 1 +
> >>>  1 file changed, 1 insertion(+)
> >>>
> >>> diff --git a/target/arm/helper.c b/target/arm/helper.c
> >>> index df4276f5f6ca..edba94004e0b 100644
> >>> --- a/target/arm/helper.c
> >>> +++ b/target/arm/helper.c
> >>> @@ -5319,6 +5319,7 @@ static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
> >>>      int new_len;
> >>>  
> >>>      /* Bits other than [3:0] are RAZ/WI.  */
> >>> +    QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
> >> Can you document in the commit message why this check is critical?
> > 
> > Sure. I can copy+paste the email subject into the commit message :-)
> Well that's not what I asked for. Are you enforcing an architectural
> maximum of 2048 bits or is the limitation due to some data structs in
> the existing code, ... For a non expert reviewer as I am it is not
> totally obvious.

How's this for the commit message

    The current implementation of ZCR_ELx matches the architecture, only
    implementing the lower four bits, with the rest RAZ/WI. This puts
    a strict limit on ARM_MAX_VQ of 16. Make sure we don't let ARM_MAX_VQ
    grow without a corresponding update here.

Thanks,
drew

> 
> Thanks
> 
> Eric
> > 
> > drew
> > 
> >>
> >> Thanks
> >>
> >> Eric
> >>>      raw_write(env, ri, value & 0xf);
> >>>  
> >>>      /*
> >>>
> >>

Re: [Qemu-devel] [PATCH v2 05/14] target/arm/helper: zcr: Add build bug next to value range assumption

[Auger Eric]

Hi,

On 6/26/19 3:58 PM, Andrew Jones wrote:
> On Wed, Jun 26, 2019 at 03:40:11PM +0200, Auger Eric wrote:
>> Hi,
>>
>> On 6/26/19 3:28 PM, Andrew Jones wrote:
>>> On Wed, Jun 26, 2019 at 12:01:10PM +0200, Auger Eric wrote:
>>>> Hi Drew,
>>>>
>>>> On 6/21/19 6:34 PM, Andrew Jones wrote:
>>>>> Suggested-by: Dave Martin <Dave.Martin@arm.com>
>>>>> Signed-off-by: Andrew Jones <drjones@redhat.com>
>>>>> ---
>>>>>  target/arm/helper.c | 1 +
>>>>>  1 file changed, 1 insertion(+)
>>>>>
>>>>> diff --git a/target/arm/helper.c b/target/arm/helper.c
>>>>> index df4276f5f6ca..edba94004e0b 100644
>>>>> --- a/target/arm/helper.c
>>>>> +++ b/target/arm/helper.c
>>>>> @@ -5319,6 +5319,7 @@ static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
>>>>>      int new_len;
>>>>>  
>>>>>      /* Bits other than [3:0] are RAZ/WI.  */
>>>>> +    QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
>>>> Can you document in the commit message why this check is critical?
>>>
>>> Sure. I can copy+paste the email subject into the commit message :-)
>> Well that's not what I asked for. Are you enforcing an architectural
>> maximum of 2048 bits or is the limitation due to some data structs in
>> the existing code, ... For a non expert reviewer as I am it is not
>> totally obvious.
> 
> How's this for the commit message
> 
>     The current implementation of ZCR_ELx matches the architecture, only
>     implementing the lower four bits, with the rest RAZ/WI. This puts
>     a strict limit on ARM_MAX_VQ of 16. Make sure we don't let ARM_MAX_VQ
>     grow without a corresponding update here.
Yep perfect. Thanks

Reviewed-by: Eric Auger <eric.auger@redhat.com>

Eric

> 
> Thanks,
> drew
> 
>>
>> Thanks
>>
>> Eric
>>>
>>> drew
>>>
>>>>
>>>> Thanks
>>>>
>>>> Eric
>>>>>      raw_write(env, ri, value & 0xf);
>>>>>  
>>>>>      /*
>>>>>
>>>>
> 

Re: [Qemu-devel] [PATCH v2 05/14] target/arm/helper: zcr: Add build bug next to value range assumption

[Richard Henderson]

On 6/21/19 6:34 PM, Andrew Jones wrote:
> Suggested-by: Dave Martin <Dave.Martin@arm.com>
> Signed-off-by: Andrew Jones <drjones@redhat.com>
> ---
>  target/arm/helper.c | 1 +
>  1 file changed, 1 insertion(+)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>

> diff --git a/target/arm/helper.c b/target/arm/helper.c
> index df4276f5f6ca..edba94004e0b 100644
> --- a/target/arm/helper.c
> +++ b/target/arm/helper.c
> @@ -5319,6 +5319,7 @@ static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
>      int new_len;
>  
>      /* Bits other than [3:0] are RAZ/WI.  */
> +    QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
>      raw_write(env, ri, value & 0xf);
>  

Re down-thread conversation, I think this is the nice easy way to make sure
that the 0xf is modified if we ever decide to support larger vectors.

I *think* that we could write ARM_MAX_VQ - 1 here, but I'm pretty sure there
are a few other places where we assume that we only need 4 bits to store this
value.  Anyway, we'd definitely need to audit the code to allow ARM_MAX_VQ to
change.


r~

[Qemu-devel] [PATCH v2 06/14] target/arm: Allow SVE to be disabled via a CPU property

[Andrew Jones]

Since 97a28b0eeac14 ("target/arm: Allow VFP and Neon to be disabled via
a CPU property") we can disable the 'max' cpu model's VFP and neon
features, but there's no way to disable SVE. Add the 'sve=on|off'
property to give it that flexibility. We also rename
cpu_max_get/set_sve_vq to cpu_max_get/set_sve_max_vq in order for them
to follow the typical *_get/set_<property-name> pattern.

Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 target/arm/cpu.c         | 10 +++++-
 target/arm/cpu64.c       | 72 ++++++++++++++++++++++++++++++++++------
 target/arm/helper.c      |  8 +++--
 target/arm/monitor.c     |  2 +-
 tests/arm-cpu-features.c |  1 +
 5 files changed, 78 insertions(+), 15 deletions(-)

diff --git a/target/arm/cpu.c b/target/arm/cpu.c
index 858f668d226e..f08e178fc84b 100644
--- a/target/arm/cpu.c
+++ b/target/arm/cpu.c
@@ -198,7 +198,7 @@ static void arm_cpu_reset(CPUState *s)
         env->cp15.cpacr_el1 = deposit64(env->cp15.cpacr_el1, 16, 2, 3);
         env->cp15.cptr_el[3] |= CPTR_EZ;
         /* with maximum vector length */
-        env->vfp.zcr_el[1] = cpu->sve_max_vq - 1;
+        env->vfp.zcr_el[1] = cpu->sve_max_vq ? cpu->sve_max_vq - 1 : 0;
         env->vfp.zcr_el[2] = env->vfp.zcr_el[1];
         env->vfp.zcr_el[3] = env->vfp.zcr_el[1];
         /*
@@ -1129,6 +1129,14 @@ static void arm_cpu_realizefn(DeviceState *dev, Error **errp)
         cpu->isar.mvfr0 = u;
     }
 
+    if (!cpu->sve_max_vq) {
+        uint64_t t;
+
+        t = cpu->isar.id_aa64pfr0;
+        t = FIELD_DP64(t, ID_AA64PFR0, SVE, 0);
+        cpu->isar.id_aa64pfr0 = t;
+    }
+
     if (arm_feature(env, ARM_FEATURE_M) && !cpu->has_dsp) {
         uint32_t u;
 
diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
index 946994838d8a..02ada65f240c 100644
--- a/target/arm/cpu64.c
+++ b/target/arm/cpu64.c
@@ -257,27 +257,75 @@ static void aarch64_a72_initfn(Object *obj)
     define_arm_cp_regs(cpu, cortex_a72_a57_a53_cp_reginfo);
 }
 
-static void cpu_max_get_sve_vq(Object *obj, Visitor *v, const char *name,
-                               void *opaque, Error **errp)
+static void cpu_max_get_sve_max_vq(Object *obj, Visitor *v, const char *name,
+                                   void *opaque, Error **errp)
 {
     ARMCPU *cpu = ARM_CPU(obj);
     visit_type_uint32(v, name, &cpu->sve_max_vq, errp);
 }
 
-static void cpu_max_set_sve_vq(Object *obj, Visitor *v, const char *name,
-                               void *opaque, Error **errp)
+static void cpu_max_set_sve_max_vq(Object *obj, Visitor *v, const char *name,
+                                   void *opaque, Error **errp)
 {
     ARMCPU *cpu = ARM_CPU(obj);
     Error *err = NULL;
+    uint32_t value;
 
-    visit_type_uint32(v, name, &cpu->sve_max_vq, &err);
+    visit_type_uint32(v, name, &value, &err);
+    if (err) {
+        error_propagate(errp, err);
+        return;
+    }
 
-    if (!err && (cpu->sve_max_vq == 0 || cpu->sve_max_vq > ARM_MAX_VQ)) {
-        error_setg(&err, "unsupported SVE vector length");
-        error_append_hint(&err, "Valid sve-max-vq in range [1-%d]\n",
+    if (!cpu->sve_max_vq) {
+        error_setg(errp, "cannot set sve-max-vq");
+        error_append_hint(errp, "SVE has been disabled with sve=off\n");
+        return;
+    }
+
+    cpu->sve_max_vq = value;
+
+    if (cpu->sve_max_vq == 0 || cpu->sve_max_vq > ARM_MAX_VQ) {
+        error_setg(errp, "unsupported SVE vector length");
+        error_append_hint(errp, "Valid sve-max-vq in range [1-%d]\n",
                           ARM_MAX_VQ);
     }
-    error_propagate(errp, err);
+}
+
+static void cpu_arm_get_sve(Object *obj, Visitor *v, const char *name,
+                            void *opaque, Error **errp)
+{
+    ARMCPU *cpu = ARM_CPU(obj);
+    bool value = !!cpu->sve_max_vq;
+
+    visit_type_bool(v, name, &value, errp);
+}
+
+static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
+                            void *opaque, Error **errp)
+{
+    ARMCPU *cpu = ARM_CPU(obj);
+    Error *err = NULL;
+    bool value;
+
+    visit_type_bool(v, name, &value, &err);
+    if (err) {
+        error_propagate(errp, err);
+        return;
+    }
+
+    if (value) {
+        /*
+         * We handle the -cpu <cpu>,sve=off,sve=on case by reinitializing,
+         * but otherwise we don't do anything as an sve=on could come after
+         * a sve-max-vq setting.
+         */
+        if (!cpu->sve_max_vq) {
+            cpu->sve_max_vq = ARM_MAX_VQ;
+        }
+    } else {
+        cpu->sve_max_vq = 0;
+    }
 }
 
 /* -cpu max: if KVM is enabled, like -cpu host (best possible with this host);
@@ -373,8 +421,10 @@ static void aarch64_max_initfn(Object *obj)
 #endif
 
         cpu->sve_max_vq = ARM_MAX_VQ;
-        object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_vq,
-                            cpu_max_set_sve_vq, NULL, NULL, &error_fatal);
+        object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_max_vq,
+                            cpu_max_set_sve_max_vq, NULL, NULL, &error_fatal);
+        object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
+                            cpu_arm_set_sve, NULL, NULL, &error_fatal);
     }
 }
 
diff --git a/target/arm/helper.c b/target/arm/helper.c
index edba94004e0b..f500ccb6d31b 100644
--- a/target/arm/helper.c
+++ b/target/arm/helper.c
@@ -5314,9 +5314,13 @@ uint32_t sve_zcr_len_for_el(CPUARMState *env, int el)
 static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
                       uint64_t value)
 {
+    ARMCPU *cpu = env_archcpu(env);
     int cur_el = arm_current_el(env);
-    int old_len = sve_zcr_len_for_el(env, cur_el);
-    int new_len;
+    int old_len, new_len;
+
+    assert(cpu->sve_max_vq);
+
+    old_len = sve_zcr_len_for_el(env, cur_el);
 
     /* Bits other than [3:0] are RAZ/WI.  */
     QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
diff --git a/target/arm/monitor.c b/target/arm/monitor.c
index 19e3120eef95..157c487a1551 100644
--- a/target/arm/monitor.c
+++ b/target/arm/monitor.c
@@ -90,7 +90,7 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)
 }
 
 static const char *cpu_model_advertised_features[] = {
-    "aarch64", "pmu",
+    "aarch64", "pmu", "sve",
     NULL
 };
 
diff --git a/tests/arm-cpu-features.c b/tests/arm-cpu-features.c
index 31b1c15bb979..509e458e9c2f 100644
--- a/tests/arm-cpu-features.c
+++ b/tests/arm-cpu-features.c
@@ -158,6 +158,7 @@ static void test_query_cpu_model_expansion(const void *data)
 
     if (g_str_equal(qtest_get_arch(), "aarch64")) {
         assert_has_feature(qts, "max", "aarch64");
+        assert_has_feature(qts, "max", "sve");
         assert_has_feature(qts, "cortex-a57", "pmu");
         assert_has_feature(qts, "cortex-a57", "aarch64");
 
-- 
2.20.1

Re: [Qemu-devel] [PATCH v2 06/14] target/arm: Allow SVE to be disabled via a CPU property

[Philippe Mathieu-Daudé]

Hi Drew,

On 6/21/19 6:34 PM, Andrew Jones wrote:
> Since 97a28b0eeac14 ("target/arm: Allow VFP and Neon to be disabled via
> a CPU property") we can disable the 'max' cpu model's VFP and neon
> features, but there's no way to disable SVE. Add the 'sve=on|off'
> property to give it that flexibility. We also rename
> cpu_max_get/set_sve_vq to cpu_max_get/set_sve_max_vq in order for them
> to follow the typical *_get/set_<property-name> pattern.
> 
> Signed-off-by: Andrew Jones <drjones@redhat.com>
> ---
>  target/arm/cpu.c         | 10 +++++-
>  target/arm/cpu64.c       | 72 ++++++++++++++++++++++++++++++++++------
>  target/arm/helper.c      |  8 +++--
>  target/arm/monitor.c     |  2 +-
>  tests/arm-cpu-features.c |  1 +
>  5 files changed, 78 insertions(+), 15 deletions(-)
> 
> diff --git a/target/arm/cpu.c b/target/arm/cpu.c
> index 858f668d226e..f08e178fc84b 100644
> --- a/target/arm/cpu.c
> +++ b/target/arm/cpu.c
> @@ -198,7 +198,7 @@ static void arm_cpu_reset(CPUState *s)
>          env->cp15.cpacr_el1 = deposit64(env->cp15.cpacr_el1, 16, 2, 3);
>          env->cp15.cptr_el[3] |= CPTR_EZ;
>          /* with maximum vector length */
> -        env->vfp.zcr_el[1] = cpu->sve_max_vq - 1;
> +        env->vfp.zcr_el[1] = cpu->sve_max_vq ? cpu->sve_max_vq - 1 : 0;
>          env->vfp.zcr_el[2] = env->vfp.zcr_el[1];
>          env->vfp.zcr_el[3] = env->vfp.zcr_el[1];
>          /*
> @@ -1129,6 +1129,14 @@ static void arm_cpu_realizefn(DeviceState *dev, Error **errp)
>          cpu->isar.mvfr0 = u;
>      }
>  
> +    if (!cpu->sve_max_vq) {
> +        uint64_t t;
> +
> +        t = cpu->isar.id_aa64pfr0;
> +        t = FIELD_DP64(t, ID_AA64PFR0, SVE, 0);
> +        cpu->isar.id_aa64pfr0 = t;
> +    }
> +
>      if (arm_feature(env, ARM_FEATURE_M) && !cpu->has_dsp) {
>          uint32_t u;
>  
> diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
> index 946994838d8a..02ada65f240c 100644
> --- a/target/arm/cpu64.c
> +++ b/target/arm/cpu64.c
> @@ -257,27 +257,75 @@ static void aarch64_a72_initfn(Object *obj)
>      define_arm_cp_regs(cpu, cortex_a72_a57_a53_cp_reginfo);
>  }
>  
> -static void cpu_max_get_sve_vq(Object *obj, Visitor *v, const char *name,
> -                               void *opaque, Error **errp)
> +static void cpu_max_get_sve_max_vq(Object *obj, Visitor *v, const char *name,
> +                                   void *opaque, Error **errp)
>  {
>      ARMCPU *cpu = ARM_CPU(obj);
>      visit_type_uint32(v, name, &cpu->sve_max_vq, errp);
>  }
>  
> -static void cpu_max_set_sve_vq(Object *obj, Visitor *v, const char *name,
> -                               void *opaque, Error **errp)
> +static void cpu_max_set_sve_max_vq(Object *obj, Visitor *v, const char *name,
> +                                   void *opaque, Error **errp)
>  {
>      ARMCPU *cpu = ARM_CPU(obj);
>      Error *err = NULL;
> +    uint32_t value;
>  
> -    visit_type_uint32(v, name, &cpu->sve_max_vq, &err);
> +    visit_type_uint32(v, name, &value, &err);
> +    if (err) {
> +        error_propagate(errp, err);
> +        return;
> +    }
>  
> -    if (!err && (cpu->sve_max_vq == 0 || cpu->sve_max_vq > ARM_MAX_VQ)) {
> -        error_setg(&err, "unsupported SVE vector length");
> -        error_append_hint(&err, "Valid sve-max-vq in range [1-%d]\n",
> +    if (!cpu->sve_max_vq) {
> +        error_setg(errp, "cannot set sve-max-vq");
> +        error_append_hint(errp, "SVE has been disabled with sve=off\n");
> +        return;
> +    }
> +
> +    cpu->sve_max_vq = value;
> +
> +    if (cpu->sve_max_vq == 0 || cpu->sve_max_vq > ARM_MAX_VQ) {
> +        error_setg(errp, "unsupported SVE vector length");
> +        error_append_hint(errp, "Valid sve-max-vq in range [1-%d]\n",
>                            ARM_MAX_VQ);
>      }
> -    error_propagate(errp, err);
> +}
> +
> +static void cpu_arm_get_sve(Object *obj, Visitor *v, const char *name,
> +                            void *opaque, Error **errp)
> +{
> +    ARMCPU *cpu = ARM_CPU(obj);
> +    bool value = !!cpu->sve_max_vq;
> +
> +    visit_type_bool(v, name, &value, errp);
> +}
> +
> +static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
> +                            void *opaque, Error **errp)
> +{
> +    ARMCPU *cpu = ARM_CPU(obj);
> +    Error *err = NULL;
> +    bool value;
> +
> +    visit_type_bool(v, name, &value, &err);
> +    if (err) {
> +        error_propagate(errp, err);
> +        return;
> +    }
> +
> +    if (value) {
> +        /*
> +         * We handle the -cpu <cpu>,sve=off,sve=on case by reinitializing,
> +         * but otherwise we don't do anything as an sve=on could come after
> +         * a sve-max-vq setting.

I don't understand why would someone use that...

For the rest:
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> +         */
> +        if (!cpu->sve_max_vq) {
> +            cpu->sve_max_vq = ARM_MAX_VQ;
> +        }
> +    } else {
> +        cpu->sve_max_vq = 0;
> +    }
>  }
>  
>  /* -cpu max: if KVM is enabled, like -cpu host (best possible with this host);
> @@ -373,8 +421,10 @@ static void aarch64_max_initfn(Object *obj)
>  #endif
>  
>          cpu->sve_max_vq = ARM_MAX_VQ;
> -        object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_vq,
> -                            cpu_max_set_sve_vq, NULL, NULL, &error_fatal);
> +        object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_max_vq,
> +                            cpu_max_set_sve_max_vq, NULL, NULL, &error_fatal);
> +        object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
> +                            cpu_arm_set_sve, NULL, NULL, &error_fatal);
>      }
>  }
>  
> diff --git a/target/arm/helper.c b/target/arm/helper.c
> index edba94004e0b..f500ccb6d31b 100644
> --- a/target/arm/helper.c
> +++ b/target/arm/helper.c
> @@ -5314,9 +5314,13 @@ uint32_t sve_zcr_len_for_el(CPUARMState *env, int el)
>  static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
>                        uint64_t value)
>  {
> +    ARMCPU *cpu = env_archcpu(env);
>      int cur_el = arm_current_el(env);
> -    int old_len = sve_zcr_len_for_el(env, cur_el);
> -    int new_len;
> +    int old_len, new_len;
> +
> +    assert(cpu->sve_max_vq);
> +
> +    old_len = sve_zcr_len_for_el(env, cur_el);
>  
>      /* Bits other than [3:0] are RAZ/WI.  */
>      QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
> diff --git a/target/arm/monitor.c b/target/arm/monitor.c
> index 19e3120eef95..157c487a1551 100644
> --- a/target/arm/monitor.c
> +++ b/target/arm/monitor.c
> @@ -90,7 +90,7 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)
>  }
>  
>  static const char *cpu_model_advertised_features[] = {
> -    "aarch64", "pmu",
> +    "aarch64", "pmu", "sve",
>      NULL
>  };
>  
> diff --git a/tests/arm-cpu-features.c b/tests/arm-cpu-features.c
> index 31b1c15bb979..509e458e9c2f 100644
> --- a/tests/arm-cpu-features.c
> +++ b/tests/arm-cpu-features.c
> @@ -158,6 +158,7 @@ static void test_query_cpu_model_expansion(const void *data)
>  
>      if (g_str_equal(qtest_get_arch(), "aarch64")) {
>          assert_has_feature(qts, "max", "aarch64");
> +        assert_has_feature(qts, "max", "sve");
>          assert_has_feature(qts, "cortex-a57", "pmu");
>          assert_has_feature(qts, "cortex-a57", "aarch64");
>  
> 

Re: [Qemu-devel] [PATCH v2 06/14] target/arm: Allow SVE to be disabled via a CPU property

[Andrew Jones]

On Fri, Jun 21, 2019 at 06:55:02PM +0200, Philippe Mathieu-Daudé wrote:
> Hi Drew,
> 
> On 6/21/19 6:34 PM, Andrew Jones wrote:
> > Since 97a28b0eeac14 ("target/arm: Allow VFP and Neon to be disabled via
> > a CPU property") we can disable the 'max' cpu model's VFP and neon
> > features, but there's no way to disable SVE. Add the 'sve=on|off'
> > property to give it that flexibility. We also rename
> > cpu_max_get/set_sve_vq to cpu_max_get/set_sve_max_vq in order for them
> > to follow the typical *_get/set_<property-name> pattern.
> > 
> > Signed-off-by: Andrew Jones <drjones@redhat.com>
> > ---
> >  target/arm/cpu.c         | 10 +++++-
> >  target/arm/cpu64.c       | 72 ++++++++++++++++++++++++++++++++++------
> >  target/arm/helper.c      |  8 +++--
> >  target/arm/monitor.c     |  2 +-
> >  tests/arm-cpu-features.c |  1 +
> >  5 files changed, 78 insertions(+), 15 deletions(-)
> > 
> > diff --git a/target/arm/cpu.c b/target/arm/cpu.c
> > index 858f668d226e..f08e178fc84b 100644
> > --- a/target/arm/cpu.c
> > +++ b/target/arm/cpu.c
> > @@ -198,7 +198,7 @@ static void arm_cpu_reset(CPUState *s)
> >          env->cp15.cpacr_el1 = deposit64(env->cp15.cpacr_el1, 16, 2, 3);
> >          env->cp15.cptr_el[3] |= CPTR_EZ;
> >          /* with maximum vector length */
> > -        env->vfp.zcr_el[1] = cpu->sve_max_vq - 1;
> > +        env->vfp.zcr_el[1] = cpu->sve_max_vq ? cpu->sve_max_vq - 1 : 0;
> >          env->vfp.zcr_el[2] = env->vfp.zcr_el[1];
> >          env->vfp.zcr_el[3] = env->vfp.zcr_el[1];
> >          /*
> > @@ -1129,6 +1129,14 @@ static void arm_cpu_realizefn(DeviceState *dev, Error **errp)
> >          cpu->isar.mvfr0 = u;
> >      }
> >  
> > +    if (!cpu->sve_max_vq) {
> > +        uint64_t t;
> > +
> > +        t = cpu->isar.id_aa64pfr0;
> > +        t = FIELD_DP64(t, ID_AA64PFR0, SVE, 0);
> > +        cpu->isar.id_aa64pfr0 = t;
> > +    }
> > +
> >      if (arm_feature(env, ARM_FEATURE_M) && !cpu->has_dsp) {
> >          uint32_t u;
> >  
> > diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
> > index 946994838d8a..02ada65f240c 100644
> > --- a/target/arm/cpu64.c
> > +++ b/target/arm/cpu64.c
> > @@ -257,27 +257,75 @@ static void aarch64_a72_initfn(Object *obj)
> >      define_arm_cp_regs(cpu, cortex_a72_a57_a53_cp_reginfo);
> >  }
> >  
> > -static void cpu_max_get_sve_vq(Object *obj, Visitor *v, const char *name,
> > -                               void *opaque, Error **errp)
> > +static void cpu_max_get_sve_max_vq(Object *obj, Visitor *v, const char *name,
> > +                                   void *opaque, Error **errp)
> >  {
> >      ARMCPU *cpu = ARM_CPU(obj);
> >      visit_type_uint32(v, name, &cpu->sve_max_vq, errp);
> >  }
> >  
> > -static void cpu_max_set_sve_vq(Object *obj, Visitor *v, const char *name,
> > -                               void *opaque, Error **errp)
> > +static void cpu_max_set_sve_max_vq(Object *obj, Visitor *v, const char *name,
> > +                                   void *opaque, Error **errp)
> >  {
> >      ARMCPU *cpu = ARM_CPU(obj);
> >      Error *err = NULL;
> > +    uint32_t value;
> >  
> > -    visit_type_uint32(v, name, &cpu->sve_max_vq, &err);
> > +    visit_type_uint32(v, name, &value, &err);
> > +    if (err) {
> > +        error_propagate(errp, err);
> > +        return;
> > +    }
> >  
> > -    if (!err && (cpu->sve_max_vq == 0 || cpu->sve_max_vq > ARM_MAX_VQ)) {
> > -        error_setg(&err, "unsupported SVE vector length");
> > -        error_append_hint(&err, "Valid sve-max-vq in range [1-%d]\n",
> > +    if (!cpu->sve_max_vq) {
> > +        error_setg(errp, "cannot set sve-max-vq");
> > +        error_append_hint(errp, "SVE has been disabled with sve=off\n");
> > +        return;
> > +    }
> > +
> > +    cpu->sve_max_vq = value;
> > +
> > +    if (cpu->sve_max_vq == 0 || cpu->sve_max_vq > ARM_MAX_VQ) {
> > +        error_setg(errp, "unsupported SVE vector length");
> > +        error_append_hint(errp, "Valid sve-max-vq in range [1-%d]\n",
> >                            ARM_MAX_VQ);
> >      }
> > -    error_propagate(errp, err);
> > +}
> > +
> > +static void cpu_arm_get_sve(Object *obj, Visitor *v, const char *name,
> > +                            void *opaque, Error **errp)
> > +{
> > +    ARMCPU *cpu = ARM_CPU(obj);
> > +    bool value = !!cpu->sve_max_vq;
> > +
> > +    visit_type_bool(v, name, &value, errp);
> > +}
> > +
> > +static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
> > +                            void *opaque, Error **errp)
> > +{
> > +    ARMCPU *cpu = ARM_CPU(obj);
> > +    Error *err = NULL;
> > +    bool value;
> > +
> > +    visit_type_bool(v, name, &value, &err);
> > +    if (err) {
> > +        error_propagate(errp, err);
> > +        return;
> > +    }
> > +
> > +    if (value) {
> > +        /*
> > +         * We handle the -cpu <cpu>,sve=off,sve=on case by reinitializing,
> > +         * but otherwise we don't do anything as an sve=on could come after
> > +         * a sve-max-vq setting.
> 
> I don't understand why would someone use that...

Command line generators can append something like 'feat1=off,feat2=off,...'
to the cpu feature property list by default, and then, based on user input,
rather than parsing and modifying the default command line they may just
append the new value, like 'feat1=off,feat2=off,...,feat1=on' to change it.

> 
> For the rest:
> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

Thanks!
drew

> 
> > +         */
> > +        if (!cpu->sve_max_vq) {
> > +            cpu->sve_max_vq = ARM_MAX_VQ;
> > +        }
> > +    } else {
> > +        cpu->sve_max_vq = 0;
> > +    }
> >  }
> >  
> >  /* -cpu max: if KVM is enabled, like -cpu host (best possible with this host);
> > @@ -373,8 +421,10 @@ static void aarch64_max_initfn(Object *obj)
> >  #endif
> >  
> >          cpu->sve_max_vq = ARM_MAX_VQ;
> > -        object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_vq,
> > -                            cpu_max_set_sve_vq, NULL, NULL, &error_fatal);
> > +        object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_max_vq,
> > +                            cpu_max_set_sve_max_vq, NULL, NULL, &error_fatal);
> > +        object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
> > +                            cpu_arm_set_sve, NULL, NULL, &error_fatal);
> >      }
> >  }
> >  
> > diff --git a/target/arm/helper.c b/target/arm/helper.c
> > index edba94004e0b..f500ccb6d31b 100644
> > --- a/target/arm/helper.c
> > +++ b/target/arm/helper.c
> > @@ -5314,9 +5314,13 @@ uint32_t sve_zcr_len_for_el(CPUARMState *env, int el)
> >  static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
> >                        uint64_t value)
> >  {
> > +    ARMCPU *cpu = env_archcpu(env);
> >      int cur_el = arm_current_el(env);
> > -    int old_len = sve_zcr_len_for_el(env, cur_el);
> > -    int new_len;
> > +    int old_len, new_len;
> > +
> > +    assert(cpu->sve_max_vq);
> > +
> > +    old_len = sve_zcr_len_for_el(env, cur_el);
> >  
> >      /* Bits other than [3:0] are RAZ/WI.  */
> >      QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
> > diff --git a/target/arm/monitor.c b/target/arm/monitor.c
> > index 19e3120eef95..157c487a1551 100644
> > --- a/target/arm/monitor.c
> > +++ b/target/arm/monitor.c
> > @@ -90,7 +90,7 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)
> >  }
> >  
> >  static const char *cpu_model_advertised_features[] = {
> > -    "aarch64", "pmu",
> > +    "aarch64", "pmu", "sve",
> >      NULL
> >  };
> >  
> > diff --git a/tests/arm-cpu-features.c b/tests/arm-cpu-features.c
> > index 31b1c15bb979..509e458e9c2f 100644
> > --- a/tests/arm-cpu-features.c
> > +++ b/tests/arm-cpu-features.c
> > @@ -158,6 +158,7 @@ static void test_query_cpu_model_expansion(const void *data)
> >  
> >      if (g_str_equal(qtest_get_arch(), "aarch64")) {
> >          assert_has_feature(qts, "max", "aarch64");
> > +        assert_has_feature(qts, "max", "sve");
> >          assert_has_feature(qts, "cortex-a57", "pmu");
> >          assert_has_feature(qts, "cortex-a57", "aarch64");
> >  
> > 
> 

Re: [Qemu-devel] [PATCH v2 06/14] target/arm: Allow SVE to be disabled via a CPU property

[Auger Eric]

Hi Drew,

On 6/21/19 6:34 PM, Andrew Jones wrote:
> Since 97a28b0eeac14 ("target/arm: Allow VFP and Neon to be disabled via
> a CPU property") we can disable the 'max' cpu model's VFP and neon
> features, but there's no way to disable SVE. Add the 'sve=on|off'
> property to give it that flexibility. We also rename
> cpu_max_get/set_sve_vq to cpu_max_get/set_sve_max_vq in order for them
> to follow the typical *_get/set_<property-name> pattern.
> 
> Signed-off-by: Andrew Jones <drjones@redhat.com>
> ---
>  target/arm/cpu.c         | 10 +++++-
>  target/arm/cpu64.c       | 72 ++++++++++++++++++++++++++++++++++------
>  target/arm/helper.c      |  8 +++--
>  target/arm/monitor.c     |  2 +-
>  tests/arm-cpu-features.c |  1 +
>  5 files changed, 78 insertions(+), 15 deletions(-)
> 
> diff --git a/target/arm/cpu.c b/target/arm/cpu.c
> index 858f668d226e..f08e178fc84b 100644
> --- a/target/arm/cpu.c
> +++ b/target/arm/cpu.c
> @@ -198,7 +198,7 @@ static void arm_cpu_reset(CPUState *s)
>          env->cp15.cpacr_el1 = deposit64(env->cp15.cpacr_el1, 16, 2, 3);
>          env->cp15.cptr_el[3] |= CPTR_EZ;
>          /* with maximum vector length */
> -        env->vfp.zcr_el[1] = cpu->sve_max_vq - 1;
> +        env->vfp.zcr_el[1] = cpu->sve_max_vq ? cpu->sve_max_vq - 1 : 0;
>          env->vfp.zcr_el[2] = env->vfp.zcr_el[1];
>          env->vfp.zcr_el[3] = env->vfp.zcr_el[1];
>          /*
> @@ -1129,6 +1129,14 @@ static void arm_cpu_realizefn(DeviceState *dev, Error **errp)
>          cpu->isar.mvfr0 = u;
>      }
>  
> +    if (!cpu->sve_max_vq) {
> +        uint64_t t;
> +
> +        t = cpu->isar.id_aa64pfr0;
> +        t = FIELD_DP64(t, ID_AA64PFR0, SVE, 0);
> +        cpu->isar.id_aa64pfr0 = t;
> +    }
> +
>      if (arm_feature(env, ARM_FEATURE_M) && !cpu->has_dsp) {
>          uint32_t u;
>  
> diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
> index 946994838d8a..02ada65f240c 100644
> --- a/target/arm/cpu64.c
> +++ b/target/arm/cpu64.c
> @@ -257,27 +257,75 @@ static void aarch64_a72_initfn(Object *obj)
>      define_arm_cp_regs(cpu, cortex_a72_a57_a53_cp_reginfo);
>  }
>  
> -static void cpu_max_get_sve_vq(Object *obj, Visitor *v, const char *name,
> -                               void *opaque, Error **errp)
> +static void cpu_max_get_sve_max_vq(Object *obj, Visitor *v, const char *name,
> +                                   void *opaque, Error **errp)
>  {
>      ARMCPU *cpu = ARM_CPU(obj);
>      visit_type_uint32(v, name, &cpu->sve_max_vq, errp);
>  }
>  
> -static void cpu_max_set_sve_vq(Object *obj, Visitor *v, const char *name,
> -                               void *opaque, Error **errp)
> +static void cpu_max_set_sve_max_vq(Object *obj, Visitor *v, const char *name,
> +                                   void *opaque, Error **errp)
>  {
>      ARMCPU *cpu = ARM_CPU(obj);
>      Error *err = NULL;
> +    uint32_t value;
>  
> -    visit_type_uint32(v, name, &cpu->sve_max_vq, &err);
> +    visit_type_uint32(v, name, &value, &err);
> +    if (err) {
> +        error_propagate(errp, err);
> +        return;
> +    }
>  
> -    if (!err && (cpu->sve_max_vq == 0 || cpu->sve_max_vq > ARM_MAX_VQ)) {
> -        error_setg(&err, "unsupported SVE vector length");
> -        error_append_hint(&err, "Valid sve-max-vq in range [1-%d]\n",
> +    if (!cpu->sve_max_vq) {
> +        error_setg(errp, "cannot set sve-max-vq");
> +        error_append_hint(errp, "SVE has been disabled with sve=off\n");
> +        return;
> +    }
> +
> +    cpu->sve_max_vq = value;
> +
> +    if (cpu->sve_max_vq == 0 || cpu->sve_max_vq > ARM_MAX_VQ) {
> +        error_setg(errp, "unsupported SVE vector length");
> +        error_append_hint(errp, "Valid sve-max-vq in range [1-%d]\n",
>                            ARM_MAX_VQ);
>      }
> -    error_propagate(errp, err);
> +}
> +
> +static void cpu_arm_get_sve(Object *obj, Visitor *v, const char *name,
> +                            void *opaque, Error **errp)
> +{
> +    ARMCPU *cpu = ARM_CPU(obj);
> +    bool value = !!cpu->sve_max_vq;
> +
> +    visit_type_bool(v, name, &value, errp);
> +}
> +
> +static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
> +                            void *opaque, Error **errp)
> +{
> +    ARMCPU *cpu = ARM_CPU(obj);
> +    Error *err = NULL;
> +    bool value;
> +
> +    visit_type_bool(v, name, &value, &err);
> +    if (err) {
> +        error_propagate(errp, err);
> +        return;
> +    }
> +
> +    if (value) {
> +        /*
> +         * We handle the -cpu <cpu>,sve=off,sve=on case by reinitializing,
> +         * but otherwise we don't do anything as an sve=on could come after
> +         * a sve-max-vq setting.
> +         */
> +        if (!cpu->sve_max_vq) {
> +            cpu->sve_max_vq = ARM_MAX_VQ;> +        }
> +    } else {
> +        cpu->sve_max_vq = 0;
> +    }
>  }
>  
>  /* -cpu max: if KVM is enabled, like -cpu host (best possible with this host);
> @@ -373,8 +421,10 @@ static void aarch64_max_initfn(Object *obj)
>  #endif
>  
>          cpu->sve_max_vq = ARM_MAX_VQ;
> -        object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_vq,
> -                            cpu_max_set_sve_vq, NULL, NULL, &error_fatal);
> +        object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_max_vq,
> +                            cpu_max_set_sve_max_vq, NULL, NULL, &error_fatal);
> +        object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
> +                            cpu_arm_set_sve, NULL, NULL, &error_fatal);
Wouldn't it be possible to allow 0 as a valid value for sve-max-vq and
interpret this as sve=off?
>      }
>  }
>  
> diff --git a/target/arm/helper.c b/target/arm/helper.c
> index edba94004e0b..f500ccb6d31b 100644
> --- a/target/arm/helper.c
> +++ b/target/arm/helper.c
> @@ -5314,9 +5314,13 @@ uint32_t sve_zcr_len_for_el(CPUARMState *env, int el)
>  static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
>                        uint64_t value)
>  {
> +    ARMCPU *cpu = env_archcpu(env);
>      int cur_el = arm_current_el(env);
> -    int old_len = sve_zcr_len_for_el(env, cur_el);
> -    int new_len;
> +    int old_len, new_len;
> +
> +    assert(cpu->sve_max_vq);
> +
> +    old_len = sve_zcr_len_for_el(env, cur_el);
The comment
    /*
     * Because we arrived here, we know both FP and SVE are enabled;
     * otherwise we would have trapped access to the ZCR_ELn register.
     */
gives the impression we are sure SVE is enabled. So is it mandated to
test sve_max_vq? Otherwise adapt the comment?

Thanks

Eric
>  
>      /* Bits other than [3:0] are RAZ/WI.  */
>      QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
> diff --git a/target/arm/monitor.c b/target/arm/monitor.c
> index 19e3120eef95..157c487a1551 100644
> --- a/target/arm/monitor.c
> +++ b/target/arm/monitor.c
> @@ -90,7 +90,7 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)
>  }
>  
>  static const char *cpu_model_advertised_features[] = {
> -    "aarch64", "pmu",
> +    "aarch64", "pmu", "sve",
>      NULL
>  };
>  
> diff --git a/tests/arm-cpu-features.c b/tests/arm-cpu-features.c
> index 31b1c15bb979..509e458e9c2f 100644
> --- a/tests/arm-cpu-features.c
> +++ b/tests/arm-cpu-features.c
> @@ -158,6 +158,7 @@ static void test_query_cpu_model_expansion(const void *data)
>  
>      if (g_str_equal(qtest_get_arch(), "aarch64")) {
>          assert_has_feature(qts, "max", "aarch64");
> +        assert_has_feature(qts, "max", "sve");
>          assert_has_feature(qts, "cortex-a57", "pmu");
>          assert_has_feature(qts, "cortex-a57", "aarch64");
>  
> 

Re: [Qemu-devel] [PATCH v2 06/14] target/arm: Allow SVE to be disabled via a CPU property

[Andrew Jones]

On Wed, Jun 26, 2019 at 12:00:54PM +0200, Auger Eric wrote:
> Hi Drew,
> 
> On 6/21/19 6:34 PM, Andrew Jones wrote:
> > Since 97a28b0eeac14 ("target/arm: Allow VFP and Neon to be disabled via
> > a CPU property") we can disable the 'max' cpu model's VFP and neon
> > features, but there's no way to disable SVE. Add the 'sve=on|off'
> > property to give it that flexibility. We also rename
> > cpu_max_get/set_sve_vq to cpu_max_get/set_sve_max_vq in order for them
> > to follow the typical *_get/set_<property-name> pattern.
> > 
> > Signed-off-by: Andrew Jones <drjones@redhat.com>
> > ---
> >  target/arm/cpu.c         | 10 +++++-
> >  target/arm/cpu64.c       | 72 ++++++++++++++++++++++++++++++++++------
> >  target/arm/helper.c      |  8 +++--
> >  target/arm/monitor.c     |  2 +-
> >  tests/arm-cpu-features.c |  1 +
> >  5 files changed, 78 insertions(+), 15 deletions(-)
> > 
> > diff --git a/target/arm/cpu.c b/target/arm/cpu.c
> > index 858f668d226e..f08e178fc84b 100644
> > --- a/target/arm/cpu.c
> > +++ b/target/arm/cpu.c
> > @@ -198,7 +198,7 @@ static void arm_cpu_reset(CPUState *s)
> >          env->cp15.cpacr_el1 = deposit64(env->cp15.cpacr_el1, 16, 2, 3);
> >          env->cp15.cptr_el[3] |= CPTR_EZ;
> >          /* with maximum vector length */
> > -        env->vfp.zcr_el[1] = cpu->sve_max_vq - 1;
> > +        env->vfp.zcr_el[1] = cpu->sve_max_vq ? cpu->sve_max_vq - 1 : 0;
> >          env->vfp.zcr_el[2] = env->vfp.zcr_el[1];
> >          env->vfp.zcr_el[3] = env->vfp.zcr_el[1];
> >          /*
> > @@ -1129,6 +1129,14 @@ static void arm_cpu_realizefn(DeviceState *dev, Error **errp)
> >          cpu->isar.mvfr0 = u;
> >      }
> >  
> > +    if (!cpu->sve_max_vq) {
> > +        uint64_t t;
> > +
> > +        t = cpu->isar.id_aa64pfr0;
> > +        t = FIELD_DP64(t, ID_AA64PFR0, SVE, 0);
> > +        cpu->isar.id_aa64pfr0 = t;
> > +    }
> > +
> >      if (arm_feature(env, ARM_FEATURE_M) && !cpu->has_dsp) {
> >          uint32_t u;
> >  
> > diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
> > index 946994838d8a..02ada65f240c 100644
> > --- a/target/arm/cpu64.c
> > +++ b/target/arm/cpu64.c
> > @@ -257,27 +257,75 @@ static void aarch64_a72_initfn(Object *obj)
> >      define_arm_cp_regs(cpu, cortex_a72_a57_a53_cp_reginfo);
> >  }
> >  
> > -static void cpu_max_get_sve_vq(Object *obj, Visitor *v, const char *name,
> > -                               void *opaque, Error **errp)
> > +static void cpu_max_get_sve_max_vq(Object *obj, Visitor *v, const char *name,
> > +                                   void *opaque, Error **errp)
> >  {
> >      ARMCPU *cpu = ARM_CPU(obj);
> >      visit_type_uint32(v, name, &cpu->sve_max_vq, errp);
> >  }
> >  
> > -static void cpu_max_set_sve_vq(Object *obj, Visitor *v, const char *name,
> > -                               void *opaque, Error **errp)
> > +static void cpu_max_set_sve_max_vq(Object *obj, Visitor *v, const char *name,
> > +                                   void *opaque, Error **errp)
> >  {
> >      ARMCPU *cpu = ARM_CPU(obj);
> >      Error *err = NULL;
> > +    uint32_t value;
> >  
> > -    visit_type_uint32(v, name, &cpu->sve_max_vq, &err);
> > +    visit_type_uint32(v, name, &value, &err);
> > +    if (err) {
> > +        error_propagate(errp, err);
> > +        return;
> > +    }
> >  
> > -    if (!err && (cpu->sve_max_vq == 0 || cpu->sve_max_vq > ARM_MAX_VQ)) {
> > -        error_setg(&err, "unsupported SVE vector length");
> > -        error_append_hint(&err, "Valid sve-max-vq in range [1-%d]\n",
> > +    if (!cpu->sve_max_vq) {
> > +        error_setg(errp, "cannot set sve-max-vq");
> > +        error_append_hint(errp, "SVE has been disabled with sve=off\n");
> > +        return;
> > +    }
> > +
> > +    cpu->sve_max_vq = value;
> > +
> > +    if (cpu->sve_max_vq == 0 || cpu->sve_max_vq > ARM_MAX_VQ) {
> > +        error_setg(errp, "unsupported SVE vector length");
> > +        error_append_hint(errp, "Valid sve-max-vq in range [1-%d]\n",
> >                            ARM_MAX_VQ);
> >      }
> > -    error_propagate(errp, err);
> > +}
> > +
> > +static void cpu_arm_get_sve(Object *obj, Visitor *v, const char *name,
> > +                            void *opaque, Error **errp)
> > +{
> > +    ARMCPU *cpu = ARM_CPU(obj);
> > +    bool value = !!cpu->sve_max_vq;
> > +
> > +    visit_type_bool(v, name, &value, errp);
> > +}
> > +
> > +static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
> > +                            void *opaque, Error **errp)
> > +{
> > +    ARMCPU *cpu = ARM_CPU(obj);
> > +    Error *err = NULL;
> > +    bool value;
> > +
> > +    visit_type_bool(v, name, &value, &err);
> > +    if (err) {
> > +        error_propagate(errp, err);
> > +        return;
> > +    }
> > +
> > +    if (value) {
> > +        /*
> > +         * We handle the -cpu <cpu>,sve=off,sve=on case by reinitializing,
> > +         * but otherwise we don't do anything as an sve=on could come after
> > +         * a sve-max-vq setting.
> > +         */
> > +        if (!cpu->sve_max_vq) {
> > +            cpu->sve_max_vq = ARM_MAX_VQ;> +        }
> > +    } else {
> > +        cpu->sve_max_vq = 0;
> > +    }
> >  }
> >  
> >  /* -cpu max: if KVM is enabled, like -cpu host (best possible with this host);
> > @@ -373,8 +421,10 @@ static void aarch64_max_initfn(Object *obj)
> >  #endif
> >  
> >          cpu->sve_max_vq = ARM_MAX_VQ;
> > -        object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_vq,
> > -                            cpu_max_set_sve_vq, NULL, NULL, &error_fatal);
> > +        object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_max_vq,
> > +                            cpu_max_set_sve_max_vq, NULL, NULL, &error_fatal);
> > +        object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
> > +                            cpu_arm_set_sve, NULL, NULL, &error_fatal);
> Wouldn't it be possible to allow 0 as a valid value for sve-max-vq and
> interpret this as sve=off?

No, because it wouldn't be a very nice interface considering all other cpu
features are enabled/disabled with booleans, all the sve<vl-bits>
properties are in vl-bits rather than vq's - so it'd be nicer to just
forget about sve-max-vq altogether, and indeed it wouldn't work for cpu
'host', because that cpu type doesn't have the sve-max-vq property at all.

> >      }
> >  }
> >  
> > diff --git a/target/arm/helper.c b/target/arm/helper.c
> > index edba94004e0b..f500ccb6d31b 100644
> > --- a/target/arm/helper.c
> > +++ b/target/arm/helper.c
> > @@ -5314,9 +5314,13 @@ uint32_t sve_zcr_len_for_el(CPUARMState *env, int el)
> >  static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
> >                        uint64_t value)
> >  {
> > +    ARMCPU *cpu = env_archcpu(env);
> >      int cur_el = arm_current_el(env);
> > -    int old_len = sve_zcr_len_for_el(env, cur_el);
> > -    int new_len;
> > +    int old_len, new_len;
> > +
> > +    assert(cpu->sve_max_vq);
> > +
> > +    old_len = sve_zcr_len_for_el(env, cur_el);
> The comment
>     /*
>      * Because we arrived here, we know both FP and SVE are enabled;
>      * otherwise we would have trapped access to the ZCR_ELn register.
>      */
> gives the impression we are sure SVE is enabled. So is it mandated to
> test sve_max_vq? Otherwise adapt the comment?

Nope, it's not mandatory, that was just some extra paranoia I can remove
now.

Thanks,
drew

> 
> Thanks
> 
> Eric
> >  
> >      /* Bits other than [3:0] are RAZ/WI.  */
> >      QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
> > diff --git a/target/arm/monitor.c b/target/arm/monitor.c
> > index 19e3120eef95..157c487a1551 100644
> > --- a/target/arm/monitor.c
> > +++ b/target/arm/monitor.c
> > @@ -90,7 +90,7 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)
> >  }
> >  
> >  static const char *cpu_model_advertised_features[] = {
> > -    "aarch64", "pmu",
> > +    "aarch64", "pmu", "sve",
> >      NULL
> >  };
> >  
> > diff --git a/tests/arm-cpu-features.c b/tests/arm-cpu-features.c
> > index 31b1c15bb979..509e458e9c2f 100644
> > --- a/tests/arm-cpu-features.c
> > +++ b/tests/arm-cpu-features.c
> > @@ -158,6 +158,7 @@ static void test_query_cpu_model_expansion(const void *data)
> >  
> >      if (g_str_equal(qtest_get_arch(), "aarch64")) {
> >          assert_has_feature(qts, "max", "aarch64");
> > +        assert_has_feature(qts, "max", "sve");
> >          assert_has_feature(qts, "cortex-a57", "pmu");
> >          assert_has_feature(qts, "cortex-a57", "aarch64");
> >  
> > 
> 

Re: [Qemu-devel] [PATCH v2 06/14] target/arm: Allow SVE to be disabled via a CPU property

[Richard Henderson]

On 6/21/19 6:34 PM, Andrew Jones wrote:
> Since 97a28b0eeac14 ("target/arm: Allow VFP and Neon to be disabled via
> a CPU property") we can disable the 'max' cpu model's VFP and neon
> features, but there's no way to disable SVE. Add the 'sve=on|off'
> property to give it that flexibility. We also rename
> cpu_max_get/set_sve_vq to cpu_max_get/set_sve_max_vq in order for them
> to follow the typical *_get/set_<property-name> pattern.

I think perhaps the new property should not be overloaded on cpu->sve_max_vq.

At present you are generating an error for

    -cpu max,sve=off,sve_max_vq=2

but not for

    -cpu max,sve_max_vq=2,sve=off

and then there's the issue of

    -cpu max,sve_max_vq=2,sve=off,sve=on

discarding the earlier sve_max_vq setting.


> @@ -1129,6 +1129,14 @@ static void arm_cpu_realizefn(DeviceState *dev, Error **errp)
>          cpu->isar.mvfr0 = u;
>      }
>  
> +    if (!cpu->sve_max_vq) {
> +        uint64_t t;
> +
> +        t = cpu->isar.id_aa64pfr0;
> +        t = FIELD_DP64(t, ID_AA64PFR0, SVE, 0);
> +        cpu->isar.id_aa64pfr0 = t;
> +    }


I suppse the isar bits are initialized too late for you to be able to re-use
the ID_AA64PFR0.SVE field *as* the property?


>  static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
>                        uint64_t value)
>  {
> +    ARMCPU *cpu = env_archcpu(env);
>      int cur_el = arm_current_el(env);
> -    int old_len = sve_zcr_len_for_el(env, cur_el);
> -    int new_len;
> +    int old_len, new_len;
> +
> +    assert(cpu->sve_max_vq);

Certainly there's no reason for this assert, given the above.


r~

Re: [Qemu-devel] [PATCH v2 06/14] target/arm: Allow SVE to be disabled via a CPU property

[Andrew Jones]

On Wed, Jun 26, 2019 at 12:20:29PM +0200, Richard Henderson wrote:
> On 6/21/19 6:34 PM, Andrew Jones wrote:
> > Since 97a28b0eeac14 ("target/arm: Allow VFP and Neon to be disabled via
> > a CPU property") we can disable the 'max' cpu model's VFP and neon
> > features, but there's no way to disable SVE. Add the 'sve=on|off'
> > property to give it that flexibility. We also rename
> > cpu_max_get/set_sve_vq to cpu_max_get/set_sve_max_vq in order for them
> > to follow the typical *_get/set_<property-name> pattern.
> 
> I think perhaps the new property should not be overloaded on cpu->sve_max_vq.
> 
> At present you are generating an error for
> 
>     -cpu max,sve=off,sve_max_vq=2
> 
> but not for
> 
>     -cpu max,sve_max_vq=2,sve=off
> 
> and then there's the issue of
> 
>     -cpu max,sve_max_vq=2,sve=off,sve=on
> 
> discarding the earlier sve_max_vq setting.

Yeah, it might be best to add a new boolean in order for that last example
to work as expected. At least my expectation would be that we'd set the
max vq to 2, when sve is disabled nothing happens to it, but then when sve
is reenabled we'll still have that max vq 2. I'm guessing you're expecting
that too, since you brought it up. I think the other two examples above
are behaving as I'd expect them to.

> 
> 
> > @@ -1129,6 +1129,14 @@ static void arm_cpu_realizefn(DeviceState *dev, Error **errp)
> >          cpu->isar.mvfr0 = u;
> >      }
> >  
> > +    if (!cpu->sve_max_vq) {
> > +        uint64_t t;
> > +
> > +        t = cpu->isar.id_aa64pfr0;
> > +        t = FIELD_DP64(t, ID_AA64PFR0, SVE, 0);
> > +        cpu->isar.id_aa64pfr0 = t;
> > +    }
> 
> 
> I suppse the isar bits are initialized too late for you to be able to re-use
> the ID_AA64PFR0.SVE field *as* the property?

Hmm, that's probably worth trying. Also reworking vfp and neon to use the
fields as the properties, putting the sanity checks directly in the
property set accessor may work too. pmu and aarch64 could work like that
too, but those still have ARM_FEATURE_* bits, so they're not really the
same [yet].

> 
> 
> >  static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
> >                        uint64_t value)
> >  {
> > +    ARMCPU *cpu = env_archcpu(env);
> >      int cur_el = arm_current_el(env);
> > -    int old_len = sve_zcr_len_for_el(env, cur_el);
> > -    int new_len;
> > +    int old_len, new_len;
> > +
> > +    assert(cpu->sve_max_vq);
> 
> Certainly there's no reason for this assert, given the above.

Yeah, I'll remove for v3.

Thanks,
drew

Re: [Qemu-devel] [PATCH v2 06/14] target/arm: Allow SVE to be disabled via a CPU property

[Andrew Jones]

On Wed, Jun 26, 2019 at 03:52:44PM +0200, Andrew Jones wrote:
> On Wed, Jun 26, 2019 at 12:20:29PM +0200, Richard Henderson wrote:
> > On 6/21/19 6:34 PM, Andrew Jones wrote:
> > > Since 97a28b0eeac14 ("target/arm: Allow VFP and Neon to be disabled via
> > > a CPU property") we can disable the 'max' cpu model's VFP and neon
> > > features, but there's no way to disable SVE. Add the 'sve=on|off'
> > > property to give it that flexibility. We also rename
> > > cpu_max_get/set_sve_vq to cpu_max_get/set_sve_max_vq in order for them
> > > to follow the typical *_get/set_<property-name> pattern.
> > 
> > I think perhaps the new property should not be overloaded on cpu->sve_max_vq.
> > 
> > At present you are generating an error for
> > 
> >     -cpu max,sve=off,sve_max_vq=2
> > 
> > but not for
> > 
> >     -cpu max,sve_max_vq=2,sve=off
> > 
> > and then there's the issue of
> > 
> >     -cpu max,sve_max_vq=2,sve=off,sve=on
> > 
> > discarding the earlier sve_max_vq setting.
> 
> Yeah, it might be best to add a new boolean in order for that last example
> to work as expected. At least my expectation would be that we'd set the
> max vq to 2, when sve is disabled nothing happens to it, but then when sve
> is reenabled we'll still have that max vq 2. I'm guessing you're expecting
> that too, since you brought it up. I think the other two examples above
> are behaving as I'd expect them to.
> 
> > 
> > 
> > > @@ -1129,6 +1129,14 @@ static void arm_cpu_realizefn(DeviceState *dev, Error **errp)
> > >          cpu->isar.mvfr0 = u;
> > >      }
> > >  
> > > +    if (!cpu->sve_max_vq) {
> > > +        uint64_t t;
> > > +
> > > +        t = cpu->isar.id_aa64pfr0;
> > > +        t = FIELD_DP64(t, ID_AA64PFR0, SVE, 0);
> > > +        cpu->isar.id_aa64pfr0 = t;
> > > +    }
> > 
> > 
> > I suppse the isar bits are initialized too late for you to be able to re-use
> > the ID_AA64PFR0.SVE field *as* the property?
> 
> Hmm, that's probably worth trying. Also reworking vfp and neon to use the
> fields as the properties, putting the sanity checks directly in the
> property set accessor may work too. pmu and aarch64 could work like that
> too, but those still have ARM_FEATURE_* bits, so they're not really the
> same [yet].

I played with this for the PMU, i.e. I removed 'has_pmu' and used its bits
in id_aa64dfr0. Everything worked, so it's possible. I think the changes
to convert the PMU, VFP, and NEON are too far outside the scope of this
series to do them now, but I'll attempt to add a has-sve boolean without
adding a "has_sve" boolean in this series by using id_aa64pfr0.

Thanks,
drew

[Qemu-devel] [PATCH v2 07/14] target/arm/cpu64: max cpu: Introduce sve<vl-bits> properties

[Andrew Jones]

Introduce cpu properties to give fine control over SVE vector lengths.
We introduce a property for each valid length up to the current
maximum supported, which is 2048-bits. The properties are named, e.g.
sve128, sve256, sve512, ..., where the number is the number of bits.

It's now possible to do something like -cpu max,sve-max-vq=4,sve384=off
to provide a guest vector lengths 128, 256, and 512 bits. The resulting
set must conform to the architectural constraint of having all power-of-2
lengths smaller than the maximum length present. It's also possible to
only provide sve<vl-bits> properties, e.g. -cpu max,sve512=on. That
example provides the machine with 128, 256, and 512 bit vector lengths.
It doesn't hurt to explicitly ask for all expected vector lengths,
which is what, for example, libvirt should do.

Note1, it is not possible to use sve<vl-bits> properties before
sve-max-vq, e.g. -cpu max,sve384=off,sve-max-vq=4, as supporting
that overly complicates the user input validation.

Note2, while one might expect -cpu max,sve-max-vq=4,sve512=on to be the
same as -cpu max,sve512=on, they are not. The former enables all vector
lengths 512 bits and smaller, while the latter only sets the 512-bit
length and its smaller power-of-2 lengths. It's probably best not to use
sve-max-vq with sve<vl-bits> properties, but it can't be completely
forbidden as we want qmp_query_cpu_model_expansion to work with guests
launched with e.g. -cpu max,sve-max-vq=8 on their command line.

Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 target/arm/cpu.c         |   6 +
 target/arm/cpu.h         |  14 ++
 target/arm/cpu64.c       | 360 ++++++++++++++++++++++++++++++++++++++-
 target/arm/helper.c      |  11 +-
 target/arm/monitor.c     |  16 ++
 tests/arm-cpu-features.c | 217 +++++++++++++++++++++++
 6 files changed, 620 insertions(+), 4 deletions(-)

diff --git a/target/arm/cpu.c b/target/arm/cpu.c
index f08e178fc84b..e060a0d9df0e 100644
--- a/target/arm/cpu.c
+++ b/target/arm/cpu.c
@@ -1019,6 +1019,12 @@ static void arm_cpu_realizefn(DeviceState *dev, Error **errp)
         return;
     }
 
+    arm_cpu_sve_finalize(cpu, &local_err);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return;
+    }
+
     if (arm_feature(env, ARM_FEATURE_AARCH64) &&
         cpu->has_vfp != cpu->has_neon) {
         /*
diff --git a/target/arm/cpu.h b/target/arm/cpu.h
index f9da672be575..cbb155cf72a5 100644
--- a/target/arm/cpu.h
+++ b/target/arm/cpu.h
@@ -184,8 +184,13 @@ typedef struct {
 
 #ifdef TARGET_AARCH64
 # define ARM_MAX_VQ    16
+void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp);
+uint32_t arm_cpu_vq_map_next_smaller(ARMCPU *cpu, uint32_t vq);
 #else
 # define ARM_MAX_VQ    1
+static inline void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp) { }
+static inline uint32_t arm_cpu_vq_map_next_smaller(ARMCPU *cpu, uint32_t vq)
+{ return 0; }
 #endif
 
 typedef struct ARMVectorReg {
@@ -915,6 +920,15 @@ struct ARMCPU {
 
     /* Used to set the maximum vector length the cpu will support.  */
     uint32_t sve_max_vq;
+
+    /*
+     * In sve_vq_map each set bit is a supported vector length of
+     * (bit-number + 1) * 16 bytes, i.e. each bit number + 1 is the vector
+     * length in quadwords. We need a map size twice the maximum
+     * quadword length though because we use two bits for each vector
+     * length in order to track three states: uninitialized, off, and on.
+     */
+    DECLARE_BITMAP(sve_vq_map, ARM_MAX_VQ * 2);
 };
 
 void arm_cpu_post_init(Object *obj);
diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
index 02ada65f240c..5def82111dee 100644
--- a/target/arm/cpu64.c
+++ b/target/arm/cpu64.c
@@ -257,6 +257,149 @@ static void aarch64_a72_initfn(Object *obj)
     define_arm_cp_regs(cpu, cortex_a72_a57_a53_cp_reginfo);
 }
 
+/*
+ * While we eventually use cpu->sve_vq_map as a typical bitmap, where each vq
+ * has only two states (off/on), until we've finalized the map at realize time
+ * we use an extra bit, at the vq - 1 + ARM_MAX_VQ bit number, to also allow
+ * tracking of the uninitialized state. The arm_vq_state typedef and following
+ * functions allow us to more easily work with the bitmap. Also, while the map
+ * is still initializing, sve-max-vq has an additional three states, bringing
+ * the number of its states to five, which are the following:
+ *
+ * sve-max-vq:
+ *   0:    SVE is disabled. The default value for a vq in the map is 'OFF'.
+ *  -1:    SVE is enabled, but neither sve-max-vq nor sve<vl-bits> properties
+ *         have yet been specified by the user. The default value for a vq in
+ *         the map is 'ON'.
+ *  -2:    SVE is enabled and one or more sve<vl-bits> properties have been
+ *         set to 'OFF' by the user, but no sve<vl-bits> properties have yet
+ *         been set to 'ON'. The user is now blocked from setting sve-max-vq
+ *         and the default value for a vq in the map is 'ON'.
+ *  -3:    SVE is enabled and one or more sve<vl-bits> properties have been
+ *         set to 'ON' by the user. The user is blocked from setting sve-max-vq
+ *         and the default value for a vq in the map is 'OFF'. sve-max-vq never
+ *         transitions back to -2, even if later inputs disable the vector
+ *         lengths that initially transitioned sve-max-vq to this state. This
+ *         avoids the default values from flip-flopping.
+ *  [1-ARM_MAX_VQ]: SVE is enabled and the user has specified a valid
+ *                  sve-max-vq. The sve-max-vq specified vq and all smaller
+ *                  vq's will be initially enabled. All larger vq's will have
+ *                  a default of 'OFF'.
+ */
+#define ARM_SVE_INIT          -1
+#define ARM_VQ_DEFAULT_ON     -2
+#define ARM_VQ_DEFAULT_OFF    -3
+
+#define arm_sve_have_max_vq(cpu) ((int32_t)(cpu)->sve_max_vq > 0)
+
+typedef enum arm_vq_state {
+    ARM_VQ_OFF,
+    ARM_VQ_ON,
+    ARM_VQ_UNINITIALIZED,
+} arm_vq_state;
+
+static arm_vq_state arm_cpu_vq_map_get(ARMCPU *cpu, int vq)
+{
+    assert(vq <= ARM_MAX_VQ);
+
+    return test_bit(vq - 1, cpu->sve_vq_map) |
+           test_bit(vq - 1 + ARM_MAX_VQ, cpu->sve_vq_map) << 1;
+}
+
+static void arm_cpu_vq_map_set(ARMCPU *cpu, int vq, arm_vq_state state)
+{
+    assert(state == ARM_VQ_OFF || state == ARM_VQ_ON);
+    assert(vq <= ARM_MAX_VQ);
+
+    clear_bit(vq - 1 + ARM_MAX_VQ, cpu->sve_vq_map);
+
+    if (state == ARM_VQ_ON) {
+        set_bit(vq - 1, cpu->sve_vq_map);
+    } else {
+        clear_bit(vq - 1, cpu->sve_vq_map);
+    }
+}
+
+static void arm_cpu_vq_map_init(ARMCPU *cpu)
+{
+    bitmap_zero(cpu->sve_vq_map, ARM_MAX_VQ * 2);
+    bitmap_set(cpu->sve_vq_map, ARM_MAX_VQ, ARM_MAX_VQ);
+}
+
+static bool arm_cpu_vq_map_is_finalized(ARMCPU *cpu)
+{
+    DECLARE_BITMAP(map, ARM_MAX_VQ * 2);
+
+    bitmap_zero(map, ARM_MAX_VQ * 2);
+    bitmap_set(map, ARM_MAX_VQ, ARM_MAX_VQ);
+    bitmap_and(map, map, cpu->sve_vq_map, ARM_MAX_VQ * 2);
+
+    return bitmap_empty(map, ARM_MAX_VQ * 2);
+}
+
+static void arm_cpu_vq_map_finalize(ARMCPU *cpu)
+{
+    Error *err = NULL;
+    char name[8];
+    uint32_t vq;
+    bool value;
+
+    /*
+     * We use the property get accessor because it knows what default
+     * values to return for uninitialized vector lengths.
+     */
+    for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
+        sprintf(name, "sve%d", vq * 128);
+        value = object_property_get_bool(OBJECT(cpu), name, &err);
+        assert(!err);
+        if (value) {
+            arm_cpu_vq_map_set(cpu, vq, ARM_VQ_ON);
+        } else {
+            arm_cpu_vq_map_set(cpu, vq, ARM_VQ_OFF);
+        }
+    }
+
+    assert(arm_cpu_vq_map_is_finalized(cpu));
+}
+
+void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp)
+{
+    Error *err = NULL;
+
+    if (!cpu->sve_max_vq) {
+        bitmap_zero(cpu->sve_vq_map, ARM_MAX_VQ * 2);
+        return;
+    }
+
+    if (cpu->sve_max_vq == ARM_SVE_INIT) {
+        object_property_set_uint(OBJECT(cpu), ARM_MAX_VQ, "sve-max-vq", &err);
+        if (err) {
+            error_propagate(errp, err);
+            return;
+        }
+        assert(cpu->sve_max_vq == ARM_MAX_VQ);
+        arm_cpu_vq_map_finalize(cpu);
+    } else {
+        arm_cpu_vq_map_finalize(cpu);
+        if (!arm_sve_have_max_vq(cpu)) {
+            cpu->sve_max_vq = arm_cpu_vq_map_next_smaller(cpu, ARM_MAX_VQ + 1);
+        }
+    }
+
+    assert(cpu->sve_max_vq == arm_cpu_vq_map_next_smaller(cpu, ARM_MAX_VQ + 1));
+}
+
+uint32_t arm_cpu_vq_map_next_smaller(ARMCPU *cpu, uint32_t vq)
+{
+    uint32_t bitnum;
+
+    assert(vq <= ARM_MAX_VQ + 1);
+    assert(arm_cpu_vq_map_is_finalized(cpu));
+
+    bitnum = find_last_bit(cpu->sve_vq_map, vq - 1);
+    return bitnum == vq - 1 ? 0 : bitnum + 1;
+}
+
 static void cpu_max_get_sve_max_vq(Object *obj, Visitor *v, const char *name,
                                    void *opaque, Error **errp)
 {
@@ -283,12 +426,203 @@ static void cpu_max_set_sve_max_vq(Object *obj, Visitor *v, const char *name,
         return;
     }
 
+    /*
+     * It gets complicated trying to support both sve-max-vq and
+     * sve<vl-bits> properties together, so we mostly don't. We
+     * do allow both if sve-max-vq is specified first and only once
+     * though.
+     */
+    if (cpu->sve_max_vq != ARM_SVE_INIT) {
+        error_setg(errp, "sve<vl-bits> in use or sve-max-vq already "
+                   "specified");
+        error_append_hint(errp, "sve-max-vq must come before all "
+                          "sve<vl-bits> properties and it must only "
+                          "be specified once.\n");
+        return;
+    }
+
     cpu->sve_max_vq = value;
 
     if (cpu->sve_max_vq == 0 || cpu->sve_max_vq > ARM_MAX_VQ) {
         error_setg(errp, "unsupported SVE vector length");
         error_append_hint(errp, "Valid sve-max-vq in range [1-%d]\n",
                           ARM_MAX_VQ);
+    } else {
+        uint32_t vq;
+
+        for (vq = 1; vq <= cpu->sve_max_vq; ++vq) {
+            char name[8];
+            sprintf(name, "sve%d", vq * 128);
+            object_property_set_bool(obj, true, name, &err);
+            if (err) {
+                error_propagate(errp, err);
+                return;
+            }
+        }
+    }
+}
+
+static void cpu_arm_get_sve_vq(Object *obj, Visitor *v, const char *name,
+                               void *opaque, Error **errp)
+{
+    ARMCPU *cpu = ARM_CPU(obj);
+    int vq = atoi(&name[3]) / 128;
+    arm_vq_state vq_state;
+    bool value;
+
+    vq_state = arm_cpu_vq_map_get(cpu, vq);
+
+    if (!cpu->sve_max_vq) {
+        /* All vector lengths are disabled when SVE is off. */
+        value = false;
+    } else if (vq_state == ARM_VQ_ON) {
+        value = true;
+    } else if (vq_state == ARM_VQ_OFF) {
+        value = false;
+    } else {
+        /*
+         * vq is uninitialized. We pick a default here based on the
+         * the state of sve-max-vq and other sve<vl-bits> properties.
+         */
+        if (arm_sve_have_max_vq(cpu)) {
+            /*
+             * If we have sve-max-vq, then all remaining uninitialized
+             * vq's are 'OFF'.
+             */
+            value = false;
+        } else {
+            switch (cpu->sve_max_vq) {
+            case ARM_SVE_INIT:
+            case ARM_VQ_DEFAULT_ON:
+                value = true;
+                break;
+            case ARM_VQ_DEFAULT_OFF:
+                value = false;
+                break;
+            }
+        }
+    }
+
+    visit_type_bool(v, name, &value, errp);
+}
+
+static void cpu_arm_set_sve_vq(Object *obj, Visitor *v, const char *name,
+                               void *opaque, Error **errp)
+{
+    ARMCPU *cpu = ARM_CPU(obj);
+    int vq = atoi(&name[3]) / 128;
+    arm_vq_state vq_state;
+    Error *err = NULL;
+    uint32_t max_vq = 0;
+    bool value;
+
+    visit_type_bool(v, name, &value, &err);
+    if (err) {
+        error_propagate(errp, err);
+        return;
+    }
+
+    if (value && !cpu->sve_max_vq) {
+        error_setg(errp, "cannot enable %s", name);
+        error_append_hint(errp, "SVE has been disabled with sve=off\n");
+        return;
+    } else if (!cpu->sve_max_vq) {
+        /*
+         * We don't complain about disabling vector lengths when SVE
+         * is off, but we don't do anything either.
+         */
+        return;
+    }
+
+    if (arm_sve_have_max_vq(cpu)) {
+        max_vq = cpu->sve_max_vq;
+    } else {
+        if (value) {
+            cpu->sve_max_vq = ARM_VQ_DEFAULT_OFF;
+        } else if (cpu->sve_max_vq != ARM_VQ_DEFAULT_OFF) {
+            cpu->sve_max_vq = ARM_VQ_DEFAULT_ON;
+        }
+    }
+
+    /*
+     * We need to know the maximum vector length, which may just currently
+     * be the maximum length, in order to validate the enabling/disabling
+     * of this vector length. We use the property get accessor in order to
+     * get the appropriate default value for any uninitialized lengths.
+     */
+    if (!max_vq) {
+        char tmp[8];
+        bool s;
+
+        for (max_vq = ARM_MAX_VQ; max_vq >= 1; --max_vq) {
+            sprintf(tmp, "sve%d", max_vq * 128);
+            s = object_property_get_bool(OBJECT(cpu), tmp, &err);
+            assert(!err);
+            if (s) {
+                break;
+            }
+        }
+    }
+
+    if (arm_sve_have_max_vq(cpu) && value && vq > cpu->sve_max_vq) {
+        error_setg(errp, "cannot enable %s", name);
+        error_append_hint(errp, "vq=%d (%d bits) is larger than the "
+                          "maximum vector length, sve-max-vq=%d "
+                          "(%d bits)\n", vq, vq * 128,
+                          cpu->sve_max_vq, cpu->sve_max_vq * 128);
+    } else if (arm_sve_have_max_vq(cpu) && !value && vq == cpu->sve_max_vq) {
+        error_setg(errp, "cannot disable %s", name);
+        error_append_hint(errp, "The maximum vector length must be "
+                          "enabled, sve-max-vq=%d (%d bits)\n",
+                          cpu->sve_max_vq, cpu->sve_max_vq * 128);
+    } else if (arm_sve_have_max_vq(cpu) && !value && vq < cpu->sve_max_vq &&
+               is_power_of_2(vq)) {
+        error_setg(errp, "cannot disable %s", name);
+        error_append_hint(errp, "vq=%d (%d bits) is required as it is a "
+                          "power-of-2 length smaller than the maximum, "
+                          "sve-max-vq=%d (%d bits)\n", vq, vq * 128,
+                          cpu->sve_max_vq, cpu->sve_max_vq * 128);
+    } else if (!value && vq < max_vq && is_power_of_2(vq)) {
+        error_setg(errp, "cannot disable %s", name);
+        error_append_hint(errp, "Vector length %d-bits is required as it "
+                          "is a power-of-2 length smaller than another "
+                          "enabled vector length. Disable all larger vector "
+                          "lengths first.\n", vq * 128);
+    } else {
+        if (value) {
+            bool fail = false;
+            uint32_t s;
+
+            /*
+             * Enabling a vector length automatically enables all
+             * uninitialized power-of-2 lengths smaller than it, as
+             * per the architecture.
+             */
+            for (s = 1; s < vq; ++s) {
+                if (is_power_of_2(s)) {
+                    vq_state = arm_cpu_vq_map_get(cpu, s);
+                    if (vq_state == ARM_VQ_UNINITIALIZED) {
+                        arm_cpu_vq_map_set(cpu, s, ARM_VQ_ON);
+                    } else if (vq_state == ARM_VQ_OFF) {
+                        fail = true;
+                        break;
+                    }
+                }
+            }
+
+            if (fail) {
+                error_setg(errp, "cannot enable %s", name);
+                error_append_hint(errp, "Vector length %d-bits is disabled "
+                                  "and is a power-of-2 length smaller than "
+                                  "%s. All power-of-2 vector lengths smaller "
+                                  "than the maximum length are required.\n",
+                                  s * 128, name);
+            } else {
+                arm_cpu_vq_map_set(cpu, vq, ARM_VQ_ON);
+            }
+        } else {
+            arm_cpu_vq_map_set(cpu, vq, ARM_VQ_OFF);
+        }
     }
 }
 
@@ -318,10 +652,11 @@ static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
         /*
          * We handle the -cpu <cpu>,sve=off,sve=on case by reinitializing,
          * but otherwise we don't do anything as an sve=on could come after
-         * a sve-max-vq setting.
+         * a sve-max-vq or sve<vl-bits> setting.
          */
         if (!cpu->sve_max_vq) {
-            cpu->sve_max_vq = ARM_MAX_VQ;
+            cpu->sve_max_vq = ARM_SVE_INIT;
+            arm_cpu_vq_map_init(cpu);
         }
     } else {
         cpu->sve_max_vq = 0;
@@ -336,6 +671,7 @@ static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
 static void aarch64_max_initfn(Object *obj)
 {
     ARMCPU *cpu = ARM_CPU(obj);
+    uint32_t vq;
 
     if (kvm_enabled()) {
         kvm_arm_set_cpu_features_from_host(cpu);
@@ -420,11 +756,29 @@ static void aarch64_max_initfn(Object *obj)
         cpu->dcz_blocksize = 7; /*  512 bytes */
 #endif
 
-        cpu->sve_max_vq = ARM_MAX_VQ;
+        /*
+         * sve_max_vq is initially unspecified, but must be initialized to a
+         * non-zero value (ARM_SVE_INIT) to indicate that this cpu type has
+         * SVE. It will be finalized in arm_cpu_realizefn().
+         */
+        cpu->sve_max_vq = ARM_SVE_INIT;
         object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_max_vq,
                             cpu_max_set_sve_max_vq, NULL, NULL, &error_fatal);
         object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
                             cpu_arm_set_sve, NULL, NULL, &error_fatal);
+
+        /*
+         * sve_vq_map uses a special state while setting properties, so
+         * we initialize it here with its init function and finalize it
+         * in arm_cpu_realizefn().
+         */
+        arm_cpu_vq_map_init(cpu);
+        for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
+            char name[8];
+            sprintf(name, "sve%d", vq * 128);
+            object_property_add(obj, name, "bool", cpu_arm_get_sve_vq,
+                                cpu_arm_set_sve_vq, NULL, NULL, &error_fatal);
+        }
     }
 }
 
diff --git a/target/arm/helper.c b/target/arm/helper.c
index f500ccb6d31b..b7b719dba57f 100644
--- a/target/arm/helper.c
+++ b/target/arm/helper.c
@@ -5324,7 +5324,16 @@ static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
 
     /* Bits other than [3:0] are RAZ/WI.  */
     QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
-    raw_write(env, ri, value & 0xf);
+    value &= 0xf;
+
+    if (value) {
+        /* get next vq that is smaller than or equal to value's vq */
+        uint32_t vq = value + 1;
+        vq = arm_cpu_vq_map_next_smaller(cpu, vq + 1);
+        value = vq - 1;
+    }
+
+    raw_write(env, ri, value);
 
     /*
      * Because we arrived here, we know both FP and SVE are enabled;
diff --git a/target/arm/monitor.c b/target/arm/monitor.c
index 157c487a1551..1e213906fd8f 100644
--- a/target/arm/monitor.c
+++ b/target/arm/monitor.c
@@ -89,8 +89,24 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)
     return head;
 }
 
+QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
+
+/*
+ * These are cpu model features we want to advertise. The order here
+ * matters as this is the order in which qmp_query_cpu_model_expansion
+ * will attempt to set them. If there are dependencies between features,
+ * as there are with the sve<vl-bits> features, then the order that
+ * considers those dependencies must be used.
+ *
+ * The sve<vl-bits> features need to be in reverse order in order to
+ * enable/disable the largest vector lengths first, ensuring all
+ * power-of-2 vector lengths smaller can also be enabled/disabled.
+ */
 static const char *cpu_model_advertised_features[] = {
     "aarch64", "pmu", "sve",
+    "sve2048", "sve1920", "sve1792", "sve1664", "sve1536", "sve1408",
+    "sve1280", "sve1152", "sve1024", "sve896", "sve768", "sve640",
+    "sve512", "sve384", "sve256", "sve128",
     NULL
 };
 
diff --git a/tests/arm-cpu-features.c b/tests/arm-cpu-features.c
index 509e458e9c2f..a4bf6aec00df 100644
--- a/tests/arm-cpu-features.c
+++ b/tests/arm-cpu-features.c
@@ -13,6 +13,18 @@
 #include "qapi/qmp/qdict.h"
 #include "qapi/qmp/qjson.h"
 
+#if __SIZEOF_LONG__ == 8
+#define BIT(n) (1UL << (n))
+#else
+#define BIT(n) (1ULL << (n))
+#endif
+
+/*
+ * We expect the SVE max-vq to be 16. Also it must be <= 64
+ * for our test code, otherwise 'vls' can't just be a uint64_t.
+ */
+#define SVE_MAX_VQ 16
+
 #define MACHINE    "-machine virt,gic-version=max "
 #define QUERY_HEAD "{ 'execute': 'query-cpu-model-expansion', " \
                      "'arguments': { 'type': 'full', "
@@ -137,6 +149,201 @@ static void assert_bad_props(QTestState *qts, const char *cpu_type)
     qobject_unref(resp);
 }
 
+static void resp_get_sve_vls(QDict *resp, uint64_t *vls, uint32_t *max_vq)
+{
+    const QDictEntry *e;
+    QDict *qdict;
+    int n = 0;
+
+    *vls = 0;
+
+    qdict = resp_get_props(resp);
+
+    for (e = qdict_first(qdict); e; e = qdict_next(qdict, e)) {
+        if (strlen(e->key) > 3 && !strncmp(e->key, "sve", 3) &&
+            g_ascii_isdigit(e->key[3])) {
+            char *endptr;
+            int bits;
+
+            bits = g_ascii_strtoll(&e->key[3], &endptr, 10);
+            if (!bits || *endptr != '\0') {
+                continue;
+            }
+
+            if (qdict_get_bool(qdict, e->key)) {
+                *vls |= BIT((bits / 128) - 1);
+            }
+            ++n;
+        }
+    }
+
+    g_assert(n == SVE_MAX_VQ);
+
+    *max_vq = !*vls ? 0 : 64 - __builtin_clzll(*vls);
+}
+
+static uint64_t sve_get_vls(QTestState *qts, const char *cpu_type,
+                            const char *fmt, ...)
+{
+    QDict *resp;
+    uint64_t vls;
+    uint32_t max_vq;
+
+    if (fmt) {
+        QDict *args;
+        va_list ap;
+
+        va_start(ap, fmt);
+        args = qdict_from_vjsonf_nofail(fmt, ap);
+        va_end(ap);
+
+        resp = qtest_qmp(qts, QUERY_HEAD "'model': { 'name': %s, "
+                                                    "'props': %p }"
+                              QUERY_TAIL, cpu_type, args);
+    } else {
+        resp = do_query_no_props(qts, cpu_type);
+    }
+
+    g_assert(resp);
+    resp_get_sve_vls(resp, &vls, &max_vq);
+    qobject_unref(resp);
+
+    return vls;
+}
+
+#define assert_sve_vls(qts, cpu_type, expected_vls, fmt, ...) \
+    g_assert(sve_get_vls(qts, cpu_type, fmt, ##__VA_ARGS__) == expected_vls)
+
+static void sve_tests_default(QTestState *qts, const char *cpu_type)
+{
+    /*
+     * With no sve-max-vq or sve<vl-bits> properties on the command line
+     * the default is to have all vector lengths enabled.
+     */
+    assert_sve_vls(qts, cpu_type, BIT(SVE_MAX_VQ) - 1, NULL);
+
+    /*
+     * -------------------------------------------------------------------
+     *               power-of-2(vq)   all-power-            can      can
+     *                                of-2(< vq)          enable   disable
+     * -------------------------------------------------------------------
+     * vq < max_vq      no            MUST*                yes      yes
+     * vq < max_vq      yes           MUST*                yes      no
+     * -------------------------------------------------------------------
+     * vq == max_vq     n/a           MUST*                yes**    yes**
+     * -------------------------------------------------------------------
+     * vq > max_vq      n/a           no                   no       yes
+     * vq > max_vq      n/a           yes                  yes      yes
+     * -------------------------------------------------------------------
+     *
+     * [*] "MUST" means this requirement must already be satisfied,
+     *     otherwise 'max_vq' couldn't itself be enabled.
+     *
+     * [**] Not testable with the QMP interface, only with the command line.
+     */
+
+    /* max_vq := 8 */
+    assert_sve_vls(qts, cpu_type, 0x8b, "{ 'sve1024': true }");
+
+    /* max_vq := 8, vq < max_vq, !power-of-2(vq) */
+    assert_sve_vls(qts, cpu_type, 0x8f,
+                   "{ 'sve1024': true, 'sve384': true }");
+    assert_sve_vls(qts, cpu_type, 0x8b,
+                   "{ 'sve1024': true, 'sve384': false }");
+
+    /* max_vq := 8, vq < max_vq, power-of-2(vq) */
+    assert_sve_vls(qts, cpu_type, 0x8b,
+                   "{ 'sve1024': true, 'sve256': true }");
+    assert_error(qts, cpu_type, "cannot disable sve256",
+                 "{ 'sve1024': true, 'sve256': false }");
+
+    /*
+     * max_vq := 3, vq > max_vq, !all-power-of-2(< vq)
+     *
+     * If given sve384=on,sve512=off,sve640=on the command line error would be
+     * "cannot enable sve640", but QMP visits the vector lengths in reverse
+     * order, so we get "cannot disable sve512" instead. The command line would
+     * also give that error if given sve384=on,sve640=on,sve512=off, so this is
+     * all fine. The important thing is that we get an error.
+     */
+    assert_error(qts, cpu_type, "cannot disable sve512",
+                 "{ 'sve384': true, 'sve512': false, 'sve640': true }");
+
+    /*
+     * We can disable power-of-2 vector lengths when all larger lengths
+     * are also disabled. The shorter, sve384=on,sve512=off,sve640=off
+     * works on the command line, but QMP doesn't know that all the
+     * vector lengths larger than 384-bits will be disabled until it
+     * sees the enabling of sve384, which comes near the end since it
+     * visits the lengths in reverse order. So we just have to explicitly
+     * disable them all.
+     */
+    assert_sve_vls(qts, cpu_type, 0x7,
+                   "{ 'sve384': true, 'sve512': false, 'sve640': false, "
+                   "  'sve768': false, 'sve896': false, 'sve1024': false, "
+                   "  'sve1152': false, 'sve1280': false, 'sve1408': false, "
+                   "  'sve1536': false, 'sve1664': false, 'sve1792': false, "
+                   "  'sve1920': false, 'sve2048': false }");
+
+    /* max_vq := 3, vq > max_vq, all-power-of-2(< vq) */
+    assert_sve_vls(qts, cpu_type, 0x1f,
+                   "{ 'sve384': true, 'sve512': true, 'sve640': true }");
+    assert_sve_vls(qts, cpu_type, 0xf,
+                   "{ 'sve384': true, 'sve512': true, 'sve640': false }");
+}
+
+static void sve_tests_sve_max_vq_8(const void *data)
+{
+    QTestState *qts;
+
+    qts = qtest_init(MACHINE "-cpu max,sve-max-vq=8");
+
+    assert_sve_vls(qts, "max", BIT(8) - 1, NULL);
+
+    /*
+     * Disabling the max-vq set by sve-max-vq is not allowed, but
+     * of course enabling it is OK.
+     */
+    assert_error(qts, "max", "cannot disable sve1024", "{ 'sve1024': false }");
+    assert_sve_vls(qts, "max", 0xff, "{ 'sve1024': true }");
+
+    /*
+     * Enabling anything larger than max-vq set by sve-max-vq is not
+     * allowed, but of course disabling everything larger is OK.
+     */
+    assert_error(qts, "max", "cannot enable sve1152", "{ 'sve1152': true }");
+    assert_sve_vls(qts, "max", 0xff, "{ 'sve1152': false }");
+
+    /*
+     * We can disable non power-of-2 lengths smaller than the max-vq
+     * set by sve-max-vq, but not power-of-2 lengths.
+     */
+    assert_sve_vls(qts, "max", 0xfb, "{ 'sve384': false }");
+    assert_error(qts, "max", "cannot disable sve256", "{ 'sve256': false }");
+
+    qtest_quit(qts);
+}
+
+static void sve_tests_sve_off(const void *data)
+{
+    QTestState *qts;
+
+    qts = qtest_init(MACHINE "-cpu max,sve=off");
+
+    /*
+     * SVE is off, so the map should be empty.
+     */
+    assert_sve_vls(qts, "max", 0, NULL);
+
+    /*
+     * We can't turn anything on, but off is OK.
+     */
+    assert_error(qts, "max", "cannot enable sve128", "{ 'sve128': true }");
+    assert_sve_vls(qts, "max", 0, "{ 'sve128': false }");
+
+    qtest_quit(qts);
+}
+
 static void test_query_cpu_model_expansion(const void *data)
 {
     QTestState *qts;
@@ -159,9 +366,12 @@ static void test_query_cpu_model_expansion(const void *data)
     if (g_str_equal(qtest_get_arch(), "aarch64")) {
         assert_has_feature(qts, "max", "aarch64");
         assert_has_feature(qts, "max", "sve");
+        assert_has_feature(qts, "max", "sve128");
         assert_has_feature(qts, "cortex-a57", "pmu");
         assert_has_feature(qts, "cortex-a57", "aarch64");
 
+        sve_tests_default(qts, "max");
+
         /* Test that features that depend on KVM generate errors without. */
         assert_error(qts, "max",
                      "'aarch64' feature cannot be disabled "
@@ -213,6 +423,13 @@ int main(int argc, char **argv)
     qtest_add_data_func("/arm/query-cpu-model-expansion",
                         NULL, test_query_cpu_model_expansion);
 
+    if (g_str_equal(qtest_get_arch(), "aarch64")) {
+        qtest_add_data_func("/arm/max/query-cpu-model-expansion/sve-max-vq-8",
+                            NULL, sve_tests_sve_max_vq_8);
+        qtest_add_data_func("/arm/max/query-cpu-model-expansion/sve-off",
+                            NULL, sve_tests_sve_off);
+    }
+
     if (kvm_available) {
         qtest_add_data_func("/arm/kvm/query-cpu-model-expansion",
                             NULL, test_query_cpu_model_expansion_kvm);
-- 
2.20.1

Re: [Qemu-devel] [PATCH v2 07/14] target/arm/cpu64: max cpu: Introduce sve<vl-bits> properties

[Dave Martin]

On Fri, Jun 21, 2019 at 05:34:15PM +0100, Andrew Jones wrote:
> Introduce cpu properties to give fine control over SVE vector lengths.
> We introduce a property for each valid length up to the current
> maximum supported, which is 2048-bits. The properties are named, e.g.
> sve128, sve256, sve512, ..., where the number is the number of bits.
> 
> It's now possible to do something like -cpu max,sve-max-vq=4,sve384=off
> to provide a guest vector lengths 128, 256, and 512 bits. The resulting
> set must conform to the architectural constraint of having all power-of-2
> lengths smaller than the maximum length present. It's also possible to
> only provide sve<vl-bits> properties, e.g. -cpu max,sve512=on. That
> example provides the machine with 128, 256, and 512 bit vector lengths.
> It doesn't hurt to explicitly ask for all expected vector lengths,
> which is what, for example, libvirt should do.
> 
> Note1, it is not possible to use sve<vl-bits> properties before
> sve-max-vq, e.g. -cpu max,sve384=off,sve-max-vq=4, as supporting
> that overly complicates the user input validation.
> 
> Note2, while one might expect -cpu max,sve-max-vq=4,sve512=on to be the
> same as -cpu max,sve512=on, they are not. The former enables all vector
> lengths 512 bits and smaller, while the latter only sets the 512-bit
> length and its smaller power-of-2 lengths. It's probably best not to use
> sve-max-vq with sve<vl-bits> properties, but it can't be completely
> forbidden as we want qmp_query_cpu_model_expansion to work with guests
> launched with e.g. -cpu max,sve-max-vq=8 on their command line.

Supporting both options together in a non-idempotent way seems to
complicate things.

Would it be simpler to allow sve-max-vq only when there are no sve<bits>
options?

Alternatively, the two options would be defined so that their meanings
are independent of parse order.

So, way sve-max-vq=<max> means that:

 * all VQs up to max for which there is no corresponding sve<bits>=off,
   are enabled; and

 * VQ max is enabled; and

 * all VQs exceeding max are disabled.

While sve<bits>=(on|off) means

 * the VQ coresponding to <bits> is enabled (for on) or disabled (for
   off).
 
After parsing all the options, you check that the sve-max-vq and
sve<bits> optinos are consistent.  If you disallow duplicate sve-max-vq
or sve<bits> options, then there is no possibility of ambiguity and the
order or options doesn't matter.

(There may be constraints on the way qemu options parsing works that
make this hard, though...)

Having sve-max-vq in 128-bit units while sve<bits> are named in terms of
bit counts also feels a bit odd now.

[...]

Cheers
---Dave

Re: [Qemu-devel] [PATCH v2 07/14] target/arm/cpu64: max cpu: Introduce sve<vl-bits> properties

[Andrew Jones]

On Mon, Jun 24, 2019 at 12:05:26PM +0100, Dave Martin wrote:
> On Fri, Jun 21, 2019 at 05:34:15PM +0100, Andrew Jones wrote:
> > Introduce cpu properties to give fine control over SVE vector lengths.
> > We introduce a property for each valid length up to the current
> > maximum supported, which is 2048-bits. The properties are named, e.g.
> > sve128, sve256, sve512, ..., where the number is the number of bits.
> > 
> > It's now possible to do something like -cpu max,sve-max-vq=4,sve384=off
> > to provide a guest vector lengths 128, 256, and 512 bits. The resulting
> > set must conform to the architectural constraint of having all power-of-2
> > lengths smaller than the maximum length present. It's also possible to
> > only provide sve<vl-bits> properties, e.g. -cpu max,sve512=on. That
> > example provides the machine with 128, 256, and 512 bit vector lengths.
> > It doesn't hurt to explicitly ask for all expected vector lengths,
> > which is what, for example, libvirt should do.
> > 
> > Note1, it is not possible to use sve<vl-bits> properties before
> > sve-max-vq, e.g. -cpu max,sve384=off,sve-max-vq=4, as supporting
> > that overly complicates the user input validation.
> > 
> > Note2, while one might expect -cpu max,sve-max-vq=4,sve512=on to be the
> > same as -cpu max,sve512=on, they are not. The former enables all vector
> > lengths 512 bits and smaller, while the latter only sets the 512-bit
> > length and its smaller power-of-2 lengths. It's probably best not to use
> > sve-max-vq with sve<vl-bits> properties, but it can't be completely
> > forbidden as we want qmp_query_cpu_model_expansion to work with guests
> > launched with e.g. -cpu max,sve-max-vq=8 on their command line.
> 
> Supporting both options together in a non-idempotent way seems to
> complicate things.
> 
> Would it be simpler to allow sve-max-vq only when there are no sve<bits>
> options?

Not really. Since we can't simply remove sve-max-vq from the 'max' cpu
type, then we'd still need conditions to check when we can use it. The
restriction that it has to come first reduces the complexity
substantially, and I think restricting to not being allowed
at all, when sve<vl-bits> are used, would only give us a net check
reduction of one or two.

> 
> Alternatively, the two options would be defined so that their meanings
> are independent of parse order.
> 
> So, way sve-max-vq=<max> means that:
> 
>  * all VQs up to max for which there is no corresponding sve<bits>=off,
>    are enabled; and
> 
>  * VQ max is enabled; and
> 
>  * all VQs exceeding max are disabled.
> 
> While sve<bits>=(on|off) means
> 
>  * the VQ coresponding to <bits> is enabled (for on) or disabled (for
>    off).
>  
> After parsing all the options, you check that the sve-max-vq and
> sve<bits> optinos are consistent.  If you disallow duplicate sve-max-vq
> or sve<bits> options, then there is no possibility of ambiguity and the
> order or options doesn't matter.

I don't want to put any final checks at the end of parsing all options,
because that won't work with the QMP query.

> 
> (There may be constraints on the way qemu options parsing works that
> make this hard, though...)

I can't think of any issue with the parsing, but the QMP query only using
the property get accessors, without any finalizing, does put constraints
on what we can do.

> 
> Having sve-max-vq in 128-bit units while sve<bits> are named in terms of
> bit counts also feels a bit odd now.

sve-max-vq already exists, so it'd have to be deprecated if we don't want
it anymore. And I think sve<vl-bits> is the right naming to go with for
that. Of course using sve-max-vq is completely optional. If you don't want
it for backward compatible reasons, or as a shorthand to restrict the
lengths, then you can just use the sve<vl-bits> properties. Indeed, with
KVM, if you use the 'host' cpu type (the default for use with KVM), then
you won't even have the sve-max-vq property. As 'host' never had it, I
didn't have to keep it, nor adopt it. And of course I didn't want to
adopt it.

Thanks,
drew

Re: [Qemu-devel] [PATCH v2 07/14] target/arm/cpu64: max cpu: Introduce sve<vl-bits> properties

[Andrew Jones]

On Mon, Jun 24, 2019 at 01:49:11PM +0200, Andrew Jones wrote:
> On Mon, Jun 24, 2019 at 12:05:26PM +0100, Dave Martin wrote:
> > On Fri, Jun 21, 2019 at 05:34:15PM +0100, Andrew Jones wrote:
> > > Introduce cpu properties to give fine control over SVE vector lengths.
> > > We introduce a property for each valid length up to the current
> > > maximum supported, which is 2048-bits. The properties are named, e.g.
> > > sve128, sve256, sve512, ..., where the number is the number of bits.
> > > 
> > > It's now possible to do something like -cpu max,sve-max-vq=4,sve384=off
> > > to provide a guest vector lengths 128, 256, and 512 bits. The resulting
> > > set must conform to the architectural constraint of having all power-of-2
> > > lengths smaller than the maximum length present. It's also possible to
> > > only provide sve<vl-bits> properties, e.g. -cpu max,sve512=on. That
> > > example provides the machine with 128, 256, and 512 bit vector lengths.
> > > It doesn't hurt to explicitly ask for all expected vector lengths,
> > > which is what, for example, libvirt should do.
> > > 
> > > Note1, it is not possible to use sve<vl-bits> properties before
> > > sve-max-vq, e.g. -cpu max,sve384=off,sve-max-vq=4, as supporting
> > > that overly complicates the user input validation.
> > > 
> > > Note2, while one might expect -cpu max,sve-max-vq=4,sve512=on to be the
> > > same as -cpu max,sve512=on, they are not. The former enables all vector
> > > lengths 512 bits and smaller, while the latter only sets the 512-bit
> > > length and its smaller power-of-2 lengths. It's probably best not to use
> > > sve-max-vq with sve<vl-bits> properties, but it can't be completely
> > > forbidden as we want qmp_query_cpu_model_expansion to work with guests
> > > launched with e.g. -cpu max,sve-max-vq=8 on their command line.
> > 
> > Supporting both options together in a non-idempotent way seems to
> > complicate things.
> > 
> > Would it be simpler to allow sve-max-vq only when there are no sve<bits>
> > options?
> 
> Not really. Since we can't simply remove sve-max-vq from the 'max' cpu
> type, then we'd still need conditions to check when we can use it. The
> restriction that it has to come first reduces the complexity
> substantially, and I think restricting to not being allowed
> at all, when sve<vl-bits> are used, would only give us a net check
> reduction of one or two.
> 
> > 
> > Alternatively, the two options would be defined so that their meanings
> > are independent of parse order.
> > 
> > So, way sve-max-vq=<max> means that:
> > 
> >  * all VQs up to max for which there is no corresponding sve<bits>=off,
> >    are enabled; and
> > 
> >  * VQ max is enabled; and
> > 
> >  * all VQs exceeding max are disabled.
> > 
> > While sve<bits>=(on|off) means
> > 
> >  * the VQ coresponding to <bits> is enabled (for on) or disabled (for
> >    off).
> >  
> > After parsing all the options, you check that the sve-max-vq and
> > sve<bits> optinos are consistent.  If you disallow duplicate sve-max-vq
> > or sve<bits> options, then there is no possibility of ambiguity and the
> > order or options doesn't matter.
> 
> I don't want to put any final checks at the end of parsing all options,
> because that won't work with the QMP query.

Actually, I think I can allow sve-max-vq to come after sve<vl-bits>
without adding much code, and without requiring a final check. I
could try that if you'd like, but I'm not sure it's worth it. Also, I
feel like I tried this once already and rejected it for some reason,
but atm I can't remember why.

> 
> > 
> > (There may be constraints on the way qemu options parsing works that
> > make this hard, though...)
> 
> I can't think of any issue with the parsing, but the QMP query only using
> the property get accessors, without any finalizing, does put constraints
> on what we can do.
> 
> > 
> > Having sve-max-vq in 128-bit units while sve<bits> are named in terms of
> > bit counts also feels a bit odd now.
> 
> sve-max-vq already exists, so it'd have to be deprecated if we don't want
> it anymore. And I think sve<vl-bits> is the right naming to go with for
> that. Of course using sve-max-vq is completely optional. If you don't want
> it for backward compatible reasons, or as a shorthand to restrict the
> lengths, then you can just use the sve<vl-bits> properties. Indeed, with
> KVM, if you use the 'host' cpu type (the default for use with KVM), then
> you won't even have the sve-max-vq property. As 'host' never had it, I
> didn't have to keep it, nor adopt it. And of course I didn't want to
> adopt it.
> 
> Thanks,
> drew
> 

Re: [Qemu-devel] [PATCH v2 07/14] target/arm/cpu64: max cpu: Introduce sve<vl-bits> properties

[Dave Martin]

On Mon, Jun 24, 2019 at 01:10:41PM +0100, Andrew Jones wrote:
> On Mon, Jun 24, 2019 at 01:49:11PM +0200, Andrew Jones wrote:
> > On Mon, Jun 24, 2019 at 12:05:26PM +0100, Dave Martin wrote:
> > > On Fri, Jun 21, 2019 at 05:34:15PM +0100, Andrew Jones wrote:
> > > > Introduce cpu properties to give fine control over SVE vector lengths.
> > > > We introduce a property for each valid length up to the current
> > > > maximum supported, which is 2048-bits. The properties are named, e.g.
> > > > sve128, sve256, sve512, ..., where the number is the number of bits.
> > > > 
> > > > It's now possible to do something like -cpu max,sve-max-vq=4,sve384=off
> > > > to provide a guest vector lengths 128, 256, and 512 bits. The resulting
> > > > set must conform to the architectural constraint of having all power-of-2
> > > > lengths smaller than the maximum length present. It's also possible to
> > > > only provide sve<vl-bits> properties, e.g. -cpu max,sve512=on. That
> > > > example provides the machine with 128, 256, and 512 bit vector lengths.
> > > > It doesn't hurt to explicitly ask for all expected vector lengths,
> > > > which is what, for example, libvirt should do.
> > > > 
> > > > Note1, it is not possible to use sve<vl-bits> properties before
> > > > sve-max-vq, e.g. -cpu max,sve384=off,sve-max-vq=4, as supporting
> > > > that overly complicates the user input validation.
> > > > 
> > > > Note2, while one might expect -cpu max,sve-max-vq=4,sve512=on to be the
> > > > same as -cpu max,sve512=on, they are not. The former enables all vector
> > > > lengths 512 bits and smaller, while the latter only sets the 512-bit
> > > > length and its smaller power-of-2 lengths. It's probably best not to use
> > > > sve-max-vq with sve<vl-bits> properties, but it can't be completely
> > > > forbidden as we want qmp_query_cpu_model_expansion to work with guests
> > > > launched with e.g. -cpu max,sve-max-vq=8 on their command line.
> > > 
> > > Supporting both options together in a non-idempotent way seems to
> > > complicate things.
> > > 
> > > Would it be simpler to allow sve-max-vq only when there are no sve<bits>
> > > options?
> > 
> > Not really. Since we can't simply remove sve-max-vq from the 'max' cpu
> > type, then we'd still need conditions to check when we can use it. The
> > restriction that it has to come first reduces the complexity
> > substantially, and I think restricting to not being allowed
> > at all, when sve<vl-bits> are used, would only give us a net check
> > reduction of one or two.
> > 
> > > 
> > > Alternatively, the two options would be defined so that their meanings
> > > are independent of parse order.
> > > 
> > > So, way sve-max-vq=<max> means that:
> > > 
> > >  * all VQs up to max for which there is no corresponding sve<bits>=off,
> > >    are enabled; and
> > > 
> > >  * VQ max is enabled; and
> > > 
> > >  * all VQs exceeding max are disabled.
> > > 
> > > While sve<bits>=(on|off) means
> > > 
> > >  * the VQ coresponding to <bits> is enabled (for on) or disabled (for
> > >    off).
> > >  
> > > After parsing all the options, you check that the sve-max-vq and
> > > sve<bits> optinos are consistent.  If you disallow duplicate sve-max-vq
> > > or sve<bits> options, then there is no possibility of ambiguity and the
> > > order or options doesn't matter.
> > 
> > I don't want to put any final checks at the end of parsing all options,
> > because that won't work with the QMP query.

Ack, I'm not familiar with the QMP side of things.

> Actually, I think I can allow sve-max-vq to come after sve<vl-bits>
> without adding much code, and without requiring a final check. I
> could try that if you'd like, but I'm not sure it's worth it. Also, I
> feel like I tried this once already and rejected it for some reason,
> but atm I can't remember why.

Probably not.  I was aiming to simplify things, but if my suggestions
don't make anything simpler, then it's obviously not worth it :)

> > > (There may be constraints on the way qemu options parsing works that
> > > make this hard, though...)
> > 
> > I can't think of any issue with the parsing, but the QMP query only using
> > the property get accessors, without any finalizing, does put constraints
> > on what we can do.
> > 
> > > Having sve-max-vq in 128-bit units while sve<bits> are named in terms of
> > > bit counts also feels a bit odd now.
> > 
> > sve-max-vq already exists, so it'd have to be deprecated if we don't want
> > it anymore. And I think sve<vl-bits> is the right naming to go with for
> > that. Of course using sve-max-vq is completely optional. If you don't want
> > it for backward compatible reasons, or as a shorthand to restrict the
> > lengths, then you can just use the sve<vl-bits> properties. Indeed, with
> > KVM, if you use the 'host' cpu type (the default for use with KVM), then
> > you won't even have the sve-max-vq property. As 'host' never had it, I
> > didn't have to keep it, nor adopt it. And of course I didn't want to
> > adopt it.

I hadn't realised the sve-max-vq option was already merged.  I agree
that if there are established semantics for this, we shouldn't mess with
it now.

So, keep it as you proposed, I guess.

Cheers
---Dave

Re: [Qemu-devel] [PATCH v2 07/14] target/arm/cpu64: max cpu: Introduce sve<vl-bits> properties

[Auger Eric]

Hi Drew,

On 6/21/19 6:34 PM, Andrew Jones wrote:
> Introduce cpu properties to give fine control over SVE vector lengths.
> We introduce a property for each valid length up to the current
> maximum supported, which is 2048-bits. The properties are named, e.g.
> sve128, sve256, sve512, ..., where the number is the number of bits.
sve384 then in the natural order, otherwise it gives the impression you
can only specify * 128bit pow of 2 sizes at this stage of the reading.

> 
> It's now possible to do something like -cpu max,sve-max-vq=4,sve384=off
Document the fact sve512 cannot be turned to off, which sounds fully
sensible (by reading the code). By the way, I think an actual
documentation should be provided in qemu. Maybe as spec to agree on.
> to provide a guest vector lengths 128, 256, and 512 bits. The resulting
> set must conform to the architectural constraint of having all power-of-2
> lengths smaller than the maximum length present. It's also possible to
> only provide sve<vl-bits> properties, e.g. -cpu max,sve512=on> That example provides the machine with 128, 256, and 512 bit vector
lengths.
> It doesn't hurt to explicitly ask for all expected vector lengths,
> which is what, for example, libvirt should do.>
> Note1, it is not possible to use sve<vl-bits> properties before
> sve-max-vq, e.g. -cpu max,sve384=off,sve-max-vq=4, as supporting
> that overly complicates the user input validation.
> 
> Note2, while one might expect -cpu max,sve-max-vq=4,sve512=on to be the
> same as -cpu max,sve512=on, they are not.
yep it is a bit weird

Didn't you consider -cpu max,sve-max-vq=4,req_only=true removing non
power of 2 values and sve<vl-bits> setting a single VLbit?
> The former enables all vector lengths 512 bits and smaller
( required and permitted)
> while the latter only sets the 512-bit
> length and its smaller power-of-2 lengths. It's probably best not to use
> sve-max-vq with sve<vl-bits> properties, but it can't be completely
> forbidden as we want qmp_query_cpu_model_expansion to work with guests
> launched with e.g. -cpu max,sve-max-vq=8 on their command line.
what does happen if you specify -cpu max,sve384=on? (seems to be allowed
in the code?)

> 
> Signed-off-by: Andrew Jones <drjones@redhat.com>
> ---
>  target/arm/cpu.c         |   6 +
>  target/arm/cpu.h         |  14 ++
>  target/arm/cpu64.c       | 360 ++++++++++++++++++++++++++++++++++++++-
>  target/arm/helper.c      |  11 +-
>  target/arm/monitor.c     |  16 ++
>  tests/arm-cpu-features.c | 217 +++++++++++++++++++++++
>  6 files changed, 620 insertions(+), 4 deletions(-)
> 
> diff --git a/target/arm/cpu.c b/target/arm/cpu.c
> index f08e178fc84b..e060a0d9df0e 100644
> --- a/target/arm/cpu.c
> +++ b/target/arm/cpu.c
> @@ -1019,6 +1019,12 @@ static void arm_cpu_realizefn(DeviceState *dev, Error **errp)
>          return;
>      }
>  
> +    arm_cpu_sve_finalize(cpu, &local_err);
> +    if (local_err) {
> +        error_propagate(errp, local_err);
> +        return;
> +    }
> +
>      if (arm_feature(env, ARM_FEATURE_AARCH64) &&
>          cpu->has_vfp != cpu->has_neon) {
>          /*
> diff --git a/target/arm/cpu.h b/target/arm/cpu.h
> index f9da672be575..cbb155cf72a5 100644
> --- a/target/arm/cpu.h
> +++ b/target/arm/cpu.h
> @@ -184,8 +184,13 @@ typedef struct {
>  
>  #ifdef TARGET_AARCH64
>  # define ARM_MAX_VQ    16
> +void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp);
> +uint32_t arm_cpu_vq_map_next_smaller(ARMCPU *cpu, uint32_t vq);
>  #else
>  # define ARM_MAX_VQ    1
> +static inline void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp) { }
> +static inline uint32_t arm_cpu_vq_map_next_smaller(ARMCPU *cpu, uint32_t vq)
> +{ return 0; }
>  #endif
>  
>  typedef struct ARMVectorReg {
> @@ -915,6 +920,15 @@ struct ARMCPU {
>  
>      /* Used to set the maximum vector length the cpu will support.  */
>      uint32_t sve_max_vq;
> +
> +    /*
> +     * In sve_vq_map each set bit is a supported vector length of
> +     * (bit-number + 1) * 16 bytes, i.e. each bit number + 1 is the vector> +     * length in quadwords. We need a map size twice the maximum
> +     * quadword length though because we use two bits for each vector
> +     * length in order to track three states: uninitialized, off, and on.
> +     */
> +    DECLARE_BITMAP(sve_vq_map, ARM_MAX_VQ * 2);
>  };
>  
>  void arm_cpu_post_init(Object *obj);
> diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
> index 02ada65f240c..5def82111dee 100644
> --- a/target/arm/cpu64.c
> +++ b/target/arm/cpu64.c
> @@ -257,6 +257,149 @@ static void aarch64_a72_initfn(Object *obj)
>      define_arm_cp_regs(cpu, cortex_a72_a57_a53_cp_reginfo);
>  }
>  
> +/*
> + * While we eventually use cpu->sve_vq_map as a typical bitmap, where each vq
> + * has only two states (off/on), until we've finalized the map at realize time
> + * we use an extra bit, at the vq - 1 + ARM_MAX_VQ bit number, to also allow
> + * tracking of the uninitialized state. The arm_vq_state typedef and following
> + * functions allow us to more easily work with the bitmap. Also, while the map
> + * is still initializing, sve-max-vq has an additional three states, bringing
> + * the number of its states to five, which are the following:
> + *
> + * sve-max-vq:
> + *   0:    SVE is disabled. The default value for a vq in the map is 'OFF'.
> + *  -1:    SVE is enabled, but neither sve-max-vq nor sve<vl-bits> properties
> + *         have yet been specified by the user. The default value for a vq in
> + *         the map is 'ON'.
> + *  -2:    SVE is enabled and one or more sve<vl-bits> properties have been
> + *         set to 'OFF' by the user, but no sve<vl-bits> properties have yet
> + *         been set to 'ON'. The user is now blocked from setting sve-max-vq
> + *         and the default value for a vq in the map is 'ON'.
> + *  -3:    SVE is enabled and one or more sve<vl-bits> properties have been
> + *         set to 'ON' by the user. The user is blocked from setting sve-max-vq
> + *         and the default value for a vq in the map is 'OFF'. sve-max-vq never
> + *         transitions back to -2, even if later inputs disable the vector
> + *         lengths that initially transitioned sve-max-vq to this state. This
> + *         avoids the default values from flip-flopping.
> + *  [1-ARM_MAX_VQ]: SVE is enabled and the user has specified a valid
> + *                  sve-max-vq. The sve-max-vq specified vq and all smaller
> + *                  vq's will be initially enabled. All larger vq's will have
> + *                  a default of 'OFF'.
> + */
> +#define ARM_SVE_INIT          -1
> +#define ARM_VQ_DEFAULT_ON     -2
> +#define ARM_VQ_DEFAULT_OFF    -3
> +
> +#define arm_sve_have_max_vq(cpu) ((int32_t)(cpu)->sve_max_vq > 0)
> +
> +typedef enum arm_vq_state {
> +    ARM_VQ_OFF,
> +    ARM_VQ_ON,
> +    ARM_VQ_UNINITIALIZED,
> +} arm_vq_state;
> +
> +static arm_vq_state arm_cpu_vq_map_get(ARMCPU *cpu, int vq)> +{
> +    assert(vq <= ARM_MAX_VQ);
> +
> +    return test_bit(vq - 1, cpu->sve_vq_map) |
> +           test_bit(vq - 1 + ARM_MAX_VQ, cpu->sve_vq_map) << 1;
> +}> +
> +static void arm_cpu_vq_map_set(ARMCPU *cpu, int vq, arm_vq_state state)
> +{
> +    assert(state == ARM_VQ_OFF || state == ARM_VQ_ON);
> +    assert(vq <= ARM_MAX_VQ);
> +
> +    clear_bit(vq - 1 + ARM_MAX_VQ, cpu->sve_vq_map);
> +
> +    if (state == ARM_VQ_ON) {
> +        set_bit(vq - 1, cpu->sve_vq_map);
> +    } else {
> +        clear_bit(vq - 1, cpu->sve_vq_map);
> +    }
> +}
> +
> +static void arm_cpu_vq_map_init(ARMCPU *cpu)
> +{
> +    bitmap_zero(cpu->sve_vq_map, ARM_MAX_VQ * 2);
nit: bitmap_clear(map, 0, ARM_MAX_VQ);
/* all VLs OFF */
> +    bitmap_set(cpu->sve_vq_map, ARM_MAX_VQ, ARM_MAX_VQ);
/* all VLs uninitialized */
> +}
> +
> +static bool arm_cpu_vq_map_is_finalized(ARMCPU *cpu)
> +{
> +    DECLARE_BITMAP(map, ARM_MAX_VQ * 2);
> +
> +    bitmap_zero(map, ARM_MAX_VQ * 2);
same
> +    bitmap_set(map, ARM_MAX_VQ, ARM_MAX_VQ);
> +    bitmap_and(map, map, cpu->sve_vq_map, ARM_MAX_VQ * 2);
> +
> +    return bitmap_empty(map, ARM_MAX_VQ * 2);
> +}
> +
> +static void arm_cpu_vq_map_finalize(ARMCPU *cpu)
> +{
> +    Error *err = NULL;
> +    char name[8];
> +    uint32_t vq;
> +    bool value;
> +
> +    /*
> +     * We use the property get accessor because it knows what default
> +     * values to return for uninitialized vector lengths.
> +     */
> +    for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
> +        sprintf(name, "sve%d", vq * 128);
> +        value = object_property_get_bool(OBJECT(cpu), name, &err);
> +        assert(!err);
> +        if (value) {
> +            arm_cpu_vq_map_set(cpu, vq, ARM_VQ_ON);
> +        } else {
> +            arm_cpu_vq_map_set(cpu, vq, ARM_VQ_OFF);
> +        }
> +    }
> +
> +    assert(arm_cpu_vq_map_is_finalized(cpu));
this can be removed
> +}
> +
> +void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp)
> +{
> +    Error *err = NULL;
> +
> +    if (!cpu->sve_max_vq) {
> +        bitmap_zero(cpu->sve_vq_map, ARM_MAX_VQ * 2);
> +        return;
> +    }
> +
> +    if (cpu->sve_max_vq == ARM_SVE_INIT) {
> +        object_property_set_uint(OBJECT(cpu), ARM_MAX_VQ, "sve-max-vq", &err);
> +        if (err) {
> +            error_propagate(errp, err);
> +            return;
> +        }
> +        assert(cpu->sve_max_vq == ARM_MAX_VQ);
I guess those asserts can be removed now?
> +        arm_cpu_vq_map_finalize(cpu);
move the arm_cpu_vq_map_finalize out of the if, at the end.
> +    } else {
> +        arm_cpu_vq_map_finalize(cpu);
> +        if (!arm_sve_have_max_vq(cpu)) {
> +            cpu->sve_max_vq = arm_cpu_vq_map_next_smaller(cpu, ARM_MAX_VQ + 1);
> +        }
> +    }
> +
> +    assert(cpu->sve_max_vq == arm_cpu_vq_map_next_smaller(cpu, ARM_MAX_VQ));
same here
> +}
> +
> +uint32_t arm_cpu_vq_map_next_smaller(ARMCPU *cpu, uint32_t vq)
> +{
> +    uint32_t bitnum;
> +
> +    assert(vq <= ARM_MAX_VQ + 1);
> +    assert(arm_cpu_vq_map_is_finalized(cpu));
> +
> +    bitnum = find_last_bit(cpu->sve_vq_map, vq - 1);
why do you pass ARM_MAX_VQ + 1 and then do vq -1? doesn't
find_last_bit() take the size which is ARM_MAX_VQ in this case?
> +    return bitnum == vq - 1 ? 0 : bitnum + 1;
> +}
> +
>  static void cpu_max_get_sve_max_vq(Object *obj, Visitor *v, const char *name,
>                                     void *opaque, Error **errp)
>  {
> @@ -283,12 +426,203 @@ static void cpu_max_set_sve_max_vq(Object *obj, Visitor *v, const char *name,
>          return;
>      }
>  
> +    /*
> +     * It gets complicated trying to support both sve-max-vq and
> +     * sve<vl-bits> properties together, so we mostly don't. We
> +     * do allow both if sve-max-vq is specified first and only once
> +     * though.
> +     */
> +    if (cpu->sve_max_vq != ARM_SVE_INIT) {
> +        error_setg(errp, "sve<vl-bits> in use or sve-max-vq already "
> +                   "specified");
> +        error_append_hint(errp, "sve-max-vq must come before all "
> +                          "sve<vl-bits> properties and it must only "
> +                          "be specified once.\n");
> +        return;
> +    }
> +
>      cpu->sve_max_vq = value;
>  
>      if (cpu->sve_max_vq == 0 || cpu->sve_max_vq > ARM_MAX_VQ) {
>          error_setg(errp, "unsupported SVE vector length");
>          error_append_hint(errp, "Valid sve-max-vq in range [1-%d]\n",
>                            ARM_MAX_VQ);
> +    } else {
> +        uint32_t vq;
> +
> +        for (vq = 1; vq <= cpu->sve_max_vq; ++vq) {
> +            char name[8];
> +            sprintf(name, "sve%d", vq * 128);
> +            object_property_set_bool(obj, true, name, &err);
> +            if (err) {
> +                error_propagate(errp, err);
> +                return;
> +            }
> +        }
> +    }
> +}
> +
> +static void cpu_arm_get_sve_vq(Object *obj, Visitor *v, const char *name,
> +                               void *opaque, Error **errp)
> +{
> +    ARMCPU *cpu = ARM_CPU(obj);
> +    int vq = atoi(&name[3]) / 128;
> +    arm_vq_state vq_state;
> +    bool value;
> +
> +    vq_state = arm_cpu_vq_map_get(cpu, vq);
> +
> +    if (!cpu->sve_max_vq) {
> +        /* All vector lengths are disabled when SVE is off. */
> +        value = false;
> +    } else if (vq_state == ARM_VQ_ON) {
> +        value = true;
> +    } else if (vq_state == ARM_VQ_OFF) {
> +        value = false;
> +    } else {
> +        /*
> +         * vq is uninitialized. We pick a default here based on the
> +         * the state of sve-max-vq and other sve<vl-bits> properties.
> +         */
> +        if (arm_sve_have_max_vq(cpu)) {
> +            /*
> +             * If we have sve-max-vq, then all remaining uninitialized
> +             * vq's are 'OFF'.
> +             */
> +            value = false;
> +        } else {
> +            switch (cpu->sve_max_vq) {
> +            case ARM_SVE_INIT:
> +            case ARM_VQ_DEFAULT_ON:
> +                value = true;
> +                break;
> +            case ARM_VQ_DEFAULT_OFF:
> +                value = false;
> +                break;
> +            }
> +        }
> +    }
> +
> +    visit_type_bool(v, name, &value, errp);
> +}
> +
> +static void cpu_arm_set_sve_vq(Object *obj, Visitor *v, const char *name,
> +                               void *opaque, Error **errp)
> +{
> +    ARMCPU *cpu = ARM_CPU(obj);
> +    int vq = atoi(&name[3]) / 128;
> +    arm_vq_state vq_state;
> +    Error *err = NULL;
> +    uint32_t max_vq = 0;
> +    bool value;
> +
> +    visit_type_bool(v, name, &value, &err);
> +    if (err) {
> +        error_propagate(errp, err);
> +        return;
> +    }
> +
> +    if (value && !cpu->sve_max_vq) {
> +        error_setg(errp, "cannot enable %s", name);
> +        error_append_hint(errp, "SVE has been disabled with sve=off\n");
> +        return;
> +    } else if (!cpu->sve_max_vq) {
> +        /*
> +         * We don't complain about disabling vector lengths when SVE
> +         * is off, but we don't do anything either.
> +         */
> +        return;
> +    }
> +
> +    if (arm_sve_have_max_vq(cpu)) {
> +        max_vq = cpu->sve_max_vq;
> +    } else {
> +        if (value) {
> +            cpu->sve_max_vq = ARM_VQ_DEFAULT_OFF;
> +        } else if (cpu->sve_max_vq != ARM_VQ_DEFAULT_OFF) {
> +            cpu->sve_max_vq = ARM_VQ_DEFAULT_ON;
> +        }
> +    }
> +
> +    /*
> +     * We need to know the maximum vector length, which may just currently
> +     * be the maximum length, in order to validate the enabling/disabling
> +     * of this vector length. We use the property get accessor in order to
> +     * get the appropriate default value for any uninitialized lengths.
> +     */
> +    if (!max_vq) {
> +        char tmp[8];
> +        bool s;
> +
> +        for (max_vq = ARM_MAX_VQ; max_vq >= 1; --max_vq) {
> +            sprintf(tmp, "sve%d", max_vq * 128);
> +            s = object_property_get_bool(OBJECT(cpu), tmp, &err);
> +            assert(!err);
> +            if (s) {
> +                break;
> +            }
> +        }
> +    }
> +
> +    if (arm_sve_have_max_vq(cpu) && value && vq > cpu->sve_max_vq) {
> +        error_setg(errp, "cannot enable %s", name);
> +        error_append_hint(errp, "vq=%d (%d bits) is larger than the "
> +                          "maximum vector length, sve-max-vq=%d "
> +                          "(%d bits)\n", vq, vq * 128,
> +                          cpu->sve_max_vq, cpu->sve_max_vq * 128);
> +    } else if (arm_sve_have_max_vq(cpu) && !value && vq == cpu->sve_max_vq) {
> +        error_setg(errp, "cannot disable %s", name);
> +        error_append_hint(errp, "The maximum vector length must be "
> +                          "enabled, sve-max-vq=%d (%d bits)\n",
> +                          cpu->sve_max_vq, cpu->sve_max_vq * 128);
> +    } else if (arm_sve_have_max_vq(cpu) && !value && vq < cpu->sve_max_vq &&
> +               is_power_of_2(vq)) {
> +        error_setg(errp, "cannot disable %s", name);
> +        error_append_hint(errp, "vq=%d (%d bits) is required as it is a "
> +                          "power-of-2 length smaller than the maximum, "
> +                          "sve-max-vq=%d (%d bits)\n", vq, vq * 128,
> +                          cpu->sve_max_vq, cpu->sve_max_vq * 128);
> +    } else if (!value && vq < max_vq && is_power_of_2(vq)) {
> +        error_setg(errp, "cannot disable %s", name);
> +        error_append_hint(errp, "Vector length %d-bits is required as it "
> +                          "is a power-of-2 length smaller than another "
> +                          "enabled vector length. Disable all larger vector "
> +                          "lengths first.\n", vq * 128);
> +    } else {
adding return in each if/elsif would allow to avoid this indent.
> +        if (value) {
> +            bool fail = false;
> +            uint32_t s;
> +
> +            /*
> +             * Enabling a vector length automatically enables all
> +             * uninitialized power-of-2 lengths smaller than it, as
> +             * per the architecture.
> +             */
Test we are not attempting to enable a !is_power_of_2
> +            for (s = 1; s < vq; ++s) {
> +                if (is_power_of_2(s)) {
> +                    vq_state = arm_cpu_vq_map_get(cpu, s);
> +                    if (vq_state == ARM_VQ_UNINITIALIZED) {
> +                        arm_cpu_vq_map_set(cpu, s, ARM_VQ_ON);
> +                    } else if (vq_state == ARM_VQ_OFF) {
> +                        fail = true;
> +                        break;
> +                    }
> +                }
> +            }
> +
> +            if (fail) {
> +                error_setg(errp, "cannot enable %s", name);
> +                error_append_hint(errp, "Vector length %d-bits is disabled "
> +                                  "and is a power-of-2 length smaller than "
> +                                  "%s. All power-of-2 vector lengths smaller "
> +                                  "than the maximum length are required.\n",
> +                                  s * 128, name);
> +            } else {
> +                arm_cpu_vq_map_set(cpu, vq, ARM_VQ_ON);
> +            }
> +        } else {
> +            arm_cpu_vq_map_set(cpu, vq, ARM_VQ_OFF);
> +        }
>      }
>  }
>  
> @@ -318,10 +652,11 @@ static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
>          /*
>           * We handle the -cpu <cpu>,sve=off,sve=on case by reinitializing,
>           * but otherwise we don't do anything as an sve=on could come after
> -         * a sve-max-vq setting.
> +         * a sve-max-vq or sve<vl-bits> setting.
>           */
>          if (!cpu->sve_max_vq) {
> -            cpu->sve_max_vq = ARM_MAX_VQ;
> +            cpu->sve_max_vq = ARM_SVE_INIT;
> +            arm_cpu_vq_map_init(cpu);
>          }
>      } else {
>          cpu->sve_max_vq = 0;
> @@ -336,6 +671,7 @@ static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
>  static void aarch64_max_initfn(Object *obj)
>  {
>      ARMCPU *cpu = ARM_CPU(obj);
> +    uint32_t vq;
>  
>      if (kvm_enabled()) {
>          kvm_arm_set_cpu_features_from_host(cpu);
> @@ -420,11 +756,29 @@ static void aarch64_max_initfn(Object *obj)
>          cpu->dcz_blocksize = 7; /*  512 bytes */
>  #endif
>  
> -        cpu->sve_max_vq = ARM_MAX_VQ;
> +        /*
> +         * sve_max_vq is initially unspecified, but must be initialized to a
> +         * non-zero value (ARM_SVE_INIT) to indicate that this cpu type has
> +         * SVE. It will be finalized in arm_cpu_realizefn().
> +         */
> +        cpu->sve_max_vq = ARM_SVE_INIT;
>          object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_max_vq,
>                              cpu_max_set_sve_max_vq, NULL, NULL, &error_fatal);
>          object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
>                              cpu_arm_set_sve, NULL, NULL, &error_fatal);
> +
> +        /*
> +         * sve_vq_map uses a special state while setting properties, so
> +         * we initialize it here with its init function and finalize it
> +         * in arm_cpu_realizefn().
> +         */
> +        arm_cpu_vq_map_init(cpu);
> +        for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
> +            char name[8];
> +            sprintf(name, "sve%d", vq * 128);
> +            object_property_add(obj, name, "bool", cpu_arm_get_sve_vq,
> +                                cpu_arm_set_sve_vq, NULL, NULL, &error_fatal);
> +        }
>      }
>  }
>  
> diff --git a/target/arm/helper.c b/target/arm/helper.c
> index f500ccb6d31b..b7b719dba57f 100644
> --- a/target/arm/helper.c
> +++ b/target/arm/helper.c
> @@ -5324,7 +5324,16 @@ static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
>  
>      /* Bits other than [3:0] are RAZ/WI.  */
>      QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
> -    raw_write(env, ri, value & 0xf);
> +    value &= 0xf;
> +
> +    if (value) {> +        /* get next vq that is smaller than or equal to value's vq */
> +        uint32_t vq = value + 1;
ditto
> +        vq = arm_cpu_vq_map_next_smaller(cpu, vq + 1);
> +        value = vq - 1;
> +    }
> +
> +    raw_write(env, ri, value);
>  
>      /*
>       * Because we arrived here, we know both FP and SVE are enabled;
> diff --git a/target/arm/monitor.c b/target/arm/monitor.c
> index 157c487a1551..1e213906fd8f 100644
> --- a/target/arm/monitor.c
> +++ b/target/arm/monitor.c
> @@ -89,8 +89,24 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)
>      return head;
>  }
>  
> +QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
> +
> +/*
> + * These are cpu model features we want to advertise. The order here
> + * matters as this is the order in which qmp_query_cpu_model_expansion
> + * will attempt to set them. If there are dependencies between features,
> + * as there are with the sve<vl-bits> features, then the order that
> + * considers those dependencies must be used.
> + *
> + * The sve<vl-bits> features need to be in reverse order in order to
> + * enable/disable the largest vector lengths first, ensuring all
> + * power-of-2 vector lengths smaller can also be enabled/disabled.
> + */
>  static const char *cpu_model_advertised_features[] = {
>      "aarch64", "pmu", "sve",
> +    "sve2048", "sve1920", "sve1792", "sve1664", "sve1536", "sve1408",
> +    "sve1280", "sve1152", "sve1024", "sve896", "sve768", "sve640",
> +    "sve512", "sve384", "sve256", "sve128",
>      NULL
>  };
>  
> diff --git a/tests/arm-cpu-features.c b/tests/arm-cpu-features.c
Please move all the tests in a separate patch.
Each day has enough trouble of its own ;-) sweat...

Thanks

Eric
> index 509e458e9c2f..a4bf6aec00df 100644
> --- a/tests/arm-cpu-features.c
> +++ b/tests/arm-cpu-features.c
> @@ -13,6 +13,18 @@
>  #include "qapi/qmp/qdict.h"
>  #include "qapi/qmp/qjson.h"
>  
> +#if __SIZEOF_LONG__ == 8
> +#define BIT(n) (1UL << (n))
> +#else
> +#define BIT(n) (1ULL << (n))
> +#endif
> +
> +/*
> + * We expect the SVE max-vq to be 16. Also it must be <= 64
> + * for our test code, otherwise 'vls' can't just be a uint64_t.
> + */
> +#define SVE_MAX_VQ 16
> +
>  #define MACHINE    "-machine virt,gic-version=max "
>  #define QUERY_HEAD "{ 'execute': 'query-cpu-model-expansion', " \
>                       "'arguments': { 'type': 'full', "
> @@ -137,6 +149,201 @@ static void assert_bad_props(QTestState *qts, const char *cpu_type)
>      qobject_unref(resp);
>  }
>  
> +static void resp_get_sve_vls(QDict *resp, uint64_t *vls, uint32_t *max_vq)
> +{
> +    const QDictEntry *e;
> +    QDict *qdict;
> +    int n = 0;
> +
> +    *vls = 0;
> +
> +    qdict = resp_get_props(resp);
> +
> +    for (e = qdict_first(qdict); e; e = qdict_next(qdict, e)) {
> +        if (strlen(e->key) > 3 && !strncmp(e->key, "sve", 3) &&
> +            g_ascii_isdigit(e->key[3])) {
> +            char *endptr;
> +            int bits;
> +
> +            bits = g_ascii_strtoll(&e->key[3], &endptr, 10);
> +            if (!bits || *endptr != '\0') {
> +                continue;
> +            }
> +
> +            if (qdict_get_bool(qdict, e->key)) {
> +                *vls |= BIT((bits / 128) - 1);
> +            }
> +            ++n;
> +        }
> +    }
> +
> +    g_assert(n == SVE_MAX_VQ);
> +
> +    *max_vq = !*vls ? 0 : 64 - __builtin_clzll(*vls);
> +}
> +
> +static uint64_t sve_get_vls(QTestState *qts, const char *cpu_type,
> +                            const char *fmt, ...)
> +{
> +    QDict *resp;
> +    uint64_t vls;
> +    uint32_t max_vq;
> +
> +    if (fmt) {
> +        QDict *args;
> +        va_list ap;
> +
> +        va_start(ap, fmt);
> +        args = qdict_from_vjsonf_nofail(fmt, ap);
> +        va_end(ap);
> +
> +        resp = qtest_qmp(qts, QUERY_HEAD "'model': { 'name': %s, "
> +                                                    "'props': %p }"
> +                              QUERY_TAIL, cpu_type, args);
> +    } else {
> +        resp = do_query_no_props(qts, cpu_type);
> +    }
> +
> +    g_assert(resp);
> +    resp_get_sve_vls(resp, &vls, &max_vq);
> +    qobject_unref(resp);
> +
> +    return vls;
> +}
> +
> +#define assert_sve_vls(qts, cpu_type, expected_vls, fmt, ...) \
> +    g_assert(sve_get_vls(qts, cpu_type, fmt, ##__VA_ARGS__) == expected_vls)
> +
> +static void sve_tests_default(QTestState *qts, const char *cpu_type)
> +{
> +    /*
> +     * With no sve-max-vq or sve<vl-bits> properties on the command line
> +     * the default is to have all vector lengths enabled.
> +     */
> +    assert_sve_vls(qts, cpu_type, BIT(SVE_MAX_VQ) - 1, NULL);
> +
> +    /*
> +     * -------------------------------------------------------------------
> +     *               power-of-2(vq)   all-power-            can      can
> +     *                                of-2(< vq)          enable   disable
> +     * -------------------------------------------------------------------
> +     * vq < max_vq      no            MUST*                yes      yes
> +     * vq < max_vq      yes           MUST*                yes      no
> +     * -------------------------------------------------------------------
> +     * vq == max_vq     n/a           MUST*                yes**    yes**
> +     * -------------------------------------------------------------------
> +     * vq > max_vq      n/a           no                   no       yes
> +     * vq > max_vq      n/a           yes                  yes      yes
> +     * -------------------------------------------------------------------
> +     *
> +     * [*] "MUST" means this requirement must already be satisfied,
> +     *     otherwise 'max_vq' couldn't itself be enabled.
> +     *
> +     * [**] Not testable with the QMP interface, only with the command line.
> +     */
> +
> +    /* max_vq := 8 */
> +    assert_sve_vls(qts, cpu_type, 0x8b, "{ 'sve1024': true }");
> +
> +    /* max_vq := 8, vq < max_vq, !power-of-2(vq) */
> +    assert_sve_vls(qts, cpu_type, 0x8f,
> +                   "{ 'sve1024': true, 'sve384': true }");
> +    assert_sve_vls(qts, cpu_type, 0x8b,
> +                   "{ 'sve1024': true, 'sve384': false }");
> +
> +    /* max_vq := 8, vq < max_vq, power-of-2(vq) */
> +    assert_sve_vls(qts, cpu_type, 0x8b,
> +                   "{ 'sve1024': true, 'sve256': true }");
> +    assert_error(qts, cpu_type, "cannot disable sve256",
> +                 "{ 'sve1024': true, 'sve256': false }");
> +
> +    /*
> +     * max_vq := 3, vq > max_vq, !all-power-of-2(< vq)
> +     *
> +     * If given sve384=on,sve512=off,sve640=on the command line error would be
> +     * "cannot enable sve640", but QMP visits the vector lengths in reverse
> +     * order, so we get "cannot disable sve512" instead. The command line would
> +     * also give that error if given sve384=on,sve640=on,sve512=off, so this is
> +     * all fine. The important thing is that we get an error.
> +     */
> +    assert_error(qts, cpu_type, "cannot disable sve512",
> +                 "{ 'sve384': true, 'sve512': false, 'sve640': true }");
> +
> +    /*
> +     * We can disable power-of-2 vector lengths when all larger lengths
> +     * are also disabled. The shorter, sve384=on,sve512=off,sve640=off
> +     * works on the command line, but QMP doesn't know that all the
> +     * vector lengths larger than 384-bits will be disabled until it
> +     * sees the enabling of sve384, which comes near the end since it
> +     * visits the lengths in reverse order. So we just have to explicitly
> +     * disable them all.
> +     */
> +    assert_sve_vls(qts, cpu_type, 0x7,
> +                   "{ 'sve384': true, 'sve512': false, 'sve640': false, "
> +                   "  'sve768': false, 'sve896': false, 'sve1024': false, "
> +                   "  'sve1152': false, 'sve1280': false, 'sve1408': false, "
> +                   "  'sve1536': false, 'sve1664': false, 'sve1792': false, "
> +                   "  'sve1920': false, 'sve2048': false }");
> +
> +    /* max_vq := 3, vq > max_vq, all-power-of-2(< vq) */
> +    assert_sve_vls(qts, cpu_type, 0x1f,
> +                   "{ 'sve384': true, 'sve512': true, 'sve640': true }");
> +    assert_sve_vls(qts, cpu_type, 0xf,
> +                   "{ 'sve384': true, 'sve512': true, 'sve640': false }");
> +}
> +
> +static void sve_tests_sve_max_vq_8(const void *data)
> +{
> +    QTestState *qts;
> +
> +    qts = qtest_init(MACHINE "-cpu max,sve-max-vq=8");
> +
> +    assert_sve_vls(qts, "max", BIT(8) - 1, NULL);
> +
> +    /*
> +     * Disabling the max-vq set by sve-max-vq is not allowed, but
> +     * of course enabling it is OK.
> +     */
> +    assert_error(qts, "max", "cannot disable sve1024", "{ 'sve1024': false }");
> +    assert_sve_vls(qts, "max", 0xff, "{ 'sve1024': true }");
> +
> +    /*
> +     * Enabling anything larger than max-vq set by sve-max-vq is not
> +     * allowed, but of course disabling everything larger is OK.
> +     */
> +    assert_error(qts, "max", "cannot enable sve1152", "{ 'sve1152': true }");
> +    assert_sve_vls(qts, "max", 0xff, "{ 'sve1152': false }");
> +
> +    /*
> +     * We can disable non power-of-2 lengths smaller than the max-vq
> +     * set by sve-max-vq, but not power-of-2 lengths.
> +     */
> +    assert_sve_vls(qts, "max", 0xfb, "{ 'sve384': false }");
> +    assert_error(qts, "max", "cannot disable sve256", "{ 'sve256': false }");
> +
> +    qtest_quit(qts);
> +}
> +
> +static void sve_tests_sve_off(const void *data)
> +{
> +    QTestState *qts;
> +
> +    qts = qtest_init(MACHINE "-cpu max,sve=off");
> +
> +    /*
> +     * SVE is off, so the map should be empty.
> +     */
> +    assert_sve_vls(qts, "max", 0, NULL);
> +
> +    /*
> +     * We can't turn anything on, but off is OK.
> +     */
> +    assert_error(qts, "max", "cannot enable sve128", "{ 'sve128': true }");
> +    assert_sve_vls(qts, "max", 0, "{ 'sve128': false }");
> +
> +    qtest_quit(qts);
> +}
> +
>  static void test_query_cpu_model_expansion(const void *data)
>  {
>      QTestState *qts;
> @@ -159,9 +366,12 @@ static void test_query_cpu_model_expansion(const void *data)
>      if (g_str_equal(qtest_get_arch(), "aarch64")) {
>          assert_has_feature(qts, "max", "aarch64");
>          assert_has_feature(qts, "max", "sve");
> +        assert_has_feature(qts, "max", "sve128");
>          assert_has_feature(qts, "cortex-a57", "pmu");
>          assert_has_feature(qts, "cortex-a57", "aarch64");
>  
> +        sve_tests_default(qts, "max");
> +
>          /* Test that features that depend on KVM generate errors without. */
>          assert_error(qts, "max",
>                       "'aarch64' feature cannot be disabled "
> @@ -213,6 +423,13 @@ int main(int argc, char **argv)
>      qtest_add_data_func("/arm/query-cpu-model-expansion",
>                          NULL, test_query_cpu_model_expansion);
>  
> +    if (g_str_equal(qtest_get_arch(), "aarch64")) {
> +        qtest_add_data_func("/arm/max/query-cpu-model-expansion/sve-max-vq-8",
> +                            NULL, sve_tests_sve_max_vq_8);
> +        qtest_add_data_func("/arm/max/query-cpu-model-expansion/sve-off",
> +                            NULL, sve_tests_sve_off);
> +    }
> +
>      if (kvm_available) {
>          qtest_add_data_func("/arm/kvm/query-cpu-model-expansion",
>                              NULL, test_query_cpu_model_expansion_kvm);
> 

Re: [Qemu-devel] [PATCH v2 07/14] target/arm/cpu64: max cpu: Introduce sve<vl-bits> properties

[Andrew Jones]

On Wed, Jun 26, 2019 at 04:58:05PM +0200, Auger Eric wrote:
> Hi Drew,
> 
> On 6/21/19 6:34 PM, Andrew Jones wrote:
> > Introduce cpu properties to give fine control over SVE vector lengths.
> > We introduce a property for each valid length up to the current
> > maximum supported, which is 2048-bits. The properties are named, e.g.
> > sve128, sve256, sve512, ..., where the number is the number of bits.
> sve384 then in the natural order, otherwise it gives the impression you
> can only specify * 128bit pow of 2 sizes at this stage of the reading.

OK

> 
> > 
> > It's now possible to do something like -cpu max,sve-max-vq=4,sve384=off
> Document the fact sve512 cannot be turned to off, which sounds fully
> sensible (by reading the code). By the way, I think an actual
> documentation should be provided in qemu. Maybe as spec to agree on.

Actually, maybe I could just allow it to be disabled. It would be
a strange command line to do '-cpu max,sve-max-vq=4,sve512=off', but if
that's what the user wants, then there's not really any good reason to
block it. As I say below, mixing these types of properties on the command
line isn't really a good idea, but it's not completely blocked because we
want a user that prefers launching their (most likely TCG) guest with,
e.g.  '-cpu max,sve-max-vq=4', to also have a functional QMP CPU model
expansion, but the CPU model expansion for SVE vector lengths depends on
the sve<vl-bits> properties, thus mixing sve-max-vq with sve<vl-bits>,
where sve-max-vq comes first. They're just not mixed on the command line.

> > to provide a guest vector lengths 128, 256, and 512 bits. The resulting
> > set must conform to the architectural constraint of having all power-of-2
> > lengths smaller than the maximum length present. It's also possible to
> > only provide sve<vl-bits> properties, e.g. -cpu max,sve512=on> That example provides the machine with 128, 256, and 512 bit vector
> lengths.
> > It doesn't hurt to explicitly ask for all expected vector lengths,
> > which is what, for example, libvirt should do.>
> > Note1, it is not possible to use sve<vl-bits> properties before
> > sve-max-vq, e.g. -cpu max,sve384=off,sve-max-vq=4, as supporting
> > that overly complicates the user input validation.
> > 
> > Note2, while one might expect -cpu max,sve-max-vq=4,sve512=on to be the
> > same as -cpu max,sve512=on, they are not.
> yep it is a bit weird
> 
> Didn't you consider -cpu max,sve-max-vq=4,req_only=true removing non
> power of 2 values and sve<vl-bits> setting a single VLbit?

Nope. I mostly considered making sure sve-max-vq was supported to the
level necessary to work with the new properties, and to not change its
current semantics, but I don't want to extend it in any way, nor to
require it to be used with the new properties at all. sve-max-vq isn't
even going to be supported by '-cpu host', so we can't rely on it being
part of the SVE vector length selection for normal KVM guests.

Keep in mind that the current semantics of sve-max-vq are to enable all
vector lengths up to and including that specified maximum. That's not
a safe way to select vector lengths on KVM, as it may include vector
lengths that the KVM host does not support, thus it's not something KVM
users should use. It's already there for the 'max' cpu type though, so
we work with it in this series the best we can for 'max', but not at all
for 'host'.

Re: documentation for this. I suppose I could write something up that
clarifies the properties and also suggests best practices regarding
sve-max-vq.

> > The former enables all vector lengths 512 bits and smaller
> ( required and permitted)
> > while the latter only sets the 512-bit
> > length and its smaller power-of-2 lengths. It's probably best not to use
> > sve-max-vq with sve<vl-bits> properties, but it can't be completely
> > forbidden as we want qmp_query_cpu_model_expansion to work with guests
> > launched with e.g. -cpu max,sve-max-vq=8 on their command line.
> what does happen if you specify -cpu max,sve384=on? (seems to be allowed
> in the code?)

That's perfectly valid with tcg, you'll get 128,256,384. It's also valid
with KVM if you're host supports it.

> 
> > 
> > Signed-off-by: Andrew Jones <drjones@redhat.com>
> > ---
> >  target/arm/cpu.c         |   6 +
> >  target/arm/cpu.h         |  14 ++
> >  target/arm/cpu64.c       | 360 ++++++++++++++++++++++++++++++++++++++-
> >  target/arm/helper.c      |  11 +-
> >  target/arm/monitor.c     |  16 ++
> >  tests/arm-cpu-features.c | 217 +++++++++++++++++++++++
> >  6 files changed, 620 insertions(+), 4 deletions(-)
> > 
> > diff --git a/target/arm/cpu.c b/target/arm/cpu.c
> > index f08e178fc84b..e060a0d9df0e 100644
> > --- a/target/arm/cpu.c
> > +++ b/target/arm/cpu.c
> > @@ -1019,6 +1019,12 @@ static void arm_cpu_realizefn(DeviceState *dev, Error **errp)
> >          return;
> >      }
> >  
> > +    arm_cpu_sve_finalize(cpu, &local_err);
> > +    if (local_err) {
> > +        error_propagate(errp, local_err);
> > +        return;
> > +    }
> > +
> >      if (arm_feature(env, ARM_FEATURE_AARCH64) &&
> >          cpu->has_vfp != cpu->has_neon) {
> >          /*
> > diff --git a/target/arm/cpu.h b/target/arm/cpu.h
> > index f9da672be575..cbb155cf72a5 100644
> > --- a/target/arm/cpu.h
> > +++ b/target/arm/cpu.h
> > @@ -184,8 +184,13 @@ typedef struct {
> >  
> >  #ifdef TARGET_AARCH64
> >  # define ARM_MAX_VQ    16
> > +void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp);
> > +uint32_t arm_cpu_vq_map_next_smaller(ARMCPU *cpu, uint32_t vq);
> >  #else
> >  # define ARM_MAX_VQ    1
> > +static inline void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp) { }
> > +static inline uint32_t arm_cpu_vq_map_next_smaller(ARMCPU *cpu, uint32_t vq)
> > +{ return 0; }
> >  #endif
> >  
> >  typedef struct ARMVectorReg {
> > @@ -915,6 +920,15 @@ struct ARMCPU {
> >  
> >      /* Used to set the maximum vector length the cpu will support.  */
> >      uint32_t sve_max_vq;
> > +
> > +    /*
> > +     * In sve_vq_map each set bit is a supported vector length of
> > +     * (bit-number + 1) * 16 bytes, i.e. each bit number + 1 is the vector> +     * length in quadwords. We need a map size twice the maximum
> > +     * quadword length though because we use two bits for each vector
> > +     * length in order to track three states: uninitialized, off, and on.
> > +     */
> > +    DECLARE_BITMAP(sve_vq_map, ARM_MAX_VQ * 2);
> >  };
> >  
> >  void arm_cpu_post_init(Object *obj);
> > diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
> > index 02ada65f240c..5def82111dee 100644
> > --- a/target/arm/cpu64.c
> > +++ b/target/arm/cpu64.c
> > @@ -257,6 +257,149 @@ static void aarch64_a72_initfn(Object *obj)
> >      define_arm_cp_regs(cpu, cortex_a72_a57_a53_cp_reginfo);
> >  }
> >  
> > +/*
> > + * While we eventually use cpu->sve_vq_map as a typical bitmap, where each vq
> > + * has only two states (off/on), until we've finalized the map at realize time
> > + * we use an extra bit, at the vq - 1 + ARM_MAX_VQ bit number, to also allow
> > + * tracking of the uninitialized state. The arm_vq_state typedef and following
> > + * functions allow us to more easily work with the bitmap. Also, while the map
> > + * is still initializing, sve-max-vq has an additional three states, bringing
> > + * the number of its states to five, which are the following:
> > + *
> > + * sve-max-vq:
> > + *   0:    SVE is disabled. The default value for a vq in the map is 'OFF'.
> > + *  -1:    SVE is enabled, but neither sve-max-vq nor sve<vl-bits> properties
> > + *         have yet been specified by the user. The default value for a vq in
> > + *         the map is 'ON'.
> > + *  -2:    SVE is enabled and one or more sve<vl-bits> properties have been
> > + *         set to 'OFF' by the user, but no sve<vl-bits> properties have yet
> > + *         been set to 'ON'. The user is now blocked from setting sve-max-vq
> > + *         and the default value for a vq in the map is 'ON'.
> > + *  -3:    SVE is enabled and one or more sve<vl-bits> properties have been
> > + *         set to 'ON' by the user. The user is blocked from setting sve-max-vq
> > + *         and the default value for a vq in the map is 'OFF'. sve-max-vq never
> > + *         transitions back to -2, even if later inputs disable the vector
> > + *         lengths that initially transitioned sve-max-vq to this state. This
> > + *         avoids the default values from flip-flopping.
> > + *  [1-ARM_MAX_VQ]: SVE is enabled and the user has specified a valid
> > + *                  sve-max-vq. The sve-max-vq specified vq and all smaller
> > + *                  vq's will be initially enabled. All larger vq's will have
> > + *                  a default of 'OFF'.
> > + */
> > +#define ARM_SVE_INIT          -1
> > +#define ARM_VQ_DEFAULT_ON     -2
> > +#define ARM_VQ_DEFAULT_OFF    -3
> > +
> > +#define arm_sve_have_max_vq(cpu) ((int32_t)(cpu)->sve_max_vq > 0)
> > +
> > +typedef enum arm_vq_state {
> > +    ARM_VQ_OFF,
> > +    ARM_VQ_ON,
> > +    ARM_VQ_UNINITIALIZED,
> > +} arm_vq_state;
> > +
> > +static arm_vq_state arm_cpu_vq_map_get(ARMCPU *cpu, int vq)> +{
> > +    assert(vq <= ARM_MAX_VQ);
> > +
> > +    return test_bit(vq - 1, cpu->sve_vq_map) |
> > +           test_bit(vq - 1 + ARM_MAX_VQ, cpu->sve_vq_map) << 1;
> > +}> +
> > +static void arm_cpu_vq_map_set(ARMCPU *cpu, int vq, arm_vq_state state)
> > +{
> > +    assert(state == ARM_VQ_OFF || state == ARM_VQ_ON);
> > +    assert(vq <= ARM_MAX_VQ);
> > +
> > +    clear_bit(vq - 1 + ARM_MAX_VQ, cpu->sve_vq_map);
> > +
> > +    if (state == ARM_VQ_ON) {
> > +        set_bit(vq - 1, cpu->sve_vq_map);
> > +    } else {
> > +        clear_bit(vq - 1, cpu->sve_vq_map);
> > +    }
> > +}
> > +
> > +static void arm_cpu_vq_map_init(ARMCPU *cpu)
> > +{
> > +    bitmap_zero(cpu->sve_vq_map, ARM_MAX_VQ * 2);
> nit: bitmap_clear(map, 0, ARM_MAX_VQ);

bitmap_clear will only clear the specified bits, leaving the upper bits
uninitialized, which could cause problems. It's not a problem here
because we're using a zero initialized cpu state member, but if it was
a bitmap like below, declared on the stack, then we need the zeroing at
least once. I'd prefer to use zero here too, to keep it consistent.

> /* all VLs OFF */
> > +    bitmap_set(cpu->sve_vq_map, ARM_MAX_VQ, ARM_MAX_VQ);
> /* all VLs uninitialized */

I can add one comment that says

"Set all VQ's to ARM_VQ_UNINITIALIZED" above both lines.

> > +}
> > +
> > +static bool arm_cpu_vq_map_is_finalized(ARMCPU *cpu)
> > +{
> > +    DECLARE_BITMAP(map, ARM_MAX_VQ * 2);
> > +
> > +    bitmap_zero(map, ARM_MAX_VQ * 2);
> same

Here me must use bitmap_zero.

> > +    bitmap_set(map, ARM_MAX_VQ, ARM_MAX_VQ);
> > +    bitmap_and(map, map, cpu->sve_vq_map, ARM_MAX_VQ * 2);
> > +
> > +    return bitmap_empty(map, ARM_MAX_VQ * 2);
> > +}
> > +
> > +static void arm_cpu_vq_map_finalize(ARMCPU *cpu)
> > +{
> > +    Error *err = NULL;
> > +    char name[8];
> > +    uint32_t vq;
> > +    bool value;
> > +
> > +    /*
> > +     * We use the property get accessor because it knows what default
> > +     * values to return for uninitialized vector lengths.
> > +     */
> > +    for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
> > +        sprintf(name, "sve%d", vq * 128);
> > +        value = object_property_get_bool(OBJECT(cpu), name, &err);
> > +        assert(!err);
> > +        if (value) {
> > +            arm_cpu_vq_map_set(cpu, vq, ARM_VQ_ON);
> > +        } else {
> > +            arm_cpu_vq_map_set(cpu, vq, ARM_VQ_OFF);
> > +        }
> > +    }
> > +
> > +    assert(arm_cpu_vq_map_is_finalized(cpu));
> this can be removed

Yes, it can be today, as the code works fine. The idea was that if
somebody came in here and modified this in someway that broke the
finalized state, then we'd catch it here before going off and doing
weird things. This isn't a hot path, so I'd prefer we keep it, but
if QEMU maintainers prefer to limit defensive code, then I can
certainly remove it.

> > +}
> > +
> > +void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp)
> > +{
> > +    Error *err = NULL;
> > +
> > +    if (!cpu->sve_max_vq) {
> > +        bitmap_zero(cpu->sve_vq_map, ARM_MAX_VQ * 2);
> > +        return;
> > +    }
> > +
> > +    if (cpu->sve_max_vq == ARM_SVE_INIT) {
> > +        object_property_set_uint(OBJECT(cpu), ARM_MAX_VQ, "sve-max-vq", &err);
> > +        if (err) {
> > +            error_propagate(errp, err);
> > +            return;
> > +        }
> > +        assert(cpu->sve_max_vq == ARM_MAX_VQ);
> I guess those asserts can be removed now?

I'd prefer to keep it. It's critical that we get this right, so if
somebody changes something somewhere in the property set/get code
that would break this, then it's best we know right away. Again,
though, if there's some reason to limit defensive code, even on the
init path, then I can.

> > +        arm_cpu_vq_map_finalize(cpu);
> move the arm_cpu_vq_map_finalize out of the if, at the end.

That wouldn't work. In this branch arm_cpu_vq_map_finalize() comes
after setting cpu->sve_max_vq. In the else branch it must come first.

> > +    } else {
> > +        arm_cpu_vq_map_finalize(cpu);
> > +        if (!arm_sve_have_max_vq(cpu)) {
> > +            cpu->sve_max_vq = arm_cpu_vq_map_next_smaller(cpu, ARM_MAX_VQ + 1);
> > +        }
> > +    }
> > +
> > +    assert(cpu->sve_max_vq == arm_cpu_vq_map_next_smaller(cpu, ARM_MAX_VQ));

What happened to my '+ 1' here? I see it's in the patch, but somehow got
removed in your quote of the patch? For a second there I thought something
was really wrong, since that assert should have fired for every full map.

> same here

Anyway my same argument that we don't want to leave arm_cpu_sve_finalize()
without knowing we got this right applies.

> > +}
> > +
> > +uint32_t arm_cpu_vq_map_next_smaller(ARMCPU *cpu, uint32_t vq)
> > +{
> > +    uint32_t bitnum;
> > +
> > +    assert(vq <= ARM_MAX_VQ + 1);
> > +    assert(arm_cpu_vq_map_is_finalized(cpu));
> > +
> > +    bitnum = find_last_bit(cpu->sve_vq_map, vq - 1);
> why do you pass ARM_MAX_VQ + 1 and then do vq -1? doesn't
> find_last_bit() take the size which is ARM_MAX_VQ in this case?

The size is ARM_MAX_VQ + 1, when you want to also check the bitnum
corresponding to ARM_MAX_VQ. The bitnum for a VQ is always VQ - 1.
Recall VQ=1, which is 128-bits. It takes the bit position zero.
VQ=ARM_MAX_VQ=16 takes the bit position 15. As for the 'vq - 1' here
in the find_last_bit() call, that's required because that's what the
function says it does. It finds the next smaller VQ. So if you pass
in, for example, vq=2, it shouldn't search anything but bit position
zero:

 vq=2 => max next smaller is vq=1, bitnum = 0
      => search the bitmap of size 1 for the last set bit

Or, if you want to search the whole map, including the maximum
possibly VQ, as is done above, then you need to pass ARM_MAX_VQ + 1,
since the max next smaller VQ is ARM_MAX_VQ.

> > +    return bitnum == vq - 1 ? 0 : bitnum + 1;
> > +}
> > +
> >  static void cpu_max_get_sve_max_vq(Object *obj, Visitor *v, const char *name,
> >                                     void *opaque, Error **errp)
> >  {
> > @@ -283,12 +426,203 @@ static void cpu_max_set_sve_max_vq(Object *obj, Visitor *v, const char *name,
> >          return;
> >      }
> >  
> > +    /*
> > +     * It gets complicated trying to support both sve-max-vq and
> > +     * sve<vl-bits> properties together, so we mostly don't. We
> > +     * do allow both if sve-max-vq is specified first and only once
> > +     * though.
> > +     */
> > +    if (cpu->sve_max_vq != ARM_SVE_INIT) {
> > +        error_setg(errp, "sve<vl-bits> in use or sve-max-vq already "
> > +                   "specified");
> > +        error_append_hint(errp, "sve-max-vq must come before all "
> > +                          "sve<vl-bits> properties and it must only "
> > +                          "be specified once.\n");
> > +        return;
> > +    }
> > +
> >      cpu->sve_max_vq = value;
> >  
> >      if (cpu->sve_max_vq == 0 || cpu->sve_max_vq > ARM_MAX_VQ) {
> >          error_setg(errp, "unsupported SVE vector length");
> >          error_append_hint(errp, "Valid sve-max-vq in range [1-%d]\n",
> >                            ARM_MAX_VQ);
> > +    } else {
> > +        uint32_t vq;
> > +
> > +        for (vq = 1; vq <= cpu->sve_max_vq; ++vq) {
> > +            char name[8];
> > +            sprintf(name, "sve%d", vq * 128);
> > +            object_property_set_bool(obj, true, name, &err);
> > +            if (err) {
> > +                error_propagate(errp, err);
> > +                return;
> > +            }
> > +        }
> > +    }
> > +}
> > +
> > +static void cpu_arm_get_sve_vq(Object *obj, Visitor *v, const char *name,
> > +                               void *opaque, Error **errp)
> > +{
> > +    ARMCPU *cpu = ARM_CPU(obj);
> > +    int vq = atoi(&name[3]) / 128;
> > +    arm_vq_state vq_state;
> > +    bool value;
> > +
> > +    vq_state = arm_cpu_vq_map_get(cpu, vq);
> > +
> > +    if (!cpu->sve_max_vq) {
> > +        /* All vector lengths are disabled when SVE is off. */
> > +        value = false;
> > +    } else if (vq_state == ARM_VQ_ON) {
> > +        value = true;
> > +    } else if (vq_state == ARM_VQ_OFF) {
> > +        value = false;
> > +    } else {
> > +        /*
> > +         * vq is uninitialized. We pick a default here based on the
> > +         * the state of sve-max-vq and other sve<vl-bits> properties.
> > +         */
> > +        if (arm_sve_have_max_vq(cpu)) {
> > +            /*
> > +             * If we have sve-max-vq, then all remaining uninitialized
> > +             * vq's are 'OFF'.
> > +             */
> > +            value = false;
> > +        } else {
> > +            switch (cpu->sve_max_vq) {
> > +            case ARM_SVE_INIT:
> > +            case ARM_VQ_DEFAULT_ON:
> > +                value = true;
> > +                break;
> > +            case ARM_VQ_DEFAULT_OFF:
> > +                value = false;
> > +                break;
> > +            }
> > +        }
> > +    }
> > +
> > +    visit_type_bool(v, name, &value, errp);
> > +}
> > +
> > +static void cpu_arm_set_sve_vq(Object *obj, Visitor *v, const char *name,
> > +                               void *opaque, Error **errp)
> > +{
> > +    ARMCPU *cpu = ARM_CPU(obj);
> > +    int vq = atoi(&name[3]) / 128;
> > +    arm_vq_state vq_state;
> > +    Error *err = NULL;
> > +    uint32_t max_vq = 0;
> > +    bool value;
> > +
> > +    visit_type_bool(v, name, &value, &err);
> > +    if (err) {
> > +        error_propagate(errp, err);
> > +        return;
> > +    }
> > +
> > +    if (value && !cpu->sve_max_vq) {
> > +        error_setg(errp, "cannot enable %s", name);
> > +        error_append_hint(errp, "SVE has been disabled with sve=off\n");
> > +        return;
> > +    } else if (!cpu->sve_max_vq) {
> > +        /*
> > +         * We don't complain about disabling vector lengths when SVE
> > +         * is off, but we don't do anything either.
> > +         */
> > +        return;
> > +    }
> > +
> > +    if (arm_sve_have_max_vq(cpu)) {
> > +        max_vq = cpu->sve_max_vq;
> > +    } else {
> > +        if (value) {
> > +            cpu->sve_max_vq = ARM_VQ_DEFAULT_OFF;
> > +        } else if (cpu->sve_max_vq != ARM_VQ_DEFAULT_OFF) {
> > +            cpu->sve_max_vq = ARM_VQ_DEFAULT_ON;
> > +        }
> > +    }
> > +
> > +    /*
> > +     * We need to know the maximum vector length, which may just currently
> > +     * be the maximum length, in order to validate the enabling/disabling
> > +     * of this vector length. We use the property get accessor in order to
> > +     * get the appropriate default value for any uninitialized lengths.
> > +     */
> > +    if (!max_vq) {
> > +        char tmp[8];
> > +        bool s;
> > +
> > +        for (max_vq = ARM_MAX_VQ; max_vq >= 1; --max_vq) {
> > +            sprintf(tmp, "sve%d", max_vq * 128);
> > +            s = object_property_get_bool(OBJECT(cpu), tmp, &err);
> > +            assert(!err);
> > +            if (s) {
> > +                break;
> > +            }
> > +        }
> > +    }
> > +
> > +    if (arm_sve_have_max_vq(cpu) && value && vq > cpu->sve_max_vq) {
> > +        error_setg(errp, "cannot enable %s", name);
> > +        error_append_hint(errp, "vq=%d (%d bits) is larger than the "
> > +                          "maximum vector length, sve-max-vq=%d "
> > +                          "(%d bits)\n", vq, vq * 128,
> > +                          cpu->sve_max_vq, cpu->sve_max_vq * 128);
> > +    } else if (arm_sve_have_max_vq(cpu) && !value && vq == cpu->sve_max_vq) {
> > +        error_setg(errp, "cannot disable %s", name);
> > +        error_append_hint(errp, "The maximum vector length must be "
> > +                          "enabled, sve-max-vq=%d (%d bits)\n",
> > +                          cpu->sve_max_vq, cpu->sve_max_vq * 128);
> > +    } else if (arm_sve_have_max_vq(cpu) && !value && vq < cpu->sve_max_vq &&
> > +               is_power_of_2(vq)) {
> > +        error_setg(errp, "cannot disable %s", name);
> > +        error_append_hint(errp, "vq=%d (%d bits) is required as it is a "
> > +                          "power-of-2 length smaller than the maximum, "
> > +                          "sve-max-vq=%d (%d bits)\n", vq, vq * 128,
> > +                          cpu->sve_max_vq, cpu->sve_max_vq * 128);
> > +    } else if (!value && vq < max_vq && is_power_of_2(vq)) {
> > +        error_setg(errp, "cannot disable %s", name);
> > +        error_append_hint(errp, "Vector length %d-bits is required as it "
> > +                          "is a power-of-2 length smaller than another "
> > +                          "enabled vector length. Disable all larger vector "
> > +                          "lengths first.\n", vq * 128);
> > +    } else {
> adding return in each if/elsif would allow to avoid this indent.

Yeah, but I didn't really mind the indent :-)

> > +        if (value) {
> > +            bool fail = false;
> > +            uint32_t s;
> > +
> > +            /*
> > +             * Enabling a vector length automatically enables all
> > +             * uninitialized power-of-2 lengths smaller than it, as
> > +             * per the architecture.
> > +             */
> Test we are not attempting to enable a !is_power_of_2

I'm not sure what this means. In this context 'vq' can be a !power-of-2
if it wants to be, and there's no way for 's' to be a !power-of-2 because
we filter the vq's with is_power_of_2(s).

> > +            for (s = 1; s < vq; ++s) {
> > +                if (is_power_of_2(s)) {
> > +                    vq_state = arm_cpu_vq_map_get(cpu, s);
> > +                    if (vq_state == ARM_VQ_UNINITIALIZED) {
> > +                        arm_cpu_vq_map_set(cpu, s, ARM_VQ_ON);
> > +                    } else if (vq_state == ARM_VQ_OFF) {
> > +                        fail = true;
> > +                        break;
> > +                    }
> > +                }
> > +            }
> > +
> > +            if (fail) {
> > +                error_setg(errp, "cannot enable %s", name);
> > +                error_append_hint(errp, "Vector length %d-bits is disabled "
> > +                                  "and is a power-of-2 length smaller than "
> > +                                  "%s. All power-of-2 vector lengths smaller "
> > +                                  "than the maximum length are required.\n",
> > +                                  s * 128, name);
> > +            } else {
> > +                arm_cpu_vq_map_set(cpu, vq, ARM_VQ_ON);
> > +            }
> > +        } else {
> > +            arm_cpu_vq_map_set(cpu, vq, ARM_VQ_OFF);
> > +        }
> >      }
> >  }
> >  
> > @@ -318,10 +652,11 @@ static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
> >          /*
> >           * We handle the -cpu <cpu>,sve=off,sve=on case by reinitializing,
> >           * but otherwise we don't do anything as an sve=on could come after
> > -         * a sve-max-vq setting.
> > +         * a sve-max-vq or sve<vl-bits> setting.
> >           */
> >          if (!cpu->sve_max_vq) {
> > -            cpu->sve_max_vq = ARM_MAX_VQ;
> > +            cpu->sve_max_vq = ARM_SVE_INIT;
> > +            arm_cpu_vq_map_init(cpu);
> >          }
> >      } else {
> >          cpu->sve_max_vq = 0;
> > @@ -336,6 +671,7 @@ static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
> >  static void aarch64_max_initfn(Object *obj)
> >  {
> >      ARMCPU *cpu = ARM_CPU(obj);
> > +    uint32_t vq;
> >  
> >      if (kvm_enabled()) {
> >          kvm_arm_set_cpu_features_from_host(cpu);
> > @@ -420,11 +756,29 @@ static void aarch64_max_initfn(Object *obj)
> >          cpu->dcz_blocksize = 7; /*  512 bytes */
> >  #endif
> >  
> > -        cpu->sve_max_vq = ARM_MAX_VQ;
> > +        /*
> > +         * sve_max_vq is initially unspecified, but must be initialized to a
> > +         * non-zero value (ARM_SVE_INIT) to indicate that this cpu type has
> > +         * SVE. It will be finalized in arm_cpu_realizefn().
> > +         */
> > +        cpu->sve_max_vq = ARM_SVE_INIT;
> >          object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_max_vq,
> >                              cpu_max_set_sve_max_vq, NULL, NULL, &error_fatal);
> >          object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
> >                              cpu_arm_set_sve, NULL, NULL, &error_fatal);
> > +
> > +        /*
> > +         * sve_vq_map uses a special state while setting properties, so
> > +         * we initialize it here with its init function and finalize it
> > +         * in arm_cpu_realizefn().
> > +         */
> > +        arm_cpu_vq_map_init(cpu);
> > +        for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
> > +            char name[8];
> > +            sprintf(name, "sve%d", vq * 128);
> > +            object_property_add(obj, name, "bool", cpu_arm_get_sve_vq,
> > +                                cpu_arm_set_sve_vq, NULL, NULL, &error_fatal);
> > +        }
> >      }
> >  }
> >  
> > diff --git a/target/arm/helper.c b/target/arm/helper.c
> > index f500ccb6d31b..b7b719dba57f 100644
> > --- a/target/arm/helper.c
> > +++ b/target/arm/helper.c
> > @@ -5324,7 +5324,16 @@ static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
> >  
> >      /* Bits other than [3:0] are RAZ/WI.  */
> >      QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
> > -    raw_write(env, ri, value & 0xf);
> > +    value &= 0xf;
> > +
> > +    if (value) {> +        /* get next vq that is smaller than or equal to value's vq */
> > +        uint32_t vq = value + 1;
> ditto

What does this 'ditto' map to? Oh, probably the discussion about vq's
getting +1's and -1's. In this context 'value' is the ZCR_ELx
representation of a VQ, which is VQ - 1, just like in the bitmap. To
translate that to a VQ we need to do a '+ 1'. As I wrote in the comment
here, we want to find the next smaller or equal VQ here, because this is
the input VQ from the guest which may itself be a valid VQ, but if it's
not, we need to step down to the next valid one. So we ask
arm_cpu_vq_map_next_smaller() for 'vq + 1' in order to ensure 'vq' will
be in the searched range. After we get a valid vq from
arm_cpu_vq_map_next_smaller(), which must be at least '1', since that's
required by the architecture and thus will always be present, we need to
translate it back to the ZCR_ELx representation with the subtraction.
Phew... We programmers do love adding and subtracting by one, right :-)

> > +        vq = arm_cpu_vq_map_next_smaller(cpu, vq + 1);
> > +        value = vq - 1;
> > +    }
> > +
> > +    raw_write(env, ri, value);
> >  
> >      /*
> >       * Because we arrived here, we know both FP and SVE are enabled;
> > diff --git a/target/arm/monitor.c b/target/arm/monitor.c
> > index 157c487a1551..1e213906fd8f 100644
> > --- a/target/arm/monitor.c
> > +++ b/target/arm/monitor.c
> > @@ -89,8 +89,24 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)
> >      return head;
> >  }
> >  
> > +QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
> > +
> > +/*
> > + * These are cpu model features we want to advertise. The order here
> > + * matters as this is the order in which qmp_query_cpu_model_expansion
> > + * will attempt to set them. If there are dependencies between features,
> > + * as there are with the sve<vl-bits> features, then the order that
> > + * considers those dependencies must be used.
> > + *
> > + * The sve<vl-bits> features need to be in reverse order in order to
> > + * enable/disable the largest vector lengths first, ensuring all
> > + * power-of-2 vector lengths smaller can also be enabled/disabled.
> > + */
> >  static const char *cpu_model_advertised_features[] = {
> >      "aarch64", "pmu", "sve",
> > +    "sve2048", "sve1920", "sve1792", "sve1664", "sve1536", "sve1408",
> > +    "sve1280", "sve1152", "sve1024", "sve896", "sve768", "sve640",
> > +    "sve512", "sve384", "sve256", "sve128",
> >      NULL
> >  };
> >  
> > diff --git a/tests/arm-cpu-features.c b/tests/arm-cpu-features.c
> Please move all the tests in a separate patch.

I'd prefer not to. I like keeping the test cases that test the new code in
the same patch. It self documents what the test cases map to what code and
allows immediate testing of the patch when bisecting. Also I'm not really
sure how it makes review worse, as another patch would look the same, just
in a different email. 

> Each day has enough trouble of its own ;-) sweat...

I really appreciate the review! I realize it's generating sweat,
especially with the European heat wave! You don't have to finish
a patch before sending comments. I'm fine with multiple replies to
the same patch if you want to break your review up into smaller
bites.

Thanks,
drew

> 
> Thanks
> 
> Eric
> > index 509e458e9c2f..a4bf6aec00df 100644
> > --- a/tests/arm-cpu-features.c
> > +++ b/tests/arm-cpu-features.c
> > @@ -13,6 +13,18 @@
> >  #include "qapi/qmp/qdict.h"
> >  #include "qapi/qmp/qjson.h"
> >  
> > +#if __SIZEOF_LONG__ == 8
> > +#define BIT(n) (1UL << (n))
> > +#else
> > +#define BIT(n) (1ULL << (n))
> > +#endif
> > +
> > +/*
> > + * We expect the SVE max-vq to be 16. Also it must be <= 64
> > + * for our test code, otherwise 'vls' can't just be a uint64_t.
> > + */
> > +#define SVE_MAX_VQ 16
> > +
> >  #define MACHINE    "-machine virt,gic-version=max "
> >  #define QUERY_HEAD "{ 'execute': 'query-cpu-model-expansion', " \
> >                       "'arguments': { 'type': 'full', "
> > @@ -137,6 +149,201 @@ static void assert_bad_props(QTestState *qts, const char *cpu_type)
> >      qobject_unref(resp);
> >  }
> >  
> > +static void resp_get_sve_vls(QDict *resp, uint64_t *vls, uint32_t *max_vq)
> > +{
> > +    const QDictEntry *e;
> > +    QDict *qdict;
> > +    int n = 0;
> > +
> > +    *vls = 0;
> > +
> > +    qdict = resp_get_props(resp);
> > +
> > +    for (e = qdict_first(qdict); e; e = qdict_next(qdict, e)) {
> > +        if (strlen(e->key) > 3 && !strncmp(e->key, "sve", 3) &&
> > +            g_ascii_isdigit(e->key[3])) {
> > +            char *endptr;
> > +            int bits;
> > +
> > +            bits = g_ascii_strtoll(&e->key[3], &endptr, 10);
> > +            if (!bits || *endptr != '\0') {
> > +                continue;
> > +            }
> > +
> > +            if (qdict_get_bool(qdict, e->key)) {
> > +                *vls |= BIT((bits / 128) - 1);
> > +            }
> > +            ++n;
> > +        }
> > +    }
> > +
> > +    g_assert(n == SVE_MAX_VQ);
> > +
> > +    *max_vq = !*vls ? 0 : 64 - __builtin_clzll(*vls);
> > +}
> > +
> > +static uint64_t sve_get_vls(QTestState *qts, const char *cpu_type,
> > +                            const char *fmt, ...)
> > +{
> > +    QDict *resp;
> > +    uint64_t vls;
> > +    uint32_t max_vq;
> > +
> > +    if (fmt) {
> > +        QDict *args;
> > +        va_list ap;
> > +
> > +        va_start(ap, fmt);
> > +        args = qdict_from_vjsonf_nofail(fmt, ap);
> > +        va_end(ap);
> > +
> > +        resp = qtest_qmp(qts, QUERY_HEAD "'model': { 'name': %s, "
> > +                                                    "'props': %p }"
> > +                              QUERY_TAIL, cpu_type, args);
> > +    } else {
> > +        resp = do_query_no_props(qts, cpu_type);
> > +    }
> > +
> > +    g_assert(resp);
> > +    resp_get_sve_vls(resp, &vls, &max_vq);
> > +    qobject_unref(resp);
> > +
> > +    return vls;
> > +}
> > +
> > +#define assert_sve_vls(qts, cpu_type, expected_vls, fmt, ...) \
> > +    g_assert(sve_get_vls(qts, cpu_type, fmt, ##__VA_ARGS__) == expected_vls)
> > +
> > +static void sve_tests_default(QTestState *qts, const char *cpu_type)
> > +{
> > +    /*
> > +     * With no sve-max-vq or sve<vl-bits> properties on the command line
> > +     * the default is to have all vector lengths enabled.
> > +     */
> > +    assert_sve_vls(qts, cpu_type, BIT(SVE_MAX_VQ) - 1, NULL);
> > +
> > +    /*
> > +     * -------------------------------------------------------------------
> > +     *               power-of-2(vq)   all-power-            can      can
> > +     *                                of-2(< vq)          enable   disable
> > +     * -------------------------------------------------------------------
> > +     * vq < max_vq      no            MUST*                yes      yes
> > +     * vq < max_vq      yes           MUST*                yes      no
> > +     * -------------------------------------------------------------------
> > +     * vq == max_vq     n/a           MUST*                yes**    yes**
> > +     * -------------------------------------------------------------------
> > +     * vq > max_vq      n/a           no                   no       yes
> > +     * vq > max_vq      n/a           yes                  yes      yes
> > +     * -------------------------------------------------------------------
> > +     *
> > +     * [*] "MUST" means this requirement must already be satisfied,
> > +     *     otherwise 'max_vq' couldn't itself be enabled.
> > +     *
> > +     * [**] Not testable with the QMP interface, only with the command line.
> > +     */
> > +
> > +    /* max_vq := 8 */
> > +    assert_sve_vls(qts, cpu_type, 0x8b, "{ 'sve1024': true }");
> > +
> > +    /* max_vq := 8, vq < max_vq, !power-of-2(vq) */
> > +    assert_sve_vls(qts, cpu_type, 0x8f,
> > +                   "{ 'sve1024': true, 'sve384': true }");
> > +    assert_sve_vls(qts, cpu_type, 0x8b,
> > +                   "{ 'sve1024': true, 'sve384': false }");
> > +
> > +    /* max_vq := 8, vq < max_vq, power-of-2(vq) */
> > +    assert_sve_vls(qts, cpu_type, 0x8b,
> > +                   "{ 'sve1024': true, 'sve256': true }");
> > +    assert_error(qts, cpu_type, "cannot disable sve256",
> > +                 "{ 'sve1024': true, 'sve256': false }");
> > +
> > +    /*
> > +     * max_vq := 3, vq > max_vq, !all-power-of-2(< vq)
> > +     *
> > +     * If given sve384=on,sve512=off,sve640=on the command line error would be
> > +     * "cannot enable sve640", but QMP visits the vector lengths in reverse
> > +     * order, so we get "cannot disable sve512" instead. The command line would
> > +     * also give that error if given sve384=on,sve640=on,sve512=off, so this is
> > +     * all fine. The important thing is that we get an error.
> > +     */
> > +    assert_error(qts, cpu_type, "cannot disable sve512",
> > +                 "{ 'sve384': true, 'sve512': false, 'sve640': true }");
> > +
> > +    /*
> > +     * We can disable power-of-2 vector lengths when all larger lengths
> > +     * are also disabled. The shorter, sve384=on,sve512=off,sve640=off
> > +     * works on the command line, but QMP doesn't know that all the
> > +     * vector lengths larger than 384-bits will be disabled until it
> > +     * sees the enabling of sve384, which comes near the end since it
> > +     * visits the lengths in reverse order. So we just have to explicitly
> > +     * disable them all.
> > +     */
> > +    assert_sve_vls(qts, cpu_type, 0x7,
> > +                   "{ 'sve384': true, 'sve512': false, 'sve640': false, "
> > +                   "  'sve768': false, 'sve896': false, 'sve1024': false, "
> > +                   "  'sve1152': false, 'sve1280': false, 'sve1408': false, "
> > +                   "  'sve1536': false, 'sve1664': false, 'sve1792': false, "
> > +                   "  'sve1920': false, 'sve2048': false }");
> > +
> > +    /* max_vq := 3, vq > max_vq, all-power-of-2(< vq) */
> > +    assert_sve_vls(qts, cpu_type, 0x1f,
> > +                   "{ 'sve384': true, 'sve512': true, 'sve640': true }");
> > +    assert_sve_vls(qts, cpu_type, 0xf,
> > +                   "{ 'sve384': true, 'sve512': true, 'sve640': false }");
> > +}
> > +
> > +static void sve_tests_sve_max_vq_8(const void *data)
> > +{
> > +    QTestState *qts;
> > +
> > +    qts = qtest_init(MACHINE "-cpu max,sve-max-vq=8");
> > +
> > +    assert_sve_vls(qts, "max", BIT(8) - 1, NULL);
> > +
> > +    /*
> > +     * Disabling the max-vq set by sve-max-vq is not allowed, but
> > +     * of course enabling it is OK.
> > +     */
> > +    assert_error(qts, "max", "cannot disable sve1024", "{ 'sve1024': false }");
> > +    assert_sve_vls(qts, "max", 0xff, "{ 'sve1024': true }");
> > +
> > +    /*
> > +     * Enabling anything larger than max-vq set by sve-max-vq is not
> > +     * allowed, but of course disabling everything larger is OK.
> > +     */
> > +    assert_error(qts, "max", "cannot enable sve1152", "{ 'sve1152': true }");
> > +    assert_sve_vls(qts, "max", 0xff, "{ 'sve1152': false }");
> > +
> > +    /*
> > +     * We can disable non power-of-2 lengths smaller than the max-vq
> > +     * set by sve-max-vq, but not power-of-2 lengths.
> > +     */
> > +    assert_sve_vls(qts, "max", 0xfb, "{ 'sve384': false }");
> > +    assert_error(qts, "max", "cannot disable sve256", "{ 'sve256': false }");
> > +
> > +    qtest_quit(qts);
> > +}
> > +
> > +static void sve_tests_sve_off(const void *data)
> > +{
> > +    QTestState *qts;
> > +
> > +    qts = qtest_init(MACHINE "-cpu max,sve=off");
> > +
> > +    /*
> > +     * SVE is off, so the map should be empty.
> > +     */
> > +    assert_sve_vls(qts, "max", 0, NULL);
> > +
> > +    /*
> > +     * We can't turn anything on, but off is OK.
> > +     */
> > +    assert_error(qts, "max", "cannot enable sve128", "{ 'sve128': true }");
> > +    assert_sve_vls(qts, "max", 0, "{ 'sve128': false }");
> > +
> > +    qtest_quit(qts);
> > +}
> > +
> >  static void test_query_cpu_model_expansion(const void *data)
> >  {
> >      QTestState *qts;
> > @@ -159,9 +366,12 @@ static void test_query_cpu_model_expansion(const void *data)
> >      if (g_str_equal(qtest_get_arch(), "aarch64")) {
> >          assert_has_feature(qts, "max", "aarch64");
> >          assert_has_feature(qts, "max", "sve");
> > +        assert_has_feature(qts, "max", "sve128");
> >          assert_has_feature(qts, "cortex-a57", "pmu");
> >          assert_has_feature(qts, "cortex-a57", "aarch64");
> >  
> > +        sve_tests_default(qts, "max");
> > +
> >          /* Test that features that depend on KVM generate errors without. */
> >          assert_error(qts, "max",
> >                       "'aarch64' feature cannot be disabled "
> > @@ -213,6 +423,13 @@ int main(int argc, char **argv)
> >      qtest_add_data_func("/arm/query-cpu-model-expansion",
> >                          NULL, test_query_cpu_model_expansion);
> >  
> > +    if (g_str_equal(qtest_get_arch(), "aarch64")) {
> > +        qtest_add_data_func("/arm/max/query-cpu-model-expansion/sve-max-vq-8",
> > +                            NULL, sve_tests_sve_max_vq_8);
> > +        qtest_add_data_func("/arm/max/query-cpu-model-expansion/sve-off",
> > +                            NULL, sve_tests_sve_off);
> > +    }
> > +
> >      if (kvm_available) {
> >          qtest_add_data_func("/arm/kvm/query-cpu-model-expansion",
> >                              NULL, test_query_cpu_model_expansion_kvm);
> > 
> 

Re: [Qemu-devel] [PATCH v2 07/14] target/arm/cpu64: max cpu: Introduce sve<vl-bits> properties

[Auger Eric]

Hi Drew,
On 6/27/19 11:40 AM, Andrew Jones wrote:
> On Wed, Jun 26, 2019 at 04:58:05PM +0200, Auger Eric wrote:
>> Hi Drew,
>>
>> On 6/21/19 6:34 PM, Andrew Jones wrote:
>>> Introduce cpu properties to give fine control over SVE vector lengths.
>>> We introduce a property for each valid length up to the current
>>> maximum supported, which is 2048-bits. The properties are named, e.g.
>>> sve128, sve256, sve512, ..., where the number is the number of bits.
>> sve384 then in the natural order, otherwise it gives the impression you
>> can only specify * 128bit pow of 2 sizes at this stage of the reading.
> 
> OK
> 
>>
>>>
>>> It's now possible to do something like -cpu max,sve-max-vq=4,sve384=off
>> Document the fact sve512 cannot be turned to off, which sounds fully
>> sensible (by reading the code). By the way, I think an actual
>> documentation should be provided in qemu. Maybe as spec to agree on.
> 
> Actually, maybe I could just allow it to be disabled. It would be
> a strange command line to do '-cpu max,sve-max-vq=4,sve512=off', but if
> that's what the user wants, then there's not really any good reason to
> block it. As I say below, mixing these types of properties on the command
> line isn't really a good idea, but it's not completely blocked because we
> want a user that prefers launching their (most likely TCG) guest with,
> e.g.  '-cpu max,sve-max-vq=4', to also have a functional QMP CPU model
> expansion, but the CPU model expansion for SVE vector lengths depends on
> the sve<vl-bits> properties, thus mixing sve-max-vq with sve<vl-bits>,
> where sve-max-vq comes first. They're just not mixed on the command line.
OK
> 
>>> to provide a guest vector lengths 128, 256, and 512 bits. The resulting
>>> set must conform to the architectural constraint of having all power-of-2
>>> lengths smaller than the maximum length present. It's also possible to
>>> only provide sve<vl-bits> properties, e.g. -cpu max,sve512=on> That example provides the machine with 128, 256, and 512 bit vector
>> lengths.
>>> It doesn't hurt to explicitly ask for all expected vector lengths,
>>> which is what, for example, libvirt should do.>
>>> Note1, it is not possible to use sve<vl-bits> properties before
>>> sve-max-vq, e.g. -cpu max,sve384=off,sve-max-vq=4, as supporting
>>> that overly complicates the user input validation.
>>>
>>> Note2, while one might expect -cpu max,sve-max-vq=4,sve512=on to be the
>>> same as -cpu max,sve512=on, they are not.
>> yep it is a bit weird
>>
>> Didn't you consider -cpu max,sve-max-vq=4,req_only=true removing non
>> power of 2 values and sve<vl-bits> setting a single VLbit?
> 
> Nope. I mostly considered making sure sve-max-vq was supported to the
> level necessary to work with the new properties, and to not change its
> current semantics, but I don't want to extend it in any way, nor to
> require it to be used with the new properties at all. sve-max-vq isn't
> even going to be supported by '-cpu host', so we can't rely on it being
> part of the SVE vector length selection for normal KVM guests.
OK, as long as it is documented somewhere.
> 
> Keep in mind that the current semantics of sve-max-vq are to enable all
> vector lengths up to and including that specified maximum. That's not
> a safe way to select vector lengths on KVM, as it may include vector
> lengths that the KVM host does not support, thus it's not something KVM
> users should use. It's already there for the 'max' cpu type though, so
> we work with it in this series the best we can for 'max', but not at all
> for 'host'.
Do you expect hosts to expose only a few of the permitted values? I
imagined it would generally expose none or all of them actually. If you
discourage people to use sve-max-vq and sve<>=on only sets 2^n values,
userspace will need to set manually all intermediate !2^n values
> 
> Re: documentation for this. I suppose I could write something up that
> clarifies the properties and also suggests best practices regarding
> sve-max-vq.

To me this is really helpful to have a global understanding
> 
>>> The former enables all vector lengths 512 bits and smaller
>> ( required and permitted)
>>> while the latter only sets the 512-bit
>>> length and its smaller power-of-2 lengths. It's probably best not to use
>>> sve-max-vq with sve<vl-bits> properties, but it can't be completely
>>> forbidden as we want qmp_query_cpu_model_expansion to work with guests
>>> launched with e.g. -cpu max,sve-max-vq=8 on their command line.
>> what does happen if you specify -cpu max,sve384=on? (seems to be allowed
>> in the code?)
> 
> That's perfectly valid with tcg, you'll get 128,256,384. It's also valid
> with KVM if you're host supports it.
OK so it exposes the maximum configurable vector length + all the
corresponding required values.
> 
>>
>>>
>>> Signed-off-by: Andrew Jones <drjones@redhat.com>
>>> ---
>>>  target/arm/cpu.c         |   6 +
>>>  target/arm/cpu.h         |  14 ++
>>>  target/arm/cpu64.c       | 360 ++++++++++++++++++++++++++++++++++++++-
>>>  target/arm/helper.c      |  11 +-
>>>  target/arm/monitor.c     |  16 ++
>>>  tests/arm-cpu-features.c | 217 +++++++++++++++++++++++
>>>  6 files changed, 620 insertions(+), 4 deletions(-)
>>>
>>> diff --git a/target/arm/cpu.c b/target/arm/cpu.c
>>> index f08e178fc84b..e060a0d9df0e 100644
>>> --- a/target/arm/cpu.c
>>> +++ b/target/arm/cpu.c
>>> @@ -1019,6 +1019,12 @@ static void arm_cpu_realizefn(DeviceState *dev, Error **errp)
>>>          return;
>>>      }
>>>  
>>> +    arm_cpu_sve_finalize(cpu, &local_err);
>>> +    if (local_err) {
>>> +        error_propagate(errp, local_err);
>>> +        return;
>>> +    }
>>> +
>>>      if (arm_feature(env, ARM_FEATURE_AARCH64) &&
>>>          cpu->has_vfp != cpu->has_neon) {
>>>          /*
>>> diff --git a/target/arm/cpu.h b/target/arm/cpu.h
>>> index f9da672be575..cbb155cf72a5 100644
>>> --- a/target/arm/cpu.h
>>> +++ b/target/arm/cpu.h
>>> @@ -184,8 +184,13 @@ typedef struct {
>>>  
>>>  #ifdef TARGET_AARCH64
>>>  # define ARM_MAX_VQ    16
>>> +void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp);
>>> +uint32_t arm_cpu_vq_map_next_smaller(ARMCPU *cpu, uint32_t vq);
>>>  #else
>>>  # define ARM_MAX_VQ    1
>>> +static inline void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp) { }
>>> +static inline uint32_t arm_cpu_vq_map_next_smaller(ARMCPU *cpu, uint32_t vq)
>>> +{ return 0; }
>>>  #endif
>>>  
>>>  typedef struct ARMVectorReg {
>>> @@ -915,6 +920,15 @@ struct ARMCPU {
>>>  
>>>      /* Used to set the maximum vector length the cpu will support.  */
>>>      uint32_t sve_max_vq;
>>> +
>>> +    /*
>>> +     * In sve_vq_map each set bit is a supported vector length of
>>> +     * (bit-number + 1) * 16 bytes, i.e. each bit number + 1 is the vector> +     * length in quadwords. We need a map size twice the maximum
>>> +     * quadword length though because we use two bits for each vector
>>> +     * length in order to track three states: uninitialized, off, and on.
>>> +     */
>>> +    DECLARE_BITMAP(sve_vq_map, ARM_MAX_VQ * 2);
>>>  };
>>>  
>>>  void arm_cpu_post_init(Object *obj);
>>> diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
>>> index 02ada65f240c..5def82111dee 100644
>>> --- a/target/arm/cpu64.c
>>> +++ b/target/arm/cpu64.c
>>> @@ -257,6 +257,149 @@ static void aarch64_a72_initfn(Object *obj)
>>>      define_arm_cp_regs(cpu, cortex_a72_a57_a53_cp_reginfo);
>>>  }
>>>  
>>> +/*
>>> + * While we eventually use cpu->sve_vq_map as a typical bitmap, where each vq
>>> + * has only two states (off/on), until we've finalized the map at realize time
>>> + * we use an extra bit, at the vq - 1 + ARM_MAX_VQ bit number, to also allow
>>> + * tracking of the uninitialized state. The arm_vq_state typedef and following
>>> + * functions allow us to more easily work with the bitmap. Also, while the map
>>> + * is still initializing, sve-max-vq has an additional three states, bringing
>>> + * the number of its states to five, which are the following:
>>> + *
>>> + * sve-max-vq:
>>> + *   0:    SVE is disabled. The default value for a vq in the map is 'OFF'.
>>> + *  -1:    SVE is enabled, but neither sve-max-vq nor sve<vl-bits> properties
>>> + *         have yet been specified by the user. The default value for a vq in
>>> + *         the map is 'ON'.
>>> + *  -2:    SVE is enabled and one or more sve<vl-bits> properties have been
>>> + *         set to 'OFF' by the user, but no sve<vl-bits> properties have yet
>>> + *         been set to 'ON'. The user is now blocked from setting sve-max-vq
>>> + *         and the default value for a vq in the map is 'ON'.
>>> + *  -3:    SVE is enabled and one or more sve<vl-bits> properties have been
>>> + *         set to 'ON' by the user. The user is blocked from setting sve-max-vq
>>> + *         and the default value for a vq in the map is 'OFF'. sve-max-vq never
>>> + *         transitions back to -2, even if later inputs disable the vector
>>> + *         lengths that initially transitioned sve-max-vq to this state. This
>>> + *         avoids the default values from flip-flopping.
>>> + *  [1-ARM_MAX_VQ]: SVE is enabled and the user has specified a valid
>>> + *                  sve-max-vq. The sve-max-vq specified vq and all smaller
>>> + *                  vq's will be initially enabled. All larger vq's will have
>>> + *                  a default of 'OFF'.
>>> + */
>>> +#define ARM_SVE_INIT          -1
>>> +#define ARM_VQ_DEFAULT_ON     -2
>>> +#define ARM_VQ_DEFAULT_OFF    -3
>>> +
>>> +#define arm_sve_have_max_vq(cpu) ((int32_t)(cpu)->sve_max_vq > 0)
>>> +
>>> +typedef enum arm_vq_state {
>>> +    ARM_VQ_OFF,
>>> +    ARM_VQ_ON,
>>> +    ARM_VQ_UNINITIALIZED,
>>> +} arm_vq_state;
>>> +
>>> +static arm_vq_state arm_cpu_vq_map_get(ARMCPU *cpu, int vq)> +{
>>> +    assert(vq <= ARM_MAX_VQ);
>>> +
>>> +    return test_bit(vq - 1, cpu->sve_vq_map) |
>>> +           test_bit(vq - 1 + ARM_MAX_VQ, cpu->sve_vq_map) << 1;
>>> +}> +
>>> +static void arm_cpu_vq_map_set(ARMCPU *cpu, int vq, arm_vq_state state)
>>> +{
>>> +    assert(state == ARM_VQ_OFF || state == ARM_VQ_ON);
>>> +    assert(vq <= ARM_MAX_VQ);
>>> +
>>> +    clear_bit(vq - 1 + ARM_MAX_VQ, cpu->sve_vq_map);
>>> +
>>> +    if (state == ARM_VQ_ON) {
>>> +        set_bit(vq - 1, cpu->sve_vq_map);
>>> +    } else {
>>> +        clear_bit(vq - 1, cpu->sve_vq_map);
>>> +    }
>>> +}
>>> +
>>> +static void arm_cpu_vq_map_init(ARMCPU *cpu)
>>> +{
>>> +    bitmap_zero(cpu->sve_vq_map, ARM_MAX_VQ * 2);
>> nit: bitmap_clear(map, 0, ARM_MAX_VQ);
> 
> bitmap_clear will only clear the specified bits, leaving the upper bits
> uninitialized, which could cause problems. It's not a problem here
> because we're using a zero initialized cpu state member, but if it was
> a bitmap like below, declared on the stack, then we need the zeroing at
> least once. I'd prefer to use zero here too, to keep it consistent.
Sorry I don't get it. I would have expected the above bitmap_clear would
0 initialize 0 - ARM_MAX_VQ-1 bits and the bitmap_set below would
initialize ARM_MAX_VQ - 2 * ARM_MAX_VQ -1 with 1's?

> 
>> /* all VLs OFF */
>>> +    bitmap_set(cpu->sve_vq_map, ARM_MAX_VQ, ARM_MAX_VQ);
>> /* all VLs uninitialized */
> 
> I can add one comment that says
> 
> "Set all VQ's to ARM_VQ_UNINITIALIZED" above both lines.
> 
>>> +}
>>> +
>>> +static bool arm_cpu_vq_map_is_finalized(ARMCPU *cpu)
>>> +{
>>> +    DECLARE_BITMAP(map, ARM_MAX_VQ * 2);
>>> +
>>> +    bitmap_zero(map, ARM_MAX_VQ * 2);
>> same
> 
> Here me must use bitmap_zero.
> 
>>> +    bitmap_set(map, ARM_MAX_VQ, ARM_MAX_VQ);
>>> +    bitmap_and(map, map, cpu->sve_vq_map, ARM_MAX_VQ * 2);
>>> +
>>> +    return bitmap_empty(map, ARM_MAX_VQ * 2);
>>> +}
>>> +
>>> +static void arm_cpu_vq_map_finalize(ARMCPU *cpu)
>>> +{
>>> +    Error *err = NULL;
>>> +    char name[8];
>>> +    uint32_t vq;
>>> +    bool value;
>>> +
>>> +    /*
>>> +     * We use the property get accessor because it knows what default
>>> +     * values to return for uninitialized vector lengths.
>>> +     */
>>> +    for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
>>> +        sprintf(name, "sve%d", vq * 128);
>>> +        value = object_property_get_bool(OBJECT(cpu), name, &err);
>>> +        assert(!err);
>>> +        if (value) {
>>> +            arm_cpu_vq_map_set(cpu, vq, ARM_VQ_ON);
>>> +        } else {
>>> +            arm_cpu_vq_map_set(cpu, vq, ARM_VQ_OFF);
>>> +        }
>>> +    }
>>> +
>>> +    assert(arm_cpu_vq_map_is_finalized(cpu));
>> this can be removed
> 
> Yes, it can be today, as the code works fine. The idea was that if
> somebody came in here and modified this in someway that broke the
> finalized state, then we'd catch it here before going off and doing
> weird things. This isn't a hot path, so I'd prefer we keep it, but
> if QEMU maintainers prefer to limit defensive code, then I can
> certainly remove it.
Well I understand this was useful was developing and to check some
tricky states but some of the asserts correspond to some checks that
look rather obvious, like this one. But well I let others give their
opinions and it is not a big deal either. Then we can wonder when one
needs a trace mesg before asserting ...
> 
>>> +}
>>> +
>>> +void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp)
>>> +{
>>> +    Error *err = NULL;
>>> +
>>> +    if (!cpu->sve_max_vq) {
>>> +        bitmap_zero(cpu->sve_vq_map, ARM_MAX_VQ * 2);
>>> +        return;
>>> +    }
>>> +
>>> +    if (cpu->sve_max_vq == ARM_SVE_INIT) {
>>> +        object_property_set_uint(OBJECT(cpu), ARM_MAX_VQ, "sve-max-vq", &err);
>>> +        if (err) {
>>> +            error_propagate(errp, err);
>>> +            return;
>>> +        }
>>> +        assert(cpu->sve_max_vq == ARM_MAX_VQ);
>> I guess those asserts can be removed now?
> 
> I'd prefer to keep it. It's critical that we get this right, so if
> somebody changes something somewhere in the property set/get code
> that would break this, then it's best we know right away. Again,
> though, if there's some reason to limit defensive code, even on the
> init path, then I can.
Here I would expect that if the set did not fail, sve-max-vq effectively
if set to the expected value. But maybe I don't remember the cases where
sve_max-vq can be set to some other value without reprting an error.

> 
>>> +        arm_cpu_vq_map_finalize(cpu);
>> move the arm_cpu_vq_map_finalize out of the if, at the end.
> 
> That wouldn't work. In this branch arm_cpu_vq_map_finalize() comes
> after setting cpu->sve_max_vq. In the else branch it must come first.
That's right sorry
> 
>>> +    } else {
>>> +        arm_cpu_vq_map_finalize(cpu);
>>> +        if (!arm_sve_have_max_vq(cpu)) {
>>> +            cpu->sve_max_vq = arm_cpu_vq_map_next_smaller(cpu, ARM_MAX_VQ + 1);
>>> +        }
>>> +    }
>>> +
>>> +    assert(cpu->sve_max_vq == arm_cpu_vq_map_next_smaller(cpu, ARM_MAX_VQ));>
> What happened to my '+ 1' here? I see it's in the patch, but somehow got
> removed in your quote of the patch? For a second there I thought something
> was really wrong, since that assert should have fired for every full map.
Yep sorry for the cut :-(
> 
>> same here
> 
> Anyway my same argument that we don't want to leave arm_cpu_sve_finalize()
> without knowing we got this right applies.

> 
>>> +}
>>> +
>>> +uint32_t arm_cpu_vq_map_next_smaller(ARMCPU *cpu, uint32_t vq)
>>> +{
>>> +    uint32_t bitnum;
>>> +
>>> +    assert(vq <= ARM_MAX_VQ + 1);
>>> +    assert(arm_cpu_vq_map_is_finalized(cpu));
>>> +
>>> +    bitnum = find_last_bit(cpu->sve_vq_map, vq - 1);
>> why do you pass ARM_MAX_VQ + 1 and then do vq -1? doesn't
>> find_last_bit() take the size which is ARM_MAX_VQ in this case?
> 
> The size is ARM_MAX_VQ + 1, when you want to also check the bitnum
> corresponding to ARM_MAX_VQ. The bitnum for a VQ is always VQ - 1.
> Recall VQ=1, which is 128-bits. It takes the bit position zero.
> VQ=ARM_MAX_VQ=16 takes the bit position 15. As for the 'vq - 1' here
> in the find_last_bit() call, that's required because that's what the
> function says it does. It finds the next smaller VQ. So if you pass
> in, for example, vq=2, it shouldn't search anything but bit position
> zero:
> 
>  vq=2 => max next smaller is vq=1, bitnum = 0
>       => search the bitmap of size 1 for the last set bit
> 
> Or, if you want to search the whole map, including the maximum
> possibly VQ, as is done above, then you need to pass ARM_MAX_VQ + 1,
> since the max next smaller VQ is ARM_MAX_VQ.
OK I get it now. Maybe renaming the function into something like
arm_cpu_vq_map_last_bit() would have avoided that. the first assert
looks strange typically.
> 
>>> +    return bitnum == vq - 1 ? 0 : bitnum + 1;
>>> +}
>>> +
>>>  static void cpu_max_get_sve_max_vq(Object *obj, Visitor *v, const char *name,
>>>                                     void *opaque, Error **errp)
>>>  {
>>> @@ -283,12 +426,203 @@ static void cpu_max_set_sve_max_vq(Object *obj, Visitor *v, const char *name,
>>>          return;
>>>      }
>>>  
>>> +    /*
>>> +     * It gets complicated trying to support both sve-max-vq and
>>> +     * sve<vl-bits> properties together, so we mostly don't. We
>>> +     * do allow both if sve-max-vq is specified first and only once
>>> +     * though.
>>> +     */
>>> +    if (cpu->sve_max_vq != ARM_SVE_INIT) {
>>> +        error_setg(errp, "sve<vl-bits> in use or sve-max-vq already "
>>> +                   "specified");
>>> +        error_append_hint(errp, "sve-max-vq must come before all "
>>> +                          "sve<vl-bits> properties and it must only "
>>> +                          "be specified once.\n");
>>> +        return;
>>> +    }
>>> +
>>>      cpu->sve_max_vq = value;
>>>  
>>>      if (cpu->sve_max_vq == 0 || cpu->sve_max_vq > ARM_MAX_VQ) {
>>>          error_setg(errp, "unsupported SVE vector length");
>>>          error_append_hint(errp, "Valid sve-max-vq in range [1-%d]\n",
>>>                            ARM_MAX_VQ);
>>> +    } else {
>>> +        uint32_t vq;
>>> +
>>> +        for (vq = 1; vq <= cpu->sve_max_vq; ++vq) {
>>> +            char name[8];
>>> +            sprintf(name, "sve%d", vq * 128);
>>> +            object_property_set_bool(obj, true, name, &err);
>>> +            if (err) {
>>> +                error_propagate(errp, err);
>>> +                return;
>>> +            }
>>> +        }
>>> +    }
>>> +}
>>> +
>>> +static void cpu_arm_get_sve_vq(Object *obj, Visitor *v, const char *name,
>>> +                               void *opaque, Error **errp)
>>> +{
>>> +    ARMCPU *cpu = ARM_CPU(obj);
>>> +    int vq = atoi(&name[3]) / 128;
>>> +    arm_vq_state vq_state;
>>> +    bool value;
>>> +
>>> +    vq_state = arm_cpu_vq_map_get(cpu, vq);
>>> +
>>> +    if (!cpu->sve_max_vq) {
>>> +        /* All vector lengths are disabled when SVE is off. */
>>> +        value = false;
>>> +    } else if (vq_state == ARM_VQ_ON) {
>>> +        value = true;
>>> +    } else if (vq_state == ARM_VQ_OFF) {
>>> +        value = false;
>>> +    } else {
>>> +        /*
>>> +         * vq is uninitialized. We pick a default here based on the
>>> +         * the state of sve-max-vq and other sve<vl-bits> properties.
>>> +         */
>>> +        if (arm_sve_have_max_vq(cpu)) {
>>> +            /*
>>> +             * If we have sve-max-vq, then all remaining uninitialized
>>> +             * vq's are 'OFF'.
>>> +             */
>>> +            value = false;
>>> +        } else {
>>> +            switch (cpu->sve_max_vq) {
>>> +            case ARM_SVE_INIT:
>>> +            case ARM_VQ_DEFAULT_ON:
>>> +                value = true;
>>> +                break;
>>> +            case ARM_VQ_DEFAULT_OFF:
>>> +                value = false;
>>> +                break;
>>> +            }
>>> +        }
>>> +    }
>>> +
>>> +    visit_type_bool(v, name, &value, errp);
>>> +}
>>> +
>>> +static void cpu_arm_set_sve_vq(Object *obj, Visitor *v, const char *name,
>>> +                               void *opaque, Error **errp)
>>> +{
>>> +    ARMCPU *cpu = ARM_CPU(obj);
>>> +    int vq = atoi(&name[3]) / 128;
>>> +    arm_vq_state vq_state;
>>> +    Error *err = NULL;
>>> +    uint32_t max_vq = 0;
>>> +    bool value;
>>> +
>>> +    visit_type_bool(v, name, &value, &err);
>>> +    if (err) {
>>> +        error_propagate(errp, err);
>>> +        return;
>>> +    }
>>> +
>>> +    if (value && !cpu->sve_max_vq) {
>>> +        error_setg(errp, "cannot enable %s", name);
>>> +        error_append_hint(errp, "SVE has been disabled with sve=off\n");
>>> +        return;
>>> +    } else if (!cpu->sve_max_vq) {
>>> +        /*
>>> +         * We don't complain about disabling vector lengths when SVE
>>> +         * is off, but we don't do anything either.
>>> +         */
>>> +        return;
>>> +    }
>>> +
>>> +    if (arm_sve_have_max_vq(cpu)) {
>>> +        max_vq = cpu->sve_max_vq;
>>> +    } else {
>>> +        if (value) {
>>> +            cpu->sve_max_vq = ARM_VQ_DEFAULT_OFF;
>>> +        } else if (cpu->sve_max_vq != ARM_VQ_DEFAULT_OFF) {
>>> +            cpu->sve_max_vq = ARM_VQ_DEFAULT_ON;
>>> +        }
>>> +    }
>>> +
>>> +    /*
>>> +     * We need to know the maximum vector length, which may just currently
>>> +     * be the maximum length, in order to validate the enabling/disabling
>>> +     * of this vector length. We use the property get accessor in order to
>>> +     * get the appropriate default value for any uninitialized lengths.
>>> +     */
>>> +    if (!max_vq) {
>>> +        char tmp[8];
>>> +        bool s;
>>> +
>>> +        for (max_vq = ARM_MAX_VQ; max_vq >= 1; --max_vq) {
>>> +            sprintf(tmp, "sve%d", max_vq * 128);
>>> +            s = object_property_get_bool(OBJECT(cpu), tmp, &err);
>>> +            assert(!err);
>>> +            if (s) {
>>> +                break;
>>> +            }
>>> +        }
>>> +    }
>>> +
>>> +    if (arm_sve_have_max_vq(cpu) && value && vq > cpu->sve_max_vq) {
>>> +        error_setg(errp, "cannot enable %s", name);
>>> +        error_append_hint(errp, "vq=%d (%d bits) is larger than the "
>>> +                          "maximum vector length, sve-max-vq=%d "
>>> +                          "(%d bits)\n", vq, vq * 128,
>>> +                          cpu->sve_max_vq, cpu->sve_max_vq * 128);
>>> +    } else if (arm_sve_have_max_vq(cpu) && !value && vq == cpu->sve_max_vq) {
>>> +        error_setg(errp, "cannot disable %s", name);
>>> +        error_append_hint(errp, "The maximum vector length must be "
>>> +                          "enabled, sve-max-vq=%d (%d bits)\n",
>>> +                          cpu->sve_max_vq, cpu->sve_max_vq * 128);
>>> +    } else if (arm_sve_have_max_vq(cpu) && !value && vq < cpu->sve_max_vq &&
>>> +               is_power_of_2(vq)) {
>>> +        error_setg(errp, "cannot disable %s", name);
>>> +        error_append_hint(errp, "vq=%d (%d bits) is required as it is a "
>>> +                          "power-of-2 length smaller than the maximum, "
>>> +                          "sve-max-vq=%d (%d bits)\n", vq, vq * 128,
>>> +                          cpu->sve_max_vq, cpu->sve_max_vq * 128);
>>> +    } else if (!value && vq < max_vq && is_power_of_2(vq)) {
>>> +        error_setg(errp, "cannot disable %s", name);
>>> +        error_append_hint(errp, "Vector length %d-bits is required as it "
>>> +                          "is a power-of-2 length smaller than another "
>>> +                          "enabled vector length. Disable all larger vector "
>>> +                          "lengths first.\n", vq * 128);
>>> +    } else {
>> adding return in each if/elsif would allow to avoid this indent.
> 
> Yeah, but I didn't really mind the indent :-)
> 
>>> +        if (value) {
>>> +            bool fail = false;
>>> +            uint32_t s;
>>> +
>>> +            /*
>>> +             * Enabling a vector length automatically enables all
>>> +             * uninitialized power-of-2 lengths smaller than it, as
>>> +             * per the architecture.
>>> +             */
>> Test we are not attempting to enable a !is_power_of_2
> 
> I'm not sure what this means. In this context 'vq' can be a !power-of-2
> if it wants to be, and there's no way for 's' to be a !power-of-2 because
> we filter the vq's with is_power_of_2(s).
Yep got it now
> 
>>> +            for (s = 1; s < vq; ++s) {
>>> +                if (is_power_of_2(s)) {
>>> +                    vq_state = arm_cpu_vq_map_get(cpu, s);
>>> +                    if (vq_state == ARM_VQ_UNINITIALIZED) {
>>> +                        arm_cpu_vq_map_set(cpu, s, ARM_VQ_ON);
>>> +                    } else if (vq_state == ARM_VQ_OFF) {
>>> +                        fail = true;
>>> +                        break;
>>> +                    }
>>> +                }
>>> +            }
>>> +
>>> +            if (fail) {
>>> +                error_setg(errp, "cannot enable %s", name);
>>> +                error_append_hint(errp, "Vector length %d-bits is disabled "
>>> +                                  "and is a power-of-2 length smaller than "
>>> +                                  "%s. All power-of-2 vector lengths smaller "
>>> +                                  "than the maximum length are required.\n",
>>> +                                  s * 128, name);
>>> +            } else {
>>> +                arm_cpu_vq_map_set(cpu, vq, ARM_VQ_ON);
>>> +            }
>>> +        } else {
>>> +            arm_cpu_vq_map_set(cpu, vq, ARM_VQ_OFF);
>>> +        }
>>>      }
>>>  }
>>>  
>>> @@ -318,10 +652,11 @@ static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
>>>          /*
>>>           * We handle the -cpu <cpu>,sve=off,sve=on case by reinitializing,
>>>           * but otherwise we don't do anything as an sve=on could come after
>>> -         * a sve-max-vq setting.
>>> +         * a sve-max-vq or sve<vl-bits> setting.
>>>           */
>>>          if (!cpu->sve_max_vq) {
>>> -            cpu->sve_max_vq = ARM_MAX_VQ;
>>> +            cpu->sve_max_vq = ARM_SVE_INIT;
>>> +            arm_cpu_vq_map_init(cpu);
>>>          }
>>>      } else {
>>>          cpu->sve_max_vq = 0;
>>> @@ -336,6 +671,7 @@ static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
>>>  static void aarch64_max_initfn(Object *obj)
>>>  {
>>>      ARMCPU *cpu = ARM_CPU(obj);
>>> +    uint32_t vq;
>>>  
>>>      if (kvm_enabled()) {
>>>          kvm_arm_set_cpu_features_from_host(cpu);
>>> @@ -420,11 +756,29 @@ static void aarch64_max_initfn(Object *obj)
>>>          cpu->dcz_blocksize = 7; /*  512 bytes */
>>>  #endif
>>>  
>>> -        cpu->sve_max_vq = ARM_MAX_VQ;
>>> +        /*
>>> +         * sve_max_vq is initially unspecified, but must be initialized to a
>>> +         * non-zero value (ARM_SVE_INIT) to indicate that this cpu type has
>>> +         * SVE. It will be finalized in arm_cpu_realizefn().
>>> +         */
>>> +        cpu->sve_max_vq = ARM_SVE_INIT;
>>>          object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_max_vq,
>>>                              cpu_max_set_sve_max_vq, NULL, NULL, &error_fatal);
>>>          object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
>>>                              cpu_arm_set_sve, NULL, NULL, &error_fatal);
>>> +
>>> +        /*
>>> +         * sve_vq_map uses a special state while setting properties, so
>>> +         * we initialize it here with its init function and finalize it
>>> +         * in arm_cpu_realizefn().
>>> +         */
>>> +        arm_cpu_vq_map_init(cpu);
>>> +        for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
>>> +            char name[8];
>>> +            sprintf(name, "sve%d", vq * 128);
>>> +            object_property_add(obj, name, "bool", cpu_arm_get_sve_vq,
>>> +                                cpu_arm_set_sve_vq, NULL, NULL, &error_fatal);
>>> +        }
>>>      }
>>>  }
>>>  
>>> diff --git a/target/arm/helper.c b/target/arm/helper.c
>>> index f500ccb6d31b..b7b719dba57f 100644
>>> --- a/target/arm/helper.c
>>> +++ b/target/arm/helper.c
>>> @@ -5324,7 +5324,16 @@ static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
>>>  
>>>      /* Bits other than [3:0] are RAZ/WI.  */
>>>      QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
>>> -    raw_write(env, ri, value & 0xf);
>>> +    value &= 0xf;
>>> +
>>> +    if (value) {> +        /* get next vq that is smaller than or equal to value's vq */
>>> +        uint32_t vq = value + 1;
>> ditto
> 
> What does this 'ditto' map to? Oh, probably the discussion about vq's
> getting +1's and -1's. In this context 'value' is the ZCR_ELx
> representation of a VQ, which is VQ - 1, just like in the bitmap. To
> translate that to a VQ we need to do a '+ 1'.

Until here I follow. By the way in which doc did you find the
description of ZCR_ELx?

 As I wrote in the comment
> here, we want to find the next smaller or equal VQ here, because this is
> the input VQ from the guest which may itself be a valid VQ, but if it's
> not, we need to step down to the next valid one. So we ask
> arm_cpu_vq_map_next_smaller() for 'vq + 1' in order to ensure 'vq' will
> be in the searched range. After we get a valid vq from
> arm_cpu_vq_map_next_smaller(), which must be at least '1', since that's
> required by the architecture and thus will always be present, we need to
> translate it back to the ZCR_ELx representation with the subtraction.
> Phew... We programmers do love adding and subtracting by one, right :-)
Got it now. but well I scratched my head
> 
>>> +        vq = arm_cpu_vq_map_next_smaller(cpu, vq + 1);
>>> +        value = vq - 1;
>>> +    }
>>> +
>>> +    raw_write(env, ri, value);
>>>  
>>>      /*
>>>       * Because we arrived here, we know both FP and SVE are enabled;
>>> diff --git a/target/arm/monitor.c b/target/arm/monitor.c
>>> index 157c487a1551..1e213906fd8f 100644
>>> --- a/target/arm/monitor.c
>>> +++ b/target/arm/monitor.c
>>> @@ -89,8 +89,24 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)
>>>      return head;
>>>  }
>>>  
>>> +QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
>>> +
>>> +/*
>>> + * These are cpu model features we want to advertise. The order here
>>> + * matters as this is the order in which qmp_query_cpu_model_expansion
>>> + * will attempt to set them. If there are dependencies between features,
>>> + * as there are with the sve<vl-bits> features, then the order that
>>> + * considers those dependencies must be used.
>>> + *
>>> + * The sve<vl-bits> features need to be in reverse order in order to
>>> + * enable/disable the largest vector lengths first, ensuring all
>>> + * power-of-2 vector lengths smaller can also be enabled/disabled.
>>> + */
>>>  static const char *cpu_model_advertised_features[] = {
>>>      "aarch64", "pmu", "sve",
>>> +    "sve2048", "sve1920", "sve1792", "sve1664", "sve1536", "sve1408",
>>> +    "sve1280", "sve1152", "sve1024", "sve896", "sve768", "sve640",
>>> +    "sve512", "sve384", "sve256", "sve128",
>>>      NULL
>>>  };
>>>  
>>> diff --git a/tests/arm-cpu-features.c b/tests/arm-cpu-features.c
>> Please move all the tests in a separate patch.
> 
> I'd prefer not to. I like keeping the test cases that test the new code in
> the same patch. It self documents what the test cases map to what code and
> allows immediate testing of the patch when bisecting. Also I'm not really
> sure how it makes review worse, as another patch would look the same, just
> in a different email. 
A reviewier might not be familiar with both standard code and test and
feel reluctant to invest reading boths, hence reducing the chances to
collect R-b's. But that's my humble opinion.
> 
>> Each day has enough trouble of its own ;-) sweat...
> 
> I really appreciate the review! I realize it's generating sweat,
> especially with the European heat wave! You don't have to finish
> a patch before sending comments. I'm fine with multiple replies to
> the same patch if you want to break your review up into smaller
> bites.

Thanks. Yes breathing times are needed due to the heat and digestion of
the code ;-)

Thanks

Eric
> 
> Thanks,
> drew
> 
>>
>> Thanks
>>
>> Eric
>>> index 509e458e9c2f..a4bf6aec00df 100644
>>> --- a/tests/arm-cpu-features.c
>>> +++ b/tests/arm-cpu-features.c
>>> @@ -13,6 +13,18 @@
>>>  #include "qapi/qmp/qdict.h"
>>>  #include "qapi/qmp/qjson.h"
>>>  
>>> +#if __SIZEOF_LONG__ == 8
>>> +#define BIT(n) (1UL << (n))
>>> +#else
>>> +#define BIT(n) (1ULL << (n))
>>> +#endif
>>> +
>>> +/*
>>> + * We expect the SVE max-vq to be 16. Also it must be <= 64
>>> + * for our test code, otherwise 'vls' can't just be a uint64_t.
>>> + */
>>> +#define SVE_MAX_VQ 16
>>> +
>>>  #define MACHINE    "-machine virt,gic-version=max "
>>>  #define QUERY_HEAD "{ 'execute': 'query-cpu-model-expansion', " \
>>>                       "'arguments': { 'type': 'full', "
>>> @@ -137,6 +149,201 @@ static void assert_bad_props(QTestState *qts, const char *cpu_type)
>>>      qobject_unref(resp);
>>>  }
>>>  
>>> +static void resp_get_sve_vls(QDict *resp, uint64_t *vls, uint32_t *max_vq)
>>> +{
>>> +    const QDictEntry *e;
>>> +    QDict *qdict;
>>> +    int n = 0;
>>> +
>>> +    *vls = 0;
>>> +
>>> +    qdict = resp_get_props(resp);
>>> +
>>> +    for (e = qdict_first(qdict); e; e = qdict_next(qdict, e)) {
>>> +        if (strlen(e->key) > 3 && !strncmp(e->key, "sve", 3) &&
>>> +            g_ascii_isdigit(e->key[3])) {
>>> +            char *endptr;
>>> +            int bits;
>>> +
>>> +            bits = g_ascii_strtoll(&e->key[3], &endptr, 10);
>>> +            if (!bits || *endptr != '\0') {
>>> +                continue;
>>> +            }
>>> +
>>> +            if (qdict_get_bool(qdict, e->key)) {
>>> +                *vls |= BIT((bits / 128) - 1);
>>> +            }
>>> +            ++n;
>>> +        }
>>> +    }
>>> +
>>> +    g_assert(n == SVE_MAX_VQ);
>>> +
>>> +    *max_vq = !*vls ? 0 : 64 - __builtin_clzll(*vls);
>>> +}
>>> +
>>> +static uint64_t sve_get_vls(QTestState *qts, const char *cpu_type,
>>> +                            const char *fmt, ...)
>>> +{
>>> +    QDict *resp;
>>> +    uint64_t vls;
>>> +    uint32_t max_vq;
>>> +
>>> +    if (fmt) {
>>> +        QDict *args;
>>> +        va_list ap;
>>> +
>>> +        va_start(ap, fmt);
>>> +        args = qdict_from_vjsonf_nofail(fmt, ap);
>>> +        va_end(ap);
>>> +
>>> +        resp = qtest_qmp(qts, QUERY_HEAD "'model': { 'name': %s, "
>>> +                                                    "'props': %p }"
>>> +                              QUERY_TAIL, cpu_type, args);
>>> +    } else {
>>> +        resp = do_query_no_props(qts, cpu_type);
>>> +    }
>>> +
>>> +    g_assert(resp);
>>> +    resp_get_sve_vls(resp, &vls, &max_vq);
>>> +    qobject_unref(resp);
>>> +
>>> +    return vls;
>>> +}
>>> +
>>> +#define assert_sve_vls(qts, cpu_type, expected_vls, fmt, ...) \
>>> +    g_assert(sve_get_vls(qts, cpu_type, fmt, ##__VA_ARGS__) == expected_vls)
>>> +
>>> +static void sve_tests_default(QTestState *qts, const char *cpu_type)
>>> +{
>>> +    /*
>>> +     * With no sve-max-vq or sve<vl-bits> properties on the command line
>>> +     * the default is to have all vector lengths enabled.
>>> +     */
>>> +    assert_sve_vls(qts, cpu_type, BIT(SVE_MAX_VQ) - 1, NULL);
>>> +
>>> +    /*
>>> +     * -------------------------------------------------------------------
>>> +     *               power-of-2(vq)   all-power-            can      can
>>> +     *                                of-2(< vq)          enable   disable
>>> +     * -------------------------------------------------------------------
>>> +     * vq < max_vq      no            MUST*                yes      yes
>>> +     * vq < max_vq      yes           MUST*                yes      no
>>> +     * -------------------------------------------------------------------
>>> +     * vq == max_vq     n/a           MUST*                yes**    yes**
>>> +     * -------------------------------------------------------------------
>>> +     * vq > max_vq      n/a           no                   no       yes
>>> +     * vq > max_vq      n/a           yes                  yes      yes
>>> +     * -------------------------------------------------------------------
>>> +     *
>>> +     * [*] "MUST" means this requirement must already be satisfied,
>>> +     *     otherwise 'max_vq' couldn't itself be enabled.
>>> +     *
>>> +     * [**] Not testable with the QMP interface, only with the command line.
>>> +     */
>>> +
>>> +    /* max_vq := 8 */
>>> +    assert_sve_vls(qts, cpu_type, 0x8b, "{ 'sve1024': true }");
>>> +
>>> +    /* max_vq := 8, vq < max_vq, !power-of-2(vq) */
>>> +    assert_sve_vls(qts, cpu_type, 0x8f,
>>> +                   "{ 'sve1024': true, 'sve384': true }");
>>> +    assert_sve_vls(qts, cpu_type, 0x8b,
>>> +                   "{ 'sve1024': true, 'sve384': false }");
>>> +
>>> +    /* max_vq := 8, vq < max_vq, power-of-2(vq) */
>>> +    assert_sve_vls(qts, cpu_type, 0x8b,
>>> +                   "{ 'sve1024': true, 'sve256': true }");
>>> +    assert_error(qts, cpu_type, "cannot disable sve256",
>>> +                 "{ 'sve1024': true, 'sve256': false }");
>>> +
>>> +    /*
>>> +     * max_vq := 3, vq > max_vq, !all-power-of-2(< vq)
>>> +     *
>>> +     * If given sve384=on,sve512=off,sve640=on the command line error would be
>>> +     * "cannot enable sve640", but QMP visits the vector lengths in reverse
>>> +     * order, so we get "cannot disable sve512" instead. The command line would
>>> +     * also give that error if given sve384=on,sve640=on,sve512=off, so this is
>>> +     * all fine. The important thing is that we get an error.
>>> +     */
>>> +    assert_error(qts, cpu_type, "cannot disable sve512",
>>> +                 "{ 'sve384': true, 'sve512': false, 'sve640': true }");
>>> +
>>> +    /*
>>> +     * We can disable power-of-2 vector lengths when all larger lengths
>>> +     * are also disabled. The shorter, sve384=on,sve512=off,sve640=off
>>> +     * works on the command line, but QMP doesn't know that all the
>>> +     * vector lengths larger than 384-bits will be disabled until it
>>> +     * sees the enabling of sve384, which comes near the end since it
>>> +     * visits the lengths in reverse order. So we just have to explicitly
>>> +     * disable them all.
>>> +     */
>>> +    assert_sve_vls(qts, cpu_type, 0x7,
>>> +                   "{ 'sve384': true, 'sve512': false, 'sve640': false, "
>>> +                   "  'sve768': false, 'sve896': false, 'sve1024': false, "
>>> +                   "  'sve1152': false, 'sve1280': false, 'sve1408': false, "
>>> +                   "  'sve1536': false, 'sve1664': false, 'sve1792': false, "
>>> +                   "  'sve1920': false, 'sve2048': false }");
>>> +
>>> +    /* max_vq := 3, vq > max_vq, all-power-of-2(< vq) */
>>> +    assert_sve_vls(qts, cpu_type, 0x1f,
>>> +                   "{ 'sve384': true, 'sve512': true, 'sve640': true }");
>>> +    assert_sve_vls(qts, cpu_type, 0xf,
>>> +                   "{ 'sve384': true, 'sve512': true, 'sve640': false }");
>>> +}
>>> +
>>> +static void sve_tests_sve_max_vq_8(const void *data)
>>> +{
>>> +    QTestState *qts;
>>> +
>>> +    qts = qtest_init(MACHINE "-cpu max,sve-max-vq=8");
>>> +
>>> +    assert_sve_vls(qts, "max", BIT(8) - 1, NULL);
>>> +
>>> +    /*
>>> +     * Disabling the max-vq set by sve-max-vq is not allowed, but
>>> +     * of course enabling it is OK.
>>> +     */
>>> +    assert_error(qts, "max", "cannot disable sve1024", "{ 'sve1024': false }");
>>> +    assert_sve_vls(qts, "max", 0xff, "{ 'sve1024': true }");
>>> +
>>> +    /*
>>> +     * Enabling anything larger than max-vq set by sve-max-vq is not
>>> +     * allowed, but of course disabling everything larger is OK.
>>> +     */
>>> +    assert_error(qts, "max", "cannot enable sve1152", "{ 'sve1152': true }");
>>> +    assert_sve_vls(qts, "max", 0xff, "{ 'sve1152': false }");
>>> +
>>> +    /*
>>> +     * We can disable non power-of-2 lengths smaller than the max-vq
>>> +     * set by sve-max-vq, but not power-of-2 lengths.
>>> +     */
>>> +    assert_sve_vls(qts, "max", 0xfb, "{ 'sve384': false }");
>>> +    assert_error(qts, "max", "cannot disable sve256", "{ 'sve256': false }");
>>> +
>>> +    qtest_quit(qts);
>>> +}
>>> +
>>> +static void sve_tests_sve_off(const void *data)
>>> +{
>>> +    QTestState *qts;
>>> +
>>> +    qts = qtest_init(MACHINE "-cpu max,sve=off");
>>> +
>>> +    /*
>>> +     * SVE is off, so the map should be empty.
>>> +     */
>>> +    assert_sve_vls(qts, "max", 0, NULL);
>>> +
>>> +    /*
>>> +     * We can't turn anything on, but off is OK.
>>> +     */
>>> +    assert_error(qts, "max", "cannot enable sve128", "{ 'sve128': true }");
>>> +    assert_sve_vls(qts, "max", 0, "{ 'sve128': false }");
>>> +
>>> +    qtest_quit(qts);
>>> +}
>>> +
>>>  static void test_query_cpu_model_expansion(const void *data)
>>>  {
>>>      QTestState *qts;
>>> @@ -159,9 +366,12 @@ static void test_query_cpu_model_expansion(const void *data)
>>>      if (g_str_equal(qtest_get_arch(), "aarch64")) {
>>>          assert_has_feature(qts, "max", "aarch64");
>>>          assert_has_feature(qts, "max", "sve");
>>> +        assert_has_feature(qts, "max", "sve128");
>>>          assert_has_feature(qts, "cortex-a57", "pmu");
>>>          assert_has_feature(qts, "cortex-a57", "aarch64");
>>>  
>>> +        sve_tests_default(qts, "max");
>>> +
>>>          /* Test that features that depend on KVM generate errors without. */
>>>          assert_error(qts, "max",
>>>                       "'aarch64' feature cannot be disabled "
>>> @@ -213,6 +423,13 @@ int main(int argc, char **argv)
>>>      qtest_add_data_func("/arm/query-cpu-model-expansion",
>>>                          NULL, test_query_cpu_model_expansion);
>>>  
>>> +    if (g_str_equal(qtest_get_arch(), "aarch64")) {
>>> +        qtest_add_data_func("/arm/max/query-cpu-model-expansion/sve-max-vq-8",
>>> +                            NULL, sve_tests_sve_max_vq_8);
>>> +        qtest_add_data_func("/arm/max/query-cpu-model-expansion/sve-off",
>>> +                            NULL, sve_tests_sve_off);
>>> +    }
>>> +
>>>      if (kvm_available) {
>>>          qtest_add_data_func("/arm/kvm/query-cpu-model-expansion",
>>>                              NULL, test_query_cpu_model_expansion_kvm);
>>>
>>

Re: [Qemu-devel] [PATCH v2 07/14] target/arm/cpu64: max cpu: Introduce sve<vl-bits> properties

[Andrew Jones]

On Thu, Jun 27, 2019 at 12:51:10PM +0200, Auger Eric wrote:
> Hi Drew,
> On 6/27/19 11:40 AM, Andrew Jones wrote:
> > On Wed, Jun 26, 2019 at 04:58:05PM +0200, Auger Eric wrote:
> >> Hi Drew,
> >>
> >> On 6/21/19 6:34 PM, Andrew Jones wrote:
> >>> Introduce cpu properties to give fine control over SVE vector lengths.
> >>> We introduce a property for each valid length up to the current
> >>> maximum supported, which is 2048-bits. The properties are named, e.g.
> >>> sve128, sve256, sve512, ..., where the number is the number of bits.
> >> sve384 then in the natural order, otherwise it gives the impression you
> >> can only specify * 128bit pow of 2 sizes at this stage of the reading.
> > 
> > OK
> > 
> >>
> >>>
> >>> It's now possible to do something like -cpu max,sve-max-vq=4,sve384=off
> >> Document the fact sve512 cannot be turned to off, which sounds fully
> >> sensible (by reading the code). By the way, I think an actual
> >> documentation should be provided in qemu. Maybe as spec to agree on.
> > 
> > Actually, maybe I could just allow it to be disabled. It would be
> > a strange command line to do '-cpu max,sve-max-vq=4,sve512=off', but if
> > that's what the user wants, then there's not really any good reason to
> > block it. As I say below, mixing these types of properties on the command
> > line isn't really a good idea, but it's not completely blocked because we
> > want a user that prefers launching their (most likely TCG) guest with,
> > e.g.  '-cpu max,sve-max-vq=4', to also have a functional QMP CPU model
> > expansion, but the CPU model expansion for SVE vector lengths depends on
> > the sve<vl-bits> properties, thus mixing sve-max-vq with sve<vl-bits>,
> > where sve-max-vq comes first. They're just not mixed on the command line.
> OK
> > 
> >>> to provide a guest vector lengths 128, 256, and 512 bits. The resulting
> >>> set must conform to the architectural constraint of having all power-of-2
> >>> lengths smaller than the maximum length present. It's also possible to
> >>> only provide sve<vl-bits> properties, e.g. -cpu max,sve512=on> That example provides the machine with 128, 256, and 512 bit vector
> >> lengths.
> >>> It doesn't hurt to explicitly ask for all expected vector lengths,
> >>> which is what, for example, libvirt should do.>
> >>> Note1, it is not possible to use sve<vl-bits> properties before
> >>> sve-max-vq, e.g. -cpu max,sve384=off,sve-max-vq=4, as supporting
> >>> that overly complicates the user input validation.
> >>>
> >>> Note2, while one might expect -cpu max,sve-max-vq=4,sve512=on to be the
> >>> same as -cpu max,sve512=on, they are not.
> >> yep it is a bit weird
> >>
> >> Didn't you consider -cpu max,sve-max-vq=4,req_only=true removing non
> >> power of 2 values and sve<vl-bits> setting a single VLbit?
> > 
> > Nope. I mostly considered making sure sve-max-vq was supported to the
> > level necessary to work with the new properties, and to not change its
> > current semantics, but I don't want to extend it in any way, nor to
> > require it to be used with the new properties at all. sve-max-vq isn't
> > even going to be supported by '-cpu host', so we can't rely on it being
> > part of the SVE vector length selection for normal KVM guests.
> OK, as long as it is documented somewhere.
> > 
> > Keep in mind that the current semantics of sve-max-vq are to enable all
> > vector lengths up to and including that specified maximum. That's not
> > a safe way to select vector lengths on KVM, as it may include vector
> > lengths that the KVM host does not support, thus it's not something KVM
> > users should use. It's already there for the 'max' cpu type though, so
> > we work with it in this series the best we can for 'max', but not at all
> > for 'host'.
> Do you expect hosts to expose only a few of the permitted values? I
> imagined it would generally expose none or all of them actually. If you

You'd expect wrong then :-) There's already a real host out there that
implements SVE, but they don't implement any of the optional lengths.

> discourage people to use sve-max-vq and sve<>=on only sets 2^n values,
> userspace will need to set manually all intermediate !2^n values

For TCG, yes. But what I recommend userspace to do isn't just to set the
max with sve<max-vl-bits>=on and then all the optional vector lengths,
too, if they want them. I recommend they set *all* vector lengths they
want, e.g. 128,256,384,... even though some of them would be auto-enabled.
If you don't do that, then you don't have a truly migratable command line.
The auto-enabling of required vector lengths makes sense to me, though,
because the code has to detect it anyway, in order to error out when
they're not present. And doing so also gives developers, who may not care
about migration, a more concise command line.

> > 
> > Re: documentation for this. I suppose I could write something up that
> > clarifies the properties and also suggests best practices regarding
> > sve-max-vq.
> 
> To me this is really helpful to have a global understanding
> > 
> >>> The former enables all vector lengths 512 bits and smaller
> >> ( required and permitted)
> >>> while the latter only sets the 512-bit
> >>> length and its smaller power-of-2 lengths. It's probably best not to use
> >>> sve-max-vq with sve<vl-bits> properties, but it can't be completely
> >>> forbidden as we want qmp_query_cpu_model_expansion to work with guests
> >>> launched with e.g. -cpu max,sve-max-vq=8 on their command line.
> >> what does happen if you specify -cpu max,sve384=on? (seems to be allowed
> >> in the code?)
> > 
> > That's perfectly valid with tcg, you'll get 128,256,384. It's also valid
> > with KVM if you're host supports it.
> OK so it exposes the maximum configurable vector length + all the
> corresponding required values.

Correct

> > 
> >>
> >>>
> >>> Signed-off-by: Andrew Jones <drjones@redhat.com>
> >>> ---
> >>>  target/arm/cpu.c         |   6 +
> >>>  target/arm/cpu.h         |  14 ++
> >>>  target/arm/cpu64.c       | 360 ++++++++++++++++++++++++++++++++++++++-
> >>>  target/arm/helper.c      |  11 +-
> >>>  target/arm/monitor.c     |  16 ++
> >>>  tests/arm-cpu-features.c | 217 +++++++++++++++++++++++
> >>>  6 files changed, 620 insertions(+), 4 deletions(-)
> >>>
> >>> diff --git a/target/arm/cpu.c b/target/arm/cpu.c
> >>> index f08e178fc84b..e060a0d9df0e 100644
> >>> --- a/target/arm/cpu.c
> >>> +++ b/target/arm/cpu.c
> >>> @@ -1019,6 +1019,12 @@ static void arm_cpu_realizefn(DeviceState *dev, Error **errp)
> >>>          return;
> >>>      }
> >>>  
> >>> +    arm_cpu_sve_finalize(cpu, &local_err);
> >>> +    if (local_err) {
> >>> +        error_propagate(errp, local_err);
> >>> +        return;
> >>> +    }
> >>> +
> >>>      if (arm_feature(env, ARM_FEATURE_AARCH64) &&
> >>>          cpu->has_vfp != cpu->has_neon) {
> >>>          /*
> >>> diff --git a/target/arm/cpu.h b/target/arm/cpu.h
> >>> index f9da672be575..cbb155cf72a5 100644
> >>> --- a/target/arm/cpu.h
> >>> +++ b/target/arm/cpu.h
> >>> @@ -184,8 +184,13 @@ typedef struct {
> >>>  
> >>>  #ifdef TARGET_AARCH64
> >>>  # define ARM_MAX_VQ    16
> >>> +void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp);
> >>> +uint32_t arm_cpu_vq_map_next_smaller(ARMCPU *cpu, uint32_t vq);
> >>>  #else
> >>>  # define ARM_MAX_VQ    1
> >>> +static inline void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp) { }
> >>> +static inline uint32_t arm_cpu_vq_map_next_smaller(ARMCPU *cpu, uint32_t vq)
> >>> +{ return 0; }
> >>>  #endif
> >>>  
> >>>  typedef struct ARMVectorReg {
> >>> @@ -915,6 +920,15 @@ struct ARMCPU {
> >>>  
> >>>      /* Used to set the maximum vector length the cpu will support.  */
> >>>      uint32_t sve_max_vq;
> >>> +
> >>> +    /*
> >>> +     * In sve_vq_map each set bit is a supported vector length of
> >>> +     * (bit-number + 1) * 16 bytes, i.e. each bit number + 1 is the vector> +     * length in quadwords. We need a map size twice the maximum
> >>> +     * quadword length though because we use two bits for each vector
> >>> +     * length in order to track three states: uninitialized, off, and on.
> >>> +     */
> >>> +    DECLARE_BITMAP(sve_vq_map, ARM_MAX_VQ * 2);
> >>>  };
> >>>  
> >>>  void arm_cpu_post_init(Object *obj);
> >>> diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
> >>> index 02ada65f240c..5def82111dee 100644
> >>> --- a/target/arm/cpu64.c
> >>> +++ b/target/arm/cpu64.c
> >>> @@ -257,6 +257,149 @@ static void aarch64_a72_initfn(Object *obj)
> >>>      define_arm_cp_regs(cpu, cortex_a72_a57_a53_cp_reginfo);
> >>>  }
> >>>  
> >>> +/*
> >>> + * While we eventually use cpu->sve_vq_map as a typical bitmap, where each vq
> >>> + * has only two states (off/on), until we've finalized the map at realize time
> >>> + * we use an extra bit, at the vq - 1 + ARM_MAX_VQ bit number, to also allow
> >>> + * tracking of the uninitialized state. The arm_vq_state typedef and following
> >>> + * functions allow us to more easily work with the bitmap. Also, while the map
> >>> + * is still initializing, sve-max-vq has an additional three states, bringing
> >>> + * the number of its states to five, which are the following:
> >>> + *
> >>> + * sve-max-vq:
> >>> + *   0:    SVE is disabled. The default value for a vq in the map is 'OFF'.
> >>> + *  -1:    SVE is enabled, but neither sve-max-vq nor sve<vl-bits> properties
> >>> + *         have yet been specified by the user. The default value for a vq in
> >>> + *         the map is 'ON'.
> >>> + *  -2:    SVE is enabled and one or more sve<vl-bits> properties have been
> >>> + *         set to 'OFF' by the user, but no sve<vl-bits> properties have yet
> >>> + *         been set to 'ON'. The user is now blocked from setting sve-max-vq
> >>> + *         and the default value for a vq in the map is 'ON'.
> >>> + *  -3:    SVE is enabled and one or more sve<vl-bits> properties have been
> >>> + *         set to 'ON' by the user. The user is blocked from setting sve-max-vq
> >>> + *         and the default value for a vq in the map is 'OFF'. sve-max-vq never
> >>> + *         transitions back to -2, even if later inputs disable the vector
> >>> + *         lengths that initially transitioned sve-max-vq to this state. This
> >>> + *         avoids the default values from flip-flopping.
> >>> + *  [1-ARM_MAX_VQ]: SVE is enabled and the user has specified a valid
> >>> + *                  sve-max-vq. The sve-max-vq specified vq and all smaller
> >>> + *                  vq's will be initially enabled. All larger vq's will have
> >>> + *                  a default of 'OFF'.
> >>> + */
> >>> +#define ARM_SVE_INIT          -1
> >>> +#define ARM_VQ_DEFAULT_ON     -2
> >>> +#define ARM_VQ_DEFAULT_OFF    -3
> >>> +
> >>> +#define arm_sve_have_max_vq(cpu) ((int32_t)(cpu)->sve_max_vq > 0)
> >>> +
> >>> +typedef enum arm_vq_state {
> >>> +    ARM_VQ_OFF,
> >>> +    ARM_VQ_ON,
> >>> +    ARM_VQ_UNINITIALIZED,
> >>> +} arm_vq_state;
> >>> +
> >>> +static arm_vq_state arm_cpu_vq_map_get(ARMCPU *cpu, int vq)> +{
> >>> +    assert(vq <= ARM_MAX_VQ);
> >>> +
> >>> +    return test_bit(vq - 1, cpu->sve_vq_map) |
> >>> +           test_bit(vq - 1 + ARM_MAX_VQ, cpu->sve_vq_map) << 1;
> >>> +}> +
> >>> +static void arm_cpu_vq_map_set(ARMCPU *cpu, int vq, arm_vq_state state)
> >>> +{
> >>> +    assert(state == ARM_VQ_OFF || state == ARM_VQ_ON);
> >>> +    assert(vq <= ARM_MAX_VQ);
> >>> +
> >>> +    clear_bit(vq - 1 + ARM_MAX_VQ, cpu->sve_vq_map);
> >>> +
> >>> +    if (state == ARM_VQ_ON) {
> >>> +        set_bit(vq - 1, cpu->sve_vq_map);
> >>> +    } else {
> >>> +        clear_bit(vq - 1, cpu->sve_vq_map);
> >>> +    }
> >>> +}
> >>> +
> >>> +static void arm_cpu_vq_map_init(ARMCPU *cpu)
> >>> +{
> >>> +    bitmap_zero(cpu->sve_vq_map, ARM_MAX_VQ * 2);
> >> nit: bitmap_clear(map, 0, ARM_MAX_VQ);
> > 
> > bitmap_clear will only clear the specified bits, leaving the upper bits
> > uninitialized, which could cause problems. It's not a problem here
> > because we're using a zero initialized cpu state member, but if it was
> > a bitmap like below, declared on the stack, then we need the zeroing at
> > least once. I'd prefer to use zero here too, to keep it consistent.
> Sorry I don't get it. I would have expected the above bitmap_clear would
> 0 initialize 0 - ARM_MAX_VQ-1 bits and the bitmap_set below would
> initialize ARM_MAX_VQ - 2 * ARM_MAX_VQ -1 with 1's?

That's true, but what about the bits > 2 * ARM_MAX_VQ - 1? The bitmap is
implemented as an array of longs, so if you don't use a multiple of
sizeof(long)*8 bits, then you're going to have uninitialized parts of
your bitmap. It's safest to use bitmap_clear() for the whole thing at
least once. That said, I'll agree it's a bit subtle and maybe not even
the right way to approach it. It might be better just to audit all the
uses to ensure there's no chance of misinterpretation of unused one
bits > 2 * ARM_MAX_VQ - 1. We could possibly sprinkle more asserts around
to maintain that.

> 
> > 
> >> /* all VLs OFF */
> >>> +    bitmap_set(cpu->sve_vq_map, ARM_MAX_VQ, ARM_MAX_VQ);
> >> /* all VLs uninitialized */
> > 
> > I can add one comment that says
> > 
> > "Set all VQ's to ARM_VQ_UNINITIALIZED" above both lines.
> > 
> >>> +}
> >>> +
> >>> +static bool arm_cpu_vq_map_is_finalized(ARMCPU *cpu)
> >>> +{
> >>> +    DECLARE_BITMAP(map, ARM_MAX_VQ * 2);
> >>> +
> >>> +    bitmap_zero(map, ARM_MAX_VQ * 2);
> >> same
> > 
> > Here me must use bitmap_zero.
> > 
> >>> +    bitmap_set(map, ARM_MAX_VQ, ARM_MAX_VQ);
> >>> +    bitmap_and(map, map, cpu->sve_vq_map, ARM_MAX_VQ * 2);
> >>> +
> >>> +    return bitmap_empty(map, ARM_MAX_VQ * 2);
> >>> +}
> >>> +
> >>> +static void arm_cpu_vq_map_finalize(ARMCPU *cpu)
> >>> +{
> >>> +    Error *err = NULL;
> >>> +    char name[8];
> >>> +    uint32_t vq;
> >>> +    bool value;
> >>> +
> >>> +    /*
> >>> +     * We use the property get accessor because it knows what default
> >>> +     * values to return for uninitialized vector lengths.
> >>> +     */
> >>> +    for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
> >>> +        sprintf(name, "sve%d", vq * 128);
> >>> +        value = object_property_get_bool(OBJECT(cpu), name, &err);
> >>> +        assert(!err);
> >>> +        if (value) {
> >>> +            arm_cpu_vq_map_set(cpu, vq, ARM_VQ_ON);
> >>> +        } else {
> >>> +            arm_cpu_vq_map_set(cpu, vq, ARM_VQ_OFF);
> >>> +        }
> >>> +    }
> >>> +
> >>> +    assert(arm_cpu_vq_map_is_finalized(cpu));
> >> this can be removed
> > 
> > Yes, it can be today, as the code works fine. The idea was that if
> > somebody came in here and modified this in someway that broke the
> > finalized state, then we'd catch it here before going off and doing
> > weird things. This isn't a hot path, so I'd prefer we keep it, but
> > if QEMU maintainers prefer to limit defensive code, then I can
> > certainly remove it.
> Well I understand this was useful was developing and to check some
> tricky states but some of the asserts correspond to some checks that
> look rather obvious, like this one. But well I let others give their
> opinions and it is not a big deal either. Then we can wonder when one
> needs a trace mesg before asserting ...

OK, this one is probably fine to remove.

> > 
> >>> +}
> >>> +
> >>> +void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp)
> >>> +{
> >>> +    Error *err = NULL;
> >>> +
> >>> +    if (!cpu->sve_max_vq) {
> >>> +        bitmap_zero(cpu->sve_vq_map, ARM_MAX_VQ * 2);
> >>> +        return;
> >>> +    }
> >>> +
> >>> +    if (cpu->sve_max_vq == ARM_SVE_INIT) {
> >>> +        object_property_set_uint(OBJECT(cpu), ARM_MAX_VQ, "sve-max-vq", &err);
> >>> +        if (err) {
> >>> +            error_propagate(errp, err);
> >>> +            return;
> >>> +        }
> >>> +        assert(cpu->sve_max_vq == ARM_MAX_VQ);
> >> I guess those asserts can be removed now?
> > 
> > I'd prefer to keep it. It's critical that we get this right, so if
> > somebody changes something somewhere in the property set/get code
> > that would break this, then it's best we know right away. Again,
> > though, if there's some reason to limit defensive code, even on the
> > init path, then I can.
> Here I would expect that if the set did not fail, sve-max-vq effectively
> if set to the expected value. But maybe I don't remember the cases where
> sve_max-vq can be set to some other value without reprting an error.

No, you're right, no other value can occur if the 'set' doesn't fail. I
suppose it's 100% safe to remove this assert, but then we'd likely want
to state in a comment that sve_max_vq is now ARM_MAX_VQ, just to make
that clear. Besides catching issues early, asserts are good for
documenting.

> 
> > 
> >>> +        arm_cpu_vq_map_finalize(cpu);
> >> move the arm_cpu_vq_map_finalize out of the if, at the end.
> > 
> > That wouldn't work. In this branch arm_cpu_vq_map_finalize() comes
> > after setting cpu->sve_max_vq. In the else branch it must come first.
> That's right sorry
> > 
> >>> +    } else {
> >>> +        arm_cpu_vq_map_finalize(cpu);
> >>> +        if (!arm_sve_have_max_vq(cpu)) {
> >>> +            cpu->sve_max_vq = arm_cpu_vq_map_next_smaller(cpu, ARM_MAX_VQ + 1);
> >>> +        }
> >>> +    }
> >>> +
> >>> +    assert(cpu->sve_max_vq == arm_cpu_vq_map_next_smaller(cpu, ARM_MAX_VQ));>
> > What happened to my '+ 1' here? I see it's in the patch, but somehow got
> > removed in your quote of the patch? For a second there I thought something
> > was really wrong, since that assert should have fired for every full map.
> Yep sorry for the cut :-(
> > 
> >> same here
> > 
> > Anyway my same argument that we don't want to leave arm_cpu_sve_finalize()
> > without knowing we got this right applies.
> 
> > 
> >>> +}
> >>> +
> >>> +uint32_t arm_cpu_vq_map_next_smaller(ARMCPU *cpu, uint32_t vq)
> >>> +{
> >>> +    uint32_t bitnum;
> >>> +
> >>> +    assert(vq <= ARM_MAX_VQ + 1);
> >>> +    assert(arm_cpu_vq_map_is_finalized(cpu));
> >>> +
> >>> +    bitnum = find_last_bit(cpu->sve_vq_map, vq - 1);
> >> why do you pass ARM_MAX_VQ + 1 and then do vq -1? doesn't
> >> find_last_bit() take the size which is ARM_MAX_VQ in this case?
> > 
> > The size is ARM_MAX_VQ + 1, when you want to also check the bitnum
> > corresponding to ARM_MAX_VQ. The bitnum for a VQ is always VQ - 1.
> > Recall VQ=1, which is 128-bits. It takes the bit position zero.
> > VQ=ARM_MAX_VQ=16 takes the bit position 15. As for the 'vq - 1' here
> > in the find_last_bit() call, that's required because that's what the
> > function says it does. It finds the next smaller VQ. So if you pass
> > in, for example, vq=2, it shouldn't search anything but bit position
> > zero:
> > 
> >  vq=2 => max next smaller is vq=1, bitnum = 0
> >       => search the bitmap of size 1 for the last set bit
> > 
> > Or, if you want to search the whole map, including the maximum
> > possibly VQ, as is done above, then you need to pass ARM_MAX_VQ + 1,
> > since the max next smaller VQ is ARM_MAX_VQ.
> OK I get it now. Maybe renaming the function into something like
> arm_cpu_vq_map_last_bit() would have avoided that. the first assert
> looks strange typically.

I'm not sure "last bit" is descriptive enough. Does it mean last bit
not including the VQ passed in? Or does it include the passed in VQ?
I prefer "next smaller". I'll add a comment above the first assert.

> > 
> >>> +    return bitnum == vq - 1 ? 0 : bitnum + 1;
> >>> +}
> >>> +
> >>>  static void cpu_max_get_sve_max_vq(Object *obj, Visitor *v, const char *name,
> >>>                                     void *opaque, Error **errp)
> >>>  {
> >>> @@ -283,12 +426,203 @@ static void cpu_max_set_sve_max_vq(Object *obj, Visitor *v, const char *name,
> >>>          return;
> >>>      }
> >>>  
> >>> +    /*
> >>> +     * It gets complicated trying to support both sve-max-vq and
> >>> +     * sve<vl-bits> properties together, so we mostly don't. We
> >>> +     * do allow both if sve-max-vq is specified first and only once
> >>> +     * though.
> >>> +     */
> >>> +    if (cpu->sve_max_vq != ARM_SVE_INIT) {
> >>> +        error_setg(errp, "sve<vl-bits> in use or sve-max-vq already "
> >>> +                   "specified");
> >>> +        error_append_hint(errp, "sve-max-vq must come before all "
> >>> +                          "sve<vl-bits> properties and it must only "
> >>> +                          "be specified once.\n");
> >>> +        return;
> >>> +    }
> >>> +
> >>>      cpu->sve_max_vq = value;
> >>>  
> >>>      if (cpu->sve_max_vq == 0 || cpu->sve_max_vq > ARM_MAX_VQ) {
> >>>          error_setg(errp, "unsupported SVE vector length");
> >>>          error_append_hint(errp, "Valid sve-max-vq in range [1-%d]\n",
> >>>                            ARM_MAX_VQ);
> >>> +    } else {
> >>> +        uint32_t vq;
> >>> +
> >>> +        for (vq = 1; vq <= cpu->sve_max_vq; ++vq) {
> >>> +            char name[8];
> >>> +            sprintf(name, "sve%d", vq * 128);
> >>> +            object_property_set_bool(obj, true, name, &err);
> >>> +            if (err) {
> >>> +                error_propagate(errp, err);
> >>> +                return;
> >>> +            }
> >>> +        }
> >>> +    }
> >>> +}
> >>> +
> >>> +static void cpu_arm_get_sve_vq(Object *obj, Visitor *v, const char *name,
> >>> +                               void *opaque, Error **errp)
> >>> +{
> >>> +    ARMCPU *cpu = ARM_CPU(obj);
> >>> +    int vq = atoi(&name[3]) / 128;
> >>> +    arm_vq_state vq_state;
> >>> +    bool value;
> >>> +
> >>> +    vq_state = arm_cpu_vq_map_get(cpu, vq);
> >>> +
> >>> +    if (!cpu->sve_max_vq) {
> >>> +        /* All vector lengths are disabled when SVE is off. */
> >>> +        value = false;
> >>> +    } else if (vq_state == ARM_VQ_ON) {
> >>> +        value = true;
> >>> +    } else if (vq_state == ARM_VQ_OFF) {
> >>> +        value = false;
> >>> +    } else {
> >>> +        /*
> >>> +         * vq is uninitialized. We pick a default here based on the
> >>> +         * the state of sve-max-vq and other sve<vl-bits> properties.
> >>> +         */
> >>> +        if (arm_sve_have_max_vq(cpu)) {
> >>> +            /*
> >>> +             * If we have sve-max-vq, then all remaining uninitialized
> >>> +             * vq's are 'OFF'.
> >>> +             */
> >>> +            value = false;
> >>> +        } else {
> >>> +            switch (cpu->sve_max_vq) {
> >>> +            case ARM_SVE_INIT:
> >>> +            case ARM_VQ_DEFAULT_ON:
> >>> +                value = true;
> >>> +                break;
> >>> +            case ARM_VQ_DEFAULT_OFF:
> >>> +                value = false;
> >>> +                break;
> >>> +            }
> >>> +        }
> >>> +    }
> >>> +
> >>> +    visit_type_bool(v, name, &value, errp);
> >>> +}
> >>> +
> >>> +static void cpu_arm_set_sve_vq(Object *obj, Visitor *v, const char *name,
> >>> +                               void *opaque, Error **errp)
> >>> +{
> >>> +    ARMCPU *cpu = ARM_CPU(obj);
> >>> +    int vq = atoi(&name[3]) / 128;
> >>> +    arm_vq_state vq_state;
> >>> +    Error *err = NULL;
> >>> +    uint32_t max_vq = 0;
> >>> +    bool value;
> >>> +
> >>> +    visit_type_bool(v, name, &value, &err);
> >>> +    if (err) {
> >>> +        error_propagate(errp, err);
> >>> +        return;
> >>> +    }
> >>> +
> >>> +    if (value && !cpu->sve_max_vq) {
> >>> +        error_setg(errp, "cannot enable %s", name);
> >>> +        error_append_hint(errp, "SVE has been disabled with sve=off\n");
> >>> +        return;
> >>> +    } else if (!cpu->sve_max_vq) {
> >>> +        /*
> >>> +         * We don't complain about disabling vector lengths when SVE
> >>> +         * is off, but we don't do anything either.
> >>> +         */
> >>> +        return;
> >>> +    }
> >>> +
> >>> +    if (arm_sve_have_max_vq(cpu)) {
> >>> +        max_vq = cpu->sve_max_vq;
> >>> +    } else {
> >>> +        if (value) {
> >>> +            cpu->sve_max_vq = ARM_VQ_DEFAULT_OFF;
> >>> +        } else if (cpu->sve_max_vq != ARM_VQ_DEFAULT_OFF) {
> >>> +            cpu->sve_max_vq = ARM_VQ_DEFAULT_ON;
> >>> +        }
> >>> +    }
> >>> +
> >>> +    /*
> >>> +     * We need to know the maximum vector length, which may just currently
> >>> +     * be the maximum length, in order to validate the enabling/disabling
> >>> +     * of this vector length. We use the property get accessor in order to
> >>> +     * get the appropriate default value for any uninitialized lengths.
> >>> +     */
> >>> +    if (!max_vq) {
> >>> +        char tmp[8];
> >>> +        bool s;
> >>> +
> >>> +        for (max_vq = ARM_MAX_VQ; max_vq >= 1; --max_vq) {
> >>> +            sprintf(tmp, "sve%d", max_vq * 128);
> >>> +            s = object_property_get_bool(OBJECT(cpu), tmp, &err);
> >>> +            assert(!err);
> >>> +            if (s) {
> >>> +                break;
> >>> +            }
> >>> +        }
> >>> +    }
> >>> +
> >>> +    if (arm_sve_have_max_vq(cpu) && value && vq > cpu->sve_max_vq) {
> >>> +        error_setg(errp, "cannot enable %s", name);
> >>> +        error_append_hint(errp, "vq=%d (%d bits) is larger than the "
> >>> +                          "maximum vector length, sve-max-vq=%d "
> >>> +                          "(%d bits)\n", vq, vq * 128,
> >>> +                          cpu->sve_max_vq, cpu->sve_max_vq * 128);
> >>> +    } else if (arm_sve_have_max_vq(cpu) && !value && vq == cpu->sve_max_vq) {
> >>> +        error_setg(errp, "cannot disable %s", name);
> >>> +        error_append_hint(errp, "The maximum vector length must be "
> >>> +                          "enabled, sve-max-vq=%d (%d bits)\n",
> >>> +                          cpu->sve_max_vq, cpu->sve_max_vq * 128);
> >>> +    } else if (arm_sve_have_max_vq(cpu) && !value && vq < cpu->sve_max_vq &&
> >>> +               is_power_of_2(vq)) {
> >>> +        error_setg(errp, "cannot disable %s", name);
> >>> +        error_append_hint(errp, "vq=%d (%d bits) is required as it is a "
> >>> +                          "power-of-2 length smaller than the maximum, "
> >>> +                          "sve-max-vq=%d (%d bits)\n", vq, vq * 128,
> >>> +                          cpu->sve_max_vq, cpu->sve_max_vq * 128);
> >>> +    } else if (!value && vq < max_vq && is_power_of_2(vq)) {
> >>> +        error_setg(errp, "cannot disable %s", name);
> >>> +        error_append_hint(errp, "Vector length %d-bits is required as it "
> >>> +                          "is a power-of-2 length smaller than another "
> >>> +                          "enabled vector length. Disable all larger vector "
> >>> +                          "lengths first.\n", vq * 128);
> >>> +    } else {
> >> adding return in each if/elsif would allow to avoid this indent.
> > 
> > Yeah, but I didn't really mind the indent :-)
> > 
> >>> +        if (value) {
> >>> +            bool fail = false;
> >>> +            uint32_t s;
> >>> +
> >>> +            /*
> >>> +             * Enabling a vector length automatically enables all
> >>> +             * uninitialized power-of-2 lengths smaller than it, as
> >>> +             * per the architecture.
> >>> +             */
> >> Test we are not attempting to enable a !is_power_of_2
> > 
> > I'm not sure what this means. In this context 'vq' can be a !power-of-2
> > if it wants to be, and there's no way for 's' to be a !power-of-2 because
> > we filter the vq's with is_power_of_2(s).
> Yep got it now
> > 
> >>> +            for (s = 1; s < vq; ++s) {
> >>> +                if (is_power_of_2(s)) {
> >>> +                    vq_state = arm_cpu_vq_map_get(cpu, s);
> >>> +                    if (vq_state == ARM_VQ_UNINITIALIZED) {
> >>> +                        arm_cpu_vq_map_set(cpu, s, ARM_VQ_ON);
> >>> +                    } else if (vq_state == ARM_VQ_OFF) {
> >>> +                        fail = true;
> >>> +                        break;
> >>> +                    }
> >>> +                }
> >>> +            }
> >>> +
> >>> +            if (fail) {
> >>> +                error_setg(errp, "cannot enable %s", name);
> >>> +                error_append_hint(errp, "Vector length %d-bits is disabled "
> >>> +                                  "and is a power-of-2 length smaller than "
> >>> +                                  "%s. All power-of-2 vector lengths smaller "
> >>> +                                  "than the maximum length are required.\n",
> >>> +                                  s * 128, name);
> >>> +            } else {
> >>> +                arm_cpu_vq_map_set(cpu, vq, ARM_VQ_ON);
> >>> +            }
> >>> +        } else {
> >>> +            arm_cpu_vq_map_set(cpu, vq, ARM_VQ_OFF);
> >>> +        }
> >>>      }
> >>>  }
> >>>  
> >>> @@ -318,10 +652,11 @@ static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
> >>>          /*
> >>>           * We handle the -cpu <cpu>,sve=off,sve=on case by reinitializing,
> >>>           * but otherwise we don't do anything as an sve=on could come after
> >>> -         * a sve-max-vq setting.
> >>> +         * a sve-max-vq or sve<vl-bits> setting.
> >>>           */
> >>>          if (!cpu->sve_max_vq) {
> >>> -            cpu->sve_max_vq = ARM_MAX_VQ;
> >>> +            cpu->sve_max_vq = ARM_SVE_INIT;
> >>> +            arm_cpu_vq_map_init(cpu);
> >>>          }
> >>>      } else {
> >>>          cpu->sve_max_vq = 0;
> >>> @@ -336,6 +671,7 @@ static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
> >>>  static void aarch64_max_initfn(Object *obj)
> >>>  {
> >>>      ARMCPU *cpu = ARM_CPU(obj);
> >>> +    uint32_t vq;
> >>>  
> >>>      if (kvm_enabled()) {
> >>>          kvm_arm_set_cpu_features_from_host(cpu);
> >>> @@ -420,11 +756,29 @@ static void aarch64_max_initfn(Object *obj)
> >>>          cpu->dcz_blocksize = 7; /*  512 bytes */
> >>>  #endif
> >>>  
> >>> -        cpu->sve_max_vq = ARM_MAX_VQ;
> >>> +        /*
> >>> +         * sve_max_vq is initially unspecified, but must be initialized to a
> >>> +         * non-zero value (ARM_SVE_INIT) to indicate that this cpu type has
> >>> +         * SVE. It will be finalized in arm_cpu_realizefn().
> >>> +         */
> >>> +        cpu->sve_max_vq = ARM_SVE_INIT;
> >>>          object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_max_vq,
> >>>                              cpu_max_set_sve_max_vq, NULL, NULL, &error_fatal);
> >>>          object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
> >>>                              cpu_arm_set_sve, NULL, NULL, &error_fatal);
> >>> +
> >>> +        /*
> >>> +         * sve_vq_map uses a special state while setting properties, so
> >>> +         * we initialize it here with its init function and finalize it
> >>> +         * in arm_cpu_realizefn().
> >>> +         */
> >>> +        arm_cpu_vq_map_init(cpu);
> >>> +        for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
> >>> +            char name[8];
> >>> +            sprintf(name, "sve%d", vq * 128);
> >>> +            object_property_add(obj, name, "bool", cpu_arm_get_sve_vq,
> >>> +                                cpu_arm_set_sve_vq, NULL, NULL, &error_fatal);
> >>> +        }
> >>>      }
> >>>  }
> >>>  
> >>> diff --git a/target/arm/helper.c b/target/arm/helper.c
> >>> index f500ccb6d31b..b7b719dba57f 100644
> >>> --- a/target/arm/helper.c
> >>> +++ b/target/arm/helper.c
> >>> @@ -5324,7 +5324,16 @@ static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
> >>>  
> >>>      /* Bits other than [3:0] are RAZ/WI.  */
> >>>      QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
> >>> -    raw_write(env, ri, value & 0xf);
> >>> +    value &= 0xf;
> >>> +
> >>> +    if (value) {> +        /* get next vq that is smaller than or equal to value's vq */
> >>> +        uint32_t vq = value + 1;
> >> ditto
> > 
> > What does this 'ditto' map to? Oh, probably the discussion about vq's
> > getting +1's and -1's. In this context 'value' is the ZCR_ELx
> > representation of a VQ, which is VQ - 1, just like in the bitmap. To
> > translate that to a VQ we need to do a '+ 1'.
> 
> Until here I follow. By the way in which doc did you find the
> description of ZCR_ELx?

It definitely wasn't "ARM Architecture Reference Manual Supplement The
Scalable Vector Extension (SVE), for ARMv8-A". I'm not being sarcastic!
That manual *should* describe the register, and it *does* list the
register in section 6.1.2 SVE System registers, but there is *no* register
description in the document...

My knowledge of ZCR_ELx comes from reading kernel and qemu code. And
actually seeing it work in practice on real hardware.

> 
>  As I wrote in the comment
> > here, we want to find the next smaller or equal VQ here, because this is
> > the input VQ from the guest which may itself be a valid VQ, but if it's
> > not, we need to step down to the next valid one. So we ask
> > arm_cpu_vq_map_next_smaller() for 'vq + 1' in order to ensure 'vq' will
> > be in the searched range. After we get a valid vq from
> > arm_cpu_vq_map_next_smaller(), which must be at least '1', since that's
> > required by the architecture and thus will always be present, we need to
> > translate it back to the ZCR_ELx representation with the subtraction.
> > Phew... We programmers do love adding and subtracting by one, right :-)
> Got it now. but well I scratched my head
> > 
> >>> +        vq = arm_cpu_vq_map_next_smaller(cpu, vq + 1);
> >>> +        value = vq - 1;
> >>> +    }
> >>> +
> >>> +    raw_write(env, ri, value);
> >>>  
> >>>      /*
> >>>       * Because we arrived here, we know both FP and SVE are enabled;
> >>> diff --git a/target/arm/monitor.c b/target/arm/monitor.c
> >>> index 157c487a1551..1e213906fd8f 100644
> >>> --- a/target/arm/monitor.c
> >>> +++ b/target/arm/monitor.c
> >>> @@ -89,8 +89,24 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)
> >>>      return head;
> >>>  }
> >>>  
> >>> +QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
> >>> +
> >>> +/*
> >>> + * These are cpu model features we want to advertise. The order here
> >>> + * matters as this is the order in which qmp_query_cpu_model_expansion
> >>> + * will attempt to set them. If there are dependencies between features,
> >>> + * as there are with the sve<vl-bits> features, then the order that
> >>> + * considers those dependencies must be used.
> >>> + *
> >>> + * The sve<vl-bits> features need to be in reverse order in order to
> >>> + * enable/disable the largest vector lengths first, ensuring all
> >>> + * power-of-2 vector lengths smaller can also be enabled/disabled.
> >>> + */
> >>>  static const char *cpu_model_advertised_features[] = {
> >>>      "aarch64", "pmu", "sve",
> >>> +    "sve2048", "sve1920", "sve1792", "sve1664", "sve1536", "sve1408",
> >>> +    "sve1280", "sve1152", "sve1024", "sve896", "sve768", "sve640",
> >>> +    "sve512", "sve384", "sve256", "sve128",
> >>>      NULL
> >>>  };
> >>>  
> >>> diff --git a/tests/arm-cpu-features.c b/tests/arm-cpu-features.c
> >> Please move all the tests in a separate patch.
> > 
> > I'd prefer not to. I like keeping the test cases that test the new code in
> > the same patch. It self documents what the test cases map to what code and
> > allows immediate testing of the patch when bisecting. Also I'm not really
> > sure how it makes review worse, as another patch would look the same, just
> > in a different email. 
> A reviewier might not be familiar with both standard code and test and
> feel reluctant to invest reading boths, hence reducing the chances to
> collect R-b's. But that's my humble opinion.

That's a valid point. But with you reviewing, I'm not worried :-)

Thanks,
drew

> > 
> >> Each day has enough trouble of its own ;-) sweat...
> > 
> > I really appreciate the review! I realize it's generating sweat,
> > especially with the European heat wave! You don't have to finish
> > a patch before sending comments. I'm fine with multiple replies to
> > the same patch if you want to break your review up into smaller
> > bites.
> 
> Thanks. Yes breathing times are needed due to the heat and digestion of
> the code ;-)
> 
> Thanks
> 
> Eric
> > 
> > Thanks,
> > drew
> > 
> >>
> >> Thanks
> >>
> >> Eric
> >>> index 509e458e9c2f..a4bf6aec00df 100644
> >>> --- a/tests/arm-cpu-features.c
> >>> +++ b/tests/arm-cpu-features.c
> >>> @@ -13,6 +13,18 @@
> >>>  #include "qapi/qmp/qdict.h"
> >>>  #include "qapi/qmp/qjson.h"
> >>>  
> >>> +#if __SIZEOF_LONG__ == 8
> >>> +#define BIT(n) (1UL << (n))
> >>> +#else
> >>> +#define BIT(n) (1ULL << (n))
> >>> +#endif
> >>> +
> >>> +/*
> >>> + * We expect the SVE max-vq to be 16. Also it must be <= 64
> >>> + * for our test code, otherwise 'vls' can't just be a uint64_t.
> >>> + */
> >>> +#define SVE_MAX_VQ 16
> >>> +
> >>>  #define MACHINE    "-machine virt,gic-version=max "
> >>>  #define QUERY_HEAD "{ 'execute': 'query-cpu-model-expansion', " \
> >>>                       "'arguments': { 'type': 'full', "
> >>> @@ -137,6 +149,201 @@ static void assert_bad_props(QTestState *qts, const char *cpu_type)
> >>>      qobject_unref(resp);
> >>>  }
> >>>  
> >>> +static void resp_get_sve_vls(QDict *resp, uint64_t *vls, uint32_t *max_vq)
> >>> +{
> >>> +    const QDictEntry *e;
> >>> +    QDict *qdict;
> >>> +    int n = 0;
> >>> +
> >>> +    *vls = 0;
> >>> +
> >>> +    qdict = resp_get_props(resp);
> >>> +
> >>> +    for (e = qdict_first(qdict); e; e = qdict_next(qdict, e)) {
> >>> +        if (strlen(e->key) > 3 && !strncmp(e->key, "sve", 3) &&
> >>> +            g_ascii_isdigit(e->key[3])) {
> >>> +            char *endptr;
> >>> +            int bits;
> >>> +
> >>> +            bits = g_ascii_strtoll(&e->key[3], &endptr, 10);
> >>> +            if (!bits || *endptr != '\0') {
> >>> +                continue;
> >>> +            }
> >>> +
> >>> +            if (qdict_get_bool(qdict, e->key)) {
> >>> +                *vls |= BIT((bits / 128) - 1);
> >>> +            }
> >>> +            ++n;
> >>> +        }
> >>> +    }
> >>> +
> >>> +    g_assert(n == SVE_MAX_VQ);
> >>> +
> >>> +    *max_vq = !*vls ? 0 : 64 - __builtin_clzll(*vls);
> >>> +}
> >>> +
> >>> +static uint64_t sve_get_vls(QTestState *qts, const char *cpu_type,
> >>> +                            const char *fmt, ...)
> >>> +{
> >>> +    QDict *resp;
> >>> +    uint64_t vls;
> >>> +    uint32_t max_vq;
> >>> +
> >>> +    if (fmt) {
> >>> +        QDict *args;
> >>> +        va_list ap;
> >>> +
> >>> +        va_start(ap, fmt);
> >>> +        args = qdict_from_vjsonf_nofail(fmt, ap);
> >>> +        va_end(ap);
> >>> +
> >>> +        resp = qtest_qmp(qts, QUERY_HEAD "'model': { 'name': %s, "
> >>> +                                                    "'props': %p }"
> >>> +                              QUERY_TAIL, cpu_type, args);
> >>> +    } else {
> >>> +        resp = do_query_no_props(qts, cpu_type);
> >>> +    }
> >>> +
> >>> +    g_assert(resp);
> >>> +    resp_get_sve_vls(resp, &vls, &max_vq);
> >>> +    qobject_unref(resp);
> >>> +
> >>> +    return vls;
> >>> +}
> >>> +
> >>> +#define assert_sve_vls(qts, cpu_type, expected_vls, fmt, ...) \
> >>> +    g_assert(sve_get_vls(qts, cpu_type, fmt, ##__VA_ARGS__) == expected_vls)
> >>> +
> >>> +static void sve_tests_default(QTestState *qts, const char *cpu_type)
> >>> +{
> >>> +    /*
> >>> +     * With no sve-max-vq or sve<vl-bits> properties on the command line
> >>> +     * the default is to have all vector lengths enabled.
> >>> +     */
> >>> +    assert_sve_vls(qts, cpu_type, BIT(SVE_MAX_VQ) - 1, NULL);
> >>> +
> >>> +    /*
> >>> +     * -------------------------------------------------------------------
> >>> +     *               power-of-2(vq)   all-power-            can      can
> >>> +     *                                of-2(< vq)          enable   disable
> >>> +     * -------------------------------------------------------------------
> >>> +     * vq < max_vq      no            MUST*                yes      yes
> >>> +     * vq < max_vq      yes           MUST*                yes      no
> >>> +     * -------------------------------------------------------------------
> >>> +     * vq == max_vq     n/a           MUST*                yes**    yes**
> >>> +     * -------------------------------------------------------------------
> >>> +     * vq > max_vq      n/a           no                   no       yes
> >>> +     * vq > max_vq      n/a           yes                  yes      yes
> >>> +     * -------------------------------------------------------------------
> >>> +     *
> >>> +     * [*] "MUST" means this requirement must already be satisfied,
> >>> +     *     otherwise 'max_vq' couldn't itself be enabled.
> >>> +     *
> >>> +     * [**] Not testable with the QMP interface, only with the command line.
> >>> +     */
> >>> +
> >>> +    /* max_vq := 8 */
> >>> +    assert_sve_vls(qts, cpu_type, 0x8b, "{ 'sve1024': true }");
> >>> +
> >>> +    /* max_vq := 8, vq < max_vq, !power-of-2(vq) */
> >>> +    assert_sve_vls(qts, cpu_type, 0x8f,
> >>> +                   "{ 'sve1024': true, 'sve384': true }");
> >>> +    assert_sve_vls(qts, cpu_type, 0x8b,
> >>> +                   "{ 'sve1024': true, 'sve384': false }");
> >>> +
> >>> +    /* max_vq := 8, vq < max_vq, power-of-2(vq) */
> >>> +    assert_sve_vls(qts, cpu_type, 0x8b,
> >>> +                   "{ 'sve1024': true, 'sve256': true }");
> >>> +    assert_error(qts, cpu_type, "cannot disable sve256",
> >>> +                 "{ 'sve1024': true, 'sve256': false }");
> >>> +
> >>> +    /*
> >>> +     * max_vq := 3, vq > max_vq, !all-power-of-2(< vq)
> >>> +     *
> >>> +     * If given sve384=on,sve512=off,sve640=on the command line error would be
> >>> +     * "cannot enable sve640", but QMP visits the vector lengths in reverse
> >>> +     * order, so we get "cannot disable sve512" instead. The command line would
> >>> +     * also give that error if given sve384=on,sve640=on,sve512=off, so this is
> >>> +     * all fine. The important thing is that we get an error.
> >>> +     */
> >>> +    assert_error(qts, cpu_type, "cannot disable sve512",
> >>> +                 "{ 'sve384': true, 'sve512': false, 'sve640': true }");
> >>> +
> >>> +    /*
> >>> +     * We can disable power-of-2 vector lengths when all larger lengths
> >>> +     * are also disabled. The shorter, sve384=on,sve512=off,sve640=off
> >>> +     * works on the command line, but QMP doesn't know that all the
> >>> +     * vector lengths larger than 384-bits will be disabled until it
> >>> +     * sees the enabling of sve384, which comes near the end since it
> >>> +     * visits the lengths in reverse order. So we just have to explicitly
> >>> +     * disable them all.
> >>> +     */
> >>> +    assert_sve_vls(qts, cpu_type, 0x7,
> >>> +                   "{ 'sve384': true, 'sve512': false, 'sve640': false, "
> >>> +                   "  'sve768': false, 'sve896': false, 'sve1024': false, "
> >>> +                   "  'sve1152': false, 'sve1280': false, 'sve1408': false, "
> >>> +                   "  'sve1536': false, 'sve1664': false, 'sve1792': false, "
> >>> +                   "  'sve1920': false, 'sve2048': false }");
> >>> +
> >>> +    /* max_vq := 3, vq > max_vq, all-power-of-2(< vq) */
> >>> +    assert_sve_vls(qts, cpu_type, 0x1f,
> >>> +                   "{ 'sve384': true, 'sve512': true, 'sve640': true }");
> >>> +    assert_sve_vls(qts, cpu_type, 0xf,
> >>> +                   "{ 'sve384': true, 'sve512': true, 'sve640': false }");
> >>> +}
> >>> +
> >>> +static void sve_tests_sve_max_vq_8(const void *data)
> >>> +{
> >>> +    QTestState *qts;
> >>> +
> >>> +    qts = qtest_init(MACHINE "-cpu max,sve-max-vq=8");
> >>> +
> >>> +    assert_sve_vls(qts, "max", BIT(8) - 1, NULL);
> >>> +
> >>> +    /*
> >>> +     * Disabling the max-vq set by sve-max-vq is not allowed, but
> >>> +     * of course enabling it is OK.
> >>> +     */
> >>> +    assert_error(qts, "max", "cannot disable sve1024", "{ 'sve1024': false }");
> >>> +    assert_sve_vls(qts, "max", 0xff, "{ 'sve1024': true }");
> >>> +
> >>> +    /*
> >>> +     * Enabling anything larger than max-vq set by sve-max-vq is not
> >>> +     * allowed, but of course disabling everything larger is OK.
> >>> +     */
> >>> +    assert_error(qts, "max", "cannot enable sve1152", "{ 'sve1152': true }");
> >>> +    assert_sve_vls(qts, "max", 0xff, "{ 'sve1152': false }");
> >>> +
> >>> +    /*
> >>> +     * We can disable non power-of-2 lengths smaller than the max-vq
> >>> +     * set by sve-max-vq, but not power-of-2 lengths.
> >>> +     */
> >>> +    assert_sve_vls(qts, "max", 0xfb, "{ 'sve384': false }");
> >>> +    assert_error(qts, "max", "cannot disable sve256", "{ 'sve256': false }");
> >>> +
> >>> +    qtest_quit(qts);
> >>> +}
> >>> +
> >>> +static void sve_tests_sve_off(const void *data)
> >>> +{
> >>> +    QTestState *qts;
> >>> +
> >>> +    qts = qtest_init(MACHINE "-cpu max,sve=off");
> >>> +
> >>> +    /*
> >>> +     * SVE is off, so the map should be empty.
> >>> +     */
> >>> +    assert_sve_vls(qts, "max", 0, NULL);
> >>> +
> >>> +    /*
> >>> +     * We can't turn anything on, but off is OK.
> >>> +     */
> >>> +    assert_error(qts, "max", "cannot enable sve128", "{ 'sve128': true }");
> >>> +    assert_sve_vls(qts, "max", 0, "{ 'sve128': false }");
> >>> +
> >>> +    qtest_quit(qts);
> >>> +}
> >>> +
> >>>  static void test_query_cpu_model_expansion(const void *data)
> >>>  {
> >>>      QTestState *qts;
> >>> @@ -159,9 +366,12 @@ static void test_query_cpu_model_expansion(const void *data)
> >>>      if (g_str_equal(qtest_get_arch(), "aarch64")) {
> >>>          assert_has_feature(qts, "max", "aarch64");
> >>>          assert_has_feature(qts, "max", "sve");
> >>> +        assert_has_feature(qts, "max", "sve128");
> >>>          assert_has_feature(qts, "cortex-a57", "pmu");
> >>>          assert_has_feature(qts, "cortex-a57", "aarch64");
> >>>  
> >>> +        sve_tests_default(qts, "max");
> >>> +
> >>>          /* Test that features that depend on KVM generate errors without. */
> >>>          assert_error(qts, "max",
> >>>                       "'aarch64' feature cannot be disabled "
> >>> @@ -213,6 +423,13 @@ int main(int argc, char **argv)
> >>>      qtest_add_data_func("/arm/query-cpu-model-expansion",
> >>>                          NULL, test_query_cpu_model_expansion);
> >>>  
> >>> +    if (g_str_equal(qtest_get_arch(), "aarch64")) {
> >>> +        qtest_add_data_func("/arm/max/query-cpu-model-expansion/sve-max-vq-8",
> >>> +                            NULL, sve_tests_sve_max_vq_8);
> >>> +        qtest_add_data_func("/arm/max/query-cpu-model-expansion/sve-off",
> >>> +                            NULL, sve_tests_sve_off);
> >>> +    }
> >>> +
> >>>      if (kvm_available) {
> >>>          qtest_add_data_func("/arm/kvm/query-cpu-model-expansion",
> >>>                              NULL, test_query_cpu_model_expansion_kvm);
> >>>
> >>
> 

Re: [Qemu-devel] [PATCH v2 07/14] target/arm/cpu64: max cpu: Introduce sve<vl-bits> properties

[Auger Eric]

Hi,
On 6/21/19 6:34 PM, Andrew Jones wrote:
> Introduce cpu properties to give fine control over SVE vector lengths.
> We introduce a property for each valid length up to the current
> maximum supported, which is 2048-bits. The properties are named, e.g.
> sve128, sve256, sve512, ..., where the number is the number of bits.
> 
> It's now possible to do something like -cpu max,sve-max-vq=4,sve384=off
> to provide a guest vector lengths 128, 256, and 512 bits. The resulting
> set must conform to the architectural constraint of having all power-of-2
> lengths smaller than the maximum length present. It's also possible to
> only provide sve<vl-bits> properties, e.g. -cpu max,sve512=on. That
> example provides the machine with 128, 256, and 512 bit vector lengths.
> It doesn't hurt to explicitly ask for all expected vector lengths,
> which is what, for example, libvirt should do.
> 
> Note1, it is not possible to use sve<vl-bits> properties before
> sve-max-vq, e.g. -cpu max,sve384=off,sve-max-vq=4, as supporting
> that overly complicates the user input validation.
> 
> Note2, while one might expect -cpu max,sve-max-vq=4,sve512=on to be the
> same as -cpu max,sve512=on, they are not. The former enables all vector
> lengths 512 bits and smaller, while the latter only sets the 512-bit
> length and its smaller power-of-2 lengths. It's probably best not to use
> sve-max-vq with sve<vl-bits> properties, but it can't be completely
> forbidden as we want qmp_query_cpu_model_expansion to work with guests
> launched with e.g. -cpu max,sve-max-vq=8 on their command line.
> 
> Signed-off-by: Andrew Jones <drjones@redhat.com>
> ---
>  target/arm/cpu.c         |   6 +
>  target/arm/cpu.h         |  14 ++
>  target/arm/cpu64.c       | 360 ++++++++++++++++++++++++++++++++++++++-
>  target/arm/helper.c      |  11 +-
>  target/arm/monitor.c     |  16 ++
>  tests/arm-cpu-features.c | 217 +++++++++++++++++++++++
>  6 files changed, 620 insertions(+), 4 deletions(-)
> 
> diff --git a/target/arm/cpu.c b/target/arm/cpu.c
> index f08e178fc84b..e060a0d9df0e 100644
> --- a/target/arm/cpu.c
> +++ b/target/arm/cpu.c
> @@ -1019,6 +1019,12 @@ static void arm_cpu_realizefn(DeviceState *dev, Error **errp)
>          return;
>      }
>  
> +    arm_cpu_sve_finalize(cpu, &local_err);
> +    if (local_err) {
> +        error_propagate(errp, local_err);
> +        return;
> +    }
> +
>      if (arm_feature(env, ARM_FEATURE_AARCH64) &&
>          cpu->has_vfp != cpu->has_neon) {
>          /*
> diff --git a/target/arm/cpu.h b/target/arm/cpu.h
> index f9da672be575..cbb155cf72a5 100644
> --- a/target/arm/cpu.h
> +++ b/target/arm/cpu.h
> @@ -184,8 +184,13 @@ typedef struct {
>  
>  #ifdef TARGET_AARCH64
>  # define ARM_MAX_VQ    16
> +void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp);
> +uint32_t arm_cpu_vq_map_next_smaller(ARMCPU *cpu, uint32_t vq);
>  #else
>  # define ARM_MAX_VQ    1
> +static inline void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp) { }
> +static inline uint32_t arm_cpu_vq_map_next_smaller(ARMCPU *cpu, uint32_t vq)
> +{ return 0; }
>  #endif
>  
>  typedef struct ARMVectorReg {
> @@ -915,6 +920,15 @@ struct ARMCPU {
>  
>      /* Used to set the maximum vector length the cpu will support.  */
>      uint32_t sve_max_vq;
> +
> +    /*
> +     * In sve_vq_map each set bit is a supported vector length of
> +     * (bit-number + 1) * 16 bytes, i.e. each bit number + 1 is the vector
> +     * length in quadwords. We need a map size twice the maximum
> +     * quadword length though because we use two bits for each vector
> +     * length in order to track three states: uninitialized, off, and on.
> +     */
> +    DECLARE_BITMAP(sve_vq_map, ARM_MAX_VQ * 2);
>  };
>  
>  void arm_cpu_post_init(Object *obj);
> diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
> index 02ada65f240c..5def82111dee 100644
> --- a/target/arm/cpu64.c
> +++ b/target/arm/cpu64.c
> @@ -257,6 +257,149 @@ static void aarch64_a72_initfn(Object *obj)
>      define_arm_cp_regs(cpu, cortex_a72_a57_a53_cp_reginfo);
>  }
>  
> +/*
> + * While we eventually use cpu->sve_vq_map as a typical bitmap, where each vq
> + * has only two states (off/on), until we've finalized the map at realize time
> + * we use an extra bit, at the vq - 1 + ARM_MAX_VQ bit number, to also allow
> + * tracking of the uninitialized state. The arm_vq_state typedef and following
> + * functions allow us to more easily work with the bitmap. Also, while the map
> + * is still initializing, sve-max-vq has an additional three states, bringing
> + * the number of its states to five, which are the following:
> + *
> + * sve-max-vq:
> + *   0:    SVE is disabled. The default value for a vq in the map is 'OFF'.
> + *  -1:    SVE is enabled, but neither sve-max-vq nor sve<vl-bits> properties
> + *         have yet been specified by the user. The default value for a vq in
> + *         the map is 'ON'.
> + *  -2:    SVE is enabled and one or more sve<vl-bits> properties have been
> + *         set to 'OFF' by the user, but no sve<vl-bits> properties have yet
> + *         been set to 'ON'. The user is now blocked from setting sve-max-vq
> + *         and the default value for a vq in the map is 'ON'.
> + *  -3:    SVE is enabled and one or more sve<vl-bits> properties have been
> + *         set to 'ON' by the user. The user is blocked from setting sve-max-vq
> + *         and the default value for a vq in the map is 'OFF'. sve-max-vq never
> + *         transitions back to -2, even if later inputs disable the vector
> + *         lengths that initially transitioned sve-max-vq to this state. This
> + *         avoids the default values from flip-flopping.
> + *  [1-ARM_MAX_VQ]: SVE is enabled and the user has specified a valid
> + *                  sve-max-vq. The sve-max-vq specified vq and all smaller
> + *                  vq's will be initially enabled. All larger vq's will have
> + *                  a default of 'OFF'.
> + */
> +#define ARM_SVE_INIT          -1
> +#define ARM_VQ_DEFAULT_ON     -2
> +#define ARM_VQ_DEFAULT_OFF    -3
> +
> +#define arm_sve_have_max_vq(cpu) ((int32_t)(cpu)->sve_max_vq > 0)
> +
> +typedef enum arm_vq_state {
> +    ARM_VQ_OFF,
> +    ARM_VQ_ON,
> +    ARM_VQ_UNINITIALIZED,
> +} arm_vq_state;
> +
> +static arm_vq_state arm_cpu_vq_map_get(ARMCPU *cpu, int vq)
> +{
> +    assert(vq <= ARM_MAX_VQ);
> +
> +    return test_bit(vq - 1, cpu->sve_vq_map) |
> +           test_bit(vq - 1 + ARM_MAX_VQ, cpu->sve_vq_map) << 1;
> +}
> +
> +static void arm_cpu_vq_map_set(ARMCPU *cpu, int vq, arm_vq_state state)
> +{
> +    assert(state == ARM_VQ_OFF || state == ARM_VQ_ON);
> +    assert(vq <= ARM_MAX_VQ);
> +
> +    clear_bit(vq - 1 + ARM_MAX_VQ, cpu->sve_vq_map);
> +
> +    if (state == ARM_VQ_ON) {
> +        set_bit(vq - 1, cpu->sve_vq_map);
> +    } else {
> +        clear_bit(vq - 1, cpu->sve_vq_map);
> +    }
> +}
> +
> +static void arm_cpu_vq_map_init(ARMCPU *cpu)
> +{
> +    bitmap_zero(cpu->sve_vq_map, ARM_MAX_VQ * 2);
> +    bitmap_set(cpu->sve_vq_map, ARM_MAX_VQ, ARM_MAX_VQ);
> +}
> +
> +static bool arm_cpu_vq_map_is_finalized(ARMCPU *cpu)
> +{
> +    DECLARE_BITMAP(map, ARM_MAX_VQ * 2);
> +
> +    bitmap_zero(map, ARM_MAX_VQ * 2);
> +    bitmap_set(map, ARM_MAX_VQ, ARM_MAX_VQ);
> +    bitmap_and(map, map, cpu->sve_vq_map, ARM_MAX_VQ * 2);
> +
> +    return bitmap_empty(map, ARM_MAX_VQ * 2);
> +}
> +
> +static void arm_cpu_vq_map_finalize(ARMCPU *cpu)
> +{
> +    Error *err = NULL;
> +    char name[8];
> +    uint32_t vq;
> +    bool value;
> +
> +    /*
> +     * We use the property get accessor because it knows what default
> +     * values to return for uninitialized vector lengths.
> +     */
> +    for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
> +        sprintf(name, "sve%d", vq * 128);
> +        value = object_property_get_bool(OBJECT(cpu), name, &err);
> +        assert(!err);
> +        if (value) {
> +            arm_cpu_vq_map_set(cpu, vq, ARM_VQ_ON);
> +        } else {
> +            arm_cpu_vq_map_set(cpu, vq, ARM_VQ_OFF);
> +        }
> +    }
> +
> +    assert(arm_cpu_vq_map_is_finalized(cpu));
> +}
> +
> +void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp)
> +{
> +    Error *err = NULL;
> +
> +    if (!cpu->sve_max_vq) {
> +        bitmap_zero(cpu->sve_vq_map, ARM_MAX_VQ * 2);
> +        return;
> +    }
> +
> +    if (cpu->sve_max_vq == ARM_SVE_INIT) {
> +        object_property_set_uint(OBJECT(cpu), ARM_MAX_VQ, "sve-max-vq", &err);
> +        if (err) {
> +            error_propagate(errp, err);
> +            return;
> +        }
> +        assert(cpu->sve_max_vq == ARM_MAX_VQ);
> +        arm_cpu_vq_map_finalize(cpu);
> +    } else {
> +        arm_cpu_vq_map_finalize(cpu);
> +        if (!arm_sve_have_max_vq(cpu)) {
> +            cpu->sve_max_vq = arm_cpu_vq_map_next_smaller(cpu, ARM_MAX_VQ + 1);
> +        }
> +    }
> +
> +    assert(cpu->sve_max_vq == arm_cpu_vq_map_next_smaller(cpu, ARM_MAX_VQ + 1));
> +}
> +
> +uint32_t arm_cpu_vq_map_next_smaller(ARMCPU *cpu, uint32_t vq)
> +{
> +    uint32_t bitnum;
> +
> +    assert(vq <= ARM_MAX_VQ + 1);
> +    assert(arm_cpu_vq_map_is_finalized(cpu));
> +
> +    bitnum = find_last_bit(cpu->sve_vq_map, vq - 1);
> +    return bitnum == vq - 1 ? 0 : bitnum + 1;
> +}
> +
>  static void cpu_max_get_sve_max_vq(Object *obj, Visitor *v, const char *name,
>                                     void *opaque, Error **errp)
>  {
> @@ -283,12 +426,203 @@ static void cpu_max_set_sve_max_vq(Object *obj, Visitor *v, const char *name,
>          return;
>      }
>  
> +    /*
> +     * It gets complicated trying to support both sve-max-vq and
> +     * sve<vl-bits> properties together, so we mostly don't. We
> +     * do allow both if sve-max-vq is specified first and only once
> +     * though.
> +     */
> +    if (cpu->sve_max_vq != ARM_SVE_INIT) {
> +        error_setg(errp, "sve<vl-bits> in use or sve-max-vq already "
> +                   "specified");
> +        error_append_hint(errp, "sve-max-vq must come before all "
> +                          "sve<vl-bits> properties and it must only "
> +                          "be specified once.\n");
> +        return;
> +    }
> +
>      cpu->sve_max_vq = value;
>  
>      if (cpu->sve_max_vq == 0 || cpu->sve_max_vq > ARM_MAX_VQ) {
>          error_setg(errp, "unsupported SVE vector length");
>          error_append_hint(errp, "Valid sve-max-vq in range [1-%d]\n",
>                            ARM_MAX_VQ);
> +    } else {
> +        uint32_t vq;
> +
> +        for (vq = 1; vq <= cpu->sve_max_vq; ++vq) {
> +            char name[8];
> +            sprintf(name, "sve%d", vq * 128);
> +            object_property_set_bool(obj, true, name, &err);
> +            if (err) {
> +                error_propagate(errp, err);
> +                return;
> +            }
> +        }
> +    }
> +}
> +
> +static void cpu_arm_get_sve_vq(Object *obj, Visitor *v, const char *name,
> +                               void *opaque, Error **errp)
> +{
> +    ARMCPU *cpu = ARM_CPU(obj);
> +    int vq = atoi(&name[3]) / 128;
> +    arm_vq_state vq_state;
> +    bool value;
> +
> +    vq_state = arm_cpu_vq_map_get(cpu, vq);
> +
> +    if (!cpu->sve_max_vq) {
> +        /* All vector lengths are disabled when SVE is off. */
> +        value = false;
> +    } else if (vq_state == ARM_VQ_ON) {
> +        value = true;
> +    } else if (vq_state == ARM_VQ_OFF) {
> +        value = false;
> +    } else {
> +        /*
> +         * vq is uninitialized. We pick a default here based on the
> +         * the state of sve-max-vq and other sve<vl-bits> properties.
> +         */
> +        if (arm_sve_have_max_vq(cpu)) {
> +            /*
> +             * If we have sve-max-vq, then all remaining uninitialized
> +             * vq's are 'OFF'.
> +             */
> +            value = false;
> +        } else {
> +            switch (cpu->sve_max_vq) {
> +            case ARM_SVE_INIT:
> +            case ARM_VQ_DEFAULT_ON:
> +                value = true;
> +                break;
> +            case ARM_VQ_DEFAULT_OFF:
> +                value = false;
> +                break;
> +            }
> +        }
> +    }
> +
> +    visit_type_bool(v, name, &value, errp);
> +}
> +
> +static void cpu_arm_set_sve_vq(Object *obj, Visitor *v, const char *name,
> +                               void *opaque, Error **errp)
> +{
> +    ARMCPU *cpu = ARM_CPU(obj);
> +    int vq = atoi(&name[3]) / 128;
> +    arm_vq_state vq_state;
> +    Error *err = NULL;
> +    uint32_t max_vq = 0;
> +    bool value;
> +
> +    visit_type_bool(v, name, &value, &err);
> +    if (err) {
> +        error_propagate(errp, err);
> +        return;
> +    }
> +
> +    if (value && !cpu->sve_max_vq) {
> +        error_setg(errp, "cannot enable %s", name);
> +        error_append_hint(errp, "SVE has been disabled with sve=off\n");
> +        return;
> +    } else if (!cpu->sve_max_vq) {
> +        /*
> +         * We don't complain about disabling vector lengths when SVE
> +         * is off, but we don't do anything either.
> +         */
> +        return;
> +    }
> +
> +    if (arm_sve_have_max_vq(cpu)) {
> +        max_vq = cpu->sve_max_vq;
> +    } else {
> +        if (value) {
> +            cpu->sve_max_vq = ARM_VQ_DEFAULT_OFF;
> +        } else if (cpu->sve_max_vq != ARM_VQ_DEFAULT_OFF) {
> +            cpu->sve_max_vq = ARM_VQ_DEFAULT_ON;
> +        }
> +    }
> +
> +    /*
> +     * We need to know the maximum vector length, which may just currently
> +     * be the maximum length, in order to validate the enabling/disabling
> +     * of this vector length. We use the property get accessor in order to
> +     * get the appropriate default value for any uninitialized lengths.
> +     */
> +    if (!max_vq) {
> +        char tmp[8];
> +        bool s;
> +
> +        for (max_vq = ARM_MAX_VQ; max_vq >= 1; --max_vq) {
> +            sprintf(tmp, "sve%d", max_vq * 128);
> +            s = object_property_get_bool(OBJECT(cpu), tmp, &err);
> +            assert(!err);
> +            if (s) {
> +                break;
> +            }
> +        }
> +    }
> +
> +    if (arm_sve_have_max_vq(cpu) && value && vq > cpu->sve_max_vq) {
> +        error_setg(errp, "cannot enable %s", name);
> +        error_append_hint(errp, "vq=%d (%d bits) is larger than the "
> +                          "maximum vector length, sve-max-vq=%d "
> +                          "(%d bits)\n", vq, vq * 128,
> +                          cpu->sve_max_vq, cpu->sve_max_vq * 128);
> +    } else if (arm_sve_have_max_vq(cpu) && !value && vq == cpu->sve_max_vq) {
> +        error_setg(errp, "cannot disable %s", name);
> +        error_append_hint(errp, "The maximum vector length must be "
> +                          "enabled, sve-max-vq=%d (%d bits)\n",
> +                          cpu->sve_max_vq, cpu->sve_max_vq * 128);
> +    } else if (arm_sve_have_max_vq(cpu) && !value && vq < cpu->sve_max_vq &&
> +               is_power_of_2(vq)) {
> +        error_setg(errp, "cannot disable %s", name);
> +        error_append_hint(errp, "vq=%d (%d bits) is required as it is a "
> +                          "power-of-2 length smaller than the maximum, "
> +                          "sve-max-vq=%d (%d bits)\n", vq, vq * 128,
> +                          cpu->sve_max_vq, cpu->sve_max_vq * 128);
> +    } else if (!value && vq < max_vq && is_power_of_2(vq)) {
> +        error_setg(errp, "cannot disable %s", name);
> +        error_append_hint(errp, "Vector length %d-bits is required as it "
> +                          "is a power-of-2 length smaller than another "
> +                          "enabled vector length. Disable all larger vector "
> +                          "lengths first.\n", vq * 128);
> +    } else {
> +        if (value) {
> +            bool fail = false;
> +            uint32_t s;
> +
> +            /*
> +             * Enabling a vector length automatically enables all
> +             * uninitialized power-of-2 lengths smaller than it, as
> +             * per the architecture.
> +             */
> +            for (s = 1; s < vq; ++s) {
> +                if (is_power_of_2(s)) {
> +                    vq_state = arm_cpu_vq_map_get(cpu, s);
> +                    if (vq_state == ARM_VQ_UNINITIALIZED) {
> +                        arm_cpu_vq_map_set(cpu, s, ARM_VQ_ON);
> +                    } else if (vq_state == ARM_VQ_OFF) {
> +                        fail = true;
> +                        break;
> +                    }
> +                }
> +            }
> +
> +            if (fail) {
> +                error_setg(errp, "cannot enable %s", name);
> +                error_append_hint(errp, "Vector length %d-bits is disabled "
> +                                  "and is a power-of-2 length smaller than "
> +                                  "%s. All power-of-2 vector lengths smaller "
> +                                  "than the maximum length are required.\n",
> +                                  s * 128, name);
> +            } else {
> +                arm_cpu_vq_map_set(cpu, vq, ARM_VQ_ON);
> +            }
> +        } else {
> +            arm_cpu_vq_map_set(cpu, vq, ARM_VQ_OFF);
> +        }
>      }
>  }
>  
> @@ -318,10 +652,11 @@ static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
>          /*
>           * We handle the -cpu <cpu>,sve=off,sve=on case by reinitializing,
>           * but otherwise we don't do anything as an sve=on could come after
> -         * a sve-max-vq setting.
> +         * a sve-max-vq or sve<vl-bits> setting.
>           */
>          if (!cpu->sve_max_vq) {
> -            cpu->sve_max_vq = ARM_MAX_VQ;
> +            cpu->sve_max_vq = ARM_SVE_INIT;
> +            arm_cpu_vq_map_init(cpu);
>          }
>      } else {
>          cpu->sve_max_vq = 0;
> @@ -336,6 +671,7 @@ static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
>  static void aarch64_max_initfn(Object *obj)
>  {
>      ARMCPU *cpu = ARM_CPU(obj);
> +    uint32_t vq;
>  
>      if (kvm_enabled()) {
>          kvm_arm_set_cpu_features_from_host(cpu);
> @@ -420,11 +756,29 @@ static void aarch64_max_initfn(Object *obj)
>          cpu->dcz_blocksize = 7; /*  512 bytes */
>  #endif
>  
> -        cpu->sve_max_vq = ARM_MAX_VQ;
> +        /*
> +         * sve_max_vq is initially unspecified, but must be initialized to a
> +         * non-zero value (ARM_SVE_INIT) to indicate that this cpu type has
> +         * SVE. It will be finalized in arm_cpu_realizefn().
> +         */
> +        cpu->sve_max_vq = ARM_SVE_INIT;
>          object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_max_vq,
>                              cpu_max_set_sve_max_vq, NULL, NULL, &error_fatal);
>          object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
>                              cpu_arm_set_sve, NULL, NULL, &error_fatal);
> +
> +        /*
> +         * sve_vq_map uses a special state while setting properties, so
> +         * we initialize it here with its init function and finalize it
> +         * in arm_cpu_realizefn().
> +         */
> +        arm_cpu_vq_map_init(cpu);
> +        for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
> +            char name[8];
> +            sprintf(name, "sve%d", vq * 128);
> +            object_property_add(obj, name, "bool", cpu_arm_get_sve_vq,
> +                                cpu_arm_set_sve_vq, NULL, NULL, &error_fatal);
> +        }
>      }
>  }
>  
> diff --git a/target/arm/helper.c b/target/arm/helper.c
> index f500ccb6d31b..b7b719dba57f 100644
> --- a/target/arm/helper.c
> +++ b/target/arm/helper.c
> @@ -5324,7 +5324,16 @@ static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
>  
>      /* Bits other than [3:0] are RAZ/WI.  */
>      QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
> -    raw_write(env, ri, value & 0xf);
> +    value &= 0xf;
> +
> +    if (value) {
> +        /* get next vq that is smaller than or equal to value's vq */
> +        uint32_t vq = value + 1;
> +        vq = arm_cpu_vq_map_next_smaller(cpu, vq + 1);
> +        value = vq - 1;
spec says:

"if an unsupported vector length is requested in ZCR_ELx, the
implementation is required to select the largest
supported vector length that is less than the requested length. This
does not alter the value of ZCR_ELx.LEN.
"

So I understand the value written in the reg should not be unmodified.

Thanks

Eric

> +    }
> +
> +    raw_write(env, ri, value);
>  
>      /*
>       * Because we arrived here, we know both FP and SVE are enabled;
> diff --git a/target/arm/monitor.c b/target/arm/monitor.c
> index 157c487a1551..1e213906fd8f 100644
> --- a/target/arm/monitor.c
> +++ b/target/arm/monitor.c
> @@ -89,8 +89,24 @@ GICCapabilityList *qmp_query_gic_capabilities(Error **errp)
>      return head;
>  }
>  
> +QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
> +
> +/*
> + * These are cpu model features we want to advertise. The order here
> + * matters as this is the order in which qmp_query_cpu_model_expansion
> + * will attempt to set them. If there are dependencies between features,
> + * as there are with the sve<vl-bits> features, then the order that
> + * considers those dependencies must be used.
> + *
> + * The sve<vl-bits> features need to be in reverse order in order to
> + * enable/disable the largest vector lengths first, ensuring all
> + * power-of-2 vector lengths smaller can also be enabled/disabled.
> + */
>  static const char *cpu_model_advertised_features[] = {
>      "aarch64", "pmu", "sve",
> +    "sve2048", "sve1920", "sve1792", "sve1664", "sve1536", "sve1408",
> +    "sve1280", "sve1152", "sve1024", "sve896", "sve768", "sve640",
> +    "sve512", "sve384", "sve256", "sve128",
>      NULL
>  };
>  
> diff --git a/tests/arm-cpu-features.c b/tests/arm-cpu-features.c
> index 509e458e9c2f..a4bf6aec00df 100644
> --- a/tests/arm-cpu-features.c
> +++ b/tests/arm-cpu-features.c
> @@ -13,6 +13,18 @@
>  #include "qapi/qmp/qdict.h"
>  #include "qapi/qmp/qjson.h"
>  
> +#if __SIZEOF_LONG__ == 8
> +#define BIT(n) (1UL << (n))
> +#else
> +#define BIT(n) (1ULL << (n))
> +#endif
> +
> +/*
> + * We expect the SVE max-vq to be 16. Also it must be <= 64
> + * for our test code, otherwise 'vls' can't just be a uint64_t.
> + */
> +#define SVE_MAX_VQ 16
> +
>  #define MACHINE    "-machine virt,gic-version=max "
>  #define QUERY_HEAD "{ 'execute': 'query-cpu-model-expansion', " \
>                       "'arguments': { 'type': 'full', "
> @@ -137,6 +149,201 @@ static void assert_bad_props(QTestState *qts, const char *cpu_type)
>      qobject_unref(resp);
>  }
>  
> +static void resp_get_sve_vls(QDict *resp, uint64_t *vls, uint32_t *max_vq)
> +{
> +    const QDictEntry *e;
> +    QDict *qdict;
> +    int n = 0;
> +
> +    *vls = 0;
> +
> +    qdict = resp_get_props(resp);
> +
> +    for (e = qdict_first(qdict); e; e = qdict_next(qdict, e)) {
> +        if (strlen(e->key) > 3 && !strncmp(e->key, "sve", 3) &&
> +            g_ascii_isdigit(e->key[3])) {
> +            char *endptr;
> +            int bits;
> +
> +            bits = g_ascii_strtoll(&e->key[3], &endptr, 10);
> +            if (!bits || *endptr != '\0') {
> +                continue;
> +            }
> +
> +            if (qdict_get_bool(qdict, e->key)) {
> +                *vls |= BIT((bits / 128) - 1);
> +            }
> +            ++n;
> +        }
> +    }
> +
> +    g_assert(n == SVE_MAX_VQ);
> +
> +    *max_vq = !*vls ? 0 : 64 - __builtin_clzll(*vls);
> +}
> +
> +static uint64_t sve_get_vls(QTestState *qts, const char *cpu_type,
> +                            const char *fmt, ...)
> +{
> +    QDict *resp;
> +    uint64_t vls;
> +    uint32_t max_vq;
> +
> +    if (fmt) {
> +        QDict *args;
> +        va_list ap;
> +
> +        va_start(ap, fmt);
> +        args = qdict_from_vjsonf_nofail(fmt, ap);
> +        va_end(ap);
> +
> +        resp = qtest_qmp(qts, QUERY_HEAD "'model': { 'name': %s, "
> +                                                    "'props': %p }"
> +                              QUERY_TAIL, cpu_type, args);
> +    } else {
> +        resp = do_query_no_props(qts, cpu_type);
> +    }
> +
> +    g_assert(resp);
> +    resp_get_sve_vls(resp, &vls, &max_vq);
> +    qobject_unref(resp);
> +
> +    return vls;
> +}
> +
> +#define assert_sve_vls(qts, cpu_type, expected_vls, fmt, ...) \
> +    g_assert(sve_get_vls(qts, cpu_type, fmt, ##__VA_ARGS__) == expected_vls)
> +
> +static void sve_tests_default(QTestState *qts, const char *cpu_type)
> +{
> +    /*
> +     * With no sve-max-vq or sve<vl-bits> properties on the command line
> +     * the default is to have all vector lengths enabled.
> +     */
> +    assert_sve_vls(qts, cpu_type, BIT(SVE_MAX_VQ) - 1, NULL);
> +
> +    /*
> +     * -------------------------------------------------------------------
> +     *               power-of-2(vq)   all-power-            can      can
> +     *                                of-2(< vq)          enable   disable
> +     * -------------------------------------------------------------------
> +     * vq < max_vq      no            MUST*                yes      yes
> +     * vq < max_vq      yes           MUST*                yes      no
> +     * -------------------------------------------------------------------
> +     * vq == max_vq     n/a           MUST*                yes**    yes**
> +     * -------------------------------------------------------------------
> +     * vq > max_vq      n/a           no                   no       yes
> +     * vq > max_vq      n/a           yes                  yes      yes
> +     * -------------------------------------------------------------------
> +     *
> +     * [*] "MUST" means this requirement must already be satisfied,
> +     *     otherwise 'max_vq' couldn't itself be enabled.
> +     *
> +     * [**] Not testable with the QMP interface, only with the command line.
> +     */
> +
> +    /* max_vq := 8 */
> +    assert_sve_vls(qts, cpu_type, 0x8b, "{ 'sve1024': true }");
> +
> +    /* max_vq := 8, vq < max_vq, !power-of-2(vq) */
> +    assert_sve_vls(qts, cpu_type, 0x8f,
> +                   "{ 'sve1024': true, 'sve384': true }");
> +    assert_sve_vls(qts, cpu_type, 0x8b,
> +                   "{ 'sve1024': true, 'sve384': false }");
> +
> +    /* max_vq := 8, vq < max_vq, power-of-2(vq) */
> +    assert_sve_vls(qts, cpu_type, 0x8b,
> +                   "{ 'sve1024': true, 'sve256': true }");
> +    assert_error(qts, cpu_type, "cannot disable sve256",
> +                 "{ 'sve1024': true, 'sve256': false }");
> +
> +    /*
> +     * max_vq := 3, vq > max_vq, !all-power-of-2(< vq)
> +     *
> +     * If given sve384=on,sve512=off,sve640=on the command line error would be
> +     * "cannot enable sve640", but QMP visits the vector lengths in reverse
> +     * order, so we get "cannot disable sve512" instead. The command line would
> +     * also give that error if given sve384=on,sve640=on,sve512=off, so this is
> +     * all fine. The important thing is that we get an error.
> +     */
> +    assert_error(qts, cpu_type, "cannot disable sve512",
> +                 "{ 'sve384': true, 'sve512': false, 'sve640': true }");
> +
> +    /*
> +     * We can disable power-of-2 vector lengths when all larger lengths
> +     * are also disabled. The shorter, sve384=on,sve512=off,sve640=off
> +     * works on the command line, but QMP doesn't know that all the
> +     * vector lengths larger than 384-bits will be disabled until it
> +     * sees the enabling of sve384, which comes near the end since it
> +     * visits the lengths in reverse order. So we just have to explicitly
> +     * disable them all.
> +     */
> +    assert_sve_vls(qts, cpu_type, 0x7,
> +                   "{ 'sve384': true, 'sve512': false, 'sve640': false, "
> +                   "  'sve768': false, 'sve896': false, 'sve1024': false, "
> +                   "  'sve1152': false, 'sve1280': false, 'sve1408': false, "
> +                   "  'sve1536': false, 'sve1664': false, 'sve1792': false, "
> +                   "  'sve1920': false, 'sve2048': false }");
> +
> +    /* max_vq := 3, vq > max_vq, all-power-of-2(< vq) */
> +    assert_sve_vls(qts, cpu_type, 0x1f,
> +                   "{ 'sve384': true, 'sve512': true, 'sve640': true }");
> +    assert_sve_vls(qts, cpu_type, 0xf,
> +                   "{ 'sve384': true, 'sve512': true, 'sve640': false }");
> +}
> +
> +static void sve_tests_sve_max_vq_8(const void *data)
> +{
> +    QTestState *qts;
> +
> +    qts = qtest_init(MACHINE "-cpu max,sve-max-vq=8");
> +
> +    assert_sve_vls(qts, "max", BIT(8) - 1, NULL);
> +
> +    /*
> +     * Disabling the max-vq set by sve-max-vq is not allowed, but
> +     * of course enabling it is OK.
> +     */
> +    assert_error(qts, "max", "cannot disable sve1024", "{ 'sve1024': false }");
> +    assert_sve_vls(qts, "max", 0xff, "{ 'sve1024': true }");
> +
> +    /*
> +     * Enabling anything larger than max-vq set by sve-max-vq is not
> +     * allowed, but of course disabling everything larger is OK.
> +     */
> +    assert_error(qts, "max", "cannot enable sve1152", "{ 'sve1152': true }");
> +    assert_sve_vls(qts, "max", 0xff, "{ 'sve1152': false }");
> +
> +    /*
> +     * We can disable non power-of-2 lengths smaller than the max-vq
> +     * set by sve-max-vq, but not power-of-2 lengths.
> +     */
> +    assert_sve_vls(qts, "max", 0xfb, "{ 'sve384': false }");
> +    assert_error(qts, "max", "cannot disable sve256", "{ 'sve256': false }");
> +
> +    qtest_quit(qts);
> +}
> +
> +static void sve_tests_sve_off(const void *data)
> +{
> +    QTestState *qts;
> +
> +    qts = qtest_init(MACHINE "-cpu max,sve=off");
> +
> +    /*
> +     * SVE is off, so the map should be empty.
> +     */
> +    assert_sve_vls(qts, "max", 0, NULL);
> +
> +    /*
> +     * We can't turn anything on, but off is OK.
> +     */
> +    assert_error(qts, "max", "cannot enable sve128", "{ 'sve128': true }");
> +    assert_sve_vls(qts, "max", 0, "{ 'sve128': false }");
> +
> +    qtest_quit(qts);
> +}
> +
>  static void test_query_cpu_model_expansion(const void *data)
>  {
>      QTestState *qts;
> @@ -159,9 +366,12 @@ static void test_query_cpu_model_expansion(const void *data)
>      if (g_str_equal(qtest_get_arch(), "aarch64")) {
>          assert_has_feature(qts, "max", "aarch64");
>          assert_has_feature(qts, "max", "sve");
> +        assert_has_feature(qts, "max", "sve128");
>          assert_has_feature(qts, "cortex-a57", "pmu");
>          assert_has_feature(qts, "cortex-a57", "aarch64");
>  
> +        sve_tests_default(qts, "max");
> +
>          /* Test that features that depend on KVM generate errors without. */
>          assert_error(qts, "max",
>                       "'aarch64' feature cannot be disabled "
> @@ -213,6 +423,13 @@ int main(int argc, char **argv)
>      qtest_add_data_func("/arm/query-cpu-model-expansion",
>                          NULL, test_query_cpu_model_expansion);
>  
> +    if (g_str_equal(qtest_get_arch(), "aarch64")) {
> +        qtest_add_data_func("/arm/max/query-cpu-model-expansion/sve-max-vq-8",
> +                            NULL, sve_tests_sve_max_vq_8);
> +        qtest_add_data_func("/arm/max/query-cpu-model-expansion/sve-off",
> +                            NULL, sve_tests_sve_off);
> +    }
> +
>      if (kvm_available) {
>          qtest_add_data_func("/arm/kvm/query-cpu-model-expansion",
>                              NULL, test_query_cpu_model_expansion_kvm);
> 

Re: [Qemu-devel] [PATCH v2 07/14] target/arm/cpu64: max cpu: Introduce sve<vl-bits> properties

[Andrew Jones]

On Wed, Jun 26, 2019 at 06:56:54PM +0200, Auger Eric wrote:
> > diff --git a/target/arm/helper.c b/target/arm/helper.c
> > index f500ccb6d31b..b7b719dba57f 100644
> > --- a/target/arm/helper.c
> > +++ b/target/arm/helper.c
> > @@ -5324,7 +5324,16 @@ static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
> >  
> >      /* Bits other than [3:0] are RAZ/WI.  */
> >      QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
> > -    raw_write(env, ri, value & 0xf);
> > +    value &= 0xf;
> > +
> > +    if (value) {
> > +        /* get next vq that is smaller than or equal to value's vq */
> > +        uint32_t vq = value + 1;
> > +        vq = arm_cpu_vq_map_next_smaller(cpu, vq + 1);
> > +        value = vq - 1;
> spec says:
> 
> "if an unsupported vector length is requested in ZCR_ELx, the
> implementation is required to select the largest
> supported vector length that is less than the requested length. This
> does not alter the value of ZCR_ELx.LEN.
> "
> 
> So I understand the value written in the reg should not be unmodified.
>

Sorry, I can't parse what you're trying to tell me here. Here we have
to write 'value', because that's what the guest is trying to do. As the
spec says in your quote, we have to pick the length the guest wants, or
the next smaller valid one, so that's what the code above does. So are
you just stating that you agree with this hunk of the code?

Thanks,
drew

Re: [Qemu-devel] [PATCH v2 07/14] target/arm/cpu64: max cpu: Introduce sve<vl-bits> properties

[Auger Eric]

Hi,

On 6/27/19 12:46 PM, Andrew Jones wrote:
> On Wed, Jun 26, 2019 at 06:56:54PM +0200, Auger Eric wrote:
>>> diff --git a/target/arm/helper.c b/target/arm/helper.c
>>> index f500ccb6d31b..b7b719dba57f 100644
>>> --- a/target/arm/helper.c
>>> +++ b/target/arm/helper.c
>>> @@ -5324,7 +5324,16 @@ static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
>>>  
>>>      /* Bits other than [3:0] are RAZ/WI.  */
>>>      QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
>>> -    raw_write(env, ri, value & 0xf);
>>> +    value &= 0xf;
>>> +
>>> +    if (value) {
>>> +        /* get next vq that is smaller than or equal to value's vq */
>>> +        uint32_t vq = value + 1;
>>> +        vq = arm_cpu_vq_map_next_smaller(cpu, vq + 1);
>>> +        value = vq - 1;
>> spec says:
>>
>> "if an unsupported vector length is requested in ZCR_ELx, the
>> implementation is required to select the largest
>> supported vector length that is less than the requested length. This
>> does not alter the value of ZCR_ELx.LEN.
>> "
>>
>> So I understand the value written in the reg should not be unmodified.
>>
> 
> Sorry, I can't parse what you're trying to tell me here. Here we have
> to write 'value', because that's what the guest is trying to do. As the
> spec says in your quote, we have to pick the length the guest wants, or
> the next smaller valid one, so that's what the code above does. So are
> you just stating that you agree with this hunk of the code?
What we are writing into the reg is arm_cpu_vq_map_next_smaller(cpu, vq
+ 1) -1. Maybe I misunderstand the whole wording but I would have
expected the original unmodified value to be written in the reg instead?

Thanks

Eric
> 
> Thanks,
> drew
> 

Re: [Qemu-devel] [PATCH v2 07/14] target/arm/cpu64: max cpu: Introduce sve<vl-bits> properties

[Andrew Jones]

On Thu, Jun 27, 2019 at 01:00:27PM +0200, Auger Eric wrote:
> Hi,
> 
> On 6/27/19 12:46 PM, Andrew Jones wrote:
> > On Wed, Jun 26, 2019 at 06:56:54PM +0200, Auger Eric wrote:
> >>> diff --git a/target/arm/helper.c b/target/arm/helper.c
> >>> index f500ccb6d31b..b7b719dba57f 100644
> >>> --- a/target/arm/helper.c
> >>> +++ b/target/arm/helper.c
> >>> @@ -5324,7 +5324,16 @@ static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
> >>>  
> >>>      /* Bits other than [3:0] are RAZ/WI.  */
> >>>      QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
> >>> -    raw_write(env, ri, value & 0xf);
> >>> +    value &= 0xf;
> >>> +
> >>> +    if (value) {
> >>> +        /* get next vq that is smaller than or equal to value's vq */
> >>> +        uint32_t vq = value + 1;
> >>> +        vq = arm_cpu_vq_map_next_smaller(cpu, vq + 1);
> >>> +        value = vq - 1;
> >> spec says:
> >>
> >> "if an unsupported vector length is requested in ZCR_ELx, the
> >> implementation is required to select the largest
> >> supported vector length that is less than the requested length. This
> >> does not alter the value of ZCR_ELx.LEN.
> >> "
> >>
> >> So I understand the value written in the reg should not be unmodified.
> >>
> > 
> > Sorry, I can't parse what you're trying to tell me here. Here we have
> > to write 'value', because that's what the guest is trying to do. As the
> > spec says in your quote, we have to pick the length the guest wants, or
> > the next smaller valid one, so that's what the code above does. So are
> > you just stating that you agree with this hunk of the code?
> What we are writing into the reg is arm_cpu_vq_map_next_smaller(cpu, vq
> + 1) -1. Maybe I misunderstand the whole wording but I would have
> expected the original unmodified value to be written in the reg instead?

Hmm... So maybe we need more changes to the emulation in order for it to
have an acting value and a register value? Maybe Richard knows what we
should do here.

Thanks,
drew

Re: [Qemu-devel] [PATCH v2 07/14] target/arm/cpu64: max cpu: Introduce sve<vl-bits> properties

[Dave Martin]

On Thu, Jun 27, 2019 at 12:47:01PM +0100, Andrew Jones wrote:
> On Thu, Jun 27, 2019 at 01:00:27PM +0200, Auger Eric wrote:
> > Hi,
> > 
> > On 6/27/19 12:46 PM, Andrew Jones wrote:
> > > On Wed, Jun 26, 2019 at 06:56:54PM +0200, Auger Eric wrote:
> > >>> diff --git a/target/arm/helper.c b/target/arm/helper.c
> > >>> index f500ccb6d31b..b7b719dba57f 100644
> > >>> --- a/target/arm/helper.c
> > >>> +++ b/target/arm/helper.c
> > >>> @@ -5324,7 +5324,16 @@ static void zcr_write(CPUARMState *env, const ARMCPRegInfo *ri,
> > >>>  
> > >>>      /* Bits other than [3:0] are RAZ/WI.  */
> > >>>      QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
> > >>> -    raw_write(env, ri, value & 0xf);
> > >>> +    value &= 0xf;
> > >>> +
> > >>> +    if (value) {
> > >>> +        /* get next vq that is smaller than or equal to value's vq */
> > >>> +        uint32_t vq = value + 1;
> > >>> +        vq = arm_cpu_vq_map_next_smaller(cpu, vq + 1);
> > >>> +        value = vq - 1;
> > >> spec says:
> > >>
> > >> "if an unsupported vector length is requested in ZCR_ELx, the
> > >> implementation is required to select the largest
> > >> supported vector length that is less than the requested length. This
> > >> does not alter the value of ZCR_ELx.LEN.
> > >> "
> > >>
> > >> So I understand the value written in the reg should not be unmodified.
> > >>
> > > 
> > > Sorry, I can't parse what you're trying to tell me here. Here we have
> > > to write 'value', because that's what the guest is trying to do. As the
> > > spec says in your quote, we have to pick the length the guest wants, or
> > > the next smaller valid one, so that's what the code above does. So are
> > > you just stating that you agree with this hunk of the code?
> > What we are writing into the reg is arm_cpu_vq_map_next_smaller(cpu, vq
> > + 1) -1. Maybe I misunderstand the whole wording but I would have
> > expected the original unmodified value to be written in the reg instead?
> 
> Hmm... So maybe we need more changes to the emulation in order for it to
> have an acting value and a register value? Maybe Richard knows what we
> should do here.

The "effective" value of an individual ZCR_ELx.LEN field is a bit
nonsensical: the effective vector length comes from the minimum
LEN value at any relevant EL (depending on which ELs are implemented,
which security state you're in, VHE controls etc.).

This is tedious to compute, so I'd expect you want to cache it,
recomputing it only when a ZCR_ELx.LEN changes or when you switch to a
different EL.


The architecture says:

"For all purposes other than returning the result of a direct read of
ZCR_EL1 then this field behaves as if it is set to the minimum of the
stored value and the constrained length inherited from more privileged
Exception levels in the current Security state, rounded down to the
nearest implemented vector length."

I think the behaviour of a direct read is implied: the LEN bits yielded
by an MRS should contain exactly what was last written to them via MSR.

Cheers
---Dave

Re: [Qemu-devel] [PATCH v2 07/14] target/arm/cpu64: max cpu: Introduce sve<vl-bits> properties

[Richard Henderson]

On 6/27/19 5:16 PM, Dave Martin wrote:
> The architecture says:
> 
> "For all purposes other than returning the result of a direct read of
> ZCR_EL1 then this field behaves as if it is set to the minimum of the
> stored value and the constrained length inherited from more privileged
> Exception levels in the current Security state, rounded down to the
> nearest implemented vector length."
> 
> I think the behaviour of a direct read is implied: the LEN bits yielded
> by an MRS should contain exactly what was last written to them via MSR.

I agree.

Moreover, the value written to ZCR_ELx.LEN should not be directly adjusted
because the effective value also depends on ZCR_EL(x+1).LEN, and if the
higher-level EL register changes, the lower-level EL must see the effect.

The function that should be modified instead is sve_zcr_len_for_el().


r~

Re: [Qemu-devel] [PATCH v2 07/14] target/arm/cpu64: max cpu: Introduce sve<vl-bits> properties

[Richard Henderson]

On 6/21/19 6:34 PM, Andrew Jones wrote:
> +    /*
> +     * In sve_vq_map each set bit is a supported vector length of
> +     * (bit-number + 1) * 16 bytes, i.e. each bit number + 1 is the vector
> +     * length in quadwords. We need a map size twice the maximum
> +     * quadword length though because we use two bits for each vector
> +     * length in order to track three states: uninitialized, off, and on.
> +     */
> +    DECLARE_BITMAP(sve_vq_map, ARM_MAX_VQ * 2);

I don't see that having one continuous bitmap is more convenient than two.
Indeed, there appear to be several places that would be clearer with two.

> +static arm_vq_state arm_cpu_vq_map_get(ARMCPU *cpu, int vq)
> +{
> +    assert(vq <= ARM_MAX_VQ);
> +
> +    return test_bit(vq - 1, cpu->sve_vq_map) |
> +           test_bit(vq - 1 + ARM_MAX_VQ, cpu->sve_vq_map) << 1;
> +}

Neither easier nor more complex w/ one or two bitmaps.

> +static void arm_cpu_vq_map_init(ARMCPU *cpu)
> +{
> +    bitmap_zero(cpu->sve_vq_map, ARM_MAX_VQ * 2);
> +    bitmap_set(cpu->sve_vq_map, ARM_MAX_VQ, ARM_MAX_VQ);
> +}

Clearer with two bitmaps.

	bitmap_zero(cpu->sve_vq_map, ARM_MAX_VQ);
	bitmap_set(cpu->sve_vq_uninit_map, 0, ARM_MAX_VQ);

> +static bool arm_cpu_vq_map_is_finalized(ARMCPU *cpu)
> +{
> +    DECLARE_BITMAP(map, ARM_MAX_VQ * 2);
> +
> +    bitmap_zero(map, ARM_MAX_VQ * 2);
> +    bitmap_set(map, ARM_MAX_VQ, ARM_MAX_VQ);
> +    bitmap_and(map, map, cpu->sve_vq_map, ARM_MAX_VQ * 2);
> +
> +    return bitmap_empty(map, ARM_MAX_VQ * 2);
> +}

Definitely clearer w/ 2 bitmaps,

	return bitmap_empty(cpu->sve_vq_uninit_map);


As for how sve-max-vq=Y and sveX={on,off} interoperate...  I wonder if we can
just remove cpu->sve_max_vq.  That might simplify your code a bit.

What if sve-max-vq=Y "expands" to

	for (X = 1; X <= Y; X++) { sve(X*128)=on }

Then you've got a reasonable in-order definition of how those two sets of
switches interoperate.

The uses of cpu->sve_max_vq within cpu.c and cpu64.c are all initialization
stuff that you're replacing.

The use within sve_zcr_len_for_el can be replaced by AVR_MAX_VQ.  Your "select
supported vector size not larger than" code goes at the end of that function,
such that we select a supported maximum no larger than the raw .LEN values.

The use within aarch64_sve_narrow_vq should in fact assert that the given vq is
set within cpu->sve_vq_map.


r~

Re: [Qemu-devel] [PATCH v2 07/14] target/arm/cpu64: max cpu: Introduce sve<vl-bits> properties

[Andrew Jones]

On Thu, Jun 27, 2019 at 06:49:02PM +0200, Richard Henderson wrote:
> On 6/21/19 6:34 PM, Andrew Jones wrote:
> > +    /*
> > +     * In sve_vq_map each set bit is a supported vector length of
> > +     * (bit-number + 1) * 16 bytes, i.e. each bit number + 1 is the vector
> > +     * length in quadwords. We need a map size twice the maximum
> > +     * quadword length though because we use two bits for each vector
> > +     * length in order to track three states: uninitialized, off, and on.
> > +     */
> > +    DECLARE_BITMAP(sve_vq_map, ARM_MAX_VQ * 2);
> 
> I don't see that having one continuous bitmap is more convenient than two.
> Indeed, there appear to be several places that would be clearer with two.
> 
> > +static arm_vq_state arm_cpu_vq_map_get(ARMCPU *cpu, int vq)
> > +{
> > +    assert(vq <= ARM_MAX_VQ);
> > +
> > +    return test_bit(vq - 1, cpu->sve_vq_map) |
> > +           test_bit(vq - 1 + ARM_MAX_VQ, cpu->sve_vq_map) << 1;
> > +}
> 
> Neither easier nor more complex w/ one or two bitmaps.
> 
> > +static void arm_cpu_vq_map_init(ARMCPU *cpu)
> > +{
> > +    bitmap_zero(cpu->sve_vq_map, ARM_MAX_VQ * 2);
> > +    bitmap_set(cpu->sve_vq_map, ARM_MAX_VQ, ARM_MAX_VQ);
> > +}
> 
> Clearer with two bitmaps.
> 
> 	bitmap_zero(cpu->sve_vq_map, ARM_MAX_VQ);
> 	bitmap_set(cpu->sve_vq_uninit_map, 0, ARM_MAX_VQ);
> 
> > +static bool arm_cpu_vq_map_is_finalized(ARMCPU *cpu)
> > +{
> > +    DECLARE_BITMAP(map, ARM_MAX_VQ * 2);
> > +
> > +    bitmap_zero(map, ARM_MAX_VQ * 2);
> > +    bitmap_set(map, ARM_MAX_VQ, ARM_MAX_VQ);
> > +    bitmap_and(map, map, cpu->sve_vq_map, ARM_MAX_VQ * 2);
> > +
> > +    return bitmap_empty(map, ARM_MAX_VQ * 2);
> > +}
> 
> Definitely clearer w/ 2 bitmaps,
> 
> 	return bitmap_empty(cpu->sve_vq_uninit_map);

I guess I don't have a strong opinion on one or two bitmaps. I'm not a big
fan of adding temporary variables to data structures (even if the same
amount of storage is being allocated a different way), but I can change
this for v3.

> 
> 
> As for how sve-max-vq=Y and sveX={on,off} interoperate...  I wonder if we can
> just remove cpu->sve_max_vq.  That might simplify your code a bit.
> 
> What if sve-max-vq=Y "expands" to
> 
> 	for (X = 1; X <= Y; X++) { sve(X*128)=on }
> 
> Then you've got a reasonable in-order definition of how those two sets of
> switches interoperate.
> 
> The uses of cpu->sve_max_vq within cpu.c and cpu64.c are all initialization
> stuff that you're replacing.

Hmm, I can look at removing cpu->sve_max_vq, by keeping the property and
expanding it, as you suggest. However I still need a few initialization
states to track what the default vq value should be, so I'm not sure
how I'd implement that without it or some other cpu state, which would
be another temporary cpu state member. Also, while it's true we can always
get the max vq with next-smaller(ARM_MAX_VQ + 1), having it cached in
cpu->sve_max_vq is convenient. That said, I think we'd rather keep it.

> 
> The use within sve_zcr_len_for_el can be replaced by AVR_MAX_VQ.  Your "select
> supported vector size not larger than" code goes at the end of that function,
> such that we select a supported maximum no larger than the raw .LEN values.
> 
> The use within aarch64_sve_narrow_vq should in fact assert that the given vq is
> set within cpu->sve_vq_map.

Yeah, I'm glad Eric pointed out that I had a bug there in the emulation.
I'll get that fixed for v3.

Thanks,
drew

Re: [Qemu-devel] [PATCH v2 07/14] target/arm/cpu64: max cpu: Introduce sve<vl-bits> properties

[Andrew Jones]

On Fri, Jun 28, 2019 at 09:27:39AM +0200, Andrew Jones wrote:
> On Thu, Jun 27, 2019 at 06:49:02PM +0200, Richard Henderson wrote:
> > On 6/21/19 6:34 PM, Andrew Jones wrote:
> > > +    /*
> > > +     * In sve_vq_map each set bit is a supported vector length of
> > > +     * (bit-number + 1) * 16 bytes, i.e. each bit number + 1 is the vector
> > > +     * length in quadwords. We need a map size twice the maximum
> > > +     * quadword length though because we use two bits for each vector
> > > +     * length in order to track three states: uninitialized, off, and on.
> > > +     */
> > > +    DECLARE_BITMAP(sve_vq_map, ARM_MAX_VQ * 2);
> > 
> > I don't see that having one continuous bitmap is more convenient than two.
> > Indeed, there appear to be several places that would be clearer with two.
> > 
> > > +static arm_vq_state arm_cpu_vq_map_get(ARMCPU *cpu, int vq)
> > > +{
> > > +    assert(vq <= ARM_MAX_VQ);
> > > +
> > > +    return test_bit(vq - 1, cpu->sve_vq_map) |
> > > +           test_bit(vq - 1 + ARM_MAX_VQ, cpu->sve_vq_map) << 1;
> > > +}
> > 
> > Neither easier nor more complex w/ one or two bitmaps.
> > 
> > > +static void arm_cpu_vq_map_init(ARMCPU *cpu)
> > > +{
> > > +    bitmap_zero(cpu->sve_vq_map, ARM_MAX_VQ * 2);
> > > +    bitmap_set(cpu->sve_vq_map, ARM_MAX_VQ, ARM_MAX_VQ);
> > > +}
> > 
> > Clearer with two bitmaps.
> > 
> > 	bitmap_zero(cpu->sve_vq_map, ARM_MAX_VQ);
> > 	bitmap_set(cpu->sve_vq_uninit_map, 0, ARM_MAX_VQ);
> > 
> > > +static bool arm_cpu_vq_map_is_finalized(ARMCPU *cpu)
> > > +{
> > > +    DECLARE_BITMAP(map, ARM_MAX_VQ * 2);
> > > +
> > > +    bitmap_zero(map, ARM_MAX_VQ * 2);
> > > +    bitmap_set(map, ARM_MAX_VQ, ARM_MAX_VQ);
> > > +    bitmap_and(map, map, cpu->sve_vq_map, ARM_MAX_VQ * 2);
> > > +
> > > +    return bitmap_empty(map, ARM_MAX_VQ * 2);
> > > +}
> > 
> > Definitely clearer w/ 2 bitmaps,
> > 
> > 	return bitmap_empty(cpu->sve_vq_uninit_map);
> 
> I guess I don't have a strong opinion on one or two bitmaps. I'm not a big
> fan of adding temporary variables to data structures (even if the same
> amount of storage is being allocated a different way), but I can change
> this for v3.

On second thought, since in this case there is storage savings (one less
long), then I'd rather we keep it a single bitmap. Maybe I can just add
some comments to these bitmap operations?

Thanks,
drew

Re: [Qemu-devel] [PATCH v2 07/14] target/arm/cpu64: max cpu: Introduce sve<vl-bits> properties

[Richard Henderson]

On 6/28/19 9:27 AM, Andrew Jones wrote:
> Also, while it's true we can always
> get the max vq with next-smaller(ARM_MAX_VQ + 1), having it cached in
> cpu->sve_max_vq is convenient. That said, I think we'd rather keep it.

When is it convenient, and for what?

Certainly the only thing that we check after boot is the largest enabled vq not
larger than x.  And for that I don't see that sve_max_vq is relevant at all.

Oh, something that we should also think about is -cpu foo, where foo is one of
the Fujitsu thingumies.  We should be able to default sve_vq_map to that which
a real bit of hw actually supports.  I, for one, welcome our typedef long
overlords.  ;-)


r~

Re: [Qemu-devel] [PATCH v2 07/14] target/arm/cpu64: max cpu: Introduce sve<vl-bits> properties

[Andrew Jones]

On Sat, Jun 29, 2019 at 02:10:28AM +0200, Richard Henderson wrote:
> On 6/28/19 9:27 AM, Andrew Jones wrote:
> > Also, while it's true we can always
> > get the max vq with next-smaller(ARM_MAX_VQ + 1), having it cached in
> > cpu->sve_max_vq is convenient. That said, I think we'd rather keep it.
> 
> When is it convenient, and for what?

It's used for the upper boundary in several loops in target/arm/kvm64.c

> 
> Certainly the only thing that we check after boot is the largest enabled vq not
> larger than x.  And for that I don't see that sve_max_vq is relevant at all.
> 
> Oh, something that we should also think about is -cpu foo, where foo is one of
> the Fujitsu thingumies.  We should be able to default sve_vq_map to that which
> a real bit of hw actually supports.  I, for one, welcome our typedef long
> overlords.  ;-)

I don't think we need to implement an A64FX cpu model with this series,
although that's a nice idea for people that focus on TCG to do as a
follow-up series. This series fully enables the guest to use SVE on the
A64FX when KVM is enabled, without the need to special case it in any way.

Thanks,
drew

[Qemu-devel] [PATCH v2 08/14] target/arm/kvm64: Fix error returns

[Andrew Jones]

A couple return -EINVAL's forgot their '-'s.

Signed-off-by: Andrew Jones <drjones@redhat.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
---
 target/arm/kvm64.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
index 45ccda589903..9ca9a0ce821d 100644
--- a/target/arm/kvm64.c
+++ b/target/arm/kvm64.c
@@ -856,7 +856,7 @@ int kvm_arch_put_registers(CPUState *cs, int level)
     write_cpustate_to_list(cpu, true);
 
     if (!write_list_to_kvmstate(cpu, level)) {
-        return EINVAL;
+        return -EINVAL;
     }
 
     kvm_arm_sync_mpstate_to_kvm(cpu);
@@ -997,7 +997,7 @@ int kvm_arch_get_registers(CPUState *cs)
     }
 
     if (!write_kvmstate_to_list(cpu)) {
-        return EINVAL;
+        return -EINVAL;
     }
     /* Note that it's OK to have registers which aren't in CPUState,
      * so we can ignore a failure return here.
-- 
2.20.1

Re: [Qemu-devel] [PATCH v2 08/14] target/arm/kvm64: Fix error returns

[Richard Henderson]

On 6/21/19 6:34 PM, Andrew Jones wrote:
> A couple return -EINVAL's forgot their '-'s.
> 
> Signed-off-by: Andrew Jones <drjones@redhat.com>
> Reviewed-by: Eric Auger <eric.auger@redhat.com>
> ---
>  target/arm/kvm64.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>


r~

Re: [Qemu-devel] [PATCH v2 08/14] target/arm/kvm64: Fix error returns

[Richard Henderson]

On 6/21/19 6:34 PM, Andrew Jones wrote:
> A couple return -EINVAL's forgot their '-'s.
> 
> Signed-off-by: Andrew Jones <drjones@redhat.com>
> Reviewed-by: Eric Auger <eric.auger@redhat.com>
> ---
>  target/arm/kvm64.c | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>


r~

[Qemu-devel] [PATCH v2 09/14] target/arm/kvm64: Move the get/put of fpsimd registers out

[Andrew Jones]

Move the getting/putting of the fpsimd registers out of
kvm_arch_get/put_registers() into their own helper functions
to prepare for alternatively getting/putting SVE registers.

No functional change.

Signed-off-by: Andrew Jones <drjones@redhat.com>
Reviewed-by: Eric Auger <eric.auger@redhat.com>
---
 target/arm/kvm64.c | 148 +++++++++++++++++++++++++++------------------
 1 file changed, 88 insertions(+), 60 deletions(-)

diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
index 9ca9a0ce821d..a2485d447e6a 100644
--- a/target/arm/kvm64.c
+++ b/target/arm/kvm64.c
@@ -721,13 +721,53 @@ int kvm_arm_cpreg_level(uint64_t regidx)
 #define AARCH64_SIMD_CTRL_REG(x)   (KVM_REG_ARM64 | KVM_REG_SIZE_U32 | \
                  KVM_REG_ARM_CORE | KVM_REG_ARM_CORE_REG(x))
 
+static int kvm_arch_put_fpsimd(CPUState *cs)
+{
+    ARMCPU *cpu = ARM_CPU(cs);
+    CPUARMState *env = &cpu->env;
+    struct kvm_one_reg reg;
+    uint32_t fpr;
+    int i, ret;
+
+    for (i = 0; i < 32; i++) {
+        uint64_t *q = aa64_vfp_qreg(env, i);
+#ifdef HOST_WORDS_BIGENDIAN
+        uint64_t fp_val[2] = { q[1], q[0] };
+        reg.addr = (uintptr_t)fp_val;
+#else
+        reg.addr = (uintptr_t)q;
+#endif
+        reg.id = AARCH64_SIMD_CORE_REG(fp_regs.vregs[i]);
+        ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
+        if (ret) {
+            return ret;
+        }
+    }
+
+    reg.addr = (uintptr_t)(&fpr);
+    fpr = vfp_get_fpsr(env);
+    reg.id = AARCH64_SIMD_CTRL_REG(fp_regs.fpsr);
+    ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
+    if (ret) {
+        return ret;
+    }
+
+    reg.addr = (uintptr_t)(&fpr);
+    fpr = vfp_get_fpcr(env);
+    reg.id = AARCH64_SIMD_CTRL_REG(fp_regs.fpcr);
+    ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
+    if (ret) {
+        return ret;
+    }
+
+    return 0;
+}
+
 int kvm_arch_put_registers(CPUState *cs, int level)
 {
     struct kvm_one_reg reg;
-    uint32_t fpr;
     uint64_t val;
-    int i;
-    int ret;
+    int i, ret;
     unsigned int el;
 
     ARMCPU *cpu = ARM_CPU(cs);
@@ -817,33 +857,7 @@ int kvm_arch_put_registers(CPUState *cs, int level)
         }
     }
 
-    /* Advanced SIMD and FP registers. */
-    for (i = 0; i < 32; i++) {
-        uint64_t *q = aa64_vfp_qreg(env, i);
-#ifdef HOST_WORDS_BIGENDIAN
-        uint64_t fp_val[2] = { q[1], q[0] };
-        reg.addr = (uintptr_t)fp_val;
-#else
-        reg.addr = (uintptr_t)q;
-#endif
-        reg.id = AARCH64_SIMD_CORE_REG(fp_regs.vregs[i]);
-        ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
-        if (ret) {
-            return ret;
-        }
-    }
-
-    reg.addr = (uintptr_t)(&fpr);
-    fpr = vfp_get_fpsr(env);
-    reg.id = AARCH64_SIMD_CTRL_REG(fp_regs.fpsr);
-    ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
-    if (ret) {
-        return ret;
-    }
-
-    fpr = vfp_get_fpcr(env);
-    reg.id = AARCH64_SIMD_CTRL_REG(fp_regs.fpcr);
-    ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
+    ret = kvm_arch_put_fpsimd(cs);
     if (ret) {
         return ret;
     }
@@ -864,14 +878,54 @@ int kvm_arch_put_registers(CPUState *cs, int level)
     return ret;
 }
 
+static int kvm_arch_get_fpsimd(CPUState *cs)
+{
+    ARMCPU *cpu = ARM_CPU(cs);
+    CPUARMState *env = &cpu->env;
+    struct kvm_one_reg reg;
+    uint32_t fpr;
+    int i, ret;
+
+    for (i = 0; i < 32; i++) {
+        uint64_t *q = aa64_vfp_qreg(env, i);
+        reg.id = AARCH64_SIMD_CORE_REG(fp_regs.vregs[i]);
+        reg.addr = (uintptr_t)q;
+        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
+        if (ret) {
+            return ret;
+        } else {
+#ifdef HOST_WORDS_BIGENDIAN
+            uint64_t t;
+            t = q[0], q[0] = q[1], q[1] = t;
+#endif
+        }
+    }
+
+    reg.addr = (uintptr_t)(&fpr);
+    reg.id = AARCH64_SIMD_CTRL_REG(fp_regs.fpsr);
+    ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
+    if (ret) {
+        return ret;
+    }
+    vfp_set_fpsr(env, fpr);
+
+    reg.addr = (uintptr_t)(&fpr);
+    reg.id = AARCH64_SIMD_CTRL_REG(fp_regs.fpcr);
+    ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
+    if (ret) {
+        return ret;
+    }
+    vfp_set_fpcr(env, fpr);
+
+    return 0;
+}
+
 int kvm_arch_get_registers(CPUState *cs)
 {
     struct kvm_one_reg reg;
     uint64_t val;
-    uint32_t fpr;
     unsigned int el;
-    int i;
-    int ret;
+    int i, ret;
 
     ARMCPU *cpu = ARM_CPU(cs);
     CPUARMState *env = &cpu->env;
@@ -960,36 +1014,10 @@ int kvm_arch_get_registers(CPUState *cs)
         env->spsr = env->banked_spsr[i];
     }
 
-    /* Advanced SIMD and FP registers */
-    for (i = 0; i < 32; i++) {
-        uint64_t *q = aa64_vfp_qreg(env, i);
-        reg.id = AARCH64_SIMD_CORE_REG(fp_regs.vregs[i]);
-        reg.addr = (uintptr_t)q;
-        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
-        if (ret) {
-            return ret;
-        } else {
-#ifdef HOST_WORDS_BIGENDIAN
-            uint64_t t;
-            t = q[0], q[0] = q[1], q[1] = t;
-#endif
-        }
-    }
-
-    reg.addr = (uintptr_t)(&fpr);
-    reg.id = AARCH64_SIMD_CTRL_REG(fp_regs.fpsr);
-    ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
-    if (ret) {
-        return ret;
-    }
-    vfp_set_fpsr(env, fpr);
-
-    reg.id = AARCH64_SIMD_CTRL_REG(fp_regs.fpcr);
-    ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
+    ret = kvm_arch_get_fpsimd(cs);
     if (ret) {
         return ret;
     }
-    vfp_set_fpcr(env, fpr);
 
     ret = kvm_get_vcpu_events(cpu);
     if (ret) {
-- 
2.20.1

Re: [Qemu-devel] [PATCH v2 09/14] target/arm/kvm64: Move the get/put of fpsimd registers out

[Richard Henderson]

On 6/21/19 6:34 PM, Andrew Jones wrote:
> Move the getting/putting of the fpsimd registers out of
> kvm_arch_get/put_registers() into their own helper functions
> to prepare for alternatively getting/putting SVE registers.
> 
> No functional change.
> 
> Signed-off-by: Andrew Jones <drjones@redhat.com>
> Reviewed-by: Eric Auger <eric.auger@redhat.com>
> ---
>  target/arm/kvm64.c | 148 +++++++++++++++++++++++++++------------------
>  1 file changed, 88 insertions(+), 60 deletions(-)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>


r~

[Qemu-devel] [PATCH v2 10/14] target/arm/kvm64: Add kvm_arch_get/put_sve

[Andrew Jones]

These are the SVE equivalents to kvm_arch_get/put_fpsimd. Note, the
swabbing is different than it is for fpsmid because the vector format
is a little-endian stream of words.

Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 target/arm/kvm64.c | 135 +++++++++++++++++++++++++++++++++++++++++++--
 1 file changed, 131 insertions(+), 4 deletions(-)

diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
index a2485d447e6a..706541327491 100644
--- a/target/arm/kvm64.c
+++ b/target/arm/kvm64.c
@@ -673,11 +673,12 @@ int kvm_arch_destroy_vcpu(CPUState *cs)
 bool kvm_arm_reg_syncs_via_cpreg_list(uint64_t regidx)
 {
     /* Return true if the regidx is a register we should synchronize
-     * via the cpreg_tuples array (ie is not a core reg we sync by
-     * hand in kvm_arch_get/put_registers())
+     * via the cpreg_tuples array (ie is not a core or sve reg that
+     * we sync by hand in kvm_arch_get/put_registers())
      */
     switch (regidx & KVM_REG_ARM_COPROC_MASK) {
     case KVM_REG_ARM_CORE:
+    case KVM_REG_ARM64_SVE:
         return false;
     default:
         return true;
@@ -763,6 +764,70 @@ static int kvm_arch_put_fpsimd(CPUState *cs)
     return 0;
 }
 
+/*
+ * If ARM_MAX_VQ is increased to be greater than 16, then we can no
+ * longer hard code slices to 1 in kvm_arch_put/get_sve().
+ */
+QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
+
+static int kvm_arch_put_sve(CPUState *cs)
+{
+    ARMCPU *cpu = ARM_CPU(cs);
+    CPUARMState *env = &cpu->env;
+    struct kvm_one_reg reg;
+    int slices = 1;
+    int i, n, ret;
+
+    for (i = 0; i < slices; i++) {
+        for (n = 0; n < KVM_ARM64_SVE_NUM_ZREGS; n++) {
+            uint64_t *q = aa64_vfp_qreg(env, n);
+#ifdef HOST_WORDS_BIGENDIAN
+            uint64_t d[ARM_MAX_VQ * 2];
+            int j;
+            for (j = 0; j < cpu->sve_max_vq * 2; j++) {
+                d[j] = bswap64(q[j]);
+            }
+            reg.addr = (uintptr_t)d;
+#else
+            reg.addr = (uintptr_t)q;
+#endif
+            reg.id = KVM_REG_ARM64_SVE_ZREG(n, i);
+            ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
+            if (ret) {
+                return ret;
+            }
+        }
+
+        for (n = 0; n < KVM_ARM64_SVE_NUM_PREGS; n++) {
+            uint64_t *q = &env->vfp.pregs[n].p[0];
+#ifdef HOST_WORDS_BIGENDIAN
+            uint64_t d[ARM_MAX_VQ * 2 / 8];
+            int j;
+            for (j = 0; j < cpu->sve_max_vq * 2 / 8; j++) {
+                d[j] = bswap64(q[j]);
+            }
+            reg.addr = (uintptr_t)d;
+#else
+            reg.addr = (uintptr_t)q;
+#endif
+            reg.id = KVM_REG_ARM64_SVE_PREG(n, i);
+            ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
+            if (ret) {
+                return ret;
+            }
+        }
+
+        reg.addr = (uintptr_t)&env->vfp.pregs[FFR_PRED_NUM].p[0];
+        reg.id = KVM_REG_ARM64_SVE_FFR(i);
+        ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
+        if (ret) {
+            return ret;
+        }
+    }
+
+    return 0;
+}
+
 int kvm_arch_put_registers(CPUState *cs, int level)
 {
     struct kvm_one_reg reg;
@@ -857,7 +922,11 @@ int kvm_arch_put_registers(CPUState *cs, int level)
         }
     }
 
-    ret = kvm_arch_put_fpsimd(cs);
+    if (!cpu->sve_max_vq) {
+        ret = kvm_arch_put_fpsimd(cs);
+    } else {
+        ret = kvm_arch_put_sve(cs);
+    }
     if (ret) {
         return ret;
     }
@@ -920,6 +989,60 @@ static int kvm_arch_get_fpsimd(CPUState *cs)
     return 0;
 }
 
+static int kvm_arch_get_sve(CPUState *cs)
+{
+    ARMCPU *cpu = ARM_CPU(cs);
+    CPUARMState *env = &cpu->env;
+    struct kvm_one_reg reg;
+    int slices = 1;
+    int i, n, ret;
+
+    for (i = 0; i < slices; i++) {
+        for (n = 0; n < KVM_ARM64_SVE_NUM_ZREGS; n++) {
+            uint64_t *q = aa64_vfp_qreg(env, n);
+            reg.id = KVM_REG_ARM64_SVE_ZREG(n, i);
+            reg.addr = (uintptr_t)q;
+            ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
+            if (ret) {
+                return ret;
+            } else {
+#ifdef HOST_WORDS_BIGENDIAN
+                int j;
+                for (j = 0; j < cpu->sve_max_vq * 2; j++) {
+                    q[j] = bswap64(q[j]);
+                }
+#endif
+            }
+        }
+
+        for (n = 0; n < KVM_ARM64_SVE_NUM_PREGS; n++) {
+            uint64_t *q = &env->vfp.pregs[n].p[0];
+            reg.id = KVM_REG_ARM64_SVE_PREG(n, i);
+            reg.addr = (uintptr_t)q;
+            ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
+            if (ret) {
+                return ret;
+            } else {
+#ifdef HOST_WORDS_BIGENDIAN
+                int j;
+                for (j = 0; j < cpu->sve_max_vq * 2 / 8; j++) {
+                    q[j] = bswap64(q[j]);
+                }
+#endif
+            }
+        }
+
+        reg.addr = (uintptr_t)&env->vfp.pregs[FFR_PRED_NUM].p[0];
+        reg.id = KVM_REG_ARM64_SVE_FFR(i);
+        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
+        if (ret) {
+            return ret;
+        }
+    }
+
+    return 0;
+}
+
 int kvm_arch_get_registers(CPUState *cs)
 {
     struct kvm_one_reg reg;
@@ -1014,7 +1137,11 @@ int kvm_arch_get_registers(CPUState *cs)
         env->spsr = env->banked_spsr[i];
     }
 
-    ret = kvm_arch_get_fpsimd(cs);
+    if (!cpu->sve_max_vq) {
+        ret = kvm_arch_get_fpsimd(cs);
+    } else {
+        ret = kvm_arch_get_sve(cs);
+    }
     if (ret) {
         return ret;
     }
-- 
2.20.1

Re: [Qemu-devel] [PATCH v2 10/14] target/arm/kvm64: Add kvm_arch_get/put_sve

[Dave Martin]

On Fri, Jun 21, 2019 at 05:34:18PM +0100, Andrew Jones wrote:
> These are the SVE equivalents to kvm_arch_get/put_fpsimd. Note, the
> swabbing is different than it is for fpsmid because the vector format
> is a little-endian stream of words.

Note, on big-endian hosts the FPSIMD view Vn and the SVE view Zn[127:0]
of the FPSIMD/SVE common register bits has the opposite endianness for
SVE_{GET,SET}_ONE_REG.

This only matters if mixing the two views: just from this patch I don't
know whether this is an issue for QEMU or not.

The kernel and gdb were recently found to be broken in this regard for
userspace [1] but the KVM interface should be unaffected.

Cheers
---Dave

[1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/arch/arm64/kernel?id=41040cf7c5f0f26c368bc5d3016fed3a9ca6dba4

Re: [Qemu-devel] [PATCH v2 10/14] target/arm/kvm64: Add kvm_arch_get/put_sve

[Andrew Jones]

On Mon, Jun 24, 2019 at 12:05:35PM +0100, Dave Martin wrote:
> On Fri, Jun 21, 2019 at 05:34:18PM +0100, Andrew Jones wrote:
> > These are the SVE equivalents to kvm_arch_get/put_fpsimd. Note, the
> > swabbing is different than it is for fpsmid because the vector format
> > is a little-endian stream of words.
> 
> Note, on big-endian hosts the FPSIMD view Vn and the SVE view Zn[127:0]
> of the FPSIMD/SVE common register bits has the opposite endianness for
> SVE_{GET,SET}_ONE_REG.
> 
> This only matters if mixing the two views: just from this patch I don't
> know whether this is an issue for QEMU or not.

I don't know either. My experience with the emulation side of QEMU is
mostly the zcr_write tweak in this series. And, TBH, I didn't put too
much thought into the endianness stuff, nor test this series with big
endian.

Hopefully Richard can chime in on this.

Thanks,
drew

> 
> The kernel and gdb were recently found to be broken in this regard for
> userspace [1] but the KVM interface should be unaffected.
> 
> Cheers
> ---Dave
> 
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/arch/arm64/kernel?id=41040cf7c5f0f26c368bc5d3016fed3a9ca6dba4

Re: [Qemu-devel] [PATCH v2 10/14] target/arm/kvm64: Add kvm_arch_get/put_sve

[Dave Martin]

On Mon, Jun 24, 2019 at 12:55:53PM +0100, Andrew Jones wrote:
> On Mon, Jun 24, 2019 at 12:05:35PM +0100, Dave Martin wrote:
> > On Fri, Jun 21, 2019 at 05:34:18PM +0100, Andrew Jones wrote:
> > > These are the SVE equivalents to kvm_arch_get/put_fpsimd. Note, the
> > > swabbing is different than it is for fpsmid because the vector format
> > > is a little-endian stream of words.
> > 
> > Note, on big-endian hosts the FPSIMD view Vn and the SVE view Zn[127:0]
> > of the FPSIMD/SVE common register bits has the opposite endianness for
> > SVE_{GET,SET}_ONE_REG.
> > 
> > This only matters if mixing the two views: just from this patch I don't
> > know whether this is an issue for QEMU or not.
> 
> I don't know either. My experience with the emulation side of QEMU is
> mostly the zcr_write tweak in this series. And, TBH, I didn't put too
> much thought into the endianness stuff, nor test this series with big
> endian.

Neither did I (at least beyond the "does it boot" level) -- hence the
bug ;)

And of course, few people are using big-endian, so nobody complained.

Just flagging it up so it doesn't get missed!

> Hopefully Richard can chime in on this.

It would be interesting to know whether you do hit this issue
somewhere, or whether you have a strong view about the clarification to
the KVM ABI.

Cheers
---Dave

Re: [Qemu-devel] [PATCH v2 10/14] target/arm/kvm64: Add kvm_arch_get/put_sve

[Richard Henderson]

On 6/21/19 6:34 PM, Andrew Jones wrote:
> +/*
> + * If ARM_MAX_VQ is increased to be greater than 16, then we can no
> + * longer hard code slices to 1 in kvm_arch_put/get_sve().
> + */
> +QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);

This seems easy to fix, or simply drop the slices entirely for now, as
otherwise they are a teeny bit confusing.

It's a shame that these slices exist at all.  It seems like the kernel could
use the negotiated max sve size to grab the data all at once.

> +        for (n = 0; n < KVM_ARM64_SVE_NUM_ZREGS; n++) {
> +            uint64_t *q = aa64_vfp_qreg(env, n);
> +#ifdef HOST_WORDS_BIGENDIAN
> +            uint64_t d[ARM_MAX_VQ * 2];
> +            int j;
> +            for (j = 0; j < cpu->sve_max_vq * 2; j++) {
> +                d[j] = bswap64(q[j]);
> +            }
> +            reg.addr = (uintptr_t)d;
> +#else
> +            reg.addr = (uintptr_t)q;
> +#endif
> +            reg.id = KVM_REG_ARM64_SVE_ZREG(n, i);
> +            ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);

It might be worth splitting this...

> +        for (n = 0; n < KVM_ARM64_SVE_NUM_PREGS; n++) {
> +            uint64_t *q = &env->vfp.pregs[n].p[0];
> +#ifdef HOST_WORDS_BIGENDIAN
> +            uint64_t d[ARM_MAX_VQ * 2 / 8];
> +            int j;
> +            for (j = 0; j < cpu->sve_max_vq * 2 / 8; j++) {
> +                d[j] = bswap64(q[j]);
> +            }
> +            reg.addr = (uintptr_t)d;
> +#else
> +            reg.addr = (uintptr_t)q;
> +#endif
> +            reg.id = KVM_REG_ARM64_SVE_PREG(n, i);
> +            ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);

... and this (unified w/ reg + size parameters?) to a function because ...

> +        reg.addr = (uintptr_t)&env->vfp.pregs[FFR_PRED_NUM].p[0];
> +        reg.id = KVM_REG_ARM64_SVE_FFR(i);
> +        ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);

... you forgot to apply the bswap here.

Likewise for the other direction.


r~


PS: It's also tempting to drop the ifdefs and, since we know the host supports
sve instructions, and that the host supports sve_max_vq, do the reformatting as

    uint64_t scratch[ARM_MAX_VQ * 2];
    asm("whilelo  p0.d, xzr, %2\n\t"
        "ld1d     z0.d, p0/z [%1]\n\t"
        "str      z0, [%0]"
        : "=Q"(scratch)
        : "Q"(*aa64_vfp_qreg(env, n)),
          "r"(cpu->sve_max_vq)
        : "p0", "v0");

PPS: Ideally, this would be further cleaned up with acle builtins, but those
are still under development for GCC.

Re: [Qemu-devel] [PATCH v2 10/14] target/arm/kvm64: Add kvm_arch_get/put_sve

[Dave Martin]

On Wed, Jun 26, 2019 at 04:22:34PM +0100, Richard Henderson wrote:
> On 6/21/19 6:34 PM, Andrew Jones wrote:
> > +/*
> > + * If ARM_MAX_VQ is increased to be greater than 16, then we can no
> > + * longer hard code slices to 1 in kvm_arch_put/get_sve().
> > + */
> > +QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
> 
> This seems easy to fix, or simply drop the slices entirely for now, as
> otherwise they are a teeny bit confusing.
> 
> It's a shame that these slices exist at all.  It seems like the kernel could
> use the negotiated max sve size to grab the data all at once.

The aim here was to be forwards compatible while fitting within the
existing ABI.

The ABI doesn't allow variable-sized registers, and if the vq can
someday grow above 16 then the individual registers could become pretty
big.

Inside the kernel, we took the view that if that ever happens, it's
sufficiently far out that we can skip implementing the support today,
providing that the ABI can accommodate the change.

For qemu, if you don't actually care what's in the regs, you could just
enumerate then using KVM_GET_REG_LIST instead of manufacturing the IDs
by hand.  That way, you don't have to care what slices exist.  For
save/restore/migrate purposes, the regs are just data, so that's
probably enough.

Debug, and exchanging vector registers between the guest and, say, and
SMC trapped to userspace, would still need to examine specific regs.
I don't know what QEMU does in this area though.

> > +        for (n = 0; n < KVM_ARM64_SVE_NUM_ZREGS; n++) {
> > +            uint64_t *q = aa64_vfp_qreg(env, n);
> > +#ifdef HOST_WORDS_BIGENDIAN
> > +            uint64_t d[ARM_MAX_VQ * 2];
> > +            int j;
> > +            for (j = 0; j < cpu->sve_max_vq * 2; j++) {
> > +                d[j] = bswap64(q[j]);
> > +            }
> > +            reg.addr = (uintptr_t)d;
> > +#else
> > +            reg.addr = (uintptr_t)q;
> > +#endif
> > +            reg.id = KVM_REG_ARM64_SVE_ZREG(n, i);
> > +            ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
> 
> It might be worth splitting this...
> 
> > +        for (n = 0; n < KVM_ARM64_SVE_NUM_PREGS; n++) {
> > +            uint64_t *q = &env->vfp.pregs[n].p[0];
> > +#ifdef HOST_WORDS_BIGENDIAN
> > +            uint64_t d[ARM_MAX_VQ * 2 / 8];
> > +            int j;
> > +            for (j = 0; j < cpu->sve_max_vq * 2 / 8; j++) {
> > +                d[j] = bswap64(q[j]);
> > +            }
> > +            reg.addr = (uintptr_t)d;
> > +#else
> > +            reg.addr = (uintptr_t)q;
> > +#endif
> > +            reg.id = KVM_REG_ARM64_SVE_PREG(n, i);
> > +            ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);

It's for QEMU to choose, but does it actually matter what byteorder you
store a Z-reg or P-reg in?  Maybe the byteswap isn't really needed.

I don't know how this works when migrating from a little-endian to a
big-endian host or vice-versa (or if that is even supported...)

[...]

Cheers
---Dave

Re: [Qemu-devel] [PATCH v2 10/14] target/arm/kvm64: Add kvm_arch_get/put_sve

[Richard Henderson]

On 6/27/19 12:59 PM, Dave Martin wrote:
>> It's a shame that these slices exist at all.  It seems like the kernel could
>> use the negotiated max sve size to grab the data all at once.
> 
> The aim here was to be forwards compatible while fitting within the
> existing ABI.
> 
> The ABI doesn't allow variable-sized registers, and if the vq can
> someday grow above 16 then the individual registers could become pretty
> big.

The ABI doesn't appear to have fixed sized data blocks.  Since that's the case,
it didn't seem to me that variable sized blocks was so different, given that
the actual size is constrained by the hardware on which we're running.

And if VQ does grow large, then do we really want oodles of syscalls in order
to transfer the data for each register?  With the 9 bits reserved for this
field, we could require a maximum of 1024 syscalls to transfer the whole
register set.

> It's for QEMU to choose, but does it actually matter what byteorder you
> store a Z-reg or P-reg in?  Maybe the byteswap isn't really needed.

I think the only sensible order for the kernel is that in which LDR/STR itself
saves the data.  Which is a byte stream.

Within QEMU, it has so far made sense to keep the data in 64-bit hunks in
host-endian order.  That's how the AdvSIMD code was written originally, and it
turned out to be easy enough to continue that for SVE.

Which does mean that if we want to pass data to the kernel as the
aforementioned byte stream that a big-endian host does need to bswap the data
in 64-bit hunks.

> I don't know how this works when migrating from a little-endian to a
> big-endian host or vice-versa (or if that is even supported...)

The data is stored canonically within the vmsave, so such migrations are
supposed to work.


r~

Re: [Qemu-devel] [PATCH v2 10/14] target/arm/kvm64: Add kvm_arch_get/put_sve

[Dave Martin]

On Thu, Jun 27, 2019 at 12:26:06PM +0100, Richard Henderson wrote:
> On 6/27/19 12:59 PM, Dave Martin wrote:
> >> It's a shame that these slices exist at all.  It seems like the kernel could
> >> use the negotiated max sve size to grab the data all at once.
> > 
> > The aim here was to be forwards compatible while fitting within the
> > existing ABI.
> > 
> > The ABI doesn't allow variable-sized registers, and if the vq can
> > someday grow above 16 then the individual registers could become pretty
> > big.
> 
> The ABI doesn't appear to have fixed sized data blocks.  Since that's
> the case, it didn't seem to me that variable sized blocks was so
> different, given that the actual size is constrained by the hardware
> on which we're running.

I'm not sure what you mean here.

For KVM_GET_ONE_REG, the size is determined by the reg size field in
the register ID, so size is deemed to be a fixed property of a
particular register.

Having the register IDs vary according to the vector length seemed a
step too far.

> And if VQ does grow large, then do we really want oodles of syscalls in order
> to transfer the data for each register?  With the 9 bits reserved for this
> field, we could require a maximum of 1024 syscalls to transfer the whole
> register set.

A save/restore requires oodles of syscalls in any case, and for SVE
there is a rapid dropoff of probabilities: VQ < 16 is much likelier than
VQ == 32 is likelier than VQ == 64 etc.

The reg access API has some shortcomings, and we might find at some
point that the whole thing needs redesigning.

I suppose we could have taken the view that the KVM ABI would not even
try to support VQ > 16 in a forwards compatible way.  In the end we
decided to at least have things workable.

Either way, it's entirely reasonable for userspace not to try to support
additional slices for now.  We'll have plenty of time to plan away
across that bridge when we spot it on the horizon...

> > It's for QEMU to choose, but does it actually matter what byteorder you
> > store a Z-reg or P-reg in?  Maybe the byteswap isn't really needed.
> 
> I think the only sensible order for the kernel is that in which LDR/STR itself
> saves the data.  Which is a byte stream.

We have a choice of STRs though.  Anyway, yes, it is the way it is, now.

> Within QEMU, it has so far made sense to keep the data in 64-bit hunks in
> host-endian order.  That's how the AdvSIMD code was written originally, and it
> turned out to be easy enough to continue that for SVE.

Fair enough.  It's entirely up to QEMU to decide -- I just wanted to
check that there was no misunderstanding about this issue in the ABI.

> Which does mean that if we want to pass data to the kernel as the
> aforementioned byte stream that a big-endian host does need to bswap the data
> in 64-bit hunks.
> 
> > I don't know how this works when migrating from a little-endian to a
> > big-endian host or vice-versa (or if that is even supported...)
> 
> The data is stored canonically within the vmsave, so such migrations are
> supposed to work.

Right, I was wondering about that.  Could be fun to test :)

Cheers
---Dave

Re: [Qemu-devel] [PATCH v2 10/14] target/arm/kvm64: Add kvm_arch_get/put_sve

[Andrew Jones]

On Thu, Jun 27, 2019 at 04:02:24PM +0100, Dave Martin wrote:
> Either way, it's entirely reasonable for userspace not to try to support
> additional slices for now.  We'll have plenty of time to plan away
> across that bridge when we spot it on the horizon...

Which makes me inclined to keep the get/put register code the way it is
in this patch, at least with regards to the hard coded number of slices
and the build-bug. The way it's written (to me) serves to document the
state of things, rather than truly implement anything, but also (to me)
it's easier to understand that code than would be a couple of paragraphs
of actual documentation trying to explain it.

> > Within QEMU, it has so far made sense to keep the data in 64-bit hunks in
> > host-endian order.  That's how the AdvSIMD code was written originally, and it
> > turned out to be easy enough to continue that for SVE.
> 
> Fair enough.  It's entirely up to QEMU to decide -- I just wanted to
> check that there was no misunderstanding about this issue in the ABI.

We do need to use/swap-to host-endian when we implement the monitor's
dump-guest-memory command, at it also creates ELF notes for the general
and VFP (and, coming soon, SVE) registers. Implementing those ELF notes
for SVE is on my TODO, right after this series.

Thanks,
drew

Re: [Qemu-devel] [PATCH v2 10/14] target/arm/kvm64: Add kvm_arch_get/put_sve

[Andrew Jones]

On Wed, Jun 26, 2019 at 05:22:34PM +0200, Richard Henderson wrote:
> On 6/21/19 6:34 PM, Andrew Jones wrote:
> > +/*
> > + * If ARM_MAX_VQ is increased to be greater than 16, then we can no
> > + * longer hard code slices to 1 in kvm_arch_put/get_sve().
> > + */
> > +QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
> 
> This seems easy to fix, or simply drop the slices entirely for now, as
> otherwise they are a teeny bit confusing.

I can do that, but as I replied down thread, I sort of like it this way
for documentation purposes. Anyway, I don't have a strong opinion here,
so I'm happy to make reviewers happy :-)

> 
> It's a shame that these slices exist at all.  It seems like the kernel could
> use the negotiated max sve size to grab the data all at once.
> 
> > +        for (n = 0; n < KVM_ARM64_SVE_NUM_ZREGS; n++) {
> > +            uint64_t *q = aa64_vfp_qreg(env, n);
> > +#ifdef HOST_WORDS_BIGENDIAN
> > +            uint64_t d[ARM_MAX_VQ * 2];
> > +            int j;
> > +            for (j = 0; j < cpu->sve_max_vq * 2; j++) {
> > +                d[j] = bswap64(q[j]);
> > +            }
> > +            reg.addr = (uintptr_t)d;
> > +#else
> > +            reg.addr = (uintptr_t)q;
> > +#endif
> > +            reg.id = KVM_REG_ARM64_SVE_ZREG(n, i);
> > +            ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
> 
> It might be worth splitting this...
> 
> > +        for (n = 0; n < KVM_ARM64_SVE_NUM_PREGS; n++) {
> > +            uint64_t *q = &env->vfp.pregs[n].p[0];
> > +#ifdef HOST_WORDS_BIGENDIAN
> > +            uint64_t d[ARM_MAX_VQ * 2 / 8];
> > +            int j;
> > +            for (j = 0; j < cpu->sve_max_vq * 2 / 8; j++) {
> > +                d[j] = bswap64(q[j]);
> > +            }
> > +            reg.addr = (uintptr_t)d;
> > +#else
> > +            reg.addr = (uintptr_t)q;
> > +#endif
> > +            reg.id = KVM_REG_ARM64_SVE_PREG(n, i);
> > +            ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
> 
> ... and this (unified w/ reg + size parameters?) to a function because ...
> 
> > +        reg.addr = (uintptr_t)&env->vfp.pregs[FFR_PRED_NUM].p[0];
> > +        reg.id = KVM_REG_ARM64_SVE_FFR(i);
> > +        ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
> 
> ... you forgot to apply the bswap here.

Ah, thanks for catching this. I'll fix it for v3, possibly with the
factoring, as you suggest.

> 
> Likewise for the other direction.
> 
> 
> r~
> 
> 
> PS: It's also tempting to drop the ifdefs and, since we know the host supports
> sve instructions, and that the host supports sve_max_vq, do the reformatting as
> 
>     uint64_t scratch[ARM_MAX_VQ * 2];
>     asm("whilelo  p0.d, xzr, %2\n\t"
>         "ld1d     z0.d, p0/z [%1]\n\t"
>         "str      z0, [%0]"
>         : "=Q"(scratch)
>         : "Q"(*aa64_vfp_qreg(env, n)),
>           "r"(cpu->sve_max_vq)
>         : "p0", "v0");

This is nice, but as we don't have any other asm's in this file, I'm
inclined to leave it with the loops/swaps until we can use a builtin,
as you suggest below.

> 
> PPS: Ideally, this would be further cleaned up with acle builtins, but those
> are still under development for GCC.
> 

Thanks,
drew

Re: [Qemu-devel] [PATCH v2 10/14] target/arm/kvm64: Add kvm_arch_get/put_sve

[Auger Eric]

Hi,

On 6/21/19 6:34 PM, Andrew Jones wrote:
> These are the SVE equivalents to kvm_arch_get/put_fpsimd. Note, the
> swabbing is different than it is for fpsmid because the vector format
> is a little-endian stream of words.

some cosmetic changes besides Richard's comments
> 
> Signed-off-by: Andrew Jones <drjones@redhat.com>
> ---
>  target/arm/kvm64.c | 135 +++++++++++++++++++++++++++++++++++++++++++--
>  1 file changed, 131 insertions(+), 4 deletions(-)
> 
> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> index a2485d447e6a..706541327491 100644
> --- a/target/arm/kvm64.c
> +++ b/target/arm/kvm64.c
> @@ -673,11 +673,12 @@ int kvm_arch_destroy_vcpu(CPUState *cs)
>  bool kvm_arm_reg_syncs_via_cpreg_list(uint64_t regidx)
>  {
>      /* Return true if the regidx is a register we should synchronize
> -     * via the cpreg_tuples array (ie is not a core reg we sync by
> -     * hand in kvm_arch_get/put_registers())
> +     * via the cpreg_tuples array (ie is not a core or sve reg that
> +     * we sync by hand in kvm_arch_get/put_registers())
>       */
>      switch (regidx & KVM_REG_ARM_COPROC_MASK) {
>      case KVM_REG_ARM_CORE:
> +    case KVM_REG_ARM64_SVE:
>          return false;
>      default:
>          return true;
> @@ -763,6 +764,70 @@ static int kvm_arch_put_fpsimd(CPUState *cs)
>      return 0;
>  }
>  
> +/*
> + * If ARM_MAX_VQ is increased to be greater than 16, then we can no
> + * longer hard code slices to 1 in kvm_arch_put/get_sve().
> + */
> +QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
if the code is ready to support slices, I guess you could have a define
and compute the slice number from ARM_MAX_VQ?
> +
> +static int kvm_arch_put_sve(CPUState *cs)
> +{
> +    ARMCPU *cpu = ARM_CPU(cs);
> +    CPUARMState *env = &cpu->env;
> +    struct kvm_one_reg reg;
> +    int slices = 1;
> +    int i, n, ret;
> +
> +    for (i = 0; i < slices; i++) {
> +        for (n = 0; n < KVM_ARM64_SVE_NUM_ZREGS; n++) {
> +            uint64_t *q = aa64_vfp_qreg(env, n);
> +#ifdef HOST_WORDS_BIGENDIAN
> +            uint64_t d[ARM_MAX_VQ * 2];
> +            int j;
line to be added
> +            for (j = 0; j < cpu->sve_max_vq * 2; j++) {
> +                d[j] = bswap64(q[j]);
> +            }
> +            reg.addr = (uintptr_t)d;
> +#else
> +            reg.addr = (uintptr_t)q;
> +#endif
> +            reg.id = KVM_REG_ARM64_SVE_ZREG(n, i);
> +            ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
> +            if (ret) {
> +                return ret;
> +            }
> +        }
> +
> +        for (n = 0; n < KVM_ARM64_SVE_NUM_PREGS; n++) {
> +            uint64_t *q = &env->vfp.pregs[n].p[0];
> +#ifdef HOST_WORDS_BIGENDIAN
> +            uint64_t d[ARM_MAX_VQ * 2 / 8];
> +            int j;
line
> +            for (j = 0; j < cpu->sve_max_vq * 2 / 8; j++) {
> +                d[j] = bswap64(q[j]);
> +            }
> +            reg.addr = (uintptr_t)d;
> +#else
> +            reg.addr = (uintptr_t)q;
> +#endif
> +            reg.id = KVM_REG_ARM64_SVE_PREG(n, i);
> +            ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
> +            if (ret) {
> +                return ret;
> +            }
> +        }
> +
> +        reg.addr = (uintptr_t)&env->vfp.pregs[FFR_PRED_NUM].p[0];
> +        reg.id = KVM_REG_ARM64_SVE_FFR(i);
> +        ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
> +        if (ret) {
> +            return ret;
> +        }
> +    }
> +
> +    return 0;
> +}
> +
>  int kvm_arch_put_registers(CPUState *cs, int level)
>  {
>      struct kvm_one_reg reg;
> @@ -857,7 +922,11 @@ int kvm_arch_put_registers(CPUState *cs, int level)
>          }
>      }
>  
> -    ret = kvm_arch_put_fpsimd(cs);
> +    if (!cpu->sve_max_vq) {
> +        ret = kvm_arch_put_fpsimd(cs);
> +    } else {
> +        ret = kvm_arch_put_sve(cs);
> +    }
>      if (ret) {
>          return ret;
>      }
> @@ -920,6 +989,60 @@ static int kvm_arch_get_fpsimd(CPUState *cs)
>      return 0;
>  }
>  
> +static int kvm_arch_get_sve(CPUState *cs)
> +{
> +    ARMCPU *cpu = ARM_CPU(cs);
> +    CPUARMState *env = &cpu->env;
> +    struct kvm_one_reg reg;
> +    int slices = 1;
> +    int i, n, ret;
> +
> +    for (i = 0; i < slices; i++) {
> +        for (n = 0; n < KVM_ARM64_SVE_NUM_ZREGS; n++) {
> +            uint64_t *q = aa64_vfp_qreg(env, n);
extra line needed
> +            reg.id = KVM_REG_ARM64_SVE_ZREG(n, i);
> +            reg.addr = (uintptr_t)q;
> +            ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
> +            if (ret) {
> +                return ret;
> +            } else {> +#ifdef HOST_WORDS_BIGENDIAN
> +                int j;
line
> +                for (j = 0; j < cpu->sve_max_vq * 2; j++) {
> +                    q[j] = bswap64(q[j]);
> +                }
> +#endif
> +            }
> +        }
> +
> +        for (n = 0; n < KVM_ARM64_SVE_NUM_PREGS; n++) {
> +            uint64_t *q = &env->vfp.pregs[n].p[0];
extra line needed
> +            reg.id = KVM_REG_ARM64_SVE_PREG(n, i);
> +            reg.addr = (uintptr_t)q;
> +            ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
> +            if (ret) {
> +                return ret;
> +            } else {> +#ifdef HOST_WORDS_BIGENDIAN
> +                int j;
line
> +                for (j = 0; j < cpu->sve_max_vq * 2 / 8; j++) {
> +                    q[j] = bswap64(q[j]);
> +                }
> +#endif
> +            }
> +        }
> +
> +        reg.addr = (uintptr_t)&env->vfp.pregs[FFR_PRED_NUM].p[0];
> +        reg.id = KVM_REG_ARM64_SVE_FFR(i);
> +        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
> +        if (ret) {
> +            return ret;
> +        }
> +    }
> +
> +    return 0;
> +}
> +
>  int kvm_arch_get_registers(CPUState *cs)
>  {
>      struct kvm_one_reg reg;
> @@ -1014,7 +1137,11 @@ int kvm_arch_get_registers(CPUState *cs)
>          env->spsr = env->banked_spsr[i];
>      }
>  
> -    ret = kvm_arch_get_fpsimd(cs);
> +    if (!cpu->sve_max_vq) {
> +        ret = kvm_arch_get_fpsimd(cs);
> +    } else {
> +        ret = kvm_arch_get_sve(cs);
> +    }
>      if (ret) {
>          return ret;
>      }
> 


Thanks

Eric

Re: [Qemu-devel] [PATCH v2 10/14] target/arm/kvm64: Add kvm_arch_get/put_sve

[Andrew Jones]

On Thu, Jun 27, 2019 at 08:56:21AM +0200, Auger Eric wrote:
> Hi,
> 
> On 6/21/19 6:34 PM, Andrew Jones wrote:
> > These are the SVE equivalents to kvm_arch_get/put_fpsimd. Note, the
> > swabbing is different than it is for fpsmid because the vector format
> > is a little-endian stream of words.
> 
> some cosmetic changes besides Richard's comments
> > 
> > Signed-off-by: Andrew Jones <drjones@redhat.com>
> > ---
> >  target/arm/kvm64.c | 135 +++++++++++++++++++++++++++++++++++++++++++--
> >  1 file changed, 131 insertions(+), 4 deletions(-)
> > 
> > diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> > index a2485d447e6a..706541327491 100644
> > --- a/target/arm/kvm64.c
> > +++ b/target/arm/kvm64.c
> > @@ -673,11 +673,12 @@ int kvm_arch_destroy_vcpu(CPUState *cs)
> >  bool kvm_arm_reg_syncs_via_cpreg_list(uint64_t regidx)
> >  {
> >      /* Return true if the regidx is a register we should synchronize
> > -     * via the cpreg_tuples array (ie is not a core reg we sync by
> > -     * hand in kvm_arch_get/put_registers())
> > +     * via the cpreg_tuples array (ie is not a core or sve reg that
> > +     * we sync by hand in kvm_arch_get/put_registers())
> >       */
> >      switch (regidx & KVM_REG_ARM_COPROC_MASK) {
> >      case KVM_REG_ARM_CORE:
> > +    case KVM_REG_ARM64_SVE:
> >          return false;
> >      default:
> >          return true;
> > @@ -763,6 +764,70 @@ static int kvm_arch_put_fpsimd(CPUState *cs)
> >      return 0;
> >  }
> >  
> > +/*
> > + * If ARM_MAX_VQ is increased to be greater than 16, then we can no
> > + * longer hard code slices to 1 in kvm_arch_put/get_sve().
> > + */
> > +QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
> if the code is ready to support slices, I guess you could have a define
> and compute the slice number from ARM_MAX_VQ?

Yeah, that should be do-able. Thanks for the suggestion.

> > +
> > +static int kvm_arch_put_sve(CPUState *cs)
> > +{
> > +    ARMCPU *cpu = ARM_CPU(cs);
> > +    CPUARMState *env = &cpu->env;
> > +    struct kvm_one_reg reg;
> > +    int slices = 1;
> > +    int i, n, ret;
> > +
> > +    for (i = 0; i < slices; i++) {
> > +        for (n = 0; n < KVM_ARM64_SVE_NUM_ZREGS; n++) {
> > +            uint64_t *q = aa64_vfp_qreg(env, n);
> > +#ifdef HOST_WORDS_BIGENDIAN
> > +            uint64_t d[ARM_MAX_VQ * 2];
> > +            int j;
> line to be added
> > +            for (j = 0; j < cpu->sve_max_vq * 2; j++) {
> > +                d[j] = bswap64(q[j]);
> > +            }
> > +            reg.addr = (uintptr_t)d;
> > +#else
> > +            reg.addr = (uintptr_t)q;
> > +#endif
> > +            reg.id = KVM_REG_ARM64_SVE_ZREG(n, i);
> > +            ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
> > +            if (ret) {
> > +                return ret;
> > +            }
> > +        }
> > +
> > +        for (n = 0; n < KVM_ARM64_SVE_NUM_PREGS; n++) {
> > +            uint64_t *q = &env->vfp.pregs[n].p[0];
> > +#ifdef HOST_WORDS_BIGENDIAN
> > +            uint64_t d[ARM_MAX_VQ * 2 / 8];
> > +            int j;
> line
> > +            for (j = 0; j < cpu->sve_max_vq * 2 / 8; j++) {
> > +                d[j] = bswap64(q[j]);
> > +            }
> > +            reg.addr = (uintptr_t)d;
> > +#else
> > +            reg.addr = (uintptr_t)q;
> > +#endif
> > +            reg.id = KVM_REG_ARM64_SVE_PREG(n, i);
> > +            ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
> > +            if (ret) {
> > +                return ret;
> > +            }
> > +        }
> > +
> > +        reg.addr = (uintptr_t)&env->vfp.pregs[FFR_PRED_NUM].p[0];
> > +        reg.id = KVM_REG_ARM64_SVE_FFR(i);
> > +        ret = kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
> > +        if (ret) {
> > +            return ret;
> > +        }
> > +    }
> > +
> > +    return 0;
> > +}
> > +
> >  int kvm_arch_put_registers(CPUState *cs, int level)
> >  {
> >      struct kvm_one_reg reg;
> > @@ -857,7 +922,11 @@ int kvm_arch_put_registers(CPUState *cs, int level)
> >          }
> >      }
> >  
> > -    ret = kvm_arch_put_fpsimd(cs);
> > +    if (!cpu->sve_max_vq) {
> > +        ret = kvm_arch_put_fpsimd(cs);
> > +    } else {
> > +        ret = kvm_arch_put_sve(cs);
> > +    }
> >      if (ret) {
> >          return ret;
> >      }
> > @@ -920,6 +989,60 @@ static int kvm_arch_get_fpsimd(CPUState *cs)
> >      return 0;
> >  }
> >  
> > +static int kvm_arch_get_sve(CPUState *cs)
> > +{
> > +    ARMCPU *cpu = ARM_CPU(cs);
> > +    CPUARMState *env = &cpu->env;
> > +    struct kvm_one_reg reg;
> > +    int slices = 1;
> > +    int i, n, ret;
> > +
> > +    for (i = 0; i < slices; i++) {
> > +        for (n = 0; n < KVM_ARM64_SVE_NUM_ZREGS; n++) {
> > +            uint64_t *q = aa64_vfp_qreg(env, n);
> extra line needed
> > +            reg.id = KVM_REG_ARM64_SVE_ZREG(n, i);
> > +            reg.addr = (uintptr_t)q;
> > +            ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
> > +            if (ret) {
> > +                return ret;
> > +            } else {> +#ifdef HOST_WORDS_BIGENDIAN
> > +                int j;
> line
> > +                for (j = 0; j < cpu->sve_max_vq * 2; j++) {
> > +                    q[j] = bswap64(q[j]);
> > +                }
> > +#endif
> > +            }
> > +        }
> > +
> > +        for (n = 0; n < KVM_ARM64_SVE_NUM_PREGS; n++) {
> > +            uint64_t *q = &env->vfp.pregs[n].p[0];
> extra line needed
> > +            reg.id = KVM_REG_ARM64_SVE_PREG(n, i);
> > +            reg.addr = (uintptr_t)q;
> > +            ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
> > +            if (ret) {
> > +                return ret;
> > +            } else {> +#ifdef HOST_WORDS_BIGENDIAN
> > +                int j;
> line
> > +                for (j = 0; j < cpu->sve_max_vq * 2 / 8; j++) {
> > +                    q[j] = bswap64(q[j]);
> > +                }
> > +#endif
> > +            }
> > +        }
> > +
> > +        reg.addr = (uintptr_t)&env->vfp.pregs[FFR_PRED_NUM].p[0];
> > +        reg.id = KVM_REG_ARM64_SVE_FFR(i);
> > +        ret = kvm_vcpu_ioctl(cs, KVM_GET_ONE_REG, &reg);
> > +        if (ret) {
> > +            return ret;
> > +        }
> > +    }
> > +
> > +    return 0;
> > +}
> > +
> >  int kvm_arch_get_registers(CPUState *cs)
> >  {
> >      struct kvm_one_reg reg;
> > @@ -1014,7 +1137,11 @@ int kvm_arch_get_registers(CPUState *cs)
> >          env->spsr = env->banked_spsr[i];
> >      }
> >  
> > -    ret = kvm_arch_get_fpsimd(cs);
> > +    if (!cpu->sve_max_vq) {
> > +        ret = kvm_arch_get_fpsimd(cs);
> > +    } else {
> > +        ret = kvm_arch_get_sve(cs);
> > +    }
> >      if (ret) {
> >          return ret;
> >      }
> > 
> 
>

I'll see if I can clean/improve this patch with yours and Richard's
suggestions.

Thanks,
drew

[Qemu-devel] [PATCH v2 11/14] target/arm/kvm64: max cpu: Enable SVE when available

[Andrew Jones]

Enable SVE in the KVM guest when the 'max' cpu type is configured
and KVM supports it. KVM SVE requires use of the new finalize
vcpu ioctl, so we add that now too. For starters SVE can only be
turned on or off, getting all vector lengths the host CPU supports
when on. We'll add the other SVE CPU properties in later patches.

Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 target/arm/cpu64.c       | 24 ++++++++++++++++++++++--
 target/arm/kvm.c         |  5 +++++
 target/arm/kvm64.c       | 25 ++++++++++++++++++++++++-
 target/arm/kvm_arm.h     | 27 +++++++++++++++++++++++++++
 tests/arm-cpu-features.c |  1 +
 5 files changed, 79 insertions(+), 3 deletions(-)

diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
index 5def82111dee..2e595ad53137 100644
--- a/target/arm/cpu64.c
+++ b/target/arm/cpu64.c
@@ -371,6 +371,11 @@ void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp)
         return;
     }
 
+    /* sve-max-vq and sve<vl-bits> properties not yet implemented for KVM */
+    if (kvm_enabled()) {
+        return;
+    }
+
     if (cpu->sve_max_vq == ARM_SVE_INIT) {
         object_property_set_uint(OBJECT(cpu), ARM_MAX_VQ, "sve-max-vq", &err);
         if (err) {
@@ -632,6 +637,10 @@ static void cpu_arm_get_sve(Object *obj, Visitor *v, const char *name,
     ARMCPU *cpu = ARM_CPU(obj);
     bool value = !!cpu->sve_max_vq;
 
+    if (kvm_enabled() && !kvm_arm_sve_supported(CPU(cpu))) {
+        value = false;
+    }
+
     visit_type_bool(v, name, &value, errp);
 }
 
@@ -649,6 +658,11 @@ static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
     }
 
     if (value) {
+        if (kvm_enabled() && !kvm_arm_sve_supported(CPU(cpu))) {
+            error_setg(errp, "'sve' feature not supported by KVM on this host");
+            return;
+        }
+
         /*
          * We handle the -cpu <cpu>,sve=off,sve=on case by reinitializing,
          * but otherwise we don't do anything as an sve=on could come after
@@ -675,6 +689,11 @@ static void aarch64_max_initfn(Object *obj)
 
     if (kvm_enabled()) {
         kvm_arm_set_cpu_features_from_host(cpu);
+        /*
+         * KVM doesn't yet support the sve-max-vq property, but
+         * setting cpu->sve_max_vq is also used to turn SVE on.
+         */
+        cpu->sve_max_vq = ARM_SVE_INIT;
     } else {
         uint64_t t;
         uint32_t u;
@@ -764,8 +783,6 @@ static void aarch64_max_initfn(Object *obj)
         cpu->sve_max_vq = ARM_SVE_INIT;
         object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_max_vq,
                             cpu_max_set_sve_max_vq, NULL, NULL, &error_fatal);
-        object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
-                            cpu_arm_set_sve, NULL, NULL, &error_fatal);
 
         /*
          * sve_vq_map uses a special state while setting properties, so
@@ -780,6 +797,9 @@ static void aarch64_max_initfn(Object *obj)
                                 cpu_arm_set_sve_vq, NULL, NULL, &error_fatal);
         }
     }
+
+    object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
+                        cpu_arm_set_sve, NULL, NULL, &error_fatal);
 }
 
 struct ARMCPUInfo {
diff --git a/target/arm/kvm.c b/target/arm/kvm.c
index 69c961a4c62c..60645a196d3d 100644
--- a/target/arm/kvm.c
+++ b/target/arm/kvm.c
@@ -49,6 +49,11 @@ int kvm_arm_vcpu_init(CPUState *cs)
     return kvm_vcpu_ioctl(cs, KVM_ARM_VCPU_INIT, &init);
 }
 
+int kvm_arm_vcpu_finalize(CPUState *cs, int feature)
+{
+    return kvm_vcpu_ioctl(cs, KVM_ARM_VCPU_FINALIZE, &feature);
+}
+
 void kvm_arm_init_serror_injection(CPUState *cs)
 {
     cap_has_inject_serror_esr = kvm_check_extension(cs->kvm_state,
diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
index 706541327491..9fc7f078cf68 100644
--- a/target/arm/kvm64.c
+++ b/target/arm/kvm64.c
@@ -604,6 +604,15 @@ bool kvm_arm_aarch32_supported(CPUState *cpu)
     return ret > 0;
 }
 
+bool kvm_arm_sve_supported(CPUState *cpu)
+{
+    KVMState *s = KVM_STATE(current_machine->accelerator);
+    int ret;
+
+    ret = kvm_check_extension(s, KVM_CAP_ARM_SVE);
+    return ret > 0;
+}
+
 #define ARM_CPU_ID_MPIDR       3, 0, 0, 0, 5
 
 int kvm_arch_init_vcpu(CPUState *cs)
@@ -632,13 +641,20 @@ int kvm_arch_init_vcpu(CPUState *cs)
         cpu->kvm_init_features[0] |= 1 << KVM_ARM_VCPU_EL1_32BIT;
     }
     if (!kvm_check_extension(cs->kvm_state, KVM_CAP_ARM_PMU_V3)) {
-            cpu->has_pmu = false;
+        cpu->has_pmu = false;
     }
     if (cpu->has_pmu) {
         cpu->kvm_init_features[0] |= 1 << KVM_ARM_VCPU_PMU_V3;
     } else {
         unset_feature(&env->features, ARM_FEATURE_PMU);
     }
+    if (cpu->sve_max_vq) {
+        if (!kvm_arm_sve_supported(cs)) {
+            cpu->sve_max_vq = 0;
+        } else {
+            cpu->kvm_init_features[0] |= 1 << KVM_ARM_VCPU_SVE;
+        }
+    }
 
     /* Do KVM_ARM_VCPU_INIT ioctl */
     ret = kvm_arm_vcpu_init(cs);
@@ -646,6 +662,13 @@ int kvm_arch_init_vcpu(CPUState *cs)
         return ret;
     }
 
+    if (cpu->sve_max_vq) {
+        ret = kvm_arm_vcpu_finalize(cs, KVM_ARM_VCPU_SVE);
+        if (ret) {
+            return ret;
+        }
+    }
+
     /*
      * When KVM is in use, PSCI is emulated in-kernel and not by qemu.
      * Currently KVM has its own idea about MPIDR assignment, so we
diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
index e0ded3607996..2367f8ab78ed 100644
--- a/target/arm/kvm_arm.h
+++ b/target/arm/kvm_arm.h
@@ -27,6 +27,20 @@
  */
 int kvm_arm_vcpu_init(CPUState *cs);
 
+/**
+ * kvm_arm_vcpu_finalize
+ * @cs: CPUState
+ * @feature: int
+ *
+ * Finalizes the configuration of the specified VCPU feature by
+ * invoking the KVM_ARM_VCPU_FINALIZE ioctl. Features requiring
+ * this are documented in the "KVM_ARM_VCPU_FINALIZE" section of
+ * KVM's API documentation.
+ *
+ * Returns: 0 if success else < 0 error code
+ */
+int kvm_arm_vcpu_finalize(CPUState *cs, int feature);
+
 /**
  * kvm_arm_register_device:
  * @mr: memory region for this device
@@ -224,6 +238,14 @@ bool kvm_arm_aarch32_supported(CPUState *cs);
  */
 bool kvm_arm_pmu_supported(CPUState *cs);
 
+/**
+ * bool kvm_arm_sve_supported:
+ * @cs: CPUState
+ *
+ * Returns true if the KVM VCPU can enable SVE and false otherwise.
+ */
+bool kvm_arm_sve_supported(CPUState *cs);
+
 /**
  * kvm_arm_get_max_vm_ipa_size - Returns the number of bits in the
  * IPA address space supported by KVM
@@ -274,6 +296,11 @@ static inline bool kvm_arm_pmu_supported(CPUState *cs)
     return false;
 }
 
+static inline bool kvm_arm_sve_supported(CPUState *cs)
+{
+    return false;
+}
+
 static inline int kvm_arm_get_max_vm_ipa_size(MachineState *ms)
 {
     return -ENOENT;
diff --git a/tests/arm-cpu-features.c b/tests/arm-cpu-features.c
index a4bf6aec00df..67ad5f2b78d5 100644
--- a/tests/arm-cpu-features.c
+++ b/tests/arm-cpu-features.c
@@ -393,6 +393,7 @@ static void test_query_cpu_model_expansion_kvm(const void *data)
 
     if (g_str_equal(qtest_get_arch(), "aarch64")) {
         assert_has_feature(qts, "host", "aarch64");
+        assert_has_feature(qts, "max", "sve");
 
         assert_error(qts, "cortex-a15",
             "The CPU definition 'cortex-a15' cannot "
-- 
2.20.1

Re: [Qemu-devel] [PATCH v2 11/14] target/arm/kvm64: max cpu: Enable SVE when available

[Richard Henderson]

On 6/21/19 6:34 PM, Andrew Jones wrote:
> @@ -675,6 +689,11 @@ static void aarch64_max_initfn(Object *obj)
>  
>      if (kvm_enabled()) {
>          kvm_arm_set_cpu_features_from_host(cpu);
> +        /*
> +         * KVM doesn't yet support the sve-max-vq property, but
> +         * setting cpu->sve_max_vq is also used to turn SVE on.
> +         */
> +        cpu->sve_max_vq = ARM_SVE_INIT;

Can we support this value with KVM_GET/SET_ONE_REG on ZCR_EL2?  (IIRC KVM
requires VHE to support SVE, so the host is always EL2 and the guest is always
EL1.)

Or do we need to probe this via normal userland prctl?

Or am I getting ahead of the patches to follow?


r~

Re: [Qemu-devel] [PATCH v2 11/14] target/arm/kvm64: max cpu: Enable SVE when available

[Andrew Jones]

On Wed, Jun 26, 2019 at 01:09:45PM +0200, Richard Henderson wrote:
> On 6/21/19 6:34 PM, Andrew Jones wrote:
> > @@ -675,6 +689,11 @@ static void aarch64_max_initfn(Object *obj)
> >  
> >      if (kvm_enabled()) {
> >          kvm_arm_set_cpu_features_from_host(cpu);
> > +        /*
> > +         * KVM doesn't yet support the sve-max-vq property, but
> > +         * setting cpu->sve_max_vq is also used to turn SVE on.
> > +         */
> > +        cpu->sve_max_vq = ARM_SVE_INIT;
> 
> Can we support this value with KVM_GET/SET_ONE_REG on ZCR_EL2?  (IIRC KVM
> requires VHE to support SVE, so the host is always EL2 and the guest is always
> EL1.)
> 
> Or do we need to probe this via normal userland prctl?
> 
> Or am I getting ahead of the patches to follow?

Hopefully mostly this, because I didn't understand the concern with VHE.
If it still looks like something is missing after looking at other
patches, then please come back to this to help me understand.

Thanks,
drew

Re: [Qemu-devel] [PATCH v2 11/14] target/arm/kvm64: max cpu: Enable SVE when available

[Auger Eric]

Hi Drew,

On 6/21/19 6:34 PM, Andrew Jones wrote:
> Enable SVE in the KVM guest when the 'max' cpu type is configured
> and KVM supports it. KVM SVE requires use of the new finalize
> vcpu ioctl, so we add that now too. For starters SVE can only be
> turned on or off, getting all vector lengths the host CPU supports
> when on. We'll add the other SVE CPU properties in later patches.
> 
> Signed-off-by: Andrew Jones <drjones@redhat.com>
> ---
>  target/arm/cpu64.c       | 24 ++++++++++++++++++++++--
>  target/arm/kvm.c         |  5 +++++
>  target/arm/kvm64.c       | 25 ++++++++++++++++++++++++-
>  target/arm/kvm_arm.h     | 27 +++++++++++++++++++++++++++
>  tests/arm-cpu-features.c |  1 +
>  5 files changed, 79 insertions(+), 3 deletions(-)
> 
> diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
> index 5def82111dee..2e595ad53137 100644
> --- a/target/arm/cpu64.c
> +++ b/target/arm/cpu64.c
> @@ -371,6 +371,11 @@ void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp)
>          return;
>      }
>  
> +    /* sve-max-vq and sve<vl-bits> properties not yet implemented for KVM */
> +    if (kvm_enabled()) {
> +        return;
> +    }
> +
>      if (cpu->sve_max_vq == ARM_SVE_INIT) {
>          object_property_set_uint(OBJECT(cpu), ARM_MAX_VQ, "sve-max-vq", &err);
>          if (err) {
> @@ -632,6 +637,10 @@ static void cpu_arm_get_sve(Object *obj, Visitor *v, const char *name,
>      ARMCPU *cpu = ARM_CPU(obj);
>      bool value = !!cpu->sve_max_vq;
>  
> +    if (kvm_enabled() && !kvm_arm_sve_supported(CPU(cpu))) {
> +        value = false;
> +    }
> +
>      visit_type_bool(v, name, &value, errp);
>  }
>  
> @@ -649,6 +658,11 @@ static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
>      }
>  
>      if (value) {
> +        if (kvm_enabled() && !kvm_arm_sve_supported(CPU(cpu))) {
> +            error_setg(errp, "'sve' feature not supported by KVM on this host");
> +            return;
> +        }
> +
>          /*
>           * We handle the -cpu <cpu>,sve=off,sve=on case by reinitializing,
>           * but otherwise we don't do anything as an sve=on could come after
> @@ -675,6 +689,11 @@ static void aarch64_max_initfn(Object *obj)
>  
>      if (kvm_enabled()) {
>          kvm_arm_set_cpu_features_from_host(cpu);
> +        /*
> +         * KVM doesn't yet support the sve-max-vq property, but
> +         * setting cpu->sve_max_vq is also used to turn SVE on.
> +         */
> +        cpu->sve_max_vq = ARM_SVE_INIT;
>      } else {
>          uint64_t t;
>          uint32_t u;
> @@ -764,8 +783,6 @@ static void aarch64_max_initfn(Object *obj)
>          cpu->sve_max_vq = ARM_SVE_INIT;
>          object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_max_vq,
>                              cpu_max_set_sve_max_vq, NULL, NULL, &error_fatal);
> -        object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
> -                            cpu_arm_set_sve, NULL, NULL, &error_fatal);
>  
>          /*
>           * sve_vq_map uses a special state while setting properties, so
> @@ -780,6 +797,9 @@ static void aarch64_max_initfn(Object *obj)
>                                  cpu_arm_set_sve_vq, NULL, NULL, &error_fatal);
>          }
>      }
> +
> +    object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
> +                        cpu_arm_set_sve, NULL, NULL, &error_fatal);
>  }
>  
>  struct ARMCPUInfo {
> diff --git a/target/arm/kvm.c b/target/arm/kvm.c
> index 69c961a4c62c..60645a196d3d 100644
> --- a/target/arm/kvm.c
> +++ b/target/arm/kvm.c
> @@ -49,6 +49,11 @@ int kvm_arm_vcpu_init(CPUState *cs)
>      return kvm_vcpu_ioctl(cs, KVM_ARM_VCPU_INIT, &init);
>  }
>  
> +int kvm_arm_vcpu_finalize(CPUState *cs, int feature)
> +{
> +    return kvm_vcpu_ioctl(cs, KVM_ARM_VCPU_FINALIZE, &feature);
> +}
> +
>  void kvm_arm_init_serror_injection(CPUState *cs)
>  {
>      cap_has_inject_serror_esr = kvm_check_extension(cs->kvm_state,
> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> index 706541327491..9fc7f078cf68 100644
> --- a/target/arm/kvm64.c
> +++ b/target/arm/kvm64.c
> @@ -604,6 +604,15 @@ bool kvm_arm_aarch32_supported(CPUState *cpu)
>      return ret > 0;
>  }
>  
> +bool kvm_arm_sve_supported(CPUState *cpu)
> +{
> +    KVMState *s = KVM_STATE(current_machine->accelerator);
> +    int ret;
> +
> +    ret = kvm_check_extension(s, KVM_CAP_ARM_SVE);
> +    return ret > 0;
return kvm_check_extension()

Eric
> +}
> +
>  #define ARM_CPU_ID_MPIDR       3, 0, 0, 0, 5
>  
>  int kvm_arch_init_vcpu(CPUState *cs)
> @@ -632,13 +641,20 @@ int kvm_arch_init_vcpu(CPUState *cs)
>          cpu->kvm_init_features[0] |= 1 << KVM_ARM_VCPU_EL1_32BIT;
>      }
>      if (!kvm_check_extension(cs->kvm_state, KVM_CAP_ARM_PMU_V3)) {
> -            cpu->has_pmu = false;
> +        cpu->has_pmu = false;
>      }
>      if (cpu->has_pmu) {
>          cpu->kvm_init_features[0] |= 1 << KVM_ARM_VCPU_PMU_V3;
>      } else {
>          unset_feature(&env->features, ARM_FEATURE_PMU);
>      }
> +    if (cpu->sve_max_vq) {
> +        if (!kvm_arm_sve_supported(cs)) {
> +            cpu->sve_max_vq = 0;
> +        } else {
> +            cpu->kvm_init_features[0] |= 1 << KVM_ARM_VCPU_SVE;
> +        }
> +    }
>  
>      /* Do KVM_ARM_VCPU_INIT ioctl */
>      ret = kvm_arm_vcpu_init(cs);
> @@ -646,6 +662,13 @@ int kvm_arch_init_vcpu(CPUState *cs)
>          return ret;
>      }
>  
> +    if (cpu->sve_max_vq) {
> +        ret = kvm_arm_vcpu_finalize(cs, KVM_ARM_VCPU_SVE);
> +        if (ret) {
> +            return ret;
> +        }
> +    }
> +
>      /*
>       * When KVM is in use, PSCI is emulated in-kernel and not by qemu.
>       * Currently KVM has its own idea about MPIDR assignment, so we
> diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
> index e0ded3607996..2367f8ab78ed 100644
> --- a/target/arm/kvm_arm.h
> +++ b/target/arm/kvm_arm.h
> @@ -27,6 +27,20 @@
>   */
>  int kvm_arm_vcpu_init(CPUState *cs);
>  
> +/**
> + * kvm_arm_vcpu_finalize
> + * @cs: CPUState
> + * @feature: int
> + *
> + * Finalizes the configuration of the specified VCPU feature by
> + * invoking the KVM_ARM_VCPU_FINALIZE ioctl. Features requiring
> + * this are documented in the "KVM_ARM_VCPU_FINALIZE" section of
> + * KVM's API documentation.
> + *
> + * Returns: 0 if success else < 0 error code
> + */
> +int kvm_arm_vcpu_finalize(CPUState *cs, int feature);
> +
>  /**
>   * kvm_arm_register_device:
>   * @mr: memory region for this device
> @@ -224,6 +238,14 @@ bool kvm_arm_aarch32_supported(CPUState *cs);
>   */
>  bool kvm_arm_pmu_supported(CPUState *cs);
>  
> +/**
> + * bool kvm_arm_sve_supported:
> + * @cs: CPUState
> + *
> + * Returns true if the KVM VCPU can enable SVE and false otherwise.
> + */
> +bool kvm_arm_sve_supported(CPUState *cs);
> +
>  /**
>   * kvm_arm_get_max_vm_ipa_size - Returns the number of bits in the
>   * IPA address space supported by KVM
> @@ -274,6 +296,11 @@ static inline bool kvm_arm_pmu_supported(CPUState *cs)
>      return false;
>  }
>  
> +static inline bool kvm_arm_sve_supported(CPUState *cs)
> +{
> +    return false;
> +}
> +
>  static inline int kvm_arm_get_max_vm_ipa_size(MachineState *ms)
>  {
>      return -ENOENT;
> diff --git a/tests/arm-cpu-features.c b/tests/arm-cpu-features.c
> index a4bf6aec00df..67ad5f2b78d5 100644
> --- a/tests/arm-cpu-features.c
> +++ b/tests/arm-cpu-features.c
> @@ -393,6 +393,7 @@ static void test_query_cpu_model_expansion_kvm(const void *data)
>  
>      if (g_str_equal(qtest_get_arch(), "aarch64")) {
>          assert_has_feature(qts, "host", "aarch64");
> +        assert_has_feature(qts, "max", "sve");
>  
>          assert_error(qts, "cortex-a15",
>              "The CPU definition 'cortex-a15' cannot "
> 

[Qemu-devel] [PATCH v2 12/14] target/arm/kvm: scratch vcpu: Preserve input kvm_vcpu_init features

[Andrew Jones]

kvm_arm_create_scratch_host_vcpu() takes a struct kvm_vcpu_init
parameter. Rather than just using it as an output parameter to
pass back the preferred target, use it also as an input parameter,
allowing a caller to pass a selected target if they wish and to
also pass cpu features. If the caller doesn't want to select a
target they can pass -1 for the target which indicates they want
to use the preferred target and have it passed back like before.

Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 target/arm/kvm.c   | 20 +++++++++++++++-----
 target/arm/kvm32.c |  6 +++++-
 target/arm/kvm64.c |  6 +++++-
 3 files changed, 25 insertions(+), 7 deletions(-)

diff --git a/target/arm/kvm.c b/target/arm/kvm.c
index 60645a196d3d..66c0c198604a 100644
--- a/target/arm/kvm.c
+++ b/target/arm/kvm.c
@@ -64,7 +64,7 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
                                       int *fdarray,
                                       struct kvm_vcpu_init *init)
 {
-    int ret, kvmfd = -1, vmfd = -1, cpufd = -1;
+    int ret = 0, kvmfd = -1, vmfd = -1, cpufd = -1;
 
     kvmfd = qemu_open("/dev/kvm", O_RDWR);
     if (kvmfd < 0) {
@@ -84,7 +84,14 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
         goto finish;
     }
 
-    ret = ioctl(vmfd, KVM_ARM_PREFERRED_TARGET, init);
+    if (init->target == -1) {
+        struct kvm_vcpu_init preferred;
+
+        ret = ioctl(vmfd, KVM_ARM_PREFERRED_TARGET, &preferred);
+        if (!ret) {
+            init->target = preferred.target;
+        }
+    }
     if (ret >= 0) {
         ret = ioctl(cpufd, KVM_ARM_VCPU_INIT, init);
         if (ret < 0) {
@@ -96,10 +103,12 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
          * creating one kind of guest CPU which is its preferred
          * CPU type.
          */
+        struct kvm_vcpu_init try;
+
         while (*cpus_to_try != QEMU_KVM_ARM_TARGET_NONE) {
-            init->target = *cpus_to_try++;
-            memset(init->features, 0, sizeof(init->features));
-            ret = ioctl(cpufd, KVM_ARM_VCPU_INIT, init);
+            try.target = *cpus_to_try++;
+            memcpy(try.features, init->features, sizeof(init->features));
+            ret = ioctl(cpufd, KVM_ARM_VCPU_INIT, &try);
             if (ret >= 0) {
                 break;
             }
@@ -107,6 +116,7 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
         if (ret < 0) {
             goto err;
         }
+        init->target = try.target;
     } else {
         /* Treat a NULL cpus_to_try argument the same as an empty
          * list, which means we will fail the call since this must
diff --git a/target/arm/kvm32.c b/target/arm/kvm32.c
index 51f78f722b18..d007f6bd34d7 100644
--- a/target/arm/kvm32.c
+++ b/target/arm/kvm32.c
@@ -54,7 +54,11 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
         QEMU_KVM_ARM_TARGET_CORTEX_A15,
         QEMU_KVM_ARM_TARGET_NONE
     };
-    struct kvm_vcpu_init init;
+    /*
+     * target = -1 informs kvm_arm_create_scratch_host_vcpu()
+     * to use the preferred target
+     */
+    struct kvm_vcpu_init init = { .target = -1, };
 
     if (!kvm_arm_create_scratch_host_vcpu(cpus_to_try, fdarray, &init)) {
         return false;
diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
index 9fc7f078cf68..2821135a4d0e 100644
--- a/target/arm/kvm64.c
+++ b/target/arm/kvm64.c
@@ -502,7 +502,11 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
         KVM_ARM_TARGET_CORTEX_A57,
         QEMU_KVM_ARM_TARGET_NONE
     };
-    struct kvm_vcpu_init init;
+    /*
+     * target = -1 informs kvm_arm_create_scratch_host_vcpu()
+     * to use the preferred target
+     */
+    struct kvm_vcpu_init init = { .target = -1, };
 
     if (!kvm_arm_create_scratch_host_vcpu(cpus_to_try, fdarray, &init)) {
         return false;
-- 
2.20.1

Re: [Qemu-devel] [PATCH v2 12/14] target/arm/kvm: scratch vcpu: Preserve input kvm_vcpu_init features

[Richard Henderson]

On 6/21/19 6:34 PM, Andrew Jones wrote:
> kvm_arm_create_scratch_host_vcpu() takes a struct kvm_vcpu_init
> parameter. Rather than just using it as an output parameter to
> pass back the preferred target, use it also as an input parameter,
> allowing a caller to pass a selected target if they wish and to
> also pass cpu features. If the caller doesn't want to select a
> target they can pass -1 for the target which indicates they want
> to use the preferred target and have it passed back like before.
> 
> Signed-off-by: Andrew Jones <drjones@redhat.com>
> ---
>  target/arm/kvm.c   | 20 +++++++++++++++-----
>  target/arm/kvm32.c |  6 +++++-
>  target/arm/kvm64.c |  6 +++++-
>  3 files changed, 25 insertions(+), 7 deletions(-)

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>


r~

Re: [Qemu-devel] [PATCH v2 12/14] target/arm/kvm: scratch vcpu: Preserve input kvm_vcpu_init features

[Auger Eric]

Hi Drew,

On 6/21/19 6:34 PM, Andrew Jones wrote:
> kvm_arm_create_scratch_host_vcpu() takes a struct kvm_vcpu_init
> parameter. Rather than just using it as an output parameter to
> pass back the preferred target, use it also as an input parameter,
> allowing a caller to pass a selected target if they wish and to
> also pass cpu features. If the caller doesn't want to select a
> target they can pass -1 for the target which indicates they want
> to use the preferred target and have it passed back like before.
> 
> Signed-off-by: Andrew Jones <drjones@redhat.com>
> ---
>  target/arm/kvm.c   | 20 +++++++++++++++-----
>  target/arm/kvm32.c |  6 +++++-
>  target/arm/kvm64.c |  6 +++++-
>  3 files changed, 25 insertions(+), 7 deletions(-)
> 
> diff --git a/target/arm/kvm.c b/target/arm/kvm.c
> index 60645a196d3d..66c0c198604a 100644
> --- a/target/arm/kvm.c
> +++ b/target/arm/kvm.c
> @@ -64,7 +64,7 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
>                                        int *fdarray,
>                                        struct kvm_vcpu_init *init)
>  {
> -    int ret, kvmfd = -1, vmfd = -1, cpufd = -1;
> +    int ret = 0, kvmfd = -1, vmfd = -1, cpufd = -1;
>  
>      kvmfd = qemu_open("/dev/kvm", O_RDWR);
>      if (kvmfd < 0) {
> @@ -84,7 +84,14 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
>          goto finish;
>      }
>  
> -    ret = ioctl(vmfd, KVM_ARM_PREFERRED_TARGET, init);
> +    if (init->target == -1) {
> +        struct kvm_vcpu_init preferred;
> +
> +        ret = ioctl(vmfd, KVM_ARM_PREFERRED_TARGET, &preferred);
> +        if (!ret) {
> +            init->target = preferred.target;
wouldn't it be safe to copy the whole struct. Kernel code says:
        /*
         * For now, we don't return any features.
         * In future, we might use features to return target
         * specific features available for the preferred
         * target type.
         */

> +        }
> +    }
>      if (ret >= 0) {
>          ret = ioctl(cpufd, KVM_ARM_VCPU_INIT, init);
>          if (ret < 0) {
> @@ -96,10 +103,12 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
>           * creating one kind of guest CPU which is its preferred
>           * CPU type.
>           */
> +        struct kvm_vcpu_init try;
> +
>          while (*cpus_to_try != QEMU_KVM_ARM_TARGET_NONE) {
> -            init->target = *cpus_to_try++;
> -            memset(init->features, 0, sizeof(init->features));
> -            ret = ioctl(cpufd, KVM_ARM_VCPU_INIT, init);
> +            try.target = *cpus_to_try++;
> +            memcpy(try.features, init->features, sizeof(init->features));
> +            ret = ioctl(cpufd, KVM_ARM_VCPU_INIT, &try);
>              if (ret >= 0) {
>                  break;
>              }
> @@ -107,6 +116,7 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
>          if (ret < 0) {
>              goto err;
>          }
> +        init->target = try.target;
>      } else {
>          /* Treat a NULL cpus_to_try argument the same as an empty
>           * list, which means we will fail the call since this must
> diff --git a/target/arm/kvm32.c b/target/arm/kvm32.c
> index 51f78f722b18..d007f6bd34d7 100644
> --- a/target/arm/kvm32.c
> +++ b/target/arm/kvm32.c
> @@ -54,7 +54,11 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
>          QEMU_KVM_ARM_TARGET_CORTEX_A15,
>          QEMU_KVM_ARM_TARGET_NONE
>      };
> -    struct kvm_vcpu_init init;
> +    /*
> +     * target = -1 informs kvm_arm_create_scratch_host_vcpu()
> +     * to use the preferred target
> +     */
> +    struct kvm_vcpu_init init = { .target = -1, };
>  
>      if (!kvm_arm_create_scratch_host_vcpu(cpus_to_try, fdarray, &init)) {
>          return false;
> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> index 9fc7f078cf68..2821135a4d0e 100644
> --- a/target/arm/kvm64.c
> +++ b/target/arm/kvm64.c
> @@ -502,7 +502,11 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
>          KVM_ARM_TARGET_CORTEX_A57,
>          QEMU_KVM_ARM_TARGET_NONE
>      };
> -    struct kvm_vcpu_init init;
> +    /*
> +     * target = -1 informs kvm_arm_create_scratch_host_vcpu()
> +     * to use the preferred target
> +     */
> +    struct kvm_vcpu_init init = { .target = -1, };
>  
>      if (!kvm_arm_create_scratch_host_vcpu(cpus_to_try, fdarray, &init)) {
>          return false;
> 

Besides

Reviewed-by: Eric Auger <eric.auger@redhat.com>

Thanks

Eric

Re: [Qemu-devel] [PATCH v2 12/14] target/arm/kvm: scratch vcpu: Preserve input kvm_vcpu_init features

[Andrew Jones]

On Thu, Jun 27, 2019 at 09:30:57AM +0200, Auger Eric wrote:
> Hi Drew,
> 
> On 6/21/19 6:34 PM, Andrew Jones wrote:
> > kvm_arm_create_scratch_host_vcpu() takes a struct kvm_vcpu_init
> > parameter. Rather than just using it as an output parameter to
> > pass back the preferred target, use it also as an input parameter,
> > allowing a caller to pass a selected target if they wish and to
> > also pass cpu features. If the caller doesn't want to select a
> > target they can pass -1 for the target which indicates they want
> > to use the preferred target and have it passed back like before.
> > 
> > Signed-off-by: Andrew Jones <drjones@redhat.com>
> > ---
> >  target/arm/kvm.c   | 20 +++++++++++++++-----
> >  target/arm/kvm32.c |  6 +++++-
> >  target/arm/kvm64.c |  6 +++++-
> >  3 files changed, 25 insertions(+), 7 deletions(-)
> > 
> > diff --git a/target/arm/kvm.c b/target/arm/kvm.c
> > index 60645a196d3d..66c0c198604a 100644
> > --- a/target/arm/kvm.c
> > +++ b/target/arm/kvm.c
> > @@ -64,7 +64,7 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
> >                                        int *fdarray,
> >                                        struct kvm_vcpu_init *init)
> >  {
> > -    int ret, kvmfd = -1, vmfd = -1, cpufd = -1;
> > +    int ret = 0, kvmfd = -1, vmfd = -1, cpufd = -1;
> >  
> >      kvmfd = qemu_open("/dev/kvm", O_RDWR);
> >      if (kvmfd < 0) {
> > @@ -84,7 +84,14 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
> >          goto finish;
> >      }
> >  
> > -    ret = ioctl(vmfd, KVM_ARM_PREFERRED_TARGET, init);
> > +    if (init->target == -1) {
> > +        struct kvm_vcpu_init preferred;
> > +
> > +        ret = ioctl(vmfd, KVM_ARM_PREFERRED_TARGET, &preferred);
> > +        if (!ret) {
> > +            init->target = preferred.target;
> wouldn't it be safe to copy the whole struct. Kernel code says:
>         /*
>          * For now, we don't return any features.
>          * In future, we might use features to return target
>          * specific features available for the preferred
>          * target type.
>          */

Indeed, we could copy the features into 'preferred', in case KVM starts to
do something with them. Based on that KVM comment, though, it looks like
we'd need to make other changes as well to QEMU at that time. Some sort
of feature matching/filtering code. It probably makes sense to do all that
at once, when/if that time comes.

> 
> > +        }
> > +    }
> >      if (ret >= 0) {
> >          ret = ioctl(cpufd, KVM_ARM_VCPU_INIT, init);
> >          if (ret < 0) {
> > @@ -96,10 +103,12 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
> >           * creating one kind of guest CPU which is its preferred
> >           * CPU type.
> >           */
> > +        struct kvm_vcpu_init try;
> > +
> >          while (*cpus_to_try != QEMU_KVM_ARM_TARGET_NONE) {
> > -            init->target = *cpus_to_try++;
> > -            memset(init->features, 0, sizeof(init->features));
> > -            ret = ioctl(cpufd, KVM_ARM_VCPU_INIT, init);
> > +            try.target = *cpus_to_try++;
> > +            memcpy(try.features, init->features, sizeof(init->features));
> > +            ret = ioctl(cpufd, KVM_ARM_VCPU_INIT, &try);
> >              if (ret >= 0) {
> >                  break;
> >              }
> > @@ -107,6 +116,7 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
> >          if (ret < 0) {
> >              goto err;
> >          }
> > +        init->target = try.target;
> >      } else {
> >          /* Treat a NULL cpus_to_try argument the same as an empty
> >           * list, which means we will fail the call since this must
> > diff --git a/target/arm/kvm32.c b/target/arm/kvm32.c
> > index 51f78f722b18..d007f6bd34d7 100644
> > --- a/target/arm/kvm32.c
> > +++ b/target/arm/kvm32.c
> > @@ -54,7 +54,11 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
> >          QEMU_KVM_ARM_TARGET_CORTEX_A15,
> >          QEMU_KVM_ARM_TARGET_NONE
> >      };
> > -    struct kvm_vcpu_init init;
> > +    /*
> > +     * target = -1 informs kvm_arm_create_scratch_host_vcpu()
> > +     * to use the preferred target
> > +     */
> > +    struct kvm_vcpu_init init = { .target = -1, };
> >  
> >      if (!kvm_arm_create_scratch_host_vcpu(cpus_to_try, fdarray, &init)) {
> >          return false;
> > diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> > index 9fc7f078cf68..2821135a4d0e 100644
> > --- a/target/arm/kvm64.c
> > +++ b/target/arm/kvm64.c
> > @@ -502,7 +502,11 @@ bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf)
> >          KVM_ARM_TARGET_CORTEX_A57,
> >          QEMU_KVM_ARM_TARGET_NONE
> >      };
> > -    struct kvm_vcpu_init init;
> > +    /*
> > +     * target = -1 informs kvm_arm_create_scratch_host_vcpu()
> > +     * to use the preferred target
> > +     */
> > +    struct kvm_vcpu_init init = { .target = -1, };
> >  
> >      if (!kvm_arm_create_scratch_host_vcpu(cpus_to_try, fdarray, &init)) {
> >          return false;
> > 
> 
> Besides
> 
> Reviewed-by: Eric Auger <eric.auger@redhat.com>
>

Thanks,
drew 

Re: [Qemu-devel] [PATCH v2 12/14] target/arm/kvm: scratch vcpu: Preserve input kvm_vcpu_init features

[Dave Martin]

On Thu, Jun 27, 2019 at 08:30:57AM +0100, Auger Eric wrote:
> Hi Drew,
> 
> On 6/21/19 6:34 PM, Andrew Jones wrote:
> > kvm_arm_create_scratch_host_vcpu() takes a struct kvm_vcpu_init
> > parameter. Rather than just using it as an output parameter to
> > pass back the preferred target, use it also as an input parameter,
> > allowing a caller to pass a selected target if they wish and to
> > also pass cpu features. If the caller doesn't want to select a
> > target they can pass -1 for the target which indicates they want
> > to use the preferred target and have it passed back like before.
> > 
> > Signed-off-by: Andrew Jones <drjones@redhat.com>
> > ---
> >  target/arm/kvm.c   | 20 +++++++++++++++-----
> >  target/arm/kvm32.c |  6 +++++-
> >  target/arm/kvm64.c |  6 +++++-
> >  3 files changed, 25 insertions(+), 7 deletions(-)
> > 
> > diff --git a/target/arm/kvm.c b/target/arm/kvm.c
> > index 60645a196d3d..66c0c198604a 100644
> > --- a/target/arm/kvm.c
> > +++ b/target/arm/kvm.c
> > @@ -64,7 +64,7 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
> >                                        int *fdarray,
> >                                        struct kvm_vcpu_init *init)
> >  {
> > -    int ret, kvmfd = -1, vmfd = -1, cpufd = -1;
> > +    int ret = 0, kvmfd = -1, vmfd = -1, cpufd = -1;
> >  
> >      kvmfd = qemu_open("/dev/kvm", O_RDWR);
> >      if (kvmfd < 0) {
> > @@ -84,7 +84,14 @@ bool kvm_arm_create_scratch_host_vcpu(const uint32_t *cpus_to_try,
> >          goto finish;
> >      }
> >  
> > -    ret = ioctl(vmfd, KVM_ARM_PREFERRED_TARGET, init);
> > +    if (init->target == -1) {
> > +        struct kvm_vcpu_init preferred;
> > +
> > +        ret = ioctl(vmfd, KVM_ARM_PREFERRED_TARGET, &preferred);
> > +        if (!ret) {
> > +            init->target = preferred.target;
> wouldn't it be safe to copy the whole struct. Kernel code says:
>         /*
>          * For now, we don't return any features.
>          * In future, we might use features to return target
>          * specific features available for the preferred
>          * target type.
>          */

Marc or Christoffer should preferably comment on this.

I think the spirit of the ABI is that can use the whole return of
KVM_ARM_PREFERRED_TARGET as a reasonable template for KVM_VCPU_INIT
without it blowing up in your face.

I initially tried to report SVE as available through this route,
but we decided against it precisely because userspace might be doing
the above.

[...]

Cheers
---Dave

[Qemu-devel] [PATCH v2 13/14] target/arm/cpu64: max cpu: Support sve properties with KVM

[Andrew Jones]

Extend the SVE vq map initialization and validation with KVM's
supported vector lengths when KVM is enabled. In order to determine
and select supported lengths we add two new KVM functions for getting
and setting the KVM_REG_ARM64_SVE_VLS pseudo-register.

Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 target/arm/cpu.h         |   3 +-
 target/arm/cpu64.c       | 171 +++++++++++++++++++++++++++------------
 target/arm/kvm64.c       | 117 +++++++++++++++++++++++++--
 target/arm/kvm_arm.h     |  19 +++++
 target/arm/monitor.c     |   2 +-
 tests/arm-cpu-features.c |  86 +++++++++++++++++---
 6 files changed, 331 insertions(+), 67 deletions(-)

diff --git a/target/arm/cpu.h b/target/arm/cpu.h
index cbb155cf72a5..8a1c6c66a462 100644
--- a/target/arm/cpu.h
+++ b/target/arm/cpu.h
@@ -926,7 +926,8 @@ struct ARMCPU {
      * (bit-number + 1) * 16 bytes, i.e. each bit number + 1 is the vector
      * length in quadwords. We need a map size twice the maximum
      * quadword length though because we use two bits for each vector
-     * length in order to track three states: uninitialized, off, and on.
+     * length in order to track four states: uninitialized, uninitialized
+     * but supported by KVM, off, and on.
      */
     DECLARE_BITMAP(sve_vq_map, ARM_MAX_VQ * 2);
 };
diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
index 2e595ad53137..6e92aa54b9c8 100644
--- a/target/arm/cpu64.c
+++ b/target/arm/cpu64.c
@@ -261,10 +261,11 @@ static void aarch64_a72_initfn(Object *obj)
  * While we eventually use cpu->sve_vq_map as a typical bitmap, where each vq
  * has only two states (off/on), until we've finalized the map at realize time
  * we use an extra bit, at the vq - 1 + ARM_MAX_VQ bit number, to also allow
- * tracking of the uninitialized state. The arm_vq_state typedef and following
- * functions allow us to more easily work with the bitmap. Also, while the map
- * is still initializing, sve-max-vq has an additional three states, bringing
- * the number of its states to five, which are the following:
+ * tracking of the uninitialized state and the uninitialized but supported by
+ * KVM state. The arm_vq_state typedef and following functions allow us to more
+ * easily work with the bitmap. Also, while the map is still initializing,
+ * sve-max-vq has an additional three states, bringing the number of its states
+ * to five, which are the following:
  *
  * sve-max-vq:
  *   0:    SVE is disabled. The default value for a vq in the map is 'OFF'.
@@ -296,6 +297,11 @@ typedef enum arm_vq_state {
     ARM_VQ_OFF,
     ARM_VQ_ON,
     ARM_VQ_UNINITIALIZED,
+    ARM_VQ_UNINITIALIZED_KVM_SUPPORTED
+    /*
+     * More states cannot be added without adding bits to sve_vq_map
+     * and modifying its supporting functions.
+     */
 } arm_vq_state;
 
 static arm_vq_state arm_cpu_vq_map_get(ARMCPU *cpu, int vq)
@@ -324,6 +330,23 @@ static void arm_cpu_vq_map_init(ARMCPU *cpu)
 {
     bitmap_zero(cpu->sve_vq_map, ARM_MAX_VQ * 2);
     bitmap_set(cpu->sve_vq_map, ARM_MAX_VQ, ARM_MAX_VQ);
+
+    if (kvm_enabled()) {
+        DECLARE_BITMAP(kvm_supported, ARM_MAX_VQ);
+        uint32_t kvm_max_vq;
+
+        bitmap_zero(kvm_supported, ARM_MAX_VQ);
+
+        kvm_arm_sve_get_vls(CPU(cpu), kvm_supported, ARM_MAX_VQ, &kvm_max_vq);
+
+        if (kvm_max_vq > ARM_MAX_VQ) {
+            warn_report("KVM supports vector lengths larger than "
+                        "QEMU can enable");
+        }
+
+        bitmap_or(cpu->sve_vq_map, cpu->sve_vq_map,
+                  kvm_supported, ARM_MAX_VQ);
+    }
 }
 
 static bool arm_cpu_vq_map_is_finalized(ARMCPU *cpu)
@@ -371,12 +394,7 @@ void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp)
         return;
     }
 
-    /* sve-max-vq and sve<vl-bits> properties not yet implemented for KVM */
-    if (kvm_enabled()) {
-        return;
-    }
-
-    if (cpu->sve_max_vq == ARM_SVE_INIT) {
+    if (!kvm_enabled() && cpu->sve_max_vq == ARM_SVE_INIT) {
         object_property_set_uint(OBJECT(cpu), ARM_MAX_VQ, "sve-max-vq", &err);
         if (err) {
             error_propagate(errp, err);
@@ -431,6 +449,11 @@ static void cpu_max_set_sve_max_vq(Object *obj, Visitor *v, const char *name,
         return;
     }
 
+    if (kvm_enabled() && !kvm_arm_sve_supported(CPU(cpu))) {
+        error_setg(errp, "'sve' feature not supported by KVM on this host");
+        return;
+    }
+
     /*
      * It gets complicated trying to support both sve-max-vq and
      * sve<vl-bits> properties together, so we mostly don't. We
@@ -460,6 +483,12 @@ static void cpu_max_set_sve_max_vq(Object *obj, Visitor *v, const char *name,
             sprintf(name, "sve%d", vq * 128);
             object_property_set_bool(obj, true, name, &err);
             if (err) {
+                if (kvm_enabled()) {
+                    error_append_hint(&err, "It is not possible to use "
+                                      "sve-max-vq with this KVM host. Try "
+                                      "using only sve<vl-bits> "
+                                      "properties.\n");
+                }
                 error_propagate(errp, err);
                 return;
             }
@@ -484,6 +513,12 @@ static void cpu_arm_get_sve_vq(Object *obj, Visitor *v, const char *name,
         value = true;
     } else if (vq_state == ARM_VQ_OFF) {
         value = false;
+    } else if (kvm_enabled() && vq_state == ARM_VQ_UNINITIALIZED) {
+        /*
+         * When KVM is enabled, anything not supported by the host must have
+         * 'OFF' for the default.
+         */
+        value = false;
     } else {
         /*
          * vq is uninitialized. We pick a default here based on the
@@ -539,6 +574,11 @@ static void cpu_arm_set_sve_vq(Object *obj, Visitor *v, const char *name,
         return;
     }
 
+    if (value && kvm_enabled() && !kvm_arm_sve_supported(CPU(cpu))) {
+        error_setg(errp, "'sve' feature not supported by KVM on this host");
+        return;
+    }
+
     if (arm_sve_have_max_vq(cpu)) {
         max_vq = cpu->sve_max_vq;
     } else {
@@ -569,6 +609,8 @@ static void cpu_arm_set_sve_vq(Object *obj, Visitor *v, const char *name,
         }
     }
 
+    vq_state = arm_cpu_vq_map_get(cpu, vq);
+
     if (arm_sve_have_max_vq(cpu) && value && vq > cpu->sve_max_vq) {
         error_setg(errp, "cannot enable %s", name);
         error_append_hint(errp, "vq=%d (%d bits) is larger than the "
@@ -580,19 +622,31 @@ static void cpu_arm_set_sve_vq(Object *obj, Visitor *v, const char *name,
         error_append_hint(errp, "The maximum vector length must be "
                           "enabled, sve-max-vq=%d (%d bits)\n",
                           cpu->sve_max_vq, cpu->sve_max_vq * 128);
-    } else if (arm_sve_have_max_vq(cpu) && !value && vq < cpu->sve_max_vq &&
-               is_power_of_2(vq)) {
+    } else if (!kvm_enabled() && arm_sve_have_max_vq(cpu) && !value &&
+               vq < cpu->sve_max_vq && is_power_of_2(vq)) {
         error_setg(errp, "cannot disable %s", name);
         error_append_hint(errp, "vq=%d (%d bits) is required as it is a "
                           "power-of-2 length smaller than the maximum, "
                           "sve-max-vq=%d (%d bits)\n", vq, vq * 128,
                           cpu->sve_max_vq, cpu->sve_max_vq * 128);
-    } else if (!value && vq < max_vq && is_power_of_2(vq)) {
+    } else if (!kvm_enabled() && !value && vq < max_vq && is_power_of_2(vq)) {
         error_setg(errp, "cannot disable %s", name);
         error_append_hint(errp, "Vector length %d-bits is required as it "
                           "is a power-of-2 length smaller than another "
                           "enabled vector length. Disable all larger vector "
                           "lengths first.\n", vq * 128);
+    } else if (kvm_enabled() && value && vq_state == ARM_VQ_UNINITIALIZED) {
+        error_setg(errp, "cannot enable %s", name);
+        error_append_hint(errp, "This KVM host does not support "
+                          "the vector length %d-bits.\n", vq * 128);
+    } else if (kvm_enabled() && !value && vq < max_vq &&
+               (vq_state == ARM_VQ_ON ||
+                vq_state == ARM_VQ_UNINITIALIZED_KVM_SUPPORTED)) {
+        error_setg(errp, "cannot disable %s", name);
+        error_append_hint(errp, "Vector length %d-bits is a KVM supported "
+                          "length smaller than another enabled vector "
+                          "length. Disable all larger vector lengths "
+                          "first.\n", vq * 128);
     } else {
         if (value) {
             bool fail = false;
@@ -602,31 +656,53 @@ static void cpu_arm_set_sve_vq(Object *obj, Visitor *v, const char *name,
              * Enabling a vector length automatically enables all
              * uninitialized power-of-2 lengths smaller than it, as
              * per the architecture.
+             *
+             * For KVM we have to automatically enable all supported,
+             * uninitialized lengths smaller than this length, even
+             * when it's not a power-of-2.
              */
             for (s = 1; s < vq; ++s) {
-                if (is_power_of_2(s)) {
-                    vq_state = arm_cpu_vq_map_get(cpu, s);
-                    if (vq_state == ARM_VQ_UNINITIALIZED) {
-                        arm_cpu_vq_map_set(cpu, s, ARM_VQ_ON);
-                    } else if (vq_state == ARM_VQ_OFF) {
-                        fail = true;
-                        break;
-                    }
+                vq_state = arm_cpu_vq_map_get(cpu, s);
+                if (!kvm_enabled() && is_power_of_2(s) &&
+                    vq_state == ARM_VQ_UNINITIALIZED) {
+                    arm_cpu_vq_map_set(cpu, s, ARM_VQ_ON);
+                } else if (vq_state == ARM_VQ_UNINITIALIZED_KVM_SUPPORTED) {
+                    assert(kvm_enabled());
+                    arm_cpu_vq_map_set(cpu, s, ARM_VQ_ON);
+                } else if ((kvm_enabled() || is_power_of_2(s)) &&
+                           vq_state == ARM_VQ_OFF) {
+                    fail = true;
+                    break;
                 }
             }
 
-            if (fail) {
+            if (!kvm_enabled() && fail) {
                 error_setg(errp, "cannot enable %s", name);
                 error_append_hint(errp, "Vector length %d-bits is disabled "
                                   "and is a power-of-2 length smaller than "
                                   "%s. All power-of-2 vector lengths smaller "
                                   "than the maximum length are required.\n",
                                   s * 128, name);
+
+            } else if (fail) {
+                error_setg(errp, "cannot enable %s", name);
+                error_append_hint(errp, "Vector length %d-bits is disabled "
+                                  "and the KVM host requires all supported "
+                                  "vector lengths smaller than %s to also be "
+                                  "enabled.\n", s * 128, name);
             } else {
                 arm_cpu_vq_map_set(cpu, vq, ARM_VQ_ON);
             }
         } else {
-            arm_cpu_vq_map_set(cpu, vq, ARM_VQ_OFF);
+            /*
+             * For KVM if the vq wasn't supported then we leave it in
+             * the ARM_VQ_UNINITIALIZED state in order to keep that
+             * unsupported information. It'll be set to OFF later when
+             * we finalize the map.
+             */
+            if (!kvm_enabled() || vq_state != ARM_VQ_UNINITIALIZED) {
+                arm_cpu_vq_map_set(cpu, vq, ARM_VQ_OFF);
+            }
         }
     }
 }
@@ -689,11 +765,6 @@ static void aarch64_max_initfn(Object *obj)
 
     if (kvm_enabled()) {
         kvm_arm_set_cpu_features_from_host(cpu);
-        /*
-         * KVM doesn't yet support the sve-max-vq property, but
-         * setting cpu->sve_max_vq is also used to turn SVE on.
-         */
-        cpu->sve_max_vq = ARM_SVE_INIT;
     } else {
         uint64_t t;
         uint32_t u;
@@ -774,32 +845,32 @@ static void aarch64_max_initfn(Object *obj)
         cpu->ctr = 0x80038003; /* 32 byte I and D cacheline size, VIPT icache */
         cpu->dcz_blocksize = 7; /*  512 bytes */
 #endif
-
-        /*
-         * sve_max_vq is initially unspecified, but must be initialized to a
-         * non-zero value (ARM_SVE_INIT) to indicate that this cpu type has
-         * SVE. It will be finalized in arm_cpu_realizefn().
-         */
-        cpu->sve_max_vq = ARM_SVE_INIT;
-        object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_max_vq,
-                            cpu_max_set_sve_max_vq, NULL, NULL, &error_fatal);
-
-        /*
-         * sve_vq_map uses a special state while setting properties, so
-         * we initialize it here with its init function and finalize it
-         * in arm_cpu_realizefn().
-         */
-        arm_cpu_vq_map_init(cpu);
-        for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
-            char name[8];
-            sprintf(name, "sve%d", vq * 128);
-            object_property_add(obj, name, "bool", cpu_arm_get_sve_vq,
-                                cpu_arm_set_sve_vq, NULL, NULL, &error_fatal);
-        }
     }
 
     object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
                         cpu_arm_set_sve, NULL, NULL, &error_fatal);
+
+    /*
+     * sve_max_vq is initially unspecified, but must be initialized to a
+     * non-zero value (ARM_SVE_INIT) to indicate that this cpu type has
+     * SVE. It will be finalized in arm_cpu_realizefn().
+     */
+    cpu->sve_max_vq = ARM_SVE_INIT;
+    object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_max_vq,
+                        cpu_max_set_sve_max_vq, NULL, NULL, &error_fatal);
+
+    /*
+     * sve_vq_map uses a special state while setting properties, so
+     * we initialize it here with its init function and finalize it
+     * in arm_cpu_realizefn().
+     */
+    arm_cpu_vq_map_init(cpu);
+    for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
+        char name[8];
+        sprintf(name, "sve%d", vq * 128);
+        object_property_add(obj, name, "bool", cpu_arm_get_sve_vq,
+                            cpu_arm_set_sve_vq, NULL, NULL, &error_fatal);
+    }
 }
 
 struct ARMCPUInfo {
diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
index 2821135a4d0e..5b0707e1192b 100644
--- a/target/arm/kvm64.c
+++ b/target/arm/kvm64.c
@@ -617,6 +617,110 @@ bool kvm_arm_sve_supported(CPUState *cpu)
     return ret > 0;
 }
 
+QEMU_BUILD_BUG_ON(KVM_ARM64_SVE_VQ_MIN != 1);
+
+void kvm_arm_sve_get_vls(CPUState *cs, unsigned long *map,
+                         uint32_t qemu_max_vq, uint32_t *kvm_max_vq)
+{
+    static uint64_t vls[KVM_ARM64_SVE_VLS_WORDS];
+    static uint32_t host_max_vq = -1;
+    uint32_t vq;
+    int i, j;
+
+    bitmap_clear(map, 0, qemu_max_vq);
+    *kvm_max_vq = 0;
+
+    /*
+     * KVM ensures all host CPUs support the same set of vector lengths.
+     * So we only need to create a scratch VCPU once and then cache the
+     * results.
+     */
+    if (host_max_vq == -1) {
+        int fdarray[3], ret = -1;
+
+        if (!kvm_arm_create_scratch_host_vcpu(NULL, fdarray, NULL)) {
+            error_report("failed to create scratch vcpu");
+            abort();
+        }
+
+        if (ioctl(fdarray[0], KVM_CHECK_EXTENSION, KVM_CAP_ARM_SVE) > 0) {
+            struct kvm_vcpu_init init = {
+                .target = -1,
+                .features[0] = (1 << KVM_ARM_VCPU_SVE),
+            };
+            struct kvm_one_reg reg = {
+                .id = KVM_REG_ARM64_SVE_VLS,
+                .addr = (uint64_t)&vls[0],
+            };
+
+            kvm_arm_destroy_scratch_host_vcpu(fdarray);
+
+            if (!kvm_arm_create_scratch_host_vcpu(NULL, fdarray, &init)) {
+                error_report("failed to create scratch vcpu");
+                abort();
+            }
+
+            ret = ioctl(fdarray[2], KVM_GET_ONE_REG, &reg);
+            if (ret) {
+                error_report("failed to get KVM_REG_ARM64_SVE_VLS: %s",
+                             strerror(errno));
+                abort();
+            }
+        }
+
+        kvm_arm_destroy_scratch_host_vcpu(fdarray);
+
+        if (ret) {
+            /* The host doesn't support SVE. */
+            return;
+        }
+    }
+
+    for (i = KVM_ARM64_SVE_VLS_WORDS - 1; i >= 0; --i) {
+        if (!vls[i]) {
+            continue;
+        }
+        if (host_max_vq == -1) {
+            host_max_vq = 64 - clz64(vls[i]) + i * 64;
+        }
+        for (j = 1; j <= 64; ++j) {
+            vq = j + i * 64;
+            if (vq > qemu_max_vq) {
+                break;
+            }
+            if (vls[i] & (1UL << (j - 1))) {
+                set_bit(vq - 1, map);
+            }
+        }
+    }
+
+    *kvm_max_vq = host_max_vq;
+}
+
+static int kvm_arm_sve_set_vls(CPUState *cs)
+{
+    uint64_t vls[KVM_ARM64_SVE_VLS_WORDS] = {0};
+    struct kvm_one_reg reg = {
+        .id = KVM_REG_ARM64_SVE_VLS,
+        .addr = (uint64_t)&vls[0],
+    };
+    ARMCPU *cpu = ARM_CPU(cs);
+    uint32_t vq;
+    int i, j;
+
+    assert(cpu->sve_max_vq <= KVM_ARM64_SVE_VQ_MAX);
+
+    for (vq = 1; vq <= cpu->sve_max_vq; ++vq) {
+        if (test_bit(vq - 1, cpu->sve_vq_map)) {
+            i = (vq - 1) / 64;
+            j = (vq - 1) % 64;
+            vls[i] |= 1UL << j;
+        }
+    }
+
+    return kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
+}
+
 #define ARM_CPU_ID_MPIDR       3, 0, 0, 0, 5
 
 int kvm_arch_init_vcpu(CPUState *cs)
@@ -628,7 +732,7 @@ int kvm_arch_init_vcpu(CPUState *cs)
 
     if (cpu->kvm_target == QEMU_KVM_ARM_TARGET_NONE ||
         !object_dynamic_cast(OBJECT(cpu), TYPE_AARCH64_CPU)) {
-        fprintf(stderr, "KVM is not supported for this guest CPU type\n");
+        error_report("KVM is not supported for this guest CPU type");
         return -EINVAL;
     }
 
@@ -653,11 +757,8 @@ int kvm_arch_init_vcpu(CPUState *cs)
         unset_feature(&env->features, ARM_FEATURE_PMU);
     }
     if (cpu->sve_max_vq) {
-        if (!kvm_arm_sve_supported(cs)) {
-            cpu->sve_max_vq = 0;
-        } else {
-            cpu->kvm_init_features[0] |= 1 << KVM_ARM_VCPU_SVE;
-        }
+        assert(kvm_arm_sve_supported(cs));
+        cpu->kvm_init_features[0] |= 1 << KVM_ARM_VCPU_SVE;
     }
 
     /* Do KVM_ARM_VCPU_INIT ioctl */
@@ -667,6 +768,10 @@ int kvm_arch_init_vcpu(CPUState *cs)
     }
 
     if (cpu->sve_max_vq) {
+        ret = kvm_arm_sve_set_vls(cs);
+        if (ret) {
+            return ret;
+        }
         ret = kvm_arm_vcpu_finalize(cs, KVM_ARM_VCPU_SVE);
         if (ret) {
             return ret;
diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
index 2367f8ab78ed..d5eb341e906d 100644
--- a/target/arm/kvm_arm.h
+++ b/target/arm/kvm_arm.h
@@ -212,6 +212,22 @@ typedef struct ARMHostCPUFeatures {
  */
 bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf);
 
+/**
+ * kvm_arm_sve_get_vls:
+ * @cs: CPUState
+ * @map: bitmap to fill in
+ * @qemu_max_vq: the maximum vector length QEMU supports in quadwords
+ *               (size of the bitmap to fill in)
+ * @kvm_max_vq: the maximum vector length KVM supports in quadwords
+ *
+ * Get all the SVE vector lengths supported by the KVM host, setting
+ * the bits corresponding to their length in quadwords minus one
+ * (vq - 1) in @map up to @qemu_max_vq. Also assign @kvm_max_vq to the
+ * maximum vector length the KVM host supports.
+ */
+void kvm_arm_sve_get_vls(CPUState *cs, unsigned long *map,
+                         uint32_t qemu_max_vq, uint32_t *kvm_max_vq);
+
 /**
  * kvm_arm_set_cpu_features_from_host:
  * @cpu: ARMCPU to set the features for
@@ -314,6 +330,9 @@ static inline int kvm_arm_vgic_probe(void)
 static inline void kvm_arm_pmu_set_irq(CPUState *cs, int irq) {}
 static inline void kvm_arm_pmu_init(CPUState *cs) {}
 
+static inline void kvm_arm_sve_get_vls(CPUState *cs, unsigned long *map,
+                                       uint32_t qemu_max_vq,
+                                       uint32_t *kvm_max_vq) {}
 #endif
 
 static inline const char *gic_class_name(void)
diff --git a/target/arm/monitor.c b/target/arm/monitor.c
index 1e213906fd8f..284818fb4a51 100644
--- a/target/arm/monitor.c
+++ b/target/arm/monitor.c
@@ -100,7 +100,7 @@ QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
  *
  * The sve<vl-bits> features need to be in reverse order in order to
  * enable/disable the largest vector lengths first, ensuring all
- * power-of-2 vector lengths smaller can also be enabled/disabled.
+ * smaller required vector lengths can also be enabled/disabled.
  */
 static const char *cpu_model_advertised_features[] = {
     "aarch64", "pmu", "sve",
diff --git a/tests/arm-cpu-features.c b/tests/arm-cpu-features.c
index 67ad5f2b78d5..349bd0dca6d1 100644
--- a/tests/arm-cpu-features.c
+++ b/tests/arm-cpu-features.c
@@ -324,23 +324,35 @@ static void sve_tests_sve_max_vq_8(const void *data)
     qtest_quit(qts);
 }
 
-static void sve_tests_sve_off(const void *data)
+static void sve_tests_off(QTestState *qts, const char *cpu_type)
 {
-    QTestState *qts;
-
-    qts = qtest_init(MACHINE "-cpu max,sve=off");
-
     /*
      * SVE is off, so the map should be empty.
      */
-    assert_sve_vls(qts, "max", 0, NULL);
+    assert_sve_vls(qts, cpu_type, 0, NULL);
 
     /*
      * We can't turn anything on, but off is OK.
      */
-    assert_error(qts, "max", "cannot enable sve128", "{ 'sve128': true }");
-    assert_sve_vls(qts, "max", 0, "{ 'sve128': false }");
+    assert_error(qts, cpu_type, "cannot enable sve128", "{ 'sve128': true }");
+    assert_sve_vls(qts, cpu_type, 0, "{ 'sve128': false }");
+}
 
+static void sve_tests_sve_off(const void *data)
+{
+    QTestState *qts;
+
+    qts = qtest_init(MACHINE "-cpu max,sve=off");
+    sve_tests_off(qts, "max");
+    qtest_quit(qts);
+}
+
+static void sve_tests_sve_off_kvm(const void *data)
+{
+    QTestState *qts;
+
+    qts = qtest_init(MACHINE "-accel kvm -cpu max,sve=off");
+    sve_tests_off(qts, "max");
     qtest_quit(qts);
 }
 
@@ -392,12 +404,66 @@ static void test_query_cpu_model_expansion_kvm(const void *data)
     assert_has_feature(qts, "host", "pmu");
 
     if (g_str_equal(qtest_get_arch(), "aarch64")) {
+        bool kvm_supports_sve;
+        uint32_t max_vq, vq;
+        uint64_t vls;
+        char name[8];
+        QDict *resp;
+        char *error;
+
         assert_has_feature(qts, "host", "aarch64");
-        assert_has_feature(qts, "max", "sve");
 
         assert_error(qts, "cortex-a15",
             "The CPU definition 'cortex-a15' cannot "
             "be used with KVM on this host", NULL);
+
+        assert_has_feature(qts, "max", "sve");
+        resp = do_query_no_props(qts, "max");
+        g_assert(resp);
+        kvm_supports_sve = qdict_get_bool(resp_get_props(resp), "sve");
+        qobject_unref(resp);
+
+        if (kvm_supports_sve) {
+            resp = do_query_no_props(qts, "max");
+            resp_get_sve_vls(resp, &vls, &max_vq);
+            g_assert(max_vq != 0);
+            qobject_unref(resp);
+
+            /* Enabling a supported length is of course fine. */
+            sprintf(name, "sve%d", max_vq * 128);
+            assert_sve_vls(qts, "max", vls, "{ %s: true }", name);
+
+            /* Also disabling the largest lengths is fine. */
+            assert_sve_vls(qts, "max", (vls & ~BIT(max_vq - 1)),
+                           "{ %s: false }", name);
+
+            for (vq = 1; vq <= max_vq; ++vq) {
+                if (!(vls & BIT(vq - 1))) {
+                    /* vq is unsupported */
+                    break;
+                }
+            }
+            if (vq <= SVE_MAX_VQ) {
+                sprintf(name, "sve%d", vq * 128);
+                error = g_strdup_printf("cannot enable %s", name);
+                assert_error(qts, "max", error, "{ %s: true }", name);
+                g_free(error);
+            }
+
+            if (max_vq > 1) {
+                /* The next smaller, supported vq is required */
+                vq = 64 - __builtin_clzll(vls & ~BIT(max_vq - 1));
+                sprintf(name, "sve%d", vq * 128);
+                error = g_strdup_printf("cannot disable %s", name);
+                assert_error(qts, "max", error, "{ %s: false }", name);
+                g_free(error);
+            }
+        } else {
+            resp = do_query_no_props(qts, "max");
+            resp_get_sve_vls(resp, &vls, &max_vq);
+            g_assert(max_vq == 0);
+            qobject_unref(resp);
+        }
     } else {
         assert_error(qts, "host",
                      "'pmu' feature not supported by KVM on this host",
@@ -434,6 +500,8 @@ int main(int argc, char **argv)
     if (kvm_available) {
         qtest_add_data_func("/arm/kvm/query-cpu-model-expansion",
                             NULL, test_query_cpu_model_expansion_kvm);
+        qtest_add_data_func("/arm/kvm/query-cpu-model-expansion/sve-off",
+                            NULL, sve_tests_sve_off_kvm);
     }
 
     return g_test_run();
-- 
2.20.1

Re: [Qemu-devel] [PATCH v2 13/14] target/arm/cpu64: max cpu: Support sve properties with KVM

[Auger Eric]

Hi Drew,

On 6/21/19 6:34 PM, Andrew Jones wrote:
> Extend the SVE vq map initialization and validation with KVM's
> supported vector lengths when KVM is enabled. In order to determine
> and select supported lengths we add two new KVM functions for getting
> and setting the KVM_REG_ARM64_SVE_VLS pseudo-register.
> 
> Signed-off-by: Andrew Jones <drjones@redhat.com>
> ---
>  target/arm/cpu.h         |   3 +-
>  target/arm/cpu64.c       | 171 +++++++++++++++++++++++++++------------
>  target/arm/kvm64.c       | 117 +++++++++++++++++++++++++--
>  target/arm/kvm_arm.h     |  19 +++++
>  target/arm/monitor.c     |   2 +-
>  tests/arm-cpu-features.c |  86 +++++++++++++++++---
>  6 files changed, 331 insertions(+), 67 deletions(-)
> 
> diff --git a/target/arm/cpu.h b/target/arm/cpu.h
> index cbb155cf72a5..8a1c6c66a462 100644
> --- a/target/arm/cpu.h
> +++ b/target/arm/cpu.h
> @@ -926,7 +926,8 @@ struct ARMCPU {
>       * (bit-number + 1) * 16 bytes, i.e. each bit number + 1 is the vector
>       * length in quadwords. We need a map size twice the maximum
>       * quadword length though because we use two bits for each vector
> -     * length in order to track three states: uninitialized, off, and on.
> +     * length in order to track four states: uninitialized, uninitialized
> +     * but supported by KVM, off, and on.
>       */
>      DECLARE_BITMAP(sve_vq_map, ARM_MAX_VQ * 2);
>  };
> diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
> index 2e595ad53137..6e92aa54b9c8 100644
> --- a/target/arm/cpu64.c
> +++ b/target/arm/cpu64.c
> @@ -261,10 +261,11 @@ static void aarch64_a72_initfn(Object *obj)
>   * While we eventually use cpu->sve_vq_map as a typical bitmap, where each vq
>   * has only two states (off/on), until we've finalized the map at realize time
>   * we use an extra bit, at the vq - 1 + ARM_MAX_VQ bit number, to also allow
> - * tracking of the uninitialized state. The arm_vq_state typedef and following
> - * functions allow us to more easily work with the bitmap. Also, while the map
> - * is still initializing, sve-max-vq has an additional three states, bringing
> - * the number of its states to five, which are the following:
> + * tracking of the uninitialized state and the uninitialized but supported by
> + * KVM state. The arm_vq_state typedef and following functions allow us to more
> + * easily work with the bitmap. Also, while the map is still initializing,
> + * sve-max-vq has an additional three states, bringing the number of its states
> + * to five, which are the following:
>   *
>   * sve-max-vq:
>   *   0:    SVE is disabled. The default value for a vq in the map is 'OFF'.
> @@ -296,6 +297,11 @@ typedef enum arm_vq_state {
>      ARM_VQ_OFF,
>      ARM_VQ_ON,
>      ARM_VQ_UNINITIALIZED,
> +    ARM_VQ_UNINITIALIZED_KVM_SUPPORTED
> +    /*
> +     * More states cannot be added without adding bits to sve_vq_map
> +     * and modifying its supporting functions.
> +     */
>  } arm_vq_state;
>  
>  static arm_vq_state arm_cpu_vq_map_get(ARMCPU *cpu, int vq)
> @@ -324,6 +330,23 @@ static void arm_cpu_vq_map_init(ARMCPU *cpu)
>  {
>      bitmap_zero(cpu->sve_vq_map, ARM_MAX_VQ * 2);
>      bitmap_set(cpu->sve_vq_map, ARM_MAX_VQ, ARM_MAX_VQ);
> +
> +    if (kvm_enabled()) {
> +        DECLARE_BITMAP(kvm_supported, ARM_MAX_VQ);
> +        uint32_t kvm_max_vq;
> +
> +        bitmap_zero(kvm_supported, ARM_MAX_VQ);
> +
> +        kvm_arm_sve_get_vls(CPU(cpu), kvm_supported, ARM_MAX_VQ, &kvm_max_vq);
> +
> +        if (kvm_max_vq > ARM_MAX_VQ) {
> +            warn_report("KVM supports vector lengths larger than "
> +                        "QEMU can enable");
> +        }
> +
> +        bitmap_or(cpu->sve_vq_map, cpu->sve_vq_map,
> +                  kvm_supported, ARM_MAX_VQ);
So you store the KVM supported state in the 1st half of the bitmap
(ON/OFF)? That's confusing actually. And the second half of the bitmap
still is used to encode whether the state is UNITIALIZED? As Richard
suggested would be clearer with separate bitmaps including for the KVM
supported state.
> +    }
>  }
>  
>  static bool arm_cpu_vq_map_is_finalized(ARMCPU *cpu)
> @@ -371,12 +394,7 @@ void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp)
>          return;
>      }
>  
> -    /* sve-max-vq and sve<vl-bits> properties not yet implemented for KVM */
> -    if (kvm_enabled()) {
> -        return;
> -    }
> -
> -    if (cpu->sve_max_vq == ARM_SVE_INIT) {
> +    if (!kvm_enabled() && cpu->sve_max_vq == ARM_SVE_INIT) {
>          object_property_set_uint(OBJECT(cpu), ARM_MAX_VQ, "sve-max-vq", &err);
>          if (err) {
>              error_propagate(errp, err);
> @@ -431,6 +449,11 @@ static void cpu_max_set_sve_max_vq(Object *obj, Visitor *v, const char *name,
>          return;
>      }
>  
> +    if (kvm_enabled() && !kvm_arm_sve_supported(CPU(cpu))) {
> +        error_setg(errp, "'sve' feature not supported by KVM on this host");
> +        return;
> +    }
> +
>      /*
>       * It gets complicated trying to support both sve-max-vq and
>       * sve<vl-bits> properties together, so we mostly don't. We
> @@ -460,6 +483,12 @@ static void cpu_max_set_sve_max_vq(Object *obj, Visitor *v, const char *name,
>              sprintf(name, "sve%d", vq * 128);
>              object_property_set_bool(obj, true, name, &err);
>              if (err) {
> +                if (kvm_enabled()) {
> +                    error_append_hint(&err, "It is not possible to use "
> +                                      "sve-max-vq with this KVM host. Try "
> +                                      "using only sve<vl-bits> "
> +                                      "properties.\n");
If I understand correctly, the problem is that setting sve-max-vq to a
given value implies all <= required lengths need to be supported and at
least one isn't. In that case the object_property_set_bool will produce
a "cannot enable %s", with a hint "This KVM host does not support
                the vector length %d-bits". So here you may simply say
sve-max-vq cannot be used as one of the requuired 128b length inferior
or equal to sve-max-vq is not supported by the host.


> +                }
>                  error_propagate(errp, err);
>                  return;
>              }
> @@ -484,6 +513,12 @@ static void cpu_arm_get_sve_vq(Object *obj, Visitor *v, const char *name,
>          value = true;
>      } else if (vq_state == ARM_VQ_OFF) {
>          value = false;
> +    } else if (kvm_enabled() && vq_state == ARM_VQ_UNINITIALIZED) {
> +        /*
> +         * When KVM is enabled, anything not supported by the host must have
> +         * 'OFF' for the default.
> +         */
> +        value = false;
>      } else {
>          /*
>           * vq is uninitialized. We pick a default here based on the
> @@ -539,6 +574,11 @@ static void cpu_arm_set_sve_vq(Object *obj, Visitor *v, const char *name,
>          return;
>      }
>  
> +    if (value && kvm_enabled() && !kvm_arm_sve_supported(CPU(cpu))) {
> +        error_setg(errp, "'sve' feature not supported by KVM on this host");
> +        return;
> +    }
> +
>      if (arm_sve_have_max_vq(cpu)) {
>          max_vq = cpu->sve_max_vq;
>      } else {
> @@ -569,6 +609,8 @@ static void cpu_arm_set_sve_vq(Object *obj, Visitor *v, const char *name,
>          }
>      }
>  
> +    vq_state = arm_cpu_vq_map_get(cpu, vq);
> +
>      if (arm_sve_have_max_vq(cpu) && value && vq > cpu->sve_max_vq) {
>          error_setg(errp, "cannot enable %s", name);
>          error_append_hint(errp, "vq=%d (%d bits) is larger than the "
> @@ -580,19 +622,31 @@ static void cpu_arm_set_sve_vq(Object *obj, Visitor *v, const char *name,
>          error_append_hint(errp, "The maximum vector length must be "
>                            "enabled, sve-max-vq=%d (%d bits)\n",
>                            cpu->sve_max_vq, cpu->sve_max_vq * 128);
> -    } else if (arm_sve_have_max_vq(cpu) && !value && vq < cpu->sve_max_vq &&
> -               is_power_of_2(vq)) {
> +    } else if (!kvm_enabled() && arm_sve_have_max_vq(cpu) && !value &&
> +               vq < cpu->sve_max_vq && is_power_of_2(vq)) {
>          error_setg(errp, "cannot disable %s", name);
>          error_append_hint(errp, "vq=%d (%d bits) is required as it is a "
>                            "power-of-2 length smaller than the maximum, "
>                            "sve-max-vq=%d (%d bits)\n", vq, vq * 128,
>                            cpu->sve_max_vq, cpu->sve_max_vq * 128);
> -    } else if (!value && vq < max_vq && is_power_of_2(vq)) {
> +    } else if (!kvm_enabled() && !value && vq < max_vq && is_power_of_2(vq)) {
>          error_setg(errp, "cannot disable %s", name);
>          error_append_hint(errp, "Vector length %d-bits is required as it "
>                            "is a power-of-2 length smaller than another "
>                            "enabled vector length. Disable all larger vector "
>                            "lengths first.\n", vq * 128);
> +    } else if (kvm_enabled() && value && vq_state == ARM_VQ_UNINITIALIZED) {
> +        error_setg(errp, "cannot enable %s", name);
> +        error_append_hint(errp, "This KVM host does not support "
> +                          "the vector length %d-bits.\n", vq * 128);
> +    } else if (kvm_enabled() && !value && vq < max_vq &&
> +               (vq_state == ARM_VQ_ON ||
> +                vq_state == ARM_VQ_UNINITIALIZED_KVM_SUPPORTED)) {
> +        error_setg(errp, "cannot disable %s", name);
> +        error_append_hint(errp, "Vector length %d-bits is a KVM supported "
> +                          "length smaller than another enabled vector "
> +                          "length. Disable all larger vector lengths "
> +                          "first.\n", vq * 128);
all those checks may be more readable if segment into kvm / !kvm, value
/ !value
>      } else {
>          if (value) {
>              bool fail = false;
> @@ -602,31 +656,53 @@ static void cpu_arm_set_sve_vq(Object *obj, Visitor *v, const char *name,
>               * Enabling a vector length automatically enables all
>               * uninitialized power-of-2 lengths smaller than it, as
>               * per the architecture.
> +             *
> +             * For KVM we have to automatically enable all supported,
> +             * uninitialized lengths smaller than this length, even
> +             * when it's not a power-of-2.
>               */
>              for (s = 1; s < vq; ++s) {
> -                if (is_power_of_2(s)) {
> -                    vq_state = arm_cpu_vq_map_get(cpu, s);
> -                    if (vq_state == ARM_VQ_UNINITIALIZED) {
> -                        arm_cpu_vq_map_set(cpu, s, ARM_VQ_ON);
> -                    } else if (vq_state == ARM_VQ_OFF) {
> -                        fail = true;
> -                        break;
> -                    }
> +                vq_state = arm_cpu_vq_map_get(cpu, s);
> +                if (!kvm_enabled() && is_power_of_2(s) &&
> +                    vq_state == ARM_VQ_UNINITIALIZED) {
> +                    arm_cpu_vq_map_set(cpu, s, ARM_VQ_ON);
> +                } else if (vq_state == ARM_VQ_UNINITIALIZED_KVM_SUPPORTED) {
> +                    assert(kvm_enabled());
> +                    arm_cpu_vq_map_set(cpu, s, ARM_VQ_ON);
> +                } else if ((kvm_enabled() || is_power_of_2(s)) &&
> +                           vq_state == ARM_VQ_OFF) {
> +                    fail = true;
> +                    break;
>                  }
>              }
>  
> -            if (fail) {
> +            if (!kvm_enabled() && fail) {
>                  error_setg(errp, "cannot enable %s", name);
>                  error_append_hint(errp, "Vector length %d-bits is disabled "
>                                    "and is a power-of-2 length smaller than "
>                                    "%s. All power-of-2 vector lengths smaller "
>                                    "than the maximum length are required.\n",
>                                    s * 128, name);
> +
> +            } else if (fail) {
> +                error_setg(errp, "cannot enable %s", name);
> +                error_append_hint(errp, "Vector length %d-bits is disabled "
> +                                  "and the KVM host requires all supported "
> +                                  "vector lengths smaller than %s to also be "
> +                                  "enabled.\n", s * 128, name);
>              } else {
>                  arm_cpu_vq_map_set(cpu, vq, ARM_VQ_ON);
>              }
>          } else {
> -            arm_cpu_vq_map_set(cpu, vq, ARM_VQ_OFF);
> +            /*
> +             * For KVM if the vq wasn't supported then we leave it in
> +             * the ARM_VQ_UNINITIALIZED state in order to keep that
> +             * unsupported information. It'll be set to OFF later when
> +             * we finalize the map.
> +             */
> +            if (!kvm_enabled() || vq_state != ARM_VQ_UNINITIALIZED) {
> +                arm_cpu_vq_map_set(cpu, vq, ARM_VQ_OFF);
> +            }
>          }
>      }
>  }
> @@ -689,11 +765,6 @@ static void aarch64_max_initfn(Object *obj)
>  
>      if (kvm_enabled()) {
>          kvm_arm_set_cpu_features_from_host(cpu);
> -        /*
> -         * KVM doesn't yet support the sve-max-vq property, but
> -         * setting cpu->sve_max_vq is also used to turn SVE on.
> -         */
> -        cpu->sve_max_vq = ARM_SVE_INIT;
>      } else {
>          uint64_t t;
>          uint32_t u;
> @@ -774,32 +845,32 @@ static void aarch64_max_initfn(Object *obj)
>          cpu->ctr = 0x80038003; /* 32 byte I and D cacheline size, VIPT icache */
>          cpu->dcz_blocksize = 7; /*  512 bytes */
>  #endif
> -
> -        /*
> -         * sve_max_vq is initially unspecified, but must be initialized to a
> -         * non-zero value (ARM_SVE_INIT) to indicate that this cpu type has
> -         * SVE. It will be finalized in arm_cpu_realizefn().
> -         */
> -        cpu->sve_max_vq = ARM_SVE_INIT;
> -        object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_max_vq,
> -                            cpu_max_set_sve_max_vq, NULL, NULL, &error_fatal);
> -
> -        /*
> -         * sve_vq_map uses a special state while setting properties, so
> -         * we initialize it here with its init function and finalize it
> -         * in arm_cpu_realizefn().
> -         */
> -        arm_cpu_vq_map_init(cpu);
> -        for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
> -            char name[8];
> -            sprintf(name, "sve%d", vq * 128);
> -            object_property_add(obj, name, "bool", cpu_arm_get_sve_vq,
> -                                cpu_arm_set_sve_vq, NULL, NULL, &error_fatal);
> -        }
>      }
>  
>      object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
>                          cpu_arm_set_sve, NULL, NULL, &error_fatal);
> +
> +    /*
> +     * sve_max_vq is initially unspecified, but must be initialized to a
> +     * non-zero value (ARM_SVE_INIT) to indicate that this cpu type has
> +     * SVE. It will be finalized in arm_cpu_realizefn().
> +     */
> +    cpu->sve_max_vq = ARM_SVE_INIT;
> +    object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_max_vq,
> +                        cpu_max_set_sve_max_vq, NULL, NULL, &error_fatal);
> +
> +    /*
> +     * sve_vq_map uses a special state while setting properties, so
> +     * we initialize it here with its init function and finalize it
> +     * in arm_cpu_realizefn().
> +     */
> +    arm_cpu_vq_map_init(cpu);
> +    for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
> +        char name[8];
> +        sprintf(name, "sve%d", vq * 128);
> +        object_property_add(obj, name, "bool", cpu_arm_get_sve_vq,
> +                            cpu_arm_set_sve_vq, NULL, NULL, &error_fatal);
> +    }
>  }
>  
>  struct ARMCPUInfo {
> diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> index 2821135a4d0e..5b0707e1192b 100644
> --- a/target/arm/kvm64.c
> +++ b/target/arm/kvm64.c
> @@ -617,6 +617,110 @@ bool kvm_arm_sve_supported(CPUState *cpu)
>      return ret > 0;
>  }
>  
> +QEMU_BUILD_BUG_ON(KVM_ARM64_SVE_VQ_MIN != 1);
> +
> +void kvm_arm_sve_get_vls(CPUState *cs, unsigned long *map,
> +                         uint32_t qemu_max_vq, uint32_t *kvm_max_vq)
you could have kvm_max_vq as a returned value?
> +{
> +    static uint64_t vls[KVM_ARM64_SVE_VLS_WORDS];
> +    static uint32_t host_max_vq = -1;
> +    uint32_t vq;
> +    int i, j;
> +
> +    bitmap_clear(map, 0, qemu_max_vq);
> +    *kvm_max_vq = 0;
> +
> +    /*
> +     * KVM ensures all host CPUs support the same set of vector lengths.
> +     * So we only need to create a scratch VCPU once and then cache the
> +     * results.
> +     */
> +    if (host_max_vq == -1) {
always true?
> +        int fdarray[3], ret = -1;
> +
> +        if (!kvm_arm_create_scratch_host_vcpu(NULL, fdarray, NULL)) {
> +            error_report("failed to create scratch vcpu");
> +            abort();
> +        }
> +
> +        if (ioctl(fdarray[0], KVM_CHECK_EXTENSION, KVM_CAP_ARM_SVE) > 0) {
suggestion:

has_sve = ioctl(fdarray[0], KVM_CHECK_EXTENSION, KVM_CAP_ARM_SVE);
kvm_arm_destroy_scratch_host_vcpu(fdarray);
if (!has_sve) {
    return;
}

/* host supports SVE */
../..
> +            struct kvm_vcpu_init init = {
> +                .target = -1,
> +                .features[0] = (1 << KVM_ARM_VCPU_SVE),
> +            };
> +            struct kvm_one_reg reg = {
> +                .id = KVM_REG_ARM64_SVE_VLS,
> +                .addr = (uint64_t)&vls[0],
> +            };
> +
> +            kvm_arm_destroy_scratch_host_vcpu(fdarray);
> +
> +            if (!kvm_arm_create_scratch_host_vcpu(NULL, fdarray, &init)) {
> +                error_report("failed to create scratch vcpu");
> +                abort();
> +            }
> +
> +            ret = ioctl(fdarray[2], KVM_GET_ONE_REG, &reg);
> +            if (ret) {
> +                error_report("failed to get KVM_REG_ARM64_SVE_VLS: %s",
> +                             strerror(errno));
> +                abort();
> +            }
> +        }
> +
> +        kvm_arm_destroy_scratch_host_vcpu(fdarray);
> +
> +        if (ret) {
> +            /* The host doesn't support SVE. */
> +            return;
> +        }
> +    }
> +
> +    for (i = KVM_ARM64_SVE_VLS_WORDS - 1; i >= 0; --i) {
> +        if (!vls[i]) {
> +            continue;
> +        }
> +        if (host_max_vq == -1) {
> +            host_max_vq = 64 - clz64(vls[i]) + i * 64;
> +        }
> +        for (j = 1; j <= 64; ++j) {
> +            vq = j + i * 64;
> +            if (vq > qemu_max_vq) {
> +                break;
> +            }
> +            if (vls[i] & (1UL << (j - 1))) {
> +                set_bit(vq - 1, map);
> +            }
> +        }
> +    }
> +
> +    *kvm_max_vq = host_max_vq;
> +}
> +
> +static int kvm_arm_sve_set_vls(CPUState *cs)
> +{
> +    uint64_t vls[KVM_ARM64_SVE_VLS_WORDS] = {0};
> +    struct kvm_one_reg reg = {
> +        .id = KVM_REG_ARM64_SVE_VLS,
> +        .addr = (uint64_t)&vls[0],
> +    };
> +    ARMCPU *cpu = ARM_CPU(cs);
> +    uint32_t vq;
> +    int i, j;
> +
> +    assert(cpu->sve_max_vq <= KVM_ARM64_SVE_VQ_MAX);
> +
> +    for (vq = 1; vq <= cpu->sve_max_vq; ++vq) {
> +        if (test_bit(vq - 1, cpu->sve_vq_map)) {
> +            i = (vq - 1) / 64;
> +            j = (vq - 1) % 64;
> +            vls[i] |= 1UL << j;
> +        }
> +    }
> +
> +    return kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
> +}
> +
>  #define ARM_CPU_ID_MPIDR       3, 0, 0, 0, 5
>  
>  int kvm_arch_init_vcpu(CPUState *cs)
> @@ -628,7 +732,7 @@ int kvm_arch_init_vcpu(CPUState *cs)
>  
>      if (cpu->kvm_target == QEMU_KVM_ARM_TARGET_NONE ||
>          !object_dynamic_cast(OBJECT(cpu), TYPE_AARCH64_CPU)) {
> -        fprintf(stderr, "KVM is not supported for this guest CPU type\n");
> +        error_report("KVM is not supported for this guest CPU type");
>          return -EINVAL;
>      }
>  
> @@ -653,11 +757,8 @@ int kvm_arch_init_vcpu(CPUState *cs)
>          unset_feature(&env->features, ARM_FEATURE_PMU);
>      }
>      if (cpu->sve_max_vq) {
> -        if (!kvm_arm_sve_supported(cs)) {
> -            cpu->sve_max_vq = 0;
> -        } else {
> -            cpu->kvm_init_features[0] |= 1 << KVM_ARM_VCPU_SVE;
> -        }
> +        assert(kvm_arm_sve_supported(cs));
> +        cpu->kvm_init_features[0] |= 1 << KVM_ARM_VCPU_SVE;
>      }
>  
>      /* Do KVM_ARM_VCPU_INIT ioctl */
> @@ -667,6 +768,10 @@ int kvm_arch_init_vcpu(CPUState *cs)
>      }
>  
>      if (cpu->sve_max_vq) {
> +        ret = kvm_arm_sve_set_vls(cs);
> +        if (ret) {
> +            return ret;
> +        }
>          ret = kvm_arm_vcpu_finalize(cs, KVM_ARM_VCPU_SVE);
>          if (ret) {
>              return ret;
> diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
> index 2367f8ab78ed..d5eb341e906d 100644
> --- a/target/arm/kvm_arm.h
> +++ b/target/arm/kvm_arm.h
> @@ -212,6 +212,22 @@ typedef struct ARMHostCPUFeatures {
>   */
>  bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf);
>  
> +/**
> + * kvm_arm_sve_get_vls:
kernel-doc comment?
> + * @cs: CPUState
> + * @map: bitmap to fill in
> + * @qemu_max_vq: the maximum vector length QEMU supports in quadwords
> + *               (size of the bitmap to fill in)
> + * @kvm_max_vq: the maximum vector length KVM supports in quadwords
> + *
> + * Get all the SVE vector lengths supported by the KVM host, setting
> + * the bits corresponding to their length in quadwords minus one
> + * (vq - 1) in @map up to @qemu_max_vq. Also assign @kvm_max_vq to the
> + * maximum vector length the KVM host supports.
> + */
> +void kvm_arm_sve_get_vls(CPUState *cs, unsigned long *map,
> +                         uint32_t qemu_max_vq, uint32_t *kvm_max_vq);
> +
>  /**
>   * kvm_arm_set_cpu_features_from_host:
>   * @cpu: ARMCPU to set the features for
> @@ -314,6 +330,9 @@ static inline int kvm_arm_vgic_probe(void)
>  static inline void kvm_arm_pmu_set_irq(CPUState *cs, int irq) {}
>  static inline void kvm_arm_pmu_init(CPUState *cs) {}
>  
> +static inline void kvm_arm_sve_get_vls(CPUState *cs, unsigned long *map,
> +                                       uint32_t qemu_max_vq,
> +                                       uint32_t *kvm_max_vq) {}
>  #endif
>  
>  static inline const char *gic_class_name(void)
> diff --git a/target/arm/monitor.c b/target/arm/monitor.c
> index 1e213906fd8f..284818fb4a51 100644
> --- a/target/arm/monitor.c
> +++ b/target/arm/monitor.c
> @@ -100,7 +100,7 @@ QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
>   *
>   * The sve<vl-bits> features need to be in reverse order in order to
>   * enable/disable the largest vector lengths first, ensuring all
> - * power-of-2 vector lengths smaller can also be enabled/disabled.
> + * smaller required vector lengths can also be enabled/disabled.
>   */
>  static const char *cpu_model_advertised_features[] = {
>      "aarch64", "pmu", "sve",
> diff --git a/tests/arm-cpu-features.c b/tests/arm-cpu-features.c
> index 67ad5f2b78d5..349bd0dca6d1 100644
> --- a/tests/arm-cpu-features.c
> +++ b/tests/arm-cpu-features.c
> @@ -324,23 +324,35 @@ static void sve_tests_sve_max_vq_8(const void *data)
>      qtest_quit(qts);
>  }
>  
> -static void sve_tests_sve_off(const void *data)
> +static void sve_tests_off(QTestState *qts, const char *cpu_type)
>  {
> -    QTestState *qts;
> -
> -    qts = qtest_init(MACHINE "-cpu max,sve=off");
> -
>      /*
>       * SVE is off, so the map should be empty.
>       */
> -    assert_sve_vls(qts, "max", 0, NULL);
> +    assert_sve_vls(qts, cpu_type, 0, NULL);
>  
>      /*
>       * We can't turn anything on, but off is OK.
>       */
> -    assert_error(qts, "max", "cannot enable sve128", "{ 'sve128': true }");
> -    assert_sve_vls(qts, "max", 0, "{ 'sve128': false }");
> +    assert_error(qts, cpu_type, "cannot enable sve128", "{ 'sve128': true }");
> +    assert_sve_vls(qts, cpu_type, 0, "{ 'sve128': false }");
> +}
>  
> +static void sve_tests_sve_off(const void *data)
> +{
> +    QTestState *qts;
> +
> +    qts = qtest_init(MACHINE "-cpu max,sve=off");
> +    sve_tests_off(qts, "max");
> +    qtest_quit(qts);
> +}
> +
> +static void sve_tests_sve_off_kvm(const void *data)
> +{
> +    QTestState *qts;
> +
> +    qts = qtest_init(MACHINE "-accel kvm -cpu max,sve=off");
> +    sve_tests_off(qts, "max");
>      qtest_quit(qts);
>  }
>  
> @@ -392,12 +404,66 @@ static void test_query_cpu_model_expansion_kvm(const void *data)
>      assert_has_feature(qts, "host", "pmu");
>  
>      if (g_str_equal(qtest_get_arch(), "aarch64")) {
> +        bool kvm_supports_sve;
> +        uint32_t max_vq, vq;
> +        uint64_t vls;
> +        char name[8];
> +        QDict *resp;
> +        char *error;
> +
>          assert_has_feature(qts, "host", "aarch64");
> -        assert_has_feature(qts, "max", "sve");
>  
>          assert_error(qts, "cortex-a15",
>              "The CPU definition 'cortex-a15' cannot "
>              "be used with KVM on this host", NULL);
> +
> +        assert_has_feature(qts, "max", "sve");
> +        resp = do_query_no_props(qts, "max");
> +        g_assert(resp);
> +        kvm_supports_sve = qdict_get_bool(resp_get_props(resp), "sve");
> +        qobject_unref(resp);
> +
> +        if (kvm_supports_sve) {
> +            resp = do_query_no_props(qts, "max");
> +            resp_get_sve_vls(resp, &vls, &max_vq);
> +            g_assert(max_vq != 0);
> +            qobject_unref(resp);
> +
> +            /* Enabling a supported length is of course fine. */
> +            sprintf(name, "sve%d", max_vq * 128);
> +            assert_sve_vls(qts, "max", vls, "{ %s: true }", name);
> +
> +            /* Also disabling the largest lengths is fine. */
> +            assert_sve_vls(qts, "max", (vls & ~BIT(max_vq - 1)),
> +                           "{ %s: false }", name);
> +
> +            for (vq = 1; vq <= max_vq; ++vq) {
> +                if (!(vls & BIT(vq - 1))) {
> +                    /* vq is unsupported */
> +                    break;
> +                }
> +            }
> +            if (vq <= SVE_MAX_VQ) {
> +                sprintf(name, "sve%d", vq * 128);
> +                error = g_strdup_printf("cannot enable %s", name);
> +                assert_error(qts, "max", error, "{ %s: true }", name);
> +                g_free(error);
> +            }
> +
> +            if (max_vq > 1) {
> +                /* The next smaller, supported vq is required */
> +                vq = 64 - __builtin_clzll(vls & ~BIT(max_vq - 1));
> +                sprintf(name, "sve%d", vq * 128);
> +                error = g_strdup_printf("cannot disable %s", name);
> +                assert_error(qts, "max", error, "{ %s: false }", name);
> +                g_free(error);
> +            }
> +        } else {
> +            resp = do_query_no_props(qts, "max");
> +            resp_get_sve_vls(resp, &vls, &max_vq);
> +            g_assert(max_vq == 0);
> +            qobject_unref(resp);
> +        }
>      } else {
>          assert_error(qts, "host",
>                       "'pmu' feature not supported by KVM on this host",
> @@ -434,6 +500,8 @@ int main(int argc, char **argv)
>      if (kvm_available) {
>          qtest_add_data_func("/arm/kvm/query-cpu-model-expansion",
>                              NULL, test_query_cpu_model_expansion_kvm);
> +        qtest_add_data_func("/arm/kvm/query-cpu-model-expansion/sve-off",
> +                            NULL, sve_tests_sve_off_kvm);
>      }
>  
>      return g_test_run();
> 

Thanks

Eric

Re: [Qemu-devel] [PATCH v2 13/14] target/arm/cpu64: max cpu: Support sve properties with KVM

[Andrew Jones]

On Fri, Jun 28, 2019 at 05:55:50PM +0200, Auger Eric wrote:
> Hi Drew,
> 
> On 6/21/19 6:34 PM, Andrew Jones wrote:
> > Extend the SVE vq map initialization and validation with KVM's
> > supported vector lengths when KVM is enabled. In order to determine
> > and select supported lengths we add two new KVM functions for getting
> > and setting the KVM_REG_ARM64_SVE_VLS pseudo-register.
> > 
> > Signed-off-by: Andrew Jones <drjones@redhat.com>
> > ---
> >  target/arm/cpu.h         |   3 +-
> >  target/arm/cpu64.c       | 171 +++++++++++++++++++++++++++------------
> >  target/arm/kvm64.c       | 117 +++++++++++++++++++++++++--
> >  target/arm/kvm_arm.h     |  19 +++++
> >  target/arm/monitor.c     |   2 +-
> >  tests/arm-cpu-features.c |  86 +++++++++++++++++---
> >  6 files changed, 331 insertions(+), 67 deletions(-)
> > 
> > diff --git a/target/arm/cpu.h b/target/arm/cpu.h
> > index cbb155cf72a5..8a1c6c66a462 100644
> > --- a/target/arm/cpu.h
> > +++ b/target/arm/cpu.h
> > @@ -926,7 +926,8 @@ struct ARMCPU {
> >       * (bit-number + 1) * 16 bytes, i.e. each bit number + 1 is the vector
> >       * length in quadwords. We need a map size twice the maximum
> >       * quadword length though because we use two bits for each vector
> > -     * length in order to track three states: uninitialized, off, and on.
> > +     * length in order to track four states: uninitialized, uninitialized
> > +     * but supported by KVM, off, and on.
> >       */
> >      DECLARE_BITMAP(sve_vq_map, ARM_MAX_VQ * 2);
> >  };
> > diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
> > index 2e595ad53137..6e92aa54b9c8 100644
> > --- a/target/arm/cpu64.c
> > +++ b/target/arm/cpu64.c
> > @@ -261,10 +261,11 @@ static void aarch64_a72_initfn(Object *obj)
> >   * While we eventually use cpu->sve_vq_map as a typical bitmap, where each vq
> >   * has only two states (off/on), until we've finalized the map at realize time
> >   * we use an extra bit, at the vq - 1 + ARM_MAX_VQ bit number, to also allow
> > - * tracking of the uninitialized state. The arm_vq_state typedef and following
> > - * functions allow us to more easily work with the bitmap. Also, while the map
> > - * is still initializing, sve-max-vq has an additional three states, bringing
> > - * the number of its states to five, which are the following:
> > + * tracking of the uninitialized state and the uninitialized but supported by
> > + * KVM state. The arm_vq_state typedef and following functions allow us to more
> > + * easily work with the bitmap. Also, while the map is still initializing,
> > + * sve-max-vq has an additional three states, bringing the number of its states
> > + * to five, which are the following:
> >   *
> >   * sve-max-vq:
> >   *   0:    SVE is disabled. The default value for a vq in the map is 'OFF'.
> > @@ -296,6 +297,11 @@ typedef enum arm_vq_state {
> >      ARM_VQ_OFF,
> >      ARM_VQ_ON,
> >      ARM_VQ_UNINITIALIZED,
> > +    ARM_VQ_UNINITIALIZED_KVM_SUPPORTED
> > +    /*
> > +     * More states cannot be added without adding bits to sve_vq_map
> > +     * and modifying its supporting functions.
> > +     */
> >  } arm_vq_state;
> >  
> >  static arm_vq_state arm_cpu_vq_map_get(ARMCPU *cpu, int vq)
> > @@ -324,6 +330,23 @@ static void arm_cpu_vq_map_init(ARMCPU *cpu)
> >  {
> >      bitmap_zero(cpu->sve_vq_map, ARM_MAX_VQ * 2);
> >      bitmap_set(cpu->sve_vq_map, ARM_MAX_VQ, ARM_MAX_VQ);
> > +
> > +    if (kvm_enabled()) {
> > +        DECLARE_BITMAP(kvm_supported, ARM_MAX_VQ);
> > +        uint32_t kvm_max_vq;
> > +
> > +        bitmap_zero(kvm_supported, ARM_MAX_VQ);
> > +
> > +        kvm_arm_sve_get_vls(CPU(cpu), kvm_supported, ARM_MAX_VQ, &kvm_max_vq);
> > +
> > +        if (kvm_max_vq > ARM_MAX_VQ) {
> > +            warn_report("KVM supports vector lengths larger than "
> > +                        "QEMU can enable");
> > +        }
> > +
> > +        bitmap_or(cpu->sve_vq_map, cpu->sve_vq_map,
> > +                  kvm_supported, ARM_MAX_VQ);
> So you store the KVM supported state in the 1st half of the bitmap
> (ON/OFF)? That's confusing actually. And the second half of the bitmap
> still is used to encode whether the state is UNITIALIZED? As Richard
> suggested would be clearer with separate bitmaps including for the KVM
> supported state.

You can think of it as two states, which is indeed confusing, or you can
think of it as one state: 3, where the two bits that compose '3' come from
two different parts of the bitmap, rather than from contiguous bits.
That's admittedly a bit complex too, but that's why I introduced helper
functions and a typedef. Perhaps a comment here explaining that this
bitmap_or() creates ARM_VQ_UNINITIALIZED_KVM_SUPPORTED entries would help?

> > +    }
> >  }
> >  
> >  static bool arm_cpu_vq_map_is_finalized(ARMCPU *cpu)
> > @@ -371,12 +394,7 @@ void arm_cpu_sve_finalize(ARMCPU *cpu, Error **errp)
> >          return;
> >      }
> >  
> > -    /* sve-max-vq and sve<vl-bits> properties not yet implemented for KVM */
> > -    if (kvm_enabled()) {
> > -        return;
> > -    }
> > -
> > -    if (cpu->sve_max_vq == ARM_SVE_INIT) {
> > +    if (!kvm_enabled() && cpu->sve_max_vq == ARM_SVE_INIT) {
> >          object_property_set_uint(OBJECT(cpu), ARM_MAX_VQ, "sve-max-vq", &err);
> >          if (err) {
> >              error_propagate(errp, err);
> > @@ -431,6 +449,11 @@ static void cpu_max_set_sve_max_vq(Object *obj, Visitor *v, const char *name,
> >          return;
> >      }
> >  
> > +    if (kvm_enabled() && !kvm_arm_sve_supported(CPU(cpu))) {
> > +        error_setg(errp, "'sve' feature not supported by KVM on this host");
> > +        return;
> > +    }
> > +
> >      /*
> >       * It gets complicated trying to support both sve-max-vq and
> >       * sve<vl-bits> properties together, so we mostly don't. We
> > @@ -460,6 +483,12 @@ static void cpu_max_set_sve_max_vq(Object *obj, Visitor *v, const char *name,
> >              sprintf(name, "sve%d", vq * 128);
> >              object_property_set_bool(obj, true, name, &err);
> >              if (err) {
> > +                if (kvm_enabled()) {
> > +                    error_append_hint(&err, "It is not possible to use "
> > +                                      "sve-max-vq with this KVM host. Try "
> > +                                      "using only sve<vl-bits> "
> > +                                      "properties.\n");
> If I understand correctly, the problem is that setting sve-max-vq to a
> given value implies all <= required lengths need to be supported and at
> least one isn't. In that case the object_property_set_bool will produce
> a "cannot enable %s", with a hint "This KVM host does not support
>                 the vector length %d-bits". So here you may simply say
> sve-max-vq cannot be used as one of the requuired 128b length inferior
> or equal to sve-max-vq is not supported by the host.

Which is more or less what I thought I was doing here. Are you suggesting
I change the error text to something else? If so, can you please provide
a suggestion?

> 
> 
> > +                }
> >                  error_propagate(errp, err);
> >                  return;
> >              }
> > @@ -484,6 +513,12 @@ static void cpu_arm_get_sve_vq(Object *obj, Visitor *v, const char *name,
> >          value = true;
> >      } else if (vq_state == ARM_VQ_OFF) {
> >          value = false;
> > +    } else if (kvm_enabled() && vq_state == ARM_VQ_UNINITIALIZED) {
> > +        /*
> > +         * When KVM is enabled, anything not supported by the host must have
> > +         * 'OFF' for the default.
> > +         */
> > +        value = false;
> >      } else {
> >          /*
> >           * vq is uninitialized. We pick a default here based on the
> > @@ -539,6 +574,11 @@ static void cpu_arm_set_sve_vq(Object *obj, Visitor *v, const char *name,
> >          return;
> >      }
> >  
> > +    if (value && kvm_enabled() && !kvm_arm_sve_supported(CPU(cpu))) {
> > +        error_setg(errp, "'sve' feature not supported by KVM on this host");
> > +        return;
> > +    }
> > +
> >      if (arm_sve_have_max_vq(cpu)) {
> >          max_vq = cpu->sve_max_vq;
> >      } else {
> > @@ -569,6 +609,8 @@ static void cpu_arm_set_sve_vq(Object *obj, Visitor *v, const char *name,
> >          }
> >      }
> >  
> > +    vq_state = arm_cpu_vq_map_get(cpu, vq);
> > +
> >      if (arm_sve_have_max_vq(cpu) && value && vq > cpu->sve_max_vq) {
> >          error_setg(errp, "cannot enable %s", name);
> >          error_append_hint(errp, "vq=%d (%d bits) is larger than the "
> > @@ -580,19 +622,31 @@ static void cpu_arm_set_sve_vq(Object *obj, Visitor *v, const char *name,
> >          error_append_hint(errp, "The maximum vector length must be "
> >                            "enabled, sve-max-vq=%d (%d bits)\n",
> >                            cpu->sve_max_vq, cpu->sve_max_vq * 128);
> > -    } else if (arm_sve_have_max_vq(cpu) && !value && vq < cpu->sve_max_vq &&
> > -               is_power_of_2(vq)) {
> > +    } else if (!kvm_enabled() && arm_sve_have_max_vq(cpu) && !value &&
> > +               vq < cpu->sve_max_vq && is_power_of_2(vq)) {
> >          error_setg(errp, "cannot disable %s", name);
> >          error_append_hint(errp, "vq=%d (%d bits) is required as it is a "
> >                            "power-of-2 length smaller than the maximum, "
> >                            "sve-max-vq=%d (%d bits)\n", vq, vq * 128,
> >                            cpu->sve_max_vq, cpu->sve_max_vq * 128);
> > -    } else if (!value && vq < max_vq && is_power_of_2(vq)) {
> > +    } else if (!kvm_enabled() && !value && vq < max_vq && is_power_of_2(vq)) {
> >          error_setg(errp, "cannot disable %s", name);
> >          error_append_hint(errp, "Vector length %d-bits is required as it "
> >                            "is a power-of-2 length smaller than another "
> >                            "enabled vector length. Disable all larger vector "
> >                            "lengths first.\n", vq * 128);
> > +    } else if (kvm_enabled() && value && vq_state == ARM_VQ_UNINITIALIZED) {
> > +        error_setg(errp, "cannot enable %s", name);
> > +        error_append_hint(errp, "This KVM host does not support "
> > +                          "the vector length %d-bits.\n", vq * 128);
> > +    } else if (kvm_enabled() && !value && vq < max_vq &&
> > +               (vq_state == ARM_VQ_ON ||
> > +                vq_state == ARM_VQ_UNINITIALIZED_KVM_SUPPORTED)) {
> > +        error_setg(errp, "cannot disable %s", name);
> > +        error_append_hint(errp, "Vector length %d-bits is a KVM supported "
> > +                          "length smaller than another enabled vector "
> > +                          "length. Disable all larger vector lengths "
> > +                          "first.\n", vq * 128);
> all those checks may be more readable if segment into kvm / !kvm, value
> / !value

I don't understand this suggestion. Can you provide examples of more
readable checks?

> >      } else {
> >          if (value) {
> >              bool fail = false;
> > @@ -602,31 +656,53 @@ static void cpu_arm_set_sve_vq(Object *obj, Visitor *v, const char *name,
> >               * Enabling a vector length automatically enables all
> >               * uninitialized power-of-2 lengths smaller than it, as
> >               * per the architecture.
> > +             *
> > +             * For KVM we have to automatically enable all supported,
> > +             * uninitialized lengths smaller than this length, even
> > +             * when it's not a power-of-2.
> >               */
> >              for (s = 1; s < vq; ++s) {
> > -                if (is_power_of_2(s)) {
> > -                    vq_state = arm_cpu_vq_map_get(cpu, s);
> > -                    if (vq_state == ARM_VQ_UNINITIALIZED) {
> > -                        arm_cpu_vq_map_set(cpu, s, ARM_VQ_ON);
> > -                    } else if (vq_state == ARM_VQ_OFF) {
> > -                        fail = true;
> > -                        break;
> > -                    }
> > +                vq_state = arm_cpu_vq_map_get(cpu, s);
> > +                if (!kvm_enabled() && is_power_of_2(s) &&
> > +                    vq_state == ARM_VQ_UNINITIALIZED) {
> > +                    arm_cpu_vq_map_set(cpu, s, ARM_VQ_ON);
> > +                } else if (vq_state == ARM_VQ_UNINITIALIZED_KVM_SUPPORTED) {
> > +                    assert(kvm_enabled());
> > +                    arm_cpu_vq_map_set(cpu, s, ARM_VQ_ON);
> > +                } else if ((kvm_enabled() || is_power_of_2(s)) &&
> > +                           vq_state == ARM_VQ_OFF) {
> > +                    fail = true;
> > +                    break;
> >                  }
> >              }
> >  
> > -            if (fail) {
> > +            if (!kvm_enabled() && fail) {
> >                  error_setg(errp, "cannot enable %s", name);
> >                  error_append_hint(errp, "Vector length %d-bits is disabled "
> >                                    "and is a power-of-2 length smaller than "
> >                                    "%s. All power-of-2 vector lengths smaller "
> >                                    "than the maximum length are required.\n",
> >                                    s * 128, name);
> > +
> > +            } else if (fail) {
> > +                error_setg(errp, "cannot enable %s", name);
> > +                error_append_hint(errp, "Vector length %d-bits is disabled "
> > +                                  "and the KVM host requires all supported "
> > +                                  "vector lengths smaller than %s to also be "
> > +                                  "enabled.\n", s * 128, name);
> >              } else {
> >                  arm_cpu_vq_map_set(cpu, vq, ARM_VQ_ON);
> >              }
> >          } else {
> > -            arm_cpu_vq_map_set(cpu, vq, ARM_VQ_OFF);
> > +            /*
> > +             * For KVM if the vq wasn't supported then we leave it in
> > +             * the ARM_VQ_UNINITIALIZED state in order to keep that
> > +             * unsupported information. It'll be set to OFF later when
> > +             * we finalize the map.
> > +             */
> > +            if (!kvm_enabled() || vq_state != ARM_VQ_UNINITIALIZED) {
> > +                arm_cpu_vq_map_set(cpu, vq, ARM_VQ_OFF);
> > +            }
> >          }
> >      }
> >  }
> > @@ -689,11 +765,6 @@ static void aarch64_max_initfn(Object *obj)
> >  
> >      if (kvm_enabled()) {
> >          kvm_arm_set_cpu_features_from_host(cpu);
> > -        /*
> > -         * KVM doesn't yet support the sve-max-vq property, but
> > -         * setting cpu->sve_max_vq is also used to turn SVE on.
> > -         */
> > -        cpu->sve_max_vq = ARM_SVE_INIT;
> >      } else {
> >          uint64_t t;
> >          uint32_t u;
> > @@ -774,32 +845,32 @@ static void aarch64_max_initfn(Object *obj)
> >          cpu->ctr = 0x80038003; /* 32 byte I and D cacheline size, VIPT icache */
> >          cpu->dcz_blocksize = 7; /*  512 bytes */
> >  #endif
> > -
> > -        /*
> > -         * sve_max_vq is initially unspecified, but must be initialized to a
> > -         * non-zero value (ARM_SVE_INIT) to indicate that this cpu type has
> > -         * SVE. It will be finalized in arm_cpu_realizefn().
> > -         */
> > -        cpu->sve_max_vq = ARM_SVE_INIT;
> > -        object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_max_vq,
> > -                            cpu_max_set_sve_max_vq, NULL, NULL, &error_fatal);
> > -
> > -        /*
> > -         * sve_vq_map uses a special state while setting properties, so
> > -         * we initialize it here with its init function and finalize it
> > -         * in arm_cpu_realizefn().
> > -         */
> > -        arm_cpu_vq_map_init(cpu);
> > -        for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
> > -            char name[8];
> > -            sprintf(name, "sve%d", vq * 128);
> > -            object_property_add(obj, name, "bool", cpu_arm_get_sve_vq,
> > -                                cpu_arm_set_sve_vq, NULL, NULL, &error_fatal);
> > -        }
> >      }
> >  
> >      object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
> >                          cpu_arm_set_sve, NULL, NULL, &error_fatal);
> > +
> > +    /*
> > +     * sve_max_vq is initially unspecified, but must be initialized to a
> > +     * non-zero value (ARM_SVE_INIT) to indicate that this cpu type has
> > +     * SVE. It will be finalized in arm_cpu_realizefn().
> > +     */
> > +    cpu->sve_max_vq = ARM_SVE_INIT;
> > +    object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_max_vq,
> > +                        cpu_max_set_sve_max_vq, NULL, NULL, &error_fatal);
> > +
> > +    /*
> > +     * sve_vq_map uses a special state while setting properties, so
> > +     * we initialize it here with its init function and finalize it
> > +     * in arm_cpu_realizefn().
> > +     */
> > +    arm_cpu_vq_map_init(cpu);
> > +    for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
> > +        char name[8];
> > +        sprintf(name, "sve%d", vq * 128);
> > +        object_property_add(obj, name, "bool", cpu_arm_get_sve_vq,
> > +                            cpu_arm_set_sve_vq, NULL, NULL, &error_fatal);
> > +    }
> >  }
> >  
> >  struct ARMCPUInfo {
> > diff --git a/target/arm/kvm64.c b/target/arm/kvm64.c
> > index 2821135a4d0e..5b0707e1192b 100644
> > --- a/target/arm/kvm64.c
> > +++ b/target/arm/kvm64.c
> > @@ -617,6 +617,110 @@ bool kvm_arm_sve_supported(CPUState *cpu)
> >      return ret > 0;
> >  }
> >  
> > +QEMU_BUILD_BUG_ON(KVM_ARM64_SVE_VQ_MIN != 1);
> > +
> > +void kvm_arm_sve_get_vls(CPUState *cs, unsigned long *map,
> > +                         uint32_t qemu_max_vq, uint32_t *kvm_max_vq)
> you could have kvm_max_vq as a returned value?

Actually I'll just move the warn_report() to kvm_arm_sve_get_vls() and
then completely drop the kvm_max_vq parameter/return for v3. That's a
cleanup that didn't cross my mind until after posting.

> > +{
> > +    static uint64_t vls[KVM_ARM64_SVE_VLS_WORDS];
> > +    static uint32_t host_max_vq = -1;
> > +    uint32_t vq;
> > +    int i, j;
> > +
> > +    bitmap_clear(map, 0, qemu_max_vq);
> > +    *kvm_max_vq = 0;
> > +
> > +    /*
> > +     * KVM ensures all host CPUs support the same set of vector lengths.
> > +     * So we only need to create a scratch VCPU once and then cache the
> > +     * results.
> > +     */
> > +    if (host_max_vq == -1) {
> always true?

If KVM changes, then we can change this too, but for now it's always true.

> > +        int fdarray[3], ret = -1;
> > +
> > +        if (!kvm_arm_create_scratch_host_vcpu(NULL, fdarray, NULL)) {
> > +            error_report("failed to create scratch vcpu");
> > +            abort();
> > +        }
> > +
> > +        if (ioctl(fdarray[0], KVM_CHECK_EXTENSION, KVM_CAP_ARM_SVE) > 0) {
> suggestion:
> 
> has_sve = ioctl(fdarray[0], KVM_CHECK_EXTENSION, KVM_CAP_ARM_SVE);
> kvm_arm_destroy_scratch_host_vcpu(fdarray);
> if (!has_sve) {
>     return;
> }
> 
> /* host supports SVE */
> ../..

I prefer it the way it is, because then I get to tuck these two temporary
structures below into the "supports SVE" block. If I do it the way you
suggest, then I'll have to move them up under the last 'if', which doesn't
fit as nicely. I could add the /* Host supports SVE */ comment to the
block I have though, if you'd like.

> > +            struct kvm_vcpu_init init = {
> > +                .target = -1,
> > +                .features[0] = (1 << KVM_ARM_VCPU_SVE),
> > +            };
> > +            struct kvm_one_reg reg = {
> > +                .id = KVM_REG_ARM64_SVE_VLS,
> > +                .addr = (uint64_t)&vls[0],
> > +            };
> > +
> > +            kvm_arm_destroy_scratch_host_vcpu(fdarray);
> > +
> > +            if (!kvm_arm_create_scratch_host_vcpu(NULL, fdarray, &init)) {
> > +                error_report("failed to create scratch vcpu");
> > +                abort();
> > +            }
> > +
> > +            ret = ioctl(fdarray[2], KVM_GET_ONE_REG, &reg);
> > +            if (ret) {
> > +                error_report("failed to get KVM_REG_ARM64_SVE_VLS: %s",
> > +                             strerror(errno));
> > +                abort();
> > +            }
> > +        }
> > +
> > +        kvm_arm_destroy_scratch_host_vcpu(fdarray);
> > +
> > +        if (ret) {
> > +            /* The host doesn't support SVE. */
> > +            return;
> > +        }
> > +    }
> > +
> > +    for (i = KVM_ARM64_SVE_VLS_WORDS - 1; i >= 0; --i) {
> > +        if (!vls[i]) {
> > +            continue;
> > +        }
> > +        if (host_max_vq == -1) {
> > +            host_max_vq = 64 - clz64(vls[i]) + i * 64;
> > +        }
> > +        for (j = 1; j <= 64; ++j) {
> > +            vq = j + i * 64;
> > +            if (vq > qemu_max_vq) {
> > +                break;
> > +            }
> > +            if (vls[i] & (1UL << (j - 1))) {
> > +                set_bit(vq - 1, map);
> > +            }
> > +        }
> > +    }
> > +
> > +    *kvm_max_vq = host_max_vq;
> > +}
> > +
> > +static int kvm_arm_sve_set_vls(CPUState *cs)
> > +{
> > +    uint64_t vls[KVM_ARM64_SVE_VLS_WORDS] = {0};
> > +    struct kvm_one_reg reg = {
> > +        .id = KVM_REG_ARM64_SVE_VLS,
> > +        .addr = (uint64_t)&vls[0],
> > +    };
> > +    ARMCPU *cpu = ARM_CPU(cs);
> > +    uint32_t vq;
> > +    int i, j;
> > +
> > +    assert(cpu->sve_max_vq <= KVM_ARM64_SVE_VQ_MAX);
> > +
> > +    for (vq = 1; vq <= cpu->sve_max_vq; ++vq) {
> > +        if (test_bit(vq - 1, cpu->sve_vq_map)) {
> > +            i = (vq - 1) / 64;
> > +            j = (vq - 1) % 64;
> > +            vls[i] |= 1UL << j;
> > +        }
> > +    }
> > +
> > +    return kvm_vcpu_ioctl(cs, KVM_SET_ONE_REG, &reg);
> > +}
> > +
> >  #define ARM_CPU_ID_MPIDR       3, 0, 0, 0, 5
> >  
> >  int kvm_arch_init_vcpu(CPUState *cs)
> > @@ -628,7 +732,7 @@ int kvm_arch_init_vcpu(CPUState *cs)
> >  
> >      if (cpu->kvm_target == QEMU_KVM_ARM_TARGET_NONE ||
> >          !object_dynamic_cast(OBJECT(cpu), TYPE_AARCH64_CPU)) {
> > -        fprintf(stderr, "KVM is not supported for this guest CPU type\n");
> > +        error_report("KVM is not supported for this guest CPU type");
> >          return -EINVAL;
> >      }
> >  
> > @@ -653,11 +757,8 @@ int kvm_arch_init_vcpu(CPUState *cs)
> >          unset_feature(&env->features, ARM_FEATURE_PMU);
> >      }
> >      if (cpu->sve_max_vq) {
> > -        if (!kvm_arm_sve_supported(cs)) {
> > -            cpu->sve_max_vq = 0;
> > -        } else {
> > -            cpu->kvm_init_features[0] |= 1 << KVM_ARM_VCPU_SVE;
> > -        }
> > +        assert(kvm_arm_sve_supported(cs));
> > +        cpu->kvm_init_features[0] |= 1 << KVM_ARM_VCPU_SVE;
> >      }
> >  
> >      /* Do KVM_ARM_VCPU_INIT ioctl */
> > @@ -667,6 +768,10 @@ int kvm_arch_init_vcpu(CPUState *cs)
> >      }
> >  
> >      if (cpu->sve_max_vq) {
> > +        ret = kvm_arm_sve_set_vls(cs);
> > +        if (ret) {
> > +            return ret;
> > +        }
> >          ret = kvm_arm_vcpu_finalize(cs, KVM_ARM_VCPU_SVE);
> >          if (ret) {
> >              return ret;
> > diff --git a/target/arm/kvm_arm.h b/target/arm/kvm_arm.h
> > index 2367f8ab78ed..d5eb341e906d 100644
> > --- a/target/arm/kvm_arm.h
> > +++ b/target/arm/kvm_arm.h
> > @@ -212,6 +212,22 @@ typedef struct ARMHostCPUFeatures {
> >   */
> >  bool kvm_arm_get_host_cpu_features(ARMHostCPUFeatures *ahcf);
> >  
> > +/**
> > + * kvm_arm_sve_get_vls:
> kernel-doc comment?

afaict I've formatted this comment in the same style as the other
functions in this file.

> > + * @cs: CPUState
> > + * @map: bitmap to fill in
> > + * @qemu_max_vq: the maximum vector length QEMU supports in quadwords
> > + *               (size of the bitmap to fill in)
> > + * @kvm_max_vq: the maximum vector length KVM supports in quadwords
> > + *
> > + * Get all the SVE vector lengths supported by the KVM host, setting
> > + * the bits corresponding to their length in quadwords minus one
> > + * (vq - 1) in @map up to @qemu_max_vq. Also assign @kvm_max_vq to the
> > + * maximum vector length the KVM host supports.
> > + */
> > +void kvm_arm_sve_get_vls(CPUState *cs, unsigned long *map,
> > +                         uint32_t qemu_max_vq, uint32_t *kvm_max_vq);
> > +
> >  /**
> >   * kvm_arm_set_cpu_features_from_host:
> >   * @cpu: ARMCPU to set the features for
> > @@ -314,6 +330,9 @@ static inline int kvm_arm_vgic_probe(void)
> >  static inline void kvm_arm_pmu_set_irq(CPUState *cs, int irq) {}
> >  static inline void kvm_arm_pmu_init(CPUState *cs) {}
> >  
> > +static inline void kvm_arm_sve_get_vls(CPUState *cs, unsigned long *map,
> > +                                       uint32_t qemu_max_vq,
> > +                                       uint32_t *kvm_max_vq) {}
> >  #endif
> >  
> >  static inline const char *gic_class_name(void)
> > diff --git a/target/arm/monitor.c b/target/arm/monitor.c
> > index 1e213906fd8f..284818fb4a51 100644
> > --- a/target/arm/monitor.c
> > +++ b/target/arm/monitor.c
> > @@ -100,7 +100,7 @@ QEMU_BUILD_BUG_ON(ARM_MAX_VQ > 16);
> >   *
> >   * The sve<vl-bits> features need to be in reverse order in order to
> >   * enable/disable the largest vector lengths first, ensuring all
> > - * power-of-2 vector lengths smaller can also be enabled/disabled.
> > + * smaller required vector lengths can also be enabled/disabled.
> >   */
> >  static const char *cpu_model_advertised_features[] = {
> >      "aarch64", "pmu", "sve",
> > diff --git a/tests/arm-cpu-features.c b/tests/arm-cpu-features.c
> > index 67ad5f2b78d5..349bd0dca6d1 100644
> > --- a/tests/arm-cpu-features.c
> > +++ b/tests/arm-cpu-features.c
> > @@ -324,23 +324,35 @@ static void sve_tests_sve_max_vq_8(const void *data)
> >      qtest_quit(qts);
> >  }
> >  
> > -static void sve_tests_sve_off(const void *data)
> > +static void sve_tests_off(QTestState *qts, const char *cpu_type)
> >  {
> > -    QTestState *qts;
> > -
> > -    qts = qtest_init(MACHINE "-cpu max,sve=off");
> > -
> >      /*
> >       * SVE is off, so the map should be empty.
> >       */
> > -    assert_sve_vls(qts, "max", 0, NULL);
> > +    assert_sve_vls(qts, cpu_type, 0, NULL);
> >  
> >      /*
> >       * We can't turn anything on, but off is OK.
> >       */
> > -    assert_error(qts, "max", "cannot enable sve128", "{ 'sve128': true }");
> > -    assert_sve_vls(qts, "max", 0, "{ 'sve128': false }");
> > +    assert_error(qts, cpu_type, "cannot enable sve128", "{ 'sve128': true }");
> > +    assert_sve_vls(qts, cpu_type, 0, "{ 'sve128': false }");
> > +}
> >  
> > +static void sve_tests_sve_off(const void *data)
> > +{
> > +    QTestState *qts;
> > +
> > +    qts = qtest_init(MACHINE "-cpu max,sve=off");
> > +    sve_tests_off(qts, "max");
> > +    qtest_quit(qts);
> > +}
> > +
> > +static void sve_tests_sve_off_kvm(const void *data)
> > +{
> > +    QTestState *qts;
> > +
> > +    qts = qtest_init(MACHINE "-accel kvm -cpu max,sve=off");
> > +    sve_tests_off(qts, "max");
> >      qtest_quit(qts);
> >  }
> >  
> > @@ -392,12 +404,66 @@ static void test_query_cpu_model_expansion_kvm(const void *data)
> >      assert_has_feature(qts, "host", "pmu");
> >  
> >      if (g_str_equal(qtest_get_arch(), "aarch64")) {
> > +        bool kvm_supports_sve;
> > +        uint32_t max_vq, vq;
> > +        uint64_t vls;
> > +        char name[8];
> > +        QDict *resp;
> > +        char *error;
> > +
> >          assert_has_feature(qts, "host", "aarch64");
> > -        assert_has_feature(qts, "max", "sve");
> >  
> >          assert_error(qts, "cortex-a15",
> >              "The CPU definition 'cortex-a15' cannot "
> >              "be used with KVM on this host", NULL);
> > +
> > +        assert_has_feature(qts, "max", "sve");
> > +        resp = do_query_no_props(qts, "max");
> > +        g_assert(resp);
> > +        kvm_supports_sve = qdict_get_bool(resp_get_props(resp), "sve");
> > +        qobject_unref(resp);
> > +
> > +        if (kvm_supports_sve) {
> > +            resp = do_query_no_props(qts, "max");
> > +            resp_get_sve_vls(resp, &vls, &max_vq);
> > +            g_assert(max_vq != 0);
> > +            qobject_unref(resp);
> > +
> > +            /* Enabling a supported length is of course fine. */
> > +            sprintf(name, "sve%d", max_vq * 128);
> > +            assert_sve_vls(qts, "max", vls, "{ %s: true }", name);
> > +
> > +            /* Also disabling the largest lengths is fine. */
> > +            assert_sve_vls(qts, "max", (vls & ~BIT(max_vq - 1)),
> > +                           "{ %s: false }", name);
> > +
> > +            for (vq = 1; vq <= max_vq; ++vq) {
> > +                if (!(vls & BIT(vq - 1))) {
> > +                    /* vq is unsupported */
> > +                    break;
> > +                }
> > +            }
> > +            if (vq <= SVE_MAX_VQ) {
> > +                sprintf(name, "sve%d", vq * 128);
> > +                error = g_strdup_printf("cannot enable %s", name);
> > +                assert_error(qts, "max", error, "{ %s: true }", name);
> > +                g_free(error);
> > +            }
> > +
> > +            if (max_vq > 1) {
> > +                /* The next smaller, supported vq is required */
> > +                vq = 64 - __builtin_clzll(vls & ~BIT(max_vq - 1));
> > +                sprintf(name, "sve%d", vq * 128);
> > +                error = g_strdup_printf("cannot disable %s", name);
> > +                assert_error(qts, "max", error, "{ %s: false }", name);
> > +                g_free(error);
> > +            }
> > +        } else {
> > +            resp = do_query_no_props(qts, "max");
> > +            resp_get_sve_vls(resp, &vls, &max_vq);
> > +            g_assert(max_vq == 0);
> > +            qobject_unref(resp);
> > +        }
> >      } else {
> >          assert_error(qts, "host",
> >                       "'pmu' feature not supported by KVM on this host",
> > @@ -434,6 +500,8 @@ int main(int argc, char **argv)
> >      if (kvm_available) {
> >          qtest_add_data_func("/arm/kvm/query-cpu-model-expansion",
> >                              NULL, test_query_cpu_model_expansion_kvm);
> > +        qtest_add_data_func("/arm/kvm/query-cpu-model-expansion/sve-off",
> > +                            NULL, sve_tests_sve_off_kvm);
> >      }
> >  
> >      return g_test_run();
> > 
> 
> Thanks
> 
> Eric
> 

[Qemu-devel] [PATCH v2 14/14] target/arm/kvm: host cpu: Add support for sve<vl-bits> properties

[Andrew Jones]

Allow cpu 'host' to enable SVE when it's available, unless the
user chooses to disable it with the added 'sve=off' cpu property.
Also give the user the ability to select vector lengths with the
sve<vl-bits> properties. We don't adopt 'max' cpu's other sve
property, sve-max-vq, because that property is difficult to
use with KVM. That property assumes all vector lengths in the
range from 1 up to and including the specified maximum length are
supported, but there may be optional lengths not supported by the
host in that range. With KVM one must be more specific when
enabling vector lengths.

Signed-off-by: Andrew Jones <drjones@redhat.com>
---
 target/arm/cpu.c         |  1 +
 target/arm/cpu.h         |  2 ++
 target/arm/cpu64.c       | 47 ++++++++++++++++++++++++++--------------
 tests/arm-cpu-features.c | 21 +++++++++---------
 4 files changed, 45 insertions(+), 26 deletions(-)

diff --git a/target/arm/cpu.c b/target/arm/cpu.c
index e060a0d9df0e..9d05291cb5f6 100644
--- a/target/arm/cpu.c
+++ b/target/arm/cpu.c
@@ -2407,6 +2407,7 @@ static void arm_host_initfn(Object *obj)
     ARMCPU *cpu = ARM_CPU(obj);
 
     kvm_arm_set_cpu_features_from_host(cpu);
+    aarch64_add_sve_properties(obj);
     arm_cpu_post_init(obj);
 }
 
diff --git a/target/arm/cpu.h b/target/arm/cpu.h
index 8a1c6c66a462..52a6b219b74a 100644
--- a/target/arm/cpu.h
+++ b/target/arm/cpu.h
@@ -974,11 +974,13 @@ int aarch64_cpu_gdb_write_register(CPUState *cpu, uint8_t *buf, int reg);
 void aarch64_sve_narrow_vq(CPUARMState *env, unsigned vq);
 void aarch64_sve_change_el(CPUARMState *env, int old_el,
                            int new_el, bool el0_a64);
+void aarch64_add_sve_properties(Object *obj);
 #else
 static inline void aarch64_sve_narrow_vq(CPUARMState *env, unsigned vq) { }
 static inline void aarch64_sve_change_el(CPUARMState *env, int o,
                                          int n, bool a)
 { }
+static inline void aarch64_add_sve_properties(Object *obj) { }
 #endif
 
 target_ulong do_arm_semihosting(CPUARMState *env);
diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
index 6e92aa54b9c8..89396a7729ec 100644
--- a/target/arm/cpu64.c
+++ b/target/arm/cpu64.c
@@ -753,6 +753,36 @@ static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
     }
 }
 
+void aarch64_add_sve_properties(Object *obj)
+{
+    ARMCPU *cpu = ARM_CPU(obj);
+    uint32_t vq;
+
+    object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
+                        cpu_arm_set_sve, NULL, NULL, &error_fatal);
+
+    /*
+     * sve_max_vq is initially unspecified, but must be initialized to a
+     * non-zero value (ARM_SVE_INIT) to indicate that this cpu type has
+     * SVE. It will be finalized in arm_cpu_realizefn().
+     */
+    assert(!cpu->sve_max_vq || cpu->sve_max_vq == ARM_SVE_INIT);
+    cpu->sve_max_vq = ARM_SVE_INIT;
+
+    /*
+     * sve_vq_map uses a special state while setting properties, so
+     * we initialize it here with its init function and finalize it
+     * in arm_cpu_realizefn().
+     */
+    arm_cpu_vq_map_init(cpu);
+    for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
+        char name[8];
+        sprintf(name, "sve%d", vq * 128);
+        object_property_add(obj, name, "bool", cpu_arm_get_sve_vq,
+                            cpu_arm_set_sve_vq, NULL, NULL, &error_fatal);
+    }
+}
+
 /* -cpu max: if KVM is enabled, like -cpu host (best possible with this host);
  * otherwise, a CPU with as many features enabled as our emulation supports.
  * The version of '-cpu max' for qemu-system-arm is defined in cpu.c;
@@ -761,7 +791,6 @@ static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
 static void aarch64_max_initfn(Object *obj)
 {
     ARMCPU *cpu = ARM_CPU(obj);
-    uint32_t vq;
 
     if (kvm_enabled()) {
         kvm_arm_set_cpu_features_from_host(cpu);
@@ -847,9 +876,6 @@ static void aarch64_max_initfn(Object *obj)
 #endif
     }
 
-    object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
-                        cpu_arm_set_sve, NULL, NULL, &error_fatal);
-
     /*
      * sve_max_vq is initially unspecified, but must be initialized to a
      * non-zero value (ARM_SVE_INIT) to indicate that this cpu type has
@@ -859,18 +885,7 @@ static void aarch64_max_initfn(Object *obj)
     object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_max_vq,
                         cpu_max_set_sve_max_vq, NULL, NULL, &error_fatal);
 
-    /*
-     * sve_vq_map uses a special state while setting properties, so
-     * we initialize it here with its init function and finalize it
-     * in arm_cpu_realizefn().
-     */
-    arm_cpu_vq_map_init(cpu);
-    for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
-        char name[8];
-        sprintf(name, "sve%d", vq * 128);
-        object_property_add(obj, name, "bool", cpu_arm_get_sve_vq,
-                            cpu_arm_set_sve_vq, NULL, NULL, &error_fatal);
-    }
+    aarch64_add_sve_properties(obj);
 }
 
 struct ARMCPUInfo {
diff --git a/tests/arm-cpu-features.c b/tests/arm-cpu-features.c
index 349bd0dca6d1..dfe83f104b27 100644
--- a/tests/arm-cpu-features.c
+++ b/tests/arm-cpu-features.c
@@ -351,8 +351,8 @@ static void sve_tests_sve_off_kvm(const void *data)
 {
     QTestState *qts;
 
-    qts = qtest_init(MACHINE "-accel kvm -cpu max,sve=off");
-    sve_tests_off(qts, "max");
+    qts = qtest_init(MACHINE "-accel kvm -cpu host,sve=off");
+    sve_tests_off(qts, "host");
     qtest_quit(qts);
 }
 
@@ -417,24 +417,24 @@ static void test_query_cpu_model_expansion_kvm(const void *data)
             "The CPU definition 'cortex-a15' cannot "
             "be used with KVM on this host", NULL);
 
-        assert_has_feature(qts, "max", "sve");
-        resp = do_query_no_props(qts, "max");
+        assert_has_feature(qts, "host", "sve");
+        resp = do_query_no_props(qts, "host");
         g_assert(resp);
         kvm_supports_sve = qdict_get_bool(resp_get_props(resp), "sve");
         qobject_unref(resp);
 
         if (kvm_supports_sve) {
-            resp = do_query_no_props(qts, "max");
+            resp = do_query_no_props(qts, "host");
             resp_get_sve_vls(resp, &vls, &max_vq);
             g_assert(max_vq != 0);
             qobject_unref(resp);
 
             /* Enabling a supported length is of course fine. */
             sprintf(name, "sve%d", max_vq * 128);
-            assert_sve_vls(qts, "max", vls, "{ %s: true }", name);
+            assert_sve_vls(qts, "host", vls, "{ %s: true }", name);
 
             /* Also disabling the largest lengths is fine. */
-            assert_sve_vls(qts, "max", (vls & ~BIT(max_vq - 1)),
+            assert_sve_vls(qts, "host", (vls & ~BIT(max_vq - 1)),
                            "{ %s: false }", name);
 
             for (vq = 1; vq <= max_vq; ++vq) {
@@ -446,7 +446,7 @@ static void test_query_cpu_model_expansion_kvm(const void *data)
             if (vq <= SVE_MAX_VQ) {
                 sprintf(name, "sve%d", vq * 128);
                 error = g_strdup_printf("cannot enable %s", name);
-                assert_error(qts, "max", error, "{ %s: true }", name);
+                assert_error(qts, "host", error, "{ %s: true }", name);
                 g_free(error);
             }
 
@@ -455,16 +455,17 @@ static void test_query_cpu_model_expansion_kvm(const void *data)
                 vq = 64 - __builtin_clzll(vls & ~BIT(max_vq - 1));
                 sprintf(name, "sve%d", vq * 128);
                 error = g_strdup_printf("cannot disable %s", name);
-                assert_error(qts, "max", error, "{ %s: false }", name);
+                assert_error(qts, "host", error, "{ %s: false }", name);
                 g_free(error);
             }
         } else {
-            resp = do_query_no_props(qts, "max");
+            resp = do_query_no_props(qts, "host");
             resp_get_sve_vls(resp, &vls, &max_vq);
             g_assert(max_vq == 0);
             qobject_unref(resp);
         }
     } else {
+        assert_has_not_feature(qts, "host", "sve");
         assert_error(qts, "host",
                      "'pmu' feature not supported by KVM on this host",
                      "{ 'pmu': true }");
-- 
2.20.1

Re: [Qemu-devel] [PATCH v2 14/14] target/arm/kvm: host cpu: Add support for sve<vl-bits> properties

[Auger Eric]

Hi Drew,
On 6/21/19 6:34 PM, Andrew Jones wrote:
> Allow cpu 'host' to enable SVE when it's available, unless the
> user chooses to disable it with the added 'sve=off' cpu property.
> Also give the user the ability to select vector lengths with the
> sve<vl-bits> properties. We don't adopt 'max' cpu's other sve
> property, sve-max-vq, because that property is difficult to
> use with KVM. That property assumes all vector lengths in the
> range from 1 up to and including the specified maximum length are
> supported, but there may be optional lengths not supported by the
> host in that range. With KVM one must be more specific when
> enabling vector lengths.
> 
> Signed-off-by: Andrew Jones <drjones@redhat.com>
> ---
>  target/arm/cpu.c         |  1 +
>  target/arm/cpu.h         |  2 ++
>  target/arm/cpu64.c       | 47 ++++++++++++++++++++++++++--------------
>  tests/arm-cpu-features.c | 21 +++++++++---------
>  4 files changed, 45 insertions(+), 26 deletions(-)
> 
> diff --git a/target/arm/cpu.c b/target/arm/cpu.c
> index e060a0d9df0e..9d05291cb5f6 100644
> --- a/target/arm/cpu.c
> +++ b/target/arm/cpu.c
> @@ -2407,6 +2407,7 @@ static void arm_host_initfn(Object *obj)
>      ARMCPU *cpu = ARM_CPU(obj);
>  
>      kvm_arm_set_cpu_features_from_host(cpu);
> +    aarch64_add_sve_properties(obj);
>      arm_cpu_post_init(obj);
>  }
>  
> diff --git a/target/arm/cpu.h b/target/arm/cpu.h
> index 8a1c6c66a462..52a6b219b74a 100644
> --- a/target/arm/cpu.h
> +++ b/target/arm/cpu.h
> @@ -974,11 +974,13 @@ int aarch64_cpu_gdb_write_register(CPUState *cpu, uint8_t *buf, int reg);
>  void aarch64_sve_narrow_vq(CPUARMState *env, unsigned vq);
>  void aarch64_sve_change_el(CPUARMState *env, int old_el,
>                             int new_el, bool el0_a64);
> +void aarch64_add_sve_properties(Object *obj);
>  #else
>  static inline void aarch64_sve_narrow_vq(CPUARMState *env, unsigned vq) { }
>  static inline void aarch64_sve_change_el(CPUARMState *env, int o,
>                                           int n, bool a)
>  { }
> +static inline void aarch64_add_sve_properties(Object *obj) { }
>  #endif
>  
>  target_ulong do_arm_semihosting(CPUARMState *env);
> diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
> index 6e92aa54b9c8..89396a7729ec 100644
> --- a/target/arm/cpu64.c
> +++ b/target/arm/cpu64.c
> @@ -753,6 +753,36 @@ static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
>      }
>  }
>  
> +void aarch64_add_sve_properties(Object *obj)
> +{
> +    ARMCPU *cpu = ARM_CPU(obj);
> +    uint32_t vq;
> +
> +    object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
> +                        cpu_arm_set_sve, NULL, NULL, &error_fatal);
> +
> +    /*
> +     * sve_max_vq is initially unspecified, but must be initialized to a
> +     * non-zero value (ARM_SVE_INIT) to indicate that this cpu type has
> +     * SVE. It will be finalized in arm_cpu_realizefn().
> +     */
this comment is duplicated in aarch64_max_initfn(). Also sve_max_vq may
be already initialized to SVE_INIT so what do you mean by unspecified?
> +    assert(!cpu->sve_max_vq || cpu->sve_max_vq == ARM_SVE_INIT);> +    cpu->sve_max_vq = ARM_SVE_INIT;
maybe you could move this assignment in arm_cpu_vq_map_init().

> +
> +    /*
> +     * sve_vq_map uses a special state while setting properties, so
> +     * we initialize it here with its init function and finalize it
> +     * in arm_cpu_realizefn().
> +     */
> +    arm_cpu_vq_map_init(cpu);
> +    for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
> +        char name[8];
> +        sprintf(name, "sve%d", vq * 128);
> +        object_property_add(obj, name, "bool", cpu_arm_get_sve_vq,
> +                            cpu_arm_set_sve_vq, NULL, NULL, &error_fatal);> +    }
> +}
> +
>  /* -cpu max: if KVM is enabled, like -cpu host (best possible with this host);
>   * otherwise, a CPU with as many features enabled as our emulation supports.
>   * The version of '-cpu max' for qemu-system-arm is defined in cpu.c;
> @@ -761,7 +791,6 @@ static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
>  static void aarch64_max_initfn(Object *obj)
>  {
>      ARMCPU *cpu = ARM_CPU(obj);
> -    uint32_t vq;
>  
>      if (kvm_enabled()) {
>          kvm_arm_set_cpu_features_from_host(cpu);
> @@ -847,9 +876,6 @@ static void aarch64_max_initfn(Object *obj)
>  #endif
>      }
>  
> -    object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
> -                        cpu_arm_set_sve, NULL, NULL, &error_fatal);
> -
>      /*
>       * sve_max_vq is initially unspecified, but must be initialized to a
>       * non-zero value (ARM_SVE_INIT) to indicate that this cpu type has
> @@ -859,18 +885,7 @@ static void aarch64_max_initfn(Object *obj)
>      object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_max_vq,
>                          cpu_max_set_sve_max_vq, NULL, NULL, &error_fatal);
>  
> -    /*
> -     * sve_vq_map uses a special state while setting properties, so
> -     * we initialize it here with its init function and finalize it
> -     * in arm_cpu_realizefn().
> -     */
> -    arm_cpu_vq_map_init(cpu);
> -    for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
> -        char name[8];
> -        sprintf(name, "sve%d", vq * 128);
> -        object_property_add(obj, name, "bool", cpu_arm_get_sve_vq,
> -                            cpu_arm_set_sve_vq, NULL, NULL, &error_fatal);
> -    }
> +    aarch64_add_sve_properties(obj);
>  }
>  
>  struct ARMCPUInfo {
> diff --git a/tests/arm-cpu-features.c b/tests/arm-cpu-features.c
> index 349bd0dca6d1..dfe83f104b27 100644
> --- a/tests/arm-cpu-features.c
> +++ b/tests/arm-cpu-features.c
> @@ -351,8 +351,8 @@ static void sve_tests_sve_off_kvm(const void *data)
>  {
>      QTestState *qts;
>  
> -    qts = qtest_init(MACHINE "-accel kvm -cpu max,sve=off");
> -    sve_tests_off(qts, "max");
> +    qts = qtest_init(MACHINE "-accel kvm -cpu host,sve=off");
> +    sve_tests_off(qts, "host");
>      qtest_quit(qts);
>  }
>  
> @@ -417,24 +417,24 @@ static void test_query_cpu_model_expansion_kvm(const void *data)
>              "The CPU definition 'cortex-a15' cannot "
>              "be used with KVM on this host", NULL);
>  
> -        assert_has_feature(qts, "max", "sve");
> -        resp = do_query_no_props(qts, "max");
> +        assert_has_feature(qts, "host", "sve");
> +        resp = do_query_no_props(qts, "host");
>          g_assert(resp);
>          kvm_supports_sve = qdict_get_bool(resp_get_props(resp), "sve");
>          qobject_unref(resp);
>  
>          if (kvm_supports_sve) {
> -            resp = do_query_no_props(qts, "max");
> +            resp = do_query_no_props(qts, "host");
>              resp_get_sve_vls(resp, &vls, &max_vq);
>              g_assert(max_vq != 0);
>              qobject_unref(resp);
>  
>              /* Enabling a supported length is of course fine. */
>              sprintf(name, "sve%d", max_vq * 128);
> -            assert_sve_vls(qts, "max", vls, "{ %s: true }", name);
> +            assert_sve_vls(qts, "host", vls, "{ %s: true }", name);
>  
>              /* Also disabling the largest lengths is fine. */
> -            assert_sve_vls(qts, "max", (vls & ~BIT(max_vq - 1)),
> +            assert_sve_vls(qts, "host", (vls & ~BIT(max_vq - 1)),
>                             "{ %s: false }", name);
>  
>              for (vq = 1; vq <= max_vq; ++vq) {
> @@ -446,7 +446,7 @@ static void test_query_cpu_model_expansion_kvm(const void *data)
>              if (vq <= SVE_MAX_VQ) {
>                  sprintf(name, "sve%d", vq * 128);
>                  error = g_strdup_printf("cannot enable %s", name);
> -                assert_error(qts, "max", error, "{ %s: true }", name);
> +                assert_error(qts, "host", error, "{ %s: true }", name);
>                  g_free(error);
>              }
>  
> @@ -455,16 +455,17 @@ static void test_query_cpu_model_expansion_kvm(const void *data)
>                  vq = 64 - __builtin_clzll(vls & ~BIT(max_vq - 1));
>                  sprintf(name, "sve%d", vq * 128);
>                  error = g_strdup_printf("cannot disable %s", name);
> -                assert_error(qts, "max", error, "{ %s: false }", name);
> +                assert_error(qts, "host", error, "{ %s: false }", name);
>                  g_free(error);
>              }
>          } else {
> -            resp = do_query_no_props(qts, "max");
> +            resp = do_query_no_props(qts, "host");
>              resp_get_sve_vls(resp, &vls, &max_vq);
>              g_assert(max_vq == 0);
>              qobject_unref(resp);
>          }
>      } else {
> +        assert_has_not_feature(qts, "host", "sve");
>          assert_error(qts, "host",
>                       "'pmu' feature not supported by KVM on this host",
>                       "{ 'pmu': true }");
> 
Thanks

Eric

Re: [Qemu-devel] [PATCH v2 14/14] target/arm/kvm: host cpu: Add support for sve<vl-bits> properties

[Andrew Jones]

On Thu, Jun 27, 2019 at 07:15:13PM +0200, Auger Eric wrote:
> Hi Drew,
> On 6/21/19 6:34 PM, Andrew Jones wrote:
> > Allow cpu 'host' to enable SVE when it's available, unless the
> > user chooses to disable it with the added 'sve=off' cpu property.
> > Also give the user the ability to select vector lengths with the
> > sve<vl-bits> properties. We don't adopt 'max' cpu's other sve
> > property, sve-max-vq, because that property is difficult to
> > use with KVM. That property assumes all vector lengths in the
> > range from 1 up to and including the specified maximum length are
> > supported, but there may be optional lengths not supported by the
> > host in that range. With KVM one must be more specific when
> > enabling vector lengths.
> > 
> > Signed-off-by: Andrew Jones <drjones@redhat.com>
> > ---
> >  target/arm/cpu.c         |  1 +
> >  target/arm/cpu.h         |  2 ++
> >  target/arm/cpu64.c       | 47 ++++++++++++++++++++++++++--------------
> >  tests/arm-cpu-features.c | 21 +++++++++---------
> >  4 files changed, 45 insertions(+), 26 deletions(-)
> > 
> > diff --git a/target/arm/cpu.c b/target/arm/cpu.c
> > index e060a0d9df0e..9d05291cb5f6 100644
> > --- a/target/arm/cpu.c
> > +++ b/target/arm/cpu.c
> > @@ -2407,6 +2407,7 @@ static void arm_host_initfn(Object *obj)
> >      ARMCPU *cpu = ARM_CPU(obj);
> >  
> >      kvm_arm_set_cpu_features_from_host(cpu);
> > +    aarch64_add_sve_properties(obj);
> >      arm_cpu_post_init(obj);
> >  }
> >  
> > diff --git a/target/arm/cpu.h b/target/arm/cpu.h
> > index 8a1c6c66a462..52a6b219b74a 100644
> > --- a/target/arm/cpu.h
> > +++ b/target/arm/cpu.h
> > @@ -974,11 +974,13 @@ int aarch64_cpu_gdb_write_register(CPUState *cpu, uint8_t *buf, int reg);
> >  void aarch64_sve_narrow_vq(CPUARMState *env, unsigned vq);
> >  void aarch64_sve_change_el(CPUARMState *env, int old_el,
> >                             int new_el, bool el0_a64);
> > +void aarch64_add_sve_properties(Object *obj);
> >  #else
> >  static inline void aarch64_sve_narrow_vq(CPUARMState *env, unsigned vq) { }
> >  static inline void aarch64_sve_change_el(CPUARMState *env, int o,
> >                                           int n, bool a)
> >  { }
> > +static inline void aarch64_add_sve_properties(Object *obj) { }
> >  #endif
> >  
> >  target_ulong do_arm_semihosting(CPUARMState *env);
> > diff --git a/target/arm/cpu64.c b/target/arm/cpu64.c
> > index 6e92aa54b9c8..89396a7729ec 100644
> > --- a/target/arm/cpu64.c
> > +++ b/target/arm/cpu64.c
> > @@ -753,6 +753,36 @@ static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
> >      }
> >  }
> >  
> > +void aarch64_add_sve_properties(Object *obj)
> > +{
> > +    ARMCPU *cpu = ARM_CPU(obj);
> > +    uint32_t vq;
> > +
> > +    object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
> > +                        cpu_arm_set_sve, NULL, NULL, &error_fatal);
> > +
> > +    /*
> > +     * sve_max_vq is initially unspecified, but must be initialized to a
> > +     * non-zero value (ARM_SVE_INIT) to indicate that this cpu type has
> > +     * SVE. It will be finalized in arm_cpu_realizefn().
> > +     */
> this comment is duplicated in aarch64_max_initfn().
> Also sve_max_vq may
> be already initialized to SVE_INIT so what do you mean by unspecified?

Unspecified means it's not set to a valid maximum VQ.

> > +    assert(!cpu->sve_max_vq || cpu->sve_max_vq == ARM_SVE_INIT);> +    cpu->sve_max_vq = ARM_SVE_INIT;
> maybe you could move this assignment in arm_cpu_vq_map_init().

I guess so. And then I guess I could just drop the comment, as
the new location would make it clear what we're doing.

> 
> > +
> > +    /*
> > +     * sve_vq_map uses a special state while setting properties, so
> > +     * we initialize it here with its init function and finalize it
> > +     * in arm_cpu_realizefn().
> > +     */
> > +    arm_cpu_vq_map_init(cpu);
> > +    for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
> > +        char name[8];
> > +        sprintf(name, "sve%d", vq * 128);
> > +        object_property_add(obj, name, "bool", cpu_arm_get_sve_vq,
> > +                            cpu_arm_set_sve_vq, NULL, NULL, &error_fatal);> +    }
> > +}
> > +
> >  /* -cpu max: if KVM is enabled, like -cpu host (best possible with this host);
> >   * otherwise, a CPU with as many features enabled as our emulation supports.
> >   * The version of '-cpu max' for qemu-system-arm is defined in cpu.c;
> > @@ -761,7 +791,6 @@ static void cpu_arm_set_sve(Object *obj, Visitor *v, const char *name,
> >  static void aarch64_max_initfn(Object *obj)
> >  {
> >      ARMCPU *cpu = ARM_CPU(obj);
> > -    uint32_t vq;
> >  
> >      if (kvm_enabled()) {
> >          kvm_arm_set_cpu_features_from_host(cpu);
> > @@ -847,9 +876,6 @@ static void aarch64_max_initfn(Object *obj)
> >  #endif
> >      }
> >  
> > -    object_property_add(obj, "sve", "bool", cpu_arm_get_sve,
> > -                        cpu_arm_set_sve, NULL, NULL, &error_fatal);
> > -
> >      /*
> >       * sve_max_vq is initially unspecified, but must be initialized to a
> >       * non-zero value (ARM_SVE_INIT) to indicate that this cpu type has
> > @@ -859,18 +885,7 @@ static void aarch64_max_initfn(Object *obj)
> >      object_property_add(obj, "sve-max-vq", "uint32", cpu_max_get_sve_max_vq,
> >                          cpu_max_set_sve_max_vq, NULL, NULL, &error_fatal);
> >  
> > -    /*
> > -     * sve_vq_map uses a special state while setting properties, so
> > -     * we initialize it here with its init function and finalize it
> > -     * in arm_cpu_realizefn().
> > -     */
> > -    arm_cpu_vq_map_init(cpu);
> > -    for (vq = 1; vq <= ARM_MAX_VQ; ++vq) {
> > -        char name[8];
> > -        sprintf(name, "sve%d", vq * 128);
> > -        object_property_add(obj, name, "bool", cpu_arm_get_sve_vq,
> > -                            cpu_arm_set_sve_vq, NULL, NULL, &error_fatal);
> > -    }
> > +    aarch64_add_sve_properties(obj);
> >  }
> >  
> >  struct ARMCPUInfo {
> > diff --git a/tests/arm-cpu-features.c b/tests/arm-cpu-features.c
> > index 349bd0dca6d1..dfe83f104b27 100644
> > --- a/tests/arm-cpu-features.c
> > +++ b/tests/arm-cpu-features.c
> > @@ -351,8 +351,8 @@ static void sve_tests_sve_off_kvm(const void *data)
> >  {
> >      QTestState *qts;
> >  
> > -    qts = qtest_init(MACHINE "-accel kvm -cpu max,sve=off");
> > -    sve_tests_off(qts, "max");
> > +    qts = qtest_init(MACHINE "-accel kvm -cpu host,sve=off");
> > +    sve_tests_off(qts, "host");
> >      qtest_quit(qts);
> >  }
> >  
> > @@ -417,24 +417,24 @@ static void test_query_cpu_model_expansion_kvm(const void *data)
> >              "The CPU definition 'cortex-a15' cannot "
> >              "be used with KVM on this host", NULL);
> >  
> > -        assert_has_feature(qts, "max", "sve");
> > -        resp = do_query_no_props(qts, "max");
> > +        assert_has_feature(qts, "host", "sve");
> > +        resp = do_query_no_props(qts, "host");
> >          g_assert(resp);
> >          kvm_supports_sve = qdict_get_bool(resp_get_props(resp), "sve");
> >          qobject_unref(resp);
> >  
> >          if (kvm_supports_sve) {
> > -            resp = do_query_no_props(qts, "max");
> > +            resp = do_query_no_props(qts, "host");
> >              resp_get_sve_vls(resp, &vls, &max_vq);
> >              g_assert(max_vq != 0);
> >              qobject_unref(resp);
> >  
> >              /* Enabling a supported length is of course fine. */
> >              sprintf(name, "sve%d", max_vq * 128);
> > -            assert_sve_vls(qts, "max", vls, "{ %s: true }", name);
> > +            assert_sve_vls(qts, "host", vls, "{ %s: true }", name);
> >  
> >              /* Also disabling the largest lengths is fine. */
> > -            assert_sve_vls(qts, "max", (vls & ~BIT(max_vq - 1)),
> > +            assert_sve_vls(qts, "host", (vls & ~BIT(max_vq - 1)),
> >                             "{ %s: false }", name);
> >  
> >              for (vq = 1; vq <= max_vq; ++vq) {
> > @@ -446,7 +446,7 @@ static void test_query_cpu_model_expansion_kvm(const void *data)
> >              if (vq <= SVE_MAX_VQ) {
> >                  sprintf(name, "sve%d", vq * 128);
> >                  error = g_strdup_printf("cannot enable %s", name);
> > -                assert_error(qts, "max", error, "{ %s: true }", name);
> > +                assert_error(qts, "host", error, "{ %s: true }", name);
> >                  g_free(error);
> >              }
> >  
> > @@ -455,16 +455,17 @@ static void test_query_cpu_model_expansion_kvm(const void *data)
> >                  vq = 64 - __builtin_clzll(vls & ~BIT(max_vq - 1));
> >                  sprintf(name, "sve%d", vq * 128);
> >                  error = g_strdup_printf("cannot disable %s", name);
> > -                assert_error(qts, "max", error, "{ %s: false }", name);
> > +                assert_error(qts, "host", error, "{ %s: false }", name);
> >                  g_free(error);
> >              }
> >          } else {
> > -            resp = do_query_no_props(qts, "max");
> > +            resp = do_query_no_props(qts, "host");
> >              resp_get_sve_vls(resp, &vls, &max_vq);
> >              g_assert(max_vq == 0);
> >              qobject_unref(resp);
> >          }
> >      } else {
> > +        assert_has_not_feature(qts, "host", "sve");
> >          assert_error(qts, "host",
> >                       "'pmu' feature not supported by KVM on this host",
> >                       "{ 'pmu': true }");
> > 
> Thanks
> 
> Eric
>