About the kvm-no-adjvtime CPU property

About the kvm-no-adjvtime CPU property

[Ying Fang]

About the kvm-no-adjvtime CPU property

Hi Andrew,
To adjust virutal time, a new kvm cpu property kvm-no-adjvtime
was introduced to 5.0 virt machine types. However the cpu
property was enabled only for host-passthrough and max cpu model.
As for other cpu model like cortex-a57, cortex-a53, cortex-a72,
this kvm-adjvtime is not enabled by default, which means the
virutal time can not be adjust for them.

Here, for example, if VM is configured with kvm enabled:

   <cpu mode='custom' match='exact' check='partial'>
     <model fallback='allow'>cortex-a72</model>
     <topology sockets='2' dies='1' cores='2' threads='1'/>
     <numa>
       <cell id='0' cpus='0-1' memory='16777216' unit='KiB'/>
       <cell id='1' cpus='2-3' memory='16777216' unit='KiB'/>
     </numa>
   </cpu>

We cannot adjust virtual time even if 5.0 virt machine is used.
So i'd like to add it to other cpu model, do you have any
suggestions here ?

Re: About the kvm-no-adjvtime CPU property

[Andrew Jones]

On Sat, May 30, 2020 at 04:56:26PM +0800, Ying Fang wrote:
> About the kvm-no-adjvtime CPU property
> 
> Hi Andrew,
> To adjust virutal time, a new kvm cpu property kvm-no-adjvtime
> was introduced to 5.0 virt machine types. However the cpu
> property was enabled only for host-passthrough and max cpu model.
> As for other cpu model like cortex-a57, cortex-a53, cortex-a72,
> this kvm-adjvtime is not enabled by default, which means the
> virutal time can not be adjust for them.
> 
> Here, for example, if VM is configured with kvm enabled:
> 
>   <cpu mode='custom' match='exact' check='partial'>
>     <model fallback='allow'>cortex-a72</model>
>     <topology sockets='2' dies='1' cores='2' threads='1'/>
>     <numa>
>       <cell id='0' cpus='0-1' memory='16777216' unit='KiB'/>
>       <cell id='1' cpus='2-3' memory='16777216' unit='KiB'/>
>     </numa>
>   </cpu>
> 
> We cannot adjust virtual time even if 5.0 virt machine is used.
> So i'd like to add it to other cpu model, do you have any
> suggestions here ?
> 
>

Hi Fang,

The cpu feature only requires kvm.  If a cpu model may be used with kvm,
then the feature can be allowed to be used with the model.  What I find
interesting is that the cpu model is being used with kvm instead of 'host'
or 'max'.  Can you explain the reasons for that?  Currently, when using
kvm, the guest will always effectively get 'host' anyway, even when a
model is provided. Indeed, for a model to work, kvm requires it to exactly
match the host cpu and even then the ID registers are passed through from
the host CPU, not taken from QEMU's model.

Thanks,
drew

Re: About the kvm-no-adjvtime CPU property

[Ying Fang]

On 2020/6/1 16:07, Andrew Jones wrote:
> On Sat, May 30, 2020 at 04:56:26PM +0800, Ying Fang wrote:
>> About the kvm-no-adjvtime CPU property
>>
>> Hi Andrew,
>> To adjust virutal time, a new kvm cpu property kvm-no-adjvtime
>> was introduced to 5.0 virt machine types. However the cpu
>> property was enabled only for host-passthrough and max cpu model.
>> As for other cpu model like cortex-a57, cortex-a53, cortex-a72,
>> this kvm-adjvtime is not enabled by default, which means the
>> virutal time can not be adjust for them.
>>
>> Here, for example, if VM is configured with kvm enabled:
>>
>>    <cpu mode='custom' match='exact' check='partial'>
>>      <model fallback='allow'>cortex-a72</model>
>>      <topology sockets='2' dies='1' cores='2' threads='1'/>
>>      <numa>
>>        <cell id='0' cpus='0-1' memory='16777216' unit='KiB'/>
>>        <cell id='1' cpus='2-3' memory='16777216' unit='KiB'/>
>>      </numa>
>>    </cpu>
>>
>> We cannot adjust virtual time even if 5.0 virt machine is used.
>> So i'd like to add it to other cpu model, do you have any
>> suggestions here ?
>>
>>
> 
> Hi Fang,
> 
> The cpu feature only requires kvm.  If a cpu model may be used with kvm,
> then the feature can be allowed to be used with the model.  What I find
> interesting is that the cpu model is being used with kvm instead of 'host'
> or 'max'.  Can you explain the reasons for that?  Currently, when using
yes，the cpu model is indeed used with kvm.

There is a situation where the host cpu model is Cortex-A72 and
a 'custom' cpu mode is used to keep insrtuction set compatible between
the source and destination host machine when doing live migration.
So the host physical machine cpu model is Cortex-A72 but
host-passthrough model is mode used here.

> kvm, the guest will always effectively get 'host' anyway, even when a
> model is provided. Indeed, for a model to work, kvm requires it to exactly
> match the host cpu and even then the ID registers are passed through from
> the host CPU, not taken from QEMU's model.
In the above scenario, the cpu mode is Cortex-A72 but maybe only a 
subset of cpu feature is presented to the guest.

Since the cpu model is not 'host' here, the kvm-adjvtime is not enabled 
by default. The guest virtual time will not be adjusted.

> 
> Thanks,
> drew
> 
> 
> .
> 

Re: About the kvm-no-adjvtime CPU property

[Andrew Jones]

On Mon, Jun 01, 2020 at 08:07:31PM +0800, Ying Fang wrote:
> 
> 
> On 2020/6/1 16:07, Andrew Jones wrote:
> > On Sat, May 30, 2020 at 04:56:26PM +0800, Ying Fang wrote:
> > > About the kvm-no-adjvtime CPU property
> > > 
> > > Hi Andrew,
> > > To adjust virutal time, a new kvm cpu property kvm-no-adjvtime
> > > was introduced to 5.0 virt machine types. However the cpu
> > > property was enabled only for host-passthrough and max cpu model.
> > > As for other cpu model like cortex-a57, cortex-a53, cortex-a72,
> > > this kvm-adjvtime is not enabled by default, which means the
> > > virutal time can not be adjust for them.
> > > 
> > > Here, for example, if VM is configured with kvm enabled:
> > > 
> > >    <cpu mode='custom' match='exact' check='partial'>
> > >      <model fallback='allow'>cortex-a72</model>
> > >      <topology sockets='2' dies='1' cores='2' threads='1'/>
> > >      <numa>
> > >        <cell id='0' cpus='0-1' memory='16777216' unit='KiB'/>
> > >        <cell id='1' cpus='2-3' memory='16777216' unit='KiB'/>
> > >      </numa>
> > >    </cpu>
> > > 
> > > We cannot adjust virtual time even if 5.0 virt machine is used.
> > > So i'd like to add it to other cpu model, do you have any
> > > suggestions here ?
> > > 
> > > 
> > 
> > Hi Fang,
> > 
> > The cpu feature only requires kvm.  If a cpu model may be used with kvm,
> > then the feature can be allowed to be used with the model.  What I find
> > interesting is that the cpu model is being used with kvm instead of 'host'
> > or 'max'.  Can you explain the reasons for that?  Currently, when using
> yes，the cpu model is indeed used with kvm.
> 
> There is a situation where the host cpu model is Cortex-A72 and
> a 'custom' cpu mode is used to keep insrtuction set compatible between
> the source and destination host machine when doing live migration.
> So the host physical machine cpu model is Cortex-A72 but
> host-passthrough model is mode used here.

Are the source and destinations hosts used in the migration identical?
If so, then the guest can use cpu 'host' and disable cpu features that
should not be exposed (e.g. -cpu host,pmu=off).  If the source and
destination hosts are not identical, then I'm curious what those exact
differences are.  With the way AArch64 KVM works today, even using the
Cortex-A72 cpu model should require identical hosts when migrating.  Or,
at least both hosts must be compatible with Cortex-A72 and any difference
in ID registers must be somehow getting hidden from the guest.

Thanks,
drew

Re: About the kvm-no-adjvtime CPU property

[Ying Fang]

On 2020/6/1 20:29, Andrew Jones wrote:
> On Mon, Jun 01, 2020 at 08:07:31PM +0800, Ying Fang wrote:
>>
>>
>> On 2020/6/1 16:07, Andrew Jones wrote:
>>> On Sat, May 30, 2020 at 04:56:26PM +0800, Ying Fang wrote:
>>>> About the kvm-no-adjvtime CPU property
>>>>
>>>> Hi Andrew,
>>>> To adjust virutal time, a new kvm cpu property kvm-no-adjvtime
>>>> was introduced to 5.0 virt machine types. However the cpu
>>>> property was enabled only for host-passthrough and max cpu model.
>>>> As for other cpu model like cortex-a57, cortex-a53, cortex-a72,
>>>> this kvm-adjvtime is not enabled by default, which means the
>>>> virutal time can not be adjust for them.
>>>>
>>>> Here, for example, if VM is configured with kvm enabled:
>>>>
>>>>     <cpu mode='custom' match='exact' check='partial'>
>>>>       <model fallback='allow'>cortex-a72</model>
>>>>       <topology sockets='2' dies='1' cores='2' threads='1'/>
>>>>       <numa>
>>>>         <cell id='0' cpus='0-1' memory='16777216' unit='KiB'/>
>>>>         <cell id='1' cpus='2-3' memory='16777216' unit='KiB'/>
>>>>       </numa>
>>>>     </cpu>
>>>>
>>>> We cannot adjust virtual time even if 5.0 virt machine is used.
>>>> So i'd like to add it to other cpu model, do you have any
>>>> suggestions here ?
>>>>
>>>>
>>>
>>> Hi Fang,
>>>
>>> The cpu feature only requires kvm.  If a cpu model may be used with kvm,
>>> then the feature can be allowed to be used with the model.  What I find
>>> interesting is that the cpu model is being used with kvm instead of 'host'
>>> or 'max'.  Can you explain the reasons for that?  Currently, when using
>> yes，the cpu model is indeed used with kvm.
>>
>> There is a situation where the host cpu model is Cortex-A72 and
>> a 'custom' cpu mode is used to keep insrtuction set compatible between
>> the source and destination host machine when doing live migration.
>> So the host physical machine cpu model is Cortex-A72 but
>> host-passthrough model is mode used here.
I mean host-passthrough model is 'not' used here. Sorry to make it 
confusing.
> 
> Are the source and destinations hosts used in the migration identical?
> If so, then the guest can use cpu 'host' and disable cpu features that
> should not be exposed (e.g. -cpu host,pmu=off).  If the source and
> destination hosts are not identical, then I'm curious what those exact
> differences are.  With the way AArch64 KVM works today, even using the
> Cortex-A72 cpu model should require identical hosts when migrating.  Or,
> at least both hosts must be compatible with Cortex-A72 and any difference
> in ID registers must be somehow getting hidden from the guest.
Yes, you are right.
We have AAarch64 server with cpu based on Cortex-A72 and some extra 
instruction set added. Source host with cpu based on V1 and destination 
host with cpu based on V2 and they are compatible with Cortex-A72. We 
want to use a 'custom' cpu mode here to make it possible to do live 
migration between them. This is the scenario where the 'host' cpu model 
is not used since a 'custom' cpu model Cortex-A72 is used here . However 
the kvm-adjvtime feature is also need. So I think we should move 
kvm_arm_add_vcpu_properties to arm_cpu_post_init, instead of limited to 
'host' and 'max' cpu model[1].

1: https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg00091.html
> 
> Thanks,
> drew
> 
> 
> 

Re: About the kvm-no-adjvtime CPU property

[Andrew Jones]

On Tue, Jun 02, 2020 at 03:47:22PM +0800, Ying Fang wrote:
> 
> 
> On 2020/6/1 20:29, Andrew Jones wrote:
> > On Mon, Jun 01, 2020 at 08:07:31PM +0800, Ying Fang wrote:
> > > 
> > > 
> > > On 2020/6/1 16:07, Andrew Jones wrote:
> > > > On Sat, May 30, 2020 at 04:56:26PM +0800, Ying Fang wrote:
> > > > > About the kvm-no-adjvtime CPU property
> > > > > 
> > > > > Hi Andrew,
> > > > > To adjust virutal time, a new kvm cpu property kvm-no-adjvtime
> > > > > was introduced to 5.0 virt machine types. However the cpu
> > > > > property was enabled only for host-passthrough and max cpu model.
> > > > > As for other cpu model like cortex-a57, cortex-a53, cortex-a72,
> > > > > this kvm-adjvtime is not enabled by default, which means the
> > > > > virutal time can not be adjust for them.
> > > > > 
> > > > > Here, for example, if VM is configured with kvm enabled:
> > > > > 
> > > > >     <cpu mode='custom' match='exact' check='partial'>
> > > > >       <model fallback='allow'>cortex-a72</model>
> > > > >       <topology sockets='2' dies='1' cores='2' threads='1'/>
> > > > >       <numa>
> > > > >         <cell id='0' cpus='0-1' memory='16777216' unit='KiB'/>
> > > > >         <cell id='1' cpus='2-3' memory='16777216' unit='KiB'/>
> > > > >       </numa>
> > > > >     </cpu>
> > > > > 
> > > > > We cannot adjust virtual time even if 5.0 virt machine is used.
> > > > > So i'd like to add it to other cpu model, do you have any
> > > > > suggestions here ?
> > > > > 
> > > > > 
> > > > 
> > > > Hi Fang,
> > > > 
> > > > The cpu feature only requires kvm.  If a cpu model may be used with kvm,
> > > > then the feature can be allowed to be used with the model.  What I find
> > > > interesting is that the cpu model is being used with kvm instead of 'host'
> > > > or 'max'.  Can you explain the reasons for that?  Currently, when using
> > > yes，the cpu model is indeed used with kvm.
> > > 
> > > There is a situation where the host cpu model is Cortex-A72 and
> > > a 'custom' cpu mode is used to keep insrtuction set compatible between
> > > the source and destination host machine when doing live migration.
> > > So the host physical machine cpu model is Cortex-A72 but
> > > host-passthrough model is mode used here.
> I mean host-passthrough model is 'not' used here. Sorry to make it
> confusing.

I guessed as much.

> > 
> > Are the source and destinations hosts used in the migration identical?
> > If so, then the guest can use cpu 'host' and disable cpu features that
> > should not be exposed (e.g. -cpu host,pmu=off).  If the source and
> > destination hosts are not identical, then I'm curious what those exact
> > differences are.  With the way AArch64 KVM works today, even using the
> > Cortex-A72 cpu model should require identical hosts when migrating.  Or,
> > at least both hosts must be compatible with Cortex-A72 and any difference
> > in ID registers must be somehow getting hidden from the guest.
> Yes, you are right.
> We have AAarch64 server with cpu based on Cortex-A72 and some extra
> instruction set added. Source host with cpu based on V1 and destination host
> with cpu based on V2 and they are compatible with Cortex-A72. We want to use
> a 'custom' cpu mode here to make it possible to do live migration between
> them. This is the scenario where the 'host' cpu model is not used since a
> 'custom' cpu model Cortex-A72 is used here .

What you've described here is indeed the reason to use CPU models. I.e.
enabling the migration from a host of one type to another by creating a
guest that only enables the features contained in both hosts (as well as
maintaining all state that describes the CPU type, e.g. MIDR).
Unfortunately, unless your KVM has patches that aren't upstream, then that
doesn't work on AArch64 KVM (more on that below). It may appear to be
working for you, because your guest kernel and userspace don't mind the
slight differences exposed to it between the hosts, or those differences
are limited to explicitly disabled features. If that's the case, then I
would guess that using '-cpu host' and disabling the same features would
"work" as well.

Here's some more details on why the Cortex-A72 CPU model doesn't matter
with upstream KVM. First, upstream AArch64 KVM doesn't support CPU models,
and it doesn't even have a Cortex-A72 preferred target. For Cortex-A72
it will use "KVM_ARM_TARGET_GENERIC_V8", which is the same thing 'host'
would do when running on a Cortex-A72. Second, if V2 of the Cortex-A72-
based CPU you're using changed the revision of the MIDR, or any other
state that gets passed directly through to the guest like the MIDR, then
that state will change on migration. If a guest looks before migration and
again after migration, then it could get confused. A guest kernel may only
look once on boot and then not notice, but anything exposed to userspace
is extra risky, as userspace may check more frequently.

In short, without KVM patches that aren't upstream, then it's risky to
migrate between machines with V1 and V2 of these CPUs. And, it doesn't
help to use the Cortex-A72 CPU model.

Thanks,
drew


> However the kvm-adjvtime
> feature is also need. So I think we should move kvm_arm_add_vcpu_properties
> to arm_cpu_post_init, instead of limited to 'host' and 'max' cpu model[1].
> 
> 1: https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg00091.html
> > 
> > Thanks,
> > drew
> > 
> > 
> > 
> 
> 

Re: About the kvm-no-adjvtime CPU property

[Ying Fang]

On 6/3/2020 4:53 PM, Andrew Jones wrote:
> On Tue, Jun 02, 2020 at 03:47:22PM +0800, Ying Fang wrote:
>>
>>
>> On 2020/6/1 20:29, Andrew Jones wrote:
>>> On Mon, Jun 01, 2020 at 08:07:31PM +0800, Ying Fang wrote:
>>>>
>>>>
>>>> On 2020/6/1 16:07, Andrew Jones wrote:
>>>>> On Sat, May 30, 2020 at 04:56:26PM +0800, Ying Fang wrote:
>>>>>> About the kvm-no-adjvtime CPU property
>>>>>>
>>>>>> Hi Andrew,
>>>>>> To adjust virutal time, a new kvm cpu property kvm-no-adjvtime
>>>>>> was introduced to 5.0 virt machine types. However the cpu
>>>>>> property was enabled only for host-passthrough and max cpu model.
>>>>>> As for other cpu model like cortex-a57, cortex-a53, cortex-a72,
>>>>>> this kvm-adjvtime is not enabled by default, which means the
>>>>>> virutal time can not be adjust for them.
>>>>>>
>>>>>> Here, for example, if VM is configured with kvm enabled:
>>>>>>
>>>>>>      <cpu mode='custom' match='exact' check='partial'>
>>>>>>        <model fallback='allow'>cortex-a72</model>
>>>>>>        <topology sockets='2' dies='1' cores='2' threads='1'/>
>>>>>>        <numa>
>>>>>>          <cell id='0' cpus='0-1' memory='16777216' unit='KiB'/>
>>>>>>          <cell id='1' cpus='2-3' memory='16777216' unit='KiB'/>
>>>>>>        </numa>
>>>>>>      </cpu>
>>>>>>
>>>>>> We cannot adjust virtual time even if 5.0 virt machine is used.
>>>>>> So i'd like to add it to other cpu model, do you have any
>>>>>> suggestions here ?
>>>>>>
>>>>>>
>>>>>
>>>>> Hi Fang,
>>>>>
>>>>> The cpu feature only requires kvm.  If a cpu model may be used with kvm,
>>>>> then the feature can be allowed to be used with the model.  What I find
>>>>> interesting is that the cpu model is being used with kvm instead of 'host'
>>>>> or 'max'.  Can you explain the reasons for that?  Currently, when using
>>>> yes，the cpu model is indeed used with kvm.
>>>>
>>>> There is a situation where the host cpu model is Cortex-A72 and
>>>> a 'custom' cpu mode is used to keep insrtuction set compatible between
>>>> the source and destination host machine when doing live migration.
>>>> So the host physical machine cpu model is Cortex-A72 but
>>>> host-passthrough model is mode used here.
>> I mean host-passthrough model is 'not' used here. Sorry to make it
>> confusing.
> 
> I guessed as much.
> 
>>>
>>> Are the source and destinations hosts used in the migration identical?
>>> If so, then the guest can use cpu 'host' and disable cpu features that
>>> should not be exposed (e.g. -cpu host,pmu=off).  If the source and
>>> destination hosts are not identical, then I'm curious what those exact
>>> differences are.  With the way AArch64 KVM works today, even using the
>>> Cortex-A72 cpu model should require identical hosts when migrating.  Or,
>>> at least both hosts must be compatible with Cortex-A72 and any difference
>>> in ID registers must be somehow getting hidden from the guest.
>> Yes, you are right.
>> We have AAarch64 server with cpu based on Cortex-A72 and some extra
>> instruction set added. Source host with cpu based on V1 and destination host
>> with cpu based on V2 and they are compatible with Cortex-A72. We want to use
>> a 'custom' cpu mode here to make it possible to do live migration between
>> them. This is the scenario where the 'host' cpu model is not used since a
>> 'custom' cpu model Cortex-A72 is used here .
> 
> What you've described here is indeed the reason to use CPU models. I.e.
> enabling the migration from a host of one type to another by creating a
> guest that only enables the features contained in both hosts (as well as
> maintaining all state that describes the CPU type, e.g. MIDR).
> Unfortunately, unless your KVM has patches that aren't upstream, then that
> doesn't work on AArch64 KVM (more on that below). It may appear to be
> working for you, because your guest kernel and userspace don't mind the
> slight differences exposed to it between the hosts, or those differences
> are limited to explicitly disabled features. If that's the case, then I
> would guess that using '-cpu host' and disabling the same features would
> "work" as well.

Yes, upstream KVM currently does not support it. We are planning to add
support for the aarch64 platform since we have the situation where it is
needed for our hardware.

@Marc Zyngier, is there anyone who doing on this?
> 
> Here's some more details on why the Cortex-A72 CPU model doesn't matter
> with upstream KVM. First, upstream AArch64 KVM doesn't support CPU models,
> and it doesn't even have a Cortex-A72 preferred target. For Cortex-A72
> it will use "KVM_ARM_TARGET_GENERIC_V8", which is the same thing 'host'
> would do when running on a Cortex-A72. Second, if V2 of the Cortex-A72-
> based CPU you're using changed the revision of the MIDR, or any other
> state that gets passed directly through to the guest like the MIDR, then
> that state will change on migration. If a guest looks before migration and
> again after migration, then it could get confused. A guest kernel may only
> look once on boot and then not notice, but anything exposed to userspace
> is extra risky, as userspace may check more frequently.

Yes, just as explained here.

> 
> In short, without KVM patches that aren't upstream, then it's risky to
> migrate between machines with V1 and V2 of these CPUs. And, it doesn't
> help to use the Cortex-A72 CPU model.
Thanks for your detailed introduction.

> 
> Thanks,
> drew
> 
> 
>> However the kvm-adjvtime
>> feature is also need. So I think we should move kvm_arm_add_vcpu_properties
>> to arm_cpu_post_init, instead of limited to 'host' and 'max' cpu model[1].
>>
>> 1: https://lists.gnu.org/archive/html/qemu-devel/2020-06/msg00091.html
>>>
>>> Thanks,
>>> drew
>>>
>>>
>>>
>>
>>
> 
> 
> .
>