Re: Timekeeping on ARM guests/hosts

[Miriam Zimmerman <mutexlox@google.com>]

On Fri, Nov 2, 2018 at 7:34 AM Christoffer Dall
<christoffer.dall@arm.com> wrote:
>
> On Wed, Oct 31, 2018 at 11:49:05AM -0700, Miriam Zimmerman wrote:
>
> [...]
>
> >
> > > CONFIG_KVM_GUEST fixes this by providing the guest with information on
> > > the host time and informs it when the guest is paused (see
> > > MSR_KVM_SYSTEM_TIME_NEW). Arm doesn't (yet) have para-virtualised time.
> > >
> > > On Arm, as far as I know, the guest's view of time is purely from the
> > > virtual counter. Since nothing saves/restores this during the pause, the
> > > counter continues to increment and the jump in time is visible to the guest.
> > What is this virtual counter? How can one configure or enable it?
>
> The 'virtual counter' is an architectural concept in the Arm
> architecture, not a virtual (in the software VM sense) counter.
>
> The Arm Generic Timer Architecture defines two counters:
>  - The physical counter, accessed via CNTPCT_EL0
>  - The virtual counter, accessed via CNTVCT_EL0
>
> And a number of timers.  All timers are associated with the physical
> counter, except the 'EL1 Virtual Timer' which uses the count in the
> virtual counter for its comparison.
>
> The virtual counter yields the value of the physical counter minus an
> offset.
>
> That offset is controlled by a hypervisor running at EL2 which can
> program the offset value in CNTVOFF_EL2.
>
> The basic idea is that a VM can use the virtual timer/counter to keep
> track of some notion of virtual time, and the physical timer/counter to
> keep track of something that always relates to wall-clock time.
>
> This all breaks quite horribly with migration where the physical counter
> is not meaningful across physical systems, and where the counter
> frequencies can vary.
>
> But we can use the offset for the virtual counter to communicate some
> other measure of time to VMs.

That makes sense. Thanks for the primer! I haven't done any ARM kernel
work before, so this is all great to know.

> > My understanding of what's going on now is that the guest kernel right
> > now has roughly no awareness that it's inside of a VM, so it just
> > tracks time using the assumption that it is native and is unaware that
> > the host ever suspends. (It just doesn't see any CPU time then.)
> >
> > >
> > > Whether the guest sees time progress depends on what happens to that
> > > counter during suspend. kvmtool pause/resume simply prevents the vCPU
> > > threads from continuing, but the system counter is still running.
> > >
> > > If you save the state of a VM to a file then the counter value is
> > > saved/restored so the guest won't see any change in time.
> > I don't believe that we're explicitly saving any state to a file on
> > suspend right now, though I could be wrong; I haven't looked into that
> > codepath yet.
>
> The key is whether the userspace program that controls the KVM VM
> (kvmtool, QEMU, crosvm) uses the KVM_REG_ARM_TIMER_CNT ioctl to save the
> VM view of virtual time, and to retore that at a later time.
>
> KVM adjusts the CNTVOFF_EL2 for the VM on which the ioctl is executed to
> represent the value written by userspace to the VM when the VM reads
> CNTVCT_EL0.
>

Ah hah. I haven't looked yet, but I bet this is the missing piece.
Thanks very much, Christoffer!

>
> Thanks,
>
>     Christoffer