Which clocksource does KVM use?

Which clocksource does KVM use?

[Arnabjyoti Kalita]

I am running QEMU with KVM using the below command line -

sudo ./qemu-system-x86_64 -m 1024 -machine pc-i440fx-3.0
-cpu qemu64,-kvmclock -accel kvm -netdev
tap,id=tap1,ifname=tap0,script=no,downscript=no
-device virtio-net-pci,netdev=tap1,mac=00:00:00:00:00:00
-drive file=ubuntu-16.04.server.qcow2,format=qcow2,if=none,id=img-direct
-device virtio-blk-pci,drive=img-direct


I can see that the current_clocksource used by the VM that is spawned
is reported as "tsc".

/sys/devices/system/clocksource/clocksource0$ cat current_clocksource
tsc

I collect a trace of the guest execution using IntelPT after running
the below command -

sudo ./perf kvm --guest --guestkallsyms=guest-kallsyms
--guestmodules=guest-modules
record -e intel_pt// -m ,65536

The IntelPT trace reveals that the function "read_hpet" gets called continuously
while I expected the function "read_tsc" to be called.

Does KVM change the clocksource in any way? I expect the clocksource
to be set at boot time,
how and why did the clocksource change later? Does KVM not support the
tsc clocksource ?

Note:

I am using QEMU version 3.0. The guest runs a 4.4.0-116-generic linux kernel.
Both my qemu host as well as the target architecture is x86_64. The
host machine is
also using "tsc" clocksource.

Best Regards,
Arnab

Re: Which clocksource does KVM use?

[Paolo Bonzini]

On 25/09/20 04:36, Arnabjyoti Kalita wrote:
> 
> Does KVM change the clocksource in any way? I expect the clocksource
> to be set at boot time,
> how and why did the clocksource change later? Does KVM not support the
> tsc clocksource ?

The TSC clocksource might fail, in which case Linux will fallback to
HPET.  Using kvmclock (which you have disabled) avoids the fallback.

Paolo

Re: Which clocksource does KVM use?

[Sean Christopherson]

On Fri, Sep 25, 2020 at 08:06:44AM +0530, Arnabjyoti Kalita wrote:
> I am running QEMU with KVM using the below command line -
> 
> sudo ./qemu-system-x86_64 -m 1024 -machine pc-i440fx-3.0
> -cpu qemu64,-kvmclock -accel kvm -netdev
> tap,id=tap1,ifname=tap0,script=no,downscript=no
> -device virtio-net-pci,netdev=tap1,mac=00:00:00:00:00:00
> -drive file=ubuntu-16.04.server.qcow2,format=qcow2,if=none,id=img-direct
> -device virtio-blk-pci,drive=img-direct
> 
> 
> I can see that the current_clocksource used by the VM that is spawned
> is reported as "tsc".
> 
> /sys/devices/system/clocksource/clocksource0$ cat current_clocksource
> tsc
> 
> I collect a trace of the guest execution using IntelPT after running
> the below command -
> 
> sudo ./perf kvm --guest --guestkallsyms=guest-kallsyms
> --guestmodules=guest-modules
> record -e intel_pt// -m ,65536
> 
> The IntelPT trace reveals that the function "read_hpet" gets called continuously
> while I expected the function "read_tsc" to be called.
> 
> Does KVM change the clocksource in any way?
 
KVM's paravirt clock affects the clocksource, but that's disable in via
"-kvmclock" in your command line.

> I expect the clocksource to be set at boot time, how and why did the
> clocksource change later? Does KVM not support the tsc clocksource ?

QEMU/KVM exposes TSC to the guest, but the Linux kernel's decision on whether
or not to use the TSC as its clocksource isn't exactly straightforward.

At a minimum, the TSC needs to be constant (count at the same rate regardless
of p-state, i.e. core frequency), which is referred to as invtsc by QEMU.  At
a glance, I don't think "-cpu qemu64" advertises support for invtsc.  This can
be forced via "-cpu qemu64,+invtsc", though that may spit out some warnings if
the host CPU itself doesn't have a constant TSC.  You can also override this
in the guest kernel by adding "tsc=reliable" to your kernel params.

C-states are another possible issu.  The kernel will mark the TSC as unstable
if C2 or deeper is supported (and maybe even C1 with MWAIT?) and the TSC isn't
marked as nonstop.  I don't _think_ this is relevant?  QEMU/KVM doesn't
support advertising a nonstop TSC, but I assume QEMU also doesn't advertise C2
or deeper (I've never actually looked), or MONITOR/MWAIT by default.

If the above are ruled out, the kernel can also mark the TSC as unstable and
switch to the HPET for a variety of other reasons.  You can check for this
by grepping for "Marking TSC unstable due to" in the guest kernel logs.

> Note:
> 
> I am using QEMU version 3.0. The guest runs a 4.4.0-116-generic linux kernel.
> Both my qemu host as well as the target architecture is x86_64. The
> host machine is
> also using "tsc" clocksource.
> 
> Best Regards,
> Arnab

Re: Which clocksource does KVM use?

[Arnabjyoti Kalita]

Thank you Paolo for this information. I believe setting the default
clocksource to HPET would be a more deterministic option in my case.

On Sat, Sep 26, 2020 at 1:49 AM Paolo Bonzini <pbonzini@redhat.com> wrote:
>
> On 25/09/20 04:36, Arnabjyoti Kalita wrote:
> >
> > Does KVM change the clocksource in any way? I expect the clocksource
> > to be set at boot time,
> > how and why did the clocksource change later? Does KVM not support the
> > tsc clocksource ?
>
> The TSC clocksource might fail, in which case Linux will fallback to
> HPET.  Using kvmclock (which you have disabled) avoids the fallback.
>
> Paolo
>

Re: Which clocksource does KVM use?

[Arnabjyoti Kalita]

Thank you Sean for the detailed information!

On Sat, Sep 26, 2020 at 1:54 AM Sean Christopherson
<sean.j.christopherson@intel.com> wrote:
>
> On Fri, Sep 25, 2020 at 08:06:44AM +0530, Arnabjyoti Kalita wrote:
> > I am running QEMU with KVM using the below command line -
> >
> > sudo ./qemu-system-x86_64 -m 1024 -machine pc-i440fx-3.0
> > -cpu qemu64,-kvmclock -accel kvm -netdev
> > tap,id=tap1,ifname=tap0,script=no,downscript=no
> > -device virtio-net-pci,netdev=tap1,mac=00:00:00:00:00:00
> > -drive file=ubuntu-16.04.server.qcow2,format=qcow2,if=none,id=img-direct
> > -device virtio-blk-pci,drive=img-direct
> >
> >
> > I can see that the current_clocksource used by the VM that is spawned
> > is reported as "tsc".
> >
> > /sys/devices/system/clocksource/clocksource0$ cat current_clocksource
> > tsc
> >
> > I collect a trace of the guest execution using IntelPT after running
> > the below command -
> >
> > sudo ./perf kvm --guest --guestkallsyms=guest-kallsyms
> > --guestmodules=guest-modules
> > record -e intel_pt// -m ,65536
> >
> > The IntelPT trace reveals that the function "read_hpet" gets called continuously
> > while I expected the function "read_tsc" to be called.
> >
> > Does KVM change the clocksource in any way?
>
> KVM's paravirt clock affects the clocksource, but that's disable in via
> "-kvmclock" in your command line.
>
> > I expect the clocksource to be set at boot time, how and why did the
> > clocksource change later? Does KVM not support the tsc clocksource ?
>
> QEMU/KVM exposes TSC to the guest, but the Linux kernel's decision on whether
> or not to use the TSC as its clocksource isn't exactly straightforward.
>
> At a minimum, the TSC needs to be constant (count at the same rate regardless
> of p-state, i.e. core frequency), which is referred to as invtsc by QEMU.  At
> a glance, I don't think "-cpu qemu64" advertises support for invtsc.  This can
> be forced via "-cpu qemu64,+invtsc", though that may spit out some warnings if
> the host CPU itself doesn't have a constant TSC.  You can also override this
> in the guest kernel by adding "tsc=reliable" to your kernel params.
>
> C-states are another possible issu.  The kernel will mark the TSC as unstable
> if C2 or deeper is supported (and maybe even C1 with MWAIT?) and the TSC isn't
> marked as nonstop.  I don't _think_ this is relevant?  QEMU/KVM doesn't
> support advertising a nonstop TSC, but I assume QEMU also doesn't advertise C2
> or deeper (I've never actually looked), or MONITOR/MWAIT by default.
>
> If the above are ruled out, the kernel can also mark the TSC as unstable and
> switch to the HPET for a variety of other reasons.  You can check for this
> by grepping for "Marking TSC unstable due to" in the guest kernel logs.
>
> > Note:
> >
> > I am using QEMU version 3.0. The guest runs a 4.4.0-116-generic linux kernel.
> > Both my qemu host as well as the target architecture is x86_64. The
> > host machine is
> > also using "tsc" clocksource.
> >
> > Best Regards,
> > Arnab