TSC deadline timer in guests vs. migration?

TSC deadline timer in guests vs. migration?

[Paolo Bonzini]

Can bad things happen if a guest using the TSC deadline timer is
migrated?  The guest doesn't re-calibrate the TSC after migration, and
the TSC frequency can and will change unless your data center is
perfectly homogeneous.

What do you think?

Paolo

Re: TSC deadline timer in guests vs. migration?

[Eduardo Habkost]

On Mon, Jul 04, 2016 at 01:01:42PM +0200, Paolo Bonzini wrote:
> Can bad things happen if a guest using the TSC deadline timer is
> migrated?  The guest doesn't re-calibrate the TSC after migration, and
> the TSC frequency can and will change unless your data center is
> perfectly homogeneous.

I believe it can. Also: what about everything else the guest may
try to use the TSC for? (even without TSC deadline timer)

>From a quick look at Linux code, it looks like the TSC
clocksource code is more strict (but the exact rules are hard to
follow), but it will still use the TSC in native_sched_clock().

What about kvmclock? Do we have a mechanism to trigger a re-read
of TSC frequency from the hypervisor?

-- 
Eduardo

Re: TSC deadline timer in guests vs. migration?

[Marcelo Tosatti]

On Mon, Jul 04, 2016 at 01:01:42PM +0200, Paolo Bonzini wrote:
> Can bad things happen if a guest using the TSC deadline timer is
> migrated?  The guest doesn't re-calibrate the TSC after migration, and
> the TSC frequency can and will change unless your data center is
> perfectly homogeneous.

It can fire earlier if the destination runs at a higher frequency.
It will fire past the configured time if the destination runs at a slower frequency.

Suppose the first case is worse.

Should convert the expiration time to nanoseconds i suppose, and then 
convert back on the destination.

Re: TSC deadline timer in guests vs. migration?

[Eduardo Habkost]

On Mon, Jul 04, 2016 at 04:30:08PM -0300, Marcelo Tosatti wrote:
> On Mon, Jul 04, 2016 at 01:01:42PM +0200, Paolo Bonzini wrote:
> > Can bad things happen if a guest using the TSC deadline timer is
> > migrated?  The guest doesn't re-calibrate the TSC after migration, and
> > the TSC frequency can and will change unless your data center is
> > perfectly homogeneous.
> 
> It can fire earlier if the destination runs at a higher frequency.
> It will fire past the configured time if the destination runs at a slower frequency.
> 
> Suppose the first case is worse.
> 
> Should convert the expiration time to nanoseconds i suppose, and then 
> convert back on the destination.

This won't make any difference if the guest sets up a new timer
after migration (but using the old TSC frequency), will it?

-- 
Eduardo

Re: TSC deadline timer in guests vs. migration?

[Paolo Bonzini]

> On Mon, Jul 04, 2016 at 04:30:08PM -0300, Marcelo Tosatti wrote:
> > On Mon, Jul 04, 2016 at 01:01:42PM +0200, Paolo Bonzini wrote:
> > > Can bad things happen if a guest using the TSC deadline timer is
> > > migrated?  The guest doesn't re-calibrate the TSC after migration, and
> > > the TSC frequency can and will change unless your data center is
> > > perfectly homogeneous.
> > 
> > It can fire earlier if the destination runs at a higher frequency.
> > It will fire past the configured time if the destination runs at a slower
> > frequency.
> > 
> > Suppose the first case is worse.
> > 
> > Should convert the expiration time to nanoseconds i suppose, and then
> > convert back on the destination.
> 
> This won't make any difference if the guest sets up a new timer
> after migration (but using the old TSC frequency), will it?

Indeed.  We need a way to compute the deadline that is customized
for kvmclock, and uses the kvmclock page instead of tsc_khz.

Paolo

Re: TSC deadline timer in guests vs. migration?

[Paolo Bonzini]

On 04/07/2016 21:07, Eduardo Habkost wrote:
> On Mon, Jul 04, 2016 at 01:01:42PM +0200, Paolo Bonzini wrote:
>> Can bad things happen if a guest using the TSC deadline timer is
>> migrated?  The guest doesn't re-calibrate the TSC after migration, and
>> the TSC frequency can and will change unless your data center is
>> perfectly homogeneous.
> 
> I believe it can. Also: what about everything else the guest may
> try to use the TSC for? (even without TSC deadline timer)

Luckily there isn't much that uses it directly.  Userspace should not
trust the TSC.  In practice there's quite some code that uses it for
performance measurement, but probably not for timekeeping.

> From a quick look at Linux code, it looks like the TSC
> clocksource code is more strict (but the exact rules are hard to
> follow), but it will still use the TSC in native_sched_clock().

This is not a big problem because KVM pvops override sched_clock to
kvm_clock_read().

> What about kvmclock? Do we have a mechanism to trigger a re-read
> of TSC frequency from the hypervisor?

Yes, that's the point of kvmclock.  Every time the TSC frequency changes
(which includes the first time you start the VCPU on a new host), the
kvmclock pages are updated with new data.

Paolo

Re: TSC deadline timer in guests vs. migration?

[Marcelo Tosatti]

On Mon, Jul 04, 2016 at 04:45:06PM -0300, Eduardo Habkost wrote:
> On Mon, Jul 04, 2016 at 04:30:08PM -0300, Marcelo Tosatti wrote:
> > On Mon, Jul 04, 2016 at 01:01:42PM +0200, Paolo Bonzini wrote:
> > > Can bad things happen if a guest using the TSC deadline timer is
> > > migrated?  The guest doesn't re-calibrate the TSC after migration, and
> > > the TSC frequency can and will change unless your data center is
> > > perfectly homogeneous.
> > 
> > It can fire earlier if the destination runs at a higher frequency.
> > It will fire past the configured time if the destination runs at a slower frequency.
> > 
> > Suppose the first case is worse.
> > 
> > Should convert the expiration time to nanoseconds i suppose, and then 
> > convert back on the destination.
> 
> This won't make any difference if the guest sets up a new timer
> after migration (but using the old TSC frequency), will it?

It does, because the timer setup traps to the host, where you can
convert it to the proper value:

                guest_tsc = kvm_x86_ops->read_l1_tsc(vcpu,
native_read_tsc());
                if (likely(tscdeadline > guest_tsc)) {
                        ns = (tscdeadline - guest_tsc) * 1000000ULL;
                        do_div(ns, this_tsc_khz);
                        expire = ktime_add_ns(now, ns);
                        expire = ktime_sub_ns(expire,
lapic_timer_advance_ns);
                        hrtimer_start(&apic->lapic_timer.timer,
                                      expire, HRTIMER_MODE_ABS);
                } else

I've opened a BZ: feel free to take it, or make a reference to it
if you submit a patch to fix it (including the CC'ed Google guys).

https://bugzilla.redhat.com/show_bug.cgi?id=1352909

Re: TSC deadline timer in guests vs. migration?

[Eduardo Habkost]

On Tue, Jul 05, 2016 at 09:51:41AM -0300, Marcelo Tosatti wrote:
> On Mon, Jul 04, 2016 at 04:45:06PM -0300, Eduardo Habkost wrote:
> > On Mon, Jul 04, 2016 at 04:30:08PM -0300, Marcelo Tosatti wrote:
> > > On Mon, Jul 04, 2016 at 01:01:42PM +0200, Paolo Bonzini wrote:
> > > > Can bad things happen if a guest using the TSC deadline timer is
> > > > migrated?  The guest doesn't re-calibrate the TSC after migration, and
> > > > the TSC frequency can and will change unless your data center is
> > > > perfectly homogeneous.
> > > 
> > > It can fire earlier if the destination runs at a higher frequency.
> > > It will fire past the configured time if the destination runs at a slower frequency.
> > > 
> > > Suppose the first case is worse.
> > > 
> > > Should convert the expiration time to nanoseconds i suppose, and then 
> > > convert back on the destination.
> > 
> > This won't make any difference if the guest sets up a new timer
> > after migration (but using the old TSC frequency), will it?
> 
> It does, because the timer setup traps to the host, where you can
> convert it to the proper value:

To convert it to the proper value, you need to know what's the
TSC frequency the guest is assuming. How would you do that?

-- 
Eduardo

Re: TSC deadline timer in guests vs. migration?

[Paolo Bonzini]

On 05/07/2016 15:04, Eduardo Habkost wrote:
> On Tue, Jul 05, 2016 at 09:51:41AM -0300, Marcelo Tosatti wrote:
>> On Mon, Jul 04, 2016 at 04:45:06PM -0300, Eduardo Habkost wrote:
>>> On Mon, Jul 04, 2016 at 04:30:08PM -0300, Marcelo Tosatti wrote:
>>>> On Mon, Jul 04, 2016 at 01:01:42PM +0200, Paolo Bonzini wrote:
>>>>> Can bad things happen if a guest using the TSC deadline timer is
>>>>> migrated?  The guest doesn't re-calibrate the TSC after migration, and
>>>>> the TSC frequency can and will change unless your data center is
>>>>> perfectly homogeneous.
>>>>
>>>> It can fire earlier if the destination runs at a higher frequency.
>>>> It will fire past the configured time if the destination runs at a slower frequency.
>>>>
>>>> Suppose the first case is worse.
>>>>
>>>> Should convert the expiration time to nanoseconds i suppose, and then 
>>>> convert back on the destination.
>>>
>>> This won't make any difference if the guest sets up a new timer
>>> after migration (but using the old TSC frequency), will it?
>>
>> It does, because the timer setup traps to the host, where you can
>> convert it to the proper value:
> 
> To convert it to the proper value, you need to know what's the
> TSC frequency the guest is assuming. How would you do that?

In theory with KVM_SET_TSC_KHZ, but it has to be specified manually and
I'm not sure anyone has tested this recently.  I'm also not sure how
robust it is, but I'm sure it's fairly slow because it triggers a
kvmclock update on every vmentry.

tsc_always_catchup is high on the list of things that I hate in KVM.
I'd like to make KVM_SET_TSC_KHZ fail if TSC scaling is not supported by
the processor.

Paolo

Re: TSC deadline timer in guests vs. migration?

[Eduardo Habkost]

On Tue, Jul 05, 2016 at 03:34:20PM +0200, Paolo Bonzini wrote:
> On 05/07/2016 15:04, Eduardo Habkost wrote:
> > On Tue, Jul 05, 2016 at 09:51:41AM -0300, Marcelo Tosatti wrote:
> >> On Mon, Jul 04, 2016 at 04:45:06PM -0300, Eduardo Habkost wrote:
> >>> On Mon, Jul 04, 2016 at 04:30:08PM -0300, Marcelo Tosatti wrote:
> >>>> On Mon, Jul 04, 2016 at 01:01:42PM +0200, Paolo Bonzini wrote:
> >>>>> Can bad things happen if a guest using the TSC deadline timer is
> >>>>> migrated?  The guest doesn't re-calibrate the TSC after migration, and
> >>>>> the TSC frequency can and will change unless your data center is
> >>>>> perfectly homogeneous.
> >>>>
> >>>> It can fire earlier if the destination runs at a higher frequency.
> >>>> It will fire past the configured time if the destination runs at a slower frequency.
> >>>>
> >>>> Suppose the first case is worse.
> >>>>
> >>>> Should convert the expiration time to nanoseconds i suppose, and then 
> >>>> convert back on the destination.
> >>>
> >>> This won't make any difference if the guest sets up a new timer
> >>> after migration (but using the old TSC frequency), will it?
> >>
> >> It does, because the timer setup traps to the host, where you can
> >> convert it to the proper value:
> > 
> > To convert it to the proper value, you need to know what's the
> > TSC frequency the guest is assuming. How would you do that?
> 
> In theory with KVM_SET_TSC_KHZ, but it has to be specified manually and
> I'm not sure anyone has tested this recently.  I'm also not sure how
> robust it is, but I'm sure it's fairly slow because it triggers a
> kvmclock update on every vmentry.
> 

Since QEMU 2.6 it doesn't need to be specified manually: the TSC
scaling patches included in 2.6 will call KVM_SET_TSC_KHZ every
time the TSC frequency in the new host doesn't match the source.
See QEMU commit 36f96c.

> tsc_always_catchup is high on the list of things that I hate in KVM.
> I'd like to make KVM_SET_TSC_KHZ fail if TSC scaling is not supported by
> the processor.
> 

Hmm, I assumed KVM_SET_TSC_KHZ would fail if TSC scaling wasn't
supported.

In this case, if TSC scaling is not supported, is it preferable
to have the TSC frequency suddenly changing without the guest
being aware of it, or to run in tsc_always_catchup mode? QEMU 2.5
did the former, QEMU 2.6 does the latter.

-- 
Eduardo

Re: TSC deadline timer in guests vs. migration?

[Marcelo Tosatti]

On Tue, Jul 05, 2016 at 03:34:20PM +0200, Paolo Bonzini wrote:
> 
> 
> On 05/07/2016 15:04, Eduardo Habkost wrote:
> > On Tue, Jul 05, 2016 at 09:51:41AM -0300, Marcelo Tosatti wrote:
> >> On Mon, Jul 04, 2016 at 04:45:06PM -0300, Eduardo Habkost wrote:
> >>> On Mon, Jul 04, 2016 at 04:30:08PM -0300, Marcelo Tosatti wrote:
> >>>> On Mon, Jul 04, 2016 at 01:01:42PM +0200, Paolo Bonzini wrote:
> >>>>> Can bad things happen if a guest using the TSC deadline timer is
> >>>>> migrated?  The guest doesn't re-calibrate the TSC after migration, and
> >>>>> the TSC frequency can and will change unless your data center is
> >>>>> perfectly homogeneous.
> >>>>
> >>>> It can fire earlier if the destination runs at a higher frequency.
> >>>> It will fire past the configured time if the destination runs at a slower frequency.
> >>>>
> >>>> Suppose the first case is worse.
> >>>>
> >>>> Should convert the expiration time to nanoseconds i suppose, and then 
> >>>> convert back on the destination.
> >>>
> >>> This won't make any difference if the guest sets up a new timer
> >>> after migration (but using the old TSC frequency), will it?
> >>
> >> It does, because the timer setup traps to the host, where you can
> >> convert it to the proper value:
> > 
> > To convert it to the proper value, you need to know what's the
> > TSC frequency the guest is assuming. How would you do that?
> 
> In theory with KVM_SET_TSC_KHZ, but it has to be specified manually and

Just read the TSC value where the guest has booted and migrate that.

> I'm not sure anyone has tested this recently.  I'm also not sure how
> robust it is, but I'm sure it's fairly slow because it triggers a
> kvmclock update on every vmentry.

But agreed on that the rest of TSC scaling is poorly thought through.

> tsc_always_catchup is high on the list of things that I hate in KVM.

It can be dropped probably. I don't see a need for it (tsc scaling 
was never supported properly because of kvmclock).

> I'd like to make KVM_SET_TSC_KHZ fail if TSC scaling is not supported by
> the processor.
> 
> Paolo

Thats a good idea.

Re: TSC deadline timer in guests vs. migration?

[Paolo Bonzini]

On 05/07/2016 20:11, Eduardo Habkost wrote:
> In this case, if TSC scaling is not supported, is it preferable
> to have the TSC frequency suddenly changing without the guest
> being aware of it, or to run in tsc_always_catchup mode? QEMU 2.5
> did the former, QEMU 2.6 does the latter.

I suspect the former is better just because we don't know the exact
effect of tsc_always_catchup.

Paolo

Re: TSC deadline timer in guests vs. migration?

[Paolo Bonzini]

On 05/07/2016 22:39, Marcelo Tosatti wrote:
> On Tue, Jul 05, 2016 at 03:34:20PM +0200, Paolo Bonzini wrote:
>>
>>
>> On 05/07/2016 15:04, Eduardo Habkost wrote:
>>> On Tue, Jul 05, 2016 at 09:51:41AM -0300, Marcelo Tosatti wrote:
>>>> On Mon, Jul 04, 2016 at 04:45:06PM -0300, Eduardo Habkost wrote:
>>>>> On Mon, Jul 04, 2016 at 04:30:08PM -0300, Marcelo Tosatti wrote:
>>>>>> On Mon, Jul 04, 2016 at 01:01:42PM +0200, Paolo Bonzini wrote:
>>>>>>> Can bad things happen if a guest using the TSC deadline timer is
>>>>>>> migrated?  The guest doesn't re-calibrate the TSC after migration, and
>>>>>>> the TSC frequency can and will change unless your data center is
>>>>>>> perfectly homogeneous.
>>>>>>
>>>>>> It can fire earlier if the destination runs at a higher frequency.
>>>>>> It will fire past the configured time if the destination runs at a slower frequency.
>>>>>>
>>>>>> Suppose the first case is worse.
>>>>>>
>>>>>> Should convert the expiration time to nanoseconds i suppose, and then 
>>>>>> convert back on the destination.
>>>>>
>>>>> This won't make any difference if the guest sets up a new timer
>>>>> after migration (but using the old TSC frequency), will it?
>>>>
>>>> It does, because the timer setup traps to the host, where you can
>>>> convert it to the proper value:
>>>
>>> To convert it to the proper value, you need to know what's the
>>> TSC frequency the guest is assuming. How would you do that?
>>
>> In theory with KVM_SET_TSC_KHZ, but it has to be specified manually and
> 
> Just read the TSC value where the guest has booted and migrate that.

Right, QEMU 2.6 does that.

But even worse, instead of tsc_always_catchup you get a WARN if the user
requests a TSC rate below hardware spped.

>> I'd like to make KVM_SET_TSC_KHZ fail if TSC scaling is not supported by
>> the processor.
> 
> Thats a good idea.

Good!

Paolo

Re: TSC deadline timer in guests vs. migration?

[Eduardo Habkost]

On Tue, Jul 05, 2016 at 11:40:22PM +0200, Paolo Bonzini wrote:
> 
> 
> On 05/07/2016 20:11, Eduardo Habkost wrote:
> > In this case, if TSC scaling is not supported, is it preferable
> > to have the TSC frequency suddenly changing without the guest
> > being aware of it, or to run in tsc_always_catchup mode? QEMU 2.5
> > did the former, QEMU 2.6 does the latter.
> 
> I suspect the former is better just because we don't know the exact
> effect of tsc_always_catchup.

Forget what I said. QEMU 2.6 checks for KVM_CAP_TSC_CONTROL
before calling KVM_SET_TSC_KHZ. In this case, it will never
trigger tsc_always_catchup, right?

-- 
Eduardo

Re: TSC deadline timer in guests vs. migration?

[Paolo Bonzini]

On 06/07/2016 00:12, Eduardo Habkost wrote:
> On Tue, Jul 05, 2016 at 11:40:22PM +0200, Paolo Bonzini wrote:
>>
>>
>> On 05/07/2016 20:11, Eduardo Habkost wrote:
>>> In this case, if TSC scaling is not supported, is it preferable
>>> to have the TSC frequency suddenly changing without the guest
>>> being aware of it, or to run in tsc_always_catchup mode? QEMU 2.5
>>> did the former, QEMU 2.6 does the latter.
>>
>> I suspect the former is better just because we don't know the exact
>> effect of tsc_always_catchup.
> 
> Forget what I said. QEMU 2.6 checks for KVM_CAP_TSC_CONTROL
> before calling KVM_SET_TSC_KHZ. In this case, it will never
> trigger tsc_always_catchup, right?

Yes, I checked right after writing. :)

Paolo