Re: x86: kvm: Revert "remove sched notifier for cross-cpu migrations"

From: Andy Lutomirski <luto@amacapital.net>
To: Marcelo Tosatti <mtosatti@redhat.com>
Cc: kvm list <kvm@vger.kernel.org>, Radim Krcmar <rkrcmar@redhat.com>,
	stable <stable@vger.kernel.org>,
	Paolo Bonzini <pbonzini@redhat.com>
Subject: Re: x86: kvm: Revert "remove sched notifier for cross-cpu migrations"
Date: Thu, 26 Mar 2015 11:51:00 -0700	[thread overview]
Message-ID: <CALCETrWpSFcPN+-e6h+BD9djgtXSDGPNxW=ynSnmo0dEWxaZRg@mail.gmail.com> (raw)
In-Reply-To: <20150326112907.GA15098@amt.cnet>

On Thu, Mar 26, 2015 at 4:29 AM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
> On Wed, Mar 25, 2015 at 04:22:03PM -0700, Andy Lutomirski wrote:
>> On Wed, Mar 25, 2015 at 4:13 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
>> > On Wed, Mar 25, 2015 at 03:48:02PM -0700, Andy Lutomirski wrote:
>> >> On Wed, Mar 25, 2015 at 3:41 PM, Marcelo Tosatti <mtosatti@redhat.com> wrote:
>> >> > On Wed, Mar 25, 2015 at 03:33:10PM -0700, Andy Lutomirski wrote:
>> >> >> On Mar 25, 2015 2:29 PM, "Marcelo Tosatti" <mtosatti@redhat.com> wrote:
>> >> >> >
>> >> >> > On Wed, Mar 25, 2015 at 01:52:15PM +0100, Radim Krčmář wrote:
>> >> >> > > 2015-03-25 12:08+0100, Radim Krčmář:
>> >> >> > > > Reverting the patch protects us from any migration, but I don't think we
>> >> >> > > > need to care about changing VCPUs as long as we read a consistent data
>> >> >> > > > from kvmclock.  (VCPU can change outside of this loop too, so it doesn't
>> >> >> > > > matter if we return a value not fit for this VCPU.)
>> >> >> > > >
>> >> >> > > > I think we could drop the second __getcpu if our kvmclock was being
>> >> >> > > > handled better;  maybe with a patch like the one below:
>> >> >> > >
>> >> >> > > The second __getcpu is not neccessary, but I forgot about rdtsc.
>> >> >> > > We need to either use rtdscp, know the host has synchronized tsc, or
>> >> >> > > monitor VCPU migrations.  Only the last one works everywhere.
>> >> >> >
>> >> >> > The vdso code is only used if host has synchronized tsc.
>> >> >> >
>> >> >> > But you have to handle the case where host goes from synchronized tsc to
>> >> >> > unsynchronized tsc (see the clocksource notifier in the host side).
>> >> >> >
>> >> >>
>> >> >> Can't we change the host to freeze all vcpus and clear the stable bit
>> >> >> on all of them if this happens?  This would simplify and speed up
>> >> >> vclock_gettime.
>> >> >>
>> >> >> --Andy
>> >> >
>> >> > Seems interesting to do on 512-vcpus, but sure, could be done.
>> >> >
>> >>
>> >> If you have a 512-vcpu system that switches between stable and
>> >> unstable more than once per migration, then I expect that you have
>> >> serious problems and this is the least of your worries.
>> >>
>> >> Personally, I'd *much* rather we just made vcpu 0's pvti authoritative
>> >> if we're stable.  If nothing else, I'm not even remotely convinced
>> >> that the current scheme gives monotonic timing due to skew between
>> >> when the updates happen on different vcpus.
>> >
>> > Can you write down the problem ?
>> >
>>
>> I can try.
>>
>> Suppose we start out with all vcpus agreeing on their pvti and perfect
>> invariant TSCs.  Now the host updates its frequency (due to NTP or
>> whatever).  KVM updates vcpu 0's pvti.  Before KVM updates vcpu 1's
>> pvti, guest code on vcpus 0 and 1 see synced TSCs but different pvti.
>> They'll disagree on the time, and one of them will be ahead until vcpu
>> 1's pvti gets updated.
>
> The masterclock scheme enforces the same system_timestamp/tsc_timestamp pairs
> to be visible at one time, for all vcpus.
>
>
>  * That is, when timespec0 != timespec1, M < N. Unfortunately that is
>  * not
>  * always the case (the difference between two distinct xtime instances
>  * might be smaller then the difference between corresponding TSC reads,
>  * when updating guest vcpus pvclock areas).
>  *
>  * To avoid that problem, do not allow visibility of distinct
>  * system_timestamp/tsc_timestamp values simultaneously: use a master
>  * copy of host monotonic time values. Update that master copy
>  * in lockstep.
>
>

[resend without HTML]

Yuck.  So we have per cpu timing data, but the protocol is only usable
for monotonic timing because we forcibly freeze all vcpus when we
update the nominally per cpu data.

The obvious guest implementations are still unnecessarily slow,
though.  It would be nice if the guest could get away without using
any getcpu operation at all.

Even if we fixed the host to increment version as advertised, I think
we can't avoid two getcpu ops.  We need one before rdtsc to figure out
which pvti to look at, and we need another to make sure that we were
actually on that cpu at the time we did rdtsc.  (Rdtscp doesn't help
-- we need to check version before rdtsc, and we don't know what
version to check until we do a getcpu.). The migration hook has the
same issue -- we need to check the migration count, then confirm we're
on that cpu, then check the migration count again, and we can't do
that until we know what cpu we're on.

If, on the other hand, we could rely on having all of these things in
sync, then this complication goes away, and we go down from two getcpu
ops to zero.

--Andy