From: Marcelo Tosatti <mtosatti@redhat.com>
To: Thomas Gleixner <tglx@linutronix.de>
Cc: Maxim Levitsky <mlevitsk@redhat.com>,
kvm@vger.kernel.org, "H. Peter Anvin" <hpa@zytor.com>,
Paolo Bonzini <pbonzini@redhat.com>,
Jonathan Corbet <corbet@lwn.net>,
Jim Mattson <jmattson@google.com>,
Wanpeng Li <wanpengli@tencent.com>,
"open list:KERNEL SELFTEST FRAMEWORK"
<linux-kselftest@vger.kernel.org>,
Vitaly Kuznetsov <vkuznets@redhat.com>,
Sean Christopherson <sean.j.christopherson@intel.com>,
open list <linux-kernel@vger.kernel.org>,
Ingo Molnar <mingo@redhat.com>,
"maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT)"
<x86@kernel.org>, Joerg Roedel <joro@8bytes.org>,
Borislav Petkov <bp@alien8.de>, Shuah Khan <shuah@kernel.org>,
Andrew Jones <drjones@redhat.com>,
Oliver Upton <oupton@google.com>,
"open list:DOCUMENTATION" <linux-doc@vger.kernel.org>
Subject: Re: [PATCH v2 1/3] KVM: x86: implement KVM_{GET|SET}_TSC_STATE
Date: Fri, 11 Dec 2020 11:18:22 -0300 [thread overview]
Message-ID: <20201211141822.GA67764@fuller.cnet> (raw)
In-Reply-To: <87v9d8h3lx.fsf@nanos.tec.linutronix.de>
On Fri, Dec 11, 2020 at 02:30:34PM +0100, Thomas Gleixner wrote:
> On Thu, Dec 10 2020 at 21:27, Marcelo Tosatti wrote:
> > On Thu, Dec 10, 2020 at 10:48:10PM +0100, Thomas Gleixner wrote:
> >> You really all live in a seperate universe creating your own rules how
> >> things which other people work hard on to get it correct can be screwed
> >> over.
> >
> > 1. T = read timestamp.
> > 2. migrate (VM stops for a certain period).
> > 3. use timestamp T.
>
> This is exactly the problem. Time stops at pause and continues where it
> stopped on resume.
>
> But CLOCK_REALTIME and CLOCK_TAI advanced in reality. So up to the point
> where NTP fixes this - if there is NTP at all - the guest CLOCK_REALTIME
> and CLOCK_TAI are off by tpause.
>
> Now the application gets a packet from the outside world with a
> CLOCK_REALTIME timestamp which is suddenly ahead of the value it reads
> from clock_gettime(CLOCK_REALTIME) by tpause. So what is it supposed to
> do with that? Make stupid assumptions that the other end screwed up
> timekeeping, throw an error that the system it is running on screwed up
> timekeeping? And a second later when NTP catched up it gets the next
> surprise because the systems CLOCK_REALTIME jumped forward unexpectedly
> or if there is no NTP it's confused forever.
This can happen even with a "perfect" solution that syncs time
instantly on the migration destination. See steps 1,2,3.
Unless you notify applications to invalidate their time reads,
i can't see a way to fix this.
Therefore if you use VM migration in the first place, a certain amount of
timestamp accuracy error must be tolerated.
> How can you even assume that this is correct?
As noted above, even without a window of unsynchronized time (due to
delay for NTP to sync time), time reads can be stale.
> It is exactly the same problem as we had many years ago with hardware
> clocks suddenly stopping to tick which caused quite some stuff to go
> belly up.
Customers complained when it was 5 seconds off, now its 0.1ms (and
people seem happy).
> In a proper suspend/resume scenario CLOCK_REALTIME/TAI are advanced
> (with a certain degree of accuracy) to compensate for the sleep time, so
> the other end of a communication is at least in the same ballpark, but
> not 50 seconds off.
Its 100ms off with migration, and can be reduced further (customers
complained about 5 seconds but seem happy with 0.1ms).
> >> This features first, correctness later frenzy is insane and it better
> >> stops now before you pile even more crap on the existing steaming pile
> >> of insanities.
> >
> > Sure.
>
> I wish that would be true. OS people - you should know that - are
> fighting forever with hardware people over feature madness and the
> attitude of 'we can fix that in software' which turns often enough out
> to be wrong.
>
> Now sadly enough people who suffered from that madness work on
> virtualization and instead of trying to avoid the same problem they go
> off and make it even worse.
So you think its important to reduce the 100ms offset?
> It's the same problem again as with hardware people. Not talking to the
> other people _before_ making uninformed assumptions and decisions.
>
> We did it that way because big customer asked for it is not a
> justification for inflicting this on everybody else and thereby
> violating correctness. Works for me and my big customer is not a proof
> of correctness either.
>
> It's another proof that this industry just "works" by chance.
>
> Thanks,
>
> tglx
OK, makes sense, then reducing the 0.1ms window even further
is a useful thing to do. What would be an acceptable
CLOCK_REALTIME accuracy error, on migration?
next prev parent reply other threads:[~2020-12-11 17:42 UTC|newest]
Thread overview: 77+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-12-03 17:11 [PATCH v2 0/3] RFC: Precise TSC migration Maxim Levitsky
2020-12-03 17:11 ` [PATCH v2 1/3] KVM: x86: implement KVM_{GET|SET}_TSC_STATE Maxim Levitsky
2020-12-06 16:19 ` Thomas Gleixner
2020-12-07 12:16 ` Maxim Levitsky
2020-12-07 13:16 ` Vitaly Kuznetsov
2020-12-07 17:41 ` Thomas Gleixner
2020-12-08 9:48 ` Peter Zijlstra
2020-12-10 11:42 ` Paolo Bonzini
2020-12-10 12:14 ` Peter Zijlstra
2020-12-10 12:22 ` Paolo Bonzini
2020-12-10 13:01 ` Peter Zijlstra
2020-12-10 20:20 ` Thomas Gleixner
2020-12-07 16:38 ` Thomas Gleixner
2020-12-07 16:53 ` Andy Lutomirski
2020-12-07 17:00 ` Maxim Levitsky
2020-12-07 18:04 ` Andy Lutomirski
2020-12-07 23:11 ` Marcelo Tosatti
2020-12-08 17:43 ` Andy Lutomirski
2020-12-08 19:24 ` Thomas Gleixner
2020-12-08 20:32 ` Andy Lutomirski
2020-12-09 0:19 ` Thomas Gleixner
2020-12-09 4:08 ` Andy Lutomirski
2020-12-09 10:14 ` Thomas Gleixner
2020-12-10 23:42 ` Andy Lutomirski
2020-12-08 11:24 ` Maxim Levitsky
2020-12-08 9:35 ` Peter Zijlstra
2020-12-07 23:34 ` Marcelo Tosatti
2020-12-07 17:29 ` Oliver Upton
2020-12-08 11:13 ` Maxim Levitsky
2020-12-08 15:57 ` Oliver Upton
2020-12-08 15:58 ` Oliver Upton
2020-12-08 17:10 ` Maxim Levitsky
2020-12-08 16:40 ` Thomas Gleixner
2020-12-08 17:08 ` Maxim Levitsky
2020-12-10 11:48 ` Paolo Bonzini
2020-12-10 14:25 ` Maxim Levitsky
2020-12-07 23:29 ` Marcelo Tosatti
2020-12-08 14:50 ` Maxim Levitsky
2020-12-08 16:02 ` Thomas Gleixner
2020-12-08 16:25 ` Maxim Levitsky
2020-12-08 17:33 ` Andy Lutomirski
2020-12-08 21:25 ` Thomas Gleixner
2020-12-08 18:12 ` Marcelo Tosatti
2020-12-08 21:35 ` Thomas Gleixner
2020-12-08 21:20 ` Thomas Gleixner
2020-12-10 11:48 ` Paolo Bonzini
2020-12-10 14:52 ` Maxim Levitsky
2020-12-10 15:16 ` Andy Lutomirski
2020-12-10 17:59 ` Oliver Upton
2020-12-10 18:05 ` Paolo Bonzini
2020-12-10 18:13 ` Oliver Upton
2020-12-10 21:25 ` Thomas Gleixner
2020-12-10 22:01 ` Andy Lutomirski
2020-12-10 22:28 ` Thomas Gleixner
2020-12-10 23:19 ` Andy Lutomirski
2020-12-11 0:03 ` Thomas Gleixner
2020-12-08 18:11 ` Marcelo Tosatti
2020-12-08 21:33 ` Thomas Gleixner
2020-12-09 16:34 ` Marcelo Tosatti
2020-12-09 20:58 ` Thomas Gleixner
2020-12-10 15:26 ` Marcelo Tosatti
2020-12-10 21:48 ` Thomas Gleixner
2020-12-11 0:27 ` Marcelo Tosatti
2020-12-11 13:30 ` Thomas Gleixner
2020-12-11 14:18 ` Marcelo Tosatti [this message]
2020-12-11 21:04 ` Thomas Gleixner
2020-12-11 21:59 ` Paolo Bonzini
2020-12-12 13:03 ` Thomas Gleixner
2020-12-15 10:59 ` Marcelo Tosatti
2020-12-15 16:55 ` Andy Lutomirski
2020-12-15 22:34 ` Thomas Gleixner
2020-12-11 13:37 ` Paolo Bonzini
2020-12-08 17:35 ` Marcelo Tosatti
2020-12-03 17:11 ` [PATCH v2 2/3] KVM: x86: introduce KVM_X86_QUIRK_TSC_HOST_ACCESS Maxim Levitsky
2020-12-03 17:11 ` [PATCH v2 3/3] kvm/selftests: update tsc_msrs_test to cover KVM_X86_QUIRK_TSC_HOST_ACCESS Maxim Levitsky
2020-12-07 23:16 ` [PATCH v2 0/3] RFC: Precise TSC migration Marcelo Tosatti
2020-12-10 11:48 ` Paolo Bonzini
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201211141822.GA67764@fuller.cnet \
--to=mtosatti@redhat.com \
--cc=bp@alien8.de \
--cc=corbet@lwn.net \
--cc=drjones@redhat.com \
--cc=hpa@zytor.com \
--cc=jmattson@google.com \
--cc=joro@8bytes.org \
--cc=kvm@vger.kernel.org \
--cc=linux-doc@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=mingo@redhat.com \
--cc=mlevitsk@redhat.com \
--cc=oupton@google.com \
--cc=pbonzini@redhat.com \
--cc=sean.j.christopherson@intel.com \
--cc=shuah@kernel.org \
--cc=tglx@linutronix.de \
--cc=vkuznets@redhat.com \
--cc=wanpengli@tencent.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).