Re: [PATCH v2 1/3] KVM: x86: implement KVM_{GET|SET}_TSC_STATE

From: Maxim Levitsky <mlevitsk@redhat.com>
To: Andy Lutomirski <luto@amacapital.net>,
	Thomas Gleixner <tglx@linutronix.de>
Cc: kvm@vger.kernel.org, "H. Peter Anvin" <hpa@zytor.com>,
	Paolo Bonzini <pbonzini@redhat.com>,
	Jonathan Corbet <corbet@lwn.net>,
	Jim Mattson <jmattson@google.com>,
	Wanpeng Li <wanpengli@tencent.com>,
	"open list:KERNEL SELFTEST FRAMEWORK" 
	<linux-kselftest@vger.kernel.org>,
	Vitaly Kuznetsov <vkuznets@redhat.com>,
	Marcelo Tosatti <mtosatti@redhat.com>,
	Sean Christopherson <sean.j.christopherson@intel.com>,
	open list <linux-kernel@vger.kernel.org>,
	Ingo Molnar <mingo@redhat.com>,
	"maintainer:X86 ARCHITECTURE (32-BIT AND 64-BIT)"
	<x86@kernel.org>, Joerg Roedel <joro@8bytes.org>,
	Borislav Petkov <bp@alien8.de>, Shuah Khan <shuah@kernel.org>,
	Andrew Jones <drjones@redhat.com>,
	Oliver Upton <oupton@google.com>,
	"open list:DOCUMENTATION" <linux-doc@vger.kernel.org>
Subject: Re: [PATCH v2 1/3] KVM: x86: implement KVM_{GET|SET}_TSC_STATE
Date: Mon, 07 Dec 2020 19:00:01 +0200	[thread overview]
Message-ID: <636fecc20b0143128b484f159ff795ff65d05b82.camel@redhat.com> (raw)
In-Reply-To: <905DFDCE-71A5-4711-A31B-9FCFEA2CFC52@amacapital.net>

On Mon, 2020-12-07 at 08:53 -0800, Andy Lutomirski wrote:
> > On Dec 7, 2020, at 8:38 AM, Thomas Gleixner <tglx@linutronix.de> wrote:
> > 
> > On Mon, Dec 07 2020 at 14:16, Maxim Levitsky wrote:
> > > > On Sun, 2020-12-06 at 17:19 +0100, Thomas Gleixner wrote:
> > > > From a timekeeping POV and the guests expectation of TSC this is
> > > > fundamentally wrong:
> > > > 
> > > >      tscguest = scaled(hosttsc) + offset
> > > > 
> > > > The TSC has to be viewed systemwide and not per CPU. It's systemwide
> > > > used for timekeeping and for that to work it has to be synchronized. 
> > > > 
> > > > Why would this be different on virt? Just because it's virt or what? 
> > > > 
> > > > Migration is a guest wide thing and you're not migrating single vCPUs.
> > > > 
> > > > This hackery just papers over he underlying design fail that KVM looks
> > > > at the TSC per vCPU which is the root cause and that needs to be fixed.
> > > 
> > > I don't disagree with you.
> > > As far as I know the main reasons that kvm tracks TSC per guest are
> > > 
> > > 1. cases when host tsc is not stable 
> > > (hopefully rare now, and I don't mind making
> > > the new API just refuse to work when this is detected, and revert to old way
> > > of doing things).
> > 
> > That's a trainwreck to begin with and I really would just not support it
> > for anything new which aims to be more precise and correct.  TSC has
> > become pretty reliable over the years.
> > 
> > > 2. (theoretical) ability of the guest to introduce per core tsc offfset
> > > by either using TSC_ADJUST (for which I got recently an idea to stop
> > > advertising this feature to the guest), or writing TSC directly which
> > > is allowed by Intel's PRM:
> > 
> > For anything halfways modern the write to TSC is reflected in TSC_ADJUST
> > which means you get the precise offset.
> > 
> > The general principle still applies from a system POV.
> > 
> >     TSC base (systemwide view) - The sane case
> > 
> >     TSC CPU  = TSC base + TSC_ADJUST
> > 
> > The guest TSC base is a per guest constant offset to the host TSC.
> > 
> >     TSC guest base = TSC host base + guest base offset
> > 
> > If the guest want's this different per vCPU by writing to the MSR or to
> > TSC_ADJUST then you still can have a per vCPU offset in TSC_ADJUST which
> > is the offset to the TSC base of the guest.
> 
> How about, if the guest wants to write TSC_ADJUST, it can turn off all paravirt features and keep both pieces?
> 

This is one of the things I had in mind recently.

Even better, we can stop advertising TSC_ADJUST in CPUID to the guest 
and forbid it from writing it at all.

And if the guest insists and writes to the TSC itself, 
then indeed let it keep both pieces (or invoke some failback code).

I have nothing against this solution.

Best regards,
	Maxim Levitsky