Re: [PATCH v2 3/3] intel_idle: add C0.2 state for Sapphire Rapids Xeon

From: Peter Zijlstra <peterz@infradead.org>
To: Andy Lutomirski <luto@kernel.org>
Cc: Artem Bityutskiy <dedekind1@gmail.com>,
	the arch/x86 maintainers <x86@kernel.org>,
	Linux PM Mailing List <linux-pm@vger.kernel.org>,
	"luto@amacapital.net" <luto@amacapital.net>
Subject: Re: [PATCH v2 3/3] intel_idle: add C0.2 state for Sapphire Rapids Xeon
Date: Wed, 22 Mar 2023 11:18:08 +0100	[thread overview]
Message-ID: <20230322101808.GU2017917@hirez.programming.kicks-ass.net> (raw)
In-Reply-To: <05fdea5a-bfad-4d7c-9925-9a45b2882782@app.fastmail.com>

On Mon, Mar 20, 2023 at 11:27:54AM -0700, Andy Lutomirski wrote:

> This is all busted.

Well, yes.

> UMWAIT has a U for *user mode*.  We have the MSR set to a small value
> because USER waits are a big can of worms, and long user waits don't
> actually behave in any particularly intelligent manner unless that
> core is dedicated to just one user task, and no virt is involved, and
> the user code involved is extremely careful.

Idem for virt. [U]MWAIT only really works for virt when the CPU is
dedicated to the one (vcpu) task.

> But now UMWAIT got extended in a way to make it useful for the kernel,
> but it's controlled by the same MSR.  And this is busted.  What we
> want is for CPL0 UMWAIT to ignore the MSR or use a different MSR (for
> virt, sigh, except that this whole mechanism is presuambly still
> useless on virt).  Or for a different instruction to be used from the
> kernel, maybe spelled MWAIT.

Yes, CPL0 usage should not be subject to the same limit. I'm not sure if
there's a good argument to have a different limit on virt vs random
other userspace.

> Also, this series needs to be tested on virt.  Because UMWAIT, if it
> works at all on virt, is going to have all manner of odd concequences
> due to the fact that the hypervisor hasn't the faintest clue what's
> going on because there's no feedback.  For all that UiPI is nasty and
> half-baked, at least it tries to notify the next privilege level up as
> to what's going on.  Explicit wakeups virtualize much better than
> cacheline monitors.

Virt is supposedly a big trumpet case for UMWAIT because VMMs
(rightfully) restrict MWAIT. At the same time, you *REALLY* do not want
your vcpu task doing long UMWAITs when there's other vcpu threads
waiting to do real work, so the MSR value *SHOULD* be really low.

Increasing this value randomly is bad.. increasing it beyond 1 tick is
abysmal.

Unless dedicated vcpu:cpu relations, in which case kvm already should be
exposing MWAIT in any case.