linux-pm.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Artem Bityutskiy <dedekind1@gmail.com>
To: x86@kernel.org, "Rafael J. Wysocki" <rafael@kernel.org>
Cc: Linux PM Mailing List <linux-pm@vger.kernel.org>,
	Arjan van de Ven <arjan@linux.intel.com>,
	Artem Bityutskiy <dedekind1@gmail.com>,
	Thomas Gleixner <tglx@linutronix.de>
Subject: [PATCH v4 2/4] x86/mwait: Add support for idle via umwait
Date: Mon, 10 Jul 2023 12:30:58 +0300	[thread overview]
Message-ID: <20230710093100.918337-3-dedekind1@gmail.com> (raw)
In-Reply-To: <20230710093100.918337-1-dedekind1@gmail.com>

From: Artem Bityutskiy <artem.bityutskiy@linux.intel.com>

On Intel platforms, C-states are requested using the 'monitor/mwait'
instructions pair, as implemented in 'mwait_idle_with_hints()'. This
mechanism allows for entering C1 and deeper C-states.

Sapphire Rapids Xeon supports new idle states - C0.1 and C0.2 (later
C0.x). These idle states have lower latency comparing to C1, and can be
requested with either 'tpause' or 'umwait' instructions.

Linux kernel already supports the 'tpause' instruction and uses it in
delay functions like 'udelay()'. Add 'umwait' support by implementing
the 'umwait_idle()' function. This function is analogous to
'mwait_idle_with_hints()', but instead of requesting a C-state with
'monitor/mwait', it requests C0.x with 'umonitor/umwait'.

Tested with both gcc/binutils and clang/llvm.

Signed-off-by: Artem Bityutskiy <artem.bityutskiy@linux.intel.com>
---
 arch/x86/include/asm/mwait.h | 67 ++++++++++++++++++++++++++++++++++++
 1 file changed, 67 insertions(+)

diff --git a/arch/x86/include/asm/mwait.h b/arch/x86/include/asm/mwait.h
index 03bef2bc28d4..48210f4d7c77 100644
--- a/arch/x86/include/asm/mwait.h
+++ b/arch/x86/include/asm/mwait.h
@@ -141,4 +141,71 @@ static inline void __tpause(u32 ecx, u32 edx, u32 eax)
 	}
 }
 
+#ifdef CONFIG_X86_64
+/*
+ * Monitor a memory address at 'rcx' using the 'umonitor' instruction.
+ */
+static __always_inline void __umonitor(const void *rcx)
+{
+	/* "umonitor %rcx" */
+	if (IS_ENABLED(CONFIG_AS_TPAUSE)) {
+		asm volatile("umonitor %%rcx\n"
+			     :
+			     : "c"(rcx));
+	} else {
+		asm volatile(".byte 0xf3, 0x0f, 0xae, 0xf1\t\n"
+			     :
+			     : "c"(rcx));
+	}
+}
+
+/*
+ * Same as '__tpause()', but uses the 'umwait' instruction. It is very
+ * similar to 'tpause', but also breaks out if the data at the address
+ * monitored with 'umonitor' is modified.
+ */
+static __always_inline void __umwait(u32 ecx, u32 edx, u32 eax)
+{
+	/* "umwait %ecx, %edx, %eax;" */
+	if (IS_ENABLED(CONFIG_AS_TPAUSE)) {
+		asm volatile("umwait %%ecx\n"
+			     :
+			     : "c"(ecx), "d"(edx), "a"(eax));
+	} else {
+		asm volatile(".byte 0xf2, 0x0f, 0xae, 0xf1\t\n"
+			     :
+			     : "c"(ecx), "d"(edx), "a"(eax));
+	}
+}
+
+/*
+ * Enter C0.1 or C0.2 state and stay there until an event happens (an interrupt
+ * or 'need_resched()'), the explicit deadline is reached, or the implicit
+ * global limit is reached.
+ *
+ * The deadline is the absolute TSC value to exit the idle state at. If it
+ * exceeds the global limit in the 'IA32_UMWAIT_CONTROL' register, the global
+ * limit prevails, and the idle state is exited earlier than the deadline.
+ */
+static __always_inline void umwait_idle(u64 deadline, u32 state)
+{
+	if (!current_set_polling_and_test()) {
+		u32 eax, edx;
+
+		eax = lower_32_bits(deadline);
+		edx = upper_32_bits(deadline);
+
+		__umonitor(&current_thread_info()->flags);
+		if (!need_resched())
+			__umwait(state, edx, eax);
+	}
+	current_clr_polling();
+}
+#else /* CONFIG_X86_64 */
+static __always_inline void umwait_idle(u64 deadline, u32 state)
+{
+	WARN_ONCE(1, "umwait CPU instruction is not supported");
+}
+#endif /* !CONFIG_X86_64 */
+
 #endif /* _ASM_X86_MWAIT_H */
-- 
2.40.1


  parent reply	other threads:[~2023-07-10  9:33 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2023-07-10  9:30 [PATCH v4 0/4] Sapphire Rapids C0.x idle states support Artem Bityutskiy
2023-07-10  9:30 ` [PATCH v4 1/4] x86/umwait: use 'IS_ENABLED()' Artem Bityutskiy
2023-07-10  9:30 ` Artem Bityutskiy [this message]
2023-07-10  9:30 ` [PATCH v4 3/4] intel_idle: rename 'intel_idle_hlt_irq_on()' Artem Bityutskiy
2023-07-14 15:34   ` Rafael J. Wysocki
2023-07-14 15:39     ` Arjan van de Ven
2023-07-14 18:11     ` Artem Bityutskiy
2023-07-14 21:01     ` Peter Zijlstra
2023-07-14 21:02       ` Arjan van de Ven
2023-07-14 21:12         ` Peter Zijlstra
2023-07-10  9:31 ` [PATCH v4 4/4] intel_idle: add C0.2 state for Sapphire Rapids Xeon Artem Bityutskiy
2023-07-20 18:35 ` [PATCH v4 0/4] Sapphire Rapids C0.x idle states support Rafael J. Wysocki
2023-08-28 16:43 ` Artem Bityutskiy
2023-09-13 11:37   ` Artem Bityutskiy
2023-09-13 12:34     ` Rafael J. Wysocki
2023-09-13 12:49       ` Artem Bityutskiy
2023-09-13 12:55         ` Rafael J. Wysocki

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20230710093100.918337-3-dedekind1@gmail.com \
    --to=dedekind1@gmail.com \
    --cc=arjan@linux.intel.com \
    --cc=linux-pm@vger.kernel.org \
    --cc=rafael@kernel.org \
    --cc=tglx@linutronix.de \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).