From: Andy Lutomirski <luto@kernel.org>
To: x86@kernel.org
Cc: One Thousand Gnomes <gnomes@lxorguk.ukuu.org.uk>,
Borislav Petkov <bp@alien8.de>,
"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
Brian Gerst <brgerst@gmail.com>,
Matthew Whitehead <tedheadster@gmail.com>,
Henrique de Moraes Holschuh <hmh@hmh.eng.br>,
Peter Zijlstra <peterz@infradead.org>,
xen-devel <Xen-devel@lists.xen.org>,
Juergen Gross <jgross@suse.com>,
Boris Ostrovsky <boris.ostrovsky@oracle.com>,
Andrew Cooper <andrew.cooper3@citrix.com>,
Andy Lutomirski <luto@kernel.org>,
"H. Peter Anvin" <hpa@zytor.com>
Subject: [PATCH v3 4/4] x86/asm: Rewrite sync_core() to use IRET-to-self
Date: Mon, 5 Dec 2016 13:32:43 -0800 [thread overview]
Message-ID: <c452beb3aaa5a0232b5ef27f443f0c27b6c65fe2.1480973468.git.luto@kernel.org> (raw)
In-Reply-To: <cover.1480973468.git.luto@kernel.org>
In-Reply-To: <cover.1480973468.git.luto@kernel.org>
Aside from being excessively slow, CPUID is problematic: Linux runs
on a handful of CPUs that don't have CPUID. Use IRET-to-self
instead. IRET-to-self works everywhere, so it makes testing easy.
For reference, On my laptop, IRET-to-self is ~110ns,
CPUID(eax=1, ecx=0) is ~83ns on native and very very slow under KVM,
and MOV-to-CR2 is ~42ns.
While we're at it: sync_core() serves a very specific purpose.
Document it.
Cc: "H. Peter Anvin" <hpa@zytor.com>
Signed-off-by: Andy Lutomirski <luto@kernel.org>
---
arch/x86/include/asm/processor.h | 77 ++++++++++++++++++++++++++++------------
1 file changed, 55 insertions(+), 22 deletions(-)
diff --git a/arch/x86/include/asm/processor.h b/arch/x86/include/asm/processor.h
index 64fbc937d586..201a956e345f 100644
--- a/arch/x86/include/asm/processor.h
+++ b/arch/x86/include/asm/processor.h
@@ -590,33 +590,66 @@ static __always_inline void cpu_relax(void)
#define cpu_relax_lowlatency() cpu_relax()
-/* Stop speculative execution and prefetching of modified code. */
+/*
+ * This function forces the icache and prefetched instruction stream to
+ * catch up with reality in two very specific cases:
+ *
+ * a) Text was modified using one virtual address and is about to be executed
+ * from the same physical page at a different virtual address.
+ *
+ * b) Text was modified on a different CPU, may subsequently be
+ * executed on this CPU, and you want to make sure the new version
+ * gets executed. This generally means you're calling this in a IPI.
+ *
+ * If you're calling this for a different reason, you're probably doing
+ * it wrong.
+ */
static inline void sync_core(void)
{
- int tmp;
-
-#ifdef CONFIG_X86_32
/*
- * Do a CPUID if available, otherwise do a jump. The jump
- * can conveniently enough be the jump around CPUID.
+ * There are quite a few ways to do this. IRET-to-self is nice
+ * because it works on every CPU, at any CPL (so it's compatible
+ * with paravirtualization), and it never exits to a hypervisor.
+ * The only down sides are that it's a bit slow (it seems to be
+ * a bit more than 2x slower than the fastest options) and that
+ * it unmasks NMIs. The "push %cs" is needed because, in
+ * paravirtual environments, __KERNEL_CS may not be a valid CS
+ * value when we do IRET directly.
+ *
+ * In case NMI unmasking or performance every becomes a problem,
+ * the next best option appears to be MOV-to-CR2 and an
+ * unconditional jump. That sequence also works on all CPUs,
+ * but it will fault at CPL3.
+ *
+ * CPUID is the conventional way, but it's nasty: it doesn't
+ * exist on some 486-like CPUs, and it usually exits to a
+ * hypervisor.
*/
- asm volatile("cmpl %2,%1\n\t"
- "jl 1f\n\t"
- "cpuid\n"
- "1:"
- : "=a" (tmp)
- : "rm" (boot_cpu_data.cpuid_level), "ri" (0), "0" (1)
- : "ebx", "ecx", "edx", "memory");
+ register void *__sp asm(_ASM_SP);
+
+#ifdef CONFIG_X86_32
+ asm volatile (
+ "pushfl\n\t"
+ "pushl %%cs\n\t"
+ "pushl $1f\n\t"
+ "iret\n\t"
+ "1:"
+ : "+r" (__sp) : : "cc", "memory");
#else
- /*
- * CPUID is a barrier to speculative execution.
- * Prefetched instructions are automatically
- * invalidated when modified.
- */
- asm volatile("cpuid"
- : "=a" (tmp)
- : "0" (1)
- : "ebx", "ecx", "edx", "memory");
+ unsigned long tmp;
+
+ asm volatile (
+ "movq %%ss, %0\n\t"
+ "pushq %0\n\t"
+ "pushq %%rsp\n\t"
+ "addq $8, (%%rsp)\n\t"
+ "pushfq\n\t"
+ "movq %%cs, %0\n\t"
+ "pushq %0\n\t"
+ "pushq $1f\n\t"
+ "iretq\n\t"
+ "1:"
+ : "=r" (tmp), "+r" (__sp) : : "cc", "memory");
#endif
}
--
2.9.3
next prev parent reply other threads:[~2016-12-05 21:33 UTC|newest]
Thread overview: 11+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-12-05 21:32 [PATCH v3 0/4] CPUID-less CPU/sync_core fixes and improvements Andy Lutomirski
2016-12-05 21:32 ` [PATCH v3 1/4] x86/asm/32: Make sync_core() handle missing CPUID on all 32-bit kernels Andy Lutomirski
2016-12-05 21:32 ` [PATCH v3 2/4] Revert "x86/boot: Fail the boot if !M486 and CPUID is missing" Andy Lutomirski
2016-12-05 21:32 ` [PATCH v3 3/4] x86/microcode/intel: Replace sync_core() with native_cpuid() Andy Lutomirski
2016-12-05 21:32 ` Andy Lutomirski [this message]
2016-12-06 7:52 ` [PATCH v3 4/4] x86/asm: Rewrite sync_core() to use IRET-to-self Borislav Petkov
2016-12-06 17:49 ` Andy Lutomirski
2016-12-06 8:46 ` [Xen-devel] " Jan Beulich
2016-12-06 9:25 ` Peter Zijlstra
[not found] ` <584697BA0200007800125855@prv-mh.provo.novell.com>
2016-12-06 17:46 ` Andy Lutomirski
2016-12-06 19:32 ` H. Peter Anvin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=c452beb3aaa5a0232b5ef27f443f0c27b6c65fe2.1480973468.git.luto@kernel.org \
--to=luto@kernel.org \
--cc=Xen-devel@lists.xen.org \
--cc=andrew.cooper3@citrix.com \
--cc=boris.ostrovsky@oracle.com \
--cc=bp@alien8.de \
--cc=brgerst@gmail.com \
--cc=gnomes@lxorguk.ukuu.org.uk \
--cc=hmh@hmh.eng.br \
--cc=hpa@zytor.com \
--cc=jgross@suse.com \
--cc=linux-kernel@vger.kernel.org \
--cc=peterz@infradead.org \
--cc=tedheadster@gmail.com \
--cc=x86@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).