LKML Archive on lore.kernel.org
 help / color / Atom feed
From: "tip-bot2 for Arvind Sankar" <tip-bot2@linutronix.de>
To: linux-tip-commits@vger.kernel.org
Cc: Arvind Sankar <nivedita@alum.mit.edu>,
	Borislav Petkov <bp@suse.de>, Kees Cook <keescook@chromium.org>,
	Miguel Ojeda <miguel.ojeda.sandonis@gmail.com>,
	Nathan Chancellor <natechancellor@gmail.com>,
	Sedat Dilek <sedat.dilek@gmail.com>, x86 <x86@kernel.org>,
	LKML <linux-kernel@vger.kernel.org>
Subject: [tip: x86/asm] x86/asm: Replace __force_order with a memory clobber
Date: Tue, 13 Oct 2020 09:30:52 -0000
Message-ID: <160258145262.7002.8053536940451102515.tip-bot2@tip-bot2> (raw)
In-Reply-To: <20200902232152.3709896-1-nivedita@alum.mit.edu>

The following commit has been merged into the x86/asm branch of tip:

Commit-ID:     3e626682046e30282979f7d71e054cd4c00069a7
Gitweb:        https://git.kernel.org/tip/3e626682046e30282979f7d71e054cd4c00069a7
Author:        Arvind Sankar <nivedita@alum.mit.edu>
AuthorDate:    Wed, 02 Sep 2020 19:21:52 -04:00
Committer:     Borislav Petkov <bp@suse.de>
CommitterDate: Tue, 13 Oct 2020 11:23:15 +02:00

x86/asm: Replace __force_order with a memory clobber

The CRn accessor functions use __force_order as a dummy operand to
prevent the compiler from reordering CRn reads/writes with respect to
each other.

The fact that the asm is volatile should be enough to prevent this:
volatile asm statements should be executed in program order. However GCC
4.9.x and 5.x have a bug that might result in reordering. This was fixed
in 8.1, 7.3 and 6.5. Versions prior to these, including 5.x and 4.9.x,
may reorder volatile asm statements with respect to each other.

There are some issues with __force_order as implemented:
- It is used only as an input operand for the write functions, and hence
  doesn't do anything additional to prevent reordering writes.
- It allows memory accesses to be cached/reordered across write
  functions, but CRn writes affect the semantics of memory accesses, so
  this could be dangerous.
- __force_order is not actually defined in the kernel proper, but the
  LLVM toolchain can in some cases require a definition: LLVM (as well
  as GCC 4.9) requires it for PIE code, which is why the compressed
  kernel has a definition, but also the clang integrated assembler may
  consider the address of __force_order to be significant, resulting in
  a reference that requires a definition.

Fix this by:
- Using a memory clobber for the write functions to additionally prevent
  caching/reordering memory accesses across CRn writes.
- Using a dummy input operand with an arbitrary constant address for the
  read functions, instead of a global variable. This will prevent reads
  from being reordered across writes, while allowing memory loads to be
  cached/reordered across CRn reads, which should be safe.

Signed-off-by: Arvind Sankar <nivedita@alum.mit.edu>
Signed-off-by: Borislav Petkov <bp@suse.de>
Reviewed-by: Kees Cook <keescook@chromium.org>
Reviewed-by: Miguel Ojeda <miguel.ojeda.sandonis@gmail.com>
Tested-by: Nathan Chancellor <natechancellor@gmail.com>
Tested-by: Sedat Dilek <sedat.dilek@gmail.com>
Link: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=82602
Link: https://lore.kernel.org/lkml/20200527135329.1172644-1-arnd@arndb.de/
Link: https://lkml.kernel.org/r/20200902232152.3709896-1-nivedita@alum.mit.edu
---
 arch/x86/boot/compressed/pgtable_64.c |  9 +--------
 arch/x86/include/asm/special_insns.h  | 28 +++++++++++++-------------
 arch/x86/kernel/cpu/common.c          |  4 ++--
 3 files changed, 17 insertions(+), 24 deletions(-)

diff --git a/arch/x86/boot/compressed/pgtable_64.c b/arch/x86/boot/compressed/pgtable_64.c
index c886269..7d0394f 100644
--- a/arch/x86/boot/compressed/pgtable_64.c
+++ b/arch/x86/boot/compressed/pgtable_64.c
@@ -5,15 +5,6 @@
 #include "pgtable.h"
 #include "../string.h"
 
-/*
- * __force_order is used by special_insns.h asm code to force instruction
- * serialization.
- *
- * It is not referenced from the code, but GCC < 5 with -fPIE would fail
- * due to an undefined symbol. Define it to make these ancient GCCs work.
- */
-unsigned long __force_order;
-
 #define BIOS_START_MIN		0x20000U	/* 128K, less than this is insane */
 #define BIOS_START_MAX		0x9f000U	/* 640K, absolute maximum */
 
diff --git a/arch/x86/include/asm/special_insns.h b/arch/x86/include/asm/special_insns.h
index 59a3e13..d6e3bb9 100644
--- a/arch/x86/include/asm/special_insns.h
+++ b/arch/x86/include/asm/special_insns.h
@@ -11,45 +11,47 @@
 #include <linux/jump_label.h>
 
 /*
- * Volatile isn't enough to prevent the compiler from reordering the
- * read/write functions for the control registers and messing everything up.
- * A memory clobber would solve the problem, but would prevent reordering of
- * all loads stores around it, which can hurt performance. Solution is to
- * use a variable and mimic reads and writes to it to enforce serialization
+ * The compiler should not reorder volatile asm statements with respect to each
+ * other: they should execute in program order. However GCC 4.9.x and 5.x have
+ * a bug (which was fixed in 8.1, 7.3 and 6.5) where they might reorder
+ * volatile asm. The write functions are not affected since they have memory
+ * clobbers preventing reordering. To prevent reads from being reordered with
+ * respect to writes, use a dummy memory operand.
  */
-extern unsigned long __force_order;
+
+#define __FORCE_ORDER "m"(*(unsigned int *)0x1000UL)
 
 void native_write_cr0(unsigned long val);
 
 static inline unsigned long native_read_cr0(void)
 {
 	unsigned long val;
-	asm volatile("mov %%cr0,%0\n\t" : "=r" (val), "=m" (__force_order));
+	asm volatile("mov %%cr0,%0\n\t" : "=r" (val) : __FORCE_ORDER);
 	return val;
 }
 
 static __always_inline unsigned long native_read_cr2(void)
 {
 	unsigned long val;
-	asm volatile("mov %%cr2,%0\n\t" : "=r" (val), "=m" (__force_order));
+	asm volatile("mov %%cr2,%0\n\t" : "=r" (val) : __FORCE_ORDER);
 	return val;
 }
 
 static __always_inline void native_write_cr2(unsigned long val)
 {
-	asm volatile("mov %0,%%cr2": : "r" (val), "m" (__force_order));
+	asm volatile("mov %0,%%cr2": : "r" (val) : "memory");
 }
 
 static inline unsigned long __native_read_cr3(void)
 {
 	unsigned long val;
-	asm volatile("mov %%cr3,%0\n\t" : "=r" (val), "=m" (__force_order));
+	asm volatile("mov %%cr3,%0\n\t" : "=r" (val) : __FORCE_ORDER);
 	return val;
 }
 
 static inline void native_write_cr3(unsigned long val)
 {
-	asm volatile("mov %0,%%cr3": : "r" (val), "m" (__force_order));
+	asm volatile("mov %0,%%cr3": : "r" (val) : "memory");
 }
 
 static inline unsigned long native_read_cr4(void)
@@ -64,10 +66,10 @@ static inline unsigned long native_read_cr4(void)
 	asm volatile("1: mov %%cr4, %0\n"
 		     "2:\n"
 		     _ASM_EXTABLE(1b, 2b)
-		     : "=r" (val), "=m" (__force_order) : "0" (0));
+		     : "=r" (val) : "0" (0), __FORCE_ORDER);
 #else
 	/* CR4 always exists on x86_64. */
-	asm volatile("mov %%cr4,%0\n\t" : "=r" (val), "=m" (__force_order));
+	asm volatile("mov %%cr4,%0\n\t" : "=r" (val) : __FORCE_ORDER);
 #endif
 	return val;
 }
diff --git a/arch/x86/kernel/cpu/common.c b/arch/x86/kernel/cpu/common.c
index c5d6f17..178499f 100644
--- a/arch/x86/kernel/cpu/common.c
+++ b/arch/x86/kernel/cpu/common.c
@@ -359,7 +359,7 @@ void native_write_cr0(unsigned long val)
 	unsigned long bits_missing = 0;
 
 set_register:
-	asm volatile("mov %0,%%cr0": "+r" (val), "+m" (__force_order));
+	asm volatile("mov %0,%%cr0": "+r" (val) : : "memory");
 
 	if (static_branch_likely(&cr_pinning)) {
 		if (unlikely((val & X86_CR0_WP) != X86_CR0_WP)) {
@@ -378,7 +378,7 @@ void native_write_cr4(unsigned long val)
 	unsigned long bits_changed = 0;
 
 set_register:
-	asm volatile("mov %0,%%cr4": "+r" (val), "+m" (cr4_pinned_bits));
+	asm volatile("mov %0,%%cr4": "+r" (val) : : "memory");
 
 	if (static_branch_likely(&cr_pinning)) {
 		if (unlikely((val & cr4_pinned_mask) != cr4_pinned_bits)) {

  parent reply index

Thread overview: 63+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-05-27 13:53 [PATCH] x86: work around clang IAS bug referencing __force_order Arnd Bergmann
2020-08-01 11:50 ` Sedat Dilek
2020-08-06 22:13   ` Thomas Gleixner
2020-08-07  7:03     ` Sedat Dilek
2020-08-04  0:09 ` Nick Desaulniers
2020-08-14 17:29   ` Sedat Dilek
2020-08-14 21:19     ` Sedat Dilek
2020-08-14 22:57       ` Nick Desaulniers
2020-08-15  0:26         ` Nick Desaulniers
2020-08-15  3:28           ` Sedat Dilek
2020-08-15  8:23             ` Sedat Dilek
2020-08-15 10:46               ` Sedat Dilek
2020-08-15 14:39                 ` Sedat Dilek
2020-08-16  9:37                   ` Sedat Dilek
2020-08-06 22:11 ` Thomas Gleixner
2020-08-13  0:12   ` Nick Desaulniers
2020-08-13  8:49     ` David Laight
2020-08-13 17:20     ` Arvind Sankar
2020-08-13 17:28     ` Thomas Gleixner
2020-08-13 17:37       ` Paul E. McKenney
2020-08-13 18:09         ` Arvind Sankar
2020-08-13 18:20           ` Paul E. McKenney
2020-08-20 10:44           ` Thomas Gleixner
2020-08-20 13:06             ` Arvind Sankar
2020-08-21  0:37               ` Thomas Gleixner
2020-08-21 23:04                 ` Arvind Sankar
2020-08-21 23:16                   ` Nick Desaulniers
2020-08-21 23:25                     ` Arvind Sankar
2020-08-22  0:43                     ` Thomas Gleixner
2020-08-22  3:55                       ` Arvind Sankar
2020-08-22  8:41                         ` Segher Boessenkool
2020-08-22  9:23                           ` Sedat Dilek
2020-08-22  9:51                             ` Sedat Dilek
2020-08-22 10:26                               ` Segher Boessenkool
2020-08-22 10:35                                 ` Arnd Bergmann
2020-08-22 18:17                               ` Miguel Ojeda
2020-08-22 21:08                                 ` Linus Torvalds
2020-08-22 23:10                                   ` Arvind Sankar
2020-08-23  0:10                                     ` Linus Torvalds
2020-08-23  1:16                                       ` Arvind Sankar
2020-08-23 21:25                                         ` [PATCH] x86/asm: Replace __force_order with memory clobber Arvind Sankar
2020-08-24 17:50                                           ` Nathan Chancellor
2020-08-24 19:13                                           ` Miguel Ojeda
2020-08-25 15:19                                             ` Arvind Sankar
2020-08-25 15:21                                               ` Sedat Dilek
2020-09-02 15:33                                           ` [PATCH v2] " Arvind Sankar
2020-09-02 15:58                                             ` David Laight
2020-09-02 16:14                                               ` Arvind Sankar
2020-09-02 16:08                                             ` Arvind Sankar
2020-09-02 20:26                                               ` David Laight
2020-09-02 17:16                                             ` Segher Boessenkool
2020-09-02 17:36                                               ` Arvind Sankar
2020-09-02 18:19                                             ` Miguel Ojeda
2020-09-02 18:24                                               ` Arvind Sankar
2020-09-02 23:21                                           ` [PATCH v3] " Arvind Sankar
2020-09-03  2:17                                             ` Kees Cook
2020-09-03  5:34                                             ` Miguel Ojeda
2020-09-30 20:50                                             ` Kees Cook
2020-10-01 10:12                                             ` [tip: x86/asm] x86/asm: Replace __force_order with a " tip-bot2 for Arvind Sankar
2020-10-13  9:30                                             ` tip-bot2 for Arvind Sankar [this message]
2020-08-22 21:17                                 ` [PATCH] x86: work around clang IAS bug referencing __force_order Arvind Sankar
2020-08-23 13:31                                   ` David Laight
2020-09-08 22:25                               ` Pavel Machek

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=160258145262.7002.8053536940451102515.tip-bot2@tip-bot2 \
    --to=tip-bot2@linutronix.de \
    --cc=bp@suse.de \
    --cc=keescook@chromium.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-tip-commits@vger.kernel.org \
    --cc=miguel.ojeda.sandonis@gmail.com \
    --cc=natechancellor@gmail.com \
    --cc=nivedita@alum.mit.edu \
    --cc=sedat.dilek@gmail.com \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git
	git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git
	git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git
	git clone --mirror https://lore.kernel.org/lkml/9 lkml/git/9.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org
	public-inbox-index lkml

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git