LKML Archive on lore.kernel.org
 help / color / Atom feed
From: Will Deacon <will@kernel.org>
To: linux-kernel@vger.kernel.org
Cc: linux-arch@vger.kernel.org, kernel-team@android.com,
	Will Deacon <will@kernel.org>,
	Michael Ellerman <mpe@ellerman.id.au>,
	Peter Zijlstra <peterz@infradead.org>,
	Linus Torvalds <torvalds@linux-foundation.org>,
	Segher Boessenkool <segher@kernel.crashing.org>,
	Christian Borntraeger <borntraeger@de.ibm.com>,
	Luc Van Oostenryck <luc.vanoostenryck@gmail.com>,
	Arnd Bergmann <arnd@arndb.de>,
	Peter Oberparleiter <oberpar@linux.ibm.com>,
	Masahiro Yamada <masahiroy@kernel.org>,
	Nick Desaulniers <ndesaulniers@google.com>
Subject: [PATCH v2 06/10] READ_ONCE: Drop pointer qualifiers when reading from scalar types
Date: Thu, 23 Jan 2020 15:33:37 +0000
Message-ID: <20200123153341.19947-7-will@kernel.org> (raw)
In-Reply-To: <20200123153341.19947-1-will@kernel.org>

Passing a volatile-qualified pointer to READ_ONCE() is an absolute
trainwreck for code generation: the use of 'typeof()' to define a
temporary variable inside the macro means that the final evaluation in
macro scope ends up forcing a read back from the stack. When stack
protector is enabled (the default for arm64, at least), this causes
the compiler to vomit up all sorts of junk.

Unfortunately, dropping pointer qualifiers inside the macro poses quite
a challenge, especially since the pointed-to type is permitted to be an
aggregate, and this is relied upon by mm/ code accessing things like
'pmd_t'. Based on numerous hacks and discussions on the mailing list,
this is the best I've managed to come up with.

Introduce '__unqual_scalar_typeof()' which takes an expression and, if
the expression is an optionally qualified 8, 16, 32 or 64-bit scalar
type, evaluates to the unqualified type. Other input types, including
aggregates, remain unchanged. Hopefully READ_ONCE() on volatile aggregate
pointers isn't something we do on a fast-path.

Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Arnd Bergmann <arnd@arndb.de>
Suggested-by: Linus Torvalds <torvalds@linux-foundation.org>
Reported-by: Michael Ellerman <mpe@ellerman.id.au>
Signed-off-by: Will Deacon <will@kernel.org>
---
 include/linux/compiler.h       |  6 +++---
 include/linux/compiler_types.h | 21 +++++++++++++++++++++
 2 files changed, 24 insertions(+), 3 deletions(-)

diff --git a/include/linux/compiler.h b/include/linux/compiler.h
index a7b2195f2655..994c35638584 100644
--- a/include/linux/compiler.h
+++ b/include/linux/compiler.h
@@ -203,13 +203,13 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
  * atomicity or dependency ordering guarantees. Note that this may result
  * in tears!
  */
-#define __READ_ONCE(x)	(*(const volatile typeof(x) *)&(x))
+#define __READ_ONCE(x)	(*(const volatile __unqual_scalar_typeof(x) *)&(x))
 
 #define __READ_ONCE_SCALAR(x)						\
 ({									\
-	typeof(x) __x = __READ_ONCE(x);					\
+	__unqual_scalar_typeof(x) __x = __READ_ONCE(x);			\
 	smp_read_barrier_depends();					\
-	__x;								\
+	(typeof(x))__x;							\
 })
 
 /*
diff --git a/include/linux/compiler_types.h b/include/linux/compiler_types.h
index 72393a8c1a6c..58361f2d3b98 100644
--- a/include/linux/compiler_types.h
+++ b/include/linux/compiler_types.h
@@ -219,6 +219,27 @@ struct ftrace_likely_data {
 /* Are two types/vars the same type (ignoring qualifiers)? */
 #define __same_type(a, b) __builtin_types_compatible_p(typeof(a), typeof(b))
 
+/*
+ * __unqual_scalar_typeof(x) - Declare an unqualified scalar type, leaving
+ *			       non-scalar types unchanged.
+ *
+ * We build this out of a couple of helper macros in a vain attempt to
+ * help you keep your lunch down while reading it.
+ */
+#define __pick_scalar_type(x, type, otherwise)					\
+	__builtin_choose_expr(__same_type(x, type), (type)0, otherwise)
+
+#define __pick_integer_type(x, type, otherwise)					\
+	__pick_scalar_type(x, unsigned type,					\
+		__pick_scalar_type(x, signed type, otherwise))
+
+#define __unqual_scalar_typeof(x) typeof(					\
+	__pick_integer_type(x, char,						\
+		__pick_integer_type(x, short,					\
+			__pick_integer_type(x, int,				\
+				__pick_integer_type(x, long,			\
+					__pick_integer_type(x, long long, x))))))
+
 /* Is this type a native word size -- useful for atomic operations */
 #define __native_word(t) \
 	(sizeof(t) == sizeof(char) || sizeof(t) == sizeof(short) || \
-- 
2.25.0.341.g760bfbb309-goog


  parent reply index

Thread overview: 41+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-01-23 15:33 [PATCH v2 00/10] Rework READ_ONCE() to improve codegen Will Deacon
2020-01-23 15:33 ` [PATCH v2 01/10] compiler/gcc: Emit build-time warning for GCC prior to version 4.8 Will Deacon
2020-01-23 15:33 ` [PATCH v2 02/10] netfilter: Avoid assigning 'const' pointer to non-const pointer Will Deacon
2020-01-23 19:07   ` Nick Desaulniers
2020-01-24  8:24     ` Peter Zijlstra
2020-01-24 17:20       ` Nick Desaulniers
2020-01-27 12:04         ` David Laight
2020-01-24 17:36       ` Linus Torvalds
2020-01-24 22:00         ` Peter Zijlstra
2020-01-27 12:21         ` David Laight
2020-01-23 15:33 ` [PATCH v2 03/10] fault_inject: Don't rely on "return value" from WRITE_ONCE() Will Deacon
2020-01-23 15:33 ` [PATCH v2 04/10] READ_ONCE: Simplify implementations of {READ,WRITE}_ONCE() Will Deacon
2020-01-23 15:33 ` [PATCH v2 05/10] READ_ONCE: Enforce atomicity for {READ,WRITE}_ONCE() memory accesses Will Deacon
2020-01-25  8:27   ` Peter Zijlstra
2020-01-29 10:49     ` Peter Zijlstra
2020-01-23 15:33 ` Will Deacon [this message]
2020-01-23 15:33 ` [PATCH v2 07/10] locking/barriers: Use '__unqual_scalar_typeof' for load-acquire macros Will Deacon
2020-01-23 15:33 ` [PATCH v2 08/10] arm64: barrier: Use '__unqual_scalar_typeof' for acquire/release macros Will Deacon
2020-01-23 15:33 ` [PATCH v2 09/10] compiler/gcc: Raise minimum GCC version for kernel builds to 4.8 Will Deacon
2020-01-23 18:36   ` Nick Desaulniers
2020-01-24  8:26     ` Peter Zijlstra
2020-01-24 17:05       ` Nick Desaulniers
2020-01-24 23:29         ` Peter Zijlstra
2020-01-25 10:34         ` Michael Ellerman
2020-01-23 15:33 ` [PATCH v2 10/10] gcov: Remove old GCC 3.4 support Will Deacon
2020-01-23 18:51   ` Nick Desaulniers
2020-01-28 14:56   ` Peter Oberparleiter
2020-01-23 17:07 ` [PATCH v2 00/10] Rework READ_ONCE() to improve codegen David Laight
2020-01-23 17:16   ` Will Deacon
2020-01-23 17:32     ` David Laight
2020-01-23 18:45       ` Nick Desaulniers
2020-01-23 19:01         ` Arvind Sankar
2020-01-24 10:11           ` David Laight
2020-01-26  1:10           ` Qais Yousef
2020-01-27  7:26             ` Arnd Bergmann
2020-01-23 17:59 ` Linus Torvalds
2020-01-24  8:33   ` Peter Zijlstra
2020-01-24 10:41     ` Peter Zijlstra
2020-02-10  9:50     ` Masahiro Yamada
2020-02-10  9:59       ` Will Deacon
2020-01-31 10:20 ` David Howells

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200123153341.19947-7-will@kernel.org \
    --to=will@kernel.org \
    --cc=arnd@arndb.de \
    --cc=borntraeger@de.ibm.com \
    --cc=kernel-team@android.com \
    --cc=linux-arch@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=luc.vanoostenryck@gmail.com \
    --cc=masahiroy@kernel.org \
    --cc=mpe@ellerman.id.au \
    --cc=ndesaulniers@google.com \
    --cc=oberpar@linux.ibm.com \
    --cc=peterz@infradead.org \
    --cc=segher@kernel.crashing.org \
    --cc=torvalds@linux-foundation.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

LKML Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/lkml/0 lkml/git/0.git
	git clone --mirror https://lore.kernel.org/lkml/1 lkml/git/1.git
	git clone --mirror https://lore.kernel.org/lkml/2 lkml/git/2.git
	git clone --mirror https://lore.kernel.org/lkml/3 lkml/git/3.git
	git clone --mirror https://lore.kernel.org/lkml/4 lkml/git/4.git
	git clone --mirror https://lore.kernel.org/lkml/5 lkml/git/5.git
	git clone --mirror https://lore.kernel.org/lkml/6 lkml/git/6.git
	git clone --mirror https://lore.kernel.org/lkml/7 lkml/git/7.git
	git clone --mirror https://lore.kernel.org/lkml/8 lkml/git/8.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 lkml lkml/ https://lore.kernel.org/lkml \
		linux-kernel@vger.kernel.org
	public-inbox-index lkml

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git