From: Will Deacon <will@kernel.org>
To: linux-kernel@vger.kernel.org
Cc: Will Deacon <will@kernel.org>, Joel Fernandes <joelaf@google.com>,
Sami Tolvanen <samitolvanen@google.com>,
Nick Desaulniers <ndesaulniers@google.com>,
Kees Cook <keescook@chromium.org>, Marco Elver <elver@google.com>,
"Paul E. McKenney" <paulmck@kernel.org>,
Matt Turner <mattst88@gmail.com>,
Ivan Kokshaysky <ink@jurassic.park.msu.ru>,
Richard Henderson <rth@twiddle.net>,
Peter Zijlstra <peterz@infradead.org>,
Alan Stern <stern@rowland.harvard.edu>,
"Michael S. Tsirkin" <mst@redhat.com>,
Jason Wang <jasowang@redhat.com>, Arnd Bergmann <arnd@arndb.de>,
Boqun Feng <boqun.feng@gmail.com>,
Catalin Marinas <catalin.marinas@arm.com>,
Mark Rutland <mark.rutland@arm.com>,
linux-arm-kernel@lists.infradead.org,
linux-alpha@vger.kernel.org,
virtualization@lists.linux-foundation.org,
kernel-team@android.com
Subject: [PATCH v3 19/19] arm64: lto: Strengthen READ_ONCE() to acquire when CONFIG_LTO=y
Date: Fri, 10 Jul 2020 17:52:03 +0100 [thread overview]
Message-ID: <20200710165203.31284-20-will@kernel.org> (raw)
In-Reply-To: <20200710165203.31284-1-will@kernel.org>
When building with LTO, there is an increased risk of the compiler
converting an address dependency headed by a READ_ONCE() invocation
into a control dependency and consequently allowing for harmful
reordering by the CPU.
Ensure that such transformations are harmless by overriding the generic
READ_ONCE() definition with one that provides acquire semantics when
building with LTO.
Signed-off-by: Will Deacon <will@kernel.org>
---
arch/arm64/include/asm/rwonce.h | 63 +++++++++++++++++++++++++++++++
arch/arm64/kernel/vdso/Makefile | 2 +-
arch/arm64/kernel/vdso32/Makefile | 2 +-
3 files changed, 65 insertions(+), 2 deletions(-)
create mode 100644 arch/arm64/include/asm/rwonce.h
diff --git a/arch/arm64/include/asm/rwonce.h b/arch/arm64/include/asm/rwonce.h
new file mode 100644
index 000000000000..d78eb4cb795b
--- /dev/null
+++ b/arch/arm64/include/asm/rwonce.h
@@ -0,0 +1,63 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2020 Google LLC.
+ */
+#ifndef __ASM_RWONCE_H
+#define __ASM_RWONCE_H
+
+#ifdef CONFIG_LTO
+
+#include <linux/compiler_types.h>
+#include <asm/alternative-macros.h>
+
+#ifndef BUILD_VDSO
+
+#ifdef CONFIG_AS_HAS_LDAPR
+#define __LOAD_RCPC(sfx, regs...) \
+ ALTERNATIVE( \
+ "ldar" #sfx "\t" #regs, \
+ ".arch_extension rcpc\n" \
+ "ldapr" #sfx "\t" #regs, \
+ ARM64_HAS_LDAPR)
+#else
+#define __LOAD_RCPC(sfx, regs...) "ldar" #sfx "\t" #regs
+#endif /* CONFIG_AS_HAS_LDAPR */
+
+#define __READ_ONCE(x) \
+({ \
+ typeof(&(x)) __x = &(x); \
+ int atomic = 1; \
+ union { __unqual_scalar_typeof(*__x) __val; char __c[1]; } __u; \
+ switch (sizeof(x)) { \
+ case 1: \
+ asm volatile(__LOAD_RCPC(b, %w0, %1) \
+ : "=r" (*(__u8 *)__u.__c) \
+ : "Q" (*__x) : "memory"); \
+ break; \
+ case 2: \
+ asm volatile(__LOAD_RCPC(h, %w0, %1) \
+ : "=r" (*(__u16 *)__u.__c) \
+ : "Q" (*__x) : "memory"); \
+ break; \
+ case 4: \
+ asm volatile(__LOAD_RCPC(, %w0, %1) \
+ : "=r" (*(__u32 *)__u.__c) \
+ : "Q" (*__x) : "memory"); \
+ break; \
+ case 8: \
+ asm volatile(__LOAD_RCPC(, %0, %1) \
+ : "=r" (*(__u64 *)__u.__c) \
+ : "Q" (*__x) : "memory"); \
+ break; \
+ default: \
+ atomic = 0; \
+ } \
+ atomic ? (typeof(*__x))__u.__val : (*(volatile typeof(__x))__x);\
+})
+
+#endif /* !BUILD_VDSO */
+#endif /* CONFIG_LTO */
+
+#include <asm-generic/rwonce.h>
+
+#endif /* __ASM_RWONCE_H */
diff --git a/arch/arm64/kernel/vdso/Makefile b/arch/arm64/kernel/vdso/Makefile
index 45d5cfe46429..60df97f2e7de 100644
--- a/arch/arm64/kernel/vdso/Makefile
+++ b/arch/arm64/kernel/vdso/Makefile
@@ -28,7 +28,7 @@ ldflags-y := -shared -nostdlib -soname=linux-vdso.so.1 --hash-style=sysv \
$(btildflags-y) -T
ccflags-y := -fno-common -fno-builtin -fno-stack-protector -ffixed-x18
-ccflags-y += -DDISABLE_BRANCH_PROFILING
+ccflags-y += -DDISABLE_BRANCH_PROFILING -DBUILD_VDSO
CFLAGS_REMOVE_vgettimeofday.o = $(CC_FLAGS_FTRACE) -Os $(CC_FLAGS_SCS) $(GCC_PLUGINS_CFLAGS)
KBUILD_CFLAGS += $(DISABLE_LTO)
diff --git a/arch/arm64/kernel/vdso32/Makefile b/arch/arm64/kernel/vdso32/Makefile
index d88148bef6b0..4fdf3754a058 100644
--- a/arch/arm64/kernel/vdso32/Makefile
+++ b/arch/arm64/kernel/vdso32/Makefile
@@ -43,7 +43,7 @@ cc32-as-instr = $(call try-run,\
# As a result we set our own flags here.
# KBUILD_CPPFLAGS and NOSTDINC_FLAGS from top-level Makefile
-VDSO_CPPFLAGS := -D__KERNEL__ -nostdinc -isystem $(shell $(CC_COMPAT) -print-file-name=include)
+VDSO_CPPFLAGS := -DBUILD_VDSO -D__KERNEL__ -nostdinc -isystem $(shell $(CC_COMPAT) -print-file-name=include)
VDSO_CPPFLAGS += $(LINUXINCLUDE)
# Common C and assembly flags
--
2.27.0.383.g050319c2ae-goog
next prev parent reply other threads:[~2020-07-10 16:53 UTC|newest]
Thread overview: 27+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-07-10 16:51 [PATCH 00/18] Allow architectures to override __READ_ONCE() Will Deacon
2020-07-10 16:51 ` [PATCH v3 01/19] tools: bpf: Use local copy of headers including uapi/linux/filter.h Will Deacon
2020-07-10 16:51 ` [PATCH v3 02/19] compiler.h: Split {READ,WRITE}_ONCE definitions out into rwonce.h Will Deacon
2020-07-13 12:23 ` boqun.feng
2020-07-20 15:55 ` Will Deacon
2020-07-10 16:51 ` [PATCH v3 03/19] asm/rwonce: Allow __READ_ONCE to be overridden by the architecture Will Deacon
2020-07-10 16:51 ` [PATCH v3 04/19] alpha: Override READ_ONCE() with barriered implementation Will Deacon
2020-07-10 16:51 ` [PATCH v3 05/19] asm/rwonce: Remove smp_read_barrier_depends() invocation Will Deacon
2020-07-10 16:51 ` [PATCH v3 06/19] asm/rwonce: Don't pull <asm/barrier.h> into 'asm-generic/rwonce.h' Will Deacon
2020-07-10 17:06 ` Nick Desaulniers
2020-07-10 17:15 ` Will Deacon
2020-07-10 16:51 ` [PATCH v3 07/19] vhost: Remove redundant use of read_barrier_depends() barrier Will Deacon
2020-07-13 11:27 ` Michael S. Tsirkin
2020-07-10 16:51 ` [PATCH v3 08/19] alpha: Replace smp_read_barrier_depends() usage with smp_[r]mb() Will Deacon
2020-07-10 16:51 ` [PATCH v3 09/19] locking/barriers: Remove definitions for [smp_]read_barrier_depends() Will Deacon
2020-07-10 16:51 ` [PATCH v3 10/19] Documentation/barriers: Remove references to [smp_]read_barrier_depends() Will Deacon
2020-07-10 16:51 ` [PATCH v3 11/19] Documentation/barriers/kokr: " Will Deacon
2020-07-10 16:51 ` [PATCH v3 12/19] tools/memory-model: Remove smp_read_barrier_depends() from informal doc Will Deacon
2020-07-10 16:51 ` [PATCH v3 13/19] include/linux: Remove smp_read_barrier_depends() from comments Will Deacon
2020-07-10 16:51 ` [PATCH v3 14/19] checkpatch: Remove checks relating to [smp_]read_barrier_depends() Will Deacon
2020-07-10 16:51 ` [PATCH v3 15/19] arm64: Reduce the number of header files pulled into vmlinux.lds.S Will Deacon
2020-07-10 16:52 ` [PATCH v3 16/19] arm64: alternatives: Split up alternative.h Will Deacon
2020-07-10 16:52 ` [PATCH v3 17/19] arm64: cpufeatures: Add capability for LDAPR instruction Will Deacon
2020-07-10 16:52 ` [PATCH v3 18/19] arm64: alternatives: Remove READ_ONCE() usage during patch operation Will Deacon
2020-07-10 16:52 ` Will Deacon [this message]
2020-07-28 20:40 ` [PATCH v3 19/19] arm64: lto: Strengthen READ_ONCE() to acquire when CONFIG_LTO=y Pavel Machek
2020-07-13 10:34 ` [PATCH 00/18] Allow architectures to override __READ_ONCE() Peter Zijlstra
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200710165203.31284-20-will@kernel.org \
--to=will@kernel.org \
--cc=arnd@arndb.de \
--cc=boqun.feng@gmail.com \
--cc=catalin.marinas@arm.com \
--cc=elver@google.com \
--cc=ink@jurassic.park.msu.ru \
--cc=jasowang@redhat.com \
--cc=joelaf@google.com \
--cc=keescook@chromium.org \
--cc=kernel-team@android.com \
--cc=linux-alpha@vger.kernel.org \
--cc=linux-arm-kernel@lists.infradead.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mark.rutland@arm.com \
--cc=mattst88@gmail.com \
--cc=mst@redhat.com \
--cc=ndesaulniers@google.com \
--cc=paulmck@kernel.org \
--cc=peterz@infradead.org \
--cc=rth@twiddle.net \
--cc=samitolvanen@google.com \
--cc=stern@rowland.harvard.edu \
--cc=virtualization@lists.linux-foundation.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).