* [PATCH v4 0/7] x86, kasan: add KASAN checks to atomic operations
@ 2017-06-17 9:15 Dmitry Vyukov
2017-06-17 9:15 ` [PATCH v4 1/7] x86: un-macro-ify atomic ops implementation Dmitry Vyukov
` (6 more replies)
0 siblings, 7 replies; 38+ messages in thread
From: Dmitry Vyukov @ 2017-06-17 9:15 UTC (permalink / raw)
To: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin,
kasan-dev, x86, linux-kernel
Cc: Dmitry Vyukov
KASAN uses compiler instrumentation to intercept all memory accesses.
But it does not see memory accesses done in assembly code.
One notable user of assembly code is atomic operations. Frequently,
for example, an atomic reference decrement is the last access to an
object and a good candidate for a racy use-after-free.
Atomic operations are defined in arch files, but KASAN instrumentation
is required for several archs that support KASAN. Later we will need
similar hooks for KMSAN (uninit use detector) and KTSAN (data race
detector).
This change introduces wrappers around atomic operations that can be
used to add KASAN/KMSAN/KTSAN instrumentation across several archs,
and adds KASAN checks to them.
This patch uses the wrappers only for x86 arch. Arm64 will be switched
later. And we also plan to instrument bitops in a similar way.
Within a day it has found its first bug:
BUG: KASAN: use-after-free in atomic_dec_and_test
arch/x86/include/asm/atomic.h:123 [inline] at addr ffff880079c30158
Write of size 4 by task syz-executor6/25698
CPU: 2 PID: 25698 Comm: syz-executor6 Not tainted 4.10.0+ #302
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
kasan_check_write+0x14/0x20 mm/kasan/kasan.c:344
atomic_dec_and_test arch/x86/include/asm/atomic.h:123 [inline]
put_task_struct include/linux/sched/task.h:93 [inline]
put_ctx+0xcf/0x110 kernel/events/core.c:1131
perf_event_release_kernel+0x3ad/0xc90 kernel/events/core.c:4322
perf_release+0x37/0x50 kernel/events/core.c:4338
__fput+0x332/0x800 fs/file_table.c:209
____fput+0x15/0x20 fs/file_table.c:245
task_work_run+0x197/0x260 kernel/task_work.c:116
exit_task_work include/linux/task_work.h:21 [inline]
do_exit+0xb38/0x29c0 kernel/exit.c:880
do_group_exit+0x149/0x420 kernel/exit.c:984
get_signal+0x7e0/0x1820 kernel/signal.c:2318
do_signal+0xd2/0x2190 arch/x86/kernel/signal.c:808
exit_to_usermode_loop+0x200/0x2a0 arch/x86/entry/common.c:157
syscall_return_slowpath arch/x86/entry/common.c:191 [inline]
do_syscall_64+0x6fc/0x930 arch/x86/entry/common.c:286
entry_SYSCALL64_slow_path+0x25/0x25
RIP: 0033:0x4458d9
RSP: 002b:00007f3f07187cf8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
RAX: fffffffffffffe00 RBX: 00000000007080c8 RCX: 00000000004458d9
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00000000007080c8
RBP: 00000000007080a8 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 00007f3f071889c0 R15: 00007f3f07188700
Object at ffff880079c30140, in cache task_struct size: 5376
Allocated:
PID = 25681
kmem_cache_alloc_node+0x122/0x6f0 mm/slab.c:3662
alloc_task_struct_node kernel/fork.c:153 [inline]
dup_task_struct kernel/fork.c:495 [inline]
copy_process.part.38+0x19c8/0x4aa0 kernel/fork.c:1560
copy_process kernel/fork.c:1531 [inline]
_do_fork+0x200/0x1010 kernel/fork.c:1994
SYSC_clone kernel/fork.c:2104 [inline]
SyS_clone+0x37/0x50 kernel/fork.c:2098
do_syscall_64+0x2e8/0x930 arch/x86/entry/common.c:281
return_from_SYSCALL_64+0x0/0x7a
Freed:
PID = 25681
__cache_free mm/slab.c:3514 [inline]
kmem_cache_free+0x71/0x240 mm/slab.c:3774
free_task_struct kernel/fork.c:158 [inline]
free_task+0x151/0x1d0 kernel/fork.c:370
copy_process.part.38+0x18e5/0x4aa0 kernel/fork.c:1931
copy_process kernel/fork.c:1531 [inline]
_do_fork+0x200/0x1010 kernel/fork.c:1994
SYSC_clone kernel/fork.c:2104 [inline]
SyS_clone+0x37/0x50 kernel/fork.c:2098
do_syscall_64+0x2e8/0x930 arch/x86/entry/common.c:281
return_from_SYSCALL_64+0x0/0x7a
Changes since v1:
- dropped "x86: remove unused atomic_inc_short()" patch
it is mailed separately
- rebased on top of tip/locking/core head
- other changes noted within individual patches
Changes since v2:
- rebased on top of tip/locking/core head
- dropped a pervasive "x86: use long long for 64-bit atomic ops" commit,
instead use s64 type in wrappers
- added "x86: use s64* for old arg of atomic64_try_cmpxchg()" commit
Changes since v3 are noted in individual commits.
Tested:
- build/boot x86_64 defconfig
- build/boot x86_64 defconfig+KASAN
- build i686/powerpc defconfig
Dmitry Vyukov (7):
x86: un-macro-ify atomic ops implementation
x86: use s64* for old arg of atomic64_try_cmpxchg()
asm-generic: add atomic-instrumented.h
x86: switch atomic.h to use atomic-instrumented.h
kasan: allow kasan_check_read/write() to accept pointers to volatiles
asm-generic: add KASAN instrumentation to atomic operations
asm-generic, x86: add comments for atomic instrumentation
arch/x86/include/asm/atomic.h | 150 ++++++-----
arch/x86/include/asm/atomic64_32.h | 153 ++++++-----
arch/x86/include/asm/atomic64_64.h | 151 ++++++-----
arch/x86/include/asm/cmpxchg.h | 14 +-
arch/x86/include/asm/cmpxchg_32.h | 8 +-
arch/x86/include/asm/cmpxchg_64.h | 4 +-
include/asm-generic/atomic-instrumented.h | 414 ++++++++++++++++++++++++++++++
include/linux/kasan-checks.h | 10 +-
mm/kasan/kasan.c | 4 +-
9 files changed, 702 insertions(+), 206 deletions(-)
create mode 100644 include/asm-generic/atomic-instrumented.h
--
2.13.1.518.g3df882009-goog
^ permalink raw reply [flat|nested] 38+ messages in thread
* [PATCH v4 1/7] x86: un-macro-ify atomic ops implementation
2017-06-17 9:15 [PATCH v4 0/7] x86, kasan: add KASAN checks to atomic operations Dmitry Vyukov
@ 2017-06-17 9:15 ` Dmitry Vyukov
2017-06-22 11:04 ` [tip:locking/core] locking/atomic/x86: Un-macro-ify " tip-bot for Dmitry Vyukov
2017-07-25 13:54 ` tip-bot for Dmitry Vyukov
2017-06-17 9:15 ` [PATCH v4 2/7] x86: use s64* for old arg of atomic64_try_cmpxchg() Dmitry Vyukov
` (5 subsequent siblings)
6 siblings, 2 replies; 38+ messages in thread
From: Dmitry Vyukov @ 2017-06-17 9:15 UTC (permalink / raw)
To: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin,
kasan-dev, x86, linux-kernel
Cc: Dmitry Vyukov, Thomas Gleixner, Andrew Morton
CPP turns perfectly readable code into an unreadable,
unmaintainable mess. Ingo suggested to write them out as-is.
Do this.
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Suggested-by: Ingo Molnar <mingo@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: linux-kernel@vger.kernel.org
Cc: x86@kernel.org
Cc: kasan-dev@googlegroups.com
---
arch/x86/include/asm/atomic.h | 80 ++++++++++++++++++++++++--------------
arch/x86/include/asm/atomic64_32.h | 77 ++++++++++++++++++++++++------------
arch/x86/include/asm/atomic64_64.h | 67 ++++++++++++++++++++-----------
3 files changed, 148 insertions(+), 76 deletions(-)
diff --git a/arch/x86/include/asm/atomic.h b/arch/x86/include/asm/atomic.h
index 33380b871463..8d7f6e579be4 100644
--- a/arch/x86/include/asm/atomic.h
+++ b/arch/x86/include/asm/atomic.h
@@ -197,35 +197,57 @@ static inline int atomic_xchg(atomic_t *v, int new)
return xchg(&v->counter, new);
}
-#define ATOMIC_OP(op) \
-static inline void atomic_##op(int i, atomic_t *v) \
-{ \
- asm volatile(LOCK_PREFIX #op"l %1,%0" \
- : "+m" (v->counter) \
- : "ir" (i) \
- : "memory"); \
-}
-
-#define ATOMIC_FETCH_OP(op, c_op) \
-static inline int atomic_fetch_##op(int i, atomic_t *v) \
-{ \
- int val = atomic_read(v); \
- do { \
- } while (!atomic_try_cmpxchg(v, &val, val c_op i)); \
- return val; \
-}
-
-#define ATOMIC_OPS(op, c_op) \
- ATOMIC_OP(op) \
- ATOMIC_FETCH_OP(op, c_op)
-
-ATOMIC_OPS(and, &)
-ATOMIC_OPS(or , |)
-ATOMIC_OPS(xor, ^)
-
-#undef ATOMIC_OPS
-#undef ATOMIC_FETCH_OP
-#undef ATOMIC_OP
+static inline void atomic_and(int i, atomic_t *v)
+{
+ asm volatile(LOCK_PREFIX "andl %1,%0"
+ : "+m" (v->counter)
+ : "ir" (i)
+ : "memory");
+}
+
+static inline int atomic_fetch_and(int i, atomic_t *v)
+{
+ int val = atomic_read(v);
+
+ do {
+ } while (!atomic_try_cmpxchg(v, &val, val & i));
+ return val;
+}
+
+static inline void atomic_or(int i, atomic_t *v)
+{
+ asm volatile(LOCK_PREFIX "orl %1,%0"
+ : "+m" (v->counter)
+ : "ir" (i)
+ : "memory");
+}
+
+static inline int atomic_fetch_or(int i, atomic_t *v)
+{
+ int val = atomic_read(v);
+
+ do {
+ } while (!atomic_try_cmpxchg(v, &val, val | i));
+ return val;
+}
+
+
+static inline void atomic_xor(int i, atomic_t *v)
+{
+ asm volatile(LOCK_PREFIX "xorl %1,%0"
+ : "+m" (v->counter)
+ : "ir" (i)
+ : "memory");
+}
+
+static inline int atomic_fetch_xor(int i, atomic_t *v)
+{
+ int val = atomic_read(v);
+
+ do {
+ } while (!atomic_try_cmpxchg(v, &val, val ^ i));
+ return val;
+}
/**
* __atomic_add_unless - add unless the number is already a given value
diff --git a/arch/x86/include/asm/atomic64_32.h b/arch/x86/include/asm/atomic64_32.h
index 71d7705fb303..f107fef7bfcc 100644
--- a/arch/x86/include/asm/atomic64_32.h
+++ b/arch/x86/include/asm/atomic64_32.h
@@ -312,37 +312,66 @@ static inline long long atomic64_dec_if_positive(atomic64_t *v)
#undef alternative_atomic64
#undef __alternative_atomic64
-#define ATOMIC64_OP(op, c_op) \
-static inline void atomic64_##op(long long i, atomic64_t *v) \
-{ \
- long long old, c = 0; \
- while ((old = atomic64_cmpxchg(v, c, c c_op i)) != c) \
- c = old; \
+static inline void atomic64_and(long long i, atomic64_t *v)
+{
+ long long old, c = 0;
+
+ while ((old = atomic64_cmpxchg(v, c, c & i)) != c)
+ c = old;
}
-#define ATOMIC64_FETCH_OP(op, c_op) \
-static inline long long atomic64_fetch_##op(long long i, atomic64_t *v) \
-{ \
- long long old, c = 0; \
- while ((old = atomic64_cmpxchg(v, c, c c_op i)) != c) \
- c = old; \
- return old; \
+static inline long long atomic64_fetch_and(long long i, atomic64_t *v)
+{
+ long long old, c = 0;
+
+ while ((old = atomic64_cmpxchg(v, c, c & i)) != c)
+ c = old;
+ return old;
}
-ATOMIC64_FETCH_OP(add, +)
+static inline void atomic64_or(long long i, atomic64_t *v)
+{
+ long long old, c = 0;
-#define atomic64_fetch_sub(i, v) atomic64_fetch_add(-(i), (v))
+ while ((old = atomic64_cmpxchg(v, c, c | i)) != c)
+ c = old;
+}
+
+static inline long long atomic64_fetch_or(long long i, atomic64_t *v)
+{
+ long long old, c = 0;
+
+ while ((old = atomic64_cmpxchg(v, c, c | i)) != c)
+ c = old;
+ return old;
+}
-#define ATOMIC64_OPS(op, c_op) \
- ATOMIC64_OP(op, c_op) \
- ATOMIC64_FETCH_OP(op, c_op)
+static inline void atomic64_xor(long long i, atomic64_t *v)
+{
+ long long old, c = 0;
+
+ while ((old = atomic64_cmpxchg(v, c, c ^ i)) != c)
+ c = old;
+}
-ATOMIC64_OPS(and, &)
-ATOMIC64_OPS(or, |)
-ATOMIC64_OPS(xor, ^)
+static inline long long atomic64_fetch_xor(long long i, atomic64_t *v)
+{
+ long long old, c = 0;
+
+ while ((old = atomic64_cmpxchg(v, c, c ^ i)) != c)
+ c = old;
+ return old;
+}
-#undef ATOMIC64_OPS
-#undef ATOMIC64_FETCH_OP
-#undef ATOMIC64_OP
+static inline long long atomic64_fetch_add(long long i, atomic64_t *v)
+{
+ long long old, c = 0;
+
+ while ((old = atomic64_cmpxchg(v, c, c + i)) != c)
+ c = old;
+ return old;
+}
+
+#define atomic64_fetch_sub(i, v) atomic64_fetch_add(-(i), (v))
#endif /* _ASM_X86_ATOMIC64_32_H */
diff --git a/arch/x86/include/asm/atomic64_64.h b/arch/x86/include/asm/atomic64_64.h
index 6189a433c9a9..8db8879a6d8c 100644
--- a/arch/x86/include/asm/atomic64_64.h
+++ b/arch/x86/include/asm/atomic64_64.h
@@ -226,34 +226,55 @@ static inline long atomic64_dec_if_positive(atomic64_t *v)
return dec;
}
-#define ATOMIC64_OP(op) \
-static inline void atomic64_##op(long i, atomic64_t *v) \
-{ \
- asm volatile(LOCK_PREFIX #op"q %1,%0" \
- : "+m" (v->counter) \
- : "er" (i) \
- : "memory"); \
+static inline void atomic64_and(long i, atomic64_t *v)
+{
+ asm volatile(LOCK_PREFIX "andq %1,%0"
+ : "+m" (v->counter)
+ : "er" (i)
+ : "memory");
}
-#define ATOMIC64_FETCH_OP(op, c_op) \
-static inline long atomic64_fetch_##op(long i, atomic64_t *v) \
-{ \
- long val = atomic64_read(v); \
- do { \
- } while (!atomic64_try_cmpxchg(v, &val, val c_op i)); \
- return val; \
+static inline long atomic64_fetch_and(long i, atomic64_t *v)
+{
+ long val = atomic64_read(v);
+
+ do {
+ } while (!atomic64_try_cmpxchg(v, &val, val & i));
+ return val;
}
-#define ATOMIC64_OPS(op, c_op) \
- ATOMIC64_OP(op) \
- ATOMIC64_FETCH_OP(op, c_op)
+static inline void atomic64_or(long i, atomic64_t *v)
+{
+ asm volatile(LOCK_PREFIX "orq %1,%0"
+ : "+m" (v->counter)
+ : "er" (i)
+ : "memory");
+}
-ATOMIC64_OPS(and, &)
-ATOMIC64_OPS(or, |)
-ATOMIC64_OPS(xor, ^)
+static inline long atomic64_fetch_or(long i, atomic64_t *v)
+{
+ long val = atomic64_read(v);
-#undef ATOMIC64_OPS
-#undef ATOMIC64_FETCH_OP
-#undef ATOMIC64_OP
+ do {
+ } while (!atomic64_try_cmpxchg(v, &val, val | i));
+ return val;
+}
+
+static inline void atomic64_xor(long i, atomic64_t *v)
+{
+ asm volatile(LOCK_PREFIX "xorq %1,%0"
+ : "+m" (v->counter)
+ : "er" (i)
+ : "memory");
+}
+
+static inline long atomic64_fetch_xor(long i, atomic64_t *v)
+{
+ long val = atomic64_read(v);
+
+ do {
+ } while (!atomic64_try_cmpxchg(v, &val, val ^ i));
+ return val;
+}
#endif /* _ASM_X86_ATOMIC64_64_H */
--
2.13.1.518.g3df882009-goog
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v4 2/7] x86: use s64* for old arg of atomic64_try_cmpxchg()
2017-06-17 9:15 [PATCH v4 0/7] x86, kasan: add KASAN checks to atomic operations Dmitry Vyukov
2017-06-17 9:15 ` [PATCH v4 1/7] x86: un-macro-ify atomic ops implementation Dmitry Vyukov
@ 2017-06-17 9:15 ` Dmitry Vyukov
2017-06-22 11:04 ` [tip:locking/core] locking/atomic/x86: Use 's64 *' for 'old' argument " tip-bot for Dmitry Vyukov
2017-07-25 13:55 ` tip-bot for Dmitry Vyukov
2017-06-17 9:15 ` [PATCH v4 3/7] asm-generic: add atomic-instrumented.h Dmitry Vyukov
` (4 subsequent siblings)
6 siblings, 2 replies; 38+ messages in thread
From: Dmitry Vyukov @ 2017-06-17 9:15 UTC (permalink / raw)
To: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin,
kasan-dev, x86, linux-kernel
Cc: Dmitry Vyukov, Andrew Morton, linux-mm
atomic64_try_cmpxchg() declares old argument as long*,
this makes it impossible to use it in portable code.
If caller passes long*, it becomes 32-bits on 32-bit arches.
If caller passes s64*, it does not compile on x86_64.
Change type of old arg to s64*.
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: kasan-dev@googlegroups.com
Cc: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org
Cc: x86@kernel.org
---
arch/x86/include/asm/atomic64_64.h | 12 ++++++------
arch/x86/include/asm/cmpxchg.h | 2 +-
2 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/arch/x86/include/asm/atomic64_64.h b/arch/x86/include/asm/atomic64_64.h
index 8db8879a6d8c..5d9de36a2f04 100644
--- a/arch/x86/include/asm/atomic64_64.h
+++ b/arch/x86/include/asm/atomic64_64.h
@@ -177,7 +177,7 @@ static inline long atomic64_cmpxchg(atomic64_t *v, long old, long new)
}
#define atomic64_try_cmpxchg atomic64_try_cmpxchg
-static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, long *old, long new)
+static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, long new)
{
return try_cmpxchg(&v->counter, old, new);
}
@@ -198,7 +198,7 @@ static inline long atomic64_xchg(atomic64_t *v, long new)
*/
static inline bool atomic64_add_unless(atomic64_t *v, long a, long u)
{
- long c = atomic64_read(v);
+ s64 c = atomic64_read(v);
do {
if (unlikely(c == u))
return false;
@@ -217,7 +217,7 @@ static inline bool atomic64_add_unless(atomic64_t *v, long a, long u)
*/
static inline long atomic64_dec_if_positive(atomic64_t *v)
{
- long dec, c = atomic64_read(v);
+ s64 dec, c = atomic64_read(v);
do {
dec = c - 1;
if (unlikely(dec < 0))
@@ -236,7 +236,7 @@ static inline void atomic64_and(long i, atomic64_t *v)
static inline long atomic64_fetch_and(long i, atomic64_t *v)
{
- long val = atomic64_read(v);
+ s64 val = atomic64_read(v);
do {
} while (!atomic64_try_cmpxchg(v, &val, val & i));
@@ -253,7 +253,7 @@ static inline void atomic64_or(long i, atomic64_t *v)
static inline long atomic64_fetch_or(long i, atomic64_t *v)
{
- long val = atomic64_read(v);
+ s64 val = atomic64_read(v);
do {
} while (!atomic64_try_cmpxchg(v, &val, val | i));
@@ -270,7 +270,7 @@ static inline void atomic64_xor(long i, atomic64_t *v)
static inline long atomic64_fetch_xor(long i, atomic64_t *v)
{
- long val = atomic64_read(v);
+ s64 val = atomic64_read(v);
do {
} while (!atomic64_try_cmpxchg(v, &val, val ^ i));
diff --git a/arch/x86/include/asm/cmpxchg.h b/arch/x86/include/asm/cmpxchg.h
index d90296d061e8..b5069e802d5c 100644
--- a/arch/x86/include/asm/cmpxchg.h
+++ b/arch/x86/include/asm/cmpxchg.h
@@ -157,7 +157,7 @@ extern void __add_wrong_size(void)
#define __raw_try_cmpxchg(_ptr, _pold, _new, size, lock) \
({ \
bool success; \
- __typeof__(_ptr) _old = (_pold); \
+ __typeof__(_ptr) _old = (__typeof__(_ptr))(_pold); \
__typeof__(*(_ptr)) __old = *_old; \
__typeof__(*(_ptr)) __new = (_new); \
switch (size) { \
--
2.13.1.518.g3df882009-goog
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v4 3/7] asm-generic: add atomic-instrumented.h
2017-06-17 9:15 [PATCH v4 0/7] x86, kasan: add KASAN checks to atomic operations Dmitry Vyukov
2017-06-17 9:15 ` [PATCH v4 1/7] x86: un-macro-ify atomic ops implementation Dmitry Vyukov
2017-06-17 9:15 ` [PATCH v4 2/7] x86: use s64* for old arg of atomic64_try_cmpxchg() Dmitry Vyukov
@ 2017-06-17 9:15 ` Dmitry Vyukov
2017-06-19 10:50 ` Mark Rutland
` (2 more replies)
2017-06-17 9:15 ` [PATCH v4 4/7] x86: switch atomic.h to use atomic-instrumented.h Dmitry Vyukov
` (3 subsequent siblings)
6 siblings, 3 replies; 38+ messages in thread
From: Dmitry Vyukov @ 2017-06-17 9:15 UTC (permalink / raw)
To: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin,
kasan-dev, x86, linux-kernel
Cc: Dmitry Vyukov, Andrew Morton, linux-mm
The new header allows to wrap per-arch atomic operations
and add common functionality to all of them.
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: kasan-dev@googlegroups.com
Cc: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org
Cc: x86@kernel.org
---
include/asm-generic/atomic-instrumented.h | 316 ++++++++++++++++++++++++++++++
1 file changed, 316 insertions(+)
diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h
new file mode 100644
index 000000000000..50401d925290
--- /dev/null
+++ b/include/asm-generic/atomic-instrumented.h
@@ -0,0 +1,316 @@
+#ifndef _LINUX_ATOMIC_INSTRUMENTED_H
+#define _LINUX_ATOMIC_INSTRUMENTED_H
+
+static __always_inline int atomic_read(const atomic_t *v)
+{
+ return arch_atomic_read(v);
+}
+
+static __always_inline s64 atomic64_read(const atomic64_t *v)
+{
+ return arch_atomic64_read(v);
+}
+
+static __always_inline void atomic_set(atomic_t *v, int i)
+{
+ arch_atomic_set(v, i);
+}
+
+static __always_inline void atomic64_set(atomic64_t *v, s64 i)
+{
+ arch_atomic64_set(v, i);
+}
+
+static __always_inline int atomic_xchg(atomic_t *v, int i)
+{
+ return arch_atomic_xchg(v, i);
+}
+
+static __always_inline s64 atomic64_xchg(atomic64_t *v, s64 i)
+{
+ return arch_atomic64_xchg(v, i);
+}
+
+static __always_inline int atomic_cmpxchg(atomic_t *v, int old, int new)
+{
+ return arch_atomic_cmpxchg(v, old, new);
+}
+
+static __always_inline s64 atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new)
+{
+ return arch_atomic64_cmpxchg(v, old, new);
+}
+
+#ifdef arch_atomic_try_cmpxchg
+#define atomic_try_cmpxchg atomic_try_cmpxchg
+static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new)
+{
+ return arch_atomic_try_cmpxchg(v, old, new);
+}
+#endif
+
+#ifdef arch_atomic64_try_cmpxchg
+#define atomic64_try_cmpxchg atomic64_try_cmpxchg
+static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, s64 new)
+{
+ return arch_atomic64_try_cmpxchg(v, old, new);
+}
+#endif
+
+static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u)
+{
+ return __arch_atomic_add_unless(v, a, u);
+}
+
+
+static __always_inline bool atomic64_add_unless(atomic64_t *v, s64 a, s64 u)
+{
+ return arch_atomic64_add_unless(v, a, u);
+}
+
+static __always_inline void atomic_inc(atomic_t *v)
+{
+ arch_atomic_inc(v);
+}
+
+static __always_inline void atomic64_inc(atomic64_t *v)
+{
+ arch_atomic64_inc(v);
+}
+
+static __always_inline void atomic_dec(atomic_t *v)
+{
+ arch_atomic_dec(v);
+}
+
+static __always_inline void atomic64_dec(atomic64_t *v)
+{
+ arch_atomic64_dec(v);
+}
+
+static __always_inline void atomic_add(int i, atomic_t *v)
+{
+ arch_atomic_add(i, v);
+}
+
+static __always_inline void atomic64_add(s64 i, atomic64_t *v)
+{
+ arch_atomic64_add(i, v);
+}
+
+static __always_inline void atomic_sub(int i, atomic_t *v)
+{
+ arch_atomic_sub(i, v);
+}
+
+static __always_inline void atomic64_sub(s64 i, atomic64_t *v)
+{
+ arch_atomic64_sub(i, v);
+}
+
+static __always_inline void atomic_and(int i, atomic_t *v)
+{
+ arch_atomic_and(i, v);
+}
+
+static __always_inline void atomic64_and(s64 i, atomic64_t *v)
+{
+ arch_atomic64_and(i, v);
+}
+
+static __always_inline void atomic_or(int i, atomic_t *v)
+{
+ arch_atomic_or(i, v);
+}
+
+static __always_inline void atomic64_or(s64 i, atomic64_t *v)
+{
+ arch_atomic64_or(i, v);
+}
+
+static __always_inline void atomic_xor(int i, atomic_t *v)
+{
+ arch_atomic_xor(i, v);
+}
+
+static __always_inline void atomic64_xor(s64 i, atomic64_t *v)
+{
+ arch_atomic64_xor(i, v);
+}
+
+static __always_inline int atomic_inc_return(atomic_t *v)
+{
+ return arch_atomic_inc_return(v);
+}
+
+static __always_inline s64 atomic64_inc_return(atomic64_t *v)
+{
+ return arch_atomic64_inc_return(v);
+}
+
+static __always_inline int atomic_dec_return(atomic_t *v)
+{
+ return arch_atomic_dec_return(v);
+}
+
+static __always_inline s64 atomic64_dec_return(atomic64_t *v)
+{
+ return arch_atomic64_dec_return(v);
+}
+
+static __always_inline s64 atomic64_inc_not_zero(atomic64_t *v)
+{
+ return arch_atomic64_inc_not_zero(v);
+}
+
+static __always_inline s64 atomic64_dec_if_positive(atomic64_t *v)
+{
+ return arch_atomic64_dec_if_positive(v);
+}
+
+static __always_inline bool atomic_dec_and_test(atomic_t *v)
+{
+ return arch_atomic_dec_and_test(v);
+}
+
+static __always_inline bool atomic64_dec_and_test(atomic64_t *v)
+{
+ return arch_atomic64_dec_and_test(v);
+}
+
+static __always_inline bool atomic_inc_and_test(atomic_t *v)
+{
+ return arch_atomic_inc_and_test(v);
+}
+
+static __always_inline bool atomic64_inc_and_test(atomic64_t *v)
+{
+ return arch_atomic64_inc_and_test(v);
+}
+
+static __always_inline int atomic_add_return(int i, atomic_t *v)
+{
+ return arch_atomic_add_return(i, v);
+}
+
+static __always_inline s64 atomic64_add_return(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_add_return(i, v);
+}
+
+static __always_inline int atomic_sub_return(int i, atomic_t *v)
+{
+ return arch_atomic_sub_return(i, v);
+}
+
+static __always_inline s64 atomic64_sub_return(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_sub_return(i, v);
+}
+
+static __always_inline int atomic_fetch_add(int i, atomic_t *v)
+{
+ return arch_atomic_fetch_add(i, v);
+}
+
+static __always_inline s64 atomic64_fetch_add(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_fetch_add(i, v);
+}
+
+static __always_inline int atomic_fetch_sub(int i, atomic_t *v)
+{
+ return arch_atomic_fetch_sub(i, v);
+}
+
+static __always_inline s64 atomic64_fetch_sub(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_fetch_sub(i, v);
+}
+
+static __always_inline int atomic_fetch_and(int i, atomic_t *v)
+{
+ return arch_atomic_fetch_and(i, v);
+}
+
+static __always_inline s64 atomic64_fetch_and(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_fetch_and(i, v);
+}
+
+static __always_inline int atomic_fetch_or(int i, atomic_t *v)
+{
+ return arch_atomic_fetch_or(i, v);
+}
+
+static __always_inline s64 atomic64_fetch_or(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_fetch_or(i, v);
+}
+
+static __always_inline int atomic_fetch_xor(int i, atomic_t *v)
+{
+ return arch_atomic_fetch_xor(i, v);
+}
+
+static __always_inline s64 atomic64_fetch_xor(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_fetch_xor(i, v);
+}
+
+static __always_inline bool atomic_sub_and_test(int i, atomic_t *v)
+{
+ return arch_atomic_sub_and_test(i, v);
+}
+
+static __always_inline bool atomic64_sub_and_test(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_sub_and_test(i, v);
+}
+
+static __always_inline bool atomic_add_negative(int i, atomic_t *v)
+{
+ return arch_atomic_add_negative(i, v);
+}
+
+static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_add_negative(i, v);
+}
+
+#define cmpxchg(ptr, old, new) \
+({ \
+ arch_cmpxchg((ptr), (old), (new)); \
+})
+
+#define sync_cmpxchg(ptr, old, new) \
+({ \
+ arch_sync_cmpxchg((ptr), (old), (new)); \
+})
+
+#define cmpxchg_local(ptr, old, new) \
+({ \
+ arch_cmpxchg_local((ptr), (old), (new)); \
+})
+
+#define cmpxchg64(ptr, old, new) \
+({ \
+ arch_cmpxchg64((ptr), (old), (new)); \
+})
+
+#define cmpxchg64_local(ptr, old, new) \
+({ \
+ arch_cmpxchg64_local((ptr), (old), (new)); \
+})
+
+#define cmpxchg_double(p1, p2, o1, o2, n1, n2) \
+({ \
+ arch_cmpxchg_double((p1), (p2), (o1), (o2), (n1), (n2)); \
+})
+
+#define cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \
+({ \
+ arch_cmpxchg_double_local((p1), (p2), (o1), (o2), (n1), (n2)); \
+})
+
+#endif /* _LINUX_ATOMIC_INSTRUMENTED_H */
--
2.13.1.518.g3df882009-goog
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v4 4/7] x86: switch atomic.h to use atomic-instrumented.h
2017-06-17 9:15 [PATCH v4 0/7] x86, kasan: add KASAN checks to atomic operations Dmitry Vyukov
` (2 preceding siblings ...)
2017-06-17 9:15 ` [PATCH v4 3/7] asm-generic: add atomic-instrumented.h Dmitry Vyukov
@ 2017-06-17 9:15 ` Dmitry Vyukov
2017-06-17 9:15 ` [PATCH v4 5/7] kasan: allow kasan_check_read/write() to accept pointers to volatiles Dmitry Vyukov
` (2 subsequent siblings)
6 siblings, 0 replies; 38+ messages in thread
From: Dmitry Vyukov @ 2017-06-17 9:15 UTC (permalink / raw)
To: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin,
kasan-dev, x86, linux-kernel
Cc: Dmitry Vyukov, Andrew Morton, linux-mm
Add arch_ prefix to all atomic operations and include
<asm-generic/atomic-instrumented.h>. This will allow
to add KASAN instrumentation to all atomic ops.
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: kasan-dev@googlegroups.com
Cc: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org
Cc: x86@kernel.org
---
Changes since v1:
- reverted unnecessary change in __raw_try_cmpxchg()
- reverted s/try_cmpxchg/arch_try_cmpxchg/ change
try_cmpxchg is x86 implementation detail
Changes since v3:
- drop comment in arch_atomic_set()
---
arch/x86/include/asm/atomic.h | 102 ++++++++++++++++++-----------------
arch/x86/include/asm/atomic64_32.h | 106 ++++++++++++++++++------------------
arch/x86/include/asm/atomic64_64.h | 108 ++++++++++++++++++-------------------
arch/x86/include/asm/cmpxchg.h | 12 ++---
arch/x86/include/asm/cmpxchg_32.h | 8 +--
arch/x86/include/asm/cmpxchg_64.h | 4 +-
6 files changed, 172 insertions(+), 168 deletions(-)
diff --git a/arch/x86/include/asm/atomic.h b/arch/x86/include/asm/atomic.h
index 8d7f6e579be4..304f4f676cce 100644
--- a/arch/x86/include/asm/atomic.h
+++ b/arch/x86/include/asm/atomic.h
@@ -16,36 +16,36 @@
#define ATOMIC_INIT(i) { (i) }
/**
- * atomic_read - read atomic variable
+ * arch_atomic_read - read atomic variable
* @v: pointer of type atomic_t
*
* Atomically reads the value of @v.
*/
-static __always_inline int atomic_read(const atomic_t *v)
+static __always_inline int arch_atomic_read(const atomic_t *v)
{
return READ_ONCE((v)->counter);
}
/**
- * atomic_set - set atomic variable
+ * arch_atomic_set - set atomic variable
* @v: pointer of type atomic_t
* @i: required value
*
* Atomically sets the value of @v to @i.
*/
-static __always_inline void atomic_set(atomic_t *v, int i)
+static __always_inline void arch_atomic_set(atomic_t *v, int i)
{
WRITE_ONCE(v->counter, i);
}
/**
- * atomic_add - add integer to atomic variable
+ * arch_atomic_add - add integer to atomic variable
* @i: integer value to add
* @v: pointer of type atomic_t
*
* Atomically adds @i to @v.
*/
-static __always_inline void atomic_add(int i, atomic_t *v)
+static __always_inline void arch_atomic_add(int i, atomic_t *v)
{
asm volatile(LOCK_PREFIX "addl %1,%0"
: "+m" (v->counter)
@@ -53,13 +53,13 @@ static __always_inline void atomic_add(int i, atomic_t *v)
}
/**
- * atomic_sub - subtract integer from atomic variable
+ * arch_atomic_sub - subtract integer from atomic variable
* @i: integer value to subtract
* @v: pointer of type atomic_t
*
* Atomically subtracts @i from @v.
*/
-static __always_inline void atomic_sub(int i, atomic_t *v)
+static __always_inline void arch_atomic_sub(int i, atomic_t *v)
{
asm volatile(LOCK_PREFIX "subl %1,%0"
: "+m" (v->counter)
@@ -67,7 +67,7 @@ static __always_inline void atomic_sub(int i, atomic_t *v)
}
/**
- * atomic_sub_and_test - subtract value from variable and test result
+ * arch_atomic_sub_and_test - subtract value from variable and test result
* @i: integer value to subtract
* @v: pointer of type atomic_t
*
@@ -75,63 +75,63 @@ static __always_inline void atomic_sub(int i, atomic_t *v)
* true if the result is zero, or false for all
* other cases.
*/
-static __always_inline bool atomic_sub_and_test(int i, atomic_t *v)
+static __always_inline bool arch_atomic_sub_and_test(int i, atomic_t *v)
{
GEN_BINARY_RMWcc(LOCK_PREFIX "subl", v->counter, "er", i, "%0", e);
}
/**
- * atomic_inc - increment atomic variable
+ * arch_atomic_inc - increment atomic variable
* @v: pointer of type atomic_t
*
* Atomically increments @v by 1.
*/
-static __always_inline void atomic_inc(atomic_t *v)
+static __always_inline void arch_atomic_inc(atomic_t *v)
{
asm volatile(LOCK_PREFIX "incl %0"
: "+m" (v->counter));
}
/**
- * atomic_dec - decrement atomic variable
+ * arch_atomic_dec - decrement atomic variable
* @v: pointer of type atomic_t
*
* Atomically decrements @v by 1.
*/
-static __always_inline void atomic_dec(atomic_t *v)
+static __always_inline void arch_atomic_dec(atomic_t *v)
{
asm volatile(LOCK_PREFIX "decl %0"
: "+m" (v->counter));
}
/**
- * atomic_dec_and_test - decrement and test
+ * arch_atomic_dec_and_test - decrement and test
* @v: pointer of type atomic_t
*
* Atomically decrements @v by 1 and
* returns true if the result is 0, or false for all other
* cases.
*/
-static __always_inline bool atomic_dec_and_test(atomic_t *v)
+static __always_inline bool arch_atomic_dec_and_test(atomic_t *v)
{
GEN_UNARY_RMWcc(LOCK_PREFIX "decl", v->counter, "%0", e);
}
/**
- * atomic_inc_and_test - increment and test
+ * arch_atomic_inc_and_test - increment and test
* @v: pointer of type atomic_t
*
* Atomically increments @v by 1
* and returns true if the result is zero, or false for all
* other cases.
*/
-static __always_inline bool atomic_inc_and_test(atomic_t *v)
+static __always_inline bool arch_atomic_inc_and_test(atomic_t *v)
{
GEN_UNARY_RMWcc(LOCK_PREFIX "incl", v->counter, "%0", e);
}
/**
- * atomic_add_negative - add and test if negative
+ * arch_atomic_add_negative - add and test if negative
* @i: integer value to add
* @v: pointer of type atomic_t
*
@@ -139,65 +139,65 @@ static __always_inline bool atomic_inc_and_test(atomic_t *v)
* if the result is negative, or false when
* result is greater than or equal to zero.
*/
-static __always_inline bool atomic_add_negative(int i, atomic_t *v)
+static __always_inline bool arch_atomic_add_negative(int i, atomic_t *v)
{
GEN_BINARY_RMWcc(LOCK_PREFIX "addl", v->counter, "er", i, "%0", s);
}
/**
- * atomic_add_return - add integer and return
+ * arch_atomic_add_return - add integer and return
* @i: integer value to add
* @v: pointer of type atomic_t
*
* Atomically adds @i to @v and returns @i + @v
*/
-static __always_inline int atomic_add_return(int i, atomic_t *v)
+static __always_inline int arch_atomic_add_return(int i, atomic_t *v)
{
return i + xadd(&v->counter, i);
}
/**
- * atomic_sub_return - subtract integer and return
+ * arch_atomic_sub_return - subtract integer and return
* @v: pointer of type atomic_t
* @i: integer value to subtract
*
* Atomically subtracts @i from @v and returns @v - @i
*/
-static __always_inline int atomic_sub_return(int i, atomic_t *v)
+static __always_inline int arch_atomic_sub_return(int i, atomic_t *v)
{
- return atomic_add_return(-i, v);
+ return arch_atomic_add_return(-i, v);
}
-#define atomic_inc_return(v) (atomic_add_return(1, v))
-#define atomic_dec_return(v) (atomic_sub_return(1, v))
+#define arch_atomic_inc_return(v) (arch_atomic_add_return(1, v))
+#define arch_atomic_dec_return(v) (arch_atomic_sub_return(1, v))
-static __always_inline int atomic_fetch_add(int i, atomic_t *v)
+static __always_inline int arch_atomic_fetch_add(int i, atomic_t *v)
{
return xadd(&v->counter, i);
}
-static __always_inline int atomic_fetch_sub(int i, atomic_t *v)
+static __always_inline int arch_atomic_fetch_sub(int i, atomic_t *v)
{
return xadd(&v->counter, -i);
}
-static __always_inline int atomic_cmpxchg(atomic_t *v, int old, int new)
+static __always_inline int arch_atomic_cmpxchg(atomic_t *v, int old, int new)
{
- return cmpxchg(&v->counter, old, new);
+ return arch_cmpxchg(&v->counter, old, new);
}
-#define atomic_try_cmpxchg atomic_try_cmpxchg
-static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new)
+#define arch_atomic_try_cmpxchg arch_atomic_try_cmpxchg
+static __always_inline bool arch_atomic_try_cmpxchg(atomic_t *v, int *old, int new)
{
return try_cmpxchg(&v->counter, old, new);
}
-static inline int atomic_xchg(atomic_t *v, int new)
+static inline int arch_atomic_xchg(atomic_t *v, int new)
{
return xchg(&v->counter, new);
}
-static inline void atomic_and(int i, atomic_t *v)
+static inline void arch_atomic_and(int i, atomic_t *v)
{
asm volatile(LOCK_PREFIX "andl %1,%0"
: "+m" (v->counter)
@@ -205,16 +205,16 @@ static inline void atomic_and(int i, atomic_t *v)
: "memory");
}
-static inline int atomic_fetch_and(int i, atomic_t *v)
+static inline int arch_atomic_fetch_and(int i, atomic_t *v)
{
- int val = atomic_read(v);
+ int val = arch_atomic_read(v);
do {
- } while (!atomic_try_cmpxchg(v, &val, val & i));
+ } while (!arch_atomic_try_cmpxchg(v, &val, val & i));
return val;
}
-static inline void atomic_or(int i, atomic_t *v)
+static inline void arch_atomic_or(int i, atomic_t *v)
{
asm volatile(LOCK_PREFIX "orl %1,%0"
: "+m" (v->counter)
@@ -222,17 +222,17 @@ static inline void atomic_or(int i, atomic_t *v)
: "memory");
}
-static inline int atomic_fetch_or(int i, atomic_t *v)
+static inline int arch_atomic_fetch_or(int i, atomic_t *v)
{
- int val = atomic_read(v);
+ int val = arch_atomic_read(v);
do {
- } while (!atomic_try_cmpxchg(v, &val, val | i));
+ } while (!arch_atomic_try_cmpxchg(v, &val, val | i));
return val;
}
-static inline void atomic_xor(int i, atomic_t *v)
+static inline void arch_atomic_xor(int i, atomic_t *v)
{
asm volatile(LOCK_PREFIX "xorl %1,%0"
: "+m" (v->counter)
@@ -240,17 +240,17 @@ static inline void atomic_xor(int i, atomic_t *v)
: "memory");
}
-static inline int atomic_fetch_xor(int i, atomic_t *v)
+static inline int arch_atomic_fetch_xor(int i, atomic_t *v)
{
- int val = atomic_read(v);
+ int val = arch_atomic_read(v);
do {
- } while (!atomic_try_cmpxchg(v, &val, val ^ i));
+ } while (!arch_atomic_try_cmpxchg(v, &val, val ^ i));
return val;
}
/**
- * __atomic_add_unless - add unless the number is already a given value
+ * __arch_atomic_add_unless - add unless the number is already a given value
* @v: pointer of type atomic_t
* @a: the amount to add to v...
* @u: ...unless v is equal to u.
@@ -258,13 +258,13 @@ static inline int atomic_fetch_xor(int i, atomic_t *v)
* Atomically adds @a to @v, so long as @v was not already @u.
* Returns the old value of @v.
*/
-static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u)
+static __always_inline int __arch_atomic_add_unless(atomic_t *v, int a, int u)
{
- int c = atomic_read(v);
+ int c = arch_atomic_read(v);
do {
if (unlikely(c == u))
break;
- } while (!atomic_try_cmpxchg(v, &c, c + a));
+ } while (!arch_atomic_try_cmpxchg(v, &c, c + a));
return c;
}
@@ -274,4 +274,6 @@ static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u)
# include <asm/atomic64_64.h>
#endif
+#include <asm-generic/atomic-instrumented.h>
+
#endif /* _ASM_X86_ATOMIC_H */
diff --git a/arch/x86/include/asm/atomic64_32.h b/arch/x86/include/asm/atomic64_32.h
index f107fef7bfcc..8501e4fc5054 100644
--- a/arch/x86/include/asm/atomic64_32.h
+++ b/arch/x86/include/asm/atomic64_32.h
@@ -61,7 +61,7 @@ ATOMIC64_DECL(add_unless);
#undef ATOMIC64_EXPORT
/**
- * atomic64_cmpxchg - cmpxchg atomic64 variable
+ * arch_atomic64_cmpxchg - cmpxchg atomic64 variable
* @v: pointer to type atomic64_t
* @o: expected value
* @n: new value
@@ -70,20 +70,21 @@ ATOMIC64_DECL(add_unless);
* the old value.
*/
-static inline long long atomic64_cmpxchg(atomic64_t *v, long long o, long long n)
+static inline long long arch_atomic64_cmpxchg(atomic64_t *v, long long o,
+ long long n)
{
- return cmpxchg64(&v->counter, o, n);
+ return arch_cmpxchg64(&v->counter, o, n);
}
/**
- * atomic64_xchg - xchg atomic64 variable
+ * arch_atomic64_xchg - xchg atomic64 variable
* @v: pointer to type atomic64_t
* @n: value to assign
*
* Atomically xchgs the value of @v to @n and returns
* the old value.
*/
-static inline long long atomic64_xchg(atomic64_t *v, long long n)
+static inline long long arch_atomic64_xchg(atomic64_t *v, long long n)
{
long long o;
unsigned high = (unsigned)(n >> 32);
@@ -95,13 +96,13 @@ static inline long long atomic64_xchg(atomic64_t *v, long long n)
}
/**
- * atomic64_set - set atomic64 variable
+ * arch_atomic64_set - set atomic64 variable
* @v: pointer to type atomic64_t
* @i: value to assign
*
* Atomically sets the value of @v to @n.
*/
-static inline void atomic64_set(atomic64_t *v, long long i)
+static inline void arch_atomic64_set(atomic64_t *v, long long i)
{
unsigned high = (unsigned)(i >> 32);
unsigned low = (unsigned)i;
@@ -111,12 +112,12 @@ static inline void atomic64_set(atomic64_t *v, long long i)
}
/**
- * atomic64_read - read atomic64 variable
+ * arch_atomic64_read - read atomic64 variable
* @v: pointer to type atomic64_t
*
* Atomically reads the value of @v and returns it.
*/
-static inline long long atomic64_read(const atomic64_t *v)
+static inline long long arch_atomic64_read(const atomic64_t *v)
{
long long r;
alternative_atomic64(read, "=&A" (r), "c" (v) : "memory");
@@ -124,13 +125,13 @@ static inline long long atomic64_read(const atomic64_t *v)
}
/**
- * atomic64_add_return - add and return
+ * arch_atomic64_add_return - add and return
* @i: integer value to add
* @v: pointer to type atomic64_t
*
* Atomically adds @i to @v and returns @i + *@v
*/
-static inline long long atomic64_add_return(long long i, atomic64_t *v)
+static inline long long arch_atomic64_add_return(long long i, atomic64_t *v)
{
alternative_atomic64(add_return,
ASM_OUTPUT2("+A" (i), "+c" (v)),
@@ -141,7 +142,7 @@ static inline long long atomic64_add_return(long long i, atomic64_t *v)
/*
* Other variants with different arithmetic operators:
*/
-static inline long long atomic64_sub_return(long long i, atomic64_t *v)
+static inline long long arch_atomic64_sub_return(long long i, atomic64_t *v)
{
alternative_atomic64(sub_return,
ASM_OUTPUT2("+A" (i), "+c" (v)),
@@ -149,7 +150,7 @@ static inline long long atomic64_sub_return(long long i, atomic64_t *v)
return i;
}
-static inline long long atomic64_inc_return(atomic64_t *v)
+static inline long long arch_atomic64_inc_return(atomic64_t *v)
{
long long a;
alternative_atomic64(inc_return, "=&A" (a),
@@ -157,7 +158,7 @@ static inline long long atomic64_inc_return(atomic64_t *v)
return a;
}
-static inline long long atomic64_dec_return(atomic64_t *v)
+static inline long long arch_atomic64_dec_return(atomic64_t *v)
{
long long a;
alternative_atomic64(dec_return, "=&A" (a),
@@ -166,13 +167,13 @@ static inline long long atomic64_dec_return(atomic64_t *v)
}
/**
- * atomic64_add - add integer to atomic64 variable
+ * arch_atomic64_add - add integer to atomic64 variable
* @i: integer value to add
* @v: pointer to type atomic64_t
*
* Atomically adds @i to @v.
*/
-static inline long long atomic64_add(long long i, atomic64_t *v)
+static inline long long arch_atomic64_add(long long i, atomic64_t *v)
{
__alternative_atomic64(add, add_return,
ASM_OUTPUT2("+A" (i), "+c" (v)),
@@ -181,13 +182,13 @@ static inline long long atomic64_add(long long i, atomic64_t *v)
}
/**
- * atomic64_sub - subtract the atomic64 variable
+ * arch_atomic64_sub - subtract the atomic64 variable
* @i: integer value to subtract
* @v: pointer to type atomic64_t
*
* Atomically subtracts @i from @v.
*/
-static inline long long atomic64_sub(long long i, atomic64_t *v)
+static inline long long arch_atomic64_sub(long long i, atomic64_t *v)
{
__alternative_atomic64(sub, sub_return,
ASM_OUTPUT2("+A" (i), "+c" (v)),
@@ -196,7 +197,7 @@ static inline long long atomic64_sub(long long i, atomic64_t *v)
}
/**
- * atomic64_sub_and_test - subtract value from variable and test result
+ * arch_atomic64_sub_and_test - subtract value from variable and test result
* @i: integer value to subtract
* @v: pointer to type atomic64_t
*
@@ -204,46 +205,46 @@ static inline long long atomic64_sub(long long i, atomic64_t *v)
* true if the result is zero, or false for all
* other cases.
*/
-static inline int atomic64_sub_and_test(long long i, atomic64_t *v)
+static inline int arch_atomic64_sub_and_test(long long i, atomic64_t *v)
{
- return atomic64_sub_return(i, v) == 0;
+ return arch_atomic64_sub_return(i, v) == 0;
}
/**
- * atomic64_inc - increment atomic64 variable
+ * arch_atomic64_inc - increment atomic64 variable
* @v: pointer to type atomic64_t
*
* Atomically increments @v by 1.
*/
-static inline void atomic64_inc(atomic64_t *v)
+static inline void arch_atomic64_inc(atomic64_t *v)
{
__alternative_atomic64(inc, inc_return, /* no output */,
"S" (v) : "memory", "eax", "ecx", "edx");
}
/**
- * atomic64_dec - decrement atomic64 variable
+ * arch_atomic64_dec - decrement atomic64 variable
* @v: pointer to type atomic64_t
*
* Atomically decrements @v by 1.
*/
-static inline void atomic64_dec(atomic64_t *v)
+static inline void arch_atomic64_dec(atomic64_t *v)
{
__alternative_atomic64(dec, dec_return, /* no output */,
"S" (v) : "memory", "eax", "ecx", "edx");
}
/**
- * atomic64_dec_and_test - decrement and test
+ * arch_atomic64_dec_and_test - decrement and test
* @v: pointer to type atomic64_t
*
* Atomically decrements @v by 1 and
* returns true if the result is 0, or false for all other
* cases.
*/
-static inline int atomic64_dec_and_test(atomic64_t *v)
+static inline int arch_atomic64_dec_and_test(atomic64_t *v)
{
- return atomic64_dec_return(v) == 0;
+ return arch_atomic64_dec_return(v) == 0;
}
/**
@@ -254,13 +255,13 @@ static inline int atomic64_dec_and_test(atomic64_t *v)
* and returns true if the result is zero, or false for all
* other cases.
*/
-static inline int atomic64_inc_and_test(atomic64_t *v)
+static inline int arch_atomic64_inc_and_test(atomic64_t *v)
{
- return atomic64_inc_return(v) == 0;
+ return arch_atomic64_inc_return(v) == 0;
}
/**
- * atomic64_add_negative - add and test if negative
+ * arch_atomic64_add_negative - add and test if negative
* @i: integer value to add
* @v: pointer to type atomic64_t
*
@@ -268,13 +269,13 @@ static inline int atomic64_inc_and_test(atomic64_t *v)
* if the result is negative, or false when
* result is greater than or equal to zero.
*/
-static inline int atomic64_add_negative(long long i, atomic64_t *v)
+static inline int arch_atomic64_add_negative(long long i, atomic64_t *v)
{
- return atomic64_add_return(i, v) < 0;
+ return arch_atomic64_add_return(i, v) < 0;
}
/**
- * atomic64_add_unless - add unless the number is a given value
+ * arch_atomic64_add_unless - add unless the number is a given value
* @v: pointer of type atomic64_t
* @a: the amount to add to v...
* @u: ...unless v is equal to u.
@@ -282,7 +283,8 @@ static inline int atomic64_add_negative(long long i, atomic64_t *v)
* Atomically adds @a to @v, so long as it was not @u.
* Returns non-zero if the add was done, zero otherwise.
*/
-static inline int atomic64_add_unless(atomic64_t *v, long long a, long long u)
+static inline int arch_atomic64_add_unless(atomic64_t *v, long long a,
+ long long u)
{
unsigned low = (unsigned)u;
unsigned high = (unsigned)(u >> 32);
@@ -293,7 +295,7 @@ static inline int atomic64_add_unless(atomic64_t *v, long long a, long long u)
}
-static inline int atomic64_inc_not_zero(atomic64_t *v)
+static inline int arch_atomic64_inc_not_zero(atomic64_t *v)
{
int r;
alternative_atomic64(inc_not_zero, "=&a" (r),
@@ -301,7 +303,7 @@ static inline int atomic64_inc_not_zero(atomic64_t *v)
return r;
}
-static inline long long atomic64_dec_if_positive(atomic64_t *v)
+static inline long long arch_atomic64_dec_if_positive(atomic64_t *v)
{
long long r;
alternative_atomic64(dec_if_positive, "=&A" (r),
@@ -312,66 +314,66 @@ static inline long long atomic64_dec_if_positive(atomic64_t *v)
#undef alternative_atomic64
#undef __alternative_atomic64
-static inline void atomic64_and(long long i, atomic64_t *v)
+static inline void arch_atomic64_and(long long i, atomic64_t *v)
{
long long old, c = 0;
- while ((old = atomic64_cmpxchg(v, c, c & i)) != c)
+ while ((old = arch_atomic64_cmpxchg(v, c, c & i)) != c)
c = old;
}
-static inline long long atomic64_fetch_and(long long i, atomic64_t *v)
+static inline long long arch_atomic64_fetch_and(long long i, atomic64_t *v)
{
long long old, c = 0;
- while ((old = atomic64_cmpxchg(v, c, c & i)) != c)
+ while ((old = arch_atomic64_cmpxchg(v, c, c & i)) != c)
c = old;
return old;
}
-static inline void atomic64_or(long long i, atomic64_t *v)
+static inline void arch_atomic64_or(long long i, atomic64_t *v)
{
long long old, c = 0;
- while ((old = atomic64_cmpxchg(v, c, c | i)) != c)
+ while ((old = arch_atomic64_cmpxchg(v, c, c | i)) != c)
c = old;
}
-static inline long long atomic64_fetch_or(long long i, atomic64_t *v)
+static inline long long arch_atomic64_fetch_or(long long i, atomic64_t *v)
{
long long old, c = 0;
- while ((old = atomic64_cmpxchg(v, c, c | i)) != c)
+ while ((old = arch_atomic64_cmpxchg(v, c, c | i)) != c)
c = old;
return old;
}
-static inline void atomic64_xor(long long i, atomic64_t *v)
+static inline void arch_atomic64_xor(long long i, atomic64_t *v)
{
long long old, c = 0;
- while ((old = atomic64_cmpxchg(v, c, c ^ i)) != c)
+ while ((old = arch_atomic64_cmpxchg(v, c, c ^ i)) != c)
c = old;
}
-static inline long long atomic64_fetch_xor(long long i, atomic64_t *v)
+static inline long long arch_atomic64_fetch_xor(long long i, atomic64_t *v)
{
long long old, c = 0;
- while ((old = atomic64_cmpxchg(v, c, c ^ i)) != c)
+ while ((old = arch_atomic64_cmpxchg(v, c, c ^ i)) != c)
c = old;
return old;
}
-static inline long long atomic64_fetch_add(long long i, atomic64_t *v)
+static inline long long arch_atomic64_fetch_add(long long i, atomic64_t *v)
{
long long old, c = 0;
- while ((old = atomic64_cmpxchg(v, c, c + i)) != c)
+ while ((old = arch_atomic64_cmpxchg(v, c, c + i)) != c)
c = old;
return old;
}
-#define atomic64_fetch_sub(i, v) atomic64_fetch_add(-(i), (v))
+#define arch_atomic64_fetch_sub(i, v) arch_atomic64_fetch_add(-(i), (v))
#endif /* _ASM_X86_ATOMIC64_32_H */
diff --git a/arch/x86/include/asm/atomic64_64.h b/arch/x86/include/asm/atomic64_64.h
index 5d9de36a2f04..5883b109d053 100644
--- a/arch/x86/include/asm/atomic64_64.h
+++ b/arch/x86/include/asm/atomic64_64.h
@@ -10,37 +10,37 @@
#define ATOMIC64_INIT(i) { (i) }
/**
- * atomic64_read - read atomic64 variable
+ * arch_atomic64_read - read atomic64 variable
* @v: pointer of type atomic64_t
*
* Atomically reads the value of @v.
* Doesn't imply a read memory barrier.
*/
-static inline long atomic64_read(const atomic64_t *v)
+static inline long arch_atomic64_read(const atomic64_t *v)
{
return READ_ONCE((v)->counter);
}
/**
- * atomic64_set - set atomic64 variable
+ * arch_atomic64_set - set atomic64 variable
* @v: pointer to type atomic64_t
* @i: required value
*
* Atomically sets the value of @v to @i.
*/
-static inline void atomic64_set(atomic64_t *v, long i)
+static inline void arch_atomic64_set(atomic64_t *v, long i)
{
WRITE_ONCE(v->counter, i);
}
/**
- * atomic64_add - add integer to atomic64 variable
+ * arch_atomic64_add - add integer to atomic64 variable
* @i: integer value to add
* @v: pointer to type atomic64_t
*
* Atomically adds @i to @v.
*/
-static __always_inline void atomic64_add(long i, atomic64_t *v)
+static __always_inline void arch_atomic64_add(long i, atomic64_t *v)
{
asm volatile(LOCK_PREFIX "addq %1,%0"
: "=m" (v->counter)
@@ -48,13 +48,13 @@ static __always_inline void atomic64_add(long i, atomic64_t *v)
}
/**
- * atomic64_sub - subtract the atomic64 variable
+ * arch_atomic64_sub - subtract the atomic64 variable
* @i: integer value to subtract
* @v: pointer to type atomic64_t
*
* Atomically subtracts @i from @v.
*/
-static inline void atomic64_sub(long i, atomic64_t *v)
+static inline void arch_atomic64_sub(long i, atomic64_t *v)
{
asm volatile(LOCK_PREFIX "subq %1,%0"
: "=m" (v->counter)
@@ -62,7 +62,7 @@ static inline void atomic64_sub(long i, atomic64_t *v)
}
/**
- * atomic64_sub_and_test - subtract value from variable and test result
+ * arch_atomic64_sub_and_test - subtract value from variable and test result
* @i: integer value to subtract
* @v: pointer to type atomic64_t
*
@@ -70,18 +70,18 @@ static inline void atomic64_sub(long i, atomic64_t *v)
* true if the result is zero, or false for all
* other cases.
*/
-static inline bool atomic64_sub_and_test(long i, atomic64_t *v)
+static inline bool arch_atomic64_sub_and_test(long i, atomic64_t *v)
{
GEN_BINARY_RMWcc(LOCK_PREFIX "subq", v->counter, "er", i, "%0", e);
}
/**
- * atomic64_inc - increment atomic64 variable
+ * arch_atomic64_inc - increment atomic64 variable
* @v: pointer to type atomic64_t
*
* Atomically increments @v by 1.
*/
-static __always_inline void atomic64_inc(atomic64_t *v)
+static __always_inline void arch_atomic64_inc(atomic64_t *v)
{
asm volatile(LOCK_PREFIX "incq %0"
: "=m" (v->counter)
@@ -89,12 +89,12 @@ static __always_inline void atomic64_inc(atomic64_t *v)
}
/**
- * atomic64_dec - decrement atomic64 variable
+ * arch_atomic64_dec - decrement atomic64 variable
* @v: pointer to type atomic64_t
*
* Atomically decrements @v by 1.
*/
-static __always_inline void atomic64_dec(atomic64_t *v)
+static __always_inline void arch_atomic64_dec(atomic64_t *v)
{
asm volatile(LOCK_PREFIX "decq %0"
: "=m" (v->counter)
@@ -102,33 +102,33 @@ static __always_inline void atomic64_dec(atomic64_t *v)
}
/**
- * atomic64_dec_and_test - decrement and test
+ * arch_atomic64_dec_and_test - decrement and test
* @v: pointer to type atomic64_t
*
* Atomically decrements @v by 1 and
* returns true if the result is 0, or false for all other
* cases.
*/
-static inline bool atomic64_dec_and_test(atomic64_t *v)
+static inline bool arch_atomic64_dec_and_test(atomic64_t *v)
{
GEN_UNARY_RMWcc(LOCK_PREFIX "decq", v->counter, "%0", e);
}
/**
- * atomic64_inc_and_test - increment and test
+ * arch_atomic64_inc_and_test - increment and test
* @v: pointer to type atomic64_t
*
* Atomically increments @v by 1
* and returns true if the result is zero, or false for all
* other cases.
*/
-static inline bool atomic64_inc_and_test(atomic64_t *v)
+static inline bool arch_atomic64_inc_and_test(atomic64_t *v)
{
GEN_UNARY_RMWcc(LOCK_PREFIX "incq", v->counter, "%0", e);
}
/**
- * atomic64_add_negative - add and test if negative
+ * arch_atomic64_add_negative - add and test if negative
* @i: integer value to add
* @v: pointer to type atomic64_t
*
@@ -136,59 +136,59 @@ static inline bool atomic64_inc_and_test(atomic64_t *v)
* if the result is negative, or false when
* result is greater than or equal to zero.
*/
-static inline bool atomic64_add_negative(long i, atomic64_t *v)
+static inline bool arch_atomic64_add_negative(long i, atomic64_t *v)
{
GEN_BINARY_RMWcc(LOCK_PREFIX "addq", v->counter, "er", i, "%0", s);
}
/**
- * atomic64_add_return - add and return
+ * arch_atomic64_add_return - add and return
* @i: integer value to add
* @v: pointer to type atomic64_t
*
* Atomically adds @i to @v and returns @i + @v
*/
-static __always_inline long atomic64_add_return(long i, atomic64_t *v)
+static __always_inline long arch_atomic64_add_return(long i, atomic64_t *v)
{
return i + xadd(&v->counter, i);
}
-static inline long atomic64_sub_return(long i, atomic64_t *v)
+static inline long arch_atomic64_sub_return(long i, atomic64_t *v)
{
- return atomic64_add_return(-i, v);
+ return arch_atomic64_add_return(-i, v);
}
-static inline long atomic64_fetch_add(long i, atomic64_t *v)
+static inline long arch_atomic64_fetch_add(long i, atomic64_t *v)
{
return xadd(&v->counter, i);
}
-static inline long atomic64_fetch_sub(long i, atomic64_t *v)
+static inline long arch_atomic64_fetch_sub(long i, atomic64_t *v)
{
return xadd(&v->counter, -i);
}
-#define atomic64_inc_return(v) (atomic64_add_return(1, (v)))
-#define atomic64_dec_return(v) (atomic64_sub_return(1, (v)))
+#define arch_atomic64_inc_return(v) (arch_atomic64_add_return(1, (v)))
+#define arch_atomic64_dec_return(v) (arch_atomic64_sub_return(1, (v)))
-static inline long atomic64_cmpxchg(atomic64_t *v, long old, long new)
+static inline long arch_atomic64_cmpxchg(atomic64_t *v, long old, long new)
{
- return cmpxchg(&v->counter, old, new);
+ return arch_cmpxchg(&v->counter, old, new);
}
-#define atomic64_try_cmpxchg atomic64_try_cmpxchg
-static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, long new)
+#define arch_atomic64_try_cmpxchg arch_atomic64_try_cmpxchg
+static __always_inline bool arch_atomic64_try_cmpxchg(atomic64_t *v, s64 *old, long new)
{
return try_cmpxchg(&v->counter, old, new);
}
-static inline long atomic64_xchg(atomic64_t *v, long new)
+static inline long arch_atomic64_xchg(atomic64_t *v, long new)
{
return xchg(&v->counter, new);
}
/**
- * atomic64_add_unless - add unless the number is a given value
+ * arch_atomic64_add_unless - add unless the number is a given value
* @v: pointer of type atomic64_t
* @a: the amount to add to v...
* @u: ...unless v is equal to u.
@@ -196,37 +196,37 @@ static inline long atomic64_xchg(atomic64_t *v, long new)
* Atomically adds @a to @v, so long as it was not @u.
* Returns the old value of @v.
*/
-static inline bool atomic64_add_unless(atomic64_t *v, long a, long u)
+static inline bool arch_atomic64_add_unless(atomic64_t *v, long a, long u)
{
- s64 c = atomic64_read(v);
+ s64 c = arch_atomic64_read(v);
do {
if (unlikely(c == u))
return false;
- } while (!atomic64_try_cmpxchg(v, &c, c + a));
+ } while (!arch_atomic64_try_cmpxchg(v, &c, c + a));
return true;
}
-#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0)
+#define arch_atomic64_inc_not_zero(v) arch_atomic64_add_unless((v), 1, 0)
/*
- * atomic64_dec_if_positive - decrement by 1 if old value positive
+ * arch_atomic64_dec_if_positive - decrement by 1 if old value positive
* @v: pointer of type atomic_t
*
* The function returns the old value of *v minus 1, even if
* the atomic variable, v, was not decremented.
*/
-static inline long atomic64_dec_if_positive(atomic64_t *v)
+static inline long arch_atomic64_dec_if_positive(atomic64_t *v)
{
- s64 dec, c = atomic64_read(v);
+ s64 dec, c = arch_atomic64_read(v);
do {
dec = c - 1;
if (unlikely(dec < 0))
break;
- } while (!atomic64_try_cmpxchg(v, &c, dec));
+ } while (!arch_atomic64_try_cmpxchg(v, &c, dec));
return dec;
}
-static inline void atomic64_and(long i, atomic64_t *v)
+static inline void arch_atomic64_and(long i, atomic64_t *v)
{
asm volatile(LOCK_PREFIX "andq %1,%0"
: "+m" (v->counter)
@@ -234,16 +234,16 @@ static inline void atomic64_and(long i, atomic64_t *v)
: "memory");
}
-static inline long atomic64_fetch_and(long i, atomic64_t *v)
+static inline long arch_atomic64_fetch_and(long i, atomic64_t *v)
{
- s64 val = atomic64_read(v);
+ s64 val = arch_atomic64_read(v);
do {
- } while (!atomic64_try_cmpxchg(v, &val, val & i));
+ } while (!arch_atomic64_try_cmpxchg(v, &val, val & i));
return val;
}
-static inline void atomic64_or(long i, atomic64_t *v)
+static inline void arch_atomic64_or(long i, atomic64_t *v)
{
asm volatile(LOCK_PREFIX "orq %1,%0"
: "+m" (v->counter)
@@ -251,16 +251,16 @@ static inline void atomic64_or(long i, atomic64_t *v)
: "memory");
}
-static inline long atomic64_fetch_or(long i, atomic64_t *v)
+static inline long arch_atomic64_fetch_or(long i, atomic64_t *v)
{
- s64 val = atomic64_read(v);
+ s64 val = arch_atomic64_read(v);
do {
- } while (!atomic64_try_cmpxchg(v, &val, val | i));
+ } while (!arch_atomic64_try_cmpxchg(v, &val, val | i));
return val;
}
-static inline void atomic64_xor(long i, atomic64_t *v)
+static inline void arch_atomic64_xor(long i, atomic64_t *v)
{
asm volatile(LOCK_PREFIX "xorq %1,%0"
: "+m" (v->counter)
@@ -268,12 +268,12 @@ static inline void atomic64_xor(long i, atomic64_t *v)
: "memory");
}
-static inline long atomic64_fetch_xor(long i, atomic64_t *v)
+static inline long arch_atomic64_fetch_xor(long i, atomic64_t *v)
{
- s64 val = atomic64_read(v);
+ s64 val = arch_atomic64_read(v);
do {
- } while (!atomic64_try_cmpxchg(v, &val, val ^ i));
+ } while (!arch_atomic64_try_cmpxchg(v, &val, val ^ i));
return val;
}
diff --git a/arch/x86/include/asm/cmpxchg.h b/arch/x86/include/asm/cmpxchg.h
index b5069e802d5c..9e2faa85eb02 100644
--- a/arch/x86/include/asm/cmpxchg.h
+++ b/arch/x86/include/asm/cmpxchg.h
@@ -144,13 +144,13 @@ extern void __add_wrong_size(void)
# include <asm/cmpxchg_64.h>
#endif
-#define cmpxchg(ptr, old, new) \
+#define arch_cmpxchg(ptr, old, new) \
__cmpxchg(ptr, old, new, sizeof(*(ptr)))
-#define sync_cmpxchg(ptr, old, new) \
+#define arch_sync_cmpxchg(ptr, old, new) \
__sync_cmpxchg(ptr, old, new, sizeof(*(ptr)))
-#define cmpxchg_local(ptr, old, new) \
+#define arch_cmpxchg_local(ptr, old, new) \
__cmpxchg_local(ptr, old, new, sizeof(*(ptr)))
@@ -220,7 +220,7 @@ extern void __add_wrong_size(void)
#define __try_cmpxchg(ptr, pold, new, size) \
__raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
-#define try_cmpxchg(ptr, pold, new) \
+#define try_cmpxchg(ptr, pold, new) \
__try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
/*
@@ -249,10 +249,10 @@ extern void __add_wrong_size(void)
__ret; \
})
-#define cmpxchg_double(p1, p2, o1, o2, n1, n2) \
+#define arch_cmpxchg_double(p1, p2, o1, o2, n1, n2) \
__cmpxchg_double(LOCK_PREFIX, p1, p2, o1, o2, n1, n2)
-#define cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \
+#define arch_cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \
__cmpxchg_double(, p1, p2, o1, o2, n1, n2)
#endif /* ASM_X86_CMPXCHG_H */
diff --git a/arch/x86/include/asm/cmpxchg_32.h b/arch/x86/include/asm/cmpxchg_32.h
index e4959d023af8..d897291d2bf9 100644
--- a/arch/x86/include/asm/cmpxchg_32.h
+++ b/arch/x86/include/asm/cmpxchg_32.h
@@ -35,10 +35,10 @@ static inline void set_64bit(volatile u64 *ptr, u64 value)
}
#ifdef CONFIG_X86_CMPXCHG64
-#define cmpxchg64(ptr, o, n) \
+#define arch_cmpxchg64(ptr, o, n) \
((__typeof__(*(ptr)))__cmpxchg64((ptr), (unsigned long long)(o), \
(unsigned long long)(n)))
-#define cmpxchg64_local(ptr, o, n) \
+#define arch_cmpxchg64_local(ptr, o, n) \
((__typeof__(*(ptr)))__cmpxchg64_local((ptr), (unsigned long long)(o), \
(unsigned long long)(n)))
#endif
@@ -75,7 +75,7 @@ static inline u64 __cmpxchg64_local(volatile u64 *ptr, u64 old, u64 new)
* to simulate the cmpxchg8b on the 80386 and 80486 CPU.
*/
-#define cmpxchg64(ptr, o, n) \
+#define arch_cmpxchg64(ptr, o, n) \
({ \
__typeof__(*(ptr)) __ret; \
__typeof__(*(ptr)) __old = (o); \
@@ -92,7 +92,7 @@ static inline u64 __cmpxchg64_local(volatile u64 *ptr, u64 old, u64 new)
__ret; })
-#define cmpxchg64_local(ptr, o, n) \
+#define arch_cmpxchg64_local(ptr, o, n) \
({ \
__typeof__(*(ptr)) __ret; \
__typeof__(*(ptr)) __old = (o); \
diff --git a/arch/x86/include/asm/cmpxchg_64.h b/arch/x86/include/asm/cmpxchg_64.h
index caa23a34c963..fafaebacca2d 100644
--- a/arch/x86/include/asm/cmpxchg_64.h
+++ b/arch/x86/include/asm/cmpxchg_64.h
@@ -6,13 +6,13 @@ static inline void set_64bit(volatile u64 *ptr, u64 val)
*ptr = val;
}
-#define cmpxchg64(ptr, o, n) \
+#define arch_cmpxchg64(ptr, o, n) \
({ \
BUILD_BUG_ON(sizeof(*(ptr)) != 8); \
cmpxchg((ptr), (o), (n)); \
})
-#define cmpxchg64_local(ptr, o, n) \
+#define arch_cmpxchg64_local(ptr, o, n) \
({ \
BUILD_BUG_ON(sizeof(*(ptr)) != 8); \
cmpxchg_local((ptr), (o), (n)); \
--
2.13.1.518.g3df882009-goog
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v4 5/7] kasan: allow kasan_check_read/write() to accept pointers to volatiles
2017-06-17 9:15 [PATCH v4 0/7] x86, kasan: add KASAN checks to atomic operations Dmitry Vyukov
` (3 preceding siblings ...)
2017-06-17 9:15 ` [PATCH v4 4/7] x86: switch atomic.h to use atomic-instrumented.h Dmitry Vyukov
@ 2017-06-17 9:15 ` Dmitry Vyukov
2017-06-19 10:50 ` Mark Rutland
2017-06-17 9:15 ` [PATCH v4 6/7] asm-generic: add KASAN instrumentation to atomic operations Dmitry Vyukov
2017-06-17 9:15 ` [PATCH v4 7/7] asm-generic, x86: add comments for atomic instrumentation Dmitry Vyukov
6 siblings, 1 reply; 38+ messages in thread
From: Dmitry Vyukov @ 2017-06-17 9:15 UTC (permalink / raw)
To: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin,
kasan-dev, x86, linux-kernel
Cc: Dmitry Vyukov, Thomas Gleixner, Andrew Morton, linux-mm
Currently kasan_check_read/write() accept 'const void*', make them
accept 'const volatile void*'. This is required for instrumentation
of atomic operations and there is just no reason to not allow that.
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: "H. Peter Anvin" <hpa@zytor.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: linux-kernel@vger.kernel.org
Cc: x86@kernel.org
Cc: linux-mm@kvack.org
Cc: kasan-dev@googlegroups.com
---
include/linux/kasan-checks.h | 10 ++++++----
mm/kasan/kasan.c | 4 ++--
2 files changed, 8 insertions(+), 6 deletions(-)
diff --git a/include/linux/kasan-checks.h b/include/linux/kasan-checks.h
index b7f8aced7870..41960fecf783 100644
--- a/include/linux/kasan-checks.h
+++ b/include/linux/kasan-checks.h
@@ -2,11 +2,13 @@
#define _LINUX_KASAN_CHECKS_H
#ifdef CONFIG_KASAN
-void kasan_check_read(const void *p, unsigned int size);
-void kasan_check_write(const void *p, unsigned int size);
+void kasan_check_read(const volatile void *p, unsigned int size);
+void kasan_check_write(const volatile void *p, unsigned int size);
#else
-static inline void kasan_check_read(const void *p, unsigned int size) { }
-static inline void kasan_check_write(const void *p, unsigned int size) { }
+static inline void kasan_check_read(const volatile void *p, unsigned int size)
+{ }
+static inline void kasan_check_write(const volatile void *p, unsigned int size)
+{ }
#endif
#endif
diff --git a/mm/kasan/kasan.c b/mm/kasan/kasan.c
index c81549d5c833..edacd161c0e5 100644
--- a/mm/kasan/kasan.c
+++ b/mm/kasan/kasan.c
@@ -333,13 +333,13 @@ static void check_memory_region(unsigned long addr,
check_memory_region_inline(addr, size, write, ret_ip);
}
-void kasan_check_read(const void *p, unsigned int size)
+void kasan_check_read(const volatile void *p, unsigned int size)
{
check_memory_region((unsigned long)p, size, false, _RET_IP_);
}
EXPORT_SYMBOL(kasan_check_read);
-void kasan_check_write(const void *p, unsigned int size)
+void kasan_check_write(const volatile void *p, unsigned int size)
{
check_memory_region((unsigned long)p, size, true, _RET_IP_);
}
--
2.13.1.518.g3df882009-goog
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v4 6/7] asm-generic: add KASAN instrumentation to atomic operations
2017-06-17 9:15 [PATCH v4 0/7] x86, kasan: add KASAN checks to atomic operations Dmitry Vyukov
` (4 preceding siblings ...)
2017-06-17 9:15 ` [PATCH v4 5/7] kasan: allow kasan_check_read/write() to accept pointers to volatiles Dmitry Vyukov
@ 2017-06-17 9:15 ` Dmitry Vyukov
2017-06-19 10:51 ` Mark Rutland
2017-06-17 9:15 ` [PATCH v4 7/7] asm-generic, x86: add comments for atomic instrumentation Dmitry Vyukov
6 siblings, 1 reply; 38+ messages in thread
From: Dmitry Vyukov @ 2017-06-17 9:15 UTC (permalink / raw)
To: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin,
kasan-dev, x86, linux-kernel
Cc: Dmitry Vyukov, Andrew Morton, linux-mm
KASAN uses compiler instrumentation to intercept all memory accesses.
But it does not see memory accesses done in assembly code.
One notable user of assembly code is atomic operations. Frequently,
for example, an atomic reference decrement is the last access to an
object and a good candidate for a racy use-after-free.
Add manual KASAN checks to atomic operations.
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Will Deacon <will.deacon@arm.com>,
Cc: Andrew Morton <akpm@linux-foundation.org>,
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>,
Cc: Ingo Molnar <mingo@redhat.com>,
Cc: kasan-dev@googlegroups.com
Cc: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org
Cc: x86@kernel.org
---
include/asm-generic/atomic-instrumented.h | 76 +++++++++++++++++++++++++++++--
1 file changed, 72 insertions(+), 4 deletions(-)
diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h
index 50401d925290..a0f5b7525bb2 100644
--- a/include/asm-generic/atomic-instrumented.h
+++ b/include/asm-generic/atomic-instrumented.h
@@ -1,43 +1,53 @@
#ifndef _LINUX_ATOMIC_INSTRUMENTED_H
#define _LINUX_ATOMIC_INSTRUMENTED_H
+#include <linux/kasan-checks.h>
+
static __always_inline int atomic_read(const atomic_t *v)
{
+ kasan_check_read(v, sizeof(*v));
return arch_atomic_read(v);
}
static __always_inline s64 atomic64_read(const atomic64_t *v)
{
+ kasan_check_read(v, sizeof(*v));
return arch_atomic64_read(v);
}
static __always_inline void atomic_set(atomic_t *v, int i)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_set(v, i);
}
static __always_inline void atomic64_set(atomic64_t *v, s64 i)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_set(v, i);
}
static __always_inline int atomic_xchg(atomic_t *v, int i)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_xchg(v, i);
}
static __always_inline s64 atomic64_xchg(atomic64_t *v, s64 i)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_xchg(v, i);
}
static __always_inline int atomic_cmpxchg(atomic_t *v, int old, int new)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_cmpxchg(v, old, new);
}
static __always_inline s64 atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_cmpxchg(v, old, new);
}
@@ -45,6 +55,8 @@ static __always_inline s64 atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new)
#define atomic_try_cmpxchg atomic_try_cmpxchg
static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new)
{
+ kasan_check_write(v, sizeof(*v));
+ kasan_check_read(old, sizeof(*old));
return arch_atomic_try_cmpxchg(v, old, new);
}
#endif
@@ -53,254 +65,310 @@ static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new)
#define atomic64_try_cmpxchg atomic64_try_cmpxchg
static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, s64 new)
{
+ kasan_check_write(v, sizeof(*v));
+ kasan_check_read(old, sizeof(*old));
return arch_atomic64_try_cmpxchg(v, old, new);
}
#endif
static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u)
{
+ kasan_check_write(v, sizeof(*v));
return __arch_atomic_add_unless(v, a, u);
}
static __always_inline bool atomic64_add_unless(atomic64_t *v, s64 a, s64 u)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_add_unless(v, a, u);
}
static __always_inline void atomic_inc(atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_inc(v);
}
static __always_inline void atomic64_inc(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_inc(v);
}
static __always_inline void atomic_dec(atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_dec(v);
}
static __always_inline void atomic64_dec(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_dec(v);
}
static __always_inline void atomic_add(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_add(i, v);
}
static __always_inline void atomic64_add(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_add(i, v);
}
static __always_inline void atomic_sub(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_sub(i, v);
}
static __always_inline void atomic64_sub(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_sub(i, v);
}
static __always_inline void atomic_and(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_and(i, v);
}
static __always_inline void atomic64_and(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_and(i, v);
}
static __always_inline void atomic_or(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_or(i, v);
}
static __always_inline void atomic64_or(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_or(i, v);
}
static __always_inline void atomic_xor(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_xor(i, v);
}
static __always_inline void atomic64_xor(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_xor(i, v);
}
static __always_inline int atomic_inc_return(atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_inc_return(v);
}
static __always_inline s64 atomic64_inc_return(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_inc_return(v);
}
static __always_inline int atomic_dec_return(atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_dec_return(v);
}
static __always_inline s64 atomic64_dec_return(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_dec_return(v);
}
static __always_inline s64 atomic64_inc_not_zero(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_inc_not_zero(v);
}
static __always_inline s64 atomic64_dec_if_positive(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_dec_if_positive(v);
}
static __always_inline bool atomic_dec_and_test(atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_dec_and_test(v);
}
static __always_inline bool atomic64_dec_and_test(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_dec_and_test(v);
}
static __always_inline bool atomic_inc_and_test(atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_inc_and_test(v);
}
static __always_inline bool atomic64_inc_and_test(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_inc_and_test(v);
}
static __always_inline int atomic_add_return(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_add_return(i, v);
}
static __always_inline s64 atomic64_add_return(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_add_return(i, v);
}
static __always_inline int atomic_sub_return(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_sub_return(i, v);
}
static __always_inline s64 atomic64_sub_return(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_sub_return(i, v);
}
static __always_inline int atomic_fetch_add(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_fetch_add(i, v);
}
static __always_inline s64 atomic64_fetch_add(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_fetch_add(i, v);
}
static __always_inline int atomic_fetch_sub(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_fetch_sub(i, v);
}
static __always_inline s64 atomic64_fetch_sub(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_fetch_sub(i, v);
}
static __always_inline int atomic_fetch_and(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_fetch_and(i, v);
}
static __always_inline s64 atomic64_fetch_and(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_fetch_and(i, v);
}
static __always_inline int atomic_fetch_or(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_fetch_or(i, v);
}
static __always_inline s64 atomic64_fetch_or(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_fetch_or(i, v);
}
static __always_inline int atomic_fetch_xor(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_fetch_xor(i, v);
}
static __always_inline s64 atomic64_fetch_xor(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_fetch_xor(i, v);
}
static __always_inline bool atomic_sub_and_test(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_sub_and_test(i, v);
}
static __always_inline bool atomic64_sub_and_test(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_sub_and_test(i, v);
}
static __always_inline bool atomic_add_negative(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_add_negative(i, v);
}
static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_add_negative(i, v);
}
#define cmpxchg(ptr, old, new) \
({ \
+ __typeof__(ptr) ___ptr = (ptr); \
+ kasan_check_write(___ptr, sizeof(*___ptr)); \
arch_cmpxchg((ptr), (old), (new)); \
})
#define sync_cmpxchg(ptr, old, new) \
({ \
- arch_sync_cmpxchg((ptr), (old), (new)); \
+ __typeof__(ptr) ___ptr = (ptr); \
+ kasan_check_write(___ptr, sizeof(*___ptr)); \
+ arch_sync_cmpxchg(___ptr, (old), (new)); \
})
#define cmpxchg_local(ptr, old, new) \
({ \
- arch_cmpxchg_local((ptr), (old), (new)); \
+ __typeof__(ptr) ____ptr = (ptr); \
+ kasan_check_write(____ptr, sizeof(*____ptr)); \
+ arch_cmpxchg_local(____ptr, (old), (new)); \
})
#define cmpxchg64(ptr, old, new) \
({ \
- arch_cmpxchg64((ptr), (old), (new)); \
+ __typeof__(ptr) ____ptr = (ptr); \
+ kasan_check_write(____ptr, sizeof(*____ptr)); \
+ arch_cmpxchg64(____ptr, (old), (new)); \
})
#define cmpxchg64_local(ptr, old, new) \
({ \
- arch_cmpxchg64_local((ptr), (old), (new)); \
+ __typeof__(ptr) ____ptr = (ptr); \
+ kasan_check_write(____ptr, sizeof(*____ptr)); \
+ arch_cmpxchg64_local(____ptr, (old), (new)); \
})
#define cmpxchg_double(p1, p2, o1, o2, n1, n2) \
--
2.13.1.518.g3df882009-goog
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v4 7/7] asm-generic, x86: add comments for atomic instrumentation
2017-06-17 9:15 [PATCH v4 0/7] x86, kasan: add KASAN checks to atomic operations Dmitry Vyukov
` (5 preceding siblings ...)
2017-06-17 9:15 ` [PATCH v4 6/7] asm-generic: add KASAN instrumentation to atomic operations Dmitry Vyukov
@ 2017-06-17 9:15 ` Dmitry Vyukov
2017-06-19 10:54 ` Mark Rutland
6 siblings, 1 reply; 38+ messages in thread
From: Dmitry Vyukov @ 2017-06-17 9:15 UTC (permalink / raw)
To: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin,
kasan-dev, x86, linux-kernel
Cc: Dmitry Vyukov, Andrew Morton, linux-mm
The comments are factored out from the code changes to make them
easier to read. Add them separately to explain some non-obvious
aspects.
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: kasan-dev@googlegroups.com
Cc: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org
Cc: x86@kernel.org
---
Changes since v3:
- rephrase comment in arch_atomic_read()
---
arch/x86/include/asm/atomic.h | 4 ++++
include/asm-generic/atomic-instrumented.h | 30 ++++++++++++++++++++++++++++++
2 files changed, 34 insertions(+)
diff --git a/arch/x86/include/asm/atomic.h b/arch/x86/include/asm/atomic.h
index 304f4f676cce..219c49b4d3aa 100644
--- a/arch/x86/include/asm/atomic.h
+++ b/arch/x86/include/asm/atomic.h
@@ -23,6 +23,10 @@
*/
static __always_inline int arch_atomic_read(const atomic_t *v)
{
+ /*
+ * Note for KASAN: we deliberately don't use READ_ONCE_NOCHECK() here,
+ * it's non-inlined function that increases binary size and stack usage.
+ */
return READ_ONCE((v)->counter);
}
diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h
index a0f5b7525bb2..5771439e7a31 100644
--- a/include/asm-generic/atomic-instrumented.h
+++ b/include/asm-generic/atomic-instrumented.h
@@ -1,3 +1,15 @@
+/*
+ * This file provides wrappers with KASAN instrumentation for atomic operations.
+ * To use this functionality an arch's atomic.h file needs to define all
+ * atomic operations with arch_ prefix (e.g. arch_atomic_read()) and include
+ * this file at the end. This file provides atomic_read() that forwards to
+ * arch_atomic_read() for actual atomic operation.
+ * Note: if an arch atomic operation is implemented by means of other atomic
+ * operations (e.g. atomic_read()/atomic_cmpxchg() loop), then it needs to use
+ * arch_ variants (i.e. arch_atomic_read()/arch_atomic_cmpxchg()) to avoid
+ * double instrumentation.
+ */
+
#ifndef _LINUX_ATOMIC_INSTRUMENTED_H
#define _LINUX_ATOMIC_INSTRUMENTED_H
@@ -336,6 +348,15 @@ static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v)
return arch_atomic64_add_negative(i, v);
}
+/*
+ * In the following macros we need to be careful to not clash with arch_ macros.
+ * arch_xchg() can be defined as an extended statement expression as well,
+ * if we define a __ptr variable, and arch_xchg() also defines __ptr variable,
+ * and we pass __ptr as an argument to arch_xchg(), it will use own __ptr
+ * instead of ours. This leads to unpleasant crashes. To avoid the problem
+ * the following macros declare variables with lots of underscores.
+ */
+
#define cmpxchg(ptr, old, new) \
({ \
__typeof__(ptr) ___ptr = (ptr); \
@@ -371,6 +392,15 @@ static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v)
arch_cmpxchg64_local(____ptr, (old), (new)); \
})
+/*
+ * Originally we had the following code here:
+ * __typeof__(p1) ____p1 = (p1);
+ * kasan_check_write(____p1, 2 * sizeof(*____p1));
+ * arch_cmpxchg_double(____p1, (p2), (o1), (o2), (n1), (n2));
+ * But it leads to compilation failures (see gcc issue 72873).
+ * So for now it's left non-instrumented.
+ * There are few callers of cmpxchg_double(), so it's not critical.
+ */
#define cmpxchg_double(p1, p2, o1, o2, n1, n2) \
({ \
arch_cmpxchg_double((p1), (p2), (o1), (o2), (n1), (n2)); \
--
2.13.1.518.g3df882009-goog
^ permalink raw reply related [flat|nested] 38+ messages in thread
* Re: [PATCH v4 5/7] kasan: allow kasan_check_read/write() to accept pointers to volatiles
2017-06-17 9:15 ` [PATCH v4 5/7] kasan: allow kasan_check_read/write() to accept pointers to volatiles Dmitry Vyukov
@ 2017-06-19 10:50 ` Mark Rutland
2017-06-19 13:11 ` Dmitry Vyukov
0 siblings, 1 reply; 38+ messages in thread
From: Mark Rutland @ 2017-06-19 10:50 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: peterz, mingo, will.deacon, hpa, aryabinin, kasan-dev, x86,
linux-kernel, Thomas Gleixner, Andrew Morton, linux-mm
On Sat, Jun 17, 2017 at 11:15:31AM +0200, Dmitry Vyukov wrote:
> Currently kasan_check_read/write() accept 'const void*', make them
> accept 'const volatile void*'. This is required for instrumentation
> of atomic operations and there is just no reason to not allow that.
>
> Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
> Cc: Thomas Gleixner <tglx@linutronix.de>
> Cc: "H. Peter Anvin" <hpa@zytor.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: linux-kernel@vger.kernel.org
> Cc: x86@kernel.org
> Cc: linux-mm@kvack.org
> Cc: kasan-dev@googlegroups.com
Looks sane to me, and I can confirm this doesn't advervsely affect
arm64. FWIW:
Acked-by: Mark Rutland <mark.rutland@arm.com>
Mark.
> ---
> include/linux/kasan-checks.h | 10 ++++++----
> mm/kasan/kasan.c | 4 ++--
> 2 files changed, 8 insertions(+), 6 deletions(-)
>
> diff --git a/include/linux/kasan-checks.h b/include/linux/kasan-checks.h
> index b7f8aced7870..41960fecf783 100644
> --- a/include/linux/kasan-checks.h
> +++ b/include/linux/kasan-checks.h
> @@ -2,11 +2,13 @@
> #define _LINUX_KASAN_CHECKS_H
>
> #ifdef CONFIG_KASAN
> -void kasan_check_read(const void *p, unsigned int size);
> -void kasan_check_write(const void *p, unsigned int size);
> +void kasan_check_read(const volatile void *p, unsigned int size);
> +void kasan_check_write(const volatile void *p, unsigned int size);
> #else
> -static inline void kasan_check_read(const void *p, unsigned int size) { }
> -static inline void kasan_check_write(const void *p, unsigned int size) { }
> +static inline void kasan_check_read(const volatile void *p, unsigned int size)
> +{ }
> +static inline void kasan_check_write(const volatile void *p, unsigned int size)
> +{ }
> #endif
>
> #endif
> diff --git a/mm/kasan/kasan.c b/mm/kasan/kasan.c
> index c81549d5c833..edacd161c0e5 100644
> --- a/mm/kasan/kasan.c
> +++ b/mm/kasan/kasan.c
> @@ -333,13 +333,13 @@ static void check_memory_region(unsigned long addr,
> check_memory_region_inline(addr, size, write, ret_ip);
> }
>
> -void kasan_check_read(const void *p, unsigned int size)
> +void kasan_check_read(const volatile void *p, unsigned int size)
> {
> check_memory_region((unsigned long)p, size, false, _RET_IP_);
> }
> EXPORT_SYMBOL(kasan_check_read);
>
> -void kasan_check_write(const void *p, unsigned int size)
> +void kasan_check_write(const volatile void *p, unsigned int size)
> {
> check_memory_region((unsigned long)p, size, true, _RET_IP_);
> }
> --
> 2.13.1.518.g3df882009-goog
>
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v4 3/7] asm-generic: add atomic-instrumented.h
2017-06-17 9:15 ` [PATCH v4 3/7] asm-generic: add atomic-instrumented.h Dmitry Vyukov
@ 2017-06-19 10:50 ` Mark Rutland
2017-06-22 11:05 ` [tip:locking/core] locking/atomic: Add asm-generic/atomic-instrumented.h tip-bot for Dmitry Vyukov
2018-03-12 12:23 ` [tip:locking/core] locking/atomic, asm-generic: " tip-bot for Dmitry Vyukov
2 siblings, 0 replies; 38+ messages in thread
From: Mark Rutland @ 2017-06-19 10:50 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: peterz, mingo, will.deacon, hpa, aryabinin, kasan-dev, x86,
linux-kernel, Andrew Morton, linux-mm
On Sat, Jun 17, 2017 at 11:15:29AM +0200, Dmitry Vyukov wrote:
> The new header allows to wrap per-arch atomic operations
> and add common functionality to all of them.
>
> Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Will Deacon <will.deacon@arm.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: kasan-dev@googlegroups.com
> Cc: linux-mm@kvack.org
> Cc: linux-kernel@vger.kernel.org
> Cc: x86@kernel.org
Looks sane to me, and I can see how this can be extended to cover all
the ordering vairants arm64 will require. I've pushed my WIP [1].
FWIW:
Acked-by: Mark Rutland <mark.rutland@arm.com>
Mark.
[1] https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/kasan-atomic
> ---
> include/asm-generic/atomic-instrumented.h | 316 ++++++++++++++++++++++++++++++
> 1 file changed, 316 insertions(+)
>
> diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h
> new file mode 100644
> index 000000000000..50401d925290
> --- /dev/null
> +++ b/include/asm-generic/atomic-instrumented.h
> @@ -0,0 +1,316 @@
> +#ifndef _LINUX_ATOMIC_INSTRUMENTED_H
> +#define _LINUX_ATOMIC_INSTRUMENTED_H
> +
> +static __always_inline int atomic_read(const atomic_t *v)
> +{
> + return arch_atomic_read(v);
> +}
> +
> +static __always_inline s64 atomic64_read(const atomic64_t *v)
> +{
> + return arch_atomic64_read(v);
> +}
> +
> +static __always_inline void atomic_set(atomic_t *v, int i)
> +{
> + arch_atomic_set(v, i);
> +}
> +
> +static __always_inline void atomic64_set(atomic64_t *v, s64 i)
> +{
> + arch_atomic64_set(v, i);
> +}
> +
> +static __always_inline int atomic_xchg(atomic_t *v, int i)
> +{
> + return arch_atomic_xchg(v, i);
> +}
> +
> +static __always_inline s64 atomic64_xchg(atomic64_t *v, s64 i)
> +{
> + return arch_atomic64_xchg(v, i);
> +}
> +
> +static __always_inline int atomic_cmpxchg(atomic_t *v, int old, int new)
> +{
> + return arch_atomic_cmpxchg(v, old, new);
> +}
> +
> +static __always_inline s64 atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new)
> +{
> + return arch_atomic64_cmpxchg(v, old, new);
> +}
> +
> +#ifdef arch_atomic_try_cmpxchg
> +#define atomic_try_cmpxchg atomic_try_cmpxchg
> +static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new)
> +{
> + return arch_atomic_try_cmpxchg(v, old, new);
> +}
> +#endif
> +
> +#ifdef arch_atomic64_try_cmpxchg
> +#define atomic64_try_cmpxchg atomic64_try_cmpxchg
> +static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, s64 new)
> +{
> + return arch_atomic64_try_cmpxchg(v, old, new);
> +}
> +#endif
> +
> +static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u)
> +{
> + return __arch_atomic_add_unless(v, a, u);
> +}
> +
> +
> +static __always_inline bool atomic64_add_unless(atomic64_t *v, s64 a, s64 u)
> +{
> + return arch_atomic64_add_unless(v, a, u);
> +}
> +
> +static __always_inline void atomic_inc(atomic_t *v)
> +{
> + arch_atomic_inc(v);
> +}
> +
> +static __always_inline void atomic64_inc(atomic64_t *v)
> +{
> + arch_atomic64_inc(v);
> +}
> +
> +static __always_inline void atomic_dec(atomic_t *v)
> +{
> + arch_atomic_dec(v);
> +}
> +
> +static __always_inline void atomic64_dec(atomic64_t *v)
> +{
> + arch_atomic64_dec(v);
> +}
> +
> +static __always_inline void atomic_add(int i, atomic_t *v)
> +{
> + arch_atomic_add(i, v);
> +}
> +
> +static __always_inline void atomic64_add(s64 i, atomic64_t *v)
> +{
> + arch_atomic64_add(i, v);
> +}
> +
> +static __always_inline void atomic_sub(int i, atomic_t *v)
> +{
> + arch_atomic_sub(i, v);
> +}
> +
> +static __always_inline void atomic64_sub(s64 i, atomic64_t *v)
> +{
> + arch_atomic64_sub(i, v);
> +}
> +
> +static __always_inline void atomic_and(int i, atomic_t *v)
> +{
> + arch_atomic_and(i, v);
> +}
> +
> +static __always_inline void atomic64_and(s64 i, atomic64_t *v)
> +{
> + arch_atomic64_and(i, v);
> +}
> +
> +static __always_inline void atomic_or(int i, atomic_t *v)
> +{
> + arch_atomic_or(i, v);
> +}
> +
> +static __always_inline void atomic64_or(s64 i, atomic64_t *v)
> +{
> + arch_atomic64_or(i, v);
> +}
> +
> +static __always_inline void atomic_xor(int i, atomic_t *v)
> +{
> + arch_atomic_xor(i, v);
> +}
> +
> +static __always_inline void atomic64_xor(s64 i, atomic64_t *v)
> +{
> + arch_atomic64_xor(i, v);
> +}
> +
> +static __always_inline int atomic_inc_return(atomic_t *v)
> +{
> + return arch_atomic_inc_return(v);
> +}
> +
> +static __always_inline s64 atomic64_inc_return(atomic64_t *v)
> +{
> + return arch_atomic64_inc_return(v);
> +}
> +
> +static __always_inline int atomic_dec_return(atomic_t *v)
> +{
> + return arch_atomic_dec_return(v);
> +}
> +
> +static __always_inline s64 atomic64_dec_return(atomic64_t *v)
> +{
> + return arch_atomic64_dec_return(v);
> +}
> +
> +static __always_inline s64 atomic64_inc_not_zero(atomic64_t *v)
> +{
> + return arch_atomic64_inc_not_zero(v);
> +}
> +
> +static __always_inline s64 atomic64_dec_if_positive(atomic64_t *v)
> +{
> + return arch_atomic64_dec_if_positive(v);
> +}
> +
> +static __always_inline bool atomic_dec_and_test(atomic_t *v)
> +{
> + return arch_atomic_dec_and_test(v);
> +}
> +
> +static __always_inline bool atomic64_dec_and_test(atomic64_t *v)
> +{
> + return arch_atomic64_dec_and_test(v);
> +}
> +
> +static __always_inline bool atomic_inc_and_test(atomic_t *v)
> +{
> + return arch_atomic_inc_and_test(v);
> +}
> +
> +static __always_inline bool atomic64_inc_and_test(atomic64_t *v)
> +{
> + return arch_atomic64_inc_and_test(v);
> +}
> +
> +static __always_inline int atomic_add_return(int i, atomic_t *v)
> +{
> + return arch_atomic_add_return(i, v);
> +}
> +
> +static __always_inline s64 atomic64_add_return(s64 i, atomic64_t *v)
> +{
> + return arch_atomic64_add_return(i, v);
> +}
> +
> +static __always_inline int atomic_sub_return(int i, atomic_t *v)
> +{
> + return arch_atomic_sub_return(i, v);
> +}
> +
> +static __always_inline s64 atomic64_sub_return(s64 i, atomic64_t *v)
> +{
> + return arch_atomic64_sub_return(i, v);
> +}
> +
> +static __always_inline int atomic_fetch_add(int i, atomic_t *v)
> +{
> + return arch_atomic_fetch_add(i, v);
> +}
> +
> +static __always_inline s64 atomic64_fetch_add(s64 i, atomic64_t *v)
> +{
> + return arch_atomic64_fetch_add(i, v);
> +}
> +
> +static __always_inline int atomic_fetch_sub(int i, atomic_t *v)
> +{
> + return arch_atomic_fetch_sub(i, v);
> +}
> +
> +static __always_inline s64 atomic64_fetch_sub(s64 i, atomic64_t *v)
> +{
> + return arch_atomic64_fetch_sub(i, v);
> +}
> +
> +static __always_inline int atomic_fetch_and(int i, atomic_t *v)
> +{
> + return arch_atomic_fetch_and(i, v);
> +}
> +
> +static __always_inline s64 atomic64_fetch_and(s64 i, atomic64_t *v)
> +{
> + return arch_atomic64_fetch_and(i, v);
> +}
> +
> +static __always_inline int atomic_fetch_or(int i, atomic_t *v)
> +{
> + return arch_atomic_fetch_or(i, v);
> +}
> +
> +static __always_inline s64 atomic64_fetch_or(s64 i, atomic64_t *v)
> +{
> + return arch_atomic64_fetch_or(i, v);
> +}
> +
> +static __always_inline int atomic_fetch_xor(int i, atomic_t *v)
> +{
> + return arch_atomic_fetch_xor(i, v);
> +}
> +
> +static __always_inline s64 atomic64_fetch_xor(s64 i, atomic64_t *v)
> +{
> + return arch_atomic64_fetch_xor(i, v);
> +}
> +
> +static __always_inline bool atomic_sub_and_test(int i, atomic_t *v)
> +{
> + return arch_atomic_sub_and_test(i, v);
> +}
> +
> +static __always_inline bool atomic64_sub_and_test(s64 i, atomic64_t *v)
> +{
> + return arch_atomic64_sub_and_test(i, v);
> +}
> +
> +static __always_inline bool atomic_add_negative(int i, atomic_t *v)
> +{
> + return arch_atomic_add_negative(i, v);
> +}
> +
> +static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v)
> +{
> + return arch_atomic64_add_negative(i, v);
> +}
> +
> +#define cmpxchg(ptr, old, new) \
> +({ \
> + arch_cmpxchg((ptr), (old), (new)); \
> +})
> +
> +#define sync_cmpxchg(ptr, old, new) \
> +({ \
> + arch_sync_cmpxchg((ptr), (old), (new)); \
> +})
> +
> +#define cmpxchg_local(ptr, old, new) \
> +({ \
> + arch_cmpxchg_local((ptr), (old), (new)); \
> +})
> +
> +#define cmpxchg64(ptr, old, new) \
> +({ \
> + arch_cmpxchg64((ptr), (old), (new)); \
> +})
> +
> +#define cmpxchg64_local(ptr, old, new) \
> +({ \
> + arch_cmpxchg64_local((ptr), (old), (new)); \
> +})
> +
> +#define cmpxchg_double(p1, p2, o1, o2, n1, n2) \
> +({ \
> + arch_cmpxchg_double((p1), (p2), (o1), (o2), (n1), (n2)); \
> +})
> +
> +#define cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \
> +({ \
> + arch_cmpxchg_double_local((p1), (p2), (o1), (o2), (n1), (n2)); \
> +})
> +
> +#endif /* _LINUX_ATOMIC_INSTRUMENTED_H */
> --
> 2.13.1.518.g3df882009-goog
>
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v4 6/7] asm-generic: add KASAN instrumentation to atomic operations
2017-06-17 9:15 ` [PATCH v4 6/7] asm-generic: add KASAN instrumentation to atomic operations Dmitry Vyukov
@ 2017-06-19 10:51 ` Mark Rutland
0 siblings, 0 replies; 38+ messages in thread
From: Mark Rutland @ 2017-06-19 10:51 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: peterz, mingo, will.deacon, hpa, aryabinin, kasan-dev, x86,
linux-kernel, Andrew Morton, linux-mm
On Sat, Jun 17, 2017 at 11:15:32AM +0200, Dmitry Vyukov wrote:
> KASAN uses compiler instrumentation to intercept all memory accesses.
> But it does not see memory accesses done in assembly code.
> One notable user of assembly code is atomic operations. Frequently,
> for example, an atomic reference decrement is the last access to an
> object and a good candidate for a racy use-after-free.
>
> Add manual KASAN checks to atomic operations.
>
> Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Will Deacon <will.deacon@arm.com>,
> Cc: Andrew Morton <akpm@linux-foundation.org>,
> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>,
> Cc: Ingo Molnar <mingo@redhat.com>,
> Cc: kasan-dev@googlegroups.com
> Cc: linux-mm@kvack.org
> Cc: linux-kernel@vger.kernel.org
> Cc: x86@kernel.org
FWIW:
Acked-by: Mark Rutland <mark.rutland@arm.com>
Mark.
> ---
> include/asm-generic/atomic-instrumented.h | 76 +++++++++++++++++++++++++++++--
> 1 file changed, 72 insertions(+), 4 deletions(-)
>
> diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h
> index 50401d925290..a0f5b7525bb2 100644
> --- a/include/asm-generic/atomic-instrumented.h
> +++ b/include/asm-generic/atomic-instrumented.h
> @@ -1,43 +1,53 @@
> #ifndef _LINUX_ATOMIC_INSTRUMENTED_H
> #define _LINUX_ATOMIC_INSTRUMENTED_H
>
> +#include <linux/kasan-checks.h>
> +
> static __always_inline int atomic_read(const atomic_t *v)
> {
> + kasan_check_read(v, sizeof(*v));
> return arch_atomic_read(v);
> }
>
> static __always_inline s64 atomic64_read(const atomic64_t *v)
> {
> + kasan_check_read(v, sizeof(*v));
> return arch_atomic64_read(v);
> }
>
> static __always_inline void atomic_set(atomic_t *v, int i)
> {
> + kasan_check_write(v, sizeof(*v));
> arch_atomic_set(v, i);
> }
>
> static __always_inline void atomic64_set(atomic64_t *v, s64 i)
> {
> + kasan_check_write(v, sizeof(*v));
> arch_atomic64_set(v, i);
> }
>
> static __always_inline int atomic_xchg(atomic_t *v, int i)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic_xchg(v, i);
> }
>
> static __always_inline s64 atomic64_xchg(atomic64_t *v, s64 i)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic64_xchg(v, i);
> }
>
> static __always_inline int atomic_cmpxchg(atomic_t *v, int old, int new)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic_cmpxchg(v, old, new);
> }
>
> static __always_inline s64 atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic64_cmpxchg(v, old, new);
> }
>
> @@ -45,6 +55,8 @@ static __always_inline s64 atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new)
> #define atomic_try_cmpxchg atomic_try_cmpxchg
> static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new)
> {
> + kasan_check_write(v, sizeof(*v));
> + kasan_check_read(old, sizeof(*old));
> return arch_atomic_try_cmpxchg(v, old, new);
> }
> #endif
> @@ -53,254 +65,310 @@ static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new)
> #define atomic64_try_cmpxchg atomic64_try_cmpxchg
> static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, s64 new)
> {
> + kasan_check_write(v, sizeof(*v));
> + kasan_check_read(old, sizeof(*old));
> return arch_atomic64_try_cmpxchg(v, old, new);
> }
> #endif
>
> static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u)
> {
> + kasan_check_write(v, sizeof(*v));
> return __arch_atomic_add_unless(v, a, u);
> }
>
>
> static __always_inline bool atomic64_add_unless(atomic64_t *v, s64 a, s64 u)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic64_add_unless(v, a, u);
> }
>
> static __always_inline void atomic_inc(atomic_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> arch_atomic_inc(v);
> }
>
> static __always_inline void atomic64_inc(atomic64_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> arch_atomic64_inc(v);
> }
>
> static __always_inline void atomic_dec(atomic_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> arch_atomic_dec(v);
> }
>
> static __always_inline void atomic64_dec(atomic64_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> arch_atomic64_dec(v);
> }
>
> static __always_inline void atomic_add(int i, atomic_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> arch_atomic_add(i, v);
> }
>
> static __always_inline void atomic64_add(s64 i, atomic64_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> arch_atomic64_add(i, v);
> }
>
> static __always_inline void atomic_sub(int i, atomic_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> arch_atomic_sub(i, v);
> }
>
> static __always_inline void atomic64_sub(s64 i, atomic64_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> arch_atomic64_sub(i, v);
> }
>
> static __always_inline void atomic_and(int i, atomic_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> arch_atomic_and(i, v);
> }
>
> static __always_inline void atomic64_and(s64 i, atomic64_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> arch_atomic64_and(i, v);
> }
>
> static __always_inline void atomic_or(int i, atomic_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> arch_atomic_or(i, v);
> }
>
> static __always_inline void atomic64_or(s64 i, atomic64_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> arch_atomic64_or(i, v);
> }
>
> static __always_inline void atomic_xor(int i, atomic_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> arch_atomic_xor(i, v);
> }
>
> static __always_inline void atomic64_xor(s64 i, atomic64_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> arch_atomic64_xor(i, v);
> }
>
> static __always_inline int atomic_inc_return(atomic_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic_inc_return(v);
> }
>
> static __always_inline s64 atomic64_inc_return(atomic64_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic64_inc_return(v);
> }
>
> static __always_inline int atomic_dec_return(atomic_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic_dec_return(v);
> }
>
> static __always_inline s64 atomic64_dec_return(atomic64_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic64_dec_return(v);
> }
>
> static __always_inline s64 atomic64_inc_not_zero(atomic64_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic64_inc_not_zero(v);
> }
>
> static __always_inline s64 atomic64_dec_if_positive(atomic64_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic64_dec_if_positive(v);
> }
>
> static __always_inline bool atomic_dec_and_test(atomic_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic_dec_and_test(v);
> }
>
> static __always_inline bool atomic64_dec_and_test(atomic64_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic64_dec_and_test(v);
> }
>
> static __always_inline bool atomic_inc_and_test(atomic_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic_inc_and_test(v);
> }
>
> static __always_inline bool atomic64_inc_and_test(atomic64_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic64_inc_and_test(v);
> }
>
> static __always_inline int atomic_add_return(int i, atomic_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic_add_return(i, v);
> }
>
> static __always_inline s64 atomic64_add_return(s64 i, atomic64_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic64_add_return(i, v);
> }
>
> static __always_inline int atomic_sub_return(int i, atomic_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic_sub_return(i, v);
> }
>
> static __always_inline s64 atomic64_sub_return(s64 i, atomic64_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic64_sub_return(i, v);
> }
>
> static __always_inline int atomic_fetch_add(int i, atomic_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic_fetch_add(i, v);
> }
>
> static __always_inline s64 atomic64_fetch_add(s64 i, atomic64_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic64_fetch_add(i, v);
> }
>
> static __always_inline int atomic_fetch_sub(int i, atomic_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic_fetch_sub(i, v);
> }
>
> static __always_inline s64 atomic64_fetch_sub(s64 i, atomic64_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic64_fetch_sub(i, v);
> }
>
> static __always_inline int atomic_fetch_and(int i, atomic_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic_fetch_and(i, v);
> }
>
> static __always_inline s64 atomic64_fetch_and(s64 i, atomic64_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic64_fetch_and(i, v);
> }
>
> static __always_inline int atomic_fetch_or(int i, atomic_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic_fetch_or(i, v);
> }
>
> static __always_inline s64 atomic64_fetch_or(s64 i, atomic64_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic64_fetch_or(i, v);
> }
>
> static __always_inline int atomic_fetch_xor(int i, atomic_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic_fetch_xor(i, v);
> }
>
> static __always_inline s64 atomic64_fetch_xor(s64 i, atomic64_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic64_fetch_xor(i, v);
> }
>
> static __always_inline bool atomic_sub_and_test(int i, atomic_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic_sub_and_test(i, v);
> }
>
> static __always_inline bool atomic64_sub_and_test(s64 i, atomic64_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic64_sub_and_test(i, v);
> }
>
> static __always_inline bool atomic_add_negative(int i, atomic_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic_add_negative(i, v);
> }
>
> static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v)
> {
> + kasan_check_write(v, sizeof(*v));
> return arch_atomic64_add_negative(i, v);
> }
>
> #define cmpxchg(ptr, old, new) \
> ({ \
> + __typeof__(ptr) ___ptr = (ptr); \
> + kasan_check_write(___ptr, sizeof(*___ptr)); \
> arch_cmpxchg((ptr), (old), (new)); \
> })
>
> #define sync_cmpxchg(ptr, old, new) \
> ({ \
> - arch_sync_cmpxchg((ptr), (old), (new)); \
> + __typeof__(ptr) ___ptr = (ptr); \
> + kasan_check_write(___ptr, sizeof(*___ptr)); \
> + arch_sync_cmpxchg(___ptr, (old), (new)); \
> })
>
> #define cmpxchg_local(ptr, old, new) \
> ({ \
> - arch_cmpxchg_local((ptr), (old), (new)); \
> + __typeof__(ptr) ____ptr = (ptr); \
> + kasan_check_write(____ptr, sizeof(*____ptr)); \
> + arch_cmpxchg_local(____ptr, (old), (new)); \
> })
>
> #define cmpxchg64(ptr, old, new) \
> ({ \
> - arch_cmpxchg64((ptr), (old), (new)); \
> + __typeof__(ptr) ____ptr = (ptr); \
> + kasan_check_write(____ptr, sizeof(*____ptr)); \
> + arch_cmpxchg64(____ptr, (old), (new)); \
> })
>
> #define cmpxchg64_local(ptr, old, new) \
> ({ \
> - arch_cmpxchg64_local((ptr), (old), (new)); \
> + __typeof__(ptr) ____ptr = (ptr); \
> + kasan_check_write(____ptr, sizeof(*____ptr)); \
> + arch_cmpxchg64_local(____ptr, (old), (new)); \
> })
>
> #define cmpxchg_double(p1, p2, o1, o2, n1, n2) \
> --
> 2.13.1.518.g3df882009-goog
>
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v4 7/7] asm-generic, x86: add comments for atomic instrumentation
2017-06-17 9:15 ` [PATCH v4 7/7] asm-generic, x86: add comments for atomic instrumentation Dmitry Vyukov
@ 2017-06-19 10:54 ` Mark Rutland
0 siblings, 0 replies; 38+ messages in thread
From: Mark Rutland @ 2017-06-19 10:54 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: peterz, mingo, will.deacon, hpa, aryabinin, kasan-dev, x86,
linux-kernel, Andrew Morton, linux-mm
On Sat, Jun 17, 2017 at 11:15:33AM +0200, Dmitry Vyukov wrote:
> The comments are factored out from the code changes to make them
> easier to read. Add them separately to explain some non-obvious
> aspects.
>
> Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Peter Zijlstra <peterz@infradead.org>
> Cc: Will Deacon <will.deacon@arm.com>
> Cc: Andrew Morton <akpm@linux-foundation.org>
> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
> Cc: Ingo Molnar <mingo@redhat.com>
> Cc: kasan-dev@googlegroups.com
> Cc: linux-mm@kvack.org
> Cc: linux-kernel@vger.kernel.org
> Cc: x86@kernel.org
>
The comments look sane to me.
When arm64 support comes round, it would be nice to instrument
cmpxchg_double(), since I think we're not affected by the compiler
issue. We can solve that as and when.
FWIW:
Acked-by: Mark Rutland <mark.rutland@arm.com>
Mark.
> ---
>
> Changes since v3:
> - rephrase comment in arch_atomic_read()
> ---
> arch/x86/include/asm/atomic.h | 4 ++++
> include/asm-generic/atomic-instrumented.h | 30 ++++++++++++++++++++++++++++++
> 2 files changed, 34 insertions(+)
>
> diff --git a/arch/x86/include/asm/atomic.h b/arch/x86/include/asm/atomic.h
> index 304f4f676cce..219c49b4d3aa 100644
> --- a/arch/x86/include/asm/atomic.h
> +++ b/arch/x86/include/asm/atomic.h
> @@ -23,6 +23,10 @@
> */
> static __always_inline int arch_atomic_read(const atomic_t *v)
> {
> + /*
> + * Note for KASAN: we deliberately don't use READ_ONCE_NOCHECK() here,
> + * it's non-inlined function that increases binary size and stack usage.
> + */
> return READ_ONCE((v)->counter);
> }
>
> diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h
> index a0f5b7525bb2..5771439e7a31 100644
> --- a/include/asm-generic/atomic-instrumented.h
> +++ b/include/asm-generic/atomic-instrumented.h
> @@ -1,3 +1,15 @@
> +/*
> + * This file provides wrappers with KASAN instrumentation for atomic operations.
> + * To use this functionality an arch's atomic.h file needs to define all
> + * atomic operations with arch_ prefix (e.g. arch_atomic_read()) and include
> + * this file at the end. This file provides atomic_read() that forwards to
> + * arch_atomic_read() for actual atomic operation.
> + * Note: if an arch atomic operation is implemented by means of other atomic
> + * operations (e.g. atomic_read()/atomic_cmpxchg() loop), then it needs to use
> + * arch_ variants (i.e. arch_atomic_read()/arch_atomic_cmpxchg()) to avoid
> + * double instrumentation.
> + */
> +
> #ifndef _LINUX_ATOMIC_INSTRUMENTED_H
> #define _LINUX_ATOMIC_INSTRUMENTED_H
>
> @@ -336,6 +348,15 @@ static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v)
> return arch_atomic64_add_negative(i, v);
> }
>
> +/*
> + * In the following macros we need to be careful to not clash with arch_ macros.
> + * arch_xchg() can be defined as an extended statement expression as well,
> + * if we define a __ptr variable, and arch_xchg() also defines __ptr variable,
> + * and we pass __ptr as an argument to arch_xchg(), it will use own __ptr
> + * instead of ours. This leads to unpleasant crashes. To avoid the problem
> + * the following macros declare variables with lots of underscores.
> + */
> +
> #define cmpxchg(ptr, old, new) \
> ({ \
> __typeof__(ptr) ___ptr = (ptr); \
> @@ -371,6 +392,15 @@ static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v)
> arch_cmpxchg64_local(____ptr, (old), (new)); \
> })
>
> +/*
> + * Originally we had the following code here:
> + * __typeof__(p1) ____p1 = (p1);
> + * kasan_check_write(____p1, 2 * sizeof(*____p1));
> + * arch_cmpxchg_double(____p1, (p2), (o1), (o2), (n1), (n2));
> + * But it leads to compilation failures (see gcc issue 72873).
> + * So for now it's left non-instrumented.
> + * There are few callers of cmpxchg_double(), so it's not critical.
> + */
> #define cmpxchg_double(p1, p2, o1, o2, n1, n2) \
> ({ \
> arch_cmpxchg_double((p1), (p2), (o1), (o2), (n1), (n2)); \
> --
> 2.13.1.518.g3df882009-goog
>
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v4 5/7] kasan: allow kasan_check_read/write() to accept pointers to volatiles
2017-06-19 10:50 ` Mark Rutland
@ 2017-06-19 13:11 ` Dmitry Vyukov
2017-06-22 8:25 ` Ingo Molnar
0 siblings, 1 reply; 38+ messages in thread
From: Dmitry Vyukov @ 2017-06-19 13:11 UTC (permalink / raw)
To: Mark Rutland
Cc: Peter Zijlstra, Ingo Molnar, Will Deacon, H. Peter Anvin,
Andrey Ryabinin, kasan-dev, x86, LKML, Thomas Gleixner,
Andrew Morton, linux-mm
On Mon, Jun 19, 2017 at 12:50 PM, Mark Rutland <mark.rutland@arm.com> wrote:
> On Sat, Jun 17, 2017 at 11:15:31AM +0200, Dmitry Vyukov wrote:
>> Currently kasan_check_read/write() accept 'const void*', make them
>> accept 'const volatile void*'. This is required for instrumentation
>> of atomic operations and there is just no reason to not allow that.
>>
>> Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
>> Cc: Mark Rutland <mark.rutland@arm.com>
>> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
>> Cc: Thomas Gleixner <tglx@linutronix.de>
>> Cc: "H. Peter Anvin" <hpa@zytor.com>
>> Cc: Peter Zijlstra <peterz@infradead.org>
>> Cc: Andrew Morton <akpm@linux-foundation.org>
>> Cc: linux-kernel@vger.kernel.org
>> Cc: x86@kernel.org
>> Cc: linux-mm@kvack.org
>> Cc: kasan-dev@googlegroups.com
>
> Looks sane to me, and I can confirm this doesn't advervsely affect
> arm64. FWIW:
>
> Acked-by: Mark Rutland <mark.rutland@arm.com>
>
> Mark.
Great! Thanks for testing.
Ingo, what are your thoughts? Are you taking this to locking tree? When?
>> ---
>> include/linux/kasan-checks.h | 10 ++++++----
>> mm/kasan/kasan.c | 4 ++--
>> 2 files changed, 8 insertions(+), 6 deletions(-)
>>
>> diff --git a/include/linux/kasan-checks.h b/include/linux/kasan-checks.h
>> index b7f8aced7870..41960fecf783 100644
>> --- a/include/linux/kasan-checks.h
>> +++ b/include/linux/kasan-checks.h
>> @@ -2,11 +2,13 @@
>> #define _LINUX_KASAN_CHECKS_H
>>
>> #ifdef CONFIG_KASAN
>> -void kasan_check_read(const void *p, unsigned int size);
>> -void kasan_check_write(const void *p, unsigned int size);
>> +void kasan_check_read(const volatile void *p, unsigned int size);
>> +void kasan_check_write(const volatile void *p, unsigned int size);
>> #else
>> -static inline void kasan_check_read(const void *p, unsigned int size) { }
>> -static inline void kasan_check_write(const void *p, unsigned int size) { }
>> +static inline void kasan_check_read(const volatile void *p, unsigned int size)
>> +{ }
>> +static inline void kasan_check_write(const volatile void *p, unsigned int size)
>> +{ }
>> #endif
>>
>> #endif
>> diff --git a/mm/kasan/kasan.c b/mm/kasan/kasan.c
>> index c81549d5c833..edacd161c0e5 100644
>> --- a/mm/kasan/kasan.c
>> +++ b/mm/kasan/kasan.c
>> @@ -333,13 +333,13 @@ static void check_memory_region(unsigned long addr,
>> check_memory_region_inline(addr, size, write, ret_ip);
>> }
>>
>> -void kasan_check_read(const void *p, unsigned int size)
>> +void kasan_check_read(const volatile void *p, unsigned int size)
>> {
>> check_memory_region((unsigned long)p, size, false, _RET_IP_);
>> }
>> EXPORT_SYMBOL(kasan_check_read);
>>
>> -void kasan_check_write(const void *p, unsigned int size)
>> +void kasan_check_write(const volatile void *p, unsigned int size)
>> {
>> check_memory_region((unsigned long)p, size, true, _RET_IP_);
>> }
>> --
>> 2.13.1.518.g3df882009-goog
>>
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v4 5/7] kasan: allow kasan_check_read/write() to accept pointers to volatiles
2017-06-19 13:11 ` Dmitry Vyukov
@ 2017-06-22 8:25 ` Ingo Molnar
2017-06-22 14:15 ` Dmitry Vyukov
0 siblings, 1 reply; 38+ messages in thread
From: Ingo Molnar @ 2017-06-22 8:25 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: Mark Rutland, Peter Zijlstra, Ingo Molnar, Will Deacon,
H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML,
Thomas Gleixner, Andrew Morton, linux-mm, Linus Torvalds
* Dmitry Vyukov <dvyukov@google.com> wrote:
> On Mon, Jun 19, 2017 at 12:50 PM, Mark Rutland <mark.rutland@arm.com> wrote:
> > On Sat, Jun 17, 2017 at 11:15:31AM +0200, Dmitry Vyukov wrote:
> >> Currently kasan_check_read/write() accept 'const void*', make them
> >> accept 'const volatile void*'. This is required for instrumentation
> >> of atomic operations and there is just no reason to not allow that.
> >>
> >> Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
> >> Cc: Mark Rutland <mark.rutland@arm.com>
> >> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
> >> Cc: Thomas Gleixner <tglx@linutronix.de>
> >> Cc: "H. Peter Anvin" <hpa@zytor.com>
> >> Cc: Peter Zijlstra <peterz@infradead.org>
> >> Cc: Andrew Morton <akpm@linux-foundation.org>
> >> Cc: linux-kernel@vger.kernel.org
> >> Cc: x86@kernel.org
> >> Cc: linux-mm@kvack.org
> >> Cc: kasan-dev@googlegroups.com
> >
> > Looks sane to me, and I can confirm this doesn't advervsely affect
> > arm64. FWIW:
> >
> > Acked-by: Mark Rutland <mark.rutland@arm.com>
> >
> > Mark.
>
>
> Great! Thanks for testing.
>
> Ingo, what are your thoughts? Are you taking this to locking tree? When?
Yeah, it all looks pretty clean to me too. I've applied the first three patches to
the locking tree, but did some minor stylistic cleanups to the first patch to
harmonize the style of the code - which made the later patches not apply cleanly.
Mind sending the remaining patches against the locking tree, tip:locking/core?
(Please also add in all the acks you got.)
This should also give people (Peter, Linus?) a last minute chance to object to my
suggestion of increasing the linecount in patch #1:
0f2376eb0ff8: locking/atomic/x86: Un-macro-ify atomic ops implementation
arch/x86/include/asm/atomic.h | 69 ++++++++++++++++++++++++++++++++++++++++++++++-----------------------
arch/x86/include/asm/atomic64_32.h | 81 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------------
arch/x86/include/asm/atomic64_64.h | 67 ++++++++++++++++++++++++++++++++++++++++++++-----------------------
3 files changed, 147 insertions(+), 70 deletions(-)
... to me the end result looks much more readable despite the +70 lines of code,
but if anyone feels strongly about this please holler!
Thanks,
Ingo
^ permalink raw reply [flat|nested] 38+ messages in thread
* [tip:locking/core] locking/atomic/x86: Un-macro-ify atomic ops implementation
2017-06-17 9:15 ` [PATCH v4 1/7] x86: un-macro-ify atomic ops implementation Dmitry Vyukov
@ 2017-06-22 11:04 ` tip-bot for Dmitry Vyukov
2017-07-25 13:54 ` tip-bot for Dmitry Vyukov
1 sibling, 0 replies; 38+ messages in thread
From: tip-bot for Dmitry Vyukov @ 2017-06-22 11:04 UTC (permalink / raw)
To: linux-tip-commits
Cc: mark.rutland, aryabinin, linux-kernel, dvyukov, mingo, torvalds,
peterz, hpa, tglx, akpm
Commit-ID: 0f2376eb0ff8851124c876eb81806d7ec1b421d1
Gitweb: http://git.kernel.org/tip/0f2376eb0ff8851124c876eb81806d7ec1b421d1
Author: Dmitry Vyukov <dvyukov@google.com>
AuthorDate: Sat, 17 Jun 2017 11:15:27 +0200
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Thu, 22 Jun 2017 10:19:56 +0200
locking/atomic/x86: Un-macro-ify atomic ops implementation
CPP turns perfectly readable code into a much harder to read syntactic soup.
Ingo suggested to write them out as-is in C and ignore the higher linecount.
Do this.
(As a side effect, plain C functions will be easier to KASAN-instrument as well.)
Suggested-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: kasan-dev@googlegroups.com
Cc: will.deacon@arm.com
Link: http://lkml.kernel.org/r/a35b983dd3be937a3cf63c4e2db487de2cdc7b8f.1497690003.git.dvyukov@google.com
[ Beautified the C code some more and twiddled the changelog
to mention the linecount increase and the KASAN benefit. ]
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/x86/include/asm/atomic.h | 69 +++++++++++++++++++++-----------
arch/x86/include/asm/atomic64_32.h | 81 +++++++++++++++++++++++++++-----------
arch/x86/include/asm/atomic64_64.h | 67 ++++++++++++++++++++-----------
3 files changed, 147 insertions(+), 70 deletions(-)
diff --git a/arch/x86/include/asm/atomic.h b/arch/x86/include/asm/atomic.h
index 33380b8..0874ebd 100644
--- a/arch/x86/include/asm/atomic.h
+++ b/arch/x86/include/asm/atomic.h
@@ -197,35 +197,56 @@ static inline int atomic_xchg(atomic_t *v, int new)
return xchg(&v->counter, new);
}
-#define ATOMIC_OP(op) \
-static inline void atomic_##op(int i, atomic_t *v) \
-{ \
- asm volatile(LOCK_PREFIX #op"l %1,%0" \
- : "+m" (v->counter) \
- : "ir" (i) \
- : "memory"); \
+static inline void atomic_and(int i, atomic_t *v)
+{
+ asm volatile(LOCK_PREFIX "andl %1,%0"
+ : "+m" (v->counter)
+ : "ir" (i)
+ : "memory");
+}
+
+static inline int atomic_fetch_and(int i, atomic_t *v)
+{
+ int val = atomic_read(v);
+
+ do { } while (!atomic_try_cmpxchg(v, &val, val & i));
+
+ return val;
}
-#define ATOMIC_FETCH_OP(op, c_op) \
-static inline int atomic_fetch_##op(int i, atomic_t *v) \
-{ \
- int val = atomic_read(v); \
- do { \
- } while (!atomic_try_cmpxchg(v, &val, val c_op i)); \
- return val; \
+static inline void atomic_or(int i, atomic_t *v)
+{
+ asm volatile(LOCK_PREFIX "orl %1,%0"
+ : "+m" (v->counter)
+ : "ir" (i)
+ : "memory");
}
-#define ATOMIC_OPS(op, c_op) \
- ATOMIC_OP(op) \
- ATOMIC_FETCH_OP(op, c_op)
+static inline int atomic_fetch_or(int i, atomic_t *v)
+{
+ int val = atomic_read(v);
-ATOMIC_OPS(and, &)
-ATOMIC_OPS(or , |)
-ATOMIC_OPS(xor, ^)
+ do { } while (!atomic_try_cmpxchg(v, &val, val | i));
-#undef ATOMIC_OPS
-#undef ATOMIC_FETCH_OP
-#undef ATOMIC_OP
+ return val;
+}
+
+static inline void atomic_xor(int i, atomic_t *v)
+{
+ asm volatile(LOCK_PREFIX "xorl %1,%0"
+ : "+m" (v->counter)
+ : "ir" (i)
+ : "memory");
+}
+
+static inline int atomic_fetch_xor(int i, atomic_t *v)
+{
+ int val = atomic_read(v);
+
+ do { } while (!atomic_try_cmpxchg(v, &val, val ^ i));
+
+ return val;
+}
/**
* __atomic_add_unless - add unless the number is already a given value
@@ -239,10 +260,12 @@ ATOMIC_OPS(xor, ^)
static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u)
{
int c = atomic_read(v);
+
do {
if (unlikely(c == u))
break;
} while (!atomic_try_cmpxchg(v, &c, c + a));
+
return c;
}
diff --git a/arch/x86/include/asm/atomic64_32.h b/arch/x86/include/asm/atomic64_32.h
index 71d7705..9e206f3 100644
--- a/arch/x86/include/asm/atomic64_32.h
+++ b/arch/x86/include/asm/atomic64_32.h
@@ -312,37 +312,70 @@ static inline long long atomic64_dec_if_positive(atomic64_t *v)
#undef alternative_atomic64
#undef __alternative_atomic64
-#define ATOMIC64_OP(op, c_op) \
-static inline void atomic64_##op(long long i, atomic64_t *v) \
-{ \
- long long old, c = 0; \
- while ((old = atomic64_cmpxchg(v, c, c c_op i)) != c) \
- c = old; \
+static inline void atomic64_and(long long i, atomic64_t *v)
+{
+ long long old, c = 0;
+
+ while ((old = atomic64_cmpxchg(v, c, c & i)) != c)
+ c = old;
}
-#define ATOMIC64_FETCH_OP(op, c_op) \
-static inline long long atomic64_fetch_##op(long long i, atomic64_t *v) \
-{ \
- long long old, c = 0; \
- while ((old = atomic64_cmpxchg(v, c, c c_op i)) != c) \
- c = old; \
- return old; \
+static inline long long atomic64_fetch_and(long long i, atomic64_t *v)
+{
+ long long old, c = 0;
+
+ while ((old = atomic64_cmpxchg(v, c, c & i)) != c)
+ c = old;
+
+ return old;
}
-ATOMIC64_FETCH_OP(add, +)
+static inline void atomic64_or(long long i, atomic64_t *v)
+{
+ long long old, c = 0;
-#define atomic64_fetch_sub(i, v) atomic64_fetch_add(-(i), (v))
+ while ((old = atomic64_cmpxchg(v, c, c | i)) != c)
+ c = old;
+}
+
+static inline long long atomic64_fetch_or(long long i, atomic64_t *v)
+{
+ long long old, c = 0;
+
+ while ((old = atomic64_cmpxchg(v, c, c | i)) != c)
+ c = old;
+
+ return old;
+}
-#define ATOMIC64_OPS(op, c_op) \
- ATOMIC64_OP(op, c_op) \
- ATOMIC64_FETCH_OP(op, c_op)
+static inline void atomic64_xor(long long i, atomic64_t *v)
+{
+ long long old, c = 0;
+
+ while ((old = atomic64_cmpxchg(v, c, c ^ i)) != c)
+ c = old;
+}
-ATOMIC64_OPS(and, &)
-ATOMIC64_OPS(or, |)
-ATOMIC64_OPS(xor, ^)
+static inline long long atomic64_fetch_xor(long long i, atomic64_t *v)
+{
+ long long old, c = 0;
+
+ while ((old = atomic64_cmpxchg(v, c, c ^ i)) != c)
+ c = old;
+
+ return old;
+}
-#undef ATOMIC64_OPS
-#undef ATOMIC64_FETCH_OP
-#undef ATOMIC64_OP
+static inline long long atomic64_fetch_add(long long i, atomic64_t *v)
+{
+ long long old, c = 0;
+
+ while ((old = atomic64_cmpxchg(v, c, c + i)) != c)
+ c = old;
+
+ return old;
+}
+
+#define atomic64_fetch_sub(i, v) atomic64_fetch_add(-(i), (v))
#endif /* _ASM_X86_ATOMIC64_32_H */
diff --git a/arch/x86/include/asm/atomic64_64.h b/arch/x86/include/asm/atomic64_64.h
index 6189a43..8db8879a 100644
--- a/arch/x86/include/asm/atomic64_64.h
+++ b/arch/x86/include/asm/atomic64_64.h
@@ -226,34 +226,55 @@ static inline long atomic64_dec_if_positive(atomic64_t *v)
return dec;
}
-#define ATOMIC64_OP(op) \
-static inline void atomic64_##op(long i, atomic64_t *v) \
-{ \
- asm volatile(LOCK_PREFIX #op"q %1,%0" \
- : "+m" (v->counter) \
- : "er" (i) \
- : "memory"); \
+static inline void atomic64_and(long i, atomic64_t *v)
+{
+ asm volatile(LOCK_PREFIX "andq %1,%0"
+ : "+m" (v->counter)
+ : "er" (i)
+ : "memory");
}
-#define ATOMIC64_FETCH_OP(op, c_op) \
-static inline long atomic64_fetch_##op(long i, atomic64_t *v) \
-{ \
- long val = atomic64_read(v); \
- do { \
- } while (!atomic64_try_cmpxchg(v, &val, val c_op i)); \
- return val; \
+static inline long atomic64_fetch_and(long i, atomic64_t *v)
+{
+ long val = atomic64_read(v);
+
+ do {
+ } while (!atomic64_try_cmpxchg(v, &val, val & i));
+ return val;
}
-#define ATOMIC64_OPS(op, c_op) \
- ATOMIC64_OP(op) \
- ATOMIC64_FETCH_OP(op, c_op)
+static inline void atomic64_or(long i, atomic64_t *v)
+{
+ asm volatile(LOCK_PREFIX "orq %1,%0"
+ : "+m" (v->counter)
+ : "er" (i)
+ : "memory");
+}
-ATOMIC64_OPS(and, &)
-ATOMIC64_OPS(or, |)
-ATOMIC64_OPS(xor, ^)
+static inline long atomic64_fetch_or(long i, atomic64_t *v)
+{
+ long val = atomic64_read(v);
-#undef ATOMIC64_OPS
-#undef ATOMIC64_FETCH_OP
-#undef ATOMIC64_OP
+ do {
+ } while (!atomic64_try_cmpxchg(v, &val, val | i));
+ return val;
+}
+
+static inline void atomic64_xor(long i, atomic64_t *v)
+{
+ asm volatile(LOCK_PREFIX "xorq %1,%0"
+ : "+m" (v->counter)
+ : "er" (i)
+ : "memory");
+}
+
+static inline long atomic64_fetch_xor(long i, atomic64_t *v)
+{
+ long val = atomic64_read(v);
+
+ do {
+ } while (!atomic64_try_cmpxchg(v, &val, val ^ i));
+ return val;
+}
#endif /* _ASM_X86_ATOMIC64_64_H */
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [tip:locking/core] locking/atomic/x86: Use 's64 *' for 'old' argument of atomic64_try_cmpxchg()
2017-06-17 9:15 ` [PATCH v4 2/7] x86: use s64* for old arg of atomic64_try_cmpxchg() Dmitry Vyukov
@ 2017-06-22 11:04 ` tip-bot for Dmitry Vyukov
2017-07-25 13:55 ` tip-bot for Dmitry Vyukov
1 sibling, 0 replies; 38+ messages in thread
From: tip-bot for Dmitry Vyukov @ 2017-06-22 11:04 UTC (permalink / raw)
To: linux-tip-commits
Cc: hpa, linux-kernel, tglx, akpm, aryabinin, mingo, peterz,
will.deacon, torvalds, mark.rutland, dvyukov
Commit-ID: f6dda790094b0d658b59cf108c52805f1f7c11e6
Gitweb: http://git.kernel.org/tip/f6dda790094b0d658b59cf108c52805f1f7c11e6
Author: Dmitry Vyukov <dvyukov@google.com>
AuthorDate: Sat, 17 Jun 2017 11:15:28 +0200
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Thu, 22 Jun 2017 10:19:56 +0200
locking/atomic/x86: Use 's64 *' for 'old' argument of atomic64_try_cmpxchg()
atomic64_try_cmpxchg() declares old argument as 'long *',
this makes it impossible to use it in portable code.
If caller passes 'long *', it becomes 32-bits on 32-bit arches.
If caller passes 's64 *', it does not compile on x86_64.
Change type of old argument to 's64 *' instead.
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Will Deacon <will.deacon@arm.com>
Cc: kasan-dev@googlegroups.com
Cc: linux-mm@kvack.org
Link: http://lkml.kernel.org/r/fa6f77f2375150d26ea796a77e8b59195fd2ab13.1497690003.git.dvyukov@google.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/x86/include/asm/atomic64_64.h | 12 ++++++------
arch/x86/include/asm/cmpxchg.h | 2 +-
2 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/arch/x86/include/asm/atomic64_64.h b/arch/x86/include/asm/atomic64_64.h
index 8db8879a..5d9de36 100644
--- a/arch/x86/include/asm/atomic64_64.h
+++ b/arch/x86/include/asm/atomic64_64.h
@@ -177,7 +177,7 @@ static inline long atomic64_cmpxchg(atomic64_t *v, long old, long new)
}
#define atomic64_try_cmpxchg atomic64_try_cmpxchg
-static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, long *old, long new)
+static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, long new)
{
return try_cmpxchg(&v->counter, old, new);
}
@@ -198,7 +198,7 @@ static inline long atomic64_xchg(atomic64_t *v, long new)
*/
static inline bool atomic64_add_unless(atomic64_t *v, long a, long u)
{
- long c = atomic64_read(v);
+ s64 c = atomic64_read(v);
do {
if (unlikely(c == u))
return false;
@@ -217,7 +217,7 @@ static inline bool atomic64_add_unless(atomic64_t *v, long a, long u)
*/
static inline long atomic64_dec_if_positive(atomic64_t *v)
{
- long dec, c = atomic64_read(v);
+ s64 dec, c = atomic64_read(v);
do {
dec = c - 1;
if (unlikely(dec < 0))
@@ -236,7 +236,7 @@ static inline void atomic64_and(long i, atomic64_t *v)
static inline long atomic64_fetch_and(long i, atomic64_t *v)
{
- long val = atomic64_read(v);
+ s64 val = atomic64_read(v);
do {
} while (!atomic64_try_cmpxchg(v, &val, val & i));
@@ -253,7 +253,7 @@ static inline void atomic64_or(long i, atomic64_t *v)
static inline long atomic64_fetch_or(long i, atomic64_t *v)
{
- long val = atomic64_read(v);
+ s64 val = atomic64_read(v);
do {
} while (!atomic64_try_cmpxchg(v, &val, val | i));
@@ -270,7 +270,7 @@ static inline void atomic64_xor(long i, atomic64_t *v)
static inline long atomic64_fetch_xor(long i, atomic64_t *v)
{
- long val = atomic64_read(v);
+ s64 val = atomic64_read(v);
do {
} while (!atomic64_try_cmpxchg(v, &val, val ^ i));
diff --git a/arch/x86/include/asm/cmpxchg.h b/arch/x86/include/asm/cmpxchg.h
index d90296d..b5069e8 100644
--- a/arch/x86/include/asm/cmpxchg.h
+++ b/arch/x86/include/asm/cmpxchg.h
@@ -157,7 +157,7 @@ extern void __add_wrong_size(void)
#define __raw_try_cmpxchg(_ptr, _pold, _new, size, lock) \
({ \
bool success; \
- __typeof__(_ptr) _old = (_pold); \
+ __typeof__(_ptr) _old = (__typeof__(_ptr))(_pold); \
__typeof__(*(_ptr)) __old = *_old; \
__typeof__(*(_ptr)) __new = (_new); \
switch (size) { \
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [tip:locking/core] locking/atomic: Add asm-generic/atomic-instrumented.h
2017-06-17 9:15 ` [PATCH v4 3/7] asm-generic: add atomic-instrumented.h Dmitry Vyukov
2017-06-19 10:50 ` Mark Rutland
@ 2017-06-22 11:05 ` tip-bot for Dmitry Vyukov
2018-03-12 12:23 ` [tip:locking/core] locking/atomic, asm-generic: " tip-bot for Dmitry Vyukov
2 siblings, 0 replies; 38+ messages in thread
From: tip-bot for Dmitry Vyukov @ 2017-06-22 11:05 UTC (permalink / raw)
To: linux-tip-commits
Cc: dvyukov, akpm, peterz, tglx, torvalds, will.deacon, mark.rutland,
mingo, linux-kernel, hpa, aryabinin
Commit-ID: ff5060d9b2a73917dbf52654e29b08018e22a1f9
Gitweb: http://git.kernel.org/tip/ff5060d9b2a73917dbf52654e29b08018e22a1f9
Author: Dmitry Vyukov <dvyukov@google.com>
AuthorDate: Sat, 17 Jun 2017 11:15:29 +0200
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Thu, 22 Jun 2017 10:20:12 +0200
locking/atomic: Add asm-generic/atomic-instrumented.h
The new header allows to wrap per-arch atomic operations
and add common functionality to all of them.
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Will Deacon <will.deacon@arm.com>
Cc: kasan-dev@googlegroups.com
Cc: linux-mm@kvack.org
Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
include/asm-generic/atomic-instrumented.h | 316 ++++++++++++++++++++++++++++++
1 file changed, 316 insertions(+)
diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h
new file mode 100644
index 0000000..50401d9
--- /dev/null
+++ b/include/asm-generic/atomic-instrumented.h
@@ -0,0 +1,316 @@
+#ifndef _LINUX_ATOMIC_INSTRUMENTED_H
+#define _LINUX_ATOMIC_INSTRUMENTED_H
+
+static __always_inline int atomic_read(const atomic_t *v)
+{
+ return arch_atomic_read(v);
+}
+
+static __always_inline s64 atomic64_read(const atomic64_t *v)
+{
+ return arch_atomic64_read(v);
+}
+
+static __always_inline void atomic_set(atomic_t *v, int i)
+{
+ arch_atomic_set(v, i);
+}
+
+static __always_inline void atomic64_set(atomic64_t *v, s64 i)
+{
+ arch_atomic64_set(v, i);
+}
+
+static __always_inline int atomic_xchg(atomic_t *v, int i)
+{
+ return arch_atomic_xchg(v, i);
+}
+
+static __always_inline s64 atomic64_xchg(atomic64_t *v, s64 i)
+{
+ return arch_atomic64_xchg(v, i);
+}
+
+static __always_inline int atomic_cmpxchg(atomic_t *v, int old, int new)
+{
+ return arch_atomic_cmpxchg(v, old, new);
+}
+
+static __always_inline s64 atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new)
+{
+ return arch_atomic64_cmpxchg(v, old, new);
+}
+
+#ifdef arch_atomic_try_cmpxchg
+#define atomic_try_cmpxchg atomic_try_cmpxchg
+static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new)
+{
+ return arch_atomic_try_cmpxchg(v, old, new);
+}
+#endif
+
+#ifdef arch_atomic64_try_cmpxchg
+#define atomic64_try_cmpxchg atomic64_try_cmpxchg
+static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, s64 new)
+{
+ return arch_atomic64_try_cmpxchg(v, old, new);
+}
+#endif
+
+static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u)
+{
+ return __arch_atomic_add_unless(v, a, u);
+}
+
+
+static __always_inline bool atomic64_add_unless(atomic64_t *v, s64 a, s64 u)
+{
+ return arch_atomic64_add_unless(v, a, u);
+}
+
+static __always_inline void atomic_inc(atomic_t *v)
+{
+ arch_atomic_inc(v);
+}
+
+static __always_inline void atomic64_inc(atomic64_t *v)
+{
+ arch_atomic64_inc(v);
+}
+
+static __always_inline void atomic_dec(atomic_t *v)
+{
+ arch_atomic_dec(v);
+}
+
+static __always_inline void atomic64_dec(atomic64_t *v)
+{
+ arch_atomic64_dec(v);
+}
+
+static __always_inline void atomic_add(int i, atomic_t *v)
+{
+ arch_atomic_add(i, v);
+}
+
+static __always_inline void atomic64_add(s64 i, atomic64_t *v)
+{
+ arch_atomic64_add(i, v);
+}
+
+static __always_inline void atomic_sub(int i, atomic_t *v)
+{
+ arch_atomic_sub(i, v);
+}
+
+static __always_inline void atomic64_sub(s64 i, atomic64_t *v)
+{
+ arch_atomic64_sub(i, v);
+}
+
+static __always_inline void atomic_and(int i, atomic_t *v)
+{
+ arch_atomic_and(i, v);
+}
+
+static __always_inline void atomic64_and(s64 i, atomic64_t *v)
+{
+ arch_atomic64_and(i, v);
+}
+
+static __always_inline void atomic_or(int i, atomic_t *v)
+{
+ arch_atomic_or(i, v);
+}
+
+static __always_inline void atomic64_or(s64 i, atomic64_t *v)
+{
+ arch_atomic64_or(i, v);
+}
+
+static __always_inline void atomic_xor(int i, atomic_t *v)
+{
+ arch_atomic_xor(i, v);
+}
+
+static __always_inline void atomic64_xor(s64 i, atomic64_t *v)
+{
+ arch_atomic64_xor(i, v);
+}
+
+static __always_inline int atomic_inc_return(atomic_t *v)
+{
+ return arch_atomic_inc_return(v);
+}
+
+static __always_inline s64 atomic64_inc_return(atomic64_t *v)
+{
+ return arch_atomic64_inc_return(v);
+}
+
+static __always_inline int atomic_dec_return(atomic_t *v)
+{
+ return arch_atomic_dec_return(v);
+}
+
+static __always_inline s64 atomic64_dec_return(atomic64_t *v)
+{
+ return arch_atomic64_dec_return(v);
+}
+
+static __always_inline s64 atomic64_inc_not_zero(atomic64_t *v)
+{
+ return arch_atomic64_inc_not_zero(v);
+}
+
+static __always_inline s64 atomic64_dec_if_positive(atomic64_t *v)
+{
+ return arch_atomic64_dec_if_positive(v);
+}
+
+static __always_inline bool atomic_dec_and_test(atomic_t *v)
+{
+ return arch_atomic_dec_and_test(v);
+}
+
+static __always_inline bool atomic64_dec_and_test(atomic64_t *v)
+{
+ return arch_atomic64_dec_and_test(v);
+}
+
+static __always_inline bool atomic_inc_and_test(atomic_t *v)
+{
+ return arch_atomic_inc_and_test(v);
+}
+
+static __always_inline bool atomic64_inc_and_test(atomic64_t *v)
+{
+ return arch_atomic64_inc_and_test(v);
+}
+
+static __always_inline int atomic_add_return(int i, atomic_t *v)
+{
+ return arch_atomic_add_return(i, v);
+}
+
+static __always_inline s64 atomic64_add_return(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_add_return(i, v);
+}
+
+static __always_inline int atomic_sub_return(int i, atomic_t *v)
+{
+ return arch_atomic_sub_return(i, v);
+}
+
+static __always_inline s64 atomic64_sub_return(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_sub_return(i, v);
+}
+
+static __always_inline int atomic_fetch_add(int i, atomic_t *v)
+{
+ return arch_atomic_fetch_add(i, v);
+}
+
+static __always_inline s64 atomic64_fetch_add(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_fetch_add(i, v);
+}
+
+static __always_inline int atomic_fetch_sub(int i, atomic_t *v)
+{
+ return arch_atomic_fetch_sub(i, v);
+}
+
+static __always_inline s64 atomic64_fetch_sub(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_fetch_sub(i, v);
+}
+
+static __always_inline int atomic_fetch_and(int i, atomic_t *v)
+{
+ return arch_atomic_fetch_and(i, v);
+}
+
+static __always_inline s64 atomic64_fetch_and(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_fetch_and(i, v);
+}
+
+static __always_inline int atomic_fetch_or(int i, atomic_t *v)
+{
+ return arch_atomic_fetch_or(i, v);
+}
+
+static __always_inline s64 atomic64_fetch_or(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_fetch_or(i, v);
+}
+
+static __always_inline int atomic_fetch_xor(int i, atomic_t *v)
+{
+ return arch_atomic_fetch_xor(i, v);
+}
+
+static __always_inline s64 atomic64_fetch_xor(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_fetch_xor(i, v);
+}
+
+static __always_inline bool atomic_sub_and_test(int i, atomic_t *v)
+{
+ return arch_atomic_sub_and_test(i, v);
+}
+
+static __always_inline bool atomic64_sub_and_test(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_sub_and_test(i, v);
+}
+
+static __always_inline bool atomic_add_negative(int i, atomic_t *v)
+{
+ return arch_atomic_add_negative(i, v);
+}
+
+static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_add_negative(i, v);
+}
+
+#define cmpxchg(ptr, old, new) \
+({ \
+ arch_cmpxchg((ptr), (old), (new)); \
+})
+
+#define sync_cmpxchg(ptr, old, new) \
+({ \
+ arch_sync_cmpxchg((ptr), (old), (new)); \
+})
+
+#define cmpxchg_local(ptr, old, new) \
+({ \
+ arch_cmpxchg_local((ptr), (old), (new)); \
+})
+
+#define cmpxchg64(ptr, old, new) \
+({ \
+ arch_cmpxchg64((ptr), (old), (new)); \
+})
+
+#define cmpxchg64_local(ptr, old, new) \
+({ \
+ arch_cmpxchg64_local((ptr), (old), (new)); \
+})
+
+#define cmpxchg_double(p1, p2, o1, o2, n1, n2) \
+({ \
+ arch_cmpxchg_double((p1), (p2), (o1), (o2), (n1), (n2)); \
+})
+
+#define cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \
+({ \
+ arch_cmpxchg_double_local((p1), (p2), (o1), (o2), (n1), (n2)); \
+})
+
+#endif /* _LINUX_ATOMIC_INSTRUMENTED_H */
^ permalink raw reply related [flat|nested] 38+ messages in thread
* Re: [PATCH v4 5/7] kasan: allow kasan_check_read/write() to accept pointers to volatiles
2017-06-22 8:25 ` Ingo Molnar
@ 2017-06-22 14:15 ` Dmitry Vyukov
0 siblings, 0 replies; 38+ messages in thread
From: Dmitry Vyukov @ 2017-06-22 14:15 UTC (permalink / raw)
To: Ingo Molnar
Cc: Mark Rutland, Peter Zijlstra, Ingo Molnar, Will Deacon,
H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML,
Thomas Gleixner, Andrew Morton, linux-mm, Linus Torvalds
On Thu, Jun 22, 2017 at 10:25 AM, Ingo Molnar <mingo@kernel.org> wrote:
>
> * Dmitry Vyukov <dvyukov@google.com> wrote:
>
>> On Mon, Jun 19, 2017 at 12:50 PM, Mark Rutland <mark.rutland@arm.com> wrote:
>> > On Sat, Jun 17, 2017 at 11:15:31AM +0200, Dmitry Vyukov wrote:
>> >> Currently kasan_check_read/write() accept 'const void*', make them
>> >> accept 'const volatile void*'. This is required for instrumentation
>> >> of atomic operations and there is just no reason to not allow that.
>> >>
>> >> Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
>> >> Cc: Mark Rutland <mark.rutland@arm.com>
>> >> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
>> >> Cc: Thomas Gleixner <tglx@linutronix.de>
>> >> Cc: "H. Peter Anvin" <hpa@zytor.com>
>> >> Cc: Peter Zijlstra <peterz@infradead.org>
>> >> Cc: Andrew Morton <akpm@linux-foundation.org>
>> >> Cc: linux-kernel@vger.kernel.org
>> >> Cc: x86@kernel.org
>> >> Cc: linux-mm@kvack.org
>> >> Cc: kasan-dev@googlegroups.com
>> >
>> > Looks sane to me, and I can confirm this doesn't advervsely affect
>> > arm64. FWIW:
>> >
>> > Acked-by: Mark Rutland <mark.rutland@arm.com>
>> >
>> > Mark.
>>
>>
>> Great! Thanks for testing.
>>
>> Ingo, what are your thoughts? Are you taking this to locking tree? When?
>
> Yeah, it all looks pretty clean to me too. I've applied the first three patches to
> the locking tree, but did some minor stylistic cleanups to the first patch to
> harmonize the style of the code - which made the later patches not apply cleanly.
>
> Mind sending the remaining patches against the locking tree, tip:locking/core?
> (Please also add in all the acks you got.)
Mailed v5 rebased on tip:locking/core (now only 4 patches).
Added Acked/Reviewed-By that I got.
> This should also give people (Peter, Linus?) a last minute chance to object to my
> suggestion of increasing the linecount in patch #1:
>
> 0f2376eb0ff8: locking/atomic/x86: Un-macro-ify atomic ops implementation
>
> arch/x86/include/asm/atomic.h | 69 ++++++++++++++++++++++++++++++++++++++++++++++-----------------------
> arch/x86/include/asm/atomic64_32.h | 81 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++------------------------
> arch/x86/include/asm/atomic64_64.h | 67 ++++++++++++++++++++++++++++++++++++++++++++-----------------------
> 3 files changed, 147 insertions(+), 70 deletions(-)
>
> ... to me the end result looks much more readable despite the +70 lines of code,
> but if anyone feels strongly about this please holler!
>
> Thanks,
>
> Ingo
^ permalink raw reply [flat|nested] 38+ messages in thread
* [tip:locking/core] locking/atomic/x86: Un-macro-ify atomic ops implementation
2017-06-17 9:15 ` [PATCH v4 1/7] x86: un-macro-ify atomic ops implementation Dmitry Vyukov
2017-06-22 11:04 ` [tip:locking/core] locking/atomic/x86: Un-macro-ify " tip-bot for Dmitry Vyukov
@ 2017-07-25 13:54 ` tip-bot for Dmitry Vyukov
1 sibling, 0 replies; 38+ messages in thread
From: tip-bot for Dmitry Vyukov @ 2017-07-25 13:54 UTC (permalink / raw)
To: linux-tip-commits
Cc: aryabinin, tglx, mark.rutland, peterz, linux-kernel, hpa,
dvyukov, akpm, mingo, torvalds
Commit-ID: ba1c9f83f6330a34cc77ef989e183f54c4fe732e
Gitweb: http://git.kernel.org/tip/ba1c9f83f6330a34cc77ef989e183f54c4fe732e
Author: Dmitry Vyukov <dvyukov@google.com>
AuthorDate: Sat, 17 Jun 2017 11:15:27 +0200
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Wed, 28 Jun 2017 18:55:55 +0200
locking/atomic/x86: Un-macro-ify atomic ops implementation
CPP turns perfectly readable code into a much harder to read syntactic soup.
Ingo suggested to write them out as-is in C and ignore the higher linecount.
Do this.
(As a side effect, plain C functions will be easier to KASAN-instrument as well.)
Suggested-by: Ingo Molnar <mingo@kernel.org>
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: kasan-dev@googlegroups.com
Cc: will.deacon@arm.com
Link: http://lkml.kernel.org/r/a35b983dd3be937a3cf63c4e2db487de2cdc7b8f.1497690003.git.dvyukov@google.com
[ Beautified the C code some more and twiddled the changelog
to mention the linecount increase and the KASAN benefit. ]
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/x86/include/asm/atomic.h | 69 +++++++++++++++++++++-----------
arch/x86/include/asm/atomic64_32.h | 81 +++++++++++++++++++++++++++-----------
arch/x86/include/asm/atomic64_64.h | 67 ++++++++++++++++++++-----------
3 files changed, 147 insertions(+), 70 deletions(-)
diff --git a/arch/x86/include/asm/atomic.h b/arch/x86/include/asm/atomic.h
index 33380b8..0874ebd 100644
--- a/arch/x86/include/asm/atomic.h
+++ b/arch/x86/include/asm/atomic.h
@@ -197,35 +197,56 @@ static inline int atomic_xchg(atomic_t *v, int new)
return xchg(&v->counter, new);
}
-#define ATOMIC_OP(op) \
-static inline void atomic_##op(int i, atomic_t *v) \
-{ \
- asm volatile(LOCK_PREFIX #op"l %1,%0" \
- : "+m" (v->counter) \
- : "ir" (i) \
- : "memory"); \
+static inline void atomic_and(int i, atomic_t *v)
+{
+ asm volatile(LOCK_PREFIX "andl %1,%0"
+ : "+m" (v->counter)
+ : "ir" (i)
+ : "memory");
+}
+
+static inline int atomic_fetch_and(int i, atomic_t *v)
+{
+ int val = atomic_read(v);
+
+ do { } while (!atomic_try_cmpxchg(v, &val, val & i));
+
+ return val;
}
-#define ATOMIC_FETCH_OP(op, c_op) \
-static inline int atomic_fetch_##op(int i, atomic_t *v) \
-{ \
- int val = atomic_read(v); \
- do { \
- } while (!atomic_try_cmpxchg(v, &val, val c_op i)); \
- return val; \
+static inline void atomic_or(int i, atomic_t *v)
+{
+ asm volatile(LOCK_PREFIX "orl %1,%0"
+ : "+m" (v->counter)
+ : "ir" (i)
+ : "memory");
}
-#define ATOMIC_OPS(op, c_op) \
- ATOMIC_OP(op) \
- ATOMIC_FETCH_OP(op, c_op)
+static inline int atomic_fetch_or(int i, atomic_t *v)
+{
+ int val = atomic_read(v);
-ATOMIC_OPS(and, &)
-ATOMIC_OPS(or , |)
-ATOMIC_OPS(xor, ^)
+ do { } while (!atomic_try_cmpxchg(v, &val, val | i));
-#undef ATOMIC_OPS
-#undef ATOMIC_FETCH_OP
-#undef ATOMIC_OP
+ return val;
+}
+
+static inline void atomic_xor(int i, atomic_t *v)
+{
+ asm volatile(LOCK_PREFIX "xorl %1,%0"
+ : "+m" (v->counter)
+ : "ir" (i)
+ : "memory");
+}
+
+static inline int atomic_fetch_xor(int i, atomic_t *v)
+{
+ int val = atomic_read(v);
+
+ do { } while (!atomic_try_cmpxchg(v, &val, val ^ i));
+
+ return val;
+}
/**
* __atomic_add_unless - add unless the number is already a given value
@@ -239,10 +260,12 @@ ATOMIC_OPS(xor, ^)
static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u)
{
int c = atomic_read(v);
+
do {
if (unlikely(c == u))
break;
} while (!atomic_try_cmpxchg(v, &c, c + a));
+
return c;
}
diff --git a/arch/x86/include/asm/atomic64_32.h b/arch/x86/include/asm/atomic64_32.h
index 71d7705..9e206f3 100644
--- a/arch/x86/include/asm/atomic64_32.h
+++ b/arch/x86/include/asm/atomic64_32.h
@@ -312,37 +312,70 @@ static inline long long atomic64_dec_if_positive(atomic64_t *v)
#undef alternative_atomic64
#undef __alternative_atomic64
-#define ATOMIC64_OP(op, c_op) \
-static inline void atomic64_##op(long long i, atomic64_t *v) \
-{ \
- long long old, c = 0; \
- while ((old = atomic64_cmpxchg(v, c, c c_op i)) != c) \
- c = old; \
+static inline void atomic64_and(long long i, atomic64_t *v)
+{
+ long long old, c = 0;
+
+ while ((old = atomic64_cmpxchg(v, c, c & i)) != c)
+ c = old;
}
-#define ATOMIC64_FETCH_OP(op, c_op) \
-static inline long long atomic64_fetch_##op(long long i, atomic64_t *v) \
-{ \
- long long old, c = 0; \
- while ((old = atomic64_cmpxchg(v, c, c c_op i)) != c) \
- c = old; \
- return old; \
+static inline long long atomic64_fetch_and(long long i, atomic64_t *v)
+{
+ long long old, c = 0;
+
+ while ((old = atomic64_cmpxchg(v, c, c & i)) != c)
+ c = old;
+
+ return old;
}
-ATOMIC64_FETCH_OP(add, +)
+static inline void atomic64_or(long long i, atomic64_t *v)
+{
+ long long old, c = 0;
-#define atomic64_fetch_sub(i, v) atomic64_fetch_add(-(i), (v))
+ while ((old = atomic64_cmpxchg(v, c, c | i)) != c)
+ c = old;
+}
+
+static inline long long atomic64_fetch_or(long long i, atomic64_t *v)
+{
+ long long old, c = 0;
+
+ while ((old = atomic64_cmpxchg(v, c, c | i)) != c)
+ c = old;
+
+ return old;
+}
-#define ATOMIC64_OPS(op, c_op) \
- ATOMIC64_OP(op, c_op) \
- ATOMIC64_FETCH_OP(op, c_op)
+static inline void atomic64_xor(long long i, atomic64_t *v)
+{
+ long long old, c = 0;
+
+ while ((old = atomic64_cmpxchg(v, c, c ^ i)) != c)
+ c = old;
+}
-ATOMIC64_OPS(and, &)
-ATOMIC64_OPS(or, |)
-ATOMIC64_OPS(xor, ^)
+static inline long long atomic64_fetch_xor(long long i, atomic64_t *v)
+{
+ long long old, c = 0;
+
+ while ((old = atomic64_cmpxchg(v, c, c ^ i)) != c)
+ c = old;
+
+ return old;
+}
-#undef ATOMIC64_OPS
-#undef ATOMIC64_FETCH_OP
-#undef ATOMIC64_OP
+static inline long long atomic64_fetch_add(long long i, atomic64_t *v)
+{
+ long long old, c = 0;
+
+ while ((old = atomic64_cmpxchg(v, c, c + i)) != c)
+ c = old;
+
+ return old;
+}
+
+#define atomic64_fetch_sub(i, v) atomic64_fetch_add(-(i), (v))
#endif /* _ASM_X86_ATOMIC64_32_H */
diff --git a/arch/x86/include/asm/atomic64_64.h b/arch/x86/include/asm/atomic64_64.h
index 6189a43..8db8879a 100644
--- a/arch/x86/include/asm/atomic64_64.h
+++ b/arch/x86/include/asm/atomic64_64.h
@@ -226,34 +226,55 @@ static inline long atomic64_dec_if_positive(atomic64_t *v)
return dec;
}
-#define ATOMIC64_OP(op) \
-static inline void atomic64_##op(long i, atomic64_t *v) \
-{ \
- asm volatile(LOCK_PREFIX #op"q %1,%0" \
- : "+m" (v->counter) \
- : "er" (i) \
- : "memory"); \
+static inline void atomic64_and(long i, atomic64_t *v)
+{
+ asm volatile(LOCK_PREFIX "andq %1,%0"
+ : "+m" (v->counter)
+ : "er" (i)
+ : "memory");
}
-#define ATOMIC64_FETCH_OP(op, c_op) \
-static inline long atomic64_fetch_##op(long i, atomic64_t *v) \
-{ \
- long val = atomic64_read(v); \
- do { \
- } while (!atomic64_try_cmpxchg(v, &val, val c_op i)); \
- return val; \
+static inline long atomic64_fetch_and(long i, atomic64_t *v)
+{
+ long val = atomic64_read(v);
+
+ do {
+ } while (!atomic64_try_cmpxchg(v, &val, val & i));
+ return val;
}
-#define ATOMIC64_OPS(op, c_op) \
- ATOMIC64_OP(op) \
- ATOMIC64_FETCH_OP(op, c_op)
+static inline void atomic64_or(long i, atomic64_t *v)
+{
+ asm volatile(LOCK_PREFIX "orq %1,%0"
+ : "+m" (v->counter)
+ : "er" (i)
+ : "memory");
+}
-ATOMIC64_OPS(and, &)
-ATOMIC64_OPS(or, |)
-ATOMIC64_OPS(xor, ^)
+static inline long atomic64_fetch_or(long i, atomic64_t *v)
+{
+ long val = atomic64_read(v);
-#undef ATOMIC64_OPS
-#undef ATOMIC64_FETCH_OP
-#undef ATOMIC64_OP
+ do {
+ } while (!atomic64_try_cmpxchg(v, &val, val | i));
+ return val;
+}
+
+static inline void atomic64_xor(long i, atomic64_t *v)
+{
+ asm volatile(LOCK_PREFIX "xorq %1,%0"
+ : "+m" (v->counter)
+ : "er" (i)
+ : "memory");
+}
+
+static inline long atomic64_fetch_xor(long i, atomic64_t *v)
+{
+ long val = atomic64_read(v);
+
+ do {
+ } while (!atomic64_try_cmpxchg(v, &val, val ^ i));
+ return val;
+}
#endif /* _ASM_X86_ATOMIC64_64_H */
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [tip:locking/core] locking/atomic/x86: Use 's64 *' for 'old' argument of atomic64_try_cmpxchg()
2017-06-17 9:15 ` [PATCH v4 2/7] x86: use s64* for old arg of atomic64_try_cmpxchg() Dmitry Vyukov
2017-06-22 11:04 ` [tip:locking/core] locking/atomic/x86: Use 's64 *' for 'old' argument " tip-bot for Dmitry Vyukov
@ 2017-07-25 13:55 ` tip-bot for Dmitry Vyukov
1 sibling, 0 replies; 38+ messages in thread
From: tip-bot for Dmitry Vyukov @ 2017-07-25 13:55 UTC (permalink / raw)
To: linux-tip-commits
Cc: akpm, tglx, dvyukov, will.deacon, torvalds, mark.rutland, hpa,
aryabinin, mingo, linux-kernel, peterz
Commit-ID: 007d185b44620f6ffa58f52476bed6e6d7d69d3b
Gitweb: http://git.kernel.org/tip/007d185b44620f6ffa58f52476bed6e6d7d69d3b
Author: Dmitry Vyukov <dvyukov@google.com>
AuthorDate: Sat, 17 Jun 2017 11:15:28 +0200
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Wed, 28 Jun 2017 18:55:55 +0200
locking/atomic/x86: Use 's64 *' for 'old' argument of atomic64_try_cmpxchg()
atomic64_try_cmpxchg() declares old argument as 'long *',
this makes it impossible to use it in portable code.
If caller passes 'long *', it becomes 32-bits on 32-bit arches.
If caller passes 's64 *', it does not compile on x86_64.
Change type of old argument to 's64 *' instead.
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Will Deacon <will.deacon@arm.com>
Cc: kasan-dev@googlegroups.com
Cc: linux-mm@kvack.org
Link: http://lkml.kernel.org/r/fa6f77f2375150d26ea796a77e8b59195fd2ab13.1497690003.git.dvyukov@google.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/x86/include/asm/atomic64_64.h | 12 ++++++------
arch/x86/include/asm/cmpxchg.h | 2 +-
2 files changed, 7 insertions(+), 7 deletions(-)
diff --git a/arch/x86/include/asm/atomic64_64.h b/arch/x86/include/asm/atomic64_64.h
index 8db8879a..5d9de36 100644
--- a/arch/x86/include/asm/atomic64_64.h
+++ b/arch/x86/include/asm/atomic64_64.h
@@ -177,7 +177,7 @@ static inline long atomic64_cmpxchg(atomic64_t *v, long old, long new)
}
#define atomic64_try_cmpxchg atomic64_try_cmpxchg
-static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, long *old, long new)
+static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, long new)
{
return try_cmpxchg(&v->counter, old, new);
}
@@ -198,7 +198,7 @@ static inline long atomic64_xchg(atomic64_t *v, long new)
*/
static inline bool atomic64_add_unless(atomic64_t *v, long a, long u)
{
- long c = atomic64_read(v);
+ s64 c = atomic64_read(v);
do {
if (unlikely(c == u))
return false;
@@ -217,7 +217,7 @@ static inline bool atomic64_add_unless(atomic64_t *v, long a, long u)
*/
static inline long atomic64_dec_if_positive(atomic64_t *v)
{
- long dec, c = atomic64_read(v);
+ s64 dec, c = atomic64_read(v);
do {
dec = c - 1;
if (unlikely(dec < 0))
@@ -236,7 +236,7 @@ static inline void atomic64_and(long i, atomic64_t *v)
static inline long atomic64_fetch_and(long i, atomic64_t *v)
{
- long val = atomic64_read(v);
+ s64 val = atomic64_read(v);
do {
} while (!atomic64_try_cmpxchg(v, &val, val & i));
@@ -253,7 +253,7 @@ static inline void atomic64_or(long i, atomic64_t *v)
static inline long atomic64_fetch_or(long i, atomic64_t *v)
{
- long val = atomic64_read(v);
+ s64 val = atomic64_read(v);
do {
} while (!atomic64_try_cmpxchg(v, &val, val | i));
@@ -270,7 +270,7 @@ static inline void atomic64_xor(long i, atomic64_t *v)
static inline long atomic64_fetch_xor(long i, atomic64_t *v)
{
- long val = atomic64_read(v);
+ s64 val = atomic64_read(v);
do {
} while (!atomic64_try_cmpxchg(v, &val, val ^ i));
diff --git a/arch/x86/include/asm/cmpxchg.h b/arch/x86/include/asm/cmpxchg.h
index d90296d..b5069e8 100644
--- a/arch/x86/include/asm/cmpxchg.h
+++ b/arch/x86/include/asm/cmpxchg.h
@@ -157,7 +157,7 @@ extern void __add_wrong_size(void)
#define __raw_try_cmpxchg(_ptr, _pold, _new, size, lock) \
({ \
bool success; \
- __typeof__(_ptr) _old = (_pold); \
+ __typeof__(_ptr) _old = (__typeof__(_ptr))(_pold); \
__typeof__(*(_ptr)) __old = *_old; \
__typeof__(*(_ptr)) __new = (_new); \
switch (size) { \
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v6 0/4] x86, kasan: add KASAN checks to atomic operations
@ 2018-01-29 17:26 Dmitry Vyukov
2018-01-29 17:26 ` [PATCH v6 1/4] locking/atomic: Add asm-generic/atomic-instrumented.h Dmitry Vyukov
` (5 more replies)
0 siblings, 6 replies; 38+ messages in thread
From: Dmitry Vyukov @ 2018-01-29 17:26 UTC (permalink / raw)
To: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin,
kasan-dev, x86, linux-kernel, tglx
Cc: Dmitry Vyukov
KASAN uses compiler instrumentation to intercept all memory accesses.
But it does not see memory accesses done in assembly code.
One notable user of assembly code is atomic operations. Frequently,
for example, an atomic reference decrement is the last access to an
object and a good candidate for a racy use-after-free.
Atomic operations are defined in arch files, but KASAN instrumentation
is required for several archs that support KASAN. Later we will need
similar hooks for KMSAN (uninit use detector) and KTSAN (data race
detector).
This change introduces wrappers around atomic operations that can be
used to add KASAN/KMSAN/KTSAN instrumentation across several archs,
and adds KASAN checks to them.
This patch uses the wrappers only for x86 arch. Arm64 will be switched
later. And we also plan to instrument bitops in a similar way.
Within a day it has found its first bug:
BUG: KASAN: use-after-free in atomic_dec_and_test
arch/x86/include/asm/atomic.h:123 [inline] at addr ffff880079c30158
Write of size 4 by task syz-executor6/25698
CPU: 2 PID: 25698 Comm: syz-executor6 Not tainted 4.10.0+ #302
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
kasan_check_write+0x14/0x20 mm/kasan/kasan.c:344
atomic_dec_and_test arch/x86/include/asm/atomic.h:123 [inline]
put_task_struct include/linux/sched/task.h:93 [inline]
put_ctx+0xcf/0x110 kernel/events/core.c:1131
perf_event_release_kernel+0x3ad/0xc90 kernel/events/core.c:4322
perf_release+0x37/0x50 kernel/events/core.c:4338
__fput+0x332/0x800 fs/file_table.c:209
____fput+0x15/0x20 fs/file_table.c:245
task_work_run+0x197/0x260 kernel/task_work.c:116
exit_task_work include/linux/task_work.h:21 [inline]
do_exit+0xb38/0x29c0 kernel/exit.c:880
do_group_exit+0x149/0x420 kernel/exit.c:984
get_signal+0x7e0/0x1820 kernel/signal.c:2318
do_signal+0xd2/0x2190 arch/x86/kernel/signal.c:808
exit_to_usermode_loop+0x200/0x2a0 arch/x86/entry/common.c:157
syscall_return_slowpath arch/x86/entry/common.c:191 [inline]
do_syscall_64+0x6fc/0x930 arch/x86/entry/common.c:286
entry_SYSCALL64_slow_path+0x25/0x25
RIP: 0033:0x4458d9
RSP: 002b:00007f3f07187cf8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
RAX: fffffffffffffe00 RBX: 00000000007080c8 RCX: 00000000004458d9
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00000000007080c8
RBP: 00000000007080a8 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 00007f3f071889c0 R15: 00007f3f07188700
Object at ffff880079c30140, in cache task_struct size: 5376
Allocated:
PID = 25681
kmem_cache_alloc_node+0x122/0x6f0 mm/slab.c:3662
alloc_task_struct_node kernel/fork.c:153 [inline]
dup_task_struct kernel/fork.c:495 [inline]
copy_process.part.38+0x19c8/0x4aa0 kernel/fork.c:1560
copy_process kernel/fork.c:1531 [inline]
_do_fork+0x200/0x1010 kernel/fork.c:1994
SYSC_clone kernel/fork.c:2104 [inline]
SyS_clone+0x37/0x50 kernel/fork.c:2098
do_syscall_64+0x2e8/0x930 arch/x86/entry/common.c:281
return_from_SYSCALL_64+0x0/0x7a
Freed:
PID = 25681
__cache_free mm/slab.c:3514 [inline]
kmem_cache_free+0x71/0x240 mm/slab.c:3774
free_task_struct kernel/fork.c:158 [inline]
free_task+0x151/0x1d0 kernel/fork.c:370
copy_process.part.38+0x18e5/0x4aa0 kernel/fork.c:1931
copy_process kernel/fork.c:1531 [inline]
_do_fork+0x200/0x1010 kernel/fork.c:1994
SYSC_clone kernel/fork.c:2104 [inline]
SyS_clone+0x37/0x50 kernel/fork.c:2098
do_syscall_64+0x2e8/0x930 arch/x86/entry/common.c:281
return_from_SYSCALL_64+0x0/0x7a
Changes since v1:
- dropped "x86: remove unused atomic_inc_short()" patch
it is mailed separately
- rebased on top of tip/locking/core head
- other changes noted within individual patches
Changes since v2:
- rebased on top of tip/locking/core head
- dropped a pervasive "x86: use long long for 64-bit atomic ops" commit,
instead use s64 type in wrappers
- added "x86: use s64* for old arg of atomic64_try_cmpxchg()" commit
Changes since v3 are noted in individual commits.
Changes since v4:
- rebased on tip/locking/core HEAD
Changes since v5:
- rework cmpxchg* implementations so that we have less
code in macros and more code in functions
Tested:
- build/boot x86_64 defconfig
- build/boot x86_64 defconfig+KASAN
- build i686/powerpc defconfig
Dmitry Vyukov (4):
locking/atomic: Add asm-generic/atomic-instrumented.h
x86: switch atomic.h to use atomic-instrumented.h
asm-generic: add KASAN instrumentation to atomic operations
asm-generic, x86: add comments for atomic instrumentation
arch/x86/include/asm/atomic.h | 106 +++----
arch/x86/include/asm/atomic64_32.h | 106 +++----
arch/x86/include/asm/atomic64_64.h | 108 +++----
arch/x86/include/asm/cmpxchg.h | 12 +-
arch/x86/include/asm/cmpxchg_32.h | 8 +-
arch/x86/include/asm/cmpxchg_64.h | 4 +-
include/asm-generic/atomic-instrumented.h | 476 ++++++++++++++++++++++++++++++
7 files changed, 652 insertions(+), 168 deletions(-)
create mode 100644 include/asm-generic/atomic-instrumented.h
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply [flat|nested] 38+ messages in thread
* [PATCH v6 1/4] locking/atomic: Add asm-generic/atomic-instrumented.h
2018-01-29 17:26 [PATCH v6 0/4] x86, kasan: add KASAN checks to atomic operations Dmitry Vyukov
@ 2018-01-29 17:26 ` Dmitry Vyukov
2018-01-29 17:26 ` [PATCH v6 2/4] x86: switch atomic.h to use atomic-instrumented.h Dmitry Vyukov
` (4 subsequent siblings)
5 siblings, 0 replies; 38+ messages in thread
From: Dmitry Vyukov @ 2018-01-29 17:26 UTC (permalink / raw)
To: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin,
kasan-dev, x86, linux-kernel, tglx
Cc: Dmitry Vyukov, Andrew Morton, Linus Torvalds, linux-mm, Ingo Molnar
The new header allows to wrap per-arch atomic operations
and add common functionality to all of them.
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Will Deacon <will.deacon@arm.com>
Cc: kasan-dev@googlegroups.com
Cc: linux-mm@kvack.org
Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
Changes since v5:
- rework cmpxchg* implementations so that we have less
code in macros and more code in functions.
---
include/asm-generic/atomic-instrumented.h | 393 ++++++++++++++++++++++++++++++
1 file changed, 393 insertions(+)
diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h
new file mode 100644
index 000000000000..b966194d120a
--- /dev/null
+++ b/include/asm-generic/atomic-instrumented.h
@@ -0,0 +1,393 @@
+#ifndef _LINUX_ATOMIC_INSTRUMENTED_H
+#define _LINUX_ATOMIC_INSTRUMENTED_H
+
+#include <linux/build_bug.h>
+
+static __always_inline int atomic_read(const atomic_t *v)
+{
+ return arch_atomic_read(v);
+}
+
+static __always_inline s64 atomic64_read(const atomic64_t *v)
+{
+ return arch_atomic64_read(v);
+}
+
+static __always_inline void atomic_set(atomic_t *v, int i)
+{
+ arch_atomic_set(v, i);
+}
+
+static __always_inline void atomic64_set(atomic64_t *v, s64 i)
+{
+ arch_atomic64_set(v, i);
+}
+
+static __always_inline int atomic_xchg(atomic_t *v, int i)
+{
+ return arch_atomic_xchg(v, i);
+}
+
+static __always_inline s64 atomic64_xchg(atomic64_t *v, s64 i)
+{
+ return arch_atomic64_xchg(v, i);
+}
+
+static __always_inline int atomic_cmpxchg(atomic_t *v, int old, int new)
+{
+ return arch_atomic_cmpxchg(v, old, new);
+}
+
+static __always_inline s64 atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new)
+{
+ return arch_atomic64_cmpxchg(v, old, new);
+}
+
+#ifdef arch_atomic_try_cmpxchg
+#define atomic_try_cmpxchg atomic_try_cmpxchg
+static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new)
+{
+ return arch_atomic_try_cmpxchg(v, old, new);
+}
+#endif
+
+#ifdef arch_atomic64_try_cmpxchg
+#define atomic64_try_cmpxchg atomic64_try_cmpxchg
+static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, s64 new)
+{
+ return arch_atomic64_try_cmpxchg(v, old, new);
+}
+#endif
+
+static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u)
+{
+ return __arch_atomic_add_unless(v, a, u);
+}
+
+
+static __always_inline bool atomic64_add_unless(atomic64_t *v, s64 a, s64 u)
+{
+ return arch_atomic64_add_unless(v, a, u);
+}
+
+static __always_inline void atomic_inc(atomic_t *v)
+{
+ arch_atomic_inc(v);
+}
+
+static __always_inline void atomic64_inc(atomic64_t *v)
+{
+ arch_atomic64_inc(v);
+}
+
+static __always_inline void atomic_dec(atomic_t *v)
+{
+ arch_atomic_dec(v);
+}
+
+static __always_inline void atomic64_dec(atomic64_t *v)
+{
+ arch_atomic64_dec(v);
+}
+
+static __always_inline void atomic_add(int i, atomic_t *v)
+{
+ arch_atomic_add(i, v);
+}
+
+static __always_inline void atomic64_add(s64 i, atomic64_t *v)
+{
+ arch_atomic64_add(i, v);
+}
+
+static __always_inline void atomic_sub(int i, atomic_t *v)
+{
+ arch_atomic_sub(i, v);
+}
+
+static __always_inline void atomic64_sub(s64 i, atomic64_t *v)
+{
+ arch_atomic64_sub(i, v);
+}
+
+static __always_inline void atomic_and(int i, atomic_t *v)
+{
+ arch_atomic_and(i, v);
+}
+
+static __always_inline void atomic64_and(s64 i, atomic64_t *v)
+{
+ arch_atomic64_and(i, v);
+}
+
+static __always_inline void atomic_or(int i, atomic_t *v)
+{
+ arch_atomic_or(i, v);
+}
+
+static __always_inline void atomic64_or(s64 i, atomic64_t *v)
+{
+ arch_atomic64_or(i, v);
+}
+
+static __always_inline void atomic_xor(int i, atomic_t *v)
+{
+ arch_atomic_xor(i, v);
+}
+
+static __always_inline void atomic64_xor(s64 i, atomic64_t *v)
+{
+ arch_atomic64_xor(i, v);
+}
+
+static __always_inline int atomic_inc_return(atomic_t *v)
+{
+ return arch_atomic_inc_return(v);
+}
+
+static __always_inline s64 atomic64_inc_return(atomic64_t *v)
+{
+ return arch_atomic64_inc_return(v);
+}
+
+static __always_inline int atomic_dec_return(atomic_t *v)
+{
+ return arch_atomic_dec_return(v);
+}
+
+static __always_inline s64 atomic64_dec_return(atomic64_t *v)
+{
+ return arch_atomic64_dec_return(v);
+}
+
+static __always_inline s64 atomic64_inc_not_zero(atomic64_t *v)
+{
+ return arch_atomic64_inc_not_zero(v);
+}
+
+static __always_inline s64 atomic64_dec_if_positive(atomic64_t *v)
+{
+ return arch_atomic64_dec_if_positive(v);
+}
+
+static __always_inline bool atomic_dec_and_test(atomic_t *v)
+{
+ return arch_atomic_dec_and_test(v);
+}
+
+static __always_inline bool atomic64_dec_and_test(atomic64_t *v)
+{
+ return arch_atomic64_dec_and_test(v);
+}
+
+static __always_inline bool atomic_inc_and_test(atomic_t *v)
+{
+ return arch_atomic_inc_and_test(v);
+}
+
+static __always_inline bool atomic64_inc_and_test(atomic64_t *v)
+{
+ return arch_atomic64_inc_and_test(v);
+}
+
+static __always_inline int atomic_add_return(int i, atomic_t *v)
+{
+ return arch_atomic_add_return(i, v);
+}
+
+static __always_inline s64 atomic64_add_return(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_add_return(i, v);
+}
+
+static __always_inline int atomic_sub_return(int i, atomic_t *v)
+{
+ return arch_atomic_sub_return(i, v);
+}
+
+static __always_inline s64 atomic64_sub_return(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_sub_return(i, v);
+}
+
+static __always_inline int atomic_fetch_add(int i, atomic_t *v)
+{
+ return arch_atomic_fetch_add(i, v);
+}
+
+static __always_inline s64 atomic64_fetch_add(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_fetch_add(i, v);
+}
+
+static __always_inline int atomic_fetch_sub(int i, atomic_t *v)
+{
+ return arch_atomic_fetch_sub(i, v);
+}
+
+static __always_inline s64 atomic64_fetch_sub(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_fetch_sub(i, v);
+}
+
+static __always_inline int atomic_fetch_and(int i, atomic_t *v)
+{
+ return arch_atomic_fetch_and(i, v);
+}
+
+static __always_inline s64 atomic64_fetch_and(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_fetch_and(i, v);
+}
+
+static __always_inline int atomic_fetch_or(int i, atomic_t *v)
+{
+ return arch_atomic_fetch_or(i, v);
+}
+
+static __always_inline s64 atomic64_fetch_or(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_fetch_or(i, v);
+}
+
+static __always_inline int atomic_fetch_xor(int i, atomic_t *v)
+{
+ return arch_atomic_fetch_xor(i, v);
+}
+
+static __always_inline s64 atomic64_fetch_xor(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_fetch_xor(i, v);
+}
+
+static __always_inline bool atomic_sub_and_test(int i, atomic_t *v)
+{
+ return arch_atomic_sub_and_test(i, v);
+}
+
+static __always_inline bool atomic64_sub_and_test(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_sub_and_test(i, v);
+}
+
+static __always_inline bool atomic_add_negative(int i, atomic_t *v)
+{
+ return arch_atomic_add_negative(i, v);
+}
+
+static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_add_negative(i, v);
+}
+
+static __always_inline unsigned long
+cmpxchg_size(volatile void *ptr, unsigned long old, unsigned long new, int size)
+{
+ switch (size) {
+ case 1:
+ return arch_cmpxchg((u8 *)ptr, (u8)old, (u8)new);
+ case 2:
+ return arch_cmpxchg((u16 *)ptr, (u16)old, (u16)new);
+ case 4:
+ return arch_cmpxchg((u32 *)ptr, (u32)old, (u32)new);
+ case 8:
+ BUILD_BUG_ON(sizeof(unsigned long) != 8);
+ return arch_cmpxchg((u64 *)ptr, (u64)old, (u64)new);
+ }
+ BUILD_BUG();
+ return 0;
+}
+
+#define cmpxchg(ptr, old, new) \
+({ \
+ ((__typeof__(*(ptr)))cmpxchg_size((ptr), (unsigned long)(old), \
+ (unsigned long)(new), sizeof(*(ptr)))); \
+})
+
+static __always_inline unsigned long
+sync_cmpxchg_size(volatile void *ptr, unsigned long old, unsigned long new,
+ int size)
+{
+ switch (size) {
+ case 1:
+ return arch_sync_cmpxchg((u8 *)ptr, (u8)old, (u8)new);
+ case 2:
+ return arch_sync_cmpxchg((u16 *)ptr, (u16)old, (u16)new);
+ case 4:
+ return arch_sync_cmpxchg((u32 *)ptr, (u32)old, (u32)new);
+ case 8:
+ BUILD_BUG_ON(sizeof(unsigned long) != 8);
+ return arch_sync_cmpxchg((u64 *)ptr, (u64)old, (u64)new);
+ }
+ BUILD_BUG();
+ return 0;
+}
+
+#define sync_cmpxchg(ptr, old, new) \
+({ \
+ ((__typeof__(*(ptr)))sync_cmpxchg_size((ptr), \
+ (unsigned long)(old), (unsigned long)(new), \
+ sizeof(*(ptr)))); \
+})
+
+static __always_inline unsigned long
+cmpxchg_local_size(volatile void *ptr, unsigned long old, unsigned long new,
+ int size)
+{
+ switch (size) {
+ case 1:
+ return arch_cmpxchg_local((u8 *)ptr, (u8)old, (u8)new);
+ case 2:
+ return arch_cmpxchg_local((u16 *)ptr, (u16)old, (u16)new);
+ case 4:
+ return arch_cmpxchg_local((u32 *)ptr, (u32)old, (u32)new);
+ case 8:
+ BUILD_BUG_ON(sizeof(unsigned long) != 8);
+ return arch_cmpxchg_local((u64 *)ptr, (u64)old, (u64)new);
+ }
+ BUILD_BUG();
+ return 0;
+}
+
+#define cmpxchg_local(ptr, old, new) \
+({ \
+ ((__typeof__(*(ptr)))cmpxchg_local_size((ptr), \
+ (unsigned long)(old), (unsigned long)(new), \
+ sizeof(*(ptr)))); \
+})
+
+static __always_inline u64
+cmpxchg64_size(volatile u64 *ptr, u64 old, u64 new)
+{
+ return arch_cmpxchg64(ptr, old, new);
+}
+
+#define cmpxchg64(ptr, old, new) \
+({ \
+ ((__typeof__(*(ptr)))cmpxchg64_size((ptr), (u64)(old), \
+ (u64)(new))); \
+})
+
+static __always_inline u64
+cmpxchg64_local_size(volatile u64 *ptr, u64 old, u64 new)
+{
+ return arch_cmpxchg64_local(ptr, old, new);
+}
+
+#define cmpxchg64_local(ptr, old, new) \
+({ \
+ ((__typeof__(*(ptr)))cmpxchg64_local_size((ptr), (u64)(old), \
+ (u64)(new))); \
+})
+
+#define cmpxchg_double(p1, p2, o1, o2, n1, n2) \
+({ \
+ arch_cmpxchg_double((p1), (p2), (o1), (o2), (n1), (n2)); \
+})
+
+#define cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \
+({ \
+ arch_cmpxchg_double_local((p1), (p2), (o1), (o2), (n1), (n2)); \
+})
+
+#endif /* _LINUX_ATOMIC_INSTRUMENTED_H */
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v6 2/4] x86: switch atomic.h to use atomic-instrumented.h
2018-01-29 17:26 [PATCH v6 0/4] x86, kasan: add KASAN checks to atomic operations Dmitry Vyukov
2018-01-29 17:26 ` [PATCH v6 1/4] locking/atomic: Add asm-generic/atomic-instrumented.h Dmitry Vyukov
@ 2018-01-29 17:26 ` Dmitry Vyukov
2018-03-12 12:24 ` [tip:locking/core] locking/atomic/x86: Switch " tip-bot for Dmitry Vyukov
2018-01-29 17:26 ` [PATCH v6 3/4] asm-generic: add KASAN instrumentation to atomic operations Dmitry Vyukov
` (3 subsequent siblings)
5 siblings, 1 reply; 38+ messages in thread
From: Dmitry Vyukov @ 2018-01-29 17:26 UTC (permalink / raw)
To: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin,
kasan-dev, x86, linux-kernel, tglx
Cc: Dmitry Vyukov, Andrew Morton, linux-mm
Add arch_ prefix to all atomic operations and include
<asm-generic/atomic-instrumented.h>. This will allow
to add KASAN instrumentation to all atomic ops.
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: kasan-dev@googlegroups.com
Cc: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org
Cc: x86@kernel.org
---
Changes since v1:
- reverted unnecessary change in __raw_try_cmpxchg()
- reverted s/try_cmpxchg/arch_try_cmpxchg/ change
try_cmpxchg is x86 implementation detail
Changes since v3:
- drop comment in arch_atomic_set()
---
arch/x86/include/asm/atomic.h | 102 ++++++++++++++++++-----------------
arch/x86/include/asm/atomic64_32.h | 106 ++++++++++++++++++------------------
arch/x86/include/asm/atomic64_64.h | 108 ++++++++++++++++++-------------------
arch/x86/include/asm/cmpxchg.h | 12 ++---
arch/x86/include/asm/cmpxchg_32.h | 8 +--
arch/x86/include/asm/cmpxchg_64.h | 4 +-
6 files changed, 172 insertions(+), 168 deletions(-)
diff --git a/arch/x86/include/asm/atomic.h b/arch/x86/include/asm/atomic.h
index 72759f131cc5..33afc966d6a9 100644
--- a/arch/x86/include/asm/atomic.h
+++ b/arch/x86/include/asm/atomic.h
@@ -17,36 +17,36 @@
#define ATOMIC_INIT(i) { (i) }
/**
- * atomic_read - read atomic variable
+ * arch_atomic_read - read atomic variable
* @v: pointer of type atomic_t
*
* Atomically reads the value of @v.
*/
-static __always_inline int atomic_read(const atomic_t *v)
+static __always_inline int arch_atomic_read(const atomic_t *v)
{
return READ_ONCE((v)->counter);
}
/**
- * atomic_set - set atomic variable
+ * arch_atomic_set - set atomic variable
* @v: pointer of type atomic_t
* @i: required value
*
* Atomically sets the value of @v to @i.
*/
-static __always_inline void atomic_set(atomic_t *v, int i)
+static __always_inline void arch_atomic_set(atomic_t *v, int i)
{
WRITE_ONCE(v->counter, i);
}
/**
- * atomic_add - add integer to atomic variable
+ * arch_atomic_add - add integer to atomic variable
* @i: integer value to add
* @v: pointer of type atomic_t
*
* Atomically adds @i to @v.
*/
-static __always_inline void atomic_add(int i, atomic_t *v)
+static __always_inline void arch_atomic_add(int i, atomic_t *v)
{
asm volatile(LOCK_PREFIX "addl %1,%0"
: "+m" (v->counter)
@@ -54,13 +54,13 @@ static __always_inline void atomic_add(int i, atomic_t *v)
}
/**
- * atomic_sub - subtract integer from atomic variable
+ * arch_atomic_sub - subtract integer from atomic variable
* @i: integer value to subtract
* @v: pointer of type atomic_t
*
* Atomically subtracts @i from @v.
*/
-static __always_inline void atomic_sub(int i, atomic_t *v)
+static __always_inline void arch_atomic_sub(int i, atomic_t *v)
{
asm volatile(LOCK_PREFIX "subl %1,%0"
: "+m" (v->counter)
@@ -68,7 +68,7 @@ static __always_inline void atomic_sub(int i, atomic_t *v)
}
/**
- * atomic_sub_and_test - subtract value from variable and test result
+ * arch_atomic_sub_and_test - subtract value from variable and test result
* @i: integer value to subtract
* @v: pointer of type atomic_t
*
@@ -76,63 +76,63 @@ static __always_inline void atomic_sub(int i, atomic_t *v)
* true if the result is zero, or false for all
* other cases.
*/
-static __always_inline bool atomic_sub_and_test(int i, atomic_t *v)
+static __always_inline bool arch_atomic_sub_and_test(int i, atomic_t *v)
{
GEN_BINARY_RMWcc(LOCK_PREFIX "subl", v->counter, "er", i, "%0", e);
}
/**
- * atomic_inc - increment atomic variable
+ * arch_atomic_inc - increment atomic variable
* @v: pointer of type atomic_t
*
* Atomically increments @v by 1.
*/
-static __always_inline void atomic_inc(atomic_t *v)
+static __always_inline void arch_atomic_inc(atomic_t *v)
{
asm volatile(LOCK_PREFIX "incl %0"
: "+m" (v->counter));
}
/**
- * atomic_dec - decrement atomic variable
+ * arch_atomic_dec - decrement atomic variable
* @v: pointer of type atomic_t
*
* Atomically decrements @v by 1.
*/
-static __always_inline void atomic_dec(atomic_t *v)
+static __always_inline void arch_atomic_dec(atomic_t *v)
{
asm volatile(LOCK_PREFIX "decl %0"
: "+m" (v->counter));
}
/**
- * atomic_dec_and_test - decrement and test
+ * arch_atomic_dec_and_test - decrement and test
* @v: pointer of type atomic_t
*
* Atomically decrements @v by 1 and
* returns true if the result is 0, or false for all other
* cases.
*/
-static __always_inline bool atomic_dec_and_test(atomic_t *v)
+static __always_inline bool arch_atomic_dec_and_test(atomic_t *v)
{
GEN_UNARY_RMWcc(LOCK_PREFIX "decl", v->counter, "%0", e);
}
/**
- * atomic_inc_and_test - increment and test
+ * arch_atomic_inc_and_test - increment and test
* @v: pointer of type atomic_t
*
* Atomically increments @v by 1
* and returns true if the result is zero, or false for all
* other cases.
*/
-static __always_inline bool atomic_inc_and_test(atomic_t *v)
+static __always_inline bool arch_atomic_inc_and_test(atomic_t *v)
{
GEN_UNARY_RMWcc(LOCK_PREFIX "incl", v->counter, "%0", e);
}
/**
- * atomic_add_negative - add and test if negative
+ * arch_atomic_add_negative - add and test if negative
* @i: integer value to add
* @v: pointer of type atomic_t
*
@@ -140,65 +140,65 @@ static __always_inline bool atomic_inc_and_test(atomic_t *v)
* if the result is negative, or false when
* result is greater than or equal to zero.
*/
-static __always_inline bool atomic_add_negative(int i, atomic_t *v)
+static __always_inline bool arch_atomic_add_negative(int i, atomic_t *v)
{
GEN_BINARY_RMWcc(LOCK_PREFIX "addl", v->counter, "er", i, "%0", s);
}
/**
- * atomic_add_return - add integer and return
+ * arch_atomic_add_return - add integer and return
* @i: integer value to add
* @v: pointer of type atomic_t
*
* Atomically adds @i to @v and returns @i + @v
*/
-static __always_inline int atomic_add_return(int i, atomic_t *v)
+static __always_inline int arch_atomic_add_return(int i, atomic_t *v)
{
return i + xadd(&v->counter, i);
}
/**
- * atomic_sub_return - subtract integer and return
+ * arch_atomic_sub_return - subtract integer and return
* @v: pointer of type atomic_t
* @i: integer value to subtract
*
* Atomically subtracts @i from @v and returns @v - @i
*/
-static __always_inline int atomic_sub_return(int i, atomic_t *v)
+static __always_inline int arch_atomic_sub_return(int i, atomic_t *v)
{
- return atomic_add_return(-i, v);
+ return arch_atomic_add_return(-i, v);
}
-#define atomic_inc_return(v) (atomic_add_return(1, v))
-#define atomic_dec_return(v) (atomic_sub_return(1, v))
+#define arch_atomic_inc_return(v) (arch_atomic_add_return(1, v))
+#define arch_atomic_dec_return(v) (arch_atomic_sub_return(1, v))
-static __always_inline int atomic_fetch_add(int i, atomic_t *v)
+static __always_inline int arch_atomic_fetch_add(int i, atomic_t *v)
{
return xadd(&v->counter, i);
}
-static __always_inline int atomic_fetch_sub(int i, atomic_t *v)
+static __always_inline int arch_atomic_fetch_sub(int i, atomic_t *v)
{
return xadd(&v->counter, -i);
}
-static __always_inline int atomic_cmpxchg(atomic_t *v, int old, int new)
+static __always_inline int arch_atomic_cmpxchg(atomic_t *v, int old, int new)
{
- return cmpxchg(&v->counter, old, new);
+ return arch_cmpxchg(&v->counter, old, new);
}
-#define atomic_try_cmpxchg atomic_try_cmpxchg
-static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new)
+#define arch_atomic_try_cmpxchg arch_atomic_try_cmpxchg
+static __always_inline bool arch_atomic_try_cmpxchg(atomic_t *v, int *old, int new)
{
return try_cmpxchg(&v->counter, old, new);
}
-static inline int atomic_xchg(atomic_t *v, int new)
+static inline int arch_atomic_xchg(atomic_t *v, int new)
{
return xchg(&v->counter, new);
}
-static inline void atomic_and(int i, atomic_t *v)
+static inline void arch_atomic_and(int i, atomic_t *v)
{
asm volatile(LOCK_PREFIX "andl %1,%0"
: "+m" (v->counter)
@@ -206,16 +206,16 @@ static inline void atomic_and(int i, atomic_t *v)
: "memory");
}
-static inline int atomic_fetch_and(int i, atomic_t *v)
+static inline int arch_atomic_fetch_and(int i, atomic_t *v)
{
- int val = atomic_read(v);
+ int val = arch_atomic_read(v);
- do { } while (!atomic_try_cmpxchg(v, &val, val & i));
+ do { } while (!arch_atomic_try_cmpxchg(v, &val, val & i));
return val;
}
-static inline void atomic_or(int i, atomic_t *v)
+static inline void arch_atomic_or(int i, atomic_t *v)
{
asm volatile(LOCK_PREFIX "orl %1,%0"
: "+m" (v->counter)
@@ -223,16 +223,16 @@ static inline void atomic_or(int i, atomic_t *v)
: "memory");
}
-static inline int atomic_fetch_or(int i, atomic_t *v)
+static inline int arch_atomic_fetch_or(int i, atomic_t *v)
{
- int val = atomic_read(v);
+ int val = arch_atomic_read(v);
- do { } while (!atomic_try_cmpxchg(v, &val, val | i));
+ do { } while (!arch_atomic_try_cmpxchg(v, &val, val | i));
return val;
}
-static inline void atomic_xor(int i, atomic_t *v)
+static inline void arch_atomic_xor(int i, atomic_t *v)
{
asm volatile(LOCK_PREFIX "xorl %1,%0"
: "+m" (v->counter)
@@ -240,17 +240,17 @@ static inline void atomic_xor(int i, atomic_t *v)
: "memory");
}
-static inline int atomic_fetch_xor(int i, atomic_t *v)
+static inline int arch_atomic_fetch_xor(int i, atomic_t *v)
{
- int val = atomic_read(v);
+ int val = arch_atomic_read(v);
- do { } while (!atomic_try_cmpxchg(v, &val, val ^ i));
+ do { } while (!arch_atomic_try_cmpxchg(v, &val, val ^ i));
return val;
}
/**
- * __atomic_add_unless - add unless the number is already a given value
+ * __arch_atomic_add_unless - add unless the number is already a given value
* @v: pointer of type atomic_t
* @a: the amount to add to v...
* @u: ...unless v is equal to u.
@@ -258,14 +258,14 @@ static inline int atomic_fetch_xor(int i, atomic_t *v)
* Atomically adds @a to @v, so long as @v was not already @u.
* Returns the old value of @v.
*/
-static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u)
+static __always_inline int __arch_atomic_add_unless(atomic_t *v, int a, int u)
{
- int c = atomic_read(v);
+ int c = arch_atomic_read(v);
do {
if (unlikely(c == u))
break;
- } while (!atomic_try_cmpxchg(v, &c, c + a));
+ } while (!arch_atomic_try_cmpxchg(v, &c, c + a));
return c;
}
@@ -276,4 +276,6 @@ static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u)
# include <asm/atomic64_64.h>
#endif
+#include <asm-generic/atomic-instrumented.h>
+
#endif /* _ASM_X86_ATOMIC_H */
diff --git a/arch/x86/include/asm/atomic64_32.h b/arch/x86/include/asm/atomic64_32.h
index 97c46b8169b7..46e1ef17d92d 100644
--- a/arch/x86/include/asm/atomic64_32.h
+++ b/arch/x86/include/asm/atomic64_32.h
@@ -62,7 +62,7 @@ ATOMIC64_DECL(add_unless);
#undef ATOMIC64_EXPORT
/**
- * atomic64_cmpxchg - cmpxchg atomic64 variable
+ * arch_atomic64_cmpxchg - cmpxchg atomic64 variable
* @v: pointer to type atomic64_t
* @o: expected value
* @n: new value
@@ -71,20 +71,21 @@ ATOMIC64_DECL(add_unless);
* the old value.
*/
-static inline long long atomic64_cmpxchg(atomic64_t *v, long long o, long long n)
+static inline long long arch_atomic64_cmpxchg(atomic64_t *v, long long o,
+ long long n)
{
- return cmpxchg64(&v->counter, o, n);
+ return arch_cmpxchg64(&v->counter, o, n);
}
/**
- * atomic64_xchg - xchg atomic64 variable
+ * arch_atomic64_xchg - xchg atomic64 variable
* @v: pointer to type atomic64_t
* @n: value to assign
*
* Atomically xchgs the value of @v to @n and returns
* the old value.
*/
-static inline long long atomic64_xchg(atomic64_t *v, long long n)
+static inline long long arch_atomic64_xchg(atomic64_t *v, long long n)
{
long long o;
unsigned high = (unsigned)(n >> 32);
@@ -96,13 +97,13 @@ static inline long long atomic64_xchg(atomic64_t *v, long long n)
}
/**
- * atomic64_set - set atomic64 variable
+ * arch_atomic64_set - set atomic64 variable
* @v: pointer to type atomic64_t
* @i: value to assign
*
* Atomically sets the value of @v to @n.
*/
-static inline void atomic64_set(atomic64_t *v, long long i)
+static inline void arch_atomic64_set(atomic64_t *v, long long i)
{
unsigned high = (unsigned)(i >> 32);
unsigned low = (unsigned)i;
@@ -112,12 +113,12 @@ static inline void atomic64_set(atomic64_t *v, long long i)
}
/**
- * atomic64_read - read atomic64 variable
+ * arch_atomic64_read - read atomic64 variable
* @v: pointer to type atomic64_t
*
* Atomically reads the value of @v and returns it.
*/
-static inline long long atomic64_read(const atomic64_t *v)
+static inline long long arch_atomic64_read(const atomic64_t *v)
{
long long r;
alternative_atomic64(read, "=&A" (r), "c" (v) : "memory");
@@ -125,13 +126,13 @@ static inline long long atomic64_read(const atomic64_t *v)
}
/**
- * atomic64_add_return - add and return
+ * arch_atomic64_add_return - add and return
* @i: integer value to add
* @v: pointer to type atomic64_t
*
* Atomically adds @i to @v and returns @i + *@v
*/
-static inline long long atomic64_add_return(long long i, atomic64_t *v)
+static inline long long arch_atomic64_add_return(long long i, atomic64_t *v)
{
alternative_atomic64(add_return,
ASM_OUTPUT2("+A" (i), "+c" (v)),
@@ -142,7 +143,7 @@ static inline long long atomic64_add_return(long long i, atomic64_t *v)
/*
* Other variants with different arithmetic operators:
*/
-static inline long long atomic64_sub_return(long long i, atomic64_t *v)
+static inline long long arch_atomic64_sub_return(long long i, atomic64_t *v)
{
alternative_atomic64(sub_return,
ASM_OUTPUT2("+A" (i), "+c" (v)),
@@ -150,7 +151,7 @@ static inline long long atomic64_sub_return(long long i, atomic64_t *v)
return i;
}
-static inline long long atomic64_inc_return(atomic64_t *v)
+static inline long long arch_atomic64_inc_return(atomic64_t *v)
{
long long a;
alternative_atomic64(inc_return, "=&A" (a),
@@ -158,7 +159,7 @@ static inline long long atomic64_inc_return(atomic64_t *v)
return a;
}
-static inline long long atomic64_dec_return(atomic64_t *v)
+static inline long long arch_atomic64_dec_return(atomic64_t *v)
{
long long a;
alternative_atomic64(dec_return, "=&A" (a),
@@ -167,13 +168,13 @@ static inline long long atomic64_dec_return(atomic64_t *v)
}
/**
- * atomic64_add - add integer to atomic64 variable
+ * arch_atomic64_add - add integer to atomic64 variable
* @i: integer value to add
* @v: pointer to type atomic64_t
*
* Atomically adds @i to @v.
*/
-static inline long long atomic64_add(long long i, atomic64_t *v)
+static inline long long arch_atomic64_add(long long i, atomic64_t *v)
{
__alternative_atomic64(add, add_return,
ASM_OUTPUT2("+A" (i), "+c" (v)),
@@ -182,13 +183,13 @@ static inline long long atomic64_add(long long i, atomic64_t *v)
}
/**
- * atomic64_sub - subtract the atomic64 variable
+ * arch_atomic64_sub - subtract the atomic64 variable
* @i: integer value to subtract
* @v: pointer to type atomic64_t
*
* Atomically subtracts @i from @v.
*/
-static inline long long atomic64_sub(long long i, atomic64_t *v)
+static inline long long arch_atomic64_sub(long long i, atomic64_t *v)
{
__alternative_atomic64(sub, sub_return,
ASM_OUTPUT2("+A" (i), "+c" (v)),
@@ -197,7 +198,7 @@ static inline long long atomic64_sub(long long i, atomic64_t *v)
}
/**
- * atomic64_sub_and_test - subtract value from variable and test result
+ * arch_atomic64_sub_and_test - subtract value from variable and test result
* @i: integer value to subtract
* @v: pointer to type atomic64_t
*
@@ -205,46 +206,46 @@ static inline long long atomic64_sub(long long i, atomic64_t *v)
* true if the result is zero, or false for all
* other cases.
*/
-static inline int atomic64_sub_and_test(long long i, atomic64_t *v)
+static inline int arch_atomic64_sub_and_test(long long i, atomic64_t *v)
{
- return atomic64_sub_return(i, v) == 0;
+ return arch_atomic64_sub_return(i, v) == 0;
}
/**
- * atomic64_inc - increment atomic64 variable
+ * arch_atomic64_inc - increment atomic64 variable
* @v: pointer to type atomic64_t
*
* Atomically increments @v by 1.
*/
-static inline void atomic64_inc(atomic64_t *v)
+static inline void arch_atomic64_inc(atomic64_t *v)
{
__alternative_atomic64(inc, inc_return, /* no output */,
"S" (v) : "memory", "eax", "ecx", "edx");
}
/**
- * atomic64_dec - decrement atomic64 variable
+ * arch_atomic64_dec - decrement atomic64 variable
* @v: pointer to type atomic64_t
*
* Atomically decrements @v by 1.
*/
-static inline void atomic64_dec(atomic64_t *v)
+static inline void arch_atomic64_dec(atomic64_t *v)
{
__alternative_atomic64(dec, dec_return, /* no output */,
"S" (v) : "memory", "eax", "ecx", "edx");
}
/**
- * atomic64_dec_and_test - decrement and test
+ * arch_atomic64_dec_and_test - decrement and test
* @v: pointer to type atomic64_t
*
* Atomically decrements @v by 1 and
* returns true if the result is 0, or false for all other
* cases.
*/
-static inline int atomic64_dec_and_test(atomic64_t *v)
+static inline int arch_atomic64_dec_and_test(atomic64_t *v)
{
- return atomic64_dec_return(v) == 0;
+ return arch_atomic64_dec_return(v) == 0;
}
/**
@@ -255,13 +256,13 @@ static inline int atomic64_dec_and_test(atomic64_t *v)
* and returns true if the result is zero, or false for all
* other cases.
*/
-static inline int atomic64_inc_and_test(atomic64_t *v)
+static inline int arch_atomic64_inc_and_test(atomic64_t *v)
{
- return atomic64_inc_return(v) == 0;
+ return arch_atomic64_inc_return(v) == 0;
}
/**
- * atomic64_add_negative - add and test if negative
+ * arch_atomic64_add_negative - add and test if negative
* @i: integer value to add
* @v: pointer to type atomic64_t
*
@@ -269,13 +270,13 @@ static inline int atomic64_inc_and_test(atomic64_t *v)
* if the result is negative, or false when
* result is greater than or equal to zero.
*/
-static inline int atomic64_add_negative(long long i, atomic64_t *v)
+static inline int arch_atomic64_add_negative(long long i, atomic64_t *v)
{
- return atomic64_add_return(i, v) < 0;
+ return arch_atomic64_add_return(i, v) < 0;
}
/**
- * atomic64_add_unless - add unless the number is a given value
+ * arch_atomic64_add_unless - add unless the number is a given value
* @v: pointer of type atomic64_t
* @a: the amount to add to v...
* @u: ...unless v is equal to u.
@@ -283,7 +284,8 @@ static inline int atomic64_add_negative(long long i, atomic64_t *v)
* Atomically adds @a to @v, so long as it was not @u.
* Returns non-zero if the add was done, zero otherwise.
*/
-static inline int atomic64_add_unless(atomic64_t *v, long long a, long long u)
+static inline int arch_atomic64_add_unless(atomic64_t *v, long long a,
+ long long u)
{
unsigned low = (unsigned)u;
unsigned high = (unsigned)(u >> 32);
@@ -294,7 +296,7 @@ static inline int atomic64_add_unless(atomic64_t *v, long long a, long long u)
}
-static inline int atomic64_inc_not_zero(atomic64_t *v)
+static inline int arch_atomic64_inc_not_zero(atomic64_t *v)
{
int r;
alternative_atomic64(inc_not_zero, "=&a" (r),
@@ -302,7 +304,7 @@ static inline int atomic64_inc_not_zero(atomic64_t *v)
return r;
}
-static inline long long atomic64_dec_if_positive(atomic64_t *v)
+static inline long long arch_atomic64_dec_if_positive(atomic64_t *v)
{
long long r;
alternative_atomic64(dec_if_positive, "=&A" (r),
@@ -313,70 +315,70 @@ static inline long long atomic64_dec_if_positive(atomic64_t *v)
#undef alternative_atomic64
#undef __alternative_atomic64
-static inline void atomic64_and(long long i, atomic64_t *v)
+static inline void arch_atomic64_and(long long i, atomic64_t *v)
{
long long old, c = 0;
- while ((old = atomic64_cmpxchg(v, c, c & i)) != c)
+ while ((old = arch_atomic64_cmpxchg(v, c, c & i)) != c)
c = old;
}
-static inline long long atomic64_fetch_and(long long i, atomic64_t *v)
+static inline long long arch_atomic64_fetch_and(long long i, atomic64_t *v)
{
long long old, c = 0;
- while ((old = atomic64_cmpxchg(v, c, c & i)) != c)
+ while ((old = arch_atomic64_cmpxchg(v, c, c & i)) != c)
c = old;
return old;
}
-static inline void atomic64_or(long long i, atomic64_t *v)
+static inline void arch_atomic64_or(long long i, atomic64_t *v)
{
long long old, c = 0;
- while ((old = atomic64_cmpxchg(v, c, c | i)) != c)
+ while ((old = arch_atomic64_cmpxchg(v, c, c | i)) != c)
c = old;
}
-static inline long long atomic64_fetch_or(long long i, atomic64_t *v)
+static inline long long arch_atomic64_fetch_or(long long i, atomic64_t *v)
{
long long old, c = 0;
- while ((old = atomic64_cmpxchg(v, c, c | i)) != c)
+ while ((old = arch_atomic64_cmpxchg(v, c, c | i)) != c)
c = old;
return old;
}
-static inline void atomic64_xor(long long i, atomic64_t *v)
+static inline void arch_atomic64_xor(long long i, atomic64_t *v)
{
long long old, c = 0;
- while ((old = atomic64_cmpxchg(v, c, c ^ i)) != c)
+ while ((old = arch_atomic64_cmpxchg(v, c, c ^ i)) != c)
c = old;
}
-static inline long long atomic64_fetch_xor(long long i, atomic64_t *v)
+static inline long long arch_atomic64_fetch_xor(long long i, atomic64_t *v)
{
long long old, c = 0;
- while ((old = atomic64_cmpxchg(v, c, c ^ i)) != c)
+ while ((old = arch_atomic64_cmpxchg(v, c, c ^ i)) != c)
c = old;
return old;
}
-static inline long long atomic64_fetch_add(long long i, atomic64_t *v)
+static inline long long arch_atomic64_fetch_add(long long i, atomic64_t *v)
{
long long old, c = 0;
- while ((old = atomic64_cmpxchg(v, c, c + i)) != c)
+ while ((old = arch_atomic64_cmpxchg(v, c, c + i)) != c)
c = old;
return old;
}
-#define atomic64_fetch_sub(i, v) atomic64_fetch_add(-(i), (v))
+#define arch_atomic64_fetch_sub(i, v) arch_atomic64_fetch_add(-(i), (v))
#endif /* _ASM_X86_ATOMIC64_32_H */
diff --git a/arch/x86/include/asm/atomic64_64.h b/arch/x86/include/asm/atomic64_64.h
index 738495caf05f..6106b59d3260 100644
--- a/arch/x86/include/asm/atomic64_64.h
+++ b/arch/x86/include/asm/atomic64_64.h
@@ -11,37 +11,37 @@
#define ATOMIC64_INIT(i) { (i) }
/**
- * atomic64_read - read atomic64 variable
+ * arch_atomic64_read - read atomic64 variable
* @v: pointer of type atomic64_t
*
* Atomically reads the value of @v.
* Doesn't imply a read memory barrier.
*/
-static inline long atomic64_read(const atomic64_t *v)
+static inline long arch_atomic64_read(const atomic64_t *v)
{
return READ_ONCE((v)->counter);
}
/**
- * atomic64_set - set atomic64 variable
+ * arch_atomic64_set - set atomic64 variable
* @v: pointer to type atomic64_t
* @i: required value
*
* Atomically sets the value of @v to @i.
*/
-static inline void atomic64_set(atomic64_t *v, long i)
+static inline void arch_atomic64_set(atomic64_t *v, long i)
{
WRITE_ONCE(v->counter, i);
}
/**
- * atomic64_add - add integer to atomic64 variable
+ * arch_atomic64_add - add integer to atomic64 variable
* @i: integer value to add
* @v: pointer to type atomic64_t
*
* Atomically adds @i to @v.
*/
-static __always_inline void atomic64_add(long i, atomic64_t *v)
+static __always_inline void arch_atomic64_add(long i, atomic64_t *v)
{
asm volatile(LOCK_PREFIX "addq %1,%0"
: "=m" (v->counter)
@@ -49,13 +49,13 @@ static __always_inline void atomic64_add(long i, atomic64_t *v)
}
/**
- * atomic64_sub - subtract the atomic64 variable
+ * arch_atomic64_sub - subtract the atomic64 variable
* @i: integer value to subtract
* @v: pointer to type atomic64_t
*
* Atomically subtracts @i from @v.
*/
-static inline void atomic64_sub(long i, atomic64_t *v)
+static inline void arch_atomic64_sub(long i, atomic64_t *v)
{
asm volatile(LOCK_PREFIX "subq %1,%0"
: "=m" (v->counter)
@@ -63,7 +63,7 @@ static inline void atomic64_sub(long i, atomic64_t *v)
}
/**
- * atomic64_sub_and_test - subtract value from variable and test result
+ * arch_atomic64_sub_and_test - subtract value from variable and test result
* @i: integer value to subtract
* @v: pointer to type atomic64_t
*
@@ -71,18 +71,18 @@ static inline void atomic64_sub(long i, atomic64_t *v)
* true if the result is zero, or false for all
* other cases.
*/
-static inline bool atomic64_sub_and_test(long i, atomic64_t *v)
+static inline bool arch_atomic64_sub_and_test(long i, atomic64_t *v)
{
GEN_BINARY_RMWcc(LOCK_PREFIX "subq", v->counter, "er", i, "%0", e);
}
/**
- * atomic64_inc - increment atomic64 variable
+ * arch_atomic64_inc - increment atomic64 variable
* @v: pointer to type atomic64_t
*
* Atomically increments @v by 1.
*/
-static __always_inline void atomic64_inc(atomic64_t *v)
+static __always_inline void arch_atomic64_inc(atomic64_t *v)
{
asm volatile(LOCK_PREFIX "incq %0"
: "=m" (v->counter)
@@ -90,12 +90,12 @@ static __always_inline void atomic64_inc(atomic64_t *v)
}
/**
- * atomic64_dec - decrement atomic64 variable
+ * arch_atomic64_dec - decrement atomic64 variable
* @v: pointer to type atomic64_t
*
* Atomically decrements @v by 1.
*/
-static __always_inline void atomic64_dec(atomic64_t *v)
+static __always_inline void arch_atomic64_dec(atomic64_t *v)
{
asm volatile(LOCK_PREFIX "decq %0"
: "=m" (v->counter)
@@ -103,33 +103,33 @@ static __always_inline void atomic64_dec(atomic64_t *v)
}
/**
- * atomic64_dec_and_test - decrement and test
+ * arch_atomic64_dec_and_test - decrement and test
* @v: pointer to type atomic64_t
*
* Atomically decrements @v by 1 and
* returns true if the result is 0, or false for all other
* cases.
*/
-static inline bool atomic64_dec_and_test(atomic64_t *v)
+static inline bool arch_atomic64_dec_and_test(atomic64_t *v)
{
GEN_UNARY_RMWcc(LOCK_PREFIX "decq", v->counter, "%0", e);
}
/**
- * atomic64_inc_and_test - increment and test
+ * arch_atomic64_inc_and_test - increment and test
* @v: pointer to type atomic64_t
*
* Atomically increments @v by 1
* and returns true if the result is zero, or false for all
* other cases.
*/
-static inline bool atomic64_inc_and_test(atomic64_t *v)
+static inline bool arch_atomic64_inc_and_test(atomic64_t *v)
{
GEN_UNARY_RMWcc(LOCK_PREFIX "incq", v->counter, "%0", e);
}
/**
- * atomic64_add_negative - add and test if negative
+ * arch_atomic64_add_negative - add and test if negative
* @i: integer value to add
* @v: pointer to type atomic64_t
*
@@ -137,59 +137,59 @@ static inline bool atomic64_inc_and_test(atomic64_t *v)
* if the result is negative, or false when
* result is greater than or equal to zero.
*/
-static inline bool atomic64_add_negative(long i, atomic64_t *v)
+static inline bool arch_atomic64_add_negative(long i, atomic64_t *v)
{
GEN_BINARY_RMWcc(LOCK_PREFIX "addq", v->counter, "er", i, "%0", s);
}
/**
- * atomic64_add_return - add and return
+ * arch_atomic64_add_return - add and return
* @i: integer value to add
* @v: pointer to type atomic64_t
*
* Atomically adds @i to @v and returns @i + @v
*/
-static __always_inline long atomic64_add_return(long i, atomic64_t *v)
+static __always_inline long arch_atomic64_add_return(long i, atomic64_t *v)
{
return i + xadd(&v->counter, i);
}
-static inline long atomic64_sub_return(long i, atomic64_t *v)
+static inline long arch_atomic64_sub_return(long i, atomic64_t *v)
{
- return atomic64_add_return(-i, v);
+ return arch_atomic64_add_return(-i, v);
}
-static inline long atomic64_fetch_add(long i, atomic64_t *v)
+static inline long arch_atomic64_fetch_add(long i, atomic64_t *v)
{
return xadd(&v->counter, i);
}
-static inline long atomic64_fetch_sub(long i, atomic64_t *v)
+static inline long arch_atomic64_fetch_sub(long i, atomic64_t *v)
{
return xadd(&v->counter, -i);
}
-#define atomic64_inc_return(v) (atomic64_add_return(1, (v)))
-#define atomic64_dec_return(v) (atomic64_sub_return(1, (v)))
+#define arch_atomic64_inc_return(v) (arch_atomic64_add_return(1, (v)))
+#define arch_atomic64_dec_return(v) (arch_atomic64_sub_return(1, (v)))
-static inline long atomic64_cmpxchg(atomic64_t *v, long old, long new)
+static inline long arch_atomic64_cmpxchg(atomic64_t *v, long old, long new)
{
- return cmpxchg(&v->counter, old, new);
+ return arch_cmpxchg(&v->counter, old, new);
}
-#define atomic64_try_cmpxchg atomic64_try_cmpxchg
-static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, long new)
+#define arch_atomic64_try_cmpxchg arch_atomic64_try_cmpxchg
+static __always_inline bool arch_atomic64_try_cmpxchg(atomic64_t *v, s64 *old, long new)
{
return try_cmpxchg(&v->counter, old, new);
}
-static inline long atomic64_xchg(atomic64_t *v, long new)
+static inline long arch_atomic64_xchg(atomic64_t *v, long new)
{
return xchg(&v->counter, new);
}
/**
- * atomic64_add_unless - add unless the number is a given value
+ * arch_atomic64_add_unless - add unless the number is a given value
* @v: pointer of type atomic64_t
* @a: the amount to add to v...
* @u: ...unless v is equal to u.
@@ -197,37 +197,37 @@ static inline long atomic64_xchg(atomic64_t *v, long new)
* Atomically adds @a to @v, so long as it was not @u.
* Returns the old value of @v.
*/
-static inline bool atomic64_add_unless(atomic64_t *v, long a, long u)
+static inline bool arch_atomic64_add_unless(atomic64_t *v, long a, long u)
{
- s64 c = atomic64_read(v);
+ s64 c = arch_atomic64_read(v);
do {
if (unlikely(c == u))
return false;
- } while (!atomic64_try_cmpxchg(v, &c, c + a));
+ } while (!arch_atomic64_try_cmpxchg(v, &c, c + a));
return true;
}
-#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0)
+#define arch_atomic64_inc_not_zero(v) arch_atomic64_add_unless((v), 1, 0)
/*
- * atomic64_dec_if_positive - decrement by 1 if old value positive
+ * arch_atomic64_dec_if_positive - decrement by 1 if old value positive
* @v: pointer of type atomic_t
*
* The function returns the old value of *v minus 1, even if
* the atomic variable, v, was not decremented.
*/
-static inline long atomic64_dec_if_positive(atomic64_t *v)
+static inline long arch_atomic64_dec_if_positive(atomic64_t *v)
{
- s64 dec, c = atomic64_read(v);
+ s64 dec, c = arch_atomic64_read(v);
do {
dec = c - 1;
if (unlikely(dec < 0))
break;
- } while (!atomic64_try_cmpxchg(v, &c, dec));
+ } while (!arch_atomic64_try_cmpxchg(v, &c, dec));
return dec;
}
-static inline void atomic64_and(long i, atomic64_t *v)
+static inline void arch_atomic64_and(long i, atomic64_t *v)
{
asm volatile(LOCK_PREFIX "andq %1,%0"
: "+m" (v->counter)
@@ -235,16 +235,16 @@ static inline void atomic64_and(long i, atomic64_t *v)
: "memory");
}
-static inline long atomic64_fetch_and(long i, atomic64_t *v)
+static inline long arch_atomic64_fetch_and(long i, atomic64_t *v)
{
- s64 val = atomic64_read(v);
+ s64 val = arch_atomic64_read(v);
do {
- } while (!atomic64_try_cmpxchg(v, &val, val & i));
+ } while (!arch_atomic64_try_cmpxchg(v, &val, val & i));
return val;
}
-static inline void atomic64_or(long i, atomic64_t *v)
+static inline void arch_atomic64_or(long i, atomic64_t *v)
{
asm volatile(LOCK_PREFIX "orq %1,%0"
: "+m" (v->counter)
@@ -252,16 +252,16 @@ static inline void atomic64_or(long i, atomic64_t *v)
: "memory");
}
-static inline long atomic64_fetch_or(long i, atomic64_t *v)
+static inline long arch_atomic64_fetch_or(long i, atomic64_t *v)
{
- s64 val = atomic64_read(v);
+ s64 val = arch_atomic64_read(v);
do {
- } while (!atomic64_try_cmpxchg(v, &val, val | i));
+ } while (!arch_atomic64_try_cmpxchg(v, &val, val | i));
return val;
}
-static inline void atomic64_xor(long i, atomic64_t *v)
+static inline void arch_atomic64_xor(long i, atomic64_t *v)
{
asm volatile(LOCK_PREFIX "xorq %1,%0"
: "+m" (v->counter)
@@ -269,12 +269,12 @@ static inline void atomic64_xor(long i, atomic64_t *v)
: "memory");
}
-static inline long atomic64_fetch_xor(long i, atomic64_t *v)
+static inline long arch_atomic64_fetch_xor(long i, atomic64_t *v)
{
- s64 val = atomic64_read(v);
+ s64 val = arch_atomic64_read(v);
do {
- } while (!atomic64_try_cmpxchg(v, &val, val ^ i));
+ } while (!arch_atomic64_try_cmpxchg(v, &val, val ^ i));
return val;
}
diff --git a/arch/x86/include/asm/cmpxchg.h b/arch/x86/include/asm/cmpxchg.h
index 56bd436ed01b..e3efd8a06066 100644
--- a/arch/x86/include/asm/cmpxchg.h
+++ b/arch/x86/include/asm/cmpxchg.h
@@ -145,13 +145,13 @@ extern void __add_wrong_size(void)
# include <asm/cmpxchg_64.h>
#endif
-#define cmpxchg(ptr, old, new) \
+#define arch_cmpxchg(ptr, old, new) \
__cmpxchg(ptr, old, new, sizeof(*(ptr)))
-#define sync_cmpxchg(ptr, old, new) \
+#define arch_sync_cmpxchg(ptr, old, new) \
__sync_cmpxchg(ptr, old, new, sizeof(*(ptr)))
-#define cmpxchg_local(ptr, old, new) \
+#define arch_cmpxchg_local(ptr, old, new) \
__cmpxchg_local(ptr, old, new, sizeof(*(ptr)))
@@ -221,7 +221,7 @@ extern void __add_wrong_size(void)
#define __try_cmpxchg(ptr, pold, new, size) \
__raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
-#define try_cmpxchg(ptr, pold, new) \
+#define try_cmpxchg(ptr, pold, new) \
__try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
/*
@@ -250,10 +250,10 @@ extern void __add_wrong_size(void)
__ret; \
})
-#define cmpxchg_double(p1, p2, o1, o2, n1, n2) \
+#define arch_cmpxchg_double(p1, p2, o1, o2, n1, n2) \
__cmpxchg_double(LOCK_PREFIX, p1, p2, o1, o2, n1, n2)
-#define cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \
+#define arch_cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \
__cmpxchg_double(, p1, p2, o1, o2, n1, n2)
#endif /* ASM_X86_CMPXCHG_H */
diff --git a/arch/x86/include/asm/cmpxchg_32.h b/arch/x86/include/asm/cmpxchg_32.h
index 1732704f0445..1a2eafca7038 100644
--- a/arch/x86/include/asm/cmpxchg_32.h
+++ b/arch/x86/include/asm/cmpxchg_32.h
@@ -36,10 +36,10 @@ static inline void set_64bit(volatile u64 *ptr, u64 value)
}
#ifdef CONFIG_X86_CMPXCHG64
-#define cmpxchg64(ptr, o, n) \
+#define arch_cmpxchg64(ptr, o, n) \
((__typeof__(*(ptr)))__cmpxchg64((ptr), (unsigned long long)(o), \
(unsigned long long)(n)))
-#define cmpxchg64_local(ptr, o, n) \
+#define arch_cmpxchg64_local(ptr, o, n) \
((__typeof__(*(ptr)))__cmpxchg64_local((ptr), (unsigned long long)(o), \
(unsigned long long)(n)))
#endif
@@ -76,7 +76,7 @@ static inline u64 __cmpxchg64_local(volatile u64 *ptr, u64 old, u64 new)
* to simulate the cmpxchg8b on the 80386 and 80486 CPU.
*/
-#define cmpxchg64(ptr, o, n) \
+#define arch_cmpxchg64(ptr, o, n) \
({ \
__typeof__(*(ptr)) __ret; \
__typeof__(*(ptr)) __old = (o); \
@@ -93,7 +93,7 @@ static inline u64 __cmpxchg64_local(volatile u64 *ptr, u64 old, u64 new)
__ret; })
-#define cmpxchg64_local(ptr, o, n) \
+#define arch_cmpxchg64_local(ptr, o, n) \
({ \
__typeof__(*(ptr)) __ret; \
__typeof__(*(ptr)) __old = (o); \
diff --git a/arch/x86/include/asm/cmpxchg_64.h b/arch/x86/include/asm/cmpxchg_64.h
index 03cad196a301..bfca3b346c74 100644
--- a/arch/x86/include/asm/cmpxchg_64.h
+++ b/arch/x86/include/asm/cmpxchg_64.h
@@ -7,13 +7,13 @@ static inline void set_64bit(volatile u64 *ptr, u64 val)
*ptr = val;
}
-#define cmpxchg64(ptr, o, n) \
+#define arch_cmpxchg64(ptr, o, n) \
({ \
BUILD_BUG_ON(sizeof(*(ptr)) != 8); \
cmpxchg((ptr), (o), (n)); \
})
-#define cmpxchg64_local(ptr, o, n) \
+#define arch_cmpxchg64_local(ptr, o, n) \
({ \
BUILD_BUG_ON(sizeof(*(ptr)) != 8); \
cmpxchg_local((ptr), (o), (n)); \
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v6 3/4] asm-generic: add KASAN instrumentation to atomic operations
2018-01-29 17:26 [PATCH v6 0/4] x86, kasan: add KASAN checks to atomic operations Dmitry Vyukov
2018-01-29 17:26 ` [PATCH v6 1/4] locking/atomic: Add asm-generic/atomic-instrumented.h Dmitry Vyukov
2018-01-29 17:26 ` [PATCH v6 2/4] x86: switch atomic.h to use atomic-instrumented.h Dmitry Vyukov
@ 2018-01-29 17:26 ` Dmitry Vyukov
2018-03-12 12:24 ` [tip:locking/core] locking/atomic, asm-generic: Add " tip-bot for Dmitry Vyukov
2018-01-29 17:26 ` [PATCH v6 4/4] asm-generic, x86: add comments for atomic instrumentation Dmitry Vyukov
` (2 subsequent siblings)
5 siblings, 1 reply; 38+ messages in thread
From: Dmitry Vyukov @ 2018-01-29 17:26 UTC (permalink / raw)
To: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin,
kasan-dev, x86, linux-kernel, tglx
Cc: Dmitry Vyukov, Andrew Morton, linux-mm
KASAN uses compiler instrumentation to intercept all memory accesses. But it does
not see memory accesses done in assembly code. One notable user of assembly code
is atomic operations. Frequently, for example, an atomic reference decrement is
the last access to an object and a good candidate for a racy use-after-free.
Add manual KASAN checks to atomic operations.
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Will Deacon <will.deacon@arm.com>,
Cc: Andrew Morton <akpm@linux-foundation.org>,
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>,
Cc: Ingo Molnar <mingo@redhat.com>,
Cc: kasan-dev@googlegroups.com
Cc: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org
Cc: x86@kernel.org
---
include/asm-generic/atomic-instrumented.h | 62 +++++++++++++++++++++++++++++++
1 file changed, 62 insertions(+)
diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h
index b966194d120a..82e080505982 100644
--- a/include/asm-generic/atomic-instrumented.h
+++ b/include/asm-generic/atomic-instrumented.h
@@ -2,44 +2,53 @@
#define _LINUX_ATOMIC_INSTRUMENTED_H
#include <linux/build_bug.h>
+#include <linux/kasan-checks.h>
static __always_inline int atomic_read(const atomic_t *v)
{
+ kasan_check_read(v, sizeof(*v));
return arch_atomic_read(v);
}
static __always_inline s64 atomic64_read(const atomic64_t *v)
{
+ kasan_check_read(v, sizeof(*v));
return arch_atomic64_read(v);
}
static __always_inline void atomic_set(atomic_t *v, int i)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_set(v, i);
}
static __always_inline void atomic64_set(atomic64_t *v, s64 i)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_set(v, i);
}
static __always_inline int atomic_xchg(atomic_t *v, int i)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_xchg(v, i);
}
static __always_inline s64 atomic64_xchg(atomic64_t *v, s64 i)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_xchg(v, i);
}
static __always_inline int atomic_cmpxchg(atomic_t *v, int old, int new)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_cmpxchg(v, old, new);
}
static __always_inline s64 atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_cmpxchg(v, old, new);
}
@@ -47,6 +56,8 @@ static __always_inline s64 atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new)
#define atomic_try_cmpxchg atomic_try_cmpxchg
static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new)
{
+ kasan_check_write(v, sizeof(*v));
+ kasan_check_read(old, sizeof(*old));
return arch_atomic_try_cmpxchg(v, old, new);
}
#endif
@@ -55,234 +66,281 @@ static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new)
#define atomic64_try_cmpxchg atomic64_try_cmpxchg
static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, s64 new)
{
+ kasan_check_write(v, sizeof(*v));
+ kasan_check_read(old, sizeof(*old));
return arch_atomic64_try_cmpxchg(v, old, new);
}
#endif
static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u)
{
+ kasan_check_write(v, sizeof(*v));
return __arch_atomic_add_unless(v, a, u);
}
static __always_inline bool atomic64_add_unless(atomic64_t *v, s64 a, s64 u)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_add_unless(v, a, u);
}
static __always_inline void atomic_inc(atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_inc(v);
}
static __always_inline void atomic64_inc(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_inc(v);
}
static __always_inline void atomic_dec(atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_dec(v);
}
static __always_inline void atomic64_dec(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_dec(v);
}
static __always_inline void atomic_add(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_add(i, v);
}
static __always_inline void atomic64_add(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_add(i, v);
}
static __always_inline void atomic_sub(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_sub(i, v);
}
static __always_inline void atomic64_sub(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_sub(i, v);
}
static __always_inline void atomic_and(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_and(i, v);
}
static __always_inline void atomic64_and(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_and(i, v);
}
static __always_inline void atomic_or(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_or(i, v);
}
static __always_inline void atomic64_or(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_or(i, v);
}
static __always_inline void atomic_xor(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_xor(i, v);
}
static __always_inline void atomic64_xor(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_xor(i, v);
}
static __always_inline int atomic_inc_return(atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_inc_return(v);
}
static __always_inline s64 atomic64_inc_return(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_inc_return(v);
}
static __always_inline int atomic_dec_return(atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_dec_return(v);
}
static __always_inline s64 atomic64_dec_return(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_dec_return(v);
}
static __always_inline s64 atomic64_inc_not_zero(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_inc_not_zero(v);
}
static __always_inline s64 atomic64_dec_if_positive(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_dec_if_positive(v);
}
static __always_inline bool atomic_dec_and_test(atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_dec_and_test(v);
}
static __always_inline bool atomic64_dec_and_test(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_dec_and_test(v);
}
static __always_inline bool atomic_inc_and_test(atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_inc_and_test(v);
}
static __always_inline bool atomic64_inc_and_test(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_inc_and_test(v);
}
static __always_inline int atomic_add_return(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_add_return(i, v);
}
static __always_inline s64 atomic64_add_return(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_add_return(i, v);
}
static __always_inline int atomic_sub_return(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_sub_return(i, v);
}
static __always_inline s64 atomic64_sub_return(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_sub_return(i, v);
}
static __always_inline int atomic_fetch_add(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_fetch_add(i, v);
}
static __always_inline s64 atomic64_fetch_add(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_fetch_add(i, v);
}
static __always_inline int atomic_fetch_sub(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_fetch_sub(i, v);
}
static __always_inline s64 atomic64_fetch_sub(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_fetch_sub(i, v);
}
static __always_inline int atomic_fetch_and(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_fetch_and(i, v);
}
static __always_inline s64 atomic64_fetch_and(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_fetch_and(i, v);
}
static __always_inline int atomic_fetch_or(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_fetch_or(i, v);
}
static __always_inline s64 atomic64_fetch_or(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_fetch_or(i, v);
}
static __always_inline int atomic_fetch_xor(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_fetch_xor(i, v);
}
static __always_inline s64 atomic64_fetch_xor(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_fetch_xor(i, v);
}
static __always_inline bool atomic_sub_and_test(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_sub_and_test(i, v);
}
static __always_inline bool atomic64_sub_and_test(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_sub_and_test(i, v);
}
static __always_inline bool atomic_add_negative(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_add_negative(i, v);
}
static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_add_negative(i, v);
}
static __always_inline unsigned long
cmpxchg_size(volatile void *ptr, unsigned long old, unsigned long new, int size)
{
+ kasan_check_write(ptr, size);
switch (size) {
case 1:
return arch_cmpxchg((u8 *)ptr, (u8)old, (u8)new);
@@ -308,6 +366,7 @@ static __always_inline unsigned long
sync_cmpxchg_size(volatile void *ptr, unsigned long old, unsigned long new,
int size)
{
+ kasan_check_write(ptr, size);
switch (size) {
case 1:
return arch_sync_cmpxchg((u8 *)ptr, (u8)old, (u8)new);
@@ -334,6 +393,7 @@ static __always_inline unsigned long
cmpxchg_local_size(volatile void *ptr, unsigned long old, unsigned long new,
int size)
{
+ kasan_check_write(ptr, size);
switch (size) {
case 1:
return arch_cmpxchg_local((u8 *)ptr, (u8)old, (u8)new);
@@ -359,6 +419,7 @@ cmpxchg_local_size(volatile void *ptr, unsigned long old, unsigned long new,
static __always_inline u64
cmpxchg64_size(volatile u64 *ptr, u64 old, u64 new)
{
+ kasan_check_write(ptr, sizeof(*ptr));
return arch_cmpxchg64(ptr, old, new);
}
@@ -371,6 +432,7 @@ cmpxchg64_size(volatile u64 *ptr, u64 old, u64 new)
static __always_inline u64
cmpxchg64_local_size(volatile u64 *ptr, u64 old, u64 new)
{
+ kasan_check_write(ptr, sizeof(*ptr));
return arch_cmpxchg64_local(ptr, old, new);
}
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [PATCH v6 4/4] asm-generic, x86: add comments for atomic instrumentation
2018-01-29 17:26 [PATCH v6 0/4] x86, kasan: add KASAN checks to atomic operations Dmitry Vyukov
` (2 preceding siblings ...)
2018-01-29 17:26 ` [PATCH v6 3/4] asm-generic: add KASAN instrumentation to atomic operations Dmitry Vyukov
@ 2018-01-29 17:26 ` Dmitry Vyukov
2018-03-12 12:25 ` [tip:locking/core] locking/atomic, asm-generic, x86: Add " tip-bot for Dmitry Vyukov
2018-01-30 9:23 ` [PATCH v6 0/4] x86, kasan: add KASAN checks to atomic operations Dmitry Vyukov
2018-01-30 15:36 ` Will Deacon
5 siblings, 1 reply; 38+ messages in thread
From: Dmitry Vyukov @ 2018-01-29 17:26 UTC (permalink / raw)
To: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin,
kasan-dev, x86, linux-kernel, tglx
Cc: Dmitry Vyukov, Andrew Morton, linux-mm
The comments are factored out from the code changes to make them
easier to read. Add them separately to explain some non-obvious
aspects.
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Will Deacon <will.deacon@arm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: kasan-dev@googlegroups.com
Cc: linux-mm@kvack.org
Cc: linux-kernel@vger.kernel.org
Cc: x86@kernel.org
---
Changes since v3:
- rephrase comment in arch_atomic_read()
Changes since v5:
- remove comment explaining cmpxchg macro mess,
since we don't have the mess now.
---
arch/x86/include/asm/atomic.h | 4 ++++
include/asm-generic/atomic-instrumented.h | 21 +++++++++++++++++++++
2 files changed, 25 insertions(+)
diff --git a/arch/x86/include/asm/atomic.h b/arch/x86/include/asm/atomic.h
index 33afc966d6a9..0db6bec95489 100644
--- a/arch/x86/include/asm/atomic.h
+++ b/arch/x86/include/asm/atomic.h
@@ -24,6 +24,10 @@
*/
static __always_inline int arch_atomic_read(const atomic_t *v)
{
+ /*
+ * Note for KASAN: we deliberately don't use READ_ONCE_NOCHECK() here,
+ * it's non-inlined function that increases binary size and stack usage.
+ */
return READ_ONCE((v)->counter);
}
diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h
index 82e080505982..ec07f23678ea 100644
--- a/include/asm-generic/atomic-instrumented.h
+++ b/include/asm-generic/atomic-instrumented.h
@@ -1,3 +1,15 @@
+/*
+ * This file provides wrappers with KASAN instrumentation for atomic operations.
+ * To use this functionality an arch's atomic.h file needs to define all
+ * atomic operations with arch_ prefix (e.g. arch_atomic_read()) and include
+ * this file at the end. This file provides atomic_read() that forwards to
+ * arch_atomic_read() for actual atomic operation.
+ * Note: if an arch atomic operation is implemented by means of other atomic
+ * operations (e.g. atomic_read()/atomic_cmpxchg() loop), then it needs to use
+ * arch_ variants (i.e. arch_atomic_read()/arch_atomic_cmpxchg()) to avoid
+ * double instrumentation.
+ */
+
#ifndef _LINUX_ATOMIC_INSTRUMENTED_H
#define _LINUX_ATOMIC_INSTRUMENTED_H
@@ -442,6 +454,15 @@ cmpxchg64_local_size(volatile u64 *ptr, u64 old, u64 new)
(u64)(new))); \
})
+/*
+ * Originally we had the following code here:
+ * __typeof__(p1) ____p1 = (p1);
+ * kasan_check_write(____p1, 2 * sizeof(*____p1));
+ * arch_cmpxchg_double(____p1, (p2), (o1), (o2), (n1), (n2));
+ * But it leads to compilation failures (see gcc issue 72873).
+ * So for now it's left non-instrumented.
+ * There are few callers of cmpxchg_double(), so it's not critical.
+ */
#define cmpxchg_double(p1, p2, o1, o2, n1, n2) \
({ \
arch_cmpxchg_double((p1), (p2), (o1), (o2), (n1), (n2)); \
--
2.16.0.rc1.238.g530d649a79-goog
^ permalink raw reply related [flat|nested] 38+ messages in thread
* Re: [PATCH v6 0/4] x86, kasan: add KASAN checks to atomic operations
2018-01-29 17:26 [PATCH v6 0/4] x86, kasan: add KASAN checks to atomic operations Dmitry Vyukov
` (3 preceding siblings ...)
2018-01-29 17:26 ` [PATCH v6 4/4] asm-generic, x86: add comments for atomic instrumentation Dmitry Vyukov
@ 2018-01-30 9:23 ` Dmitry Vyukov
2018-01-30 9:27 ` Dmitry Vyukov
2018-01-30 15:36 ` Will Deacon
5 siblings, 1 reply; 38+ messages in thread
From: Dmitry Vyukov @ 2018-01-30 9:23 UTC (permalink / raw)
To: Mark Rutland, Peter Zijlstra, Ingo Molnar, Will Deacon,
H. Peter Anvin, Andrey Ryabinin, kasan-dev,
the arch/x86 maintainers, LKML, Thomas Gleixner
Cc: Dmitry Vyukov
On Mon, Jan 29, 2018 at 6:26 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
> KASAN uses compiler instrumentation to intercept all memory accesses.
> But it does not see memory accesses done in assembly code.
> One notable user of assembly code is atomic operations. Frequently,
> for example, an atomic reference decrement is the last access to an
> object and a good candidate for a racy use-after-free.
>
> Atomic operations are defined in arch files, but KASAN instrumentation
> is required for several archs that support KASAN. Later we will need
> similar hooks for KMSAN (uninit use detector) and KTSAN (data race
> detector).
>
> This change introduces wrappers around atomic operations that can be
> used to add KASAN/KMSAN/KTSAN instrumentation across several archs,
> and adds KASAN checks to them.
>
> This patch uses the wrappers only for x86 arch. Arm64 will be switched
> later. And we also plan to instrument bitops in a similar way.
>
> Within a day it has found its first bug:
>
> BUG: KASAN: use-after-free in atomic_dec_and_test
> arch/x86/include/asm/atomic.h:123 [inline] at addr ffff880079c30158
> Write of size 4 by task syz-executor6/25698
> CPU: 2 PID: 25698 Comm: syz-executor6 Not tainted 4.10.0+ #302
> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
> Call Trace:
> kasan_check_write+0x14/0x20 mm/kasan/kasan.c:344
> atomic_dec_and_test arch/x86/include/asm/atomic.h:123 [inline]
> put_task_struct include/linux/sched/task.h:93 [inline]
> put_ctx+0xcf/0x110 kernel/events/core.c:1131
> perf_event_release_kernel+0x3ad/0xc90 kernel/events/core.c:4322
> perf_release+0x37/0x50 kernel/events/core.c:4338
> __fput+0x332/0x800 fs/file_table.c:209
> ____fput+0x15/0x20 fs/file_table.c:245
> task_work_run+0x197/0x260 kernel/task_work.c:116
> exit_task_work include/linux/task_work.h:21 [inline]
> do_exit+0xb38/0x29c0 kernel/exit.c:880
> do_group_exit+0x149/0x420 kernel/exit.c:984
> get_signal+0x7e0/0x1820 kernel/signal.c:2318
> do_signal+0xd2/0x2190 arch/x86/kernel/signal.c:808
> exit_to_usermode_loop+0x200/0x2a0 arch/x86/entry/common.c:157
> syscall_return_slowpath arch/x86/entry/common.c:191 [inline]
> do_syscall_64+0x6fc/0x930 arch/x86/entry/common.c:286
> entry_SYSCALL64_slow_path+0x25/0x25
> RIP: 0033:0x4458d9
> RSP: 002b:00007f3f07187cf8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
> RAX: fffffffffffffe00 RBX: 00000000007080c8 RCX: 00000000004458d9
> RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00000000007080c8
> RBP: 00000000007080a8 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
> R13: 0000000000000000 R14: 00007f3f071889c0 R15: 00007f3f07188700
> Object at ffff880079c30140, in cache task_struct size: 5376
> Allocated:
> PID = 25681
> kmem_cache_alloc_node+0x122/0x6f0 mm/slab.c:3662
> alloc_task_struct_node kernel/fork.c:153 [inline]
> dup_task_struct kernel/fork.c:495 [inline]
> copy_process.part.38+0x19c8/0x4aa0 kernel/fork.c:1560
> copy_process kernel/fork.c:1531 [inline]
> _do_fork+0x200/0x1010 kernel/fork.c:1994
> SYSC_clone kernel/fork.c:2104 [inline]
> SyS_clone+0x37/0x50 kernel/fork.c:2098
> do_syscall_64+0x2e8/0x930 arch/x86/entry/common.c:281
> return_from_SYSCALL_64+0x0/0x7a
> Freed:
> PID = 25681
> __cache_free mm/slab.c:3514 [inline]
> kmem_cache_free+0x71/0x240 mm/slab.c:3774
> free_task_struct kernel/fork.c:158 [inline]
> free_task+0x151/0x1d0 kernel/fork.c:370
> copy_process.part.38+0x18e5/0x4aa0 kernel/fork.c:1931
> copy_process kernel/fork.c:1531 [inline]
> _do_fork+0x200/0x1010 kernel/fork.c:1994
> SYSC_clone kernel/fork.c:2104 [inline]
> SyS_clone+0x37/0x50 kernel/fork.c:2098
> do_syscall_64+0x2e8/0x930 arch/x86/entry/common.c:281
> return_from_SYSCALL_64+0x0/0x7a
>
> Changes since v1:
> - dropped "x86: remove unused atomic_inc_short()" patch
> it is mailed separately
> - rebased on top of tip/locking/core head
> - other changes noted within individual patches
>
> Changes since v2:
> - rebased on top of tip/locking/core head
> - dropped a pervasive "x86: use long long for 64-bit atomic ops" commit,
> instead use s64 type in wrappers
> - added "x86: use s64* for old arg of atomic64_try_cmpxchg()" commit
>
> Changes since v3 are noted in individual commits.
>
> Changes since v4:
> - rebased on tip/locking/core HEAD
>
> Changes since v5:
> - rework cmpxchg* implementations so that we have less
> code in macros and more code in functions
Some context.
This revives a half-year old patch. v5 of this was applied to
tip/locking/core, but then reverted due to a reported crash:
https://groups.google.com/d/msg/kasan-dev/ZJl66N7smmk/lJY99HmmAgAJ
The root cause was in the cmpxchg macros:
#define cmpxchg64(ptr, old, new) \
({ \
__typeof__(ptr) ____ptr = (ptr); \
kasan_check_write(____ptr, sizeof(*____ptr)); \
arch_cmpxchg64(____ptr, (old), (new)); \
})
I had to introduce the new ____ptr variable, so that
kasan_check_write() and arch_cmpxchg64() don't evaluate ptr twice. But
there are multiple layers of macros and ____ptr ended up referring to
something else (to itself).
v6 changes cmpxchg macros to (as was suggested by Mark and Thomas):
static __always_inline unsigned long
cmpxchg_size(volatile void *ptr, unsigned long old, unsigned long new, int size)
{
kasan_check_write(ptr, size);
switch (size) {
case 1:
return arch_cmpxchg((u8 *)ptr, (u8)old, (u8)new);
case 2:
return arch_cmpxchg((u16 *)ptr, (u16)old, (u16)new);
case 4:
return arch_cmpxchg((u32 *)ptr, (u32)old, (u32)new);
case 8:
BUILD_BUG_ON(sizeof(unsigned long) != 8);
return arch_cmpxchg((u64 *)ptr, (u64)old, (u64)new);
}
BUILD_BUG();
return 0;
}
#define cmpxchg(ptr, old, new) \
({ \
((__typeof__(*(ptr)))cmpxchg_size((ptr), (unsigned long)(old), \
(unsigned long)(new), sizeof(*(ptr)))); \
})
Otherwise the patch series were rebased without any conflicts (surprisingly).
There is now some duplication between cmpxchg_size, cmpxchg_local_size
and sync_cmpxchg_size, but I thought that employing more macros while
trying to resolve macro mess bugs is not the best idea.
> Tested:
> - build/boot x86_64 defconfig
> - build/boot x86_64 defconfig+KASAN
> - build i686/powerpc defconfig
>
> Dmitry Vyukov (4):
> locking/atomic: Add asm-generic/atomic-instrumented.h
> x86: switch atomic.h to use atomic-instrumented.h
> asm-generic: add KASAN instrumentation to atomic operations
> asm-generic, x86: add comments for atomic instrumentation
>
> arch/x86/include/asm/atomic.h | 106 +++----
> arch/x86/include/asm/atomic64_32.h | 106 +++----
> arch/x86/include/asm/atomic64_64.h | 108 +++----
> arch/x86/include/asm/cmpxchg.h | 12 +-
> arch/x86/include/asm/cmpxchg_32.h | 8 +-
> arch/x86/include/asm/cmpxchg_64.h | 4 +-
> include/asm-generic/atomic-instrumented.h | 476 ++++++++++++++++++++++++++++++
> 7 files changed, 652 insertions(+), 168 deletions(-)
> create mode 100644 include/asm-generic/atomic-instrumented.h
>
> --
> 2.16.0.rc1.238.g530d649a79-goog
>
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v6 0/4] x86, kasan: add KASAN checks to atomic operations
2018-01-30 9:23 ` [PATCH v6 0/4] x86, kasan: add KASAN checks to atomic operations Dmitry Vyukov
@ 2018-01-30 9:27 ` Dmitry Vyukov
0 siblings, 0 replies; 38+ messages in thread
From: Dmitry Vyukov @ 2018-01-30 9:27 UTC (permalink / raw)
To: Mark Rutland, Peter Zijlstra, Ingo Molnar, Will Deacon,
H. Peter Anvin, Andrey Ryabinin, kasan-dev,
the arch/x86 maintainers, LKML, Thomas Gleixner
Cc: Dmitry Vyukov
On Tue, Jan 30, 2018 at 10:23 AM, Dmitry Vyukov <dvyukov@google.com> wrote:
> On Mon, Jan 29, 2018 at 6:26 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
>> KASAN uses compiler instrumentation to intercept all memory accesses.
>> But it does not see memory accesses done in assembly code.
>> One notable user of assembly code is atomic operations. Frequently,
>> for example, an atomic reference decrement is the last access to an
>> object and a good candidate for a racy use-after-free.
>>
>> Atomic operations are defined in arch files, but KASAN instrumentation
>> is required for several archs that support KASAN. Later we will need
>> similar hooks for KMSAN (uninit use detector) and KTSAN (data race
>> detector).
>>
>> This change introduces wrappers around atomic operations that can be
>> used to add KASAN/KMSAN/KTSAN instrumentation across several archs,
>> and adds KASAN checks to them.
>>
>> This patch uses the wrappers only for x86 arch. Arm64 will be switched
>> later. And we also plan to instrument bitops in a similar way.
>>
>> Within a day it has found its first bug:
>>
>> BUG: KASAN: use-after-free in atomic_dec_and_test
>> arch/x86/include/asm/atomic.h:123 [inline] at addr ffff880079c30158
>> Write of size 4 by task syz-executor6/25698
>> CPU: 2 PID: 25698 Comm: syz-executor6 Not tainted 4.10.0+ #302
>> Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
>> Call Trace:
>> kasan_check_write+0x14/0x20 mm/kasan/kasan.c:344
>> atomic_dec_and_test arch/x86/include/asm/atomic.h:123 [inline]
>> put_task_struct include/linux/sched/task.h:93 [inline]
>> put_ctx+0xcf/0x110 kernel/events/core.c:1131
>> perf_event_release_kernel+0x3ad/0xc90 kernel/events/core.c:4322
>> perf_release+0x37/0x50 kernel/events/core.c:4338
>> __fput+0x332/0x800 fs/file_table.c:209
>> ____fput+0x15/0x20 fs/file_table.c:245
>> task_work_run+0x197/0x260 kernel/task_work.c:116
>> exit_task_work include/linux/task_work.h:21 [inline]
>> do_exit+0xb38/0x29c0 kernel/exit.c:880
>> do_group_exit+0x149/0x420 kernel/exit.c:984
>> get_signal+0x7e0/0x1820 kernel/signal.c:2318
>> do_signal+0xd2/0x2190 arch/x86/kernel/signal.c:808
>> exit_to_usermode_loop+0x200/0x2a0 arch/x86/entry/common.c:157
>> syscall_return_slowpath arch/x86/entry/common.c:191 [inline]
>> do_syscall_64+0x6fc/0x930 arch/x86/entry/common.c:286
>> entry_SYSCALL64_slow_path+0x25/0x25
>> RIP: 0033:0x4458d9
>> RSP: 002b:00007f3f07187cf8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
>> RAX: fffffffffffffe00 RBX: 00000000007080c8 RCX: 00000000004458d9
>> RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00000000007080c8
>> RBP: 00000000007080a8 R08: 0000000000000000 R09: 0000000000000000
>> R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
>> R13: 0000000000000000 R14: 00007f3f071889c0 R15: 00007f3f07188700
>> Object at ffff880079c30140, in cache task_struct size: 5376
>> Allocated:
>> PID = 25681
>> kmem_cache_alloc_node+0x122/0x6f0 mm/slab.c:3662
>> alloc_task_struct_node kernel/fork.c:153 [inline]
>> dup_task_struct kernel/fork.c:495 [inline]
>> copy_process.part.38+0x19c8/0x4aa0 kernel/fork.c:1560
>> copy_process kernel/fork.c:1531 [inline]
>> _do_fork+0x200/0x1010 kernel/fork.c:1994
>> SYSC_clone kernel/fork.c:2104 [inline]
>> SyS_clone+0x37/0x50 kernel/fork.c:2098
>> do_syscall_64+0x2e8/0x930 arch/x86/entry/common.c:281
>> return_from_SYSCALL_64+0x0/0x7a
>> Freed:
>> PID = 25681
>> __cache_free mm/slab.c:3514 [inline]
>> kmem_cache_free+0x71/0x240 mm/slab.c:3774
>> free_task_struct kernel/fork.c:158 [inline]
>> free_task+0x151/0x1d0 kernel/fork.c:370
>> copy_process.part.38+0x18e5/0x4aa0 kernel/fork.c:1931
>> copy_process kernel/fork.c:1531 [inline]
>> _do_fork+0x200/0x1010 kernel/fork.c:1994
>> SYSC_clone kernel/fork.c:2104 [inline]
>> SyS_clone+0x37/0x50 kernel/fork.c:2098
>> do_syscall_64+0x2e8/0x930 arch/x86/entry/common.c:281
>> return_from_SYSCALL_64+0x0/0x7a
>>
>> Changes since v1:
>> - dropped "x86: remove unused atomic_inc_short()" patch
>> it is mailed separately
>> - rebased on top of tip/locking/core head
>> - other changes noted within individual patches
>>
>> Changes since v2:
>> - rebased on top of tip/locking/core head
>> - dropped a pervasive "x86: use long long for 64-bit atomic ops" commit,
>> instead use s64 type in wrappers
>> - added "x86: use s64* for old arg of atomic64_try_cmpxchg()" commit
>>
>> Changes since v3 are noted in individual commits.
>>
>> Changes since v4:
>> - rebased on tip/locking/core HEAD
>>
>> Changes since v5:
>> - rework cmpxchg* implementations so that we have less
>> code in macros and more code in functions
>
> Some context.
> This revives a half-year old patch. v5 of this was applied to
> tip/locking/core, but then reverted due to a reported crash:
> https://groups.google.com/d/msg/kasan-dev/ZJl66N7smmk/lJY99HmmAgAJ
>
> The root cause was in the cmpxchg macros:
>
> #define cmpxchg64(ptr, old, new) \
> ({ \
> __typeof__(ptr) ____ptr = (ptr); \
> kasan_check_write(____ptr, sizeof(*____ptr)); \
> arch_cmpxchg64(____ptr, (old), (new)); \
> })
>
> I had to introduce the new ____ptr variable, so that
> kasan_check_write() and arch_cmpxchg64() don't evaluate ptr twice. But
> there are multiple layers of macros and ____ptr ended up referring to
> something else (to itself).
>
> v6 changes cmpxchg macros to (as was suggested by Mark and Thomas):
>
> static __always_inline unsigned long
> cmpxchg_size(volatile void *ptr, unsigned long old, unsigned long new, int size)
> {
> kasan_check_write(ptr, size);
> switch (size) {
> case 1:
> return arch_cmpxchg((u8 *)ptr, (u8)old, (u8)new);
> case 2:
> return arch_cmpxchg((u16 *)ptr, (u16)old, (u16)new);
> case 4:
> return arch_cmpxchg((u32 *)ptr, (u32)old, (u32)new);
> case 8:
> BUILD_BUG_ON(sizeof(unsigned long) != 8);
> return arch_cmpxchg((u64 *)ptr, (u64)old, (u64)new);
> }
> BUILD_BUG();
> return 0;
> }
>
> #define cmpxchg(ptr, old, new) \
> ({ \
> ((__typeof__(*(ptr)))cmpxchg_size((ptr), (unsigned long)(old), \
> (unsigned long)(new), sizeof(*(ptr)))); \
> })
>
> Otherwise the patch series were rebased without any conflicts (surprisingly).
>
> There is now some duplication between cmpxchg_size, cmpxchg_local_size
> and sync_cmpxchg_size, but I thought that employing more macros while
> trying to resolve macro mess bugs is not the best idea.
Why I am reviving this patch now: besides the fact that it needs to be
done sooner or later anyway (also useful for KMSAN/KTSAN), now there
is another reason. Any silent heap corruptions produce very negative
effect on syzbot. If we miss a corruption on atomic access, syzbot can
report a dozen of induced crashes in random places. These look broken,
unexplainable, non-reproducible and cause complaints. It's better to
proactively prevent at this this known source of silent corruptions.
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v6 0/4] x86, kasan: add KASAN checks to atomic operations
2018-01-29 17:26 [PATCH v6 0/4] x86, kasan: add KASAN checks to atomic operations Dmitry Vyukov
` (4 preceding siblings ...)
2018-01-30 9:23 ` [PATCH v6 0/4] x86, kasan: add KASAN checks to atomic operations Dmitry Vyukov
@ 2018-01-30 15:36 ` Will Deacon
2018-01-31 7:28 ` Ingo Molnar
5 siblings, 1 reply; 38+ messages in thread
From: Will Deacon @ 2018-01-30 15:36 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: mark.rutland, peterz, mingo, hpa, aryabinin, kasan-dev, x86,
linux-kernel, tglx
Hi Dmitry,
On Mon, Jan 29, 2018 at 06:26:03PM +0100, Dmitry Vyukov wrote:
> KASAN uses compiler instrumentation to intercept all memory accesses.
> But it does not see memory accesses done in assembly code.
> One notable user of assembly code is atomic operations. Frequently,
> for example, an atomic reference decrement is the last access to an
> object and a good candidate for a racy use-after-free.
>
> Atomic operations are defined in arch files, but KASAN instrumentation
> is required for several archs that support KASAN. Later we will need
> similar hooks for KMSAN (uninit use detector) and KTSAN (data race
> detector).
>
> This change introduces wrappers around atomic operations that can be
> used to add KASAN/KMSAN/KTSAN instrumentation across several archs,
> and adds KASAN checks to them.
>
> This patch uses the wrappers only for x86 arch. Arm64 will be switched
> later. And we also plan to instrument bitops in a similar way.
One way you could reduce the intrusivness for each architecture would be
to leave the existing macro names as-is, and redefine them in the
asm-generic header. It's certainly ugly, but it makes the porting work
a lot smaller. Apologies if you've considered this approach before, but
I figured it was worth mentioning just in case.
e.g. for atomic[64]_read, your asm-generic header looks like:
#ifndef _LINUX_ATOMIC_INSTRUMENTED_H
#define _LINUX_ATOMIC_INSTRUMENTED_H
#include <linux/build_bug.h>
#include <linux/kasan-checks.h>
static __always_inline int __atomic_read_instrumented(const atomic_t *v)
{
kasan_check_read(v, sizeof(*v));
return atomic_read(v);
}
static __always_inline s64 __atomic64_read_instrumented(const atomic64_t *v)
{
kasan_check_read(v, sizeof(*v));
return atomic64_read(v);
}
#undef atomic_read
#undef atomic64_read
#define atomic_read __atomic_read_instrumented
#define atomic64_read __atomic64_read_instrumented
#endif /* _LINUX_ATOMIC_INSTRUMENTED_H */
and the arch code just includes that in asm/atomic.h once it's done with
its definitions.
What do you think? Too stinky?
Will
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v6 0/4] x86, kasan: add KASAN checks to atomic operations
2018-01-30 15:36 ` Will Deacon
@ 2018-01-31 7:28 ` Ingo Molnar
2018-01-31 8:53 ` Dmitry Vyukov
0 siblings, 1 reply; 38+ messages in thread
From: Ingo Molnar @ 2018-01-31 7:28 UTC (permalink / raw)
To: Will Deacon
Cc: Dmitry Vyukov, mark.rutland, peterz, mingo, hpa, aryabinin,
kasan-dev, x86, linux-kernel, tglx
* Will Deacon <will.deacon@arm.com> wrote:
> Hi Dmitry,
>
> On Mon, Jan 29, 2018 at 06:26:03PM +0100, Dmitry Vyukov wrote:
> > KASAN uses compiler instrumentation to intercept all memory accesses.
> > But it does not see memory accesses done in assembly code.
> > One notable user of assembly code is atomic operations. Frequently,
> > for example, an atomic reference decrement is the last access to an
> > object and a good candidate for a racy use-after-free.
> >
> > Atomic operations are defined in arch files, but KASAN instrumentation
> > is required for several archs that support KASAN. Later we will need
> > similar hooks for KMSAN (uninit use detector) and KTSAN (data race
> > detector).
> >
> > This change introduces wrappers around atomic operations that can be
> > used to add KASAN/KMSAN/KTSAN instrumentation across several archs,
> > and adds KASAN checks to them.
> >
> > This patch uses the wrappers only for x86 arch. Arm64 will be switched
> > later. And we also plan to instrument bitops in a similar way.
>
> One way you could reduce the intrusivness for each architecture would be
> to leave the existing macro names as-is, and redefine them in the
> asm-generic header. It's certainly ugly, but it makes the porting work
> a lot smaller. Apologies if you've considered this approach before, but
> I figured it was worth mentioning just in case.
>
> e.g. for atomic[64]_read, your asm-generic header looks like:
>
> #ifndef _LINUX_ATOMIC_INSTRUMENTED_H
> #define _LINUX_ATOMIC_INSTRUMENTED_H
>
> #include <linux/build_bug.h>
> #include <linux/kasan-checks.h>
>
> static __always_inline int __atomic_read_instrumented(const atomic_t *v)
> {
> kasan_check_read(v, sizeof(*v));
> return atomic_read(v);
> }
>
> static __always_inline s64 __atomic64_read_instrumented(const atomic64_t *v)
> {
> kasan_check_read(v, sizeof(*v));
> return atomic64_read(v);
> }
>
> #undef atomic_read
> #undef atomic64_read
>
> #define atomic_read __atomic_read_instrumented
> #define atomic64_read __atomic64_read_instrumented
>
> #endif /* _LINUX_ATOMIC_INSTRUMENTED_H */
>
> and the arch code just includes that in asm/atomic.h once it's done with
> its definitions.
>
> What do you think? Too stinky?
Hm, so while this could work - I actually *like* the low level changes: they are
straightforward, trivial, easy to read and they add the arch_ prefix that makes it
abundantly clear that this isn't the highest level interface.
The KASAN callbacks in the generic methods are also abundantly clear and very easy
to read. I could literally verify the sanity of the series while still being only
half awake. ;-)
Also note that the arch renaming should be 'trivial', in the sense that any
missing rename results in a clear build breakage. Plus any architecture making use
of this new KASAN feature should probably be tested before it's enabled - and the
renaming of the low level atomic APIs kind of forces that too.
So while this approach creates some churn, this series is IMHO a marked
improvement over the previous iterations.
Thanks,
Ingo
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v6 0/4] x86, kasan: add KASAN checks to atomic operations
2018-01-31 7:28 ` Ingo Molnar
@ 2018-01-31 8:53 ` Dmitry Vyukov
2018-01-31 16:17 ` Will Deacon
0 siblings, 1 reply; 38+ messages in thread
From: Dmitry Vyukov @ 2018-01-31 8:53 UTC (permalink / raw)
To: Ingo Molnar
Cc: Will Deacon, Mark Rutland, Peter Zijlstra, Ingo Molnar,
H. Peter Anvin, Andrey Ryabinin, kasan-dev,
the arch/x86 maintainers, LKML, Thomas Gleixner
On Wed, Jan 31, 2018 at 8:28 AM, Ingo Molnar <mingo@kernel.org> wrote:
>
> * Will Deacon <will.deacon@arm.com> wrote:
>
>> Hi Dmitry,
>>
>> On Mon, Jan 29, 2018 at 06:26:03PM +0100, Dmitry Vyukov wrote:
>> > KASAN uses compiler instrumentation to intercept all memory accesses.
>> > But it does not see memory accesses done in assembly code.
>> > One notable user of assembly code is atomic operations. Frequently,
>> > for example, an atomic reference decrement is the last access to an
>> > object and a good candidate for a racy use-after-free.
>> >
>> > Atomic operations are defined in arch files, but KASAN instrumentation
>> > is required for several archs that support KASAN. Later we will need
>> > similar hooks for KMSAN (uninit use detector) and KTSAN (data race
>> > detector).
>> >
>> > This change introduces wrappers around atomic operations that can be
>> > used to add KASAN/KMSAN/KTSAN instrumentation across several archs,
>> > and adds KASAN checks to them.
>> >
>> > This patch uses the wrappers only for x86 arch. Arm64 will be switched
>> > later. And we also plan to instrument bitops in a similar way.
>>
>> One way you could reduce the intrusivness for each architecture would be
>> to leave the existing macro names as-is, and redefine them in the
>> asm-generic header. It's certainly ugly, but it makes the porting work
>> a lot smaller. Apologies if you've considered this approach before, but
>> I figured it was worth mentioning just in case.
>>
>> e.g. for atomic[64]_read, your asm-generic header looks like:
>>
>> #ifndef _LINUX_ATOMIC_INSTRUMENTED_H
>> #define _LINUX_ATOMIC_INSTRUMENTED_H
>>
>> #include <linux/build_bug.h>
>> #include <linux/kasan-checks.h>
>>
>> static __always_inline int __atomic_read_instrumented(const atomic_t *v)
>> {
>> kasan_check_read(v, sizeof(*v));
>> return atomic_read(v);
>> }
>>
>> static __always_inline s64 __atomic64_read_instrumented(const atomic64_t *v)
>> {
>> kasan_check_read(v, sizeof(*v));
>> return atomic64_read(v);
>> }
>>
>> #undef atomic_read
>> #undef atomic64_read
>>
>> #define atomic_read __atomic_read_instrumented
>> #define atomic64_read __atomic64_read_instrumented
>>
>> #endif /* _LINUX_ATOMIC_INSTRUMENTED_H */
>>
>> and the arch code just includes that in asm/atomic.h once it's done with
>> its definitions.
>>
>> What do you think? Too stinky?
>
> Hm, so while this could work - I actually *like* the low level changes: they are
> straightforward, trivial, easy to read and they add the arch_ prefix that makes it
> abundantly clear that this isn't the highest level interface.
>
> The KASAN callbacks in the generic methods are also abundantly clear and very easy
> to read. I could literally verify the sanity of the series while still being only
> half awake. ;-)
>
> Also note that the arch renaming should be 'trivial', in the sense that any
> missing rename results in a clear build breakage. Plus any architecture making use
> of this new KASAN feature should probably be tested before it's enabled - and the
> renaming of the low level atomic APIs kind of forces that too.
>
> So while this approach creates some churn, this series is IMHO a marked
> improvement over the previous iterations.
I think I mildly leaning towards Ingo's point.
I guess people will first find the version in arch (because that's
where they used to be), but that version is actually not the one that
is called.
The renaming is mechanical and you get build errors if anything is
wrong. It's macros that caused hard to debug runtime crashes and
multiple revisions of this series.
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v6 0/4] x86, kasan: add KASAN checks to atomic operations
2018-01-31 8:53 ` Dmitry Vyukov
@ 2018-01-31 16:17 ` Will Deacon
2018-02-07 14:17 ` Dmitry Vyukov
0 siblings, 1 reply; 38+ messages in thread
From: Will Deacon @ 2018-01-31 16:17 UTC (permalink / raw)
To: Dmitry Vyukov
Cc: Ingo Molnar, Mark Rutland, Peter Zijlstra, Ingo Molnar,
H. Peter Anvin, Andrey Ryabinin, kasan-dev,
the arch/x86 maintainers, LKML, Thomas Gleixner
On Wed, Jan 31, 2018 at 09:53:10AM +0100, Dmitry Vyukov wrote:
> On Wed, Jan 31, 2018 at 8:28 AM, Ingo Molnar <mingo@kernel.org> wrote:
> > * Will Deacon <will.deacon@arm.com> wrote:
> >> e.g. for atomic[64]_read, your asm-generic header looks like:
> >>
> >> #ifndef _LINUX_ATOMIC_INSTRUMENTED_H
> >> #define _LINUX_ATOMIC_INSTRUMENTED_H
> >>
> >> #include <linux/build_bug.h>
> >> #include <linux/kasan-checks.h>
> >>
> >> static __always_inline int __atomic_read_instrumented(const atomic_t *v)
> >> {
> >> kasan_check_read(v, sizeof(*v));
> >> return atomic_read(v);
> >> }
> >>
> >> static __always_inline s64 __atomic64_read_instrumented(const atomic64_t *v)
> >> {
> >> kasan_check_read(v, sizeof(*v));
> >> return atomic64_read(v);
> >> }
> >>
> >> #undef atomic_read
> >> #undef atomic64_read
> >>
> >> #define atomic_read __atomic_read_instrumented
> >> #define atomic64_read __atomic64_read_instrumented
> >>
> >> #endif /* _LINUX_ATOMIC_INSTRUMENTED_H */
> >>
> >> and the arch code just includes that in asm/atomic.h once it's done with
> >> its definitions.
> >>
> >> What do you think? Too stinky?
> >
> > Hm, so while this could work - I actually *like* the low level changes: they are
> > straightforward, trivial, easy to read and they add the arch_ prefix that makes it
> > abundantly clear that this isn't the highest level interface.
> >
> > The KASAN callbacks in the generic methods are also abundantly clear and very easy
> > to read. I could literally verify the sanity of the series while still being only
> > half awake. ;-)
> >
> > Also note that the arch renaming should be 'trivial', in the sense that any
> > missing rename results in a clear build breakage. Plus any architecture making use
> > of this new KASAN feature should probably be tested before it's enabled - and the
> > renaming of the low level atomic APIs kind of forces that too.
> >
> > So while this approach creates some churn, this series is IMHO a marked
> > improvement over the previous iterations.
>
>
> I think I mildly leaning towards Ingo's point.
> I guess people will first find the version in arch (because that's
> where they used to be), but that version is actually not the one that
> is called.
> The renaming is mechanical and you get build errors if anything is
> wrong. It's macros that caused hard to debug runtime crashes and
> multiple revisions of this series.
Sure, and it sounds like you're proposing to do the arm64 changes anyway so
I'm not complaining! Just thought I'd float the alternative to see what
people think.
Will
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v6 0/4] x86, kasan: add KASAN checks to atomic operations
2018-01-31 16:17 ` Will Deacon
@ 2018-02-07 14:17 ` Dmitry Vyukov
2018-02-20 10:40 ` Dmitry Vyukov
0 siblings, 1 reply; 38+ messages in thread
From: Dmitry Vyukov @ 2018-02-07 14:17 UTC (permalink / raw)
To: Will Deacon
Cc: Ingo Molnar, Mark Rutland, Peter Zijlstra, Ingo Molnar,
H. Peter Anvin, Andrey Ryabinin, kasan-dev,
the arch/x86 maintainers, LKML, Thomas Gleixner
On Wed, Jan 31, 2018 at 5:17 PM, Will Deacon <will.deacon@arm.com> wrote:
>> > * Will Deacon <will.deacon@arm.com> wrote:
>> >> e.g. for atomic[64]_read, your asm-generic header looks like:
>> >>
>> >> #ifndef _LINUX_ATOMIC_INSTRUMENTED_H
>> >> #define _LINUX_ATOMIC_INSTRUMENTED_H
>> >>
>> >> #include <linux/build_bug.h>
>> >> #include <linux/kasan-checks.h>
>> >>
>> >> static __always_inline int __atomic_read_instrumented(const atomic_t *v)
>> >> {
>> >> kasan_check_read(v, sizeof(*v));
>> >> return atomic_read(v);
>> >> }
>> >>
>> >> static __always_inline s64 __atomic64_read_instrumented(const atomic64_t *v)
>> >> {
>> >> kasan_check_read(v, sizeof(*v));
>> >> return atomic64_read(v);
>> >> }
>> >>
>> >> #undef atomic_read
>> >> #undef atomic64_read
>> >>
>> >> #define atomic_read __atomic_read_instrumented
>> >> #define atomic64_read __atomic64_read_instrumented
>> >>
>> >> #endif /* _LINUX_ATOMIC_INSTRUMENTED_H */
>> >>
>> >> and the arch code just includes that in asm/atomic.h once it's done with
>> >> its definitions.
>> >>
>> >> What do you think? Too stinky?
>> >
>> > Hm, so while this could work - I actually *like* the low level changes: they are
>> > straightforward, trivial, easy to read and they add the arch_ prefix that makes it
>> > abundantly clear that this isn't the highest level interface.
>> >
>> > The KASAN callbacks in the generic methods are also abundantly clear and very easy
>> > to read. I could literally verify the sanity of the series while still being only
>> > half awake. ;-)
>> >
>> > Also note that the arch renaming should be 'trivial', in the sense that any
>> > missing rename results in a clear build breakage. Plus any architecture making use
>> > of this new KASAN feature should probably be tested before it's enabled - and the
>> > renaming of the low level atomic APIs kind of forces that too.
>> >
>> > So while this approach creates some churn, this series is IMHO a marked
>> > improvement over the previous iterations.
>>
>>
>> I think I mildly leaning towards Ingo's point.
>> I guess people will first find the version in arch (because that's
>> where they used to be), but that version is actually not the one that
>> is called.
>> The renaming is mechanical and you get build errors if anything is
>> wrong. It's macros that caused hard to debug runtime crashes and
>> multiple revisions of this series.
>
> Sure, and it sounds like you're proposing to do the arm64 changes anyway so
> I'm not complaining! Just thought I'd float the alternative to see what
> people think.
Any other comments?
Ingo, will you take this to locking tree?
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v6 0/4] x86, kasan: add KASAN checks to atomic operations
2018-02-07 14:17 ` Dmitry Vyukov
@ 2018-02-20 10:40 ` Dmitry Vyukov
2018-02-26 12:52 ` Dmitry Vyukov
0 siblings, 1 reply; 38+ messages in thread
From: Dmitry Vyukov @ 2018-02-20 10:40 UTC (permalink / raw)
To: Will Deacon
Cc: Ingo Molnar, Mark Rutland, Peter Zijlstra, Ingo Molnar,
H. Peter Anvin, Andrey Ryabinin, kasan-dev,
the arch/x86 maintainers, LKML, Thomas Gleixner
On Wed, Feb 7, 2018 at 3:17 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
> On Wed, Jan 31, 2018 at 5:17 PM, Will Deacon <will.deacon@arm.com> wrote:
>>> > * Will Deacon <will.deacon@arm.com> wrote:
>>> >> e.g. for atomic[64]_read, your asm-generic header looks like:
>>> >>
>>> >> #ifndef _LINUX_ATOMIC_INSTRUMENTED_H
>>> >> #define _LINUX_ATOMIC_INSTRUMENTED_H
>>> >>
>>> >> #include <linux/build_bug.h>
>>> >> #include <linux/kasan-checks.h>
>>> >>
>>> >> static __always_inline int __atomic_read_instrumented(const atomic_t *v)
>>> >> {
>>> >> kasan_check_read(v, sizeof(*v));
>>> >> return atomic_read(v);
>>> >> }
>>> >>
>>> >> static __always_inline s64 __atomic64_read_instrumented(const atomic64_t *v)
>>> >> {
>>> >> kasan_check_read(v, sizeof(*v));
>>> >> return atomic64_read(v);
>>> >> }
>>> >>
>>> >> #undef atomic_read
>>> >> #undef atomic64_read
>>> >>
>>> >> #define atomic_read __atomic_read_instrumented
>>> >> #define atomic64_read __atomic64_read_instrumented
>>> >>
>>> >> #endif /* _LINUX_ATOMIC_INSTRUMENTED_H */
>>> >>
>>> >> and the arch code just includes that in asm/atomic.h once it's done with
>>> >> its definitions.
>>> >>
>>> >> What do you think? Too stinky?
>>> >
>>> > Hm, so while this could work - I actually *like* the low level changes: they are
>>> > straightforward, trivial, easy to read and they add the arch_ prefix that makes it
>>> > abundantly clear that this isn't the highest level interface.
>>> >
>>> > The KASAN callbacks in the generic methods are also abundantly clear and very easy
>>> > to read. I could literally verify the sanity of the series while still being only
>>> > half awake. ;-)
>>> >
>>> > Also note that the arch renaming should be 'trivial', in the sense that any
>>> > missing rename results in a clear build breakage. Plus any architecture making use
>>> > of this new KASAN feature should probably be tested before it's enabled - and the
>>> > renaming of the low level atomic APIs kind of forces that too.
>>> >
>>> > So while this approach creates some churn, this series is IMHO a marked
>>> > improvement over the previous iterations.
>>>
>>>
>>> I think I mildly leaning towards Ingo's point.
>>> I guess people will first find the version in arch (because that's
>>> where they used to be), but that version is actually not the one that
>>> is called.
>>> The renaming is mechanical and you get build errors if anything is
>>> wrong. It's macros that caused hard to debug runtime crashes and
>>> multiple revisions of this series.
>>
>> Sure, and it sounds like you're proposing to do the arm64 changes anyway so
>> I'm not complaining! Just thought I'd float the alternative to see what
>> people think.
>
>
> Any other comments?
> Ingo, will you take this to locking tree?
Any other comments?
Ingo, will you take this to locking tree?
^ permalink raw reply [flat|nested] 38+ messages in thread
* Re: [PATCH v6 0/4] x86, kasan: add KASAN checks to atomic operations
2018-02-20 10:40 ` Dmitry Vyukov
@ 2018-02-26 12:52 ` Dmitry Vyukov
0 siblings, 0 replies; 38+ messages in thread
From: Dmitry Vyukov @ 2018-02-26 12:52 UTC (permalink / raw)
To: Peter Zijlstra, Ingo Molnar
Cc: Ingo Molnar, Mark Rutland, H. Peter Anvin, Andrey Ryabinin,
kasan-dev, the arch/x86 maintainers, LKML, Thomas Gleixner,
Will Deacon
On Tue, Feb 20, 2018 at 11:40 AM, Dmitry Vyukov <dvyukov@google.com> wrote:
> On Wed, Feb 7, 2018 at 3:17 PM, Dmitry Vyukov <dvyukov@google.com> wrote:
>> On Wed, Jan 31, 2018 at 5:17 PM, Will Deacon <will.deacon@arm.com> wrote:
>>>> > * Will Deacon <will.deacon@arm.com> wrote:
>>>> >> e.g. for atomic[64]_read, your asm-generic header looks like:
>>>> >>
>>>> >> #ifndef _LINUX_ATOMIC_INSTRUMENTED_H
>>>> >> #define _LINUX_ATOMIC_INSTRUMENTED_H
>>>> >>
>>>> >> #include <linux/build_bug.h>
>>>> >> #include <linux/kasan-checks.h>
>>>> >>
>>>> >> static __always_inline int __atomic_read_instrumented(const atomic_t *v)
>>>> >> {
>>>> >> kasan_check_read(v, sizeof(*v));
>>>> >> return atomic_read(v);
>>>> >> }
>>>> >>
>>>> >> static __always_inline s64 __atomic64_read_instrumented(const atomic64_t *v)
>>>> >> {
>>>> >> kasan_check_read(v, sizeof(*v));
>>>> >> return atomic64_read(v);
>>>> >> }
>>>> >>
>>>> >> #undef atomic_read
>>>> >> #undef atomic64_read
>>>> >>
>>>> >> #define atomic_read __atomic_read_instrumented
>>>> >> #define atomic64_read __atomic64_read_instrumented
>>>> >>
>>>> >> #endif /* _LINUX_ATOMIC_INSTRUMENTED_H */
>>>> >>
>>>> >> and the arch code just includes that in asm/atomic.h once it's done with
>>>> >> its definitions.
>>>> >>
>>>> >> What do you think? Too stinky?
>>>> >
>>>> > Hm, so while this could work - I actually *like* the low level changes: they are
>>>> > straightforward, trivial, easy to read and they add the arch_ prefix that makes it
>>>> > abundantly clear that this isn't the highest level interface.
>>>> >
>>>> > The KASAN callbacks in the generic methods are also abundantly clear and very easy
>>>> > to read. I could literally verify the sanity of the series while still being only
>>>> > half awake. ;-)
>>>> >
>>>> > Also note that the arch renaming should be 'trivial', in the sense that any
>>>> > missing rename results in a clear build breakage. Plus any architecture making use
>>>> > of this new KASAN feature should probably be tested before it's enabled - and the
>>>> > renaming of the low level atomic APIs kind of forces that too.
>>>> >
>>>> > So while this approach creates some churn, this series is IMHO a marked
>>>> > improvement over the previous iterations.
>>>>
>>>>
>>>> I think I mildly leaning towards Ingo's point.
>>>> I guess people will first find the version in arch (because that's
>>>> where they used to be), but that version is actually not the one that
>>>> is called.
>>>> The renaming is mechanical and you get build errors if anything is
>>>> wrong. It's macros that caused hard to debug runtime crashes and
>>>> multiple revisions of this series.
>>>
>>> Sure, and it sounds like you're proposing to do the arm64 changes anyway so
>>> I'm not complaining! Just thought I'd float the alternative to see what
>>> people think.
>>
>>
>> Any other comments?
>> Ingo, will you take this to locking tree?
>
>
> Any other comments?
> Ingo, will you take this to locking tree?
Peter, Ingo, ping.
^ permalink raw reply [flat|nested] 38+ messages in thread
* [tip:locking/core] locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h
2017-06-17 9:15 ` [PATCH v4 3/7] asm-generic: add atomic-instrumented.h Dmitry Vyukov
2017-06-19 10:50 ` Mark Rutland
2017-06-22 11:05 ` [tip:locking/core] locking/atomic: Add asm-generic/atomic-instrumented.h tip-bot for Dmitry Vyukov
@ 2018-03-12 12:23 ` tip-bot for Dmitry Vyukov
2 siblings, 0 replies; 38+ messages in thread
From: tip-bot for Dmitry Vyukov @ 2018-03-12 12:23 UTC (permalink / raw)
To: linux-tip-commits
Cc: tglx, peterz, mark.rutland, will.deacon, akpm, aryabinin, hpa,
mingo, dvyukov, linux-kernel, torvalds
Commit-ID: b06ed71a624ba088a3e3e3ac7d4185f48c7c1660
Gitweb: https://git.kernel.org/tip/b06ed71a624ba088a3e3e3ac7d4185f48c7c1660
Author: Dmitry Vyukov <dvyukov@google.com>
AuthorDate: Mon, 29 Jan 2018 18:26:04 +0100
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Mon, 12 Mar 2018 12:15:27 +0100
locking/atomic, asm-generic: Add asm-generic/atomic-instrumented.h
The new header allows to wrap per-arch atomic operations
and add common functionality to all of them.
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Acked-by: Mark Rutland <mark.rutland@arm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Will Deacon <will.deacon@arm.com>
Cc: kasan-dev@googlegroups.com
Cc: linux-mm@kvack.org
Link: http://lkml.kernel.org/r/31040b4e126bce801d2cc85a9c444b4332a88aa8.1517246437.git.dvyukov@google.com
Link: http://lkml.kernel.org/r/4ffbfa72c29134ac87b1f69da1506a5720590b5d.1497690003.git.dvyukov@google.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
include/asm-generic/atomic-instrumented.h | 393 ++++++++++++++++++++++++++++++
1 file changed, 393 insertions(+)
diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h
new file mode 100644
index 000000000000..b966194d120a
--- /dev/null
+++ b/include/asm-generic/atomic-instrumented.h
@@ -0,0 +1,393 @@
+#ifndef _LINUX_ATOMIC_INSTRUMENTED_H
+#define _LINUX_ATOMIC_INSTRUMENTED_H
+
+#include <linux/build_bug.h>
+
+static __always_inline int atomic_read(const atomic_t *v)
+{
+ return arch_atomic_read(v);
+}
+
+static __always_inline s64 atomic64_read(const atomic64_t *v)
+{
+ return arch_atomic64_read(v);
+}
+
+static __always_inline void atomic_set(atomic_t *v, int i)
+{
+ arch_atomic_set(v, i);
+}
+
+static __always_inline void atomic64_set(atomic64_t *v, s64 i)
+{
+ arch_atomic64_set(v, i);
+}
+
+static __always_inline int atomic_xchg(atomic_t *v, int i)
+{
+ return arch_atomic_xchg(v, i);
+}
+
+static __always_inline s64 atomic64_xchg(atomic64_t *v, s64 i)
+{
+ return arch_atomic64_xchg(v, i);
+}
+
+static __always_inline int atomic_cmpxchg(atomic_t *v, int old, int new)
+{
+ return arch_atomic_cmpxchg(v, old, new);
+}
+
+static __always_inline s64 atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new)
+{
+ return arch_atomic64_cmpxchg(v, old, new);
+}
+
+#ifdef arch_atomic_try_cmpxchg
+#define atomic_try_cmpxchg atomic_try_cmpxchg
+static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new)
+{
+ return arch_atomic_try_cmpxchg(v, old, new);
+}
+#endif
+
+#ifdef arch_atomic64_try_cmpxchg
+#define atomic64_try_cmpxchg atomic64_try_cmpxchg
+static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, s64 new)
+{
+ return arch_atomic64_try_cmpxchg(v, old, new);
+}
+#endif
+
+static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u)
+{
+ return __arch_atomic_add_unless(v, a, u);
+}
+
+
+static __always_inline bool atomic64_add_unless(atomic64_t *v, s64 a, s64 u)
+{
+ return arch_atomic64_add_unless(v, a, u);
+}
+
+static __always_inline void atomic_inc(atomic_t *v)
+{
+ arch_atomic_inc(v);
+}
+
+static __always_inline void atomic64_inc(atomic64_t *v)
+{
+ arch_atomic64_inc(v);
+}
+
+static __always_inline void atomic_dec(atomic_t *v)
+{
+ arch_atomic_dec(v);
+}
+
+static __always_inline void atomic64_dec(atomic64_t *v)
+{
+ arch_atomic64_dec(v);
+}
+
+static __always_inline void atomic_add(int i, atomic_t *v)
+{
+ arch_atomic_add(i, v);
+}
+
+static __always_inline void atomic64_add(s64 i, atomic64_t *v)
+{
+ arch_atomic64_add(i, v);
+}
+
+static __always_inline void atomic_sub(int i, atomic_t *v)
+{
+ arch_atomic_sub(i, v);
+}
+
+static __always_inline void atomic64_sub(s64 i, atomic64_t *v)
+{
+ arch_atomic64_sub(i, v);
+}
+
+static __always_inline void atomic_and(int i, atomic_t *v)
+{
+ arch_atomic_and(i, v);
+}
+
+static __always_inline void atomic64_and(s64 i, atomic64_t *v)
+{
+ arch_atomic64_and(i, v);
+}
+
+static __always_inline void atomic_or(int i, atomic_t *v)
+{
+ arch_atomic_or(i, v);
+}
+
+static __always_inline void atomic64_or(s64 i, atomic64_t *v)
+{
+ arch_atomic64_or(i, v);
+}
+
+static __always_inline void atomic_xor(int i, atomic_t *v)
+{
+ arch_atomic_xor(i, v);
+}
+
+static __always_inline void atomic64_xor(s64 i, atomic64_t *v)
+{
+ arch_atomic64_xor(i, v);
+}
+
+static __always_inline int atomic_inc_return(atomic_t *v)
+{
+ return arch_atomic_inc_return(v);
+}
+
+static __always_inline s64 atomic64_inc_return(atomic64_t *v)
+{
+ return arch_atomic64_inc_return(v);
+}
+
+static __always_inline int atomic_dec_return(atomic_t *v)
+{
+ return arch_atomic_dec_return(v);
+}
+
+static __always_inline s64 atomic64_dec_return(atomic64_t *v)
+{
+ return arch_atomic64_dec_return(v);
+}
+
+static __always_inline s64 atomic64_inc_not_zero(atomic64_t *v)
+{
+ return arch_atomic64_inc_not_zero(v);
+}
+
+static __always_inline s64 atomic64_dec_if_positive(atomic64_t *v)
+{
+ return arch_atomic64_dec_if_positive(v);
+}
+
+static __always_inline bool atomic_dec_and_test(atomic_t *v)
+{
+ return arch_atomic_dec_and_test(v);
+}
+
+static __always_inline bool atomic64_dec_and_test(atomic64_t *v)
+{
+ return arch_atomic64_dec_and_test(v);
+}
+
+static __always_inline bool atomic_inc_and_test(atomic_t *v)
+{
+ return arch_atomic_inc_and_test(v);
+}
+
+static __always_inline bool atomic64_inc_and_test(atomic64_t *v)
+{
+ return arch_atomic64_inc_and_test(v);
+}
+
+static __always_inline int atomic_add_return(int i, atomic_t *v)
+{
+ return arch_atomic_add_return(i, v);
+}
+
+static __always_inline s64 atomic64_add_return(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_add_return(i, v);
+}
+
+static __always_inline int atomic_sub_return(int i, atomic_t *v)
+{
+ return arch_atomic_sub_return(i, v);
+}
+
+static __always_inline s64 atomic64_sub_return(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_sub_return(i, v);
+}
+
+static __always_inline int atomic_fetch_add(int i, atomic_t *v)
+{
+ return arch_atomic_fetch_add(i, v);
+}
+
+static __always_inline s64 atomic64_fetch_add(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_fetch_add(i, v);
+}
+
+static __always_inline int atomic_fetch_sub(int i, atomic_t *v)
+{
+ return arch_atomic_fetch_sub(i, v);
+}
+
+static __always_inline s64 atomic64_fetch_sub(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_fetch_sub(i, v);
+}
+
+static __always_inline int atomic_fetch_and(int i, atomic_t *v)
+{
+ return arch_atomic_fetch_and(i, v);
+}
+
+static __always_inline s64 atomic64_fetch_and(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_fetch_and(i, v);
+}
+
+static __always_inline int atomic_fetch_or(int i, atomic_t *v)
+{
+ return arch_atomic_fetch_or(i, v);
+}
+
+static __always_inline s64 atomic64_fetch_or(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_fetch_or(i, v);
+}
+
+static __always_inline int atomic_fetch_xor(int i, atomic_t *v)
+{
+ return arch_atomic_fetch_xor(i, v);
+}
+
+static __always_inline s64 atomic64_fetch_xor(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_fetch_xor(i, v);
+}
+
+static __always_inline bool atomic_sub_and_test(int i, atomic_t *v)
+{
+ return arch_atomic_sub_and_test(i, v);
+}
+
+static __always_inline bool atomic64_sub_and_test(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_sub_and_test(i, v);
+}
+
+static __always_inline bool atomic_add_negative(int i, atomic_t *v)
+{
+ return arch_atomic_add_negative(i, v);
+}
+
+static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v)
+{
+ return arch_atomic64_add_negative(i, v);
+}
+
+static __always_inline unsigned long
+cmpxchg_size(volatile void *ptr, unsigned long old, unsigned long new, int size)
+{
+ switch (size) {
+ case 1:
+ return arch_cmpxchg((u8 *)ptr, (u8)old, (u8)new);
+ case 2:
+ return arch_cmpxchg((u16 *)ptr, (u16)old, (u16)new);
+ case 4:
+ return arch_cmpxchg((u32 *)ptr, (u32)old, (u32)new);
+ case 8:
+ BUILD_BUG_ON(sizeof(unsigned long) != 8);
+ return arch_cmpxchg((u64 *)ptr, (u64)old, (u64)new);
+ }
+ BUILD_BUG();
+ return 0;
+}
+
+#define cmpxchg(ptr, old, new) \
+({ \
+ ((__typeof__(*(ptr)))cmpxchg_size((ptr), (unsigned long)(old), \
+ (unsigned long)(new), sizeof(*(ptr)))); \
+})
+
+static __always_inline unsigned long
+sync_cmpxchg_size(volatile void *ptr, unsigned long old, unsigned long new,
+ int size)
+{
+ switch (size) {
+ case 1:
+ return arch_sync_cmpxchg((u8 *)ptr, (u8)old, (u8)new);
+ case 2:
+ return arch_sync_cmpxchg((u16 *)ptr, (u16)old, (u16)new);
+ case 4:
+ return arch_sync_cmpxchg((u32 *)ptr, (u32)old, (u32)new);
+ case 8:
+ BUILD_BUG_ON(sizeof(unsigned long) != 8);
+ return arch_sync_cmpxchg((u64 *)ptr, (u64)old, (u64)new);
+ }
+ BUILD_BUG();
+ return 0;
+}
+
+#define sync_cmpxchg(ptr, old, new) \
+({ \
+ ((__typeof__(*(ptr)))sync_cmpxchg_size((ptr), \
+ (unsigned long)(old), (unsigned long)(new), \
+ sizeof(*(ptr)))); \
+})
+
+static __always_inline unsigned long
+cmpxchg_local_size(volatile void *ptr, unsigned long old, unsigned long new,
+ int size)
+{
+ switch (size) {
+ case 1:
+ return arch_cmpxchg_local((u8 *)ptr, (u8)old, (u8)new);
+ case 2:
+ return arch_cmpxchg_local((u16 *)ptr, (u16)old, (u16)new);
+ case 4:
+ return arch_cmpxchg_local((u32 *)ptr, (u32)old, (u32)new);
+ case 8:
+ BUILD_BUG_ON(sizeof(unsigned long) != 8);
+ return arch_cmpxchg_local((u64 *)ptr, (u64)old, (u64)new);
+ }
+ BUILD_BUG();
+ return 0;
+}
+
+#define cmpxchg_local(ptr, old, new) \
+({ \
+ ((__typeof__(*(ptr)))cmpxchg_local_size((ptr), \
+ (unsigned long)(old), (unsigned long)(new), \
+ sizeof(*(ptr)))); \
+})
+
+static __always_inline u64
+cmpxchg64_size(volatile u64 *ptr, u64 old, u64 new)
+{
+ return arch_cmpxchg64(ptr, old, new);
+}
+
+#define cmpxchg64(ptr, old, new) \
+({ \
+ ((__typeof__(*(ptr)))cmpxchg64_size((ptr), (u64)(old), \
+ (u64)(new))); \
+})
+
+static __always_inline u64
+cmpxchg64_local_size(volatile u64 *ptr, u64 old, u64 new)
+{
+ return arch_cmpxchg64_local(ptr, old, new);
+}
+
+#define cmpxchg64_local(ptr, old, new) \
+({ \
+ ((__typeof__(*(ptr)))cmpxchg64_local_size((ptr), (u64)(old), \
+ (u64)(new))); \
+})
+
+#define cmpxchg_double(p1, p2, o1, o2, n1, n2) \
+({ \
+ arch_cmpxchg_double((p1), (p2), (o1), (o2), (n1), (n2)); \
+})
+
+#define cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \
+({ \
+ arch_cmpxchg_double_local((p1), (p2), (o1), (o2), (n1), (n2)); \
+})
+
+#endif /* _LINUX_ATOMIC_INSTRUMENTED_H */
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [tip:locking/core] locking/atomic/x86: Switch atomic.h to use atomic-instrumented.h
2018-01-29 17:26 ` [PATCH v6 2/4] x86: switch atomic.h to use atomic-instrumented.h Dmitry Vyukov
@ 2018-03-12 12:24 ` tip-bot for Dmitry Vyukov
0 siblings, 0 replies; 38+ messages in thread
From: tip-bot for Dmitry Vyukov @ 2018-03-12 12:24 UTC (permalink / raw)
To: linux-tip-commits
Cc: tglx, linux-kernel, will.deacon, mark.rutland, akpm, torvalds,
mingo, dvyukov, hpa, peterz, aryabinin
Commit-ID: 8bf705d130396e69c04cd8e6e010244ad2ce71f4
Gitweb: https://git.kernel.org/tip/8bf705d130396e69c04cd8e6e010244ad2ce71f4
Author: Dmitry Vyukov <dvyukov@google.com>
AuthorDate: Mon, 29 Jan 2018 18:26:05 +0100
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Mon, 12 Mar 2018 12:15:35 +0100
locking/atomic/x86: Switch atomic.h to use atomic-instrumented.h
Add arch_ prefix to all atomic operations and include
<asm-generic/atomic-instrumented.h>. This will allow
to add KASAN instrumentation to all atomic ops.
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Will Deacon <will.deacon@arm.com>
Cc: kasan-dev@googlegroups.com
Cc: linux-mm@kvack.org
Link: http://lkml.kernel.org/r/54f0eb64260b84199e538652e079a89b5423ad41.1517246437.git.dvyukov@google.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/x86/include/asm/atomic.h | 102 ++++++++++++++++++-----------------
arch/x86/include/asm/atomic64_32.h | 106 ++++++++++++++++++------------------
arch/x86/include/asm/atomic64_64.h | 108 ++++++++++++++++++-------------------
arch/x86/include/asm/cmpxchg.h | 12 ++---
arch/x86/include/asm/cmpxchg_32.h | 8 +--
arch/x86/include/asm/cmpxchg_64.h | 4 +-
6 files changed, 172 insertions(+), 168 deletions(-)
diff --git a/arch/x86/include/asm/atomic.h b/arch/x86/include/asm/atomic.h
index 72759f131cc5..33afc966d6a9 100644
--- a/arch/x86/include/asm/atomic.h
+++ b/arch/x86/include/asm/atomic.h
@@ -17,36 +17,36 @@
#define ATOMIC_INIT(i) { (i) }
/**
- * atomic_read - read atomic variable
+ * arch_atomic_read - read atomic variable
* @v: pointer of type atomic_t
*
* Atomically reads the value of @v.
*/
-static __always_inline int atomic_read(const atomic_t *v)
+static __always_inline int arch_atomic_read(const atomic_t *v)
{
return READ_ONCE((v)->counter);
}
/**
- * atomic_set - set atomic variable
+ * arch_atomic_set - set atomic variable
* @v: pointer of type atomic_t
* @i: required value
*
* Atomically sets the value of @v to @i.
*/
-static __always_inline void atomic_set(atomic_t *v, int i)
+static __always_inline void arch_atomic_set(atomic_t *v, int i)
{
WRITE_ONCE(v->counter, i);
}
/**
- * atomic_add - add integer to atomic variable
+ * arch_atomic_add - add integer to atomic variable
* @i: integer value to add
* @v: pointer of type atomic_t
*
* Atomically adds @i to @v.
*/
-static __always_inline void atomic_add(int i, atomic_t *v)
+static __always_inline void arch_atomic_add(int i, atomic_t *v)
{
asm volatile(LOCK_PREFIX "addl %1,%0"
: "+m" (v->counter)
@@ -54,13 +54,13 @@ static __always_inline void atomic_add(int i, atomic_t *v)
}
/**
- * atomic_sub - subtract integer from atomic variable
+ * arch_atomic_sub - subtract integer from atomic variable
* @i: integer value to subtract
* @v: pointer of type atomic_t
*
* Atomically subtracts @i from @v.
*/
-static __always_inline void atomic_sub(int i, atomic_t *v)
+static __always_inline void arch_atomic_sub(int i, atomic_t *v)
{
asm volatile(LOCK_PREFIX "subl %1,%0"
: "+m" (v->counter)
@@ -68,7 +68,7 @@ static __always_inline void atomic_sub(int i, atomic_t *v)
}
/**
- * atomic_sub_and_test - subtract value from variable and test result
+ * arch_atomic_sub_and_test - subtract value from variable and test result
* @i: integer value to subtract
* @v: pointer of type atomic_t
*
@@ -76,63 +76,63 @@ static __always_inline void atomic_sub(int i, atomic_t *v)
* true if the result is zero, or false for all
* other cases.
*/
-static __always_inline bool atomic_sub_and_test(int i, atomic_t *v)
+static __always_inline bool arch_atomic_sub_and_test(int i, atomic_t *v)
{
GEN_BINARY_RMWcc(LOCK_PREFIX "subl", v->counter, "er", i, "%0", e);
}
/**
- * atomic_inc - increment atomic variable
+ * arch_atomic_inc - increment atomic variable
* @v: pointer of type atomic_t
*
* Atomically increments @v by 1.
*/
-static __always_inline void atomic_inc(atomic_t *v)
+static __always_inline void arch_atomic_inc(atomic_t *v)
{
asm volatile(LOCK_PREFIX "incl %0"
: "+m" (v->counter));
}
/**
- * atomic_dec - decrement atomic variable
+ * arch_atomic_dec - decrement atomic variable
* @v: pointer of type atomic_t
*
* Atomically decrements @v by 1.
*/
-static __always_inline void atomic_dec(atomic_t *v)
+static __always_inline void arch_atomic_dec(atomic_t *v)
{
asm volatile(LOCK_PREFIX "decl %0"
: "+m" (v->counter));
}
/**
- * atomic_dec_and_test - decrement and test
+ * arch_atomic_dec_and_test - decrement and test
* @v: pointer of type atomic_t
*
* Atomically decrements @v by 1 and
* returns true if the result is 0, or false for all other
* cases.
*/
-static __always_inline bool atomic_dec_and_test(atomic_t *v)
+static __always_inline bool arch_atomic_dec_and_test(atomic_t *v)
{
GEN_UNARY_RMWcc(LOCK_PREFIX "decl", v->counter, "%0", e);
}
/**
- * atomic_inc_and_test - increment and test
+ * arch_atomic_inc_and_test - increment and test
* @v: pointer of type atomic_t
*
* Atomically increments @v by 1
* and returns true if the result is zero, or false for all
* other cases.
*/
-static __always_inline bool atomic_inc_and_test(atomic_t *v)
+static __always_inline bool arch_atomic_inc_and_test(atomic_t *v)
{
GEN_UNARY_RMWcc(LOCK_PREFIX "incl", v->counter, "%0", e);
}
/**
- * atomic_add_negative - add and test if negative
+ * arch_atomic_add_negative - add and test if negative
* @i: integer value to add
* @v: pointer of type atomic_t
*
@@ -140,65 +140,65 @@ static __always_inline bool atomic_inc_and_test(atomic_t *v)
* if the result is negative, or false when
* result is greater than or equal to zero.
*/
-static __always_inline bool atomic_add_negative(int i, atomic_t *v)
+static __always_inline bool arch_atomic_add_negative(int i, atomic_t *v)
{
GEN_BINARY_RMWcc(LOCK_PREFIX "addl", v->counter, "er", i, "%0", s);
}
/**
- * atomic_add_return - add integer and return
+ * arch_atomic_add_return - add integer and return
* @i: integer value to add
* @v: pointer of type atomic_t
*
* Atomically adds @i to @v and returns @i + @v
*/
-static __always_inline int atomic_add_return(int i, atomic_t *v)
+static __always_inline int arch_atomic_add_return(int i, atomic_t *v)
{
return i + xadd(&v->counter, i);
}
/**
- * atomic_sub_return - subtract integer and return
+ * arch_atomic_sub_return - subtract integer and return
* @v: pointer of type atomic_t
* @i: integer value to subtract
*
* Atomically subtracts @i from @v and returns @v - @i
*/
-static __always_inline int atomic_sub_return(int i, atomic_t *v)
+static __always_inline int arch_atomic_sub_return(int i, atomic_t *v)
{
- return atomic_add_return(-i, v);
+ return arch_atomic_add_return(-i, v);
}
-#define atomic_inc_return(v) (atomic_add_return(1, v))
-#define atomic_dec_return(v) (atomic_sub_return(1, v))
+#define arch_atomic_inc_return(v) (arch_atomic_add_return(1, v))
+#define arch_atomic_dec_return(v) (arch_atomic_sub_return(1, v))
-static __always_inline int atomic_fetch_add(int i, atomic_t *v)
+static __always_inline int arch_atomic_fetch_add(int i, atomic_t *v)
{
return xadd(&v->counter, i);
}
-static __always_inline int atomic_fetch_sub(int i, atomic_t *v)
+static __always_inline int arch_atomic_fetch_sub(int i, atomic_t *v)
{
return xadd(&v->counter, -i);
}
-static __always_inline int atomic_cmpxchg(atomic_t *v, int old, int new)
+static __always_inline int arch_atomic_cmpxchg(atomic_t *v, int old, int new)
{
- return cmpxchg(&v->counter, old, new);
+ return arch_cmpxchg(&v->counter, old, new);
}
-#define atomic_try_cmpxchg atomic_try_cmpxchg
-static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new)
+#define arch_atomic_try_cmpxchg arch_atomic_try_cmpxchg
+static __always_inline bool arch_atomic_try_cmpxchg(atomic_t *v, int *old, int new)
{
return try_cmpxchg(&v->counter, old, new);
}
-static inline int atomic_xchg(atomic_t *v, int new)
+static inline int arch_atomic_xchg(atomic_t *v, int new)
{
return xchg(&v->counter, new);
}
-static inline void atomic_and(int i, atomic_t *v)
+static inline void arch_atomic_and(int i, atomic_t *v)
{
asm volatile(LOCK_PREFIX "andl %1,%0"
: "+m" (v->counter)
@@ -206,16 +206,16 @@ static inline void atomic_and(int i, atomic_t *v)
: "memory");
}
-static inline int atomic_fetch_and(int i, atomic_t *v)
+static inline int arch_atomic_fetch_and(int i, atomic_t *v)
{
- int val = atomic_read(v);
+ int val = arch_atomic_read(v);
- do { } while (!atomic_try_cmpxchg(v, &val, val & i));
+ do { } while (!arch_atomic_try_cmpxchg(v, &val, val & i));
return val;
}
-static inline void atomic_or(int i, atomic_t *v)
+static inline void arch_atomic_or(int i, atomic_t *v)
{
asm volatile(LOCK_PREFIX "orl %1,%0"
: "+m" (v->counter)
@@ -223,16 +223,16 @@ static inline void atomic_or(int i, atomic_t *v)
: "memory");
}
-static inline int atomic_fetch_or(int i, atomic_t *v)
+static inline int arch_atomic_fetch_or(int i, atomic_t *v)
{
- int val = atomic_read(v);
+ int val = arch_atomic_read(v);
- do { } while (!atomic_try_cmpxchg(v, &val, val | i));
+ do { } while (!arch_atomic_try_cmpxchg(v, &val, val | i));
return val;
}
-static inline void atomic_xor(int i, atomic_t *v)
+static inline void arch_atomic_xor(int i, atomic_t *v)
{
asm volatile(LOCK_PREFIX "xorl %1,%0"
: "+m" (v->counter)
@@ -240,17 +240,17 @@ static inline void atomic_xor(int i, atomic_t *v)
: "memory");
}
-static inline int atomic_fetch_xor(int i, atomic_t *v)
+static inline int arch_atomic_fetch_xor(int i, atomic_t *v)
{
- int val = atomic_read(v);
+ int val = arch_atomic_read(v);
- do { } while (!atomic_try_cmpxchg(v, &val, val ^ i));
+ do { } while (!arch_atomic_try_cmpxchg(v, &val, val ^ i));
return val;
}
/**
- * __atomic_add_unless - add unless the number is already a given value
+ * __arch_atomic_add_unless - add unless the number is already a given value
* @v: pointer of type atomic_t
* @a: the amount to add to v...
* @u: ...unless v is equal to u.
@@ -258,14 +258,14 @@ static inline int atomic_fetch_xor(int i, atomic_t *v)
* Atomically adds @a to @v, so long as @v was not already @u.
* Returns the old value of @v.
*/
-static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u)
+static __always_inline int __arch_atomic_add_unless(atomic_t *v, int a, int u)
{
- int c = atomic_read(v);
+ int c = arch_atomic_read(v);
do {
if (unlikely(c == u))
break;
- } while (!atomic_try_cmpxchg(v, &c, c + a));
+ } while (!arch_atomic_try_cmpxchg(v, &c, c + a));
return c;
}
@@ -276,4 +276,6 @@ static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u)
# include <asm/atomic64_64.h>
#endif
+#include <asm-generic/atomic-instrumented.h>
+
#endif /* _ASM_X86_ATOMIC_H */
diff --git a/arch/x86/include/asm/atomic64_32.h b/arch/x86/include/asm/atomic64_32.h
index 97c46b8169b7..46e1ef17d92d 100644
--- a/arch/x86/include/asm/atomic64_32.h
+++ b/arch/x86/include/asm/atomic64_32.h
@@ -62,7 +62,7 @@ ATOMIC64_DECL(add_unless);
#undef ATOMIC64_EXPORT
/**
- * atomic64_cmpxchg - cmpxchg atomic64 variable
+ * arch_atomic64_cmpxchg - cmpxchg atomic64 variable
* @v: pointer to type atomic64_t
* @o: expected value
* @n: new value
@@ -71,20 +71,21 @@ ATOMIC64_DECL(add_unless);
* the old value.
*/
-static inline long long atomic64_cmpxchg(atomic64_t *v, long long o, long long n)
+static inline long long arch_atomic64_cmpxchg(atomic64_t *v, long long o,
+ long long n)
{
- return cmpxchg64(&v->counter, o, n);
+ return arch_cmpxchg64(&v->counter, o, n);
}
/**
- * atomic64_xchg - xchg atomic64 variable
+ * arch_atomic64_xchg - xchg atomic64 variable
* @v: pointer to type atomic64_t
* @n: value to assign
*
* Atomically xchgs the value of @v to @n and returns
* the old value.
*/
-static inline long long atomic64_xchg(atomic64_t *v, long long n)
+static inline long long arch_atomic64_xchg(atomic64_t *v, long long n)
{
long long o;
unsigned high = (unsigned)(n >> 32);
@@ -96,13 +97,13 @@ static inline long long atomic64_xchg(atomic64_t *v, long long n)
}
/**
- * atomic64_set - set atomic64 variable
+ * arch_atomic64_set - set atomic64 variable
* @v: pointer to type atomic64_t
* @i: value to assign
*
* Atomically sets the value of @v to @n.
*/
-static inline void atomic64_set(atomic64_t *v, long long i)
+static inline void arch_atomic64_set(atomic64_t *v, long long i)
{
unsigned high = (unsigned)(i >> 32);
unsigned low = (unsigned)i;
@@ -112,12 +113,12 @@ static inline void atomic64_set(atomic64_t *v, long long i)
}
/**
- * atomic64_read - read atomic64 variable
+ * arch_atomic64_read - read atomic64 variable
* @v: pointer to type atomic64_t
*
* Atomically reads the value of @v and returns it.
*/
-static inline long long atomic64_read(const atomic64_t *v)
+static inline long long arch_atomic64_read(const atomic64_t *v)
{
long long r;
alternative_atomic64(read, "=&A" (r), "c" (v) : "memory");
@@ -125,13 +126,13 @@ static inline long long atomic64_read(const atomic64_t *v)
}
/**
- * atomic64_add_return - add and return
+ * arch_atomic64_add_return - add and return
* @i: integer value to add
* @v: pointer to type atomic64_t
*
* Atomically adds @i to @v and returns @i + *@v
*/
-static inline long long atomic64_add_return(long long i, atomic64_t *v)
+static inline long long arch_atomic64_add_return(long long i, atomic64_t *v)
{
alternative_atomic64(add_return,
ASM_OUTPUT2("+A" (i), "+c" (v)),
@@ -142,7 +143,7 @@ static inline long long atomic64_add_return(long long i, atomic64_t *v)
/*
* Other variants with different arithmetic operators:
*/
-static inline long long atomic64_sub_return(long long i, atomic64_t *v)
+static inline long long arch_atomic64_sub_return(long long i, atomic64_t *v)
{
alternative_atomic64(sub_return,
ASM_OUTPUT2("+A" (i), "+c" (v)),
@@ -150,7 +151,7 @@ static inline long long atomic64_sub_return(long long i, atomic64_t *v)
return i;
}
-static inline long long atomic64_inc_return(atomic64_t *v)
+static inline long long arch_atomic64_inc_return(atomic64_t *v)
{
long long a;
alternative_atomic64(inc_return, "=&A" (a),
@@ -158,7 +159,7 @@ static inline long long atomic64_inc_return(atomic64_t *v)
return a;
}
-static inline long long atomic64_dec_return(atomic64_t *v)
+static inline long long arch_atomic64_dec_return(atomic64_t *v)
{
long long a;
alternative_atomic64(dec_return, "=&A" (a),
@@ -167,13 +168,13 @@ static inline long long atomic64_dec_return(atomic64_t *v)
}
/**
- * atomic64_add - add integer to atomic64 variable
+ * arch_atomic64_add - add integer to atomic64 variable
* @i: integer value to add
* @v: pointer to type atomic64_t
*
* Atomically adds @i to @v.
*/
-static inline long long atomic64_add(long long i, atomic64_t *v)
+static inline long long arch_atomic64_add(long long i, atomic64_t *v)
{
__alternative_atomic64(add, add_return,
ASM_OUTPUT2("+A" (i), "+c" (v)),
@@ -182,13 +183,13 @@ static inline long long atomic64_add(long long i, atomic64_t *v)
}
/**
- * atomic64_sub - subtract the atomic64 variable
+ * arch_atomic64_sub - subtract the atomic64 variable
* @i: integer value to subtract
* @v: pointer to type atomic64_t
*
* Atomically subtracts @i from @v.
*/
-static inline long long atomic64_sub(long long i, atomic64_t *v)
+static inline long long arch_atomic64_sub(long long i, atomic64_t *v)
{
__alternative_atomic64(sub, sub_return,
ASM_OUTPUT2("+A" (i), "+c" (v)),
@@ -197,7 +198,7 @@ static inline long long atomic64_sub(long long i, atomic64_t *v)
}
/**
- * atomic64_sub_and_test - subtract value from variable and test result
+ * arch_atomic64_sub_and_test - subtract value from variable and test result
* @i: integer value to subtract
* @v: pointer to type atomic64_t
*
@@ -205,46 +206,46 @@ static inline long long atomic64_sub(long long i, atomic64_t *v)
* true if the result is zero, or false for all
* other cases.
*/
-static inline int atomic64_sub_and_test(long long i, atomic64_t *v)
+static inline int arch_atomic64_sub_and_test(long long i, atomic64_t *v)
{
- return atomic64_sub_return(i, v) == 0;
+ return arch_atomic64_sub_return(i, v) == 0;
}
/**
- * atomic64_inc - increment atomic64 variable
+ * arch_atomic64_inc - increment atomic64 variable
* @v: pointer to type atomic64_t
*
* Atomically increments @v by 1.
*/
-static inline void atomic64_inc(atomic64_t *v)
+static inline void arch_atomic64_inc(atomic64_t *v)
{
__alternative_atomic64(inc, inc_return, /* no output */,
"S" (v) : "memory", "eax", "ecx", "edx");
}
/**
- * atomic64_dec - decrement atomic64 variable
+ * arch_atomic64_dec - decrement atomic64 variable
* @v: pointer to type atomic64_t
*
* Atomically decrements @v by 1.
*/
-static inline void atomic64_dec(atomic64_t *v)
+static inline void arch_atomic64_dec(atomic64_t *v)
{
__alternative_atomic64(dec, dec_return, /* no output */,
"S" (v) : "memory", "eax", "ecx", "edx");
}
/**
- * atomic64_dec_and_test - decrement and test
+ * arch_atomic64_dec_and_test - decrement and test
* @v: pointer to type atomic64_t
*
* Atomically decrements @v by 1 and
* returns true if the result is 0, or false for all other
* cases.
*/
-static inline int atomic64_dec_and_test(atomic64_t *v)
+static inline int arch_atomic64_dec_and_test(atomic64_t *v)
{
- return atomic64_dec_return(v) == 0;
+ return arch_atomic64_dec_return(v) == 0;
}
/**
@@ -255,13 +256,13 @@ static inline int atomic64_dec_and_test(atomic64_t *v)
* and returns true if the result is zero, or false for all
* other cases.
*/
-static inline int atomic64_inc_and_test(atomic64_t *v)
+static inline int arch_atomic64_inc_and_test(atomic64_t *v)
{
- return atomic64_inc_return(v) == 0;
+ return arch_atomic64_inc_return(v) == 0;
}
/**
- * atomic64_add_negative - add and test if negative
+ * arch_atomic64_add_negative - add and test if negative
* @i: integer value to add
* @v: pointer to type atomic64_t
*
@@ -269,13 +270,13 @@ static inline int atomic64_inc_and_test(atomic64_t *v)
* if the result is negative, or false when
* result is greater than or equal to zero.
*/
-static inline int atomic64_add_negative(long long i, atomic64_t *v)
+static inline int arch_atomic64_add_negative(long long i, atomic64_t *v)
{
- return atomic64_add_return(i, v) < 0;
+ return arch_atomic64_add_return(i, v) < 0;
}
/**
- * atomic64_add_unless - add unless the number is a given value
+ * arch_atomic64_add_unless - add unless the number is a given value
* @v: pointer of type atomic64_t
* @a: the amount to add to v...
* @u: ...unless v is equal to u.
@@ -283,7 +284,8 @@ static inline int atomic64_add_negative(long long i, atomic64_t *v)
* Atomically adds @a to @v, so long as it was not @u.
* Returns non-zero if the add was done, zero otherwise.
*/
-static inline int atomic64_add_unless(atomic64_t *v, long long a, long long u)
+static inline int arch_atomic64_add_unless(atomic64_t *v, long long a,
+ long long u)
{
unsigned low = (unsigned)u;
unsigned high = (unsigned)(u >> 32);
@@ -294,7 +296,7 @@ static inline int atomic64_add_unless(atomic64_t *v, long long a, long long u)
}
-static inline int atomic64_inc_not_zero(atomic64_t *v)
+static inline int arch_atomic64_inc_not_zero(atomic64_t *v)
{
int r;
alternative_atomic64(inc_not_zero, "=&a" (r),
@@ -302,7 +304,7 @@ static inline int atomic64_inc_not_zero(atomic64_t *v)
return r;
}
-static inline long long atomic64_dec_if_positive(atomic64_t *v)
+static inline long long arch_atomic64_dec_if_positive(atomic64_t *v)
{
long long r;
alternative_atomic64(dec_if_positive, "=&A" (r),
@@ -313,70 +315,70 @@ static inline long long atomic64_dec_if_positive(atomic64_t *v)
#undef alternative_atomic64
#undef __alternative_atomic64
-static inline void atomic64_and(long long i, atomic64_t *v)
+static inline void arch_atomic64_and(long long i, atomic64_t *v)
{
long long old, c = 0;
- while ((old = atomic64_cmpxchg(v, c, c & i)) != c)
+ while ((old = arch_atomic64_cmpxchg(v, c, c & i)) != c)
c = old;
}
-static inline long long atomic64_fetch_and(long long i, atomic64_t *v)
+static inline long long arch_atomic64_fetch_and(long long i, atomic64_t *v)
{
long long old, c = 0;
- while ((old = atomic64_cmpxchg(v, c, c & i)) != c)
+ while ((old = arch_atomic64_cmpxchg(v, c, c & i)) != c)
c = old;
return old;
}
-static inline void atomic64_or(long long i, atomic64_t *v)
+static inline void arch_atomic64_or(long long i, atomic64_t *v)
{
long long old, c = 0;
- while ((old = atomic64_cmpxchg(v, c, c | i)) != c)
+ while ((old = arch_atomic64_cmpxchg(v, c, c | i)) != c)
c = old;
}
-static inline long long atomic64_fetch_or(long long i, atomic64_t *v)
+static inline long long arch_atomic64_fetch_or(long long i, atomic64_t *v)
{
long long old, c = 0;
- while ((old = atomic64_cmpxchg(v, c, c | i)) != c)
+ while ((old = arch_atomic64_cmpxchg(v, c, c | i)) != c)
c = old;
return old;
}
-static inline void atomic64_xor(long long i, atomic64_t *v)
+static inline void arch_atomic64_xor(long long i, atomic64_t *v)
{
long long old, c = 0;
- while ((old = atomic64_cmpxchg(v, c, c ^ i)) != c)
+ while ((old = arch_atomic64_cmpxchg(v, c, c ^ i)) != c)
c = old;
}
-static inline long long atomic64_fetch_xor(long long i, atomic64_t *v)
+static inline long long arch_atomic64_fetch_xor(long long i, atomic64_t *v)
{
long long old, c = 0;
- while ((old = atomic64_cmpxchg(v, c, c ^ i)) != c)
+ while ((old = arch_atomic64_cmpxchg(v, c, c ^ i)) != c)
c = old;
return old;
}
-static inline long long atomic64_fetch_add(long long i, atomic64_t *v)
+static inline long long arch_atomic64_fetch_add(long long i, atomic64_t *v)
{
long long old, c = 0;
- while ((old = atomic64_cmpxchg(v, c, c + i)) != c)
+ while ((old = arch_atomic64_cmpxchg(v, c, c + i)) != c)
c = old;
return old;
}
-#define atomic64_fetch_sub(i, v) atomic64_fetch_add(-(i), (v))
+#define arch_atomic64_fetch_sub(i, v) arch_atomic64_fetch_add(-(i), (v))
#endif /* _ASM_X86_ATOMIC64_32_H */
diff --git a/arch/x86/include/asm/atomic64_64.h b/arch/x86/include/asm/atomic64_64.h
index 738495caf05f..6106b59d3260 100644
--- a/arch/x86/include/asm/atomic64_64.h
+++ b/arch/x86/include/asm/atomic64_64.h
@@ -11,37 +11,37 @@
#define ATOMIC64_INIT(i) { (i) }
/**
- * atomic64_read - read atomic64 variable
+ * arch_atomic64_read - read atomic64 variable
* @v: pointer of type atomic64_t
*
* Atomically reads the value of @v.
* Doesn't imply a read memory barrier.
*/
-static inline long atomic64_read(const atomic64_t *v)
+static inline long arch_atomic64_read(const atomic64_t *v)
{
return READ_ONCE((v)->counter);
}
/**
- * atomic64_set - set atomic64 variable
+ * arch_atomic64_set - set atomic64 variable
* @v: pointer to type atomic64_t
* @i: required value
*
* Atomically sets the value of @v to @i.
*/
-static inline void atomic64_set(atomic64_t *v, long i)
+static inline void arch_atomic64_set(atomic64_t *v, long i)
{
WRITE_ONCE(v->counter, i);
}
/**
- * atomic64_add - add integer to atomic64 variable
+ * arch_atomic64_add - add integer to atomic64 variable
* @i: integer value to add
* @v: pointer to type atomic64_t
*
* Atomically adds @i to @v.
*/
-static __always_inline void atomic64_add(long i, atomic64_t *v)
+static __always_inline void arch_atomic64_add(long i, atomic64_t *v)
{
asm volatile(LOCK_PREFIX "addq %1,%0"
: "=m" (v->counter)
@@ -49,13 +49,13 @@ static __always_inline void atomic64_add(long i, atomic64_t *v)
}
/**
- * atomic64_sub - subtract the atomic64 variable
+ * arch_atomic64_sub - subtract the atomic64 variable
* @i: integer value to subtract
* @v: pointer to type atomic64_t
*
* Atomically subtracts @i from @v.
*/
-static inline void atomic64_sub(long i, atomic64_t *v)
+static inline void arch_atomic64_sub(long i, atomic64_t *v)
{
asm volatile(LOCK_PREFIX "subq %1,%0"
: "=m" (v->counter)
@@ -63,7 +63,7 @@ static inline void atomic64_sub(long i, atomic64_t *v)
}
/**
- * atomic64_sub_and_test - subtract value from variable and test result
+ * arch_atomic64_sub_and_test - subtract value from variable and test result
* @i: integer value to subtract
* @v: pointer to type atomic64_t
*
@@ -71,18 +71,18 @@ static inline void atomic64_sub(long i, atomic64_t *v)
* true if the result is zero, or false for all
* other cases.
*/
-static inline bool atomic64_sub_and_test(long i, atomic64_t *v)
+static inline bool arch_atomic64_sub_and_test(long i, atomic64_t *v)
{
GEN_BINARY_RMWcc(LOCK_PREFIX "subq", v->counter, "er", i, "%0", e);
}
/**
- * atomic64_inc - increment atomic64 variable
+ * arch_atomic64_inc - increment atomic64 variable
* @v: pointer to type atomic64_t
*
* Atomically increments @v by 1.
*/
-static __always_inline void atomic64_inc(atomic64_t *v)
+static __always_inline void arch_atomic64_inc(atomic64_t *v)
{
asm volatile(LOCK_PREFIX "incq %0"
: "=m" (v->counter)
@@ -90,12 +90,12 @@ static __always_inline void atomic64_inc(atomic64_t *v)
}
/**
- * atomic64_dec - decrement atomic64 variable
+ * arch_atomic64_dec - decrement atomic64 variable
* @v: pointer to type atomic64_t
*
* Atomically decrements @v by 1.
*/
-static __always_inline void atomic64_dec(atomic64_t *v)
+static __always_inline void arch_atomic64_dec(atomic64_t *v)
{
asm volatile(LOCK_PREFIX "decq %0"
: "=m" (v->counter)
@@ -103,33 +103,33 @@ static __always_inline void atomic64_dec(atomic64_t *v)
}
/**
- * atomic64_dec_and_test - decrement and test
+ * arch_atomic64_dec_and_test - decrement and test
* @v: pointer to type atomic64_t
*
* Atomically decrements @v by 1 and
* returns true if the result is 0, or false for all other
* cases.
*/
-static inline bool atomic64_dec_and_test(atomic64_t *v)
+static inline bool arch_atomic64_dec_and_test(atomic64_t *v)
{
GEN_UNARY_RMWcc(LOCK_PREFIX "decq", v->counter, "%0", e);
}
/**
- * atomic64_inc_and_test - increment and test
+ * arch_atomic64_inc_and_test - increment and test
* @v: pointer to type atomic64_t
*
* Atomically increments @v by 1
* and returns true if the result is zero, or false for all
* other cases.
*/
-static inline bool atomic64_inc_and_test(atomic64_t *v)
+static inline bool arch_atomic64_inc_and_test(atomic64_t *v)
{
GEN_UNARY_RMWcc(LOCK_PREFIX "incq", v->counter, "%0", e);
}
/**
- * atomic64_add_negative - add and test if negative
+ * arch_atomic64_add_negative - add and test if negative
* @i: integer value to add
* @v: pointer to type atomic64_t
*
@@ -137,59 +137,59 @@ static inline bool atomic64_inc_and_test(atomic64_t *v)
* if the result is negative, or false when
* result is greater than or equal to zero.
*/
-static inline bool atomic64_add_negative(long i, atomic64_t *v)
+static inline bool arch_atomic64_add_negative(long i, atomic64_t *v)
{
GEN_BINARY_RMWcc(LOCK_PREFIX "addq", v->counter, "er", i, "%0", s);
}
/**
- * atomic64_add_return - add and return
+ * arch_atomic64_add_return - add and return
* @i: integer value to add
* @v: pointer to type atomic64_t
*
* Atomically adds @i to @v and returns @i + @v
*/
-static __always_inline long atomic64_add_return(long i, atomic64_t *v)
+static __always_inline long arch_atomic64_add_return(long i, atomic64_t *v)
{
return i + xadd(&v->counter, i);
}
-static inline long atomic64_sub_return(long i, atomic64_t *v)
+static inline long arch_atomic64_sub_return(long i, atomic64_t *v)
{
- return atomic64_add_return(-i, v);
+ return arch_atomic64_add_return(-i, v);
}
-static inline long atomic64_fetch_add(long i, atomic64_t *v)
+static inline long arch_atomic64_fetch_add(long i, atomic64_t *v)
{
return xadd(&v->counter, i);
}
-static inline long atomic64_fetch_sub(long i, atomic64_t *v)
+static inline long arch_atomic64_fetch_sub(long i, atomic64_t *v)
{
return xadd(&v->counter, -i);
}
-#define atomic64_inc_return(v) (atomic64_add_return(1, (v)))
-#define atomic64_dec_return(v) (atomic64_sub_return(1, (v)))
+#define arch_atomic64_inc_return(v) (arch_atomic64_add_return(1, (v)))
+#define arch_atomic64_dec_return(v) (arch_atomic64_sub_return(1, (v)))
-static inline long atomic64_cmpxchg(atomic64_t *v, long old, long new)
+static inline long arch_atomic64_cmpxchg(atomic64_t *v, long old, long new)
{
- return cmpxchg(&v->counter, old, new);
+ return arch_cmpxchg(&v->counter, old, new);
}
-#define atomic64_try_cmpxchg atomic64_try_cmpxchg
-static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, long new)
+#define arch_atomic64_try_cmpxchg arch_atomic64_try_cmpxchg
+static __always_inline bool arch_atomic64_try_cmpxchg(atomic64_t *v, s64 *old, long new)
{
return try_cmpxchg(&v->counter, old, new);
}
-static inline long atomic64_xchg(atomic64_t *v, long new)
+static inline long arch_atomic64_xchg(atomic64_t *v, long new)
{
return xchg(&v->counter, new);
}
/**
- * atomic64_add_unless - add unless the number is a given value
+ * arch_atomic64_add_unless - add unless the number is a given value
* @v: pointer of type atomic64_t
* @a: the amount to add to v...
* @u: ...unless v is equal to u.
@@ -197,37 +197,37 @@ static inline long atomic64_xchg(atomic64_t *v, long new)
* Atomically adds @a to @v, so long as it was not @u.
* Returns the old value of @v.
*/
-static inline bool atomic64_add_unless(atomic64_t *v, long a, long u)
+static inline bool arch_atomic64_add_unless(atomic64_t *v, long a, long u)
{
- s64 c = atomic64_read(v);
+ s64 c = arch_atomic64_read(v);
do {
if (unlikely(c == u))
return false;
- } while (!atomic64_try_cmpxchg(v, &c, c + a));
+ } while (!arch_atomic64_try_cmpxchg(v, &c, c + a));
return true;
}
-#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0)
+#define arch_atomic64_inc_not_zero(v) arch_atomic64_add_unless((v), 1, 0)
/*
- * atomic64_dec_if_positive - decrement by 1 if old value positive
+ * arch_atomic64_dec_if_positive - decrement by 1 if old value positive
* @v: pointer of type atomic_t
*
* The function returns the old value of *v minus 1, even if
* the atomic variable, v, was not decremented.
*/
-static inline long atomic64_dec_if_positive(atomic64_t *v)
+static inline long arch_atomic64_dec_if_positive(atomic64_t *v)
{
- s64 dec, c = atomic64_read(v);
+ s64 dec, c = arch_atomic64_read(v);
do {
dec = c - 1;
if (unlikely(dec < 0))
break;
- } while (!atomic64_try_cmpxchg(v, &c, dec));
+ } while (!arch_atomic64_try_cmpxchg(v, &c, dec));
return dec;
}
-static inline void atomic64_and(long i, atomic64_t *v)
+static inline void arch_atomic64_and(long i, atomic64_t *v)
{
asm volatile(LOCK_PREFIX "andq %1,%0"
: "+m" (v->counter)
@@ -235,16 +235,16 @@ static inline void atomic64_and(long i, atomic64_t *v)
: "memory");
}
-static inline long atomic64_fetch_and(long i, atomic64_t *v)
+static inline long arch_atomic64_fetch_and(long i, atomic64_t *v)
{
- s64 val = atomic64_read(v);
+ s64 val = arch_atomic64_read(v);
do {
- } while (!atomic64_try_cmpxchg(v, &val, val & i));
+ } while (!arch_atomic64_try_cmpxchg(v, &val, val & i));
return val;
}
-static inline void atomic64_or(long i, atomic64_t *v)
+static inline void arch_atomic64_or(long i, atomic64_t *v)
{
asm volatile(LOCK_PREFIX "orq %1,%0"
: "+m" (v->counter)
@@ -252,16 +252,16 @@ static inline void atomic64_or(long i, atomic64_t *v)
: "memory");
}
-static inline long atomic64_fetch_or(long i, atomic64_t *v)
+static inline long arch_atomic64_fetch_or(long i, atomic64_t *v)
{
- s64 val = atomic64_read(v);
+ s64 val = arch_atomic64_read(v);
do {
- } while (!atomic64_try_cmpxchg(v, &val, val | i));
+ } while (!arch_atomic64_try_cmpxchg(v, &val, val | i));
return val;
}
-static inline void atomic64_xor(long i, atomic64_t *v)
+static inline void arch_atomic64_xor(long i, atomic64_t *v)
{
asm volatile(LOCK_PREFIX "xorq %1,%0"
: "+m" (v->counter)
@@ -269,12 +269,12 @@ static inline void atomic64_xor(long i, atomic64_t *v)
: "memory");
}
-static inline long atomic64_fetch_xor(long i, atomic64_t *v)
+static inline long arch_atomic64_fetch_xor(long i, atomic64_t *v)
{
- s64 val = atomic64_read(v);
+ s64 val = arch_atomic64_read(v);
do {
- } while (!atomic64_try_cmpxchg(v, &val, val ^ i));
+ } while (!arch_atomic64_try_cmpxchg(v, &val, val ^ i));
return val;
}
diff --git a/arch/x86/include/asm/cmpxchg.h b/arch/x86/include/asm/cmpxchg.h
index 56bd436ed01b..e3efd8a06066 100644
--- a/arch/x86/include/asm/cmpxchg.h
+++ b/arch/x86/include/asm/cmpxchg.h
@@ -145,13 +145,13 @@ extern void __add_wrong_size(void)
# include <asm/cmpxchg_64.h>
#endif
-#define cmpxchg(ptr, old, new) \
+#define arch_cmpxchg(ptr, old, new) \
__cmpxchg(ptr, old, new, sizeof(*(ptr)))
-#define sync_cmpxchg(ptr, old, new) \
+#define arch_sync_cmpxchg(ptr, old, new) \
__sync_cmpxchg(ptr, old, new, sizeof(*(ptr)))
-#define cmpxchg_local(ptr, old, new) \
+#define arch_cmpxchg_local(ptr, old, new) \
__cmpxchg_local(ptr, old, new, sizeof(*(ptr)))
@@ -221,7 +221,7 @@ extern void __add_wrong_size(void)
#define __try_cmpxchg(ptr, pold, new, size) \
__raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX)
-#define try_cmpxchg(ptr, pold, new) \
+#define try_cmpxchg(ptr, pold, new) \
__try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr)))
/*
@@ -250,10 +250,10 @@ extern void __add_wrong_size(void)
__ret; \
})
-#define cmpxchg_double(p1, p2, o1, o2, n1, n2) \
+#define arch_cmpxchg_double(p1, p2, o1, o2, n1, n2) \
__cmpxchg_double(LOCK_PREFIX, p1, p2, o1, o2, n1, n2)
-#define cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \
+#define arch_cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \
__cmpxchg_double(, p1, p2, o1, o2, n1, n2)
#endif /* ASM_X86_CMPXCHG_H */
diff --git a/arch/x86/include/asm/cmpxchg_32.h b/arch/x86/include/asm/cmpxchg_32.h
index 1732704f0445..1a2eafca7038 100644
--- a/arch/x86/include/asm/cmpxchg_32.h
+++ b/arch/x86/include/asm/cmpxchg_32.h
@@ -36,10 +36,10 @@ static inline void set_64bit(volatile u64 *ptr, u64 value)
}
#ifdef CONFIG_X86_CMPXCHG64
-#define cmpxchg64(ptr, o, n) \
+#define arch_cmpxchg64(ptr, o, n) \
((__typeof__(*(ptr)))__cmpxchg64((ptr), (unsigned long long)(o), \
(unsigned long long)(n)))
-#define cmpxchg64_local(ptr, o, n) \
+#define arch_cmpxchg64_local(ptr, o, n) \
((__typeof__(*(ptr)))__cmpxchg64_local((ptr), (unsigned long long)(o), \
(unsigned long long)(n)))
#endif
@@ -76,7 +76,7 @@ static inline u64 __cmpxchg64_local(volatile u64 *ptr, u64 old, u64 new)
* to simulate the cmpxchg8b on the 80386 and 80486 CPU.
*/
-#define cmpxchg64(ptr, o, n) \
+#define arch_cmpxchg64(ptr, o, n) \
({ \
__typeof__(*(ptr)) __ret; \
__typeof__(*(ptr)) __old = (o); \
@@ -93,7 +93,7 @@ static inline u64 __cmpxchg64_local(volatile u64 *ptr, u64 old, u64 new)
__ret; })
-#define cmpxchg64_local(ptr, o, n) \
+#define arch_cmpxchg64_local(ptr, o, n) \
({ \
__typeof__(*(ptr)) __ret; \
__typeof__(*(ptr)) __old = (o); \
diff --git a/arch/x86/include/asm/cmpxchg_64.h b/arch/x86/include/asm/cmpxchg_64.h
index 03cad196a301..bfca3b346c74 100644
--- a/arch/x86/include/asm/cmpxchg_64.h
+++ b/arch/x86/include/asm/cmpxchg_64.h
@@ -7,13 +7,13 @@ static inline void set_64bit(volatile u64 *ptr, u64 val)
*ptr = val;
}
-#define cmpxchg64(ptr, o, n) \
+#define arch_cmpxchg64(ptr, o, n) \
({ \
BUILD_BUG_ON(sizeof(*(ptr)) != 8); \
cmpxchg((ptr), (o), (n)); \
})
-#define cmpxchg64_local(ptr, o, n) \
+#define arch_cmpxchg64_local(ptr, o, n) \
({ \
BUILD_BUG_ON(sizeof(*(ptr)) != 8); \
cmpxchg_local((ptr), (o), (n)); \
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [tip:locking/core] locking/atomic, asm-generic: Add KASAN instrumentation to atomic operations
2018-01-29 17:26 ` [PATCH v6 3/4] asm-generic: add KASAN instrumentation to atomic operations Dmitry Vyukov
@ 2018-03-12 12:24 ` tip-bot for Dmitry Vyukov
0 siblings, 0 replies; 38+ messages in thread
From: tip-bot for Dmitry Vyukov @ 2018-03-12 12:24 UTC (permalink / raw)
To: linux-tip-commits
Cc: mingo, mark.rutland, tglx, hpa, linux-kernel, peterz, dvyukov, torvalds
Commit-ID: a35353bb9eb1990a44a0d7585f99e9589bcdb682
Gitweb: https://git.kernel.org/tip/a35353bb9eb1990a44a0d7585f99e9589bcdb682
Author: Dmitry Vyukov <dvyukov@google.com>
AuthorDate: Mon, 29 Jan 2018 18:26:06 +0100
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Mon, 12 Mar 2018 12:15:35 +0100
locking/atomic, asm-generic: Add KASAN instrumentation to atomic operations
KASAN uses compiler instrumentation to intercept all memory accesses. But it does
not see memory accesses done in assembly code. One notable user of assembly code
is atomic operations. Frequently, for example, an atomic reference decrement is
the last access to an object and a good candidate for a racy use-after-free.
Add manual KASAN checks to atomic operations.
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Cc: Andrew Morton <akpm@linux-foundation.org>,
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>,
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Will Deacon <will.deacon@arm.com>,
Cc: kasan-dev@googlegroups.com
Cc: linux-mm@kvack.org
Link: http://lkml.kernel.org/r/2fa6e7f0210fd20fe404e5b67e6e9213af2b69a1.1517246437.git.dvyukov@google.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
include/asm-generic/atomic-instrumented.h | 62 +++++++++++++++++++++++++++++++
1 file changed, 62 insertions(+)
diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h
index b966194d120a..82e080505982 100644
--- a/include/asm-generic/atomic-instrumented.h
+++ b/include/asm-generic/atomic-instrumented.h
@@ -2,44 +2,53 @@
#define _LINUX_ATOMIC_INSTRUMENTED_H
#include <linux/build_bug.h>
+#include <linux/kasan-checks.h>
static __always_inline int atomic_read(const atomic_t *v)
{
+ kasan_check_read(v, sizeof(*v));
return arch_atomic_read(v);
}
static __always_inline s64 atomic64_read(const atomic64_t *v)
{
+ kasan_check_read(v, sizeof(*v));
return arch_atomic64_read(v);
}
static __always_inline void atomic_set(atomic_t *v, int i)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_set(v, i);
}
static __always_inline void atomic64_set(atomic64_t *v, s64 i)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_set(v, i);
}
static __always_inline int atomic_xchg(atomic_t *v, int i)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_xchg(v, i);
}
static __always_inline s64 atomic64_xchg(atomic64_t *v, s64 i)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_xchg(v, i);
}
static __always_inline int atomic_cmpxchg(atomic_t *v, int old, int new)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_cmpxchg(v, old, new);
}
static __always_inline s64 atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_cmpxchg(v, old, new);
}
@@ -47,6 +56,8 @@ static __always_inline s64 atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new)
#define atomic_try_cmpxchg atomic_try_cmpxchg
static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new)
{
+ kasan_check_write(v, sizeof(*v));
+ kasan_check_read(old, sizeof(*old));
return arch_atomic_try_cmpxchg(v, old, new);
}
#endif
@@ -55,234 +66,281 @@ static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new)
#define atomic64_try_cmpxchg atomic64_try_cmpxchg
static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, s64 new)
{
+ kasan_check_write(v, sizeof(*v));
+ kasan_check_read(old, sizeof(*old));
return arch_atomic64_try_cmpxchg(v, old, new);
}
#endif
static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u)
{
+ kasan_check_write(v, sizeof(*v));
return __arch_atomic_add_unless(v, a, u);
}
static __always_inline bool atomic64_add_unless(atomic64_t *v, s64 a, s64 u)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_add_unless(v, a, u);
}
static __always_inline void atomic_inc(atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_inc(v);
}
static __always_inline void atomic64_inc(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_inc(v);
}
static __always_inline void atomic_dec(atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_dec(v);
}
static __always_inline void atomic64_dec(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_dec(v);
}
static __always_inline void atomic_add(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_add(i, v);
}
static __always_inline void atomic64_add(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_add(i, v);
}
static __always_inline void atomic_sub(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_sub(i, v);
}
static __always_inline void atomic64_sub(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_sub(i, v);
}
static __always_inline void atomic_and(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_and(i, v);
}
static __always_inline void atomic64_and(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_and(i, v);
}
static __always_inline void atomic_or(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_or(i, v);
}
static __always_inline void atomic64_or(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_or(i, v);
}
static __always_inline void atomic_xor(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic_xor(i, v);
}
static __always_inline void atomic64_xor(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
arch_atomic64_xor(i, v);
}
static __always_inline int atomic_inc_return(atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_inc_return(v);
}
static __always_inline s64 atomic64_inc_return(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_inc_return(v);
}
static __always_inline int atomic_dec_return(atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_dec_return(v);
}
static __always_inline s64 atomic64_dec_return(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_dec_return(v);
}
static __always_inline s64 atomic64_inc_not_zero(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_inc_not_zero(v);
}
static __always_inline s64 atomic64_dec_if_positive(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_dec_if_positive(v);
}
static __always_inline bool atomic_dec_and_test(atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_dec_and_test(v);
}
static __always_inline bool atomic64_dec_and_test(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_dec_and_test(v);
}
static __always_inline bool atomic_inc_and_test(atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_inc_and_test(v);
}
static __always_inline bool atomic64_inc_and_test(atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_inc_and_test(v);
}
static __always_inline int atomic_add_return(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_add_return(i, v);
}
static __always_inline s64 atomic64_add_return(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_add_return(i, v);
}
static __always_inline int atomic_sub_return(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_sub_return(i, v);
}
static __always_inline s64 atomic64_sub_return(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_sub_return(i, v);
}
static __always_inline int atomic_fetch_add(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_fetch_add(i, v);
}
static __always_inline s64 atomic64_fetch_add(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_fetch_add(i, v);
}
static __always_inline int atomic_fetch_sub(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_fetch_sub(i, v);
}
static __always_inline s64 atomic64_fetch_sub(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_fetch_sub(i, v);
}
static __always_inline int atomic_fetch_and(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_fetch_and(i, v);
}
static __always_inline s64 atomic64_fetch_and(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_fetch_and(i, v);
}
static __always_inline int atomic_fetch_or(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_fetch_or(i, v);
}
static __always_inline s64 atomic64_fetch_or(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_fetch_or(i, v);
}
static __always_inline int atomic_fetch_xor(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_fetch_xor(i, v);
}
static __always_inline s64 atomic64_fetch_xor(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_fetch_xor(i, v);
}
static __always_inline bool atomic_sub_and_test(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_sub_and_test(i, v);
}
static __always_inline bool atomic64_sub_and_test(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_sub_and_test(i, v);
}
static __always_inline bool atomic_add_negative(int i, atomic_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic_add_negative(i, v);
}
static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v)
{
+ kasan_check_write(v, sizeof(*v));
return arch_atomic64_add_negative(i, v);
}
static __always_inline unsigned long
cmpxchg_size(volatile void *ptr, unsigned long old, unsigned long new, int size)
{
+ kasan_check_write(ptr, size);
switch (size) {
case 1:
return arch_cmpxchg((u8 *)ptr, (u8)old, (u8)new);
@@ -308,6 +366,7 @@ static __always_inline unsigned long
sync_cmpxchg_size(volatile void *ptr, unsigned long old, unsigned long new,
int size)
{
+ kasan_check_write(ptr, size);
switch (size) {
case 1:
return arch_sync_cmpxchg((u8 *)ptr, (u8)old, (u8)new);
@@ -334,6 +393,7 @@ static __always_inline unsigned long
cmpxchg_local_size(volatile void *ptr, unsigned long old, unsigned long new,
int size)
{
+ kasan_check_write(ptr, size);
switch (size) {
case 1:
return arch_cmpxchg_local((u8 *)ptr, (u8)old, (u8)new);
@@ -359,6 +419,7 @@ cmpxchg_local_size(volatile void *ptr, unsigned long old, unsigned long new,
static __always_inline u64
cmpxchg64_size(volatile u64 *ptr, u64 old, u64 new)
{
+ kasan_check_write(ptr, sizeof(*ptr));
return arch_cmpxchg64(ptr, old, new);
}
@@ -371,6 +432,7 @@ cmpxchg64_size(volatile u64 *ptr, u64 old, u64 new)
static __always_inline u64
cmpxchg64_local_size(volatile u64 *ptr, u64 old, u64 new)
{
+ kasan_check_write(ptr, sizeof(*ptr));
return arch_cmpxchg64_local(ptr, old, new);
}
^ permalink raw reply related [flat|nested] 38+ messages in thread
* [tip:locking/core] locking/atomic, asm-generic, x86: Add comments for atomic instrumentation
2018-01-29 17:26 ` [PATCH v6 4/4] asm-generic, x86: add comments for atomic instrumentation Dmitry Vyukov
@ 2018-03-12 12:25 ` tip-bot for Dmitry Vyukov
0 siblings, 0 replies; 38+ messages in thread
From: tip-bot for Dmitry Vyukov @ 2018-03-12 12:25 UTC (permalink / raw)
To: linux-tip-commits
Cc: torvalds, linux-kernel, tglx, mingo, akpm, will.deacon, dvyukov,
aryabinin, peterz, mark.rutland, hpa
Commit-ID: ac605bee0bfab40fd5d11964705e907d2d5a32de
Gitweb: https://git.kernel.org/tip/ac605bee0bfab40fd5d11964705e907d2d5a32de
Author: Dmitry Vyukov <dvyukov@google.com>
AuthorDate: Mon, 29 Jan 2018 18:26:07 +0100
Committer: Ingo Molnar <mingo@kernel.org>
CommitDate: Mon, 12 Mar 2018 12:15:35 +0100
locking/atomic, asm-generic, x86: Add comments for atomic instrumentation
The comments are factored out from the code changes to make them
easier to read. Add them separately to explain some non-obvious
aspects.
Signed-off-by: Dmitry Vyukov <dvyukov@google.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Andrey Ryabinin <aryabinin@virtuozzo.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Will Deacon <will.deacon@arm.com>
Cc: kasan-dev@googlegroups.com
Cc: linux-mm@kvack.org
Link: http://lkml.kernel.org/r/cc595efc644bb905407012d82d3eb8bac3368e7a.1517246437.git.dvyukov@google.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
---
arch/x86/include/asm/atomic.h | 4 ++++
include/asm-generic/atomic-instrumented.h | 21 +++++++++++++++++++++
2 files changed, 25 insertions(+)
diff --git a/arch/x86/include/asm/atomic.h b/arch/x86/include/asm/atomic.h
index 33afc966d6a9..0db6bec95489 100644
--- a/arch/x86/include/asm/atomic.h
+++ b/arch/x86/include/asm/atomic.h
@@ -24,6 +24,10 @@
*/
static __always_inline int arch_atomic_read(const atomic_t *v)
{
+ /*
+ * Note for KASAN: we deliberately don't use READ_ONCE_NOCHECK() here,
+ * it's non-inlined function that increases binary size and stack usage.
+ */
return READ_ONCE((v)->counter);
}
diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h
index 82e080505982..ec07f23678ea 100644
--- a/include/asm-generic/atomic-instrumented.h
+++ b/include/asm-generic/atomic-instrumented.h
@@ -1,3 +1,15 @@
+/*
+ * This file provides wrappers with KASAN instrumentation for atomic operations.
+ * To use this functionality an arch's atomic.h file needs to define all
+ * atomic operations with arch_ prefix (e.g. arch_atomic_read()) and include
+ * this file at the end. This file provides atomic_read() that forwards to
+ * arch_atomic_read() for actual atomic operation.
+ * Note: if an arch atomic operation is implemented by means of other atomic
+ * operations (e.g. atomic_read()/atomic_cmpxchg() loop), then it needs to use
+ * arch_ variants (i.e. arch_atomic_read()/arch_atomic_cmpxchg()) to avoid
+ * double instrumentation.
+ */
+
#ifndef _LINUX_ATOMIC_INSTRUMENTED_H
#define _LINUX_ATOMIC_INSTRUMENTED_H
@@ -442,6 +454,15 @@ cmpxchg64_local_size(volatile u64 *ptr, u64 old, u64 new)
(u64)(new))); \
})
+/*
+ * Originally we had the following code here:
+ * __typeof__(p1) ____p1 = (p1);
+ * kasan_check_write(____p1, 2 * sizeof(*____p1));
+ * arch_cmpxchg_double(____p1, (p2), (o1), (o2), (n1), (n2));
+ * But it leads to compilation failures (see gcc issue 72873).
+ * So for now it's left non-instrumented.
+ * There are few callers of cmpxchg_double(), so it's not critical.
+ */
#define cmpxchg_double(p1, p2, o1, o2, n1, n2) \
({ \
arch_cmpxchg_double((p1), (p2), (o1), (o2), (n1), (n2)); \
^ permalink raw reply related [flat|nested] 38+ messages in thread
end of thread, other threads:[~2018-03-12 12:25 UTC | newest]
Thread overview: 38+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-06-17 9:15 [PATCH v4 0/7] x86, kasan: add KASAN checks to atomic operations Dmitry Vyukov
2017-06-17 9:15 ` [PATCH v4 1/7] x86: un-macro-ify atomic ops implementation Dmitry Vyukov
2017-06-22 11:04 ` [tip:locking/core] locking/atomic/x86: Un-macro-ify " tip-bot for Dmitry Vyukov
2017-07-25 13:54 ` tip-bot for Dmitry Vyukov
2017-06-17 9:15 ` [PATCH v4 2/7] x86: use s64* for old arg of atomic64_try_cmpxchg() Dmitry Vyukov
2017-06-22 11:04 ` [tip:locking/core] locking/atomic/x86: Use 's64 *' for 'old' argument " tip-bot for Dmitry Vyukov
2017-07-25 13:55 ` tip-bot for Dmitry Vyukov
2017-06-17 9:15 ` [PATCH v4 3/7] asm-generic: add atomic-instrumented.h Dmitry Vyukov
2017-06-19 10:50 ` Mark Rutland
2017-06-22 11:05 ` [tip:locking/core] locking/atomic: Add asm-generic/atomic-instrumented.h tip-bot for Dmitry Vyukov
2018-03-12 12:23 ` [tip:locking/core] locking/atomic, asm-generic: " tip-bot for Dmitry Vyukov
2017-06-17 9:15 ` [PATCH v4 4/7] x86: switch atomic.h to use atomic-instrumented.h Dmitry Vyukov
2017-06-17 9:15 ` [PATCH v4 5/7] kasan: allow kasan_check_read/write() to accept pointers to volatiles Dmitry Vyukov
2017-06-19 10:50 ` Mark Rutland
2017-06-19 13:11 ` Dmitry Vyukov
2017-06-22 8:25 ` Ingo Molnar
2017-06-22 14:15 ` Dmitry Vyukov
2017-06-17 9:15 ` [PATCH v4 6/7] asm-generic: add KASAN instrumentation to atomic operations Dmitry Vyukov
2017-06-19 10:51 ` Mark Rutland
2017-06-17 9:15 ` [PATCH v4 7/7] asm-generic, x86: add comments for atomic instrumentation Dmitry Vyukov
2017-06-19 10:54 ` Mark Rutland
2018-01-29 17:26 [PATCH v6 0/4] x86, kasan: add KASAN checks to atomic operations Dmitry Vyukov
2018-01-29 17:26 ` [PATCH v6 1/4] locking/atomic: Add asm-generic/atomic-instrumented.h Dmitry Vyukov
2018-01-29 17:26 ` [PATCH v6 2/4] x86: switch atomic.h to use atomic-instrumented.h Dmitry Vyukov
2018-03-12 12:24 ` [tip:locking/core] locking/atomic/x86: Switch " tip-bot for Dmitry Vyukov
2018-01-29 17:26 ` [PATCH v6 3/4] asm-generic: add KASAN instrumentation to atomic operations Dmitry Vyukov
2018-03-12 12:24 ` [tip:locking/core] locking/atomic, asm-generic: Add " tip-bot for Dmitry Vyukov
2018-01-29 17:26 ` [PATCH v6 4/4] asm-generic, x86: add comments for atomic instrumentation Dmitry Vyukov
2018-03-12 12:25 ` [tip:locking/core] locking/atomic, asm-generic, x86: Add " tip-bot for Dmitry Vyukov
2018-01-30 9:23 ` [PATCH v6 0/4] x86, kasan: add KASAN checks to atomic operations Dmitry Vyukov
2018-01-30 9:27 ` Dmitry Vyukov
2018-01-30 15:36 ` Will Deacon
2018-01-31 7:28 ` Ingo Molnar
2018-01-31 8:53 ` Dmitry Vyukov
2018-01-31 16:17 ` Will Deacon
2018-02-07 14:17 ` Dmitry Vyukov
2018-02-20 10:40 ` Dmitry Vyukov
2018-02-26 12:52 ` Dmitry Vyukov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).