* [PATCH v5 0/4] x86, kasan: add KASAN checks to atomic operations
@ 2017-06-22 14:14 Dmitry Vyukov
[not found] ` <cover.1498140838.git.dvyukov@google.com>
0 siblings, 1 reply; 63+ messages in thread
From: Dmitry Vyukov @ 2017-06-22 14:14 UTC (permalink / raw)
To: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin,
kasan-dev, x86, linux-kernel
Cc: Dmitry Vyukov
KASAN uses compiler instrumentation to intercept all memory accesses.
But it does not see memory accesses done in assembly code.
One notable user of assembly code is atomic operations. Frequently,
for example, an atomic reference decrement is the last access to an
object and a good candidate for a racy use-after-free.
Atomic operations are defined in arch files, but KASAN instrumentation
is required for several archs that support KASAN. Later we will need
similar hooks for KMSAN (uninit use detector) and KTSAN (data race
detector).
This change introduces wrappers around atomic operations that can be
used to add KASAN/KMSAN/KTSAN instrumentation across several archs,
and adds KASAN checks to them.
This patch uses the wrappers only for x86 arch. Arm64 will be switched
later. And we also plan to instrument bitops in a similar way.
Within a day it has found its first bug:
BUG: KASAN: use-after-free in atomic_dec_and_test
arch/x86/include/asm/atomic.h:123 [inline] at addr ffff880079c30158
Write of size 4 by task syz-executor6/25698
CPU: 2 PID: 25698 Comm: syz-executor6 Not tainted 4.10.0+ #302
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs 01/01/2011
Call Trace:
kasan_check_write+0x14/0x20 mm/kasan/kasan.c:344
atomic_dec_and_test arch/x86/include/asm/atomic.h:123 [inline]
put_task_struct include/linux/sched/task.h:93 [inline]
put_ctx+0xcf/0x110 kernel/events/core.c:1131
perf_event_release_kernel+0x3ad/0xc90 kernel/events/core.c:4322
perf_release+0x37/0x50 kernel/events/core.c:4338
__fput+0x332/0x800 fs/file_table.c:209
____fput+0x15/0x20 fs/file_table.c:245
task_work_run+0x197/0x260 kernel/task_work.c:116
exit_task_work include/linux/task_work.h:21 [inline]
do_exit+0xb38/0x29c0 kernel/exit.c:880
do_group_exit+0x149/0x420 kernel/exit.c:984
get_signal+0x7e0/0x1820 kernel/signal.c:2318
do_signal+0xd2/0x2190 arch/x86/kernel/signal.c:808
exit_to_usermode_loop+0x200/0x2a0 arch/x86/entry/common.c:157
syscall_return_slowpath arch/x86/entry/common.c:191 [inline]
do_syscall_64+0x6fc/0x930 arch/x86/entry/common.c:286
entry_SYSCALL64_slow_path+0x25/0x25
RIP: 0033:0x4458d9
RSP: 002b:00007f3f07187cf8 EFLAGS: 00000246 ORIG_RAX: 00000000000000ca
RAX: fffffffffffffe00 RBX: 00000000007080c8 RCX: 00000000004458d9
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 00000000007080c8
RBP: 00000000007080a8 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000000 R14: 00007f3f071889c0 R15: 00007f3f07188700
Object at ffff880079c30140, in cache task_struct size: 5376
Allocated:
PID = 25681
kmem_cache_alloc_node+0x122/0x6f0 mm/slab.c:3662
alloc_task_struct_node kernel/fork.c:153 [inline]
dup_task_struct kernel/fork.c:495 [inline]
copy_process.part.38+0x19c8/0x4aa0 kernel/fork.c:1560
copy_process kernel/fork.c:1531 [inline]
_do_fork+0x200/0x1010 kernel/fork.c:1994
SYSC_clone kernel/fork.c:2104 [inline]
SyS_clone+0x37/0x50 kernel/fork.c:2098
do_syscall_64+0x2e8/0x930 arch/x86/entry/common.c:281
return_from_SYSCALL_64+0x0/0x7a
Freed:
PID = 25681
__cache_free mm/slab.c:3514 [inline]
kmem_cache_free+0x71/0x240 mm/slab.c:3774
free_task_struct kernel/fork.c:158 [inline]
free_task+0x151/0x1d0 kernel/fork.c:370
copy_process.part.38+0x18e5/0x4aa0 kernel/fork.c:1931
copy_process kernel/fork.c:1531 [inline]
_do_fork+0x200/0x1010 kernel/fork.c:1994
SYSC_clone kernel/fork.c:2104 [inline]
SyS_clone+0x37/0x50 kernel/fork.c:2098
do_syscall_64+0x2e8/0x930 arch/x86/entry/common.c:281
return_from_SYSCALL_64+0x0/0x7a
Changes since v1:
- dropped "x86: remove unused atomic_inc_short()" patch
it is mailed separately
- rebased on top of tip/locking/core head
- other changes noted within individual patches
Changes since v2:
- rebased on top of tip/locking/core head
- dropped a pervasive "x86: use long long for 64-bit atomic ops" commit,
instead use s64 type in wrappers
- added "x86: use s64* for old arg of atomic64_try_cmpxchg()" commit
Changes since v3 are noted in individual commits.
Changes since v4:
- rebased on tip/locking/core HEAD
Tested:
- build/boot x86_64 defconfig
- build/boot x86_64 defconfig+KASAN
- build i686/powerpc defconfig
Dmitry Vyukov (4):
x86: switch atomic.h to use atomic-instrumented.h
kasan: allow kasan_check_read/write() to accept pointers to volatiles
asm-generic: add KASAN instrumentation to atomic operations
asm-generic, x86: add comments for atomic instrumentation
arch/x86/include/asm/atomic.h | 106 +++++++++++++++--------------
arch/x86/include/asm/atomic64_32.h | 106 +++++++++++++++--------------
arch/x86/include/asm/atomic64_64.h | 108 +++++++++++++++---------------
arch/x86/include/asm/cmpxchg.h | 12 ++--
arch/x86/include/asm/cmpxchg_32.h | 8 +--
arch/x86/include/asm/cmpxchg_64.h | 4 +-
include/asm-generic/atomic-instrumented.h | 106 +++++++++++++++++++++++++++--
include/linux/kasan-checks.h | 10 +--
mm/kasan/kasan.c | 4 +-
9 files changed, 286 insertions(+), 178 deletions(-)
--
2.13.1.611.g7e3b11ae1-goog
^ permalink raw reply [flat|nested] 63+ messages in thread[parent not found: <cover.1498140838.git.dvyukov@google.com>]
* [PATCH v5 1/4] x86: switch atomic.h to use atomic-instrumented.h [not found] ` <cover.1498140838.git.dvyukov@google.com> @ 2017-06-22 14:14 ` Dmitry Vyukov 2017-06-22 14:14 ` Dmitry Vyukov ` (2 subsequent siblings) 3 siblings, 0 replies; 63+ messages in thread From: Dmitry Vyukov @ 2017-06-22 14:14 UTC (permalink / raw) To: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin, kasan-dev, x86, linux-kernel Cc: Dmitry Vyukov, Andrew Morton, linux-mm Add arch_ prefix to all atomic operations and include <asm-generic/atomic-instrumented.h>. This will allow to add KASAN instrumentation to all atomic ops. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Will Deacon <will.deacon@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Ingo Molnar <mingo@redhat.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org Cc: x86@kernel.org --- Changes since v1: - reverted unnecessary change in __raw_try_cmpxchg() - reverted s/try_cmpxchg/arch_try_cmpxchg/ change try_cmpxchg is x86 implementation detail Changes since v3: - drop comment in arch_atomic_set() --- arch/x86/include/asm/atomic.h | 102 ++++++++++++++++++----------------- arch/x86/include/asm/atomic64_32.h | 106 ++++++++++++++++++------------------ arch/x86/include/asm/atomic64_64.h | 108 ++++++++++++++++++------------------- arch/x86/include/asm/cmpxchg.h | 12 ++--- arch/x86/include/asm/cmpxchg_32.h | 8 +-- arch/x86/include/asm/cmpxchg_64.h | 4 +- 6 files changed, 172 insertions(+), 168 deletions(-) diff --git a/arch/x86/include/asm/atomic.h b/arch/x86/include/asm/atomic.h index 0874ebda3069..03dd2a6cf7e2 100644 --- a/arch/x86/include/asm/atomic.h +++ b/arch/x86/include/asm/atomic.h @@ -16,36 +16,36 @@ #define ATOMIC_INIT(i) { (i) } /** - * atomic_read - read atomic variable + * arch_atomic_read - read atomic variable * @v: pointer of type atomic_t * * Atomically reads the value of @v. */ -static __always_inline int atomic_read(const atomic_t *v) +static __always_inline int arch_atomic_read(const atomic_t *v) { return READ_ONCE((v)->counter); } /** - * atomic_set - set atomic variable + * arch_atomic_set - set atomic variable * @v: pointer of type atomic_t * @i: required value * * Atomically sets the value of @v to @i. */ -static __always_inline void atomic_set(atomic_t *v, int i) +static __always_inline void arch_atomic_set(atomic_t *v, int i) { WRITE_ONCE(v->counter, i); } /** - * atomic_add - add integer to atomic variable + * arch_atomic_add - add integer to atomic variable * @i: integer value to add * @v: pointer of type atomic_t * * Atomically adds @i to @v. */ -static __always_inline void atomic_add(int i, atomic_t *v) +static __always_inline void arch_atomic_add(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "addl %1,%0" : "+m" (v->counter) @@ -53,13 +53,13 @@ static __always_inline void atomic_add(int i, atomic_t *v) } /** - * atomic_sub - subtract integer from atomic variable + * arch_atomic_sub - subtract integer from atomic variable * @i: integer value to subtract * @v: pointer of type atomic_t * * Atomically subtracts @i from @v. */ -static __always_inline void atomic_sub(int i, atomic_t *v) +static __always_inline void arch_atomic_sub(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "subl %1,%0" : "+m" (v->counter) @@ -67,7 +67,7 @@ static __always_inline void atomic_sub(int i, atomic_t *v) } /** - * atomic_sub_and_test - subtract value from variable and test result + * arch_atomic_sub_and_test - subtract value from variable and test result * @i: integer value to subtract * @v: pointer of type atomic_t * @@ -75,63 +75,63 @@ static __always_inline void atomic_sub(int i, atomic_t *v) * true if the result is zero, or false for all * other cases. */ -static __always_inline bool atomic_sub_and_test(int i, atomic_t *v) +static __always_inline bool arch_atomic_sub_and_test(int i, atomic_t *v) { GEN_BINARY_RMWcc(LOCK_PREFIX "subl", v->counter, "er", i, "%0", e); } /** - * atomic_inc - increment atomic variable + * arch_atomic_inc - increment atomic variable * @v: pointer of type atomic_t * * Atomically increments @v by 1. */ -static __always_inline void atomic_inc(atomic_t *v) +static __always_inline void arch_atomic_inc(atomic_t *v) { asm volatile(LOCK_PREFIX "incl %0" : "+m" (v->counter)); } /** - * atomic_dec - decrement atomic variable + * arch_atomic_dec - decrement atomic variable * @v: pointer of type atomic_t * * Atomically decrements @v by 1. */ -static __always_inline void atomic_dec(atomic_t *v) +static __always_inline void arch_atomic_dec(atomic_t *v) { asm volatile(LOCK_PREFIX "decl %0" : "+m" (v->counter)); } /** - * atomic_dec_and_test - decrement and test + * arch_atomic_dec_and_test - decrement and test * @v: pointer of type atomic_t * * Atomically decrements @v by 1 and * returns true if the result is 0, or false for all other * cases. */ -static __always_inline bool atomic_dec_and_test(atomic_t *v) +static __always_inline bool arch_atomic_dec_and_test(atomic_t *v) { GEN_UNARY_RMWcc(LOCK_PREFIX "decl", v->counter, "%0", e); } /** - * atomic_inc_and_test - increment and test + * arch_atomic_inc_and_test - increment and test * @v: pointer of type atomic_t * * Atomically increments @v by 1 * and returns true if the result is zero, or false for all * other cases. */ -static __always_inline bool atomic_inc_and_test(atomic_t *v) +static __always_inline bool arch_atomic_inc_and_test(atomic_t *v) { GEN_UNARY_RMWcc(LOCK_PREFIX "incl", v->counter, "%0", e); } /** - * atomic_add_negative - add and test if negative + * arch_atomic_add_negative - add and test if negative * @i: integer value to add * @v: pointer of type atomic_t * @@ -139,65 +139,65 @@ static __always_inline bool atomic_inc_and_test(atomic_t *v) * if the result is negative, or false when * result is greater than or equal to zero. */ -static __always_inline bool atomic_add_negative(int i, atomic_t *v) +static __always_inline bool arch_atomic_add_negative(int i, atomic_t *v) { GEN_BINARY_RMWcc(LOCK_PREFIX "addl", v->counter, "er", i, "%0", s); } /** - * atomic_add_return - add integer and return + * arch_atomic_add_return - add integer and return * @i: integer value to add * @v: pointer of type atomic_t * * Atomically adds @i to @v and returns @i + @v */ -static __always_inline int atomic_add_return(int i, atomic_t *v) +static __always_inline int arch_atomic_add_return(int i, atomic_t *v) { return i + xadd(&v->counter, i); } /** - * atomic_sub_return - subtract integer and return + * arch_atomic_sub_return - subtract integer and return * @v: pointer of type atomic_t * @i: integer value to subtract * * Atomically subtracts @i from @v and returns @v - @i */ -static __always_inline int atomic_sub_return(int i, atomic_t *v) +static __always_inline int arch_atomic_sub_return(int i, atomic_t *v) { - return atomic_add_return(-i, v); + return arch_atomic_add_return(-i, v); } -#define atomic_inc_return(v) (atomic_add_return(1, v)) -#define atomic_dec_return(v) (atomic_sub_return(1, v)) +#define arch_atomic_inc_return(v) (arch_atomic_add_return(1, v)) +#define arch_atomic_dec_return(v) (arch_atomic_sub_return(1, v)) -static __always_inline int atomic_fetch_add(int i, atomic_t *v) +static __always_inline int arch_atomic_fetch_add(int i, atomic_t *v) { return xadd(&v->counter, i); } -static __always_inline int atomic_fetch_sub(int i, atomic_t *v) +static __always_inline int arch_atomic_fetch_sub(int i, atomic_t *v) { return xadd(&v->counter, -i); } -static __always_inline int atomic_cmpxchg(atomic_t *v, int old, int new) +static __always_inline int arch_atomic_cmpxchg(atomic_t *v, int old, int new) { - return cmpxchg(&v->counter, old, new); + return arch_cmpxchg(&v->counter, old, new); } -#define atomic_try_cmpxchg atomic_try_cmpxchg -static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new) +#define arch_atomic_try_cmpxchg arch_atomic_try_cmpxchg +static __always_inline bool arch_atomic_try_cmpxchg(atomic_t *v, int *old, int new) { return try_cmpxchg(&v->counter, old, new); } -static inline int atomic_xchg(atomic_t *v, int new) +static inline int arch_atomic_xchg(atomic_t *v, int new) { return xchg(&v->counter, new); } -static inline void atomic_and(int i, atomic_t *v) +static inline void arch_atomic_and(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "andl %1,%0" : "+m" (v->counter) @@ -205,16 +205,16 @@ static inline void atomic_and(int i, atomic_t *v) : "memory"); } -static inline int atomic_fetch_and(int i, atomic_t *v) +static inline int arch_atomic_fetch_and(int i, atomic_t *v) { - int val = atomic_read(v); + int val = arch_atomic_read(v); - do { } while (!atomic_try_cmpxchg(v, &val, val & i)); + do { } while (!arch_atomic_try_cmpxchg(v, &val, val & i)); return val; } -static inline void atomic_or(int i, atomic_t *v) +static inline void arch_atomic_or(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "orl %1,%0" : "+m" (v->counter) @@ -222,16 +222,16 @@ static inline void atomic_or(int i, atomic_t *v) : "memory"); } -static inline int atomic_fetch_or(int i, atomic_t *v) +static inline int arch_atomic_fetch_or(int i, atomic_t *v) { - int val = atomic_read(v); + int val = arch_atomic_read(v); - do { } while (!atomic_try_cmpxchg(v, &val, val | i)); + do { } while (!arch_atomic_try_cmpxchg(v, &val, val | i)); return val; } -static inline void atomic_xor(int i, atomic_t *v) +static inline void arch_atomic_xor(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "xorl %1,%0" : "+m" (v->counter) @@ -239,17 +239,17 @@ static inline void atomic_xor(int i, atomic_t *v) : "memory"); } -static inline int atomic_fetch_xor(int i, atomic_t *v) +static inline int arch_atomic_fetch_xor(int i, atomic_t *v) { - int val = atomic_read(v); + int val = arch_atomic_read(v); - do { } while (!atomic_try_cmpxchg(v, &val, val ^ i)); + do { } while (!arch_atomic_try_cmpxchg(v, &val, val ^ i)); return val; } /** - * __atomic_add_unless - add unless the number is already a given value + * __arch_atomic_add_unless - add unless the number is already a given value * @v: pointer of type atomic_t * @a: the amount to add to v... * @u: ...unless v is equal to u. @@ -257,14 +257,14 @@ static inline int atomic_fetch_xor(int i, atomic_t *v) * Atomically adds @a to @v, so long as @v was not already @u. * Returns the old value of @v. */ -static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u) +static __always_inline int __arch_atomic_add_unless(atomic_t *v, int a, int u) { - int c = atomic_read(v); + int c = arch_atomic_read(v); do { if (unlikely(c == u)) break; - } while (!atomic_try_cmpxchg(v, &c, c + a)); + } while (!arch_atomic_try_cmpxchg(v, &c, c + a)); return c; } @@ -275,4 +275,6 @@ static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u) # include <asm/atomic64_64.h> #endif +#include <asm-generic/atomic-instrumented.h> + #endif /* _ASM_X86_ATOMIC_H */ diff --git a/arch/x86/include/asm/atomic64_32.h b/arch/x86/include/asm/atomic64_32.h index 9e206f31ce2a..512377a1c2aa 100644 --- a/arch/x86/include/asm/atomic64_32.h +++ b/arch/x86/include/asm/atomic64_32.h @@ -61,7 +61,7 @@ ATOMIC64_DECL(add_unless); #undef ATOMIC64_EXPORT /** - * atomic64_cmpxchg - cmpxchg atomic64 variable + * arch_atomic64_cmpxchg - cmpxchg atomic64 variable * @v: pointer to type atomic64_t * @o: expected value * @n: new value @@ -70,20 +70,21 @@ ATOMIC64_DECL(add_unless); * the old value. */ -static inline long long atomic64_cmpxchg(atomic64_t *v, long long o, long long n) +static inline long long arch_atomic64_cmpxchg(atomic64_t *v, long long o, + long long n) { - return cmpxchg64(&v->counter, o, n); + return arch_cmpxchg64(&v->counter, o, n); } /** - * atomic64_xchg - xchg atomic64 variable + * arch_atomic64_xchg - xchg atomic64 variable * @v: pointer to type atomic64_t * @n: value to assign * * Atomically xchgs the value of @v to @n and returns * the old value. */ -static inline long long atomic64_xchg(atomic64_t *v, long long n) +static inline long long arch_atomic64_xchg(atomic64_t *v, long long n) { long long o; unsigned high = (unsigned)(n >> 32); @@ -95,13 +96,13 @@ static inline long long atomic64_xchg(atomic64_t *v, long long n) } /** - * atomic64_set - set atomic64 variable + * arch_atomic64_set - set atomic64 variable * @v: pointer to type atomic64_t * @i: value to assign * * Atomically sets the value of @v to @n. */ -static inline void atomic64_set(atomic64_t *v, long long i) +static inline void arch_atomic64_set(atomic64_t *v, long long i) { unsigned high = (unsigned)(i >> 32); unsigned low = (unsigned)i; @@ -111,12 +112,12 @@ static inline void atomic64_set(atomic64_t *v, long long i) } /** - * atomic64_read - read atomic64 variable + * arch_atomic64_read - read atomic64 variable * @v: pointer to type atomic64_t * * Atomically reads the value of @v and returns it. */ -static inline long long atomic64_read(const atomic64_t *v) +static inline long long arch_atomic64_read(const atomic64_t *v) { long long r; alternative_atomic64(read, "=&A" (r), "c" (v) : "memory"); @@ -124,13 +125,13 @@ static inline long long atomic64_read(const atomic64_t *v) } /** - * atomic64_add_return - add and return + * arch_atomic64_add_return - add and return * @i: integer value to add * @v: pointer to type atomic64_t * * Atomically adds @i to @v and returns @i + *@v */ -static inline long long atomic64_add_return(long long i, atomic64_t *v) +static inline long long arch_atomic64_add_return(long long i, atomic64_t *v) { alternative_atomic64(add_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -141,7 +142,7 @@ static inline long long atomic64_add_return(long long i, atomic64_t *v) /* * Other variants with different arithmetic operators: */ -static inline long long atomic64_sub_return(long long i, atomic64_t *v) +static inline long long arch_atomic64_sub_return(long long i, atomic64_t *v) { alternative_atomic64(sub_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -149,7 +150,7 @@ static inline long long atomic64_sub_return(long long i, atomic64_t *v) return i; } -static inline long long atomic64_inc_return(atomic64_t *v) +static inline long long arch_atomic64_inc_return(atomic64_t *v) { long long a; alternative_atomic64(inc_return, "=&A" (a), @@ -157,7 +158,7 @@ static inline long long atomic64_inc_return(atomic64_t *v) return a; } -static inline long long atomic64_dec_return(atomic64_t *v) +static inline long long arch_atomic64_dec_return(atomic64_t *v) { long long a; alternative_atomic64(dec_return, "=&A" (a), @@ -166,13 +167,13 @@ static inline long long atomic64_dec_return(atomic64_t *v) } /** - * atomic64_add - add integer to atomic64 variable + * arch_atomic64_add - add integer to atomic64 variable * @i: integer value to add * @v: pointer to type atomic64_t * * Atomically adds @i to @v. */ -static inline long long atomic64_add(long long i, atomic64_t *v) +static inline long long arch_atomic64_add(long long i, atomic64_t *v) { __alternative_atomic64(add, add_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -181,13 +182,13 @@ static inline long long atomic64_add(long long i, atomic64_t *v) } /** - * atomic64_sub - subtract the atomic64 variable + * arch_atomic64_sub - subtract the atomic64 variable * @i: integer value to subtract * @v: pointer to type atomic64_t * * Atomically subtracts @i from @v. */ -static inline long long atomic64_sub(long long i, atomic64_t *v) +static inline long long arch_atomic64_sub(long long i, atomic64_t *v) { __alternative_atomic64(sub, sub_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -196,7 +197,7 @@ static inline long long atomic64_sub(long long i, atomic64_t *v) } /** - * atomic64_sub_and_test - subtract value from variable and test result + * arch_atomic64_sub_and_test - subtract value from variable and test result * @i: integer value to subtract * @v: pointer to type atomic64_t * @@ -204,46 +205,46 @@ static inline long long atomic64_sub(long long i, atomic64_t *v) * true if the result is zero, or false for all * other cases. */ -static inline int atomic64_sub_and_test(long long i, atomic64_t *v) +static inline int arch_atomic64_sub_and_test(long long i, atomic64_t *v) { - return atomic64_sub_return(i, v) == 0; + return arch_atomic64_sub_return(i, v) == 0; } /** - * atomic64_inc - increment atomic64 variable + * arch_atomic64_inc - increment atomic64 variable * @v: pointer to type atomic64_t * * Atomically increments @v by 1. */ -static inline void atomic64_inc(atomic64_t *v) +static inline void arch_atomic64_inc(atomic64_t *v) { __alternative_atomic64(inc, inc_return, /* no output */, "S" (v) : "memory", "eax", "ecx", "edx"); } /** - * atomic64_dec - decrement atomic64 variable + * arch_atomic64_dec - decrement atomic64 variable * @v: pointer to type atomic64_t * * Atomically decrements @v by 1. */ -static inline void atomic64_dec(atomic64_t *v) +static inline void arch_atomic64_dec(atomic64_t *v) { __alternative_atomic64(dec, dec_return, /* no output */, "S" (v) : "memory", "eax", "ecx", "edx"); } /** - * atomic64_dec_and_test - decrement and test + * arch_atomic64_dec_and_test - decrement and test * @v: pointer to type atomic64_t * * Atomically decrements @v by 1 and * returns true if the result is 0, or false for all other * cases. */ -static inline int atomic64_dec_and_test(atomic64_t *v) +static inline int arch_atomic64_dec_and_test(atomic64_t *v) { - return atomic64_dec_return(v) == 0; + return arch_atomic64_dec_return(v) == 0; } /** @@ -254,13 +255,13 @@ static inline int atomic64_dec_and_test(atomic64_t *v) * and returns true if the result is zero, or false for all * other cases. */ -static inline int atomic64_inc_and_test(atomic64_t *v) +static inline int arch_atomic64_inc_and_test(atomic64_t *v) { - return atomic64_inc_return(v) == 0; + return arch_atomic64_inc_return(v) == 0; } /** - * atomic64_add_negative - add and test if negative + * arch_atomic64_add_negative - add and test if negative * @i: integer value to add * @v: pointer to type atomic64_t * @@ -268,13 +269,13 @@ static inline int atomic64_inc_and_test(atomic64_t *v) * if the result is negative, or false when * result is greater than or equal to zero. */ -static inline int atomic64_add_negative(long long i, atomic64_t *v) +static inline int arch_atomic64_add_negative(long long i, atomic64_t *v) { - return atomic64_add_return(i, v) < 0; + return arch_atomic64_add_return(i, v) < 0; } /** - * atomic64_add_unless - add unless the number is a given value + * arch_atomic64_add_unless - add unless the number is a given value * @v: pointer of type atomic64_t * @a: the amount to add to v... * @u: ...unless v is equal to u. @@ -282,7 +283,8 @@ static inline int atomic64_add_negative(long long i, atomic64_t *v) * Atomically adds @a to @v, so long as it was not @u. * Returns non-zero if the add was done, zero otherwise. */ -static inline int atomic64_add_unless(atomic64_t *v, long long a, long long u) +static inline int arch_atomic64_add_unless(atomic64_t *v, long long a, + long long u) { unsigned low = (unsigned)u; unsigned high = (unsigned)(u >> 32); @@ -293,7 +295,7 @@ static inline int atomic64_add_unless(atomic64_t *v, long long a, long long u) } -static inline int atomic64_inc_not_zero(atomic64_t *v) +static inline int arch_atomic64_inc_not_zero(atomic64_t *v) { int r; alternative_atomic64(inc_not_zero, "=&a" (r), @@ -301,7 +303,7 @@ static inline int atomic64_inc_not_zero(atomic64_t *v) return r; } -static inline long long atomic64_dec_if_positive(atomic64_t *v) +static inline long long arch_atomic64_dec_if_positive(atomic64_t *v) { long long r; alternative_atomic64(dec_if_positive, "=&A" (r), @@ -312,70 +314,70 @@ static inline long long atomic64_dec_if_positive(atomic64_t *v) #undef alternative_atomic64 #undef __alternative_atomic64 -static inline void atomic64_and(long long i, atomic64_t *v) +static inline void arch_atomic64_and(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c & i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c & i)) != c) c = old; } -static inline long long atomic64_fetch_and(long long i, atomic64_t *v) +static inline long long arch_atomic64_fetch_and(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c & i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c & i)) != c) c = old; return old; } -static inline void atomic64_or(long long i, atomic64_t *v) +static inline void arch_atomic64_or(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c | i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c | i)) != c) c = old; } -static inline long long atomic64_fetch_or(long long i, atomic64_t *v) +static inline long long arch_atomic64_fetch_or(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c | i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c | i)) != c) c = old; return old; } -static inline void atomic64_xor(long long i, atomic64_t *v) +static inline void arch_atomic64_xor(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c ^ i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c ^ i)) != c) c = old; } -static inline long long atomic64_fetch_xor(long long i, atomic64_t *v) +static inline long long arch_atomic64_fetch_xor(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c ^ i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c ^ i)) != c) c = old; return old; } -static inline long long atomic64_fetch_add(long long i, atomic64_t *v) +static inline long long arch_atomic64_fetch_add(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c + i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c + i)) != c) c = old; return old; } -#define atomic64_fetch_sub(i, v) atomic64_fetch_add(-(i), (v)) +#define arch_atomic64_fetch_sub(i, v) arch_atomic64_fetch_add(-(i), (v)) #endif /* _ASM_X86_ATOMIC64_32_H */ diff --git a/arch/x86/include/asm/atomic64_64.h b/arch/x86/include/asm/atomic64_64.h index 5d9de36a2f04..5883b109d053 100644 --- a/arch/x86/include/asm/atomic64_64.h +++ b/arch/x86/include/asm/atomic64_64.h @@ -10,37 +10,37 @@ #define ATOMIC64_INIT(i) { (i) } /** - * atomic64_read - read atomic64 variable + * arch_atomic64_read - read atomic64 variable * @v: pointer of type atomic64_t * * Atomically reads the value of @v. * Doesn't imply a read memory barrier. */ -static inline long atomic64_read(const atomic64_t *v) +static inline long arch_atomic64_read(const atomic64_t *v) { return READ_ONCE((v)->counter); } /** - * atomic64_set - set atomic64 variable + * arch_atomic64_set - set atomic64 variable * @v: pointer to type atomic64_t * @i: required value * * Atomically sets the value of @v to @i. */ -static inline void atomic64_set(atomic64_t *v, long i) +static inline void arch_atomic64_set(atomic64_t *v, long i) { WRITE_ONCE(v->counter, i); } /** - * atomic64_add - add integer to atomic64 variable + * arch_atomic64_add - add integer to atomic64 variable * @i: integer value to add * @v: pointer to type atomic64_t * * Atomically adds @i to @v. */ -static __always_inline void atomic64_add(long i, atomic64_t *v) +static __always_inline void arch_atomic64_add(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "addq %1,%0" : "=m" (v->counter) @@ -48,13 +48,13 @@ static __always_inline void atomic64_add(long i, atomic64_t *v) } /** - * atomic64_sub - subtract the atomic64 variable + * arch_atomic64_sub - subtract the atomic64 variable * @i: integer value to subtract * @v: pointer to type atomic64_t * * Atomically subtracts @i from @v. */ -static inline void atomic64_sub(long i, atomic64_t *v) +static inline void arch_atomic64_sub(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "subq %1,%0" : "=m" (v->counter) @@ -62,7 +62,7 @@ static inline void atomic64_sub(long i, atomic64_t *v) } /** - * atomic64_sub_and_test - subtract value from variable and test result + * arch_atomic64_sub_and_test - subtract value from variable and test result * @i: integer value to subtract * @v: pointer to type atomic64_t * @@ -70,18 +70,18 @@ static inline void atomic64_sub(long i, atomic64_t *v) * true if the result is zero, or false for all * other cases. */ -static inline bool atomic64_sub_and_test(long i, atomic64_t *v) +static inline bool arch_atomic64_sub_and_test(long i, atomic64_t *v) { GEN_BINARY_RMWcc(LOCK_PREFIX "subq", v->counter, "er", i, "%0", e); } /** - * atomic64_inc - increment atomic64 variable + * arch_atomic64_inc - increment atomic64 variable * @v: pointer to type atomic64_t * * Atomically increments @v by 1. */ -static __always_inline void atomic64_inc(atomic64_t *v) +static __always_inline void arch_atomic64_inc(atomic64_t *v) { asm volatile(LOCK_PREFIX "incq %0" : "=m" (v->counter) @@ -89,12 +89,12 @@ static __always_inline void atomic64_inc(atomic64_t *v) } /** - * atomic64_dec - decrement atomic64 variable + * arch_atomic64_dec - decrement atomic64 variable * @v: pointer to type atomic64_t * * Atomically decrements @v by 1. */ -static __always_inline void atomic64_dec(atomic64_t *v) +static __always_inline void arch_atomic64_dec(atomic64_t *v) { asm volatile(LOCK_PREFIX "decq %0" : "=m" (v->counter) @@ -102,33 +102,33 @@ static __always_inline void atomic64_dec(atomic64_t *v) } /** - * atomic64_dec_and_test - decrement and test + * arch_atomic64_dec_and_test - decrement and test * @v: pointer to type atomic64_t * * Atomically decrements @v by 1 and * returns true if the result is 0, or false for all other * cases. */ -static inline bool atomic64_dec_and_test(atomic64_t *v) +static inline bool arch_atomic64_dec_and_test(atomic64_t *v) { GEN_UNARY_RMWcc(LOCK_PREFIX "decq", v->counter, "%0", e); } /** - * atomic64_inc_and_test - increment and test + * arch_atomic64_inc_and_test - increment and test * @v: pointer to type atomic64_t * * Atomically increments @v by 1 * and returns true if the result is zero, or false for all * other cases. */ -static inline bool atomic64_inc_and_test(atomic64_t *v) +static inline bool arch_atomic64_inc_and_test(atomic64_t *v) { GEN_UNARY_RMWcc(LOCK_PREFIX "incq", v->counter, "%0", e); } /** - * atomic64_add_negative - add and test if negative + * arch_atomic64_add_negative - add and test if negative * @i: integer value to add * @v: pointer to type atomic64_t * @@ -136,59 +136,59 @@ static inline bool atomic64_inc_and_test(atomic64_t *v) * if the result is negative, or false when * result is greater than or equal to zero. */ -static inline bool atomic64_add_negative(long i, atomic64_t *v) +static inline bool arch_atomic64_add_negative(long i, atomic64_t *v) { GEN_BINARY_RMWcc(LOCK_PREFIX "addq", v->counter, "er", i, "%0", s); } /** - * atomic64_add_return - add and return + * arch_atomic64_add_return - add and return * @i: integer value to add * @v: pointer to type atomic64_t * * Atomically adds @i to @v and returns @i + @v */ -static __always_inline long atomic64_add_return(long i, atomic64_t *v) +static __always_inline long arch_atomic64_add_return(long i, atomic64_t *v) { return i + xadd(&v->counter, i); } -static inline long atomic64_sub_return(long i, atomic64_t *v) +static inline long arch_atomic64_sub_return(long i, atomic64_t *v) { - return atomic64_add_return(-i, v); + return arch_atomic64_add_return(-i, v); } -static inline long atomic64_fetch_add(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_add(long i, atomic64_t *v) { return xadd(&v->counter, i); } -static inline long atomic64_fetch_sub(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_sub(long i, atomic64_t *v) { return xadd(&v->counter, -i); } -#define atomic64_inc_return(v) (atomic64_add_return(1, (v))) -#define atomic64_dec_return(v) (atomic64_sub_return(1, (v))) +#define arch_atomic64_inc_return(v) (arch_atomic64_add_return(1, (v))) +#define arch_atomic64_dec_return(v) (arch_atomic64_sub_return(1, (v))) -static inline long atomic64_cmpxchg(atomic64_t *v, long old, long new) +static inline long arch_atomic64_cmpxchg(atomic64_t *v, long old, long new) { - return cmpxchg(&v->counter, old, new); + return arch_cmpxchg(&v->counter, old, new); } -#define atomic64_try_cmpxchg atomic64_try_cmpxchg -static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, long new) +#define arch_atomic64_try_cmpxchg arch_atomic64_try_cmpxchg +static __always_inline bool arch_atomic64_try_cmpxchg(atomic64_t *v, s64 *old, long new) { return try_cmpxchg(&v->counter, old, new); } -static inline long atomic64_xchg(atomic64_t *v, long new) +static inline long arch_atomic64_xchg(atomic64_t *v, long new) { return xchg(&v->counter, new); } /** - * atomic64_add_unless - add unless the number is a given value + * arch_atomic64_add_unless - add unless the number is a given value * @v: pointer of type atomic64_t * @a: the amount to add to v... * @u: ...unless v is equal to u. @@ -196,37 +196,37 @@ static inline long atomic64_xchg(atomic64_t *v, long new) * Atomically adds @a to @v, so long as it was not @u. * Returns the old value of @v. */ -static inline bool atomic64_add_unless(atomic64_t *v, long a, long u) +static inline bool arch_atomic64_add_unless(atomic64_t *v, long a, long u) { - s64 c = atomic64_read(v); + s64 c = arch_atomic64_read(v); do { if (unlikely(c == u)) return false; - } while (!atomic64_try_cmpxchg(v, &c, c + a)); + } while (!arch_atomic64_try_cmpxchg(v, &c, c + a)); return true; } -#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0) +#define arch_atomic64_inc_not_zero(v) arch_atomic64_add_unless((v), 1, 0) /* - * atomic64_dec_if_positive - decrement by 1 if old value positive + * arch_atomic64_dec_if_positive - decrement by 1 if old value positive * @v: pointer of type atomic_t * * The function returns the old value of *v minus 1, even if * the atomic variable, v, was not decremented. */ -static inline long atomic64_dec_if_positive(atomic64_t *v) +static inline long arch_atomic64_dec_if_positive(atomic64_t *v) { - s64 dec, c = atomic64_read(v); + s64 dec, c = arch_atomic64_read(v); do { dec = c - 1; if (unlikely(dec < 0)) break; - } while (!atomic64_try_cmpxchg(v, &c, dec)); + } while (!arch_atomic64_try_cmpxchg(v, &c, dec)); return dec; } -static inline void atomic64_and(long i, atomic64_t *v) +static inline void arch_atomic64_and(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "andq %1,%0" : "+m" (v->counter) @@ -234,16 +234,16 @@ static inline void atomic64_and(long i, atomic64_t *v) : "memory"); } -static inline long atomic64_fetch_and(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_and(long i, atomic64_t *v) { - s64 val = atomic64_read(v); + s64 val = arch_atomic64_read(v); do { - } while (!atomic64_try_cmpxchg(v, &val, val & i)); + } while (!arch_atomic64_try_cmpxchg(v, &val, val & i)); return val; } -static inline void atomic64_or(long i, atomic64_t *v) +static inline void arch_atomic64_or(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "orq %1,%0" : "+m" (v->counter) @@ -251,16 +251,16 @@ static inline void atomic64_or(long i, atomic64_t *v) : "memory"); } -static inline long atomic64_fetch_or(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_or(long i, atomic64_t *v) { - s64 val = atomic64_read(v); + s64 val = arch_atomic64_read(v); do { - } while (!atomic64_try_cmpxchg(v, &val, val | i)); + } while (!arch_atomic64_try_cmpxchg(v, &val, val | i)); return val; } -static inline void atomic64_xor(long i, atomic64_t *v) +static inline void arch_atomic64_xor(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "xorq %1,%0" : "+m" (v->counter) @@ -268,12 +268,12 @@ static inline void atomic64_xor(long i, atomic64_t *v) : "memory"); } -static inline long atomic64_fetch_xor(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_xor(long i, atomic64_t *v) { - s64 val = atomic64_read(v); + s64 val = arch_atomic64_read(v); do { - } while (!atomic64_try_cmpxchg(v, &val, val ^ i)); + } while (!arch_atomic64_try_cmpxchg(v, &val, val ^ i)); return val; } diff --git a/arch/x86/include/asm/cmpxchg.h b/arch/x86/include/asm/cmpxchg.h index b5069e802d5c..9e2faa85eb02 100644 --- a/arch/x86/include/asm/cmpxchg.h +++ b/arch/x86/include/asm/cmpxchg.h @@ -144,13 +144,13 @@ extern void __add_wrong_size(void) # include <asm/cmpxchg_64.h> #endif -#define cmpxchg(ptr, old, new) \ +#define arch_cmpxchg(ptr, old, new) \ __cmpxchg(ptr, old, new, sizeof(*(ptr))) -#define sync_cmpxchg(ptr, old, new) \ +#define arch_sync_cmpxchg(ptr, old, new) \ __sync_cmpxchg(ptr, old, new, sizeof(*(ptr))) -#define cmpxchg_local(ptr, old, new) \ +#define arch_cmpxchg_local(ptr, old, new) \ __cmpxchg_local(ptr, old, new, sizeof(*(ptr))) @@ -220,7 +220,7 @@ extern void __add_wrong_size(void) #define __try_cmpxchg(ptr, pold, new, size) \ __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX) -#define try_cmpxchg(ptr, pold, new) \ +#define try_cmpxchg(ptr, pold, new) \ __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr))) /* @@ -249,10 +249,10 @@ extern void __add_wrong_size(void) __ret; \ }) -#define cmpxchg_double(p1, p2, o1, o2, n1, n2) \ +#define arch_cmpxchg_double(p1, p2, o1, o2, n1, n2) \ __cmpxchg_double(LOCK_PREFIX, p1, p2, o1, o2, n1, n2) -#define cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \ +#define arch_cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \ __cmpxchg_double(, p1, p2, o1, o2, n1, n2) #endif /* ASM_X86_CMPXCHG_H */ diff --git a/arch/x86/include/asm/cmpxchg_32.h b/arch/x86/include/asm/cmpxchg_32.h index e4959d023af8..d897291d2bf9 100644 --- a/arch/x86/include/asm/cmpxchg_32.h +++ b/arch/x86/include/asm/cmpxchg_32.h @@ -35,10 +35,10 @@ static inline void set_64bit(volatile u64 *ptr, u64 value) } #ifdef CONFIG_X86_CMPXCHG64 -#define cmpxchg64(ptr, o, n) \ +#define arch_cmpxchg64(ptr, o, n) \ ((__typeof__(*(ptr)))__cmpxchg64((ptr), (unsigned long long)(o), \ (unsigned long long)(n))) -#define cmpxchg64_local(ptr, o, n) \ +#define arch_cmpxchg64_local(ptr, o, n) \ ((__typeof__(*(ptr)))__cmpxchg64_local((ptr), (unsigned long long)(o), \ (unsigned long long)(n))) #endif @@ -75,7 +75,7 @@ static inline u64 __cmpxchg64_local(volatile u64 *ptr, u64 old, u64 new) * to simulate the cmpxchg8b on the 80386 and 80486 CPU. */ -#define cmpxchg64(ptr, o, n) \ +#define arch_cmpxchg64(ptr, o, n) \ ({ \ __typeof__(*(ptr)) __ret; \ __typeof__(*(ptr)) __old = (o); \ @@ -92,7 +92,7 @@ static inline u64 __cmpxchg64_local(volatile u64 *ptr, u64 old, u64 new) __ret; }) -#define cmpxchg64_local(ptr, o, n) \ +#define arch_cmpxchg64_local(ptr, o, n) \ ({ \ __typeof__(*(ptr)) __ret; \ __typeof__(*(ptr)) __old = (o); \ diff --git a/arch/x86/include/asm/cmpxchg_64.h b/arch/x86/include/asm/cmpxchg_64.h index caa23a34c963..fafaebacca2d 100644 --- a/arch/x86/include/asm/cmpxchg_64.h +++ b/arch/x86/include/asm/cmpxchg_64.h @@ -6,13 +6,13 @@ static inline void set_64bit(volatile u64 *ptr, u64 val) *ptr = val; } -#define cmpxchg64(ptr, o, n) \ +#define arch_cmpxchg64(ptr, o, n) \ ({ \ BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ cmpxchg((ptr), (o), (n)); \ }) -#define cmpxchg64_local(ptr, o, n) \ +#define arch_cmpxchg64_local(ptr, o, n) \ ({ \ BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ cmpxchg_local((ptr), (o), (n)); \ -- 2.13.1.611.g7e3b11ae1-goog ^ permalink raw reply related [flat|nested] 63+ messages in thread
* [PATCH v5 1/4] x86: switch atomic.h to use atomic-instrumented.h @ 2017-06-22 14:14 ` Dmitry Vyukov 0 siblings, 0 replies; 63+ messages in thread From: Dmitry Vyukov @ 2017-06-22 14:14 UTC (permalink / raw) To: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin, kasan-dev, x86, linux-kernel Cc: Dmitry Vyukov, Andrew Morton, linux-mm Add arch_ prefix to all atomic operations and include <asm-generic/atomic-instrumented.h>. This will allow to add KASAN instrumentation to all atomic ops. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Will Deacon <will.deacon@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Ingo Molnar <mingo@redhat.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org Cc: x86@kernel.org --- Changes since v1: - reverted unnecessary change in __raw_try_cmpxchg() - reverted s/try_cmpxchg/arch_try_cmpxchg/ change try_cmpxchg is x86 implementation detail Changes since v3: - drop comment in arch_atomic_set() --- arch/x86/include/asm/atomic.h | 102 ++++++++++++++++++----------------- arch/x86/include/asm/atomic64_32.h | 106 ++++++++++++++++++------------------ arch/x86/include/asm/atomic64_64.h | 108 ++++++++++++++++++------------------- arch/x86/include/asm/cmpxchg.h | 12 ++--- arch/x86/include/asm/cmpxchg_32.h | 8 +-- arch/x86/include/asm/cmpxchg_64.h | 4 +- 6 files changed, 172 insertions(+), 168 deletions(-) diff --git a/arch/x86/include/asm/atomic.h b/arch/x86/include/asm/atomic.h index 0874ebda3069..03dd2a6cf7e2 100644 --- a/arch/x86/include/asm/atomic.h +++ b/arch/x86/include/asm/atomic.h @@ -16,36 +16,36 @@ #define ATOMIC_INIT(i) { (i) } /** - * atomic_read - read atomic variable + * arch_atomic_read - read atomic variable * @v: pointer of type atomic_t * * Atomically reads the value of @v. */ -static __always_inline int atomic_read(const atomic_t *v) +static __always_inline int arch_atomic_read(const atomic_t *v) { return READ_ONCE((v)->counter); } /** - * atomic_set - set atomic variable + * arch_atomic_set - set atomic variable * @v: pointer of type atomic_t * @i: required value * * Atomically sets the value of @v to @i. */ -static __always_inline void atomic_set(atomic_t *v, int i) +static __always_inline void arch_atomic_set(atomic_t *v, int i) { WRITE_ONCE(v->counter, i); } /** - * atomic_add - add integer to atomic variable + * arch_atomic_add - add integer to atomic variable * @i: integer value to add * @v: pointer of type atomic_t * * Atomically adds @i to @v. */ -static __always_inline void atomic_add(int i, atomic_t *v) +static __always_inline void arch_atomic_add(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "addl %1,%0" : "+m" (v->counter) @@ -53,13 +53,13 @@ static __always_inline void atomic_add(int i, atomic_t *v) } /** - * atomic_sub - subtract integer from atomic variable + * arch_atomic_sub - subtract integer from atomic variable * @i: integer value to subtract * @v: pointer of type atomic_t * * Atomically subtracts @i from @v. */ -static __always_inline void atomic_sub(int i, atomic_t *v) +static __always_inline void arch_atomic_sub(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "subl %1,%0" : "+m" (v->counter) @@ -67,7 +67,7 @@ static __always_inline void atomic_sub(int i, atomic_t *v) } /** - * atomic_sub_and_test - subtract value from variable and test result + * arch_atomic_sub_and_test - subtract value from variable and test result * @i: integer value to subtract * @v: pointer of type atomic_t * @@ -75,63 +75,63 @@ static __always_inline void atomic_sub(int i, atomic_t *v) * true if the result is zero, or false for all * other cases. */ -static __always_inline bool atomic_sub_and_test(int i, atomic_t *v) +static __always_inline bool arch_atomic_sub_and_test(int i, atomic_t *v) { GEN_BINARY_RMWcc(LOCK_PREFIX "subl", v->counter, "er", i, "%0", e); } /** - * atomic_inc - increment atomic variable + * arch_atomic_inc - increment atomic variable * @v: pointer of type atomic_t * * Atomically increments @v by 1. */ -static __always_inline void atomic_inc(atomic_t *v) +static __always_inline void arch_atomic_inc(atomic_t *v) { asm volatile(LOCK_PREFIX "incl %0" : "+m" (v->counter)); } /** - * atomic_dec - decrement atomic variable + * arch_atomic_dec - decrement atomic variable * @v: pointer of type atomic_t * * Atomically decrements @v by 1. */ -static __always_inline void atomic_dec(atomic_t *v) +static __always_inline void arch_atomic_dec(atomic_t *v) { asm volatile(LOCK_PREFIX "decl %0" : "+m" (v->counter)); } /** - * atomic_dec_and_test - decrement and test + * arch_atomic_dec_and_test - decrement and test * @v: pointer of type atomic_t * * Atomically decrements @v by 1 and * returns true if the result is 0, or false for all other * cases. */ -static __always_inline bool atomic_dec_and_test(atomic_t *v) +static __always_inline bool arch_atomic_dec_and_test(atomic_t *v) { GEN_UNARY_RMWcc(LOCK_PREFIX "decl", v->counter, "%0", e); } /** - * atomic_inc_and_test - increment and test + * arch_atomic_inc_and_test - increment and test * @v: pointer of type atomic_t * * Atomically increments @v by 1 * and returns true if the result is zero, or false for all * other cases. */ -static __always_inline bool atomic_inc_and_test(atomic_t *v) +static __always_inline bool arch_atomic_inc_and_test(atomic_t *v) { GEN_UNARY_RMWcc(LOCK_PREFIX "incl", v->counter, "%0", e); } /** - * atomic_add_negative - add and test if negative + * arch_atomic_add_negative - add and test if negative * @i: integer value to add * @v: pointer of type atomic_t * @@ -139,65 +139,65 @@ static __always_inline bool atomic_inc_and_test(atomic_t *v) * if the result is negative, or false when * result is greater than or equal to zero. */ -static __always_inline bool atomic_add_negative(int i, atomic_t *v) +static __always_inline bool arch_atomic_add_negative(int i, atomic_t *v) { GEN_BINARY_RMWcc(LOCK_PREFIX "addl", v->counter, "er", i, "%0", s); } /** - * atomic_add_return - add integer and return + * arch_atomic_add_return - add integer and return * @i: integer value to add * @v: pointer of type atomic_t * * Atomically adds @i to @v and returns @i + @v */ -static __always_inline int atomic_add_return(int i, atomic_t *v) +static __always_inline int arch_atomic_add_return(int i, atomic_t *v) { return i + xadd(&v->counter, i); } /** - * atomic_sub_return - subtract integer and return + * arch_atomic_sub_return - subtract integer and return * @v: pointer of type atomic_t * @i: integer value to subtract * * Atomically subtracts @i from @v and returns @v - @i */ -static __always_inline int atomic_sub_return(int i, atomic_t *v) +static __always_inline int arch_atomic_sub_return(int i, atomic_t *v) { - return atomic_add_return(-i, v); + return arch_atomic_add_return(-i, v); } -#define atomic_inc_return(v) (atomic_add_return(1, v)) -#define atomic_dec_return(v) (atomic_sub_return(1, v)) +#define arch_atomic_inc_return(v) (arch_atomic_add_return(1, v)) +#define arch_atomic_dec_return(v) (arch_atomic_sub_return(1, v)) -static __always_inline int atomic_fetch_add(int i, atomic_t *v) +static __always_inline int arch_atomic_fetch_add(int i, atomic_t *v) { return xadd(&v->counter, i); } -static __always_inline int atomic_fetch_sub(int i, atomic_t *v) +static __always_inline int arch_atomic_fetch_sub(int i, atomic_t *v) { return xadd(&v->counter, -i); } -static __always_inline int atomic_cmpxchg(atomic_t *v, int old, int new) +static __always_inline int arch_atomic_cmpxchg(atomic_t *v, int old, int new) { - return cmpxchg(&v->counter, old, new); + return arch_cmpxchg(&v->counter, old, new); } -#define atomic_try_cmpxchg atomic_try_cmpxchg -static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new) +#define arch_atomic_try_cmpxchg arch_atomic_try_cmpxchg +static __always_inline bool arch_atomic_try_cmpxchg(atomic_t *v, int *old, int new) { return try_cmpxchg(&v->counter, old, new); } -static inline int atomic_xchg(atomic_t *v, int new) +static inline int arch_atomic_xchg(atomic_t *v, int new) { return xchg(&v->counter, new); } -static inline void atomic_and(int i, atomic_t *v) +static inline void arch_atomic_and(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "andl %1,%0" : "+m" (v->counter) @@ -205,16 +205,16 @@ static inline void atomic_and(int i, atomic_t *v) : "memory"); } -static inline int atomic_fetch_and(int i, atomic_t *v) +static inline int arch_atomic_fetch_and(int i, atomic_t *v) { - int val = atomic_read(v); + int val = arch_atomic_read(v); - do { } while (!atomic_try_cmpxchg(v, &val, val & i)); + do { } while (!arch_atomic_try_cmpxchg(v, &val, val & i)); return val; } -static inline void atomic_or(int i, atomic_t *v) +static inline void arch_atomic_or(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "orl %1,%0" : "+m" (v->counter) @@ -222,16 +222,16 @@ static inline void atomic_or(int i, atomic_t *v) : "memory"); } -static inline int atomic_fetch_or(int i, atomic_t *v) +static inline int arch_atomic_fetch_or(int i, atomic_t *v) { - int val = atomic_read(v); + int val = arch_atomic_read(v); - do { } while (!atomic_try_cmpxchg(v, &val, val | i)); + do { } while (!arch_atomic_try_cmpxchg(v, &val, val | i)); return val; } -static inline void atomic_xor(int i, atomic_t *v) +static inline void arch_atomic_xor(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "xorl %1,%0" : "+m" (v->counter) @@ -239,17 +239,17 @@ static inline void atomic_xor(int i, atomic_t *v) : "memory"); } -static inline int atomic_fetch_xor(int i, atomic_t *v) +static inline int arch_atomic_fetch_xor(int i, atomic_t *v) { - int val = atomic_read(v); + int val = arch_atomic_read(v); - do { } while (!atomic_try_cmpxchg(v, &val, val ^ i)); + do { } while (!arch_atomic_try_cmpxchg(v, &val, val ^ i)); return val; } /** - * __atomic_add_unless - add unless the number is already a given value + * __arch_atomic_add_unless - add unless the number is already a given value * @v: pointer of type atomic_t * @a: the amount to add to v... * @u: ...unless v is equal to u. @@ -257,14 +257,14 @@ static inline int atomic_fetch_xor(int i, atomic_t *v) * Atomically adds @a to @v, so long as @v was not already @u. * Returns the old value of @v. */ -static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u) +static __always_inline int __arch_atomic_add_unless(atomic_t *v, int a, int u) { - int c = atomic_read(v); + int c = arch_atomic_read(v); do { if (unlikely(c == u)) break; - } while (!atomic_try_cmpxchg(v, &c, c + a)); + } while (!arch_atomic_try_cmpxchg(v, &c, c + a)); return c; } @@ -275,4 +275,6 @@ static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u) # include <asm/atomic64_64.h> #endif +#include <asm-generic/atomic-instrumented.h> + #endif /* _ASM_X86_ATOMIC_H */ diff --git a/arch/x86/include/asm/atomic64_32.h b/arch/x86/include/asm/atomic64_32.h index 9e206f31ce2a..512377a1c2aa 100644 --- a/arch/x86/include/asm/atomic64_32.h +++ b/arch/x86/include/asm/atomic64_32.h @@ -61,7 +61,7 @@ ATOMIC64_DECL(add_unless); #undef ATOMIC64_EXPORT /** - * atomic64_cmpxchg - cmpxchg atomic64 variable + * arch_atomic64_cmpxchg - cmpxchg atomic64 variable * @v: pointer to type atomic64_t * @o: expected value * @n: new value @@ -70,20 +70,21 @@ ATOMIC64_DECL(add_unless); * the old value. */ -static inline long long atomic64_cmpxchg(atomic64_t *v, long long o, long long n) +static inline long long arch_atomic64_cmpxchg(atomic64_t *v, long long o, + long long n) { - return cmpxchg64(&v->counter, o, n); + return arch_cmpxchg64(&v->counter, o, n); } /** - * atomic64_xchg - xchg atomic64 variable + * arch_atomic64_xchg - xchg atomic64 variable * @v: pointer to type atomic64_t * @n: value to assign * * Atomically xchgs the value of @v to @n and returns * the old value. */ -static inline long long atomic64_xchg(atomic64_t *v, long long n) +static inline long long arch_atomic64_xchg(atomic64_t *v, long long n) { long long o; unsigned high = (unsigned)(n >> 32); @@ -95,13 +96,13 @@ static inline long long atomic64_xchg(atomic64_t *v, long long n) } /** - * atomic64_set - set atomic64 variable + * arch_atomic64_set - set atomic64 variable * @v: pointer to type atomic64_t * @i: value to assign * * Atomically sets the value of @v to @n. */ -static inline void atomic64_set(atomic64_t *v, long long i) +static inline void arch_atomic64_set(atomic64_t *v, long long i) { unsigned high = (unsigned)(i >> 32); unsigned low = (unsigned)i; @@ -111,12 +112,12 @@ static inline void atomic64_set(atomic64_t *v, long long i) } /** - * atomic64_read - read atomic64 variable + * arch_atomic64_read - read atomic64 variable * @v: pointer to type atomic64_t * * Atomically reads the value of @v and returns it. */ -static inline long long atomic64_read(const atomic64_t *v) +static inline long long arch_atomic64_read(const atomic64_t *v) { long long r; alternative_atomic64(read, "=&A" (r), "c" (v) : "memory"); @@ -124,13 +125,13 @@ static inline long long atomic64_read(const atomic64_t *v) } /** - * atomic64_add_return - add and return + * arch_atomic64_add_return - add and return * @i: integer value to add * @v: pointer to type atomic64_t * * Atomically adds @i to @v and returns @i + *@v */ -static inline long long atomic64_add_return(long long i, atomic64_t *v) +static inline long long arch_atomic64_add_return(long long i, atomic64_t *v) { alternative_atomic64(add_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -141,7 +142,7 @@ static inline long long atomic64_add_return(long long i, atomic64_t *v) /* * Other variants with different arithmetic operators: */ -static inline long long atomic64_sub_return(long long i, atomic64_t *v) +static inline long long arch_atomic64_sub_return(long long i, atomic64_t *v) { alternative_atomic64(sub_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -149,7 +150,7 @@ static inline long long atomic64_sub_return(long long i, atomic64_t *v) return i; } -static inline long long atomic64_inc_return(atomic64_t *v) +static inline long long arch_atomic64_inc_return(atomic64_t *v) { long long a; alternative_atomic64(inc_return, "=&A" (a), @@ -157,7 +158,7 @@ static inline long long atomic64_inc_return(atomic64_t *v) return a; } -static inline long long atomic64_dec_return(atomic64_t *v) +static inline long long arch_atomic64_dec_return(atomic64_t *v) { long long a; alternative_atomic64(dec_return, "=&A" (a), @@ -166,13 +167,13 @@ static inline long long atomic64_dec_return(atomic64_t *v) } /** - * atomic64_add - add integer to atomic64 variable + * arch_atomic64_add - add integer to atomic64 variable * @i: integer value to add * @v: pointer to type atomic64_t * * Atomically adds @i to @v. */ -static inline long long atomic64_add(long long i, atomic64_t *v) +static inline long long arch_atomic64_add(long long i, atomic64_t *v) { __alternative_atomic64(add, add_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -181,13 +182,13 @@ static inline long long atomic64_add(long long i, atomic64_t *v) } /** - * atomic64_sub - subtract the atomic64 variable + * arch_atomic64_sub - subtract the atomic64 variable * @i: integer value to subtract * @v: pointer to type atomic64_t * * Atomically subtracts @i from @v. */ -static inline long long atomic64_sub(long long i, atomic64_t *v) +static inline long long arch_atomic64_sub(long long i, atomic64_t *v) { __alternative_atomic64(sub, sub_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -196,7 +197,7 @@ static inline long long atomic64_sub(long long i, atomic64_t *v) } /** - * atomic64_sub_and_test - subtract value from variable and test result + * arch_atomic64_sub_and_test - subtract value from variable and test result * @i: integer value to subtract * @v: pointer to type atomic64_t * @@ -204,46 +205,46 @@ static inline long long atomic64_sub(long long i, atomic64_t *v) * true if the result is zero, or false for all * other cases. */ -static inline int atomic64_sub_and_test(long long i, atomic64_t *v) +static inline int arch_atomic64_sub_and_test(long long i, atomic64_t *v) { - return atomic64_sub_return(i, v) == 0; + return arch_atomic64_sub_return(i, v) == 0; } /** - * atomic64_inc - increment atomic64 variable + * arch_atomic64_inc - increment atomic64 variable * @v: pointer to type atomic64_t * * Atomically increments @v by 1. */ -static inline void atomic64_inc(atomic64_t *v) +static inline void arch_atomic64_inc(atomic64_t *v) { __alternative_atomic64(inc, inc_return, /* no output */, "S" (v) : "memory", "eax", "ecx", "edx"); } /** - * atomic64_dec - decrement atomic64 variable + * arch_atomic64_dec - decrement atomic64 variable * @v: pointer to type atomic64_t * * Atomically decrements @v by 1. */ -static inline void atomic64_dec(atomic64_t *v) +static inline void arch_atomic64_dec(atomic64_t *v) { __alternative_atomic64(dec, dec_return, /* no output */, "S" (v) : "memory", "eax", "ecx", "edx"); } /** - * atomic64_dec_and_test - decrement and test + * arch_atomic64_dec_and_test - decrement and test * @v: pointer to type atomic64_t * * Atomically decrements @v by 1 and * returns true if the result is 0, or false for all other * cases. */ -static inline int atomic64_dec_and_test(atomic64_t *v) +static inline int arch_atomic64_dec_and_test(atomic64_t *v) { - return atomic64_dec_return(v) == 0; + return arch_atomic64_dec_return(v) == 0; } /** @@ -254,13 +255,13 @@ static inline int atomic64_dec_and_test(atomic64_t *v) * and returns true if the result is zero, or false for all * other cases. */ -static inline int atomic64_inc_and_test(atomic64_t *v) +static inline int arch_atomic64_inc_and_test(atomic64_t *v) { - return atomic64_inc_return(v) == 0; + return arch_atomic64_inc_return(v) == 0; } /** - * atomic64_add_negative - add and test if negative + * arch_atomic64_add_negative - add and test if negative * @i: integer value to add * @v: pointer to type atomic64_t * @@ -268,13 +269,13 @@ static inline int atomic64_inc_and_test(atomic64_t *v) * if the result is negative, or false when * result is greater than or equal to zero. */ -static inline int atomic64_add_negative(long long i, atomic64_t *v) +static inline int arch_atomic64_add_negative(long long i, atomic64_t *v) { - return atomic64_add_return(i, v) < 0; + return arch_atomic64_add_return(i, v) < 0; } /** - * atomic64_add_unless - add unless the number is a given value + * arch_atomic64_add_unless - add unless the number is a given value * @v: pointer of type atomic64_t * @a: the amount to add to v... * @u: ...unless v is equal to u. @@ -282,7 +283,8 @@ static inline int atomic64_add_negative(long long i, atomic64_t *v) * Atomically adds @a to @v, so long as it was not @u. * Returns non-zero if the add was done, zero otherwise. */ -static inline int atomic64_add_unless(atomic64_t *v, long long a, long long u) +static inline int arch_atomic64_add_unless(atomic64_t *v, long long a, + long long u) { unsigned low = (unsigned)u; unsigned high = (unsigned)(u >> 32); @@ -293,7 +295,7 @@ static inline int atomic64_add_unless(atomic64_t *v, long long a, long long u) } -static inline int atomic64_inc_not_zero(atomic64_t *v) +static inline int arch_atomic64_inc_not_zero(atomic64_t *v) { int r; alternative_atomic64(inc_not_zero, "=&a" (r), @@ -301,7 +303,7 @@ static inline int atomic64_inc_not_zero(atomic64_t *v) return r; } -static inline long long atomic64_dec_if_positive(atomic64_t *v) +static inline long long arch_atomic64_dec_if_positive(atomic64_t *v) { long long r; alternative_atomic64(dec_if_positive, "=&A" (r), @@ -312,70 +314,70 @@ static inline long long atomic64_dec_if_positive(atomic64_t *v) #undef alternative_atomic64 #undef __alternative_atomic64 -static inline void atomic64_and(long long i, atomic64_t *v) +static inline void arch_atomic64_and(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c & i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c & i)) != c) c = old; } -static inline long long atomic64_fetch_and(long long i, atomic64_t *v) +static inline long long arch_atomic64_fetch_and(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c & i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c & i)) != c) c = old; return old; } -static inline void atomic64_or(long long i, atomic64_t *v) +static inline void arch_atomic64_or(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c | i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c | i)) != c) c = old; } -static inline long long atomic64_fetch_or(long long i, atomic64_t *v) +static inline long long arch_atomic64_fetch_or(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c | i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c | i)) != c) c = old; return old; } -static inline void atomic64_xor(long long i, atomic64_t *v) +static inline void arch_atomic64_xor(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c ^ i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c ^ i)) != c) c = old; } -static inline long long atomic64_fetch_xor(long long i, atomic64_t *v) +static inline long long arch_atomic64_fetch_xor(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c ^ i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c ^ i)) != c) c = old; return old; } -static inline long long atomic64_fetch_add(long long i, atomic64_t *v) +static inline long long arch_atomic64_fetch_add(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c + i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c + i)) != c) c = old; return old; } -#define atomic64_fetch_sub(i, v) atomic64_fetch_add(-(i), (v)) +#define arch_atomic64_fetch_sub(i, v) arch_atomic64_fetch_add(-(i), (v)) #endif /* _ASM_X86_ATOMIC64_32_H */ diff --git a/arch/x86/include/asm/atomic64_64.h b/arch/x86/include/asm/atomic64_64.h index 5d9de36a2f04..5883b109d053 100644 --- a/arch/x86/include/asm/atomic64_64.h +++ b/arch/x86/include/asm/atomic64_64.h @@ -10,37 +10,37 @@ #define ATOMIC64_INIT(i) { (i) } /** - * atomic64_read - read atomic64 variable + * arch_atomic64_read - read atomic64 variable * @v: pointer of type atomic64_t * * Atomically reads the value of @v. * Doesn't imply a read memory barrier. */ -static inline long atomic64_read(const atomic64_t *v) +static inline long arch_atomic64_read(const atomic64_t *v) { return READ_ONCE((v)->counter); } /** - * atomic64_set - set atomic64 variable + * arch_atomic64_set - set atomic64 variable * @v: pointer to type atomic64_t * @i: required value * * Atomically sets the value of @v to @i. */ -static inline void atomic64_set(atomic64_t *v, long i) +static inline void arch_atomic64_set(atomic64_t *v, long i) { WRITE_ONCE(v->counter, i); } /** - * atomic64_add - add integer to atomic64 variable + * arch_atomic64_add - add integer to atomic64 variable * @i: integer value to add * @v: pointer to type atomic64_t * * Atomically adds @i to @v. */ -static __always_inline void atomic64_add(long i, atomic64_t *v) +static __always_inline void arch_atomic64_add(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "addq %1,%0" : "=m" (v->counter) @@ -48,13 +48,13 @@ static __always_inline void atomic64_add(long i, atomic64_t *v) } /** - * atomic64_sub - subtract the atomic64 variable + * arch_atomic64_sub - subtract the atomic64 variable * @i: integer value to subtract * @v: pointer to type atomic64_t * * Atomically subtracts @i from @v. */ -static inline void atomic64_sub(long i, atomic64_t *v) +static inline void arch_atomic64_sub(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "subq %1,%0" : "=m" (v->counter) @@ -62,7 +62,7 @@ static inline void atomic64_sub(long i, atomic64_t *v) } /** - * atomic64_sub_and_test - subtract value from variable and test result + * arch_atomic64_sub_and_test - subtract value from variable and test result * @i: integer value to subtract * @v: pointer to type atomic64_t * @@ -70,18 +70,18 @@ static inline void atomic64_sub(long i, atomic64_t *v) * true if the result is zero, or false for all * other cases. */ -static inline bool atomic64_sub_and_test(long i, atomic64_t *v) +static inline bool arch_atomic64_sub_and_test(long i, atomic64_t *v) { GEN_BINARY_RMWcc(LOCK_PREFIX "subq", v->counter, "er", i, "%0", e); } /** - * atomic64_inc - increment atomic64 variable + * arch_atomic64_inc - increment atomic64 variable * @v: pointer to type atomic64_t * * Atomically increments @v by 1. */ -static __always_inline void atomic64_inc(atomic64_t *v) +static __always_inline void arch_atomic64_inc(atomic64_t *v) { asm volatile(LOCK_PREFIX "incq %0" : "=m" (v->counter) @@ -89,12 +89,12 @@ static __always_inline void atomic64_inc(atomic64_t *v) } /** - * atomic64_dec - decrement atomic64 variable + * arch_atomic64_dec - decrement atomic64 variable * @v: pointer to type atomic64_t * * Atomically decrements @v by 1. */ -static __always_inline void atomic64_dec(atomic64_t *v) +static __always_inline void arch_atomic64_dec(atomic64_t *v) { asm volatile(LOCK_PREFIX "decq %0" : "=m" (v->counter) @@ -102,33 +102,33 @@ static __always_inline void atomic64_dec(atomic64_t *v) } /** - * atomic64_dec_and_test - decrement and test + * arch_atomic64_dec_and_test - decrement and test * @v: pointer to type atomic64_t * * Atomically decrements @v by 1 and * returns true if the result is 0, or false for all other * cases. */ -static inline bool atomic64_dec_and_test(atomic64_t *v) +static inline bool arch_atomic64_dec_and_test(atomic64_t *v) { GEN_UNARY_RMWcc(LOCK_PREFIX "decq", v->counter, "%0", e); } /** - * atomic64_inc_and_test - increment and test + * arch_atomic64_inc_and_test - increment and test * @v: pointer to type atomic64_t * * Atomically increments @v by 1 * and returns true if the result is zero, or false for all * other cases. */ -static inline bool atomic64_inc_and_test(atomic64_t *v) +static inline bool arch_atomic64_inc_and_test(atomic64_t *v) { GEN_UNARY_RMWcc(LOCK_PREFIX "incq", v->counter, "%0", e); } /** - * atomic64_add_negative - add and test if negative + * arch_atomic64_add_negative - add and test if negative * @i: integer value to add * @v: pointer to type atomic64_t * @@ -136,59 +136,59 @@ static inline bool atomic64_inc_and_test(atomic64_t *v) * if the result is negative, or false when * result is greater than or equal to zero. */ -static inline bool atomic64_add_negative(long i, atomic64_t *v) +static inline bool arch_atomic64_add_negative(long i, atomic64_t *v) { GEN_BINARY_RMWcc(LOCK_PREFIX "addq", v->counter, "er", i, "%0", s); } /** - * atomic64_add_return - add and return + * arch_atomic64_add_return - add and return * @i: integer value to add * @v: pointer to type atomic64_t * * Atomically adds @i to @v and returns @i + @v */ -static __always_inline long atomic64_add_return(long i, atomic64_t *v) +static __always_inline long arch_atomic64_add_return(long i, atomic64_t *v) { return i + xadd(&v->counter, i); } -static inline long atomic64_sub_return(long i, atomic64_t *v) +static inline long arch_atomic64_sub_return(long i, atomic64_t *v) { - return atomic64_add_return(-i, v); + return arch_atomic64_add_return(-i, v); } -static inline long atomic64_fetch_add(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_add(long i, atomic64_t *v) { return xadd(&v->counter, i); } -static inline long atomic64_fetch_sub(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_sub(long i, atomic64_t *v) { return xadd(&v->counter, -i); } -#define atomic64_inc_return(v) (atomic64_add_return(1, (v))) -#define atomic64_dec_return(v) (atomic64_sub_return(1, (v))) +#define arch_atomic64_inc_return(v) (arch_atomic64_add_return(1, (v))) +#define arch_atomic64_dec_return(v) (arch_atomic64_sub_return(1, (v))) -static inline long atomic64_cmpxchg(atomic64_t *v, long old, long new) +static inline long arch_atomic64_cmpxchg(atomic64_t *v, long old, long new) { - return cmpxchg(&v->counter, old, new); + return arch_cmpxchg(&v->counter, old, new); } -#define atomic64_try_cmpxchg atomic64_try_cmpxchg -static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, long new) +#define arch_atomic64_try_cmpxchg arch_atomic64_try_cmpxchg +static __always_inline bool arch_atomic64_try_cmpxchg(atomic64_t *v, s64 *old, long new) { return try_cmpxchg(&v->counter, old, new); } -static inline long atomic64_xchg(atomic64_t *v, long new) +static inline long arch_atomic64_xchg(atomic64_t *v, long new) { return xchg(&v->counter, new); } /** - * atomic64_add_unless - add unless the number is a given value + * arch_atomic64_add_unless - add unless the number is a given value * @v: pointer of type atomic64_t * @a: the amount to add to v... * @u: ...unless v is equal to u. @@ -196,37 +196,37 @@ static inline long atomic64_xchg(atomic64_t *v, long new) * Atomically adds @a to @v, so long as it was not @u. * Returns the old value of @v. */ -static inline bool atomic64_add_unless(atomic64_t *v, long a, long u) +static inline bool arch_atomic64_add_unless(atomic64_t *v, long a, long u) { - s64 c = atomic64_read(v); + s64 c = arch_atomic64_read(v); do { if (unlikely(c == u)) return false; - } while (!atomic64_try_cmpxchg(v, &c, c + a)); + } while (!arch_atomic64_try_cmpxchg(v, &c, c + a)); return true; } -#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0) +#define arch_atomic64_inc_not_zero(v) arch_atomic64_add_unless((v), 1, 0) /* - * atomic64_dec_if_positive - decrement by 1 if old value positive + * arch_atomic64_dec_if_positive - decrement by 1 if old value positive * @v: pointer of type atomic_t * * The function returns the old value of *v minus 1, even if * the atomic variable, v, was not decremented. */ -static inline long atomic64_dec_if_positive(atomic64_t *v) +static inline long arch_atomic64_dec_if_positive(atomic64_t *v) { - s64 dec, c = atomic64_read(v); + s64 dec, c = arch_atomic64_read(v); do { dec = c - 1; if (unlikely(dec < 0)) break; - } while (!atomic64_try_cmpxchg(v, &c, dec)); + } while (!arch_atomic64_try_cmpxchg(v, &c, dec)); return dec; } -static inline void atomic64_and(long i, atomic64_t *v) +static inline void arch_atomic64_and(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "andq %1,%0" : "+m" (v->counter) @@ -234,16 +234,16 @@ static inline void atomic64_and(long i, atomic64_t *v) : "memory"); } -static inline long atomic64_fetch_and(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_and(long i, atomic64_t *v) { - s64 val = atomic64_read(v); + s64 val = arch_atomic64_read(v); do { - } while (!atomic64_try_cmpxchg(v, &val, val & i)); + } while (!arch_atomic64_try_cmpxchg(v, &val, val & i)); return val; } -static inline void atomic64_or(long i, atomic64_t *v) +static inline void arch_atomic64_or(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "orq %1,%0" : "+m" (v->counter) @@ -251,16 +251,16 @@ static inline void atomic64_or(long i, atomic64_t *v) : "memory"); } -static inline long atomic64_fetch_or(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_or(long i, atomic64_t *v) { - s64 val = atomic64_read(v); + s64 val = arch_atomic64_read(v); do { - } while (!atomic64_try_cmpxchg(v, &val, val | i)); + } while (!arch_atomic64_try_cmpxchg(v, &val, val | i)); return val; } -static inline void atomic64_xor(long i, atomic64_t *v) +static inline void arch_atomic64_xor(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "xorq %1,%0" : "+m" (v->counter) @@ -268,12 +268,12 @@ static inline void atomic64_xor(long i, atomic64_t *v) : "memory"); } -static inline long atomic64_fetch_xor(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_xor(long i, atomic64_t *v) { - s64 val = atomic64_read(v); + s64 val = arch_atomic64_read(v); do { - } while (!atomic64_try_cmpxchg(v, &val, val ^ i)); + } while (!arch_atomic64_try_cmpxchg(v, &val, val ^ i)); return val; } diff --git a/arch/x86/include/asm/cmpxchg.h b/arch/x86/include/asm/cmpxchg.h index b5069e802d5c..9e2faa85eb02 100644 --- a/arch/x86/include/asm/cmpxchg.h +++ b/arch/x86/include/asm/cmpxchg.h @@ -144,13 +144,13 @@ extern void __add_wrong_size(void) # include <asm/cmpxchg_64.h> #endif -#define cmpxchg(ptr, old, new) \ +#define arch_cmpxchg(ptr, old, new) \ __cmpxchg(ptr, old, new, sizeof(*(ptr))) -#define sync_cmpxchg(ptr, old, new) \ +#define arch_sync_cmpxchg(ptr, old, new) \ __sync_cmpxchg(ptr, old, new, sizeof(*(ptr))) -#define cmpxchg_local(ptr, old, new) \ +#define arch_cmpxchg_local(ptr, old, new) \ __cmpxchg_local(ptr, old, new, sizeof(*(ptr))) @@ -220,7 +220,7 @@ extern void __add_wrong_size(void) #define __try_cmpxchg(ptr, pold, new, size) \ __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX) -#define try_cmpxchg(ptr, pold, new) \ +#define try_cmpxchg(ptr, pold, new) \ __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr))) /* @@ -249,10 +249,10 @@ extern void __add_wrong_size(void) __ret; \ }) -#define cmpxchg_double(p1, p2, o1, o2, n1, n2) \ +#define arch_cmpxchg_double(p1, p2, o1, o2, n1, n2) \ __cmpxchg_double(LOCK_PREFIX, p1, p2, o1, o2, n1, n2) -#define cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \ +#define arch_cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \ __cmpxchg_double(, p1, p2, o1, o2, n1, n2) #endif /* ASM_X86_CMPXCHG_H */ diff --git a/arch/x86/include/asm/cmpxchg_32.h b/arch/x86/include/asm/cmpxchg_32.h index e4959d023af8..d897291d2bf9 100644 --- a/arch/x86/include/asm/cmpxchg_32.h +++ b/arch/x86/include/asm/cmpxchg_32.h @@ -35,10 +35,10 @@ static inline void set_64bit(volatile u64 *ptr, u64 value) } #ifdef CONFIG_X86_CMPXCHG64 -#define cmpxchg64(ptr, o, n) \ +#define arch_cmpxchg64(ptr, o, n) \ ((__typeof__(*(ptr)))__cmpxchg64((ptr), (unsigned long long)(o), \ (unsigned long long)(n))) -#define cmpxchg64_local(ptr, o, n) \ +#define arch_cmpxchg64_local(ptr, o, n) \ ((__typeof__(*(ptr)))__cmpxchg64_local((ptr), (unsigned long long)(o), \ (unsigned long long)(n))) #endif @@ -75,7 +75,7 @@ static inline u64 __cmpxchg64_local(volatile u64 *ptr, u64 old, u64 new) * to simulate the cmpxchg8b on the 80386 and 80486 CPU. */ -#define cmpxchg64(ptr, o, n) \ +#define arch_cmpxchg64(ptr, o, n) \ ({ \ __typeof__(*(ptr)) __ret; \ __typeof__(*(ptr)) __old = (o); \ @@ -92,7 +92,7 @@ static inline u64 __cmpxchg64_local(volatile u64 *ptr, u64 old, u64 new) __ret; }) -#define cmpxchg64_local(ptr, o, n) \ +#define arch_cmpxchg64_local(ptr, o, n) \ ({ \ __typeof__(*(ptr)) __ret; \ __typeof__(*(ptr)) __old = (o); \ diff --git a/arch/x86/include/asm/cmpxchg_64.h b/arch/x86/include/asm/cmpxchg_64.h index caa23a34c963..fafaebacca2d 100644 --- a/arch/x86/include/asm/cmpxchg_64.h +++ b/arch/x86/include/asm/cmpxchg_64.h @@ -6,13 +6,13 @@ static inline void set_64bit(volatile u64 *ptr, u64 val) *ptr = val; } -#define cmpxchg64(ptr, o, n) \ +#define arch_cmpxchg64(ptr, o, n) \ ({ \ BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ cmpxchg((ptr), (o), (n)); \ }) -#define cmpxchg64_local(ptr, o, n) \ +#define arch_cmpxchg64_local(ptr, o, n) \ ({ \ BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ cmpxchg_local((ptr), (o), (n)); \ -- 2.13.1.611.g7e3b11ae1-goog -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply related [flat|nested] 63+ messages in thread
* Re: [PATCH v5 1/4] x86: switch atomic.h to use atomic-instrumented.h 2017-06-22 14:14 ` Dmitry Vyukov @ 2017-06-22 21:14 ` Andrew Morton -1 siblings, 0 replies; 63+ messages in thread From: Andrew Morton @ 2017-06-22 21:14 UTC (permalink / raw) To: Dmitry Vyukov Cc: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin, kasan-dev, x86, linux-kernel, linux-mm On Thu, 22 Jun 2017 16:14:16 +0200 Dmitry Vyukov <dvyukov@google.com> wrote: > Add arch_ prefix to all atomic operations and include > <asm-generic/atomic-instrumented.h>. This will allow > to add KASAN instrumentation to all atomic ops. This gets a large number of (simple) rejects when applied to linux-next. Can you please redo against -next? ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH v5 1/4] x86: switch atomic.h to use atomic-instrumented.h @ 2017-06-22 21:14 ` Andrew Morton 0 siblings, 0 replies; 63+ messages in thread From: Andrew Morton @ 2017-06-22 21:14 UTC (permalink / raw) To: Dmitry Vyukov Cc: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin, kasan-dev, x86, linux-kernel, linux-mm On Thu, 22 Jun 2017 16:14:16 +0200 Dmitry Vyukov <dvyukov@google.com> wrote: > Add arch_ prefix to all atomic operations and include > <asm-generic/atomic-instrumented.h>. This will allow > to add KASAN instrumentation to all atomic ops. This gets a large number of (simple) rejects when applied to linux-next. Can you please redo against -next? -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH v5 1/4] x86: switch atomic.h to use atomic-instrumented.h 2017-06-22 21:14 ` Andrew Morton @ 2017-06-23 8:23 ` Dmitry Vyukov -1 siblings, 0 replies; 63+ messages in thread From: Dmitry Vyukov @ 2017-06-23 8:23 UTC (permalink / raw) To: Andrew Morton Cc: Mark Rutland, Peter Zijlstra, Ingo Molnar, Will Deacon, H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML, linux-mm On Thu, Jun 22, 2017 at 11:14 PM, Andrew Morton <akpm@linux-foundation.org> wrote: > On Thu, 22 Jun 2017 16:14:16 +0200 Dmitry Vyukov <dvyukov@google.com> wrote: > >> Add arch_ prefix to all atomic operations and include >> <asm-generic/atomic-instrumented.h>. This will allow >> to add KASAN instrumentation to all atomic ops. > > This gets a large number of (simple) rejects when applied to > linux-next. Can you please redo against -next? This is based on tip/locking tree. Ingo already took a part of these series. The plan is that he takes the rest, and this applies on tip/locking without conflicts. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH v5 1/4] x86: switch atomic.h to use atomic-instrumented.h @ 2017-06-23 8:23 ` Dmitry Vyukov 0 siblings, 0 replies; 63+ messages in thread From: Dmitry Vyukov @ 2017-06-23 8:23 UTC (permalink / raw) To: Andrew Morton Cc: Mark Rutland, Peter Zijlstra, Ingo Molnar, Will Deacon, H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML, linux-mm On Thu, Jun 22, 2017 at 11:14 PM, Andrew Morton <akpm@linux-foundation.org> wrote: > On Thu, 22 Jun 2017 16:14:16 +0200 Dmitry Vyukov <dvyukov@google.com> wrote: > >> Add arch_ prefix to all atomic operations and include >> <asm-generic/atomic-instrumented.h>. This will allow >> to add KASAN instrumentation to all atomic ops. > > This gets a large number of (simple) rejects when applied to > linux-next. Can you please redo against -next? This is based on tip/locking tree. Ingo already took a part of these series. The plan is that he takes the rest, and this applies on tip/locking without conflicts. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH v5 1/4] x86: switch atomic.h to use atomic-instrumented.h 2017-06-23 8:23 ` Dmitry Vyukov @ 2017-06-23 8:54 ` Ingo Molnar -1 siblings, 0 replies; 63+ messages in thread From: Ingo Molnar @ 2017-06-23 8:54 UTC (permalink / raw) To: Dmitry Vyukov Cc: Andrew Morton, Mark Rutland, Peter Zijlstra, Ingo Molnar, Will Deacon, H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML, linux-mm * Dmitry Vyukov <dvyukov@google.com> wrote: > On Thu, Jun 22, 2017 at 11:14 PM, Andrew Morton > <akpm@linux-foundation.org> wrote: > > On Thu, 22 Jun 2017 16:14:16 +0200 Dmitry Vyukov <dvyukov@google.com> wrote: > > > >> Add arch_ prefix to all atomic operations and include > >> <asm-generic/atomic-instrumented.h>. This will allow > >> to add KASAN instrumentation to all atomic ops. > > > > This gets a large number of (simple) rejects when applied to > > linux-next. Can you please redo against -next? > > > This is based on tip/locking tree. Ingo already took a part of these > series. The plan is that he takes the rest, and this applies on > tip/locking without conflicts. Yeah, so I've taken the rest as well, it all looks very clean now. Should show up in the next -next, if it passes my (arguably limited) testing. Andrew, is this workflow fine with you? You usually take KASAN patches, but I was unhappy with the atomics instrumention of the earlier patches, and ended up reviewing the followup variants, and felt that if I hinder a patchset I might as well test and apply it once I'm happy with them! ;-) Should be a special exception for this series only. Thanks, Ingo ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH v5 1/4] x86: switch atomic.h to use atomic-instrumented.h @ 2017-06-23 8:54 ` Ingo Molnar 0 siblings, 0 replies; 63+ messages in thread From: Ingo Molnar @ 2017-06-23 8:54 UTC (permalink / raw) To: Dmitry Vyukov Cc: Andrew Morton, Mark Rutland, Peter Zijlstra, Ingo Molnar, Will Deacon, H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML, linux-mm * Dmitry Vyukov <dvyukov@google.com> wrote: > On Thu, Jun 22, 2017 at 11:14 PM, Andrew Morton > <akpm@linux-foundation.org> wrote: > > On Thu, 22 Jun 2017 16:14:16 +0200 Dmitry Vyukov <dvyukov@google.com> wrote: > > > >> Add arch_ prefix to all atomic operations and include > >> <asm-generic/atomic-instrumented.h>. This will allow > >> to add KASAN instrumentation to all atomic ops. > > > > This gets a large number of (simple) rejects when applied to > > linux-next. Can you please redo against -next? > > > This is based on tip/locking tree. Ingo already took a part of these > series. The plan is that he takes the rest, and this applies on > tip/locking without conflicts. Yeah, so I've taken the rest as well, it all looks very clean now. Should show up in the next -next, if it passes my (arguably limited) testing. Andrew, is this workflow fine with you? You usually take KASAN patches, but I was unhappy with the atomics instrumention of the earlier patches, and ended up reviewing the followup variants, and felt that if I hinder a patchset I might as well test and apply it once I'm happy with them! ;-) Should be a special exception for this series only. Thanks, Ingo -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH v5 1/4] x86: switch atomic.h to use atomic-instrumented.h 2017-06-23 8:54 ` Ingo Molnar @ 2017-06-23 19:00 ` Andrew Morton -1 siblings, 0 replies; 63+ messages in thread From: Andrew Morton @ 2017-06-23 19:00 UTC (permalink / raw) To: Ingo Molnar Cc: Dmitry Vyukov, Mark Rutland, Peter Zijlstra, Ingo Molnar, Will Deacon, H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML, linux-mm On Fri, 23 Jun 2017 10:54:02 +0200 Ingo Molnar <mingo@kernel.org> wrote: > > * Dmitry Vyukov <dvyukov@google.com> wrote: > > > On Thu, Jun 22, 2017 at 11:14 PM, Andrew Morton > > <akpm@linux-foundation.org> wrote: > > > On Thu, 22 Jun 2017 16:14:16 +0200 Dmitry Vyukov <dvyukov@google.com> wrote: > > > > > >> Add arch_ prefix to all atomic operations and include > > >> <asm-generic/atomic-instrumented.h>. This will allow > > >> to add KASAN instrumentation to all atomic ops. > > > > > > This gets a large number of (simple) rejects when applied to > > > linux-next. Can you please redo against -next? > > > > > > This is based on tip/locking tree. Ingo already took a part of these > > series. The plan is that he takes the rest, and this applies on > > tip/locking without conflicts. > > Yeah, so I've taken the rest as well, it all looks very clean now. Should show up > in the next -next, if it passes my (arguably limited) testing. > > Andrew, is this workflow fine with you? You usually take KASAN patches, but I was > unhappy with the atomics instrumention of the earlier patches, and ended up > reviewing the followup variants, and felt that if I hinder a patchset I might as > well test and apply it once I'm happy with them! ;-) > Sure.. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH v5 1/4] x86: switch atomic.h to use atomic-instrumented.h @ 2017-06-23 19:00 ` Andrew Morton 0 siblings, 0 replies; 63+ messages in thread From: Andrew Morton @ 2017-06-23 19:00 UTC (permalink / raw) To: Ingo Molnar Cc: Dmitry Vyukov, Mark Rutland, Peter Zijlstra, Ingo Molnar, Will Deacon, H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML, linux-mm On Fri, 23 Jun 2017 10:54:02 +0200 Ingo Molnar <mingo@kernel.org> wrote: > > * Dmitry Vyukov <dvyukov@google.com> wrote: > > > On Thu, Jun 22, 2017 at 11:14 PM, Andrew Morton > > <akpm@linux-foundation.org> wrote: > > > On Thu, 22 Jun 2017 16:14:16 +0200 Dmitry Vyukov <dvyukov@google.com> wrote: > > > > > >> Add arch_ prefix to all atomic operations and include > > >> <asm-generic/atomic-instrumented.h>. This will allow > > >> to add KASAN instrumentation to all atomic ops. > > > > > > This gets a large number of (simple) rejects when applied to > > > linux-next. Can you please redo against -next? > > > > > > This is based on tip/locking tree. Ingo already took a part of these > > series. The plan is that he takes the rest, and this applies on > > tip/locking without conflicts. > > Yeah, so I've taken the rest as well, it all looks very clean now. Should show up > in the next -next, if it passes my (arguably limited) testing. > > Andrew, is this workflow fine with you? You usually take KASAN patches, but I was > unhappy with the atomics instrumention of the earlier patches, and ended up > reviewing the followup variants, and felt that if I hinder a patchset I might as > well test and apply it once I'm happy with them! ;-) > Sure.. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 63+ messages in thread
* [tip:locking/core] locking/atomic/x86: Switch atomic.h to use atomic-instrumented.h 2017-06-22 14:14 ` Dmitry Vyukov (?) (?) @ 2017-06-23 13:59 ` tip-bot for Dmitry Vyukov -1 siblings, 0 replies; 63+ messages in thread From: tip-bot for Dmitry Vyukov @ 2017-06-23 13:59 UTC (permalink / raw) To: linux-tip-commits Cc: akpm, torvalds, aryabinin, mingo, will.deacon, hpa, peterz, mark.rutland, tglx, linux-kernel, dvyukov Commit-ID: 1a870f0aa88faf2e5c72303c89a58ebb8638270a Gitweb: http://git.kernel.org/tip/1a870f0aa88faf2e5c72303c89a58ebb8638270a Author: Dmitry Vyukov <dvyukov@google.com> AuthorDate: Thu, 22 Jun 2017 16:14:16 +0200 Committer: Ingo Molnar <mingo@kernel.org> CommitDate: Fri, 23 Jun 2017 10:50:19 +0200 locking/atomic/x86: Switch atomic.h to use atomic-instrumented.h Add arch_ prefix to all atomic operations and include <asm-generic/atomic-instrumented.h>. This will allow to add KASAN instrumentation to all atomic ops. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Reviewed-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/ff85407a7476ac41bfbdd46a35a93b8f57fa4b1e.1498140838.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org> --- arch/x86/include/asm/atomic.h | 102 ++++++++++++++++++----------------- arch/x86/include/asm/atomic64_32.h | 106 ++++++++++++++++++------------------ arch/x86/include/asm/atomic64_64.h | 108 ++++++++++++++++++------------------- arch/x86/include/asm/cmpxchg.h | 12 ++--- arch/x86/include/asm/cmpxchg_32.h | 8 +-- arch/x86/include/asm/cmpxchg_64.h | 4 +- 6 files changed, 172 insertions(+), 168 deletions(-) diff --git a/arch/x86/include/asm/atomic.h b/arch/x86/include/asm/atomic.h index 0874ebd..03dd2a6 100644 --- a/arch/x86/include/asm/atomic.h +++ b/arch/x86/include/asm/atomic.h @@ -16,36 +16,36 @@ #define ATOMIC_INIT(i) { (i) } /** - * atomic_read - read atomic variable + * arch_atomic_read - read atomic variable * @v: pointer of type atomic_t * * Atomically reads the value of @v. */ -static __always_inline int atomic_read(const atomic_t *v) +static __always_inline int arch_atomic_read(const atomic_t *v) { return READ_ONCE((v)->counter); } /** - * atomic_set - set atomic variable + * arch_atomic_set - set atomic variable * @v: pointer of type atomic_t * @i: required value * * Atomically sets the value of @v to @i. */ -static __always_inline void atomic_set(atomic_t *v, int i) +static __always_inline void arch_atomic_set(atomic_t *v, int i) { WRITE_ONCE(v->counter, i); } /** - * atomic_add - add integer to atomic variable + * arch_atomic_add - add integer to atomic variable * @i: integer value to add * @v: pointer of type atomic_t * * Atomically adds @i to @v. */ -static __always_inline void atomic_add(int i, atomic_t *v) +static __always_inline void arch_atomic_add(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "addl %1,%0" : "+m" (v->counter) @@ -53,13 +53,13 @@ static __always_inline void atomic_add(int i, atomic_t *v) } /** - * atomic_sub - subtract integer from atomic variable + * arch_atomic_sub - subtract integer from atomic variable * @i: integer value to subtract * @v: pointer of type atomic_t * * Atomically subtracts @i from @v. */ -static __always_inline void atomic_sub(int i, atomic_t *v) +static __always_inline void arch_atomic_sub(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "subl %1,%0" : "+m" (v->counter) @@ -67,7 +67,7 @@ static __always_inline void atomic_sub(int i, atomic_t *v) } /** - * atomic_sub_and_test - subtract value from variable and test result + * arch_atomic_sub_and_test - subtract value from variable and test result * @i: integer value to subtract * @v: pointer of type atomic_t * @@ -75,63 +75,63 @@ static __always_inline void atomic_sub(int i, atomic_t *v) * true if the result is zero, or false for all * other cases. */ -static __always_inline bool atomic_sub_and_test(int i, atomic_t *v) +static __always_inline bool arch_atomic_sub_and_test(int i, atomic_t *v) { GEN_BINARY_RMWcc(LOCK_PREFIX "subl", v->counter, "er", i, "%0", e); } /** - * atomic_inc - increment atomic variable + * arch_atomic_inc - increment atomic variable * @v: pointer of type atomic_t * * Atomically increments @v by 1. */ -static __always_inline void atomic_inc(atomic_t *v) +static __always_inline void arch_atomic_inc(atomic_t *v) { asm volatile(LOCK_PREFIX "incl %0" : "+m" (v->counter)); } /** - * atomic_dec - decrement atomic variable + * arch_atomic_dec - decrement atomic variable * @v: pointer of type atomic_t * * Atomically decrements @v by 1. */ -static __always_inline void atomic_dec(atomic_t *v) +static __always_inline void arch_atomic_dec(atomic_t *v) { asm volatile(LOCK_PREFIX "decl %0" : "+m" (v->counter)); } /** - * atomic_dec_and_test - decrement and test + * arch_atomic_dec_and_test - decrement and test * @v: pointer of type atomic_t * * Atomically decrements @v by 1 and * returns true if the result is 0, or false for all other * cases. */ -static __always_inline bool atomic_dec_and_test(atomic_t *v) +static __always_inline bool arch_atomic_dec_and_test(atomic_t *v) { GEN_UNARY_RMWcc(LOCK_PREFIX "decl", v->counter, "%0", e); } /** - * atomic_inc_and_test - increment and test + * arch_atomic_inc_and_test - increment and test * @v: pointer of type atomic_t * * Atomically increments @v by 1 * and returns true if the result is zero, or false for all * other cases. */ -static __always_inline bool atomic_inc_and_test(atomic_t *v) +static __always_inline bool arch_atomic_inc_and_test(atomic_t *v) { GEN_UNARY_RMWcc(LOCK_PREFIX "incl", v->counter, "%0", e); } /** - * atomic_add_negative - add and test if negative + * arch_atomic_add_negative - add and test if negative * @i: integer value to add * @v: pointer of type atomic_t * @@ -139,65 +139,65 @@ static __always_inline bool atomic_inc_and_test(atomic_t *v) * if the result is negative, or false when * result is greater than or equal to zero. */ -static __always_inline bool atomic_add_negative(int i, atomic_t *v) +static __always_inline bool arch_atomic_add_negative(int i, atomic_t *v) { GEN_BINARY_RMWcc(LOCK_PREFIX "addl", v->counter, "er", i, "%0", s); } /** - * atomic_add_return - add integer and return + * arch_atomic_add_return - add integer and return * @i: integer value to add * @v: pointer of type atomic_t * * Atomically adds @i to @v and returns @i + @v */ -static __always_inline int atomic_add_return(int i, atomic_t *v) +static __always_inline int arch_atomic_add_return(int i, atomic_t *v) { return i + xadd(&v->counter, i); } /** - * atomic_sub_return - subtract integer and return + * arch_atomic_sub_return - subtract integer and return * @v: pointer of type atomic_t * @i: integer value to subtract * * Atomically subtracts @i from @v and returns @v - @i */ -static __always_inline int atomic_sub_return(int i, atomic_t *v) +static __always_inline int arch_atomic_sub_return(int i, atomic_t *v) { - return atomic_add_return(-i, v); + return arch_atomic_add_return(-i, v); } -#define atomic_inc_return(v) (atomic_add_return(1, v)) -#define atomic_dec_return(v) (atomic_sub_return(1, v)) +#define arch_atomic_inc_return(v) (arch_atomic_add_return(1, v)) +#define arch_atomic_dec_return(v) (arch_atomic_sub_return(1, v)) -static __always_inline int atomic_fetch_add(int i, atomic_t *v) +static __always_inline int arch_atomic_fetch_add(int i, atomic_t *v) { return xadd(&v->counter, i); } -static __always_inline int atomic_fetch_sub(int i, atomic_t *v) +static __always_inline int arch_atomic_fetch_sub(int i, atomic_t *v) { return xadd(&v->counter, -i); } -static __always_inline int atomic_cmpxchg(atomic_t *v, int old, int new) +static __always_inline int arch_atomic_cmpxchg(atomic_t *v, int old, int new) { - return cmpxchg(&v->counter, old, new); + return arch_cmpxchg(&v->counter, old, new); } -#define atomic_try_cmpxchg atomic_try_cmpxchg -static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new) +#define arch_atomic_try_cmpxchg arch_atomic_try_cmpxchg +static __always_inline bool arch_atomic_try_cmpxchg(atomic_t *v, int *old, int new) { return try_cmpxchg(&v->counter, old, new); } -static inline int atomic_xchg(atomic_t *v, int new) +static inline int arch_atomic_xchg(atomic_t *v, int new) { return xchg(&v->counter, new); } -static inline void atomic_and(int i, atomic_t *v) +static inline void arch_atomic_and(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "andl %1,%0" : "+m" (v->counter) @@ -205,16 +205,16 @@ static inline void atomic_and(int i, atomic_t *v) : "memory"); } -static inline int atomic_fetch_and(int i, atomic_t *v) +static inline int arch_atomic_fetch_and(int i, atomic_t *v) { - int val = atomic_read(v); + int val = arch_atomic_read(v); - do { } while (!atomic_try_cmpxchg(v, &val, val & i)); + do { } while (!arch_atomic_try_cmpxchg(v, &val, val & i)); return val; } -static inline void atomic_or(int i, atomic_t *v) +static inline void arch_atomic_or(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "orl %1,%0" : "+m" (v->counter) @@ -222,16 +222,16 @@ static inline void atomic_or(int i, atomic_t *v) : "memory"); } -static inline int atomic_fetch_or(int i, atomic_t *v) +static inline int arch_atomic_fetch_or(int i, atomic_t *v) { - int val = atomic_read(v); + int val = arch_atomic_read(v); - do { } while (!atomic_try_cmpxchg(v, &val, val | i)); + do { } while (!arch_atomic_try_cmpxchg(v, &val, val | i)); return val; } -static inline void atomic_xor(int i, atomic_t *v) +static inline void arch_atomic_xor(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "xorl %1,%0" : "+m" (v->counter) @@ -239,17 +239,17 @@ static inline void atomic_xor(int i, atomic_t *v) : "memory"); } -static inline int atomic_fetch_xor(int i, atomic_t *v) +static inline int arch_atomic_fetch_xor(int i, atomic_t *v) { - int val = atomic_read(v); + int val = arch_atomic_read(v); - do { } while (!atomic_try_cmpxchg(v, &val, val ^ i)); + do { } while (!arch_atomic_try_cmpxchg(v, &val, val ^ i)); return val; } /** - * __atomic_add_unless - add unless the number is already a given value + * __arch_atomic_add_unless - add unless the number is already a given value * @v: pointer of type atomic_t * @a: the amount to add to v... * @u: ...unless v is equal to u. @@ -257,14 +257,14 @@ static inline int atomic_fetch_xor(int i, atomic_t *v) * Atomically adds @a to @v, so long as @v was not already @u. * Returns the old value of @v. */ -static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u) +static __always_inline int __arch_atomic_add_unless(atomic_t *v, int a, int u) { - int c = atomic_read(v); + int c = arch_atomic_read(v); do { if (unlikely(c == u)) break; - } while (!atomic_try_cmpxchg(v, &c, c + a)); + } while (!arch_atomic_try_cmpxchg(v, &c, c + a)); return c; } @@ -275,4 +275,6 @@ static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u) # include <asm/atomic64_64.h> #endif +#include <asm-generic/atomic-instrumented.h> + #endif /* _ASM_X86_ATOMIC_H */ diff --git a/arch/x86/include/asm/atomic64_32.h b/arch/x86/include/asm/atomic64_32.h index 9e206f3..512377a 100644 --- a/arch/x86/include/asm/atomic64_32.h +++ b/arch/x86/include/asm/atomic64_32.h @@ -61,7 +61,7 @@ ATOMIC64_DECL(add_unless); #undef ATOMIC64_EXPORT /** - * atomic64_cmpxchg - cmpxchg atomic64 variable + * arch_atomic64_cmpxchg - cmpxchg atomic64 variable * @v: pointer to type atomic64_t * @o: expected value * @n: new value @@ -70,20 +70,21 @@ ATOMIC64_DECL(add_unless); * the old value. */ -static inline long long atomic64_cmpxchg(atomic64_t *v, long long o, long long n) +static inline long long arch_atomic64_cmpxchg(atomic64_t *v, long long o, + long long n) { - return cmpxchg64(&v->counter, o, n); + return arch_cmpxchg64(&v->counter, o, n); } /** - * atomic64_xchg - xchg atomic64 variable + * arch_atomic64_xchg - xchg atomic64 variable * @v: pointer to type atomic64_t * @n: value to assign * * Atomically xchgs the value of @v to @n and returns * the old value. */ -static inline long long atomic64_xchg(atomic64_t *v, long long n) +static inline long long arch_atomic64_xchg(atomic64_t *v, long long n) { long long o; unsigned high = (unsigned)(n >> 32); @@ -95,13 +96,13 @@ static inline long long atomic64_xchg(atomic64_t *v, long long n) } /** - * atomic64_set - set atomic64 variable + * arch_atomic64_set - set atomic64 variable * @v: pointer to type atomic64_t * @i: value to assign * * Atomically sets the value of @v to @n. */ -static inline void atomic64_set(atomic64_t *v, long long i) +static inline void arch_atomic64_set(atomic64_t *v, long long i) { unsigned high = (unsigned)(i >> 32); unsigned low = (unsigned)i; @@ -111,12 +112,12 @@ static inline void atomic64_set(atomic64_t *v, long long i) } /** - * atomic64_read - read atomic64 variable + * arch_atomic64_read - read atomic64 variable * @v: pointer to type atomic64_t * * Atomically reads the value of @v and returns it. */ -static inline long long atomic64_read(const atomic64_t *v) +static inline long long arch_atomic64_read(const atomic64_t *v) { long long r; alternative_atomic64(read, "=&A" (r), "c" (v) : "memory"); @@ -124,13 +125,13 @@ static inline long long atomic64_read(const atomic64_t *v) } /** - * atomic64_add_return - add and return + * arch_atomic64_add_return - add and return * @i: integer value to add * @v: pointer to type atomic64_t * * Atomically adds @i to @v and returns @i + *@v */ -static inline long long atomic64_add_return(long long i, atomic64_t *v) +static inline long long arch_atomic64_add_return(long long i, atomic64_t *v) { alternative_atomic64(add_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -141,7 +142,7 @@ static inline long long atomic64_add_return(long long i, atomic64_t *v) /* * Other variants with different arithmetic operators: */ -static inline long long atomic64_sub_return(long long i, atomic64_t *v) +static inline long long arch_atomic64_sub_return(long long i, atomic64_t *v) { alternative_atomic64(sub_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -149,7 +150,7 @@ static inline long long atomic64_sub_return(long long i, atomic64_t *v) return i; } -static inline long long atomic64_inc_return(atomic64_t *v) +static inline long long arch_atomic64_inc_return(atomic64_t *v) { long long a; alternative_atomic64(inc_return, "=&A" (a), @@ -157,7 +158,7 @@ static inline long long atomic64_inc_return(atomic64_t *v) return a; } -static inline long long atomic64_dec_return(atomic64_t *v) +static inline long long arch_atomic64_dec_return(atomic64_t *v) { long long a; alternative_atomic64(dec_return, "=&A" (a), @@ -166,13 +167,13 @@ static inline long long atomic64_dec_return(atomic64_t *v) } /** - * atomic64_add - add integer to atomic64 variable + * arch_atomic64_add - add integer to atomic64 variable * @i: integer value to add * @v: pointer to type atomic64_t * * Atomically adds @i to @v. */ -static inline long long atomic64_add(long long i, atomic64_t *v) +static inline long long arch_atomic64_add(long long i, atomic64_t *v) { __alternative_atomic64(add, add_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -181,13 +182,13 @@ static inline long long atomic64_add(long long i, atomic64_t *v) } /** - * atomic64_sub - subtract the atomic64 variable + * arch_atomic64_sub - subtract the atomic64 variable * @i: integer value to subtract * @v: pointer to type atomic64_t * * Atomically subtracts @i from @v. */ -static inline long long atomic64_sub(long long i, atomic64_t *v) +static inline long long arch_atomic64_sub(long long i, atomic64_t *v) { __alternative_atomic64(sub, sub_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -196,7 +197,7 @@ static inline long long atomic64_sub(long long i, atomic64_t *v) } /** - * atomic64_sub_and_test - subtract value from variable and test result + * arch_atomic64_sub_and_test - subtract value from variable and test result * @i: integer value to subtract * @v: pointer to type atomic64_t * @@ -204,46 +205,46 @@ static inline long long atomic64_sub(long long i, atomic64_t *v) * true if the result is zero, or false for all * other cases. */ -static inline int atomic64_sub_and_test(long long i, atomic64_t *v) +static inline int arch_atomic64_sub_and_test(long long i, atomic64_t *v) { - return atomic64_sub_return(i, v) == 0; + return arch_atomic64_sub_return(i, v) == 0; } /** - * atomic64_inc - increment atomic64 variable + * arch_atomic64_inc - increment atomic64 variable * @v: pointer to type atomic64_t * * Atomically increments @v by 1. */ -static inline void atomic64_inc(atomic64_t *v) +static inline void arch_atomic64_inc(atomic64_t *v) { __alternative_atomic64(inc, inc_return, /* no output */, "S" (v) : "memory", "eax", "ecx", "edx"); } /** - * atomic64_dec - decrement atomic64 variable + * arch_atomic64_dec - decrement atomic64 variable * @v: pointer to type atomic64_t * * Atomically decrements @v by 1. */ -static inline void atomic64_dec(atomic64_t *v) +static inline void arch_atomic64_dec(atomic64_t *v) { __alternative_atomic64(dec, dec_return, /* no output */, "S" (v) : "memory", "eax", "ecx", "edx"); } /** - * atomic64_dec_and_test - decrement and test + * arch_atomic64_dec_and_test - decrement and test * @v: pointer to type atomic64_t * * Atomically decrements @v by 1 and * returns true if the result is 0, or false for all other * cases. */ -static inline int atomic64_dec_and_test(atomic64_t *v) +static inline int arch_atomic64_dec_and_test(atomic64_t *v) { - return atomic64_dec_return(v) == 0; + return arch_atomic64_dec_return(v) == 0; } /** @@ -254,13 +255,13 @@ static inline int atomic64_dec_and_test(atomic64_t *v) * and returns true if the result is zero, or false for all * other cases. */ -static inline int atomic64_inc_and_test(atomic64_t *v) +static inline int arch_atomic64_inc_and_test(atomic64_t *v) { - return atomic64_inc_return(v) == 0; + return arch_atomic64_inc_return(v) == 0; } /** - * atomic64_add_negative - add and test if negative + * arch_atomic64_add_negative - add and test if negative * @i: integer value to add * @v: pointer to type atomic64_t * @@ -268,13 +269,13 @@ static inline int atomic64_inc_and_test(atomic64_t *v) * if the result is negative, or false when * result is greater than or equal to zero. */ -static inline int atomic64_add_negative(long long i, atomic64_t *v) +static inline int arch_atomic64_add_negative(long long i, atomic64_t *v) { - return atomic64_add_return(i, v) < 0; + return arch_atomic64_add_return(i, v) < 0; } /** - * atomic64_add_unless - add unless the number is a given value + * arch_atomic64_add_unless - add unless the number is a given value * @v: pointer of type atomic64_t * @a: the amount to add to v... * @u: ...unless v is equal to u. @@ -282,7 +283,8 @@ static inline int atomic64_add_negative(long long i, atomic64_t *v) * Atomically adds @a to @v, so long as it was not @u. * Returns non-zero if the add was done, zero otherwise. */ -static inline int atomic64_add_unless(atomic64_t *v, long long a, long long u) +static inline int arch_atomic64_add_unless(atomic64_t *v, long long a, + long long u) { unsigned low = (unsigned)u; unsigned high = (unsigned)(u >> 32); @@ -293,7 +295,7 @@ static inline int atomic64_add_unless(atomic64_t *v, long long a, long long u) } -static inline int atomic64_inc_not_zero(atomic64_t *v) +static inline int arch_atomic64_inc_not_zero(atomic64_t *v) { int r; alternative_atomic64(inc_not_zero, "=&a" (r), @@ -301,7 +303,7 @@ static inline int atomic64_inc_not_zero(atomic64_t *v) return r; } -static inline long long atomic64_dec_if_positive(atomic64_t *v) +static inline long long arch_atomic64_dec_if_positive(atomic64_t *v) { long long r; alternative_atomic64(dec_if_positive, "=&A" (r), @@ -312,70 +314,70 @@ static inline long long atomic64_dec_if_positive(atomic64_t *v) #undef alternative_atomic64 #undef __alternative_atomic64 -static inline void atomic64_and(long long i, atomic64_t *v) +static inline void arch_atomic64_and(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c & i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c & i)) != c) c = old; } -static inline long long atomic64_fetch_and(long long i, atomic64_t *v) +static inline long long arch_atomic64_fetch_and(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c & i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c & i)) != c) c = old; return old; } -static inline void atomic64_or(long long i, atomic64_t *v) +static inline void arch_atomic64_or(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c | i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c | i)) != c) c = old; } -static inline long long atomic64_fetch_or(long long i, atomic64_t *v) +static inline long long arch_atomic64_fetch_or(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c | i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c | i)) != c) c = old; return old; } -static inline void atomic64_xor(long long i, atomic64_t *v) +static inline void arch_atomic64_xor(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c ^ i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c ^ i)) != c) c = old; } -static inline long long atomic64_fetch_xor(long long i, atomic64_t *v) +static inline long long arch_atomic64_fetch_xor(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c ^ i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c ^ i)) != c) c = old; return old; } -static inline long long atomic64_fetch_add(long long i, atomic64_t *v) +static inline long long arch_atomic64_fetch_add(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c + i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c + i)) != c) c = old; return old; } -#define atomic64_fetch_sub(i, v) atomic64_fetch_add(-(i), (v)) +#define arch_atomic64_fetch_sub(i, v) arch_atomic64_fetch_add(-(i), (v)) #endif /* _ASM_X86_ATOMIC64_32_H */ diff --git a/arch/x86/include/asm/atomic64_64.h b/arch/x86/include/asm/atomic64_64.h index 5d9de36..5883b10 100644 --- a/arch/x86/include/asm/atomic64_64.h +++ b/arch/x86/include/asm/atomic64_64.h @@ -10,37 +10,37 @@ #define ATOMIC64_INIT(i) { (i) } /** - * atomic64_read - read atomic64 variable + * arch_atomic64_read - read atomic64 variable * @v: pointer of type atomic64_t * * Atomically reads the value of @v. * Doesn't imply a read memory barrier. */ -static inline long atomic64_read(const atomic64_t *v) +static inline long arch_atomic64_read(const atomic64_t *v) { return READ_ONCE((v)->counter); } /** - * atomic64_set - set atomic64 variable + * arch_atomic64_set - set atomic64 variable * @v: pointer to type atomic64_t * @i: required value * * Atomically sets the value of @v to @i. */ -static inline void atomic64_set(atomic64_t *v, long i) +static inline void arch_atomic64_set(atomic64_t *v, long i) { WRITE_ONCE(v->counter, i); } /** - * atomic64_add - add integer to atomic64 variable + * arch_atomic64_add - add integer to atomic64 variable * @i: integer value to add * @v: pointer to type atomic64_t * * Atomically adds @i to @v. */ -static __always_inline void atomic64_add(long i, atomic64_t *v) +static __always_inline void arch_atomic64_add(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "addq %1,%0" : "=m" (v->counter) @@ -48,13 +48,13 @@ static __always_inline void atomic64_add(long i, atomic64_t *v) } /** - * atomic64_sub - subtract the atomic64 variable + * arch_atomic64_sub - subtract the atomic64 variable * @i: integer value to subtract * @v: pointer to type atomic64_t * * Atomically subtracts @i from @v. */ -static inline void atomic64_sub(long i, atomic64_t *v) +static inline void arch_atomic64_sub(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "subq %1,%0" : "=m" (v->counter) @@ -62,7 +62,7 @@ static inline void atomic64_sub(long i, atomic64_t *v) } /** - * atomic64_sub_and_test - subtract value from variable and test result + * arch_atomic64_sub_and_test - subtract value from variable and test result * @i: integer value to subtract * @v: pointer to type atomic64_t * @@ -70,18 +70,18 @@ static inline void atomic64_sub(long i, atomic64_t *v) * true if the result is zero, or false for all * other cases. */ -static inline bool atomic64_sub_and_test(long i, atomic64_t *v) +static inline bool arch_atomic64_sub_and_test(long i, atomic64_t *v) { GEN_BINARY_RMWcc(LOCK_PREFIX "subq", v->counter, "er", i, "%0", e); } /** - * atomic64_inc - increment atomic64 variable + * arch_atomic64_inc - increment atomic64 variable * @v: pointer to type atomic64_t * * Atomically increments @v by 1. */ -static __always_inline void atomic64_inc(atomic64_t *v) +static __always_inline void arch_atomic64_inc(atomic64_t *v) { asm volatile(LOCK_PREFIX "incq %0" : "=m" (v->counter) @@ -89,12 +89,12 @@ static __always_inline void atomic64_inc(atomic64_t *v) } /** - * atomic64_dec - decrement atomic64 variable + * arch_atomic64_dec - decrement atomic64 variable * @v: pointer to type atomic64_t * * Atomically decrements @v by 1. */ -static __always_inline void atomic64_dec(atomic64_t *v) +static __always_inline void arch_atomic64_dec(atomic64_t *v) { asm volatile(LOCK_PREFIX "decq %0" : "=m" (v->counter) @@ -102,33 +102,33 @@ static __always_inline void atomic64_dec(atomic64_t *v) } /** - * atomic64_dec_and_test - decrement and test + * arch_atomic64_dec_and_test - decrement and test * @v: pointer to type atomic64_t * * Atomically decrements @v by 1 and * returns true if the result is 0, or false for all other * cases. */ -static inline bool atomic64_dec_and_test(atomic64_t *v) +static inline bool arch_atomic64_dec_and_test(atomic64_t *v) { GEN_UNARY_RMWcc(LOCK_PREFIX "decq", v->counter, "%0", e); } /** - * atomic64_inc_and_test - increment and test + * arch_atomic64_inc_and_test - increment and test * @v: pointer to type atomic64_t * * Atomically increments @v by 1 * and returns true if the result is zero, or false for all * other cases. */ -static inline bool atomic64_inc_and_test(atomic64_t *v) +static inline bool arch_atomic64_inc_and_test(atomic64_t *v) { GEN_UNARY_RMWcc(LOCK_PREFIX "incq", v->counter, "%0", e); } /** - * atomic64_add_negative - add and test if negative + * arch_atomic64_add_negative - add and test if negative * @i: integer value to add * @v: pointer to type atomic64_t * @@ -136,59 +136,59 @@ static inline bool atomic64_inc_and_test(atomic64_t *v) * if the result is negative, or false when * result is greater than or equal to zero. */ -static inline bool atomic64_add_negative(long i, atomic64_t *v) +static inline bool arch_atomic64_add_negative(long i, atomic64_t *v) { GEN_BINARY_RMWcc(LOCK_PREFIX "addq", v->counter, "er", i, "%0", s); } /** - * atomic64_add_return - add and return + * arch_atomic64_add_return - add and return * @i: integer value to add * @v: pointer to type atomic64_t * * Atomically adds @i to @v and returns @i + @v */ -static __always_inline long atomic64_add_return(long i, atomic64_t *v) +static __always_inline long arch_atomic64_add_return(long i, atomic64_t *v) { return i + xadd(&v->counter, i); } -static inline long atomic64_sub_return(long i, atomic64_t *v) +static inline long arch_atomic64_sub_return(long i, atomic64_t *v) { - return atomic64_add_return(-i, v); + return arch_atomic64_add_return(-i, v); } -static inline long atomic64_fetch_add(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_add(long i, atomic64_t *v) { return xadd(&v->counter, i); } -static inline long atomic64_fetch_sub(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_sub(long i, atomic64_t *v) { return xadd(&v->counter, -i); } -#define atomic64_inc_return(v) (atomic64_add_return(1, (v))) -#define atomic64_dec_return(v) (atomic64_sub_return(1, (v))) +#define arch_atomic64_inc_return(v) (arch_atomic64_add_return(1, (v))) +#define arch_atomic64_dec_return(v) (arch_atomic64_sub_return(1, (v))) -static inline long atomic64_cmpxchg(atomic64_t *v, long old, long new) +static inline long arch_atomic64_cmpxchg(atomic64_t *v, long old, long new) { - return cmpxchg(&v->counter, old, new); + return arch_cmpxchg(&v->counter, old, new); } -#define atomic64_try_cmpxchg atomic64_try_cmpxchg -static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, long new) +#define arch_atomic64_try_cmpxchg arch_atomic64_try_cmpxchg +static __always_inline bool arch_atomic64_try_cmpxchg(atomic64_t *v, s64 *old, long new) { return try_cmpxchg(&v->counter, old, new); } -static inline long atomic64_xchg(atomic64_t *v, long new) +static inline long arch_atomic64_xchg(atomic64_t *v, long new) { return xchg(&v->counter, new); } /** - * atomic64_add_unless - add unless the number is a given value + * arch_atomic64_add_unless - add unless the number is a given value * @v: pointer of type atomic64_t * @a: the amount to add to v... * @u: ...unless v is equal to u. @@ -196,37 +196,37 @@ static inline long atomic64_xchg(atomic64_t *v, long new) * Atomically adds @a to @v, so long as it was not @u. * Returns the old value of @v. */ -static inline bool atomic64_add_unless(atomic64_t *v, long a, long u) +static inline bool arch_atomic64_add_unless(atomic64_t *v, long a, long u) { - s64 c = atomic64_read(v); + s64 c = arch_atomic64_read(v); do { if (unlikely(c == u)) return false; - } while (!atomic64_try_cmpxchg(v, &c, c + a)); + } while (!arch_atomic64_try_cmpxchg(v, &c, c + a)); return true; } -#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0) +#define arch_atomic64_inc_not_zero(v) arch_atomic64_add_unless((v), 1, 0) /* - * atomic64_dec_if_positive - decrement by 1 if old value positive + * arch_atomic64_dec_if_positive - decrement by 1 if old value positive * @v: pointer of type atomic_t * * The function returns the old value of *v minus 1, even if * the atomic variable, v, was not decremented. */ -static inline long atomic64_dec_if_positive(atomic64_t *v) +static inline long arch_atomic64_dec_if_positive(atomic64_t *v) { - s64 dec, c = atomic64_read(v); + s64 dec, c = arch_atomic64_read(v); do { dec = c - 1; if (unlikely(dec < 0)) break; - } while (!atomic64_try_cmpxchg(v, &c, dec)); + } while (!arch_atomic64_try_cmpxchg(v, &c, dec)); return dec; } -static inline void atomic64_and(long i, atomic64_t *v) +static inline void arch_atomic64_and(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "andq %1,%0" : "+m" (v->counter) @@ -234,16 +234,16 @@ static inline void atomic64_and(long i, atomic64_t *v) : "memory"); } -static inline long atomic64_fetch_and(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_and(long i, atomic64_t *v) { - s64 val = atomic64_read(v); + s64 val = arch_atomic64_read(v); do { - } while (!atomic64_try_cmpxchg(v, &val, val & i)); + } while (!arch_atomic64_try_cmpxchg(v, &val, val & i)); return val; } -static inline void atomic64_or(long i, atomic64_t *v) +static inline void arch_atomic64_or(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "orq %1,%0" : "+m" (v->counter) @@ -251,16 +251,16 @@ static inline void atomic64_or(long i, atomic64_t *v) : "memory"); } -static inline long atomic64_fetch_or(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_or(long i, atomic64_t *v) { - s64 val = atomic64_read(v); + s64 val = arch_atomic64_read(v); do { - } while (!atomic64_try_cmpxchg(v, &val, val | i)); + } while (!arch_atomic64_try_cmpxchg(v, &val, val | i)); return val; } -static inline void atomic64_xor(long i, atomic64_t *v) +static inline void arch_atomic64_xor(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "xorq %1,%0" : "+m" (v->counter) @@ -268,12 +268,12 @@ static inline void atomic64_xor(long i, atomic64_t *v) : "memory"); } -static inline long atomic64_fetch_xor(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_xor(long i, atomic64_t *v) { - s64 val = atomic64_read(v); + s64 val = arch_atomic64_read(v); do { - } while (!atomic64_try_cmpxchg(v, &val, val ^ i)); + } while (!arch_atomic64_try_cmpxchg(v, &val, val ^ i)); return val; } diff --git a/arch/x86/include/asm/cmpxchg.h b/arch/x86/include/asm/cmpxchg.h index b5069e8..9e2faa8 100644 --- a/arch/x86/include/asm/cmpxchg.h +++ b/arch/x86/include/asm/cmpxchg.h @@ -144,13 +144,13 @@ extern void __add_wrong_size(void) # include <asm/cmpxchg_64.h> #endif -#define cmpxchg(ptr, old, new) \ +#define arch_cmpxchg(ptr, old, new) \ __cmpxchg(ptr, old, new, sizeof(*(ptr))) -#define sync_cmpxchg(ptr, old, new) \ +#define arch_sync_cmpxchg(ptr, old, new) \ __sync_cmpxchg(ptr, old, new, sizeof(*(ptr))) -#define cmpxchg_local(ptr, old, new) \ +#define arch_cmpxchg_local(ptr, old, new) \ __cmpxchg_local(ptr, old, new, sizeof(*(ptr))) @@ -220,7 +220,7 @@ extern void __add_wrong_size(void) #define __try_cmpxchg(ptr, pold, new, size) \ __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX) -#define try_cmpxchg(ptr, pold, new) \ +#define try_cmpxchg(ptr, pold, new) \ __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr))) /* @@ -249,10 +249,10 @@ extern void __add_wrong_size(void) __ret; \ }) -#define cmpxchg_double(p1, p2, o1, o2, n1, n2) \ +#define arch_cmpxchg_double(p1, p2, o1, o2, n1, n2) \ __cmpxchg_double(LOCK_PREFIX, p1, p2, o1, o2, n1, n2) -#define cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \ +#define arch_cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \ __cmpxchg_double(, p1, p2, o1, o2, n1, n2) #endif /* ASM_X86_CMPXCHG_H */ diff --git a/arch/x86/include/asm/cmpxchg_32.h b/arch/x86/include/asm/cmpxchg_32.h index e4959d0..d897291 100644 --- a/arch/x86/include/asm/cmpxchg_32.h +++ b/arch/x86/include/asm/cmpxchg_32.h @@ -35,10 +35,10 @@ static inline void set_64bit(volatile u64 *ptr, u64 value) } #ifdef CONFIG_X86_CMPXCHG64 -#define cmpxchg64(ptr, o, n) \ +#define arch_cmpxchg64(ptr, o, n) \ ((__typeof__(*(ptr)))__cmpxchg64((ptr), (unsigned long long)(o), \ (unsigned long long)(n))) -#define cmpxchg64_local(ptr, o, n) \ +#define arch_cmpxchg64_local(ptr, o, n) \ ((__typeof__(*(ptr)))__cmpxchg64_local((ptr), (unsigned long long)(o), \ (unsigned long long)(n))) #endif @@ -75,7 +75,7 @@ static inline u64 __cmpxchg64_local(volatile u64 *ptr, u64 old, u64 new) * to simulate the cmpxchg8b on the 80386 and 80486 CPU. */ -#define cmpxchg64(ptr, o, n) \ +#define arch_cmpxchg64(ptr, o, n) \ ({ \ __typeof__(*(ptr)) __ret; \ __typeof__(*(ptr)) __old = (o); \ @@ -92,7 +92,7 @@ static inline u64 __cmpxchg64_local(volatile u64 *ptr, u64 old, u64 new) __ret; }) -#define cmpxchg64_local(ptr, o, n) \ +#define arch_cmpxchg64_local(ptr, o, n) \ ({ \ __typeof__(*(ptr)) __ret; \ __typeof__(*(ptr)) __old = (o); \ diff --git a/arch/x86/include/asm/cmpxchg_64.h b/arch/x86/include/asm/cmpxchg_64.h index caa23a3..fafaeba 100644 --- a/arch/x86/include/asm/cmpxchg_64.h +++ b/arch/x86/include/asm/cmpxchg_64.h @@ -6,13 +6,13 @@ static inline void set_64bit(volatile u64 *ptr, u64 val) *ptr = val; } -#define cmpxchg64(ptr, o, n) \ +#define arch_cmpxchg64(ptr, o, n) \ ({ \ BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ cmpxchg((ptr), (o), (n)); \ }) -#define cmpxchg64_local(ptr, o, n) \ +#define arch_cmpxchg64_local(ptr, o, n) \ ({ \ BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ cmpxchg_local((ptr), (o), (n)); \ ^ permalink raw reply related [flat|nested] 63+ messages in thread
* [PATCH v5 2/4] kasan: allow kasan_check_read/write() to accept pointers to volatiles [not found] ` <cover.1498140838.git.dvyukov@google.com> @ 2017-06-22 14:14 ` Dmitry Vyukov 2017-06-22 14:14 ` Dmitry Vyukov ` (2 subsequent siblings) 3 siblings, 0 replies; 63+ messages in thread From: Dmitry Vyukov @ 2017-06-22 14:14 UTC (permalink / raw) To: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin, kasan-dev, x86, linux-kernel Cc: Dmitry Vyukov, Thomas Gleixner, Andrew Morton, linux-mm Currently kasan_check_read/write() accept 'const void*', make them accept 'const volatile void*'. This is required for instrumentation of atomic operations and there is just no reason to not allow that. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: linux-kernel@vger.kernel.org Cc: x86@kernel.org Cc: linux-mm@kvack.org Cc: kasan-dev@googlegroups.com --- include/linux/kasan-checks.h | 10 ++++++---- mm/kasan/kasan.c | 4 ++-- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/include/linux/kasan-checks.h b/include/linux/kasan-checks.h index b7f8aced7870..41960fecf783 100644 --- a/include/linux/kasan-checks.h +++ b/include/linux/kasan-checks.h @@ -2,11 +2,13 @@ #define _LINUX_KASAN_CHECKS_H #ifdef CONFIG_KASAN -void kasan_check_read(const void *p, unsigned int size); -void kasan_check_write(const void *p, unsigned int size); +void kasan_check_read(const volatile void *p, unsigned int size); +void kasan_check_write(const volatile void *p, unsigned int size); #else -static inline void kasan_check_read(const void *p, unsigned int size) { } -static inline void kasan_check_write(const void *p, unsigned int size) { } +static inline void kasan_check_read(const volatile void *p, unsigned int size) +{ } +static inline void kasan_check_write(const volatile void *p, unsigned int size) +{ } #endif #endif diff --git a/mm/kasan/kasan.c b/mm/kasan/kasan.c index c81549d5c833..edacd161c0e5 100644 --- a/mm/kasan/kasan.c +++ b/mm/kasan/kasan.c @@ -333,13 +333,13 @@ static void check_memory_region(unsigned long addr, check_memory_region_inline(addr, size, write, ret_ip); } -void kasan_check_read(const void *p, unsigned int size) +void kasan_check_read(const volatile void *p, unsigned int size) { check_memory_region((unsigned long)p, size, false, _RET_IP_); } EXPORT_SYMBOL(kasan_check_read); -void kasan_check_write(const void *p, unsigned int size) +void kasan_check_write(const volatile void *p, unsigned int size) { check_memory_region((unsigned long)p, size, true, _RET_IP_); } -- 2.13.1.611.g7e3b11ae1-goog ^ permalink raw reply related [flat|nested] 63+ messages in thread
* [PATCH v5 2/4] kasan: allow kasan_check_read/write() to accept pointers to volatiles @ 2017-06-22 14:14 ` Dmitry Vyukov 0 siblings, 0 replies; 63+ messages in thread From: Dmitry Vyukov @ 2017-06-22 14:14 UTC (permalink / raw) To: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin, kasan-dev, x86, linux-kernel Cc: Dmitry Vyukov, Thomas Gleixner, Andrew Morton, linux-mm Currently kasan_check_read/write() accept 'const void*', make them accept 'const volatile void*'. This is required for instrumentation of atomic operations and there is just no reason to not allow that. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: "H. Peter Anvin" <hpa@zytor.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: linux-kernel@vger.kernel.org Cc: x86@kernel.org Cc: linux-mm@kvack.org Cc: kasan-dev@googlegroups.com --- include/linux/kasan-checks.h | 10 ++++++---- mm/kasan/kasan.c | 4 ++-- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/include/linux/kasan-checks.h b/include/linux/kasan-checks.h index b7f8aced7870..41960fecf783 100644 --- a/include/linux/kasan-checks.h +++ b/include/linux/kasan-checks.h @@ -2,11 +2,13 @@ #define _LINUX_KASAN_CHECKS_H #ifdef CONFIG_KASAN -void kasan_check_read(const void *p, unsigned int size); -void kasan_check_write(const void *p, unsigned int size); +void kasan_check_read(const volatile void *p, unsigned int size); +void kasan_check_write(const volatile void *p, unsigned int size); #else -static inline void kasan_check_read(const void *p, unsigned int size) { } -static inline void kasan_check_write(const void *p, unsigned int size) { } +static inline void kasan_check_read(const volatile void *p, unsigned int size) +{ } +static inline void kasan_check_write(const volatile void *p, unsigned int size) +{ } #endif #endif diff --git a/mm/kasan/kasan.c b/mm/kasan/kasan.c index c81549d5c833..edacd161c0e5 100644 --- a/mm/kasan/kasan.c +++ b/mm/kasan/kasan.c @@ -333,13 +333,13 @@ static void check_memory_region(unsigned long addr, check_memory_region_inline(addr, size, write, ret_ip); } -void kasan_check_read(const void *p, unsigned int size) +void kasan_check_read(const volatile void *p, unsigned int size) { check_memory_region((unsigned long)p, size, false, _RET_IP_); } EXPORT_SYMBOL(kasan_check_read); -void kasan_check_write(const void *p, unsigned int size) +void kasan_check_write(const volatile void *p, unsigned int size) { check_memory_region((unsigned long)p, size, true, _RET_IP_); } -- 2.13.1.611.g7e3b11ae1-goog -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply related [flat|nested] 63+ messages in thread
* [tip:locking/core] kasan: Allow kasan_check_read/write() to accept pointers to volatiles 2017-06-22 14:14 ` Dmitry Vyukov (?) @ 2017-06-23 14:00 ` tip-bot for Dmitry Vyukov -1 siblings, 0 replies; 63+ messages in thread From: tip-bot for Dmitry Vyukov @ 2017-06-23 14:00 UTC (permalink / raw) To: linux-tip-commits Cc: tglx, peterz, torvalds, linux-kernel, akpm, mark.rutland, mingo, aryabinin, hpa, dvyukov Commit-ID: 342b3b2cbf10c267c75a9608c8a2a0316162cbd4 Gitweb: http://git.kernel.org/tip/342b3b2cbf10c267c75a9608c8a2a0316162cbd4 Author: Dmitry Vyukov <dvyukov@google.com> AuthorDate: Thu, 22 Jun 2017 16:14:17 +0200 Committer: Ingo Molnar <mingo@kernel.org> CommitDate: Fri, 23 Jun 2017 10:50:20 +0200 kasan: Allow kasan_check_read/write() to accept pointers to volatiles Currently kasan_check_read/write() accept 'const void*', make them accept 'const volatile void*'. This is required for instrumentation of atomic operations and there is just no reason to not allow that. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Reviewed-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Cc: will.deacon@arm.com Link: http://lkml.kernel.org/r/33e5ec275c1ee89299245b2ebbccd63709c6021f.1498140838.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org> --- include/linux/kasan-checks.h | 10 ++++++---- mm/kasan/kasan.c | 4 ++-- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/include/linux/kasan-checks.h b/include/linux/kasan-checks.h index b7f8ace..41960fe 100644 --- a/include/linux/kasan-checks.h +++ b/include/linux/kasan-checks.h @@ -2,11 +2,13 @@ #define _LINUX_KASAN_CHECKS_H #ifdef CONFIG_KASAN -void kasan_check_read(const void *p, unsigned int size); -void kasan_check_write(const void *p, unsigned int size); +void kasan_check_read(const volatile void *p, unsigned int size); +void kasan_check_write(const volatile void *p, unsigned int size); #else -static inline void kasan_check_read(const void *p, unsigned int size) { } -static inline void kasan_check_write(const void *p, unsigned int size) { } +static inline void kasan_check_read(const volatile void *p, unsigned int size) +{ } +static inline void kasan_check_write(const volatile void *p, unsigned int size) +{ } #endif #endif diff --git a/mm/kasan/kasan.c b/mm/kasan/kasan.c index c81549d..edacd16 100644 --- a/mm/kasan/kasan.c +++ b/mm/kasan/kasan.c @@ -333,13 +333,13 @@ static void check_memory_region(unsigned long addr, check_memory_region_inline(addr, size, write, ret_ip); } -void kasan_check_read(const void *p, unsigned int size) +void kasan_check_read(const volatile void *p, unsigned int size) { check_memory_region((unsigned long)p, size, false, _RET_IP_); } EXPORT_SYMBOL(kasan_check_read); -void kasan_check_write(const void *p, unsigned int size) +void kasan_check_write(const volatile void *p, unsigned int size) { check_memory_region((unsigned long)p, size, true, _RET_IP_); } ^ permalink raw reply related [flat|nested] 63+ messages in thread
* [tip:locking/core] kasan: Allow kasan_check_read/write() to accept pointers to volatiles 2017-06-22 14:14 ` Dmitry Vyukov (?) (?) @ 2017-07-26 12:10 ` tip-bot for Dmitry Vyukov -1 siblings, 0 replies; 63+ messages in thread From: tip-bot for Dmitry Vyukov @ 2017-07-26 12:10 UTC (permalink / raw) To: linux-tip-commits Cc: tglx, linux-kernel, hpa, mingo, torvalds, akpm, aryabinin, mark.rutland, dvyukov, peterz Commit-ID: f06e8c584fa0d05312c11ea66194f3d2efb93c21 Gitweb: http://git.kernel.org/tip/f06e8c584fa0d05312c11ea66194f3d2efb93c21 Author: Dmitry Vyukov <dvyukov@google.com> AuthorDate: Thu, 22 Jun 2017 16:14:17 +0200 Committer: Ingo Molnar <mingo@kernel.org> CommitDate: Wed, 26 Jul 2017 13:08:54 +0200 kasan: Allow kasan_check_read/write() to accept pointers to volatiles Currently kasan_check_read/write() accept 'const void*', make them accept 'const volatile void*'. This is required for instrumentation of atomic operations and there is just no reason to not allow that. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Reviewed-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Cc: will.deacon@arm.com Link: http://lkml.kernel.org/r/33e5ec275c1ee89299245b2ebbccd63709c6021f.1498140838.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org> --- include/linux/kasan-checks.h | 10 ++++++---- mm/kasan/kasan.c | 4 ++-- 2 files changed, 8 insertions(+), 6 deletions(-) diff --git a/include/linux/kasan-checks.h b/include/linux/kasan-checks.h index b7f8ace..41960fe 100644 --- a/include/linux/kasan-checks.h +++ b/include/linux/kasan-checks.h @@ -2,11 +2,13 @@ #define _LINUX_KASAN_CHECKS_H #ifdef CONFIG_KASAN -void kasan_check_read(const void *p, unsigned int size); -void kasan_check_write(const void *p, unsigned int size); +void kasan_check_read(const volatile void *p, unsigned int size); +void kasan_check_write(const volatile void *p, unsigned int size); #else -static inline void kasan_check_read(const void *p, unsigned int size) { } -static inline void kasan_check_write(const void *p, unsigned int size) { } +static inline void kasan_check_read(const volatile void *p, unsigned int size) +{ } +static inline void kasan_check_write(const volatile void *p, unsigned int size) +{ } #endif #endif diff --git a/mm/kasan/kasan.c b/mm/kasan/kasan.c index ca11bc4..6f319fb 100644 --- a/mm/kasan/kasan.c +++ b/mm/kasan/kasan.c @@ -267,13 +267,13 @@ static void check_memory_region(unsigned long addr, check_memory_region_inline(addr, size, write, ret_ip); } -void kasan_check_read(const void *p, unsigned int size) +void kasan_check_read(const volatile void *p, unsigned int size) { check_memory_region((unsigned long)p, size, false, _RET_IP_); } EXPORT_SYMBOL(kasan_check_read); -void kasan_check_write(const void *p, unsigned int size) +void kasan_check_write(const volatile void *p, unsigned int size) { check_memory_region((unsigned long)p, size, true, _RET_IP_); } ^ permalink raw reply related [flat|nested] 63+ messages in thread
* [PATCH v5 3/4] asm-generic: add KASAN instrumentation to atomic operations [not found] ` <cover.1498140838.git.dvyukov@google.com> @ 2017-06-22 14:14 ` Dmitry Vyukov 2017-06-22 14:14 ` Dmitry Vyukov ` (2 subsequent siblings) 3 siblings, 0 replies; 63+ messages in thread From: Dmitry Vyukov @ 2017-06-22 14:14 UTC (permalink / raw) To: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin, kasan-dev, x86, linux-kernel Cc: Dmitry Vyukov, Andrew Morton, linux-mm KASAN uses compiler instrumentation to intercept all memory accesses. But it does not see memory accesses done in assembly code. One notable user of assembly code is atomic operations. Frequently, for example, an atomic reference decrement is the last access to an object and a good candidate for a racy use-after-free. Add manual KASAN checks to atomic operations. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Will Deacon <will.deacon@arm.com>, Cc: Andrew Morton <akpm@linux-foundation.org>, Cc: Ingo Molnar <mingo@redhat.com>, Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org Cc: x86@kernel.org --- include/asm-generic/atomic-instrumented.h | 76 +++++++++++++++++++++++++++++-- 1 file changed, 72 insertions(+), 4 deletions(-) diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h index 50401d925290..a0f5b7525bb2 100644 --- a/include/asm-generic/atomic-instrumented.h +++ b/include/asm-generic/atomic-instrumented.h @@ -1,43 +1,53 @@ #ifndef _LINUX_ATOMIC_INSTRUMENTED_H #define _LINUX_ATOMIC_INSTRUMENTED_H +#include <linux/kasan-checks.h> + static __always_inline int atomic_read(const atomic_t *v) { + kasan_check_read(v, sizeof(*v)); return arch_atomic_read(v); } static __always_inline s64 atomic64_read(const atomic64_t *v) { + kasan_check_read(v, sizeof(*v)); return arch_atomic64_read(v); } static __always_inline void atomic_set(atomic_t *v, int i) { + kasan_check_write(v, sizeof(*v)); arch_atomic_set(v, i); } static __always_inline void atomic64_set(atomic64_t *v, s64 i) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_set(v, i); } static __always_inline int atomic_xchg(atomic_t *v, int i) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_xchg(v, i); } static __always_inline s64 atomic64_xchg(atomic64_t *v, s64 i) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_xchg(v, i); } static __always_inline int atomic_cmpxchg(atomic_t *v, int old, int new) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_cmpxchg(v, old, new); } static __always_inline s64 atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_cmpxchg(v, old, new); } @@ -45,6 +55,8 @@ static __always_inline s64 atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new) #define atomic_try_cmpxchg atomic_try_cmpxchg static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new) { + kasan_check_write(v, sizeof(*v)); + kasan_check_read(old, sizeof(*old)); return arch_atomic_try_cmpxchg(v, old, new); } #endif @@ -53,254 +65,310 @@ static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new) #define atomic64_try_cmpxchg atomic64_try_cmpxchg static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, s64 new) { + kasan_check_write(v, sizeof(*v)); + kasan_check_read(old, sizeof(*old)); return arch_atomic64_try_cmpxchg(v, old, new); } #endif static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u) { + kasan_check_write(v, sizeof(*v)); return __arch_atomic_add_unless(v, a, u); } static __always_inline bool atomic64_add_unless(atomic64_t *v, s64 a, s64 u) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_add_unless(v, a, u); } static __always_inline void atomic_inc(atomic_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic_inc(v); } static __always_inline void atomic64_inc(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_inc(v); } static __always_inline void atomic_dec(atomic_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic_dec(v); } static __always_inline void atomic64_dec(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_dec(v); } static __always_inline void atomic_add(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic_add(i, v); } static __always_inline void atomic64_add(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_add(i, v); } static __always_inline void atomic_sub(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic_sub(i, v); } static __always_inline void atomic64_sub(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_sub(i, v); } static __always_inline void atomic_and(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic_and(i, v); } static __always_inline void atomic64_and(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_and(i, v); } static __always_inline void atomic_or(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic_or(i, v); } static __always_inline void atomic64_or(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_or(i, v); } static __always_inline void atomic_xor(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic_xor(i, v); } static __always_inline void atomic64_xor(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_xor(i, v); } static __always_inline int atomic_inc_return(atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_inc_return(v); } static __always_inline s64 atomic64_inc_return(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_inc_return(v); } static __always_inline int atomic_dec_return(atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_dec_return(v); } static __always_inline s64 atomic64_dec_return(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_dec_return(v); } static __always_inline s64 atomic64_inc_not_zero(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_inc_not_zero(v); } static __always_inline s64 atomic64_dec_if_positive(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_dec_if_positive(v); } static __always_inline bool atomic_dec_and_test(atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_dec_and_test(v); } static __always_inline bool atomic64_dec_and_test(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_dec_and_test(v); } static __always_inline bool atomic_inc_and_test(atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_inc_and_test(v); } static __always_inline bool atomic64_inc_and_test(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_inc_and_test(v); } static __always_inline int atomic_add_return(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_add_return(i, v); } static __always_inline s64 atomic64_add_return(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_add_return(i, v); } static __always_inline int atomic_sub_return(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_sub_return(i, v); } static __always_inline s64 atomic64_sub_return(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_sub_return(i, v); } static __always_inline int atomic_fetch_add(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_fetch_add(i, v); } static __always_inline s64 atomic64_fetch_add(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_fetch_add(i, v); } static __always_inline int atomic_fetch_sub(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_fetch_sub(i, v); } static __always_inline s64 atomic64_fetch_sub(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_fetch_sub(i, v); } static __always_inline int atomic_fetch_and(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_fetch_and(i, v); } static __always_inline s64 atomic64_fetch_and(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_fetch_and(i, v); } static __always_inline int atomic_fetch_or(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_fetch_or(i, v); } static __always_inline s64 atomic64_fetch_or(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_fetch_or(i, v); } static __always_inline int atomic_fetch_xor(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_fetch_xor(i, v); } static __always_inline s64 atomic64_fetch_xor(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_fetch_xor(i, v); } static __always_inline bool atomic_sub_and_test(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_sub_and_test(i, v); } static __always_inline bool atomic64_sub_and_test(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_sub_and_test(i, v); } static __always_inline bool atomic_add_negative(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_add_negative(i, v); } static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_add_negative(i, v); } #define cmpxchg(ptr, old, new) \ ({ \ + __typeof__(ptr) ___ptr = (ptr); \ + kasan_check_write(___ptr, sizeof(*___ptr)); \ arch_cmpxchg((ptr), (old), (new)); \ }) #define sync_cmpxchg(ptr, old, new) \ ({ \ - arch_sync_cmpxchg((ptr), (old), (new)); \ + __typeof__(ptr) ___ptr = (ptr); \ + kasan_check_write(___ptr, sizeof(*___ptr)); \ + arch_sync_cmpxchg(___ptr, (old), (new)); \ }) #define cmpxchg_local(ptr, old, new) \ ({ \ - arch_cmpxchg_local((ptr), (old), (new)); \ + __typeof__(ptr) ____ptr = (ptr); \ + kasan_check_write(____ptr, sizeof(*____ptr)); \ + arch_cmpxchg_local(____ptr, (old), (new)); \ }) #define cmpxchg64(ptr, old, new) \ ({ \ - arch_cmpxchg64((ptr), (old), (new)); \ + __typeof__(ptr) ____ptr = (ptr); \ + kasan_check_write(____ptr, sizeof(*____ptr)); \ + arch_cmpxchg64(____ptr, (old), (new)); \ }) #define cmpxchg64_local(ptr, old, new) \ ({ \ - arch_cmpxchg64_local((ptr), (old), (new)); \ + __typeof__(ptr) ____ptr = (ptr); \ + kasan_check_write(____ptr, sizeof(*____ptr)); \ + arch_cmpxchg64_local(____ptr, (old), (new)); \ }) #define cmpxchg_double(p1, p2, o1, o2, n1, n2) \ -- 2.13.1.611.g7e3b11ae1-goog ^ permalink raw reply related [flat|nested] 63+ messages in thread
* [PATCH v5 3/4] asm-generic: add KASAN instrumentation to atomic operations @ 2017-06-22 14:14 ` Dmitry Vyukov 0 siblings, 0 replies; 63+ messages in thread From: Dmitry Vyukov @ 2017-06-22 14:14 UTC (permalink / raw) To: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin, kasan-dev, x86, linux-kernel Cc: Dmitry Vyukov, Andrew Morton, linux-mm KASAN uses compiler instrumentation to intercept all memory accesses. But it does not see memory accesses done in assembly code. One notable user of assembly code is atomic operations. Frequently, for example, an atomic reference decrement is the last access to an object and a good candidate for a racy use-after-free. Add manual KASAN checks to atomic operations. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Will Deacon <will.deacon@arm.com>, Cc: Andrew Morton <akpm@linux-foundation.org>, Cc: Ingo Molnar <mingo@redhat.com>, Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org Cc: x86@kernel.org --- include/asm-generic/atomic-instrumented.h | 76 +++++++++++++++++++++++++++++-- 1 file changed, 72 insertions(+), 4 deletions(-) diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h index 50401d925290..a0f5b7525bb2 100644 --- a/include/asm-generic/atomic-instrumented.h +++ b/include/asm-generic/atomic-instrumented.h @@ -1,43 +1,53 @@ #ifndef _LINUX_ATOMIC_INSTRUMENTED_H #define _LINUX_ATOMIC_INSTRUMENTED_H +#include <linux/kasan-checks.h> + static __always_inline int atomic_read(const atomic_t *v) { + kasan_check_read(v, sizeof(*v)); return arch_atomic_read(v); } static __always_inline s64 atomic64_read(const atomic64_t *v) { + kasan_check_read(v, sizeof(*v)); return arch_atomic64_read(v); } static __always_inline void atomic_set(atomic_t *v, int i) { + kasan_check_write(v, sizeof(*v)); arch_atomic_set(v, i); } static __always_inline void atomic64_set(atomic64_t *v, s64 i) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_set(v, i); } static __always_inline int atomic_xchg(atomic_t *v, int i) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_xchg(v, i); } static __always_inline s64 atomic64_xchg(atomic64_t *v, s64 i) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_xchg(v, i); } static __always_inline int atomic_cmpxchg(atomic_t *v, int old, int new) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_cmpxchg(v, old, new); } static __always_inline s64 atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_cmpxchg(v, old, new); } @@ -45,6 +55,8 @@ static __always_inline s64 atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new) #define atomic_try_cmpxchg atomic_try_cmpxchg static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new) { + kasan_check_write(v, sizeof(*v)); + kasan_check_read(old, sizeof(*old)); return arch_atomic_try_cmpxchg(v, old, new); } #endif @@ -53,254 +65,310 @@ static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new) #define atomic64_try_cmpxchg atomic64_try_cmpxchg static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, s64 new) { + kasan_check_write(v, sizeof(*v)); + kasan_check_read(old, sizeof(*old)); return arch_atomic64_try_cmpxchg(v, old, new); } #endif static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u) { + kasan_check_write(v, sizeof(*v)); return __arch_atomic_add_unless(v, a, u); } static __always_inline bool atomic64_add_unless(atomic64_t *v, s64 a, s64 u) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_add_unless(v, a, u); } static __always_inline void atomic_inc(atomic_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic_inc(v); } static __always_inline void atomic64_inc(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_inc(v); } static __always_inline void atomic_dec(atomic_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic_dec(v); } static __always_inline void atomic64_dec(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_dec(v); } static __always_inline void atomic_add(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic_add(i, v); } static __always_inline void atomic64_add(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_add(i, v); } static __always_inline void atomic_sub(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic_sub(i, v); } static __always_inline void atomic64_sub(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_sub(i, v); } static __always_inline void atomic_and(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic_and(i, v); } static __always_inline void atomic64_and(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_and(i, v); } static __always_inline void atomic_or(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic_or(i, v); } static __always_inline void atomic64_or(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_or(i, v); } static __always_inline void atomic_xor(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic_xor(i, v); } static __always_inline void atomic64_xor(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_xor(i, v); } static __always_inline int atomic_inc_return(atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_inc_return(v); } static __always_inline s64 atomic64_inc_return(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_inc_return(v); } static __always_inline int atomic_dec_return(atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_dec_return(v); } static __always_inline s64 atomic64_dec_return(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_dec_return(v); } static __always_inline s64 atomic64_inc_not_zero(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_inc_not_zero(v); } static __always_inline s64 atomic64_dec_if_positive(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_dec_if_positive(v); } static __always_inline bool atomic_dec_and_test(atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_dec_and_test(v); } static __always_inline bool atomic64_dec_and_test(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_dec_and_test(v); } static __always_inline bool atomic_inc_and_test(atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_inc_and_test(v); } static __always_inline bool atomic64_inc_and_test(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_inc_and_test(v); } static __always_inline int atomic_add_return(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_add_return(i, v); } static __always_inline s64 atomic64_add_return(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_add_return(i, v); } static __always_inline int atomic_sub_return(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_sub_return(i, v); } static __always_inline s64 atomic64_sub_return(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_sub_return(i, v); } static __always_inline int atomic_fetch_add(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_fetch_add(i, v); } static __always_inline s64 atomic64_fetch_add(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_fetch_add(i, v); } static __always_inline int atomic_fetch_sub(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_fetch_sub(i, v); } static __always_inline s64 atomic64_fetch_sub(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_fetch_sub(i, v); } static __always_inline int atomic_fetch_and(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_fetch_and(i, v); } static __always_inline s64 atomic64_fetch_and(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_fetch_and(i, v); } static __always_inline int atomic_fetch_or(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_fetch_or(i, v); } static __always_inline s64 atomic64_fetch_or(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_fetch_or(i, v); } static __always_inline int atomic_fetch_xor(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_fetch_xor(i, v); } static __always_inline s64 atomic64_fetch_xor(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_fetch_xor(i, v); } static __always_inline bool atomic_sub_and_test(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_sub_and_test(i, v); } static __always_inline bool atomic64_sub_and_test(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_sub_and_test(i, v); } static __always_inline bool atomic_add_negative(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_add_negative(i, v); } static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_add_negative(i, v); } #define cmpxchg(ptr, old, new) \ ({ \ + __typeof__(ptr) ___ptr = (ptr); \ + kasan_check_write(___ptr, sizeof(*___ptr)); \ arch_cmpxchg((ptr), (old), (new)); \ }) #define sync_cmpxchg(ptr, old, new) \ ({ \ - arch_sync_cmpxchg((ptr), (old), (new)); \ + __typeof__(ptr) ___ptr = (ptr); \ + kasan_check_write(___ptr, sizeof(*___ptr)); \ + arch_sync_cmpxchg(___ptr, (old), (new)); \ }) #define cmpxchg_local(ptr, old, new) \ ({ \ - arch_cmpxchg_local((ptr), (old), (new)); \ + __typeof__(ptr) ____ptr = (ptr); \ + kasan_check_write(____ptr, sizeof(*____ptr)); \ + arch_cmpxchg_local(____ptr, (old), (new)); \ }) #define cmpxchg64(ptr, old, new) \ ({ \ - arch_cmpxchg64((ptr), (old), (new)); \ + __typeof__(ptr) ____ptr = (ptr); \ + kasan_check_write(____ptr, sizeof(*____ptr)); \ + arch_cmpxchg64(____ptr, (old), (new)); \ }) #define cmpxchg64_local(ptr, old, new) \ ({ \ - arch_cmpxchg64_local((ptr), (old), (new)); \ + __typeof__(ptr) ____ptr = (ptr); \ + kasan_check_write(____ptr, sizeof(*____ptr)); \ + arch_cmpxchg64_local(____ptr, (old), (new)); \ }) #define cmpxchg_double(p1, p2, o1, o2, n1, n2) \ -- 2.13.1.611.g7e3b11ae1-goog -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply related [flat|nested] 63+ messages in thread
* [tip:locking/core] locking/atomics, asm-generic: Add KASAN instrumentation to atomic operations 2017-06-22 14:14 ` Dmitry Vyukov (?) @ 2017-06-23 14:01 ` tip-bot for Dmitry Vyukov -1 siblings, 0 replies; 63+ messages in thread From: tip-bot for Dmitry Vyukov @ 2017-06-23 14:01 UTC (permalink / raw) To: linux-tip-commits Cc: torvalds, mingo, hpa, tglx, linux-kernel, aryabinin, dvyukov, peterz, mark.rutland Commit-ID: 235a93822a21ef6459f4f8eeb58f7bf48e02b450 Gitweb: http://git.kernel.org/tip/235a93822a21ef6459f4f8eeb58f7bf48e02b450 Author: Dmitry Vyukov <dvyukov@google.com> AuthorDate: Thu, 22 Jun 2017 16:14:18 +0200 Committer: Ingo Molnar <mingo@kernel.org> CommitDate: Fri, 23 Jun 2017 10:50:20 +0200 locking/atomics, asm-generic: Add KASAN instrumentation to atomic operations KASAN uses compiler instrumentation to intercept all memory accesses. But it does not see memory accesses done in assembly code. One notable user of assembly code is atomic operations. Frequently, for example, an atomic reference decrement is the last access to an object and a good candidate for a racy use-after-free. Add manual KASAN checks to atomic operations. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Reviewed-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org>, Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com>, Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/85d51d3551b676ba1fc40e8fbddd2eadd056d8dd.1498140838.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org> --- include/asm-generic/atomic-instrumented.h | 76 +++++++++++++++++++++++++++++-- 1 file changed, 72 insertions(+), 4 deletions(-) diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h index 50401d9..a0f5b75 100644 --- a/include/asm-generic/atomic-instrumented.h +++ b/include/asm-generic/atomic-instrumented.h @@ -1,43 +1,53 @@ #ifndef _LINUX_ATOMIC_INSTRUMENTED_H #define _LINUX_ATOMIC_INSTRUMENTED_H +#include <linux/kasan-checks.h> + static __always_inline int atomic_read(const atomic_t *v) { + kasan_check_read(v, sizeof(*v)); return arch_atomic_read(v); } static __always_inline s64 atomic64_read(const atomic64_t *v) { + kasan_check_read(v, sizeof(*v)); return arch_atomic64_read(v); } static __always_inline void atomic_set(atomic_t *v, int i) { + kasan_check_write(v, sizeof(*v)); arch_atomic_set(v, i); } static __always_inline void atomic64_set(atomic64_t *v, s64 i) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_set(v, i); } static __always_inline int atomic_xchg(atomic_t *v, int i) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_xchg(v, i); } static __always_inline s64 atomic64_xchg(atomic64_t *v, s64 i) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_xchg(v, i); } static __always_inline int atomic_cmpxchg(atomic_t *v, int old, int new) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_cmpxchg(v, old, new); } static __always_inline s64 atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_cmpxchg(v, old, new); } @@ -45,6 +55,8 @@ static __always_inline s64 atomic64_cmpxchg(atomic64_t *v, s64 old, s64 new) #define atomic_try_cmpxchg atomic_try_cmpxchg static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new) { + kasan_check_write(v, sizeof(*v)); + kasan_check_read(old, sizeof(*old)); return arch_atomic_try_cmpxchg(v, old, new); } #endif @@ -53,254 +65,310 @@ static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new) #define atomic64_try_cmpxchg atomic64_try_cmpxchg static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, s64 new) { + kasan_check_write(v, sizeof(*v)); + kasan_check_read(old, sizeof(*old)); return arch_atomic64_try_cmpxchg(v, old, new); } #endif static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u) { + kasan_check_write(v, sizeof(*v)); return __arch_atomic_add_unless(v, a, u); } static __always_inline bool atomic64_add_unless(atomic64_t *v, s64 a, s64 u) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_add_unless(v, a, u); } static __always_inline void atomic_inc(atomic_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic_inc(v); } static __always_inline void atomic64_inc(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_inc(v); } static __always_inline void atomic_dec(atomic_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic_dec(v); } static __always_inline void atomic64_dec(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_dec(v); } static __always_inline void atomic_add(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic_add(i, v); } static __always_inline void atomic64_add(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_add(i, v); } static __always_inline void atomic_sub(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic_sub(i, v); } static __always_inline void atomic64_sub(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_sub(i, v); } static __always_inline void atomic_and(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic_and(i, v); } static __always_inline void atomic64_and(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_and(i, v); } static __always_inline void atomic_or(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic_or(i, v); } static __always_inline void atomic64_or(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_or(i, v); } static __always_inline void atomic_xor(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic_xor(i, v); } static __always_inline void atomic64_xor(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); arch_atomic64_xor(i, v); } static __always_inline int atomic_inc_return(atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_inc_return(v); } static __always_inline s64 atomic64_inc_return(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_inc_return(v); } static __always_inline int atomic_dec_return(atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_dec_return(v); } static __always_inline s64 atomic64_dec_return(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_dec_return(v); } static __always_inline s64 atomic64_inc_not_zero(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_inc_not_zero(v); } static __always_inline s64 atomic64_dec_if_positive(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_dec_if_positive(v); } static __always_inline bool atomic_dec_and_test(atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_dec_and_test(v); } static __always_inline bool atomic64_dec_and_test(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_dec_and_test(v); } static __always_inline bool atomic_inc_and_test(atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_inc_and_test(v); } static __always_inline bool atomic64_inc_and_test(atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_inc_and_test(v); } static __always_inline int atomic_add_return(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_add_return(i, v); } static __always_inline s64 atomic64_add_return(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_add_return(i, v); } static __always_inline int atomic_sub_return(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_sub_return(i, v); } static __always_inline s64 atomic64_sub_return(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_sub_return(i, v); } static __always_inline int atomic_fetch_add(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_fetch_add(i, v); } static __always_inline s64 atomic64_fetch_add(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_fetch_add(i, v); } static __always_inline int atomic_fetch_sub(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_fetch_sub(i, v); } static __always_inline s64 atomic64_fetch_sub(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_fetch_sub(i, v); } static __always_inline int atomic_fetch_and(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_fetch_and(i, v); } static __always_inline s64 atomic64_fetch_and(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_fetch_and(i, v); } static __always_inline int atomic_fetch_or(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_fetch_or(i, v); } static __always_inline s64 atomic64_fetch_or(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_fetch_or(i, v); } static __always_inline int atomic_fetch_xor(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_fetch_xor(i, v); } static __always_inline s64 atomic64_fetch_xor(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_fetch_xor(i, v); } static __always_inline bool atomic_sub_and_test(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_sub_and_test(i, v); } static __always_inline bool atomic64_sub_and_test(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_sub_and_test(i, v); } static __always_inline bool atomic_add_negative(int i, atomic_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic_add_negative(i, v); } static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v) { + kasan_check_write(v, sizeof(*v)); return arch_atomic64_add_negative(i, v); } #define cmpxchg(ptr, old, new) \ ({ \ + __typeof__(ptr) ___ptr = (ptr); \ + kasan_check_write(___ptr, sizeof(*___ptr)); \ arch_cmpxchg((ptr), (old), (new)); \ }) #define sync_cmpxchg(ptr, old, new) \ ({ \ - arch_sync_cmpxchg((ptr), (old), (new)); \ + __typeof__(ptr) ___ptr = (ptr); \ + kasan_check_write(___ptr, sizeof(*___ptr)); \ + arch_sync_cmpxchg(___ptr, (old), (new)); \ }) #define cmpxchg_local(ptr, old, new) \ ({ \ - arch_cmpxchg_local((ptr), (old), (new)); \ + __typeof__(ptr) ____ptr = (ptr); \ + kasan_check_write(____ptr, sizeof(*____ptr)); \ + arch_cmpxchg_local(____ptr, (old), (new)); \ }) #define cmpxchg64(ptr, old, new) \ ({ \ - arch_cmpxchg64((ptr), (old), (new)); \ + __typeof__(ptr) ____ptr = (ptr); \ + kasan_check_write(____ptr, sizeof(*____ptr)); \ + arch_cmpxchg64(____ptr, (old), (new)); \ }) #define cmpxchg64_local(ptr, old, new) \ ({ \ - arch_cmpxchg64_local((ptr), (old), (new)); \ + __typeof__(ptr) ____ptr = (ptr); \ + kasan_check_write(____ptr, sizeof(*____ptr)); \ + arch_cmpxchg64_local(____ptr, (old), (new)); \ }) #define cmpxchg_double(p1, p2, o1, o2, n1, n2) \ ^ permalink raw reply related [flat|nested] 63+ messages in thread
* [PATCH] locking/atomics: don't alias ____ptr 2017-06-22 14:14 ` Dmitry Vyukov @ 2017-06-28 10:02 ` Sebastian Andrzej Siewior -1 siblings, 0 replies; 63+ messages in thread From: Sebastian Andrzej Siewior @ 2017-06-28 10:02 UTC (permalink / raw) To: Dmitry Vyukov, Ingo Molnar Cc: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin, kasan-dev, x86, linux-kernel, Andrew Morton, linux-mm, Linus Torvalds, Thomas Gleixner Trying to boot tip/master resulted in: |DMAR: dmar0: Using Queued invalidation |DMAR: dmar1: Using Queued invalidation |DMAR: Setting RMRR: |DMAR: Setting identity map for device 0000:00:1a.0 [0xbdcf9000 - 0xbdd1dfff] |BUG: unable to handle kernel NULL pointer dereference at (null) |IP: __domain_mapping+0x10f/0x3d0 |PGD 0 |P4D 0 | |Oops: 0002 [#1] PREEMPT SMP |Modules linked in: |CPU: 19 PID: 1 Comm: swapper/0 Not tainted 4.12.0-rc6-00117-g235a93822a21 #113 |task: ffff8805271c2c80 task.stack: ffffc90000058000 |RIP: 0010:__domain_mapping+0x10f/0x3d0 |RSP: 0000:ffffc9000005bca0 EFLAGS: 00010246 |RAX: 0000000000000000 RBX: 00000000bdcf9003 RCX: 0000000000000000 |RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000001 |RBP: ffffc9000005bd00 R08: ffff880a243e9780 R09: ffff8805259e67c8 |R10: 00000000000bdcf9 R11: 0000000000000000 R12: 0000000000000025 |R13: 0000000000000025 R14: 0000000000000000 R15: 00000000000bdcf9 |FS: 0000000000000000(0000) GS:ffff88052acc0000(0000) knlGS:0000000000000000 |CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 |CR2: 0000000000000000 CR3: 0000000001c0f000 CR4: 00000000000406e0 |Call Trace: | iommu_domain_identity_map+0x5a/0x80 | domain_prepare_identity_map+0x9f/0x160 | iommu_prepare_identity_map+0x7e/0x9b bisect points to commit 235a93822a21 ("locking/atomics, asm-generic: Add KASAN instrumentation to atomic operations"), RIP is at tmp = cmpxchg64_local(&pte->val, 0ULL, pteval); in drivers/iommu/intel-iommu.c. The assembly for this inline assembly is: xor %edx,%edx xor %eax,%eax cmpxchg %rbx,(%rdx) and as you see edx is set to zero and used later as a pointer via the full register. This happens with gcc-6, 5 and 8 (snapshot from last week). After a longer while of searching and swearing I figured out that this bug occures once cmpxchg64_local() and cmpxchg_local() uses the same ____ptr macro and they are shadow somehow. What I don't know why edx is set to zero. Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> --- include/asm-generic/atomic-instrumented.h | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h index a0f5b7525bb2..ac6155362b39 100644 --- a/include/asm-generic/atomic-instrumented.h +++ b/include/asm-generic/atomic-instrumented.h @@ -359,16 +359,16 @@ static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v) #define cmpxchg64(ptr, old, new) \ ({ \ - __typeof__(ptr) ____ptr = (ptr); \ - kasan_check_write(____ptr, sizeof(*____ptr)); \ - arch_cmpxchg64(____ptr, (old), (new)); \ + __typeof__(ptr) ____ptr64 = (ptr); \ + kasan_check_write(____ptr64, sizeof(*____ptr64));\ + arch_cmpxchg64(____ptr64, (old), (new)); \ }) #define cmpxchg64_local(ptr, old, new) \ ({ \ - __typeof__(ptr) ____ptr = (ptr); \ - kasan_check_write(____ptr, sizeof(*____ptr)); \ - arch_cmpxchg64_local(____ptr, (old), (new)); \ + __typeof__(ptr) ____ptr64 = (ptr); \ + kasan_check_write(____ptr64, sizeof(*____ptr64));\ + arch_cmpxchg64_local(____ptr64, (old), (new)); \ }) #define cmpxchg_double(p1, p2, o1, o2, n1, n2) \ -- 2.13.2 ^ permalink raw reply related [flat|nested] 63+ messages in thread
* [PATCH] locking/atomics: don't alias ____ptr @ 2017-06-28 10:02 ` Sebastian Andrzej Siewior 0 siblings, 0 replies; 63+ messages in thread From: Sebastian Andrzej Siewior @ 2017-06-28 10:02 UTC (permalink / raw) To: Dmitry Vyukov, Ingo Molnar Cc: mark.rutland, peterz, will.deacon, hpa, aryabinin, kasan-dev, x86, linux-kernel, Andrew Morton, linux-mm, Linus Torvalds, Thomas Gleixner Trying to boot tip/master resulted in: |DMAR: dmar0: Using Queued invalidation |DMAR: dmar1: Using Queued invalidation |DMAR: Setting RMRR: |DMAR: Setting identity map for device 0000:00:1a.0 [0xbdcf9000 - 0xbdd1dfff] |BUG: unable to handle kernel NULL pointer dereference at (null) |IP: __domain_mapping+0x10f/0x3d0 |PGD 0 |P4D 0 | |Oops: 0002 [#1] PREEMPT SMP |Modules linked in: |CPU: 19 PID: 1 Comm: swapper/0 Not tainted 4.12.0-rc6-00117-g235a93822a21 #113 |task: ffff8805271c2c80 task.stack: ffffc90000058000 |RIP: 0010:__domain_mapping+0x10f/0x3d0 |RSP: 0000:ffffc9000005bca0 EFLAGS: 00010246 |RAX: 0000000000000000 RBX: 00000000bdcf9003 RCX: 0000000000000000 |RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000001 |RBP: ffffc9000005bd00 R08: ffff880a243e9780 R09: ffff8805259e67c8 |R10: 00000000000bdcf9 R11: 0000000000000000 R12: 0000000000000025 |R13: 0000000000000025 R14: 0000000000000000 R15: 00000000000bdcf9 |FS: 0000000000000000(0000) GS:ffff88052acc0000(0000) knlGS:0000000000000000 |CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 |CR2: 0000000000000000 CR3: 0000000001c0f000 CR4: 00000000000406e0 |Call Trace: | iommu_domain_identity_map+0x5a/0x80 | domain_prepare_identity_map+0x9f/0x160 | iommu_prepare_identity_map+0x7e/0x9b bisect points to commit 235a93822a21 ("locking/atomics, asm-generic: Add KASAN instrumentation to atomic operations"), RIP is at tmp = cmpxchg64_local(&pte->val, 0ULL, pteval); in drivers/iommu/intel-iommu.c. The assembly for this inline assembly is: xor %edx,%edx xor %eax,%eax cmpxchg %rbx,(%rdx) and as you see edx is set to zero and used later as a pointer via the full register. This happens with gcc-6, 5 and 8 (snapshot from last week). After a longer while of searching and swearing I figured out that this bug occures once cmpxchg64_local() and cmpxchg_local() uses the same ____ptr macro and they are shadow somehow. What I don't know why edx is set to zero. Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> --- include/asm-generic/atomic-instrumented.h | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h index a0f5b7525bb2..ac6155362b39 100644 --- a/include/asm-generic/atomic-instrumented.h +++ b/include/asm-generic/atomic-instrumented.h @@ -359,16 +359,16 @@ static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v) #define cmpxchg64(ptr, old, new) \ ({ \ - __typeof__(ptr) ____ptr = (ptr); \ - kasan_check_write(____ptr, sizeof(*____ptr)); \ - arch_cmpxchg64(____ptr, (old), (new)); \ + __typeof__(ptr) ____ptr64 = (ptr); \ + kasan_check_write(____ptr64, sizeof(*____ptr64));\ + arch_cmpxchg64(____ptr64, (old), (new)); \ }) #define cmpxchg64_local(ptr, old, new) \ ({ \ - __typeof__(ptr) ____ptr = (ptr); \ - kasan_check_write(____ptr, sizeof(*____ptr)); \ - arch_cmpxchg64_local(____ptr, (old), (new)); \ + __typeof__(ptr) ____ptr64 = (ptr); \ + kasan_check_write(____ptr64, sizeof(*____ptr64));\ + arch_cmpxchg64_local(____ptr64, (old), (new)); \ }) #define cmpxchg_double(p1, p2, o1, o2, n1, n2) \ -- 2.13.2 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply related [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr 2017-06-28 10:02 ` Sebastian Andrzej Siewior @ 2017-06-28 10:16 ` Dmitry Vyukov -1 siblings, 0 replies; 63+ messages in thread From: Dmitry Vyukov @ 2017-06-28 10:16 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: Ingo Molnar, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds, Thomas Gleixner On Wed, Jun 28, 2017 at 12:02 PM, Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote: > Trying to boot tip/master resulted in: > |DMAR: dmar0: Using Queued invalidation > |DMAR: dmar1: Using Queued invalidation > |DMAR: Setting RMRR: > |DMAR: Setting identity map for device 0000:00:1a.0 [0xbdcf9000 - 0xbdd1dfff] > |BUG: unable to handle kernel NULL pointer dereference at (null) > |IP: __domain_mapping+0x10f/0x3d0 > |PGD 0 > |P4D 0 > | > |Oops: 0002 [#1] PREEMPT SMP > |Modules linked in: > |CPU: 19 PID: 1 Comm: swapper/0 Not tainted 4.12.0-rc6-00117-g235a93822a21 #113 > |task: ffff8805271c2c80 task.stack: ffffc90000058000 > |RIP: 0010:__domain_mapping+0x10f/0x3d0 > |RSP: 0000:ffffc9000005bca0 EFLAGS: 00010246 > |RAX: 0000000000000000 RBX: 00000000bdcf9003 RCX: 0000000000000000 > |RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000001 > |RBP: ffffc9000005bd00 R08: ffff880a243e9780 R09: ffff8805259e67c8 > |R10: 00000000000bdcf9 R11: 0000000000000000 R12: 0000000000000025 > |R13: 0000000000000025 R14: 0000000000000000 R15: 00000000000bdcf9 > |FS: 0000000000000000(0000) GS:ffff88052acc0000(0000) knlGS:0000000000000000 > |CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > |CR2: 0000000000000000 CR3: 0000000001c0f000 CR4: 00000000000406e0 > |Call Trace: > | iommu_domain_identity_map+0x5a/0x80 > | domain_prepare_identity_map+0x9f/0x160 > | iommu_prepare_identity_map+0x7e/0x9b > > bisect points to commit 235a93822a21 ("locking/atomics, asm-generic: Add KASAN > instrumentation to atomic operations"), RIP is at > tmp = cmpxchg64_local(&pte->val, 0ULL, pteval); > in drivers/iommu/intel-iommu.c. The assembly for this inline assembly > is: > xor %edx,%edx > xor %eax,%eax > cmpxchg %rbx,(%rdx) > > and as you see edx is set to zero and used later as a pointer via the > full register. This happens with gcc-6, 5 and 8 (snapshot from last > week). > After a longer while of searching and swearing I figured out that this > bug occures once cmpxchg64_local() and cmpxchg_local() uses the same > ____ptr macro and they are shadow somehow. What I don't know why edx is > set to zero. > > Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> > --- > include/asm-generic/atomic-instrumented.h | 12 ++++++------ > 1 file changed, 6 insertions(+), 6 deletions(-) > > diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h > index a0f5b7525bb2..ac6155362b39 100644 > --- a/include/asm-generic/atomic-instrumented.h > +++ b/include/asm-generic/atomic-instrumented.h > @@ -359,16 +359,16 @@ static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v) > > #define cmpxchg64(ptr, old, new) \ > ({ \ > - __typeof__(ptr) ____ptr = (ptr); \ > - kasan_check_write(____ptr, sizeof(*____ptr)); \ > - arch_cmpxchg64(____ptr, (old), (new)); \ > + __typeof__(ptr) ____ptr64 = (ptr); \ > + kasan_check_write(____ptr64, sizeof(*____ptr64));\ > + arch_cmpxchg64(____ptr64, (old), (new)); \ > }) > > #define cmpxchg64_local(ptr, old, new) \ > ({ \ > - __typeof__(ptr) ____ptr = (ptr); \ > - kasan_check_write(____ptr, sizeof(*____ptr)); \ > - arch_cmpxchg64_local(____ptr, (old), (new)); \ > + __typeof__(ptr) ____ptr64 = (ptr); \ > + kasan_check_write(____ptr64, sizeof(*____ptr64));\ > + arch_cmpxchg64_local(____ptr64, (old), (new)); \ > }) > > #define cmpxchg_double(p1, p2, o1, o2, n1, n2) \ Doh! Thanks for fixing this. I think I've a similar crash in a different place when I developed the patch. The problem is that when we do: __typeof__(ptr) ____ptr = (ptr); \ arch_cmpxchg64_local(____ptr, (old), (new)); \ We don't necessary pass value of our just declared ____ptr to arch_cmpxchg64_local(). We just pass a symbolic identifier. So if arch_cmpxchg64_local() declares own ____ptr and then tries to use what we passed ("____ptr") it will actually refer to own variable declared rather than to what we wanted to pass in. In my case I ended up with something like: __typeof__(foo) __ptr = __ptr; which compiler decided to turn into 0. Thank you, macros. We can add more underscores, but the problem can happen again. Should we prefix current function/macro name to all local vars?.. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr @ 2017-06-28 10:16 ` Dmitry Vyukov 0 siblings, 0 replies; 63+ messages in thread From: Dmitry Vyukov @ 2017-06-28 10:16 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: Ingo Molnar, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds, Thomas Gleixner On Wed, Jun 28, 2017 at 12:02 PM, Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote: > Trying to boot tip/master resulted in: > |DMAR: dmar0: Using Queued invalidation > |DMAR: dmar1: Using Queued invalidation > |DMAR: Setting RMRR: > |DMAR: Setting identity map for device 0000:00:1a.0 [0xbdcf9000 - 0xbdd1dfff] > |BUG: unable to handle kernel NULL pointer dereference at (null) > |IP: __domain_mapping+0x10f/0x3d0 > |PGD 0 > |P4D 0 > | > |Oops: 0002 [#1] PREEMPT SMP > |Modules linked in: > |CPU: 19 PID: 1 Comm: swapper/0 Not tainted 4.12.0-rc6-00117-g235a93822a21 #113 > |task: ffff8805271c2c80 task.stack: ffffc90000058000 > |RIP: 0010:__domain_mapping+0x10f/0x3d0 > |RSP: 0000:ffffc9000005bca0 EFLAGS: 00010246 > |RAX: 0000000000000000 RBX: 00000000bdcf9003 RCX: 0000000000000000 > |RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000001 > |RBP: ffffc9000005bd00 R08: ffff880a243e9780 R09: ffff8805259e67c8 > |R10: 00000000000bdcf9 R11: 0000000000000000 R12: 0000000000000025 > |R13: 0000000000000025 R14: 0000000000000000 R15: 00000000000bdcf9 > |FS: 0000000000000000(0000) GS:ffff88052acc0000(0000) knlGS:0000000000000000 > |CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > |CR2: 0000000000000000 CR3: 0000000001c0f000 CR4: 00000000000406e0 > |Call Trace: > | iommu_domain_identity_map+0x5a/0x80 > | domain_prepare_identity_map+0x9f/0x160 > | iommu_prepare_identity_map+0x7e/0x9b > > bisect points to commit 235a93822a21 ("locking/atomics, asm-generic: Add KASAN > instrumentation to atomic operations"), RIP is at > tmp = cmpxchg64_local(&pte->val, 0ULL, pteval); > in drivers/iommu/intel-iommu.c. The assembly for this inline assembly > is: > xor %edx,%edx > xor %eax,%eax > cmpxchg %rbx,(%rdx) > > and as you see edx is set to zero and used later as a pointer via the > full register. This happens with gcc-6, 5 and 8 (snapshot from last > week). > After a longer while of searching and swearing I figured out that this > bug occures once cmpxchg64_local() and cmpxchg_local() uses the same > ____ptr macro and they are shadow somehow. What I don't know why edx is > set to zero. > > Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> > --- > include/asm-generic/atomic-instrumented.h | 12 ++++++------ > 1 file changed, 6 insertions(+), 6 deletions(-) > > diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h > index a0f5b7525bb2..ac6155362b39 100644 > --- a/include/asm-generic/atomic-instrumented.h > +++ b/include/asm-generic/atomic-instrumented.h > @@ -359,16 +359,16 @@ static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v) > > #define cmpxchg64(ptr, old, new) \ > ({ \ > - __typeof__(ptr) ____ptr = (ptr); \ > - kasan_check_write(____ptr, sizeof(*____ptr)); \ > - arch_cmpxchg64(____ptr, (old), (new)); \ > + __typeof__(ptr) ____ptr64 = (ptr); \ > + kasan_check_write(____ptr64, sizeof(*____ptr64));\ > + arch_cmpxchg64(____ptr64, (old), (new)); \ > }) > > #define cmpxchg64_local(ptr, old, new) \ > ({ \ > - __typeof__(ptr) ____ptr = (ptr); \ > - kasan_check_write(____ptr, sizeof(*____ptr)); \ > - arch_cmpxchg64_local(____ptr, (old), (new)); \ > + __typeof__(ptr) ____ptr64 = (ptr); \ > + kasan_check_write(____ptr64, sizeof(*____ptr64));\ > + arch_cmpxchg64_local(____ptr64, (old), (new)); \ > }) > > #define cmpxchg_double(p1, p2, o1, o2, n1, n2) \ Doh! Thanks for fixing this. I think I've a similar crash in a different place when I developed the patch. The problem is that when we do: __typeof__(ptr) ____ptr = (ptr); \ arch_cmpxchg64_local(____ptr, (old), (new)); \ We don't necessary pass value of our just declared ____ptr to arch_cmpxchg64_local(). We just pass a symbolic identifier. So if arch_cmpxchg64_local() declares own ____ptr and then tries to use what we passed ("____ptr") it will actually refer to own variable declared rather than to what we wanted to pass in. In my case I ended up with something like: __typeof__(foo) __ptr = __ptr; which compiler decided to turn into 0. Thank you, macros. We can add more underscores, but the problem can happen again. Should we prefix current function/macro name to all local vars?.. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr 2017-06-28 10:16 ` Dmitry Vyukov @ 2017-06-28 11:10 ` Thomas Gleixner -1 siblings, 0 replies; 63+ messages in thread From: Thomas Gleixner @ 2017-06-28 11:10 UTC (permalink / raw) To: Dmitry Vyukov Cc: Sebastian Andrzej Siewior, Ingo Molnar, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, 28 Jun 2017, Dmitry Vyukov wrote: > On Wed, Jun 28, 2017 at 12:02 PM, Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote: > > @@ -359,16 +359,16 @@ static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v) > > > > #define cmpxchg64(ptr, old, new) \ > > ({ \ > > - __typeof__(ptr) ____ptr = (ptr); \ > > - kasan_check_write(____ptr, sizeof(*____ptr)); \ > > - arch_cmpxchg64(____ptr, (old), (new)); \ > > + __typeof__(ptr) ____ptr64 = (ptr); \ > > + kasan_check_write(____ptr64, sizeof(*____ptr64));\ > > + arch_cmpxchg64(____ptr64, (old), (new)); \ > > }) > > > > #define cmpxchg64_local(ptr, old, new) \ > > ({ \ > > - __typeof__(ptr) ____ptr = (ptr); \ > > - kasan_check_write(____ptr, sizeof(*____ptr)); \ > > - arch_cmpxchg64_local(____ptr, (old), (new)); \ > > + __typeof__(ptr) ____ptr64 = (ptr); \ > > + kasan_check_write(____ptr64, sizeof(*____ptr64));\ > > + arch_cmpxchg64_local(____ptr64, (old), (new)); \ > > }) > > > > #define cmpxchg_double(p1, p2, o1, o2, n1, n2) \ > > > Doh! Thanks for fixing this. I think I've a similar crash in a > different place when I developed the patch. > The problem is that when we do: > > __typeof__(ptr) ____ptr = (ptr); \ > arch_cmpxchg64_local(____ptr, (old), (new)); \ > > We don't necessary pass value of our just declared ____ptr to > arch_cmpxchg64_local(). We just pass a symbolic identifier. So if > arch_cmpxchg64_local() declares own ____ptr and then tries to use what > we passed ("____ptr") it will actually refer to own variable declared > rather than to what we wanted to pass in. > > In my case I ended up with something like: > > __typeof__(foo) __ptr = __ptr; > > which compiler decided to turn into 0. > > Thank you, macros. > > We can add more underscores, but the problem can happen again. Should > we prefix current function/macro name to all local vars?.. Actually we can void that ___ptr dance completely. Thanks, tglx 8<-------------------- --- a/include/asm-generic/atomic-instrumented.h +++ b/include/asm-generic/atomic-instrumented.h @@ -359,37 +359,32 @@ static __always_inline bool atomic64_add #define cmpxchg(ptr, old, new) \ ({ \ - __typeof__(ptr) ___ptr = (ptr); \ - kasan_check_write(___ptr, sizeof(*___ptr)); \ + kasan_check_write((ptr), sizeof(*(ptr))); \ arch_cmpxchg((ptr), (old), (new)); \ }) #define sync_cmpxchg(ptr, old, new) \ ({ \ - __typeof__(ptr) ___ptr = (ptr); \ - kasan_check_write(___ptr, sizeof(*___ptr)); \ - arch_sync_cmpxchg(___ptr, (old), (new)); \ + kasan_check_write((ptr), sizeof(*(ptr))); \ + arch_sync_cmpxchg((ptr), (old), (new)); \ }) #define cmpxchg_local(ptr, old, new) \ ({ \ - __typeof__(ptr) ____ptr = (ptr); \ - kasan_check_write(____ptr, sizeof(*____ptr)); \ - arch_cmpxchg_local(____ptr, (old), (new)); \ + kasan_check_write((ptr), sizeof(*(ptr))); \ + arch_cmpxchg_local((ptr), (old), (new)); \ }) #define cmpxchg64(ptr, old, new) \ ({ \ - __typeof__(ptr) ____ptr = (ptr); \ - kasan_check_write(____ptr, sizeof(*____ptr)); \ - arch_cmpxchg64(____ptr, (old), (new)); \ + kasan_check_write((ptr), sizeof(*(ptr))); \ + arch_cmpxchg64((ptr), (old), (new)); \ }) #define cmpxchg64_local(ptr, old, new) \ ({ \ - __typeof__(ptr) ____ptr = (ptr); \ - kasan_check_write(____ptr, sizeof(*____ptr)); \ - arch_cmpxchg64_local(____ptr, (old), (new)); \ + kasan_check_write((ptr), sizeof(*(ptr))); \ + arch_cmpxchg64_local((ptr), (old), (new)); \ }) /* ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr @ 2017-06-28 11:10 ` Thomas Gleixner 0 siblings, 0 replies; 63+ messages in thread From: Thomas Gleixner @ 2017-06-28 11:10 UTC (permalink / raw) To: Dmitry Vyukov Cc: Sebastian Andrzej Siewior, Ingo Molnar, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, 28 Jun 2017, Dmitry Vyukov wrote: > On Wed, Jun 28, 2017 at 12:02 PM, Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote: > > @@ -359,16 +359,16 @@ static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v) > > > > #define cmpxchg64(ptr, old, new) \ > > ({ \ > > - __typeof__(ptr) ____ptr = (ptr); \ > > - kasan_check_write(____ptr, sizeof(*____ptr)); \ > > - arch_cmpxchg64(____ptr, (old), (new)); \ > > + __typeof__(ptr) ____ptr64 = (ptr); \ > > + kasan_check_write(____ptr64, sizeof(*____ptr64));\ > > + arch_cmpxchg64(____ptr64, (old), (new)); \ > > }) > > > > #define cmpxchg64_local(ptr, old, new) \ > > ({ \ > > - __typeof__(ptr) ____ptr = (ptr); \ > > - kasan_check_write(____ptr, sizeof(*____ptr)); \ > > - arch_cmpxchg64_local(____ptr, (old), (new)); \ > > + __typeof__(ptr) ____ptr64 = (ptr); \ > > + kasan_check_write(____ptr64, sizeof(*____ptr64));\ > > + arch_cmpxchg64_local(____ptr64, (old), (new)); \ > > }) > > > > #define cmpxchg_double(p1, p2, o1, o2, n1, n2) \ > > > Doh! Thanks for fixing this. I think I've a similar crash in a > different place when I developed the patch. > The problem is that when we do: > > __typeof__(ptr) ____ptr = (ptr); \ > arch_cmpxchg64_local(____ptr, (old), (new)); \ > > We don't necessary pass value of our just declared ____ptr to > arch_cmpxchg64_local(). We just pass a symbolic identifier. So if > arch_cmpxchg64_local() declares own ____ptr and then tries to use what > we passed ("____ptr") it will actually refer to own variable declared > rather than to what we wanted to pass in. > > In my case I ended up with something like: > > __typeof__(foo) __ptr = __ptr; > > which compiler decided to turn into 0. > > Thank you, macros. > > We can add more underscores, but the problem can happen again. Should > we prefix current function/macro name to all local vars?.. Actually we can void that ___ptr dance completely. Thanks, tglx 8<-------------------- --- a/include/asm-generic/atomic-instrumented.h +++ b/include/asm-generic/atomic-instrumented.h @@ -359,37 +359,32 @@ static __always_inline bool atomic64_add #define cmpxchg(ptr, old, new) \ ({ \ - __typeof__(ptr) ___ptr = (ptr); \ - kasan_check_write(___ptr, sizeof(*___ptr)); \ + kasan_check_write((ptr), sizeof(*(ptr))); \ arch_cmpxchg((ptr), (old), (new)); \ }) #define sync_cmpxchg(ptr, old, new) \ ({ \ - __typeof__(ptr) ___ptr = (ptr); \ - kasan_check_write(___ptr, sizeof(*___ptr)); \ - arch_sync_cmpxchg(___ptr, (old), (new)); \ + kasan_check_write((ptr), sizeof(*(ptr))); \ + arch_sync_cmpxchg((ptr), (old), (new)); \ }) #define cmpxchg_local(ptr, old, new) \ ({ \ - __typeof__(ptr) ____ptr = (ptr); \ - kasan_check_write(____ptr, sizeof(*____ptr)); \ - arch_cmpxchg_local(____ptr, (old), (new)); \ + kasan_check_write((ptr), sizeof(*(ptr))); \ + arch_cmpxchg_local((ptr), (old), (new)); \ }) #define cmpxchg64(ptr, old, new) \ ({ \ - __typeof__(ptr) ____ptr = (ptr); \ - kasan_check_write(____ptr, sizeof(*____ptr)); \ - arch_cmpxchg64(____ptr, (old), (new)); \ + kasan_check_write((ptr), sizeof(*(ptr))); \ + arch_cmpxchg64((ptr), (old), (new)); \ }) #define cmpxchg64_local(ptr, old, new) \ ({ \ - __typeof__(ptr) ____ptr = (ptr); \ - kasan_check_write(____ptr, sizeof(*____ptr)); \ - arch_cmpxchg64_local(____ptr, (old), (new)); \ + kasan_check_write((ptr), sizeof(*(ptr))); \ + arch_cmpxchg64_local((ptr), (old), (new)); \ }) /* -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr 2017-06-28 11:10 ` Thomas Gleixner @ 2017-06-28 11:12 ` Dmitry Vyukov -1 siblings, 0 replies; 63+ messages in thread From: Dmitry Vyukov @ 2017-06-28 11:12 UTC (permalink / raw) To: Thomas Gleixner Cc: Sebastian Andrzej Siewior, Ingo Molnar, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, Jun 28, 2017 at 1:10 PM, Thomas Gleixner <tglx@linutronix.de> wrote: > On Wed, 28 Jun 2017, Dmitry Vyukov wrote: >> On Wed, Jun 28, 2017 at 12:02 PM, Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote: >> > @@ -359,16 +359,16 @@ static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v) >> > >> > #define cmpxchg64(ptr, old, new) \ >> > ({ \ >> > - __typeof__(ptr) ____ptr = (ptr); \ >> > - kasan_check_write(____ptr, sizeof(*____ptr)); \ >> > - arch_cmpxchg64(____ptr, (old), (new)); \ >> > + __typeof__(ptr) ____ptr64 = (ptr); \ >> > + kasan_check_write(____ptr64, sizeof(*____ptr64));\ >> > + arch_cmpxchg64(____ptr64, (old), (new)); \ >> > }) >> > >> > #define cmpxchg64_local(ptr, old, new) \ >> > ({ \ >> > - __typeof__(ptr) ____ptr = (ptr); \ >> > - kasan_check_write(____ptr, sizeof(*____ptr)); \ >> > - arch_cmpxchg64_local(____ptr, (old), (new)); \ >> > + __typeof__(ptr) ____ptr64 = (ptr); \ >> > + kasan_check_write(____ptr64, sizeof(*____ptr64));\ >> > + arch_cmpxchg64_local(____ptr64, (old), (new)); \ >> > }) >> > >> > #define cmpxchg_double(p1, p2, o1, o2, n1, n2) \ >> >> >> Doh! Thanks for fixing this. I think I've a similar crash in a >> different place when I developed the patch. >> The problem is that when we do: >> >> __typeof__(ptr) ____ptr = (ptr); \ >> arch_cmpxchg64_local(____ptr, (old), (new)); \ >> >> We don't necessary pass value of our just declared ____ptr to >> arch_cmpxchg64_local(). We just pass a symbolic identifier. So if >> arch_cmpxchg64_local() declares own ____ptr and then tries to use what >> we passed ("____ptr") it will actually refer to own variable declared >> rather than to what we wanted to pass in. >> >> In my case I ended up with something like: >> >> __typeof__(foo) __ptr = __ptr; >> >> which compiler decided to turn into 0. >> >> Thank you, macros. >> >> We can add more underscores, but the problem can happen again. Should >> we prefix current function/macro name to all local vars?.. > > Actually we can void that ___ptr dance completely. > > Thanks, > > tglx > > 8<-------------------- > > --- a/include/asm-generic/atomic-instrumented.h > +++ b/include/asm-generic/atomic-instrumented.h > @@ -359,37 +359,32 @@ static __always_inline bool atomic64_add > > #define cmpxchg(ptr, old, new) \ > ({ \ > - __typeof__(ptr) ___ptr = (ptr); \ > - kasan_check_write(___ptr, sizeof(*___ptr)); \ > + kasan_check_write((ptr), sizeof(*(ptr))); \ > arch_cmpxchg((ptr), (old), (new)); \ > }) > > #define sync_cmpxchg(ptr, old, new) \ > ({ \ > - __typeof__(ptr) ___ptr = (ptr); \ > - kasan_check_write(___ptr, sizeof(*___ptr)); \ > - arch_sync_cmpxchg(___ptr, (old), (new)); \ > + kasan_check_write((ptr), sizeof(*(ptr))); \ > + arch_sync_cmpxchg((ptr), (old), (new)); \ > }) > > #define cmpxchg_local(ptr, old, new) \ > ({ \ > - __typeof__(ptr) ____ptr = (ptr); \ > - kasan_check_write(____ptr, sizeof(*____ptr)); \ > - arch_cmpxchg_local(____ptr, (old), (new)); \ > + kasan_check_write((ptr), sizeof(*(ptr))); \ > + arch_cmpxchg_local((ptr), (old), (new)); \ /\/\/\/\/\/\/\/\/\/\/\/\ These are macros. If ptr is foo(), then we will call foo() twice. > }) > > #define cmpxchg64(ptr, old, new) \ > ({ \ > - __typeof__(ptr) ____ptr = (ptr); \ > - kasan_check_write(____ptr, sizeof(*____ptr)); \ > - arch_cmpxchg64(____ptr, (old), (new)); \ > + kasan_check_write((ptr), sizeof(*(ptr))); \ > + arch_cmpxchg64((ptr), (old), (new)); \ > }) > > #define cmpxchg64_local(ptr, old, new) \ > ({ \ > - __typeof__(ptr) ____ptr = (ptr); \ > - kasan_check_write(____ptr, sizeof(*____ptr)); \ > - arch_cmpxchg64_local(____ptr, (old), (new)); \ > + kasan_check_write((ptr), sizeof(*(ptr))); \ > + arch_cmpxchg64_local((ptr), (old), (new)); \ > }) > > /* > ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr @ 2017-06-28 11:12 ` Dmitry Vyukov 0 siblings, 0 replies; 63+ messages in thread From: Dmitry Vyukov @ 2017-06-28 11:12 UTC (permalink / raw) To: Thomas Gleixner Cc: Sebastian Andrzej Siewior, Ingo Molnar, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, Jun 28, 2017 at 1:10 PM, Thomas Gleixner <tglx@linutronix.de> wrote: > On Wed, 28 Jun 2017, Dmitry Vyukov wrote: >> On Wed, Jun 28, 2017 at 12:02 PM, Sebastian Andrzej Siewior <bigeasy@linutronix.de> wrote: >> > @@ -359,16 +359,16 @@ static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v) >> > >> > #define cmpxchg64(ptr, old, new) \ >> > ({ \ >> > - __typeof__(ptr) ____ptr = (ptr); \ >> > - kasan_check_write(____ptr, sizeof(*____ptr)); \ >> > - arch_cmpxchg64(____ptr, (old), (new)); \ >> > + __typeof__(ptr) ____ptr64 = (ptr); \ >> > + kasan_check_write(____ptr64, sizeof(*____ptr64));\ >> > + arch_cmpxchg64(____ptr64, (old), (new)); \ >> > }) >> > >> > #define cmpxchg64_local(ptr, old, new) \ >> > ({ \ >> > - __typeof__(ptr) ____ptr = (ptr); \ >> > - kasan_check_write(____ptr, sizeof(*____ptr)); \ >> > - arch_cmpxchg64_local(____ptr, (old), (new)); \ >> > + __typeof__(ptr) ____ptr64 = (ptr); \ >> > + kasan_check_write(____ptr64, sizeof(*____ptr64));\ >> > + arch_cmpxchg64_local(____ptr64, (old), (new)); \ >> > }) >> > >> > #define cmpxchg_double(p1, p2, o1, o2, n1, n2) \ >> >> >> Doh! Thanks for fixing this. I think I've a similar crash in a >> different place when I developed the patch. >> The problem is that when we do: >> >> __typeof__(ptr) ____ptr = (ptr); \ >> arch_cmpxchg64_local(____ptr, (old), (new)); \ >> >> We don't necessary pass value of our just declared ____ptr to >> arch_cmpxchg64_local(). We just pass a symbolic identifier. So if >> arch_cmpxchg64_local() declares own ____ptr and then tries to use what >> we passed ("____ptr") it will actually refer to own variable declared >> rather than to what we wanted to pass in. >> >> In my case I ended up with something like: >> >> __typeof__(foo) __ptr = __ptr; >> >> which compiler decided to turn into 0. >> >> Thank you, macros. >> >> We can add more underscores, but the problem can happen again. Should >> we prefix current function/macro name to all local vars?.. > > Actually we can void that ___ptr dance completely. > > Thanks, > > tglx > > 8<-------------------- > > --- a/include/asm-generic/atomic-instrumented.h > +++ b/include/asm-generic/atomic-instrumented.h > @@ -359,37 +359,32 @@ static __always_inline bool atomic64_add > > #define cmpxchg(ptr, old, new) \ > ({ \ > - __typeof__(ptr) ___ptr = (ptr); \ > - kasan_check_write(___ptr, sizeof(*___ptr)); \ > + kasan_check_write((ptr), sizeof(*(ptr))); \ > arch_cmpxchg((ptr), (old), (new)); \ > }) > > #define sync_cmpxchg(ptr, old, new) \ > ({ \ > - __typeof__(ptr) ___ptr = (ptr); \ > - kasan_check_write(___ptr, sizeof(*___ptr)); \ > - arch_sync_cmpxchg(___ptr, (old), (new)); \ > + kasan_check_write((ptr), sizeof(*(ptr))); \ > + arch_sync_cmpxchg((ptr), (old), (new)); \ > }) > > #define cmpxchg_local(ptr, old, new) \ > ({ \ > - __typeof__(ptr) ____ptr = (ptr); \ > - kasan_check_write(____ptr, sizeof(*____ptr)); \ > - arch_cmpxchg_local(____ptr, (old), (new)); \ > + kasan_check_write((ptr), sizeof(*(ptr))); \ > + arch_cmpxchg_local((ptr), (old), (new)); \ /\/\/\/\/\/\/\/\/\/\/\/\ These are macros. If ptr is foo(), then we will call foo() twice. > }) > > #define cmpxchg64(ptr, old, new) \ > ({ \ > - __typeof__(ptr) ____ptr = (ptr); \ > - kasan_check_write(____ptr, sizeof(*____ptr)); \ > - arch_cmpxchg64(____ptr, (old), (new)); \ > + kasan_check_write((ptr), sizeof(*(ptr))); \ > + arch_cmpxchg64((ptr), (old), (new)); \ > }) > > #define cmpxchg64_local(ptr, old, new) \ > ({ \ > - __typeof__(ptr) ____ptr = (ptr); \ > - kasan_check_write(____ptr, sizeof(*____ptr)); \ > - arch_cmpxchg64_local(____ptr, (old), (new)); \ > + kasan_check_write((ptr), sizeof(*(ptr))); \ > + arch_cmpxchg64_local((ptr), (old), (new)); \ > }) > > /* > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr 2017-06-28 11:12 ` Dmitry Vyukov @ 2017-06-28 11:21 ` Thomas Gleixner -1 siblings, 0 replies; 63+ messages in thread From: Thomas Gleixner @ 2017-06-28 11:21 UTC (permalink / raw) To: Dmitry Vyukov Cc: Sebastian Andrzej Siewior, Ingo Molnar, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, 28 Jun 2017, Dmitry Vyukov wrote: > On Wed, Jun 28, 2017 at 1:10 PM, Thomas Gleixner <tglx@linutronix.de> wrote: > >> In my case I ended up with something like: > >> > >> __typeof__(foo) __ptr = __ptr; > >> > >> which compiler decided to turn into 0. > >> > >> Thank you, macros. > >> > >> We can add more underscores, but the problem can happen again. Should > >> we prefix current function/macro name to all local vars?.. > > > > Actually we can void that ___ptr dance completely. > > > > Thanks, > > > > tglx > > > > 8<-------------------- > > > > --- a/include/asm-generic/atomic-instrumented.h > > +++ b/include/asm-generic/atomic-instrumented.h > > @@ -359,37 +359,32 @@ static __always_inline bool atomic64_add > > > > #define cmpxchg(ptr, old, new) \ > > ({ \ > > - __typeof__(ptr) ___ptr = (ptr); \ > > - kasan_check_write(___ptr, sizeof(*___ptr)); \ > > + kasan_check_write((ptr), sizeof(*(ptr))); \ > > arch_cmpxchg((ptr), (old), (new)); \ > > }) > > > > #define sync_cmpxchg(ptr, old, new) \ > > ({ \ > > - __typeof__(ptr) ___ptr = (ptr); \ > > - kasan_check_write(___ptr, sizeof(*___ptr)); \ > > - arch_sync_cmpxchg(___ptr, (old), (new)); \ > > + kasan_check_write((ptr), sizeof(*(ptr))); \ > > + arch_sync_cmpxchg((ptr), (old), (new)); \ > > }) > > > > #define cmpxchg_local(ptr, old, new) \ > > ({ \ > > - __typeof__(ptr) ____ptr = (ptr); \ > > - kasan_check_write(____ptr, sizeof(*____ptr)); \ > > - arch_cmpxchg_local(____ptr, (old), (new)); \ > > + kasan_check_write((ptr), sizeof(*(ptr))); \ > > + arch_cmpxchg_local((ptr), (old), (new)); \ > > > /\/\/\/\/\/\/\/\/\/\/\/\ > > These are macros. > If ptr is foo(), then we will call foo() twice. Sigh, is that actually used? That's all insane. The whole crap gets worse because: cmpxchg() can be used on u8, u16, u32 .... There is a reason why type safety matters. Thanks, tglx ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr @ 2017-06-28 11:21 ` Thomas Gleixner 0 siblings, 0 replies; 63+ messages in thread From: Thomas Gleixner @ 2017-06-28 11:21 UTC (permalink / raw) To: Dmitry Vyukov Cc: Sebastian Andrzej Siewior, Ingo Molnar, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, 28 Jun 2017, Dmitry Vyukov wrote: > On Wed, Jun 28, 2017 at 1:10 PM, Thomas Gleixner <tglx@linutronix.de> wrote: > >> In my case I ended up with something like: > >> > >> __typeof__(foo) __ptr = __ptr; > >> > >> which compiler decided to turn into 0. > >> > >> Thank you, macros. > >> > >> We can add more underscores, but the problem can happen again. Should > >> we prefix current function/macro name to all local vars?.. > > > > Actually we can void that ___ptr dance completely. > > > > Thanks, > > > > tglx > > > > 8<-------------------- > > > > --- a/include/asm-generic/atomic-instrumented.h > > +++ b/include/asm-generic/atomic-instrumented.h > > @@ -359,37 +359,32 @@ static __always_inline bool atomic64_add > > > > #define cmpxchg(ptr, old, new) \ > > ({ \ > > - __typeof__(ptr) ___ptr = (ptr); \ > > - kasan_check_write(___ptr, sizeof(*___ptr)); \ > > + kasan_check_write((ptr), sizeof(*(ptr))); \ > > arch_cmpxchg((ptr), (old), (new)); \ > > }) > > > > #define sync_cmpxchg(ptr, old, new) \ > > ({ \ > > - __typeof__(ptr) ___ptr = (ptr); \ > > - kasan_check_write(___ptr, sizeof(*___ptr)); \ > > - arch_sync_cmpxchg(___ptr, (old), (new)); \ > > + kasan_check_write((ptr), sizeof(*(ptr))); \ > > + arch_sync_cmpxchg((ptr), (old), (new)); \ > > }) > > > > #define cmpxchg_local(ptr, old, new) \ > > ({ \ > > - __typeof__(ptr) ____ptr = (ptr); \ > > - kasan_check_write(____ptr, sizeof(*____ptr)); \ > > - arch_cmpxchg_local(____ptr, (old), (new)); \ > > + kasan_check_write((ptr), sizeof(*(ptr))); \ > > + arch_cmpxchg_local((ptr), (old), (new)); \ > > > /\/\/\/\/\/\/\/\/\/\/\/\ > > These are macros. > If ptr is foo(), then we will call foo() twice. Sigh, is that actually used? That's all insane. The whole crap gets worse because: cmpxchg() can be used on u8, u16, u32 .... There is a reason why type safety matters. Thanks, tglx -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr 2017-06-28 11:21 ` Thomas Gleixner @ 2017-06-28 12:45 ` Mark Rutland -1 siblings, 0 replies; 63+ messages in thread From: Mark Rutland @ 2017-06-28 12:45 UTC (permalink / raw) To: Thomas Gleixner Cc: Dmitry Vyukov, Sebastian Andrzej Siewior, Ingo Molnar, Peter Zijlstra, Will Deacon, H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, Jun 28, 2017 at 01:21:43PM +0200, Thomas Gleixner wrote: > On Wed, 28 Jun 2017, Dmitry Vyukov wrote: > > On Wed, Jun 28, 2017 at 1:10 PM, Thomas Gleixner <tglx@linutronix.de> wrote: > > >> In my case I ended up with something like: > > >> > > >> __typeof__(foo) __ptr = __ptr; > > >> > > >> which compiler decided to turn into 0. > > >> > > >> Thank you, macros. > > >> > > >> We can add more underscores, but the problem can happen again. Should > > >> we prefix current function/macro name to all local vars?.. > > > > > > Actually we can void that ___ptr dance completely. > > > > > > Thanks, > > > > > > tglx > > > > > > 8<-------------------- > > > > > > --- a/include/asm-generic/atomic-instrumented.h > > > +++ b/include/asm-generic/atomic-instrumented.h > > > @@ -359,37 +359,32 @@ static __always_inline bool atomic64_add > > > > > > #define cmpxchg(ptr, old, new) \ > > > ({ \ > > > - __typeof__(ptr) ___ptr = (ptr); \ > > > - kasan_check_write(___ptr, sizeof(*___ptr)); \ > > > + kasan_check_write((ptr), sizeof(*(ptr))); \ > > > arch_cmpxchg((ptr), (old), (new)); \ > > > }) > > > > > > #define sync_cmpxchg(ptr, old, new) \ > > > ({ \ > > > - __typeof__(ptr) ___ptr = (ptr); \ > > > - kasan_check_write(___ptr, sizeof(*___ptr)); \ > > > - arch_sync_cmpxchg(___ptr, (old), (new)); \ > > > + kasan_check_write((ptr), sizeof(*(ptr))); \ > > > + arch_sync_cmpxchg((ptr), (old), (new)); \ > > > }) > > > > > > #define cmpxchg_local(ptr, old, new) \ > > > ({ \ > > > - __typeof__(ptr) ____ptr = (ptr); \ > > > - kasan_check_write(____ptr, sizeof(*____ptr)); \ > > > - arch_cmpxchg_local(____ptr, (old), (new)); \ > > > + kasan_check_write((ptr), sizeof(*(ptr))); \ > > > + arch_cmpxchg_local((ptr), (old), (new)); \ > > > > > > /\/\/\/\/\/\/\/\/\/\/\/\ > > > > These are macros. > > If ptr is foo(), then we will call foo() twice. > > Sigh, is that actually used? For better or worse, we can't rule it out. We'd risk even more subtle bugs in future trying to rely on that not being the case. :/ > That's all insane. The whole crap gets worse because: > > cmpxchg() can be used on u8, u16, u32 .... Yup, that's the whole reason for the macro insanity in the fist place. Anoother option is something like: static inline unsigned long cmpxchg_size(unsigned long *ptr, unsigned long old, unsigned long new, int size) { kasan_check_write(ptr, size); switch (size) { case 1: return arch_cmpxchg((u8 *)ptr, (u8)old, (u8)new); case 2: return arch_cmpxchg((u16 *)ptr, (u16)old, (u16)new); case 4: return arch_cmpxchg((u32 *)ptr, (u32)old, (u32)new); case 8: return arch_cmpxchg((u64 *)ptr, (u64)old, (u64)new); } BUILD_BUG(); } #define cmpxchg(ptr, old, new) \ cmpxchg_size(ptr, old, new, sizeof(*ptr)) Thanks, Mark. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr @ 2017-06-28 12:45 ` Mark Rutland 0 siblings, 0 replies; 63+ messages in thread From: Mark Rutland @ 2017-06-28 12:45 UTC (permalink / raw) To: Thomas Gleixner Cc: Dmitry Vyukov, Sebastian Andrzej Siewior, Ingo Molnar, Peter Zijlstra, Will Deacon, H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, Jun 28, 2017 at 01:21:43PM +0200, Thomas Gleixner wrote: > On Wed, 28 Jun 2017, Dmitry Vyukov wrote: > > On Wed, Jun 28, 2017 at 1:10 PM, Thomas Gleixner <tglx@linutronix.de> wrote: > > >> In my case I ended up with something like: > > >> > > >> __typeof__(foo) __ptr = __ptr; > > >> > > >> which compiler decided to turn into 0. > > >> > > >> Thank you, macros. > > >> > > >> We can add more underscores, but the problem can happen again. Should > > >> we prefix current function/macro name to all local vars?.. > > > > > > Actually we can void that ___ptr dance completely. > > > > > > Thanks, > > > > > > tglx > > > > > > 8<-------------------- > > > > > > --- a/include/asm-generic/atomic-instrumented.h > > > +++ b/include/asm-generic/atomic-instrumented.h > > > @@ -359,37 +359,32 @@ static __always_inline bool atomic64_add > > > > > > #define cmpxchg(ptr, old, new) \ > > > ({ \ > > > - __typeof__(ptr) ___ptr = (ptr); \ > > > - kasan_check_write(___ptr, sizeof(*___ptr)); \ > > > + kasan_check_write((ptr), sizeof(*(ptr))); \ > > > arch_cmpxchg((ptr), (old), (new)); \ > > > }) > > > > > > #define sync_cmpxchg(ptr, old, new) \ > > > ({ \ > > > - __typeof__(ptr) ___ptr = (ptr); \ > > > - kasan_check_write(___ptr, sizeof(*___ptr)); \ > > > - arch_sync_cmpxchg(___ptr, (old), (new)); \ > > > + kasan_check_write((ptr), sizeof(*(ptr))); \ > > > + arch_sync_cmpxchg((ptr), (old), (new)); \ > > > }) > > > > > > #define cmpxchg_local(ptr, old, new) \ > > > ({ \ > > > - __typeof__(ptr) ____ptr = (ptr); \ > > > - kasan_check_write(____ptr, sizeof(*____ptr)); \ > > > - arch_cmpxchg_local(____ptr, (old), (new)); \ > > > + kasan_check_write((ptr), sizeof(*(ptr))); \ > > > + arch_cmpxchg_local((ptr), (old), (new)); \ > > > > > > /\/\/\/\/\/\/\/\/\/\/\/\ > > > > These are macros. > > If ptr is foo(), then we will call foo() twice. > > Sigh, is that actually used? For better or worse, we can't rule it out. We'd risk even more subtle bugs in future trying to rely on that not being the case. :/ > That's all insane. The whole crap gets worse because: > > cmpxchg() can be used on u8, u16, u32 .... Yup, that's the whole reason for the macro insanity in the fist place. Anoother option is something like: static inline unsigned long cmpxchg_size(unsigned long *ptr, unsigned long old, unsigned long new, int size) { kasan_check_write(ptr, size); switch (size) { case 1: return arch_cmpxchg((u8 *)ptr, (u8)old, (u8)new); case 2: return arch_cmpxchg((u16 *)ptr, (u16)old, (u16)new); case 4: return arch_cmpxchg((u32 *)ptr, (u32)old, (u32)new); case 8: return arch_cmpxchg((u64 *)ptr, (u64)old, (u64)new); } BUILD_BUG(); } #define cmpxchg(ptr, old, new) \ cmpxchg_size(ptr, old, new, sizeof(*ptr)) Thanks, Mark. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr 2017-06-28 11:12 ` Dmitry Vyukov @ 2017-06-28 12:24 ` Thomas Gleixner -1 siblings, 0 replies; 63+ messages in thread From: Thomas Gleixner @ 2017-06-28 12:24 UTC (permalink / raw) To: Dmitry Vyukov Cc: Sebastian Andrzej Siewior, Ingo Molnar, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, 28 Jun 2017, Dmitry Vyukov wrote: > On Wed, Jun 28, 2017 at 1:10 PM, Thomas Gleixner <tglx@linutronix.de> wrote: > > #define cmpxchg_local(ptr, old, new) \ > > ({ \ > > - __typeof__(ptr) ____ptr = (ptr); \ > > - kasan_check_write(____ptr, sizeof(*____ptr)); \ > > - arch_cmpxchg_local(____ptr, (old), (new)); \ > > + kasan_check_write((ptr), sizeof(*(ptr))); \ > > + arch_cmpxchg_local((ptr), (old), (new)); \ > > > /\/\/\/\/\/\/\/\/\/\/\/\ > > These are macros. > If ptr is foo(), then we will call foo() twice. If that's true, the foo() will be evaluated a gazillion more times down the way to the end of this macro maze. Thanks, tglx ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr @ 2017-06-28 12:24 ` Thomas Gleixner 0 siblings, 0 replies; 63+ messages in thread From: Thomas Gleixner @ 2017-06-28 12:24 UTC (permalink / raw) To: Dmitry Vyukov Cc: Sebastian Andrzej Siewior, Ingo Molnar, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, 28 Jun 2017, Dmitry Vyukov wrote: > On Wed, Jun 28, 2017 at 1:10 PM, Thomas Gleixner <tglx@linutronix.de> wrote: > > #define cmpxchg_local(ptr, old, new) \ > > ({ \ > > - __typeof__(ptr) ____ptr = (ptr); \ > > - kasan_check_write(____ptr, sizeof(*____ptr)); \ > > - arch_cmpxchg_local(____ptr, (old), (new)); \ > > + kasan_check_write((ptr), sizeof(*(ptr))); \ > > + arch_cmpxchg_local((ptr), (old), (new)); \ > > > /\/\/\/\/\/\/\/\/\/\/\/\ > > These are macros. > If ptr is foo(), then we will call foo() twice. If that's true, the foo() will be evaluated a gazillion more times down the way to the end of this macro maze. Thanks, tglx -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr 2017-06-28 12:24 ` Thomas Gleixner @ 2017-06-28 12:27 ` Dmitry Vyukov -1 siblings, 0 replies; 63+ messages in thread From: Dmitry Vyukov @ 2017-06-28 12:27 UTC (permalink / raw) To: Thomas Gleixner Cc: Sebastian Andrzej Siewior, Ingo Molnar, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, Jun 28, 2017 at 2:24 PM, Thomas Gleixner <tglx@linutronix.de> wrote: > On Wed, 28 Jun 2017, Dmitry Vyukov wrote: >> On Wed, Jun 28, 2017 at 1:10 PM, Thomas Gleixner <tglx@linutronix.de> wrote: >> > #define cmpxchg_local(ptr, old, new) \ >> > ({ \ >> > - __typeof__(ptr) ____ptr = (ptr); \ >> > - kasan_check_write(____ptr, sizeof(*____ptr)); \ >> > - arch_cmpxchg_local(____ptr, (old), (new)); \ >> > + kasan_check_write((ptr), sizeof(*(ptr))); \ >> > + arch_cmpxchg_local((ptr), (old), (new)); \ >> >> >> /\/\/\/\/\/\/\/\/\/\/\/\ >> >> These are macros. >> If ptr is foo(), then we will call foo() twice. > > If that's true, the foo() will be evaluated a gazillion more times down the > way to the end of this macro maze. No. If we do: __typeof__(ptr) ____ptr = (ptr); and then only use ____ptr, then ptr is evaluated only once regardless of what the rest of macros do. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr @ 2017-06-28 12:27 ` Dmitry Vyukov 0 siblings, 0 replies; 63+ messages in thread From: Dmitry Vyukov @ 2017-06-28 12:27 UTC (permalink / raw) To: Thomas Gleixner Cc: Sebastian Andrzej Siewior, Ingo Molnar, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, Jun 28, 2017 at 2:24 PM, Thomas Gleixner <tglx@linutronix.de> wrote: > On Wed, 28 Jun 2017, Dmitry Vyukov wrote: >> On Wed, Jun 28, 2017 at 1:10 PM, Thomas Gleixner <tglx@linutronix.de> wrote: >> > #define cmpxchg_local(ptr, old, new) \ >> > ({ \ >> > - __typeof__(ptr) ____ptr = (ptr); \ >> > - kasan_check_write(____ptr, sizeof(*____ptr)); \ >> > - arch_cmpxchg_local(____ptr, (old), (new)); \ >> > + kasan_check_write((ptr), sizeof(*(ptr))); \ >> > + arch_cmpxchg_local((ptr), (old), (new)); \ >> >> >> /\/\/\/\/\/\/\/\/\/\/\/\ >> >> These are macros. >> If ptr is foo(), then we will call foo() twice. > > If that's true, the foo() will be evaluated a gazillion more times down the > way to the end of this macro maze. No. If we do: __typeof__(ptr) ____ptr = (ptr); and then only use ____ptr, then ptr is evaluated only once regardless of what the rest of macros do. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr 2017-06-28 12:27 ` Dmitry Vyukov @ 2017-06-28 13:33 ` Thomas Gleixner -1 siblings, 0 replies; 63+ messages in thread From: Thomas Gleixner @ 2017-06-28 13:33 UTC (permalink / raw) To: Dmitry Vyukov Cc: Sebastian Andrzej Siewior, Ingo Molnar, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, 28 Jun 2017, Dmitry Vyukov wrote: > On Wed, Jun 28, 2017 at 2:24 PM, Thomas Gleixner <tglx@linutronix.de> wrote: > > On Wed, 28 Jun 2017, Dmitry Vyukov wrote: > >> On Wed, Jun 28, 2017 at 1:10 PM, Thomas Gleixner <tglx@linutronix.de> wrote: > >> > #define cmpxchg_local(ptr, old, new) \ > >> > ({ \ > >> > - __typeof__(ptr) ____ptr = (ptr); \ > >> > - kasan_check_write(____ptr, sizeof(*____ptr)); \ > >> > - arch_cmpxchg_local(____ptr, (old), (new)); \ > >> > + kasan_check_write((ptr), sizeof(*(ptr))); \ > >> > + arch_cmpxchg_local((ptr), (old), (new)); \ > >> > >> > >> /\/\/\/\/\/\/\/\/\/\/\/\ > >> > >> These are macros. > >> If ptr is foo(), then we will call foo() twice. > > > > If that's true, the foo() will be evaluated a gazillion more times down the > > way to the end of this macro maze. > > No. If we do: > > __typeof__(ptr) ____ptr = (ptr); > > and then only use ____ptr, then ptr is evaluated only once regardless > of what the rest of macros do. What I meant is, that we have today nested macros which do a lot of that same nonsense even w/o that patch. Thanks, tglx ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr @ 2017-06-28 13:33 ` Thomas Gleixner 0 siblings, 0 replies; 63+ messages in thread From: Thomas Gleixner @ 2017-06-28 13:33 UTC (permalink / raw) To: Dmitry Vyukov Cc: Sebastian Andrzej Siewior, Ingo Molnar, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, Andrey Ryabinin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, 28 Jun 2017, Dmitry Vyukov wrote: > On Wed, Jun 28, 2017 at 2:24 PM, Thomas Gleixner <tglx@linutronix.de> wrote: > > On Wed, 28 Jun 2017, Dmitry Vyukov wrote: > >> On Wed, Jun 28, 2017 at 1:10 PM, Thomas Gleixner <tglx@linutronix.de> wrote: > >> > #define cmpxchg_local(ptr, old, new) \ > >> > ({ \ > >> > - __typeof__(ptr) ____ptr = (ptr); \ > >> > - kasan_check_write(____ptr, sizeof(*____ptr)); \ > >> > - arch_cmpxchg_local(____ptr, (old), (new)); \ > >> > + kasan_check_write((ptr), sizeof(*(ptr))); \ > >> > + arch_cmpxchg_local((ptr), (old), (new)); \ > >> > >> > >> /\/\/\/\/\/\/\/\/\/\/\/\ > >> > >> These are macros. > >> If ptr is foo(), then we will call foo() twice. > > > > If that's true, the foo() will be evaluated a gazillion more times down the > > way to the end of this macro maze. > > No. If we do: > > __typeof__(ptr) ____ptr = (ptr); > > and then only use ____ptr, then ptr is evaluated only once regardless > of what the rest of macros do. What I meant is, that we have today nested macros which do a lot of that same nonsense even w/o that patch. Thanks, tglx -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr 2017-06-28 10:16 ` Dmitry Vyukov @ 2017-06-28 11:15 ` Andrey Ryabinin -1 siblings, 0 replies; 63+ messages in thread From: Andrey Ryabinin @ 2017-06-28 11:15 UTC (permalink / raw) To: Dmitry Vyukov, Sebastian Andrzej Siewior Cc: Ingo Molnar, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds, Thomas Gleixner On 06/28/2017 01:16 PM, Dmitry Vyukov wrote: > On Wed, Jun 28, 2017 at 12:02 PM, Sebastian Andrzej Siewior > <bigeasy@linutronix.de> wrote: >> Trying to boot tip/master resulted in: >> |DMAR: dmar0: Using Queued invalidation >> |DMAR: dmar1: Using Queued invalidation >> |DMAR: Setting RMRR: >> |DMAR: Setting identity map for device 0000:00:1a.0 [0xbdcf9000 - 0xbdd1dfff] >> |BUG: unable to handle kernel NULL pointer dereference at (null) >> |IP: __domain_mapping+0x10f/0x3d0 >> |PGD 0 >> |P4D 0 >> | >> |Oops: 0002 [#1] PREEMPT SMP >> |Modules linked in: >> |CPU: 19 PID: 1 Comm: swapper/0 Not tainted 4.12.0-rc6-00117-g235a93822a21 #113 >> |task: ffff8805271c2c80 task.stack: ffffc90000058000 >> |RIP: 0010:__domain_mapping+0x10f/0x3d0 >> |RSP: 0000:ffffc9000005bca0 EFLAGS: 00010246 >> |RAX: 0000000000000000 RBX: 00000000bdcf9003 RCX: 0000000000000000 >> |RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000001 >> |RBP: ffffc9000005bd00 R08: ffff880a243e9780 R09: ffff8805259e67c8 >> |R10: 00000000000bdcf9 R11: 0000000000000000 R12: 0000000000000025 >> |R13: 0000000000000025 R14: 0000000000000000 R15: 00000000000bdcf9 >> |FS: 0000000000000000(0000) GS:ffff88052acc0000(0000) knlGS:0000000000000000 >> |CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> |CR2: 0000000000000000 CR3: 0000000001c0f000 CR4: 00000000000406e0 >> |Call Trace: >> | iommu_domain_identity_map+0x5a/0x80 >> | domain_prepare_identity_map+0x9f/0x160 >> | iommu_prepare_identity_map+0x7e/0x9b >> >> bisect points to commit 235a93822a21 ("locking/atomics, asm-generic: Add KASAN >> instrumentation to atomic operations"), RIP is at >> tmp = cmpxchg64_local(&pte->val, 0ULL, pteval); >> in drivers/iommu/intel-iommu.c. The assembly for this inline assembly >> is: >> xor %edx,%edx >> xor %eax,%eax >> cmpxchg %rbx,(%rdx) >> >> and as you see edx is set to zero and used later as a pointer via the >> full register. This happens with gcc-6, 5 and 8 (snapshot from last >> week). >> After a longer while of searching and swearing I figured out that this >> bug occures once cmpxchg64_local() and cmpxchg_local() uses the same >> ____ptr macro and they are shadow somehow. What I don't know why edx is >> set to zero. >> >> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> >> --- >> include/asm-generic/atomic-instrumented.h | 12 ++++++------ >> 1 file changed, 6 insertions(+), 6 deletions(-) >> >> diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h >> index a0f5b7525bb2..ac6155362b39 100644 >> --- a/include/asm-generic/atomic-instrumented.h >> +++ b/include/asm-generic/atomic-instrumented.h >> @@ -359,16 +359,16 @@ static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v) >> >> #define cmpxchg64(ptr, old, new) \ >> ({ \ >> - __typeof__(ptr) ____ptr = (ptr); \ >> - kasan_check_write(____ptr, sizeof(*____ptr)); \ >> - arch_cmpxchg64(____ptr, (old), (new)); \ >> + __typeof__(ptr) ____ptr64 = (ptr); \ >> + kasan_check_write(____ptr64, sizeof(*____ptr64));\ >> + arch_cmpxchg64(____ptr64, (old), (new)); \ >> }) >> >> #define cmpxchg64_local(ptr, old, new) \ >> ({ \ >> - __typeof__(ptr) ____ptr = (ptr); \ >> - kasan_check_write(____ptr, sizeof(*____ptr)); \ >> - arch_cmpxchg64_local(____ptr, (old), (new)); \ >> + __typeof__(ptr) ____ptr64 = (ptr); \ >> + kasan_check_write(____ptr64, sizeof(*____ptr64));\ >> + arch_cmpxchg64_local(____ptr64, (old), (new)); \ >> }) >> >> #define cmpxchg_double(p1, p2, o1, o2, n1, n2) \ > > > Doh! Thanks for fixing this. I think I've a similar crash in a > different place when I developed the patch. > The problem is that when we do: > > __typeof__(ptr) ____ptr = (ptr); \ > arch_cmpxchg64_local(____ptr, (old), (new)); \ > > We don't necessary pass value of our just declared ____ptr to > arch_cmpxchg64_local(). We just pass a symbolic identifier. So if > arch_cmpxchg64_local() declares own ____ptr and then tries to use what > we passed ("____ptr") it will actually refer to own variable declared > rather than to what we wanted to pass in. > > In my case I ended up with something like: > > __typeof__(foo) __ptr = __ptr; > > which compiler decided to turn into 0. > > Thank you, macros. > > We can add more underscores, but the problem can happen again. Should > we prefix current function/macro name to all local vars?.. > The main problem here is that arch_cmpxchg64_local() calls cmpxhg_local() instead of using arch_cmpxchg_local(). So, the patch bellow should fix the problem, also this will fix double instrumentation of cmpcxchg64[_local](). But I haven't tested this patch yet. --- arch/x86/include/asm/cmpxchg_64.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/cmpxchg_64.h b/arch/x86/include/asm/cmpxchg_64.h index fafaebacca2d..7046a3cc2493 100644 --- a/arch/x86/include/asm/cmpxchg_64.h +++ b/arch/x86/include/asm/cmpxchg_64.h @@ -9,13 +9,13 @@ static inline void set_64bit(volatile u64 *ptr, u64 val) #define arch_cmpxchg64(ptr, o, n) \ ({ \ BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ - cmpxchg((ptr), (o), (n)); \ + arch_cmpxchg((ptr), (o), (n)); \ }) #define arch_cmpxchg64_local(ptr, o, n) \ ({ \ BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ - cmpxchg_local((ptr), (o), (n)); \ + arch_cmpxchg_local((ptr), (o), (n)); \ }) #define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX16) -- 2.13.0 ^ permalink raw reply related [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr @ 2017-06-28 11:15 ` Andrey Ryabinin 0 siblings, 0 replies; 63+ messages in thread From: Andrey Ryabinin @ 2017-06-28 11:15 UTC (permalink / raw) To: Dmitry Vyukov, Sebastian Andrzej Siewior Cc: Ingo Molnar, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds, Thomas Gleixner On 06/28/2017 01:16 PM, Dmitry Vyukov wrote: > On Wed, Jun 28, 2017 at 12:02 PM, Sebastian Andrzej Siewior > <bigeasy@linutronix.de> wrote: >> Trying to boot tip/master resulted in: >> |DMAR: dmar0: Using Queued invalidation >> |DMAR: dmar1: Using Queued invalidation >> |DMAR: Setting RMRR: >> |DMAR: Setting identity map for device 0000:00:1a.0 [0xbdcf9000 - 0xbdd1dfff] >> |BUG: unable to handle kernel NULL pointer dereference at (null) >> |IP: __domain_mapping+0x10f/0x3d0 >> |PGD 0 >> |P4D 0 >> | >> |Oops: 0002 [#1] PREEMPT SMP >> |Modules linked in: >> |CPU: 19 PID: 1 Comm: swapper/0 Not tainted 4.12.0-rc6-00117-g235a93822a21 #113 >> |task: ffff8805271c2c80 task.stack: ffffc90000058000 >> |RIP: 0010:__domain_mapping+0x10f/0x3d0 >> |RSP: 0000:ffffc9000005bca0 EFLAGS: 00010246 >> |RAX: 0000000000000000 RBX: 00000000bdcf9003 RCX: 0000000000000000 >> |RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000001 >> |RBP: ffffc9000005bd00 R08: ffff880a243e9780 R09: ffff8805259e67c8 >> |R10: 00000000000bdcf9 R11: 0000000000000000 R12: 0000000000000025 >> |R13: 0000000000000025 R14: 0000000000000000 R15: 00000000000bdcf9 >> |FS: 0000000000000000(0000) GS:ffff88052acc0000(0000) knlGS:0000000000000000 >> |CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >> |CR2: 0000000000000000 CR3: 0000000001c0f000 CR4: 00000000000406e0 >> |Call Trace: >> | iommu_domain_identity_map+0x5a/0x80 >> | domain_prepare_identity_map+0x9f/0x160 >> | iommu_prepare_identity_map+0x7e/0x9b >> >> bisect points to commit 235a93822a21 ("locking/atomics, asm-generic: Add KASAN >> instrumentation to atomic operations"), RIP is at >> tmp = cmpxchg64_local(&pte->val, 0ULL, pteval); >> in drivers/iommu/intel-iommu.c. The assembly for this inline assembly >> is: >> xor %edx,%edx >> xor %eax,%eax >> cmpxchg %rbx,(%rdx) >> >> and as you see edx is set to zero and used later as a pointer via the >> full register. This happens with gcc-6, 5 and 8 (snapshot from last >> week). >> After a longer while of searching and swearing I figured out that this >> bug occures once cmpxchg64_local() and cmpxchg_local() uses the same >> ____ptr macro and they are shadow somehow. What I don't know why edx is >> set to zero. >> >> Signed-off-by: Sebastian Andrzej Siewior <bigeasy@linutronix.de> >> --- >> include/asm-generic/atomic-instrumented.h | 12 ++++++------ >> 1 file changed, 6 insertions(+), 6 deletions(-) >> >> diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h >> index a0f5b7525bb2..ac6155362b39 100644 >> --- a/include/asm-generic/atomic-instrumented.h >> +++ b/include/asm-generic/atomic-instrumented.h >> @@ -359,16 +359,16 @@ static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v) >> >> #define cmpxchg64(ptr, old, new) \ >> ({ \ >> - __typeof__(ptr) ____ptr = (ptr); \ >> - kasan_check_write(____ptr, sizeof(*____ptr)); \ >> - arch_cmpxchg64(____ptr, (old), (new)); \ >> + __typeof__(ptr) ____ptr64 = (ptr); \ >> + kasan_check_write(____ptr64, sizeof(*____ptr64));\ >> + arch_cmpxchg64(____ptr64, (old), (new)); \ >> }) >> >> #define cmpxchg64_local(ptr, old, new) \ >> ({ \ >> - __typeof__(ptr) ____ptr = (ptr); \ >> - kasan_check_write(____ptr, sizeof(*____ptr)); \ >> - arch_cmpxchg64_local(____ptr, (old), (new)); \ >> + __typeof__(ptr) ____ptr64 = (ptr); \ >> + kasan_check_write(____ptr64, sizeof(*____ptr64));\ >> + arch_cmpxchg64_local(____ptr64, (old), (new)); \ >> }) >> >> #define cmpxchg_double(p1, p2, o1, o2, n1, n2) \ > > > Doh! Thanks for fixing this. I think I've a similar crash in a > different place when I developed the patch. > The problem is that when we do: > > __typeof__(ptr) ____ptr = (ptr); \ > arch_cmpxchg64_local(____ptr, (old), (new)); \ > > We don't necessary pass value of our just declared ____ptr to > arch_cmpxchg64_local(). We just pass a symbolic identifier. So if > arch_cmpxchg64_local() declares own ____ptr and then tries to use what > we passed ("____ptr") it will actually refer to own variable declared > rather than to what we wanted to pass in. > > In my case I ended up with something like: > > __typeof__(foo) __ptr = __ptr; > > which compiler decided to turn into 0. > > Thank you, macros. > > We can add more underscores, but the problem can happen again. Should > we prefix current function/macro name to all local vars?.. > The main problem here is that arch_cmpxchg64_local() calls cmpxhg_local() instead of using arch_cmpxchg_local(). So, the patch bellow should fix the problem, also this will fix double instrumentation of cmpcxchg64[_local](). But I haven't tested this patch yet. --- arch/x86/include/asm/cmpxchg_64.h | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/arch/x86/include/asm/cmpxchg_64.h b/arch/x86/include/asm/cmpxchg_64.h index fafaebacca2d..7046a3cc2493 100644 --- a/arch/x86/include/asm/cmpxchg_64.h +++ b/arch/x86/include/asm/cmpxchg_64.h @@ -9,13 +9,13 @@ static inline void set_64bit(volatile u64 *ptr, u64 val) #define arch_cmpxchg64(ptr, o, n) \ ({ \ BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ - cmpxchg((ptr), (o), (n)); \ + arch_cmpxchg((ptr), (o), (n)); \ }) #define arch_cmpxchg64_local(ptr, o, n) \ ({ \ BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ - cmpxchg_local((ptr), (o), (n)); \ + arch_cmpxchg_local((ptr), (o), (n)); \ }) #define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX16) -- 2.13.0 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply related [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr 2017-06-28 11:15 ` Andrey Ryabinin @ 2017-06-28 12:12 ` Sebastian Andrzej Siewior -1 siblings, 0 replies; 63+ messages in thread From: Sebastian Andrzej Siewior @ 2017-06-28 12:12 UTC (permalink / raw) To: Andrey Ryabinin, Ingo Molnar Cc: Dmitry Vyukov, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds, Thomas Gleixner On 2017-06-28 14:15:18 [+0300], Andrey Ryabinin wrote: > The main problem here is that arch_cmpxchg64_local() calls cmpxhg_local() instead of using arch_cmpxchg_local(). > > So, the patch bellow should fix the problem, also this will fix double instrumentation of cmpcxchg64[_local](). > But I haven't tested this patch yet. tested, works. Next step? > --- > arch/x86/include/asm/cmpxchg_64.h | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/arch/x86/include/asm/cmpxchg_64.h b/arch/x86/include/asm/cmpxchg_64.h > index fafaebacca2d..7046a3cc2493 100644 > --- a/arch/x86/include/asm/cmpxchg_64.h > +++ b/arch/x86/include/asm/cmpxchg_64.h > @@ -9,13 +9,13 @@ static inline void set_64bit(volatile u64 *ptr, u64 val) > #define arch_cmpxchg64(ptr, o, n) \ > ({ \ > BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ > - cmpxchg((ptr), (o), (n)); \ > + arch_cmpxchg((ptr), (o), (n)); \ > }) > > #define arch_cmpxchg64_local(ptr, o, n) \ > ({ \ > BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ > - cmpxchg_local((ptr), (o), (n)); \ > + arch_cmpxchg_local((ptr), (o), (n)); \ > }) > > #define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX16) Sebastian ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr @ 2017-06-28 12:12 ` Sebastian Andrzej Siewior 0 siblings, 0 replies; 63+ messages in thread From: Sebastian Andrzej Siewior @ 2017-06-28 12:12 UTC (permalink / raw) To: Andrey Ryabinin, Ingo Molnar Cc: Dmitry Vyukov, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds, Thomas Gleixner On 2017-06-28 14:15:18 [+0300], Andrey Ryabinin wrote: > The main problem here is that arch_cmpxchg64_local() calls cmpxhg_local() instead of using arch_cmpxchg_local(). > > So, the patch bellow should fix the problem, also this will fix double instrumentation of cmpcxchg64[_local](). > But I haven't tested this patch yet. tested, works. Next step? > --- > arch/x86/include/asm/cmpxchg_64.h | 4 ++-- > 1 file changed, 2 insertions(+), 2 deletions(-) > > diff --git a/arch/x86/include/asm/cmpxchg_64.h b/arch/x86/include/asm/cmpxchg_64.h > index fafaebacca2d..7046a3cc2493 100644 > --- a/arch/x86/include/asm/cmpxchg_64.h > +++ b/arch/x86/include/asm/cmpxchg_64.h > @@ -9,13 +9,13 @@ static inline void set_64bit(volatile u64 *ptr, u64 val) > #define arch_cmpxchg64(ptr, o, n) \ > ({ \ > BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ > - cmpxchg((ptr), (o), (n)); \ > + arch_cmpxchg((ptr), (o), (n)); \ > }) > > #define arch_cmpxchg64_local(ptr, o, n) \ > ({ \ > BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ > - cmpxchg_local((ptr), (o), (n)); \ > + arch_cmpxchg_local((ptr), (o), (n)); \ > }) > > #define system_has_cmpxchg_double() boot_cpu_has(X86_FEATURE_CX16) Sebastian -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr 2017-06-28 12:12 ` Sebastian Andrzej Siewior @ 2017-06-28 13:20 ` Thomas Gleixner -1 siblings, 0 replies; 63+ messages in thread From: Thomas Gleixner @ 2017-06-28 13:20 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: Andrey Ryabinin, Ingo Molnar, Dmitry Vyukov, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, 28 Jun 2017, Sebastian Andrzej Siewior wrote: > On 2017-06-28 14:15:18 [+0300], Andrey Ryabinin wrote: > > The main problem here is that arch_cmpxchg64_local() calls cmpxhg_local() instead of using arch_cmpxchg_local(). > > > > So, the patch bellow should fix the problem, also this will fix double instrumentation of cmpcxchg64[_local](). > > But I haven't tested this patch yet. > > tested, works. Next step? Check all other implementations in every architecture whether there is a similar problem ..... But this really want's a proper cleanup unless we want to waste the time over and over again with the next hard to figure out macro expansion fail. First of all, cmpxchg64[_local]() can be implemented as inlines right away. For cmpxchg*(), the situation is slightly different, but the sizeof() evaluation should be done at the top most level, even if we do it further down in the low level arch/asm-generic implementation once more. Something along the lines of: static inline unsigned long cmpxchg_varsize(void *ptr, unsigned long old, unsigned long new, int size) { switch (size) { case 1: case 2: case 4: break; case 8: if (sizeof(unsigned long) == 8) break; default: BUILD_BUG_ON(1); } kasan_check(ptr, size); return arch_cmpxchg(ptr, old, new); } #define cmpxchg(ptr, o, n) \ ({ \ ((__typeof__(*(ptr)))cmpxchg_varsize((ptr), (unsigned long)(o), \ (unsigned long)(n), sizeof(*(ptr)))); \ }) That's the first step to cure the actual mess. Ideally we get rid of that whole macro maze and convert everything to proper inlines with actual cmpxchg8/16/32/64() variants, but that's going to take some time. As an intermediate step we can at least propagate 'size' to arch_cmpxchg(), which is not that much of an effort. Thanks, tglx ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr @ 2017-06-28 13:20 ` Thomas Gleixner 0 siblings, 0 replies; 63+ messages in thread From: Thomas Gleixner @ 2017-06-28 13:20 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: Andrey Ryabinin, Ingo Molnar, Dmitry Vyukov, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, 28 Jun 2017, Sebastian Andrzej Siewior wrote: > On 2017-06-28 14:15:18 [+0300], Andrey Ryabinin wrote: > > The main problem here is that arch_cmpxchg64_local() calls cmpxhg_local() instead of using arch_cmpxchg_local(). > > > > So, the patch bellow should fix the problem, also this will fix double instrumentation of cmpcxchg64[_local](). > > But I haven't tested this patch yet. > > tested, works. Next step? Check all other implementations in every architecture whether there is a similar problem ..... But this really want's a proper cleanup unless we want to waste the time over and over again with the next hard to figure out macro expansion fail. First of all, cmpxchg64[_local]() can be implemented as inlines right away. For cmpxchg*(), the situation is slightly different, but the sizeof() evaluation should be done at the top most level, even if we do it further down in the low level arch/asm-generic implementation once more. Something along the lines of: static inline unsigned long cmpxchg_varsize(void *ptr, unsigned long old, unsigned long new, int size) { switch (size) { case 1: case 2: case 4: break; case 8: if (sizeof(unsigned long) == 8) break; default: BUILD_BUG_ON(1); } kasan_check(ptr, size); return arch_cmpxchg(ptr, old, new); } #define cmpxchg(ptr, o, n) \ ({ \ ((__typeof__(*(ptr)))cmpxchg_varsize((ptr), (unsigned long)(o), \ (unsigned long)(n), sizeof(*(ptr)))); \ }) That's the first step to cure the actual mess. Ideally we get rid of that whole macro maze and convert everything to proper inlines with actual cmpxchg8/16/32/64() variants, but that's going to take some time. As an intermediate step we can at least propagate 'size' to arch_cmpxchg(), which is not that much of an effort. Thanks, tglx -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr 2017-06-28 13:20 ` Thomas Gleixner @ 2017-06-28 13:54 ` Thomas Gleixner -1 siblings, 0 replies; 63+ messages in thread From: Thomas Gleixner @ 2017-06-28 13:54 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: Andrey Ryabinin, Ingo Molnar, Dmitry Vyukov, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, 28 Jun 2017, Thomas Gleixner wrote: > On Wed, 28 Jun 2017, Sebastian Andrzej Siewior wrote: > > On 2017-06-28 14:15:18 [+0300], Andrey Ryabinin wrote: > > > The main problem here is that arch_cmpxchg64_local() calls cmpxhg_local() instead of using arch_cmpxchg_local(). > > > > > > So, the patch bellow should fix the problem, also this will fix double instrumentation of cmpcxchg64[_local](). > > > But I haven't tested this patch yet. > > > > tested, works. Next step? > > Check all other implementations in every architecture whether there is a > similar problem ..... > > But this really want's a proper cleanup unless we want to waste the time > over and over again with the next hard to figure out macro expansion fail. > > First of all, cmpxchg64[_local]() can be implemented as inlines right away. > > For cmpxchg*(), the situation is slightly different, but the sizeof() > evaluation should be done at the top most level, even if we do it further > down in the low level arch/asm-generic implementation once more. > > Something along the lines of: > > static inline unsigned long cmpxchg_varsize(void *ptr, unsigned long old, > unsigned long new, int size) > { > switch (size) { > case 1: > case 2: > case 4: > break; > case 8: > if (sizeof(unsigned long) == 8) > break; > default: > BUILD_BUG_ON(1); > } > kasan_check(ptr, size); > return arch_cmpxchg(ptr, old, new); > } > > #define cmpxchg(ptr, o, n) \ > ({ \ > ((__typeof__(*(ptr)))cmpxchg_varsize((ptr), (unsigned long)(o), \ > (unsigned long)(n), sizeof(*(ptr)))); \ > }) > > That's the first step to cure the actual mess. > > Ideally we get rid of that whole macro maze and convert everything to > proper inlines with actual cmpxchg8/16/32/64() variants, but that's going > to take some time. As an intermediate step we can at least propagate 'size' > to arch_cmpxchg(), which is not that much of an effort. And to be honest. That should have be done in the first place _BEFORE_ adding that atomic-instrumented stuff. I'm tempted to revert that mess instead of 'fixing' it half arsed. As a side note, we have files (aside of x86/asm/atomic.h) which include asm/cmpxchg.h ... net/sunrpc/xprtmultipath.c:#include <asm/cmpxchg.h> arch/x86/kvm/mmu.c:#include <asm/cmpxchg.h> arch/x86/um/asm/barrier.h:#include <asm/cmpxchg.h> I'm really tired of all this featuritis crammed into the code without much thought. Dammit, can we please stop this and clean up the existing mess first before duct taping more mess on top of it. Thanks, tglx ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr @ 2017-06-28 13:54 ` Thomas Gleixner 0 siblings, 0 replies; 63+ messages in thread From: Thomas Gleixner @ 2017-06-28 13:54 UTC (permalink / raw) To: Sebastian Andrzej Siewior Cc: Andrey Ryabinin, Ingo Molnar, Dmitry Vyukov, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, 28 Jun 2017, Thomas Gleixner wrote: > On Wed, 28 Jun 2017, Sebastian Andrzej Siewior wrote: > > On 2017-06-28 14:15:18 [+0300], Andrey Ryabinin wrote: > > > The main problem here is that arch_cmpxchg64_local() calls cmpxhg_local() instead of using arch_cmpxchg_local(). > > > > > > So, the patch bellow should fix the problem, also this will fix double instrumentation of cmpcxchg64[_local](). > > > But I haven't tested this patch yet. > > > > tested, works. Next step? > > Check all other implementations in every architecture whether there is a > similar problem ..... > > But this really want's a proper cleanup unless we want to waste the time > over and over again with the next hard to figure out macro expansion fail. > > First of all, cmpxchg64[_local]() can be implemented as inlines right away. > > For cmpxchg*(), the situation is slightly different, but the sizeof() > evaluation should be done at the top most level, even if we do it further > down in the low level arch/asm-generic implementation once more. > > Something along the lines of: > > static inline unsigned long cmpxchg_varsize(void *ptr, unsigned long old, > unsigned long new, int size) > { > switch (size) { > case 1: > case 2: > case 4: > break; > case 8: > if (sizeof(unsigned long) == 8) > break; > default: > BUILD_BUG_ON(1); > } > kasan_check(ptr, size); > return arch_cmpxchg(ptr, old, new); > } > > #define cmpxchg(ptr, o, n) \ > ({ \ > ((__typeof__(*(ptr)))cmpxchg_varsize((ptr), (unsigned long)(o), \ > (unsigned long)(n), sizeof(*(ptr)))); \ > }) > > That's the first step to cure the actual mess. > > Ideally we get rid of that whole macro maze and convert everything to > proper inlines with actual cmpxchg8/16/32/64() variants, but that's going > to take some time. As an intermediate step we can at least propagate 'size' > to arch_cmpxchg(), which is not that much of an effort. And to be honest. That should have be done in the first place _BEFORE_ adding that atomic-instrumented stuff. I'm tempted to revert that mess instead of 'fixing' it half arsed. As a side note, we have files (aside of x86/asm/atomic.h) which include asm/cmpxchg.h ... net/sunrpc/xprtmultipath.c:#include <asm/cmpxchg.h> arch/x86/kvm/mmu.c:#include <asm/cmpxchg.h> arch/x86/um/asm/barrier.h:#include <asm/cmpxchg.h> I'm really tired of all this featuritis crammed into the code without much thought. Dammit, can we please stop this and clean up the existing mess first before duct taping more mess on top of it. Thanks, tglx -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr 2017-06-28 13:54 ` Thomas Gleixner @ 2017-06-28 14:14 ` Mark Rutland -1 siblings, 0 replies; 63+ messages in thread From: Mark Rutland @ 2017-06-28 14:14 UTC (permalink / raw) To: Thomas Gleixner Cc: Sebastian Andrzej Siewior, Andrey Ryabinin, Ingo Molnar, Dmitry Vyukov, Peter Zijlstra, Will Deacon, H. Peter Anvin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, Jun 28, 2017 at 03:54:42PM +0200, Thomas Gleixner wrote: > On Wed, 28 Jun 2017, Thomas Gleixner wrote: > > On Wed, 28 Jun 2017, Sebastian Andrzej Siewior wrote: > > > On 2017-06-28 14:15:18 [+0300], Andrey Ryabinin wrote: > > > > The main problem here is that arch_cmpxchg64_local() calls cmpxhg_local() instead of using arch_cmpxchg_local(). > > > > > > > > So, the patch bellow should fix the problem, also this will fix double instrumentation of cmpcxchg64[_local](). > > > > But I haven't tested this patch yet. > > > > > > tested, works. Next step? > > > > Check all other implementations in every architecture whether there is a > > similar problem ..... FWIW, as x86 is the only user of atomic-instrumented.h, any similar issues are unrelated to this series. That's not to say they don't exist, just that they're orthognal to this. I've been reworking things for arm64 [1], but there's more cleanup needed first. > > But this really want's a proper cleanup unless we want to waste the time > > over and over again with the next hard to figure out macro expansion fail. > > > > First of all, cmpxchg64[_local]() can be implemented as inlines right away. > > > > For cmpxchg*(), the situation is slightly different, but the sizeof() > > evaluation should be done at the top most level, even if we do it further > > down in the low level arch/asm-generic implementation once more. > > > > Something along the lines of: > > > > static inline unsigned long cmpxchg_varsize(void *ptr, unsigned long old, > > unsigned long new, int size) > > { > > switch (size) { > > case 1: > > case 2: > > case 4: > > break; > > case 8: > > if (sizeof(unsigned long) == 8) > > break; > > default: > > BUILD_BUG_ON(1); > > } > > kasan_check(ptr, size); > > return arch_cmpxchg(ptr, old, new); > > } This'll need to re-cast things before the call to arch_cmpxchg(), and we can move the check above the switch, as in [2]. > > #define cmpxchg(ptr, o, n) \ > > ({ \ > > ((__typeof__(*(ptr)))cmpxchg_varsize((ptr), (unsigned long)(o), \ > > (unsigned long)(n), sizeof(*(ptr)))); \ > > }) > > > > That's the first step to cure the actual mess. > > > > Ideally we get rid of that whole macro maze and convert everything to > > proper inlines with actual cmpxchg8/16/32/64() variants, but that's going > > to take some time. As an intermediate step we can at least propagate 'size' > > to arch_cmpxchg(), which is not that much of an effort. > > And to be honest. That should have be done in the first place _BEFORE_ > adding that atomic-instrumented stuff. I'm tempted to revert that mess > instead of 'fixing' it half arsed. Sure. Let's figure out what this *should* look like first. If that's sufficiently different to what we have now, we revert this and clean things up first. > As a side note, we have files (aside of x86/asm/atomic.h) which include > asm/cmpxchg.h ... > > net/sunrpc/xprtmultipath.c:#include <asm/cmpxchg.h> > arch/x86/kvm/mmu.c:#include <asm/cmpxchg.h> > arch/x86/um/asm/barrier.h:#include <asm/cmpxchg.h> Ugh. I'd sent out a patch [3] for the first of these a while back, as I spotted that when experimenting with arm64, but tht got dropped on the floor. I can resend that, if you like? I guess it'd also make sense to fix the x86 bits at the same time, so I'm fine with tahat being folded with other fixes. > I'm really tired of all this featuritis crammed into the code without much > thought. Dammit, can we please stop this and clean up the existing mess > first before duct taping more mess on top of it. Sorry for adding to the mess here. Thanks, Mark. [1] https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/kasan-atomic [2] https://lkml.kernel.org/r/20170628124552.GG5981@leverpostej [3] http://lkml.kernel.org/r/1489574142-20856-1-git-send-email-mark.rutland@arm.com ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr @ 2017-06-28 14:14 ` Mark Rutland 0 siblings, 0 replies; 63+ messages in thread From: Mark Rutland @ 2017-06-28 14:14 UTC (permalink / raw) To: Thomas Gleixner Cc: Sebastian Andrzej Siewior, Andrey Ryabinin, Ingo Molnar, Dmitry Vyukov, Peter Zijlstra, Will Deacon, H. Peter Anvin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, Jun 28, 2017 at 03:54:42PM +0200, Thomas Gleixner wrote: > On Wed, 28 Jun 2017, Thomas Gleixner wrote: > > On Wed, 28 Jun 2017, Sebastian Andrzej Siewior wrote: > > > On 2017-06-28 14:15:18 [+0300], Andrey Ryabinin wrote: > > > > The main problem here is that arch_cmpxchg64_local() calls cmpxhg_local() instead of using arch_cmpxchg_local(). > > > > > > > > So, the patch bellow should fix the problem, also this will fix double instrumentation of cmpcxchg64[_local](). > > > > But I haven't tested this patch yet. > > > > > > tested, works. Next step? > > > > Check all other implementations in every architecture whether there is a > > similar problem ..... FWIW, as x86 is the only user of atomic-instrumented.h, any similar issues are unrelated to this series. That's not to say they don't exist, just that they're orthognal to this. I've been reworking things for arm64 [1], but there's more cleanup needed first. > > But this really want's a proper cleanup unless we want to waste the time > > over and over again with the next hard to figure out macro expansion fail. > > > > First of all, cmpxchg64[_local]() can be implemented as inlines right away. > > > > For cmpxchg*(), the situation is slightly different, but the sizeof() > > evaluation should be done at the top most level, even if we do it further > > down in the low level arch/asm-generic implementation once more. > > > > Something along the lines of: > > > > static inline unsigned long cmpxchg_varsize(void *ptr, unsigned long old, > > unsigned long new, int size) > > { > > switch (size) { > > case 1: > > case 2: > > case 4: > > break; > > case 8: > > if (sizeof(unsigned long) == 8) > > break; > > default: > > BUILD_BUG_ON(1); > > } > > kasan_check(ptr, size); > > return arch_cmpxchg(ptr, old, new); > > } This'll need to re-cast things before the call to arch_cmpxchg(), and we can move the check above the switch, as in [2]. > > #define cmpxchg(ptr, o, n) \ > > ({ \ > > ((__typeof__(*(ptr)))cmpxchg_varsize((ptr), (unsigned long)(o), \ > > (unsigned long)(n), sizeof(*(ptr)))); \ > > }) > > > > That's the first step to cure the actual mess. > > > > Ideally we get rid of that whole macro maze and convert everything to > > proper inlines with actual cmpxchg8/16/32/64() variants, but that's going > > to take some time. As an intermediate step we can at least propagate 'size' > > to arch_cmpxchg(), which is not that much of an effort. > > And to be honest. That should have be done in the first place _BEFORE_ > adding that atomic-instrumented stuff. I'm tempted to revert that mess > instead of 'fixing' it half arsed. Sure. Let's figure out what this *should* look like first. If that's sufficiently different to what we have now, we revert this and clean things up first. > As a side note, we have files (aside of x86/asm/atomic.h) which include > asm/cmpxchg.h ... > > net/sunrpc/xprtmultipath.c:#include <asm/cmpxchg.h> > arch/x86/kvm/mmu.c:#include <asm/cmpxchg.h> > arch/x86/um/asm/barrier.h:#include <asm/cmpxchg.h> Ugh. I'd sent out a patch [3] for the first of these a while back, as I spotted that when experimenting with arm64, but tht got dropped on the floor. I can resend that, if you like? I guess it'd also make sense to fix the x86 bits at the same time, so I'm fine with tahat being folded with other fixes. > I'm really tired of all this featuritis crammed into the code without much > thought. Dammit, can we please stop this and clean up the existing mess > first before duct taping more mess on top of it. Sorry for adding to the mess here. Thanks, Mark. [1] https://git.kernel.org/pub/scm/linux/kernel/git/mark/linux.git/log/?h=arm64/kasan-atomic [2] https://lkml.kernel.org/r/20170628124552.GG5981@leverpostej [3] http://lkml.kernel.org/r/1489574142-20856-1-git-send-email-mark.rutland@arm.com -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr 2017-06-28 14:14 ` Mark Rutland @ 2017-06-28 15:24 ` Thomas Gleixner -1 siblings, 0 replies; 63+ messages in thread From: Thomas Gleixner @ 2017-06-28 15:24 UTC (permalink / raw) To: Mark Rutland Cc: Sebastian Andrzej Siewior, Andrey Ryabinin, Ingo Molnar, Dmitry Vyukov, Peter Zijlstra, Will Deacon, H. Peter Anvin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, 28 Jun 2017, Mark Rutland wrote: > On Wed, Jun 28, 2017 at 03:54:42PM +0200, Thomas Gleixner wrote: > > > static inline unsigned long cmpxchg_varsize(void *ptr, unsigned long old, > > > unsigned long new, int size) > > > { > > > switch (size) { > > > case 1: > > > case 2: > > > case 4: > > > break; > > > case 8: > > > if (sizeof(unsigned long) == 8) > > > break; > > > default: > > > BUILD_BUG_ON(1); > > > } > > > kasan_check(ptr, size); > > > return arch_cmpxchg(ptr, old, new); > > > } > > This'll need to re-cast things before the call to arch_cmpxchg(), and we > can move the check above the switch, as in [2]. Sure, but I rather see that changed to: 1) Create arch_cmpxchg8/16/32/64() inlines first 2) Add that varsize wrapper: static inline unsigned long cmpxchg_varsize(void *ptr, unsigned long old, unsigned long new, int size) { switch (size) { case 1: kasan_check_write(ptr, size); return arch_cmpxchg8((u8 *)ptr, (u8) old, (u8)new); case 2: kasan_check_write(ptr, size); return arch_cmpxchg16((u16 *)ptr, (u16) old, (u16)new); case 4: kasan_check_write(ptr, size); return arch_cmpxchg32((u32 *)ptr, (u32) old, (u32)new); case 8: if (sizeof(unsigned long) == 8) { kasan_check_write(ptr, size); return arch_cmpxchg64((u64 *)ptr, (u64) old, (u64)new); } default: BUILD_BUG(); } } #define cmpxchg(ptr, o, n) \ ({ \ ((__typeof__(*(ptr)))cmpxchg_varsize((ptr), (unsigned long)(o), \ (unsigned long)(n), sizeof(*(ptr)))); \ }) Which allows us to create: static inline u8 cmpxchg8(u8 *ptr, u8 old, u8 new) { kasan_check_write(ptr, sizeof(old)); return arch_cmpxchg8(ptr, old, new); } and friends as well and later migrate the existing users away from that untyped macro mess. And instead of adding #include <asm/atomic-instrumented.h> to the architecture code, we rather do # mv arch/xxx/include/asm/atomic.h mv arch/xxx/include/asm/arch_atomic.h # echo '#include <asm-generic/atomic.h>' >arch/xxx/include/asm/atomic.h # mv include/asm-generic/atomic.h include/asm-generic/atomic_up.h and create a new include/asm-generic/atomic.h #ifndef __ASM_GENERIC_ATOMIC_H #define __ASM_GENERIC_ATOMIC_H #ifdef CONFIG_ATOMIC_INSTRUMENTED_H #include <asm-generic/atomic_instrumented.h> #else #include <asm-generic/atomic_up.h> #endif #endif Thanks, tglx ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr @ 2017-06-28 15:24 ` Thomas Gleixner 0 siblings, 0 replies; 63+ messages in thread From: Thomas Gleixner @ 2017-06-28 15:24 UTC (permalink / raw) To: Mark Rutland Cc: Sebastian Andrzej Siewior, Andrey Ryabinin, Ingo Molnar, Dmitry Vyukov, Peter Zijlstra, Will Deacon, H. Peter Anvin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, 28 Jun 2017, Mark Rutland wrote: > On Wed, Jun 28, 2017 at 03:54:42PM +0200, Thomas Gleixner wrote: > > > static inline unsigned long cmpxchg_varsize(void *ptr, unsigned long old, > > > unsigned long new, int size) > > > { > > > switch (size) { > > > case 1: > > > case 2: > > > case 4: > > > break; > > > case 8: > > > if (sizeof(unsigned long) == 8) > > > break; > > > default: > > > BUILD_BUG_ON(1); > > > } > > > kasan_check(ptr, size); > > > return arch_cmpxchg(ptr, old, new); > > > } > > This'll need to re-cast things before the call to arch_cmpxchg(), and we > can move the check above the switch, as in [2]. Sure, but I rather see that changed to: 1) Create arch_cmpxchg8/16/32/64() inlines first 2) Add that varsize wrapper: static inline unsigned long cmpxchg_varsize(void *ptr, unsigned long old, unsigned long new, int size) { switch (size) { case 1: kasan_check_write(ptr, size); return arch_cmpxchg8((u8 *)ptr, (u8) old, (u8)new); case 2: kasan_check_write(ptr, size); return arch_cmpxchg16((u16 *)ptr, (u16) old, (u16)new); case 4: kasan_check_write(ptr, size); return arch_cmpxchg32((u32 *)ptr, (u32) old, (u32)new); case 8: if (sizeof(unsigned long) == 8) { kasan_check_write(ptr, size); return arch_cmpxchg64((u64 *)ptr, (u64) old, (u64)new); } default: BUILD_BUG(); } } #define cmpxchg(ptr, o, n) \ ({ \ ((__typeof__(*(ptr)))cmpxchg_varsize((ptr), (unsigned long)(o), \ (unsigned long)(n), sizeof(*(ptr)))); \ }) Which allows us to create: static inline u8 cmpxchg8(u8 *ptr, u8 old, u8 new) { kasan_check_write(ptr, sizeof(old)); return arch_cmpxchg8(ptr, old, new); } and friends as well and later migrate the existing users away from that untyped macro mess. And instead of adding #include <asm/atomic-instrumented.h> to the architecture code, we rather do # mv arch/xxx/include/asm/atomic.h mv arch/xxx/include/asm/arch_atomic.h # echo '#include <asm-generic/atomic.h>' >arch/xxx/include/asm/atomic.h # mv include/asm-generic/atomic.h include/asm-generic/atomic_up.h and create a new include/asm-generic/atomic.h #ifndef __ASM_GENERIC_ATOMIC_H #define __ASM_GENERIC_ATOMIC_H #ifdef CONFIG_ATOMIC_INSTRUMENTED_H #include <asm-generic/atomic_instrumented.h> #else #include <asm-generic/atomic_up.h> #endif #endif Thanks, tglx -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr 2017-06-28 15:24 ` Thomas Gleixner @ 2017-06-28 15:54 ` Mark Rutland -1 siblings, 0 replies; 63+ messages in thread From: Mark Rutland @ 2017-06-28 15:54 UTC (permalink / raw) To: Thomas Gleixner Cc: Sebastian Andrzej Siewior, Andrey Ryabinin, Ingo Molnar, Dmitry Vyukov, Peter Zijlstra, Will Deacon, H. Peter Anvin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, Jun 28, 2017 at 05:24:24PM +0200, Thomas Gleixner wrote: > On Wed, 28 Jun 2017, Mark Rutland wrote: > > On Wed, Jun 28, 2017 at 03:54:42PM +0200, Thomas Gleixner wrote: > > > > static inline unsigned long cmpxchg_varsize(void *ptr, unsigned long old, > > > > unsigned long new, int size) > > > > { > > > > switch (size) { > > > > case 1: > > > > case 2: > > > > case 4: > > > > break; > > > > case 8: > > > > if (sizeof(unsigned long) == 8) > > > > break; > > > > default: > > > > BUILD_BUG_ON(1); > > > > } > > > > kasan_check(ptr, size); > > > > return arch_cmpxchg(ptr, old, new); > > > > } > > > > This'll need to re-cast things before the call to arch_cmpxchg(), and we > > can move the check above the switch, as in [2]. > > Sure, but I rather see that changed to: > > 1) Create arch_cmpxchg8/16/32/64() inlines first > > 2) Add that varsize wrapper: > > static inline unsigned long cmpxchg_varsize(void *ptr, unsigned long old, > unsigned long new, int size) > { > switch (size) { > case 1: > kasan_check_write(ptr, size); > return arch_cmpxchg8((u8 *)ptr, (u8) old, (u8)new); > case 2: > kasan_check_write(ptr, size); > return arch_cmpxchg16((u16 *)ptr, (u16) old, (u16)new); > case 4: > kasan_check_write(ptr, size); > return arch_cmpxchg32((u32 *)ptr, (u32) old, (u32)new); > case 8: > if (sizeof(unsigned long) == 8) { > kasan_check_write(ptr, size); > return arch_cmpxchg64((u64 *)ptr, (u64) old, (u64)new); > } > default: > BUILD_BUG(); > } > } > > #define cmpxchg(ptr, o, n) \ > ({ \ > ((__typeof__(*(ptr)))cmpxchg_varsize((ptr), (unsigned long)(o), \ > (unsigned long)(n), sizeof(*(ptr)))); \ > }) > > Which allows us to create: > > static inline u8 cmpxchg8(u8 *ptr, u8 old, u8 new) > { > kasan_check_write(ptr, sizeof(old)); > return arch_cmpxchg8(ptr, old, new); > } > > and friends as well and later migrate the existing users away from that > untyped macro mess. Sure, that makes sense to me. > > And instead of adding > > #include <asm/atomic-instrumented.h> > > to the architecture code, we rather do > > # mv arch/xxx/include/asm/atomic.h mv arch/xxx/include/asm/arch_atomic.h > # echo '#include <asm-generic/atomic.h>' >arch/xxx/include/asm/atomic.h > > # mv include/asm-generic/atomic.h include/asm-generic/atomic_up.h > > and create a new include/asm-generic/atomic.h > > #ifndef __ASM_GENERIC_ATOMIC_H > #define __ASM_GENERIC_ATOMIC_H > > #ifdef CONFIG_ATOMIC_INSTRUMENTED_H > #include <asm-generic/atomic_instrumented.h> > #else > #include <asm-generic/atomic_up.h> > #endif > > #endif Given we're gonig to clean things up, we may as well avoid the backwards include of <asm-generic/atomic_instrumented.h>, whcih was only there as a bodge: For the UP arches we do: # echo '#include <asm-generic/atomic_up.h>' >arch/xxx/include/asm/atomic.h # mv include/asm-generic/atomic.h include/asm-generic/atomic_up.h Then we add a <linux/atomic_instrumented.h>: #ifndef __LINUX_ATOMIC_INSTRUMENTED_H #define __LINUX_ATOMIC INSTRUMENTED_H #include <asm/atomic.h> #if CONFIG_ATOMIC_INSTRUMENTED_H <instrumentation> #endif #endif /* __LINUX_ATOMIC_ARCH_H */ ... and make <linux/atomic.h> incldue that rather than <asm/atomic.h>. That way the instrumentation's orthogonal to the UP-ness of the arch, and we can fold any other instrumentation in there, or later move it directly into <linux/atomic.h> Thanks, Mark. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr @ 2017-06-28 15:54 ` Mark Rutland 0 siblings, 0 replies; 63+ messages in thread From: Mark Rutland @ 2017-06-28 15:54 UTC (permalink / raw) To: Thomas Gleixner Cc: Sebastian Andrzej Siewior, Andrey Ryabinin, Ingo Molnar, Dmitry Vyukov, Peter Zijlstra, Will Deacon, H. Peter Anvin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, Jun 28, 2017 at 05:24:24PM +0200, Thomas Gleixner wrote: > On Wed, 28 Jun 2017, Mark Rutland wrote: > > On Wed, Jun 28, 2017 at 03:54:42PM +0200, Thomas Gleixner wrote: > > > > static inline unsigned long cmpxchg_varsize(void *ptr, unsigned long old, > > > > unsigned long new, int size) > > > > { > > > > switch (size) { > > > > case 1: > > > > case 2: > > > > case 4: > > > > break; > > > > case 8: > > > > if (sizeof(unsigned long) == 8) > > > > break; > > > > default: > > > > BUILD_BUG_ON(1); > > > > } > > > > kasan_check(ptr, size); > > > > return arch_cmpxchg(ptr, old, new); > > > > } > > > > This'll need to re-cast things before the call to arch_cmpxchg(), and we > > can move the check above the switch, as in [2]. > > Sure, but I rather see that changed to: > > 1) Create arch_cmpxchg8/16/32/64() inlines first > > 2) Add that varsize wrapper: > > static inline unsigned long cmpxchg_varsize(void *ptr, unsigned long old, > unsigned long new, int size) > { > switch (size) { > case 1: > kasan_check_write(ptr, size); > return arch_cmpxchg8((u8 *)ptr, (u8) old, (u8)new); > case 2: > kasan_check_write(ptr, size); > return arch_cmpxchg16((u16 *)ptr, (u16) old, (u16)new); > case 4: > kasan_check_write(ptr, size); > return arch_cmpxchg32((u32 *)ptr, (u32) old, (u32)new); > case 8: > if (sizeof(unsigned long) == 8) { > kasan_check_write(ptr, size); > return arch_cmpxchg64((u64 *)ptr, (u64) old, (u64)new); > } > default: > BUILD_BUG(); > } > } > > #define cmpxchg(ptr, o, n) \ > ({ \ > ((__typeof__(*(ptr)))cmpxchg_varsize((ptr), (unsigned long)(o), \ > (unsigned long)(n), sizeof(*(ptr)))); \ > }) > > Which allows us to create: > > static inline u8 cmpxchg8(u8 *ptr, u8 old, u8 new) > { > kasan_check_write(ptr, sizeof(old)); > return arch_cmpxchg8(ptr, old, new); > } > > and friends as well and later migrate the existing users away from that > untyped macro mess. Sure, that makes sense to me. > > And instead of adding > > #include <asm/atomic-instrumented.h> > > to the architecture code, we rather do > > # mv arch/xxx/include/asm/atomic.h mv arch/xxx/include/asm/arch_atomic.h > # echo '#include <asm-generic/atomic.h>' >arch/xxx/include/asm/atomic.h > > # mv include/asm-generic/atomic.h include/asm-generic/atomic_up.h > > and create a new include/asm-generic/atomic.h > > #ifndef __ASM_GENERIC_ATOMIC_H > #define __ASM_GENERIC_ATOMIC_H > > #ifdef CONFIG_ATOMIC_INSTRUMENTED_H > #include <asm-generic/atomic_instrumented.h> > #else > #include <asm-generic/atomic_up.h> > #endif > > #endif Given we're gonig to clean things up, we may as well avoid the backwards include of <asm-generic/atomic_instrumented.h>, whcih was only there as a bodge: For the UP arches we do: # echo '#include <asm-generic/atomic_up.h>' >arch/xxx/include/asm/atomic.h # mv include/asm-generic/atomic.h include/asm-generic/atomic_up.h Then we add a <linux/atomic_instrumented.h>: #ifndef __LINUX_ATOMIC_INSTRUMENTED_H #define __LINUX_ATOMIC INSTRUMENTED_H #include <asm/atomic.h> #if CONFIG_ATOMIC_INSTRUMENTED_H <instrumentation> #endif #endif /* __LINUX_ATOMIC_ARCH_H */ ... and make <linux/atomic.h> incldue that rather than <asm/atomic.h>. That way the instrumentation's orthogonal to the UP-ness of the arch, and we can fold any other instrumentation in there, or later move it directly into <linux/atomic.h> Thanks, Mark. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr 2017-06-28 15:54 ` Mark Rutland @ 2017-06-28 16:56 ` Ingo Molnar -1 siblings, 0 replies; 63+ messages in thread From: Ingo Molnar @ 2017-06-28 16:56 UTC (permalink / raw) To: Mark Rutland Cc: Thomas Gleixner, Sebastian Andrzej Siewior, Andrey Ryabinin, Dmitry Vyukov, Peter Zijlstra, Will Deacon, H. Peter Anvin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds * Mark Rutland <mark.rutland@arm.com> wrote: > > And instead of adding > > > > #include <asm/atomic-instrumented.h> > > > > to the architecture code, we rather do > > > > # mv arch/xxx/include/asm/atomic.h mv arch/xxx/include/asm/arch_atomic.h > > # echo '#include <asm-generic/atomic.h>' >arch/xxx/include/asm/atomic.h > > > > # mv include/asm-generic/atomic.h include/asm-generic/atomic_up.h > > > > and create a new include/asm-generic/atomic.h > > > > #ifndef __ASM_GENERIC_ATOMIC_H > > #define __ASM_GENERIC_ATOMIC_H > > > > #ifdef CONFIG_ATOMIC_INSTRUMENTED_H > > #include <asm-generic/atomic_instrumented.h> > > #else > > #include <asm-generic/atomic_up.h> > > #endif > > > > #endif > > Given we're gonig to clean things up, we may as well avoid the backwards > include of <asm-generic/atomic_instrumented.h>, whcih was only there as > a bodge: So, since the final v4.12 release is so close, I've put the following KASAN commits aside into tip:WIP.locking/atomics: 4b47cc154eed: locking/atomic/x86, asm-generic: Add comments for atomic instrumentation 35787d9d7ca4: locking/atomics, asm-generic: Add KASAN instrumentation to atomic operations 68c1ed1fdb0a: kasan: Allow kasan_check_read/write() to accept pointers to volatiles f1c3049f6729: locking/atomic/x86: Switch atomic.h to use atomic-instrumented.h d079eebb3958: locking/atomic: Add asm-generic/atomic-instrumented.h 007d185b4462: locking/atomic/x86: Use 's64 *' for 'old' argument of atomic64_try_cmpxchg() ba1c9f83f633: locking/atomic/x86: Un-macro-ify atomic ops implementation (Note, I had to rebase these freshly, to decouple them from a refcount_t commit that we need.) and won't send them to Linus unless the cleanups are done and acked by Thomas. Thanks, Ingo ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr @ 2017-06-28 16:56 ` Ingo Molnar 0 siblings, 0 replies; 63+ messages in thread From: Ingo Molnar @ 2017-06-28 16:56 UTC (permalink / raw) To: Mark Rutland Cc: Thomas Gleixner, Sebastian Andrzej Siewior, Andrey Ryabinin, Dmitry Vyukov, Peter Zijlstra, Will Deacon, H. Peter Anvin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds * Mark Rutland <mark.rutland@arm.com> wrote: > > And instead of adding > > > > #include <asm/atomic-instrumented.h> > > > > to the architecture code, we rather do > > > > # mv arch/xxx/include/asm/atomic.h mv arch/xxx/include/asm/arch_atomic.h > > # echo '#include <asm-generic/atomic.h>' >arch/xxx/include/asm/atomic.h > > > > # mv include/asm-generic/atomic.h include/asm-generic/atomic_up.h > > > > and create a new include/asm-generic/atomic.h > > > > #ifndef __ASM_GENERIC_ATOMIC_H > > #define __ASM_GENERIC_ATOMIC_H > > > > #ifdef CONFIG_ATOMIC_INSTRUMENTED_H > > #include <asm-generic/atomic_instrumented.h> > > #else > > #include <asm-generic/atomic_up.h> > > #endif > > > > #endif > > Given we're gonig to clean things up, we may as well avoid the backwards > include of <asm-generic/atomic_instrumented.h>, whcih was only there as > a bodge: So, since the final v4.12 release is so close, I've put the following KASAN commits aside into tip:WIP.locking/atomics: 4b47cc154eed: locking/atomic/x86, asm-generic: Add comments for atomic instrumentation 35787d9d7ca4: locking/atomics, asm-generic: Add KASAN instrumentation to atomic operations 68c1ed1fdb0a: kasan: Allow kasan_check_read/write() to accept pointers to volatiles f1c3049f6729: locking/atomic/x86: Switch atomic.h to use atomic-instrumented.h d079eebb3958: locking/atomic: Add asm-generic/atomic-instrumented.h 007d185b4462: locking/atomic/x86: Use 's64 *' for 'old' argument of atomic64_try_cmpxchg() ba1c9f83f633: locking/atomic/x86: Un-macro-ify atomic ops implementation (Note, I had to rebase these freshly, to decouple them from a refcount_t commit that we need.) and won't send them to Linus unless the cleanups are done and acked by Thomas. Thanks, Ingo -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr 2017-06-28 15:54 ` Mark Rutland @ 2017-06-28 18:21 ` Thomas Gleixner -1 siblings, 0 replies; 63+ messages in thread From: Thomas Gleixner @ 2017-06-28 18:21 UTC (permalink / raw) To: Mark Rutland Cc: Sebastian Andrzej Siewior, Andrey Ryabinin, Ingo Molnar, Dmitry Vyukov, Peter Zijlstra, Will Deacon, H. Peter Anvin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, 28 Jun 2017, Mark Rutland wrote: > On Wed, Jun 28, 2017 at 05:24:24PM +0200, Thomas Gleixner wrote: > Given we're gonig to clean things up, we may as well avoid the backwards > include of <asm-generic/atomic_instrumented.h>, whcih was only there as > a bodge: > > For the UP arches we do: > # echo '#include <asm-generic/atomic_up.h>' >arch/xxx/include/asm/atomic.h > # mv include/asm-generic/atomic.h include/asm-generic/atomic_up.h > > Then we add a <linux/atomic_instrumented.h>: > > #ifndef __LINUX_ATOMIC_INSTRUMENTED_H > #define __LINUX_ATOMIC INSTRUMENTED_H > > #include <asm/atomic.h> > > #if CONFIG_ATOMIC_INSTRUMENTED_H > <instrumentation> > #endif > > #endif /* __LINUX_ATOMIC_ARCH_H */ > > ... and make <linux/atomic.h> incldue that rather than <asm/atomic.h>. > > That way the instrumentation's orthogonal to the UP-ness of the arch, > and we can fold any other instrumentation in there, or later move it > directly into <linux/atomic.h> Sounds like a plan. Thanks, tglx ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr @ 2017-06-28 18:21 ` Thomas Gleixner 0 siblings, 0 replies; 63+ messages in thread From: Thomas Gleixner @ 2017-06-28 18:21 UTC (permalink / raw) To: Mark Rutland Cc: Sebastian Andrzej Siewior, Andrey Ryabinin, Ingo Molnar, Dmitry Vyukov, Peter Zijlstra, Will Deacon, H. Peter Anvin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, 28 Jun 2017, Mark Rutland wrote: > On Wed, Jun 28, 2017 at 05:24:24PM +0200, Thomas Gleixner wrote: > Given we're gonig to clean things up, we may as well avoid the backwards > include of <asm-generic/atomic_instrumented.h>, whcih was only there as > a bodge: > > For the UP arches we do: > # echo '#include <asm-generic/atomic_up.h>' >arch/xxx/include/asm/atomic.h > # mv include/asm-generic/atomic.h include/asm-generic/atomic_up.h > > Then we add a <linux/atomic_instrumented.h>: > > #ifndef __LINUX_ATOMIC_INSTRUMENTED_H > #define __LINUX_ATOMIC INSTRUMENTED_H > > #include <asm/atomic.h> > > #if CONFIG_ATOMIC_INSTRUMENTED_H > <instrumentation> > #endif > > #endif /* __LINUX_ATOMIC_ARCH_H */ > > ... and make <linux/atomic.h> incldue that rather than <asm/atomic.h>. > > That way the instrumentation's orthogonal to the UP-ness of the arch, > and we can fold any other instrumentation in there, or later move it > directly into <linux/atomic.h> Sounds like a plan. Thanks, tglx -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr 2017-06-28 18:21 ` Thomas Gleixner @ 2017-06-29 6:47 ` Thomas Gleixner -1 siblings, 0 replies; 63+ messages in thread From: Thomas Gleixner @ 2017-06-29 6:47 UTC (permalink / raw) To: Mark Rutland Cc: Sebastian Andrzej Siewior, Andrey Ryabinin, Ingo Molnar, Dmitry Vyukov, Peter Zijlstra, Will Deacon, H. Peter Anvin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, 28 Jun 2017, Thomas Gleixner wrote: > On Wed, 28 Jun 2017, Mark Rutland wrote: > > On Wed, Jun 28, 2017 at 05:24:24PM +0200, Thomas Gleixner wrote: > > Given we're gonig to clean things up, we may as well avoid the backwards > > include of <asm-generic/atomic_instrumented.h>, whcih was only there as > > a bodge: > > > > For the UP arches we do: > > # echo '#include <asm-generic/atomic_up.h>' >arch/xxx/include/asm/atomic.h > > # mv include/asm-generic/atomic.h include/asm-generic/atomic_up.h > > > > Then we add a <linux/atomic_instrumented.h>: > > > > #ifndef __LINUX_ATOMIC_INSTRUMENTED_H > > #define __LINUX_ATOMIC INSTRUMENTED_H > > > > #include <asm/atomic.h> > > > > #if CONFIG_ATOMIC_INSTRUMENTED_H > > <instrumentation> > > #endif > > > > #endif /* __LINUX_ATOMIC_ARCH_H */ > > > > ... and make <linux/atomic.h> incldue that rather than <asm/atomic.h>. > > > > That way the instrumentation's orthogonal to the UP-ness of the arch, > > and we can fold any other instrumentation in there, or later move it > > directly into <linux/atomic.h> > > Sounds like a plan. Actually we should make it slightly different and make asm-generic/atomic.h the central point for everything. It should contain the wrapper macro and the central inlines including the kasan stuff and include either arch/arch_atomic.h or asm-generic/atomic_up.h. That way all potential instrumentation happens in the generic header (which is a NOP for archs which do not support it) and pull in the appropriate arch specific or generic UP low level implementations. Thanks, tglx ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr @ 2017-06-29 6:47 ` Thomas Gleixner 0 siblings, 0 replies; 63+ messages in thread From: Thomas Gleixner @ 2017-06-29 6:47 UTC (permalink / raw) To: Mark Rutland Cc: Sebastian Andrzej Siewior, Andrey Ryabinin, Ingo Molnar, Dmitry Vyukov, Peter Zijlstra, Will Deacon, H. Peter Anvin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On Wed, 28 Jun 2017, Thomas Gleixner wrote: > On Wed, 28 Jun 2017, Mark Rutland wrote: > > On Wed, Jun 28, 2017 at 05:24:24PM +0200, Thomas Gleixner wrote: > > Given we're gonig to clean things up, we may as well avoid the backwards > > include of <asm-generic/atomic_instrumented.h>, whcih was only there as > > a bodge: > > > > For the UP arches we do: > > # echo '#include <asm-generic/atomic_up.h>' >arch/xxx/include/asm/atomic.h > > # mv include/asm-generic/atomic.h include/asm-generic/atomic_up.h > > > > Then we add a <linux/atomic_instrumented.h>: > > > > #ifndef __LINUX_ATOMIC_INSTRUMENTED_H > > #define __LINUX_ATOMIC INSTRUMENTED_H > > > > #include <asm/atomic.h> > > > > #if CONFIG_ATOMIC_INSTRUMENTED_H > > <instrumentation> > > #endif > > > > #endif /* __LINUX_ATOMIC_ARCH_H */ > > > > ... and make <linux/atomic.h> incldue that rather than <asm/atomic.h>. > > > > That way the instrumentation's orthogonal to the UP-ness of the arch, > > and we can fold any other instrumentation in there, or later move it > > directly into <linux/atomic.h> > > Sounds like a plan. Actually we should make it slightly different and make asm-generic/atomic.h the central point for everything. It should contain the wrapper macro and the central inlines including the kasan stuff and include either arch/arch_atomic.h or asm-generic/atomic_up.h. That way all potential instrumentation happens in the generic header (which is a NOP for archs which do not support it) and pull in the appropriate arch specific or generic UP low level implementations. Thanks, tglx -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr 2017-06-28 13:20 ` Thomas Gleixner @ 2017-06-28 14:00 ` Andrey Ryabinin -1 siblings, 0 replies; 63+ messages in thread From: Andrey Ryabinin @ 2017-06-28 14:00 UTC (permalink / raw) To: Thomas Gleixner, Sebastian Andrzej Siewior Cc: Ingo Molnar, Dmitry Vyukov, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On 06/28/2017 04:20 PM, Thomas Gleixner wrote: > On Wed, 28 Jun 2017, Sebastian Andrzej Siewior wrote: >> On 2017-06-28 14:15:18 [+0300], Andrey Ryabinin wrote: >>> The main problem here is that arch_cmpxchg64_local() calls cmpxhg_local() instead of using arch_cmpxchg_local(). >>> >>> So, the patch bellow should fix the problem, also this will fix double instrumentation of cmpcxchg64[_local](). >>> But I haven't tested this patch yet. >> >> tested, works. Next step? > > Check all other implementations in every architecture whether there is a > similar problem ..... > This and similar problems could have been caught by -Wshadow warning: In file included from ../arch/x86/include/asm/atomic.h:282:0, from ../include/linux/atomic.h:4, from ../include/linux/jump_label.h:183, from ../arch/x86/include/asm/string_64.h:5, from ../arch/x86/include/asm/string.h:4, from ../include/linux/string.h:18, from ../include/linux/bitmap.h:8, from ../drivers/iommu/intel-iommu.c:24: ../include/asm-generic/atomic-instrumented.h:376:18: warning: declaration of ‘____ptr’ shadows a previous local [-Wshadow] __typeof__(ptr) ____ptr = (ptr); \ ^ ../arch/x86/include/asm/cmpxchg_64.h:18:2: note: in expansion of macro ‘cmpxchg_local’ cmpxchg_local((ptr), (o), (n)); \ ^ ../include/asm-generic/atomic-instrumented.h:392:2: note: in expansion of macro ‘arch_cmpxchg64_local’ arch_cmpxchg64_local(____ptr, (old), (new)); \ ^ ../drivers/iommu/intel-iommu.c:2290:9: note: in expansion of macro ‘cmpxchg64_local’ tmp = cmpxchg64_local(&pte->val, 0ULL, pteval); ^ ../include/asm-generic/atomic-instrumented.h:390:18: note: shadowed declaration is here __typeof__(ptr) ____ptr = (ptr); \ ^ ../drivers/iommu/intel-iommu.c:2290:9: note: in expansion of macro ‘cmpxchg64_local’ tmp = cmpxchg64_local(&pte->val, 0ULL, pteval); ^ But for some reason we use -Wshadow only on W=2 level. ^ permalink raw reply [flat|nested] 63+ messages in thread
* Re: [PATCH] locking/atomics: don't alias ____ptr @ 2017-06-28 14:00 ` Andrey Ryabinin 0 siblings, 0 replies; 63+ messages in thread From: Andrey Ryabinin @ 2017-06-28 14:00 UTC (permalink / raw) To: Thomas Gleixner, Sebastian Andrzej Siewior Cc: Ingo Molnar, Dmitry Vyukov, Mark Rutland, Peter Zijlstra, Will Deacon, H. Peter Anvin, kasan-dev, x86, LKML, Andrew Morton, linux-mm, Linus Torvalds On 06/28/2017 04:20 PM, Thomas Gleixner wrote: > On Wed, 28 Jun 2017, Sebastian Andrzej Siewior wrote: >> On 2017-06-28 14:15:18 [+0300], Andrey Ryabinin wrote: >>> The main problem here is that arch_cmpxchg64_local() calls cmpxhg_local() instead of using arch_cmpxchg_local(). >>> >>> So, the patch bellow should fix the problem, also this will fix double instrumentation of cmpcxchg64[_local](). >>> But I haven't tested this patch yet. >> >> tested, works. Next step? > > Check all other implementations in every architecture whether there is a > similar problem ..... > This and similar problems could have been caught by -Wshadow warning: In file included from ../arch/x86/include/asm/atomic.h:282:0, from ../include/linux/atomic.h:4, from ../include/linux/jump_label.h:183, from ../arch/x86/include/asm/string_64.h:5, from ../arch/x86/include/asm/string.h:4, from ../include/linux/string.h:18, from ../include/linux/bitmap.h:8, from ../drivers/iommu/intel-iommu.c:24: ../include/asm-generic/atomic-instrumented.h:376:18: warning: declaration of a??____ptra?? shadows a previous local [-Wshadow] __typeof__(ptr) ____ptr = (ptr); \ ^ ../arch/x86/include/asm/cmpxchg_64.h:18:2: note: in expansion of macro a??cmpxchg_locala?? cmpxchg_local((ptr), (o), (n)); \ ^ ../include/asm-generic/atomic-instrumented.h:392:2: note: in expansion of macro a??arch_cmpxchg64_locala?? arch_cmpxchg64_local(____ptr, (old), (new)); \ ^ ../drivers/iommu/intel-iommu.c:2290:9: note: in expansion of macro a??cmpxchg64_locala?? tmp = cmpxchg64_local(&pte->val, 0ULL, pteval); ^ ../include/asm-generic/atomic-instrumented.h:390:18: note: shadowed declaration is here __typeof__(ptr) ____ptr = (ptr); \ ^ ../drivers/iommu/intel-iommu.c:2290:9: note: in expansion of macro a??cmpxchg64_locala?? tmp = cmpxchg64_local(&pte->val, 0ULL, pteval); ^ But for some reason we use -Wshadow only on W=2 level. -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply [flat|nested] 63+ messages in thread
* [PATCH v5 4/4] asm-generic, x86: add comments for atomic instrumentation [not found] ` <cover.1498140838.git.dvyukov@google.com> @ 2017-06-22 14:14 ` Dmitry Vyukov 2017-06-22 14:14 ` Dmitry Vyukov ` (2 subsequent siblings) 3 siblings, 0 replies; 63+ messages in thread From: Dmitry Vyukov @ 2017-06-22 14:14 UTC (permalink / raw) To: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin, kasan-dev, x86, linux-kernel Cc: Dmitry Vyukov, Andrew Morton, linux-mm The comments are factored out from the code changes to make them easier to read. Add them separately to explain some non-obvious aspects. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Will Deacon <will.deacon@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Ingo Molnar <mingo@redhat.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org Cc: x86@kernel.org --- Changes since v3: - rephrase comment in arch_atomic_read() --- arch/x86/include/asm/atomic.h | 4 ++++ include/asm-generic/atomic-instrumented.h | 30 ++++++++++++++++++++++++++++++ 2 files changed, 34 insertions(+) diff --git a/arch/x86/include/asm/atomic.h b/arch/x86/include/asm/atomic.h index 03dd2a6cf7e2..b1cd05da5d8a 100644 --- a/arch/x86/include/asm/atomic.h +++ b/arch/x86/include/asm/atomic.h @@ -23,6 +23,10 @@ */ static __always_inline int arch_atomic_read(const atomic_t *v) { + /* + * Note for KASAN: we deliberately don't use READ_ONCE_NOCHECK() here, + * it's non-inlined function that increases binary size and stack usage. + */ return READ_ONCE((v)->counter); } diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h index a0f5b7525bb2..5771439e7a31 100644 --- a/include/asm-generic/atomic-instrumented.h +++ b/include/asm-generic/atomic-instrumented.h @@ -1,3 +1,15 @@ +/* + * This file provides wrappers with KASAN instrumentation for atomic operations. + * To use this functionality an arch's atomic.h file needs to define all + * atomic operations with arch_ prefix (e.g. arch_atomic_read()) and include + * this file at the end. This file provides atomic_read() that forwards to + * arch_atomic_read() for actual atomic operation. + * Note: if an arch atomic operation is implemented by means of other atomic + * operations (e.g. atomic_read()/atomic_cmpxchg() loop), then it needs to use + * arch_ variants (i.e. arch_atomic_read()/arch_atomic_cmpxchg()) to avoid + * double instrumentation. + */ + #ifndef _LINUX_ATOMIC_INSTRUMENTED_H #define _LINUX_ATOMIC_INSTRUMENTED_H @@ -336,6 +348,15 @@ static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v) return arch_atomic64_add_negative(i, v); } +/* + * In the following macros we need to be careful to not clash with arch_ macros. + * arch_xchg() can be defined as an extended statement expression as well, + * if we define a __ptr variable, and arch_xchg() also defines __ptr variable, + * and we pass __ptr as an argument to arch_xchg(), it will use own __ptr + * instead of ours. This leads to unpleasant crashes. To avoid the problem + * the following macros declare variables with lots of underscores. + */ + #define cmpxchg(ptr, old, new) \ ({ \ __typeof__(ptr) ___ptr = (ptr); \ @@ -371,6 +392,15 @@ static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v) arch_cmpxchg64_local(____ptr, (old), (new)); \ }) +/* + * Originally we had the following code here: + * __typeof__(p1) ____p1 = (p1); + * kasan_check_write(____p1, 2 * sizeof(*____p1)); + * arch_cmpxchg_double(____p1, (p2), (o1), (o2), (n1), (n2)); + * But it leads to compilation failures (see gcc issue 72873). + * So for now it's left non-instrumented. + * There are few callers of cmpxchg_double(), so it's not critical. + */ #define cmpxchg_double(p1, p2, o1, o2, n1, n2) \ ({ \ arch_cmpxchg_double((p1), (p2), (o1), (o2), (n1), (n2)); \ -- 2.13.1.611.g7e3b11ae1-goog ^ permalink raw reply related [flat|nested] 63+ messages in thread
* [PATCH v5 4/4] asm-generic, x86: add comments for atomic instrumentation @ 2017-06-22 14:14 ` Dmitry Vyukov 0 siblings, 0 replies; 63+ messages in thread From: Dmitry Vyukov @ 2017-06-22 14:14 UTC (permalink / raw) To: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin, kasan-dev, x86, linux-kernel Cc: Dmitry Vyukov, Andrew Morton, linux-mm The comments are factored out from the code changes to make them easier to read. Add them separately to explain some non-obvious aspects. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Will Deacon <will.deacon@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Ingo Molnar <mingo@redhat.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org Cc: x86@kernel.org --- Changes since v3: - rephrase comment in arch_atomic_read() --- arch/x86/include/asm/atomic.h | 4 ++++ include/asm-generic/atomic-instrumented.h | 30 ++++++++++++++++++++++++++++++ 2 files changed, 34 insertions(+) diff --git a/arch/x86/include/asm/atomic.h b/arch/x86/include/asm/atomic.h index 03dd2a6cf7e2..b1cd05da5d8a 100644 --- a/arch/x86/include/asm/atomic.h +++ b/arch/x86/include/asm/atomic.h @@ -23,6 +23,10 @@ */ static __always_inline int arch_atomic_read(const atomic_t *v) { + /* + * Note for KASAN: we deliberately don't use READ_ONCE_NOCHECK() here, + * it's non-inlined function that increases binary size and stack usage. + */ return READ_ONCE((v)->counter); } diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h index a0f5b7525bb2..5771439e7a31 100644 --- a/include/asm-generic/atomic-instrumented.h +++ b/include/asm-generic/atomic-instrumented.h @@ -1,3 +1,15 @@ +/* + * This file provides wrappers with KASAN instrumentation for atomic operations. + * To use this functionality an arch's atomic.h file needs to define all + * atomic operations with arch_ prefix (e.g. arch_atomic_read()) and include + * this file at the end. This file provides atomic_read() that forwards to + * arch_atomic_read() for actual atomic operation. + * Note: if an arch atomic operation is implemented by means of other atomic + * operations (e.g. atomic_read()/atomic_cmpxchg() loop), then it needs to use + * arch_ variants (i.e. arch_atomic_read()/arch_atomic_cmpxchg()) to avoid + * double instrumentation. + */ + #ifndef _LINUX_ATOMIC_INSTRUMENTED_H #define _LINUX_ATOMIC_INSTRUMENTED_H @@ -336,6 +348,15 @@ static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v) return arch_atomic64_add_negative(i, v); } +/* + * In the following macros we need to be careful to not clash with arch_ macros. + * arch_xchg() can be defined as an extended statement expression as well, + * if we define a __ptr variable, and arch_xchg() also defines __ptr variable, + * and we pass __ptr as an argument to arch_xchg(), it will use own __ptr + * instead of ours. This leads to unpleasant crashes. To avoid the problem + * the following macros declare variables with lots of underscores. + */ + #define cmpxchg(ptr, old, new) \ ({ \ __typeof__(ptr) ___ptr = (ptr); \ @@ -371,6 +392,15 @@ static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v) arch_cmpxchg64_local(____ptr, (old), (new)); \ }) +/* + * Originally we had the following code here: + * __typeof__(p1) ____p1 = (p1); + * kasan_check_write(____p1, 2 * sizeof(*____p1)); + * arch_cmpxchg_double(____p1, (p2), (o1), (o2), (n1), (n2)); + * But it leads to compilation failures (see gcc issue 72873). + * So for now it's left non-instrumented. + * There are few callers of cmpxchg_double(), so it's not critical. + */ #define cmpxchg_double(p1, p2, o1, o2, n1, n2) \ ({ \ arch_cmpxchg_double((p1), (p2), (o1), (o2), (n1), (n2)); \ -- 2.13.1.611.g7e3b11ae1-goog -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a> ^ permalink raw reply related [flat|nested] 63+ messages in thread
* [tip:locking/core] locking/atomic/x86, asm-generic: Add comments for atomic instrumentation 2017-06-22 14:14 ` Dmitry Vyukov (?) @ 2017-06-23 14:01 ` tip-bot for Dmitry Vyukov -1 siblings, 0 replies; 63+ messages in thread From: tip-bot for Dmitry Vyukov @ 2017-06-23 14:01 UTC (permalink / raw) To: linux-tip-commits Cc: mark.rutland, tglx, aryabinin, linux-kernel, akpm, mingo, hpa, dvyukov, torvalds, will.deacon, peterz Commit-ID: 3f04aae282662e4075b6c3824aa3e1e1dabacd31 Gitweb: http://git.kernel.org/tip/3f04aae282662e4075b6c3824aa3e1e1dabacd31 Author: Dmitry Vyukov <dvyukov@google.com> AuthorDate: Thu, 22 Jun 2017 16:14:19 +0200 Committer: Ingo Molnar <mingo@kernel.org> CommitDate: Fri, 23 Jun 2017 10:50:20 +0200 locking/atomic/x86, asm-generic: Add comments for atomic instrumentation The comments are factored out from the code changes to make them easier to read. Add them separately to explain some non-obvious aspects. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Reviewed-by: Andrey Ryabinin <aryabinin@virtuozzo.com> Acked-by: Mark Rutland <mark.rutland@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/65058e2d09cf0920769ca72a932d9de4f613249d.1498140838.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org> --- arch/x86/include/asm/atomic.h | 4 ++++ include/asm-generic/atomic-instrumented.h | 30 ++++++++++++++++++++++++++++++ 2 files changed, 34 insertions(+) diff --git a/arch/x86/include/asm/atomic.h b/arch/x86/include/asm/atomic.h index 03dd2a6..b1cd05d 100644 --- a/arch/x86/include/asm/atomic.h +++ b/arch/x86/include/asm/atomic.h @@ -23,6 +23,10 @@ */ static __always_inline int arch_atomic_read(const atomic_t *v) { + /* + * Note for KASAN: we deliberately don't use READ_ONCE_NOCHECK() here, + * it's non-inlined function that increases binary size and stack usage. + */ return READ_ONCE((v)->counter); } diff --git a/include/asm-generic/atomic-instrumented.h b/include/asm-generic/atomic-instrumented.h index a0f5b75..5771439 100644 --- a/include/asm-generic/atomic-instrumented.h +++ b/include/asm-generic/atomic-instrumented.h @@ -1,3 +1,15 @@ +/* + * This file provides wrappers with KASAN instrumentation for atomic operations. + * To use this functionality an arch's atomic.h file needs to define all + * atomic operations with arch_ prefix (e.g. arch_atomic_read()) and include + * this file at the end. This file provides atomic_read() that forwards to + * arch_atomic_read() for actual atomic operation. + * Note: if an arch atomic operation is implemented by means of other atomic + * operations (e.g. atomic_read()/atomic_cmpxchg() loop), then it needs to use + * arch_ variants (i.e. arch_atomic_read()/arch_atomic_cmpxchg()) to avoid + * double instrumentation. + */ + #ifndef _LINUX_ATOMIC_INSTRUMENTED_H #define _LINUX_ATOMIC_INSTRUMENTED_H @@ -336,6 +348,15 @@ static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v) return arch_atomic64_add_negative(i, v); } +/* + * In the following macros we need to be careful to not clash with arch_ macros. + * arch_xchg() can be defined as an extended statement expression as well, + * if we define a __ptr variable, and arch_xchg() also defines __ptr variable, + * and we pass __ptr as an argument to arch_xchg(), it will use own __ptr + * instead of ours. This leads to unpleasant crashes. To avoid the problem + * the following macros declare variables with lots of underscores. + */ + #define cmpxchg(ptr, old, new) \ ({ \ __typeof__(ptr) ___ptr = (ptr); \ @@ -371,6 +392,15 @@ static __always_inline bool atomic64_add_negative(s64 i, atomic64_t *v) arch_cmpxchg64_local(____ptr, (old), (new)); \ }) +/* + * Originally we had the following code here: + * __typeof__(p1) ____p1 = (p1); + * kasan_check_write(____p1, 2 * sizeof(*____p1)); + * arch_cmpxchg_double(____p1, (p2), (o1), (o2), (n1), (n2)); + * But it leads to compilation failures (see gcc issue 72873). + * So for now it's left non-instrumented. + * There are few callers of cmpxchg_double(), so it's not critical. + */ #define cmpxchg_double(p1, p2, o1, o2, n1, n2) \ ({ \ arch_cmpxchg_double((p1), (p2), (o1), (o2), (n1), (n2)); \ ^ permalink raw reply related [flat|nested] 63+ messages in thread
* [PATCH v6 2/4] x86: switch atomic.h to use atomic-instrumented.h @ 2018-01-29 17:26 Dmitry Vyukov 2018-03-12 12:24 ` [tip:locking/core] locking/atomic/x86: Switch " tip-bot for Dmitry Vyukov 0 siblings, 1 reply; 63+ messages in thread From: Dmitry Vyukov @ 2018-01-29 17:26 UTC (permalink / raw) To: mark.rutland, peterz, mingo, will.deacon, hpa, aryabinin, kasan-dev, x86, linux-kernel, tglx Cc: Dmitry Vyukov, Andrew Morton, linux-mm Add arch_ prefix to all atomic operations and include <asm-generic/atomic-instrumented.h>. This will allow to add KASAN instrumentation to all atomic ops. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Will Deacon <will.deacon@arm.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Ingo Molnar <mingo@redhat.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org Cc: x86@kernel.org --- Changes since v1: - reverted unnecessary change in __raw_try_cmpxchg() - reverted s/try_cmpxchg/arch_try_cmpxchg/ change try_cmpxchg is x86 implementation detail Changes since v3: - drop comment in arch_atomic_set() --- arch/x86/include/asm/atomic.h | 102 ++++++++++++++++++----------------- arch/x86/include/asm/atomic64_32.h | 106 ++++++++++++++++++------------------ arch/x86/include/asm/atomic64_64.h | 108 ++++++++++++++++++------------------- arch/x86/include/asm/cmpxchg.h | 12 ++--- arch/x86/include/asm/cmpxchg_32.h | 8 +-- arch/x86/include/asm/cmpxchg_64.h | 4 +- 6 files changed, 172 insertions(+), 168 deletions(-) diff --git a/arch/x86/include/asm/atomic.h b/arch/x86/include/asm/atomic.h index 72759f131cc5..33afc966d6a9 100644 --- a/arch/x86/include/asm/atomic.h +++ b/arch/x86/include/asm/atomic.h @@ -17,36 +17,36 @@ #define ATOMIC_INIT(i) { (i) } /** - * atomic_read - read atomic variable + * arch_atomic_read - read atomic variable * @v: pointer of type atomic_t * * Atomically reads the value of @v. */ -static __always_inline int atomic_read(const atomic_t *v) +static __always_inline int arch_atomic_read(const atomic_t *v) { return READ_ONCE((v)->counter); } /** - * atomic_set - set atomic variable + * arch_atomic_set - set atomic variable * @v: pointer of type atomic_t * @i: required value * * Atomically sets the value of @v to @i. */ -static __always_inline void atomic_set(atomic_t *v, int i) +static __always_inline void arch_atomic_set(atomic_t *v, int i) { WRITE_ONCE(v->counter, i); } /** - * atomic_add - add integer to atomic variable + * arch_atomic_add - add integer to atomic variable * @i: integer value to add * @v: pointer of type atomic_t * * Atomically adds @i to @v. */ -static __always_inline void atomic_add(int i, atomic_t *v) +static __always_inline void arch_atomic_add(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "addl %1,%0" : "+m" (v->counter) @@ -54,13 +54,13 @@ static __always_inline void atomic_add(int i, atomic_t *v) } /** - * atomic_sub - subtract integer from atomic variable + * arch_atomic_sub - subtract integer from atomic variable * @i: integer value to subtract * @v: pointer of type atomic_t * * Atomically subtracts @i from @v. */ -static __always_inline void atomic_sub(int i, atomic_t *v) +static __always_inline void arch_atomic_sub(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "subl %1,%0" : "+m" (v->counter) @@ -68,7 +68,7 @@ static __always_inline void atomic_sub(int i, atomic_t *v) } /** - * atomic_sub_and_test - subtract value from variable and test result + * arch_atomic_sub_and_test - subtract value from variable and test result * @i: integer value to subtract * @v: pointer of type atomic_t * @@ -76,63 +76,63 @@ static __always_inline void atomic_sub(int i, atomic_t *v) * true if the result is zero, or false for all * other cases. */ -static __always_inline bool atomic_sub_and_test(int i, atomic_t *v) +static __always_inline bool arch_atomic_sub_and_test(int i, atomic_t *v) { GEN_BINARY_RMWcc(LOCK_PREFIX "subl", v->counter, "er", i, "%0", e); } /** - * atomic_inc - increment atomic variable + * arch_atomic_inc - increment atomic variable * @v: pointer of type atomic_t * * Atomically increments @v by 1. */ -static __always_inline void atomic_inc(atomic_t *v) +static __always_inline void arch_atomic_inc(atomic_t *v) { asm volatile(LOCK_PREFIX "incl %0" : "+m" (v->counter)); } /** - * atomic_dec - decrement atomic variable + * arch_atomic_dec - decrement atomic variable * @v: pointer of type atomic_t * * Atomically decrements @v by 1. */ -static __always_inline void atomic_dec(atomic_t *v) +static __always_inline void arch_atomic_dec(atomic_t *v) { asm volatile(LOCK_PREFIX "decl %0" : "+m" (v->counter)); } /** - * atomic_dec_and_test - decrement and test + * arch_atomic_dec_and_test - decrement and test * @v: pointer of type atomic_t * * Atomically decrements @v by 1 and * returns true if the result is 0, or false for all other * cases. */ -static __always_inline bool atomic_dec_and_test(atomic_t *v) +static __always_inline bool arch_atomic_dec_and_test(atomic_t *v) { GEN_UNARY_RMWcc(LOCK_PREFIX "decl", v->counter, "%0", e); } /** - * atomic_inc_and_test - increment and test + * arch_atomic_inc_and_test - increment and test * @v: pointer of type atomic_t * * Atomically increments @v by 1 * and returns true if the result is zero, or false for all * other cases. */ -static __always_inline bool atomic_inc_and_test(atomic_t *v) +static __always_inline bool arch_atomic_inc_and_test(atomic_t *v) { GEN_UNARY_RMWcc(LOCK_PREFIX "incl", v->counter, "%0", e); } /** - * atomic_add_negative - add and test if negative + * arch_atomic_add_negative - add and test if negative * @i: integer value to add * @v: pointer of type atomic_t * @@ -140,65 +140,65 @@ static __always_inline bool atomic_inc_and_test(atomic_t *v) * if the result is negative, or false when * result is greater than or equal to zero. */ -static __always_inline bool atomic_add_negative(int i, atomic_t *v) +static __always_inline bool arch_atomic_add_negative(int i, atomic_t *v) { GEN_BINARY_RMWcc(LOCK_PREFIX "addl", v->counter, "er", i, "%0", s); } /** - * atomic_add_return - add integer and return + * arch_atomic_add_return - add integer and return * @i: integer value to add * @v: pointer of type atomic_t * * Atomically adds @i to @v and returns @i + @v */ -static __always_inline int atomic_add_return(int i, atomic_t *v) +static __always_inline int arch_atomic_add_return(int i, atomic_t *v) { return i + xadd(&v->counter, i); } /** - * atomic_sub_return - subtract integer and return + * arch_atomic_sub_return - subtract integer and return * @v: pointer of type atomic_t * @i: integer value to subtract * * Atomically subtracts @i from @v and returns @v - @i */ -static __always_inline int atomic_sub_return(int i, atomic_t *v) +static __always_inline int arch_atomic_sub_return(int i, atomic_t *v) { - return atomic_add_return(-i, v); + return arch_atomic_add_return(-i, v); } -#define atomic_inc_return(v) (atomic_add_return(1, v)) -#define atomic_dec_return(v) (atomic_sub_return(1, v)) +#define arch_atomic_inc_return(v) (arch_atomic_add_return(1, v)) +#define arch_atomic_dec_return(v) (arch_atomic_sub_return(1, v)) -static __always_inline int atomic_fetch_add(int i, atomic_t *v) +static __always_inline int arch_atomic_fetch_add(int i, atomic_t *v) { return xadd(&v->counter, i); } -static __always_inline int atomic_fetch_sub(int i, atomic_t *v) +static __always_inline int arch_atomic_fetch_sub(int i, atomic_t *v) { return xadd(&v->counter, -i); } -static __always_inline int atomic_cmpxchg(atomic_t *v, int old, int new) +static __always_inline int arch_atomic_cmpxchg(atomic_t *v, int old, int new) { - return cmpxchg(&v->counter, old, new); + return arch_cmpxchg(&v->counter, old, new); } -#define atomic_try_cmpxchg atomic_try_cmpxchg -static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new) +#define arch_atomic_try_cmpxchg arch_atomic_try_cmpxchg +static __always_inline bool arch_atomic_try_cmpxchg(atomic_t *v, int *old, int new) { return try_cmpxchg(&v->counter, old, new); } -static inline int atomic_xchg(atomic_t *v, int new) +static inline int arch_atomic_xchg(atomic_t *v, int new) { return xchg(&v->counter, new); } -static inline void atomic_and(int i, atomic_t *v) +static inline void arch_atomic_and(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "andl %1,%0" : "+m" (v->counter) @@ -206,16 +206,16 @@ static inline void atomic_and(int i, atomic_t *v) : "memory"); } -static inline int atomic_fetch_and(int i, atomic_t *v) +static inline int arch_atomic_fetch_and(int i, atomic_t *v) { - int val = atomic_read(v); + int val = arch_atomic_read(v); - do { } while (!atomic_try_cmpxchg(v, &val, val & i)); + do { } while (!arch_atomic_try_cmpxchg(v, &val, val & i)); return val; } -static inline void atomic_or(int i, atomic_t *v) +static inline void arch_atomic_or(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "orl %1,%0" : "+m" (v->counter) @@ -223,16 +223,16 @@ static inline void atomic_or(int i, atomic_t *v) : "memory"); } -static inline int atomic_fetch_or(int i, atomic_t *v) +static inline int arch_atomic_fetch_or(int i, atomic_t *v) { - int val = atomic_read(v); + int val = arch_atomic_read(v); - do { } while (!atomic_try_cmpxchg(v, &val, val | i)); + do { } while (!arch_atomic_try_cmpxchg(v, &val, val | i)); return val; } -static inline void atomic_xor(int i, atomic_t *v) +static inline void arch_atomic_xor(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "xorl %1,%0" : "+m" (v->counter) @@ -240,17 +240,17 @@ static inline void atomic_xor(int i, atomic_t *v) : "memory"); } -static inline int atomic_fetch_xor(int i, atomic_t *v) +static inline int arch_atomic_fetch_xor(int i, atomic_t *v) { - int val = atomic_read(v); + int val = arch_atomic_read(v); - do { } while (!atomic_try_cmpxchg(v, &val, val ^ i)); + do { } while (!arch_atomic_try_cmpxchg(v, &val, val ^ i)); return val; } /** - * __atomic_add_unless - add unless the number is already a given value + * __arch_atomic_add_unless - add unless the number is already a given value * @v: pointer of type atomic_t * @a: the amount to add to v... * @u: ...unless v is equal to u. @@ -258,14 +258,14 @@ static inline int atomic_fetch_xor(int i, atomic_t *v) * Atomically adds @a to @v, so long as @v was not already @u. * Returns the old value of @v. */ -static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u) +static __always_inline int __arch_atomic_add_unless(atomic_t *v, int a, int u) { - int c = atomic_read(v); + int c = arch_atomic_read(v); do { if (unlikely(c == u)) break; - } while (!atomic_try_cmpxchg(v, &c, c + a)); + } while (!arch_atomic_try_cmpxchg(v, &c, c + a)); return c; } @@ -276,4 +276,6 @@ static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u) # include <asm/atomic64_64.h> #endif +#include <asm-generic/atomic-instrumented.h> + #endif /* _ASM_X86_ATOMIC_H */ diff --git a/arch/x86/include/asm/atomic64_32.h b/arch/x86/include/asm/atomic64_32.h index 97c46b8169b7..46e1ef17d92d 100644 --- a/arch/x86/include/asm/atomic64_32.h +++ b/arch/x86/include/asm/atomic64_32.h @@ -62,7 +62,7 @@ ATOMIC64_DECL(add_unless); #undef ATOMIC64_EXPORT /** - * atomic64_cmpxchg - cmpxchg atomic64 variable + * arch_atomic64_cmpxchg - cmpxchg atomic64 variable * @v: pointer to type atomic64_t * @o: expected value * @n: new value @@ -71,20 +71,21 @@ ATOMIC64_DECL(add_unless); * the old value. */ -static inline long long atomic64_cmpxchg(atomic64_t *v, long long o, long long n) +static inline long long arch_atomic64_cmpxchg(atomic64_t *v, long long o, + long long n) { - return cmpxchg64(&v->counter, o, n); + return arch_cmpxchg64(&v->counter, o, n); } /** - * atomic64_xchg - xchg atomic64 variable + * arch_atomic64_xchg - xchg atomic64 variable * @v: pointer to type atomic64_t * @n: value to assign * * Atomically xchgs the value of @v to @n and returns * the old value. */ -static inline long long atomic64_xchg(atomic64_t *v, long long n) +static inline long long arch_atomic64_xchg(atomic64_t *v, long long n) { long long o; unsigned high = (unsigned)(n >> 32); @@ -96,13 +97,13 @@ static inline long long atomic64_xchg(atomic64_t *v, long long n) } /** - * atomic64_set - set atomic64 variable + * arch_atomic64_set - set atomic64 variable * @v: pointer to type atomic64_t * @i: value to assign * * Atomically sets the value of @v to @n. */ -static inline void atomic64_set(atomic64_t *v, long long i) +static inline void arch_atomic64_set(atomic64_t *v, long long i) { unsigned high = (unsigned)(i >> 32); unsigned low = (unsigned)i; @@ -112,12 +113,12 @@ static inline void atomic64_set(atomic64_t *v, long long i) } /** - * atomic64_read - read atomic64 variable + * arch_atomic64_read - read atomic64 variable * @v: pointer to type atomic64_t * * Atomically reads the value of @v and returns it. */ -static inline long long atomic64_read(const atomic64_t *v) +static inline long long arch_atomic64_read(const atomic64_t *v) { long long r; alternative_atomic64(read, "=&A" (r), "c" (v) : "memory"); @@ -125,13 +126,13 @@ static inline long long atomic64_read(const atomic64_t *v) } /** - * atomic64_add_return - add and return + * arch_atomic64_add_return - add and return * @i: integer value to add * @v: pointer to type atomic64_t * * Atomically adds @i to @v and returns @i + *@v */ -static inline long long atomic64_add_return(long long i, atomic64_t *v) +static inline long long arch_atomic64_add_return(long long i, atomic64_t *v) { alternative_atomic64(add_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -142,7 +143,7 @@ static inline long long atomic64_add_return(long long i, atomic64_t *v) /* * Other variants with different arithmetic operators: */ -static inline long long atomic64_sub_return(long long i, atomic64_t *v) +static inline long long arch_atomic64_sub_return(long long i, atomic64_t *v) { alternative_atomic64(sub_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -150,7 +151,7 @@ static inline long long atomic64_sub_return(long long i, atomic64_t *v) return i; } -static inline long long atomic64_inc_return(atomic64_t *v) +static inline long long arch_atomic64_inc_return(atomic64_t *v) { long long a; alternative_atomic64(inc_return, "=&A" (a), @@ -158,7 +159,7 @@ static inline long long atomic64_inc_return(atomic64_t *v) return a; } -static inline long long atomic64_dec_return(atomic64_t *v) +static inline long long arch_atomic64_dec_return(atomic64_t *v) { long long a; alternative_atomic64(dec_return, "=&A" (a), @@ -167,13 +168,13 @@ static inline long long atomic64_dec_return(atomic64_t *v) } /** - * atomic64_add - add integer to atomic64 variable + * arch_atomic64_add - add integer to atomic64 variable * @i: integer value to add * @v: pointer to type atomic64_t * * Atomically adds @i to @v. */ -static inline long long atomic64_add(long long i, atomic64_t *v) +static inline long long arch_atomic64_add(long long i, atomic64_t *v) { __alternative_atomic64(add, add_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -182,13 +183,13 @@ static inline long long atomic64_add(long long i, atomic64_t *v) } /** - * atomic64_sub - subtract the atomic64 variable + * arch_atomic64_sub - subtract the atomic64 variable * @i: integer value to subtract * @v: pointer to type atomic64_t * * Atomically subtracts @i from @v. */ -static inline long long atomic64_sub(long long i, atomic64_t *v) +static inline long long arch_atomic64_sub(long long i, atomic64_t *v) { __alternative_atomic64(sub, sub_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -197,7 +198,7 @@ static inline long long atomic64_sub(long long i, atomic64_t *v) } /** - * atomic64_sub_and_test - subtract value from variable and test result + * arch_atomic64_sub_and_test - subtract value from variable and test result * @i: integer value to subtract * @v: pointer to type atomic64_t * @@ -205,46 +206,46 @@ static inline long long atomic64_sub(long long i, atomic64_t *v) * true if the result is zero, or false for all * other cases. */ -static inline int atomic64_sub_and_test(long long i, atomic64_t *v) +static inline int arch_atomic64_sub_and_test(long long i, atomic64_t *v) { - return atomic64_sub_return(i, v) == 0; + return arch_atomic64_sub_return(i, v) == 0; } /** - * atomic64_inc - increment atomic64 variable + * arch_atomic64_inc - increment atomic64 variable * @v: pointer to type atomic64_t * * Atomically increments @v by 1. */ -static inline void atomic64_inc(atomic64_t *v) +static inline void arch_atomic64_inc(atomic64_t *v) { __alternative_atomic64(inc, inc_return, /* no output */, "S" (v) : "memory", "eax", "ecx", "edx"); } /** - * atomic64_dec - decrement atomic64 variable + * arch_atomic64_dec - decrement atomic64 variable * @v: pointer to type atomic64_t * * Atomically decrements @v by 1. */ -static inline void atomic64_dec(atomic64_t *v) +static inline void arch_atomic64_dec(atomic64_t *v) { __alternative_atomic64(dec, dec_return, /* no output */, "S" (v) : "memory", "eax", "ecx", "edx"); } /** - * atomic64_dec_and_test - decrement and test + * arch_atomic64_dec_and_test - decrement and test * @v: pointer to type atomic64_t * * Atomically decrements @v by 1 and * returns true if the result is 0, or false for all other * cases. */ -static inline int atomic64_dec_and_test(atomic64_t *v) +static inline int arch_atomic64_dec_and_test(atomic64_t *v) { - return atomic64_dec_return(v) == 0; + return arch_atomic64_dec_return(v) == 0; } /** @@ -255,13 +256,13 @@ static inline int atomic64_dec_and_test(atomic64_t *v) * and returns true if the result is zero, or false for all * other cases. */ -static inline int atomic64_inc_and_test(atomic64_t *v) +static inline int arch_atomic64_inc_and_test(atomic64_t *v) { - return atomic64_inc_return(v) == 0; + return arch_atomic64_inc_return(v) == 0; } /** - * atomic64_add_negative - add and test if negative + * arch_atomic64_add_negative - add and test if negative * @i: integer value to add * @v: pointer to type atomic64_t * @@ -269,13 +270,13 @@ static inline int atomic64_inc_and_test(atomic64_t *v) * if the result is negative, or false when * result is greater than or equal to zero. */ -static inline int atomic64_add_negative(long long i, atomic64_t *v) +static inline int arch_atomic64_add_negative(long long i, atomic64_t *v) { - return atomic64_add_return(i, v) < 0; + return arch_atomic64_add_return(i, v) < 0; } /** - * atomic64_add_unless - add unless the number is a given value + * arch_atomic64_add_unless - add unless the number is a given value * @v: pointer of type atomic64_t * @a: the amount to add to v... * @u: ...unless v is equal to u. @@ -283,7 +284,8 @@ static inline int atomic64_add_negative(long long i, atomic64_t *v) * Atomically adds @a to @v, so long as it was not @u. * Returns non-zero if the add was done, zero otherwise. */ -static inline int atomic64_add_unless(atomic64_t *v, long long a, long long u) +static inline int arch_atomic64_add_unless(atomic64_t *v, long long a, + long long u) { unsigned low = (unsigned)u; unsigned high = (unsigned)(u >> 32); @@ -294,7 +296,7 @@ static inline int atomic64_add_unless(atomic64_t *v, long long a, long long u) } -static inline int atomic64_inc_not_zero(atomic64_t *v) +static inline int arch_atomic64_inc_not_zero(atomic64_t *v) { int r; alternative_atomic64(inc_not_zero, "=&a" (r), @@ -302,7 +304,7 @@ static inline int atomic64_inc_not_zero(atomic64_t *v) return r; } -static inline long long atomic64_dec_if_positive(atomic64_t *v) +static inline long long arch_atomic64_dec_if_positive(atomic64_t *v) { long long r; alternative_atomic64(dec_if_positive, "=&A" (r), @@ -313,70 +315,70 @@ static inline long long atomic64_dec_if_positive(atomic64_t *v) #undef alternative_atomic64 #undef __alternative_atomic64 -static inline void atomic64_and(long long i, atomic64_t *v) +static inline void arch_atomic64_and(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c & i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c & i)) != c) c = old; } -static inline long long atomic64_fetch_and(long long i, atomic64_t *v) +static inline long long arch_atomic64_fetch_and(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c & i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c & i)) != c) c = old; return old; } -static inline void atomic64_or(long long i, atomic64_t *v) +static inline void arch_atomic64_or(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c | i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c | i)) != c) c = old; } -static inline long long atomic64_fetch_or(long long i, atomic64_t *v) +static inline long long arch_atomic64_fetch_or(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c | i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c | i)) != c) c = old; return old; } -static inline void atomic64_xor(long long i, atomic64_t *v) +static inline void arch_atomic64_xor(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c ^ i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c ^ i)) != c) c = old; } -static inline long long atomic64_fetch_xor(long long i, atomic64_t *v) +static inline long long arch_atomic64_fetch_xor(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c ^ i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c ^ i)) != c) c = old; return old; } -static inline long long atomic64_fetch_add(long long i, atomic64_t *v) +static inline long long arch_atomic64_fetch_add(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c + i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c + i)) != c) c = old; return old; } -#define atomic64_fetch_sub(i, v) atomic64_fetch_add(-(i), (v)) +#define arch_atomic64_fetch_sub(i, v) arch_atomic64_fetch_add(-(i), (v)) #endif /* _ASM_X86_ATOMIC64_32_H */ diff --git a/arch/x86/include/asm/atomic64_64.h b/arch/x86/include/asm/atomic64_64.h index 738495caf05f..6106b59d3260 100644 --- a/arch/x86/include/asm/atomic64_64.h +++ b/arch/x86/include/asm/atomic64_64.h @@ -11,37 +11,37 @@ #define ATOMIC64_INIT(i) { (i) } /** - * atomic64_read - read atomic64 variable + * arch_atomic64_read - read atomic64 variable * @v: pointer of type atomic64_t * * Atomically reads the value of @v. * Doesn't imply a read memory barrier. */ -static inline long atomic64_read(const atomic64_t *v) +static inline long arch_atomic64_read(const atomic64_t *v) { return READ_ONCE((v)->counter); } /** - * atomic64_set - set atomic64 variable + * arch_atomic64_set - set atomic64 variable * @v: pointer to type atomic64_t * @i: required value * * Atomically sets the value of @v to @i. */ -static inline void atomic64_set(atomic64_t *v, long i) +static inline void arch_atomic64_set(atomic64_t *v, long i) { WRITE_ONCE(v->counter, i); } /** - * atomic64_add - add integer to atomic64 variable + * arch_atomic64_add - add integer to atomic64 variable * @i: integer value to add * @v: pointer to type atomic64_t * * Atomically adds @i to @v. */ -static __always_inline void atomic64_add(long i, atomic64_t *v) +static __always_inline void arch_atomic64_add(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "addq %1,%0" : "=m" (v->counter) @@ -49,13 +49,13 @@ static __always_inline void atomic64_add(long i, atomic64_t *v) } /** - * atomic64_sub - subtract the atomic64 variable + * arch_atomic64_sub - subtract the atomic64 variable * @i: integer value to subtract * @v: pointer to type atomic64_t * * Atomically subtracts @i from @v. */ -static inline void atomic64_sub(long i, atomic64_t *v) +static inline void arch_atomic64_sub(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "subq %1,%0" : "=m" (v->counter) @@ -63,7 +63,7 @@ static inline void atomic64_sub(long i, atomic64_t *v) } /** - * atomic64_sub_and_test - subtract value from variable and test result + * arch_atomic64_sub_and_test - subtract value from variable and test result * @i: integer value to subtract * @v: pointer to type atomic64_t * @@ -71,18 +71,18 @@ static inline void atomic64_sub(long i, atomic64_t *v) * true if the result is zero, or false for all * other cases. */ -static inline bool atomic64_sub_and_test(long i, atomic64_t *v) +static inline bool arch_atomic64_sub_and_test(long i, atomic64_t *v) { GEN_BINARY_RMWcc(LOCK_PREFIX "subq", v->counter, "er", i, "%0", e); } /** - * atomic64_inc - increment atomic64 variable + * arch_atomic64_inc - increment atomic64 variable * @v: pointer to type atomic64_t * * Atomically increments @v by 1. */ -static __always_inline void atomic64_inc(atomic64_t *v) +static __always_inline void arch_atomic64_inc(atomic64_t *v) { asm volatile(LOCK_PREFIX "incq %0" : "=m" (v->counter) @@ -90,12 +90,12 @@ static __always_inline void atomic64_inc(atomic64_t *v) } /** - * atomic64_dec - decrement atomic64 variable + * arch_atomic64_dec - decrement atomic64 variable * @v: pointer to type atomic64_t * * Atomically decrements @v by 1. */ -static __always_inline void atomic64_dec(atomic64_t *v) +static __always_inline void arch_atomic64_dec(atomic64_t *v) { asm volatile(LOCK_PREFIX "decq %0" : "=m" (v->counter) @@ -103,33 +103,33 @@ static __always_inline void atomic64_dec(atomic64_t *v) } /** - * atomic64_dec_and_test - decrement and test + * arch_atomic64_dec_and_test - decrement and test * @v: pointer to type atomic64_t * * Atomically decrements @v by 1 and * returns true if the result is 0, or false for all other * cases. */ -static inline bool atomic64_dec_and_test(atomic64_t *v) +static inline bool arch_atomic64_dec_and_test(atomic64_t *v) { GEN_UNARY_RMWcc(LOCK_PREFIX "decq", v->counter, "%0", e); } /** - * atomic64_inc_and_test - increment and test + * arch_atomic64_inc_and_test - increment and test * @v: pointer to type atomic64_t * * Atomically increments @v by 1 * and returns true if the result is zero, or false for all * other cases. */ -static inline bool atomic64_inc_and_test(atomic64_t *v) +static inline bool arch_atomic64_inc_and_test(atomic64_t *v) { GEN_UNARY_RMWcc(LOCK_PREFIX "incq", v->counter, "%0", e); } /** - * atomic64_add_negative - add and test if negative + * arch_atomic64_add_negative - add and test if negative * @i: integer value to add * @v: pointer to type atomic64_t * @@ -137,59 +137,59 @@ static inline bool atomic64_inc_and_test(atomic64_t *v) * if the result is negative, or false when * result is greater than or equal to zero. */ -static inline bool atomic64_add_negative(long i, atomic64_t *v) +static inline bool arch_atomic64_add_negative(long i, atomic64_t *v) { GEN_BINARY_RMWcc(LOCK_PREFIX "addq", v->counter, "er", i, "%0", s); } /** - * atomic64_add_return - add and return + * arch_atomic64_add_return - add and return * @i: integer value to add * @v: pointer to type atomic64_t * * Atomically adds @i to @v and returns @i + @v */ -static __always_inline long atomic64_add_return(long i, atomic64_t *v) +static __always_inline long arch_atomic64_add_return(long i, atomic64_t *v) { return i + xadd(&v->counter, i); } -static inline long atomic64_sub_return(long i, atomic64_t *v) +static inline long arch_atomic64_sub_return(long i, atomic64_t *v) { - return atomic64_add_return(-i, v); + return arch_atomic64_add_return(-i, v); } -static inline long atomic64_fetch_add(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_add(long i, atomic64_t *v) { return xadd(&v->counter, i); } -static inline long atomic64_fetch_sub(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_sub(long i, atomic64_t *v) { return xadd(&v->counter, -i); } -#define atomic64_inc_return(v) (atomic64_add_return(1, (v))) -#define atomic64_dec_return(v) (atomic64_sub_return(1, (v))) +#define arch_atomic64_inc_return(v) (arch_atomic64_add_return(1, (v))) +#define arch_atomic64_dec_return(v) (arch_atomic64_sub_return(1, (v))) -static inline long atomic64_cmpxchg(atomic64_t *v, long old, long new) +static inline long arch_atomic64_cmpxchg(atomic64_t *v, long old, long new) { - return cmpxchg(&v->counter, old, new); + return arch_cmpxchg(&v->counter, old, new); } -#define atomic64_try_cmpxchg atomic64_try_cmpxchg -static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, long new) +#define arch_atomic64_try_cmpxchg arch_atomic64_try_cmpxchg +static __always_inline bool arch_atomic64_try_cmpxchg(atomic64_t *v, s64 *old, long new) { return try_cmpxchg(&v->counter, old, new); } -static inline long atomic64_xchg(atomic64_t *v, long new) +static inline long arch_atomic64_xchg(atomic64_t *v, long new) { return xchg(&v->counter, new); } /** - * atomic64_add_unless - add unless the number is a given value + * arch_atomic64_add_unless - add unless the number is a given value * @v: pointer of type atomic64_t * @a: the amount to add to v... * @u: ...unless v is equal to u. @@ -197,37 +197,37 @@ static inline long atomic64_xchg(atomic64_t *v, long new) * Atomically adds @a to @v, so long as it was not @u. * Returns the old value of @v. */ -static inline bool atomic64_add_unless(atomic64_t *v, long a, long u) +static inline bool arch_atomic64_add_unless(atomic64_t *v, long a, long u) { - s64 c = atomic64_read(v); + s64 c = arch_atomic64_read(v); do { if (unlikely(c == u)) return false; - } while (!atomic64_try_cmpxchg(v, &c, c + a)); + } while (!arch_atomic64_try_cmpxchg(v, &c, c + a)); return true; } -#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0) +#define arch_atomic64_inc_not_zero(v) arch_atomic64_add_unless((v), 1, 0) /* - * atomic64_dec_if_positive - decrement by 1 if old value positive + * arch_atomic64_dec_if_positive - decrement by 1 if old value positive * @v: pointer of type atomic_t * * The function returns the old value of *v minus 1, even if * the atomic variable, v, was not decremented. */ -static inline long atomic64_dec_if_positive(atomic64_t *v) +static inline long arch_atomic64_dec_if_positive(atomic64_t *v) { - s64 dec, c = atomic64_read(v); + s64 dec, c = arch_atomic64_read(v); do { dec = c - 1; if (unlikely(dec < 0)) break; - } while (!atomic64_try_cmpxchg(v, &c, dec)); + } while (!arch_atomic64_try_cmpxchg(v, &c, dec)); return dec; } -static inline void atomic64_and(long i, atomic64_t *v) +static inline void arch_atomic64_and(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "andq %1,%0" : "+m" (v->counter) @@ -235,16 +235,16 @@ static inline void atomic64_and(long i, atomic64_t *v) : "memory"); } -static inline long atomic64_fetch_and(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_and(long i, atomic64_t *v) { - s64 val = atomic64_read(v); + s64 val = arch_atomic64_read(v); do { - } while (!atomic64_try_cmpxchg(v, &val, val & i)); + } while (!arch_atomic64_try_cmpxchg(v, &val, val & i)); return val; } -static inline void atomic64_or(long i, atomic64_t *v) +static inline void arch_atomic64_or(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "orq %1,%0" : "+m" (v->counter) @@ -252,16 +252,16 @@ static inline void atomic64_or(long i, atomic64_t *v) : "memory"); } -static inline long atomic64_fetch_or(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_or(long i, atomic64_t *v) { - s64 val = atomic64_read(v); + s64 val = arch_atomic64_read(v); do { - } while (!atomic64_try_cmpxchg(v, &val, val | i)); + } while (!arch_atomic64_try_cmpxchg(v, &val, val | i)); return val; } -static inline void atomic64_xor(long i, atomic64_t *v) +static inline void arch_atomic64_xor(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "xorq %1,%0" : "+m" (v->counter) @@ -269,12 +269,12 @@ static inline void atomic64_xor(long i, atomic64_t *v) : "memory"); } -static inline long atomic64_fetch_xor(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_xor(long i, atomic64_t *v) { - s64 val = atomic64_read(v); + s64 val = arch_atomic64_read(v); do { - } while (!atomic64_try_cmpxchg(v, &val, val ^ i)); + } while (!arch_atomic64_try_cmpxchg(v, &val, val ^ i)); return val; } diff --git a/arch/x86/include/asm/cmpxchg.h b/arch/x86/include/asm/cmpxchg.h index 56bd436ed01b..e3efd8a06066 100644 --- a/arch/x86/include/asm/cmpxchg.h +++ b/arch/x86/include/asm/cmpxchg.h @@ -145,13 +145,13 @@ extern void __add_wrong_size(void) # include <asm/cmpxchg_64.h> #endif -#define cmpxchg(ptr, old, new) \ +#define arch_cmpxchg(ptr, old, new) \ __cmpxchg(ptr, old, new, sizeof(*(ptr))) -#define sync_cmpxchg(ptr, old, new) \ +#define arch_sync_cmpxchg(ptr, old, new) \ __sync_cmpxchg(ptr, old, new, sizeof(*(ptr))) -#define cmpxchg_local(ptr, old, new) \ +#define arch_cmpxchg_local(ptr, old, new) \ __cmpxchg_local(ptr, old, new, sizeof(*(ptr))) @@ -221,7 +221,7 @@ extern void __add_wrong_size(void) #define __try_cmpxchg(ptr, pold, new, size) \ __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX) -#define try_cmpxchg(ptr, pold, new) \ +#define try_cmpxchg(ptr, pold, new) \ __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr))) /* @@ -250,10 +250,10 @@ extern void __add_wrong_size(void) __ret; \ }) -#define cmpxchg_double(p1, p2, o1, o2, n1, n2) \ +#define arch_cmpxchg_double(p1, p2, o1, o2, n1, n2) \ __cmpxchg_double(LOCK_PREFIX, p1, p2, o1, o2, n1, n2) -#define cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \ +#define arch_cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \ __cmpxchg_double(, p1, p2, o1, o2, n1, n2) #endif /* ASM_X86_CMPXCHG_H */ diff --git a/arch/x86/include/asm/cmpxchg_32.h b/arch/x86/include/asm/cmpxchg_32.h index 1732704f0445..1a2eafca7038 100644 --- a/arch/x86/include/asm/cmpxchg_32.h +++ b/arch/x86/include/asm/cmpxchg_32.h @@ -36,10 +36,10 @@ static inline void set_64bit(volatile u64 *ptr, u64 value) } #ifdef CONFIG_X86_CMPXCHG64 -#define cmpxchg64(ptr, o, n) \ +#define arch_cmpxchg64(ptr, o, n) \ ((__typeof__(*(ptr)))__cmpxchg64((ptr), (unsigned long long)(o), \ (unsigned long long)(n))) -#define cmpxchg64_local(ptr, o, n) \ +#define arch_cmpxchg64_local(ptr, o, n) \ ((__typeof__(*(ptr)))__cmpxchg64_local((ptr), (unsigned long long)(o), \ (unsigned long long)(n))) #endif @@ -76,7 +76,7 @@ static inline u64 __cmpxchg64_local(volatile u64 *ptr, u64 old, u64 new) * to simulate the cmpxchg8b on the 80386 and 80486 CPU. */ -#define cmpxchg64(ptr, o, n) \ +#define arch_cmpxchg64(ptr, o, n) \ ({ \ __typeof__(*(ptr)) __ret; \ __typeof__(*(ptr)) __old = (o); \ @@ -93,7 +93,7 @@ static inline u64 __cmpxchg64_local(volatile u64 *ptr, u64 old, u64 new) __ret; }) -#define cmpxchg64_local(ptr, o, n) \ +#define arch_cmpxchg64_local(ptr, o, n) \ ({ \ __typeof__(*(ptr)) __ret; \ __typeof__(*(ptr)) __old = (o); \ diff --git a/arch/x86/include/asm/cmpxchg_64.h b/arch/x86/include/asm/cmpxchg_64.h index 03cad196a301..bfca3b346c74 100644 --- a/arch/x86/include/asm/cmpxchg_64.h +++ b/arch/x86/include/asm/cmpxchg_64.h @@ -7,13 +7,13 @@ static inline void set_64bit(volatile u64 *ptr, u64 val) *ptr = val; } -#define cmpxchg64(ptr, o, n) \ +#define arch_cmpxchg64(ptr, o, n) \ ({ \ BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ cmpxchg((ptr), (o), (n)); \ }) -#define cmpxchg64_local(ptr, o, n) \ +#define arch_cmpxchg64_local(ptr, o, n) \ ({ \ BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ cmpxchg_local((ptr), (o), (n)); \ -- 2.16.0.rc1.238.g530d649a79-goog ^ permalink raw reply related [flat|nested] 63+ messages in thread
* [tip:locking/core] locking/atomic/x86: Switch atomic.h to use atomic-instrumented.h 2018-01-29 17:26 [PATCH v6 2/4] x86: switch atomic.h to use atomic-instrumented.h Dmitry Vyukov @ 2018-03-12 12:24 ` tip-bot for Dmitry Vyukov 0 siblings, 0 replies; 63+ messages in thread From: tip-bot for Dmitry Vyukov @ 2018-03-12 12:24 UTC (permalink / raw) To: linux-tip-commits Cc: tglx, linux-kernel, will.deacon, mark.rutland, akpm, torvalds, mingo, dvyukov, hpa, peterz, aryabinin Commit-ID: 8bf705d130396e69c04cd8e6e010244ad2ce71f4 Gitweb: https://git.kernel.org/tip/8bf705d130396e69c04cd8e6e010244ad2ce71f4 Author: Dmitry Vyukov <dvyukov@google.com> AuthorDate: Mon, 29 Jan 2018 18:26:05 +0100 Committer: Ingo Molnar <mingo@kernel.org> CommitDate: Mon, 12 Mar 2018 12:15:35 +0100 locking/atomic/x86: Switch atomic.h to use atomic-instrumented.h Add arch_ prefix to all atomic operations and include <asm-generic/atomic-instrumented.h>. This will allow to add KASAN instrumentation to all atomic ops. Signed-off-by: Dmitry Vyukov <dvyukov@google.com> Cc: Andrew Morton <akpm@linux-foundation.org> Cc: Andrey Ryabinin <aryabinin@virtuozzo.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Mark Rutland <mark.rutland@arm.com> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Will Deacon <will.deacon@arm.com> Cc: kasan-dev@googlegroups.com Cc: linux-mm@kvack.org Link: http://lkml.kernel.org/r/54f0eb64260b84199e538652e079a89b5423ad41.1517246437.git.dvyukov@google.com Signed-off-by: Ingo Molnar <mingo@kernel.org> --- arch/x86/include/asm/atomic.h | 102 ++++++++++++++++++----------------- arch/x86/include/asm/atomic64_32.h | 106 ++++++++++++++++++------------------ arch/x86/include/asm/atomic64_64.h | 108 ++++++++++++++++++------------------- arch/x86/include/asm/cmpxchg.h | 12 ++--- arch/x86/include/asm/cmpxchg_32.h | 8 +-- arch/x86/include/asm/cmpxchg_64.h | 4 +- 6 files changed, 172 insertions(+), 168 deletions(-) diff --git a/arch/x86/include/asm/atomic.h b/arch/x86/include/asm/atomic.h index 72759f131cc5..33afc966d6a9 100644 --- a/arch/x86/include/asm/atomic.h +++ b/arch/x86/include/asm/atomic.h @@ -17,36 +17,36 @@ #define ATOMIC_INIT(i) { (i) } /** - * atomic_read - read atomic variable + * arch_atomic_read - read atomic variable * @v: pointer of type atomic_t * * Atomically reads the value of @v. */ -static __always_inline int atomic_read(const atomic_t *v) +static __always_inline int arch_atomic_read(const atomic_t *v) { return READ_ONCE((v)->counter); } /** - * atomic_set - set atomic variable + * arch_atomic_set - set atomic variable * @v: pointer of type atomic_t * @i: required value * * Atomically sets the value of @v to @i. */ -static __always_inline void atomic_set(atomic_t *v, int i) +static __always_inline void arch_atomic_set(atomic_t *v, int i) { WRITE_ONCE(v->counter, i); } /** - * atomic_add - add integer to atomic variable + * arch_atomic_add - add integer to atomic variable * @i: integer value to add * @v: pointer of type atomic_t * * Atomically adds @i to @v. */ -static __always_inline void atomic_add(int i, atomic_t *v) +static __always_inline void arch_atomic_add(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "addl %1,%0" : "+m" (v->counter) @@ -54,13 +54,13 @@ static __always_inline void atomic_add(int i, atomic_t *v) } /** - * atomic_sub - subtract integer from atomic variable + * arch_atomic_sub - subtract integer from atomic variable * @i: integer value to subtract * @v: pointer of type atomic_t * * Atomically subtracts @i from @v. */ -static __always_inline void atomic_sub(int i, atomic_t *v) +static __always_inline void arch_atomic_sub(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "subl %1,%0" : "+m" (v->counter) @@ -68,7 +68,7 @@ static __always_inline void atomic_sub(int i, atomic_t *v) } /** - * atomic_sub_and_test - subtract value from variable and test result + * arch_atomic_sub_and_test - subtract value from variable and test result * @i: integer value to subtract * @v: pointer of type atomic_t * @@ -76,63 +76,63 @@ static __always_inline void atomic_sub(int i, atomic_t *v) * true if the result is zero, or false for all * other cases. */ -static __always_inline bool atomic_sub_and_test(int i, atomic_t *v) +static __always_inline bool arch_atomic_sub_and_test(int i, atomic_t *v) { GEN_BINARY_RMWcc(LOCK_PREFIX "subl", v->counter, "er", i, "%0", e); } /** - * atomic_inc - increment atomic variable + * arch_atomic_inc - increment atomic variable * @v: pointer of type atomic_t * * Atomically increments @v by 1. */ -static __always_inline void atomic_inc(atomic_t *v) +static __always_inline void arch_atomic_inc(atomic_t *v) { asm volatile(LOCK_PREFIX "incl %0" : "+m" (v->counter)); } /** - * atomic_dec - decrement atomic variable + * arch_atomic_dec - decrement atomic variable * @v: pointer of type atomic_t * * Atomically decrements @v by 1. */ -static __always_inline void atomic_dec(atomic_t *v) +static __always_inline void arch_atomic_dec(atomic_t *v) { asm volatile(LOCK_PREFIX "decl %0" : "+m" (v->counter)); } /** - * atomic_dec_and_test - decrement and test + * arch_atomic_dec_and_test - decrement and test * @v: pointer of type atomic_t * * Atomically decrements @v by 1 and * returns true if the result is 0, or false for all other * cases. */ -static __always_inline bool atomic_dec_and_test(atomic_t *v) +static __always_inline bool arch_atomic_dec_and_test(atomic_t *v) { GEN_UNARY_RMWcc(LOCK_PREFIX "decl", v->counter, "%0", e); } /** - * atomic_inc_and_test - increment and test + * arch_atomic_inc_and_test - increment and test * @v: pointer of type atomic_t * * Atomically increments @v by 1 * and returns true if the result is zero, or false for all * other cases. */ -static __always_inline bool atomic_inc_and_test(atomic_t *v) +static __always_inline bool arch_atomic_inc_and_test(atomic_t *v) { GEN_UNARY_RMWcc(LOCK_PREFIX "incl", v->counter, "%0", e); } /** - * atomic_add_negative - add and test if negative + * arch_atomic_add_negative - add and test if negative * @i: integer value to add * @v: pointer of type atomic_t * @@ -140,65 +140,65 @@ static __always_inline bool atomic_inc_and_test(atomic_t *v) * if the result is negative, or false when * result is greater than or equal to zero. */ -static __always_inline bool atomic_add_negative(int i, atomic_t *v) +static __always_inline bool arch_atomic_add_negative(int i, atomic_t *v) { GEN_BINARY_RMWcc(LOCK_PREFIX "addl", v->counter, "er", i, "%0", s); } /** - * atomic_add_return - add integer and return + * arch_atomic_add_return - add integer and return * @i: integer value to add * @v: pointer of type atomic_t * * Atomically adds @i to @v and returns @i + @v */ -static __always_inline int atomic_add_return(int i, atomic_t *v) +static __always_inline int arch_atomic_add_return(int i, atomic_t *v) { return i + xadd(&v->counter, i); } /** - * atomic_sub_return - subtract integer and return + * arch_atomic_sub_return - subtract integer and return * @v: pointer of type atomic_t * @i: integer value to subtract * * Atomically subtracts @i from @v and returns @v - @i */ -static __always_inline int atomic_sub_return(int i, atomic_t *v) +static __always_inline int arch_atomic_sub_return(int i, atomic_t *v) { - return atomic_add_return(-i, v); + return arch_atomic_add_return(-i, v); } -#define atomic_inc_return(v) (atomic_add_return(1, v)) -#define atomic_dec_return(v) (atomic_sub_return(1, v)) +#define arch_atomic_inc_return(v) (arch_atomic_add_return(1, v)) +#define arch_atomic_dec_return(v) (arch_atomic_sub_return(1, v)) -static __always_inline int atomic_fetch_add(int i, atomic_t *v) +static __always_inline int arch_atomic_fetch_add(int i, atomic_t *v) { return xadd(&v->counter, i); } -static __always_inline int atomic_fetch_sub(int i, atomic_t *v) +static __always_inline int arch_atomic_fetch_sub(int i, atomic_t *v) { return xadd(&v->counter, -i); } -static __always_inline int atomic_cmpxchg(atomic_t *v, int old, int new) +static __always_inline int arch_atomic_cmpxchg(atomic_t *v, int old, int new) { - return cmpxchg(&v->counter, old, new); + return arch_cmpxchg(&v->counter, old, new); } -#define atomic_try_cmpxchg atomic_try_cmpxchg -static __always_inline bool atomic_try_cmpxchg(atomic_t *v, int *old, int new) +#define arch_atomic_try_cmpxchg arch_atomic_try_cmpxchg +static __always_inline bool arch_atomic_try_cmpxchg(atomic_t *v, int *old, int new) { return try_cmpxchg(&v->counter, old, new); } -static inline int atomic_xchg(atomic_t *v, int new) +static inline int arch_atomic_xchg(atomic_t *v, int new) { return xchg(&v->counter, new); } -static inline void atomic_and(int i, atomic_t *v) +static inline void arch_atomic_and(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "andl %1,%0" : "+m" (v->counter) @@ -206,16 +206,16 @@ static inline void atomic_and(int i, atomic_t *v) : "memory"); } -static inline int atomic_fetch_and(int i, atomic_t *v) +static inline int arch_atomic_fetch_and(int i, atomic_t *v) { - int val = atomic_read(v); + int val = arch_atomic_read(v); - do { } while (!atomic_try_cmpxchg(v, &val, val & i)); + do { } while (!arch_atomic_try_cmpxchg(v, &val, val & i)); return val; } -static inline void atomic_or(int i, atomic_t *v) +static inline void arch_atomic_or(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "orl %1,%0" : "+m" (v->counter) @@ -223,16 +223,16 @@ static inline void atomic_or(int i, atomic_t *v) : "memory"); } -static inline int atomic_fetch_or(int i, atomic_t *v) +static inline int arch_atomic_fetch_or(int i, atomic_t *v) { - int val = atomic_read(v); + int val = arch_atomic_read(v); - do { } while (!atomic_try_cmpxchg(v, &val, val | i)); + do { } while (!arch_atomic_try_cmpxchg(v, &val, val | i)); return val; } -static inline void atomic_xor(int i, atomic_t *v) +static inline void arch_atomic_xor(int i, atomic_t *v) { asm volatile(LOCK_PREFIX "xorl %1,%0" : "+m" (v->counter) @@ -240,17 +240,17 @@ static inline void atomic_xor(int i, atomic_t *v) : "memory"); } -static inline int atomic_fetch_xor(int i, atomic_t *v) +static inline int arch_atomic_fetch_xor(int i, atomic_t *v) { - int val = atomic_read(v); + int val = arch_atomic_read(v); - do { } while (!atomic_try_cmpxchg(v, &val, val ^ i)); + do { } while (!arch_atomic_try_cmpxchg(v, &val, val ^ i)); return val; } /** - * __atomic_add_unless - add unless the number is already a given value + * __arch_atomic_add_unless - add unless the number is already a given value * @v: pointer of type atomic_t * @a: the amount to add to v... * @u: ...unless v is equal to u. @@ -258,14 +258,14 @@ static inline int atomic_fetch_xor(int i, atomic_t *v) * Atomically adds @a to @v, so long as @v was not already @u. * Returns the old value of @v. */ -static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u) +static __always_inline int __arch_atomic_add_unless(atomic_t *v, int a, int u) { - int c = atomic_read(v); + int c = arch_atomic_read(v); do { if (unlikely(c == u)) break; - } while (!atomic_try_cmpxchg(v, &c, c + a)); + } while (!arch_atomic_try_cmpxchg(v, &c, c + a)); return c; } @@ -276,4 +276,6 @@ static __always_inline int __atomic_add_unless(atomic_t *v, int a, int u) # include <asm/atomic64_64.h> #endif +#include <asm-generic/atomic-instrumented.h> + #endif /* _ASM_X86_ATOMIC_H */ diff --git a/arch/x86/include/asm/atomic64_32.h b/arch/x86/include/asm/atomic64_32.h index 97c46b8169b7..46e1ef17d92d 100644 --- a/arch/x86/include/asm/atomic64_32.h +++ b/arch/x86/include/asm/atomic64_32.h @@ -62,7 +62,7 @@ ATOMIC64_DECL(add_unless); #undef ATOMIC64_EXPORT /** - * atomic64_cmpxchg - cmpxchg atomic64 variable + * arch_atomic64_cmpxchg - cmpxchg atomic64 variable * @v: pointer to type atomic64_t * @o: expected value * @n: new value @@ -71,20 +71,21 @@ ATOMIC64_DECL(add_unless); * the old value. */ -static inline long long atomic64_cmpxchg(atomic64_t *v, long long o, long long n) +static inline long long arch_atomic64_cmpxchg(atomic64_t *v, long long o, + long long n) { - return cmpxchg64(&v->counter, o, n); + return arch_cmpxchg64(&v->counter, o, n); } /** - * atomic64_xchg - xchg atomic64 variable + * arch_atomic64_xchg - xchg atomic64 variable * @v: pointer to type atomic64_t * @n: value to assign * * Atomically xchgs the value of @v to @n and returns * the old value. */ -static inline long long atomic64_xchg(atomic64_t *v, long long n) +static inline long long arch_atomic64_xchg(atomic64_t *v, long long n) { long long o; unsigned high = (unsigned)(n >> 32); @@ -96,13 +97,13 @@ static inline long long atomic64_xchg(atomic64_t *v, long long n) } /** - * atomic64_set - set atomic64 variable + * arch_atomic64_set - set atomic64 variable * @v: pointer to type atomic64_t * @i: value to assign * * Atomically sets the value of @v to @n. */ -static inline void atomic64_set(atomic64_t *v, long long i) +static inline void arch_atomic64_set(atomic64_t *v, long long i) { unsigned high = (unsigned)(i >> 32); unsigned low = (unsigned)i; @@ -112,12 +113,12 @@ static inline void atomic64_set(atomic64_t *v, long long i) } /** - * atomic64_read - read atomic64 variable + * arch_atomic64_read - read atomic64 variable * @v: pointer to type atomic64_t * * Atomically reads the value of @v and returns it. */ -static inline long long atomic64_read(const atomic64_t *v) +static inline long long arch_atomic64_read(const atomic64_t *v) { long long r; alternative_atomic64(read, "=&A" (r), "c" (v) : "memory"); @@ -125,13 +126,13 @@ static inline long long atomic64_read(const atomic64_t *v) } /** - * atomic64_add_return - add and return + * arch_atomic64_add_return - add and return * @i: integer value to add * @v: pointer to type atomic64_t * * Atomically adds @i to @v and returns @i + *@v */ -static inline long long atomic64_add_return(long long i, atomic64_t *v) +static inline long long arch_atomic64_add_return(long long i, atomic64_t *v) { alternative_atomic64(add_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -142,7 +143,7 @@ static inline long long atomic64_add_return(long long i, atomic64_t *v) /* * Other variants with different arithmetic operators: */ -static inline long long atomic64_sub_return(long long i, atomic64_t *v) +static inline long long arch_atomic64_sub_return(long long i, atomic64_t *v) { alternative_atomic64(sub_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -150,7 +151,7 @@ static inline long long atomic64_sub_return(long long i, atomic64_t *v) return i; } -static inline long long atomic64_inc_return(atomic64_t *v) +static inline long long arch_atomic64_inc_return(atomic64_t *v) { long long a; alternative_atomic64(inc_return, "=&A" (a), @@ -158,7 +159,7 @@ static inline long long atomic64_inc_return(atomic64_t *v) return a; } -static inline long long atomic64_dec_return(atomic64_t *v) +static inline long long arch_atomic64_dec_return(atomic64_t *v) { long long a; alternative_atomic64(dec_return, "=&A" (a), @@ -167,13 +168,13 @@ static inline long long atomic64_dec_return(atomic64_t *v) } /** - * atomic64_add - add integer to atomic64 variable + * arch_atomic64_add - add integer to atomic64 variable * @i: integer value to add * @v: pointer to type atomic64_t * * Atomically adds @i to @v. */ -static inline long long atomic64_add(long long i, atomic64_t *v) +static inline long long arch_atomic64_add(long long i, atomic64_t *v) { __alternative_atomic64(add, add_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -182,13 +183,13 @@ static inline long long atomic64_add(long long i, atomic64_t *v) } /** - * atomic64_sub - subtract the atomic64 variable + * arch_atomic64_sub - subtract the atomic64 variable * @i: integer value to subtract * @v: pointer to type atomic64_t * * Atomically subtracts @i from @v. */ -static inline long long atomic64_sub(long long i, atomic64_t *v) +static inline long long arch_atomic64_sub(long long i, atomic64_t *v) { __alternative_atomic64(sub, sub_return, ASM_OUTPUT2("+A" (i), "+c" (v)), @@ -197,7 +198,7 @@ static inline long long atomic64_sub(long long i, atomic64_t *v) } /** - * atomic64_sub_and_test - subtract value from variable and test result + * arch_atomic64_sub_and_test - subtract value from variable and test result * @i: integer value to subtract * @v: pointer to type atomic64_t * @@ -205,46 +206,46 @@ static inline long long atomic64_sub(long long i, atomic64_t *v) * true if the result is zero, or false for all * other cases. */ -static inline int atomic64_sub_and_test(long long i, atomic64_t *v) +static inline int arch_atomic64_sub_and_test(long long i, atomic64_t *v) { - return atomic64_sub_return(i, v) == 0; + return arch_atomic64_sub_return(i, v) == 0; } /** - * atomic64_inc - increment atomic64 variable + * arch_atomic64_inc - increment atomic64 variable * @v: pointer to type atomic64_t * * Atomically increments @v by 1. */ -static inline void atomic64_inc(atomic64_t *v) +static inline void arch_atomic64_inc(atomic64_t *v) { __alternative_atomic64(inc, inc_return, /* no output */, "S" (v) : "memory", "eax", "ecx", "edx"); } /** - * atomic64_dec - decrement atomic64 variable + * arch_atomic64_dec - decrement atomic64 variable * @v: pointer to type atomic64_t * * Atomically decrements @v by 1. */ -static inline void atomic64_dec(atomic64_t *v) +static inline void arch_atomic64_dec(atomic64_t *v) { __alternative_atomic64(dec, dec_return, /* no output */, "S" (v) : "memory", "eax", "ecx", "edx"); } /** - * atomic64_dec_and_test - decrement and test + * arch_atomic64_dec_and_test - decrement and test * @v: pointer to type atomic64_t * * Atomically decrements @v by 1 and * returns true if the result is 0, or false for all other * cases. */ -static inline int atomic64_dec_and_test(atomic64_t *v) +static inline int arch_atomic64_dec_and_test(atomic64_t *v) { - return atomic64_dec_return(v) == 0; + return arch_atomic64_dec_return(v) == 0; } /** @@ -255,13 +256,13 @@ static inline int atomic64_dec_and_test(atomic64_t *v) * and returns true if the result is zero, or false for all * other cases. */ -static inline int atomic64_inc_and_test(atomic64_t *v) +static inline int arch_atomic64_inc_and_test(atomic64_t *v) { - return atomic64_inc_return(v) == 0; + return arch_atomic64_inc_return(v) == 0; } /** - * atomic64_add_negative - add and test if negative + * arch_atomic64_add_negative - add and test if negative * @i: integer value to add * @v: pointer to type atomic64_t * @@ -269,13 +270,13 @@ static inline int atomic64_inc_and_test(atomic64_t *v) * if the result is negative, or false when * result is greater than or equal to zero. */ -static inline int atomic64_add_negative(long long i, atomic64_t *v) +static inline int arch_atomic64_add_negative(long long i, atomic64_t *v) { - return atomic64_add_return(i, v) < 0; + return arch_atomic64_add_return(i, v) < 0; } /** - * atomic64_add_unless - add unless the number is a given value + * arch_atomic64_add_unless - add unless the number is a given value * @v: pointer of type atomic64_t * @a: the amount to add to v... * @u: ...unless v is equal to u. @@ -283,7 +284,8 @@ static inline int atomic64_add_negative(long long i, atomic64_t *v) * Atomically adds @a to @v, so long as it was not @u. * Returns non-zero if the add was done, zero otherwise. */ -static inline int atomic64_add_unless(atomic64_t *v, long long a, long long u) +static inline int arch_atomic64_add_unless(atomic64_t *v, long long a, + long long u) { unsigned low = (unsigned)u; unsigned high = (unsigned)(u >> 32); @@ -294,7 +296,7 @@ static inline int atomic64_add_unless(atomic64_t *v, long long a, long long u) } -static inline int atomic64_inc_not_zero(atomic64_t *v) +static inline int arch_atomic64_inc_not_zero(atomic64_t *v) { int r; alternative_atomic64(inc_not_zero, "=&a" (r), @@ -302,7 +304,7 @@ static inline int atomic64_inc_not_zero(atomic64_t *v) return r; } -static inline long long atomic64_dec_if_positive(atomic64_t *v) +static inline long long arch_atomic64_dec_if_positive(atomic64_t *v) { long long r; alternative_atomic64(dec_if_positive, "=&A" (r), @@ -313,70 +315,70 @@ static inline long long atomic64_dec_if_positive(atomic64_t *v) #undef alternative_atomic64 #undef __alternative_atomic64 -static inline void atomic64_and(long long i, atomic64_t *v) +static inline void arch_atomic64_and(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c & i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c & i)) != c) c = old; } -static inline long long atomic64_fetch_and(long long i, atomic64_t *v) +static inline long long arch_atomic64_fetch_and(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c & i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c & i)) != c) c = old; return old; } -static inline void atomic64_or(long long i, atomic64_t *v) +static inline void arch_atomic64_or(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c | i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c | i)) != c) c = old; } -static inline long long atomic64_fetch_or(long long i, atomic64_t *v) +static inline long long arch_atomic64_fetch_or(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c | i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c | i)) != c) c = old; return old; } -static inline void atomic64_xor(long long i, atomic64_t *v) +static inline void arch_atomic64_xor(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c ^ i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c ^ i)) != c) c = old; } -static inline long long atomic64_fetch_xor(long long i, atomic64_t *v) +static inline long long arch_atomic64_fetch_xor(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c ^ i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c ^ i)) != c) c = old; return old; } -static inline long long atomic64_fetch_add(long long i, atomic64_t *v) +static inline long long arch_atomic64_fetch_add(long long i, atomic64_t *v) { long long old, c = 0; - while ((old = atomic64_cmpxchg(v, c, c + i)) != c) + while ((old = arch_atomic64_cmpxchg(v, c, c + i)) != c) c = old; return old; } -#define atomic64_fetch_sub(i, v) atomic64_fetch_add(-(i), (v)) +#define arch_atomic64_fetch_sub(i, v) arch_atomic64_fetch_add(-(i), (v)) #endif /* _ASM_X86_ATOMIC64_32_H */ diff --git a/arch/x86/include/asm/atomic64_64.h b/arch/x86/include/asm/atomic64_64.h index 738495caf05f..6106b59d3260 100644 --- a/arch/x86/include/asm/atomic64_64.h +++ b/arch/x86/include/asm/atomic64_64.h @@ -11,37 +11,37 @@ #define ATOMIC64_INIT(i) { (i) } /** - * atomic64_read - read atomic64 variable + * arch_atomic64_read - read atomic64 variable * @v: pointer of type atomic64_t * * Atomically reads the value of @v. * Doesn't imply a read memory barrier. */ -static inline long atomic64_read(const atomic64_t *v) +static inline long arch_atomic64_read(const atomic64_t *v) { return READ_ONCE((v)->counter); } /** - * atomic64_set - set atomic64 variable + * arch_atomic64_set - set atomic64 variable * @v: pointer to type atomic64_t * @i: required value * * Atomically sets the value of @v to @i. */ -static inline void atomic64_set(atomic64_t *v, long i) +static inline void arch_atomic64_set(atomic64_t *v, long i) { WRITE_ONCE(v->counter, i); } /** - * atomic64_add - add integer to atomic64 variable + * arch_atomic64_add - add integer to atomic64 variable * @i: integer value to add * @v: pointer to type atomic64_t * * Atomically adds @i to @v. */ -static __always_inline void atomic64_add(long i, atomic64_t *v) +static __always_inline void arch_atomic64_add(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "addq %1,%0" : "=m" (v->counter) @@ -49,13 +49,13 @@ static __always_inline void atomic64_add(long i, atomic64_t *v) } /** - * atomic64_sub - subtract the atomic64 variable + * arch_atomic64_sub - subtract the atomic64 variable * @i: integer value to subtract * @v: pointer to type atomic64_t * * Atomically subtracts @i from @v. */ -static inline void atomic64_sub(long i, atomic64_t *v) +static inline void arch_atomic64_sub(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "subq %1,%0" : "=m" (v->counter) @@ -63,7 +63,7 @@ static inline void atomic64_sub(long i, atomic64_t *v) } /** - * atomic64_sub_and_test - subtract value from variable and test result + * arch_atomic64_sub_and_test - subtract value from variable and test result * @i: integer value to subtract * @v: pointer to type atomic64_t * @@ -71,18 +71,18 @@ static inline void atomic64_sub(long i, atomic64_t *v) * true if the result is zero, or false for all * other cases. */ -static inline bool atomic64_sub_and_test(long i, atomic64_t *v) +static inline bool arch_atomic64_sub_and_test(long i, atomic64_t *v) { GEN_BINARY_RMWcc(LOCK_PREFIX "subq", v->counter, "er", i, "%0", e); } /** - * atomic64_inc - increment atomic64 variable + * arch_atomic64_inc - increment atomic64 variable * @v: pointer to type atomic64_t * * Atomically increments @v by 1. */ -static __always_inline void atomic64_inc(atomic64_t *v) +static __always_inline void arch_atomic64_inc(atomic64_t *v) { asm volatile(LOCK_PREFIX "incq %0" : "=m" (v->counter) @@ -90,12 +90,12 @@ static __always_inline void atomic64_inc(atomic64_t *v) } /** - * atomic64_dec - decrement atomic64 variable + * arch_atomic64_dec - decrement atomic64 variable * @v: pointer to type atomic64_t * * Atomically decrements @v by 1. */ -static __always_inline void atomic64_dec(atomic64_t *v) +static __always_inline void arch_atomic64_dec(atomic64_t *v) { asm volatile(LOCK_PREFIX "decq %0" : "=m" (v->counter) @@ -103,33 +103,33 @@ static __always_inline void atomic64_dec(atomic64_t *v) } /** - * atomic64_dec_and_test - decrement and test + * arch_atomic64_dec_and_test - decrement and test * @v: pointer to type atomic64_t * * Atomically decrements @v by 1 and * returns true if the result is 0, or false for all other * cases. */ -static inline bool atomic64_dec_and_test(atomic64_t *v) +static inline bool arch_atomic64_dec_and_test(atomic64_t *v) { GEN_UNARY_RMWcc(LOCK_PREFIX "decq", v->counter, "%0", e); } /** - * atomic64_inc_and_test - increment and test + * arch_atomic64_inc_and_test - increment and test * @v: pointer to type atomic64_t * * Atomically increments @v by 1 * and returns true if the result is zero, or false for all * other cases. */ -static inline bool atomic64_inc_and_test(atomic64_t *v) +static inline bool arch_atomic64_inc_and_test(atomic64_t *v) { GEN_UNARY_RMWcc(LOCK_PREFIX "incq", v->counter, "%0", e); } /** - * atomic64_add_negative - add and test if negative + * arch_atomic64_add_negative - add and test if negative * @i: integer value to add * @v: pointer to type atomic64_t * @@ -137,59 +137,59 @@ static inline bool atomic64_inc_and_test(atomic64_t *v) * if the result is negative, or false when * result is greater than or equal to zero. */ -static inline bool atomic64_add_negative(long i, atomic64_t *v) +static inline bool arch_atomic64_add_negative(long i, atomic64_t *v) { GEN_BINARY_RMWcc(LOCK_PREFIX "addq", v->counter, "er", i, "%0", s); } /** - * atomic64_add_return - add and return + * arch_atomic64_add_return - add and return * @i: integer value to add * @v: pointer to type atomic64_t * * Atomically adds @i to @v and returns @i + @v */ -static __always_inline long atomic64_add_return(long i, atomic64_t *v) +static __always_inline long arch_atomic64_add_return(long i, atomic64_t *v) { return i + xadd(&v->counter, i); } -static inline long atomic64_sub_return(long i, atomic64_t *v) +static inline long arch_atomic64_sub_return(long i, atomic64_t *v) { - return atomic64_add_return(-i, v); + return arch_atomic64_add_return(-i, v); } -static inline long atomic64_fetch_add(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_add(long i, atomic64_t *v) { return xadd(&v->counter, i); } -static inline long atomic64_fetch_sub(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_sub(long i, atomic64_t *v) { return xadd(&v->counter, -i); } -#define atomic64_inc_return(v) (atomic64_add_return(1, (v))) -#define atomic64_dec_return(v) (atomic64_sub_return(1, (v))) +#define arch_atomic64_inc_return(v) (arch_atomic64_add_return(1, (v))) +#define arch_atomic64_dec_return(v) (arch_atomic64_sub_return(1, (v))) -static inline long atomic64_cmpxchg(atomic64_t *v, long old, long new) +static inline long arch_atomic64_cmpxchg(atomic64_t *v, long old, long new) { - return cmpxchg(&v->counter, old, new); + return arch_cmpxchg(&v->counter, old, new); } -#define atomic64_try_cmpxchg atomic64_try_cmpxchg -static __always_inline bool atomic64_try_cmpxchg(atomic64_t *v, s64 *old, long new) +#define arch_atomic64_try_cmpxchg arch_atomic64_try_cmpxchg +static __always_inline bool arch_atomic64_try_cmpxchg(atomic64_t *v, s64 *old, long new) { return try_cmpxchg(&v->counter, old, new); } -static inline long atomic64_xchg(atomic64_t *v, long new) +static inline long arch_atomic64_xchg(atomic64_t *v, long new) { return xchg(&v->counter, new); } /** - * atomic64_add_unless - add unless the number is a given value + * arch_atomic64_add_unless - add unless the number is a given value * @v: pointer of type atomic64_t * @a: the amount to add to v... * @u: ...unless v is equal to u. @@ -197,37 +197,37 @@ static inline long atomic64_xchg(atomic64_t *v, long new) * Atomically adds @a to @v, so long as it was not @u. * Returns the old value of @v. */ -static inline bool atomic64_add_unless(atomic64_t *v, long a, long u) +static inline bool arch_atomic64_add_unless(atomic64_t *v, long a, long u) { - s64 c = atomic64_read(v); + s64 c = arch_atomic64_read(v); do { if (unlikely(c == u)) return false; - } while (!atomic64_try_cmpxchg(v, &c, c + a)); + } while (!arch_atomic64_try_cmpxchg(v, &c, c + a)); return true; } -#define atomic64_inc_not_zero(v) atomic64_add_unless((v), 1, 0) +#define arch_atomic64_inc_not_zero(v) arch_atomic64_add_unless((v), 1, 0) /* - * atomic64_dec_if_positive - decrement by 1 if old value positive + * arch_atomic64_dec_if_positive - decrement by 1 if old value positive * @v: pointer of type atomic_t * * The function returns the old value of *v minus 1, even if * the atomic variable, v, was not decremented. */ -static inline long atomic64_dec_if_positive(atomic64_t *v) +static inline long arch_atomic64_dec_if_positive(atomic64_t *v) { - s64 dec, c = atomic64_read(v); + s64 dec, c = arch_atomic64_read(v); do { dec = c - 1; if (unlikely(dec < 0)) break; - } while (!atomic64_try_cmpxchg(v, &c, dec)); + } while (!arch_atomic64_try_cmpxchg(v, &c, dec)); return dec; } -static inline void atomic64_and(long i, atomic64_t *v) +static inline void arch_atomic64_and(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "andq %1,%0" : "+m" (v->counter) @@ -235,16 +235,16 @@ static inline void atomic64_and(long i, atomic64_t *v) : "memory"); } -static inline long atomic64_fetch_and(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_and(long i, atomic64_t *v) { - s64 val = atomic64_read(v); + s64 val = arch_atomic64_read(v); do { - } while (!atomic64_try_cmpxchg(v, &val, val & i)); + } while (!arch_atomic64_try_cmpxchg(v, &val, val & i)); return val; } -static inline void atomic64_or(long i, atomic64_t *v) +static inline void arch_atomic64_or(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "orq %1,%0" : "+m" (v->counter) @@ -252,16 +252,16 @@ static inline void atomic64_or(long i, atomic64_t *v) : "memory"); } -static inline long atomic64_fetch_or(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_or(long i, atomic64_t *v) { - s64 val = atomic64_read(v); + s64 val = arch_atomic64_read(v); do { - } while (!atomic64_try_cmpxchg(v, &val, val | i)); + } while (!arch_atomic64_try_cmpxchg(v, &val, val | i)); return val; } -static inline void atomic64_xor(long i, atomic64_t *v) +static inline void arch_atomic64_xor(long i, atomic64_t *v) { asm volatile(LOCK_PREFIX "xorq %1,%0" : "+m" (v->counter) @@ -269,12 +269,12 @@ static inline void atomic64_xor(long i, atomic64_t *v) : "memory"); } -static inline long atomic64_fetch_xor(long i, atomic64_t *v) +static inline long arch_atomic64_fetch_xor(long i, atomic64_t *v) { - s64 val = atomic64_read(v); + s64 val = arch_atomic64_read(v); do { - } while (!atomic64_try_cmpxchg(v, &val, val ^ i)); + } while (!arch_atomic64_try_cmpxchg(v, &val, val ^ i)); return val; } diff --git a/arch/x86/include/asm/cmpxchg.h b/arch/x86/include/asm/cmpxchg.h index 56bd436ed01b..e3efd8a06066 100644 --- a/arch/x86/include/asm/cmpxchg.h +++ b/arch/x86/include/asm/cmpxchg.h @@ -145,13 +145,13 @@ extern void __add_wrong_size(void) # include <asm/cmpxchg_64.h> #endif -#define cmpxchg(ptr, old, new) \ +#define arch_cmpxchg(ptr, old, new) \ __cmpxchg(ptr, old, new, sizeof(*(ptr))) -#define sync_cmpxchg(ptr, old, new) \ +#define arch_sync_cmpxchg(ptr, old, new) \ __sync_cmpxchg(ptr, old, new, sizeof(*(ptr))) -#define cmpxchg_local(ptr, old, new) \ +#define arch_cmpxchg_local(ptr, old, new) \ __cmpxchg_local(ptr, old, new, sizeof(*(ptr))) @@ -221,7 +221,7 @@ extern void __add_wrong_size(void) #define __try_cmpxchg(ptr, pold, new, size) \ __raw_try_cmpxchg((ptr), (pold), (new), (size), LOCK_PREFIX) -#define try_cmpxchg(ptr, pold, new) \ +#define try_cmpxchg(ptr, pold, new) \ __try_cmpxchg((ptr), (pold), (new), sizeof(*(ptr))) /* @@ -250,10 +250,10 @@ extern void __add_wrong_size(void) __ret; \ }) -#define cmpxchg_double(p1, p2, o1, o2, n1, n2) \ +#define arch_cmpxchg_double(p1, p2, o1, o2, n1, n2) \ __cmpxchg_double(LOCK_PREFIX, p1, p2, o1, o2, n1, n2) -#define cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \ +#define arch_cmpxchg_double_local(p1, p2, o1, o2, n1, n2) \ __cmpxchg_double(, p1, p2, o1, o2, n1, n2) #endif /* ASM_X86_CMPXCHG_H */ diff --git a/arch/x86/include/asm/cmpxchg_32.h b/arch/x86/include/asm/cmpxchg_32.h index 1732704f0445..1a2eafca7038 100644 --- a/arch/x86/include/asm/cmpxchg_32.h +++ b/arch/x86/include/asm/cmpxchg_32.h @@ -36,10 +36,10 @@ static inline void set_64bit(volatile u64 *ptr, u64 value) } #ifdef CONFIG_X86_CMPXCHG64 -#define cmpxchg64(ptr, o, n) \ +#define arch_cmpxchg64(ptr, o, n) \ ((__typeof__(*(ptr)))__cmpxchg64((ptr), (unsigned long long)(o), \ (unsigned long long)(n))) -#define cmpxchg64_local(ptr, o, n) \ +#define arch_cmpxchg64_local(ptr, o, n) \ ((__typeof__(*(ptr)))__cmpxchg64_local((ptr), (unsigned long long)(o), \ (unsigned long long)(n))) #endif @@ -76,7 +76,7 @@ static inline u64 __cmpxchg64_local(volatile u64 *ptr, u64 old, u64 new) * to simulate the cmpxchg8b on the 80386 and 80486 CPU. */ -#define cmpxchg64(ptr, o, n) \ +#define arch_cmpxchg64(ptr, o, n) \ ({ \ __typeof__(*(ptr)) __ret; \ __typeof__(*(ptr)) __old = (o); \ @@ -93,7 +93,7 @@ static inline u64 __cmpxchg64_local(volatile u64 *ptr, u64 old, u64 new) __ret; }) -#define cmpxchg64_local(ptr, o, n) \ +#define arch_cmpxchg64_local(ptr, o, n) \ ({ \ __typeof__(*(ptr)) __ret; \ __typeof__(*(ptr)) __old = (o); \ diff --git a/arch/x86/include/asm/cmpxchg_64.h b/arch/x86/include/asm/cmpxchg_64.h index 03cad196a301..bfca3b346c74 100644 --- a/arch/x86/include/asm/cmpxchg_64.h +++ b/arch/x86/include/asm/cmpxchg_64.h @@ -7,13 +7,13 @@ static inline void set_64bit(volatile u64 *ptr, u64 val) *ptr = val; } -#define cmpxchg64(ptr, o, n) \ +#define arch_cmpxchg64(ptr, o, n) \ ({ \ BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ cmpxchg((ptr), (o), (n)); \ }) -#define cmpxchg64_local(ptr, o, n) \ +#define arch_cmpxchg64_local(ptr, o, n) \ ({ \ BUILD_BUG_ON(sizeof(*(ptr)) != 8); \ cmpxchg_local((ptr), (o), (n)); \ ^ permalink raw reply related [flat|nested] 63+ messages in thread
end of thread, other threads:[~2018-03-12 12:24 UTC | newest]
Thread overview: 63+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-06-22 14:14 [PATCH v5 0/4] x86, kasan: add KASAN checks to atomic operations Dmitry Vyukov
[not found] ` <cover.1498140838.git.dvyukov@google.com>
2017-06-22 14:14 ` [PATCH v5 1/4] x86: switch atomic.h to use atomic-instrumented.h Dmitry Vyukov
2017-06-22 14:14 ` Dmitry Vyukov
2017-06-22 21:14 ` Andrew Morton
2017-06-22 21:14 ` Andrew Morton
2017-06-23 8:23 ` Dmitry Vyukov
2017-06-23 8:23 ` Dmitry Vyukov
2017-06-23 8:54 ` Ingo Molnar
2017-06-23 8:54 ` Ingo Molnar
2017-06-23 19:00 ` Andrew Morton
2017-06-23 19:00 ` Andrew Morton
2017-06-23 13:59 ` [tip:locking/core] locking/atomic/x86: Switch " tip-bot for Dmitry Vyukov
2017-06-22 14:14 ` [PATCH v5 2/4] kasan: allow kasan_check_read/write() to accept pointers to volatiles Dmitry Vyukov
2017-06-22 14:14 ` Dmitry Vyukov
2017-06-23 14:00 ` [tip:locking/core] kasan: Allow " tip-bot for Dmitry Vyukov
2017-07-26 12:10 ` tip-bot for Dmitry Vyukov
2017-06-22 14:14 ` [PATCH v5 3/4] asm-generic: add KASAN instrumentation to atomic operations Dmitry Vyukov
2017-06-22 14:14 ` Dmitry Vyukov
2017-06-23 14:01 ` [tip:locking/core] locking/atomics, asm-generic: Add " tip-bot for Dmitry Vyukov
2017-06-28 10:02 ` [PATCH] locking/atomics: don't alias ____ptr Sebastian Andrzej Siewior
2017-06-28 10:02 ` Sebastian Andrzej Siewior
2017-06-28 10:16 ` Dmitry Vyukov
2017-06-28 10:16 ` Dmitry Vyukov
2017-06-28 11:10 ` Thomas Gleixner
2017-06-28 11:10 ` Thomas Gleixner
2017-06-28 11:12 ` Dmitry Vyukov
2017-06-28 11:12 ` Dmitry Vyukov
2017-06-28 11:21 ` Thomas Gleixner
2017-06-28 11:21 ` Thomas Gleixner
2017-06-28 12:45 ` Mark Rutland
2017-06-28 12:45 ` Mark Rutland
2017-06-28 12:24 ` Thomas Gleixner
2017-06-28 12:24 ` Thomas Gleixner
2017-06-28 12:27 ` Dmitry Vyukov
2017-06-28 12:27 ` Dmitry Vyukov
2017-06-28 13:33 ` Thomas Gleixner
2017-06-28 13:33 ` Thomas Gleixner
2017-06-28 11:15 ` Andrey Ryabinin
2017-06-28 11:15 ` Andrey Ryabinin
2017-06-28 12:12 ` Sebastian Andrzej Siewior
2017-06-28 12:12 ` Sebastian Andrzej Siewior
2017-06-28 13:20 ` Thomas Gleixner
2017-06-28 13:20 ` Thomas Gleixner
2017-06-28 13:54 ` Thomas Gleixner
2017-06-28 13:54 ` Thomas Gleixner
2017-06-28 14:14 ` Mark Rutland
2017-06-28 14:14 ` Mark Rutland
2017-06-28 15:24 ` Thomas Gleixner
2017-06-28 15:24 ` Thomas Gleixner
2017-06-28 15:54 ` Mark Rutland
2017-06-28 15:54 ` Mark Rutland
2017-06-28 16:56 ` Ingo Molnar
2017-06-28 16:56 ` Ingo Molnar
2017-06-28 18:21 ` Thomas Gleixner
2017-06-28 18:21 ` Thomas Gleixner
2017-06-29 6:47 ` Thomas Gleixner
2017-06-29 6:47 ` Thomas Gleixner
2017-06-28 14:00 ` Andrey Ryabinin
2017-06-28 14:00 ` Andrey Ryabinin
2017-06-22 14:14 ` [PATCH v5 4/4] asm-generic, x86: add comments for atomic instrumentation Dmitry Vyukov
2017-06-22 14:14 ` Dmitry Vyukov
2017-06-23 14:01 ` [tip:locking/core] locking/atomic/x86, asm-generic: Add " tip-bot for Dmitry Vyukov
2018-01-29 17:26 [PATCH v6 2/4] x86: switch atomic.h to use atomic-instrumented.h Dmitry Vyukov
2018-03-12 12:24 ` [tip:locking/core] locking/atomic/x86: Switch " tip-bot for Dmitry Vyukov
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.