linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH] READ_ONCE, WRITE_ONCE, kcsan: Perform checks in __*_ONCE variants
@ 2020-05-12 18:38 Marco Elver
  2020-05-12 19:09 ` Peter Zijlstra
  0 siblings, 1 reply; 11+ messages in thread
From: Marco Elver @ 2020-05-12 18:38 UTC (permalink / raw)
  To: elver
  Cc: linux-kernel, kasan-dev, Will Deacon, Thomas Gleixner,
	Paul E . McKenney, Ingo Molnar, Peter Zijlstra

If left plain, using __READ_ONCE and __WRITE_ONCE will result in many
false positives with KCSAN due to being instrumented normally. To fix,
we should move the kcsan_check and data_race into __*_ONCE.

Cc: Will Deacon <will@kernel.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Paul E. McKenney <paulmck@kernel.org>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Marco Elver <elver@google.com>
---
A proposal to fix the problem with __READ_ONCE/__WRITE_ONCE and KCSAN
false positives.

Will, please feel free to take this patch and fiddle with it until it
looks like what you want if this is completely off.

Note: Currently __WRITE_ONCE_SCALAR seems to serve no real purpose. Do
we still need it?
---
 include/linux/compiler.h | 15 +++++++++------
 1 file changed, 9 insertions(+), 6 deletions(-)

diff --git a/include/linux/compiler.h b/include/linux/compiler.h
index 741c93c62ecf..e902ca5de811 100644
--- a/include/linux/compiler.h
+++ b/include/linux/compiler.h
@@ -224,13 +224,16 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
  * atomicity or dependency ordering guarantees. Note that this may result
  * in tears!
  */
-#define __READ_ONCE(x)	(*(const volatile __unqual_scalar_typeof(x) *)&(x))
+#define __READ_ONCE(x)							\
+({									\
+	kcsan_check_atomic_read(&(x), sizeof(x));			\
+	data_race((*(const volatile __unqual_scalar_typeof(x) *)&(x))); \
+})
 
 #define __READ_ONCE_SCALAR(x)						\
 ({									\
 	typeof(x) *__xp = &(x);						\
-	__unqual_scalar_typeof(x) __x = data_race(__READ_ONCE(*__xp));	\
-	kcsan_check_atomic_read(__xp, sizeof(*__xp));			\
+	__unqual_scalar_typeof(x) __x = __READ_ONCE(*__xp);		\
 	smp_read_barrier_depends();					\
 	(typeof(x))__x;							\
 })
@@ -243,14 +246,14 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
 
 #define __WRITE_ONCE(x, val)						\
 do {									\
-	*(volatile typeof(x) *)&(x) = (val);				\
+	kcsan_check_atomic_write(&(x), sizeof(x));			\
+	data_race(*(volatile typeof(x) *)&(x) = (val));			\
 } while (0)
 
 #define __WRITE_ONCE_SCALAR(x, val)					\
 do {									\
 	typeof(x) *__xp = &(x);						\
-	kcsan_check_atomic_write(__xp, sizeof(*__xp));			\
-	data_race(({ __WRITE_ONCE(*__xp, val); 0; }));			\
+	__WRITE_ONCE(*__xp, val);					\
 } while (0)
 
 #define WRITE_ONCE(x, val)						\
-- 
2.26.2.645.ge9eca65c58-goog


^ permalink raw reply related	[flat|nested] 11+ messages in thread

* Re: [PATCH] READ_ONCE, WRITE_ONCE, kcsan: Perform checks in __*_ONCE variants
  2020-05-12 18:38 [PATCH] READ_ONCE, WRITE_ONCE, kcsan: Perform checks in __*_ONCE variants Marco Elver
@ 2020-05-12 19:09 ` Peter Zijlstra
  2020-05-19 21:10   ` Qian Cai
  0 siblings, 1 reply; 11+ messages in thread
From: Peter Zijlstra @ 2020-05-12 19:09 UTC (permalink / raw)
  To: Marco Elver
  Cc: linux-kernel, kasan-dev, Will Deacon, Thomas Gleixner,
	Paul E . McKenney, Ingo Molnar

On Tue, May 12, 2020 at 08:38:39PM +0200, Marco Elver wrote:
> diff --git a/include/linux/compiler.h b/include/linux/compiler.h
> index 741c93c62ecf..e902ca5de811 100644
> --- a/include/linux/compiler.h
> +++ b/include/linux/compiler.h
> @@ -224,13 +224,16 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
>   * atomicity or dependency ordering guarantees. Note that this may result
>   * in tears!
>   */
> -#define __READ_ONCE(x)	(*(const volatile __unqual_scalar_typeof(x) *)&(x))
> +#define __READ_ONCE(x)							\
> +({									\
> +	kcsan_check_atomic_read(&(x), sizeof(x));			\
> +	data_race((*(const volatile __unqual_scalar_typeof(x) *)&(x))); \
> +})

NAK

This will actively insert instrumentation into __READ_ONCE() and I need
it to not have any.



^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH] READ_ONCE, WRITE_ONCE, kcsan: Perform checks in __*_ONCE variants
  2020-05-12 19:09 ` Peter Zijlstra
@ 2020-05-19 21:10   ` Qian Cai
  2020-05-19 21:25     ` Marco Elver
  0 siblings, 1 reply; 11+ messages in thread
From: Qian Cai @ 2020-05-19 21:10 UTC (permalink / raw)
  To: Peter Zijlstra
  Cc: Marco Elver, Linux Kernel Mailing List, kasan-dev, Will Deacon,
	Thomas Gleixner, Paul E . McKenney, Ingo Molnar

On Tue, May 12, 2020 at 3:09 PM Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Tue, May 12, 2020 at 08:38:39PM +0200, Marco Elver wrote:
> > diff --git a/include/linux/compiler.h b/include/linux/compiler.h
> > index 741c93c62ecf..e902ca5de811 100644
> > --- a/include/linux/compiler.h
> > +++ b/include/linux/compiler.h
> > @@ -224,13 +224,16 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
> >   * atomicity or dependency ordering guarantees. Note that this may result
> >   * in tears!
> >   */
> > -#define __READ_ONCE(x)       (*(const volatile __unqual_scalar_typeof(x) *)&(x))
> > +#define __READ_ONCE(x)                                                       \
> > +({                                                                   \
> > +     kcsan_check_atomic_read(&(x), sizeof(x));                       \
> > +     data_race((*(const volatile __unqual_scalar_typeof(x) *)&(x))); \
> > +})
>
> NAK
>
> This will actively insert instrumentation into __READ_ONCE() and I need
> it to not have any.

Any way to move this forward? Due to linux-next commit 6bcc8f459fe7
(locking/atomics: Flip fallbacks and instrumentation), it triggers a
lots of KCSAN warnings due to atomic ops are no longer marked. For
example,

[  197.318288][ T1041] write to 0xffff9302764ccc78 of 8 bytes by task
1048 on cpu 47:
[  197.353119][ T1041]  down_read_trylock+0x9e/0x1e0
atomic_long_set(&sem->owner, val);
__rwsem_set_reader_owned at kernel/locking/rwsem.c:205
(inlined by) rwsem_set_reader_owned at kernel/locking/rwsem.c:213
(inlined by) __down_read_trylock at kernel/locking/rwsem.c:1373
(inlined by) down_read_trylock at kernel/locking/rwsem.c:1517
[  197.374641][ T1041]  page_lock_anon_vma_read+0x19d/0x3c0
[  197.398894][ T1041]  rmap_walk_anon+0x30e/0x620

[  197.924695][ T1041] read to 0xffff9302764ccc78 of 8 bytes by task
1041 on cpu 43:
[  197.959501][ T1041]  up_read+0xb8/0x41a
arch_atomic64_read at arch/x86/include/asm/atomic64_64.h:22
(inlined by) atomic64_read at include/asm-generic/atomic-instrumented.h:838
(inlined by) atomic_long_read at include/asm-generic/atomic-long.h:29
(inlined by) rwsem_clear_reader_owned at kernel/locking/rwsem.c:242
(inlined by) __up_read at kernel/locking/rwsem.c:1433
(inlined by) up_read at kernel/locking/rwsem.c:1574
[  197.977728][ T1041]  rmap_walk_anon+0x2f2/0x620
[  197.999055][ T1041]  rmap_walk+0xb5/0xe0

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH] READ_ONCE, WRITE_ONCE, kcsan: Perform checks in __*_ONCE variants
  2020-05-19 21:10   ` Qian Cai
@ 2020-05-19 21:25     ` Marco Elver
  2020-05-19 21:45       ` Qian Cai
  0 siblings, 1 reply; 11+ messages in thread
From: Marco Elver @ 2020-05-19 21:25 UTC (permalink / raw)
  To: Qian Cai
  Cc: Peter Zijlstra, Linux Kernel Mailing List, kasan-dev,
	Will Deacon, Thomas Gleixner, Paul E . McKenney, Ingo Molnar

On Tue, 19 May 2020 at 23:10, Qian Cai <cai@lca.pw> wrote:
>
> On Tue, May 12, 2020 at 3:09 PM Peter Zijlstra <peterz@infradead.org> wrote:
> >
> > On Tue, May 12, 2020 at 08:38:39PM +0200, Marco Elver wrote:
> > > diff --git a/include/linux/compiler.h b/include/linux/compiler.h
> > > index 741c93c62ecf..e902ca5de811 100644
> > > --- a/include/linux/compiler.h
> > > +++ b/include/linux/compiler.h
> > > @@ -224,13 +224,16 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
> > >   * atomicity or dependency ordering guarantees. Note that this may result
> > >   * in tears!
> > >   */
> > > -#define __READ_ONCE(x)       (*(const volatile __unqual_scalar_typeof(x) *)&(x))
> > > +#define __READ_ONCE(x)                                                       \
> > > +({                                                                   \
> > > +     kcsan_check_atomic_read(&(x), sizeof(x));                       \
> > > +     data_race((*(const volatile __unqual_scalar_typeof(x) *)&(x))); \
> > > +})
> >
> > NAK
> >
> > This will actively insert instrumentation into __READ_ONCE() and I need
> > it to not have any.
>
> Any way to move this forward? Due to linux-next commit 6bcc8f459fe7
> (locking/atomics: Flip fallbacks and instrumentation), it triggers a
> lots of KCSAN warnings due to atomic ops are no longer marked.

This is no longer the right solution we believe due to the various
requirements that Peter also mentioned. See the discussion here:
    https://lkml.kernel.org/r/CANpmjNOGFqhtDa9wWpXs2kztQsSozbwsuMO5BqqW0c0g0zGfSA@mail.gmail.com

The new solution is here:
    https://lkml.kernel.org/r/20200515150338.190344-1-elver@google.com
While it's a little inconvenient that we'll require Clang 11
(currently available by building yourself from LLVM repo), but until
we get GCC fixed (my patch there still pending :-/), this is probably
the right solution going forward.   If possible, please do test!

Thanks,
-- Marco

> For
> example,
> [  197.318288][ T1041] write to 0xffff9302764ccc78 of 8 bytes by task
> 1048 on cpu 47:
> [  197.353119][ T1041]  down_read_trylock+0x9e/0x1e0
> atomic_long_set(&sem->owner, val);
> __rwsem_set_reader_owned at kernel/locking/rwsem.c:205
> (inlined by) rwsem_set_reader_owned at kernel/locking/rwsem.c:213
> (inlined by) __down_read_trylock at kernel/locking/rwsem.c:1373
> (inlined by) down_read_trylock at kernel/locking/rwsem.c:1517
> [  197.374641][ T1041]  page_lock_anon_vma_read+0x19d/0x3c0
> [  197.398894][ T1041]  rmap_walk_anon+0x30e/0x620
>
> [  197.924695][ T1041] read to 0xffff9302764ccc78 of 8 bytes by task
> 1041 on cpu 43:
> [  197.959501][ T1041]  up_read+0xb8/0x41a
> arch_atomic64_read at arch/x86/include/asm/atomic64_64.h:22
> (inlined by) atomic64_read at include/asm-generic/atomic-instrumented.h:838
> (inlined by) atomic_long_read at include/asm-generic/atomic-long.h:29
> (inlined by) rwsem_clear_reader_owned at kernel/locking/rwsem.c:242
> (inlined by) __up_read at kernel/locking/rwsem.c:1433
> (inlined by) up_read at kernel/locking/rwsem.c:1574
> [  197.977728][ T1041]  rmap_walk_anon+0x2f2/0x620
> [  197.999055][ T1041]  rmap_walk+0xb5/0xe0

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH] READ_ONCE, WRITE_ONCE, kcsan: Perform checks in __*_ONCE variants
  2020-05-19 21:25     ` Marco Elver
@ 2020-05-19 21:45       ` Qian Cai
  2020-05-19 22:05         ` Thomas Gleixner
  0 siblings, 1 reply; 11+ messages in thread
From: Qian Cai @ 2020-05-19 21:45 UTC (permalink / raw)
  To: Marco Elver
  Cc: Peter Zijlstra, Linux Kernel Mailing List, kasan-dev,
	Will Deacon, Thomas Gleixner, Paul E . McKenney, Ingo Molnar

On Tue, May 19, 2020 at 5:26 PM Marco Elver <elver@google.com> wrote:
>
> On Tue, 19 May 2020 at 23:10, Qian Cai <cai@lca.pw> wrote:
> >
> > On Tue, May 12, 2020 at 3:09 PM Peter Zijlstra <peterz@infradead.org> wrote:
> > >
> > > On Tue, May 12, 2020 at 08:38:39PM +0200, Marco Elver wrote:
> > > > diff --git a/include/linux/compiler.h b/include/linux/compiler.h
> > > > index 741c93c62ecf..e902ca5de811 100644
> > > > --- a/include/linux/compiler.h
> > > > +++ b/include/linux/compiler.h
> > > > @@ -224,13 +224,16 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
> > > >   * atomicity or dependency ordering guarantees. Note that this may result
> > > >   * in tears!
> > > >   */
> > > > -#define __READ_ONCE(x)       (*(const volatile __unqual_scalar_typeof(x) *)&(x))
> > > > +#define __READ_ONCE(x)                                                       \
> > > > +({                                                                   \
> > > > +     kcsan_check_atomic_read(&(x), sizeof(x));                       \
> > > > +     data_race((*(const volatile __unqual_scalar_typeof(x) *)&(x))); \
> > > > +})
> > >
> > > NAK
> > >
> > > This will actively insert instrumentation into __READ_ONCE() and I need
> > > it to not have any.
> >
> > Any way to move this forward? Due to linux-next commit 6bcc8f459fe7
> > (locking/atomics: Flip fallbacks and instrumentation), it triggers a
> > lots of KCSAN warnings due to atomic ops are no longer marked.
>
> This is no longer the right solution we believe due to the various
> requirements that Peter also mentioned. See the discussion here:
>     https://lkml.kernel.org/r/CANpmjNOGFqhtDa9wWpXs2kztQsSozbwsuMO5BqqW0c0g0zGfSA@mail.gmail.com
>
> The new solution is here:
>     https://lkml.kernel.org/r/20200515150338.190344-1-elver@google.com
> While it's a little inconvenient that we'll require Clang 11
> (currently available by building yourself from LLVM repo), but until
> we get GCC fixed (my patch there still pending :-/), this is probably
> the right solution going forward.   If possible, please do test!

That would be quite unfortunate. The version here is still gcc-8.3.1
and clang-9.0.1 on RHEL 8.2 here. It will probably need many years to
be able to get the fixed compilers having versions that high. Sigh...
Also, I want to avoid compiling compilers on my own.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH] READ_ONCE, WRITE_ONCE, kcsan: Perform checks in __*_ONCE variants
  2020-05-19 21:45       ` Qian Cai
@ 2020-05-19 22:05         ` Thomas Gleixner
  2020-05-20  2:28           ` Qian Cai
  0 siblings, 1 reply; 11+ messages in thread
From: Thomas Gleixner @ 2020-05-19 22:05 UTC (permalink / raw)
  To: Qian Cai, Marco Elver
  Cc: Peter Zijlstra, Linux Kernel Mailing List, kasan-dev,
	Will Deacon, Paul E . McKenney, Ingo Molnar

Qian Cai <cai@lca.pw> writes:
> On Tue, May 19, 2020 at 5:26 PM Marco Elver <elver@google.com> wrote:
>> The new solution is here:
>>     https://lkml.kernel.org/r/20200515150338.190344-1-elver@google.com
>> While it's a little inconvenient that we'll require Clang 11
>> (currently available by building yourself from LLVM repo), but until
>> we get GCC fixed (my patch there still pending :-/), this is probably
>> the right solution going forward.   If possible, please do test!
>
> That would be quite unfortunate. The version here is still gcc-8.3.1
> and clang-9.0.1 on RHEL 8.2 here. It will probably need many years to
> be able to get the fixed compilers having versions that high. Sigh...
> Also, I want to avoid compiling compilers on my own.

Yes, it's unfortunate, but we have to stop making major concessions just
because tools are not up to the task.

We've done that way too much in the past and this particular problem
clearly demonstrates that there are limits.

Making brand new technology depend on sane tools is not asked too
much. And yes, it's inconvenient, but all of us have to build tools
every now and then to get our job done. It's not the end of the world.

Building clang is trivial enough and pointing the make to the right
compiler is not rocket science either.

Thanks,

        tglx

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH] READ_ONCE, WRITE_ONCE, kcsan: Perform checks in __*_ONCE variants
  2020-05-19 22:05         ` Thomas Gleixner
@ 2020-05-20  2:28           ` Qian Cai
  2020-05-20  2:47             ` Nathan Chancellor
  0 siblings, 1 reply; 11+ messages in thread
From: Qian Cai @ 2020-05-20  2:28 UTC (permalink / raw)
  To: Thomas Gleixner
  Cc: Marco Elver, Peter Zijlstra, Linux Kernel Mailing List,
	kasan-dev, Will Deacon, Paul E . McKenney, Ingo Molnar



> On May 19, 2020, at 6:05 PM, Thomas Gleixner <tglx@linutronix.de> wrote:
> 
> Yes, it's unfortunate, but we have to stop making major concessions just
> because tools are not up to the task.
> 
> We've done that way too much in the past and this particular problem
> clearly demonstrates that there are limits.
> 
> Making brand new technology depend on sane tools is not asked too
> much. And yes, it's inconvenient, but all of us have to build tools
> every now and then to get our job done. It's not the end of the world.
> 
> Building clang is trivial enough and pointing the make to the right
> compiler is not rocket science either.

Yes, it all make sense from that angle. On the other hand, I want to be focus on kernel rather than compilers by using a stable and rocket-solid version. Not mentioned the time lost by compiling and properly manage my own toolchain in an automated environment, using such new version of compilers means that I have to inevitably deal with compiler bugs occasionally. Anyway, it is just some other more bugs I have to deal with, and I don’t have a better solution to offer right now.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH] READ_ONCE, WRITE_ONCE, kcsan: Perform checks in __*_ONCE variants
  2020-05-20  2:28           ` Qian Cai
@ 2020-05-20  2:47             ` Nathan Chancellor
  2020-05-20  3:16               ` Qian Cai
  0 siblings, 1 reply; 11+ messages in thread
From: Nathan Chancellor @ 2020-05-20  2:47 UTC (permalink / raw)
  To: Qian Cai
  Cc: Thomas Gleixner, Marco Elver, Peter Zijlstra,
	Linux Kernel Mailing List, kasan-dev, Will Deacon,
	Paul E . McKenney, Ingo Molnar, clang-built-linux

On Tue, May 19, 2020 at 10:28:41PM -0400, Qian Cai wrote:
> 
> 
> > On May 19, 2020, at 6:05 PM, Thomas Gleixner <tglx@linutronix.de> wrote:
> > 
> > Yes, it's unfortunate, but we have to stop making major concessions just
> > because tools are not up to the task.
> > 
> > We've done that way too much in the past and this particular problem
> > clearly demonstrates that there are limits.
> > 
> > Making brand new technology depend on sane tools is not asked too
> > much. And yes, it's inconvenient, but all of us have to build tools
> > every now and then to get our job done. It's not the end of the world.
> > 
> > Building clang is trivial enough and pointing the make to the right
> > compiler is not rocket science either.
> 
> Yes, it all make sense from that angle. On the other hand, I want to be focus on kernel rather than compilers by using a stable and rocket-solid version. Not mentioned the time lost by compiling and properly manage my own toolchain in an automated environment, using such new version of compilers means that I have to inevitably deal with compiler bugs occasionally. Anyway, it is just some other more bugs I have to deal with, and I don’t have a better solution to offer right now.

Hi Qian,

Shameless plug but I have made a Python script to efficiently configure
then build clang specifically for building the kernel (turn off a lot of
different things that the kernel does not need).

https://github.com/ClangBuiltLinux/tc-build

I added an option '--use-good-revision', which uses an older master
version (basically somewhere between clang-10 and current master) that
has been qualified against the kernel. I currently update it every
Linux release but I am probably going to start doing it every month as
I have written a pretty decent framework to ensure that nothing is
breaking on either the LLVM or kernel side.

$ ./build-llvm.py --use-good-revision

should be all you need to get off the ground and running if you wanted
to give it a shot. The script is completely self contained by default so
it won't mess with the rest of your system. Additionally, leaving off
'--use-good-revision' will just use the master branch, which can
definitely be broken but not as often as you would think (although I
totally understand wanting to focus on kernel regressions only).

Cheers,
Nathan

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH] READ_ONCE, WRITE_ONCE, kcsan: Perform checks in __*_ONCE variants
  2020-05-20  2:47             ` Nathan Chancellor
@ 2020-05-20  3:16               ` Qian Cai
  2020-05-20  3:44                 ` Nathan Chancellor
  0 siblings, 1 reply; 11+ messages in thread
From: Qian Cai @ 2020-05-20  3:16 UTC (permalink / raw)
  To: Nathan Chancellor
  Cc: Thomas Gleixner, Marco Elver, Peter Zijlstra,
	Linux Kernel Mailing List, kasan-dev, Will Deacon,
	Paul E . McKenney, Ingo Molnar, clang-built-linux

On Tue, May 19, 2020 at 10:47 PM Nathan Chancellor
<natechancellor@gmail.com> wrote:
>
> On Tue, May 19, 2020 at 10:28:41PM -0400, Qian Cai wrote:
> >
> >
> > > On May 19, 2020, at 6:05 PM, Thomas Gleixner <tglx@linutronix.de> wrote:
> > >
> > > Yes, it's unfortunate, but we have to stop making major concessions just
> > > because tools are not up to the task.
> > >
> > > We've done that way too much in the past and this particular problem
> > > clearly demonstrates that there are limits.
> > >
> > > Making brand new technology depend on sane tools is not asked too
> > > much. And yes, it's inconvenient, but all of us have to build tools
> > > every now and then to get our job done. It's not the end of the world.
> > >
> > > Building clang is trivial enough and pointing the make to the right
> > > compiler is not rocket science either.
> >
> > Yes, it all make sense from that angle. On the other hand, I want to be focus on kernel rather than compilers by using a stable and rocket-solid version. Not mentioned the time lost by compiling and properly manage my own toolchain in an automated environment, using such new version of compilers means that I have to inevitably deal with compiler bugs occasionally. Anyway, it is just some other more bugs I have to deal with, and I don’t have a better solution to offer right now.
>
> Hi Qian,
>
> Shameless plug but I have made a Python script to efficiently configure
> then build clang specifically for building the kernel (turn off a lot of
> different things that the kernel does not need).
>
> https://github.com/ClangBuiltLinux/tc-build
>
> I added an option '--use-good-revision', which uses an older master
> version (basically somewhere between clang-10 and current master) that
> has been qualified against the kernel. I currently update it every
> Linux release but I am probably going to start doing it every month as
> I have written a pretty decent framework to ensure that nothing is
> breaking on either the LLVM or kernel side.
>
> $ ./build-llvm.py --use-good-revision
>
> should be all you need to get off the ground and running if you wanted
> to give it a shot. The script is completely self contained by default so
> it won't mess with the rest of your system. Additionally, leaving off
> '--use-good-revision' will just use the master branch, which can
> definitely be broken but not as often as you would think (although I
> totally understand wanting to focus on kernel regressions only).

Great, thanks. I'll try it in a bit.

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH] READ_ONCE, WRITE_ONCE, kcsan: Perform checks in __*_ONCE variants
  2020-05-20  3:16               ` Qian Cai
@ 2020-05-20  3:44                 ` Nathan Chancellor
  2020-05-20  7:28                   ` Marco Elver
  0 siblings, 1 reply; 11+ messages in thread
From: Nathan Chancellor @ 2020-05-20  3:44 UTC (permalink / raw)
  To: Qian Cai
  Cc: Thomas Gleixner, Marco Elver, Peter Zijlstra,
	Linux Kernel Mailing List, kasan-dev, Will Deacon,
	Paul E . McKenney, Ingo Molnar, clang-built-linux

On Tue, May 19, 2020 at 11:16:24PM -0400, Qian Cai wrote:
> On Tue, May 19, 2020 at 10:47 PM Nathan Chancellor
> <natechancellor@gmail.com> wrote:
> >
> > On Tue, May 19, 2020 at 10:28:41PM -0400, Qian Cai wrote:
> > >
> > >
> > > > On May 19, 2020, at 6:05 PM, Thomas Gleixner <tglx@linutronix.de> wrote:
> > > >
> > > > Yes, it's unfortunate, but we have to stop making major concessions just
> > > > because tools are not up to the task.
> > > >
> > > > We've done that way too much in the past and this particular problem
> > > > clearly demonstrates that there are limits.
> > > >
> > > > Making brand new technology depend on sane tools is not asked too
> > > > much. And yes, it's inconvenient, but all of us have to build tools
> > > > every now and then to get our job done. It's not the end of the world.
> > > >
> > > > Building clang is trivial enough and pointing the make to the right
> > > > compiler is not rocket science either.
> > >
> > > Yes, it all make sense from that angle. On the other hand, I want to be focus on kernel rather than compilers by using a stable and rocket-solid version. Not mentioned the time lost by compiling and properly manage my own toolchain in an automated environment, using such new version of compilers means that I have to inevitably deal with compiler bugs occasionally. Anyway, it is just some other more bugs I have to deal with, and I don’t have a better solution to offer right now.
> >
> > Hi Qian,
> >
> > Shameless plug but I have made a Python script to efficiently configure
> > then build clang specifically for building the kernel (turn off a lot of
> > different things that the kernel does not need).
> >
> > https://github.com/ClangBuiltLinux/tc-build
> >
> > I added an option '--use-good-revision', which uses an older master
> > version (basically somewhere between clang-10 and current master) that
> > has been qualified against the kernel. I currently update it every
> > Linux release but I am probably going to start doing it every month as
> > I have written a pretty decent framework to ensure that nothing is
> > breaking on either the LLVM or kernel side.
> >
> > $ ./build-llvm.py --use-good-revision
> >
> > should be all you need to get off the ground and running if you wanted
> > to give it a shot. The script is completely self contained by default so
> > it won't mess with the rest of your system. Additionally, leaving off
> > '--use-good-revision' will just use the master branch, which can
> > definitely be broken but not as often as you would think (although I
> > totally understand wanting to focus on kernel regressions only).
> 
> Great, thanks. I'll try it in a bit.

Please let me know if there are any issues!

Do note that in order to get support for Marco's series, you will need
to have a version of LLVM that includes [1], which the current
--use-good-revision does not. You can checkout that revision exactly
through the '-b' ('--branch') parameter:

$ ./build-llvm.py -b 5a2c31116f412c3b6888be361137efd705e05814

I also see another patch in LLVM that concerns KCSAN [2] but that does
not appear used in Marco's series. Still might be worth having available
in your version of clang.

I'll try to bump the hash that '--use-good-revision' uses soon. I might
wait until 5.7 final so that I can do both at the same time like I
usually do but we'll see how much time I have.

[1]: https://github.com/llvm/llvm-project/commit/5a2c31116f412c3b6888be361137efd705e05814
[2]: https://github.com/llvm/llvm-project/commit/151ed6aa38a3ec6c01973b35f684586b6e1c0f7e

Cheers,
Nathan

^ permalink raw reply	[flat|nested] 11+ messages in thread

* Re: [PATCH] READ_ONCE, WRITE_ONCE, kcsan: Perform checks in __*_ONCE variants
  2020-05-20  3:44                 ` Nathan Chancellor
@ 2020-05-20  7:28                   ` Marco Elver
  0 siblings, 0 replies; 11+ messages in thread
From: Marco Elver @ 2020-05-20  7:28 UTC (permalink / raw)
  To: Nathan Chancellor
  Cc: Qian Cai, Thomas Gleixner, Peter Zijlstra,
	Linux Kernel Mailing List, kasan-dev, Will Deacon,
	Paul E . McKenney, Ingo Molnar, clang-built-linux

On Wed, 20 May 2020 at 05:44, Nathan Chancellor
<natechancellor@gmail.com> wrote:
>
> On Tue, May 19, 2020 at 11:16:24PM -0400, Qian Cai wrote:
> > On Tue, May 19, 2020 at 10:47 PM Nathan Chancellor
> > <natechancellor@gmail.com> wrote:
> > >
> > > On Tue, May 19, 2020 at 10:28:41PM -0400, Qian Cai wrote:
> > > >
> > > >
> > > > > On May 19, 2020, at 6:05 PM, Thomas Gleixner <tglx@linutronix.de> wrote:
> > > > >
> > > > > Yes, it's unfortunate, but we have to stop making major concessions just
> > > > > because tools are not up to the task.
> > > > >
> > > > > We've done that way too much in the past and this particular problem
> > > > > clearly demonstrates that there are limits.
> > > > >
> > > > > Making brand new technology depend on sane tools is not asked too
> > > > > much. And yes, it's inconvenient, but all of us have to build tools
> > > > > every now and then to get our job done. It's not the end of the world.
> > > > >
> > > > > Building clang is trivial enough and pointing the make to the right
> > > > > compiler is not rocket science either.
> > > >
> > > > Yes, it all make sense from that angle. On the other hand, I want to be focus on kernel rather than compilers by using a stable and rocket-solid version. Not mentioned the time lost by compiling and properly manage my own toolchain in an automated environment, using such new version of compilers means that I have to inevitably deal with compiler bugs occasionally. Anyway, it is just some other more bugs I have to deal with, and I don’t have a better solution to offer right now.
> > >
> > > Hi Qian,
> > >
> > > Shameless plug but I have made a Python script to efficiently configure
> > > then build clang specifically for building the kernel (turn off a lot of
> > > different things that the kernel does not need).
> > >
> > > https://github.com/ClangBuiltLinux/tc-build
> > >
> > > I added an option '--use-good-revision', which uses an older master
> > > version (basically somewhere between clang-10 and current master) that
> > > has been qualified against the kernel. I currently update it every
> > > Linux release but I am probably going to start doing it every month as
> > > I have written a pretty decent framework to ensure that nothing is
> > > breaking on either the LLVM or kernel side.
> > >
> > > $ ./build-llvm.py --use-good-revision
> > >
> > > should be all you need to get off the ground and running if you wanted
> > > to give it a shot. The script is completely self contained by default so
> > > it won't mess with the rest of your system. Additionally, leaving off
> > > '--use-good-revision' will just use the master branch, which can
> > > definitely be broken but not as often as you would think (although I
> > > totally understand wanting to focus on kernel regressions only).
> >
> > Great, thanks. I'll try it in a bit.
>
> Please let me know if there are any issues!
>
> Do note that in order to get support for Marco's series, you will need
> to have a version of LLVM that includes [1], which the current
> --use-good-revision does not. You can checkout that revision exactly
> through the '-b' ('--branch') parameter:
>
> $ ./build-llvm.py -b 5a2c31116f412c3b6888be361137efd705e05814
>
> I also see another patch in LLVM that concerns KCSAN [2] but that does
> not appear used in Marco's series. Still might be worth having available
> in your version of clang.
>
> I'll try to bump the hash that '--use-good-revision' uses soon. I might
> wait until 5.7 final so that I can do both at the same time like I
> usually do but we'll see how much time I have.
>
> [1]: https://github.com/llvm/llvm-project/commit/5a2c31116f412c3b6888be361137efd705e05814
> [2]: https://github.com/llvm/llvm-project/commit/151ed6aa38a3ec6c01973b35f684586b6e1c0f7e

Thanks for sharing the script, this is very useful!

Note that [2] above is used, but optional:
https://lore.kernel.org/lkml/20200515150338.190344-5-elver@google.com/
It's not required for KCSAN to function correctly, but if it's
available it'll help find more data races with the default config.

Thanks,
-- Marco

^ permalink raw reply	[flat|nested] 11+ messages in thread

end of thread, other threads:[~2020-05-20  7:28 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-05-12 18:38 [PATCH] READ_ONCE, WRITE_ONCE, kcsan: Perform checks in __*_ONCE variants Marco Elver
2020-05-12 19:09 ` Peter Zijlstra
2020-05-19 21:10   ` Qian Cai
2020-05-19 21:25     ` Marco Elver
2020-05-19 21:45       ` Qian Cai
2020-05-19 22:05         ` Thomas Gleixner
2020-05-20  2:28           ` Qian Cai
2020-05-20  2:47             ` Nathan Chancellor
2020-05-20  3:16               ` Qian Cai
2020-05-20  3:44                 ` Nathan Chancellor
2020-05-20  7:28                   ` Marco Elver

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).