linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Marco Elver <elver@google.com>
To: Peter Zijlstra <peterz@infradead.org>
Cc: Will Deacon <will@kernel.org>,
	LKML <linux-kernel@vger.kernel.org>,
	Thomas Gleixner <tglx@linutronix.de>,
	"Paul E. McKenney" <paulmck@kernel.org>,
	Ingo Molnar <mingo@kernel.org>,
	Dmitry Vyukov <dvyukov@google.com>
Subject: Re: [PATCH v5 00/18] Rework READ_ONCE() to improve codegen
Date: Wed, 13 May 2020 13:48:41 +0200	[thread overview]
Message-ID: <CANpmjNMariz3-keqwUsLHVrpk2r7ThLSKtkhHxTDa3SEGeznhA@mail.gmail.com> (raw)
In-Reply-To: <20200513111057.GN2957@hirez.programming.kicks-ass.net>

On Wed, 13 May 2020 at 13:11, Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Tue, May 12, 2020 at 10:31:44PM +0200, Marco Elver wrote:
> > On Tue, 12 May 2020 at 21:08, Peter Zijlstra <peterz@infradead.org> wrote:
>
> > > data_race() will include active calls to kcsan_{dis,en}able_current(),
> > > and this must not happen.
> >
> > Only if instrumentation is enabled for the compilation unit. If you
> > have KCSAN_SANITIZE_foo.c := n, no calls are emitted not even to
> > kcsan_{dis,en}able_current(). Does that help?
> >
> > By default, right now __READ_ONCE() will still generate a call due to
> > instrumentation (call to __tsan_readX).
>
> Ah, so looking at:
>
> #define data_race(expr)                                                 \
> ({                                                                      \
>         __kcsan_disable_current();                                      \
>         ({                                                              \
>                 __unqual_scalar_typeof(({ expr; })) __v = ({ expr; });  \
>                 __kcsan_enable_current();                               \
>                 __v;                                                    \
>         });                                                             \
> })
>
> had me confused, but then you've got this squirreled away in another
> header:
>
> #ifdef __SANITIZE_THREAD__
> /*
>  * Only calls into the runtime when the particular compilation unit has KCSAN
>  * instrumentation enabled. May be used in header files.
>  */
> #define kcsan_check_access __kcsan_check_access
>
> /*
>  * Only use these to disable KCSAN for accesses in the current compilation unit;
>  * calls into libraries may still perform KCSAN checks.
>  */
> #define __kcsan_disable_current kcsan_disable_current
> #define __kcsan_enable_current kcsan_enable_current_nowarn
> #else
> static inline void kcsan_check_access(const volatile void *ptr, size_t size,
>                                       int type) { }
> static inline void __kcsan_enable_current(void)  { }
> static inline void __kcsan_disable_current(void) { }
> #endif
>
> And I suppose KCSAN_SANITIZE := n, results in __SANITIZE_THREAD__ not
> being defined.
>
> I really hate the function attribute situation, that is some ill
> considered trainwreck.
>
> Looking at this more, I found you already have:
>
> arch/x86/kernel/Makefile:KCSAN_SANITIZE := n
> arch/x86/kernel/Makefile:KCOV_INSTRUMENT                := n
> arch/x86/mm/Makefile:KCSAN_SANITIZE := n
>
> So how about I complete that and kill everhthing for all arch/x86/ that
> has DEFINE_IDTENTRY*() in.
>
> That avoids me having to do a lot of work to split up the tricky bits.
> You didn't think it was important, so why should I bother.
>
> So then I end up with something like the below, and I've validated that
> does not generate instrumentation... HOWEVER, I now need ~10g of memory
> and many seconds to compile each file in arch/x86/kernel/.
>
> That is, when I do 'make arch/x86/kernel/ -j8', it is slow enough that I
> can run top and grab:
>
> 31249 root      20   0 6128580   4.1g  13092 R 100.0  13.1   0:16.29 cc1
> 31278 root      20   0 6259456   4.4g  12932 R 100.0  13.9   0:16.27 cc1
> 31286 root      20   0 7243160   4.9g  13028 R 100.0  15.5   0:16.26 cc1
> 31289 root      20   0 5933824   4.0g  12936 R 100.0  12.8   0:16.26 cc1
> 31331 root      20   0 4250924   2.9g  13016 R 100.0   9.3   0:09.54 cc1
> 31346 root      20   0 1939552   1.3g  13028 R 100.0   4.1   0:07.01 cc1
> 31238 root      20   0 6293524   4.1g  13008 R 100.0  13.0   0:16.29 cc1
> 31259 root      20   0 6817076   4.7g  12956 R 100.0  14.9   0:16.27 cc1
>
> and it then triggers OOMs, while previously I could build kernels with
> -j80 on that machine:
>
> 31289 root      20   0   10.8g   6.2g    884 R 100.0  19.7   1:01.56 cc1
> 31249 root      20   0   10.2g   6.1g    484 R 100.0  19.3   1:00.10 cc1
> 31331 root      20   0   10.3g   7.2g    496 R 100.0  23.1   0:53.95 cc1
>
> Only 3 left, because the others OOM'ed.
>
> This is gcc-8.3, the situation with gcc-10 seems marginally better, but
> still atrocious.
>
> ---
> diff --git a/arch/x86/entry/Makefile b/arch/x86/entry/Makefile
> index b7a5790d8d63..ff959f0209e7 100644
> --- a/arch/x86/entry/Makefile
> +++ b/arch/x86/entry/Makefile
> @@ -6,6 +6,7 @@
>  KASAN_SANITIZE := n
>  UBSAN_SANITIZE := n
>  KCOV_INSTRUMENT := n
> +KCSAN_INSTRUMENT := n
>
>  CFLAGS_REMOVE_common.o = $(CC_FLAGS_FTRACE) -fstack-protector -fstack-protector-strong
>  CFLAGS_REMOVE_syscall_32.o = $(CC_FLAGS_FTRACE) -fstack-protector -fstack-protector-strong
> diff --git a/arch/x86/kernel/Makefile b/arch/x86/kernel/Makefile
> index d6d61c4455fa..f2a46a87026e 100644
> --- a/arch/x86/kernel/Makefile
> +++ b/arch/x86/kernel/Makefile
> @@ -22,15 +22,18 @@ CFLAGS_REMOVE_early_printk.o = -pg
>  CFLAGS_REMOVE_head64.o = -pg
>  endif
>
> -KASAN_SANITIZE_head$(BITS).o                           := n
> -KASAN_SANITIZE_dumpstack.o                             := n
> -KASAN_SANITIZE_dumpstack_$(BITS).o                     := n
> -KASAN_SANITIZE_stacktrace.o                            := n
> -KASAN_SANITIZE_paravirt.o                              := n
> -
> -# With some compiler versions the generated code results in boot hangs, caused
> -# by several compilation units. To be safe, disable all instrumentation.
> -KCSAN_SANITIZE := n
> +#
> +# You cannot instrument entry code, that results in definite problems.
> +# In particular, anything with DEFINE_IDTENTRY*() in must not have
> +# instrumentation on.
> +#
> +# If only function attributes and inlining would work properly, without
> +# that untangling this is a giant trainwreck, don't attempt.
> +#
> +KASAN_SANITIZE := n
> +UBSAN_SANITIZE := n
> +KCOV_INSTRUMENT := n
> +KCSAN_INSTRUMENT := n
>
>  OBJECT_FILES_NON_STANDARD_test_nx.o                    := y
>  OBJECT_FILES_NON_STANDARD_paravirt_patch.o             := y
> @@ -39,11 +42,6 @@ ifdef CONFIG_FRAME_POINTER
>  OBJECT_FILES_NON_STANDARD_ftrace_$(BITS).o             := y
>  endif
>
> -# If instrumentation of this dir is enabled, boot hangs during first second.
> -# Probably could be more selective here, but note that files related to irqs,
> -# boot, dumpstack/stacktrace, etc are either non-interesting or can lead to
> -# non-deterministic coverage.
> -KCOV_INSTRUMENT                := n
>
>  CFLAGS_irq.o := -I $(srctree)/$(src)/../include/asm/trace
>
> diff --git a/arch/x86/mm/Makefile b/arch/x86/mm/Makefile
> index f7fd0e868c9c..f8d7e7432847 100644
> --- a/arch/x86/mm/Makefile
> +++ b/arch/x86/mm/Makefile
> @@ -1,15 +1,17 @@
>  # SPDX-License-Identifier: GPL-2.0
> -# Kernel does not boot with instrumentation of tlb.c and mem_encrypt*.c
> -KCOV_INSTRUMENT_tlb.o                  := n
> -KCOV_INSTRUMENT_mem_encrypt.o          := n
> -KCOV_INSTRUMENT_mem_encrypt_identity.o := n
>
> -KASAN_SANITIZE_mem_encrypt.o           := n
> -KASAN_SANITIZE_mem_encrypt_identity.o  := n
> -
> -# Disable KCSAN entirely, because otherwise we get warnings that some functions
> -# reference __initdata sections.
> -KCSAN_SANITIZE := n
> +#
> +# You cannot instrument entry code, that results in definite problems.
> +# In particular, anything with DEFINE_IDTENTRY*() in must not have
> +# instrumentation on.
> +#
> +# If only function attributes and inlining would work properly, without
> +# that untangling this is a giant trainwreck, don't attempt.
> +#
> +KASAN_SANITIZE := n
> +UBSAN_SANITIZE := n
> +KCOV_INSTRUMENT := n
> +KCSAN_INSTRUMENT := n
>
>  ifdef CONFIG_FUNCTION_TRACER
>  CFLAGS_REMOVE_mem_encrypt.o            = -pg
> diff --git a/include/linux/compiler.h b/include/linux/compiler.h
> index 3bb962959d8b..48f85d1d2db6 100644
> --- a/include/linux/compiler.h
> +++ b/include/linux/compiler.h
> @@ -241,7 +241,7 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
>   * atomicity or dependency ordering guarantees. Note that this may result
>   * in tears!
>   */
> -#define __READ_ONCE(x) (*(const volatile __unqual_scalar_typeof(x) *)&(x))
> +#define __READ_ONCE(x) data_race((*(const volatile __unqual_scalar_typeof(x) *)&(x)))
>
>  #define __READ_ONCE_SCALAR(x)                                          \
>  ({                                                                     \
> @@ -260,7 +260,7 @@ void ftrace_likely_update(struct ftrace_likely_data *f, int val,
>
>  #define __WRITE_ONCE(x, val)                                           \
>  do {                                                                   \
> -       *(volatile typeof(x) *)&(x) = (val);                            \
> +       data_race(*(volatile typeof(x) *)&(x) = (val));                 \
>  } while (0)
>
>  #define __WRITE_ONCE_SCALAR(x, val)                                    \
>

Disabling most instrumentation for arch/x86 is reasonable. Also fine
with the __READ_ONCE/__WRITE_ONCE changes (your improved
compiler-friendlier version).

We likely can't have both: still instrument __READ_ONCE/__WRITE_ONCE
(as Will suggested) *and* avoid double-instrumentation in arch_atomic.
If most use-cases of __READ_ONCE/__WRITE_ONCE are likely to use
data_race() or KCSAN_SANITIZE := n anyway, I'd say it's reasonable for
now.

Thanks,
-- Marco

  parent reply	other threads:[~2020-05-13 11:48 UTC|newest]

Thread overview: 127+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-05-11 20:41 Will Deacon
2020-05-11 20:41 ` [PATCH v5 01/18] sparc32: mm: Fix argument checking in __srmmu_get_nocache() Will Deacon
2020-05-12 14:37   ` [tip: locking/kcsan] " tip-bot2 for Will Deacon
2020-05-11 20:41 ` [PATCH v5 02/18] sparc32: mm: Restructure sparc32 MMU page-table layout Will Deacon
2020-05-12 14:37   ` [tip: locking/kcsan] " tip-bot2 for Will Deacon
2020-05-11 20:41 ` [PATCH v5 03/18] sparc32: mm: Change pgtable_t type to pte_t * instead of struct page * Will Deacon
2020-05-12 14:36   ` [tip: locking/kcsan] " tip-bot2 for Will Deacon
2020-05-11 20:41 ` [PATCH v5 04/18] sparc32: mm: Reduce allocation size for PMD and PTE tables Will Deacon
2020-05-12 14:36   ` [tip: locking/kcsan] " tip-bot2 for Will Deacon
2020-05-17  0:00   ` [PATCH v5 04/18] " Guenter Roeck
2020-05-17  0:07     ` Guenter Roeck
2020-05-18  8:37       ` Will Deacon
2020-05-18  9:18         ` Mike Rapoport
2020-05-18  9:48         ` Guenter Roeck
2020-05-18 14:23           ` Mike Rapoport
2020-05-18 16:08             ` Guenter Roeck
2020-05-18 18:11               ` Ira Weiny
2020-05-18 18:14               ` Ira Weiny
2020-05-18 18:09             ` Guenter Roeck
2020-05-18 18:21               ` Ira Weiny
2020-05-18 19:15               ` Mike Rapoport
2020-05-19 16:40                 ` Guenter Roeck
2020-05-20 17:03         ` Mike Rapoport
2020-05-20 19:03           ` Guenter Roeck
2020-05-20 19:51             ` Mike Rapoport
2020-05-21 23:02               ` Guenter Roeck
2020-05-24 12:32                 ` Mike Rapoport
2020-05-24 14:01                   ` Guenter Roeck
2020-05-26 13:26                   ` Will Deacon
2020-05-26 14:01                     ` Will Deacon
2020-05-26 15:21                       ` Mike Rapoport
2020-05-26 16:18                       ` Guenter Roeck
2020-05-26 16:29                         ` Mike Rapoport
2020-05-26 17:15                           ` Guenter Roeck
2020-05-11 20:41 ` [PATCH v5 05/18] compiler/gcc: Raise minimum GCC version for kernel builds to 4.8 Will Deacon
2020-05-12 14:36   ` [tip: locking/kcsan] " tip-bot2 for Will Deacon
2020-05-11 20:41 ` [PATCH v5 06/18] netfilter: Avoid assigning 'const' pointer to non-const pointer Will Deacon
2020-05-12 14:36   ` [tip: locking/kcsan] " tip-bot2 for Will Deacon
2020-05-11 20:41 ` [PATCH v5 07/18] net: tls: " Will Deacon
2020-05-12 14:36   ` [tip: locking/kcsan] " tip-bot2 for Will Deacon
2020-05-11 20:41 ` [PATCH v5 08/18] fault_inject: Don't rely on "return value" from WRITE_ONCE() Will Deacon
2020-05-12 14:36   ` [tip: locking/kcsan] " tip-bot2 for Will Deacon
2020-05-11 20:41 ` [PATCH v5 09/18] arm64: csum: Disable KASAN for do_csum() Will Deacon
2020-05-12 14:36   ` [tip: locking/kcsan] " tip-bot2 for Will Deacon
2020-05-11 20:41 ` [PATCH v5 10/18] READ_ONCE: Simplify implementations of {READ,WRITE}_ONCE() Will Deacon
2020-05-12 14:36   ` [tip: locking/kcsan] " tip-bot2 for Will Deacon
2020-05-11 20:41 ` [PATCH v5 11/18] READ_ONCE: Enforce atomicity for {READ,WRITE}_ONCE() memory accesses Will Deacon
2020-05-12 14:36   ` [tip: locking/kcsan] " tip-bot2 for Will Deacon
2020-05-11 20:41 ` [PATCH v5 12/18] READ_ONCE: Drop pointer qualifiers when reading from scalar types Will Deacon
2020-05-12 14:36   ` [tip: locking/kcsan] " tip-bot2 for Will Deacon
2020-05-11 20:41 ` [PATCH v5 13/18] locking/barriers: Use '__unqual_scalar_typeof' for load-acquire macros Will Deacon
2020-05-12 14:36   ` [tip: locking/kcsan] " tip-bot2 for Will Deacon
2020-05-11 20:41 ` [PATCH v5 14/18] arm64: barrier: Use '__unqual_scalar_typeof' for acquire/release macros Will Deacon
2020-05-12 14:36   ` [tip: locking/kcsan] " tip-bot2 for Will Deacon
2020-05-11 20:41 ` [PATCH v5 15/18] gcov: Remove old GCC 3.4 support Will Deacon
2020-05-12 14:36   ` [tip: locking/kcsan] " tip-bot2 for Will Deacon
2020-05-11 20:41 ` [PATCH v5 16/18] kcsan: Rework data_race() so that it can be used by READ_ONCE() Will Deacon
2020-05-12 14:36   ` [tip: locking/kcsan] " tip-bot2 for Will Deacon
2020-05-11 20:41 ` [PATCH v5 17/18] READ_ONCE: Use data_race() to avoid KCSAN instrumentation Will Deacon
2020-05-12  8:23   ` Peter Zijlstra
2020-05-12  9:49     ` Will Deacon
2020-05-12 14:36   ` [tip: locking/kcsan] " tip-bot2 for Will Deacon
2020-05-20 22:17     ` Borislav Petkov
2020-05-20 22:30       ` Marco Elver
2020-05-21  7:25         ` Borislav Petkov
2020-05-21  9:37           ` Marco Elver
2020-05-21  3:30       ` Nathan Chancellor
2020-05-22 16:08       ` [tip: locking/kcsan] compiler.h: Avoid nested statement expression in data_race() tip-bot2 for Marco Elver
2020-05-11 20:41 ` [PATCH v5 18/18] linux/compiler.h: Remove redundant '#else' Will Deacon
2020-05-12 14:36   ` [tip: locking/kcsan] " tip-bot2 for Will Deacon
2020-05-12  8:18 ` [PATCH v5 00/18] Rework READ_ONCE() to improve codegen Peter Zijlstra
2020-05-12 17:53   ` Marco Elver
2020-05-12 18:55     ` Marco Elver
2020-05-12 19:07     ` Peter Zijlstra
2020-05-12 20:31       ` Marco Elver
2020-05-13 11:10         ` Peter Zijlstra
2020-05-13 11:14           ` Peter Zijlstra
2020-05-13 11:48           ` Marco Elver [this message]
2020-05-13 12:32             ` Peter Zijlstra
2020-05-13 12:40               ` Will Deacon
2020-05-13 13:15                 ` Marco Elver
2020-05-13 13:24                   ` Peter Zijlstra
2020-05-13 13:58                     ` Marco Elver
2020-05-14 11:21                       ` Peter Zijlstra
2020-05-14 11:24                         ` Peter Zijlstra
2020-05-14 11:35                         ` Peter Zijlstra
2020-05-14 12:01                         ` Will Deacon
2020-05-14 12:27                           ` Peter Zijlstra
2020-05-14 13:07                             ` Marco Elver
2020-05-14 13:14                               ` Peter Zijlstra
2020-05-14 12:20                         ` Peter Zijlstra
2020-05-14 14:13                       ` Peter Zijlstra
2020-05-14 14:20                         ` Marco Elver
2020-05-15  9:20                           ` Peter Zijlstra
2020-05-13 16:50                   ` Will Deacon
2020-05-13 17:32                     ` Marco Elver
2020-05-13 17:47                       ` Will Deacon
2020-05-13 18:54                         ` Marco Elver
2020-05-13 21:25                           ` Will Deacon
2020-05-14  7:31                             ` Marco Elver
2020-05-14 11:05                               ` Will Deacon
2020-05-14 13:35                                 ` Marco Elver
2020-05-14 13:47                                   ` Peter Zijlstra
2020-05-14 13:50                                   ` Peter Zijlstra
2020-05-14 13:56                                   ` Peter Zijlstra
2020-05-14 14:24                                   ` Peter Zijlstra
2020-05-14 15:09                                     ` Thomas Gleixner
2020-05-14 15:29                                       ` Marco Elver
2020-05-14 19:37                                         ` Thomas Gleixner
2020-05-15 13:55                                     ` David Laight
2020-05-15 14:04                                       ` Marco Elver
2020-05-15 14:07                                       ` Peter Zijlstra
2020-05-14 15:38                                   ` Paul E. McKenney
2020-05-22 16:08                                   ` [tip: locking/kcsan] kcsan: Restrict supported compilers tip-bot2 for Marco Elver
2020-06-03 18:52                                 ` [PATCH v5 00/18] Rework READ_ONCE() to improve codegen Borislav Petkov
2020-06-03 19:23                                   ` Marco Elver
2020-06-03 22:05                                     ` Borislav Petkov
2020-06-08 17:32                                     ` Martin Liška
2020-06-08 19:56                                       ` Marco Elver
2020-06-09 11:55                                         ` Martin Liška
2020-06-09 12:36                                           ` Martin Liška
2020-06-09 13:45                                             ` Marco Elver
2020-05-22 16:08                           ` [tip: locking/kcsan] kcsan: Remove 'noinline' from __no_kcsan_or_inline tip-bot2 for Marco Elver
2020-05-13 13:21                 ` [PATCH v5 00/18] Rework READ_ONCE() to improve codegen David Laight
2020-05-13 16:32                   ` Thomas Gleixner
2020-05-12 21:14       ` Will Deacon
2020-05-12 22:00         ` Marco Elver

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CANpmjNMariz3-keqwUsLHVrpk2r7ThLSKtkhHxTDa3SEGeznhA@mail.gmail.com \
    --to=elver@google.com \
    --cc=dvyukov@google.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mingo@kernel.org \
    --cc=paulmck@kernel.org \
    --cc=peterz@infradead.org \
    --cc=tglx@linutronix.de \
    --cc=will@kernel.org \
    --subject='Re: [PATCH v5 00/18] Rework READ_ONCE() to improve codegen' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is a public inbox, see mirroring instructions
on how to clone and mirror all data and code used for this inbox