Linux-EFI Archive on lore.kernel.org
 help / color / Atom feed
From: Marco Elver <elver@google.com>
To: Qian Cai <cai@lca.pw>
Cc: LKMM Maintainers -- Akira Yokosawa <akiyks@gmail.com>,
	Alan Stern <stern@rowland.harvard.edu>,
	Alexander Potapenko <glider@google.com>,
	Andrea Parri <parri.andrea@gmail.com>,
	Andrey Konovalov <andreyknvl@google.com>,
	Andy Lutomirski <luto@kernel.org>,
	Ard Biesheuvel <ard.biesheuvel@linaro.org>,
	Arnd Bergmann <arnd@arndb.de>, Boqun Feng <boqun.feng@gmail.com>,
	Borislav Petkov <bp@alien8.de>, Daniel Axtens <dja@axtens.net>,
	Daniel Lustig <dlustig@nvidia.com>,
	Dave Hansen <dave.hansen@linux.intel.com>,
	David Howells <dhowells@redhat.com>,
	Dmitry Vyukov <dvyukov@google.com>,
	"H. Peter Anvin" <hpa@zytor.com>, Ingo Molnar <mingo@redhat.com>,
	Jade Alglave <j.alglave@ucl.ac.uk>,
	Joel Fernandes <joel@joelfernandes.org>,
	Jonathan Corbet <corbet@lwn.net>,
	Josh Poimboeuf <jpoimboe@redhat.com>,
	Luc Maranget <luc.maranget@inria.fr>,
	Mark Rutland <mark.rutland@arm.com>,
	Nicholas Piggin <npiggin@gmail.com>,
	"Paul E. McKenney" <paulmck@kernel.org>,
	Peter Zijlstra <peterz@infradead.org>,
	Thomas Gleixner <tglx@linutronix.de>,
	Will Deacon <will@kernel.org>, Eric Dumazet <edumazet@google.com>,
	kasan-dev <kasan-dev@googlegroups.com>,
	linux-arch <linux-arch@vger.kernel.org>,
	"open list:DOCUMENTATION" <linux-doc@vger.kernel.org>,
	linux-efi@vger.kernel.org,
	Linux Kbuild mailing list <linux-kbuild@vger.kernel.org>,
	LKML <linux-kernel@vger.kernel.org>,
	Linux Memory Management List <linux-mm@kvack.org>,
	"the arch/x86 maintainers" <x86@kernel.org>
Subject: Re: [PATCH v4 01/10] kcsan: Add Kernel Concurrency Sanitizer infrastructure
Date: Mon, 6 Jan 2020 13:46:57 +0100
Message-ID: <CANpmjNOC2PYFsE_TK2SYmKcHxyG+2arWc8x_fmeWPOMi0+ot8g@mail.gmail.com> (raw)
In-Reply-To: <BAB5F853-95FA-4623-A067-4E62B90721D3@lca.pw>

On Fri, 3 Jan 2020 at 06:13, Qian Cai <cai@lca.pw> wrote:
>
>
>
> > On Nov 14, 2019, at 1:02 PM, 'Marco Elver' via kasan-dev <kasan-dev@googlegroups.com> wrote:
> > +static noinline void kcsan_setup_watchpoint(const volatile void *ptr,
> > +                                         size_t size, bool is_write)
> > +{
> > +     atomic_long_t *watchpoint;
> > +     union {
> > +             u8 _1;
> > +             u16 _2;
> > +             u32 _4;
> > +             u64 _8;
> > +     } expect_value;
> > +     bool value_change = false;
> > +     unsigned long ua_flags = user_access_save();
> > +     unsigned long irq_flags;
> > +
> > +     /*
> > +      * Always reset kcsan_skip counter in slow-path to avoid underflow; see
> > +      * should_watch().
> > +      */
> > +     reset_kcsan_skip();
> > +
> > +     if (!kcsan_is_enabled())
> > +             goto out;
> > +
> > +     if (!check_encodable((unsigned long)ptr, size)) {
> > +             kcsan_counter_inc(KCSAN_COUNTER_UNENCODABLE_ACCESSES);
> > +             goto out;
> > +     }
> > +
> > +     /*
> > +      * Disable interrupts & preemptions to avoid another thread on the same
> > +      * CPU accessing memory locations for the set up watchpoint; this is to
> > +      * avoid reporting races to e.g. CPU-local data.
> > +      *
> > +      * An alternative would be adding the source CPU to the watchpoint
> > +      * encoding, and checking that watchpoint-CPU != this-CPU. There are
> > +      * several problems with this:
> > +      *   1. we should avoid stealing more bits from the watchpoint encoding
> > +      *      as it would affect accuracy, as well as increase performance
> > +      *      overhead in the fast-path;
> > +      *   2. if we are preempted, but there *is* a genuine data race, we
> > +      *      would *not* report it -- since this is the common case (vs.
> > +      *      CPU-local data accesses), it makes more sense (from a data race
> > +      *      detection point of view) to simply disable preemptions to ensure
> > +      *      as many tasks as possible run on other CPUs.
> > +      */
> > +     local_irq_save(irq_flags);
>
> Enabling KCSAN will now generate a warning during boot here.
>
> Config (need to deselect KASAN and select KCSAN):
>
> https://raw.githubusercontent.com/cailca/linux-mm/master/x86.config

Thanks, I'll look into KCSAN + lockdep compatibility. It's probably
missing some KCSAN_SANITIZE := n in some Makefile.

> [   13.358813][    T0] Spectre V2 : Spectre v2 / SpectreRSB mitigation: Filling RSB on context switch
> [   13.361606][    T0] Speculative Store Bypass: Vulnerable
> [   13.363254][    T0] TAA: Vulnerable: Clear CPU buffers attempted, no microcode
> [   13.366836][    T0] MDS: Vulnerable: Clear CPU buffers attempted, no microcode
> [   13.369877][    T0] debug: unmapping init [mem 0xffffffff8dd83000-0xffffffff8dd87fff]
> [   13.415028][    T1] ------------[ cut here ]------------
> [   13.416814][    T1] DEBUG_LOCKS_WARN_ON(!current->hardirqs_enabled)
> [   13.416814][    T1] WARNING: CPU: 0 PID: 1 at kernel/locking/lockdep.c:4406 check_flags.part.26+0x102/0x240
> [   13.416814][    T1] Modules linked in:
> [   13.416814][    T1] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.5.0-rc2-next-20191220+ #4
> [   13.416814][    T1] Hardware name: HP ProLiant DL580 Gen9/ProLiant DL580 Gen9, BIOS U17 07/21/2016
> [   13.416814][    T1] RIP: 0010:check_flags.part.26+0x102/0x240
> [   13.416814][    T1] Code: bc 8d e8 51 a1 15 00 44 8b 05 2a a0 46 01 45 85 c0 0f 85 57 76 00 00 48 c7 c6 5d fa 7b 8d 48 c7 c7 b1 54 7b 8d e8 10 91 f5 ff <0f> 0b e9 3d 76 00 00 65 48 8b 3c 25 40 7f 01 00 e8 89 f0 ff ff e8
> [   13.416814][    T1] RSP: 0000:ffff9d3206287ce8 EFLAGS: 00010082
> [   13.416814][    T1] RAX: 0000000000000000 RBX: ffff8e5b8541e040 RCX: 0000000000000000
> [   13.416814][    T1] RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000
> [   13.416814][    T1] RBP: ffff9d3206287cf0 R08: 0000000000000000 R09: 0000ffff8dbcc254
> [   13.416814][    T1] R10: 0000ffffffffffff R11: 0000ffff8dbcc257 R12: 0000000000000235
> [   13.416814][    T1] R13: 0000000000000000 R14: 0000000000000246 R15: 000000000000001b
> [   13.416814][    T1] FS:  0000000000000000(0000) GS:ffff8e61e3200000(0000) knlGS:0000000000000000
> [   13.416814][    T1] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [   13.416814][    T1] CR2: ffff8e79f07ff000 CR3: 0000001284c0e001 CR4: 00000000003606f0
> [   13.416814][    T1] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> [   13.416814][    T1] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> [   13.416814][    T1] Call Trace:
> [   13.416814][    T1]  lock_is_held_type+0x66/0x160
> [   13.416814][    T1]  ___might_sleep+0xc1/0x1d0
> [   13.416814][    T1]  __might_sleep+0x5b/0xa0
> [   13.416814][    T1]  slab_pre_alloc_hook+0x7b/0xa0
> [   13.416814][    T1]  __kmalloc_node+0x60/0x300
> [   13.416814   T1]  ? alloc_cpumask_var_node+0x44/0x70
> [   13.416814][    T1]  ? topology_phys_to_logical_die+0x7e/0x180
> [   13.416814][    T1]  alloc_cpumask_var_node+0x44/0x70
> [   13.416814][    T1]  zalloc_cpumask_var+0x2a/0x40
> [   13.416814][    T1]  native_smp_prepare_cpus+0x246/0x425
> [   13.416814][    T1]  kernel_init_freeable+0x1b8/0x496
> [   13.416814][    T1]  ? rest_init+0x381/0x381
> [   13.416814][    T1]  kernel_init+0x18/0x17f
> [   13.416814][    T1]  ? rest_init+0x381/0x381
> [   13.416814][    T1]  ret_from_fork+0x3a/0x50
> [   13.416814][    T1] irq event stamp: 910
> [   13.416814][    T1] hardirqs last  enabled at (909): [<ffffffff8d1240f3>] _raw_write_unlock_irqrestore+0x53/0x57
> [   13.416814][    T1] hardirqs last disabled at (910): [<ffffffff8c8bba76>] kcsan_setup_watchpoint+0x96/0x460
> [   13.416814][    T1] softirqs last  enabled at (0): [<ffffffff8c6b697a>] copy_process+0x11fa/0x34f0
> [   13.416814][    T1] softirqs last disabled at (0): [<0000000000000000>] 0x0
> [   13.416814][    T1] ---[ end trace 7d1df66da055aa92 ]---
> [   13.416814][    T1] possible reason: unannotated irqs-on.
> [   13.416814][ent stamp: 910
> [   13.416814][    T1] hardirqs last  enabled at (909): [<ffffffff8d1240f3>] _raw_write_unlock_irqrestore+0x53/0x57
> [   13.416814][    T1] hardirqs last disabled at (910): [<ffffffff8c8bba76>] kcsan_setup_watchpoint+0x96/0x460
> [   13.416814][    T1] softirqs last  enabled at (0): [<ffffffff8c6b697a>] copy_process+0x11fa/0x34f0
> [   13.416814][    T1] softirqs last disabled at (0): [<0000000000000000>] 0x0
>
>
> The other issue is that the system is unable to boot due to endless of those messages.

Apart from fixing the data races, I can add a feature to KCSAN to
limit reporting too often (will send patch).

> [   17.976814][  T578] Reported by Kernel Concurrency Sanitizer on:
> [   17.976814][  T578] CPU: 12 PID: 578 Comm: pgdatinit1 Tainted: G        W         5.5.0-rc2-next-20191220+ #4
> [   17.976814][  T578] Hardware name: HP ProLiant DL580 Gen9/ProLiant DL580 Gen9, BIOS U17 07/21/2016
> [   17.976814][  T578] ==================================================================
> [   17.976814][  T578] ==================================================================
> [   17.976814][  T578] BUG: KCSAN: data-race in __change_page_attr / __change_page_attr
> [   17.976814][  T578]
> [   17.976814][  T578] write to 0xffffffff8dda0de0 of 8 bytes by task 577 on cpu 2:
> [   17.976814][  T578]  __change_page_attr+0xef7/0x16a0
> [   17.976814][  T578]  __change_page_attr_set_clr+0xec/0x4f0
> [   17.97681pages_np+0xcc/0x100
> [   17.976814][  T578]  __kernel_map_pages+0xd6/0xdb
> [   17.976814][  T578]  __free_pages_ok+0x1a8/0x730
> [   17.976814][  T578]  __free_pages+0x51/0x90
> [   17.976814][  T578]  __free_pages_core+0x1c7/0x2c0
> [   17.976814][  T578]  deferred_free_range+0x59/0x8f
> [   17.976814][  T578]  deferred_init_maxorder+0x1d6/0x21d
> [   17.976814][  T578]  deferred_init_memmap+0x14a/0x1c1
> [   17.976814][  T578]  kthread+0x1e0/0x200
> [   17.976814][  T578]  ret_from_fork+0x3a/0x50
> [   17.976814][  T578]
> [   17.976814][  T578] read to 0xffffffff8dda0de0 of 8 bytes by task 578 on cpu 12:
> [   17.976814][  T578]  __change_page_attr+0xed1/0x16a0
> [   17.976814][  T578]  __change_page_attr_set_clr+0xec/0x4f0
> [   17.976814][  T578]  __set_pages_np+0xcc/0x100
> [   17.976814][  T578]  __kernel_map_pages+0xd6/0xdb
> [   17.976814][  T578]  __free_pages_ok+0x1a8/0x730
> [   17.976814][  T578]  __free_pages+0x51/0x90
> [   17.976814][  T578]  __free_pages_core+0x1c7/0x2c0
> [   17.976814][  T578]  deferred_free_range+0x59/0x8f
> [   17.976814][  T578]  deferred_init_maxorder+0x1aa/0x21d
> [   17.976814][  T578]  deferred_init_memmap+0x14a/0x1c1
> [   17.976814][  T578]  kthread+0x1e0/0x200
> [   17.976814][  T578]  ret_from_fork+0x3a/0x50
>
> # ./scripts/faddr2line vmlinux __change_page_attr+0xef7/0x16a0
> __change_page_attr+0xef7/0x16a0:
> static_protections at arch/x86/mm/pat/set_memory.c:528
> (inlined by) __change_page_attr at arch/x86/mm/pat/set_memory.c:1516
>
> # ./scripts/faddr2line vmlinux __change_page_attr+0xed1/0x16a0
> __change_page_attr+0xed1/0x16a0:
> cpa_inc_4k_install at arch/x86/mm/pat/set_memory.c:131
> (inlined by) __change_page_attr at arch/x86/mm/pat/set_memory.c:1514

Thanks,
-- Marco

  reply index

Thread overview: 40+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-11-14 18:02 [PATCH v4 00/10] Add Kernel Concurrency Sanitizer (KCSAN) Marco Elver
2019-11-14 18:02 ` [PATCH v4 01/10] kcsan: Add Kernel Concurrency Sanitizer infrastructure Marco Elver
2019-11-19 19:27   ` Qian Cai
2019-11-19 19:54     ` Marco Elver
2019-11-19 21:42       ` Qian Cai
2019-11-19 21:53         ` Marco Elver
2020-01-03  5:13   ` Qian Cai
2020-01-06 12:46     ` Marco Elver [this message]
2020-01-14 11:08       ` Qian Cai
2020-01-14 12:51         ` Marco Elver
2020-01-14 19:22         ` Paul E. McKenney
2020-01-14 20:30           ` Qian Cai
2020-01-14 21:34             ` Paul E. McKenney
2020-01-14 21:48               ` Qian Cai
2020-01-14 22:09                 ` Paul E. McKenney
2019-11-14 18:02 ` [PATCH v4 02/10] include/linux/compiler.h: Introduce data_race(expr) macro Marco Elver
2019-11-14 18:02 ` [PATCH v4 03/10] kcsan: Add Documentation entry in dev-tools Marco Elver
2019-11-14 18:02 ` [PATCH v4 04/10] objtool, kcsan: Add KCSAN runtime functions to whitelist Marco Elver
2019-11-14 18:02 ` [PATCH v4 05/10] build, kcsan: Add KCSAN build exceptions Marco Elver
2019-11-14 18:02 ` [PATCH v4 06/10] seqlock, kcsan: Add annotations for KCSAN Marco Elver
2019-11-14 18:03 ` [PATCH v4 07/10] seqlock: Require WRITE_ONCE surrounding raw_seqcount_barrier Marco Elver
2019-11-14 18:03 ` [PATCH v4 08/10] asm-generic, kcsan: Add KCSAN instrumentation for bitops Marco Elver
2019-11-15 11:55   ` Marco Elver
2019-11-14 18:03 ` [PATCH v4 09/10] locking/atomics, kcsan: Add KCSAN instrumentation Marco Elver
2019-11-14 18:03 ` [PATCH v4 10/10] x86, kcsan: Enable KCSAN for x86 Marco Elver
2019-11-14 19:50 ` [PATCH v4 00/10] Add Kernel Concurrency Sanitizer (KCSAN) Paul E. McKenney
2019-11-14 21:33   ` Marco Elver
2019-11-14 22:15     ` Paul E. McKenney
2019-11-15 12:02       ` Marco Elver
2019-11-15 16:41         ` Paul E. McKenney
2019-11-15 17:14           ` Marco Elver
2019-11-15 20:43             ` Paul E. McKenney
2019-11-16  8:20               ` Marco Elver
2019-11-16 15:34                 ` Paul E. McKenney
2019-11-16 18:09                   ` Marco Elver
2019-11-16 18:28                     ` Paul E. McKenney
2019-11-19 19:50 ` Qian Cai
2019-11-19 20:12 ` Qian Cai
2019-11-19 21:50   ` Marco Elver
2019-11-20 15:54     ` Marco Elver

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CANpmjNOC2PYFsE_TK2SYmKcHxyG+2arWc8x_fmeWPOMi0+ot8g@mail.gmail.com \
    --to=elver@google.com \
    --cc=akiyks@gmail.com \
    --cc=andreyknvl@google.com \
    --cc=ard.biesheuvel@linaro.org \
    --cc=arnd@arndb.de \
    --cc=boqun.feng@gmail.com \
    --cc=bp@alien8.de \
    --cc=cai@lca.pw \
    --cc=corbet@lwn.net \
    --cc=dave.hansen@linux.intel.com \
    --cc=dhowells@redhat.com \
    --cc=dja@axtens.net \
    --cc=dlustig@nvidia.com \
    --cc=dvyukov@google.com \
    --cc=edumazet@google.com \
    --cc=glider@google.com \
    --cc=hpa@zytor.com \
    --cc=j.alglave@ucl.ac.uk \
    --cc=joel@joelfernandes.org \
    --cc=jpoimboe@redhat.com \
    --cc=kasan-dev@googlegroups.com \
    --cc=linux-arch@vger.kernel.org \
    --cc=linux-doc@vger.kernel.org \
    --cc=linux-efi@vger.kernel.org \
    --cc=linux-kbuild@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=luc.maranget@inria.fr \
    --cc=luto@kernel.org \
    --cc=mark.rutland@arm.com \
    --cc=mingo@redhat.com \
    --cc=npiggin@gmail.com \
    --cc=parri.andrea@gmail.com \
    --cc=paulmck@kernel.org \
    --cc=peterz@infradead.org \
    --cc=stern@rowland.harvard.edu \
    --cc=tglx@linutronix.de \
    --cc=will@kernel.org \
    --cc=x86@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Linux-EFI Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/linux-efi/0 linux-efi/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 linux-efi linux-efi/ https://lore.kernel.org/linux-efi \
		linux-efi@vger.kernel.org
	public-inbox-index linux-efi

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.linux-efi


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git