From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.3 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54AB9C43461 for ; Fri, 16 Apr 2021 07:52:00 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 37AC760FF1 for ; Fri, 16 Apr 2021 07:52:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S239761AbhDPHwX (ORCPT ); Fri, 16 Apr 2021 03:52:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:39468 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S239747AbhDPHwW (ORCPT ); Fri, 16 Apr 2021 03:52:22 -0400 Received: from mail-qk1-x731.google.com (mail-qk1-x731.google.com [IPv6:2607:f8b0:4864:20::731]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 3145EC061574 for ; Fri, 16 Apr 2021 00:51:58 -0700 (PDT) Received: by mail-qk1-x731.google.com with SMTP id t80so767737qke.13 for ; Fri, 16 Apr 2021 00:51:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=BizjiGX7CK7NjPG3BovU7t24ZMIVaovvNxFJE+wsBXQ=; b=vI+aMtELxlNIzIJMPLcC4V1frvgUhkH01PvdlxpYg06klmvQYA8kuSK04/cL8O+eRS ZRHLP2IHb780+a8Ktl+bIwClqMU0OehGGsouVxVmaBMY3SffvyIS5lD/MrDasJerbJpM cIVzQKb9xrP9SHk6UAVNbIYTMi3bPL36arW6GfWfni31nPD2lpgl8IakVKULxpFNCEM1 F7NM5A2IOXY9x34QknLbjPXRIGMEMCEEaQR68wSEg1Kjxzq4kL4DinCio3HkUmRaHDbr /tluu2R2zZ14/piduklaqwLEgY59DmM+8uTMetfuSi4z1DhGre8OsH5nuKuPZxYOGpgo AMsw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=BizjiGX7CK7NjPG3BovU7t24ZMIVaovvNxFJE+wsBXQ=; b=a9rBBgBSCbiekvt9ucuESq5KQ2QmYm80SlB274gj19nHxXXobvvcqKSe1l/FsQylkK RjmF/6gQxhkNrBA/4CZLxI+AuWsyDum3ynmF6aO3uuSN+E6ptIkgTLm2eaKVfPBetZrQ SGo6CkBZAoocNft4DEoHAJ59pcyDa8zyICmm9Os74Z9q/3LmiXpSNUNo7MySU5ZO0C+u JSFL6x7MCIUEP9Oex58YbUy859e2/8eD1i6jjDRixfAGVx4sO8W7lzdO01ySW/fuaIRn bqEOw80FuQGcpaEPntSzWLYithryY2th0UcVG8hKYFiTysfVEoW/GyHkQ32yvZXCfBzC vBZg== X-Gm-Message-State: AOAM533SPCpT2MmBx/Po/2S/7MGx/5a/5mCGSX7PyD5Cwou4HEZ3Sr/f 28ymTkJLAHuQhSMNsPr5glbKfDWoW2nkG+KEh4hgmQ== X-Google-Smtp-Source: ABdhPJwEtiakJYwu3/9foJYpXSrj8LxrAzHvwF1aRhkw1e9k2vb8/qow2YEeRFwq+MdgQSvXF7YZSoZaigW7DwlgsdU= X-Received: by 2002:a37:4042:: with SMTP id n63mr3263745qka.501.1618559517123; Fri, 16 Apr 2021 00:51:57 -0700 (PDT) MIME-Version: 1.0 References: <000000000000ae236f05bfde0678@google.com> <20210413134147.54556d9d@gandalf.local.home> <20210413134314.16068eeb@gandalf.local.home> <20210413144009.6ed2feb8@gandalf.local.home> <20210413144335.4ff14cf2@gandalf.local.home> In-Reply-To: <20210413144335.4ff14cf2@gandalf.local.home> From: Dmitry Vyukov Date: Fri, 16 Apr 2021 09:51:45 +0200 Message-ID: Subject: Bisections with different bug manifestations To: Steven Rostedt Cc: syzkaller , syzbot , Andrew Morton , Jens Axboe , Borislav Petkov , "H. Peter Anvin" , Jim Mattson , Joerg Roedel , KVM list , linux-block , LKML , Mark Rutland , masahiroy@kernel.org, Ingo Molnar , Paolo Bonzini , Peter Zijlstra , rafael.j.wysocki@intel.com, Sean Christopherson , syzkaller-bugs , Thomas Gleixner , Vitaly Kuznetsov , Wanpeng Li , Will Deacon , "the arch/x86 maintainers" Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-block@vger.kernel.org On Tue, Apr 13, 2021 at 8:43 PM Steven Rostedt wrote: > > On Tue, 13 Apr 2021 14:40:09 -0400 > Steven Rostedt wrote: > > > ------------[ cut here ]------------ > > raw_local_irq_restore() called with IRQs enabled > > WARNING: CPU: 0 PID: 8777 at kernel/locking/irqflag-debug.c:9 warn_bogus_irq_restore kernel/locking/irqflag-debug.c:9 [inline] > > WARNING: CPU: 0 PID: 8777 at kernel/locking/irqflag-debug.c:9 warn_bogus_irq_restore+0x1d/0x20 kernel/locking/irqflag-debug.c:7 > > In fact, when you have the above, which is a WARN() with text: > > "raw_local_irq_restore() called with IRQs enabled" > > It is pretty much guaranteed that all triggers of this bug will have the > above warning with the same text. This looks nice on paper and looking at only 1 bisection log. But unfortunately in practice theory and practice are different... This was discussed at length multiple times: https://groups.google.com/g/syzkaller/search?q=bisection+different+manifestations https://groups.google.com/g/syzkaller-bugs/c/nFeC8-UG1gg/m/y6gUEsvAAgAJ https://groups.google.com/g/syzkaller/c/sR8aAXaWEF4/m/tTWYRgvmAwAJ https://groups.google.com/g/syzkaller/c/9NdprHsGBqo/m/Yj9uWRDgBQAJ If you look at substantial base of bisection logs, you will find lots of cases where bug types, functions don't match. Kernel crashes differently even on the same revision. And obviously things change if you change revisions. Also if you see presumably a different bug, what does it say regarding the original bug. I would very much like to improve automatic bisection quality, but it does not look trivial at all. Some random examples where, say, your hypothesis of WARN-to-WARN, BUG-to-BUG does not hold even on the same kernel revision (add to this different revisions and the fact that a different bug does not give info regarding the original bug): run #0: crashed: KASAN: use-after-free Read in fuse_dev_do_read run #1: crashed: WARNING in request_end run #2: crashed: KASAN: use-after-free Read in fuse_dev_do_read run #3: OK run #4: OK run #0: crashed: KASAN: slab-out-of-bounds Read in __ip_append_data run #1: crashed: inconsistent lock state in rhashtable_walk_enter run #2: crashed: inconsistent lock state in rhashtable_walk_enter run #3: crashed: inconsistent lock state in rhashtable_walk_enter run #4: crashed: inconsistent lock state in rhashtable_walk_enter run #5: crashed: inconsistent lock state in rhashtable_walk_enter run #6: crashed: inconsistent lock state in rhashtable_walk_enter run #7: crashed: kernel BUG at arch/x86/mm/physaddr.c:LINE! run #8: crashed: inconsistent lock state in rhashtable_walk_enter run #9: crashed: inconsistent lock state in rhashtable_walk_enter run #0: crashed: kernel BUG at arch/x86/mm/physaddr.c:LINE! run #1: crashed: inconsistent lock state in rhashtable_walk_enter run #2: crashed: inconsistent lock state in rhashtable_walk_enter run #3: crashed: inconsistent lock state in rhashtable_walk_enter run #4: crashed: inconsistent lock state in rhashtable_walk_enter run #5: crashed: kernel BUG at arch/x86/mm/physaddr.c:LINE! run #6: crashed: inconsistent lock state in rhashtable_walk_enter run #7: crashed: kernel BUG at arch/x86/mm/physaddr.c:LINE! run #8: crashed: inconsistent lock state in rhashtable_walk_enter run #9: crashed: inconsistent lock state in rhashtable_walk_enter run #0: crashed: KASAN: use-after-free Read in __vb2_perform_fileio run #1: crashed: KASAN: use-after-free Write in __vb2_cleanup_fileio run #2: crashed: KASAN: use-after-free Read in __vb2_perform_fileio run #3: crashed: KASAN: use-after-free Read in __vb2_perform_fileio run #4: crashed: INFO: task hung in vivid_stop_generating_vid_cap run #5: crashed: INFO: task hung in vivid_stop_generating_vid_cap run #6: crashed: INFO: task hung in vivid_stop_generating_vid_cap run #7: crashed: INFO: task hung in vivid_stop_generating_vid_cap run #8: crashed: INFO: task hung in vivid_stop_generating_vid_cap run #9: crashed: INFO: task hung in vivid_stop_generating_vid_cap run #0: crashed: general protection fault in sctp_assoc_rwnd_increase run #1: crashed: general protection fault in sctp_ulpevent_free run #2: crashed: general protection fault in sctp_assoc_rwnd_increase run #3: crashed: general protection fault in sctp_assoc_rwnd_increase run #4: crashed: general protection fault in sctp_assoc_rwnd_increase run #5: crashed: general protection fault in sctp_assoc_rwnd_increase run #6: crashed: general protection fault in sctp_assoc_rwnd_increase run #7: crashed: general protection fault in sctp_assoc_rwnd_increase run #0: crashed: general protection fault in sctp_assoc_rwnd_increase run #1: crashed: general protection fault in sctp_assoc_rwnd_increase run #2: crashed: general protection fault in sctp_assoc_rwnd_increase run #3: crashed: general protection fault in sctp_assoc_rwnd_increase run #4: crashed: general protection fault in corrupted run #5: crashed: general protection fault in sctp_assoc_rwnd_increase run #6: crashed: general protection fault in sctp_assoc_rwnd_increase run #7: crashed: general protection fault in corrupted run #0: crashed: INFO: rcu detected stall in corrupted run #1: crashed: INFO: rcu detected stall in ext4_file_write_iter run #2: crashed: INFO: rcu detected stall in sys_sendfile64 run #3: crashed: INFO: rcu detected stall in corrupted run #4: crashed: INFO: rcu detected stall in ext4_file_write_iter run #5: crashed: INFO: rcu detected stall in corrupted run #6: crashed: INFO: rcu detected stall in corrupted run #7: crashed: INFO: rcu detected stall in ext4_file_write_iter run #8: crashed: INFO: rcu detected stall in sys_sendfile64 run #9: crashed: INFO: rcu detected stall in ext4_file_write_iter run #0: crashed: INFO: rcu detected stall in sys_sendfile64 run #1: crashed: INFO: rcu detected stall in corrupted run #2: crashed: INFO: rcu detected stall in corrupted run #3: crashed: INFO: rcu detected stall in sys_sendfile64 run #4: crashed: INFO: rcu detected stall in corrupted run #5: crashed: INFO: rcu detected stall in corrupted run #6: crashed: INFO: rcu detected stall in corrupted run #7: crashed: INFO: rcu detected stall in corrupted run #8: crashed: INFO: rcu detected stall in sys_sendfile64 run #9: crashed: INFO: rcu detected stall in corrupted run #0: crashed: INFO: rcu detected stall in rw_verify_area run #1: crashed: INFO: rcu detected stall in ext4_file_write_iter run #2: crashed: INFO: rcu detected stall in corrupted run #3: crashed: INFO: rcu detected stall in corrupted run #4: crashed: INFO: rcu detected stall in ext4_file_write_iter run #5: crashed: INFO: rcu detected stall in ext4_file_write_iter run #6: crashed: INFO: rcu detected stall in corrupted run #7: crashed: INFO: rcu detected stall in ext4_file_write_iter run #8: crashed: INFO: rcu detected stall in ext4_file_write_iter run #9: crashed: INFO: rcu detected stall in rw_verify_area run #0: crashed: INFO: rcu detected stall in ext4_file_write_iter run #1: crashed: INFO: rcu detected stall in corrupted run #2: crashed: INFO: rcu detected stall in sys_sendfile64 run #3: crashed: INFO: rcu detected stall in sys_sendfile64 run #4: crashed: INFO: rcu detected stall in corrupted run #5: crashed: INFO: rcu detected stall in sys_sendfile64 run #6: crashed: INFO: rcu detected stall in sys_sendfile64 run #7: crashed: INFO: rcu detected stall in corrupted run #8: crashed: INFO: rcu detected stall in corrupted run #9: crashed: INFO: rcu detected stall in sys_sendfile64 run #0: crashed: KASAN: use-after-free Read in link_path_walk run #1: crashed: KASAN: use-after-free Read in link_path_walk run #2: crashed: KASAN: use-after-free Read in trailing_symlink run #3: crashed: KASAN: use-after-free Read in trailing_symlink run #4: crashed: KASAN: use-after-free Read in trailing_symlink run #5: crashed: KASAN: use-after-free Read in link_path_walk run #6: crashed: KASAN: use-after-free Read in link_path_walk run #7: crashed: KASAN: use-after-free Read in link_path_walk run #8: crashed: KASAN: use-after-free Read in trailing_symlink run #9: crashed: KASAN: use-after-free Read in trailing_symlink run #0: crashed: BUG: unable to handle kernel NULL pointer dereference in mrvl_setup run #1: crashed: BUG: unable to handle kernel NULL pointer dereference in mrvl_setup run #2: crashed: BUG: unable to handle kernel NULL pointer dereference in mrvl_setup run #3: crashed: BUG: unable to handle kernel NULL pointer dereference in mrvl_setup run #4: crashed: WARNING: ODEBUG bug in corrupted run #5: crashed: BUG: unable to handle kernel NULL pointer dereference in mrvl_setup run #6: crashed: BUG: unable to handle kernel NULL pointer dereference in mrvl_setup run #7: crashed: BUG: unable to handle kernel NULL pointer dereference in mrvl_setup run #8: crashed: BUG: unable to handle kernel NULL pointer dereference in mrvl_setup run #9: crashed: BUG: unable to handle kernel NULL pointer dereference in mrvl_setup run #0: crashed: KASAN: use-after-free Read in delayed_uprobe_remove run #1: crashed: KASAN: use-after-free Read in delayed_uprobe_remove run #2: crashed: general protection fault in delayed_uprobe_remove run #3: crashed: KASAN: use-after-free Read in delayed_uprobe_remove run #4: crashed: general protection fault in delayed_uprobe_remove run #5: crashed: KASAN: use-after-free Read in delayed_uprobe_remove run #6: OK run #7: OK run #8: OK run #9: OK run #0: crashed: general protection fault in delayed_uprobe_remove run #1: crashed: KASAN: use-after-free Read in delayed_uprobe_remove run #2: crashed: KASAN: use-after-free Read in delayed_uprobe_remove run #3: crashed: KASAN: use-after-free Read in delayed_uprobe_remove run #4: crashed: general protection fault in delayed_uprobe_remove run #5: crashed: KASAN: use-after-free Read in delayed_uprobe_remove run #6: OK run #7: OK run #8: OK run #9: OK