From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.2 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_IN_DEF_DKIM_WL autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CE38FC433E0 for ; Thu, 11 Feb 2021 13:39:56 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id 888D064E87 for ; Thu, 11 Feb 2021 13:39:56 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231769AbhBKNjt (ORCPT ); Thu, 11 Feb 2021 08:39:49 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53144 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S230242AbhBKNTF (ORCPT ); Thu, 11 Feb 2021 08:19:05 -0500 Received: from mail-qk1-x72d.google.com (mail-qk1-x72d.google.com [IPv6:2607:f8b0:4864:20::72d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 2A444C061756 for ; Thu, 11 Feb 2021 05:18:23 -0800 (PST) Received: by mail-qk1-x72d.google.com with SMTP id w19so4590576qki.13 for ; Thu, 11 Feb 2021 05:18:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=TNbrNGEpNHvneHu+xk19nOO78YSnguGzc+qrUt8Ns9I=; b=ACfPd8VaxtXZss6ecIfSTRn1q0RQ50rMA9La8hOx9/ojVo7ooRxL/3afPAct9nxVkR ALypNL9GdLhiOKikzl3blb5AqkzlaSjvbNg/0Lcnc79VBGyEciT2KLTprEFdaJvz+u/Q TfrShJIW6WjQ0ympnQ3AIpnUxBJqja46lebBO6j0zYWK+4tOXis4AAHRWGdvoLE6m5DN mva5hVY43HnDlYh7NLmxtb10P5dPAYB7DNQGzCHl8fBoWc/id6xeRCuC5d08nQHze07y g9iJe8/MIrGXuYbDy08DQG92v5Xrpb7kdb+KCvz2z/gBE197KxwIxaqvFlrqifTxiHmv LUbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=TNbrNGEpNHvneHu+xk19nOO78YSnguGzc+qrUt8Ns9I=; b=e9jnSp0HPpfAD6Zn/VbQmrosLwEZ1HIc44QVU2tWRJ5etFJeFf9ek9uIncPyFS18YB VJNSQpeHnUBKM2GjopqGBLcaF3HW7mCmiaoc5SrV6WpM9lkKY33h5DGCS3+NgK9V9nA3 R3k8on7h6lSFslx8zTTDN5E5n9FCsCHtLccOoLpWbp+g7TqdfNKUEsYjqXVkWqZdIBH0 bw+lMAzDvSovQsBvIgkl4rseFd6LHol0EtXc+kFVeL7kEDNaiHKDvy6kkWJSqoP6uytg 2UQJtJrKYWWyA7ErL7LoIf0ZVp6p2NjHSojZhrFqm5Ird5JJFCKuUbiQtvt1IsZWPbSn Qzjw== X-Gm-Message-State: AOAM532K1KVhc8rDYshFLIVJP8y8SDN40atiqNZTAnjr2wTA8qDWfdk+ cj3hRo5JaXqw9TA9+2fp9SDAls9iTepL5mGu8OqdsQ== X-Google-Smtp-Source: ABdhPJxA6pzkZq0hAk0lQqeUPpSmGOTEnqyjDG6MHORn1Z6d6T5PVA9k6A2/h2zgWi0Z+OCZSulMRKjO+Rbx/+6EILM= X-Received: by 2002:a05:620a:49:: with SMTP id t9mr8683728qkt.231.1613049502035; Thu, 11 Feb 2021 05:18:22 -0800 (PST) MIME-Version: 1.0 References: <000000000000563a0205bafb7970@google.com> <20210211104947.GL19070@quack2.suse.cz> <20210211121020.GO19070@quack2.suse.cz> <20210211125717.GH308988@casper.infradead.org> In-Reply-To: <20210211125717.GH308988@casper.infradead.org> From: Dmitry Vyukov Date: Thu, 11 Feb 2021 14:18:09 +0100 Message-ID: Subject: Re: possible deadlock in start_this_handle (2) To: Matthew Wilcox Cc: Michal Hocko , Jan Kara , syzbot , Jan Kara , linux-ext4@vger.kernel.org, LKML , syzkaller-bugs , "Theodore Ts'o" , Linux-MM Content-Type: text/plain; charset="UTF-8" Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, Feb 11, 2021 at 1:57 PM Matthew Wilcox wrote: > > > > > > Hello, > > > > > > > > > > > > added mm guys to CC. > > > > > > > > > > > > On Wed 10-02-21 05:35:18, syzbot wrote: > > > > > > > HEAD commit: 1e0d27fc Merge branch 'akpm' (patches from Andrew) > > > > > > > git tree: upstream > > > > > > > console output: https://syzkaller.appspot.com/x/log.txt?x=15cbce90d00000 > > > > > > > kernel config: https://syzkaller.appspot.com/x/.config?x=bd1f72220b2e57eb > > > > > > > dashboard link: https://syzkaller.appspot.com/bug?extid=bfdded10ab7dcd7507ae > > > > > > > userspace arch: i386 > > > > > > > > > > > > > > Unfortunately, I don't have any reproducer for this issue yet. > > > > > > > > > > > > > > IMPORTANT: if you fix the issue, please add the following tag to the commit: > > > > > > > Reported-by: syzbot+bfdded10ab7dcd7507ae@syzkaller.appspotmail.com > > > > > > > > > > > > > > ====================================================== > > > > > > > WARNING: possible circular locking dependency detected > > > > > > > 5.11.0-rc6-syzkaller #0 Not tainted > > > > > > > ------------------------------------------------------ > > > > > > > kswapd0/2246 is trying to acquire lock: > > > > > > > ffff888041a988e0 (jbd2_handle){++++}-{0:0}, at: start_this_handle+0xf81/0x1380 fs/jbd2/transaction.c:444 > > > > > > > > > > > > > > but task is already holding lock: > > > > > > > ffffffff8be892c0 (fs_reclaim){+.+.}-{0:0}, at: __fs_reclaim_acquire+0x0/0x30 mm/page_alloc.c:5195 > > > > > > > > > > > > > > which lock already depends on the new lock. > > > > > > > > > > > > > > the existing dependency chain (in reverse order) is: > > > > > > > > > > > > > > -> #2 (fs_reclaim){+.+.}-{0:0}: > > > > > > > __fs_reclaim_acquire mm/page_alloc.c:4326 [inline] > > > > > > > fs_reclaim_acquire+0x117/0x150 mm/page_alloc.c:4340 > > > > > > > might_alloc include/linux/sched/mm.h:193 [inline] > > > > > > > slab_pre_alloc_hook mm/slab.h:493 [inline] > > > > > > > slab_alloc_node mm/slub.c:2817 [inline] > > > > > > > __kmalloc_node+0x5f/0x430 mm/slub.c:4015 > > > > > > > kmalloc_node include/linux/slab.h:575 [inline] > > > > > > > kvmalloc_node+0x61/0xf0 mm/util.c:587 > > > > > > > kvmalloc include/linux/mm.h:781 [inline] > > > > > > > ext4_xattr_inode_cache_find fs/ext4/xattr.c:1465 [inline] > > > > > > > ext4_xattr_inode_lookup_create fs/ext4/xattr.c:1508 [inline] > > > > > > > ext4_xattr_set_entry+0x1ce6/0x3780 fs/ext4/xattr.c:1649 > > > > > > > ext4_xattr_ibody_set+0x78/0x2b0 fs/ext4/xattr.c:2224 > > > > > > > ext4_xattr_set_handle+0x8f4/0x13e0 fs/ext4/xattr.c:2380 > > > > > > > ext4_xattr_set+0x13a/0x340 fs/ext4/xattr.c:2493 > > > > > > > ext4_xattr_user_set+0xbc/0x100 fs/ext4/xattr_user.c:40 > > > > > > > __vfs_setxattr+0x10e/0x170 fs/xattr.c:177 > > > > > > > __vfs_setxattr_noperm+0x11a/0x4c0 fs/xattr.c:208 > > > > > > > __vfs_setxattr_locked+0x1bf/0x250 fs/xattr.c:266 > > > > > > > vfs_setxattr+0x135/0x320 fs/xattr.c:291 > > > > > > > setxattr+0x1ff/0x290 fs/xattr.c:553 > > > > > > > path_setxattr+0x170/0x190 fs/xattr.c:572 > > > > > > > __do_sys_setxattr fs/xattr.c:587 [inline] > > > > > > > __se_sys_setxattr fs/xattr.c:583 [inline] > > > > > > > __ia32_sys_setxattr+0xbc/0x150 fs/xattr.c:583 > > > > > > > do_syscall_32_irqs_on arch/x86/entry/common.c:77 [inline] > > > > > > > __do_fast_syscall_32+0x56/0x80 arch/x86/entry/common.c:139 > > > > > > > do_fast_syscall_32+0x2f/0x70 arch/x86/entry/common.c:164 > > > > > > > entry_SYSENTER_compat_after_hwframe+0x4d/0x5c > > > > > > > > > > > > This stacktrace should never happen. ext4_xattr_set() starts a transaction. > > > > > > That internally goes through start_this_handle() which calls: > > > > > > > > > > > > handle->saved_alloc_context = memalloc_nofs_save(); > > > > > > > > > > > > and we restore the allocation context only in stop_this_handle() when > > > > > > stopping the handle. And with this fs_reclaim_acquire() should remove > > > > > > __GFP_FS from the mask and not call __fs_reclaim_acquire(). > > > > > > > > > > > > Now I have no idea why something here didn't work out. Given we don't have > > > > > > a reproducer it will be probably difficult to debug this. I'd note that > > > > > > about year and half ago similar report happened (got autoclosed) so it may > > > > > > be something real somewhere but it may also be just some HW glitch or > > > > > > something like that. > > > > > > > > > > HW glitch is theoretically possible. But if we are considering such > > > > > causes, I would say a kernel memory corruption is way more likely, we > > > > > have hundreds of known memory-corruption-capable bugs open. In most > > > > > cases they are caught by KASAN before doing silent damage. But KASAN > > > > > can miss some cases. > > > > > > > > > > I see at least 4 existing bugs with similar stack: > > > > > https://syzkaller.appspot.com/bug?extid=bfdded10ab7dcd7507ae > > > > > https://syzkaller.appspot.com/bug?extid=a7ab8df042baaf42ae3c > > > > > https://syzkaller.appspot.com/bug?id=c814a55a728493959328551c769ede4c8ff72aab > > > > > https://syzkaller.appspot.com/bug?id=426ad9adca053dafcd698f3a48ad5406dccc972b > > > > > > > > > > All in all, I would not assume it's a memory corruption. When we had > > > > > bugs that actually caused silent memory corruption, that caused a > > > > > spike of random one-time crashes all over the kernel. This does not > > > > > look like it. > > > > > > > > I wonder if memalloc_nofs_save (or any other manipulation of > > > > current->flags) could have been invoked from interrupt context? I > > > > think it could cause the failure mode we observe (extremely rare > > > > disappearing flags). It may be useful to add a check for task context > > > > there. > > > > > > That's an interesting idea. I'm not sure if anything does manipulate > > > current->flags from inside an interrupt (definitely memalloc_nofs_save() > > > doesn't seem to be) but I'd think that in fully preemtible kernel, > > > scheduler could preempt the task inside memalloc_nofs_save() and the > > > current->flags manipulation could also clash with a manipulation of these > > > flags by the scheduler if there's any? > > > > current->flags should be always manipulated from the user context. But > > who knows maybe there is a bug and some interrupt handler is calling it. > > This should be easy to catch no? > > Why would it matter if it were? We save the current value of the nofs > flag and then restore it. That would happen before the end of the > interrupt handler. So the interrupt isn't going to change the observed > value of the flag by the task which is interrupted. Good question. I just think that fixing some of these assumptions as runtime checks is useful, as it will allow us to reduce infinite space of possibilities. What is called from what context. Maybe checking that PF_MEMALLOC_NOFS is indeed set when we enter memalloc_nofs_restore().