From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-2.7 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 54DFBC2BCC4 for ; Wed, 12 May 2021 16:31:20 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id AE68F61288 for ; Wed, 12 May 2021 16:31:17 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org AE68F61288 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 16ACC6B006C; Wed, 12 May 2021 12:31:17 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 141936B006E; Wed, 12 May 2021 12:31:17 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EFE6B6B0070; Wed, 12 May 2021 12:31:16 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0020.hostedemail.com [216.40.44.20]) by kanga.kvack.org (Postfix) with ESMTP id BD6B96B006C for ; Wed, 12 May 2021 12:31:16 -0400 (EDT) Received: from smtpin34.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 5B78818044754 for ; Wed, 12 May 2021 16:31:16 +0000 (UTC) X-FDA: 78133118952.34.EBF692B Received: from mail-ed1-f42.google.com (mail-ed1-f42.google.com [209.85.208.42]) by imf29.hostedemail.com (Postfix) with ESMTP id 1B13B2BEB for ; Wed, 12 May 2021 16:31:08 +0000 (UTC) Received: by mail-ed1-f42.google.com with SMTP id v5so16737207edc.8 for ; Wed, 12 May 2021 09:31:15 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=NrO2f3Y5bX0Ld5eTZyN7edfkVSC2ChqprPIZjMQEH38=; b=XM5b9z+QHDlAD8+VuFk/8xLmDkYNr6ofjsbETZe5gaYBKrPyrY7/S7XIA+LxQvsRbv Y66nsH2OcHDkFOl2SZ84KBdW6ch9JeRisPzCldcPeWFAcl97xUbXNeTyDd++MpI+QRJF qCejx25S/2puNmP9Y7V9i7MGlaWDo1wS9k29sHKOB9xt4nneHxFAlCZEkJi3+3IZ+h9X 3AFsdGYkp2P+AkD1WLKesLtH8ux1MhfEnLcOJ8APx3+pcI/lGyGo9ByxgLZwSJtG+MQ6 PsSU0aRJWT99lCtC6lqdCxMo9APjfVm9C4kIMnT1FpsVPk8t7D7La+sWMZTOr3yagGwy h83A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=NrO2f3Y5bX0Ld5eTZyN7edfkVSC2ChqprPIZjMQEH38=; b=K+zBRc0RXzpqWsOirPr6hTKwV5r49LDuaAJ6kRu29sG2mvyVKRiCDMik2/hEhm7wQU D/V/QRhkwOX+oj5M6W9R8+wQKbcBftc9/1Ih7Kk7BtrV7B9E6V6OVN0ZPPe1/fjaql2J gUvBEaVf32onxsvzPLz5ggCo9IuCKD+0qzMsHLUzDAcQtTqHAAUX/brI7c2jfN0fqlYQ qhu+l+XaPiJxorc1weSeOlHL4H3uuqYa6UARfrfNxrZl3YCPna2oidUMS1MMed9idjvv 47eBb1VPQvis68B8Xh1dqt0vS/GxQNKt93px2MF8OmciIE7CtcTuUXj28tWz3LmWMrSy mayQ== X-Gm-Message-State: AOAM531rPfzTR1N00FNJRdUNayNoEHNGXtgKCJBmwit9lJa60ATfZTZx U6wAS0f6WMr6pHaQF8OlE083pVCLRsTj1FAfZZM= X-Google-Smtp-Source: ABdhPJxNXHQWVUndkl0WL4YeAcYGb4YO8Fmbh6Xp6cwte+zostGw4ZiulMbZmxqW/DhkkriAB35s4xLosS6LYsyt1Zg= X-Received: by 2002:a05:6402:51ce:: with SMTP id r14mr44923278edd.151.1620837074600; Wed, 12 May 2021 09:31:14 -0700 (PDT) MIME-Version: 1.0 References: <921e53f3-4b13-aab8-4a9e-e83ff15371e4@nec.com> In-Reply-To: From: Yang Shi Date: Wed, 12 May 2021 09:31:02 -0700 Message-ID: Subject: Re: [REGRESSION v5.13-rc1] NULL dereference in do_shrink_slab() To: Shakeel Butt Cc: =?UTF-8?B?Tk9NVVJBIEpVTklDSEko6YeO5p2RIOa3s+S4gCk=?= , Tejun Heo , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" , "vbabka@suse.cz" , "ktkhai@virtuozzo.com" , "guro@fb.com" , "david@fromorbit.com" , "hannes@cmpxchg.org" , "mhocko@suse.com" , "akpm@linux-foundation.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20161025 header.b=XM5b9z+Q; spf=pass (imf29.hostedemail.com: domain of shy828301@gmail.com designates 209.85.208.42 as permitted sender) smtp.mailfrom=shy828301@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: 1B13B2BEB X-Stat-Signature: ca57wpddo64kdf6jy47ujuuuctt3yeca Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf29; identity=mailfrom; envelope-from=""; helo=mail-ed1-f42.google.com; client-ip=209.85.208.42 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1620837068-939315 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, May 12, 2021 at 5:36 AM Shakeel Butt wrote: > > +Tejun Heo > > On Wed, May 12, 2021 at 3:48 AM NOMURA JUNICHI(=E9=87=8E=E6=9D=91=E3=80= =80=E6=B7=B3=E4=B8=80) > wrote: > > > > v5.13-rc1 sometimes causes NULL pointer dereference during kdump, where > > memcg is disabled with "cgroup_disable=3Dmemory" boot option. > > I haven't seen this problem with v5.12, so it looks like regression. > > > > [ 73.199590] BUG: kernel NULL pointer dereference, address: 000000000= 0000000 > > [ 73.206593] #PF: supervisor write access in kernel mode > > [ 73.211845] #PF: error_code(0x0002) - not-present page > > [ 73.217010] PGD 0 P4D 0 > > [ 73.219556] Oops: 0002 [#1] SMP NOPTI > > [ 73.223236] CPU: 0 PID: 95 Comm: kswapd0 Tainted: G I = 5.13.0-rc1 #1 > > [ 73.239418] RIP: 0010:do_shrink_slab+0x85/0x2d0 > > [ 73.243977] Code: 49 63 44 24 04 be 00 00 00 00 49 8b 4c 24 18 f6 c2= 02 48 0f 44 c6 48 85 c9 74 09 83 e2 04 0f 85 19 02 00 00 49 8b 4f 38 31 d2= <48> 87 14 c1 48 89 55 b8 41 8b 77 18 4c 89 f0 85 f6 0f 84 82 01 00 > > [ 73.262856] RSP: 0018:ffffc900001abc18 EFLAGS: 00010246 > > [ 73.268108] RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000= 000000000 > > [ 73.275281] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000= 000000064 > > [ 73.282454] RBP: ffffc900001abc70 R08: 28f5c28f5c28f5c3 R09: 0000000= 000000000 > > [ 73.289628] R10: 0000000000000000 R11: 0000000000000004 R12: ffffc90= 0001abca0 > > [ 73.296800] R13: 0000000000000400 R14: 0000000000000002 R15: ffff888= 05344bc10 > > [ 73.303972] FS: 0000000000000000(0000) GS:ffff888072c00000(0000) kn= lGS:0000000000000000 > > [ 73.312108] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 73.317883] CR2: 0000000000000000 CR3: 000000005cf68004 CR4: 0000000= 0007706b0 > > [ 73.325055] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000= 000000000 > > [ 73.332227] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000= 000000400 > > [ 73.339400] PKRU: 55555554 > > [ 73.342117] Call Trace: > > [ 73.344576] shrink_slab+0xa9/0x2b0 > > [ 73.348083] ? __update_load_avg_se+0x298/0x320 > > [ 73.352640] shrink_node+0x248/0x6f0 > > [ 73.356234] balance_pgdat+0x303/0x5f0 > > [ 73.360002] kswapd+0x20b/0x390 > > [ 73.363157] ? finish_wait+0x80/0x80 > > [ 73.366752] ? balance_pgdat+0x5f0/0x5f0 > > [ 73.370693] kthread+0x124/0x140 > > [ 73.373937] ? kthread_park+0x90/0x90 > > [ 73.377617] ret_from_fork+0x1f/0x30 > > [ 73.381215] Modules linked in: xfs libcrc32c sd_mod t10_pi sr_mod cd= rom sg crc32c_intel ahci libahci libata smartpqi scsi_transport_sas overlay= squashfs loop > > [ 73.395386] CR2: 0000000000000000 > > [ 73.398716] ---[ end trace 9752d71309d33c00 ]--- > > > > The code around do_shrink_slab+0x85 is: > > 0xffffffff9d094925 : mov 0x18(%r12),%rcx > > 0xffffffff9d09492a : test $0x2,%dl > > 0xffffffff9d09492d : cmove %rsi,%rax > > 0xffffffff9d094931 : test %rcx,%rcx > > 0xffffffff9d094934 : je 0xffffffff9d0949= 3f > > 0xffffffff9d094936 : and $0x4,%edx > > 0xffffffff9d094939 : jne 0xffffffff9d094b= 58 > > 0xffffffff9d09493f : mov 0x38(%r15),%rcx > > 0xffffffff9d094943 : xor %edx,%edx > > 0xffffffff9d094945 : xchg %rdx,(%rcx,%rax,= 8) > > > > The NULL dereference occurred at here in in-lined xchg_nr_deferred(): > > > > return atomic_long_xchg(&shrinker->nr_deferred[nid], 0); > > > > that means "shrinker->nr_deferred" was NULL. > > > > Though I haven't fully bisected between v5.12 and v5.13-rc1, I can repr= oduce > > the problem with this commit: > > > > 476b30a0949a mm: vmscan: don't need allocate shrinker->nr_deferred f= or memcg aware shrinkers > > > > but not with this previous commit: > > > > 867508304685 mm: vmscan: use per memcg nr_deferred of shrinker > > > > With the commit 476b30a0949a, if a memcg-aware shrinker is registered b= efore > > cgroup_init(), shrinker->nr_deferred is NULL. However xchg_nr_deferred= () > > tries to use it as memcg is turned off via "cgroup_disable=3Dmemory". > > > > Any thoughts? Thanks for the report. > > Is there a way to find the call chain of "memcg-aware shrinker is > registered before cgroup_init()"? Other than adding some printk in prealloc_memcg_shrinker() then checking out the output of dmesg I didn't think of a better way. Not sure if we have something like early trace. > > Irrespective I think we can revert a3e72739b7a7e ("cgroup: fix too > early usage of static_branch_disable()") as 6041186a3258 ("init: > initialize jump labels before command line option parsing") has moved > the initialization of jump labels before command line parsing. Seems make sense to me. If some memcg aware shrinker is registered before cgroup_init(), the mem_cgroup_disabled() check in prealloc_memcg_shrinker() would return false negative. And I don't think any shrinker could be registered before parsing boot commandline.