From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FD37C0502C for ; Wed, 31 Aug 2022 14:42:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B66EF8D0003; Wed, 31 Aug 2022 10:42:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B16698D0001; Wed, 31 Aug 2022 10:42:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9B6ED8D0003; Wed, 31 Aug 2022 10:42:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 8A2E08D0001 for ; Wed, 31 Aug 2022 10:42:18 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 4B4B7AB4D2 for ; Wed, 31 Aug 2022 14:42:18 +0000 (UTC) X-FDA: 79860153156.07.A085386 Received: from wout5-smtp.messagingengine.com (wout5-smtp.messagingengine.com [64.147.123.21]) by imf03.hostedemail.com (Postfix) with ESMTP id D4BC220033 for ; Wed, 31 Aug 2022 14:42:16 +0000 (UTC) Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by mailout.west.internal (Postfix) with ESMTP id 5864932007D7; Wed, 31 Aug 2022 10:42:12 -0400 (EDT) Received: from mailfrontend2 ([10.202.2.163]) by compute3.internal (MEProxy); Wed, 31 Aug 2022 10:42:15 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cerno.tech; h=cc :cc:content-type:date:date:from:from:in-reply-to:message-id :mime-version:reply-to:sender:subject:subject:to:to; s=fm3; t= 1661956930; x=1662043330; bh=wgddTPMrnv5MJa6nI4oUHuH16TmALCsoINA DyvR3vOM=; b=bEEJXBO5F4bg/rhDlik9lU4mJq35Xq06kI/Ay8gtE9R5I/Pqg+Y 01K9rxdCVMbJ1Q83HaUuHcQ8km4PO68AEJnAP5OIZr0QzaZ2Et7kEDsGs1gVG/JK H1ZitwxG8j0JZ+r3LvVzNpzCDjEzgwJnjXWUGK6VaXlVT24Az2DmuOtKOG59YCU/ ljEHx3mA/DHSN9D0HQO49dvTY2CFzWxjXKbZuOy92IR6/pYmhgiYOeBeiiCfwUAh ewWzCxntg3wMwckU926xU+2bHsyp4P9KIAavrqXe4QcxokmDEKJ+DnyyTkdjiU7G ohy/2eAUN6/dezcEEJmdlCpipDSs3S0m5HQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:cc:content-type:date:date:feedback-id :feedback-id:from:from:in-reply-to:message-id:mime-version :reply-to:sender:subject:subject:to:to:x-me-proxy:x-me-proxy :x-me-sender:x-me-sender:x-sasl-enc; s=fm1; t=1661956930; x= 1662043330; bh=wgddTPMrnv5MJa6nI4oUHuH16TmALCsoINADyvR3vOM=; b=4 B3s0nf2VTHp3uC2IkmvyWvrrLSUPEXw6ZoYAv7I2p7k9MXAojEm99iG+vS53tu3v E9Mwf2lgxx2xjhKjQB0r99TakUTnsSKT34qyMAgU3UKktesWYDsxG1G1f471pWMw 0RP78eOBxcNBW10o3XwE6swoaMYXAeHu/rlLtlchZ+iA07gTKs8rlVRYTOE6W9zh iJOaDyvsmltm4P4oAkIeYKhxk9OmixXxU5F+zzHJacRu1rjRkQZu1ZW90zfXh910 ZWnlPyKVjHMxrJjJGOQnSaaPO6kV1PnmDQ100Ok1BfkH/04hq3oU2nMR2yJCVpCu YMa4BCbzjnNN336kNtHqQ== X-ME-Sender: X-ME-Received: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedvfedrvdekiedghedvucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfqfgfvpdfurfetoffkrfgpnffqhgen uceurghilhhouhhtmecufedttdenucesvcftvggtihhpihgvnhhtshculddquddttddmne cujfgurhepfffhvfevuffkgggtugesthdtredttddtvdenucfhrhhomhepofgrgihimhgv ucftihhprghrugcuoehmrgigihhmvgestggvrhhnohdrthgvtghhqeenucggtffrrghtth gvrhhnpedtleevgfehjeejfeekgeelffeiveektdeguddujeetiedvhfdvuedtheehvdfg jeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfhhrohhmpehmrg igihhmvgestggvrhhnohdrthgvtghh X-ME-Proxy: Feedback-ID: i8771445c:Fastmail Received: by mail.messagingengine.com (Postfix) with ESMTPA; Wed, 31 Aug 2022 10:42:07 -0400 (EDT) Date: Wed, 31 Aug 2022 16:42:05 +0200 From: Maxime Ripard To: "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Arnd Bergmann , Olof Johansson , Catalin Marinas , Will Deacon , Daniel Vetter , David Airlie , Maarten Lankhorst , Thomas Zimmermann , Maxime Ripard , Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Vlastimil Babka , Roman Gushchin Cc: Martin Roukala , netdev@vger.kernel.org, linux-arm-kernel@lists.infradead.org, dri-devel@lists.freedesktop.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Dave Stevenson , Phil Elwell , Tim Gover , Dom Cobley Subject: RaspberryPi4 Panic in net_ns_init() Message-ID: <20220831144205.iirdun6bf3j5v6q4@houat> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=cerno.tech header.s=fm3 header.b=bEEJXBO5; dkim=pass header.d=messagingengine.com header.s=fm1 header.b="4 B3s0nf"; spf=pass (imf03.hostedemail.com: domain of maxime@cerno.tech designates 64.147.123.21 as permitted sender) smtp.mailfrom=maxime@cerno.tech; dmarc=pass (policy=none) header.from=cerno.tech ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1661956937; a=rsa-sha256; cv=none; b=GCc19bJtJMKeHQcR02tf65NBNFUBN/fi31HwYUvI8rOHUOBeXUAwdoWhb0WRvHLan94SaF RsWQSuESliDqPiS5jhBUbkLUwBaApRFoydKQoAin0SemapvbgQyBqlI2QkxOSuXBvF7IZi 5yzLtskvPJs3Nkowkg4mlq8vSlCwKo0= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1661956937; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding:in-reply-to: references:dkim-signature; bh=wgddTPMrnv5MJa6nI4oUHuH16TmALCsoINADyvR3vOM=; b=cvSlUoYgnAYVKH/R17EHhlBTeCU3pfz8NlTjo5m8wuBxPCRqL1JFxb5xdSnQ17+1Gg6E/e 5Cv6vjh/I6p7r4SgPnQUl4lfCUw5O+GvFbqoFC11J9H5/iNGyBkTg7KxoWIpvGZnFMaoW4 7eia3GjyUhxpXSH0Ek8DDfxJh5P51kI= X-Rspam-User: X-Rspamd-Queue-Id: D4BC220033 Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=cerno.tech header.s=fm3 header.b=bEEJXBO5; dkim=pass header.d=messagingengine.com header.s=fm1 header.b="4 B3s0nf"; spf=pass (imf03.hostedemail.com: domain of maxime@cerno.tech designates 64.147.123.21 as permitted sender) smtp.mailfrom=maxime@cerno.tech; dmarc=pass (policy=none) header.from=cerno.tech X-Stat-Signature: n833swiraemoprnuzanc5fw5k99h4ei9 X-Rspamd-Server: rspam11 X-HE-Tag: 1661956936-383554 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi, Sorry for the fairly broad list of recipients, I'm not entirely sure where the issue lies exactly, and it seems like multiple areas are involved. Martin reported me an issue discovered with the VC4 DRM driver that would prevent the RaspberryPi4 from booting entirely. At boot, and apparently before the console initialization, the board would just die. It first appeared when both DYNAMIC_DEBUG and DRM_VC4 were built-in. We started to look into what configuration would trigger it. It looks like a good reproducer is: ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make -j18 defconfig mod2yesconfig ./scripts/config -e CONFIG_DYNAMIC_DEBUG ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- make -j18 olddefconfig If we enable earlycon, we end up with: [ 0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd083] [ 0.000000] Linux version 6.0.0-rc3 (max@houat) (aarch64-linux-gnu-gcc (GCC) 12.1.1 20220507 (Red Hat Cross 12.1.1-1), GNU ld version 2.37-7.fc36) #52 SMP PREEMPT Wed Aug 31 14:28:41 CEST 2022 [ 0.000000] random: crng init done [ 0.000000] Machine model: Raspberry Pi 4 Model B Rev 1.1 [ 0.000000] earlycon: uart8250 at MMIO32 0x00000000fe215040 (options '') [ 0.000000] printk: bootconsole [uart8250] enabled [ 0.000000] efi: UEFI not found. [ 0.000000] Reserved memory: bypass linux,cma node, using cmdline CMA params instead [ 0.000000] OF: reserved mem: node linux,cma compatible matching fail [ 0.000000] NUMA: No NUMA configuration found [ 0.000000] NUMA: Faking a node at [mem 0x0000000000000000-0x00000000fbffffff] [ 0.000000] NUMA: NODE_DATA [mem 0xfb815b40-0xfb817fff] [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000000000-0x000000003fffffff] [ 0.000000] DMA32 [mem 0x0000000040000000-0x00000000fbffffff] [ 0.000000] Normal empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000000000-0x000000003b3fffff] [ 0.000000] node 0: [mem 0x0000000040000000-0x00000000fbffffff] [ 0.000000] Initmem setup node 0 [mem 0x0000000000000000-0x00000000fbffffff] [ 0.000000] On node 0, zone DMA32: 19456 pages in unavailable ranges [ 0.000000] On node 0, zone DMA32: 16384 pages in unavailable ranges [ 0.000000] cma: Reserved 512 MiB at 0x000000000ee00000 [ 0.000000] percpu: Embedded 21 pages/cpu s48040 r8192 d29784 u86016 [ 0.000000] Detected PIPT I-cache on CPU0 [ 0.000000] CPU features: detected: Spectre-v2 [ 0.000000] CPU features: detected: Spectre-v3a [ 0.000000] CPU features: detected: Spectre-v4 [ 0.000000] CPU features: detected: Spectre-BHB [ 0.000000] CPU features: detected: Kernel page table isolation (KPTI) [ 0.000000] CPU features: detected: ARM erratum 1742098 [ 0.000000] CPU features: detected: ARM errata 1165522, 1319367, or 1530923 [ 0.000000] Fallback order for Node 0: 0 [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 996912 [ 0.000000] Policy zone: DMA32 [ 0.000000] Kernel command line: video=Composite-1:720x480@60i,margin_left=32,margin_right=32,margin_top=32,margin_bottom=32 dma.dmachans=0x37f5 bcm2709.boardrev=0xc03111 bcm2709.serial=0xb7f44626 bcm2709.uart_clock=48000000 bcm2709.disk_led_gpio=42 bcm2709.disk_led_active_low=0 smsc95xx.macaddr=DC:A6:32:0E:F7:01 vc_mem.mem_base=0x3ec00000 vc_mem.mem_size=0x40000000 root=/dev/nfs nfsroot=192.168.20.10:/srv/nfs/rpi/bullseye64 rw 8250.nr_uarts=1 cma=512M ip=dhcp console=ttyS0,115200 earlycon=uart8250,mmio32,0xfe215040 [ 0.000000] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes, linear) [ 0.000000] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes, linear) [ 0.000000] mem auto-init: stack:all(zero), heap alloc:off, heap free:off [ 0.000000] software IO TLB: area num 4. [ 0.000000] software IO TLB: mapped [mem 0x0000000037400000-0x000000003b400000] (64MB) [ 0.000000] Memory: 3312220K/4050944K available (30656K kernel code, 5924K rwdata, 18912K rodata, 11584K init, 672K bss, 214436K reserved, 524288K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] rcu: Preemptible hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=256 to nr_cpu_ids=4. [ 0.000000] Trampoline variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 25 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 64, nr_irqs: 64, preallocated irqs: 0 [ 0.000000] Root IRQ handler: gic_handle_irq [ 0.000000] GIC: Using split EOI/Deactivate mode [ 0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention. [ 0.000000] arch_timer: cp15 timer(s) running at 54.00MHz (phys). [ 0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0xc743ce346, max_idle_ns: 440795203123 ns [ 0.000001] sched_clock: 56 bits at 54MHz, resolution 18ns, wraps every 4398046511102ns [ 0.008648] Console: colour dummy device 80x25 [ 0.013237] Calibrating delay loop (skipped), value calculated using timer frequency.. 108.00 BogoMIPS (lpj=216000) [ 0.023803] pid_max: default: 32768 minimum: 301 [ 0.028540] LSM: Security Framework initializing [ 0.033252] Kernel panic - not syncing: Could not allocate generic netns [ 0.040026] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 6.0.0-rc3 #52 [ 0.046363] Hardware name: Raspberry Pi 4 Model B Rev 1.1 (DT) [ 0.052255] Call trace: [ 0.054721] dump_backtrace+0xe4/0x124 [ 0.058525] show_stack+0x1c/0x5c [ 0.061878] dump_stack_lvl+0x64/0x80 [ 0.065582] dump_stack+0x1c/0x38 [ 0.068932] panic+0x170/0x328 [ 0.072020] net_ns_init+0x88/0x134 [ 0.075548] start_kernel+0x628/0x69c [ 0.079251] __primary_switched+0xbc/0xc4 [ 0.083311] ---[ end Kernel panic - not syncing: Could not allocate generic netns ]--- So it seems that net_alloc_generic() fails, and the only reason I could see is if kzalloc() fails, so now I'm super confused. It looks like the board has plenty (~3GB) of RAM available at boot, and most importantly I don't see the relationship between a DRM driver, DYNAMIC_DEBUG, and SLAB or the network namespace. After a bit more experiments, * ./scripts/config -e CONFIG_DYNAMIC_DEBUG -d CONFIG_DRM_VC4 still has that panic, so it looks like VC4 itself isn't involved. * ./scripts/config -e CONFIG_DYNAMIC_DEBUG -d CONFIG_DRM works, so DRM seems to be involved somehow. It has a number of memory management dependencies, so it's probably a side effect of DRM being there. * make defconfig mod2yesconfig (so without DYNAMIC_DEBUG, with DRM) works too. So it looks to me like there's indeed some interaction between DRM, DYNAMIC_DEBUG, SLAB and/or the network namespace, but I'm not entirely sure where to go from there. Any ideas? Maxime