From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id C5B2DC433EF for ; Thu, 12 May 2022 05:41:28 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1350016AbiELFlW (ORCPT ); Thu, 12 May 2022 01:41:22 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:48432 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1350000AbiELFlM (ORCPT ); Thu, 12 May 2022 01:41:12 -0400 Received: from muru.com (muru.com [72.249.23.125]) by lindbergh.monkeyblade.net (Postfix) with ESMTP id 135625641A; Wed, 11 May 2022 22:41:10 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by muru.com (Postfix) with ESMTPS id CAE928027; Thu, 12 May 2022 05:37:34 +0000 (UTC) Date: Thu, 12 May 2022 08:41:08 +0300 From: Tony Lindgren To: Yegor Yefremov , Ard Biesheuvel , Arnd Bergmann Cc: Linux-OMAP , linux-clk@vger.kernel.org, Stephen Boyd , linux-arm-kernel@lists.infradead.org Subject: Re: am335x: 5.18.x: system stalling Message-ID: References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: linux-clk@vger.kernel.org Hi, Adding Ard and Arnd for vmap stack. * Yegor Yefremov [220511 14:16]: > Hi Tony, > > On Thu, May 5, 2022 at 7:08 AM Tony Lindgren wrote: > > > > Hi, > > > > * Yegor Yefremov [220504 10:35]: > > > Hi Tony, all, > > > > > > since kernel 5.18.x (5.17.x doesn't show this behavior), the system > > > stalls as soon as I invoke the following commands (initializing > > > USB-to-CAN converter): > > > > > > slcand -o -s8 -t hw -S 3000000 /dev/ttyUSB0 > > > ip link set slcan0 up > > > > > > Have you already seen such an issue? Should I try to bisect this? > > > > No have not seen this one either, yes please bisect if you can. > > > > Note that v5.18-rc1 has revert commit 859c2c7b1d06 ("Revert "clk: Drop > > the rate range on clk_put()"") that you may need to carry along in the > > bisect. > > I had to skip a lot of commits due to assembler related build issues: > > /tmp/cc5p087h.s: Assembler messages: > /tmp/cc5p087h.s:500: Error: invalid literal constant: pool needs to be closer > > Hence, I don't have the exact commit: > > #There are only 'skip'ped commits left to test. > The first bad commit could be any of: > 9cf72c358a20b95e040e6a54a03baf6d264e0719 > cafc0eab168917ec9c0cd47d530a40cd40eb2928 > 23d9a9280efea105852de358f21d69231992ae73 > 9c46929e7989efacc1dd0a1dd662a839897ea2b6 > 5fe41793bc78d9bb47fea37d1a16984ad6cf294b > We cannot bisect more! Sounds like you would need to carry some fixes along with the bisect to avoid multiple bugs.. Note that for smc calls we needed 8cf8df89678a ("ARM: OMAP2+: Fix regression for smc calls for vmap stack"), but that should only affect am3/4 for system suspend. > git bisect log > git bisect start > # good: [f443e374ae131c168a065ea1748feac6b2e76613] Linux 5.17 > git bisect good f443e374ae131c168a065ea1748feac6b2e76613 > # bad: [672c0c5173427e6b3e2a9bbb7be51ceeec78093a] Linux 5.18-rc5 > git bisect bad 672c0c5173427e6b3e2a9bbb7be51ceeec78093a > # bad: [25fd2d41b505d0640bdfe67aa77c549de2d3c18a] selftests: kselftest > framework: provide "finished" helper > git bisect bad 25fd2d41b505d0640bdfe67aa77c549de2d3c18a > # bad: [b4bc93bd76d4da32600795cd323c971f00a2e788] Merge tag > 'arm-drivers-5.18' of > git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc > git bisect bad b4bc93bd76d4da32600795cd323c971f00a2e788 > # good: [3fe2f7446f1e029b220f7f650df6d138f91651f2] Merge tag > 'sched-core-2022-03-22' of > git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip > git bisect good 3fe2f7446f1e029b220f7f650df6d138f91651f2 > # good: [182966e1cd74ec0e326cd376de241803ee79741b] Merge tag > 'media/v5.18-1' of > git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media > git bisect good 182966e1cd74ec0e326cd376de241803ee79741b > # good: [49a24e9d9c740d3bd8b1200f225f67d45e3d68a5] Make the SOF > control, PCM and PM code IPC agnostic > git bisect good 49a24e9d9c740d3bd8b1200f225f67d45e3d68a5 > # bad: [8ffa5709e577385a1c8d20fb434cb02732f1d991] Merge tag > 'arm-defconfig-5.18' of > git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc > git bisect bad 8ffa5709e577385a1c8d20fb434cb02732f1d991 > # good: [e6aef3496a00a12e78a571f61d98300cf0a86e6a] Merge tag > 'm68knommu-for-v5.18' of > git://git.kernel.org/pub/scm/linux/kernel/git/gerg/m68knommu > git bisect good e6aef3496a00a12e78a571f61d98300cf0a86e6a > # bad: [9c0e6a89b592f4c4e4d769dbc22d399ab0685159] Merge tag > 'for-linus' of git://git.armlinux.org.uk/~rmk/linux-arm > git bisect bad 9c0e6a89b592f4c4e4d769dbc22d399ab0685159 > # skip: [cafc0eab168917ec9c0cd47d530a40cd40eb2928] ARM: v7m: enable > support for IRQ stacks > git bisect skip cafc0eab168917ec9c0cd47d530a40cd40eb2928 > # skip: [54f481a2308efab49d2b14c3f8263b34fdb1c65e] ARM: remove > old-style irq entry > git bisect skip 54f481a2308efab49d2b14c3f8263b34fdb1c65e > # good: [8cdfdf7fe4fec5a952edfb8927ee7cc639c58184] ARM: export > dump_mem() to other objects > git bisect good 8cdfdf7fe4fec5a952edfb8927ee7cc639c58184 > # bad: [5fe41793bc78d9bb47fea37d1a16984ad6cf294b] ARM: 9176/1: avoid > literal references in inline assembly > git bisect bad 5fe41793bc78d9bb47fea37d1a16984ad6cf294b > # good: [90890f17ccd2aa96350abd1f4d37d4667e09027f] ARM: footbridge: > use GENERIC_IRQ_MULTI_HANDLER > git bisect good 90890f17ccd2aa96350abd1f4d37d4667e09027f > # good: [4e918ab13eaf40f19938659cb5a22c93172778a8] ARM: assembler: add > optimized ldr/str macros to load variables from memory > git bisect good 4e918ab13eaf40f19938659cb5a22c93172778a8 > # skip: [9c46929e7989efacc1dd0a1dd662a839897ea2b6] ARM: implement > THREAD_INFO_IN_TASK for uniprocessor systems > git bisect skip 9c46929e7989efacc1dd0a1dd662a839897ea2b6 > # good: [c2755910373bb5dfb9aa68ba2924036686815c9e] ARM: smp: defer > TPIDRURO update for SMP v6 configurations too > git bisect good c2755910373bb5dfb9aa68ba2924036686815c9e > # skip: [9cf72c358a20b95e040e6a54a03baf6d264e0719] Merge tag > 'arm-irq-and-vmap-stacks-for-rmk' of > git://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux into > devel-stable > git bisect skip 9cf72c358a20b95e040e6a54a03baf6d264e0719 > # skip: [23d9a9280efea105852de358f21d69231992ae73] ARM: 9177/1: > disable vmap'ed stacks on suspend-capable SMP configs > git bisect skip 23d9a9280efea105852de358f21d69231992ae73 > # only skipped commits left to test > # possible first bad commit: > [5fe41793bc78d9bb47fea37d1a16984ad6cf294b] ARM: 9176/1: avoid literal > references in inline assembly > # possible first bad commit: > [23d9a9280efea105852de358f21d69231992ae73] ARM: 9177/1: disable > vmap'ed stacks on suspend-capable SMP configs > # possible first bad commit: > [9cf72c358a20b95e040e6a54a03baf6d264e0719] Merge tag > 'arm-irq-and-vmap-stacks-for-rmk' of > git://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux into > devel-stable > # possible first bad commit: > [cafc0eab168917ec9c0cd47d530a40cd40eb2928] ARM: v7m: enable support > for IRQ stacks > # possible first bad commit: > [9c46929e7989efacc1dd0a1dd662a839897ea2b6] ARM: implement > THREAD_INFO_IN_TASK for uniprocessor systems Maybe Ard and Arnd have some ideas what might be going wrong here. Basically anything trying to use a physical address on stack will fail in weird ways like we've seen for smc and wl1251. Regards, Tony > Best regards, > Yegor > > > > [ 88.408578] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: > > > [ 88.415777] (detected by 0, t=2602 jiffies, g=2529, q=17) > > > [ 88.422026] rcu: All QSes seen, last rcu_sched kthread activity > > > 2602 (-21160--23762), jiffies_till_next_fqs=1, root ->qsmask 0x0 > > > [ 88.434445] rcu: rcu_sched kthread starved for 2602 jiffies! g2529 > > > f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 > > > [ 88.445274] rcu: Unless rcu_sched kthread gets sufficient CPU > > > time, OOM is now expected behavior. > > > [ 88.454859] rcu: RCU grace-period kthread stack dump: > > > [ 88.460446] task:rcu_sched state:R running task stack: > > > 0 pid: 11 ppid: 2 flags:0x00000000 > > > [ 88.471840] __schedule from schedule+0x58/0xcc > > > [ 88.477680] schedule from schedule_timeout+0x78/0xf8 > > > [ 88.483754] schedule_timeout from rcu_gp_fqs_loop+0x108/0x3cc > > > [ 88.490629] rcu_gp_fqs_loop from rcu_gp_kthread+0xa8/0x134 > > > [ 88.497187] rcu_gp_kthread from kthread+0xe4/0x104 > > > [ 88.503061] kthread from ret_from_fork+0x14/0x28 > > > [ 88.508627] Exception stack(0xd0041fb0 to 0xd0041ff8) > > > [ 88.514443] 1fa0: 00000000 > > > 00000000 00000000 00000000 > > > [ 88.523433] 1fc0: 00000000 00000000 00000000 00000000 00000000 > > > 00000000 00000000 00000000 > > > [ 88.532374] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000 > > > [ 88.539639] rcu: Stack dump where RCU GP kthread last ran: > > > [ 88.545694] NMI backtrace for cpu 0 > > > [ 88.549779] CPU: 0 PID: 58 Comm: kworker/0:8 Not tainted 5.18.0-rc5 #1 > > > [ 88.557103] Hardware name: Generic AM33XX (Flattened Device Tree) > > > [ 88.563822] Workqueue: events dbs_work_handler > > > [ 88.569398] unwind_backtrace from show_stack+0x10/0x14 > > > [ 88.575662] show_stack from dump_stack_lvl+0x58/0x70 > > > [ 88.581627] dump_stack_lvl from nmi_cpu_backtrace+0xe0/0x128 > > > [ 88.588345] nmi_cpu_backtrace from nmi_trigger_cpumask_backtrace+0xec/0x184 > > > [ 88.596339] nmi_trigger_cpumask_backtrace from > > > trigger_single_cpu_backtrace+0x20/0x2c > > > [ 88.605221] trigger_single_cpu_backtrace from > > > rcu_check_gp_kthread_starvation+0xf4/0x148 > > > [ 88.614328] rcu_check_gp_kthread_starvation from > > > rcu_sched_clock_irq+0xdf0/0xf7c > > > [ 88.622778] rcu_sched_clock_irq from update_process_times+0x88/0xc0 > > > [ 88.630182] update_process_times from tick_sched_handle+0x48/0x54 > > > [ 88.637293] tick_sched_handle from tick_sched_timer+0x48/0xac > > > [ 88.643993] tick_sched_timer from __hrtimer_run_queues+0x244/0x4d8 > > > [ 88.651212] __hrtimer_run_queues from hrtimer_interrupt+0x128/0x2c8 > > > [ 88.658582] hrtimer_interrupt from dmtimer_clockevent_interrupt+0x24/0x2c > > > [ 88.666506] dmtimer_clockevent_interrupt from > > > __handle_irq_event_percpu+0x98/0x334 > > > [ 88.675241] __handle_irq_event_percpu from handle_irq_event+0x38/0xc0 > > > [ 88.682749] handle_irq_event from handle_level_irq+0xb4/0x1a8 > > > [ 88.689639] handle_level_irq from handle_irq_desc+0x1c/0x2c > > > [ 88.696253] handle_irq_desc from generic_handle_arch_irq+0x2c/0x64 > > > [ 88.703524] generic_handle_arch_irq from __irq_svc+0x90/0xbc > > > [ 88.710195] Exception stack(0xd0001f58 to 0xd0001fa0) > > > [ 88.715947] 1f40: > > > c01015c8 00000000 > > > [ 88.724939] 1f60: 0eae9000 00000000 fffffffe 60000013 ffffffff > > > d0385d74 00000000 c2702a80 > > > [ 88.733926] 1f80: 00000002 c2702a80 00000000 d0001fa8 c01015c8 > > > c01015d0 60000113 ffffffff > > > [ 88.742765] __irq_svc from __do_softirq+0xa0/0x604 > > > [ 88.748533] __do_softirq from __irq_exit_rcu+0x138/0x178 > > > [ 88.754961] __irq_exit_rcu from irq_exit+0x8/0x28 > > > [ 88.760758] irq_exit from call_with_stack+0x18/0x20 > > > [ 88.766687] call_with_stack from __irq_svc+0x9c/0xbc > > > [ 88.772576] Exception stack(0xd0385d40 to 0xd0385d88) > > > [ 88.778458] 5d40: 00000005 00000488 00000000 00000000 c208c0c0 > > > 00006402 c208b800 c1874ff0 > > > [ 88.787451] 5d60: 00000000 c208c0c0 c1109210 c208c0d8 00000000 > > > d0385d90 c06e068c c06e08a4 > > > [ 88.796305] 5d80: 60000013 ffffffff > > > [ 88.800369] __irq_svc from omap3_noncore_dpll_program+0x3f8/0x5ec > > > [ 88.807588] omap3_noncore_dpll_program from clk_change_rate+0x23c/0x4f8 > > > [ 88.815375] clk_change_rate from clk_core_set_rate_nolock+0x1b0/0x29c > > > [ 88.822936] clk_core_set_rate_nolock from clk_set_rate+0x30/0x64 > > > [ 88.830056] clk_set_rate from _set_opp+0x254/0x51c > > > [ 88.835835] _set_opp from dev_pm_opp_set_rate+0xec/0x228 > > > [ 88.842073] dev_pm_opp_set_rate from __cpufreq_driver_target+0x584/0x700 > > > [ 88.849792] __cpufreq_driver_target from od_dbs_update+0xb4/0x168 > > > [ 88.856953] od_dbs_update from dbs_work_handler+0x2c/0x60 > > > [ 88.863441] dbs_work_handler from process_one_work+0x284/0x72c > > > [ 88.870411] process_one_work from worker_thread+0x28/0x4b0 > > > [ 88.876973] worker_thread from kthread+0xe4/0x104 > > > [ 88.882692] kthread from ret_from_fork+0x14/0x28 > > > [ 88.888225] Exception stack(0xd0385fb0 to 0xd0385ff8) > > > [ 88.893998] 5fa0: 00000000 > > > 00000000 00000000 00000000 > > > [ 88.902971] 5fc0: 00000000 00000000 00000000 00000000 00000000 > > > 00000000 00000000 00000000 > > > [ 88.911888] 5fe0: 00000000 00000000 00000000 00000000 00000013 00000000 > > > > > > Regards, > > > Yegor From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7AF16C433EF for ; Thu, 12 May 2022 05:42:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=DFxIlMgFhI+ID0Q95TaZ6enVi4bXUF+7QVukJRZWCDQ=; b=XzAp37rNWrjREI pbfBV9Vejnl5xG258D2+Xcxu3DIH+8U55zZezw7d2GsPm+v+Fammr28HNNX5FRCFYN8OgClU9pijQ 8CBC8/b4ytde1GMjqn32RdBVBhJ5K/BlP1GaNl9yP/e6beJ3IC1tkz/DwkF6Irg9tMPL/9upK/5Ag xueN863UBQapGoMR6GkwhJ7aPahEt9+giGnX5e11+XbPGOhVLfmM0rKvkgE8EKOYD0Ma4mkjzKALg rLGQspmvNw7kxXcq0GMn1FX3HZQ6B0F4jU5tg6jZ9+uDHcjC9WdkGB4loLLLqg+oYQyIc21FP/wcP gf/Tl8rDqvy3aeXOvaig==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1np1Zh-00AB2Y-LA; Thu, 12 May 2022 05:41:22 +0000 Received: from muru.com ([72.249.23.125]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1np1ZW-00AAyN-J1 for linux-arm-kernel@lists.infradead.org; Thu, 12 May 2022 05:41:18 +0000 Received: from localhost (localhost [127.0.0.1]) by muru.com (Postfix) with ESMTPS id CAE928027; Thu, 12 May 2022 05:37:34 +0000 (UTC) Date: Thu, 12 May 2022 08:41:08 +0300 From: Tony Lindgren To: Yegor Yefremov , Ard Biesheuvel , Arnd Bergmann Cc: Linux-OMAP , linux-clk@vger.kernel.org, Stephen Boyd , linux-arm-kernel@lists.infradead.org Subject: Re: am335x: 5.18.x: system stalling Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220511_224110_725103_0F684C5B X-CRM114-Status: GOOD ( 28.15 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi, Adding Ard and Arnd for vmap stack. * Yegor Yefremov [220511 14:16]: > Hi Tony, > > On Thu, May 5, 2022 at 7:08 AM Tony Lindgren wrote: > > > > Hi, > > > > * Yegor Yefremov [220504 10:35]: > > > Hi Tony, all, > > > > > > since kernel 5.18.x (5.17.x doesn't show this behavior), the system > > > stalls as soon as I invoke the following commands (initializing > > > USB-to-CAN converter): > > > > > > slcand -o -s8 -t hw -S 3000000 /dev/ttyUSB0 > > > ip link set slcan0 up > > > > > > Have you already seen such an issue? Should I try to bisect this? > > > > No have not seen this one either, yes please bisect if you can. > > > > Note that v5.18-rc1 has revert commit 859c2c7b1d06 ("Revert "clk: Drop > > the rate range on clk_put()"") that you may need to carry along in the > > bisect. > > I had to skip a lot of commits due to assembler related build issues: > > /tmp/cc5p087h.s: Assembler messages: > /tmp/cc5p087h.s:500: Error: invalid literal constant: pool needs to be closer > > Hence, I don't have the exact commit: > > #There are only 'skip'ped commits left to test. > The first bad commit could be any of: > 9cf72c358a20b95e040e6a54a03baf6d264e0719 > cafc0eab168917ec9c0cd47d530a40cd40eb2928 > 23d9a9280efea105852de358f21d69231992ae73 > 9c46929e7989efacc1dd0a1dd662a839897ea2b6 > 5fe41793bc78d9bb47fea37d1a16984ad6cf294b > We cannot bisect more! Sounds like you would need to carry some fixes along with the bisect to avoid multiple bugs.. Note that for smc calls we needed 8cf8df89678a ("ARM: OMAP2+: Fix regression for smc calls for vmap stack"), but that should only affect am3/4 for system suspend. > git bisect log > git bisect start > # good: [f443e374ae131c168a065ea1748feac6b2e76613] Linux 5.17 > git bisect good f443e374ae131c168a065ea1748feac6b2e76613 > # bad: [672c0c5173427e6b3e2a9bbb7be51ceeec78093a] Linux 5.18-rc5 > git bisect bad 672c0c5173427e6b3e2a9bbb7be51ceeec78093a > # bad: [25fd2d41b505d0640bdfe67aa77c549de2d3c18a] selftests: kselftest > framework: provide "finished" helper > git bisect bad 25fd2d41b505d0640bdfe67aa77c549de2d3c18a > # bad: [b4bc93bd76d4da32600795cd323c971f00a2e788] Merge tag > 'arm-drivers-5.18' of > git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc > git bisect bad b4bc93bd76d4da32600795cd323c971f00a2e788 > # good: [3fe2f7446f1e029b220f7f650df6d138f91651f2] Merge tag > 'sched-core-2022-03-22' of > git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip > git bisect good 3fe2f7446f1e029b220f7f650df6d138f91651f2 > # good: [182966e1cd74ec0e326cd376de241803ee79741b] Merge tag > 'media/v5.18-1' of > git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media > git bisect good 182966e1cd74ec0e326cd376de241803ee79741b > # good: [49a24e9d9c740d3bd8b1200f225f67d45e3d68a5] Make the SOF > control, PCM and PM code IPC agnostic > git bisect good 49a24e9d9c740d3bd8b1200f225f67d45e3d68a5 > # bad: [8ffa5709e577385a1c8d20fb434cb02732f1d991] Merge tag > 'arm-defconfig-5.18' of > git://git.kernel.org/pub/scm/linux/kernel/git/soc/soc > git bisect bad 8ffa5709e577385a1c8d20fb434cb02732f1d991 > # good: [e6aef3496a00a12e78a571f61d98300cf0a86e6a] Merge tag > 'm68knommu-for-v5.18' of > git://git.kernel.org/pub/scm/linux/kernel/git/gerg/m68knommu > git bisect good e6aef3496a00a12e78a571f61d98300cf0a86e6a > # bad: [9c0e6a89b592f4c4e4d769dbc22d399ab0685159] Merge tag > 'for-linus' of git://git.armlinux.org.uk/~rmk/linux-arm > git bisect bad 9c0e6a89b592f4c4e4d769dbc22d399ab0685159 > # skip: [cafc0eab168917ec9c0cd47d530a40cd40eb2928] ARM: v7m: enable > support for IRQ stacks > git bisect skip cafc0eab168917ec9c0cd47d530a40cd40eb2928 > # skip: [54f481a2308efab49d2b14c3f8263b34fdb1c65e] ARM: remove > old-style irq entry > git bisect skip 54f481a2308efab49d2b14c3f8263b34fdb1c65e > # good: [8cdfdf7fe4fec5a952edfb8927ee7cc639c58184] ARM: export > dump_mem() to other objects > git bisect good 8cdfdf7fe4fec5a952edfb8927ee7cc639c58184 > # bad: [5fe41793bc78d9bb47fea37d1a16984ad6cf294b] ARM: 9176/1: avoid > literal references in inline assembly > git bisect bad 5fe41793bc78d9bb47fea37d1a16984ad6cf294b > # good: [90890f17ccd2aa96350abd1f4d37d4667e09027f] ARM: footbridge: > use GENERIC_IRQ_MULTI_HANDLER > git bisect good 90890f17ccd2aa96350abd1f4d37d4667e09027f > # good: [4e918ab13eaf40f19938659cb5a22c93172778a8] ARM: assembler: add > optimized ldr/str macros to load variables from memory > git bisect good 4e918ab13eaf40f19938659cb5a22c93172778a8 > # skip: [9c46929e7989efacc1dd0a1dd662a839897ea2b6] ARM: implement > THREAD_INFO_IN_TASK for uniprocessor systems > git bisect skip 9c46929e7989efacc1dd0a1dd662a839897ea2b6 > # good: [c2755910373bb5dfb9aa68ba2924036686815c9e] ARM: smp: defer > TPIDRURO update for SMP v6 configurations too > git bisect good c2755910373bb5dfb9aa68ba2924036686815c9e > # skip: [9cf72c358a20b95e040e6a54a03baf6d264e0719] Merge tag > 'arm-irq-and-vmap-stacks-for-rmk' of > git://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux into > devel-stable > git bisect skip 9cf72c358a20b95e040e6a54a03baf6d264e0719 > # skip: [23d9a9280efea105852de358f21d69231992ae73] ARM: 9177/1: > disable vmap'ed stacks on suspend-capable SMP configs > git bisect skip 23d9a9280efea105852de358f21d69231992ae73 > # only skipped commits left to test > # possible first bad commit: > [5fe41793bc78d9bb47fea37d1a16984ad6cf294b] ARM: 9176/1: avoid literal > references in inline assembly > # possible first bad commit: > [23d9a9280efea105852de358f21d69231992ae73] ARM: 9177/1: disable > vmap'ed stacks on suspend-capable SMP configs > # possible first bad commit: > [9cf72c358a20b95e040e6a54a03baf6d264e0719] Merge tag > 'arm-irq-and-vmap-stacks-for-rmk' of > git://git.kernel.org/pub/scm/linux/kernel/git/ardb/linux into > devel-stable > # possible first bad commit: > [cafc0eab168917ec9c0cd47d530a40cd40eb2928] ARM: v7m: enable support > for IRQ stacks > # possible first bad commit: > [9c46929e7989efacc1dd0a1dd662a839897ea2b6] ARM: implement > THREAD_INFO_IN_TASK for uniprocessor systems Maybe Ard and Arnd have some ideas what might be going wrong here. Basically anything trying to use a physical address on stack will fail in weird ways like we've seen for smc and wl1251. Regards, Tony > Best regards, > Yegor > > > > [ 88.408578] rcu: INFO: rcu_sched detected stalls on CPUs/tasks: > > > [ 88.415777] (detected by 0, t=2602 jiffies, g=2529, q=17) > > > [ 88.422026] rcu: All QSes seen, last rcu_sched kthread activity > > > 2602 (-21160--23762), jiffies_till_next_fqs=1, root ->qsmask 0x0 > > > [ 88.434445] rcu: rcu_sched kthread starved for 2602 jiffies! g2529 > > > f0x2 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=0 > > > [ 88.445274] rcu: Unless rcu_sched kthread gets sufficient CPU > > > time, OOM is now expected behavior. > > > [ 88.454859] rcu: RCU grace-period kthread stack dump: > > > [ 88.460446] task:rcu_sched state:R running task stack: > > > 0 pid: 11 ppid: 2 flags:0x00000000 > > > [ 88.471840] __schedule from schedule+0x58/0xcc > > > [ 88.477680] schedule from schedule_timeout+0x78/0xf8 > > > [ 88.483754] schedule_timeout from rcu_gp_fqs_loop+0x108/0x3cc > > > [ 88.490629] rcu_gp_fqs_loop from rcu_gp_kthread+0xa8/0x134 > > > [ 88.497187] rcu_gp_kthread from kthread+0xe4/0x104 > > > [ 88.503061] kthread from ret_from_fork+0x14/0x28 > > > [ 88.508627] Exception stack(0xd0041fb0 to 0xd0041ff8) > > > [ 88.514443] 1fa0: 00000000 > > > 00000000 00000000 00000000 > > > [ 88.523433] 1fc0: 00000000 00000000 00000000 00000000 00000000 > > > 00000000 00000000 00000000 > > > [ 88.532374] 1fe0: 00000000 00000000 00000000 00000000 00000013 00000000 > > > [ 88.539639] rcu: Stack dump where RCU GP kthread last ran: > > > [ 88.545694] NMI backtrace for cpu 0 > > > [ 88.549779] CPU: 0 PID: 58 Comm: kworker/0:8 Not tainted 5.18.0-rc5 #1 > > > [ 88.557103] Hardware name: Generic AM33XX (Flattened Device Tree) > > > [ 88.563822] Workqueue: events dbs_work_handler > > > [ 88.569398] unwind_backtrace from show_stack+0x10/0x14 > > > [ 88.575662] show_stack from dump_stack_lvl+0x58/0x70 > > > [ 88.581627] dump_stack_lvl from nmi_cpu_backtrace+0xe0/0x128 > > > [ 88.588345] nmi_cpu_backtrace from nmi_trigger_cpumask_backtrace+0xec/0x184 > > > [ 88.596339] nmi_trigger_cpumask_backtrace from > > > trigger_single_cpu_backtrace+0x20/0x2c > > > [ 88.605221] trigger_single_cpu_backtrace from > > > rcu_check_gp_kthread_starvation+0xf4/0x148 > > > [ 88.614328] rcu_check_gp_kthread_starvation from > > > rcu_sched_clock_irq+0xdf0/0xf7c > > > [ 88.622778] rcu_sched_clock_irq from update_process_times+0x88/0xc0 > > > [ 88.630182] update_process_times from tick_sched_handle+0x48/0x54 > > > [ 88.637293] tick_sched_handle from tick_sched_timer+0x48/0xac > > > [ 88.643993] tick_sched_timer from __hrtimer_run_queues+0x244/0x4d8 > > > [ 88.651212] __hrtimer_run_queues from hrtimer_interrupt+0x128/0x2c8 > > > [ 88.658582] hrtimer_interrupt from dmtimer_clockevent_interrupt+0x24/0x2c > > > [ 88.666506] dmtimer_clockevent_interrupt from > > > __handle_irq_event_percpu+0x98/0x334 > > > [ 88.675241] __handle_irq_event_percpu from handle_irq_event+0x38/0xc0 > > > [ 88.682749] handle_irq_event from handle_level_irq+0xb4/0x1a8 > > > [ 88.689639] handle_level_irq from handle_irq_desc+0x1c/0x2c > > > [ 88.696253] handle_irq_desc from generic_handle_arch_irq+0x2c/0x64 > > > [ 88.703524] generic_handle_arch_irq from __irq_svc+0x90/0xbc > > > [ 88.710195] Exception stack(0xd0001f58 to 0xd0001fa0) > > > [ 88.715947] 1f40: > > > c01015c8 00000000 > > > [ 88.724939] 1f60: 0eae9000 00000000 fffffffe 60000013 ffffffff > > > d0385d74 00000000 c2702a80 > > > [ 88.733926] 1f80: 00000002 c2702a80 00000000 d0001fa8 c01015c8 > > > c01015d0 60000113 ffffffff > > > [ 88.742765] __irq_svc from __do_softirq+0xa0/0x604 > > > [ 88.748533] __do_softirq from __irq_exit_rcu+0x138/0x178 > > > [ 88.754961] __irq_exit_rcu from irq_exit+0x8/0x28 > > > [ 88.760758] irq_exit from call_with_stack+0x18/0x20 > > > [ 88.766687] call_with_stack from __irq_svc+0x9c/0xbc > > > [ 88.772576] Exception stack(0xd0385d40 to 0xd0385d88) > > > [ 88.778458] 5d40: 00000005 00000488 00000000 00000000 c208c0c0 > > > 00006402 c208b800 c1874ff0 > > > [ 88.787451] 5d60: 00000000 c208c0c0 c1109210 c208c0d8 00000000 > > > d0385d90 c06e068c c06e08a4 > > > [ 88.796305] 5d80: 60000013 ffffffff > > > [ 88.800369] __irq_svc from omap3_noncore_dpll_program+0x3f8/0x5ec > > > [ 88.807588] omap3_noncore_dpll_program from clk_change_rate+0x23c/0x4f8 > > > [ 88.815375] clk_change_rate from clk_core_set_rate_nolock+0x1b0/0x29c > > > [ 88.822936] clk_core_set_rate_nolock from clk_set_rate+0x30/0x64 > > > [ 88.830056] clk_set_rate from _set_opp+0x254/0x51c > > > [ 88.835835] _set_opp from dev_pm_opp_set_rate+0xec/0x228 > > > [ 88.842073] dev_pm_opp_set_rate from __cpufreq_driver_target+0x584/0x700 > > > [ 88.849792] __cpufreq_driver_target from od_dbs_update+0xb4/0x168 > > > [ 88.856953] od_dbs_update from dbs_work_handler+0x2c/0x60 > > > [ 88.863441] dbs_work_handler from process_one_work+0x284/0x72c > > > [ 88.870411] process_one_work from worker_thread+0x28/0x4b0 > > > [ 88.876973] worker_thread from kthread+0xe4/0x104 > > > [ 88.882692] kthread from ret_from_fork+0x14/0x28 > > > [ 88.888225] Exception stack(0xd0385fb0 to 0xd0385ff8) > > > [ 88.893998] 5fa0: 00000000 > > > 00000000 00000000 00000000 > > > [ 88.902971] 5fc0: 00000000 00000000 00000000 00000000 00000000 > > > 00000000 00000000 00000000 > > > [ 88.911888] 5fe0: 00000000 00000000 00000000 00000000 00000013 00000000 > > > > > > Regards, > > > Yegor _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel