From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7070C7618D for ; Mon, 20 Mar 2023 14:56:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 26A1E6B0074; Mon, 20 Mar 2023 10:56:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 21A266B0078; Mon, 20 Mar 2023 10:56:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0E2896B007B; Mon, 20 Mar 2023 10:56:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 000586B0074 for ; Mon, 20 Mar 2023 10:56:33 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id B319CA0D42 for ; Mon, 20 Mar 2023 14:56:33 +0000 (UTC) X-FDA: 80589577866.01.77CB5DF Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf11.hostedemail.com (Postfix) with ESMTP id F1B0940013 for ; Mon, 20 Mar 2023 14:56:30 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of mark.rutland@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=mark.rutland@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1679324191; a=rsa-sha256; cv=none; b=mvANmGUvGuEvfG/2y8fECyP9eM5MK2EurU54DNt8zaz0PFBTK+z7PpAJ4w6FLm3th3RR97 kfYXoQYXd2Sng9qQAymqLvPpyCmkkxbdNHlkfG4n4iex0d3unS1Hc3z+uo+KZICiJdppYk 3suqWTzoOhMdk4GRlPmtvjbZGWa2qAI= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=none; spf=pass (imf11.hostedemail.com: domain of mark.rutland@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=mark.rutland@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1679324191; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=6uyHtPzbLs9F82J0HVWdgBO/Lsb4YVLVbr58qGA9kOk=; b=adzM8b3Wb47Twyt2PO1wE/Mkxxqg/SG0V5TSsUK6cN6QgIni1BNPMJ5zVMi+HM4nc2GJmc cBgd4azMlihTRVInCEuQ/05X8T0IbkmtIKAsg86SKoUY9VpREt/CFCIyCQAY6CaKCuzJdA WQ7QaNZkAPcyunJLWCTK8LSJ+Wk755E= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 8FD62AD7; Mon, 20 Mar 2023 07:57:13 -0700 (PDT) Received: from FVFF77S0Q05N.cambridge.arm.com (FVFF77S0Q05N.cambridge.arm.com [10.1.35.35]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id AAA413F71E; Mon, 20 Mar 2023 07:56:26 -0700 (PDT) Date: Mon, 20 Mar 2023 14:56:20 +0000 From: Mark Rutland To: Geert Uytterhoeven Cc: Peter Zijlstra , Frederic Weisbecker , Guo Ren , "Rafael J. Wysocki" , Kajetan Puchalski , Tony Lindgren , Ulf Hansson , Ingo Molnar , linux@armlinux.org.uk, linux-imx@nxp.com, linux-kernel@vger.kernel.org, linux-omap@vger.kernel.org, linux-samsung-soc@vger.kernel.org, linux-perf-users@vger.kernel.org, linux-pm@vger.kernel.org, linux-clk@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-tegra@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, kasan-dev@googlegroups.com, linux-renesas-soc@vger.kernel.org Subject: Re: [PATCH v3 07/51] cpuidle,psci: Push RCU-idle into driver Message-ID: References: <20230112194314.845371875@infradead.org> <20230112195539.760296658@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspam-User: X-Rspamd-Queue-Id: F1B0940013 X-Rspamd-Server: rspam01 X-Stat-Signature: 1jcibut8ss99shfp99dftwhnhfdys1fd X-HE-Tag: 1679324190-166360 X-HE-Meta: U2FsdGVkX1+DVHXTV1fqnez4Xsvw8beP6gvgTVVhKDvybBUxyz5GzucS3BtgauinS+basxmrrfMlUF0S5vgqSFqtMU9iFiElE0u0Wszkr5ky5GFCTsQg0khl6bDupm2aMto7rdS5yKs1bDkmkAGUvJAEYCBXVPfBPlUkrTUXSAiZMjOoJfwkZFWwDWrdypG/4SAvOE1e1/I1OgDdhhhChUSwxyY1FbaD8CsIa+HBfO98P2HKv7QBsmyMslRpsFcMTSzmWNyOkYqNrvLqEEwj82i67aiNQw1ibFIunLt5d/XgKhHzbQ8NZy0XIZ5PZo+a951Mtd7/IeKdNfJQJlyruXTBYOAtKsvPdiUHwaD3Fa8RtHPxWv9u8TIefDmW+V8wl5MjD3UBpk9aIsnvf5v5mA9w1kLghUMAse/bglyPmI0oYYwYPUMtqBTaH3PL59MYofwHVpNsGKAhdgud7SwGIKW19UuUVh8wZBNlK0TK3gwwhS/O0/EJy/zJlYHVUA+zTXtSEej4aSfxrfDmh1Zc94Z+Y1TiuiLKfQom93/dvtFnRxpdJB8jpUCQIGvbLfhQ+2YLTfxUwp5Zx61TITcshCiO7ODdjUw1JqlM7BAEZsSk6pFPZiJdYHj1mdqqtQcXh0BSOh/21tMW3f3UfIoH40J5NruSaE8DPSTK4VOT6pB4gjx+hZW82WT3rNZ7ngaPUBnDx+GfAoh1v9IbPRwdHQLmM3fwdlbrzael/R+wAu1OkSyROU3BlVg6otqATA2je+Po4xWCJHAvkqQMZvB+OWbx4xbVn9r6itJks8TbL9lnoVrm3ZjdhtQQTpqj4OATNw2fAOqOX0Sy5fC1YZ4Ll/+vUg4otIMufzdn+86lsvMj6JIGgAHezDsvZO6Gmih3IlRwXcooEg+Kh3H17NAJDElFlwJRZWbsm27DioBF3vCmb/HHefwJppbhT1QdQFSPcDHnzI4m+5stB6nyP3u 2Cwcv3KM iOlFrSUtVY6SJqhY5VIrYl6rPkN7LduJRQNx+cMOo3U80RPy0oQdx7keRNEx+NFjna/uGmjK15vcrbsaeoQtlmIlmP4Fadf4GDX2GO7abQ0El04u1fp1N1Y95agBQQPq0gEp5e4TazZ0MPwhyB7VS8VZHhbt/wwdhxyFCxWSEQx+A79usW/rMnQlIXKN2Zr45v4QHUJ4CQQ5VLoSOgJ9cYEJIFnPo4QllN/72h+ZX6XUJ82zV6eqVk/QvhhmyxQQnSWJ+Qn2W5mBCbicwtDnfgy27aIZhNY3hz+nI6o1JdW5lR1p7kyNUdacRPLb6LFdaD8+SK9EdWgAw+w6ze450CEAhfnxSNGyNv9wVGHG/HasSuMhhcKzl4p4MYlrD8x0cRwWZm1WY6YF+QQFEaTDGpkimLd+s7A7RuPY5/Qa2nGMZF59LUYbnHQog28BMjfAPJu0JdhhbYeu5kyZy+CtyTHkFYLvrmYRLmex9KR5XuLkuHvThpIM1N65Jk0tadSZP2kv+l0zD5u1jhGdhG13YtPFP3DaCV+pmXnv7eUIZDQKHL1148rMs7ppaKB7DgKfJiNMBg582O+78UvcZpB96E4g8yw7xhAl1Zfmc X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hi Geert, On Tue, Mar 07, 2023 at 05:40:08PM +0100, Geert Uytterhoeven wrote: > Hoi Peter, > > (reduced the insane CC list) Helpfully you dropped me from Cc, so I missed this until just now... > On Thu, 12 Jan 2023, Peter Zijlstra wrote: > > Doing RCU-idle outside the driver, only to then temporarily enable it > > again, at least twice, before going idle is daft. > > > > Notably once implicitly through the cpu_pm_*() calls and once > > explicitly doing ct_irq_*_irqon(). > > > > Signed-off-by: Peter Zijlstra (Intel) > > Reviewed-by: Frederic Weisbecker > > Reviewed-by: Guo Ren > > Acked-by: Rafael J. Wysocki > > Tested-by: Kajetan Puchalski > > Tested-by: Tony Lindgren > > Tested-by: Ulf Hansson > > Thanks for your patch, which is now commit e038f7b8028a1d1b ("cpuidle, > psci: Push RCU-idle into driver") in v6.3-rc1. > > I have bisected a PSCI checker regression on Renesas R-Car Gen3/4 SoCs > to commit a01353cf1896ea5b ("cpuidle: Fix ct_idle_*() usage") (the 7 > commits before that do not compile): > > psci_checker: PSCI checker started using 2 CPUs > psci_checker: Starting hotplug tests > psci_checker: Trying to turn off and on again all CPUs > psci: CPU0 killed (polled 0 ms) > Detected PIPT I-cache on CPU0 > CPU0: Booted secondary processor 0x0000000000 [0x411fd073] > psci_checker: Trying to turn off and on again group 0 (CPUs 0-1) > psci: CPU0 killed (polled 0 ms) > Detected PIPT I-cache on CPU0 > CPU0: Booted secondary processor 0x0000000000 [0x411fd073] > psci_checker: Hotplug tests passed OK > psci_checker: Starting suspend tests (10 cycles per state) > psci_checker: CPU 0 entering suspend cycles, states 1 through 1 > psci_checker: CPU 1 entering suspend cycles, states 1 through 1 > ------------[ cut here ]------------ > WARNING: CPU: 1 PID: 177 at kernel/context_tracking.c:141 ct_kernel_exit.constprop.0+0xd8/0xf4 So that's: WARN_ON_ONCE(IS_ENABLED(CONFIG_RCU_EQS_DEBUG) && !user && !is_idle_task(current)); ... and the PSCI checker doens't run in the context of the idle thread, so the warning is correct, and we're violating the expectation of the context tracking code. The PSCI checker is very much a special case, and I'm not sure how we can fix this without removing the warning in the cases we want it. It'd be nicer if we could "queue" the idle into the relevant idle thread. :/ I'm very tempted to say we should just rip the checker code out, rather than contorting the rest of the code to make that work. Thanks, Mark. > Modules linked in: > CPU: 1 PID: 177 Comm: psci_suspend_te Not tainted 6.2.0-rc1-salvator-x-00052-ga01353cf1896 #1415 > Hardware name: Renesas Salvator-X 2nd version board based on r8a77965 (DT) > pstate: 604000c5 (nZCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--) > pc : ct_kernel_exit.constprop.0+0xd8/0xf4 > lr : ct_kernel_exit.constprop.0+0xc8/0xf4 > sp : ffffffc00b73bd30 > x29: ffffffc00b73bd30 x28: ffffff807fbadc90 x27: 0000000000000000 > x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000 > x23: ffffff800981e140 x22: 0000000000000001 x21: 0000000000010000 > x20: ffffffc0086be1d8 x19: ffffff807fbac070 x18: 0000000000000000 > x17: ffffff80083d1000 x16: ffffffc00841fff8 x15: ffffffc00b73b990 > x14: ffffffc00895be78 x13: 0000000000000001 x12: 0000000000000000 > x11: 00000000000001aa x10: 00000000ffffffea x9 : 000000000000000f > x8 : ffffffc00b73bb68 x7 : ffffffc00b73be18 x6 : ffffffc00815ff34 > x5 : ffffffc00a6a0c30 x4 : ffffffc00801ce00 x3 : 0000000000000000 > x2 : ffffffc008dc3070 x1 : ffffffc008dc3078 x0 : 0000000004208040 > Call trace: > ct_kernel_exit.constprop.0+0xd8/0xf4 > ct_idle_enter+0x18/0x20 > psci_enter_idle_state+0xa4/0xfc > suspend_test_thread+0x238/0x2f0 > kthread+0xd8/0xe8 > ret_from_fork+0x10/0x20 > irq event stamp: 0 > hardirqs last enabled at (0): [<0000000000000000>] 0x0 > hardirqs last disabled at (0): [] copy_process+0x608/0x13dc > softirqs last enabled at (0): [] copy_process+0x608/0x13dc > softirqs last disabled at (0): [<0000000000000000>] 0x0 > ---[ end trace 0000000000000000 ]--- > ------------[ cut here ]------------ > WARNING: CPU: 1 PID: 177 at kernel/context_tracking.c:186 ct_kernel_enter.constprop.0+0x78/0xa4 > Modules linked in: > CPU: 1 PID: 177 Comm: psci_suspend_te Tainted: G W 6.2.0-rc1-salvator-x-00052-ga01353cf1896 #1415 > Hardware name: Renesas Salvator-X 2nd version board based on r8a77965 (DT) > pstate: 604000c5 (nZCv daIF +PAN -UAO -TCO -DIT -SSBS BTYPE=--) > pc : ct_kernel_enter.constprop.0+0x78/0xa4 > lr : ct_kernel_enter.constprop.0+0x68/0xa4 > sp : ffffffc00b73bd30 > x29: ffffffc00b73bd30 x28: ffffff807fbadc90 x27: 0000000000000000 > x26: 0000000000000000 x25: 0000000000000000 x24: 0000000000000000 > x23: ffffff800981e140 x22: 0000000000000001 x21: 00000000ffffffa1 > x20: ffffffc0086be1d8 x19: 00000000000000c0 x18: 0000000000000000 > x17: ffffff80083d1000 x16: ffffffc00841fff8 x15: ffffffc00b73b990 > x14: ffffffc00895be78 x13: ffffff800e325180 x12: ffffffc076de9000 > x11: 0000000034d4d91d x10: 0000000000000008 x9 : 0000000000001000 > x8 : ffffffc008012800 x7 : 0000000000000000 x6 : ffffff807fbac070 > x5 : ffffffc008dc3070 x4 : 0000000000000000 x3 : 000000000001a9fc > x2 : 0000000000000003 x1 : ffffffc008dc3070 x0 : 0000000004208040 > Call trace: > ct_kernel_enter.constprop.0+0x78/0xa4 > ct_idle_exit+0x18/0x38 > psci_enter_idle_state+0xdc/0xfc > suspend_test_thread+0x238/0x2f0 > kthread+0xd8/0xe8 > ret_from_fork+0x10/0x20 > irq event stamp: 0 > hardirqs last enabled at (0): [<0000000000000000>] 0x0 > hardirqs last disabled at (0): [] copy_process+0x608/0x13dc > softirqs last enabled at (0): [] copy_process+0x608/0x13dc > softirqs last disabled at (0): [<0000000000000000>] 0x0 > ---[ end trace 0000000000000000 ]--- > psci_checker: Failed to suspend CPU 1: error -1 (requested state 1, cycle 0) > psci_checker: CPU 0 suspend test results: success 0, shallow states 10, errors 0 > mmcblk0rpmb: mmc0:0001 BGSD3R 4.00 MiB, chardev (243:0) > psci_checker: CPU 1 suspend test results: success 0, shallow states 9, errors 1 > psci_checker: 1 error(s) encountered in suspend tests > psci_checker: PSCI checker completed > > > --- > > drivers/cpuidle/cpuidle-psci.c | 9 +++++---- > > 1 file changed, 5 insertions(+), 4 deletions(-) > > > > --- a/drivers/cpuidle/cpuidle-psci.c > > +++ b/drivers/cpuidle/cpuidle-psci.c > > @@ -69,12 +69,12 @@ static int __psci_enter_domain_idle_stat > > return -1; > > > > /* Do runtime PM to manage a hierarchical CPU toplogy. */ > > - ct_irq_enter_irqson(); > > if (s2idle) > > dev_pm_genpd_suspend(pd_dev); > > else > > pm_runtime_put_sync_suspend(pd_dev); > > - ct_irq_exit_irqson(); > > + > > + ct_idle_enter(); > > > > state = psci_get_domain_state(); > > if (!state) > > @@ -82,12 +82,12 @@ static int __psci_enter_domain_idle_stat > > > > ret = psci_cpu_suspend_enter(state) ? -1 : idx; > > > > - ct_irq_enter_irqson(); > > + ct_idle_exit(); > > + > > if (s2idle) > > dev_pm_genpd_resume(pd_dev); > > else > > pm_runtime_get_sync(pd_dev); > > - ct_irq_exit_irqson(); > > > > cpu_pm_exit(); > > > > @@ -240,6 +240,7 @@ static int psci_dt_cpu_init_topology(str > > * of a shared state for the domain, assumes the domain states are all > > * deeper states. > > */ > > + drv->states[state_count - 1].flags |= CPUIDLE_FLAG_RCU_IDLE; > > drv->states[state_count - 1].enter = psci_enter_domain_idle_state; > > drv->states[state_count - 1].enter_s2idle = psci_enter_s2idle_domain_idle_state; > > psci_cpuidle_use_cpuhp = true; > > Gr{oetje,eeting}s, > > Geert > > -- > Geert Uytterhoeven -- There's lots of Linux beyond ia32 -- geert@linux-m68k.org > > In personal conversations with technical people, I call myself a hacker. But > when I'm talking to journalists I just say "programmer" or something like that. > -- Linus Torvalds >