From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 76010C677F1 for ; Tue, 17 Jan 2023 13:16:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E6EA16B0071; Tue, 17 Jan 2023 08:16:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E1E346B0073; Tue, 17 Jan 2023 08:16:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C98196B0074; Tue, 17 Jan 2023 08:16:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id B2EF46B0071 for ; Tue, 17 Jan 2023 08:16:47 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 8EF2D160B02 for ; Tue, 17 Jan 2023 13:16:47 +0000 (UTC) X-FDA: 80364340854.09.DF4E535 Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf09.hostedemail.com (Postfix) with ESMTP id 0B365140015 for ; Tue, 17 Jan 2023 13:16:44 +0000 (UTC) Authentication-Results: imf09.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf09.hostedemail.com: domain of mark.rutland@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=mark.rutland@arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673961405; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=XBR2j2WobSI/ezrkv6zcggGb+f68YhK+NbVBn7so8co=; b=PgYzyiEAg3/0gwsehnCMYq0rp3ZQfJHmpY9cpufP554FPeqntlAUoK/drFQupX2je3lwz2 TxfDlvQ9syZTJOWSIOsvNr5UUSpWfY/1HolqGrnvK/gCzzIMS0UIS8aA3nutJwQ+GBYpKx rGqwSH85ZKKMSNSg+yuhxNlpb/uDmKU= ARC-Authentication-Results: i=1; imf09.hostedemail.com; dkim=none; dmarc=pass (policy=none) header.from=arm.com; spf=pass (imf09.hostedemail.com: domain of mark.rutland@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=mark.rutland@arm.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673961405; a=rsa-sha256; cv=none; b=hy3QNPCpmpeXXIjwXs7PXpqngYqSdmf7X77dp6r2fHUBOqTvbTsSCKgPt/smNjkHCLw1Hz mGWRzcjgJGCtVw/9Lv7ar7V9ahthAfVS5MmgzOqTIOA6SMk9ivYBaVxjGKV9vd2ui9SqrB 43+kk5D+5KFPlxMw5BXnR/JvHWXyXdg= Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 020EA4B3; Tue, 17 Jan 2023 05:17:26 -0800 (PST) Received: from FVFF77S0Q05N.cambridge.arm.com (FVFF77S0Q05N.cambridge.arm.com [10.1.31.153]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 223C03F67D; Tue, 17 Jan 2023 05:16:27 -0800 (PST) Date: Tue, 17 Jan 2023 13:16:21 +0000 From: Mark Rutland To: Peter Zijlstra Cc: richard.henderson@linaro.org, ink@jurassic.park.msu.ru, mattst88@gmail.com, vgupta@kernel.org, linux@armlinux.org.uk, nsekhar@ti.com, brgl@bgdev.pl, ulli.kroll@googlemail.com, linus.walleij@linaro.org, shawnguo@kernel.org, Sascha Hauer , kernel@pengutronix.de, festevam@gmail.com, linux-imx@nxp.com, tony@atomide.com, khilman@kernel.org, krzysztof.kozlowski@linaro.org, alim.akhtar@samsung.com, catalin.marinas@arm.com, will@kernel.org, guoren@kernel.org, bcain@quicinc.com, chenhuacai@kernel.org, kernel@xen0n.name, geert@linux-m68k.org, sammy@sammy.net, monstr@monstr.eu, tsbogend@alpha.franken.de, dinguyen@kernel.org, jonas@southpole.se, stefan.kristiansson@saunalahti.fi, shorne@gmail.com, James.Bottomley@hansenpartnership.com, deller@gmx.de, mpe@ellerman.id.au, npiggin@gmail.com, christophe.leroy@csgroup.eu, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, ysato@users.sourceforge.jp, dalias@libc.org, davem@davemloft.net, richard@nod.at, anton.ivanov@cambridgegreys.com, johannes@sipsolutions.net, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, acme@kernel.org, alexander.shishkin@linux.intel.com, jolsa@kernel.org, namhyung@kernel.org, jgross@suse.com, srivatsa@csail.mit.edu, amakhalov@vmware.com, pv-drivers@vmware.com, boris.ostrovsky@oracle.com, chris@zankel.net, jcmvbkbc@gmail.com, rafael@kernel.org, lenb@kernel.org, pavel@ucw.cz, gregkh@linuxfoundation.org, mturquette@baylibre.com, sboyd@kernel.org, daniel.lezcano@linaro.org, lpieralisi@kernel.org, sudeep.holla@arm.com, agross@kernel.org, andersson@kernel.org, konrad.dybcio@linaro.org, anup@brainfault.org, thierry.reding@gmail.com, jonathanh@nvidia.com, jacob.jun.pan@linux.intel.com, atishp@atishpatra.org, Arnd Bergmann , yury.norov@gmail.com, andriy.shevchenko@linux.intel.com, linux@rasmusvillemoes.dk, dennis@kernel.org, tj@kernel.org, cl@linux.com, rostedt@goodmis.org, mhiramat@kernel.org, frederic@kernel.org, paulmck@kernel.org, pmladek@suse.com, senozhatsky@chromium.org, john.ogness@linutronix.de, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, ryabinin.a.a@gmail.com, glider@google.com, andreyknvl@gmail.com, dvyukov@google.com, vincenzo.frascino@arm.com, Andrew Morton , jpoimboe@kernel.org, linux-alpha@vger.kernel.org, linux-kernel@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-omap@vger.kernel.org, linux-samsung-soc@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-ia64@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, openrisc@lists.librecores.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-um@lists.infradead.org, linux-perf-users@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-xtensa@linux-xtensa.org, linux-acpi@vger.kernel.org, linux-pm@vger.kernel.org, linux-clk@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-tegra@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, kasan-dev@googlegroups.com Subject: Re: [PATCH v3 00/51] cpuidle,rcu: Clean up the mess Message-ID: References: <20230112194314.845371875@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 0B365140015 X-Stat-Signature: edym8fzjfuyaw1jusd7rrdbsi34ijsth X-HE-Tag: 1673961404-726419 X-HE-Meta: U2FsdGVkX1/DkqoN/dX8gDOhTpFT9ONlKUlgBvvXggJRTI0yW77xJOtYQTxVbAMxSyUxi1teTWOTSLWU1AuEsK5ZoMQrS8M029uYxKD6WCh+4lednxKu2S9TRlIe5JkGIv8eGozvfh2C26k0YsnyJHqHp8XGi7Euzq/31/ZGZq3vybY/S48dkwj2ALWC7B1J0lCUPsJ/6/vtLLjRqGCSjaMTaa5zCOjABe2o51LYtEul87NDF1cs7Pg0xvtc0xWu8NNTDjSldF8lSRTcvensEFJDCWnlhoQRQCa9DovrhvSkTqA+V3uVALmClrJSZzNmvCrZZYIP4R11MiaLRPkQNlYBDBhblX3uwo8bXYfqAEIfTF5h3QEsoYEGYPUTZ+JRQQqERUKJyfcJ+hDJKycd/OAxGRa4u7mFWBHxdKZKZU8JBIZ6hAMJm+sWxeBaOpke8M675GEPcNCdErk/GE5yXpsVDjlK1fq/4ze4Azp3KBSfhrjoU8S68aizp6cPapNG6w8EVWDLu85Z7TBVHfPFdpAQkCSDjQj86jmdeD4NawEjFcF106xtWJUbdd0IHFnFdUOPHPcw1DgRTAA6dcZO5DzTc6h7e9M2Yd/fGxqyR+1+vKKr0+6yHo790iJJB3f4YTLVAHAjzyr/No5cTm5FThMvHM6KqP4XuRPBiPb68uaahYMjPeXg4lZkpGfSXt8mNz5Cxillylb/lSfnYYdmhduwmr/7f7fKFe3Pn/kya06ZwSwIi3LL0EmClrnOgtTAcfFZ0s/RoEyxlzqcXScalDSNEr/sbemK7ZDLQZttgd3xog0eLHnJIcyop/ZZJYNbtJNluwR0JcgzKpXsyjkbauoXmJHm4WdVTaUChjlPar3f9JhRZuwS1iqFDgLaQJj/5chasRHue9o6s3peMb6YXU+deoUBwhMVaRzuU1Kv0efTF+kAytLC2p2A5xtsQ/yMeAW00Mzu1eWkqX/mHZ+ UWaBuqZP LFM9L6ueENOZERTfpZIZBYTkCfdk28uIy6J2HGTSW1KpfLVFA+Z6hMFyqtZM2EVlxcLkJ/qkEkoQYpQ5OHvdiO6hSn4F/ge+/za8wGvWuuCOIs7LZu9AWzyMhmnj9h6tQs9r7/8xyxTg8vNhLfTtJaEXBNxHhX+mRrCmnlPmFHpL6LYzdL3ytaAJ+LdBKXz6CEXdErz20A1ZblXxQxdJJaLT4u/GI4xXs9Vqcm7o5BrLaaV1KsjsuXscjVEqI40wr9IVnjTcMN5K13xqg+7qlUbqtzVFRJd5MS+RBkZGhbu0SqmSKgufK2kPkWFtdp36b2iK+SxunEPFwpsz4JpqlFEG8kNoj0exlXp/QBKNRIcyHGKM3VwY3t+gntMor2s9AlxulRQ2v1Y6Hldv1FPal2DnvnV9jnWcP/dL8C5vDcIlHhHSGv+dlJGsO8+R/7UAhIe+7dEndYScClAR8oiRZc5lZqeqvh+fLSD9mFmmp7WfO5QanLQdxCU3c8DLdeC5CDvCQ+9bOnGJ0+pu3uQ63BWBb+xPMxsdMwiBx/PZb/zfk2qCrQKlQgwPCGVX+6+vegQkbhXSyGIETMcnCMhRdijuFDDHfMF4/T7PZh/rT0Pmyxb0fJFg2Gnz3tq4PEl9R7dvUArqnPdnjCm3n6i8xsLljPpMqcLRwkajqgFg7BW8Cu5l01tUzgkd4GzP+XyX3yp5WiMpXct6GBAoQXuQI/+1/qjGwQVOHLcDd+aTAmfTl3PKZ+JL9UJXmukKlDOnilBJrGq6QrpK3fkWkKEGaKW23sP7c50pqNzwDL9CR9zzOuAq1h6cRcvtEoQJ4gF7LiQrP9fWBrlyfTtyUfBdvj8gNDy2XzrNvU1nDKugZXX/z9oMLhuvrJveo5uJcA+68I+xniB4YcRc9wM2vdTWvOIpQTzkmRa3OEwV5GzMO4uuyX+q56CCNUPnzYk3Ww27+S9E+ahjnZrcUqpZhc2qwQT6NpgyQ cZjvcgRG VOQS21N1Be9UbsTERjSGy7+tREV8Y0CjgwhMpdMJs7MJZs5yJR9lqNYEzXbLRBt3iBxpNG8V9VEwDRl/6Xlw/Q5q2QzOP46h8wh1YrKNHHD03dErLuS8cU7wQ2VAAuIWAkaDF9iV9ry9Ngo6cfw9n3P2pBnJeDGHVTsuJYnPPqX4DbynU4bJaj5Eg+tWZWRZ8c6IHIj2uAiq0vYtWgAkUoGbF7qHjLEm5rLOrAH6hthDkCYONoxH+km5CmBspQTIPSURi5yTDjvXV3D8DUbDmSple/4lXZ6+uJMx8m6uXTlqlJadHV1YAKKa3NHHKwtnQpZvXPJATXup187/91ODHATp0/K6sxmCYlo8JWjR+72gSOcuTzKb7hOMROMfQ2SuCP6j/2Zr/SaDyqbf7nOWy3Z6BU9ivJK8hfSxEAoLMf8NZ897YmswkAXu4hxQ/93GclHptIHsKinhkHvLmtQIwKkyzmLtJhZ4yh85sVIqsp2nj7UJ3AsgL0cZFrU9TDxsjWSMaN+RE2blVyyhmgQsNN03KF8Pas6exo6kUMVnv/xPQBbbH1lAwmVE9U5vlDk4QsvPcaIbKiCFRl+J2o66TICmlB/FXidUmLYr5RjuYlPWCwnLrBsyRFrin8u0443c3CZHyNzAG080M4UZ+aJCHb8kX/zw2ZpYrq4heUqS8QYrcUsUGzz9Fnwel0ewN2n7tFMWCzGH83ttfQRyeR20GEab8kYj7YLzO+YJqqS9X3ssu/H01Ws6cn0hgU/SVUHpdrLCrkUchQkN+fszzgUG87K7hoA5KaiNwciFTjz6YS/ggbkL+XLjl1R8mcgPO07Nu5eX3/OFhPIZ0J+nUmDj8mHCb0x9sTB04g+uAN9ckdK7P9wFjSxGCZUQnRRyANxOKFdW5EP+QD87/KLivWPP3Pc9dITp3qlSz6b1dD6dfZJTVUcr+VGyC3viumIw5vIxNgBP3w6jvdH4PzQa+/M2jFgSRqr4g v3peC4+2 w8z8JaynBDRLM/OLOpWEV0i0cf6jVL0liVEY6WIkZHzu/gy2SxrYb/xxhMsdrbgLA6wtMWdVMaR9B09ud3QtfDz5yCIwH1YnlQn7/fud3VZFkW1RstO9Aqm0gFRU1pbBPSvYcQEmDz1RP0iZNHdGkH5JngFkXpQysfOPtkKZ3y783L4AZIlenZqyX65As0N7ujWxfj53PvCxMSs8MSfQfzfUJVo5N15EZLYNLYpMTWvhWSzLd1drwWyFZ+Bu9w9fKuoEUlIPZignLeEjk1y7QdhPTn/GxC3wsSDzzIOGOWX2vlthZYaA1JL7uGMT2+PuQEmcSYTPYvZmKHsDwjA4We+PgvV3JTQAoZg5yDRPkUqtia+O8CxHmGcWBwM3pfJ9X8ZjXyGoSSVavB/Tg6rsAXYgl+MTlKNv21jL+Au5E9vMs8YVEYrziW9QnlNRjukv/6otSngJjG+0MiTKe+sHBoLcjDt3LBmaM4pEBXwhSH+24KTP6BHdej5O/w3x9cT9l158rr9C1pIztKQNFgOYnDFmD8omfgjYCLz+XT7V+yrYDVVegHIt0MFMWM3pcW3yMukdBstWWRlbs4D9LKvFdhYstLIjNu7+Pwo1D2liL3jgtUMmpPUVZrC2Xh40Q2n5nAVxsai23aZ1so2VMPRlyO4ug4fP1APQWAOwoVZ1ZEn2M9WQPYQ3yoVOi1kOpySzE5u0bGZZSZQ+CicoRXXdz6b67s6nE95L0esZDMf6TLsDg40E39+QYZnkoe73ktjOZi4j4Mhgr/FgGyj8gvn+4gQCSWfo5cIXAKbisGJOjIfe9uLH49XhrI2CEbfcFNSpEykHvrsidGbB8Q3xgrk5g8GOEy+YYJo/Rjz7LohVgh+3JQWC1oVoJnxMjb8CZIMnafazPhyRImj07ZPYjCp/+AL1I1XVe1PG2rmhXEm7CMIAgYoUaU4QvVdtVkF3gv/kyaP8encWehGiZtcTrg+/rjGMqxiHY oG9u4y+x W01loSktYkFZH/puqMSBa2ewBSuPG8/3xh5CRcas90Ph/12CRr+/rwFfNItUz2WgX7SdbotMcXndAgiw7VRfPdtkS4JX9L1yU2XsdNTU0cmi2EEmJn4ptsUITfa1oPtZ/MrfkERkeQmA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Jan 17, 2023 at 11:26:29AM +0100, Peter Zijlstra wrote: > On Mon, Jan 16, 2023 at 04:59:04PM +0000, Mark Rutland wrote: > > > I'm sorry to have to bear some bad news on that front. :( > > Moo, something had to give.. > > > > IIUC what's happenign here is the PSCI cpuidle driver has entered idle and RCU > > is no longer watching when arm64's cpu_suspend() manipulates DAIF. Our > > local_daif_*() helpers poke lockdep and tracing, hence the call to > > trace_hardirqs_off() and the RCU usage. > > Right, strictly speaking not needed at this point, IRQs should have been > traced off a long time ago. True, but there are some other calls around here that *might* end up invoking RCU stuff (e.g. the MTE code). That all needs a noinstr cleanup too, which I'll sort out as a follow-up. > > I think we need RCU to be watching all the way down to cpu_suspend(), and it's > > cpu_suspend() that should actually enter/exit idle context. That and we need to > > make cpu_suspend() and the low-level PSCI invocation noinstr. > > > > I'm not sure whether 32-bit will have a similar issue or not. > > I'm not seeing 32bit or Risc-V have similar issues here, but who knows, > maybe I missed somsething. I reckon if they do, the core changes here give us the infrastructure to fix them if/when we get reports. > In any case, the below ought to cure the ARM64 case and remove that last > known RCU_NONIDLE() user as a bonus. The below works for me testing on a Juno R1 board with PSCI, using defconfig + CONFIG_PROVE_LOCKING=y + CONFIG_DEBUG_LOCKDEP=y + CONFIG_DEBUG_ATOMIC_SLEEP=y. I'm not sure how to test the LPI / FFH part, but it looks good to me. FWIW: Reviewed-by: Mark Rutland Tested-by: Mark Rutland Sudeep, would you be able to give the LPI/FFH side a spin with the kconfig options above? Thanks, Mark. > > --- > diff --git a/arch/arm64/kernel/cpuidle.c b/arch/arm64/kernel/cpuidle.c > index 41974a1a229a..42e19fff40ee 100644 > --- a/arch/arm64/kernel/cpuidle.c > +++ b/arch/arm64/kernel/cpuidle.c > @@ -67,10 +67,10 @@ __cpuidle int acpi_processor_ffh_lpi_enter(struct acpi_lpi_state *lpi) > u32 state = lpi->address; > > if (ARM64_LPI_IS_RETENTION_STATE(lpi->arch_flags)) > - return CPU_PM_CPU_IDLE_ENTER_RETENTION_PARAM(psci_cpu_suspend_enter, > + return CPU_PM_CPU_IDLE_ENTER_RETENTION_PARAM_RCU(psci_cpu_suspend_enter, > lpi->index, state); > else > - return CPU_PM_CPU_IDLE_ENTER_PARAM(psci_cpu_suspend_enter, > + return CPU_PM_CPU_IDLE_ENTER_PARAM_RCU(psci_cpu_suspend_enter, > lpi->index, state); > } > #endif > diff --git a/arch/arm64/kernel/suspend.c b/arch/arm64/kernel/suspend.c > index e7163f31f716..0fbdf5fe64d8 100644 > --- a/arch/arm64/kernel/suspend.c > +++ b/arch/arm64/kernel/suspend.c > @@ -4,6 +4,7 @@ > #include > #include > #include > +#include > #include > #include > #include > @@ -104,6 +105,10 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long)) > * From this point debug exceptions are disabled to prevent > * updates to mdscr register (saved and restored along with > * general purpose registers) from kernel debuggers. > + * > + * Strictly speaking the trace_hardirqs_off() here is superfluous, > + * hardirqs should be firmly off by now. This really ought to use > + * something like raw_local_daif_save(). > */ > flags = local_daif_save(); > > @@ -120,6 +125,8 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long)) > */ > arm_cpuidle_save_irq_context(&context); > > + ct_cpuidle_enter(); > + > if (__cpu_suspend_enter(&state)) { > /* Call the suspend finisher */ > ret = fn(arg); > @@ -133,8 +140,11 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long)) > */ > if (!ret) > ret = -EOPNOTSUPP; > + > + ct_cpuidle_exit(); > } else { > - RCU_NONIDLE(__cpu_suspend_exit()); > + ct_cpuidle_exit(); > + __cpu_suspend_exit(); > } > > arm_cpuidle_restore_irq_context(&context); > diff --git a/drivers/cpuidle/cpuidle-psci.c b/drivers/cpuidle/cpuidle-psci.c > index 4fc4e0381944..312a34ef28dc 100644 > --- a/drivers/cpuidle/cpuidle-psci.c > +++ b/drivers/cpuidle/cpuidle-psci.c > @@ -69,16 +69,12 @@ static __cpuidle int __psci_enter_domain_idle_state(struct cpuidle_device *dev, > else > pm_runtime_put_sync_suspend(pd_dev); > > - ct_cpuidle_enter(); > - > state = psci_get_domain_state(); > if (!state) > state = states[idx]; > > ret = psci_cpu_suspend_enter(state) ? -1 : idx; > > - ct_cpuidle_exit(); > - > if (s2idle) > dev_pm_genpd_resume(pd_dev); > else > @@ -192,7 +188,7 @@ static __cpuidle int psci_enter_idle_state(struct cpuidle_device *dev, > { > u32 *state = __this_cpu_read(psci_cpuidle_data.psci_states); > > - return CPU_PM_CPU_IDLE_ENTER_PARAM(psci_cpu_suspend_enter, idx, state[idx]); > + return CPU_PM_CPU_IDLE_ENTER_PARAM_RCU(psci_cpu_suspend_enter, idx, state[idx]); > } > > static const struct of_device_id psci_idle_state_match[] = { > diff --git a/drivers/firmware/psci/psci.c b/drivers/firmware/psci/psci.c > index e7bcfca4159f..f3a044fa4652 100644 > --- a/drivers/firmware/psci/psci.c > +++ b/drivers/firmware/psci/psci.c > @@ -462,11 +462,22 @@ int psci_cpu_suspend_enter(u32 state) > if (!psci_power_state_loses_context(state)) { > struct arm_cpuidle_irq_context context; > > + ct_cpuidle_enter(); > arm_cpuidle_save_irq_context(&context); > ret = psci_ops.cpu_suspend(state, 0); > arm_cpuidle_restore_irq_context(&context); > + ct_cpuidle_exit(); > } else { > + /* > + * ARM64 cpu_suspend() wants to do ct_cpuidle_*() itself. > + */ > + if (!IS_ENABLED(CONFIG_ARM64)) > + ct_cpuidle_enter(); > + > ret = cpu_suspend(state, psci_suspend_finisher); > + > + if (!IS_ENABLED(CONFIG_ARM64)) > + ct_cpuidle_exit(); > } > > return ret; > diff --git a/include/linux/cpuidle.h b/include/linux/cpuidle.h > index 630c879143c7..3183aeb7f5b4 100644 > --- a/include/linux/cpuidle.h > +++ b/include/linux/cpuidle.h > @@ -307,7 +307,7 @@ extern s64 cpuidle_governor_latency_req(unsigned int cpu); > #define __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, \ > idx, \ > state, \ > - is_retention) \ > + is_retention, is_rcu) \ > ({ \ > int __ret = 0; \ > \ > @@ -319,9 +319,11 @@ extern s64 cpuidle_governor_latency_req(unsigned int cpu); > if (!is_retention) \ > __ret = cpu_pm_enter(); \ > if (!__ret) { \ > - ct_cpuidle_enter(); \ > + if (!is_rcu) \ > + ct_cpuidle_enter(); \ > __ret = low_level_idle_enter(state); \ > - ct_cpuidle_exit(); \ > + if (!is_rcu) \ > + ct_cpuidle_exit(); \ > if (!is_retention) \ > cpu_pm_exit(); \ > } \ > @@ -330,15 +332,21 @@ extern s64 cpuidle_governor_latency_req(unsigned int cpu); > }) > > #define CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx) \ > - __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, idx, 0) > + __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, idx, 0, 0) > > #define CPU_PM_CPU_IDLE_ENTER_RETENTION(low_level_idle_enter, idx) \ > - __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, idx, 1) > + __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, idx, 1, 0) > > #define CPU_PM_CPU_IDLE_ENTER_PARAM(low_level_idle_enter, idx, state) \ > - __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, state, 0) > + __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, state, 0, 0) > + > +#define CPU_PM_CPU_IDLE_ENTER_PARAM_RCU(low_level_idle_enter, idx, state) \ > + __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, state, 0, 1) > > #define CPU_PM_CPU_IDLE_ENTER_RETENTION_PARAM(low_level_idle_enter, idx, state) \ > - __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, state, 1) > + __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, state, 1, 0) > + > +#define CPU_PM_CPU_IDLE_ENTER_RETENTION_PARAM_RCU(low_level_idle_enter, idx, state) \ > + __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, state, 1, 1) > > #endif /* _LINUX_CPUIDLE_H */