From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2D028C678D4 for ; Tue, 17 Jan 2023 10:26:48 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 41ABD6B0074; Tue, 17 Jan 2023 05:26:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 3C9706B0075; Tue, 17 Jan 2023 05:26:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2446F6B0078; Tue, 17 Jan 2023 05:26:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 15FB06B0074 for ; Tue, 17 Jan 2023 05:26:47 -0500 (EST) Received: from smtpin30.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id CDA9CA0588 for ; Tue, 17 Jan 2023 10:26:46 +0000 (UTC) X-FDA: 80363912412.30.86977F4 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf08.hostedemail.com (Postfix) with ESMTP id 0258D160017 for ; Tue, 17 Jan 2023 10:26:43 +0000 (UTC) Authentication-Results: imf08.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=GQsHqdtN; spf=none (imf08.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1673951205; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oxrJBZkAaCeFAHBiyHLFWLeqn2s4gXmFVHT68O/MsFk=; b=epZXYzHZdQ4DcWoRjuFYm+46x7H+fqNq/IsTv/rcVVnmLOa1cfFVCF2r2oGDQvWOVkza7F ExYmJSw9I4Pi5Z+keNip373rQ5coyXNtItwwVLTKZe9IIn0K7eqKSBLj0L+BS7z50RRFun fQu9tb9EIvnPaUlVN7eZpe+jZRz8/Lw= ARC-Authentication-Results: i=1; imf08.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=GQsHqdtN; spf=none (imf08.hostedemail.com: domain of peterz@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=peterz@infradead.org; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1673951205; a=rsa-sha256; cv=none; b=BvieL7VZ62hWSX++hJREMWwrF22z1YruF9bF7OO4Gh6YZW3bPUjstjUdCuoMSvxa6zOddB FbO3b2OaokrG1e2vsLBjOzgZhJRO/RqFMcaXbJRG0yRzOOLdkI4P53jwFJMae/+B0RKWEL ZQ8uHt+lO5Xk7fD+Eshb/V/vJKIx9RY= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:Sender:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description; bh=oxrJBZkAaCeFAHBiyHLFWLeqn2s4gXmFVHT68O/MsFk=; b=GQsHqdtN/iREDQ8vi1O4Be1MrZ rseA9/3ogzxQ6xZE4SqLUZG4sV/QDZjvMovg65N0xIIgCnRPbkND1+nwy51nHnBc5DWNJS4XUBAN9 DGhO4XvtgUfXh3DGTlKQTq1yWaZoEocA9mKp7m7181bsUQeP0yUWn9Ocr+1xM7l1a0UxxyRm82Zw+ z3mRAtrbo2NEuxCbyqDel031QfZM7HOcrBLOGqBY1RekqMov5sENJSTJquxMwFQpH1Z5aFuaq2SxP yues13giahhBFOzG58JwmbeFuKGXFy4M+qN2H/Xcy0ClAmlPum6BOEWDZwYNw11AStmRYs4JMNKUo mE/sBfbw==; Received: from j130084.upc-j.chello.nl ([24.132.130.84] helo=noisy.programming.kicks-ass.net) by casper.infradead.org with esmtpsa (Exim 4.94.2 #2 (Red Hat Linux)) id 1pHjB0-009alj-Ie; Tue, 17 Jan 2023 10:26:46 +0000 Received: from hirez.programming.kicks-ass.net (hirez.programming.kicks-ass.net [192.168.1.225]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) by noisy.programming.kicks-ass.net (Postfix) with ESMTPS id E6A7A300129; Tue, 17 Jan 2023 11:26:29 +0100 (CET) Received: by hirez.programming.kicks-ass.net (Postfix, from userid 1000) id C3475201C94B3; Tue, 17 Jan 2023 11:26:29 +0100 (CET) Date: Tue, 17 Jan 2023 11:26:29 +0100 From: Peter Zijlstra To: Mark Rutland Cc: richard.henderson@linaro.org, ink@jurassic.park.msu.ru, mattst88@gmail.com, vgupta@kernel.org, linux@armlinux.org.uk, nsekhar@ti.com, brgl@bgdev.pl, ulli.kroll@googlemail.com, linus.walleij@linaro.org, shawnguo@kernel.org, Sascha Hauer , kernel@pengutronix.de, festevam@gmail.com, linux-imx@nxp.com, tony@atomide.com, khilman@kernel.org, krzysztof.kozlowski@linaro.org, alim.akhtar@samsung.com, catalin.marinas@arm.com, will@kernel.org, guoren@kernel.org, bcain@quicinc.com, chenhuacai@kernel.org, kernel@xen0n.name, geert@linux-m68k.org, sammy@sammy.net, monstr@monstr.eu, tsbogend@alpha.franken.de, dinguyen@kernel.org, jonas@southpole.se, stefan.kristiansson@saunalahti.fi, shorne@gmail.com, James.Bottomley@hansenpartnership.com, deller@gmx.de, mpe@ellerman.id.au, npiggin@gmail.com, christophe.leroy@csgroup.eu, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, hca@linux.ibm.com, gor@linux.ibm.com, agordeev@linux.ibm.com, borntraeger@linux.ibm.com, svens@linux.ibm.com, ysato@users.sourceforge.jp, dalias@libc.org, davem@davemloft.net, richard@nod.at, anton.ivanov@cambridgegreys.com, johannes@sipsolutions.net, tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, hpa@zytor.com, acme@kernel.org, alexander.shishkin@linux.intel.com, jolsa@kernel.org, namhyung@kernel.org, jgross@suse.com, srivatsa@csail.mit.edu, amakhalov@vmware.com, pv-drivers@vmware.com, boris.ostrovsky@oracle.com, chris@zankel.net, jcmvbkbc@gmail.com, rafael@kernel.org, lenb@kernel.org, pavel@ucw.cz, gregkh@linuxfoundation.org, mturquette@baylibre.com, sboyd@kernel.org, daniel.lezcano@linaro.org, lpieralisi@kernel.org, sudeep.holla@arm.com, agross@kernel.org, andersson@kernel.org, konrad.dybcio@linaro.org, anup@brainfault.org, thierry.reding@gmail.com, jonathanh@nvidia.com, jacob.jun.pan@linux.intel.com, atishp@atishpatra.org, Arnd Bergmann , yury.norov@gmail.com, andriy.shevchenko@linux.intel.com, linux@rasmusvillemoes.dk, dennis@kernel.org, tj@kernel.org, cl@linux.com, rostedt@goodmis.org, mhiramat@kernel.org, frederic@kernel.org, paulmck@kernel.org, pmladek@suse.com, senozhatsky@chromium.org, john.ogness@linutronix.de, juri.lelli@redhat.com, vincent.guittot@linaro.org, dietmar.eggemann@arm.com, bsegall@google.com, mgorman@suse.de, bristot@redhat.com, vschneid@redhat.com, ryabinin.a.a@gmail.com, glider@google.com, andreyknvl@gmail.com, dvyukov@google.com, vincenzo.frascino@arm.com, Andrew Morton , jpoimboe@kernel.org, linux-alpha@vger.kernel.org, linux-kernel@vger.kernel.org, linux-snps-arc@lists.infradead.org, linux-omap@vger.kernel.org, linux-samsung-soc@vger.kernel.org, linux-csky@vger.kernel.org, linux-hexagon@vger.kernel.org, linux-ia64@vger.kernel.org, loongarch@lists.linux.dev, linux-m68k@lists.linux-m68k.org, linux-mips@vger.kernel.org, openrisc@lists.librecores.org, linux-parisc@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-s390@vger.kernel.org, linux-sh@vger.kernel.org, sparclinux@vger.kernel.org, linux-um@lists.infradead.org, linux-perf-users@vger.kernel.org, virtualization@lists.linux-foundation.org, linux-xtensa@linux-xtensa.org, linux-acpi@vger.kernel.org, linux-pm@vger.kernel.org, linux-clk@vger.kernel.org, linux-arm-msm@vger.kernel.org, linux-tegra@vger.kernel.org, linux-arch@vger.kernel.org, linux-mm@kvack.org, linux-trace-kernel@vger.kernel.org, kasan-dev@googlegroups.com Subject: Re: [PATCH v3 00/51] cpuidle,rcu: Clean up the mess Message-ID: References: <20230112194314.845371875@infradead.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 0258D160017 X-Rspam-User: X-Stat-Signature: iok8d9jxsdstbds4p8ycw5gnoxeax9kd X-HE-Tag: 1673951203-750717 X-HE-Meta: U2FsdGVkX1/MN11c71Kxp8F4JW/O9gQvgzn4N590t9WxVgizmRWXdj9E9bOeILb7KiUwolwgtyiJtBZcaoeIqpHqB0VDpGWhARSHjIeYGHSJmm4yCZMCzNF2bve/mNfVJ0t2a/XTmVJxyqfvHaFODsjwTZvFR5nHY3eeWFlvHbDSZbb1Zyqvp45QfL8do+h60fVNZTskrAWPG/x+7hMgLAIXjIKZd1Nwz1IkbUsYqdJVenXqQb30d0xIN3IR84DlBOw1TjbbTGwuNkwKkBlCtLSOqTEYnPcXvGGyPHvIxgk4s49gHEKsy0oTDGu6ZBhuq3a3oNRDN6LxJGLf36lKR49gpmcjqjcSOK+cmesiTI/YBjunmWlOnaraqFdf8L6kVvLeqZJiX8IRnMtZkudLGdpAW2Sc2jE+TVZL03800IrSa8JPA9O/RRS0UJE8XeRC6Ub28la+N+7Iv9Wy+I49EmOYL+ZGroDyPV50Om3PdgYpjd6JsdVa3TY2O7N8P37pO0b2ii6ZGW1O4/PE3fblYXIdSYGKK5+QEy6blOOFl8uN3PtWyVfQMt0VfLDHp1JenIJEAJMoIy1BW+IkD9au/lx88iL4KYvnATGHFkYlunEy3n32JHZfjZSNLGawrw/Owo3dPDLF3BJnWw65qjW+1fb+8xdEYwf6lcob8dDImN5oLLrpfbgdviCFwgKekotTwgFfN9pfclbzyoHKUyy/8bk0lDIh8eawr6QXxn1ziCvTpa24oXpsArJiVkbqeziRpb3PS8qUwczBptZjHj7Lr1LC5dRIClA/5q8rD8/GQO4cGsYmK7yV3d8gMHnwwi40eVjn3iNA3G0WLzvXiwTMaJnDvCj46xwvdUc+GgNyIvSUDJqhiKWKprePTqsC9MAI8VapYsjCNnHVYwhD4zbtqoK/gV3p0HAbd40L7EVaR4IEdJx+/Q47QUAoNhq+K6EdftA/CKAm1UP9p5OFPwL E1SHSr41 uH+23bVRhyaW/KoAEh7vAnyh1RsZTniPJOMLUEas2nzyTrLyBsomqII+2pRsVG8QlVYsfqjl3Otcn0Zyc652W0grxMfN0PFRxYDmoUV4cadeJf4kZ5jGiwS/ruFqH2NKx19mIwIAKuFbB280Vh3qoMVRZ+sMWCAbD4dBDmz88h3ts5bBMCLIBfCscVko9XbYQd/S8fQ0p+h1RFRQt8JJ1nwQB9gKjf5DQAsheVIBjRHmRbkox1AJU3dkq2p4vjpN8AqKIxcrJQSjD41afHn2SljntUxiGTqQAXfOyheC2TO7g366Yate5UWXXW6uxxCp9D87VeqUAB3jgJgZZ5RwM4Wv2Cnj46d+I9vAA6u2TDS02ByVqMTKKG1Qg8S6MrZBvNmlh64r6eutwW/XWClv/wimEq8QBJ3fIA+ebd217P6im/0bIsCL3JI/Ky8p8jh2rw9KGIFKZeKKSq86MXwayOc+AX399THDpwlyzXigwOod4ApLqAx3plGz6oInBHKPizgjZbIlCNr6Os4WRbKngMxJzmlYm0+g9yUzlTOYbuFcqHra07caBnUVpN4lt7rSaoGJD7gvVHv0QP1HFqDyIkxOpFy6JTcin/t0KTnvAqEZUk2mySJ6VirAUg2Mft3+PUt7ybbrYd99SZ71h08sM3PGXHiSB6jfRS/mJG2DyQchhq9exog88oQe6mYd4nvI9ri30uSMWnPRhMqFqrFaRbW0s4z8uo5YHG+4TLyYOY2tp2dYQQPkeDSWkMNLnFXfrxQtcNuvLsaUKalKACiav4AS4pdrSWQ+3hmsiQmpmIIFlHerYE7WmBt0/BhhkXgxejy9QwQECizxiXHbYlOBELMCAdtk3Fu8xuTeYH2zdXTtYQ7pxgbSOAYv9Hdnd7VJpdTo3WCQi174HUuh7cti1a/OekB/2wlu6Prek2Y3AGXikCb8uKuCFVI2srk4KGccet1nN9oBbtu5WNS5FEvcwfA2jAxWE i4Mu0TU9 2MejDVbsUg7wAgqcKAtIF80HnoJECXRxFHySljTE2gHU2/MjmUwxKjXGxD11XV6OHn+lchdd3QO1owenL2Avf54adpPl/kEY9uwwgvAbfdMXCz310qxzN0oL02kKlRvv6qbFJOVLf8eDXPO2MpcsedOjlr577pf/QbwBUsENwyDfHHfjdNj+A9vQBhBWRLq3JFtk5BmqVzcJFht3sh5MhjFbP0q73c4Oh5ZzrrR7b+8s4YxnOR+N0WtThDEgRrr2LZ3fXaWJmxvVTmhZ/78oDhvFGm3KaoBBxoZ4zTqtlH6z4pQcYGZlAMdrrnb8NZOVjfwgSGB81epAlzMMY0RrMetUgs78oQn9sNhFSbed24Yrvj6h8A8Kqi+G9XIjQcmVG3grGJV/Bla6gGJT0Ec8ZX5obryFUmFTpqgPiIHlOROmOEh2K7r8QBJ0RuDDvrPFKLsBZGwG0be9s1YwLAfiUrW9gWHSmNFU8b/D530Cw88LZSJzpS7DeCpPoDaRsxQpnj7VIXx1mN2rx/OZGoAmCRtrTwLGr/qVb7EdvyQpRmN5dbiXNhc3hKwuSNENlPZdlzVQgw7MfCn/sxq+YlBne+fOuSGcC6DixxcCQZ9Cn5w7qNIXHNqjn2CcMqonq7DlnBcu2vviiYmelDrK3y5CQoUTq5fN9Y3NWwxkxR7P8iKOYp1D6v0SmtcuZoTq3qbVwMf95DqWHDcXpzzilbO7xEnKbD0O6vB6dRlIK4H3ueTEY82d0XjE7lOVtmVxLwwOZ/Fb3Y/wTHZDvBh/5/zK6F4Rgig8h4E+Ey/+w6Mvcc+RkyDtfVfASCjMIrGoGOxi6jM/G7rPS2XY8Gr0mAR6X//R9NENqRxYKBZtgmJm2sZjbSLeCbNsMoXEVXVpk7tQQVsOLmTESvRzxe1whQvKjavBdKhwiCnv70yZ2U94oW2EVGlb2I6sYIr7l9G4qxmp3Owahf3GjQyPqIb4SE7gt8SrvRQRd ehFlvx9f bMyVIHryJ9OVHY3UxREJG5CvFTktcCpEcj4SsXzhLRmI8kA9Ys1e1Z0tZVh0VN+xJKvwodiy/1xN/Qcoeo+NHrZkDhPBox02oGS5KBK25ObYxg24Buug4v+ng3Qa2gRO8eLErSeDWlYDi9qQjdj0Yh39sVBPohRqpxIzwUV+QCY5yLcSyaqhSVldu4306DKb6D0GhwF9bRQeRBbzq6vRVM7phj3SyM3bgdPvkq3/Ozb9BcC92AJthPz2hf4gtX0LP15UIqsFcrf+42uzx/+iXnUXgWu5a9HtJG9DhShtpCpUY8MGH32MdK01zU+SHoKqFl3tQlrR3/qVh/pvhdoZkP3KLPXbsZlfX6YV7IXDMOxumfkDt6jf1wL2oeEhJSz6e816fKyU7vxS5u02MWuFm24vF64F1CaZ0AfEl24cSGQWdjYtIspeiFtj2H5l9AU8roHSGHbQtffZhYT8T9YEpCU3XF9k7/A0JFS/qMQVmGe3Cy3ZnS9NFkphHhqmPH/lHfrr/8b5yZiSaGvcsdclNOfIeq4ViNWAM9unO5u2dVgQkrQ9k+X3La8H8QKKpGknecWEi+ZAVSvRRMCn8kpQ/kpjYy6JDz2+3bv18tN7ol9/XP6agv/iPWhw0nrghc6AH1VTW9XyeMiDgpBJKJpNtO5bL1nP6mTiAS99HCQOwPtAJeWuFnh2N6XUBYeoNQC/b7xCnhHHRjQtT0MBx/vlA1YoVvJyAEuOa0geR5pnVPLMpyMvGxYaj9DtdVM8ZLKve93gNw7PD9Q/94tHMX2ePPegPomTrQ1gpofNKBM3BWJa03IMaARrMmzV1G7Om2Zwash4E/SOzqNgUG1Lut6rBqTrt7ZjkE7zaC53tu10DmjdML6P5gtWOTc478riVw5DdjNprpPPLFjJD72fBnRNPyZFRNKW34/WQ75stIaUmVfYHUHomoWxGAwIbnHcckI+qsqnIZiyA7U3Lk6wsxdK5M0ZQlbMC iNCl92vB NjBvUTgaGhsUHYCn0t5tnd2BuKD/RdE0x8Jb9ettWJvd1+CniBEgD0MT5WSr+xusZ/82EdcTUhZw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Jan 16, 2023 at 04:59:04PM +0000, Mark Rutland wrote: > I'm sorry to have to bear some bad news on that front. :( Moo, something had to give.. > IIUC what's happenign here is the PSCI cpuidle driver has entered idle and RCU > is no longer watching when arm64's cpu_suspend() manipulates DAIF. Our > local_daif_*() helpers poke lockdep and tracing, hence the call to > trace_hardirqs_off() and the RCU usage. Right, strictly speaking not needed at this point, IRQs should have been traced off a long time ago. > I think we need RCU to be watching all the way down to cpu_suspend(), and it's > cpu_suspend() that should actually enter/exit idle context. That and we need to > make cpu_suspend() and the low-level PSCI invocation noinstr. > > I'm not sure whether 32-bit will have a similar issue or not. I'm not seeing 32bit or Risc-V have similar issues here, but who knows, maybe I missed somsething. In any case, the below ought to cure the ARM64 case and remove that last known RCU_NONIDLE() user as a bonus. --- diff --git a/arch/arm64/kernel/cpuidle.c b/arch/arm64/kernel/cpuidle.c index 41974a1a229a..42e19fff40ee 100644 --- a/arch/arm64/kernel/cpuidle.c +++ b/arch/arm64/kernel/cpuidle.c @@ -67,10 +67,10 @@ __cpuidle int acpi_processor_ffh_lpi_enter(struct acpi_lpi_state *lpi) u32 state = lpi->address; if (ARM64_LPI_IS_RETENTION_STATE(lpi->arch_flags)) - return CPU_PM_CPU_IDLE_ENTER_RETENTION_PARAM(psci_cpu_suspend_enter, + return CPU_PM_CPU_IDLE_ENTER_RETENTION_PARAM_RCU(psci_cpu_suspend_enter, lpi->index, state); else - return CPU_PM_CPU_IDLE_ENTER_PARAM(psci_cpu_suspend_enter, + return CPU_PM_CPU_IDLE_ENTER_PARAM_RCU(psci_cpu_suspend_enter, lpi->index, state); } #endif diff --git a/arch/arm64/kernel/suspend.c b/arch/arm64/kernel/suspend.c index e7163f31f716..0fbdf5fe64d8 100644 --- a/arch/arm64/kernel/suspend.c +++ b/arch/arm64/kernel/suspend.c @@ -4,6 +4,7 @@ #include #include #include +#include #include #include #include @@ -104,6 +105,10 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long)) * From this point debug exceptions are disabled to prevent * updates to mdscr register (saved and restored along with * general purpose registers) from kernel debuggers. + * + * Strictly speaking the trace_hardirqs_off() here is superfluous, + * hardirqs should be firmly off by now. This really ought to use + * something like raw_local_daif_save(). */ flags = local_daif_save(); @@ -120,6 +125,8 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long)) */ arm_cpuidle_save_irq_context(&context); + ct_cpuidle_enter(); + if (__cpu_suspend_enter(&state)) { /* Call the suspend finisher */ ret = fn(arg); @@ -133,8 +140,11 @@ int cpu_suspend(unsigned long arg, int (*fn)(unsigned long)) */ if (!ret) ret = -EOPNOTSUPP; + + ct_cpuidle_exit(); } else { - RCU_NONIDLE(__cpu_suspend_exit()); + ct_cpuidle_exit(); + __cpu_suspend_exit(); } arm_cpuidle_restore_irq_context(&context); diff --git a/drivers/cpuidle/cpuidle-psci.c b/drivers/cpuidle/cpuidle-psci.c index 4fc4e0381944..312a34ef28dc 100644 --- a/drivers/cpuidle/cpuidle-psci.c +++ b/drivers/cpuidle/cpuidle-psci.c @@ -69,16 +69,12 @@ static __cpuidle int __psci_enter_domain_idle_state(struct cpuidle_device *dev, else pm_runtime_put_sync_suspend(pd_dev); - ct_cpuidle_enter(); - state = psci_get_domain_state(); if (!state) state = states[idx]; ret = psci_cpu_suspend_enter(state) ? -1 : idx; - ct_cpuidle_exit(); - if (s2idle) dev_pm_genpd_resume(pd_dev); else @@ -192,7 +188,7 @@ static __cpuidle int psci_enter_idle_state(struct cpuidle_device *dev, { u32 *state = __this_cpu_read(psci_cpuidle_data.psci_states); - return CPU_PM_CPU_IDLE_ENTER_PARAM(psci_cpu_suspend_enter, idx, state[idx]); + return CPU_PM_CPU_IDLE_ENTER_PARAM_RCU(psci_cpu_suspend_enter, idx, state[idx]); } static const struct of_device_id psci_idle_state_match[] = { diff --git a/drivers/firmware/psci/psci.c b/drivers/firmware/psci/psci.c index e7bcfca4159f..f3a044fa4652 100644 --- a/drivers/firmware/psci/psci.c +++ b/drivers/firmware/psci/psci.c @@ -462,11 +462,22 @@ int psci_cpu_suspend_enter(u32 state) if (!psci_power_state_loses_context(state)) { struct arm_cpuidle_irq_context context; + ct_cpuidle_enter(); arm_cpuidle_save_irq_context(&context); ret = psci_ops.cpu_suspend(state, 0); arm_cpuidle_restore_irq_context(&context); + ct_cpuidle_exit(); } else { + /* + * ARM64 cpu_suspend() wants to do ct_cpuidle_*() itself. + */ + if (!IS_ENABLED(CONFIG_ARM64)) + ct_cpuidle_enter(); + ret = cpu_suspend(state, psci_suspend_finisher); + + if (!IS_ENABLED(CONFIG_ARM64)) + ct_cpuidle_exit(); } return ret; diff --git a/include/linux/cpuidle.h b/include/linux/cpuidle.h index 630c879143c7..3183aeb7f5b4 100644 --- a/include/linux/cpuidle.h +++ b/include/linux/cpuidle.h @@ -307,7 +307,7 @@ extern s64 cpuidle_governor_latency_req(unsigned int cpu); #define __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, \ idx, \ state, \ - is_retention) \ + is_retention, is_rcu) \ ({ \ int __ret = 0; \ \ @@ -319,9 +319,11 @@ extern s64 cpuidle_governor_latency_req(unsigned int cpu); if (!is_retention) \ __ret = cpu_pm_enter(); \ if (!__ret) { \ - ct_cpuidle_enter(); \ + if (!is_rcu) \ + ct_cpuidle_enter(); \ __ret = low_level_idle_enter(state); \ - ct_cpuidle_exit(); \ + if (!is_rcu) \ + ct_cpuidle_exit(); \ if (!is_retention) \ cpu_pm_exit(); \ } \ @@ -330,15 +332,21 @@ extern s64 cpuidle_governor_latency_req(unsigned int cpu); }) #define CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx) \ - __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, idx, 0) + __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, idx, 0, 0) #define CPU_PM_CPU_IDLE_ENTER_RETENTION(low_level_idle_enter, idx) \ - __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, idx, 1) + __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, idx, 1, 0) #define CPU_PM_CPU_IDLE_ENTER_PARAM(low_level_idle_enter, idx, state) \ - __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, state, 0) + __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, state, 0, 0) + +#define CPU_PM_CPU_IDLE_ENTER_PARAM_RCU(low_level_idle_enter, idx, state) \ + __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, state, 0, 1) #define CPU_PM_CPU_IDLE_ENTER_RETENTION_PARAM(low_level_idle_enter, idx, state) \ - __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, state, 1) + __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, state, 1, 0) + +#define CPU_PM_CPU_IDLE_ENTER_RETENTION_PARAM_RCU(low_level_idle_enter, idx, state) \ + __CPU_PM_CPU_IDLE_ENTER(low_level_idle_enter, idx, state, 1, 1) #endif /* _LINUX_CPUIDLE_H */