Re: [RFC 0/7] cpuidle: Add poking mechanism to support non-IPI wakeup

From: Leonard Crestez <leonard.crestez@nxp.com>
To: "l.stach@pengutronix.de" <l.stach@pengutronix.de>,
	"marc.zyngier@arm.com" <marc.zyngier@arm.com>,
	Abel Vesa <abel.vesa@nxp.com>, Jacky Bai <ping.bai@nxp.com>
Cc: dl-linux-imx <linux-imx@nxp.com>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	Aisheng Dong <aisheng.dong@nxp.com>,
	"linux-pm@vger.kernel.org" <linux-pm@vger.kernel.org>,
	"lorenzo.pieralisi@arm.com" <lorenzo.pieralisi@arm.com>,
	Fabio Estevam <fabio.estevam@nxp.com>,
	"mark.rutland@arm.com" <mark.rutland@arm.com>,
	"rjw@rjwysocki.net" <rjw@rjwysocki.net>,
	"catalin.marinas@arm.com" <catalin.marinas@arm.com>,
	"will.deacon@arm.com" <will.deacon@arm.com>,
	"robh@kernel.org" <robh@kernel.org>,
	"shawnguo@kernel.org" <shawnguo@kernel.org>,
	"linux-arm-kernel@lists.infradead.org" 
	<linux-arm-kernel@lists.infradead.org>,
	"sudeep.holla@arm.com" <sudeep.holla@arm.com>,
	Anson Huang <anson.huang@nxp.com>,
	"kernel@pengutronix.de" <kernel@pengutronix.de>
Subject: Re: [RFC 0/7] cpuidle: Add poking mechanism to support non-IPI wakeup
Date: Wed, 27 Mar 2019 18:40:07 +0000	[thread overview]
Message-ID: <cb2e5521cc0d29b7c3ac42a6717256ec2e8d35e6.camel@nxp.com> (raw)
In-Reply-To: <85c91392-9cbf-a5fc-b037-3d58f2b0ac9c@arm.com>

On Wed, 2019-03-27 at 17:45 +0000, Marc Zyngier wrote:
> On 27/03/2019 16:06, Lucas Stach wrote:
> > Am Mittwoch, den 27.03.2019, 15:57 +0000 schrieb Marc Zyngier:
> > > On 27/03/2019 15:44, Lucas Stach wrote:
> > > > Am Mittwoch, den 27.03.2019, 13:21 +0000 schrieb Abel Vesa:
> > > > > This work is a workaround I'm looking into (more as a background task)
> > > > > in order to add support for cpuidle on i.MX8MQ based platforms.
> > > > > 
> > > > > The main idea here is getting around the missing GIC wake_request signal
> > > > > (due to integration design issue) by waking up a each individual core through
> > > > > some dedicated SW power-up bits inside the power controller (GPC) right before
> > > > > every IPI is requested for that each individual core.
> > > > 
> > > > Just a general comment, without going into the details of this series:
> > > > this issue is not only affecting IPIs, but also MSIs terminated at the
> > > > GIC. Currently MSIs are terminated at the PCIe core, but terminating
> > > > them at the GIC is clearly preferable, as this allows assigning CPU
> > > > affinity to individual MSIs and lowers IRQ service overhead.
> > > > 
> > > > I'm not sure what the consequences are for upstream Linux support yet,
> > > > but we should keep in mind that having a workaround for IPIs is only
> > > > solving part of the issue.
> > > 
> > > If this erratum is affecting more than just IPIs, then indeed I don't
> > > see how this patch series solves anything.
> > > 
> > > But the erratum documentation seems to imply that only SGIs are
> > > affected, and goes as far as suggesting to use an external interrupt
> > > would solve it. How comes this is not the case? Or is it that anything
> > > directly routed to a redistributor is also affected? This would break
> > > LPIs (and thus MSIs) and PPIs (the CPU timer, among others).
> > 
> > Anything that isn't visible to the GPC and requires the GIC
> > wake_request signal to behave as specified is broken by this erratum.
> 
> I really wonder how a timer interrupt (a PPI, hence not routed through
> the GPC) can wake up the CPU in this case. It really feels like
> something like "program CNTV_CVAL_EL0 to expire at some later point;
> WFI" could result in the CPU going to a deep sleep state, and not
> wake-up at all.

This is already a common issue for cpuidle implementions handled by the
"local-timer-stop" property. imx has other timer blocks in the SOC,
they generate SPIs which are connected to GPC.

> This would indicate that not only cpuidle is broken with this, but
> absolutely every interrupt that is not routed through the GPC.

Yes, cpuidle is broken for irqs not routed through GPC. However:

* All SPIs are connected to GPC in a 1:1 mapping
* This series deals with SGIs
* The timer PPIs are not required; covered by local-timer-stop
* LPIs are currently unused (I understand imx-pci uses SPI by default
from Lucas)

Anything missing?

My understanding is that this wake request feature via GIC is new in v3
and this is maybe why HW team missed it during integration. Older
imx6/7 has GICv2 and has deep idle states which always rely on GPC to
wakeup so the approach can work.

--
Regards,
Leonard