From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=HEADER_FROM_DIFFERENT_DOMAINS, MAILING_LIST_MULTI,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63C93C43381 for ; Wed, 27 Mar 2019 17:12:16 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 3B486206BA for ; Wed, 27 Mar 2019 17:12:16 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728069AbfC0RMO (ORCPT ); Wed, 27 Mar 2019 13:12:14 -0400 Received: from metis.ext.pengutronix.de ([85.220.165.71]:33127 "EHLO metis.ext.pengutronix.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727234AbfC0RMO (ORCPT ); Wed, 27 Mar 2019 13:12:14 -0400 Received: from kresse.hi.pengutronix.de ([2001:67c:670:100:1d::2a]) by metis.ext.pengutronix.de with esmtp (Exim 4.89) (envelope-from ) id 1h9C5d-0007FP-8C; Wed, 27 Mar 2019 18:11:49 +0100 Message-ID: <1553706708.2561.42.camel@pengutronix.de> Subject: Re: [RFC 0/7] cpuidle: Add poking mechanism to support non-IPI wakeup From: Lucas Stach To: Leonard Crestez , "marc.zyngier@arm.com" , Richard Zhu Cc: Fabio Estevam , Cosmin Samoila , Robin Gong , Mircea Pop , Daniel Baluta , "catalin.marinas@arm.com" , Aisheng Dong , "shawnguo@kernel.org" , Robert Chiras , Anson Huang , Jun Li , Abel Vesa , "robh@kernel.org" , Zening Wang , dl-linux-imx , BOUGH CHEN , Horia Geanta , Peter Chen , Joakim Zhang , "rjw@rjwysocki.net" , Leo Zhang , Shenwei Wang , "linux-pm@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , Ranjani Vaidyanathan , Han Xu , "will.deacon@arm.com" , Iuliana Prodan , "sudeep.holla@arm.com" , "lorenzo.pieralisi@arm.com" , Jacky Bai , "linux-kernel@vger.kernel.org" , "mark.rutland@arm.com" , Peng Fan , "kernel@pengutronix.de" , Viorel Suman Date: Wed, 27 Mar 2019 18:11:48 +0100 In-Reply-To: References: <1553692845-20983-1-git-send-email-abel.vesa@nxp.com> <1553701479.2561.38.camel@pengutronix.de> <564216aa-1144-71de-e887-00c58f466bf5@arm.com> <1553702767.2561.40.camel@pengutronix.de> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.22.6-1+deb9u1 Mime-Version: 1.0 Content-Transfer-Encoding: 7bit X-SA-Exim-Connect-IP: 2001:67c:670:100:1d::2a X-SA-Exim-Mail-From: l.stach@pengutronix.de X-SA-Exim-Scanned: No (on metis.ext.pengutronix.de); SAEximRunCond expanded to false X-PTX-Original-Recipient: linux-kernel@vger.kernel.org Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Am Mittwoch, den 27.03.2019, 17:00 +0000 schrieb Leonard Crestez: > On Wed, 2019-03-27 at 17:06 +0100, Lucas Stach wrote: > > Am Mittwoch, den 27.03.2019, 15:57 +0000 schrieb Marc Zyngier: > > > On 27/03/2019 15:44, Lucas Stach wrote: > > > > Am Mittwoch, den 27.03.2019, 13:21 +0000 schrieb Abel Vesa: > > > > > This work is a workaround I'm looking into (more as a > > > > > background task) > > > > > in order to add support for cpuidle on i.MX8MQ based > > > > > platforms. > > > > > > > > > > The main idea here is getting around the missing GIC > > > > > wake_request signal > > > > > (due to integration design issue) by waking up a each > > > > > individual core through > > > > > some dedicated SW power-up bits inside the power controller > > > > > (GPC) right before > > > > > every IPI is requested for that each individual core. > > > > > > > > Just a general comment, without going into the details of this > > > > series: > > > > this issue is not only affecting IPIs, but also MSIs terminated > > > > at the > > > > GIC. Currently MSIs are terminated at the PCIe core, but > > > > terminating > > > > them at the GIC is clearly preferable, as this allows assigning > > > > CPU > > > > affinity to individual MSIs and lowers IRQ service overhead. > > > > > > > > I'm not sure what the consequences are for upstream Linux > > > > support yet, > > > > but we should keep in mind that having a workaround for IPIs is > > > > only > > > > solving part of the issue. > > > > > > If this erratum is affecting more than just IPIs, then indeed I > > > don't > > > see how this patch series solves anything. > > > > > > But the erratum documentation seems to imply that only SGIs are > > > affected, and goes as far as suggesting to use an external > > > interrupt > > > would solve it. How comes this is not the case? Or is it that > > > anything > > > directly routed to a redistributor is also affected? This would > > > break > > > LPIs (and thus MSIs) and PPIs (the CPU timer, among others). > > > > > > What is the *exact* status of this thing? I have the ugly feeling > > > that > > > the true workaround is just to disable cpuidle. > > > > As far as I understand the erratum, the basic issue is that the GIC > > wake_request signals are not connected to the GPC (the > > CPU/peripheral > > power sequencer). The SPIs are routed through the GPC and thus are > > visible as wakeup sources, which is why the workaround of using an > > external SPI as wakeup trigger for the IPI works. > > We had a kernel workaround for IPIs in our internal tree for a long > time and I don't think we do anything special for PCI. Does PCI MSI > really bypass the GPC on 8mq? > > Adding Richard/Jacky, they might know about this. Currently the MSIs are terminated at the PCIe controller and routed to the CPU via a normal interrupt line that is going through the GPC, so there are no workaround required today. But then this setup severely limits the usefulness of PCI MSIs, as they incur an additional overhead of working with the DWC MSI controller and are unable to target a specific CPU, as they are all routed via a single IRQ line. > This seems like something of a corner case to me, don't many imx > boards > ship without PCI; especially for low-power scenarios? If required it > might be reasonable to add an additional workaround to disable all > cpuidle if pci msis are used. I don't know how common using PCIe with the i.MX8M is, but even the reference board ships with the WLAN connected to PCIe. I'm working with a design that has both a multi-queue and TSN capable ethernet card connected to one PCIe controller and a NVMe SSD with multiple queues connected to the second controller. Being able to terminate the MSIs at the GIC level and have proper CPU affinity makes a lot of sense in that scenario. Regards, Lucas From mboxrd@z Thu Jan 1 00:00:00 1970 From: Lucas Stach Subject: Re: [RFC 0/7] cpuidle: Add poking mechanism to support non-IPI wakeup Date: Wed, 27 Mar 2019 18:11:48 +0100 Message-ID: <1553706708.2561.42.camel@pengutronix.de> References: <1553692845-20983-1-git-send-email-abel.vesa@nxp.com> <1553701479.2561.38.camel@pengutronix.de> <564216aa-1144-71de-e887-00c58f466bf5@arm.com> <1553702767.2561.40.camel@pengutronix.de> Mime-Version: 1.0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: Sender: linux-kernel-owner@vger.kernel.org To: Leonard Crestez , "marc.zyngier@arm.com" , Richard Zhu Cc: Fabio Estevam , Cosmin Samoila , Robin Gong , Mircea Pop , Daniel Baluta , "catalin.marinas@arm.com" , Aisheng Dong , "shawnguo@kernel.org" , Robert Chiras , Anson Huang , Jun Li , Abel Vesa , "robh@kernel.org" , Zening Wang , dl-linux-imx , BOUGH CHEN , Horia Geanta , Peter Chen , Joakim Zhang , "rjw@rjwysocki.net" , Leo List-Id: linux-pm@vger.kernel.org Am Mittwoch, den 27.03.2019, 17:00 +0000 schrieb Leonard Crestez: > On Wed, 2019-03-27 at 17:06 +0100, Lucas Stach wrote: > > Am Mittwoch, den 27.03.2019, 15:57 +0000 schrieb Marc Zyngier: > > > On 27/03/2019 15:44, Lucas Stach wrote: > > > > Am Mittwoch, den 27.03.2019, 13:21 +0000 schrieb Abel Vesa: > > > > > This work is a workaround I'm looking into (more as a > > > > > background task) > > > > > in order to add support for cpuidle on i.MX8MQ based > > > > > platforms. > > > > > > > > > > The main idea here is getting around the missing GIC > > > > > wake_request signal > > > > > (due to integration design issue) by waking up a each > > > > > individual core through > > > > > some dedicated SW power-up bits inside the power controller > > > > > (GPC) right before > > > > > every IPI is requested for that each individual core. > > > > > > > > Just a general comment, without going into the details of this > > > > series: > > > > this issue is not only affecting IPIs, but also MSIs terminated > > > > at the > > > > GIC. Currently MSIs are terminated at the PCIe core, but > > > > terminating > > > > them at the GIC is clearly preferable, as this allows assigning > > > > CPU > > > > affinity to individual MSIs and lowers IRQ service overhead. > > > > > > > > I'm not sure what the consequences are for upstream Linux > > > > support yet, > > > > but we should keep in mind that having a workaround for IPIs is > > > > only > > > > solving part of the issue. > > > > > > If this erratum is affecting more than just IPIs, then indeed I > > > don't > > > see how this patch series solves anything. > > > > > > But the erratum documentation seems to imply that only SGIs are > > > affected, and goes as far as suggesting to use an external > > > interrupt > > > would solve it. How comes this is not the case? Or is it that > > > anything > > > directly routed to a redistributor is also affected? This would > > > break > > > LPIs (and thus MSIs) and PPIs (the CPU timer, among others). > > > > > > What is the *exact* status of this thing? I have the ugly feeling > > > that > > > the true workaround is just to disable cpuidle. > > > > As far as I understand the erratum, the basic issue is that the GIC > > wake_request signals are not connected to the GPC (the > > CPU/peripheral > > power sequencer). The SPIs are routed through the GPC and thus are > > visible as wakeup sources, which is why the workaround of using an > > external SPI as wakeup trigger for the IPI works. > > We had a kernel workaround for IPIs in our internal tree for a long > time and I don't think we do anything special for PCI. Does PCI MSI > really bypass the GPC on 8mq? > > Adding Richard/Jacky, they might know about this. Currently the MSIs are terminated at the PCIe controller and routed to the CPU via a normal interrupt line that is going through the GPC, so there are no workaround required today. But then this setup severely limits the usefulness of PCI MSIs, as they incur an additional overhead of working with the DWC MSI controller and are unable to target a specific CPU, as they are all routed via a single IRQ line. > This seems like something of a corner case to me, don't many imx > boards > ship without PCI; especially for low-power scenarios? If required it > might be reasonable to add an additional workaround to disable all > cpuidle if pci msis are used. I don't know how common using PCIe with the i.MX8M is, but even the reference board ships with the WLAN connected to PCIe. I'm working with a design that has both a multi-queue and TSN capable ethernet card connected to one PCIe controller and a NVMe SSD with multiple queues connected to the second controller. Being able to terminate the MSIs at the GIC level and have proper CPU affinity makes a lot of sense in that scenario. Regards, Lucas From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-1.0 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id C06D5C43381 for ; Wed, 27 Mar 2019 17:12:19 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 8F0E5206BA for ; Wed, 27 Mar 2019 17:12:19 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="PFRYtPoJ" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8F0E5206BA Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=pengutronix.de Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20170209; h=Sender: Content-Transfer-Encoding:Content-Type:Cc:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Mime-Version:References:In-Reply-To: Date:To:From:Subject:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=WixyUueUWyRba/67uh/RdrqXJQ3udaz6KaeJip2CTq0=; b=PFRYtPoJz7DuKG Z8xqp67IgrlQRSGpk/gyQsSSp4/afFwneXOBe5SfjlHRyFYe1Ut1xJNrJsXWmSwDMDOzf2lmIXsND tmLUqyrDaKn6FIGZb7YvcYeVtwM8huWAiW4Giw/n0NW2Au63S1lMrUuSlYwzjHp6UuHFjLQ6WOu8I zvJNqGk8Mn8n6fXjYUccU3XVqkiPalH2TRkoh+ApeYArKS3br9ClxG2QF5iRs9ywb+isqrF4VUGH3 gjXImHn7z8WKnIxR0xEK2qPsTRheqPbREnrvu/uh8O/B9pDvg7YqSj6XrtIi0/m1W+SR3nKHv9Bl/ GPN2KbauNOJQmj89zN8A==; Received: from localhost ([127.0.0.1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.90_1 #2 (Red Hat Linux)) id 1h9C62-000243-Hb; Wed, 27 Mar 2019 17:12:14 +0000 Received: from metis.ext.pengutronix.de ([2001:67c:670:201:290:27ff:fe1d:cc33]) by bombadil.infradead.org with esmtps (Exim 4.90_1 #2 (Red Hat Linux)) id 1h9C5y-00022v-PO for linux-arm-kernel@lists.infradead.org; Wed, 27 Mar 2019 17:12:12 +0000 Received: from kresse.hi.pengutronix.de ([2001:67c:670:100:1d::2a]) by metis.ext.pengutronix.de with esmtp (Exim 4.89) (envelope-from ) id 1h9C5d-0007FP-8C; Wed, 27 Mar 2019 18:11:49 +0100 Message-ID: <1553706708.2561.42.camel@pengutronix.de> Subject: Re: [RFC 0/7] cpuidle: Add poking mechanism to support non-IPI wakeup From: Lucas Stach To: Leonard Crestez , "marc.zyngier@arm.com" , Richard Zhu Date: Wed, 27 Mar 2019 18:11:48 +0100 In-Reply-To: References: <1553692845-20983-1-git-send-email-abel.vesa@nxp.com> <1553701479.2561.38.camel@pengutronix.de> <564216aa-1144-71de-e887-00c58f466bf5@arm.com> <1553702767.2561.40.camel@pengutronix.de> X-Mailer: Evolution 3.22.6-1+deb9u1 Mime-Version: 1.0 X-SA-Exim-Connect-IP: 2001:67c:670:100:1d::2a X-SA-Exim-Mail-From: l.stach@pengutronix.de X-SA-Exim-Scanned: No (on metis.ext.pengutronix.de); SAEximRunCond expanded to false X-PTX-Original-Recipient: linux-arm-kernel@lists.infradead.org X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20190327_101210_981987_FE3BAB9D X-CRM114-Status: GOOD ( 30.07 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "mark.rutland@arm.com" , Peter Chen , Peng Fan , Jacky Bai , "catalin.marinas@arm.com" , "will.deacon@arm.com" , Leo Zhang , "linux-kernel@vger.kernel.org" , Mircea Pop , Robin Gong , "robh@kernel.org" , "lorenzo.pieralisi@arm.com" , Abel Vesa , Anson Huang , BOUGH CHEN , Shenwei Wang , dl-linux-imx , Viorel Suman , Cosmin Samoila , "linux-pm@vger.kernel.org" , Iuliana Prodan , Zening Wang , Robert Chiras , Han Xu , Daniel Baluta , "linux-arm-kernel@lists.infradead.org" , Aisheng Dong , Horia Geanta , "rjw@rjwysocki.net" , Joakim Zhang , Ranjani Vaidyanathan , "kernel@pengutronix.de" , "sudeep.holla@arm.com" , Fabio Estevam , "shawnguo@kernel.org" , Jun Li Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+infradead-linux-arm-kernel=archiver.kernel.org@lists.infradead.org Am Mittwoch, den 27.03.2019, 17:00 +0000 schrieb Leonard Crestez: > On Wed, 2019-03-27 at 17:06 +0100, Lucas Stach wrote: > > Am Mittwoch, den 27.03.2019, 15:57 +0000 schrieb Marc Zyngier: > > > On 27/03/2019 15:44, Lucas Stach wrote: > > > > Am Mittwoch, den 27.03.2019, 13:21 +0000 schrieb Abel Vesa: > > > > > This work is a workaround I'm looking into (more as a > > > > > background task) > > > > > in order to add support for cpuidle on i.MX8MQ based > > > > > platforms. > > > > > > > > > > The main idea here is getting around the missing GIC > > > > > wake_request signal > > > > > (due to integration design issue) by waking up a each > > > > > individual core through > > > > > some dedicated SW power-up bits inside the power controller > > > > > (GPC) right before > > > > > every IPI is requested for that each individual core. > > > > > > > > Just a general comment, without going into the details of this > > > > series: > > > > this issue is not only affecting IPIs, but also MSIs terminated > > > > at the > > > > GIC. Currently MSIs are terminated at the PCIe core, but > > > > terminating > > > > them at the GIC is clearly preferable, as this allows assigning > > > > CPU > > > > affinity to individual MSIs and lowers IRQ service overhead. > > > > > > > > I'm not sure what the consequences are for upstream Linux > > > > support yet, > > > > but we should keep in mind that having a workaround for IPIs is > > > > only > > > > solving part of the issue. > > > > > > If this erratum is affecting more than just IPIs, then indeed I > > > don't > > > see how this patch series solves anything. > > > > > > But the erratum documentation seems to imply that only SGIs are > > > affected, and goes as far as suggesting to use an external > > > interrupt > > > would solve it. How comes this is not the case? Or is it that > > > anything > > > directly routed to a redistributor is also affected? This would > > > break > > > LPIs (and thus MSIs) and PPIs (the CPU timer, among others). > > > > > > What is the *exact* status of this thing? I have the ugly feeling > > > that > > > the true workaround is just to disable cpuidle. > > > > As far as I understand the erratum, the basic issue is that the GIC > > wake_request signals are not connected to the GPC (the > > CPU/peripheral > > power sequencer). The SPIs are routed through the GPC and thus are > > visible as wakeup sources, which is why the workaround of using an > > external SPI as wakeup trigger for the IPI works. > > We had a kernel workaround for IPIs in our internal tree for a long > time and I don't think we do anything special for PCI. Does PCI MSI > really bypass the GPC on 8mq? > > Adding Richard/Jacky, they might know about this. Currently the MSIs are terminated at the PCIe controller and routed to the CPU via a normal interrupt line that is going through the GPC, so there are no workaround required today. But then this setup severely limits the usefulness of PCI MSIs, as they incur an additional overhead of working with the DWC MSI controller and are unable to target a specific CPU, as they are all routed via a single IRQ line. > This seems like something of a corner case to me, don't many imx > boards > ship without PCI; especially for low-power scenarios? If required it > might be reasonable to add an additional workaround to disable all > cpuidle if pci msis are used. I don't know how common using PCIe with the i.MX8M is, but even the reference board ships with the WLAN connected to PCIe. I'm working with a design that has both a multi-queue and TSN capable ethernet card connected to one PCIe controller and a NVMe SSD with multiple queues connected to the second controller. Being able to terminate the MSIs at the GIC level and have proper CPU affinity makes a lot of sense in that scenario. Regards, Lucas _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel