From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69F09C77B73 for ; Wed, 31 May 2023 08:35:18 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S234690AbjEaIfQ (ORCPT ); Wed, 31 May 2023 04:35:16 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53944 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S235112AbjEaIfL (ORCPT ); Wed, 31 May 2023 04:35:11 -0400 Received: from dfw.source.kernel.org (dfw.source.kernel.org [IPv6:2604:1380:4641:c500::1]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id A8E61122; Wed, 31 May 2023 01:35:07 -0700 (PDT) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 3556862871; Wed, 31 May 2023 08:35:07 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7E457C433EF; Wed, 31 May 2023 08:35:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685522106; bh=NvnORJxFYk7Gbi9d+lRiKRLianbRU171EPYVxebojBQ=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=N5TFcW7QfAm4i7Jr8Eu5N6ipqg9wVM7yUZYxuGQqkY2ughfXFASU8XWJ607C2vmtK DKZR3sj3VLJAUrwCizJPjNLU9rNDtWbhDE2ZT2cFKGu8yd+80HEgusu3FPFpJjMpqG QHcWMBFhM5KEKbtiVkZYchimZvjfNNIS66Y/BNe2U1yY/XzzY8uHNMLyG8+Kwf6pUR 7ADPZGhTGh953nl+J0EzPGjqQW6fg6fN5QwR4EePiqtr3wPMz9easemvNKYNi5g6NR v4yfjo6KdAcxbsRl/B+MbsxtACM7m4HtB/S07V4JNhJN9RXTL5N8tG0XrTpqHnZsbM kS4PDHWOxrkYQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1q4HIO-001YmU-AJ; Wed, 31 May 2023 09:35:04 +0100 Date: Wed, 31 May 2023 09:35:03 +0100 Message-ID: <861qiwdhl4.wl-maz@kernel.org> From: Marc Zyngier To: Thomas Gleixner Cc: LKML , Will Deacon , linux-pci@vger.kernel.org, Bjorn Helgaas , Lorenzo Pieralisi , Greg Kroah-Hartman , Jason Gunthorpe , Andrew Lunn , Gregory Clement , Sebastian Hesselbarth , Ammar Faizi , Robin Murphy , Lorenzo Pieralisi , Nishanth Menon , Tero Kristo , Santosh Shilimkar , linux-arm-kernel@lists.infradead.org, Vinod Koul , Sinan Kaya , Andy Gross , Bjorn Andersson , Mark Rutland , Shameerali Kolothum Thodi , Zenghui Yu , Shawn Guo , Sascha Hauer , Fabio Estevam , Anna-Maria Behnsen Subject: Re: [patch V2 06/40] PCI/MSI: Provide static key for parent mask/unmask In-Reply-To: <87lehfurij.ffs@tglx> References: <20221121135653.208611233@linutronix.de> <20221121140048.659849460@linutronix.de> <8635a8o65q.wl-maz@kernel.org> <87bkowcx0z.ffs@tglx> <86zgcgmpzl.wl-maz@kernel.org> <87v8n3c2qy.ffs@tglx> <87ttw4wiro.ffs@tglx> <86r0r7cpks.wl-maz@kernel.org> <87lehfurij.ffs@tglx> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/28.2 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: tglx@linutronix.de, linux-kernel@vger.kernel.org, will@kernel.org, linux-pci@vger.kernel.org, bhelgaas@google.com, lorenzo.pieralisi@arm.com, gregkh@linuxfoundation.org, jgg@mellanox.com, andrew@lunn.ch, gregory.clement@bootlin.com, sebastian.hesselbarth@gmail.com, ammarfaizi2@gnuweeb.org, robin.murphy@arm.com, lpieralisi@kernel.org, nm@ti.com, kristo@kernel.org, ssantosh@kernel.org, linux-arm-kernel@lists.infradead.org, vkoul@kernel.org, okaya@kernel.org, agross@kernel.org, andersson@kernel.org, mark.rutland@arm.com, shameerali.kolothum.thodi@huawei.com, yuzenghui@huawei.com, shawnguo@kernel.org, s.hauer@pengutronix.de, festevam@gmail.com, anna-maria.behnsen@linutronix.de X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, 23 May 2023 14:05:56 +0100, Thomas Gleixner wrote: > > On Tue, May 23 2023 at 11:25, Marc Zyngier wrote: > > On Mon, 22 May 2023 15:19:39 +0100, > > Thomas Gleixner wrote: > >> On the other hand for PCI/MSI[x] the mask/unmask operations are not in > >> the hot path as PCI/MSI[x] are strictly edge. Mask/unmask is only > >> happening on startup, shutdown and when an interrupt arrives after > >> disable_irq() incremented the lazy disable counter. > >> > >> For regular interrupt handling mask/unmask is not involved. > >> > >> So to avoid that global key we can let the parent domain set a new flag, > >> e.g. MSI_FLAG_PCI_MSI_MASK_PARENT, in msi_parent_ops::supported_flags > >> and let the PCI/MSI core code query that information when the per device > >> domain is created and select the appropriate template or fixup the > >> callbacks after the domain is created. > >> > >> Does that address your concerns? > > > > It does to a certain extent. > > > > But what I'd really like is that in the most common case where the > > interrupt controller is capable of masking MSIs, the PCI/MSI > > *enabling* becomes the responsibility of the PCI core code and not the > > IRQ code. > > > > The IRQ code should ideally only be concerned with the masking of the > > interrupt at the irqchip level, and not beyond that. And that'd solve > > the Xen problem by merely ignoring it. > > > > If we have HW out there that cannot mask MSIs at the interrupt > > controller level, then we'd have to fallback to device-side masking, > > which doesn't really work in general (MultiMSI being my favourite > > example). My gut feeling is that this is rare, but I'm pretty sure it > > exists. > > Sure. There are 3 parts involved: > > [Device]--->[PCI/MSI]---->[GIC] > irqchip irqchip > > Controlling the interrupt machinery in the device happens at the device > driver level and is conceptually independent of the interrupt > manangement code. The device driver has no access to the PCI/MSI irqchip > and all it can do is to enable/disable the source of the interrupt in > the device. > > For the interrupt management code the job is to ensure that an interrupt > can be prevented from disrupting the OS operation independent of the > device driver correctness. > > As a matter of fact we know that PCI/MSI masking ranges from not > possible over flaky to properly working. So we can't reliably prevent > that a rougue device spams the PCIe bus with messages. > > Which means that we should utilize the fact that the next interrupt chip > in the hierarchy can mask reliably. I wish I could disable individual > vectors at the local APIC level on x86... > > Now the question is whether we want to make this conditional depending > on what the PCI/MSI[X] hardware advertises or just keep it simple and do > it unconditionally. I think this should be unconditional if the root irqchip (the GIC in this instance) is capable of it. So a suggestion where the root irqchip exposes its masking capability, which upon detection by the upper layer (whateverbusyouwant/MSI) makes it stop playing with its own device-level mask has my full support (and now breathe normally). Thanks, M. -- Without deviation from the norm, progress is not possible. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5AD46C7EE2F for ; Wed, 31 May 2023 08:35:37 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Subject:Cc:To:From:Message-ID:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=Hf3EMjmVOsuaUn+nb9i1Y7DL9i8otZZA/yrPIlnaff4=; b=Vu3BqAOQiLwh74 mn/B6/DHB5CpT0NwLt8R4dgWMQ47QPc5xkk3zuZTV4Syg36MTeTQH22kaqz7PeVb22C2fCUOATDIS EgPDyXjWE5epGSLzPuaX4TdwBUfb/qoFPGbJ+eYy/NK/S4u005vYj40IpDp3karEBfD+2e+sfl1FS gmfHhiyedOeN9LfUEU7SgdoDKdxuA4pWgKAjIU3MuViyBUKbQNZ/GJ7mDy8vKNtgIgt6vFX/GG/Om Z1NKsbXAD3rffZxfUf0yNXAShAHYuIVPwQPtd6G5AWI1nzJXOIJZc8+e3FItuIUP8+YWf+Rq1gpIX F3kg8yhftl3bQQNH856g==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q4HIX-00GdNp-0D; Wed, 31 May 2023 08:35:13 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q4HIT-00GdNB-27 for linux-arm-kernel@lists.infradead.org; Wed, 31 May 2023 08:35:11 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 2B8BC619CC; Wed, 31 May 2023 08:35:07 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 7E457C433EF; Wed, 31 May 2023 08:35:06 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1685522106; bh=NvnORJxFYk7Gbi9d+lRiKRLianbRU171EPYVxebojBQ=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=N5TFcW7QfAm4i7Jr8Eu5N6ipqg9wVM7yUZYxuGQqkY2ughfXFASU8XWJ607C2vmtK DKZR3sj3VLJAUrwCizJPjNLU9rNDtWbhDE2ZT2cFKGu8yd+80HEgusu3FPFpJjMpqG QHcWMBFhM5KEKbtiVkZYchimZvjfNNIS66Y/BNe2U1yY/XzzY8uHNMLyG8+Kwf6pUR 7ADPZGhTGh953nl+J0EzPGjqQW6fg6fN5QwR4EePiqtr3wPMz9easemvNKYNi5g6NR v4yfjo6KdAcxbsRl/B+MbsxtACM7m4HtB/S07V4JNhJN9RXTL5N8tG0XrTpqHnZsbM kS4PDHWOxrkYQ== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1q4HIO-001YmU-AJ; Wed, 31 May 2023 09:35:04 +0100 Date: Wed, 31 May 2023 09:35:03 +0100 Message-ID: <861qiwdhl4.wl-maz@kernel.org> From: Marc Zyngier To: Thomas Gleixner Cc: LKML , Will Deacon , linux-pci@vger.kernel.org, Bjorn Helgaas , Lorenzo Pieralisi , Greg Kroah-Hartman , Jason Gunthorpe , Andrew Lunn , Gregory Clement , Sebastian Hesselbarth , Ammar Faizi , Robin Murphy , Lorenzo Pieralisi , Nishanth Menon , Tero Kristo , Santosh Shilimkar , linux-arm-kernel@lists.infradead.org, Vinod Koul , Sinan Kaya , Andy Gross , Bjorn Andersson , Mark Rutland , Shameerali Kolothum Thodi , Zenghui Yu , Shawn Guo , Sascha Hauer , Fabio Estevam , Anna-Maria Behnsen Subject: Re: [patch V2 06/40] PCI/MSI: Provide static key for parent mask/unmask In-Reply-To: <87lehfurij.ffs@tglx> References: <20221121135653.208611233@linutronix.de> <20221121140048.659849460@linutronix.de> <8635a8o65q.wl-maz@kernel.org> <87bkowcx0z.ffs@tglx> <86zgcgmpzl.wl-maz@kernel.org> <87v8n3c2qy.ffs@tglx> <87ttw4wiro.ffs@tglx> <86r0r7cpks.wl-maz@kernel.org> <87lehfurij.ffs@tglx> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/28.2 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: tglx@linutronix.de, linux-kernel@vger.kernel.org, will@kernel.org, linux-pci@vger.kernel.org, bhelgaas@google.com, lorenzo.pieralisi@arm.com, gregkh@linuxfoundation.org, jgg@mellanox.com, andrew@lunn.ch, gregory.clement@bootlin.com, sebastian.hesselbarth@gmail.com, ammarfaizi2@gnuweeb.org, robin.murphy@arm.com, lpieralisi@kernel.org, nm@ti.com, kristo@kernel.org, ssantosh@kernel.org, linux-arm-kernel@lists.infradead.org, vkoul@kernel.org, okaya@kernel.org, agross@kernel.org, andersson@kernel.org, mark.rutland@arm.com, shameerali.kolothum.thodi@huawei.com, yuzenghui@huawei.com, shawnguo@kernel.org, s.hauer@pengutronix.de, festevam@gmail.com, anna-maria.behnsen@linutronix.de X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230531_013509_801881_E4A6B101 X-CRM114-Status: GOOD ( 42.64 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, 23 May 2023 14:05:56 +0100, Thomas Gleixner wrote: > > On Tue, May 23 2023 at 11:25, Marc Zyngier wrote: > > On Mon, 22 May 2023 15:19:39 +0100, > > Thomas Gleixner wrote: > >> On the other hand for PCI/MSI[x] the mask/unmask operations are not in > >> the hot path as PCI/MSI[x] are strictly edge. Mask/unmask is only > >> happening on startup, shutdown and when an interrupt arrives after > >> disable_irq() incremented the lazy disable counter. > >> > >> For regular interrupt handling mask/unmask is not involved. > >> > >> So to avoid that global key we can let the parent domain set a new flag, > >> e.g. MSI_FLAG_PCI_MSI_MASK_PARENT, in msi_parent_ops::supported_flags > >> and let the PCI/MSI core code query that information when the per device > >> domain is created and select the appropriate template or fixup the > >> callbacks after the domain is created. > >> > >> Does that address your concerns? > > > > It does to a certain extent. > > > > But what I'd really like is that in the most common case where the > > interrupt controller is capable of masking MSIs, the PCI/MSI > > *enabling* becomes the responsibility of the PCI core code and not the > > IRQ code. > > > > The IRQ code should ideally only be concerned with the masking of the > > interrupt at the irqchip level, and not beyond that. And that'd solve > > the Xen problem by merely ignoring it. > > > > If we have HW out there that cannot mask MSIs at the interrupt > > controller level, then we'd have to fallback to device-side masking, > > which doesn't really work in general (MultiMSI being my favourite > > example). My gut feeling is that this is rare, but I'm pretty sure it > > exists. > > Sure. There are 3 parts involved: > > [Device]--->[PCI/MSI]---->[GIC] > irqchip irqchip > > Controlling the interrupt machinery in the device happens at the device > driver level and is conceptually independent of the interrupt > manangement code. The device driver has no access to the PCI/MSI irqchip > and all it can do is to enable/disable the source of the interrupt in > the device. > > For the interrupt management code the job is to ensure that an interrupt > can be prevented from disrupting the OS operation independent of the > device driver correctness. > > As a matter of fact we know that PCI/MSI masking ranges from not > possible over flaky to properly working. So we can't reliably prevent > that a rougue device spams the PCIe bus with messages. > > Which means that we should utilize the fact that the next interrupt chip > in the hierarchy can mask reliably. I wish I could disable individual > vectors at the local APIC level on x86... > > Now the question is whether we want to make this conditional depending > on what the PCI/MSI[X] hardware advertises or just keep it simple and do > it unconditionally. I think this should be unconditional if the root irqchip (the GIC in this instance) is capable of it. So a suggestion where the root irqchip exposes its masking capability, which upon detection by the upper layer (whateverbusyouwant/MSI) makes it stop playing with its own device-level mask has my full support (and now breathe normally). Thanks, M. -- Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel