From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 8471DC54EE9 for ; Tue, 20 Sep 2022 10:11:03 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=mQA7P4zCxaXNI8RLQ0iAGJn84m2uvwsHfdB54EHeywY=; b=VweBecsCWHeAkn zANGLQYEX++WkzoLDtil2M69wRrjAu9y4KgIweWNO8SsWkq/xCiZPkK1wfm+xCA9sOUSpgH/qLpl5 NHH0Y3X3iHetMp9/tJkMcaSKPaT9ENHdZFCKPOQINWV6xo3LQ9vfn33++yVltxebydm7zhiEcec54 oI2SVGWPZ12gqAUhWQAf+kqRwZPLjNWTozrVJENisvcOjKjeasASE4rDcIZW//9NRCihzwQOKzo24 Xuo/mRRmug90msfEYTfUZXJ2OhcYfjq3P++cuQs0MWb7oKC09A5FAnOwRKRgy0URx7+0Cuug0tpAM Qofvt0C8bnpY7O5VA8wg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1oaaCL-002nUp-TR; Tue, 20 Sep 2022 10:09:50 +0000 Received: from pandora.armlinux.org.uk ([2001:4d48:ad52:32c8:5054:ff:fe00:142]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1oaaCH-002nQ3-CA for linux-arm-kernel@lists.infradead.org; Tue, 20 Sep 2022 10:09:47 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=armlinux.org.uk; s=pandora-2019; h=Sender:In-Reply-To:Content-Type: MIME-Version:References:Message-ID:Subject:Cc:To:From:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Id: List-Help:List-Unsubscribe:List-Subscribe:List-Post:List-Owner:List-Archive; bh=lJCfj6/8lxb6ydDoOUAnnyZWAit7dTBRoj8QS3FHggE=; b=OKLAN5b3Ub3M2uUVAnCNufR1zN 9S9Ti834XJe57RozQl3DDg7cYKxBktevS/OvxXoe2G9GgpeYc8SeOOuW7HavFClDYx7WUVd4CjQqm k1BEIiTwqDerlpmZ/EYj7xg33MPtdTOkEGXWhJhNifybzcCiRH9ZOWlEc115JB4OUfo9QBlTLW/4l 0UiycZuG59DcFrcxRH5GJ5Jd8Axeo8jkpGGPqMXuBfkXxLBj0if/P5xz71wy+smYBI3OwygfboUSn lFLeJU9szjRQOcDs16yDVbsAay2QSXVz7E6Y5Qqqzli0AddMbhQaHNrPrhqhmvL/CkKACFjIW1Dm5 fYrfzh+Q==; Received: from shell.armlinux.org.uk ([fd8f:7570:feb6:1:5054:ff:fe00:4ec]:34412) by pandora.armlinux.org.uk with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1oaaCC-0001rk-32; Tue, 20 Sep 2022 11:09:40 +0100 Received: from linux by shell.armlinux.org.uk with local (Exim 4.94.2) (envelope-from ) id 1oaaCA-0007Fq-Iw; Tue, 20 Sep 2022 11:09:38 +0100 Date: Tue, 20 Sep 2022 11:09:38 +0100 From: "Russell King (Oracle)" To: Li Chen Cc: Arnd Bergmann , linux-arm-kernel Subject: Re: Why GICD_ITARGETSR is not used by Linux Message-ID: References: <183588fd5a7.1074ac6241209386.2442883306267550831@linux.beauty> <45d1a6b4-eb28-4392-84a3-d30a2f8e150a@www.fastmail.com> <1835a49844d.111e968dc1302721.8856919484302602882@linux.beauty> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <1835a49844d.111e968dc1302721.8856919484302602882@linux.beauty> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220920_030945_454831_A1962F1C X-CRM114-Status: GOOD ( 39.69 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Sep 20, 2022 at 11:45:10AM +0200, Li Chen wrote: > Hi Arnd, > > ---- On Tue, 20 Sep 2022 09:04:16 +0200 Arnd Bergmann wrote --- > > On Tue, Sep 20, 2022, at 3:42 AM, Li Chen wrote: > > > Hi Arnd, > > > > > > I noticed GIC has GICD_ITARGETSR to distribute IRQ to different CPUs, > > > but currently, it is not used by Linux. > > > > > > There was a patchset from MTK people: > > > http://archive.lwn.net:8080/linux-kernel/1606486531-25719-1-git-send-email-hanks.chen@mediatek.com/T/#t > > > which implements GIC-level IRQ distributor using GICD_ITARGETSR, but it > > > is > > > not accepted because the maintainer thinks it will break existing codes > > > and not provide benefits compared with the existing affinity mechanism. > > > > > > IIUC, Linux only relies on affinity/irqbalance to distribute IRQ > > > instead of architecture-specific solutions like GIC's distributor. > > > > > > Maybe latency can somewhat get improved, but there is no benchmark yet. > > > > > > I have two questions here: > > > 1. Now that Linux doesn't use GICD_ITARGETSR, where does it set CPU 0 > > > to be the only IRQ distributor core? > > > 2. Do you know any other reasons that GICD_ITARGETSR is not used by > > > Linux? > > > > Hi Li, > > > > It looks like the original submitter never followed up > > with a new version of the patch that addresses the > > issues found in review. I would assume they gave up either > > because it did not show any real-world advantage, or they > > could not address all of the concerns. > > Thanks for your reply. > > FYI, here is another thread about this topic: https://lore.kernel.org/linux-arm-kernel/20191120105017.GN25745@shell.armlinux.org.uk/ Oh god, not this again. The behaviour of the GIC is as follows. If you set two CPUs in GICD_ITARGETSRn, then the interrupt will be delivered to _both_ of those CPUs. Not just one selected at random or determined by some algorithm, but both CPUs. Both CPUs get woken up if they're in sleep, and both CPUs attempt to process the interrupt. One CPU will win the lock, while the other CPU spins waiting for the lock to process the interrupt. The winning CPU will process the interrupt, clear it on the device, release the lock and acknowledge it at the GIC CPU interface. The CPU that lost the previous race can now proceed to process the very same interrupt, discovers that it's no longer pending on the device, and signals IRQ_NONE as it appears to be a spurious interrupt. The result is that the losing CPU ends up wasting CPU cycles, and if the losing CPU was in a low power idle state, needlessly wakes up to process this interrupt. If you have more CPUs involved, you have more CPUs wasting CPU cycles, being woken up wasting power - not just occasionally, but almost every single interrupt that is raised from a device in the system. On architectures such as x86, the PICs distribute the interrupts in hardware amongst the CPUs. So if a single interrupt is set to be sent to multiple CPUs, only _one_ of the CPUs is actually interrupted. Hence, x86 can have multiple CPUs selected as a destination, and the hardware delivers the interrupt across all CPUs. On ARM, we don't have that. We have a thundering herd of CPUs if we set more than one CPU to process the interrupt, which is grossly inefficient. As I said in the reply you linked to above, I did attempt to implement several ideas in software, where the kernel would attempt to distribute dynamically the interrupt amongst the CPUs in the affinity mask, but I could never get what appeared to be a good behaviour on the platforms I was trying and performance wasn't as good. So I abandoned it. This doesn't preclude someone else having a go at solving that problem, but the problem is not solved by setting multiple CPU bits in the GICD_ITARGETSRn registers. As I said above, that just gets you a thundering herd problem, less performance, and worse power consumption. -- RMK's Patch system: https://www.armlinux.org.uk/developer/patches/ FTTP is here! 40Mbps down 10Mbps up. Decent connectivity at last! _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel