From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.0 required=3.0 tests=BAYES_00,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E2C52C433DB for ; Mon, 29 Mar 2021 09:56:22 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A5B606191F for ; Mon, 29 Mar 2021 09:56:22 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S231854AbhC2Jzu (ORCPT ); Mon, 29 Mar 2021 05:55:50 -0400 Received: from mail.kernel.org ([198.145.29.99]:35316 "EHLO mail.kernel.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232087AbhC2Jz0 (ORCPT ); Mon, 29 Mar 2021 05:55:26 -0400 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 7A90C61554; Mon, 29 Mar 2021 09:55:26 +0000 (UTC) Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94) (envelope-from ) id 1lQocG-004Pp5-CY; Mon, 29 Mar 2021 10:55:24 +0100 Date: Mon, 29 Mar 2021 10:55:23 +0100 Message-ID: <87wntqqo6s.wl-maz@kernel.org> From: Marc Zyngier To: Jingyi Wang Cc: , , , , , Subject: Re: [RFC PATCH 1/3] irqchip/gic-v3: Make use of ICC_SGI1R IRM bit In-Reply-To: <20210329085210.11524-2-wangjingyi11@huawei.com> References: <20210329085210.11524-1-wangjingyi11@huawei.com> <20210329085210.11524-2-wangjingyi11@huawei.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: wangjingyi11@huawei.com, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, tglx@linutronix.de, wanghaibin.wang@huawei.com, yuzenghui@huawei.com, zhukeqian1@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Mon, 29 Mar 2021 09:52:08 +0100, Jingyi Wang wrote: > > IRM, bit[40] in ICC_SGI1R, determines how the generated SGIs > are distributed to PEs. If the bit is set, interrupts are routed > to all PEs in the system excluding "self". We use cpumask to > determine if this bit should be set and make use of that. > > This will reduce vm trap when broadcast IPIs are sent. I remember writing similar code about 4 years ago, only to realise what: - the cost of computing the resulting mask is pretty high for large machines - Linux almost never sends broadcast IPIs, so the complexity was all in vain What changed? Please provide supporting data showing how many IPIs we actually save, and for which workload. > > Signed-off-by: Jingyi Wang > --- > drivers/irqchip/irq-gic-v3.c | 12 ++++++++++++ > 1 file changed, 12 insertions(+) > > diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c > index eb0ee356a629..8ecc1b274ea8 100644 > --- a/drivers/irqchip/irq-gic-v3.c > +++ b/drivers/irqchip/irq-gic-v3.c > @@ -1127,6 +1127,7 @@ static void gic_send_sgi(u64 cluster_id, u16 tlist, unsigned int irq) > static void gic_ipi_send_mask(struct irq_data *d, const struct cpumask *mask) > { > int cpu; > + cpumask_t tmp; > > if (WARN_ON(d->hwirq >= 16)) > return; > @@ -1137,6 +1138,17 @@ static void gic_ipi_send_mask(struct irq_data *d, const struct cpumask *mask) > */ > wmb(); > > + if (!cpumask_and(&tmp, mask, cpumask_of(smp_processor_id()))) { Are you sure this does the right thing? This is checking that the current CPU is not part of the mask. But it not checking that the mask is actually "all but self". This means you are potentially sending IPIs to CPUs that are not part of the mask, making performance potentially worse. Thanks, M. > + /* Set Interrupt Routing Mode bit */ > + u64 val; > + val = (d->hwirq) << ICC_SGI1R_SGI_ID_SHIFT; > + val |= BIT_ULL(ICC_SGI1R_IRQ_ROUTING_MODE_BIT); > + gic_write_sgi1r(val); > + > + isb(); > + return; > + } > + > for_each_cpu(cpu, mask) { > u64 cluster_id = MPIDR_TO_SGI_CLUSTER_ID(cpu_logical_map(cpu)); > u16 tlist; > -- > 2.19.1 > > -- Without deviation from the norm, progress is not possible. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D8E0EC433C1 for ; Mon, 29 Mar 2021 16:43:14 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 76AE76191F for ; Mon, 29 Mar 2021 16:43:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 76AE76191F Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Subject:Cc:To: From:Message-ID:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=x3tZVOYAOqrEFpWJ/6SOKR4KUTYYfMWCyRGtscYXqVw=; b=bvjZV6UFkm+QiRQTgSrun2siG FmmqtoY/lR3POe7lNpmrJ5Phu5ZiTkU7AHCwmuxnBE2eKf44cT8P3DDyw7ez6aoJyPNN5WD5lq7wE x9eHMtzhli3utYfVOo2W5qdoulNTcQLi7Qz+FSBJz71rmA3VjwE6ZFnkSeGRISBdUXSyb08Q23jQq vlaj0pKAuDNG2bmpsK2jTBai2HXJdtJZTKPd/Lag2DhCjtvNF3QriNCKbHUrw2adC02u0/HW6pjO9 ePO1AVFqrGiuRqZ2IASKeafoLf/BBsA4CK5viA7myscKa26Njj6fp5T43u9mWo6dHIHVZNssFN7xp hEiRyCBqA==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1lQuvi-000lko-Mn; Mon, 29 Mar 2021 16:39:54 +0000 Received: from mail.kernel.org ([198.145.29.99]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1lQocS-000Yek-Rb for linux-arm-kernel@lists.infradead.org; Mon, 29 Mar 2021 09:55:41 +0000 Received: from disco-boy.misterjones.org (disco-boy.misterjones.org [51.254.78.96]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 7A90C61554; Mon, 29 Mar 2021 09:55:26 +0000 (UTC) Received: from 78.163-31-62.static.virginmediabusiness.co.uk ([62.31.163.78] helo=why.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94) (envelope-from ) id 1lQocG-004Pp5-CY; Mon, 29 Mar 2021 10:55:24 +0100 Date: Mon, 29 Mar 2021 10:55:23 +0100 Message-ID: <87wntqqo6s.wl-maz@kernel.org> From: Marc Zyngier To: Jingyi Wang Cc: , , , , , Subject: Re: [RFC PATCH 1/3] irqchip/gic-v3: Make use of ICC_SGI1R IRM bit In-Reply-To: <20210329085210.11524-2-wangjingyi11@huawei.com> References: <20210329085210.11524-1-wangjingyi11@huawei.com> <20210329085210.11524-2-wangjingyi11@huawei.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 62.31.163.78 X-SA-Exim-Rcpt-To: wangjingyi11@huawei.com, linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org, tglx@linutronix.de, wanghaibin.wang@huawei.com, yuzenghui@huawei.com, zhukeqian1@huawei.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210329_105538_591142_9DD94AFD X-CRM114-Status: GOOD ( 25.44 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, 29 Mar 2021 09:52:08 +0100, Jingyi Wang wrote: > > IRM, bit[40] in ICC_SGI1R, determines how the generated SGIs > are distributed to PEs. If the bit is set, interrupts are routed > to all PEs in the system excluding "self". We use cpumask to > determine if this bit should be set and make use of that. > > This will reduce vm trap when broadcast IPIs are sent. I remember writing similar code about 4 years ago, only to realise what: - the cost of computing the resulting mask is pretty high for large machines - Linux almost never sends broadcast IPIs, so the complexity was all in vain What changed? Please provide supporting data showing how many IPIs we actually save, and for which workload. > > Signed-off-by: Jingyi Wang > --- > drivers/irqchip/irq-gic-v3.c | 12 ++++++++++++ > 1 file changed, 12 insertions(+) > > diff --git a/drivers/irqchip/irq-gic-v3.c b/drivers/irqchip/irq-gic-v3.c > index eb0ee356a629..8ecc1b274ea8 100644 > --- a/drivers/irqchip/irq-gic-v3.c > +++ b/drivers/irqchip/irq-gic-v3.c > @@ -1127,6 +1127,7 @@ static void gic_send_sgi(u64 cluster_id, u16 tlist, unsigned int irq) > static void gic_ipi_send_mask(struct irq_data *d, const struct cpumask *mask) > { > int cpu; > + cpumask_t tmp; > > if (WARN_ON(d->hwirq >= 16)) > return; > @@ -1137,6 +1138,17 @@ static void gic_ipi_send_mask(struct irq_data *d, const struct cpumask *mask) > */ > wmb(); > > + if (!cpumask_and(&tmp, mask, cpumask_of(smp_processor_id()))) { Are you sure this does the right thing? This is checking that the current CPU is not part of the mask. But it not checking that the mask is actually "all but self". This means you are potentially sending IPIs to CPUs that are not part of the mask, making performance potentially worse. Thanks, M. > + /* Set Interrupt Routing Mode bit */ > + u64 val; > + val = (d->hwirq) << ICC_SGI1R_SGI_ID_SHIFT; > + val |= BIT_ULL(ICC_SGI1R_IRQ_ROUTING_MODE_BIT); > + gic_write_sgi1r(val); > + > + isb(); > + return; > + } > + > for_each_cpu(cpu, mask) { > u64 cluster_id = MPIDR_TO_SGI_CLUSTER_ID(cpu_logical_map(cpu)); > u16 tlist; > -- > 2.19.1 > > -- Without deviation from the norm, progress is not possible. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel