From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-11.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE, SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id DB77DC2B9F8 for ; Tue, 25 May 2021 17:33:24 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id B96DF61400 for ; Tue, 25 May 2021 17:33:24 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233784AbhEYRew (ORCPT ); Tue, 25 May 2021 13:34:52 -0400 Received: from foss.arm.com ([217.140.110.172]:33014 "EHLO foss.arm.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231266AbhEYReq (ORCPT ); Tue, 25 May 2021 13:34:46 -0400 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 921CB168F; Tue, 25 May 2021 10:33:15 -0700 (PDT) Received: from e113632-lin.cambridge.arm.com (e113632-lin.cambridge.arm.com [10.1.194.46]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 7CCBF3F73B; Tue, 25 May 2021 10:33:14 -0700 (PDT) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Marc Zyngier , Thomas Gleixner , Lorenzo Pieralisi , Vincenzo Frascino Subject: [RFC PATCH v2 00/10] irqchip/irq-gic: Optimize masking by leveraging EOImode=1 Date: Tue, 25 May 2021 18:32:45 +0100 Message-Id: <20210525173255.620606-1-valentin.schneider@arm.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi folks! This is the spiritual successor to [1], which was over 6 years ago (!). Revisions ========= RFCv1 -> RFCv2 ++++++++++++++ o Rebased against latest tip/irq/core o Applied cleanups suggested by Thomas o Collected some performance results Background ========== GIC mechanics +++++++++++++ There are three IRQ operations: o Acknowledge. This gives us the IRQ number that interrupted us, and also - raises the running priority of the CPU interface to that of the IRQ - sets the active bit of the IRQ o Priority Drop. This "clears" the running priority. o Deactivate. This clears the active bit of the IRQ. o The CPU interface has a running priority value. No interrupt of lower or equal priority will be signaled to the CPU attached to that interface. On Linux, we only have two priority values: pNMIs at highest priority, and everything else at the other priority. o Most GIC interrupts have an "active" bit. This bit is set on Acknowledge and cleared on Deactivate. A given interrupt cannot be re-signaled to a CPU if it has its active bit set (i.e. if it "fires" again while it's being handled). EOImode fun +++++++++++ In EOImode=0, Priority Drop and Deactivate are undissociable. The (simplified) interrupt handling flow is as follows: <~IRQ> Acknowledge Priority Drop + Deactivate With EOImode=1, we can invoke each operation individually. This gives us: <~IRQ> Acknowledge Priority Drop <*other* interrupts can be signaled from here, once interrupts are re-enabled> Deactivate <*this* interrupt can be signaled again> What this means is that with EOImode=1, any interrupt is kept "masked" by its active bit between Priority Drop and Deactivate. Threaded IRQs and ONESHOT ========================= ONESHOT threaded IRQs must remain masked between the main handler and the threaded handler. Right now we do this using the conventional irq_mask() operations, which looks like this: Acknowledge Priority Drop irq_mask() Deactivate irq_unmask() However, masking for the GICs means poking the distributor, and there's no sysreg for that - it's an MMIO access. We've seen above that our IRQ handling can give us masking "for free", and this is what this patch set is all about. It turns the above handling into: Acknowledge Priority Drop Deactivate No irq_mask() => fewer MMIO accesses => happier users (or so I've been told). This is especially relevant to PREEMPT_RT which forces threaded IRQs. Functional testing ================== GICv2 +++++ I've tested this on my Juno with forced irqthreads. This makes the pl011 IRQ into a threaded ONESHOT IRQ, so I spammed my keyboard into the console and verified via ftrace that there were no irq_mask() / irq_unmask() involved. GICv3 +++++ I've tested this on my Ampere eMAG, which uncovered "fun" interactions with the MSI domains. Did the same trick as the Juno with the pl011. pNMIs cause said eMAG to freeze, but that's true even without my patches. I did try them out under QEMU+KVM and that looked fine, although that means I only got to test EOImode=0. I'll try to dig into this when I get some more cycles. Performance impact ================== Benchmark +++++++++ Finding a benchmark that leverages a force-threaded IRQ has proved to be somewhat of a pain, so I crafted my own. It's a bit daft, but so are most benchmarks (though this one might win a prize). Long story short, I'm picking an unused IRQ and have it be force-threaded. The benchmark then is: loop: irq_set_irqchip_state(irq, IRQCHIP_STATE_PENDING, true); wait_for_completion(&done); complete(&done); A more complete picture would be: raise IRQ wait run flow handler wake IRQ thread finish handling wake bench thread Letting this run for a fixed amount of time lets me measure an entire IRQ handling cycle, which is what I'm after since there's one less mask() in the flow handler and one less unmask() in the threaded handler. You'll note there's some potential "noise" in there due to scheduling both the benchmark thread and the IRQ thread. However, the IRQ thread is pinned to the IRQ's affinity, and I also pinned the benchmark thread in my tests, which should keep this noise to a minimum. Results +++++++ On a Juno r0, 20 iterations of 5 seconds of that benchmark yields (measuring irqs/sec): | mean | median | 90th percentile | 99th percentile | |------+--------+-----------------+-----------------| | +11% | +11% | +12% | +14% | On an Ampere eMAG, 20 iterations of 5 seconds of that benchmark yields (measuring irqs/sec): | mean | median | 90th percentile | 99th percentile | |------+--------+-----------------+-----------------| | +20% | +20% | +20% | +20% | This is still quite "artificial", but it reassures me in that skipping those (un)mask operations can yield some measurable improvement. Valentin Schneider (10): genirq: Add chip flag to denote automatic IRQ (un)masking genirq: Define irq_ack() and irq_eoi() helpers genirq: Employ ack_irq() and eoi_irq() where relevant genirq: Add handle_strict_flow_irq() flow handler genirq: Let purely flow-masked ONESHOT irqs through unmask_threaded_irq() genirq: Don't mask IRQ within flow handler if IRQ is flow-masked genirq, irq-gic-v3: Make NMI flow handlers use ->irq_ack() if available irqchip/gic-v3-its: Use irq_chip_ack_parent() irqchip/gic: Convert to handle_strict_flow_irq() irqchip/gic-v3: Convert to handle_strict_flow_irq() drivers/irqchip/irq-gic-v3-its-pci-msi.c | 1 + drivers/irqchip/irq-gic-v3-its.c | 1 + drivers/irqchip/irq-gic-v3.c | 27 +++-- drivers/irqchip/irq-gic.c | 14 ++- include/linux/irq.h | 15 ++- kernel/irq/chip.c | 122 ++++++++++++++++++++--- kernel/irq/debugfs.c | 2 + kernel/irq/internals.h | 7 ++ kernel/irq/manage.c | 2 +- 9 files changed, 159 insertions(+), 32 deletions(-) -- 2.25.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.1 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7FCC8C2B9F8 for ; Tue, 25 May 2021 17:35:47 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 4B45B61157 for ; Tue, 25 May 2021 17:35:47 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 4B45B61157 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=arm.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-Id:Date:Subject:Cc :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=MC1uCrvwg/Z2YqheaEbilePlvwTIzyT7k5GLbOIE4Jk=; b=EhtJ5UzF2xECOW C3FkVlg6JU78ug0gRtX9znmSaOY6RfDCw2s5DSuPM8GLlC4xN/1YQyVPYHzU1FEqz+8+uqEL//gKR WTBSq0xNLsIJciPecbwuNUwr1ma6x9pQrTMzivg0hezxn9zsB3JNCwzuWhVT8eCDxBu0ziGS2w/ft Odb4Iko+QV073OVCEaNqyXmf2DVQcpwDwVV833hL+A8xcGY1zTtM9TdT3Xoyvp/sDyj3wgQn++iwT EiSNG++MFXFZphywfW15YwOukBLErOWh7El+u9LaUgGOkmnkytZcPxBjxalYoqFfOPxzrv4iLzmrL ZYlKC+RK86hdgCEj0qnA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1llaw6-006uqW-3Y; Tue, 25 May 2021 17:33:46 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1llave-006udT-GU for linux-arm-kernel@lists.infradead.org; Tue, 25 May 2021 17:33:20 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 921CB168F; Tue, 25 May 2021 10:33:15 -0700 (PDT) Received: from e113632-lin.cambridge.arm.com (e113632-lin.cambridge.arm.com [10.1.194.46]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPA id 7CCBF3F73B; Tue, 25 May 2021 10:33:14 -0700 (PDT) From: Valentin Schneider To: linux-kernel@vger.kernel.org, linux-arm-kernel@lists.infradead.org Cc: Marc Zyngier , Thomas Gleixner , Lorenzo Pieralisi , Vincenzo Frascino Subject: [RFC PATCH v2 00/10] irqchip/irq-gic: Optimize masking by leveraging EOImode=1 Date: Tue, 25 May 2021 18:32:45 +0100 Message-Id: <20210525173255.620606-1-valentin.schneider@arm.com> X-Mailer: git-send-email 2.25.1 MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210525_103318_722500_97A1E455 X-CRM114-Status: GOOD ( 17.78 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi folks! This is the spiritual successor to [1], which was over 6 years ago (!). Revisions ========= RFCv1 -> RFCv2 ++++++++++++++ o Rebased against latest tip/irq/core o Applied cleanups suggested by Thomas o Collected some performance results Background ========== GIC mechanics +++++++++++++ There are three IRQ operations: o Acknowledge. This gives us the IRQ number that interrupted us, and also - raises the running priority of the CPU interface to that of the IRQ - sets the active bit of the IRQ o Priority Drop. This "clears" the running priority. o Deactivate. This clears the active bit of the IRQ. o The CPU interface has a running priority value. No interrupt of lower or equal priority will be signaled to the CPU attached to that interface. On Linux, we only have two priority values: pNMIs at highest priority, and everything else at the other priority. o Most GIC interrupts have an "active" bit. This bit is set on Acknowledge and cleared on Deactivate. A given interrupt cannot be re-signaled to a CPU if it has its active bit set (i.e. if it "fires" again while it's being handled). EOImode fun +++++++++++ In EOImode=0, Priority Drop and Deactivate are undissociable. The (simplified) interrupt handling flow is as follows: <~IRQ> Acknowledge Priority Drop + Deactivate With EOImode=1, we can invoke each operation individually. This gives us: <~IRQ> Acknowledge Priority Drop <*other* interrupts can be signaled from here, once interrupts are re-enabled> Deactivate <*this* interrupt can be signaled again> What this means is that with EOImode=1, any interrupt is kept "masked" by its active bit between Priority Drop and Deactivate. Threaded IRQs and ONESHOT ========================= ONESHOT threaded IRQs must remain masked between the main handler and the threaded handler. Right now we do this using the conventional irq_mask() operations, which looks like this: Acknowledge Priority Drop irq_mask() Deactivate irq_unmask() However, masking for the GICs means poking the distributor, and there's no sysreg for that - it's an MMIO access. We've seen above that our IRQ handling can give us masking "for free", and this is what this patch set is all about. It turns the above handling into: Acknowledge Priority Drop Deactivate No irq_mask() => fewer MMIO accesses => happier users (or so I've been told). This is especially relevant to PREEMPT_RT which forces threaded IRQs. Functional testing ================== GICv2 +++++ I've tested this on my Juno with forced irqthreads. This makes the pl011 IRQ into a threaded ONESHOT IRQ, so I spammed my keyboard into the console and verified via ftrace that there were no irq_mask() / irq_unmask() involved. GICv3 +++++ I've tested this on my Ampere eMAG, which uncovered "fun" interactions with the MSI domains. Did the same trick as the Juno with the pl011. pNMIs cause said eMAG to freeze, but that's true even without my patches. I did try them out under QEMU+KVM and that looked fine, although that means I only got to test EOImode=0. I'll try to dig into this when I get some more cycles. Performance impact ================== Benchmark +++++++++ Finding a benchmark that leverages a force-threaded IRQ has proved to be somewhat of a pain, so I crafted my own. It's a bit daft, but so are most benchmarks (though this one might win a prize). Long story short, I'm picking an unused IRQ and have it be force-threaded. The benchmark then is: loop: irq_set_irqchip_state(irq, IRQCHIP_STATE_PENDING, true); wait_for_completion(&done); complete(&done); A more complete picture would be: raise IRQ wait run flow handler wake IRQ thread finish handling wake bench thread Letting this run for a fixed amount of time lets me measure an entire IRQ handling cycle, which is what I'm after since there's one less mask() in the flow handler and one less unmask() in the threaded handler. You'll note there's some potential "noise" in there due to scheduling both the benchmark thread and the IRQ thread. However, the IRQ thread is pinned to the IRQ's affinity, and I also pinned the benchmark thread in my tests, which should keep this noise to a minimum. Results +++++++ On a Juno r0, 20 iterations of 5 seconds of that benchmark yields (measuring irqs/sec): | mean | median | 90th percentile | 99th percentile | |------+--------+-----------------+-----------------| | +11% | +11% | +12% | +14% | On an Ampere eMAG, 20 iterations of 5 seconds of that benchmark yields (measuring irqs/sec): | mean | median | 90th percentile | 99th percentile | |------+--------+-----------------+-----------------| | +20% | +20% | +20% | +20% | This is still quite "artificial", but it reassures me in that skipping those (un)mask operations can yield some measurable improvement. Valentin Schneider (10): genirq: Add chip flag to denote automatic IRQ (un)masking genirq: Define irq_ack() and irq_eoi() helpers genirq: Employ ack_irq() and eoi_irq() where relevant genirq: Add handle_strict_flow_irq() flow handler genirq: Let purely flow-masked ONESHOT irqs through unmask_threaded_irq() genirq: Don't mask IRQ within flow handler if IRQ is flow-masked genirq, irq-gic-v3: Make NMI flow handlers use ->irq_ack() if available irqchip/gic-v3-its: Use irq_chip_ack_parent() irqchip/gic: Convert to handle_strict_flow_irq() irqchip/gic-v3: Convert to handle_strict_flow_irq() drivers/irqchip/irq-gic-v3-its-pci-msi.c | 1 + drivers/irqchip/irq-gic-v3-its.c | 1 + drivers/irqchip/irq-gic-v3.c | 27 +++-- drivers/irqchip/irq-gic.c | 14 ++- include/linux/irq.h | 15 ++- kernel/irq/chip.c | 122 ++++++++++++++++++++--- kernel/irq/debugfs.c | 2 + kernel/irq/internals.h | 7 ++ kernel/irq/manage.c | 2 +- 9 files changed, 159 insertions(+), 32 deletions(-) -- 2.25.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel