From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E9F7AC433EF for ; Fri, 1 Oct 2021 14:44:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id CEE5761507 for ; Fri, 1 Oct 2021 14:44:39 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1354628AbhJAOqX (ORCPT ); Fri, 1 Oct 2021 10:46:23 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:57424 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231679AbhJAOqU (ORCPT ); Fri, 1 Oct 2021 10:46:20 -0400 Received: from mail-pl1-x62d.google.com (mail-pl1-x62d.google.com [IPv6:2607:f8b0:4864:20::62d]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 264C8C06177C for ; Fri, 1 Oct 2021 07:44:36 -0700 (PDT) Received: by mail-pl1-x62d.google.com with SMTP id j15so6443566plh.7 for ; Fri, 01 Oct 2021 07:44:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ad+1NfwHnbLkN/4DhPh8YfoW0rtkGTB5w67G5jKzAqA=; b=m4AoOUSeWpSf+BmE/E5JE7QAW2tuRX2lPWbrPrr9rFDDwcu+fkJv5fHqWTYvkVAfS9 nyDfJG623FAQ6TzwfPGjIphoAVqWn78S0YkkNvAAb+1PHVvRaSrBRQZfPHsu8FVIv1CT Q51DGGxE2xiabv82hDlYmW68H6KIgUrpAOdy2t7vnat5ZE/huyPGWCW0aiNpgFcIhtz+ qpVWzBO7OuPd2gOHVDwoApKG/vpWkUXTbJm4K1epfsDsV/C7XecY0uYXOa2dkf827S0D AofPgMwSxn2N60kBaOsTOwv4iN7kHPH+41iT3sZ13QeDkVHt45byN15UmonuU2qzpvH3 8P8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ad+1NfwHnbLkN/4DhPh8YfoW0rtkGTB5w67G5jKzAqA=; b=OBTpS+JbFj89qDbMfWh2z4bWtyLFsn4OLXyGbuzawldkBkao5nZSi5jXmSdYQ2s9Ws lBMywJsGedUYnXs5a7dLk+xU6na0+zacR32GBjRI7P6EssOYtwpMvaqt3wXBu28SEBr/ XnvW0x1+ffEYbpdR684bqxVrsYraTlCqeRS6QO+A7RZiAd/U7O5PE6WesxmPzcPtlwjU ex+d79zzvIuar+NCGjoQJtKA2guKzHblQppwgRgcPfBtrbjAMB8SrvWt360MZZcaNc0m N4D0cYWAYQFNcrddqDIXEdmJ38f5doD+05wzaAWjNHSAx12UIqHV3HxlbiWTtbiairGT haaA== X-Gm-Message-State: AOAM533JBbVR5WrYsTZ/4eLCdp7sYml7yp93JzXDMBBKkD3rRIz2ZTKJ bo428cY5X/KotTSdbeNyag== X-Google-Smtp-Source: ABdhPJzRN0EOIHvRUOUv9bkTENMQoAeWuRf06FqsKjZbaWa0u8oeHc3rRZvzDN47mdSXjksUi2ampA== X-Received: by 2002:a17:902:708c:b0:13e:b42:88b5 with SMTP id z12-20020a170902708c00b0013e0b4288b5mr9862002plk.4.1633099475725; Fri, 01 Oct 2021 07:44:35 -0700 (PDT) Received: from piliu.users.ipa.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id z12sm6766203pge.16.2021.10.01.07.44.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Oct 2021 07:44:35 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Mark Rutland , Marc Zyngier , "Paul E. McKenney" , Catalin Marinas , Will Deacon , Joey Gouly , Sami Tolvanen , Julien Thierry , Thomas Gleixner , Yuichi Ito , linux-kernel@vger.kernel.org Subject: [PATCHv4 3/3] arm64: entry: avoid double-accounting IRQ RCU entry Date: Fri, 1 Oct 2021 22:44:06 +0800 Message-Id: <20211001144406.7719-4-kernelfans@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211001144406.7719-1-kernelfans@gmail.com> References: <20211001144406.7719-1-kernelfans@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org When an IRQ is taken, some accounting needs to be performed to enter and exit IRQ context around the IRQ handler. On arm64 some of this accounting is performed by both the architecture code and the IRQ domain code, resulting in calling rcu_irq_enter() twice per exception entry, violating the expectations of the core RCU code, and resulting in failing to identify quiescent periods correctly (e.g. in rcu_is_cpu_rrupt_from_idle()). To fix this, we must perform all the accounting from the architecture code. We prevent the IRQ domain code from performing any accounting by selecting HAVE_ARCH_IRQENTRY, and must call irq_enter_rcu() and irq_exit_rcu() around invoking the root IRQ handler. When we take a pNMI from a context with IRQs disabled, we'll perform the necessary accounting as part of arm64_enter_nmi() and arm64_exit_nmi(), and should only call irq_enter_rcu() and irq_exit_rcu() when we may have taken a regular interrupt. Signed-off-by: Pingfan Liu Reviewed-by: Mark Rutland Reviewed-by: Marc Zyngier Cc: "Paul E. McKenney" Cc: Catalin Marinas Cc: Will Deacon Cc: Joey Gouly Cc: Sami Tolvanen Cc: Julien Thierry Cc: Thomas Gleixner Cc: Yuichi Ito Cc: linux-kernel@vger.kernel.org To: linux-arm-kernel@lists.infradead.org --- arch/arm64/Kconfig | 1 + arch/arm64/kernel/entry-common.c | 4 ++++ 2 files changed, 5 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 5c7ae4c3954b..c0ba052116b2 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -144,6 +144,7 @@ config ARM64 select HAVE_ARCH_BITREVERSE select HAVE_ARCH_COMPILER_H select HAVE_ARCH_HUGE_VMAP + select HAVE_ARCH_IRQENTRY select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index fecf046f0708..af931e63dc59 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -428,7 +428,9 @@ static __always_inline void __el1_interrupt(struct pt_regs *regs, void (*handler)(struct pt_regs *)) { enter_from_kernel_mode(regs); + irq_enter_rcu(); do_interrupt_handler(regs, handler); + irq_exit_rcu(); /* * Note: thread_info::preempt_count includes both thread_info::count * and thread_info::need_resched, and is not equivalent to @@ -666,7 +668,9 @@ static void noinstr el0_interrupt(struct pt_regs *regs, if (regs->pc & BIT(55)) arm64_apply_bp_hardening(); + irq_enter_rcu(); do_interrupt_handler(regs, handler); + irq_exit_rcu(); exit_to_user_mode(regs); } -- 2.31.1 From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id A05EAC433EF for ; Fri, 1 Oct 2021 14:47:39 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 5D76361A55 for ; Fri, 1 Oct 2021 14:47:39 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 5D76361A55 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To: Message-Id:Date:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=7s67LHpaJ8JQLrTtNZxBb62pL4TA49QuW2P5flPV4L0=; b=mdWdf2xRS0nJ+l TXKkkuHniC0oM3QNcFO2mIj3eVqccsKZTrHOQ+KT5DnNb68UkI9M0hg2CefGnA4Js6thfMChXtgPh XUQfYfO5TbV5uGLKiNATv9av5/wyVxnanHocazmzMy2Lk1wBt/mNZtk/+OX99bfbsYePmPzXI6eyH PdmzQUZtyfK/rd19Ir7uz+WPkzeG0RX31OMYhBEuipU3gVMhcYueYo36KTeNMMfz0eVs84ehG8OiZ KxLqfvAjKl619sKjl7C8O8xzlHSGyseSepIJSzq9uwHimGj9mW1sxe4Ec+6KbxDtzAkmwWuvNBpQP pJvoRRhBz51vtalxYwqA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mWJmZ-000baR-4L; Fri, 01 Oct 2021 14:45:03 +0000 Received: from mail-pl1-x630.google.com ([2607:f8b0:4864:20::630]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mWJm9-000bUd-5z for linux-arm-kernel@lists.infradead.org; Fri, 01 Oct 2021 14:44:38 +0000 Received: by mail-pl1-x630.google.com with SMTP id b22so6460725pls.1 for ; Fri, 01 Oct 2021 07:44:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=ad+1NfwHnbLkN/4DhPh8YfoW0rtkGTB5w67G5jKzAqA=; b=m4AoOUSeWpSf+BmE/E5JE7QAW2tuRX2lPWbrPrr9rFDDwcu+fkJv5fHqWTYvkVAfS9 nyDfJG623FAQ6TzwfPGjIphoAVqWn78S0YkkNvAAb+1PHVvRaSrBRQZfPHsu8FVIv1CT Q51DGGxE2xiabv82hDlYmW68H6KIgUrpAOdy2t7vnat5ZE/huyPGWCW0aiNpgFcIhtz+ qpVWzBO7OuPd2gOHVDwoApKG/vpWkUXTbJm4K1epfsDsV/C7XecY0uYXOa2dkf827S0D AofPgMwSxn2N60kBaOsTOwv4iN7kHPH+41iT3sZ13QeDkVHt45byN15UmonuU2qzpvH3 8P8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=ad+1NfwHnbLkN/4DhPh8YfoW0rtkGTB5w67G5jKzAqA=; b=UkFVmX+DILZGFLIpbvbP1j4PunXGH/7FWtFIZOF+wpW5JN6FdSvsRWjRAyfvvw6Z8a ExgqPYJQaaESFhlROICIwQdh5aXkivCeFLEQISvOO3QBGhD6SZQ0lWQLxU6nfUlXwAfe +vte5lchUDjE3p6CQcEjzUElCKLvu8yalKI59FJj3lejXWkylOPsqsrRDKLlyW7aVYAt GxIOly1k5teQNFHDK/y5XTOo8um7xs6i/icKHrggHsy+6t1mKPk1G37j9Zh+2jOZjjZU bB6oydmBJrcmV4iHX6cN4y0VdqkIcLFQlZtrP705tGc9A4P/Vr5sMuZFlSltiZT+Ak+o dYNA== X-Gm-Message-State: AOAM533c5cOOwwxOHSd+hXK10hEALgsvhu+2j6i36Gv7qrAIBTdIcEGh 6rtGsbmExifM8BM1t20vW94UQ/cXDA== X-Google-Smtp-Source: ABdhPJzRN0EOIHvRUOUv9bkTENMQoAeWuRf06FqsKjZbaWa0u8oeHc3rRZvzDN47mdSXjksUi2ampA== X-Received: by 2002:a17:902:708c:b0:13e:b42:88b5 with SMTP id z12-20020a170902708c00b0013e0b4288b5mr9862002plk.4.1633099475725; Fri, 01 Oct 2021 07:44:35 -0700 (PDT) Received: from piliu.users.ipa.redhat.com ([209.132.188.80]) by smtp.gmail.com with ESMTPSA id z12sm6766203pge.16.2021.10.01.07.44.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 01 Oct 2021 07:44:35 -0700 (PDT) From: Pingfan Liu To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu , Mark Rutland , Marc Zyngier , "Paul E. McKenney" , Catalin Marinas , Will Deacon , Joey Gouly , Sami Tolvanen , Julien Thierry , Thomas Gleixner , Yuichi Ito , linux-kernel@vger.kernel.org Subject: [PATCHv4 3/3] arm64: entry: avoid double-accounting IRQ RCU entry Date: Fri, 1 Oct 2021 22:44:06 +0800 Message-Id: <20211001144406.7719-4-kernelfans@gmail.com> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20211001144406.7719-1-kernelfans@gmail.com> References: <20211001144406.7719-1-kernelfans@gmail.com> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211001_074437_247525_34249204 X-CRM114-Status: GOOD ( 16.83 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org When an IRQ is taken, some accounting needs to be performed to enter and exit IRQ context around the IRQ handler. On arm64 some of this accounting is performed by both the architecture code and the IRQ domain code, resulting in calling rcu_irq_enter() twice per exception entry, violating the expectations of the core RCU code, and resulting in failing to identify quiescent periods correctly (e.g. in rcu_is_cpu_rrupt_from_idle()). To fix this, we must perform all the accounting from the architecture code. We prevent the IRQ domain code from performing any accounting by selecting HAVE_ARCH_IRQENTRY, and must call irq_enter_rcu() and irq_exit_rcu() around invoking the root IRQ handler. When we take a pNMI from a context with IRQs disabled, we'll perform the necessary accounting as part of arm64_enter_nmi() and arm64_exit_nmi(), and should only call irq_enter_rcu() and irq_exit_rcu() when we may have taken a regular interrupt. Signed-off-by: Pingfan Liu Reviewed-by: Mark Rutland Reviewed-by: Marc Zyngier Cc: "Paul E. McKenney" Cc: Catalin Marinas Cc: Will Deacon Cc: Joey Gouly Cc: Sami Tolvanen Cc: Julien Thierry Cc: Thomas Gleixner Cc: Yuichi Ito Cc: linux-kernel@vger.kernel.org To: linux-arm-kernel@lists.infradead.org --- arch/arm64/Kconfig | 1 + arch/arm64/kernel/entry-common.c | 4 ++++ 2 files changed, 5 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 5c7ae4c3954b..c0ba052116b2 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -144,6 +144,7 @@ config ARM64 select HAVE_ARCH_BITREVERSE select HAVE_ARCH_COMPILER_H select HAVE_ARCH_HUGE_VMAP + select HAVE_ARCH_IRQENTRY select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index fecf046f0708..af931e63dc59 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -428,7 +428,9 @@ static __always_inline void __el1_interrupt(struct pt_regs *regs, void (*handler)(struct pt_regs *)) { enter_from_kernel_mode(regs); + irq_enter_rcu(); do_interrupt_handler(regs, handler); + irq_exit_rcu(); /* * Note: thread_info::preempt_count includes both thread_info::count * and thread_info::need_resched, and is not equivalent to @@ -666,7 +668,9 @@ static void noinstr el0_interrupt(struct pt_regs *regs, if (regs->pc & BIT(55)) arm64_apply_bp_hardening(); + irq_enter_rcu(); do_interrupt_handler(regs, handler); + irq_exit_rcu(); exit_to_user_mode(regs); } -- 2.31.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel