From: Pingfan Liu <kernelfans@gmail.com> To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu <kernelfans@gmail.com>, Mark Rutland <mark.rutland@arm.com>, Marc Zyngier <maz@kernel.org>, "Paul E. McKenney" <paulmck@kernel.org>, Catalin Marinas <catalin.marinas@arm.com>, Will Deacon <will@kernel.org>, Joey Gouly <joey.gouly@arm.com>, Sami Tolvanen <samitolvanen@google.com>, Julien Thierry <julien.thierry@arm.com>, Thomas Gleixner <tglx@linutronix.de>, Yuichi Ito <ito-yuichi@fujitsu.com>, linux-kernel@vger.kernel.org Subject: [PATCHv4 3/3] arm64: entry: avoid double-accounting IRQ RCU entry Date: Fri, 1 Oct 2021 22:44:06 +0800 [thread overview] Message-ID: <20211001144406.7719-4-kernelfans@gmail.com> (raw) In-Reply-To: <20211001144406.7719-1-kernelfans@gmail.com> When an IRQ is taken, some accounting needs to be performed to enter and exit IRQ context around the IRQ handler. On arm64 some of this accounting is performed by both the architecture code and the IRQ domain code, resulting in calling rcu_irq_enter() twice per exception entry, violating the expectations of the core RCU code, and resulting in failing to identify quiescent periods correctly (e.g. in rcu_is_cpu_rrupt_from_idle()). To fix this, we must perform all the accounting from the architecture code. We prevent the IRQ domain code from performing any accounting by selecting HAVE_ARCH_IRQENTRY, and must call irq_enter_rcu() and irq_exit_rcu() around invoking the root IRQ handler. When we take a pNMI from a context with IRQs disabled, we'll perform the necessary accounting as part of arm64_enter_nmi() and arm64_exit_nmi(), and should only call irq_enter_rcu() and irq_exit_rcu() when we may have taken a regular interrupt. Signed-off-by: Pingfan Liu <kernelfans@gmail.com> Reviewed-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Marc Zyngier <maz@kernel.org> Cc: "Paul E. McKenney" <paulmck@kernel.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will@kernel.org> Cc: Joey Gouly <joey.gouly@arm.com> Cc: Sami Tolvanen <samitolvanen@google.com> Cc: Julien Thierry <julien.thierry@arm.com> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Yuichi Ito <ito-yuichi@fujitsu.com> Cc: linux-kernel@vger.kernel.org To: linux-arm-kernel@lists.infradead.org --- arch/arm64/Kconfig | 1 + arch/arm64/kernel/entry-common.c | 4 ++++ 2 files changed, 5 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 5c7ae4c3954b..c0ba052116b2 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -144,6 +144,7 @@ config ARM64 select HAVE_ARCH_BITREVERSE select HAVE_ARCH_COMPILER_H select HAVE_ARCH_HUGE_VMAP + select HAVE_ARCH_IRQENTRY select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index fecf046f0708..af931e63dc59 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -428,7 +428,9 @@ static __always_inline void __el1_interrupt(struct pt_regs *regs, void (*handler)(struct pt_regs *)) { enter_from_kernel_mode(regs); + irq_enter_rcu(); do_interrupt_handler(regs, handler); + irq_exit_rcu(); /* * Note: thread_info::preempt_count includes both thread_info::count * and thread_info::need_resched, and is not equivalent to @@ -666,7 +668,9 @@ static void noinstr el0_interrupt(struct pt_regs *regs, if (regs->pc & BIT(55)) arm64_apply_bp_hardening(); + irq_enter_rcu(); do_interrupt_handler(regs, handler); + irq_exit_rcu(); exit_to_user_mode(regs); } -- 2.31.1
WARNING: multiple messages have this Message-ID (diff)
From: Pingfan Liu <kernelfans@gmail.com> To: linux-arm-kernel@lists.infradead.org Cc: Pingfan Liu <kernelfans@gmail.com>, Mark Rutland <mark.rutland@arm.com>, Marc Zyngier <maz@kernel.org>, "Paul E. McKenney" <paulmck@kernel.org>, Catalin Marinas <catalin.marinas@arm.com>, Will Deacon <will@kernel.org>, Joey Gouly <joey.gouly@arm.com>, Sami Tolvanen <samitolvanen@google.com>, Julien Thierry <julien.thierry@arm.com>, Thomas Gleixner <tglx@linutronix.de>, Yuichi Ito <ito-yuichi@fujitsu.com>, linux-kernel@vger.kernel.org Subject: [PATCHv4 3/3] arm64: entry: avoid double-accounting IRQ RCU entry Date: Fri, 1 Oct 2021 22:44:06 +0800 [thread overview] Message-ID: <20211001144406.7719-4-kernelfans@gmail.com> (raw) In-Reply-To: <20211001144406.7719-1-kernelfans@gmail.com> When an IRQ is taken, some accounting needs to be performed to enter and exit IRQ context around the IRQ handler. On arm64 some of this accounting is performed by both the architecture code and the IRQ domain code, resulting in calling rcu_irq_enter() twice per exception entry, violating the expectations of the core RCU code, and resulting in failing to identify quiescent periods correctly (e.g. in rcu_is_cpu_rrupt_from_idle()). To fix this, we must perform all the accounting from the architecture code. We prevent the IRQ domain code from performing any accounting by selecting HAVE_ARCH_IRQENTRY, and must call irq_enter_rcu() and irq_exit_rcu() around invoking the root IRQ handler. When we take a pNMI from a context with IRQs disabled, we'll perform the necessary accounting as part of arm64_enter_nmi() and arm64_exit_nmi(), and should only call irq_enter_rcu() and irq_exit_rcu() when we may have taken a regular interrupt. Signed-off-by: Pingfan Liu <kernelfans@gmail.com> Reviewed-by: Mark Rutland <mark.rutland@arm.com> Reviewed-by: Marc Zyngier <maz@kernel.org> Cc: "Paul E. McKenney" <paulmck@kernel.org> Cc: Catalin Marinas <catalin.marinas@arm.com> Cc: Will Deacon <will@kernel.org> Cc: Joey Gouly <joey.gouly@arm.com> Cc: Sami Tolvanen <samitolvanen@google.com> Cc: Julien Thierry <julien.thierry@arm.com> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: Yuichi Ito <ito-yuichi@fujitsu.com> Cc: linux-kernel@vger.kernel.org To: linux-arm-kernel@lists.infradead.org --- arch/arm64/Kconfig | 1 + arch/arm64/kernel/entry-common.c | 4 ++++ 2 files changed, 5 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 5c7ae4c3954b..c0ba052116b2 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -144,6 +144,7 @@ config ARM64 select HAVE_ARCH_BITREVERSE select HAVE_ARCH_COMPILER_H select HAVE_ARCH_HUGE_VMAP + select HAVE_ARCH_IRQENTRY select HAVE_ARCH_JUMP_LABEL select HAVE_ARCH_JUMP_LABEL_RELATIVE select HAVE_ARCH_KASAN if !(ARM64_16K_PAGES && ARM64_VA_BITS_48) diff --git a/arch/arm64/kernel/entry-common.c b/arch/arm64/kernel/entry-common.c index fecf046f0708..af931e63dc59 100644 --- a/arch/arm64/kernel/entry-common.c +++ b/arch/arm64/kernel/entry-common.c @@ -428,7 +428,9 @@ static __always_inline void __el1_interrupt(struct pt_regs *regs, void (*handler)(struct pt_regs *)) { enter_from_kernel_mode(regs); + irq_enter_rcu(); do_interrupt_handler(regs, handler); + irq_exit_rcu(); /* * Note: thread_info::preempt_count includes both thread_info::count * and thread_info::need_resched, and is not equivalent to @@ -666,7 +668,9 @@ static void noinstr el0_interrupt(struct pt_regs *regs, if (regs->pc & BIT(55)) arm64_apply_bp_hardening(); + irq_enter_rcu(); do_interrupt_handler(regs, handler); + irq_exit_rcu(); exit_to_user_mode(regs); } -- 2.31.1 _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel
next prev parent reply other threads:[~2021-10-01 14:44 UTC|newest] Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top 2021-10-01 14:44 [PATCHv4 0/3] arm64/irqentry: remove duplicate housekeeping of rcu Pingfan Liu 2021-10-01 14:44 ` Pingfan Liu 2021-10-01 14:44 ` [PATCHv4 1/3] kernel/irq: make irq_{enter,exit}() in handle_domain_irq() arch optional Pingfan Liu 2021-10-01 14:44 ` [PATCHv4 1/3] kernel/irq: make irq_{enter, exit}() " Pingfan Liu 2021-10-04 16:30 ` [PATCHv4 1/3] kernel/irq: make irq_{enter,exit}() " Marc Zyngier 2021-10-04 16:30 ` [PATCHv4 1/3] kernel/irq: make irq_{enter, exit}() " Marc Zyngier 2021-10-01 14:44 ` [PATCHv4 2/3] arm64: entry: refactor EL1 interrupt entry logic Pingfan Liu 2021-10-01 14:44 ` Pingfan Liu 2021-10-01 14:44 ` Pingfan Liu [this message] 2021-10-01 14:44 ` [PATCHv4 3/3] arm64: entry: avoid double-accounting IRQ RCU entry Pingfan Liu 2021-10-04 16:31 ` [PATCHv4 0/3] arm64/irqentry: remove duplicate housekeeping of rcu Marc Zyngier 2021-10-04 16:31 ` Marc Zyngier 2021-10-05 17:17 ` Catalin Marinas 2021-10-05 17:17 ` Catalin Marinas
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=20211001144406.7719-4-kernelfans@gmail.com \ --to=kernelfans@gmail.com \ --cc=catalin.marinas@arm.com \ --cc=ito-yuichi@fujitsu.com \ --cc=joey.gouly@arm.com \ --cc=julien.thierry@arm.com \ --cc=linux-arm-kernel@lists.infradead.org \ --cc=linux-kernel@vger.kernel.org \ --cc=mark.rutland@arm.com \ --cc=maz@kernel.org \ --cc=paulmck@kernel.org \ --cc=samitolvanen@google.com \ --cc=tglx@linutronix.de \ --cc=will@kernel.org \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.