From: Nicholas Piggin <npiggin@gmail.com>
To: linuxppc-dev@lists.ozlabs.org
Cc: Nicholas Piggin <npiggin@gmail.com>
Subject: [RFC PATCH 8/9] powerpc/64s: interrupt soft-enable race fix
Date: Sat, 7 Nov 2020 01:59:28 +1000 [thread overview]
Message-ID: <20201106155929.2246055-9-npiggin@gmail.com> (raw)
In-Reply-To: <20201106155929.2246055-1-npiggin@gmail.com>
Prevent interrupt restore from allowing racing hard interrupts going
ahead of previous soft-pending ones, by using the soft-masked restart
handler to allow a store to clear the soft-mask while knowing nothing
is soft-pending.
This probably doesn't matter much in practice, but it's a simple
demonstrator / test case to exercise the restart table logic.
XXX: 64e can't do this unless it adds restart table support
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
---
arch/powerpc/kernel/irq.c | 79 +++++++++++++++++++++++++--------------
1 file changed, 50 insertions(+), 29 deletions(-)
diff --git a/arch/powerpc/kernel/irq.c b/arch/powerpc/kernel/irq.c
index c8185f709d26..9671468b2c51 100644
--- a/arch/powerpc/kernel/irq.c
+++ b/arch/powerpc/kernel/irq.c
@@ -317,52 +317,73 @@ notrace void arch_local_irq_restore(unsigned long mask)
unsigned char irq_happened;
/* Write the new soft-enabled value */
- irq_soft_mask_set(mask);
- if (mask)
+ if (mask) {
+ irq_soft_mask_set(mask);
return;
+ }
/*
- * From this point onward, we can take interrupts, preempt,
- * etc... unless we got hard-disabled. We check if an event
- * happened. If none happened, we know we can just return.
- *
- * We may have preempted before the check below, in which case
- * we are checking the "new" CPU instead of the old one. This
- * is only a problem if an event happened on the "old" CPU.
+ * After the stb, interrupts are unmasked and there are no interrupts
+ * pending replay. The restart sequence makes this atomic with
+ * respect to soft-masked interrupts. If this was just a simple code
+ * sequence, a soft-masked interrupt could become pending right after
+ * the comparison and before the stb.
*
- * External interrupt events will have caused interrupts to
- * be hard-disabled, so there is no problem, we
- * cannot have preempted.
+ * This allows interrupts to be unmasked without hard disabling, and
+ * also without new hard interrupts coming in ahead of pending ones.
*/
+ asm_volatile_goto(
+"1: \n"
+" lbz 9,%0(13) \n"
+" cmpwi 9,0 \n"
+" bne %l[happened] \n"
+" stb 9,%1(13) \n"
+"2: \n"
+ RESTART_TABLE(1b, 2b, 1b)
+ : : "i" (offsetof(struct paca_struct, irq_happened)),
+ "i" (offsetof(struct paca_struct, irq_soft_mask))
+ : "cr0", "r9"
+ : happened);
+
+ if (IS_ENABLED(CONFIG_PPC_IRQ_SOFT_MASK_DEBUG))
+ WARN_ON_ONCE(!(mfmsr() & MSR_EE));
+
+ return;
+
+happened:
irq_happened = get_irq_happened();
- if (!irq_happened) {
+ if (IS_ENABLED(CONFIG_PPC_IRQ_SOFT_MASK_DEBUG))
+ WARN_ON_ONCE(!irq_happened);
+
+ if (irq_happened == PACA_IRQ_HARD_DIS) {
if (IS_ENABLED(CONFIG_PPC_IRQ_SOFT_MASK_DEBUG))
- WARN_ON_ONCE(!(mfmsr() & MSR_EE));
+ WARN_ON_ONCE(mfmsr() & MSR_EE);
+ irq_soft_mask_set(IRQS_ENABLED);
+ local_paca->irq_happened = 0;
+ __hard_irq_enable();
return;
}
- /* We need to hard disable to replay. */
+ /* Have interrupts to replay, need to hard disable first */
if (!(irq_happened & PACA_IRQ_HARD_DIS)) {
- if (IS_ENABLED(CONFIG_PPC_IRQ_SOFT_MASK_DEBUG))
- WARN_ON_ONCE(!(mfmsr() & MSR_EE));
+ if (IS_ENABLED(CONFIG_PPC_IRQ_SOFT_MASK_DEBUG)) {
+ if (!(mfmsr() & MSR_EE)) {
+ /*
+ * An interrupt could have come in and cleared
+ * MSR[EE] and set IRQ_HARD_DIS, so check
+ * IRQ_HARD_DIS again and warn if it is still
+ * clear.
+ */
+ irq_happened = get_irq_happened();
+ WARN_ON_ONCE(!(irq_happened & PACA_IRQ_HARD_DIS));
+ }
+ }
__hard_irq_disable();
} else {
- /*
- * We should already be hard disabled here. We had bugs
- * where that wasn't the case so let's dbl check it and
- * warn if we are wrong. Only do that when IRQ tracing
- * is enabled as mfmsr() can be costly.
- */
if (IS_ENABLED(CONFIG_PPC_IRQ_SOFT_MASK_DEBUG)) {
if (WARN_ON_ONCE(mfmsr() & MSR_EE))
__hard_irq_disable();
}
-
- if (irq_happened == PACA_IRQ_HARD_DIS) {
- local_paca->irq_happened = 0;
- __hard_irq_enable();
- return;
- }
}
/*
--
2.23.0
next prev parent reply other threads:[~2020-11-06 16:18 UTC|newest]
Thread overview: 14+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-11-06 15:59 [RFC PATCH 0/9] powerpc/64s: fast interrupt exit Nicholas Piggin
2020-11-06 15:59 ` [RFC PATCH 1/9] powerpc/64s: syscall real mode entry use mtmsrd rather than rfid Nicholas Piggin
2020-11-06 15:59 ` [RFC PATCH 2/9] powerpc/64s: system call avoid setting MSR[RI] until we set MSR[EE] Nicholas Piggin
2020-11-06 15:59 ` [RFC PATCH 3/9] powerpc/64s: introduce different functions to return from SRR vs HSRR interrupts Nicholas Piggin
2020-11-06 15:59 ` [RFC PATCH 4/9] powerpc/64s: avoid reloading (H)SRR registers if they are still valid Nicholas Piggin
2020-11-06 15:59 ` [RFC PATCH 5/9] powerpc/64: move interrupt return asm to interrupt_64.S Nicholas Piggin
2020-11-06 15:59 ` [RFC PATCH 6/9] powerpc/64s: save one more register in the masked interrupt handler Nicholas Piggin
2020-11-06 15:59 ` [RFC PATCH 7/9] powerpc/64s: allow alternate return locations for soft-masked interrupts Nicholas Piggin
2020-11-06 15:59 ` Nicholas Piggin [this message]
2020-11-06 15:59 ` [RFC PATCH 9/9] powerpc/64s: use interrupt restart table to speed up return from interrupt Nicholas Piggin
2020-11-07 10:35 ` [RFC PATCH 0/9] powerpc/64s: fast interrupt exit Christophe Leroy
2020-11-10 8:49 ` Nicholas Piggin
2020-11-10 11:31 ` Christophe Leroy
2020-11-11 4:49 ` Nicholas Piggin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20201106155929.2246055-9-npiggin@gmail.com \
--to=npiggin@gmail.com \
--cc=linuxppc-dev@lists.ozlabs.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).