> On 07-Mar-2022, at 8:21 PM, Nicholas Piggin wrote: > > When a synchronous interrupt[1] is taken in a local_irq_disable() region > which has MSR[EE]=1, the interrupt handler will enable MSR[EE] as part > of enabling MSR[RI], for peformance and profiling reasons. > > [1] Typically a hash fault, but in error cases this could be a page > fault or facility unavailable as well. > > If an asynchronous interrupt hits here and its masked handler requires > MSR[EE] to be cleared (it is a PACA_IRQ_MUST_HARD_MASK interrupt), then > MSR[EE] must remain disabled until that pending interrupt is replayed. > The problem is that the MSR of the original context has MSR[EE]=1, so > returning directly to that causes MSR[EE] to be enabled while the > interrupt is still pending. > > This issue was hacked around in the interrupt return code by just > clearing the hard mask to avoid a warning, and taking the masked > interrupt again immediately in the return context, which would disable > MSR[EE]. However in the case of a pending PMI, it is possible that it is > not maked in the calling context so the full handler will be run while > there is a PMI pending, and this confuses the perf code and causes > warnings with its PMI pending management. > > Fix this by removing the hack, and adjusting the return MSR if it has > MSR[EE]=1 and there is a PACA_IRQ_MUST_HARD_MASK interrupt pending. > > Fixes: 4423eb5ae32e ("powerpc/64/interrupt: make normal synchronous interrupts enable MSR[EE] if possible") > Signed-off-by: Nicholas Piggin > --- > arch/powerpc/kernel/interrupt.c | 10 --------- > arch/powerpc/kernel/interrupt_64.S | 34 +++++++++++++++++++++++++++--- > 2 files changed, 31 insertions(+), 13 deletions(-) With this patch on top of powerpc/merge following rcu stalls are seen while running powerpc selftests (mitigation-patching) on P9. I don’t see this issue on P10. [ 1841.248838] link-stack-flush: flush disabled. [ 1841.248905] count-cache-flush: software flush enabled. [ 1841.248911] link-stack-flush: software flush enabled. [ 1901.249668] rcu: INFO: rcu_sched self-detected stall on CPU [ 1901.249703] rcu: 12-...!: (5999 ticks this GP) idle=d0f/1/0x4000000000000002 softirq=37019/37027 fqs=0 [ 1901.249720] (t=6000 jiffies g=106273 q=1624) [ 1901.249729] rcu: rcu_sched kthread starved for 6000 jiffies! g106273 f0x0 RCU_GP_WAIT_FQS(5) ->state=0x0 ->cpu=6 [ 1901.249743] rcu: Unless rcu_sched kthread gets sufficient CPU time, OOM is now expected behavior. [ 1901.249752] rcu: RCU grace-period kthread stack dump: [ 1901.249759] task:rcu_sched state:R running task stack: 0 pid: 11 ppid: 2 flags:0x00000800 [ 1901.249775] Call Trace: [ 1901.249781] [c0000000076ab870] [0000000000000001] 0x1 (unreliable) [ 1901.249795] [c0000000076aba60] [c00000000001e508] __switch_to+0x288/0x4a0 [ 1901.249811] [c0000000076abac0] [c000000000d15950] __schedule+0x2c0/0x950 [ 1901.249824] [c0000000076abb80] [c000000000d16048] schedule+0x68/0x130 [ 1901.249836] [c0000000076abbb0] [c000000000d1df1c] schedule_timeout+0x25c/0x3f0 [ 1901.249849] [c0000000076abc90] [c00000000021522c] rcu_gp_fqs_loop+0x2fc/0x3e0 [ 1901.249863] [c0000000076abd40] [c00000000021a0fc] rcu_gp_kthread+0x13c/0x180 [ 1901.249875] [c0000000076abdc0] [c00000000018ce94] kthread+0x124/0x130 [ 1901.249887] [c0000000076abe10] [c00000000000cec0] ret_from_kernel_thread+0x5c/0x64 [ 1901.249900] rcu: Stack dump where RCU GP kthread last ran: [ 1901.249908] Sending NMI from CPU 12 to CPUs 6: [ 1901.249944] NMI backtrace for cpu 6 [ 1901.249957] CPU: 6 PID: 40 Comm: migration/6 Not tainted 5.17.0-rc6-00327-g782b30d101f6-dirty #3 [ 1901.249971] Stopper: multi_cpu_stop+0x0/0x230 <- stop_machine_cpuslocked+0x188/0x1e0 [ 1901.249987] NIP: c000000000d14e0c LR: c000000000214280 CTR: c0000000002914f0 [ 1901.249996] REGS: c00000000785b980 TRAP: 0500 Not tainted (5.17.0-rc6-00327-g782b30d101f6-dirty) [ 1901.250007] MSR: 800000000280b033 CR: 48002822 XER: 00000000 [ 1901.250038] CFAR: 0000000000000000 IRQMASK: 0 [ 1901.250038] GPR00: c00000000029165c c00000000785bc20 c000000002a20000 0000000000000002 [ 1901.250038] GPR04: 0000000000000000 c0000009fb60ab80 c0000009fb60ab70 c00000000001e508 [ 1901.250038] GPR08: 0000000000000000 c0000009fb68f5a8 00000009f94c0000 000000000098967f [ 1901.250038] GPR12: 0000000000000000 c00000001ec57a00 c00000000018cd78 c000000007234f80 [ 1901.250038] GPR16: 0000000000000000 0000000000000000 0000000000000000 0000000000000000 [ 1901.250038] GPR20: 0000000000000000 0000000000000000 0000000000000000 0000000000000001 [ 1901.250038] GPR24: 0000000000000002 0000000000000003 0000000000000000 c000000002a62138 [ 1901.250038] GPR28: c0000000ee70faf8 0000000000000001 c0000000ee70fb1c 0000000000000001 [ 1901.250157] NIP [c000000000d14e0c] rcu_dynticks_inc+0x1c/0x40 [ 1901.250168] LR [c000000000214280] rcu_momentary_dyntick_idle+0x30/0x60 [ 1901.250180] Call Trace: [ 1901.250185] [c00000000785bc20] [c000000007a96738] 0xc000000007a96738 (unreliable) [ 1901.250198] [c00000000785bc40] [c00000000029165c] multi_cpu_stop+0x16c/0x230 [ 1901.250210] [c00000000785bcb0] [c000000000291244] cpu_stopper_thread+0xe4/0x240 [ 1901.250223] [c00000000785bd60] [c000000000193214] smpboot_thread_fn+0x1e4/0x250 [ 1901.250237] [c00000000785bdc0] [c00000000018ce94] kthread+0x124/0x130 [ 1901.250249] [c00000000785be10] [c00000000000cec0] ret_from_kernel_thread+0x5c/0x64 [ 1901.250262] Instruction dump: [ 1901.250269] f821ffa1 4b43b81d 00000000 00000000 00000000 3c4c01d1 3842b210 e94d0030 [ 1901.250290] 3d22ff7b 3929f5a8 7d295214 7c0004ac <7d404828> 7d4a1814 7d40492d 40c2fff4 [ 1901.250313] Sending NMI from CPU 12 to CPUs 0: [ 1901.250328] NMI backtrace for cpu 0 Have attached o/p captured during mitigation-patching test run. Thanks -Sachin